Shelley Ross1, Deena Hamza2, Rosslynn Zulla3, Samantha Stasiuk4, Darren Nichols5. 1. is Professor, Department of Family Medicine, University of Alberta, Edmonton, AB, Canada. 2. is Competency-Based Medical Education Evaluation Lead for Postgraduate Medical Education, University of Alberta, Edmonton, AB, Canada. 3. is a Specialist/Advisor, Faculty of Social Work, University of Calgary, AB, Canada. 4. is Clinical Assistant Professor, Department of Family Practice, University of British Columbia, BC, Canada. 5. is Associate Professor, Department of Family Medicine, University of Alberta, Edmonton, AB, Canada.
Abstract
BACKGROUND: Narrative feedback, like verbal feedback, is essential to learning. Regardless of form, all feedback should be of high quality. This is becoming even more important as programs incorporate narrative feedback into the constellation of evidence used for summative decision-making. Continuously improving the quality of narrative feedback requires tools for evaluating it, and time to score. A tool is needed that does not require clinical educator expertise so scoring can be delegated to others. OBJECTIVE: To develop an evidence-based tool to evaluate the quality of documented feedback that could be reliably used by clinical educators and non-experts. METHODS: Following a literature review to identify elements of high-quality feedback, an expert consensus panel developed the scoring tool. Messick's unified concept of construct validity guided the collection of validity evidence throughout development and piloting (2013-2020). RESULTS: The Evaluation of Feedback Captured Tool (EFeCT) contains 5 categories considered to be essential elements of high-quality feedback. Preliminary validity evidence supports content, substantive, and consequential validity facets. Generalizability evidence supports that EFeCT scores assigned to feedback samples show consistent interrater reliability scores between raters across 5 sessions, regardless of level of medical education or clinical expertise (Session 1: n=3, ICC=0.94; Session 2: n=6, ICC=0.90; Session 3: n=5, ICC=0.91; Session 4: n=6, ICC=0.89; Session 5: n=6, ICC=0.92). CONCLUSIONS: There is preliminary validity evidence for the EFeCT as a useful tool for scoring the quality of documented feedback captured on assessment forms. Generalizability evidence indicated comparable EFeCT scores by raters regardless of level of expertise.
BACKGROUND: Narrative feedback, like verbal feedback, is essential to learning. Regardless of form, all feedback should be of high quality. This is becoming even more important as programs incorporate narrative feedback into the constellation of evidence used for summative decision-making. Continuously improving the quality of narrative feedback requires tools for evaluating it, and time to score. A tool is needed that does not require clinical educator expertise so scoring can be delegated to others. OBJECTIVE: To develop an evidence-based tool to evaluate the quality of documented feedback that could be reliably used by clinical educators and non-experts. METHODS: Following a literature review to identify elements of high-quality feedback, an expert consensus panel developed the scoring tool. Messick's unified concept of construct validity guided the collection of validity evidence throughout development and piloting (2013-2020). RESULTS: The Evaluation of Feedback Captured Tool (EFeCT) contains 5 categories considered to be essential elements of high-quality feedback. Preliminary validity evidence supports content, substantive, and consequential validity facets. Generalizability evidence supports that EFeCT scores assigned to feedback samples show consistent interrater reliability scores between raters across 5 sessions, regardless of level of medical education or clinical expertise (Session 1: n=3, ICC=0.94; Session 2: n=6, ICC=0.90; Session 3: n=5, ICC=0.91; Session 4: n=6, ICC=0.89; Session 5: n=6, ICC=0.92). CONCLUSIONS: There is preliminary validity evidence for the EFeCT as a useful tool for scoring the quality of documented feedback captured on assessment forms. Generalizability evidence indicated comparable EFeCT scores by raters regardless of level of expertise.
Authors: Margaret L Boehler; David A Rogers; Cathy J Schwind; Ruth Mayforth; Jacquelyn Quin; Reed G Williams; Gary Dunnington Journal: Med Educ Date: 2006-08 Impact factor: 6.251
Authors: Peter Harris; Farhan Bhanji; Maureen Topps; Shelley Ross; Steven Lieberman; Jason R Frank; Linda Snell; Jonathan Sherbino Journal: Med Teach Date: 2017-06 Impact factor: 3.650
Authors: Jocelyn Lockyer; Carol Carraccio; Ming-Ka Chan; Danielle Hart; Sydney Smee; Claire Touchie; Eric S Holmboe; Jason R Frank Journal: Med Teach Date: 2017-06 Impact factor: 3.650
Authors: Shiphra Ginsburg; Christopher J Watling; Daniel J Schumacher; Andrea Gingerich; Rose Hatala Journal: Acad Med Date: 2021-07-01 Impact factor: 6.893
Authors: Suzanne Schut; Lauren A Maggio; Sylvia Heeneman; Jan van Tartwijk; Cees van der Vleuten; Erik Driessen Journal: Perspect Med Educ Date: 2020-10-21