Sonja E Raaum1, Katie Lappe2, Jorie M Colbert-Getz3, Caroline K Milne2. 1. Department of Internal Medicine, School of Medicine and George E. Whalen VA Hospital, University of Utah, Salt Lake City, UT, USA. sonja.raaum@hsc.utah.edu. 2. Department of Internal Medicine, School of Medicine and George E. Whalen VA Hospital, University of Utah, Salt Lake City, UT, USA. 3. Department of Internal Medicine, School of Medicine, University of Utah, Salt Lake City, UT, USA.
Abstract
BACKGROUND: Feedback is a critical element of graduate medical education. Narrative comments on evaluation forms are a source of feedback for residents. As a shared mental model for performance, milestone-based evaluations may impact narrative comments and resident perception of feedback. OBJECTIVE: To determine if milestone-based evaluations impacted the quality of faculty members' narrative comments on evaluations and, as an extension, residents' perception of feedback. DESIGN: Concurrent mixed methods study, including qualitative analysis of narrative comments and survey of resident perception of feedback. PARTICIPANTS: Seventy internal medicine residents and their faculty evaluators at the University of Utah. APPROACH: Faculty narrative comments from 248 evaluations pre- and post-milestone implementation were analyzed for quality and Accreditation Council for Graduate Medical Education competency by area of strength and area for improvement. Seventy residents were surveyed regarding quality of feedback pre- and post-milestone implementation. KEY RESULTS: Qualitative analysis of narrative comments revealed nearly all evaluations pre- and post-milestone implementation included comments about areas of strength but were frequently vague and not related to competencies. Few evaluations included narrative comments on areas for improvement, but these were of higher quality compared to areas of strength (p < 0.001). Overall resident perception of quality of narrative comments was low and did not change following milestone implementation (p = 0.562) for the 86% of residents (N = 60/70) who completed the pre- and post-surveys. CONCLUSIONS: The quality of narrative comments was poor, and there was no evidence of improved quality following introduction of milestone-based evaluations. Comments on areas for improvement were of higher quality than areas of strength, suggesting an area for targeted intervention. Residents' perception of feedback quality did not change following implementation of milestone-based evaluations, suggesting that in the post-milestone era, internal medicine educators need to utilize additional interventions to improve quality of feedback.
BACKGROUND: Feedback is a critical element of graduate medical education. Narrative comments on evaluation forms are a source of feedback for residents. As a shared mental model for performance, milestone-based evaluations may impact narrative comments and resident perception of feedback. OBJECTIVE: To determine if milestone-based evaluations impacted the quality of faculty members' narrative comments on evaluations and, as an extension, residents' perception of feedback. DESIGN: Concurrent mixed methods study, including qualitative analysis of narrative comments and survey of resident perception of feedback. PARTICIPANTS: Seventy internal medicine residents and their faculty evaluators at the University of Utah. APPROACH: Faculty narrative comments from 248 evaluations pre- and post-milestone implementation were analyzed for quality and Accreditation Council for Graduate Medical Education competency by area of strength and area for improvement. Seventy residents were surveyed regarding quality of feedback pre- and post-milestone implementation. KEY RESULTS: Qualitative analysis of narrative comments revealed nearly all evaluations pre- and post-milestone implementation included comments about areas of strength but were frequently vague and not related to competencies. Few evaluations included narrative comments on areas for improvement, but these were of higher quality compared to areas of strength (p < 0.001). Overall resident perception of quality of narrative comments was low and did not change following milestone implementation (p = 0.562) for the 86% of residents (N = 60/70) who completed the pre- and post-surveys. CONCLUSIONS: The quality of narrative comments was poor, and there was no evidence of improved quality following introduction of milestone-based evaluations. Comments on areas for improvement were of higher quality than areas of strength, suggesting an area for targeted intervention. Residents' perception of feedback quality did not change following implementation of milestone-based evaluations, suggesting that in the post-milestone era, internal medicine educators need to utilize additional interventions to improve quality of feedback.
Entities:
Keywords:
graduate medical education; medical education qualitative methods; milestones
Authors: Eva Aagaard; Gregory C Kane; Lisa Conforti; Sarah Hood; Kelly J Caverzagie; Cynthia Smith; Davoren A Chick; Eric S Holmboe; William F Iobst Journal: J Grad Med Educ Date: 2013-09
Authors: Christopher Nabors; Stephen J Peterson; Leanne Forman; Gary W Stallings; Arif Mumtaz; Sachin Sule; Tushar Shah; Wilbert Aronow; Lawrence Delorenzo; Dipak Chandy; Stuart G Lehrman; William H Frishman; Eric Holmboe Journal: J Grad Med Educ Date: 2013-03
Authors: Jennifer R Kogan; Lisa N Conforti; Elizabeth C Bernabeo; Steven J Durning; Karen E Hauer; Eric S Holmboe Journal: Med Educ Date: 2012-02 Impact factor: 6.251