BACKGROUND: Rater errors, such as halo/reverse halo, range restriction, and leniency errors, are frequently cited as threats to the validity of resident assessment by faculty. OBJECTIVE: We studied whether participation in faculty development on the use of a new Milestone-based assessment tool reduced rater error for participants compared to individuals who did not participate. METHODS: We reviewed evaluations of resident Milestones completed by faculty at the end of rotations between July 2012 and June 2013. The 2 Milestones in each competency with the greatest number of ratings were selected for analysis. RESULTS: A total of 412 evaluations were analyzed, including 217 completed by faculty who participated in the development activity, and 240 completed by nonparticipant faculty. All evaluations that contained identical scores for all Milestones (16%) were completed by nonparticipant faculty (χ(2) = 37.498, P < .001). Faculty who had participated in development assigned a wider range of scores and lower minimum scores to residents, and provided the highest ratings for residents less frequently (P < .001) than nonparticipants. CONCLUSIONS: Faculty who participated in education about the Milestones demonstrated significantly less halo, range restriction, and leniency errors than faculty members who did not participate. These findings support a recommendation to develop a cadre of "core faculty" by training them in the use of Milestone assessment tools, and making them responsible for a significant portion of resident assessments.
BACKGROUND: Rater errors, such as halo/reverse halo, range restriction, and leniency errors, are frequently cited as threats to the validity of resident assessment by faculty. OBJECTIVE: We studied whether participation in faculty development on the use of a new Milestone-based assessment tool reduced rater error for participants compared to individuals who did not participate. METHODS: We reviewed evaluations of resident Milestones completed by faculty at the end of rotations between July 2012 and June 2013. The 2 Milestones in each competency with the greatest number of ratings were selected for analysis. RESULTS: A total of 412 evaluations were analyzed, including 217 completed by faculty who participated in the development activity, and 240 completed by nonparticipant faculty. All evaluations that contained identical scores for all Milestones (16%) were completed by nonparticipant faculty (χ(2) = 37.498, P < .001). Faculty who had participated in development assigned a wider range of scores and lower minimum scores to residents, and provided the highest ratings for residents less frequently (P < .001) than nonparticipants. CONCLUSIONS: Faculty who participated in education about the Milestones demonstrated significantly less halo, range restriction, and leniency errors than faculty members who did not participate. These findings support a recommendation to develop a cadre of "core faculty" by training them in the use of Milestone assessment tools, and making them responsible for a significant portion of resident assessments.
Authors: Michael L Green; Eva M Aagaard; Kelly J Caverzagie; Davoren A Chick; Eric Holmboe; Gregory Kane; Cynthia D Smith; William Iobst Journal: J Grad Med Educ Date: 2009-09
Authors: Eric S Holmboe; Denham S Ward; Richard K Reznick; Peter J Katsufrakis; Karen M Leslie; Vimla L Patel; Donna D Ray; Elizabeth A Nelson Journal: Acad Med Date: 2011-04 Impact factor: 6.893
Authors: Matthew R Thomas; Thomas J Beckman; Karen F Mauck; Stephen S Cha; Kris G Thomas Journal: J Gen Intern Med Date: 2011-03-03 Impact factor: 5.128
Authors: William Iobst; Eve Aagaard; Hasan Bazari; Timothy Brigham; Roger W Bush; Kelly Caverzagie; Davoren Chick; Michael Green; Kevin Hinchey; Eric Holmboe; Sarah Hood; Gregory Kane; Lynne Kirk; Lauren Meade; Cynthia Smith; Susan Swing Journal: J Grad Med Educ Date: 2013-03
Authors: David A Cook; Denise M Dupras; Thomas J Beckman; Kris G Thomas; V Shane Pankratz Journal: J Gen Intern Med Date: 2008-11-11 Impact factor: 5.128
Authors: Eric S Holmboe; Kenji Yamazaki; Laura Edgar; Lisa Conforti; Nicholas Yaghmour; Rebecca S Miller; Stanley J Hamstra Journal: J Grad Med Educ Date: 2015-09
Authors: Melinda Fleming; Danika Vautour; Michael McMullen; Nicholas Cofie; Nancy Dalgarno; Rachel Phelan; Glenio B Mizubuti Journal: Can Med Educ J Date: 2021-09-14