BACKGROUND: Individual faculty assessments of resident competency are complicated by inconsistent application of standards, lack of reliability, and the "halo" effect. OBJECTIVE: We determined whether the addition of faculty group assessments of residents in an ambulatory clinic, compared with individual faculty-of-resident assessments alone, have better reliability and reduced halo effects. DESIGN: This prospective, longitudinal study was performed in the outpatient continuity clinics of a large internal medicine residency program. MAIN MEASURES: Faculty-on-resident and group faculty-on-resident assessment scores were used for comparison. KEY RESULTS: Overall mean scores were significantly higher for group than individual assessments (3.92 ± 0.51 vs. 3.83 ± 0.38, p = 0.0001). Overall inter-rater reliability increased when combining group and individual assessments compared to individual assessments alone (intraclass correlation coefficient, 95% CI = 0.828, 0.785-0.866 vs. 0.749, 0.686-0.804). Inter-item correlations were less for group (0.49) than individual (0.68) assessments. CONCLUSIONS: This study demonstrates improved inter-rater reliability and reduced range restriction (halo effect) of resident assessment across multiple performance domains by adding the group assessment method to traditional individual faculty-on-resident assessment. This feasible model could help graduate medical education programs achieve more reliable and discriminating resident assessments.
BACKGROUND: Individual faculty assessments of resident competency are complicated by inconsistent application of standards, lack of reliability, and the "halo" effect. OBJECTIVE: We determined whether the addition of faculty group assessments of residents in an ambulatory clinic, compared with individual faculty-of-resident assessments alone, have better reliability and reduced halo effects. DESIGN: This prospective, longitudinal study was performed in the outpatient continuity clinics of a large internal medicine residency program. MAIN MEASURES: Faculty-on-resident and group faculty-on-resident assessment scores were used for comparison. KEY RESULTS: Overall mean scores were significantly higher for group than individual assessments (3.92 ± 0.51 vs. 3.83 ± 0.38, p = 0.0001). Overall inter-rater reliability increased when combining group and individual assessments compared to individual assessments alone (intraclass correlation coefficient, 95% CI = 0.828, 0.785-0.866 vs. 0.749, 0.686-0.804). Inter-item correlations were less for group (0.49) than individual (0.68) assessments. CONCLUSIONS: This study demonstrates improved inter-rater reliability and reduced range restriction (halo effect) of resident assessment across multiple performance domains by adding the group assessment method to traditional individual faculty-on-resident assessment. This feasible model could help graduate medical education programs achieve more reliable and discriminating resident assessments.
Authors: Michael J Battistone; Caroline Milne; Merle A Sande; Louis N Pangaro; Paul A Hemmer; T Samuel Shomaker Journal: Teach Learn Med Date: 2002 Impact factor: 2.414
Authors: Karen E Hauer; Olle Ten Cate; Christy K Boscardin; William Iobst; Eric S Holmboe; Benjamin Chesluk; Robert B Baron; Patricia S O'Sullivan Journal: J Grad Med Educ Date: 2016-05
Authors: Marrigje E Duitsman; Irene A Slootweg; Imke C van der Marel; Marianne Ten Kate-Booij; Jacqueline de Graaf; Cornelia Fluit; Debbie Jaarsma Journal: J Grad Med Educ Date: 2019-08