PURPOSE: To investigate the measurement characteristics of standardized clinical evaluation forms (CEFs) used to assign grades for clerkship performance. METHOD: In 1996-97, the authors reviewed 5,168 CEFs completed for 175 students in eight clerkships. Limiting their analysis to the three clerkships that produced the most CEFs, the authors conducted a generalizability study to determine the five variance components for each clerkship. A decision study then calculated the generalizability coefficients and standard errors of measurement in each clerkship for varied numbers of raters and CEF items. RESULTS: The generalizability study found large variance components attributable to rater and rating context. The decision study found that, when three or more raters completed CEFs for a student, the generalizability coefficient and standard error of measurement reached levels acceptable for grading. Increasing the number of items on the CEF had no significant effect. CONCLUSION: The reliability of assigning students clerkship grades based on single CEFs is unacceptably low. However, CEFs can accurately measure students' clerkship performances if completed by three or more raters.
PURPOSE: To investigate the measurement characteristics of standardized clinical evaluation forms (CEFs) used to assign grades for clerkship performance. METHOD: In 1996-97, the authors reviewed 5,168 CEFs completed for 175 students in eight clerkships. Limiting their analysis to the three clerkships that produced the most CEFs, the authors conducted a generalizability study to determine the five variance components for each clerkship. A decision study then calculated the generalizability coefficients and standard errors of measurement in each clerkship for varied numbers of raters and CEF items. RESULTS: The generalizability study found large variance components attributable to rater and rating context. The decision study found that, when three or more raters completed CEFs for a student, the generalizability coefficient and standard error of measurement reached levels acceptable for grading. Increasing the number of items on the CEF had no significant effect. CONCLUSION: The reliability of assigning students clerkship grades based on single CEFs is unacceptably low. However, CEFs can accurately measure students' clerkship performances if completed by three or more raters.
Authors: Tina Hsu; Flávia De Angelis; Sohaib Al-Asaaed; Sanraj K Basi; Anna Tomiak; Debjani Grenier; Nazik Hammad; Jan-Willem Henning; Scott Berry; Xinni Song; Som D Mukherjee Journal: Can Med Educ J Date: 2021-04-30
Authors: Andrew W Phillips; David Diller; Sarah Williams; Yoon Soo Park; Jonathan Fisher; Kevin Biese; Jacob Ufberg Journal: AEM Educ Train Date: 2017-09-21