CONTEXT: Faculty members often use global rating scales as a method of assessing various characteristics of medical students' clinical performance. The purpose of this study was to determine if some performance characteristics are more highly associated with the overall faculty grade than others. METHODS: The clinical performance of 211 surgery clerkship students was evaluated by 2 or 3 faculty preceptors. Faculty rated students on 10 specific performance characteristics, using a 5-point scale. Faculty then assigned a numerical grade summarising the faculty's view of the student's performance. Reliability of the ratings was estimated by the intraclass correlation, and 1-way (analysis of variance) anova was used to test for differences among the students' mean ratings. Logistic regression was employed to determine the accuracy of each performance measure in predicting students' grades (A or B). Stepwise logistic regression was used to determine if there was a combination of performance characteristics that best predicted students' grades. RESULTS: The inter-rater reliabilities were low (<or= 0.33). Four of the 10 performance characteristics were each found to predict the assigned grade most accurately (80%). However, the least predictive characteristic achieved 75% accuracy. The stepwise logistic regressions indicated that combinations of 2 or 3 of the 10 performance characteristics increased prediction accuracy by <or= 4%. CONCLUSIONS: The rating on any single characteristic predicts a student's overall grade very accurately. Additional measures do not greatly increase the accuracy of prediction. Our results suggest that faculty make a relatively undifferentiated judgement in assigning student grades, and there is little to be gained by using multi-item forms in assessing medical students' clinical performance.
CONTEXT: Faculty members often use global rating scales as a method of assessing various characteristics of medical students' clinical performance. The purpose of this study was to determine if some performance characteristics are more highly associated with the overall faculty grade than others. METHODS: The clinical performance of 211 surgery clerkship students was evaluated by 2 or 3 faculty preceptors. Faculty rated students on 10 specific performance characteristics, using a 5-point scale. Faculty then assigned a numerical grade summarising the faculty's view of the student's performance. Reliability of the ratings was estimated by the intraclass correlation, and 1-way (analysis of variance) anova was used to test for differences among the students' mean ratings. Logistic regression was employed to determine the accuracy of each performance measure in predicting students' grades (A or B). Stepwise logistic regression was used to determine if there was a combination of performance characteristics that best predicted students' grades. RESULTS: The inter-rater reliabilities were low (<or= 0.33). Four of the 10 performance characteristics were each found to predict the assigned grade most accurately (80%). However, the least predictive characteristic achieved 75% accuracy. The stepwise logistic regressions indicated that combinations of 2 or 3 of the 10 performance characteristics increased prediction accuracy by <or= 4%. CONCLUSIONS: The rating on any single characteristic predicts a student's overall grade very accurately. Additional measures do not greatly increase the accuracy of prediction. Our results suggest that faculty make a relatively undifferentiated judgement in assigning student grades, and there is little to be gained by using multi-item forms in assessing medical students' clinical performance.
Authors: M J B Govaerts; M W J Van de Wiel; L W T Schuwirth; C P M Van der Vleuten; A M M Muijtjens Journal: Adv Health Sci Educ Theory Pract Date: 2012-05-17 Impact factor: 3.853
Authors: Catherine Hyde; Sarah Yardley; Janet Lefroy; Simon Gay; Robert K McKinley Journal: Adv Health Sci Educ Theory Pract Date: 2020-01-29 Impact factor: 3.853