BACKGROUND: Multivariate generalizability analysis was used to investigate the performance of a commonly used clinical evaluation tool. METHOD: Practicing physicians were trained to use the mini-Clinical Skills Examination (CEX) rating form to rate performances from the United States Medical Licensing Examination Step 2 Clinical Skills examination. RESULTS: Differences in rater stringency made the greatest contribution to measurement error; more raters rating each examinee, even on fewer occasions, could enhance score stability. Substantial correlated error across the competencies suggests that decisions about one scale unduly influence those on others. CONCLUSIONS: Given the appearance of a halo effect across competencies, score interpretations that assume assessment of distinct dimensions of clinical performance should be made with caution. If the intention is to produce a single composite score by combining results across competencies, the presence of these effects may be less critical.
BACKGROUND: Multivariate generalizability analysis was used to investigate the performance of a commonly used clinical evaluation tool. METHOD: Practicing physicians were trained to use the mini-Clinical Skills Examination (CEX) rating form to rate performances from the United States Medical Licensing Examination Step 2 Clinical Skills examination. RESULTS: Differences in rater stringency made the greatest contribution to measurement error; more raters rating each examinee, even on fewer occasions, could enhance score stability. Substantial correlated error across the competencies suggests that decisions about one scale unduly influence those on others. CONCLUSIONS: Given the appearance of a halo effect across competencies, score interpretations that assume assessment of distinct dimensions of clinical performance should be made with caution. If the intention is to produce a single composite score by combining results across competencies, the presence of these effects may be less critical.
Authors: David A Cook; Denise M Dupras; Thomas J Beckman; Kris G Thomas; V Shane Pankratz Journal: J Gen Intern Med Date: 2008-11-11 Impact factor: 5.128
Authors: Alberto Alves de Lima; Diego Conde; Juan Costabel; Juan Corso; Cees Van der Vleuten Journal: Adv Health Sci Educ Theory Pract Date: 2011-12-23 Impact factor: 3.853
Authors: E A M Pelgrim; A W M Kramer; H G A Mokkink; L van den Elsen; R P T M Grol; C P M van der Vleuten Journal: Adv Health Sci Educ Theory Pract Date: 2010-06-18 Impact factor: 3.853
Authors: Catharina M Haring; Claudia C R Klaarwater; Geert A Bouwmans; Bernadette M Cools; Petra J M van Gurp; Jos W M van der Meer; Cornelis T Postma Journal: BMC Med Educ Date: 2020-06-19 Impact factor: 2.463