BACKGROUND: Rater-based judgments are widely used in graduate medical education to provide more meaningful assessments, despite concerns about rater reliability. OBJECTIVE: We introduced a statistical modeling technique that corresponds to the new rater reliability framework, and present a case example to provide an illustration of the utility of this new approach to assessing rater reliability. METHODS: We used mixed-effects models to simultaneously incorporate random effects for raters and systematic effects of rater role as fixed effects. Study data are clinical performance ratings collected from medical school graduates who were evaluated for their readiness for supervised clinical practice in authentic simulation settings at 2 medical schools in the Netherlands and Germany. RESULTS: The medical schools recruited a maximum of 30 graduates out of 60 (50%) and 180 (17%) eligible candidates, respectively. Clinician raters (n = 25) for the study were selected based on their level of expertise and experience. Graduates were assessed on 7 facets of competence (FOCs) that are considered important in supervisors' entrustment decisions across the 5 cases used. Rater role was significantly associated with 2 FOCs: (1) teamwork and collegiality, and (2) verbal communication with colleagues/supervisors. For another 2 FOCs, rater variability was only partially explained by the role of the rater (a proxy for the amount of direct interaction with the trainee). CONCLUSIONS: Consideration of raters as meaningfully idiosyncratic provides a new framework to explore their influence on assessment scores, which goes beyond considering them as random sources of variability.
BACKGROUND: Rater-based judgments are widely used in graduate medical education to provide more meaningful assessments, despite concerns about rater reliability. OBJECTIVE: We introduced a statistical modeling technique that corresponds to the new rater reliability framework, and present a case example to provide an illustration of the utility of this new approach to assessing rater reliability. METHODS: We used mixed-effects models to simultaneously incorporate random effects for raters and systematic effects of rater role as fixed effects. Study data are clinical performance ratings collected from medical school graduates who were evaluated for their readiness for supervised clinical practice in authentic simulation settings at 2 medical schools in the Netherlands and Germany. RESULTS: The medical schools recruited a maximum of 30 graduates out of 60 (50%) and 180 (17%) eligible candidates, respectively. Clinician raters (n = 25) for the study were selected based on their level of expertise and experience. Graduates were assessed on 7 facets of competence (FOCs) that are considered important in supervisors' entrustment decisions across the 5 cases used. Rater role was significantly associated with 2 FOCs: (1) teamwork and collegiality, and (2) verbal communication with colleagues/supervisors. For another 2 FOCs, rater variability was only partially explained by the role of the rater (a proxy for the amount of direct interaction with the trainee). CONCLUSIONS: Consideration of raters as meaningfully idiosyncratic provides a new framework to explore their influence on assessment scores, which goes beyond considering them as random sources of variability.
Authors: Erik W Driessen; Jan van Tartwijk; Marjan Govaerts; Pim Teunissen; Cees P M van der Vleuten Journal: Med Teach Date: 2012 Impact factor: 3.650
Authors: Karen E Hauer; Olle Ten Cate; Christy Boscardin; David M Irby; William Iobst; Patricia S O'Sullivan Journal: Adv Health Sci Educ Theory Pract Date: 2013-07-27 Impact factor: 3.853
Authors: M J B Govaerts; L W T Schuwirth; C P M Van der Vleuten; A M M Muijtjens Journal: Adv Health Sci Educ Theory Pract Date: 2010-09-30 Impact factor: 3.853