CONTEXT: Subjective rating scales for communication skills may yield more personally meaningful responses than more standardised rating schemes. It is unclear, however, whether such evaluations may be overly biased by respondents' rating styles, which may lead to unreliable measurement of examinees' communication skills. METHODS: Our study involved 212 students from the classes of 2005 and 2006 at the University of Rochester School of Medicine and Dentistry. All students were rated by actors depicting standardised patients (SPs) on the same seven cases using the 19-item Rochester Communication Rating Scale (RCRS). Different students were assigned to different actors playing the same SP. We assessed the extent to which actors' personal rating styles influenced the scores they assigned to students. Main outcome measures were: between-actor variability in responses; the degree to which actors' response styles contribute to overall scores, and improvements in reliability achieved by standardising actors' ratings. RESULTS: There were statistically significant differences between actors in their mean assigned scores. Scores aggregated over 18 separate SP cases have an expected generalisability coefficient of 0.79. If raw RCRS scores are used, a total of 27 replications of the RCRS are required to achieve a Cronbach's alpha of 0.8; standardisation reduces this number to 18. CONCLUSIONS: Although actors are variable in their use of a standardised subjective scale of communication, such differences contribute to an acceptably small proportion of the total variance if scores are combined across a large number of cases. Reliability can be markedly improved by standardising scores across raters.
CONTEXT: Subjective rating scales for communication skills may yield more personally meaningful responses than more standardised rating schemes. It is unclear, however, whether such evaluations may be overly biased by respondents' rating styles, which may lead to unreliable measurement of examinees' communication skills. METHODS: Our study involved 212 students from the classes of 2005 and 2006 at the University of Rochester School of Medicine and Dentistry. All students were rated by actors depicting standardised patients (SPs) on the same seven cases using the 19-item Rochester Communication Rating Scale (RCRS). Different students were assigned to different actors playing the same SP. We assessed the extent to which actors' personal rating styles influenced the scores they assigned to students. Main outcome measures were: between-actor variability in responses; the degree to which actors' response styles contribute to overall scores, and improvements in reliability achieved by standardising actors' ratings. RESULTS: There were statistically significant differences between actors in their mean assigned scores. Scores aggregated over 18 separate SP cases have an expected generalisability coefficient of 0.79. If raw RCRS scores are used, a total of 27 replications of the RCRS are required to achieve a Cronbach's alpha of 0.8; standardisation reduces this number to 18. CONCLUSIONS: Although actors are variable in their use of a standardised subjective scale of communication, such differences contribute to an acceptably small proportion of the total variance if scores are combined across a large number of cases. Reliability can be markedly improved by standardising scores across raters.
Authors: Patricia A Carney; Ryan T Palmer; Marissa Fuqua Miller; Erin K Thayer; Sue E Estroff; Debra K Litzelman; Frances E Biagioli; Cayla R Teal; Ann Lambros; William J Hatt; Jason M Satterfield Journal: Acad Med Date: 2016-05 Impact factor: 6.893
Authors: Kristina Schick; Sabine Reiser; Katharina Mosene; Laura Schacht; Laura Janssen; Eva Thomm; Andreas Dinkel; Andreas Fleischmann; Pascal O Berberat; Johannes Bauer; Martin Gartmeier Journal: GMS J Med Educ Date: 2020-11-16
Authors: Kevin Gruffydd-Jones; Helen Marsden; Steve Holmes; Peter Kardos; Roger Escamilla; Roberto Dal Negro; June Roberts; Gilbert Nadeau; David Leather; Paul Jones Journal: BMC Med Res Methodol Date: 2013-05-10 Impact factor: 4.615