Marian Manciu1, Roszella Trevino1, Zuber D Mulla2, Claudia Cortez2, Sanja Kupesic Plavsic2. 1. Physics Department, University of Texas at El Paso. 2. Department of Obstetrics and Gynecology, Center for Advanced Teaching and Assessment in Clinical Simulation, Paul L. Foster School of Medicine, Texas Tech University Health Sciences Center El Paso.
Abstract
OBJECTIVES: To assess the inter-rater reliability of standardized patients (SPs) as they assess the clinical skills of medical students and to detect possible rating bias in SPs. METHODS: The ratings received by 6 students examined in 4 clinical stations by 13 SPs were examined. Each SP contributed at least 3 and at most 10 pairwise ratings, with an average of approximately 5 ratings per SP. The standard Cohen' kappa statistic was calculated and the distribution of scores among SPs was compared via both ANOVA the Kruskal-Wallis H test (one-way ANOVA by ranks). Furthermore, the number of discrepancies between pairwise raters (showing either "positive" or "negative" bias in the rating) were analyzed using ANOVA and a χ2 goodness-of-fit test. RESULTS: The conventional method, which compared the statistics of kappa scores of the raters (including the prevalence-adjusted-bias adjusted kappa scores) did not reject the null hypothesis, that the raters (SPs) are similar. However, the analysis of the distribution of the discrepancies among the raters revealed that the differences between raters cannot be attributed to chance, particularly when a distinction was made between their overall "positive" and "negative" bias. A strong (p<0.001) "negative" bias was detected, and the SPs responsible for this bias have been identified. CONCLUSIONS: The statistical method suggested here, which takes into account explicitly the "positive" and the "negative" bias of the raters, is more sensitive than the conventional method (Cohens' kappa). Since the outliers (the biased SPs) affect the fairness of the grading of the medical students, it is important to detect any statistically-significant bias in the rating and to adjust correspondingly the SP's assessment.
OBJECTIVES: To assess the inter-rater reliability of standardized patients (SPs) as they assess the clinical skills of medical students and to detect possible rating bias in SPs. METHODS: The ratings received by 6 students examined in 4 clinical stations by 13 SPs were examined. Each SP contributed at least 3 and at most 10 pairwise ratings, with an average of approximately 5 ratings per SP. The standard Cohen' kappa statistic was calculated and the distribution of scores among SPs was compared via both ANOVA the Kruskal-Wallis H test (one-way ANOVA by ranks). Furthermore, the number of discrepancies between pairwise raters (showing either "positive" or "negative" bias in the rating) were analyzed using ANOVA and a χ2 goodness-of-fit test. RESULTS: The conventional method, which compared the statistics of kappa scores of the raters (including the prevalence-adjusted-bias adjusted kappa scores) did not reject the null hypothesis, that the raters (SPs) are similar. However, the analysis of the distribution of the discrepancies among the raters revealed that the differences between raters cannot be attributed to chance, particularly when a distinction was made between their overall "positive" and "negative" bias. A strong (p<0.001) "negative" bias was detected, and the SPs responsible for this bias have been identified. CONCLUSIONS: The statistical method suggested here, which takes into account explicitly the "positive" and the "negative" bias of the raters, is more sensitive than the conventional method (Cohens' kappa). Since the outliers (the biased SPs) affect the fairness of the grading of the medical students, it is important to detect any statistically-significant bias in the rating and to adjust correspondingly the SP's assessment.
Authors: Kevin Fiscella; Peter Franks; Malathi Srinivasan; Richard L Kravitz; Ronald Epstein Journal: Ann Fam Med Date: 2007 Mar-Apr Impact factor: 5.166
Authors: P Stillman; D Swanson; M B Regan; M M Philbin; V Nelson; T Ebert; B Ley; T Parrino; J Shorey; A Stillman Journal: Ann Intern Med Date: 1991-03-01 Impact factor: 25.391
Authors: Abd Moain Abu Dabrh; Mohammad Hassan Murad; Richard D Newcomb; William G Buchta; Mark W Steffen; Zhen Wang; Amanda K Lovett; Lawrence W Steinkraus Journal: BMC Med Educ Date: 2016-09-02 Impact factor: 2.463