BACKGROUND: Although the existing psychometric literature provides guidance on the best method for acquiring a reliable clinical evaluation form (CEF)-based score, it also shows that a single CEF rating has very low reliability. PURPOSE: This study examines whether experience with rating students might act as a form of rater training and hence improve the quality of CEF ratings. METHODS: Preceptors were divided into two groups based on rater experience. The univariate and multivariate G study designs used were simple rater (r)-nested-within-person (p) [r : p and r(○) : p(•)] models, and in the univariate analysis was applied separately to CEFs completed by high and low experienced raters. RESULTS: The high experienced rater group yielded a substantially higher observed reliability in both the univariate and multivariate analyses. CONCLUSIONS: These results support the hypothesis that high experienced raters produce more reliable ratings of student performance and suggest methods for improving CEF ratings.
BACKGROUND: Although the existing psychometric literature provides guidance on the best method for acquiring a reliable clinical evaluation form (CEF)-based score, it also shows that a single CEF rating has very low reliability. PURPOSE: This study examines whether experience with rating students might act as a form of rater training and hence improve the quality of CEF ratings. METHODS: Preceptors were divided into two groups based on rater experience. The univariate and multivariate G study designs used were simple rater (r)-nested-within-person (p) [r : p and r(○) : p(•)] models, and in the univariate analysis was applied separately to CEFs completed by high and low experienced raters. RESULTS: The high experienced rater group yielded a substantially higher observed reliability in both the univariate and multivariate analyses. CONCLUSIONS: These results support the hypothesis that high experienced raters produce more reliable ratings of student performance and suggest methods for improving CEF ratings.