Andrea Gingerich1, Glenn Regehr, Kevin W Eva. 1. Northern Medical Program, University of Northern British Columbia, 3333 University Way, Prince George, British Columbia V2N 4Z9. gingeri@unbc.ca
Abstract
BACKGROUND: Measurement errors are a limitation of using rater-based assessments that are commonly attributed to rater errors. Solutions targeting rater subjectivity have been largely unsuccessful. METHOD: This critical review examines investigations of rater idiosyncrasy from impression formation literatures to ask new questions for the parallel problem in rater-based assessments. RESULTS: Raters may form categorical judgments about ratees as part of impression formation. Although categorization can be idiosyncratic, raters tend to consistently construct one of a few possible interpretations of each ratee. If raters naturally form categorical judgments, an assessment system requiring ordinal or interval ratings may inadvertently introduce conversion errors due to translation techniques unique to each rater. CONCLUSIONS: Potential implications of raters forming differing categorizations of ratees combined with the use of rating scales to collect categorical judgments on measurement outcomes in rater-based assessments are explored.
BACKGROUND: Measurement errors are a limitation of using rater-based assessments that are commonly attributed to rater errors. Solutions targeting rater subjectivity have been largely unsuccessful. METHOD: This critical review examines investigations of rater idiosyncrasy from impression formation literatures to ask new questions for the parallel problem in rater-based assessments. RESULTS: Raters may form categorical judgments about ratees as part of impression formation. Although categorization can be idiosyncratic, raters tend to consistently construct one of a few possible interpretations of each ratee. If raters naturally form categorical judgments, an assessment system requiring ordinal or interval ratings may inadvertently introduce conversion errors due to translation techniques unique to each rater. CONCLUSIONS: Potential implications of raters forming differing categorizations of ratees combined with the use of rating scales to collect categorical judgments on measurement outcomes in rater-based assessments are explored.
Authors: Susan Humphrey-Murto; Timothy J Wood; Shelly Ross; Walter Tavares; Brent Kvern; Ravi Sidhu; Joan Sargeant; Claire Touchie Journal: J Grad Med Educ Date: 2017-12