BACKGROUND: The National Board of Medical Examiners is currently developing the Assessment of Professional Behaviors, a multisource feedback (MSF) tool intended for formative use with medical students and residents. This study investigated whether missing responses on this tool can be considered random; evidence that missing values are not random would suggest response bias, a significant threat to score validity. METHOD: Correlational analyses of pilot data (N = 2,149) investigated whether missing values were systematically related to global evaluations of observees. RESULTS: The percentage of missing items was correlated with global evaluations of observees; observers answered more items for preferred observees compared with nonpreferred observees. CONCLUSIONS: Missing responses on this MSF tool seem to be nonrandom and are instead systematically related to global perceptions of observees. Further research is needed to determine whether modifications to the items, the instructions, or other components of the assessment process can reduce this effect.
BACKGROUND: The National Board of Medical Examiners is currently developing the Assessment of Professional Behaviors, a multisource feedback (MSF) tool intended for formative use with medical students and residents. This study investigated whether missing responses on this tool can be considered random; evidence that missing values are not random would suggest response bias, a significant threat to score validity. METHOD: Correlational analyses of pilot data (N = 2,149) investigated whether missing values were systematically related to global evaluations of observees. RESULTS: The percentage of missing items was correlated with global evaluations of observees; observers answered more items for preferred observees compared with nonpreferred observees. CONCLUSIONS: Missing responses on this MSF tool seem to be nonrandom and are instead systematically related to global perceptions of observees. Further research is needed to determine whether modifications to the items, the instructions, or other components of the assessment process can reduce this effect.
Authors: Walther N K A van Mook; Arno M M Muijtjens; Simone L Gorter; Jan Harm Zwaveling; Lambert W Schuwirth; Cees P M van der Vleuten Journal: Adv Health Sci Educ Theory Pract Date: 2011-05-01 Impact factor: 3.853