Kevin W Eva1, Glenn Regehr. 1. Department of Clinical Epidemiology and Biostatistics, Program for Educational Research and Development, MDCL 3522, McMaster University, Hamilton, ON, L8N 3Z5, Canada. evakw@mcmaster.ca
Abstract
BACKGROUND: Although self-assessment is widely acknowledged as a vital skill for members of self-regulating professions, a ubiquitous finding in the research literature is that self-ratings are quite poor when compared with externally generated measures of ability. Many researchers have identified this as a serious problem for the concept of self-regulation in the professions. However, we question the sufficiency of the operational definitions of self-assessment on which the previous research is based. This study examines the validity of a new conceptualization of self-assessment in practice and evaluates a series of measures for capturing self-assessment ability as defined by this new conceptualization. METHOD: Using a computer-delivered free-response test, the authors generated three measures intended to capture situational awareness: (1) response times to questions, (2) the ability to avoid responding to questions for which the respondent is less likely to be correct, and (3) the ability to select questions from content areas in which respondents have greater ability. In addition, the traditional measures of self-assessment (e.g., predictions of how many questions one would answer correctly) were administered. RESULTS: Participants showed behavioral indications of being aware of the limits of their ability. They took longer to respond when their eventual answer was incorrect relative to when it was correct, they were able to avoid answering questions on which they were likely to be incorrect, and they selected content-based domains in an appropriate order given their accuracy. DISCUSSION: These results provide evidence in favor of this new framework that should reorient the way in which self-assessment "skills" are conceptualized, taught, and evaluated in medical school and beyond.
BACKGROUND: Although self-assessment is widely acknowledged as a vital skill for members of self-regulating professions, a ubiquitous finding in the research literature is that self-ratings are quite poor when compared with externally generated measures of ability. Many researchers have identified this as a serious problem for the concept of self-regulation in the professions. However, we question the sufficiency of the operational definitions of self-assessment on which the previous research is based. This study examines the validity of a new conceptualization of self-assessment in practice and evaluates a series of measures for capturing self-assessment ability as defined by this new conceptualization. METHOD: Using a computer-delivered free-response test, the authors generated three measures intended to capture situational awareness: (1) response times to questions, (2) the ability to avoid responding to questions for which the respondent is less likely to be correct, and (3) the ability to select questions from content areas in which respondents have greater ability. In addition, the traditional measures of self-assessment (e.g., predictions of how many questions one would answer correctly) were administered. RESULTS:Participants showed behavioral indications of being aware of the limits of their ability. They took longer to respond when their eventual answer was incorrect relative to when it was correct, they were able to avoid answering questions on which they were likely to be incorrect, and they selected content-based domains in an appropriate order given their accuracy. DISCUSSION: These results provide evidence in favor of this new framework that should reorient the way in which self-assessment "skills" are conceptualized, taught, and evaluated in medical school and beyond.