Brian J Hess1, Rebecca S Lipner, Valerie Thompson, Eric S Holmboe, Mark L Graber. 1. Dr. Hess is a consultant, Hess Consulting, St-Nicolas, Québec, Canada. Dr. Lipner is senior vice president of evaluation, research and development, American Board of Internal Medicine, Philadelphia, Pennsylvania. Dr. Thompson is professor of cognitive psychology, University of Saskatchewan, Saskatoon, Saskatchewan, Canada. Dr. Holmboe is senior vice president of milestone development and evaluation, Accreditation Council for Graduate Medical Education, Chicago, Illinois. Dr. Graber is senior research fellow, RTI International, Research Triangle Park, North Carolina, and professor emeritus, at SUNY Stony Brook University School of Medicine, Stony Brook, New York.
Abstract
PURPOSE: Experienced clinicians derive many diagnoses intuitively, because most new problems they see closely resemble problems they've seen before. The majority of these diagnoses, but not all, will be correct. This study determined whether further reflection regarding initial diagnoses improves diagnostic accuracy during a high-stakes board exam, a model for studying clinical decision making. METHOD: Keystroke response data were used from 500 residents who took the 2010 American Board of Internal Medicine (ABIM) Internal Medicine Certification Examination. Data included time to initial response on each question, whether the answer was correct, and whether or not the resident changed her or his initial response. The focus was on 80 diagnosis questions that comprised realistic clinical vignettes with multiple-choice single-best answers. Cognitive skill (ability) was measured using overall exam scores. Case complexity was determined using item difficulty (proportion of examinees that correctly answered the question). A hierarchical generalized linear model was used to assess the relationship between time spent on initial responses and the probability of correctly answering the questions. RESULTS: On average, residents changed their responses on 12% of all diagnosis questions (or 9.6 questions out of 80). Changing an answer from incorrect to correct was almost twice as likely as changing an answer from correct to incorrect. The relationship between response time and accuracy was complex. CONCLUSIONS: Further reflection appears to be beneficial to diagnostic accuracy, especially for more complex cases.
PURPOSE: Experienced clinicians derive many diagnoses intuitively, because most new problems they see closely resemble problems they've seen before. The majority of these diagnoses, but not all, will be correct. This study determined whether further reflection regarding initial diagnoses improves diagnostic accuracy during a high-stakes board exam, a model for studying clinical decision making. METHOD: Keystroke response data were used from 500 residents who took the 2010 American Board of Internal Medicine (ABIM) Internal Medicine Certification Examination. Data included time to initial response on each question, whether the answer was correct, and whether or not the resident changed her or his initial response. The focus was on 80 diagnosis questions that comprised realistic clinical vignettes with multiple-choice single-best answers. Cognitive skill (ability) was measured using overall exam scores. Case complexity was determined using item difficulty (proportion of examinees that correctly answered the question). A hierarchical generalized linear model was used to assess the relationship between time spent on initial responses and the probability of correctly answering the questions. RESULTS: On average, residents changed their responses on 12% of all diagnosis questions (or 9.6 questions out of 80). Changing an answer from incorrect to correct was almost twice as likely as changing an answer from correct to incorrect. The relationship between response time and accuracy was complex. CONCLUSIONS: Further reflection appears to be beneficial to diagnostic accuracy, especially for more complex cases.
Authors: Sreeja Natesan; John Bailitz; Andrew King; Sara M Krzyzaniak; Sarah K Kennedy; Albert J Kim; Richard Byyny; Michael Gottlieb Journal: West J Emerg Med Date: 2020-07-03