Kathleen Short1, S Deniz Bucak, Francine Rosenthal, Mark R Raymond. 1. K. Short is program officer, National Board of Medical Examiners, Philadelphia, Pennsylvania; ORCID: http://orcid.org/0000-0001-9602-3124. S.D. Bucak is manager, National Board of Medical Examiners, Philadelphia, Pennsylvania; http://orcid.org/0000-0002-4370-8266. F. Rosenthal is senior editor, National Board of Medical Examiners, Philadelphia, Pennsylvania. M.R. Raymond is research director and principal assessment scientist, National Board of Medical Examiners, Philadelphia, Pennsylvania.
Abstract
PURPOSE: In 2007, the United States Medical Licensing Examination embedded multimedia simulations of heart sounds into multiple-choice questions. This study investigated changes in item difficulty as determined by examinee performance over time. The data reflect outcomes obtained following initial use of multimedia items from 2007 through 2012, after which an interface change occurred. METHOD: A total of 233,157 examinees responded to 1,306 cardiology test items over the six-year period; 138 items included multimedia simulations of heart sounds, while 1,168 text-based items without multimedia served as controls. The authors compared changes in difficulty of multimedia items over time with changes in difficulty of text-based cardiology items over time. Further, they compared changes in item difficulty for both groups of items between graduates of Liaison Committee on Medical Education (LCME)-accredited and non-LCME-accredited (i.e., international) medical schools. RESULTS: Examinee performance on cardiology test items with multimedia heart sounds improved by 12.4% over the six-year period, while performance on text-based cardiology items improved by approximately 1.4%. These results were similar for graduates of LCME-accredited and non-LCME-accredited medical schools. CONCLUSIONS: Examinees' ability to interpret auscultation findings in test items that include multimedia presentations increased from 2007 to 2012.
PURPOSE: In 2007, the United States Medical Licensing Examination embedded multimedia simulations of heart sounds into multiple-choice questions. This study investigated changes in item difficulty as determined by examinee performance over time. The data reflect outcomes obtained following initial use of multimedia items from 2007 through 2012, after which an interface change occurred. METHOD: A total of 233,157 examinees responded to 1,306 cardiology test items over the six-year period; 138 items included multimedia simulations of heart sounds, while 1,168 text-based items without multimedia served as controls. The authors compared changes in difficulty of multimedia items over time with changes in difficulty of text-based cardiology items over time. Further, they compared changes in item difficulty for both groups of items between graduates of Liaison Committee on Medical Education (LCME)-accredited and non-LCME-accredited (i.e., international) medical schools. RESULTS: Examinee performance on cardiology test items with multimedia heart sounds improved by 12.4% over the six-year period, while performance on text-based cardiology items improved by approximately 1.4%. These results were similar for graduates of LCME-accredited and non-LCME-accredited medical schools. CONCLUSIONS: Examinees' ability to interpret auscultation findings in test items that include multimedia presentations increased from 2007 to 2012.