Judith M Brenner1, Jeffrey B Bird, Joanne M Willey. 1. J.M. Brenner is associate dean of curricular integration and assessment, Hofstra Northwell School of Medicine, Hempstead, New York. J.B. Bird is assessment and evaluation analyst, Hofstra Northwell School of Medicine, Hempstead, New York. J.M. Willey is professor and chair, Department of Science Education, Hofstra Northwell School of Medicine, Hempstead, New York.
Abstract
PURPOSE: The Hofstra Northwell School of Medicine (HNSOM) uses an essay-based assessment system. Recognizing the emphasis graduate medical education places on the United States Medical Licensing Examination (USMLE) Step exams, the authors developed a method to predict students at risk for lower performance on USMLE Step 1. METHOD: Beginning with the inaugural class (2015), HNSOM administered National Board of Medical Examiners (NBME) Customized Assessment Service (CAS) examinations as formative assessment at the end of each integrated course in the first two years of medical school. Using preadmission data, the first two courses in the educational program, and NBME score deviation from the national test takers' mean, a statistical model was built to predict students who scored below the Step 1 national mean. RESULTS: A regression equation using the highest Medical College Admission Test (MCAT) score and NBME score deviation predicted student Step 1 scores. The MCAT alone accounted for 21% of the variance. Adding the NBME score deviation from the first and second courses increased the variance to 40% and 50%, respectively. Adding NBME exams from later courses increased the variance to 52% and 64% by the end of years one and two, respectively. Cross-validation demonstrated the model successfully predicted 63% of at-risk students by the end of the fifth month of medical school. CONCLUSIONS: The model identified students at risk for lower performance on Step 1 using the NBME CAS. This model is applicable to schools reforming their curriculum delivery and assessment programs toward an integrated model.
PURPOSE: The Hofstra Northwell School of Medicine (HNSOM) uses an essay-based assessment system. Recognizing the emphasis graduate medical education places on the United States Medical Licensing Examination (USMLE) Step exams, the authors developed a method to predict students at risk for lower performance on USMLE Step 1. METHOD: Beginning with the inaugural class (2015), HNSOM administered National Board of Medical Examiners (NBME) Customized Assessment Service (CAS) examinations as formative assessment at the end of each integrated course in the first two years of medical school. Using preadmission data, the first two courses in the educational program, and NBME score deviation from the national test takers' mean, a statistical model was built to predict students who scored below the Step 1 national mean. RESULTS: A regression equation using the highest Medical College Admission Test (MCAT) score and NBME score deviation predicted student Step 1 scores. The MCAT alone accounted for 21% of the variance. Adding the NBME score deviation from the first and second courses increased the variance to 40% and 50%, respectively. Adding NBME exams from later courses increased the variance to 52% and 64% by the end of years one and two, respectively. Cross-validation demonstrated the model successfully predicted 63% of at-risk students by the end of the fifth month of medical school. CONCLUSIONS: The model identified students at risk for lower performance on Step 1 using the NBME CAS. This model is applicable to schools reforming their curriculum delivery and assessment programs toward an integrated model.