Daniel Jurich1, Michelle Daniel, Miguel Paniagua, Amy Fleming, Victoria Harnik, Arnyce Pock, Aubrie Swan-Sein, Michael A Barone, Sally A Santen. 1. D. Jurich is senior psychometrician, National Board of Medical Examiners, Philadelphia, Pennsylvania. M. Daniel is assistant dean for curriculum and associate professor, Departments of Emergency Medicine and Learning Health Sciences, University of Michigan Medical School, Ann Arbor, Michigan; ORCID: http://orcid.org/0000-0001-8961-7119. M. Paniagua is medical advisor, Test Development Services, National Board of Medical Examiners, and adjunct associate professor, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania. A. Fleming is associate dean for medical student affairs and associate professor of pediatrics, Vanderbilt University School of Medicine, Nashville, Tennessee. V. Harnik is associate dean for curriculum and associate professor, Department of Cell Biology, New York University School of Medicine, New York, New York. A. Pock is associate dean for curriculum and associate professor of medicine, Uniformed Services University of the Health Sciences, Bethesda, Maryland. A. Swan-Sein is director, Center for Education Research and Evaluation, and assistant professor of educational assessment, Columbia University Vagelos College of Physicians and Surgeons, New York, New York. M.A. Barone is vice president of licensure, National Board of Medical Examiners, Philadelphia, Pennsylvania. S.A. Santen is senior associate dean of evaluation, assessment and scholarship of learning, and professor of emergency medicine, Virginia Commonwealth University School of Medicine, Richmond, Virginia, and was previously assistant dean for educational research and quality improvement, University of Michigan Medical School, Ann Arbor, Michigan.
Abstract
PURPOSE: Schools undergoing curricular reform are reconsidering the optimal timing of Step 1. This study provides a psychometric investigation of the impact on United States Medical Licensing Examination Step 1 scores of changing the timing of Step 1 from after completion of the basic science curricula to after core clerkships. METHOD: Data from four schools that recently moved the examination were analyzed in a pre-post format using examinee scores from three years before and after the change. The sample included scores from 2008 through 2016. Several confounders were addressed, including rising national scores and potential differences in cohort abilities using deviation scores and analysis of covariance (ANCOVA) controlling for Medical College Admission Test (MCAT) scores. A resampling procedure compared study schools' score changes versus similar schools' in the same time period. RESULTS: The ANCOVA indicated postchange Step 1 scores were higher compared with prechange (adjusted difference = 2.67; 95% confidence interval: 1.50-3.83, P < .001; effect size = 0.14) after adjusting for MCAT scores and rising national averages. The average score increase in study schools was larger than changes seen in similar schools. Failure rates also decreased from 2.87% (n = 48) pre change to 0.39% (n = 6) post change (P < .001). CONCLUSIONS: Results suggest moving Step 1 after core clerkships yielded a small increase in scores and a reduction in failure rates. Although these small increases are unlikely to represent meaningful knowledge gains, this demonstration of "noninferiority" may allow schools to implement significant curricular reforms.
PURPOSE: Schools undergoing curricular reform are reconsidering the optimal timing of Step 1. This study provides a psychometric investigation of the impact on United States Medical Licensing Examination Step 1 scores of changing the timing of Step 1 from after completion of the basic science curricula to after core clerkships. METHOD: Data from four schools that recently moved the examination were analyzed in a pre-post format using examinee scores from three years before and after the change. The sample included scores from 2008 through 2016. Several confounders were addressed, including rising national scores and potential differences in cohort abilities using deviation scores and analysis of covariance (ANCOVA) controlling for Medical College Admission Test (MCAT) scores. A resampling procedure compared study schools' score changes versus similar schools' in the same time period. RESULTS: The ANCOVA indicated postchange Step 1 scores were higher compared with prechange (adjusted difference = 2.67; 95% confidence interval: 1.50-3.83, P < .001; effect size = 0.14) after adjusting for MCAT scores and rising national averages. The average score increase in study schools was larger than changes seen in similar schools. Failure rates also decreased from 2.87% (n = 48) pre change to 0.39% (n = 6) post change (P < .001). CONCLUSIONS: Results suggest moving Step 1 after core clerkships yielded a small increase in scores and a reduction in failure rates. Although these small increases are unlikely to represent meaningful knowledge gains, this demonstration of "noninferiority" may allow schools to implement significant curricular reforms.
Authors: Daniel Jurich; Sally A Santen; Miguel Paniagua; Amy Fleming; Victoria Harnik; Arnyce Pock; Aubrie Swan-Sein; Michael A Barone; Michelle Daniel Journal: Acad Med Date: 2020-01 Impact factor: 6.893
Authors: Matthew Fitz; William Adams; Marc Heincelman; Steve Haist; Karina Whelan; LeeAnn Cox; Uyen-Thi Cao; Susan Hingle; Amanda Raff; Bruce Houghton; Janet Fitzpatrick; Ryan Nall; Jennifer Foster; Jonathan Appelbaum; Cyril Grum; Anna Donovan; Stuart Kiken; Reeni Abraham; Marti Hlafka; Chad Miller; Saurabh Bansal; Douglas Paauw; Cindy J Lai; Amber Pincavage; Gauri Agarwal; Cynthia Burns; Horatio Holzer; Katie Lappé; Viju John; Blake Barker; Nina Mingioni; Deepti Rao; Laura Zakowski; Chayan Chakraborti; Winter Williams; William Kelly Journal: J Gen Intern Med Date: 2022-06-28 Impact factor: 6.473
Authors: Sandra B Haudek; Ingrid Bahner; Andrea N Belovich; Giulia Bonaminio; Anthony Brenneman; William S Brooks; Cassie Chinn; Nehad El-Sawi; Shafik Habal; Michele Haight; Uzoma Ikonne; Robert J McAuley; Douglas McKell; Rebecca Rowe; Tracey A H Taylor; Thomas Thesen; Richard C Vari Journal: Med Sci Educ Date: 2022-05-04
Authors: Kelly M Quesnelle; Naunihal T Zaveri; Stephen D Schneid; Joe B Blumer; John L Szarek; Marieke Kruidering; Michael W Lee Journal: Pharmacol Res Perspect Date: 2021-05