Michael S Beeson1, Eric S Holmboe2,3, Robert C Korte4, Thomas J Nasca3,5, Timothy Brigham3,5, Chad M Russ4, Cameron T Whitley6, Earl J Reisdorff4. 1. The Department of Emergency Medicine, Akron General Medical Center, Akron, OH. 2. Milestones Development and Evaluation, Chicago, IL. 3. Accreditation Council for Graduate Medical Education, Chicago, IL. 4. American Board of Emergency Medicine, East Lansing, MI. 5. Jefferson Medical Center, Philadelphia, PA. 6. The Department of Sociology, Michigan State University, East Lansing, MI.
Abstract
OBJECTIVES: The Accreditation Council for Graduate Medical Education (ACGME) Milestones describe behavioral markers for the progressive acquisition of competencies during residency. As a key component of the Next Accreditation System, all residents are evaluated for the acquisition of specialty-specific Milestones. The objective was to determine the validity and reliability of the emergency medicine (EM) Milestones. METHODS: The ACGME and the American Board of Emergency Medicine performed this single-event observational study. The data included the initial EM Milestones performance ratings of all categorical EM residents submitted to the ACGME from October 31, 2013, to January 6, 2014. Mean performance ratings were determined for all 23 subcompetencies for every year of residency training. The internal consistency (reliability) of the Milestones was determined using a standardized Cronbach's alpha coefficient. Exploratory factor analysis was conducted to determine how the subcompetencies were interrelated. RESULTS: EM Milestone performance ratings were obtained on 100% of EM residents (n = 5,805) from 162 residency programs. The mean performance ratings of the aggregate and individual subcompetency scores showed discrimination between residency years, and the factor structure further supported the validity of the EM Milestones. The reliability was α = 0.96 within each year of training. CONCLUSIONS: The EM Milestones demonstrated validity and reliability as an assessment instrument for competency acquisition. EM residents can be assured that this evaluation process has demonstrated validity and reliability; faculty can be confident that the Milestones are psychometrically sound; and stakeholders can know that the Milestones are a nationally standardized, objective measure of specialty-specific competency acquisition.
OBJECTIVES: The Accreditation Council for Graduate Medical Education (ACGME) Milestones describe behavioral markers for the progressive acquisition of competencies during residency. As a key component of the Next Accreditation System, all residents are evaluated for the acquisition of specialty-specific Milestones. The objective was to determine the validity and reliability of the emergency medicine (EM) Milestones. METHODS: The ACGME and the American Board of Emergency Medicine performed this single-event observational study. The data included the initial EM Milestones performance ratings of all categorical EM residents submitted to the ACGME from October 31, 2013, to January 6, 2014. Mean performance ratings were determined for all 23 subcompetencies for every year of residency training. The internal consistency (reliability) of the Milestones was determined using a standardized Cronbach's alpha coefficient. Exploratory factor analysis was conducted to determine how the subcompetencies were interrelated. RESULTS: EM Milestone performance ratings were obtained on 100% of EM residents (n = 5,805) from 162 residency programs. The mean performance ratings of the aggregate and individual subcompetency scores showed discrimination between residency years, and the factor structure further supported the validity of the EM Milestones. The reliability was α = 0.96 within each year of training. CONCLUSIONS: The EM Milestones demonstrated validity and reliability as an assessment instrument for competency acquisition. EM residents can be assured that this evaluation process has demonstrated validity and reliability; faculty can be confident that the Milestones are psychometrically sound; and stakeholders can know that the Milestones are a nationally standardized, objective measure of specialty-specific competency acquisition.
Authors: Eric S Holmboe; Kenji Yamazaki; Laura Edgar; Lisa Conforti; Nicholas Yaghmour; Rebecca S Miller; Stanley J Hamstra Journal: J Grad Med Educ Date: 2015-09
Authors: Robin Klein; Katherine A Julian; Erin D Snyder; Jennifer Koch; Nneka N Ufere; Anna Volerman; Ann E Vandenberg; Sarah Schaeffer; Kerri Palamara Journal: J Gen Intern Med Date: 2019-05 Impact factor: 5.128
Authors: Delia L Gold; Jennifer R Marin; Demetris Haritos; L Melissa Skaugset; Jennifer M Kline; Rachel M Stanley; David P Way; David P Bahner Journal: AEM Educ Train Date: 2017-08-29