Maura E Sullivan1, Yoon Soo Park2, Kathy Liscum3, Ajit K Sachdeva4, Patrice Gabler Blair4, Melissa Gesbeck4, Georges Bordage2. 1. Department of Surgery, Keck School of Medicine, University of Southern California, Los Angeles, CA. 2. Department of Medical Education, College of Medicine, University of Illinois at Chicago, Chicago, IL. 3. Department of Surgery, Baylor College of Medicine, Houston, TX. 4. Division of Education, American College of Surgeons, Chicago, IL.
Abstract
OBJECTIVE: To assess the readiness of entering residents for clinical responsibilities, the American College of Surgeons (ACS) Division of Education developed the "Entering Resident Readiness Assessment" (ACS-ERRA) Program. SUMMARY BACKGROUND: ACS-ERRA is an online formative assessment that uses a key features approach to measure clinical decision-making skills and focuses on cases encountered at the beginning of residency. Results can be used to develop learning plans to address areas that may need reinforcement. METHODS: A national panel of 16 content experts, 3 medical educators, and a psychometrician developed 98 short, key features cases. Each case required medical knowledge to be applied appropriately at challenging decision points during case management. Four pilot testing studies were conducted sequentially to gather validity evidence. RESULTS: Residents from programs across the United States participated in the studies (n = 58, 20, 87, 154, respectively). Results from the pilot studies enabled improvements after each pilot test. For the psychometric pilot (final pilot test), 2 parallel test forms of the ACS-ERRA were administered, each containing 40 cases, resulting in overall mean testing time of 2 hours 2 minutes (SD = 43 min). The mean test score was 61% (SD = 9%) and the G-coefficient reliability was 0.90. CONCLUSIONS: Results can be used to identify strengths and weaknesses in residents' decision-making skills and yield valuable information to create individualized learning plans. The data can also support efforts directed at the transition into residency training and inform discussions about levels of supervision. In addition, surgery program directors can use the aggregate test results to make curricular changes.
OBJECTIVE: To assess the readiness of entering residents for clinical responsibilities, the American College of Surgeons (ACS) Division of Education developed the "Entering Resident Readiness Assessment" (ACS-ERRA) Program. SUMMARY BACKGROUND: ACS-ERRA is an online formative assessment that uses a key features approach to measure clinical decision-making skills and focuses on cases encountered at the beginning of residency. Results can be used to develop learning plans to address areas that may need reinforcement. METHODS: A national panel of 16 content experts, 3 medical educators, and a psychometrician developed 98 short, key features cases. Each case required medical knowledge to be applied appropriately at challenging decision points during case management. Four pilot testing studies were conducted sequentially to gather validity evidence. RESULTS: Residents from programs across the United States participated in the studies (n = 58, 20, 87, 154, respectively). Results from the pilot studies enabled improvements after each pilot test. For the psychometric pilot (final pilot test), 2 parallel test forms of the ACS-ERRA were administered, each containing 40 cases, resulting in overall mean testing time of 2 hours 2 minutes (SD = 43 min). The mean test score was 61% (SD = 9%) and the G-coefficient reliability was 0.90. CONCLUSIONS: Results can be used to identify strengths and weaknesses in residents' decision-making skills and yield valuable information to create individualized learning plans. The data can also support efforts directed at the transition into residency training and inform discussions about levels of supervision. In addition, surgery program directors can use the aggregate test results to make curricular changes.
Authors: AlleaBelle Gongola; Jared T Gowen; Rebecca J Reif; Carol R Thrush; Hamilton Newhart; Molly Peckham; Zachary Schwartz; David Davies; Mary Katherine Kimbrough Journal: Med Sci Educ Date: 2021-06-21