Reed G Williams1, David B Swanson2, Jonathan P Fryer3, Shari L Meyerson4, Jordan D Bohnen5, Gary L Dunnington1, Rebecca E Scully6, Mary C Schuller3, Brian C George7. 1. Department of Surgery, Indiana University, Indianapolis, IN. 2. Academic Programs and Services, American Board of Medical Specialties, Chicago, IL. 3. Department of Surgery, Northwestern University, Evanston, IL. 4. Department of General and Thoracic Surgery, Northwestern University, Evanston, IL. 5. Department of General Surgery, Massachusetts General Hospital, Boston, MA. 6. Department of General Surgery, Brigham and Women's Hospital, Boston, MA. 7. Department of Surgery, University of Michigan, Ann Arbor, MI.
Abstract
OBJECTIVE: To establish the number of operative performance observations needed for reproducible assessments of operative competency. BACKGROUND: Surgical training is transitioning from a time-based to a competency-based approach, but the number of assessments needed to reliably establish operative competency remains unknown. METHODS: Using a smart phone based operative evaluation application (SIMPL), residents from 13 general surgery training programs were evaluated performing common surgical procedures. Two competency metrics were investigated separately: autonomy and overall performance. Analyses were performed for laparoscopic cholecystectomy performances alone and for all operative procedures combined. Variance component analyses determined operative performance score variance attributable to resident operative competency and measurement error. Generalizability and decision studies determined number of assessments needed to achieve desired reliability (0.80 or greater) and determine standard errors of measurement. RESULTS: For laparoscopic cholecystectomy, 23 ratings are needed to achieve reproducible autonomy ratings and 17 ratings are needed to achieve reproducible overall operative performance ratings. For the undifferentiated mix of procedures, 60 ratings are needed to achieve reproducible autonomy ratings and 40 are needed for reproducible overall operative performance ratings. CONCLUSION: The number of observations needed to achieve reproducible assessments of operative competency far exceeds current certification requirements, yet remains an important and achievable goal. Attention should also be paid to the mix of cases and raters in order to assure fair judgments about operative competency and fair comparisons of trainees.
OBJECTIVE: To establish the number of operative performance observations needed for reproducible assessments of operative competency. BACKGROUND: Surgical training is transitioning from a time-based to a competency-based approach, but the number of assessments needed to reliably establish operative competency remains unknown. METHODS: Using a smart phone based operative evaluation application (SIMPL), residents from 13 general surgery training programs were evaluated performing common surgical procedures. Two competency metrics were investigated separately: autonomy and overall performance. Analyses were performed for laparoscopic cholecystectomy performances alone and for all operative procedures combined. Variance component analyses determined operative performance score variance attributable to resident operative competency and measurement error. Generalizability and decision studies determined number of assessments needed to achieve desired reliability (0.80 or greater) and determine standard errors of measurement. RESULTS: For laparoscopic cholecystectomy, 23 ratings are needed to achieve reproducible autonomy ratings and 17 ratings are needed to achieve reproducible overall operative performance ratings. For the undifferentiated mix of procedures, 60 ratings are needed to achieve reproducible autonomy ratings and 40 are needed for reproducible overall operative performance ratings. CONCLUSION: The number of observations needed to achieve reproducible assessments of operative competency far exceeds current certification requirements, yet remains an important and achievable goal. Attention should also be paid to the mix of cases and raters in order to assure fair judgments about operative competency and fair comparisons of trainees.
Authors: Gideon Blumstein; Brian Zukotynski; Nicolas Cevallos; Chad Ishmael; Steven Zoller; Zach Burke; Samuel Clarkson; Howard Park; Nicholas Bernthal; Nelson F SooHoo Journal: J Surg Educ Date: 2020-02-05 Impact factor: 2.891
Authors: Lauren V Huckaby; Anthony R Cyr; Robert M Handzel; Eliza Beth Littleton; Lawrence R Crist; James D Luketich; Kenneth K Lee; Rajeev Dhupar Journal: Ann Thorac Surg Date: 2021-06-29 Impact factor: 4.330
Authors: Ebba K Lindqvist; Niels Noordzij; Shiv Chopra; Alfonso Navia; Laura Cappuyns; Amr Khalaf; Oscar F Fernandez Diaz; Juan Enrique Berner; Linda Monshizadeh; Robert X Murphy Journal: Plast Reconstr Surg Glob Open Date: 2022-10-07
Authors: Jenny X Chen; Elliott Kozin; Jordan Bohnen; Brian George; Daniel Deschler; Kevin Emerick; Stacey T Gray Journal: Laryngoscope Investig Otolaryngol Date: 2019-11-11