Juliet J Ray1, Joshua A Sznol2, Laura F Teisch1, Jonathan P Meizoso1, Casey J Allen1, Nicholas Namias1, Louis R Pizano1, Danny Sleeman1, Seth A Spector3, Carl I Schulman1. 1. DeWitt Daughtry Family Department of Surgery, University of Miami Miller School of Medicine, Miami, Florida. 2. Department of Public Health Sciences, University of Miami Miller School of Medicine, Miami, Florida. 3. DeWitt Daughtry Family Department of Surgery, University of Miami Miller School of Medicine, Miami, Florida3Department of Surgery, Bruce W. Carter Department of Veterans Affairs Medical Center, Miami, Florida.
Abstract
IMPORTANCE: The American Board of Surgery In-Training Examination (ABSITE) is designed to measure progress, applied medical knowledge, and clinical management; results may determine promotion and fellowship candidacy for general surgery residents. Evaluations are mandated by the Accreditation Council for Graduate Medical Education but are administered at the discretion of individual institutions and are not standardized. It is unclear whether the ABSITE and evaluations form a reasonable assessment of resident performance. OBJECTIVE: To determine whether favorable evaluations are associated with ABSITE performance. DESIGN, SETTING, AND PARTICIPANTS: Cross-sectional analysis of preliminary and categorical residents in postgraduate years (PGYs) 1 through 5 training in a single university-based general surgery program from July 1, 2011, through June 30, 2014, who took the ABSITE. EXPOSURES: Evaluation overall performance and subset evaluation performance in the following categories: patient care, technical skills, problem-based learning, interpersonal and communication skills, professionalism, systems-based practice, and medical knowledge. MAIN OUTCOMES AND MEASURES: Passing the ABSITE (≥30th percentile) and ranking in the top 30% of scores at our institution. RESULTS: The study population comprised residents in PGY 1 (n = 44), PGY 2 (n = 31), PGY 3 (n = 26), PGY 4 (n = 25), and PGY 5 (n = 24) during the 4-year study period (N = 150). Evaluations had less variation than the ABSITE percentile (SD = 5.06 vs 28.82, respectively). Neither annual nor subset evaluation scores were significantly associated with passing the ABSITE (n = 102; for annual evaluation, odds ratio = 0.949; 95% CI, 0.884-1.019; P = .15) or receiving a top 30% score (n = 45; for annual evaluation, odds ratio = 1.036; 95% CI, 0.964-1.113; P = .33). There was no difference in mean evaluation score between those who passed vs failed the ABSITE (mean [SD] evaluation score, 91.77 [5.10] vs 93.04 [4.80], respectively; P = .14) or between those who received a top 30% score vs those who did not (mean [SD] evaluation score, 92.78 [4.83] vs 91.92 [5.11], respectively; P = .33). There was no correlation between annual evaluation score and ABSITE percentile (r(2) = 0.014; P = .15), percentage correct unadjusted for PGY level (r(2) = 0.019; P = .09), or percentage correct adjusted for PGY level (r(2) = 0.429; P = .91). CONCLUSIONS AND RELEVANCE: Favorable evaluations do not correlate with ABSITE scores, nor do they predict passing. Evaluations do not show much discriminatory ability. It is unclear whether individual resident evaluations and ABSITE scores fully assess competency in residents or allow comparisons to be made across programs. Creation of a uniform evaluation system that encompasses the necessary subjective feedback from faculty with the objective measure of the ABSITE is warranted.
IMPORTANCE: The American Board of Surgery In-Training Examination (ABSITE) is designed to measure progress, applied medical knowledge, and clinical management; results may determine promotion and fellowship candidacy for general surgery residents. Evaluations are mandated by the Accreditation Council for Graduate Medical Education but are administered at the discretion of individual institutions and are not standardized. It is unclear whether the ABSITE and evaluations form a reasonable assessment of resident performance. OBJECTIVE: To determine whether favorable evaluations are associated with ABSITE performance. DESIGN, SETTING, AND PARTICIPANTS: Cross-sectional analysis of preliminary and categorical residents in postgraduate years (PGYs) 1 through 5 training in a single university-based general surgery program from July 1, 2011, through June 30, 2014, who took the ABSITE. EXPOSURES: Evaluation overall performance and subset evaluation performance in the following categories: patient care, technical skills, problem-based learning, interpersonal and communication skills, professionalism, systems-based practice, and medical knowledge. MAIN OUTCOMES AND MEASURES: Passing the ABSITE (≥30th percentile) and ranking in the top 30% of scores at our institution. RESULTS: The study population comprised residents in PGY 1 (n = 44), PGY 2 (n = 31), PGY 3 (n = 26), PGY 4 (n = 25), and PGY 5 (n = 24) during the 4-year study period (N = 150). Evaluations had less variation than the ABSITE percentile (SD = 5.06 vs 28.82, respectively). Neither annual nor subset evaluation scores were significantly associated with passing the ABSITE (n = 102; for annual evaluation, odds ratio = 0.949; 95% CI, 0.884-1.019; P = .15) or receiving a top 30% score (n = 45; for annual evaluation, odds ratio = 1.036; 95% CI, 0.964-1.113; P = .33). There was no difference in mean evaluation score between those who passed vs failed the ABSITE (mean [SD] evaluation score, 91.77 [5.10] vs 93.04 [4.80], respectively; P = .14) or between those who received a top 30% score vs those who did not (mean [SD] evaluation score, 92.78 [4.83] vs 91.92 [5.11], respectively; P = .33). There was no correlation between annual evaluation score and ABSITE percentile (r(2) = 0.014; P = .15), percentage correct unadjusted for PGY level (r(2) = 0.019; P = .09), or percentage correct adjusted for PGY level (r(2) = 0.429; P = .91). CONCLUSIONS AND RELEVANCE: Favorable evaluations do not correlate with ABSITE scores, nor do they predict passing. Evaluations do not show much discriminatory ability. It is unclear whether individual resident evaluations and ABSITE scores fully assess competency in residents or allow comparisons to be made across programs. Creation of a uniform evaluation system that encompasses the necessary subjective feedback from faculty with the objective measure of the ABSITE is warranted.