Joseph C Wildenberg1, Po-Hao Chen2, Mary H Scanlon2, Tessa S Cook2. 1. Department of Radiology, Hospital of the University of Pennsylvania, 3400 Spruce St., Philadelphia, PA 19104. Electronic address: joseph.wildenberg@uphs.upenn.edu. 2. Department of Radiology, Hospital of the University of Pennsylvania, 3400 Spruce St., Philadelphia, PA 19104.
Abstract
RATIONALE AND OBJECTIVES: Discrepancy rates for interpretations produced in a call situation are one metric to evaluate residents during training. Current benchmarks, reported in previous studies, do not consider the effects of practice pattern variability among attending radiologists. This study aims to investigate the impact of attending variability on resident discrepancy rates to determine if the current benchmarks are an accurate measure of resident performance and, if necessary, update discrepancy benchmarks to accurately identify residents performing below expectations. MATERIALS AND METHODS: All chest radiographs, musculoskeletal (MSK) radiographs, chest computed tomographies (CTs), abdomen and pelvis CTs, and head CTs interpreted by postgraduate year-3 residents in a call situation over 5 years were reviewed for the presence of a significant discrepancy and composite results compared to prior findings. Simulations of the expected discrepancy distribution for an "average resident" were then performed using Gibbs sampling, and this distribution was compared to the actual resident distribution. RESULTS: A strong inverse correlation between resident volume and discrepancy rates was found. There was wide variability among attendings in both overread volume and propensity to issue a discrepancy, although there was no significant correlation. Simulations show that previous benchmarks match well for chest radiographs, abdomen and pelvis CTs, and head CTs but not for MSK radiographs and chest CTs. The simulations also demonstrate a large effect of attending practice patterns on resident discrepancy rates. CONCLUSIONS: The large variability in attending practice patterns suggests direct comparison of residents using discrepancy rates is unlikely to reflect true performance. Current benchmarks for chest radiographs, abdomen and pelvis CTs, and head CTs are appropriate and correctly flag residents whose performance may benefit from additional attention, whereas those for MSK radiographs and chest CTs are likely too strict.
RATIONALE AND OBJECTIVES: Discrepancy rates for interpretations produced in a call situation are one metric to evaluate residents during training. Current benchmarks, reported in previous studies, do not consider the effects of practice pattern variability among attending radiologists. This study aims to investigate the impact of attending variability on resident discrepancy rates to determine if the current benchmarks are an accurate measure of resident performance and, if necessary, update discrepancy benchmarks to accurately identify residents performing below expectations. MATERIALS AND METHODS: All chest radiographs, musculoskeletal (MSK) radiographs, chest computed tomographies (CTs), abdomen and pelvis CTs, and head CTs interpreted by postgraduate year-3 residents in a call situation over 5 years were reviewed for the presence of a significant discrepancy and composite results compared to prior findings. Simulations of the expected discrepancy distribution for an "average resident" were then performed using Gibbs sampling, and this distribution was compared to the actual resident distribution. RESULTS: A strong inverse correlation between resident volume and discrepancy rates was found. There was wide variability among attendings in both overread volume and propensity to issue a discrepancy, although there was no significant correlation. Simulations show that previous benchmarks match well for chest radiographs, abdomen and pelvis CTs, and head CTs but not for MSK radiographs and chest CTs. The simulations also demonstrate a large effect of attending practice patterns on resident discrepancy rates. CONCLUSIONS: The large variability in attending practice patterns suggests direct comparison of residents using discrepancy rates is unlikely to reflect true performance. Current benchmarks for chest radiographs, abdomen and pelvis CTs, and head CTs are appropriate and correctly flag residents whose performance may benefit from additional attention, whereas those for MSK radiographs and chest CTs are likely too strict.
Authors: Michael Tran Duong; Andreas M Rauschecker; Jeffrey D Rudie; Po-Hao Chen; Tessa S Cook; R Nick Bryan; Suyash Mohan Journal: Br J Radiol Date: 2019-07-26 Impact factor: 3.039