Steven J Skube1, Zhen Hu2, Gyorgy J Simon3, Elizabeth C Wick4, Elliot G Arsoniadis1,2, Clifford Y Ko5,6, Genevieve B Melton1,2. 1. Department of Surgery, University of Minnesota, Minneapolis, Minnesota. 2. lnstitute for Health Informatics, University of Minnesota, Minneapolis, Minnesota. 3. Department of Medicine, University of Minnesota, Minneapolis, Minnesota. 4. Department of Surgery, University of California San Francisco, San Francisco, California. 5. Department of Surgery, University of California Los Angeles, Los Angeles, California. 6. Division of Research and Optimal Patient Care, American College of Surgeons, Chicago, Illinois.
Abstract
OBJECTIVE: To demonstrate that a semi-automated approach to health data abstraction provides significant efficiencies and high accuracy. BACKGROUND: Surgical outcome abstraction remains laborious and a barrier to the sustainment of quality improvement registries like ACS-NSQIP. A supervised machine learning algorithm developed for detecting SSi using structured and unstructured electronic health record data was tested to perform semi-automated SSI abstraction. METHODS: A Lasso-penalized logistic regression model with 2011-3 data was trained (baseline performance measured with 10-fold cross-validation). A cutoff probability score from the training data was established, dividing the subsequent evaluation dataset into "negative" and "possible" SSI groups, with manual data abstraction only performed on the "possible" group. We evaluated performance on data from 2014, 2015, and both years. RESULTS: Overall, 6188 patients were in the 2011-3 training dataset and 5132 patients in the 2014-5 evaluation dataset. With use of the semi-automated approach, applying the cut-off score decreased the amount of manual abstraction by >90%, resulting in < 1% false negatives in the "negative" group and a sensitivity of 82%. A blinded review of 10% of the "possible" group, considering only the features selected by the algorithm, resulted in high agreement with the gold standard based on full chart abstraction, pointing towards additional efficiency in the abstraction process by making it possible for abstractors to review limited, salient portions of the chart. CONCLUSION: Semi-automated machine learning-aided SSI abstraction greatly accelerates the abstraction process and achieves very good performance. This could be translated to other post-operative outcomes and reduce cost barriers for wider ACS-NSQIP adoption.
OBJECTIVE: To demonstrate that a semi-automated approach to health data abstraction provides significant efficiencies and high accuracy. BACKGROUND: Surgical outcome abstraction remains laborious and a barrier to the sustainment of quality improvement registries like ACS-NSQIP. A supervised machine learning algorithm developed for detecting SSi using structured and unstructured electronic health record data was tested to perform semi-automated SSI abstraction. METHODS: A Lasso-penalized logistic regression model with 2011-3 data was trained (baseline performance measured with 10-fold cross-validation). A cutoff probability score from the training data was established, dividing the subsequent evaluation dataset into "negative" and "possible" SSI groups, with manual data abstraction only performed on the "possible" group. We evaluated performance on data from 2014, 2015, and both years. RESULTS: Overall, 6188 patients were in the 2011-3 training dataset and 5132 patients in the 2014-5 evaluation dataset. With use of the semi-automated approach, applying the cut-off score decreased the amount of manual abstraction by >90%, resulting in < 1% false negatives in the "negative" group and a sensitivity of 82%. A blinded review of 10% of the "possible" group, considering only the features selected by the algorithm, resulted in high agreement with the gold standard based on full chart abstraction, pointing towards additional efficiency in the abstraction process by making it possible for abstractors to review limited, salient portions of the chart. CONCLUSION: Semi-automated machine learning-aided SSI abstraction greatly accelerates the abstraction process and achieves very good performance. This could be translated to other post-operative outcomes and reduce cost barriers for wider ACS-NSQIP adoption.