Oanh Kieu Nguyen1,2,3, Colin Washington4, Christopher R Clark5, Michael E Miller6, Vivek A Patel4, Ethan A Halm4,6, Anil N Makam4,6,7. 1. Department of Internal Medicine, UT Southwestern, Dallas, TX, USA. Oanh.Nguyen@ucsf.edu. 2. Department of Population and Data Sciences, UT Southwestern, Dallas, TX, USA. Oanh.Nguyen@ucsf.edu. 3. Division of Hospital Medicine at San Francisco General Hospital, University of California, San Francisco, San Francisco, CA, USA. Oanh.Nguyen@ucsf.edu. 4. Department of Internal Medicine, UT Southwestern, Dallas, TX, USA. 5. Department of Research Administration, Parkland Health and Hospital System, Dallas, TX, USA. 6. Department of Population and Data Sciences, UT Southwestern, Dallas, TX, USA. 7. Division of Hospital Medicine at San Francisco General Hospital, University of California, San Francisco, San Francisco, CA, USA.
Abstract
BACKGROUND: Electronic health record (EHR)-based readmission risk prediction models can be automated in real-time but have modest discrimination and may be missing important readmission risk factors. Clinician predictions of readmissions may incorporate information unavailable in the EHR, but the comparative usefulness is unknown. We sought to compare clinicians versus a validated EHR-based prediction model in predicting 30-day hospital readmissions. METHODS: We conducted a prospective survey of internal medicine clinicians in an urban safety-net hospital. Clinicians prospectively predicted patients' 30-day readmission risk on 5-point Likert scales, subsequently dichotomized into low- vs. high-risk. We compared human with machine predictions using discrimination, net reclassification, and diagnostic test characteristics. Observed readmissions were ascertained from a regional hospitalization database. We also developed and assessed a "human-plus-machine" logistic regression model incorporating both human and machine predictions. RESULTS: We included 1183 hospitalizations from 106 clinicians, with a readmission rate of 20.8%. Both clinicians and the EHR model had similar discrimination (C-statistic 0.66 vs. 0.66, p = 0.91). Clinicians had higher specificity (79.0% vs. 48.9%, p < 0.001) but lower sensitivity (43.9 vs. 75.2%, p < 0.001) than EHR model predictions. Compared with machine, human was better at reclassifying non-readmissions (non-event NRI + 30.1%) but worse at reclassifying readmissions (event NRI - 31.3%). A human-plus-machine approach best optimized discrimination (C-statistic 0.70, 95% CI 0.67-0.74), sensitivity (65.5%), and specificity (66.7%). CONCLUSION: Clinicians had similar discrimination but higher specificity and lower sensitivity than EHR model predictions. Human-plus-machine was better than either alone. Readmission risk prediction strategies should incorporate clinician assessments to optimize the accuracy of readmission predictions.
BACKGROUND: Electronic health record (EHR)-based readmission risk prediction models can be automated in real-time but have modest discrimination and may be missing important readmission risk factors. Clinician predictions of readmissions may incorporate information unavailable in the EHR, but the comparative usefulness is unknown. We sought to compare clinicians versus a validated EHR-based prediction model in predicting 30-day hospital readmissions. METHODS: We conducted a prospective survey of internal medicine clinicians in an urban safety-net hospital. Clinicians prospectively predicted patients' 30-day readmission risk on 5-point Likert scales, subsequently dichotomized into low- vs. high-risk. We compared human with machine predictions using discrimination, net reclassification, and diagnostic test characteristics. Observed readmissions were ascertained from a regional hospitalization database. We also developed and assessed a "human-plus-machine" logistic regression model incorporating both human and machine predictions. RESULTS: We included 1183 hospitalizations from 106 clinicians, with a readmission rate of 20.8%. Both clinicians and the EHR model had similar discrimination (C-statistic 0.66 vs. 0.66, p = 0.91). Clinicians had higher specificity (79.0% vs. 48.9%, p < 0.001) but lower sensitivity (43.9 vs. 75.2%, p < 0.001) than EHR model predictions. Compared with machine, human was better at reclassifying non-readmissions (non-event NRI + 30.1%) but worse at reclassifying readmissions (event NRI - 31.3%). A human-plus-machine approach best optimized discrimination (C-statistic 0.70, 95% CI 0.67-0.74), sensitivity (65.5%), and specificity (66.7%). CONCLUSION: Clinicians had similar discrimination but higher specificity and lower sensitivity than EHR model predictions. Human-plus-machine was better than either alone. Readmission risk prediction strategies should incorporate clinician assessments to optimize the accuracy of readmission predictions.
Authors: Paul A Harris; Robert Taylor; Robert Thielke; Jonathon Payne; Nathaniel Gonzalez; Jose G Conde Journal: J Biomed Inform Date: 2008-09-30 Impact factor: 6.317
Authors: Oanh Kieu Nguyen; Anil N Makam; Christopher Clark; Song Zhang; Bin Xie; Ferdinand Velasco; Ruben Amarasingham; Ethan A Halm Journal: J Gen Intern Med Date: 2016-08-08 Impact factor: 5.128
Authors: Amy J H Kind; Steve Jencks; Jane Brock; Menggang Yu; Christie Bartels; William Ehlenbach; Caprice Greenberg; Maureen Smith Journal: Ann Intern Med Date: 2014-12-02 Impact factor: 25.391
Authors: Amol S Navathe; Feiran Zhong; Victor J Lei; Frank Y Chang; Margarita Sordo; Maxim Topaz; Shamkant B Navathe; Roberto A Rocha; Li Zhou Journal: Health Serv Res Date: 2017-03-13 Impact factor: 3.402
Authors: Samuel Kabue; John Greene; Patricia Kipnis; Brian Lawson; Gina Rinetti-Vargas; Vincent Liu; Gabriel Escobar Journal: Med Care Date: 2019-04 Impact factor: 2.983
Authors: John A Dodson; Alexandra M Hajduk; Terrence E Murphy; Mary Geda; Harlan M Krumholz; Sui Tsang; Michael G Nanna; Mary E Tinetti; David Goldstein; Daniel E Forman; Karen P Alexander; Thomas M Gill; Sarwat I Chaudhry Journal: Circ Cardiovasc Qual Outcomes Date: 2019-05
Authors: Wei Jiang; Sauleh Siddiqui; Sean Barnes; Lili A Barouch; Frederick Korley; Diego A Martinez; Matthew Toerper; Stephanie Cabral; Eric Hamrock; Scott Levin Journal: JMIR Med Inform Date: 2019-09-16
Authors: Anita D Misra-Hebert; Christina Felix; Alex Milinovich; Michael W Kattan; Marc A Willner; Kevin Chagin; Janine Bauman; Aaron C Hamilton; Jay Alberts Journal: J Gen Intern Med Date: 2022-02-07 Impact factor: 6.473