Andrew Wong1, Erkin Otles2,3, John P Donnelly4, Andrew Krumm4, Jeffrey McCullough5, Olivia DeTroyer-Cooley6, Justin Pestrue6, Marie Phillips7, Judy Konye8, Carleen Penoza8, Muhammad Ghous4, Karandeep Singh1,4. 1. Department of Internal Medicine, University of Michigan Medical School, Ann Arbor. 2. Medical Scientist Training Program, University of Michigan Medical School, Ann Arbor. 3. Department of Industrial and Operations Engineering, University of Michigan College of Engineering, Ann Arbor. 4. Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor. 5. School of Public Health, University of Michigan, Ann Arbor. 6. Department of Quality, Michigan Medicine, Ann Arbor. 7. Health Information Technology and Services, Michigan Medicine, Ann Arbor. 8. Nursing Informatics, Michigan Medicine, Ann Arbor.
Abstract
Importance: The Epic Sepsis Model (ESM), a proprietary sepsis prediction model, is implemented at hundreds of US hospitals. The ESM's ability to identify patients with sepsis has not been adequately evaluated despite widespread use. Objective: To externally validate the ESM in the prediction of sepsis and evaluate its potential clinical value compared with usual care. Design, Setting, and Participants: This retrospective cohort study was conducted among 27 697 patients aged 18 years or older admitted to Michigan Medicine, the academic health system of the University of Michigan, Ann Arbor, with 38 455 hospitalizations between December 6, 2018, and October 20, 2019. Exposure: The ESM score, calculated every 15 minutes. Main Outcomes and Measures: Sepsis, as defined by a composite of (1) the Centers for Disease Control and Prevention surveillance criteria and (2) International Statistical Classification of Diseases and Related Health Problems, Tenth Revision diagnostic codes accompanied by 2 systemic inflammatory response syndrome criteria and 1 organ dysfunction criterion within 6 hours of one another. Model discrimination was assessed using the area under the receiver operating characteristic curve at the hospitalization level and with prediction horizons of 4, 8, 12, and 24 hours. Model calibration was evaluated with calibration plots. The potential clinical benefit associated with the ESM was assessed by evaluating the added benefit of the ESM score compared with contemporary clinical practice (based on timely administration of antibiotics). Alert fatigue was evaluated by comparing the clinical value of different alerting strategies. Results: We identified 27 697 patients who had 38 455 hospitalizations (21 904 women [57%]; median age, 56 years [interquartile range, 35-69 years]) meeting inclusion criteria, of whom sepsis occurred in 2552 (7%). The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64). The ESM identified 183 of 2552 patients with sepsis (7%) who did not receive timely administration of antibiotics, highlighting the low sensitivity of the ESM in comparison with contemporary clinical practice. The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue. Conclusions and Relevance: This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis. The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.
Importance: The Epic Sepsis Model (ESM), a proprietary sepsis prediction model, is implemented at hundreds of US hospitals. The ESM's ability to identify patients with sepsis has not been adequately evaluated despite widespread use. Objective: To externally validate the ESM in the prediction of sepsis and evaluate its potential clinical value compared with usual care. Design, Setting, and Participants: This retrospective cohort study was conducted among 27 697 patients aged 18 years or older admitted to Michigan Medicine, the academic health system of the University of Michigan, Ann Arbor, with 38 455 hospitalizations between December 6, 2018, and October 20, 2019. Exposure: The ESM score, calculated every 15 minutes. Main Outcomes and Measures: Sepsis, as defined by a composite of (1) the Centers for Disease Control and Prevention surveillance criteria and (2) International Statistical Classification of Diseases and Related Health Problems, Tenth Revision diagnostic codes accompanied by 2 systemic inflammatory response syndrome criteria and 1 organ dysfunction criterion within 6 hours of one another. Model discrimination was assessed using the area under the receiver operating characteristic curve at the hospitalization level and with prediction horizons of 4, 8, 12, and 24 hours. Model calibration was evaluated with calibration plots. The potential clinical benefit associated with the ESM was assessed by evaluating the added benefit of the ESM score compared with contemporary clinical practice (based on timely administration of antibiotics). Alert fatigue was evaluated by comparing the clinical value of different alerting strategies. Results: We identified 27 697 patients who had 38 455 hospitalizations (21 904 women [57%]; median age, 56 years [interquartile range, 35-69 years]) meeting inclusion criteria, of whom sepsis occurred in 2552 (7%). The ESM had a hospitalization-level area under the receiver operating characteristic curve of 0.63 (95% CI, 0.62-0.64). The ESM identified 183 of 2552 patients with sepsis (7%) who did not receive timely administration of antibiotics, highlighting the low sensitivity of the ESM in comparison with contemporary clinical practice. The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue. Conclusions and Relevance: This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis. The widespread adoption of the ESM despite its poor performance raises fundamental concerns about sepsis management on a national level.
Authors: Polina V Kukhareva; Charlene Weir; Guilherme Del Fiol; Gregory A Aarons; Teresa Y Taft; Chelsey R Schlechter; Thomas J Reese; Rebecca L Curran; Claude Nanjo; Damian Borbolla; Catherine J Staes; Keaton L Morgan; Heidi S Kramer; Carole H Stipelman; Julie H Shakib; Michael C Flynn; Kensaku Kawamoto Journal: J Biomed Inform Date: 2022-02-12 Impact factor: 6.317
Authors: Anthony E Solomonides; Eileen Koski; Shireen M Atabaki; Scott Weinberg; John D McGreevey; Joseph L Kannry; Carolyn Petersen; Christoph U Lehmann Journal: J Am Med Inform Assoc Date: 2022-03-15 Impact factor: 4.497
Authors: Wonsuk Oh; Pushkala Jayaraman; Ashwin S Sawant; Lili Chan; Matthew A Levin; Alexander W Charney; Patricia Kovatch; Benjamin S Glicksberg; Girish N Nadkarni Journal: J Am Med Inform Assoc Date: 2022-01-29 Impact factor: 4.497
Authors: Sunyang Fu; Andrew Wen; Gavin M Schaeferle; Patrick M Wilson; Gabriel Demuth; Xiaoyang Ruan; Sijia Liu; Curtis Storlie; Hongfang Liu Journal: AMIA Annu Symp Proc Date: 2022-05-23
Authors: Sayon Dutta; Dustin S McEvoy; David M Rubins; Anand S Dighe; Michael R Filbin; Chanu Rhee Journal: J Am Med Inform Assoc Date: 2022-09-12 Impact factor: 7.942
Authors: Sriram Ramgopal; L Nelson Sanchez-Pinto; Christopher M Horvat; Michael S Carroll; Yuan Luo; Todd A Florin Journal: Pediatr Res Date: 2022-07-29 Impact factor: 3.953