Patricia Kipnis1, Benjamin J Turk2, David A Wulf2, Juan Carlos LaGuardia2, Vincent Liu3, Matthew M Churpek4, Santiago Romero-Brufau5, Gabriel J Escobar6. 1. Kaiser Foundation Health Plan, Inc., 1950 Franklin St., 17th Floor, Oakland, CA 94612, United States; Kaiser Permanente Northern California, Division of Research, 2000 Broadway Avenue, 032 R01, Oakland, CA 94612, United States. Electronic address: patricia.kipnis@kp.org. 2. Kaiser Permanente Northern California, Division of Research, 2000 Broadway Avenue, 032 R01, Oakland, CA 94612, United States. 3. Kaiser Permanente Northern California, Division of Research, 2000 Broadway Avenue, 032 R01, Oakland, CA 94612, United States; Intensive Care Department, Kaiser Permanente Medical Center, 700 Lawrence Expressway, Santa Clara, CA 95051, United States. 4. Section of Pulmonary and Critical Care Medicine, Department of Medicine, University of Chicago, 5841 South Maryland Avenue, Chicago, IL 60637, United States. 5. Mayo Clinic Center for Innovation, 200 1st Street SW, Rochester, MN 55905, United States. 6. Kaiser Permanente Northern California, Division of Research, 2000 Broadway Avenue, 032 R01, Oakland, CA 94612, United States; Department of Inpatient Pediatrics, Kaiser Permanente Medical Center, 1425 S. Main Street Walnut Creek, CA 94596, United States.
Abstract
BACKGROUND: Patients in general medical-surgical wards who experience unplanned transfer to the intensive care unit (ICU) show evidence of physiologic derangement 6-24h prior to their deterioration. With increasing availability of electronic medical records (EMRs), automated early warning scores (EWSs) are becoming feasible. OBJECTIVE: To describe the development and performance of an automated EWS based on EMR data. MATERIALS AND METHODS: We used a discrete-time logistic regression model to obtain an hourly risk score to predict unplanned transfer to the ICU within the next 12h. The model was based on hospitalization episodes from all adult patients (18years) admitted to 21 Kaiser Permanente Northern California (KPNC) hospitals from 1/1/2010 to 12/31/2013. Eligible patients met these entry criteria: initial hospitalization occurred at a KPNC hospital; the hospitalization was not for childbirth; and the EMR had been operational at the hospital for at least 3months. We evaluated the performance of this risk score, called Advanced Alert Monitor (AAM) and compared it against two other EWSs (eCART and NEWS) in terms of their sensitivity, specificity, negative predictive value, positive predictive value, and area under the receiver operator characteristic curve (c statistic). RESULTS: A total of 649,418 hospitalization episodes involving 374,838 patients met inclusion criteria, with 19,153 of the episodes experiencing at least one outcome. The analysis data set had 48,723,248 hourly observations. Predictors included physiologic data (laboratory tests and vital signs); neurological status; severity of illness and longitudinal comorbidity indices; care directives; and health services indicators (e.g. elapsed length of stay). AAM showed better performance compared to NEWS and eCART in all the metrics and prediction intervals. The AAM AUC was 0.82 compared to 0.79 and 0.76 for eCART and NEWS, respectively. Using a threshold that generated 1 alert per day in a unit with a patient census of 35, the sensitivity of AAM was 49% (95% CI: 47.6-50.3%) compared to the sensitivities of eCART and NEWS scores of 44% (42.3-45.1) and 40% (38.2-40.9), respectively. For all three scores, about half of alerts occurred within 12h of the event, and almost two thirds within 24h of the event. CONCLUSION: The AAM score is an example of a score that takes advantage of multiple data streams now available in modern EMRs. It highlights the ability to harness complex algorithms to maximize signal extraction. The main challenge in the future is to develop detection approaches for patients in whom data are sparser because their baseline risk is lower. Copyright Â
BACKGROUND:Patients in general medical-surgical wards who experience unplanned transfer to the intensive care unit (ICU) show evidence of physiologic derangement 6-24h prior to their deterioration. With increasing availability of electronic medical records (EMRs), automated early warning scores (EWSs) are becoming feasible. OBJECTIVE: To describe the development and performance of an automated EWS based on EMR data. MATERIALS AND METHODS: We used a discrete-time logistic regression model to obtain an hourly risk score to predict unplanned transfer to the ICU within the next 12h. The model was based on hospitalization episodes from all adult patients (18years) admitted to 21 Kaiser Permanente Northern California (KPNC) hospitals from 1/1/2010 to 12/31/2013. Eligible patients met these entry criteria: initial hospitalization occurred at a KPNC hospital; the hospitalization was not for childbirth; and the EMR had been operational at the hospital for at least 3months. We evaluated the performance of this risk score, called Advanced Alert Monitor (AAM) and compared it against two other EWSs (eCART and NEWS) in terms of their sensitivity, specificity, negative predictive value, positive predictive value, and area under the receiver operator characteristic curve (c statistic). RESULTS: A total of 649,418 hospitalization episodes involving 374,838 patients met inclusion criteria, with 19,153 of the episodes experiencing at least one outcome. The analysis data set had 48,723,248 hourly observations. Predictors included physiologic data (laboratory tests and vital signs); neurological status; severity of illness and longitudinal comorbidity indices; care directives; and health services indicators (e.g. elapsed length of stay). AAM showed better performance compared to NEWS and eCART in all the metrics and prediction intervals. The AAM AUC was 0.82 compared to 0.79 and 0.76 for eCART and NEWS, respectively. Using a threshold that generated 1 alert per day in a unit with a patient census of 35, the sensitivity of AAM was 49% (95% CI: 47.6-50.3%) compared to the sensitivities of eCART and NEWS scores of 44% (42.3-45.1) and 40% (38.2-40.9), respectively. For all three scores, about half of alerts occurred within 12h of the event, and almost two thirds within 24h of the event. CONCLUSION: The AAM score is an example of a score that takes advantage of multiple data streams now available in modern EMRs. It highlights the ability to harness complex algorithms to maximize signal extraction. The main challenge in the future is to develop detection approaches for patients in whom data are sparser because their baseline risk is lower. Copyright Â
Authors: Gabriel J Escobar; John D Greene; Peter Scheirer; Marla N Gardner; David Draper; Patricia Kipnis Journal: Med Care Date: 2008-03 Impact factor: 2.983
Authors: Matthew M Churpek; Trevor C Yuen; Christopher Winslow; Ari A Robicsek; David O Meltzer; Robert D Gibbons; Dana P Edelson Journal: Am J Respir Crit Care Med Date: 2014-09-15 Impact factor: 21.405
Authors: Daniel T Linnen; Gabriel J Escobar; Xiao Hu; Elizabeth Scruth; Vincent Liu; Caroline Stephens Journal: J Hosp Med Date: 2019-03 Impact factor: 2.960
Authors: Alvin D Jeffery; Mary S Dietrich; Daniel Fabbri; Betsy Kennedy; Laurie L Novak; Joseph Coco; Lorraine C Mion Journal: Am J Crit Care Date: 2018-09 Impact factor: 2.228
Authors: Gabriel J Escobar; Vincent X Liu; Alejandro Schuler; Brian Lawson; John D Greene; Patricia Kipnis Journal: N Engl J Med Date: 2020-11-12 Impact factor: 91.245
Authors: Patrick G Lyons; Jeff Klaus; Colleen A McEvoy; Peter Westervelt; Brian F Gage; Marin H Kollef Journal: J Oncol Pract Date: 2019-07-15 Impact factor: 3.840
Authors: Kelly C Vranas; Jeffrey K Jopling; Timothy E Sweeney; Meghan C Ramsey; Arnold S Milstein; Christopher G Slatore; Gabriel J Escobar; Vincent X Liu Journal: Crit Care Med Date: 2017-10 Impact factor: 7.598