Daniel T Linnen1, Gabriel J Escobar2, Xiao Hu3, Elizabeth Scruth4, Vincent Liu2, Caroline Stephens5. 1. Kaiser Permanente Northern California, Kaiser Foundation Hospitals, Inc., Patient Care Services, Nurse Scholars Academy, Oakland, California. 2. Kaiser Permanente Northern California, The Permanente Medical Group, Inc., Division of Research, Oakland, California. 3. University of California, San Francisco, School of Nursing, Department of Physiological Nursing, San Francisco, California. 4. Kaiser Permanente Northern California, Kaiser Foundation Hospitals, Inc., Department of Quality, Oakland, California. 5. University of California, San Francisco, School of Nursing, Department of Community Health Systems, San Francisco, California.
Abstract
BACKGROUND: The clinical deterioration of patientsin general hospital wards is an important safety issue. Aggregate-weighted early warning systems (EWSs) may not detect risk until patients present with acute decline. PURPOSE: We aimed to compare the prognostic test accuracy and clinical workloads generated by EWSs using statistical modeling (multivariable regression or machine learning) versus aggregate-weighted tools. DATA SOURCES: We searched PubMed and CINAHL using terms that described clinical deterioration and use of an advanced EWS. STUDY SELECTION: The outcome was clinical deterioration (intensive care unit transfer or death) of adult patients on general hospital wards. We included studies published from January 1, 2012 to September 15, 2018. DATA EXTRACTION: Following 2015 PRIMSA systematic review protocol guidelines; 2015 TRIPOD criteria for predictive model evaluation; and the Cochrane Collaboration guidelines, we reported model performance, adjusted positive predictive value (PPV), and conducted simulations of workup-to-detection ratios. DATA SYNTHESIS: Of 285 articles, six studies reported the model performance of advanced EWSs, and five were of high quality. All EWSs using statistical modeling identified at-risk patients with greater precision than aggregate-weighted EWSs (mean AUC 0.80 vs 0.73). EWSs using statistical modeling generated 4.9 alerts to find one true positive case versus 7.1 alerts in aggregate-weighted EWSs; a nearly 50% relative workload increase for aggregate-weighted EWSs. CONCLUSIONS: Compared with aggregate-weighted tools, EWSs using statistical modeling consistently demonstrated superior prognostic performance and generated less workload to identify and treat one true positive case. A standardized approach to reporting EWS model performance is needed, including outcome definitions, pretest probability, observed and adjusted PPV, and workup-to-detection ratio.
BACKGROUND: The clinical deterioration of patientsin general hospital wards is an important safety issue. Aggregate-weighted early warning systems (EWSs) may not detect risk until patients present with acute decline. PURPOSE: We aimed to compare the prognostic test accuracy and clinical workloads generated by EWSs using statistical modeling (multivariable regression or machine learning) versus aggregate-weighted tools. DATA SOURCES: We searched PubMed and CINAHL using terms that described clinical deterioration and use of an advanced EWS. STUDY SELECTION: The outcome was clinical deterioration (intensive care unit transfer or death) of adult patients on general hospital wards. We included studies published from January 1, 2012 to September 15, 2018. DATA EXTRACTION: Following 2015 PRIMSA systematic review protocol guidelines; 2015 TRIPOD criteria for predictive model evaluation; and the Cochrane Collaboration guidelines, we reported model performance, adjusted positive predictive value (PPV), and conducted simulations of workup-to-detection ratios. DATA SYNTHESIS: Of 285 articles, six studies reported the model performance of advanced EWSs, and five were of high quality. All EWSs using statistical modeling identified at-risk patients with greater precision than aggregate-weighted EWSs (mean AUC 0.80 vs 0.73). EWSs using statistical modeling generated 4.9 alerts to find one true positive case versus 7.1 alerts in aggregate-weighted EWSs; a nearly 50% relative workload increase for aggregate-weighted EWSs. CONCLUSIONS: Compared with aggregate-weighted tools, EWSs using statistical modeling consistently demonstrated superior prognostic performance and generated less workload to identify and treat one true positive case. A standardized approach to reporting EWS model performance is needed, including outcome definitions, pretest probability, observed and adjusted PPV, and workup-to-detection ratio.
Authors: Bradford D Winters; Julius Cuong Pham; Elizabeth A Hunt; Eliseo Guallar; Sean Berenholtz; Peter J Pronovost Journal: Crit Care Med Date: 2007-05 Impact factor: 7.598
Authors: Gabriel J Escobar; Vincent X Liu; Alejandro Schuler; Brian Lawson; John D Greene; Patricia Kipnis Journal: N Engl J Med Date: 2020-11-12 Impact factor: 91.245
Authors: Kais Gadhoumi; Alex Beltran; Christopher G Scully; Ran Xiao; David O Nahmias; Xiao Hu Journal: Physiol Meas Date: 2021-06-17 Impact factor: 2.688
Authors: Stephen Gerry; Timothy Bonnici; Jacqueline Birks; Shona Kirtley; Pradeep S Virdee; Peter J Watkinson; Gary S Collins Journal: BMJ Date: 2020-05-20
Authors: Walter Nelson; Shuang Di; Sankavi Muralitharan; Michael McGillion; P J Devereaux; Neil Grant Barr; Jeremy Petch Journal: J Med Internet Res Date: 2021-02-04 Impact factor: 5.428
Authors: Brandon C Cummings; Sardar Ansari; Jonathan R Motyka; Guan Wang; Richard P Medlin; Steven L Kronick; Karandeep Singh; Pauline K Park; Lena M Napolitano; Robert P Dickson; Michael R Mathis; Michael W Sjoding; Andrew J Admon; Ross Blank; Jakob I McSparron; Kevin R Ward; Christopher E Gillies Journal: JMIR Med Inform Date: 2021-04-21
Authors: Richard John Woodman; Kimberley Bryant; Michael J Sorich; Alberto Pilotto; Arduino Aleksander Mangoni Journal: J Med Internet Res Date: 2021-06-21 Impact factor: 5.428
Authors: Sarah Collins Rossetti; Chris Knaplund; Dave Albers; Patricia C Dykes; Min Jeoung Kang; Tom Z Korach; Li Zhou; Kumiko Schnock; Jose Garcia; Jessica Schwartz; Li-Heng Fu; Jeffrey G Klann; Graham Lowenthal; Kenrick Cato Journal: J Am Med Inform Assoc Date: 2021-06-12 Impact factor: 4.497