Ben J Marafino1, W John Boscardin2, R Adams Dudley3. 1. Philip R. Lee Institute for Health Policy Studies, School of Medicine, University of California, San Francisco, United States; Center for Healthcare Value, University of California, San Francisco, United States. Electronic address: ben.marafino@ucsf.edu. 2. Department of Epidemiology and Biostatistics, University of California, San Francisco, United States; Department of Medicine, University of California, San Francisco, United States. 3. Philip R. Lee Institute for Health Policy Studies, School of Medicine, University of California, San Francisco, United States; Center for Healthcare Value, University of California, San Francisco, United States; Department of Epidemiology and Biostatistics, University of California, San Francisco, United States; Department of Medicine, University of California, San Francisco, United States.
Abstract
BACKGROUND AND SIGNIFICANCE: Sparsity is often a desirable property of statistical models, and various feature selection methods exist so as to yield sparser and interpretable models. However, their application to biomedical text classification, particularly to mortality risk stratification among intensive care unit (ICU) patients, has not been thoroughly studied. OBJECTIVE: To develop and characterize sparse classifiers based on the free text of nursing notes in order to predict ICU mortality risk and to discover text features most strongly associated with mortality. METHODS: We selected nursing notes from the first 24h of ICU admission for 25,826 adult ICU patients from the MIMIC-II database. We then developed a pair of stochastic gradient descent-based classifiers with elastic-net regularization. We also studied the performance-sparsity tradeoffs of both classifiers as their regularization parameters were varied. RESULTS: The best-performing classifier achieved a 10-fold cross-validated AUC of 0.897 under the log loss function and full L2 regularization, while full L1 regularization used just 0.00025% of candidate input features and resulted in an AUC of 0.889. Using the log loss (range of AUCs 0.889-0.897) yielded better performance compared to the hinge loss (0.850-0.876), but the latter yielded even sparser models. DISCUSSION: Most features selected by both classifiers appear clinically relevant and correspond to predictors already present in existing ICU mortality models. The sparser classifiers were also able to discover a number of informative - albeit nonclinical - features. CONCLUSION: The elastic-net-regularized classifiers perform reasonably well and are capable of reducing the number of features required by over a thousandfold, with only a modest impact on performance.
BACKGROUND AND SIGNIFICANCE: Sparsity is often a desirable property of statistical models, and various feature selection methods exist so as to yield sparser and interpretable models. However, their application to biomedical text classification, particularly to mortality risk stratification among intensive care unit (ICU) patients, has not been thoroughly studied. OBJECTIVE: To develop and characterize sparse classifiers based on the free text of nursing notes in order to predict ICU mortality risk and to discover text features most strongly associated with mortality. METHODS: We selected nursing notes from the first 24h of ICU admission for 25,826 adult ICU patients from the MIMIC-II database. We then developed a pair of stochastic gradient descent-based classifiers with elastic-net regularization. We also studied the performance-sparsity tradeoffs of both classifiers as their regularization parameters were varied. RESULTS: The best-performing classifier achieved a 10-fold cross-validated AUC of 0.897 under the log loss function and full L2 regularization, while full L1 regularization used just 0.00025% of candidate input features and resulted in an AUC of 0.889. Using the log loss (range of AUCs 0.889-0.897) yielded better performance compared to the hinge loss (0.850-0.876), but the latter yielded even sparser models. DISCUSSION: Most features selected by both classifiers appear clinically relevant and correspond to predictors already present in existing ICU mortality models. The sparser classifiers were also able to discover a number of informative - albeit nonclinical - features. CONCLUSION: The elastic-net-regularized classifiers perform reasonably well and are capable of reducing the number of features required by over a thousandfold, with only a modest impact on performance.
Authors: Gary E Weissman; Rebecca A Hubbard; Lyle H Ungar; Michael O Harhay; Casey S Greene; Blanca E Himes; Scott D Halpern Journal: Crit Care Med Date: 2018-07 Impact factor: 7.598
Authors: Robert M Cronin; Daniel Fabbri; Joshua C Denny; S Trent Rosenbloom; Gretchen Purcell Jackson Journal: Int J Med Inform Date: 2017-06-23 Impact factor: 4.046
Authors: Tomasz Oliwa; Brian Furner; Jessica Schmitt; John Schneider; Jessica P Ridgway Journal: J Am Med Inform Assoc Date: 2021-01-15 Impact factor: 4.497
Authors: Gary E Weissman; Lyle H Ungar; Michael O Harhay; Katherine R Courtright; Scott D Halpern Journal: J Biomed Inform Date: 2018-12-14 Impact factor: 6.317
Authors: Danqing Hu; Zhengxing Huang; Tak-Ming Chan; Wei Dong; Xudong Lu; Huilong Duan Journal: Int J Environ Res Public Health Date: 2016-09-13 Impact factor: 3.390