Siru Liu1, Kensaku Kawamoto1, Guilherme Del Fiol1, Charlene Weir1, Daniel C Malone2, Thomas J Reese1,3, Keaton Morgan1, David ElHalta4, Samir Abdelrahman1,5. 1. Department of Biomedical Informatics, University of Utah, Salt Lake City, Utah, USA. 2. Department of Pharmacotherapy, Skaggs College of Pharmacy, University of Utah, Salt Lake City, Utah, USA. 3. Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA. 4. Pharmacy Services, University of Utah, Salt Lake City, Utah, USA. 5. Computer Science Department, Faculty of Computers and Artificial Intelligence, Cairo University, Giza, Egypt.
Abstract
OBJECTIVE: To evaluate the potential for machine learning to predict medication alerts that might be ignored by a user, and intelligently filter out those alerts from the user's view. MATERIALS AND METHODS: We identified features (eg, patient and provider characteristics) proposed to modulate user responses to medication alerts through the literature; these features were then refined through expert review. Models were developed using rule-based and machine learning techniques (logistic regression, random forest, support vector machine, neural network, and LightGBM). We collected log data on alerts shown to users throughout 2019 at University of Utah Health. We sought to maximize precision while maintaining a false-negative rate <0.01, a threshold predefined through discussion with physicians and pharmacists. We developed models while maintaining a sensitivity of 0.99. Two null hypotheses were developed: H1-there is no difference in precision among prediction models; and H2-the removal of any feature category does not change precision. RESULTS: A total of 3,481,634 medication alerts with 751 features were evaluated. With sensitivity fixed at 0.99, LightGBM achieved the highest precision of 0.192 and less than 0.01 for the pre-defined maximal false-negative rate by subject-matter experts (H1) (P < 0.001). This model could reduce alert volume by 54.1%. We removed different combinations of features (H2) and found that not all features significantly contributed to precision. Removing medication order features (eg, dosage) most significantly decreased precision (-0.147, P = 0.001). CONCLUSIONS: Machine learning potentially enables the intelligent filtering of medication alerts.
OBJECTIVE: To evaluate the potential for machine learning to predict medication alerts that might be ignored by a user, and intelligently filter out those alerts from the user's view. MATERIALS AND METHODS: We identified features (eg, patient and provider characteristics) proposed to modulate user responses to medication alerts through the literature; these features were then refined through expert review. Models were developed using rule-based and machine learning techniques (logistic regression, random forest, support vector machine, neural network, and LightGBM). We collected log data on alerts shown to users throughout 2019 at University of Utah Health. We sought to maximize precision while maintaining a false-negative rate <0.01, a threshold predefined through discussion with physicians and pharmacists. We developed models while maintaining a sensitivity of 0.99. Two null hypotheses were developed: H1-there is no difference in precision among prediction models; and H2-the removal of any feature category does not change precision. RESULTS: A total of 3,481,634 medication alerts with 751 features were evaluated. With sensitivity fixed at 0.99, LightGBM achieved the highest precision of 0.192 and less than 0.01 for the pre-defined maximal false-negative rate by subject-matter experts (H1) (P < 0.001). This model could reduce alert volume by 54.1%. We removed different combinations of features (H2) and found that not all features significantly contributed to precision. Removing medication order features (eg, dosage) most significantly decreased precision (-0.147, P = 0.001). CONCLUSIONS: Machine learning potentially enables the intelligent filtering of medication alerts.
Authors: Jerome A Osheroff; Jonathan M Teich; Blackford Middleton; Elaine B Steen; Adam Wright; Don E Detmer Journal: J Am Med Inform Assoc Date: 2007-01-09 Impact factor: 4.497
Authors: Hanna M Seidling; Shobha Phansalkar; Diane L Seger; Marilyn D Paterno; Shimon Shaykevich; Walter E Haefeli; David W Bates Journal: J Am Med Inform Assoc Date: 2011-05-12 Impact factor: 4.497
Authors: Hanna M Seidling; Ulrike Klein; Matthias Schaier; David Czock; Dirk Theile; Markus G Pruszydlo; Jens Kaltschmidt; Gerd Mikus; Walter E Haefeli Journal: Int J Med Inform Date: 2014-01-13 Impact factor: 4.046
Authors: Sandra L Kane-Gill; Michael F O'Connor; Jeffrey M Rothschild; Nicholas M Selby; Barbara McLean; Christopher P Bonafide; Maria M Cvach; Xiao Hu; Avinash Konkani; Michele M Pelter; Bradford D Winters Journal: Crit Care Med Date: 2017-09 Impact factor: 7.598
Authors: Daniel Riedmann; Martin Jung; Werner O Hackl; Wolf Stühlinger; Heleen van der Sijs; Elske Ammenwerth Journal: BMC Med Inform Decis Mak Date: 2011-05-25 Impact factor: 2.796