Jonathan H Chen1, Tanya Podchiyska2, Russ B Altman3. 1. Center for Innovation to Implementation (Ci2i), Veterans Affairs Palo Alto Health Care System, Palo Alto, CA, USA Center for Primary Care and Outcomes Research (PCOR), Stanford University, Stanford, CA, USA. 2. Biomedical Informatics Training Program, Stanford University, Stanford, CA, USA. 3. Departments of Bioengineering, Genetics, and Medicine, Stanford University, Stanford, CA, USA russ.altman@stanford.edu.
Abstract
OBJECTIVE: To answer a "grand challenge" in clinical decision support, the authors produced a recommender system that automatically data-mines inpatient decision support from electronic medical records (EMR), analogous to Netflix or Amazon.com's product recommender. MATERIALS AND METHODS: EMR data were extracted from 1 year of hospitalizations (>18K patients with >5.4M structured items including clinical orders, lab results, and diagnosis codes). Association statistics were counted for the ∼1.5K most common items to drive an order recommender. The authors assessed the recommender's ability to predict hospital admission orders and outcomes based on initial encounter data from separate validation patients. RESULTS: Compared to a reference benchmark of using the overall most common orders, the recommender using temporal relationships improves precision at 10 recommendations from 33% to 38% (P < 10(-10)) for hospital admission orders. Relative risk-based association methods improve inverse frequency weighted recall from 4% to 16% (P < 10(-16)). The framework yields a prediction receiver operating characteristic area under curve (c-statistic) of 0.84 for 30 day mortality, 0.84 for 1 week need for ICU life support, 0.80 for 1 week hospital discharge, and 0.68 for 30-day readmission. DISCUSSION: Recommender results quantitatively improve on reference benchmarks and qualitatively appear clinically reasonable. The method assumes that aggregate decision making converges appropriately, but ongoing evaluation is necessary to discern common behaviors from "correct" ones. CONCLUSIONS: Collaborative filtering recommender algorithms generate clinical decision support that is predictive of real practice patterns and clinical outcomes. Incorporating temporal relationships improves accuracy. Different evaluation metrics satisfy different goals (predicting likely events vs. "interesting" suggestions). Published by Oxford University Press on behalf of the American Medical Informatics Association 2015. This work is written by US Government employees and is in the public domain in the US.
OBJECTIVE: To answer a "grand challenge" in clinical decision support, the authors produced a recommender system that automatically data-mines inpatient decision support from electronic medical records (EMR), analogous to Netflix or Amazon.com's product recommender. MATERIALS AND METHODS: EMR data were extracted from 1 year of hospitalizations (>18K patients with >5.4M structured items including clinical orders, lab results, and diagnosis codes). Association statistics were counted for the ∼1.5K most common items to drive an order recommender. The authors assessed the recommender's ability to predict hospital admission orders and outcomes based on initial encounter data from separate validation patients. RESULTS: Compared to a reference benchmark of using the overall most common orders, the recommender using temporal relationships improves precision at 10 recommendations from 33% to 38% (P < 10(-10)) for hospital admission orders. Relative risk-based association methods improve inverse frequency weighted recall from 4% to 16% (P < 10(-16)). The framework yields a prediction receiver operating characteristic area under curve (c-statistic) of 0.84 for 30 day mortality, 0.84 for 1 week need for ICU life support, 0.80 for 1 week hospital discharge, and 0.68 for 30-day readmission. DISCUSSION: Recommender results quantitatively improve on reference benchmarks and qualitatively appear clinically reasonable. The method assumes that aggregate decision making converges appropriately, but ongoing evaluation is necessary to discern common behaviors from "correct" ones. CONCLUSIONS: Collaborative filtering recommender algorithms generate clinical decision support that is predictive of real practice patterns and clinical outcomes. Incorporating temporal relationships improves accuracy. Different evaluation metrics satisfy different goals (predicting likely events vs. "interesting" suggestions). Published by Oxford University Press on behalf of the American Medical Informatics Association 2015. This work is written by US Government employees and is in the public domain in the US.
Entities:
Keywords:
clinical decision support systems; collaborative filtering; data-mining; electronic health records; order sets; practice guidelines; practice variability; recommender algorithms
Authors: Nazzareno Galiè; Marius M Hoeper; Marc Humbert; Adam Torbicki; Jean-Luc Vachiery; Joan Albert Barbera; Maurice Beghetti; Paul Corris; Sean Gaine; J Simon Gibbs; Miguel Angel Gomez-Sanchez; Guillaume Jondeau; Walter Klepetko; Christian Opitz; Andrew Peacock; Lewis Rubin; Michael Zellweger; Gerald Simonneau Journal: Eur Heart J Date: 2009-08-27 Impact factor: 29.983
Authors: Jonathan H Chen; Muthuraman Alagappan; Mary K Goldstein; Steven M Asch; Russ B Altman Journal: Int J Med Inform Date: 2017-03-18 Impact factor: 4.046
Authors: Jason K Wang; Alejandro Schuler; Nigam H Shah; Michael T M Baiocchi; Jonathan H Chen Journal: AMIA Jt Summits Transl Sci Proc Date: 2018-05-18