Teresa B Gibson1, Michael D Nguyen2, Timothy Burrell1, Frank Yoon1, Jenna Wong3, Sai Dharmarajan4, Rita Ouellet-Hellstrom5, Wei Hua2, Yong Ma6, Elande Baro7, Sarah Bloemers1, Cory Pack1, Adee Kennedy3, Sengwee Toh3, Robert Ball8. 1. Government Health and Human Services, IBM Watson Health, Bethesda, Maryland, USA. 2. Food and Drug Administration, Silver Spring, Maryland, USA. 3. Harvard Medical School and Harvard Pilgrim Health Care Institute, Harvard Medical School, Boston, Massachusetts, USA. 4. Office of Biostatistics, Center for Drug Evaluation and Research, Food and Drug Administration, Silver Spring, Maryland, USA. 5. Division of Epidemiology II, Office of Pharmacovigilance and Epidemiology, Office of Surveillance and Epidemiology, Center for Drug Evaluation and Research, Food and Drug Administration, Silver Spring, Maryland, USA. 6. Center for Drug Evaluation and Research, Food and Drug Administration, Silver Spring, Maryland, USA. 7. Department of Population Medicine, Harvard Medical School and Harvard Pilgrim Health Care Institute, Boston, Massachusetts, USA. 8. Office of Surveillance and Epidemiology Center for Drug Evaluation and Research U.S. Food and Drug Administration, Silver Spring, Maryland, USA.
Abstract
OBJECTIVE: Claims-based algorithms are used in the Food and Drug Administration Sentinel Active Risk Identification and Analysis System to identify occurrences of health outcomes of interest (HOIs) for medical product safety assessment. This project aimed to apply machine learning classification techniques to demonstrate the feasibility of developing a claims-based algorithm to predict an HOI in structured electronic health record (EHR) data. MATERIALS AND METHODS: We used the 2015-2019 IBM MarketScan Explorys Claims-EMR Data Set, linking administrative claims and EHR data at the patient level. We focused on a single HOI, rhabdomyolysis, defined by EHR laboratory test results. Using claims-based predictors, we applied machine learning techniques to predict the HOI: logistic regression, LASSO (least absolute shrinkage and selection operator), random forests, support vector machines, artificial neural nets, and an ensemble method (Super Learner). RESULTS: The study cohort included 32 956 patients and 39 499 encounters. Model performance (positive predictive value [PPV], sensitivity, specificity, area under the receiver-operating characteristic curve) varied considerably across techniques. The area under the receiver-operating characteristic curve exceeded 0.80 in most model variations. DISCUSSION: For the main Food and Drug Administration use case of assessing risk of rhabdomyolysis after drug use, a model with a high PPV is typically preferred. The Super Learner ensemble model without adjustment for class imbalance achieved a PPV of 75.6%, substantially better than a previously used human expert-developed model (PPV = 44.0%). CONCLUSIONS: It is feasible to use machine learning methods to predict an EHR-derived HOI with claims-based predictors. Modeling strategies can be adapted for intended uses, including surveillance, identification of cases for chart review, and outcomes research.
OBJECTIVE: Claims-based algorithms are used in the Food and Drug Administration Sentinel Active Risk Identification and Analysis System to identify occurrences of health outcomes of interest (HOIs) for medical product safety assessment. This project aimed to apply machine learning classification techniques to demonstrate the feasibility of developing a claims-based algorithm to predict an HOI in structured electronic health record (EHR) data. MATERIALS AND METHODS: We used the 2015-2019 IBM MarketScan Explorys Claims-EMR Data Set, linking administrative claims and EHR data at the patient level. We focused on a single HOI, rhabdomyolysis, defined by EHR laboratory test results. Using claims-based predictors, we applied machine learning techniques to predict the HOI: logistic regression, LASSO (least absolute shrinkage and selection operator), random forests, support vector machines, artificial neural nets, and an ensemble method (Super Learner). RESULTS: The study cohort included 32 956 patients and 39 499 encounters. Model performance (positive predictive value [PPV], sensitivity, specificity, area under the receiver-operating characteristic curve) varied considerably across techniques. The area under the receiver-operating characteristic curve exceeded 0.80 in most model variations. DISCUSSION: For the main Food and Drug Administration use case of assessing risk of rhabdomyolysis after drug use, a model with a high PPV is typically preferred. The Super Learner ensemble model without adjustment for class imbalance achieved a PPV of 75.6%, substantially better than a previously used human expert-developed model (PPV = 44.0%). CONCLUSIONS: It is feasible to use machine learning methods to predict an EHR-derived HOI with claims-based predictors. Modeling strategies can be adapted for intended uses, including surveillance, identification of cases for chart review, and outcomes research.
Authors: Richard Platt; Jeffrey S Brown; Melissa Robb; Mark McClellan; Robert Ball; Michael D Nguyen; Rachel E Sherman Journal: N Engl J Med Date: 2018-11-29 Impact factor: 91.245
Authors: Anna Ostropolets; Christian Reich; Patrick Ryan; Ning Shang; George Hripcsak; Chunhua Weng Journal: J Biomed Inform Date: 2019-12-19 Impact factor: 6.317
Authors: Rishi J Desai; Michael E Matheny; Kevin Johnson; Keith Marsolo; Lesley H Curtis; Jennifer C Nelson; Patrick J Heagerty; Judith Maro; Jeffery Brown; Sengwee Toh; Michael Nguyen; Robert Ball; Gerald Dal Pan; Shirley V Wang; Joshua J Gagne; Sebastian Schneeweiss Journal: NPJ Digit Med Date: 2021-12-20