Gang Fang1, Izabela E Annis1, Jennifer Elston-Lafata1, Samuel Cykert2. 1. Division of Pharmaceutical Outcomes and Policy, UNC Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA. 2. Program for Health and Clinical Informatics, School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, USA.
Abstract
OBJECTIVE: We aimed to investigate bias in applying machine learning to predict real-world individual treatment effects. MATERIALS AND METHODS: Using a virtual patient cohort, we simulated real-world healthcare data and applied random forest and gradient boosting classifiers to develop prediction models. Treatment effect was estimated as the difference between the predicted outcomes of a treatment and a control. We evaluated the impact of predictors (ie, treatment predictors [X1], confounders [X2], treatment effects modifiers [X3], and other outcome risk factors [X4]) with known effects on treatment and outcome using real-world data, and outcome imbalance on predicting individual outcome. Using counterfactuals, we evaluated percentage of patients with biased predicted individual treatment effects. RESULTS: The X4 had relatively more impact on model performance than X2 and X3 did. No effects were observed from X1. Moderate-to-severe outcome imbalance had a significantly negative impact on model performance, particularly among subgroups in which an outcome occurred. Bias in predicting individual treatment effects was significant and persisted even when the models had a 100% accuracy in predicting health outcome. DISCUSSION: Inadequate inclusion of the X2, X3, and X4 and moderate-to-severe outcome imbalance may affect model performance in predicting individual outcome and subsequently bias in predicting individual treatment effects. Machine learning models with all features and high performance for predicting individual outcome still yielded biased individual treatment effects. CONCLUSIONS: Direct application of machine learning might not adequately address bias in predicting individual treatment effects. Further method development is needed to advance machine learning to support individualized treatment selection.
OBJECTIVE: We aimed to investigate bias in applying machine learning to predict real-world individual treatment effects. MATERIALS AND METHODS: Using a virtual patient cohort, we simulated real-world healthcare data and applied random forest and gradient boosting classifiers to develop prediction models. Treatment effect was estimated as the difference between the predicted outcomes of a treatment and a control. We evaluated the impact of predictors (ie, treatment predictors [X1], confounders [X2], treatment effects modifiers [X3], and other outcome risk factors [X4]) with known effects on treatment and outcome using real-world data, and outcome imbalance on predicting individual outcome. Using counterfactuals, we evaluated percentage of patients with biased predicted individual treatment effects. RESULTS: The X4 had relatively more impact on model performance than X2 and X3 did. No effects were observed from X1. Moderate-to-severe outcome imbalance had a significantly negative impact on model performance, particularly among subgroups in which an outcome occurred. Bias in predicting individual treatment effects was significant and persisted even when the models had a 100% accuracy in predicting health outcome. DISCUSSION: Inadequate inclusion of the X2, X3, and X4 and moderate-to-severe outcome imbalance may affect model performance in predicting individual outcome and subsequently bias in predicting individual treatment effects. Machine learning models with all features and high performance for predicting individual outcome still yielded biased individual treatment effects. CONCLUSIONS: Direct application of machine learning might not adequately address bias in predicting individual treatment effects. Further method development is needed to advance machine learning to support individualized treatment selection.
Authors: Shinji Tarumi; Wataru Takeuchi; George Chalkidis; Salvador Rodriguez-Loya; Junichi Kuwata; Michael Flynn; Kyle M Turner; Farrant H Sakaguchi; Charlene Weir; Heidi Kramer; David E Shields; Phillip B Warner; Polina Kukhareva; Hideyuki Ban; Kensaku Kawamoto Journal: Methods Inf Med Date: 2021-05-11 Impact factor: 2.176