Qingyang Xu1,2, Elaheh Ahmadi3,4, Alexander Amini3,4, Daniela Rus3,4, Andrew W Lo5,6,7,8,9. 1. MIT Laboratory for Financial Engineering, Cambridge, MA, USA. 2. MIT Operations Research Center, Cambridge, MA, USA. 3. MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA, USA. 4. MIT Department of Electrical Engineering and Computer Science, Cambridge, MA, USA. 5. MIT Laboratory for Financial Engineering, Cambridge, MA, USA. alo-admin@mit.edu. 6. MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA, USA. alo-admin@mit.edu. 7. MIT Department of Electrical Engineering and Computer Science, Cambridge, MA, USA. alo-admin@mit.edu. 8. MIT Operations Research Center, Cambridge, MA, USA. alo-admin@mit.edu. 9. Sante Fe Institute, Santa Fe, NM, USA. alo-admin@mit.edu.
Abstract
INTRODUCTION: Machine learning models are increasingly applied to predict the drug development outcomes based on intermediary clinical trial results. A key challenge to this task is to address various forms of bias in the historical drug approval data. OBJECTIVE: We aimed to identify and mitigate the bias in drug approval predictions and quantify the impacts of debiasing in terms of financial value and drug safety. METHODS: We instantiated the Debiasing Variational Autoencoder, the state-of-the-art model for automated debiasing. We trained and evaluated the model on the Citeline dataset provided by Informa Pharma Intelligence to predict the final drug development outcome from phase II trial results. RESULTS: The debiased Debiasing Variational Autoencoder model achieved better performance (measured by the [Formula: see text] score 0.48) in predicting the drug development outcomes than its un-debiased baseline ([Formula: see text] score 0.25). It had a much higher true-positive rate than baseline (60% vs 15%), while its true-negative rate was slightly lower (88% vs 99%). The Debiasing Variational Autoencoder distinguished between drugs developed by large pharmaceutical firms and those by small biotech companies. The model prediction is strongly influenced by multiple factors such as prior approval of the drug for another indication, whether the trial meets the positive/negative endpoints, and the year when the trial is completed. We estimate that the debiased model generates financial value for the drug developer in six major therapeutic areas, with a range of US$763-1,365 million. CONCLUSIONS: Our analysis shows that debiasing improves the financial efficiency of late-stage drug development. From the pharmacovigilance perspective, the debiased model is more likely to identify drugs that are both safe and effective. Meanwhile, it may predict a higher probability of success for drugs with potential adverse effects (because of its lower true-negative rate), thus it must be used with caution to predict the development outcomes of drug candidates currently in the pipeline.
INTRODUCTION: Machine learning models are increasingly applied to predict the drug development outcomes based on intermediary clinical trial results. A key challenge to this task is to address various forms of bias in the historical drug approval data. OBJECTIVE: We aimed to identify and mitigate the bias in drug approval predictions and quantify the impacts of debiasing in terms of financial value and drug safety. METHODS: We instantiated the Debiasing Variational Autoencoder, the state-of-the-art model for automated debiasing. We trained and evaluated the model on the Citeline dataset provided by Informa Pharma Intelligence to predict the final drug development outcome from phase II trial results. RESULTS: The debiased Debiasing Variational Autoencoder model achieved better performance (measured by the [Formula: see text] score 0.48) in predicting the drug development outcomes than its un-debiased baseline ([Formula: see text] score 0.25). It had a much higher true-positive rate than baseline (60% vs 15%), while its true-negative rate was slightly lower (88% vs 99%). The Debiasing Variational Autoencoder distinguished between drugs developed by large pharmaceutical firms and those by small biotech companies. The model prediction is strongly influenced by multiple factors such as prior approval of the drug for another indication, whether the trial meets the positive/negative endpoints, and the year when the trial is completed. We estimate that the debiased model generates financial value for the drug developer in six major therapeutic areas, with a range of US$763-1,365 million. CONCLUSIONS: Our analysis shows that debiasing improves the financial efficiency of late-stage drug development. From the pharmacovigilance perspective, the debiased model is more likely to identify drugs that are both safe and effective. Meanwhile, it may predict a higher probability of success for drugs with potential adverse effects (because of its lower true-negative rate), thus it must be used with caution to predict the development outcomes of drug candidates currently in the pipeline.
Authors: J A DiMasi; J C Hermann; K Twyman; R K Kondru; S Stergiopoulos; K A Getz; W Rackoff Journal: Clin Pharmacol Ther Date: 2015-09-24 Impact factor: 6.875
Authors: Laeeq Malik; Alex Mejia; Helen Parsons; Benjamin Ehler; Devalingam Mahalingam; Andrew Brenner; John Sarantopoulos; Steven Weitman Journal: Cancer Chemother Pharmacol Date: 2014-09-23 Impact factor: 3.333