Jean Feng1, Alexej Gossmann2, Berkman Sahiner2, Romain Pirracchio3. 1. Department of Epidemiology and Biostatistics, University of California, San Francisco, San Francisco, California, USA. 2. CDRH-Center for Devices and Radiological Health, Food and Drug Administration, Silver Spring, Maryland, USA. 3. Department of Anesthesia and Perioperative Care, University of California, San Francisco, San Francisco, California, USA.
Abstract
OBJECTIVE: After deploying a clinical prediction model, subsequently collected data can be used to fine-tune its predictions and adapt to temporal shifts. Because model updating carries risks of over-updating/fitting, we study online methods with performance guarantees. MATERIALS AND METHODS: We introduce 2 procedures for continual recalibration or revision of an underlying prediction model: Bayesian logistic regression (BLR) and a Markov variant that explicitly models distribution shifts (MarBLR). We perform empirical evaluation via simulations and a real-world study predicting Chronic Obstructive Pulmonary Disease (COPD) risk. We derive "Type I and II" regret bounds, which guarantee the procedures are noninferior to a static model and competitive with an oracle logistic reviser in terms of the average loss. RESULTS: Both procedures consistently outperformed the static model and other online logistic revision methods. In simulations, the average estimated calibration index (aECI) of the original model was 0.828 (95%CI, 0.818-0.938). Online recalibration using BLR and MarBLR improved the aECI towards the ideal value of zero, attaining 0.265 (95%CI, 0.230-0.300) and 0.241 (95%CI, 0.216-0.266), respectively. When performing more extensive logistic model revisions, BLR and MarBLR increased the average area under the receiver-operating characteristic curve (aAUC) from 0.767 (95%CI, 0.765-0.769) to 0.800 (95%CI, 0.798-0.802) and 0.799 (95%CI, 0.797-0.801), respectively, in stationary settings and protected against substantial model decay. In the COPD study, BLR and MarBLR dynamically combined the original model with a continually refitted gradient boosted tree to achieve aAUCs of 0.924 (95%CI, 0.913-0.935) and 0.925 (95%CI, 0.914-0.935), compared to the static model's aAUC of 0.904 (95%CI, 0.892-0.916). DISCUSSION: Despite its simplicity, BLR is highly competitive with MarBLR. MarBLR outperforms BLR when its prior better reflects the data. CONCLUSIONS: BLR and MarBLR can improve the transportability of clinical prediction models and maintain their performance over time. Published by Oxford University Press on behalf of the American Medical Informatics Association 2022. This work is written by US Government employees and is in the public domain in the US.
OBJECTIVE: After deploying a clinical prediction model, subsequently collected data can be used to fine-tune its predictions and adapt to temporal shifts. Because model updating carries risks of over-updating/fitting, we study online methods with performance guarantees. MATERIALS AND METHODS: We introduce 2 procedures for continual recalibration or revision of an underlying prediction model: Bayesian logistic regression (BLR) and a Markov variant that explicitly models distribution shifts (MarBLR). We perform empirical evaluation via simulations and a real-world study predicting Chronic Obstructive Pulmonary Disease (COPD) risk. We derive "Type I and II" regret bounds, which guarantee the procedures are noninferior to a static model and competitive with an oracle logistic reviser in terms of the average loss. RESULTS: Both procedures consistently outperformed the static model and other online logistic revision methods. In simulations, the average estimated calibration index (aECI) of the original model was 0.828 (95%CI, 0.818-0.938). Online recalibration using BLR and MarBLR improved the aECI towards the ideal value of zero, attaining 0.265 (95%CI, 0.230-0.300) and 0.241 (95%CI, 0.216-0.266), respectively. When performing more extensive logistic model revisions, BLR and MarBLR increased the average area under the receiver-operating characteristic curve (aAUC) from 0.767 (95%CI, 0.765-0.769) to 0.800 (95%CI, 0.798-0.802) and 0.799 (95%CI, 0.797-0.801), respectively, in stationary settings and protected against substantial model decay. In the COPD study, BLR and MarBLR dynamically combined the original model with a continually refitted gradient boosted tree to achieve aAUCs of 0.924 (95%CI, 0.913-0.935) and 0.925 (95%CI, 0.914-0.935), compared to the static model's aAUC of 0.904 (95%CI, 0.892-0.916). DISCUSSION: Despite its simplicity, BLR is highly competitive with MarBLR. MarBLR outperforms BLR when its prior better reflects the data. CONCLUSIONS: BLR and MarBLR can improve the transportability of clinical prediction models and maintain their performance over time. Published by Oxford University Press on behalf of the American Medical Informatics Association 2022. This work is written by US Government employees and is in the public domain in the US.
Entities:
Keywords:
Bayesian model updating; clinical prediction models; machine learning; model recalibration
Authors: Sharon E Davis; Robert A Greevy; Christopher Fonnesbeck; Thomas A Lasko; Colin G Walsh; Michael E Matheny Journal: J Am Med Inform Assoc Date: 2019-12-01 Impact factor: 4.497
Authors: Yvonne Vergouwe; Daan Nieboer; Rianne Oostenbrink; Thomas P A Debray; Gordon D Murray; Michael W Kattan; Hendrik Koffijberg; Karel G M Moons; Ewout W Steyerberg Journal: Stat Med Date: 2016-11-28 Impact factor: 2.373
Authors: Sharon E Davis; Robert A Greevy; Thomas A Lasko; Colin G Walsh; Michael E Matheny Journal: J Biomed Inform Date: 2020-11-04 Impact factor: 6.317
Authors: Andreas N Strobl; Andrew J Vickers; Ben Van Calster; Ewout Steyerberg; Robin J Leach; Ian M Thompson; Donna P Ankerst Journal: J Biomed Inform Date: 2015-05-16 Impact factor: 6.317
Authors: Sharon E Davis; Thomas A Lasko; Guanhua Chen; Edward D Siew; Michael E Matheny Journal: J Am Med Inform Assoc Date: 2017-11-01 Impact factor: 4.497
Authors: Jean Feng; Rachael V Phillips; Ivana Malenica; Andrew Bishara; Alan E Hubbard; Leo A Celi; Romain Pirracchio Journal: NPJ Digit Med Date: 2022-05-31