Sei J Lee1,2, Alexander K Smith1,2, L Grisell Diaz-Ramirez1,2, Kenneth E Covinsky1,2, Siqi Gan1,2, Catherine L Chen3,4, William J Boscardin1,2. 1. Department of Medicine, Division of Geriatrics, University of California, San Francisco. 2. San Francisco Veterans Affairs Medical Center. 3. Department of Anesthesia & Perioperative Care, University of California. 4. Philip R. Lee Institute for Health Policy Studies at UCSF, San Francisco, CA.
Abstract
BACKGROUND: Guidelines recommend that clinicians use clinical prediction models to estimate future risk to guide decisions. For example, predicted fracture risk is a major factor in the decision to initiate bisphosphonate medications. However, current methods for developing prediction models often lead to models that are accurate but difficult to use in clinical settings. OBJECTIVE: The objective of this study was to develop and test whether a new metric that explicitly balances model accuracy with clinical usability leads to accurate, easier-to-use prediction models. METHODS: We propose a new metric called the Time-cost Information Criterion (TCIC) that will penalize potential predictor variables that take a long time to obtain in clinical settings. To demonstrate how the TCIC can be used to develop models that are easier-to-use in clinical settings, we use data from the 2000 wave of the Health and Retirement Study (n=6311) to develop and compare time to mortality prediction models using a traditional metric (Bayesian Information Criterion or BIC) and the TCIC. RESULTS: We found that the TCIC models utilized predictors that could be obtained more quickly than BIC models while achieving similar discrimination. For example, the TCIC identified a 7-predictor model with a total time-cost of 44 seconds, while the BIC identified a 7-predictor model with a time-cost of 119 seconds. The Harrell C-statistic of the TCIC and BIC 7-predictor models did not differ (0.7065 vs. 0.7088, P=0.11). CONCLUSION: Accounting for the time-costs of potential predictor variables through the use of the TCIC led to the development of an easier-to-use mortality prediction model with similar discrimination.
BACKGROUND: Guidelines recommend that clinicians use clinical prediction models to estimate future risk to guide decisions. For example, predicted fracture risk is a major factor in the decision to initiate bisphosphonate medications. However, current methods for developing prediction models often lead to models that are accurate but difficult to use in clinical settings. OBJECTIVE: The objective of this study was to develop and test whether a new metric that explicitly balances model accuracy with clinical usability leads to accurate, easier-to-use prediction models. METHODS: We propose a new metric called the Time-cost Information Criterion (TCIC) that will penalize potential predictor variables that take a long time to obtain in clinical settings. To demonstrate how the TCIC can be used to develop models that are easier-to-use in clinical settings, we use data from the 2000 wave of the Health and Retirement Study (n=6311) to develop and compare time to mortality prediction models using a traditional metric (Bayesian Information Criterion or BIC) and the TCIC. RESULTS: We found that the TCIC models utilized predictors that could be obtained more quickly than BIC models while achieving similar discrimination. For example, the TCIC identified a 7-predictor model with a total time-cost of 44 seconds, while the BIC identified a 7-predictor model with a time-cost of 119 seconds. The Harrell C-statistic of the TCIC and BIC 7-predictor models did not differ (0.7065 vs. 0.7088, P=0.11). CONCLUSION: Accounting for the time-costs of potential predictor variables through the use of the TCIC led to the development of an easier-to-use mortality prediction model with similar discrimination.
Authors: Mark A van de Wiel; Tonje G Lien; Wina Verlaat; Wessel N van Wieringen; Saskia M Wilting Journal: Stat Med Date: 2015-09-13 Impact factor: 2.373
Authors: Pauline M Camacho; Steven M Petak; Neil Binkley; Bart L Clarke; Steven T Harris; Daniel L Hurley; Michael Kleerekoper; E Michael Lewiecki; Paul D Miller; Harmeet S Narula; Rachel Pessah-Pollack; Vin Tangpricha; Sunil J Wimalawansa; Nelson B Watts Journal: Endocr Pract Date: 2016-09 Impact factor: 3.443
Authors: Jennifer S Lin; Margaret A Piper; Leslie A Perdue; Carolyn M Rutter; Elizabeth M Webber; Elizabeth O'Connor; Ning Smith; Evelyn P Whitlock Journal: JAMA Date: 2016-06-21 Impact factor: 56.272
Authors: L Grisell Diaz-Ramirez; Sei J Lee; Alexander K Smith; Siqi Gan; W John Boscardin Journal: Comput Methods Programs Biomed Date: 2021-03-27 Impact factor: 5.428