Aasthaa Bansal1, Nicole Mayer-Hamblett2, Christopher H Goss3, Lingtak N Chan4, Patrick J Heagerty5. 1. The Comparative Health Outcomes, Policy, and Economic (CHOICE) Institute, School of Pharmacy, University of Washington, Box 357630, 1959 NE Pacific Ave, H-375B, Seattle, WA, USA, 98195. 2. Departments of Pediatrics and Biostatistics, University of Washington, Seattle, WA. 3. Division of Pulmonary and Critical Care Medicine, Departments of Medicine and Pediatrics, University of Washington, Seattle WA, USA. 4. Department of Pharmacy, University of Washington, Seattle WA, USA. 5. Department of Biostatistics, University of Washington, Seattle WA, USA.
Abstract
BACKGROUND: Effective transplantation recommendations in cystic fibrosis (CF) require accurate survival predictions, so that high-risk patients may be prioritized for transplantation. In practice, decisions about transplantation are made dynamically, using routinely updated assessments. We present a novel tool for evaluating risk prediction models that, unlike traditional methods, captures classification accuracy in identifying high-risk patients in a dynamic fashion. METHODS: Predicted risk is used as a score to rank incident deaths versus patients who survive, with the goal of ranking the deaths higher. The mean rank across deaths at a given time measures time-specific predictive accuracy; when assessed over time, it reflects time-varying accuracy. RESULTS: Applying this approach to CF Registry data on patients followed from 1993-2011, we show that traditional methods do not capture the performance of models used dynamically in the clinical setting. Previously proposed multivariate risk scores perform no better than forced expiratory volume in 1 second as a percentage of predicted normal (FEV1%) alone. Despite its value for survival prediction, FEV1% has a low sensitivity of 45% over time (for fixed specificity of 95%), leaving room for improvement in prediction. Finally, prediction accuracy with annually-updated FEV1% shows minor differences compared to FEV1% updated every 2 years, which may have clinical implications regarding the optimal frequency of updating clinical information. CONCLUSIONS: It is imperative to continue to develop models that accurately predict survival in CF. Our proposed approach can serve as the basis for evaluating the predictive ability of these models by better accounting for their dynamic clinical use.
BACKGROUND: Effective transplantation recommendations in cystic fibrosis (CF) require accurate survival predictions, so that high-risk patients may be prioritized for transplantation. In practice, decisions about transplantation are made dynamically, using routinely updated assessments. We present a novel tool for evaluating risk prediction models that, unlike traditional methods, captures classification accuracy in identifying high-risk patients in a dynamic fashion. METHODS: Predicted risk is used as a score to rank incident deaths versus patients who survive, with the goal of ranking the deaths higher. The mean rank across deaths at a given time measures time-specific predictive accuracy; when assessed over time, it reflects time-varying accuracy. RESULTS: Applying this approach to CF Registry data on patients followed from 1993-2011, we show that traditional methods do not capture the performance of models used dynamically in the clinical setting. Previously proposed multivariate risk scores perform no better than forced expiratory volume in 1 second as a percentage of predicted normal (FEV1%) alone. Despite its value for survival prediction, FEV1% has a low sensitivity of 45% over time (for fixed specificity of 95%), leaving room for improvement in prediction. Finally, prediction accuracy with annually-updated FEV1% shows minor differences compared to FEV1% updated every 2 years, which may have clinical implications regarding the optimal frequency of updating clinical information. CONCLUSIONS: It is imperative to continue to develop models that accurately predict survival in CF. Our proposed approach can serve as the basis for evaluating the predictive ability of these models by better accounting for their dynamic clinical use.
Authors: Jonathan B Orens; Marc Estenne; Selim Arcasoy; John V Conte; Paul Corris; Jim J Egan; Thomas Egan; Shaf Keshavjee; Christiane Knoop; Robert Kotloff; Fernando J Martinez; Steven Nathan; Scott Palmer; Alec Patterson; Lianne Singer; Gregory Snell; Sean Studer; J L Vachiery; Allan R Glanville Journal: J Heart Lung Transplant Date: 2006-07 Impact factor: 10.247
Authors: Nicole Mayer-Hamblett; Margaret Rosenfeld; Julia Emerson; Christopher H Goss; Moira L Aitken Journal: Am J Respir Crit Care Med Date: 2002-08-15 Impact factor: 21.405
Authors: Ewout W Steyerberg; Andrew J Vickers; Nancy R Cook; Thomas Gerds; Mithat Gonen; Nancy Obuchowski; Michael J Pencina; Michael W Kattan Journal: Epidemiology Date: 2010-01 Impact factor: 4.822
Authors: B Whitehead; P Helms; M Goodwin; I Martin; J P Scott; R L Smyth; T W Higenbottam; J Wallwork; M Elliott; M de Leval Journal: Arch Dis Child Date: 1991-09 Impact factor: 3.791