Peter C Austin1,2,3, Douglas S Lee1,2,4,5, Dennis T Ko1,2,3,4, Ian R White6. 1. ICES, Toronto, ON, Canada (P.C.A., D.S.L., D.T.K.). 2. Institute of Health Management, Policy and Evaluation, University of Toronto, ON, Canada (P.C.A., D.S.L., D.T.K.). 3. Sunnybrook Research Institute, Toronto, ON, Canada (P.C.A., D.T.K.). 4. Department of Medicine, University of Toronto, ON, Canada (D.S.L., D.T.K.). 5. Peter Munk Cardiac Centre, University Health Network, Toronto, ON, Canada (D.S.L.). 6. Medical Research Council Clinical Trials Unit, University College London, United Kingdom (I.R.W.).
Abstract
BACKGROUND: Variable selection is an important issue when developing prognostic models. Missing data occur frequently in clinical research. Multiple imputation is increasingly used to address the presence of missing data in clinical research. The effect of different variable selection strategies with multiply imputed data on the external performance of derived prognostic models has not been well examined. METHODS AND RESULTS: We used backward variable selection with 9 different ways to handle multiply imputed data in a derivation sample to develop logistic regression models for predicting death within 1 year of hospitalization with an acute myocardial infarction. We assessed the prognostic accuracy of each derived model in a temporally distinct validation sample. The derivation and validation samples consisted of 11 524 patients hospitalized between 1999 and 2001 and 7889 patients hospitalized between 2004 and 2005, respectively. We considered 41 candidate predictor variables. Missing data occurred frequently, with only 13% of patients in the derivation sample and 31% of patients in the validation sample having complete data. Regardless of the significance level for variable selection, the prognostic model developed using only the complete cases in the derivation sample had substantially worse performance in the validation sample than did the models for which variables were selected using the multiply imputed versions of the derivation sample. The other 8 approaches to handling multiply imputed data resulted in prognostic models with performance similar to one another. CONCLUSIONS: Ignoring missing data and using only subjects with complete data can result in the derivation of prognostic models with poor performance. Multiple imputation should be used to account for missing data when developing prognostic models.
BACKGROUND: Variable selection is an important issue when developing prognostic models. Missing data occur frequently in clinical research. Multiple imputation is increasingly used to address the presence of missing data in clinical research. The effect of different variable selection strategies with multiply imputed data on the external performance of derived prognostic models has not been well examined. METHODS AND RESULTS: We used backward variable selection with 9 different ways to handle multiply imputed data in a derivation sample to develop logistic regression models for predicting death within 1 year of hospitalization with an acute myocardial infarction. We assessed the prognostic accuracy of each derived model in a temporally distinct validation sample. The derivation and validation samples consisted of 11 524 patients hospitalized between 1999 and 2001 and 7889 patients hospitalized between 2004 and 2005, respectively. We considered 41 candidate predictor variables. Missing data occurred frequently, with only 13% of patients in the derivation sample and 31% of patients in the validation sample having complete data. Regardless of the significance level for variable selection, the prognostic model developed using only the complete cases in the derivation sample had substantially worse performance in the validation sample than did the models for which variables were selected using the multiply imputed versions of the derivation sample. The other 8 approaches to handling multiply imputed data resulted in prognostic models with performance similar to one another. CONCLUSIONS: Ignoring missing data and using only subjects with complete data can result in the derivation of prognostic models with poor performance. Multiple imputation should be used to account for missing data when developing prognostic models.
Entities:
Keywords:
death; hospitalization; incidence; myocardial infarction; probability
Authors: W James Deardorff; Deborah E Barnes; Sun Y Jeon; W John Boscardin; Kenneth M Langa; Kenneth E Covinsky; Susan L Mitchell; Elizabeth L Whitlock; Alexander K Smith; Sei J Lee Journal: JAMA Intern Med Date: 2022-09-26 Impact factor: 44.409
Authors: Ruben J Eck; Lisa Hulshof; Renske Wiersema; Chris H L Thio; Bart Hiemstra; Niels C Gritters van den Oever; Reinold O B Gans; Iwan C C van der Horst; Karina Meijer; Frederik Keus Journal: Crit Care Date: 2021-01-12 Impact factor: 9.097