Ewout W Steyerberg1, Marinus J C Eijkemans, Eric Boersma, J D F Habbema. 1. Department of Public Health, Center for Clinical Decision Sciences, Ee2093, Erasmus MC, University Medical Center Rotterdam, PO Box 1738, 3000 DR Rotterdam, The Netherlands. E.Steyerberg@ErasmusMC.nl
Abstract
OBJECTIVE: Models that predict mortality after acute myocardial infarction (AMI) contain different predictors and are based on different populations. We studied the agreement and validity of predictions for individual patients. STUDY DESIGN AND SETTING: We compared predictions from five predictive logistic regression models for short-term mortality after AMI. Three models were developed previously, and two models were developed in the GUSTO-I data, where all five models were applied (n =40,830, 7.0% 30-day mortality). Agreement was studied with weighted kappa statistics of categorized predictions. Validity was assessed by comparing observed frequencies with predictions (indicating calibration) and by the area under the receiver operating characteristic curve (AUC), indicating discriminative ability. RESULTS: The predictions from the five models varied considerably for individual patients, with low agreement between most (kappa <0.6). Risk predictions from the three previously developed models were on average too high, which could be corrected by re-calibration of the model intercept. The AUC ranged from 0.76-0.78 and increased to 0.78-0.79 with re-estimated regression coefficients that were optimal for the GUSTO-I patients. The two more detailed GUSTO-I based models performed better (AUC approximately 0.82). CONCLUSION: Models with different predictors may have a similar validity while the agreement between predictions for individual patients is poor. The main concerns in the applicability of predictive models for AMI should relate to the selected predictors and average calibration.
OBJECTIVE: Models that predict mortality after acute myocardial infarction (AMI) contain different predictors and are based on different populations. We studied the agreement and validity of predictions for individual patients. STUDY DESIGN AND SETTING: We compared predictions from five predictive logistic regression models for short-term mortality after AMI. Three models were developed previously, and two models were developed in the GUSTO-I data, where all five models were applied (n =40,830, 7.0% 30-day mortality). Agreement was studied with weighted kappa statistics of categorized predictions. Validity was assessed by comparing observed frequencies with predictions (indicating calibration) and by the area under the receiver operating characteristic curve (AUC), indicating discriminative ability. RESULTS: The predictions from the five models varied considerably for individual patients, with low agreement between most (kappa <0.6). Risk predictions from the three previously developed models were on average too high, which could be corrected by re-calibration of the model intercept. The AUC ranged from 0.76-0.78 and increased to 0.78-0.79 with re-estimated regression coefficients that were optimal for the GUSTO-I patients. The two more detailed GUSTO-I based models performed better (AUC approximately 0.82). CONCLUSION: Models with different predictors may have a similar validity while the agreement between predictions for individual patients is poor. The main concerns in the applicability of predictive models for AMI should relate to the selected predictors and average calibration.
Authors: Kevin Ten Haaf; Jihyoun Jeon; Martin C Tammemägi; Summer S Han; Chung Yin Kong; Sylvia K Plevritis; Eric J Feuer; Harry J de Koning; Ewout W Steyerberg; Rafael Meza Journal: PLoS Med Date: 2017-04-04 Impact factor: 11.069