Use of reclassification for assessment of improved prediction: an empirical evaluation.

BACKGROUND: An increasing number of studies evaluate the ability of predictors to change risk stratification and alter medical decisions, i.e. reclassification performance. We examined the reported design and analysis of recent studies of reclassification and the robustness of their claims for improved reclassification.
METHODS: Two independent investigators searched PubMed and citations to the article that introduced the currently most popular reclassification metric (net reclassification index, NRI) to identify studies performing reclassification analysis (January 2006-January 2010). We focused on articles that included any analyses comparing the performance of a baseline predictive model vs the baseline model plus some additional predictor for a prospectively assessed outcome. We recorded information on the baseline model used, outcomes assessed, choice of risk thresholds and features of reclassification analyses.
RESULTS: Of 58 baseline models used in 51 eligible papers, only 14 (24%) were previously described, used as described and had same outcomes as originally intended. Calibration was examined in 53% of the studies. Sixteen studies (31%) provided a reference for the choice of risk thresholds and only six used the previously proposed categories or justified the use of alternative thresholds. Only 14 studies (27%) stated that the chosen risk thresholds had different therapeutic intervention implications. NRI was calculated in 38 studies and was smaller in studies with adequately referenced or justified risk thresholds vs others (P < 0.0001).
CONCLUSIONS: Reclassification studies would benefit from more rigorous methodological standards; otherwise claims for improved reclassification may remain spurious.