BACKGROUND: New markers may improve prediction of diagnostic and prognostic outcomes. We review various measures to quantify the incremental value of markers over standard, readily available characteristics. METHODS: Widely used traditional measures include the improvement in model fit or in the area under the receiver operating characteristic (ROC) curve (AUC). New measures include the net reclassification index (NRI) and decision-analytic measures, such as the fraction of true-positive classifications penalized for false-positive classifications [net benefit (NB)]. For illustration, we discuss a case study on the presence of residual tumour vs. benign tissue in 544 patients with testicular cancer. We assessed three tumour markers [Alpha-fetoprotein (AFP), Human chorionic gonadotropin (HCG) and Lactate dehydrogenase (LDH)] for their incremental value over currently standard clinical predictors. RESULTS: AUC and R(2) values suggested adding continuous LDH and AFP whereas NB only favoured HCG as a potentially promising marker at a clinically defendable decision threshold of 20% risk. The NRI suggested reclassification potential of all three markers. CONCLUSIONS: The improvement in standard discrimination measures, which focus on finding variables that might be promising across all decision thresholds, may not detect the most informative markers at a specific threshold of particular clinical relevance. When a marker is intended to support decision-making, calculation of the improvement in a decision-analytic measure, such as NB, is preferable over an overall judgment as obtained from the AUC in ROC analysis.
BACKGROUND: New markers may improve prediction of diagnostic and prognostic outcomes. We review various measures to quantify the incremental value of markers over standard, readily available characteristics. METHODS: Widely used traditional measures include the improvement in model fit or in the area under the receiver operating characteristic (ROC) curve (AUC). New measures include the net reclassification index (NRI) and decision-analytic measures, such as the fraction of true-positive classifications penalized for false-positive classifications [net benefit (NB)]. For illustration, we discuss a case study on the presence of residual tumour vs. benign tissue in 544 patients with testicular cancer. We assessed three tumour markers [Alpha-fetoprotein (AFP), Human chorionic gonadotropin (HCG) and Lactate dehydrogenase (LDH)] for their incremental value over currently standard clinical predictors. RESULTS: AUC and R(2) values suggested adding continuous LDH and AFP whereas NB only favoured HCG as a potentially promising marker at a clinically defendable decision threshold of 20% risk. The NRI suggested reclassification potential of all three markers. CONCLUSIONS: The improvement in standard discrimination measures, which focus on finding variables that might be promising across all decision thresholds, may not detect the most informative markers at a specific threshold of particular clinical relevance. When a marker is intended to support decision-making, calculation of the improvement in a decision-analytic measure, such as NB, is preferable over an overall judgment as obtained from the AUC in ROC analysis.
Authors: Margaret S Pepe; Ziding Feng; Ying Huang; Gary Longton; Ross Prentice; Ian M Thompson; Yingye Zheng Journal: Am J Epidemiol Date: 2007-11-02 Impact factor: 4.897
Authors: Ewout W Steyerberg; Andrew J Vickers; Nancy R Cook; Thomas Gerds; Mithat Gonen; Nancy Obuchowski; Michael J Pencina; Michael W Kattan Journal: Epidemiology Date: 2010-01 Impact factor: 4.822
Authors: E W Steyerberg; H J Keizer; S D Fosså; D T Sleijfer; G C Toner; H Schraffordt Koops; P F Mulders; J E Messemer; K Ney; J P Donohue Journal: J Clin Oncol Date: 1995-05 Impact factor: 44.544
Authors: Jose López-Torrecilla; Anna Boladeras; María Angeles Cabeza; Almudena Zapatero; Josep Jove; Luis M Esteban; Ivan Henriquez; Manuel Casaña; Carmen González-San Segundo; Antonio Gómez-Caamaño; Jose Luis Mengual; Asunción Hervás; Julia Luisa Muñoz; Gerardo Sanz Journal: Strahlenther Onkol Date: 2015-07-09 Impact factor: 3.621
Authors: John F Heitner; Raymond J Kim; Han W Kim; Igor Klem; Dipan J Shah; Dany Debs; Afshin Farzaneh-Far; Venkateshwar Polsani; Jiwon Kim; Jonathan Weinsaft; Chetan Shenoy; Andrew Hughes; Preston Cargile; Jean Ho; Robert O Bonow; Elizabeth Jenista; Michele Parker; Robert M Judd Journal: JAMA Cardiol Date: 2019-03-01 Impact factor: 14.676
Authors: Ben Van Calster; Andrew J Vickers; Michael J Pencina; Stuart G Baker; Dirk Timmerman; Ewout W Steyerberg Journal: Med Decis Making Date: 2013-01-11 Impact factor: 2.583