David van Klaveren1, Ewout W Steyerberg2, Patrick W Serruys3, David M Kent4. 1. Predictive Analytics and Comparative Effectiveness Center, Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, 800 Washington St, Boston, MA 02111, USA; Department of Medical Statistics, Leiden University Medical Center, Albinusdreef 2, Leiden 2333 ZA, The Netherlands. Electronic address: d.van_klaveren@lumc.nl. 2. Department of Medical Statistics, Leiden University Medical Center, Albinusdreef 2, Leiden 2333 ZA, The Netherlands; Department of Public Health, Erasmus University Medical Center, 's-Gravendijkwal 230, Rotterdam 3015 CE, The Netherlands. 3. National Heart and Lung Institute, Imperial College London, Dovehouse Street, London SW3 6LR, United Kingdom. 4. Predictive Analytics and Comparative Effectiveness Center, Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, 800 Washington St, Boston, MA 02111, USA.
Abstract
OBJECTIVES: Clinical prediction models that support treatment decisions are usually evaluated for their ability to predict the risk of an outcome rather than treatment benefit-the difference between outcome risk with vs. without therapy. We aimed to define performance metrics for a model's ability to predict treatment benefit. STUDY DESIGN AND SETTING: We analyzed data of the Synergy between Percutaneous Coronary Intervention with Taxus and Cardiac Surgery (SYNTAX) trial and of three recombinant tissue plasminogen activator trials. We assessed alternative prediction models with a conventional risk concordance-statistic (c-statistic) and a novel c-statistic for benefit. We defined observed treatment benefit by the outcomes in pairs of patients matched on predicted benefit but discordant for treatment assignment. The 'c-for-benefit' represents the probability that from two randomly chosen matched patient pairs with unequal observed benefit, the pair with greater observed benefit also has a higher predicted benefit. RESULTS: Compared to a model without treatment interactions, the SYNTAX score II had improved ability to discriminate treatment benefit (c-for-benefit 0.590 vs. 0.552), despite having similar risk discrimination (c-statistic 0.725 vs. 0.719). However, for the simplified stroke-thrombolytic predictive instrument (TPI) vs. the original stroke-TPI, the c-for-benefit (0.584 vs. 0.578) was similar. CONCLUSION: The proposed methodology has the potential to measure a model's ability to predict treatment benefit not captured with conventional performance metrics.
OBJECTIVES: Clinical prediction models that support treatment decisions are usually evaluated for their ability to predict the risk of an outcome rather than treatment benefit-the difference between outcome risk with vs. without therapy. We aimed to define performance metrics for a model's ability to predict treatment benefit. STUDY DESIGN AND SETTING: We analyzed data of the Synergy between Percutaneous Coronary Intervention with Taxus and Cardiac Surgery (SYNTAX) trial and of three recombinant tissue plasminogen activator trials. We assessed alternative prediction models with a conventional risk concordance-statistic (c-statistic) and a novel c-statistic for benefit. We defined observed treatment benefit by the outcomes in pairs of patients matched on predicted benefit but discordant for treatment assignment. The 'c-for-benefit' represents the probability that from two randomly chosen matched patient pairs with unequal observed benefit, the pair with greater observed benefit also has a higher predicted benefit. RESULTS: Compared to a model without treatment interactions, the SYNTAX score II had improved ability to discriminate treatment benefit (c-for-benefit 0.590 vs. 0.552), despite having similar risk discrimination (c-statistic 0.725 vs. 0.719). However, for the simplified stroke-thrombolytic predictive instrument (TPI) vs. the original stroke-TPI, the c-for-benefit (0.584 vs. 0.578) was similar. CONCLUSION: The proposed methodology has the potential to measure a model's ability to predict treatment benefit not captured with conventional performance metrics.
Authors: Andreas D Meid; Carmen Ruff; Lucas Wirbka; Felicitas Stoll; Hanna M Seidling; Andreas Groll; Walter E Haefeli Journal: Clin Epidemiol Date: 2020-11-02 Impact factor: 4.790
Authors: David M Kent; David van Klaveren; Jessica K Paulus; Ralph D'Agostino; Steve Goodman; Rodney Hayward; John P A Ioannidis; Bray Patrick-Lake; Sally Morton; Michael Pencina; Gowri Raman; Joseph S Ross; Harry P Selker; Ravi Varadhan; Andrew Vickers; John B Wong; Ewout W Steyerberg Journal: Ann Intern Med Date: 2019-11-12 Impact factor: 25.391
Authors: Adam P Bress; Tom Greene; Catherine G Derington; Jincheng Shen; Yizhe Xu; Yiyi Zhang; Jian Ying; Brandon K Bellows; William C Cushman; Paul K Whelton; Nicholas M Pajewski; David Reboussin; Srinivasan Beddu; Rachel Hess; Jennifer S Herrick; Zugui Zhang; Paul Kolm; Robert W Yeh; Sanjay Basu; William S Weintraub; Andrew E Moran Journal: J Am Coll Cardiol Date: 2021-04-27 Impact factor: 24.094
Authors: E Hope Weissler; Tristan Naumann; Tomas Andersson; Rajesh Ranganath; Olivier Elemento; Yuan Luo; Daniel F Freitag; James Benoit; Michael C Hughes; Faisal Khan; Paul Slater; Khader Shameer; Matthew Roe; Emmette Hutchison; Scott H Kollins; Uli Broedl; Zhaoling Meng; Jennifer L Wong; Lesley Curtis; Erich Huang; Marzyeh Ghassemi Journal: Trials Date: 2021-08-16 Impact factor: 2.279
Authors: Alexandros Rekkas; Jessica K Paulus; Gowri Raman; John B Wong; Ewout W Steyerberg; Peter R Rijnbeek; David M Kent; David van Klaveren Journal: BMC Med Res Methodol Date: 2020-10-23 Impact factor: 4.615
Authors: Judith Heinz; Christian Röver; Ghefar Furaijat; Yvonne Kaußner; Eva Hummers; Thomas Debray; Alastair D Hay; Stefan Heytens; Ingvild Vik; Paul Little; Michael Moore; Beth Stuart; Florian Wagenlehner; Andreas Kronenberg; Sven Ferry; Tor Monsen; Morten Lindbaek; Tim Friede; Ildiko Gagyor Journal: BMJ Open Date: 2020-10-01 Impact factor: 2.692