Tony Duan1, Pranav Rajpurkar1, Dillon Laird1, Andrew Y Ng1, Sanjay Basu2. 1. Department of Computer Science (T.D., P.R., D.L., A.Y.N.), Stanford University, Stanford, CA. 2. Center for Primary Care and Outcomes Research and Center for Population Health Sciences, Departments of Medicine and of Health Research and Policy (S.B.), Stanford University, Stanford, CA.
Abstract
BACKGROUND: The absolute risk reduction (ARR) in cardiovascular events from therapy is generally assumed to be proportional to baseline risk-such that high-risk patients benefit most. Yet newer analyses have proposed using randomized trial data to develop models that estimate individual treatment effects. We tested 2 hypotheses: first, that models of individual treatment effects would reveal that benefit from intensive blood pressure therapy is proportional to baseline risk; and second, that a machine learning approach designed to predict heterogeneous treatment effects-the X-learner meta-algorithm-is equivalent to a conventional logistic regression approach. METHODS AND RESULTS: We compared conventional logistic regression to the X-learner approach for prediction of 3-year cardiovascular disease event risk reduction from intensive (target systolic blood pressure <120 mm Hg) versus standard (target <140 mm Hg) blood pressure treatment, using individual participant data from the SPRINT (Systolic Blood Pressure Intervention Trial; N=9361) and ACCORD BP (Action to Control Cardiovascular Risk in Diabetes Blood Pressure; N=4733) trials. Each model incorporated 17 covariates, an indicator for treatment arm, and interaction terms between covariates and treatment. Logistic regression had lower C statistic for benefit than the X-learner (0.51 [95% CI, 0.49-0.53] versus 0.60 [95% CI, 0.58-0.63], respectively). Following the logistic regression's recommendation for individualized therapy produced restricted mean time until cardiovascular disease event of 1065.47 days (95% CI, 1061.04-1069.35), while following the X-learner's recommendation improved mean time until cardiovascular disease event to 1068.71 days (95% CI, 1065.42-1072.08). Calibration was worse for logistic regression; it over-estimated ARR attributable to intensive treatment (slope between predicted and observed ARR of 0.73 [95% CI, 0.30-1.14] versus 1.06 [95% CI, 0.74-1.32] for the X-learner, compared with the ideal of 1). Predicted ARRs using logistic regression were generally proportional to baseline pretreatment cardiovascular risk, whereas the X-learner observed-correctly-that individual treatment effects were often not proportional to baseline risk. CONCLUSIONS: Predictions for individual treatment effects from trial data reveal that patients may experience ARRs not simply proportional to baseline cardiovascular disease risk. Machine learning methods may improve discrimination and calibration of individualized treatment effect estimates from clinical trial data. CLINICAL TRIAL REGISTRATION: URL: https://www.clinicaltrials.gov . Unique identifiers: NCT01206062; NCT00000620.
BACKGROUND: The absolute risk reduction (ARR) in cardiovascular events from therapy is generally assumed to be proportional to baseline risk-such that high-risk patients benefit most. Yet newer analyses have proposed using randomized trial data to develop models that estimate individual treatment effects. We tested 2 hypotheses: first, that models of individual treatment effects would reveal that benefit from intensive blood pressure therapy is proportional to baseline risk; and second, that a machine learning approach designed to predict heterogeneous treatment effects-the X-learner meta-algorithm-is equivalent to a conventional logistic regression approach. METHODS AND RESULTS: We compared conventional logistic regression to the X-learner approach for prediction of 3-year cardiovascular disease event risk reduction from intensive (target systolic blood pressure <120 mm Hg) versus standard (target <140 mm Hg) blood pressure treatment, using individual participant data from the SPRINT (Systolic Blood Pressure Intervention Trial; N=9361) and ACCORD BP (Action to Control Cardiovascular Risk in Diabetes Blood Pressure; N=4733) trials. Each model incorporated 17 covariates, an indicator for treatment arm, and interaction terms between covariates and treatment. Logistic regression had lower C statistic for benefit than the X-learner (0.51 [95% CI, 0.49-0.53] versus 0.60 [95% CI, 0.58-0.63], respectively). Following the logistic regression's recommendation for individualized therapy produced restricted mean time until cardiovascular disease event of 1065.47 days (95% CI, 1061.04-1069.35), while following the X-learner's recommendation improved mean time until cardiovascular disease event to 1068.71 days (95% CI, 1065.42-1072.08). Calibration was worse for logistic regression; it over-estimated ARR attributable to intensive treatment (slope between predicted and observed ARR of 0.73 [95% CI, 0.30-1.14] versus 1.06 [95% CI, 0.74-1.32] for the X-learner, compared with the ideal of 1). Predicted ARRs using logistic regression were generally proportional to baseline pretreatment cardiovascular risk, whereas the X-learner observed-correctly-that individual treatment effects were often not proportional to baseline risk. CONCLUSIONS: Predictions for individual treatment effects from trial data reveal that patients may experience ARRs not simply proportional to baseline cardiovascular disease risk. Machine learning methods may improve discrimination and calibration of individualized treatment effect estimates from clinical trial data. CLINICAL TRIAL REGISTRATION: URL: https://www.clinicaltrials.gov . Unique identifiers: NCT01206062; NCT00000620.
Authors: Joseph Yeboah; Raimund Erbel; Joseph Chris Delaney; Robin Nance; Mengye Guo; Alain G Bertoni; Matthew Budoff; Susanne Moebus; Karl-Heinz Jöckel; Gregory L Burke; Nathan D Wong; Nils Lehmann; David M Herrington; Stefan Möhlenkamp; Philip Greenland Journal: Atherosclerosis Date: 2014-08-14 Impact factor: 5.162
Authors: David C Goff; Donald M Lloyd-Jones; Glen Bennett; Sean Coady; Ralph B D'Agostino; Raymond Gibbons; Philip Greenland; Daniel T Lackland; Daniel Levy; Christopher J O'Donnell; Jennifer G Robinson; J Sanford Schwartz; Susan T Shero; Sidney C Smith; Paul Sorlie; Neil J Stone; Peter W F Wilson; Harmon S Jordan; Lev Nevo; Janusz Wnek; Jeffrey L Anderson; Jonathan L Halperin; Nancy M Albert; Biykem Bozkurt; Ralph G Brindis; Lesley H Curtis; David DeMets; Judith S Hochman; Richard J Kovacs; E Magnus Ohman; Susan J Pressler; Frank W Sellke; Win-Kuang Shen; Sidney C Smith; Gordon F Tomaselli Journal: Circulation Date: 2013-11-12 Impact factor: 29.690
Authors: Robert A Phillips; Jiaqiong Xu; Leif E Peterson; Ryan M Arnold; Joseph A Diamond; Adam E Schussheim Journal: J Am Coll Cardiol Date: 2018-03-07 Impact factor: 24.094
Authors: William C Cushman; Gregory W Evans; Robert P Byington; David C Goff; Richard H Grimm; Jeffrey A Cutler; Denise G Simons-Morton; Jan N Basile; Marshall A Corson; Jeffrey L Probstfield; Lois Katz; Kevin A Peterson; William T Friedewald; John B Buse; J Thomas Bigger; Hertzel C Gerstein; Faramarz Ismail-Beigi Journal: N Engl J Med Date: 2010-03-14 Impact factor: 91.245
Authors: Ralph B D'Agostino; Ramachandran S Vasan; Michael J Pencina; Philip A Wolf; Mark Cobain; Joseph M Massaro; William B Kannel Journal: Circulation Date: 2008-01-22 Impact factor: 29.690
Authors: Johannes A N Dorresteijn; Frank L J Visseren; Paul M Ridker; Annemarie M J Wassink; Nina P Paynter; Ewout W Steyerberg; Yolanda van der Graaf; Nancy R Cook Journal: BMJ Date: 2011-10-03
Authors: Bohdan B Khomtchouk; Diem-Trang Tran; Kasra A Vand; Matthew Might; Or Gozani; Themistocles L Assimes Journal: Brief Bioinform Date: 2020-12-01 Impact factor: 11.622
Authors: Andreas D Meid; Carmen Ruff; Lucas Wirbka; Felicitas Stoll; Hanna M Seidling; Andreas Groll; Walter E Haefeli Journal: Clin Epidemiol Date: 2020-11-02 Impact factor: 4.790