Validation of the WATCH-DM and TRS-HFDM Risk Scores to Predict the Risk of Incident Hospitalization for Heart Failure Among Adults With Type 2 Diabetes: A Multicohort Analysis.

Background The WATCH-DM (weight [body mass index], age, hypertension, creatinine, high-density lipoprotein cholesterol, diabetes control [fasting plasma glucose], ECG QRS duration, myocardial infarction, and coronary artery bypass grafting) and TRS-HFDM (Thrombolysis in Myocardial Infarction [TIMI] risk score for heart failure in diabetes) risk scores were developed to predict risk of heart failure (HF) among individuals with type 2 diabetes. WATCH-DM was developed to predict incident HF, whereas TRS-HFDM predicts HF hospitalization among patients with and without a prior HF history. We evaluated the model performance of both scores to predict incident HF events among patients with type 2 diabetes and no history of HF hospitalization across different cohorts and clinical settings with varying baseline risk. Methods and Results Incident HF risk was estimated by the integer-based WATCH-DM and TRS-HFDM scores in participants with type 2 diabetes free of baseline HF from 2 randomized clinical trials (TECOS [Trial Evaluating Cardiovascular Outcomes With Sitagliptin], N=12 028; and Look AHEAD [Look Action for Health in Diabetes] trial, N=4867). The integer-based WATCH-DM score was also validated in electronic health record data from a single large health care system (N=7475). Model discrimination was assessed by the Harrell concordance index and calibration by the Greenwood-Nam-D'Agostino statistic. HF incidence rate was 7.5, 3.9, and 4.1 per 1000 person-years in the TECOS, Look AHEAD trial, and electronic health record cohorts, respectively. Integer-based WATCH-DM and TRS-HFDM scores had similar discrimination and calibration for predicting 5-year HF risk in the Look AHEAD trial cohort (concordance indexes=0.70; Greenwood-Nam-D'Agostino P>0.30 for both). Both scores had lower discrimination and underpredicted HF risk in the TECOS cohort (concordance indexes=0.65 and 0.66, respectively; Greenwood-Nam-D'Agostino P<0.001 for both). In the electronic health record cohort, the integer-based WATCH-DM score demonstrated a concordance index of 0.73 with adequate calibration (Greenwood-Nam-D'Agostino P=0.96). TRS-HFDM score could not be validated in the electronic health record because of unavailability of data on urine albumin/creatinine ratio in most patients in the contemporary clinical practice. Conclusions The WATCH-DM and TRS-HFDM risk scores can discriminate risk of HF among intermediate-risk populations with type 2 diabetes.

fasting plasma glucose

Look Action for Health in Diabetes

sodium‐glucose cotransporter‐2 inhibitors

type 2 diabetes

Trial Evaluating Cardiovascular Outcomes With Sitagliptin

Thrombolysis in Myocardial Infarction (TIMI) risk score for heart failure in diabetes

urine albumin/creatinine ratio

weight (body mass index), age, hypertension, creatinine, high‐density lipoprotein cholesterol, diabetes control (fasting plasma glucose), ECG QRS duration, myocardial infarction, and coronary artery bypass grafting

Clinical Perspective

What Is New?

This study demonstrates that the WATCH‐DM (weight [body mass index], age, hypertension, creatinine, high‐density lipoprotein cholesterol, diabetes control [fasting plasma glucose], ECG QRS duration, myocardial infarction, and coronary artery bypass grafting) and TRS‐HFDM (Thrombolysis in Myocardial Infarction [TIMI] risk score for heart failure in diabetes) risk scores can discriminate risk of heart failure (HF) among low‐ and intermediate‐risk populations with type 2 diabetes.

Among high‐risk cohorts, neither risk score was well calibrated, and they tended to underestimate HF risk.

The WATCH‐DM risk score can be calculated from data routinely collected in the electronic health record.

What Are the Clinical Implications?

The WATCH‐DM and TRS‐HFDM risk scores were able to stratify HF risk among adults with type 2 diabetes and differing degrees of baseline cardiovascular disease risk.

Future studies are needed to evaluate whether diabetes‐specific risk scores can improve use of effective preventive interventions, such as sodium‐glucose cotransporter‐2 inhibitors, to lower HF risk.

Type 2 diabetes (T2D) affects >30 million adults in the United States and is an independent risk factor for heart failure (HF). , Over the past 2 decades, there has been a shift in cardiovascular complications observed in T2D with greater hospitalizations for HF compared with atherosclerotic cardiovascular disease. , Moreover, individuals with both T2D and HF are subject to a higher risk of all‐cause and cardiovascular death. Even with adequate control of glycemic status and other cardiovascular risk factors, the increased risk of HF among individuals with T2D persists, highlighting the need for novel approaches to its prevention. , Recent therapeutic advances in pharmacotherapies, such as sodium‐glucose cotransporter‐2 inhibitors (SGLT2i), have shown to be beneficial in preventing HF among patients with T2D. However, the uptake of these therapies in patients with T2D has been low. Identification of individuals with T2D who are at the highest risk of developing HF is key for efficient and cost‐effective allocation of preventive therapies. To this end, recent studies have focused on developing simple and accurate risk scores using clinical, laboratory, and electrocardiographic variables to predict risk of HF development over short‐term follow‐up. , , ,

Among HF risk scores developed specifically for patients with T2D, the WATCH‐DM (weight [body mass index], age, hypertension, creatinine, high‐density lipoprotein cholesterol, diabetes control [fasting plasma glucose {FPG}], ECG QRS duration, myocardial infarction [MI], and coronary artery bypass grafting [CABG]) and TRS‐HFDM (Thrombolysis in Myocardial Infarction [TIMI] risk score for heart failure in diabetes) risk scores have demonstrated good performance to predict short‐term risk of HF among adults with T2D. , WATCH‐DM was developed to predict 5‐year incident HF risk using data from the Action to Control Cardiovascular Risk in Diabetes (ACCORD) trial (median follow‐up, 4.9 years) and externally validated among participants with diabetes from the Antihypertensive and Lipid‐Lowering Treatment to Prevent Heart Attack Trial (ALLHAT) (median follow‐up, 4.8 years) and pooled community cohorts (up to 5 years). , , TRS‐HFDM was developed in the Saxagliptin Assessment of Vascular Outcomes Recorded in Patients With Diabetes Mellitus‐Thrombolysis in Myocardial Infarction 53 (SAVOR‐TIMI 53) trial (median follow up, 2.1 years) and externally validated in Dapagliflozin Effect on Cardiovascular Events‐Thrombolysis in Myocardial Infarction 58 (DECLARE‐TIMI 58) (median follow up, 4.2 years) and ACCORD trial cohorts to predict risk of HF hospitalization among patients with as well as without a history of HF. , HF risk stratification is more relevant in patients without a history of HF, especially among those with T2D, in whom there is considerable heterogeneity in risk for developing HF and SGLT2i have emerged as important preventive therapies. In contrast, patients with established HF are at high risk for complications and warrant aggressive implementation of guideline‐directed medical therapy, including SGLT2i. Both scores have demonstrated good performance in predicting HF risk among individuals without history of HF, highlighting their utility for risk stratification. However, performance of the WATCH‐DM and TRS‐HFDM scores to predict incident HF risk across cohorts with different baseline risk and among contemporary patients outside the clinical trial setting is uncertain. Accordingly, in this study, we aimed to evaluate the performance of the WATCH‐DM and TRS‐HFDM risk scores for incident HF risk prediction in 3 separate cohorts: participants with higher atherosclerotic cardiovascular disease burden from the TECOS (Trial Evaluating Cardiovascular Outcomes With Sitagliptin) and participants with lower atherosclerotic cardiovascular disease risk from the Look AHEAD (Look Action for Health in Diabetes) trial, and a real‐world cohort of patients from the University of Texas (UT) Southwestern Medical Center.

Methods

Our study data will not be made available to other researchers for purposes of reproducing the results because of institutional review board and clinical trial restrictions. We analyzed data from the TECOS and Look AHEAD clinical trials and from electronic health records (EHRs) at UT Southwestern Medical Center. In all cohorts comprising adults with T2D, participants with a history of HF were excluded. Data from the Look AHEAD trial were obtained from the National Institute of Diabetes and Digestive and Kidney Disease Repository. The TECOS database is located at the Duke Clinical Research Institute (Durham, NC). All trial participants provided written informed consent, and the studies were approved by the ethics committees for each participating trial site. Analyses were performed at UT Southwestern and the Duke Clinical Research Institute. The analyses of the UT Southwestern EHR data and the combined analyses presented herein were deemed exempt by the Institutional Review Board at the UT Southwestern Medical Center (Dallas, TX).

Study Populations

The details of the Look AHEAD trial have previously been published and are described (Data S1). Briefly, the Look AHEAD trial was a randomized, multicenter, clinical trial involving 5145 participants with T2D randomized to either an intensive lifestyle intervention focused on weight loss by reduced caloric intake plus increased physical activity (intervention arm) or diabetes support and education alone (control arm) to determine the impact of an intensive lifestyle intervention on the development of cardiovascular disease. Among 5145 participants in the Look AHEAD trial cohort, 239 were not included in the National Institute of Diabetes and Digestive and Kidney Disease data set. We further excluded 39 participants with a history of HF. The final analysis cohort included 4867 participants.

The design and results of TECOS have previously been published and are detailed (Data S1). , Briefly, TECOS was a double‐blind, multicenter, randomized trial of 14 671 participants evaluating the cardiovascular safety of the dipeptidyl peptidase 4 inhibitor sitagliptin versus placebo. Among 14 671 participants in the TECOS cohort, we excluded 2643 with a history of HF. The final analysis cohort included 12 028 participants.

The EHR cohort included patients at the UT Southwestern Medical Center, who were registered in the institution’s diabetes registry as of December 31, 2014. Patients were included in the diabetes registry if they had an active problem of T2D listed in the EHR. Patients who were also listed in the institution’s HF registry as of December 31, 2014, were excluded. Patients were similarly added to the HF registry if they had an active problem of HF listed or had a previous clinical encounter, either ambulatory or inpatient, for HF. The study cohort included 17 929 patients with a diagnosis of T2D and free of HF as of December 31, 2014. We excluded 4413 patients who did not have at least one recorded clinic visit before December 31, 2019, 3807 patients with laboratory data not recorded within 12 months of each other, and 2234 patients with >20% missingness of the WATCH‐DM covariates. The final analysis cohort included 7475 patients.

Clinical Covariates

Clinical and laboratory values for the Look AHEAD trial and TECOS cohorts were obtained using a standardized protocol, as published previously and described (Data S1). , For the EHR cohort, the baseline data were abstracted from the health record from the closest visit before start of the follow‐up period. A detailed description is provided (Data S1). Among participants from the Look AHEAD trial and EHR cohorts, missing data were imputed using random forest imputation. In the TECOS cohort, missing data were imputed on the basis of the fully conditional specification method, taking into consideration the joint distribution of other variables.

Outcomes of Interest

The primary outcome of interest for the present analysis was incident HF hospitalization. In the Look AHEAD trial cohort, incident HF events were adjudicated by a committee of physicians blinded to randomization after reviewing medical records, including the medical history, test results, and medication use, as reported previously. The median follow‐up in the Look AHEAD trial data available through the National Institute of Diabetes and Digestive and Kidney Disease was 9.6 (interquartile range [IQR], 8.9–10.3) years. Because the WATCH‐DM and TRS‐HFDM risk scores were developed to predict short‐term risk of HF (up to 5‐year risk assessed in WATCH‐DM and up to 4.8‐year risk for TRS‐HFDM across different cohorts), , the Look AHEAD trial follow‐up was censored at 5 years for the present analysis.

In the TECOS cohort, incident hospitalization for HF was adjudicated as a prespecified secondary end point of the trial. Incident hospitalization for HF was defined as an inpatient admission or emergency department visit >12 hours with clinical manifestations of HF and additional treatment with a diuretic, inotrope, vasodilator therapy, or mechanical or surgical intervention for hemodynamic support. All events were prospectively collected and centrally adjudicated by an independent clinical committee of physicians masked to treatment assignment. The median follow‐up was 3.0 (IQR, 2.3–3.7) years, and follow‐up was censored at 4 years.

In the EHR cohort, the first episode of HF hospitalization was considered as an incident HF event. Time to the first HF event was calculated from the latest clinic visit date before December 31, 2014 (start of the outcome assessment period) to the first hospitalization with HF recorded as a physician‐entered diagnosis on the hospital encounter form. Diagnoses were selected from the clinical terms available within the EHR (Intelligent Medical Objects, Chicago, IL) that were mapped to both Systematized Nomenclature of Medicine ‐ Clinical Terms (SNOMED CT) and International Classification of Diseases, Ninth Revision, and International Classification of Diseases, Tenth Revision, codes, as previously described. HF was defined with SNOMED CT hierarchies for HF using the specific codes listed in Tables S1 and S2. All patient outcomes were censored at December 31, 2019, or at 5 years after the latest clinic visit date before December 31, 2014. The median follow‐up was 5.7 (IQR, 4.0–6.0) years.

WATCH‐DM Risk Score

The details and derivation of the WATCH‐DM risk score have been previously published. Briefly, the WATCH‐DM risk score was developed to predict risk of incident HF and incorporates clinical, laboratory, and ECG parameters. Depending on the clinical use case, separate integer‐, regression‐, and machine learning (ML)–based models (henceforth referred to as WATCH‐DM[i], WATCH‐DM[r], and WATCH‐DM[ml], respectively) were developed. Specifically, the integer‐based model was designed to facilitate the ease of use in clinical settings without the need for a web‐based platform or programming into the EHR system. The WATCH‐DM score was developed using data from the ACCORD trial and validated in participants with T2D from the ALLHAT trial. , The risk score includes 10 variables, of which 3 are binary (QRS >120 milliseconds, history of MI, and history of CABG) and 7 are continuous (body mass index, age, systolic blood pressure, diastolic blood pressure, serum creatinine, high‐density lipoprotein cholesterol, and FPG). For the present analyses, because ECG data were not available in Look AHEAD trial National Institute of Diabetes and Digestive and Kidney Disease data set and ECGs and FPG were not available in TECOS, the WATCH‐DM risk score was rederived excluding QRS duration and replacing FPG for hemoglobin A1c using the same methods as the original score and described in the Supplemental Methods (Data S1). The rederived WATCH‐DM risk integer score is shown in Figure S1. The performance of the rederived WATCH‐DM risk scores is shown in Figure S2. To facilitate clinical use of the WATCH‐DM(ml) model, prevent sharing of protected health data, and adhere to data use agreements, the model was implemented using an application programming interface, as detailed (Data S1).

TRS‐HFDM Risk Score

Details of the TRS‐HFDM risk score have been previously described. TRS‐HFDM is an integer‐based risk score to predict risk of HF hospitalization with the following variables and corresponding points: prior HF (2 points), history of atrial fibrillation (1 point), coronary artery disease (1 point), estimated glomerular filtration rate <60 mL/min per 1.73 m2 (1 point), and urine albumin/creatinine ratio (UACR) >300 mg/g (2 points) or 30 to 300 mg/g (1 point). The risk score was developed using data from the SAVOR‐TIMI 53 trial and validated in participants from the DECLARE‐TIMI 58 and ACCORD trials. , , All participants were assigned 0 points for prior HF because history of HF was an exclusion criterion for the present study.

Statistical Analysis

Baseline characteristics for each of the cohorts were compared across quintiles of WATCH‐DM(i) and categories of TRS‐HFDM scores (0, 1, 2, and ≥3 point categories) and summarized as median (25th–75th percentiles) for continuous and number (percentage) for categorical variables. The unadjusted 5‐year risk of incident HF was estimated in the EHR and Look AHEAD trial cohorts using Kaplan‐Meier curves. Because of shorter follow‐up in TECOS, the 4‐year risk of incident HF was calculated. In addition, given UACR was missing in 63.4% of participants in TECOS, analysis of TRS‐HFDM was restricted to only those with available UACR data (n=4408). The cumulative risk of incident HF was assessed across WATCH‐DM(i) quintiles and TRS‐HFDM categories. Model performance was evaluated according to discrimination, assessed by the Harrell concordance index (C‐index). Calibration was assessed by the Greenwood‐Nam‐D’Agostino method, with adequate calibration defined a priori as Greenwood‐Nam‐D’Agostino P>0.05. , , Unadjusted Cox proportional hazard models were constructed to evaluate the association of score categories with risk of incident HF. Sensitivity analyses were also conducted to (1) compare predicted versus observed event rates across original WATCH‐DM(i) score categories ; and (2) assess the performance of the WATCH‐DM(ml) model in a less restrictive EHR cohort (ie, not requiring patients to have variables collected within 12 months of the baseline visit and not excluding patients because of missing variables).

Performance of WATCH‐DM(i) was compared with TRS‐HFDM risk score in the TECOS and Look AHEAD trial cohorts using the same model performance metrics discussed previously. TRS‐HFDM performance was unable to be assessed in the EHR cohort because of lack of UACR data. Decision curve analysis, a measure between the number of true‐positive cases identified without an increase in false‐positive rate, was performed to compare the clinical net benefit between models. Given the lack of a consensus patient risk threshold for HF treatment, harm was removed from the decision curve calculation and risk was assessed at the cohort‐specific event rate.

Analyses were performed using either R version 3.6.3 (R Foundation, Vienna, Austria) for the Look AHEAD trial and EHR analyses or SAS version 14.3 (SAS Institute, Cary, NC) for the TECOS analysis, with a 2‐sided P<0.05 indicating statistical significance.

Results

Baseline characteristics of the participants stratified by cohort are shown in Table 1. Participants in the TECOS were older and more likely to be men (Table 1). Among the 3 study cohorts, participants in the TECOS also had a higher average blood pressure, longer duration of diabetes diagnosis, higher serum creatinine, and higher prevalence of prior MI and CABG. Conversely, participants of the Look AHEAD trial were younger, had the lowest percentage of men, had the lowest baseline blood pressure and serum creatinine, and had the lowest prevalence of prior MI and CABG. Patients in the EHR cohort had a median age of 60 years, half were men (49.8%), and the group had the lowest percentage of self‐reported White race (54.6%) and highest median hemoglobin A1c (7.3%) (Table 1). HF incidence rate was 7.5, 3.9, and 4.1 per 1000 person‐years in the TECOS, Look AHEAD trial, and electronic health record cohorts, respectively.

Table 1

Baseline Characteristics of Participants in the Look AHEAD Trial, TECOS, and EHR Cohorts

Characteristic	Look AHEAD trial	TECOS	EHR
Total No.	4867	12 028	7475
Age, y	59 (55–63)	65 (59–71)	60 (50–68)
Men	2022 (41.5)	8668 (72.1)	3724 (49.8)
White race	3228 (66.3)	7761 (64.5)	4083 (54.6)
Black race	795 (16.3)	399 (3.3)	1650 (22.1)
Others or unknown	844 (17.3)	3868 (32.2)	1742 (23.3)
Body mass index, kg/m2	34.9 (31.5–39.4)	29.1 (26.0–32.8)	30.8 (26.6–36.3)
Systolic BP, mm Hg	129 (117–141)	133 (124–145)	131 (120–145)
Diastolic BP, mm Hg	70 (64–77)	78 (70–83)	76 (68–83)
Diabetes duration, y	5 (2–10)	10 (5–16)	…
Serum creatinine, mg/dL	0.8 (0.7–0.9)	1.0 (0.8–1.1)	0.9 (0.7–1.2)
HDL‐c, mg/dL	42 (35–50)	42 (35–50)	45 (37–56)
HbA1c, %	7.1 (6.4–7.9)	7.2 (6.8–7.7)	7.3 (6.5–7.9)
Prior MI	287 (5.9)	4667 (38.8)	992 (13.3)
Prior CABG	119 (2.4)	2946 (24.5)	592 (7.9)
Insulin use	896 (18.4)	2639 (21.9)	…
WATCH‐DM(i) score	13 (11–15)	14 (12–17)	10 (8–13)
TRS‐HFDM score	0 (0–1)	1 (1–2)	…

Values are displayed as median (25th–75th percentiles) for continuous and number (percentage) for categorical variables. BP indicates blood pressure; CABG, coronary artery bypass grafting; EHR, electronic health record; HbA1c, hemoglobin A1c; HDL‐c, high‐density lipoprotein cholesterol; Look AHEAD, Look Action for Health in Diabetes; MI, myocardial infarction; TECOS, Trial Evaluating Cardiovascular Outcomes With Sitagliptin; TRS‐HFDM, Thrombolysis in Myocardial Infarction (TIMI) risk score for heart failure in diabetes; and WATCH‐DM(i), integer‐based weight (body mass index), age, hypertension, creatinine, high‐density lipoprotein cholesterol, diabetes control (fasting plasma glucose), ECG QRS duration, myocardial infarction, and coronary artery bypass grafting.

Performance of WATCH‐DM and TRS‐HFDM in the Look AHEAD Trial Cohort

Among 4867 participants from the Look AHEAD trial, 91 developed HF within 5 years from enrollment, with a 5‐year Kaplan‐Meier risk estimate of 1.83%. The median WATCH‐DM(i) score was 13 (IQR, 11–15), with an observed range of 3 to 26. Baseline characteristics of participants stratified by WATCH‐DM(i) quintiles are shown in Table S3. The WATCH‐DM(i) score demonstrated good discrimination, with a C‐index of 0.70 (95% CI, 0.64–0.76) and adequate calibration (P=0.39) for predicting HF risk in the Look AHEAD trial (Table 2 and Figure S3). Similar results were observed across original WATCH‐DM(i) score categories (Figure S4). The 5‐year incidence of HF increased across quintiles of WATCH‐DM(i), ranging from 0.90% in quintile 1 to 5.97% in quintile 5 (Figure 1A). Event rates across quintiles are shown in Table S4. The risk of incident HF was almost 7‐fold higher in quintile 5 compared with quintile 1 (hazard ratio [HR], 6.79; 95% CI, 3.32–15.27) (Table S5). The WATCH‐DM(ml) and WATCH‐DM(r) scores demonstrated superior discrimination, with C‐indexes of 0.76 (95% CI, 0.70–0.82) and 0.73 (95% CI, 0.67–0.78), respectively (Table 2), and adequate calibration (P=0.61 and P=0.16, respectively) (Figure S3).

Table 2

Discrimination and Calibration Metrics of the WATCH‐DM(i), WATCH‐DM(r), WATCH‐DM(ml), and TRS‐HFDM Scores for Predicting Risk of Incident HF in Each Cohort Analyzed

Variable	Look AHEAD trial		TECOS*		EHR
	C‐index (95% CI)	GND P value	C‐index (95% CI)	GND P value	C‐index (95% CI)	GND P value
WATCH‐DM(i)	0.70 (0.64–0.76)	0.39	0.65 (0.61–0.68)	<0.001	0.73 (0.69–0.77)	0.96
WATCH‐DM(r)	0.73 (0.67–0.78)	0.16	0.67 (0.63–0.70)	<0.001	0.73 (0.69–0.78)	0.27
WATCH‐DM(ml)	0.76 (0.70–0.82)	0.61	0.63 (0.59–0.67)	<0.001	0.77 (0.73–0.80)	0.42
TRS‐HFDM	0.70 (0.65–0.75)	0.84	0.66 (0.60–0.72)	<0.001	Could not be assessed

The 5‐year risk of HF was assessed in the Look AHEAD trial and EHR cohorts and 4‐year risk in the TECOS cohort. C‐index indicates concordance index; EHR, electronic health record; GND, Greenwood‐Nam‐D’Agostino; HF, heart failure; Look AHEAD, Look Action for Health in Diabetes; TECOS, Trial Evaluating Cardiovascular Outcomes With Sitagliptin; TRS‐HFDM, Thrombolysis in Myocardial Infarction (TIMI) risk score for heart failure in diabetes; WATCH‐DM, weight (body mass index), age, hypertension, creatinine, high‐density lipoprotein cholesterol, diabetes control (fasting plasma glucose), ECG QRS duration, myocardial infarction, and coronary artery bypass grafting; WATCH‐DM(i), integer‐based WATCH‐DM; WATCH‐DM(ml), machine learning–based WATCH‐DM; and WATCH‐DM(r), regression‐based WATCH‐DM.

Because of limited availability of urine albumin/creatinine ratio data, risk score performance in TECOS was assessed in 12 028 participants for WATCH‐DM and 4408 participants in TECOS.

Figure 1

Cumulative incidence of heart failure (HF) in the Look AHEAD (Look Action for Health in Diabetes) trial (A and B), TECOS (Trial Evaluating Cardiovascular Outcomes With Sitagliptin) (C and D), and electronic health record (EHR) (E) validation cohorts across WATCH‐DM(i) (integer‐based weight [body mass index], age, hypertension, creatinine, high‐density lipoprotein cholesterol, diabetes control [fasting plasma glucose], ECG QRS duration, myocardial infarction, and coronary artery bypass grafting) and TRS‐HFDM (Thrombolysis in Myocardial Infarction (TIMI) risk score for heart failure in diabetes) risk scores.

The 5‐year risk was assessed in the Look AHEAD trial and EHR cohorts and 4‐year risk in the TECOS cohort.

The median TRS‐HFDM score was 0 (IQR, 0–1), with an observed range of 0 to 1. Baseline characteristics of participants stratified by TRS‐HFDM are shown in Table S6. The TRS‐HFDM score demonstrated similar performance to the WATCH‐DM(i) score, with a C‐index of 0.70 (95% CI, 0.65–0.75) and adequate calibration (P=0.84) (Table 2 and Figure S5A). The incidence of HF increased from 0.91% for score of 0 to 16.10% for scores of ≥3 (Figure 1B and Table S4). The risk of incident HF was significantly higher for participants with scores of ≥3 compared with score of 0 (HR, 20.04; 95% CI, 9.49–42.34) (Table S5).

In decision curve analysis, WATCH‐DM(i) identified 2 additional HF events per 1000 participants compared with the TRS‐HFDM risk score (Figure 2A).

Figure 2

Decision curve analysis of the WATCH‐DM(i) (integer‐based weight [body mass index], age, hypertension, creatinine, high‐density lipoprotein cholesterol, diabetes control [fasting plasma glucose], ECG QRS duration, myocardial infarction, and coronary artery bypass grafting) and TRS‐HFDM (Thrombolysis in Myocardial Infarction (TIMI) risk score for heart failure in diabetes) risk scores in the Look AHEAD (Look Action for Health in Diabetes) trial (A) and TECOS (Trial Evaluating Cardiovascular Outcomes With Sitagliptin) (B) validation cohorts.

At a 1.9% risk threshold (the overall heart failure event rate) in the Look AHEAD trial cohort, the WATCH‐DM(i) risk score identified 2 additional heart failure events per 1000 individuals compared with the TRS‐HFDM risk score. Similarly, at a 3% risk threshold in the TECOS cohort, the WATCH‐DM(i) risk score identified 2 additional heart failure events per 1000 individuals compared with the TRS‐HFDM risk score.

Performance of WATCH‐DM and TRS‐HFDM in the TECOS cohort

In 12 028 participants from the TECOS, 266 developed HF during follow‐up, with a 4‐year Kaplan‐Meier risk of 3.1%. The median WATCH‐DM(i) score was 14 (IQR, 12–17), with an observed range of 3 to 30. Baseline characteristics of participants across quintiles of WATCH‐DM(i) are shown in Table S7. The WATCH‐DM(i), WATCH‐DM(r), and WATCH‐DM(ml) models demonstrated modest discrimination for predicting risk of HF in the TECOS, with C‐indexes of 0.65 (95% CI, 0.61–0.68), 0.67 (95% CI, 0.63–0.70), and 0.63 (95% CI, 0.59–0.67), respectively (Table 2). Evidence of miscalibration was observed (P<0.001) in all models particularly underpredicting observed risk in the highest deciles and categories (Figures S3 through S4). The WATCH‐DM(i) score was able to stratify the incidence of HF at year 4 from 1.85% in quintile 1 to 6.66% in quintile 5 (Figure 1C and Table S4). Similarly, the risk of incident HF was nearly 4‐fold higher for quintile 5 when compared with quintile 1 (HR, 3.78; 95% CI, 2.54–5.61) (Table S5).

Among 4408 participants with available UACR data, the median TRS‐HFDM score was 1 (IQR, 1–2), with an observed range of 0 to 5. Baseline characteristics across score categories are shown in Table S8. The TRS‐HFDM score demonstrated modest discrimination, with a C‐index of 0.66 (95% CI, 0.60–0.72) for predicting HF risk in the TECOS cohort (Table 2). Evidence of miscalibration was observed with consistent underestimated HF risk and a Greenwood‐Nam‐D’Agostino P<0.001 (Table 2 and Figure S5B). The incidence of HF increased from 1.16% for a score of 0 to 6.42% for scores of ≥3 (Figure 1D and Table S4). Compared with participants with a score of 0, the risk of incident HF was >6‐fold higher for scores of ≥3 (HR, 6.71; 95% CI, 2.71–16.60) (Table S5). In decision curve analysis, WATCH‐DM(i) identified 2 additional HF events per 1000 individuals compared with the TRS‐HFDM risk score (Figure 2B).

External Validation of WATCH‐DM in the EHR Cohort

Among 7475 patients from the UT Southwestern Medical Center EHR cohort, 133 developed incident HF within 5 years from the baseline clinic visit, with a Kaplan‐Meier risk of 1.88%. The median WATCH‐DM(i) score was 10 (IQR, 8–13), with an observed range of 1 to 28. Baseline characteristics of participants stratified by WATCH‐DM(i) quintiles are shown in Table S9. The WATCH‐DM(i) score demonstrated good discrimination, with a C‐index of 0.73 (95% CI, 0.69–0.77) and adequate calibration (P=0.96) (Table 2 and Figures S3 and S4). The incidence of HF increased across quintiles of WATCH‐DM(i), ranging from 0.56% in quintile 1 to 5.29% in quintile 5 (Figure 1E and Table S4). The risk of incident HF was nearly 9‐fold higher in quintile 5 compared with quintile 1 (HR, 8.98; 95% CI, 4.46–18.06) (Table S5). The WATCH‐DM(r) and WATCH‐DM(ml) scores also demonstrated good discrimination, with C‐indexes of 0.73 (95% CI, 0.69–0.78) and 0.77 (95% CI, 0.73–0.80) (Table 2), and adequate calibration (P=0.27 and P=0.42, respectively) (Figure S3). In sensitivity analysis, we liberalized the cohort to include patients with laboratory values collected at different times (ie, >12 months of the baseline visit) or patients with >20% variable missingness. The analysis cohort included 13 516 patients (75.4% of candidate cohort). Among 13 516 patients, 8947 (66.2%) were missing variables required for WATCH‐DM (average participant variable missingness, 10% [IQR, 0%–30%]). Even in the presence of considerable missingness, the WATCH‐DM(ml) score demonstrated modest discrimination, with C‐index of 0.70 (95% CI, 0.67–0.72), and adequate calibration (P=0.11).

Discussion

In the present study, we evaluated the performance of the novel HF risk scores for T2D, the WATCH‐DM score and the TRS‐HFDM score, in large external cohorts of individuals with varying cardiovascular risk profiles. Among adults with T2D and intermediate cardiovascular disease risk from the Look AHEAD trial cohort, both risk scores predicted 5‐year risk of incident HF with adequate discrimination and calibration. Conversely, among the high‐risk cohort of adults with T2D in the TECOS, neither WATCH‐DM nor TRS‐HFDM was well calibrated and tended to underestimate HF risk. The WATCH‐DM risk score detected an additional 2 HF events per 1000 individuals compared with the TRS‐HFDM risk score in both the Look AHEAD trial and TECOS cohorts. Finally, we demonstrated that the WATCH‐DM risk score can be calculated from data acquired from routine clinical care in the EHR.

Over the past 2 decades, among adults with T2D, there has been substantial progress in reducing hospitalizations for ischemic heart disease but less so for HF. Primary cardiovascular disease prevention guidelines endorse use of the pooled cohort equations to estimate 10‐year risk of atherosclerotic cardiovascular disease to inform allocation of preventive therapies to reduce risk, but there is no specific instrument recommended for HF risk assessment to similarly inform clinical decision making. Multiple HF risk prediction tools are available, but none has been widely accepted because, in part, of the need for robust validation. The pooled cohort equations to prevent HF were developed in pooled epidemiologic cohorts to predict incident HF and were validated in a low‐risk cohort free of cardiovascular disease. , However, these prior HF risk prediction equations were not specific for T2D and excluded patients with prevalent coronary heart disease, a common comorbidity and HF risk factor among patients with T2D, thereby limiting their generalizability. The integer‐based TRS‐HFDM risk score was developed using clinical trial data to predict HF hospitalization in T2D and was externally validated in a separate trial cohort. , However, the TRS‐HFDM risk score requires UACR to estimate risk, but this is not routinely assessed in clinical practice, limiting its use.

In the present study, we evaluated the performance of the WATCH‐DM(i) and TRS‐HFDM risk scores in cohorts with intermediate (Look AHEAD trial) and high baseline cardiovascular risk (TECOS). Both the WATCH‐DM and TRS‐HFDM scores identified a wide gradient of risk across categories with good discrimination across cohorts with variable baseline cardiovascular disease risk. Both scores also had adequate calibration in the intermediate‐ but not high‐risk cohort. In the high‐risk cohort, the WATCH‐DM risk score underestimated HF risk, especially in the highest deciles of risk. For WATCH‐DM, miscalibration was likely related to differences in baseline cardiovascular disease risk among the derivation (ACCORD) and validation study (TECOS) populations. Notably, the TECOS cohort included primarily men (72%) with high risk for developing atherosclerotic cardiovascular disease. Recalibration of the contribution of history of MI and history of CABG in a higher‐risk cohort or including other complications of T2D as variables, such as cerebrovascular disease or peripheral vascular disease, may yield improved results. The TRS‐HFDM risk score underestimated HF risk in TECOS compared with lower‐risk cohort, such as the Look AHEAD trial and ACCROD trial cohorts. Overall, both risk scores may have better performance in lower‐ than in higher‐risk populations. Even in the SAVOR‐TIMI cohort and the DECLARE‐TIMI cohort, where the TIMI‐HFDM was developed, was initially validated, and demonstrated good performance, the proportion of participants with prior MI, peripheral vascular disease, and cerebrovascular disease was lower than that observed in the TECOS validation cohort. These differences in ischemic vascular disease and associated risk of HF may explain the variability in performance of TRS‐HFDM across cohorts. It is also noteworthy that TRS‐HFDM risk score was developed to predict HF hospitalization among patients with diabetes with or without history of HF, with prevalent HF contributing 2 points (of maximum 8 points) toward the risk score. In the present study, we used TRS‐HFDM to predict incident HF with 0 points assigned to the history of HF criteria, which may have further limited its performance in TECOS compared with other high‐risk cohorts, such as DECLARE‐TIMI cohort.

There are several strengths to the WATCH‐DM risk score with respect to its implementation for management of patients with T2D in the contemporary clinical setting.

We demonstrated easy applicability of the risk score in a health system’s EHR for efficient risk estimation in clinical practice. WATCH‐DM could be calculated for most individuals without significant exclusions required for data missingness. Comparatively, because of lack of available UACR data, the TRS‐HFDM score could not be calculated in the EHR cohort and only in <40% of participants in TECOS. Clinical decision support tools in prevalent HF are associated with improved medication adherence and appropriate referrals for advanced therapies, but less is known about their impact on HF prevention. , In addition to the feasibility from an implementation standpoint, WATCH‐DM demonstrated adequate discrimination and calibration for predicting HF risk in a real‐world contemporary clinical cohort of patients with T2D. These observations highlight the potential for wide generalizability and applicability of the WATCH‐DM risk score. Participants included in the EHR cohort are not subject to strict enrollment criteria that often challenge the generalizability of findings from clinical trials. In addition, the clinical data used to estimate 5‐year incident HF risk were obtained as part of routine clinical care rather than standardized protocols used in research studies.

ML models also afford several advantages compared with traditional risk modeling techniques. Notably, ML models can be updated to tailor to the available data, making them more usable across different populations. Similar to other maximum likelihood‐based modeling approaches, ML models can continue to function even in the presence of considerable missing data, as would be common in real‐world or EHR data registries. In the present study, we observed that the WATCH‐DM(ml) model performed well even in an EHR cohort with 65% of patients missing the required variables. In addition, we validated a proof‐of‐concept application programming interface to share the WATCH‐DM(ml) model while preserving data use agreements. Such a model allows for WATCH‐DM(ml) to be directly accessed by researchers or applications without directly sharing patient‐level data.

The present study findings have important clinical implications. Clinical practice recommendations suggest select individuals with T2D should be considered for weight loss therapies, including intensive lifestyle interventions and metabolic surgery, as well as specific medications, such as SGLT2i, based on comorbidity burden and risk for HF. , , Intentional weight loss, particularly targeting central adiposity, and prescription of SGLT2i are associated with lower risk of HF, but, currently, there are no validated risk scores recommended for HF risk stratification. , , Both WATCH‐DM and TRS‐HFDM are HF risk prediction tools that incorporate routinely assessed clinical data and are now validated in multiple cohorts. In the present analyses, we demonstrated that novel risk scores could help target preventive HF therapies, such as intentional weight loss interventions and SGLT2i, to individuals who have the highest risk for developing HF and are therefore most likely to experience the greatest absolute risk reductions for incident HF. Clinical decision support tools incorporating risk assessment may help target HF preventive therapies to individuals at the greatest risk for incident HF in a cost‐effective manner. In our study, both the WATCH‐DM and TRS‐HFDM risk scores had relatively worse performance in the high‐risk TECOS cohort. However, high‐risk patients, such as those in the TECOS cohort, are less in need for risk stratification and would benefit from preventive therapies because of high baseline event rate. The application of risk stratification models, such as WATCH‐DM and TRS‐HFDM, is more relevant for low‐ and intermediate‐risk populations, where these scores demonstrated adequate to good performance. Future studies are needed to determine if implementing these risk score in the EMR may improve uptake of evidence‐based therapies for HF prevention in patients with T2D.

This study has several notable strengths, including comparison of 2 novel T2D‐specific risk scores, validation in 2 large clinical trial cohorts with study populations who have varied baseline cardiovascular disease risk, additional validation in a real‐world EHR data set, inclusion of a diverse study population, and rigorous adjudication of HF events in the clinical trial cohorts, according to standardized protocols. However, the study findings should be interpreted in the context of several limitations. First, the WATCH‐DM risk score assessed in the present study was modified from its original derivation. ECGs and fasting blood samples are not routinely collected in clinical practice. The WATCH‐DM risk score was rederived excluding ECG data and substituting hemoglobin A1c for FPG using similar methods as the original analysis to ensure generalizability of the risk prediction tool in clinical practice. The rederived WATCH‐DM risk score was validated and had similar model performance for predicting HF risk as the original score (C‐index range, 0.72–0.76; Figure S2) ; however, with fewer variables, the ML‐based model was more prone to overfitting. Second, data from the EHR were collected according to routine clinical practice. WATCH‐DM was assessed in the real‐world EHR data set without strict data requirements, suggesting that this risk score may have generalizable use and implementation may be feasible. Also, UACR data were not captured consistently in the EHR data set to allow for evaluation of the TRS‐HFDM risk score. Similarly, <40% of participants in TECOS had available UACR data. As such, evaluation of TRS‐HFDM was limited to only those with available data. Third, because of the required ECG variables, we were unable to assess the performance of the pooled cohort equations to prevent HF risk score.13 Finally, both risk scores do not incorporate blood‐based biomarkers, such as high‐sensitivity cardiac troponin and natriuretic peptide levels, that are well‐established predictors of HF risk. , , Similarly, data on cardiometabolic measures, such as fat mass, cardiorespiratory fitness, and visceral adiposity, which are also well‐established risk factors of HF, were not included in either risk score. , , These factors are not commonly assessed in routine clinical practice and thus are not readily available in all patients with T2D for risk assessment. Future studies are needed to determine whether incorporation of these markers of risk can improve the predictive performance of the WATCH‐DM and TRS‐HFDM risk scores.

In conclusion, the WATCH‐DM and TRS‐HFDM risk scores were able to stratify HF risk among adults with T2D and differing degrees of baseline cardiovascular disease risk in 2 large, multicenter, clinical trials. The WATCH‐DM risk score was also validated in a contemporary EHR cohort from a large health system. Future studies are needed to further evaluate whether use of the WATCH‐DM or TRS‐HFDM risk scores can improve use of effective preventive interventions, such as SGLT2i, to lower the risk of HF.

Sources of Funding

The TECOS (Trial Evaluating Cardiovascular Outcomes With Sitagliptin) was funded by Merck Sharp & Dohme Corp, a subsidiary of Merck & Co, Inc, Kenilworth, NJ. Dr Vaduganathan is supported by the KL2/Catalyst Medical Research Investigator Training award from Harvard Catalyst (National Institutes of Health/National Center for Advancing Translational Sciences Award UL 1TR002541) and serves on advisory boards or has received research grant support from American Regent, Amgen, AstraZeneca, Baxter Healthcare, Bayer AG, Boehringer Ingelheim, Cytokinetics, and Relypsa. Dr Holman reports research support from AstraZeneca, Bayer, and Merck Sharp & Dohme, and personal fees from Anji Pharmaceuticals, AstraZeneca, Novartis, and Novo Nordisk. Dr. Kolkailah was supported by the National Heart, Lung, And Blood Institute of the National Institutes of Health under Award Number T32HL125247. The content is solely the responsibility of the authors and does not necessarily represent the official view of the National Institutes of Health. Dr McGuire has had leadership roles in clinical trials for AstraZeneca, Boehringer Ingelheim, Eisai, Esperion, GlaxoSmithKline, Janssen, Lexicon, Merck & Co, Inc, Novo Nordisk, CSL Behring, and Sanofi USA; and has received consultancy fees from AstraZeneca, Boehringer Ingelheim, Lilly USA, Merck & Co, Inc, Pfizer, Novo Nordisk, Metavant, Afimmune, and Sanofi. Dr Pandey received grant funding outside the present study from Applied Therapeutics; has received honoraria outside of the present study as an advisor/consultant for Tricog Health Inc, Lilly, USA, Rivus, and Roche Diagnostics; and has received nonfinancial support from Pfizer and Merck. Dr Pandey is supported by the Texas Health Resources Clinical Scholarship, Gilead Sciences Research Scholar Program, the National Institute of Aging GEMSSTAR Grant (1R03AG067960‐01), and grant support from Applied Therapeutics. Dr Peterson receives consulting from: advisory committees for Novo Nordisk, Novartis, Janssen, Pfizer, Bayer; and receives research support from: Janssen, Amgen, Esperion, BMS

Disclosures

All other authors report no disclosures or sources of funding.

Data S1

Tables S1–S9

Figures S1–S5

References 46–49

Click here for additional data file.