Clinical Prediction Models for Heart Failure Hospitalization in Type 2 Diabetes: A Systematic Review and Meta-Analysis.

Background Clinical prediction models have been developed for hospitalization for heart failure in type 2 diabetes. However, a systematic evaluation of these models' performance, applicability, and clinical impact is absent. Methods and Results We searched Embase, MEDLINE, Web of Science, Google Scholar, and Tufts' clinical prediction registry through February 2021. Studies needed to report the development, validation, clinical impact, or update of a prediction model for hospitalization for heart failure in type 2 diabetes with measures of model performance and sufficient information for clinical use. Model assessment was done with the Prediction Model Risk of Bias Assessment Tool, and meta-analyses of model discrimination were performed. We included 15 model development and 3 external validation studies with data from 999 167 people with type 2 diabetes. Of the 15 models, 6 had undergone external validation and only 1 had low concern for risk of bias and applicability (Risk Equations for Complications of Type 2 Diabetes). Seven models were presented in a clinically useful manner (eg, risk score, online calculator) and 2 models were classified as the most suitable for clinical use based on study design, external validity, and point-of-care usability. These were Risk Equations for Complications of Type 2 Diabetes (meta-analyzed c-statistic, 0.76) and the Thrombolysis in Myocardial Infarction Risk Score for Heart Failure in Diabetes (meta-analyzed c-statistic, 0.78), which was the simplest model with only 5 variables. No studies reported clinical impact. Conclusions Most prediction models for hospitalization for heart failure in patients with type 2 diabetes have potential concerns with risk of bias or applicability, and uncertain external validity and clinical impact. Future research is needed to address these knowledge gaps.

Action to Control Cardiovascular Risk in Diabetes trial

hospitalization for heart failure

Prediction Model Risk of Bias Assessment Tool

Risk Equations for Complications of Type 2 Diabetes

Thrombolysis in Myocardial Infarction Risk Score for HF in Diabetes

(Weight [BMI], Age, Hypertension, Creatinine, HDL‐C, Diabetes Control [fasting plasma glucose], QRS Duration, MI, and CABG) risk score

Clinical Perspective

What Is New?

Available evidence suggests that several widely available clinical risk tools can robustly prognosticate the risk of heart failure hospitalization in people with type 2 diabetes and identify those who may benefit most from novel guideline‐directed medical therapies.

The effect of most clinical prediction models on clinical outcomes, patient care, provider behaviors has largely been under investigated, as a result, health, economic, and clinical investigations of these tools are warranted.

What Are the Clinical Implications?

Given that several clinical prediction models have demonstrated robust prognostic accuracy, their implementation in clinical settings might facilitate the identification of high‐risk individuals with type 2 diabetes who may benefit most from guideline‐directed therapies such as sodium‐glucose co‐transporter 2 inhibitors.

Type 2 diabetes (T2D) contributes to >1.5 million annual deaths worldwide. This high rate of mortality can be partly attributed to the metabolic alterations that precipitate diabetic cardiomyopathy and its attendant complications. Correspondingly, the development of heart failure (HF) has emerged as one of the most common and important manifestations of cardiovascular disease in individuals with diabetes. The concern over HF initially stemmed from safety troubles with certain antidiabetic treatments (eg, thiazolidinediones), and the fact that hospitalizations for heart failure (HHF) portend a poor prognosis. , Thus, the use of prediction models for identifying patients with diabetes at particularly high risk for developing HF and HHF has become clinically important.

Clinical prediction models and risk prediction can facilitate shared decision‐making, recruitment into clinical trials, and the selection of patients who may benefit most from therapies that reduce the risk of HHF events (eg, sodium‐glucose cotransporter 2 inhibitors). , In addition, such models may be used to support clinical trial design and cost‐effectiveness studies by simplifying risk stratification. Although several studies have developed prediction models for HHF in people with T2D, no models have yet been incorporated in guideline‐directed care. Therefore, our review aimed to evaluate the performance, applicability, and clinical impact of existing clinical prediction models for HHF in adults with T2D to facilitate the selection of models for clinical implementation.

Methods

The authors declare that all supporting data are available within the article. This review was conducted according to a pre‐specified protocol that was developed with clinical experts and registered in the Open Science Framework (osf.io/na26x). The review adhered to the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses and the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies guidelines. ,

Search Strategy

We searched Embase (Ovid), MEDLINE (Ovid), Web of Science, Google Scholar (first 200 citations), and Tufts’ Clinical Prediction Model Registry from database inception to February 24, 2021. The complete search strategy, detailed in Table S1, was developed and peer‐reviewed with 2 academic librarians following the Peer Review of Electronic Search Strategies guidelines. Briefly, a 3‐concept search strategy using medical subject headings, Emtree terms, and keywords related to T2D, HF, and prediction modeling were applied. Additional citations were included by consulting content experts and reviewing the reference and citation lists of included studies (Web of Science), related reviews, and conference proceedings. No restrictions were placed on the language, date, or status of publications.

Study Selection and Eligibility Criteria

The titles and abstracts of retrieved citations were screened against prespecified inclusion criteria by 2 independent reviewers (A.R., E.O). References marked as potentially eligible proceeded to secondary full‐text assessment. Disagreements were resolved by consensus or when needed, by a third reviewer (A.S.). Eligible studies needed to report the development, validation, or update of a multivariable prediction model for HHF with or without the competing risk of death in patients with T2D (>90% population prevalence). Because our review aimed to identify models apt for clinical use, eligible studies also needed to report measures of model performance (≥1) and sufficient information for clinical use. Eligible performance measures inlcuded but were not limited to assessments of model accuracy, concordance, Brier score, sensitivity, specificity, discrimination, calibration (eg, Hosmer–Lemeshow test), or R2. Sufficient information for clinical use constituted regression coefficients or measures of association (eg, hazard ratios) for quantifiable and non‐arbitrary variables. Studies not reporting original data (ie, reviews) were excluded.

Data Extraction

Two reviewers (A.R., E.O.) independently extracted the characteristics of the studies and cohorts used to develop, validate, or update eligible models. Extracted study and model characteristics included the first author, year of publication, model data source, model derivation and validation methods, number of predictors screened, variables included in the final model, measures of model performance, outcome details (ie, time horizon, definition), and model presentation. Extracted cohort characteristics included patients’ geographic region, age, sex, comorbidities (ie, HF, coronary artery disease), follow‐up period, and number of cumulative events. In instances where multiple models were reported in a single study, data were extracted from the best performing model. Discrepancies between the reviewers’ results were resolved through consensus or a third reviewer (A.S.).

Model Evaluation

Included studies were evaluated with the Prediction Model Risk of Bias Assessment Tool (PROBAST). PROBAST comprises 4 domains designed to identify methodological limitations in model development and validation with respect to selected participants, predictors, outcomes, and analyses. PROBAST also includes domains designed to assess the applicability of developed models with respect to the included participants, predictors, and outcomes. As the purpose of this review was to identify models apt for clinical use, many variables (>7) and the inclusion of continuous variables were separately considered as barriers to routine use after discussions with clinical experts. Based on the PROBAST domains, potential concern with the risk of bias or applicability of primary studies were classified as either high, low, or unclear. Risk of bias assessments were done in duplicate by 2 independent reviewers (A.R., E.O.), and disagreements were resolved by consensus, or a third reviewer as needed (A.S.).

Performance Measures and Model Validation

A clinical prediction models’ performance may be evaluated in internal‐ or external validation through measures of discrimination and calibration. Discrimination reflects a model’s ability to distinguish between patients who do and do not experience an outcome of interest. Calibration conversely reflects a model’s predictive accuracy: ie, the agreement between the predicted probability of events and the actual proportion of events observed. A detailed explanation of internal‐ and external validation and commonly used measures that evaluate these metrics is available in Data S1.

Data Synthesis Statistical Analysis

We constructed evidence tables with details of the identified clinical prediction models and their derivation, validation, or updated (eg, presentation, performance, included variables, risk of bias, and applicability). For the analysis of overall discrimination, meta‐analyses were performed using random‐effect models in instances where multiple c‐statistics (≥3) were available for the same prediction model in internal‐ or external validation. For primary studies where meta‐analysis was not indicated, the results were synthesized qualitatively using a narrative approach. Furthermore, in response to expert review, our protocol was modified to synthesize the modeling studies according to HHF and new‐onset HHF, respectively. Statistical analyses were performed using R version 4.0.3 (R Foundation for Statistical Computing, Vienna, Austria).

Results

The literature search identified 6192 citations (Figure 1). After screening, a total of 18 studies , , , , , , , , , , , , , , , , , published between 2008 and 2021 were included in the review, with data from 999 167 unique patients with T2D. Of these 18 studies, , , , , , , , , , , , , , , , , , 2 were identified by screening reference lists and conference abstracts, respectively. A flow diagram of study selection is shown in Figure 1 and the list of excluded full‐text citations is available in Table S2.

Figure 1

PRISMA flow diagram for study selection.

PRISMA indicates Preferred Reporting Items for Systematic Reviews and Meta‐Analyses.

Study and Model Characteristics

Together 15 multivariable prediction models , , , , , , , , , , , , , , were assessed as 3 studies , , exclusively performed external validation. Most modeling studies used randomized controlled trials (9 of 15) , , , , , , , , for model development, 3 of which were the ACCORD (Action to Control Cardiovascular Risk in Diabetes) trial. , , The median number of participants and events used for model development was 8756 (interquartile range, 5184‒16 013) and 258 (interquartile range, 223‒420), respectively (Table 1). The median number of included variables was 10 (interquartile range, 7‒12) with the most common being age (12 of 15), body mass index (8 of 15), and systolic blood pressure (8 of 15; Figure 2). A minority of models incorporated race and ethnicity (4 of 15) or sex (3 of 15); estimated glycated hemoglobin was more common in models predicting incident HHF (4 of 6) whereas glomerular filtration rate was more common in models predicting incident or recurrent HHF (5 of 9). Tables 2 and 3 summarize the complete details of the included models and their internal and external performance.

Table 1

Characteristics of the Cohorts and Methods Used in Studies Reporting the Development of a Clinical Prediction Model for Heart Failure Hospitalization in Type 2 Diabetes

Reference (Model)	Data source	Follow‐up (y)	No.*	Country (n)	Age (mean), y	HF (%)	CAD (%)	Model derivation	Variables screened	Outcome	No. events
Heart failure hospitalization
Willis 2021 15	CANVAS	3.6	10 142	Multinational (30)	63.3	14.4	56.4	Weibull regression	52	HHF	243
Sharma 2020 16	EXAMINE	1.6	5154	Multinational (49)	61.0	27.9	100	Cox regression	NR	HHF	195
Berg 2019 17 (TRS‐HFDM)	SAVOR‐TIMI 53	2.1	8212	Multinational (26)	65	12.8	62.4	Cox regression	25	HHF	228
Kim 2019 18	EMR	5.0	81 091	United States	60.4	7.0	22.0	Multi‐task learning	45	HHF	NR
Fraty 2018 19	SURDIAGENE	5.3	1438	France	65.0	NR	26.7	Fine and Gray regression	24	HHF	206
Shao 2018 20 (BRAVO)	ACCORD	4.7	10 251	United States and Canada	62.8	4.8	35.2	Weibull regression	28	HHF or HF death	454
Basu 2017 21 (RECODe)	ACCORD	4.7	9635	United States and Canada	62.8	4.8	35.2	Cox regression	33	HHF or HF death	454
Wolsk 2017 22	ELIXA	2.2	5525	Multinational (49)	60.3	22.4	100	Cox regression	45	HHF	221
Kiadaliri 2013 23	EMR	5.0	21 775	Sweden	56.1	NR	NR	Weibull regression	11	HHF	I: 1366 R: 947
Incident heart failure hospitalization
Williams 2020 24	EMR	6.6	54 452	United States	60.0	0.0 †	21.0	Cox regression	80	New‐onset HHF	1884
Segar 2019 25 (WATCH‐DM)	ACCORD	4.9	8756	United States and Canada	62.7	0.0 †	35.2	Random survival forests	147	New‐onset HHF or HF death	319
Halon 2017 26	Cohort study	8.4	735	Israel	63.4	0.0	0.0	Cox regression	39	New‐onset HHF or cardiovascular death	41
Hippisley‐Cox 2015 27 (QDiabetes)	EMR	15.0	437 806	England	60.0	0.0 †	17.4	Cox regression	21	New‐onset HHF	274
Pfister 2011 28	PROactive	2.9	5238	Multinational (19)	61.7	0.0 †	94.7	Cox regression	34	New‐onset HHF or HF death	233
Yang 2008 29	EMR	5.5	3456	China	57	0.0 †	4.4	Cox regression	26	New‐onset HHF	274

ACCORD indicates Action to Control Cardiovascular Risk in Diabetes trial; CAD, coronary artery disease; CANVAS, Canagliflozin Cardiovascular Assessment Study; DECLARE‐TIMI 58, The Dapagliflozin Effect on Cardiovascular Events–Thrombolysis in Myocardial Infarction 58 trial; ELIXA, Evaluation of Lixisenatide in Acute Coronary Syndrome trial; EMR, electronic medical records; EXAMINE, Examination of Cardiovascular Outcomes With Alogliptin Versus Standard of Care trial; HF, heart failure; HHF, hospitalization for heart failure; I, incident HF; NR, not reported; PROactive, Prospective Pioglitazone Clinical Trsial in Macrovascular Events trial; R, recurrent HF; SURDIAGENE, Survival Diabetes and Genetics cohort; and WATCH‐DM, Weight [BMI], Age, Hypertension, Creatinine, HDL‐C, Diabetes Control [fasting plasma glucose], QRS Duration, MI, and CABG) risk score.

Training data set sample size.

Excluded from training data set.

Figure 2

Matrix of risk predictors for heart failure hospitalization in included model development studies.

Afib indicates atrial fibrillation; BMI, body mass index; BUN, blood urea nitrogen; CABG, coronary artery bypass grafting; CAD, coronary artery disease; CBV, cerebrovascular disease; CKD, chronic kidney disease; Cr, Creatinine; CRF, chronic renal failure; DBP, diastolic blood pressure; FPG, free plasma glucose; GAL3, Galectin‐3; GDF‐15, Growth‐Differentiation‐Factor‐15; GFR, Glomerular Filtration Rate; HBA1c, glycated hemoglobin A1c; HDL, high‐density lipoprotein; HHF, hospitalization for heart failure; HF, heart failure; HsTNI, high‐sensitivity troponin; LA/RA, left atrium / right atrium; LDL, low‐density lipoprotein; MR‐pro‐ADM, Mid‐regional pro‐ADM; NT‐proBNP, N‐terminal pro‐B‐type natriuretic peptide; PVD, peripheral vascular disease; and SBP, systolic blood pressure.

Table 2

Characteristics and Internal Performance of Clinical Prediction Models for Heart Failure Hospitalization in Type 2 Diabetes

Reference (Model)	Model presentation	Model variables (n)	Internal validation	Time horizon	Internal Model Performance		Heart failure hospitalization definition †	Risk of bias|applicability ‡
					Calibration	Discrimination*
Heart Failure Hospitalization
Willis 2021 15	Regression coefficients	Age, race and ethnicity, BMI, eGFR, LDL, ulcer history, AF, PVD, micro‐or macroalbuminuria, amputation history (10)	Apparent	7‐y risk	O/E ratio: 0.98	0.83	Hospitalization >24 hours with new or worsening clinical and physical signs of HF with need for therapy.	High|High §
Sharma 2020 16	Regression coefficients	NT‐proBNP for consistency , HF history, hsTNI, GDF15, Hypercholesterolemia, hypertension, MI history, DM duration, GAL3, SBP, age, sex, smoke, adiponectin, eGFR (15)	Apparent	6‐mo risk	NR	0.83	Hospitalization >12 hours with new or worsening clinical, radiological, or physical signs of HF with need for therapy.	High|High
Berg 2019 (TRS‐HFDM) 17	Integer score and online calculator	HF history, AF, CAD, eGFR, uACR (5)	Bootstrapping	4‐y risk	NR	0.81	Hospitalization >12 hours with evidence of new or worsening HF and need for additional or increased therapy.	High|Low
Kim 2019 18	Regression coefficients	GFR, normal GFR, BMI, HR, smoke, age, region, hypertension therapy, Hypercholesterolemia treatment, insulin treatment, CAD, CBV, PVD, CKD, CRF (15)	Bootstrapping	5‐y risk	NR	0.81	Hospital discharge with ICD‐9 codes: 428.0, 428.1, 428.2, 428.3, 428.4, 428.9.	High|High
Fraty 2018 19	Regression coefficients	Age, DM duration, eGFR, uACR, CAD, MR‐proADM, NT‐proBNP (7)	Apparent	NR	NR	0.84	Definition as provided by the ESC guidelines 2012.	High|High
Shao 2018 20 (BRAVO)	Online calculator	HbA1c, SBP, BMI, age, hypoglycemia, education, MI history, HF history, revascularization history (9)	Cross‐validation	NA ¶	Brier score: 0.008	0.79	See definition provided in Segar 2019 above.	High|Low
Basu 2017 21 (RECODe)	Online calculator	Age, sex, race and ethnicity, smoke, SBP, CAD, hypertension therapy, statin treatment, anticoagulant treatment, HbA1c, TC, HDL, Serum creatinine, uACR (14)	Cross‐validation	10‐y risk	Calibration slope/intercept/p: 1.01/−0.0004/0.93	0.75	See definition provided in Segar 2019 above.	Low|Low
Wolsk 2017 22	Regression coefficients	NT‐proBNP, BMI, NSTEMI, HF history, MI history (5)	Apparent	NR	NR	0.77	Hospitalization with new or worsening clinical and physical signs of HF with need for therapy.	High|High
Kiadaliri 2013 23	Regression coefficients	Sex, age, HbA1c, SBP, TC/HDL, BMI, micro/macroalbuminuria, smoke, HF history, DM duration (10)	Split‐sample	5‐y risk	HL test Training: P=0.30 Test: P=0.14	Training: 0.84 Test: 0.84	Fatal or non‐fatal HF hospitalization with ICD‐10 code I50.	High|High §
Incident heart failure hospitalization
Williams 2020 24	Integer score	Age, CAD, BUN, AF, HbA1c, albumin, SBP, CKD, smoke (9)	Apparent	1, 3, 5‐y risk	NR	0.78	EMR documented hospital admission with HF as the primary diagnosis in the absence of prior HF diagnosis.	High|High
Segar 2019 25 (WATCH‐DM)	Online calculator & integer score	Age, BMI, SPB, FPG, QRS, Serum Cr, DBP, HDL‐C, MI history, CABG history for consistency (10)	Split‐sample	5‐y risk	HL test: P=0.29	Training: 0.74 Test: 0.77	Hospitalization with clinical and radiologic evidence of HF; or death due to HF or cardiogenic shock with evidence of HF in the absence of acute ischemic event	High|High
Halon 2017 26	Regression coefficients	LA/RA volume ratio, microvascular disease, SBP (3)	Apparent	NR	NR	0.79	Hospitalization requiring >1 of: typical HF symptoms and findings on examination, dyspnea and radiological evidence, or dyspnea and HF diagnosis requiring IV therapy with furosemide.	High|High
Hippisley‐Cox 2015 27 (QDiabetes)	Online calculator	Age, sex, deprivation, race and ethnicity, smoke, DM type, DM duration, CKD, AF, HbA1c, TC/HDL, SBP, BMI (13)	Split‐sample	10‐y risk	R2: 39.8 ‐ 40.0 ||	0.77 ‐ 0.77 ||	Primary care record codes: G58%, G5yy9, G5yyA, 662f, 662g, 662h, 662i; ICD‐10 codes: I110, I130, I42, I50 for cases of HF from hospital and mortality records.	High|High
Pfister 2011 28	Integer score	Age, total creatinine, diuretic treatment, HbA1c, DM duration, LDL, HR, left BBB, right BBB, MI also history for consistency, microalbuminuria, pioglitazone treatment (12)	Bootstrapping	NR	HL test P=0.62	0.75	Serious HF event requiring hospitalization or prolongation of stay, was fatal or life‐threatening, or resulted in significant disability.	High|High
Yang 2008 29	Regression coefficients	Age, BMI, HBA1c, uACR, hemoglobin, CAD (6)	Split‐sample	5‐y risk	HL test P>0.20	0.85	Hospital discharge with ICD‐9 code 428.X.	High|High

Model validation methods and model performance measures are delineated according to internal‐ and external validation. AF indicates atrial fibrillation; BBB, bundle branch block; BMI, body mass index; BUN, blood urea nitrogen; CABG, coronary artery bypass grafting; CAD, coronary artery disease; CBV, cerebrovascular disease; CKD, chronic kidney disease; CRF, chronic renal failure; DBP, diastolic blood pressure; EMR, electronic medical records; FPG, free plasma glucose; GAL3, Galectin‐3; GDF‐15, growth‐differentiation factor‐15; GFR, glomerular filtration rate; HBA1c, glycated hemoglobin A1c; HDL, high‐density lipoprotein; HF, heart failure; HL, Hosmer‐Lemeshow; HsTNI, high‐sensitivity troponin; ICD‐9, International Classification of Diseases, Ninth Revision; ICD‐10, International Classification of Diseases, Tenth Revision; LA/RA, left atrium/right atrium; LDL, low‐density lipoprotein; MI, myocardial infarction; MR‐pro‐ADM, Mid‐regional pro‐ADM; NSTEMI, non–ST‐segment–elevation myocardial infarction; NT‐proBNP, N‐terminal pro‐B‐type natriuretic peptide; O/E ratio, predicted mean event rate/observed mean event rate; PVD, peripheral vascular disease; SBP, systolic blood pressure; TC, total cholesterol; uACR, urine albumin to creatinine ratio; and WATCH‐DM, (Weight [BMI], Age, hyperTension, Creatinine, HDL‐C, Diabetes control [fasting plasma glucose], QRS Duration, MI, and CABG) risk score.

Discrimination is measured as c‐statistics unless specified otherwise.

The definition for heart failure hospitalization derived from the training set used for model development.

High and low ratings relate to the degree of concern associated with the risk of bias or applicability. These domains were assessed with the Prediction Model Risk of Bias Assessment Tool.

Downgraded to high concern with applicability because the study only reported regression coefficients.

Range across men and women, respectively.

Time horizon is not applicable as the model is a discrete‐time patient‐level microsimulation.

Table 3

External Validity of Identified Clinical Prediction Model for Heart Failure Hospitalization in Type 2 Diabetes

Reference	Data Source	No.	Countries (n, sites)	Age, y*	HF (%)	CAD (%)	External validation	Event time horizon	External model performance		Heart failure hospitalization definition ‡	Risk of bias|applicability §
									Calibration	Discrimination †
TRS‐HFDM
Berg 2020 17	DECLARE‐TIMI 58	8578	Multinational (882)	63.9	10.0	40.6	Temporal	4‐y risk	Nam D’Agostino: P=0.20	0.78	Hospitalization >24 hours with new or worsening clinical and physical signs of HF with need for therapy.	High|Low
Elharram 2020 31	ACCORD	5123	United States and Canada (77)	62.7	4.9	35.2	Independent	7‐y risk	Nam D'Agostino: P=0.13	0.78	See ACCORD’s definition in Table 2.	High|Low
Razaghizad 2021 30	EXAMINE	5380	Multinational (898)	61.0	27.9	100	Independent	6‐ and 30‐mo risk	Calibration slope/intercept/p:0.81/−0.17/0.33; 0.77/−0.18 /0.06	0.75	See EXAMINE’s definition in Table 2.	High|High
RECODe
Basu 2017 21	Look AHEAD	4760	United States (16)	57.5	NR	14.0	Independent	10‐y risk	Calibration slope/intercept/p: 1.13/−0.011/0.07	0.76	Hospitalization with clinical and radiologic evidence of HF with need for therapy or ventricular dysfunction.	Low|Low
Basu 2018 32	MESA	1555	United States (6)	63.0	0.0 ‖	NR	Independent	10‐y risk	Calibration slope/intercept/p: 1.01/0.005/0.42	0.80	ICD‐9 code 428 discharge or underlying cause of death I50; radiologic evidence of HF; or autopsy finding of pulmonary edema or HF.	Low|Low
Basu 2018 32	JHS	1746	United States (3)	57.5	NR	NR	Independent	10‐y risk	Calibration slope/intercept/p: 0.72/0.091/0.07	0.73	Hospitalization with clinical and radiologic evidence of HF with need for therapy, and dilated ventricle or poor heart function assessed by echocardiography or ventriculography.
QDiabetes
Hippisley‐Cox 2015 27	CPRD	197 905	United Kingdom (254)	61.0	0.0 ‖	19.1	Geographic	10‐y risk	R2: 41.1 – 38.7 ¶	0.77 – 0.78 ¶	See Hippisley‐Cox’s definition in Table 2.	High|Low
WATCH‐DM
Segar 2019 25	ALLHAT	10 819	United States and Canada (623)	67.0	0.0 ‖	NR	Temporal	5‐y risk	HL test: P=0.20	0.74	HHF was primarily based on clinic investigator reports.	High|Low
BRAVO
Shao 2018 20	ASPEN	2410	Multinational (70)	61.1	0.0 ‖	NR	Geographic	NR	NR	NR	NR	High|High
Shao 2018 20	ADVANCE	11 140	Multinational (215)	66.3	NR	NR	Geographic	NR	NR	NR	Death attributable to HF, HHF, or worsening New York Heart Association class.
Shao 2018 20	CARDS	2838	United Kingdom (132)	61.7	NR	NR	Geographic	NR	NR	NR	NR

ACCORD indicates Action to Control Cardiovascular Risk in Diabetes trial; ADVANCE, the Action in Diabetes and Vascular Disease: Preterax and Diamicron Modified Release Controlled Evaluation trial; ALLHAT, the Antihypertensive and Lipid‐Lowering Treatment to Prevent Heart Attack trial; ASPEN, Atorvastatin Study for Prevention of Coronary Heart Disease Endpoints in Non‐Insulin‐Dependent Diabetes Mellitus trial; CARDS, the Collaborative Atorvastatin Diabetes Study trial; CPRD, Clinical Research Practice Datalink database; DECLARE‐TIMI 58, The Dapagliflozin Effect on Cardiovascular Events–Thrombolysis in Myocardial Infarction 58 trial; EXAMINE, Examination of Cardiovascular Outcomes With Alogliptin Versus Standard of Care trial; HF, heart failure; HHF, hospitalization for heart failure; HL, Hosmer‐Lemeshow; JHS, Jackson Heart Study; Look AHEAD, Action for Health in Diabetes trial; MESA, The Multi‐Ethnic Study of Atherosclerosis trial; NR, not reported; and WATCH‐DM, (Weight [BMI], Age, Hypertension, Creatinine, HDL‐C, Diabetes Control [fasting plasma glucose], QRS Duration, MI, and CABG) risk score.

Mean or median as available in the publication.

Discrimination is measured as c‐statistics unless specified otherwise.

The definition for heart failure hospitalization derived from the data set used for model validation.

High and low ratings relate to the degree of concern associated with the risk of bias or applicability. These domains were assessed with the Prediction Model Risk of Bias Assessment Tool.

Patients with prevalent heart failure excluded.

Range across men and women, respectively.

(%)

Berg 2019

Shao 2018

Basu 2017

Segar 2019

Models With Clinical Utility and External Validation

Of the 15 multivariable models, , , , , , , , , , , , , , , 7 were presented in a useful manner (eg, risk score, online calculator). , , , , , , Of these, 5 had their performance evaluated in external validation (Table 3). , , , , These were the Thrombolysis in Myocardial Infarction (TIMI) Risk Score for HF in Diabetes (TRS‐HFDM) ; the BRAVO (Building, Relating, Assessing, and Validating Outcomes) risk engine ; the Risk Equations for Complications of Type 2 Diabetes (RECODe) ; the Weight, Age, Hypertension, Creatinine, HDL‐C, Diabetes Control, QRS Duration, MI (myocardial infarction), and CABG (WATCH‐DM) risk score ; and QDiabetes. Two models strictly predicted incident HHF, , whereas the other 3 predicted incident or recurrent HHF in patients with and without prevalent HF (Table 1). , ,

TRS‐HFDM Risk Score

The TRS‐HFDM was developed in SAVOR‐TIMI 53 (Saxagliptin Assessment of Vascular Outcomes Recorded in Patients with Diabetes Mellitus‐TIMI 53) trial (Table 1) and externally validated in 3 trials (N=19 081; Table 3). , , The development cohort had a mean age of 65.0 years and a median follow‐up time of 2.1 years; the prevalence of HF and coronary artery disease was 12.8% and 62.4%, respectively. The TRS‐HFDM is an integer‐based risk score between 0 to 7 that uses 5 clinical variables (Table 2) and predicts the 4‐year risk of incident or recurrent HHF. Proteomic biomarkers were purposely excluded to prioritize parsimony and ease‐of‐use. Good discriminative performance was observed in internal validation (c‐statistic: 0.81; Table 2). In addition, external validation studies showed moderate‐to‐good discrimination and calibration for 0.5‐, 2.5‐, 4‐ and 7‐year event predictions (Table 3). , The overall c‐statistic was 0.78 (95% CI, 0.76‒0.80; Figure 3). However, 1 external validation study demonstrated the model modestly overestimated the risk of events for individuals with diabetes and acute coronary syndrome who are classified at severe risk.

Figure 3

Meta‐analysis of externally validated clinical prediction models’ discrimination.

ACCORD indicates Action to Control Cardiovascular Risk in Diabetes trial; CPRD, Clinical Research Practice Datalink database; DECLARE‐TIMI 58, The Dapagliflozin Effect on Cardiovascular Events–Thrombolysis in Myocardial Infarction 58 trial; EXAMINE, Examination of Cardiovascular Outcomes With Alogliptin versus Standard of Care trial; JHS, Jackson Heart Study; Look AHEAD, Action for Health in Diabetes trial; MESA, The Multi‐Ethnic Study of Atherosclerosis trial; RECODe, Risk Equations for Complications of Type 2 Diabetes; SAVOR‐TIMI 53, Saxagliptin Assessment of Vascular Outcomes Recorded in Patients with Diabetes Mellitus–Thrombolysis in Myocardial Infarction 53 trial; and TRS‐HFDM, Thrombolysis in Myocardial Infarction (TIMI) Risk Score for Heart Failure in Diabetes.

WATCH‐DM Risk Score

The Weight, Age, Hypertension, Creatinine, HDL‐C, Diabetes Control, QRS Duration, MI, and CABG risk score was developed in the ACCORD trial (Table 1) and externally validated in 1 trial (N= 10 819; Table 3). The development cohort had a mean age of 62.7 and a median follow‐up time of 4.9 years; the prevalence of HF and coronary artery disease was 0% and 35.2%, respectively. The model predicts the 5‐year risk of incident HHF and is available as an online calculator as well as an integer‐based risk score that can range between 0 to 34. The model used machine learning for development, and it included 7 multinomous clinical variables (Table 2). In external validation, the online calculator had a c‐statistic of 0.74 and Hosmer‐Lemeshow χ2 of 11.1 (P=0.20) while the risk score had a c‐statistic of 0.70 and Hosmer‐Lemeshow χ2 of 10.0 (P=0.29). Markedly, the risk score showed better performance for incident HF with reduced ejection fraction versus preserved ejection fraction in external validation (c‐statistic, 0.72 versus 0.64, respectively, P<0.001).

Building, Relating, Assessing, and Validating Outcomes Risk Engine

The Building, Relating, Assessing, and Validating Outcomes risk engine was developed in ACCORD and externally validated in 3 trials (N = 16 388; Table 3). The development cohort had a mean age of 62.8 and a median follow‐up time of 4.7 years; the prevalence of HF and coronary artery disease was 4.8% and 35.2%, respectively. Building, Relating, Assessing, and Validating Outcomes predicts the annual risk of incident or recurrent HHF alongside stroke, MI, angina, revascularization, severe pressure loss, end‐stage renal disease, blindness, all‐cause mortality, and cardiovascular death. The model is accessible as an online calculator that requires 18 variables, 9 of which are for HHF (Table 2). Internal validation demonstrated good performance: for HHF the Brier score was 0.008 and the c‐statistic was 0.79 (95% CI, 0.77 to 0.81). In external validation, the risk engine demonstrated good calibration across the 28 end points (calibration slope: 1.07; R2: 0.86), however, HHF specific calibration was not reported.

RECODe Risk Equations

RECODe was developed in ACCORD and validated in 3 cohorts (N = 8 061; Table 3). , The development cohort had a mean age of 62.8 and a median follow‐up time of 4.7 years; the prevalence of HF and coronary artery disease was 4.8% and 35.2%, respectively. The risk equation is available as an online calculator that predicts incident or recurrent HHF alongside the 10‐year risk of nephropathy, retinopathy, neuropathy, myocardial infarction or stroke, and all‐cause mortality. The online calculator requires 16 clinical variables, 14 of which are for HHF (Table 2). RECODe showed moderate‐to‐good discrimination and calibration in internal and external validation. In ACCORD, the c‐statistic was 0.75 and the calibration slope was 1.01 (P=0.93 [insignificant p‐values indicate acceptable calibration]). In external validation, the c‐statistic ranged between 0.73 to 0.80 and the calibration slopes ranged between 0.72 to 1.13 for the 10‐year estimated risk of HHF (Table 3). The overall c‐statistic for RECODe was 0.76 (95% CI, 0.73‒0.79; Figure 3).

QDiabetes Risk Calculator

QDiabetes was developed using primary care data from 437 806 people and then validated in 2 cohorts (N = 334 933, Table 3). The development cohort had a mean age of 60.0 and a follow‐up time of 15 years; the prevalence of HF and coronary artery disease was 0% and 17.4%, respectively. The risk calculator is available as a sex‐specific online calculator that predicts the 10‐year risk of incident HHF. The risk calculator uses 12 variables (Table 2). The model demonstrated moderate performance in internal and external validation. In internal validation the c‐statistic was 0.76 and 0.77 in men and women, respectively, whilst in external validation the c‐statistic was 0.77 and 0.78. Calibration, which was assessed visually with calibration plots, was satisfactory as there was good agreement between the predicted and observed risks in each validation cohort. The overall c‐statistic for the QDiabetes Risk Calculator was 0.77 (95% CI, 0.76‒0.78; Figure 3).

Models With Incomplete Evaluation

The 10 remaining clinical prediction models either never underwent external validation (9 of 10) , , , , , , , , or were externally validated but not presented in a clinically practical form (1 of 10; Table 1). Therefore, these models' clinical utility was judged as less applicable for the review. As an exception, 2 models were presented as integer‐based risk scores that could be relevant in certain settings. , The first model by William and colleagues used routine care data to predict the 1‐, 3‐ and 5‐year risk of HHF. The model used 9 variables (Table 2) and had moderate discrimination in patients without prior HF (c‐statistic: 0.78). The second model by Pfister and colleagues used data from PROactive (Prospective Pioglitazone Clinical Trial in Macrovascular Events) to develop an 11 variable risk score (Table 2). Internal model performance with bootstrap re‐sampling was moderate (c‐statistic: 0.75): however, there were some concerns with applicability (eg, inclusion of variables with limited availability).

Risk of Bias and Applicability

The most prevalent sources of potential bias in model development included use of unblinded outcome adjudication (12 of 15), , , , , , , , , , , , handling of missing data (11 of 15),* management of competing risk (8 of 15), , , , , , , , and consideration of model overfitting (8 of 15). , , , , , , , Applicability concerns were model predictor accessibility (6 of 15) , , , , , and generalizability of the derivation cohorts (6 of 15). , , , , , In validation, sources of bias included unblinded outcome adjudication (8 of 9) , , , , , , , and evaluation of model performance (4 of 9). , , , All but one model (RECODe) by Basu and colleagues had potential for risk of bias during development, and all but 3 models , , had potential applicability concerns (Figures S1 through S4).

Discussion

This review was designed to evaluate existing clinical prediction models for HHF in adults with T2D to facilitate clinical model selection. Only studies reporting performance measures and sufficient information for model use were included to highlight tools with the most clinical utility. Altogether 15 models were identified, of which only 5 were externally validated and presented in a clinically practical form. Most identified models (n=15) included >7 variables (75%) or only reported regression coefficients (53%), limiting clinical applicability. Moreover, most identified clinical prediction models had potential concerns with risk of bias (93%) or applicability (67%), highlighting the need for improved methods in modeling studies.

Prieto‐Merino and Pocock have proposed that 3 features of risk models should be favored: (1) relative simplicity with reasonably easy‐to‐obtain variables; (2) clinical relevance in the context of the disease state; and (3) overall generalizability to other settings. Models with these characteristics should be favored compared with complex mathematical models. Based on these criteria and our appraisal of the included models’ risk of bias, the RECODe risk equations and the TRS‐HFDM demonstrated the most clinical potential. Both models were easy‐to‐use (eg, as risk scores and online calculators) and were externally validated (ie, generalized) in 3 large cardiovascular safety trials. Although RECODe was the only model with low potential for risk of bias, it requires the input of 16 variables, which restricts its potential for routine use. Nevertheless, RECODe provides risk estimates for several diabetes‐specific complications, making it potentially apt for comprehensive metabolic, cardiovascular, and renal risk prediction. The TRS‐HFDM, in contrast, was one of the simplest model for HHF as it was an integer‐based risk score that only required 5 common variables. Berg and colleagues, who developed the model, demonstrated it could identify a 20‐fold risk gradient and patients who derive greatest absolute benefit from sodium‐glucose co‐transporter 2 inhibition. , Despite these strengths, the TRS‐HFDM has yet to be externally validated in a low‐risk population‐based cohort. Future research may seek to simplify the RECODe risk equations or externally validate the TRS‐HFDM in a low‐risk non‐clinical trial population.

Defining HF Hospitalization

At present, it is unclear if research should focus on developing new models for incident HHF or validating and using models that already predict new‐onset HHF alongside recurrent events (eg, RECODe, TRS‐HFDM). On one hand, future prediction modeling studies may need to focus on incident HHF as the therapeutics options (eg, quadruple therapy, sodium‐glucose co‐transporter 2 inhibitors) for individuals with diabetes who have prevalent HF are already well established. , , , The prediction of new‐onset HHF, in addition, may be particularly important as the use of sodium‐glucose co‐transporter 2 inhibitors to prevent the development HF has traditionally been overlooked compared with the prevention of traditional major adverse cardiovascular events. Therefore, the prediction of new‐onset HF could offer more avenues for clinician‐patient discussions to facilitate the implementation of therapies that prevent HF development. On the other hand, predicting new‐onset or recurrent HHF provides generalizable models that are applicable for a larger segment of individuals with diabetes who are at risk of HF. Ultimately, both avenues present a good path to increase the implementation of guideline‐directed medical therapy. However, at present there is no consensus on this issue.

Overcoming Barriers to Adoption

Despite the proliferation of prediction models for HHF in individuals with T2D, few models have been successfully incorporated into routine care. A previous systematic review of mortality models for patients with HF highlighted deficiencies that have limited the adoption of clinical prediction models. The review, by Alba and colleagues, prompted a former FDA commissioner to critique the fact that most models have insufficient validation and the fact that those with the greatest levels of validation are derived from populations with limited generalizability (eg, clinical trials).

Our review adds to this literature as it shows that after almost 10 years the same problems have been left unresolved. Furthermore, we have identified that few models are applicable for point‐of‐care use because of either the large number of variables required, or the sole reporting of regression coefficients. As a result, in the absence of electronic medical records that can integrate complex algorithms for decision support, we encourage investigators to develop models that can readily be implemented at the point‐of‐care.

Our data demonstrated that 80% of clinical predictions models for HHF in T2D included at least one measure of renal function (eg, albuminuria, estimated glomerular filtration rate, serum creatinine, blood urea nitrogen), underscoring its importance in HHF prediction. However, our data also demonstrated that cardiac biomarkers (eg, natriuretic peptides, troponins) may currently be underutilized. As a result, future modeling studies may consider further leveraging cardiac biomarkers to improve predictive performance and thus the impetus for applying predictive models in practice. However, it may be important to develop biomarker‐ and non‐biomarkers versions to facilitate adoption in resource‐limited settings.

Methodological and Reporting Issues

As part of the systematic review, a detailed critical appraisal of the risk of bias and applicability of the included studies was performed. This was done to identify potential methodological issues in the conduct of included studies. Our data demonstrated that most studies developing models for HHF did not or were not able to blind outcome adjudication to candidate baseline variables as most studies used data already collected from clinical trials. This was not a major concern in a few studies (ie, those that accounted for model overfitting). However, models which included prior HF as a risk predictor were at high risk of biased predictive accuracy as knowledge of predictors can influence outcome determination. , , We acknowledge blinding adjudication to patients’ baseline clinical information would likely present forbidding practical challenges. Therefore, in response, we recommend investigators planning to use data from established clinical trials to include statistical methods to address model overfitting.

In the included studies, information relevant to the handling of missing data were also often excluded. Few studies implemented statistical imputation for dealing with missing data and even fewer reported the characteristics of patients with missing data. This can severely affect model validity as patients who are lost to follow‐up can differ significantly from the target population. , Similarly, few studies considered competing risks (eg, death) in model development, which can lead to informative censoring and risk overestimation. , As the complications of HF and diabetes can disproportionality affect older patients, it is important to use techniques (eg, Fine and Gray regression, cause specific hazard models) to account for competing risk and maximize model performance. To ensure model quality, investigators should reference reporting standards (eg, the Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) and risk of bias tools (eg, PROBAST) during model development. ,

Strengths and Limitations

This review adhered to numerous best practices for systematic reviews. To ensure transparent reporting and analysis, the protocol was publicly registered in Open Science Framework. A highly sensitive search strategy was also developed to provide a comprehensive overview of existing models. The search included several resources including bibliographic databases, conference proceedings, content experts, and a clinical prediction model registry. Likewise, no restrictions were placed on the date, status, or language of publications. All aspects of the review were also done in duplicate including the data extraction, which followed Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies guidance, and the risk of bias and applicability assessment, which was done with PROBAST. Consequently, to our knowledge, this systematic review constitutes an evidence map of the highest available evidence on clinical prediction models for HHF in adults with T2D to date. The review may thus facilitate evidence‐based decisions at the point of care, which may have been hampered in the past by a lack of clarity in the literature.

Despite the strengths, there were limitations with the review. First, most models that we included were judged to have potential concerns with risk of bias or applicability in their development, stemming largely from their methods of analysis and predictor selection. However, this fact was one of the most important findings of the review as it motivated our proposed guidance for future model development. In addition, it is important to highlight that potential for bias does not mean that the included models were significantly flawed. Likewise, nor does it mean that studies with fewer potential concerns are more valid than studies with multiple potential flaws.

Second, relevant studies that did not mention or did not include index terms for T2D may have been missed. Third, as noticed in this review, the clinical utility or net benefit of the identified models was not evaluated in any of the studies. Therefore, the effect of these models on real patient outcomes remains unclear and warrants evaluation in external validation and impact studies. Finally, because only 1 study was judged at low risk of bias in development, no association between predicted outcomes and methodological quality could be inferred.

Future Directions

Clinical prediction models may improve health outcomes and resource usage only to the degree that they affect individual patients’ or health care providers’ behaviors. As mentioned, no studies evaluated the effect of an eligible model on behavioral change or patient events. As a result, cluster randomized controlled trials evaluating the impact of the identified models may be warranted. However, because clinical trials can be expensive, other study designs may be leveraged to conduct initial feasibility assessments. For instance, studies assessing health care providers’ judgments before‐and‐after being presented a model could be a cost‐effective study design for such means. Likewise, decision curve analysis, a method for evaluating prediction models, may be used to evaluate potential net benefit (eg, number of unnecessary treatments avoided) because of attributable to risk prediction. , Although a detailed explanation of the methodology is outside the scope of this review, positive findings compared with standard‐of‐care could inform the implementation of risk tools into guideline‐directed T2D management.

Lastly, 2 studies by Segar and colleagues developed machine learning models according to sex, race and ethnicity, and HF subtype. , Their data showed risk prediction models may yield significantly different results between these groups. Therefore, future model development and validation studies should aim to validate model performance according to sex, race and ethnicity, and HF subtype as the risk factors and approaches to mitigate risk may differ. As sex was under‐represented in the prediction models included in this review, it will be particularly important for future studies to underscore the role of sex as T2D confers an excess of risk for HF in women than in men.

Conclusions

Evaluating cardiovascular risk is critical for guiding the selection of preventive therapies in patients with T2D. While there has been a proliferation of cardiovascular prediction models for HHF in these patients, there is a lack of external validation studies to ensure their performance and generalizability. Moreover, most models for HHF have potential concerns with risk of bias and applicability. In terms of the best available models, the TRS‐HFDM was identified as particularly apt for routine point‐of‐care use, while RECODe may be helpful in instances where holistic cardiovascular risk assessment is required. Nevertheless, the actual effect of even the best models remains unclear because of an absence of clinical impact studies. As a result, studies evaluating model‐based judgments in comparison with existing clinical practice may be warranted to evaluate clinical prediction model utility in real‐world practice before implementation into guideline‐directed medical care.

Sources of Funding

None.

Disclosures

Mr Razaghizad has received a Graduate Excellence Award in Medicine and a Ethelwyn and Ernie Nyman Fellowship from McGill University. Dr Greene has received research support from the Duke University Department of Medicine Chair’s Research Award, American Heart Association, National Heart Lung and Blood Institute, Amgen, AztraZeneca, Bristol‐Myers Squibb, Cytokinetics, Merck, Novartis, and Pfizer; has served on advisory boards for Amgen, AstraZeneca, and Cytokinetics; and has served as a consultant for Amgen, Bayer, Bristol‐Myers Squibb, Merck, and Vifor. Dr Fudim was supported by the National Heart, Lung, and Blood Institute (K23HL151744), the American Heart Association (20IPA35310955), Mario Family Award, Duke Chair’s Award, Translating Duke Health Award, Bayer, Bodyport, and BTG Specialty Pharmaceuticals. He receives consulting fees from AxonTherapies, Bodyport, Boston Scientific, CVRx, Daxor, Edwards LifeSciences, Fire1, NXT Biomedical, Viscardia, Zoll. Dr Brophy reports receiving support from the Fonds de Recherche Santé Quebec. Dr Sharma reports receiving support from the Canadian Institutes of Health Research (175095), Fonds de Recherche Santé Quebec, Alberta Innovates Health Solution, European Society of Cardiology young investigator grant, Roche Diagnostics, Boeringer‐Ingelheim, Novartis, and Takeda. Dr Fudim reports consulting fees from AxonTherapies and Daxor. Dr Tsoukas reports no disclosures in relation to this work but has received speaker honoraria from NovoNordisk, Eli Lilly, Boehringer‐Ingelheim, and AstraZeneca. Dr Mavrakanas has received honoraria from Daiichi Sankyo, BMS Canada, Janssen, and Pfizer and has served on advisory boards for Boehringer Ingelheim outside the submitted work. He is supported from the Fonds de Recherche Santé Quebec (FRSQ) Junior 1 clinician scholar program and from the McGill University Department of Medicine salary support program. Dr Felker has received research grants from NHLBI, American Heart Association, Amgen, Bayer, BMS, Merck, Cytokinetics, and CSL‐Behring; he has acted as a consultant to Novartis, Amgen, BMS, Cytokinetics, Medtronic, Cardionomic, Boehringer‐Ingelheim, American Regent, Abbott, Astra‐Zeneca, Reprieve, Myovant, Sequana, Windtree Therapuetics, and Whiteswell, and has served on clinical endpoint committees/data safety monitoring boards for Amgen, Merck, Medtronic, EBR Systems, V‐Wave, LivaNova, Siemens, and Rocket Pharma. Dr Peters has received support from the Fonds de Recherche Santé Quebec. Dr Ezekowitz reports research grants or consulting work outside this work from American Regent, Amgen, AstraZeneca, Bayer, Boehringer‐Ingelheim, Bristol‐Myers Squibb/Pfizer, eko.ai, US2.ai, Merck, Novartis, Otsuka, Sanofi, and Servier. The remaining authors have no disclosures to report.

Data S1

Tables S1–S2

Figures S1–S4

References 56, 57, 58, 59, 60, 61

Click here for additional data file.