Performance of the MAGGIC heart failure risk score and its modification with the addition of discharge natriuretic peptides.

AIMS: Predictive models for heart failure patients are widely used in the clinical practice to stratify patients' mortality and enable clinicians to tailor and intensify their approach. However, such models have not been validated internationally. In addition, biomarkers are now frequently measured to obtain prognostic information, and the implications of this practice are not known. In this study, we aimed to validate the model performance of the Meta-analysis Global Group in Chronic Heart Failure (MAGGIC) score in a Japanese acute heart failure registry and further explore the incremental prognostic value of discharge B-type natriuretic peptide (BNP) level. METHODS AND
RESULTS: In this study, we evaluated the registered data of 2215 consecutive acute HF patients (with 694 119 person-years follow-up) from a prospective multicentre registry (the West Tokyo Heart Failure) conducted in Japan from April 2006 to August 2016. The mean age was 73.0 ± 13.0, and 61.2% were male. The MAGGIC score demonstrated modest discrimination (c-index = 0.71, 95% confidence interval 0.67-0.74) and good calibration (R2 value = 0.97); there was constant overestimation for 1 year mortality. However, when the BNP level was added to the original MAGGIC variables, the model demonstrated good discrimination (c-index = 0.74, 95% confidence interval 0.70-0.78) with adequate calibration (R2 value = 0.91). The modified MAGGIC BNP score was externally validated in a separate Japanese registry (NaDEF) and demonstrated moderate discrimination (c-index = 0.69, 95% confidence interval 0.65-0.73) and calibration (R2 value = 0.85).
CONCLUSION: The original MAGGIC score performed modestly in Japanese patients, but the addition of discharge BNP level enhanced model performance. The addition of objective biomarkers may result in effective modification of preexisting internationally recognized risk models and aid in multinational comparisons of heart failure patients' outcomes.

Introduction

Heart failure (HF) is a growing epidemic worldwide particularly in countries with a rapidly ageing population.1 In Japan, with over 10.5 million octogenarians at present, the number of HF patients is expected to grow proportionately over the coming decades. To best identify patients at risk and concentrate limited resources, high‐performing internationally validated risk prediction models are necessary.2

Predicting 1 year mortality in HF patients has become extremely important in order to stratify patients that require heart transplantation and ventricular assist devices. Peak oxygen consumption (VO2) values measured during cardiopulmonary exercise testing are considered as an important tool for patient selection in advanced HF patients,3 and its use has been recommended in various guidelines. However, in reality, a significant proportion of these patients are too frail to undergo rigorous exercise testing. In this regard, risk prediction models such as the Seattle Heart Failure Model4 and the Meta‐analysis Global Group in Chronic Heart Failure (MAGGIC) score are particularly useful because their calculation are easy to use in the outpatient clinics or bedsides. The MAGGIC score was developed to predict survival of HF patients after hospitalization; from 39 372 patients with HF, from 30 cohort studies, six of which were clinical trials.5 To date, MAGGIC score has been validated externally in Western countries displaying good overall performance;6, 7 however, its performance in other regions of the world remains largely unknown.

Additionally, biomarkers such as B‐type natriuretic peptide (BNP) have become widely used in the management of HF patients. BNP measured at the timing of discharge has been proven to be prognostically important in recent studies.8, 9, 10 In the current study, we aimed to validate the performance of the MAGGIC score and further explore the incremental prognostic value of discharge BNP level in Japanese acute HF patients.

Methods

Study population

In this study, two registries were used as derivation and validation cohorts for evaluating the performance of the prediction models. The derivation cohort was the West Tokyo Heart Failure (WET‐HF) registry, and the validation cohort was the National Cerebral and Cardiovascular center acute DEcompensated heart Failure study (NaDEF). In both registries, acute HF was defined as rapid‐onset HF or a change in the signs and symptoms of HF requiring urgent therapy and hospitalization, based on the Framingham criteria.11 HF patients presenting with coexisting acute coronary syndrome were not included because of exclusion criterion. Individual cardiologists at each institution made the clinical diagnosis of HF. The institutional review boards at each site approved the study protocol, and research was conducted in accordance with the Declaration of Helsinki. Written informed consent was obtained from each subject before the study.

The derivation cohort was the WET‐HF registry: a large, ongoing, prospective, multicenter cohort registry conducted in Tokyo, Japan (East Japan), from June 2005 to August 2016. Details of the registry have been described previously.12, 13, 14, 15 Dedicated clinical research coordinators collected clinical data by reviewing individual patient medical records. Exclusive on‐site auditing by the investigators (Y.S. and S.K.) ensured proper registration of each patient. The objectives and detailed design are provided on the University Hospital Medical Information Network (UMIN000001171).

The validation cohort for our study was the NaDEF study: an ongoing, prospective, single‐centre acute HF registry conducted at the National Cerebral and Cardiovascular Center in Osaka, Japan (West Japan), that began from January 2013. The objectives and detailed design are provided on the University Hospital Medical Information Network (UMIN000017024).16, 17

Patient selection and calculation of the MAGGIC score

Patients were excluded for missing data required for calculation of the MAGGIC score (Figure ). The 1 year mortality rate was calculated using the MAGGIC score including 13 clinical variables: age, ejection fraction as categorical variable (≥40%, 35–39%, 30–34%, 25–29%, <20%), New York Heart Association (NYHA) functional class, serum creatinine, diabetes, non‐prescription of beta‐blocker, systolic blood pressure (SBP), body mass index, time since diagnosis, current smoker, chronic obstructive pulmonary disease, male gender, and non‐prescription of angiotensin‐converting enzyme inhibitor or angiotensin receptor blockers for the remaining patients. Discharge NYHA class status was imputed as NYHA class II for 187 (8.4%) of the patients with missing data. Data on time since first diagnosis of HF or HF duration were not collected in our current database; thus, patients with previous history of HF hospitalization were regarded as having HF duration of more than 18 months. The study population was divided into six risk categories (Groups 1–6) based on the MAGGIC score according to the original article.5

Figure 1

Study population.

Statistical analysis

The patient demographics, the predicted cardiovascular death risk according to the MAGGIC score, and the observed mortality rate were compared between those who died during the first year of follow‐up and those who did not by using the Student's t‐tests or the Wilcoxon rank‐sum test for continuous variables and χ2 test for categorical variables.

Time‐to‐event data were calculated as the time in days from the date of discharge after initial hospitalization for acute HF to the date of all‐cause death. Cumulative survival was computed according to the Kaplan–Meier method. We fit a Cox regression model with the calculated MAGGIC score as the only independent variable, and we calculated the c‐index, which is regarded as a measure for model discrimination equivalent to the area under curve for binary dependent variables. A c‐index equal or above 0.6 was considered modest, a c‐index above 0.7 was considered reasonable, and a c‐index above 0.80 was considered strong. Receiver operating characteristic curves were computed by using the MAGGIC score as the reference variable and the 1 year observed mortality as the binary reference variable, and the area under the receiver operating characteristic curve were calculated. To measure model calibration, we compared the predicted mortality rate to the observed mortality rate on the calibration plot and evaluated the coefficient of determination (R 2 value). Further, we compared the regression line drawn within the plotted points to the line of unity and calculated the best fit linear regression formula according to the observed mortality rate.

The modified MAGGIC score was created by adding the log‐transformed discharge BNP value as an ordinal covariate and incorporated along with the MAGGIC score as independent variables and 1 year mortality rate as the dependent variable. We evaluated model discrimination and calibration similarly as stated previously. Model improvement was assessed by the Akaike information criterion (AIC) and Bayesian information criterion (BIC). Furthermore, the modified MAGGIC score was further evaluated in an independent Japanese HF cohort for external validation of the newly created risk prediction model. Model discrimination and calibration were evaluated similarly as stated previously. All statistical analyses were performed by Stata/IC version 13.1 for Macintosh (StataCorp, College Station, TX).

Results

Patient demographics

Overall, 2215 consecutive acute HF patients (with 694 119 person‐years follow‐up) in the WET‐HF registry were evaluated in this study. In brief, the mean age was 73.1 ± 13.5, 62.3% (n = 1136) were male, mean body mass index was 23.3 ± 4.4, 43.9% had a left ventricular ejection fraction below 40%, mean SBP upon admission was 139 ± 34 mmHg, creatinine level was 127 ± 26 μmol/L, 32.3% (n = 572) had a previous history of HF admissions, 25.5% (n = 459) had an ischaemic aetiology for HF, and the median discharge BNP level was 248 μg/dL (Q1–3, 129–528 μg/dL) (Table 1 and 2). The study patients were divided into those who were alive or dead after 1 year of follow‐up. Patients who died (n = 178) were older, leaner, had a lower SBP, haemogloblin level, and sodium level, higher creatinine, proportion of cardiac resynchronization therapy insertion, HF with ischaemic aetiology, and discharge BNP with longer length of stay during the initial hospitalization.

Table 1

Baseline characteristics of patients who died and survived 1 year after discharge

			Total			Dead			Alive
			n = 2215			n = 241			n = 1974
		n	Mean or %	SD	n	Mean or %	SD	n	Mean or %	SD	P value
Age (years)		2215	73.0	13.0		79	11		73	14	<0.001
Male, %		2215	61.2%		147	61.0%		1209	61.2%		0.94
BMI (kg/m2)		2067	23.5	4.4	222	22.0	4.0	1845	23.6	4.4	<0.001
EF, %		2215	44.7	15.5	241	41.5	4.0	1974	45	15	<0.001
EF <40%		2215	40%		116	48%		767	39%		0.005
Previous heart failure admission		2215			101	41.9%		575	29.1%		<0.001
Atrial fibrillation		2086			102	42.3%		885	44.8%		0.87
Medical history, %
Diabetes, %		2215	34.7%		92	38.2%		677	34.3%		0.23
Hypertension, %		2210	67.7%		154	63.9%		1,342	68.0%		0.22
Stroke, %		2208	12.5%		36	14.9%		240	12.2%		0.23
Current smoker, %		2215	43.0%		102	42.3%		850	43.1%		0.83
COPD, %		2215	5.6%		24	10.0%		99	5.0%		0.002
Length of stay, days		2212	14	(10–22)	241	17	(11–28)	14	(10–22)		<0.001
Systolic BP (mmHg)		2209	112	18	239	109	19	1970	112	18	0.02
NYHA class, %		2028
I		188	9%		4	1.7%		184	9.3%
II		1312	65%		108	44.8%		1204	61.0%
III		497	25%		85	35.3%		412	20.9%
IV		31	2%		9	3.7%		22	1.1%		<0.001
Haemoglobin (g/L)		2212	12.2	2.2	241	11.0	1.9	1,971	12.3	2.2	<0.001
Creatinine (mg/dL)		2215	1.3	1.4	241	1.7	1.6	1974	1.0	1.3	<0.001
BUN (mg/dL)		2148	26.9	15.4	237	35.7	20	1911	25.7	14	<0.001
Sodium (mmol/L)		2211	138.5	4.5	240	137.7	4	1971	138.6	5	<0.001
BNP (pg/mL)		1400	139	(137–141)	152	532	(242–1157)	1248	246	(123–503)	<0.001
Medications, %
ACE inhibitor, %		2215	29.3%	650	67	27.8%		583	29.5%		0.58
ARB, %		2215	37.9%	840	65	27.0%		775	39.3%		<0.001
Beta‐blocker, %		2215	77.4%	1715	175	72.6%		1540	78.0%		0.058
Aldosterone antagonist, %		2207	36.0%	795	82	34.0%		713	36.1%		0.49
Loop diuretics, %		2215	77.2%	1711	203	84.2%		1508	76.4%		0.004
Digoxin, %		2215	10.9%	241	12	5.0%		170	8.6%		0.053
MAGGIC score		2215	25	(21–29)	241	29	(25–33)	1974	25	(20–28)	<0.001

ACE, angiotensin‐converting enzyme; ARB, angiotensin receptor blockers; BMI, body mass index; BP, blood pressure; BUN, blood urea nitrogen; COPD, chronic obstructive pulmonary disease; EF, ejection fraction; MAGGIC, Meta‐analysis Global Group in Chronic Heart Failure; NYHA, New York Heart Association; SD, standard deviation.

Table 2

Extent of missing data

		n	%
Age (years)		0	0.0
Male, %		0	0.0
BMI (kg/m2)		148	6.7
EF, %		0	0.0
EF < 40%		0	0.0
Previous heart failure admission		0	0.0
Atrial fibrillation		129	5.8
Medical history, %
Diabetes, %		0	0.0
Hypertension, %		5	0.2
Stroke, %		7	0.3
Current smoker, %		0	0.0
COPD, %		0	0.0
Length of stay, days		3	0.14
Systolic BP (mmHg)		6	0.3
NYHA class, %		187	8.4
I
II
III
IV
Haemoglobin (g/L)		3	0.1
Creatinine (mg/dL)		0	0.0
BUN (mg/dL)		67	3.0
Sodium (mmol/L)		4	0.2
BNP (pg/mL)		815	36.8
Medications, %
ACE inhibitor, %		0	0.0
ARB, %		0	0.0
Beta‐blocker, %		0	0.0
Aldosterone antagonist, %		8	0.4
Loop diuretics, %		0	0.0
Digoxin, %		0	0.0
MAGGIC score		0	0.0

ACE, angiotensin‐converting enzyme; ARB, angiotensin receptor blockers; BMI, body mass index; BP, blood pressure; BUN, blood urea nitrogen; COPD, chronic obstructive pulmonary disease; EF, ejection fraction; MAGGIC, Meta‐analysis Global Group in Chronic Heart Failure; NYHA, New York Heart Association.

The MAGGIC score was available for 2215 patients with a median score of 25 points with the first (Q1) and third (Q3) interquartile ranging from 21 to 29 points (Figure ). The overall predicted median 1 year mortality rate was 16.0%. Patients who survived (n = 1974) had a median score of 25 (20–28) points (predicted mortality rate, 19.1%), and those who died (n = 241) had a median score of 29 (25–33) points (predicted mortality rate, 22.7%), respectively. A Kaplan–Meier mortality curve has been drawn for the six risk categories divided by the MAGGIC score displaying a distinct difference in 1 year mortality (log‐rank P < 0.001: Figure ).

Figure 2

Distribution of the MAGGIC score within the WET‐HF registry.

Figure 3

Kaplan–Meier curve of all‐cause mortality divided by MAGGIC risk groups.

Within the WET‐HF study population, the MAGGIC score demonstrated modest discrimination (c‐index = 0.71, 95% confidence interval 0.67–0.74) and good calibration (R 2 value = 0.96) (Figure A,B) with a constant overestimation throughout all six risk groups for 1 year mortality (Supporting Information, ). The regression formula drawn linearly on the calibration plot was as follows:

Figure 4

Performance of the original and modified MAGGIC score within the WET‐HF registry.

When discharge BNP level was added to the MAGGIC score, the modified model demonstrated improved discrimination (c‐index = 0.74, 95% confidence interval 0.70–0.78) and good calibration (R 2 value = 0.91). The AIC and BIC for the original MAGGIC score (AIC = 1413.818, BIC = 1425.224) improved after modification (AIC = 862.454, BIC = 878.1867). The regression formula drawn linearly on the calibration plot was as follows (Figure C,D):

Finally, the modified MAGGIC score was externally validated in an independent HF registry cohort (NaDEF registry). The modified model demonstrated modest discrimination (c‐index = 0.69; 95% confidence interval, 0.65–0.73) and good calibration (R 2 value = 0.85) (Figure ). Similarly, the AIC and BIC for the original MAGGIC score (AIC = 369.0387, BIC = 377.7717) improved after modification (AIC = 344.7423, BIC = 353.4195) within the NaDEF registry, as well.

Figure 5

Performance of the original and modified MAGGIC score within the NaDEF registry.

Discussion

To our knowledge, this is the first reporting of an extensive external validation of the MAGGIC score outside the European states. In this Japanese HF registry, the MAGGIC score demonstrated modest discrimination and calibration with a constant overestimation upon risk prediction of 1 year mortality. To our knowledge, this is the first reporting of an extensive external validation of the MAGGIC score outside the European states. Additionally, when discharge BNP was added to the MAGGIC score, the modified risk model demonstrated improved discrimination without lowering calibration that was further confirmed by external validation in an independent HF registry cohort. These results demonstrate that the original MAGGIC score can modestly predict 1 year mortality, but the addition of BNP greatly enhances its performance, thereby re‐emphasizing the importance of the prognostic impact given by discharge BNP levels in Japanese HF patients. These findings suggest that the addition of objective biomarkers may result in effective modification of preexisting internationally recognized risk prediction models and multinational comparisons of HF patients' outcomes.

Previously, we reported the performance of two HF risk models that are widely used in the Western HF community: the Seattle Heart Failure Model4 and the Get With The Guideline–Heart Failure risk score18 within the Japanese HF population. While the Get With The Guideline–Heart Failure risk score was designed to give estimates on short‐term inhospital mortality, the Seattle Heart Failure Model predicts long‐term post‐discharge 1 and 2 year mortality similar to the MAGGIC score. Notably, the Seattle Heart Failure Model demonstrated modest performance with a c‐index of 0.666 and 0.721, respectively, in a subset of 492 Japanese HF patients.19 Although the Seattle Heart Failure Model demonstrated modest performance, it does have its own disadvantages. The calculation of the Seattle Heart Failure Model necessities 20 unique variables such as the use of allopurinol or lymphocyte percentage that are not commonly assessed in modern‐day care. Because the MAGGIC score requires only clinically routinely collected variables and also gives predictions on 1 and 3 year mortality, it has its advantages over the Seattle Heart Failure Model, thus requires rigorous validation in other ethnicities from around the globe.

Several distinct differences between the MAGGIC study population and the WET‐HF registry have been notified. Within the 31 studies utilized to create the MAGGIC study, six studies mostly conducted during the 1990s (DIAMOND,20 DIG,21 CHARM,22 and ECHOS23 trials and IN‐CHF24 and HOLA registries25) contributed to over three quarters of the entire meta‐analysis population and were larger or equal to the population size of our current study population. Despite showing a similar distribution in the baseline MAGGIC scores, the observed mean mortality rate was lower than the predicted mortality rate in our database. A similar trend was observed in the external validation European cohort as well.6 Two main reasons could have affected this trend: (i) a disproportionately high score appointed to one or more variables used to calculate the MAGGIC score or (ii) an important prognostic factor not incorporated in the MAGGIC score. An example for the first reason is ejection fraction, and examples for the second reason are renal function (e.g. blood urea nitrogen and/or glomerular filtration rate), serum sodium level, exercise capacity, and frailty.26, 27 Other unmeasured covariates such as the intensity or quality of care after the initial HF hospitalization and patient adherence to essential medications may affect 1 year outcome. Of note, within the MAGGIC meta‐analysis, there was one Japanese HF registry conducted by Tsutsui et al. that consisted 172 HF patients recruited during the year 1997.28 There were notable differences in the baseline demographics such as higher age, higher prevalence of atrial fibrillation, and smokers commonly seen in our current registry; however, the 1 year mortality rate did not differ significantly indicating a better prognosis in our database reflecting modern‐day HF patients.

Natriuretic peptide‐guided treatment of HF has been known to reduce all‐cause mortality in relatively young HF patients and overall reduce HF hospitalizations.29 Although the emergence of other biomarkers such as soluble ST2, growth differentiation factor 15, cystatin‐C, galectin‐3, and high sensitivity C‐reactive protein30 has slightly shifted physicians away from measuring BNP or N terminal proBNP in recent years, the prognostic value of serum natriuretic peptides remains robust, which has been shown in previous studies.31, 32, 33 In this study, we observed a significant additive value of discharge BNP and an overall improvement of the MAGGIC score with this modification. This result points to the importance of BNP measurement upon risk prediction as in fact mentioned in the discussion section by Pocock et al. within the original MAGGIC article.5 Nonetheless, discharge BNP levels should be interpreted with caution because the length of stay is significantly longer in Japan as compared with Western countries. Discharge BNP in Japanese HF patients is likely to reflect patient status after achieving full haemodynamic compensation. Thus, high discharge BNP is likely to indicate a more hazardous state despite rigorous medical attempts to achieve stable status under the Japanese healthcare system. Although the issue of costs associated with BNP measurements remains an issue, its prognostic importance is robust upon predicting post‐discharge mortality beyond racial or ethnic differences.34 With the widespread use of angiotensin receptor neprilysin inhibitor, the use of N terminal‐proBNP over traditional BNP may be more valid in recent days. Nevertheless, our results suggest that future HF risk models ought to incorporate natriuretic peptides as a covariate to meet high performance in various ethnicities.

Limitations

There are several limitations to our current study. Although more than 2000 patients were evaluated in the current study, this population may not be representative of the entire Japanese population because the registry was conducted within the metropolitan area or its residing districts of Tokyo. Nevertheless, the model performed modestly in a single‐centre NaDEF registry consisted of patients mostly living in Osaka, the second largest city situated in the western part of Japan. Validation of the model may be necessary in other regions or larger sized registries because variation in patient characteristic, treatment, and practice pattern may exist. Second, we inevitably encountered substantial missing data especially for discharge BNP, and the model performance was only assessed for those with complete data without statistical imputation. This may have resulted in overestimation/underestimation of the modified MAGGIC score, although there were no statistical significant differences in the average MAGGIC score between those with and without discharge BNP. Third, we were able to assess model performance for 1 year mortality but not for 3 year mortality because our database was designed to follow up patients up to 2 years.

Conclusions

The performance of the MAGGIC HF risk score is reasonable in the Japanese HF population over a single year time frame. The addition of discharge BNP significantly enhances model discrimination and calibration that was further confirmed in an external validation cohort. Future HF risk models ought to incorporate BNP as a covariate to meet high performance in various ethnicities.

Conflict of interest

S.K. received lecture fees from Pfizer Japan Inc. and also received an unrestricted research grant for the Department of Cardiology, Keio University School of Medicine, from Bayer Pharmaceutical Co., Ltd. The authors report no other conflict of interest.

Author contributions

M.S., Y.S., and S.K. conceived and designed the research and drafted the manuscript; M.S., Y.S., and S.K. analysed and interpreted the data; M.S. performed the statistical analysis; S.K., T.N., T.A., K.F., and T.Y. handled funding and supervision; A.G., A.M., Y.S., Y.N., T.K., and T.Y. made critical revisions of the manuscript for important intellectual content.

Funding

This study was supported by the Grant‐in‐Aid for Scientific Research (C) (23591062 and 26461088), the Health Labour Sciences Research Grant (14528506), and the Sakakibara Clinical Research Grant for Promotion of Sciences, 2012, 2013, 2014.

Table S1. Baseline characteristics of patients with and without discharge BNP value.

Figure S1. Calibration of the MAGGIC score.

Click here for additional data file.