A statistical predictive model consistent within a 5-year follow-up period for patients with acute heart failure.

BACKGROUND: Acute heart failure (AHF) is a major and rapidly growing health problem responsible for millions of hospitalizations annually. Due to a high proportion of in-hospital mortality and postdischarge rehospitalization and mortality, a prompt strategy for risk stratification and subsequently tailored therapy is desirable to help improve clinical outcomes. The AHEAD (A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus) and AHEAD-U (A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus, U: uric acid) are popular prognostic scoring systems. However, only a specific follow-up period is considered in these systems, and whether their predictive capability is still accurate in a significantly shorter or longer follow-up period is not known.
METHODS: In this research, we adapted extensive statistical approaches based on the Cox model to explore consistent risk factors in various follow-up durations. Results showed that six factors, namely, hemoglobin level, age, sodium level, blood urea nitrogen level, atrial fibrillation, and high-density lipoprotein level could be used to establish a new prognostic model, which was referred to as HANBAH. For a simple clinical application, the HANBAH scoring system, with scores from 0 to 6, was developed using several statistical models.
RESULTS: Based on an evaluation using the conventional statistical approaches, such as the Akaike information criterion, concordance statistic, and Cox area under the curve, the HANBAH scoring system consistently outperformed other strategies in predicting short- and long-term mortality. Notably, an independent replication study also revealed similar results. In addition, a modern machine learning technique using the support vector machine confirmed its superior performance.
CONCLUSION: The use of the HANBAH scoring system, which is a clinically friendly tool, was proposed, and its efficacy in predicting the mortality rates of patients with AHF regardless of the follow-up duration was independently validated.

1. INTRODUCTION

Acute heart failure (AHF) is a leading cause of hospitalization in developed centuries,[1] and it is associated with an in-hospital mortality rate of 4%–6.7%.[2-4] In addition, the readmission rate within 6 months is >50%,[5] and the mortality rate within 1 year after discharge ranges from 20% to 30%.[6,7] Because AHF has a poor prognosis, a rapid risk classification, followed by tailored treatments, is required to improve clinical outcomes.[8]

The efficacy of the Acute Decompensated Heart Failure National Registry (ADHERE) scoring system[8] for risk stratifications in patients with AHF has been validated. However, the practicability of the ADHERE scoring system is limited because of complex calculations, which jeopardize its clinical application. Based on the Acute Heart Failure Database Network registry, which comprises 5846 inpatients with AHF, the Cox proportional hazard model[9] with forward stepwise selection and time-dependent receiver operating characteristic (ROC) curve came up with the simple AHEAD (A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus) scoring system.[6] Then, Chen et al.[10] assessed the Heart Failure Registry of Taipei Veterans General Hospital and identified 2143 patients with AHF. The Cox regression analysis revealed that a uric acid level >8.6 mg/dL could significantly improve risk stratification. Hence, the use of the AHEAD-U (A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus, U: uric acid) scoring system was proposed.[10] Previous studies focused on a fixed time frame, and the predictive capability of this system would be lower in different follow-up durations. Thus, this study aimed to develop a time-invariant statistical model with a consistent superior predictive capability for patients with AHF regardless of the follow-up durations.

To evaluate for statistical performances, the Akaike information criterion (AIC), concordance statistic (C-statistic), and COX area under the curve (AUC) were used for comparison. In 1974, Akaike established the AIC: AIC = –2log(likelihood) + 2d, where d is the number of parameter.[11] The C-statistic is a popular reference for the evaluations of discrimination and calibration using the logistic regression models.[12,13] Moreover, its score ranges from 0.5 to 1, and a higher value indicates a better model. However, in the analysis of survival time, it may not be suitable. Therefore, a time-dependent ROC curve analysis was performed, and the COX AUC was used to assess discrimination ability.[14-19]

2. METHODS

Data were collected from the Heart Failure Registry of Taipei Veterans General Hospital (HARVEST registry). We enrolled patients hospitalized due to AHF, defined as new-onset or gradually or rapidly worsening heart failure symptoms and abnormal vital signs requiring urgent therapy. A total of 1024 patients with AHF were included.[20] Data about descriptive statistics and other details are available in a previous publication. The COX proportional hazard model[9] was fitted with 30 parameters in five different follow-up durations (90 days, 1 year, 2 years, 3 years, and 5 years). The performance of various methods was compared using AIC to evaluate model fitness, Harrell’s C to consider time-dependent features and censored data, and the time-dependent Cox ROC and AUC to assess accuracy at different time points. All-cause mortality is a nonreversible outcome; therefore, subjects expired at an earlier follow-up time are also considered as mortality cases in the later time frames. Similarly, censoring status was assessed at each time point.

First, in addition to the variables in the AHEAD or AHEAD-U scoring systems, some variables, such as high-density lipoprotein (HDL) and blood urea nitrogen (BUN) levels, remained significant. Therefore, we hypothesized that the two scoring systems might not be capable to identify all significant predictors, and different statistical approaches were used to identify each potential risk factor. After removing the AHEAD and AHEAD-U scoring systems from the stepwise selection, different factors were identified at each time point using the Cox models. According to the frequency of significant inclusions, atrial fibrillation (AF) (five times), BUN (five times), age (four times), hemoglobin level (four times), Natrium (NA) level (four times), and HDL level (three times) were included in the new model. The new approach, which was based on six factors, was referred to as HANBAH. Left ventricular ejection fraction (three times) was excluded because it is generally used to identify the subtypes of cardiovascular disorders.

The six factors of the HANBAH and AHEAD-U scoring systems were compared. Then, age, AF, and hemoglobin level were included in the AHEAD-U scoring system. The other factors (such as creatinine level, diabetes mellitus [DM], and uric acid level) were substituted with BUN, NA, and HDL levels.

Instead of the HANBAH scoring system with the original six factors, a simple scoring system that can be used by clinicians in predicting either short- or long-term mortality is preferred. Equal weights were assigned to each of the six variables since this approach was adopted in both the AHEAD and AHEAD-U scoring systems. To achieve this goal, we assessed the ROC curve based on the Cox proportional hazard model and the distribution of mortality rates from 90 days up to 5 years to identify the time invariant optimal cutoffs for BUN, NA, and HDL levels. For the three factors included in both the HANBAH and AHEAD-U scoring systems, the previous guideline was followed. Thus, age was dichotomized as elderly (>70 years), the definition of AF was not changed, and hemoglobin score was defined as hemoglobin level <130 g/L for men and <120 g/L for women.

BUN was dichotomized by the joint evaluations of the five ROC curves at 90-day, 1-year, 2-year, 3-year, and 5-year follow-up for men and women. We identified the sex-specific cutoffs that can consistently provide a sensitivity >50% and specificity >50% at all follow-up periods. Hence, male patients with a BUN level >26 mg/dL was provided a score of 1; otherwise, a score of 0 was assigned. Meanwhile, female patients with a BUN level >28 mg/dL was provided a score of 1; otherwise, a score of 0 was assigned.

Unlike BUN level, which has a linear effect on mortality, the distributions of mortality rates according to the cumulative percentage of HDL had a nonlinear trend in both sexes. Hence, HDL level was dichotomized based on the distributions of mortality rate at various follow-up periods. The time invariant cutoff for men was 24 mg/dL, which represents approximately 7.5 percentile of all samples. Similarly, the cutoff for women was 25 mg/dL. Since the difference was not discernible, we dichotomized HDL level according to a fixed value of 25 mg/dL for a simplified clinical application. Hence, patients with an HDL level <25 mg/dL were provided with a score of 1; otherwise, a score of 0 was assigned regardless of gender.

A similar approach was used in the analysis of NA levels since a sex-specific distribution revealed that the most effective cutoff is 20 percentile, which corresponds to an NA value of 135 mg/dL. The summary of the HANBAH scores is presented in Fig. 1.

Fig. 1

Cutoff points for the HANBAH score. Each variable is scored 1 if the condition is satisfied. AF = atrial fibrillation; BUN = blood urea nitrogen; HDLC = high-density lipoprotein cholesterol; NA = Natrium.

The AHEAD, AHEAD-U, and HANBAH scoring systems were evaluated using the support vector machine (SVM)[21] which is a powerful and popular machine learning tool used to assess predictive capability. Since this technique is not involved with the generation of the HANBAH scoring system, the results were more accurate and provided unbiased conclusions. SVM is a supervised learning model for two-group classification problems, which conceptually implement the following procedure: the input vectors are non-linearly mapped to an extremely high dimension feature space. In the feature space, a linear decision surface is established. Based on the training data, each sample is marked as belonging to one of the two categories, and an SVM training algorithm builds a model that assigns new examples to one category or the other such that SVM is a nonprobabilistic binary linear classifier.

3. RESULTS

Data about descriptive statistics are available in our previous study (Tables 1–5)[10] since we used the same study population. According to the complex longitudinal assessment of the HANBAH scoring system, the ordinal total score was developed, and its performance was evaluated using various statistical tools, such as AIC, C statistic, and Cox AUC. In Tables 1–3, the differences between the original variables in the HANBAH scoring system and the modified HANBAH scoring system were not significant, indicating that a simple scoring system is a great alternative.

Table 1

Akaike information criterion at different follow-up periods

	90 d	1 y	2 y	3 y	5 y
AHEAD	1235.548	2885.857	4391.341	5217.445	6177.062
AHEAD-U	1231.855	2882.272	4382.747	5212.784	6170.641
HANBAH (original data)	1195.829	2857.213	4348.015	5175.882	6130.588
HANBAH score	1208.179	2841.517	4341.354	5172.639	6138.543
Ratio: AHEAD-U/AHEAD	0.9970	0.9988	0.9980	0.9991	0.9990
Ratio: HANBAH score/AHEAD	0.9778	0.9846	0.9886	0.9914	0.9938

AHEAD = A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus; AHEAD-U = A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus U: uric acid; HANBAH = hemoglobin level, age, sodium level, blood urea nitrogen level, atrial fibrillation, and high-density lipoprotein.

Table 5

Replication study: concordance statistic at different follow-up periods

	90 d	1 y	2 y	3 y	5 y
AHEAD	0.6301	0.6180	0.6187	0.6210	0.6214
AHEAD-U	0.6660	0.6429	0.6382	0.6368	0.6347
HANBAH (continuous data)	0.6746	0.6440	0.6424	0.6384	0.6388
HANBAH score	0.6615	0.6445	0.6412	0.6382	0.6380
Ratio: AHEAD-U/AHEAD	1.0570	1.0403	1.0315	1.0254	1.0214
Ratio: HANBAH score/AHEAD	1.0498	1.0429	1.0364	1.0277	1.0267

AHEAD = A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus; AHEAD-U = A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus U: uric acid; HANBAH = hemoglobin level, age, sodium level, blood urea nitrogen level, atrial fibrillation, and high-density lipoprotein.

Table 3

Cox area under the curve at different follow-up periods

	90 d	1 y	2 y	3 y	5 y
AHEAD	0.6607	0.6216	0.6208	0.6445	0.6485
AHEAD-U	0.6740	0.6253	0.6385	0.6543	0.6627
HANBAH (original data)	0.7170	0.6446	0.6582	0.6857	0.6870
HANBAH score	0.6805	0.6719	0.6739	0.6897	0.6841
Ratio: AHEAD-U/AHEAD	1.0201	1.0060	1.0285	1.0152	1.0219
Ratio: HANBAH score/AHEAD	1.0300	1.0809	1.0855	1.0701	1.0549

AHEAD = A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus; AHEAD-U = A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus U: uric acid; HANBAH = hemoglobin level, age, sodium level, blood urea nitrogen level, atrial fibrillation, and high-density lipoprotein.

Regardless of the follow-up periods, Table 1 shows that the AIC values of the AHEAD and AHEAD-U scoring systems were consistently larger than those of the HANBAH scoring system. Therefore, the HANBAH scoring system had a better model fit since a smaller AIC indicates a better fitted result. A HANBAH/AHEAD score ratio <1 indicates that the HANBAH scoring system had a lower AIC score than the AHEAD scoring system.

In Table 2, a larger C-statistics indicated a better performance. In predicting either short- or long-term mortality, the HANBAH scoring system was consistently superior to the AHEAD and AHEAD-U scoring systems. A HANBAH/AHEAD score ratio >1 indicated the quantity of improvement using the HANBAH scoring system. The most significant improvement was observed at a 1-year follow-up period when the HANBAH score was 8% higher than that of the AHEAD score.

Table 2

Concordance statistic at different follow-up periods

	90 d	1 y	2 y	3 y	5 y
AHEAD	0.6614	0.6180	0.6139	0.6191	0.6167
AHEAD-U	0.6717	0.6237	0.6230	0.6255	0.6233
HANBAH (original data)	0.7477	0.6646	0.6603	0.6613	0.6577
HANBAH score	0.6995	0.6662	0.6545	0.6533	0.6481
Ratio: AHEAD-U/AHEAD	1.0156	1.0092	1.0148	1.0103	1.0107
Ratio: HANBAH score/AHEAD	1.0576	1.0780	1.0661	1.0552	1.0509

AHEAD = A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus; AHEAD-U = A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus U: uric acid; HANBAH = hemoglobin level, age, sodium level, blood urea nitrogen level, atrial fibrillation, and high-density lipoprotein.

In Table 3, a larger Cox AUC indicated a better predictive capability. Moreover, the HANBAH score consistently had a better pattern, and the HANBAH/AHEAD score ratio was > 1. In Tables 1–3, the AHEAD-U scoring system outperformed the AHEAD scoring system, and this result was in accordance with that of previous studies.

A replication study further confirmed the superior predictive capability of the new HANBAH scoring system. An independent sample of patients with AHF (n = 827) was randomly selected from the hospital database and included in a replication study. Notably, the HANBAH scoring system used a subset of 646 participants due to missing data introduced by the different sets of predictors if the sample size for the AHEAD and AHEAD-U scoring system is 827.

In Table 4, the AIC values decreased in the order of AHEAD, AHEAD-U, and then HANBAH. Although the sample size was small, HANBAH still had the best performance. The HANBAH/AHEAD score ratio was significantly lower than the AHEAD-U/AHEAD score ratio. This result indicated that the HANBAH scoring system had the best model fit.

Table 4

Replication study: Akaike information criterion at different follow-up periods

	90 d	1 y	2 y	3 y	5 y
AHEAD	747.635	2236.613	2822.399	3453.127	3907.320
AHEAD-U	738.404	2221.862	2810.007	3442.224	3902.099
HANBAH (original data)	745.435	2235.536	2821.086	3454.243	3906.578
HANBAH score	739.997	2220.942	2807.224	3442.215	3894.785
Ratio: AHEAD-U/AHEAD	0.9877	0.9934	0.9956	0.9968	0.9987
Ratio: HANBAH score/AHEAD	0.9898	0.9930	0.9946	0.9968	0.9968

AHEAD = A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus; AHEAD-U = A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus U: uric acid; HANBAH = hemoglobin level, age, sodium level, blood urea nitrogen level, atrial fibrillation, and high-density lipoprotein.

In Table 5, the C-statistics showed that the HANBAH score had the best performance, and a ratio >1 indicated a better improvement. In Table 6, the Cox AUC revealed similar patterns. That is, the HANBAH score was consistently higher.

Table 6

Replication study: Cox area under the curve at different follow-up periods

	90 d	1 y	2 y	3 y	5 y
AHEAD	0.6395	0.6350	0.6421	0.6633	0.6868
AHEAD-U	0.6784	0.6655	0.6639	0.6711	0.6757
HANBAH (original data)	0.6501	0.6597	0.6604	0.6664	0.6925
HANBAH score	0.6755	0.6663	0.6669	0.6759	0.6966
Ratio: AHEAD-U/AHEAD	1.0608	1.0480	1.0340	1.0118	0.9838
Ratio: HANBAH score/AHEAD	1.0563	1.0493	1.0386	1.0190	1.0143

AHEAD = A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus; AHEAD-U = A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus U: uric acid; HANBAH = hemoglobin level, age, sodium level, blood urea nitrogen level, atrial fibrillation, and high-density lipoprotein.

Finally, the SVM was an independent tool that can be used to further evaluate performance in the replication study. The SVM model adopted the Radial basis (RBF) kernel function (γ = 0.5, C = 10), with 70% training set and 30% testing set. The SVM model tuning parameter was chosen based on a 10-fold cross-validation that minimized the mean squared error. The SVM revealed that the HANBAH score consistently outperformed the other two methods, and it had the highest accuracy rates (Fig. 2). The superiority was more evident in terms of sensitivity rates (Fig. 3).

Fig. 2

Accuracy of the support vector machine model based on the three methods. AHEAD = A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus; AHEAD-U: A: atrial fibrillation; H: hemoglobin; E: elderly; A: abnormal renal parameters; D: diabetes mellitus; U: uric acid; HANBAH: hemoglobin level, age, sodium level, blood urea nitrogen level, atrial fibrillation, and high-density lipoprotein.

Fig. 3

Sensitivity of the support vector machine model based on the three methods.

4. DISCUSSION

In this research, we used the AHEAD and AHEAD-U scoring systems in shorter (90-days) and longer (5-year) follow-up periods. Moreover, new and important factors that could be great surrogates for the original ones and could have a better predictive capability were identified.

In addition to the statistical rationale that led to the new scoring system, we discussed the clinical background of the new factors. First, the BUN level is correlated to the CREA score, which is a stable indicator of kidney function and is used to evaluate kidney damage because BUN is metabolized by protein and is excreted by the kidney through the urine. BUN is also an indicator of kidney function. In clinical practice, a BUN/creatinine ratio of 10 is used as an estimate of fluid status. When the ratio is <10, azotemia is usually introduced by kidney-related factors. By contrast, if the ratio is >10, the malfunction is not caused by the kidneys.[22,23]

Second, NA replaces the role of DM in the model. According to an epidemiologic survey, older and obese patients with diabetes are more likely to be hypertensive. Thus, blood pressure control is as important as sugar intake control for patients with DM. However, an excess intake of NA will result in elevated blood pressure level and hardening of blood vessels.[24,25] A recent study showed that in addition to hyponatremia, a worsening sodium level is associated with long-term outcomes in patients hospitalized due to AHF.[26]

Finally, HDL level, which refers to good cholesterol and is associated with heart diseases, is a novel factor added to the AHEAD-U scoring system.[27] HDL is considered a scavenger that cruises the bloodstream and removes harmful cholesterol from where it does not belong. Hence, a higher HDL level is associated with a lower risk of heart diseases.

Thus, HANBAH, which is a new scoring tool, underwent clinical and statistical assessments. The modification of the HANBAH factors also underwent a deliberate process using several statistical techniques at various time frames. Such process has not yet been implemented in the previous work.

After an evaluation using the AIC, C-statistic, and Cox AUC, the HANBAH scoring system outperformed the AHEAD and AHEAD-U scoring systems both in short- and long-term follow-ups. In addition to the model generation, a replication study of an independent sample further confirmed the improved predictive capability of the HANBAH scoring system. Notably, the SVM, which is a modern machine learning tool, also confirmed that the HANBAH scoring system had a better predictive capability. Thus, the new strategy could be a reliable prediction tool in clinical applications and could contribute significantly in the field.

In conclusion, the HANBAH score, which is a clinically friendly tool, was developed for patients with AHF, and another independent sample was used to validate its performance using conventional statistical tools and modern machine learning models (SVM). Results consistently showed that the HANBAH score can better predict mortality rates regardless of the follow-up duration.

This research was conducted on an Asian population. Hence, a replication study of Caucasians or other ethnicities should be conducted in the future to promote the use of this new scoring tool. The modification of the HANBAH scoring system is dependent on separate dichotomizations. The scoring system can be improved by implementing the joint distribution of six factors instead of the stepwise selection that can evaluate predictors individually. However, further research should be performed to validate this notion.

ACKNOWLEDGMENTS

This study received grants from the Ministry of Health and Welfare (MOHW104-TDU-B-211-113-003, MOHW106-TDU-B-211-113001) and an intramural grant from National Yang-Ming University (E107F-M01-0501). This work was supported by Yin Yen-Liang Foundation Development and Construction Plan of the School of Medicine, National Yang-Ming University.