Comparison of the performance of the CRUSADE, ACUITY-HORIZONS, and ACTION bleeding scores in ACS patients undergoing PCI: insights from a cohort of 4939 patients in China.

BACKGROUND: The CRUSADE, ACTION and ACUITY-HORIZONS scores are commonly used for predicting in-hospital major bleeding events in patients with acute coronary syndrome (ACS), but the homogeneous nature of these models' population limits simple extrapolation to other local population. We aimed to compare the performance of the three risk models in Chinese patients.
METHODS: We evaluated the performance of the three predicting scores for predicting in-hospital major bleeding events defined by thrombolysis in myocardial infarction (TIMI) serious (major and minor) episodes, in a cohort of Chinese ACS patients with either non-ST-elevation ACS (NSTE-ACS) or ST-elevation myocardial infarction (STEMI). Calibration and discrimination of the three risk models were evaluated by the Hosmer-Lemeshow test and C-statistic, respectively. We compared the predictive accuracy of the risk scores by the Delong non-parametric test.
RESULTS: TIMI serious bleeding rate was 1.1% overall (1.9% and 0.86% for STEMI and NSTE-ACS, respectively). The CRUSADE, ACTION and ACUTIY-HORIZONS scores showed an adequate discriminatory capacity for major bleeding: in overall patients, the C-statistic was 0.80, 0.77, and 0.70, respectively; in NSTE-ACS patients, the C-statistic was 0.73, 0.72, and 0.64, respectively; in STEMI patients, the C-statistic was 0.91, 0.92, and 0.75, respectively. The C-statistic for the ACUITY-HORIZONS model was significantly lower than those of the CRUSADE and ACTION scores for the prediction of TIMI serious bleeding in overall patients (compared with CRUSADE, z = 3.83, P = 0.02; compared with ACTION, z = 3.51, P = 0.03); in NSTE-ACS patients (compared with CRUSADE, z = 2.37, P = 0.01; compared with ACTION, z = 2.11, P = 0.04), and in STEMI patients (compared with CRUSADE, z = 2.6.77, P = 0.02; compared with ACTION, z = 7.91, P = 0.002). No differences were observed when the CRUSADE and ACTION models were compared to each other, regardless of overall patients (z = 0.68, P = 0.31) and both of ACS types (NSTE-ACS, z = 0.52, P = 0.60), and STEMI patients (z = 0.36, P = 0.74). However, the three risk scores all overestimated the absolute major bleeding risk in each risk stratification in our study. For example, the predicted rate of CRUSADE score at high risk stratification was 11.9% vs. an actual rate of 5.3%.
CONCLUSIONS: The CRUSADE and ACTION scores had a greater calibration and discrimination for in-hospital major bleeding compared with the ACUITY-HORIZONS score in Chinese patients with ACS undergoing PCI. However, they all overestimated the bleeding risk rate for Chinese populations. Calibration of these risk scores would be useful for the generalization in Chinese populations.

Introduction

Bleeding is most common non-cardiac complication among patients with acute coronary syndrome (ACS). In- hospital major bleeding is associated with short- and long-term death, myocardial infarction, stroke, blood transfusion, as well as increased length of hospitalization and cost.[1],[2] Given its clinical importance, bleeding risk scores have become a key part of management of patients presenting with ACS.

Currently, three bleeding risk scores have been developed and validated to assess the bleeding risk in ACS patients: the Can Rapid Risk Stratification of Unstable Angina Patients Suppress Adverse Outcome with Early Implementation (CRUSADE of the American College of Cardiology/ American Heart Association Guidelines) bleeding score,[3] the Acute Coronary Treatment and Intervention Outcomes Network Registry-Get with the Guidelines (ACTION Registry-GWTG) risk score,[4] and the Acute catheterization and urgent intervention triage strategy and The Harmonizing Outcomes with Revascularization and Stents in Acute Myocardial Infarction (ACUITY-HORIZONS) risk score.[5] However, almost all of the participants of the three models were European descent, and the models functions have not be evaluated in Chinese population. Directly applying these bleeding prediction models may over or under estimate the bleeding risk.

Calibrating the three bleeding risk models can substantially improve predictive abilities and generalize the models to other populations with different genetic, social, and cultural backgrounds. Therefore, we investigated the performance of CRUSADE, ACTION, and ACUITY-HORIZON scores in Chinese patients, aiming to obtain evidence of which score is the most accurate and reliable quantitative clinical tool for predicting in-hospital major bleeding among ACS patients treated by percutaneous coronary intervention(PCI) in China.

Methods

Study population

This was a retrospective study. The subjects were admitted consecutively between January 1st and June 30th, 2015 to the Beijing Anzhen Hospital and were all diagnosed of ACS. The initial cohort of the study comprised 8,926 patients. Patients underwent coronary artery bypass graft (CABG, n = 2340), conservative therapy (n = 1647) were excluded from this study (n = 3987). Thus, there were 4939 patients constituted the final study cohort.

The details of clinical characteristics, antithrombotic therapy, angiography parameters, biochemical and electrocardiography, in-hospital complications were collected and recorded. Patients were classified as ST-elevation myocardial infarction (STEMI) or non-ST-elevation ACS [NSTE-ACS, unstable angina and non ST-elevation myocardial infarction (NSTEMI)].

Clinical endpoint and definitions

The primary endpoint was the predictive accuracy of the three contemporary risk scores for classifying the risk of in-hospital bleeding in ACS patients undergoing PCI. We recorded the information of bleeding localization, imaging tests, admission and nadir hemoglobin/hematocrit levels, and blood transfusion. The major bleeding complications defined according to thrombolysis in myocardial infarction (TIMI)[6] as a “gold standard” to compare the predictive ability of these scores. TIMI minor bleeding was defined as any clinically overt bleeding (including on imaging), resulting in hemoglobin drop of 3−5 g/dL. TIMI major bleeding was defined as intracranial hemorrhage, fatal bleeding (bleeding that directly results in death within seven days), or clinically overt bleeding (including on imaging) associated with a drop in hemoglobin of ≥ 5 g/dL. The diagnosis of intracranial bleeding required confirmation by computed tomography or magnetic resonance imaging of the head. In our study, TIMI serious bleeding was defined as TIMI major or TIMI minor bleeding.

Statistical analysis

Continuous variables are presented either as median (25th and 75th percentiles) and categorical variables are presented as n (%) and were compared by the χ2 test or Fisher Exact test, as appropriate. Risk scores were calculated in each patient from the corresponding prognostic variables scores. Both CRUSADE and ACTION models were constructed to assign patient into five risk strata (very low, low, moderate, high, and very high risk). In the ACUITY-HORIZONS, patients were stratified in four risk categories for bleeding (low, moderate, high, and very high risk). In our study, patients were categorized into three bleeding risk strata for all scores by considering the very high and high risk categories as a unique category (high risk) and the very low and low risk categories as a low-risk category.

Both calibration and discrimination of the model were assessed with respect to the all patients. Risk model calibration was assessed using the Hosmer-Lemeshow goodness-of-fit test.[7] Discrimination was evaluated by receiver operating characteristics (ROC) curves and expressed by c-statistics and > 0.70 has acceptable discriminatory capacity.[8] The C-statistics for the three risk models were compared to each other using a nonparametric test.[9] A separate analysis was performed for the patients' diagnosis NSTE-ACS and STEMI. A two-sided P-value < 0.05 was considered to indicate a statistically significant difference. All statistical analyses were performed using the SPSS software (version 23; IBM Corp., USA). This study complies with the Declaration of Helsinki. The local ethics committee has approved our research protocol.

Result

Baseline characteristics

A total of 4939 ACS patients underwent PCI were included in this current analysis. Baseline clinical characteristics are shown in Table 1. The overall ACS population age was 58 (interquartile ranged: 51 to 64) years old, and women was 22.7%. Patients presented with STEMI in 22.5% of cases, NSTE-ACS in 77.5%. PCI was performed via radial artery access in 90.4% of cases. Patients received thrombolysis in 0.8%.

Table 1.

Clinical Characteristics and in-hospital management of study population.

	Overall (n = 4939)
Clinical characteristics
Age, yrs	58 (51–64)
Female	1120 (22.7%)
Weight, kg	75 (68–80)
Hypertension	2558 (51.7%)
Diabetes mellitus	1563 (31.6%)
Dyslipidemia	2038 (41.2%)
Chronic kidney disease	94 (1.9%)
Previous PCI	617 (12.5%)
Previous CABG	84 (1.7%)
Previous MI	294 (5.9%)
Previous ischemic stroke	202 (4.1%)
Peripheral arterial disease	132 (2.7%)
On-admission data
STEMI	1113 (22.5%)
NST-ACS	3830 (77.5%)
Cardiac function class
I	4459 (90.2%)
II	369 (7.5%)
III	77 (1.6%)
IV	38 (0.8%)
Systolic blood pressure, mmHg	125 (120–130)
Heart rate, beats/min	68 (66–75)
Hemoglobin, g/dL	142 (133–152)
Hematocrit	41.4 (38.8–43.9)
Anemia	755 (15.3%)
Serum creatinine, mg/dL	91.0 (73.3–111.1)
*Creatinine clearance, mL/min	0.9 (0.8–1.0)
Leucocyte, g/L	6.6 (5.6–8.0)
In-hospital management
Aspirin	4942 (100%)
P2Y12 inhibitors
Clopidogrel	2624 (53.1%)
Ticagrelor	2319 (46.9%)
Heparin	2799 (56.6%)
Fondaparinux	34 (0.7%)
Bivalirudin	52 (1.1%)
Glycoprotein IIb/IIIa inhibitors	961 (19.4%)
Two antithrombotic agents	1848 (37.4%)
Three antithrombotic agents	2403 (48.7%)
Four antithrombotic agents	692 (14.0%)
Vascular access site
Radial	4469 (90.4%)
Femoral	474 (9.6%)
Thrombolysis	40 (0.8%)
Home warfarin	5 (0.1%)

Data were presented as n (%) or median (interquartile range). *Calculated by Cockcroft-Gault formula. CABG: coronary artery bypass graft; GPI: Glycoprotein IIb/IIIa inhibitor; MI: myocardial infarction; NSTE-ACS: non ST-segment elevation acute coronary syndrome; PCI: percutaneous coronary intervention; STEMI: ST-segment elevation myocardial infarction.

Incidence of TIMI serious bleeding

Figure 1 shows details of the TIMI serious bleeding types. In-hospital TIMI serious bleeding events was in 54 cases overall. The most prevalent cause of major bleeding complications was gastrointestinal bleeding in 44 (81.5%) cases, followed by intracranial hemorrhage in 6 (11.1%) cases, others were retroperitoneal bleeding 1 (1.9%) and vascular access hematomas 2 (3.7%). In gastrointestinal bleeding, almost one half had a hemoglobin drop of > 5 mg/dL. Two-thirds of intracranial bleeding complications were related to fatal.

Figure 1.

Detail of TIMI serious bleeding.

Comparative performance of risk scores using TIMI serious bleeding definition. TIMI: thrombolysis in myocardial infarction.

TIMI serious bleeding rate was 1.1% overall (1.9% and 0.86% for STEMI and NSTE-ACS, respectively). As shown in Figure 2, in our study, the major bleeding rate of overall patients in the three scores categories were: CRUSADE score low risk 0.6%, moderate risk 2.1%, high risk 5.3%; ACTION score low risk 0.6%, moderate risk 5.5%, high risk 27%; and ACUITY-HORIZONS score low risk 0.5%, moderate risk 1.3%, and high risk 3%. The distribution of the in-hospital TIMI serious bleeding rates in the different risk groups showed a consistent gradient of risk for each risk scores in overall and both ACS patients (Figure 2), the rate of Chinese patients who experienced in-hospital major bleeding were obviously lower than the rates reported in the three risk models original researches.

Figure 2.

Distribution of TIMI serious bleeding rates with respect to risk categories for CRUSADE, ACTION, and ACUITY-HORIZONS risk scores, in ACS (A), NSTE-ACS (B), and STEMI (C) patients.

*P-values for comparisons across the different risk categories of each score. NSTE-ACS: non ST-segment elevation acute coronary syndrome; STEMI: ST-segment elevation myocardial infarction; TIMI: thrombolysis in myocardial infarction.

Table 2 summarized the calibration and discrimination of the three risk scores. The calibration was good for our study population, as shown by the non-significant results of the Hosmer−Lemeshow tests. The CRUSADE, ACTION and ACUTIY-HORIZONS showed an adequate discriminatory capacity for TIMI serious bleeding: in overall patients, the c-statistic was 0.80, 0.77, and 0.70, respectively (Figure 3A); in NSTE-ACS patients, the c-statistic was 0.73, 0.72, and 0.64, respectively (Figure 3B); in STEMI patients, the c-statistic was 0.91, 0.92, and 0.75, respectively (Figure 3C).

Table 2

Calibration and discrimination for the three risk scores of in-hospital major bleeding events in STEMI and NSTE-ACS.

Risk score	Risk group	n	HL		C-statistic (95% CI)
			χ2	P
CRUSADE	Overall	4943	8.2	0.41	0.80 (0.74–0.85)
	NSTE-ACS	3830	10.1	0.26	0.73 (0.64–0.81)
	STEMI	1113	11.7	0.20	0.91 (0.88–0.94)
ACTION	Overall	4943	14.9	0.06	0.77 (0.70–0.85)
	NSTE-ACS	3830	14.6	0.07	0.72 (0.63–0.81)
	STEMI	1113	3.9	0.87	0.92 (0.85–0.97)
ACUITY-HORIZONS	Overall	4943	6.7	0.57	0.70 (0.62–0.77)
	NSTE-ACS	3830	4.2	0.84	0.64 (0.54–0.74)
	STEMI	1113	4.3	0.63	0.75 (0.66–0.83)

HL: Hosmer−Lemeshow test; NSTE-ACS: Non ST-segment elevation acute coronary syndrome; STEMI: ST-segment elevation myocardial infarction.

Figure 3.

Receiver operating characteristics curves of the CRUSADE, ACTION, and ACUITY-HORIZONS models for major bleeding prediction in ACS (A), NSTE-ACS (B), and STEMI (C) patients.

ACS: acute coronary syndrome; NSTE-ACS: Non ST-segment elevation acute coronary syndrome; STEMI: ST-segment elevation myocardial infarction.

The C-statistic for the ACUITY-HORIZONS model was significantly lower than the CRUSADE and ACTION scores for the prediction of TIMI serious bleeding in overall patients (compared with CRUSADE, z = 3.83, P = 0.02; compared with ACTION, z = 3.51, P = 0.03); in NSTE-ACS patients (compared with CRUSADE, z = 2.37, P = 0.01; compared with ACTION, z = 2.11, P = 0.04), and in STEMI patients (compared with CRUSADE, z = 2.6.77, P = 0.02; compared with ACTION, z = 7.91, P = 0.002). No differences were observed when the CRUSADE and ACTION models were compared to each other, regardless of overall patients (z = 0.68, P = 0.31) and both of ACS types (NSTE-ACS, z = 0.52, P = 0.60), and STEMI patients (z = 0.36, P = 0.74 Table 3).

Table 3.

Comparisons of the discriminative power of the tree risk scores for predicting TIMI serious bleeding.

Risk group	Comparisons	z	P
Overall (n = 4943)	CRUSADE vs. ACTION	0.68	0.31
	CRUSADE vs. ACUITY-HORIZONS	3.83	0.02
	ACTION vs. ACUITY-HORIZONS	3.51	0.03
NSTE-ACS (n = 3830)	CRUSADE vs. ACTION	0.52	0.60
	CRUSADE vs. ACUITY-HORIZONS	2.37	0.01
	ACTION vs. ACUITY-HORIZONS	2.11	0.04
STEMI (n = 1113)	CRUSADE vs. ACTION	0.36	0.74
	CRUSADE vs. ACUITY-HORIZONS	6.77	0.02
	ACTION vs. ACUITY-HORIZONS	7.91	0.002

The null-hypothesis z-test result is shown for the comparisons of the C-statistic for the three risk models and the respective P-value, obtained by the DeLong nonparametric method. NSTE-ACS: Non ST-segment elevation acute coronary syndrome; STEMI: ST-segment elevation myocardial infarction.

Discussion

In this study, we evaluated the performance of the CRUSADE, ACUITY-HORIZONS, and ACTION models' functions in a Chinese cohort of ACS treated by PCI. By using the neutral definition of TIMI serious bleeding, the CRUSADE and ACTION scores showed good calibration and discrimination abilities for Chinese population. The present study is the first to compare predictive value in the three contemporary bleeding risk scores, using an independent dataset of Chinese ACS patients.

The first finding in this analysis is the CRUSADE and ACTION risk scores performed equally better predictive ability for in-hospital major bleeding than the ACUITY-HORIZONS model regard to overall ACS patients and NSTE-ACS patients. This is an important finding since the three scores were originally developed in westerners. The differences of discrimination between them might be explained by different variables and subjects comprised by each model: the CRUSADE and ACTION models have similar variables, such as female sex, renal function, and anemia, heart rate, systolic blood pressure, heart failure, diabetes, and vascular disease, which are clearly associated with bleeding in ACS, Nevertheless, the ACUITY-HORIZONS risk model did not include these variables. Besides, in the original research, the CRUSADE model was developed to estimate the patients' baseline risk of in-hospital major bleeding during NSTEMI, and recommended to apply before initiating a management strategy by NSTE-ACS guidelines.[10],[11] The ACTION score focused on STEMI and NSTEMI patients, considered excess antithrombotic dosing related to in-hospital bleeding, which took efforts in improving the management for those have a high risk ischemic events. While ACUITY-HORIZONS score offered an advantage of providing quantitative tool in a broader spectrum of ACS patients treated heparin + Glycoprotein IIb/IIIa inhibitor (GPI) or bivalirudin alone, but considered treatment in determining bleeding risk stratification. Moreover, the CRUSADE and ACTION models are derived from ‘real-world’ population, in contrast to the more selected derivation population of the ACUITY-HORIZONS model which might limit its generalizability.

The second finding in this study is that we found the CRUSADE score showed good predictive ability similar to ACTIOIN score for in-hospital bleeding risk in STEMI patients. They all seem a superiority over the ACUITY-HORIZONS score. Previously, the external validity of the CRUSADE score for STEMI had been less evaluated in Chinese patients. In fact, the CRUSADE score and ACTION score have only minor differences in their composition. The ACTION model comprises prior warfarin use and ECG changes, which are not included in the CRUSADE model. However, none of these differences seems to be relevant for major bleeding prediction in our STEMI cohort. Despite the concerns regarding the impact of antithrombotic therapy in the management of STEMI, home warfarin use was uncommon in STEMI group, having no impact in major bleeding. Given the inclusion of ECG changes as defined by researchers, ACTION risk score showed a little better performance than CRUSADE score in our analysis (CRUSADE c = 0.91 vs. ACTION c = 0.92). Actually, we support the use of CRUSADE score in the STEMI because of its' simplicity, superiority and generalizability, not only in the NSTE-ACS, which been also found by Kadakia, et al.[12] and other studies[13],[14] It is enough to prove that CRUSADE model is also potent prognosticators of bleeding development in the STEMI scenario. It will be convince a busy physician to use CRUSADE for STEMI as well as for NSTE-ACS patients.

The third finding is the in-hospital major bleeding rate in Chinese ACS cohort was 1.1%, which was similar with the incidences reported between 0.2% to 9.1% in previously studies.[15],[16] However, in the original reports of the three risk scores, the corresponding crude incidence rates of CRUSADE score was at least 3.1%, 8.6% and 11.9%, ACTION score was at least 4.8%, 16%, 28%, and the ACUITY-HORIZONS score was 1.9%, 3.3%, and 6.9%, respectively. Compared to our observation, the major bleeding rate in the three scores categories were: CRUSADE score low risk 0.6%, moderate risk 2.1%, high risk 5.3%; ACTION score low risk 0.6%, moderate risk 5.5%, high risk 27%; and ACUITY-HORIZONS score low risk 0.5%, moderate risk 1.3%, and high risk 3%. The variations in incidence are due to the differences in study subjects, indication for PCI, received medication and single versus multicenter studies. Even though the Chinese population is more likely lower weight, more smoked, and higher prevalence rates for diabetes, but all other risk factor levels of baseline examination were lower than westerners.[17] for example, the baseline heart rate, systolic blood pressure, anemia and history of peripheral arteriopathy proportion. On the other hand, as the advance of coronary invention technology and medication strategies, the access-site bleeding rate was decline. Our analysis showed major bleeding of vascular access site only in 3.7%, obviously lower than previously study reported of 33%.[18],[19] In contemporary era, the Chinese clinicians pay more attention to the medication strategy for ACS to minimize the bleeding risk, and balance the ischemic risk, such as in our cardiology center, radial access rate was more than 90%, patients received thrombolysis only in 0.8% cases, GPI use less than 20% and only in the conditions of heavy thrombosis and no-reflow, and operators preferred to use of bivalirudin in high bleeding risk patients.[20],[21] In addition, we enrolled almost half proportion patients received the new antiplatelet agent ticagrelor, which showed higher rates of bleeding events than clopidogrel, but decreased the ischemic events.[22]

In the modern era of risk-tailored and personalized cardiovascular treatment, studies on risk prediction models have been inclined to multiple and competing risk scores in different race of population. Therefore, the clinicians advocate to perform these risk scores in independent cohort before application.[23],[24] It is important because the models with insufficient accuracy would increase adverse events by underestimating the real risks. In our study, we found the scores still have overestimated bleeding risk rate in Chinese populations, which maybe mislead the physician to choose the strategy of obviating intensive antithrombotic therapy in patients at moderate or high ischemic risk stratifications, yet increase risk of ischemic complications who would potentially benefit from the use of effective treatments. The future research may need to focus on testing the ischemic models and bleeding models in Chinese cohort and try to develop a new risk prediction score for Chinese ACS patients.

Limitations

One limitation of this study is that it is a retrospective study and the data came from a single center registry, as the management of patients is relatively homogenous. Second, we did not compare performance of the three models by using their own major bleeding definitions. Finally, the performance of the ACUITY-HORIZONS score was assessed regarding in-hospital bleeding events while it was originally designed to predict 30-day bleeding.

Conclusions

The CRUSADE model, followed by the ACTION score had a greater calibration and discrimination for in-hospital major bleeding than the ACUITY-HORIZONS risk score in Chinese patients with ACS undergoing PCI, also apply in STEMI patients. However, they all overestimate the bleeding risk for Chinese populations. This would be useful for the generalization of these prediction risk models in other populations, and remind us to develop a bleeding risk model suited for local practice of prevention in China.