Comparison of ACUITY and CRUSADE Scores in Predicting Major Bleeding during Acute Coronary Syndrome.

BACKGROUND: The ACUITY and CRUSADE scores are validated models for prediction of major bleeding events in acute coronary syndrome (ACS). However, the comparative performances of these scores are not known.
OBJECTIVE: To compare the accuracy of ACUITY and CRUSADE in predicting major bleeding events during ACS.
METHODS: This study included 519 patients consecutively admitted for unstable angina, non-ST-elevation or ST-elevation myocardial infarction. The scores were calculated based on admission data. We considered major bleeding events during hospitalization and not related to cardiac surgery, according to the Bleeding Academic Research Consortium (BARC) criteria (type 3 or 5: hemodynamic instability, need for transfusion, drop in hemoglobin ≥ 3 g, and intracranial, intraocular or fatal bleeding).
RESULTS: Major bleeding was observed in 31 patients (23 caused by femoral puncture, 5 digestive, 3 in other sites), an incidence of 6%. While both scores were associated with bleeding, ACUITY demonstrated better C-statistics (0.73, 95% CI = 0.63 - 0.82) as compared with CRUSADE (0.62, 95% CI = 0.53 - 0.71; p = 0.04). The best performance of ACUITY was also reflected by a net reclassification improvement of + 0.19 (p = 0.02) over CRUSADE’s definition of low or high risk. Exploratory analysis suggested that the presence of the variables ‘age’ and ‘type of ACS’ in ACUITY was the main reason for its superiority.
CONCLUSION: The ACUITY Score is a better predictor of major bleeding when compared with the CRUSADE Score in patients hospitalized for ACS.

Introduction

Individuals admitted with acute coronary syndrome (ACS) are at considerable risk of ischemic complications during the acute phase. Thus, aggressive pharmacological and interventional therapies are adopted to minimize the likelihood of recurrent events, such as refractory angina, re-infarction or cardiovascular death. However, the same interventions designed to protect against ischemic complications are the ones to increase the likelihood of major bleeding during hospitalization.

Since major bleeding is associated with mortality[1], clinical decision should balance the risk of recurrent ischemia and that of bleeding. Multivariate models for risk prediction of cardiovascular events in ACS were validated early in the last decade[2,3]. Those scores have been compared with each other, the GRACE (Global Registry of Acute Coronary Events) model showing the best accuracy[4,5]. On the other hand, bleeding scores, such as ACUITY (Acute Catheterization and Urgent Intervention Triage strategY) and CRUSADE (Can Rapid risk stratification of Unstable angina patients Suppress ADverse outcomes with Early implementation of the ACC/AHA guidelines), have been validated only recently, but not yet compared.

To assess whether there is any superiority of one bleeding score over the other, we evaluated the agreement between ACUITY and CRUSADE, compared their C-statistics and analyzed their reclassification. A prospective cohort of 519 consecutive patients with ACS had bleeding scores calculated on admission, and major bleeding registered according to the Bleeding Academic Research Consortium (BARC) criteria[6].

Methods

Sample Selection

This is an analysis of the Registry of Acute Coronary Syndromes (REACS). In this Registry, consecutive patients with rest onset of typical chest discomfort within the previous 48 hours, admitted to the coronary care units of two tertiary hospitals in the city of Salvador, Brazil, between August 2007 and December 2011, were evaluated for inclusion in the REACS. To include patients with non-ST-elevation ACS, at least one of the three objective criteria should be present: electrocardiographic changes consisting of transient ST‑segment depression (≥ 0.05 mV) or T-wave inversion (≥ 0.1 mV); troponin change to a level beyond the 99th percentile threshold of a healthy reference population, with 10% coefficient of variability[7]; or previous documentation of coronary artery disease, defined as a definitive history of myocardial infarction or coronary obstruction ≥ 50% on angiography. For inclusion of ST‑elevation acute myocardial infarction, a persistent ST‑segment elevation of at least 0.1 mV in at least two contiguous leads or a left bundle‑branch block, with subsequent Q wave formation and elevation of serum marker of myocardial necrosis, was required. Patient’s option not to participate in the Registry was the sole exclusion criterion. All participants provided written informed consent.

Study Protocol

The study protocol conforms to the ethical guidelines of the 1975 Declaration of Helsinki as reflected in a priori approval by the institution's human research ethics committee. Based on clinical and laboratory data collected on admission, the ACUITY and CRUSADE scores were calculated, according to the original studies’ definition[8,9]. During hospitalization, patients were prospectively followed up for the detection of major bleeding, as our primary endpoint. Major bleeding was defined as type 3 or type 5 of the Bleeding Academic Research Consortium (BARC)[6]. The criteria for type 3 bleeding are as follows: hemoglobin drop of 3-5 g% or need for blood transfusion (type 3a); a drop in hemoglobin ≥ 5 g%, cardiac tamponade, need for surgical treatment or hemodynamic instability (type 3b); and intracranial or intraocular bleeding (type 3c). Type 5 is a definitive fatal bleeding (direct causal link, type 5a) or probable fatal bleeding (indirect causal link, type 5b). CABG-related bleeding (type 4) was not taken into account in our bleeding definition. Bleeding was also classified according to site (femoral, gastrointestinal or other).

ACUITY and CRUSADE Scores

Briefly, the ACUITY Score consists of seven variables, three dichotomous (female sex, presence of anemia, use of bivalirudin), one nominal variable (type of ACS: unstable angina, non-ST-elevation or ST-elevation acute myocardial infarction) and three semiquantitative variables (age, serum creatinine and white blood cell count, all analyzed as ordinal categories). In this score, predisposing factors for bleeding are female sex, presence of anemia, advanced age, elevated creatinine, high white blood cell count and ACS type (ST‑elevation myocardial infarction being the higher risk, followed by non-ST- myocardial infarction and unstable angina). The use of bivalirudin is supposed to be a protective factor, but this variable was never present in our patients, because this drug is not commercially available in Brazil[9].

The CRUSADE Score consists of eight variables, four dichotomous (female sex, heart failure signs, diabetes and peripheral artery disease) and four semiquantitative (baseline hematocrit, creatinine clearance, heart rate, systolic blood pressure, all analyzed as ordinal categories). All dichotomous variables were predisposition factors, as were low hematocrit, creatinine clearance and blood pressure, and high heart rate[8]. Creatinine clearance was calculated according to the Cockcroft & Gault formula.

In both scores, points are attributed according to the values of each variable, and the sum of all variables corresponds to previously defined risk levels.

Statistical Analysis

As ordinal variables, the bleeding scores were described as medians and interquartile ranges. Considering their normal distribution, we additionally described the scores as mean ± standard deviation. Agreement between the scores in defining high risk of bleeding was evaluated by Kappa test. In this analysis, high risk was defined by the threshold ≥ 10% according to the validation studies of each score (ACUITY > 20 and CRUSADE > 40)[8,9]. Agreement was also evaluated in the definition of low, intermediate or high risk, according to the observed scores’ tertiles.

Secondly, the scores’ predictive performances were evaluated and compared with each other. Calibration was evaluated by Hosmer and Lemeshow’s test, with a calibrated score defined by a p value ≥ 0.05. Most importantly, discrimination was assessed by C-statistics, as the area under the ROC curve of each score for predicting major bleeding. Scores’ C-statistics were compared by the method of Henley and McNeil[10]. The optimal cut-off point in the ROC curve was identified by the maximal difference between sensitivity and 1 - specificity. Then, sensitivity and specificity according to the optimized cut-off were compared between the scores by the McNemar’s test.

After identifying the score that performed best, it was used to reclassify the definition of high or low risk initially assessed by the other score. The definition of low and high risk for each score was performed in two ways: primarily, using the optimal cut-off point on the ROC curve; and secondarily, using the cut-off points for the threshold of risk ≥ 10% indicated by the validation studies of each score (ACUITY > 20 and CRUSADE > 40)[8,9]. The impact of the reclassification procedure by using the superior score was assessed by using the method of net reclassification improvement (NRI). Briefly, this method focuses on reclassification tables constructed separately for participants with and without events, and quantifies the correct (upwards for events and downwards for non‑events) and incorrect (downwards for events and upwards for non-events) movements between categories. The net reclassification is the balance between correct and incorrect movements. Positive values of NRI indicate a predominance of correct reclassification, while negative values indicate a predominance of incorrect reclassification. p value < 0.05 rejects the null hypothesis of NRI = 0[11].

To evaluate whether the scores have complementary predictive value to each other, we performed a logistic regression analysis taking bleeding as the dependent variable and the scores (entered as numeric variables) as independent variables. Independent prediction of both scores would suggest complementary value.

Finally, an exploratory analysis was performed to evaluate which components of the scores were mostly associated with bleeding. Student’s t test was used to compare numeric variables between bleeding and non-bleeding patients, and Pearson’s chi-square to compare categorical variables. The Statistical Package for the Social Sciences (SPSS) software (SPSS Inc., Chicago, Illinois, USA), version 9.0, was used for data analysis, and final statistical significance was defined as p < 0.05 in all cases.

Results

Sample Characteristics

This study assessed 519 patients (mean age of 67 ± 13 years, 54% males), 37% of whom with an index diagnosis of unstable angina, 47% with non-ST-elevation acute myocardial infarction and 22% with ST-elevation acute myocardial infarction. During hospitalization, most patients underwent dual antiplatelet treatment plus full anticoagulation, while only 5.7% received antagonists of glycoprotein IIb/IIIa. No patient used bivalirudin. Percutaneous coronary intervention was performed in 37% of the patients, 41% underwent coronary angiography without intervention, and the remaining 22% did not undergo any percutaneous procedure (Table 1). During the study period, all percutaneous procedures were performed via the femoral site.

Table 1

Baseline characteristics and treatment during hospitalization

Baseline characteristics
Sample size	519
Age (years)	67 ± 13
Female sex	237 (46%)
Index diagnosis
Unstable angina	191 (37%)
NSTEMI	215 (41%)
STEMI	113 (22%)
Serum creatinine (mg/dL)	1.16 ± 1.0
Killip Class > 1	87 (17%)
Grace score (median, IQR)	119 (96 - 148)
GRACE low risk	221 (43%)
GRACE intermediate risk	161 (31%)
GRACE high risk	137 (26%)
Gensini angiographic score (median, IQR)	109 (71 - 166)
Severe coronary disease*	187 (36%)
In-hospital Treatment
Aspirin	512 (99%)
Clopidogrel	465 (90%)
Ticagrelor	9 (0.2%)
Abxicimab	14 (2.6%)
Tirofiban	16 (3.1%)
Enoxaparin	434 (84%)
Fondaparinux	16 (3.1%)
Unfractionated heparin	17 (3.3%)
Percutaneous coronary intervention	193 (37%)
Coronary angiography w/o intervention	210 (41%)
Coronary artery bypass surgery	44 (8.5%)

STEMI: ST-elevation myocardial infarction; NSTEMI: non-ST-elevation myocardial infarction; IQR: interquartile range.

left main or triple vessel disease

The ACUITY Score had a normal distribution [mean of 16 ± 7.0, median of 16 (interquartile range, 11 - 21)]. The CRUSADE Score also had a normal distribution [mean of 39 ± 15, median of 40 (interquartile range, 29 - 50)]. During hospitalization, major bleeding was observed in 31 patients, leading to an incidence of 6%. Of those, 23 events were related to the femoral puncture site, 5 to gastrointestinal bleeding, and 3 to other sites. Only one major bleeding was fatal (type 5).

Agreement between ACUITY and CRUSADE

According to ACUITY, 26% of the patients were defined as at high risk of bleeding (risk ≥ 10%), while CRUSADE classified 48% of the individuals as high risk. There was 69% agreement between the two in defining high risk, with a modest Kappa coefficient of 0.36 (95% CI = 0.28 - 0.43; p < 0.001). Of the patients with disagreement between the two scores, 85% were characterized as high risk by CRUSADE, as opposed to ACUITY. When the bleeding risk was defined as low, moderate or high (according to scores’ tertiles), agreement between CRUSADE and ACUITY was also modest (57%, Kappa = 0.36; 95% CI = 0.30 - 0.43; p < 0.001).

Predictive Value of ACUITY versus CRUSADE

Calibration

In predicting the bleeding incidence, ACUITY and CRUSADE were similarly calibrated, according to Hosmer and Lemeshow’s chi-square of 7.1 (p = 0.42) and 7.5 (p = 0.38), respectively.

Discrimination

The discriminatory ability of the ACUITY Score for bleeding events was demonstrated by a C-statistics of 0.73 (95% CI = 0.63 - 0.82), significantly better than CRUSADE’s C-statistics of 0.62 (95% CI = 0.53 - 0.71; p = 0.04 for the comparison between the scores) (Figure 1). The optimal cut-off points for ACUITY and CRUSADE were 19 and 31, respectively. Based on these points, CRUSADE had a better sensitivity (90%; 95% CI = 80% - 100%) as compared with ACUITY (71%; 95% CI = 55% - 87%; p = 0.004), but at the expense of much worse specificity of CRUSADE (32%; 95% CI = 28% - 36%) in relation to ACUITY (71%; 95% CI = 67% - 75%; p < 0.001).

Figure 1

Receiver-Operating Characteristic Curves of the Bleeding Risk according to the ACUITY and CRUSADE scores. Caption: ACUITY’s C-statistics was significantly better than CRUSADE’s

The C-statistics’ superiority of ACUITY over CRUSADE was consistent across non-ST-elevation ACS (0.66 versus 0.57, respectively) and ST-elevation ACS (0.87 versus 0.80, respectively) (Figure 2).

Figure 2

Receiver-Operating Characteristic Curves of the Bleeding Risk according to the ACUITY and CRUSADE scores, in the subgroups of non-ST and ST-elevation acute coronary syndromes. Caption: ACUITY’s C-statistics is consistently better than CRUSADE’s, independent of the acute coronary syndrome presentation

Reclassification by ACUITY

Of the 488 individuals without bleeding, CRUSADE incorrectly classified 330 as high risk according to optimal cut-off point. ACUITY correctly reclassified 200 of these patients as low risk. Of the 158 correctly classified by CRUSADE as low risk, ACUITY incorrectly reclassified 11 patients as high risk. Thus, more correct than incorrect reclassification was provided by ACUITY over CRUSADE in patients without bleeding. This provided a significant NRI of 0.38 for patients without bleeding (p < 0.001).

Of the 31 patients with bleeding events, CRUSADE incorrectly classified 3 as low risk; none of those were reclassified by ACUITY. Of the 28 patients correctly classified by CRUSADE as high risk, ACUITY incorrectly reclassified 6 as high risk. Thus, more incorrect than correct reclassification was provided by ACUITY over CRUSADE in patients with bleeding. This provided a NRI of - 0.19 for patients with bleeding (p = 0.01).

Since the positive NRI in patients without bleeding was higher than the negative NRI in patients with bleeding, a global NRI of + 0.19 (P = 0.02) indicated a balance in favor of correct reclassification by ACUITY over CRUSADE (Table 2).

Table 2

Net reclassification improvement obtained by ACUITY over CRUSADE risk stratification

Observed events	CRUSADE	Reclassification by ACUITY		NRI	Z Score	p Value
		Low Risk	High Risk
Bleeding (31)	Low risk = 3	3	0	- 0.19	2.49	0.01
	High risk = 28	6	22
No bleeding (488)	Low risk = 158	147	11	+ 0.38	13	< 0.001
	High risk = 330	200	130
Global NRI by ACUITY Score				+ 0.19	2.34	0.02

NRI: net reclassification improvement

When cut-off points from previous validation studies were used as thresholds of risk ≥ 10% (ACUITY > 20 and CRUSADE > 40), a similar global NRI of + 0.20 (p = 0.03) was observed. It resulted from a non-significant NRI of - 0.03 among patients with events (p = 0.92) and a significant NRI of + 0.23 among those without events.

Independent Predictive Value

When both ACUITY and CRUSADE were entered as numeric covariates in a logistic regression model for predicting major bleeding, CRUSADE lost statistical significance (p = 0.66), while ACUITY remained a predictor of bleeding events (OR = 1.12; 95% CI = 1.06 - 1.19; p < 0.001).

Reasons for ACUITY Superiority: Exploratory Analysis

Of the four variables exclusive of the ACUITY Score, the following three were significantly associated with bleeding: mean age, higher in bleeding than in non-bleeding individuals (76 ± 11 years versus 67 ± 13 years, respectively; p < 0.001); white blood cell count, higher in bleeding than in non-bleeding individuals (10,876 ± 3,735 versus 9,062 ± 4,510, respectively; p = 0.03); and infarction of any type, more prevalent in bleeding patients, while unstable angina was less prevalent (p = 0.005). Only creatinine was not associated with bleeding (Table 3).

Table 3

Association of the score’s individual variables and major bleeding

	Bleeding	No bleeding	p Value
Sample size	31	488
Common criteria
Female sex	15 (48%)	267 (55%)	0.49
Hematocrit (%)	40 ± 4.7	40 ± 5.5	0.53
CRUSADE exclusive criteria
Heart rate (bpm)	86 ± 22	78 19	0.02
Creatinine clearance (mL/min)	47 ± 25	60 ± 26	0.008
Diabetes	14 (45%)	172 (35%)	0.27
Heart failure	9 (29%)	78 (16%)	0.06
Vascular disease	18 (58%)	279 (58%)	0.97
ACUITY exclusive criteria
Age (years)	76 ± 11	67 ± 13	< 0.001
White blood cells	10.876 ± 3.735	9.062 ± 4.510	0.03
Presentation			0.005
STEMI	9 (29%)	104 (21%)
NSTEMI	19 (61%)	196 (40%)
Unstable angina	3 (9.7%)	188 (39%)
Serum creatinine (mg/dL)	1.3 ± 1.2	1.2 ± 0.9	0.27

STEMI: ST-elevation myocardial infarction; NSTEMI: non-ST-elevation myocardial infarction; IQR: interquartile range

On the contrary, of the five exclusive variables of CRUSADE, only heart rate (86 ± 22 versus 78 ± 19; p = 0.02) and creatinine clearance (47 ± 25 mL/min versus 60 ± 26 mL/min; p = 0.008) differed between the two groups. Diabetes, heart failure and vascular disease did not reach statistical significance.

Finally, the two variables common to ACUITY and CRUSADE (sex and hematocrit) did not differ between bleeding and non-bleeding individuals. Thus, the cluster of variables best related to bleeding is present in the ACUITY Score, serving as a reason for its predictor superiority (Table 3).

Discussion

The present study provides the first head-to-head comparison of the two best validated tools for predicting major bleeding events in patients with ACS. In our sample population, the ACUITY Score performed better as compared with the CRUSADE Score. ACUITY’s superiority was indicated by an absolute 0.11 difference in C-statistics and a net reclassification improvement of 0.19 over CRUSADE’s classification. Moreover, the CRUSADE Score did not sustain its significance in multivariate analysis, while the predictive value of ACUITY was reinforced by its independent association with bleeding.

Prevention of recurrent ischemic events in ACS is effectively achieved through aggressive antithrombotic therapy and early coronary intervention. However, this effectiveness is achieved at the expense of increased bleeding events[12,13]. Since major bleeding is associated with increased mortality and morbidity[1,14-16], clinical decision should balance the ischemic risk against the hemorrhagic risk[17]. Multivariate models for predicting ischemic events in ACS are well calibrated and have good discriminatory performance. Previous studies have compared the two most popular models, the TIMI and GRACE Scores[4,5]. These studies have established the GRACE Score as the best model for predicting outcomes in ACS patients. On the contrary, bleeding scores have never been compared with each other.

Prior to the present study, one could hypothesize the CRUSADE Score to be the best predictor of bleeding. First, because it has more variables, which are disposed in a more quantitative fashion than ACUITY[8,9]. Second, CRUSADE derived from an observational registry of greater sample size, as opposed to ACUITY, which was created from an interventional clinical trial. Therefore, we should explore the reasons for ACUITY outperforming CRUSADE. As demonstrated in our Results section, disagreement between the two scores resulted from an overestimation of risk by CRUSADE as compared with ACUITY. This led to a greater sensitivity of CRUSADE in predicting bleeding, at the expense of a much lower specificity. It suggests that the greater number of variables in CRUSADE promoted an excessive number of patients characterized as vulnerable to bleeding. In addition, two important variables are only present in ACUITY: age and type of ACS. The former is universally present in all bleeding models to date, except CRUSADE. The latter is clinically related to the aggressiveness of treatment, which predisposes to bleeding. However, the type of ACS was not even considered in CRUSADE’s univariate analysis. Finally, we demonstrated that ACUITY’s greater number of exclusive variables, as compared with CRUSADE, was associated with bleeding. This indicates that the ACUITY model had a better choice of candidate variables at the initial step of univariate analysis.

One concern is that the present results were driven by our study being more similar to the ACUITY’s than CRUSADE’s studies - characteristics depicted on Table 4. However, this does not seem to be the case, since our study population is actually closer to the CRUSADE Registry, as opposed to the ACUITY. Similar to our study, CRUSADE is a real world registry, while ACUITY is an interventional clinical trial; CRUSADE included both ST-elevation and non-ST-elevation ACS, as ours did, while ACUITY included only non-ST-elevation ACS. Although the definition of major bleeding was similar in both studies, CRUSADE presented a higher incidence of bleeding as compared with ACUITY (9.6% versus 3.8%). Our incidence was halfway between the two. Therefore, the better performance of ACUITY did not result from the greater similarities between our sample and that of the ACUITY Registry.

Table 4

Differences between the ACUITY and CRUSADE scores

	ACUITY	CRUSADE
Validation study
Sample size	17,000	71,000
ACS type	Only NSTEMI	STEMI and NSTEMI
Study type	Randomized clinical trial	Observational cohort
Bleeding Incidence	3%	9%
Number of variables	6	8
Age computed	Yes	No
ACS type computed	Yes	No

STEMI: ST-elevation myocardial infarction; NSTEMI: non-ST-elevation myocardial infarction; IQR: interquartile range; ACS: acute coronary syndrome.

Our net reclassification table indicated that 330 patients without bleeding were mistakenly classified by CRUSADE as at high risk of bleeding, due to its low specificity. Thus, the use of CRUSADE leads to clinical decisions towards a more conservative approach, not offering full pharmacological and invasive therapy to patients who could benefit from it. Instead, the use of ACUITY corrected that mistake in 200 of 330 patients. On the other hand, CRUSADE has better sensitivity than ACUITY in detecting those vulnerable to bleeding. But its advantage in sensitivity is much smaller than its disadvantage in specificity, leading to a net improvement of ACUITY over CRUSADE.

The applicability of the present findings should be discussed in light of the representativeness of our sample population. We selected a sample of consecutive patients with well-defined criteria for ACS, with a risk profile equally distributed into low, intermediate or high risk by the GRACE Score, as expected according to the risk definition based on tertiles of that score. Moreover, the mean age was typical of the ACS population, and individuals were symmetrically distributed into male and female sex. Thus, we believe our sample represents the average patient population with ACS.

The major limitation of our study was the relatively modest number of bleeding events. It implies imprecision in the magnitude of ACUITY superiority. On the other hand, as the major problem of a small sample size, the type II error did not take place. The difference of C-statistics between the two studies reached significance. Although not definitive, the present study is a first indication that ACUITY is a more promising tool in the risk/benefit stratification of patients with ACS.

In conclusion, as the first comparison between ACUITY and CRUSADE bleeding scores, the present study suggests that the former has a better accuracy as compared with the latter.