Prognostic value of TIMI score versus GRACE score in ST-segment elevation myocardial infarction.

BACKGROUND: The TIMI Score for ST-segment elevation myocardial infarction (STEMI) was created and validated specifically for this clinical scenario, while the GRACE score is generic to any type of acute coronary syndrome.
OBJECTIVE: Between TIMI and GRACE scores, identify the one of better prognostic performance in patients with STEMI.
METHODS: We included 152 individuals consecutively admitted for STEMI. The TIMI and GRACE scores were tested for their discriminatory ability (C-statistics) and calibration (Hosmer-Lemeshow) in relation to hospital death.
RESULTS: The TIMI score showed equal distribution of patients in the ranges of low, intermediate and high risk (39 %, 27 % and 34 %, respectively), as opposed to the GRACE Score that showed predominant distribution at low risk (80 %, 13 % and 7%, respectively). Case-fatality was 11%. The C-statistics of the TIMI score was 0.87 (95%CI = 0.76 to 0.98), similar to GRACE (0.87, 95%CI = 0.75 to 0.99) - p = 0.71. The TIMI score showed satisfactory calibration represented by χ2 = 1.4 (p = 0.92), well above the calibration of the GRACE score, which showed χ2 = 14 (p = 0.08). This calibration is reflected in the expected incidence ranges for low, intermediate and high risk, according to the TIMI score (0 %, 4.9 % and 25 %, respectively), differently to GRACE (2.4%, 25% and 73%), which featured middle range incidence inappropriately.
CONCLUSION: Although the scores show similar discriminatory capacity for hospital death, the TIMI score had better calibration than GRACE. These findings need to be validated populations of different risk profiles.

Introduction

Risk scores correspond to multivariate models used in clinical practice to estimate the individual probability of unwanted outcomes. Risk estimates have implications for clinical management, particularly with regard to broad-spectrum diseases, such as acute coronary syndromes (ACS). In this context, the main risk scores have been developed by the TIMI[1] and GRACE[2] groups. Our group[3] and others[4,5] have previously compared these two scores in patients with acute coronary syndromes without ST-segment elevation, and it was evident that the GRACE score had better prognostic accuracy. However, in patients with ST-segment elevation myocardial infraction it has not been established which of the two models has better prognostic performance with regard to in-hospital death.

The GRACE score was derived from a sample composed of a combination of patients with ACS without ST-segment elevation and patients with ST-segment elevation myocardial infarction. The former clinical condition was predominant in the sample[2]. Thus, the same score is used in both conditions. On the other hand, the TIMI score has distinct models for each of these clinical conditions, validated in distinct samples. Therefore, the TIMI score for ST-segment elevation myocardial infarction[6] comprises variables that are more specific for this clinical scenario.

To determine which of the two scores has better performance in patients with ST-segment elevation myocardial infarction, we studied the individuals who were part of the Acute Coronary Syndromes Registry (RESCA) and were admitted with this condition. In this hospital cohort, with regard to in-hospital death prediction, the discriminatory ability and calibration of the TIMI score was compared to those of the GRACE score.

Methods

Sample Selection

The patients who participated in this study were part of the Acute Coronary Syndromes Registry (Registro de Síndromes Coronarianas Agudas - RESCA) of our hospital and provided their signed informed consent document. This registry was approved by the Hospital's Research Ethics Committee, according to Resolution 466/2012 of the National Health Council. RESCA was started in August 2007 and includes patients consecutively admitted to the coronary unit of this hospital on the basis of the objective diagnostic criteria. The criterion for ST-segment elevation myocardial infarction was defined as chest pain associated with persistent ST-segment elevation of ≥ 0.1 mv (not relieved by nitrates) in two contiguous leads and confirmed by elevation of myocardial necrosis markers. In the presence of third-degree left bundle branch block, the diagnosis is based on the presented symptoms and on necrosis markers. The only predefined exclusion criterion in this study was the patient's refusal to participate in this registry, which did not occur.

Study Protocol

In this study, the outcome predicted by both scores was in-hospital death. The patients' data on presentation in the emergency room, the electrocardiographic exams performed in the first six hours of hospitalization, troponin T or I levels measured during the first 12 hours of hospitalization, and the first serum creatinine levels were used to calculate the scores. The elevation of myocardial necrosis markers, which is one of the components of the scores, was defined as troponin above the 99th percentile[7]. The Killip classification[8] was applied at time of admission.

The TIMI score was created and validated in a sample of patients with ST-segment elevation myocardial infarction, and thus, has specific variables for this clinical condition. In brief, this score consists of eight dichotomic variables, with the exception of age that adds points in two distinct strata. The presence of previous infarction (or left bundle branch block esquerdo), major risk factors (hypertension, diabetes, or dyslipidemia), weight <67 kg, and reperfusion time >4 hours adds one point each. The variables Killip > 1, heart rate >100 bpm, and systolic pressure < 100 mmHg adds 2 points each. Age adds 2 points if between 65 and 74 years and 3 points if ≥75 years. The final score may vary between 0 and 14[6].

The GRACE score was created and validated in a sample of patients with or without ST-segment elevation (1/3 and 2/3 of patients, respectively). Therefore, its variables are common to both clinical conditions. This score consists of eight variables; five of them are analyzed semiquantitatively: different weight for each age range, systolic arterial pressure, heart rate, serum creatinine, and Killip class; three of them are analyzed dichotomically: ST-segment depression, elevation of myocardial necrosis markers, and cardiac arrest on admission. The final score may vary between 0 and 372[2].

Statistical Analysis

For calculating sample size, we estimated a statistical power of 80% (alpha of 5%) for the detection of a difference of 0.10 between the ROC curves of both scores, considering a correlation between the scores of 0.80. Hence, a sample of 94 patients was thus calculated[9]. To optimize the statistical power, the number of patients with the outcome should be similar to the number of patients without the outcome. Because this was not expected in our sample, we empirically increased the size of the sample by 50%, thus obtaining a total of 141 patients.

The statistical analysis is explained in the diagram of Figure 1. The discriminatory ability of the scores with regard to risk of in-hospital death was assessed using ROC curves. The area under the ROC curve (C-statistics) represented the accuracy of each score to discriminate the survivors from the non survivors. C-statistics of the two scores were compared pairwise using the Hanley-McNeil test[10]. Next, the optimal cutoff-points for each score were determined. Specificity was defined as the maximum difference between sensitivity and 1. These cutoff-points were used to calculate the prognostic sensitivity and specificity (confidence intervals set at 95%). The McNemar test was used to compare these parameters in the two scores. Score calibration was evaluated by the Hosmer-Lemeshow test and by the scatter plot of predicted mortality by risk deciles versus observed mortality.

Figure 1

Statistical analysis diagram.

In a secondary analysis, score performance was assessed on the basis of cutoff-points used in the clinical practice, which in turn were derived from the studies that developed these scores. For this analysis we defined high risk of in-hospital death as a likelihood of > 8%, which is above the mortality value reported for patients with ST-segment elevation myocardial infarction[2,6]. Low risk of death was defined as a likelihood of < 3%, which is below 50% of the mortality reported. Based on these definitions, the strata that define low, intermediate, and high risk in the TIMI score are < 3, 3-4, and > 4, respectively. In the GRACE score, the risk strata correspond to < 141, 141-172, and > 172, respectively. Initially, the prevalence of patients in each risk stratum was described for each score. Subsequently, the agreement between the scores in relation to the definition of these three risk strata was assessed using the Kappa test and the Kendall correlation coefficient. Next, considering the definition of high risk as outcome predictor, we described sensitivity, specificity, and odds ratios. The Youden index was used to evaluate overall accuracy and is calculated as (sensitivity + specificity) - 1; a perfect test would have a Youden index of 1. In addition, we used the diagnostic odds ratio, defined by the equation (true positive/false negative)/(false positive/true negative). Finally, in an explanatory phase, the components of each score were compared between the patients who died and those who survived.

For statistical analysis the SPSS software, version 21 was used. The Hanley-McNeil test (comparison of ROC curves) was performed using the MedCalc software, version 12. A value of p < 0.05 was considered statistically significant in all analyses.

Results

Sample characterization

The study included 152 patients, with a mean age of 63 ± 13 years; 72% were men, admitted to hospital within a median of 171 minutes [interquartile interval (IIQ) = 66-402] after the onset of symptoms. Of these, 78% were subjected to reperfusion in a timely manner, the majority via primary coronary intervention. The median door-to-balloon time for primary intervention was 132 minutes (IIQ = 95-175). Other characteristics are described in Table 1. Sixteen patients died during hospitalization (lethality of 11%). The causes of death were cardiogenic shock in 50% of cases, arrhythmic death in 12.5%, renal dysfunction post cardiac catheterization in 25% (in patients with hypoperfusion), and infection in 12.5%.

Table 1

Sample characteristics

Variables	Values
Sample Size	152
Age (years)	63 ± 13
Male sex	110 (72%)
Weight (kg)	76 ± 12
Previous infarction	75 (49%)
Symptom-to-door time (minutes)*	171 (66-402)
Door-to-balloon time (minutes)*	132 (95-175)
Absent or >4-hour reperfusion therapy	96 (63%)
Killip class	1.3 ± 0.78
Killip I	124 (82%)
Killip II	9 (5.9%)
Killip III	14 (9.2%)
Killip IV	5 (3.3%)
Pro-BNP (pg/ml)*	712 (274-1817)
Elevated pro-BNP	67 (44%)
LV ejection fraction <45%	35 (23%)
Tri-arterial disease or left coronary artery	70/147 (48%)
Serum creatinine (mg/dl)	1.1 ± 0.81
Heart rate (bpm)	81 ± 18
Systolic arterial pressure (mmHg)	150 ± 31
Previous coronary disease	26 (17%)
Diabetes	49 (32%)
Systemic arterial hypertension	108 (71%)
Reperfusion therapy
Absence of indication due to late arrival	17 (11%)
Indication for reperfusion	135 (89%)
Primary angioplasty	104 (77%)
Thrombolysis	15 (11%)
Coronarography without indication for angioplasty	9 (6.7%)
Absence of strategy due to diagnostic error	7 (5.2%)
Adjuvant pharmacological therapy
Aspirin	150 (99%)
Clopidrogrel or ticagrelor	143 (94%)
Statin	143 (94%)
ACE inhibitor	111 (73%)
Beta blocker	99 (65%)
Risk scores
GRACE score	116 ± 36
TIMI score	3.7 ± 2.3

'Variables expressed as median and interquartile interval. The remaining numerical variables are expressed as mean and standard deviation. Elevated pro-BNP: > 450 pg/ml if age < 50 years; > 900 pg/ml if age ≥ 50 years; If distribution não normal, numerical variables expressed as median (interquartile interval). LV: Left ventricle; ACE: Angiotensin converting enzyme.

The values of the TIMI and GRACE scores followed normal distribution. The mean TIMI score was 3.7 ± 2.3, which corresponded to a mortality of 7%[6] and indicated intermediate risk, whereas the mean GRACE score was 116 ± 36, which corresponded to a mortality of 1.6%[2] and suggested low risk. We used the cutoff-points determined by the score validation studies to define low (< 3%), intermediate (3%-8%), and high (> 8%) risk, and we observed that the prevalence of these risk strata according to TIMI were 39%, 27%, and 34%, respectively, which showed a balanced distribution of patients across risk strata. On the other hand, the distribution of patients according to the cutoff-points of the GRACE score was asymmetric, with 80% in the low-risk stratum, 13% in the intermediate-risk stratum, and 7% in the high-risk stratum (Figure 2).

Figure 2

Distribution of the risk strata according to the TIMI and GRACE scores.

Prognostic Accuracy of Scores

The analysis of the discriminatory capacity with regard to the outcome in-hospital death showed similarity between the TIMI and GRACE scores, represented by C-statistics of 0.867 (95% CI = 0.76-0.98) and 0.871 (95% CI = 0.75 - 0.99), respectively, which indicated good accuracy in both cases (p = 0.71; Figure 3). In our sample, the optimal cutoff-points according to the analysis of the ROC curve were TIMI > 5 and GRACE >170. Based on these values, the two scores exhibited similar sensitivities: 75% (95% CI = 48%-93%) for the TIMI score and 81% (95% CI = 54%-96%) for the GRACE score (p = 1.0). Similarly, the specificities were 86% (95% CI = 79%-91%) and 88% (95% CI = 81%-93%), respectively (p = 0.82; Table 2).

Figure 3

ROC curves for the TIMI and GRACE scores for predicting in-hospital death. Area under the curve for the TIMI score = 0.867 and area under the curve for the GRACE score = 0.871 (p = 0.71).

Table 2

Comparison between the TIMI and GRACE scores with regard to discriminatory capacity, calibration, and accuracy according to cutoff-points

	TIMI Score	GRACE Score	p Value*
Discrimination
C-Statistics (95% CI)	0.867 (0.76-0.98)	0.871 (0.5-0.99)	0.71
Calibration
X2 Hosmer-Lemeshow	1.4	14
p Value	0.92	0.08
Optimal Cutoff-Point†	TIMI >5	GRACE >142
Sensitivity (95% CI)	75% (48%-93%)	81% (54%-96%)	1.0
Specificity (95% CI)	86% (79%-91%)	88% (81%-93%)	0.82
Positive Odds Ratio	5.4	6.5
Negative Odds Ratio	0.29	0.21
Definition of High Risk (> 8%)‡	TIMI > 4	GRACE >170
Sensitivity (95% CI)	88% (71%-100%)	50% (26%-75%)	< 0.001
Specificity (95% CI)	72% (65%-80%)	98% (97%-100%)	0.03
Positive Odds Ratio	3.1	23
Negative Odds Ratio	0.17	0.51

Optimal cutoff-point: best accuracy determined by the ROC curve.

Cutoff-point of the validation studies that defined risk of death > 8%, with GRACE score > 170 and TIMI score > 4.

Statistical comparison between the two scores using the Hanley-McNeil test for the C-statistics and the McNemar test for sensitivity and specificity. CI: Confidence interval.

The TIMI score exhibited satisfactory calibration, represented by χ2 of 1.4 using the Hosmer-Lemeshow test (p = 0.92), well better than the calibration of the GRACE score, which was χ2 de 14 (p = 0.08). Figure 4 depicts the scatter plots of estimated probability versus observed mortality by risk deciles. TIMI score data points showed an ascending distribution pattern, close to the identity line, which was not the case for the GRACE score; this demonstrated that TIMI calibration was better than the GRACE calibration.

Figure 4

Correlation between the mortality predicted by the TIMI and GRACE scores and the observed mortality (calibration analysis).

Analysis of the Prevalidated Cutoff-Points

The two scores categorized 59% of patients in the same risk stratum (low, intermediate, or high), corresponding to a kappa of 0.32 (95% CI = 0.24-0.41), which indicated discrete agreement. The Kendall correlation coefficient was 0.58 (95% CI = 0.49-0.66).

The analysis of death incidence in the low-, intermediate-, and high-risk strata showed an increase in mortality according to the TIMI score (0%, 4.9%, and 25%, respectively). With regard to the GRACE score, the increase in mortality from the low-risk stratum to the intermediate-risk stratum was excessive (2.4%, 25%, and 73%, respectively), which inadequately characterized the predefined stratum as intermediate risk.

In a dichotomic analysis, the definition of high risk according to the TIMI score exhibited sensitivity of 88% (95% CI = 71% - 100%) in predicting which patients would present the outcome. This value was significantly higher than that of the GRACE score (sensitivity 50%; 95% CI = 26%-75%; p < 0.001). On the other hand, the specificity of TIMI was 72% (95% CI = 65%-80%), moderately lower than that of GRACE (specificity 98%; 95% CI = 97%-100%; p = 0.03). These values of sensitivity and specificity resulted in a positive odds ratios and negative odds ratio of 3.1 and 0.17 for TIMI and 23 and 0.51 for GRACE, respectively. Thus, positive and negative TIMI results influence the risk of death, whereas only positive results of GRACE have a significant influence. With regard to overall accuracy, the Youden indexes of the TIMI and GRACE scores were 0.6 and 0.48, respectively. The diagnostic odds ratio of TIMI (42) was higher than that of GRACE (19). Therefore, the accuracy of TIMI using prevalidated cutoff-points was higher than that of GRACE.

Exploratory Analysis of Score Components

Of the four variables that are common to both scores, three exhibited higher values (age, arterial pressure, Killip) in patients who died than in those who survived. Heart rate values were similar in these two groups of patients.

Of the four variables exclusive to TIMI, three (previous infarction, late reperfusion, and risk factors ≥ 3) were more frequent in patients who died than in those who survived. Weight (<67 kg) was similar in both groups. Of the four exclusive variables of GRACE, only creatinine differed between the groups. ST-segment deviation and positive necrosis marker are universal characteristics of patients with ST-segment elevation myocardial infarction. The fourth variable, death after hospital admission, was not observed in any patient. These data are shown in Table 3.

Table 3

Comparison of score components between patients who died and patients who survived

Variables	Death	Survival
Sample Size	16	136
Common variables
Age (years)	77 ± 13	61 ± 12
Systolic Arterial Pressure (mmHg)	129 ± 38	152 ± 29
Killip	2.4 ± 1.3	1.2 ± 0.59
Heart Rate (bpm)	82 ± 22	80 ± 17
Variables Exclusive to TIMI
Previous Infarction	5 (31%)	27 (20%)
Risk Factors ≥3	15 (94%)	106 (78%)
Reperfusion >4 hours	12 (75%)	84 (62%)
Weight < 67 kg	3 (19%)	30 (22%)
Variables Exclusive to GRACE
Serum Creatinine (mg/dl)	1.35 ± 0,69	1.06 ± 0,82
ST segment deviation	16 (100%)	16 (100%)
Elevation of necrosis marker	16 (100%)	136 (100%)
Cardiac arrest	0	0

Discussion

In the present study we compared the performance of the two main scores, for predicting death, applied to patients with ST-segment elevation myocardial infarction. Our primary analysis considered all the ranges of numerical values of the scores and demonstrated similar discriminatory ability. However, TIMI score calibration was better than that of GRACE. As a result, the accuracy of TIMI was better when the cutoff-points predefined in the score validation studies were used.

Our exploratory analysis explains these findings: the TIMI score has more variables associated with death. This may result from the fact that, unlike GRACE, TIMI was derived from a specific sample of patients with ST-segment elevation myocardial infraction. The combination of better accuracy (although not excessive) and greater simplicity of calculation led us to deem TIMI as the preferred score in this clinical scenario.

It is worth revising the significance of the prognostic factors under study: discrimination and calibration. Discriminatory capacity (C-statistics) describes the score's ability to individually distinguish patients who will experience the outcome from patients who will not. Calibration, on the other hand, relates to the score's ability to predict mortality in a given population. It is a collective estimate rather than an individual one. Therefore, one might expect the TIMI score, which is derived from a sample of patients with ST-segment elevation myocardial infarction, to be better calibrated than GRACE, which was derived from a sample with only 1/3 of patients with this condition.

The similar discriminatory ability observed in our primary analysis suggests that the scores are interchangeable with regard to distinguishing the patients who will experience the outcome from those who will not, provided the cutoff-points are adapted to our population. On the other hand, the calibration analysis indicated that the performance of the TIMI score with regard to estimation of probability of outcome is better. By using a basal probability to calculate the absolute impact of a clinical decision, this estimate would prove useful for clinical decision trees. We acknowledge that, in ST-segment elevation myocardial infarction, risk prediction does not modify the initial treatment because the indication for reperfusion is universal. Still, a well calibrated score has practical advantages: low-risk patients subjected to primary intervention can be discharged earlier; beta blockers should be used carefully used in high-risk patients who tend to be hemodynamically unstable; complete revascularization via percutaneous intervention may be better than simply intervening on the involved vessel in high-risk patients[11] (this hypothesis needs to be tested in large randomized studies). Second, risk estimation is useful for assistance quality control, namely by comparing the predicted mortality in a population (score) with the observed mortality in our sample.

Another aspect that suggested greater usefulness of TIMI was the balanced distribution of patients in the low-, intermediate-, and high-risk strata, when considering the cutoff-points used in the clinical practice. This allows applying differentiated treatments to a larger number of patients. In contrast, the GRACE score classified almost all patients as low risk, which would hinder differentiated risk-based treatment.

Recently, Kozieradzka et al[12] compared these scores in ST-segment elevation myocardial infarction and observed similar discriminatory ability. However, these authors did not analyze score calibration. In addition, that study analyzed long-term outcomes (five-year evolution)[12]. We, on the other hand, focused our approach on hospital outcomes, for two reasons: first, these scores were originally created and validated for this type of prediction; second, the prediction of hospital outcomes has greater ability to influence acute phase treatment. This characteristic and our calibration analysis represent the original contribution of our study.

Next, we discuss the significance of the relatively high death incidence (11%) in our sample. As shown in Figure 4 (calibration graph), the two deciles of patients who experienced higher mortality are the same patients whose TIMI and GRACE scores predicted high mortality. On the other hand, the graph shows that mortality was low in the other eight deciles of predicted low risk. That is, when we demonstrated that the calibration was good, mortality was high among high-risk patients (according to the scores) and was low among low-risk patients. Therefore, it is evident that the observed mortality was the result of the characteristics of some high-risk subgroups. This is the statistical explanation for the observed mortality. In line with this idea, Table 3 indicates that the severity characteristics of both scores were more present in the patients who died. Moreover, it should be noted that our sample was limited to patients with ST-segment elevation myocardial infarction, a group in which mortality was expected to be higher than that in the overall GRACE cohort.

On this subject it should be said that the calibration of a prediction model can vary with the incidence of the outcome in the population. Therefore, our findings cannot be extrapolated with a high degree of certainty to populations that are at a significantly different risk for outcome.

With regard to the high frequency of late reperfusion, it should be noted that this reperfusion time does not refer to door-to-balloon time but rather the time elapsed between onset of symptoms and reperfusion. Thus, it is expected that most patients exhibit a prolonged reperfusion time. The original work on the TIMI score (which includes this variable) used a sample from a randomized clinical trial, i.e., a selected sample, which may explain the favorable rates of early reperfusion.

Ideally, the evaluation of a risk marker should not be performed in a context where this marker is used by the team who cares for the patient because this leads to the underestimation of the observed predictive value. However, these scores are known and have been adopted in the clinical practice, which makes it impossible to guarantee that they are not used in the current registries. Although we acknowledge this limitation, we note that it probably occurred with the two scores. This lessens this potential bias because the main objective was to compare the scores.

Conclusion

The discriminatory abilities of the TIMI and GRACE scores were similar; however, the calibration of TIMI was better than that of GRACE. This result, combined with the greater simplicity of the TIMI score, makes this the preferred score in patients with ST-segment elevation myocardial infarction. In future studies, this conclusion should be validated for populations at different risks for the outcome.