Indirect comparison of TIMI, HEART and GRACE for predicting major cardiovascular events in patients admitted to the emergency department with acute chest pain: a systematic review and meta-analysis.

BACKGROUND: The study aimed to compare the predictive values of the thrombolysis in myocardial infarction (TIMI); History, Electrocardiography, Age, Risk factors and Troponin (HEART) and Global Registry in Acute Coronary Events (GRACE) scoring systems for major adverse cardiovascular events (MACEs) in acute chest pain (ACP) patients admitted to the emergency department (ED).
METHODS: We systematically searched PubMed, Embase and the Cochrane Library from their inception to June 2020; we compared the following parameters: sensitivity, specificity, positive and negative likelihood ratios (PLR and NLR), diagnostic OR (DOR) and area under the receiver operating characteristic curves (AUC).
RESULTS: The pooled sensitivity and specificity for TIMI, HEART and GRACE were 0.95 and 0.36, 0.96 and 0.50, and 0.78 and 0.56, respectively. The pooled PLR and NLR for TIMI, HEART and GRACE were 1.49 and 0.13, 1.94 and 0.08, and 1.77 and 0.40, respectively. The pooled DOR for TIMI, HEART and GRACE was 9.18, 17.92 and 4.00, respectively. The AUC for TIMI, HEART and GRACE was 0.80, 0.80 and 0.70, respectively. Finally, the results of indirect comparison suggested the superiority of values of TIMI and HEART to those of GRACE for predicting MACEs, while there were no significant differences between TIMI and HEART for predicting MACEs.
CONCLUSIONS: TIMI and HEART were superior to GRACE for predicting MACE risk in ACP patients admitted to the ED. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The analysis is based on prospective studies and used consistent cut-off values.

The pooled results were stable owing to a large sample size.

The indirect comparisons among thrombolysis in myocardial infarction, History, Electrocardiography, Age, Risk factors and Troponin and Global Registry in Acute Coronary Events scoring systems were provided.

The analysis is based on crude data; the predictive values could be affected by covariates.

Substantial heterogeneity was not fully explained.

Introduction

Acute chest pain (ACP) is a common symptom accounting for a significant proportion of attendance and burden in the emergency department (ED).1 ACP patients require effective risk stratification to ensure timely initiation of proper treatment in high-risk cases to achieve better prognoses. The early identification of cardiovascular disease (CVD) in ACP patients is important although CVD accounts for only a small proportion of ACP patients with ECGs on presentation.2–4 Moreover, patients diagnosed with acute coronary syndrome (ACS) should remain hospitalised, whereas non-ACS patients are unnecessarily admitted to hospitals due to the heavy burden on resource constraints.5 Therefore, accurate risk stratification for ACP patients is essential to improve hospital efficacy by administering timely interventions to high-risk patients, avoiding unnecessary tests and minimising admissions for low-risk patients.

Currently, the thrombolysis in myocardial infarction (TIMI); History, ECG, Age, Risk factors and Troponin (HEART) and Global Registry in Acute Coronary Events (GRACE) scores are widely used for the risk stratification of ACP patients; however, the predictive values using these methods on major adverse cardiovascular events (MACEs) have not been elucidated. The TIMI scoring system was established in 2000 for evaluating patients with unstable angina or non-ST-segment elevation myocardial infarction.6–10 The HEART score, developed in 2008, aims to improve the accuracy of diagnosing ACS for patients with undifferentiated chest pain.11 The GRACE score was developed in 2001 for adults with symptoms of ACS; it comprises the following factors: age, vital signs, kidney function, ECG and troponin levels.12 13 However, the predictive values of risk stratification measured by the TIMI, HEART and GRACE scoring systems on MACEs have not been fully compared. Therefore, this study was conducted based on prospective cohort studies to provide comprehensive results regarding the risk stratification assessed by the TIMI, HEART and GRACE scoring systems on MACEs in ACP patients admitted to the ED. Furthermore, the predictive values of risk stratification assessed by the TIMI, HEART and GRACE scoring systems on MACEs were compared through an indirect analytic approach.

Methods

Data sources, search strategy and selection criteria

This study was conducted and reported following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Statement issued in 2009.14 Any prospective cohort studies investigating the predictive value of TIMI, HEART and GRACE on MACEs in ACP patients were eligible for inclusion in this study. No restrictions were placed on publishing language and status. The electronic databases of PubMed, Embase and the Cochrane library were systematically searched for studies from their inception up to June 2020, and the search strategy was performed using the following terms with Medical Subject Heading and free words: (“TIMI” or “HEART” or “GRACE”) and “emergency department” and “chest pain” and (“prospective” or “cohort”). The search strategy details are summarised in online supplemental file. The reference lists of retrieved studies were also searched manually to find new eligible studies.

Two authors independently performed the literature search and study selection; any conflicts were resolved through group discussion until a consensus was reached. A study was included if they met the following inclusion criteria: (1) study design: the study had a prospective design; (2) patients: ACP patients admitted to the ED; (3) risk stratifying tools: TIMI, HEART or GRACE; (4) outcomes: the study had to report the incidence of MACEs and provided clear definitions of MACEs; (5) data abstracted: true and false positives or negatives, or data could transform into the above information must be reported and (6) cut-off value: the cut-off value of TIMI and HEART was 0–3, and the cut-off value of GRACE was 55–110. Retrospective studies were excluded due to various confounding factors. Additionally, studies that used other cut-off values were excluded.

Data collection and quality assessment

Two authors independently abstracted data items and assessed the quality of the included studies, and any disagreement was settled by an additional author reviewing the original article. The collected information from retrieved studies including the first author’s name, publication year, country, sample size, age at baseline, percentage of males, risk stratifying tools, patients’ status, MACE definition, follow-up duration and true and false positives/negatives. Study quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2), which is based on patient selection, index test, reference standard, risk of bias and concerns about applicability.15

Statistical analysis

The pooled sensitivity, specificity, positive likelihood ratio (PLR), negative likelihood ratio (NLR) and the area under the receiver operating characteristic curves (AUC) for the risk stratification by TIMI, HEART and GRACE on MACEs were calculated using a bivariate generalised linear mixed model,16 17 while the pooled diagnostic OR (DOR) was calculated using a random-effects model.18 The I2 and Q statistic were used for assessing the heterogeneity across included studies, and p<0.10 was considered as significant heterogeneity.19 20 The robustness of pooled results was also assessed by sensitivity analyses, which were also performed for studies using all three scoring systems and the endpoint MACE.21 TIMI-based, HEART-based and GRACE-based risk stratification were assessed using an indirect comparison analysis, and the ratios among these scoring systems were calculated.22 Furthermore, subgroup analyses for the predictive values of TIMI, HEART and GRACE on subsequent MACE risk were also estimated based on country, mean age, percentage of males, follow-up duration and study quality. The funnel plots and Deeks’ asymmetry tests were used to assess publication bias.23 24 The inspection level for pooled diagnostic parameters was two sided, and p<0.05 was considered statistically significant. All statistical analyses were performed using Stata software (V.10.0; Stata).

Patient and public involvement

There was no patient or public involvement in the design or conduct of this study.

Results

Literature search

The details regarding the literature search and study selection of eligible studies are presented in figure 1. A total of 2794 articles were identified through the electronic search from PubMed, Embase and the Cochrane library, and 1981 were excluded because of term duplications. Subsequently, the remaining 813 studies were selected through title and abstract review; 732 were excluded based on irrelevance. A total of 81 full texts were retrieved for further evaluation, and 48 studies were excluded due to the following reasons: used other cut-off values (n=21), retrospective study design (n=14) and insufficient data (n=13). An additional 135 potential studies identified from the reference lists of retrieved studies were excluded because of duplication with the electronic search. Subsequently, 33 prospective cohort studies that recruited 40 262 ACP patients were selected for final quantitative meta-analysis.13 25–56

Figure 1

Flow diagram of the literature search and study selection process.

Study characteristics

The baseline characteristics of included studies are summarised in table 1. The retrieved studies were published from 2005 to 2020, and 255–4333 ACP patients were included in each study. Nine studies were conducted in Eastern countries, and the remaining 24 studies were conducted in Western countries. The mean age of enrolled patients ranged from 48.0 to 69.0 years, and the percentage of males ranged from 40.0% to 68.8%. Risk stratification by the TIMI score was available in 25 studies published between 2005 and 2020, 16 studies used the HEART score and were published between 2013 and 2020 and and 16 studies employed GRACE and were published between 2007 and 2020.55 The definition of MACEs across included studies contained all-cause death, cardiac death, myocardial infarction, revascularisation, cardiac arrest, cardiogenic shock, unstable angina, ACS, percutaneous coronary intervention, coronary artery bypass grafting, coronary angiography revealing procedurally correctable stenosis managed conservatively, ventricular arrhythmia needing intervention, high-degree atrioventricular block needing intervention and life-threatening arrhythmias requiring emergency intervention. The study quality of the included studies was assessed by QUADAS-2 (figure 2).

Table 1

Baseline characteristics of studies included in the systematic review and meta-analysis

Study	Country	Sample size	Mean age (years)	Percentage male (%)	Risk stratifying tools	Patients’ status	MACE definition	Follow-up
Tong 2005 25	USA	957	60.0	52.0	TIMI	Chest pain and a nondiagnostic ECG	Death and MI	30 days
Sanchis 200526	Spain	646	64.0	65.8	TIMI	Acute chest pain	Death, MI or urgent revascularisation	14 days
Pollack 200627	USA	3929	51.6	40.0	TIMI	Chest pain in the ED	Death, acute MI and revascularisation	30 days
Pelliccia 200628	Italy	4333	58.4	68.8	TIMI	Acute chest pain	MI	In-hospital
Lyon 200729	UK	954	60.0	62.0	TIMI, GRACE	Undifferentiated chest pain	MI, cardiac arrest, revascularisation, unstable angina with myocardial damage and death	30 days
Ramsay 200713	UK	347	65.2	62.3	TIMI, GRACE	Suspected cardiac pain	Death, non-fatal MI and emergency revascularisation	3.0 months
Body 200930	UK	796	58.9	60.4	TIMI	Chest pain in the ED	Death, acute MI or urgent coronary revascularisation	30 days
Campbell 200931	USA	3169	53.6	45.0	TIMI	Chest pain in the ED	Death, MI or revascularisation	30 days
Hess 201032	Canada	1017	59.3	60.6	TIMI	ED patients with chest pain and possible ACS	Acute MI, revascularisation or death	30 days
Stracke 201033	Germany	1014	66.0	55.0	GRACE	Chest pain in the ED	Death	In-hospital
van der Zee 201134	The Netherlands	524	57.7	60.5	GRACE	Chest pain in the ED	Death	9.4 years
Graham 201335	China	315	69.0	54.9	TIMI	Chest pain in the ED	Death, MI, troponin positive ACS andPCI	30 days
Holly 201336	USA	552	54.1	46.0	TIMI	Chest pain in the ED	MI, revascularisation or death	30 days
Backus 201337	The Netherlands	2388	60.6	57.5	HEART	Chest pain in the ED	Acute MI, PCI, CABG, coronary angiography revealing procedurally correctable stenosis managed conservatively and death	6 weeks
Cullen 201338	Australia	948	54.0	59.9	TIMI, and GRACE	Chest pain in the ED	Cardiac death, acute MI and unstable angina	30 days
Graham 201439	China	925	68.0	51.7	TIMI	Chest pain in the ED	Death, readmission with MI, ACS not diagnosed at initial ED presentation and coronary revascularisation	30 days
Visser 201540	The Netherlands	255	64.0	56.0	HEART	Chest pain in the ED	MI, or PCI, or CABG, or coronary angiography revealing significant stenosis or death	6 weeks
Boubaker 201541	Tunisia	3125	57.7	58.3	TIMI, GRACE	Chest pain in the ED	All-cause mortality, ACS and coronary non-ED planned revascularisation	30 days
Wang 201642	China	986	54.0	55.0	TIMI, HEART, GRACE	Chest pain in the ED	Death, MI and/or the need for revascularisation by CABG or PCI	6 months
Chen 201643	China	833	65.1	55.3	TIMI, HEART, GRACE	Chest pain in the ED	Coronary revascularisation, ventricular arrhythmia needing intervention and high-degree atrioventricular block needing intervention	30 days
Sakamoto 201644	Singapore	604	60.8	69.2	TIMI, HEART, GRACE	Chest pain in the ED	Death, acute MI, PCI, CABG	30 days
Leung 201745	China	602	66.0	48.8	TIMI, HEART	Chest pain in the ED	Death, cardiac arrest, MI and cardiogenic shock	30 days
Poldervaart 201746	The Netherlands	1748	62.0	54.0	TIMI, HEART, GRACE	Chest pain in the ED	UA, MI, PCI, CABG, stenosis managed conservatively and death	3.0 months
McCord 201747	Europe, Australia and the USA	661	58.3	58.2	HEART	Chest pain in the ED	Death or acute MI	30 days
Reaney 201848	UK	1000	62.5	57.6	TIMI, HEART, GRACE	Chest pain in the ED	Acute MI, PCI, CABG, cardiac death, cardiogenic shock and life-threatening arrhythmias requiring emergency intervention.	30 days
Greenslade 201849	Australia	1760	60.4	59.3	TIMI	Chest pain in the ED	Cardiac death, cardiac arrest, cardiogenic shock, acute MI, UA, emergency or urgent revascularisation, high-level atrioventricular block and ventricular arrhythmias	30 days
Moumneh 201850	France	641	53.3	53.4	HEART	Non-traumatic chest pain patients	MI, coronary angioplasty, coronary bypass and sudden unexplained death	6 weeks
Ishak 201851	The Netherlands	1127	63.8	57.7	HEART	Chest pain in the ED	Acute MI, PCI, CABG or death	30 days
Wong 201852	China	1081	48.0	52.3	TIMI, HEART, GRACE	Undifferentiated chest pain	Acute MI, PCI, CABG and death	30 days
Al-Zaiti 201953	USA	750	59.0	58.0	TIMI, HEART, GRACE	Chest pain in the ED	Resuscitated or unresuscitated sudden cardiac arrest, all-cause death, postdischarge reinfarction requiring cardiac revascularisation	30 days
Huang 202054	China	509	59.8	53.4	HEART, GRACE	Chest pain in the ED	Acute MI, PCI, CABG, cardiac death, cardiogenic shock or life-threatening arrhythmias requiring emergency intervention or resulting in mortality	30 days
Torralba 202055	Colombia	519	64.3	56.1	TIMI, HEART, GRACE	Chest pain in the ED	Death from any cause, MI and surgical or percutaneous myocardial revascularisation	30 days
Shin 202056	Korea	1247	62.0	60.8	TIMI, HEART, GRACE	Chest pain in the ED	Acute MI, PCI, CABG or death from cardiac causes	30 days

ACS, acute coronary syndrome; CABG, coronary artery bypass grafting; ED, emergency department; GRACE, Global Registry in Acute Coronary Events; HEART, History, Electrocardiography, Age, Risk factors, and Troponin; MI, myocardial infarction; PCI, percutaneous coronary intervention; TIMI, thrombolysis in myocardial infarction; UA, unstable angina.

Figure 2

QUADAS-2 scoring of included studies. QUADAS-2, Quality Assessment of Diagnostic Accuracy Studies 2.

Thrombolysis in myocardial infarction

The predictive value of risk stratification by the TIMI score on MACEs in ACP patients was available in 25 studies. The pooled sensitivity and specificity of the TIMI score for predicting MACEs were 0.95 (95% CI: 0.91 to 0.98; I2=98.10%) and 0.36 (95% CI: 0.24 to 0.50; I2=99.64%), respectively (online supplemental file). Moreover, the pooled PLR and NLR of the TIMI score for predicting MACEs were 1.49 (95% CI: 1.25 to 1.79; I2=99.21%) and 0.13 (95% CI: 0.07 to 0.21; I2=95.38%), respectively (online supplemental file). The pooled DOR of the TIMI score for predicting MACEs was 9.18 (95% CI: 6.22 to 13.55; p<0.001) with significant heterogeneity across the included studies (I2=87.6%; p<0.001) (online supplemental file). The AUC of the TIMI score for predicting MACEs was 0.80 (95% CI: 0.76 to 0.83; figure 3). No significant publication bias for the TIMI score was detected (p=0.17; online supplemental file).

Figure 3

The summary receiver operating characteristic curve (SROC) of risk stratification assessed by the TIMI score. AUC, area under the curve; TIMI, thrombolysis in myocardial infarction.

History, electrocardiography, age, risk factors and troponin

The predictive value of risk stratification by the HEART score on MACEs in ACP patients was available in 16 studies. The pooled sensitivity and specificity of the HEART score for predicting MACEs were 0.96 (95% CI: 0.91 to 0.98; I2=94.87%) and 0.50 (95% CI: 0.41 to 0.60; I2=98.84%), respectively (online supplemental file). The pooled PLR and NLR of the HEART score for predicting MACEs were 1.94 (95% CI: 1.61 to 2.35; I2=98.01%) and 0.08 (95% CI: 0.03 to 0.17; I=94.65%), respectively (online supplemental file). The pooled DOR for the HEART score was 17.92 (95% CI: 9.40 to 34.18; p<0.001) with significant heterogeneity across the included studies (I2=88.9%; p<0.001) (online supplemental file). The AUC of the HEART score for predicting MACEs was 0.80 (95% CI: 0.77 to 0.84; figure 4). There was no significant publication bias for the HEART score (p=0.98; online supplemental file).

Figure 4

The summary receiver operating characteristic curve (SROC) of risk stratification assessed by the HEART score. AUC, area under the curve; HEART, History, Electrocardiography, Age, Risk factors and Troponin.

Global Registry in Acute Coronary Events

The predictive value of risk stratification by the GRACE score on MACEs in ACP patients was available in 16 studies. The pooled sensitivity and specificity of the GRACE score for predicting MACEs were 0.78 (95% CI: 0.64 to 0.87; I2=96.78%) and 0.56 (95% CI: 0.46 to 0.66; I2=99.39%), respectively (online supplemental file). The pooled PLR and NLR of the GRACE score for predicting MACEs were 1.77 (95% CI: 1.51 to 2.08; I=96.34%) and 0.12 (95% CI: 0.06 to 0.26; I=94.07%), respectively (online supplemental file). The DOR of the GRACE score for predicting MACEs was 4.00 (95% CI: 2.78 to 5.74; p<0.001) with significant heterogeneity across the included studies (I2=88.7%; p<0.001) (online supplemental file). The AUC of the GRACE score for predicting MACEs was 0.70 (95% CI: 0.66 to 0.74; figure 5). No significant publication bias for the GRACE score was observed (p=0.36; online supplemental file).

Figure 5

The summary receiver operating characteristic curve (SROC) of risk stratification assessed by the grace score. AUC, area under the curve.

Indirect comparisons

Indirect comparisons of the diagnostic parameters (sensitivity, specificity, PLR, NLR, DOR and AUC) among the TIMI, HEART and GRACE scoring systems for predicting MACEs are summarised in table 2. First, the sensitivity of TIMI (ratio: 1.22; 95% CI: 1.04 to 1.43) and HEART (ratio: 1.23; 95% CI: 1.05 to 1.44) was significantly higher than that of GRACE for predicting MACEs. Second, the specificity of TIMI was significantly lower than that of GRACE for predicting MACEs (ratio: 0.64; 95% CI: 0.43 to 0.97). Third, the PLR of TIMI was lower than that of HEART for predicting MACEs (ratio: 0.77; 95% CI: 0.59 to 1.00). Fourth, TIMI (ratio: 0.32; 95% CI: 0.17 to 0.64) and HEART (ratio: 0.20; 95% CI: 0.08 to 0.52) were associated with a lower NLR than GRACE for predicting MACEs. Fifth, the DOR of TIMI (ratio: 2.29; 95% CI: 1.35 to 3.91) and HEART (ratio: 4.48; 95% CI: 2.14 to 9.39) was significantly higher than that of GRACE for predicting MACEs. Finally, the AUC of TIMI (ratio: 1.14; 95% CI: 1.06 to 1.23) and HEART (ratio: 1.14; 95% CI: 1.06 to 1.23) was significantly higher than that of GRACE for predicting MACEs.

Table 2

Systematic comparisons of the sensitivity, specificity, PLR and NLR, DOR and the AUC of risk stratifying measured by TIMI, HEART and GRACE

Parameters	TIMI	HEART	GRACE	TIMI versus HEART	TIMI versus GRACE	HEART versus GRACE
Sensitivity and 95% CI	0.95 (0.91–0.98)	0.96 (0.91–0.98)	0.78 (0.64–0.87)	0.99 (0.94–1.04)	1.22 (1.04–1.43)	1.23 (1.05–1.44)
Specificity and 95% CI	0.36 (0.24–0.50)	0.50 (0.41–0.60)	0.56 (0.46–0.66)	0.72 (0.48–1.09)	0.64 (0.43–0.97)	0.89 (0.69–1.16)
PLR and 95% CI	1.49 (1.25–1.79)	1.94 (1.61–2.35)	1.77 (1.51–2.08)	0.77 (0.59–1.00)	0.84 (0.66–1.07)	1.10 (0.86–1.40)
NLR and 95% CI	0.13 (0.07–0.21)	0.08 (0.03–0.17)	0.40 (0.27–0.59)	0.46 (0.15–1.39)	0.32 (0.17–0.64)	0.20 (0.08–0.52)
DOR and 95% CI	9.18 (6.22–13.55)	17.92 (9.40–34.18)	4.00 (2.78–5.74)	0.51 (0.24–1.09)	2.29 (1.35–3.91)	4.48 (2.14–9.39)
AUC and 95% CI	0.80 (0.76–0.83)	0.80 (0.77–0.84)	0.70 (0.66–0.74)	1.00 (0.94–1.06)	1.14 (1.06–1.23)	1.14 (1.06–1.23)

AUC, the area under the receiver operating characteristic curve; DOR, diagnostic OR; GRACE, Global Registry in Acute Coronary Events; HEART, History, Electrocardiography, Age, Risk factors and Troponin; NLR, negative likelihood ratio; PLR, positive likelihood ratio; TIMI, thrombolysis in myocardial infarction.

Sensitivity and subgroup analysis

The results of sensitivity analyses found that the predictive values of TIMI, HEART and GRACE for predicting MACEs were stable and unaltered by sequential removal of one study from the overall analysis (data not shown). Sensitivity analyses were also performed after removing studies not using all three scoring systems (table 3). We noted that TIMI had lower sensitivity that that of HEART (ratio: 0.86; 95% CI: 0.75 to 0.99). Moreover, TIMI had lower NLR (ratio: 0.55; 95% CI: 0.31 to 0.99), higher DOR (ratio: 2.71; 95% CI: 1.17 to 6.24) and higher AUC (ratio: 1.19; 95% CI: 1.11 to 1.27), compared with GRACE. Furthermore, HEART had higher sensitivity (ratio: 1.32; 95% CI: 1.11 to 1.58), DOR (ratio: 4.36; 95% CI: 1.63 to 11.64) and AUC (ratio: 1.14; 95% CI: 1.06 to 1.23) and lower NLR (ratio: 0.22; 95% CI: 0.09 to 0.56) as compared with GRACE. Additionally, table 4 presents the subgroup analysis results for the predictive values of TIMI, HEART and GRACE. First, TIMI has lower specificity than HEART in the pooled studies from Western countries. Furthermore, TIMI versus HEART showed lower PLR in the pooled studies of Western countries (mean age <60.0 years). Moreover, the DOR of TIMI was lower than of HEART in the pooled studies of Eastern countries, while TIMI had a lower AUC than HEART for mean age <60.0 years. Second, TIMI with higher sensitivity than GRACE when pooled studies conducted in Eastern countries, percentage of males≥60.0%, follow-up duration ≤30.0 days and studies of moderate quality. Moreover, TIMI versus GRACE showed lower specificity if mean age <60.0 years, percentage of males ≥60.0% and follow-up duration ≤30.0 days. Furthermore, TIMI has lower PLR than GRACE if percentage of males ≥60.0%, while TIMI has lower NLR than GRACE in the pooled studies of Eastern countries, irrespective of the mean age or percentage male status, follow-up duration ≤30.0 days, and studies of moderate quality. In addition, TIMI versus GRACE showed higher DOR and AUC in most subgroups. Third, HEART versus GRACE showed higher sensitivity in the pooled studies of Eastern countries, mean age ≥60.0 years, follow-up duration ≤30.0 days and studies of moderate quality. Moreover, HEART has lower NLR than GRACE in most subgroups, except that of mean age <60.0 years. Furthermore, HEART versus GRACE showed higher DOR and AUC in most subgroups.

Table 3

Sensitivity analysis for direct comparisons of the sensitivity, specificity, PLR and NLR, DOR, and the AUC of risk stratifying measured by TIMI, HEART and GRACE

Parameters	TIMI	HEART	GRACE	TIMI versus HEART	TIMI versus GRACE	HEART versus GRACE
Sensitivity and 95% CI	0.81 (0.70–0.89)	0.94 (0.84–0.98)	0.71 (0.58–0.81)	0.86 (0.75–0.99)	1.14 (0.94–1.39)	1.32 (1.11–1.58)
Specificity and 95% CI	0.70 (0.54–0.83)	0.53 (0.39–0.66)	0.59 (0.44–0.73)	1.32 (0.94–1.86)	1.19 (0.85–1.65)	0.90 (0.62–1.29)
PLR and 95% CI	2.72 (1.67–4.41)	1.99 (1.51–2.63)	1.74 (1.33–2.28)	1.37 (0.78–2.39)	1.56 (0.90–2.72)	1.14 (0.78–1.68)
NLR and 95% CI	0.27 (0.16–0.44)	0.11 (0.05–0.28)	0.49 (0.36–0.66)	2.45 (0.90–6.66)	0.55 (0.31–0.99)	0.22 (0.09–0.56)
DOR and 95% CI	9.45 (4.77–18.73)	15.21 (6.45–35.88)	3.49 (2.16–5.63)	0.62 (0.21–1.86)	2.71 (1.17–6.24)	4.36 (1.63–11.64)
AUC and 95% CI	0.83 (0.80–0.86)	0.80 (0.76–0.83)	0.70 (0.66–0.74)	1.04 (0.98–1.10)	1.19 (1.11–1.27)	1.14 (1.06–1.23)

AUC, the area under the receiver operating characteristic curve; DOR, diagnostic OR; GRACE, Global Registry in Acute Coronary Events; HEART, History, Electrocardiography, Age, Risk factors, and Troponin; NLR, negative likelihood ratio; PLR, positive likelihood ratio; TIMI, thrombolysis in myocardial infarction.

Table 4

Subgroup analyses

Parameters	Factors	Groups	TIMI	HEART	GRACE	TIMI versus HEART	TIMI versus GRACE	HEART versus GRACE
Sensitivity	Country	Western	0.97 (0.93–0.99)	0.97 (0.93–0.99)	0.88 (0.72–0.95)	1.00 (0.96–1.05)	1.10 (0.96–1.27)	1.10 (0.96–1.27)
		Eastern	0.86 (0.69–0.95)	0.94 (0.79–0.98)	0.60 (0.49–0.70)	0.91 (0.75–1.11)	1.43 (1.13–1.82)	1.57 (1.27–1.93)
	Mean age (years)	≥60.0	0.97 (0.92–0.99)	0.97 (0.94–0.99)	0.83 (0.67–0.92)	1.00 (0.96–1.05)	1.17 (0.99–1.38)	1.17 (1.00–1.37)
		<60.0	0.93 (0.81–0.97)	0.91 (0.70–0.98)	0.68 (0.44–0.85)	1.02 (0.84–1.24)	1.37 (0.97–1.92)	1.34 (0.92–1.94)
	Percentage male (%)	≥60.0	0.98 (0.92–0.99)	–	0.73 (0.50–0.88)	–	1.34 (1.01–1.79)	–
		<60.0	0.92 (0.85–0.96)	0.96 (0.91–0.99)	0.80 (0.62–0.90)	0.96 (0.89–1.03)	1.15 (0.95–1.40)	1.20 (0.99–1.45)
	Follow-up duration	≤30.0 days	0.96 (0.91–0.98)	0.97 (0.92–0.99)	0.76 (0.61–0.87)	0.99 (0.94–1.04)	1.26 (1.05–1.51)	1.28 (1.06–1.53)
		>30.0 days	–	0.93 (0.78–0.98)	–	–	–	–
	Study quality	High	0.94 (0.83–0.98)	0.97 (0.91–0.99)	0.77 (0.50–0.92)	0.97 (0.88–1.06)	1.22 (0.89–1.67)	1.26 (0.93–1.71)
		Moderate	0.96 (0.90–0.98)	0.95 (0.85–0.98)	0.78 (0.61–0.89)	1.01 (0.93–1.10)	1.23 (1.01–1.49)	1.22 (1.00–1.49)
Specificity	Country	Western	0.33 (0.20–0.50)	0.56 (0.47–0.64)	0.50 (0.34–0.66)	0.59 (0.36–0.96)	0.66 (0.37–1.16)	1.12 (0.78–1.61)
		Eastern	0.46 (0.27–0.67)	0.44 (0.27–0.62)	0.64 (0.55–0.73)	1.05 (0.56–1.94)	0.72 (0.45–1.16)	0.69 (0.44–1.07)
	Mean age (years)	≥60.0	0.35 (0.19–0.55)	0.45 (0.34–0.56)	0.51 (0.37–0.64)	0.78 (0.43–1.40)	0.69 (0.38–1.25)	0.88 (0.61–1.28)
		<60.0	0.38 (0.23–0.56)	0.60 (0.44–0.74)	0.63 (0.49–0.76)	0.63 (0.38–1.06)	0.60 (0.37–0.99)	0.95 (0.68–1.34)
	Percentage male (%)	≥60.0	0.21 (0.10–0.37)	–	0.56 (0.45–0.66)	–	0.38 (0.19–0.74)	–
		<60.0	0.45 (0.30–0.61)	0.52 (0.42–0.63)	0.56 (0.42–0.70)	0.87 (0.58–1.30)	0.80 (0.52–1.24)	0.93 (0.67–1.29)
	Follow-up duration	≤30.0 days	0.35 (0.23–0.50)	0.46 (0.35–0.57)	0.56 (0.45–0.67)	0.76 (0.48–1.20)	0.63 (0.40–0.97)	0.82 (0.60–1.13)
		>30.0 days	–	0.60 (0.43–0.75)	–	–	–	–
	Study quality	High	0.41 (0.24–0.62)	0.47 (0.35–0.60)	0.61 (0.36–0.81)	0.87 (0.51–1.51)	0.67 (0.36–1.25)	0.77 (0.47–1.25)
		Moderate	0.33 (0.19–0.51)	0.54 (0.39–0.68)	0.54 (0.44–0.64)	0.61 (0.35–1.08)	0.61 (0.36–1.04)	1.00 (0.72–1.40)
PLR	Country	Western	1.45 (1.16–1.80)	2.20 (1.79–2.70)	1.75 (1.37–2.23)	0.66 (0.49–0.89)	0.83 (0.60–1.15)	1.26 (0.91–1.73)
		Eastern	1.61 (1.24–2.08)	1.66 (1.30–2.13)	1.67 (1.37–2.03)	0.97 (0.68–1.39)	0.96 (0.70–1.33)	0.99 (0.72–1.36)
	Mean age (years)	≥60.0	1.49 (1.13–1.96)	1.76 (1.44–2.16)	1.67 (1.34–2.09)	0.85 (0.60–1.19)	0.89 (0.63–1.27)	1.05 (0.78–1.42)
		<60.0	1.50 (1.23–1.83)	2.26 (1.61–3.18)	1.86 (1.61–2.15)	0.66 (0.45–0.98)	0.81 (0.63–1.03)	1.22 (0.84–1.02)
	Percentage male (%)	≥60.0	1.23 (1.06–1.43)	–	1.67 (1.44–1.93)	–	0.74 (0.60–0.91)	–
		<60.0	1.68 (1.29–2.19)	2.02 (1.64–2.50)	1.81 (1.45–2.28)	0.83 (0.59–1.17)	0.93 (0.66–1.31)	1.12 (0.82–1.52)
	Follow-up duration	≤30.0 days	1.47 (1.22–1.79)	1.79 (1.47–2.19)	1.74 (1.45–2.09)	0.82 (0.62–1.08)	0.84 (0.65–1.10)	1.03 (0.78–1.35)
		>30.0 days	–	2.34 (1.63–3.35)	–	–	–	–
	Study quality	High	1.61 (1.20–2.17)	1.84 (1.45–2.34)	1.97 (1.32–2.95)	0.88 (0.60–1.28)	0.82 (0.50–1.35)	0.93 (0.58–1.49)
		Moderate	1.44 (1.15–1.79)	2.04 (1.53–2.72)	1.70 (1.46–1.97)	0.71 (0.49–1.01)	0.85 (0.65–1.11)	1.20 (0.87–1.66)
NLR	Country	Western	0.10 (0.05–0.18)	0.05 (0.02–0.14)	0.25 (0.13–0.49)	2.00 (0.62–6.41)	0.40 (0.16–1.01)	0.20 (0.06–0.65)
		Eastern	0.30 (0.18–0.48)	0.14 (0.06–0.37)	0.63 (0.51–0.77)	2.14 (0.76–6.02)	0.48 (0.28–0.81)	0.22 (0.09–0.56)
	Mean age (years)	≥60.0	0.08 (0.03–0.20)	0.06 (0.02–0.14)	0.34 (0.19–0.60)	1.33 (0.34–5.19)	0.24 (0.08–0.71)	0.18 (0.06–0.55)
		<60.0	0.19 (0.11–0.35)	0.15 (0.04–0.51)	0.50 (0.31–0.81)	1.27 (0.31–5.13)	0.38 (0.18–0.81)	0.30 (0.08–1.17)
	Percentage male (%)	≥60.0	0.11 (0.05–0.24)	–	0.48 (0.27–0.87)	–	0.23 (0.09–0.61)	–
		<60.0	0.17 (0.10–0.28)	0.07 (0.03–0.17)	0.36 (0.22–0.60)	2.43 (0.89–6.66)	0.47 (0.23–0.97)	0.19 (0.07–0.53)
	Follow-up duration	≤30.0 days	0.12 (0.07–0.22)	0.07 (0.02–0.18)	0.42 (0.27–0.65)	1.71 (0.50–5.92)	0.29 (0.14–0.59)	0.17 (0.05–0.54)
		>30.0 days	–	0.12 (0.04–0.35)	–	–	–	–
	Study quality	High	0.13 (0.05–0.33)	0.06 (0.02–0.20)	0.38 (0.20–0.71)	2.17 (0.49–9.60)	0.34 (0.11–1.07)	0.16 (0.04–0.59)
		Moderate	0.12 (0.06–0.24)	0.10 (0.04–0.26)	0.40 (0.24–0.69)	1.20 (0.37–3.85)	0.30 (0.13–0.72)	0.25 (0.09–0.73)
DOR	Country	Western	12.68 (7.19–22.38)	30.41 (10.41–88.82)	6.32 (3.35–11.92)	0.42 (0.12–1.40)	2.01 (0.86–4.70)	4.81 (1.38–16.72)
		Eastern	4.56 (3.67–5.65)	10.05 (5.30–19.06)	2.59 (1.83–3.67)	0.45 (0.23–0.89)	1.76 (1.17–2.65)	3.88 (1.87–8.04)
	Mean age (years)	≥60.0	12.86 (6.11–27.07)	23.01 (10.84–48.85)	4.55 (2.50–8.28)	0.56 (0.19–1.61)	2.83 (1.09–7.35)	5.06 (1.93–13.23)
		<60.0	6.24 (4.46–8.75)	11.52 (3.44–38.55)	3.41 (2.25–5.16)	0.54 (0.15–1.90)	1.83 (1.07–3.12)	3.38 (0.94–12.12)
	Percentage male (%)	≥60.0	8.06 (4.08–15.91)	12.32 (1.74–87.48)	3.08 (1.76–5.37)	0.65 (0.08–5.20)	2.62 (1.09–6.31)	4.00 (0.52–30.66)
		<60.0	9.16 (5.69–14.75)	19.66 (9.31–41.51)	4.51 (2.82–7.22)	0.47 (0.19–1.13)	2.03 (1.04–3.97)	4.36 (1.80–10.54)
	Follow-up duration	≤30.0 days	9.45 (6.19–14.41)	19.21 (8.02–45.99)	3.70 (2.46–5.55)	0.49 (0.19–1.30)	2.55 (1.42–4.59)	5.19 (1.98–13.61)
		>30.0 days	8.42 (5.14–13.80)	15.90 (8.10–31.21)	5.53 (2.34–13.03)	0.53 (0.23–1.22)	1.52 (0.57–4.10)	2.88 (0.96–8.57)
	Study quality	High	7.96 (4.90–12.91)	21.21 (7.10–63.34)	5.17 (2.51–10.64)	0.38 (0.11–1.24)	1.54 (0.65–3.67)	4.10 (1.11–15.22)
		Moderate	9.44 (5.17–17.23)	16.25 (6.88–38.41)	3.33 (2.22–5.01)	0.58 (0.20–1.66)	2.83 (1.37–5.86)	4.88 (1.88–12.63)
AUC	Country	Western	0.85 (0.82–0.88)	0.82 (0.78–0.85)	0.74 (0.70–0.78)	1.04 (0.98–1.10)	1.15 (1.08–1.23)	1.11 (1.03–1.19)
		Eastern	0.73 (0.69–0.77)	0.74 (0.70–0.78)	0.66 (0.62–0.70)	0.99 (0.91–1.07)	1.11 (1.02–1.20)	1.12 (1.03–1.22)
	Mean age (years)	≥60.0	0.85 (0.82–0.88)	0.83 (0.80–0.86)	0.70 (0.66–0.74)	1.02 (0.97–1.08)	1.21 (1.14–1.30)	1.19 (1.11–1.27)
		<60.0	0.73 (0.69–0.77)	0.79 (0.75–0.82)	0.70 (0.65–0.73)	0.92 (0.86–0.99)	1.04 (0.96–1.13)	1.13 (1.05–1.21)
	Percentage male (%)	≥60.0	0.74 (0.70–0.77)	–	0.65 (0.60–0.69)	–	1.14 (1.05–1.24)	–
		<60.0	0.82 (0.79–0.85)	0.81 (0.78–0.85)	0.72 (0.68–0.76)	1.01 (0.96–1.07)	1.14 (1.07–1.22)	1.13 (1.05–1.21)
	Follow-up duration	≤30.0 days	0.81 (0.77–0.84)	0.79 (0.75–0.82)	0.69 (0.65–0.73)	1.03 (0.96–1.09)	1.17 (1.09–1.26)	1.14 (1.06–1.23)
		>30.0 days	–	0.83 (0.79–0.86)	–	–	–	–
	Study quality	High	0.79 (0.75–0.83)	0.80 (0.76–0.83)	0.74 (0.70–0.78)	0.99 (0.92–1.06)	1.07 (0.99–1.15)	1.08 (1.01–1.16)
		Moderate	0.80 (0.76–0.83)	0.81 (0.77–0.84)	0.67 (0.62–0.71)	0.99 (0.93–1.05)	1.19 (1.10–1.29)	1.21 (1.05–1.14)

AUC, area under the receiver operating characteristic curves; GRACE, Global Registry in Acute Coronary Events; HEART, History, Electrocardiography, Age, Risk factors and Troponin; NLR, negative likelihood ratio; PLR, positive likelihood ratio; TIMI, thrombolysis in myocardial infarction.

Discussion

This study was the first meta-analysis to conduct indirect comparisons of the predictive values of risk stratification assessed by the TIMI, HEART and GRACE scores on MACEs in ACP patients. The current study included a total of 40 262 ACP patients from 33 prospective cohort studies and across a wide range of patient characteristics. The findings of this study suggest that the predictive values of TIMI, HEART and GRACE scoring systems were better for MACEs in ACP patients admitted to the ED. Moreover, an indirect analysis indicated that the predictive value of TIMI and HEART was superior to that of GRACE for predicting MACEs, while there were no significant differences between TIMI and HEART for predicting MACEs. The results of sensitivity analyses for studies using all three scoring systems were consistent with those of the overall analysis. Meanwhile, we noted that the sensitivity of TIMI was lower than HEART for predicting MACEs.

Several systematic reviews and meta-analyses have illustrated the predictive values of the TIMI, HEART and GRACE scoring systems on MACEs in ACP patients.57–59 Hess et al included eight prospective studies and found that the TIMI score provided effective risk stratification for predicting MACEs in potential ACS patients, whereas it should not be used as the sole means for determining patient disposition.57 Van Den Berg et al identified 2 prospective and 10 retrospective cohort studies and suggested that the HEART score could be used to identify MACEs in patients with a suspected diagnosis of ACS.58 Roche et al included 11 studies and found that using 100 as the cut-off value of the GRACE score could predict the discharge of nearly 70% of presentations, while the predictive value for subsequent MACE risk was not obtained.59 However, the above studies only reported the diagnostic value of a single scoring system for predicting MACEs in ACS patients. Therefore, we performed the current meta-analysis of prospective studies to evaluate the predictive values of the TIMI, HEART and GRACE scoring systems on the risk of MACEs in ACP patients and systematically compared the predictive values among them.

The predictive value of the TIMI score for MACEs in ACP patients was statistically significant, whereas several studies reported inconsistent results. Sanchis et al found that the TIMI score was not associated with the risk of MACEs when 0 was used as the cut-off value.26 Graham et al found that a low TIMI score could not rule out cardiac causes of chest pain.35 Holly et al suggested that the TIMI score was not associated with the risk of MACE at 30 days when 0 was used as the cut-off value.36 Leung et al indicated that a modified TIMI score of 0 could not rule out 30-day MACEs in ACP patients admitted to the ED.45 The potential reasons for this could be that the TIMI score was designed for risk stratification in patients with non-ST-segment elevation ACS, which is mainly based on appropriate ECG changes or elevations of biomarkers of necrosis. Moreover, the presentation characteristics in ACP patients were not entered into the TIMI score. Finally, the prevalence of MACEs during the follow-up in these studies was lower than expected, resulting in broad 95% CIs, that is, no statistically significant difference.26 35 36 45

The predictive value of the HEART score for predicting MACEs in ACP patients was statistically significant. Nearly all included studies reported a similar conclusion, whereas the study conducted by McCord et al suggested that the HEART score after 4 hours of the presentation was associated with marginal predictive values for the risk of MACEs.47 The potential reason for this may be that this study used a modified HEART score, and the original HEART score only considered the initial cTn value, without taking serial sampling into account, which is associated with a lower prevalence of MACEs.60

The predictive value of the GRACE score for predicting MACE in ACP patients was statistically significant, and all included studies reported similar conclusions for the predictive value of the GRACE score on the risk of MACEs. Especially, the GRACE score was initially designed for post-ACS risk stratification, including unstable angina and non-ST-elevation ACS.11 The American Heart Association suggested the use of the GRACE score for admission and discharge of ACS patients. Moreover, the risk assessment for patients evaluated outside the hospital should be recommended to use GRACE.61 Therefore, the predictive value of the GRACE score in low-risk individuals was restricted, which should be addressed in clinical practice.

We noted that the predictive value of the GRACE score was inferior to that of TIMI and HEART scores. Sensitivity analyses were performed for studies reporting all three scoring systems, which included nine studies,42–44 46 48 52 53 55 56 and the results indicated TIMI and HEART having higher predictive values for MACEs than GRACE. The results of the sensitivity analyses were more reliable owing to the analysis of three scoring systems based on direct comparisons. Moreover, the GRACE score was initially developed for ACS patients but not for ACP patients, and the potential risk factors for the progression of MACEs were not considered in the GRACE score. Interestingly, we noted no significant difference between TIMI and HEART for predicting MACEs, while several studies reported that the predictive value of HEART for MACEs was superior to that of TIMI.44 46 48 52 53 55 56 Subgroup analysis found TIMI with lower AUC compared with HEART if the mean age of patients was <60.0 years. The potential explanation could be the use of HEART score in the absence of exact definitions for medical history across included studies,11 and the predictive value of HEART was more suitable in low-risk individuals.

Three strengths of this quantitative meta-analysis should be highlighted: (1) the study was based on prospective cohort studies and used relatively uniform cut-off values, which were associated with lower selective and informative biases; (2) the analysis of this study was based on a large sample size, and the findings in our study were more robust than any individual study and (3) the predictive values of the TIMI, HEART and GRACE scoring systems on the risk of MACEs in ACP patients were compared through an indirect analytical approach.

Despite the above-mentioned findings, the predictive values of the TIMI, HEART and GRACE could be affected by the definitions of MACEs and the ranges from a single endpoint (death or myocardial infarction) to a composite endpoint. Subgroup analysis based on MACE definition were not performed because of the definition of MACE across included studies are various. Therefore, MACE definition could affect the predictive value and follow-up durations owing to these factors that attribute to the weight of pooled conclusion. Moreover, this meta-analysis was based on crude data, and the adjusted results were not available. Furthermore, substantial heterogeneity was detected across the included studies, and the heterogeneity was not fully explained by sensitivity and subgroup analyses. In addition, the analysis was conducted on published articles, which causes inevitable publication biases. Finally, the current study was based on indirect comparisons between the predictive values of the TIMI, HEART and GRACE scoring systems as direct comparisons were not available.

The findings of this study indicated that risk stratification assessed by the TIMI, HEART and GRACE scores provides relatively appropriate predictive values for MACEs in ACP patients. The results of indirect comparison analysis indicated that TIMI and HEART had relatively relative better predictive values than GRACE on subsequent MACE risk. Further prospective cohort studies should be conducted to directly compare the predictive values of TIMI, HEART and GRACE on MACEs in ACP patients.