Secondary analysis of frequency, circumstances and consequences of calculation errors of the HEART (history, ECG, age, risk factors and troponin) score at the emergency departments of nine hospitals in the Netherlands.

OBJECTIVE: The HEART score can accurately stratify the risk of major adverse cardiac events (MACE) in patients with chest pain. We investigated the frequency, circumstances and potential consequences of errors in its calculation.
METHODS: We performed a secondary analysis of a stepped wedge trial of patients with chest pain presenting to nine Dutch emergency departments. We recalculated HEART scores for all patients by re-evaluating the elements age (A), risk factors (R) and troponin (T) and compared these new scores with those given by physicians in daily practice. We investigated which circumstances increased the probability of incorrect scoring and explored the potential consequences.
RESULTS: The HEART score was incorrectly scored in 266 out of 1752 patients (15.2%; 95% CI 13.5% to 16.9%). Most errors occurred in the R ('Risk factors') element (61%). Time of admission, and patient's age or gender did not contribute to errors, but more errors were made in patients with higher scores. In 102 patients (5.8%, 95% CI 4.7% to 6.9%) the incorrect HEART score resulted in incorrect risk categorisation (too low or too high). Patients with an incorrectly calculated HEART score had a higher risk of MACE (OR 1.85; 95% CI 1.37 to 2.50), which was largely related to more errors being made in patients with higher HEART scores.
CONCLUSIONS: Our results show that the HEART score was incorrectly calculated in 15% of patients, leading to inappropriate risk categorisation in 5.8% which may have led to suboptimal clinical decision-making and management. Actions should be taken to improve the score's use in daily practice. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

RCT Entities:   Population Interventions Outcomes

The use of prospectively collected data from a large multicentre trial.

This trial had a pragmatic design, ensuring minimal interference with daily practice.

Errors in the ECG or history component were not assessed.

Errors were identified by verifying against originally recorded data which might contain some errors as well.

Introduction

Approximately 6% of patients admitted to the emergency department (ED) present with chest pain as their main symptom.1 2 It is a challenge for physicians to identify the patients with acute coronary syndrome (ACS) among the many patients with chest pain presenting to the ED. Patients with a missed diagnosis of ACS are at increased risk of morbidity and mortality.3 The challenge particularly is not to miss ACS in these patients while avoiding unnecessary diagnostic procedures.4

Over the years several cardiac risk scores have been developed.5 6 The HEART score was developed to stratify the risk of major adverse cardiac events (MACE) in patients with chest pain at the ED.7 The HEART score has been extensively and internationally validated. Furthermore, it has been compared with other risk scores in various studies including a systematic review, mostly showing superiority over other risk scores in predicting the occurrence of MACE and proportion of patients to be discharged in an early stage from the ED.8–14 HEART is an acronym for history, ECG, age, risk factors and troponin (see online supplementary appendix A). Each of these five elements can be scored with 0, 1 or 2 points, resulting in total scores ranging from 0 to 10. Patients with a score of 0–3 are considered low risk, a score of 4–6 is considered intermediate risk and a score of 7–10 is considered high risk. The proposed policy for a low risk score is discharge from the ED without further testing or observation, the policy for an intermediate score is non-invasive testing and for a high risk score invasive testing and therapy.7 10 15

Risk scores are now increasingly being used in daily practice and recommended in the guidelines of the European Society of Cardiology.16 However, there has been no attention for the practical use of these scores (ie, the actual calculation), which can be flawed depending on complexity of the incorporated elements. Moreover, it is very well imaginable that in a chaotic, time-constrained setting of an ED, incorrect calculations of such a score may occur. These incorrect calculations might result in suboptimal clinical management.

The aim of the present study is threefold. First, we will assess the frequency and type of errors made in calculating the HEART score by physicians at the ED. Second, we will investigate the circumstances in whom these errors occurred. Third, we will explore potential consequences in terms of incorrect risk categorisation and correlation of incorrect scores with the occurrence of MACE.

Methods

Study design

We performed a secondary analysis of prospectively collected data available from the HEART-Impact trial.17 The HEART-Impact trial was a stepped wedge cluster randomised trial, in which nine hospitals in the Netherlands participated. In short, all hospitals (clusters) started with an initial period of usual care. Subsequently, at regular intervals of 6 weeks (‘steps’), each hospital switched (in a randomised order) to using the HEART score until all hospitals had crossed over. Details on the HEART-Impact trial’s design and results have been previously published.17 18

Study population and setting

All patients (≥18 years) presenting with chest pain to the ED were included from 1 July 2013 to 31 August 2014. Exclusion criteria were ST-segment elevation myocardial infarction, recurrent presentation with chest pain and unwillingness or inability to give informed consent. Only patients who were allocated to HEART care (ie, the period where application of the HEART’s score was implemented into daily work-up of patients with chest pain) were included in current secondary analysis (thus, patients in the usual care period were excluded). A key characteristic of the stepped wedge design is a stepwise implementation of the intervention: all clusters will have a period of application of the HEART score. In our trial, every 6 weeks a new cluster switched to the HEART care period, until in the end all clusters had switched.

Patients with missing HEART scores in the trial were excluded, even though the score could be calculated from available data, since we wished to investigate the frequency of incorrect scores given by physicians at the ED. Also excluded were patients with missing data on risk factors or troponin measurements, necessary for us to calculate corrected scores on the individual elements of the HEART score.

Informed consent forms were retrieved for all patients.

Calculation of HEART score

During the HEART-Impact trial physicians at participating EDs were trained to calculate and interpret the HEART score. One week before implementation of the HEART score in a hospital, a presentation on the trial was given in the morning meeting. Physicians and nurses were instructed personally and during a meeting physicians could practise with exercises on the calculation of the HEART score. Furthermore, small pocket cards with the HEART score and the proposed policies for each score were handed out to residents, nurses and cardiologists. The materials and documents used for the training (in Dutch) are shown in online supplementary appendices B1–B4.

From the database of the HEART-Impact trial, the total HEART score and the scoring for each of the five clinical elements of all patients were available for current analysis. However, the information needed to calculate corrected H and E elements for all patients was not available from these data. The H element is subjective and the physicians are responsible to estimate the probability of ACS by judging the patient’s anamnesis. Also, during the HEART-Impact trial, ECGs were not centrally scored and stored. Therefore, we could not calculate corrected scores for the H and E elements of the HEART score. Information on age, risk factors and troponin measurements was available from medical records and/or discharge letters, thus we were able to recalculate scores for the A, R and T elements in patients and compare these with the HEART score given by the physicians.

Concerning the measurement of troponin, the nine participating hospitals all used different analysers with different upper limits of a normal troponin concentration (see online supplementary appendix C). When we calculated the corrected scores for the T element in each patient, these different upper limits were taken into account. The score for the T element should be based on the first performed troponin measurement, as instructed in the original HEART score.7

Frequency and type of errors in HEART scores

The primary outcome for current study was the frequency of incorrect scores given to the individual elements of the HEART score. We also investigated whether the scores on the individual elements of the HEART score were correctly added up. Furthermore, the direction and possible causes for errors were examined.

Circumstances associated with the frequency of errors

We investigated whether time of admission, hospital and patient characteristics increased the probability of receiving an incorrect score. To assess the possible influence of time of admission, we divided days into the following periods: night (00:00–07:59), morning (08:00–12:59), afternoon (13:00–17:59) and evening (18:00–23:59). These periods were based on current working shifts in Dutch hospitals.

Potential consequences of incorrect HEART scores

To explore potential consequences, we investigated whether the incorrect total HEART score led to a difference in risk categorisation, as three relevant risk groups have been defined for the HEART score: low (0–3), intermediate (4–6), high (7–10).

We also examined whether the presence of errors was associated with the 6-week occurrence of MACE. The composite endpoint of MACE was defined as: ST-segment elevation myocardial infarction, non-ST-elevation myocardial infarction, unstable angina, percutaneous coronary intervention, coronary arterial bypass grafting, coronary stenosis managed conservatively or death due to any cause. All cases possibly indicating a MACE were reviewed by two independent cardiologists for classification using the latest guidelines.19 In case of disagreement between two adjudicating cardiologists, the case was discussed in a meeting with at least three cardiologists.

We also calculated the diagnostic accuracy statistics for the uncorrected and corrected HEART score categories and investigated whether there was a significant difference in the accuracy of both calculations.

Statistical analysis

The frequency of errors with corresponding 95% CIs in the HEART scores given by physicians was calculated by comparing their scores with the recalculated scores. Differences in characteristics between the group of patients with an incorrect HEART score and the group with a correct HEART score were examined with Χ2 tests (categorical variables) and independent samples t-tests (continuous variables). Diagnostic accuracy statistics were calculated using 2×2 tables. Logistic regression was used to assess the association between making an error in the HEART score and the occurrence of MACE. The strength of the association was assessed by ORs and 95% CIs. We examined whether the strength of the possible association between making an error in the HEART score and MACE changed after adjusting for the value of the corrected HEART score by adding the corrected HEART score to the logistic regression model. The corrected HEART score was added to the model as a continuous predictor (linear relationship) after checking its functional form with the outcome. Statistical analysis was performed using IBM SPSS Statistics for Windows, V.21.

Results

Study population

Of the 3666 patients included in the HEART-Impact trial, 1821 received HEART care during the trial. Sixty-nine patients had one or more missing scores of individual elements of the HEART score and therefore 1752 patients remained for analysis (figure 1).

Figure 1

Patient’s flow chart of secondary analyses of the HEART-Impact trial.

The baseline characteristics are shown in table 1. In total, MACE occurred in 325 patients (18.6%) within 6 weeks after presentation at the ED.

Table 1

Baseline characteristics of patients of HEART-Impact trial included in analyses (n=1752)

Demographics	Total (n=1752)	HEART incorrect (n=266)	HEART correct (n=1486)
Male	940 (54%)	145 (55%)	795 (54%)
Mean age (SD)	62 (14)	64 (15)	61 (14)
Risk factors for cardiovascular disease
Positive family history	631 (36%)	92 (35%)	539 (36%)
Recent smoking (<3 months)	437 (25%)	58 (22%)	379 (26%)
Diabetes mellitus	270 (15%)	36 (14%)	234 (16%)
Obesity (BMI>30)	321 (18%)	38 (14%)	283 (19%)
(Currently treated) hypercholesterolaemia	559 (32%)	78 (29%)	481 (32%)
(Currently treated) hypertension	842 (48%)	127 (48%)	715 (48%)
History of cardiovascular disease
Any cardiac ischaemic disease*	476 (27%)	89 (34%)	387 (26%)
Any cardiovascular ischemic disease†	578 (33%)	124 (47%)	454 (31%)
Hospital
Academic hospital	484 (28%)	93 (35%)	391 (26%)
Time of admission
Night (00:00–07:59)	129 (7%)	19 (7%)	110 (7%)
Morning (08:00–12:59)	674 (39%)	106 (40%)	568 (39%)
Afternoon (13:00–17:59)	473 (27%)	68 (26%)	405 (27%)
Evening (18:00–23:59)	215 (12%)	36 (14%)	179 (12%)
Missing time of admission	261 (15%)	37 (14%)	224 (15%)
Corrected HEART score
Mean uncorrected HEART score (SD)	4.0 (1.9)	4.7 (1.9)	4.0 (1.9)
Low risk category (HEART score 0–3)	713 (41%)	68 (27%)	608 (41%)
Intermediate risk category (4–6)	849 (49%)	153 (58%)	726 (49%)
High risk category (7–10)	190 (11%)	25 (10%)	152 (10%)
Outcome
MACE within 6 weeks	325 (19%)	83 (29%)	252 (17%)

Values given are either means with SD in brackets or frequencies with percentages in brackets.

*Any cardiac ischaemic disease includes acute myocardial infarction, percutaneous coronary intervention, coronary artery bypass grafting.

†Any cardiovascular ischaemic disease includes acute myocardial infarction, percutaneous coronary intervention, coronary artery bypass grafting, stroke and peripheral artery disease.

BMI, body mass index; MACE, major adverse cardiac event.

A total of 296 elements were scored incorrectly in 275 different patients (table 2). In 19 patients, two elements were scored incorrectly and in 1 patient, three elements were scored incorrectly.

Incorrect calculations of the individual elements in the HEART score resulted in a total of 266 patients with incorrect total HEART scores (15.2%; 95% CI 13.5% to 16.9%) (see online supplementary appendix D). In nine patients, the incorrect calculations of two individual elements cancelled out each other. Notably, no HEART scores were incorrectly added up by the physicians.

The most incorrectly scored element of the HEART score was the R element (61%). The A element was scored incorrectly in 52 (3%) patients, the R element in 182 (10%) patients and the T element in 62 (4%) patients. The frequency of incorrect calculations of the R due to not scoring 2 points to patients with a history of atherosclerosis was 89/182 (5%), which accounts for half of the total frequency of incorrect calculations of the R element. The incorrect scoring of the T element occurred in 24/62 (39%) patients because the second troponin measurement seemed to be used for calculation. The second troponin value was higher than the first in 16 of these 24 patients (67%). In all of these 16 cases, there was an increase in the troponin value of at least 20% between the first and second measurements.

Circumstances and patient characteristics associated with the frequency of errors

There were significant differences in baseline characteristics between the groups with correct and incorrect HEART scores. Incidence of a history of atherosclerotic disease and use of anticoagulants and the mean age and total HEART score were higher in patients with an incorrect HEART score. Incidence of smoking, obesity and hypercholesterolaemia were higher in patients with correct HEART scores.

There was a significant difference in the percentages of incorrect HEART scores between participating hospitals (range 9.6%–21.6%, p=0.010). Patients from academic hospitals had a statistically significant higher chance of receiving incorrect HEART scores (19.2%) compared with patients from non-academic hospitals (13.6%; p=0.004).

No significant correlation was found between time of admission and incidence of incorrect HEART scores. The percentages of incorrectly scored HEART scores ranged from 14.4% between 13:00 and 17:59 and 16.7% between 18:00 and 23:59 (table 1).

Incorrect HEART score elements were calculated in 275 patients, leading to 266 patients with an incorrect total HEART score, which led to an inappropriate risk category in 102 patients (5.8%; 95% CI 4.7% to 6.9%) (table 3).

Table 3

Classification of inappropriate risk categories of the HEART score

Risk category as classified by physicians	Risk category based on corrected score			Total
	Low	Intermediate	High
Low (HEART score 0–3)	663 (12)	50 (2)	–	713 (14)
Intermediate (HEART score 4–6)	13 (3)	813 (156)	23 (12)	849 (171)
High (HEART score 7–10)	–	16 (12)	174 (128)	190 (140)
Total	676 (15)	879 (170)	197 (140)	1752 (325)

The figures in brackets represent the amount of patients with major adverse cardiac events in that risk category.

Forty-six patients were incorrectly given a HEART score of 3 points and should have been given 4 points or higher and thus classified as intermediate risk. Out of these patients, two (4.3%) had a MACE. This incidence of MACE was not significantly higher than the incidence of MACE among all patients with a corrected HEART score of 3, which was 3.9% (12/307, p=0.887). The 6-week incidence of MACE in the low risk category in the original HEART-Impact trial was 2.0%. After applying the corrected risk categories to all patients, the incidence of MACE in the low risk category was 2.2% (95% CI 1.1% to 3.3%). The characteristics of the low-risk patients with MACE are displayed in online supplementary appendix E. The diagnostic accuracy (sensitivity, specificity, positive predictive value and negative predictive value) of the uncorrected and corrected HEART scores is displayed in online supplementary appendix F. None of these parameters differed significantly between both scores.

Of the 73 patients who received an incorrect score and had MACE, 40 (55%) had received a HEART score that was too low. In patients with an incorrect HEART score that was too low the incidence of MACE was 22% whereas in patients with an incorrect HEART score that was too high the incidence was 18%.

Of the 266 patients who received an incorrect HEART score, 73 (27%) had MACE within 6 weeks compared with 252 out of 1486 patients (17%) with a correct score. This association was statistically significant (OR 1.85; 95% CI 1.37 to 2.50; p<0.001). After adding the corrected HEART score to the regression model, the association between making an error in the HEART score and MACE became less strong and was no longer statistically significant (OR 1.34; 95% CI 0.94 to 1.93; p=0.11).

Discussion

This is the first study to investigate the frequency of incorrect calculations of the HEART score, showing a substantial percentage (15%) of incorrect scoring. Most errors occurred for the R (risk factors) element with 61%, followed by T (troponin) with 21% and A (age) with 18%. Incorrect calculations of the R element were mostly caused by not taking a history of atherosclerotic disease into consideration, thereby scoring it too low. Furthermore, the score on the T element was often inappropriately based on the second troponin measurement. An error in the HEART score led to an inappropriate risk category in 5.8% of the patients.

Several circumstances may have contributed to the incorrect scoring of the HEART score. First, ED residents are commonly (typically) under time constraints when assessing patients. This makes the score more vulnerable to incorrect scoring. Second, physicians could deliberately score some elements incorrectly, because they have a strong gut feeling telling them the patient is less or more likely to have ACS. Third, the score could be misinterpreted by the physician. For example, the physician could think that the T element should be based on the second troponin measurement or that a history of atherosclerotic disease counts as one risk factor. Last, the ≥ and ≤ signs could be mistaken while scoring the A and T elements. All in all, it is hard to pinpoint the ‘true’ underlying cause(s) of incorrectly calculated scores.

In this study, patients with an incorrectly calculated HEART score had a higher chance of MACE within 6 weeks. However, this relationship diminished after adding the corrected HEART score to the model and the association was no longer statistically significant. This indicates that more errors were made in patients with higher HEART scores (table 1), which in turn lead to the relatively strong association between making an error and MACE in the crude data.

Literature on incorrect clinical use of risk scores is scarce. Taylor and Mancini performed a prospective analysis for the TIMI (Thrombolysis in Myocardial Infarction) score among patients with non-traumatic chest pain presenting at a chest pain unit and found that the scores given by clinicians equalled the scores given by trained researchers in only 43% of patients, which the authors attribute to time constraints at the ED.20 Furthermore, incidence of incorrect use of the well-known Wells score for diagnosis of pulmonary embolism and deep venous thrombosis has been investigated. Roy et al found inappropriate diagnostic management based on the Wells score in 43% of patients with suspected pulmonary embolism.21 Kingma et al found incorrect guideline use in 32% of patients with suspected deep vein thrombosis.22

The percentage of incorrect calculations in our study (15%) is considerably lower than those calculated in the mentioned studies. This may be due to the relatively simple format of the HEART risk score: five elements, all scored 0, 1 or 2 points. However, we were only able to correct the HEART score for the A, R and T elements. The frequency of errors may well be greater when correcting for the H and E elements, see also section on limitations.

Concerning strengths of this study, the HEART-Impact trial had broad inclusion criteria. Furthermore, the trial was designed as a pragmatic study meaning limited interference and additional procedures compared with daily practice. Our estimate of the frequency of errors is therefore likely to be a good representation of the frequency in real life. Also, the study population appears to be a good representation of patients presenting with chest pain at the ED in daily clinical practice, when compared with other studies.10–12 23–25 Our percentage of patients with MACE (18.6%) is comparable to those found in other European studies (15%–20%).10 11 24 25

Our study has some limitations. First, we were not able to account for all possible incorrect calculations of the HEART score. The scores on the H and E elements could not be recalculated, because the information necessary to score the H and E elements was not available from our data. The score on the H element is based on history taking (anamnesis) of the patient and this information was often not (or not sufficiently) documented in medical records or discharge letters. Also, the H element is typically assessed by attending physicians at the ED, where they can take into account all information including their gut feeling, while attending to the patient. Correcting for the H element based on notes from the medical record would exclude the clinical view of the physician from the HEART score calculation process, which we think is an important strength of the score. Concerning the E element, ECGs were not routinely collected during the trial. Moreover, ECG interpretation is shown to have a high interobserver variability.26 27 In other words, the H and E elements are always subject to subjectivity, while age, risk factors and troponin concentration are objective measurements. Second, it is hard to assess whether the recalculated HEART scores are in fact correct in each patient, since the used information from medical records could contain mistakes as well. However, by combining multiple sources we tried to minimise this problem.

Our results show that incorrect calculation of the HEART score occurs regularly, possibly resulting in inappropriate clinical decision-making and suboptimal care of these patients ranging from an increase in the number of unnecessary admissions and diagnostic tests being performed to a higher risk of MACE. We should be aware that even a relatively simple score as the HEART score may be calculated incorrectly. To reduce the frequency of errors when calculating HEART scores, more attention should be paid during anamnesis to scoring the R element. It is also important to stress that the scoring of the T element should be based on the first troponin measurement. A computer program aiding the physician in calculating the HEART score might also reduce the frequency of errors, since it could remove some steps from the calculation process.

Table 2

(A) Comparison of the A (age) element; (B) Comparison of the R (risk factors) element; (C) Comparison of the T (troponin) element of the HEART score given by the physicians and the corrected HEART score

A element scored by physicians	Corrected A score			Total
	0	1	2
0	230 (96%)	7 (3%)	2 (1%)	239
1	13 (2%)	751 (96%)	20 (3%)	784
2	0 (0%)	10 (1%)	719 (99%)	729
Total	243	769	742	1752

R element scored by physicians	Corrected R score			Total
	0	1	2
0	204 (83%)	35 (13%)	8 (3%)	247
1	12 (2%)	661 (85%)	108 (14%)	781
2	0 (0%)	19 (3%)	705 (97%)	724
Total	216	717	821	1752

T element scored by physicians	Corrected T score			Total
	0	1	2
0	1458 (99%)	10 (1%)	2 (0%)	1470
1	15 (9%)	143 (83%)	14 (8%)	172
2	7 (6%)	14 (13%)	89 (81%)	110
Total	1480	167	105	1752