Systematic Review and Meta-Analysis of Diagnostic Accuracy to Identify ST-Segment Elevation Myocardial Infarction on Interpretations of Prehospital Electrocardiograms.

Background: The aim of this study was to assess and discuss the diagnostic accuracy of prehospital ECG interpretation through systematic review and meta-analyses. Methods and 
Results: Relevant literature published up to July 2020 was identified using PubMed. All human studies of prehospital adult patients suspected of ST-segment elevation myocardial infarction in which prehospital electrocardiogram (ECG) interpretation by paramedics or computers was evaluated and reporting all 4 (true-positive, false-positive, false-negative, and true-negative) values were included. Meta-analyses were conducted separately for the diagnostic accuracy of prehospital ECG interpretation by paramedics (Clinical Question [CQ] 1) and computers (CQ2). After screening, 4 studies for CQ1 and 6 studies for CQ2 were finally included in the meta-analysis. Regarding CQ1, the pooled sensitivity and specificity were 95.5% (95% confidence interval [CI] 82.5-99.0%) and 95.8% (95% CI 82.3-99.1%), respectively. Regarding CQ2, the pooled sensitivity and specificity were 85.4% (95% CI 74.1-92.3%) and 95.4% (95% CI 87.3-98.4%), respectively. Conclusions: This meta-analysis suggests that the diagnostic accuracy of paramedic prehospital ECG interpretations is favorable, with high pooled sensitivity and specificity, with an acceptable estimated number of false positives and false negatives. Computer-assisted ECG interpretation showed high pooled specificity with an acceptable estimated number of false positives, whereas the pooled sensitivity was relatively low.

Early diagnosis and reperfusion therapy are vital steps for a better prognosis in the management of patients with ST-segment elevation myocardial infarction (STEMI)., Prehospital attempts to identify STEMI are useful and critical in reducing the time until reperfusion, with various approaches tried, including the interpretation of prehospital electrocardiograms (ECGs) by paramedics or computers. For such prehospital procedures to be adopted in clinical practice, diagnostic accuracy is a crucial factor. Accurate early diagnosis can lead to prompt cardiac catheterization laboratory activation, which can reduce mortality and/or morbidity. Low diagnostic accuracy and high rates of false-positive results, which are overidentifications of STEMI, can have significant adverse effects on resource utilization. However, high rates of false-negative results can also be problematic because they can lead to interruption of early diagnosis and a delay in STEMI notification. The diagnostic accuracy of STEMI identification on prehospital ECGs by paramedics or computers has not been fully elucidated.

Objectives

The aim of this study was to assess and discuss the diagnostic accuracy of prehospital ECG interpretation through a systematic review and meta-analysis.

Methods

The Japan Resuscitation Council (JRC) Acute Coronary Syndrome (ACS) Task Force was established for the JRC guidelines 2020 and was organized by the Japanese Circulation Society, the Japanese Association of Acute Medicine, and the Japanese Society of Internal Medicine. The Task Force set 12 clinical questions (CQs), with 9 systematic reviews newly conducted.

The JRC ACS Task Force used the Population Intervention Comparator Outcome Study design and Time frame (PICOST) to define 2 CQs:

CQ1: can prehospital ECG interpretation by paramedics diagnose STEMI?

Using PICOST, CQ1 was defined as follows:

P (patients): prehospital adult patients suspected of STEMI

I (intervention): interpretation of prehospital 12-lead ECG by paramedics

C (comparison): 12-lead ECG or clinical diagnosis of STEMI by a physician

O (outcomes): diagnostic accuracy of STEMI, including false negatives, which can interrupt early diagnosis, and false positives, which can cause unnecessary catheterization laboratory activation

S (study design): all human studies, regardless of whether randomized or non-randomized, prospective or retrospective, showing all 4 values (i.e., true-positive, false-positive, false-negative, and true-negative values)

T (time frame): all published literature until July 15, 2020.

CQ2: can computer-assisted interpretation of prehospital 12-lead ECG diagnose STEMI?

Using PICOST, CQ2 was defined as follows:

P (patients): prehospital adult patients suspected of STEMI

I (intervention): computer-assisted interpretation of prehospital 12-lead ECG

C (comparison): 12-lead ECG or clinical diagnosis of STEMI by a physician

O (outcomes): diagnostic accuracy of STEMI, including false negatives, which can interrupt early diagnosis, and false positives, which can cause unnecessary catheterization laboratory activation

S (study design): all human studies, regardless of whether randomized or non-randomized, prospective or retrospective, showing all 4 values (i.e., true-positive, false-positive, false-negative, and true-negative values)

T (time frame): all published literature until July 15, 2020.

The meta-analyses were performed in accordance with the Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies Statement.

Search Strategy and Data Extraction

We included studies published in English that fulfilled all the components of the PICOST described above. Studies dealing with only true-positive and false-positive values and without true-negative and false-negative results, those with prehospital ECG transmission to experts, and those with all acute coronary syndrome patients were excluded from the meta-analysis. Relevant literature was identified by searching PubMed, from inception to July 2020. In addition, the reference lists of identified articles were reviewed to identify any further relevant articles. The search formula used by the International Liaison Committee on Resuscitation (ILCOR) in 2015, was used in the present analysis (). Two investigators (A.T., K.M.) independently screened all the titles and abstracts of the relevant literature; after excluding obviously non-applicable articles, case reports, case series, review articles, editorials, and clinical guidelines, they assessed the full text of the included articles. Any disagreement regarding eligibility was resolved by consensus.

Risk of Bias Assessment

Quality Assessment of Diagnostic Accuracy-2 (QUADAS-2) was used to evaluate the methodological quality of the included studies. The QUADAS-2 tool includes 4 domains (patient selection, the index test, the reference standard, and flow and timing), which were evaluated in 2 categories (risk of bias and applicability concern). The risk of bias and applicability concerns were judged as low, high, or unclear. These results are presented as figures, prepared using Review Manager version 5.3 (The Nordic Cochrane Center, The Cochrane Collaboration, Copenhagen, Denmark).

Rating the Certainty of Evidence

The GRADEpro system for diagnostic studies was used to assess the quality of evidence, which evaluates the risk of bias, indirectness, inconsistency, imprecision, and publication bias. The certainty of evidence was presented as high, moderate, low, or very low.

Statistical Analysis

Meta-analyses were performed and pooled sensitivity and specificity were calculated using STATA 17.0 SE (StataCorp, College Station, TX, USA). All analyses were performed using random-effects models. Statistical heterogeneity was assessed using the I2 statistic. The estimated absolute numbers of test positives (true positives and false positives) and test negatives (true negatives and false negatives) per 1,000 people were calculated using the pooled sensitivity and specificity analyses and the pretest probabilities of the target population in the GRADEpro system. The meta-analyses were performed based on all published data.

Results

Study Selection

shows a flowchart of the study selection process for both CQ1 and CQ2. After database searching and record screening, 35 and 29 articles were included for full-text assessment for CQ1 and CQ2, respectively. After excluding articles without data regarding negative test results (false negatives and true negatives) and dealing with different subjects/objects, 4 studies– were included in the meta-analysis for CQ1 and 6 studies– were included in the meta-analysis for CQ2.

Figure 1.

Flowchart of the study selection process for Clinical Question (CQ) 1 and CQ2.

Study Characteristics

shows the characteristics of the included studies for CQ1 and CQ2. For CQ1, 1,414 patients were included in 4 prospective cohort studies. Each study included 155–703 patients, with the prevalence of STEMI ranging from 12% to 33%.– In all 4 studies, each participating paramedic was trained for ECG interpretation before the study period. For CQ2, 47,717 patients were included in 6 retrospective cohort studies. Each study included 200–44,611 patients, with the prevalence of STEMI ranging from 1.2% to 50%.–

Table 1.

Characteristics of the Included Studies

	Study type	Sample size	Paramedic type	Reference standard	TP	FP	FN	TN	Prevalence (%)
CQ1: Paramedic interpretation of prehospital ECG
Ducas et al7(2012)	Prospective cohort	703	EMS personnel	Physician (cardiologist/ER physician)	228	152	1	322	33
Feldman et al9(2005)	Prospective cohort	151	Paramedic	Physician (cardiologist)	20	4	5	122	17
Foster et al10(1994)	Prospective cohort	149	ALS provider	Physician (ED physician)	17	0	1	131	12
Le May et al8(2006)	Prospective cohort	411	ACP	Physician (cardiologist/emergency physician)	60	13	3	335	15
CQ2: Computer interpretation of prehospital ECG
Bhalla et al14(2013)	Retrospective cohort	200		Physician (ED physician)	58	0	42	100	50
Bosson et al11(2017)	Retrospective cohort	44,611		Physician	482	711	47	43,371	1.2
Clark et al15(2010)	Retrospective cohort	912		Hospital clinical diagnosis	241	55	68	548	34
Garvey et al12(2016)	Retrospective cohort	500		CAG	118	33	27	322	29
Kudenchuk et al16(1991)	Retrospective cohort	1,189		Electrocardiographer	202	189	13	785	18
Wilson et al13(2013)	Retrospective cohort	305		Physician	22	15	1	267	8

ACP, advanced care paramedics; ALS, advanced life support; CAG, coronary angiography; ED, emergency department; EMS, emergency medical services; ER, emergency room; FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Risk of Bias in the Included Studies

shows the QUADAS-2 quality assessment results for each study in CQ1 and CQ2. For both CQs, most of the studies had a high risk of bias within the “reference standard” domain. A summary of the findings and an assessment of the evidence quality from the GRADEpro system for CQ1 and CQ2 are presented in . Regarding factors that may decrease the certainty of evidence in CQ1, the risk of bias was considered “very serious” in both sensitivity and specificity because the results of the reference standard (physician diagnosis) should not be blindly interpreted without the results of the index test in all studies (). Furthermore, ECG interpretation by a physician was used as a reference standard in all studies, and incompleteness may exist. Inconsistency was considered “serious” due to high heterogeneity (sensitivity: I2=98%; specificity: I2=99%). Subsequently, the certainty of the evidence was determined to be “very low” for CQ1 (). Regarding factors that may decrease the certainty of the evidence in CQ2 (), the risk of bias was considered “very serious” in both sensitivity and specificity for the same reasons as mentioned for CQ1. Inconsistency was considered “serious” due to high heterogeneity (sensitivity: I2=98%; specificity: I2=99%). As a result, the certainty of the evidence was determined to be “very low” for CQ2 ().

Figure 2.

Summaries of the risk of bias and applicability concerns for Clinical Question (CQ) 1 and CQ2. ECG, electrocardiogram.

Table 2.

Summary of Findings Regarding 2 Clinical Questions, (A) CQ1: Paramedic Electrocardiogram Interpretation, (B) CQ2: Computer Electrocardiogram Interpretation

(A) Test result	No. participants (no. studies)	No. results per 1,000 tested (95% CI)			Factors that may decrease certainty of evidence					Test accuracy CoE
		Baseline risk 30%*	Baseline risk 20%*	Baseline risk 10%*	Risk of bias	Indirectness	Inconsistency	Imprecision	Publication bias
True positives	1,414 (4)	287 (247–297)	191 (165–198)	96 (83–99)	Very seriousA	Not serious	SeriousB	Not serious	Not serious	⊕○○○ Very low
False negatives		13 (3–53)	9 (2–35)	4 (1–17)
True negatives	1,414 (4)	671 (576–694)	766 (658–793)	862 (741–892)	Very seriousA	Not serious	SeriousB	Not serious	Not serious	⊕○○○ Very low
False positives		29 (6–124)	34 (7–142)	38 (8–159)
(B) Test result	No. participants (no. studies)	No. results per 1,000 tested (95% CI)			Factors that may decrease certainty of evidence					Test accuracy CoE
		Baseline risk 50%*	Baseline risk 25%*	Baseline risk 1%*	Risk of bias	Indirectness	Inconsistency	Imprecision	Publication bias
True positives	47,717 (6)	427 (371–462)	214 (185–231)	9 (7–9)	Very seriousA	Not serious	SeriousB	Not serious	Not serious	⊕○○○ Very low
False negatives		73 (38–129)	36 (19–65)	1 (1–3)
True negatives	47,717 (6)	477 (437–492)	716 (655–738)	944 (864–974)	Very seriousA	Not serious	SeriousB	Not serious	Not serious	⊕○○○ Very low
False positives		23 (8–63)	34 (12–95)	46 (16–126)

(A) The pooled sensitivity and pooled specificity for Clinical Question 1 were 95.5% (95% confidence interval [CI] 82.5–99.0%) and 95.8% (95% CI 82.3–99.1%), respectively. *Prevalences of 30%, 20%, and 10% were assumed according to the prevalences of ST-elevation myocardial infarction (STEMI) in the 4 included studies, which ranged from 12% to 33%. AThe results of the reference standard (physician diagnosis) may not be interpreted without the results of the index test in all studies. Further, electrocardiogram interpretation by a physician was used as the reference standard in all studies, and there may be incompleteness in the reference standard. BDue to high heterogeneity (sensitivity: I2=98%; specificity: I2=99%). CoE, certainty of evidence.

(B) The pooled sensitivity and pooled specificity for Clinical Question 2 were 85.4% (95% confidence interval [CI] 74.1–92.3%) and 95.4% (95% CI 87.3–98.4%), respectively. *Prevalences of 50%, 25%, and 1% were assumed according to the prevalences of ST-elevation myocardial infarction in the 6 included studies, which ranged from 1% to 50%. AThe results of the reference standard (physician diagnosis) may not be interpreted without the results of the index test in all studies. Further, electrocardiogram interpretation by a physician was used as the reference standard in most studies, and there may be incompleteness in the reference standard. BDue to high heterogeneity (sensitivity: I2=95%; specificity: I2=99.8%). CoE, certainty of evidence.

Results of Syntheses

shows forest plots summarizing the sensitivity and specificity values of the included studies and the pooled sensitivity and specificity values for CQ1 and CQ2. In CQ1 (paramedic ECG interpretation), the pooled sensitivity and specificity were 95.5% (95% confidence interval [CI] 82.5–99.0%) and 95.8% (95% CI 82.3–99.1%), respectively. In CQ2 (computer-assisted ECG interpretation), the pooled sensitivity and specificity were 85.4% (95% CI 74.1–92.3%) and 95.4% (95% CI 87.3–98.4%), respectively.

Figure 3.

Forest plots summarizing the sensitivity and specificity values of the included studies and pooled sensitivity and specificity values for Clinical Question (CQ) 1 and CQ2. CI, confidence interval.

shows the estimated absolute numbers of test positives (true positives and false positives) and test negatives (true negatives and false negatives) per 1,000 tested people calculated using the pooled sensitivity and specificity in CQ1 and CQ2. The assumed pretest probabilities of the target population in CQ1 were 30%, 20%, and 10% (), adopted according to the prevalence of STEMI in 4 included studies ranging from 12% to 33%. The estimated number of false positives was 38 per 1,000 (95% CI 8–159 per 1,000), with an assumed baseline risk of 10%. The estimated number of false negatives was 13 per 1,000 (95% CI 3–53 per 1,000), with an assumed baseline risk of 30%. The assumed pretest probabilities of the target population in CQ2 were 50%, 25%, and 1% (), adopted according to the prevalence of STEMI in 6 included studies ranging from 1.2% to 50%. The estimated number of false positives was 46 per 1,000 (95% CI 16–126 per 1,000), with an assumed baseline risk of 1%. The estimated number of false negatives was 73 per 1,000 (95% CI 38–129 per 1,000), with an assumed baseline risk of 50%.

Discussion

Using a systematic review, we investigated the diagnostic accuracy of identifying STEMI based on paramedic and computer-assisted ECG interpretation of prehospital 12-lead ECGs. Meta-analysis of paramedic ECG interpretation from 4 studies showed a pooled sensitivity and specificity of 95.5% and 95.8%, respectively. The estimated number of false positives was 38 per 1,000, with an assumed baseline risk of 10% as the maximum false positive rate, and the estimated number of false negatives was 13 per 1,000, with an assumed baseline risk of 30% as the maximum false negative rate. A meta-analysis of computer-assisted ECG interpretation from 6 studies showed a pooled sensitivity and specificity of 85.4% and 95.4%, respectively. The estimated number of false positives was 46 per 1,000, with an assumed baseline risk of 1% as the maximum false positive rate, and the estimated number of false negatives was 73 per 1,000, with an assumed baseline risk of 50% as the maximum false negative rate.

First, our results suggest that the diagnostic accuracy of paramedic prehospital ECG interpretation is favorable, with high pooled sensitivity and specificity, and an acceptable estimated number of false positives and negatives. However, careful interpretation is required when considering using the tool in clinical practice in certain areas because this meta-analysis consisted of only 4 observational studies over a prolonged time; furthermore, the paramedics in the 4 studies had received ECG training just before the studies. Different medical systems and different pretest probabilities of the target population may lead to different results. During the study selection process, we identified 8 studies (four prospective– and 4 retrospective cohort– studies) showing only the number of positive test results (true positives and false positives), which were excluded from the meta-analysis. When considering a total of 12 studies, consisting of the 8 excluded studies and the 4 studies included in the meta-analysis, the occurrence of false positives in all test positives (true positive plus false positive) ranged from 0% to 51%, suggesting considerable variability in the diagnostic accuracy of paramedic interpretation of prehospital ECGs among various medical systems. Therefore, when discussing whether to adopt a clinical practice in a specific medical area, individual assessment of diagnostic accuracy of each emergency medical service is required. However, it may be difficult to assess the diagnostic accuracy in patients suspected of STEMI in actual clinical practice. Some prior studies have used simulation tests for this assessment and have also shown an improvement in diagnostic accuracy with ECG training or computer-assisted interpretation.– Simulation tests could be useful in assessing the applicability of paramedic ECG interpretation and in helping introduce the new system in each emergency medical service. After appropriate education and assessment, paramedic ECG interpretation can be considered a helpful prehospital tool for the early diagnosis of STEMI.

Second, our results showed high pooled specificity with an acceptable estimated number of false positives for computer-assisted ECG interpretations. However, the pooled sensitivity was relatively low and the estimated number of false negatives was relatively high. This result may because we included studies with low prevalence or low sensitivity. Even then, these studies should not be excluded according to the predefined criteria. However, even if we excluded the study of Bosson et al because of the low prevalence of STEMI and the study of Bhalla et al because of low sensitivity, the sensitivity remained similar to that of the original meta-analysis and the specificity decreased; the pooled sensitivity and specificity were 84.5% (95% CI 74.7–90.9%) and 89.6% (95% CI 84.5–93.2%), respectively (). In addition, the result could be due to different computer algorithms providing a wide range of results. Computer programs have generally been improving, which could lead to lower false-positive and false-negative results. During the study selection process, we identified 6 retrospective cohort studies– reporting only the number of positive test results (true positives and false positives); these studies were excluded from the meta-analysis. However, when considering a total of 12 studies, consisting of the 6 excluded and 6 included studies, the occurrence of false positives in test positives (true positive plus false positive) ranged from 0% to 70%. This suggests that there is greater variability in diagnostic accuracy among the various computer algorithms. Further, a prior study directly compared diagnostic accuracies among different automated interpretation algorithms and showed significant differences in sensitivity and specificity. Therefore, the results do not refute the usefulness of computer-assisted ECG interpretations. When adopting a computer-assisted interpretation in a medical system, it is necessary to assess the diagnostic accuracy of the computer algorithm available in that area. At the very least, computer-assisted ECG interpretation could be useful, especially as an assistant tool.

In this study, paramedic- and computer-assisted ECG interpretations were investigated as potential prehospital approaches for the early diagnosis of STEMI. Other approaches can also be considered, such as prehospital ECG transmission to experts.– The most appropriate approach would differ and depend on the medical system and medical resources available in a particular country or area. In each medical system, a reasonable and feasible approach and/or composite should be considered to achieve an early STEMI diagnosis, leading to better patient prognosis.

Study Limitations

This study has several limitations. First, only 4 observational studies were included in the meta-analysis for CQ1 and 6 studies were included in the meta-analysis for CQ2 over a long study period from the 1990 s to 2020. The small number of studies included and high heterogeneity led to very low certainty of evidence. Second, there may have been undescribed computer-assisted ECG interpretation in the studies included for CQ1 that could have had an effect. Third, we extracted citations from the PubMed database only. Moreover, the situations in individual medical systems were not considered. Further accumulation of data is required to address these issues.

Conclusions

The results of our meta-analyses suggest that the diagnostic accuracy of paramedic prehospital ECG interpretation is favorable, with high pooled sensitivity and specificity. Furthermore, the estimated numbers of false positives and false negatives were considered acceptable. Computer-assisted ECG interpretations also showed high pooled specificity with an acceptable estimated number of false positives, although the pooled sensitivity was relatively low. Further accumulation of data is essential to establish high-quality evidence regarding these issues.

Sources of Funding

Funding was provided by the Japan Resuscitation Council and the Japanese Circulation Society Emergency and Critical Care Committee.

Disclosures

T. Matoba is a member of Circulation Reports’ Editorial Team. The other authors have no conflicts of interest to declare with regard to this article.

Author Contributions

All authors were involved in the study design. A.T. and M.K. identified the studies included in the meta-analysis and analyzed the data. A.T. and M.K. drafted the manuscript. Y.T., M.K., T. Matoba and H.N. reviewed the manuscript. All authors were involved in data interpretation and discussion. All authors had full access to all data (including statistical reports and tables) in the study and take responsibility for the integrity of the data and the accuracy of the analysis, and have read and approved the final manuscript.

Supplementary Text. Searching Formula Supplementary Figure. (CQ2) Computer ECG interpretation.