A Retrospective Clinical Evaluation of an Artificial Intelligence Screening Method for Early Detection of STEMI in the Emergency Department.

BACKGROUND: Rapid revascularization is the key to better patient outcomes in ST-elevation myocardial infarction (STEMI). Direct activation of cardiac catheterization laboratory (CCL) using artificial intelligence (AI) interpretation of initial electrocardiography (ECG) might help reduce door-to-balloon (D2B) time. To prove that this approach is feasible and beneficial, we assessed the non-inferiority of such a process over conventional evaluation and estimated its clinical benefits, including a reduction in D2B time, medical cost, and 1-year mortality.
METHODS: This is a single-center retrospective study of emergency department (ED) patients suspected of having STEMI from January 2021 to June 2021. Quantitative ECG (QCG™), a comprehensive cardiovascular evaluation system, was used for screening. The non-inferiority of the AI-driven CCL activation over joint clinical evaluation by emergency physicians and cardiologists was tested using a 5% non-inferiority margin.
RESULTS: Eighty patients (STEMI, 54 patients [67.5%]) were analyzed. The area under the curve of QCG score was 0.947. Binned at 50 (binary QCG), the sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) were 98.1% (95% confidence interval [CI], 94.6%, 100.0%), 76.9% (95% CI, 60.7%, 93.1%), 89.8% (95% CI, 82.1%, 97.5%) and 95.2% (95% CI, 86.1%, 100.0%), respectively. The difference in sensitivity and specificity between binary QCG and the joint clinical decision was 3.7% (95% CI, -3.5%, 10.9%) and 19.2% (95% CI, -4.7%, 43.1%), respectively, confirming the non-inferiority. The estimated median reduction in D2B time, evaluation cost, and the relative risk of 1-year mortality were 11.0 minutes (interquartile range [IQR], 7.3-20.0 minutes), 26,902.2 KRW (22.78 USD) per STEMI patient, and 12.39% (IQR, 7.51-22.54%), respectively.
CONCLUSION: AI-assisted CCL activation using initial ECG is feasible. If such a policy is implemented, it would be reasonable to expect some reduction in D2B time, medical cost, and 1-year mortality.

INTRODUCTION

ST-elevation myocardial infarction (STEMI) is a cardiovascular emergency with high mortality and morbidity.12 Rapid revascularization is the key to better patient outcomes. Delayed recognition of STEMI in the emergency department (ED) is the primary cause of delayed revascularization.3 However, forcing the clinicians to reduce the delay will increase false alarms because of the subtleness of electrocardiography (ECG) changes in the early phase of STEMI and many benign patterns mimicking STEMI.4 Therefore, many institutions use secondary confirmation by cardiologists to activate their cardiac catheterization laboratory (CCL). However, this may also require significant time, effort, and costs.

Artificial intelligence (AI) systems powered by deep learning technology have innovated many industries5 and are being actively adopted in the medical field.678910 If there is an AI system that can predict STEMI as accurate as a human cardiologist using initial ECG alone so that a triage nurse or an ECG technician can activate CCL directly, we can expect a significant reduction in door-to-balloon (D2B) time and cost as well as a significant improvement in patient outcomes.

It has been reported AI algorithms can outperform human experts in detecting some ECG abnormalities.511 However, it is unlikely that a human expert depends only on a single piece of information (e.g., initial ECG) when making a diagnosis of STEMI. Other information that might be considered includes vital signs, symptom description, and past medical history, as well as serial ECG measurements, echocardiogram, and even cardiac enzyme measurements. Currently, it is unknown whether an AI algorithm using only initial ECG can safely replace the clinical diagnostic process. In addition, it is also unknown how much benefit such replacement would provide additionally.

The objective of this study was two-fold. The first objective was to assess the non-inferiority of screening STEMI using AI interpretation of initial ECG only compared to the conventional screening process. The second objective was to estimate the clinical benefits, including a reduction in D2B time, medical cost, and 1-year mortality such process would provide.

METHODS

This is a retrospective study of ED patients suspected of having STEMI. The primary objective was to assess whether an AI system can achieve non-inferior diagnostic performance compared to the concerted screening efforts by emergency physicians (EPs) and cardiologists. The secondary goal was to estimate the benefits of the AI screening, such as reduction in D2B time, evaluation cost, and 1-year mortality.

The AI algorithm

The AI algorithm tested was a CNN-based binary classifier. It is a part of a previously built deep-learning system called Quantitative ECG (QCG™) capable of diagnosing various conditions, including shock, cardiac arrest, acute coronary syndrome, STEMI, non-specific myocardial injury, left heart failure, right heart failure, large pericardial effusion, pulmonary hypertension, hyperkalemia and 35 types of heart rhythms with various accuracies. The AI algorithm was trained using a transfer learning scheme where a modified CNN-based algorithm was pretrained on various open ECG datasets (49,731 recordings total) using a self-supervised learning scheme and fine-tuned on a clinical dataset of 47,194 annotated ECG images of over 32,968 patients who visited Seoul National University Bundang Hospital ED from 2017 to 2019. The algorithm has a signal extraction part using a series of morphological operation procedures and a multi-channel CNN network with 16 layers of convolution layers and a non-local network block with a single sigmoid activation function at the end. The probability output from the sigmoid function was calibrated using focal loss and temperature scaling method as described in a previous study.1213 The STEMI classifier outputs an estimated risk of STEMI (QCG score, a quantitative score ranging from 0 to 100), and we interpreted a score of 50 or more as positive for STEMI.

Study setting and participants and data preparation

The study facility was a tertiary academic hospital with over 80,000 annual patient visits. The ED’s acute chest pain protocol includes the following rules; 1) any patient with acute chest pain undergo ECG at triage and be screened for STEMI by emergency medicine (EM) physicians (EM professors or postgraduate year 3 to 5 EM residents), 2) if a patient is suspected of having STEMI a pre-activation warning call is made to the cardiologist on duty, 3) If the ECG is typical for STEMI, this leads to immediate CCL activation. Otherwise, the cardiologist examines the patient to decide whether to activate the CCL or cancel the whole process.

Patients with pre-activation warning call from January 1, 2021 to June 30, 2021 were included. Patients with delayed initial ECG (tested over 30 minutes after ED arrival) or insufficient coronary evaluation were excluded. The captured images of the initial ECGs of the patients were labeled based on their final diagnosis as assessed by reviewing the patient’s discharge note and outpatient follow-up visit records rather than ECG morphologic criteria. The labeling work was done by an emergency physician and was checked and confirmed by an interventional cardiologist.

For comparison, we assessed the performance of EPs’ interpretation of initial ECGs and the results of joint clinical evaluation by EPs and cardiologists. EPs’ interpretation of the initial ECGs was assumed to be positive if the pre-activation call was made within 30 minutes after the initial ECGs and negative otherwise. The results of the concerted efforts by EPs and cardiologists were assumed to be positive if the cardiologist confirmed the STEMI and activated the CCL and negative otherwise.

Statistical analysis

The performance of the AI classifier was evaluated based on sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the curve (AUC) of the receiver operating characteristic (ROC) curve. The non-inferiority of the AI classifier against the concerted clinical effort was tested with a priori non-inferiority margin of 5% in sensitivity and specificity. It was done by calculating 95% confidence interval (CI) of absolute difference (AI performance minus clinician performance) in sensitivity and specificity. If the lower margin of the CI is bigger than −5%, we can be sure at least 95% that the AI classifier is not inferior over 5% in sensitivity (or specificity) compared to the clinical efforts.

The estimated reduction in D2B time was calculated assuming that direct CCL activation is made within 3 minutes after initial ECG acquisition. The assumption was based on an internal survey on how many minutes will be required for brief history taking to get essential information for the activation. The average reduction of D2B time was calculated following this equation.

The average reduction in cost was calculated based on the sum of the expenses for additional diagnostic tests, including ECG (6,200 KRW = 5.27 USD, November 16, 2021) and bedside portable echocardiography (72,440 KRW = 61.62 USD, November 16, 2021) during the interval between the initial ECG and CCL activation in confirmed STEMI cases following this equation.

The reduction in 1-year mortality if the algorithm was implemented was estimated based on the results of a previous study on Korea Acute Myocardial Infarction Registry (KAMIR).14 The study estimates the 1-year mortality benefit of reducing 30 minutes of D2B time for each D2B time category. Assuming 1-hour interval between CCL activation and ballooning event, we estimated median absolute risk reduction of 1-year mortality for each patient. Its 95% CI was calculated using bootstrapping with 5,000 repetitions.

Categorical variables were reported using frequencies and proportions. Continuous variables were reported using mean and standard deviation (SD) or median and interquartile range (IQR) as appropriately. The performance of the AI algorithm was compared to its human counterparts (EPs and concerted effort by EPs and cardiologists) based on AUC, sensitivity, specificity, PPV, and NPV. AUC was compared using DeLong’s method.15 Sensitivity and specificity were compared using the McNemar test, and PPV and NPV were compared using relative predictive values, as proposed by Moskowitz and Pepe.16 The CI of difference of sensitivity and specificity between two diagnostic modalities was calculated using Wald method as implemented in R-package, DTComPair. P values < 0.05 were considered significant.17 All data handling and statistical analyses were performed using R-packages version 3.3.2 (R Foundation for Statistical Computing, Vienna, Austria).

Ethics statement

The Institutional Review Board of Seoul National Bundang Hospital approved the analysis and waived the informed consent requirement (IRB Number: B-2111-723-114).

RESULTS

After excluding six patients with delayed initial ECG and two patients who had expired before coronary artery evaluation (one acute type-A aortic dissection and the other one with isolated AVR lead ST elevation), a total of 80 patients were included (Table 1). The mean age was 64.7 (SD, 13.8), and STEMI was confirmed in 54 (73.8%) patients. The most common presenting symptom was chest pain (n = 71, 88.8%), and the most common diagnosis other than STEMI was NSTEMI. The mean and median time required for CCL activation, minutes between initial ECG and cardiologists’ confirmation, was 34.1 (SD, 73.4) and 14.0 (IQR, 10.3–23.0) minutes, respectively.

Table 1

Patient characteristics

Characteristics			Values
Age, yr			64.7 ± 13.8
Sex, No. (%)
	Female		17 (21.2)
	Male		63 (78.8)
Chief complaints, No. (%)
	Chest pain		71 (88.8)
	Dyspnea		7 (8.8)
	Epigastric pain		2 (2.5)
Troponin I
	Not STEMI		0.0 (0.0–0.5)
	STEMI		0.8 (0.1–9.8)
Binary troponin, No. (%)
	Not Elevated		48 (60.0)
	Elevated (≥ 0.6)		32 (40.0)
Time delays in STEMI cases, minutes
	Door to ECG		4.5 (3.0–7.8)
	ECG to pre-activation		25.7 ± 72.8
		Median (IQR)	7.5 (5.0–12.0)
	ECG to CCL activation		34.1 ± 73.4
		Median (IQR)	14.0 (10.3–23.0)
Additional tests for STEMI patients, minutes
	Bedside echocardiography		18 (33.3)
	Number of additional ECG		24 (44.4)
		1	19 (35.2)
		2	3 (5.6)
		3	2 (3.7)
QCG score (initial ECG)			68.5 ± 38.6
Binary QCG (initial ECG), No. (%)
	Not STEMI		21 (26.2)
	STEMI		59 (73.8)
EPs’ decisions, No. (%)
	Pre-activation done		35 (43.8)
	Delayed		45 (56.3)
Cardiologists’ decisions (Joint evaluation), No. (%)
	Not STEMI		18 (22.5)
	STEMI		62 (77.5)
Final clinical diagnosis, No. (%)
	STEMI		54 (67.5)
	Not STEMI		26 (32.5)
		Atrial flutter	1 (1.2)
		Brugada syndrome	1 (1.2)
		DCM	1 (1.2)
		HCM	2 (2.5)
		Lung cancer	1 (1.2)
		Myocarditis	2 (2.5)
		Non-cardiac	1 (1.2)
		NSTEMI	9 (11.2)
		Pancreatic cancer	1 (1.2)
		Pericarditis	2 (2.5)
		Pneumonia	1 (1.2)
		Pulmonary thromboembolism	1 (1.2)
		Unstable angina	2 (2.5)
		Variant angina	1 (1.2)

Values are expressed as mean ± standard deviation, number (%), or median (IQR).

STEMI = ST-elevation myocardial infarction, IQR = interquartile range, ECG = electrocardiography, QCG = quantitative electrocardiography, EP = emergency physician, DCM = dilated cardiomyopathy, HCM = hypertrophic cardiomyopathy.

The AUC of QCG score was 0.947 (Fig. 1), which was significantly higher than that of EPs’ initial decision (0.710, P < 0.001) and joint decision by EPs and cardiologists (0.761, P < 0.001). If the QCG score was binned at 50 (binary QCG), so that the result is positive if the score is 50 or more and negative otherwise, the AUC was 0.875, which was still the highest. However, the difference was statistically significant only when compared to EPs’ initial decision (P = 0.011). The sensitivity, specificity, PPV and NPV of binary QCG were 98.1% (95% CI, 94.6%, 100.0%), 76.9% (95% CI, 60.7%, 93.1%), 89.8% (82.1%, 97.5%), and 95.2% (86.1%, 100.0%), respectively (Table 2). The sensitivity and NPV were significantly higher than EPs’ (both P < 0.001).

Fig. 1

ROC curve of QCG score (black solid line), binary QCG (score ≥ 50, blue dashed line), EPs (brown dashed line), and cardiologists (joint evaluation, red dashed line) in the prediction of STEMI. Binary QCG had the highest AUC, which was significantly higher than that of EPs.

ROC = receiver operating characteristic, QCG = quantitative electrocardiography, EP = emergency physician, STEMI = ST-elevation myocardial infarction, AUC = area under the curve.

Table 2

Diagnostic performance of EPs, joint evaluation, and binary QCG

Metrics	EPs’ single readinga	Joint evaluationb	Binary QCG of initial ECGs	P value (vs. EPs)	P value (vs. Joint evaluation)
Sensitivity	57.4 (44.2–70.6)	94.4 (88.3–100.0)	98.1 (94.6–100.0)	< 0.001	0.317
Specificity	84.6 (70.7–98.5)	57.7 (38.7–76.7)	76.9 (60.7–93.1)	0.480	0.132
PPV	88.6 (78.0–99.1)	82.3 (72.7–91.8)	89.8 (82.1–97.5)	0.828	0.113
NPV	48.9 (34.3–63.5)	83.3 (66.1–100.0)	95.2 (86.1–100.0)	< 0.001	0.239

EP = emergency physician, ECG = electrocardiography, QCG = quantitative electrocardiography, PPV = positive predictive value, NPV = negative predictive value, CCL = cardiac catheterization laboratory.

aAssumed to be positive if the pre-activation call was made within 30 minutes after the initial ECGs; bAssumed to be positive if the cardiologist confirmed the activation of the CCL. Additional serial ECGs and bedside echocardiography were done in some patients before the confirmation.

The difference in sensitivity and specificity between Binary QCG and the joint clinical decision was 3.7% (−3.5%, 10.9%) and 19.2% (−4.7%, 43.1%), respectively (Table 3). As the absolute values of their lower margins were smaller than the non-inferiority margin of 5%, it was confirmed that the Binary QCG was at least as accurate as the joint decision by EPs and cardiologists.

Table 3

Non-inferiority test of binary QCG against the joint evaluation

Metrics	Joint evaluation	Binary QCG	Difference	Non-inferiority
Sensitivity	94.4 (88.3, 100.0)	98.1 (94.6, 100.0)	3.7 (−3.5, 10.9)	Confirmed (−3.5 > −5)
Specificity	57.7 (38.7, 76.7)	76.9 (60.7, 93.1)	19.2 (−4.7, 43.1)	Confirmed (−4.7 > −5)

QCG = quantitative electrocardiography.

The average (median) reduction in D2B time was estimated to be 11.0 (IQR, 7.3–20.0) minutes (Table 1, Fig. 2). There were 18 bedside echocardiography and 24 additional ECGs, whose overall cost was 1,452,720 KRW in 54 STEMI patients, and the estimated reduction in evaluation cost was 26,902.2 KRW (22.78 USD) per STEMI patient. The average (median) relative risk reduction of 1-year mortality was calculated to be 12.39 (IQR, 7.51–22.54) %. There were 11 (42.3%) emergency CCL activations for the false-positive cases when the joint clinical decision was used, which averages 25.4 minutes and 244,467 KRW per patient without STEMI. If the Binary QCG were to be applied to the same patients, the number of false-positive CCL activations decreases to 6 (23.1%) with an estimated decrease in the time and the cost of 45.5%.

Fig. 2

Event progress of study subjects in ED. (A) Event progress since ED arrival in confirmed STEMI patients: Circles, time of initial ECG; Vertical line, time of call by EM physicians; Squares, time of confirmation by cardiologists; Cyan, predicted as STEMI; Pink, predicted as not STEMI. (B) Event progress since ED arrival in patients confirmed to be free of STEMI: Circles, time of initial ECG; Vertical line, time of call by EM physicians; Squares, time of CP confirmation by cardiologists; Cyan, predicted as STEMI; Pink, predicted as not STEMI.

ED = emergency department, STEMI = ST-elevation myocardial infarction, ECG = electrocardiography, EM = emergency medicine, CP = coronary perforation.

DISCUSSION

This study reports that AI interpretation of initial ECG to screen STEMI in ED is non-inferior to joint clinical evaluation by EPs and cardiologists using bedside echocardiography and serial ECGs if required. In addition, we could estimate the possible benefits of CCL activation using the AI system such as reduction in D2B time, medical cost, and 1-year mortality risk.

Rapid revascularization is essential in improving patient outcomes in STEMI.14181920 One of the most critical factors in reducing D2B time is rapid diagnosis. The 2017 European clinical guideline for STEMI states that the delay between first medical contact (FMC) and STEMI diagnosis should be ≤ 10 minutes both in hospitals and emergency medical services (EMSs). However, the goal is challenging even in typical STEMI cases considering the minimum time required for ECG tests and brief history.

We think AI algorithms can play vital roles if the following qualities are met in this situation. First, the algorithm should be as accurate as healthcare professionals armed with additional diagnostic information such as patient history and physical examination, bedside echocardiography, and serial ECG measurements. Second, such an algorithm should provide probabilistic information so that clinical policies incorporating the AI system can deal with uncertainty related to its diagnosis. Third, the algorithm should be universally available to users to consistently apply such a policy. The algorithm tested in this study satisfied all three of the requirements. It showed non-inferior accuracy using only initial ECG. It provides a probabilistic output so that users can apply different decision thresholds that can accommodate various clinical situations. Lastly, it can be used universally by using printed ECG images instead of raw signals.

The cost change we estimated only considered the reduction in serial ECGs and bedside echocardiography tests. However, reduction in D2B and thus earlier life-saving treatment might lead to other types of cost reduction such as reduced length of stay or overall treatment cost.21222324 It would be worthwhile to see the effect of highly-performing AI decision aids on overall healthcare cost in real-life situations through prospective studies.

The AI algorithm used 12-lead ECG images instead of raw waveform data. This type of technology can be applied in many clinical situations where raw waveform data is not available to the users. For example, emergency medical technicians can use their existing ECG machines and analyze the printed materials using their smartphones to activate CCL directly. The core benefits of this approach are the high cost-effectiveness where there is no need to buy new AI-enabled ECG machines and the high scalability where already existing application distribution services such as Apple or Android app store can be utilized to distribute the technology. Interestingly, there had been a mobile phone-based approach using crowdsourcing instead of AI interpretation.25 We think combining these two approaches involving both human experts and AI systems will be synergistic in promoting both technologies.

This study has several limitations. First, this is a retrospective study. We do not think there would be any significant difference in performance measurements by study design, prospective or retrospective, as the input data will be the same. However, it is possible the estimation of the potential benefits of AI application could be affected. Second, this is a single-center study. Although the ECG report formats used for 12 lead ECGs are almost identical among hospitals in South Korea, there could be some exceptions that might affect the algorithm's performance. Third, the findings in the study should be externally validated, preferably on a more diverse population from multiple hospitals, countries, and races. Although we think 12-lead ECG is a well-standardized clinical test, some racial differences in ECG findings have been reported.26 Future studies on how the algorithm performs in multicenter settings, especially in non-east Asian populations would be important.

In conclusion, AI interpretation of initial ECGs was non-inferior to joint clinical evaluation by EPs and cardiologists in screening STEMI in ED. Therefore, CCL activation based only on AI interpretation of initial ECG is feasible. If such a policy is implemented, it would be reasonable to expect some reduction in D2B time, medical cost, and 1-year mortality.