Detection and Classification of Cardiac Arrhythmias by a Challenge-Best Deep Learning Neural Network Model.

Electrocardiograms (ECGs) are widely used to clinically detect cardiac arrhythmias (CAs). They are also being used to develop computer-assisted methods for heart disease diagnosis. We have developed a convolution neural network model to detect and classify CAs, using a large 12-lead ECG dataset (6,877 recordings) provided by the China Physiological Signal Challenge (CPSC) 2018. Our model, which was ranked first in the challenge competition, achieved a median overall F1-score of 0.84 for the nine-type CA classification of CPSC2018's hidden test set of 2,954 ECG recordings. Further analysis showed that concurrent CAs were adequately predictive for 476 patients with multiple types of CA diagnoses in the dataset. Using only single-lead data yielded a performance that was only slightly worse than using the full 12-lead data, with leads aVR and V1 being the most prominent. We extensively consider these results in the context of their agreement with and relevance to clinical observations.

Introduction

Cardiac arrhythmias (CAs) are harbingers of cardiovascular diseases and the potential associated mortality (Kibos et al., 2013). CAs are usually diagnosed from electrocardiograms (ECGs), a noninvasive, inexpensive, and widely used clinical method for monitoring heart function. The diagnosis of CAs is based on wave-like features, such as the P wave, QRS wave, and T wave, of ECGs. A complete ECG usually contains recordings from six limb leads (I, II, III, aVR, aVL, aVF) and six chest leads (V1, V2, V3, V4, V5, V6), with each lead measuring electrical activity from a different angle of the heart, covering both the vertical plane (limb leads) and the horizontal plane (chest leads) (Malmivuo et al., 1995, Wilson et al., 1954).

The different leads exhibit distinct features of ECG signals that are associated with specific types of CA. For example, atrial fibrillation (AF) is characterized by the fibrillatory atrial waves and irregular conduction of QRS (Bayes de Luna et al., 1988, Platonov et al., 2012). Left bundle branch block (LBBB) is diagnosed by the distinct QRS morphology at leads I, aVL, V1, V2, V5, and V6, whereas right bundle branch block (RBBB) is diagnosed by the rsR′ pattern at V1 and V2 (Surawicz et al., 2009). First-degree atrioventricular block (I-AVB) is defined as constant PR intervals longer than 0.2 s (Wesley, 2016). The premature atrial contraction (PAC) and premature ventricular contraction (PVC) indicate the electrical impulse from an abnormal site; specifically, the P wave or QRS morphology of PAC and PVC differs from that in normal heart beats (Garcia and Miller, 2004, Kobayashi, 2018). ST segment is abnormal if either ST-segment elevation (STE) or ST-segment depression (STD) is greater than 0.1 mV (Hanna and Glancy, 2011).

To reliably recognize these complex CA-associated ECG characteristics, considerable training is required. Indeed, studies have shown that internists or cardiologists sometimes misdiagnose CA types (Hannun et al., 2019, Shiyovich et al., 2010). The significant growth of ECG examination, which increases physicians' workload, exacerbates the problem. This problem might be alleviated by developing computer algorithms that produce accurate and automatic diagnosis to assist the physicians. Although such a task would be difficult owing to the large variance in the geometrical and physiological features of ECG signals (Hoekema et al., 2001), significant progress has been made, especially in recent years (Lyon et al., 2018).

Two general approaches are available for developing an automatic CA diagnostic tool. The first splits ECG signals into units of the heartbeat, or cycles of the characteristic ECG waveforms. Thus, even with a small number of subjects, this beat-based approach can generate a large amount of data for machine learning to train predictive classification models. However, extracting ECG morphological features to delineate ECG signals is challenging because it is often an imprecise undertaking (Lyon et al., 2018). Although prediction accuracies as high as >99% have been reported in beat-based studies (Lyon et al., 2018), they could be confounded by both training and test beats coming from the same individuals. This issue is illustrated by a study in which test beats were taken from patients who were not included in the training set; the cross-validation accuracy of classification for six types of CA decreased from 99.7% to 81.5% (Qin et al., 2017).

The MIT-BIH Arrhythmia Database (MIT-BIH AD) (Goldberger et al., 2000, Moody and Mark, 2001) and the UCI Machine Learning Repository: Arrhythmia Data Set (UCIAD) (Guvenir et al., 1997), which, respectively, contain only 48 and 452 subjects, have been the source of publicly available ECG data for most previous CA prediction studies. However, databases with such small numbers of subjects can cause over-fitting problems for classification, especially for neural network algorithms (Begg, 2006). Data over-fitting can also arise from significantly unbalanced data, as with one or a few CA types being over-represented among cases. These problems can produce biased results from analyses of MIT-BIH AD and UCIAD (Mustaqeem et al., 2018, Nayak et al., 2016). For instance, in a study using UCIAD, a high accuracy (92%) of CA classification was achieved when 80% of the data were used for the training set and the remaining 20% for the test set, but the accuracy dropped to only 60% when the training-test split was 50-50 (Mustaqeem et al., 2018). Additional drawbacks are that ECG data in MIT-BIH AD only include two leads (e.g., leads II and V1, II and V5, II and V4, and V2 and V4), whereas the UCIAD only has extracted features available (average width of Q, amplitude of Q, etc.), not the raw data from 12-lead ECGs.

The second approach provides an end-to-end solution, avoiding the main difficulty of the beat-based approach. This approach requires a very large ECG database as well as the construction of a suitable deep learning artificial neural network to take advantage of it. Developments in both factors in recent years have made the second approach increasingly attractive. For example, to promote open-source research, the PhysioNet/Computing in Cardiology Challenge 2017 (CinC2017) released single-lead (lead I) ECG data from 8,528 subjects with four types of heart rhythms (AF, normal, other rhythms, noise) to the public (Clifford et al., 2017). Using convolutional neural network (CNN) plus three layers of long short-term memory (LSTM, one kind of recurrent neural network [RNN]), Xiong et al. (2018) produced the top performance in CinC2017 with an F1 score (the harmonic mean of the precision and recall) of 0.82 on its hidden test set (3,658 subjects).

Similar to CinC2017, the China Physiological Signal Challenge 2018 (CPSC2018), hosted by the seventh International Conference on Biomedical Engineering and Biotechnology (Liu et al., 2018), released a large ECG database for free download and set aside a hidden test set to assess models submitted by challenge participants from around the world. In contrast to CinC2017, CPSC2018 used 12-lead ECG data and subjects were grouped according to normal heart rhythm and eight types of CA: AF, I-AVB, LBBB, RBBB, PAC, PVC, STD, and STE. This represents the largest 12-lead ECG database with the most labeled CA types in the public domain to date. Here, we report a deep learning artificial neural network modeling of the CPSC2018 ECG data, and the results that won the first place in the competition.

Results

Construction of a CNN-Based Model of CAs

Figure 1 depicts the architecture of our model for automating recognition of the CAs labeled in the CPSC2018 dataset. The model and the data used are described in more detail in Transparent Methods of the Supplemental Information. Briefly, the model consists of five CNN blocks, with several other types of neural network layers appended to achieve optimal performance while reducing over-fitting. To derive the model, CPSC2018's open-source ECG dataset was randomly divided into 10 equal folds, with 8 of the 10 folds serving as the training set and each of the two remaining folds serving as the validation set and the test set, respectively. This 10-fold cross-validation procedure of machine learning was repeated to produce hundreds of trained models, and the models producing the best validation results were selected for further evaluation on the test folds. For each model, a subject, identifiable by a unique ID number, would appear only once, exclusively in one of the three subsets (training, validation, or test). Single-lead models with the same architecture were similarly derived using this procedure. An ensemble model combining the best validation models of both 12-lead models and single-lead models was submitted to compete in the CPSC2018 challenge.

Figure 1

The Architecture of Deep Learning Artificial Neural Network for 12-Lead ECG CA Detection and Classification

Layers and blocks are specified in rectangle boxes; “X5” indicates that five CNN blocks are tandem connected before connecting to the bidirectional RNN layer, which is a gated recurrent unit layer. The output layer at the bottom contains the probabilities predicted by the model for each of the nine types of the CA classification. The type with the highest probability is the type predicted by the model for the input ECG recording.

Best Validation Models on 10-Fold Tests and Ensemble Model on Hidden Test

In Table 1, for each CA type the median accuracy, AUC (area under the receiver operating characteristic curve), and F1-score for the ten 10-fold tests from the best validation models are compared with those of the ensemble model, as well as with the F1-score of the ensemble model on the hidden test set of CPSC2018. The comparisons show that the ensemble model performed somewhat better than the best validation models, which is expected because the former combined and optimized the latter to produce the best 10-fold test results (see Methods). In addition, the ensemble model's performance was quite stable across all CA types, from the publicly available data to the hidden test data, reflecting the fairly similar compositions of the two sets of data, as mentioned in Transparent Methods.

Table 1

Comparison of Model Performances on Tests

	Best Validation Models			Ensemble Model
CA Typ	Median Accuracy	Median AUC (95% CI)	Median F1-Score	Median Accuracy	Median AUC (95% CI)	Median F1-Score	Hidden Set F1-Score
Normal	0.940	0.890 (0.810–0.942)	0.795	0.949	0.867 (0.832–0.973)	0.808	0.801
AF	0.969	0.928 (0.902–0.985)	0.897	0.983	0.963 (0.914–0.993)	0.944	0.933
I-AVB	0.972	0.899 (0.864–0.988)	0.865	0.977	0.950 (0.875–0.990)	0.899	0.875
LBBB	0.990	0.914 (0.748–1.000)	0.821	0.995	0.942 (0.763–1.000)	0.899	0.884
RBBB	0.955	0.956 (0.887–0.988)	0.911	0.952	0.946 (0.871–0.976)	0.903	0.910
PAC	0.957	0.867 (0.749–0.955)	0.734	0.963	0.920 (0.779–0.981)	0.797	0.826
PVC	0.970	0.928 (0.841–0.988)	0.852	0.977	0.932 (0.864–0.996)	0.874	0.869
STD	0.951	0.878 (0.797–0.972)	0.788	0.959	0.906 (0.815–0.970)	0.834	0.811
STE	0.976	0.707 (0.558–0.995)	0.509	0.977	0.773 (0.603–0.993)	0.600	0.624

Results are from the best validation models and the ensemble model on the ten 10-fold tests, except for those in the last column (boldfaced), which are the ensemble model's median F1-scores for the hidden test set of CPSC2018 reported at its website http://2018.icbeb.org/Challenge.html, which did not provide accuracy or AUC results.

Table 1 also reveals differential difficulties in predicting CA types. Namely, the prediction accuracy decreased from AF, bundle branch blocks, and premature contractions to ST abnormalities, with the normal type being one of the more difficult-to-predict types. The model's prediction for STE had the lowest F1-score (0.5–0.6), which may be due in part to physicians' variable opinions on how to diagnose STE (McCabe et al., 2013). The same trend, including the prediction of the normal type, was observed in all other top-performing models of CPSC2018 (Table S1). Indeed, almost all the top models produced very high F1-scores (>0.9) for AF and bundle branch blocks. Our model had significantly better predictions than the other models for several CA types, especially PAC, PVC, STD, and STE. This outcome explains how we outperformed others (Table S1). However, it should be noted that all top models performed well (overall F1-score > 0.8), and the difference between our model and the second-place model was minimal (Table S1).

Concurrent CA Types

One reason that models perform less accurately for certain CA types is that multiple CA types are predicted with almost equal probabilities for some patients. Figure 2 displays the probabilities output by the best validation models for ECG subjects when they were in the test fold of the 10-fold tests. As may be seen, normal, STD, and STE lack a probability score that can make them stand out from the other eight types, which is consistent with the model's performance results presented in Table 1. Further analysis on model probabilities showed that, for many patients with AF, a common concurrent CA was RBBB, whereas RBBB was often concurrent with PAC and PVC, in addition to AF (Figure 2). These probability results of concurrent CAs agreed well with the statistics for the 476 multi-labeled subjects; specifically, the three most multi-labeled incidences in these subjects were AF/RBBB, RBBB/PAC, and RBBB/PVC (Table 2). An ensemble model without these 476 multi-labeled subjects being added back to the training set (see Methods) performed well in predicting these multiple types of CA (Tables S3 and S4), indicating the model's ability to capture ECG features of concurrent CAs. These results are also generally compatible with clinical observations that rate-dependent (phase 3) block during ectopic atrial beats or AF can lead to RBBB (Nielsen et al., 2010, Gertsch, 2016). However, a larger dataset of multi-labeled subjects is required to fully evaluate our model's performance on concurrent CA diagnoses.

Figure 2

Probabilities Output by the Best Validation Models in the Test Folds of the 10-Fold Tests

On the right is the color-coded probability scale.

Table 2

Label Count Statistics of the 476 Multi-Labeled Subjects in the Released CPSC2018 Dataset

	AF	I-AVB	LBBB	RBBB	PAC	PVC	STD	STE
AF	0	0	29	172	4	8	33	2
I-AVB		0	8	10	3	5	6	4
LBBB			0	0	10	6	3	4
RBBB				0	55	51	20	19
PAC					2	3	6	5
PVC						0	18	2
STD							0	2
STE								0

Only the upper triangle portion of the symmetrical concurrent CA label counts is shown. The three largest counts are boldfaced.

Model Performances with Single-Lead Data

The median F1-scores for models for a single lead in the 10-fold tests are presented in Figure 3. The performances for the best validation models using the 12-lead data in Table 1 were largely replicated by those using only single-lead data. In most cases, only minimal changes of F1-scores for the classification of individual CA types were noted between the analyses of 12-lead and single-lead ECGs. The results also indicate aVR was one of the best-performing single leads, with its performance ranking first in the overall average and the three individual CA types (normal, AF and STD), as well as within the top three for all CA types except STE and PAC. Another well-performing single lead is lead V1, which ranked first in three types (I-AVB, RBBB, and PAC) but did worse than most other leads for other types. In comparison, lead I, which was used by Apple Watch (Apple, 2019), was not as remarkable in our tests. Lead II, which is favored among the 12 leads by physicians for a quick impression of an ECG recording due to its clearest signal (Beebe and Myers, 2012), ranked fifth in the overall average but was statistically no different from the best performing leads (p value of paired t test < 0.05). These results are largely supported by a Bayes factor analysis (Goodman, 1999) that rigorously assessed statistical differences between these leads (see Tables S5–S7).

Figure 3

The Ranked F1-Score Results of Single-Lead Models

The F1-scores (on the y axis) are from the single-lead models performed on the 10-fold tests (see Methods). Lead aVR is shown in red, V1 in green, I in blue, and II in orange.

These performance rankings suggest that the current model identified the lead-specific morphology of the various CA types. For examples, the deep and broad S-waves in lead V1 and the broad clumsy R-waves in V6 have been used for the diagnosis of LBBB (Podrid et al., 2015), and V1 and V6 were identified as having the leading performance among the single leads. Meanwhile, the diagnostic criteria of RBBB included the rSR′ pattern in leads V1 and V2 (Chugh, 2014), which were also selected as top-performing single leads.

Discussion

In recent years, deep learning of artificial intelligence (AI) has been successfully used to make medical diagnoses (Esteva et al., 2019). The present work for CA detection and classification is related to the competition in CinC2017 (Clifford et al., 2017) and recently published studies (Clifford et al., 2017, Hannun et al., 2019). A direct comparison of performance between different studies is difficult because not all of them used publicly available ECG data and different CA types and type numbers were predicted. The complexity of these deep learning models also differed; for example, our model had a total of 18 neural network layers, compared with 33 (Hannun et al., 2019) and 5–7 (Clifford et al., 2017) in others.

Our model can be best compared with one reported very recently (Yao et al., 2020). The model of that study used an architecture that is quite similar to ours, albeit with some significant differences in detail as discussed in Transparent Methods, and its performance on the same CPSC2018 hidden test set was reported. As the comparisons in Table S8 showed, Yao et al.’s model achieved a better F1-score than our best validation models for most of the CA types, but the reverse was true when it was compared with our ensemble model. This reinforces the notion that an ensemble model can often achieve an even better performance than those achievable by individual models alone.

Interestingly, all these recent studies achieved an overall F1-score of 0.82–0.84 in predicting CA types. Although not fully tested in real-world scenarios, AI-based ECG diagnosis of CAs has been shown to significantly improve diagnosis accuracy, compared with general physicians and cardiologists (Hannun et al., 2019, Shiyovich et al., 2010) (also see Table S2 for a very small sampling). Therefore, these AI models are capable of reducing erroneous diagnoses and medical overload. Although this outcome is very encouraging, it is sobering to remember that until most of the “ground truth” CA diagnoses used to derive AI models are made by expert cardiologists, further improvement of model accuracy may be limited.

Our analyses suggest that the models built on single-lead information could predict CA types with minimal differences in performance from those based on 12 leads. The clinical diagnostic criteria of CA types are often lead specific. The top-ranking single lead for RBBB or LBBB in our model was compatible with the leads in the diagnostic criteria for RBBB and LBBB (Surawicz et al., 2009), solidifying the validity of the present AI diagnostic model. The performance of aVR, a lead that is often clinically ignored, in our AI model is intriguing and deserves attention. The leads I, II, and V1 are conventionally used as the modified leads in continuous monitoring or mobile devices for ECG (Apple, 2019, Brunner et al., 2010). In our AI model, aVR could predict several CA types with better performance than the conventional leads. Rather than considering reciprocal information from the left lateral side, the purpose of lead aVR is to obtain specific information from the right upper side of the heart, including the outflow tract of the right ventricle and the basal part of the septum. The vector of lead aVR is parallel to the anatomical and corresponding electrical axis from atrial base to ventricular apex, and thus it may maximize the electrical signals of atrial and ventricular depolarization. These factors may give the unique role of aVR to diagnose CAs with a potential mechanism to outperform the other leads. In comparison, lead I, which is used in Apple Watch for AF detection (Krueger, 2018), did not perform as well in our analysis. Our results suggest that the best predictive single lead for different CA types could be different for clinical applications. Our results may provide an impetus for future studies to investigate the potential use of lead aVR in different CA types and ECG devices (wearable or portable).

CAs are complex and concurrent CA types are not uncommon, especially for those that are related in cardiac electrophysiology. Although ECG-based CA diagnostic models have so far focused only on single-type predictions, our analysis shows that AI is capable of multi-type CA diagnosis. Detection and classification of concurrent CAs should be a subject for future studies, and our model is a first step in that direction. ECGs have been shown to be capable of disease/health detection beyond CAs, including, for example, the prediction of asymptomatic left ventricular dysfunction (Attia et al., 2019) and non-invasive potassium tracking (Attia et al., 2016). As methods of AI machine learning continue to advance and become friendlier for non-AI specialists to employ, we can expect ECGs to be explored for their diagnostic power in many more diseases and clinical applications.

Limitations of the Study

Although our model produced state-of-the-art accuracy on the hidden test set of CPSC2018, which was collected from 11 hospitals in China, it will most likely still need further refinement to achieve the same level of performance for other ECG datasets. However, a recent study (Hannun et al., 2019) has shown that this will probably only require transfer learning, that is, keeping the same model architecture while retraining network weights for the new data. Additionally, other deep learning architectures may exist that can achieve even better CA prediction accuracy and, as the comparisons with Yao et al.’s study (Yao et al., 2020) showed, even within a similar network architecture, different models (e.g., using different number of trainable parameters) exist to achieve similar performances. Furthermore, although different CA types might be better modeled by using information from different single leads, it remains to be studied whether different methods, or different network architectures, should be used for different single leads. In the absence of a systematic evaluation approach, as well as the lack of a truly gold standard training set, as alluded to above, these limitations are difficult to address at present.

Methods

All methods can be found in the accompanying Transparent Methods supplemental file.