Diagnostic utility of artificial intelligence for left ventricular scar identification using cardiac magnetic resonance imaging-A systematic review.

Background: Accurate, rapid quantification of ventricular scar using cardiac magnetic resonance imaging (CMR) carries importance in arrhythmia management and patient prognosis. Artificial intelligence (AI) has been applied to other radiological challenges with success. Objective: We aimed to assess AI methodologies used for left ventricular scar identification in CMR, imaging sequences used for training, and its diagnostic evaluation.
Methods: Following PRISMA recommendations, a systematic search of PubMed, Embase, Web of Science, CINAHL, OpenDissertations, arXiv, and IEEE Xplore was undertaken to June 2021 for full-text publications assessing left ventricular scar identification algorithms. No pre-registration was undertaken. Random-effect meta-analysis was performed to assess Dice Coefficient (DSC) overlap of learning vs predefined thresholding methods.
Results: Thirty-five articles were included for final review. Supervised and unsupervised learning models had similar DSC compared to predefined threshold models (0.616 vs 0.633, P = .14) but had higher sensitivity, specificity, and accuracy. Meta-analysis of 4 studies revealed standardized mean difference of 1.11; 95% confidence interval -0.16 to 2.38, P = .09, I2 = 98% favoring learning methods.
Conclusion: Feasibility of applying AI to the task of scar detection in CMR has been demonstrated, but model evaluation remains heterogenous. Progression toward clinical application requires detailed, transparent, standardized model comparison and increased model generalizability.

Background and objectives

Accurate identification of cardiac scar is growing in importance. Previous studies demonstrate that ventricular scar volume/mass is associated with ventricular arrhythmia episode risk and response to medical and device therapies. Descriptive delineation can improve electrophysiological procedural outcomes. Subsequently, scar metrics are increasingly integrated into clinical practice as a decision aid guiding patient therapy.

Cardiac magnetic resonance imaging (CMR) is the current gold standard for tissue characterization. However, expert manual delineation is time-consuming, with significant interoperator variability. Methods to improve objective quantification through means such as thresholding and full-width half-maximum can reduce time and improve reproducibility. However, outcome variability between these methods exists and none are preferentially recommended., Moreover, these techniques are often solely based on signal intensity and may require further human postprocessing (Figure 1).

Figure 1

Simplified comparison of distinct cardiac magnetic resonance image segmentation methods. Far-left image demonstrates a short-axis view of transmural septal scar. In subsequent images, green represents epicardial border, magenta endocardial, and purple segmented scar for thresholding (top) and manual (bottom) techniques.

Artificial intelligence (AI) is a broad term encompassing a multitude of functions, including the undertaking of tasks normally requiring human intelligence. Subsets including machine learning (ML) and deep learning (DL) have been investigated as solutions to clinical challenges. The use of AI in general medical and cardiac imaging research has developed rapidly owing to technological advances and the data-rich environment of imaging. Models ranging from simple task automation to deep investigative associations have been developed.

Classical AI methods typically make decisions based on hand-crafted rules or predefined thresholds, whereas ML models can “learn” rules and associations from patterns statistically discerned from datasets. DL is a subcategory of ML that specifically uses multilayer artificial neural networks for decision-making. Training can be supervised or unsupervised. The proposed advantage of AI tools for scar identification includes rapid precision-based undertaking of routine tasks utilizing the wealth of data available with CMR.

We undertook a targeted systematic review of publications including AI methods for the identification of left ventricular (LV) cardiac scar in CMR to answer the following questions:

What AI methods are being employed for ventricular scar assessment in those with cardiac disease on CMR?

What CMR sequences are being utilized?

What are the diagnostic evaluations of these methods?

Methods

This review adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) recommendations (Supplemental Appendices A and B). No pre-registration was undertaken.

Search strategy and data sources

We performed a systematic search up to June 2021 of PubMed, Embase, Web of Science, Cumulative Index to Nursing and Allied Health Literature (CINAHL), OpenDissertations, arXiv, and IEEE Xplore. Full Boolean criteria appear in Supplemental Appendix C.

Records were imported into Rayyan (Qatar Computing Research Institute, Doha, Qatar) and duplicates were manually removed.

Study selection

Articles underwent independent abstract screening for eligibility and full-text review by at least 2 reviewers (NJ, AP, AS), with conflicts resolved by a third.

Full-text articles evaluating AI algorithms for LV scar identification with CMR annotations were included. Exclusion criteria included animal-based studies, algorithms primarily utilizing predefined thresholds, absence of scar annotations, non-CMR imaging, letters to editors, editorials, abstract-only publications/posters, studies not in the English language, and non-LV scar studies.

Data extraction and analysis

We extracted following variables, where available: AI Algorithm characteristics, dataset characteristics, primary cardiac disease, CMR characteristics, ground-truth assessment, and evaluation measures.

Owing to the unique complexities of individual algorithms and to enable study comparison, methodologies were broadly categorized into supervised and unsupervised methods, with further subcategorization if appreciable in the main text. The definition of “fully automated” required no human interaction from image input to result output.

CMR sequence categorization accounted for nomenclature among different manufacturers. Dataset size was assessed through total caseload, with cases defined as individual CMR scans. Primary cardiac disease was broadly categorized, with mixed-etiology studies highlighted.

Where required, conversion to mean was undertaken., Grouped means were calculated using Cochrane’s formula. Welch test was used for comparison of means. Random-effect meta-analyses of continuous primary outcome (Dice coefficient) was expressed as standardized mean difference with a 95% confidence interval. Comparison between proposed and comparator predefined thresholding methods was undertaken. Analysis was visualized by a forest plot. The extent of between-study heterogeneity was assessed with the I2 statistic. Funnel plots were used to assess publication bias. The ROBINS-I tool was used to evaluate the risk of bias of each study included in quantitative analysis by independent assessors and the robvis tool was used for visualization. P values were 2-tailed, with values <.05 considered statistically significant. Extracted data are available in Supplemental Appendix C.

Results

Of 6156 results, 35 were included for qualitative analysis. Four contained predefined thresholding comparators for quantitative analysis (Figure 2).

Figure 2

PRISMA flowchart of study selection process.

Articles included were published between the years 2010 and 2021, with increasing publications in recent years, with the highest in 2020 (n = 8).

CMR dataset

Sequences and magnet strength

All studies used CMR; however, 1 study did not specify sequences.

All 34 articles describing CMR sequences utilized late gadolinium–enhanced short-axis imaging as a minimum, with most undertaken with 2D imaging and breath-holds (82.35%). Of the 6 of 34 (17.65%) utilizing 3D late gadolinium enhancement, 4 utilized 3D whole-heart imaging, with the remaining consisting of breath-hold sequences.

Additional sequences were only used in combination with 2D late gadolinium enhancement (25/28, 89.29%), with the most frequent being a variation of cine imaging.

Ten articles did not declare CMR magnet strength. Of the remaining, 12 used only 1.5 Tesla (T), 6 only 3 T, and 7 a combination.

Manufacturers

CMR manufacturer was declared in 25 of 35 articles: 59.3%, 25.9%, and 18.5% for Siemens, Philips, and General Electric, respectively. Only 5 studies used multiple manufacturers; 1 utilized all 3 manufacturers, 2 used Philips and Siemens, and 2 were unclear on designations.

Dataset size

Excluding postprocessing transformations, a total of 3856 human CMR studies in 32 of 35 articles were utilized, with a range of 3–1073 and a median (interquartile range) of 45 (24–143.5). Three articles had unclear datasets.

Nine studies employed augmentation to increase the number of examples for model training.

Cardiac diseases

The most common cardiac disease examined was ischemic heart disease (21/35); however, only 14 contained solely ischemic images. The remaining ischemic cohorts were combined with “normal” images (4), tetralogy of Fallot (1), unspecified (1), or a combination (1). Four studies assessed hypertrophic cardiomyopathy (11.4%) and 9 did not specify any disease etiology (25.7%). Intracardiac device presence was not declared.

AI methods

DL and ML were represented in 60% and 40%, respectively. Subcategorization of DL algorithms were convolutional neural networks based on 2D models (12/21), 3D models (4/21), or other/unspecified (5/21). Most DL approaches used a convolutional neural network with U-Net architecture. ML algorithms that referred to classical ML methods using hand-crafted features included 13 various models with at least 6 studies employing multiple models.

Only 4 papers utilized unsupervised primary methods—all classic ML. Twenty-four of 35 ML/DL methods were fully automated, with 9 of 11 nonautomated algorithms utilizing ML methods (Table 1).

Table 1

Summary of reviewed studies

Author, year of publication	Supervised learning?	Full Automation?	Final code available?	MRI sequences	Dataset condition cohort
Abramson, 202013	Yes	Yes	No	Cine, 2D LGE	Ischemic
Brahim, 202014	Yes	Yes	No	2D LGE	Mixed – Ischemic, healthy, not specified
Brahim, 202115	Yes	Yes	No	2D LGE	Mixed – Ischemic, healthy
Brahim, 202116	Yes	Yes	No	2D LGE	Mixed – Ischemic, healthy
Campello, 202017	Yes	Yes	No	Cine, 2D LGE, T2	Not specified
Carminati, 201518	No	No	No	2D LGE	Ischemic
Carminati, 201619	No	No	No	2D LGE	Ischemic
De la Rosa, 201920	Yes	Yes	No	Cine, 2D LGE	Mixed – Ischemic, healthy
Engblom, 201621	Yes	Yes	No	2D LGE	Ischemic
Fadil, 202122	Yes	Yes	Yes	Cine, 2D LGE, pre- & postcontrast T1, T2	Mixed – Ischemic, healthy, not specified
Fahmy, 202023	Yes	Yes	Yes	Cine, 2D LGE	Hypertrophic cardiomyopathy
Fahmy, 202124	Yes	Yes	Yes	2D LGE	Hypertrophic cardiomyopathy
Heidenreich, 202125	Yes	Yes	Yes	2D LGE	Ischemic
Kotu, 201126	Yes	Yes	No	2D LGE	Not specified
Kurzendorfer, 201827	Yes	No	No	3D LGE	Not specified
Larroza, 201728	Yes	No	Yes	Cine, 2D LGE	Ischemic
Larroza, 201829	Yes	No	No	Cine, 2D LGE	Ischemic
Lau, 201830	Yes	Yes	No	2D LGE	Not specified
Mantilla, 201531	Yes	Yes	No	2D LGE	Hypertrophic cardiomyopathy
Merino-Caviedes, 201632	Yes	No	No	Cine, 2D LGE	Hypertrophic cardiomyopathy
Metwally, 201033	No	Yes	No	2D LGE	Not specified
Moccia, 201834	Yes	Yes	No	2D LGE	Ischemic
Moccia, 201935	Yes	Yes	No	2D LGE	Ischemic
Moccia, 202036	Yes	Yes	No	Cine, 2D LGE	Ischemic
Morisi, 201537	Yes	No	No	3D LGE	Not specified
Rajchl, 201438	No	No	No	3D LGE (WH)	Mixed – Ischemic, Tetralogy of Fallot
Rukundo, 202039	Yes	Yes	No	2D LGE	Not specified
Wang, 201140	Yes	Yes	No	2D LGE	Ischaemic
Wang, 202041	Yes	Yes	No	Not specified	Mixed – Ischemic, healthy
Zabihollahy, 201842	Yes	No	No	3D LGE (WH)	Ischemic
Zabihollahy, 201943	Yes	No	No	3D LGE (WH)	Ischemic
Zabihollahy, 202044	Yes	Yes	No	3D LGE (WH)	Ischemic
Zhang Z, 202045	Yes	Yes	No	Cine, 2D LGE, T2	Not specified
Zhang X, 202046	Yes	Yes	No	Cine, 2D LGE, T2	Not specified
Zhuang, 201947	Yes	No	No	Cine, 2D LGE, T2	Not specified

LGE = late gadolinium enhancement; WH = whole-heart imaging.

Diagnostic evaluation

Ground truth

Ground-truth segmentations were all acquired from human delineation, 9 of which highlighted semiautomated thresholding techniques as segmentation aids. No histological confirmation was undertaken.

Evaluation metrics

All methods were compared to a minimum of human delineated ground truth. Four studies compared performance of proposed and predefined thresholding methods.19, 22, 43, 44

Twenty-seven different reported evaluation metrics were utilized. Common metrics existed in 3 groups: overlap, distance, and volume metrics (Table 2).

Table 2

Summary of evaluation metrics utilized

Reported evaluation metrics
Overlap
Dice coefficient, Jaccard index/intersection over union, Sensitivity specificity, Accuracy, Precision, F-score, Mean BF1, Recall, Segment overlap, Repeatability, True/false positive & negative
Distance
Haussdorf distance, Surface distance, Average contour distance, Root-mean-squared area
Volume
Left ventricular volume, Scar/infarct volume, Scar mass, Absolute volume difference ± normalization, Total volume error, Percentage volume error, Scar as myocardial percentage, Mean absolute error ± normalization, Left ventricular mass

DSC was the most common statistical metric in 54% of studies, followed by sensitivity (17.1%).

Tables 3 and 4 compare results of 5 evaluation methods in studies with comparable data. No statistically significant difference was seen in DSC between predefined threshold models compared to supervised and unsupervised learning methods (0.633 vs 0.616, P = .14), with unsupervised methods performing better than supervised (0.732 vs 0.599, P < .05). However, substantial dataset size variance is present.

Table 3

Comparison of reported Dice coefficient and Haussdorf distance for scar segmentation

	Dice coefficient			Haussdorf distance
	Mean (SD)	Total test cases	No. of studies	Mean	Total test cases	No. of studies
Predefined threshold	0.633† (0.15)	306	4	37.973	82	2
Supervised and unsupervised learning	0.616† (0.256)	1125	13	18.135	230	3
Supervised learning	0.599‡ (0.264)	984	9	18.135	230	3
Unsupervised learning	0.732‡ (0.153)	141	4	-	-	-

P = .14.

P < .05.

Table 4

Comparison of reported mean sensitivity, specificity, and accuracy for scar segmentation

	Sensitivity	Total test cases	No. of studies	Specificity	Total test cases	No. of studies	Accuracy	Total test cases	No. of studies
Predefined threshold	83.91	160	1	90.79	160	1	88.51	160	1
Supervised and unsupervised learning	91.4	882	4	97.11	870	3	92.99	1296	6
Supervised learning	95.09	520	2	98.95	520	2	94.03	776	4
Unsupervised learning	83.79	372	5	94.38	350	3	88.4	520	3

Higher sensitivity, specificity, and accuracy were seen in proposed methods, namely supervised, compared to predefined thresholding with an associated lower Haussdorf distance.

In 4 studies with direct predefined thresholding vs supervised/unsupervised learning comparisons, weak evidence of a small effect with high heterogeneity was seen in learning models with higher DSC, standardized mean difference = 1.11; 95% CI -0.16 to 2.38, P = .09 (Figure 3). Visual inspection of funnel plots did not reveal asymmetry in studies, though sparse data existed (Figure 4). Studies had low (n = 2) and moderate (n = 2) risk of bias (Supplemental Appendix C).

Figure 3

Forest plot of supervised and unsupervised learning vs predefined thresholding models.

Figure 4

Funnel plot. Visual analysis suggests no bias, though data points are sparse. SE = standard error; SMD = standardized mean difference.

Discussion

In this article, we reviewed published literature surrounding methods for CMR scar segmentation. As expected, with technological advances, increasing numbers of studies using learning methods have been published internationally, highlighting the subject’s global interest and clinical potential.

With the recent rapid expansion of AI, many state-of-the-art reviews exist covering current and potential applications for general cardiology. Our review focuses on progress within 1 specific aspect of cardiology to provide clinically relatable interpretation. For this purpose, it is important to consider evaluation, practicality, and patient/scanner generalizability.

Evaluation

Our review has not demonstrated clear benefit of supervised/unsupervised learning vs predefined thresholding methods for LV scar segmentation methods. Unsupervised models performed better than other models when considering volume overlap (DSC), but the inverse was seen in sensitivity, specificity, and Haussdorf distance, with supervised methods outperforming others. However, limited comparisons and significant heterogeneity in evaluation methods was evident. In direct comparison from 4 studies, there was weak evidence favoring supervised/unsupervised learning methods. Furthermore, it is questionable whether the small differences demonstrated would translate to clinical benefit.

Three evaluation categories were present: overlap, distance, and volume metrics. Combinations of metrics are required to extensively assess segmentation models. Unfortunately, owing to absence of clear guidance regarding minimal and/or preferable metrics, many models described to date may not be directly comparable. The use of segmentation challenges mitigates some of these issues with standardized datasets and reporting measures but is associated with its own caveats.

Moreover, studies predominantly compared their proposed model to initial ground-truth or other supervised models, with little comparison to various clinically utilized non–learning methods.

For integration into clinical practice, future research must compare standard clinical applications against novel methods through robust multimodal comparable metrics.

Practicality

A noteworthy benefit of the described AI methods is automaticity. Guidelines recommend objective quantification of cardiac structures and function. Hence, full automation of scar assessment is desirable to reduce operator burden. Many existent predefined thresholding algorithms for scar quantification are applied once myocardial borders are delineated—a semiautomatic process., Myocardial border annotation deviations may lead to scar misevaluation and this issue extends to all methodologies dependent on predefined ground-truth borders. Subsequently, published segmentation results may not be directly transferable clinically when reliant on border segmentation quality. Fully automated AI methods for both myocardial and scar segmentation may result in more reproducible and objective segmentation labels.

However, the black-box problem of DL decision-making processes being essentially noninterpretable remains a considerable issue. Until this problem is solved, human oversight and manual input for segmentation and correction remains essential to ensure patient safety in the clinical context, currently limiting the potential of a truly automatic approach outside the research environment. Unsupervised methods and rule-based ML algorithms have an advantage in this respect owing to relatively explainable methodologies, which may be important for clinical confidence.

Generalizability

Clinical use and external generalizability of models requires applicability to a variety of patients/scanners and is dependent on image or model complexity and training datasets.

Scar patterns vary across different cardiomyopathies. Ischemic scar arises from the subendocardium, with epicardial progression compared with a more heterogenous distribution in nonischemic cardiomyopathies. With the high prevalence of ischemic heart disease and its associated significant mortality, predominance of ischemic scar models is understandable and clinically necessary. Nonischemic etiologies have been investigated but are more sparse., Of greater concern is that 25.7% of publications did not specify disease etiology.

Training datasets have significant impact on model performance. Hence, detailed descriptions of dataset disease etiologies are of great importance for clinical utility, especially as studies assessing model transferability across disease cohorts without retraining are sparse.

Metadata can vary between manufacturers and magnet strengths, and models trained with specific data may not be generally applicable. Only a small subset of studies included multiple CMR manufactures to mitigate this risk.,21, 22, 23,, Training data on various field strengths would allow greater generalizability clinically, but 1.5 T remains the current standard, with 3 T employed mainly by more experienced imaging centers. Similarly, regional manufacturer predominance may exist. Inclusion of multiple manufacturers and field strengths for generalizability is to be commended and should be actively sought; however, data suggest 1.5 T Siemens scans have the largest data support for clinical application.

Small datasets are the main limitation for optimal Algorithm creation. Access to data remains a limitation to researchers explaining, in part, the large variation in dataset size range.

Reducing barriers to data accessibility may reduce training/testing data variability and improving model comparability. Collaborative sharing and access to such data is an important consideration. Critically, labeled data are required to avoid time-consuming reanalysis and promote transparent result comparison. Other smaller datasets exist in the form of CMR challenges for more standardized model comparison. Further options to improve generalizability within existing datasets include augmentation through image transformations or using generative adversarial networks to produce synthetic images.,

Conclusion

Feasibility of applying AI to the task of scar segmentation in CMR has been demonstrated. Progression toward clinical application requires dataset transparency, evaluation, standardization, and model generalizability.

Funding Sources

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Disclosures

The authors have no conflicts of interest to declare.

Authorship

All authors attest they meet the current ICMJE criteria for authorship.

Patient Consent

Not applicable.

Ethics Statement

This review adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) recommendations (Supplemental Appendices A and B). No pre-registration was undertaken.

Artificial intelligence models have been shown to be feasible for the segmentation of cardiac structures including scar.

There is no clear benefit of supervised or unsupervised learning models over predefined thresholding models. Random-effect analysis may suggest a benefit of learning models, but high heterogeneity exists.

Comparison of models to identify superior methodologies remains challenging in humans owing to a lack of true scar “ground truth” as reference, with significant variation in assessment methods utilized and limited access to final Algorithm codes.

Standardization of model evaluation, transparency in training/testing data, and highly generalizable models are recommended before transition to clinical practice can be considered.