Computer-aided analysis of 64-slice coronary computed tomography angiography: a comparison with manual interpretation.

Coronary computed tomography angiography (CCTA) is increasingly used for the assessment of coronary heart disease (CHD) in symptomatic patients. Software applications have recently been developed to facilitate efficient and accurate analysis of CCTA. This study aims to evaluate the clinical application of computer-aided diagnosis (CAD) software for the detection of significant coronary stenosis on CCTA in populations with low (8%), moderate (13%), and high (27%) CHD prevalence. A total of 341 consecutive patients underwent 64-slice CCTA at 3 clinical sites in the United States. CAD software performed automatic detection of significant coronary lesions (>50% stenosis). CAD results were then compared to the consensus manual interpretation of 2 imaging experts. Data analysis was conducted for each patient and segment. The CAD had 100% sensitivity per patient across all 3 clinical sites. Specificity in the low, moderate, and high CHD prevalence populations was 64%, 41%, and 38%, respectively. The negative predictive value at the 3 clinical sites was 100%. The positive predictive value was 22%, 21%, and 38% for the low, moderate, and high CHD prevalence populations, respectively. This study demonstrates the utility of CAD software in 3 distinct clinical settings. In a low-prevalence population, such as seen in the emergency department, CAD can be used as a Computer-Aided Simple Triage tool to assist in diagnostic delineation of acute chest pain. In a higher prevalence population, CAD software is useful as an adjunct for both the experienced and inexperienced reader.

Introduction

Coronary computed tomography angiography (CCTA) is increasingly used for the non-invasive assessment of coronary artery disease in stable symptomatic patients. Several multicenter clinical trials comparing 64-slice CCTA with invasive catheter-based coronary angiography have shown remarkable diagnostic accuracy for the detection of obstructive coronary heart disease (CHD) (>50% stenosis).[1,2] In addition, a negative CCTA has been demonstrated in multiple studies to improve diagnostic certainty for ruling out significant coronary artery disease in a low-risk acute chest pain population, as well as reduce hospital length of stay and healthcare costs when compared with the standard of care.[3,4]

The increased utilization of CCTA has resulted in a need to rapidly interpret a large volume of images in an accurate and efficient manner. To meet this growing demand for expert image interpretation, computer-aided diagnosis (CAD) software applications have been developed to expedite the evaluation of coronary heart disease with CCTA. The multiple benefits of automated software programs have been previously demonstrated in many non-cardiac diagnostic imaging modalities. However, the clinical utility of CAD for the assessment of CHD is still in its infancy. While various software applications have been validated for the automated evaluation of CCTA and automated aortic calcium scoring, a useful commercially available product has yet to be established.[8,9]

Here we evaluate a dedicated CAD software designed for the automated evaluation of CCTA: the COR Analyzer by Rcadia Medical Imaging. Recently, 3 studies were published which evaluated the COR Analyzer performance and reported promising system sensitivity and negative predictive values. Furthermore, the 3 studies proved to have specificities of over 60%, which is significant when considering the potential for the software to be used as a Computer-Aided Simple Triage (CAST) tool. A successful CAST tool can be employed in the emergency department as an initial rapid read to allow for triage of patients. This is a step forward from the typical CAD role as a confirmatory second reader. However, the past studies evaluating the COR Analyzer were limited in their size as well as in their patient populations and further confirmation of these results is necessary.

The current study aims to further evaluate the applicability of CAD software for the detection of significant coronary stenosis by 64-slice CCTA. We aimed to determine the utility and accuracy of the COR Analyzer compared to manual CCTA analysis in various clinical sites using different types of equipment (CT scanners) and among populations with varying CHD prevalence.

Materials and Methods

Coronary CTA was performed on consecutive patients at 3 clinical sites within the United States: Emergency Room, NY (ER); Imaging Center, WA (IC); Outpatient Office, NY (OP). All CCTAs were acquired using one of three 64-slice multi-detector computed tomography scanners: 64-detector row Lightspeed VCT scanner (GE Healthcare, Milwaukee, Wisconsin); 64-detector row Sensation 64 scanner (Siemens Health Care, Forchheim, Germany); 64-detector row Aquilion-64 scanner (Toshiba Medical, Tochigi, Japan). The 3 sites were assigned a disease prevalence category based on the relative prevalence to the other sites included in the study; ER was considered low risk (<10% prevalence), IC was considered intermediate risk (11-20% prevalence) and OP was considered high risk (>20% prevalence). A standard CCTA protocol was employed at all sites. Patients were pre-treated with oral and/or intravenous beta-blockers as necessary to achieve a resting heart rate target of less than 70 beats/min. All patients received one 4 mg sublingual nitroglycerin tablet five minutes prior to the CCTA scan. Electrocardiographic (ECG) prospective trigger or retrospective gating was used for all scans. The scanning protocol was adjusted for patient weight and heart rate, but typically included: 64x0.5 mm collimation, 80-120 kV tube voltage, 400-700 mAs of current for prospective scans and current modulation during retrospective scans. The entire volume of the heart was acquired in 8-9 s during a single breath-hold. For axial images, an effective slice thickness of 0.75 mm with a reconstruction increment of 0.5 mm was used. Optimal images and reformations were reconstructed on a high-resolution matrix of 512x512 pixels and during several cardiac phases (various percentages of the R-R interval) when available.

Each CCTA study was independently interpreted by 2 ACC/ACR credentialed level III CCTA imaging experts. Images were qualitatively classified as excellent, good or suboptimal studies. The CCTA studies were analyzed for the presence of significant stenosis (defined as >50%) using a combination of axial images, 3D volume-rendered images, multiplanar reformations, and maximum intensity projections. A 10-segment model of the coronary artery tree was used for per segment analysis: left main and the proximal, mid, and distal segments of the 3 major coronary arteries, which corresponded to the CAD reporting scheme. Vessel segments less than 1 mm in diameter were excluded. All CCTA studies performed were included in the final analysis irrespective of study quality.

Following manual interpretation, the CCTA studies were transferred to a separate workstation equipped with the CAD software (COR Analyzer©, version 1.8.1070 by Rcadia Medical Imaging, Haifa, Israel). The CAD software performed automatic segmentation of the coronary artery tree, labeling of coronary arteries, and detection of significant coronary lesions (defined as the most severe lesion in each segment with >50% stenosis).Figure 1 shows both normal and obstructive reads by the CAD software with the corresponding manual CCTA reads for reference. Since the discussion about image processing and machine learning algorithms used by the CAD system is beyond the scope of this journal, for detailed technical description we refer the interested reader elsewhere in the literature.

Figure 1.

(Upper panels) MIP of a normal coronary computed tomography angiography (left) and a normal Rcadia computer-aided diagnosis image output (right) of the same patient. (Lower panels) MIP (left) and Rcadia computer-aided diagnosis image and findings output (right) of a coronary computed tomography angiography with 2-vessel non-calcified obstructive (>50% stenosis) disease. Red arrows point to the proximal LAD lesion and blue arrows point to the OM branch lesion.

In addition to reporting a positive or negative study the CAD system may also report a warning when there is an error in the automatic analysis or the system is not confident about the result. Warnings are issued for a specific blood vessel segment and are usually caused by various imaging artifacts, low image quality, or unrelated software errors in vessel tracking. A warning is not an indication for a lesion detected by the system, but rather an invitation for the user to pay extra attention to the specified area. In case of a warning on a negative study, the human observer should confirm the lack of findings and declare it negative, while in the case of a warning on a positive study, the reader should detect a lesion by inspecting the indicated area and declare it positive. Therefore, warnings on negative studies were not counted as false alarms and warnings on positive studies were considered as true positives (hits).

The guidelines for the evaluation of diagnostic performance of CAD were provided by the interpretative recommendations in the CCTA consensus manual. The analysis was performed for each patient and segment; sensitivity, specificity, negative predicative value and positive predicative value were calculated. Approval from the IRB committee at the coordinating site was obtained prior to study initiation.

Results

A total of 341 patients with appropriate indications for 64-slice CCTA at 3 clinical sites in the United States were included in the study. Of these, 96 patients underwent CCTA in the emergency room (ER) of a suburban tertiary medical center, 196 patients were scanned in a suburban outpatient cardiac imaging center (IC), and 49 patients had CCTA at an urban outpatient cardiology office (OP). The prevalence of CHD (>50% stenosis in one or more vessels) in the population at these 3 locations was low (8%), moderate (13%), and high (27%) for ER, IC and OP, respectively. Two of the 341 CCTAs were characterized as suboptimal in quality, but were still included in the analysis.

The CAD results categorized the CCTA as positive for obstructive CHD in 37 of the 96 scans performed at ER, 122 of the 196 at IC, and 34 of the 49 in OP. In the segmental analysis, segments were classified as positive in 77 of 960 ER segments, 278 of the 1960 IC segments and 76 of 490 OP segments. The consensus manual interpretation found that 8 of the 96 CCTAs at ER, 26 of the 196 CCTAs at IC, and 13 of the 49 CCTAs at OP had findings consistent with obstructive CHD. On a per segment basis, 19 out of the 960 ER segments, 29 out of the 1960 IC segments, and 27 out of the 490 OP segments were found to be representative of significant coronary stenosis. In the per patient analysis, there were 7 warnings on negative studies in both the ER and IC sites and 0 warnings on negative in the OP site. In the per vessel analysis, there were 24 warning on negative studies in the ER site, 20 in the IC site, and 6 in the OP site (Table 1).

Table 1.

Warnings on negative for each site, in both the per patient and the per vessel analysis.

	Coronary artery disease prevalence (%)
	Low (8%)Emergency room	Moderate (13%)Imaging center	High (27%)Outpatient office
Per patient	7/96	7/196	0/49
Per vessel	24/960	20/1960	6/490

The CAD had a per patient sensitivity of 100% across all 3 clinical sites. Specificity in the low, moderate, and high CHD prevalence populations was 64%, 41%, and 38%, respectively. The negative predictive value (NPV) at the 3 clinical sites was 100% (this is by default when a test results in a sensitivity of 100% and a specificity >0%). The positive predictive value (PPV) was 22%, 21%, and 38% for the low, moderate, and high CHD prevalence populations, respectively (Table 2). The segmental analysis yielded results with a similar trend (Table 3); however, the absolute values were lower due to misclassification in predominantly distal segments and segments affected by significant motion artifact.

Table 2.

Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for the identification of significant stenosis (>50%) on coronary computed tomography angiography by computer-aided analysis when compared with manual visual interpretation on a per patient analysis in 3 populations of varying disease prevalence.

	Coronary artery disease prevalence (%)
	Low (8%)Emergency room	Moderate (13%)Imaging center	High (27%)Outpatient office
Sensitivity	100% (8/8)	100% (26/26)	100% (13/13)
Specificity	64.2% (52/81)	41.1% (67/163)	38.2% (13/34)
PPV	21.6% (8/37)	21.3% (26/122)	38.2% (13/34)
NPV	100% (52/52)	100% (67/67)	100% (13/13)

PPV, positive predictive value; NPV, negative predictive value.

Table 3.

Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for the identification of significant stenosis (>50%) on coronary computed tomography angiography by computer-aided analysis when compared with manual visual interpretation on segmental analysis in 3 populations of varying disease prevalence

	Coronary artery disease prevalence (%)
	Low (8%)Emergency room	Moderate (13%)Imaging center	High (27%)Outpatient office
Sensitivity	73.7% (14/19)	82.8% (24/29)	100% (27/27)
Specificity	93.1% (847/910)	86.7% (1657/1911)	89.0% (397/446)
PPV	18.2% (14/77)	8.6% (24/278)	35.5% (27/76)
NPV	99.4% (847/852)	99.7% (1657/1662)	100% (397/397)

PPV, positive predictive value; NPV, negative predictive value.

Discussion

With the introduction of CCTA computer-aided detection software for the automatic detection of significant coronary heart disease, the potential for rapid discharge of patients who present to the emergency department with acute non-myocardial infarction (MI) chest pain is promising. Due to the low-risk nature of the acute non-MI chest pain patient, patients often endure long waiting times for various diagnostic procedures only to have a normal result. Computer-aided detection of CHD in the emergency setting has the potential to efficiently rule out CHD in acute non-MI chest pain patients, and thus can expedite the discharge of a large portion of patients who are in fact negative for CHD. While studies have validated the potential for CAD software for evaluation of CHD,[8,9] to date, few studies have explored commercially available CAD software.

A past study evaluating the diagnostic accuracy of the COR Analyzer software by comparison with quantitative angiography (QCA) reported a high sensitivity and negative predictive value. However, this study was limited by its small population (n=59) which was restricted to patients referred for QCA; thus, low- to intermediate-risk patients were not assessed. Additionally, a recent mid-sized, 2-center study and a large single center study compared the accuracy of the COR Analyzer software on 64-slice and 256-slice CCTA by comparison with manual interpretation of CCTA, again demonstrated a high sensitivity and negative predictive value.[11,12] These 3 studies compliment the current, so far largest, multicenter comparison of the CCTA CAD on 64-slice CCTA compared to manual CCTA interpretation.

The present study illustrates that the CAD system can identify abnormal coronary segments when employed in 3 distinct clinical sites with varying incidences of CHD. There was good correlation between manual interpretation and CAD results for the exclusion of significant CHD across a spectrum of disease prevalence. Specifically there were no false negatives in any of the populations studied.

While the high sensitivity and negative predicative value for CAD using CCTA is promising, both the specificity and positive predictive value were rather low by human analysis standards. On the other hand, when comparing to other known computer-aided analysis systems (e.g. for mammography, colonoscopy, lung nodules, chest X rays, etc.), the reported specificity and PPV (especially in the low-risk population, 8% prevalence) is extremely high. All other computer-aided diagnosis systems available today generate several false alarms on average per every case, yielding a close to zero per patient specificity and PPV.

The system reported in this study, yielding a 64% specificity for a low-risk population, falls into a different category of computer-aided analysis tools: the Computer-Aided Simple Triage (CAST) recently introduced by Goldenberg and Peled. As a CAST tool, it can be used for chest pain patient triage in the emergency department or for reading sequence prioritization in high-volume practices with low-risk populations. For moderate- and high-risk populations, the CAD system yielded lower specificities and cannot, therefore, be used as CAST in these groups. However, a CAST tool is most useful in the setting of the emergency department where rapid triage of patients is critical, and it is well documented that emergency departments are a low-risk population with regards to coronary artery disease. Therefore, CAD has the potential to significantly aid the experienced practitioner with accurate and efficient CCTA interpretations and maintain throughput, especially in high-volume emergency departments where a rapid evaluation for CHD is crucial. A recent study using the COR Analyzer software supports the current study’s results in the low-risk population, demonstrating a 63% specificity, further supporting the usage of the COR Analyzer software as a potential CAST tool.

Although the evaluated CAD system cannot be used as a CAST tool for moderate- (13% prevalence) and high-risk (27% prevalence) populations due to the subpar specificities and PPV, it still has a role in these categories of patients. In populations in which false positives are common, inexperienced radiologists can still benefit from having the second reader. Furthermore, it is important to note that the experienced reader can also benefit from the CAD software in populations where false positives reads by the CAD software are frequent. Every study reported as positive by the computer system is to be immediately reviewed by a human reader. Therefore, the most important characteristic for all practical purposes is not the stand-alone specificity/PPV of the computer, but the combined computer-human diagnostic performance. In this regard, it is important to mention that the majority of false alarms produced by the system were related to calcifications, imaging artifacts, and vessel tracking errors. Those can be easily and rapidly discarded by a human reader, yielding high combined computer-human specificity and PPV. These results are supported by findings in the previous comparison of manual CCTA and automated CCTA.[11,12]

We should note that the moderate prevalence population had results that are a cause of concern and that should be further discussed. The low-prevalence population has a very low PPV that is counterbalanced by the strong specificity in this population. Also, in the high-prevalence population, the low specificity is counterbalanced by an improved PPV. The moderate prevalence population saw comparative reductions in both specificity and PPV, bringing into question the utility of CAD for CHD in this population. However, considering the previous discussion regarding the nature of the false positive calls, there is still a possibility that CAD might be useful in the moderate prevalence population and, as such, more research must be conducted before eliminating the role of CAD in this population.

Furthermore, with the introduction of the 320-slice CT scanner, we hope for a large improvement in the specificity and positive predictive value of CAD for coronary heart disease, mainly due to its wider volume coverage resulting in significantly less stacking artifact. Improvements in the software, including proper analysis of heavily calcified vessels, as well as improved vessel tracking, can further advance the diagnostic ability of the CAD system.

Limitations

The retrospective application of the CAD software is a limitation of this study. A prospective evaluation could assess objectively whether the combined use of CAD and manual interpretation is superior to manual interpretation alone, as has been previously demonstrated in other non-cardiovascular imaging modalities using CAD, such as mammography. Additionally, a prospective study can explore the logistics and utility of applying the CAD as a CAST tool in a high-volume emergency department. An important aspect to consider when designing a prospective study is that the mean time for CAD (approx. 7 min per phase) exceeds the time required for manual interpretation if more than 2-3 phases are sent for the automatic analysis (mean reading time of an expert SCCT level III reader is approx. 5-10 min; Poon M, unpublished data). While this has the potential to be the rate-limiting step when employed in a combined prospective work flow, the ability to facilitate the triage process in a high-volume emergency department setting where the majority of the acute chest pain cases are normal is still promising. Especially when considering nights and weekends when it may be difficult to reach an expert SCCT reader and long patient waiting times is the norm.Another important limitation is the utilization of an expert manual interpretation of CCTA as the reference standard rather than the typical QCA. However, in the emergency setting, most clinical decisions are made upon manual CCTA analysis, as opposed to QCA, and thus a comparison of CAD using CCTA and manual CCTA is appropriate in this study. Furthermore, a previous study has confirmed that CAD utilizing 64-slice CCTA has high sensitivity and negative predicative value when compared to QCA that nicely complements the results of the current study.

A final limitation to consider is the exclusion of large coronary side branches during the automated analysis. At the launch of this trial, the software did not have the capability to analyze side branches and so significant lesions may have been missed. However, since the launch of this trial, the COR analyzer has been up-dated and now incorporates side branches in the coronary analysis with promising initial results.

Conclusions

In this multi-center, retrospective study, we have shown the utility of the CAD for analysis of coronary heart disease using CCTA. By applying the software in 3 distinct clinical sites, we demonstrated that the CAD system is compatible with scans from 3 major CT scanner manufacturers as well as in varying patient populations with different incidences of CHD.

In a low-risk population, the CAD system shows as a CAST tool. This is particularly exciting when considering the low prevalence of CHD typically encountered in the emergency department. While both the moderate and high prevalence populations are not viable candidates for the usage of CAD as a CAST tool, the importance in these populations is not lost. First, the inexperienced reader can benefit from a CAD second reader in ruling out false positives the inexperienced reader might encounter. Additionally, an experienced reader can rapidly disregard many of the false positives read by the CAD software leading to an efficient human-computer interface in the diagnosis of CHD using CCTA.

Overall, computer-aided diagnosis in the evaluation of coronary heart disease is starting to become a reality. With this study, we have shown the commercially available CAD system to be a potential candidate to take on this role in various clinical sites with various incidences of coronary heart disease.