Clinical Application of Computer-Aided Diagnosis for Breast Ultrasonography: Factors That Lead to Discordant Results in Radial and Antiradial Planes.

OBJECTIVE: To investigate factors that may lead to discordant results in radial and antiradial planes when applying a computer-aided diagnostic system (S-DetectTM) for breast ultrasound (US).
METHODS: From May 2019 to September 2019, a total of 288 breast lesions from 286 consecutive women were analyzed. Diagnostic performance and diagnostic agreement of the CAD system between the radial and antiradial planes were calculated. Based on the CAD results in the radial and antiradial planes, the lesions were classified into two groups: the agreement group and the discordant group. Ultrasound imaging and clinicopathologic factors in the two groups were compared.
RESULTS: Of the 288 breast lesions, 187 (64.7%) were benign, and 101 (35.3%) were malignant. There was no difference in diagnostic performance of the CAD system between the radial and antiradial planes. The diagnostic agreement of the CAD system between these two orthogonal planes was good (ĸ = 0.645). Compared to category 3, 4C and 5 lesions, category 4A and 4B lesions were more likely to have discordant CAD results. In the subgroup of malignant tumors, lesions diagnosed as carcinoma in situ (P = 0.014), staged as T1 (P = 0.013) and with low Ki-67 status (P = 0.024) were significantly associated with discordant CAD results.
CONCLUSION: The CAD system for breast ultrasound has a favorable diagnostic performance, and discordant CAD results between the radial and antiradial planes were more frequent for BI-RADS category 4A-4B lesions and less invasive malignant lesions.

Introduction

Recently, the occurrence of breast cancer has been increasing in both developed and developing countries.1 Early diagnosis can improve the prognosis of breast cancer patients, and breast ultrasound (US) plays an important role in detecting breast cancer. As a convenient, cost-effective, and widely available diagnostic tool, conventional US is able to achieve a sensitivity greater than 90%.2,3 Nevertheless, this method is limited by its high dependence on operator experience.4,5

In recent years, artificial intelligence (AI) technology has been greatly developed. Preliminary studies have shown a wide AI application in imaging the lung, thyroid, fetus, carotid, liver, etc. Several deep learning technologies for ultrasound computer-aided diagnosis systems (CADs) have been frequently utilized in investigations to provide a second objective opinion to assist radiologists in image interpretation and diagnosis.6 Among them, a deep learning-based CAD system for breast US, also known as S-Detect (Samsung Medison Co. Ltd., Seoul, Korea), has become commercially available.7 This system can analyze the morphologic features of a target breast mass automatically and quickly provide a final diagnosis in a dichotomic form of “possibly benign” or “possibly malignant”. This system has high diagnostic performance in the diagnosis of breast lesions and is distinguished for its high specificity.8–13 Many previous studies have demonstrated that adding S-Detect to breast US can improve the specificity, positive predictive value, and accuracy in the diagnosis of breast lesions and can reduce variability in radiologist interpretations.8–10,13

Breast tumors often show different morphologies on grayscale images in different examination planes due to their intratumoral heterogeneity and three-dimensional structures, which leads to a high probability that the CAD system would display different results in different acquisition planes when being applied.14,15 Currently, to the best of our knowledge, there are no standard guidelines for CAD systems with regard to image acquisition planes. In previous studies,8,13 a single representative image of the breast tumor was selected for CAD evaluation. However, the high interobserver variability in the acquisition of representative images seemed inevitable. In some other studies,9,11 there was no description of whether the CAD examination was performed in the antiradial plane or the radial plane when classifying the tumors as “possibly malignant”. Thus, to decrease the difficulties in interpreting CAD results in clinical practice, it is necessary to assess whether and in what circumstances the orientation of imaging acquisition influences the diagnostic performances of CAD systems.

According to the American College of Radiology practice guidelines for breast US, image acquisition in two orthogonal planes is recommended as a standard procedure to achieve better lesion features.16 Therefore, we used a CAD system (S-Detect) to evaluate the frequency of discordant CAD results in the radial and antiradial planes when differentiating breast lesions and to investigate the imaging and clinicopathologic factors that lead to the discordant results.

Patients and Methods

Patients

This prospective study, carried out from May 2019 to September 2019, was approved by the Ethics Committee of the Shanghai Ruijin Hospital of Huangpu District. Written informed consent was obtained from the patients or parents of patients under 18 years of age, and this study was conducted in compliance with the Declaration of Helsinki. Three hundred thirty breast lesions in 328 consecutive patients who were scheduled for ultrasound-guided core needle biopsy or ultrasound-guided vacuum-assisted excision were included in this study. Among them, 42 patients were excluded from the study for the following reasons: 7 patients had nonmass lesions on ultrasound imaging; 19 lesions were currently being treated with neoadjuvant chemotherapy; 9 lesions were larger than 4 cm; 4 lesions were biopsied before the US examination; and 3 women were pregnant or lactating. Finally, the final cohort for this study included 288 breast lesions in 286 women (mean age, 47.3 ± 13.8 years; age range, 19–86 years) (Figure 1). One hundred and twenty-six patients (44.1%) were asymptomatic, while the remaining 160 patients (55.9%) presented with symptoms such as a palpable mass, breast pain and bloody nipple discharge.

Figure 1

Flowchart of the breast lesion selection process.

Conventional Ultrasound Examinations and the Application of the Computer-Aided Diagnosis System

An RS80A system (Samsung Medison Co., Ltd.) equipped with a deep learning-based CAD system software for breast (S-Detect; Samsung Medison, Co. Ltd., Seoul, Korea) was applied for breast imaging with a 5–12 MHz linear high-frequency transducer. To remove any bias arising from multiple operators performing CAD and reduce interobserver variability, the examinations of the study population were performed by a single expert radiologist (J.Q.Z) with 18 years of experience in breast US imaging who has performed more than 100 CAD examinations.

Informed of the patients’ clinical information and mammography results, the radiologist performed the bilateral breast US examinations using the routine scanning protocol. When the target lesion was detected, Figure 2 radial and antiradial static images (Figure 2), radial and antiradial video clips, representative color Doppler images were recorded for further imaging evaluation. After the conventional US examination, the S-Detect examination was performed for the static images in the radial and antiradial planes by the same radiologist who performed the breast US examination. When the image was selected, a region of interest (ROI) was automatically drawn along the lesion margin by the S-detect software. If the automatically generated ROI was considered inaccurate by the radiologist, it was manually adjusted. After the ROI was delineated, the final evaluations from the CAD system were displayed in dichotomized form as “probably benign” or “possibly malignant”.

Figure 2

Image acquisition directions for the breast lesions. (A) Antiradial plane; (B) radial plane.

Imaging Review

Two radiologists (Y.Z. and J.X.H) with more than 8 years of experience in breast US were asked to review the static images and video clips of the breast lesions. They had no access to the patients’ clinical information or CAD results but were aware of the patients’ age. If disagreements occurred, they jointly reviewed the images, discussed the findings, and reached a consensus. The US features, including shape, orientation, margin, echo pattern, posterior features, calcifications, changes in the surrounding tissue, degree of blood flow, tumor location and tissue composition of the breast gland adjacent to the lesions, were described. In this study, the degree of blood flow in the breast lesions was divided as follows according to the grading by Adler et al17 absent, no blood flow was visualized; minimal, 1 or 2 pixels containing blood flow were visualized; moderate, a main vessel and several small vessels were visualized; and marked, 4 or more vessels were visualized. Finally, each target lesion was classified as category 3, 4A, 4B, 4C, or 5 according to the fifth edition of the Breast Imaging Reporting and Data System (BI-RADS) lexicon.

Statistical Analysis

Statistical analyses were performed by using SPSS version 16.0 (SPSS Inc, Chicago, IL) and MedCalc for Windows, version 12.2.0.0 (MedCalc Software, Mariakerke, Belgium). Descriptive variables are presented as the mean ± SD. Categorical variables are described as proportions. Using histologic results as the reference, the diagnostic sensitivity, specificity, and area under the ROC curve (AUC) of the CAD system were calculated. The kappa coefficient (k) was calculated to determine the agreement between CAD results in the radial and antiradial planes. The k values were interpreted as follows using the guidelines of Landis and Koch:18 poor agreement, k =0.00–0.20; fair agreement, k =0.21–0.40; moderate agreement, k = 0.41–0.60; good agreement k =0.61–0.80; and excellent agreement, k =0.81–1.00. Based on the CAD results in the radial and antiradial planes, the lesions were classified into two groups: the agreement group, in which the tumor had the same CAD results in the two orthogonal planes; and the discordant group, in which the tumor had different CAD results in the two orthogonal planes. Chi-square tests and Fisher’s exact tests were applied to compare the categorical variables. Statistical significance was determined at a P value < 0.05.

Results

Demographic Data and Histologic Findings

Of the 288 breast lesions, 187 (64.7%) were benign, and 101 (35.3%) were malignant. The malignant lesions were significantly larger than the benign lesions: 18.42 ±7.48 mm (9.00–40.00 mm) vs 15.68 ± 7.60 mm (7.00–39.10 mm) (P=0.004). The patients with malignant lesions were significantly older than those with benign lesions (mean age: 55.9 ± 12.9 years vs 42.4 ± 11.7 years, P < 0.001). The pathological results of the breast lesions are shown in Figure 1.

Diagnostic Agreement of the CAD Evaluations in the Radial and Antiradial Planes

Among the 288 lesions, 105 cases had the same CAD results of “possibly malignant” and 131 cases had the same CAD results of “probably benign” lesions in the radial and antiradial planes, while 52 cases had discordant CAD results, including 22 cases with “possibly malignant” evaluations in only the antiradial plane and 30 cases with “possibly malignant” evaluations in only the radial plane. The diagnostic agreement of the CAD results based on the images in the radial and antiradial planes was good, with a ĸ value of 0.645 (95% CI: 0.557–0.733). In addition, compared with those based on the images in the antiradial planes, CAD evaluations based on the ultrasound images in the radial planes had a similar sensitivity (79.4% vs 85.3%), specificity (75.4% vs 72.2%) and AUC (0.774 vs 0.790) in predicting malignant lesions (P >0.05 for all).

Correlation with Ultrasound Features

There was a significant association between BI-RADS category and discordant CAD results for breast lesions in the radial and antiradial planes (P= 0.011). Our data showed that the rates of malignancy were 3.20% (three of 103) for category 3 lesions, 12.2% (10 of 82) for category 4A lesions, 59.6% (28 of 47) for category 4B lesions, 88.9% (five of 45) for category 4c lesions, and 100% (21 of 21) for category 5 lesions. Compared to category 3, 4C and 5 lesions, category 4A and 4B lesions were more likely to have discordant CAD results in the two orthogonal planes (Table 1). In this study, 127 lesions were categorized as BI-RADS 4A or 4B, and 34 (26.8%) of these had discordant CAD results (Figures 3 and 4).

Table 1

Relationship of Agreement of CAD Results in Orthogonal Planes with Pathology Findings and Imaging Features

	Agreement Group (n=236)	Discordant Group (n=52)	p value
Age			0.429
<50y (n=180)	150 (83.3%)	30 (16.7%)
≥50y (n=108)	86 (79.6%)	22 (20.4%)
Pathology			0.126
Benign (n=187)	158 (84.5%)	29 (15.5%)
Malignant (n=101)	78 (77.2%)	23 (15.5%)
Manually adjusted ROI			0.580
Yes (n=69)	55 (79.7%)	14 (20.3%)
No (n=219)	181 (82.6%)	38 (17.4%)
Size			0.664
≤ 20 mm (n=209)	170 (81.3)	39 (18.7%)
>20 mm (n=79)	66 (83.5%)	13 (16.5%)
Location			0.315
Upper outer quadrant	104 (83.2%)	21 (16.8%)
Lower outer quadrant	41 (87.2%)	6 (12.8%)
Lower inner quadrant	29 (78.4%)	8 (21.6%)
Upper inner quadrant	48 (75.0%)	16 (25.0%)
Central quadrant	14 (93.3%)	1 (6.7%)
Tissue composition			0.109
Homogeneous background echotexture – fat (n=37)	26 (70.3%)	11 (29.7%)
Homogeneous background echotexture – fibroglandular (n=91)	74 (81.3%)	17 (18.7%)
Heterogeneous background echotexture (n=160)	136 (81.9%)	24 (15.0%)
Shape			0.324
Oval (n=119)	102 (85.7%)	17 (14.3%)
Round (n=7)	5 (71.4%)	2 (28.6%)
Irregular (n=162)	129 (79.6%)	33 (20.4%)
Orientation			0.075
Parallel (n=257)	207 (80.5%)	50 (19.5%)
Not parallel (n=31)	29 (93.5%)	2 (6.5%)
Margin			0.110
Circumscribed (n=117)	101 (86.3%)	16 (13.7%)
Not circumscribed (n=171)	135 (78.9%)	36 (21.1%)
Echo pattern			0.717
Hypoechoic (n=256)	211 (82.4%)	45 (17.6%)
Hyperechoic (n=16)	12 (75.0%)	4 (25.0%)
Complex cystic and solid (n=14)	12 (85.7%)	2 (14.3%)
Heterogeneous (n=2)	1 (50%)	1 (50%)
Posterior features			0.546
None (n=182)	147 (80.8%)	35 (19.2%)
Shadowing (n=27)	23 (85.2%)	4 (14.8%)
Enhancement (n=71)	58 (81.7%)	13 (18.3%)
Combined pattern (n=8)	8 (100%)	0 (0%)
Microcalcifications in the mass			0.177
Present (n=52)	46 (88.5%)	6 (11.5%)
Absent (n=236)	190 (80.5%)	46 (19.5%)
Degree of vascularity in the mass			0.193
Absent (n=103)	85 (81.6%)	19 (18.4%)
Minimal (n=55)	40 (72.7%)	15 (27.3%)
Moderate (n=71)	61 (85.9%)	10 (14.1%)
Marked (n=59)	51 (86.4%)	8 (13.6%)
Changes in the surrounding tissue			0.158
Present (n=53)	47 (88.7%)	6 (11.3%)
Absent (n=235)	189 (80.4%)	46 (19.6%)
US-based BI-RADS category			0.011
3 (n=98)	85 (86.7%)	13 (13.3%)
4A (n=84)	63 (75.0%)	21 (25.0%)
4B (n=43)	30 (69.8%)	13 (30.2%)
4C (n=42)	38 (90.5%)	4 (9.5%)
5 (n=21)	20 (95.2%)	1 (4.8%)

Note: Values are presented as number (%).

Figure 3

US images in a 40-year-old woman with fibroadenoma. Conventional ultrasound images show 13-mm hypoechoic mass with irregular shape and macrocalcification (A and B), which was classified as BI-RADS 4A. The CAD system diagnosed it as “possibly malignant” based on the image in the radial plane (C) and “probably benign” based on the image in the antiradial plane (D).

Figure 4

US images in a 54-year-old woman with Her-2 positive invasive ductal carcinoma. Conventional ultrasound images show 15-mm hypoechoic mass with irregular shape (A and B), which was classified as BI-RADS 4B. The CAD system diagnosed it as “probably benign” based on the image in the antiradial plane (C) and “possibly malignant” based on the image in the radial plane (D).

During the application of S-Detect, a region of interest (ROI) along the border of the mass was drawn automatically by the CAD system or was adjusted manually by the radiologist. We did not find that the radiologist’s adjustments to the ROI led to bias in the final CAD results. In this study, the radiologist manually adjusted the ROIs in 69 (24.0%) cases, including 25 benign tumors and 44 malignant tumors. Among them, only one case proven to be DCIS had a changed CAD result of “possibly malignant” in the radial plane. The other cases had no changes in CAD results before and after adjusting the ROI for CAD analysis. Accordingly, the adjustment of the ROIs was not related to the agreement of the CAD results between the two orthogonal planes (Table 1). No correlation was found among the discordant CAD results, tumor size on ultrasonography, tissue composition of the breast gland adjacent to the lesions and other ultrasound features of the breast lesions (Table 1).

Correlation with Clinicopathologic Features

In this study, the risk of malignancy and patient age showed no significant differences between the agreement and discordant group. However, in the subgroup of malignant tumors, discordant CAD results between the two orthogonal planes were more frequent among lesions proven as carcinoma in situ by pathology (P=0.014), staged as T1 (P=0.013) and with low Ki-67 status (P=0.024) (Table 2), and the discordant results were not related to the expression of ER, PR or CerBb-2.

Table 2

Relationship Between Agreement of CAD Results in the Orthogonal Planes and Pathology Findings Among Malignant Lesions (n=101)

	Agreement Group (n=78)	Disagreement Group (n=23)	p value
Size			0.013
≤ 20 mm (n=66)	46 (69.7%)	20 (30.3%)
>20 mm (n=35)	32 (91.4%)	3 (8.6%)
Pathological type			0.014
Carcinoma in situ (n=21)	12 (57.1%)	9 (42.9%)
Invasive cancer (n=80)	66 (82.5%)	14 (17.5%)
ER and/or PR			0.168
Negative (n=34)	29 (85.3%)	5 (14.7%)
Positive (n=67)	49 (73.1%)	18 (26.9%)
CerbB-2			0.510
0 or 1+ (n=38)	28 (73.7%)	10 (26.3%)
2++ or 3+++ (n=63)	50 (79.4%)	13 (20.6%)
Ki67 status			0.024
Low (<14%; n=37)	24 (64.9%)	13 (35.1%)
High (≥14%;n=64)	54 (84.4%)	10 (15.6%)

Note: Values are presented as number (%).

Discussion

At present, CAD systems for breast ultrasound are considered to have good prospects for wide clinical application.19–21 S-Detect (Samsung Medison Co. Ltd., Seoul, Korea) is a deep learning-based CAD system that is commercially available and has become one of the most increasingly used CAD systems for the diagnosis of breast cancer. Consistent with previous studies,8–12 our study confirmed that this CAD system exhibited a favorable diagnostic value in assisting physicians to discriminate benign and malignant breast lesions, with a sensitivity of 79.4%-85.3%, a specificity of 72.7%-75.4% and an AUC of 0.774-0.790.

Because of their heterogeneous and three-dimensional structures, breast tumors might have different CAD results in different scanning planes.14,15 After a thorough evaluation of the literature, it appears that our study is the first to evaluate the agreement of CAD results between two orientations of image acquisition. Our data showed good agreement (k=0.645) between the CAD results for lesions in the antiradial and radial planes. We also found that the diagnostic performance of S-Detect based on ultrasound images in the antiradial plane did not significantly differ from that based on images in the radial plane. Even so, a better understanding of the factors related to discordant CAD findings between the two orthogonal planes might be helpful in the use of this CAD system and the interpretation of its results. Therefore, in this study, we analyzed the clinicopathologic and imaging features of the lesions related to discordant CAD results.

Although malignant breast lesions appear more likely to have complicated and variable intratumor heterogeneity than benign lesions,14 we did not find a significant correlation between discordant CAD results between the two orthogonal planes and tumor pathology. Notably, our data showed that malignant lesions diagnosed as carcinoma in situ, with a smaller size and with low Ki-67 status were more likely to have discordant CAD results in the orthogonal planes. The main reason might be related to their different growth rates in different directions and their high frequency for displaying nonspecific US features.22,23 This subgroup analysis indicated that the observed disagreement for CAD results of malignant lesions may correlate with the invasive potential of the lesion. Of course, this has to be proven further in a large-scale study.

Moreover, our results found that lesions categorized as BI-RADS 4A or 4B were more likely to have different CAD results between two orthogonal planes. Previous studies have comfirmed that radiologists had an improved specificity,  positive predictive value (PPV) and accuracy after integrating CAD results into their initial evaluations.9–12 But according to our experience, they might encounter difficulties in integrating CAD findings into their initial conventional ultrasound results when considering images of breast lesions from two orthogonal view rather than form a single view. Considering that there are no specific guidelines on the application of CAD for the breast, this decision might inevitably lead to high heterogeneity when discordant CAD results are obtained for BI-RADS category 4A and 4B lesions. Thus, the optimal strategy for adding CAD results to improve radiologists’ diagnostic performance needs to be clarified.

The CAD system of S-Detect can can perform autosegmentation and interpretation based on US-based morphological descriptions. However, before the classification of lesions is displayed by the S-Detect system, radiologists need to delineate the ROI. The ROI for CAD analysis is not automatically segmented in the full sense and needs to be adjusted manually if the outline is not adequate. Jeong et al24 found that radiologists with different experience levels had different accuracies in applying the S-Detect system for the thyroid because of differences in the identified ROI. In contrast, we found that manual adjustment of the ROI by an experienced radiologist had little influence on the results of this CAD system and accordingly had little effect on the agreement of the CAD results between two orthogonal planes. In this study, the ROI was manually adjusted in 24.0% of cases (69/288). Among them, only one case proven to be DCIS had a changed CAD result in the radial plane as “possibly malignant”. The main reasons might be as follows: 1) the differences in the outline detected by the CAD system and the manual adjustments were typically small;25 and 2) this CAD system focuses not only on the ROI area but also on the peripheral area within 180 pixels when analyzing the ultrasound image of breast lesions,7 which might reduce the influence of manual adjustments on the core analysis area. Moreover, other potential factors that preclude the application of this CAD system for evaluating breast lesions, including changes in the surrounding tissue, microcalcifications and degree of vascularity in the mass, were also discussed in this study, and we did not find any correlations with discordant CAD results between two orthogonal planes.

There were a few limitations in our study. First, this study was a single-center, retrospective study, although care was taken to minimize bias by blinding the radiologists to the final pathology of the lesions, and the radiologists performing the BI-RADS classification were did not perform the CAD examination. Second, there was a risk for potential bias with respect to data collection, as only biopsy-confirmed lesions were included. Therefore, lesions categorized as BI-RADS 2 were not included. Third, this study was solely based on an investigation with one CAD system (S-Detect), and the results might vary with other systems. In the future, the performance of deep learning-based CAD systems may be improved with technical developments, especially when CAD systems based on the video clips rather than static images are developed. However, even then, an evaluation of discordant CAD findings between radial and antiradial planes would still be of interest.

In conclusion, the deep learning-based CAD system (S-Detect) is an effective tool for aiding US-based diagnosis of breast lesions and have a favorable diagnostic performance. The agreement between the CAD results based on the radial and antiradial planes was good. Discordant CAD results were more frequent for lesions categorized as BI-RADS 4a or 4b and for less invasive malignant lesions. The recognition of discordant CAD results can help to avoid inaccurate evaluations of breast cancer with low invasiveness and decrease the difficulty in interpreting CAD results in clinical practice.