Artificial intelligence in breast ultrasound.

Artificial intelligence (AI) is gaining extensive attention for its excellent performance in image-recognition tasks and increasingly applied in breast ultrasound. AI can conduct a quantitative assessment by recognizing imaging information automatically and make more accurate and reproductive imaging diagnosis. Breast cancer is the most commonly diagnosed cancer in women, severely threatening women's health, the early screening of which is closely related to the prognosis of patients. Therefore, utilization of AI in breast cancer screening and detection is of great significance, which can not only save time for radiologists, but also make up for experience and skill deficiency on some beginners. This article illustrates the basic technical knowledge regarding AI in breast ultrasound, including early machine learning algorithms and deep learning algorithms, and their application in the differential diagnosis of benign and malignant masses. At last, we talk about the future perspectives of AI in breast ultrasound.

Core tip: Artificial intelligence (AI) is gaining extensive attention for its excellent performance in image-recognition tasks and increasingly applied in breast ultrasound. In this review, we summarize the current knowledge of AI in breast ultrasound, including the technical aspects, and its applications in the differentiation between benign and malignant breast masses. In the meanwhile, we also discuss the future perspectives, such as combining with elastography and contrast-enhanced ultrasound, to improve the performance of AI in breast ultrasound.

INTRODUCTION

Breast cancer is the most common malignant tumor and the second leading cause of cancer death among women in the United States[1]. In recent years, the incidence and mortality of breast cancer have increased year by year[2,3]. Mortality can be reduced by early detection and timely therapy. Therefore, its early and correct diagnosis has received significant attention. There are several predominant diagnostic methods for breast cancer, such as X-ray mammography, ultrasound, and magnetic resonance imaging (MRI).

Ultrasound is a first-line imaging tool for breast lesion characterization for its high availability, cost-effectiveness, acceptable diagnostic performance, and noninvasive and real-time capabilities. In addition to B-mode ultrasound, new techniques such as color Doppler, spectral Doppler, contrast-enhanced ultrasound, and elastography can also help ultrasound doctors obtain more accurate information. However, it suffers from operator dependence[4].

In recent years, artificial intelligence (AI), particularly deep learning (DL) algorithms, is gaining extensive attention for its extremely excellent performance in image-recognition tasks. AI can make a quantitative assessment by recognizing imaging information automatically so as to improve ultrasound performance in imaging breast lesions[5].

The use of AI in breast ultrasound has also been combined with other novel technology, such as ultrasound radiofrequency (RF) time series analysis[6], multimodality GPU-based computer-assisted diagnosis of breast cancer using ultrasound and digital mammography image[7], optical breast imaging[8,9], QT-based breast tissue volume imaging[10], and automated breast volume scanning (ABVS)[11].

So far, most studies on the use of AI in breast ultrasound focus on the differentiation of benign and malignant breast masses based on the B-mode ultrasound features of the masses. There is a need of a review to summarize the current status and future perspectives of the use of AI in breast ultrasound. In this paper, we introduce the applications of AI for breast mass detection and diagnosis with ultrasound.

EARLY AI

Early AI mainly refers to traditional machine learning. It solves problems with two steps: object detection and object recognition. First, the machine uses a bounding box detection algorithm to scan the entire image to find the possible area of the object; second, the object recognition algorithm identifies and recognizes the object based on the previous step.

In the identification process, experts need to determine certain features and encode them into a data type. The machine extracts such features through images, performs quantitative analysis processing and then gives a judgment. It will be able to assist the radiologist to discover and analyze the lesions and improve the accuracy and efficiency of the diagnosis.

In the 1980s, computer-aided diagnosis (CAD) technology developed rapidly in medical imaging diagnosis. The workflow of the CAD system is roughly divided into several processes: data preprocessing, image segmentation-feature, extraction, selection and classification recognition, and result output (Figure 1).

Figure 1

Workflow of machine learining algorithm.

FEATURE EXTRACTION

In traditional machine learning, most applied features of a breast mass on ultrasound, including shape, texture, location, orientation and so on, require experts to identify and encode each as a data type. Therefore, the performance of machine learning algorithms depends on the accuracy of the extracted features of benign and malignant breast masses.

Identifying effective computable features from the Breast Imaging Reporting and Data System (BI-RADS) can help distinguish between benign and potential malignant lesions by different machine learning methods. Lesion margin and orientation were optimum features in almost all of the different machine learning methods[12].

CAD model can also be used to classify benign and metastatic lymph nodes in patients with breast tumor. Zhang et al[13] proposed a computer-assisted method through dual-modal features extracted from real-time elastography (RTE) and B-mode ultrasound. With the assistance of computer, five morphological features describing the hilum, size, shape, and echogenic uniformity of a lymph node were extracted from B-mode ultrasound, and three elastic features consisting of hard area ratio, strain ratio, and coefficient of variance were extracted from RTE. This computer-assisted method is proved to be valuable for the identification of benign and metastatic lymph nodes.

SEGMENTATION

Recently, great progress has been made in processing and segmentation of images and selection of regions of interest (ROIs) in CAD. Feng et al[14] proposed a method of adaptively utilizing neighboring information, which can effectively improve the breast tumor segmentation performance on ultrasound images. Cai et al[15] proposed a phased congruency-based binary pattern texture descriptor, which is effective and robust to segament and classify B-mode ultrasound images regardless of image grey-scale variation.

CLASSIFICATION AND RECOGNITION

According to the similarity of algorithm functions and forms, machine learning generally includes support vector machine, fuzzy logic, artificial neural network, etc., and each has its own advantages and disadvantages. Bing et al[16] proposed a novel method based on sparse representation for breast ultrasound image classification under the framework of multi-instance learning (MIL). Compared with state-of-the-art MIL method, this method achieved its obvious superiority in classification accuracy.

Lee et al[17] studied a novel Fourier-based shape feature extraction technique and proved that this technique provides higher classification accuracy for breast tumors in computer-aided B-mode ultrasound diagnosis system.

Otherwise, more features extracted and trained may benefit the recognition effciency. De et al[18] questioned the claim that training of machines with a simplified set of features would have a better effect on recognition. They conducted related experiments, and the results showed that the performance obtained with all 22 features in this experiment was slightly better than that obtained with a reduced set of features.

DL ALGORITHMS

In contrast to traditional machine learning algorithms, DL algorithms do not rely on the features and ROIs that humans set in advance[19,20]. On the contrary, it prefers carrying out all the task processions on its own. Taking the convolutional neural networks (CNNs), the most popular architecture in DL for medical imaging, as an example, input layers, hidden layers, and output layers constitute the whole model, among which hidden layers are the key determinant of accomplishing the recognition. Hidden layers consist of quantities of convolutional layers and the fully connected layer. Convolutional layers handle different and massive problems that the machine raise itself on the basis of the input task, and the fully connected layer then connects them to be a complex system so as to output the outcome easily[21]. It has been proved that DL won an overwhelming victory over other architectures in computer vision completion despite its excessive data and hardware dependencies[22]. In medical imaging, besides ultrasound[23], studies have found that DL methods also perform perfectly on computed tomography[24] and MRI[25] (Figure 2).

Figure 2

Workflow of deep learining algorithm.

Becker et al[26] conducted a retrospective study to evaluate the performance of generic DL software (DLS) in classifying breast cancer based on ultrasound images. They found that the accuracy of DLS to diagnose breast cancer is comparable to that of radiologists, and DLS can learn better and faster than a human reader without prior experience.

Zhang et al[27] established a DL architecture that could automatically extract image features from shear-wave elastography and evaluated the DL architecture in differentiation between benign and malignant breast tumors. The results showed that DL achieved better classification performance with an accuracy of 93.4%, a sensitivity of 88.6%, a specificity of 97.1%, and an area under the receiver operating characteristic curve (AUC) of 0.947.

Han et al[28] used CNN DL framework to differentiate the distinctive types of lesions and nodules on breast images acquired by ultrasound. The networks showed an accuracy of about 0.9, a sensitivity of 0.86, and a specificity of 0.96. This method shows promising results to classify malignant lesions in a short time and supports the diagnosis of radiologists in discriminating malignant lesions. Therefore, the proposed method can work in tandem with human radiologists to improve performance.

TRANSFERRED DEEP NEURAL NETWORKS

CNN has proven to be an effective task classifier, while it requires a large amount of training data, which can be a difficult task. Transferred deep neural networks are powerful tools for training deeper networks without overfitting and they may have better performance than CNN. Xiao et al[29] compared the performance of three transferred models, a CNN model, and a traditional machine learning-based model to differentiate benign and malignant tumors from breast ultrasound data and found that the transfer learning method outperformed the traditional machine learning model and the CNN model, where the transferred InceptionV3 achieved the best performance with an accuracy of 85.13% and an AUC of 0.91. Moreover, they built the model with combined features extracted from all three transferred models, which achieved the best performance with an accuracy of 89.44% and an AUC of 0.93 on an independent test set.

Yap et al[30] studied the use of three DL methods (patch-based LeNet, U-Net, and a transfer learning approach with a pretrained FCN-AlexNet) for breast ultrasound lesion detection and compared their performance against four state-of-the-art lesion detection algorithms. The results demonstrate that the transfer learning method showed the best performance over the other two DL approaches when assessed on two datasets in terms of true positive fraction, false positives per image, and F-measure.

AI EQUIPPED IN ULTRASOUND SYSTEM

Images are usually uploaded from the ultrasonic machine to the workstation for image re-processing, while a DL technique (S-detect) can directly identify and mark breast masses on the ultrasound system. S-detect is a tool equipped in the Samsung RS80A ultrasound system, and based on the DL algorithm, it performs lesion segmentation, feature analysis, and descriptions according to the BI-RADS 2003 or BI-RADS 2013 lexicon. It can give immediate judgment of benignity or malignancy in the freezd images on the ultrasound machine after choosing ROI automatically or manually (Figure 3). Kim et al[31] evaluated the diagnostic performance of S-detect for the differentiation of benign from malignant breast lesions. When the cutoff was set at category 4a in BI-RADS, the specificity, PPV, and accuracy were significantly higher in S-detect compared to the radiologist (P < 0.05 for all), and the AUC was 0.725 compared to 0.653 (P = 0.038).

Figure 3

S-detect technique in the Samsung RS80A ultrasound system. A and B: In a 47-year-old woman with left invasive breast cancer on B-mode ultrasound (A), S-detect correctly concluded that it is “Possibly Malignant” based on the lesion features listed on the right column (B); C and D: In a 55-year-old woman with fibroadenoma of left breast on B-mode ultrasound (C), S-detect correctly concluded that it is “Possibly Benign” based on the lesion features listed on the right column (D).

Di Segni et al[32] also evaluated the diagnostic performance of S-detect in the assessment of focal breast lesions. S-detect showed a sensitivity > 90% and a 70.8% specificity, with inter-rater agreement ranging from moderate to good. S-detect may be a feasible tool for the characterization of breast lesions and assist physicians in making clinical decisions.

CONCLUSION

AI has been increasingly applied in ultrasound and proved to be a powerful tool to provide a reliable diagnosis with higher accuracy and efficiency and reduce the workload of pyhsicians. It is roughly divided into early machine learning controlled by manual input algorithms, and DL, with which software can self-study. There is still no guidelines to recommend the application of AI with ultrasound in clinical practice, and more studies are required to explore more advanced methods and to prove their usefulness.

In the near future, we believe that AI in breast ultrasound can not only distinguish between benign and malignant breast masses, but also further classify specific benign diseases, such as inflammative breast mass and fibroplasia. In addition, AI in ultrasound may also predict Tumor Node Metastasis classification[33], prognosis, and the treatment response for patients with breast cancer. Last but not the least, the accuracy of AI on ultrasound to differentiate benign from malignant breast lesions may not only be based on B-mode ultrasound images, but also could combine images from other advanced techniqes, such as ABVS, elastography, and contrast-enhanced ultrasound.