Overview of Deep Learning in Gastrointestinal Endoscopy.

Artificial intelligence is likely to perform several roles currently performed by humans, and the adoption of artificial intelligence-based medicine in gastroenterology practice is expected in the near future. Medical image-based diagnoses, such as pathology, radiology, and endoscopy, are expected to be the first in the medical field to be affected by artificial intelligence. A convolutional neural network, a kind of deep-learning method with multilayer perceptrons designed to use minimal preprocessing, was recently reported as being highly beneficial in the field of endoscopy, including esophagogastroduodenoscopy, colonoscopy, and capsule endoscopy. A convolutional neural network-based diagnostic program was challenged to recognize anatomical locations in esophagogastroduodenoscopy images, Helicobacter pylori infection, and gastric cancer for esophagogastroduodenoscopy; to detect and classify colorectal polyps; to recognize celiac disease and hookworm; and to perform small intestine motility characterization of capsule endoscopy images. Artificial intelligence is expected to help endoscopists provide a more accurate diagnosis by automatically detecting and classifying lesions; therefore, it is essential that endoscopists focus on this novel technology. In this review, we describe the effects of artificial intelligence on gastroenterology with a special focus on automatic diagnosis, based on endoscopic findings.

INTRODUCTION

Artificial intelligence (AI) can be defined by an intelligence demonstrated by machines in contrast to the natural intelligence displayed by humans and other animals.1 AI is generally applied when a machine mimics “cognitive” functions that humans associate with other human minds, such as “learning” and “problem solving.”2 The concept of AI first appeared at the Dartmouth Conference held in 1956 by McCarthy et al.3 AI pioneers dreamed about developing a super-complex computer with characteristics similar to human intelligence, for example, AI that thinks like human beings with human senses and thinking power. However, today’s AI is characterized by its ability to perform certain tasks, such as image classification or face recognition functions. The recent explosion of AI became possible by the introduction of graphic processing units that offer fast and robust parallel processing capabilities of computer and big data analysis. The three approaches to AI include symbolism (rule based, such as IBM Watson), connectionism (network- and connection-based, such as deep learning or artificial neural net), and Bayesian (based on the Bayesian theorem).4

In March 2016, “Baduk (board game Go) AI (AlphaGo)” and Korea’s professional Baduk artist (former world Go champion) had five confrontations, and the result was human defeat.5 This confrontation led to the realization that AI is a reality rather than the distant future story, as perceived in movies and novels. Today’s AI offerings struggle to understand natural human language, compete at the highest level in strategic game systems, such as chess or the board game Go, drive autonomous cars, and resolve content delivery network and military simulations. AI is likely to perform many roles performed by humans, although the adoption of AI-based medicine in gastroenterology practice is expected in the near future. Medical image-based diagnoses, such as pathology, radiology, and endoscopy, will be the first in the medical field to be affected by AI. The rapid advancement in AI technology requires physicians to be knowledgeable about AI to aid in the understanding of AI and how it might change and influence the medical field in the near future. Based on advanced deep learning (DL) technology, many startups in South Korea are developing businesses in the medical image analysis area. The rapid growth of domestic start-ups in a short period of time is probably due to the abundance of data at the core of DL, which is among the strengths of Korea. Korea has been storing digital medical images for a long time and has the advantage of having large amounts of electrical medical records stored because it can receive medical services at a much lower cost than other countries.

In this review, the effects of AI on gastroenterology will be described with specific focus on the automatic diagnosis of endoscopic findings.

DEVELOPMENT OF MACHINE LEARNING

Physicians may be confused about the precise concept of AI, machine learning (ML), and DL (Fig. 1).6 In AI, computers can imitate human beings and show intelligence similar to that of human beings. Murphy7 defined ML as a set of methods that automatically detect data patterns and then use the detected patterns to predict future data or enable decision-making in uncertain conditions. Thus, it ultimately aims to learn how to do things by “learning” using massive amounts of data and algorithms rather than coding specific instructions directly into the software. The most representative characteristic of ML is that it is driven by data, and the decision process is accomplished with minimal human interventions. Today’s ML has achieved great results in some fields such as computer vision; however, it has the limitation that a certain amount of human instruction is needed in the process. The image recognition rate of ML is sufficient for commercialization, but it remains low in certain fields; this is why image recognition skills are still inferior compared to human capabilities.

Fig. 1

Machine learning is a manner of achieving artificial intelligence, and deep learning is one of many machine learning methods. Adapted from NVIDIA. Available from: https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/.6

DL is the process by which a computer collects, analyzes, and rapidly processes the data it needs without having to accept formal data when performing certain tasks. Therefore, DL is the way a machine responds to create a learning model based on vast amounts of data without receiving instructions from humans. DL is characterized by self-learning; once a training data-set has been provided, the program can extract the key features and quantities without human indications by using a back-propagation algorithm and changing the internal parameters of each neural network layer.8 DL is a type of ML that is developed in an artificial neural network (ANN) and uses data input and output hierarchies similar to brain neurons to learn data. In fact, the ANN was inspired by the biological properties of the human brain, particularly neuronal connections. An ANN is a network in which several nodes (which is corresponding to human brain neurons) are connected, and various types of networks can be created depending on how these nodes are connected. Because ANN requires a great deal of computation power, DL was finally completed by acceleration of neural network computation speed from the development of supercomputers. DL has the potential to automatically detect lesions, suggest differential diagnoses, and compose preliminary medical reports. When ANN was introduced in 1950, it had many limitations, such as a vanishing gradient, overfitting problems, a lack of computing power, and the absence of big data to train the neural network.4 The deep neural network (DNN) is an ANN with several layers hidden between the input and output layers; this allows for the modeling of complex data with fewer nodes than a similar ANN. In 2012, DNN was implemented by Google and professor Andrew NG of Stanford University with more than a billion neural networks with 16,000 computers. In ImageNet Large Scale Visual Recognition Challenge 2012, Super Vision, which was proposed by Geoffrey Hinton, professor at the University of Toronto, used a different type of DNN and achieved 84% accuracy, 10% higher than that in the previous year. DL was used to develop computer-aided systems that can support physicians’ diagnoses, and previous reports reported a high level of performance for radiological diagnosis,9–14 skin cancer classification,15 diabetic retinopathy,16,17 and histopathology.18

CONVOLUTIONAL NEURAL NETWORK

Image recognition using DNN has been attempted in some medical fields since it is well suited for medical big data. Convolutional neural network (CNN) is a DNN based on the principle that the visual cortex of the human brain processes and recognizes images. The CNN contains multilayer perceptrons (artificial neurons) and is designed to use minimal preprocessing. CNN leverages the multiple network layers (consecutive convolutional layers followed by pooling layers) to extract the key features from an image and provide a final classification through the fully connected layers as the output.8 The convolutional layer, which extracts features from the input data, is composed of a filter that extracts the features and an activation function that converts the value of the filter to a non-linear value (Fig. 2). Because there are many features in the input value, multiple filters are used in the CNN. The combination of multiple filters that extract different features can be applied to the CNN to determine the characteristics of the original data. The filter is automatically created after recognition of the features through learning from the learning data. Once the feature map has been extracted through the filters, the activation function is applied to make the quantitative value non-linear (yes or no value). Compared to other DL structures, CNN is a popular method for image recognition since it offers good performance in both video and audio applications. For example, CNN has the best performance for image classification in a large image repository, such as ImageNet.19 Furthermore, CNN is easier to train than other ANN techniques and has the advantage of using fewer parameters. The more data an ML algorithm has access to, the more effective it can be; however, medical images often have insufficient data because they involve privacy concerns. To overcome this, several data augmentation techniques have been proposed to reduce overfitting on models by increasing the amount of training data.

Fig. 2

A convolutional neural network is a multilayer construct of perceptrons designed to use minimal preprocessing. The convolutional layer is composed of a filter for extracting features and an activation function for converting the filter’s value to a nonlinear value.

DL-BASED AUTOMATIC DETECTION AND CLASSIFICATION FOR ENDOSCOPY

CNN was recently reported to be highly beneficial in the field of endoscopy, including esophagogastroduodenoscopy (EGD), colonoscopy, and capsule endoscopy (CE). In EGD, a CNN-based diagnostic program was challenged to recognize the anatomical location in EGD images,8 Helicobacter pylori (HP) infection,20,21 and gastric cancer.22 Takiyama et al.8 reported that the trained CNN can recognize the anatomical location in EGD images. They constructed a CNN-based diagnostic program using 27,335 EGD images from 1,750 patients and validated their program’s performance using 17,081 independent EGD images from 435 patients. In this study, the performance of the trained CNN to classify the anatomical location in EGD images provided an area under the curve (AUC) of 1.00 for the larynx and esophagus images and an AUC of 0.99 for the stomach and duodenum images. Furthermore, the trained CNN could recognize specific anatomical locations within the stomach, with an AUC of 0.99 for the upper, middle, and lower stomach. This study highlighted the significant potential for future application as a DL-aided EGD diagnostic system. Two Japanese studies20,21 suggested that a CNN-based diagnostic program could recognize HP gastritis with higher accuracy. Itoh et al.20 used 179 EGD images obtained from 139 patients (65 HP-positive and 74 HP-negative) for the development of the CNN; of them, 149 were used as training images and 30 as test images. The sensitivity and specificity for the CNN-based detection of HP infection were 86.7% and 86.7%, respectively, and the AUC was 0.956. Shichijo et al.21 used a dataset of 32,208 EGD images (HP-positive or -negative) for the first CNN and trained secondary CNN using images classified according to eight anatomical locations. In this study, separate test datasets (11,481 images) from 397 patients was evaluated independently by the CNN and 23 endoscopists. The sensitivity, specificity, and accuracy were 81.9%, 83.4%, and 83.1% for the first CNN and 88.9%, 87.4%, and 87.7% for the secondary CNN, respectively. The secondary CNN had significantly higher accuracy than the endoscopists (by 5.3%; 95% confidence interval, 0.3 to 10.2). Hirasawa et al.22 recently challenged the Single Shot MultiBox Detector–based diagnosis of gastric cancer using 13,584 EGD images of gastric cancer and an independent 2,296 test gastric images. The sensitivity and positive predictive value of the CNN for gastric cancer detection were 92.2% and 30.6%, respectively. Furthermore, 70 of the 71 lesions (98.6%) ≥6 mm as well as all invasive cancers were correctly detected. However, 69.4% of the false-positive lesions were gastritis with redness or an irregular mucosal surface. All missed lesions were superficially depressed and differentiated-type intramucosal cancers that were difficult for even experienced endoscopists to distinguish from gastritis. This study was very promising, but it had limitations since only high-quality endoscopic images were used from the same endoscope type and endoscopic video system.

For colonoscopy, a CNN-based diagnostic program was applied to detect and classify colorectal polyps.23–26 In 2017, Komeda et al.23 reported on CNN-based diagnosis of colorectal polyps using 1,200 colonoscopy images and 10 additional video images of unlearned processes. The accuracy of the 10-hold cross-validation was 0.751, where the accuracy is the ratio of the number of correct answers to the number of all answers produced by the CNN. Byrne et al.24 developed a deep CNN model for the real-time assessment of colonoscopic video images of colorectal polyps. To train and validate the model, they used only narrow-band image (NBI) video frames (split equally between relevant multi-classes) and unaltered videos from routine exams not specifically designed or adapted for CNN classification. The model was tested on a separate series of 125 videos of 106 consecutively encountered diminutive polyps. The accuracy, sensitivity, specificity, negative predictive value, and positive predictive value of the CNN for the detection of adenoma were 94%, 98%, 83%, 97%, and 90%, respectively. Zhang et al.25 reported automatic detection and classification of colorectal polyps by transferring low-level CNN features from the nonmedical domain. They used 1,104 non-polyp images taken under both white-light and NBI endoscopy and 826 NBI endoscopic polyp images (263 hyperplastic polyps and 563 adenomas). CNN-based diagnoses showed similar precision (87.3% vs 86.4%) but a higher recall rate (87.6% vs 77.0%) and accuracy (85.9% vs 74.3%) than endoscopist diagnoses. Yu et al.26 proposed a novel offline and online three-dimensional (3D) DL integration framework by leveraging the fully 3D CNN for automated polyp detection. Compared with the hand-crafted features or two-dimensional CNN, the fully 3D CNN is capable of learning more representative spatiotemporal features from colonoscopy videos. They integrated offline and online learning to effectively reduce the number of false positives generated by the offline network and further improve detection performance using the dataset of the 2015 Medical Image Computing and Computer Assisted Intervention (MICCAI) Challenge on Polyp Detection.

In the field of CE, a CNN-based diagnostic program was challenged to recognize celiac disease,27 hookworm,28 and small intestine motility characterization.29 A CNN-based diagnostic program using CE is difficult to develop because CE image quality is usually poor due to hardware and light limitations and low resolution (320×320 pixels) (Fig. 3). CE image quality is further limited by various orientations because of the free motion of the capsule and various extraneous matters such as bile, bubble, food, and fecal material. Zhou et al.27 attempted to develop the automatic diagnosis of celiac disease using CE clips from six celiac disease patients and five controls preprocessed for training. The clips of CE from five additional celiac disease patients and five additional control patients were used for testing, and the GoogLeNet achieved 100% sensitivity and specificity for the testing set. A t-test confirmed significant evaluation confidence to distinguish the celiac disease patients from controls. He et al.28 designed a CNN-based program of two integrated networks (edge extraction and hookworm classification) to detect hookworm in CE images. The edge extraction network was used to produce the edge maps for tubular region detection since tubular structures are important clues for hookworm detection.28 They achieved 84.6% sensitivity and approximately 10% to 15% improvement in sensitivity, specificity, and accuracy compared with the previous program. Wu et al.30 demonstrated high sensitivity and accuracy for hookworm detection using one of the largest CE datasets (440,000 images of 11 patients). In this study, accuracy, sensitivity, and specificity reach around 78%. Seguí et al.29 introduced a deep CNN system for small intestine motility characterization. They demonstrated the superiority of the learned features over alternative classifiers and a mean classification accuracy of 96% for six intestinal motility events, thereby outperforming the other classifiers by a large margin (a 14% relative performance increase).

Fig. 3

Sample of a deep convolutional neural network-based ulcer detection algorithm using capsule endoscopy.

Despite the promising results of a CNN-based diagnosis program for endoscopic images, its diagnostic ability is highly dependent on training data quality and amount. More training images might result in a more accurate diagnostic ability of the CNN; however, there may be legal or ethical issues surrounding the commercial use of high-quality endoscopy images. When a CNN-based system makes a diagnostic error, legal liability issues could be raised because it is difficult to explain the technical and logical methodology of the system.

LIMITATIONS AND FUTURE DIRECTION

It is not sufficiently verified whether it is possible to improve the medical performance, to reduce the medical cost, and to improve the satisfaction of the patient and medical staff using AI in the medical field. Furthermore, treatment results may vary depending on the indications and range of application, even using the same AI. Therefore, it is necessary to demonstrate the clinical efficacy of AI, to develop doctor-friendly interface of AI, and to educate physicians to utilize AI. However, it is very difficult to demonstrate the efficacy of AI with clinical trials and to establish guidelines for applying AI. Furthermore, there is a problem that the number of medical staff who can educate AI for physicians is very limited. AI will help physicians in their care because the physician is only responsible for the results of the treatment using AI. In the interpretation of endoscopic images using AI, AI will help physician to reduce medical errors and to improve medical efficiency in the interpretation of endoscopic images. Therefore, physicians should seek best ways to provide better care to their patient with the help of AI. However, it is necessary to make a social consensus for medicolegal issues and to educate future physicians for AI, because AI can cause legal and ethical issues as future AI may exclude physician’s decision for the interpretation of endoscopic images.

CONCLUSIONS

DL-based medical technology, such as the automatic detection of cancer or pathological endoscopic findings and more accurate decision-making, are currently or soon to be in use in our daily lives. The CNN-based automatic diagnosis of endoscopic findings is expected to be a mainstream DL technology in the next few decades. DL is expected to help endoscopists provide a more accurate diagnosis by automatically detecting and classifying endoscopic lesions. One of the most important factors for the development of DL for endoscopy would be the availability of large amounts of high-quality endoscopic images, as well as an increased understanding of the technology by the endoscopist. Therefore, it is essential that endoscopists focus on this novel technology.