Application of Artificial Intelligence in the Detection and Characterization of Colorectal Neoplasm.

Endoscpists always have tried to pursue a perfect colonoscopy, and application of artificial intelligence (AI) using deep-learning algorithms is one of the promising supportive options for detection and characterization of colorectal polyps during colonoscopy. Many retrospective studies conducted with real-time application of AI using convolutional neural networks have shown improved colorectal polyp detection. Moreover, a recent randomized clinical trial reported additional polyp detection with shorter analysis time. Studies conducted regarding polyp characterization provided additional promising results. Application of AI with narrow band imaging in real-time prediction of the pathology of diminutive polyps resulted in high diagnostic accuracy. In addition, application of AI with endocytoscopy or confocal laser endomicroscopy was investigated for realtime cellular diagnosis, and the diagnostic accuracy of some studies was comparable to that of pathologists. With AI technology, we can expect a higher polyp detection rate with reduced time and cost by avoiding unnecessary procedures, resulting in enhanced colonoscopy efficiency. However, for AI application in actual daily clinical practice, more prospective studies with minimized selection bias, consensus on standardized utilization, and regulatory approval are needed.

INTRODUCTION

The incidence of colorectal cancer (CRC), the third most common cancer worldwide,1 has been steadily increasing in the Republic of Korea recently. Because most of the CRC arise from adenomas, detection and complete removal of these precancerous lesions can reduce the incidence and mortality associated with CRC.2,3 However, for the effective prevention of CRC, high-quality colonoscopy that detects all the polyps, is a prerequisite. Adenoma detection rate (ADR) has been considered as one of the important quality indicators of colonoscopy, and the inverse association between ADR and incidence of interval CRC has been established.4,5 Because colonoscopy may not be perfect, many efforts to enhance ADR have been made.6 Artificial intelligence (AI), one of the promising technologies, can mitigate the shortcomings of colonoscopy. As it is easy to obtain polyp images to provide enough data for AI training, many studies using this technology have been reported lately.7 Well-trained AI modelling can increase polyp identification and optical diagnosis. Especially, its use can increase the detection rate of polyps in the right colon, where a higher miss rate is understandably anticipated with conventional colonoscopy.8

In this review, we aim to contemplate the current status and future directions of AI applications, including technologies, applications, efficacy, and unmet needs, regarding colorectal neoplasm.

COMPUTER-AIDED DIAGNOSIS AND DEEP-LEARNING FOR DETECTION AND CHARACTERIZATION OF THE COLORECTAL POLYPS

AI, having machine intelligence that is different from the natural intelligence displayed by humans and other animals,9 can learn and solve problems.10 Earlier, machine learning (ML) had been the main focus of AI. A computer-vision algorithm, developed by computer scientists based on earlier ML research for the detection of colorectal polyps was “hand-crafted,” based on the features adopted by the designers.11 In other words, human efforts or instructions were needed for extracting features, such as color, shape, or texture of polyps, because the earlier ML could learn only the classification of the extracted images.12 Although the “hand-crafted” algorithm showed high accuracy, the risk of missing the lesions without those extracted features or obtaining false positive results was of concern because it was designed to detect the lesions with certain features chosen by designers. In addition, actual clinical application was limited due to the difference in image quality and also slow processing time.11

Deep-learning (DL) algorithm, one of the subtype of ML introduced in the 1980s, could overcome the limitation of earlier ML by combining both the extraction and classification of image features using deep neural networks (DNN).13 The innovative method of DNN gained significant attention because of self-learning capability of DL that could automatically identify polyp and non-polyp features from the huge dataset, instead of capturing specific features of a polyp using several networks. Self-extraction, the key feature, was achieved using backpropagation algorithm and changing the internal parameters of each neural network layer.14

Among the variable classes of DNN for image and video applications, convolutional neural network (CNN), the most popular method, can carry out layers of convolutions and can completely connect layers to unite all features in the final outcome.15,16 Provided with sufficient annotated data, CNNs can be trained to describe, in detail, what they see and discriminate polyps from non-polypoid lesions.11 Currently, AI can be trained with enough input data, thanks to the easy accessibility of big data, aiding rapid progress in the research and application of AI in colonoscopic polyp detection and characterization.17

STUDIES RELATED TO APPLICATION OF AI IN COLONOSCOPY

1. AI for detection of colorectal polyp

Since the initial study regarding computer-aided detection based on “hand-crafted” data,18 the performance of AI has improved, especially with the introduction of DL, for colorectal neoplasm detection.19,20 Table 1 shows the summary of the clinical studies of AI for the detection of colorectal polyps.18-26

Table 1

Clinical Studies of Artificial Intelligence for the Detection of Colorectal Polyps

Author (year)	Study design	Algorithm type	Dataset	Processing time	Results
Wang et al. (2019) 21	Randomized controlled study	Convolutional neural network	5,545 Images	25 fps with 77 ms latency	9% Increase of ADR
Klare et al. (2019) 22	Prospective In vivo	Convolutional neural network	55 Live colonoscopies	50 ms latency	Sensitivity 75%/polypADR 29% (31% in endoscopist)
Urban et al. (2018) 20	Retrospective Ex vivo	Convolutional neural network	Image dataset: 8,641 polypsVideo: 20 colonoscopies	10 ms/frame (real-time)	Image dataset: accuracy 96.4%AUROC 0.991
Misawa et al. (2018)19	Retrospective Ex vivo	Convolutional neural network	135 Video clips	No description	Sensitivity 90%Specificity 63.3%Accuracy 76.5%
Zhang et al. (2017) 23	Retrospective Ex vivo	Convolutional neural network	150 Random+30 NBI images	No description	Sensitivity 98%PPV 99%AUROC 1.00
Yu et al. (2017) 24	Retrospective Ex vivo	Convolutional neural network	ASU-Mayo 20 videos	1.23 s/frame	Sensitivity 7%PPV 88%
Angermann et al. (2017) 25	Retrospective Ex vivo	Hand-crafted	No description	20–185 ms 0.3-1.8 s delay	Sensitivity 100%/polypPPV50%
Tajbakhsh et al. (2015) 26	Retrospective Ex vivo	Hand-crafted	No description	2.6 s/frame	Sensitivity 48% on proprietary databaseSensitivity 88% in CVC-colon DB
Karkanis et al. (2003) 18	Retrospective Ex vivo	Hand-crafted	180 Still images	1.5 m/video	Sensitivity 94%Specificity 99%

ADR, adenoma detection rate; AUROC, area under the receiver operating characteristics; NBI, narrow band imaging; PPV, positive predictive value; ASU, Arizona State University; CVC, computer vision center; DB, data base.

Urban et al. 20 reported the first real-time application. They initially pretrained AI using ImageNet and then trained the deep CNNs. The algorithm was tested with multiple sets of colonoscopic images and 11 challenging videos sets. The result was very promising with 97% sensitivity, 95% specificity, and 96% overall accuracy. For the practical utilization in real-world, the CNN-assisted video was reviewed by experts. During the index colonoscopy, 28 polyps were noted in nine standard colonoscopic videos. Experts could identify 36 polyps without and 45 polyps with CNN assistance. The additional 17 polyps identified with CNN assistance were not larger than 10 mm in size. Further, the algorithm was faster than the real-time analysis by endoscopists (10 ms/frame vs 33–40 ms/frame). Meanwhile, Yu et al. 24 developed a novel three-dimensional CNN algorithm that could learn more representative spatiotemporal features and improve performance of automated polyp detection.

Wang et al. 21 developed a CNN system that processed data in real-time with a 77 millisecond delay on monitor, in which, a blue box appeared around the area of the polyp upon its detection, along with an alarm. They randomized the colonoscopy of about 1,058 patients into those using this system and the standard one and proved an increased ADR in AI (29.1% vs 20.3%, respectively) with minimal delay in examining time. However, the robust detection rate was limited to polyps smaller than 10 mm in size, and 43.6% of the polyps were hyperplastic polyps with low malignant potential.27,28 The most recent meta-analysis of three randomized controlled trial of AI-assisted colonoscopy reported 32.9% of ADR (20.8% in standard colonoscopy, risk ratio [RR]=1.58, p<0.001) and 43.0% of polyp detection rate (27.8% in standard colonoscopy, RR=1.55, p<0.001).29 AI utility in screening colonoscopy to improve ADR looks optimistic.

2. AI for characterization of colorectal neoplasm

Accurate histologic diagnosis of the colorectal polyp before resection is desired because endoscopic resection and pathologic evaluation of lesions with very low risk of malignant potential may result in waste of time and cost. On the other hand, incomplete resection of lesions with high risk of malignancy should be avoided. Therefore, various advanced endoscopic techniques with image enhancement for optical diagnosis have been introduced, and various studies concerning the application of AI to these endoscopic systems have also been steadily reported to date (Table 2).

Table 2

Clinical Studies of Artificial Intelligence for Characterization of Colorectal Polyps

Author (year)	Study design	Classification target and base	Algorithm type	Image modality	Dataset	Results
Byrne et al. (2019) 35	Retrospective	Histology of diminutive polyp	Convolutional neural network	NBI video frames	125 Diminutive polyp videos	Sensitivity 98%Specificity 83%Accuracy 94%
Chen et al. (2018) 36	Retrospective	Neoplastic or hyperplastic polyp <5 mm	Convolutional neural network	Magnifying NBI	284 Diminutive polyps image	Sensitivity 96.3%Specificity 78.1%Accuracy 90.1%
Mori et al. (2018) 44	Prospective	Diagnosis of neoplastic diminutive polyp	SVM	Endocytoscopy with NBI and stained images	466 Diminutive polyps from 325 patients	Prediction rate 98.1%
Takeda et al. (2017) 43	Retrospective	Invasive CRC	SVM	Endocytoscopy with NBI and stained images	200 Images	Sensitivity 89.4%Specificity 98.9%Accuracy 94.1%
Kominami et al. (2016) 33	Prospective	Histology	SVM with logistic regression	Magnifying NBI	118 Colorectal lesions	Sensitivity 95.9%Specificity 93.3%Accuracy 94.9%
Misawa et al. (2016) 46	Retrospective	Microvascular findings	SVM	Endocytoscopy with NBI	100 Images	Sensitivity 84.5%Specificity 97.6%Accuracy 90.0%
Mori et al. (2015) 42	Retrospective	Neoplastic changes in small polyps	Multivariate regression analysis	Endocytoscopy	176 Polyps from 152 patients	Sensitivity 92%Specificity 79.5%Accuracy 89.2%
Takemura et al. (2012) 57	Retrospective	Pit pattern	SVM	Magnifying NBI	371 Images	Sensitivity 97.8%Specificity 97.9%Accuracy 97.8%
Gross et al. (2011) 32	Prospective	Small colonic polyp <10 mm	SVM	Magnifying NBI	434 Polyps from 214 patients	Sensitivity 95%Specificity 90.3%Accuracy 93.1%
Tischendorf et al. (2010) 31	Prospective pilot	Vascularization features	SVM	Magnifying NBI	209 Polyps from 128 patients	Sensitivity 90%Specificity 70.2%Accuracy 85.3%
Takemura et al. (2010) 39	Retrospective	Pit pattern	HuPAS software version 1.3	Magnifying NBI with chromoendoscopy (crystal violet)	134 Images	Accuracy 98.5%

NBI, narrow band imaging; SVM, support vector machine; CRC, colorectal cancer.

Narrow band imaging (NBI; Olympus Corp., Tokyo, Japan), an image-enhanced endoscopy used for the observation of microstructure or capillaries of colorectal neoplasm, could discriminate each polyp well, based on the pit and vascular pattern.30 Tischendorf et al. 31 and Gross et al. 32 investigated AI application to NBI (excluding its performance enhancement) for the characterization of colon polyps. They both extracted nine vessel features from the NBI images in a similar manner and sorted them into neoplastic and non-neoplastic lesions using a support vector machine (SVM), a discriminated ML model.15 The diagnostic accuracies of the two studies were 85.3% and 93.1%, respectively. In addition, Gross et al. 32 proved superior accuracy of AI-assisted diagnosis compared to that of nonexperts in differentiating colorectal polyps smaller than 10 mm in size, suggesting that AI could be a good support for the beginners of endoscopy.

Another Japanese group developed a real-time image recognition system that could achieve 93.2% accuracy in real-time prediction of diminutive polyp pathology.33 In addition, the follow-up recommendation based on the prediction by that model showed 92.7% consistency with Preservation and Incorporation of Valuable Endoscopic Innovations (PIVI)-2 criteria of American Society for Gastrointestinal Endoscopy for “resect-and-discard” strategy.34 Resect and discard strategy can bring a substantial economic benefit.

Two groups carried out retrospective studies based on DL. Byrne et al. 35 developed CNN model using NBI video frames. Although the model could not make adequate credence of histology prediction in 15% of polyps, it could differentiate diminutive adenomas from hyperplastic polyps with 94% accuracy. The sensitivity and specificity for detecting adenoma were 98% and 83%, respectively. Chen et al. 36 also developed a similar model, a DNN-computer aided diagnosis (CAD), with 2,157 images for identification of about 284 neoplastic or hyperplastic polyps smaller than 5 mm in size, with 89.6% positive predictive value (PPV) and 91.5% negative predictive value (NPV). Consequently, among 117 tubular adenomas, DNN-CAD could diagnose high grade dysplasia with 100% sensitivity and 94% specificity. The performance of DNN-CAD, when compared to that of four endoscopists with less than 1 year of experience, was superior with a shorter procedure time and perfect intra-observer agreement (kappa score of 1). The result of the above-mentioned two studies could satisfy the PIVI-2 threshold (90% NPV for adenoma detection) for “leave-in-place” strategy for diminutive hyperplastic polyps.34

Although there’s a lack of recent further studies, CAD of pit pattern by magnifying chromoendoscopy was done by quantitative analysis of pit structure or texture analysis of endoscopic images.37,38 Takemura et al. 39 reported 98.5% diagnostic accuracy after automatically evaluating the area, perimeter, major/minor fit ellipse and circularity of the pit using software.

Recently, novel introduction of in vivo contact microscopic imaging modalities, such as endocytoscopy (H290ECI; Olympus Corp.) and confocal laser endomicroscopy (Cellvizio; Mauna Kea Technologies Inc, Paris, France), enabled real-time diagnosis of cellular images.40,41 Because both endocytoscopy and endomicroscopy could enhance image analysis with focused fixed-size images, they were ideal to be used in combination with the AI system. They magnified the image with 500- or 1,000-fold power, respectively, during colonoscopy and showed diagnostic accuracy comparable to that of pathologists. The first application of AI in endocytoscopy was assessed by quantitative analysis of six nuclear features, and the accuracy for the detection of neoplastic change was 89.2%.42 Takeda et al. 43 trained the computer-aided ultrahigh (approximately ×400) magnification endoscopy system for the diagnosis of invasive CRC with 5,543 endocytoscopic images. This system, when assessed using 200 endocytoscopic test images, could discriminate invasive cancers with 89.4% sensitivity, 98.9% specificity, and 94.1% accuracy. Mori et al. 44 assessed the efficacy of an endocytoscopy-based CAD with 466 cases of diminutive polyps, and the NPV for diminutive rectosigmoid adenomas was 93.7%. They also proved that polyp diagnosis with AI, an add-on analysis, could reduce the cost of annual reimbursement for colonoscopy by 18.9%, by leaving 145 rectosigmoid diminutive polyps based on the AI support.45 However, AI endocytoscopy had a limitation–it needed pre-staining with crystal violet and methylene blue before the extraction of images. Misawa et al. 46 upgraded the system by combining endocytoscopy with NBI, eliminating the pre-staining step and resulting in 90.0% overall accuracy and 84.5% sensitivity within 0.3 seconds.

AI application with confocal endomicroscopy was assessed in several studies, in experimental setting, with promising accuracy.47 50 AI in other advanced endoscopies, including laser-induced fluorescence spectroscopy and autofluorescence endoscopy, was also evaluated retrospectively or prospectively.51-57 We can expect that the use of AI in these advanced endoscopies for optical diagnosis could aid the real-time decision making of endoscopists.

3. AI for combination of detection and characterization of colorectal polyps

For an endoscopist, both polyp detection and characterization are essential in clinical practice. An ideal scenario would be an AI-assisted immediate detection and characterization of the colorectal polyp. A Japanese group developed novel technologies that included two algorithm systems–one, based on DL algorithm, for the detection of polyps in white light images, and the other, for the prediction of pathology by endocytoscopic images generated by a photograph.58 According to the most recent study using CNN, AI system (Single Shot Multibox Detector) could detect 1,246 polyps with 92% sensitivity, 86% PPV, and 83% accuracy in polyp classification.59 Although more studies are needed, an ideal colonoscopy for the detection and characterization of colorectal neoplasm seems to be achieved with AI assistance.

4. Prediction of prognosis

Prior to DL, SVM was a highly efficient computational tool that classified and regressed best by optimizing a hyperplane with largest functional margin.15 Ichimasa et al. 60 evaluated the predictive factors for lymph node metastasis from endoscopically resected T1 CRC using SVM. Their result showed better sensitivity (100%), specificity (66%), and accuracy (69%) of the AI model than most of the current guidelines. In addition, this model could reduce the unnecessary additional colectomy after endoscopic resection of T1 CRC compared to the current guidelines that lead to misdiagnosis.

UNMET NEEDS AND FUTURE PERSPECTIVES OF AI FOR COLORECTAL NEOPLASM

Although AI-assisted detection and diagnosis of colorectal neoplasm are promising, most of the studies are retrospective and covered lesions which might have been selected with bias. Well-designed prospective studies that present more reliable data compared to the previous retrospective studies are needed. Most of the studies regarding efficacy of AI deal with polypoid lesions. However, for practical utilization, the efficacy of AI needs to be consistent irrespective of the shape of the polyps. Therefore, more studies with all types of polyps, including polypoid, depressed, or flat type, are needed because these non-polypoid type lesions are often more aggressive.61

In addition, as AI is trained with high-quality images, the system has to overcome the blurry vision, inadequate preparation status, and variable unpredictable hurdles observed in actual practice. There is a need for real-time application of CAD and randomized controlled comparative study between usage and avoidance of AI.

Previous studies, conducted in a variable design, resulted in different primary outcomes. Because AI development needed both engineers, who dealt with the software, and clinicians, who contributed to the clinical use, the outcomes and study design could be different depending on the study conductors.11 Communication and collaboration between these different groups are also needed.

To apply AI in real clinical practice, a regulatory approval of AI-based decision making, the rules of which vary for each country, is an essential step. For that, we need to prove the minimal risk of treatment failure due to the misdiagnosis by AI.22,62 So, we need the evaluation of risk stratification for AI system through well-designed randomized clinical trials.

CONCLUSION

Because colonoscopy cannot completely prevent CRC, AI application in the field of colorectal neoplasm could be one of the promising options for enhancing the efficiency of colonosocpy. Owing to easy accessibility of big data and computer science, the AI technologies for the detection and classification of colorectal polyps have developed rapidly, with studies supporting the advantage of AI use in colonoscopy. However, obstacles, such as insufficient evidence of practical clinical usefulness, lack of consensus on standardized utilization, and need for regulatory approval, exist. For the ideal implementation of AI in actual clinical practice, comprehensive understanding of the strengths and weaknesses of the technology, qualified real-time studies, and accumulation of experience are warranted.