Current status and limitations of artificial intelligence in colonoscopy.

BACKGROUND: Artificial intelligence (AI) using deep learning methods for polyp detection (CADe) and characterization (CADx) is on the verge of clinical application. CADe already implied its potential use in randomized controlled trials. Further efforts are needed to take CADx to the next level of development. AIM: This work aims to give an overview of the current status of AI in colonoscopy, without going into too much technical detail.
METHODS: A literature search to identify important studies exploring the use of AI in colonoscopy was performed.
RESULTS: This review focuses on AI performance in screening colonoscopy summarizing the first prospective trials for CADe, the state of research in CADx as well as current limitations of those systems and legal issues.

INTRODUCTION

The advance of artificial intelligence (AI) using deep learning (DL) methods opened a new scope in medical imaging analysis. Regarding possible applications in gastroenterology prevention of colorectal cancer (CRC) by screening colonoscopy has always been a main field of activity and research for endoscopists. An increase of adenoma detection rate (ADR) is expected to lead to a decrease of cancer risk but simultaneously burdens workload, costs, and risk of additional polypectomy to the patient.

The use of computer‐aided detection (CADe) and characterization (CADx) systems promise both augmented performance by increased ADR and higher efficiency by firm identification of nonadenomatous polyps. Thus, for example, hyperplastic polyps in the rectosigmoid can be left unresected. Yet physicians' sentiment toward AI is rather ambiguous. Strong interest and concerns about unnecessary workload, development of operator deskilling, and over‐reliance on AI are facing one another. Several commercial CAD systems have entered the market already (Table 1). The ESGE states that incorporation of these systems in clinical routine is possible as soon as high‐quality multicenter studies demonstrated acceptable accuracy. This review focuses on AI performance in screening colonoscopy. It provides an overview of the first prospective trials in CADe, the state of research in CADx as well as current limitations of those systems and legal issues.

TABLE 1

Overview of commercial CAD systems

Name	EndoBrain	GI Genius	Discovery	REiLI CAD EYE	ai4gi	EndoScreener
Company	Cybernet Systems Co.	Medtronic	Pentax Medical	Fujifilm	Imagia, Satis	Shanghai Wision AI Co
Integration	CF‐H290ECI, Olympus Corp.	Different vendors possible	Pentax colonoscopes	Fujifilm's 700 series colonoscopes	Different vendors possible	Different vendors possible
Dual monitor in study	No	No	Yes	No	No	Yes
Study data	Mori Y et al. (2016), Misawa M et al. (2016)	Repici A et al. (2020)	n/a	Non	Byrne M et al. (2017)	Wang P et al. (2020)
Mode	CADe and CADx	CADe	CADe	CADe and CADx	CADe and CADx	CADe
Regulatory	Approved by PMDA 12/2018	CE‐approved 07/2019	CE‐approved 01/2020	CE‐approved 02/2020	No	No

POLYP DETECTION (CADe)

The main contribution of screening colonoscopy to CRC prevention is the ability to detect precancerous lesions and the possibility to remove them. The most intensively studied AI that was built using DL and that bridged the gap from an experimental tool tested on retrospectively collected colonoscopy videos to the clinical practice is the image‐based polyp detection. Until now, six prospective randomized trials analyze the performance of such systems in clinical practice , , , , , (Table S1).

Five of those randomized controlled trials have ADR as the primary endpoint. One tandem trial analyses the adenoma miss rate (AMR) as a primary endpoint. The most important finding is that all trials powered for ADR comparison present a significant increase in ADR. Pooling the data from those trials present an increase in ADR of 6–15.2 percentage points (median 12.4), depending on the experience of the examiners and the inclusion criteria (Table 2). The range of ADR using AI rose in one multicenter study up to 54.8% in the CADe group compared to 40.4% in the control group. Thus, extrapolating those results to the studies that correlated the ADR to the rate of interval cancer and death due to interval cancer would suggest that there is a huge benefit if this new technology is applied in clinical practice. A more detailed analysis of the randomized trials with ADR as a primary endpoint reveals that all report on a significant increase in detection of diminutive adenomas of 5 mm and lower. Only one study reported a significant increase in the detection of adenomas of 6–9 mm. None of those studies present an increase in the detection of larger adenoma.

TABLE 2

Qualitative overview of AI capability for polyp detection (CADe)

	Adenoma detection rate improvement	Polyp size (mm)			Location		Appearance
		<6	6–9	>9	Proximal	Distal	Flat	sessile	Pedunculated
AI benefit	+++	+++	+		++	+++	+++	+++

Note: Increase in plus signs resembles higher number of studies supporting this feature.

Abbreviation: AI, artificial intelligence.

Interestingly the withdrawal time without the duration of polyp removal did not significantly differ between examinations with CADe support or without in any of the studies.

Taken together, the trend toward an increase in ADR will go on and will be pushed even further due to the introduction of CADe.

QUALITY CONTROL OF ENDOSCOPY (QC)

Key performance measures for lower gastrointestinal endoscopy aim to improve quality. AI might help to objectively record key performance measures and thus contribute to an improved quality of the procedure. One of the most important quality measures is the withdrawal time. It corresponds with the ADR and with death due to interval cancer. The recommended time for withdrawal is six minutes or more. , However, there is no recommendation on the time the examiner should spend per colon segment. In practice, some examiners move quicker in the ascending colon and then slow down after realizing that the withdrawal time will be too short resulting in a more thorough examination of the distal colon. The withdrawal speed and time spent per optically covered surface might be a more reliable measurement tool. In the study of Gong D et al., the authors could correlate similarity scores of images to each other and relate them to withdrawal speed. Setting thresholds for withdrawal speed in this randomized trial resulted in a higher ADR of 16% compared to the control group with 8% obtained in a non‐screening population. An additional feature of the system was to recognize endoscope slipping during withdrawal. To help resume colonoscopy at the point the endoscope slipped, it presented the last ten frames before slipping at the bottom of the screen.

Another important quality measurement is the adequacy of bowel cleansing. The Boston Bowel Preparation Scale (BBPS) became the most commonly used tool for this estimation. The scale is applied during withdrawal after cleansing maneuvers including washing of mucosa and suction of residual fluids have been applied. Poor bowel preparation is associated with a reduced ADR. , Thus, different European guidelines recommend a BBPS value of 6 and more as adequate with none of the three segments being below 2. , , One disadvantage of this scale is that the examiners recall the scoring impression from their memory. A more detailed and precise subdivision might better reflect reality. Zhou J et al. trained an AI to estimate the BBPS every 30 s and thus present an overview of the bowel preparation. A further step forward was performed by Thakkar et al. who developed an algorithm that displays quality of examination metrics comprising total percentage of visualized colonic surface area, colon preparation, clarity of the endoscopic view, and colon distension in real‐time to the endoscopist. After this proof‐of‐concept study, further efforts are needed to quantify the contribution of such AIs to colonoscopy performance. Additionally, the potential of distraction from the endoscopic image introduced by such digital assistants needs to be evaluated.

POLYP CHARACTERIZATION (CADx)

The vast majority of polyps that are detected during screening colonoscopy are sized less than 5 mm. Those diminutive polyps rarely progress to advanced adenomatous histology. , Therefore removal and histopathological examination burden costs and workload to the health systems.

Optical biopsy means distinguishing adenomatous and nonadenomatous lesions by real‐time assessment of polyps using advanced endoscopy imaging techniques. Many classifications were introduced to distinguish adenoma's different grades of dysplasia or sessile serrated adenoma (SSA) from hyperplastic tissue. However, their consequent implementation in the clinical routine is rare. , , , Identification of hyperplastic polyps enables “resect‐and‐discard” and “diagnose and leave” strategies resulting in shorter procedure time, fewer adverse events, and lower costs. To implement these strategies, quality directives of the American Society for Gastrointestinal Endoscopy PIVI have been determined for diminutive polyps in the rectosigmoid. For resect and discard polyps the threshold of ≥90% agreement in the assignment of post‐polypectomy surveillance compared to histopathology has to be achieved. Leaving polyps requires a negative predictive value (NPV) ≥90% for adenomatous histology with high confidence. These thresholds are not reached outside expert centers. , , This leads to dissent of international (ASGE/ESGE) and national guidelines. ,

CADx as a virtual assistant can overcome interobserver variability. CADx systems do not rely on optical clinician‐based classifications. They rely on histological findings although this is also prone to interobserver variability. Especially, pathologists often do not agree regarding the differentiation between SAA and hyperplastic polyps.

To evaluate practical approaches, we focus in this review on WLE and so‐called “push the button technologies” (Table S2).

Renner et al. presented in a prospective study evaluating 100 images of polyps, an accuracy of 78% in differentiating between adenomatous and nonadenomatous histology. Predominantly diminutive polyps were analyzed, and the NPV was 88.2%. Chen et al. added full magnification imaging modality and achieved a substantially higher specificity in a set of 384 diminutive polyps resulting in an NPV of 92.6% in their validation set. The performance of the convolutional neural network (CNN) was equal to expert colonoscopists but superior to novices.

Using a training data set of >5000 retrospectively collected images, Zachariah et al. could reach an accuracy of 93.6% with an NPV of 92.6% with their CNN. There was no significant difference in using narrow‐band‐imaging or HD‐WLE images for analysis.

To implement CADx systems in clinical routine, real‐time characterization of polyps detected during colonoscopy is a prerequisite. One CNN developed by Komeda et al. showed an accuracy of 75.1% in the characterization of 180 polyp images in real‐time. Byrne et al. validated their system with 125 videos and achieved a high accuracy of 94%. However, 15% of the videos could not be characterized with high confidence.

In conclusion, the feasibility of CADx systems and thereby improvement of optical biopsy with accuracy >90% have been shown (Table 3). Confirmation of those results have to be shown in real life settings.

TABLE 3

Qualitative overview of AI capability for polyp characterization (CADx)

	Adenoma/Non‐adenoma (mm)			SSA/Hyperplastic	Invasiveness depth CRC
Differentiation	<6	6–9	>9	Any size	TA	HGD/T1	>=T2
AI benefit	+++	+	+	n/a	+		+

Note: Increase in plus signs resembles higher number of studies supporting this feature.

Abbreviations: AI, artificial intelligence; CRC, colorectal cancer; HGD, high‐grade dysplasia; TA, tubular adenoma.

LIMITATIONS

Although there have been great advances of AI in GI‐endoscopy, its contribution to improved CRC surveillance is still uncertain. Limitations include that existing data of CAD systems has exclusively been collected using in‐hospital colonoscopy settings, usually resembling a mixture of inpatient and outpatient colonoscopy procedures. All randomized prospective studies for polyp detection except one were done in China. A comparison of the ADR in those studies performed in China with the one study done in Italy reveals a marked difference in baseline ADR. , , , , , A reason for this difference might have been different expertise of the participating endoscopists. Furthermore, the studies performed in China were all single‐center studies. These limitations regarding the studies' populations prevent the extrapolation of the current data to the outpatient CRC‐surveillance cohorts in different regions. Multicenter studies in outpatient settings have to confirm the significant improvements in ADR.

Moreover, improvement of ADR in colonoscopy mainly by the identification of diminutive adenomas should be critically scrutinized to what extent it adds clinical benefit to the patient. CADe systems report an increase of ADR but no increase was shown for detection of advanced adenoma. One reason might be that the studies were not powered to detect an increase in recognition of advanced adenomas. High‐grade dysplasia is found in less than 1% of diminutive adenomas, defined as smaller than 6 mm. Their natural history is not as clearly associated with interval cancer as in the case of adenomas 1 cm and greater in size. , , The well‐established link between improved ADR and decrease of interval cancer could be caused by enhanced thoroughness of inspection or enhanced mucosal visualization. CADe can only contribute to the first point. Thus, future studies are necessary to reveal if the usage of AI for polyp detection is associated with increased survival due to a lower rate of interval cancer.

The sole use of CADe system has the potential to increase endoscopists' workload due to unnecessary resection of additional detected hyperplastic polyps. In particular, inexperienced examiners could be prone to that problem. Combined CADx systems can encounter this problem but their development is at an earlier stage. Existing study data mainly relies on the analysis of high‐definition images, which have been in part retrospectively collected and reviewed for their quality. , , Prospective multicenter studies analyzing data of real‐world images in real time have to confirm these results. A further limitation is the lack of data for the differentiation of SAAs and hyperplastic polyps. SSAs were excluded in those studies because of the similarity of their surface pattern to hyperplastic polyps.

In particular, nonexpert colonoscopists are expected to benefit from support by CAD systems. Improved detection rates and confidence in the characterization of lesions are expected. However, the learning phase could potentially be hindered by the black‐box nature of DL algorithms when using the CAD as a substitute for a skilled mentor who teaches patterns of polyp classification. Furthermore, concentration distraction (Figure 1) and over‐reliance on AI systems are potential sequelae. Studies involving examiners at different levels of experience have to evaluate this point.

FIGURE 1

Example of images obtained using a commercial CADe system. Left image: Recorded raw video signal presenting a polyp that was undetected by the endoscopist and highlighted in the middle image by the CADe. Right image: Distraction of the endoscopist by the CADe due to the highlighting of a small mucosal protrusion instead of the arrow marked polyp

Evaluation of AI systems is of urgent need as several commercial systems have entered the market already. There is no standardized way of describing algorithm architecture, data sets, as well as training, validation, and testing of CAD systems. Technical details pose many challenges to clinicians. Differing test data sets resembling various clinical challenges prevent direct comparability of the performance between different systems. Benchmarking data sets containing heterogeneous data, representing the natural variability of polyps, could overcome that problem if used to evaluate multiple systems the same way.

Until today, there is no standard definition for false positive (FP) or false negative (FN) detections. Some research groups defined a qualitative definition in which they considered a FP detection any activation continuously traced by the system or a consistent detection. , , , Other research groups consider FP detections as any activation, irrespectively of the number of frames. By using a frame‐by‐frame approach, meaning counting an FP as any false activation and FN as any missed activation in a frame, the comparison between CADe systems quality would be more accurate. In this sense, Bernal et al. proposed “temporal coherence” on the method's response as a way to evaluate these features that would solve such a problem. Standards on AI validation have to be established. Furthermore, clinical relevance of FP detections has not been established so far. Hassan et al. proposed a classification system to address this problem, which should be evaluated in further studies.

COMMERCIAL CAD SYSTEMS AND REGULATIONS

Several commercial CAD systems have now entered the market (Table 1). Usage of any device in Europe requires a CE mark. From May 2021, a new Medical Devices Regulation (MDR) will be applied in Europe. Devices will still be classified in different risk classes (Class I, Class IIa, Class IIb and Class III) according to their intended purpose. Class I are noninvasive low‐risk devices. Class IIa are low to medium risk, Class IIb are medium to high and Class III are high‐risk devices.

The first established CADx system was EndoBRAIN from Cybernet Systems Co., Japan, that has been developed in collaboration with Kudo S and Mori K et al. It is based on endocytoscopy and a conventional machine learning algorithm. Analyzing images taken at 520‐fold magnification the CADx accuracy in distinguishing neoplasms from non‐neoplasms is about 98%. An additional CADe system called EndoBRAIN‐EYE was released in 2020.

GI Genius was the first commercial CADe system. It obtained CE approval in July 2019 and is marked by Medtronic, Ireland. The CADe system has been developed by Cosmo Pharmaceuticals and validated by Hassan C and Repici A et al. Its utilization in real‐time colonoscopy increased significantly ADR in a multicenter trial. DISCOVERY from Pentax Medical, Japan, is another CADe system that was CE approved in January 2020. Both systems were classified as Class I. In their case, the intended purpose is to work as an adjunct highlighting region such as colorectal polyps, but under no circumstances, their output can override that of the endoscopist. In such cases, the clinical evidence required is less stringent.

Under the new MDR, almost all software will be up‐classified into higher risk classes. Software intended to provide information that is used to make decisions with diagnosis or therapeutic purposes is classified as Class IIa, or even higher depending on the risk associated with this decision. Recently, United European Gastroenterology recommended that all AI‐based systems for polyp detection should be classified as IIa‐products and those used for differentiation as IIb‐products.

The corresponding system from Fujifilm is called CAD EYE. It supports polyp detection in white light endoscopy (WLE) and Linked color imaging as well as polyp characterization in Blue Light Imaging (BLI). The CADe system obtained a CE mark in February 2020.

Ai4gi was developed in collaboration with Byrne M and Rex D et al. It is commercialized by a joint venture between Satis Operations and Imagia, USA. The CADe system automatically switches to CADx mode when NBI light is detected. An accuracy of 97.6% for differentiating between NICE 1 and NICE 2 polyps has been shown recently.

The EndoScreener from Shanghai Wision AI Co., China, is a CADe system developed in a joint work with Wang P et al. Although lacking CE approval, it has already been evaluated in prospective clinical trials in China. By its use, increased ADR for diminutive polyps and a significantly lower AMR compared to experienced colonoscopists was shown. ,

Approval of future CAD systems will require a significantly higher level of evidence and they must provide post‐market surveillance and prepare a periodic safety update report (‘PSUR’)51.

A White Paper on AI was released by the European Commission to present policy options. A High‐Level Expert Group published nonbinding risk‐based Guidelines on trustworthy AI. They conclude that the future regulation should deal with the following: data and record‐keeping to allow traceback and verification, robustness and accuracy in results, gender and racial bias, and personal data and privacy protection.

OUTLOOK

AI in polyp detection and characterization is on the verge of clinical application. CADe already implied its potential use in prospective trials. Further efforts are needed to take CADx to the next level of development. The prediction of clinical impact in CRC surveillance is limited by the current study data. Future devices will encounter more challenging requirements for approval and will be categorized into higher MDR classes.

CONFLICT OF INTEREST

The authors Alexander Hann and Daniel Fitting receive public funding from the state government of Baden‐Württemberg, Germany (Funding cluster “Forum Gesundheitsstandort Baden‐Württemberg”) to research and develop artificial intelligence applications for polyp detection in screening colonoscopy.

AUTHOR CONTRIBUTIONS

Conception and design: all authors; drafting of the article: all authors; critical revision of the article for important intellectual content: all authors; final approval of the article: all authors.

Supporting Information 1

Click here for additional data file.