Endoscopic prediction of deeply submucosal invasive carcinoma with use of artificial intelligence.

Background and study aims  We evaluated use of artificial intelligence (AI) assisted image classifier in determining the feasibility of curative endoscopic resection of large colonic lesion based on non-magnified endoscopic images Methods  AI image classifier was trained by 8,000 endoscopic images of large (≥ 2 cm) colonic lesions. The independent validation set consisted of 567 endoscopic images from 76 colonic lesions. Histology of the resected specimens was used as gold standard. Curative endoscopic resection was defined as histology no more advanced than well-differentiated adenocarcinoma, ≤ 1 mm submucosal invasion and without lymphovascular invasion, whereas non-curative resection was defined as any lesion that could not meet the above requirements. Performance of the trained AI image classifier was compared with that of endoscopists. Results  In predicting endoscopic curative resection, AI had an overall accuracy of 85.5 %. Images from narrow band imaging (NBI) had significantly higher accuracy (94.3 % vs 76.0 %; P  < 0.00001) and area under the ROC curve (AUROC) (0.934 vs 0.758; P  = 0.002) than images from white light imaging (WLI). AI was superior to two junior endoscopists in terms of accuracy (85.5 % vs 61.9 % or 82.0 %, P  < 0.05), AUROC (0.837 vs 0.638 or 0.717, P  < 0.05) and confidence level (90.1 % vs 83.7 % or 78.3 %, P  < 0.05). However, there was no statistical difference in accuracy and AUROC between AI and a senior endoscopist. Conclusions  The trained AI image classifier based on non-magnified images can accurately predict probability of curative resection of large colonic lesions and is better than junior endoscopists. NBI images have better accuracy than WLI for AI prediction.

Introduction

Use of artificial intelligence (AI)-assisted image analysis recently has been shown to be able to distinguish diminutive colorectal adenoma from hyperplastic polyps with high accuracy 1 2 3 4 , further enhancing adoption of “resect and discard” strategy by less experienced endoscopists. Another important diagnostic challenge faced by endoscopists is whether it would be curative (R0 resection) to remove large colorectal neoplasms by endoscopic means including endoscopic submucosal dissection (ESD) or mucosal resection (EMR). Traditional features of endoscopic appearance of colonic lesions suggestive of non-curative endoscopic resection include deep depression, fold convergence, and irregular bottom in a depressed surface 5 . A highly irregular, heterogenous and avascular area in the microvascular pattern of a large colonic lesion is also suggestive of a deep invasive component 7 8 9 10 11 . There are well-established indications for endoscopic resection including histological subtypes and risk of submucosal invasion and presence of lymphovascular permeation exist 1 2 , but experienced endoscopists nevertheless are usually required to achieve resection. Herein, we evaluated use of an AI- assisted image classifier in predicting feasibility of curative endoscopic resection of large colonic lesions (≥ 2 cm), based on non-magnified endoscopic images.

Methods

This was a retrospective, single-center, endoscopic image and pathology correlation study, by means of novel AI deep learning algorithms conducted in the Integrated Endoscopy Center of the Queen Mary Hospital of Hong Kong, which is a major regional hospital serving the Hong Kong West Cluster and a university teaching hospital. The study protocol was approved by the Hospital Authority Hong Kong West Cluster & Hong Kong University Institutional Review Board. All baseline endoscopic procedures were performed by experienced endoscopists with non-optical magnifying colonoscopes (CF-H260, CF-HQ290 model and CV-260 or CV-290 video system, Olympus, Tokyo, Japan).

Definition

Endoscopically curable lesions (ECL) were defined as those that had no risk of lymph node metastasis, and thus could be completely managed by endoscopic resection. These lesions included sessile serrated adenoma, tubular adenoma with or without villous component, intramucosal adenocarcinoma and shallow submucosal adenocarcinoma that fulfill the following criteria: submucosal invasion depth ≤ 1000 um, papillary or tubular adenocarcinoma and no lymphovascular permeation. Endoscopically incurable lesions (EIL) were defined as adenocarcinoma that did not satisfy any of the three above criteria 2 .

Creation of the AI image classifier and training set

The AI image classifier we developed was built on a convolutional neural network (CNN) (with 5 convolutional layers and 3 fully connected layers) by using still endoscopic images of large colonic lesions obtained between January 2006 and July 2017. The AI image classifier was trained based on a pre-trained ResNet CNN backbone. All the still images were collected via the electronic patient record system or captured from colonoscopy video. Image resolution was at least 720 × 526 pixels. All the training images collected were pre-screened by an expert who had intensive endoscopic training (TKLL) and had performed more than 2000 image-enhanced colonoscopies with narrow band imaging (NBI; Olympus) and more than 400 ESD cases. Multiple images per lesion were obtained in the training set. The region of interest (ROI) within the endoscopic images, which included all endoscopically visible abnormal mucosa area, was highlighted (by TKLL). ROI that focused on adenomatous areas only in the non-curative lesions from the training images were excluded. All images that had motion – artifact, were out of focus, insufficiently bright or covered with mucus were also excluded.

The final training set consisted of 8,000 ROI images, including 4,000 ROI images from 1,159 endoscopically curable lesions and 4,000 ROI images from 493 endoscopically incurable lesions. Among these training images, 4,000 ROI were taken with NBI and 4,000 ROI images were taken under white light imaging (WLI) to ensure balance of training data. A total of 10 % of all training images were randomly chosen as an internal validation set with 100 % internal accuracy achieved for the trained AI image classifier.

Testing set

The independent testing set consisted of 76 large colonic lesions that fulfilled the indications for ESD 1 and were removed between August 2017 and July 2018. Endoscopic en bloc resection was a requirement but snare EMR was difficult to apply. Large depressed-type tumors and large protruded-type lesions suspected to be carcinoma were included. Heterogenous lateral spreading tumors with mixed adenomatous and suspected invasive components were excluded. The region of interest (ROI) within the endoscopic images, which included all endoscopically visible abnormal mucosal areas, was again highlighted by an expert who had intensive training (TKLL). To minimize selection bias, three ROIs were randomly selected from each endoscopic image of a lesion. The ROI images were then analyzed by the trained AI image classifier for the mean probability of endoscopically curative resection ( Fig. 1 ). Histology of the final resected specimens was used as the gold standard when determining performance of the trained AI image classifier.

Fig. 1

 Diagnostic flow from an endoscopic image to final AI result.

The performance of AI was also compared with that of three endoscopists. Two of them (Junior Endoscopist A & Junior Endoscopist B) had performed more than 500 colonoscopies and had special NBI training with tutorials and endoscopic image education. Another more experienced endoscopist (Senior Endoscopist C) had performed more than 2000 NBI colonoscopies and more than 300 ESD cases. All three endoscopists reviewed the testing set endoscopic images using a standard monitor with a resolution of at least 1920 × 1080 to determine whether the colonic lesions were ECL or EIL and the probability level.

Statistical analysis

Categorical data were compared by the χ2-test or Fisher Exact test where appropriate. Numerical data were analyzed with the Student’s t -test. Statistical significance was taken as two-sided ( P value < 0.05). A two-by-two table was constructed using the predicted and actual outcome to calculate different domains in the diagnostic test of sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and accuracy, and area under ROC (AUROC). Confidence intervals (CIs) used for sensitivity, specificity and accuracy were Clopper-Pearson confidence intervals. CIs for the predictive values were the standard logit confidence intervals. All statistical analysis was performed by SPSS statistics software (version 19.0, SPSS, Chicago, Illinois, United States).

Results

Mean size of the 76 endoscopically removed lesions in the testing set was 26.6 mm (range: 20 to 60 mm). More than half of the lesions were located in the sigmoid colon (53.9 %). The rest of the lesions were located in the ascending colon (14.5 %), transverse colon (10.5 %), cecum (7.9 %), descending colon (7.9 %) and rectum (5.3 %). Based on final histology of the resected specimens, 56 (74.6 %) were endoscopically curable including 28 (50 %) tubular adenoma, 10 (17.9 %) tubulovillous adenoma, eight (14.3 %) sessile serrated adenoma and 10 (17.9 %) adenocarcinoma ( Table 1 ).

Characteristics of large colonic lesions in the final testing set.

	Endoscopically curable lesion	Endoscopically incurable lesion	Total
Number of lesions	56	20	76
Location
Cecum	5 (8.9 %)	1 (5 %)	6 (7.9 %)
Ascending colon	10 (17.8 %)	1 (5 %)	11 (14.5 %)
Transverse colon	8 (14.3 %)	0 (0 %)	8 (10.5 %)
Descending colon	4 (7.1 %)	2 (10 %)	6 (7.9 %)
Sigmoid	28 (50 %)	13 (65 %)	41 (53.9 %)
Rectum	1(1.8 %)	3 (15 %)	4 (5.3 %)
Histology
Invasive adenocarcinoma (SM2) 1	0 (0 %)	20 (100 %)	20 (26.3 %)
Slightly invasive adenocarcinoma (SM1)	10 (17.9 %)	0 (%)	10 (13.1 %)
Tubulovillous adenoma	8 (14.3 %)	0 (0 %)	8 (10.5 %)
Tubular adenoma	28 (50 %)	0 (0 %)	28 (36.8 %)
Sessile serrated adenoma	10 (17.9 %)	0 (0 %)	10 (13.2 %)

12 lesions had presence of lymphovascular invasion

The testing set comprised a total of 567 ROIs (296 NBI and 271 WLI images). Overall, the trained AI image classifier had a sensitivity of 88.2 % (95 %CI: 84.7 % to 91.1 %) in differentiating ECL vs EIL with an overall specificity of 77.9 % (95 %CI: 70.3 % to 84.4 %) and accuracy of 85.5 % (95 %CI: 82.4 % to 88.3 %). When comparing the two different imaging modalities, NBI had significantly higher sensitivity (94.6 % vs 78.2 %; P < 0.0001), specificity (92.3 % vs 72.6 %; P  = 0.05), PPV (98.8 vs 81.7 %; P  < 0.0001), accuracy (94.3 % vs 76.0 %; P  < 0.00001) and AUROC (0.934 vs 0.758; P  = 0.002) than WLI in predicting complete endoscopic curative resection ( Table 2 ).

Performance of the AI image classifier in predicting ECL based on non-magnified endoscopic images.

	All	NBI	WLI	P value
Sensitivity	88.2 % (95 % CI: 84.7 %- 91.1 %)	94.6 % (95 % CI:91.0 %-97.0 %)	78.2 % (95 % CI:71.1 %-84.2 %)	< 0.0001
Specificity	77.9 % (95 % CI: 70.3 % – 84.4 %)	92.3 % (95 % CI:79.1 % – 98.4 %)	72.6 % (95 % CI:63.1 %-80.9 %)	0.05
PPV	92.1 % (95 % CI: 89.5 % – 94.1 %)	98.8 % (95 % CI:96.5 %-99.6 %)	81.7 % (95 % CI:76.4 %-86.0 %)	< 0.0001
NPV	69.3 % (95 % CI: 63.2 %-74.8 %)	72.0 % (95 % CI:60.5 %-81.2 %)	68.1 % (95 % CI:61.0 %-74.5 %)	0.41
Accuracy	85.5 % (95 % CI:82.4 % – 88.3 %)	94.3 % (95 % CI:91.0 %-96.2 %)	76.0 % (95 % CI:70.5 %-81.0 %)	< 0.00001
AUROC	0.837 (95 % CI: 0.794 – 0.880)	0.934 (95 % CI: 0.883 – 0.985)	0.758 (95 % CI: 0.697 – 0.819)	0.002

AI, artificial intelligence; ECL, endoscopically curable lesion; CI, confidence interval; NBI, narrow-band imaging; WLI, white light imaging; PPV, positive predictive value; NPV, negative predictive value; AUROC, area under the receiver operator curve

P value: NBI vs WLI

The accuracy of AI image classifier was similar for left-sided and right-sided colon lesions (83.5 % vs 88.0 %; P  = 0.07), and the performance was similar in other domains ( Table 3 ). The main reason for incorrect results (14.5 %, n = 82 ROI) was unclear vascular pattern on the surface of the lesion, especially with WLI endoscopic images ( Fig. 2 ). Mean probability level for AI prediction was 0.90 ± 0.13SD with skewness of –1.50 and Kurtosis 1.22. Comparing average probability of the AI for correctly and incorrectly predicting images, the AI had significantly higher confidence in correctly predicted images (0.912 vs 0.864, P  < 0.001). The AI was also more confident in making ECL predictions than EIL predictions (0.924 vs 0.844, P  < 0.001), and when using NBI images for prediction than WLI (0.926 vs 0.873, P  < 0.001).

Performance of the AI image classifier in predicting ECL based on non-magnified endoscopic images according to location.

	All n = 76	Left-sided colon n = 45	Right sided colon n = 31	P value
Sensitivity	88.2 % (95 % CI: 84.7 %- 91.1 %)	84.9 % (95 % CI:79.5 %-89.5 %)	90.3 % (95 % CI:85.6 %-93.9 %)	0.213
Specificity	77.9 % (95 % CI: 70.3 % – 84.4 %)	79.2 % (95 % CI:68.0 % – 87.8 %)	80.3 % (95 % CI:68.7 %-89.1 %)	0.871
PPV	92.1 % (95 % CI: 89.5 % – 94.1 %)	92.4 % (95 % CI:88.5 %-95.0 %)	93.8 % (95 % CI:90.2 %-96.1 %)	0.403
NPV	69.3 % (95 % CI: 63.2 %-74.8 %)	64.0 % (95 % CI:55.9 %-71.5 %)	71.6 % (95 % CI:62.3 %-79.4 %)	0.116
Accuracy	85.5 % (95 % CI:82.4 % – 88.3 %)	83.5 % (95 % CI:78.7 %-87.6 %)	88.0 % (95 % CI:83.6 %-91.5 %)	0.317
AUROC	0.837 (95 % CI: 0.794 – 0.880)	0.821 (95 % CI:0.760 – 0.882)	0.853 (95 % CI: 0.792 – 0.913)	0.548

AI, artificial intelligence; ECL, endoscopically curable lesion; CI, confidence interval; PPV, positive predictive value; NPV, negative predictive value; AUROC, area under the receiver operator curve

P value: left-sided colon vs right-sided colon

Fig. 2 

Example of inaccurate results.

Compared to junior endoscopists, AI was superior in terms of PPV (92.1 % vs 85.1 % or 86.2 %, P  < 0.05), accuracy (85.5 % vs 61.9 % or 82.0 %, P  < 0.05), and AUROC (0.837 vs 0.638 or 0.717, P  < 0.05) ( Table 4 ). AI was also more confident than the two human endoscopists in predicting curative endoscopic resection (0.901 vs 0.837 or 0.783, P  < 0.05). However, AI and the senior endoscopist had no statistical difference in performance of almost all domains except specificity (77.9 % vs 52.6 %, P  < 0.05) and NPV (69.3 % vs 50.6 %, P  < 0.05), for which AI seems to have performed slightly better statistically ( Table 4 ).

Performance of the AI image classifier vs human endoscopists in predicting ECL based on non-magnified endoscopic images.

	Al	Junior Endoscopist A	Junior Endoscopist B	Senior Endoscopist C
Sensitivity 1	88.2 % (95 % CI: 84.7 %- 91.1 %)	60.1 % (95 % CI:55.3 %-64.8 %)	87.6 % (95 % CI:84.2 %-90.6 %)	91.8 % (95 % CI:89.1 %-94.1 %)
Specificity 2 3	77.9 % (95 % CI: 70.3 % – 84.4 %)	67.1 % (95 % CI:58.9 % – 75.1 %)	56.4 % (95 % CI:47.5 %-64.7 %)	52.6 % (95 % CI:40.9 %-64.0 %)
PPV 2	92.1 % (95 % CI: 89.5 % – 94.1 %)	85.1 % (95 % CI:81.6 %-88.6 %)	86.2 % (95 % CI:83.8 %-88.4 %)	92.4 % (95 % CI:90.6 %-93.9 %)
NPV 1 3	69.3 % (95 % CI: 63.2 %-74.8 %)	35.2 % (95 % CI:31.6 %-39.1 %)	59.2 % (95 % CI:52.0 %-66.1 %)	50.6 % (95 % CI:41.5 %-59.6 %)
Accuracy 1 2	85.5 % (95 % CI:82.4 % – 88.3 %)	61.9 % (95 % CI:57.8 %-65.9 %)	80.0 % (95 % CI:76.5 %-82.0 %)	86.4 % (95 % CI:83.4 %-89.2 %)
AUROC 1 2	0.837 (95 % CI: 0.794 – 0.880)	0.638 (95 % CI:0.585 – 0.690)	0.717 (95 % CI: 0.663 – 0.771)	0.873 (95 % CI: 0.739 – 0.907)
Mean probability 1 2	0.901	0.837	0.783	0.803

AI, artificial intelligence; ECL, endoscopically curable lesion; CI confidence interval/ PPV, positive predictive value; NPV, negative predictive value; AUROC, area under the receiver operator curve

AI is statistically better than Junior Endoscopist A ( P  < 0.05)

AI is statistically better than Junior Endoscopist B ( P  < 0.05)

AI is statistically better than Senior Endoscopist C ( P  < 0.05)

Discussion

In this pilot study, we showed that a trained AI image classifier can accurately predict probability of ECL. Unlike previous studies that focused on diminutive polyps 3 4 6 , this study targeted larger endoscopic lesions with risk of submucosal invasion and lymphovascular permeation. The overall sensitivity of this AI image classifier was 88.2 % and specificity was 77.9 %. However, use of NBI endoscopic images was found to have significantly better performance than ordinary WLI images with sensitivity and specificity higher than 92 %. This is the first study to show that use of NBI was superior to WLI for AI-assisted endoscopic diagnosis. Moreover, instead of using magnified chromoendoscopy images, we used non-magnified endoscopic images as in usual daily practice. The better performance of the NBI images for AI discrimination could be accounted for by the highlights of the vascular patterns of the colonic lesions with NBI, which have been well reported in previous studies 7 8 9 10 11 . Specifically, previous studies have demonstrated the accuracy of endoscopic prediction of invasive carcinoma by using the Narrow-Band Imaging International Colorectal Endoscopic (NICE) classification with an overall sensitivity, specificity, PPV and NPV of around 90 % 6 7 . Those results were achieved by expert endoscopists using magnified NBI images. The current study also included a head-to-head comparison between AI and human endoscopists with special training on NBI and found that the AI outperformed our endoscopists, further confirming the robust differentiating power of the trained AI image classifier.

One of the important applications of this AI technology is determining whether a large colonic lesion should be treated by endoscopic resection or surgery, without the need for a biopsy. Because expertise is usually required to select the appropriate colonic lesion for ESD, many community endoscopists still rely on biopsy to guide their management. However, biopsy may miss the most advanced part of the lesion and multiple biopsies will result in significant submucosal fibrosis, which will make subsequent ESD more difficult with longer procedure time and higher risk of potential complications 12 . Moreover, AI technology can enable a quick and accurate optical diagnosis. It can potentially shorten procedure time for standard image-enhanced endoscopy as a learning curve is always needed for less-experienced endoscopists to differentiate between vascular and pit patterns on NBI 13 . The AI technology can eliminate the need for this kind of training without affecting the clinical decision. Moreover, there is no interobserver variation with an AI image classifier and a confidence or probability level could be provided for each lesion to facilitate better communication of the treatment plan to the patient.

There are several limitations of this study. First, it was retrospective, which could be subjected to selection bias, particularly on the external independent testing set. Second, the ROI was selected by a single experienced endoscopist (TKLL), which may hinder its application in a real-time situation when a more experienced endoscopist is not available. Third, because of the retrospective nature of the endoscopy image database, the image-pathological correlation could not take into consideration whether the lesions were similarly and profoundly examined endoscopically or the difficulty of endoscopic resection of particular lesions.

To implement this application of AI in clinical practice, an interface software between the AI image classifier and the current endoscopic system is needed. Although the endocytoscope has also been shown to be effective in prediction of histology of colonic lesions 14 , that system is expensive and is not readily available in most centers. Moreover, certain skills are still required to operate the endocytoscope and the AI system using standard endoscopy would be a simpler option for reliable prediction of histology. Besides the technical issue, any new technology needs to be approved by the regulatory authority before any real clinical use. Therefore, further studies about the AI’s accuracy, safety, and cost-effectiveness analysis in prospective, real-time fashion are needed to convince the involved parties that approval of this technology is warranted.

Conclusion

In summary, this study showed that the trained AI image classifier is highly accurate in predicting feasibility of curative endoscopic resection of large colonic lesions. While development of AI using CNN is excellent for pattern recognition and image classification 15 , it is anticipated that the prediction performance might outperform an expert human endoscopist in the near future 16 17 18 .