Endoscopists performance in optical diagnosis of colorectal polyps in artificial intelligence studies.

Widespread adoption of optical diagnosis of colorectal neoplasia is prevented by suboptimal endoscopist performance and lack of standardized training and competence evaluation. We aimed to assess diagnostic accuracy of endoscopists in optical diagnosis of colorectal neoplasia in the framework of artificial intelligence (AI) validation studies. Literature searches of databases (PubMed/MEDLINE, EMBASE, Scopus) up to April 2022 were performed to identify articles evaluating accuracy of individual endoscopists in performing optical diagnosis of colorectal neoplasia within studies validating AI against a histologically verified ground-truth. The main outcomes were endoscopists' pooled sensitivity, specificity, positive and negative predictive value (PPV/NPV), positive and negative likelihood ratio (LR) and area under the curve (AUC for sROC) for predicting adenomas versus non-adenomas. Six studies with 67 endoscopists and 2085 (IQR: 115-243,5) patients were evaluated. Pooled sensitivity and specificity for adenomatous histology was respectively 84.5% (95% CI 80.3%-88%) and 83% (95% CI 79.6%-85.9%), corresponding to a PPV, NPV, LR+, LR- of 89.5% (95% CI 87.1%-91.5%), 75.7% (95% CI 70.1%-80.7%), 5 (95% CI 3.9%-6.2%) and 0.19 (95% CI 0.14%-0.25%). The AUC was 0.82 (CI 0.76-0.90). Expert endoscopists showed a higher sensitivity than non-experts (90.5%, [95% CI 87.6%-92.7%] vs. 75.5%, [95% CI 66.5%-82.7%], p < 0.001), and Eastern endoscopists showed a higher sensitivity than Western (85%, [95% CI 80.5%-88.6%] vs. 75.8%, [95% CI 70.2%-80.6%]). Quality was graded high for 3 studies and low for 3 studies. We show that human accuracy for diagnosis of colorectal neoplasia in the setting of AI studies is suboptimal. Educational interventions could benefit by AI validation settings which seem a feasible framework for competence assessment.

INTRODUCTION

A substantial proportion of the cost of population‐based Colorectal Cancer (CRC) screening program is due to removal and subsequent pathology assessment of diminutive colorectal polyps that represent more than 80% of all the detectable lesions. , , , Optical diagnosis has been shown to be able to in vivo predict histology of these diminutive lesions in expert centers, opening the way to cost‐saving strategies, namely the Leave‐in‐Situ for ≤5 mm rectosigmoid hyperplastic lesions, and Resect and Discard for all the others. ,

Disappointingly, implementation of these cost‐saving strategies has been hampered by suboptimal results in community‐based controlled trials, questioning on the actual accuracy of endoscopists in the optical diagnosis of diminutive lesions. However, a direct assessment of the accuracy of individual endoscopists in optical diagnosis is limited to few studies, leaving uncertainty on the actual need of educational interventions as well as on the best approach.

Artificial Intelligence (AI) has been claimed to predict histology of diminutive polyps in real‐time endoscopy. For this reason, several AI‐algorithms have been tested in standalone performance studies against a ground‐truth generally represented by pathologically verified polyps selected by expert endoscopists, resulting in an overall accuracy of over 90%. In order to better define it, AI performance has been generally benchmarked against multiple endoscopists with different degrees of competence which were administered the same sets of images/videos analyzed by AI. , This framework provides a unique modality of assessing the performance of human endoscopists when dealing with optical diagnosis, and could be used as testing ground for the application of PIVI criteria.

Aim of our study was to evaluate the accuracy of human endoscopists in performing optical diagnosis of colorectal polyps extracting their performances from studies on standalone performance of AI systems, as well as on possible associated factors. Such analysis could set the grounds for new modalities of training and competence evaluation in colorectal lesion evaluation.

METHODS

The methods of our analysis and inclusion criteria were based on Preferred Reporting Items for Systematic Reviews and MetaAnalyses (PRISMA) recommendations.

The PRISMA Checklist is available in Supporting Information S1.

Study registration

This study was registered on the PROSPERO international database (University of York Centre for Reviews, www.crd.york.ac.uk/prospero/). Number: 279321.

Inclusion and exclusion criteria

Only original full articles published in English have been considered for the study. Abstract, letter or review articles were excluded. All studies reporting the use of AI for characterization of colorectal adenoma compared to human characterization with histological confirmation as ground truth have been included.

Search strategy and data extraction

We performed a comprehensive literature search of two scientific databases (PubMed/Medline and Scopus) up to April 2022 to identify full articles evaluating the diagnostic accuracy of AI‐assisted colonoscopy for characterization of colorectal adenoma compared to “human control” of expert and non‐expert endoscopists. Electronic searches were supplemented by manual searches of references of included studies. Complete search strategy and search strings used are available in Supporting Information S2. Two authors (SP and GA) independently evaluated all titles and abstracts of the identified articles to exclude papers not strictly related to the aim of the study or meeting inclusion criteria. Remaining abstract and full text were further screened for eligibility. Finally, any disagreement was discussed and solved with senior authors. Data extraction from eligible study was performed using the following scheme (a blank example of our data extraction table is available in Supporting Information S3):

the total number of images/cases and the number of total positive images/cases (predicted as adenoma/non adenoma by endoscopists and confirmed by histology)

the numbers of images/cases classified as true positive (images/cases showing colorectal lesion predicted‐as‐adenoma by AI), true negative (images/cases showing non‐neoplastic mucosa without AI detection or lesions predicted as non‐neoplastic), false positive (FP, images/cases showing non‐neoplastic mucosa or lesions detected/predicted‐as‐neoplastic by AI) or false negative (images/cases showing a neoplastic lesion missed by AI or predicted as non‐neoplastic)

In addition, country of provenience, type of study, number of patients, characteristics of polyps detected were also considered. Corresponding authors were contacted for data extraction in case of missing information from published studies.

Study outcomes

Primary endpoint of the study was the pooled diagnostic endoscopists' accuracy for the characterization of colorectal adenoma in terms of sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV), likelihood ratio (LR+) and negative likelihood ratio (LR). The accuracy of endoscopists was defined as “hierarchical summary receiver‐operating characteristic (SROC) curve (area under the curve; AUC).”

Secondary outcomes were the diagnostic performance according to study design, endoscopist's level of expertise and country of provenience.

Quality of studies

The degree of bias was assessed using a modified version of the QUADAS (quality for assessment of diagnostic studies score) , score that was already used in previous publications. We include specific bias domains for diagnostic studies in AI. We divided in two main domains and respective subdomains, namely Training set bias (subdomains: Selection bias, Spectrum Bias and Operator bias) and Validation set bias (subdomains: Overfitting bias and Operator bias). For Overfitting bias we considered at low risk of bias papers explicitly describing the use of overfitting mitigation techniques as data augmentation, dropout, batch normalization, regularization, early stopping, and transfer learning from large datasets.

Statistical analysis

We computed summary estimates of sensitivity, specificity, LR+ and LR− of GI endoscopists on a “per‐endoscopist” basis, through the bivariate mixed‐effects regression model proposed by Reitsma et al ; 95% Confidence Intervals (CIs) for the diagnostic accuracy parameters were computed through the bivariate model, as well. Positive predictive value (PPV) and negative predictive value (NPV) were obtained for the pooled prevalence of lesions. Forest plots for sensitivity and specificity, and summary receiving operating characteristic (SROC) curve were drawn. Positive and negative likelihood ratios were applied to the pooled prevalence of the various types of UGI premalignant and malignant lesions (i.e. pre‐test probability), to derive the post‐test probability in case of a positive or negative test result; a Fagan's plot was derived, accordingly.

Heterogeneity was assessed through visual inspection of forest plots and SROC curve, and quantified by the between‐study standard deviation (SD) for logit‐transformed sensitivity and specificity. We assessed heterogeneity through sensitivity analyses based on subgroup meta‐analyses and bivariate meta‐regression models. Variables which might have influenced the diagnostic accuracy of GI endoscopists were defined a priori at two levels: (i) the “endoscopist” level, that is, the experience of the endoscopists participating to the included studies as dichotomized into expert and non‐expert according to study definitions; (ii) the “study” level, that is, study size, mono versus multicenter studies, country, number of images provided, percentage of lesions in the right colon, percentage of adenomas, and quality of studies. All the analyses were performed with the package mada for R.

RESULTS

Search data

The search strategy yielded a total of 1267 studies. Once duplicates were removed, a total of 987 studies were screened by analysis of title and abstract and 959 studies were removed because not related to the study topic or not meeting inclusion criteria. Then, 28 studies were entirely evaluated for eligibility and among them 20 were excluded, all for the absence of histological confirmation and two of them were excluded since accuracy data of performance for each endoscopy was not available in the full text. Finally, six articles , , , , , were included in the statistical analysis (Figure 1—Study Flowchart).

FIGURE 1

Flow‐chart of included studies

Study details

Among 6 studies included (Table 1), 4 had a single center design, , , , while 2 had a multicenter design , (6). Regarding geographical area, 4 studies were performed in Eastern centers , , , and two in Western centers. , Olympus endoscopes with narrow‐band imaging (NBI) filter were used by five out of six studies and two of them added endocitoscopy (EC). All studies were based on characterization of adenoma/non‐adenoma lesions and had histopathological evaluation as standard reference. Median number of included patients was 208.5 (IQR: 115–243.5). All studies reported the number of total images used for adenoma/non adenoma characterization, accounting for a total of 1368 (median: 209; IQR: 108.5–296). Regarding polyp details, all studies considered colorectal polyps <10 mm (for one study the data was missing). Polyp morphology was protruded type (Paris type Is, Isp or Ip) and slightly elevated type (Paris type IIa) in five studies, while an Eastern study included also slightly depressed type polyps (Paris type IIc). Complete characteristics of polyps are reported in Table 2.

TABLE 1

Details of included studies

First Author, Year	Design	Country	Patients (n)	Consecutive Y/N	Images (n)	Endoscopists (n)	Expert (n)	Non expert (n)	AI type	Imaging type	Setting
Chen, 2018	U	E	193	Y	284	6	2	4	CAOB	HDWL/magnifying NBI	Experimental images only
Renner, 2018	U	W	250	Y	100	2	2	0	DNN‐CAD	HDWL/NBI	Experimental images only
Mori et al., 2018	U	E	320	Y	450	4	2	2	CAD system	EC‐NBI	Real time images and videos
Kudo, 2020	M	E	89	N	100	30	10	20	EndoBRAIN	WL/EC‐NBI	Experimental images only
Jin, 2020	U	E	224	N	300	22	15	7	CNN system	NBI/near‐focus	Experimental images only
Weigt, 2021	M	W	80	Y	134	3	3	0	CAD‐EYE	WL/LCI/BLI	Experimental images and videos

Abbreviations: BLI, Blue Light Imaging; CAD, Computer Aided Detection; CAOB, computer‐assisted optical biopsy; CNN, convolutional neural network; DNN, deep neural network; E, Eastern; EC, endocitoscopy; HDWL, high definition white light; LCI, Linked Color Imaging; M, multicentric; NBI, narrow band imaging; U, unicentric; W, Western.

TABLE 2

Polyps characteristics

Polyps characteristics
Study	Size (mm)	Shape (Paris class)	Location % (right/left colon)
Chen, 2018 18	<5	Is – Isp – IIa	34.8/65.2
Renner, 2018 19	<5	Is – Ip – IIa	51/49
Mori, 2018 22	<5	Is – Ip – IIa – IIc	40.4/59.6
Kudo, 2020 21	<10	Is – Isp – IIa	38/62
Jin, 2020 20	<5	Is – Isp – IIa	54.3/45.7
Weigt, 2021 23	‐	Is – IIa	44/35.5 (20.9 missing)

Endoscopists characteristics

Overall, 67 endoscopists from 6 studies were included in the analysis. Of these, 5/67 (7.46%) endoscopists came from Western centers and 62/67 (92.55%) from Eastern centers. Expert endoscopists were 34/67 (50.75%) while 33/67 (49.25%) were considered non‐experts and were all located in Eastern countries.

Primary outcome

The pooled prevalence of colorectal adenoma among all images shown to endoscopists was 8576/13705 (63.2%; 95% CI 62.1%–64.4%). Overall, 67 endoscopists from 6 studies had a pooled sensitivity and specificity of 84.5% (95% CI 80.3%–88%) and 83% (95% CI 79.6%–85.9%), respectively for adenomatous histology. In addition, PPV and NPV were 89.5% (95% CI 87.1%–91.5%) and 75.7% (95% CI 70.1%–80.7%), respectively, corresponding to positive and negative likelihood ratio (LR+/LR) of 5 (95% CI 3.9%–6.2%) and 0.19 (95% CI 0.14%–0.25%), with AUC of 0.82 (95% CI 0.76–0.90). Relative SROC curve is available in Figure 2.

FIGURE 2

Summary receiver‐operating characteristic curve

Secondary outcomes

Experienced endoscopists had a significantly higher sensitivity (90.5%, [95% CI 87.6%–92.7%] vs. 75.5%, [95% CI 66.5%–82.7%], p < 0.001) and specificity than non‐expert endoscopists (84.8% [95% CI 82.3%–87.8%] vs. 81.4% [95% CI 75.1%–86.4%], p < 0.84), corresponding to a NPV of 84% (95% CI 79.4%–87.6%) versus 66.1% (95% CI 64.9%–80.5%). The forest plots for sensitivity and specificity can be found in Figure 3.

FIGURE 3

Forest plots for sensitivity and specificity by study

Eastern endoscopists showed higher sensitivity than Western endoscopists (85%, [95% CI 80.5%–88.6%] vs. 75.8%, [95% CI 70.2%–80.6%]) and higher specificity (83.6%, [95% CI 80%–86.6%] vs. 76.7%, [95% CI 65.9%–84.8%]). The forest plots is available in Figure 4. Moreover, sensitivity is significantly higher for endoscopists of single center design studies (90.2% [95% CI 86.9%–92.8%]) than multicenter studies (76.2% [95% CI 76.2%–88.8%], p < 0.001), while on the contrary, endoscopists of multicenter studies seem to have a better rate of specificity (88.8% [95% CI 84.5%–92.1%]) than single center studies (78.5% [95% CI 73.7%–82.7%], p = 0.001). Details in Table 3.

FIGURE 4

Forest plots for sensitivity and specificity by experience and country

TABLE 3

Subgroup meta‐analyses for summary diagnostic accuracy measures of endoscopists for adenoma characterization at colonoscopy, according to study variables

Study variable (n of endoscopists)	Sensitivity (95% CI)	p‐value for sensitivity	Specificity (95% CI)	p‐value for specificity
Endoscopists' experience
Experienced (n = 34)	90.5 (87.6–92.7)	<0.001	84.8 (82.3–87.8)	0.084
Inexperienced (n = 33)	75.5 (66.5–82.7)		81.4 (75.1–86.4)
Country
Eastern (n = 62)	85 (80.5–88.6)	0.436	83.6 (80–86.6)	0.28
Western (n = 5)	75.8 (70.2–80.6)		76.7 (65.9–84.8)
Study design
Monocenter (n = 34)	90.2 (86.9–92.8)	<0.001	78.5 (73.7–82.7)	0.001
Multicenter (n = 33)	76.2 (67.7–83.1)		88.8 (84.5–92.1)
Study quality
High (n = 56)	83.6 (78.6–87.6)	0.359	84.6 (80.9–87.8)	0.051
Low (n = 11)	89 (80.8–93.9)		75.6 (70.1–80.4)

Additional analysis

Meta‐regression analysis for other studies variables showed a positive relation between the number of images and sensitivity (p = 0.002) and a negative relation with specificity (p = 0.032). Also the rate of right colon lesions had a significant impact on sensitivity (p < 0.001). Details available in Table 4.

TABLE 4

Meta‐regression analysis for continuous moderators

Study variable	Coefficient for sensitivity (95% CI)	p‐value for impact on sensitivity	Coefficient for 1‐specificity (95% CI)	p‐value for impact on specificity
Number of images	0.004 (0.001–0.006)	0.002	0.002 (0.001–0.004)	0.032
Percentage of right colon lesions	−0.081 (−0.126–−0.037)	<0.001	0.086 (−1.084–1.256)	0.886
Relative frequency of adenomas	−5.310 (−11.429–0.809)	0.089	−1.305 (−6.053–3.443)	0.590

Study quality assessment according to the modified QUADAS score is available in Table 5. In detail, 3 out of 6 studies , , were considered of High quality, and 3 studies , , was considered of Low quality. There was a tendency to spectrum bias in the included studies, as often the images were only selected among high quality images or best framing of the polyp. Meta regression analysis including study quality is available in Table 3.

TABLE 5

Quality assessment

	Reference standard/Training set			Index test/Validation set
Study	Selection bias	Spectrum bias	Operator bias	Overfitting bias	Operator bias	Overall quality
Chen, 2018						Low
Renner, 2018						Low
Mori et al., 2018						High
Kudo, 2020						High
Jin, 2020						High
Weigt, 2021						Low

Note: low risk of bias high risk of bias.

DISCUSSION

By exploiting the artificial setting represented by AI validation studies, we measured a suboptimal performance of human endoscopists in the optical diagnosis of diminutive to small polyps that appears to be not compatible with the implementation of clinical strategies based on a human‐alone evaluation. Even though sensitivity and specificity overall would be slightly over the 80% threshold recently proposed by ESGE for the resect‐and‐discard strategy (24), the 76% NPV for adenomatous histology is disappointingly far from the 90% cut‐off universally recognized as the minimum cut‐off to implement the leave‐in‐situ strategy. , , This result is not fully unexpected since previous literature , has shown how, especially in the community setting, endoscopists fail to reach required thresholds. However, we show for the first time how the development framework of CADx systems may be an optimal platform to assess endoscopist competence.

The main clinical relevance of our study is the intimate association between the level of competence as defined by the degree of experience and the accuracy of individual endoscopists. Remarkably, the much higher sensitivity of experts versus non‐experts—90.5% versus 75.5%—indicates a much higher risk of false‐negative cases for adenomatous histology that is adenomas misinterpreted as hyperplastic polyps by non‐expert endoscopists. This is by far the worst error that can come from an inaccurate in vivo prediction as potentially condemning a high‐risk patient with multiple adenomas (i.e., ≥3 low‐risk adenomas) who needs an intensive post‐polypectomy surveillance to a low‐risk category without the necessary endoscopic surveillance. In addition, we showed that an Eastern location of the endoscopists is also associated with a higher accuracy in optical diagnosis, irrespectively of the level of experience. This shows that the training approach that is much more meticulous and image‐based in Eastern as compared to Western school is critical in the development of adequate skills in polyp characterization. Thus, a dedicated image‐based training is needed for Western endoscopists, and in this regard the artificial setting adopted in our pooled studies could be at least a good benchmarking when testing the outcome of such educational interventions.

Our study indirectly supports the validity of community‐based studies on optical diagnosis of diminutive polyps showing a suboptimal performance not matching the required standards. Indeed, recent meta‐analysis on training modalities for optical diagnosis has shown pre‐training accuracy levels as low as 68.1%, as well as an unsatisfactory post‐training performance ranging from 77.1% to 81.6%. Such results are likely to be the direct consequence of the low sensitivity we measured rather than related with the clinical setting where such accuracy was tested, that is, distraction related with real‐life endoscopy, blurred or out‐of‐focus images, and difficult polyp position.

The strength of our study is to show that an artificial setting, that is, exposure of multiple endoscopists against images of histologically‐verified lesions, may be suitable to assess the skills of individual endoscopists in polyp characterization, as much as in benchmarking them against experts or standalone performance of Artificial Intelligence algorithms. In this regard, a recent meta‐analysis comprising 7680 images of colorectal polyps from 18 studies showed an accuracy (AUC) of AI of 96% (95% CI 0.95–0.98), corresponding to a sensitivity of 92.3% (95% CI 88.8%–94.9%) and a specificity of 89.8% (95% CI 85.3%–93.0%). When compared with our pooled estimate of 84% for endoscopist‐based sensitivity, this would suggest a relevant role for AI in assisting human endoscopists for polyp characterization. Secondly, all studies included diminutive to small polyps that is exactly what is required for the proposed cost‐saving resect‐and‐discard and leave‐in‐situ strategies.

The main limitations of our study is the per‐polyp rather than per‐patient analysis due to the image‐oriented rather than patient‐oriented collection of cases. However, a high per‐polyp accuracy should be the base rather than the consequence of a successful clinically‐oriented strategy rather than vice versa, and the cost‐saving strategies proposed by the PIVI document are indeed polyp‐ rather than patient‐based. Secondly, the actual number of western endoscopists was low, prompting the need for additional data. Third, our quality assessment found a possible spectrum‐related bias: indeed, endoscopists were shown images or video frames of lesions specifically selected for the purpose of characterization, which may be an ideal setting. Further, the prevalence of the disease (i.e. adenomas) may not represent clinical practice. Nevertheless, sensitivity and specificity are independent from the prevalence of the disease, therefore such estimates have external validity. Fourth, we cannot fully rule out an underperformance of benchmarking endoscopists leading to investigator bias. However, it must also be noted that benchmarking human endoscopists are ofthen not involved in the study conduction in the first place. Indeed, whether or not benchmarking is undergone by endoscopists from different centers and not involved in the data acquisition and annotation is a major quality indicator of a pre‐clinical AI paper. , Fifth, current CADx systems do not account for sessile serrated lesions as adenomatous, leading to a possible partial reduction of their accuracy. However, it must be noted that the primary aims of these systems have been first of all to implement cost saving strategies for diminutive polyps. This limits the impact of serrated lesions as their prevalence in the RS tract is negligible and all serrated lesions >5 mm of the whole colon are to be in any case resected and sent to pathology. Last, we provided diagnostic accuracy for adenomatous lesions irrespective of colonic site, therefore the inference on leave‐in‐situ strategy may be biased. However, although we could not separately assess diagnostic accuracy for rectosigmoid lesions, we performed a metaregression analysis showing that sensitivity tended to reduce when a higher proportion of lesions in the proximal colon was shown to endoscopists. This is in line with current recommendations suggesting to limit the leave‐in‐situ strategy to rectosigmoid lesions.

In conclusion, we show a disappointingly low accuracy in optical diagnosis of diminutive to small polyps when extracting them from the artificial setting of AI standalone performance studies. The exploitation of the AI development framework for endoscopist competence assessment is feasible and effective.

AUTHOR CONTRIBUTIONS

Silvia Pecere, Giulio Antonelli, Cesare Hassan: conception and design; Silvia Pecere, Giulio Antonelli, Yuichi Mori, Cesare Hassan: data extraction and interpretation; Lorenzo Fuccio, Leonardo Frazzoni: statistical analysis; Silvia Pecere, Giulio Antonelli, Cesare Hassan, Leonardo Frazzoni: drafting of the article; Raf Bisschops, Mario Dinis‐Ribeiro, Helmut Messmann, Yuichi Mori, Federico Iacopini, Lucio Petruzziello, Eun Hyo Jin, Yutaka Saito, Masayoshi Yamada, Alessandro Repici, Prateek Sharma, Guido Costamagna, Cristiano Spada: data provision and/or critical revision of the article for important intellectual content. All authors read and approved the final version of the manuscript.

CONFLICT OF INTEREST

The authors declare no COI relevant to this paper.

Supporting Information S1

Click here for additional data file.

Supporting Information S2

Click here for additional data file.

Supporting Information S3

Click here for additional data file.