Leonardo Frazzoni1, Giulio Antonelli2,3, Julia Arribas4, Diogo Libanio4, Alanna Ebigbo5, Fons van der Sommen6, Albert Jeroen de Groof7, Hiromu Fukuda8, Masayasu Ohmori8, Ryu Ishihara8, Lianlian Wu9, Honggang Yu9, Yuichi Mori10,11, Alessandro Repici12,13, Jacques J G H M Bergman7, Prateek Sharma14, Helmut Messmann5, Cesare Hassan2, Lorenzo Fuccio1, Mário Dinis-Ribeiro15. 1. Department of Medical and Surgical Sciences (DIMEC), University of Bologna, S. Orsola-Malpighi Hospital, Bologna, Italy. 2. Gastroenterology Unit, Nuovo Regina Margherita Hospital, Rome, Italy. 3. Department of Translational and Precision Medicine, "Sapienza" University of Rome, Rome, Italy. 4. CIDES/CINTESIS, Faculty of Medicine, University of Porto, Porto, Portugal. 5. III Medizinische Klinik, Universitatsklinikum Augsburg, Augsburg, Germany. 6. Department of Electrical Engineering, VCA group, Eindhoven University of Technology, Eindhoven, The Netherlands. 7. Department of Gastroenterology and Hepatology, Amsterdam UMC, Amsterdam, The Netherlands. 8. Department of Gastrointestinal Oncology, Osaka International Cancer Institute, Osaka, Japan. 9. Department of Gastroenterology, Renmin Hospital of Wuhan University, Institute for Gastroenterology and Hepatology, Wuhan University, Wuhan, China. 10. Clinical Effectiveness Research Group, University of Oslo, Oslo, Norway. 11. Digestive Disease Center, Showa University Northern Yokohama Hospital, Yokohama, Japan. 12. Digestive Endoscopy Unit, Humanitas Research Hospital - IRCCS, Milan, Italy. 13. Department of Biomedical Sciences, Humanitas University, Pieve Emanuele, Milan, Italy. 14. Department of Gastroenterology and Hepatology, University of Kansas Medical Center, Kansas City, Kansas, USA. 15. Gastroenterology Department, Portuguese Oncology Institute of Porto, Porto, Portugal.
Abstract
BACKGROUND: Estimates on miss rates for upper gastrointestinal neoplasia (UGIN) rely on registry data or old studies. Quality assurance programs for upper GI endoscopy are not fully established owing to the lack of infrastructure to measure endoscopists' competence. We aimed to assess endoscopists' accuracy for the recognition of UGIN exploiting the framework of artificial intelligence (AI) validation studies. METHODS: Literature searches of databases (PubMed/MEDLINE, EMBASE, Scopus) up to August 2020 were performed to identify articles evaluating the accuracy of individual endoscopists for the recognition of UGIN within studies validating AI against a histologically verified expert-annotated ground-truth. The main outcomes were endoscopists' pooled sensitivity, specificity, positive and negative predictive value (PPV/NPV), and area under the curve (AUC) for all UGIN, for esophageal squamous cell neoplasia (ESCN), Barrett esophagus-related neoplasia (BERN), and gastric adenocarcinoma (GAC). RESULTS: Seven studies (2 ESCN, 3 BERN, 1 GAC, 1 UGIN overall) with 122 endoscopists were included. The pooled endoscopists' sensitivity and specificity for UGIN were 82 % (95 % confidence interval [CI] 80 %-84 %) and 79 % (95 %CI 76 %-81 %), respectively. Endoscopists' accuracy was higher for GAC detection (AUC 0.95 [95 %CI 0.93-0.98]) than for ESCN (AUC 0.90 [95 %CI 0.88-0.92]) and BERN detection (AUC 0.86 [95 %CI 0.84-0.88]). Sensitivity was higher for Eastern vs. Western endoscopists (87 % [95 %CI 84 %-89 %] vs. 75 % [95 %CI 72 %-78 %]), and for expert vs. non-expert endoscopists (85 % [95 %CI 83 %-87 %] vs. 71 % [95 %CI 67 %-75 %]). CONCLUSION: We show suboptimal accuracy of endoscopists for the recognition of UGIN even within a framework that included a higher prevalence and disease awareness. Future AI validation studies represent a framework to assess endoscopist competence. Thieme. All rights reserved.
BACKGROUND: Estimates on miss rates for upper gastrointestinal neoplasia (UGIN) rely on registry data or old studies. Quality assurance programs for upper GI endoscopy are not fully established owing to the lack of infrastructure to measure endoscopists' competence. We aimed to assess endoscopists' accuracy for the recognition of UGIN exploiting the framework of artificial intelligence (AI) validation studies. METHODS: Literature searches of databases (PubMed/MEDLINE, EMBASE, Scopus) up to August 2020 were performed to identify articles evaluating the accuracy of individual endoscopists for the recognition of UGIN within studies validating AI against a histologically verified expert-annotated ground-truth. The main outcomes were endoscopists' pooled sensitivity, specificity, positive and negative predictive value (PPV/NPV), and area under the curve (AUC) for all UGIN, for esophageal squamous cell neoplasia (ESCN), Barrett esophagus-related neoplasia (BERN), and gastric adenocarcinoma (GAC). RESULTS: Seven studies (2 ESCN, 3 BERN, 1 GAC, 1 UGIN overall) with 122 endoscopists were included. The pooled endoscopists' sensitivity and specificity for UGIN were 82 % (95 % confidence interval [CI] 80 %-84 %) and 79 % (95 %CI 76 %-81 %), respectively. Endoscopists' accuracy was higher for GAC detection (AUC 0.95 [95 %CI 0.93-0.98]) than for ESCN (AUC 0.90 [95 %CI 0.88-0.92]) and BERN detection (AUC 0.86 [95 %CI 0.84-0.88]). Sensitivity was higher for Eastern vs. Western endoscopists (87 % [95 %CI 84 %-89 %] vs. 75 % [95 %CI 72 %-78 %]), and for expert vs. non-expert endoscopists (85 % [95 %CI 83 %-87 %] vs. 71 % [95 %CI 67 %-75 %]). CONCLUSION: We show suboptimal accuracy of endoscopists for the recognition of UGIN even within a framework that included a higher prevalence and disease awareness. Future AI validation studies represent a framework to assess endoscopist competence. Thieme. All rights reserved.