Katherine Elfer1,2, Sarah Dudgeon3,4, Victor Garcia1, Kim Blenman5,6, Evangelos Hytopoulos7, Si Wen1, Xiaoxian Li8, Amy Ly9, Bruce Werness10,11, Manasi S Sheth12, Mohamed Amgad13, Rajarsi Gupta14, Joel Saltz14,15, Matthew G Hanna16, Anna Ehinger17, Dieter Peeters18,19, Roberto Salgado20,21, Brandon D Gallas1. 1. United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics & Software Reliability, Silver Spring, Maryland, United States. 2. National Institutes of Health, National Cancer Institute, Division of Cancer Prevention, Cancer Prevention Fellowship Program, Bethesda, Maryland, United States. 3. Yale University Computational Biology and Bioinformatics, New Haven, Connecticut, United States. 4. Yale New Haven Hospital, Center for Outcomes Research and Evaluation, New Haven, Connecticut, United States. 5. School of Medicine, Yale Cancer Center, Department of Internal Medicine, Section of Medical Oncology, New Haven, Connecticut, United States. 6. Yale University, School of Engineering and Applied Science, Department of Computer Science, New Haven, Connecticut, United States. 7. iRhythm Technologies Inc., San Francisco, California, United States. 8. Emory University School of Medicine, Department of Pathology and Laboratory Medicine, Atlanta, Georgia, United States. 9. Massachusetts General Hospital, Boston, Massachusetts, United States. 10. Inova Health System Department of Pathology, Falls Church, Virginia, United States. 11. Arrive Bio LLC, San Francisco, California, United States. 12. United States Food and Drug Administration (FDA), Center for Devices and Radiologic Health, Office of Product Evaluation and Quality, Office of Clinical Evidence and Analysis, Division of Biostatistics, White Oak, Maryland, United States. 13. Northwestern University Feinberg School of Medicine, Department of Pathology, Chicago, Illinois, United States. 14. SUNY Stony Brook Medicine, Department of Biomedical Informatics, Stony Brook, New York, United States. 15. SUNY Stony Brook Medicine, Department of Pathology, Stony Brook, New York, United States. 16. Memorial Sloan Kettering Cancer Center, New York, New York, United States. 17. Lund University, Laboratory Medicine, Region Skåne, Department of Genetics and Pathology, Lund, Sweden. 18. Sint-Maarten Hospital, Department of Pathology, Mechelen, Belgium. 19. University of Antwerp, Department of Biomedical Sciences, Antwerp, Belgium. 20. Peter Mac Callum Cancer Centre, Division of Research, Melbourne, Australia. 21. GZA-ZNA Hospitals, Department of Pathology, Antwerp, Belgium.
Abstract
Purpose: Validation of artificial intelligence (AI) algorithms in digital pathology with a reference standard is necessary before widespread clinical use, but few examples focus on creating a reference standard based on pathologist annotations. This work assesses the results of a pilot study that collects density estimates of stromal tumor-infiltrating lymphocytes (sTILs) in breast cancer biopsy specimens. This work will inform the creation of a validation dataset for the evaluation of AI algorithms fit for a regulatory purpose. Approach: Collaborators and crowdsourced pathologists contributed glass slides, digital images, and annotations. Here, "annotations" refer to any marks, segmentations, measurements, or labels a pathologist adds to a report, image, region of interest (ROI), or biological feature. Pathologists estimated sTILs density in 640 ROIs from hematoxylin and eosin stained slides of 64 patients via two modalities: an optical light microscope and two digital image viewing platforms. Results: The pilot study generated 7373 sTILs density estimates from 29 pathologists. Analysis of annotations found the variability of density estimates per ROI increases with the mean; the root mean square differences were 4.46, 14.25, and 26.25 as the mean density ranged from 0% to 10%, 11% to 40%, and 41% to 100%, respectively. The pilot study informs three areas of improvement for future work: technical workflows, annotation platforms, and agreement analysis methods. Upgrades to the workflows and platforms will improve operability and increase annotation speed and consistency. Conclusions: Exploratory data analysis demonstrates the need to develop new statistical approaches for agreement. The pilot study dataset and analysis methods are publicly available to allow community feedback. The development and results of the validation dataset will be publicly available to serve as an instructive tool that can be replicated by developers and researchers.
Purpose: Validation of artificial intelligence (AI) algorithms in digital pathology with a reference standard is necessary before widespread clinical use, but few examples focus on creating a reference standard based on pathologist annotations. This work assesses the results of a pilot study that collects density estimates of stromal tumor-infiltrating lymphocytes (sTILs) in breast cancer biopsy specimens. This work will inform the creation of a validation dataset for the evaluation of AI algorithms fit for a regulatory purpose. Approach: Collaborators and crowdsourced pathologists contributed glass slides, digital images, and annotations. Here, "annotations" refer to any marks, segmentations, measurements, or labels a pathologist adds to a report, image, region of interest (ROI), or biological feature. Pathologists estimated sTILs density in 640 ROIs from hematoxylin and eosin stained slides of 64 patients via two modalities: an optical light microscope and two digital image viewing platforms. Results: The pilot study generated 7373 sTILs density estimates from 29 pathologists. Analysis of annotations found the variability of density estimates per ROI increases with the mean; the root mean square differences were 4.46, 14.25, and 26.25 as the mean density ranged from 0% to 10%, 11% to 40%, and 41% to 100%, respectively. The pilot study informs three areas of improvement for future work: technical workflows, annotation platforms, and agreement analysis methods. Upgrades to the workflows and platforms will improve operability and increase annotation speed and consistency. Conclusions: Exploratory data analysis demonstrates the need to develop new statistical approaches for agreement. The pilot study dataset and analysis methods are publicly available to allow community feedback. The development and results of the validation dataset will be publicly available to serve as an instructive tool that can be replicated by developers and researchers.
Authors: Brandon D Gallas; Marios A Gavrielides; Catherine M Conway; Adam Ivansky; Tyler C Keay; Wei-Chung Cheng; Jason Hipp; Stephen M Hewitt Journal: J Med Imaging (Bellingham) Date: 2014-11-12
Authors: Robyn D Gartrell; Douglas K Marks; Thomas D Hart; Gen Li; Danielle R Davari; Alan Wu; Zoë Blake; Yan Lu; Kayleigh N Askin; Anthea Monod; Camden L Esancy; Edward C Stack; Dan Tong Jia; Paul M Armenta; Yichun Fu; Daisuke Izaki; Bret Taback; Raul Rabadan; Howard L Kaufman; Charles G Drake; Basil A Horst; Yvonne M Saenger Journal: Cancer Immunol Res Date: 2018-02-21 Impact factor: 11.151
Authors: Joel Saltz; Ashish Sharma; Ganesh Iyer; Erich Bremer; Feiqiao Wang; Alina Jasniewski; Tammy DiPrima; Jonas S Almeida; Yi Gao; Tianhao Zhao; Mary Saltz; Tahsin Kurc Journal: Cancer Res Date: 2017-11-01 Impact factor: 12.701
Authors: Kimberly H Allison; Lisa M Reisch; Patricia A Carney; Donald L Weaver; Stuart J Schnitt; Frances P O'Malley; Berta M Geller; Joann G Elmore Journal: Histopathology Date: 2014-04-02 Impact factor: 5.087
Authors: Wei Chang Colin Tan; Sanjna Nilesh Nerurkar; Hai Yun Cai; Harry Ho Man Ng; Duoduo Wu; Yu Ting Felicia Wee; Jeffrey Chun Tatt Lim; Joe Yeong; Tony Kiat Hon Lim Journal: Cancer Commun (Lond) Date: 2020-04-17
Authors: Serdar Altinay; Laurent Arnould; Maschenka Balkenhol; Glenn Broeckx; Octavio Burguès; Cecile Colpaert; Franceska Dedeurwaerdere; Benjamin Dessauvagie; Valérie Duwel; Giuseppe Floris; Stephen Fox; Clara Gerosa; Delfyne Hastir; Shabnam Jaffer; Eline Kurpershoek; Magali Lacroix-Triki; Andoni Laka; Kathleen Lambein; Gaëtan Marie MacGrogan; Caterina Marchiò; Maria-Dolores Martin Martinez; Sharon Nofech-Mozes; Dieter Peeters; Alberto Ravarino; Emily Reisenbichler; Erika Resetkova; Souzan Sanati; Anne-Marie Schelfhout; Vera Schelfhout; Abeer Shaaban; Renata Sinke; Claudia M Stanciu-Pop; Carolien H M van Deurzen; Koen K Van de Vijver; Anne-Sophie Van Rompuy; Anne Vincent-Salomon; Hannah Y Wen; Serena Wong; Mieke R Van Bockstal; Aline François; Caroline Bouzin; Christine Galant Journal: Mod Pathol Date: 2021-07-03 Impact factor: 7.842