J Peter Campbell1, Michael F Chiang2, Jimmy S Chen3, Darius M Moshfeghi4, Eric Nudleman5, Paisan Ruambivoonsuk6, Hunter Cherwek7, Carol Y Cheung8, Praveer Singh9, Jayashree Kalpathy-Cramer9, Susan Ostmo3, Malvina Eydelman10, R V Paul Chan11, Antonio Capone12. 1. Casey Eye Institute, Department of Ophthalmology, Oregon Health & Science University, Portland, Oregon. Electronic address: campbelp@ohsu.edu. 2. National Eye Institute, National Institutes of Health, Bethesda, Maryland. 3. Casey Eye Institute, Department of Ophthalmology, Oregon Health & Science University, Portland, Oregon. 4. Byers Eye Institute, Horngren Family Vitreoretinal Center, Department of Ophthalmology, Stanford University, Palo Alto, California. 5. Department of Ophthalmology, University of California, San Diego, California. 6. Department of Ophthalmology, Rajavithi Hospital, Bangkok, Thailand. 7. Orbis International, New York, New York. 8. Department of Ophthalmology and Visual Sciences, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong, China. 9. Department of Radiology, MGH/Harvard Medical School, Charlestown, Massachusetts; Massachusetts General Hospital & Brigham and Women's Hospital Center for Clinical Data Science, Boston, Massachusetts. 10. Center for Devices and Radiological Health, US Food and Drug Administration, Silver Spring, Maryland. 11. Department of Ophthalmology and Visual Sciences, University of Illinois at Chicago, Chicago, Illinois. 12. Associated Retinal Consultants, Oakland University William Beaumont School of Medicine, Royal Oak, Michigan.
Abstract
PURPOSE: To validate a vascular severity score as an appropriate output for artificial intelligence (AI) Software as a Medical Device (SaMD) for retinopathy of prematurity (ROP) through comparison with ordinal disease severity labels for stage and plus disease assigned by the International Classification of Retinopathy of Prematurity, Third Edition (ICROP3), committee. DESIGN: Validation study of an AI-based ROP vascular severity score. PARTICIPANTS: A total of 34 ROP experts from the ICROP3 committee. METHODS: Two separate datasets of 30 fundus photographs each for stage (0-5) and plus disease (plus, preplus, neither) were labeled by members of the ICROP3 committee using an open-source platform. Averaging these results produced a continuous label for plus (1-9) and stage (1-3) for each image. Experts were also asked to compare each image to each other in terms of relative severity for plus disease. Each image was also labeled with a vascular severity score from the Imaging and Informatics in ROP deep learning system, which was compared with each grader's diagnostic labels for correlation, as well as the ophthalmoscopic diagnosis of stage. MAIN OUTCOME MEASURES: Weighted kappa and Pearson correlation coefficients (CCs) were calculated between each pair of grader classification labels for stage and plus disease. The Elo algorithm was also used to convert pairwise comparisons for each expert into an ordered set of images from least to most severe. RESULTS: The mean weighted kappa and CC for all interobserver pairs for plus disease image comparison were 0.67 and 0.88, respectively. The vascular severity score was found to be highly correlated with both the average plus disease classification (CC = 0.90, P < 0.001) and the ophthalmoscopic diagnosis of stage (P < 0.001 by analysis of variance) among all experts. CONCLUSIONS: The ROP vascular severity score correlates well with the International Classification of Retinopathy of Prematurity committee member's labels for plus disease and stage, which had significant intergrader variability. Generation of a consensus for a validated scoring system for ROP SaMD can facilitate global innovation and regulatory authorization of these technologies.
PURPOSE: To validate a vascular severity score as an appropriate output for artificial intelligence (AI) Software as a Medical Device (SaMD) for retinopathy of prematurity (ROP) through comparison with ordinal disease severity labels for stage and plus disease assigned by the International Classification of Retinopathy of Prematurity, Third Edition (ICROP3), committee. DESIGN: Validation study of an AI-based ROP vascular severity score. PARTICIPANTS: A total of 34 ROP experts from the ICROP3 committee. METHODS: Two separate datasets of 30 fundus photographs each for stage (0-5) and plus disease (plus, preplus, neither) were labeled by members of the ICROP3 committee using an open-source platform. Averaging these results produced a continuous label for plus (1-9) and stage (1-3) for each image. Experts were also asked to compare each image to each other in terms of relative severity for plus disease. Each image was also labeled with a vascular severity score from the Imaging and Informatics in ROP deep learning system, which was compared with each grader's diagnostic labels for correlation, as well as the ophthalmoscopic diagnosis of stage. MAIN OUTCOME MEASURES: Weighted kappa and Pearson correlation coefficients (CCs) were calculated between each pair of grader classification labels for stage and plus disease. The Elo algorithm was also used to convert pairwise comparisons for each expert into an ordered set of images from least to most severe. RESULTS: The mean weighted kappa and CC for all interobserver pairs for plus disease image comparison were 0.67 and 0.88, respectively. The vascular severity score was found to be highly correlated with both the average plus disease classification (CC = 0.90, P < 0.001) and the ophthalmoscopic diagnosis of stage (P < 0.001 by analysis of variance) among all experts. CONCLUSIONS: The ROP vascular severity score correlates well with the International Classification of Retinopathy of Prematurity committee member's labels for plus disease and stage, which had significant intergrader variability. Generation of a consensus for a validated scoring system for ROP SaMD can facilitate global innovation and regulatory authorization of these technologies.
Authors: Michael C Ryan; Susan Ostmo; Karyn Jonas; Audina Berrocal; Kimberly Drenser; Jason Horowitz; Thomas C Lee; Charles Simmons; Maria-Ana Martinez-Castellanos; R V Paul Chan; Michael F Chiang Journal: AMIA Annu Symp Proc Date: 2014-11-14
Authors: James M Brown; J Peter Campbell; Andrew Beers; Ken Chang; Susan Ostmo; R V Paul Chan; Jennifer Dy; Deniz Erdogmus; Stratis Ioannidis; Jayashree Kalpathy-Cramer; Michael F Chiang Journal: JAMA Ophthalmol Date: 2018-07-01 Impact factor: 7.389
Authors: J Peter Campbell; Michael C Ryan; Emily Lore; Peng Tian; Susan Ostmo; Karyn Jonas; R V Paul Chan; Michael F Chiang Journal: Ophthalmology Date: 2016-05-27 Impact factor: 12.079
Authors: J Peter Campbell; Jayashree Kalpathy-Cramer; Deniz Erdogmus; Peng Tian; Dharanish Kedarisetti; Chace Moleta; James D Reynolds; Kelly Hutcheson; Michael J Shapiro; Michael X Repka; Philip Ferrone; Kimberly Drenser; Jason Horowitz; Kemal Sonmez; Ryan Swan; Susan Ostmo; Karyn E Jonas; R V Paul Chan; Michael F Chiang Journal: Ophthalmology Date: 2016-08-31 Impact factor: 12.079
Authors: Jayashree Kalpathy-Cramer; J Peter Campbell; Deniz Erdogmus; Peng Tian; Dharanish Kedarisetti; Chace Moleta; James D Reynolds; Kelly Hutcheson; Michael J Shapiro; Michael X Repka; Philip Ferrone; Kimberly Drenser; Jason Horowitz; Kemal Sonmez; Ryan Swan; Susan Ostmo; Karyn E Jonas; R V Paul Chan; Michael F Chiang Journal: Ophthalmology Date: 2016-08-24 Impact factor: 12.079
Authors: Mrinali Patel Gupta; R V Paul Chan; Rachelle Anzures; Susan Ostmo; Karyn Jonas; Michael F Chiang Journal: Am J Ophthalmol Date: 2015-12-15 Impact factor: 5.258
Authors: Michael F Chiang; Graham E Quinn; Alistair R Fielder; Susan R Ostmo; R V Paul Chan; Audina Berrocal; Gil Binenbaum; Michael Blair; J Peter Campbell; Antonio Capone; Yi Chen; Shuan Dai; Anna Ells; Brian W Fleck; William V Good; M Elizabeth Hartnett; Gerd Holmstrom; Shunji Kusaka; Andrés Kychenthal; Domenico Lepore; Birgit Lorenz; Maria Ana Martinez-Castellanos; Şengül Özdek; Dupe Ademola-Popoola; James D Reynolds; Parag K Shah; Michael Shapiro; Andreas Stahl; Cynthia Toth; Anand Vinekar; Linda Visser; David K Wallace; Wei-Chi Wu; Peiquan Zhao; Andrea Zin Journal: Ophthalmology Date: 2021-07-08 Impact factor: 12.079