Rory Sayres1, Ankur Taly1, Ehsan Rahimy2, Katy Blumer1, David Coz1, Naama Hammel1, Jonathan Krause1, Arunachalam Narayanaswamy1, Zahra Rastegar1, Derek Wu1, Shawn Xu3, Scott Barb4, Anthony Joseph5, Michael Shumski6, Jesse Smith7, Arjun B Sood8, Greg S Corrado1, Lily Peng9, Dale R Webster1. 1. Google Research, Google, LLC, Mountain View, California. 2. Department of Ophthalmology, Palo Alto Medical Foundation, Palo Alto, California. 3. Verily Life Sciences, South San Francisco, California. 4. Department of Ophthalmology, Emory University, Atlanta, Georgia. 5. Ophthalmic Consultants of Boston, Boston, Massachusetts. 6. Magruder Laser Vision, Orlando, Florida. 7. Denver Health Medical Center, Denver, Colorado; Department of Ophthalmology, University of Colorado School of Medicine, Aurora, Colorado. 8. Department of Ophthalmology, Massachusetts Eye and Ear Infirmary, Harvard Medical School, Boston, Massachusetts. 9. Google Research, Google, LLC, Mountain View, California. Electronic address: lhpeng@google.com.
Abstract
PURPOSE: To understand the impact of deep learning diabetic retinopathy (DR) algorithms on physician readers in computer-assisted settings. DESIGN: Evaluation of diagnostic technology. PARTICIPANTS: One thousand seven hundred ninety-six retinal fundus images from 1612 diabetic patients. METHODS: Ten ophthalmologists (5 general ophthalmologists, 4 retina specialists, 1 retina fellow) read images for DR severity based on the International Clinical Diabetic Retinopathy disease severity scale in each of 3 conditions: unassisted, grades only, or grades plus heatmap. Grades-only assistance comprised a histogram of DR predictions (grades) from a trained deep-learning model. For grades plus heatmap, we additionally showed explanatory heatmaps. MAIN OUTCOME MEASURES: For each experiment arm, we computed sensitivity and specificity of each reader and the algorithm for different levels of DR severity against an adjudicated reference standard. We also measured accuracy (exact 5-class level agreement and Cohen's quadratically weighted κ), reader-reported confidence (5-point Likert scale), and grading time. RESULTS: Readers graded more accurately with model assistance than without for the grades-only condition (P < 0.001). Grades plus heatmaps improved accuracy for patients with DR (P < 0.001), but reduced accuracy for patients without DR (P = 0.006). Both forms of assistance increased readers' sensitivity moderate-or-worse DR: unassisted: mean, 79.4% [95% confidence interval (CI), 72.3%-86.5%]; grades only: mean, 87.5% [95% CI, 85.1%-89.9%]; grades plus heatmap: mean, 88.7% [95% CI, 84.9%-92.5%] without a corresponding drop in specificity (unassisted: mean, 96.6% [95% CI, 95.9%-97.4%]; grades only: mean, 96.1% [95% CI, 95.5%-96.7%]; grades plus heatmap: mean, 95.5% [95% CI, 94.8%-96.1%]). Algorithmic assistance increased the accuracy of retina specialists above that of the unassisted reader or model alone; and increased grading confidence and grading time across all readers. For most cases, grades plus heatmap was only as effective as grades only. Over the course of the experiment, grading time decreased across all conditions, although most sharply for grades plus heatmap. CONCLUSIONS: Deep learning algorithms can improve the accuracy of, and confidence in, DR diagnosis in an assisted read setting. They also may increase grading time, although these effects may be ameliorated with experience.
PURPOSE: To understand the impact of deep learning diabetic retinopathy (DR) algorithms on physician readers in computer-assisted settings. DESIGN: Evaluation of diagnostic technology. PARTICIPANTS: One thousand seven hundred ninety-six retinal fundus images from 1612 diabeticpatients. METHODS: Ten ophthalmologists (5 general ophthalmologists, 4 retina specialists, 1 retina fellow) read images for DR severity based on the International Clinical Diabetic Retinopathy disease severity scale in each of 3 conditions: unassisted, grades only, or grades plus heatmap. Grades-only assistance comprised a histogram of DR predictions (grades) from a trained deep-learning model. For grades plus heatmap, we additionally showed explanatory heatmaps. MAIN OUTCOME MEASURES: For each experiment arm, we computed sensitivity and specificity of each reader and the algorithm for different levels of DR severity against an adjudicated reference standard. We also measured accuracy (exact 5-class level agreement and Cohen's quadratically weighted κ), reader-reported confidence (5-point Likert scale), and grading time. RESULTS: Readers graded more accurately with model assistance than without for the grades-only condition (P < 0.001). Grades plus heatmaps improved accuracy for patients with DR (P < 0.001), but reduced accuracy for patients without DR (P = 0.006). Both forms of assistance increased readers' sensitivity moderate-or-worse DR: unassisted: mean, 79.4% [95% confidence interval (CI), 72.3%-86.5%]; grades only: mean, 87.5% [95% CI, 85.1%-89.9%]; grades plus heatmap: mean, 88.7% [95% CI, 84.9%-92.5%] without a corresponding drop in specificity (unassisted: mean, 96.6% [95% CI, 95.9%-97.4%]; grades only: mean, 96.1% [95% CI, 95.5%-96.7%]; grades plus heatmap: mean, 95.5% [95% CI, 94.8%-96.1%]). Algorithmic assistance increased the accuracy of retina specialists above that of the unassisted reader or model alone; and increased grading confidence and grading time across all readers. For most cases, grades plus heatmap was only as effective as grades only. Over the course of the experiment, grading time decreased across all conditions, although most sharply for grades plus heatmap. CONCLUSIONS:Deep learning algorithms can improve the accuracy of, and confidence in, DR diagnosis in an assisted read setting. They also may increase grading time, although these effects may be ameliorated with experience.
Authors: Matthew D Li; Ken Chang; Ben Bearce; Connie Y Chang; Ambrose J Huang; J Peter Campbell; James M Brown; Praveer Singh; Katharina V Hoebel; Deniz Erdoğmuş; Stratis Ioannidis; William E Palmer; Michael F Chiang; Jayashree Kalpathy-Cramer Journal: NPJ Digit Med Date: 2020-03-26
Authors: Philipp Berens; Sebastian M Waldstein; Murat Seckin Ayhan; Louis Kümmerle; Hansjürgen Agostini; Andreas Stahl; Focke Ziemssen Journal: Ophthalmologe Date: 2020-04 Impact factor: 1.059
Authors: Kim V Garvey; Kelly Jean Thomas Craig; Regina G Russell; Laurie Novak; Don Moore; Anita M Preininger; Gretchen P Jackson; Bonnie M Miller Journal: Med Sci Educ Date: 2021-09-09