Xueyang Wang1, Lucy I Mudie, Mani Baskaran, Ching-Yu Cheng, Wallace L Alward, David S Friedman, Christopher J Brady. 1. *Johns Hopkins School of Medicine, Wilmer Eye Institute, Baltimore, MD ∥Department of Ophthalmology and Visual Sciences, University of Iowa Carver College of Medicine, Iowa City, IA †Singapore National Eye Center, Singapore Eye Research Institute ‡Duke-NUS Medical School §Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore.
Abstract
PURPOSE: To assess the accuracy of crowdsourcing for grading optic nerve images for glaucoma using Amazon Mechanical Turk before and after training modules. MATERIALS AND METHODS: Images (n=60) from 2 large population studies were graded for glaucoma status and vertical cup-to-disc ratio (VCDR). In the baseline trial, users on Amazon Mechanical Turk (Turkers) graded fundus photos for glaucoma and VCDR after reviewing annotated example images. In 2 additional trials, Turkers viewed a 26-slide PowerPoint training or a 10-minute video training and passed a quiz before being permitted to grade the same 60 images. Each image was graded by 10 unique Turkers in all trials. The mode of Turker grades for each image was compared with an adjudicated expert grade to determine accuracy as well as the sensitivity and specificity of Turker grading. RESULTS: In the baseline study, 50% of the images were graded correctly for glaucoma status and the area under the receiver operating characteristic (AUROC) was 0.75 [95% confidence interval (CI), 0.64-0.87]. Post-PowerPoint training, 66.7% of the images were graded correctly with AUROC of 0.86 (95% CI, 0.78-0.95). Finally, Turker grading accuracy was 63.3% with AUROC of 0.89 (95% CI, 0.83-0.96) after video training. Overall, Turker VCDR grades for each image correlated with expert VCDR grades (Bland-Altman plot mean difference=-0.02). CONCLUSIONS: Turkers graded 60 fundus images quickly and at low cost, with grading accuracy, sensitivity, and specificity, all improving with brief training. With effective education, crowdsourcing may be an efficient tool to aid in the identification of glaucomatous changes in retinal images.
PURPOSE: To assess the accuracy of crowdsourcing for grading optic nerve images for glaucoma using Amazon Mechanical Turk before and after training modules. MATERIALS AND METHODS: Images (n=60) from 2 large population studies were graded for glaucoma status and vertical cup-to-disc ratio (VCDR). In the baseline trial, users on Amazon Mechanical Turk (Turkers) graded fundus photos for glaucoma and VCDR after reviewing annotated example images. In 2 additional trials, Turkers viewed a 26-slide PowerPoint training or a 10-minute video training and passed a quiz before being permitted to grade the same 60 images. Each image was graded by 10 unique Turkers in all trials. The mode of Turker grades for each image was compared with an adjudicated expert grade to determine accuracy as well as the sensitivity and specificity of Turker grading. RESULTS: In the baseline study, 50% of the images were graded correctly for glaucoma status and the area under the receiver operating characteristic (AUROC) was 0.75 [95% confidence interval (CI), 0.64-0.87]. Post-PowerPoint training, 66.7% of the images were graded correctly with AUROC of 0.86 (95% CI, 0.78-0.95). Finally, Turker grading accuracy was 63.3% with AUROC of 0.89 (95% CI, 0.83-0.96) after video training. Overall, Turker VCDR grades for each image correlated with expert VCDR grades (Bland-Altman plot mean difference=-0.02). CONCLUSIONS: Turkers graded 60 fundus images quickly and at low cost, with grading accuracy, sensitivity, and specificity, all improving with brief training. With effective education, crowdsourcing may be an efficient tool to aid in the identification of glaucomatous changes in retinal images.
Authors: Augusto Azuara-Blanco; L Jay Katz; George L Spaeth; Stephen A Vernon; Fiona Spencer; Ines M Lanzl Journal: Am J Ophthalmol Date: 2003-11 Impact factor: 5.258
Authors: Henry D Jampel; David Friedman; Harry Quigley; Susan Vitale; Rhonda Miller; Frederick Knezevich; Yulan Ding Journal: Am J Ophthalmol Date: 2008-09-13 Impact factor: 5.258
Authors: Jinseop S Kim; Matthew J Greene; Aleksandar Zlateski; Kisuk Lee; Mark Richardson; Srinivas C Turaga; Michael Purcaro; Matthew Balkam; Amy Robinson; Bardia F Behabadi; Michael Campos; Winfried Denk; H Sebastian Seung Journal: Nature Date: 2014-05-04 Impact factor: 49.962
Authors: Christopher John Brady; Lucy Iluka Mudie; Xueyang Wang; Eliseo Guallar; David Steven Friedman Journal: J Med Internet Res Date: 2017-06-20 Impact factor: 5.428