Melanie Ganz1, Daniel Kondermann2, Jonas Andrulis2, Gitte Moos Knudsen3, Lena Maier-Hein4. 1. Neurobiology Research Unit, Center for Integrated Molecular Brain Imaging, Rigshospitalet, Juliane Maries Vej 28, 2100, Copenhagen, Denmark. melanie.ganz@nru.dk. 2. Pallas Ludens GmbH, Im Bosseldorn 29, 69126, Heidelberg, Germany. 3. Neurobiology Research Unit, Center for Integrated Molecular Brain Imaging, Rigshospitalet, Juliane Maries Vej 28, 2100, Copenhagen, Denmark. 4. Computer-assisted Interventions, German Cancer Research Center (DKFZ), Im Neuenheimer Feld 280, 69120, Heidelberg, Germany.
Abstract
PURPOSE: With the recent trend toward big data analysis, neuroimaging datasets have grown substantially in the past years. While larger datasets potentially offer important insights for medical research, one major bottleneck is the requirement for resources of medical experts needed to validate automatic processing results. To address this issue, the goal of this paper was to assess whether anonymous nonexperts from an online community can perform quality control of MR-based cortical surface delineations derived by an automatic algorithm. METHODS: So-called knowledge workers from an online crowdsourcing platform were asked to annotate errors in automatic cortical surface delineations on 100 central, coronal slices of MR images. RESULTS: On average, annotations for 100 images were obtained in less than an hour. When using expert annotations as reference, the crowd on average achieves a sensitivity of 82 % and a precision of 42 %. Merging multiple annotations per image significantly improves the sensitivity of the crowd (up to 95 %), but leads to a decrease in precision (as low as 22 %). CONCLUSION: Our experiments show that the detection of errors in automatic cortical surface delineations generated by anonymous untrained workers is feasible. Future work will focus on increasing the sensitivity of our method further, such that the error detection tasks can be handled exclusively by the crowd and expert resources can be focused on error correction.
PURPOSE: With the recent trend toward big data analysis, neuroimaging datasets have grown substantially in the past years. While larger datasets potentially offer important insights for medical research, one major bottleneck is the requirement for resources of medical experts needed to validate automatic processing results. To address this issue, the goal of this paper was to assess whether anonymous nonexperts from an online community can perform quality control of MR-based cortical surface delineations derived by an automatic algorithm. METHODS: So-called knowledge workers from an online crowdsourcing platform were asked to annotate errors in automatic cortical surface delineations on 100 central, coronal slices of MR images. RESULTS: On average, annotations for 100 images were obtained in less than an hour. When using expert annotations as reference, the crowd on average achieves a sensitivity of 82 % and a precision of 42 %. Merging multiple annotations per image significantly improves the sensitivity of the crowd (up to 95 %), but leads to a decrease in precision (as low as 22 %). CONCLUSION: Our experiments show that the detection of errors in automatic cortical surface delineations generated by anonymous untrained workers is feasible. Future work will focus on increasing the sensitivity of our method further, such that the error detection tasks can be handled exclusively by the crowd and expert resources can be focused on error correction.
Authors: Benjamin L Ranard; Yoonhee P Ha; Zachary F Meisel; David A Asch; Shawndra S Hill; Lance B Becker; Anne K Seymour; Raina M Merchant Journal: J Gen Intern Med Date: 2013-07-11 Impact factor: 5.128
Authors: Lena Maier-Hein; Sven Mersmann; Daniel Kondermann; Christian Stock; Hannes Gotz Kenngott; Alexandro Sanchez; Martin Wagner; Anas Preukschas; Anna-Laura Wekerle; Stefanie Helfert; Sebastian Bodenstedt; Stefanie Speidel Journal: Med Image Comput Comput Assist Interv Date: 2014
Authors: Lena Maier-Hein; Sven Mersmann; Daniel Kondermann; Sebastian Bodenstedt; Alexandro Sanchez; Christian Stock; Hannes Gotz Kenngott; Mathias Eisenmann; Stefanie Speidel Journal: Med Image Comput Comput Assist Interv Date: 2014
Authors: K H Fritzsche; P F Neher; I Reicht; T van Bruggen; C Goch; M Reisert; M Nolden; S Zelzer; H-P Meinzer; B Stieltjes Journal: Methods Inf Med Date: 2012-09-28 Impact factor: 2.176
Authors: M E Haahr; D L Hansen; P M Fisher; C Svarer; D S Stenbæk; K Madsen; J Madsen; J J Holst; W F C Baaré; L Hojgaard; T Almdal; G M Knudsen Journal: J Neurosci Date: 2015-04-08 Impact factor: 6.167
Authors: Mark Jenkinson; Christian F Beckmann; Timothy E J Behrens; Mark W Woolrich; Stephen M Smith Journal: Neuroimage Date: 2011-09-16 Impact factor: 6.556