Jonathan M Mortensen1, Evan P Minty2, Michael Januszyk3, Timothy E Sweeney4, Alan L Rector5, Natalya F Noy6, Mark A Musen1. 1. Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA Biomedical Informatics Training Program, Stanford University, Stanford, California, USA. 2. Biomedical Informatics Training Program, Stanford University, Stanford, California, USA Faculty of Medicine, University of Calgary, Calgary, Canada. 3. Biomedical Informatics Training Program, Stanford University, Stanford, California, USA. 4. Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA. 5. School of Computer Science, University of Manchester, Manchester, UK. 6. Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA Google Inc., Mountain View, California, USA.
Abstract
OBJECTIVES: The verification of biomedical ontologies is an arduous process that typically involves peer review by subject-matter experts. This work evaluated the ability of crowdsourcing methods to detect errors in SNOMED CT (Systematized Nomenclature of Medicine Clinical Terms) and to address the challenges of scalable ontology verification. METHODS: We developed a methodology to crowdsource ontology verification that uses micro-tasking combined with a Bayesian classifier. We then conducted a prospective study in which both the crowd and domain experts verified a subset of SNOMED CT comprising 200 taxonomic relationships. RESULTS: The crowd identified errors as well as any single expert at about one-quarter of the cost. The inter-rater agreement (κ) between the crowd and the experts was 0.58; the inter-rater agreement between experts themselves was 0.59, suggesting that the crowd is nearly indistinguishable from any one expert. Furthermore, the crowd identified 39 previously undiscovered, critical errors in SNOMED CT (eg, 'septic shock is a soft-tissue infection'). DISCUSSION: The results show that the crowd can indeed identify errors in SNOMED CT that experts also find, and the results suggest that our method will likely perform well on similar ontologies. The crowd may be particularly useful in situations where an expert is unavailable, budget is limited, or an ontology is too large for manual error checking. Finally, our results suggest that the online anonymous crowd could successfully complete other domain-specific tasks. CONCLUSIONS: We have demonstrated that the crowd can address the challenges of scalable ontology verification, completing not only intuitive, common-sense tasks, but also expert-level, knowledge-intensive tasks.
OBJECTIVES: The verification of biomedical ontologies is an arduous process that typically involves peer review by subject-matter experts. This work evaluated the ability of crowdsourcing methods to detect errors in SNOMED CT (Systematized Nomenclature of Medicine Clinical Terms) and to address the challenges of scalable ontology verification. METHODS: We developed a methodology to crowdsource ontology verification that uses micro-tasking combined with a Bayesian classifier. We then conducted a prospective study in which both the crowd and domain experts verified a subset of SNOMED CT comprising 200 taxonomic relationships. RESULTS: The crowd identified errors as well as any single expert at about one-quarter of the cost. The inter-rater agreement (κ) between the crowd and the experts was 0.58; the inter-rater agreement between experts themselves was 0.59, suggesting that the crowd is nearly indistinguishable from any one expert. Furthermore, the crowd identified 39 previously undiscovered, critical errors in SNOMED CT (eg, 'septic shock is a soft-tissue infection'). DISCUSSION: The results show that the crowd can indeed identify errors in SNOMED CT that experts also find, and the results suggest that our method will likely perform well on similar ontologies. The crowd may be particularly useful in situations where an expert is unavailable, budget is limited, or an ontology is too large for manual error checking. Finally, our results suggest that the online anonymous crowd could successfully complete other domain-specific tasks. CONCLUSIONS: We have demonstrated that the crowd can address the challenges of scalable ontology verification, completing not only intuitive, common-sense tasks, but also expert-level, knowledge-intensive tasks.
Authors: Christopher Ochs; Yehoshua Perl; James Geller; Michael Halper; Huanying Gu; Yan Chen; Gai Elhanan Journal: AMIA Annu Symp Proc Date: 2013-11-16
Authors: Patricia L Whetzel; Natalya F Noy; Nigam H Shah; Paul R Alexander; Csongor Nyulas; Tania Tudorache; Mark A Musen Journal: Nucleic Acids Res Date: 2011-06-14 Impact factor: 16.971
Authors: Robert Hoehndorf; Michel Dumontier; John H Gennari; Sarala Wimalaratne; Bernard de Bono; Daniel L Cook; Georgios V Gkoutos Journal: BMC Syst Biol Date: 2011-08-11
Authors: Lawrence Hunter; Zhiyong Lu; James Firby; William A Baumgartner; Helen L Johnson; Philip V Ogren; K Bretonnel Cohen Journal: BMC Bioinformatics Date: 2008-01-31 Impact factor: 3.169
Authors: Julio Saez-Rodriguez; James C Costello; Stephen H Friend; Michael R Kellen; Lara Mangravite; Pablo Meyer; Thea Norman; Gustavo Stolovitzky Journal: Nat Rev Genet Date: 2016-07-15 Impact factor: 53.242
Authors: Christopher Ochs; Zhe He; Ling Zheng; James Geller; Yehoshua Perl; George Hripcsak; Mark A Musen Journal: J Biomed Inform Date: 2016-03-14 Impact factor: 6.317