Alex Gartland1, Andrew Bate2, Jeffery L Painter3, Tim A Casperson4, Gregory Eugene Powell5. 1. College of Medicine, University of Central Florida, Orlando, FL, USA. 2. Safety and Medical Governance, GlaxoSmithKline, London, UK. 3. JiveCast, Raleigh, NC, USA. 4. North American Medical Affairs, GlaxoSmithKline, Research Triangle Park, NC, USA. 5. Pharma Safety, GlaxoSmithKline, 5 Moore Dr., Research Triangle Park, NC, 27709, USA. gregory.e.powell@gsk.com.
Abstract
INTRODUCTION: Machine learning offers an alluring solution to developing automated approaches to the increasing individual case safety report burden being placed upon pharmacovigilance. Leveraging crowdsourcing to annotate unstructured data may provide accurate, efficient, and contemporaneous training data sets in support of machine learning. OBJECTIVE: The objective of this study was to evaluate whether crowdsourcing can be used to accurately and efficiently develop training data sets in support of pharmacovigilance automation. MATERIALS AND METHODS: Pharmacovigilance experts created a reference dataset by reviewing 15,490 de-identified social media posts of narratives pertaining to 15 drugs and 22 medically relevant topics. A random sampling of posts from the reference dataset was published on Amazon Turk and its users (Turkers) were asked a series of questions about those same medical concepts. Accuracy, price elasticity, and time efficiency were evaluated. RESULTS: Accuracy of crowdsourced curation exceeded 90% when compared to the reference dataset and was completed in about 5% of the time. There was an increase in time efficiency with higher pay, but there was no significant difference in accuracy. Additionally, having a social media post reviewed by more than one Turker (using a voting system) did not offer significant improvements in terms of accuracy. CONCLUSIONS: Crowdsourcing is an accurate and efficient method that can be used to develop training data sets in support of pharmacovigilance automation. More research is needed to better understand the breadth and depth of possible uses as well as strengths, limitations, and generalizability of results.
INTRODUCTION: Machine learning offers an alluring solution to developing automated approaches to the increasing individual case safety report burden being placed upon pharmacovigilance. Leveraging crowdsourcing to annotate unstructured data may provide accurate, efficient, and contemporaneous training data sets in support of machine learning. OBJECTIVE: The objective of this study was to evaluate whether crowdsourcing can be used to accurately and efficiently develop training data sets in support of pharmacovigilance automation. MATERIALS AND METHODS: Pharmacovigilance experts created a reference dataset by reviewing 15,490 de-identified social media posts of narratives pertaining to 15 drugs and 22 medically relevant topics. A random sampling of posts from the reference dataset was published on Amazon Turk and its users (Turkers) were asked a series of questions about those same medical concepts. Accuracy, price elasticity, and time efficiency were evaluated. RESULTS: Accuracy of crowdsourced curation exceeded 90% when compared to the reference dataset and was completed in about 5% of the time. There was an increase in time efficiency with higher pay, but there was no significant difference in accuracy. Additionally, having a social media post reviewed by more than one Turker (using a voting system) did not offer significant improvements in terms of accuracy. CONCLUSIONS: Crowdsourcing is an accurate and efficient method that can be used to develop training data sets in support of pharmacovigilance automation. More research is needed to better understand the breadth and depth of possible uses as well as strengths, limitations, and generalizability of results.
Authors: Carrie E Pierce; Khaled Bouri; Carol Pamer; Scott Proestel; Harold W Rodriguez; Hoa Van Le; Clark C Freifeld; John S Brownstein; Mark Walderhaug; I Ralph Edwards; Nabarun Dasgupta Journal: Drug Saf Date: 2017-04 Impact factor: 5.606
Authors: Carrie E Pierce; Sieta T de Vries; Stephanie Bodin-Parssinen; Linda Härmark; Phil Tregunno; David J Lewis; Simon Maskell; Raphael Van Eemeren; Alicia Ptaszynska-Neophytou; Victoria Newbould; Nabarun Dasgupta; Antoni F Z Wisniewski; Sara Gama; Peter G M Mol Journal: Drug Saf Date: 2019-04 Impact factor: 5.606
Authors: John van Stekelenborg; Johan Ellenius; Simon Maskell; Tomas Bergvall; Ola Caster; Nabarun Dasgupta; Juergen Dietrich; Sara Gama; David Lewis; Victoria Newbould; Sabine Brosch; Carrie E Pierce; Gregory Powell; Alicia Ptaszyńska-Neophytou; Antoni F Z Wiśniewski; Phil Tregunno; G Niklas Norén; Munir Pirmohamed Journal: Drug Saf Date: 2019-12 Impact factor: 5.606