Anna Noel-Storr1, Gordon Dooley2, Julian Elliott3, Emily Steele4, Ian Shemilt5, Chris Mavergames6, Susanna Wisniewski7, Steven McDonald8, Melissa Murano8, Julie Glanville9, Ruth Foxlee10, Deirdre Beecher6, Jennifer Ware7, James Thomas5. 1. Radcliffe Department of Medicine, University of Oxford, Level 4, Academic Block, John Radcliffe Hospital, Headington, Oxford OX3 9DU, UK; People Services Department, Cochrane, St Albans House, 57-59 Haymarket, London SW1Y 4QX, UK. Electronic address: anna.noel-storr@rdm.ox.ac.uk. 2. Metaxis Ltd, Elmbank Offices, Main Road Curbridge, Witney, Oxfordshire OX29 7NT, UK. 3. Cochrane Australia, School of Public Health and Preventive Medicine, Monash University, 553 St Kilda Road, Melbourne, Victoria 3004, Australia; Department of Infectious Diseases, Monash University and Alfred Hospital, 55 Commercial Rd, Melbourne, Victoria 3004, Australia. 4. People Services Department, Cochrane, St Albans House, 57-59 Haymarket, London SW1Y 4QX, UK. 5. EPPI-Centre, Department of Social Science, University College London, 18 Woburn Square, London, WC1H 0NR, UK. 6. Informatics and Technology Systems, Cochrane, St Albans House, 57-59 Haymarket, London SW1Y 4QX, UK. 7. Radcliffe Department of Medicine, University of Oxford, Level 4, Academic Block, John Radcliffe Hospital, Headington, Oxford OX3 9DU, UK. 8. Cochrane Australia, School of Public Health and Preventive Medicine, Monash University, 553 St Kilda Road, Melbourne, Victoria 3004, Australia. 9. York Health Economics Consortium, University of York, Enterprise House, York YO10 5NQ, UK. 10. Editorial and Methods Department, Cochrane, St Albans House, 57-59 Haymarket, London SW1Y 4QX, UK.
Abstract
BACKGROUND AND OBJECTIVES: Filtering the deluge of new research to facilitate evidence synthesis has proven to be unmanageable using current paradigms of search and retrieval. Crowdsourcing, a way of harnessing the collective effort of a "crowd" of people, has the potential to support evidence synthesis by addressing this information overload created by the exponential growth in primary research outputs. Cochrane Crowd, Cochrane's citizen science platform, offers a range of tasks aimed at identifying studies related to health care. Accompanying each task are brief, interactive training modules, and agreement algorithms that help ensure accurate collective decision-making.The aims of the study were to evaluate the performance of Cochrane Crowd in terms of its accuracy, capacity, and autonomy and to examine contributor engagement across three tasks aimed at identifying randomized trials. STUDY DESIGN AND SETTING: Crowd accuracy was evaluated by measuring the sensitivity and specificity of crowd screening decisions on a sample of titles and abstracts, compared with "quasi gold-standard" decisions about the same records using the conventional methods of dual screening. Crowd capacity, in the form of output volume, was evaluated by measuring the number of records processed by the crowd, compared with baseline. Crowd autonomy, the capability of the crowd to produce accurate collectively derived decisions without the need for expert resolution, was measured by the proportion of records that needed resolving by an expert. RESULTS: The Cochrane Crowd community currently has 18,897 contributors from 163 countries. Collectively, the Crowd has processed 1,021,227 records, helping to identify 178,437 reports of randomized controlled trials (RCTs) for Cochrane's Central Register of Controlled Trials. The sensitivity for each task was 99.1% for the RCT identification task (RCT ID), 99.7% for the RCT identification task of trials from ClinicalTrials.gov (CT ID), and 97.7% for the identification of RCTs from the International Clinical Trials Registry Platform (ICTRP ID). The specificity for each task was 99% for RCT ID, 98.6% for CT ID, and 99.1% for CT ICTRP ID. The capacity of the combined Crowd and machine learning workflow has increased fivefold in 6 years, compared with baseline. The proportion of records requiring expert resolution across the tasks ranged from 16.6% to 19.7%. CONCLUSION: Cochrane Crowd is sufficiently accurate and scalable to keep pace with the current rate of publication (and registration) of new primary studies. It has also proved to be a popular, efficient, and accurate way for a large number of people to play an important voluntary role in health evidence production. Cochrane Crowd is now an established part of Cochrane's effort to manage the deluge of primary research being produced.
BACKGROUND AND OBJECTIVES: Filtering the deluge of new research to facilitate evidence synthesis has proven to be unmanageable using current paradigms of search and retrieval. Crowdsourcing, a way of harnessing the collective effort of a "crowd" of people, has the potential to support evidence synthesis by addressing this information overload created by the exponential growth in primary research outputs. Cochrane Crowd, Cochrane's citizen science platform, offers a range of tasks aimed at identifying studies related to health care. Accompanying each task are brief, interactive training modules, and agreement algorithms that help ensure accurate collective decision-making.The aims of the study were to evaluate the performance of Cochrane Crowd in terms of its accuracy, capacity, and autonomy and to examine contributor engagement across three tasks aimed at identifying randomized trials. STUDY DESIGN AND SETTING: Crowd accuracy was evaluated by measuring the sensitivity and specificity of crowd screening decisions on a sample of titles and abstracts, compared with "quasi gold-standard" decisions about the same records using the conventional methods of dual screening. Crowd capacity, in the form of output volume, was evaluated by measuring the number of records processed by the crowd, compared with baseline. Crowd autonomy, the capability of the crowd to produce accurate collectively derived decisions without the need for expert resolution, was measured by the proportion of records that needed resolving by an expert. RESULTS: The Cochrane Crowd community currently has 18,897 contributors from 163 countries. Collectively, the Crowd has processed 1,021,227 records, helping to identify 178,437 reports of randomized controlled trials (RCTs) for Cochrane's Central Register of Controlled Trials. The sensitivity for each task was 99.1% for the RCT identification task (RCT ID), 99.7% for the RCT identification task of trials from ClinicalTrials.gov (CT ID), and 97.7% for the identification of RCTs from the International Clinical Trials Registry Platform (ICTRP ID). The specificity for each task was 99% for RCT ID, 98.6% for CT ID, and 99.1% for CT ICTRP ID. The capacity of the combined Crowd and machine learning workflow has increased fivefold in 6 years, compared with baseline. The proportion of records requiring expert resolution across the tasks ranged from 16.6% to 19.7%. CONCLUSION: Cochrane Crowd is sufficiently accurate and scalable to keep pace with the current rate of publication (and registration) of new primary studies. It has also proved to be a popular, efficient, and accurate way for a large number of people to play an important voluntary role in health evidence production. Cochrane Crowd is now an established part of Cochrane's effort to manage the deluge of primary research being produced.
Authors: Julian Elliott; Rebecca Lawrence; Jan C Minx; Olufemi T Oladapo; Philippe Ravaud; Britta Tendal Jeppesen; James Thomas; Tari Turner; Per Olav Vandvik; Jeremy M Grimshaw Journal: Nature Date: 2021-12 Impact factor: 49.962
Authors: Monika Geretsegger; Laura Fusar-Poli; Cochavit Elefant; Karin A Mössler; Giovanni Vitale; Christian Gold Journal: Cochrane Database Syst Rev Date: 2022-05-09
Authors: Elizabeth Allen; Alice R Rumbold; Amy Keir; Carmel T Collins; Jennifer Gillis; Hiroki Suganuma Journal: Cochrane Database Syst Rev Date: 2021-10-21