Gerald Gartlehner1, Lisa Affengruber2, Viktoria Titscher3, Anna Noel-Storr4, Gordon Dooley5, Nicolas Ballarini6, Franz König6. 1. Department for Evidence-based Medicine and Evaluation, Cochrane Austria, Danube University Krems, Krems, Austria; RTI-University of North Carolina Evidence-based Practice Center, RTI International, Research Triangle Park, NC, USA. Electronic address: gartlehner@cochrane.at. 2. Department for Evidence-based Medicine and Evaluation, Cochrane Austria, Danube University Krems, Krems, Austria; Department of Family Medicine, Care and Public Health Research Institute, Maastricht University, Maastricht, The Netherlands. 3. Department for Evidence-based Medicine and Evaluation, Cochrane Austria, Danube University Krems, Krems, Austria. 4. Radcliffe Department of Medicine, University of Oxford, Oxford, UK. 5. Metaxis Ltd, Curbridge, UK. 6. Section of Medical Statistics, Medical University of Vienna, Vienna, Austria.
Abstract
OBJECTIVES: To determine the accuracy of single-reviewer screening in correctly classifying abstracts as relevant or irrelevant for literature reviews. STUDY DESIGN AND SETTING: We conducted a crowd-based, parallel-group randomized controlled trial. Using the Cochrane Crowd platform, we randomly assigned eligible participants to 100 abstracts each of a pharmacological or a public health topic. After completing a training exercise, participants screened abstracts online based on predefined inclusion and exclusion criteria. We calculated sensitivities and specificities of single- and dual-reviewer screening using two published systematic reviews as reference standards. RESULTS:Two hundred and eighty participants made 24,942 screening decisions on 2,000 randomly selected abstracts from the reference standard reviews. On average, each abstract was screened 12 times. Overall, single-reviewer abstract screening missed 13% of relevant studies (sensitivity: 86.6%; 95% confidence interval [CI], 80.6%-91.2%). By comparison, dual-reviewer abstract screening missed 3% of relevant studies (sensitivity: 97.5%; 95% CI, 95.1%-98.8%). The corresponding specificities were 79.2% (95% CI, 77.4%-80.9%) and 68.7% (95% CI, 66.4%-71.0%), respectively. CONCLUSIONS: Single-reviewer abstract screening does not appear to fulfill the high methodological standards that decisionmakers expect from systematic reviews. It may be a viable option for rapid reviews, which deliberately lower methodological standards to provide decision makers with accelerated evidence synthesis products.
RCT Entities:
OBJECTIVES: To determine the accuracy of single-reviewer screening in correctly classifying abstracts as relevant or irrelevant for literature reviews. STUDY DESIGN AND SETTING: We conducted a crowd-based, parallel-group randomized controlled trial. Using the Cochrane Crowd platform, we randomly assigned eligible participants to 100 abstracts each of a pharmacological or a public health topic. After completing a training exercise, participants screened abstracts online based on predefined inclusion and exclusion criteria. We calculated sensitivities and specificities of single- and dual-reviewer screening using two published systematic reviews as reference standards. RESULTS: Two hundred and eighty participants made 24,942 screening decisions on 2,000 randomly selected abstracts from the reference standard reviews. On average, each abstract was screened 12 times. Overall, single-reviewer abstract screening missed 13% of relevant studies (sensitivity: 86.6%; 95% confidence interval [CI], 80.6%-91.2%). By comparison, dual-reviewer abstract screening missed 3% of relevant studies (sensitivity: 97.5%; 95% CI, 95.1%-98.8%). The corresponding specificities were 79.2% (95% CI, 77.4%-80.9%) and 68.7% (95% CI, 66.4%-71.0%), respectively. CONCLUSIONS: Single-reviewer abstract screening does not appear to fulfill the high methodological standards that decisionmakers expect from systematic reviews. It may be a viable option for rapid reviews, which deliberately lower methodological standards to provide decision makers with accelerated evidence synthesis products.
Authors: Francis Q S Dzakpasu; Alison Carver; Christian J Brakenridge; Flavia Cicuttini; Donna M Urquhart; Neville Owen; David W Dunstan Journal: Int J Behav Nutr Phys Act Date: 2021-12-13 Impact factor: 6.457
Authors: Jennifer Hunter; Susan Arentz; Joshua Goldenberg; Guoyan Yang; Jennifer Beardsley; Dominik Mertz; Stephen Leeder Journal: Integr Med Res Date: 2020-06-23
Authors: Dena Zeraatkar; Arrti Bhasin; Rita E Morassut; Isabella Churchill; Arnav Gupta; Daeria O Lawson; Anna Miroshnychenko; Emily Sirotich; Komal Aryal; David Mikhail; Tauseef A Khan; Vanessa Ha; John L Sievenpiper; Steven E Hanna; Joseph Beyene; Russell J de Souza Journal: Am J Clin Nutr Date: 2021-06-01 Impact factor: 7.045
Authors: Katie O'Hearn; Cameron MacDonald; Anne Tsampalieros; Leo Kadota; Ryan Sandarage; Supun Kotteduwa Jayawarden; Michele Datko; John M Reynolds; Thanh Bui; Shagufta Sultan; Margaret Sampson; Misty Pratt; Nick Barrowman; Nassr Nama; Matthew Page; James Dayre McNally Journal: BMC Med Res Methodol Date: 2021-07-08 Impact factor: 4.615