Matthew J Page1, Douglas G Altman2, Larissa Shamseer3, Joanne E McKenzie4, Nadera Ahmadzai5, Dianna Wolfe5, Fatemeh Yazdi5, Ferrán Catalá-López6, Andrea C Tricco7, David Moher3. 1. School of Public Health and Preventive Medicine, Monash University, 553 St Kilda Road, Melbourne, Victoria 3004, Australia. Electronic address: matthew.page@monash.edu. 2. UK EQUATOR Centre, Centre for Statistics in Medicine, NDORMS, University of Oxford, Windmill Road, Oxford OX3 7LD, UK. 3. Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, 501 Smyth Road, Ottawa, Ontario K1H 8L6, Canada; School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, 451 Smyth Road, Ottawa, Ontario K1H 8M5, Canada. 4. School of Public Health and Preventive Medicine, Monash University, 553 St Kilda Road, Melbourne, Victoria 3004, Australia. 5. Knowledge Synthesis Group, Clinical Epidemiology Program, Ottawa Hospital Research Institute, 451 Smyth Road, Ottawa, Ontario K1H 8M5, Canada. 6. Knowledge Synthesis Group, Clinical Epidemiology Program, Ottawa Hospital Research Institute, 451 Smyth Road, Ottawa, Ontario K1H 8M5, Canada; Department of Medicine, University of Valencia/INCLIVA Health Research Institute and CIBERSAM, Valencia 46010, Spain. 7. Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, 30 Bond Street, Ontario M5B 1W8, Canada; Epidemiology Division, Dalla Lana School of Public Health, University of Toronto, 155 College Street, Toronto, Ontario M5T 3M7, Canada.
Abstract
OBJECTIVES: To evaluate how often reproducible research practices, which allow others to recreate the findings of studies, given the original data, are used in systematic reviews (SRs) of biomedical research. STUDY DESIGN AND SETTING: We evaluated a random sample of SRs indexed in MEDLINE during February 2014, which focused on a therapeutic intervention and reported at least one meta-analysis. Data on reproducible research practices in each SR were extracted using a 26-item form by one author, with a 20% random sample extracted in duplicate. We explored whether the use of reproducible research practices was associated with an SR being a Cochrane review, as well as with the reported use of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. RESULTS: We evaluated 110 SRs of therapeutic interventions, 78 (71%) of which were non-Cochrane SRs. Across the SRs, there were 2,139 meta-analytic effects (including subgroup meta-analytic effects and sensitivity analyses), 1,551 (73%) of which were reported in sufficient detail to recreate them. Systematic reviewers reported the data needed to recreate all meta-analytic effects in 72 (65%) SRs only. This percentage was higher in Cochrane than in non-Cochrane SRs (30/32 [94%] vs. 42/78 [54%]; risk ratio 1.74, 95% confidence interval 1.39-2.18). Systematic reviewers who reported imputing, algebraically manipulating, or obtaining some data from the study author/sponsor infrequently stated which specific data were handled in this way. Only 33 (30%) SRs mentioned access to data sets and statistical code used to perform analyses. CONCLUSION: Reproducible research practices are underused in SRs of biomedical interventions. Adoption of such practices facilitates identification of errors and allows the SR data to be reanalyzed.
OBJECTIVES: To evaluate how often reproducible research practices, which allow others to recreate the findings of studies, given the original data, are used in systematic reviews (SRs) of biomedical research. STUDY DESIGN AND SETTING: We evaluated a random sample of SRs indexed in MEDLINE during February 2014, which focused on a therapeutic intervention and reported at least one meta-analysis. Data on reproducible research practices in each SR were extracted using a 26-item form by one author, with a 20% random sample extracted in duplicate. We explored whether the use of reproducible research practices was associated with an SR being a Cochrane review, as well as with the reported use of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. RESULTS: We evaluated 110 SRs of therapeutic interventions, 78 (71%) of which were non-Cochrane SRs. Across the SRs, there were 2,139 meta-analytic effects (including subgroup meta-analytic effects and sensitivity analyses), 1,551 (73%) of which were reported in sufficient detail to recreate them. Systematic reviewers reported the data needed to recreate all meta-analytic effects in 72 (65%) SRs only. This percentage was higher in Cochrane than in non-Cochrane SRs (30/32 [94%] vs. 42/78 [54%]; risk ratio 1.74, 95% confidence interval 1.39-2.18). Systematic reviewers who reported imputing, algebraically manipulating, or obtaining some data from the study author/sponsor infrequently stated which specific data were handled in this way. Only 33 (30%) SRs mentioned access to data sets and statistical code used to perform analyses. CONCLUSION: Reproducible research practices are underused in SRs of biomedical interventions. Adoption of such practices facilitates identification of errors and allows the SR data to be reanalyzed.
Authors: Dena Zeraatkar; Arrti Bhasin; Rita E Morassut; Isabella Churchill; Arnav Gupta; Daeria O Lawson; Anna Miroshnychenko; Emily Sirotich; Komal Aryal; David Mikhail; Tauseef A Khan; Vanessa Ha; John L Sievenpiper; Steven E Hanna; Joseph Beyene; Russell J de Souza Journal: Am J Clin Nutr Date: 2021-06-01 Impact factor: 7.045
Authors: Matthew J Page; Joanne E McKenzie; Patrick M Bossuyt; Isabelle Boutron; Tammy C Hoffmann; Cynthia D Mulrow; Larissa Shamseer; Jennifer M Tetzlaff; Elie A Akl; Sue E Brennan; Roger Chou; Julie Glanville; Jeremy M Grimshaw; Asbjørn Hróbjartsson; Manoj M Lalu; Tianjing Li; Elizabeth W Loder; Evan Mayo-Wilson; Steve McDonald; Luke A McGuinness; Lesley A Stewart; James Thomas; Andrea C Tricco; Vivian A Welch; Penny Whiting; David Moher Journal: Syst Rev Date: 2021-03-29
Authors: Holly K Grossetta Nardini; Janene Batten; Melissa C Funaro; Rolando Garcia-Milian; Kate Nyhan; Judy M Spak; Lei Wang; Janis G Glover Journal: Res Integr Peer Rev Date: 2019-11-27
Authors: Ferrán Catalá-López; Lisa Caulley; Manuel Ridao; Brian Hutton; Don Husereau; Michael F Drummond; Adolfo Alonso-Arroyo; Manuel Pardo-Fernández; Enrique Bernal-Delgado; Ricard Meneu; Rafael Tabarés-Seisdedos; José Ramón Repullo; David Moher Journal: BMJ Open Date: 2020-02-13 Impact factor: 2.692