Matthew J Page1, Douglas G Altman2, Joanne E McKenzie3, Larissa Shamseer4, Nadera Ahmadzai5, Dianna Wolfe5, Fatemeh Yazdi5, Ferrán Catalá-López6, Andrea C Tricco7, David Moher4. 1. School of Public Health and Preventive Medicine, Monash University, 553 St Kilda Road, Melbourne, Victoria 3004, Australia. Electronic address: matthew.page@monash.edu. 2. UK EQUATOR Centre, Centre for Statistics in Medicine, NDORMS, University of Oxford, Oxford OX3 7LD, United Kingdom. 3. School of Public Health and Preventive Medicine, Monash University, 553 St Kilda Road, Melbourne, Victoria 3004, Australia. 4. Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, K1H 8L6, Canada; School of Epidemiology, Public Health and Preventive Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, K1H 8M5, Canada. 5. Knowledge Synthesis Group, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, K1H 8L6, Canada. 6. Knowledge Synthesis Group, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, K1H 8L6, Canada; Department of Medicine, University of Valencia/INCLIVA Health Research Institute and CIBERSAM, Valencia, 46010, Spain. 7. Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, Ontario, M5B 1W8, Canada; Epidemiology Division, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, M5T 3M7, Canada.
Abstract
OBJECTIVES: The objective of the study was to investigate the application and interpretation of statistical analyses in a cross-section of systematic reviews (SRs) of therapeutic interventions, without restriction by journal, clinical condition, or specialty. STUDY DESIGN AND SETTING: We evaluated a random sample of SRs assembled previously, which were indexed in MEDLINE® during February 2014, focused on a treatment or prevention question, and reported at least one meta-analysis. The reported statistical methods used in each SR were extracted from articles and online appendices by one author, with a 20% random sample extracted in duplicate. RESULTS: We evaluated 110 SRs; 78/110 (71%) were non-Cochrane SRs and 55/110 (50%) investigated a pharmacological intervention. The SRs presented a median of 13 (interquartile range: 5-27) meta-analytic effects. When considering the index (primary or first reported) meta-analysis of each SR, just over half (62/110 [56%]) used the random-effects model, but few (5/62 [8%]) interpreted the meta-analytic effect correctly (as the average of the intervention effects across all studies). A statistical test for funnel plot asymmetry was reported in 17/110 (15%) SRs; however, in only 4/17 (24%) did the test include the recommended number of at least 10 studies of varying size. Subgroup analyses accompanied 42/110 (38%) index meta-analyses, but findings were not interpreted with respect to a test for interaction in 29/42 (69%) cases, and the issue of potential confounding in the subgroup analyses was not raised in any SR. CONCLUSIONS: There is scope for improvement in the application and interpretation of statistical analyses in SRs of therapeutic interventions. The involvement of statisticians on the SR team and establishment of partnerships between researchers with specialist expertise in SR methods and journal editors may help overcome these shortcomings.
OBJECTIVES: The objective of the study was to investigate the application and interpretation of statistical analyses in a cross-section of systematic reviews (SRs) of therapeutic interventions, without restriction by journal, clinical condition, or specialty. STUDY DESIGN AND SETTING: We evaluated a random sample of SRs assembled previously, which were indexed in MEDLINE® during February 2014, focused on a treatment or prevention question, and reported at least one meta-analysis. The reported statistical methods used in each SR were extracted from articles and online appendices by one author, with a 20% random sample extracted in duplicate. RESULTS: We evaluated 110 SRs; 78/110 (71%) were non-Cochrane SRs and 55/110 (50%) investigated a pharmacological intervention. The SRs presented a median of 13 (interquartile range: 5-27) meta-analytic effects. When considering the index (primary or first reported) meta-analysis of each SR, just over half (62/110 [56%]) used the random-effects model, but few (5/62 [8%]) interpreted the meta-analytic effect correctly (as the average of the intervention effects across all studies). A statistical test for funnel plot asymmetry was reported in 17/110 (15%) SRs; however, in only 4/17 (24%) did the test include the recommended number of at least 10 studies of varying size. Subgroup analyses accompanied 42/110 (38%) index meta-analyses, but findings were not interpreted with respect to a test for interaction in 29/42 (69%) cases, and the issue of potential confounding in the subgroup analyses was not raised in any SR. CONCLUSIONS: There is scope for improvement in the application and interpretation of statistical analyses in SRs of therapeutic interventions. The involvement of statisticians on the SR team and establishment of partnerships between researchers with specialist expertise in SR methods and journal editors may help overcome these shortcomings.
Authors: Dena Zeraatkar; Arrti Bhasin; Rita E Morassut; Isabella Churchill; Arnav Gupta; Daeria O Lawson; Anna Miroshnychenko; Emily Sirotich; Komal Aryal; David Mikhail; Tauseef A Khan; Vanessa Ha; John L Sievenpiper; Steven E Hanna; Joseph Beyene; Russell J de Souza Journal: Am J Clin Nutr Date: 2021-06-01 Impact factor: 7.045
Authors: Cathrine A McKenzie; Valerie J Page; W David Strain; Bronagh Blackwood; Marlies Ostermann; David Taylor; Peter E Spronk; Daniel F McAuley Journal: Syst Rev Date: 2020-06-05
Authors: Ferrán Catalá-López; Adolfo Alonso-Arroyo; Matthew J Page; Brian Hutton; Manuel Ridao; Rafael Tabarés-Seisdedos; Rafael Aleixandre-Benavent; David Moher Journal: BMJ Open Date: 2019-03-04 Impact factor: 2.692