Claudia Hacke1, David Nunan2. 1. Department of Pediatrics I, University Medical Center Schleswig-Holstein, Campus Kiel, Arnold-Heller-Straße 3, Kiel 24105, Germany; Christian-Albrechts-University of Kiel, Institute of Sports Science, Olshausenstraße 74, 24118 Kiel, Germany. Electronic address: claudia.hacke@uksh.de. 2. Nuffield Department of Primary Care Health Sciences, University of Oxford, Centre for Evidence-Based Medicine, Radcliffe Observatory Quarter, Woodstock Road, Oxford, OX2 6GG UK.
Abstract
OBJECTIVES: To systematically explore the methodological factors underpinning discrepancies in the pooled effect estimates from Cochrane reviews (CRs) and non-Cochrane reviews (NCRs) systematic reviews, answering the same clinical question. STUDY DESIGN AND SETTING: Quantitative and qualitative analysis of concordance in effect estimates between meta-analyses from CR and NCR matched on population, intervention, condition, and outcome. RESULTS: We identified 24 matched meta-analyses from 24 CR to 20 NCR reviews (545 randomized controlled trials [RCTs]). Compared to their CR matched-pair, pooled effects from NCR were the same in only one pair, were on average 0.12 log units (13%) higher (P = 0.012), and had a greater than twofold larger effect size in four matched-pairs. Two-thirds of CR (15/24, 70.8%) and 0/20 (0%) NCR were rated to have moderate to high confidence in their results (AMSTAR 2). Differences in pre-defined methods, including search strategy, eligibility criteria, and performance of dual screening, could explain mismatches in included studies. Disagreements in the interpretation of eligibility criteria were identified as reasons underpinning discrepant findings in 14 pairs. 23/24 meta-analyses included at least one study of its match. Only two pairs agreed on the numerical data presented for the same studies. An assessment of 50% of discrepant studies (n = 45) showed that reasons for differences in extracted data could be identified in 15 studies. CONCLUSION: On average, meta-analyses from NCR reported higher effect estimates compared with meta-analyses from CR answering the same clinical question. Methodological and author judgments and performance are key aspects underpinning poor overlap of included studies and discrepancies in reported effect estimates. The potential impacts on health care policy and clinical practice are far-reaching but still remain unknown. Reinforcing awareness and scrutiny of application of reporting guidelines and improvements in protocol registration are needed.
OBJECTIVES: To systematically explore the methodological factors underpinning discrepancies in the pooled effect estimates from Cochrane reviews (CRs) and non-Cochrane reviews (NCRs) systematic reviews, answering the same clinical question. STUDY DESIGN AND SETTING: Quantitative and qualitative analysis of concordance in effect estimates between meta-analyses from CR and NCR matched on population, intervention, condition, and outcome. RESULTS: We identified 24 matched meta-analyses from 24 CR to 20 NCR reviews (545 randomized controlled trials [RCTs]). Compared to their CR matched-pair, pooled effects from NCR were the same in only one pair, were on average 0.12 log units (13%) higher (P = 0.012), and had a greater than twofold larger effect size in four matched-pairs. Two-thirds of CR (15/24, 70.8%) and 0/20 (0%) NCR were rated to have moderate to high confidence in their results (AMSTAR 2). Differences in pre-defined methods, including search strategy, eligibility criteria, and performance of dual screening, could explain mismatches in included studies. Disagreements in the interpretation of eligibility criteria were identified as reasons underpinning discrepant findings in 14 pairs. 23/24 meta-analyses included at least one study of its match. Only two pairs agreed on the numerical data presented for the same studies. An assessment of 50% of discrepant studies (n = 45) showed that reasons for differences in extracted data could be identified in 15 studies. CONCLUSION: On average, meta-analyses from NCR reported higher effect estimates compared with meta-analyses from CR answering the same clinical question. Methodological and author judgments and performance are key aspects underpinning poor overlap of included studies and discrepancies in reported effect estimates. The potential impacts on health care policy and clinical practice are far-reaching but still remain unknown. Reinforcing awareness and scrutiny of application of reporting guidelines and improvements in protocol registration are needed.
Authors: Esther Vaquero-Álvarez; Antonio Cubero-Atienza; Pilar Ruiz-Martínez; Manuel Vaquero-Abellán; María Dolores Redel Mecías; Pilar Aparicio-Martínez Journal: Int J Environ Res Public Health Date: 2020-09-16 Impact factor: 3.390
Authors: Lara Dotto; Mateus de Azevedo Kinalski; Pablo Soares Machado; Gabriel Kalil Rocha Pereira; Rafael Sarkis-Onofre; Mateus Bertolini Fernandes Dos Santos Journal: J Evid Based Med Date: 2021-02-17
Authors: Matthew J Page; David Moher; Fiona M Fidler; Julian P T Higgins; Sue E Brennan; Neal R Haddaway; Daniel G Hamilton; Raju Kanukula; Sathya Karunananthan; Lara J Maxwell; Steve McDonald; Shinichi Nakagawa; David Nunan; Peter Tugwell; Vivian A Welch; Joanne E McKenzie Journal: Syst Rev Date: 2021-04-16