Peter W Lane1, Julian P T Higgins2, Betsy Anagnostelis3, Judith Anzures-Cabrera4, Nigel F Baker5, Joseph C Cappelleri6, Scott Haughie7, Sally Hollis8, Steff C Lewis9, Patrick Moneuse10, Anne Whitehead11. 1. Statistical Consultancy Group, GlaxoSmithKline R&D, Stevenage, UK. 2. School of Social and Community Medicine, University of Bristol, Bristol, UK. 3. Royal Free Hospital Medical Library, University College London, London, UK. 4. Biostatistics, Roche Products Ltd, Welwyn Garden City, UK. 5. Amgen Ltd, Cambridge, UK. 6. Biostatistics, Pfizer Inc, Groton, Connecticut, USA. 7. Primary Care Business Unit, Pfizer Global R&D, Sandwich, UK. 8. Global Medicines Development, AstraZeneca, Macclesfield, UK. 9. Centre for Population Health Sciences, University of Edinburgh Medical School, Teviot Place, Edinburgh, EH8 9AG, UK. 10. Biometrics, Vifor Pharma, Glattbrugg, Switzerland. 11. Medical and Pharmaceutical Statistics Research Unit, Lancaster University, Lancaster, UK.
Abstract
CONTEXT: Meta-analyses are regularly used to inform healthcare decisions. Concerns have been expressed about the quality of meta-analyses and, in particular, about those supported by the pharmaceutical industry. OBJECTIVE: The objective of this study is to compare the quality of pharmaceutical-industry-supported meta-analyses with academic meta-analyses and of meta-analyses published before and after companies started to disclose their data. DATA SOURCES: We identified industry-supported meta-analyses by searching the Scopus bibliographic database, using author affiliations. We matched each industry-supported meta-analysis with an academic meta-analysis using high-level MeSH terms in PubMed. STUDY SELECTION: We included meta-analyses of randomized trials assessing the efficacy or safety of any pharmaceutical intervention in humans, published in 2002-2004 or 2008-2009. Cochrane reviews were excluded. Two individuals independently selected papers, with discrepancies resolved by two further individuals. ASSESSMENT: We developed and piloted a quality-assessment tool, consisting of 43 questions in four domains, with a key summary question covering each domain. Two individuals independently assessed each meta-analysis. RESULTS: We examined 126 meta-analysis publications in 63 matched pairs. The average quality was low, with fewer than 50% adequate in three of the four domains. Industry-supported meta-analyses less often demonstrated adequate methods for locating studies and assessing their quality (odds ratio 0.44, 95% confidence interval 0.21 to 0.92), for analysing the included studies (0.52, 0.25 to 1.06), for undertaking meta-analyses (0.82, 0.40 to 1.68) and in reaching sound conclusions (0.62, 0.30 to 1.28). Quality generally improved over time, particularly for some aspects of industry reports. CONCLUSIONS: Academic meta-analysis papers are generally of higher quality than industry-supported ones. This is largely due to less detailed reporting in industry-supported meta-analyses and a tendency for them to take the included studies at face value, probably arising from the implicit assumption that these studies already have high methodological standards to meet licensing requirements. The improved quality over time does not appear to be due to the use of data disclosed by industry. The main limitations of this study are the small sample of papers and the subjective nature of some of the assessment processes.
CONTEXT: Meta-analyses are regularly used to inform healthcare decisions. Concerns have been expressed about the quality of meta-analyses and, in particular, about those supported by the pharmaceutical industry. OBJECTIVE: The objective of this study is to compare the quality of pharmaceutical-industry-supported meta-analyses with academic meta-analyses and of meta-analyses published before and after companies started to disclose their data. DATA SOURCES: We identified industry-supported meta-analyses by searching the Scopus bibliographic database, using author affiliations. We matched each industry-supported meta-analysis with an academic meta-analysis using high-level MeSH terms in PubMed. STUDY SELECTION: We included meta-analyses of randomized trials assessing the efficacy or safety of any pharmaceutical intervention in humans, published in 2002-2004 or 2008-2009. Cochrane reviews were excluded. Two individuals independently selected papers, with discrepancies resolved by two further individuals. ASSESSMENT: We developed and piloted a quality-assessment tool, consisting of 43 questions in four domains, with a key summary question covering each domain. Two individuals independently assessed each meta-analysis. RESULTS: We examined 126 meta-analysis publications in 63 matched pairs. The average quality was low, with fewer than 50% adequate in three of the four domains. Industry-supported meta-analyses less often demonstrated adequate methods for locating studies and assessing their quality (odds ratio 0.44, 95% confidence interval 0.21 to 0.92), for analysing the included studies (0.52, 0.25 to 1.06), for undertaking meta-analyses (0.82, 0.40 to 1.68) and in reaching sound conclusions (0.62, 0.30 to 1.28). Quality generally improved over time, particularly for some aspects of industry reports. CONCLUSIONS: Academic meta-analysis papers are generally of higher quality than industry-supported ones. This is largely due to less detailed reporting in industry-supported meta-analyses and a tendency for them to take the included studies at face value, probably arising from the implicit assumption that these studies already have high methodological standards to meet licensing requirements. The improved quality over time does not appear to be due to the use of data disclosed by industry. The main limitations of this study are the small sample of papers and the subjective nature of some of the assessment processes.
Authors: Dawid Pieper; Irma Hellbrecht; Linlu Zhao; Clemens Baur; Georgia Pick; Sarah Schneider; Thomas Harder; Kelsey Young; Andrea C Tricco; Ella Westhaver; Matthew Tunis Journal: Syst Rev Date: 2022-08-22