Ikhlaaq Ahmed1, Alexander J Sutton, Richard D Riley. 1. MRC Midlands Hub for Trials Methodology Research, School of Health and Population Sciences, University of Birmingham, Birmingham B15 2TT, UK.
Abstract
OBJECTIVE: To examine the potential for publication bias, data availability bias, and reviewer selection bias in recently published meta-analyses that use individual participant data and to investigate whether authors of such meta-analyses seemed aware of these issues. DESIGN: In a database of 383 meta-analyses of individual participant data that were published between 1991 and March 2009, we surveyed the 31 most recent meta-analyses of randomised trials that examined whether an intervention was effective. Identification of relevant articles and data extraction was undertaken by one author and checked by another. RESULTS: Only nine (29%) of the 31 meta-analyses included individual participant data from "grey literature" (such as unpublished studies) in their primary meta-analysis, and the potential for publication bias was discussed or investigated in just 10 (32%). Sixteen (52%) of the 31 meta-analyses did not obtain all the individual participant data requested, yet five of these (31%) did not mention this as a potential limitation, and only six (38%) examined how trials without individual participant data might affect the conclusions. In nine (29%) of the meta-analyses reviewer selection bias was a potential issue, as the identification of relevant trials was either not stated or based on a more selective, non-systematic approach. Investigation of four meta-analyses containing data from ≥10 trials revealed one with an asymmetric funnel plot consistent with publication bias, and the inclusion of studies without individual participant data revealed additional heterogeneity between trials. CONCLUSIONS: Publication, availability, and selection biases are a potential concern for meta-analyses of individual participant data, but many reviewers neglect to examine or discuss them. These issues warn against uncritically viewing any meta-analysis that uses individual participant data as the most reliable. Reviewers should seek individual participant data from all studies identified by a systematic review; include, where possible, aggregate data from any studies lacking individual participant data to consider their potential impact; and investigate funnel plot asymmetry in line with recent guidelines.
OBJECTIVE: To examine the potential for publication bias, data availability bias, and reviewer selection bias in recently published meta-analyses that use individual participant data and to investigate whether authors of such meta-analyses seemed aware of these issues. DESIGN: In a database of 383 meta-analyses of individual participant data that were published between 1991 and March 2009, we surveyed the 31 most recent meta-analyses of randomised trials that examined whether an intervention was effective. Identification of relevant articles and data extraction was undertaken by one author and checked by another. RESULTS: Only nine (29%) of the 31 meta-analyses included individual participant data from "grey literature" (such as unpublished studies) in their primary meta-analysis, and the potential for publication bias was discussed or investigated in just 10 (32%). Sixteen (52%) of the 31 meta-analyses did not obtain all the individual participant data requested, yet five of these (31%) did not mention this as a potential limitation, and only six (38%) examined how trials without individual participant data might affect the conclusions. In nine (29%) of the meta-analyses reviewer selection bias was a potential issue, as the identification of relevant trials was either not stated or based on a more selective, non-systematic approach. Investigation of four meta-analyses containing data from ≥10 trials revealed one with an asymmetric funnel plot consistent with publication bias, and the inclusion of studies without individual participant data revealed additional heterogeneity between trials. CONCLUSIONS: Publication, availability, and selection biases are a potential concern for meta-analyses of individual participant data, but many reviewers neglect to examine or discuss them. These issues warn against uncritically viewing any meta-analysis that uses individual participant data as the most reliable. Reviewers should seek individual participant data from all studies identified by a systematic review; include, where possible, aggregate data from any studies lacking individual participant data to consider their potential impact; and investigate funnel plot asymmetry in line with recent guidelines.
Authors: Veronika I Müller; Edna C Cieslik; Angela R Laird; Peter T Fox; Joaquim Radua; David Mataix-Cols; Christopher R Tench; Tal Yarkoni; Thomas E Nichols; Peter E Turkeltaub; Tor D Wager; Simon B Eickhoff Journal: Neurosci Biobehav Rev Date: 2017-11-24 Impact factor: 8.989
Authors: J E Lutomski; N J A van Exel; G I J M Kempen; E P Moll van Charante; W P J den Elzen; A P D Jansen; P F M Krabbe; B Steunenberg; E W Steyerberg; M G M Olde Rikkert; R J F Melis Journal: Qual Life Res Date: 2014-11-08 Impact factor: 4.147
Authors: Rod S Taylor; Sarah Walker; Neil A Smart; Massimo F Piepoli; Fiona C Warren; Oriana Ciani; David Whellan; Christopher O'Connor; Steven J Keteyian; Andrew Coats; Constantinos H Davos; Hasnain M Dalal; Kathleen Dracup; Lorraine S Evangelista; Kate Jolly; Jonathan Myers; Birgitta B Nilsson; Claudio Passino; Miles D Witham; Gloria Y Yeh Journal: J Am Coll Cardiol Date: 2019-04-02 Impact factor: 24.094
Authors: Christine M Parseghian; Kanwal Raghav; Robert A Wolff; Joe Ensor; James Yao; Lee M Ellis; Alda L Tam; Michael J Overman Journal: Clin Cancer Res Date: 2017-07-28 Impact factor: 12.531
Authors: N A Henriksen; E B Deerenberg; L Venclauskas; R H Fortelny; J M Garcia-Alamino; M Miserez; F E Muysoms Journal: Hernia Date: 2017-10-17 Impact factor: 4.739