Search for unpublished data by systematic reviewers: an audit.

OBJECTIVES: We audited a selection of systematic reviews published in 2013 and reported on the proportion of reviews that researched for unpublished data, included unpublished data in analysis and assessed for publication bias.
DESIGN: Audit of systematic reviews. DATA SOURCES: We searched PubMed and Ovid MEDLINE In-Process & Other Non-Indexed Citations between 1 January 2013 and 31 December 2013 for the following journals: Journal of the American Medical Association, The British Medical Journal, Lancet, Annals of Internal Medicine and the Cochrane Database of Systematic Reviews. We also searched the Cochrane Library and included 100 randomly selected Cochrane reviews. ELIGIBILITY CRITERIA: Systematic reviews published in 2013 in the selected journals were included. Methodological reviews were excluded. DATA EXTRACTION AND SYNTHESIS: Two reviewers independently reviewed each included systematic review. The following data were extracted: whether the review searched for grey literature or unpublished data, the sources searched, whether unpublished data were included in analysis, whether publication bias was assessed and whether there was evidence of publication bias. MAIN
FINDINGS: 203 reviews were included for analysis. 36% (73/203) of studies did not describe any attempt to obtain unpublished studies or to search grey literature. 89% (116/130) of studies that sought unpublished data found them. 33% (68/203) of studies included an assessment of publication bias, and 40% (27/68) of these found evidence of publication bias.
CONCLUSION: A significant fraction of systematic reviews included in our study did not search for unpublished data. Publication bias may be present in almost half the published systematic reviews that assessed for it. Exclusion of unpublished data may lead to biased estimates of efficacy or safety in systematic reviews. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

To our knowledge, this is the first study to report the proportion of systematic reviews that searched for and included unpublished data from multiple sources including registries.

Two reviewers independently reviewed each included review to minimise bias.

Only reports of trials published in selected high-impact journals were reviewed, and this may reduce the external validity of our results.

Introduction

Readers of systematic reviews should know whether unpublished data was sought as the conclusions of the review may depend on studies that have not been published. Studies with positive results have a three times higher odds of being published than those with negative or null results.1 The results of studies in favour of a new treatment also have a higher chance of being published than negative studies.2 3 Published studies are likely to have larger treatment effects than unpublished studies.4–6 An analysis of 42 meta-analyses on drug trials found that 92% of the efficacy estimates were altered when unpublished studies that were submitted to regulatory agencies were included.7 For example, agomelatine was reported to be an effective depression treatment by several systematic reviews of only published studies,8 9 but no treatment effect was demonstrated when seven unpublished studies were included.10 11 Other examples of publication bias include oseltamivir for influenza in adults,12 statins for the prevention of venous thromboembolic events,13 quinine for nocturnal leg cramps14 and reboxetine for depression.15

The Cochrane Handbook for Systematic Reviews of Interventions states that ‘the convincing evidence for the presence of several types of reporting biases demonstrates the need to search comprehensively for studies that meet the eligibility criteria for a Cochrane review’.16 Several studies have demonstrated that widely agreed on reporting standards are not always followed. Following the publication of reporting guidelines, the reporting of randomised control trials (RCTs)17 and abstracts18 remained poor. Audits of adherence to reporting standards might have improved the reporting of RCTs.19 Just as changes to reporting guidelines for RCTs improved RCT reporting, changes to reporting guidelines for systematic reviews may improve systematic review reporting. A recent editorial and other published guidance suggest that data should routinely be sought from regulatory agencies and trial registries.20 21

We performed an audit of systematic reviews of healthcare interventions published in 2013 to determine the proportion of reviews that reported searching for unpublished data. We also determined how many systematic reviews included unpublished data in the analysis and how many assessed for publication bias.

Methods

Eligibility criteria

Ethics approval was not required as the study was based on published systematic reviews. We included all full-text systematic reviews (with prespecified methods) that were published in 2013 in Journal of the American Medical Association (JAMA), British Medical Journal (BMJ), Lancet and Annals of Internal Medicine, as well as a random subset of 100 reviews published in the Cochrane Database of Systematic Reviews. Methodological reviews were excluded.

Search strategy

PubMed and Ovid MEDLINE In-Process & Other Non-Indexed Citations database were used to identify all published systematic reviews between 1 January 2013 and 31 December 2013 for the following journals: JAMA, BMJ, Lancet and Annals of Internal Medicine. The search for each source included the journal name, ‘systematic review’ and the publication date (eg, the search strategy used for JAMA in PubMed was: ‘JAMA’[journal] AND ‘systematic review’[title] AND‘2013’[pdat]). The journals were also manually searched to identify potential relevant papers that were missing from the electronic searches.

The Cochrane Library was also searched to identify literature published in the Cochrane Database of Systematic Reviews not retrieved by the previous electronic searches. The search was limited to the ‘Cochrane Database of Systematic Reviews’, the publication date was set to ‘2013 to 2013’ and the product type was identified as ‘Review’. A random number sequence generator was used to select and include 100 reviews from the Cochrane Database of Systematic Reviews. Based on our piloting of the data abstraction instrument, we believed that 100 reviews would be sufficient to determine the common approaches to unpublished data. The date of the last search for all databases was 11 February 2014.

Data extraction

Unpublished data included complete trials that have never been published as well as specific outcomes that are not reported in published trials. For this study, we considered data appearing in conference proceedings, research reports and dissertations as part of the grey literature, and these were included as a type of unpublished data.

From each systematic review included in the analysis, the following general characteristics were collected: journal name, title, first author and the date of publication. The following main outcomes were extracted: whether there was any search for unpublished data as described in the methods or evidence of unpublished data inclusion, whether unpublished data were used in meta-analysis, whether the interventions of review were pharmacological in nature and the results of any assessment of publication bias.

Two authors (HZ, NP or RZ) independently extracted data from the Abstract, Methods and Results sections using a standardised electronic form. Disagreements were discussed until consensus was achieved or resolved by a third individual. The following information was collected from the methods sections of each report: databases searched, electronic search strategy, other sources searched (conference abstracts, unpublished studies, ongoing studies, contact with study authors/experts and so on), any restrictions to publication status, language, whether publication bias was assessed and methods used to assess publication bias. For the results section, we reviewed any analysis of unpublished data and assessment of publication bias. For the discussion section, we assessed whether the authors commented on the presence of publication bias. Additionally, the following terms were searched electronically in the article text to ensure inclusion of all relevant information: ‘unpublish’, ‘publication’, ‘bias’, ‘funnel’, ‘Hegg’, ‘Egger’, ‘gray’, ‘grey’ and ‘reporting’.

Data synthesis and analysis

The primary purpose of this study was to determine the proportion of systematic reviews that reported searching for unpublished data. The secondary objective was to determine the proportion of systematic reviews that found unpublished data and included them in the meta-analysis. Pearson χ2 tests were completed to assess for any difference between systematic reviews of pharmacological versus other interventions, and for any difference between reviews published in the Cochrane Database of Systematic Reviews versus the other journals.

Results

We identified 104 systematic reviews published in JAMA (n=13), BMJ (n=40), Lancet (n=10) and Annals of Internal Medicine (n=41). The Cochrane Library search strategy yielded 1090 results, and subsequently 100 articles were randomly selected. Among the 204 systematic reviews reviewed in full text, one review was excluded as a methodological review. Our final cohort consisted of 203 reviews. Seventy-one reviews (35%) were pharmaceutical reviews.

Search for unpublished data

Of the 203 included systematic reviews, 73 (36%) did not describe in the methods whether there was any search for unpublished or grey literature data and 130 (64%) described some search for unpublished data. Overall, 42% (86/203) of reviews described searching for unpublished completed studies (with or without other types of unpublished data) in their search strategy and 22% (44/203) of reviews described searching the grey literature without specifically describing searching for unpublished studies. No reviews described only searching for unreported data from published studies included in the review in their methods.

Table 1 summarises the number of reviews that searched each source according to what was reported in the entire report including the results and discussion in addition to the methods section. National and international trial registries were the most significant source of unpublished data collection, followed by data received from contacting individuals or organisation and conference proceedings. Specifically, 42% (85/203) searched national and international trials registers, 41% (84/203) of the reviews contacted individuals or organisations (eg, sponsors or research organisations), 35% (71/203) searched conference proceedings, 8% (16/203) searched pharmaceutical industry trials registers and 2% (4/203) searched subject-specific trials registers. No review included a search for data from regulators via their websites or information requests.

Table 1

Sources of data and yield of searches in systematic reviews. Types of unpublished data are in rows, and sources of unpublished data are in columns

Type of unpublished data	National and international trial registries (n=85)	Subject-specific trial registries (n=4)	Pharmaceutical industry trials registries (n=16)	Conference proceedings (n=71)	Contacting individuals or organisations (n=84)
Unpublished completed studies	55	4	16	46	53
Grey literature only	22	0	0	25	21
Ongoing studies only	6	0	0	0	2
Unreported data from published studies only	2	0	0	0	8
Searches that yielded results (%)	75 (88)	3 (75)	15 (94)	63 (89)	76 (90)

Some reviews identified types of unpublished data that were not described in the search strategy in the Methods section (ie, they included unpublished data in the results section without describing the methodology of searching for unpublished data in the methods). Among the 130 reviews that described searching sources for unpublished data, 55 of them (43%) described only searching one source, 37 (28%) reviews described searching two sources and the remaining 38 (29%) reviews described searching three or more sources. No review described searching all sources for unpublished data. Furthermore, some included systematic reviews did not specify the source of unpublished data.

In a prespecified analysis of factors associated with performing a search for unpublished studies, we did not find any difference between reviews of pharmacological and non-pharmacological interventions. However, reviews published in the Cochrane Database of Systematic Reviews were significantly more likely to search for unpublished studies than those published in standard journals (p<0.0001).

The unpublished data discovered by systematic reviews were often not included in the results and analyses. Among the 130 reviews that included a search for unpublished or grey literature data, 89% (n=116) found such data. Of the 116 reviews that searched and found unpublished data, 46 reviews (40%) both identified and included unpublished data. The remaining 70 reviews did not include the supplementary data. Among the 46 studies that included unpublished data, 23 (50%) included data from unpublished studies. Twenty reviews (43%) included unpublished data from published studies and 7% (3/46) included data from abstracts of published papers only.

Assessment of publication bias

Thirty-three per cent (68/203) of the reviews included an assessment of publication bias. An additional 27% (55/203) of the reviews planned a publication bias assessment, but these analyses were not reported for a number of reasons: 10 did not find any eligible studies, 42 had insufficient quantity (<10) of selected studies or other reasons and three did not provide an explanation. Thirty-nine per cent (80/203) of the reviews did not describe an intent to assess publication bias.

Of the 68 systematic reviews that assessed publication bias, 58 reviews performed statistical or graphical analysis, and 10 reviews employed other methods such as a discussion of the likelihood that highly statistically significant results could be explained by publication bias. Thirty-four per cent (20/58) of the non-qualitative assessments and 70% (7/10) of the qualitative assessments were significant for publication bias (table 2). There was a trend towards systematic reviews that searched for unpublished data being less likely to indicate publication bias.

Table 2

Results of assessment for publication bias

	Reviews that searched for unpublished data (n=49)	Reviews that did not search for unpublished data (n=19)	All reviews (n=68)
Statistical or graphical analysis (n (%))
Indication of publication bias	11 (27)	9 (53)	20 (33)
No indication of publication bias	30 (73)	8 (47)	38 (51)
Qualitative analysis (n (%))
Indication of publication bias	5 (63)	2 (100)	7 (11)
No indication of publication bias	3 (38)	0	3 (5)

Discussion

Among the 203 systematic reviews published in high-impact general medical journals in 2013, 36% did not describe any attempt to search for unpublished studies, although guidelines recommend searching for unpublished data. Of the 116 reviews that completed a search and found unpublished data or grey literature, 40% included unpublished data for analysis. Thirty-three per cent of the reviews included a publication bias assessment, and 40% of these reviews revealed evidence of publication bias. The quantitative and/or qualitative suggestion of publication bias was more prevalent in reviews that did not search for unpublished data, when compared with those reviews that searched for unpublished data.

To our knowledge, this is the largest study that has investigated the proportion of systematic reviews that searched for and included unpublished data. A 2017 audit found that 52% of selected systematic reviews did not report a search of trial registries.22 A cross-sectional survey of corresponding authors of Cochrane reviews with 37% response rate found that 76% of Cochrane respondents reported searching for unpublished data, 82% of unpublished data were used in analysis and that the most common source of unpublished data was from contacting the study investigators.23 Over 10% of Cochrane reviews from 2000 to 2006 included unpublished studies.24

The results of this study highlight inadequacies in identifying unpublished data in systematic reviews. Cochrane reviews might have been more likely to search for unpublished data because of guidance in the Cochrane handbook as well as rigorous protocol review and editorial practices. Clear standards regarding the search for and inclusion of unpublished studies may help. There is some evidence that unpublished data are sometimes misleading25 and that unpublished data may not contain sufficient information to assess methodological quality and in turn may be unreliable. Conference abstracts may not always be reliable; 40%–60% may not have reported the main outcome results in the same way as the final published study.25–30 Further studies will be needed to determine whether including unpublished data improves systematic reviews as their unpublished data may be unreliable.

Current reporting guidelines for systematic reviews such as Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) require only specification of the data sources searched, but they do not recommend describing whether a search for unpublished data was performed.31 One of the 27 items is on the description of information sources, but the checklist does not mention unpublished data. The PRISMA statement explanation and elaboration indicates that describing the search for unpublished information is ‘useful’ but not mandatory. Specifically, they indicate: ‘Authors should also report if they attempted to acquire any missing information (such as on study methods or results) from investigators or sponsors; it is useful to describe briefly who was contacted and what unpublished information was obtained’.31 Two systematic review quality assessment tools, A Measurement Tool to Assess Systematic Reviews and the Risk of Bias in Systematic Reviews (ROBIS), explicitly include the search for unpublished data.32 33

The reporting of the search for unpublished data in systematic reviews may be improved by its explicit inclusion as a recommendation in the PRISMA reporting guideline. Similarly, the Consolidated Standards of Reporting Trials (CONSORT) statement was developed in response to concerns about the quality of RCTs.34 35 Evaluations of its effectiveness have suggested that journal adoption of CONSORT is associated with improved quality of reporting of RCTs.17 36–38 Furthermore, the inclusion of unpublished studies in systematic reviews might lead to those primary studies eventually being published through initiatives such as the Restoration Invisible and Abandoned Trials process.26

To reduce potential reporting bias, it is important to search for unpublished data using several sources: subject matter experts, investigators, commercial sponsors, trial registries, regulatory agency documents and conference proceedings.21 Searching for trial protocols can help to better understand the study methods and identify selective reporting of outcomes within published studies.1 If there are concerns about the quality of data in any included study, whether it is published or not, a sensitivity analysis should be conducted to determine the effect of the suspect data.

Limitations

There are some limitations to this study. Only reports of trials published in the JAMA, BMJ, Lancet, Annals of Internal Medicine and the Cochrane Database of Systematic Reviews were reviewed. Systematic reviews published in lower impact factor journals were not included. We also randomly included 100 reviews from a group of 1090 reviews in the Cochrane Library. Therefore, the findings might not be representative of all published systematic reviews in 2013. We report the searches as described in the methods section of the publications, and the actual searches may have been different. We found that two systematic reviews obtained unreported data regarding published studies from registries, although they did not describe searching these registries. Time-lag bias that may have overestimated the under-reporting of unpublished data, although time-lag bias cannot explain the poor descriptions of the search strategy.

Conclusion

The search for unpublished data in systematic reviews is still suboptimal. A significant number of systematic reviews published in 2013 did not search for grey literature and unpublished data. Inadequate reporting of unpublished data and grey literature can lead to reporting bias. Almost half of the included reviews that assessed for publication bias suggested the presence of publication bias. Improving reporting guidelines for systematic reviews and better adherence to reporting guidelines may help address this issue.