Factors predicting completion and time to publication of Cochrane reviews.

BACKGROUND: Cochrane reviews are regarded as scientifically rigorous, yet a review's time to publication can be affected by factors such as the statistical significance of the findings. When this happens, misrepresentation of the literature and subsequent inappropriate decisions may result. We aimed to examine the factors associated with the time to publication of Cochrane reviews.
METHODS: Review protocols published in issue 2, 2000, of the Cochrane Database of Systematic Reviews were included in this analysis if the corresponding review was published by issue 1, 2008. We used univariable and multivariate analyses to examine the time from publication of the protocol to publication of the first review and review-related factors predicting the time to publication.
RESULTS: Of 118 eligible protocols published in issue 2, 2000, we identified 93 Cochrane reviews that had been published by January 2008. Of these, 36 (39%) were updates. The median time to publication was 1.63 years (range 0.15-7.31 years). A change in authors between publication of the protocol and publication of the final review was associated with longer time to publication (p = 0.002), whereas updated reviews were associated with shorter time to publication (p = 0.030).
INTERPRETATION: In our study, 79% of the Cochrane protocols were published as a final review, and some Cochrane reviews took over 7 years from publication of the protocol to publication of the final review. Strategies to increase the number of published Cochrane reviews and decrease the time to publication should be considered, such as providing support to reviewers when a change in authorship occurs.

Publication bias occurs when studies with certain characteristics (e.g., statistically significant results) have a greater likelihood of being published,1 and of being published more quickly,2 than studies without those characteristics. We previously examined publication bias through a retrospective cohort study in which we followed Cochrane reviews for approximately 9 years and found that the majority of studies were published (301/372 [81%]).3 In that study, we abstracted data from Cochrane protocols and used the information to determine which factors predicted publication of the corresponding Cochrane reviews. We found that the time to publication was shorter for reviews that were subsequently updated (hazard ratio for no update v. update 1.80, 95% confidence interval [CI] 1.39–2.33) and longer for reviews with 2 published protocols, indicating changes to the review plan (hazard ratio for 1 protocol v. 2 protocols 0.33, 95% CI 0.12–0.90). These data were based on comparing published Cochrane reviews and unpublished Cochrane reviews by examining differences observed in their protocols.

For the study reported here, we aimed to examine review-related factors (e.g., statistically significant results) associated with the time to publication of Cochrane reviews. Since such review factors are not easily obtained from unpublished Cochrane reviews (i.e., this information could be acquired only by contacting the Cochrane review authors), we included only published Cochrane reviews in this study.

Methods

To allow a reasonable time for publication of the review, we first identified all new protocols that were published in issue 2, 2000, of the Cochrane Database of Systematic Reviews. We tracked the status of these protocols by searching the Cochrane Database of Systematic Reviews until January 23, 2008 (issue 1, 2008) and by contacting the corresponding authors or the Cochrane Review Group coordinator. We excluded Cochrane protocols that were split into more than 1 Cochrane review, taken over by another review group, published in the same issue as the final Cochrane review, published after publication of the corresponding review or published for the second or subsequent time in issue 2, 2000.3

We developed and pilot-tested a comprehensive data abstraction form. One of us (ACT) used this form to abstract data from the published Cochrane reviews, such as characteristics of the published review (e.g., number of authors), methodology used (e.g., number of primary outcomes, inclusion of unpublished material, assessment of publication bias) and other factors (e.g., funding, number of updates).

In addition, we classified the results and conclusions using a system reported elsewhere.4 Briefly, we classified the results of each review as negative, not statistically significant (unfavourable toward treatment intervention, p ≥ 0.05); negative, statistically significant (unfavourable toward treatment intervention, p < 0.05); neutral (effect size between 0.95 and 1.05 with the CI crossing 1); positive, not statistically significant (favourable toward treatment intervention, p ≥ 0.05); positive, statistically significant (favourable toward treatment intervention, p < 0.05); and indeterminate (unable to judge because, for example, the Cochrane review listed 10 primary outcomes, which had different results). The conclusions were classified as positive (authors stated that there is evidence of effectiveness), neutral (authors stated that there is no evidence of effectiveness or they reported no opinion), negative (authors advised against use of the intervention or it was not recommended) or indeterminate (authors stated that there is insufficient evidence or that more research is required).

We first conducted univariable analysis for the published Cochrane reviews by linear regression. The data was skewed and did not follow a normal distribution. Therefore, we used the log transformation of the time between the “most recent substantive amendment date” of the protocol and the published review to analyze the time variable.5 The characteristics reported in Table 1 were analyzed using linear regression. They were entered into a univariable analysis, and those that were statistically significant as well as those that we defined a priori as being important (i.e., results classified as favourable v. unfavourable and all others v. unfavourable; conclusions classified as negative v. positive and all others v. positive; funding; number of authors) were included in the multivariate linear regression analysis. Since only published reviews were included, none of the data were censored (i.e., losses to the sample that occur prior to the final outcome) or missing. Therefore, we did not perform a Cox proportional hazards model, which is commonly used to estimate time-related events and takes into account censored data. Statistical analyses were conducted with SAS, version 9.0 (SAS Institute, Cary, NC).

Table 1

Cochrane review characteristics

Results

A total of 130 new Cochrane protocols appeared in issue 2, 2000, of the Cochrane Database of Systematic Reviews. Twelve protocols were excluded because they did not meet the inclusion criteria, and 93 (79%) of the remaining 118 protocols were published as Cochrane reviews and were included in this study, including 1 withdrawn review (Fig. 1).

Figure 1

Study selection

A primary outcome was reported in 75 (81%) of the 93 included reviews. More than half of the reviews included studies published in any language (49/93 [53%]), and nearly three-quarters included both published and unpublished material (69/93 [74%]) (Table 1). Publication bias was found to affect the results in only 1 (11%) of the 9 Cochrane reviews that assessed the publication bias of included studies. Of the 54 Cochrane reviews in which a meta-analysis of the primary outcome was performed, the largest category for classification of results was for studies with statistically significant positive results (17/54 [31%]), whereas the largest category for classification of conclusion statements was for studies with indeterminate conclusions (40/93 [43%]). Only 36 (39%) of the 93 Cochrane reviews were subsequently updated, and 74 (80%) reported a funding source.

The median time to publication was 1.63 years (range 0.15–7.31 years). Four factors were significant predictors of publication, by univariable analysis: presence of 2 protocols, subsequent updating of the review, change in authorship between publication of the Cochrane protocol and publication of the review, and number of included studies (all p < 0.05). As noted in the Methods section, we included an additional 4 factors in the multivariate analyses on the basis of a priori consideration. Of these 8 factors, a change in authorship between publication of the protocol and publication of the review was associated with longer time to publication (p = 0.002), and updating of the review was associated with shorter time to publication (p = 0.030) (Table 2).

Table 2

Factors associated with the time to publication of Cochrane reviews, as determined by multivariate analysis

Interpretation

Our results indicate that 79% of the Cochrane protocols were published as a final review and a change in authorship between publication of the Cochrane protocol and publication of the final review was associated with longer times to publication. Since the Cochrane Collaboration is internationally recognized for how it conducts systematic reviews, it would be beneficial if all of the published protocols were also published as reviews. The Cochrane Collaboration should therefore consider strategies to improve the publication rate of Cochrane reviews and to decrease their time to publication.

Our results are inconsistent with research examining publication bias of individual studies (e.g., trials).2,6-10 In those studies, statistically significant results and funding were associated with publication. Given that our study did not find such associations, it appears that publication patterns differ between Cochrane reviews and trials. This conclusion is consistent with other research on publication bias of systematic reviews. In a survey of authors of published systematic reviews, participants reported that non-statistically significant results were not a major reason why their systematic reviews were unpublished; however, 65% (195/300) also reported that informative or statistically significant results were the “most significant” or a “significant” facilitator to publishing their reviews.11 Furthermore, the funding source was not associated with the time to publication of Cochrane reviews in our previous retrospective cohort study.3 However, our analysis of statistically significant results in the current study was based on only a little over half of the sample, since a meta-analysis had been conducted for only 54 (58%) of the 93 Cochrane reviews.

This study, like our previous one on a similar topic,3 had some limitations. A single investigator abstracted all of the data, which might have led to inaccuracies. Assessing the primary outcome, results, and conclusions of the Cochrane reviews was often difficult, and there may have been errors in these assessments.4

In conclusion, 79% of the Cochrane protocols were published as a final review, and some Cochrane reviews took over 7 years from publication of the protocol to publication of the final review. Strategies to increase the number of published Cochrane reviews and decrease the time to publication should be considered. Such strategies might include providing additional support to reviewers when a change in authorship occurs.