Impact of study outcome on submission and acceptance metrics for peer reviewed medical journals: six year retrospective review of all completed GlaxoSmithKline human drug research studies.

Objectives To determine whether the outcome of drug studies influenced submission and/or acceptance rates for publication in peer reviewed medical journals.Design A six year retrospective review of publication status by study outcome for all human drug research studies conducted by a single industry sponsor (GlaxoSmithKline) that completed from 1 January 2009 to 30 June 2014 and were therefore due for manuscript submission (per the sponsor's policy) to peer reviewed journals within 18 months of study completion-that is, 31 December 2015. In addition, manuscripts from studies completing after 30 June 2014 were included irrespective of outcome if they were submitted before 31 December 2015.Setting Studies conducted by a single industry sponsor (GlaxoSmithKline)Studies reviewed 1064 human drug research studies.Main outcome measures All studies were assigned a publication status at 26 February 2016 including (as applicable): study completion date, date of first primary manuscript submission, number of submissions, journal decision(s), and publication date. All studies were also classified with assessors blinded to publication status as "positive" (perceived favorable outcome for the drug under study), "negative" (perceived unfavorable outcome for the drug under study), mixed, or non-comparative based on the presence and outcome of the primary outcome measure(s) for each study. "Negative" studies included safety studies in which the primary outcome was achieved but was adverse for the drug under study. For the total cohort and each of the four study outcomes, measures included descriptive statistics for study phase, time from study completion to submission and publication, and number and outcome (accepted/rejected) of publication submissions.Results Of the 1064 studies (phase I-IV, interventional and non-interventional) included, 321 had study outcomes classified as positive, 155 as negative, 52 as mixed, and 536 as non-comparative. At the time of publication cut-off date (26 February 2016), 904 (85%) studies had been submitted for publication as full manuscripts and 751 (71%) had been successfully published or accepted, with 100 (9%) still under journal review. An additional 77 (7%) studies were conference abstracts and were not included in submission or publication rates. Submission rates by study outcome were 79% for the 321 studies with positive outcomes, 92% for the 155 with negative outcomes, 94% for the 52 with mixed outcomes, and 85% for the 536 non-comparative studies; while rates of publication at the cut-off date were 66%, 77%, 77%, and 71%, respectively. Median time from study completion to submission was 537 days (interquartile range 396-638 days) and 823 days (650-1063 days) from completion to publication, with similar times observed across study outcomes. First time acceptance rates were 56% for studies with positive outcomes and 48% for studies with negative outcomes. Over 10% of studies across all categories required three or more submissions to achieve successful publication. At the time of analysis, 83 studies had not been submitted for publication, including 49 bioequivalence studies with positive outcomes and 33 non-comparative studies. Most studies (98%, 1041/1064) had results posted to one or more public registers, including all studies subject to FDAAA (Food and Drug Administration Amendments Act) requirements for posting to www.clinicaltrials.govConclusions Over the period studied, there was no evidence of submission or publication bias: 92% of studies with negative outcomes were submitted for publication by the cut-off date versus 79% of those with positive outcomes. Publication rates were slightly higher for studies with a negative (that is, unfavorable) outcome compared with a positive outcome, despite a slightly lower rate of acceptance at first submission. Many studies required multiple submission attempts before they were accepted for publication. Analyses focusing solely on publication rates do not take into account unsuccessful efforts to publish. Sponsors and journal editors should share similar information to contribute to better understanding of issues and barriers to full transparency. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

Introduction

Industry and, to a lesser extent, academia have been criticized for failing to submit clinical trial data for publication, especially when the results are perceived to be “negative”—that is, unfavorable to the drug under study—leading to publication bias. Despite a commitment by many industry sponsors to publish all research results, failure to publish regardless of outcome adversely impacts the credibility of all industry sponsored research.1 2

Transparent reporting of medical research irrespective of outcome fulfills an ethical obligation to the patients taking part in a trial, advances scientific understanding, and could inform treatment decisions. Although public posting of summary results is now a legal requirement for many types of studies, publication in peer reviewed journals is still considered the ideal form of disclosure as it provides critical context that aids interpretation. Literature surveys carried out since 2010 suggest that 56-85% of trials with study protocols registered on www.clinicaltrials.gov and other public sites were eventually published.3 4 5 6 7 8 9 There has been wide variance across and within sponsor categories (such as industry, academia, government) and by study outcomes (presence versus absence of significance) in both publication rates as well as time to publication, with a general temporal trend toward increased disclosure rates.3 4 6 7 8 9 10 11 12 Studies lacking statistically significant outcomes were less likely to be published and, when they were published, took a longer time from study completion to publication.10 11

Most literaturesurveys rely on information on public websites (such as www.clinicaltrials.gov) that lack the information necessary to take into account the effect of unsuccessful attempts to publish (that is, journal rejection) on publication rates and time to publish. We analyzed study and publication outcomes using a cohort of all drug trials completed from January 2009 to June 2014 from a single pharmaceutical sponsor, GlaxoSmithKline (GSK). Since 2009, GSK policy requires that all human research studies of its drug products (whether investigational or marketed) are submitted for journal publication within 18 months of study completion unless exempted. For this analysis, we classified all studies completing during this period (including those not submitted or published) by outcome and hypothesized that there would be no difference in submission rates by study outcomes (per sponsor policy) but that studies with negative outcomes would have lower acceptance rates and require a greater number of submission attempts to be published in peer reviewed medical journals.

Methods

From the sponsor’s clinical trial management system, we compiled a comprehensive list of all human drug research studies sponsored by GSK Pharmaceuticals that completed (last participant’s final study visit, or completion of statistical analysis for observational studies) from 1 January 2009 to 30 June 2014. We also extracted and cross checked other study characteristics (study type, phase, end date) against www.clinicaltrials.gov and www.gsk-clinicalstudyregister.com. We selected 1 January 2009 as the start as this was the date from which GSK policy required all human drug research studies to be submitted for publication within 18 months of study completion. We selected 30 June 2014 as the cut off to ensure capture of the cohort of studies expected by this policy to have been submitted by 31 December 2015. To capture all manuscript submissions from 2010-15, we also included studies that completed after 30 June 2014, irrespective of outcome, if they had a manuscript submitted by 31 December 2015. We excluded studies of vaccines or consumer products, with no drug administration, with a change in sponsor, or with termination before completion of sufficient patient enrollment to conduct the primary statistical analyses, and studies submitted for publication outside 2010-15 (see fig A in appendix).

Using study results summaries posted to www.clinicaltrials.gov and/or www.gsk-clinicalstudyregister.com, we classified study outcomes into the following categories:

Positive: significant difference (P<0.05 or non-overlapping 95% confidence intervals) on the primary outcome measure prespecified in the protocol in favor of the experimental drug or, for safety studies, a lack of adverse safety findings (such as corrected QT studies/non-inferiority) or all formulations tested were within bioequivalence/non-inferiority limits. A subcategory of “pure positive” studies was also tracked for those studies meeting only the first criterion listed above

Negative: lack of significant difference (as defined above) on primary outcome measure prespecified in the protocol or appearance of an adverse safety finding or lack of bioequivalence. A subcategory of “pure negative” studies was also tracked for those studies meeting only the first criterion listed above

Mixed: both significant and non-significant results on studies with more than one primary endpoint prespecified in the protocol

Non-comparative: studies that did not meet the above criteria, including those with only descriptive statistics—that is, no preplanned formal comparisons or prospective identification of a primary endpoint (typically phase I and pharmacokinetic studies)—or studies that were terminated early without the planned statistical analysis of the primary outcome.

In classifying study outcomes we considered only the outcomes of the primary endpoint and did not consider secondary endpoints. Outcomes were assessed by an external party (Tata Consultancy Services medical writing team) and verified by one of the authors (GE), both of whom were blinded to publication status. Concordance of initial assessments was >80% with discrepancies resolved by consensus discussion.

Publication status was determined from a snapshot review of information within the sponsor’s publication tracking system as of 26 February 2016, which provided the following information for studies included in this cohort: number of submissions, outcome of each submission, and dates of submission and publication (when available). We included only submissions of full manuscripts to peer reviewed journals in the main analysis. Posting of study outcomes to public registries such as www.clinicaltrials.gov or www.gsk-clinicalstudyregister.com did not qualify as submission for publication. Disclosure through conference abstracts was also tracked but not included in the main analysis. Studies were grouped into published (including those accepted but awaiting publication), those currently under journal review, and those not published (either rejected or not submitted). The number of submissions for each study was also recorded. Resubmission to the same journal was not counted as a separate submission, but resubmission to a separate journal within or outside of a given publisher’s journal “family” was counted. Although more than one study could be combined into a single publication submission, for the purpose of this analysis we considered each study as a separate attempt to publish, given the interest in the comparison of study outcomes to publication status.

After separate compilation of study outcomes and publication status, we merged the data and generated descriptive statistics for publication status and number of submission attempts. No other formal statistical comparisons were planned but we used Fisher’s exact test post hoc for proportion of submissions, acceptance, and first time acceptance for studies with positive versus negative outcomes.

Patient involvement

No patients were involved in setting the research question or the outcome measures, nor were they involved in developing plans for implementation of the study. No patients were asked to advise on interpretation or writing up of results. There are no plans to disseminate the results of the research to study participants or the relevant patient community.

Results

The search identified 1003 studies that completed from 1 January 2009 and 30 June 2014 and were therefore expected to have a primary manuscript submitted within 18 months of study completion (that is, by the end of 2015) and an additional 61 more recently completed studies that had a manuscript submitted by the end of 2015, making a total of 1064 studies in the cohort. These studies were then classified according to study outcome (table 1); 45% (n=476) of all studies and 50% (386/779) of interventional studies could be classified as having positive or negative outcomes, including 69% of all phase IIb and phase III studies. In contrast, 65% (n=250/385) of phase I studies were classified as non-comparative and comprised nearly half of the studies within this outcome category (see table A in appendix for additional information on characteristics of non-comparative studies).

Table 1

Overall study outcome (positive, negative, mixed, non-comparative) of human drug studies by study phase and type, with percentages based on proportion of study outcomes. Figures are numbers of studies; percentage (95% CI)

	All studies	No of studies; % (95% CI)
		Positive	Negative	Mixed	Non-comparative
Study phase:
I	385	87; 23 (19 to 27)	37; 10 (7 to 13)	11; 3 (2 to 5)	250; 65 (60 to 70)
II/IIa	121	30; 25 (18 to 33)	36; 30 (22 to 38)	10; 8 (5 to 16)	45; 37 (29 to 46)
IIb	57	20; 35 (24 to 48)	21; 37 (26 to 50)	3; 5 (2 to 14)	13; 23 (14 to 35)
III/IIIa	113	57; 50 (41 to 59)	14; 12 (8 to 20)	6; 5 (2 to 11)	36; 32 (24 to 41)
IIIb	59	30; 51 (38 to 63)	16; 27 (17 to 40)	2; 3 (1 to 12)	11; 19 (11 to 30)
IV	125	46; 37 (29 to 46)	19; 15 (10 to 23)	9; 7 (4 to 13)	51; 41 (33 to 50)
NA	204	51; 25 (20 to 31)	12; 6 (3 to 10)	11; 5 (3 to 9)	130; 64 (57 to 70)
Study type:
Interventional	779	246; 32 (28 to 35)	140; 18 (15 to 21)	31; 4 (3 to 6)	362; 46 (43 to 50)
Non-interventional*	285	75; 26 (22 to 32)	15; 5 (3 to 9)	21; 7 (5 to 11)	174; 61 (55 to 67)
Total	1064	321; 30 (27 to 33)	155; 15 (13 to 17)	52; 5 (4 to 6)	536; 50 (47 to 53)

NA=not applicable.

*Includes observational studies.

Table 2 summarizes publication status by study phase: 904 (85%) of all studies had been submitted for publication as full manuscripts, 751 (71%) had been accepted and/or published as full manuscripts, and 100 (9%) were submitted but still awaiting a journal decision. Some 133 studies were combined into a total of 65 submissions to increase their scientific interest, and the 771 remaining were submitted as standalone publications. Full manuscript publication rates were highest for phase II and III studies (78-88%), whereas phase I studies had the lowest publication rate (57%, 219/385), though an additional 18% (69) were disclosed via conference abstracts.

Table 2

Overall publication status of human drug studies by study phase, with percentages based on proportion of submissions within study phase. Figures are number of studies; percentage (95% CI)

	All studies	Phase I	Phase II/IIA	Phase IIB	Phase III/IIIA	Phase IIIB	Phase IV	Phase NA
Submitted:
Published*	751; 71 (68 to 73)	219; 57 (52 to 62)	95; 79 (70 to 85)	45; 79 (67 to 88)	100; 88 (81 to 93)	46; 78 (66 to 87)	93; 74 (66 to 81)	153; 75 (69 to 80)
Pending	100; 9 (8 to 11)	28; 7 (5 to 10)	11; 9 (5 to 16)	7; 12 (6 to 23)	9; 8 (4 to 14)	10; 17 (9 to 28)	14; 11 (7 to 18)	21; 10 (7 to 15)
Rejected	53; 5 (4 to 6)	9; 2 (1 to 4)	9; 7 (4 to 14)	2; 4 (1 to 12)	1; 1 (0 to 5)	2; 3 (1 to 12)	10; 8 (4 to 14)	20; 10 (6 to 15)
Abstract	77; 7 (6 to 9)	69; 18 (14 to 22)	2; 2 (0 to 6)	0; 0 (0 to 6)	2; 2 (0 to 6)	1; 2 (0 to 9)	2; 2 (0 to 6)	1; <1 (0 to 3)
Not submitted	83; 8 (6 to 10)	60; 16 (12 to 20)	4; 3 (1 to 8)	3; 5 (2 to 14)	1; 1 (0 to 5)	0; 0 (0 to 6)	6; 5 (2 to 10)	9; 4 (2 to 8)
Total	1064	385	121	57	113	59	125	204

NA=not applicable.

*Includes 48 studies accepted but not yet published as of data cut-off date.

Of the 83studies that were not submitted for publication in any form at the time of analysis cut off, 49 were positive bioequivalence studies that showed no differences between formulations; 24 did not examine the safety or efficacy of a drug; nine were terminated early such that the primary outcome measure was not analyzed; and one was excluded because of confounding by indication. Of these, 100% (81/81) that were subject to FDAAA (Food and Drug Administration Amendments Act) or GSK requirements for public disclosure had results posted to www.clinicaltrials.gov or www.gsk-clinicalstudyregister.com (for the full cohort of 1064 studies, 1041 had results posted).

Table 3 summarizes publication status by study outcome. Rates of publication were similar across all categories of study outcome: 66-77% of all categories were published as full manuscripts by the cut-off date with a further 5-8% disclosed as abstracts only. Submission rates were lower for studies with positive outcomes than for studies with negative outcomes (79% v 92%, Fisher’s exact P<0.001), as were acceptance rates (66% v 77%, P=0.019). Rates of non-publication were similar across all categories of study outcome, whether because of journal rejections (3-10%) or lack of manuscript submission during the analysis period (6-16%). Overall, the median time from study completion to submission was 537 days (interquartile range 396-638 days) and from completion to publication was 823 days (650-1063 days). These times were broadly similar across study outcomes, though for studies with negative outcomes, median times to submission and publication were 31 and 102 days longer, respectively, than for studies with positive outcomes.

Table 3

Overall publication status of human drug studies by study outcome. Figures are number of studies; percentage based on proportion of study outcome with given publication status (95% CI), unless stated otherwise

	All studies	Positive	Negative	Mixed	Non-comparative
Submitted:
Published*	751; 71 (68 to 73)	212; 66 (61 to 71)	119; 77 (70 to 83)	40; 77 (64 to 86)	380; 71 (67 to 75)
Pending	100; 9 (8 to 11)	32; 10 (7 to 14)	13; 8 (5 to 14)	4; 8 (3 to 18)	51; 10 (7 to 12)
Rejected	53; 5 (4 to 6)	11; 3 (2 to 6)	10; 6 (4 to 11)	5; 10 (4 to 21)	27; 5 (3 to 7)
Abstract only	77; 7 (6 to 9)	16; 5 (3 to 8)	13; 8 (5 to 14)	3; 6 (2 to 16)	45; 8 (6 to 11)
Not submitted	83; 8 (6 to 10)	50; 16 (12 to 20)	0; 0 (0 to 2)	0; 0 (0 to 7)	33; 6 (4 to 9)
Mean (IQR) time to submission (days)†	537 (396-638)	504 (343-601)	535 (408-612)	538 (366-640)	543 (427-674)*
Mean (IQR) time to publication (days)‡	823 (650-1063)	774 (628-949)	876 (708-1203)	824 (694-1041)	833 (650-1103)*

IQR=interquartile range.

*Includes 48 studies accepted but not yet published as of data cut-off date.

†Based on 867 studies with study end and subsequent submission dates.

‡Based on 670 studies with study end and subsequent publication dates.

Figure 1 summarizes acceptance rates for studies with positive and negative outcomes. First time acceptance rates were 56% for positive studies and 48% for negative studies (P=0.17). These rates were also similar when we performed the same analysis using the pure positive and pure negative categories described in methods (50% v 49%). About three quarters of studies were accepted after one to two submissions for both study outcome categories (78% for positive v 73% for negative studies).

Fig 1 Rates of acceptance for publication in peer reviewed journals for human drug research studies with positive and negative outcomes

Discussion

In this systematic review of more than 1000 studies from a single industry sponsor, rates of submission for publication in peer reviewed journals were broadly similar across study types and outcomes. Over the study period, a greater proportion of studies with negative outcomes than positive outcomes were submitted (92% v 79%) and accepted (77% v 66%) for publication in a journal, indicating a lack of bias against either submission or publication of negative studies by sponsor and journals, respectively. Given the proximity of the analysis cut-off point to the end of the six year review period, journal decisions were still pending for 9% of manuscripts submitted. An additional 7% of studies were disclosed through conference abstracts only (not included in publication totals). In total, 83 studies (8%) had not been submitted for publication in any form at the time of analysis, of which 49 were bioequivalence studies with positive outcomes and 33 were non-comparative studies. Most (98%) studies had results posted to one or more public registers, including all studies subject to FDAAA requirements for posting on www.clinicaltrials.gov.

While the topic of industry bias in reporting results of negative studies had been recently and widely debated, to our knowledge no previous study has systematically covered the issue of publication bias according to study outcome.1 6 7 10 11 12 13 In our cohort, overall publication rates were broadly in line with recent estimates based solely on data obtained from www.clinicaltrials.gov.5 6 7 Median times to publication were also consistent with previously published figures.6 7 8 9 When we evaluated the impact of study outcome, first time acceptance rates for studies with positive outcomes were numerically, but not significantly, higher than for those with negative outcomes (56% v 48%). Median time from study completion to submission was 31 days longer for negative than for positive studies and time to publication was 102 days longer, suggesting that greater effort and care (such as journal selection) are required for publication of negative data. Even so, 10% of all studies and 13% of those with negative outcomes required three or more submissions to achieve journal publication. These data suggest that rates of publication discussed in the scientific literature might substantially underestimate the effort that sponsors make to publish their results as they do not capture the number of submissions and rates of journal rejection.

Our analysis also shows that studies with negative outcomes can be and have been successfully published in the scientific literature and provides no evidence of a systematic bias against their acceptance by journals. Most (92%) studies in our cohort that had an unfavorable outcome for the drug under evaluation were submitted, and 77% had been published by the analysis cut-off date. The fact that slightly more than half of studies included in our analysis could not be readily classified into those with positive or negative outcomes, however, also suggests that such a dichotomous scheme represents an oversimplification of study outcomes, particularly when early phase studies are included. The current transparency debate and efforts might be better served by focusing on ensuring full publication of all studies, irrespective of outcome.

Strengths and limitations

Although our study has several strengths—including a large systematic cohort, inclusion of all studies (irrespective of phase or outcome), rigorous methods, and metrics on submissions as well as final publication rates—it also has important limitations. Clarity in specifying endpoints and classification of study outcomes could be inexact and subject to debate. For applicable studies, we deliberately chose to classify studies as having a positive or negative outcome for the drug, rather than on the basis of significance alone, because for some types of studies (such as corrected QT safety studies) lack of significance is a favorable outcome for the drug being tested. Conversely, significant negative safety findings could preclude further development of an investigational drug. We did not examine selective outcome reporting for the 751 studies that were published, although a unified process for reporting endpoints in study summaries and reports is followed by the sponsor to ensure consistent reporting. Most importantly, the data reported here are from a single industry sponsor. Without further data from other research sponsors, it is not possible to determine whether these results generalize to other industry sponsors or to other types of sponsors (academia, government). Despite this limitation, these data provide a clear signal that submission and publication bias against studies with negative outcomes might be less widespread than has generally been assumed and should not deter efforts to publish them.

Conclusion

In summary, the conduct and publication of analyses of submissions and successful publication according to study outcome are potentially important actions needed to assess and improve actual practice, and when appropriate, to correct misperceptions regarding publication bias that adversely impact the credibility of drug research. We encourage other sponsors and journal editors to share similar information in efforts to contribute to better understanding of issues and barriers to full transparency.

Literature surveys carried out since 2010 suggest that 15-44% of studies registered on www.clinicaltrials.gov and other public sites had not been published in medical journals

Studies lacking significant outcomes were less likely to be published and, when published, took a longer time from study completion to publication

Failure of sponsors to publish all research results, regardless of outcome, adversely impacts the credibility of medical research

Rates of publication previously discussed in the scientific literature might have substantially underestimated the effort that sponsors made to publish their results as they did not capture sponsor submission and journal rejection rates

Submission and publication bias against studies with negative outcomes might be less widespread than has generally been assumed and should not deter efforts to publish them

Greater sharing of submission and rejection metrics by sponsors and journals could contribute to better understanding of issues and barriers to full transparency