Time to publication for publicly funded clinical trials in Australia: an observational study.

OBJECTIVE: To examine the length of time between receiving funding and publishing the protocol and main paper for randomised controlled trials.
DESIGN: An observational study using survival analysis.
SETTING: Publicly funded health and medical research in Australia. PARTICIPANTS: Randomised controlled trials funded by the National Health and Medical Research Council of Australia between 2008 and 2010. MAIN OUTCOME MEASURES: Time from funding to the protocol paper and main results paper. Multiple variable survival models examining whether study characteristics predicted publication times.
RESULTS: We found 77 studies with a total funding of $A59 million. The median time to publication of the protocol paper was 6.4 years after funding (95% CI 4.1 to 8.1). The proportion with a published protocol paper 8 years after funding was 0.61 (95% CI 0.48 to 0.74). The median time to publication of the main results paper was 7.1 years after funding (95% CI 6.3 to 7.6). The proportion with a published main results paper 8 years after funding was 0.72 (95% CI 0.56 to 0.87). The HRs for how study characteristics might influence timing were generally close to one with narrow CIs, the notable exception was that a longer study length lengthened the time to the main paper (HR=0.62 per extra study year, 95% CI 0.43 to 0.89).
CONCLUSIONS: Despite the widespread registration of clinical trials, there remain serious concerns of trial results not being published or being published with a long delay. We have found that these same concerns apply to protocol papers, which should be publishable soon after funding. Funding agencies could set a target of publishing the protocol paper within 18 months of funding. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/.

We had a relatively small sample size of 77 studies.

The protocol paper was examined as a time-dependent intermediate event when examining the time taken to publish the main paper.

We did not contact authors to confirm publication dates.

Introduction

Randomised controlled trials (RCTs) represent the gold standard when evaluating healthcare interventions, mainly because they minimise selection bias.1 Policymakers often depend on the published results from RCTs and systematic reviews based on RCTs to learn about the efficacy of interventions and to apply the findings in health policy.2 Previous research has found 30–50% of clinical trial results do not get published, even years after completion of the project.3–7 This missing research is a large part of the overall ‘waste in research’, with the estimate that 85% of current research investment is wasted.8

National Health and Medical Research Council (NHMRC) Project Grants are the biggest source of funding in Australia for clinical trials. Project Grants aim to support individual researchers as individuals or small teams for a defined project from 1 to 5 years. Applications are extensive, including sections on past funding, budget and progress reports, but the key sections are the researchers' track records and the nine-page detailed background and research plan. Applications for funding large clinical trials are generally peer reviewed by a separate panel of experts who only consider clinical trials.

The NHMRC have supported policies to improve clinical trial reporting. In 2005, an NHMRC grant helped establish the Australian and New Zealand Clinical Trial Registry (ANZCTR). In 2007, the NHMRC published a code for the responsible conduct of research, which stated that researchers must register clinical trials and have a responsibility to their colleagues and the wider community to disseminate a full account of their research as broadly as possible.9 In 2014, the NHMRC supported the Alltrials campaign, which calls for all past and present clinical trials to be registered and their results reported. To date, there is no NHMRC policy that mentions publishing research protocols.

The aim of this paper was to examine the time delay between funding and publication for government-funded RCTs in Australia. As well as the time to the publication of the main paper, we also examined the time to a protocol paper. The main purpose of the protocol is to predetermine the methods and primary outcomes, so that issues such as outcome switching are reduced or at least made explicit.10 Publishing a protocol paper also allows others to see what research is ongoing and so helps avoid duplicated research being funded. Ideally, protocols would be published quickly, preferably soon after the research is funded. We were also interested in whether publication of a protocol paper was associated with time to publication of the main paper and whether there was an association between the study characteristics (ie, funding amount, number of investigators and number of participants) and time to publication.

Methods

We examined studies funded by the NHMRC of Australia. We combined data on the funding amount, estimated number of research participants, number of investigators and publication times in peer-reviewed journals of the protocol paper and main results paper. We examined the time to publication of the protocol paper and the main paper, and whether characteristics of the study predicted these times.

Data collection

The NHMRC is Australia's main funding body for medical research. The NHMRC administers funding for health and medical research on behalf of the Australian Government, and provides funding for all areas of research relevant to human health and medicine. This study examined Project Grants which are the NHMRC's main scheme for individuals and teams of researchers undertaking RCTs in any field relevant to human health. Applicants can apply for grants of up to 5 years in duration.11

Every RCT should be registered. The three most relevant trial registries for this study are the: ANZCTR, International Standard Randomised Controlled Trial Number (ISRCTN) and ClinicalTrials.gov. They represent a comprehensive source for information about ongoing and completed publicly and privately funded trials within and outside Australia. All three registries use web-based systems to facilitate registration of clinical trials.12–14 The registries include compulsory and optional data items, and trials will not be registered without completion of all compulsory data items and approval by a human participant ethics review board. Additional information about the registries is available from their websites.12–14

Search strategy

We aimed to find all RCTs funded by the NHMRC between 2008 and 2010. We used these years to strike a balance between examining recent data and allowing a reasonable time for trial completion and publication. We identified 77 RCTs by searching for the words ‘RCT’, ‘randomised’ and ‘trial’ in NHMRC documents available on the NHMRC website (http://www.nhmrc.gov.au/grants-funding/outcomes-funding-rounds). From the same data we also extracted the funding amount ($A), number of investigators and planned study duration. To find the number of participants and trial registration number, we first searched for the project name and principal investigator within ANZCTR. When a study could not be located in ANZCTR, we searched for it within ISRCTN and ClinicalTrials.gov. Some studies may have been registered in multiple registries, but we did not record this. The data were entered into an Excel spreadsheet.

Two investigators (LBS and AGB) independently determined the publication status of each trial in January 2016. To locate a protocol paper or main results paper, we searched for the unique NHMRC project number and the unique trial registration number in PubMed and in Google Scholar. If we were unable to locate the papers by this method, we searched for the name of the principal investigator and the name of the project. We matched the articles that were identified to the corresponding trial using information about location, enrolment, study start and completion dates, and outcome measures described in the clinical trial registers. We also searched all available NHMRC grant summaries from 2004 to 2012 (the last year available from the NHMRC). If we found a protocol paper, we searched all the papers that cited the protocol paper to look for the main paper. Similarly, if we found the main paper, we searched the text for mention of the protocol paper. We only included protocols published in peer-reviewed journals. The data collection process is shown in figure 1.

Figure 1

Flow chart showing the data collection process. ANZCTR, Australian and New Zealand Clinical Trial Registry NHMRC, National Health and Medical Research Council; RCT, randomised controlled trial.

Publication status and time to publication

For all published papers, we calculated time to publication from the date of the funding announcement. For papers published online ahead of print, we used the date the publication was available electronically. For trials where we could not find a protocol paper or a main results paper, we used a censored follow-up time as the last day we performed our search (31 January 2016).

We were interested in whether some trials were more likely to be completed and wanted to examine whether time to publication of the protocol paper and main results paper was associated with funding amount, number of investigators or participant numbers. We were also interested in whether publishing a protocol paper influenced the time it took for the main results paper to be published.

Statistical analysis

Kaplan-Meier survival plots were used to show the time to publication of protocol paper and main results paper. The y-axes in the plots were reversed to show the increasing proportion of published papers over time. We calculated the median time to publication and 95% CIs, and the mean survival and 95% CI at the maximum follow-up time of 8 years.

We used Cox proportional hazard models with time since funding date as the time scale, and presented results as HRs with 95% CIs. The predictor variables for the publication of a protocol paper were funding amount (per $A100K), number of investigators, project duration and participant numbers (per 100). These variables were chosen as indicators of complexity that may make a trial more difficult to complete.

When examining the time to publication of the main results paper, we added the protocol paper as a time-dependent intermediate variable (figure 2).15 The alternative approach of using a binary variable for protocol published (yes/no) ignores time and assumes the protocol was published on the funding date. This lengthens the time between the protocol and main paper and would bias the HR downwards.16 All analyses were performed using R V.3.1.0.

Figure 2

The three key events over time. Studies may go straight from funding to main paper publication or first publish a protocol paper, which is therefore a time-dependent variable. The illustration shows how ignoring the timing of the protocol lengthens the time between the protocol paper and the main paper which would bias the HR downwards. The thicker part of the arrow from protocol to main paper in the right panel is the arrow length in the left panel.

Missing data

We were unable to find four studies in any of the trial registers, and so we did not have information on the number of participants. We therefore randomly imputed these missing numbers from the available participant numbers. We created 10 imputed data sets and combined the results using the ‘mitools’ package in R.17

Results

We found 77 studies with a total funding of ∼$A59 million (see online supplementary appendix for list of studies). Forty-three studies published a protocol paper and 39 studies published a main results paper (figure 3). Thirty million dollars and 50% of the funding went to studies that have not yet published the main results paper during the mean follow-up time of 6.2 years.

Figure 3

Flow chart showing publication status (protocol and main paper) of the 77 funded studies.

Characteristics of the RCTs by publication status are in table 1.

Table 1

Characteristics of the randomised controlled trials funded by the National Health and Medical Research Council (NHMRC) Project Grants 2008–2010 by publication status of main results paper

	Main results paper published
	Yes		No
Characteristic	Mean	SD	Mean	SD
Funding amount ($1000)	756	539	788	509
Participant numbers	562	680	545	1049
	Median	IQR	Median	IQR
Number of investigators	5	4–6	5	4–6
Project duration (years)	4	4–5	5	4–6
Protocol paper	n	Per cent	n	Per cent
Yes	20	51	23	61
No	19	49	15	39
Total	39	51	38	49

Time to publication

From the funding date, the median time to publication of the protocol paper was 6.4 years (figure 4), with a 95% CI from 4.1 to 8.1 years. The estimated proportion with a published protocol paper 8 years after funding was 0.61 (95% CI 0.48 to 0.74). One protocol was published before funding.

Figure 4

Kaplan-Meier survival curve (and 95% CIs as dashed lines) for time from funding to protocol paper published (n=77). Horizontal lines on the survival curve indicate censoring.

The median time to publication of the main results paper was 7.1 years (figure 5), with a 95% CI 6.3 to 7.6. The estimated proportion with a published main results paper 8 years after funding is 0.72 (95% CI 0.56 to 0.87). As expected, shorter studies were generally published faster.

Figure 5

Kaplan-Meier survival curves (and 95% CIs as dashed lines) for time from funding to main paper published for all papers combined (left) and split into relatively long and short studies by median study length (right; n=77). Horizontal lines on the survival curves indicate censoring.

Multivariable analysis

We found no evidence for an association between funding amount, number of investigators or participant numbers with time to publication of protocol paper or main results (table 2). Publication of a protocol paper was not associated with time to publication of the main results paper. Project duration was not associated with time to protocol paper, but not surprisingly longer projects took longer to publish their main results paper, with a HR of 0.62 per extra year (95% CI 0.43 to 0.89).

Table 2

HRs and 95% CIs for time to publication of protocol paper and main results paper dependent on study characteristics

Study characteristic	Protocol paper	Main paper
Funding amount (per $100K)	1.01 (0.95 to 1.07)	1.02 (0.95 to 1.09)
Number of investigators	1.03 (0.84 to 1.27)	1.03 (0.81 to 1.30)
Project duration (years)	1.00 (0.86 to 1.16)	0.62 (0.43 to 0.89)
Participant numbers (per 100)	1.01 (0.98 to 1.04)	1.01 (0.97 to 1.04)
Protocol paper as intermediate event		0.69 (0.34 to 1.39)

Discussion

The median time to publishing a protocol paper from NHMRC-funded research was 6.4 years after funding. The median time to publication of the main results paper was 7.2 years, well beyond the duration of funding for NHMRC grant schemes. It is hard to understand the reasons for these delays, because the study design should largely be finalised at the time the researchers receive funding, and a key criteria the NHMRC uses for funding research is the feasibility of the project and the investigators' track record.18 Some delays may be due to protracted peer review, which is no fault of the investigators.19 Other delays outside the investigators' control are sponsor termination and contractual issues.20

Publishing a protocol

Publishing a protocol early increases transparency and gives the research community and the public a better idea of ongoing studies. Protocols are often included in systematic reviews or assist in development of prospective meta-analyses, and can be particularly important for research questions with little current evidence. While some journals (eg, BMJ Open) will not accept protocol papers if the data collection has concluded, there are few guidelines on the timing of the publication of protocol paper in relation to funding or initiation of the study. While there is an established guideline for the reporting of a protocol,21 these only cover the content and not the timing of protocol papers. The guidelines could be extended to include a recommended target date for publication after funding, or after ethics approval for unfunded studies. A period of 18 months after funding seems a sensible target. This time allows researchers to transfer the information from the grant application into the protocol format and allows time for peer review. Delays due to peer review could be avoided by adding the protocol to a clinical trial registry. Funding agencies such as the NHMRC could mandate a target and withhold further funding if the protocol is not published or at least submitted to a journal to provide a clear incentive to publish the protocol. There is a precedent for this, as the UK National Institute for Health Research include in their funding contract that a publicly available final report must be submitted, and this agency has a 98% publication rate.22

Current editorial processes at many journals mean it is now much harder to get an RCT published without a protocol, and so the problem we identified may be a historic concern. However, the median time to protocol paper was 7.3 years for studies funded in 2008, 3.9 years for studies funded in 2009 and 6.0 years for studies funded in 2010. So our data do not show clear signs of improvement between 2008 and 2010.

Predictors of publication timing

Our sample size was relatively small, but the HRs for how study characteristics might influence timing were generally close to one with narrow CIs (table 2), which suggests that the issues with timely publication are common to large and small studies. The two exceptions were study length and protocol paper. The mean HR for the protocol paper is <1, suggesting that publishing a protocol slows the time to the main paper. This could be because not having a protocol paper allows authors to focus on statistically significant results that may not have been the primary outcome, and there is a bias towards journals accepting papers with statistically significant results,23 and it generally takes longer to publish negative results.24 However, the CIs around the HR for the protocol were wide, and a larger study is needed to examine this potential association.

Previous studies

Our results are similar to related studies. A large study of US RCTs found that only 36% had published their results within 2 years of study completion7 and another US study of clinical trials found that only 46% were published within 30 months of trial completion.4 These results and ours highlight the serious problem of missing research and the timeliness of completing studies. Registering trials has been an important step in avoiding the waste of missing research, but it has not solved the problem as there are many registered randomised trials that remain unpublished.

Ours is the first study to use Australian data and the first to additionally examine the protocol paper.

Limitations

There are a number of limitations. Our sample size was relatively small compared with related studies. We did not email authors to confirm publication status, but a related study found that emailing authors did not provide much additional information.4 Our search to identify RCTs would have missed studies that did not include the three phrases we searched for in their title.

Conclusions

In a time when there is very high competition for scarce research funds, we found that both the protocol paper and the results of clinical trials are not being published in a timely fashion. Funding bodies need to focus more on the outputs from research and timely publication of a protocol paper should be the first milestone to assess whether trials will produce results which can be added to the body of evidence used for medical decision-making.