Trial registration and time to publication in a retrospective cohort of publicly funded randomised controlled trials in New Zealand 1999-2017.

OBJECTIVES: To determine how quickly randomised controlled trials funded by the Health Research Council of New Zealand (HRC) were registered and published, and whether time to publication differed by trial result.
DESIGN: We created a retrospective cohort of trials offered funding from 1999 to 2017 by seeking lists of candidate studies using the Official Information Act 1982. These lists were supplemented by searching the HRC's online research repository and an open-access database on Figshare. One investigator searched for trial registrations and for dissemination using electronic databases, university websites and ResearchGate. One investigator extracted data from the obtained studies and a second investigator independently corroborated the data entry from a 10% random sample.
RESULTS: We identified 258 trials that were offered funding, 252 trials were conducted and 229 (90.9%) were registered, 179 prospectively by the date of the final search (24 March 2022). Overall, 236 trials were completed by the date of the last search and in 209 (88.6%) trials the results had been disseminated, 200 (84.7%) of which were by journal publication. We obtained the results for 214 trials, 91 (42.5%) of which were positive, 120 (56.1%) of which were null and 3 (1.4%) of which were negative. Median time to publication was 22.7 months for positive trials and 21.5 months for combined null or negative trials (log rank test p=0.83). Median time since trial completion in the trials that had not been published was 43.6 months (IQR 17.1-108.2 months).
CONCLUSIONS: Between 1999 and 2017, almost 9 out of every 10 HRC-funded trials had been registered and a similar proportion of completed trials had been published with no difference in time to publication based on type of result. However, only a slim majority of trials had published within the 2-year time frame set by the WHO. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study quantifies the number of trials supported by the Health Research Council of New Zealand (the country’s peak health research funder) from 1999 to 2017, and estimates the rate of trial registration and publication for such trials.

This study provides the only investigation of time to publication according to trial result (positive vs null and negative) for trials supported by a peak health research funder.

However, not all trials supported through career development awards from this funder between 1999 and 2006 may have been identified.

The Health Research Council of New Zealand only funds a small proportion of the trials conducted in New Zealand each year and thus these findings are not representative of all the conducted trials.

Introduction

Randomised controlled trials (RCTs) (henceforth referred to as ‘trials’) are the most reliable method for evaluating the effect of interventions,1 but many remain unpublished despite being registered.2 Trials can support both clinical decision-making and policy development, but only where the results of the trials are known. However, substantial proportions of completed trials are not reported,3 sometimes with unfortunate consequences such as in the case of class 1c antiarrhythmics.4 Studies with positive findings also appear more likely to be published than those with null findings.3 In an effort to counter this dissemination and publication bias, the International Committee of Medical Journal Editors (ICMJE) announced in 2004 that from 2005, the journals the ICMJE represented would only consider trials for publication if they had been registered.5 This position caused a spike in trials registration as the deadline approached6 and an ongoing increase in trials being registered thereafter.7

The Health Research Council (HRC) is the main public good funder for health research in New Zealand. Research grants range from small career development awards to large programme grants with support sought through competitive funding rounds. The HRC has supported trials registration through contributions to the Australia New Zealand Clinical Trials Registry and other initiatives. Since 2009 the national guidelines for intervention studies in New Zealand obliged investigators to register clinical trials on a WHO approved clinical trials registry,8 an initiative that was enhanced to a requirement in 2012 within the national ethics committees operating standards.9 While trials registration has been investigated in New Zealand,10 there have been no investigations of rates of dissemination or of publication bias in trials supported by the HRC. The aim of this paper was, therefore, to determine (1) the compliance with trials registration in trials funded by the HRC, (2) the rate of publication of HRC-funded trials and (3) whether there was any delay to publication by type of result in these trials.

Methods

Patient and public involvement

No patients or members of the public were involved in the design or conduct of this study.

Search strategy

We sought a list of trials from the HRC using a request under the Official Information Act 1982 via the FYI website (www.fyi.org.nz) on 24 April, 2020. That request sought the HRC reference number, title, principal investigators’ names and type of grant for all trials funded by the HRC under the project, programme and feasibility schemes since 1999. This year was chosen as trials registers (ClinicalTrials.gov and International Standard Randomised Controlled Trial Number (ISRCTN)) first became available online during 2000, meaning that trials funded from 1999 onward had the opportunity to be registered early in their conduct, either prospectively or retrospectively. The cut-off of 2017 was selected to ensure investigators funded in that year had sufficient time to have completed and published their trials.

The HRC responded they did not hold information on the study design prior to 2011. We therefore requested the lay summaries for studies funded between 1999 and 2010. Two separate lists of studies were provided by the HRC, one list including the lay summaries for studies funded 1999–2010 and the second, a list of trials funded 2011–2017. The 1999–2010 list was searched for trials by one investigator (AJ) using the terms “random”, “control” “trial” or “RCT” within the title or lay summary. These lists were supplemented by information from two additional sources as cross-checks. First, we searched the HRC Research Repository from 2011 for any support that would not have been included in our original request, for example, Emerging Researcher First Grants, Explorer Grants, Partnership Grants and other career development awards. Second, we reviewed an open access database on Figshare created from online listings of HRC awards for the period 2006 to 2013 for possible trials not otherwise identified.11 If there was ambiguity as to whether a candidate study was a trial, we reviewed the principal investigators’ professional profile on university websites and ResearchGate for contact information and sought clarification from the principal investigator in the first instance or associate investigators where the principal investigator was not contactable. Only studies described as RCTs were included in this study.

Data collection and analysis

To obtain details on trials registration, we searched three trials registers (Australia New Zealand Clinical Trials Register (ANZCTR), ClinicalTrials.gov and ISRCTN). To determine whether the trial had been disseminated, we searched Medline, Embase, the Cochrane Controlled Trials Register and Google Scholar using the principal investigator’s name, and the trial registration number, or other identifiers such as study title or acronym in separate searches. We also reviewed professional profiles on university websites and ResearchGate. If the award was in support of a postgraduate degree, we searched Digital Dissertations and the candidate’s university library catalogue for a thesis. Research dissemination could take the form of a journal publication, a publicly available thesis, a letter, a conference abstract or proceedings (but only where the final results were published), a preprint paper on MedRxiv, or the results being posted on a trials register and was only counted once in that hierarchy. Publication was defined as publication of a paper in a peer-reviewed journal; a conference abstract or letter to the editor did not qualify as a journal publication. A protocol was considered published if the complete protocol was published separately or if the complete protocol was available as a supplement to the main results being published. If we could not identify whether a trial was registered, completed or published, we contacted the principal investigator for clarification. In four instances where we could not obtain a response from an investigator and no other information was available, we obtained the final report for the grant from the HRC using the Offical Information Act.

Each trial’s findings were used to determine whether the trial was categorised as positive, negative or null. A trial was defined as positive if the between-group difference for the primary outcome was statistically significant and in favour of the exposure, or if the results were within predefined non-inferiority or equivalence bounds, or, if feasibility objectives were considered to be met by the investigators or if a pilot trial recommended a main trial. A trial was deemed null if the between-group differences were not statistically significant, or if the differences overlapped predetermined non-inferiority or equivalence bounds, or if a larger trial was not recommended where the funding was for a feasibility study. A trial was defined as negative if the between-group difference for the primary outcome was statistically significant, but against the exposure. The date of dissemination was taken from the date of presentation at a conference or date of publication. The date of publication was taken from when the publication was first available, whether that was online publication or print publication. If the actual day of publication was not identifiable for print publications, the day of publication was taken as the 15th day of the month of publication. Time to publication was obtained by adding the longest follow-up period from all the outcomes in the trial to the date the last participant was recruited, unless the trial register or journal paper provided a date for last data collect. The last search for trial registration or publication was 24 March 2022. The citation counts were obtained for each published trial from Google Scholar and other dissemination metrics (altmetric score and download counts) from each publication’s page on journal websites, all on the same date. All data was extracted into an Excel spreadsheet by the same investigator (AJ). The second investigator (NW) verified the entries for accuracy on a 10% sample of entries randomly generated using the RANDBETWEEN function in Excel. The dataset was imported into SPPS V.26 for analysis. Counts and percentages were reported for categorical data, while medians and IQR were reported for continuous data as the data was non-parametric. The time-to-event analysis was undertaken using a Kaplan-Meier plot and the log rank test.

Results

We identified 258 trials that were offered funding by the HRC between 1999 and 2017; 2 did not proceed as trials and 4 trials failed to recruit, leaving 252 trials for analysis (figure 1). Verification by the second reviewer on 10% of entries showed 98% accuracy on data extraction across all fields. The majority of identified trials (57.2%) were funded as project grants (table 1), were two arm parallel group trials (71.6%), used a single centre to recruit (53.6%), recruited adult participants (77.6%), and most frequently, involved a procedure or behavioural intervention as the exposure (46.8%). Two hundred and twenty-nine trials (90.9%) were registered across the three trials registers, 179 prospectively and 50 retrospectively. Most trials (192, 83.%) were registered on the ANZCTR, followed by ClinicalTrials.gov (31, 13.5%) and the ISRCTN register (6, 2.6%).

Figure 1

Flow diagram of search process and obtained trials.

Table 1

Characteristics of included trials

Characteristic	Not registered	Registered	Total
	(N=23)	(N=229)	(N=252)
	N, %	N, %	N, %
Type of award
Programme	3 (13.0)	18 (7.9)	21 (8.4)
Project	6 (26.1)	138 (60.3)	144 (57.2)
Feasibility	4 (17.4)	25 (10.9)	29 (11.6)
Career development awards*	5 (21.7)	23 (10.0)	28 (10.8)
IIOF	-	5 (2.2)	5 (2.0)
DHB Partnership Programme	-	16 (7.0)	16 (6.4)
Explorer	-	1 (0.4)	1 (0.4)
Missing	5 (21.7)	3 (1.3)	8 (3.2)
Type of trial
2-arm RCT	9 (39.1)	170 (74.6)	179 (71.6)
Multiarm RCT	5 (21.7)	21 (9.2)	26 (11.4)
Factorial RCT	-	7 (3.1)	7 (2.8)
Cluster RCT	7 (30.4)	18 (7.9)	25 (10.0)
Cross-over RCT	1 (4.3)	6 (2.6)	7 (2.4)
No of centres
Single centre	15 (65.2)	120 (52.4)	135 (53.6)
Multicentre	4 (17.4)	96 (41.9)	100 (39.7)
Missing	4 (17.4)	13 (5.7)	17 (6.7)
Type of intervention
Drug	2 (8.7)	89 (38.9)	91 (36.0)
Device	2 (8.7)	34 (14.8)	36 (14.4)
Procedure/behavioural	17 (73.9)	101 (44.1)	118 (46.8)
Food	2 (8.7)	2 (0.9)	4 (1.6)
Other	-	3 (1.3)	3 (1.2)
Type of participant recruited
Adult	16 (69.6)	179 (78.2)	194 (77.6)
Child†	6 (26.1)	44 (19.2)	50 (20.0)
Adult or child	-	6 (2.6)	6 (2.4)
Missing	1 (4.3)	-	1 (0.4)
Sample size
Target sample size‡	96 (31–300)	308 (104–918)	284 (100–800)
Actual sample size‡	80 (34–276)	251 (95–712)	225 (89–666)
Trials register§
ANZCTR	-	192 (83.)	192 (76.1)
ClinicalTrials.gov	-	31 (13.5)	31 (13.5)
ISRCTN	-	6 (2.6)	6 (2.4)

*Includes both junior and senior career awards.

†Includes adolescents.

‡Median and IQR.

§Missing information.

ANZCTR, Australia New Zealand Clinical Trials Register; DHB, District Health Board; IIOF, International Investment Opportunities Fund; ISRCTN, International Standard Randomised Controlled Trial Number; RCT, randomised controlled trial.

The number of funded trials increased from six awards in 1999 to 17 awards in 2017, with the greatest number (25) funded in 2009 (figure 2). Between 1999 and 2017, the HRC supported a mean of 13.3 (SD 6.4) trials per year supported from 1999 to 2017. The number of trials supported has doubled from a mean of 8.8 (SD 5.1) trials per year in the first 10 years of the period to a mean of 18.2 (SD 3.1) trials per year from 2009. Eighteen of the 23 (78.3%) unregistered trials were funded in the period to 2009.

Figure 2

Number of trials registered and unregistered by year of funding round (black equals registered and grey equals unregistered trials).

Two hundred and thirty-six trials had completed data collect by 24 March 2022 and we were able to determine that 209 (88.6%) had been disseminated (table 2). The median time to dissemination was 20.8 months (IQR 12.4–33.0 months). In 200 trials, dissemination took the form of a paper published in a peer-reviewed journal, whereas the remaining trials were disseminated via conferences (4), a letter to a journal (1), preprint paper on MedRxiv (1), results posted on the trials register (1) or as a thesis (2). For published papers, the median time to publication was 21.5 months (IQR 12.5–36.9 months), the median number of citations was 62 (IQR 18–167), the median altmetric score (n=124) was 12 (IQR 6–25) and the median number of full text accesses or downloads (n=98) was 4951 (IQR 970–19,254). The median time to censoring for the trials that had not yet been disseminated was 43.6 months (IQR 17.1–108.2 months) and 25 of these 27 trials had been registered, but no results were posted to the registeries for these trials.

Table 2

Characteristics (where known) of completed trials

	Not registered	Registered	Total
	(N=19)	(N=217)	(n=236)
Protocol published	-	104 (47.9)	104 (44.1)
Research disseminated*	17 (89.5)	192 (88.5)	209 (88.6)
Journal paper	15 (78.9)	185 (85.3)	200 (84.8)
Thesis	-	2 (0.9)	2 (0.9)
Letter	1 (5.3)	-	1 (0.4)
Conference	1 (5.3)	3 (1.4)	4 (1.7)
Preprint server	-	1 (0.5)	1 (0.4)
Posted on trials register	-	1 (0.5)	1 (0.4)
Not disseminated	2 (10.5)	25 (11.5)	27 (11.4)
Information reported on register
Funder acknowledged	-	175 (80.6)	175 (74.2)
Grant no present	-	18 (8.3)	18 (7.6)
Record of main publication	-	89 (41.0)	89 (37.7)
Results posted	-	6 (2.8)	6 (2.6)
Data sharing statement posted	-	31 (14.3)	31 (13.1)
Information reported in journal paper†
Sequence generation reported	10 (66.7)	154 (83.4)	164 (82.0)
Concealment strategy reported	7 (46.7)	124 (67.0)	131 (65.5)
Funder acknowledged	15 (100)	185 (100)	200 (100)
Grant no reported	3 (20.0)	88 (47.6)	92 (46.0)
Type of result‡
Positive result	14 (82.4)	77 (40.1)	91 (42.5)
Null result	3 (17.6)	117 (60.9)	120 (56.1)
Negative result	1 (5.9)	2 (1.0)	3 (1.4)

*Means of dissemination was only counted once in the order reported in the table.

†Denominator equals 15 in the not registered column, 185 in the Registered column and 200 in the Total column.

‡Denominator equals 17 in the not registered column, 192 in the Registered column and 214 in the total column as 5 results were obtained via personal communication with principal investigator and 209 through research dissemination for a total of 214 known results.

The information provided on trials registers and in journal publications was often incomplete (table 2). The funder was acknowledged in 175 (74.2%) of completed trials on trials registers, but the grant numbers were only reported in 18 (7.6%) of the trials. Similarly, few entries contained either a record of the main results paper (37.7%), a data-sharing statement (13.1%) or posted results (2.6%) despite these being a requirement on WHO-standard clinical trials registers. Where trials had been published in a journal, 100% had acknowledged the HRC as funder, but only 92 (46.0%) included the grant number. Of more concern is the observation that only 164 (82.0%) had reported sequence generation and 131 (65.5%) had reported allocation concealment strategies despite these being important design elements for assessing the quality of RCTs.

We were able to estimate the time to journal publication or censoring and obtain the findings in 214 (90.7%) of the 236 completed trials. The findings were positive in 91 trials (42.5%) and negative or null in 123 trials (57.5%). Fourteen of the 214 trials remained unpublished in journals at the date of the last search and were censored, 4 of which had positive findings and 10 of which had null or negative findings. The median time to publication was 22.7 months (IQR 13.3–33.0) and 21.5 months (IQR 12.6–35.8), respectively, for positive and null or negative trials. There was no significant difference in time to publication by type of result (figure 3, log rank test p=0.83).

Figure 3

Kaplan-Meier plot of time to publication in months by type of result (red curve represents positive trials and blue curve negative or null trials) for 214 HRC-funded trials (with vertical markers indicating a censored trial still unpublished after date of last search).

Discussion

The HRC support for trials appears to have doubled since the decade of 1999–2008. Most trials have been registered, with almost all trials since 2009 having been registered. Almost 9 out of every 10 completed trials had published results in a journal with the median time to publication being no different for positive trials compared with null or negative trials. However, only one-third of results publications were recorded on trials registers, and just six trials had posted results on their register entry.

Our findings suggest HRC supported trials are published in journals at a greater rate than other peak public funders worldwide. The publication rate of the 101 trials funded by the Swiss National Science Foundation (SNSF) for the 29 years between 1986 and 2015 was 60%,12 compared with the 84.8% publication rate we observed for HRC supported trials. The publication rate in a sample of completed trials funded by America’s National Institute of Health (NIH) registered on ClinicalTrials.gov was similar to that of the SNSF at 68% of the 635 trials being published.13 The rate of publication for projects funded by the UK’s National Institute of Health Research (NIHR) was also within this range at 63%, although the rate was considerably higher if publication in the NIHR’s inhouse journal was considered.14 These rates were all lower than the publication rate for primary outcomes in the 66 completed trials funded by Australia’s National Health and Medical Research Council between 2003 and 2007 at 81.8%, which was closer to the rate for the New Zealand trials.15 The median time to publication from completion in HRC supported trials was of a similar order to those in other investigations. The median time to journal publication was 23 months in NIH-funded trials and 26 months in trials conducted by academic centres in the United States,13 16 but the time to external publication was longer in NIHR-funded trials at 30.5 months.14

The impact of HRC-funded trials is at least on par if not better than that found in other investigations. The mean citation rate for medicine and health papers published in World of Science in 1995 and counted 18 years later in 2013 was 28,17 while the mean citation rate for NIHR trials was 103 cites per trial.18 We have reported medians given our data were non-parametric, but for the sake of comparison, reanalysis of the HRC-funded trials found the mean citation rate was 160, suggesting impressive citation rates compared with other investigations.

Previous studies have found evidence supporting the presumption that trials with significant or positive results are more likely to be published.19 Reasons for non-publication of trials often relate to investigator perceptions and include lack of time, difficulties between authors, loss of interest, results not being statistically significant or belief that journals would be unlikely to accept the paper.20 21 Yet our findings belie these beliefs; we found little difference in the time to publication for trials with positive findings or otherwise, and the majority of the published trials had null or negative findings.

We cannot account for why the HRC-funded trials have performed well in comparison to other peak funders internationally. HRC funding is not conditional on registration and HRC research contracts are only monitored through annual reports. Trial registration is possibly influenced by NZ having a national standard for ethics committees, which stipulated in 2011 that trials must be registered. However, compliance is not perfect and while the non-publication rate in HRC-funded trials is lower than in other jurisdictions, we draw attention to the fact that after a median of 43.6 months since trial completion, 11% of the trials identified in our study remained unpublished. It is of course possible that the SARS CoV-2 pandemic has interrupted the publication process for completed studies, but prior to the pandemic a similar pattern has been noted in NIH-funded trials where time to publication was 23 months and many trials remained unpublished after a median of 51 months since completion. A question, therefore, to be considered is what role funders might have to expedite timely publication?

The WHO has defined timely publication as the main results for a trial being submitted within 12 months of completion and published within 24 months.22 Many public good funders, including the HRC, are a signatories to this joint statement. However, just 56% of trials in our study met this criterion for timely publication, leaving much room for improvement. Currently, final reporting for trials in New Zealand is tied to the contract term for funding and anticipated completion dates. However, both funders and ethics committees could consider extending their final reporting period to incorporate the 2 years after trial completion, so that investigators could report on publication and other forms of dissemination. Just 2.6% of trials in our study had posted results on trials registers, a rate considerably lower than the 26.8% reported for trials conducted by academic centres in the USA,16 but both rates are too low considering the WHO has stated that results should be posted on trials registers within 12 months of completion. Funders could also encourage investigators’ compliance with dissemination requirements by providing the dissemination reports to scientific assessing committees when investigators seek funding for new projects. Trials registers also have a role to play in improving timely posting of results by sending reminder notices to the investigators at 12 and 24 months after anticipated trial completion dates.

We found 23 trials were not registered, despite trials registers being publicly available since 2000 and the ICMJE statements on trials registration being issued in 2004 and 2005.5 23 Nineteen of the trials have been completed and 17 have been disseminated, most as papers, so it may not be necessary for these trials to be retrospectively registered. On the other hand, not all forms of dissemination are equally or easily discoverable and that is a compelling argument for #AllTrials to be registered, even those that have been published, in order to ensure the complete evidence base for treatments is available. There are no barriers for such retrospective registration to happen, although it may be necessary to persuade past investigators to invest time in registering trials. Registries might encourage such action through the use of amnesties, such as those used by the Cochrane Collaboration in the 1990s, or ethics committees might raise a question about whether the applicant has been party to any unregistered trials on ethics applications for prospective projects.

Finally, readers of published reports rely on information sufficiency in order to make judgements about the quality of trials. Critical to these judgements are adequate reporting on sequence generation and allocation concealment, with methodological research suggesting trials that do not provide adequate information overestimate treatment effects.24 It must remain of concern that not all published trials in our study reported sequence generation strategies and fewer reported allocation concealment strategies, a pattern that was similar for both registered and unregistered trials, and one that has been observed previously.25

Our study is subject to several limitations. First, where trials did not publish the last date a participant was recruited, or did not update the same date on the trial registry, or did not provide us with the information through reasonable efforts at contacting the investigator, we had to impute the date of trial completion from other available information, for example, anticipated completion dates on the trials register. Thus some estimated completion dates may create inaccuracies for calculating time to publication. Second, our original request to the HRC did not seek career development awards that may have supported a trial. A small number of such awards were included in the supplied list of studies for 1999–2010, which we supplemented with searching two other datasets from 2006 for such awards. However, we cannnot be certain that all career development awards from 1999 to 2006 were identified and thus we may have underestimated of the number of trials supported by the HRC. Third, we did not seek the results of all the completed but as yet unpublished trials. While some results were available through various forms of dissemination and some investigators did communicate their findings, many investigators told us that they were seeking publication and we did not wish to pre-empt that process in any way. It is possible that inclusion of these results could have influenced the time-to-event analyses. Fourthly, the HRC only funds a small proportion of the number of trials conducted in New Zealand each year. Between 1999 and 2003, an average of 97 late phase trials sought ethics approval each year, while that increased to an average of 124 trial each year between 2005 and 2009.10 26 Thus, our findings should not be considered representative of all trials conducted in New Zealand.

Conclusions

Almost 9 out of 10 HRC-funded trials have been registered and a similar proportion of completed trials have been published with no difference in time to publication based on type of result. However, only a slim majority of trials had been published within the 2-year time frame set by the WHO and there is a case for funders to be more engaged in encouraging timely dissemination of results.