Publication of NIH funded trials registered in ClinicalTrials.gov: cross sectional analysis.

OBJECTIVE: To review patterns of publication of clinical trials funded by US National Institutes of Health (NIH) in peer reviewed biomedical journals indexed by Medline.
DESIGN: Cross sectional analysis.
SETTING: Clinical trials funded by NIH and registered within ClinicalTrials.gov (clinicaltrials.gov), a trial registry and results database maintained by the US National Library of Medicine, after 30 September 2005 and updated as having been completed by 31 December 2008, allowing at least 30 months for publication after completion of the trial. MAIN OUTCOME MEASURES: Publication and time to publication in the biomedical literature, as determined through Medline searches, the last of which was performed in June 2011.
RESULTS: Among 635 clinical trials completed by 31 December 2008, 294 (46%) were published in a peer reviewed biomedical journal, indexed by Medline, within 30 months of trial completion. The median period of follow-up after trial completion was 51 months (25th-75th centiles 40-68 months), and 432 (68%) were published overall. Among published trials, the median time to publication was 23 months (14-36 months). Trials completed in either 2007 or 2008 were more likely to be published within 30 months of study completion compared with trials completed before 2007 (54% (196/366) v 36% (98/269); P<0.001).
CONCLUSIONS: Despite recent improvement in timely publication, fewer than half of trials funded by NIH are published in a peer reviewed biomedical journal indexed by Medline within 30 months of trial completion. Moreover, after a median of 51 months after trial completion, a third of trials remained unpublished.

Introduction

Today, there is an increasing emphasis on the successful translation of results from research into practice. This requires the timely dissemination of findings. While research results might be submitted directly to regulatory agencies, such as the Food and Drug Administration (FDA), physicians and policy makers generally depend on peer reviewed publications to learn about findings from clinical trials. Extensive research has shown, however, that the results of studies are often not shared publicly in a timely way and that between 25% and 50% of clinical trials remain unpublished even several years after completion,1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 although this work was largely focused on industry funded studies. There are many possible reasons behind the delayed or non-publication of results from clinical trials, including lack of incentive to disseminate negative or unsupportive findings, time constraints, limited resources, changing interests, or even failure to have an article accepted by a journal.

Understanding the patterns of publication of research findings among publicly funded research, as opposed to industry funded research, is important because of the funding and the expectation for public benefit. Within the United States, the National Institutes of Health (NIH) is the leading and largest government agency responsible for biomedical and health related research and invests more than $12bn (about £7600m or €8900m) of public resources in funding research in people or in clinical research, $3.5bn explicitly on clinical trials.17 These costs do not include the considerable contributions and costs incurred by the participants in the research. Previous work suggests that half to two thirds of all government funded studies were published two or more years after completion of the trial 18 19—that is, after completion of enrolment and observation—but these estimates included both US and other non-US government agencies. To our knowledge, the last study to focus on rates of publication solely among NIH funded trials was done nearly 20 years ago and examined trials funded by NIH in 1979.4

We examined patterns of publication in peer reviewed biomedical journals indexed by Medline among a sample of NIH funded trials registered within ClinicalTrials.gov (clinicaltrials.gov), a trial registry and results database maintained by the US National Library of Medicine. Although there is no mandate by NIH that all funded research must be registered in ClinicalTrials.gov, in 2005 the International Committee of Medical Journal Editors (ICMJE) began requiring registration as a prerequisite for publication in one of its member journals,20 nearly always in ClinicalTrials.gov for US based studies, and the 2007 US FDA Amendments Act required sponsors of trials of FDA regulated products to register their trials at inception within ClinicalTrials.gov.

Methods

Data source

ClinicalTrials.gov is a publicly available trial registry and results database developed and maintained by the National Library of Medicine on behalf of the NIH. It represents the most comprehensive source for information about ongoing and completed publicly and privately funded trials within and outside the US. ClinicalTrials.gov uses a web based system to facilitate registration of clinical trials by the individual or organisation with primary responsibility for the study (such as the principal investigator or study sponsor).21 Multisite clinical trials using the same protocol are considered to be a single study in the registry. ClinicalTrials.gov includes both mandatory and optional data elements, and trials cannot be registered without adequate completion of all mandatory data elements (according to minimal quality review criteria), approval by a human participant review board (or equivalent), and conformity to the regulations of the appropriate national health authorities. Additional information about the registry is available from the National Library of Medicine.22

Study sample

Among more than 100 000 studies registered within ClinicalTrials.gov on 10 November 2010, we limited our study sample to interventional studies or “clinical trials” (n=81 197; fig 1), defined by the National Library of Medicine as “studies in human beings in which individuals are assigned by an investigator based on a protocol to receive specific interventions.”23 We further limited our sample to trials registered on or after 13 September 2005 (n=62 428), the date by which any ongoing clinical trial, and after which all newly initiated trials, needed to be registered to be considered for publication in a journal that follows the ICMJE registration policy,20 as this requirement was associated with a large increase in trial registration within ClinicalTrials.gov.24 Many trials that had already begun enrolling patients, and some that were completed, were registered on or after this date to comply with the ICMJE policy. We then limited our sample to trials primarily or partially funded by the NIH by selecting registrations that listed at least one NIH institute or centre as the sponsor or collaborator (n=6424). Finally, to provide a minimum of at least two years for investigators to prepare data for analysis, draft a manuscript, and complete the process required to publish their trial, we limited our sample to trials whose overall recruitment status had been updated to notify ClinicalTrials.gov that the trial was completed by 31 December 2008 (n=635). A trial with a status of “completed” is defined by the National Library of Medicine as a trial that has ended and participants are no longer being examined or treated (that is, the last participant’s last visit has occurred).23 25 We excluded NIH funded trials whose overall recruitment status was registered as completed but did not provide an actual study completion date because we would not be able to determine the time lapse between completion and publication for these trials.

Fig 1 Inclusion of NIH funded trials registered within ClinicalTrials.gov

Publication status

Two investigators (HX, LZ) independently determined the publication status of each trial in January 2011 using a search protocol applied in previous research.18 All searches began by first examining the “publication” field within ClinicalTrials.gov to determine if trial investigators provided a citation of an article that described trial results, as this field is used to display citations of trial results or other relevant research. This field is populated in two ways as either the sponsor or investigator can provide one or more entries or ClinicalTrials.gov will search Medline on a daily basis and extract all publications that have an indexed ClinicalTrials.gov identifier (NCT number). If no citation was provided we manually searched Medline with the NCT number. If we did not identify a publication, we searched Medline again with the intervention, condition studied, and name of the principal investigator (when provided in response to the “study official” field). The articles identified through the search were matched to the corresponding trial (when possible) using the following information from ClinicalTrials.gov: detailed description, location, enrolment, study start and completion dates, and primary and secondary outcome measures. Using the same search protocol a third investigator (JSR) independently determined the publication status in June 2011 for all trials that the previous two investigators had determined were unpublished.

Although Medline is not a complete repository of all publications in all biomedical journals, it remains the largest database of biomedical journal articles that can be searched freely with the National Library of Medicine PubMed system. As such, Medline is the source that nearly all physicians and policy makers depend on to learn about and obtain access to clinical trial findings.

Time to publication

For all trials for which a publication was identified in Medline, we determined the time to publication by calculating the duration of time (in months) that had lapsed between completion of the study and publication. The study completion date was determined from ClinicalTrials.gov, and the publication date was determined with Medline. For trials published online ahead of print, we used the date on which the publication was made available electronically as the publication date. For all trials for which a publication could not be identified, we determined the follow-up time by calculating the duration of time, in months, that had lapsed between completion of the study and our final search (June 2011).

Study variables

We obtained information on each trial reported by the study sponsor or investigator directly from the National Library of Medicine, including the following ClinicalTrials.gov data elements for each trial23: trial number, brief title, lead sponsor and collaborators, investigators and study official, study design, trial phase, intervention name, condition, population studied (age, sex, accepts healthy volunteers), target or actual enrolment, study start and completion dates, primary purpose, primary and secondary outcome measures, and publication citations. Investigators/sponsors self report data for their trials during initial registration and are expected to update the information as the trial progresses. We used the information provided in the trial classification data element to categorise study purpose as efficacy only, efficacy and safety, safety only, or indeterminate. For instance, a study design of “safety/efficacy study” was categorised as efficacy and safety, whereas a study design of “safety study” was categorised as safety only. We used the information provided in the study design data element to determine random allocation and comparator arms. We used the information provided in the trial primary purpose data element to categorise whether the trial was designed to evaluate an approach to treatment, prevention, or diagnosis or other type of intervention.

Statistical analysis

We conducted a descriptive analysis, estimating the publication rate among completed NIH funded trials registered within ClinicalTrials.gov overall and categorised by trial characteristics. Because our sample captured all completed NIH funded trials registered within ClinicalTrials.gov, we did not use inferential statistics to estimate associations among a larger population, with the exception being an examination of whether trials completed in 2007 or 2008 were published more quickly that those completed in earlier years. This analysis was prespecified and undertaken to evaluate whether or not the increased attention to research dissemination exemplified by enactment of the FDA Amendments Act had an impact on patterns of publication. Our hypothesis was that trials completed after 2007 would be published more quickly. In addition, we evaluated the time to publication using Kaplan-Meier analysis. Finally, among published and unpublished trials, we calculated the total targeted number of enrolled participants using summary statistics. Statistical analysis was performed with JMP 9.0 (SAS Institute, Cary, NC).

Results

Trial characteristics

We identified 635 completed NIH funded clinical trials for inclusion in our analysis. The National Institute of Mental Health (n=76) and the National Institute of Allergy and Infectious Diseases (n=72) funded the most trials in our sample (table). Among all trials, 66% (n=418) were of interventions for treatment and 26% (n=165) were trials of interventions for prevention or diagnostic tests; 45% (n=278) enrolled (actual or target reported) 100 participants or more. Finally, 58% (n=366) were completed during either 2007 or 2008, whereas 42% (n=269) of trials were completed before 2007.

Characteristics of completed NIH funded trials registered within ClinicalTrials.gov* and publication in peer reviewed biomedical journal indexed by Medline within 30 months of study completion, stratified by trial characteristics

	No (%) (n=635)	No (%) published
NIH funding institute†:
National Cancer Institute (NCI)	49 (8)	25 (51)
National Heart, Lung, and Blood Institute (NHLBI)	54 (9)	23 (43)
National Institute of Child Health and Human Development (NICHD)	53 (8)	23 (43)
National Institute on Aging (NIA)	28 (4)	11 (39)
National Institute of Allergy and Infectious Diseases (NIAID)	72 (11)	36 (50)
National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK)	55 (9)	30 (55)
National Institute of Mental Health (NIMH)	76 (12)	27 (36)
National Institute on Alcohol Abuse and Alcoholism (NIAAA)/National Institute on Drug Abuse (NIDA)	87 (14)	44 (51)
Other institute or not specified	161 (25)	75 (47)
Focus:
Treatment	418 (66)	206 (49)
Prevention or diagnostic	165 (26)	70 (42)
Other (such as health services research or basic science)	52 (8)	18 (35)
Purpose‡:
Safety and efficacy	149 (36)	71 (48)
Efficacy only	224 (54)	115 (51)
Safety only	20 (5)	11 (55)
Pharmacokinetics	25 (6)	9 (36)
Conducted under FDA IND or IDE‡:
Yes	79 (22)	48 (61)
No	282 (78)	140 (50)
Phase‡:
I	61 (15)	27 (44)
I/II or II	137 (33)	65 (47)
II/III or III	53 (13)	32 (60)
IV	36 (9)	18 (50)
No phase designated	131 (31)	64 (49)
Design, treatment groups§:
Single arm trial	113 (19)	45 (40)
Multiple arm trial	470 (81)	231 (49)
Design, control group among multiple arm studies§:
Placebo control group	219 (47)	105 (48)
At least one active control group	251 (53)	126 (50)
Design, group assignment among multiple arm studies§:
Randomisation used	451 (96)	221 (49)
Randomisation not used	19 (4)	10 (53)
Data monitoring board§:
Yes	176 (56)	93 (53)
No	140 (44)	64 (46)
Population studied included children:
Yes	153 (24)	68 (44)
No	482 (76)	226 (47)
Population studied included older adults (age ≥65):
Yes	331 (52)	153 (46)
No	304 (48)	141 (46)
Trial enrolment (target or actual):
<100 participants	334 (55)	149 (45)
≥100 participants	278 (45)	133 (48)
Trial location includes at least one US enrolment site:
Yes	524 (92)	241 (46)
No	45 (8)	25 (56)
Trial completion date:
Before 2007	269 (42)	98 (36)
During 2007-8	366 (58)	196 (54)
Trial results posted on ClinicalTrials.gov:
Yes	26 (4)	11 (42)
No	609 (96)	283 (47)

NIH=National Institutes of Health; FDA=Food and Drug Administration; IND=Investigational New Drug Application; IDE=Investigational Device Exemption.

*All data elements not provided for each registered trial.

†Clinical trials cofunded by multiple NIH institutes assigned to institute listed first in ClinicalTrials.gov registration.

‡Data provided only for treatment clinical trials.

§Data provided only for treatment, prevention, and diagnostic clinical trials.

For all trials in our sample, at least 30 months elapsed between completion and our search for a publication in Medline describing the trial’s findings. Publication rates in peer reviewed biomedical journals indexed by Medline increased as more time elapsed after trial completion, as 6% of trials were published within six months of study completion, 15% (n=98) within 12 months, 35% (n=223) within 24 months, and 46% (n=294) within 30 months.

As secondary analyses, we examined time to trial publication after additional periods of time (fig 2). For about 70% (n=444) of the trials in our sample, at least 42 months had elapsed between trial completion and our search for a publication describing the trial’s findings. Among these, 54% (n=241) of trials were published. Similarly, for about 44% (n=278) of the trials in our sample, at least 54 months had elapsed between trial completion and our search for a publication. Among these, 62% (n=173) were published.

Fig 2 Cumulative percentage of studies published in a peer reviewed biomedical journal indexed by Medline during 100 months after trial completion among all NIH funded clinical trials registered within ClinicalTrials.gov

Overall trial publication

Among all trials, the median period of follow-up after study completion was 51 months, (25th-75th centiles 40-68 months). Overall, 68% (n=432) of trials were published in peer reviewed biomedical journals indexed by Medline and 32% (n=203) remained unpublished. Among published trials, the median time to publication was 23 months (14-36 months). Unpublished trials had a cumulative target (or actual) enrolment of about 60 000 participants.

Trial characteristics and publication within 30 months of study completion

Few trials had a publication rate within 30 months in excess of 60% (table). Among treatment trials, 60% of trials registered as either a phase II/III or phase III study and 61% of trials conducted under an FDA Investigational New Drug Application (IND) or Investigational Device Exemption (IDE) were published within 30 months of study completion. Similarly, few trials had a publication rate within 30 months below 40%. Notably, 40% of single arm treatment trials and 39% and 36% of trials funded by the National Institute on Aging and National Institute of Mental Health, respectively, were published within 30 months of study completion. Trials completed in either 2007 or 2008 were more likely to be published within 30 months of study completion when compared with trials completed before 2007 (54% (196/366) v 36% (98/269), respectively, P<0.001; fig 3).

Fig 3 Cumulative percentage of studies published in peer reviewed biomedical journal indexed by Medline during 30 months after trial completion among NIH funded clinical trials registered within ClinicalTrials.gov, stratified by whether trial was completed before 2007 or during 2007-8

Discussion

In a sample of NIH funded clinical trials registered in ClinicalTrials.gov, we found that fewer than half of trials were published in a peer reviewed biomedical journal indexed in Medline within 30 months of trial completion, although there were more recent improvements in timely publication. Furthermore, after a median of 51 months after study completion, we found that about a third of NIH funded trials remained unpublished. Twenty years ago, a single study focused on trials funded by NIH in 1979 found that 93% of completed trials were published within 10 years, using a broader definition of publication that included research abstracts, book chapters, and other non-Medline indexed sources.4 In contrast, a larger body of work focused on industry sponsored studies suggested that between 25% and 50% of clinical trials remain unpublished even several years after completion.1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Our results suggest that patterns of publication are similar for publicly funded and for industry funded clinical trial research and that substantial amounts of publicly funded research data are not published and available to inform future research and practice.

The current culture of research needs to prioritise the timely public dissemination of research findings via peer reviewed journals among research funded by both public and private sources. Sponsors and investigators need to prioritise the publication process, managing it as closely and rigorously as any other part of the scientific process, regardless of the outcome: positive, negative, or inconclusive. In previous years, publishing negative findings from clinical trials was considered far more difficult, and trials with positive findings were more likely to be recommended for publication.26 Two factors, however, facilitate the publication of negative findings. Firstly, the growth in online only journals, such as those offered by the Public Library of Science (PLoS) and BioMed Central (BMC), and the practice of print journals publishing articles “online only” has reduced competition for space among print journals. Secondly, the ICMJE, among its uniform requirements for manuscripts submitted to biomedical journals, has issued a statement compelling the publication of trials with negative results.27

Clearly, many steps occur between the completion of data collection and the publication of the central research findings from a clinical trial, only some of which are under the control of the investigators. Preparations for publication and dissemination, however, can be made in advance of the trial’s completion, including data management and cleaning, analysis planning, and drafting of parts of the manuscript. Among those NIH funded trials that were published in a peer reviewed biomedical journal indexed by Medline, some required as little as a few months for publication, others as long as five years. Trials that required several years to be published could potentially have been moved forward more expeditiously. Clear and reasonable expectations for timely dissemination are needed; perhaps the goal should be to have results from clinical research reported publicly 12-24 months after study completion. Interestingly, in our sample, trials completed in 2007 or 2008 were published more quickly after study completion than trials completed before 2007, suggesting that the timeliness of dissemination could be improving. This issue deserves re-examination in future years.

Implications

To this end, US policies have been recently enacted that might further improve public reporting of results. Enacted in 2007, the FDA Amendments Act requires the sponsors or designated principal investigators of nearly all non-phase I trials of FDA regulated drugs, biologicals, and medical devices to register their trials at inception within ClinicalTrials.gov. Critically, this law also requires public reporting of summary results among trials of FDA approved drugs and devices within a legally mandated timeframe, often within 12 months of trial completion, regardless of publication status. Specifically, investigators must report results for all primary and secondary outcomes to the ClinicalTrials.gov results database for public posting.28 Importantly, the ICMJE has clarified that member journals should not consider reported results to be a previous publication if they were posted in ClinicalTrials.gov.29

Although the act should make findings from NIH funded research more available, there are many NIH funded trials that are not covered by it, such as trials of behavioural interventions and surgical procedures. Results from these trials can be voluntarily submitted to ClinicalTrials.gov, but there is no law or policy requiring reporting or publication of results from these studies. Since April 2008, federal law requires that any published article that describes the results of research funded wholly or in part by the NIH is to be made publicly available no more than one year after the date of publication through PubMed Central, as part of the NIH’s public access programme.30 No policies exist, however, to ensure that the public has access to results from NIH funded research that is not published, although there are policies promoting data sharing among NIH projects receiving more than $500 000 in direct costs per year.31

Lack of dissemination of research findings disrupts the scientific process and leads to redundant efforts and misconceptions about clinical evidence, undermining not only the trial in question but also the evidence available in the medical literature that forms the basis for systematic reviews and meta-analyses, evidence based clinical and policy decisions, and even institutional review board assessments of risks and benefits associated with future research studies. In addition, non-publication violates the commitment made to trial participants, all of whom took part in the research and possibly placed themselves at some risk in an effort to contribute to clinical science. International guidelines on protecting participants in research, such as the Declaration of Helsinki, also recognise the ethical obligation in publishing or otherwise making publicly available “negative and inconclusive as well as positive results.”32 There are many studies “competing” for trial volunteers, yet we found that data from about 30% of participants in trials have not contributed to the publicly available knowledge base available through Medline 30 or more months after study completion. Given the concerns raised about insufficient numbers of volunteers for clinical trial research,33 34 a key step should be to ensure that data from all those who volunteer are published and widely used to inform future research and practice.

ClinicalTrials.gov and its results database offer a unique opportunity and venue for sponsors and investigators to report results from NIH funded clinical trials, as well as all other studies, to the broader public, regardless of whether or not the results are published in a peer reviewed journal. As a central repository already being used for trial registration,24 the site will be familiar to researchers and regulators. Although the legal requirements pertain only to certain interventional studies of drugs and devices, all studies registered in ClinicalTrials.gov are eligible to use the database for reporting summary results. Finally, using the results database ensures that summary data from trials are reported with tabular data forms that can be easily searched and do not limit reporting only to specific results relevant only to particular target audiences.

Limitations

The central limitation to our study is that we focused our search for relevant publications from NIH funded research to peer reviewed biomedical journals indexed by Medline and did not search other databases, such as Embase or research conference proceedings (abstracts). Our search for publications was extensive, however, involving two independent investigators using systematic methods. In addition, some trials might have been made publicly available elsewhere, such as through investigator or institutional websites. Another factor to consider is that even though Medline initiated efforts to index publications using ClinicalTrials.gov NCT numbers in July 2005, the practice didn’t become common until later, perhaps making it more difficult to identify publications from older trials.

There are additional limitations to consider. This study relied on data in ClinicalTrials.gov that are reported by the trial sponsor or responsible investigator. The National Library of Medicine cannot verify the validity of the trial information, and there might be errors in some records. In addition, there is some uncertainty in our estimates of trial enrolment as for some trials within ClinicalTrials.gov only the target enrolment number was reported, which might differ from the actual enrolment number. Moreover, our study was limited to NIH funded trials and cannot be generalised to other international government agencies that fund clinical trial research.

Finally, there are several reasons why our overall observation that 32% of NIH funded trials were not published in a peer reviewed biomedical journal indexed by Medline is a conservative estimate of non-publication. Firstly, we limited our analysis to trials that were registered and had updated their status to completed, indicating that they were no longer recruiting or observing participants. We would expect that some proportion of initiated trials would be registered and have difficulty with recruitment or follow-up but continue to pursue these goals without formally terminating or completing the trial. Secondly, we limited our analysis to trials that were registered and provided a study completion date. Our previous work has shown that completed trials that do not provide study completion dates within ClinicalTrials.gov were less likely to be published than those providing the information.18 In addition, some institutes, such as the National Cancer Institute, were less likely to provide trial completion dates and were thus under-represented in our sample. Finally, we limited our analysis to trials that were completed by December 2008, providing a minimum of two and a half years for investigators to publish the trial findings in a peer reviewed biomedical journal indexed by Medline. We expect lower publication rates among trials that have had fewer than two years to publish the findings.

Conclusion

In conclusion, despite recent improvement in timely publication, overall we found that fewer than half of NIH funded trials in our sample registered within ClinicalTrials.gov were published in a peer reviewed biomedical journal indexed by Medline within 30 months of trial completion. Moreover, after a median of 51 months after trial completion, we found that a third of trials remained unpublished. Although there might be many reasons for lack of publication, the results database at ClinicalTrials.gov offers a complementary method of providing timely public access to study results, although the peer reviewed literature is likely to remain the principal method of communicating with clinicians and policy makers. Steps must be taken to ensure the timely dissemination of publicly funded research so that data from all those who volunteer are available to inform future research and practice.

Previous work, largely focused on industry sponsored studies, suggests that between 25% and 50% of clinical trials remain unpublished even several years after completion

Little is known about the patterns of publication of publicly funded trials

Fewer than half of NIH funded trials registered after September 2005 within ClinicalTrials.gov and completed by December 2008 were published in a peer reviewed biomedical journal indexed by Medline within 30 months of trial completion

After a median of 51 months after study completion, a third of NIH funded trials in our sample remained unpublished