Non-publication of large randomized clinical trials: cross sectional analysis.

OBJECTIVE: To estimate the frequency with which results of large randomized clinical trials registered with ClinicalTrials.gov are not available to the public.
DESIGN: Cross sectional analysis
SETTING: Trials with at least 500 participants that were prospectively registered with ClinicalTrials.gov and completed prior to January 2009. DATA SOURCES: PubMed, Google Scholar, and Embase were searched to identify published manuscripts containing trial results. The final literature search occurred in November 2012. Registry entries for unpublished trials were reviewed to determine whether results for these studies were available in the ClinicalTrials.gov results database. MAIN OUTCOME MEASURES: The frequency of non-publication of trial results and, among unpublished studies, the frequency with which results are unavailable in the ClinicalTrials.gov database.
RESULTS: Of 585 registered trials, 171 (29%) remained unpublished. These 171 unpublished trials had an estimated total enrollment of 299,763 study participants. The median time between study completion and the final literature search was 60 months for unpublished trials. Non-publication was more common among trials that received industry funding (150/468, 32%) than those that did not (21/117, 18%), P=0.003. Of the 171 unpublished trials, 133 (78%) had no results available in ClinicalTrials.gov.
CONCLUSIONS: Among this group of large clinical trials, non-publication of results was common and the availability of results in the ClinicalTrials.gov database was limited. A substantial number of study participants were exposed to the risks of trial participation without the societal benefits that accompany the dissemination of trial results.

Introduction

Randomized clinical trials are a critical means of advancing medical knowledge. Clinical trials depend on the willingness of participants to expose themselves to the risks of randomization, blinding, and unproven interventions. The ethical justification for these risks is that society will eventually benefit from the knowledge gained from the trial.1 Because the risks involved in trial participation may be significant, and because individual trial participants often do not benefit directly from trial participation, substantial safeguards have been implemented to protect the interests of study participants both prior to and during the trial.2 These safeguards take multiple forms, including oversight by institutional review boards, the informed consent process, and data and safety monitoring boards. Until recently, the protection of the interests of study participants after trial completion has received significantly less emphasis. This began to change in 1997 with the signing of the Food and Drug Administration Modernization Act in the United States, which mandated that the US Department of Health and Human Services establish a registry of clinical trials, thereby providing permanent, public access to information on the conduct of both publicly and privately funded clinical trials.3

In 2005 the International Committee of Medical Journal Editors (ICMJE) required that prospective trials involving human participants be registered prior to the beginning of study enrollment in order to be considered for publication in member journals.4 This requirement was later incorporated into the ICMJE’s “uniform requirements for manuscripts submitted to biomedical journals,” along with the updated CONSORT 2010 statement for the reporting of randomized controlled trials.5 The prospective registration of phase II-IV clinical trials subsequently became federal law in the United States in 2007 with the passage of the Food and Drug Administration Amendments Act. This legislation also expanded the scope of ClinicalTrials.gov to include a database of trial results.6 Results from all registered studies may be posted to ClinicalTrials.gov, including studies completed prior to enactment of the Food and Drug Administration Amendments Act. In addition, reporting results is now mandatory for many trials (box).7 Failure to comply with this mandate can result in substantial penalties, including civil fines of up to $10 000 (£6200; €7400) per day and withholding of funds from investigators sponsored by the National Institutes of Health.6

Prospective clinical study of health outcomes

Controlled interventional study in humans

Involves a drug, biological product, or device regulated by the Food and Drug Administration

Not a phase I drug trial or small feasibility device study*

Trial has at least one US site; or trial is conducted under an FDA investigational new drug application or investigational device exemption; or trial involves a drug, biological product, or device that is manufactured in the United States

Initiated after 27 September 2007 or initiated before 27 September 2007 but completed after 26 December 2007

*All pediatric post-marketing surveillance of device studies are included regardless of study size

The registration of clinical trials serves an important role in protecting the interests of study participants after trial completion. In addition to discouraging investigators from preferentially choosing to report statistically significant positive outcomes, trial registration can increase awareness of possible publication bias within the medical literature by allowing the public to compare the subset of trials with published results to the total number of trials that were registered and conducted. Publication bias can distort the apparent efficacy of interventions, which complicates the interpretation of the medical literature.8 9 10 11 12 13 The non-publication of trial data also violates an ethical obligation that investigators have towards study participants.14 15 When trial data remain unpublished, the societal benefit that may have motivated someone to enroll in a study remains unrealized. Systematic trial registration provides a tool that can help to assess both the magnitude and the causes of these problems. We estimated the frequency of non-publication of large randomized clinical trials and, for unpublished trials, determined the frequency with which trial results are unavailable in ClinicalTrials.gov.

Methods

ClinicalTrials.gov query

ClinicalTrials.gov is the largest existing clinical trial registry, with over 150 000 total registered studies from 185 countries as of August 2013. The regulatory background and logistical processes underlying trial registration in ClinicalTrials.gov have been previously described in detail.16 17 18 Individual trial entries in the ClinicalTrials.gov database include information about the trial registration process, study design, and participants’ enrollment. Information about the trial is entered into the database by study investigators or sponsors and may be updated as the trial progresses. We used the “advanced search” function of ClinicalTrials.gov to search for interventional trials with more than one arm that had been registered prior to 1 January 2009 and were closed to ongoing subject enrollment. ClinicalTrials.gov defines closed studies as “clinical studies that are no longer recruiting participants because they have enough participants already, because they are completed, or because they have been stopped for some reason.”19 We included trials only if the ClinicalTrials.gov recruitment status was listed as one of the following: “active, not recruiting,” indicating that participants were no longer being enrolled but that the study was considered to be ongoing; “completed,” indicating that the trial had “ended normally” (that is, per ClinicalTrials.gov, the “last subject, last visit” had occurred); or “terminated,” indicating that the trial had started to enroll participants but was stopped early for any reason. Each of these designations indicates that participants had been enrolled, that enrollment was no longer taking place, and that there were no plans to reopen enrollment in the future.19 We included trials listed as “active, not recruiting” in our analysis because investigators do not consistently update recruitment status after completing enrollment, and this listing persists indefinitely for many trials. Because of the possibility that publication of these trials was delayed owing to ongoing outcome assessments, we performed a planned subgroup analysis involving trials in which the recruitment status was listed as “completed.” We excluded trial protocols that were withdrawn prior to the beginning of study enrolment; non-randomized trials; duplicate registry entries; and studies where the trial registry listed a study completion date after 31 December 2008. Finally, we excluded trials that were registered with ClinicalTrials.gov more than one month after beginning enrolment, in order to ensure that the decision to register a study occurred separately from the decision to publish the study results. This decision was made because it has been shown that in spite of guidelines endorsing prospective trial registration, trials are often not registered for many months after the initiation of participant enrollment.20 In some of these cases, the registration of the trial is performed in preparation for publication and after the decision is made to publish the results. To the extent that this occurs, the inclusion of trials with delayed registration would bias our results. We limited this source of bias by focusing on trials for which registration was not delayed.

No consensus exists as to what constitutes a large randomized trial. Prior publications examining large trials have used cut-off points based on more than 200 participants or more than 1000 participants.21 22 23 We chose a cut-off point of 500 participants because we believe this defined a group of studies that were unlikely to be pilot studies, that trials of this size have a significant potential to inform medical decision making, and that randomized trials in this group are likely to appeal to journal editors as candidates for publication. Consequently, our search only included trials with an anticipated or actual enrollment of at least 500 participants. Additionally, we conducted a sensitivity analysis in which we calculated publication rates using only those trials with 1000 or more participants. At the time of trial registration the enrollment field lists the planned number of study participants; this field may be updated at the end of study recruitment to reflect the actual number of participants enrolled. Because investigators do not consistently update the enrollment field, we use the term “planned or actual enrolment” to refer to the number of participants listed in the enrollment field at the time of data collection for this study.

At the time of the initial registry search from ClinicalTrials.gov we downloaded a dataset containing the condition studied, funding source, study participants, and trial phase, along with other information.

Publication search

The search for published manuscripts occurred between April and November 2012. We considered a trial to be published when a journal had published a peer reviewed manuscript, either online or in print, which included any outcome data from the study in question. We classified a trial as unpublished if a manuscript containing trial results was not identified after different investigators performed three independent searches. One of these searches was conducted by a health sciences research librarian (LH or KC). The last search for each trial occurred in November 2012, thereby allowing an elapsed time of at least 46 months between trial completion and the final literature search.24

The search for manuscripts containing trial results began with a review of the ClinicalTrials.gov entry. ClinicalTrials.gov encourages trial investigators to update each registry profile with a link to PubMed indexed manuscripts containing results from the registered trials. Additionally, ClinicalTrials.gov uses the unique trial identification number (NCT number) assigned to each entry to automatically identify and link to relevant PubMed entries. For registry entries that did not include a link to a publication containing trial results, we searched Medline via PubMed by trial registration number, keywords, trial title, and investigator’s name for matching manuscripts. For those trials for which no published results were identified, we repeated the search strategy using Google Scholar and then Embase. We assessed matches between registry entries and publications identified by this search strategy by consensus based on the following trial characteristics: study title, trial design, interventions, primary and secondary outcomes, number of participants, recruitment dates, location, and funding sources. We excluded non-peer reviewed publications, including conference proceedings and publications that described study methods without reporting trial results. We considered trials that were only reported as part of a pooled analysis to be published if the manuscript included any unpooled results from the trial in question.

ClinicalTrials.gov provides an optional feature allowing investigators to include their name and contact information as part of the trial registry. When a contact person was listed in ClinicalTrials.gov and no publication was identified using the described search strategy, a study author (CWJ) attempted to contact that investigator once by email to inquire if results had been published. If the investigator indicated that non-publication was due to failed recruitment then we removed the trial from the analysis because our intent was to estimate the frequency of non-publication among trials for which enrollment was successfully completed.

Trial results

For those studies classified as unpublished at the conclusion of our literature search, we reviewed the ClinicalTrials.gov entry for results. This review was performed by a research assistant (LGK) with extensive prior research experience using ClinicalTrials.gov, who underwent training specific to the results database at the start of the present study. We considered only results uploaded prior to 1 November 2012 (corresponding to our final PubMed literature search), because only these results were known to be available in the absence of results in a peer reviewed publication.

When a trial is registered on ClinicalTrials.gov, investigators may list both primary and secondary outcome measures. We recorded whether results were provided for each of the primary outcomes detailed in the original registry. When investigators made the results of statistical analyses available for primary outcomes, we recorded whether results were statistically significant in favor of the interventions (P<0.05 was considered significant unless otherwise specified by study investigators). We assessed analyses identified as non-inferiority or equivalence tests against the non-inferiority margin specified by study investigators. Finally, for each study we determined whether results were provided for safety endpoints or adverse events.

Statistical analyses

We used χ2 tests to compare publication rates between trials grouped according to prespecified characteristics, including funding source, recruitment status, and completion date. P values of <0.05 were considered statistically significant. We used Kaplan-Meier methods to estimate cumulative publication percentages over time since trial completion, with all unpublished studies censored on the date of the final manuscript search. Statistical analyses were performed using SPSS version 20.0 (IBM, Armonk, NY).

Results

Overall, 585 randomized trials met the inclusion criteria (fig 1, see supplementary file). According to available registry data, these trials included 990 136 planned or actual study participants. Median planned or actual enrollment was 800 participants per trial (interquartile range 601-1239). The earliest registry entry was registered with ClinicalTrials.gov on 1 November 1999, and the median registration date was 25 April 2006. At least one publication containing trial results was identified for 410 of these studies during our literature search. An additional four published manuscripts were identified following email correspondence with study investigators, so published manuscripts were identified for 414 of the 585 included trials (71%). The 171 unpublished trials (29% of total) had a planned or actual enrollment of 299 763 participants. Table 1 shows the characteristics of both published and unpublished trials. Phase III studies accounted for 67% of included trials (394/585). Three studies exclusively enrolled patients aged over 65, all of which were published trials. Sixty six studies exclusively enrolled patients under 18; 16 of these trials (24%) were not published. Eleven per cent (n=64) of studies received funding from the National Institutes of Health or other federal sources; 80% (n=468) of studies were supported by industry. Non-publication was less common among trials supported by the National Institutes of Health or other federal sources compared with those without federal funding (17% v 31%, P=0.025). Non-publication was more common among trials that did than did not receive industry funding (32% v 18%, P=0.003).

Fig 1 Flowchart of included trials

Table 1 Characteristics of large randomized trials registered with ClinicalTrials.gov prior to 2009. Values are numbers (percentages) unless stated otherwise

Trial characteristics	All trials (n=585)	Published trials (n=414)	Unpublished trials (n=171)
Mean (SD) No of participants	1693 (5580)	1668 (3217)	1753 (9045)
Trial phase*:
I/II	84 (14)	43 (10)	41 (24)
III	394 (67)	290 (70)	104 (61)
IV	58 (10)	40 (10)	18 (11)
Other	49 (8)	41 (10)	8 (5)
Funding source†:
Industry	468 (80)	318 (77)	150 (88)
NIH/US government	64 (11)	53 (13)	11 (6)
Other	104 (18)	84 (20)	20 (12)
Trial completion date:
Unlisted	43 (7)	25 (6)	18 (11)
Prior to 2005	11 (2)	8 (2)	3 (2)
2005-06	93 (16)	71 (17)	22 (13)
2007	192 (33)	137 (33)	55 (32)
2008	246 (42)	173 (42)	73 (43)

NIH=National Institutes of Health.

*Trials described as “phase II/III” in their registry entry are categorized as phase III.

†Trials with multiple funding sources are listed within all relevant categories; totals therefore add to more than 100%.

For trials in which the ClinicalTrials.gov recruitment status was listed as “completed,” 26% (132/513) remained unpublished. Twenty nine trials were described as “active, not recruiting”; 10 of these (34%) had not been published. Finally, 67% (29/43) of trials that had been terminated remained unpublished. Among the subset of 225 trials with at least 1000 planned or actual participants, 53 studies (24%) were unpublished.

Among trials with a completion date listed in ClinicalTrials.gov (n=542) and that had been published (389/542, 72%), the median time from study completion to publication was 27 months (interquartile range 20-37 months). The observed rate at which these trials were published was highest between approximately 24 and 48 months after study completion (fig 2). The mean time between trial completion and the final manuscript search was similar for published trials (61 months, SD 13 months) and unpublished trials (60 months, SD 12 months).

Fig 2 Kaplan-Meier estimate of cumulative publication percentage by time elapsed from trial completion to publication

Of the 171 trials that remained unpublished as of 1 November 2012, 38 (22%) had results available on ClinicalTrials.gov (table 2). From the initial group of 585 trials, 414 (71%) had published results, 38 (6%) were unpublished but had results available on ClinicalTrials.gov, and 133 (23%) had no results available either in published form or in ClinicalTrials.gov. Fifteen (39%) of the 38 trials with results reported also provided statistical analyses for at least one primary outcome; the remaining 23 trials reported results without including statistical analyses. Nine of these 15 (60%) reported statistically significant results clearly favoring the intervention, whereas another four (27%) reported meeting a standard of non-inferiority. Of 38 trials with results provided, 37 (97%) included information regarding safety endpoints or adverse events.

Table 2 Characteristics of unpublished trials with and without results available in the ClinicalTrials.gov results database. Values are numbers (percentages) unless stated otherwise

Trial characteristics	Results in ClinicalTrials.gov
	Available (n=38)	Not available (n=133)
Mean (SD) No of participants	1199 (1049)	1911 (10 244)
Trial phase*:
I/II	5 (13)	36 (27)
III	24 (63)	80 (60)
IV	8 (21)	10 (8)
Other	1 (3)	7 (5)
Funding source†:
Industry	38 (100)	112 (84)
NIH/US government	0 (0)	11 (8)
Other	0 (0)	20 (15)
Trial completion date:
Unlisted	0 (0)	18 (11)
Prior to 2005	0 (0)	3 (2)
2005-06	3 (8)	19 (14)
2007	5 (13)	50 (38)
2008	30 (79)	43 (32)

NIH=National Institutes of Health.

*Trials described as “phase II/III” in their registry entry are categorized as phase III.

†Trials with multiple funding sources are listed within all relevant categories; totals therefore add to more than 100%.

Of the 150 unpublished industry funded studies, 38 (25%) had results available on ClinicalTrials.gov. None of the 11 unpublished trials funded by the National Institutes of Health or the 20 unpublished trials with other funding sources provided results in the results database. Of 23 unpublished trials completed during or after September 2008 (when the ClinicalTrials.gov results database became functional), 10 (43%) provided results. In total, 237 trials were listed as either being started after 27 September 2007 or completed after 26 December 2007, dates that correspond to the mandatory results reporting period established by the Food and Drug Administration Amendments Act. Of these, 67 (28%) were not published and 38 (16%) were neither published nor had results available in the ClinicalTrials.gov results database. Among all unpublished trials with posted results, the median delay between trial completion and results becoming available was 22 months (interquartile range 13-37 months).

Discussion

Among large randomized trials registered with ClinicalTrials.gov and closed prior to 2009, non-publication was common. We identified an estimated 250 000 trial participants for whom we were unable to find results either in the published literature or in the result database of the registry, which was approximately 26% of the total participants in the included trials. Our findings are consistent with previously observed rates of non-publication among trials representing a broad spectrum of clinical topics and funding sources.9 21 24 25 26 27 28 29 30 31 32 Our results add to existing work by showing that non-publication is an important problem even among large randomized trials, and by providing an estimate of the number of trial participants who accepted the risks of participation in these trials but for whom no results are publicly available. The present study also shows that among large unpublished trials completed prior to 2009, utilization of the results database remains limited. Even among studies completed after the results database became available in 2008, ClinicalTrials.gov contained results less than half of the time.

Prior investigations have shown evidence of publication bias against negative studies as one factor associated with non-publication.9 11 32 33 34 However, it has previously been unclear whether this bias is primarily due to limited motivation on the part of study investigators and sponsors to publish negative data or because journals are less interested in publishing the results of negative trials. By focusing our investigation on studies with at least 500 participants, we greatly limited the possibility that non-publication of trials in this cohort was due to rejection of manuscripts by journals or a lack of time or interest on the part of investigators or sponsors.35 Because of the considerable time and expense required to run these trials and the appeal of such studies to editors, non-publication of the results of large randomized trials is likely in most cases to be a conscious decision made by study investigators and sponsors. Reasons why investigators and sponsors might choose not to publish results include a discrepancy between observed and desired results or the protection of intellectual property rights.36 37 The considerable rate of non-publication even among this group of large trials suggests that decisions by investigators or sponsors are a significant cause of non-publication of clinical trials.

ClinicalTrials.gov has had the capacity to display trial results since September of 2008.38 While this is an important development, there is evidence that compliance with even basic requirements for trial registration still remains poor.20 39 40 41 42 43 44 Furthermore, use of the ClinicalTrials.gov results database is limited, even among trials for which results reporting is required by the Food and Drug Administration Amendments Act.45 Rates of compliance with the ClinicalTrials.gov results database have not been previously described among unpublished trials; our findings suggest that among these studies, utilization of the results database also remains limited. These findings have important implications for those who perform comprehensive literature searches. In over 6% of the large clinical trials we studied, results were available within the ClinicalTrials.gov results database but not within the published literature. Consequently, for authors of systematic reviews, meta-analyses, and clinical guidelines, the ClinicalTrials.gov results database is an important source of data that may otherwise be unavailable. Additionally, for another 23% of the studies in this analysis, results were not available in either the ClinicalTrials.gov database or in published form. For these trials, searching ClinicalTrials.gov may be the only way to determine that an unpublished study was performed. Even without access to results from these trials, information on the number and size of unpublished trials provides valuable information about the possibility of publication bias. Finally, all 38 of the unpublished trials we studied with results available on ClinicalTrials.gov were sponsored by industry. While the sponsors and investigators of these trials deserve credit for utilizing the results database, this represents a limited percentage of the unpublished trials in this group, meaning that currently the availability of results on ClinicalTrials.gov may also be affected by biases similar in nature to the publication and outcome reporting biases that affect the published literature.

Even when study investigators and sponsors make their results available in ClinicalTrials.gov, this should not absolve them from their obligation to publish data in a peer reviewed format. The peer review process plays a critical role in improving the quality of trial reporting. For example, peer review serves to identify potential sources of study bias, helps to ensure statistical rigor, and assists authors and editors with the task of placing study results into context.46 47 While the ability of ClinicalTrials.gov to display results is an important step forward, this capability is not a substitute for the peer review process. Further, publication in the peer reviewed literature also greatly facilitates access to the results by the scientific community and provides readers with the details needed to appropriately incorporate study results into systematic reviews and meta-analyses.48 49

Trial investigators and sponsors have an ethical obligation to study participants to publish trial results.50 This principle is implicit in the US Federal Policy for the Protection of Human Subjects, also known as the “Common Rule,” which outlines the scope and responsibilities of institutional review boards for overseeing research using human participants. The Common Rule states that institutional review board approval requires demonstration that “risks to subjects are reasonable in relation to anticipated benefits, if any, to subjects, and the importance of the knowledge that may reasonably be expected to result.”51 Similarly, the Declaration of Helsinki, which was instrumental in developing the modern system of oversight by institutional review boards, also acknowledges the importance of disseminating research results, stating “Authors have a duty to make publicly available the results of their research on human subjects and are accountable for the completeness and accuracy of their reports.”52 By directing institutional review boards to assess the societal importance of resulting knowledge in addition to the possible risks and harms to individual research participants, the Common Rule provides justification for institutional review board oversight of results reporting, including trial registration and publication.53 Because the involvement of institutional review boards with clinical trial oversight begins prior to participant enrollment, these institutions are uniquely positioned to protect the rights of study participants throughout all stages of trial conduct, from study planning to reporting results. Given the persistent problem of unpublished trial results despite continued emphasis on trial registration from governmental agencies, funding organizations, and the editorial community, increased institutional review board attention toward this issue may be needed.

Limitations of this study

Several study limitations should be considered when interpreting these results. Firstly, it is possible that we failed to identify published results for some of the trials that were classified as unpublished. We attempted to limit this risk by performing multiple searches for each trial, utilizing multiple databases for each search, and by attempting to contact study investigators before classifying any trial as unpublished. Given the considerable efforts taken to identify publications, any that we missed are sufficiently well hidden that they are unlikely to be identified during routine or even systematic searches. It is also likely that results from some of the currently missing trials will be published in the future, though our data show that publication more than 46 months after trial completion is rare.

An additional limitation of the present study is that registry information contained within ClinicalTrials.gov is inconsistently updated by trial investigators. Trial recruitment status, for example, is not consistently updated by investigators as trials transition from active participant enrollment to closed enrollment with ongoing data analysis to study completion. For this reason, we included in our analysis trials listed by ClinicalTrials.gov as “active, not recruiting.” The publication of these trials may be delayed pending the completion of data gathering and analysis. However, the non-publication rate of these trials (34%) was only slightly higher than that for trials known to have been completed (26%), indicating that outcome data were available for most of these studies, and that studies listed as “active, not recruiting” often proceed to publication without a corresponding change in the registered enrollment status. Study enrollment figures are also inconsistently updated by investigators. Consequently, some registry entries still contain pretrial values for the number of planned study participants rather than updated enrollment information, despite the completion of trial enrollment and follow-up. It is often not possible to tell whether trial enrollment refers to planned or actual enrollment numbers. For this reason, our estimate of the number of people who participated in studies for which results are unavailable should be considered an approximation.

To allow adequate time for study investigators to publish their results, we report on trials that were closed prior to 1 January 2009. It is possible that publication rates are different among more recently conducted trials. Finally, this analysis addresses publication rates among large registered trials. Publication rates may be different for smaller trials and those not registered in ClinicalTrials.gov.

Conclusions

We observed that non-publication is common among large randomized clinical trials. Furthermore, the sponsors and investigators of these unpublished trials infrequently utilize the ClinicalTrials.gov results database. The lack of availability of results from these trials contributes to publication bias and also constitutes a failure to honor the ethical contract that is the basis for exposing study participants to the risks inherent in trial participation. Additional safeguards are needed to ensure timely public dissemination of trial data.

Non-publication of clinical trial results is common among studies across a wide spectrum of both clinical topics and funding characteristics

Utilization of the ClinicalTrials.gov results database by investigators is also limited, even among trials for which results reporting is required by US law

Little is known about publication or availability of results in the ClinicalTrials.gov database for large clinical trials, the subset of trials with the greatest potential impact on clinical care

Among trials with at least 500 participants that were prospectively registered in ClinicalTrials.gov and completed before January 2009, 23% were neither published nor had results available in the ClinicalTrials.gov database a median of 60 months after trial completion