Publication of interventional phase 3 and 4 clinical trials in radiation oncology: an observational study.

OBJECTIVES: Clinical trials produce the best data available for decision-making in modern evidence-based medicine. We aimed to determine the rate of non-publication of interventional phase 3 and 4 clinical trials involving patients with cancer undergoing radiotherapy.
SETTING: The ClinicalTrials.gov database was searched for interventional phase 3 and 4 trials in radiotherapy with a primary completion date before 1 January 2013. We determined how many of these registry entries have not published the compulsory deposition of their results in the database and performed a systematic search for published studies in peer-reviewed journals.
RESULTS: Of 576 trials, 484 (84.0%) did not deposit a summary result in the registry. In addition, 44.9% of them did not publish their results in a peer-reviewed journal. Similar percentages were found for most cancer subtypes: brain (41%), breast (38%), cervical (66%), colorectal (38%), lung (48%), prostate (45%), bladder (56%), head and neck (56%) and lymphoma (33%).
CONCLUSION: Our results show that most trials in radiation oncology did not report the results in the registry. Almost half of these trials have not been published in the biomedical literature. This means that a large number of study participants were exposed to the risks of trial participation without the supposed benefits that sharing and publishing of results would offer to future generations of patients. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

We have considered and analysed the higher levels of evidence-based radiation oncology.

Each trial meeting the inclusion criteria was independently searched by at least two authors in order to assess its publication in a peer-reviewed journal.

ClinicalTrials.gov is, by far, the largest trial registry in the world. Any applicable medical device trial or medical drug trial planned to be market in the USA has to be registered in this registry.

Insufficient statistical power and lack of data to test for hypotheses giving a plausible explanation of non-publication in radiation oncology.

Background

Clinical trials produce the best data available for decision-making in modern evidence-based medicine. All this evidence should be both published and available, since withholding results skews the evidence and therefore dangerously distorts it. Publication of all trials conducted in radiation oncology is needed to fully determine the benefits and risks of treatments currently in use in our clinics.

Since 2005, the International Committee of Medical Journal Editors has required prospective registration of all interventional clinical studies prior to publication. It does not, however, require authors to report the results of registered trials.1 On the other hand, the US federal law, the Food and Drug Administration Amendments Act of 2007 (FDAAA 801),2 requires responsible parties of all interventional trials to submit summary results to the ClinicalTrials.gov database 12 months after the primary completion date (PCD); PCD is the term used at ClinicalTrials.gov for the ‘completion date’, as defined in FDAAA 801. Furthermore, this summary must be made publicly available, keeping with the Declaration of Helsinki, which makes it an ethical obligation to make the results of all medical research involving human subjects publicly available.3

In this work, we answered two important questions regarding the state of the evidence in radiation oncology. The first was ‘Were the trials conducted in radiation oncology in compliance with the US law and therefore did they make their results publicly available?’ The second was ‘How many of the trials conducted in radiation oncology have published their results in a peer-reviewed journal (PRJ)?’ The answers to both questions are vital to our patients, to our healthcare system (independently of the model a country has chosen as its own), and to the state of evidence we have within our reach as practitioners (are our treatments really based on evidence?).

Methods

Data source

ClinicalTrials.gov is a clinical trial registry and results database that provides the public with access to registrations and summary results information for clinical studies. This registry is maintained by the National Library of Medicine at the National Institutes of Health (NIH). As is often stated, this registry represents nowadays the most comprehensive source for information about ongoing and completed trials within and outside the USA, and we consequently chose it to conduct this research.4

Database search

We searched the ClinicalTrials.gov database for trials in radiotherapy as of 6 May 2016 that had a PCD between 1 January 2008 and 1 January 2015. We chose this date because we had to allow a minimum 12-month period for publication of the compulsory summary results in the registry (16 months in our case). When a PCD was missing, we instead used the completion date field. We used the ‘Advanced Search’ form to broaden our search (box 1).

Search terms: “radiotherapy” OR “radiation therapy” OR brachytherapy” OR “IMRT” OR “SBRT” OR “IMPT” OR “radiation oncology” (IMRT stands for intensity-modulated radiation therapy; SBRT stands for stereotactic body radiation therapy; IMPT stands for intensity-modulated proton therapy)

Study type: interventional studies

Study results: all studies

Recruitment: all studies

Additional criteria → phase: no phase was ticked since phase 3 or 4 trials concerning radiation therapy were also registered as trials without phase

For this study and within the aforementioned date range, we considered all clinical trials that met the criteria shown in box 2.

Study type: interventional studies

Interventions: radiotherapy as standard treatment or primary focus in oncology

Phase: phase 3; phase 4

Trials with a ‘Withdrawn’ status were excluded because these trials have ended early before enrolling the first patients.

Each trial registered on ClinicalTrials.gov has a unique identification code, ‘NCT’, followed by an eight-digit number. This identifier is commonly known as the NCT number. We used this NCT number to avoid trial duplicates within our final set. In order to avoid false positives, for each trial, we extracted all the information provided by ClinicalTrials.gov’s application programming interface (see table 1). We also used the uniform resource locator field in order to access all the trial information registered in the database. Two researchers (JP-A and PG) independently reviewed the information displayed by using the same search protocol and decided for each trial whether the criteria mentioned above were fully met, with a consensus discussion in case of disagreement. If they failed to reach a consensus, a third researcher (IL) took a final decision after taking into account both arguments.

Table 1

Information extracted for each interventional phase 3 and 4 trials

Information extracted
NCT number	Gender	Other IDs	Results first received
Title	Age groups	First received	Primary completion date
Recruitment	Phases	Start date	Outcome measures
Study results	Enrolment	Completion date	URL
Conditions	Funded bys	Last updated
Interventions	Study types	Last verified
Sponsor/collaborators	Study designs	Acronym

ClinicalTrials.gov’s API Information.

URL, Uniform Resource Locator.

Finally, we analysed the ‘Study Results’ field and differentiated between those studies with a ‘Has Results’ tag from those with a ‘No Results Available’ tag (see figure 1).

Figure 1

Database search. PCD, primary completion date.

Publication search in a PRJ

Because our query on ClinicalTrials.gov was conducted on 6 May 2016, we allowed a minimum of 24 months after the latest possible PCD (6 May 2014) for journal submission, peer review and editorial process until the trial was finally published in a PRJ. For those trials published electronically ahead of print, we used the date on which online publication occurred. Trials with a ‘Suspended’ or ‘Terminated’ status were excluded from this search (see figure 2).

Figure 2

Publication search in a peer-reviewed journal (PRJ). PCD, primary completion date.

Our clinical trial set was divided into four subsets. Each subset was given to a particular researcher (JP-A, PG, IL and EA). A trial was considered published if it met the criteria shown in box 3.

The trial was published in a PRJ

Results reported in the publication were a primary outcome measure or a secondary outcome measure, or both

No abstract, poster, oral communication or private communication of a trial result was considered as a valid publication

Each author searched PubMed, Google Scholar and Google by using the following characteristics: NCT number, other identification numbers provided by ClinicalTrials.gov, author names, institutions, title, official title and keywords. Matches were evaluated according to title, trial design, sample size, intervention, location, dates of recruitment and completion, study hypotheses, and primary and/or secondary outcome measures, as described in the ClinicalTrials.gov database. Matches found by each researcher were always checked by a second researcher. We then categorised our data into subsets by cancer subtype.

ClinicalTrials.gov also displayed publication citations at the bottom of the ‘Full Text View’ tab of a study record, under the ‘More Information’ heading. These citations are either submitted by sponsors or investigators, or are automatically indexed by ClinicalTrials.gov. Citations submitted by sponsors or investigators may provide background information instead of information about results. We also reviewed this linked information to evaluate whether or not the information provided by sponsors or indexed by ClinicalTrials.gov was relevant to our study. We applied the same methodology as explained in the previous paragraph.

In order to look for publication bias, we took into account all trials with results in the registry that qualified for a search in a PRJ. This set was further divided into two subsets: the first contained all trials with a summary result reported in the registry and no publication in a PRJ; the second contained all trials with a summary result reported in the registry and a publication in a PRJ. For each subset, we further analyse positive and negative result frequencies. A positive finding was defined as a result rejecting the null hypothesis in favour of the experimental arm; a negative finding, on the other hand, was defined as a result that either confirmed the null hypothesis or rejected it in favour of the control arm.

Statistical analysis

We used the χ2 test to compare publication rates in the registry between trials grouped by funding type. p Values of <0.05 were considered statistically significant. We also used the χ2 test to compare publication rates in a PRJ between trials grouped by funding type. To test for the effect of this variable on publication, we used adjusted binary logistic regression (non-publication vs publication), which produced an OR and a 95% CI; an OR larger than 1.0 indicated a greater likelihood of trial publication in this group. The main explanatory variable was funding status adjusted for number of patients in the trial and the country of the principal investigator (American Institution vs Other). These analyses was prespecified and undertaken to evaluate whether or not industry funding, enrolment or country had an impact on patterns of publication. Statistical analyses were performed by using R version V.3.3.15

Results

Overall, 583 interventional phase 3 and 4 clinical trials met the inclusion criteria. Of these 583 trials, 7 had a ‘Withdrawn’ status and were consequently excluded. Fifty-one were phase 4 trials with the remaining 525 phase 3 trials. A total of 484 (84.0%) of all the interventional phase 3 and 4 clinical trials did not publish the compulsory summary results in the ClinicalTrials.gov registry. NIH funding was significantly associated with a higher likelihood of reporting results (OR 3.23, 95% CI 1.89 to 5.57; p<0.001). Industry funding was likewise significantly associated with a higher likelihood of reporting results in the registry (OR 3.43, 95% CI 1.93 to 6.08; p<0.001). No statistically significant differences were found between NIH-funded trials and industry-funded trials (OR=1.14, 95% CI 0.64 to 2.04, p=0.66) (see table 2 and figure 3). Although we had focus in funding as our explanatory variable, we have also observed that ‘being from an American Institution’ (in principal investigator variable) was significantly associated with a higher likelihood of reporting results when adjusted by funding type and enrolment (see table 3).

Table 2

Number of trials with results not posted on ClinitalTrials.gov registry. Funded feature is not an exclusive one: trials might have been funded by a combination of the three possible options (National Institutes of Health (NIH), industry and other)

	No of trials	Results not posted on ClinicalTrials.gov registry
Phase 3	525	438 (83.4%)
Phase 4	51	46 (90.2%)
NIH funded	146	93 (63.7%)
Industry funded	85	56 (65.9%)
Other funded	502	450 (89.6%)
Total	576	484 (84.0%)

Figure 3

Distribution of trials is described in table 2. NIH, National Institutes of Health.

Table 3

Adjusted binary logistic regression (non-publication vs publication in ClinicalTrials.gov) by funding type, adjusted for the country of the principal investigator and enrolment

	Being from an American Institution p value and OR (95% CI)	Enrolment p value and OR (95% CI)
NIH funded	p<0.001, OR 3.54 (2.06 to 6.16)	p=0.011, OR 1.00 (1.00 to 1.00)
Industry funded	p<0.001, OR 5.98 (3.68 to 9.94)	p=0.06, OR 1.00 (0.99 to 1.00)
Other funded	p<0.001, OR 6.70 (3.99 to 11.58)	p=0.27, OR 1.00 (0.99 to 1.00)

NIH, National Institutes of Health.

When categorised by phase, 46 (90.2%) phase 4 trials and 438 (83.4%) phase 3 trials did not publish a deposition of their results in the registry, although this percentage difference was not significant (OR 1.75, 95% CI 0.68 to 5.99; p=0.301).

Overall, 463 interventional phase 3 and 4 clinical trials met the criteria for searching a publication in a PRJ (43 phase 4 trials and 420 phase 3 trials). A total of 255 (55.1%) trials each had at least one publication of their results in a PRJ, but 208 (44.9%) trials remained unpublished. Median and mean time to publication was 60 months. NIH funding was significantly associated with a higher likelihood of published results (OR 3.17, 95% CI 1.85 to 5.55; p<0.001). Industry funding was not significantly associated with a higher or lower likelihood of publishing results in a PRJ (OR 1.14, 95% CI 0.67 to 1.98; p=0.63) (see table 4 and figure 4). ‘Being from an American Institution’ was not significantly associated with a lower or higher likelihood of publishing results when adjusted by funding type and enrolment (see table 5).

Table 4

Number of trials with results not published on a peer-reviewed journal (PRJ). As given in table 1, the funded feature is not exclusive, and there might be trials which were funded by a combination of the three possible options (National Institutes of Health (NIH), industry and other)

	No of trials	Results not published on PRJ
Phase 3	420	181 (43.1%)
Phase 4	43	27 (62.8%)
NIH funded	113	30 (26.5%)
Industry funded	64	26 (40.6%)
Other funded	412	189 (45.9%)
Total	463	208 (44.9%)

Figure 4

Distribution of trials is described in table 4. NIH, National Institutes of Health.

Table 5

Adjusted binary logistic regression (non-publication vs publication in peer-reviewed journal) by funding type, adjusted for the country of the principal investigator and enrolment

	Being from an American Institution p value and OR (95% CI)	Enrolment p value and OR (95%CI)
NIH funded	p=0.691, OR 0.91 (0.56 to 1.46)	p=0.07, OR 1.00 (0.99 to 1.00)
Industry funded	p=0.052, OR 1.50 (1.00 to 2.26)	p=0.087, OR 1.00 (0.99 to 1.00)
Other funded	p=0.054, OR 1.49 (0.99 to 2.25)	p=0.117, OR 1.00 (0.99 to 1.00)

NIH, National Institutes of Health.

Taking into account the trial phase, 27 (62.8%) phase 4 trials and 181 (43.1%) phase 3 trials remained unpublished. This difference between phase 3 and phase 4 trials was statistically significant (OR=2.23, 95% CI 1.18 to 4.34; p=0.02).

Of these 463 trials, when taking into account cancer subtype, we found the following percentages for unpublished results in a PRJ (total number of unpublished trials is shown in parentheses): 41.2% for brain (14 of 34), 37.9% for breast (25 of 66), 61.1% for cervical (11 of 18), 37.9% for colorectal (11 of 29), 33.3% for endometrial (3 of 9), 75% for oesophagus (3 of 4), 62.5% for eye (5 of 8), 37.5% for gastric (3 of 8), 55.6% for head and neck (47 of 84), 100.0% for kidney (2 of 2), 36.0% for leukaemia (9 of 25), 50.0% for liver (4 of 8), 48.1% for lung (25 of 52), 100.0% for melanoma (1 of 1), 66.7% for myeloma (2 of 3), 80% for metastasis (4 of 5), 36.4% for pancreatic (4 of 11), 45.2% for prostate (19 of 42), 55.6% for bladder (5 of 9), 33.3% for lymphoma (7 of 21), 33.3% for sarcoma (4 of 12), 61.5% for other (8 of 13). For all subgroups, we ran a significance test to determine whether these percentages were different from the global non-publication tendency. As can be seen in table 6, no statistically significant difference was found in any of them with the exception of head and neck which showed slightly worse numbers.

Table 6

Number of trials with results not published in a peer-reviewed journal (PRJ) by cancer subtype. For those subgroups with at least 16 trials, we run a significant test in order to see if these percentages were different from the global non-publication tendency. For each cancer subtype, OR were calculated taking as reference the global set minus this cancer subtype subset

	No of trials	Results not published on PRJ	OR (95% CI)	p Value
Brain	34	14 (41.2%)	0.85 (0.42 to 1.72)	0.65
Breast	66	25 (37.9%)	0.71 (0.42 to 1.22)	0.21
Cervical	18	11 (61.1%)	1.98 (0.75 to 5.20)	0.16
Colorectal	29	11 (37.9%)	0.74 (0.34 to 1.59)	0.43
Endometrial	9	3 (33.3%)	0.61 (0.15 to 2.46)	0.48
Oesophagus	4	3 (75%)	3.72 (0.38 to 36)	0.22
Eye	8	5 (62.5%)	2.07 (0.49 to 8.76)	0.31
Gastric	8	3 (37.5%)	0.73 (0.17 to 3.10)	0.67
Head and neck	84	47 (55.6%)	1.72 (1.07 to 2.77)	0.03
Kidney	2	2 (100.0%)	NaN	0.12
Leukaemia	25	9 (36.0%)	0.68 (0.29 to 1.56)	0.36
Liver	8	4 (50.0%)	1.23 (0.30 to 4.98)	0.77
Lung	52	25 (48.1%)	1.15 (0.65 to 2.06)	0.63
Melanoma	1	1 (100.0%)	NaN	0.27
Metastasis	5	4 (80.0%)	4.98 (0.55 to 44.9)	0.11
Myeloma	3	2 (66.7%)	2.47 (0.22 to 27.39)	0.44
Pancreatic	11	4 (36.4%)	0.69 (0.20 to 2.41)	0.56
Prostate	42	19 (45.2%)	1.01 (0.54 to 1.92)	0.97
Bladder	9	5 (55.5%)	1.55 (0.41 to 5.83)	0.52
Lymphoma	21	7 (33.3%)	0.60 (0.24 to 1.51)	0.27
Sarcoma	12	4 (33.3%)	0.61 (0.18 to 2.04)	0.41
Other	13	8 (61.5%)	2 (0.64 to 6.21)	0.22

For publication bias, only 67 trials (14.4%) met the criteria: 18 trials reported a summary result but were not published in a PRJ, and 49 trials reported a summary result and were published in a PRJ. For our first subset, 8 of 18 trials (44.4%) showed a positive finding and the remaining 10 (55.6%) a negative finding; the second subset showed a similar pattern: 24 of 49 (49.0%) had a positive finding and the remaining 25 (51%) a negative finding (table 7).

Table 7

Number of trials meeting the inclusion criteria for analysing the publication bias

	No of trials	Positive results	Negative results
Results published on PRJ	18	8 (44.4%)	10 (55.6%)
Results not published on PRJ	49	24 (49.0%)	25 (51%)
Results	67	32 (47.8%)	35 (52.2%)

PRJ, peer-reviewed journal.

Discussion

Clinical trials produce the best data available for decision-making in modern evidence-based medicine. All evidence should be both published and available because withholding the results skews the evidence and therefore dangerously distorts it. When evidence is not published, those who make decisions about potential treatments do not have complete information about the outcome and the entire set of benefits and risks that a particular treatment might involve. The importance of publishing negative results has not been stressed strongly enough6; publishing these results reduces biases regarding the efficacy of a treatment and plays a huge role in helping science to move forward. Perhaps the most famous example of a negative result was the historic paper published by Michelson and Morley in 1883,7 which led a young physicist working at a patent office in Bern 22 years later, in 1905, to completely change our notion of space and time—a notion that almost one hundred years later turned out to be an essential feature in the GPS system. This young physicist was Albert Einstein. Despite the importance of knowing whether there is publication bias in radiation oncology, the present work confirms that it is not possible to assess such bias because of a massive lack of data: a mere 15% of the trials registered at ClinicalTrials.gov had published the compulsory summary result and only 45% of all trials conducted had been published in a PRJ. Rates of publication in radiation oncology were nonetheless higher than those previously reported 3 years ago in a cross-sectional analysis of large randomised clinical trials in medicine, although comparisons are hard to make because our work is an observational study in a specific medical field with substantially different inclusion criteria.8

As our results showed, a large number of interventional phase 3 and 4 trials in radiation oncology have been conducted but have not published their results. Thus, 45% of all evidence collected in our field is seemingly lost forever and raises the question about the extent to which the treatments being offered to patients are really evidence based. This problem of representation does not only concern radiation oncology, but it has also been a distinctive issue in medicine. Even if our findings are consistent with previously observed rates of non-publication in other clinical scenarios, our results add to existing work by showing that this representation problem is an essential feature of interventional phase 3 and 4 trials in radiation oncology, since studies assessing non-publication did not analyse interventional radiotherapy trials separately.9–20

It is worth noting that trials funded by NIH and industry showed a higher rate of reporting results in the registry than did other trials, even though nearly 65% of NIH-funded and industry-funded trials did not report anything in ClinicalTrials.gov. In addition, there was no statistically significant difference between trials funded by private companies or by NIH. One way to improve these reporting rates would be to apply economic sanctions against sponsors who do not comply with the regulation (such sanctions already exist in the USA by the Food and Drug Administration, although they have rarely been applied); however, economic sanctions against clinical investigators or companies might prevent them from deciding to begin a new trial if sanctions are a possibility. Having fewer trials could be damaging to the health system as a whole, as well as to future patients. A potential solution would be to institute a system whereby if clinical investigators apply for public funding, they have to disclose results of all previously conducted trials; for privately funded trials, results from all previous studies would have to be made available before the new trial could be registered.

Recently, it has been reported that fewer than half of the trials funded by NIH were published in a PRJ.4 We found a far better publishing rate within the radiation oncology field, since almost 75% of all trials with NIH funding published their results in a PRJ. We found that publication rates for industry-funded trials, on the other hand, were far worse, with 60% of them remaining unpublished. An important consideration is that, leaving aside NIH-funded trials, although this 50% rate of non-publication was higher in industry-funded than in non-industry-funded trials, the differences were not statistically significant. This result is opposite to what has been sometimes reported in the medical literature.21

We would like also to mention that principal investigators from an American Institution were more likely to report results on ClinicalTrials.gov registry and this might be because the law enforcing the registration and reporting of clinical trial results was an American one.

A study design limitation should be considered when interpreting these results. Although we allowed a minimum 24 months for publication in a PRJ, but we did not know if this period was long enough for an assessment of publication. Since all trials analysed in this study should have reported results after a 12-month period, we decided to allow for another 12 months for publishing in a PRJ. Phase 3 and 4 clinical trials provide strong evidence and are more easily accepted for publication in a PRJ. Although a 24-month period might not seem sufficient to our purposes, we have to emphasise that these 24 months was a minimum and most trials analysed in our study were given much more time to publish their results, with a median and mean ‘time to publication’ of 60 months.

It is hard to fathom the reasons underlying this non-publication. One reason might be that we are living in a ‘publish or perish’ era and most clinicians and researchers are willing to participate in a trial without questioning what is really happening with these data globally (there are more ongoing trials than ever before and, as a consequence, it is easy for investigators to participate in multiple trials at the same time; the paradox might rest on the fact that when one of those trials remain unpublished, little attention is paid to it). Another potential reason is publication bias, although it was not possible to assess it in this study. A final possibility is ‘the planning fallacy,22 23: people tend to make terrible predictions of task completion times and what once looked like a feasible trial becomes a longer and much more difficult project to undertake. Given these possibilities, it is important to highlight initiatives such as the 2013 ‘Restoring Invisible and Abandoned Trials’ statement, which was supported by a number of important journals, giving trialists an amnesty of 1 year to publish the results of previously unreported trials.24

As it has been previously stated in the Methods section, we chose ClinicalTrial.gov registry because this registry represented the most comprehensive source for information about ongoing and completed trials within and outside the USA. However, as large and important as this registry is, many trials conducted in radiotherapy have been registered in other registries. Therefore, it should be taken into account that our dataset did not represent the entire population of interventional phase 3 and 4 trials conducted in radiotherapy. On the other hand, we assumed most phase 3 and 4 trials conducted in radiotherapy would be willing to apply their results on the USA soil and therefore have to comply with the FDAAA 801.

There are additional limitations concerning our described search method in ClinicalTrials.gov registry. ClinicalTrials.gov search engine allows the user to focus its search through multiple search fields. Searching for the word ‘Radiotherapy’ did not account for all trials conducted in radiotherapy and produced an enormous amount of false-positive results. To account for all these false-negative and false-positive results, we had to extend our search terms further, including radiotherapy-related terms such as ‘radiation oncology’, ‘radiation therapy’ or ‘IMRT’. This strategy broadened the initial search and lowered considerably false-negative results in our final set, but it is likely that not all phase 3 and 4 interventional clinical trials were capture by our search strategy. On the other hand, false-positive results were easily handled performing a double check on every item at our final set.

In summary, non-publication means poor use of financial resources from funders, host institutions and commissioning bodies. It also means loss of knowledge through hidden data, makes medical practice less evidence based and risks biasing the evidence in important ways. Moreover, it means that a large number of study participants were exposed to the risks of trial participation without the supposed benefits that sharing and publishing of results would offer to future generations of patients. This ethical issue should be at the heart of our current medical practice.