Do urology journals enforce trial registration? A cross-sectional study of published trials.

Objectives (1) To assess endorsement of trial registration in author instructions of urology-related journals and (2) to assess whether randomised controlled trials (RCTs) in the field of urology were effectively registered. Design Cross-sectional study of author instructions and published trials. Setting Journals publishing in the field of urology. Participants First, the authors analysed author instructions of 55 urology-related journals indexed in 'Journal Citation Reports 2009' (12/2010). The authors divided these journals in two groups: those requiring and those not mentioning trial registration as a precondition for publication. Second, the authors chose the five journals with the highest impact factor (IF) from each group. Intervention MEDLINE search to identify RCTs published in these 10 journals in 2009 (01/2011); search of the clinical trials meta-search interface of WHO (International Clinical Trials Registry Platform) for RCTs that lacked information about registration (01-03/2011). Two authors independently assessed the information. Outcome measures Proportion of journals providing advice about trial registration and proportion of trials registered. Results Of 55 journals analysed, 26 (47.3%) provided some editorial advice about trial registration. Journals with higher IFs were more likely to mention trial registration explicitly (p=0.015). Of 106 RCTs published in 2009, 63 were registered (59.4%) with a tendency to an increase after 2005 (83.3%, p=0.035). 71.4% (30/42) of the RCTs that were published in journals mentioning and requiring registration, and 51.6% (33/64) of the RCTs that were published in journals that did not mention trial registration explicitly were registered. This difference was statistically significant (p=0.04). Conclusions The existence of a statement about trial registration in author instructions resulted in a higher proportion of registered RCTs in those journals. Journals with higher IFs were more likely to mention trial registration.

Introduction

Transparency is essential for the efficient transfer of scientific knowledge gained from research into practice. Nevertheless, recent studies have shown that transparency is impaired by non-publication or inappropriate publication of trial results. A major reason is selective publication, either of whole trials or of certain outcomes, which is referred to as publication bias or outcome reporting bias, respectively.1 2 Other reasons that hamper the generation of valid results from trials are limitations in trial methodology and reporting of trial results. A recently published Cochrane Review evaluated studies that compared the content of protocols and register entries with published reports of randomised controlled trials (RCTs). The authors found that it is common practice to add, omit or change outcome parameters in the reporting of study results, which leads to discrepancies between study protocols or register entries and reported results.3 As very recently shown by a retrospective analysis of data entries in http://clinicaltrials.gov on randomised trials on diseases in the digestive system, information available in trial registers was partly inadequate, 30% of trials were registered incorrectly after their completion date and registered information and trial publications showed discrepancies.4

The complete registration of all trials has the potential to increase transparency in scientific research. Members of the International Committee of Medical Journal Editors (ICMJE) promoted trial registration in 2004 by publishing a statement on it.5 The ICMJE defined 1 July 2005 as a key date requiring prospective trial registration (registration before recruitment started) for trials started after this date. For trials that began enrolment prior to this date, the ICMJE member journals allowed for retrospective registration by 13 September 2005.5 WHO supported this joint effort of journals by founding the International Clinical Trials Registry Platform (ICTRP), which brings together data sets provided by different primary registers worldwide. As of 1 July 2007, also the Surgery Journal Editors Group member journals required prospective registration of all clinical trials. From this date on, the Surgery Journal Editors Group member journals planned to consider trials only if they are registered before enrolment of the first study subject.6 In 2007, several editors from major medical journals stated that ‘three years ago, trial registration was the exception and now it is the rule’,7 suggesting that trial registration nowadays is in fact an integral part of clinical research.

It is unclear, however, to what extent trial registration is implemented in specialty fields such as urology. To address this question, we aimed to assess the endorsement of trial registration in journal author instructions in relevant urology journals. Second, we sought to determine whether a statement of trial registration in author instructions resulted in a higher proportion of registered RCTs published in 2009.

Methods

We included in this study all journals that were indexed in the subject categories ‘Urology & Nephrology’ or ‘Andrology’ in the ‘Journal Citation Reports 2009’.8 We examined author instructions available via journal websites (12/2010) and did not impose any language restriction. Two authors (FK and HRG) independently read each document and used relevant key words in electronic full-text searches to extract information about whether the journal (1) publishes original investigations, (2) gives editorial advice about trial registration and (3) endorses the ‘Uniform Requirements for Manuscripts Submitted to Biomedical Journals’ (URM) recommendation of the ICMJE.9 Disagreements were resolved by rechecking the respective websites and, if necessary, by discussion with a third author (JJM).

Journals reporting reviews or commissioned articles only were excluded. We noted if journals recommended or required trial registration as a precondition for publication. In addition, we assessed the context in which the URM appeared in a journal. If the journal guidelines mentioned the URM only in the context of ‘conflicts of interest’ or ‘style of references’, we classified the URM for this journal as ‘not relevant’ for advice about trial registration. If trial registration was either required or recommended or if the guidelines mentioned the URM in the context of trial registration, we classified the journal as ‘direct reference to trial registration’. If the guidelines mentioned the URM in the context of general manuscript preparation without other editorial advice on trial registration, we classified it as ‘indirect reference to trial registration’.

In a second step, we assessed whether a statement of trial registration in author instructions of uro-nephrology journals resulted in a higher proportion of registered RCTs. On the basis of our evaluation, we selected the 10 journals (five journals that mentioned and required trial registration and five journals that did not mention trial registration directly in their author instructions) with the highest impact factor (IF) in their respective group (table 1). We have chosen these journals to obtain a reasonably sized, representative sample of trials published in the major uro-nephrological journals supporting or not supporting prospective trial registration. We then searched Ovid MEDLINE (01/2011, box 1) to identify RCTs that were published in 2009 in these journals using the ‘Cochrane Highly Sensitive Search Strategy’.10

Table 1

Groups of journals giving some or no advice on trial registration in their author instructions

Trial registration mentioned and required	IF	Trial registration not mentioned directly	IF
Journal of the American Society of Nephrology	7.689	Journal of Urology	4.016
European Urology	7.667	American Journal of Nephrology	3.481
Kidney International	6.193	Nephrology Dialysis Transplantation	3.306
American Journal of Kidney Diseases	5.152	Prostate	3.081
Clinical Journal of the American Society of Nephrology	4.844	British Journal of Urology International	2.865

IF, impact factor.

Database: Ovid MEDLINE(R) In-Process & Other Non-Indexed Citations, Ovid MEDLINE(R) Daily and Ovid MEDLINE(R) 1946 to Present.

Searched: 17.01.2011.

Responsible searcher: Kunath

european urology.jn.

kidney international.jn.

american journal of kidney diseases.jn.

clinical journal of the american society of nephrology cjasn.jn.

journal of the american society of nephrology.jn.

bju international.jn.

journal of urology.jn.

american journal of nephrology.jn.

nephrology dialysis transplantation.jn.

prostate.jn.

1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10

randomized controlled trial.pt.

controlled clinical trial.pt.

randomized.ab.

placebo.ab.

clinical trials as topic.sh.

randomly.ab.

Trial.ti.

12 or 13 or 14 or 15 or 16 or 17 or 18

exp animals/ not humans.sh.

19 not 20

11 and 21

limit 22 to yr="2009"

One author (FK) screened the titles and abstracts of all retrieved references and excluded only citations that were clearly irrelevant or duplicates. Two authors (FK and HRG) then independently from each other identified RCTs and excluded non-randomised trials and discussed discrepancies with a third party (JJM). Trials recruiting patients prior to 2004 were excluded because registration was not strongly promoted at that time. The 2004 ICMJE statement promoting trial registration had defined 1 July 2005 as a key date for prospective trial registration (registration before recruitment started). To investigate possible changes in registration practice, we subdivided trials into two groups according to the start of patient recruitment: 2004–2005 and ≥2006. The ICMJE accepted retrospective registration (registration when recruitment was already happening) of trials that began before 1 July 2005.9 For this reason, we did not investigate if trial registration was done pro- or retrospectively.

We searched the ICTRP (01–03/2011) for every report of an RCT without information about trial registration by using relevant key words based on the Patients, Intervention, Control, Outcome, Time scheme.11 The ICTRP is periodically updated and contains data sets of registers from Australia, New Zealand, China, India, Republic of Korea, Germany, Iran, Japan, Africa, Sri Lanka and the Netherlands as well as from ‘ClinicalTrials.gov’ and the ‘International Standard Randomised Controlled Trial Number Register’.

The influence of the IF on trial registration endorsement was evaluated using logistic regression. Statistical tests were two sided, and p<0.05 was considered significant.

Results

We identified 69 journals in the subject categories ‘Urology & Nephrology’ (n=63) and ‘Andrology’ (n=6). Fourteen journals were excluded for various reasons (figure 1); 55 journals were evaluated. The IFs ranged from 0.054 to 7.689.

Figure 1

Selection of relevant journals from the ‘Journal Citation Reports 2009’.

Twenty journals (36.4%) mentioned trial registration as a direct reference in their author instructions, two of these (10%, IF<2.0) recommended trial registration and 18 journals (90%, IF>0.01) required trial registration as a precondition for publication (table 2).

Table 2

Endorsement of trial registration in author instructions

	All journals	IF 0.01–2.0	IF 2.01–4.0	IF>4.01
	n=55 (100%)	n=30 (100%)	n=18 (100%)	n=7 (100%)
Direct reference to trial registration	20 (36.4)	8 (26.7)	7 (38.9)	5 (71.43)
Required	18 (90)	6 (75)	7 (100)	5 (100)
Recommended	2 (10)	2 (25)	–	–
Indirect reference to trial registration	6 (10.9)	3 (10)	2 (11.1)	1 (14.29)
Any advice about trial registration	26 (47.3)	11 (36.7)	9 (50)	6 (85.72)

IF, impact factor.

A total of 32 journals (58.2%) mentioned the URM of the ICMJE. However, 18 journals (32.7%) mentioned the URM only in the context of conflict of interest (n=8) or style of references (n=10). Fourteen journals (25.5%) mentioned the URM in the context of trial registration (n=2) or general manuscript preparation (n=12). All journals that mentioned the URM in the context of trial registration also required or recommended trial registration. This was additionally the case for six of the 12 journals mentioning the URM in the context of general manuscript preparation. Thus, six journals (10.9%) gave an indirect reference to trial registration by mentioning the URM exclusively in the context of general manuscript preparation (table 2).

Twenty-six journals (47.3%) made some kind of reference to trial registration, either by directly or indirectly referring to the URM in their author instructions (table 2). Six of seven journals (85.7%) with IFs above 4.01 included such indirect editorial advice. The likelihood of a direct reference to trial registration was significantly higher with increasing IF (p=0.015).

Our search strategy for the evaluation of the registration practice for RCTs published in 2009 retrieved 429 citations; our evaluation included 106 RCTs (figure 2). Fifty-two RCTs (49%) had started patient recruitment in 2004–2005, 36 (34%) during or after 2006 and in 18 RCTs (17%), the start of patient recruitment was not specified (table 3). These 18 trials were cross-over trials, RCTs with an interventional duration of <12 weeks (n=14, 77.8%) or RCTs with a follow-up of <2 years (n=4, 22.2%).

Figure 2

Selection of randomised controlled trials (RCTs).

Table 3

Registration of RCTs

	Start of patient recruitment
	All	2004–2005	≥2006	Not specified
	n=106 (100%)	n=52 (100%)	n=36 (100%)	n=18 (100%)
Registered RCTs	63 (59.4)	33 (63.5)	30 (83.3)	0
Registration mentioned in article	48 (76.2)	25 (75.8)	23 (76.7)	0
No registration mentioned in article; registered in the ICTRP	15 (23.8)	8 (24.2)	7 (23.3)	0
Not registered in the ICTRP; no registration mentioned in article	43 (40.6)	19 (36.5)	6 (16.7)	18 (100)

ICTRP, International Clinical Trials Registry Platform; RCTs, randomised controlled trials.

In total, 63 of the 106 RCTs (59.4%) were registered, 33 with recruitment from 2004 to 2005 (33/52, 63.5%) and 30 with recruitment in 2006 or later (30/36, 83.3%) (table 3). The difference between these two proportions was statistically significant (RR 0.76, 95% CI 0.59–0.98, p=0.035). Of the 63 registered RCTs, 48 specified the registration number (76.2%). Following the search of the ICTRP, 15 additional trials (23.8%) were classified as ‘registered’, although no reference to their registration was given in their publication. We found no differences in reporting of registration between RCTs with patient recruitment in 2004–2005 and during or after 2006 (table 3, registration mentioned in article: 75.8% vs 76.7%; no registration mentioned in article but registered in the ICTRP: 24.2% vs 23.3%, respectively). Forty-three RCTs (40.6%) did not report registration and could not be identified in the ICTRP (table 3).

Of the 42 RCTs that were published in journals that mentioned and required trial registration as a precondition for publication, 71.4% (30/42) were registered and 28.6% (12/42) were not registered (table 4). In contrast to this, of the 64 RCTs that were published in journals that did not mention trial registration directly in their author instructions, only 51.6% (33/64) were registered and 48.4% (31/64) were not registered (table 4). The proportion of reports of registered trials was significantly higher in journals that mentioned and required trial registration as a precondition for publication than those that did not mention trial registration directly in their author instructions (p=0.04).

Table 4

Registration of RCTs published in journals mentioning and requiring or not mentioning trial registration in their author instructions

	RCTs (n=106)
	Registered RCTs (n=63)	Non-registered RCTs (n=43)
Trial registration mentioned and required
All (n=42, 100%)	30 (71.4)	12 (28.6)
2004–2005	18 (60)	2 (16.7)
≥2006	12 (40)	2 (16.7)
Not specified	0	8 (66.6)
Trial registration not mentioned directly
All (n=64, 100%)	33 (51.6)	31 (48.4)
2004–2005	15 (45.5)	17 (54.8)
≥2006	18 (54.5)	4 (12.9)
Not specified	0	10 (32.2)

RCTs, randomised controlled trials.

Discussion

We assessed the endorsement of trial registration in author instructions of urology-related journals. Only 36.4% of uro-nephrology journals gave direct editorial advice about trial registration. Of interest, a higher IF was significantly associated with the likelihood of giving direct advice about trial registration. More than half of all RCTs published in 2009 and the vast majority (83%) of trials with patient recruitment during or after 2006 were registered (table 3). Three quarters of the RCTs that were published in journals mentioning and requiring trial registration as a precondition for publication were registered (table 4).

Although the ICMJE, WHO and the World Medical Association support trial registration,9 12 13 the advice given in the author instructions of uro-nephrology journals does not sufficiently reflect this recommendation. This situation, however, is not as comparatively negative as it might seem at first. Meerpohl et al14 evaluated the editorial policies of paediatric journals and demonstrated that only one-quarter of all selected journals required or recommended trial registration in their author instructions. Another study compared Italian journals (n=76) indexed in Medline and randomly chosen journals from the UK (n=76) with regard to editorial advice on trial registration. Surprisingly, none of the Italian journals required trial registration, while 28% of the UK journals mentioned registration.15

The data presented here show that the reporting of registration can still be improved, and we suggest that this problem is general in the scientific literature. Reveiz et al evaluated the key methodological items of RCTs published in 55 of the highest-ranking journals. Only 36% of 148 included RCTs reported that the study was registered in any trial register.16 Awareness of trial registration was raised in the scientific community in 2005. This awareness translated into a statistically significant increase in trial registrations for RCTs with patient recruitment after 2005 in urology and better registration of trials in other medical specialties. Hamm et al17 found that only one-quarter of all paediatric RCTs published in 2007 were in fact registered. Similarly, a study of 323 RCTs from medical departments published in journals with high IFs identified only 46% of those as adequately registered trials.18 However, these studies did not differentiate between trial registration for RCTs with patient recruitment before, during or after 2005.

Potential bias in our data may have emerged when registered RCTs could not be identified in the ICTRP, and the start of patient recruitment was not mentioned in the manuscript. Because this potential limitation hinders the detection of studies in registries, it is also a handicap for transparency in research. If we assumed that patient recruitment in trials not reporting a registration number or not specifying their conduct happened in or after 2006, the proportion of registered trials would decrease. In addition, the RCTs analysed were taken from the five uro-nephrological journals with the highest IF in their respective categories. It may well be that RCTs published in these journals with higher IFs are more likely to be registered. Our results might, therefore, overestimate compliance with trial registration and be, accordingly, not representative of the uro-nephrological field as a whole.

In an ideal world, knowledge translation would be supported by the prospective registration of trials as well as the comprehensive and transparent publication of all trial results in either a register or a journal. This transparency would guarantee public access to research results. Our study points to some limitations on the road to this ideal setup. On the one hand, not all trials are registered yet. Furthermore, some registered trials are not identified as such in their publication, so to obtain access to the additional information provided by the entry in the register, a separate search is required. Consequently, a reference to the registration number of trials should be mandatory in publications. One consideration, though, is that if journal editors rigorously require trial registration prior to the start of patient recruitment as a precondition for publication, only these prospectively registered trials could be published in the future. Although such a practice would be consistent with the ICMJE recommendations, it could lead to the non-publication of non-registered trials, increasing the proportion of non-published studies.

Conclusions

A statement of registration in author instructions resulted in a higher proportion of registered RCTs. However, journal editors could still do more by giving advice about trial registration in their author instructions and by ensuring that manuscripts include registration details of trials published in their journal. It is not, however, only the editors' responsibility to ensure trial registration. In fact, the scientists and authors need to act to ensure trial registration. Researchers and sponsors should consider trial registration as an ethical imperative to enhance transparency in research and not as an obstacle to research.