Characteristics and publication fate of unregistered and retrospectively registered clinical trials submitted to The BMJ over 4 years.

OBJECTIVES: We sought to evaluate the characteristics and publication fate of improperly registered clinical trials submitted to a medical journal (The BMJ) over a 4-year period to identify common types of registration issues and their relation to publication outcomes.
DESIGN: Research articles submitted to The BMJ and identified as unregistered or retrospectively registered by editors were included if they reported outcomes of a clinical trial. Relevant data regarding the trials were then extracted from each paper. Trials were categorised as prospectively registered, registered in an unapproved registry, unregistered or other, and explanations for registration deficiencies were grouped into six categories. We searched PubMed and Google to determine whether, where and when improperly registered studies were subsequently published and whether registration issues were disclosed.
RESULTS: 123 research papers reporting apparently unregistered or retrospectively registered clinical trials were identified. 110 studies (89.4%) were retrospectively registered, nine (7.3%) were unregistered, three (2.4%) had been registered in an unapproved registry and one study originally lacking registration details was later discovered to have been prospectively registered. 82 studies (66.6%) were funded entirely or in part by government sources, and only seven studies (5.7%) received funding from industry. Of those papers submitted to The BMJ through the end of 2015, 67 of the 70 papers rejected for registration problems (95.7%) were subsequently published in another journal. The registration problem was disclosed in only 2 (2.9%).
CONCLUSIONS: Improper registration remains a problem, particularly for clinical trials that are government or foundation-funded. Nonetheless, improperly registered trials are almost always published, suggesting that medical journal editors may not actively enforce registration requirements. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

This study provides the first comprehensive look at registration issues and publication outcomes among research papers submitted to a high-impact medical journal.

This represents the experience of a single high-impact journal and may not be representative of the situation in other, particularly smaller, journals.

We relied on journal editors to report studies with registration problems; some such trials may have been missed and therefore not included in this audit.

Information about the reason for late registration or lack of registration is incomplete because not all authors provided explanations.

Background

Prospective registration of clinical trials is an important safeguard against selective reporting and non-publication of research. Since 2005, the International Committee of Medical Journal Editors (ICMJE) has insisted on prospective trial registration as a condition of publication.1 Registration must occur before enrolment of the first study participant in a trial registry that meets quality criteria developed by WHO. Hundreds of biomedical journals subsequently endorsed this policy.2 Adherence to these requirements is imperfect, however, both on the part of researchers and journal editors. A recent study found that about one-third of contemporary trials in the ClinicalTrials.gov database had been registered more than 3 months after the study began.3 In one survey, 67% of editors from journals that endorse the ICMJE registration policy nonetheless indicated that they would consider retrospectively registered studies for publication.4 Many published trials have registration deficiencies or do not report outcomes that are consistent with registry entries. This problem exists in the subspecialty literature as well as at major journals.5 6

There are many possible reasons for the slow uptake of trial registration. Among them are that in most cases there are no strong penalties for failure to properly register a trial other than the threat of non-publication in an ICMJE-adherent journal. Not all journals are equally committed to enforcing registration requirements and many editors are not persuaded of the need for registration.7 8 The influential Committee on Publication Ethics has advocated a soft line on registration, suggesting that ‘whether or not a trial is registered has little bearing on the quality or ethics of the study, and so it is up to the editor to decide whether or not a study should be published.’9 For the small proportion of trials where financial or other penalties theoretically are in place, enforcement is lax.10

Although the persistence of trial registration deficiencies has been well documented, only one previous study has described the magnitude of the problem in a family of journals.11 Of the 108 clinical trials submitted to BioMed Central journals in 2013, just 33 (31%) had been registered prospectively. The median time between enrolment of the first trial participant and registration was 356 days, with a range of 5–1677 days. This study did not describe the detailed characteristics of the papers with registration problems, or provide information about the publication fate of papers that are rejected from a journal because of such problems. This information is needed to better understand the problem of deficient trial registration and develop strategies to encourage timely registration.

Methods

As part of an internal BMJ audit of adherence to trial registration requirements, we aimed to identify all trials submitted from June 2013 to June 2017 that did not comply with ICMJE trial registration requirements. The BMJ employs four full-time staff and five part-time freelance research editors who appraise and handle submitted articles. In June 2013, EL began keeping a list of all clinical trials submitted to The BMJ that appeared to be unregistered or retrospectively registered. As one of the founding members of the ICMJE, The BMJ since 2005 has required appropriate registration of clinical trials as a condition of publication. All BMJ research editors were asked to notify EL whenever they determined that a submitted trial was unregistered, retrospectively registered (ie, registered after the study began) or registered in an unapproved registry.

All papers on this list were screened to identify those that met the following inclusion criteria: submitted to The BMJ as a research article between 1 June 2013 and 30 June 2017; reported outcomes of a clinical trial according to the ICMJE definition of a trial.12 Papers reporting on observational analyses of clinical trials were excluded. All articles meeting these criteria were entered into an Excel spreadsheet. EL searched the ScholarOne database and extracted the following information for each included paper: full title; unique BMJ paper number; year of submission; country and region of the corresponding author (Europe, North America, Africa/Middle East, Asia, Central and South America—including Mexico or Australia/New Zealand); the trial registry and number, if present; funding sources (categorised as no funding or none when authors specifically declared they had received no funding for the work, unknown funding when authors did not provide funding information, government, academic, industry, foundation or combinations) and any explanation provided by the authors for the registration problem. A second researcher, SC, verified data abstraction for a random sample of 10% of all trial entries. Disagreements were resolved by consensus.

Where a trial registration number could be located in the submitted paper, the corresponding registry entry was searched to verify the date the trial was registered, and if present, the registry-reported date of enrolment of the first participant, date of submission, and anticipated or actual start dates, if present. Additional information about these dates was also sought in the text of the article itself and this information was recorded in the Excel spreadsheet. We then categorised articles as follows:

Prospectively registered, that is, trial registration had occurred in an approved trial registry before the date of first participant enrolment as recorded in the trial registry.

Prospectively or retrospectively registered in a registry that did not meet ICMJE requirements at the time of registration (eg, European  Clinical Trials Database is a WHO-approved registry only for trials registered after 20 June 2011).1

Unregistered.

Other (eg, identified by editor as improperly registered but later found to be properly registered).

For registries such as the International Standardised Randomised Controlled Trials Number (ISRCTN) registry, where entries report both the date researchers submitted their request for registration and the date that trial registration was completed, we considered the date registration actually was assigned (not when the application was first submitted) to be the date of registration. For registries that reported only the date of registration submission, we considered that to be the date of registration. To determine the date when a study began, or when participants were enrolled, we used ‘anticipated start date’ for ISRCTN, ‘study start date’ for ClinicalTrials.gov. For the Australian New Zealand Clinical Trials Registry, which reports both an anticipated and actual date of first enrolment, we considered the actual date to be the start date. In cases where the starting date of a trial was reported differently on a trial registry than in the paper, we considered the date reported in the paper itself to be the correct date. In cases where only the month and not the date of registration or first enrolment were reported, we assumed the trial had been registered prospectively.

EL and SL searched PubMed and Google to determine whether, where and when improperly registered studies were subsequently published and, if published, whether the registration problem was disclosed. We first searched Google using the title of the rejected paper; if unsuccessful in locating a publication, we then searched Google using keywords and author names. If unsuccessful, we performed the same searches in PubMed. We searched the full-text versions of subsequently published papers for information about trial registration by first visually scanning the abstract and full text and, if unsuccessful, using the ‘find’ function in Adobe Acrobat Reader or the Google Chrome browser and the search term reg*. We recorded the impact factor of the publishing journal for the year when the study was published, using the Thomson Reuters Journal Citation Reports database.13

We determined the number and proportion of included trials that were unregistered, retrospectively registered or registered in an unapproved registry. We classified the interventions studied in each trial as regulated (eg, drug, biological, medical device or other intervention regulated by the US Food and Drug Administration—FDA) or unregulated (eg, procedures, behavioural interventions, dietary supplements or other interventions that are not subject to FDA regulation). We then evaluated the proportion of papers from different regions of the world (Europe, North America, Africa/Middle East, Asia, Central and South America—including Mexico or Australia/New Zealand). Results are presented as absolute numbers with percentages. If authors had provided explanations for the registration problem, EL and SL classified these explanations into one of six categories (ambiguity in the definition of a clinical trial; error attributed to another team member; author belief that the trial was not subject to registration requirements; registry error; unaware of or misunderstood registration requirements; technical difficulties) and selected representative quotations to illustrate each category. We removed potentially identifying information from quotations, either summarising it in a non-identifiable way in square brackets or omitting words, indicated by an ellipsis.

Results

BMJ editors identified 123 unregistered or retrospectively registered clinical trials over the 49-month period of the study. Table 1 shows the characteristics of these studies. One hundred and ten (89.4%) were retrospectively registered and nine (7.3%) were unregistered. Three studies had been registered in an unapproved registry. One study (classified as ‘Other’) was originally assumed to be unregistered because the authors did not supply registration information. However, we discovered that it had subsequently been published elsewhere and that publication included registration information showing the trial had been prospectively registered.

Table 1

Characteristics of 123 trials rejected for registration deficiencies

	#	%
Registration deficiency
Retrospective	110	89.4
Unregistered	9	7.3
Registered in unapproved registry	3	2.4
Other (mistaken as unregistered)	1	<1
Funding source
Single funder	95	77.2
Combination	28	22.8
Funding type*
Government	82	66.6
Foundation	33	26.8
Academic	22	17.9
Industry	7	5.7
No funding	5	4.1
Unknown	6	4.9
Year submitted
2013	7	5.7
2014	27	22.0
2015	36	29.3
2016	26	21.1
2017	27	22.0
Registry
CT.gov	55	48.2
ISRCTN	30	26.3
NTR	7	6.1
ANZCTR	7	6.1
EudraCT	3	2.6
ChiCTR	3	2.6
PACTR	2	1.8
German CTR	2	1.8
UMIN	1	0.8
AEARCTR	1	0.8
CTRI	1	0.8

*Projects could have more than one source of funding, so numbers do not add to 123 and percentages do not sum to 100%.

AEARCTR, American Economic Association RCT Registry; ANZCTR, Australian New Zealand Clinical Trials Registry; ChiCTR, Chinese Clinical Trial Registry; CT.gov, ClinicalTrials.gov; CTRI, Clinical Trials Registry—India; EudraCT, European Clinical Trials Database, approved registry only for trials registered after 20 June 2011; German CTR, German Clinical Trials Registry; ISRCTN, International Standard Randomised Controlled Trial Number Registry; NTR, Netherlands Trial Registry; PACTR, Pan African Clinical Trials Registry; UMIN, University Hospital Medical Information Network Clinical Trials Registry (Japan).

Over three-quarters of studies had only a single funder. Almost two-thirds were funded entirely or in part by government sources, while about a quarter received some or all of their funding from philanthropic foundations. Only seven studies (5.7%) received funding from industry, and in all but one case other funding sources were also listed; thus, only one trial in our sample was solely industry-funded. The number of improperly registered trials varied over the years of the study, but averaged 2.1 a month. From 2014 to 2016, the 3 years for which data are complete, the number ranged from 26 in 2016 to 36 in 2015. Almost three-quarters of retrospectively registered studies were registered in ClinicalTrials.gov or ISRCTN. Of the 123 trials, 19 (5.5%) in our series evaluated FDA-regulated interventions. Three (15.8%) of these trials were unregistered and 16 were retrospectively registered. Among the 104 trials in our series that reported on unregulated interventions, 6 (5.8%) were unregistered.

Table 2 shows the subsequent publication fate of papers with registration problems that were rejected by The BMJ through the end of 2015. Of the 70, 67 papers rejected for registration problems (97%) were subsequently published in another journal. We could not locate a subsequent publication for three papers. One paper had been published in another journal but then withdrawn. No reason was given for the withdrawal. Only two papers were published in a journal without an impact factor. One paper was eventually published in The BMJ, after the authors appealed and provided an explanation that was deemed to justify retrospective registration. The registration problem was explained in the paper. No paper was published in a journal with a higher-impact factor than The BMJ. The median impact factor of the journals where papers were subsequently published was 4.972 (IQR: 3.586). Only two of the subsequently published papers provided a description of the registration problem and explained why the journal chose to publish it despite the problem.

Table 2

Publication fate of the 70 papers submitted through 2015*

Publication fate	n (%)
Not published	3 (4.3%)
Published	67 (95.7%)
Published and then withdrawn	1
Published in a journal without an impact factor	2
Published in a journal with an impact factor	64
Median impact factor	4.972
First quartile of impact factors	3.057
Third quartile of impact factors	6.643
IQR of impact factors	3.586
Range of impact factors	1.556–19.697
Registration problem disclosed in subsequent publication	2(2.9%)

*Publication status determined September 2017.

We noted several inconsistencies and alterations in some of the subsequently published papers that we located (online supplementary appendix). For example, one study no longer reported a trial registration number and the research was described as an ‘implementation-effectiveness study’ rather than a clinical trial. Another paper was listed on the researcher’s website as having been published in The BMJ even though it had been rejected. In several papers, the date participants were enrolled in the study had been changed from that reported in the version of the paper submitted to The BMJ.

The majority of unregistered trials were from Europe or Asia (table 3). One-third of unregistered studies were submitted in 2017. The most common reason given for lack of registration was that the researchers did not believe their study met the ICMJE definition of a clinical trial. In all of these cases, researchers had randomised healthcare providers or groups of participants and examined outcomes that they claimed were not medical, although BMJ editors disagreed with this assessment. In several cases, authors were unaware of requirements for trial registration, or assumed that ethical approval was the same thing as registration.

Table 3

Characteristics and publication fate of the nine unregistered studies

Year	Region	Explanation	Publication fate	Funding
2013	Central and South America	Appeared unaware of requirements for trial registration. Confused ethical approval with registration.	Published but withdrawn shortly thereafter; no reason given.	Government
2015	Europe	Began before registration required but not subsequently registered prior to submission.	Published. No explanation of registration problem in the paper.	Government
2015	Europe	Confused ethical approval with registration. Did not accept that study needed to be registered as a trial.	Published. Explanation about registration is provided in the paper.	Government
2015	Europe	None	Could not locate a publication.	None
2016	Europe	Did not accept that study was a clinical trial.	Published. No explanation of registration problem in the paper.	Government
2016	Asia	None	Could not locate a publication, but it may be too soon to expect one.	Unknown (not reported in paper)
2017	Asia	Did not accept that study was a clinical trial.	Too soon to expect publication elsewhere.	Government, foundation, industry (not pharmaceutical/device).
2017	Asia	No explanation	Too soon to expect publication elsewhere.	Academic
2017	Africa/Middle East	Appeared unaware of requirements for trial registration; confused ethical approval with registration.	Too soon to expect publication elsewhere.	None

Table 4 lists common reasons for late or non-registration of trials, grouped by category and illustrated with quotations. Authors of 40/123 (32.5%) of the papers provided explanations for registration problems. Overall, the most common explanation authors gave for registration deficiencies was the belief that trials randomising health professionals did not need to be registered.

Table 4

Selected explanations for registration deficiencies

Explanation category	Sample quotes
Ambiguity of ICMJE definition of clinical trial	"We didn’t include patients in our trial, and didn’t analyze patients’ health outcomes. "In [our country] studies using prescription databases do not require ethical approval… The intervention on GPs was performed within the continuing medical education (CME) programme, according to the [country] health authority policy." "As this was not a clinical trial, no registration was obtained." "There was debate amongst the clinicians and academic staff involved in the project at the very start whether this represented research or a service evaluation, and it was decided to treat it as original research (and seek ethical approval) for completeness." Comment: In each of these cases, the authors described the study as a trial. In two cases, physicians had been randomised to an educational intervention and outcomes of prescriptions were evaluated. The third study was a cluster randomised trial of an educational intervention for physician groups and evaluated the outcome of medical errors.
Error attributed to another team member or team processes	"This resulted from a genuine oversight—simply a mix up between two people who each thought the other had registered the trial." "Due to the lack of expertise of our sponsor… at that time, ClinicalTrial registration has been initiated but not validated in time. Neither me nor my colleagues were aware and informed of the fact that this process was not well done and all the investigators were in good faith that everything was going well. You are right, the registration at ISRCTN is effective since [date]. The payment to ISRCTN got delayed at the time, which delayed the effective registration. As part of the preparation we had the intention to register it on trials.gov, but something must have gone wrong." "[Our Project Coordinator] was responsible for all registration-related tasks. Unbeknownst to the rest of the research team, [this person] was dealing with a very serious and highly personal issue…This was complicated further by the fact that [this person] was on [medical] leave… The rest of the research team were unaware at the time that anything was amiss, and we proceeded with the research on the assumption that registration was complete." "The delay of the registration was because our [partners] took more time than expected due to their bureaucratic procedures and delays for translations." Comment: ICMJE makes it clear that responsibility for compliance with registration requirements lies with the principal investigator.
Requirements should not apply	"You are hampering the possible publication of valuable clinical data long awaited by the research community because the trial protocol was not included in a register whose main aim should be to prevent underreporting." "[The junior investigator who failed to register the study] is a developing country scientist doing this important study alongside a very busy job. Drug companies have whole departments devoted to compliance with regulations and processes like this."
Registry error	"Unfortunately, our trial appeared as ’retrospectively registered' due to a database error made by those curating the ISRCTN registry website during a recent upgrade of the system. We have made the ISRCTN team aware of this and details of the trial have now been corrected." Comment: In three cases ISRCTN changed categorization of a trial from retrospective to prospective when authors supplied information showing the dates they originally provided had been incorrect or ISRCTN had made an error. The reasons for the change were clearly described in the registry entry.
Technical difficulties or misunderstanding of registry instructions	"We are somewhat unsure why the submission date say aprill [sic] 2011 on the website, given we started the application over a year before that. We did submit the information before we started enrollment. The only thing we can think [sic] of is that either we did not hit the submit button or NIH took some time to approve it." "The only thing that was not done was pressing a ’submit' button which was because of a logical (but as it happened incorrect) assumption by [a junior researcher] that the wording on the website meant it should happen when the trial had completed."
Unaware of requirements	"We humbly confess the delay in registering the clinical trial that happened partly due to the lack of knowledge about trial registration…" "About the registration of our trial, the official rule including the official registration in the faculty of Medicine before it should be reviewed by ethical committee. Today I send you the official certificate of ethical committee. Tomorrow, I will try to send you the official number of registration of this trial in our faculty." "This was a prospective randomised controlled trial… The registration of the trial being delayed does not dispute it being a prospective trial but just that there was a delay in registering the study on the clinicaltrials.gov website." "We do not know what retrospective is, but it surely does not apply to us." "However, it was registered later on clinicaltrials.gov mainly because when we considered ethic and legal issues, we came to the conclusion that this registration was not mandatory."

GP, General Practitioner; ICMJE, International Committee of Medical Journal Editors; ISRCTN, International Standardised Randomised Controlled Trials Number.

Discussion

Twelve years after trial registration requirements were imposed by ICMJE journals, compliance is imperfect. In this study of trials submitted to a single high-impact journal, it was rare that trials were unregistered but retrospective registration remains a problem. This is a concern because retrospective registration may abet selective reporting of outcomes.14 The number of trials with registration problems is small in comparison to the number of research papers received by The BMJ but ‘large in relation to zero, which should be the goal.’15

Unregistered and retrospectively registered trials come from all parts of the world, not just those with less training and tradition in research. Many such trials originated from well-established research groups in the USA, England, Canada and other countries with a stronger tradition and more training in clinical research. It is possible that this situation may improve, at least in the USA where the US FDA Amendments Act of 2007 imposed a legal mandate to register clinical trials of many FDA-regulated products. Research suggests this has increased trial registration, publication and outcome reporting fidelity, but clearly more progress is needed.16

It is notable that there are so few industry-funded trials on this list of 123 late or unregistered studies. The problem of deficient trial registration is almost entirely confined to government, foundation and academically funded studies. This is consistent with other research and with the espoused commitment of most pharmaceutical companies to trial registration.17 18 Most clinical trials submitted to The BMJ-tested interventions that are not regulated by The FDA. Previous work has shown that trials of unregulated interventions are more likely to have registration deficiencies, compared with trials that evaluate FDA-regulated interventions.19

Contrary to the worries of some authors and others, trials that have been retrospectively registered seem to be easily published elsewhere, and are not exclusively relegated to obscure, low-impact journals. This is heartening for authors of studies lacking timely registration. It suggests, however, that while many journals say they require prospective registration, they do not mean it. They are prepared to overlook registration problems, especially in borderline situations, and are encouraged to do so by authorities.4 9 Still other journals have deliberately moved away from requirements for prospective registration requirements, citing ‘the need to uncover all trials and their results.’20 These moves may put journals that do enforce registration requirements at a disadvantage in the competition to publish clinical trials and maintain the goodwill of trialists.

Authors offer a variety of reasons for failure to register trials on a timely basis, including lack of awareness of requirements, failure of a junior team member or study coordinator to register the study or problems with the registration process. In checking on the publication fate of rejected papers, we identified several instances in which authors appeared to have changed dates in trial registries or manuscripts before submission to the publishing journal. We also noted other instances of possible misbehaviour. These things suggest that registration deficiencies may be markers of poor organisation and supervision of a research project, or a disregard for accuracy and rules. This is consistent with previous research showing that small errors and discrepancies in published papers reflected more serious problems with the quality of the research.21 Also of note, in several cases authors were successful in persuading a trial registry (ISRCTN) to amend dates and change the categorisation of their study.

Our study has a number of limitations. We relied on BMJ research editors to report studies with registration problems; some such trials may have been missed and therefore not included in this audit. Information about the reason for late registration or lack of registration is incomplete because not all authors provided explanations. This represents the experience of a single journal and may not be representative of the situation in other journals. Authors submitting to higher-impact journals such as The BMJ are a self-selected group that might be more aware and observant of registration requirements. The problem might be worse at low-impact journals. High-impact journals (those with an impact factor of 10 or greater) are more likely than low-impact journals to endorse ICMJE trial registration requirements and they publish a lower proportion of unregistered trials.22

Conclusions

This paper makes clear that one notable area of success for trial registration has been in industry-funded research, where pharmaceutical companies and device manufacturers often have both a legal obligation to register and financial and reputational interests at stake when planning and publishing trials. In our sample, the majority of studies with registration problems were funded by government or private grants. Such funders appear poised to address the problem of improperly registered trials. In January 2017, the US National Institutes of Health along with 20 other funders committed themselves to providing funds only for prospectively registered trials, regardless of the intervention that is assessed.23

In their explanations for trial registration problems, many authors cited a lack of familiarity with the registration process, misunderstanding about what constitutes a clinical trial, or confusion about which member of the study team was responsible for registering the trial. Our findings suggest that the ICMJE should clarify its definition of a clinical trial, commenting specifically on two matters: (1) whether quality improvement or implementation trials are exempt from registration requirements and (2) whether changes in health provider behaviour (eg, medical errors, prescription of specific treatments) constitute medical outcomes.

Our data show that even 12 years after many medical journals have endorsed ICMJE registration requirements and indicated that they require prospective registration, adherence to this standard is not universal. The fact that trials with registration issues were almost always published in another journal, and that these journals did not disclose registration problems, indicates that journal editors, as well as researchers and funders, can play an important role in improving the registration system. By refusing to publish such trials or at least by prominently disclosing registration issues, medical journals could provide a strong motivation for researchers to register their trials going forward.

Future research

Future research should evaluate the link between registration deficiencies and the underlying quality of the scientific data included in these trials. We noted several instances of possible methodological problems or questionable researcher conduct for some of the trials we examined. Furthermore, research is needed to understand the reasons for lax enforcement of trial registration requirements by editors.