Registration and Outcome-Reporting Bias in Randomized Controlled Trials of Distal Radial Fracture Treatment.

BACKGROUND: The purpose of the present study was to systematically evaluate the completeness of trial registration and the extent of outcome-reporting bias in modern randomized controlled trials (RCTs) relating to the treatment of distal radial fracture.
METHODS: With use of 4 databases (PubMed, Cochrane CENTRAL, Embase, and PEDro), this systematic review identified all RCTs of distal radial fracture treatment published from January 1, 2010, to December 31, 2015. We independently determined the registration status of these trials in a public trial registry and compared the characteristics of registered and non-registered trials. We assessed the quality and consistency of primary outcome measure (POM) reporting between the registration data and the final published studies.
RESULTS: Ninety studies met the inclusion criteria. Of those, only 28 (31%) were registered, and only 3 (3%) were "appropriately registered" (i.e., prospectively registered and identifying and fully describing the POM). Registered trials had larger sample sizes and were more likely to be multicenter, to report funding sources, and to be published in higher-impact-factor journals. Sixteen (18%) of the 90 registered RCTs named a POM in the registry; 7 (44%) of those 16 registered RCTs stated a different POM, an additional POM, or no POM at all in the final publication than was stated in the registry data. Additionally, 13 (81%) of those 16 registered RCTs had discrepancies in the time point reported for the POM.
CONCLUSIONS: In an attempt to address publication and outcome-reporting bias, prospective trial registration in a public registry has been deemed a condition for publication by the International Committee of Medical Journal Editors (ICMJE) since 2005. This study shows poor registration rates as well as inconsistencies in the reporting of POMs of recent trials relating to the treatment of distal radial fracture, one of the most common and most investigated injuries in orthopaedic practice. CLINICAL RELEVANCE: The problems of registration and outcome-reporting bias in RCTs are important to highlight and address, and to find a solution will require the cooperation of researchers, reviewers, and journal editors. Increasing the transparency and consistency of reporting will help to increase the quality of research, which can impact patient care through evidence-based guidelines.

Distal radial fractures are the most common fracture of the upper limb[1] and the most common fracture sustained overall until the age of 75 years, at which point they are surpassed by hip fractures[2]. This injury has been recognized as a priority for clinical research, becoming the focus of a plethora of randomized controlled trials (RCTs) in the past decade. Nevertheless, there is little agreement on optimal treatment[3,4]. Interpretation of the evidence is challenging, and the lack of transparency and consistency in outcome reporting is a major limitation to the research[3,5-10].

In 2004, the International Committee of Medical Journal Editors (ICMJE) introduced a policy that required trials to be registered in a public trial registry prior to the enrollment of the first participant in order to be considered for publication[11]. Despite this effort to minimize publication bias and outcome-reporting bias, poor compliance persists. Outcome-reporting bias can increase the prevalence of false-positive findings and subsequently affect the results of systematic reviews and meta-analyses, which could overestimate the effects of treatment[12].

The present study had the following objectives involving RCTs investigating the treatment of distal radial fracture that were published from January 1, 2010, to December 31, 2015: (1) to determine the proportion of trials that were registered in a trial database; (2) to determine the proportion of trials that were “appropriately registered;” (3) to determine if there were differences in the primary outcome measure (POM) reported between the registration record and the published reports; and (4) to compare trials with and without registration for differences in journal impact factor, sample size, funding source, study design, length of follow-up, and country of origin.

To our knowledge, this is the first study to investigate these factors in the setting of a single, common, well-researched orthopaedic injury and across all journal publications.

Materials and Methods

Four electronic databases (PubMed, Cochrane CENTRAL [Wiley], Embase [OVID], and PEDro) were searched, and the inclusion and exclusion criteria were applied to the results (Tables I and II). Search terms were tailored to meet the main inclusion criteria concepts, “distal radial fracture” and “randomized controlled trial.” This study used the definition of an RCT from The Cochrane Handbook for Systematic Reviews of Interventions[13]. The search strategies were compiled with the expertise of an information specialist (D.G.) (see ). The search timeframe was from January 1, 2010, to December 31, 2015.

Two authors (S.L. and T.K.) independently screened records for eligibility. Disagreements were reviewed by a third author (A.K.) and settled by discussion. Study selection is reported in a PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)[14] flow diagram (Fig. 1).

Fig. 1

Flowchart showing the inclusion and exclusion of the study records.

When the trial registration number was included in the published report, it was used to locate the record. If the publication did not include a registration number, 2 authors (S.L. and T.K.) manually searched the World Health Organization (WHO) International Clinical Trials Registry Platform with the following criteria in 3 separate searches[15]: keywords “radius,” “radial,” and “wrist fracture” with no time limits. Duplicate records and those not pertaining to distal radial fractures were removed, and all remaining registration records were reviewed with use of key information, including the interventions, name of the principal investigator, institution, study design, and country. If a trial was not present in the WHO registry, it was considered “not registered.” The WHO registry is comprehensive and includes all major trial registries throughout the world that are endorsed by the ICMJE[16] (Table III).

Presence in the WHO registry alone, however, did not confirm that the study fully and correctly fulfilled the criteria for trial registration. As in the study by Nankervis et al., and in accordance with ICMJE recommendations, the present study defined “appropriately registered trials” as those in which (1) the trial was registered no later than the date of first participant enrollment, (2) the registration record specifically identified the POM, and (3) the registration record specifically stated the time point of primary interest for the POM[16].

The POM was defined as a prespecified outcome considered to be of greatest importance to the relevant stakeholders, compared between 2 randomized groups (control and intervention) in order to evaluate the effect of an intervention at a prespecified time point[17]. The POM had to be specifically identified as such in a published article; otherwise, the respective study was considered to not have specified a POM—for example, studies in which there were no details as to how the POM was measured (e.g., “wrist function” or “fracture healing”) and when multiple POMs were listed despite the trial not being powered for such analysis.

Continuous variables were compared with use of the t test for parametric data. Chi-square and Fisher exact tests were used for categorical variables. Significance was set at p < 0.05. The relationship of time and registration status was tested with use of logistic regression. Analysis was performed with use of Prism (version 7; GraphPad).

The PRISMA statement criteria were followed in so far as the items were applicable to the design of the study[14].

Results

For publication dates between January 1, 2010, and December 31, 2015, 1,300 studies were identified by searching 4 databases. After excluding duplicate listings and independently screening abstract and full text records, 90 RCTs were identified as fulfilling the study inclusion criteria (Fig. 1, see Appendix).

Only 28 (31%) of the 90 distal radial fracture RCTs appeared in any form in a trial registry and were considered registered, and only 3 (3%) were “appropriately registered,” meaning that they were registered prospectively and had fully specified the POM, including specifying the time point of interest.

The differing characteristics of registered and non-registered trials are shown in Table IV, and the number of registered trials per year is detailed in Table V. There was no relationship identified between registration status and year of registration (regression coefficient = 0.894; 95% confidence interval [CI], −0.885 to 2.673; p = 0.324), nor between registration status and year of study publication (regression coefficient = −1.012; 95% CI, −2.156 to 0.133; p = 0.083).

Seventeen (61%) of the 28 registered studies included the trial registration number in their final published articles, allowing a direct link from the study to the registration record, as recommended by the ICMJE[16]. Only 11 trials (39%) were prospectively registered (i.e., registered before enrollment of the first participant), therefore fulfilling ICMJE criteria. Seven trials (25%) were registered before the study end, 3 (11%) were registered after the end of the study, and 7 (25%) did not identify when the trial was registered relative to the timeframe of the study.

Sixteen (57%) of the 28 registered studies named the POM in the WHO record, with or without a time point of interest. Seven (44%) of these 16 studies had discrepancies in POM reporting, with 3 studies reporting an entirely different POM in the published study, 2 reporting additional POMs, and 2 failing to identify the POM altogether. Thirteen (81%) of the 16 registered RCTs had discrepancies in the time point of primary interest between that in the registry and that in the final report. Overall, only 3 (11%) of the 28 registered trials stated both the POM and the time point of interest, which constitutes full reporting of the POM.

Discussion

To our knowledge, this is the first study to systematically investigate the completeness of trial registration and the extent of outcome-reporting bias in RCTs relating to the treatment of distal radial fracture. Inadequate trial protocol registration can lead to outcome-reporting bias[18]. The latter occurs when study outcomes are selectively reported depending on the results, and can lead to the overestimation of treatment effects because of the overrepresentation of positive findings and the exclusion of nonsignificant results. As a result, outcome-reporting bias can induce false-positive findings in systematic reviews and meta-analyses[19-21].

The present study identified poor trial protocol registration rates in recent RCTs relating to the treatment of distal radial fracture, with only 31% (28) of 90 published trials appearing at all in a trial registry. Registered trials were more likely to be multicenter, have larger sample sizes, report a funding source, and be published in higher-impact-factor journals. These findings may represent a higher awareness of the ICMJE statement by the investigators of these studies or may be the effect of a more stringent peer-review process during the study design and publication stages. Nevertheless, only a staggering 3% (3) of 90 studies were correctly registered and therefore provided sufficient detail to assess outcome-reporting bias for the POM of that study. In the few studies that could be fully evaluated, discrepancies were prevalent between the POM reported in the registry and in the published study. These discrepancies could be mostly attributed to omissions and non-specific registration statements, rather than clear signs of biased reporting. The number of trials prespecifying the POM was so low that it is not possible to draw firm conclusions. Nonetheless, the overall findings are concerning, constitute bad science, and suggest a potential waste of resources[22,23].

To reflect the modern literature on distal radial fractures, we investigated reports of trials that took place from 2010 to 2015, resulting in a minimum of 5 and a maximum of 11 years from the implementation of the ICMJE policy in 2005[11] to the publication of the reports utilized in the present study. We believed that this timeframe would allow adequate time for the dissemination and uptake of the IMCJE recommendations and would help to avoid the influencing of results by an early “bedding-in” effect of the ICMJE statement. This belief was supported by the finding that registration status was not affected by year of registration nor by year of publication; in other words, registration rates did not appear to improve with time. This finding is in contrast to global trends in RCT registration across all disciplines, which have increased fivefold between 2004 and 2013[24].

Unlike similar reports in different settings[25-29], the present study was focused on a single condition/injury and setting, distal radial fracture intervention, which may mean that the results are not generalizable to orthopaedic interventions as a whole. However, including all trials on the treatment of an injury, rather than a sample of intervention trials published in specific journals, makes a study of this type less susceptible to selection bias. In the present study, we were able to accurately determine the proportion of registered trials and to assess the quality of registration by focusing on a well-researched orthopaedic injury, performing a comprehensive search across all journals (rather than a selection of journals), and cross-checking individual records. Furthermore, unlike authors of other studies, we did not restrict inclusion solely to trials reporting their registration in the final published report[25,26], but performed a primary search of all relevant distal radial fracture treatment trials in the WHO trial registry; we thus identified an additional 11 registered trials on our topic, making this review as comprehensive as possible.

This study had some weaknesses. We excluded 14 non-English-language publications (Fig. 1), and thus a small number of registered trials amongst those could have been missed. Our definition of “appropriately registered” trials was a generous one and focused on the POM[30]. Outcome-reporting bias for secondary outcomes has been reported in other settings[29] and has been linked to preferential reporting of significant measures[21]; exploring secondary outcomes would have been of interest but would not have changed the overall message of the present study.

The findings of this study are consistent with those in other clinical research settings. Mathieu et al. focused on trials in 3 medical areas (cardiology, rheumatology, and gastroenterology) published in the top 10 general medical or specialty journals with the highest impact factors; they reported that 147 (45.5%) of 323 trials were “adequately” registered[27]. Nankervis et al. assessed registration of eczema trials and reported that only 18 (17%) of 109 trials were “properly” registered[30]. In an evaluation of RCTs across orthopaedic surgical interventions in 10 major journals, Rongen and Hannink reported that 34 (9.3%) of 362 trials were registered adequately; the authors evaluated the POM in 26 of those 34 trials to find that 14 had 1 or more reporting discrepancy[25]. In addition, Chan and Altman reported that outcome-reporting bias is prevalent in published trials that have been approved by an ethics committee and indexed on PubMed[12].

A lack of clarity in submission guidelines might help to explain inadequate trial registration. Despite many journals requiring a registration number for publication, some journals use vague language in their instructions to authors[27]. Unclear language has also been suggested as a reason for a lack of adherence to reporting guidelines[31]. Our interpretation is that journals that published non-registered studies either do not subscribe to or do not fully adhere to the ICMJE guidance. Furthermore, even journals that do require trial registration for publication may not routinely cross-check the registration record, and peer reviewers may not be aware of the issues and how they can affect outcome-reporting bias.

Along with outcome-reporting bias, publication bias can contribute to the overestimation of treatment effects in the published literature. In 2013, Sando et al. evaluated the presence of publication bias in the available primary literature on distal radial fracture treatment[32]. The authors concluded that publication bias likely exists and that certain study characteristics—such as treatment type, external funding, and study outcome type—are associated with the reporting of positive outcomes. This combination of outcome-reporting bias and publication bias poses an increased risk of false-positive findings in current and future systematic reviews and meta-analyses relating to distal radial fracture treatment[19-21], the conclusions of which are used to inform clinical practice.

In conclusion, the present study shows poor registration of RCTs on the treatment of distal radial fracture, which is one of the most common and highly investigated injuries in orthopaedic practice, and inconsistencies in the reporting of the POM of such studies. These problems in RCT registration are important to highlight and address because a solution will require the endorsement and cooperation of researchers, reviewers, journal editors, and the scientific community as a whole, and will be achieved through improved trial registration, improved quality control procedures of trial registries, and wider implementation of the ICMJE criteria. Finally, a core outcome set for RCTs relating to the treatment of distal radial fracture would substantially increase the transparency and consistency of outcome reporting[5,6]. A core outcome set is a consensus minimum set of outcomes that should be measured and reported in all trials relating to a specific condition and is developed with the input of all relevant stakeholders, including patients, researchers, clinicians, and policy-makers.

Minimizing outcome-reporting bias will help to increase the quality of research, which can impact patient care through evidence-based clinical practice, treatment guidelines, and central resource allocation.

Appendix

A list of the database search criteria as well as a table showing the 90 included studies by name of first author, year, and journal of publication is available with the online version of this article as a data supplement at jbjs.org (http://links.lww.com/JBJSOA/A54).

TABLE I

Inclusion and Exclusion Criteria

Criteria
Inclusion criteria
Study design
Studies described as randomized controlled trials
Studies stated to be “randomized” but for which there is inadequate information about sequence generation and/or concealment of allocation
Quasi-randomized studies, such as those with alternate allocation or allocation based on day of the week or clinic
Population: adults with a distal radial fracture
Intervention: any intervention for the treatment of distal radial fractures in adults
Comparator
Any intervention for the treatment of distal radial fracture in adults
No treatment
Placebo/sham intervention
Timeframe of trial report publication: January 1, 2010, to December 31, 2015
Studies reported to be a further follow-up of a previously reported RCT
Exclusion criteria
Trial report published in languages other than English, German, French, or Greek, for which we had no resources to translate the text
Interim-analysis trial report publications
Trial protocol publications
Separate publications of further follow-up in which the primary report of the RCT was published prior to January 1, 2010
Separate publications of an economic evaluation of a primary trial
Registered RCTs published as abstracts only without an associated full text publication

TABLE II

Databases and Platforms Utilized

Database	Platform	Coverage
PubMed	PubMed	1946-2016
Cochrane Central Register of Controlled Trials (Cochrane CENTRAL)	Wiley	1999-2016
Embase	OVID	1980-2016
PEDro	PEDro	1929-2016

TABLE III

Data Providers of the WHO International Clinical Trials Registry Platform (ICTRP)

Data Providers of ICTRP Search Portal
Australian New Zealand Clinical Trials Registry
Brazilian Clinical Trials Registry
Chinese Clinical Trial Registry
Clinical Research Information Service, Republic of Korea
ClinicalTrials.gov
Clinical Trials Registry - India
Cuban Public Registry of Clinical Trials
EU Clinical Trials Register
German Clinical Trials Register
Iranian Registry of Clinical Trials
ISRCTN.org
Japan Primary Registries Network
Pan African Clinical Trials Registry
Sri Lanka Clinical Trials Registry
The Netherlands National Trial Register

TABLE IV

Characteristics of Registered and Non-Registered Trials

	Registered RCTs (N = 28)	Non-Registered RCTs (N = 62)	95% CI of Difference*	P Value
Journal impact factor*	3.1 ± 1.92	1.6 ± 1.18	0.4 to 2.5	0.008
Sample size*	109 ± 9.52	63 ± 6	20.2 to 73.2	<0.001
RCT study design				<0.001
Single-center	9 (32%)	45 (73%)
Multicenter	14 (50%)	7 (11%)
Inadequate information	5 (18%)	10 (16%)
Maximum follow-up* (days)	328 ± 15.17	253 ± 15.91	−36.6 to 186.3	0.185
Funding source				0.008
Reported	24 (86%)	35 (56%)
Not reported	4 (14%)	27 (44%)
Country of origin				0.934
Europe and North America	12 (43%)	26 (42%)
Rest of the world	16 (57%)	36 (58%)

Data are presented as the mean and standard deviation.

Data are presented as the number of studies, with the percentage in parentheses.

TABLE V

Registered Trials per Year of Publication

Year of Publication	Registered Trials	Total Trials	Percentage Registered
2010	4	15	27%
2011	8	19	42%
2012	2	11	18%
2013	4	15	27%
2014	3	13	23%
2015	7	17	41%