Selective reporting bias in randomised controlled trials from two network meta-analyses: comparison of clinical trial registrations and their respective publications.

OBJECTIVE: To determine (i) the difference in the frequency of serious adverse events (SAEs) reported in trial registrations and their respective primary publications and (ii) the effect of adding SAE data from registries to a network meta-analysis (NMA) in changing the surface under the cumulative ranking (SUCRA) curve values of interventions.
DESIGN: Secondary analysis of primary publications from two NMAs. ELIGIBILITY CRITERIA FOR SELECTING STUDIES: We included randomised trials published in English after 2005 that were included in two NMAs of pharmacological interventions for Alzheimer's disease and chronic obstructive pulmonary disease. DATA EXTRACTION: Two reviewers independently searched multiple international trial registries for registration status and abstracted data from the included study publications and ClinicalTrials.gov.
RESULTS: Of the 203 randomised trials included, 140 (69.0%) were registered with a trial registry and 72 (35.5%) posted results in the registry. The proportion of registered trials increased over time (38.5% in 2005 vs 78.6% in 2014). Of the publications with results posted in a trial registry, 14 (19.4%) had inconsistent reporting of overall SAEs; 7 (10.4%) studies did not report SAEs in the publication but did in the registry. In the 134 randomised trials with a prespecified primary outcome in the registry, 19 studies (9.4%) had a change in the primary outcome in the publication. Adding SAEs reported in registries to the NMAs did not affect the ranking of interventions.
CONCLUSION: We identified inconsistent reporting of SAEs in randomised trials that were included in two NMAs. Findings highlight the importance of including trial registries in the grey literature search and verifying safety data before incorporating it into NMAs. STUDY REGISTRATION: osf.io/mk6dr. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study analysed 203 randomised trials from two large network meta-analyses on Alzheimer’s disease and chronic pulmonary obstructive disease for consistency in reporting serious adverse events (SAEs) between trial registries and the publication.

We searched multiple international trial registries in duplicate to ensure accuracy of the trial registration information and restricted inclusion of trials to those published after 2005 reflecting implementation of mandatory reporting requirements.

A published method was used to grade changes to the primary outcome from trial registry to publication.

We included SAE results from only a single trial registry (ClinicalTrials.gov) because it is the only one that provides these data openly.

We did not contact study authors to confirm SAE counts in the trial registry.

Introduction

Selective or incomplete reporting occurs when authors do not include results for any outcome that is expected to be reported for a study.1 Selective reporting of clinical trial results can lead to erroneous conclusions about drug safety and efficacy, which can have a major impact on patients.2–6 In response, the US government created a clinical trials registry in 2000 where investigators can disclose ongoing clinical trials with information about planned outcome measurement (https://clinicaltrials.gov).7 In 2005, the International Committee of Medical Journal Editors (ICMJE) mandated the registration of clinical trials for their results to be eligible for publication in its member journals.8 In 2007, the Food and Drug Administration (FDA) required registration and reporting of summary results for all new drugs seeking regulatory approval.7 The FDA further required mandatory posting of adverse events in 2009.7 Clinical trial registries in other countries were created in subsequent years,9 for example, the European Union Clinical Trials Register was created in 2009 in accordance with Article 11 of Directive 2001/20/EC.10 The International Clinical Trials Registry Platform was created by WHO to deliver an amalgamated search across different national trial registries.11 Despite regulatory changes, compliance with trial registration and consistency in outcome reporting continue to be low.12–16

The validity of a knowledge synthesis, including network meta-analysis (NMA), is dependent on the quality of primary studies, which includes appropriate outcome reporting. Registration of clinical trials provides transparency in interpreting published results because the intended primary outcome is known. Similarly, summary results for adverse events reported in trial registries should be identical to those reported in the primary publication. Adverse events (AEs) are often incompletely and inconsistently reported in publications compared with trial registries.17 ClinicalTrials.gov defines an AE as an unfavourable change in the health of a participant, including abnormal laboratory findings, which happens during a clinical study or within a certain amount of time after the study has ended; this change may or may not be caused by the intervention being studied.18 Serious adverse events (SAEs) are AEs that result in death, are life-threatening, require inpatient hospitalisation or extend a current hospital stay, result in an ongoing or significant incapacity or interfere substantially with normal life functions or cause a congenital anomaly or birth defect.18 According to Consolidated Standards of Reporting Trials (CONSORT) clinical trial reporting standards, all AEs, and particularly SAEs, should be completely reported in publications.19

Any discrepancy between the information provided in the registry and publication that is not disclosed may indicate concealment of results because of the statistical significance, magnitude or direction of the effect, leading to selective reporting bias. For example, cognitive enhancers for managing Alzheimer’s disease (AD) are only modestly effective,20 as such, if selective outcome reporting occurs in published trials, it may significantly change the perceived risk-benefit ratio of this drug class. Selective reporting happens regardless of whether differences are explained by authors.21 Previous studies revealed changes in summary estimates with addition of trial registry data for pair-wise meta-analyses (MA)22 23; similar findings for NMAs have not been reported.

Tang et al 24 investigated the consistency between SAEs in ClinicalTrials.gov and their corresponding publications in a random sample of 300 randomised trials. SAE reporting was consistent between the two sources in only 11% of the registered trials.

We aimed to investigate the prevalence of discrepant reporting of SAEs in randomised trials included in two NMAs that compared the relative efficacy and safety of cognitive enhancers in AD NMA20 and long-acting inhaled agents for chronic obstructive pulmonary disease (COPD NMA).25 We evaluated SAEs and not all AEs because SAEs should be consistently reported in both ClinicalTrials.gov and the manuscript. We also examined the frequency of trial registration and the impact of incorporating unpublished data reported in trial registries on NMA results.

Methods

Study design

We conducted a retrospective review of the published journal articles and corresponding registration records for randomised trials included in two systematic reviews, the AD NMA20 and the COPD NMA.25 Our protocol was registered with the Open Science Framework (Identifier: osf.io/mk6dr).

Objectives

Our primary objective was to determine if there was a difference in the frequency of overall SAEs reported in the clinical trial registrations and the respective primary publications.

As secondary objectives, we investigated:

The proportion of randomised trials that reported trial registration information (eg, ID number) in the publication.

The proportion of randomised trials that were registered in any publicly accessible trial registry.

The proportion of randomised trials that had consistent reporting of the frequency of overall SAE.

The proportion of randomised trials that referred to the primary outcome specified in the trial registry.

The relative risk of each type of primary outcome change (eg, new, exclusion, upgrade, downgrade, change in definition or measure)26 in relation to whether the study conclusion is positive, determined using a previously defined method.27

The difference in ranking of the treatment groups for SAEs in the two NMAs after adding SAE data from the trial registry.

Eligibility criteria

We included randomised trials from the NMAs, since other study designs were not likely to be registered.28 Eligible randomised trials were published in English in 2005 or later, aligned with the ICMJE mandate for clinical trial registration.29

Study procedure

Eligible randomised trials were scanned by two independent reviewers (EKCW, CCL) for a trial registration number. If no number was identified, we searched for the study in the most frequently used trial registries including ClinicalTrials.gov, European Union Clinical Trials Register and International Standard Randomised Controlled Trial Number Register (via the WHO International Clinical Trials Registry Platform).30 If we could not find a trial in these registries, we searched the national registry in the country where the trial was conducted. For identification of SAEs, we only included trials registered on ClinicalTrials.gov because other registries currently do not allow for the posting of trial results. Two investigators (EKCW, CCL) independently searched the registries for studies that did not declare registration status in its publication. In each database, the search strategy included drug name (generic and brand) and condition name (eg, AD), with manual review of the trial entries.

Two investigators independently abstracted the data from the included publications and trial registrations. Abstraction of SAEs was adapted from a previously described method.24 A calibration exercise was conducted first to establish consistency in data extraction. A third reviewer (ACT) adjudicated any discrepancies that could not be resolved by the two research team members (EKCW, CCL). The following data were abstracted:

Study details: first author, year of publication, country (-ies) of trial conduct (number, name), clinical condition (AD, COPD), journal, journal impact factor (JIF), disclosure of industry funding, sample size (total and per treatment arm).

Registration details: registration status, registration number, presence of registration number in publication, presence of results on registration website.

Outcome reporting: presence of primary outcome (in registry, in publication), primary outcome study results (statistically negative or positive (unfavourable or favourable: p<0.05), non-statistically negative or positive (unfavourable or favourable: p≥0.05), neutral indeterminate/unclear, or non-comparative),26 study conclusions (positive (significant), positive (non-significant), no effect, negative (non-significant), negative (significant)).27 Outcome data were abstracted in February 2018 for all studies, which would allow >12 months mandatory reporting period for all included clinical trials.31

SAE reporting in registry and publication: definition of SAEs, total number of SAEs, number of SAEs per treatment arm, number of participants per treatment arm.

Data analysis

We used χ2 tests to determine if there were differences between the clinical trial registrations and the respective publications. If inconsistencies were found between trial registration and the publication for any outcome data, we reviewed the publication for an explanation (eg, determined if a non-pre-specified outcome with statistically significant results was substituted). Differences in the primary outcome between registry and publication were graded using a published method26 into the following categories: no change, change in definition or measure, added (completely new outcome measure), upgrade (secondary outcome changed to primary), omitted (excluded primary outcome), downgrade (primary outcome changed to secondary). The abstract was used for the primary outcome change grading. The OR (95% CI) of each type of primary outcome change was determined in relation to whether the study conclusion was positive or not. Differences between data in the registry and publication were described. Analyses were conducted using SAS V.9.4 (SAS Institute, Cary, North Carolina, USA).

NMA re-analysis

Newly identified SAE data from the registry were added to the initial dataset for re-analysis of the NMA.32 Transitivity and consistency were explored in the original NMAs.20 25 A frequentist random-effects NMA model was applied to derive the summary effect estimates, along with 95% CIs and 95% predictive intervals using Stata (V.15, StataCorp). For heterogeneity, we used a common within-network between-study variance estimated with the restricted maximum likelihood method. Surface under the cumulative ranking (SUCRA) curves were used to compare treatment rankings for total SAE33 and displayed in a rank-heat plot.34

Patient and public involvement

This research was done without patient involvement. Patients were not invited to comment on the study design and were not consulted to develop patient-relevant outcomes or interpret the results. Patients were not invited to contribute to the writing or editing of this document for readability or accuracy.

Results

Included studies

Of the 318 randomised trials in the original systematic reviews, 89 were excluded because they were published before 2005 and 26 were excluded because they were written in languages other than English (figure 1). Thus, this study included 203 studies (AD NMA n=67; COPD NMA n=136).

Figure 1

Flow diagram for included studies. AD, Alzheimer’s disease; COPD, chronic obstructive pulmonary disease; NMA, network meta-analysis; SAE, serious adverse event.

Registration status

The registry search identified that 140 (69.0%) studies from the two NMAs had registered their trials (AD 59.7%, COPD 73.5%). However, only 49.3% (n=100) of studies reported trial registration in their publication. Significantly more registered trials reported an industry funding source (80.0% vs 20.0%, p<0.001; table 1) compared with non-registered trials. All trials that did not provide a funding source were not registered. Multinational trials were more likely to be registered (90.1% vs 9.9%, p<0.001). Studies published in journals with higher impact factors were more likely to be registered (100.0% in JIF ≥15.000 vs 23.0% in JIF <2000, p<0.001). Studies with a larger sample size were more likely to be registered (100.0% with n≥1000 vs 46.3% with n<50, p<0.001). Trial registration increased with more recent publications (78.6% in 2014 vs 38.5% in 2005, figure 2).

Table 1

Study characteristics by registration status

	Registered, n (%)	Not registered, n (%)	Total	P value
	140 (69.0)	63 (31.0)	203
Industry funding
Yes	132 (80.0)	33 (20.0)	165	<0.001
No	8 (72.7)	3 (27.2)	11
Not reported	0	27 (100.0)	27
Country
Multinational	82 (90.1)	9 (9.9)	91	<0.001
Europe	26 (50.0)	26 (50.0)	78
North America	18 (64.2)	10 (35.7)	28
Asia	6 (23.1)	20 (76.8)	26
South America	2 (66.7)	1 (33.3)	3
Australia (Oceania)	0	1 (100.0)	1
Africa	1 (100.0)	0	1
Not reported	1 (100.0)	0	1
Clinical condition
Alzheimer’s disease	40 (59.7)	27 (40.3)	67	0.045
COPD	100 (73.5)	36 (26.5)	136
Journal impact factor
0–1.999	4 (23.5)	13 (76.5)	17	<0.001
2.000–4.999	86 (69.9)	37 (30.1)	123
5.000–9.999	17 (60.7)	11 (39.3)	28
10.000–14.999	14 (87.5)	2 (12.5)	16
≥15.000	19 (100.0)	0	19
Registration stated in publication
Yes	109 (100.0)	0	109	<0.001
No	31 (33.0)	63 (67.0)	94
Sample size
0–49	19 (46.3)	22 (53.7)	41	<0.001
50–99	6 (31.6)	13 (68.4)	19
100–199	17 (65.4)	9 (34.6)	26
200–499	41 (75.9)	13 (24.1)	54
500–999	27 (81.8)	6 (18.2)	33
≥1000	30 (100.0)	0	30

COPD, chronic obstructive pulmonary disease.

Figure 2

Trial registration by year of publication and total number of studies from each year. Solid blue line, registered trials; dotted red line, non-registered trials; grey bars, number of trials published in each year.

Change in trial primary outcome

Of the 140 registered trials, 134 (95.7%) reported a prespecified primary outcome in the registry (n=125 efficacy, n=7 safety and n=2 unclear). Most of the primary outcome results were statistically positive (n=74, 36.5%) or neutral (n=31, 15.3%), and a minority (n=15, 7.4%) were non-significant or negative (online supplementary appendix A, online supplementary table 1). Comparing the registry and the publication primary outcome, 19 studies (9.4%) had a change in the primary outcome. Of those studies, six studies changed the outcome definition or measure, five added a different outcome, four upgraded a secondary outcome, three omitted a primary outcome and one downgraded a primary outcome (online supplementary appendix A, online supplementary table 2).

The majority of study conclusions were positive (79.3%, online supplementary appendix A, online supplementary table 3), but the odds of having a positive conclusion was not statistically different between studies with changed or unchanged primary outcomes. Overall, more studies with changes to their primary outcome had a positive conclusion, although the absolute number of studies was small (table 2).

Table 2

Outcome changes in relation to a positive conclusion

	Positive conclusionn (%)	Non-positive conclusionn (%)	Total	OR (95% CI)
No change	86 (74.8)	29 (25.2)	115	Reference
Change in definition or measure	3 (50.0)	3 (50.0)	6	2.97 (0.57 to 15.51)
Added	4 (80.0)	1 (20.0)	5	0.74 (0.08 to 6.90)
Upgrade	3 (75.0)	1 (25.0)	4	0.99 (0.10 to 9.88)
Omitted	3 (100.0)	0	3	--
Downgrade	1 (100.0)	0	1	--

Frequency of overall SAEs

Of the 140 registered studies, 72 studies (51.4%, AD NMA n=13, COPD NMA n=59) reported SAE data in the registry. Of these 72 studies, 80.6% (n=58) had consistent reporting in the total number of SAEs (primary outcome) between the registry and publication, while 19.4% (n=14) had a discrepancy (table 3). Of the 14 studies with a discrepancy, 3 reported more, 4 reported fewer and 7 did not report SAE counts in the publication (online supplementary appendix A, online supplementary table 4). Studies published more recently (p=0.160) and in journals with higher impact factor (p=0.258) had fewer discrepancies in total SAE count. Declaring registration information in the publication was associated with fewer SAE discrepancies (17.4% vs 66.7%, p=0.035). Other study characteristics (country of origin, clinical condition and total sample size) did not show a difference in total SAE counts. All studies with registry SAE data were industry-sponsored. The overall proportion of discrepant SAE events compared with total events was small (2.9%, n=452/15 807).

Table 3

Association between study characteristics and presence of discrepancy in total SAE count between publication and trial registry

	Discrepancy in SAE, n (%)	No discrepancy in SAE, n (%)	Total	P value
	14 (19.4)	58 (80.5)	72
Year of publication
2008	0	1 (100.0)	1	0.160
2009	1 (100.0)	0	1
2010	4 (44.4)	5 (55.6)	9
2011	3 (20.0)	12 (80.0)	15
2012	2 (11.1)	16 (88.9)	18
2013	3 (14.3)	18 (85.7)	21
2014	1 (14.3)	6 (85.7)	7
Industry funding
Yes	14 (19.4)	58 (80.6)	72	N/A
No	0	0	0
Not reported	0	0	0
Country
Multinational	11 (19.3)	46 (80.7)	57
North America	2 (18.1)	9 (81.8)	11
Europe	1 (50.0)	1 (50.0)	2
Asia	0	1 (100.0)	1
Not reported	0	1 (100.0)	1
Clinical condition
Alzheimer’s disease	2 (15.4)	11 (84.6)	13	0.683
COPD	1 (20.3)	47 (79.7)	59
Journal impact factor
0–1.999	1 (100.0)	0	1	0.258
2.000–4.999	9 (19.2)	38 (80.6)	47
5.000–9.999	1 (16.7)	5 (83.3)	6
10.000–14.999	2 (28.6)	5 (71.4)	7
≥15.000	1 (9.1)	10 (90.9)	11
Registration stated in publication
Yes	12 (17.4)	57 (82.6)	69	0.035
No	2 (66.7)	1 (33.3)	3
Sample size
0–49	1 (9.1)	10 (90.9)	11	0.870
50–99	0	2 100.0	2
100–199	1 (33.3)	2 (66.7)	3
200–499	3 (18.8)	13 (81.3)	16
500–999	4 (22.2)	14 (77.8)	18
≥1000	5 (22.7)	17 (77.3)	22

COPD, chronic obstructive pulmonary; SAE, serious adverse effect.

Re-analysis of NMAs for SAEs

We re-analysed the SAE outcomes of the AD NMA by adding new data as identified from ClinicalTrials.gov. The COPD NMA was not re-analysed because the total SAE was not an outcome of interest in the original NMA. We aimed to look at specific SAEs (eg, pneumonia, fractures), but the coding of the events was different between the registry and publications, making comparisons difficult. For example, a pneumonia may be classified as cough, dyspnoea, hypoxia, lung infiltration, acute respiratory failure, lung infection or lobar pneumonia.

Only two studies had undisclosed SAE from the AD NMA (nine new events), but the overall SUCRA ranking statistics were not changed when these data were added (online supplementary appendix A, online supplementary table 5). When the SUCRA ranking statistics were calculated by registration status, different treatments were favoured between registered, non-registered and combined subgroups (online supplementary appendix A, online supplementary table 6). The rankings are presented in a rank-heat plot (figure 3).

Figure 3

Rank-heat plot for nine treatments and placebo in randomised clinical trials with patients with Alzheimer’s disease for total serious adverse events. *Each sector is coloured according to SUCRA curve of the corresponding treatment as total SAEs. The scale consists of the transformation of three colours: red (0%), yellow (50%) and green (100%). White sectors show that the underlying treatment was missing from the network meta-analysis for the particular outcome. Numbers within each sector correspond to the SUCRA values as estimated in each network meta-analysis. DONE, donepezil; GALA, galantamine; MEMA, memantine; PLAC, placebo/no treatment; RIVA_O, rivastigmine oral; RIVA_P, rivastigmine transdermal (patch); SUCRA, surface under the cumulative ranking.

Discussion

To our knowledge, this is the first study to explore selective reporting bias in trials included in NMAs by comparing the publication results to their respective trial registry data. We found that 19.4% of publications with results posted in a clinical trial registry had inconsistent reporting of overall treatment. The most common discrepancy was not reporting the SAE data in the primary publication (50.0%), which could be addressed by following the CONSORT guidelines19 to enhance reporting. This result is similar to a study that examined SAE discrepancies between ClinicalTrials.gov and publications using a random set of registry records.24 Reviews of pairwise MAs showed changes in summary estimates but not the overall conclusion when registry data were added.22 23 Although we uncovered discrepancies in reporting, we did not find significant differences in the SAE rankings of interventions when we repeated the AD NMA with the corrected data; this is likely because the relative proportion of discrepant SAE events was small overall (~3%).

Since the discrepancies in SAE counts included 15 events that were in the publication but not in the registry, the role of clerical or misclassification error in these discrepancies should be considered.29 A potential safeguard to avoid errors in SAE reporting is to ensure that the posting of results in trial registries is performed by someone who is intimately involved with the data and has methodological and/or clinical expertise, such as the principal investigator or study sponsor. Peer reviewers should also consider cross-checking key event rates with trial registries before recommending a manuscript for publication.

The frequency of trial registration in this cohort of studies was consistent with systematic reviews in other clinical areas,12 14 35–39 for example, 58.7% of oncology trials in solid tumours registered their trials.39 Only 51% of registered trials from our study included results and SAEs counts in the registry, which is more than a review of trials of diverse clinical conditions published from 2008 to 2013 (38.3%).31 Although the FDA Amendments Act requires clinical trials with any enrolment site in the USA to have trial results posted within 12 months of completion, compliance is low.31 40 Extensions may be allowed under certain circumstances, but full or partial results should be released within 2 years.41 Our study searched the registry at least 3 years after trial completion, allowing for late data to be reviewed. Despite this, overall compliance with results transparency was suboptimal.

To identify trial registrations not stated on the primary publication, we searched multiple international registries in duplicate. This strategy found a registry record for 38.8% (n=40/103) of publications not declaring trial registration, a proportion similar to the 35% in a similar study of trials in clinical geriatrics.37 In the clinical geriatrics study, the authors only searched the WHO International Clinical Trial Registry Platform, which searches multiple national registries simultaneously.42 There is no other amalgamated search tool to identify trial registrations internationally, highlighting an important gap for researchers and knowledge users.

Our study has certain limitations to consider. First, we included primary publications from NMAs investigating two clinical conditions (AD and COPD) and, therefore, results may not generalise to other fields. Second, we excluded primary publications if they were not written in English. As ClinicalTrials.gov is the most comprehensive clinical trial registry and requires results to be published in English, it is unlikely that our study conclusions would have changed had we sought to translate the non-English language articles. Third, we searched only a single trials registry (ClinicalTrials.gov) for SAE results because it is the only one that provides these data openly. Fourth, we were unable to repeat the COPD NMA because specific SAEs were classified differently in the publication and registry, and total SAE counts were not analysed in the original NMA. Fifth, we did not contact study authors to confirm SAE counts in the trial registry.

Conclusions

Our findings offer insights on how to improve transparency in trial data and comprehensiveness in knowledge synthesis. To improve consistency between trial registry data and publications, we recommend that all trial registry results are uploaded by the same responsible parties that prepare the published manuscript (ie, principal investigator, study sponsor). The authors of systematic reviews should be mindful about the possibility of reporting bias in primary publications and should routinely search trial registries to reduce the risk of reporting bias. Safety data should be verified within these registries before incorporating it into MAs and NMAs. Peer reviewers and journal editors should also consult the trial registry record of the manuscript under review to ensure that the results are consistent and accurately reported.