Published trials of TACE for HCC are often not registered and subject to outcome reporting bias.

BACKGROUND & AIMS: In 2005, the registration of all randomised controlled trials (RCTs) before enrolment of participants became a condition for publication by the International Committee of Medical Journal Editors to increase transparency in trial reporting. Among RCTs on transarterial chemoembolisation (TACE) for the treatment of hepatocellular carcinoma (HCC) published after 2007, we assess the proportion that were registered and compare registered primary outcomes (PO) with those reported in publications to determine whether primary outcome reporting bias favoured significant outcomes.
METHODS: We searched MEDLINE and EMBASE for reports of RCTs evaluating TACE for HCC treatment between 1 September 2007 and 31 March 2018. Registration and publication information for each included RCT was compared using a standardised data extraction form.
RESULTS: Thirteen out of 53 (25%) included RCTs were correctly registered (i.e. before the starting date of the RCT), 14 (26%) were registered after the RCT starting date, and 26 (49%) were not registered. Six out of 14 of the retrospectively registered RCTs (43%) were registered after their completion date. The PO was clearly reported in the published article of all registered RCTs, whereas the report was not clear in 8/26 (31%) of the non-registered RCTs (p = 0.01). Among registered RCTs, 8/27 (30%) had major discrepancies between registered and published PO. The influence of these discrepancies could be assessed in 6 of them and was shown to statistically favour significant results in 2.
CONCLUSIONS: Registration and outcome reporting in RCTs on TACE for HCC are often inadequate. Registration should be reinforced because it is a key to transparency. LAY
SUMMARY: Trial registration is fundamental to our understanding and interpretation of results, as it provides information on all relevant clinical trials (to place the results in a broader context), and on the details of their associated protocols (to ensure that the scientific plan is followed). Once a randomised controlled trial (RCT) is completed, the trial results are usually publicly shared via scientific articles that are expected to thoroughly and objectively report them. This study shows that half of the RCTs evaluating transarterial chemoembolisation for hepatocellular carcinoma were not registered, and identified major discrepancies between registered and published primary outcome favouring significant results.

Introduction

Liver cancer, which is mainly hepatocellular carcinoma (HCC), is the third leading cause of cancer-related death worldwide, and its incidence is expected to increase in the future. In 2002, in the studies by Llovet et al. and Lo et al. transarterial chemoembolisation (TACE) was shown to have a significant benefit to survival compared with the best supportive care for the treatment of HCC. Almost 2 decades later, TACE continues to be the first-line treatment for most HCC patients., At the same time, research on the efficacy and safety of TACE remains active, either by innovating TACE treatment using emerging technologies, or by evaluating new treatment strategies and combinations of TACE with other treatments.

In 2005, the International Committee of Medical Journal Editors (ICMJE) initiated a registration policy for randomised controlled trials (RCTs) to increase transparency in research. All RCTs that started recruiting on or after September 2005 were required to be registered prospectively on a publicly accessible web-based registry before participant enrolment as a prerequisite for publication in member journals. Indeed, trial registration is fundamental to our understanding and interpretation of results, as it provides information on all relevant clinical trials (to place the results in a broader context), and on the details of their associated protocols (to ensure that the scientific plan is followed)., One of the main objectives of registration is to improve the transparency of study results.

Once an RCT is completed, the trial results are usually publicly shared via scientific articles that are expected to thoroughly and objectively report them. However, there may be significant differences between trial registration and published results. The term ‘outcome reporting bias’ refers to the selective reporting of results in trial publications, depending on the nature and direction of the results. Although a study may be published in full, pre-specified outcomes may be omitted or misrepresented to report results more favourably.

The registration and selective reporting of studies on TACE for HCC have not been examined. Thus, our study had three objectives: (1) to evaluate the proportion of registered RCTs on TACE treatment for HCC among trials published after 2007, (2) to compare registered primary outcomes (POs) with those actually reported in publications, and finally (3) to determine whether primary outcome reporting bias favoured significant outcomes.

Materials and methods

We performed a methodological 2-step review of RCTs evaluating the efficacy and/or safety of TACE for the treatment of HCC. First, we identified all RCTs published after September 2007 and systematically searched for registration on online registries. Then, for each registered RCT, we assessed possible selective outcome reporting.

Search strategy for published RCTs

We systematically searched MEDLINE (PubMed interface) and EMBASE (Elsevier interface) on April 2019 to identify eligible published RCTs from 1 September 2007 to 31 March 2018 (see Supplementary material for search terms and equations). We chose September 2007 to leave a reasonable delay after the prospective registration requirement date (i.e. September 2005) by the International Committee of Medical Journal Editors. We also screened reference lists of included studies, and relevant systematic reviews and meta-analyses.[11], [12], [13], [14], [15], [16] All the records from these searches were imported via EndNote into Rayyan (a free web application for screening abstracts) to proceed with the screening.

Identification of relevant published RCTs

We identified all RCTs that assessed the efficacy and/or safety of TACE for HCC, regardless of the phase, from the retrieved records. To focus on trials in which TACE was the primary treatment and was not an associated treatment, we excluded RCTs if the same TACE procedure was combined with another treatment and was therefore present in all groups or arms. We also excluded systematic reviews or meta-analyses, diagnostic studies, methodological publications, editorial style reviews, abstracts and posters, conference papers, case reports, studies not involving human participants, or studies that were not in English. One reviewer (JG) examined each title, keywords, and abstract and then selected full-text articles according to the pre-specified eligibility criteria.

Data extraction of published RCTs

One reviewer (JG) extracted all data from the original reports and supplementary files according to a standardised data collection form (Supplementary material). For quality assurance a second reviewer (PC) collected data from a random sample of 10% of the RCTs. Disagreement was resolved by discussion.

The following characteristics were systematically recorded from the published article: study location (defined as Eastern-Asia, Western-Europe, North America, and North Africa; location of international RCTs was determined according to the primary investigator); number of involved centres (either single or multicentre); presence of a university-affiliated centre; phase of the RCT when stated; funding source (profit, non-profit, or mixed), sample size (defined as total number of randomised patients); date of journal publication; RCT start and completion dates; type of journal (i.e. radiological or clinical), median journal impact factor in the 2 years before publication; and we additionally collected the interventions to which TACE was compared in each RCT.

We also assessed the methodological quality of the included RCTs using selected items from the Cochrane risk of bias tool, in particular, random sequence generation and allocation concealment. In addition, we investigated whether the delay between randomisation and the TACE procedure, the reasons for and percentages of withdrawal or patient dropouts, and a flow chart were reported in the published article. Finally, we evaluated whether the intention-to-treat analysis was correctly performed.

RCT registration

One reviewer (JG) systematically looked to see if each published RCT was registered.

In particular we checked if the authors mentioned the registration of their trial in the published article and whether a registration number was provided.

When no registration number was reported in the published article, we searched for a corresponding registration using a combination of keywords, including a description of the interventions and population, location, responsible party, and primary outcome name. We first searched meta-platforms including the Cochrane Central Register of Controlled Trials (CENTRAL) and the International Clinical Trials Registry Platform (ICTRP). We then searched the following primary registries: Clinicaltrials.gov, Clinicaltrialsregister.eu, the Chinese Clinical Trial Registry (ChiCTR), and the University Hospital Medical Information Network Clinical Trials registry (UMIN-CTR).

RCT registrations were classified according to the ICMJE statement: (1) prospectively registered if the initial registration date was before or within 1 month after the trial start date as stated in the registry (to allow for incomplete start dates and processing delays in the registry); (2) retrospectively registered; and (3) not registered.

Selective outcome reporting

Finally, we evaluated the consistency between the primary outcomes as specified in the registered RCT (when possible in the version before patient inclusion began, including amendments, and if not in the most recent available version), and those defined in the published articles. We also recorded whether the primary outcome (PO) was clearly identified in the article (defined and graded as primary and secondary upfront), and if multiple primary outcomes were investigated in the same RCT.

We defined major discrepancies according to the classification by Chan et al., as those in which (1) a pre-specified primary outcome in the RCT registration protocol was later reported as a secondary outcome or was not labelled as either in the final published article; (2) a pre-specified primary outcome was omitted from the published articles; (3) the published primary outcome was described as a secondary outcome in the registry; and (4) a new primary outcome was introduced in the published articles.

Results of the registered and published primary outcomes

We extracted the p values from the full text article for all the registered primary outcomes and for all the outcomes reported in the article. We quoted results according to statistical significance: results significantly supporting or refuting the study intervention (i.e. p <0.05) and results that did not reach statistical significance (i.e. p ≥0.05).

A discrepancy was said to favour statistically significant results if a new statistically significant primary outcome was introduced or if a non-significant primary outcome was defined as non-primary in the published article. We also judged the discrepancy as positive when a new statistically non-significant safety primary outcome was introduced in the published article. The influence of some discrepancies could not be assessed because the published text contained no results concerning the registered primary outcome. Thus, the positivity or negativity of the discrepancy was considered to be impossible to decide, and the study was considered impossible to assess.

Statistical analysis

Quantitative variables are presented as medians (quartiles 1–3, Q1–Q3) and qualitative variables as n (%). Logistic regression was used to identify RCT characteristics associated with RCT registration status. Two-sided p <0.05 was considered to be statistically significant. All data were analysed with R version 4.2 (https://www.r-project.org, the R Foundation for Statistical Computing, Vienna, Austria), (CTAT Table, Supplementary material).

Data sharing

The final data spreadsheet, including the list of eligible RCTs, is available in the Supplementary material.

Results

The search retrieved 6,104 references. A total of 349 reports were selected after screening titles, abstracts, and keywords. A total of 75 published RCTs were eligible and 53 were included for analysis. The flow chart is reported in Fig. 1.

Fig. 1

Selection of randomised controlled trials.

HCC, hepatocellular carcinoma; TACE, transarterial chemoembolisation.

Registration of included RCTs

The description of included articles is presented in Table 1. Twenty-six (49%) RCTs published after September 2007 (i.e. 2 years after the ICMJE statement for prospective registration of RCTs) were not registered. Registration of 14 of the 27 registered RCTs was retrospective (52%). Six of these retrospectively registered RCTs (43%) were registered after their completion date. RCTs that were retrospectively registered before their completion date were more recent (median [Q1–Q3], 2011 [2010-2013]) than those registered after their completion date (2007 [2004-2009], p = 0.02).

Table 1

General characteristics of included randomised controlled trials by registration status.

Characteristics	Registeredn = 27		Not registeredn = 26	p value
	Prospectivelyn = 13	Retrospectivelyn = 14
Study location				0.63
Eastern-Asia (n = 39)	8 (21)	11 (28)	20 (51)
Western-Europe (n = 9)	2 (22)	3 (33)	4 (45)
North America (n = 4)	3 (75)	0 (–)	1 (25)
North Africa (n = 1)	0 (–)	0 (–)	1 (100)
Number of centres				0.20
Multicentre (n = 12)	4 (33)	4 (33)	4 (33)
Single-centre (n = 53)	9 (22)	10 (24)	22 (54)
Type of centre				0.21
University affiliated (n = 43)	11 (26)	12 (28)	20 (46)
Non-university affiliated (n = 3)	0 (–)	0 (–)	3 (100)
Both (n = 7)	2 (29)	2 (29)	3 (42)
Funding source				0.08
Non-profit (n = 50)	11 (22)	13 (26)	26 (52)
Profit (n = 3)	2 (66)	1 (33)	0 (–)
Mixed (n = 0)	0 (–)	0 (–)	0 (–)
Phase				0.13
II (n = 8)	3 (37)	2 (25)	3 (37)
III (n = 10)	6 (60)	2 (20)	2 (20)
n.a. (n = 35)	4 (11)	10 (29)	21 (60)
Median sample size (min–max)	189 (92–247)	131 (93–238)	97 (52–125)	<0.01
Type of journal				0.27
Radiological (n = 14)	3 (21)	2 (14)	9 (65)
Clinical (n = 39)	10 (25)	12 (31)	17 (44)
Median date of publication (Q1–Q3)	2016 (2013–2017)	2014 (2012–2016)	2011 (2009–2013)	<0.01
Median date of first inclusion (Q1–Q3)	2010 (2009–2013)	2010 (2008–2012)	2008 (2004–2011)	0.47
Median journal impact factor (min–max)	15 (7.6–20.9)	5.6 (3.4–7.2)	2.3 (1.5–2.9)	0.02

Unless otherwise noted, data are presented as n (%). Percentages are calculated per row. Univariate logistic regression analyses were performed to compare the variables.

Information on RCT registration was reported in the text of the article in 85% of the registered RCTs (13/13 of the prospectively registered and 10/14 of the retrospectively registered studies, p = 0.05).

Prospectively registered RCTs were more recently published (median [Q1–Q3], 2016 [2013-2017]) than retrospectively registered (2014 [2012-2016]) and non-registered (2011 [2009-2013], p <0.01) studies. The median (Q1–Q3) journal impact factor was 15 (7.6–20.9) for prospectively registered RCTs, 5.6 for retrospectively registered RCTs and 2.3 for non-registered RCTs (p = 0.02).

Registration rate by type of intervention TACE is compared with is presented in Fig. 2.

Fig. 2

Network graph of included randomised controlled trials showing the registration rate by type of intervention compared with TACE.

Each node represents a treatment and each edge a randomised comparison of two treatments. Thickness of each edge increases with the number of randomised comparisons. Interventions are pooled according to main treatment category. For each group of intervention, number and percentage of registered trials is presented. oTACE, other type of TACE; PA, percutaneous ablation; TAC, transarterial chemotherapy; TACE, transarterial chemoembolisation; TAE, transarterial embolisation; SBRT, stereotactic body radiation therapy; SIRT, selective internal radiotherapy; Surg, surgery.

Methodological characteristics of RCTs according to registration status

A flow chart was reported in 13/13 of the prospectively registered RCTs, 12/14 retrospectively registered RCTs (86%), and 8/26 of the non-registered RCTs (31%), (p <0.01). None of the prospectively registered RCTs reported a high-risk/unclear randomisation sequence or allocation concealment compared with 2/14 (14%) and 2/14 (14%) of the retrospectively registered RCTs, and 14/26 (54%) and 17/26 (65%) of the non-registered RCTs (p <0.01), respectively. All prospectively registered RCTs (13/13) performed an intention-to-treat analysis, compared with 13/14 (93%) of the retrospectively registered RCTs, and 15/26 (58%) of the non-registered RCTs (p <0.01) (Table 2).

Table 2

Methodological characteristics of included randomised controlled trials by registration status.

Characteristics	Registeredn = 27		Not registeredn = 26	p value
	Prospectivelyn = 13	Retrospectivelyn = 14
Sequence generation∗ (high/unclear)	0	2 (14)	4 (54)	<0.01
Allocation concealment∗ (high/unclear)	0	2 (14)	17 (65)	<0.01
Reported delay between randomisation and transarterial chemoembolisation	6 (46)	8 (57)	7 (27)	0.28
Dropouts or withdrawal reported	13 (100)	13 (93)	23 (88)	0.20
Reason of dropouts reported	13 (100)	13 (93)	20 (77)	0.13
Intention to treat performed	13 (100)	13 (93)	15 (58)	<0.01
Presence of a flow chart	13 (100)	12 (86)	8 (31)	<0.01

Data are presented as n (%). Percentages are calculated per column. Univariate logistic regression analyses were performed to compare the variables.

According to the Cochrane risk of bias tool.

Primary outcome of RCTs according to registration status

The primary outcome was clearly defined in the published article of all registered RCTs (27/27), whereas this was not reported clearly in 8/26 non-registered RCTs (31%) (p <0.01). Multiple primary outcomes were reported in 1/27 registered RCT publications (4%), and in 8/26 non-registered RCTs (31%) (p = 0.01). Among registered RCTs, multiple primary outcomes were initially registered for 0/13 prospectively registered RCTs and 4/14 retrospectively registered RCTs (29%) (p = 0.01).

Selective outcome reporting for registered RCTs

Eight of the 27 registered RCTs (30%) had major discrepancies between the registered PO and those reported in the publication article (Table 3). The frequency of discrepancies was higher in retrospectively registered RCTs (7/14, 50%) than in prospectively registered RCTs (1/13, 8%) (p = 0.01). The list of major discrepancies is shown in Box 1. Only 1 of the 4 registries that initially reported multiple POs reported the same POs in the published article; the 3 others reported a single PO.

Table 3

Proportion of RCTs with major discrepancies in the specification of primary outcomes when comparing registries and published articles.

Discrepancy between RCT report and registration (n = 27)	RCTs with discrepancies for primary outcome
Primary outcomes specified in protocols∗
Any change to registration-defined primary outcome	8 (30)
Reported as secondary in published articles	6 (22)
Omitted from published articles	3 (11)
Primary outcomes specified in published articles
Any new primary outcome defined in published articles	5 (19)
Changed from secondary in registration to primary in published articles	2 (7)
Not mentioned in protocol	2 (7)
Any discrepancy in primary outcome†	8 (30)

Data are presented as n (%).

Categories are not mutually exclusive: 1 RCT could have more than 1 type of discrepancy for different primary outcomes.

Primary outcomes defined in either registration or published articles. RCT, randomised controlled trial.

Box 1

List of major discrepancies between RCT registry and published article in the specification of primary outcomes.

RCT, randomised controlled trial.

The influence of the discrepancy could be assessed in 6/8 (75%) of the RCTs with a PO discrepancy between the registry and the published article. The registered PO was omitted from the published article in the two remaining RCTs. Two of the 6 RCTs in which the discrepancy could be assessed, had a discrepancy that favoured statistically significant results (1 had a discrepancy that favoured statistically significant secondary outcomes and 2 non-significant primary outcomes; the other study had a new statistically non-significant safety primary outcome introduced in the published article).

Discussion

In a previous study, we showed that despite the ethical commitments and public expectations for the disclosure of results, the availability of RCT results evaluating TACE for the treatment of HCC is very limited. In this study we focused on RCTs on the same topic with results published after 2007, and showed that only half were registered. It is important to note that only 25% were prospectively registered as recommended by the ICJME guidelines. We also found that one-third of registered RCTs had major discrepancies between the registered and published primary outcomes, with 7% presenting with selective outcome reporting favouring the trial results.

Registration rates have already been studied in other specialties also highlighting the low proportion of correctly registered RCTs, although with some discrepancies. Rosenthal and Dawn focused on registered RCTs published in 2010 in 3 leading general surgical journals. This study identified 55 RCTs including 4 (8%) which were not registered, 46 (84%) which were retrospectively registered, and only 5 (9%) which were prospectively registered. Bonnot et al. investigated the registration of 183 RCTs published in 2013 in 12 high-impact anaesthesia journals. These authors showed that trial registration was lacking in 70 published reports (38%), and that 60 (53%) and 53 (29%) were retrospectively and prospectively registered, respectively. Finally, Mathieu et al. evaluated 323 RCTs published in 2008 in a panel of medical specialties in high-impact journals, and reported that trial registration was lacking in 89 (28%) published reports, whereas 45 (15%) had been retrospectively registered and 147 (46%) had been prospectively registered. The fact that we did not focus on high-impact journals only probably explains the higher rate of unregistered trials in our study. Despite this difference, the proportion of prospectively registered RCTs observed in our study appears to be similar to that reported by Bonnot et al. and Mathieu et al., but higher than that of surgical RCTs published in high-impact journals. Although we covered a longer and later time period than Rosenthal and Dwan, the comparison seems valid because interventional oncology is often compared with surgery owing to similarities between the 2 disciplines, with particular difficulty in designing RCTs, and complex interventions that are difficult to evaluate.

The 2005 ICJME initiative for trial registration is a significant step towards more transparent and accountable research. However, the timing of registration must still be improved. Indeed, only 25% of RCTs were registered prospectively. This is also in line with a previous report. One possible explanation is that researchers need time to adjust to new regulations and recommendations. We chose to analyse the results of RCTs published after 2007 to take this into consideration, but integrating new habits into research protocols may take more time. The fact that prospectively registered RCTs were published more recently than retrospectively registered or unregistered studies seems to support this hypothesis. Furthermore, the involvement of institutional boards and trial funders is likely to help enforce prospective trial registration.,

It is important to note that registration and especially prospective registration was associated with higher methodological standards (e.g. less allocation concealment, more intention-to-treat analysis, more frequent flow chart, etc.). This is probably because of the higher standards of the journals publishing the results of registered RCTs, as confirmed by their higher impact factors. Indeed, clear journal submission guidelines are associated with better adherence to ICJME standards, which are mainly endorsed by journals with a high impact factor. Altogether, trial registration and publication standards create a virtuous circle that is beneficial to the scientific community and, more importantly, to patients. Nevertheless, the positive effect of trial registration on the quality of research reporting cannot prevent possible selective reporting of outcomes.

Publications have shown that discrepancies between registries and publications are frequent.,, A systematic review comparing registered and published outcomes in RCTs, including 27 studies from a panel of medical and surgical specialties, reported a median proportion of discrepancy of 31%, which is similar to our results. However, we may have underestimated the selective reporting of outcomes associated with statistical significance because we could only access registered protocols. Indeed, the extent of selective reporting in the results of unregistered RCTs remains unknown. Of course, the decision to omit outcomes from publications is not necessarily a sign of dishonest practices. It may be related to a combination of space constraints, a lack of clinical importance, and statistical results., Nevertheless, selective reporting may result in a bias of favourably reporting efficacy which could cause physicians to perform interventions with no evidence of benefit. Similarly, under-reporting toxicity could result in patients being exposed to treatments that may not be optimal, even with proven benefit.

Interventional radiologists, like other physicians, are generally not well-trained to identify outcome reporting bias. However, they should be aware that study reports may be biased, despite ICJME guidelines. One might argue that it is the role of publishers and peer reviewers to detect these types of bias to reduce their frequency in published results. Unfortunately, our results suggest that they do not sufficiently take advantage of the benefits of the transparency provided by registration. Indeed, we found some evidence of selective outcome reporting in one-third of published results. Therefore, as the consistency of outcomes between the registered protocol and published reports are available in online registries, readers should make a habit of consulting this information when a study can have a significant impact on clinical practices. Moreover, we believe that tracking registration status and selective outcome reporting are fundamental steps in the conduct of a meta-analysis as well as in the development of recommendations, as their omission could impact subsequent conclusions and statements.

Our study has limitations. First, it only focused on a specific treatment (TACE for treating HCC). Nevertheless, TACE remains the most frequent first line treatment in patients with HCC. We did not include RCTs with combinations of TACE with other treatments because it would have been methodologically difficult to analyse the heterogenous data. Also, we did not contact the authors of unregistered RCTs to identify unreported POs and determine whether they did not reach statistical significance. Finally, we did not address the issue of selective reporting of non-primary outcomes, which may be more common. The strength of this study is that it is the first to show that under-registration and selective outcome reporting are common in one of the most extensively studied and performed interventional oncology procedures.

Conclusion

Trial registration improves the transparency of research and is a safeguard against reporting bias or other deviations from the planned study. The registration and timing of registration in RCTs on TACE for HCC must still be improved, and outcome reporting is too often inadequate in the results of registered trials. The registration of RCTs in interventional oncology of the liver should be reinforced.

Financial support

JG received a PhD grant (DOC20180507331) from the .

Authors' contributions

Study design: JG, MR, IB.

Data collection: JG, PC.

Data analysis: JG, PC, MR, IB.

Manuscript drafting: JG.

Manuscript revision: all authors.

Manuscript final approval: all authors.

Data availability

The authors declare that data supporting the findings of this study are available within the article and its Supplementary material.

Conflict of interest

The authors declare no conflict of interest than pertain to this work.

Please refer to the accompanying ICMJE disclosure forms for further details.