Is there an efficacy-effectiveness gap between randomized controlled trials and real-world studies in colorectal cancer: a systematic review and meta-analysis.

BACKGROUND: To investigate whether patients with colorectal cancer (CRC) enrolled in randomized controlled trials (RCTs) and real-world studies (RWS) differ in terms of baseline characteristics, leading to an efficacy-effectiveness gap.
METHODS: A systematic literature reviews was conducted to identify RCTs and RWS with CRC, treated with bevacizumab (BEV), cetuximab (CET) or oxaliplatin combined with capecitabine (XELOX). Using random-effects meta-analyses compared the baseline characteristics and treatment effects of RCTs and RWS, overall and by drug. Correlation between treatment effects and baseline characteristics and study types were estimated using meta-regression analyses.
RESULTS: Two hundred and fifty-three studies were included. Compared with patients enrolled in RWS, the proportion of male patients in RCTs was 0.032 higher (P=0.004), the proportion of patients with Eastern Cooperative Oncology Group (ECOG) performance ≥2 was 0.085 less (P<0.001). No significant differences in treatment effects [progression-free survival (PFS), overall survival (OS), objective response rate (ORR), disease control rate (DCR)] were found by overall analysis. But the OS of patients in RCTs was 4.184 higher (P=0.023) in the CET group. Meta-regression results showed that OS difference in the CET group was related to the difference in treatment lines, not related to other baseline characteristics and study types.
CONCLUSIONS: No efficacy-effectiveness gap was found in CRC between RCTs and RWS. CRC treatment effects Between RCTs and RWS had high consistency. 2020 Translational Cancer Research. All rights reserved.

Introduction

In the process of developing clinical diagnosis and treatment guidelines and healthcare policy, it is essential to obtain valid clinical trial evidence, in which randomized controlled trials (RCTs) are recognized as the gold standard for evaluating interventions (1). In most countries, such as the United Kingdom, Canada, and South Korea, the development of health decision-making and clinical practice guidelines are based on research-based RCTs (2). With the increasingly complicated situation and high cost of cancer treatment, the conducting clinical trials in cancer are facing more challenges. People have begun to realize that RCTs do not match the real-world environment and lack external validity, due to moderately and highly standardized trial designs, strict patient inclusion and exclusion criteria, and short follow-up time (3). Unlike RCTs, real-world studies (RWS) are a type of research that reflects the actual clinical diagnosis and treatment process, based on the real-world data. Principles of its research design are mainly non-randomization, non-intervention, and openness, which are closer to the actual clinical treatment environment and have higher external validity. RWS have received an increasing amount of attention, since the United States Congress passed the 21st Century Cures Act in 2016, which made it clear that the FDA could use real-world data as evidence of approval for post-marketing research and new indications for medical devices and drugs, where appropriate. In 2018, the FDA announced Real-World Evidence Program, which presents a detailed standard for evaluating the quality of real-world evidence. Recently, the FDA approved a new indication for Pfizer’s Ibrance based on the real-world data, which is the first drug indication approved by the FDA based on real-world data. RWS immediately ignited the hot topic (4,5).

There has been much controversy about the application and differences in results between RCTs and RWS. A study by Jaksa et al. (6) showed that RWS may amplify the positive effects of interventions and allow health policymakers to make favorable decisions. A study by Naudet et al. (7) showed that RCTs are more efficient than RWS in the study of treatment for major depression. Some studies (8-12) have compared the baseline characteristics and treatment effects of patients in RCTs and RWS and showed that RCTs tend to include patients with better prognostic factors and high treatment effects. They also proposed the concept of the efficiency-effectiveness gap to describe the gap between treatment effects observed in RCTs and those observed in RWS. However, other studies (13-18) have shown that most RCTs in the same disease and treatment methods have very similar results to RWS. As the design and reporting quality of RWS improve, respectively, the consistency with the results of RCTs becomes higher.

Although there is much debate about the differences between RCTs and RWS, comparative studies for colorectal cancer (CRC) are still lacking. no valid evidence is available to indicate the difference between RCTs and RWS in CRC. Based on previous studies, we performed a meta-analysis to investigate whether patients with CRC enrolled in RCTs and RWS differ in terms of baseline characteristics, leading efficacy-effectiveness gap. Oxaliplatin combined with capecitabine (XELOX), and targeted drugs [e.g., cetuximab (CET), bevacizumab (BEV)] combined with chemotherapy should be used as effective first- and second-line treatments for chemotherapy-resistant patients with metastatic CRC according to NCCN Clinical practice guidelines in oncology (version 1.2017) (19) and The Chinese Diagnosis and Treatment Specification of Colorectal Cancer (2017 edition) (20). Therefore, this study selected XELOX, CET monotherapy or combined chemotherapy, BEV monotherapy or combined chemotherapy as the therapeutic regimens.

We present the following article in accordance with the PRISMA reporting checklist (available at http://dx.doi.org/10.21037/tcr-20-2303).

Methods

Literature search strategy

We searched Medline and Embase to find relevant articles published from 20 September 2009 to 20 September 2019 in English using the main search terms “bevacizumab”, “cetuximab”, “XELOX” and “colorectal cancer”. Considering the incomplete development of real-world research methods, the database search was limited to last 10 years of research. In addition, references for secondary research were manually retrieved to supplement the original research literature. Specific search strategies show in .

Table 1

Search strategy

No.	Search strategy
1	(colorectal cancer or CRC or Colorectal carcinoma or Colorectal neoplasms).ti,ab,ot,hw,rn.
2	(Cetuxim* or Erbitux).ti,ab,ot,hw,rn.
3	(Bevacizum$b or CAPOX-B).ti,ab,ot,hw,rn.
4	(Oxaliplatin or L-OHP or OXA).ti,ab,ot,hw,rn.
5	(capecitabine or Xeloda or ECX).ti,ab,ot,hw,rn.
6	4 and 5
7	XELOX or CapeOX.ti,ab,ot,hw,rn.
8	Or/6-7
9	Or/2,3,8
10	1 and 9
11	limit 10 to yr=“2009-current”

Study selection

Titles and abstracts of all retrieved literature were imported into the NoteExpress V3.2.0. The repeat literature was removed. Two reviewers (XZ and SF) independently performed the study selection, including screening titles and abstracts, and evaluating full-text eligibility of potentially eligible studies. Discussion or negotiation with a third party was implemented if there were divergences. If necessary, we contacted the original authors by email or phone to obtain unidentified information.

Included studies need to meet the following criteria: (I) studies that enrolled patients with CRC treated with BEV, CET or XELOX; (II) studies that reported on at least one of the following clinical outcomes: (i) primary outcomes: progression-free survival (PFS), overall survival (OS); (ii) secondary outcomes: response rate (RR) including disease control rate (DCR), objective response rate (ORR), complete response rate (CR), partial response rate (PR), and stable disease (SD) based on the measurement of cancer antigen 125 levels confirmed by radiological examination results or by combined Gynecologic Cancer InterGroup criteria.

Studies not meeting the inclusion criteria were excluded. Other exclusion criteria were: (I) studies in which BEV, CET or XELOX was used as neoadjuvant treatments; (II) studies with a sample size of less than 30; (III) non-English studies.

Data extraction

Data from each included paper were extracted into a standardized spreadsheet developed for this project by two reviewers independently with adjudication by a third reviewer: study characteristics (e.g., title, author, publication year, study design, country, study horizon, follow-up time, trial name, and registration number); treatments (e.g., drug, dose, frequency, and cycle); patient characteristics (e.g., sample size, age, gender, Eastern Cooperative Oncology Group (ECOG), treatment line, tumor location, and transfer); treatment effects (e.g., PFS, OS, RR, DCR, ORR, CR, PR, and SD). We extracted frequency number and percentages. All patients included in the study were fully enrolled in the primary studies, and no witching over treatment or treatment discontinuation.

Data synthesis and statistical analysis

Data on patient baseline characteristics (age, proportions of male, proportion of patients with ECOG ≥2, proportion of patients with second-line and above second-line treatment) and treatment effects (PFS, OS, ORR, DCR) were finally analyzed. The ORR = CR + PR and DCR = ORR + SD were used to process the tumor response results. The methods described by Wan et al. (21) were used to convert the mean and range of continuous variables such as age, PFS, and OS into mean and standard deviation, whereas the other variables were presented as ratios. We first combine the baseline characteristics and treatment effects of CRC patients in RCTs and RWS using random-effect meta-analyses, and subsequently to compare the difference of the combined results.

We used meta-regression analyses to assess the heterogeneity by including the baseline characteristics as covariates, the study design as a dichotomous covariate, and treatment effects as dependent variables. We used restricted maximum-likelihood estimation to assess between-study variance (tau-squared) and applied the Knapp-Hartung adjustment (22).

Considering the follow-up time, treatment cycle and duration would have a major impact on the treatment effects, a comparative analysis of follow up time, treatment cycle and duration between RCT and RWS was added. All analyses were done in the Stata SE15.

Results

Characteristics of included studies

We identified 6,147 records through database searching, and 2 potentially eligible studies through other sources. After duplicate checking and title and abstract screening, 369 full-text articles assessed for eligibility. Finally, 369 full-text articles assessed for eligibility. Finally, 201 articles were eventually included: 117 RCTs including 94 phase II clinical trials, 6 phase III clinical trials, and 17 unknown phase clinical trials; 84 RWS including 36 case series, 13 registry, 20 cohort, and 15 unknown category of studies. There were 102 studies on BEV treatment, 54 studies on CET treatment, and 45 studies on XELOX treatment. A total of 37,479 patients were included, with 13,889 patients in RCTs and 23,590 patients in RWS. The process and results of article selection show in . The main characteristics of all studies show in .

Figure 1

Flow chart. RCT, randomized controlled trial; RWS, real-world studies; BEV, bevacizumab; CET, cetuximab; XELOX, oxaliplatin combined with capecitabine.

Table 2

Baseline characteristics of RCTs

No.	Reference	Year	Study phase	Country/region	Sample size	Drug	Characteristics	Outcomes	Registration number
1	Kim et al. (23)	2019	Phrase II	Korea	60	BEV	1,2,3,4	5,7,8	NCT02026583
2	Cremolini et al. (24)	2019	Phrase II	Italy	117	BEV	1,2,4	5,7,8	NCT02271464
3	Suzuki et al. (25)	2019	Phrase II	Japan	51	BEV	1,2,3,4	6,8	UMIN 000009280
4	Nakayama et al. (26)	2018	Phrase II	Japan	54	BEV	1,2,4	5,6,7,8	UMIN000006478
5	Oki et al. (27)	2018	Phrase II	Japan	69	BEV	1,2,3,4	5,6,7	NCT02246049
6	Jonker et al. (28)	2018	Phrase II	Canada	51	BEV	1,2,3,4	8	NA
7	Satake et al. (29)	2018	Phrase II	Japan	62	BEV	1,2,3,4	5,6,7	NA
8	Matsuda et al. (30)	2018	Phrase II	Japan	51	BEV	2,3,4	5,6,8	NA
9	Ulivi et al. (31)	2018	Phrase I/II	Italy	65	BEV	1,2,4	5,7,8	NA
10	Venook et al. (32)	2017	NA	USA	559	BEV	1,2,3,4	5,8	NCT00265850
11	Nakayama et al. (33)	2017	Phrase II	Japan	52	BEV	1,2,3,4	5	UMIN000006478
12	Apsangikar et al. (34)	2017	NA	India	33	BEV	2,3,4	5,8	NA
13	Zhao et al. (35)	2017	Phrase II	China	122	BEV	1,2,3,4	5,8	NA
14	Baba et al. (36)	2017	Phrase I/II	Japan	256	BEV	4	7,8	NA
15	Matsui et al. (37)	2016	Phrase II	Japan	51	BEV	1,2,3,4	5,7	NA
16	Ogata et al. (38)	2016	NA	Japan	47	BEV	1,2,3,4	5,6,7	NA
17	Yamazaki et al. (39)	2016	Phrase I/II	Japan	197	BEV	1,2,3,4	8	UMIN000001396
18	van Hazel et al. (40)	2016	Phrase I/II	Australia	263	BEV	1,2,3,4	5	NA
19	Stintzing et al. (41)	2016	Phrase I/II	Germany	201	BEV	2,3,4	5,7,8	NA
20	Shitara et al. (42)	2016	Phrase II	Japan	58	BEV	1,2,3,4	8	NA
21	Hagman et al. (43)	2016	NA	Sweden	35	BEV	1,2,3,4	8	NCT01229813
22	Benson et al. (44)	2016	Phrase II	USA	88	BEV	2,3,4	5	NCT01478594
23	Shimomura et al. (45)	2016	Phrase II	Japan	55	BEV	1,2,3,4	5,6,7,8	NA
24	Passardi et al. (46)	2015	Phrase I/II	Italy	176	BEV	1,2,4	5,7,8	NCT01878422
25	Antonuzzo et al. (47)	2015	Phrase I/II	Italy	197	BEV	1,2,3,4	5,7,8	NCT00577031
26	Iwamoto et al. (48)	2015	Phrase I/II	Japan	181	BEV	1,2,3,4	–	UMIN000002557
27	Hegewisch et al. (49)	2015	Phrase I/II	Germany	158	BEV	1,2,3,4	8	NCT00973609
28	Masi et al. (50)	2015	Phrase I/II	Italy	92	BEV	1,2,3,4	5,6,8	NCT00720512
29	Cao et al. (51)	2015	Phrase II	China	65	BEV	1,2,4	5,6,8	NA
30	Wang et al. (52)	2015	NA	China	114	BEV	1,2,3,4	5,6,8	NA
31	Garcia et al. (53)	2015	Phrase II	Spain	77	BEV	1,2,3,4	5,6,7,8	NCT00875771
32	Liu et al. (54)	2015	Phrase II	China	30	BEV	1,2,3,4	5,8	NA
33	Nakayama et al. (55)	2015	Phrase II	Japan	40	BEV	1,2,3,4	5,6,7,8	UMIN000001127
34	Heinemann et al. (56)	2014	Phrase I/II	Germany	295	BEV	1,2,3,4	5,7,8	NCT00433927
35	Duran et al. (57)	2014	NA	Turkey	298	BEV	2,3,4	5,7,8	NA
36	O'Neil et al. (58)	2014	Phrase II	USA	49	BEV	1,2,3,4	5	NA
37	Uygun et al. (59)	2013	NA	Japan	64	BEV	1,2,3,4	5,8	NA
38	Schmiegel et al. (60)	2013	Phrase II	Germany	127	BEV	1,2,3,4	7,8	NA
39	Kochi et al. (61)	2013	Phrase II	Japan	39	BEV	1,2,3,4	5,6,7	NA
40	Bennouna et al. (62)	2013	Phrase I/II	France	409	BEV	1,2,3,4	5,8	NCT00700102
41	Ducreux et al. (63)	2013	Phrase II	France	72	BEV	1,2,3,4	5,7,8	NA
42	Cunningham et al. (64)	2013	NA	UK	66	BEV	2,3,4	5,8	NA
43	Yalcin et al. (65)	2013	Phrase I/II	Turkey	62	BEV	1,2,3,4	5,7,8	NA
44	Johnsson et al. (66)	2013	Phrase I/II	Sweden	80	BEV	1,2,3,4	8	NCT00598156
45	Hong et al. (67)	2013	Phrase II	Korea	57	BEV	1,2,3,4	5,8	NA
46	Stintzing et al. (68)	2012	NA	Germany	46	BEV	1,2,3,4	5,6,7,8	NCT00433927
47	Pectasides et al. (69)	2012	Phrase I/II	Australia, New Zealand	143	BEV	1,2,3,4	5,7,8	NA
48	Díaz-Rubio et al. (70)	2012	Phrase I/II	Spain	241	BEV	1,2,3,4	5,7,8	NA
49	Hurwitz et al. (71)	2012	Phrase II	USA	217	BEV	2,3,4	5,8	NCT00159432
50	Renouf et al. (72)	2012	Phrase II	Canada	50	BEV	1,2,3	5,6	NA
51	Wolff et al. (73)	2012	Phrase II	USA	58	BEV	1,2,3,4	–	NA
52	Tang et al. (74)	2012	Phrase II	NA	51	BEV	1,2,3	8	NA
53	Yamada et al. (75)	2012	Phrase II	Japan	51	BEV	1,2,3	5,6	NA
54	Wong et al. (76)	2011	Phrase I/II	NA	31	BEV	2	–	NA
55	Guan et al. (77)	2011	Phrase I/II	China	139	BEV	1,2,3,4	5,7,8	NCT00642577
56	Altomare et al. (78)	2011	Phrase II	USA	50	BEV	1,2,4	8	NCT00597506
57	Kopetz et al. (79)	2010	Phrase II	USA	43	BEV	1,2,4	5,8	NA
58	Bruera et al. (80)	2010	Phrase II	NA	50	BEV	1,2,4	5,8	NA
59	Masi et al. (81)	2010	Phrase II	Italy	57	BEV	1,2,3,4	5,6,7,8	NCT01163396
60	Tebbutt et al. (82)	2010	Phrase I/II	Australia, New Zealand	157	BEV	1,2,3,4	5,7,8	NA
61	Aranda et al. (83)	2018	Phrase II	NA	129	CET	1,2,3,4	7	NA
62	Kotake et al. (84)	2017	Phrase II	Japan	60	CET	1,2,3,4	5,6,7	NA
63	Kataoka et al. (85)	2017	Phrase II	Japan	32	CET	2,3,4	5,6	NA
64	Stintzing et al. (41)	2016	Phrase I/II	NA	199	CET	2,3	5,8	NA
65	Hazama et al. (86)	2016	Phrase II	Japan	40	CET	1,2,3,4	5,6,7,8	NA
66	Bowles et al. (87)	2016	Phrase II	NA	43	CET	1,2,3	5,6,8	NA
67	Ciardiello et al. (88)	2016	Phrase II	Italy	74	CET	1,2,4	5,6,8	NA
68	Eng et al. (89)	2016	Phrase II	NA	60	CET	1,2,3,4	5,6,7,8	NA
69	Soda et al. (90)	2015	Phrase II	Japan	62	CET	1,2,3,4	5,6,7,8	NA
70	Sclafani et al. (91)	2015	Phrase I/II	UK	119	CET	2,3,4	5,6	NA
71	Do et al. (92)	2015	Phrase II	USA	30	CET	1,2,4	5,7	NA
72	Élez et al. (93)	2015	NA	NA	72	CET	1,2,3	5,8	NA
73	Fernandez et al. (94)	2014	Phrase II	Spain	99	CET	1,2,3,4	5,6,7,8	NA
74	Heinemann et al. (56)	2014	Phrase I/II	Germany	297	CET	1,2,3,4	5,6,7,8	NA
75	Iwamoto et al. (95)	2014	Phrase II	Japan	60	CET	1,2,3,4	5,6,8	NA
76	Douillard et al. (96)	2014	Phrase II	USA	150	CET	1,2,3,4	5,6,7,8	NA
77	Ye et al. (97)	2014	Phrase II	NA	70	CET	1,2,3	5,6,8	NA
78	Siu et al. (98)	2013	NA	China	374	CET	1,2,3	5,6,8	NA
79	Brodowicz et al. (99)	2013	NA	NA	75	CET	1,2,4	5,6,7,8	NA
80	Hong et al. (100)	2013	NA	NA	40	CET	1,2,3,4	5,6,8	NA
81	Assenat et al. (101)	2011	Phrase II	France	42	CET	1,2,3,4	5,6,7	NA
82	Kullmann et al. (102)	2011	Phrase II	NA	62	CET	1,2,4	5,6,7,8	NA
83	Lim et al. (103)	2011	Phrase II	Asian, Australia	123	CET	1,2,4	5,6,8	NA
84	Van et al. (104)	2011	Phrase I/II	Europe	599	CET	1,2,3,4	5,6,7,8	NA
85	Moosmann et al. (105)	2011	Phrase II	Germany	89	CET	1,2,4	5,6	NA
86	Wong et al. (106)	2011	Phrase II	USA	30	CET	1,2,3	5,8	NA
87	Shitara et al. (107)	2011	NA	NA	30	CET	1,2,3,4	5,6,7	NA
88	Saridaki et al. (108)	2012	Phrase II	USA	30	CET	1,2,3	5,6,8	NA
89	Stintzing et al. (68)	2012	Phrase I/II	Germany	50	CET	1,2,3,4	5,6,7,8	NA
90	Shitara et al. (109)	2012	Phrase II	Japan	30	CET	1,2,3,4	5,6	NA
91	Tveit et al. (110)	2012	Phrase I/II	Europe	194	CET	1,2,3,4	5,7,8	NA
92	Mrabti et al. (111)	2009	Phrase I/II	Morocco	32	CET	1,2,4	5	NA
93	Mizushima et al. (112)	2019	Phrase II	Japan	107	XELOX	1,2,3	–	NA
94	Yoshimatsu et al. (113)	2019	Phrase II	Japan	57	XELOX	1,2	–	ID:000005427
95	Nishimura et al. (114)	2018	Phrase II	Japan	42	XELOX	1,2,3	–	NA
96	Larsen et al. (115)	2017	Phrase II	NA	52	XELOX	1,2,3	–	NCT00964457
97	Danno et al. (116)	2017	Phrase II	Japan	190	XELOX	1,2,3	5	ID:000006742
98	Azria et al. (117)	2017	NA	France	291	XELOX	1,2	–	NA
99	Liu et al. (118)	2016	Phrase II	China	47	XELOX	1,2	5,6	NCT02415829
100	Pilanci et al. (119)	2016	Phrase II	Turkey	30	XELOX	1,2,3	5,8	NO:44140529
101	Feng et al. (120)	2016	Phrase III	China	224	XELOX	1,2	–	NCT00714077
102	Sclafani et al. (121)	2016	Phrase II	UK	50	XELOX	1,2,3	–	NCT00958737
103	Kim et al. (122)	2015	Phrase II	Korea	44	XELOX	1,2,3,4	5,7,8	NCT00677144
104	Wong et al. (123)	2015	Phrase II	USA	52	XELOX	1,2,3	–	NA
105	Kim et al. (124)	2014	Phrase III	Korea	172	XELOX	2,3	–	NCT00677443
106	Zhu et al. (111)	2013	Phrase II	China	32	XELOX	1,2,4	7,8	NA
107	Gérard et al. (125)	2012	NA	France	299	XELOX	–	–	NA
108	Salazar et al. (126)	2012	Phrase II	Spain	45	XELOX	1,2,3	5,6	NA
109	Arbea et al. (127)	2012	Phrase II	Spain	100	XELOX	1,2	–	NA
110	Schou et al. (128)	2012	NA	Denmark	84	XELOX	1,2,3	–	NA
111	Ducreux et al. (129)	2011	Phrase III	France	156	XELOX	1,2,3,4	5,7,8	NA
112	Haller et al. (130)	2011	Phrase III	29countries	944	XELOX	1,2,3	–	NO16968
113	Waddell et al. (131)	2011	Phrase II	UK	45	XELOX	1,2,3	5,6,8	NA
114	Baraniskin et al. (132)	2011	Phrase III	Germany	190	XELOX	2,4	5,7,8	NA
115	Cassidy et al. (133)	2011	Phrase III	UK	317	XELOX	1,2,3,4	8	NO16966
116	Li et al. (134)	2010	Phrase II	China	124	XELOX	1,2,3,4	5,7,8	NA
117	Qvortrup et al. (135)	2010	Phrase II	Denmark	70	XELOX	1,2,3,4	8	NA

Age =1; gender =2; ECOG =3; treat-line =4; ORR =5; DCR =6; PFS =7; OS =8. UK, United Kingdom; USA, the United States of America; NA, not available; BEV, bevacizumab; CET, cetuximab; XELOX, oxaliplatin combined with capecitabine; ECOG, Eastern Cooperative Oncology Group.

Table 3

Baseline characteristics of RWS

No.	Reference	Year	Study design	Country/region	Sample size	Drug	Characteristics	Outcomes
1	Houts et al. (136)	2019	Case series	USA	264	BEV	2,4	7,8
2	Degirmencioglu et al. (137)	2019	Case series	Turkey	114	BEV	4	–
3	Khakoo et al. (138)	2019	Case series	UK	714	BEV	1,2,3,4	7,8
4	Ogata et al. (139)	2019	NA	Japan	55	BEV	1,2,3,4	5,6,8
5	Ottaiano et al. (140)	2019	Registry	NA	31	BEV	1,2,3,4	5,6,8
6	Devaux et al. (141)	2019	NA	France	99	BEV	1,2,3,4	5,6,8
7	Turpin et al. (142)	2018	NA	France	216	BEV	1,2,4	7,8
8	Matsusaka et al. (143)	2017	NA	Japan	424	BEV	1,2,4	8
9	Hasegawa et al. (144)	2017	NA	Japan	58	BEV	1,2,4	5,8
10	Sun et al. (145)	2017	Case series	China	217	BEV	2,3,4	5,6,8
11	Bennouna et al. (146)	2017	Cohort	France	521	BEV	1,2,3,4	8
12	Chapman et al. (147)	2016	Case series	Australia	292	BEV	2,4	8
13	Bai et al. (148)	2016	Registry	China	188	BEV	1,2,3,4	5,7,8
14	Dionísio de Sousa et al. (149)	2016	Case series	France	41	BEV	1,2,4	5,8
15	Kotaka et al. (150)	2016	Cohort	Japan	40	BEV	1,2,3,4	5
16	Wong et al. (151)	2016	Registry	Australia	206	BEV	2,3,4	–
17	Cabart et al. (152)	2016	NA	France	164	BEV	1,2,3,4	8
18	Kocakova et al. (153)	2015	Registry	Czech	357	BEV	1,2,3,4	6,8
19	Hammerman et al. (154)	2015	Cohort	Israel	1,052	BEV	2,4	8
20	Stein et al. (155)	2015	Cohort	Germany	1,777	BEV	1,2,3,4	5,6,8
21	Bai et al. (156)	2015	Cohort	China	175	BEV	1,2,3,4	5,6,8
22	Bencsikova et al. (157)	2015	NA	Czech	964	BEV	1,2,3,4	7,8
23	Tahover et al. (158)	2015	Cohort	Israel	216	BEV	1,2,4	5,6,7,8
24	Kubáčková et al. (159)	2015	Registry	Czech	981	BEV	1,2,4	5,6,7,8
25	Cheng et al. (160)	2015	NA	China	69	BEV	2,4	5,6,8
26	Ohhara et al. (161)	2015	Cohort	Japan	85	BEV	1,2,4	5,6
27	Yang et al. (162)	2014	Case series	Taiwan	95	BEV	2,4	5,6,8
28	Fourrier-Réglat et al. (163)	2014	Cohort	France	411	BEV	1,2,3,4	5,7,8
29	Hofheinz et al. (164)	2014	Cohort	Germany	1,297	BEV	1,2,3,4	–
30	Suenaga et al. (165)	2014	Cohort	Japan	85	BEV	1,2,4	5,6,7,8
31	Uchima et al. (166)	2014	NA	Japan	40	BEV	1,2,4	5,6,7
32	Yin et al. (167)	2014	Case series	China	87	BEV	1,2,4	7
33	Hurwitz et al. (168)	2014	Cohort	USA	1,550	BEV	1,2,3,4	7,8
34	Kiss et al. (169)	2014	Registry	Czech	3,990	BEV	1,2,4	5,7,8
35	Turan et al. (170)	2014	Case series	Turkey	52	BEV	2	–
36	Moscetti et al. (171)	2013	Case series	NA	220	BEV	1,2,3,4	5
37	Cvetanovic et al. (172)	2013	Case series	NA	51	BEV	2,4	6,7
38	Wu et al. (173)	2013	Case series	China	36	BEV	1,2,3,4	6,7,8
39	Meyerhardt et al. (174)	2012	Registry	USA	1,589	BEV	2,3,4	5,8
40	Ghiringhelli et al. (175)	2012	Case series	France	49	BEV	1,2,3	8
41	Yildiz et al. (176)	2010	NA	NA	40	BEV	2,3	5,8
42	Dranitsaris et al. (177)	2010	Case series	Holland	43	BEV	1,2,4	8
43	Rouyer et al. (178)	2018	Cohort	France	389	CET	1,2,3,4	7,8
44	Wu et al. (179)	2018	Case series	China	34	CET	1,2,4	5,7,8
45	Chapman et al. (147)	2017	Case series	Australia	134	CET	2	8
46	Jerzak, et al. (180)	2017	Registry	Canada	278	CET	2,4	8
47	Kim et al. (181)	2017	NA	Korea	147	CET	1,2,4	8
48	Ozaslan et al. (182)	2017	Case series	NA	40	CET	1,2,4	5,6,8
49	Bai et al. (148)	2016	Registry	China	101	CET	1,2,3,4	5,6,7,8
50	Derangère et al. (183)	2016	Cohort	France	52	CET	2,3	–
51	Pinto et al. (184)	2016	Case series	Italy	225	CET	2,3,4	5,6,7,8
52	Uemura et al. (185)	2016	Case series	Japan	64	CET	1,2,3,4	5,6
53	Yamaguchi et al. (186)	2016	Case series	Japan	97	CET	1,2,3,4	5,8
54	Feng et al. (187)	2016	Cohort	China	102	CET	2,3,4	5,6,8
55	Sato et al. (188)	2015	NA	Japan	109	CET	1,2,4	8
56	Wang et al. (189)	2015	NA	China	110	CET	2,3,4	5,6
57	Giampieri et al. (190)	2015	Case series	Italy	46	CET	2	5,6,8
58	Yang et al. (162)	2014	Case series	Taiwan	63	CET	2,4	5,6,7,8
59	Jehn et al. (191)	2014	Registry	Germany	247	CET	2	5,6
60	Kennecke et al. (192)	2013	Registry	Canada	37	CET	1,2,3	8
61	Chen et al. (193)	2013	Case series	Taiwan	50	CET	1,2,4	5,6
62	Santos-Ramos et al. (194)	2013	Case series	Spain	81	CET	2,3,4	–
63	Jehn et al. (195)	2012	NA	Germany	309	CET	1,2,3,4	–
64	Bouchahda et al. (196)	2011	Case series	Europe	91	CET	1,2,3,4	5,8
65	Xu et al. (197)	2019	Case series	NA	108	XELOX	1,2	–
66	Loree et al. (198)	2018	Registry	Canada	151	XELOX	1,2,3	–
67	Sha et al. (199)	2018	NA	NA	95	XELOX	2,3	–
68	van et al. (200)	2017	Case series	Holland	191	XELOX	2	–
69	Nakanishi et al. (201)	2016	Case series	Japan	53	XELOX	1,2	–
70	Karin et al. (202)	2016	Registry	NA	51	XELOX	2	8
71	Spada et al. (203)	2016	Case series	Italy	78	XELOX	1,2,3	5,8
72	Osawa et al. (204)	2014	Case series	Japan	41	XELOX	1,2,3	–
73	Osawa et al. (204)	2014	Case series	Japan	41	XELOX	1,2	–
74	Loree et al. (205)	2014	Cohort	Canada	83	XELOX	2,3	8
75	Chiu et al. (206)	2014	Case series	Hong Kong	110	XELOX	1,2,3	–
76	Loree et al. (207)	2014	Cohort	Canada	76	XELOX	1,2	–
77	Boisen et al. (208)	2014	Cohort	Denmark	211	XELOX	1,2,3	8
78	Qiu et al. (209)	2014	Cohort	China	64	XELOX	1,2,4	7,8
79	Fukuchi et al. (210)	2013	Case series	Japan	108	XELOX	1,2,3	5,6
80	Constantinidou et al. (211)	2013	Case series	UK	34	XELOX	1,2	–
81	Hansen et al. (212)	2012	Cohort	Denmark	89	XELOX	2	–
82	Satram-Hoang et al. (213)	2013	Cohort	USA	122	XELOX	2	8
83	Hansen et al. (212)	2012	Case series	Denmark	89	XELOX	2,4	8
84	Karacetin et al. (214)	2009	Case series	Turkey	34	XELOX	1,2,3	8

Age =1; gender =2; ECOG =3; treat-line =4; ORR =5; DCR =6; PFS =7; OS =8. UK, United Kingdom; USA, the United States of America; NA, not available; BEV, bevacizumab; CET, cetuximab; XELOX, oxaliplatin combined with capecitabine; ECOG, Eastern Cooperative Oncology Group.

Comparison of patient characteristics

Compared with patients enrolled in RWS, the proportion of male patients in RCTs was 0.032 higher (0.613, 0.598 to 0.628 vs. 0.581, 0.565 to 0.597; P=0.004), the proportion of patients with ECOG ≥2 was 0.085 less (0.005, 0.003 to 0.006 vs. 0.090, 0.078 to 0.103; P<0.001). No significant differences in age and treatment line were found ().

Figure 2

Comparison of patient characteristics. (A) Age; (B) gender; (C) ECOG ≥2; (D) treat-line ≥2. ECOG, Eastern Cooperative Oncology Group; RCT, randomized controlled trial; RWS, real-world studies; BEV, bevacizumab; CET, cetuximab; XELOX, oxaliplatin combined with capecitabine; ES, effect size; CI, confidence interval.

Subgroup analysis by drug showed that differences generally were in the same direction for the three drugs: the proportion of male patients in RCTs was 0.060 higher than those in RWS (0.622, 0.580 to 0.664 vs. 0.562, 0.524 to 0.600; P=0.038) in the XELOX group; the proportion of patients with ECOG ≥2 in RCTs was 0.075 less than those in RWS (0.006, 0.003 to 0.008 vs. 0.081, 0.065 to 0.98; P<0.001) in the BEV group, and similar results was also found in the CET group [0.175 less than those in RWS (0.006, 0.003 to 0.009 vs. 0.181, 0.118 to 0.245; P<0.001)]. Furthermore, patients in RCTs were 1.304 years older than those in RWS (59.205, 58.520 to 59.890 vs. 57.901, 56.839 to 58.963; P=0.043) in the BEV group; the proportion of patients with second-line and above second-line treatment in RCTs was 0.350 lower than those in RWS (0.281, 0.136 to 0.427 vs. 0.631, 0.403 to 0860; P=0.012) in the CET group (). More detailed results show in Table S1 and Figures S1−S8.

Comparison of treatment effects

Primary outcomes

No significant differences were found in OS and PFS between RCTs and RWS by overall analysis. The results of subgroup analysis by drug were mostly consistent with the overall analysis, no significant differences were found in the BEV group and XELOX group, but patients in the CET group of RCTs had an OS of 4.184 months higher than that of patients in the CET group of RWS (17.432 months, 15.118 to 19.745 vs. 13.248, 11.281 to 15.215; P=0.023) ().

Figure 3

Comparison of treatment effects. (A) PFS; (B) OS; (C) ORR; (D) DCR. PFS, progression-free survival; OS, overall survival; ORR, objective response rate; DCR, disease control rate; RCT, randomized controlled trial; RWS, real-world studies; BEV, bevacizumab; CET, cetuximab; XELOX, oxaliplatin combined with capecitabine; ES, effect size; CI, confidence interval.

Secondary outcomes

No differences in ORR and DCR were found between RCTs and RWS by overall analysis and subgroup analysis in the BEV group and CET group. However, in the XELOX group, the ORR of patients in RCTs was 0.251 higher than that of patients in RWS (0.563, 0.457 to 0.669 vs. 0.312, 0.214 to 0.410; P=0.001), and DCR was also 20.6% higher than that of patients in RWS (0.936, 0.857 to 1.016 vs. 0.730, 0.646 to 0.814; P=0.001) (). More detailed results show in Table S2 and Figures S9−S16.

Meta-regression analyses result

According to the meta-analysis results, there were OS differences between RCT and RWS in the CET group, and ORR and DCR differences in the XELOX group. Based on the previous analysis, we found no differences in age, gender, ethnicity and other baseline characteristics of the CET group, except for ECOG and treatment line. To explore the reason for OS differences, we performed meta-regression analysis by including ECOG and treatment line as covariates, OS as dependent variables in the CET group. We extracted the proportion of patients with ECOG score ≥2, and the proportion of patients with second-line or above treatment, based on baseline data from the original study. And there were only gender differences in the XELOX group, so we included the proportion of male patients as covariates, ORR and DCR as the dependent variable in the XELOX group. To explore the impact of study design on results, included the study design as a dichotomous covariate in both groups.

The regression results showed that OS differences in the CET group were related to the difference of treatment line and were not related to ECOG and study type (). In the XELOX group, differences in treatment outcomes were independent of baseline characteristics and study type ().

Table 4

Regression analyses of OS in the CET group

OS	Coef.	Std. Err.	t	P	95% CI
Study type	2.924438	2.812611	1.04	0.314	–3.038031 to 8.886906
Treatment line	10.29738	2.341684	–4.4	0.000	–15.26153 to –5.333236
ECOG	2.644013	10.23937	–0.26	0.800	–24.3505 to 19.06248
_cons	21.47765	1.143907	18.78	0.000	19.05267 to 23.90262

OS, overall survival; CET, cetuximab; Coef., coefficient; Std. Err., standard error; CI, confidence interval; ECOG, Eastern Cooperative Oncology Group.

Table 5

Regression analyses of ORR in the XELOX group

ORR	Coef.	Std. Err.	t	P	95% CI
Study type	–0.2529	0.112954	–2.24	0.052	–0.50842 to 0.002623
Gender	0.422701	0.484584	0.87	0.406	–0.673505 to 1.518906
_cons	0.262381	0.294841	0.89	0.397	–0.404595 to 0.929357

ORR, objective response rate; XELOX, oxaliplatin combined with capecitabine; Coef., coefficient; Std. Err., standard error; CI, confidence interval.

Table 6

Regression analyses of DCR in the XELOX group

DCR	Coef.	Std. Err.	t	P	95% CI
Study type	0.0055461	0.0924147	0.06	0.962	–1.168694 to 1.179786
Gender	1.428532	0.4492353	3.18	0.194	–4.279555 to 7.136596
_cons	0.1183735	0.3428134	–0.35	0.788	–4.475501 to 4.238754

DCR, disease control rate; XELOX, oxaliplatin combined with capecitabine; Coef., coefficient; Std. Err., standard error; CI, confidence interval.

In addition, although the case number of RWS reporting follow-up time, treatment cycle, and duration was lower than that of RCT, the t-test results for mean follow-up time, treatment cycle, and duration between RCT and RWS showed no significant difference ().

Table 7

T-test of follow up time, treatment cycle and duration

T-test	Study type	Case number	Mean	SD	P value
Follow up time/month	RCT	56	29.6352	16.40549	2.19228
	RWS	38	26.0416	11.32764	0.244
Treatment cycle	RCT	35	7.37143	2.587705	0.212
	RWS	18	7.31383	2.624251	0.939
Treatment duration/month	RCT	24	4.8875	2.47607	0.940
	RWS	12	5.7408	4.23496	0.449

RCT, randomized controlled trial; RWS, real-world study; SD, standard.

Discussion

Key findings

In this systematic review and meta-analysis, we found that there were slight systematic differences in patient characteristics between RCTs and RWS in CRC. The differences in baseline characteristics mainly included a higher proportion of male patients, a lower proportion of patients with ECOG score ≥2, and a lower proportion of second-line and above-second-line treatments in RCT. The reasons for these differences may be as follows: For gender, data on CRC patients collected from the Medicare database show that the proportion of men with CRC is generally higher than that of women, however, as the sample size increases, the difference will be narrowed, since the sample size of RWS is much larger than that of RCT, the proportion of male patients in RWS is closer to 50%. In addition, according to a study, men are more likely to participate in RCTs than women (215), which also leaded to a higher proportion of male patients in RCT than RWS. For ECOG score and treatment line, RCT has more strict inclusion and exclusion criteria for patients. Patients with high ECOG score and above-second-line treatments may be excluded due to poor health status and complex medical history. Therefore, the proportion of patients with ECOG score ≥2 and second-line and above-second-line treatments in RCT is lower.

Although there were slight differences in baseline characteristics, it did not lead to any difference in treatment outcomes by overall analysis, indicating that the results of RCT and RWS were highly consistent. As for the partial differences in subgroup analysis, a further meta-regression analysis showed that the higher OS value in the CET group of RCTs were due to the inclusion of more patients who are treated in frontlines, that can be reasonably interpreted as patients treated in frontlines were in better health. But no reason was found for the difference between ORR and DCR in XELOX group due to the small number of studies and the serious lack of clinical outcome data. We suggest conducting high-quality XELOX RWS for CRC patients in the future to supplement the deficiencies of the existing research.

Strengths and implications

This comparative study focused on cancer, the anticancer treatment process had relatively high standardization in drug regimens, drug compliance, and strict monitoring measures of toxicity and adverse reaction (216,217), which greatly reduced the differences in intervention measures and patients’ drug compliance and also lowered the bias of the results. Compared with several studies in the past, regression analysis was added in this study to determine the correlation between differences in baseline characteristics and differences in treatment effects, and rule out the effect of study design on the results. We believe that the differences between RCTs and RWS in different disease areas cannot be generalized. This study will be more applicable to clarify the external validity of RCTs results for CRC in real-world applications, help understanding the current status in CRC, improving research design and providing decision-making references for health decision-making departments.

Limitations

Given that this study mainly focused on the differences in patient characteristics between RCTs and RWS rather than the results of clinical trials, we did not perform quality assessment on the literature, the RWS across different countries may result in potential confounding factors. Since the OS value did not reach the upper limit in some studies, we used conservative estimation in the analysis to assume the OS values as the longest follow-up time in this study, which may lead to the underestimation of the OS values. Due to the limitations of study time, study number, and quality of the included studies, the conclusion herein need further verification.

Conclusions

No efficacy-effectiveness gap was found in CRC between RCTs and RWS. The treatment effects of RCTs and RWS in CRC patients were highly consistent, and the results of RCTs have high external validity.