Adjuvant chemotherapy in curatively resected rectal cancer: How valid are the data?

BACKGROUND: According to the result of the Cochrane review published in 2012, postoperative adjuvant chemotherapy (CTx) is associated with a survival benefit for rectal cancer patients operated for cure in comparison to patients who underwent only the surgical resection. AIM: To analyze the quality of the data supporting the advantage of adjuvant CTx after surgery for rectal cancer. In the times of increasing health care costs, it is imperative to offer the patient an evidence-based therapy that justifies potential side effects as well as costs.
METHODS: Overall survival was selected as endpoint of interest. Among the 21 included papers which analyzed this endpoint, we identified those three publications which have the highest weights to influence the final result. The validity of these papers was analyzed using the CONSORT checklist for randomized controlled trials. We performed a second meta-analysis excluding the three analyzed studies (n = 18) in order to assess their impact on the overall result of the original meta-analysis. Finally, we performed a third meta-analysis excluding all studies (n = 16) which showed a statistically improved overall survival.
RESULTS: The detailed analysis of the three most relevant RCTs according to the items of the CONSORT checklist showed several pitfalls. In up to 47% of the items, inappropriate answers were found. Generally, a lack of information regarding the randomization procedure as well as the absence of allocation concealment, blinded set-up, of intention-to-treat analysis and omission of sample size calculation were common problems of the analyzed studies. The exclusion of these three studies from the meta-analysis did not affect the general result of the meta-analysis, still confirming a survival advantage after adjuvant chemotherapy. After exclusion of single studies with a statistically significant outcome improvement, the meta-analysis of the remaining 16 studies again shows a statistically significant result due in part to a large remaining sample size.
CONCLUSION: The three most powerful publications show substantial deficits. We suggest a more critical appraisal regarding the validity of single studies because a meta-analysis cannot overcome the limitations of individual trials by pooling treatment effect estimates to generate a single best estimate. ©The Author(s) 2020. Published by Baishideng Publishing Group Inc. All rights reserved.

Core tip: The role of adjuvant chemotherapy (CTx) in curatively resected rectal cancer needs to be considered cautiously. Petersen et al published a Cochrane review in 2012 which found an improvement in survival in patients receiving adjuvant CTx after surgery in comparison to those who were treated by surgery only. The result was based on 21 studies, 5 of them supporting the advantage of CTx. Among these, we selected the three most powerful studies and assessed their validity, which was poor. Surprisingly, our meta-analysis without these studies yielded similar results as the original study (still in favor of adjuvant CTx).

INTRODUCTION

With the growing aging population, the prevalence of rectal cancer is significantly increasing[1]. Seven hundred and four thousand three hundred and seventy-six new cases of rectal cancer (3.9% of all sites) worldwide were registered in 2018 with 310394 deaths (3.2% of all deaths from all sites)[2]. Chemotherapy (CTx) after curative resection for non-metastatic rectal cancer is commonly used in the US, but this is not the case in Europe[3] with its role in improving patient survival remaining controversial[4-6], partly because many studies addressing this topic include also patients with colon cancer regardless of the biological differences of the clinical behavior of these two distinct diseases[7].

In 2012, Petersen et al[3] reported through a Cochrane review the impact of postoperative adjuvant CTx used for curatively resected rectal cancer (Tany, Nany, M0) on overall survival (OS) and disease-free survival (DFS)[3]. The authors identified 21 randomized controlled trials (RCT) reporting OS from a total of 9221 rectal cancer patients with 4854 of these patients which were randomized to adjuvant CTx (treatment arm) with the remaining 4367 patients not receiving adjuvant CTx representing the control arm. The meta-analysis of these studies highlighted a significant reduction in mortality risk (17%) among patients undergoing postoperative CTx as compared to those patients with simple follow-up observation [hazard ratio (HR) = 0.83; 95% confidence interval (CI): 0.76-0.91]. Twenty trials reported DFS with a total of 8530 patients examined. Again, the meta-analysis revealed a decrease in disease recurrence (25%) among patients undergoing adjuvant CTx when compared to the observation only group (HR = 0.75; 95%CI: 0.68-0.83)[3].

In the era of “choosing wisely” decisions, we deemed it necessary to reevaluate treatment recommendations for special tumor entities, in particular we aimed to assess the validity of studies on which meta-analysis rely and form the basis for these recommendations. As done for gastric[8] and esophageal cancer[9], the current study examined the validity of those studies within the meta-analysis of Petersen et al[3] (2012) which confirmed the benefit of post-operative CTx in rectal cancer. We do not aim to answer the clinical question about the use of adjuvant CTx after radical resection for rectal cancer, as this would imply a more extensive literature research than the critical analysis of a Cochrane review. The purpose is to critically evaluate both the results and the methodology by which the results were derived. It is imperative to offer the patient an evidence based therapy that justifies potential side effects as well as costs.

MATERIALS AND METHODS

The meta-analysis of Petersen et al[3] (2012) included a total of 21 studies with the endpoint of OS. Five studies (23.8%) (CCCSGJ[10] 1995, Krook et al[11] 1991, Quasar[12] 2007, Grage et al[13] 1981, Hamaguchi et al[14] 2011) found a statistically significant advantage in survival in curatively resected patients undergoing adjuvant CTx compared to those undergoing observation (HR < 1 with significant 95%CI because not including the 1 – this means that the graphic representation of the CI in the forest plot of the meta-analysis does not overcome the line of No Effect). The remaining 16 studies (76.2%) did not show statistically significant results. Table 1 lists the 21 studies with number of included patients in each arm as well as the weight of the study and information about statistical significance. Weight reflects the influence (in %) of each study within the overall meta-analysis, i.e., studies with high weight affect the results more than those with low weights with respect to the meta-analysis results. Weighting is determined by type of model, either fixed or random effect model, sample size (larger n = more weight), and precision of the estimate (narrower CI = more weight). The data provided in Table 1 is based on the description of the studies on pages 6-9 as well as on the tables from pages 24 to 37 of the original meta-analysis[3]. The total number of included patients was 9411. In the original publication the authors describe 9785 enrolled patients with rectal cancer and, in 9221 of them, data were available for meta-analysis.

Table 1

Sample size and significance of the 21 studies included in the Cochrane meta-analysis

Ref.	ntot	nc	ni (type of intervention)	Weight (%)	Outcome
Grage et al[13] 1981	64	31	33 (adj. CTx)	1.4	S
Fisher 1988 (NSABP)	574	191	190 (adj. RT)	6.8	NS
			193(adj. CTx+RT)
Thomas 1988 (GTSG)	106	58	48 (adj. CTx)	3.5	NS
Hafström 1990	99	56	43 (adj. CTx)	2.6	NS
Krook et al[11] 1991 (NCCTG)	204	100 (adj. RT)	104 (adj. CTx + RT)	6.8	S
Matsuda 1991 (SGACCS)	1243	598	645 (adj. CTx)	7.8	NS
CCCSGJ[10] 1995	1004	335	323 (adj. CTx)	7.6	S
			346 (intra-art CTx + adj. CTx )
Kornek 1996	57	29	28 (adj. CTx)	0.9	NS
Ito 1996 (TSGHCFU)	77	40	37 (adj. CTx)	1.6	NS
Yasutomi 1997 (JFMTC7-2)	713	356	357 (adj. CTx)	6.9	NS
Kodaira 1998 (JFMTC 7-1)	794	398	396 (adj. CTx)	7.4	NS
Taal 2001 (NACCP)	299	150	149 (adj. CTx)	4.4	NS
Kato 2002 (TACSG)	143	72	71 (adj. CTx)	1.7	NS
Cafiero 2003	218	108	110 (adj. CTx+RT)	4.0	NS
Watanabe 2004 (JFMTC15-2)	391	122	269 (adj. CTx)	3.3	NS
Glimelius 2005 (NGTATG)	691	352	339 (adj. CTX)	9.2	NS
Bosset et al[16] 2006 (EORTC)	1011	252 (neoadj. RT)	253 (neoadj. RT + adj. CTx)	8.9	NS
		253 (neoadj. CTx + RT)	253 (neoadj. CTx and RT + adj. CTx)
Sakamoto 2007 (JFMTC 15-1)	447	229	218 (adj. CTx)	5.2	NS
Quasar[12] 2007	948	474	474 (adj. CTx)	7.0	S
Koda 2009	54	29	25 (adj. CTx)	0.3	NS
Hamaguchi et al[14] 2011	274	135	139 (adj. CTx)	2.9	S
Total	94111	43681	50431	100.22	5S vs 16NS

Following the description of the studies at pages 6-9 as well as the tables from page 24 to 37 of the original meta-analysis we obtain this total number of included patients which is different from the 9785 reported in the meta-analysis. Consequently, also the total number of patients that receive surgery alone and surgery plus chemotherapy is different;

The sum of the assigned weight should be 100%. ntot: Number of patients with rectal cancer included in the study; nc: Number of patients included in the control group (surgery only); ni: Number of patients included in the intervention group (operation and adj. chemotherapy or other interventions); Outcome: Results of the study (S: Study result significant for better OS in the intervention group vs control group; NS: Not significant). CTx: Chemotherapy; RT: Radiotherapy.

In the first part of the results section, we assessed the validity of the three most powerful studies included in the Cochrane review by Petersen et al[3] (2012) which found a statistically significant advantage in survival in curatively resected patients with rectal cancer receiving adjuvant CTx compared to patients undergoing observation following surgery. These studies are those of CCCSGJ[10] 1995, Krook et al[11] 1991, Quasar[12] 2007. The assigned weights are 7.6%, 6.8% and 7.0%, respectively.

In the second part of the results section, we performed a second meta-analysis without these aforementioned three studies (n = 18, with a total of 7255 patients, 3459 in the control and 3796 intervention group, respectively), and finally we present the results of a third meta-analysis with all five statistically significant studies confirming the survival advantage for patients treated with postoperative adjuvant CTx excluded (n = 16, with a total of 6917 patients, 3293 in the control and 3624 in the intervention group, respectively). In this last case, only statistically non-significant studies were included in the meta-analysis.

Selection of the studies and assessment of their validity

As showed in Figure 1 and as previously described in detail in another publication[8], we selected three studies with the greatest power as weighted by the original authors which supported post-surgical CTx treatment among all included studies (n = 21) with endpoint overall survival: CCCSGJ[10] 1995, Krook et al[11] 1991, Quasar[12] 2007. The assigned weights were 7.6%, 6.8% and 7%, respectively. We then utilized the CONSORT checklist to assess the validity of these studies[15]. Two independent review authors (GM and FH) then examined the validity of these three publications.

Figure 1

Four steps to the analysis of validity of a systematic review. We identified the endpoint of interest (overall survival) and selected the three most powerful studies addressing this endpoint based on the assigned weights from the authors of the systematic review as these studies contributed essentially to the positive result of the systematic review. We finally assessed the validity of these studies by using the CONSORT checklist.

Meta-analysis

We performed a meta-analysis using R excluding the three analyzed studies discussed above (n = 18) and compared these results with those of the original meta-analysis comprising the entire 21 studies. Next, all single studies with a statistically significant benefit of post-operative CTx after curative resection of rectal cancer were removed and a third meta-analysis with the remaining 16 studies was performed. The meta-analysis were performed with R, version 3.2.0, with the package “meta” (http:// www.r-project.org/foundation).

RESULTS

Assessment of study validity

Table 1 provides an overview of the 21 studies of the original meta-analysis focusing on sample size, weight and statistical significance. As previously described in the methods section, this table is based on the data provided in the description of the studies on pages 6-9 as well as the tables from page 24 to 37 of the original meta-analysis[3]. According to these, the total number of included patients is 9411. In the original publication, the authors report 9785 enrolled patients with rectal cancer and in 9221 of them, data were available for meta-analysis. Table 2 presents a summary of the three analyzed papers described in the methods. Table 3 summarizes the items present in the CONSORT checklist[15] and how the studies address each evaluated component. The results are reported for each of the three included studies. In this section, we describe the issues identified through use of the CONSORT checklist evaluation.

Table 2

Summary of the three analyzed studies

	CCCSGJ[10] 1995	Krook et al[11] 1991	Quasar[12] 2007
Included patients	335 vs 323 vs 346 [3 trial arms for rectal carcinoma (IV, V, VI)]	104 vs 100	474 vs 474
Inclusion criteria	(1) Rectal carcinomas intraoperatively assessed as T3 or T4 and/or N1, N2, or N3 (before resection); (2) age ≤ 75 and no serious problems; (3) no cancer therapy in the past; (4) no other primary carcinomas; (5) satisfying laboratory tests at the beginning of treatment; and (6) consent to participation	(1) Potentially curative resection of histologically confirmed rectal adenocarcinoma; (2) T3 or T4; N1 or N2; (3) primary rectal carcinoma if extension of the carcinoma within 12 cm from of anal verge or inferior edge extended the sacral promontory; (4) anterior resection: entering in the study no later than 56 d postoperative; Abdominal perineal resection: 70 d; (5) met laboratory value requirements; (6) no prior radiation to the pelvis or CTx; and (7) no other malignancies within the last 5 years apart from superficial skin cancer and CIS of the cervix	(1) Presumably complete resection of colon or rectal cancer; no evident distant metastases; (2) no contraindications to CTx; (3) no prior CTx apart from one-week portal-vein infusion of fluorouracil after surgery; and (4) written consent before randomisation
Intervention group	(1) Trial arm IV: Intraoperative + postoperative mitomycin C iv; postoperative 5-FU po; and (2) Trial arm V: Postoperative mytomycin C iv + 5-FU po	Postoperative CTx with fluorouracil and semustine + radiation	CTx with fluorouracil and folinic acid after apparently curative (until October 1997 levamisole or placebo was added)
Control group	Trial arm VI: Surgery alone	Postoperative radiation alone	Surgery alone
Outcome (intervention vs Control): HR (95%CI)	HR (IV + V compared to VI) shown in the Cochrane meta-analysis: 0.66 [0.52-0.84]	HR shown in the Cochrane meta-analysis: 0.71 (0.55-0.92)	HR shown in the Cochrane meta-analysis: 0.77 (0.60-0.99)
Weight assigned in the Cochrane review (%)	7.6	6.8	7.0

5-FU: 5-fluorouracil; HR: Hazard Ratio; CI: Confidence interval; CIS: Carcinoma in situ; CTx: Chemotherapy.

Table 3

Assessment of validity of the analyzed studies according to the CONSORT checklist

Section/topic	Item number	CCCSGJ[10] 1995	Krook et al[11] 1991	Quasar[12] 2007
Title and abstract	1a	Yes	No	Yes
	1b	No	Yes	Yes
Introduction
Background and objectives	2a	Yes	Yes	Yes
	2b	Yes	Yes	Yes
Methods
Trial design	3a	No	Yes	Yes
	3b	NA	NA	NA
Participants	4a	Yes	Yes	Yes
	4b	No	No	No
Interventions	5	Yes	Yes	Yes
Outcomes	6a	No	No	Yes
	6b	NA	NA	NA
Sample size	7a	No	No	Yes
	7b	NA	NA	No
Randomisation	8a	Yes	No	Yes
	8b	No	No	Yes
	9	No	No	Yes
	10	No	No	Yes
Blinding	11a	NA	NA	NA
	11b	No	No	No
Statistical methods	12a	Yes	Yes	Yes
	12b	No	Yes	Yes
Results
Participant flow	13a	Yes	No	Yes
	13b	Yes	Yes	Yes
Recruitment	14a	Yes	Yes	Yes
	14b	NA	NA	NA
Baseline data	15	Yes	Yes	Yes
Numbers analysed	16	Yes	Yes	Yes
Outcomes and estimaton	17a	No	Yes	Yes
	17b	No	No	Yes
Ancillary analysis	18	Yes	Yes	Yes
Harms	19	Yes	Yes	Yes
Discussion
Limitations	20	Yes	Yes	No
Generalisability	21	No	No	No
Interpretation	22	Yes	Yes	Yes
Other information
Registration	23	No	No	Yes
Protocol	24	No	No	No
Funding	25	No	No	Yes

NA: Not applicable.

Regarding the CCCSGJ[10] study (1995), validity criteria were not met in 14 of 32 items (43.75%) while five were not applicable. 1004 patients from 140 centers over 2 years were randomized to one of the three arms with 98 assessed as non-eligible and not further analyzed. This causes a loss of the balance in the three groups used within the randomization process. In the author’s power calculation, 310 patients were needed in each of the three groups. Included in the final study were 316, 297 and 293, respectively. As the randomization procedure is not described in detail beyond mention of use of the envelope method, it is not possible to know if the allocation concealment was maintained or not. The absence of blinding limits the possibility to correctly interpret the results of the study because difference between control and intervention group may be caused by placebo effect.

In the study of Krook et al[11] (1991), we identified poor validity in 15 of the 32 items of the CONSORT checklist (47%). Five items were not applicable. Specifically, the control group was different compared to the standard control group used in the other studies included in the Cochrane meta-analysis (i.e., surgery alone) with exception of the study by Bosset et al[16] 2006. In the study of Krook et al[11], two groups were compared: Surgery plus adjuvant radiotherapy (n = 100) vs surgery plus adjuvant radiochemotherapy (n = 104), whereas in the study of Bosset et al[16], 505 patients that received preoperative radiotherapy or radiochemotherapy were compared with 506 patients receiving preoperative radiotherapy and postoperative CTx or preoperative radiochemotherapy and postoperative CTx. In the study of Krook et al[11] patients included were stratified by operation, extent of invasion, nodal involvement and time to study entry and then randomly assigned to the control or intervention group. It is thus not clear if this design reflects a randomization by strata. Moreover, information about the randomization process is absent as is a power calculation. It is not possible to understand if sample size is high enough, as this should be calculated based on the primary endpoint, which is not clearly defined. Several endpoints are listed (time to local recurrence or metastasis, local recurrence rates and metastasis, and survival). Not all randomized patients were analyzed (209 patients were enrolled in the study, only 205 analyzed). Likewise, no discussion of use of neoadjuvant treatment, if utilized, is described.

In the Quasar study[12] (2007), validity criteria were not met in 6 of 33 items (18.2%). Four items were not applicable. A pragmatic design for trial organization was adopted, with clinical teams dividing patients as having either a clear or an unclear indication for adjuvant CTx. This means that the indication for CTx was decided by each clinician after consultation with the patient. This is a source of bias. Moreover, use of the minimization method does not allow allocation concealment to be maintained. This study is a pragmatic controlled trial rather than a randomized controlled trial. Blinding could not be possible in these studies, as the control group failed to receive any type of treatment. Additionally, patients with colon and rectal cancer were analyzed together.

In the study by Krook et al[11] only patients with high risk rectal carcinoma were included. This was defined as the histological presence of an indicator associated with poor prognosis [e.g., perirectal fat invasion (T3), adjacent organ involvement (T4), or regional lymph node metastasis (N1 or N2)]. In the Quasar trial[12], patients with low risk of recurrence were included. These two studies analyzed different subgroups of patients and the results cannot therefore be compared.

Figure 2 shows meta-analysis results when the three individually analyzed studies were removed, leaving a total of 18 studies included. Two studies (Grage et al[13] 1981, Hamaguchi et al[14] 2011) showed statistically significant result in favor of post-surgical CTx following curative resection of rectal cancer. The other sixteen included studies were not statistically significant. The modified meta-analysis estimate had an HR of 0.87 with a 95%CI: 0.79-0.96. The original meta-analysis showed an HR of 0.83 with 95%CI: 0.76-0.91. Removal of the three studies did not significantly change the result of the original meta-analysis.

Figure 2

Meta-analysis of n = 18 studies after the exclusion of the three analyzed studies. Each study included in the meta-analysis is represented on the forest plot by a box [put either in the region of favors (T) or favors (C) of the graphic according to the hazard ratio] and a line which indicates the 95% confidence interval. The pooled hazard ratio of the meta-analysis is represented by a diamond. In this case the estimate of overall effect is in the region which favors the treatment [favors (T)]. HR: Hazard ratio; CI: Confidence interval; W(fixed): Weight assigned to the study by using a fixed effect model; W(random): Weight assigned to the study by using a random effect model; Favors (T): Result in favor of the treatment (intervention) group (i.e., adjuvant chemotherapy after curative surgery); Favors(C): Result in favor of the control group (no adjuvant chemotherapy after surgery).

A second meta-analysis was conducted (Figure 3) without the two studies[13,14] which showed a positive, statistically significant result. Together, the exclusion of the five studies that showed a statistically significant results by themselves still resulted in a statistically significant result (HR = 0.90, 95%CI: 0.83-0.97) with improved survival in patients receiving adjuvant CTx after resection of rectal cancer compared to patients with surgery alone, when a new meta analysis was performed.

Figure 3

Meta-analysis of n = 16 studies after the exclusion of all studies which found a statistically significant survival advantage in the experimental group. Each study included in the meta-analysis is represented on the forest plot by a box [put either in the region of favors (T) or favors (C) of the graphic according to the hazard ratio] and a line which indicates the 95% confidence interval. The pooled hazard ratio of the meta-analysis is represented by a diamond. In this case the estimate of overall effect is in the region which favors the treatment [favors (T)]. HR: Hazard ratio; CI: Confidence interval; W(fixed): Weight assigned to the study by using a fixed effect model; W(random): Weight assigned to the study by using a random effect model; Favors (T): Result in favor of the treatment (intervention) group (i.e., adjuvant chemotherapy after curative surgery); Favors(C): Result in favor of the control group (no adjuvant chemotherapy after surgery).

DISCUSSION

The present study aimed to assess the validity of the three most powerful studies (CCCSGJ[10], Krook et al[11], Quasar[12]) included in the meta-analysis by Petersen et al[3] 2012 supporting the survival advantage of adjuvant CTx which lends support to the results of enhanced survival with post-operative adjuvant CTx after surgical resection for rectal cancer. Rather than to answer the clinical question about the use of adjuvant CTx after radical resection for rectal cancer, as this would imply a more extensive literature research than the solely critical analysis of a Cochrane review, this work focus on the critical evaluation of the methodology used in the analyzed RCTs to achieve results, as bias endangers the validity of studies and needs to be evaluated. To this end, three studies which contributed the greatest to the findings of the Petersen meta-analysis were evaluated for validity using the standardized CONSORT checklist. We demonstrated that these three studies lack validity. Firstly, we would like to summarize the main and the specific problems found in the three critically revised studies. The main common problems among the three studies were the randomization procedure, the lack of a power calculation or when done this was not respected. Additionally, the absence of blinded, placebo-controlled study design limits the soundness of three of the studies and consequently the overall conclusions of the review. In the absence of placebo control, one cannot differentiate between specific pharmacological and placebo effects. Placebo effect is defined as the “response of a subject to a substance or any procedure known to be without specific therapeutic effect for the condition being treated”[17]. Patients assigned to the control group often experience disappointment when they expect to be treated. Furthermore, lack of concealment of treatment allocation prevents the randomization process leading to conscious or subconscious bias[18].

In the meta-analysis of Petersen et al[3] inclusion criteria for the studies were RCTs comparing patients receiving radical surgery for non-metastatic rectal cancer (Tany, Nany, M0) who did not receive adjuvant CTx with those receiving postoperative CTx regimen of any kind[3]. This has as consequence that the included studies are heterogeneous regarding inclusion criteria, making them difficult to compare, as different subgroups of patients were analyzed. In the study by Krook et al[11], only patients with high risk rectal carcinoma were included. This was defined as the presence in the histology of an indicator of disease progression or expansion outside the original location [invasion of perirectal fat (T3), extension to adjacent organs (T4), or metastasis to regional lymph nodes (N1 or N2)]. In the Quasar trial[12], patients with low risk of recurrence were included. Moreover, in the study of Krook et al[11], a control group treated only with surgery followed by observation does not exist. In this trial, two intervention groups are present (surgery plus radiotherapy vs surgery plus radiochemotherapy). Due to these issues, the studies by Krook et al[11] and Bosset et al[16] should not be included in the Cochrane review, as their control group differs from the control group (surgery alone) of the other 19 studies.

Regarding the single studies, some problems should be mentioned. In the CCCSGJ study[10], patients were first randomized and then assessed for eligibility, which is a methodological error because, in this way, the balance obtained through randomization is lost. Ninety-eight randomized patients were excluded because they did not meet the inclusion criteria. This prevents an intention to treat analysis from being performed.

In the study by Krook et al[11], the primary endpoint on which sample size should be calculated, is not clearly defined. Although the results seem to be clinically relevant, they probably lack sufficient power to provide usable conclusions.

The Quasar study is a pragmatic controlled trial and not a RCT. This contrasts with the declared inclusion criteria by the author of the Cochrane meta-analysis (only RCT). Additionally, in this study, a minimization method is used. Minimization[19-22] refers to a type of dynamic allocation, where the subject’s treatment regimen is dictated by the evaluation of the potential imbalance of covariates that would result if the patient were assigned to the treatment or the control group[23]. This design seeks to balance patient numbers over a large group of pre-specified prognostic factors at once. Minimization determines the group a prospective subject would be assigned and consequently the allocation concealment is impossible[24]. The European Medicines Agency’s Committee[25] states that “dynamic allocation is strongly discouraged”.

Regarding the two meta-analysis performed after exclusion of the three analyzed studies, as well as all five studies showing a survival benefit of adjuvant CTx, both confirmed the advantage in survival in patients receiving adjuvant CTx after curative surgery. In the last case, none of the single included studies (n = 16) could demonstrate this benefit. This is similar to previous evaluations by our group where postsurgical CTx improved survival in patients with gastric cancer[8]. Similar meta-analyses where overall significance exceeded individual study results are present in the literature[26] and are critically examined[27]. Meta-analyses provide for larger sample sizes by combining many individual studies, thereby enhancing the power to detect differences[28,29]. The findings here and in other studies again demonstrates the critical need of including specific studies which meet high level criteria for inclusion. On the other hand, meta-analysis will not always compensate for the limitations of individual studies. Bias can still exist within the single studies included despite sound meta-analysis techniques. Meta-analysis increase precision, but accuracy of overall findings hinges upon the individual studies included with their potential faults (e.g. bias), potentially significantly affecting the overall outcome[30]. Study inclusion criteria is only one of several questions that can seriously affect quantitative results and qualitative conclusions derived from the analysis: Measuring publication bias, quantifying risk of bias for specific domains, appropriate statistical techniques for pooled data, and the use of unpublished literature are further important themes[30]. In particular, as the inclusion criteria for trial selection in the Petersen meta-analysis were so wide (Tany, Nany, M0), it is difficult to evaluate the clinical relevance. The actual German S3 guideline for treatment of rectal cancer[31] generally recommends neoadjuvant radiochemotherapy or short radiotherapy for UICC stage II and III middle and low rectal cancer and for T4 tumors with contact to the mesorectal fascia and near the sphincter. T1/2 tumors and limited or uncertain lymph node metastasis and T3a/b tumors found in the middle rectum and small perirectal infiltration assessed by standard MRI (T3a < 1 mm; T3b 1-5 mm) without involvement of surrounding lymph nodes or extramural vessel invasion can be treated by primary surgery as recently reported[31]. Rectal carcinomas found in the upper third are generally resected without any secondary treatment. Neoadjuvant therapy, in those instances, should be utilized only in risk situations (T4, positive circumferential resection margin, clearly lymph node +)[31]. Depending on the final pathological TNM stage after surgery, adjuvant therapy should be determined by tumor boards. Cases involving T1/2 N0 rectal cancer do not warrant no adjuvant therapy based on these results. An adjuvant radiochemotherapy is indicated in presence of risk factors for recurrence (for example R1-resection, intraoperative tumor rupture, poor quality of total mesorectal excision, pT4, pT3c/d, pN2, pT3 in lower rectum)[31].

A limitation of our study is that the focus was put on the statistical and methodological aspect considering three studies of a Cochrane review rather than to answer the clinical question regarding the survival benefit of adjuvant CTx for rectal cancer operated for cure. The critical analysis of larger amount of studies and meta-analysis is mandatory in order to assess the clinical relevance of adjuvant CTx.

However, our purpose was to inform the reader of the importance of critical interpretation of the results of RCTs and meta-analysis as well as the selection process of the studies to be included in such reports, as this cannot overcome bias present in the single included studies. Most important of all, the safety of the patient with a clear benefit of the suggested treatment should remain the main goal of medical decision making.

Implications for practice

Following the results of the Cochrane review, post-surgical CTx was shown to improve survival in patients after curative resection of rectal cancer. Importantly, it is noted that a portion of the reviewed studies contain limitations which impair a definitive assessment and recommendation development and that final guidance on this topic should be delayed until additional properly designed studies are conducted. The three analyzed studies which were of highest weight in the Cochrane review had insufficient validity, based on key study features, to be included in a meta-analysis. Systematic review and/or meta-analysis quality is directly dependent on the quality of the studies included, as studies with bias risk can consequentially deteriorate the validity of the entire analysis[29]. As results of meta-analysis are often taken into big consideration in the medical community, the literature needs to be reevaluated in order to avoid unnecessary side effects for the patients as well as unnecessary costs for the health care systems.

ARTICLE HIGHLIGHTS

Research background

The use of chemotherapy (CTx) after curative surgery for non-metastatic rectal cancer and its role in improving patient survival remains controversial.

Research motivation

In 2012, Petersen et al[3] reported in a Cochrane review the effect of postoperative adjuvant CTx following curatively resected rectal cancer (Tany, Nany, M0) on overall survival. The authors identified 21 randomized controlled trials (RCT) reporting overall survival as primary endpoint. The meta-analysis of these RCTs showed a significant reduction in the risk of death (17%) among patients undergoing postoperative CTx as compared to those undergoing observation (hazard ratio = 0.83; 95% confidence interval: 0.76-0.91).

Research objectives

We aimed to analyze the quality of the data supporting the advantage of adjuvant CTx after surgery for rectal cancer.

Research methods

Using the CONSORT Checklist, the current analysis evaluated the validity of the three most powerful studies reviewed and analyzed within the Cochrane review by Petersen et al[3] 2012 which support the survival benefit of adjuvant CTx.

Research results

The detailed analysis of the three most powerful studies highlighted inconsistencies including inappropriate answers in up to 47% of the items of the CONSORT checklist. Inadequate or unclear randomization without allocation concealment, missing blinded set-up, absence of intention-to-treat analysis and omission of sample size calculation were the most common findings.

Research conclusions

We suggest a more critical appraisal regarding the validity of single RCTs, as these studies are included in meta-analysis that are the basis for guidelines.

Research perspectives

As CTx has several side effects for the patient and generates costs for the health system, it should be used only if its benefit is real.