Prevalence and Impact of Treatment Crossover in Cardiac Surgery Randomized Trials: A Meta-Epidemiologic Study.

Background Crossover dilutes treatment effect and reduces statistical power of intention-to-treat analysis. We examined incidence and impact on cardiac surgery randomized controlled trial (RCT) outcomes of crossover from experimental to control interventions, or vice versa. Methods and Results MEDLINE, EMBASE, and Cochrane Library were searched, and RCTs (≥100 patients) comparing ≥2 adult cardiac surgical interventions were included. Crossover from the initial treatment assignment and relative risks (RRs) for each trial's primary end point and mortality at longest available follow-up were extracted. All RRs were calculated as >1 favored control group and <1 favored experimental arm. Primary outcome was the effect estimate for primary end point of each RCT, and secondary outcome was all-cause mortality; both were appraised as RR at the longest follow-up available. Sixty articles reporting on 47 RCTs (25 440 patients) were identified. Median crossover rate from experimental to control group was 7.0% (first quartile, 2.0%; third quartile, 9.7%), whereas from control to experimental group, the rate was 1.3% (first quartile, 0%; third quartile, 3.6%). RRs for primary end point and mortality were higher in RCTs with higher crossover rate from experimental to control group (RR, 1.01 [95% CI, 0.94-1.07] versus RR, 0.80 [95% CI, 0.66-0.97] and RR, 1.02 [95% CI, 0.95-1.11] versus RR, 0.94 [95% CI, 0.82-1.07], respectively). Crossover from control to experimental group did not alter effect estimates for primary end point or mortality (RR, 0.82 [95% CI, 0.63-1.05] versus RR, 0.95 [95% CI, 0.86-1.04] and RR, 0.88 [95% CI, 0.73-1.07] versus RR, 1.02 [95% CI, 0.95-1.09], respectively). Conclusions Crossover from experimental to control group is associated with outcomes of cardiac surgery RCTs. Crossover should be minimized at designing stage and carefully appraised after study completion.

Clinical Perspective

What Is New?

Crossover from experimental to control group is associated with outcomes of cardiac surgery randomized controlled trials.

What Are the Clinical Implications?

Crossover should be minimized at designing stage and carefully appraised after study completion.

Introduction

Randomized controlled trials (RCTs) are considered the gold standard to compare different treatments.1, 2 If duly adhered to, the process of randomization enables the homogeneous distribution of known and unknown risk factors and confounders between treatment arms, thereby allowing for minimization of the bias around the point estimate for a given treatment effect.2 However, within RCTs, the crossover of subjects from one group to the other dilutes the treatment effect and, even if occurring randomly, reduces the statistical power of the intention‐to‐treat analysis.3, 4 With high rates of crossover, and even more so when crossover is not random but caused by specific reasons related to the intervention, such as lower confidence or expertise with the experimental therapy, the internal validity of RCTs may be jeopardized and biased effect estimates may ensue, misinforming researchers, clinicians, and patients.5, 6 Crossover is particularly worrisome when occurring prevalently from one group to the other (1‐way crossover). When the prevalent direction is from the experimental to the control arm, crossover may drive the effect estimates toward the null hypothesis, even when the experimental treatment is superior to the control intervention.7 On the other hand, crossover from control to experimental group may lead to underestimation of the adverse effects associated with a potentially more hazardous experimental treatment.

Surgical RCTs are particularly at risk of crossover issues because of the complex, intricate, and operator‐dependent nature of the intervention, the likelihood of a learning curve, the need for specific technical expertise, and the lapse between randomization and intervention.8, 9, 10 For instance, randomization to a novel technique in cardiovascular surgery is typically completed before the operation, whereas the actual feasibility of the procedure can only be fully appraised intraoperatively. Moreover, in surgical RCTs, crossover not only dilutes the potential treatment effect, but may also be a surrogate for lack of confidence or expertise with the novel treatment.10 This can potentially influence the study results, even beyond the purely statistical effect of crossover.

Given the lack of dedicated investigations on the prevalence and impact of crossover in cardiac surgery, where treatment applicability considerations are arguably among the highest in medicine, the importance and frequency of these interventions for millions of patients worldwide, and the informative, prototypical role of cardiac surgery treatments on medical practice at large, we aimed to systematically evaluate the relationship between crossover rate and outcomes in all RCTs that compared cardiac surgical interventions, by using a meta‐epidemiologic approach.11 Specifically, our main hypothesis was that crossover may unfavorably impact effect estimates, with a high crossover rate diluting clinical effects and, thus, undermining study validity and conclusions.

Methods

The data that support the findings of this study are available from the corresponding author on reasonable request.

Design

This meta‐epidemiologic study was conducted in keeping with current reviewing practice recommendations,11 and it is reported according to the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses checklist (Figure S1).12 The analysis was preregistered on the PROSPERO international database of prospectively registered systematic reviews (identifier: CRD42019123022). There was no individual patient involvement in this study; as such, research ethics board approval was not required.

Search Strategy

A comprehensive search to identify RCTs that compared ≥2 adult cardiac surgical interventions was performed in the following databases: Ovid MEDLINE (all; 1990 to February 2019); Ovid EMBASE (1990 to February 2019); and The Cochrane Library (Wiley). The search strategy included the following terms: “CABG,” “off‐pump coronary surgery,” “on‐pump coronary surgery,” “arterial grafts,” “bilateral internal mammary artery,” “bilateral internal thoracic artery,” “radial artery,” “right gastroepiploic artery,” “arterial revascularization,” “saphenous vein graft,” “mitral valve repair,” “mitral valve replacement,” “tricuspid valve,” “aortic valve replacement,” “aortic valve repair,” “heart transplant,” and “ventricular assist device” (full search strategy available in Table S1). Studies performed on patients with congenital cardiac abnormalities were not included in the search.

Study Selection

Searches across the chosen databases retrieved a total of 10 619 results. After deduplication and removal of nonpertinent articles, 2 independent reviewers (A.D.F., M.D.M.) screened a total of 152 articles. Titles and abstracts were reviewed against predefined inclusion/exclusion criteria. A third independent reviewer (G.F.) confirmed adequacy of studies on the basis of predefined inclusion criteria. Articles were considered for inclusion if they were RCTs with a planned sample size ≥100 patients, published between 1990 and February 2019, comparing ≥2 cardiac surgical interventions in adult patients, and written in English. Studies that compared different devices (not different surgical techniques), surgical versus nonsurgical interventions, and pharmacological or percutaneous treatments were not included. Studies using a noninferiority or equivalence design and for which the primary outcomes or crossover rate was not specified were also excluded. In case of several publications with overlapping study populations, the report that detailed the largest sample size with the longest follow‐up available was selected. In case of multiple publications from the same RCT providing different outcomes, all publications were included. Multiple outcomes from the same trial were included when the sample size was powered for all of them.

The full text of preliminary screened studies was retrieved for a second round of eligibility screening. Disagreements were resolved by the first author (M.G.). Reference lists of the included articles were also searched, and additional studies were included (ie, backward snowballing). The Preferred Reporting Items for Systematic Reviews and Meta‐Analyses flow diagram is shown as Figure S1.

Data Extraction

Two investigators (A.D.F., M.D.M.) performed data extraction independently, and data were later verified by a third investigator (M.G.) for accuracy. The following variables were included: number of studies, number of study reports, total number of comparisons, number of studies with co–primary end points, year of publication, number of multicenter studies, number of centers in multicenter studies, number of off‐pump coronary artery bypass graft (CABG) trials, statistical power (%), expected relative risk (RR) reduction (%), sample size, follow‐up (months), internal validity, according to Cochrane Risk of Bias Tool, crossover from experimental to control group (%), crossover from control to experimental group (%), total crossover (%), relative crossover (%), main findings, RR for primary end point, and RR for mortality. We also generated a relative crossover estimate (defined as crossover from experimental to control group divided by crossover from control to experimental group) and a total crossover estimate (defined as the sum of crossover from experimental to control group plus crossover from control to experimental group).

Assessment of the Quality of Individual Studies and the Overall Quality of Evidence

The quality of the included studies was assessed by using the Cochrane Risk of Bias Tool for assessing the internal validity of RCTs.13

Outcomes

The primary outcome was the effect estimate for the primary end point of each included RCT, and the secondary outcome was all‐cause mortality, both appraised as RR at the longest follow‐up available. Whenever possible, RR was recalculated for each study and comparison. When only hazard ratios were reported and RR calculation was unfeasible, hazard ratios were considered as adequate estimators of RR.

Statistical Analysis

Regardless of the way the individual trial was initially reported, for this review, all RRs were calculated as >1 favored the control group and <1 favored the experimental arm.

Continuous variables are reported as median (first quartile; third quartile), and categorical variables are reported as count (%). Exploratory inferential analysis was based on χ2 or Wilcoxon rank‐sum tests. The main inferential analysis was based on random effects pairwise meta‐analysis and on meta‐regression, using the logarithm of RR (LogRR) as dependent variable, with graphical synthesis by means of forest and L'Abbè plots. Inconsistency was appraised as I2, τ2, and χ2 tests, with between‐subgroup heterogeneity tested with an inverse‐variance‐weighting model. Adjusted R 2 was also used to describe predictive ability. A sensitivity analysis using absolute LogRR (ie, |LogRR|) was also conducted, as it may capture extreme effects better when treatments undergoing comparison may have been defined experimental versus standard inappropriately. In addition, we also conducted an exploratory analysis to identify features independently associated with high crossover rate. Publication bias and small study effects were explored by visual inspection of funnel plots and computing Egger and Begg tests. Effects are reported as point estimates (95% CIs). Computations were performed with Stata 13 (StataCorp, College Station, TX). Studies having a continuous variable as the primary outcome measure were not included in the main analysis.

Results

After searches and selection, 60 reports (accounting for 67 comparisons) from 47 RCTs were identified (25 440 total patients) (Table 1; references of the included RCTs are provided in the supplemental material). Most RCTs focused on off‐pump versus on‐pump CABG; 5 focused on radial versus saphenous vein grafting; 2 focused on repair versus replacement of the mitral valve; 2 focused on total arterial revascularization versus left internal thoracic artery plus saphenous vein grafting; and the remaining RCTs focused on CABG, mitral valve disease, and aortic valve disease (Table S2). Most studies (n=27 [57.4%]) were limited to 1 or 2 centers. The a priori median expected RR reduction was 29%, the median sample size was 251 subjects, and the median follow‐up was 12 months. The internal validity of included studies was variable, with few studies at low risk of bias in all appraised dimensions, and several at high risk (Table S3). Crossover from the experimental to control group occurred in 7.0% (first quartile, 2.0%; third quartile, 9.7%) of patients, whereas crossover from the control to experimental group took place in 1.3% (first quartile, 0%; third quartile, 3.6%) of patients. Inferential estimates of the primary end point favored the experimental treatment in 13 RCTs and the control group in 4, and were not significantly different in 50 cases. The median RR for primary end point was 0.95 (95% CI, 0.65–1.12), and the median RR for mortality was 0.99 (95% CI, 0.80–1.28).

Table 1

Main Features of Included Studies

Feature	Value, Count (%) or Median (First Quartile; Third Quartile)
Studies	47
Study reports	60
Total no. of comparisons	67
Studies with co–primary end points	8 (17.8)
Year of publication	2009 (2004; 2012)
Multicenter studies	20 (42.6)
No. of centers in multicenter studies	15 (7; 24)
Treatment features
Coronary artery bypass graft trials	42 (89.4)
Off‐pump coronary artery bypass graft trials	27 (57.4)
Valve trials	9 (19.2)
Experimental group more invasive than control group	14 (29.8)
End point features
Mortality end point	24 (51.1)
Clinical end point	22 (46.8)
Patency end point	13 (27.7)
Statistical power, %	80 (80; 90)
Expected relative risk reduction, %	29 (18; 37)
Sample size	251 (128; 401)
Follow‐up, mo	12 (1; 36)
Internal validity according to Cochrane Risk of Bias Tool
High	8 (11.9)
Moderate or uncertain	48 (71.6)
Low	11 (16.4)
Crossover from experimental to control group, %	7.0 (2.0; 9.7)
Crossover from control to experimental group, %	1.3 (0; 3.6)
Total crossover, %	7.8 (3.0; 10.7)
Relative crossover, %	3.6 (1.0; 19.0)
Main findings
Favors experimental group	13 (19.4)
Favors control group	4 (6.0)
Nonsignificant	50 (74.6)
Relative risk for primary end point	0.95 (0.65; 1.12)
Relative risk for mortality	0.99 (0.80; 1.28)

A meta‐epidemiologic analysis that explores the study features associated with higher versus lower than median crossover rate, from experimental to control group, is provided in Table 2. Off‐pump CABG trials showed a higher prevalence of crossover from experimental to control group, as did studies on valve surgery, allocating less invasive treatments to the experimental arm and focusing on a mortality or clinical end point, low, moderate, or uncertain internal validity, according to the Cochrane Risk of Bias Tool assessment, and nonsignificant inferential estimates for the trial's primary end point (all P<0.05 at χ2 test). Crossover from control to experimental group was significantly associated with multicenter setting, focus on a clinical end point, and large sample size (all P<0.05 at χ2 test). Accordingly, the RR for primary end point was higher in RCTs with high crossover rate from experimental to control group (P=0.040 by meta‐regression analysis) (Table 3) (Figure 1),14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49 with similar findings pertaining to the RR for mortality (P=0.015 by meta‐regression analysis) (Figure S2). However, no significant associations were found between crossover from control to experimental group and study findings (P=0.065 at χ2 test), nor RR for primary end point (P=0.726 at meta‐regression analysis) (Figure 2)14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49 or RR for mortality (P=0.871 at meta‐regression analysis) (Figure S3).

Table 2

Inferential Analysis Comparing Higher vs Lower Than Median Crossover Rate

Variable	Value, Count (%) or Median (First Quartile; Third Quartile)
	High Crossover Rate*	Low Crossover Rate†	P Value
From experimental to control group
Year of publication	2010 (2004; 2014)	2010 (2008; 2012)	0.796
Multicenter studies	20 (58.8)	14 (42.2)	0.179
Treatment features
Coronary artery bypass graft trials	31 (91.2)	30 (90.9)	0.969
Off‐pump coronary artery bypass graft trials	25 (73.5)	14 (42.4)	0.010
Valve trials	3 (8.8)	9 (27.3)	0.049
Experimental group more invasive than control group	7 (20.6)	15 (45.5)	0.030
End point features
Mortality end point	26 (76.5)	12 (36.4)	0.001
Clinical end point	22 (64.7)	11 (33.3)	0.010
Patency end point	3 (8.8)	13 (39.4)	0.003
Statistical power, %	90 (80; 90)	80 (80; 90)	0.290
Expected relative risk reduction, %	32 (28; 33)	18 (15; 40)	0.227
Sample size	290 (176; 2394)	260 (150; 339)	0.123
Follow‐up, mo	12 (1; 24)	15 (1; 35)	0.228
Crossover from control to experimental group, %	1.9 (0; 3.6)	0.9 (0; 3.7)	0.722
Total crossover, %	10.0 (8.7; 13.4)	3.0 (1.0; 5.4)	<0.001
Relative crossover	5.6 (2.3; 85.0)	1.0 (0.3; 10.0)	<0.001
Internal validity according to Cochrane Risk of Bias Tool
High	1 (2.9)	7 (21.2)	0.003
Moderate or uncertain	23 (67.7)	25 (75.8)
Low	10 (29.4)	1 (3.0)
Main findings for primary end point
Favors experimental group	2 (5.9)	11 (33.3)	0.016
Favors control group	2 (5.9)	2 (6.1)
Nonsignificant	30 (88.2)	20 (60.6)
Relative risk for primary end point	1.00 (0.91; 1.22)	0.77 (0.52; 1.00)	0.054
Relative risk for mortality	1.02 (0.93; 1.30)	0.88 (0.61; 1.03)	0.022
From control to experimental group
Year of publication	2011 (2009; 2014)	2008 (2004; 2011)	0.055
Multicenter studies	24 (70.6)	10 (30.3)	0.001
Treatment features
Coronary artery bypass graft trials	33 (97.1)	28 (84.9)	0.080
Off‐pump coronary artery bypass graft trials	19 (55.9)	20 (60.6)	0.695
Valve trials	4 (11.8)	8 (24.2)	0.183
Experimental group more invasive than control group	13 (38.2)	9 (27.3)	0.339
End point features
Mortality end point	23 (67.7)	15 (45.5)	0.067
Clinical end point	21 (61.8)	12 (36.4)	0.038
Patency end point	7 (20.6)	9 (27.3)	0.521
Statistical power, %	82 (80; 90)	80 (80; 90)	0.721
Expected relative risk reduction, %	28 (20; 33)	30 (14; 50)	0.926
Sample size	475 (301; 2394)	176 (120; 251)	<0.001
Follow‐up, mo	12 (1; 60)	12 (1; 24)	0.275
Internal validity according to Cochrane Risk of Bias Tool
High	4 (11.8)	4 (12.1)	0.642
Moderate or uncertain	23 (67.7)	25 (75.8)
Low	7 (20.6)	4 (12.1)
Crossover from experimental to control group, %	7.0 (3.9; 12.4)	7.4 (1.0; 9.0)	0.269
Total crossover, %	10.0 (8.7; 13.4)	3.0 (1.0; 5.4)	0.074
Relative crossover	1.9 (1.0; 3.4)	16.6 (5.3; 85.0)	<0.001
Main findings for primary end point
Favors experimental group	3 (8.8)	1 (3.0)	0.065
Favors control group	3 (8.8)	10 (30.3)
Nonsignificant	28 (82.4)	22 (66.7)
Relative risk for primary end point	0.98 (0.83; 1.14)	0.78 (0.52; 1.08)	0.468
Relative risk for mortality	1.01 (0.91; 1.28)	0.92 (0.70; 1.29)	0.174

Higher or equal to the median.

Lower than the median.

Table 3

Meta‐Regression Analysis Exploring Potential Moderators of Effect Sizes

Variable	Coefficient (95% CI)	P Value	τ2	Adjusted R 2
Logarithm of the relative risk of the primary end point
Year of publication	−0.002 (−0.031 to 0.027)	0.904	0.055	0.118
Multicenter studies	−0.091 (−0.366 to 0.183)	0.504	0.054	0.559
Coronary artery bypass graft trials	0.256 (0.015 to 0.497)	0.038	0.040	0.194
Off‐pump coronary artery bypass graft trials	0.256 (0.015 to 0.497)	0.038	0.040	0.513
Valve trials	−0.290 (−0.637 to 0.057)	0.099	0.045	0.075
Experimental group more invasive than control group	−0.245 (−0.489 to −0.001)	0.049	0.038	0.239
Mortality end point	0.216 (−0.040 to 0.472)	0.096	0.020	0.595
Clinical end point	0.153 (−0.102 to 0.409)	0.232	0.035	0.284
Patency end point	−0.066 (−0.376 to 0.244)	0.670	0.049	0.003
Statistical power, %	−0.070 (−2.786 to 2.646)	0.958	0.058	0.633
Expected relative risk reduction, %	−0.014 (−0.028 to 0.001)	0.067	0	0.104
Sample size	0.000 (0.000 to 0.000)	0.292	0.047	0.542
Follow‐up, mo	0.002 (−0.002 to 0.006)	0.336	0.041	0.532
High internal validity	−0.083 (0.426 to 0.260)	0.625	0.048	0.534
High, moderate, or uncertain validity	−0.076 (−0.408 to 0.256)	0.644	0.054	0.556
Crossover from experimental to control group, %	0.027 (0.001 to 0.053)	0.040	0.025	0.474
Crossover from control to experimental group, %	−0.010 (−0.068 to 0.048)	0.726	0.056	0.556
Total crossover, %	0.027 (0.002 to 0.052)	0.037	0.025	0.469
Relative crossover	−0.001 (−0.007 to 0.004)	0.640	0.052	0.556
Logarithm of the relative risk of mortality
Year of publication	−0.003 (−0.037 to 0.031)	0.842	0.099	0.488
Multicenter studies	−0.059 (−0.374 to 0.257)	0.710	0.099	0.490
Coronary artery bypass graft trials	0.346 (−0.268 to 0.959)	0.262	0.093	0.013
Off‐pump coronary artery bypass graft trials	0.313 (0.041 to 0.585)	0.025	0.062	0.324
Valve trials	−0.284 (−0.703 to 0.134)	0.178	0.092	0.001
Experimental group more invasive than control group	−0.223 (−0.537 to 0.090)	0.157	0.083	0.099
Mortality end point	0.218 (−0.125 to 0.560)	0.207	0.087	0.055
Clinical end point	0.212 (−0.108 to 0.532)	0.188	0.089	0.034
Patency end point	−0.306 (−0.700 to 0.088)	0.125	0.090	0.026
Statistical power, %	−1.945 (−4.607 to 0.716)	0.147	0.076	0.472
Expected relative risk reduction, %	−0.003 (−0.040 to 0.033)	0.824	0.027	0.573
Sample size	0.000 (0.000 to 0.000)	0.519	0.010	0.476
Follow‐up, mo	0.002 (−0.003 to 0.007)	0.446	0.099	0.488
High internal validity	−0.405 (−0.834 to 0.025)	0.064	0.079	0.439
High, moderate, or uncertain validity	0.060 (−0.341 to 0.461)	0.764	0.102	0.488
Crossover from experimental to control group, %	0.040 (0.008 to 0.071)	0.015	0.073	0.399
Crossover from control to experimental group, %	0.005 (−0.062 to 0.073)	0.871	0.100	0.086
Total crossover, %	0.037 (0.006 to 0.067)	0.021	0.076	0.405
Relative crossover	−0.001 (−0.006 to 0.005)	0.815	0.097	0.486

Figure 1

Forest plot for the relative risk of the primary end point, with studies subgrouped and then sorted according to decreasing rate of crossover from experimental group (EG) to control group (CG). All relative risks were calculated as >1 favored the CG and <1 favored the EG. Between‐group heterogeneity P=0.017. ART, Randomized Trial of Bilateral versus Single Internal‐Thoracic‐Artery Grafts; BBS, Best Bypass Surgery Trial; BHACAS, Beating Heart Against Cardioplegic Arrest Studies; CARRPO, Copenhagen Arterial Revascularization Randomized Patency and Outcome Trial; CORONARY, CABG Off or On Pump Revascularization Study; CRISP, Coronary artery bypass grafting in high‐risk patients randomised to off‐ or on‐pump surgery; CTSNET AFIB, Cardiothoracic Surgical Trials Network ‐ Surgical Ablation of Atrial Fibrillation during Mitral‐Valve Surgery; DOORS, Danish On‐Pump Versus Off‐Pump Randomization Study; GOPCABE, German Off Pump Coronary Artery Bypass in Elderly Study; MASS, Medicine, Angioplasty, or Surgery Study; OCTOPUS, A Comparison of On‐Pump and Off‐Pump Coronary Bypass Surgery in Low‐Risk Patients; PRAGUE 6, Off‐Pump Versus On Pump Coronary Artery Bypass Graft Surgery in Patients With EuroSCORE ≥6; PROMISS, Prospective Randomized Comparison of Off‐Pump and On‐Pump Multi‐vessel Coronary Artery Bypass Surgery; RAPCO SVG, Radial Artery Patency; and Clinical Outcomes trial ‐ saphenous vein graft arm; RAPCO RITA, Radial Artery Patency; and Clinical Outcomes trial ‐ right internal thoracic artery arm; RAPS, radial artery patency study; REGROUP, Randomized Endo‐Vein Graft Prospective; RESTORE‐MV, Randomized Evaluation of a Surgical Treatment for Off‐Pump Repair of the Mitral Valve; ROOBY, Randomized On/Off Bypass; RSVP, Radial Artery Versus Saphenous Vein Patency Trial; STICH, Surgical Treatment for Ischemic Heart Failure.

Figure 2

Forest plot for the relative risk of the primary end point, with studies subgrouped and then sorted according to decreasing rate of crossover from control group (CG) to experimental group (EG). All relative risks were calculated as >1 favored the CG and <1 favored the EG. Between‐group heterogeneity P=0.002. ART, Randomized Trial of Bilateral versus Single Internal‐Thoracic‐Artery Grafts; BBS, Best Bypass Surgery Trial; BHACAS, Beating Heart Against Cardioplegic Arrest Studies; CARRPO, Copenhagen Arterial Revascularization Randomized Patency and Outcome Trial; CORONARY, CABG Off or On Pump Revascularization Study; CRISP, Coronary artery bypass grafting in high‐risk patients randomised to off‐ or on‐pump surgery; CTSNET AFIB, Cardiothoracic Surgical Trials Network ‐ Surgical Ablation of Atrial Fibrillation during Mitral‐Valve Surgery; DOORS, Danish On‐Pump Versus Off‐Pump Randomization Study; GOPCABE, German Off Pump Coronary Artery Bypass in Elderly Study; MASS, Medicine, Angioplasty, or Surgery Study; OCTOPUS, A Comparison of On‐Pump and Off‐Pump Coronary Bypass Surgery in Low‐Risk Patients; PRAGUE 6, Off‐Pump Versus On Pump Coronary Artery Bypass Graft Surgery in Patients With EuroSCORE ≥6; PROMISS, Prospective Randomized Comparison of Off‐Pump and On‐Pump MultI‐vessel Coronary Artery Bypass Surgery; RAPCO RITA, Radial Artery Patency; and Clinical Outcomes trial ‐ right internal thoracic artery arm; RAPCO SVG, Radial Artery Patency; and Clinical Outcomes trial ‐ saphenous vein graft arm; RAPS, radial artery patency study; REGROUP, Randomized Endo‐Vein Graft Prospective; RESTORE‐MV, Randomized Evaluation of a Surgical Treatment for Off‐Pump Repair of the Mitral Valve; ROOBY, Randomized On/Off Bypass; RSVP, Radial Artery Versus Saphenous Vein Patency Trial; STICH, Surgical Treatment for Ischemic Heart Failure.

Another additional important finding of meta‐regression analysis was the significant association between the RR for the primary end point (P=0.038) and for mortality (P=0.025) with off‐pump CABG, with effect estimates significantly in favor of the control intervention (ie, on‐pump CABG). Other nominally significant associations with the RR for the primary end point or for mortality occurred for the following: rate of crossover from experimental to control group (Figures S4 and S5); CABG trials; and trials allocating more invasive treatments to the experimental arm. Accompanying L'Abbè plots for the association between the rate of crossover from control to experimental group and the RRs for the primary end point and mortality are provided in Figures S6 and S7. Visual inspection highlights the lack of an evident crossover threshold.

Exploratory analysis to identify multivariable predictors of crossover (Table S4) suggested that crossover from experimental to control group was mostly associated with patency as primary end point (P=0.036). In addition, meta‐regression focusing on total crossover rate instead of relative crossover confirmed the overall findings (Table 3), with significant associations found for this composite variable and the RR for the primary end point (P=0.037) and for mortality (P=0.021). In contrast, relative crossover was not significantly associated with significant effect on the RR. Last, sensitivity analysis using |LogRR| (Table S5) showed that CABG trials (P=0.012), trials with a mortality end point (P<0.001), trials with a clinical end point (P=0.044), sample size (P=0.006), follow‐up (P=0.023), study validity (P=0.022), crossover from experimental to control group (P=0.007), and total crossover (P=0.005) were significantly associated with the |LogRR| for the primary end point. Furthermore, year of publication (P=0.034), multicenter design (P<0.001), sample size (P=0.004), study validity (P=0.009), and crossover from control to experimental group (P=0.021) were significantly associated with the |LogRR| for mortality. Finally, visual inspection of funnel plots showed borderline small study effects/publication bias for the RR of the primary end point and no significant small study effects/publication bias for the RR of mortality (Figures S8 and S9).

Discussion

The optimal approach to personalized medicine and therapeutics entails a complex synthesis of diagnostic and treatment skills. Cardiac surgical management, arguably, appears particularly demanding as surgeons have to master advanced technical requirements while applying evidence‐based recommendations. As such, clinical research designed and conducted in this particularly challenging scenario, with uncertain applicability and generalizability, requires careful scrutiny of its strengths and weaknesses. By including all RCTs focusing on cardiac surgical interventions, we hereby provide a comprehensive and updated perspective on the prevalence of crossover from experimental to control group and vice versa in cardiac surgery RCTs, highlighting the detrimental impact of high crossover from experimental to control group in terms of effect dilution for the primary end point and for mortality. We corroborate these findings using a comprehensive estimate of crossover and absolute effect estimates, thus in a more generalistic context.

By definition, crossover dilutes the treatment effect in the intention‐to‐treat analysis. It has been estimated that, for a fixed sample size and power, the increase of the crossover rate from 0% to 25% may reduce the power of the study from 77% to 57%.6 In surgical RCTs, things are even more complex, as crossover rate from the experimental to the control group may be to be a marker of a lack of familiarity of the participating surgeons with the (usually new or more complex) surgical procedure performed in the experimental arm. In this case, the association between crossover and outcome may overcome the purely statistical diluting effect of crossover in general and suggests a problem with the deliverability of the procedure. In fact, crossover has been associated with learning curve, as well as with suboptimal technical and clinical results in surgical RCTs.5, 10

Crossover rate from off‐pump to on‐pump CABG was used as a surrogate for surgeon's experience by Gaudino et al.10 By pooling data from 104 RCTs (20 627 patients), the authors showed a statistically significant excess risk of late mortality in the off‐pump CABG group for studies with a crossover rate ≥10% (incidence rate ratio, 1.30; 95% CI, 1.04–1.62); moreover, at meta‐regression, there was a correlation between the proportion of patients who crossed over from off‐pump to on‐pump CABG and the rate of incomplete revascularization in the off‐pump CABG arm (β=0.0019; P=0.03).

In a systematic review of all RCTs comparing minimally invasive surgery with open surgery for gastrointestinal cancer, surgical experience showed an inverse relationship with crossover (β=−2.344; P=0.037). At multivariate analysis, a statistically significant correlation between crossover to the open surgery arm and 30‐day mortality (β=0.125; P=0.033), anastomotic leak rate (β=0.550; P=0.004), and early complications (β=1.255; P=0.001) was found.5

Our analysis cannot shed a conclusive light on the existence of a level of crossover where the results of the intention‐to‐treat analysis are by definition invalidated. Similarly, we could not find a safety threshold for crossover, below which results of an RCT can be considered bulletproof. Clearly, the complex relation between the crossover, event rate, power, and sample size must be considered on a case‐by‐case basis. For studies with few events and high crossover rates, major concerns may exist. Pragmatically, we propose that optimal crossover rates should arguably be <5%. The use of entry criteria for surgeons’ experience, pretrial training, and close monitoring of crossover rate should be implemented in every surgical RCT to ensure the methodological validity of the trial.

Yet, an RCT does not necessarily correspond to clinical practice, and the design of RCTs is typically based on phase 3 pharmacologic trials. Focusing on the percutaneous coronary intervention versus CABG comparison, for instance, randomization typically occurs after coronary angiography. In addition, it is often the case that percutaneous coronary intervention and CABG are considered complementary, rather than alternative and competing treatments, as in many hybrid revascularization studies. Furthermore, percutaneous coronary intervention after CABG and CABG after percutaneous coronary intervention may also represent disease progression, rather than failure of the initial treatment strategy. Finally, it should be borne in mind that “pure treatment” should not be equated to “intention to treat,” and that our work mainly focuses on the features of surgical RCTs according to crossover from pure treatment. Indeed, in selected instances, crossover from pure treatment may actually be correctly mirroring best clinical practice and, thus, be appropriate and justifiable.

Our analysis was also limited to cardiac surgery RCTs where one may argue that the level of technical complexity is usually higher than in other fields of surgery. It is possible that the strength of the association between crossover rate and outcome is different in other surgical or medical specialties. However, our focus on a discipline, such as cardiac surgery, that entails complex organizational and procedural hurdles and involves patients at high risk provides a powerful data set capable of informing on the impact of crossover in surgical trials. Another important limitation of our work is that the included RCTs tested different interventions with different levels of technical complexity and it is likely that the association between crossover and outcome may be different for different procedures. Indeed, statistical heterogeneity was significant in our analysis, as can be expected when appraising studies focusing on heterogeneous end points and interventions. Also, some trial was powered for and reported multiple comparisons, and our model did not adjust for possible interactions. In addition, our work shares all the common limitations of meta‐epidemiologic studies, including ecological fallacy, selective reporting, small study effect, and residual confounding. Accordingly, our analysis remains a constructive critique to modern cardiac surgery RCTs, hopefully suitable to inform the conduct, analysis, and interpretation of ongoing and future trials in this field and arguably in other medical and surgical disciplines as well.

In conclusion, we have shown that crossover, particularly from experimental to control group, is associated with outcome, including mortality, in cardiac surgery RCTs. Crossover should be minimized when designing RCTs, and carefully appraised after their completion, to maximize the chance of methodological success and efficient resource use.

Disclosures

Dr Biondi‐Zoccai has consulted for Abbott Vascular and Bayer. The remaining authors have no disclosures to report.

Table S1. Full Search Strategy

Table S2. Surgical Interventions Tested in the INCLUDED Trials

Table S3. Cochrane Risk of Bias Tool for Assessing Risk of Bias in Randomized Trials

Table S4. Exploratory Analysis for Multivariable Predictors of High Crossover (ie, higher or equal to the median)*

Table S5. Meta‐Regression Analysis Exploring Potential Moderators of Absolute Effect Sizes

Figure S1. Preferred Reporting Items for Systematic Reviews and Meta‐Analyses flow diagram.

Figure S2. Forest plot for the relative risk of mortality, with studies subgrouped and then sorted according to decreasing rate of crossover from experimental group (EG) to control group (CG).

Figure S3. Forest plot for the relative risk of mortality, with studies subgrouped and then sorted according to decreasing rate of crossover from control group (CG) to experimental group (EG).

Figure S4. L'Abbè plot for the association between crossover rate from experimental to control group and logarithm of the relative risk of the primary endpoint.

Figure S5. L'Abbè plot for the association between crossover rate from experimental to control group and logarithm of the relative risk of mortality.

Figure S6. L'Abbè plot for the association between cross‐over rate from control to experimental group and logarithm of the relative risk of the primary endpoint.

Figure S7. L'Abbè plot for the association between cross‐over rate from control to experimental group and logarithm of the relative risk of mortality.

Figure S8. Funnel plot showing borderline small study effects/publication bias for the relative risk of the primary endpoint (P=0.090 at Egger test, P=0.360 at Begg test).

Figure S9. Funnel plot showing no significant small study effects/publication bias for the relative risk of mortality (P=0.328 at Egger test, P=0.492 at Begg test).

Click here for additional data file.