Robustness of the Comparative Observational Evidence Supporting Class I and II Cardiac Surgery Procedures.

Background Current cardiac surgery guidelines give Class I and II recommendations to valve-sparing root replacement over the Bentall procedure, mitral valve (MV) repair over replacement, and multiple arterial grafting with bilateral internal thoracic artery based on observational evidence. We evaluated the robustness of the observational studies supporting these recommendations using the E value, an index of unmeasured confounding. Methods and Results Observational studies cited in the guidelines and in the 3 largest meta-analyses comparing the procedures were evaluated for statistically significant effect measures. Two E values were calculated: 1 for the effect-size estimate and 1 for the lower limit of the 95% CI. Thirty-one observational studies were identified, and E values were computed for 75 effect estimates. The observed effect estimates for improved clinical outcomes with valve-sparing root replacement versus the Bentall procedure, MV repair versus replacement, and grafting with bilateral internal thoracic artery versus single internal thoracic artery could be explained by an unmeasured confounder that was associated with both the treatment and outcome by a risk ratio of more than 16.77, 4.32, and 3.14, respectively. For MV repair versus replacement and grafting with bilateral internal thoracic artery versus single internal thoracic artery, the average E values were lower than the effect sizes of the other measured confounders in 33.3% and 60.9% of the studies, respectively. For valve-sparing root replacement versus the Bentall procedure, no study reported effect sizes for associations of other covariates with outcomes. Conclusions The E values for observational evidence supporting the use of valve-sparing root replacement, MV repair, and grafting with bilateral internal thoracic artery over the Bentall procedure, MV replacement, and grafting with single internal thoracic artery are relatively low. This suggests that small-to-moderate unmeasured confounding could explain most of the observed associations for these procedures.

bilateral internal thoracic artery

mitral valve

single internal thoracic artery

valve‐sparing root replacement

Clinical Perspective

What Is New?

The E values for observational evidence supporting the use of valve‐sparing root replacement, mitral valve repair, and grafting with bilateral internal thoracic artery over the Bentall procedure, mitral valve replacement, and grafting with single internal thoracic artery are relatively low.

This suggests that small‐to‐moderate unmeasured confounding could explain most of the observed associations for these procedures.

What Are the Clinical Implications?

The E value, or other similar sensitivity analyses, should be part of the reporting of all comparative observational studies.

Randomized controlled trials are the standard for comparing the treatment effects of different surgical procedures. However, randomized evidence is available only for a minority of questions in cardiac surgery and surgeons often must rely on observational evidence.

Current guidelines recommend (Class I; Level of Evidence C) valve‐sparing root replacement (VSRR) over the Bentall procedure, when possible, for patients with proximal aortic aneurysms. Similarly, mitral valve (MV) repair is recommended over replacement in patients with degenerative mitral regurgitation as a Class I, Level of Evidence C recommendation. In coronary surgery, multiple arterial grafting with bilateral internal thoracic artery (BITA) is a Class IIa, Level of Evidence B recommendation in patients not at increased risk of sternal wound infection. All these recommendations are solely based on observational evidence (notably for grafting with BITA; in fact, the only randomized trial suggested lack of effect).

However, observational studies may be confounded by treatment allocation bias. Although stratification, propensity matching, and regression‐based adjustments can adjust for assumed and measured confounders, there is potential for unmeasured confounders. Strategies to minimize unmeasured confounding such as the negative control method, the perturbation variable method, instrumental variable methods, sensitivity analysis, and ecological analysis require informed assumptions and are complex to perform hence, their use in clinical research is very limited.

The E value is a method used to analyze unmeasured confounding in observational studies by objectively quantifying the minimum strength of association on the risk ratio scale that an unmeasured confounder must have with both the treatment and outcome, while simultaneously considering measured covariates, to negate the observed treatment‐outcome association. Importantly, the E value does not require assumptions on the nature or prevalence of the unmeasured confounder(s).

We evaluated the robustness of observational studies comparing VSRR versus the Bentall procedure, MV repair versus replacement, and BITA versus single internal thoracic artery (SITA) grafting using the E value.

Methods

Data, analytic methods, and study materials are available upon reasonable request and approval by the authors.

Study Selection and Data Extraction

All observational studies in the most recent guidelines , , supporting VSRR versus the Bentall procedure, MV repair versus replacement, and BITA versus SITA grafting, respectively, were identified. An additional literature search was performed to identify the observational studies in the 3 largest meta‐analyses comparing these procedures. , , All observational studies were evaluated for data extraction and subsequent analysis. Articles were included if they reported any statistically significant association between the surgical procedures and at least 1 clinical outcome.

For each selected study, data were extracted by 2 independent reviewers (I.H., M.G.) on the study characteristics (study origin, year of publication, number of patients), type of surgical procedure, clinical outcomes, the effect measure (relative risk [RR], odds ratio [OR], or hazard ratio [HR]), the effect size estimate, and the associated 95% CI. Data were also extracted for the type of adjustment for risk factors in the studies and the effect estimates of other confounders associated with the outcomes in each study.

Calculation of E Value

Using previously described methodology for each effect measure, 2 E values were calculated: 1 for the effect‐size estimate and 1 for the lower limit of the 95% CI. For consistency, effect estimates were inverted where necessary, so that all relative effects were >1. For a RR, the E value was estimated as RR+√[RR×(RR−1)], with RR being the observed risk ratio estimate after adjustments for measured confounders. For an OR or HR, RR in the previous formula was replaced by OR or HR when the outcome was uncommon (<15%). If the outcome was common (≥15%), RR was replaced with √OR for the OR and by (1−0.5√HR)/(1−0.5√1/HR) for the HR.

All calculations were performed using R version 3.4.3 (R Foundation for Statistical Computing, Vienna, Austria) and using the EValue and pairwiseCI packages. ,

Statistical Analysis

Following calculation of E values for each effect measure, the averages of the E values for the different treatment‐outcome effect measures, and the corresponding lower CI limits for each surgical comparison were calculated. These were compared with the effect measures for associations of other covariates with study outcomes for each observational study reporting the surgical comparison.

For each surgical comparison, the averages of the E values of effect measures and the corresponding lower CI limits were also calculated based on type of clinical outcome, and study strategy for adjustment of confounders (none, multivariable adjustment, propensity matching, and propensity matching and multivariable adjustment [doubly robust]).

Results

Study Characteristics

Thirty‐one observational studies were identified: 4 comparing VSRR versus the Bentall procedure, 7 comparing MV repair versus replacement, and 20 comparing BITA versus SITA grafting. Twelve studies were from the United States, 5 from Canada, 3 from Japan, and the rest from other countries. E values were computed for 75 effect estimates and 64 lower CI limits (Figure 1). The details of study characteristics and the effect estimate of covariates reported for each study are summarized in Table 1. , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Figure 1

E values of effect estimates in observational studies comparing bilateral vs single internal thoracic artery grafting (BITA vs SITA), mitral valve repair vs replacement (MVr vs MVR), and valve‐sparing root replacement (VSRR) vs Bentall procedure.

 

Table 1

Demographics of Patients in the Included Studies

Study	Institution	Country	Study Period	No. of Patients	Type of Adjustment	Effect Estimates of Other Confounders Reported in Study
Benedetto et al, 2014 13	Harefield Hospital, London	United Kingdom	2001–2013	4195 (750 BITA grafts, 3445 SITA grafts)	PSM	Outcome: death No prior MI: HR, 0.48; 95% CI, 0.23–0.98 LVEF <50: HR, 0.18; 95% CI, 0.05–0.60
Berreklouw et al, 2001 14	Catharina Hospital	The Netherlands	1985–1990	482 (249 BITA, 233 SITA)	Multivariable adjustment	Outcome: angina Female sex: RR, 1.9; 95% CI, 1.2–3.0 Outcome: angina‐free survival Age: RR, 1.0; 95% CI, 1.0–1.1
Buxton et al, 1998 15	Austin and Repatriation Medical Center	Australia	1985–1995	2853 (1296 BITA grafts, 1557 SITA grafts)	Multivariable adjustment	Outcome: death PVD: RR, 2.4; 95% CI, 1.7–3.4 Prior MI: RR, 2.1; 95% CI, 1.5–3.1 Severe left ventricular dysfunction: RR, 3.9; 95% CI, 2.6–5.9 Moderate left ventricular dysfunction: RR, 2.0; 95% CI, 1.5–2.6 Age >/=70 y: RR, 3.4; 95% CI, 2.4–4.8 Age 60–69 y: RR, 1.7; 95% CI, 1.3–2.4 DM: RR, 1.7; 95% CI, 1.3–2.4 Carotid disease: RR, 1.7; 95% CI, 1.2–2.4 Outcome: composite of allcause mortality, late myocardial infarction, or late reoperation PVD: RR, 2.4; 95% CI, 1.5–2.9 Prior MI: RR, 2.1; 95% CI, 1.3–2.2 Severe left ventricular dysfunction: RR, 3.1; 95% CI, 2.1–3.4 Moderate left ventricular dysfunction: RR, 2.0; 95% CI, 1.5–2.6 Age >/=70 y: RR, 3.4; 95% CI, 1.8–3.7 Age 60–69 y: RR 1.3, 95% CI 1.0–1.7 DM: RR, 1.7; 95% CI, 1.3–2.2
Calafiore et al, 2005 16	Multicenter	Italy	1986–1999	1602 (1026 BITA, 576 SITA)	PSM and multivariable adjustment	NR
Carrier et al, 2009 17	Montreal Heart Institute	Canada	1995–2007	6655 (1235 BITA grafts, 5420 SITA grafts)	Multivariable adjustment	Outcome: death Age: HR, 1.06; 95% CI, 1.05–1.07 Sex: HR, 0.90; 95% CI, 0.78–1.04 DM: HR, 1.63; 95% CI, 1.43–1.86 Hyperlipidemia: HR, 0.83; 95% CI, 0.72–0.95 Antiplatelet agents: HR, 0.83; 95% CI, 0.73–0.95 Beta-blocker: HR, 0.74; 95% CI, 0.65–0.45 ACE-inhibitor: HR, 1.22; 95% CI, 1.05–1.42 Statin: HR, 0.73; 95% CI, 0.67–0.86
Chikwe et al, 2011 18	Mount Sanai Medical Center, Herzzentrum Universitaet	United States, Germany	1998–2008	322 (227 MVr, 95 MVR)	Multivariable adjustment	Outcome: survival Age: HR, 1.1; 95% CI, 1.0–1.2 LVEF ≤30%: HR, 1.8; 95% CI, 1.0–3.3 Renal failure: HR, 1.8; 95% CI, 1.1–2.8 Emergency surgery: HR, 2.9; 95% CI, 1.6–5.2
Endo et al, 2001 19	Tokyo Women's Medical University	Japan	1985–1998	1131 (443 BITA grafts, 688 SITA grafts grafts)	Multivariable adjustment	NR
Gogbashian et al, 2005 20	Brigham and Women's Hospital	United States	1992–2002	183 (147 MVr, 36 MVR)	Multivariable adjustment	Outcome: death NYHA I/II cardiac failure (vs III/IV): HR, 0.52; 95% CI, 0.32–0.86 COPD: HR, 2.79; 95% CI, 1.47–5.28 Cerebrovascular disease: HR, 1.52; 95% CI, 1.31–1.93 Hypercholesterolemia: HR, 2.07; 95% CI, 1.24–3.46 Chronic renal insufficiency: HR, 1.76; 95% CI, 1.22–2.57 MVR and CABG: HR, 1.66; 95% CI, 1.03–2.67 Postoperative pneumonia: HR, 1.62; 95% CI, 1.35–2.18 Postoperative stroke: HR, 1.64; 95% CI, 1.11–2.60
Grau et al, 2015 21	The Valley Columbia Heart Center	United States	1994–2013	6666 (1544 BITA grafts, 5122 SITA grafts)	PSM and multivariable adjustment	Outcome: death LVEF (%): HR, 0.97; 95% CI, 0.96–0.98 Age: HR, 1.08; 95% CI, 1.06–1.09 DM: HR, 1.59; 95% CI, 1.15–2.20 PVD: HR, 1.70; 95% CI, 1.28–2.27 History of renal failure: HR, 3.39; 95% CI, 1.43–8.04 History of smoking: HR, 1.41; 95% CI, 1.10–1.81 Surgery era (early reference) 2001–2005: HR, 0.65; 95% CI, 0.45–0.93 Total grafts placed: HR, 0.84; 95% CI, 0.74–0.97 Blood transfusion at surgery: HR, 1.43; 95% CI, 1.13–1.82
Itoh et al, 2016 22	Saitama Medical Center	Japan	1990–2014	400 (107 BITA grafts, 293 SITA grafts)	PSM	NR
Javadikasgari et al, 2017 23	Cleveland Clinic	United States	1985–2011	1071 (872 MVr, 199 MVR)	Multivariable adjustment	NR
Kelly et al, 2012 24	Queen Elizabeth II Health Sciences Center	Canada	1995–2007	7633 (1079 BITA, 6554 SITA)	Multivariable adjustment	Outcome: survival No ITA: HR, 1.42; 95% CI, 1.24–1.62 Incomplete revascularization: HR, 1.23; 95% CI, 1.10–1.38 Age 60–69 y: HR, 1.75; 95% CI, 1.49–2.06 Age 70–79 y: HR, 2.96; 95% CI, 2.52–3.48 Age ≥80 y: HR, 4.86; 95% CI, 3.96–5.98 BMI <25: HR 1.20, 95% CI 1.07–1.34 BMI >35: HR, 1.22; 95% CI, 1.04–1.43 DM: HR, 1.50; 95% CI, 1.35–1.66 Renal function: HR, 2.05; 95% CI, 1.78–2.36 PVD: HR, 1.69; 95% CI, 1.52–1.88 COPD: HR, 1.66; 95% CI, 1.48–1.85 LVEF <40%: HR, 1.80; 95% CI, 1.60–2.02 In-hospital urgent: HR, 1.34; 95% CI, 1.19–1.52 Urgent: HR, 1.78; 95% CI, 1.54–2.05 Emergency: HR, 1.83; 95% CI, 1.48–2.26
Kieser et al, 2011 25	The Province of Alberta	Canada	1995–2008	5067 (1038 BITA grafts, 4029 SITA grafts)	Multivariable adjustment	NR
Kinoshita, 2015 26	Shiga University of Medical Science	Japan	2002–2014	1203 (750 BITA grafts, 453 SITA grafts)	PSM and multivariable adjustment, multivariable adjustment only	Outcome: death (PSM and multivariable adjusted) Age per 1-SD increase: HR: 1.40: 95% CI, 1.12–1.75 BMI: HR, 0.79; 95% CI, 0.67–0.93 End-stage renal failure: HR, 3.02; 95% CI, 1.97–4.63 Peripheral arterial disease: HR, 1.90; 95% CI, 1.26–2.87 Prior MI: HR, 1.93; 95% CI, 1.31–2.84 Outcome: cardiac death (PSM and multivariable adjusted) End-stage renal failure: HR, 8.08; 95% CI, 4.23–15.43 Peripheral arterial disease: HR, 2.71; 95% CI, 1.43–5.14 Prior MI: HR, 2.99; 95% CI, 1.57–5.69 Heart failure: HR, 1.95; 95% CI, 1.04–3.66 Outcome: death (multivariable adjusted) Age per 1-SD increase: HR, 1.38; 95% CI, 1.13–1.68 End-stage renal failure: HR, 3.49; 95% CI, 2.38–5.12 Peripheral arterial disease: HR, 2.26; 95% CI, 1.57–3.25 Prior MI: HR, 1.76; 95% CI, 1.24–2.50 Heart failure: HR, 1.61; 95% CI, 1.02–2.52 Outcome: cardiac death (multivariable adjusted) End-stage renal failure: HR, 6.80; 95% CI, 3.74–12.37 Peripheral arterial disease: HR, 2.45; 95% CI, 1.34–4.47 Prior MI: HR, 2.58; 95% CI, 1.42–4.69
Kurlansky, 2010 27	Florida Heart Research Institute	United States	1972–1994	4584 (2215 BITA, 2369 SITA)	PSM and multivariable adjustment	Outcome: death Age: HR, 1.06; 95% CI, 1.06–1.07 Angina-stable: HR, 0.89; 95% CI, 0.82–0.97 Cardiac arrest: HR, 1.59; 95% CI, 1.20–2.11 CHF: HR, 1.44; 95% CI, 1.28–1.62 Cerebrovascular disease: HR, 1.45; 95% CI, 1.22–1.73 DM: HR, 1.52; 95% CI, 1.39–1.66 Dyslipidemia: HR, 0.87; 95% CI, 0.76–0.98 LVEF: HR, 1.33; 95% CI, 1.22–1.45 Female sex: HR, 0.88; 95% CI, 0.80–0.97 LM disease: HR, 1.17; 95% CI, 1.06–1.30 Prior MI: HR, 1.23; 95% CI, 1.14–1.34 Pulmonary insufficiency: HR, 1.35; 95% CI, 1.14–1.61 PVD: HR, 1.47; 95% CI, 1.24–1.73 Renal disease: HR, 1.44; 95% CI, 1.19–1.73 Perfusion time: HR, 1.00; 95% CI, 1.00–1.00 Renal insufficiency: HR, 1.99; 95% CI, 1.58–2.50 MI: HR, 1.42; 95% CI, 1.20–1.69
Lazam et al, 2017 28	Multicenter	Multinational	1980–2005	1922 (1922 MVr, 213 MVR)	PSM	NR
Lee et al, 1997 29	Papworth Hospital Regional Cardiac Center	United Kingdom	1987–1994	278 (167 MVr, 111 MVR)	Multivariable adjustment	Outcome: death Age >70 y: HR, 2.1; P=0.025 LVEF ≤40%: HR, 2.1; P=0.030 NYHA III or IV:HR, 4.8; P=0.004 Outcome: Heart failure Age >70 y: HR, 2.5; P=0.012 LVEF ≤40%: HR, 2.8; P=0.006 NYHA III or IV: HR, 5.0; P=0.010 Outcome: anticoagulation‐related hemorrhage Age >70 y: HR, 6.3; P=0.0059
Lee et al, 2018 30	Seoul National University Bundang Hospital	South Korea	1995–2013	216 (82 VSSR, 134 Bentall)	PSM, multivariable adjustment
Locker et al, 2012 31	Mayo Clinic	United States	1993–2009	8295 (860 BITA grafts, 7435 SITA grafts)	Multivariable adjustment	Outcome: Death Older age (per 1 y): HR, 1.07; 95% CI, 1.06–1.07 Low LVEF (per 1%): HR, 1.02; 95% CI, 1.02–1.02 Hypertension: HR, 1.14; 95% CI, 1.05–1.25 DM: HR, 1.55; 95% CI, 1.44–1.68 Chronic lung disease: HR, 1.66; 95% CI, 1.50–1.83 Renal failure: HR, 2.29; 95% CI, 2.01–2.62 PVD: HR, 1.45; 95% CI,1.34–1.57 MI: HR, 1.10; 95% CI, 1.02–1.19 CVA: HR,1.56; 95% CI, 1.38–1.76 LM disease >50%: HR, 1.17; 95% CI, 1.08–1.26 Urgent/emergent: HR, 1.11; 95% CI, 1.01–1.21 OPCAB: HR, 1.30; 95% CI, 1.11–1.52
Medalion et al, 2010 32	Tel Aviv Sourasky Medical Center	Israel	1996–2008	1627 (1045 BITA grafts, 582 SITA grafts)	Multivariable adjustment	Outcome: death Age 80 y: HR, 0.50; 95% CI, 0.41–0.61 Age 75–79 y: HR, 0.73; 95% CI, 0.62–0.81 DM: HR, 0.73; 95% CI, 0.64–0.84 COPD: HR, 0.58; 95% CI, 0.47–0.72 CHF: HR, 0.66; 95% CI, 0.55–0.77 Emergency operation: HR, 0.80; 95% CI, 0.68–0.99 PVD: HR, 0.80; 95% CI, 0.69–0.95 CVD: HR, 0.80; 95% CI, 0.67–0.96 Repeat operation: HR, 0.50; 95% CI, 0.35–0.70 Conduit—RA: HR, 1.36; 95% CI,1.10–1.69
Navia, 2016 33	Instituto Cardiovascular de Buenos Aires	Argentina	1996–2014	2486 (2098 BITA, 388 SITA)	Multivariable adjustment	Outcome: death Age years: HR, 1.07; 95% CI, 1.06–1.08 DM: HR, 1.69; 95% CI, 1.39–2.06 Cerebrovascular disease: HR, 2.16; 95% CI, 1.49–3.11 Previous renal dysfunction: HR, 2.12; 95% CI, 1.58–2.85 Smoking habit: HR, 1.47; 95% CI, 1.21–1.78 Elective operation: HR, 0.78; 95% CI, 0.64–0.94 Left ventricular dysfunction (moderate/severe): HR, 2.47; 95% CI, 1.92–3.19
Ouzounian et al, 2016 34	Peter Munk Cardiac Centre	Canada	1990–2010	616 (253 VSRR, 363 Bentall)	Multivariable adjustment	NR
Parsa et al, 2013 35	Duke University Medical Center	United States	1984–2009	17 609 (728 BITA grafts, 16 881 SITA grafts)	Multivariable adjustment	NR
Pettinari et al, 2015 36	Multicenter	Belgium	1972–2006	3496 (1328 BITA grafts, 2168 SITA grafts)	PSM	Outcome: death Experience: OR, 0.68; 95% CI, 0.58–0.81 Age: OR, 1.04; 95% CI, 1.00–1.07 Preop dialysis: OR, 0.07; 95% CI, 0.01–0.40 Preop creatinine: OR, 1.48; 95% CI, 1.30–1.69 LVEF: OR, 0.99; 95% CI, 0.99–0.99 FEV1: OR, 0.99; 95% CI, 0.99–0.99 Recent MI: OR, 3.57; 95% CI, 1.75–7.27 PVD: OR, 1.34; 95% CI, 1.08–1.66
Pick et al, 1997 37	Mayo Clinic	United States	1983–1986	321 (160 BITA grafts, 161 SITA grafts)	Multivariable adjustment	Outcome: angina recurrence Female sex: HR, 1.81; 95% CI, 1.22–2.69 Obesity: HR, 1.69; 95% CI, 1.21–2.19 Preop hypertension: HR, 1.54; 95% CI, 1.87–2.19 Outcome: late MI DM: HR, 3.39; 95% CI, 1.81– 6.34
Price et al, 2016 38	Johns Hopkins Hospital	United States	1997–2013	165 (98 VSRR, 67 Bentall)	PSM and multivariable adjustment	NR
Schwann et al, 2016 39	Multicenter	United States	1987–2011	5125 (641 BITA grafts, 4484 SITA grafts)	PSM and multivariable adjustment, multivariable adjustment only	NR
Stevens et al, 2004 40	Montreal Heart Institute	Canada	1985–1995	4382 (1835 BITA grafts, 2547 SITA grafts)	PSM and multivariable adjustment, multivariable adjustment only	Outcome: death Age: HR, 1.02; 95% CI, 1.01–1.03 DM: HR, 1.81; 95% CI, 1.47–2.23 Prior MI: HR, 1.36; 95% CI, 1.13–1.63 CHF: HR, 2.73; 95% CI, 1.59–4.67 PVD: HR, 2.24; 95% CI, 1.74–2.89 COPD: HR, 1.54; 95% CI, 1.12–2.11 Outcome: MI Age: HR, 0.98; 95% CI, 0.97–0.99 Diabetes: HR, 1.46; 95% CI, 1.20–1.77 Prior MI: HR, 1.40; 95% CI, 1.20–1.63 PVD: HR, 1.45; 95% CI, 1.11–1.89 Obesity: HR, 0.76; 95% CI, 0.59–0.98 Outcome: reoperation Age: HR, 0.94, 95% CI, 0.91–0.97 Preop PCI: HR, 3.28, 95% CI, 1.01–10.6 PVD: HR, 2.56, 95% CI, 1.00–6.53 Outcome: any event Age: HR, 0.98; 95% CI, 0.98–0.99 DM: HR, 1.34; 95% CI, 1.14–1.57 Prior MI: HR, 1.34; 95% CI, 1.18–1.53 Preoperative PCI: HR, 3.28; 95% CI, 1.01–10.6 IABP: HR, 1.82; 95% CI, 1.49–2.22 CHF: HR, 1.49; 95% CI, 0.95–2.36 PVD: HR, 1.48; 95% CI, 1.20–1.83 Dyslipidemia: HR, 0.85; 95% CI, 0.70–1.02 COPD: HR, 1.32; 95% CI, 1.03–1.70
Suri et al, 2006 41	Mayo Clinic	United States	1980–1999	1411 (1173 MVr, 238 MVR)	Multivariable adjustment	Outcome: death Age: HR, 1.08; 95% CI, 1.06–1.09 NYHA class: HR, 1.44; 95% CI, 1.23–1.64 CABG: HR, 1.56; 95% CI, 1.27–1.92 Preop LVESD: HR, 1.02; 95% CI, 1.00–1.03
Yang et al, 2018 42	Michigan Medicine	United States	2001–2017	135 (40 VSRR, 95 Bentall)	Multivariable adjustment
Zhou et al, 2010 43	Centre Hospitalier Universitaire de Rangueil	France	1995–2002	319 (241 MVr, 78 MVR)	Multivariable adjustment	Outcome: survival NYHA functional class (IV or III): RR, 2.69; 95% CI, 1.45–4.99 Older age (>60 y): RR, 2.33; 95% CI,1.21–4.84 Renal impairment: RR, 2.27; 95% CI, 1.42–3.45

ACE indicates angiotensin converting enzyme; BITA, bilateral internal thoracic artery; BMI, body mass index; CABG, coronary artery bypass grafting; CHF, congestive heart failure; COPD, chronic obstructive pulmonary disease; CVA, cerebrovascular accident; DM, diabetes mellitus; FEV1, forced expiratory volume; HR, hazard ratio; IABP, intra‐aortic balloon pump; ITA, internal thoracic artery; LM, left main; LVEF, left ventricular ejection fraction; LVESD, left ventricular end‐systolic disease; MI, myocardial infarction; MVr, mitral valve repair; MVR, mitral valve replacement; NR, not reported; NYHA, New York Heart Association; OPCAB, off‐pump coronary artery bypass grafting; OR, odds ratio; PSM, propensity score matching; PVD, peripheral vascular disease; RR, relative risk; SITA, single internal thoracic artery; and VSRR, valve‐sparing root replacement.

VSRR Versus the Bentall Procedure

In the 4 studies comparing VSRR versus the Bentall procedure, the sample size ranged from 135 to 616 patients. One was propensity matched, 1 used both propensity matching and multivariable adjustment, and 2 used multivariable adjustment. On average, the observed effect estimates for improved clinical outcomes with VSRR versus the Bentall procedure could be explained by an unmeasured confounder that was associated with both VSRR and the clinical outcomes by a risk ratio of more than 16.77 (E value for lower confidence bound 2.44). No study reported effect sizes for associations of other covariates with study outcomes. The mean E value for effect estimates in propensity‐matched studies was highest (22.08), followed by multivariable‐adjusted (17.14) studies, and propensity‐matched and multivariable‐adjusted (15.60) studies. Details of the mean E values for the effect estimates of different clinical outcomes for VSRR versus the Bentall procedure are summarized in Table 2.

Table 2

Summary of Average E‐value Calculations

Variable	VSRR vs Bentall Procedure	MV Repair vs Replacement	BITA vs SITA Grafting
Number of studies	4	7	20
Number of effect estimates	16	14	47
Mean E values of effect estimates for different clinical outcomes (mean E value for lower confidence bound)
All clinical outcomes	16.77 (2.44)	4.32 (1.75)	3.14 (1.78)
Death	21.35 (1.83)	4.16 (1.71)	2.56 (1.74)
Cardiac death	12.88 (‐)	4.03 (1.74)	…
Composite outcome	8.06 (‐)	…	2.69 (1.66)
Myocardial infarction	…	…	3.42 (1.48)
Survival	…	3.17 (1.79)	2.05 (1.18)
Reoperation	20.92 (3.42)	…	4.29 (1.77)
Mean E‐values of effect estimates by adjustment strategy (mean E value for lower confidence bound)
Unadjusted	14.02 (1.81)	3.33 (2.17)	5.78 (3.97)
Multivariable adjusted	17.14 (2.73)	4.49 (1.57)	2.96 (1.62)
Propensity matched	22.08 (3.79)	3.59 (2.21)	2.24 (1.33)
Propensity matched and multivariable adjusted	15.60 (2.15)	…	2.97 (1.54)

BITA indicates bilateral internal thoracic artery; SITA, single internal thoracic artery; and VSRR, valve‐sparing root replacement.

MV Repair Versus Replacement

In the 7 studies comparing MV repair versus replacement, the sample size ranged from 183 to 1922 patients. One was propensity matched, and 6 used multivariable adjustment. On average, the observed effect estimates for improved clinical outcomes with MV repair versus the MV replacement could be explained by an unmeasured confounder that was associated with both MV repair and the clinical outcomes by an effect size of >4.32. This was lower than the effect size of the other measured confounders in 33.3% of the observational studies comparing MV repair versus replacement (E value for lower confidence bound 1.74). In terms of an adjustment strategy for the computation of treatment effects, the mean E value for effect estimates in multivariable adjusted studies was highest (4.49), followed by propensity‐matched studies (3.59). There were no studies that used both propensity matching and multivariable adjustment. Details of the average E values for the effect estimates of different clinical outcomes for MV repair versus replacement are summarized in Table 2.

BITA Versus SITA Grafting

In the 20 studies comparing BITA versus SITA grafting, the sample size ranged from 321 to 17 609 patients. Three studies were propensity matched, 6 used both propensity matching and multivariable adjustment, and 11 used multivariable adjustment. On average, the observed effect estimates for improved clinical outcomes with BITA versus SITA grafting could be explained by an unmeasured confounder that was associated with both BITA grafting and the clinical outcomes by an effect size of >3.14. This was lower than the effect size of the other measured confounders in 60.9% of the observational studies comparing BITA versus SITA grafting (E value for lower confidence bound 1.78). The mean E value for effect estimates in propensity‐matched and multivariable‐adjusted (doubly robust) studies was 2.97: 2.96 for multivariable‐adjusted and 2.24 for propensity‐matched studies. Details of the average E values for the effect estimates of different clinical outcomes for BITA versus SITA grafting are summarized in Table 2.

Discussion

Using the E value, we evaluated confounding bias in 31 observational studies on 3 guideline‐recommended cardiac surgical procedures. The observed treatment benefit for VSRR versus the Bentall procedure could be explained by an unmeasured confounder associated with both VSRR and clinical outcomes by a risk ratio of 16.76, above the adjusted variables. This was much higher than the treatment‐confounder association required to explain the treatment benefit of MV repair over replacement (4.32), and BITA over SITA grafting (3.14). Although this suggests that the evidence supporting VSRR over the Bentall procedure is relatively robust, no observational study showing benefit for VSRR reported the effect sizes for the associations of other covariates with the study outcomes. By comparison, in 33.3% and 60.9% of studies comparing MV repair versus replacement, and BITA versus SITA grafting, respectively, at least 1 study covariate was associated with both the treatment arm and clinical outcomes by an effect size larger than the average E value of these studies.

Studies with smaller sample sizes typically report larger effect estimates, and larger effect estimates yield larger E values. , This can spuriously give the impression that these small studies are more robust to unmeasured confounding. For each effect estimate in our analysis, the E values based on the lower bound of the CI were also calculated, as these are less influenced by study size. The lower confidence bound E values demonstrated that the association between VSRR and improved clinical outcomes could lose statistical significance if an unmeasured outcome was associated with both VSRR and the outcomes by a relative effect as low as 2.44. For MV repair and BITA grafting, these E values were even lower, at 1.74 and 1.78, respectively. These data further suggest that the observations in favor of VSRR are relatively more robust to unmeasured confounding than MV repair and BITA‐grafting observational evidence. Centrally, however, given the relatively small strengths of the association for the lower confidence bounds, the evidence on all the procedures seems fragile to unmeasured confounding.

Observational studies suggest that BITA grafting can improve patient survival generally based on greater and more‐durable graft patency compared with the saphenous vein, as well as increased native atherosclerosis progression associated with saphenous vein grafts. , However, randomized data comparing BITA versus SITA grafting report similar results for the 2 strategies. , , Although reasons inherent to the design and conduct of these trials have been described as possible reasons for this apparent disagreement, the treatment benefit of BITA grafting in observational studies has also been potentially attributed to unmeasured confounders and treatment‐allocation bias. , The same may be true for the other procedures analyzed. A surgeon's decision to perform VSRR versus the Bentall procedure is based on careful assessment of the patient's anatomy and functional status, as well as the surgeon's expertise in the procedures, all of which are variables that are difficult to measure and account for using statistical adjustment. Similarly, MV repair is more likely to be performed in patients considered to have a longer life expectancy by the operating surgeon.

It is important to note that the use of the E value does not prove that the findings in these comparisons are wrong per se. It is theoretically possible that the uncontrolled confounding could work in the opposite direction and be strengthening, instead of denying, the reported associations. However, evaluation using the E value demonstrates that the results reported by the analyzed observational studies are not robust to the idea of uncontrolled confounding possibly explaining the results.

To evaluate the effect of covariate adjustment or propensity matching on reducing confounding bias further, for each surgical comparison, we calculated the average E values for effect estimates stratified by type of adjustment and/or matching used in the observational studies. For both VSRR versus the Bentall procedure, and MV repair versus replacement studies, multivariable adjustment or propensity matching increased the average E value of effect estimates, suggesting increased robustness of effect estimates with adjustment or propensity matching. However, for BITA versus SITA grafting, adjustment or propensity matching of covariates did not increase the E value. These findings suggest that factors unrelated to the measured confounders in BITA‐ versus SITA‐grafting observational studies may have been associated with the outcomes, and that unmatched confounders continue to be present even in matched studies. Furthermore, this suggests that even the best statistical methods currently used to minimize confounding bias in observational studies may have major limitations.

From a practical perspective, the decision to perform a surgical procedure versus another is based on a complex clinical assessment of patients' characteristics, surgical anatomy, relative effectiveness, and safety of the 2 interventions, as well as the individual surgeon’s experience. Similarly, guideline recommendations are based on the evaluation of the totality of the evidence and the overall risk: benefit ratio. Our data add to the existing knowledge an objective assessment of the solidity of the comparative results for the interventions investigated, but are not enough per se to change recommendations or indicate what intervention to use in clinical practice.

Our study has limitations. Only statistically significant associations were selected, and the use of P values and statistical significance may not be ideal for estimating causal effects. It is possible to apply the E value in the absence of evidence of association to assess how much unmeasured/residual confounding would be required to make a null association clinically significant. However, the E value is only typically applied when claims of associations or treatment benefit are made. We did not also assess for other sources of bias in our study; it is possible that various forms of reporting bias might have been present in the selected studies. It is also possible that the effect estimates could have been biased by measurement error, and we could not examine how the exposures were measured.

In conclusion, the E values for observational evidence supporting use of VSRR, MV repair, and BITA grafting over the Bentall procedure, MV replacement, and SITA grafting, respectively, are relatively low. This suggests that small‐to‐moderate unmeasured confounding could explain most of the observed associations for these cardiac surgery procedures.

The E value, or other similar sensitivity analyses, should be part of the reporting of all comparative observational studies.

Sources of Funding

Prof Fremes is supported in part by the Bernard S. Goldman Chair in Cardiovascular Surgery. Dr Tam is supported in part by a Canadian Institutes of Health Research scholarship.

Disclosures

None.