Low Reporting of Cointerventions in Recent Cardiovascular Clinical Trials: A Systematic Review.

Background A cointervention in a randomized clinical trial (RCT) is medical care given in addition to the tested intervention. If cointerventions are unbalanced between trial arms, the results may be biased. We hypothesized that cointerventions would be more adequately reported in RCTs without full blinding or at risk of bias. Methods and Results To describe the reporting of cointerventions and to evaluate the factors associated with their reporting, we did a systematic search of all RCTs evaluating pharmacological interventions on cardiovascular outcomes published in 5 high-impact journals. The reporting of cointerventions, blinding, and risk of bias were extracted and evaluated independently by 2 reviewers (E.M., L.A.). Cointerventions were inadequately reported in 87 of 123 RCTs (70.7%), with 56 (45.5%) providing no information on cointerventions and 31 (25.2%) providing only partial information. Of the RCTs, 52 (42.3%) had inadequate blinding of participants and/or personnel and 63 (51.2%) of the RCTs were judged at risk of bias. In univariable analysis, the reporting of cointerventions was not associated with blinding of participants and/or personnel (odds ratio [OR], 1.04; 95% CI, 0.47-2.27 for adequately versus inadequately blinded trials) or with risk of bias (OR, 1.47; 95% CI, 0.67-3.21 for at low risk of bias versus trials at risk of bias). In multivariable analysis, only a follow-up of <1 month was associated with the adequate reporting of cointerventions (OR, 3.63; 95% CI, 1.21-10.91). Conclusions More than two-thirds of recent major cardiovascular trials did not adequately report cointerventions. The quality of reporting was not better among trials that were not fully blinded or at risk for bias. Registration URL: https://www.crd.york.ac.uk/PROSP​ERO/. Unique identifier: CRD42018106771.

odds ratio

randomized clinical trial

relative risk

Consolidated Standards of Reporting Trials

International normalized ratio

Preferred Reporting Items for Systematic Reviews and Meta‐Analyses

Stroke Prevention Using the Oral Direct Thrombin Inhibitor Ximelagatran in Patients With Atrial Fibrillation

Clinical Perspective

What Is New?

In this systematic review of major cardiovascular trials in 5 highly influential medical journals, cointerventions were inadequately reported in more than two‐thirds of the trials, whereas the quality of reporting was not better among trials that were not fully blinded or at risk for bias.

What Are the Clinical Implications?

Cointerventions should be systematically reported in cardiovascular trials to assess the validity of the findings, particularly when trials are not fully blinded.

Because randomized clinical trial (RCT) outcomes shape clinical guidelines and daily practice,1, 2 we expect them to meet the highest standards of methodological quality and provide us with robust results.3, 4 RCTs have benefitted from continuous improvement in methodological quality,5 especially in random sequence generation and allocation concealment, which have freed them from baseline confounding.5, 6, 7 However, randomization does not eliminate differences that may arise between treatment groups during follow‐up. After randomization, bias can arise when participants receive medical care in addition to the intervention of interest (cointerventions)6, 8 if it is not provided equally to all treatment groups.8, 9, 10, 11

When one group receives more cointerventions than another, the RCT results may be compromised by bias.6, 7, 8, 11 This unequal distribution of cointerventions might be caused by a failure to adequately blind participants and/or personnel.12, 13, 14 For example, if investigators know that a participant is receiving an active substance in a trial designed to prevent myocardial infarction (eg, new antidiabetic drugs), they might suggest that the participant take other medications that reduce cardiovascular risk (eg, statins). If a family doctor knows that a patient is not receiving the active substance, he or she might feel ethically bound to prescribe effective cointerventions.8 If cointerventions affect one group more than another, the results could be biased in either direction.6, 8 To reduce the risk of bias, cointerventions should be reported in both unblinded (ie, open label) and in double‐blind trials because blinding can be compromised during the course of even a double‐blind RCT by, for example, drugs that are not adequately matched, specific side effects, or laboratory investigations (such as lipid measurements).15, 16, 17, 18, 19 It is difficult to measure unblinding in a double‐blind RCT, but we can and should quantify its possible consequences by reporting relevant cointerventions.13, 16, 17

Patients in cardiovascular trials often receive multiple treatments (eg, statins, antihypertensives, antiplatelets) beyond the studied medication, each of which could affect outcomes, so cointerventions and in particular these comedications may play an important role in cardiovascular RCTs, especially if unblinded.6, 8, 20, 21 After several years without new potent drugs for cardiovascular prevention, a number of large RCTs have demonstrated the benefit of recent drugs for cardiovascular prevention,22, 23, 24, 25, 26, 27 but in some there was risk that cointerventions were unbalanced between study groups. We designed this systematic review to evaluate the quality of cointervention reporting in recently published RCTs with cardiovascular outcomes and to evaluate potential explanatory factors for reporting. We hypothesized that cointerventions would be more adequately reported in RCTs that were not fully blinded or otherwise at risk of bias because unbalanced cointerventions between trial arms may be more likely in these studies and could compromise their findings.

Methods

Selection of Articles

We searched MEDLINE and EMBASE for RCTs evaluating pharmacological interventions on binary cardiovascular outcomes (fatal and/or nonfatal myocardial infarction, fatal and/or nonfatal stroke, mortality as well as composite outcomes) published in the 5 general medical journals with the highest impact factors (New England Journal of Medicine, Lancet, Journal of the American Medical Association, British Medical Journal, and Annals of Internal Medicine) between 2011 and 2019 (see Table S1 for details of the search strategy). Our methods conform to the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) statement for reporting systematic reviews and meta‐analyses.28 The protocol is registered on PROSPERO (CRD42018106771). One reviewer (E.M.) screened all titles and abstracts, assessed the full text of eligible abstracts and articles, and identified relevant trials. Another investigator (L.A.) independently assessed the eligible abstracts. The data that support the findings of this study are available from the corresponding author upon request.

Assessment of Included RCTs

The following information was extracted: study design (superiority versus noninferiority/equivalence trials), number of patients, type of intervention and comparator, follow‐up duration, outcomes, information concerning methods of blinding of participants and personnel, blinding of outcome assessors, information about cointerventions, implementation of study treatment, adherence to study treatment, cross‐overs, statistical analysis conducted, and funding source (industry versus nonindustry). Available information on cointerventions, blinding of participants and/or personnel, adherence to study treatment, and statistical analysis was extracted independently by 2 reviewers (E.M., L.A.). All available information was extracted from the original trial reports, supplementary material, and protocols (if available).

Definition of Cointerventions and Quality of Their Reporting

Two investigators (E.M., L.A.) independently assessed the cointervention reporting. Because we included RCTs with cardiovascular outcomes, we considered potential cointerventions whose modification has been shown to decrease cardiovascular risk (Box 1).8, 29, 30, 31, 32, 33, 34 We defined cointerventions as concomitant medications (statins, antihypertensives, antiplatelets) over follow‐up (Box 1). In addition, diuretics, antidiabetics, and anticoagulants were also included in the definition of “cointervention” if these patients were included in the trials (ie, patients with heart failure, diabetics, or atrial fibrillation). We also defined 2 special categories of cointerventions in (1) RCTs where there was an index procedure after randomization, in which case, in addition to concomitant medications (statins, antihypertensives, antiplatelets) over follow‐up, procedural characteristics and periprocedural medications between the groups would also be cointerventions29, 30, 33 (Box S1), and (2) in RCTs with an index procedure after randomization but with a follow‐up of <1 month in which case cointerventions would be procedural characteristics and periprocedural medications without considering concomitant medications (statins, antihypertensives, antiplatelets; Box S1).29, 30, 33 Although advice for smoking, diet, and physical activity are also effective cointerventions, they are difficult to quantify, are rarely assessed in the original studies, and are therefore not evaluated in the present study.

Cointerventions are defined as the following: Special conditions:

To evaluate the reporting quality of cointerventions in each RCT, cointerventions were judged as adequately reported if the authors reported all cointerventions across trial arms (as described in Box 1) or if the authors explicitly stated that cointerventions did not differ between groups or gave indirect evidence that cointerventions did not differ between groups (eg, “there were no differences between groups in blood‐pressure or cholesterol levels”) or that there were no cointerventions. We judged cointerventions as inadequately reported if information in the article or supplement was incomplete (ie, partially reported) or missing (ie, not reported). Trials that did report cointerventions were classed as either “balanced” if there were similar levels of cointerventions between both groups or “unbalanced” and were judged by 2 reviewers (E.M., L.A.) independently. Disagreements were resolved by consensus in discussions that involved a third author (M.F.).

Assessment of Blinding and the risk of bias

We independently assessed the blinding of participants and/or personnel. We based our judgments about blinding participants and/or personnel on the Cochrane Collaboration risk of bias tool 2011 (Risk of bias 1.0) and instructions from Unverzagt et al (Table S2).35 We classified RCTs into having adequate blinding or inadequate blinding.

Two authors (E.M., L.A.) used the risk of bias 2.0 tool to independently assess risk of bias caused by deviations from the intended interventions (effect of adhering to treatment),13 and classified RCTs as at high risk of bias, some concerns, or at low risk of bias. For our analysis, we grouped together RCTs judged as “some concerns” and RCTs judged as “at high risk of bias” and classed them all as “at risk of bias.”

In general, there was good agreement regarding the previous classifications: Cohen's κ score for interobserver variability was 0.84 for the reporting of cointerventions, 0.87 for blinding participants and/or personnel, and 0.76 for the RoB 2.0 assessment.

Statistical Analysis

We used descriptive statistics. Comparisons between groups were conducted using a chi‐square test. We used univariable and multivariable logistic regressions to evaluate the association of reporting of cointerventions with blinding (adequately versus inadequately), risk of bias (trials at low risk of bias versus trials at risk of bias), funding (nonindustry funded versus industry funded), design (superiority versus noninferiority/equivalence), and duration of follow‐up (≤1 month versus >1 month). Finally, in an analysis that was not prespecified in the protocol, we looked at RCTs that adequately reported cointerventions and explored the aforementioned factors for their association with balanced cointerventions between treatment arms using univariable logistic regression. P values were 2‐sided and considered significant if P<0.05. For data management, analysis, and graphics, we used Stata version 15.0.

Results

General Characteristics of Included RCTs

The literature search identified 1625 potentially eligible reports. After screening titles and abstracts, we evaluated 149 full articles, of which 123 met the inclusion criteria (Figure S1). A detailed description of the excluded trials is provided in Table S3. Table S4 describes the main characteristics of the 123 included RCTs: 83 (67.5%) were published in the New England Journal of Medicine; 27 (21.9%) had a noninferiority/equivalence design; 94 (76.4%) were industry funded; 45 (36.6%) examined antithrombotics or anticoagulants; 16 (13.0%) involved antidiabetics; 14 (11.4%) involved antihypertensives; and 17 (13.8%) were lipid‐modifying agents (Table S4). The primary end points of all trials were composite end points (Table S5), and all of the trials had blinded adjudication committees.

Reporting of Cointerventions

As seen in Table, cointerventions were inadequately reported in 87 of 123 RCTs (70.7%), with 56 (45.5%) providing no information on cointerventions and 31 (25.2%) providing only partial information (Table). Table S5 provides detailed descriptions of the potential cointerventions in the protocols, all cointerventions reported and not reported, and the time points of reporting in each RCT. As seen in Table S6, the results remained similar in a stratified analysis based on medication category. Assessing potential cointerventions at regular intervals, usually at each visit and the last visit, was often included in study protocols (Table S5). Protocols were not available in only 7 RCTs.

Table 1

Reporting of Cointerventions (n=123)

Variable*	Sample, n (%)
Adequately reported	36 (29.3)
Balanced	31/36 (86.1)
Unbalanced	5/36 (13.9)
Partially reported	31 (25.2)
Balanced	26/31 (83.9)
Unbalanced	5/31 (16.1)
Not reported	56 (45.5)

“Adequately reported” indicates if cointerventions of interest were reported across trial arms; “partially reported” indicates if only part of the information was provided; “not reported” indicates if there was no reporting on potential cointerventions in the published article or the supplements (see Box 1).

The Reporting of Cointerventions in Relation to Quality of Blinding and Risk of Bias

A total of 71 (57.7%) RCTs adequately blinded participants and/or personnel, whereas 52 (42.3%) were inadequately blinded. Of the RCTs, 60 (48.8%) were at “low risk of bias”; 63 (51.2%) were “at risk of bias” (n=28, 22.8% as “some concerns”; n=35, 28.5% as “at high risk of bias”) because they deviated from planned interventions. Among the 52 trials with inadequate blinding of participants and/or personnel, 15 (28.9%) adequately reported cointerventions versus 21 (29.6%) in those with adequate blinding (P=0.93; Figure A). Among the 63 trials “at risk of bias,” 16 (25.4%) adequately reported cointerventions versus 20 (33.3%) in those “at low risk of bias” (P=0.33; FigureB).

Figure 1

Proportion of trials reporting cointerventions according to blinding and risk of bias.

A, Proportion of trials reporting cointerventions according to blinding of participants and/or personnel (n=123). For the analysis, we grouped together the trials with no information on cointerventions and partial information and defined them as “not adequately reported”; P=0.93 for the comparison between groups. B, Proportion of trials reporting cointerventions according to risk of bias attributed to deviation of intended interventions (n=123). For the analysis, we grouped (1) trials with some concerns and at high risk of bias and defined them as “at risk of bias” attributed to the deviation of intended interventions and (2) trials with no information on cointerventions and partial information and defined them as “not adequately reported”; P=0.33 for the comparison between groups.

Factors Associated With Adequately Reporting Cointerventions

As seen in Table S7, the odds ratio (OR) in the univariable analysis for adequately reporting cointerventions was 1.04 (95% CI, 0.47–2.27) comparing adequately versus inadequately blinded trials, 1.47 (95% CI, 0.67–3.21) comparing trials “at low risk of bias” versus trials “at risk of bias,” 2.06 (95% CI, 0.86–4.92) comparing non‐industry‐funded trials versus industry‐funded trials, 0.63 (95% CI, 0.26–1.55) comparing superiority trials versus noninferiority/equivalence trials, and 4.33 (95% CI, 1.63–11.52) comparing trials with a follow‐up ≤1 month versus >1 month (Table S7). In multivariable analysis, only a follow‐up of <1 month was associated with the adequate reporting of cointerventions (OR, 3.63; 95% CI, 1.21–10.91; Table S7).

Factors Associated With Balanced Cointerventions

As seen in Table, among the 36 RCTs that adequately reported cointerventions, cointerventions were balanced in 31 and unbalanced in 5 trials. All trials with unbalanced cointerventions were judged as inadequately blinded trials and were industry funded. As seen in Table S8, no other factor was associated with unbalanced cointerventions, even though the confidence intervals were large.

DISCUSSION

In this systematic review of recent RCTs on cardiovascular outcomes, more than two‐thirds of RCTs did not adequately report cointerventions. Reporting was not better among trials that were not fully blinded nor among RCTs at risk of bias in which the reporting of cointerventions would be particularly important to assess the validity of their results. Adequate reporting of cointerventions was more common in trials that followed patients for <1 month, perhaps because cointerventions are easier to assess over a short follow‐up.

Lack of blinding could lead to biased results through many different ways. Indeed, an association between lack of blinding and positive results has been shown, especially when the outcomes were subject to ascertainment bias, that is, not “hard” outcomes.36 RCTs with inadequate blinding seem particularly at risk for unbalanced cointerventions14 and reporting cointerventions is important because if they are unbalanced between treatment arms, they could introduce bias.6, 8, 11, 13 In an earlier systematic review of 12 complementary/alternative medicine RCTs, cointerventions (use of analgesics) were reported in 7 of these studies, and it was shown that not blinding participants was associated with an 1.55 increased risk (95% CI, 0.99–2.43) of receiving cointerventions.12 The lack of blinding and cointerventions could also explain the differences in the effect sizes between SPORTIF III (Stroke Prevention Using the Oral Direct Thrombin Inhibitor Ximelagatran in Patients With Atrial Fibrillation),21 an open‐label trial evaluating the effect of ximelagatran versus warfarin on strokes and systemic embolic events and SPORTIF V,20 a trial with otherwise similar design and end points with SPORTIF III, but double‐blinded. Although the potential risk factors were well balanced across the treatment arms within each trial, the effect sizes were remarkably different between the 2 trials: SPORTIF III, primary event rate 1.6% per year with ximelagatran and 2.3% per year with warfarin (relative risk [RR], 0.71; 95% CI, 0.48–1.07) versus SPORTIF V, primary event rate 1.6% with ximelagatran per year and 1.2% with warfarin per year (RR, 1.38; 95% CI, 0.91–2.10). Outcome assessments were blinded in both trials. Indeed, in a pooled analysis of the 2 trials,37 it was shown that the differences between the trials could be attributed to differences in cointerventions such as statins and differences in other risk factors (eg, hypertension), in addition to less variability in international normalized ratio (INR) control in SPORTIF V,37, 38 although ascertainment bias cannot be excluded. In our review, the reporting of cointerventions was scarce in both RCTs with adequate and inadequate blinding, and we found no association between blinding and the reporting of cointerventions. The reasons for this could be that the reporting of cointerventions in cardiovascular trials might have received less attention and/or be less standardized. Although the Consolidated Standards of Reporting Trials (CONSORT) statement recognizes that a lack of blinding may influence the use of cointerventions, subsequent reporting of cointerventions across groups is currently not mandatory.14 However, cointerventions are among the data required to be collected in a Cochrane systematic review.13, 39

In cardiovascular medicine, cointerventions may be particularly important because participants usually receive many different treatments that could reduce cardiovascular risk and change cardiovascular outcomes.6, 8 In the Women's Health Initiative, which examined the effect of hormone therapy on cardiovascular outcomes, the differential use of statins showed significantly different effects on coronary heart disease and stroke, confounding the results.6 A recently published RCT on the effects of coronary computer tomography on cardiovascular outcomes, which did not blind participants or personnel, found that the participants assigned to the intervention group were more likely to receive additional preventive treatments for cardiovascular disease (statins, antihypertensives, antiplatelets).40 In a double‐blind RCT designed to test the effects of fenofibrate versus placebo on hard cardiovascular end points, 17% of the participants on placebo were also treated with statins versus 8% in the fenofibrate group, which may have caused the results to be biased toward the null.10 In many cardiovascular trials, depending on the type of intervention, the presence of cointerventions may reflect the effectiveness of the study treatment that occurs in a real world instead of a perfect hypothetical study scenario, and the blinding of participants and/or personnel may not always be possible. Nevertheless, as cointerventions may lead to an overestimation of treatment effect, this is of particular concern when the results of an RCT are used for the registration of a new drug. In addition, in this systematic review, we included RCTs with pharmacological interventions (and not surgery or with devices), so that in these cases blinding is usually feasible.

This study has limitations. First, the results were limited to cardiovascular trials published in major medical journals, which represent a minority of published clinical research. However, trials published in journals with high impact factors usually do better in terms of the quality of reporting5 and previous methodological reviews have used the same design.41 Second, this study did not evaluate the reporting of cointerventions in medical fields other than cardiovascular. Third, the definition of which cointerventions should be reported is (to some extent) arbitrary. We proposed a definition (Box 1) that was easy to apply, reflected by a high interobserver agreement (Cohen's κ, 0.84).

Conclusions

More than two‐thirds of recent major cardiovascular trials did not adequately report cointerventions. The quality of reporting was not better among trials that were not fully blinded or at risk of bias. Our review highlights the need for more standardized, systematic reporting of cointerventions in cardiovascular trials.

Sources of Funding

The work of E. Moutzouri and M. Feller was partly supported by a grant from the Swiss National Science Foundation (320030_172676 to N. Rodondi). This work was also supported by the Swiss Society of General Internal Medicine (grant to N. Rodondi).

Disclosures

None.

Tables S1‐S8 Box S1 Figure S1 References 29, 30, and 35

Click here for additional data file.

The reporting was adequate if all of the following elements were reported and inadequate if 1 or more elements were missing* Cointerventions are defined as the following: Concomitant medications (statins, antihypertensives, antiplatelets) over follow‐up.31, 32, 34 Special conditions: If randomization before an index procedure‡ and follow‐up >1 month: concomitant medications (statins, antihypertensives, antiplatelets†) over follow‐up and procedural characteristics and periprocedural medications.29, 30, 33 If randomization before an index procedure‡ and follow‐up <1 month: procedural characteristics and periprocedural medications.29, 30, 33, §

Information could be anywhere in main article or supplements. Cointerventions should be summarized by percentages or absolute number across groups or the authors should state explicitly in the main text that cointerventions did not differ across the groups.

Includes others depending on the condition under study, for example, antidiabetics in trials that included patients with diabetes mellitus or diuretics if heart failure or anticoagulants in trials that included patients with atrial fibrillation; see the detailed descriptions in Table S3.

Index procedures included percutaneous coronary–angiography (n=18), cardiac surgery (n=5), surgery (n=2), and ablation (n=1); see the detailed description in Table S3.

For more detailed descriptions of procedural characteristics/periprocedural medications, see Box S1.