The effect of blinding on estimates of mortality in randomised clinical trials of intensive care interventions: protocol for a systematic review and meta-analysis.

INTRODUCTION: Evidence exists that unblinded randomised clinical trials (RCTs) overestimate intervention effects compared with blinded RCTs. It has been suggested that this is less pronounced for objective (ie, not subject to interpretation) outcome measures, including mortality. This may not apply in the intensive care unit (ICU), as most deaths are preceded by decisions to withhold or withdraw treatments. Lack of blinding of physicians in RCTs of ICU interventions may potentially influence the decision towards a higher threshold for discontinuing treatment in patients who receive the investigational treatment and/or a lower threshold for discontinuing treatment in patients who receive the comparator (control). This may have important implications for patients, caregivers, researchers and society. Accordingly, we aim to assess whether lack of blinding affects mortality effect estimates in RCTs of ICU interventions. METHODS AND ANALYSIS: We will conduct a systematic review with meta-analyses and assess the effect of blinding versus no blinding on mortality effect estimates in RCTs of interventions used in adult ICU patients.We will systematically search the Cochrane Library for systematic reviews reporting mortality effect estimates of any intervention used in adult ICU patients which includes at least one RCT with 'low risk of bias' in the bias domains 'blinding of participants and personnel' and/or 'blinding of outcome assessment' and one RCT with 'unclear' or 'high risk of bias' in the same bias domain(s). For each intervention, we will compare summary mortality effect estimates in blinded versus unblinded trials. ETHICS AND DISSEMINATION: This research does not require ethical approval as we will use summary data from trials already approved by relevant ethical institutions. We will report the results in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) statement and submit the final paper to an international peer-reviewed journal. TRIAL REGISTRATION NUMBER: PROSPERO, registration number: CRD42017056212. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

This protocol for a systematic review has been prepared in accordance with the Cochrane Handbook and the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) statement.

A predefined statistical analysis plan, including sensitivity analyses, is presented.

Duplicate, independent literature search and data extraction will be performed.

Not all high-quality randomised clinical trials of interventions used in the intensive care unit may have been included in a Cochrane review.

‘High risk of bias’ in the domains ‘blinding of participants and personnel’ and ‘blinding of outcome assessment’ may be associated with ‘high risk of bias’ in other bias domains.

Introduction

Description of the condition

Randomised clinical trials (RCTs) and systematic reviews of these represent the gold standard for the assessment of benefits and harms of healthcare interventions.1 The validity of effect estimates of interventions in RCTs, and therefore in systematic reviews, depends on different aspects of the study design, each affecting the risk of bias.1 2 These aspects include random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective reporting of trial results and other risks of bias.1 2 Bias can be defined as systematic errors in results or inferences that favour one outcome over others.1 The impact of bias on trial results varies as some are insignificant while others may be substantial and drive most, if not all, of the observed intervention effect.2 The direction in which a certain bias operates is usually unknown and may differ between trials; in one trial, bias may lead to overestimation of the intervention effect whereas the same bias may lead to underestimation in another.2

Empirical evidence from meta-epidemiological studies based on collections of meta-analyses implies that RCTs with ‘high risk of bias’ due to study design flaws tend to exaggerate intervention effects compared with trials with ‘low risk of bias’.3–7 It seems that trials reporting inadequate or unclear randomisation sequence and/or allocation concealment exaggerate intervention effects estimates compared with trials reporting adequate measures in these design aspects.3 5–7 Similarly, RCTs with inadequate or unclear blinding also overestimate benefits of an intervention and underestimate harms, compared with RCTs with adequate blinding.3–7

Description of the intervention

Blinding is considered a critical component of modern medical research methodology and most researchers worldwide are familiar with the general concept. However, confusion exists regarding terminology and definitions.8 9 In RCTs, ‘blinding’ refers to techniques applied to keep participants, healthcare and trial personnel including physicians, investigators, data collectors, outcome assessors and statisticians unaware of the treatment allocation throughout the trial.1 8 10 11

Blinding can be applied in many different ways depending on the clinical setting and the intervention assessed. In trials assessing pharmacological interventions, blinding of participants and healthcare and trial personnel are often achieved with the use of placebo.8 Although blinding is notoriously harder to incorporate and maintain in non-pharmacological trials (eg, in surgical trials due to different incisions and scars between the groups),12 it is possible (eg, by using placebo dressings).13 In both pharmacological and non-pharmacological RCTs, blinding of statisticians can be accomplished using independent personnel (without knowledge of treatment allocation) and applying non-identifying terms to the groups before performing statistical analyses (eg, groups A and B).14

In some cases, blinding is not feasible or ethically justifiable. For instance, in trials comparing surgical with medicinal or no interventions, participants may sometimes only be blinded using ethically questionable methods like sham surgery.15 Furthermore, surgeons who perform the intervention are often impossible to blind. In these cases, blinding of other trial personnel (eg, personnel involved in postoperative care and outcome assessment) may reduce the risk of bias associated with inadequate blinding of trial participants and personnel.14

How the intervention might work

Blinding of participants and healthcare and trial personnel in RCTs aims to minimise the risk of performance bias and detection bias.2 Performance bias is the bias associated with unequal management of patients in the care provided (co-interventions) and unequal exposure to factors other than the intervention investigated.2 16 If unblinded, the participants’ expectations about a ‘new and exciting’ intervention or a ‘mediocre’ standard intervention may affect the psychological or physical response.11 In addition, the knowledge of the intervention assignment could be important for the participants’ corporation; participants who believe they have been assigned to an inferior intervention may be less likely to comply with the trial protocol, more likely to seek additional treatment outside the trial and more likely to leave the trial resulting in loss to follow-up (attrition bias).8 14 17 If trial personnel are unblinded, their inclinations about the investigated treatment may be directly transferred to the participants18 and knowledge of the intervention assignment may, whether conscious or unconscious, prompt differential use of co-interventions (supplemental care, interventions and diagnostic procedures) across the intervention groups.2 8 11

Blinding of data collectors and outcome assessors reduces the risk of detection bias. Detection bias or ascertainment bias refers to differential outcome assessment between the intervention groups.16 If data collectors and outcome assessors are unblinded, preconceived opinions regarding the intervention may influence the assessment; if outcome assessors believe the tested intervention is superior they may assess outcomes more generously in participants receiving that particular intervention and vice versa.19 Lack of blinding of statisticians may also introduce bias through selective use and reporting of statistical tests, and they should, therefore, remain blinded until all data analyses have been completed.14

Why it is important to do this review

The impact of blinding on effect estimates appears to depend on the outcome of interest. Subjective outcomes (eg, postoperative pain) seem highly vulnerable to the blinding status, whereas objective (ie, not subject to interpretation) outcomes including mortality may be less affected.3 6 However, this may not be true in the ICU since almost 80% of deaths are preceded by a decision to withhold or withdraw treatments.20 Lack of adequate blinding of healthcare personnel, including ICU physicians, in RCTs of intensive care interventions may potentially influence this decision in the direction of a higher threshold for discontinuing treatment in patients receiving the experimental intervention and/or a lower threshold for discontinuing treatment in patients receiving the comparator (control). Hence, there is a risk that inadequately blinded RCTs of ICU interventions overestimate benefit and underestimate harm even if mortality is the outcome.3

Recommendations for clinical practice are now commonly issued in accordance with the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology.21 The confidence in the effect estimate of any intervention is the most important determinant of the overall assessment of the quality of evidence.22 23 An incorrect assumption of ‘low risk of bias’ where the outcome in question is death maytherefore result in unwarranted recommendations of potentially harmful interventions.24 25 This may have important implications for patients, caregivers, researchers and society.

Objectives

We aim to assess the effect of blinding of participants, healthcare and trial personnel and outcome assessment on mortality effect estimates in RCTs of interventions used in adult ICU patients.

Research question: does blinding of participants, healthcare and trial personnel and outcome assessment affect mortality effect estimates in RCTs of interventions used in adult ICU patients compared with no blinding?

Methods and analysis

Study design

We will conduct a systematic review and meta-analyses in accordance with the recommendations by the Cochrane Collaboration where applicable.2

Study registration

This protocol has been prepared in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) guidelines (checklist is included as online supplementary file 1).26 27 The review has been registered in the Prospective Register of Systematic Reviews (PROSPERO); registration number CRD42017056212; available from http://www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42017056212.

Criteria for considering studies for this review

Types of studies

We will include parallel group RCTs as follows: first, we will identify relevant Cochrane reviews using the search strategy described below and assess them for eligibility (full text). All inclusion criteria must be fulfilled: (1) population: adult patients in the ICU, (2) intervention: any intervention used in the ICU, (3) outcome: mortality (reported as primary or secondary outcome) and (4) comprise at least one RCT with ‘low risk of bias’ in the bias domains ‘blinding of participants and personnel’ and/or ‘blinding of outcome assessment’ and one RCT assessing the same intervention with ‘unclear’ or ‘high risk of bias’ in the same bias domain(s), as assessed by the Cochrane review authors. If only one blinding domain has been assessed, the review must comprise at least one RCT with and without ‘low risk of bias’ in that domain. Second, RCTs included in the eligible Cochrane reviews will be used for the data synthesis (for study flow chart, see online  supplementary file 2).

We will use the same restrictions on language, publication status, time frame and inclusion of unpublished data as done in the individual Cochrane reviews.

Quasi-randomised trials, cluster randomised trials and cross-over trials will be excluded along with parallel group RCTs not fulfilling inclusion criteria 1–3 as outlined above.

Types of participants

Adult patients admitted to the ICU regardless of diagnosis or condition. We will exclude trials including children as defined in the Cochrane reviews.

Types of interventions

Intervention: RCTs assessing ICU interventions with adequate blinding defined as ‘low risk of bias’ in the bias domains ‘blinding of participants and personnel’ and ‘blinding of outcome assessment’, as assessed by the Cochrane review authors.

Control: RCTs assessing ICU interventions with inadequate blinding defined as ‘unclear’ or ‘high risk of bias’ in the bias domains ‘blinding of participants and personnel’ and/or ‘blinding of outcome assessment’, as assessed by the Cochrane review authors.

Mortality at longest follow-up, as reported in the primary publication.

In-hospital mortality.

In-ICU mortality.

Search methods for identification of studies

We will conduct an electronic search in the Cochrane Library (http://onlinelibrary.wiley.com/cochranelibrary/search) using the search string (title, abstract and keywords): ‘critically ill’ or ‘ICU’ or ‘intensive care’. The search will be limited to Cochrane reviews. No language or date limits will be imposed. Cochrane reviews are updated regularly to account of emerging evidence, and we will use the most recently updated review if more than one review on the same topic exists.

Before submission of the final draft of the manuscript to an international peer-reviewed journal, we will perform an updated search. If we identify new Cochrane reviews that comply with the inclusion criteria (see above), we will evaluate the RCTs included and incorporate the data into our analyses before submission of the final draft of the manuscript.

Data collection and analysis

Selection of studies

Two members (CTA and AG) of the review group will independently search the Cochrane Library and identify systematic reviews eligible for inclusion (see above). A third author (MHM) will arbitrate disagreements. If consensus is not achieved, we will contact study authors for clarification. We will use Covidence (https://www.covidence.org) to manage search records and document reasons for trial exclusion.

Data extraction and management

Duplicate, independent data extraction will be performed by two members (CTA and AG) of the review group using a predefined electronic case report form. Disagreements will be resolved through discussion with a third author (MHM). We will extract preaggregated data from the Cochrane reviews on trial characteristics (country, year of publication, population, intervention, comparator, outcome, single centre or multicentre, sample size), risk of bias (random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective reporting and other possible sources of bias including baseline imbalance, early stopping, bias due to vested financial interest and academic bias) as assessed by the Cochrane review authors and effect estimates (summary data for each intervention group (2×2 table): number of participants allocated to each intervention group and number of deaths in each group (mortality at longest follow-up, in-hospital mortality and in-ICU mortality)). If the data of interest is not reported (eg, assessment time point not specified) in the Cochrane review, we will consult the original publications in full text. We will not be blinded to author, institution or publication source of trials.

Assessment of risk of bias in included studies

We will use the risk of bias assessments performed by the Cochrane review authors according to the Cochrane Handbook2 for each included RCT in the following domains: random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective reporting and other bias including baseline imbalance, early stopping, bias due to vested financial interest and academic bias. For each intervention defined in the Cochrane reviews, the risk of bias assessments will be summarised in ‘risk of bias summary’ figures in trials with and without ‘low risk of bias’ in the bias domains ‘blinding of participants and personnel’ and ‘blinding of outcome assessment’ to allow an easy review of the distribution of risk of bias in the included bias domains between adequately and inadequately blinded trials.

Measures of treatment effect

For each ICU intervention defined and assessed in the eligible Cochrane reviews, the included RCTs will be divided into subgroups with and without ‘low risk of bias’ in the bias domains ‘blinding of participants and personnel’ and ‘blinding of outcome assessment’. We will provide effect estimates (risk ratios and 95% CI) on mortality for each RCT along with summary effect estimates for each of the subgroups. We will use χ2 test to test whether the subgroup summary effect estimates differ between adequately blinded (‘low risk of bias’ in both domains) versus unblinded trials (‘unclear’ or ‘high risk of bias’ in at least one domain); test of interaction. Results will be presented as forest plots for each intervention.

Dealing with missing data

We will not contact authors of RCTs for missing outcome data, as we expect that this have been done in the original Cochrane reviews. If the Cochrane reviews report additional outcome data (eg, if authors of a RCT were contacted for additional information), we will use and report these data to increase data completeness.

Assessment of heterogeneity

We will graphically evaluate forest plots for signs of heterogeneity and use D-squared and I-squared to describe statistical heterogeneity among the included trials. We will use both a fixed effect model and a random effects model, and will report the most conservative estimate.

Assessment of reporting bias

For each intervention (defined in the Cochrane reviews), we will assess reporting bias in blinded and unblinded trials separately by inspecting funnel plots. This assessment will only be feasible if 10 or more blinded or unblinded trials are included.

Data synthesis

We will use Review Manager (V.5.3) as statistical software and consider p values <0.10 as statistically significant (test of interaction).2 The strategy for the data synthesis is outlined above.

Subgroup analyses

None.

Sensitivity analysis

To assess the separate effects (if any) of risk of bias in the bias domains ‘blinding of participants and personnel’ and ‘blinding of outcome assessment’, we plan to conduct the following sensitivity analysis: (1) comparing mortality effect estimates in RCTs with ‘low risk of bias’ in the bias domain ‘blinding of participants and personnel’ and ‘unclear’ or ‘high risk of bias’ in ‘blinding of outcome assessment’ versus RCTs with ‘unclear’ or ‘high risk of bias’ in both domains (sensitivity analysis 1) and (2) comparing mortality effect estimates in RCTs with ‘low risk of bias’ in the bias domain ‘blinding of outcome assessment’ and ‘unclear’ or ‘high risk of bias’ in ‘blinding of participants and personnel’ versus RCTs with ‘unclear’ or ‘high risk of bias’ in both domains (sensitivity analysis 2).

Trials with adequate blinding (‘low risk of bias’ in the bias domains ‘blinding of participants and personnel’ and ‘blinding of outcome assessment’) may be more likely to be of better methodological quality, including an a priori higher likelihood of having ‘low risk of bias’ in the other five bias domains.3 An unequal distribution of risk of bias in other bias domains between blinded and unblinded trials (risk of bias heterogeneity) may increase the risk of confounded summary estimates. To evaluate the potential confounding by unequal distributions of risk of bias, we will exclude RCTs with ‘unclear’ or ‘high risk of bias’ in the bias domains ‘random sequence generation’ and/or “allocation concealment (sensitivity analysis 3), as risk of bias in these domains has been shown to influence intervention effect estimates markedly in large meta-epidemiologic studies.3 5–7

As separate analyses of individual interventions may decrease statistical power and increase the risk of accepting a false null-hypothesis, we also plan to conduct the following two sensitivity analyses: (1) compare mortality effect estimates of blinded versus unblinded RCTs by categorising the interventions into drug therapy, medical devices and management interventions (sensitivity analysis 4) and (2) compare mortality effect estimates of blinded versus unblinded RCTs pooling ALL interventions (sensitivity analysis 5).

Discussion

The proposed systematic review will provide important new information on whether blinding affects mortality effect estimates in RCTs of interventions used in adult ICU patients. The effects of ‘low risk of bias’ versus ‘unclear’ or ‘high risk of bias’ in the bias domains ‘blinding of trial participants and personnel’ and ‘blinding of outcome assessment’ will be evaluated combined, as well as separately, to further explore the impact of blinding different groups in RCTs of ICU interventions.

The most important limitation to this study is the risk of confounding due to unequal distribution of risk of bias in the other bias domains between blinded and unblinded trials. To account for this, we have planned a sensitivity analysis in which trials without ‘low risk of bias’ in the bias domains ‘random sequence generation’ and ‘allocation concealment’ are excluded as these two domains have been shown to affect intervention effect estimates in large meta-epidemiologic studies.3 5–7 Nonetheless, confounding may still arise from differences in the bias domains not accounted for. Another limitation may be the use of preaggregated data from the included Cochrane reviews without direct reference to the primary publications; however, we consider it sound to use data from Cochrane reviews, as we consider Cochrane reviews trustworthy. Cochrane review authors often obtain additional information regarding design, conduct and outcomes from the authors of the original papers; this may increase the accuracy and completeness of the data.

Despite limitations, we believe that the results of the proposed systematic review will have important implications for clinical practice guideline developers, as it is currently standard not to consider downgrading the quality of evidence for lack of blinding if mortality is the outcome of interest.28 Recommendations based on overly confident effect estimates may mislead ICU physicians and compromise patient safety. Therefore, we believe that there is an urgent need to address this potential issue.

Ethics and dissemination

Ethics and approvals

The proposed systematic review does not require ethical approval as it will not contain any data from any individual person.

Dissemination

We will report the results of the proposed systematic review in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) statement.29 Regardless of the findings, the results will be published in an international peer-reviewed journal and presented at relevant international conferences.