Identifying low-value clinical practices in critical care medicine: protocol for a scoping review.

INTRODUCTION: Reducing unnecessary, low-value clinical practice (ie, de-adoption) is key to improving value for money in healthcare, especially among patients admitted to intensive care units (ICUs) where resource consumption exceeds other medical and surgical populations. Research suggests that low-value clinical practices are common in medicine, however systematically and objectively identifying them is a widely cited barrier to de-adoption. We will conduct a scoping review to identify low-value clinical practices in adult critical care medicine that are candidates for de-adoption. METHODS AND ANALYSIS: We will systematically search the literature to identify all randomised controlled trials or systematic reviews that focus on diagnostic or therapeutic interventions in adult patients admitted to medical, surgical or specialty ICUs, and are published in 3 general medical journals with the highest impact factor (New England Journal of Medicine, The Lancet, Journal of the American Medical Association). 2 investigators will independently screen abstracts and full-text articles against inclusion criteria, and extract data from included citations. Included citations will be classified according to whether or not they represent a repeat examination of the given research question (ie, replication research), and whether the results are similar or contradictory to the original study. Studies with contradictory results will determine clinical practices that are candidates for de-adoption. ETHICS AND DISSEMINATION: Our scoping review will use robust methodology to systematically identify a list of clinical practices in adult critical care medicine with evidence supporting their de-adoption. In addition to adding to advancing the study of de-adoption, this review may also serve as the launching point for clinicians and researchers in critical care to begin reducing the number of low-value clinical practices. Dissemination of these results to relevant stakeholders will include tailored presentations at local, national and international meetings, and publication of a manuscript. Ethical approval is not required for this study. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/

Systematic and objective identification of low-value clinical practices in critical care, addresses one of the highly cited barriers to the de-adoption process (ie, objective identification of candidate practices).

Highly generalisable and replicable methodology may be employed to identify low-value clinical practices in most other healthcare disciplines.

Although this study will identify clinical practices in critical care that are candidates for de-adoption, the quality of the included studies will not be assessed given its scoping nature.

Introduction

Patients admitted to the intensive care unit (ICU) represent the sickest patients in the healthcare system. As a result of the severity of their illness, they are subject to twice as many interventions (diagnostic and therapeutic) as other hospitalised patients, and are frequently unable to participate in their own medical care.1 2 Data from the USA suggests that critical care medicine beds, occupancy rates and associated costs are increasing.3 4 In the USA for 2005, critical care accounted for 13.4% of all hospital costs, 4.1% of national health costs and 0.66% of the gross domestic product (GDP).3 In Canada for 2004, ICU costs were estimated to account for approximately $6 billion of the $39 billion spent on hospital services (0.5% of the 2004 GDP),5 and estimates from the Netherlands indicate ICU departments accounted for 20% of hospital budgets.6 Research suggests that up to 30% of interventions provided to patients admitted to acute care facilities may be unnecessary.7 8 Therefore, a considerable portion of this cost may be modifiable. Modifying this cost is important as projections suggest that with an ageing population, and our continuously advancing ability to treat critical illness, the demand for critical care services is likely to exceed the financial and human resource capacities of healthcare systems.4 Therefore, there is an urgent need for research to reduce any unnecessary use of these expensive and limited resources.

Research in critical care has traditionally focused on defining the pathophysiology of critical illness, evaluating the efficacy of different life-sustaining interventions (eg, high-frequency oscillatory ventilation vs conventional mechanical ventilation for patients with the acute respiratory distress syndrome (ARDS)9), or evaluating therapies targeted at specific diseases or syndromes (eg, recombinant human activated protein C for patients with severe sepsis10).11 Although this type of research contributes important information to the science that underpins critical care, there is a growing need for research that promotes the implementation of best practices into clinical care (ie, knowledge translation (KT)). This includes facilitating the adoption of effective clinical practices (eg, prone positioning in patients with severe ARDS12), and the de-adoption of those demonstrated to have no effect on patient outcomes (eg, use of hydroxyethyl starches for fluid resuscitation13).

Given existing fiscal climates, the notion of decreasing the use of unnecessary clinical practices is quickly permeating the medical literature, and has been recognised by professional societies, and governments as an integral component to the delivery of high-quality clinical care.14–17 Established clinical practices may be shown to be unnecessary or even harmful through publication of new scientific evidence. This process has recently been referred to as medical reversal.18 Medical reversal implies that patients who received the reversed practice may have experienced unnecessary harms associated with prior adoption, and may continue to be harmed until the reversed practice is de-adopted.18 19 Furthermore, the adoption of clinical practices that are later de-adopted imposes substantial inefficiencies on the healthcare system wherein resources that could have been dedicated to other purposes are instead devoted to a technology that is ineffective. Unfortunately, medical reversal and low-value clinical practices are common. A recent review of all research articles published in the New England Journal of Medicine between 2001 and 2010 found that 363 of 1344 (27%) original articles describing a medical practice examined the efficacy of an established practice (ie, replication research), among which 146 studies found evidence for practice reversal (40%).20

Although reversal appears to be common, and has negative consequences for patients, providers and healthcare systems, there is limited research to guide the systematic identification of such low-value clinical practices. While the publication of new scientific evidence can be used to identify reversal for an individual practice, this does not provide a rigorous means of capturing all potentially low-value clinical practices within a given discipline, especially those that became standard practice before the evidence-based medicine era, and thus are not supported by high-quality evidence. Nonetheless, other researchers and professional societies have developed lists of low-value clinical practices. Through a search of highly cited original clinical research citations, Ioannidis21 identified that 7 of 45 (16%) highly cited publications claiming an intervention to be effective were eventually contradicted through subsequent research. Through their health technology appraisal programme, the National Institute for Health and Care Excellence (NICE) created a programme to identify ineffective clinical practices.14 Their goal was to identify low-value interventions, which if stopped would save over £1 million per intervention. To date, their online database contains 1347 ‘do not do’ recommendations.22 In 2010, the Australian government introduced a programme for managing their Medicare Benefits Schedule (MBS), named the Comprehensive Management Framework (CMF).15 One of their key objectives was to create a systematic and transparent strategy to identify low-value clinical services whose inclusion in the MBS required review.15 Using an environmental scanning approach that involved triangulation of data from a search of peer-reviewed literature, targeted search of select databases, and opportunistic sampling of stakeholder groups, Elshaug et al15 identified 156 potentially unsafe and/or ineffective practices. Finally, the Choosing Wisely17 campaign along with its international extensions (eg, Choosing Wisely Canada23) recently published a number of specialty-specific ‘do not do’ lists, generated through combinations of literature searches, expert opinion and modified Delphi processes.

Although each of the aforementioned lists of low-value clinical practices identify opportunities from which to streamline the delivery of healthcare, they have limitations including heterogeneity in the methods used to identify the practices, which are generally difficult to replicate. In addition, aside from the Choosing Wisely programme, each approach to identifying low-value care was forced to restrict searches to a single source (eg, one major medical journal),14 20 or select a restricted subset of identified articles15 in order to manage the breadth of searching for practices across all medical and surgical specialties. Given the number of potentially low-value clinical practices that exist in medicine, rigorously identifying low-value clinical care is likely to require a specialty-specific approach. The recently published Choosing Wisely Top 5 list in Critical Care Medicine24 represents a starting point for critical care; however, the results of several recent studies in critical care suggest that this may under-represent the true incidence of medical reversal and opportunities for de-adoption within critical care.25–29 Identifying a comprehensive list of ineffective and/or harmful clinical practices in critical care medicine is important as critically ill patients are at increased risk of adverse events, consume considerable amounts of healthcare resources, and thus contribute a large portion to the costs of providing acute care medicine.1 3 5 30 Therefore, we will conduct a scoping review to systematically identify low-value clinical practices in adult critical care medicine wherein there is evidence for practice reversal, and thus candidacy for de-adoption.

Methods and analysis

Objectives

This is a protocol for a scoping review to identify clinical practices in adult critical care medicine that should be considered for de-adoption. Methods for inclusion and analysis of articles will be performed according to the recommendations from Arksey and O'Malley31 and updated by Levac et al.32 The main items in the Preferred Reporting Items for Systematic Review and Meta-analysis Protocols (PRISMA-P) guided the reporting of this protocol.33

The specific objectives of this scoping review are:

To determine the occurrence of replication research in the three highest impact general medical journals that publish critical care research.

To determine the occurrence of practice reversal in the three highest impact general medical journals that publish critical care research.

To determine clinical practices in critical care medicine that should be considered for de-adoption.

To identify opportunities for systematic reviews and empirical research that examine in greater detail the factors that may be associated with practice reversal (eg, quality of study methodology).

Studies will be classified as replication research if they represent retest replication, or approximate replication as defined by Curran et al34 (table 1). Retest replication refers to studies that repeat exactly the methodology from a previous study in another group of study participants in order to validate the original findings. Approximate replication refers to studies that repeat a previous study with only minor changes to the population, setting, treatment, outcomes and/or analyses.34 In addition to these definitions, we will also require the original and replicating studies to be of identical design (eg, both randomised controlled trials (RCTs)), and the replicating study must have a sample size that is within 10% of the original study. Replicating studies that contradict the results of the original study will define a list of reversed interventions.18 21 We will use the definition of de-adoption whereby it refers to the discontinuance of a clinical practice, where discontinuance is the decision to reject a practice after it was previously adopted.35 ICU will be defined as a distinct hospital specialty care unit staffed by specialised healthcare professionals where immediate and continuous life-sustaining treatment (eg, mechanical ventilation) is provided to hospitalised patients suffering from life-threatening conditions (eg, septic shock).36

Table 1

Key terms and definitions

Term	Definition
Replication research
Retest replication	Repeating exactly the methodology from an original study*in another group of study participants in order to validate the original findings34
Approximatereplication	Repeating an original study*with only minor changes to the population, setting, treatment, outcomes and/or analyses34
Similar results	Treatment effect from the replication study is in the same direction as the original study
Contradictory results	Treatment effect from the replication study changes direction relative to the original study
Practice reversal	Replication study demonstrates that a practice previously shown to be beneficial or with uncertain impact is ineffective or harmful18
Replacement	Replication study demonstrates that a practice previously felt to be harmful or ineffective is beneficial46
No change inpractice	Replication study demonstrates that a practice previously shown to be harmful is neither beneficial nor harmful (ie, ineffective)
De-adoption	The discontinuance of a clinical practice, where discontinuance is the decision to reject a practice after it was previously adopted35

*For randomised clinical trials: pilot trials do not count as original studies; sample size of the replication study must be within 10% of that of the original study. For systematic reviews/meta-analyses: replication study must include citations from primary research not included in the original study.

Eligibility criteria

At both level 1 and level 2 citation screening (outlined in detail below), citations will be included in this review if: (1) the study design is a RCT, or systematic review/meta-analysis of RCTs; (2) the study population includes adults (mean age ≥18 years) admitted to general medical-surgical or specialty ICUs (eg, burn, neurological, cardiac surgical); and (3) the study examines a diagnostic or therapeutic intervention. Articles will be excluded if: (1) the study participants are primarily admitted to coronary care units, defined as hospital specialty units where high-risk patients with a primary cardiac diagnosis (eg, acute myocardial infarction) are managed by a specialist cardiology team37; and (2) study design is other than a RCT, or systematic review/meta-analysis (eg, non-original research, cohort study, etc). The study will be restricted to RCTs and systematic reviews/meta-analyses as these represent the highest forms of research evidence, and should be the standard against which any clinical practice is evaluated before widespread implementation.

Search strategy

We will search MEDLINE, CENTRAL, and the American College of Physicians (ACP) Journal Club to identify all RCTs or systematic reviews of clinical trials published in the three general medical journals with the highest impact factor (New England Journal of Medicine, The Lancet, Journal of the American Medical Association) that focus on interventions in adult patients admitted to an ICU. The search will be restricted to these three general journals as they are more likely to publish research that changes clinical practice,38 and have a high readership among practitioners working in general multisystem or specialty ICUs.39 The MEDLINE and CENTRAL search will use exploded Medical Subject Heading (MeSH) terms, and text words that contain synonyms of critical care (eg, intensive care unit, intensive care, critical illness), and diseases/interventions specific to critical care (eg, septic shock, acute respiratory distress syndrome, continuous renal replacement therapy, etc). Search terms will be combined using Boolean logic, and will include wildcards to account for plural words and variations in spelling. The search strategy will also include previously validated study design filters that restrict the results to clinical trials, and systematic reviews.40 41 A similar search strategy will be developed for the ACP Journal Club, but will exclude the MeSH terms, and study design search filters. Finally, to ensure reproducibility, the search strategy will be validated by a medical librarian using the Peer Review of Electronic Search Strategies (PRESS) checklist.42 The proposed MEDLINE search strategy is outlined in the online supplementary appendix. All reference management will be performed in EndNote (V.X7, Thomson Reuters).

Study selection

Prior to the screening of titles and abstracts (level 1 screening), the citation screening form will be calibrated through pilot testing with a random sample of 50 citations from the literature search by two reviewers, independently. The inclusion/exclusion criteria will be serially revised until consistent citation selection is achieved (κ statistic, κ≥0.8).43 The same two reviewers will then independently screen citations for inclusion through a two-stage process. During level 1 screening, the titles and abstracts of all unique citations will be reviewed to determine which citations meet the inclusion criteria. The full text of any citation classified as include or unclear by either reviewer during level 1 screening will be reviewed to determine whether it meets the inclusion criteria (level 2 screening). Any eligibility disagreements encountered during level 1 or 2 screening will be resolved by consensus, or arbitration by a third reviewer. Agreement between reviewers at both stages will also be quantified using the κ statistic.43

Data extraction

In duplicate and independently, two reviewers will extract data from all included citations using a predesigned electronic form. Prior to extraction of data, the form will be pilot tested using a random sample of 10 of the included full-text citations. The data extraction form will be serially revised until data are reliably abstracted (κ≥0.8).43 The extracted data will generally pertain to methodology (eg, study design, number of centres), the study participants (eg, disease and/or syndrome under investigation), the intervention (eg, diagnostic vs therapeutic), the primary outcome (eg, mortality, length of stay, days free of a particular organ failure) and the magnitude of the intervention's effect (actual effect measure related to the primary outcome), and subsequent conclusions drawn by the authors. Any eligibility disagreements encountered during data extraction will be resolved by consensus, or arbitration by a third reviewer. Agreement between reviewers will be quantified using the κ statistic.43 All data will be managed using Microsoft Excel 2011 (Excel V.14.3.2, Microsoft Corp).

Analysis

In duplicate, and independently, two reviewers will classify included articles according to whether or not the study represents replication research (figure 1) using definitions outlined in table 1.34 Any eligibility disagreements encountered during data extraction will be resolved by consensus, or arbitration by a third reviewer. This classification will result in articles being labelled as non-replicated (eg, lung protective ventilation44), or replicated (eg, recombinant human activated protein C10). To prevent potential misclassification of non-replicated articles, we will review bibliographies of non-replicated studies, and conduct targeted searches within other high-impact general and specialty journals known to publish critical care research (Annals of Internal Medicine, British Medical Journal, American Journal of Respiratory and Critical Care Medicine, Chest, Critical Care Medicine, Intensive Care Medicine, and Critical Care).45 To ensure that non-replicated studies are not being replicated in an active, unpublished study, additional searches will be conducted in two international clinical trial registries (http://www.clinicaltrials.gov and http://www.controlled-trials.com).

Figure 1

Proposed classification scheme for included articles.

Studies classified as replication research will be further classified by comparing the results to those of the original study, and determining whether they are similar, or contradictory (figure 1 and table 1).21 This classification scheme will require a comparison of the direction of the effect of the intervention on the study's primary end point (ie, positive, neutral or negative) between the original and replication study. Replication studies will be classified as producing similar results as the original study when the treatment effect from the replication study is in the same direction as the first (eg, both beneficial). Replication studies will be classified as contradictory if the estimate of the treatment effect changes direction (eg, beneficial to harmful). Studies classified as replication research will identify practice reversal if the replication study demonstrates that a practice previously shown to be beneficial or with uncertain impact is either ineffective or harmful.18 Replication research will indicate replacement of current practice if a practice previously felt to be harmful or ineffective is shown to be beneficial.46

Extracted data (eg, number of therapeutic interventions vs number of diagnostic interventions) will be summarised using simple numerical counts and percentages where applicable. Similarly, classification of articles as described above will be summarised using simple numerical counts and percentages. All analyses will be conducted using Stata V.13.1 (Stata Corp, College Station, Texas, USA).

Ethics and dissemination

This scoping review will lay the foundation for a research programme that will seek to identify and facilitate the de-adoption of low-value clinical practices in adult critical care medicine. It will systematically and objectively identify reversed clinical practices in critical care, thereby addressing one of the frequently cited barriers to any programmatic approach to de-adoption (ie, identifying clinical practices with an empiric evidence base that supports practice reversal).14 47 48 In addition to the generation of a list of reversed practices in critical care, this review will identify opportunities for systematic reviews and empirical research that examine in greater detail the factors that may be associated with practice reversal (eg, quality of study methodology), so that the deleterious effects of early, widespread implementation of reversed technologies can be minimised through a more cautious, informed approach to adoption. All data will be obtained from publicly available databases; therefore, formal ethical approval will not be required.

There are two main end-of-synthesis outputs anticipated from this study. The main outputs include a list of clinical practices in adult critical care medicine that are ineffective and/or harmful, and a map of all potentially practice-changing randomised trials and systematic reviews in adult critical care that were published in the highest impact general medical and critical care specialty journals. These outputs will be relevant to several stakeholders including patients and their families, front-line healthcare workers that care for critically ill patients (nurses, respiratory therapists, physicians, pharmacists, physiotherapists), knowledge users and decision-makers (ICU patientcare managers and medical directors, hospital and regional decision-makers, health ministers), and researchers (critical care researchers and KT experts). Dissemination of these results to relevant stakeholders will include tailored presentations at local (departmental grand rounds), national (Knowledge Translation Canada Annual Scientific Sessions, Canadian Critical Care Trials Group quarterly meetings and the Critical Care Canada Annual Forum) and international (European Society of Intensive Care Medicine's Annual Congress) meetings. In addition, the results of this study will be published in a health services research or specialty journal whose readership will include several of the aforementioned stakeholders.

The results of this scoping review have the potential to influence the care of many patients, and advance the science of KT. We will use robust methodology to systematically identify low-value clinical practices in critical care medicine that are potential candidates for de-adoption. While successful, widespread de-adoption of these practices will require additional research to design and implement tailored KT interventions, the objective nature of this list is important as promoting the de-adoption of established practices is likely more difficult than promoting the adoption of new evidence-based practices.47 49 However, reducing the use of low-value care may release resources (financial and human) currently consumed by these practices and make them available for implementing high-value evidence-based interventions, and thus improve the overall quality of care. In addition to and as a consequence of systematically identifying low-value clinical practices, this review will also produce a list of clinical practices for which the evidence of benefit is either a RCT or systematic review/meta-analysis published in the three general medical journals with the highest impact factor, and thus should be considered to represent best practice. Finally given the use of robust, replicable methodology (ie, scoping review), with transparent reporting of our protocol and literature search, the methodology may be used within other healthcare disciplines, and thus improve the quality and value of care not only for critically ill patients, but for any patient cared for throughout the healthcare system.