Systematic review of economic analyses in patient safety: a protocol designed to measure development in the scope and quality of evidence.

INTRODUCTION: Recent avoidable failures in patient care highlight the ongoing need for evidence to support improvements in patient safety. According to the most recent reviews, there is a dearth of economic evidence related to patient safety. These reviews characterise an evidence gap in terms of the scope and quality of evidence available to support resource allocation decisions. This protocol is designed to update and improve on the reviews previously conducted to determine the extent of methodological progress in economic analyses in patient safety. METHODS AND ANALYSIS: A broad search strategy with two core themes for original research (excluding opinion pieces and systematic reviews) in 'patient safety' and 'economic analyses' has been developed. Medline, Econlit and National Health Service Economic Evaluation Database bibliographic databases will be searched from January 2007 using a combination of medical subject headings terms and research-derived search terms (see table 1). The method is informed by previous reviews on this topic, published in 2012. Screening, risk of bias assessment (using the Cochrane collaboration tool) and economic evaluation quality assessment (using the Drummond checklist) will be conducted by two independent reviewers, with arbitration by a third reviewer as needed. Studies with a low risk of bias will be assessed using the Drummond checklist. High-quality economic evaluations are those that score >20/35. A qualitative synthesis of evidence will be performed using a data collection tool to capture the study design(s) employed, population(s), setting(s), disease area(s), intervention(s) and outcome(s) studied. Methodological quality scores will be compared with previous reviews where possible. Effect size(s) and estimate uncertainty will be captured and used in a quantitative synthesis of high-quality evidence, where possible. ETHICS AND DISSEMINATION: Formal ethical approval is not required as primary data will not be collected. The results will be disseminated through a peer-reviewed publication, presentations and social media. TRIAL REGISTRATION NUMBER: CRD42017057853. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

A strength of the proposed study is the application of transparent and replicable procedures for searching health economic literature.

The breadth of the search strategy is a strength as it lends to generating findings that are translational.

A weakness of this study is the close but imperfect replication of previous reviews. This limits the comparability of this study’s findings with others’, which also limits our interpretation of the maturation of evidence.

Considering previous findings, a large sample size for statistical pooling of effect size(s) within subgroups is not anticipated. This may limit the strength of this protocol as a means of producing a quantitative synthesis.

Introduction

After recent focusing events in which avoidable failures in healthcare were identified, patient safety remains high on the clinical and policy agenda in health systems. The burden of lapses in safety is high in developed health systems. In the USA, the Office of the Inspector General (OIG) estimated an adverse event (AE) rate of 13.5% for hospitalised Medicare beneficiaries, amounting to approximately 180 000 additional deaths per annum.1 For the UK, the rate is estimated to be 10.8%, although there are limitations to this calculation and the methodology differs to that used by the US OIG.2 3 According to the most recent systematic reviews of economic analyses in patient safety, there is a dearth of high-quality evidence on the economic impact of avoidable harm and the interventions that aim to improve safety. As such, there is limited evidence to support resource allocation decisions in this area. Meltzer summarised the state of evidence in a 2012 editorial in which economic evaluation in patient safety was described as a ‘neglected necessity’.4

In light of findings by Etchells et al5 and De Rezende et al,6 there is a need to conduct a new review of economic evaluations to measure recent progress in this area. A central question is whether the research community has produced more high-quality evidence since these publications. Implicitly, this question should be approached by (1) conducting a before and after comparison of studies using the same, valid quality assessment instrument and (2) this is a prerequisite for a potential metasynthesis of recent (last 10 years) evidence.

Etchells et al5 restricted their search to comparative evaluations of 15 patient safety target conditions and 6 improvement strategies using a Medline and Agency for Healthcare Research & Quality (AHRQ) Patient Safety Network search between 2000 and 2011. Studies that did not analyse an intervention with a patient safety target or were not in the acute setting were excluded. Studies were also excluded if evidence of effectiveness was poor, according to the Cochrane collaboration risk of bias tool,7 or not cited. Those studies that were included were quality assessed using the Drummond checklist,8 which was completed for each study by two independent reviewers. Consensus was achieved based on a five-point discrepancy between assessments scores. The findings from this review show the paucity of high-quality comparative research.

From this review, only five studies were considered high quality: pharmacist-led medication reconciliation dominated a strategy of no reconciliation9; additional erythropoietin to reduce transfusion-related AEs were evaluated against the standard of care in another study10; use of chlorohexidine gluconate to reduce catheter-related infections was found to dominate povidone-iodine use11; the Keystone intensive care unit patient safety programme was evaluated using effectiveness data related to the reduction in central line-associated bloodstream infections and was found to dominate standard of care12; lastly, strategies to reduce retained surgical foreign bodies were compared and standard counting was found most likely to dominate a no counting strategy.13

De Rezende et al6 used Medline, National Health Service (NHS) Economic Evaluation Database (EED), and Econlit databases to search for English and French language economic evaluations from 2000 to 2010. Their focus was on characterising the types of AEs studied and methods used to estimate costs and cost-effectiveness. Two independent reviewers used a common template to assess AE type, evaluation method, perspective, population studied, cost components, data sources, cost calculation methods, outcome measures and results obtained. As such, burden studies, cost-effectiveness studies and intervention cost analyses were identified. A quality assessment tool was conceived and used to determine studies’ methodological quality and discrepancies were discussed by the four authors.

Using a qualitative synthesis, this review found that most studies considered healthcare-associated infections and/or infection control activities and medication-related AEs. Three economic burden/cost of-illness studies, five complete economic evaluations and two intervention cost studies were described in detail by the research group because they were considered high quality (a published instrument was not applied for quality assessment). Economic evaluations of hospital deep cleaning programmes, staff vaccination against nosocomial pertussis, three medication-related AE reduction strategies, a national methicillin-resistant Staphylococcus aureus (MRSA) reduction policy and the introduction of electronic medical records to reduce prescription errors were identified. The findings from this review provide insight on the scope of economic analyses performed in this area.

Taken together, Etchells et al5 and De Rezende et al6characterise the literature from 2000 to 2011 with a combination of scope and quality appraisal. There is a renewed evidence gap in economic evaluations of patient safety for three reasons: (1) the reviews by these research groups are no longer recent; (2) the breadth of safety priorities and interventions and the depth of objective quality assessment in these reviews have historical and scientific limitations and (3) considerable resources continue to be invested in patient safety without a renewed synthesis of evidence to inform decision-making.

The proposed review will address these three concerns by expanding the scope of the review conducted by Etchells et al5 and raising the scientific rigour compared with the De Rezende et al6 review. This approach will be essential to determine the extent to which methodological progress has been made since Meltzer’s (2012)4 editorial. To ensure that the quality assessment is valid and reliable, high-quality studies published after 2007 that were also identified in previous reviews will be included for quality assessment in this study.

Research aims and objectives

This study aims to (1) assess methodological progress in economic studies in patient safety and (2) provide a quantitative summary and synthesis of the most recent evidence in this area.

Characterise and describe the types of economic analyses performed in patient safety since 2011;

Assess the quality of empirical (descriptive and comparative/experimental) economic analyses in patient safety;

Compare the scope and quality of economic evidence in patient safety between 2007 and 2011 to the end date of the literature search.

Methods and analysis

The scope of this study is driven by the combined aspects of the Etchells et al5 and De Rezende et al6 studies. The approach is derived using resources that enhance methodological transparency and improve the systematic nature of the search strategy and evidence synthesis. These resources are described for both the economic and medical aspects of this review. Medline, Econlit and NHS EED bibliographic databases will be searched from January 2007 using a combination of medical subject headings (MeSH) terms and research-derived search terms. This approach is considered efficient for searching health economic literature.14 The primary outcomes of interest in this review are economic endpoints in patient safety research.

The overarching approach to this study is summarised in the prospective Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) (see online supplementary 1). A completed PRISMA Protocols checklist can be found in online supplementary 2. The following research stages will be followed:

Title and abstract searches using MeSH and researcher-derived search terms (provided in table 1);

Table 1

Search themes (facets) and terms derived for each theme relating to economic analyses in patient safety

Patient safety

Economic analyses

patient harm/or patient safety/or safety management/or accidental falls/or accident prevention/or infection control/or antisepsis/or asepsis/or blood safety/or sterilisation/or disinfection/or sanitation/or decontamination/or medical errors/or diagnostic errors/or medication errors/or inappropriate prescribing/or medication reconciliation/or near miss, healthcare/or Venous Thromboembolism/or Pulmonary Embolism/or Pressure Ulcer/ (nosocomial or Patient safety or iatrogenic or ((Adverse or sentinel) adj (outcome* or event* or drug event*)) or Incident*one or Error*one or (Near adj miss*2) or Close call* or (Never adj event*) or complicat* or (Mis diagnos* or misdiagnos*) or (missed adj diagnosis) or (Patient adj accident*1) or (((Inappropriate*two or unsafe or wrong) adj (transfer or discharge*)) or treatment) or ((Healthcare or healthcare) adj (acquired or associated) adj infection*1) or ((Wound or Surgical site) adj infection*) or (Fail* adj2 discontinu*five adj treatment*) or Infusion injur* or ((Missing or retained) adj (needle* or swab or instrument or sponge)) or Unplanned return to theatre or ((Maternal or mother) adj (death* or mortality or readmission)) or Venous thromboembolism* or Pulmonary embolism* or Pressure ulcer*one or ((wrong adj site surgery) or impact or prosthesis or (route adj1 administration adj2 medic*)) or (ABO-incompatible adj (blood or organ*))).ti,ab. 1 or 2

costs and cost analysis/or cost allocation/or cost-benefit analysis/or cost of illness/or health care costs/or health expenditures/or economics, hospital/or hospital charges/or hospital costs/or economics, medical/or fees, medical/ ((economic burden) adj2 (disease* or illness*)).ti,ab. OR (cost* adj2 illness*).ti,ab. OR (cost*3).ti,ab. OR ((healthcare or healthcare) adj2 (cost*3)).ti,ab. OR ((cost) adj (analy*three or evaluation)).ti,ab. OR (health resource utili#ation).ti,ab. OR (cost adj1 effective*).ti,ab. OR (cost adj1 benefit*).ti,ab. OR (cost adj utilit*).ti,ab. OR ((economic) adj (evaluation* or analysis or burden or impact)).ti,ab. OR (compar*five adj econom* adj analys*2).ti,ab. OR (incremental adj analysis).ti,ab. OR (net adj1 benefit*).ti,ab. OR ((financ*3) adj (analy*three or evaluation or manag*5)).ti,ab. (health econom*).ti,ab. OR (value adj2 money).ti,ab. OR (expenditure*).ti,ab. OR (financ*).ti,ab. OR (fiscal).ti,ab. OR (funding).ti,ab. OR (pharmacoeconomic*).ti,ab. OR (socioeconomic*).ti,ab. OR (pric*3).ti,ab. OR (fee*).ti,ab. OR (health priorit*).ti,ab. OR (resource allocation).ti,ab. OR (economic* adj aspect).ti,ab. OR (efficien*2).ti,ab. OR (technical efficien*2).ti,ab. OR (allocative efficien*2).ti,ab. OR (productivit*3).ti,ab. OR (organi?ational adj efficien*2).ti,ab. OR (economic adj endpoint).ti,ab. OR (budget*).ti,ab. 1 or 2 or 3

Combine database searches, deduplication using a prespecified procedure.

Title and abstract screening using prespecified inclusion and exclusion criteria;

Full-text and reference list screening according to pre-specified inclusion and exclusion criteria.

Risk of bias assessment and Drummond checklist assessment of economic evaluation quality;

Data extraction in accordance with research objectives.

Evidence synthesis of scope and quality of evidence;

Presentation and publication of findings;

Assessment of evidence for use in subgroup meta-analyses.

Eligibility criteria

The systematic review does not focus on a particular population, disease area or health system setting as it intends to characterise the scope of all economic evaluations in patient safety. The search criteria are shaped around two core themes for original research in ‘patient safety’ and ‘economic analyses’. We define patient safety as ‘The avoidance, prevention and amelioration of adverse outcomes or injuries stemming from the process of healthcare’.15 Economic analyses in this study broadly refer to cost-of-illness, cost-effectiveness, cost utility, cost benefit, cost consequence, cost minimisation and cost analysis studies.

Search terms will be restricted to title and abstract only at the screening stage. The time frame for study selection is decided by the completion dates of previous reviews, therefore all studies published since January 2007. This also allows for overlap between the current study and previous reviews, ensuring appropriate articles are included.

Participants/population: individuals of any age and gender, all index admissions and in all health system settings who are at risk of experiencing a safety-related AEs;

Intervention(s): this review is deliberately broad and is designed to include descriptive and intervention studies. Interventions refer to any care process that intends to reduce the incidence of safety-related AEs. These may include new drugs (or drug-presentations), procedures, guidelines and policies that are produced to improve patient safety;

Comparators: all comparison groups;

Outcomes: all economic and clinical outcomes related to patient safety will be considered.

Participants/population: articles that report populations not at risk of iatrogenic harm;

Intervention(s): any studies reporting on non-safety-related interventions that is, those that do not explicitly intend to reduce the rates of harm;

Comparators: no specific exclusion;

Outcomes: studies with outcomes that are not considered a consequence of patient safety interventions will be disregarded;

Timing: studies published during or before December 2006;

Studies that do not report economic data.

Non-English language articles;

Opinion: letters, editorials and commentaries;

Systematic reviews and book chapters;

Partial or non-peer reviewed evidence: conference abstracts, study protocols.

Other general exclusion criteria

The review will consider both experimental and epidemiological study designs that are incorporated with economic analyses. We will therefore include randomised controlled trials, non-randomised controlled trials, quasi-experimental before and after studies, prospective and retrospective cohort studies, case–control studies and analytical cross-sectional studies. Only peer-reviewed publications of original research will be considered, and these criteria are satisfied by applying the Neyt and Chalon optimised procedure for searching economic evaluations in Medline.16 This procedure may be used exclusively or in combination with the research group’s search terms.

Screening

Initial search result titles, abstracts and index or keyword terms will be screened. Studies will be screened by two reviewers acting independently. Reference lists of all identified reports and articles will be searched for other relevant studies.

After title and abstract screening, full-text reviews will be conducted.

Studies that do not investigate economic outcomes;

Studies that do not investigate patient safety.

Exclusion criterion

The two reviewers will extract relevant data on study design, funding, population(s), setting(s), disease area(s), intervention(s), outcome(s) and effect size(s). These data will be used to for the quality assessment of the review.

Quality assessment

Two reviewers will independently assess the risk of bias in studies using the Cochrane collaboration risk of bias tool.7 This tool will be applied to assess the quality of effectiveness data reported and used for cost-effectiveness analyses. It will also be used to assess epidemiological and observational findings. Overall, the quality assessment will consider the following study-level aspects:

Randomisation sequence generation: was the allocation sequence generated appropriately?

Treatment allocation concealment: were study participants, clinicians and other healthcare or research staff unaware of the intervention(s) given at the enrolment stage?

Blinding: were the personnel assessing outcomes and analysing data blinded to the intervention allocated during the trial?

Completeness of outcome data: were participant exclusions, attrition and incomplete outcome data adequately reported in the published manuscript?

Selective outcome reporting: is there evidence of selective outcome reporting and could this have biased the study results?

Other sources of bias: was the study apparently free of any other high-risk sources of bias?

Disagreements between the review authors over the risk of bias studies will be resolved by discussion, with arbitration by a third reviewer where necessary. For studies considered to have a low risk of bias and that also report economic evidence, the Drummond checklist for assessing the methodology quality of economic evaluations will be applied. Two independent reviewers will score these (out of 35). Scores will be reviewed by a second reviewer and any disagreements in score of >5 will be resolved with arbitration by a third reviewer.

Data synthesis

A descriptive (qualitative) analysis of the scope of economic evidence in patient safety will be presented. Consistent with De Rezende et al,6 the types of economic analyses will be presented as studies of either economic burden/cost of illness or cost analysis of interventions or full economic evaluations (encompassing cost and effectiveness data).

The populations (by demographic), settings (primary, secondary, community or social care), diseases areas (problems of circulation, cancers and tumours, mental disorders, musculoskeletal system and connective tissue) and interventions (safety-related AE that is targeted—either infection-related, medication AE, surgical safety, hospital accidents) of all economic analyses will be presented.

A synthesis of quality scores for risk of bias and economic evaluation quality will be given. Effect size(s) and cost-effectiveness results of high-quality papers—those scoring above 20 on the Drummond checklist—will be reported. This cut-off score is consistent with that used by Etchells et al.5 Sensitivity analysis results will also be given, if reported. The sensitivity of review findings to stratification of evidence quality will also be explored.

A quantitative synthesis will be performed if necessary data are available. High-quality findings from comparative analyses will be compared with findings from Etchells et al5 and De Rezende et al’s6 reviews. High-quality findings identified in these reviews and that were published after 2006 will be included for quantitative synthesis. Effectiveness estimates will be grouped by the type of safety-related AE that are the target of interventions identified in the review (for example, nosocomial infections). Estimates will then be pooled for statistical meta-analysis. Heterogeneity will be assessed statistically using an appropriate test statistic (such as χ2). Effect sizes expressed as OR (for categorical data) and weighted mean differences (for continuous data) and their 95% CIs will be calculated for analysis. Where statistical pooling is not possible, findings will be tabulated for comparison.

Metabiases will be determined using the risk of bias and Drummond checklist scores. These results will be explored for high, moderate, low and very low-quality evidence. Systematic biases associated with the quality of evidence will be presented.

Amendments

Amendments to this protocol will be captured and documented with reference to saved searches and analysis methods, which will be recorded in bibliographic databases (OVID), Endnote and Excel templates for data collection and synthesis. Deviations to this protocol will be documented in a research manuscript that will be submitted for peer-reviewed publication.

Discussion

In light of the findings in previous reviews of economic analyses in patient safety, this review is a timely update on a high-priority topic in healthcare. A strength of the proposed study is the application of transparent procedures for searching health economic literature. These procedures are (1) application of the Neyt and Chalonprocedure for economic literature searches in MEDLINE, which is a published procedure,16 (2) full publication of both MeSH and research-group derived search terms associated with each theme/facet, (3) use of a reproducible deduplication procedure for Endnote article management software. Second, we aim to address two aspects—scope and quality—to determine how research in this area has matured since 2011. This is important because many health systems are facing efficiency and sustainability challenges, which require robust economic evidence to support decision-making in patient safety. This review aims to support targeted decision-making in the medical profession that is, synthesising the evidence by population, setting and disease area. The breadth of the search strategy should therefore improve the translational aspects of the anticipated research findings. A weakness of this study is the close, but imperfect replication of previous reviews. This limits the comparability of this study’s findings with others’, which also limits our interpretation of the maturation of evidence. In addition, considering previous findings, a large sample size for statistical pooling of effect size(s) within subgroups is not anticipated. This may limit the strength of this protocol as a means of producing a quantitative synthesis.