Assessing the Effectiveness of Complex Interventions to Meet the Needs of VA Stakeholders: Experience of the Department of Veterans Affairs Evidence Synthesis Program.

BACKGROUND: Complex health care interventions involve multiple distinct elements that contribute to their functioning. Conducting systematic reviews of complex interventions has substantial challenges. Although methodological guidance exists, less is known about the practical strategies and approaches undertaken by systematic review groups to navigate common challenges and enhance impacts of systematic review findings.
OBJECTIVES: Describe pragmatic approaches taken by Department of Veterans Affairs Evidence Synthesis Programs (VA ESP) in conducting systematic reviews of effectiveness and implementation barriers and facilitators for complex interventions to provide VA stakeholders with evidence to guide national health care practice and policy.
RESULTS: We describe 3 systematic reviews conducted by VA ESP teams to evaluate the evidence for complex health care interventions. We summarize key findings, implications for future research needs and policy, dissemination of findings, and approaches taken to address common challenges. The VA ESP experience adds to existing systematic review methods and provides a perspective on generating rigorous and relevant reviews of complex interventions.
CONCLUSIONS: Reviews of complex interventions often encounter challenges related to sources of variability in many dimensions, and lack of clarity and information in reporting of intervention elements, local context, and implementation factors. Evidence synthesis teams should work closely with stakeholders to understand their needs, synthesize and interpret results in meaningful ways, and explore implications that are most relevant for day-to-day clinical practice and operational decisions of learning health care systems. More evaluation of the impact of systematic reviews may improve uptake of findings from future reviews and enhance translation of evidence into practice and policy.

Complex health care interventions are common and involve multiple distinct elements that contribute to the functioning of the interventions. The Agency for Healthcare Research and Quality (AHRQ) Complex Interventions Steering Committee characterized them as having intervention complexity (ie, multiple elements) and pathway complexity (ie, multiple causal pathways with possible mediators and moderators of effect); they also often involve other sources of complexity (eg, population and context complexity).1,2 The AHRQ Committee proposed adaptations to traditional systematic review approaches, particularly in defining the scope and analytic frameworks and selecting strategies for qualitative and quantitative synthesis.1–6

Key challenges in evidence synthesis of complex interventions arise primarily from variation across interventions (putatively within the same or similar category) and the diverse contexts in which such interventions are implemented and evaluated. Elements of the intervention and the rationale for selection and combination of elements may not be clearly or consistently defined by research studies; similar reporting issues affect characteristics of the local context that may be important for interpreting results. This lack of clarity in reporting contributes substantially to difficulty in evaluating key issues, such as intervention fidelity, the additional value of each element (or synergy between elements), and generalizability of findings. However, aside from challenges related to reporting of complex interventions, there are more analytic decisions in systematic reviews because of the level of complexity along multiple dimensions. For example, review teams often need to judge which combination of elements is sufficient to qualify as the complex intervention of interest. Therefore, heterogeneity of the primary research literature has led systematic reviews of similar interventions to use a wide range of approaches to frame scope and key questions, apply different selection criteria, and make different decisions about analytic strategies to evaluate and summarize intervention effects.

Although the AHRQ Committee and the Cochrane Effective Practice and Organisation of Care team have proposed guidelines for conducting and reporting systematic reviews of complex interventions,1–9 less is known about the practical strategies and approaches undertaken by systematic review groups to navigate common challenges. The Department of Veterans Affairs Evidence Synthesis Program (VA ESP) supports the needs of VA as a learning health care system by addressing important questions posed by VA operational leadership and policymakers. As VA strives to translate the best research evidence into key decisions and policy initiatives to transform health care for Veterans, there has been a growing need to evaluate complex interventions in terms of effectiveness, harms, and costs, as well as implementation barriers and facilitators. Over the past 2 years, VA ESP conducted 33 evidence reports; 12 (36%) addressed complex interventions through systematic reviews using meta-analyses, umbrella reviews (ie, reviews of reviews), and scoping reviews (eg, evidence maps). VA ESP experience with complex interventions provides an opportunity to highlight approaches taken to address key challenges that occur to generate relevant, rigorous reviews.

We describe 3 VA ESP reviews on a diverse set of complex interventions in surgical care, palliative care, and transitional care. These reports were conducted by 3 different VA ESP teams, used different review methodologies, and selected various approaches to synthesize findings. In addition to noting key report findings, including those related to implementation and policy implications, we focus on how each review team addressed a number of important challenges. Rather than propose new methodological standards or provide a prescriptive road map, our objective is to summarize the variety of approaches chosen to produce high-quality systematic reviews to support future research and clinical and policy decisions. We hope that these examples will stimulate others to report on their experience with complex interventions, and thus, together advance the methodology for evidence synthesis of complex interventions.

VA ESP EXAMPLES OF COMPLEX INTERVENTION SYSTEMATIC REVIEWS

Enhanced Recovery After Surgery Programs for Colorectal Surgery10

Intervention Topic and Key Questions

An Enhanced Recovery After Surgery (ERAS) program is a multimodal perioperative management pathway intended to reduce hospital length of stay and health care costs without increasing complications or hospital readmissions compared with standard care. Most ERAS protocols involve multiple elements for each of 4 stages of perioperative care (Table 1). The VA National Director of Surgery requested VA ESP evaluate key questions related to the comparative effectiveness and harms of ERAS interventions overall and for certain ERAS elements, clinical conditions, and surgical approaches. Additional key questions addressed the barriers and facilitators for implementation of ERAS interventions. The review was intended to guide implementation of a standardized ERAS protocol for colorectal surgery across VA. Reducing costs and improving health outcomes for this frequently performed surgery would provide substantial benefits.

TABLE 1

Intervention Topics and Key Features of Reviews on Complex Interventions

State of the Literature, Key Findings, and Implications for Research and Policy

The Minneapolis VA ESP team conducted a systematic review using quantitative meta-analysis and qualitative summaries to evaluate 25 eligible controlled clinical trials (13 used random assignment) involving elective open and/or laparoscopic colorectal surgery in adults (Table 2). ERAS programs reduced hospital length of stay and perioperative morbidity, whereas mortality, hospital readmissions, and surgical site infections were similar between ERAS and usual care. Importantly, ERAS programs and adherence varied widely and there was no evidence that directly assessed the additional benefits and harms of more versus less intensive ERAS programs. Implementation facilitators and barriers were addressed by 10 published reports of patient and provider interviews. Only 1 non-US study assessed cost despite reports on implementation barriers showing this as an important factor.

TABLE 2

Key Findings and Implications for Research and Policy

The review suggested that nationwide implementation of an ERAS protocol should be approached through careful evaluation and iterative refinement. Detailed information should be gathered regarding the elements of existing ERAS and standard perioperative care protocols within VA, and implementation factors, including roles of key champions, assessment of adherence, and current resource needs and costs. Key informant interviews should specifically address implementation barriers and facilitators at multiple levels. In future research, a trial with sequential subtraction of ERAS protocol elements and evaluation of outcomes may help identify essential elements. The lack of VA studies was also a specific gap in evidence and an area for future research.

The National Surgery Office (NSO) subsequently convened a multidisciplinary stakeholder workgroup to develop ERAS program implementation guidance; the VA ESP team also participated. NSO guidance was directed to VA physician and nursing surgical leadership to help ensure that surgery programs develop and implement an ERAS protocol on the basis of the current evidence. ERAS criteria were outlined for pre-, intra-, and postoperative settings, and included explanations of implementation criteria, level of evidence supporting the criteria, and impact on patient-centered outcomes. A “strategies for implementation sheet” was developed to explain ERAS and steps for implementation including securing buy-in from identified critical stakeholders, guiding a multidisciplinary team to champion ERAS implementation, and designing facility-specific approaches. The ESP team also presented at the NSO Leadership Conference and published a peer-reviewed manuscript to disseminate findings.

Challenges and Chosen Analytic Approach

The VA ESP team faced several challenges including a lack of standard terminology for ERAS programs (eg, multimodal optimization, multimodal program, accelerated pathway, and accelerated protocol). Although guidelines exist for ERAS protocols for colorectal surgery, variation in the number and definition of elements, as well as criteria for adherence, contributed to difficulties in identifying eligible studies. To overcome challenges related to the lack of standardized terminology, the ESP team used broad search terms across multiple electronic databases and conducted hand searching of reference lists to locate additional references. To examine comparative effectiveness across ERAS programs, including whether outcomes varied by number and types of various elements, the ESP team developed a set of potentially recommended ERAS components by merging elements from 2 recently published guidelines13,14 and charted the ERAS elements specified for each of the trials. Most ERAS protocols contained fewer than half of the recommended components and about one third of the standard care protocols contained several ERAS components. As studies were not designed to directly answer questions about the effects or value of specific elements, the ESP team addressed variation in interventions more broadly by examining more versus less intensive ERAS programs. As recommended,7,9 additional analyses looked at whether results varied by ERAS protocols with greater differentiation from standard care on the basis of a set of core elements. Because there was limited reporting of the local context for interventions, and no studies were conducted at VA, it was difficult to judge applicability related to differences in health care systems. To guide future implementation, the ESP team adapted a prior organizational framework15 to categorize barriers and facilitators according to staff, organizational, and patient factors noting their importance in policy and practice implementation.

Integrated Outpatient Palliative Care in Oncology11

More than 40,000 US Veterans are diagnosed with advanced cancer annually. Various models of integration between palliative and oncology care have been described, including co-rounding for hospitalized patients, embedded or co-located outpatient clinical services, and stand-alone clinics or services. Multiple VA stakeholders, including the Association of VA Hematology/Oncology, Palliative Care Research Committee, Geriatrics and Extended Care, and Hospice and Palliative Care Program, requested an evidence review on integrated outpatient palliative care in oncology (Table 1). In this review, the Durham VA ESP team addressed key questions related to: effects of palliative care initiated “upstream” and integrated with oncology care; intervention characteristics associated with greater patient and caregiver benefits; and barriers to implementation in VA settings.

Nine randomized controlled trials (including 2 in VA) provided limited information about integrated palliative and oncology care for patients with symptomatic or advanced cancer (Table 2). Four observational studies conducted in the VA provided information on implementation barriers and facilitators. Trials did not describe specific elements of integration between palliative and oncology care. Qualitative summaries did not show that level of integration was related to overall intervention effects, including short-term quality of life.

At the time of the ESP review, the VA did not have guidelines for standard integration of outpatient palliative and oncology care. The VA ESP review found improvements in quality of life with integrated care, thereby supporting current medical professional society guidelines that recommend routine integration of palliative and oncology care. The report also highlighted an important role for consistent palliative care integration with routine outpatient oncology care, for consideration in VA policies and clinical practices. The Palliative Care stakeholder group presented ESP review findings at VA and broader national meetings.

To address considerable gaps in evidence for some policy-relevant outcomes and critical intervention elements, the ESP review called for more clearly defined palliative care intervention elements and integration characteristics. This will allow for a more precise understanding of the impact of integrated palliative and oncology care on outcomes. The review also recommended that future studies adopt a standard set of key outcomes and recruit a more racially and ethnically diverse population.

As it was difficult to define both palliative care and integrated palliative-oncology care, the ESP team undertook extensive literature searching which required multiple search strategies and databases, refining search terms, conducting manual searches of cited references, and contacting content experts. To determine eligibility, the ESP team relied on US guideline definitions for intervention characteristics; applying those definitions to the literature was difficult. In addition, defining integrated care according to “co-location,” as originally proposed, was both conceptually inadequate and too narrow. The ESP team, therefore, adapted and applied a definition from behavioral health integration and considered the timing of service delivery. To target outcomes of greatest clinical utility, the ESP team conducted an outcome prioritization process with topic nominators and the technical expert panel. To identify intervention elements and integration characteristics (timing and co-location) associated with greater effects, the ESP team developed a theory-driven, standardized classification of each study intervention and conducted quantitative (subgroup analyses of moderator variables) and qualitative analysis. Because classification of integration could not occur on the basis of published information alone, the ESP team contacted authors for additional data; 3 studies still could not be classified because of missing information. A potential limitation of this approach was that it relied on study authors’ retrospective reports about aspects of integration that were used to classify each study.

Transitions of Care From Hospital to Home: An Overview of Systematic Reviews12

In 2011, the total cost of 30-day readmissions to the VA was $1.2 billion. In addition to being costly, hospital readmission is an indicator of acute care quality for VA and non-VA facilities. Health care systems, including the VA, have focused on efforts to reduce hospital readmissions through improving care transitions. Transitional care interventions aim to deliver coordinated and continuous care during key transitions. However, these interventions can be resource intensive and often include many different elements. For health systems that endeavor to improve transitional care, it is a challenge to define the specific elements they should adopt, as well as which patient populations they should target. The VA Primary Care Services and Office of Quality, Safety, and Value requested an evidence report on the effects of transitional care interventions, primarily on reducing hospital readmissions and mortality (Table 1). The Portland Oregon ESP team evaluated the effects of different transitional care interventions on hospital readmissions and mortality. They also attempted to address whether the effects of the interventions varied for different settings or in different patient populations. The ESP report was intended to inform a VA State of the Art Conference (SOTA) on Care Coordination.

The ESP report identified 17 eligible systematic reviews and from the included reviews, 9 primary studies conducted in VA settings (Table 2). There was no clear pattern of effect differences between studies conducted in VA and non-VA settings. Qualitative summaries were prepared for each included systematic review and 2 common approaches were found among included reviews: those that focused on different transitional care interventions in a specific patient population and reviews that focused on 1 type of intervention in different patient populations.

The lack of evidence supporting specific interventions and the likelihood that gaps in transitional care vary across health care facilities and patient populations made it difficult to recommend a specific transitional care protocol. Moreover, at least 2 large reviews showed that more recently published studies were less likely to show positive effects, suggesting that implementation of potentially resource-intensive transitional care interventions in the current health care context may not reap commensurate benefits. Rather, the ESP report proposed health care facilities and systems use a standardized process for assessing the current state of transitional care and identifying gaps in care.

The ESP report was part of references and materials for the national VA SOTA conference on care coordination. At this conference, invited experts sought to develop a research agenda focused on improvements and innovations in care coordination, and to identify evidence-based recommendations for VA policymakers. Conference participants disseminated their recommendations through briefings to VA leadership, targeted virtual seminars, and a peer-reviewed journal publication.

Sources of variability for transitional care interventions existed in many dimensions (eg, training and roles of interventionist, intensity and timing of contacts, setting, and patient population). The volume of original research on this topic was very large and numerous systematic reviews already existed. However, none adequately met the needs of VA stakeholders. To efficiently and effectively evaluate the existing evidence and provide practical information in a timely manner, the Portland ESP team conducted an umbrella review (ie, review of systematic reviews) focusing on hospital-to-home transitional care interventions. An important limitation to this approach is the inability to delve into primary research studies, and directly assess internal validity and generalizability of effects. Another challenge was identifying essential intervention elements or characteristics. For example, is the type of patient contact (home visits vs. telephone follow-up) critical for effective transitional care after hospital discharge? If so, for which patients, and delivered at what time after discharge? The ESP team dedicated considerable effort during topic refinement to estimate the range of variability across key dimensions and define core elements of transitional care. They explored existing taxonomies for complex interventions in general and for transitional care coordination specifically, but none adequately addressed the key questions. The ESP team subsequently developed a transitional care organizational map that outlined core processes and key staff members to guide transitional care improvements. This served as a valuable tool and key deliverable to assist VA stakeholders in developing, implementing, and evaluating transitional care programs.

Because of the heterogeneity across multiple dimensions, the ESP team conducted a qualitative analysis of themes across reviews. They first summarized key findings from each review in a single document, and then reviewers independently identified themes from this document. Commonly identified themes were further summarized and used to develop policy implications. The ESP team found it difficult to use established criteria to formally rate the strength of evidence for each conclusion, and few of the included systematic reviews reported the strength of evidence for their findings. The report noted these limitations and urged caution because of indirect comparisons and variability among studies. Policy and research implications were informed by the evidence and discussions with VA stakeholders on current transitional care pathways.

DISCUSSION AND LESSONS LEARNED

We summarized 3 ESP reports conducted to meet the needs of different VA stakeholders and highlighted pragmatic approaches taken to address several common challenges in the evaluation of complex interventions. We also described dissemination and important impacts of review findings. The complex health care interventions examined in these reviews all involved variability across intervention elements, participant characteristics, and settings, among other dimensions. Such interventions may face unique implementation barriers and facilitators, and relevant implementation factors for each element (or combination of elements) likely vary across participant groups and settings. Yet clinicians and operational policymakers need to decide if and how these complex interventions should be implemented in specific settings.

Challenges and approaches to conducting reviews of complex intervention reviews have been previously described.1–9,16–18 We build off this prior work and note common themes from VA ESP reviews (Table 3). Challenges included several related to heterogeneity in the evidence base and the lack of studies specifically evaluating comparative effectiveness across different versions of complex interventions or in different settings. Lack of clarity and information on intervention elements, local context, and implementation factors all contributed to difficulties in identifying, selecting, and interpreting the evidence. Additional time and resources were often needed to refine the scope and develop conceptual frameworks, improve literature searches, identify relevant studies, extract results, and apply findings to address stakeholder needs. Creating more detailed conceptual models was useful for both ESP teams and VA stakeholders; these models helped identify the information needed to inform clinical and management decisions, thus guiding interpretation of results and formulation of recommendations for policy and research. To better understand complex interventions and supplement published research studies evaluating complex interventions, ESP teams also sought additional sources of information, including qualitative studies on implementation barriers and facilitators, and queries to authors about intervention elements.

TABLE 3

Approaches to Common Challenges

Qualitative approaches were often selected to synthesize the evidence, sometimes in combination with quantitative methods, to provide a greater understanding of findings. For example, results were presented in tables and figures stratified by complex intervention categories (eg, classifications on the basis of number and types of elements). Even with better reported (and larger) studies, it remains unlikely that there will be sufficient data for standard quantitative techniques because of a large number of possible combinations for different versions of complex interventions and varying populations and settings. Therefore, qualitative approaches should be considered and for certain topics or questions, these may be more valuable than traditional meta-analyses. Lastly, collaborative relationships between ESP review teams and VA stakeholders were critical for guiding the selection of synthesis approaches and successful dissemination of findings (also enabled by established communication channels like virtual seminars and management briefs).

Evidence synthesis teams should work closely with stakeholders and understand their needs, synthesize and interpret results in meaningful ways, and explore implications that are most relevant for day-to-day clinical practice and operational decisions of learning health care systems. This can include “informed speculation” whereby the review team applies existing evidence to questions of “how” and “why” to better understand the applicability and what local adaptations may be beneficial. These interpretations of the evidence often require an in-depth understanding of the tradeoffs that stakeholders are willing to accept regarding evidence certainty for interventions and outcomes as well as the policy imperatives to act. Evidence reviews of complex interventions should help inform the selection of implementation approaches and planning for evaluation of key outcomes, including health benefits and harms, resource use, and unintended effects.

Increasingly, stakeholders acknowledge the importance of complex interventions for addressing key gaps and problems in complex health care systems. Evidence reviews of complex interventions represent an opportunity for translating current research evidence into real-world solutions. Rigorous reviews of complex interventions should provide accurate, credible, and useable information that enhance implementation of findings, or de-implementation when this reduces waste and redirects resources to more effective/efficient strategies. Evidence synthesis is critical for highlighting the limitations of existing evidence, particularly regarding implementation and unintended consequences. Unfortunately, there is limited information on the downstream results of most evidence synthesis reports on complex interventions: if report findings affected practice and policy; if interventions were implemented, were they adapted to address local factors; what were the results of implementations, including resource use, cost, and patient and provider outcomes; and were identified gaps in evidence addressed with future research including quality improvement initiatives. More evaluation and follow-up of the impact of evidence reviews may enable improved uptake of findings from future reviews, thereby enhancing translation of evidence into practice and policy.