Establishing an Evidence Synthesis Capability For Psychological Health Topics in the Military Health System.

BACKGROUND: To promote evidence-based health care, clinical providers and decision makers rely on scientific evidence to inform best practices. Evidence synthesis (ES) is a key component of this process that serves to inform health care decisions by integrating and contextualizing research findings across studies.
OBJECTIVE: This paper describes the process of establishing an ES capability in the Military Health System dedicated to psychological health topics. RESEARCH DESIGNS: The goal of establishing the current ES capability was to facilitate evidence-based decision-making among clinicians, clinic managers, research funders, and policymakers, through the production and dissemination of trustworthy ES reports. We describe how we developed this capability, provide an overview of the types of evidence syntheses products we use to respond to different stakeholders, and detail the procedures established for selecting and prioritizing synthesis topics.
RESULTS: We report on the productivity, acceptability, and impact of our efforts. Our reports were used by a variety of stakeholders and working groups, briefed to major committees, included in official reports and policies, and cited in clinical practice guidelines and the peer-reviewed literature.
CONCLUSIONS: Our experiences thus far suggest that the current ES capability offers a needed service within our health system. Our framework may help inform other agencies interested in developing or sponsoring a similar capability.

Over the past several decades there has been a pronounced shift in health care towards an evidence-based practice that brings together research evidence, clinical expertise, and patient values to inform decision-making.1 Health system decision makers rely on scientific evidence to inform best practices and policies. However, rapidly accumulating research of varying quality creates challenges for busy decision makers who need to draw evidence-informed conclusions to support timely clinical and policy efforts. Furthermore, variation in how evidence is identified, summarized, and communicated has limited the trustworthiness of existing evidence reviews.2 For these reasons, rigorous methods to synthesize and summarize evidence have been developed and endorsed across clinical, research, and policy communities.3

Evidence synthesis (ES) is the use of transparent and systematic methodologies for summarizing a body of evidence on a given topic to help inform practice, policy, and research decisions. ES centers have become valuable resources that produce and disseminate evidence reviews across many health systems and scientific professional organizations. However, the Military Health System (MHS), one of the largest integrated health system in the United States that serves >10 million beneficiaries,4 previously lacked a dedicated ES center that could inform military stakeholders on the evidence relevant to the psychological health of service members and their families.

An ES center residing within the Department of Defense (DoD) has the potential to offer synthesis reviews tailored to the unique aspects of the health system, and within a flexible timeline to meet the requirements of different stakeholders. ES reports from external evidence centers are often applicable to MHS patients and DoD policymakers. However, these external centers are not wholly dedicated to responding to DoD needs. The MHS has a unique mission relative to Veterans Affairs (VA) and civilian health systems and serves a distinct patient population.4,5 External ES reports may not address some of the key questions that are pertinent to DoD stakeholders or take into account how study samples compare to MHS patients when drawing conclusions. In addition, DoD personnel frequently receive demands by congressional stakeholders, military officers, and other government leaders for evidence reports that need to be delivered on a shortened timeline and tailored to the specific needs of the MHS.

This paper describes the establishment of an ES capability in the MHS dedicated to psychological health topics of interest. The goal of developing this “in-house” capability was to facilitate the production and dissemination of trustworthy ES reports to clinicians, clinic managers, research funders, and policymakers, with the goal of promoting evidence-based decision-making. First, we review how the ES capability was established. Next, we describe a suite of ES products and the iterative procedures we used to select topics and to respond to different stakeholder needs. We evaluated the productivity, acceptability, and impact of this effort. Results from our experience have the potential to inform other health care systems interested in supporting an ES capability.

METHODS

The decision to establish an ES team corresponded with a larger effort within the Defense Health Agency to leverage existing published research, as well as available data and analytic capabilities, to directly inform health care policy and research funding decisions. As the scope of our agency mission evolved in recent years, a need was identified for a reliable capability that could provide timely and cost-effective evidence-based recommendations in response to emerging psychological health needs. We shifted existing research staff and resources away from long-term research trials and toward the analysis of existing research and data. We designed a new research mission that encompassed 2 complementary functions to provide a greater return on investment: an ES capability (described herein) and a health services research effort making use of large administrative and personnel datasets.

A project management plan was developed to support the establishment of the ES capability (Table 1). The plan defined the overall scope and objectives of the effort, key milestones and deliverables, and objective metrics to continually monitor progress. We also defined staffing and resource needs, anticipated potential risks, and devised mitigation strategies. To standardize key processes of the synthesis capability, a set of support documents and forms were developed (Table 2).

TABLE 1

Project Management Plan Components

TABLE 2

Evidence Synthesis Center Support Documents and Forms

Important stakeholder groups were identified and methods for soliciting their input were incorporated to ensure our products addressed relevant areas of need across the MHS. The stakeholders included policymakers who rely on rigorous reviews to inform practice and policy decisions, working groups that need to know the state of evidence on a given topic within a shortened time frame, military funding agencies charged with targeting research gaps through grant award announcements, and frontline providers and clinic leads who support care delivery. We also developed methods to stay abreast of relevant military policy directions, such as executive orders, the National Defense Authorization Act that directs the DoD budget, and other major initiatives that could impact care in the MHS.

Two conceptual models informed our ES strategy. We based our overall valuation of research on levels of evidence guidelines that identify well-conducted systematic reviews and rigorously-developed clinical practice guidelines (CPGs) as the highest levels of evidence.6,7 The inverted pyramid of the knowledge to action framework informed the scope of our mission.8 The knowledge to action model describes 2 general components of knowledge translation: knowledge creation and action. Our work primarily focused on the funneling component of knowledge creation in which we sought to synthesize original research or “first-generation evidence” into more convenient products for stakeholders. This included the creation of new knowledge (second-generation knowledge) in the form of systematic reviews and the funneling of existing knowledge through the development and dissemination of products and tools such as rapid reviews and evidence briefs (third-generation knowledge).

A key step was the establishment of a team with the requisite ES skills to provide a variety of higher-quality briefs and reports to support more definitive, evidence-based responses to requests. We invested in ongoing training, engaged in consultation with experts in the field, and identified relevant resources to support our efforts.9–11 We recruited an interdisciplinary team that could conduct and manage different forms of ES and developed a core team that engaged in an array of trainings in systematic review methodology: a lead to direct the program, an information specialist with access to full-text literature, a statistician skilled in meta-analysis and simulation, and several subject matter experts. The broader interdisciplinary team with a variety of training backgrounds (eg, epidemiology, psychology, psychiatry) supported the project as needed based on individual expertise and the aim of the synthesis. We also offered training opportunities to other staff with more diverse skills within the center that could support our efforts. The core team managed ES projects, with the interdisciplinary team of subject matter experts lending expertise to individual projects as needed.

Because ES can be resource-intensive, federal agencies often sponsor specialist Evidence-based Practice Centers (EPCs) to support this work. EPCs serve federal and nonprofit organizations exclusively, and their goal is to summarize evidence independently and objectively. To supplement the internal activities of the initiative, we leveraged an existing Federally Funded Research and Develop contract to sponsor work from the Southern California EPC located at RAND. This relationship allowed us to defer more complex review topics to this greater-resourced agency that comprises a multidisciplinary team of experts and who could also provide a greater level of objectivity to reviews in which there was the potential for conflicts of interest. All systematic review protocols and final reports produced by RAND were managed and reviewed by our core team.

ES Products

We provided a range of ES products to account for stakeholder needs, the state of the literature on the given topic, and the resources required and/or available. These distinct synthesis products are described below.

Systematic reviews

Systematic reviews employ a methodology that synthesizes all available and relevant empirical evidence about a specific health care question and often incorporate a meta-analysis or data simulation. Systematic review methodology used for all of our internal and sponsored reviews is outlined by the Institute of Medicine.3 Systematic reviews integrated within a decision-making framework can directly inform clinical practice, policy decisions, and future research.12 The systematic reviews we produced and sponsored through our program differed from other externally-produced systematic reviews by being more targeted and responsive to MHS health care needs.

Rapid reviews

Rapid reviews use a broad methodology that modifies aspects of systematic reviews to enable comprehensive, yet expeditious reviews and syntheses of research literature.13 Conducted on a shorter timeline that could range from hours to weeks, rapid reviews are initiated in direct response to stakeholder requests that require timely responses to help inform specific health care decisions.

Research gap analyses

Research gap analyses inform funding agencies about important gaps in health research that may benefit from more research funding. Main features of this methodology include reviewing important reports in a specific area of psychological health research, relying on subject matter experts to conduct comprehensive searches of the literature, and engaging stakeholders to prioritize research gaps (see Otto et al14 for a full description of the methodology).

Evidence briefs

Evidence briefs are concise, 2-page or 3-page evidence summaries, which provide overviews of existing clinical guidance and scientific evidence on treatments for psychological health conditions. Evidence briefs are targeted at busy clinicians who need authoritative but quickly digestible information on the merits of a very specific treatment topic.

Topic Selection and Dissemination

Our topic selection and dissemination have varied according to stakeholder need and the type of report indicated or requested (Fig. 1). Systematic review topics are selected by applying iterative approaches that take into account the existing literature and gaps identified in authoritative and scientific documents, knowledge of the needs and pertinent policy decisions in the health system, and stakeholder input (see Hempel & Belsher15 for an example of these processes). Systematic reviews are disseminated through 2 channels: (1) internally, in the form of dissemination packages to higher-level leadership and relevant workgroups who may benefit from an awareness of the research; and (2) publically, through the peer-review process and online reports. Dissemination packages circulated to DoD leadership include executive summaries, information papers, and proposed courses of action based on the results. Rapid reviews requested by stakeholders are disseminated through individual reports delivered directly to the requesters. In selecting annual topics for research gap analysis, different DoD stakeholders are consulted to prioritize the need for research on various psychological health topics. The gap analysis reports are disseminated to groups and agencies involved in the funding of psychological health research, as well as through peer-reviewed publications. Evidence brief topics are solicited using our online voting forum open to the public on the Defense Health Agency web site (www.pdhealth.mil/research/evidence-synthesis/evidence-briefs). New evidence briefs and voting topics are publicized through blog posts, social media posts, and e-mail distribution lists.

FIGURE 1

Evidence synthesis product flowchart. This figure illustrates the different components that inform the evidence synthesis topic and diagrams of how different synthesis reports are disseminated. The components within the boundary are considered internal to the Department of Defense, whereas those outside the boundary are considered external to the Department of Defense.

Outcomes

To evaluate the success of our efforts we collected different forms of data. To assess our productivity we counted the number of ES products we developed, disseminated, and published. To assess the acceptability of our services we considered the ongoing requests we received for reports and informally inquired with stakeholders whether they found our services useful. To assess impact, we noted whether our ES products were used for any health care decision related to policy or clinical guidelines and whether our reports were cited in the scientific literature.

RESULTS

Table 3 provides examples of recent evidence syntheses we developed and sponsored, with originating needs and associated dissemination strategies. Our reviews variously addressed needs identified by executive orders, National Defense Authorization requirements, congressional requests, and other efforts related to military psychological health. The reports were used by a variety of stakeholders and working groups, briefed to major DoD committees, included in official reports and policies, and cited in CPGs and the peer-reviewed literature.

TABLE 3

Examples of Recent ES Reports Produced by the ES Team

To date, we have produced or sponsored 25 systematic reviews. Seven systematic reviews27–33 supported the guideline preparation for the VA/DoD CPGs on substance use disorders (SUD) and major depressive disorder.34 Three systematic reviews were provided to the posttraumatic stress disorder (PTSD) and chronic pain CPG workgroup champions.35–37 Two reviews supported a VA directive on best practices for treating SUD and chronic pain.30,33 Six reviews supported a VA directive on best practices for complementary and integrative health for depression, chronic pain, and PTSD.27–29,35–37 Results from these systematic reviews were briefed to military leadership and workgroups on numerous occasions. Eleven peer-reviewed articles17,38–46 from these reviews have been published in the scientific literature and have been cited repeatedly.

Several rapid reviews on a range of psychological health topics were provided to internal stakeholder groups with the goal of informing a clinical or policy decision.23 The timelines for those reviews varied from 1 week to 1 month as dictated by stakeholder need. Stakeholders indicated that they found it convenient, time-saving, and otherwise efficient to obtain rapid reviews from a single, dedicated, and specialized resource.

We completed research gap analysis reports on PTSD, depression, SUD, and adjustment disorders in 2016, 2017, and 2018, respectively.14,24,25,47,48 The primary end-users of those reports were agencies charged with developing program announcements for new research. For example, the adjustment disorders research gaps report recently informed a program announcement for a major DoD funding opportunity for the fiscal year 2019. In addition, the results of the SUD research gaps initiative were solicited and shared as the DoD response for the 2017 National Institute on Drug Abuse/National Institutes of Health (NIDA/NIH) conference on opioid use and addiction.

More than 50 evidence briefs have been completed and published online across a range of psychological health conditions including generalized anxiety disorder, PTSD, major depressive disorder, alcohol use disorder and/or SUD, adjustment disorder, obsessive-compulsive disorder, and panic disorder, with new topics, added continuously.26 Briefs have been downloaded by the public >1600 times, and >140 suggestion for new evidence brief topics have been made online.

DISCUSSION

The goal of our effort was to establish an enduring ES center capable of providing high-quality synthesis reports to inform MHS clinical and policy decisions on psychological health topics. This initiative offered tailored and timely information about psychological health care topics that are important to providers and policymakers across our health care system. The ES center incorporated input from a broad range of stakeholders that encompassed the public, frontline providers, working group members, and DoD leadership. Our experiences suggest that this ES capability offered needed evidence syntheses tailored to the unique aspects of our stakeholders, and within a flexible timeline to meet diverse requests and requirements.

We believe the following points were demonstrated as validated by the ongoing review requests, feedback we received, and the use of our reports for various efforts: (1) it is feasible to establish an ES capability within the MHS provided the needed resources and support; (2) there is a need for an ES capability within the MHS based on our targeted reviews and feedback from stakeholders who request specialized reviews on heterogeneous topics within a shortened timeline; and (3) our ES products have had their intended impact on health care decisions as indicated by their uptake by workgroups and their application across practice and policy decisions.

The need for an ES capability within our health system is likely similar to other health care organizations and hospitals promoting evidence-based practice and policy. This paper may provide a model for other systems that have specific information needs. For example, our discovery that there is a need for different ES products beyond systematic reviews may translate to other health systems and inform their ES strategies. health care leaders may determine that investing in a team specialized in ES with dedicated time and familiarity with the health system can provide a valuable return on investment. Conversely, these agencies may determine that it is more cost-effective and efficient to contract out with an established evidence review center. Our efforts were largely driven by our ability to take advantage of publically available resources on ES methodologies, retraining of existing staff, and advocating for additional support. We benefitted from leadership support, sponsored support of an EPC, availability of training and resources, an enduring team that was not dependent on grant funding, and positioning within an agency charged with informing practice, policy, and research decisions.

The current framework has the potential to extend beyond psychological health topics. Because of our agency mission and in-house subject matter expertise, we focused exclusively on psychological health. However, relying on the general skill sets of a core team, this model has the potential to synthesize evidence on a broader range of subjects if a more diverse set of subject matter experts can be recruited to support specific projects. The VA ES Program has such a model in which national subject matter experts across the VA are recruited and contribute to a diverse array of ES topics.11 For this more comprehensive model to be successful, however, system-level changes need to be enacted to support the time commitment of experts across the health system.

Although several of our ES products have been effective in informing care and policy, there have been notable challenges and limitations that are worth recognizing. First, it is difficult to measure the impact. To explore the impact of the current initiative, we relied on a narrow definition (impact as determined when our products were used to inform practice or policy decisions). Notably, we did distinguish impact from publication, dissemination, and outreach which are commendable precursors to impact. However, it is hard to know where and when knowledge impacts clinical practice; it is important for organizations to think about the impact and return of investment. Second, our reach is limited to those stakeholders who are familiar with our efforts through proactive promotion or word of mouth. The DoD comprises a vast network of branches, divisions, and services: we have reached only a minority of stakeholders who we could benefit. Finally, the sustainability of this service is unclear. Frequent reorganizational alignments and funding changes typical to the military have the ability to greatly modify, or even terminate, our activities. Minimally, we hope the demonstrated value of our synthesis center will continue to facilitate support and investment in this capability.

CONCLUSIONS

The goal of our initiative was to establish an enduring capability that could produce and disseminate high-quality evidence syntheses to support evidence-based decision-making in the MHS. Our experiences thus far suggest that this ES capability offers a needed service within our health system and has demonstrated a potential impact on health care policies and practices. Our framework may provide a useful blueprint for other agencies and investigators interested in developing similar capabilities.