"Implementation Is so Difficult": Survey of National Learning Health System Decision-makers Identifies Need For Implementation Information in Evidence Reviews.

BACKGROUND: Evidence use within learning health care systems can improve patient health outcomes. Embedded in the Veterans Health Administration (VHA) since 2007, the Veterans Affairs Evidence Synthesis Program (ESP) provides tailored evidence synthesis services to support VHA's learning health care system goals. As part of the ESP's ongoing quality improvement efforts, we have been surveying our users since 2016.
METHODS: We reviewed data from a survey of end-users received between September 5, 2016, and June 10, 2019. The survey assessed: (1) nature of decision-making needs; (2) actions resulting from the report's findings; (3) implementation timeframe; and (4) overall perception of report content.
RESULTS: Feedback was received from 66 of the 138 operational partners requesting ESP products during the fiscal year 2015 through 2018. Requests commonly informed clinical guidance (58%), identified future research needs (58%), and determined VHA-specific implementation strategy (47%). A total of 91% of responders used reports, typically within 3 months after completion (82%). Use was typically for VHA publications and/or presentations to inform VHA policy or guidance (26%), to inform intervention/strategy adoption decisions (23%) and for medical device and therapy procurement decisions (21%). Over half (53%) of respondents indicated that it would be useful for ESP reports to include more guidance on implementing findings.
CONCLUSION: Our survey of learning health system decision-makers' actual patterns and timing of evidence use provides valuable new information that can further support development of other health system and evidence producer partnerships and identifies key needs for better supporting health systems' uptake of evidence.

BACKGROUND

A landmark 2012 brief by the Institute of Medicine defined a “Learning Health System” as one designed to leverage the best available evidence in the collaborative health care choices of each patient and provider; to drive the process of discovery as a natural outgrowth of patient care; and to ensure innovation, quality, safety, and value in health care.1 This dynamic relies on the generation, utilization, and renewal of high-quality evidence syntheses to ensure that health care decisions are responsive to an environment of continually emerging evidence, knowledge, and innovation. However, there is often concern that systematic evidence reviews are not actionable due to a lack of high-quality evidence, long production times, and scopes that are not relevant to policy or health care decisions.2–5

Little is known about how health systems access and use evidence to inform decision-making. Preliminary information from a convenience sample of 4 evidence synthesis programs that serve US health systems indicates that evidence needs vary widely both within and between health systems.6 Evidence syntheses from these programs catered to the specific needs and strengths of the individual organizations. For example, reports are done by the Veterans Health Administration (VHA) often focused on care processes or policies while those done for the ECRI Institute focused on specific technologies. However, evidence gaps remain, including the need for a better understanding of how health systems actually use evidence syntheses and how these usages impact their learning health care journey.

The Veterans Affairs (VA) Evidence Synthesis Program (ESP)—one of the 4 aforementioned evidence synthesis programs—is a national multicenter research program embedded within the VA health system since 2007. Initially funded as a 2-site pilot building on the expertise of the Agency for Healthcare Research and Quality Evidence-based Practice Center Program, in 2009, the program expanded to 4 ESP Centers and a Coordinating Center, with each Center led by a VHA clinician-researcher with systematic review expertise. The ESP program is guided by operations’ information needs, and actively engages clinical health system requestors (operational partners) throughout the evidence synthesis process (Fig. 1).

FIGURE 1

Operational partner engagement in evidence synthesis process. The partnership between the health system requestor and the evidence synthesis review team is continual, engaging throughout the evidence synthesis process. Feasibility and operational needs drive report scope; all reports prioritize evidence specifically in military/Veteran populations when available. By encouraging fluid and unfettered engagement with the requestor, Evidence Synthesis Program (ESP) investigators can quickly develop a shared understanding of the context and evidence needs and can better shape a review that will inform policy and clinical decision-making.

Continuous quality improvement (QI) is a primary focus for the ESP. One of our ongoing QI initiatives includes surveying our operational partners to better understand their information needs, actions resulting from findings, implementation timeframe, perception of report content, and suggestions for improvement. Therefore, the objective of this article is to build on previous work6,7 describing the evidence needs of learning health systems by reporting our evaluation of a health system’s actual utilization of a broad range of 45 reviews completed between 2016 and 2018.

METHODS

We developed a 43-question survey instrument (ESP OP Survey Instrument, Supplemental Digital Content 1, http://links.lww.com/MLR/B846) based on ESP’s strategic goals, which were informed by the QUERI Strategic Plan, the VHA Strategic Plan (Blueprint for Excellence), and recommendations from a 2012 external evaluation of the program examining participant satisfaction.8 The survey was further refined based on feedback from the directors of the ESP Centers as well as VHA health system and research leadership. It consists of closed- and open-ended prompts that cover 4 primary domains: (1) nature of decision-making needs, (2) actions resulting from the report’s findings, (3) implementation timeframe, and (4) overall perception of report content. After extensive pilot testing, the survey was launched in 2016 in an online format (REDCap, Vanderbilt University, Nashville, TN) that automatically sent an invitation to each operational partner for a given report 6 months after report completion. The invitation consisted of an e-mail with a unique survey link, which associated end-user responses with the report in which they participated. Respondents were given 4 weeks to respond, with automatic reminders sent weekly until they responded, or the 4-week period ended. Survey completion was entirely voluntary, and we provided no incentives. We calculated usage frequencies from closed-end responses using R, version 3.6.0 (April 26, 2019). We used an inductive approach to conduct a thematic analysis of the open-ended responses. The survey was administratively reviewed and approved as QI based on VHA policy.9

RESULTS

Survey invitations were sent to all 138 operational partners who requested the 45 ESP products produced during the fiscal year 2015 through 2018. We received feedback from 48% (N=66) of those invited to respond. These responses covered 38 of the 45 completed reports, of which 23 (61%) were systematic reviews, 11 (29%) rapid reviews, and 4 (11%) evidence maps (Fig. 2). Respondents included nonacademic Subject Matter Experts (SME) with VHA operations decision-making authority, including National Program Offices, Central Office, and Chief Consultants (N=39, 59%); academic researchers charged with leading system-wide health/QI efforts (no VHA operations decision-making authority) (N=22, 33%); and nonacademic Health System Managers with VHA operations decision-making authority, such as Veterans Integrated Service Network (VISN) Directors or Chief Medical Officers (N=5, 8%). Because invitations were sent to all operational partners involved with each report, the number of responses varies per the report, ranging from 1 to 10 with a median of 3.

FIGURE 2

Survey respondents, nonrespondents, response rates, products and topic areas. VA indicates Veterans Affairs; VISN, Veterans Integrated Service Network.

ESP reports were most commonly requested to help develop clinical guidance (58%), identify future research needs (58%), and determine VHA-suited implementation strategies (47%) (Fig. 3). The majority of reports were used within 3 months after completion (82%). Respondents indicated a variety of resulting uses. The top 3 actions include using the report’s findings: (1) in VHA publications and/or presentations that informed VHA policy or guidance (26%); (2) as part of decisions on intervention/strategy adoption (23%); and to (3) inform evidence-guided procurement decisions (21%) (Fig. 4). Decision-makers considered other inputs, such as expert opinion and other stakeholder involvement, in addition to ESP reports (Fig. 5).

FIGURE 3

Need that prompted a request for report. Two respondents did not answer this question. Respondents could choose >1 option. VHA indicates Veterans Health Administration.

FIGURE 4

Actions that resulted from report. Two respondents did not answer these questions. Respondents could choose >1 option. ESP indicates Evidence Synthesis Program; RFA, Request for Applications; SOTA, State of the Art Conference; VA, Veterans Affairs.

FIGURE 5

Other factors that influence implementation. Two respondents did not answer this question. AHRQ indicates Agency for Healthcare Research and Quality; ECRI, Emergency Care Research Institute; VA, Veterans Affairs.

ESP reports were generally viewed as having the right scope (94%) and as clearly contextualizing the findings for VHA (84%). More than half (53%) of survey respondents indicated that it would be useful for ESP reports to include guidance on implementing findings. Respondents further clarified that such guidance would be helpful but only when report findings suggest a clear benefit for the intervention under investigation. For instance, 1 respondent suggested, “If there are strong findings, providing best practice tools on how to implement those finding is always helpful.” Furthermore, the vast majority of respondents said that they considered ESP reports equal to (or better than) other resources, such as Cochrane or Agency for Healthcare Research and Quality reports. There were only 6 cases in which the reports were not used (9%). Among those cases, there were no instances in which the nonuse was clearly due to the primary requestor finding the report inactionable. Finally, a preponderance of respondents (83%) said that they would request ESP reports again in the future.

DISCUSSION

Over the past decade, much work has been done by the Institute of Medicine and others to conceptualize and promote learning health system principles. However, less is known about how health systems have transitioned to learning health systems and the impacts of implementing these principles. Feedback from VHA leadership about how they have used an embedded evidence synthesis program to support VHA’s ongoing learning health system journey could help other health systems and evidence synthesis producers to develop and foster strong partnerships to enhance the integration of evidence-based practices into health care delivery.

Findings from our recent survey of learning health system evidence synthesis users indicate that our evidence products are highly actionable and commonly used within 3 months of completion to support intervention/strategy adoption decisions and medical device/therapy acquisition, and are used as the basis for VHA publications and presentations. This high actionability may be attributed to a couple of key factors noted by the Schoelles white paper6 as important for evidence to be believable and implementable by a health system: (1) continuous engagement with operational partners that fosters reciprocal trust and (2) inclusion of VHA clinicians as evidence reviewers to maximize local and operational contextualization of evidence. In addition, as 51% of our reports have also been translated into manuscript publications and published in high-impact medical journals,10–15 this extends their reach to other learning health systems.

As noted in the 2017 Schoelles white paper,6 evidence requests vary across health systems. For example, in VHA, most requestors were nonacademic SMEs with VHA operations decision-making authority; whereas, for a few of the other health systems, most requestors were clinical departments. In addition, VHA uses ESP reviews for evaluation of health system initiatives and care processes more frequently than other health systems. This difference emphasizes the importance of tailoring review content, format, and timeline to meet different needs for each health system. To satisfy the needs of our customers, the ESP has evolved streamlined methods to produce focused reviews more quickly, to create an array of report products suited to their information needs, and to move beyond using traditional systematic review methods to answer questions about the evidence base16 (Evolution of ESP, Supplemental Digital Content 2, http://links.lww.com/MLR/B847). These additional methods include incorporating key informant interviews with SMEs to supplement significant gaps in the literature and17–19 analysis of VHA data to validate or supplement findings of systematic reviews.20,21 Report products have also expanded to include visual evidence maps,18,22–30 and clinician guides to aid those in the field in understanding the current evidence base for various treatments and in engaging in evidence-informed discussions with their patients.31

Our survey has a few important limitations worth noting. First, while our response rate of 48% is consistent with the average organizational research response rate of 53%,32 there is the chance that nonrespondents’ perceptions may differ from those who completed the survey. Second, our surveys were sent out 6 months after report completion—as most users noted that they used the report within 3 months, this delay may risk respondent recall bias. Third, while respondents were invited to include open-ended clarifying comments, we received few, which prevents a deeper understanding of their experiences using ESP products. Also, it is unclear if and how others either within or outside the VHA health system who have accessed reports on our website have used them and are not included in our findings.

To further increase the actionability of our evidence products, ESP is considering several future directions for innovation. First, to gain a more in-depth understanding of how ESP products are used and valued, we plan to conduct a thematic analysis of feedback received on draft ESP reports and implement phone-based postreview debriefs with operational partners and review teams. This would solicit more impactful information on what went well and what can be improved. We also plan to explore methods for evaluating the perceptions of end-users who are not operational partners (ie, those who access the report on the program website). Second, as discussed in the paper by Christensen et al33 in this supplement, the health system decision-makers view the omission of implementation information in reports as a limitation of evidence synthesis products’ utility. Despite decision-maker enthusiasm for implementation information, discussions with ESP investigators has revealed disagreement about if and how to include this information. Although the ESP has started to pilot incorporation of implementation information in reports, it is also exploring ways of promoting scale-up and spread of promising practices, including developing guidance and “how to” toolkits, and including implementation specialists on the review teams. Finally, as discussed in the paper by Gierisch et al34 in this supplement, the program is also exploring engaging Veterans as advisors in the review process to further increase the likelihood that end products will be relevant and useful to decision-makers and support end-user uptake.

In conclusion, our survey of decision-makers working in a large national “learning health system” provides valuable new information about actual use and timing of evidence uptake in their learning cycle that can support development of other health system and evidence producer partnerships. To better support health systems’ uptake of evidence, our survey findings indicate a need for evidence syntheses to present findings with the local and operational context taken into account and to provide implementation guidance. Lack of reviewer consensus about if and how to incorporate implementation guidance in evidence syntheses indicates a need for health systems and evidence producers to sponsor methods to work to develop tools that support these efforts.

Supplemental Digital Content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal's website, www.lww-medicalcare.com.