"It Would've Been Nice if They Interpreted the Data a Little Bit. It Didn't Really Say Much, and It Didn't Really Help Us.": A Qualitative Study of VA Health System Evidence Needs.

BACKGROUND: Patient health outcomes improve when learning health care systems use evidence to implement promising services and allocate resources effectively. Here, we examine the unique environment in which Veterans Health Administration (VHA) leadership use evidence and the facilitators and barriers to using evidence synthesis products in decision-making. We end by describing the steps researchers can take to better support the needs of health system leadership.
METHODS: We conducted 20 semistructured phone interviews with individuals in VHA leadership positions. We used an inductive approach to identify themes observed across key informant interviews.
RESULTS: Key informants identified several factors that fostered the use of evidence including, timeliness, lack of bias, flexible approaches, and concise reports with a clear bottom line. Barriers included lack of relevant evidence and lack of information on how to translate evidence into practice, resistance to change among providers and within the larger health system, and political pressures to implement therapies or technologies with little evidence or uncertainty. Researchers can foster evidence uptake by developing a review scope and key questions that are important to multiple stakeholders, including frontline clinicians and health system leadership.
CONCLUSIONS: The VHA's evidence needs resemble other health systems, but evidence synthesis products should include a translational component to enhance implementation. Resistance to change and political pressures can further hinder the uptake of evidence within VHA.

BACKGROUND

Evidence Needs of Health Systems

Improving the fundamentals of health care systems—including quality, outcomes, and cost—requires evidence that meets the needs of policymakers and understanding the facilitators and barriers to using evidence. Health systems frequently seek evidence to aid decision-making for adopting new medical technologies, implementing health services, or selecting models of health care governance, financing, or delivery. Optimal evidence syntheses are succinct, timely, and specific, and include contextual information and local evidence of importance.1 Health systems may access evidence through several mechanisms, including internal quality improvement processes through the use of internal benchmarking data or internal medical librarians. Alternatively, health systems may form collaborations with outside organizations who produce evidence syntheses for specific needs.2 Increasing evidence uptake and use requires partnerships and trust between health systems and evidence synthesis producers.1

Common barriers to evidence use include long production times, lack of existing relevant research, scoping that does not match decision-making needs, and concerns about trust and credibility (ie, whether the review comes from an organization or authors that users trust to perform methodologically sound systematic reviews).3–5 In addition, lack of trust between policymakers and researchers and differences in understanding evidence synthesis methodology can impede health system use of evidence.4,5 To overcome these barriers, partnerships between health systems and evidence synthesis producers should be formed.3,4 Within these partnerships, discussion of specific evidence needs, timelines, scoping, and methodological concerns can improve evidence synthesis relevance and use.3,5

Study Aim

In this article, we focus on the evidence needs of the Veterans Health Administration (VHA) health system. We seek to understand the environment in which VA leadership use evidence as well as the barriers and facilitators in using evidence synthesis products to inform decision-making. We also examine whether the experiences and needs of VHA leadership are unique compared with other large health systems. We end with a brief discussion of what researchers can do to better support health system needs.

METHODS

We conducted a thematic analysis of semistructured phone interviews with VHA leadership to better understand evidence needs within the VHA and the facilitators and barriers to using evidence synthesis products to inform decision-making. Potential participants were identified through discussions with: Quality Enhancement Research Initiative (QUERI) and Health Services Research and Development Service (HSR&D) leadership, the Evidence Synthesis Program (ESP) Steering Committee, and ESP leadership. Our aim was to interview VHA leaders representing a variety of VHA national program offices as well as regional health system clinical and administrative managers. The ESP Deputy Director initially contacted all identified potential participants via email to describe the project and encourage participation. A sociologist trained in qualitative methodology interviewed those willing to participate. Interview participation was voluntary, and no incentives were provided. Interviews lasted 30–60 minutes and were audio-recorded and transcribed verbatim. To ensure anonymity, throughout the manuscript we refer to participants as “informants.” We arbitrarily assigned each participant a unique identification number (1–20).

We developed our semistructured interview guide (Supplemental Digital Content 1, http://links.lww.com/MLR/B835: KI Interview Questions) based on information from: (1) feedback from the ESP Steering Committee and QUERI leadership; (2) discussions with ESP researchers; and (3) domains identified in the literature examining facilitators and barriers to using evidence.

The primary researcher (V.C.) reviewed all transcripts and used an inductive approach to identify themes across the key informant interviews. Informants were provided with a draft of the project’s findings and were given the option to modify or delete their quotes. The interview guide was administratively reviewed and approved as quality improvement, intended to guide program development, based on VHA policy.6

RESULTS

Identified individuals in VA leadership positions (N=35) were invited to participate in a semistructured interview. Twenty-eight individuals initially agreed to participate, and 20 interviews were completed (4 regional health system leaders/Chief Medical Officers and 16 national program office leaders) between October 15, 2018, and December 10, 2018. Our key informant interviews revealed several common themes that contribute to and hinder the use of evidence.

Facilitators of Evidence Use

Although key informants reported several factors that fostered the use of evidence (Table 1), the 4 main themes were timeliness, avoidance of bias, flexibility in approach, and a focus on clear, concise take-home messages.

TABLE 1

Barriers and Facilitators to Evidence Use

Timeliness

Many viewed the ability to meet the tight timelines of the Veterans Affairs (VA) health system as critical. Several key informants remarked that they are often expected to make decisions or implement new care processes very quickly. Informant (8) explained: “Decision making is happening so fast and it’s so volatile that I would … aim for speed rather than perfection.” Informant (10) described how a “slow” review limited his use of a report: “I don’t have [the report] in my hands; I can’t give it to the people who it would be very impactful for … when it comes out it’s going to be very late in the game.” In addition, the pace of technology itself requires timeliness. Informant (13) mentioned: “The problem with technology is that it advances so fast. If you have to wait a year to do an evidence synthesis review, you are already two or three generations behind … it just can’t be done fast enough.”

The Importance of Unbiased Evidence

Similar to the findings of other researchers,1,3–5 our informants confirmed that for evidence syntheses to be perceived as reliable and useful, they must be viewed as unbiased. Informant (8) commented: “[I may be able to] go to Google Scholar or PubMed and pull up the article that confirms my preconceived biases and even if you guys are doing the same thing … you’re at least going into it without a preconceived bias … to get a certain answer. So, I would hope that you would always preserve independence and not massage the data … to give us the answer we want.” Informant (17) remarked: “… There’s a lot of stuff out there that is not objective. You can go to the internet and find all kinds of stuff, but I want to use information that is systematic, objective, and thorough.”

A Flexible Approach to Evidence Synthesis

To be responsive to the varying needs of health system leadership, a variety of different evidence synthesis products, beyond traditional systematic reviews, must be available. Informant (20) described evidence needs from a leadership perspective: “… when you get into the policy/leadership world, you are trying to lead a transformation in health care in a way in which we don’t have all the answers and we’re never going to have all the answers. What’s helpful to me in terms of products is something that says: ‘here’s the best we’ve got, here’s what we can and cannot say.’ To me it is a more practical application and a broader picture of what we know and don’t know. Rather than straight research in the way I was trained in as a clinician.”

Clear, Concise Evidence Reviews that Include a Practical Bottom Line

VA leadership reported they are more likely to use concise reports with clear conclusions that focus on the bottom line—with particular attention paid to what works or shows promise. Informant (18) commented: “It has to be short and sweet and digestible. If it’s 5 or 10 pages, then many people are not going to read it. It has to be something that catches people, is impactful to them …” Highlighting research that has been demonstrated to be effective and interpreting the available data are viewed as most important. When faced with the need to make decisions when evidence is sparse, stakeholders are more comfortable with reviews that include “rigorous speculation”7 and prefer guidance about possible choices in contrast to traditional systematic reviewers who focus on evaluating the quality of existing evidence. When lamenting about the lack of conclusions in reports, Informant (2) noted: “… it would’ve been nice if they interpreted the data a little bit. It didn’t really say much and it didn’t really help us.” While Informant (4) stated: “With the [AHRQ Evidence-based Practice Center] projects, 75% of the conclusions are ‘inadequate information or studies to conclude blah blah blah.’ Which isn’t helpful from an operational perspective at all …” This Informant went on to say “… from an operational perspective [identifying] the things that work are the most valuable.”

Barriers to Using Evidence in Decision-making

Two main themes emerged as barriers to evidence use: a lack of evidence around specific policy questions and a lack of information on translating available evidence into practice. Several informants mentioned 2 additional themes that lie outside the scope of evidence. These include resistance to change among providers and within the larger health system, and political pressures (from members of Congress or Veterans groups) to implement therapies or technologies with little evidence or despite evidence demonstrating lack of effectiveness. Although these lie outside the scope of “evidence,” we believe it is important to describe how such factors affect how evidence is used or ignored.

Lack of Availability of Research

Several informants commented on the lack of available research to inform their decision-making needs. Informant (2) commented: “… I try to use evidence in all of my decision-making, so the barrier to making evidence-based decisions would be the unavailability of evidence.” While Informant (4) lamented that “Common barriers are lack of evidence. Either we haven’t looked at the topic or if it was looked at, we were more concerned if it was the same population we’re talking about.”

The Need For Information on Implementation

Our informants indicated a desire for guidance on how to translate evidence into practice as part of the evidence synthesis process. Informant (16) stated: “I think it would help when research identifies a particular process [they also] outline the parameters that should be in place [for the] best chance for success. … What types of staffing is needed … what kinds of space is needed …” Informant (20) remarked: “… talk to us from an implementation side; what matters and what do you pay attention to and what are the barriers? I would think that having the best understanding one could have of that … would inform and enrich the product.” Thus, it is the practical “how to” guidance, which is lacking from traditional reviews, that VA leadership are looking for.

Resistance to Change

Informants also identified resistance to change among health care providers as a significant barrier to the successful implementation of evidence-based therapies and technologies. For instance, Informant (13) noted: “Translational practices and science is always a problem. You are always going to have a number of people who are resistant to change no matter the reason.” While Informant (12) commented: “There are a lot of … people who push back. [They] know what they know and don’t see a need to change. Informant (19) elaborated: “… when you deal with younger clinicians and younger health care providers it’s much less of an issue, but when you talk about people over 50, that can still be a big barrier.”

The reluctance of VA leadership to implement new therapies/technologies also poses a barrier to the uptake of evidence. Informant (4), for instance, remarked: “Another evidence gap is that of actually making the change. You might be presented with evidence that this particular intervention will achieve the goal that you want, but if it’s a lot of work to implement, then one might not necessarily go down that pathway.” While Informant (10) commented: “Just because you implement something doesn’t mean that it’s in practice. There is a lot of people that go out there and say that we implemented this in 150 facilities and I still have the ability as a frontline clinician to say ‘yeah, our facility isn’t using that.”

Informant (20) elaborated about the challenges of decision-making and implementation at the national level: “I feel like the biggest barrier to us being a learning organization is an inability to make decisions and execute. I don’t believe that the weakness is in our ability to understand the evidence or have the right information … But we are incapable … of making decisions and changing even basic easy stuff. Look at the commission reports and everybody else that evaluated what we need to change to fix the underlying issues … and we have failed repeatedly. We all agree on the problem, [but] we are incapable thus far of fixing [it]. We can always use more evidence [to make decisions at the national level], but that’s not the problem…we have an issue with leadership and implementation. There is compete lack of process.” For this informant, a lack of process for putting large-scale directives into practice, rather than evidence, is what is preventing system-wide change.

Political Pressures

Our informants also said political pressure, either from Congress or Veterans’ groups, directly affected the ability to use evidence in decision-making. Most often, political pressure results in the approval of treatments without supporting evidence. Informant (8) commented: “I think our governance process is fairly broken. It tends to be fairly reactive; our board of directors is any individual member of Congress, so there are 535 of them and any one of them might ask us to do something, regardless of the evidence. For political reasons we will often move forward with it even if the evidence says we shouldn’t.” Informant (8) further stated: “… We’ve done a lot of verbal and policy gymnastics to try to make sense of a situation that fundamentally doesn’t make sense. Let’s be honest why we’re doing this; we’re calling it compassionate innovation. We’re doing it because politically we’ve been told to do it …” When discussing evidence synthesis reports, Informant (6) lamented: “… there is complete politics that dictate how the policies are written … [For example,] there is a certain political pressure to offer [describes device] to Veterans with PTSD despite that there is no study to show that it is better.” In some cases, political pressure can result in not using a particular treatment/technology in spite of evidence supporting it, as Informant (16) elaborated: “The politicians said, ‘No, don’t do that.’ The evidence was there but wasn’t allowed to be used.”

Furthermore, 1 informant viewed the overarching political atmosphere as having contributed to a lack of direction and stability within the health system, directly affecting the ability of the VA to create a unified and consistent course of action. Informant (20) remarked: “Obviously, the VA hasn’t been in stable times for a very long time … Constant turnover in leadership is a big one because the work itself is hard and requires really changing what we incentivize … everything from what we reimburse, measure and value on performance … This kind of large system change is very difficult without stable, consistent leadership.” This theme has been described elsewhere, including national media outlets.8

In addition, interview participants noted that pharmaceutical companies and device vendors—either directly or through Veterans’ groups or lobbyists—push for the use of certain interventions either without evidence or based on “cherry picked” evidence. When describing the lack of processes for purchasing decisions, Informant (13) noted: “… things get thrust onto the VA and you have vendors that are pushing them. They will often cite information as evidence, but it remains to be seen. … It’s difficult to manage such tactics when a company, organization, or association contributes to the ranking member of one of your committees, and claims they offer this fabulous technology which VA is not giving to Veterans. The Department then receives bevy of congressional inquiries as to why we are not issuing a certain technology, or why aren’t we issuing certain technology at the rate at which they believe we should be using it?” Informant (1) remarked: “What chance do you think evidence has against marketing? They amplify the message by the pharmaceutical companies promoting what the guidelines from the private sectors say. … Low testosterone (Low T) is a great example, it was never evidence-based.”

What Researchers Can Do to Increase the Uptake of Evidence Synthesis Products

Develop Key Questions That Matter to Stakeholders

Developing key questions that are important to frontline clinicians as well as VA leadership is described as key for the success of an evidence synthesis product and is central to evidence uptake. Informant (5) remarked: “I think a lot of times we will call upon clinic managers to implement change, but they aren’t necessarily researchers. They’ll be interested in different kinds of things like, how many people do I need to hire? What kind of supplies do I need? Things that we don’t normally write in research papers.” Informant (15) stated: “To be useful to people it needs to answer … a question that is on people’s minds and that they care about. If we say something and no one really cares, it doesn’t matter how pretty the package.”

Engage a Variety of Stakeholders Throughout the Review Process

Informants also suggested that successful implementation begins with engaging a variety of stakeholders to inform the review scope. Informant (15) elaborated: “It can’t be the leaders of the organization that are so high up that they don’t know what’s going on. And it can’t just be people at the frontline. It needs to be a combination. I think finding the right people, in the very beginning [is important], not after we’ve compiled the results and written the report.” Informant (20) elaborated: “I think pre-interviews with somebody like me—who again isn’t a researcher, but a leader who is trying to drive change and who basically will be the end user of the product—to get an understanding of the nuances and context of the question at hand. To kind of do an interview with stakeholders: What are the questions you’re having? … that’s just again making a case for a deep dive earlier in the process, not only with how we would use the information, but what are the hidden challenges in answering the question.” From this perspective, having a clear understanding of both the end-user’s evidence needs and their plans for use—from review inception—will help researchers develop products that are more useful to decision-makers and therefore more likely to be used.

DISCUSSION

Key informants described several factors that foster the use of evidence, including timeliness, avoidance of bias, a flexible approach to evidence synthesis, and a focus on concise and clear take-home messages. Difficulties occur when evidence is sparse or indirect or when issues of implementation are not adequately addressed. These findings align with common barriers and facilitators identified in previous research in other health systems.2–5

Our interviews identified other factors, including resistance to change and political pressures, that can further stymie efforts to fully utilize evidence in decision-making. Although other large health systems are primarily accountable to a board of directors and stockholders, the VA is unique in that it is accountable to Congress, Veterans, and taxpayers. Although it may seem like a stable organization—given its longevity and size—those working within it expressed frustration with constantly changing directives and leadership as the organization attempts to react to shifts in political directions. These factors can create a chaotic environment where reactionary decisions are made with little attention paid to what can be learned from the published literature.

To our knowledge, this is the first in-depth examination of the facilitators and barriers of using evidence from the perspective of VA leadership. Our chosen methodology allowed for added insight into the current climate of the VA health system. Although our project identified several factors that affect evidence uptake, which are consistent with the published literature, our findings have the following limitations. Our findings reflect the experiences of a limited number of participants and cannot be generalized beyond the scope of this project. Further, several of our informants had previously requested ESP reviews and some had prior knowledge of systematic review methodology, which may have introduced bias—in either direction—regarding their perceptions of evidence in decision-making.

CONCLUSIONS

Our interviews highlight the need for new and innovative evidence synthesis products that address the complex and often competing demands that have been placed on health systems. VA leaders expressed a need for evidence reviews that are timely, include implementation specifics, and provide a practical bottom line. Researchers can further foster the use of evidence by developing products that take the needs of multiple end-users into account.

Supplemental Digital Content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal's website, www.lww-medicalcare.com.