Interpretation and use of evidence in state policymaking: a qualitative analysis.

INTRODUCTION: Researchers advocating for evidence-informed policy have attempted to encourage policymakers to develop a greater understanding of research and researchers to develop a better understanding of the policymaking process. Our aim was to apply findings drawn from studies of the policymaking process, specifically the theory of policy windows, to identify strategies used to integrate evidence into policymaking and points in the policymaking process where evidence was more or less relevant.
METHODS: Our observational study relied on interviews conducted with 24 policymakers from the USA who had been trained to interpret scientific research in multiple iterations of an evidence-based workshop. Participants were asked to describe cases where they had been involved in making health policy and to provide examples in which research was used, either successfully or unsuccessfully. Interviews were transcribed, independently coded by multiple members of the study team and analysed for content using key words, concepts identified by participants and concepts arising from review of the texts.
RESULTS: Our results suggest that policymakers who focused on health issues used multiple strategies to encourage evidence-informed policymaking. The respondents used a strict definition of what constituted evidence, and relied on their experience with research to discourage the use of less rigorous research. Their experience suggested that evidence was less useful in identifying problems, encouraging political action or ensuring feasibility and more useful in developing policy alternatives.
CONCLUSIONS: Past research has suggested multiple strategies to increase the use of evidence in policymaking, including the development of rapid-response research and policy-oriented summaries of data. Our findings suggest that these strategies may be most relevant to the policymaking stream, which develops policy alternatives. In addition, we identify several strategies that policymakers and researchers can apply to encourage evidence-informed policymaking. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/.

We assess a unique population, US state legislators and administrators trained in research interpretation, to better understand the role of evidence in health policymaking.

We applied policy windows theory to clarify the stages in the policymaking process where evidence is useful in making policy decisions, where it is unlikely to be useful, and where more research is needed.

The results provide clarification regarding the types of policy-relevant research that are most likely to be used.

The findings are limited by the reliance on a small sample selected for their interest in integrating research with policy, and by completion of interviews in 2005.

Introduction

In the 1990s, there was increasing concern that US patients were receiving inappropriate care due to medical errors, leading to the ‘quality movement,’ which noted that practitioners frequently did not use evidence-based practices.1 In some cases, patient evaluations of care were found to offer evidence of quality,2 which led to increasing state policymaker involvement with regulating healthcare, as these findings offered justification to listen to constituents rather than experts.3 By 2004, there were studies suggesting that evidence-based medicine-informed policymaker decisions in the USA, with systematic reviews having the greatest credibility.4 5

Building in part on these experiences, research on evidence-informed policymaking has attempted to identify how scientific findings are used in policymaking in multiple countries, and how to improve the process of research translation. One study in the UK, for example, found that policymakers defined ‘evidence’ broadly, and relied on stories while ignoring inconclusive research.6 Other research has suggested that ideas are used strategically by policymakers to support positions, not to learn new information and that their calls for more research are an avoidance tactic intended to discourage the consideration of possible policy changes.7

These findings have led to suggestions to make research more policy relevant by expanding definitions of what constitutes ‘good’ evidence in systematic reviews through the creation of ‘realist’8–11 or ‘practical’12 reviews and urging researchers to increase their community engagement and improve their dissemination strategies.13–17 Other studies have noted that policy-relevant research faces distinctive constraints, including the cost of studies, the need for rapid results and potential limits on decision makers' understanding of scientific research.18 Some studies have concluded that research, if appropriately designed, could be used to clarify problems, develop support for evidence-informed policy, and monitor and evaluate policy changes.19 20 Recent research has found that simplified methods of interpreting research are effective for policy advocates.21

Other studies of evidence-informed policymaking have focused on the strategies that researchers use when presenting information. They note that scientific presentations often do a poor job of identifying who will be harmed or saved,22 whereas policymakers are rewarded for focusing on individual constituents.23 24 Research on storytelling has found that people who are trained to rely on technical information can appear morally indifferent25 and mystify outsiders unfamiliar with the language of research.26 Those not trained to interpret research prefer stories, which make the world intelligible by turning what seems like a random aggregation of facts into a process with a cause and effect.27 28 Stories can organise policy questions,29 30 suggesting that experts are most politically useful when they combine research with anecdotes, creating ‘superior stories.’26 31

All of these proposed strategies—expanding definitions of evidence, changing the focus of studies, redesigning research presentations—focus on the nature of evidence rather than the nature of policymaking. Research conducted with policymakers themselves has concluded that some of these factors are more or less relevant; one systematic review found that personal contact, timeliness and evidence summaries were most important.32

A different perspective is drawn from research on the policymaking process from political science, specifically the theory of policy windows, which suggests that there are points in the policymaking process where evidence will be more or less relevant.33 Policy windows theory argues that three streams must align to enact policy change: first, an issue must be defined as a problem, second, a policy alternative must be found to address it and third, there must be sufficient political resources to make the change feasible.33 Policy windows theory has been used in previous work to consider the complex interactions between the roles of research evidence and political constraints on decision makers, and offers potentially useful refinements to theories about how research can be used in making policy.34 35 However, research on evidence-informed policy has not yet addressed whether evidence (however, it is defined) can be used to move any or all of the three policy streams.

In this study, we applied Kingdon's policy window theory to the question of evidence-informed policy to address previously unanswered questions about the value of evidence in creating policy change. To assess the actual and potential roles of evidence, we contacted state policymakers trained to interpret scientific research and apply it to health policymaking. They indicated how they defined evidence, assessed which of the strategies proposed by researchers in evidence-informed policymaking had been most effective in the states where they worked, and identified which of the policymaking streams were most amenable to the use of evidence. Our findings suggest that state policymakers in the USA defined evidence conservatively, and that while they found scientific research valuable in policymaking, they found that its primary use was in developing policy alternatives, rather than in problem definition or political feasibility. These findings suggest that the different strategies that have been proposed to increase the use of evidence in policymaking may be more or less useful depending on the policy stream that they affect.

Methods

We applied Kingdon's policy windows theory as a framework for interpreting and analysing qualitative data, which was collected by interviewing policymakers, about the ways they interpret and use evidence. Policy windows theory is useful for this type of analysis because it grew from research on policymakers and their incentives rather than from expectations drawn from clinical research.33 Policy windows theory has been successfully applied to studies of the policymaking process in a variety of issue areas, including health, and in multiple countries. Recent studies have noted that the theory is particularly relevant to questions of scientific research could be used in making policy; addressing this question was the objective of this study.34 35

Our study relied on interviews conducted with participants of the Rocky Mountain Evidence-Based Health Care (EBHC) Workshop, a set of seminars offered by the Reforming States Group. The Reforming States Group was founded with funding from the Milbank Foundation in 1991, when leaders in health policy from state government met to share their experience and develop solutions to healthcare problems. The group expanded its work over time to the point that a bipartisan, voluntary group of representatives from over 40 states, the Reforming States Group, met regularly. The EBHC Workshop offered seminars based on small group, experiential learning, which taught techniques for finding, analysing and interpreting scientific studies for use in policymaking. The workshop design was validated in an earlier study that sought to teach research evaluation to US state and federal judges.3 Our research on participants' application of evidence relied on a grounded theory approach.36

Research team

One of the authors (LAB) had worked with EBHC as a workshop coordinator from 2000 to 2006, and organised the study design in part to answer their questions about (1) how effective prior seminars had been and (2) how to improve education of state policymakers. Contact information for potential participants was provided by EBHC organisers. The letter of introduction was sent by the author who had taught a portion of the workshops from 2000 to 2005 (LAB) and it described the research project and interviewer. To address reflexivity concerns that could arise regarding the validity of different kinds of evidence if participants were interviewed by a past instructor for the evidence-based healthcare workshops,37 a separate interviewer, who held a PhD in social policy and a JD and developed extensive interview experience in the course of his doctoral research, was recruited to collect data for the study.

Study design

Participant characteristics

The study population consisted of USA policymakers that had attended the EBHC Workshops at least twice from 2000 to 2005. These participants offered a unique perspective on how information is used in policymaking, as they were both insider informants about policymaking with access to closed-door negotiations, and had explicitly sought to develop expertise in understanding and interpreting scientific research on health. Of the 35 potential participants who met the selection criteria, 24 chose to participate after receiving a letter of introduction and a follow-up telephone call (68.5% response rate). Of the remaining 11 potential participants, two chose not to participate, three agreed to participate but could not be scheduled for an interview and six could not be located.

The participants included 12 state legislators, 10 officials from state public health agencies and 2 people who had held both types of positions.

Those who had served as legislators represented all four census regions (two from the Northeast, two from the South, five from the Midwest and five from the West), both levels of state government (nine had served in the lower house, four in the senate, and one in both houses) and both political parties (eight Democrats and six Republicans.) The legislators' average length of government service was 16 years; six identified as women and eight as men. Three of the 14 self-identified as having a professional background in science or the health professions.

The administrators represented all four census regions (three from the Northeast, three from the South, three from the Midwest and three from the West). The average length of government service for those who had served as state health administrators was 10.5 years, three identified as women and nine as men, and seven of the 12 self-identified as having a professional background in science or the health professions. All of them had served as health-related division or programme directors or officers.

Setting

The first round of interviews was conducted in August 2005 at the EBHC Workshop. Five participants who attended the 2005 workshop were interviewed in person in private breakout rooms; the remaining participants were interviewed by telephone. Travel costs for policymakers attending the workshops were paid by the Milbank Memorial Fund. Follow-up interviews were conducted 6–9 months after the workshop by telephone with three of the five 2005 workshop participants.

In the course of each interview, participants were asked to describe their current job responsibilities, employment history and basic biographical information. For the remainder of the interview, participants were asked to describe 2–3 cases where they had been involved in making health policy, providing examples in which research was used, either successfully or unsuccessfully. For each example, respondents were asked to highlight the most important issues and participants, the types of evidence they themselves used, challenges and facilitators to using research findings, the most important factors leading to the policy outcome, and the effect, if any, of the EBHC Workshop experience on their work. Interviews lasted between 30 and 120 min, with an average length of 60 min. Each interview was audio-recorded and transcribed. The interviewer took field notes during interviews and immediately afterward, which were appended to the transcripts after they had been reviewed for accuracy by participants. Data saturation was pursued by using a replicable research design and standardised instrument,38 39 requesting information on behaviour and context,40 interviewing participants from multiple years of workshops,41 checking for new information in follow-up interviews and after coding,42 43 and using multiple coders.42 We used triangulation strategies to ensure data validity, consistent with prior research on policymakers and their decision making.44–46 Specifically, our data collection relied on multiple respondents within certain states, and we verified the existence of stated policy outcomes by assessing outcomes across policymaker types (eg, comparing a report from an administrator with that of a legislator), written documentation provided by workshop administrators and policymakers, and where relevant, by checking contemporaneous media reports. Our triangulation sources validated each other and no contradictory information appeared, providing confirmation of findings across interviews.

Data analysis

Transcripts were entered into the qualitative analysis programme NVivo for review. As suggested by Glaser and Strauss, the stages of grounded theory analysis proceeded through codes, concepts, categories and theory.36 To ensure coding reliability, three coders (the interviewer, a research assistant, DEA) independently coded the data iteratively to generate themes. As a further validation for coding reliability, the resulting codes for all interviews were independently reviewed by both authors, who discussed and resolved discrepancies (DEA and LAB.) Initial codes relied on key words (eg, ‘evidence’) and creation of specific codes identifying key concepts in statements that had been explicitly emphasised by interview participants. Coders looked for repetition of these concepts across multiple interviews. In addition, coders marked conceptual statements that were repeated across interviews even when they had not been explicitly identified as critical concepts by respondents, to ensure that concepts that may have seemed obvious to experts were not missed by analysts. Interviews were coded by nature of the issue (eg, vaccines, insurance coverage), information used (including explicit or implied definitions of the term ‘evidence,’) and stated barriers and facilitators to using research literature in policymaking. These categories were further refined using the political science framework of policy streams, addressing problem definition, policy alternatives and political feasibility.33 The results review the identified themes, ideas, reactions and expressions.47

Results

Administrators and legislators suggested strategies to generate evidence-informed policy and perceived this to be desirable. However, as policymakers, they saw a more limited and more sophisticated role for research than much of the literature on evidence-informed policy has suggested. Unlike some of the policymakers interviewed in other studies, they made an explicit distinction between evidence, by which they meant scientific research, and other forms of information. They also had clear expectations for when evidence as they defined it was relevant to policymaking, which followed the policy streams classification discussed by Kingdon's theory. Evidence was not perceived to be relevant in problem definition, yet was viewed as highly relevant in deciding among policy alternatives. And although these policymakers anticipated that evidence could be important in the political feasibility stream, they found that the available research was frequently not designed or written in ways that made it useful in this context.

‘Evidence’ is defined as scientific research

Past research on evidence-informed policymaking has suggested that policymakers use an expanded definition of evidence, including information that researchers have historically perceived to be ‘lower quality,’ such as anecdotes, case studies and public opinion.6 State policymakers in this study, however, unanimously (24 of 24 participants) stated in their interviews that they believed evidence consisted of peer-reviewed scientific research that relied on replicable research designs, large sample sizes and relevant control groups. Although, not all of the respondents identified all of these characteristics of research evidence, the distinction between research defined as evidence and other types of information was made repeatedly by respondents.

You know, there would be no way from a political point of view that we could sell at this time stopping mammography, just because politically we can't. You know, the evidence is the evidence. But the reality is… certain things you can't do in the United States, just because the people won't allow it even if the data shows otherwise. An example: if you come in and you've been in a car accident and you say your head hurts I'm going to get a skull X-ray. I would do it because I'm concerned about litigation. There are many, many other things other than pure evidence that drive the things I do. (P#02, administrator)

One legislator explicitly distinguished between evidence, meaning research and anecdotes.

The other important source of information is certainly not an evidence-based approach but there are a growing number of anecdotal reports coming really across the gamut from parents. (P#05, legislator)

Although legislators and administrators perceived that their own use of evidence was more sophisticated than that of many of their colleagues, who had no training or experience with research, in all cases they were careful to distinguish between what they perceived as evidence and what they viewed as a misinterpretation of the term.

[There is] overall interest in evidence-based practice that's floating around now. There's a lot of general discussion, both in some of the journals as well as in the government's view of how to move forward and some of their programmes. And admittedly they use the term inappropriately sometimes. (P#08, administrator)

Over half of the state administrators in our sample had advanced training in science or the health professions. Their limited definition of what constituted evidence may have reflected this training in interpreting scientific research. However, they reported that they used this definition in part because they had found it useful to distinguish between evidence, which was perceived as credible, and other information, which was perceived to be less credible. This finding suggests that using a limited definition of evidence may have practical relevance in policymaking.

Respondents also noted that decision makers who become involved in health policymaking have distinctive characteristics, tending to be less conservative than some of their colleagues. In the words of one legislator, “If somebody is interested in shaping health policy, it's not going to be a really conservative person because people who are interested in shaping health policy believe that government has a role to play. And that's a naturally selecting sort of criterion.” (P#24, legislator) In addition, several states have become single-party states in the wake of 21st century political polarisation in the USA, which may make it easier rather than harder to use evidence in policymaking, as one legislator explained.

Sometimes states controlled by [a single party, even if conservative] tend to be more progressive than states that are really tight in terms of the balance of power. Because it's like “well, we can do good things because we're going to win the elections anyway.” They start looking at the logic of it rather than the rhetoric of the political ideology. (P#17, legislator)

Research was irrelevant to problem definition

Although the policymakers in the study believed that evidence, specifically scientific research, had a role to play in policymaking, they generally did not see it was relevant to defining problems that government should address. Consistent with the 1990s ‘quality movement’ in health policy,3 state legislators and administrators viewed problem definition as something that came from constituents rather than from evidence, and legislators in particular viewed this as appropriate. Of the 24 respondents, 14 reported that new policy ideas tended to be poorly researched, or not researched at all, while 12 respondents (including 9 of the 14 legislators) explicitly stated that new policy proposals were drawn from what constituents, family members and friends wanted. Administrators also accepted that problem definition was very constituent focused, even if they might prefer that this were not the case. Representative quotes are provided in table 1.

Table 1

Research findings were not relevant to the problem definition stream

Legislators viewed problem definition as constituent driven	People will look at the evidence and say “Okay, that's all very fine, but my constituents want [this].” (P#04, legislator)
	My experience has been is that these decisions [about what is important] are seldom made on a study or review of the studies that's out there, on evidence that may be provided. It's based on a much simpler approach of gut or something someone read in an article or something like that. (P#15, legislator)
	I think [legislators] understand the concept of [evidence-informed policy], but they don't think that should be the only criterion. And the other criteria are their own personal beliefs, their own personal experience, something they heard about, what their father said worked one time, you know that kind of thing. So you always have to encounter that. (P#21, legislator/administrator)
Administrators also perceive problem definition as constituent driven	Legislators say to me “I deal with one constituent at a time…” The people I deal with don't care about numbers. (P#02, administrator)
	What I've come to find is that what really appeals to the legislators is anecdotal stuff. Passion. Rarely do I get a request from a legislative aid that asks, “What's the data on this?” Never do we get the question, “What's the research on this?” It's all about gut level feeling. That's what sells things in our legislature. (P#13, administrator)

Overall, legislators and administrators perceived the definition of health policy problems that needed to be addressed as a process driven by the needs of constituents rather than by research on health outcomes.

Research influenced assessment of policy alternatives

In contrast to the disinterest in evidence when considering problem definition, policymakers felt strongly that evidence was relevant to generating, assessing and presenting policy alternatives. This view was reflected in two different ways. First, policymakers believed, consistent with research on the role of stories, that policy alternatives could and should be made more relevant by creating ‘superior stories’ that mixed evidence and anecdotes.26–31 Second, the policymakers in this study, who had been trained in the interpretation of scientific research, felt that relying on evidence was important to assessing policy alternatives. They had also used their training to generate heuristics and shortcuts that allowed them to discredit policy alternatives that were not supported by evidence, which they defined as high-quality peer-reviewed research.

Of the 24 respondents, 14 explicitly called out the importance of stories in convincing policymakers to accept evidence as relevant to their decision making. One administrator reported that a legislator was dissuaded from creating a herpes registry in an effort to prevent the spread of sexually transmitted diseases after being told that there was no clinical difference between oral and genital herpes, and that people with oral herpes would also be listed in such a registry (the legislator in question had oral herpes and did not wish to be listed on a Sexually Transmitted Disease Registry.). Representative quotes from respondents are provided in table 2.

Table 2

Appropriately presented research findings could be relevant in the policy alternatives stream

Stories are used to make research compelling	It's important for people who use evidence, to understand that evidence in and of itself is not persuasive. They have to learn to tell stories. They have to learn to translate the evidence into something that is understandable by the average legislator, average citizen. (P#02, administrator, emphasis added)
	Now it helps to have, in addition to your statistical evidence, it helps to have a few anecdotes so that people can see it concretely. That's always useful. (P#04, legislator, emphasis added)
	I think most legislators are reasonable people if you can try to relate to them and get them to understand “this could be me”. (P#14, administrator)
	The health department would always come to me with… the research. And I always had to tell them, “Look you understand, if some jerk in the legislature has one anecdote that goes directly against this I could lose it.” The anecdote that tells you what happened to a person, only a person is just very powerful. So I used to make them go look for anecdotes on their side. The researcher thought I was a nut. I don't personally really give a crap about anecdotes. But I need one to fight politically. (P#16, legislator, emphasis added)
	It's good to combine things. You know, you take data that's good data and then you back it up with a human face on it. Because then you've got the logic and compassion going for you as a part of the argument. And I think that combination is powerful. (P#17, legislator)
Numbers are not persuasive	[Legislators] tend not to want to do numbers… when I testify I watch the eyes; use numbers, and they glaze. Because if you think about the background of most [legislators], they're not science people, they're mostly non-science, non-mathematicians, non-engineers. And so when I talk about five parts per billion, they have no concept. If you say one grain of sand on the beach at Waikiki, they kind of get it. (P#02, administrator)
	Most legislators don't understand cause and correlation. They don't have any clue about statistics. (P#03, legislator)
Simplified study assessment guidelines can guide decision making	[S]ometimes we answer fire with fire. We say “That's a great article, it's a great subject, we think that we would love to research this topic, or see more data and evidence on this topic when you get it in a peer reviewed journal, in a controlled study.” Sometimes we get it out of the press. Reporters typically ask “Doctor so and so says he's doing this study.” [We say that] we would like to see it in a reproducible study where all the variables are controlled, and we'll be more than happy to consider the evidence at that time; conclusions based on evidence. (P#02, administrator, emphasis added)
	When you talk to [certain advocacy groups] generally their sources are themselves. That's when you know that they've cooked the data. We went through some [training that said] all you have to do is ask one question: Is this statistically significant? And the answer is no. And then once you ask that question you ask, “Well, what was your sample? Who did you talk to? Are there any other corroborating studies that don't come from [your own organisation] that show this is the case?” (P#07, legislator, emphasis added)
	We've had to continually go back to, you know and each meeting we go through another set of interventions that people have come up with. I mean these families… come up with these studies where there's like five kids. They'll come to the meeting, here's this study and then we go through it. It's like, “Okay, how many children? Was there a control group?” We go through all the stuff. And it's like “Well no.” “Well no.” And it's like “Well, then we can't support it, can we?” “Well no…” You can just come back to four or five basic points. You don't need to go into a lot of scientific depth. (P#13, administrator, emphasis added)
	What you've got to find is the inconsistencies in that article and rip it right in front of their eyes. That's the only thing you can do. In the past at least no matter what the administration you'd be able to go back and say “Look in this area the CDC says this is the best way to do it.” And people would just shut up at that point. But now it's harder [and] you've got to be able to attack stupid research that isn't research. (P#16, legislator)
	One of our standard responses when a company comes and asks us to cover something, we ask them for any randomised controlled trials that they've done or anyone has done on the product. But generally they don't have it. So we don't cover it. (P#20, administrator, emphasis added)
Using research can make some policy alternatives more credible	They may say “Okay, this is a policy we want to adopt, but we want to bounce this off of somebody that really knows how to analyse and find evidence.” So they can say “All right, here are the reviews that we looked at, here's the policy that we're articulating. What are the weak points? What level of confidence can we have if we move forward with this?” (P#18, administrator)
	[W]hen you look at a study, and you see that it has had a worthwhile number of people studied, and that it has been done in basically a double blind way. So that you know there's credibility to it. When you take a look at that you have a much better opportunity to make good decisions than if you're just shooting from the hip or getting involved emotionally. (P#01, legislator, emphasis added)
	By knowing and using the scientific base you are going to be able to be more protected than if you just go out on a political limb by yourself. Even if I had enemies that are looking to tarnish our credibility, the fact that evidence for secondhand smoke exposure is pretty overwhelming almost negates whether that can grow legs and propagate. (P#09, administrator, emphasis added)

The respondents were evenly split (11/11) on the explicit question of whether research or politics was more important in assessing policy alternatives. However, the respondents who believed that research could overcome political considerations had developed simplifications of the research standards that they had been taught in evidence-based medicine workshops. Six of the respondents noted that certain methods of study assessment, specifically those relying on numbers or calculations, were not persuasive (see table 2).42 43

Legislators and administrators found that some methods of study assessment were more intuitive than numbers. Those respondents who believed that research could overcome political concerns reported that they were able to use study assessment guidelines to undercut arguments that they felt were unsupported by evidence. These included asking if studies were randomised controlled trials, used large samples, had statistically significant results or had a relevant control group. They reported successfully using these strategies to convince other policymakers and constituents. Finally, being able to understand and present scientific research, even in a simplified way, was perceived as a means of making the policymakers and their preferred policy alternatives more credible. Representative quotes are provided in table 2.

Research was inadequate to affect political feasibility

While research was perceived to be irrelevant to defining problems, and useful in assessing and choosing among policy alternatives, it was perceived as potentially useful but inadequate in the area of political feasibility. Policymakers noted that health policy decisions were made quickly, especially in the context of part-time legislatures that might meet for only a few weeks each year. Twenty of the 24 respondents explicitly noted that a lack of relevant studies was a problem when making policy. As a result, 19 of the 24 said that they had completed what they referred to as non-systematic reviews, at times in a matter of minutes; in one case an administrator was asked to make a coverage decision as a patient was heading into surgery. Of the 24 policymakers, 22 stated that they typically used staff members to collect policy-relevant research in real time, even when the staff members were not qualified as researchers. In addition, 17 policymakers explained that they had relied on trusted outsiders, including interest groups, family members and researchers at nearby universities, when they were forced to make immediate decisions. These findings were consistent with previous research on facilitators and barriers to using evidence in health policymaking, which identified personal contact, timeliness and evidence summaries as most relevant; notably, however, they were relevant to only this policy stream.32 Policymakers also expressed frustration that when they could find the kind of data that they wanted to use, that it was not relevant to the questions they needed to answer, and when they found relevant information, it was typically low quality. Representative responses are provided in table 3.

Table 3

Scientific research findings were desired but inadequate when seeking to influence the political feasibility stream

Lack of relevant, timely studies prevented research findings from being useful	The problem is when they're making decisions like that they're making them under a gun, meaning something has happened or something is happening like a budget crisis. They want quick results and you don't get quick results trying to transform a system and base it on evidence. (P#15, legislator)
	The next challenge is really having the needs of policymakers driving the research. (P#18, administrator)
	[Existing systematic reviews] really can't be tied to any policy issue that you can find. They're driven by either funding from pharmaceutical companies, or an investigator's whim of what's interesting. They're not tied to a political or policy question that has import and that we need evidence for. (P#09, administrator)
	The departments of health in particular, states frequently buy into and pay for non-evidence-based treatment programmes. Somebody's got some small programme in some state or in some town that they believe anecdotally has been eminently successful. [They] sell that programme, when in fact the studies are so small, and so poorly documented, that looking at successes is absolutely, purely anecdotal. It has no scientific basis whatsoever, and yet states are buying into those kinds of programme because there are no good studies that are conducted. (P#11, legislator)

Consistent with prior research, legislators and administrators felt that there was a lack of relevant and high-quality studies that could be used when decisions needed to be made immediately. The kind of information that was available to assess programmes and policies was largely anecdotal, despite policymaker demand. Legislators and administrators expressed a desire for research that was responsive to policy needs, rather than driven solely by the interests of researchers and conducted over long periods of time.

Discussion

Using policy windows theory as a lens to interpret evidence-informed policymaking offers insights into where health research can be influential. Although research on evidence-informed policy has suggested that evidence can be used to clarify problems and build support for policy change (the problem definition stream,)19 20 our findings suggest that given the way that policymakers define problems, this strategy may be more effective if addressed to constituents rather than to policymakers. We found that in assessing and deciding among policies (the policy alternatives stream,) the use of ‘superior stories’ that mixed research and anecdotes (included under some interpretations of realist reviews) was perceived to be effective,26–31 and that policymakers found that simplified evidence assessment heuristics21 were effective in encouraging the use of higher quality assessment. Such heuristics did not require policymakers to rely on interpreting numbers or statistics, which legislators perceived to be beyond their expertise. Finally, in the area of political feasibility (the politics stream,) the nature of current research studies and the time it took to complete them were viewed as problems that prevented scientific research from being relevant to policymaking when policy windows opened.

Researchers advocating for evidence-informed policy have attempted to encourage policymakers to develop a greater understanding of research and researchers to develop a better understanding of the policymaking process. Our study suggests some of the outcomes of pursuing this strategy, and some useful modifications based on more explicit disaggregation of the steps in the policymaking process. Specifically, we found that policymakers who had been trained in interpreting research maintained a strict definition of what constituted evidence, and that they were able to use this definition to discredit what they perceived to be poor quality or misleading evidence by asking if evidence presented to them consisted of randomised controlled trials, used large samples, included a relevant control group or had found statistically significant results. We also found that these policymakers viewed evidence as less useful in identifying health issues as problems, very useful in identifying relevant and effective policy alternatives, and currently inadequate to establish political feasibility. Our results suggest that policy-relevant research will be most valuable in identifying solutions to problems and convincing policymakers that these solutions are effective. Finally, legislators and administrators found that linking stories to research and providing simplified guidelines to assess the quality of evidence encouraged the creation of evidence-informed policy. These strategies are not specific to time or place, suggesting their long-term relevance in multiple policymaking arenas.

Limitations

This research relied on a small sample and self-selected population of legislators and administrators in the USA who sought out additional training on how to use scientific evidence in policymaking. As a result, they are not necessarily representative of policymakers in general. Nonetheless, respondents believed that they were often representative of policymakers who focus on health issues, in the USA and internationally, and in multiple cases, these respondents indicated that they provided relevant expertise to their colleagues that affected policy outcomes. In addition, as noted in methods, the interviews were conducted in 2005. In the USA, policymaking has become increasingly polarised along ideological lines in the early 21st century. Some of the strategies suggested by these policymakers may be less relevant during periods when ideology is more relevant to decision making, however, policymakers themselves indicated that evidence could be even more relevant in periods of strong polarisation given that there was no fear of losing office. Regardless, political climates vary widely within the USA and internationally, and change over time, implying that these findings will be relevant in some places even when they are not relevant in all places. In addition, although clinical research is often time-dependent and location-specific, past research on policymaking has found that relevant concepts and practices identified through surveys of political decision makers typically remain relevant for decades after data collection, and can be applied across different regions and countries.48 This widespread applicability is particularly plausible given that many policymakers in the USA and other countries do not face limitations on terms of service, unlike some of the legislators in this sample. Given that decision makers may retain their positions for decades, they are able to develop and expand on their expertise over extended terms of service.

Implications

Past research on evidence-informed policymaking has sought ways to integrate research into multiple stages of the policymaking process. In this study, we applied research on the nature of the policymaking process to better understand how research is used and could be used in the decision making on health policy. We relied on a unique sample of policymakers, who had completed training that has been proposed in multiple programmes that seek to integrate research into health policy.

Applying policy windows theory to research on the use of evidence in health policymaking offers explicit insights into how researchers who seek to integrate evidence into policy might best direct their efforts. We found that even policymakers with training in interpreting research evidence saw a limited role for evidence as they defined it in defining policy problems, an extensive role for such evidence in assessing policy alternatives, and substantial unmet needs for research in making decisions in the moments when policy windows opened and decisions were made. Policymakers can learn from these findings by refining their understanding of where to present evidence in the policymaking process, and how to make arguments about what constitutes credible evidence to colleagues and constituents who may have not have as much expertise with research. Public managers may also find the strategies identified by the administrators in this sample, such as including stories and applying simplified decision heuristics, useful in communicating with decision makers. Researchers can learn where to target their efforts from these findings, which suggest that they will have the most success when they focus on providing evidence for the policy stream by assessing policy alternatives and building ‘superior stories’ in support of proposals supported by research evidence.

Conclusions

The application of policy windows theory clarifies areas in policymaking where evidence has been useful, where it is unlikely to be useful, and where changes in research strategies could influence policy outcomes. We found that health policymakers trained in the use of research evidence maintained a strict definition of the term ‘evidence’ in their work, and used it to discredit advocacy that they believed was unreliable. In addition, they viewed evidence, as they defined it, as most useful in identifying and supporting effective policies, and less useful in identifying problems that policymakers were called on to resolve or in strategising to find politically feasible alternatives. The perception that health research was not policy focused was repeatedly expressed by policymakers when discussing the political feasibility stream. The findings from this research suggest possible approaches to addressing what past research has identified as an underuse of evidence in health policymaking. These approaches may reflect, to some extent, the nature of research in health, which often focuses on individual clinical interventions, rather than on the effects of interventions implemented at the organisational or policy level.