The utilisation of health research in policy-making: concepts, examples and methods of assessment.

The importance of health research utilisation in policy-making, and of understanding the mechanisms involved, is increasingly recognised. Recent reports calling for more resources to improve health in developing countries, and global pressures for accountability, draw greater attention to research-informed policy-making. Key utilisation issues have been described for at least twenty years, but the growing focus on health research systems creates additional dimensions.The utilisation of health research in policy-making should contribute to policies that may eventually lead to desired outcomes, including health gains. In this article, exploration of these issues is combined with a review of various forms of policy-making. When this is linked to analysis of different types of health research, it assists in building a comprehensive account of the diverse meanings of research utilisation.Previous studies report methods and conceptual frameworks that have been applied, if with varying degrees of success, to record utilisation in policy-making. These studies reveal various examples of research impact within a general picture of underutilisation.Factors potentially enhancing utilisation can be identified by exploration of: priority setting; activities of the health research system at the interface between research and policy-making; and the role of the recipients, or 'receptors', of health research. An interfaces and receptors model provides a framework for analysis.Recommendations about possible methods for assessing health research utilisation follow identification of the purposes of such assessments. Our conclusion is that research utilisation can be better understood, and enhanced, by developing assessment methods informed by conceptual analysis and review of previous studies.

Review

Introduction and Background

The Director General of the World Health Organization (WHO) has decided that the World Health Report 2004, Health Research: Knowledge for Better Health, should involve a careful reflection of how advances in health research lead to improved health and health equity. The WHO has launched a broad Health Research Systems Analysis (HRSA) Initiative that will inform the 2004 report. One component of this initiative is a project focusing on the assessment of health research utilisation. The utilisation project itself consists of various elements. This paper was commissioned by the Research Policy and Co-operation Department of WHO, Geneva, to review the issues related to the utilisation of health research in policy-making, and, based on that review, make recommendations about appropriate methods for assessment of such utilisation.

WHO is giving increased emphasis to the role of health systems [1] and attention is focusing on the importance of policy-making in achieving effective health systems [2,3]. The World Bank made estimates of the costs of attaining the health-related Millennium Development Goals of between $20 and $25 billion a year. However, the report notes that: 'these unit cost estimates only apply when the policy and institutional environment is conducive to additional health spending being effective' [2]. The importance of health policy-making, in turn, being research-informed is recognised by a growing number of bodies [3-5].

The existence of relevant research, though necessary, is not sufficient. Evidence-based policy is difficult to achieve and it is widely agreed that health policies do not reflect research evidence to the extent that in theory they could [5-11]. Examination of the policy-making process confirms it to be extremely complex, with many genuine obstacles to evidence-based policy-making at the same time as there are factors that could increase research utilisation. A full review of the many possible meanings of research impact reveals that there may be more utilisation in policy-making than is sometimes recognised. Such a review also enhances understanding of the issues, including the differential scope for utilisation associated with different types of research and policy environments. Developing a conceptual framework of the processes of utilisation should assist with the formulation of assessment tools that reveal the full picture of the way research is used in policy-making. Furthermore, it should allow the growing demands for accountability for research expenditure [12-18] to be addressed appropriately, which could also be of benefit to the research community.

There is a rich background of material for each of these areas, including key contributions from Weiss identifying the multiple meanings that can be attached to research utilisation in policy-making [19]. Their importance lies in the fact that some of these meanings, or models, point to less obvious patterns of use than those suggested by instrumental research exploitation which involves research findings being directly used in policy formulation.

About twenty years ago there was recognition of the need for analysis to combine a range of factors such as the nature of different types of health research knowledge and the diverse institutional arrangements for policy-making. In their assessment of the attempt in the 1970s to increase utilisation of research funded by the UK's Department of Health, Kogan and Henkel found, 'the interconnections between epistemologies and institutional relationships were a recurring theme' [7]. The importance of interactions across the interfaces between researchers and policy-makers was identified. The role of policy-makers as the receivers, or receptors, of research and the need for careful priority setting were highlighted. Various elements of this analysis were recently reported also to be relevant for health research in Mexico [10].

The context of the current move to attempt to increase research utilisation is important. There is now a broad coalition pressing for improvements. Various organisations came together in 2000 to support the formation of the Alliance for Health Policy and Systems Research with its 300 partner institutions. It aims to promote capacity building for, and the dissemination and impact of, research both on and for policy [20]. At the level of specific programmes within international bodies, there is a growing stress on the role of policy-making: 'Research on implementation, on policy-making or programme development is as important as basic clinical research for improving child health' [21]. Recent weeks have seen publication of the first systematic review to address research utilisation in policy-making [22]. Furthermore, the developing interest in research informed policy-making coincides with the extensive efforts being made to increase the implementation of health research findings more generally. Indeed, the emphasis on evidence-based medicine is itself generating extra pressure from practitioners that policy-makers, too, should have a duty to consider research evidence [23]. The role of research utilisation in policy-making is seen as a key element in the growing interest within WHO on research utilisation and its assessment [3,24].

A further important part of the context is that developments in the UK in the 1970s, and in other European countries [25], could be seen as early attempts to develop a system to augment the traditional individualistic determination of medical research priorities in universities and hospitals. A similar emphasis on issues such as priority setting is seen in recommendations made for middle and low income countries by the Commission on Health Research for Development [26]. The concept of Health Research Systems (HRSs) is now of growing significance [27]. One of the main elements that distinguishes a HRS is the attempt to develop mechanisms and networks to facilitate the greater use of health research.

Building on the above analysis, it is our contention that many factors need to be brought together if assessment of research impact on policy-making is to contribute to an understanding of the issues and an enhancement of utilisation. The prime focus should be the policy-maker. This paper first examines the concept of policy-making, and the underlying assumption that it is better if it is research-informed. Then we consider the range of types of health research and the levels of policy-making at which they could be applied. These strands are brought together to provide an analysis of the wide range of ways in which research can have an impact on health policy-making.

The focus then moves to examining contributions from previous studies of knowledge utilisation in health policy-making, including those using standardised measures. Various dimensions of our conceptual analysis form the next sections. We start with the interfaces, both at the priority setting stage and when findings are communicated between researchers and policy-makers. The role of policy-makers as receivers, or receptors, of research is examined along with the accompanying institutional arrangements. Incentives are also important. The material is brought together in a wide-ranging interfaces and receptor model of research utilisation in policy making. Finally, the various possible purposes of assessment of research utilisation are considered before suggestions are made about suitable methods for assessing the impact of research on policy-making. Such assessments would be best undertaken as part of a wider evaluation of the utilisation of health research by industry, medical practitioners and the public.

The nature of policy-making and its role in knowledge utilisation

Policy-making can be viewed as involving the 'authoritative allocation of values' [28], and when interpreted broadly can include people making the policy as government ministers and officials, as local health service managers, or as representatives of a professional body. Policy-making involves those in positions of authority making choices that have a special status within the group to which they will apply. The results take many forms ranging from national health policies made by the government to clinical guidelines determined by professional bodies. This broad usage of the term policy-making has advantages when conducting knowledge utilisation or payback assessments, and has contributed to a conceptual framework for a series of such studies [14,17,29]. In this article, however, the analysis mainly concentrates on public policy-making rather than that undertaken by professional bodies.

This framework consists of two elements. These are a multidimensional categorisation of benefits from health research, going through from the primary and secondary outputs to the final outcomes, and a model of how to assess them. A revised version of the model is shown as Figure 1 and consists of a series of stages. This sequence can be useful when examining how a health research project could be utilised, in policy-making and practice, in ways that result in final outcomes such as health gains and economic benefits. Public engagement with research can play a key role in research utilisation. The model incorporates the concept of the stock, or pool, of knowledge and the idea that there are various interfaces between research and the wider political, professional and social environments. These points, together with various feedback loops and forward leaps, mean that although the stages are presented in a linear form, the model recognises that the actual steps involved in utilisation and achieving final outcomes are often multidirectional and convoluted. That said, the model helps both to organise assessments and to indicate where the various elements of the multidimensional categorisation of benefits might occur.

Figure 1

The Place of Policy-Making in the Stages of Assessment of Research Utilisation and Final Outcomes.

Key:

Direct lines within the flow or feedback

Indirect lines of communication

Primary Outputs – Publications, trained researchers

Secondary Outputs – Policies from national, local and professional bodies

Final Outcomes – Health and equity gains, cost-effectiveness and economic benefits

Source: Adapted from S Hanney, S Kuruvilla: HRSPA Project 4: Utilisation of research to inform policy, practice and public understanding and improve health and health equity. WHO/Wellcome Trust Technical Workshop. London, January 2002; and S Hanney et al 2000, Evaluation, 6, published by Sage [29].

The framework is also important for the structure of this article because it demonstrates that assessment of research impact on policy-making is best undertaken as part of a wider analysis of the utilisation of research. Throughout the paper it will become increasingly clear that policy-making is itself influenced by industry, by health professionals who might be expected to apply research findings in their practice, and by the public who might engage with research, either as patients or more generally in society. The interaction of all these groups with research findings is an important consideration and the interfaces operate at many levels.

In terms of the utilisation of the knowledge, research-informed policies can be referred to as secondary outputs from research [14]. This distinguishes them from the primary, or direct, outputs of research processes such as journal articles, other publications, and trained researchers. Neither, however, are the policies the desired final outcomes; they represent a step in the process. It is sometimes possible to identify how research findings have informed policy-making even when it might be extremely difficult to trace influences at other stages in the utilisation processes. Furthermore, the approach enables the processes of research utilisation to be identified in ways that would be impossible if the analysis attempted to jump immediately to the final outcomes. In particular, detailed analysis at this stage can address the counterfactual, ie consider what might have happened without the relevant research: would the policy have been changed anyway?

Not all examples of health knowledge utilisation go through a policy-making stage, and in some cases the policy comes after partial translation of the findings into practice. For example, clinical guidelines are usually developed after leading clinicians in the field have already adopted an evidence-based practice and then seek to encourage its wider diffusion throughout the profession. Nevertheless, often a policy-making stage in knowledge utilisation is important if the final outcomes of health, health equity, and social and economic gain are to be achieved. The potential importance of a policy stage in the process of turning evidence into application is increasingly being accepted, even for clinical practice [30].

A positive case can be set out for the contribution research can make to policy-making. The basic assumption of knowledge utilisation related to policy-making is that policies which are research informed will be better than otherwise would have been the case. It is assumed that research exposes policy-making to a wider range of validated concepts and experiences than those that can be drawn from the normal time-limited and politically constrained processes of policy deliberation. It thus allows a broader choice of policy options to emerge. Research often enables policies to be generated upon technically well-informed bases. It gives warnings of reasons why some policies succeed and others fail. It can make connections between otherwise separate factors such as the nature of the substantive field and organisational patterns set up to manage them, or the power of environments over health outcomes. It legitimises some policies and throws legitimate doubts on others. Analysis of policy-making, and of research utilisation, often identifies at least three broad areas of activity: policy agenda setting, policy formulation, and policy implementation [11,31]. Potentially, research could play a part in all three areas. Evaluation is also often seen as an important activity, and one that adopts a research approach. Indeed, in this paper evaluation is primarily being viewed as a form of research, the utilisation of which will be examined in the other phases of policy-making.

Davies and Nutley, in a recent analysis of the role of evidence, or 'What Works', in a series of public services, suggest 'the research community in healthcare is truly global, and the drive to evidence-based policy and practice is pandemic' [32]. Of the public services, health care is seen as the one where, despite the difficulties, the utilisation of knowledge is most advanced. Although this analysis does examine a range of policy-making models, critics claim that the theoretical basis for evidence-based policy-making is not strong because it rests too much on a rational view of policy-making [33,34]. The debate needs to be informed by various models of policy-making such as those set out below.

Many categorisations of policy-making exist. The categorisation of policy-making presented here is not intended to describe the models comprehensively. Instead, it is based on previous analyses of public policy-making that were specifically made in the process of analysing research utilisation. The categorisation incorporates work undertaken by Kogan and Tuijnman [35], for the Organisation for Economic Co-operation and Development, and by many others [9,36-38]. The various models described are not mutually exclusive, but are included because each makes a specific contribution that is built on in later analysis:

Rational models

Rational models of policy-making assume policy-makers identify problems, then gather and review all the data about alternative possible solutions, and their consequences, and select the solution that best matches their goals. Sometimes this approach is known as ends-means rationality; it is thus different from some of the models below which might, nevertheless, also seem rational to the policy-makers involved. The various models of policy-making should be seen as a spectrum. Thus Simon [39] is sometimes seen as writing from the rationalist tradition, but he was critical of the more basic rational models and his concept of 'bounded rationality' involves concentrating the review of data on a more limited range of possible solutions.

Incrementalist models

It has long been recognised that policy-making is a complex process. It can involve scientific knowledge and a range of other factors including interests, values, established positions within institutions, and personal ambitions. Furthermore, evidence from research has to compete with what Lindblom and Cohen [40] call 'ordinary knowledge' which owes its origins to, 'common sense, casual empiricism, or thoughtful speculation and analysis'. In models such as 'disjointed incrementalism' [41] policy-making does not involve a clear movement towards predetermined goals but rather is more a series of small steps in a process of 'muddling through' [42] or 'decision accretion' [43]. Incrementalists allow for a greater role for interests in policy-making debates and emphasise the many sources of information that impinge on policy-makers.

Networks

A networks approach also highlights the role of different interests and how the relationships between such groups and policy-makers can result in an incremental policy process. The term 'policy network' is defined as a generic label for the different types of state/interest group relationships, for example 'policy communities' in which the long term relationships between government officials and representatives of leading interest groups are particularly powerful [44-46]. Other definitions of the term networks involve a wider membership and are more likely to include researchers. It is claimed that researcher involvement in 'social networks' [47] is important for research utilisation. Others suggest that leading experts who share a similar approach on an issue can be seen as an 'epistemic community' [48] and can influence policy. Analysis of health systems often suggests the influence of the medical profession over policy-making is particularly strong [49]. Its domination of the policy networks led to the use of the term 'a professionalised policy network' [50,51]. Its influence is likely to be a factor in setting agendas and determining the type of knowledge to which most notice is taken in the policy debate.

The 'garbage can' model

The 'garbage can' model of policy-making [52] looks at these issues in an idiosyncratic way. It suggests that sometimes solutions that might have been disgarded nevertheless remain in the policy-making system, and occasionally there are problems to which they become attached. Models such as this highlight the way in which policy-making can be seen as a most untidy process, rather than neatly going through a series of phases [53].

These various models of policy-making, even occasionally the final one, are likely to be found relevant to different circumstances and parts of health systems. They will have different implications for the utilisation of research and although they do not stack up as connected paradigms, or have much predictive power, they help put shape onto otherwise inchoate patterns. We shall explore how far they specifically map on to models of research utilisation after considering the range of overlapping categorisations of health research.

Categories of health research and possible levels of utilisation

The categorisation of health research discussed here has a potential importance for the analysis of utilisation. Often a broad distinction is made between basic, clinical and applied research. By its nature basic, or blue-skies, research is not often likely to be utilised until further, less basic, research has been undertaken and perhaps some synthesis with other findings has occurred. Research that follows priorities determined by the researchers themselves, according to the 'internalist' norms of science [54], is more often, though not always, going to be basic. Applied research is more likely than basic research to be following an agenda driven by forces other than the scientific imperative. Just because the research topic has been set by non-researchers does not, of course, ensure its impact. Nevertheless, where such drivers and sponsors are also the most likely potential users of the research, this provides some of the circumstances that might encourage utilisation [7,53,55].

There is generally greater resistance within health services to the use of social science, despite it often being applied and user-driven, than there is to the adoption of the findings from natural sciences [7,10] such as those used in clinical science. Possible explanations include the fact that the more highly technical and specific the research, the more there might be circumstances in which it can be utilised directly by policy makers without ideological or political considerations intervening too much. Moreover, the receptors of research are likely to place more confidence in the strictly controlled natural sciences than in the more eclectic social sciences. Much of policy-making can be seen as a craft, which draws substantially on ordinary knowledge and in which the contextual component will often be more significant than the type of evidence offered by social science research [56].

Another partially overlapping distinction is between national and international research. International research findings might be more likely to be utilised where there is greater technical content in the research and also potential for application to an issue of patient care. The report of the Commission on Health Research for Development also identified the particular contributions that national and global health research could make [26]. It developed the concept of Essential National Health Research (ENHR). This entails a strategy in which each country plans its health research according to country-specific health problems and the contribution it can make to regional and global health research. Mechanisms for the synthesis or systematic review of research might become even more important in relation to international research.

Adding to the epistemological debate about the most appropriate forms of production of knowledge intended for utilisation, Trist [57] argued that domain-based research represented a third category alongside basic and applied research. Domain-based, or policy-oriented, research is essentially interdisciplinary and the crossing of new boundaries and the creation of new syntheses may advance both knowledge and human betterment. It also entails wider reference groups, beyond the scientific or clinical communities. Along similar lines, Gibbons et al [58] claim to identify a shift from the traditional discipline-centred mode of knowledge production that they characterise as Mode 1, towards a broader conception of knowledge production described as Mode 2. In this, knowledge is generated in a context of application and addresses problems identified through continual negotiation between actors from a variety of settings. The results are communicated to those who have participated in their production. Although the degree of change described by Gibbons et al could be exaggerated [59], this general approach, as with that of Trist, is compatible with attempts to increase utilisation by focusing research production on the interests of at least some potential users.

A slightly different dimension, but one also associated with utilisation, is that of the features of specific research studies. When a particular piece of research is seen to be of high quality this might help reinforce a policy-maker's inclination to use it [14,60], as might a favourable view about the quality of the specific researcher [10]. The argument, as developed by Weiss and Bucuvalas, is that where the policies are potentially controversial the decision-makers will not want the credibility of their case undermined by critics pointing to flaws in the research behind the policy [60]. When Ministers in the UK supported actions to address gender inequalities in the medical profession, they did so with full confidence in the quality of the research that demonstrated the problem [61]. Examinations of the use of economic evaluations and Health Technology Assessments (HTAs) in policy-making have considered the importance of the quality, reliability, timeliness and comprehensiveness of research in influencing the level of utilisation [62-67]. For example, the latter two factors were highlighted as important determinants of the usefulness of the information in the context of drug formulary decisions in the USA [68].

Different types of research are likely to be most relevant for various levels and situations of policy-making, and for different aspects of those policies. There is no agreed typology of policy categories suitable for utilisation assessment [11]. Above, we suggest that the interpretation of policy being adopted here covers national policies, local health service policies and policies made by professional bodies. Along not dissimilar lines Black [34] argues that an earlier threefold categorisation [69] could be appropriate when examining health research and policy-making. The three categories are: governance policies which relate to organisational and financial structures; service policies which cover resource allocation issues and pattern of services; and practice policies which relate to the use of resources by practitioners in delivering patient care. A similar division appears in the threefold categorisation proposed by Lomas: 'legislative, administrative and clinical' [70]. Legislative policies relate to the overall framework for organising health services; administrative to the running of the service and allocation of resources within the overall framework; and clinical to the policies about what therapies are applied. These categorisations are best seen as a spectrum, but it is generally agreed that research has least impact on the first of these categories and most on the third where often the relevant knowledge comes from clinical research. This is despite the frequent delays in turning research evidence into improved patient care [71,72].

Some of the issues in this section are illustrated in relation to Health Technology Assessments (HTAs). Various features of HTAs might be associated with the sometimes quite high levels of translation into policy-making and through into the final outcomes [14,73-75]. Many HTAs are undertaken, commissioned, or produced by technology 'sponsors' specifically for agencies set up to advise governmental bodies setting policies for delivering patient care in national health systems. Frequently they address a very specific question that has been identified and prioritised by the health care system: presumably, a question to which the system wants an answer, and by implication is willing and able to accept alternative outcomes if they can be supported by evidence.

Whilst these HTAs are 'technical' in the sense that they typically relate to quantitative measures of effectiveness and cost-effectiveness of specific interventions, they do have important distributional and equity implications. Policies deriving from them may induce strong public and patient reactions (as is evident in media coverage of proposed guidance from bodies such as The National Institute for Clinical Excellence in the UK). This emphasises the need in such systems to differentiate between the research activity of health technology assessment and the decision-making (or guidance forming) process of appraisal of that evidence and its implications [76].

But even for HTA the evidence of widespread, direct impact on policy (with policy seen as entirely convergent with the research evidence) is at best patchy. A study in the Netherlands by van den Heuvel et al [77] concludes that policy decisions concerning the introduction of (new) technologies in health care are not based on the results of medical technology assessments. Rather, 'political arguments and interest groups decide the outcomes'. In a recent literature review, Barbieri and Drummond [78] found few examples of HTA impact in European health care systems. At the local level, those involved in making policies about the introduction of new medical technologies are likely to view the contribution that effectiveness research can make in different ways, depending on their professional backgrounds [79]. This discussion underlines the fact that even in the circumstances most favourable for 'rational' policy-making there are limitations upon it. This indicates the need to consider the full range of models of research utilisation.

Models of research utilisation

Having reviewed various models of policy-making in the second section, and examined different types of research in the third section, it will now be useful to consider models of research utilisation in policy-making. Then we can see how far the various strands from these sections can be drawn together and developed in our later construction of a conceptual framework. We start by looking at previous models of research utilisation, and then suggest ways in which they could be elaborated.

Following the work of Weiss [19,80], and others [7,14,37,38,55,81,82], various models of research utilisation in policy-making have been identified, and they are thought to be applicable beyond the social sciences:

The classic/purist/knowledge-driven model

This suggests a linear sequence in which research generates knowledge that impels action.

The problem-solving/engineering/policy-driven model

This also follows a linear sequence, but begins with the identification of a problem by a customer who requests the researcher to identify and assess alternative solutions. This was explicitly the model behind the changes attempted by the UK Department of Health in the 1970s [7].

The interactive/social interaction model

The process here is a set of interactions between researchers and users rather than a linear move from research to decisions. It ensures they are exposed to each other's worlds and needs.

The enlightenment/percolation/limestone model

According to this, research is more likely to be used through the gradual 'sedimentation' of insight, theories, concepts and perspectives. This model has the advantage of extending the range of ways in which research is seen to be utilised.

The political model

In this, research findings become ammunition in an adversarial system of policy making.

The tactical model

Here research is used when there is pressure for action to be taken on an issue, and policy-makers respond by announcing that they have commissioned a research study on the matter. Whilst this can sometimes be seen as a cynical delaying tactic, there are other occasions on which the commissioning of research provides the political system with a valuable breathing space, thus reducing the chances of irrational policy-making.

There is no precise overlap between the principal characteristics of policy-making models discussed earlier and the utilisation processes listed above such as would allow them to be presented in neat pairs of singletons. The first two categories of utilisation both fit with rational models of policy-making, but it is the problem-solving model that shares the same starting point: identification of a problem by a policy-maker. The more incremental models of policy-making have the longer time frame implied by interactive and enlightenment models of utilisation, but sometimes these forms of utilisation lead to paradigm shifts which are much more radical than is inherent in incrementalism.

Policy networks are seen as providing a useful framework for studying research utilisation [36]. Where researchers become part of a policy network, or find their ideas taken up by elements within it, this could be a strong version of the interactive model and be an important route for such findings to enter the policy arena. Network approaches can highlight the role of stakeholders in research utilisation [9,14]. (The network concepts could also, however, help to explain the difficulties some research faces in gaining acceptance, or even a hearing. Policy-making systems can be relatively impermeable to research findings that are contrary to the consensus developed as a result of the strong, long-term, links between departmental officials and leaders of the main interest groups).

It is of value to explore the variety of policy-making/utilisation connections because they underline the argument that it is not realistic to expect policy-making always to follow the ends-means rational model that might entail the clearest use of research. Weiss also suggests that there are three main forms in which research might appear and be utilised in policy-making: as data and findings; as ideas and criticism in the enlightenment mode; or as briefs and arguments for action [83]. Along similar lines the utilisation of research in policy-making is sometimes considered to be instrumental, conceptual or symbolic [11]. As we have seen, instrumental use involves research findings being directly used in policy formulation, conceptual use refers to a type of enlightenment mode of utilisation, and symbolic to the use of the research to support a position already taken, which may be to continue with existing policies.

Taking another of Weiss's arguments, that utilisation of research can be usefully be defined as a process of interaction between research inputs and decision outputs [43], we next elaborate the range of possible uses of research. Given the diversity of forms of knowledge and policy decision, their interaction has to be understood in the context of both the diverse values shaped by philosophies of knowledge and the practical aspects of policy-making. With regard to the former, policy-makers may privilege empirical findings against more abstract and general models of reality. In terms of policy-making it is useful to distinguish three dimensions: the nature of decisions, that is, the extent to which they are explicit and specific versus implicit and diffuse; the extent of choice available in a given situation; and the political or technical character of actors participating in decisions. These are shown on Figure 2 and developed below but it is useful to consider the categories as a spectrum.

Figure 2

Decision Context, Research Inputs and Forms of Research Utilisation in Policy-Making

Sometimes policy-makers make rational and weighted decisions along a well charted course of action, yet more often apply knowledge through largely routine or unconscious processes in response to ad hoc situations; here the context is implicit rather than explicit. A situation of choice will exist when several alternatives are perceived as viable, contrasting with a situation where a decision has been taken and the role for research is rather to support this choice. Support in turn covers two types of situation. Specific findings can be used to legitimate decisions when these have been formed, have hardened or when they are being implemented. In relation to the concept of models referred to above, support is more a matter of explicit policies being seen to be made by institutions that are research-based and therefore the policies gaining greater credibility. Political decisions are normally justified in terms of social values and understandings shaped in the political arena, but there can be a role for scientific inputs in the policy formulation. Technical decisions are those that are expected to be justified in terms of scientific or specialised methods.

The combination of diverse forms of scientific inputs and decision outputs shapes the processes of utilisation and creates specific expectations and opportunities. The components of Figure 2 are expounded in more detail as follows:

Conceptual modelling

Knowledge to inform complex situations is frequently demanded in the form of concepts to model or shape the general nature of the policy problems and possible solutions. Planning health sector reforms or identifying health policy in areas normally outside its purview, such as poverty or economic development, are likely to demand such knowledge, as they provide new disciplinary or social perspectives on a given problem and activate new associations and meanings for policy issues [84]. They can be a first step to other forms of research utilisation.

Data-based policy

This form of utilisation aims to influence courses of action on the basis of the strength of empirical findings. Scientists may take the lead through a 'knowledge-driven' approach, or policy-makers can demand such knowledge to solve specific problems ('policy-driven' model). In either case scientific rigour, robustness and objectivity would be principles trusted by both researchers and policy-makers.

Constrained modelling

Constrained political conditions give rise to utilisation that, from the perspective of researchers, uses only a restricted range of available knowledge. Likewise, policy-makers will not commission or will discourage research that, in its broad outlines, poses more political risks than benefits [85].

Strategic research

Policy is most often formulated in a context where lay (as opposed to technocratic) actors vie for power and resources. The choice of policy may be open, but only through politically controlled windows of opportunity. Under these circumstances the ultimate validity of research will be assessed together with other and often competing evidence. The aim of researchers is usually to influence policy choice or to make explicit the costs of not adopting a recommended course of action [86].

Symbolic payback

Science has become a potent cultural symbol that permeates modern life and confers privileges on its users. Likewise, there is a political pay-off in supporting research and building research capacity in strategic areas. Research has become 'an essential mode of communication and persuasion in the public arena' [43]. In complex organisations, research could be a common language used to talk across the boundaries of interests and content areas given its capacity to effectively link disparate realities and because of 'the patina of rigor that science confers to discourse'. This might suggest that policies from bodies known to be research-informed might be more likely to be supported.

Symbolic argumentation

Policy making may be based mainly on reasons of interest, ideology or intellect. Under these circumstances, however, research can still be used as ammunition to support the decisions made and being implemented. Science content is here used as a collection of arguments, rather than as data or evidence to be weighed. Arguments may be fashioned as by-products of formal research publications, particularly by policy analysis units, consulting firms and the media [87].

Paradigms

Given the large measure of unconscious elements in every-day decision making, accepted ways of interpreting reality and facing problems are the most important influence. An aggregate of normative expectations may amount to an overriding view of what is desirable health policy, such as those advanced in Welfare State thinking. Such policy paradigms may be triggered off or supported by single or grouped assumptions derived from research, which also may achieve paradigmal status. Individual policies are likely to reflect the dominant paradigms of their time.

Policy-Makers' practice wisdom

How far individual policy-makers will automatically attempt to use research findings on a regular basis will depend on multiple influences, such as training, continuous education, exposure to the media and to the demand of clients.

Although these categories are not water-tight, they help indicate the breadth of types of research utilisation and, therefore, areas on which any assessment methods should focus.

Various elements from this and preceding sections will be particularly useful as we develop our own conceptual framework for analysing and assessing research utilisation in the context of the increasing attention on HRSs. For example, the importance, but also the limitations, of the problem-solving model must be considered when examining the role of research priority setting. Furthermore, in light of the practical limitations on rational models, the importance of interactive perspectives will be highlighted as a way of encouraging policy-makers to be responsive to relevant research. Overall, both these points, and others, suggest that it is appropriate to focus on the actions that could be taken to encourage permeability at the interfaces between policy-makers and researchers. Such actions should help ensure both that researchers are aware of policy-makers' needs, and that the policy-making system is willing and able to absorb relevant research findings.

It is also possible to link some of the ideas in this section to the three phases of policy-making referred to earlier: agenda setting; policy formulation; and policy implementation. For agenda setting the research could impact in one of several ways. Research could demonstrate the existence or extent of a problem, through either specific findings or a process of enlightenment. Alternatively, it could be that, as in the knowledge-driven model, the mere generation of the findings leads to pressure to act upon the new knowledge. The use of research in policy formulation could be in either the instrumental or conceptual/ enlightenment mode. A further possibility is that it could be used as briefs to inform arguments as set out in the political model of research use. The implementation of health policies is widely acknowledged often to be difficult [37,53]. At the implementation stage, research could play some part in demonstrating the best way to implement policy and could inform decisions. It could also be of symbolic use in helping to build support for implementation through assistance with communicating or justifying the policy and being used to generate support for it in terms of financial resources, political commitment, and public opinion.

Before developing the material into a conceptual framework and methods for assessment of utilisation, it will be useful to review the focus and methods adopted in previous studies of policy-making. This review will provide examples of where wide interpretations of utilisation have been incorporated into studies of the impact of research.

Contributions from selected previous studies of health policy-making and knowledge utilisation

Studies of the nature and impacts of policy-making can each exploit a wide range of analytic methods. The area covered by them is extensive and here we review one study: that of policy change in relation to health financing reforms in South Africa and Zambia [31]. It illustrates several of the approaches that can be applied when analysing policy-making in general, and is included here as a backdrop to the accounts, immediately following this one, specifically on research utilisation in policy-making. The case study in each country was organised according to a conceptual framework consisting of a process of policy change moving from, Gilson et al state, 'agenda setting around a reform of focus, to reform design, and then through implementation to the achievement of immediate and longer term changes' [31]. The policy analysis approach of Walt [37,88] was also drawn upon so that the factors influencing each stage of the reform process were categorised and analysed according to four broad factors: context, actors, process and content.

In the two countries data were collected through: documentary analysis; key informant interviews with policy-makers and analysts; media analysis; and review of secondary sources. The data analysis techniques included: development of a timeline for each reform; stakeholder analysis; policy mapping techniques; impact analysis through use of secondary data; and a review process. The two case studies incorporated examination of the impact on the policy development made by research analysts, both inside and outside government, and found it to be 'strongly dependent upon the presence of a policy champion' [31].

This section now draws selectively upon a review of a wide range of previous studies specifically about research utilisation in health policy. From this several key themes are discussed:

• focus of the study;

• how far the study was based around a conceptual framework and how far comparisons or conclusions were drawn;

• methods and standardised measures used in assessing impacts and outcomes;

• levels of utilisation and other benefits shown.

Focus of the study

Studies that start with the research project or programme, and examine the impact that it has had [14,17,75,89,90], have the advantage of a reasonably tightly defined focus. They can be of use for research funders, but often run into the problem that any impact research makes is usually of a contributive nature and it is difficult to identify the impact of one project or programme from that of others with which the findings get mixed [16]. Other studies consider the portfolio of researchers' work over a particular period [91], or the contribution of specific health research centres [29,92-95]. Studies such as these lend themselves to network analysis.

Health policy-makers have been the focus of studies. Some examine the policy-makers' use of research in general [96]. Others can involve health policy-makers, for example in mental health, being shown research papers describing evaluations of programmes and then asked how useful they would find such research [60]. Drummond et al asked local policy-makers about their attitude to using economic evaluations in general, and whether they had used specific evaluations [65]. A major way in which impact on policy has been assessed is through studies that start with a policy area, or theme, and then seek to identify how far the policy-making or implementation has been informed by research [11,77,96-102]. Focusing on the policy area has the advantage of facilitating some assessment of how far lack of relevant research is the problem, as opposed to the underutilisation of existing research.

The two broad approaches, starting with research or starting with policy, have elements of overlap. A study of the role of research in the regulation of private health care providers in an Indian state focused on the activities of a key research centre [103]. In some studies of the impact of research on particular health care programmes, interviews with researchers produce examples of how their own research has been utilised [10,104]

Some studies have cases from a range of countries and have been organised by, or involved, international bodies including the Council on Health Research for Development (COHRED) [5], the European Union (EU) [101], and the Cochrane Collaboration [102]. Other studies explicitly take examples from two or more sub-national units [11]. Many studies cover a series of examples from the same country or sub-national unit.

Conceptual framework and comparisons

Some of the most illuminating studies are organised according to a conceptual framework. Thus Landry et al [91] attempted to operationalise Knott and Wildavsky's Ladder of Knowledge Utilisation which suggests there are six stages of utilisation [105]. These are: transmission, cognition, reference, effort, influence, and application. Generally, studies that adopt a conceptual framework involve a series of cases. Buxton and Hanney [14,17,29,106] organised several series of studies around the conceptual framework described in Figure 1. Walt's framework for analysing health policy-making [88], described earlier, was used to organise impact studies in Mexico [10]. The EU-funded set of studies in developing and developed countries [101] was based around a conceptual framework, as described by Sauerborn et al [9], that concentrates on analysis of the role of the various interests or stakeholders.

A key feature of all these studies is that the conceptual framework facilitates comparable analysis of the results. A considerable amount of this analysis is drawn upon elsewhere in this paper, but one example is given here. Lessons were learnt from the EU-funded studies regarding: factors enhancing or hindering research use; possible indicators to measure research use; and possible strategies to increase such use. However, it has become clear that making generalisations is difficult because of country specific factors such as differing contexts (including the political system and culture), differing stakeholder constellations, and differing availability and quality of research. A conclusion, therefore, is that careful consideration of these factors, rather than recipe-like approaches, will be needed for successful enhancement of the use of research. (A. Gerhardus-personal communication). Similarly, Berridge and Stanton [107], after reviewing various case studies that were not undertaken as part of a set, identify a series of factors considered important in the research/policy relationship. They go on, however, to note that 'they are necessarily historically determined, and culture and context specific, rather than part of a reproducible general formula for action' [107].

There are clear tensions in our analysis between recognition of the genuine limitations in the ability to make generalisations and a desire to learn as much as possible from comparisons. It is useful, therefore, to note the findings from a study that addresses the issue of how far analysis from developed countries might be appropriate elsewhere. Trostle et al found there were various issues where, in comparison with developed countries, there could be different emphases in Mexico and 'in other developing country contexts' [10]. Nevertheless, most of the factors that Husén and Kogan [108] had identified as encouraging utilisation of educational research in industrialised nations were, they state, 'also found to be important in our study. These included decision-makers' willingness to consider research results as input for decision-making, and political stability...and the existence of research networks or commissions which provide a favourable arena for interaction between research and decision-making' [10].

Methods and standardised measures used in assessing impacts and outcomes

The two methods used most frequently, and usually used together, come from the qualitative tradition: documentary analysis and in-depth interviews [10,11,14,63,75,77,97,98,104]. The need for having the flexibility interviewing can provide is well illustrated by studies in which the original interview schedule had to be amended to take account of the different perspectives held by policy-makers of the role of research [79]. Interviewing is useful for understanding many aspects of research utilisation, including tracing networks between researchers and users [90].

Some studies use insider knowledge [94,102] and there has been some adoption of questionnaires to researchers about the utilisation of their work [11,106]. Particularly where the policy-making is at a local level, questionnaires have been used and administered either by phone [68] or by post [65]. In the latter case, Drummond et al also attempted to assess the problem of inaccurate responses. They included two fictitious studies in the list sent to policy-makers, and almost 20% claimed to have seen these studies; some of those admitted to having been influenced by them [65]. Bibliometric analysis is sometimes incorporated into broader studies [106] and, in an analysis of the papers that were cited in clinical guidelines, Grant et al specifically adopted a bibliometric approach [109]. Historical approaches have also been adopted [99,107] and allow a more contextualised analysis. The methods used are diverse, and only partially depend on the focus and purpose of the studies. The list of studies described here partially overlaps with, but is not identical to, those 24 included in the recent systematic review [22]. The 24 include a greater proportion that use questionnaires administered either by phone or post, but still face-to-face interviews form the majority.

There have been a few recent attempts to scale or score the degree of impact. Four such studies are described below, starting with two where the focus of each case study was a specific piece of research. An assessment of the impact of HTAs in Quebec used a case study approach [75]. Initially seven levels of critical incidents were identified. The impact of each HTA was scored on the basis of documentary analysis, and the information completed and validated through contact with key witnesses. In this way, Jacob and McGregor explain, 'by taking into consideration the level and number of critical incidents, an overall estimate of impact on policy was awarded to each HTA. This was reported in a scale extending from 0 (no impact) to +++(major impact). The weight awarded to critical incidents was adjusted according the nature of the decision at issue' [75].

Despite the largely, though not exclusively, qualitative methods used by Buxton and Hanney, in one study they attempted to score the impact made by projects on the basis of material from questionnaires [106]. This was partially validated by re-scoring those projects for which more detailed information became available from case studies. The correlation was quite good at an overall level, though there were differences in both directions at the individual project level. This scoring was entirely a methodological exercise and the results for specific projects were not identified.

Lavis et al coded their interview material according to whether the research had been used at agenda setting and/or policy formulation stages, and whether the research had impacted on all the policy or only partially [11].

The above three examples involved members of the assessment team undertaking the scoring and coding. The EU-funded project described by Gerhardus et al developed a model for mapping research to policy flows based on the qualitative case studies. From this model a set of numerical indicators was devised which entailed scoring by both the assessment team and stakeholders [101]. The intention, in what is probably the most methodologically ambitious research utilisation study included in this review, was that the indicators would be used in each of the eight countries to facilitate comparisons between research utilisation before and after interventions aimed at increasing such utilisation. In each country a policy was identified along with the content, conclusions and recommendations of the relevant research. Next, a series of questions was put to the stakeholders and points allocated according to their recall of the content etc. Further points were allocated depending on the references to the research made in speeches, statements, guidelines and similar sources. Finally, the stakeholders were asked to rate on a five-point scale a range of factors, including research, that influenced their decision making [101].

Preliminary results from the study suggest that conceptually the set of indicators has proven to be helpful, but problems with computation of the indicators arose due to the generally small sample of stakeholders interviewed. There were also problems related to the data collection for applying the knowledge-related indicators. This part of the stakeholder interviewing frequently created a possibly stressful, or even embarrassing, exam-like situation and in some cases revealed problems due to the considerable recall period involved when probing about older pieces of research. The body of indicators developed in the project was quite large, and it is considered appropriate to reduce it to a simplified set of core indicators (U. Sunderbrink – personal communication).

Level of utilisation and final outcomes

The examples suggest a greater level of utilisation and final outcomes in terms of health, health equity and social and economic gain than is often assumed, whilst still showing much underutilisation. There is considerable variation, both within and between studies. The study of the role of research in child health policy and programmes in Pakistan [104] found some examples of immediate clear-cut linkage between research and decisions, but in general the view was that research was little utilised. A mixed picture was reported in the Mexican studies: biomedical or clinical research was thought to be 'a critical resource for decision-making in each of the four programmes', but the importance of other types of research varied [10]. Of the eight policies examined in two Canadian provinces, four seemed to have been influenced by research, for example, in terms of agenda setting, research identified the need for increased pneumococcal immunisation in Saskatchewan [11]. Research utilisation is also demonstrated in some of the primarily insider-accounts, including that by Phoolcharoen [94] describing the role of the Health Systems Research Institute in Thailand in enabling research to impact on the reform of the health system.

Considerable utilisation is reported in some of the studies that focus on specific pieces of research. For two of these sets of studies, a wide interpretation of utilisation in policy-making was used [14,17,89,106], and one focused explicitly on evaluative research [89]. In some, but not all, such cases there was purposive selection. The study of the HTAs in Quebec showed over 85% had had an impact on policy [75]. The latter study is also one of the comparatively few to trace through from impact on policy to actual outcomes or benefits. It was suggested there had been cost savings of between $16 and $27 million annually. The Buxton and Hanney studies also attempted to trace through to the outcomes although this proved difficult. In one case, the evidence suggested that the research had strongly influenced the policy on heart transplantations. Buxton was able to estimate the increased number of QALYs (Quality-Adjusted Life Years) that resulted from the programme being properly funded and organised, as opposed to the counterfactual which might have been a less substantial and piece-meal development of heart transplantation in the UK [110].

While not specifically measuring levels of benefits, some of the studies have clearly shown an improvement in health equity as a result of policy changes: 'research has also played an important role in the expansion of Medicaid to poor pregnant women, young children, the elderly, and disabled' [111]. Other studies have not only demonstrated a major impact on policy but also been able to describe how research led to a paradigm shift [95].

The review of previous studies is now build upon in the following section as we develop our conceptual framework for assessing the level research utilisation in policy-making.

The interfaces between the health research system and policy-makers: priority setting and research commissioning

Increasing attention is focusing on the concept of interfaces between researchers and the users of research [6,7,14,29,93,112]. This incorporates the idea that there are likely to be different values and interests between the two communities [113], with their different time-frames [6,7], and that research is less likely to be utilised in a significant way unless networks and mechanisms are established at the interfaces. One version of the interfaces concept is presented in Figure 1. The 'permeability of the interfaces' [14] becomes important given the potential problems in the transmission of views and findings between researchers and policy-makers. Issues around interfaces need to be considered at various stages including priority setting, commissioning of research and communication of the findings.

The power relationship between publicly funded researchers and policy-makers may be described in terms of an exchange relationship [114]. The policy-maker receives new knowledge, and the testing of existing knowledge, in return for providing resources and public legitimacy. If the exchange becomes imbalanced, a reduction in the value of its outcomes becomes likely. Some of the analysis below attempts to identify both ways of enhancing the exchange, and the items upon which any assessment of utilisation should focus.

As shown in Figure 1, however, the picture is broader than this because many of the research findings flow into the pool of knowledge. Furthermore, some research that is potentially of use to policy-makers will not have been funded by them. This includes research from the international stock, which highlights the role of research as a global public good [3]. If a national system is to draw on this to maximum benefit, various interface mechanisms might be needed. This section, however, describes a mechanism specifically related to user-driven research, and the next section covers the broader interfaces.

It is not necessary here to describe all the expert approaches to research priority setting – see The 10/90 Report on Health Research, 2001–2002 [4] for a recent review. Given all the current activities, however, it is important to consider problems identified in previous attempts to enhance utilisation through priority setting [7]. Resistance to priority setting comes from those who adopt the 'internalist' view of research. They share Polanyi's opinion that the best science comes from the freedom of the researcher to pursue the priorities that emerge from the scientific imperatives [115]. Most now accept the contention, as voiced by Kogan and Henkel, that if health research is 'internalist and freely sponsored, the problem for government will be that of securing adequate brokerage with it...because it has not taken part in the setting of problems' [7].

In addition to the technical questions to do with how best to identify the most important priorities in terms of health needs, the utilisation aspects of the debate perhaps revolve around two key questions:

• are priorities being set that will produce research that policy-makers and others will want to use?

• are priorities being set that will engage the interests and commitment of the research community?

Research that Policy-makers will be more likely to use

Policy-makers have not always found it easy to identify their needs or to aggregate the demands from various sources [7]. Again, the limitations on the ends-means model of rationality must be recognised and it should not be assumed that sophisticated priority setting mechanisms will automatically produce research regarded as relevant by policy-makers. This is why it is so important that the methods described do incorporate stakeholder involvement and an iterative approach [4], and that, particularly when overseas agencies/researchers are involved, efforts are made to link the research to the priorities of the national policy-makers [21]. This should boost local ownership of the research. From the perspective of the policy-maker it is important that the research not only seems relevant, but also timely. Involvement in such priority setting is itself sometimes seen as a way of informing policy [7]. Any assessment of utilisation should include identification of policy-makers' attitudes towards the priorities set.

The ability of policy-makers to set priorities, and the likelihood of them using the eventual research findings, will probably be increased if they are able to develop long-term links with researchers. This is especially the case for researchers in centres where they can build up their own shared reservoir of knowledge on the key issues and discuss this with policy-makers [14,29,103]. In these circumstances, researchers help develop the policy-makers' views about what are the important issues that should be addressed by research. Crucially, this allows researchers to play an interactive role in shaping policy-makers perceived needs.

Priorities to engage the commitment of researchers

There is a danger that the more the agenda is set unilaterally by non-researchers, the less the research community will be committed to working on it. At the commissioning interface between priority setting and the funding of specific pieces of research, there is some scope for subtle defection from the agreed priorities [116]. It is possible that the move towards Mode 2 research [58] means that an increasing number of researchers are moving away from belief in the superiority of the internalist Mode 1 approach. Where the policy-makers are working with the researchers as suggested by Trist [57] and Gibbons et al [58] this could result in research that has more chance of being utilised, but much of Gibbons et al's analysis is not related to formal priority setting exercises. Iterative research commissioning processes [34,117] and priority setting [4] might be ways of addressing both problems identified in this section.

Finally, despite the importance of priority setting, there is no monopoly of wisdom and those who wield the enormous power of government do well to foster their own critics and counter-analysis [7]. Independent research can provide critical commentaries and alternative perspectives that are important for healthy policy-making in the long term.

The interfaces between the health research system and policy-makers: transfer of research to policy-makers

Much previous work stresses the importance of interactions between policy-makers and researchers in increasing the likelihood of attention being given to the knowledge produced. This continues the above discussion and fits especially well with Weiss's interactive model [19], and with the view that policy-makers are unlikely to take much notice of research if the first they know about it is when it arrives on their desk [89]. It is claimed that previous interaction increases the possibilities of the findings moving up the Ladder of Research Utilisation [105], and that the building of bridges between researchers and policy-makers is important and could be achieved by 'decision-linked research' [6,118].

The studies described earlier provide many examples to support this analysis, including discussion of 'linkage strategies' [104], and 'interactions' [11]. A cholera researcher is quoted in the Mexican study as saying: ' "if there isn't a good relationship between a researcher and a decision-maker...it is difficult for research results to be taken into account" ' [10]. Buxton's insider account of his own work evaluating the emerging UK heart transplantation programme illustrates the benefits that can come from close liaison with the potential users [110,119]. As a result of frequent liaison the Department of Health knew the likely results of the final report. Then, on the day it was received, a major decision was made to fund a full heart transplantation programme in the UK, the benefits of which were described earlier. This demonstrates that although building interactive relationships is often a long-term endeavour, it can result in rapid policy-making.

Some of the studies provide examples of how good interaction was achieved through informal communications as a result of deliberate actions by researchers or even through chance relationships [10]. Researchers themselves sometimes provide policy briefing for policy-makers, which is seen as a useful but underdeveloped approach [53]. The existence of researchers, or research responsive members, in policy networks can also be important. These can be international [120]. These types of observations are broadly supported by some of the three most commonly mentioned facilitators of the use of research in the 24 studies included in the systematic review. The three are: personal contact between researchers and policy-makers (13/24); the timeliness and relevance of research (13/24); and the inclusion of a summary with clear recommendations (11/24) [22].

The various actions of individuals can be important, but it is desirable to consider the role of the HRS in encouraging or facilitating interactions, networks and mechanisms at a system-wide level. Priority setting approaches are one such mechanism. The development of long-term research centres focusing on particular topics [10,14,29] is one of the potentially strongest ways a HRS can take action to increase the possibilities of research being used to inform policy. Here the concept, noted above, of 'epistemic communities' [48] is useful and has explicitly been applied to assessments of the benefits from health research centres [29]. Furthermore, accounts from various countries or provinces describe the importance being attached to the creation of an institute for health research. Examples include: Mexico [6]; Thailand [94]; Canada [121] and Manitoba, Canada [93]. The desirability of such institutes engaging with stakeholders is being addressed by the Alliance for Health Policy and Systems Research [122]. Once established, such links can build on mutual respect and help develop an understanding of the differing perspectives.

HRSs could also ensure long-standing committees or fora are formally established to allow scientists and policy-makers to discuss issues. These could operate at both interfaces – feeding into the priority setting, and ensuring key policy-makers are aware of relevant research. Such approaches have been used in various countries including the UK [7] and Burkina Faso [9]. Other brokerage mechanisms that could also be provided by the HRS include arranging seminars for policy-makers, and funding individuals to act as research brokers [7,80,123], or translators [12,82]. Such individuals, who may be in key knowledge management roles within the HRS, take the findings from researchers and bring them to the attention of policy-makers and others. It is useful to think of diffusion of the findings at several levels. In addition to directing findings at policy-makers within the health system, efforts at wider diffusion might also help build support for adoption of the findings.

Whatever the direction of the dissemination, however, mechanisms are needed that review and synthesise research and attempt to identify the research that should be promoted from that that should not. HRSs have a clear responsibility in this area in terms of funding such reviews and their dissemination; the latter through a range of mechanisms including the internet. It can go further than this, however, and the attempt to provide some structure, or 'knowledge warehouse' [29], to the pool or stock of knowledge should be seen as a key knowledge management function of the HRS. The international Cochrane Collaboration plays an important part in this, and was inspired by the UK Cochrane Centre that was a mechanism funded as part of the information system of the UK's HRS [124,125]. The need to use and develop databases of evidence, and reviews of research, has been explored in relation to preparing evidence to inform policies on the reduction of health inequalities [126].

Many, but not necessarily all, of the mechanisms for transmission of the relevant national and international research are the responsibility of the HRS to provide. Some of the above considerations are important in the interface between national health systems and international research and international bodies promoting health. In drawing conclusions from the COHRED studies, Chunharas comments: 'National research co-ordinating bodies, such as the ENHR mechanisms promoted by COHRED, can also play a mediating role to better foster research to policy linkages. International agencies too have an important contribution to make as intermediaries in linking knowledge and action' [110]. The integration of research into the health care programmes of international organisations can be an effective mechanism for research-informed policies to be brought about [21].

The role of policy-makers as receptors of research

There is increased recognition of the significance of policy-makers in their role as recipients, or receptors, of research [7,9,11,104,112,127,128]. Despite the low response rate to their questionnaires, the findings from Landry et al's study illustrate this point. They claim: 'factors such as dissemination and linkage mechanisms that are generally considered to be powerful explanatory factors and to be the most efficient targets for policy interventions are less important than factors such as the receptive capacity of users when one climbs from the stage of transmission to the higher stages in the ladder of knowledge utilisation. Future research must recognize that the same factors do not explain success at all stages of knowledge utilisation' [91].

Beyer and Trice [129] also set out a series of steps policy-makers go through when using research and this has been applied to health research [11,62]. Epistemological, social and institutional issues are all relevant to the role of the research receptor [7,128]. The types of research relevant to policy-making vary greatly. The key questions could be seen as a spectrum:

• is there research available that is either relevant to policy issues, or could help bring new issues onto the agenda?

• is such research being effectively brought to the attention of policy-makers in diverse positions within the health system?

• is the policy-making system capable of absorbing the research findings?

• are there situations where the policy-makers are willing, and able, to use it?

The HRS can assist here in the various ways described, but the wider policy system has a responsibility to create the right institutional mechanisms and staff capacity. Broadly, the responsibility of the HRS is greatest in the first part of this spectrum. It is recognised that it is much more difficult to make recommendations about how to increase the use of research in the development and implementation of policy, than it is to suggest how to improve communications [10]. There is, however, no neat division of responsibility. The main thrust of our analysis is that the issues need to be addressed on a system-wide basis, and that there is a series of measures the HRS can take to maximise the possibilities of research utilisation. These include encouraging policy-makers to see the benefits in general, and in specific situations, of using research to help build a policy environment which will result in improvements in the health system.

Institutional arrangements do matter [6,7]. A policy machine must face the problems involved in using research, some of which it will not have commissioned itself. It needs a capacity to decode the results of research or to discern a policy problem that might yield to disciplined enquiry. To some extent these needs might be met by the use of scientific or policy advisers from outside the policy-making body, but they may not have full access to the generative stage of policy development. Hence the need for internal brokerage. These might be officials with either a scientific or a professional or a policy-making background. The evidence is that, whatever their provenance, they may be able to assume the skills and value-set of boundary-crossers and research enablers [7]. Some have become famous for their ability to empathise with the needs, problems and potentials of researchers whilst enabling policy-makers to secure otherwise inaccessible skills and knowledge [130].

The response of policy-makers to research varies not just with the type of issues and research being dealt with, but also with the differing attitudes they adopt towards the whole policy-making process [113]. As individuals, some policy-makers are much more receptive to research than others. The issues are wider than individual preferences, however, and also depend on: how far the research accords with the political and social zeitgeist of the time [128]; the national political and administrative culture [10]; and the institutional arrangements for policy-making. The historical study from Uruguay demonstrates the detrimental effect military dictatorship can have on research utilisation [99]. By contrast, the study from South Africa illustrates how, despite the problems, the new political environment can help foster the better use of research in the policies related to some programmes [98]. There will be clear opportunities for research findings to have greater impact when they are in tune with the wider developments of the time, but there are also dangers that such research could sometimes be accepted and acted upon without sufficient analysis to test its validity.

There are variations in patterns of bureaucratic recruitment and other characteristics that can influence research utilisation. In the countries where the research and policy connections are strongest, the relationship has been enhanced by the fact that some of the senior administrators have had research experience or interests as part of their prior education [131,128]. This should make mutual institutionalisation of the relationship easier to secure. The willingness of officials to undertake policy analysis is seen as important [7]. In some systems specific policy analysis units [132], or think tanks of researchers [9], are established in health policy-making bodies. An important determinant of their success will be their position within the policy-making organisation.

Too often, however, officials in policy-making bodies are resistant to research because they display strong distrust of information generated outside the organisation or system [133]. Furthermore, the career patterns of policy-makers are often not compatible with strong research utilisation if the latter depends on developing long term relations to boost receptivity. Given the length of many research projects, the original sponsor of research is often not in place when the findings are reported. Patton, the arch proponent of making evaluations more likely to be used through being utilisation-focused, notes that the major problem with his approach is the frequent turn-over of the primary intended users [134]. Various studies support a greater emphasis on training of policy-makers, at least those in bureaucratic positions [9,10]. If such training fosters a more positive attitude towards the use of research findings, where relevant, in the policy-making system as a whole, this could mitigate some of the problems.

There will be situations, particularly where the policies are likely to be made at local level, where there is much less likelihood that the researchers will have the opportunity to develop an interactive relationship with potential policy-makers. Several consequences flow from this. As noted previously, the characteristics of specific pieces of research can become important determinants of its uptake. There is an onus on the HRS to ensure it identifies and publicises those characteristics of research that are likely to increase its appeal to policy-makers. It should encourage such research to be undertaken.

In some countries there are specific mechanisms that lead to the incorporation into policy instruments of research such as Health Technology Assessments (HTAs) and clinical trials. This is one of the reasons noted previously for the greater likelihood of HTAs making an impact. A collaborative working group examining these issues in Europe concluded that, whilst they were able to identify occasional examples of systematic integration of HTA in decision-making structures, there was no direct link between the amount of money spent on HTA and its impact on the decision-making process [67]. Indeed, they suggest that small programmes can be involved in the core of the policy-making structure whilst larger HTA programmes have difficulty in demonstrating impact.

It seems clear that HTAs have had most impact in those situations where there are specific mechanisms in place that require research evidence to support well-defined policy decisions on provision, coverage or reimbursement (and these impact on practice where there are further mechanisms to ensure local adherence to national policy). The European countries where there is some evidence of such integration include: Germany, Spain, Switzerland, Sweden and, despite our earlier example, the Netherlands [67]. Conversely, HTA has had much less impact where these specific mechanisms are not in place and policy-makers are exposed to HTA only in a diffuse or indirect manner.

Governments that set up what could be considered rational policy-making arrangements in which primacy is given to the role of research evidence might find the results face considerable criticisms in the media. Even with a population fully engaged in the cost effectiveness/rationing debate, there would still be scope for disappointed interests to campaign against decisions. This illustrates the desirability of an integrated approach to utilisation and an awareness of all the pressures on policy-makers.

In the context of the above discussions more attention should be given to the role of incentives, both for researchers to produce utilisable research [6] and for policy-makers, at the system or individual level, to pay attention to it. In an exercise of empirically based modelling, Bardach [56] assumed classical economic rationality on the part of individual policy-makers. He showed how research reaches those for whom its utility exceeds the disutility of obtaining it and noted that co-operative relationships grow up with research consumers when producers try to reduce the cost to them of obtaining information.

Engaging in 'useful' research produces some clear benefits for researchers. It may be a source of satisfaction that one's work is being taken notice of and contributing to the formation of policy or the improvement of practice. At present, however, it is widely thought that the traditional academic criteria still dominate the crucial assessments of research performance upon which career advancement and peer recognition depend [7,29,98,135,136]. The assessment of utilisation, therefore, could become a key issue if rewards are to focus on relevance as well as research excellence [6,137].

The interfaces and receptor model

Any assessment of the utilisation of health research in policy-making has to integrate two factors: an awareness of the wider influences on policy-makers and a detailed analysis of the specific ways the HRS could contribute to improving the health system through providing the research to inform policies. An appropriate model for assessing research utilisation in policy-making is also likely to be one that combines both an emphasis on the importance of actions at the interfaces and an analysis of the role of receptors. As we have seen there are many models already in existence. We are proposing an interfaces and receptor model because it allows a range of key issues to be integrated into the analysis. These include:

A focus on the need for multi-layered analysis

Multi-directional interactions with practitioners and the public are important for policy-makes and augment the crucial interface, for research utilisation in policy-making, between the HRS and the policy system. As noted above, this interface itself has various dimensions including: priority setting; research commissioning; and the transfer of research findings to policy-makers.

An appreciation that both researchers and policy-makers have their own values and interests

Therefore, for example, priority setting has to be sophisticated to maximise the likelihood that the research community will be engaged on a research agenda producing knowledge that the policy-makers will use. Similarly, just because research centres undertake large scale dissemination does not necessarily guarantee their research will be utilised [11]. Hence the importance of analysis that goes beyond examining dissemination and considers the nature of productive interactions and the characteristics of research to which receptors are responsive.

An emphasis on the role of the receptor

This is necessary because ultimately it is up to policy-makers to make the decisions; this can be a convoluted process with many stages at which research could potentially have a role. Again as described above, there are various features of the organisation and training within the receptor (or policy-making) body that can enhance the utilisation of research. Even though responsibility lies with the receptors, the HRS should take every action possible to facilitate the use of the research. These are important considerations for any assessment of the success of the HRS in relation to utilisation. First, because they highlight the wider political context which is beyond the control of the HRS. Second, because they still leave room for assessments of the activities of the HRS, within its given context, to increase the permeability at the interfaces [14] and thus promote the uptake of the research findings by the receptors.

An approach that facilitates analysis of the key paradox highlighted by the systematic review

Innvær et al concluded that, 'two-way personal communication, the most common suggestion, may improve the appropriate use of research evidence, but it might also promote selective (inappropriate) use of research evidence' [22]. This potential problem can be addressed in several ways through the interfaces and receptor model. First, links between researchers and policy-makers should ideally develop on a long-term basis so that together at the priority setting interface they produce a research agenda that reflects some synthesis between the needs of policy-makers and the perspectives of independent research analysis. Second, the interfaces and receptor model emphasises the importance of the role of organisational and training issues such as the need for capacity to undertake systematic reviews and policy analysis within any system. While such capacity is seen as a way of enhancing the ability of the receptors to absorb research, it should also allow proper analysis of all evidence to be undertaken.

There could, therefore, be value in having assessments of utilisation that integrate the modelling of research utilisation with the epistemological, social and institutional analysis [7] inherent in concepts such as interfaces and receptor functions. This might contribute to future research policies and strategies in such a way as to promote greater utilisation.

Purposes of assessing the utilisation of research in health policy-making

Before showing how all the previous analysis could be built upon in the generation of appropriate tools for the assessment of the nature and extent of knowledge utilisation in health policy-making, it is desirable to consider the purposes of such assessments. The purpose of the assessment is likely to differ depending on the level at which it is conducted.

Buxton and Hanney [14,15] identified three main reasons for undertaking their case study, and more general, assessments of the benefits from research:

• justifying spending resources on health research;

• assisting with the prioritisation of future expenditure;

• indicating ways to improve the conduct and management of research so as to increase the likelihood or magnitude of subsequent beneficial consequences.

These considerations are particularly relevant when the assessment is related to the justification of, and accountability for, funding at a national level, even if the case studies are conducted at project or research unit level. For a body such as the WHO, there could well be an important role in conducting such assessments with the aim of providing evidence of the possibility of the effective use of research resources. This could support advocacy for greater resources to be made available for health research. Such advocacy has recently been powerfully made as part of the report from WHO's Commission on Macroeconmics and Health [138]. This report is seen as convincing [139], and thus perhaps is helping to generate a more promising climate in which research utilisation could be assessed. Cross-national studies of research utilisation around common themes might be the best way to conduct assessments that could illustrate effective ways in which health research can be used. Understanding could be gained from the comparisons between and within countries. The potential link with advocacy would be strengthened if the policies on which the studies were based were specifically in those areas where the Commission is calling for increased research funding. These areas include: reproductive health, maternal and child health, tropical diseases, and health systems research.

We noted previously the increasing WHO focus on the importance of research informing key policy areas [3,21]; this perspective is shared by WHO regions, for example, in relation to policies for improving health equity [140]. In this context it is important to recognise the claim, made in section 8, that assessment can influence the activities given priority by researchers. This is likely to be particularly relevant when the focus of the evaluation is the performance of specific research units, teams, or even individuals, especially when funding is at stake. Given this, it is argued that moves towards giving more importance to the assessment of utilisation of health research should help encourage researchers to devote effort to activities likely to stimulate impact, and reward those who are already doing so [7,29,136]. The greater the significance of the assessment, however, the more dispute there will be over the methods to use.

In particular, the role of numerical indicators needs to be considered in relation to the purposes of the assessment. It is argued that if the indicators used in performance evaluation lack 'decision relevance' they are ignored [141]. The introduction of performance indicators into a process such as research may, however, have a dysfunctional impact unless great care is taken to establish the purposes and likely consequences of assessments [13,18]. For example, an assessment system that resulted in more dissemination in general, as opposed to more targeted dissemination of relevant knowledge, would be repeating the dangers of increasing the overload on policy-makers [105]. Where indicators are involved, they can be used as either 'dials' to measure inputs and outputs accurately, or as 'tin openers' to identify issues needing further examination or to aid judgement [142]. Although the use of numerical indicators as dials has been advocated by some, in an area such as the assessment of research and its utilisation in policy-making, where measurements are so difficult to make, caution is usually recommended [13,18,142]. It would seem only sensible to use indicators as tin openers to aid judgements when the purpose of the assessment involves funding decisions.

Even when funding is not an issue, if any comparisons, especially international, are to be made, there would be dangers in using simple indicators outside of a wider qualitative assessment. They would become de-contextualised. The long-standing fears about such an assessment process include the danger of manipulation through collusion and the difficulty of making comparisons across programmes with a different composition of user groups [13]. Depending on the purpose of the assessment there is, nevertheless, scope for innovative thinking in terms of methods.

Methods for assessment of research utilisation in policy-making

Appropriate methods for assessment therefore have to be developed to reflect:

• the purposes for which the assessment is to be conducted, for example, to increase accountability, or to support advocacy for health research;

• the analysis about the various types of research, the range of utilisation possibilities, and the wider conceptual frameworks, for example, the interfaces and receptors model; and

• the different roles that can be played by retrospective assessments and ones that focus on the current position.

Various lists have been produced of the type of information that could be gathered to produce numerical indicators to inform either self-evaluation/peer review of research teams [136], or to inform regular monitoring of the benefits from work originating from a particular health research funder [12,18]. Items from these lists relevant for policy-making include a numerical record of: presentations to policy-makers; production of fact sheets; membership of advisory committees; and membership of committees issuing a policy document or a treatment guideline. These are not really measures of actual impact and although one such measure, references in policy publications, was also proposed, the list would probably need to be supplemented; in the case of regular monitoring, for example, by a set of structured case studies. When an evaluation within a country is to be used for making funding decisions, it would be unwise to use the numerical indicators as dials because of the contextual issues and possible biases described above. Instead they should be used to inform judgements.

Nor would it be sensible to use such indicators in any cross-national comparative study unless they were informing wider qualitative studies. Furthermore, to understand the peculiar difficulties of using raw questionnaire data in relation to assessing research utilisation in policy-making on specific issues, it is helpful to return to the definition of policy-making given earlier. This emphasised that those who make policies are in a particular, authoritative, position. This presents a rather different set of circumstances from those encountered when assessing utilisation of specific findings by practitioners and members of the public. In such cases a sample might be thought to be representative of a wider group, and individual characteristics and circumstances might even out within the sample. Moreover, it could be claimed that the opinions of each practitioner or member of the public are equally valid as regards the influence of research upon their own behaviour. In a study of policy-making on a specific issue, by contrast, the interviewees or questionnaire respondents will be likely to include some representatives from relevant interest groups, commentators, and researchers as well as policy-makers with varying degrees of involvement in different aspects of the making of that policy. In relation to understanding the processes involved in the policy-making, therefore, the respondents might have conflicting views that do present truthful representations of what people saw and heard. Nevertheless, depending on the respondent's degree of involvement with the specific events under consideration, these views are likely to be of varying validity in relation to providing an account of the key actions.

Such complexities no doubt help explain why qualitative interviewing and documentary analysis were used most frequently the research utilisation studies described in the review of previous work. Questionnaires could provide some information from a wider range of informants than it might be possible to interview. They could also be used to help identify aspects on which to focus detailed parts of the interviewing. In-depth interviews, however, are widely seen as the most appropriate method when there is a need to unravel situations with diverse layers and subtle nuances. According to Rossi et al, 'whereas written surveys and questions might be useful for some limited purposes, that approach lacks the flexibility to tailor the line of discussion to the expertise of the individual, probe and explore issues in depth, and engage the informant in careful reflection' [143]. The growth of health policy and systems research suggests there is an increasing number of researchers who could undertake such interviewing [122].

Our review of previous studies demonstrated the great difficulties of making generalisations about specific factors associated with high levels of utilisation. To address this in any cross-national initiative it would be useful to adopt several strategies. First, as far as possible, structure all the assessment studies around a conceptual framework such as the interfaces and receptor framework presented earlier in the report. The framework is probably sufficiently broad to allow it to be applied to many situations. It would, nevertheless, help inform any interview schedule so as to ensure the questions were focusing on both how research findings were communicated across the interfaces, and the degree of policy-maker receptivity to them. This would be done not to provide a check-list of items that it is expected would all have to be present if research utilisation is to be achieved. Rather, it would be so that the interview covered a range of items, some of which might emerge as the reasons linked to utilisation, or lack of it, in each particular study.

The second strategy would be to base the studies on common policy themes as far as possible. Possible specific topics within the areas identified in the previous section include multi-drug therapy for leprosy and equitable access to health services. For each of the common themes, a key body of international research would be identified and some of the analysis would relate to that, and some to the impact of the full body of research available to policy-makers in the specific country. Some of the potential purposes the WHO might have in conducting such a cross-national assessment were set out above. An approach that uses common elements in several detailed studies, but which also expects each study to produce its own narrative or story of what caused research utilisation in the particular situation and context, has similarities to broader approaches to the study of innovation and organisational change [144].

Analysis of documents and semi-structured interviews would appear to be appropriate methods to use in a retrospective study of research impact on policy-making related to specific issues, especially where the policy is made at national or sub-national government level. Indeed, the recently conducted systematic review recommended that future research in this field, 'should combine interviews with document analysis' [22]. Questionnaires could also have a role, particularly in securing a wide range of opinions about the current situation regarding knowledge sources for research utilisation in policy-making and the relevant HRS mechanisms. A combination of these approaches would provide triangulation of methods and data-sources. The account below focuses particularly on four main elements of the recommended methods for the retrospective part of the policy-making element of the Structured Cross-national Thematic Studies that could be undertaken in the WHO research utilisation project [24]:

• documentary analysis;

• interviews;

• application of scales reporting the level of research utilisation in policy-making;

• overall analysis.

Documentary analysis

Documentary analysis would be undertaken in each study. Initially it would be used in an attempt to identify the degree of consistency between the policy in the country and the body of international research that is being centrally collated by the WHO utilisation project team. Further documentary analysis would also cover issues such as how far policy-makers drew on research findings in speeches during the policy formulation and implementation stages, and accounts in reports from research funding bodies of their efforts at developing mechanisms to enhance research utilisation. The documentary data-sources would include: research publications and reports; legislation; administrative/executive regulations or orders; reimbursement arrangements; guidelines/advice; meeting reports and minutes (if available); policy statements, speeches, and articles; and reports from research funding bodies. A draft protocol for the first element of the documentary analysis has been prepared (see additional file 1: Elements of a protocol for documentary analysis).

Interviews

A stakeholder analysis could identify whom to interview first, and then snowball techniques, together with review of the questionnaires, would ensure other key people were approached for an interview. In devising the semi-structured interview schedule to be used for all interviews, in all the countries participating in a cross-national study, it would be most important to allow interviewer flexibility. This would be necessary to deal with local circumstances and with situations, as described above [79], where the interviewee has a much more limited conception of research informed policy than the interviewer. Despite these caveats it would also be desirable to develop a semi-structured interview schedule that covered as many as possible of the points discussed in the previous analysis. A draft semi-structured interview schedule has been developed (see additional file 2: Draft interview schedule for assessing research utilisation in policy-making), but it would have to be administered with considerable flexibility.

The interviews would allow:

• comparability across themes and countries yet sensitivity to specific contexts;

• detailed investigation of the level of research impact, in relation to the particular issue, on the three stages of the policy-making process: agenda setting; policy formulation; and implementation;

• rolling triangulation ie using later interviews to test information gathered during earlier ones;

• investigation of key HRS and other mechanism that operated at the interfaces to enhance the responsiveness of the receptors, including: priority setting and research commissioning mechanisms; the creation of research centres and facilitating links with policy-makers; encouraging and funding research brokerage/translator/promoter activities; encouraging and funding reviews and syntheses of relevant research findings and the production of policy briefs; and facilitating interaction between researchers and policy-makers at long-standing committees or one-off seminars etc;

• investigation of a wide range of other relevant issues: the role of key institutions and their mechanisms, such as policy analysis, for absorbing research and their exposure to forces in addition to research findings; the responsiveness of policy-makers to different types and sources of research knowledge; the features of specific research findings that made policy-makers more responsive to their findings; the aspects of policy-making where research was seen as most valuable; the role of networks, international bodies, practitioner and advocacy groups, NGOs, the media and the public in bringing research findings into the policy debate; and developments in the wider political system;

• collection of data for the wider assessment in the overall utilisation project about how far any research-informed policy formulation and implementation was contributing to an increase in any of the final outcomes such as health and health equity gains.

Application of scales describing the level of research utilisation in policy-making

Whilst there are reservations about the extent to which numerical indicators should be used for cross-national comparisons, it is possible to see how the type of exercises undertaken by Buxton and Hanney [106], Jacob and McCregor [75] and Lavis et al [11] could be built upon. It might be possible to develop indicators in the form of descriptive scales of the degree of utilisation. These would be used to give an account of the impact of research on the policy-making in the specific context of each of the countries participating in the WHO research utilisation project. In the three studies cited above, the scoring or coding for each example was undertaken by the same team. Even clearer agreement about interpretation of scales would be necessary in an international exercise. Before starting any initiative, it would be desirable for the scales to be agreed between the assessment teams in the participating countries.

The previous analysis indicates that it would probably be appropriate to consider developing four scales to apply to research utilisation in each policy area. The first scale would focus on a slightly narrower range of research that, as noted above, would be the international research. This would examine the consistency between the research and the policy. Previous studies illustrate, however, that consistency with research findings does not necessarily demonstrate that the particular findings influenced the policy [14]. Where the policy consists of a clinical guideline developed by a professional group there could be circumstances in which the first scale on the degree of consistency, based on documentary analysis, might be the only scale appropriate to apply. In such circumstances the analysis should probably concentrate on the quality of the evidence used in the guidelines [145].

The remaining three scales would each relate to assessing the actual role played by research in each of the three phases of policy-making described previously. The relationship between policy-making and research is often messy and varied. Therefore, it is inevitable that some research might play a part in only one of these three phases, but other research might play several roles. For example, epidemiological research might cause an issue to be placed on the policy agenda, other research that developed a specific way of improving treatment could be used in policy formulation, but might also have helped force the issue onto the policy agenda by showing improvements were possible. The details of each scale have been prepared (see additional file 3: Draft scales of the level of research utilisation in health policy-making). The key issues covered in them are described here:

(i) Consistency of policy with research findings

This scale would relate to how far the content of the policy on issue X was in agreement with the findings from a defined body of international research (irrespective of the actual degree of influence of research on the policy formulation). It would initially be applied during the documentary analysis.

(ii) Degree of influence of research on policy agenda setting

This scale would relate to the extent to which research (including local research) had been responsible for getting the issue onto the policy-makers' agenda. It would cover research that: either showed the existence/extent of a problem; or was so dramatic/decisive that it instigated action to be taken to turn it into policy; or contained findings/theoretical frameworks that gradually changed the perception of policy-makers and others as to the importance of the issue in a process of enlightenment. It would rely on interviews, questionnaires and documentary analysis.

(iii) Degree of influence of research on policy formulation

This scale would relate to the actual influence the research had in the policy formulation process. It would aim not only to confirm any instrumental use of the research (ie direct use of the findings or research theories in formulating the content of the policy) but also to capture examples of the much wider range of possible impacts on policy, including the gradual sedimentation of insights, theories, concepts and perspectives in the enlightenment mode. This scale would consider the utilisation of research both in the actual development of the policy content, and in policy discussions and debates. The scale would be based primarily on the data from the interviews, but also use survey and documentary data.

(iv) Degree of influence of research on policy implementation

The key issues for this scale would be the use of research in assisting implementation, either through findings which are used to inform decisions about how best to implement the policy, or by providing justification of the policy and being used to generate support for it in terms of financial resources, political commitment, and public opinion. The scale would be based on data primarily from interviews and documentary analysis.

Overall analysis

The interviews, questionnaires, and documentary analysis should also provide material to help identify the relative importance, in relation to the level of utilisation recorded, of each of the HRS mechanisms listed in the bullet point above. The types and sources of research used, and reasons for their use, should also be recorded and attempts made to correlate them with the previous priority setting approaches. It would be appropriate to enhance the internal validity of the judgements about the list and the scales by discussing the emerging findings with the respondents. The account of each study would also involve description both of the value given to research in the country and of the broader cultural and socio-political environment, to the extent that they seem relevant to the degree of research utilisation achieved.

The findings from the assessments in each participating country could be collated. For each research theme the analysis would compare two sets of data: the scales for level of utilisation in each country, and the contextualised lists of the HRS activities and other mechanisms and networks thought to be important. Organising the studies around common themes might assist assessment of how far the use of the international stock of knowledge was dependent on local research.

As noted previously, although the account here has focused on research impact on policy-making, the evaluations would be stronger as part of a wider analysis covering research utilisation and interactions with practitioners, industry and the public. The fuller analysis would be both most useful in itself, and provide greater understanding of the environment in which the policy-making occurred. By building on the framework described in Figure 1, it should provide a holistic approach [112] to these issues. Thus, the WHO research utilisation project was conceived as an integrated whole in which retrospective assessment of research utilisation in policy-making would examine one step in a process that should eventually lead to health and health equity gains [24].

Given appropriate and targeted topic and country selection, this approach is likely to meet the purpose of using structured methods to provide examples of effective research utilisation. It should contribute towards enhanced understanding of the issues and could provide the basis of an assessment tool which, if used widely in countries, could give a boost to the importance attached to the utilisation of health research.

Conclusions

Increasing global attention is focusing on ways to improve health systems and the contribution that research-informed policies can make to this. It has long been recognised that a range of factors is involved in the interactions between health research and policy-makers. The emerging focus on Health Research Systems (HRS) has identified additional mechanisms through which greater utilisation of research could be achieved. Assessment of the role of health research in policy-making is best undertaken as part of a wider study that also includes utilisation of health research by industry, medical practitioners, and the public.

The utilisation of health research in policy-making should eventually lead to desired outcomes, including health gains. Research can make a contribution in at least three phases of the policy-making process: agenda setting; policy formulation; and implementation. Descriptions of these processes, however, can over-estimate the degree of rationality in policy-making. Therefore, the analysis should be informed by a review of the full range of policy-making models. Various categories of research are likely to be used differently in health policy-making. Applied research might be more readily useable by a policy system than basic research, but health policy-makers tend to relate more willingly to natural sciences than social sciences. There also appears to be a greater chance of research being used in clinical policies about delivering care to patients, than in national policies on the structures of the health service.

Models of research utilisation in policy-making start with a link to rational or instrumental views of policy-making, and include descriptions of how commissioned research can help to find solutions to problems. Other models relate to an incrementalist view in which policy-making involves a series of small steps over a long period; research findings might gradually cause a shift in perceptions about an issue in a process of 'enlightenment'. Interactive models of research utilisation stress the way in which policy-makers and researchers might develop links over a long period. Research can also be used symbolically to support decisions already taken.

Many previous studies of research utilisation can provide lessons for future assessments. Two broad approaches can be identified. Some studies start with pieces, or programmes, of research and examine their impact. Others consider policy on a particular topic and assess the role of research in the policy-making. To facilitate comparison, studies of research utilisation are best organised around a conceptual framework. Despite that, the influence of contextual factors in different settings makes it difficult to generalise. The two methods used most frequently, and usually together, come from the qualitative tradition: documentary analysis and in-depth interviews. Questionnaires, bibliometric analysis, insider knowledge and historical approaches have all been applied. A few recent studies have attempted to score or scale the level of utilisation. The examples suggest there is a greater level of utilisation and final outcomes in terms of health, health equity, and social and economic gain than is often assumed, whilst still showing much underutilisation. There is considerable variation in the degree of utilisation, both within and between studies.

Increasing attention is focusing on the concept of interfaces between researchers and the users of research. This incorporates the idea that there are likely to be different values and interests between the two communities. At the prioritisation interface there are two key questions: whether priorities are being set that will produce research that policy-makers and others will want to use, and whether priorities are being set that will engage the interests and commitment of the research community.

Interactions across the interface between policy-makers and researchers are important in transferring research to policy-makers. This fits especially well with the interactive model of utilisation. Actions by individual researchers can be useful in generating interaction, but it is desirable to consider the role of the HRS in encouraging or facilitating interactions, networks and mechanisms at a system-wide level. The HRS could provide funding and organisational support for various items including: long-term research centres; research brokerage/translator mechanisms; the creation of official committees of policy-makers and researchers; and mechanisms for review and synthesis of research findings.

There is increased recognition of the significance of policy-makers in their role as the receptors of research. In relation to the perspective of policy-makers there is a spectrum of key questions. These range from whether relevant research is available and effectively being brought to their attention, to whether they are able to absorb it and willing to use it. The HRS has a responsibility, especially in the early parts of the spectrum, but the wider health system also has a responsibility to create appropriate institutional mechanisms and ensure there are staff willing and able to incorporate relevant research. More attention should be given to the role of incentives. The assessment of utilisation becomes a key issue if rewards are to focus on relevance as well as research excellence.

An appropriate model for assessing research utilisation in policy-making combines analysis of two issues: the role of receptors and the importance of actions at the interfaces. An emphasis on the role of the receptor is necessary because ultimately it is up to the policy-maker to make the decisions. Any assessment of the success of the HRS in relation to utilisation must accept that the wider political context is beyond the control of the HRS, but consider the activities of the HRS, within its given context, to enhance the utilisation of research by increasing the permeability of the interfaces.

The reasons for assessing the utilisation of research in policy-making include: advocacy, accountability, and increased understanding. For the World Health Organization there could be a role in conducting such assessments with the aim of providing evidence of the effective use of research resources. This could support advocacy for greater resources to be made available for health research. It is important that the purposes of any assessment are taken into account in planning the methods to be used.

Previous studies demonstrated the difficulties of making generalisations about specific factors associated with high levels of utilisation. To address this in any cross-national WHO initiative involving a series of studies in a range of countries, it would be desirable to structure all the studies around a conceptual framework (such as the interfaces and receptor framework considered here) and base the studies in each country on common themes. These could include policies for the adoption of multi-drug therapy for treating leprosy, and for the equitable access to health services.

Analysis of documents and semi-structured interviews would be appropriate methods in each study assessing the role of research in policy-making on a specific policy theme. Surveys could also have a role. These approaches would provide triangulation of methods and data-sources and should also provide material to help identify the relative importance, in relation to the level of utilisation recorded, of the HRS mechanisms described in the previous analysis. The types and sources of research used, and reasons for their use, should also be recorded and attempts made to correlate them with the previous priority setting approaches. It is expected that each study will produce its own narrative or story of what caused utilisation in the particular context, but the data gathered could also be applied to descriptive scales of the level research utilisation. The four scales could cover the consistency of policy with research findings, and the degree of influence of research on agenda setting, policy formulation, and implementation.

The findings from the assessments in each participating country should be collated. For each policy theme or topic the analysis would compare two sets of data: the scales for level of research utilisation in each country, and the contextualised lists of the HRS activities and other mechanisms and networks thought to be important. Although the account here has focused on research impact on policy-making, the evaluations would be stronger as part of a wider analysis covering research utilisation and interactions with practitioners, industry and the public.

Given appropriate and targeted topic and country selection, this approach is likely to meet the purpose of using structured methods to provide examples of effective research utilisation. The approach should contribute towards enhanced understanding of the issues and could provide the basis of an assessment tool which, if used widely in countries, could lead to greater utilisation of health research.

Competing interests

SH, MB, and MK were funded for this study by the Research Policy and Co-operation Department of the World Health Organization. MG-B is manager of the Alliance for Health Policy and Systems Research.

Authors' Contributions

All authors were involved in devising the structure of the article and contributed text. MG-B supplied Figure 2 and accompanying text. SH drafted and developed the article.

Additional File 1

Elements of a Protocol for Documentary Analysis

Click here for file

Additional File 2

Draft Interview Schedule for Assessing Research Utilisation in Policy-Making

Click here for file

Additional File 3

Draft Scales of the Level of Research Utilisation in Health Policy-Making

Click here for file