A checklist for health research priority setting: nine common themes of good practice.

Health research priority setting processes assist researchers and policymakers in effectively targeting research that has the greatest potential public health benefit. Many different approaches to health research prioritization exist, but there is no agreement on what might constitute best practice. Moreover, because of the many different contexts for which priorities can be set, attempting to produce one best practice is in fact not appropriate, as the optimal approach varies per exercise. Therefore, following a literature review and an analysis of health research priority setting exercises that were organized or coordinated by the World Health Organization since 2005, we propose a checklist for health research priority setting that allows for informed choices on different approaches and outlines nine common themes of good practice. It is intended to provide generic assistance for planning health research prioritization processes. The checklist explains what needs to be clarified in order to establish the context for which priorities are set; it reviews available approaches to health research priority setting; it offers discussions on stakeholder participation and information gathering; it sets out options for use of criteria and different methods for deciding upon priorities; and it emphasizes the importance of well-planned implementation, evaluation and transparency.

Introduction

Setting priorities for health research is essential to maximize the impact of investments, which is especially relevant in resource-poor environments. Health research prioritization is regarded as a key part of efforts needed to strengthen national health research systems [1-6]. Additionally, prioritization mechanisms are necessary to facilitate the current demand for increased harmonization of health research at a global level [4,7-10], particularly in combination with analyses of financial flows for health research [9,11,12] and burden of disease studies [13,14]. Numerous World Health Assembly resolutions and the 2004 and 2008 Ministerial Summits on Health Research have stressed the need for action on these issues [4,15-19].

For health research priority setting exercises to effectively target research with the greatest public health benefit, it is important that they are of high quality and so there is a need for consensus on what constitutes quality or good practice in this area [2,20]. The various approaches that are available to guide priority setting for health research differ on important aspects of the process [20-29]. Because of the different contexts for which priorities can be set, the optimal approach varies per exercise. Consensus on a gold standard or best practice for health research prioritization thus seems difficult to achieve and is, more importantly, not an appropriate response [30].

Therefore, taking the heterogeneous nature of research priority setting exercises into account, while recognizing the need for agreement on appropriate guidance for these exercises, we propose a checklist that outlines options for different approaches and defines nine common themes of good practice for health research prioritization processes. It is intended to provide assistance for planning a high quality health research priority setting exercise whether at national, regional or global level.

Methods

Several methodological approaches were combined to acquire a comprehensive overview of common views on good practices in health research priority setting. First, a literature search was conducted of Pubmed for peer-reviewed literature that discussed good practices in health research priority setting (search terms: (setting priorities [title/abstract] OR priority setting [title/abstract]) AND research [title/abstract]). Additionally, the World Health Organization (WHO) library database (WHOLIS) was sought for literature emanating from WHO on this topic (search terms: (research AND priorities) OR (research AND priority) OR (research AND agenda)) [31]. Secondly, health research priority setting exercises that were organized or coordinated by WHO headquarters since 2005 were reviewed. Documents describing these exercises were identified through the search of WHOLIS and by a manual search of all departmental websites of WHO. Methods used for prioritizing research were analysed in the 230 documents that were found [32]. Finally, a process of expert consultation was employed using in-depth and semi-structured interviews with staff in WHO and a selection of international research organizations experienced in health research priority setting.

A checklist for health research priority setting

Nine common themes for good practice in health research priority setting (i.e. elements of a health research priority setting process that are key and should not be overlooked) emerged and were combined into a checklist for health research priority setting (Table 1). The nine themes broadly fall into three different categories. Five are especially important in the preparation phase of the prioritization process, two concern methods for deciding upon priorities and two relate to work that is usually performed after priorities have been set.

Table 1

Checklist for health research priority setting

Preparatory work
1. Context
Decide which contextual factors underpin the process: What resources are available for the exercise? What is the focus of the exercise (i.e. what is the exercise about and who is it for)? What are the underlying values or principles? What is the health, research and political environment in which the process will take place?
2. Use of a comprehensive approach
Decide if use of a comprehensive approach is appropriate, or if development of own methods is the preferred choice. These approaches provide structured, detailed, step-by-step guidance for health research priority setting processes from beginning to end.
3. Inclusiveness
Decide who should be involved in setting the health research priorities and why. Is there appropriate representation of expertises and balanced gender and regional participation? Have important health sectors and other constituencies been included?
4. Information gathering
Choose what information should be gathered to inform the exercise, such as literature reviews, collection of technical data (e.g. burden of disease or cost-effectiveness data), assessment of broader stakeholder views, reviews or impact analyses of previous priority setting exercises or exercises from other geographical levels.
5. Planning for implementation
Establish plans for translation of the priorities to actual research (via policies and funding) as a priority at the beginning of the process. Who will implement the research priorities? And how?
Deciding on priorities
6. Criteria
Select relevant criteria to focus discussion around setting priorities.
7. Methods for deciding on priorities
Choose a method for deciding on priorities. Decide whether to use a consensus based approach or a metrics based approach (pooling individual rankings), or a combination.
After priorities have been set
8. Evaluation
Define when and how evaluation of the established priorities and the priority setting process will take place. Health research priority setting should not be a one-time exercise!
9. Transparency
Write a clear report that discusses the approach used: Who set the priorities? How exactly were the priorities set?

Preparatory work

1. Context

There are several contextual factors that underpin the process of research priority setting, namely practical considerations about available resources, the focus of the exercise, the values that stakeholders adhere to, and the health, research and political environment in a country. These factors influence the prioritization process and the eventual research priorities and should therefore be discussed explicitly from the beginning of the exercise [21,33].

Careful planning of the prioritization exercise is important to establish an exercise that meets the initial expectations. It is necessary to identify available financial, human and time resources [33].

A clear focus or scope must be defined for the exercise: What is the exercise about and who is it for [21,22]? Factors such as the target disease burden or risk factor (which health research areas does the exercise aim to address), the geographical scope (global, regional, national, sub-national or institutional), the intended timeframe (long-term or short-term priorities), the intended beneficiaries (e.g. children, elderly, urban/rural areas) and the target audience of the research priorities (e.g. policymakers, funding organizations, researchers) must be known before priorities can be set.

The values or principles of an exercise should also be clarified [21,33,34]. Should priorities be cost-effective or equitable, or combine both criteria [35]? Should there be an emphasis on a particular type of research (e.g. research among children)? Does the nature of the institution setting the priorities influence the values of the exercise? Are there any external demands for the exercise (e.g. political or commercial) that have an influence [33]? Diverging principles or values between different stakeholders or disciplines are likely and should be resolved in a fair and legitimate manner [21,34].

For country-level exercises scanning the health, research and political environment of the country is of particular importance [26,28]. Who has the political power to set priorities? Who has previously set priorities? How do policymakers perceive research for health? What kind of capacity exists to do, use, and fund research?

2. Use of a comprehensive approach

There exist a number of comprehensive approaches to health research priority setting. These approaches are comprehensive because they provide structured, detailed, step-by-step guidance for the entire priority setting process, covering many of the points on this checklist. They assist in the preparatory work of an exercise, in deciding on priorities, and in what to do after priorities have been set. Use of these approaches is therefore in general advantageous and their use should be at least considered.

Four commonly used comprehensive approaches are:

• 3D Combined Approach Matrix (CAM) - Focus on the structured collection of information [21,36,37]

The CAM offers a structured framework for the collection of information according to several important criteria for research priority setting and takes into account the influence of different actors and factors [36]. Recently, a dimension on equity was added to this framework [21]. The process for deciding on priorities is consensus-based. The CAM has been used for both global and national exercises.

• Essential National Health Research (ENHR) approach - Focus on health research priority setting for national-level exercises [23]

The ENHR approach provides guidance for the entire process of setting priorities for health research on a national level. It is a step-by-step manual for facilitators of a national priority setting process.

• The Child Health and Nutrition Research Initiative (CHNRI) approach - Focus on a systematic algorithm for deciding on priorities [22]

The CHNRI approach to research priority setting provides specific guidance for the entire process of setting research priorities. It offers a detailed, systematic algorithm for the identification of research priorities that pools individual scorings of research options based on five weighted criteria. The CHNRI approach has been used for both global and national exercises.

• The COHRED management process to priority setting - Focus on the management process for national-level exercises [26,28]

Recently, COHRED has developed a management approach for countries to set health research priorities. This high-level approach delineates important steps of a priority setting process for national-level exercises, and discusses a wide range of options for tools and approaches to use in the process (including 3D CAM, CHNRI and ENHR approach).

These comprehensive approaches are reviewed and compared in several documents [20,24-26,28,29]. As part of a workshop on priority setting methodologies in health research convened by WHO's Cluster on Information, Evidence and Research (IER), its Department for Research Policy and Cooperation (RPC) and the Special Programme for Research and Training in Tropical Diseases (TDR) in 2008, a matrix was developed reviewing three of these approaches in more detail, providing a summary, discussing strengths and weaknesses, and listing applications of each approach [20]. The matrix is added to this article as additional file 1.

Adhering to a comprehensive approach will in general improve the quality of an exercise, but it depends entirely on the context of the priority setting exercise in question whether use of such an approach is appropriate, or whether development of own methods is the preferred choice. Approaches can be tailored to match a specific exercise, retaining the advantages of their comprehensive and detailed methodology, while accommodating existing wishes and needs for the exercise [24,38].

The list of approaches provided here is not exhaustive. Other forms of guidance are available, for example those that were developed for specific health research priority setting situations, such as for Health Technology Assessments [39], applied health services research [40], guideline development [41], and patient/caregiver priority setting partnerships [42]. Additionally, distinct approaches are often recommended for health policy and systems research [1,30,43-46]. Objective approaches to research priority setting without stakeholder consultation [27,47,48] and foresight techniques [49,50] are also used for health research priority setting. Approaches that help set priorities for health interventions and those for prioritizing health research should not be confused [51].

3. Inclusiveness

Although objective approaches to health research prioritization that are solely based on burden of disease data or cost-effective analyses do exist, most literature on health research priority setting that was found, and the experts that were consulted, considered stakeholder involvement to be an indispensable part of the process of research prioritization [40]. It is thus important to identify which stakeholders need to be involved in the research priority setting exercise, why their opinions need to be sought and what role they should play in the process (e.g. providing opinion, providing evidence or being a part of the group that decides on priorities) [52].

Fair involvement of stakeholders is important. Priority setting exercises should strive for appropriate representation of different expertises and for balanced gender and regional participation. Different sectors and constituencies that could potentially be involved are for example civil society [53], policymakers [54-56], funders/donors, the private sector [57], and members of the public [42,58-63]. The interdisciplinary nature of public health suggests a role for many different disciplines in setting research priorities, including health researchers and medical practitioners (often several medical professions and health research disciplines have relevant knowledge) [64], economists, sociologists [65] and many others. For national exercises, tools are available to assist in the mapping of possible stakeholders [49]. A transparent method should be agreed upon to manage potential conflicts of interest in personal, professional and commercial areas.

In principle broad stakeholder involvement (multisectorial and multidisciplinary) is beneficial for the outcomes of a research priority setting exercise for several reasons. Firstly, it minimizes the chances of research options being overlooked. Different groups of stakeholders tend to prioritize research differently [9,66-70]. Secondly, participation in the exercise fosters ownership of the established priorities among those involved, thus increasing the chances of implementation of the priorities. Thirdly, broad participation makes priorities correspond to the needs of those that will implement and those that will benefit from the research priorities. As such, the prioritized research will be a better response to societal and policy needs, increasing the overall credibility of the exercise and the potential impact on health and health equity [71,72]. Finally, broad stakeholder involvement may prevent unnecessary duplication of prioritization efforts and hence wasting of resources [2].

Lastly, appropriate leadership of the priority setting process needs to be identified. This can for example be in the form of an executive committee or an advisory group that provides overall guidance on the prioritization process, while a larger core working group or decision making group actually decides on priorities [20,73]. Good leadership can be pivotal in creating and sustaining a high quality priority setting process.

4. Information gathering

There are many ways to make the priority setting process better informed and choices should be made on which types of information are necessary. These can include the collection of technical data that are often needed to inform discussion on research priorities (See Criteria), such as burden of disease, cost-effectiveness of interventions, current resource flows towards particular research areas, or determinants of disease [21]. Furthermore, in order to be able to prioritize research, one must first know where the gaps in knowledge are; a literature review to identify those gaps is often necessary. Also an initial survey of broader stakeholder views on priorities or opinions on matters related to the research area [74], or a review or impact analysis of previously established priorities [75] can serve as preparation before the actual exercise.

Research priority setting is needed at different geographical levels: global, regional, national, local within countries, and within organizations. For some health topics, priorities will be the same on all levels. For most however, priorities will reflect the context they are seeking to address. Research priorities from different levels can be used to inform each other. For global exercises, awareness of national and regional research priorities is important in reaching an inclusive research agenda that is relevant for national and regional contexts [1,72]. The development of national health research agendas in turn can benefit from awareness of local research priorities, set by primary care teams [76]. Vice versa, global or regional research priority setting exercises can be of value in informing research priority setting on a national level. To facilitate information exchange on national health research agendas in and for low- and middle-income countries, an interactive, web-based information platform on health research called the Health Research Web was recently initiated by COHRED. The platform contains a section aimed at collecting national health research priorities [77].

Finally, there are many organizations (such as COHRED, CHNRI and the Global Forum for Health Research) that have specialized in providing advice on the process of health research prioritization [21-23,26,28,36]. Other organizations such as TDR aim to support the identification of health research priorities [78]. Consulting individuals or organizations with previous experience in health research priority setting as part of the preparatory work can aid in obtaining a higher quality process for setting priorities.

5. Planning for implementation

Health research priorities that are set by an organization or country to inform its own funding policies are likely to be linked with implementation strategies. Research priority setting exercises are however often faced with considerable inherent implementation issues because priorities are set by those who are not directly responsible for their implementation. If that is the case, planning for implementation should be a priority during the initial phase of a research priority setting exercise (and not be left till after priorities are established). It is important to decide who the priorities are being set for, and what that target group need. It should be mapped out in advance which stakeholders are required to be included in the exercise for a feasible and sustainable implementation of the established research priorities. For example, the involvement of policymakers and funding organizations from the beginning means that support for the priorities is more likely and increases the opportunity for research priorities to be translated into actual research [1,30]. Other examples of facilitation of implementation are classification of priorities into themes [73], engagement of media in the exercise to increase coverage [74], adaptation of global research priorities at regional or national level [79] and writing evidence informed policy briefs [23,80]. More information on making effective use of health research evidence in policymaking can be found on the website of the WHO Evidence-Informed Policy Network (EVIPNet) [81].

Deciding on priorities

6. Criteria

Criteria are used to focus discussion around research priorities and to ensure that important considerations are not overlooked. They allow for different research dimensions to be balanced against one another depending on the identified values or principles of the exercise, which is reflected in their variation across different exercises and comprehensive approaches to research priority setting [21-23,43,51,82-85]. Examples of criteria are the magnitude of a health problem, the likelihood of reducing disease burden, cost-effectiveness, the present level of knowledge, current resource flows, the degree of equitability, sustainability, ethical aspects and local research capacity, but there are many more possibilities. Commonly, criteria can be categorized into one of three dimensions: Public health benefit (should we do it?), feasibility (can we do it?) and cost (Figure 1). Participants in the priority setting exercise should decide by consensus on appropriate criteria at the beginning of the exercise.

Figure 1

There are three common categories of criteria against which different research options can be considered.

7. Methods for deciding on priorities

There are several different methods that can be used to actually decide on priorities. These broadly fall into two groups: consensus based approaches and metrics based approaches. The former lead priorities to be decided by group consensus, the latter involve metrics or an algorithm that results in pooling of individual rankings of research options. Consensus tends to improve the acceptability of the exercise; individual ranking prevents dominance of a few participants. An example of a consensus based approach is the Combined Approach Matrix (CAM) [21,36,37]. Given that all stakeholders are typically not equal and are knowledgeable in different areas, it is especially important for consensus based approaches to take into account diverging values and viewpoints between stakeholders; there are several methods available to do so [86]. Two examples of metrics based approaches are Delphi like techniques and the method as employed by CHNRI [22,87]. Approaches that combine consensus with some form of metrics are common; research options are then first individually prioritized and consequently discussed (or vice versa). This can be an iterative process, as is possible for example in the nominal group approach [88].

If one chooses to rank priorities, this also can be achieved in different ways. Ranking can be performed per research option with the criteria as guidance for discussion and thought. Conversely, research options can be ranked per criterion. In the latter case, different criteria can even receive different weights according to their contextual importance [22,23]. Another option is to differentiate between ranking priority issues and priority research questions. The former could be performed by a broad stakeholder group up front and the latter by technical experts [1,40,65]. A detailed discussion of different ranking techniques can be found on the COHRED website [28].

After priorities have been set

8. Evaluation

The identification of health research priorities should be seen in the broader context of health research coordination and inform funding and policymaking for health research in a sustainable manner. Hence, previously set priorities should be periodically reviewed to ensure that priorities are up to date. Besides updating research priorities, other forms of evaluation can be considered. Evaluation of the process used to set priorities can increase the quality and acceptability of that process [89]. Furthermore, to make research prioritization legitimate and fair, an appeals mechanism for the established priorities can be considered, providing opportunity for feedback [63,90]. Finally, performing an impact analysis, for example in the form of a review of research performed and/or funding allocated based on previously established priorities, can be valuable [75]. Not only can this provide insight into priorities that have remained devoid of attention, but it can also enforce discussion on implementation issues.

9. Transparency

When writing a report of the exercise, being as transparent as possible is crucial. Potential implementers of health research priorities are unlikely to adopt or use priorities unless they are fully informed of all aspects of the priority setting process; transparency increases the credibility and thus the acceptability of the final result. Therefore, the report should not be limited to stating a list of priorities, but should also explain how those priorities were established, and by who. This entails providing details on which choices were made for points one through eight on this checklist, and why those choices were made.

Discussion

It is commonly accepted that health research priority setting processes assist researchers and policymakers in effectively targeting research that has the greatest potential public health benefit [17,33,91]. Particularly for low-income countries, national health research priorities can facilitate the transformation of a donor-driven research agenda to an agenda driven by countries' needs [6,92]. The establishment of such a nationally owned research agenda is consistent with the Paris Declaration and Accra Agenda for Action, in which country ownership of developmental strategies is regarded a fundamental consideration for the achievement of enhanced aid effectiveness [93].

On a global level, research for health has been prioritized comprehensively (i.e. covering all health research areas) on several occasions [8-10,94]. Additionally, numerous exercises are continuously being conducted to prioritize health research for specific health areas. Setting priorities for research globally is essential to provide more direction to the currently fragmented global approach to health research funding [7] and to reduce the inequities in allocation of funding towards research commonly articulated as the 10/90 gap [9]. There are several groups of health research funders, such as the Heads of International Research Organizations (HIROs) and Enhancing Support for Strengthening the Effectiveness of National Capacity Efforts (ESSENCE), that could bring more harmonization and alignment in funding for global health research. In order to accomplish this, consensus on mechanisms to identify common priorities will be indispensable [7].

There are a number of comprehensive approaches to health research priority setting available to guide researchers step by step in setting research priorities. These approaches have been extensively tested and have proven their value. Their detailed methodologies, although varying per approach, all ensure that the priority setting process is comprehensive and complete. However, the review that we performed of health research priority setting exercises that were organized or coordinated by WHO revealed that many researchers choose to develop their own, unique methods [32]. Existing needs and contextual particularities of priority setting exercises cannot always be accommodated by one of the existing approaches. For those who wish to develop their own methods to research priority setting, the amount of available aid has been limited to date [20,26,28,33,72,73,82,95]. This lack of guidance has had a negative result on the quality of exercises. Among the exercises we reviewed, often one or more of the elements of good practice we identified here were overlooked.

The checklist helps those seeking to undertake a health research priority setting exercise to make an informed choice as to which comprehensive approach to use or provides assistance for creating a high quality priority setting process without use of an existing approach. It lists nine common themes for good practice that deserve to be considered in any health research prioritization exercise. This element of consideration (instead of specific guidance) is key throughout the checklist: rather than suggesting a particular path, it has been developed to accommodate the flexibility required by different contexts.

The checklist was tested by informing health research priority setting exercises with a global scope at WHO [75]. It is hoped that in the future it will also prove to be of value in informing national-level exercises. Additionally, the generic framework that the checklist offers provides a useful template for future collection of more detailed information on good practices in health research prioritization. In this paper key references are provided under the respective sections, but this information is not exhaustive. More detailed guidance should be collected and compiled in one place as part of the resources available to support countries in organizing health research, in line with the WHO strategy on research for health and the Global Strategy and Plan of Action (GSPA) on Public Health, Innovation and Intellectual Property [16-18]. Work in this area has already been performed by COHRED who have collected a wide range of tools that can aid health research priority setting in national contexts [28].

Conclusion

There are as many approaches to health research prioritization as there are priority setting exercises. One gold standard or best practice is therefore not attainable, nor appropriate. The identification of common themes for good practice fulfils the need for a generic guidance on this variable and intricate process. The checklist for health research priority setting provides practical assistance for the formation of a high quality priority setting process and can aid researchers and policymakers in effectively targeting health research that is needed the most.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

RFT conceived the study concept, RFT and RFV designed the study methods, RFV performed the reviews and conducted all interviews, AG, RFT, RFV and SO analysed the information acquired through the reviews and interviews, RFV wrote the first draft of the paper, AG, RFT and SO contributed to the writing of the paper, all have read and approved the final manuscript.

Additional file 1

Summary of three commonly used research priority setting methods. This file contains a matrix that was developed as part of a workshop on priority setting methodologies in health research that was convened by WHO's Cluster on Information, Evidence and Research (IER), its Department for Research Policy and Cooperation (RPC) and the Special Programme for Research and Training in Tropical Diseases (TDR) in 2008. It reviews three comprehensive approaches in more detail, providing a summary, discussing strengths and weaknesses, and listing applications of each approach.

Click here for file