Identifying and prioritising systematic review topics with public health stakeholders: A protocol for a modified Delphi study in Switzerland to inform future research agendas.

INTRODUCTION: The Cochrane Collaboration aims to produce relevant and top priority evidence that responds to existing evidence gaps. Hence, research priority setting (RPS) is important to identify which potential research gaps are deemed most important. Moreover, RPS supports future health research to conform both health and health evidence needs. However, studies that are prioritising systematic review topics in public health are surprisingly rare. Therefore, to inform the research agenda of Cochrane Public Health Europe (CPHE), we introduce the protocol of a priority setting study on systematic review topics in several European countries, which is conceptualised as pilot. METHODS AND ANALYSIS: We will conduct a two-round modified Delphi study in Switzerland, incorporating an anonymous web-based questionnaire, to assess which topics should be prioritised for systematic reviews in public health. In the first Delphi round public health stakeholders will suggest relevant assessment criteria and potential priority topics. In the second Delphi round the participants indicate their (dis)agreement to the aggregated results of the first round and rate the potential review topics with the predetermined criteria on a four-point Likert scale. As we invite a wide variety of stakeholders we will compare the results between the different stakeholder groups. ETHICS AND DISSEMINATION: We have received ethical approval from the ethical board of the University of Bremen, Germany (principal investigation is conducted at the University of Bremen) and a certificate of non-objection from the Canton of Zurich, Switzerland (fieldwork will be conducted in Switzerland). The results of this study will be further disseminated through peer reviewed publication and will support systematic review author groups (i.a. CPHE) to improve the relevance of the groups´ future review work. Finally, the proposed priority setting study can be used as a framework by other systematic review groups when conducting a priority setting study in a different context. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

We expect that this structured research priority setting study is developed in such a way that it can serve as a framework for, and can easily be replicated by, other systematic review author groups.

Our recruitment strategy, in which organisations themselves are invited to nominate respondents from within their own organisation for participating in the study, increases the likelihood that the invited individuals will participate and allows for a participation of individuals who are considered to have the most expertise for contributing to the research.

We include a wide variety of stakeholders that are affiliated to the field of public health in order to get a clear overview of the evidence needs that are present in public health.

Although our modified Delphi technique cannot be considered a full Delphi study due to the lack of an additional feedback round, it does include a combination of metrics- and consensus-based priority setting techniques and it ensures the involvement of stakeholders who otherwise would likely be overshadowed in a full consensus-building procedure.

We use purposive sampling to include participants who have the expertise necessary to optimally respond to the questions raised. However, this might also lead to the inclusion of respondents who are interested in participating in research and who may differ from those who do not want to participate.

Introduction

Research Priority Setting

Organisations conducting or funding public health research have to select research priorities while often facing competing demands and scare resources.1 Without research priority setting (RPS) there is a risk that research topics will be chosen ad hoc or are determined by funders based on subjective goals.2–4 Hence, RPS exercises are conducted to identify and rank the most important and beneficial research options while taking current health evidence into account.5–7

Although the need to produce relevant and top priority evidence for health is widely supported a general framework for conducting a priority setting exercise in health research does not exist7–10 and might not be realisable due to contextual factors of different RPS exercises.8 Furthermore, RPS exercises specifically focused on public health topics are rare.

Research Priority Setting within Cochrane

The proposed study is conducted on behalf of Cochrane Public Health Europe (CPHE). CPHE is a subdivision of Cochrane Public Health, which aims to produce and publish Cochrane reviews regarding the effects of public health interventions. In general, Cochrane entities want to produce relevant and top priority evidence – which is also highlighted by one of their key principles for the production of systematic reviews: ‘Striving for relevance’.11 12 In order to select the most relevant and important review topics some type of RPS is necessary.

So far, Cochrane Review Groups use a wide variety of frameworks, approaches, and methods for assessing which review topics should be prioritised and a more rigorous and transparent process for priority setting is needed.12–14 One study showed that 23 out of 52 questioned Cochrane Review Groups did not have a process for prioritising review topics at all and only 13 out of 52 used a transparent and structured process for prioritising their review work.13 Of these 13 Review Groups the following prioritisation processes could be identified13:

Selecting priority topics according to health status data and policy reports.

Identifying potential priority topics from the gaps in existing evidence.

Collecting priority topics from the recommendations of e.g. clinical guidelines or other systematic reviews.

Gathering priority suggestions from stakeholders (eg, including review groups, clinicians, consumers, policy-makers).

Although the focus of these process are all important features for prioritising research topics in health, they are limiting the range of the assessment when investigated separately. Therefore, these processes should be included as separate stages or as assessment criteria in an overall structured priority setting exercise. Multiple stakeholders should be invited to participate to ensure that the process will be more transparent and not solely based on the expertise and research interests of the editorial team.

In order to develop a prioritisation process that is informed by existing evidence gaps, recommendations from other sources, health impacts, and the input from stakeholders, the processes described above need to be combined in one structured prioritisation process.

Aims

The first aim of the proposed priority setting study is to implement a structured RPS study in Switzerland to investigate which systematic review topics within the field of public health are rated highest by public health stakeholders and whether preferences vary significantly by stakeholder group. Second, we investigate which assessment criteria are perceived most important by the public health stakeholders when prioritising potential review topics.

Rational

We want to inform systematic review author groups, both within and outside of Cochrane Public Health Europe (CPHE), to improve the relevance of future review work with regards to the needs of public health stakeholders.

Finally, this is the first study in a series of future studies in other European countries. Hence, we intend to design the study in a way that allows for easy replication by other systemic review author groups and for application in different countries. This is one of the reasons to choose Switzerland: As the country is trilingual we can emulate the challenges that adaptation and translation of a survey for several language and settings pose. Another reason to first conduct this study in Switzerland is because there is a good mixture of national and international health organisations present, which increases the likelihood to gather suggestions from a wide variety of stakeholders.

Methods and analysis

Study status

We employ a multi-stage process in our study (see figure 1): We started mid-2015 with a comprehensive literature review that preceded the scoping stages of the study. In the first scoping stage we established how public health is defined for the remainder of the study, which stakeholder groups are relevant for inclusion in the study, and which assessment criteria will be used as preselection for prioritising the review topics (Pre-Stage I). In the second scoping stage we continued with a horizon screening of recommendations and statements in existing literature with regards to potential systematic reviews topics (Pre-Stage II).

Figure 1

Stages of the proposed Research Priority Setting study.

Pre-Stage I started December 2015 and has been completed in October 2016, whereas Pre-Stage II is on-going and is planned to be completed in July 2017. With regards to the comprehensive literature review and the scoping stages we communicated intensively with members of Cochrane Public Health Europe. We also established an advisory board at the beginning of Pre-Stage I and the members have provided valuable input for the first two scoping stages and will be involved for the remainder of the proposed study. We identified 120 organisations and institutions, which are affiliated to the field of public health in Switzerland, for being invited to participate in the Delphi rounds.

We will use a two-round modified Delphi technique to prioritise systematic review topics within public health (Stage III & IV). The questionnaire for the first Delphi round is planned to be sent out in August 2017 and the second Delphi round questionnaire is aimed for October 2017.

Study design

Several studies used the Delphi technique for prioritising research options.15–21 The Delphi technique is often applied when consensus is sought from a diverse group of experts on a range of topics or ideas.21–23 Participants are encouraged to provide controlled feedback in multiple rounds, which allows them to reassess and/or change their opinion according to the results of previous rounds.23 Additionally, the Delphi process allows for an easy incorporation of both metrics- and consensus-based techniques and it does not require face-to-face meetings. Therefore, the Delphi technique does not only reduces resource expenses but it also reduces potential disadvantages of a face-to-face expert panel such as social pressure and dominance of certain individuals or groups.7 21 23 24

Our modified Delphi cannot be considered as a full Delphi study because the rating exercise that will be conducted in the second questionnaire will not be available for a discussion in another follow-up round. We modified the Delphi process as such to reduce the workload for the respondents and to be able to include more stakeholders. Therefore, the results of the proposed Delphi procedure should be interpreted as an indication of preferences rather than a true consensus. On the other hand this modified process increases the involvement of stakeholders who otherwise might be overshadowed in the discussion of a full Delphi study, or who would not be invited at all as one of the top expert for a Delphi study.23 25

In the first Delphi round we will ask relevant stakeholders in an online questionnaire to propose and select systematic review topics that they consider to be potential priorities and to select which assessment criteria they deem essential for assessing the importance of potential review topics (Stage III). Finally, in the second Delphi round a second questionnaire will be distributed to the same stakeholders in which they are invited to provide feedback to the results of the first questionnaire and to rate the proposed topics from the first Delphi round according to the selected assessment criteria (Stage IV).

Advisory board

It is recommended that a RPS study is overseen and guided by an advisory or steering group.7 8 26 Hence, our proposed study will be conducted with the support of an advisory board – consisting of both subject matter experts and methodological experts from three different countries. We purposefully chose advisory board members who have a lot of experience with combining practice and research in their work. More information about the advisory board members can be found in the acknowledgments. The advisory board members oversee the design and the analysis of the study especially with regards to the scope of the study, the selection of relevant participants for the Delphi process, the questionnaire design, and the interpretation of the Delphi results.

Participant selection and recruitment

Part of the core principles of Cochrane Public Health is to produce accessible evidence which can be disseminated to potential users of public health evidence. Therefore, it is important that the needs of those potential users are also taken into account with regards to the choice of future review topics.

In our study we will explore the needs (or preferences) of potential users of public health research evidence so that their perceived needs can be incorporated in the direction of future reviews. We decided to approach a wide variety of stakeholder groups (box 1).

Public Health research and education

Policy makers at different levels

National & local health organisations (non-governmental & non-profit organisations – NGOs/NPOs)

International health organisations (NGOs & NPOs): Excluded are multilateral/bilateral agencies

Representatives of health professionals and healthcare institutions

Representatives of the public

Health insurers

As commonly adopted in Delphi studies we will use purposive sampling for both questionnaires that will be conducted in our study. In purposeful sampling the respondents do not have to represent a broader population. It only has to be ensured that the respondents have the expertise to participate and to provide insightful responses regarding the study subject.33 Nevertheless, it is important to include a wide variety of stakeholders in order to not under- or over-represent certain stakeholder groups.

Our recruitment method is based on a successful priority setting study conducted by the German Network for Healthcare Research (Deutsches Netzwerk Versorgungsforschung) in which organisations themselves are invited to nominate respondents from within their own organisation for participating in the study.20 This method increases the likelihood that the invited individuals will participate and it allows for a selection of individuals who are considered to have the most expertise for contributing to the research.

We will contact directors or heads of departments of all stakeholder organisations that are identified for the proposed study in Pre-Stage I and we will ask them to nominate individuals from their organisation that are most suitable for participating in the study. If they nominate individuals then we can see this as an informed consent from the directors or heads of departments. At the beginning of both questionnaires the respondents themselves will be further informed about that their participation is voluntarily and anonymous; the questionnaires will only start after their informed consent.

Approximately 120 organisations and institutions in Switzerland that are associated to public health will be invited to nominate participants for the proposed priority setting study. Each invited organisation and institution will be asked to nominate minimal five individuals to participate in the two Delphi rounds. The same recruitment method in the priority setting study of the German Network for Healthcare Research showed that about 75% of the organisations nominated respondents.20 Most of these nominees participated in the study (85%). Based on these results we could expect to have a maximum of 380 respondents. However it is not unlikely that participating organisations will nominate less than five individuals, hence we expect the final number of respondents to be lower than 380.

A disadvantage of the Delphi technique is that the panel attrition rates are high and increase when the number of rounds increases.34 35 Based on a study that compared online Delphi studies,34 we expect an attrition rate of 10.5% from round one to round two. The risk of dropouts is reduced due to the chosen recruitment strategy and due to the included function that allows respondents to save their progress in the questionnaire and to continue later.

Scoping stages

Pre-Stage I – Scoping the study

Before the actual prioritisation will be undertaken (stage III & IV) we needed to define the scope of the proposed study.8 36 37 With input from the advisory board we established:

The domains of public health research that needed to be included in this RPS study.

The stakeholder groups that are relevant for inclusion in the study.

The assessment criteria that needed to be preselected for the Delphi rounds.

Public health domains

We defined three domains of public health research for our proposed study (table 1). These domains are based on the classification that is made within the European SPHERE project.38 The SPHERE project (supported by the European Commission’s Sixth Framework Research Programme) serves as an excellent base for this study as it thoroughly reviewed and described public health research at the European level.

Table 1

Main domains of public health research

Domains	Key Words
Prevention	Surveillance / modelling / control of diseases & other health conditions
Health promotion	Health information / health competences / healthy behaviour & environments
Health services	Medical care / health systems & services

Stakeholder groups

Information regarding the identified stakeholder groups for the proposed priority setting exercise is described in the participant selection & recruitment section and an overview can be found in box 1.

Assessment criteria

Many RPS studies establish a preliminary list of assessment criteria for assessing the different research options.5 7 39 40 Based on recommended assessment criteria for RPS studies in health5 41–44 we included six assessment criteria as a starting point for this study (table 2).

Table 2

Preselected assessment criteria

Equity	The review topic will be likely to reduce inequity in the accessibility of health interventions.
Effectiveness	The review topic will be likely to generate/improve effective health interventions.
Disease burden reduction	The review topic will have a theoretical potential to reduce large portions of the existing disease burden.
Feasibility	The review topic will be deliverable within its context.
Novelty of the concept	The review topic represents a new or emerging area.
Lack of (good quality) evidence	Synthesised evidence for the review topic is either not existing, lacking in quality, not up to date, or only available in a different context.

Pre-Stage II – Horizon screening

Data collection

In Pre-Stage II we are gathering statements regarding potential review topics in public health from existing literature. The identified review topics will serve as examples in the first Delphi round, which creates a starting point for the respondents for their own listing of potential review topics.

As horizon screening can become an onerous task we based our approach on another RPS study in the field of public health.45 We modified their approach as the gathered review topics will only serve as the building blocks for the remainder of the study and therefore we do not need to gather an extensive list of review topics. We are screening explicit electronic databases and sources that present high quality systematic reviews (Cochrane Library, Health Evidence.org, Systematic Reviews Journal, Evidence for Policy and Practice Information and Coordinating) that fit into our set scope (Pre-Stage I).

Data analysis

During the horizon screening we analyse the recommendations in the identified reviews for statements regarding gaps in research evidence in public health. These gaps in the research evidence will be categorised into the main public health domains that have been established in Pre-Stage I.5 18 42 46

Delphi procedure

Stage III – First Delphi round – Gathering of review topics and assessment criteria

In the first Delphi round an online questionnaire (see online supplementary appendix 1) will be distributed to the stakeholders. They will be invited to propose and select systematic review topics for each public health domain separately. We compare topics within each domain separately because a cross domain comparison of topics will be less informative and can lead to interpretation problems. Furthermore, the domains will serve as guidance for the respondents allowing them to consider all the different directions within public health. The respondents need to select review topics which they think should be considered as potential priority topics. Simultaneously they can indicate if another domain should be added or if the definition of the existing domains should be adjusted.

From the selected and proposed topics the respondents will need to make top three lists. We include this quick prioritisation of three topics per domain to investigate which review topics are proposed ad hoc as top priorities without the consideration of assessment criteria. Thereafter, we ask them to select or propose criteria which they believe are of particular importance for assessing which review topics should be prioritised. We will use the selected assessment criteria in the second Delphi round for an in-depth prioritising and compare this to the results of the quick prioritisation.

In the final part of the first Delphi round the respondents will have to rate their experience with systematic reviews on a four-point Likert scale.

The questionnaire will remain online for up to 4 weeks and reminders will be sent out via electronic mail approximately every 7 days after the initial invitation. The questionnaire will be administered online using Lime Survey software. We will use IBM SPSS Statistics (Version 23) software for further data management and analysis of the anonymised database to carry out descriptive statistics (frequencies, range of scores).

The results of the first Delphi round will give an extensive list of review topics that are selected or proposed by public health stakeholders in Switzerland. The top three lists from the quick prioritisation will be pooled into a preliminary priority list per domain. Furthermore, we will conduct a domain comparison to see if certain domains are over- or under-represented in the final priority lists and a stakeholder group comparison to see if the different stakeholder groups select and prioritise different review topics. Finally, we will also count the selected and proposed assessment criteria to see which criteria are deemed most suitable for assessing review topics. These assessment criteria will be used in the second Delphi round for the in-depth prioritisation rating.

The proposed topics will be synthesised to reduce the overlap between the proposed topics and clustered into the established public health domains.5 18 42 46 Thereafter, the proposed topics will be reshaped into properly defined review topics using the PI(C)O format. In evidence-based medicine and evidence-based public health the PICO format is used to create review questions that are focused on the effect of an intervention or issue. The different elements of PICO stand for:

P = Patient, Population, or Problem.

I = Intervention, Item of Interest, exposure, or Issue

C = Comparison (if any)

O = Outcome

The synthesising and redefining of topics will reduce the length of the list of options for the second Delphi round, improve the definition and therefore the applicability of the listed topics, give insight in how many times different topics are proposed and by whom, and provide a good overview of which public health areas or domains are suggested most often.5 42

Stage IV – Second Delphi round – Rating of review topics with assessment criteria

In the second Delphi round we will distribute another questionnaire to the stakeholders (see online supplementary appendix 2) showing them the aggregated results of the first Delphi round. First the respondents are asked to indicate their (dis)agreement with the aggregated results – regarding both the quick prioritisation exercise and the selected assessment criteria. There will also be an open section where respondents are asked to comment on their (dis)agreement.

Thereafter, we ask them to rate each review topic individually per assessment criteria on a four-point Likert scale of importance. Finally, the respondents will be asked to weigh the importance of the selected assessment criteria. This scoring exercise will give a total score for all the assessed review topics: One score with and one score without including the attached criteria weights. The questionnaires will be provided in German, French, Italian, and English.

Similar as in Delphi round one the questionnaire of the second Delphi round will remain online for up to 4 weeks. We will send reminders via electronic mail approximately every 7 days. Also the second questionnaire will be administered online using Lime Survey software while questionnaire responses will be entered into IBM SPSS Statistics (Version 23). We will analyse the agreement statements and the rating exercise, both conducted on a four-point Likert scale, using multivariate statistics.

Moreover, the respondents need to weight the criteria: They will have to allocate a fictional 100 points to the selected assessment criteria based on their perception of the criterion’s importance for prioritising review topics. The allocation per assessment criterion of all the respondents will be added together and then divided by the number of respondents, which gives a total weighted score for all the separate assessment criteria. This total score shows the relative importance of every assessment criterion. table 3 shows an example on how the points allocated by the respondents to the criteria are transformed into a final criterion weight.

Table 3

Example of criterion weight calculation

	Criterion 1	Criterion 2	Criterion 3	Total
Respondent 1	50	30	20	100
Respondent 2	20	50	30	100
Respondent 3	100	0	0	100
Respondent 4	0	50	50	100
Total points (sum of points given by respondents)	170	130	100	400
Weight ((total points given to criterion / total number of respondents) * 100)	(170/400) *100 = 42.5	(130/400) *100 = 32.5	(100/400) *100 = 25

We will calculate a final score for every assessed review topic: One total score with and one total score without weighting. The review topics that received the highest score will be listed (one list for scoring with and one list for scoring without weighting) as top priority topics. Additionally, we will compare these top priorities with the priorities that were derived by the quick prioritisation exercise in the first Delphi round.

Finally, we can conduct an inter- and intra-stakeholder group comparison. The comparison between stakeholder groups allows us to analyse (univariate analysis) if there are significant differences in the preferences of the stakeholder groups. For the intra-stakeholder group comparison we want to observe if there is consensus within each stakeholder group itself. Consensus within groups (when using a four-point Likert scale) will be defined by taking the IQR and divide it by two.18 47

Ethics and dissemination

The data from the online questionnaire are automatically stored in a database on a local university server. The server is only accessible with a username and a password that are only known to the main investigators. Also the online questionnaire software provider (Lime Survey) does not have access to the database. The username and password will not be shared with other parties.

Participation in this study is voluntary. We will provide detailed information during the recruitment phase with regards to the study in general and to the tasks of the respondents so they can make an informed decision. Responses will not be traced back to the respondent. No personally identifiable information is captured; unless the respondent voluntarily offers personal or contact information in any of the comment fields (however this is not requested in any way). We will not ask the respondents to indicate which organisations or institutes they represent. Therefore, we can also not report this information in any publication.

Dissemination

This results of this study will be disseminated through a PhD Thesis, the CPHE website, peer reviewed publications, academic conferences, and possibly through formal presentations to the stakeholders. Finally, the study is designed in such a way that other systematic review teams can use it to recreate and conduct their own priority setting studies in different contexts (using free software).

Discussion

This priority setting study will involve a wide variety of stakeholders in the prioritisation process of public health review topics, as is recommended and applied by other larger scale projects such as the James Lind Alliance and the Healthy People initiative.26 44 48 Therefore the results will not only represent the expertise and preferences of editorial teams and review groups themselves, but also the evidence needs in practice. Moreover, due to the online application and the promising recruitment strategy, we believe that this study will produce a large pool of suggestions and priority topics, providing new insight for review groups working with public health related topics.

This study design has several limitations. Because we modified the Delphi process to reduce the workload for the respondents and to be able to produce quicker results once this pilot study is applied in a different context, the results should be interpreted as an indication of preferences rather than a true consensus.49 Although our online modified Delphi study allows for the inclusion of a larger panel group than most other Delphi studies, it is also reducing the chance for respondents to discuss arguments and counterarguments for prioritising certain topics. However, by using this process we provide each respondent with an equal chance to propose and reassess potential priority topics, which should ensure that the final rating is not only informed by the perspectives of single (strong) individuals. This reduces the attrition of individuals with minority views, therefore leading to a more balanced results without an overestimation in the final ‘consensus’.