The implementation of prioritization exercises in the development and update of health practice guidelines: A scoping review.

BACKGROUND: The development of trustworthy guidelines requires substantial investment of resources and time. This highlights the need to prioritize topics for guideline development and update.
OBJECTIVE: To systematically identify and describe prioritization exercises that have been conducted for the purpose of the de novo development, update or adaptation of health practice guidelines.
METHODS: We searched Medline and CINAHL electronic databases from inception to July 2019, supplemented by hand-searching Google Scholar and the reference lists of relevant studies. We included studies describing prioritization exercises that have been conducted during the de novo development, update or adaptation of guidelines addressing clinical, public health or health systems topics. Two reviewers worked independently and in duplicate to complete study selection and data extraction. We consolidated findings in a semi-quantitative and narrative way.
RESULTS: Out of 33,339 identified citations, twelve studies met the eligibility criteria. All included studies focused on prioritizing topics; none on questions or outcomes. While three exercises focused on updating guidelines, nine were on de novo development. All included studies addressed clinical topics. We adopted a framework that categorizes prioritization into 11 steps clustered in three phases (pre-prioritization, prioritization and post-prioritization). Four studies covered more than half of the 11 prioritization steps across the three phases. The most frequently reported steps for generating initial list of topics were stakeholders' input (n = 8) and literature review (n = 7). The application of criteria to determine research priorities was used in eight studies. We used and updated a common framework of 22 prioritization criteria, clustered in 6 domains. The most frequently reported criteria related to the health burden of disease (n = 9) and potential impact of the intervention on health outcomes (n = 5). All the studies involved health care providers in the prioritization exercises. Only one study involved patients. There was a variation in the number and type of the prioritization exercises' outputs.
CONCLUSIONS: This review included 12 prioritization exercises that addressed different aspects of priority setting for guideline development and update that can guide the work of researchers, funders, and other stakeholders seeking to prioritize guideline topics.

Introduction

Health practice guidelines are “systematically developed statements” intended to optimize care at the clinical, public health and health systems levels [1]. The development of trustworthy guidelines requires substantial investment of resources [2, 3] and time, often taking an average of two to three years [4, 5]. In addition, and with the expansion of medical research and the emergence of new and innovative technologies, guidelines should be updated as necessary [6]. This highlights the need to prioritize topics for guideline development and update.

In fact, the importance of prioritization for guideline development has been recognized by many guideline developing groups [7]. It ensures that limited resources are aligned with priority needs for guideline development [8]. Furthermore, prioritization exercises engaging a wide range of stakeholders enhance the relevance and potential uptake of priority topics by end users [9-11]. This represents an important step toward improving the delivery of evidence-informed care.

The guideline development process includes, in addition to prioritizing topics, the prioritization of questions and outcomes [5, 12]. Also, prioritization should be considered when adapting guidelines to select priority questions from among those addressed in the original guidelines [13]. Similarly, guideline developers need to prioritize which guidelines, guideline sections, or recommendations should be updated [14].

With the growing interest among researchers in prioritizing topics for the de novo development, update and adaptation of guidelines, several exercises have been conducted to yield explicit and transparent prioritization [15-17]. Some investigators relied primarily on the use of criteria to select priority guideline topics [18], while others have followed multicomponent prioritization processes and have used established tools and approaches [19].

While some efforts have been invested in synthesizing the evidence on prioritization for guideline updating [20], none have described prioritization for the de novo development, update or adaptation of guidelines. As such, the objective of this study was to systematically identify and describe prioritization exercises that have been conducted for the purpose of the de novo development, update or adaptation of health practice guidelines.

Methods

We conducted a scoping review of published prioritization exercises implemented as part of the de novo development, update or adaptation of health practice guidelines. We followed standard methodology and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines for reporting scoping reviews [21] (see S1 File). This study is based on a detailed protocol available in S2 File. The project’s team included a multidisciplinary group of professionals in the clinical, public health, and health policy and systems fields, with expertise in guideline development and priority setting.

Paper type: We included descriptive reports and excluded commentaries, editorials, letters, correspondences, news, and abstracts.

Scope: We included papers about the de novo development, update or adaptation of health practice guidelines addressing clinical, public health or health system topics. Also, we excluded papers reporting proposed approaches, without any applied exercise.

Focus: We included papers that aimed to prioritize one of the following: guideline topics, questions/recommendations, or outcomes. We excluded papers reporting on individual prioritization criteria or items.

Setting: We included eligible papers irrespective of the setting (low-, middle- or high-income countries; primary, secondary or tertiary healthcare facilities).

Information sources and literature search

We searched Medline and CINAHL electronic databases from their respective inception date to July 2019. We also manually searched Google Scholar in July 2019. We developed the search strategy with the assistance of a medical librarian. The search included both Medical Subject Headings (MeSH) terms and free-text words and combined various terms for health prioritization (see S3 File). We did not use any language or date restrictions. We screened the reference lists of included and other relevant papers.

Study selection

The study selection process consisted of two phases: title and abstract screening and full text screening. Teams of two reviewers (AEH, TL, AA, RZM, RF, LBK) independently and in duplicate screened the titles and abstracts of citations captured by the search for potential eligibility. The reviewers then obtained the full texts of citations judged as potentially eligible by at least one of the two reviewers. Then, they screened the full texts in duplicate and independently for eligibility using a standardized and pilot-tested screening form and following a calibration exercise. At this phase, the reviewers resolved disagreements by discussion or with the input of a third reviewer (EAA) when consensus could not be reached.

Data extraction

Prior to data extraction, we conducted two calibration exercises to enhance the validity of the process. Three reviewers (AEH, TL and AA) worked in duplicate and independently using a standardized and pilot-tested data extraction form (see S4 File). They resolved disagreements by discussion or with the help of a third reviewer (EAA). We collected the following data from each included paper:

General characteristics: authors, lead entity, target audience, year of prioritization conduct, scope of prioritization, topic (specific domain), focus of prioritization, type of guideline development, and support/funding;

Steps of prioritization:

Pre-prioritization phase (development of guiding/ethical principles, generation of initial list of topics and collection of technical data to inform discussions);

Prioritization phase (use of established prioritization methods, research gap analysis, use of criteria, prioritization/ranking);

Post-prioritization phase (refinement of priorities into guideline topics, dissemination and implementation, revision mechanism, monitoring and evaluation).

Data synthesis

Due to the descriptive nature of data, we synthesized the findings in a semi-quantitative way. We used the extracted data to come up with common categorizations of relevant concepts (e.g., prioritization steps, generation of initial list of topics), using an iterative process of review and refinement. As part of this process, we analyzed the content of each study at least twice; when drafting the initial categories, and after producing an advanced draft. We reported the results narratively and in a tabular format.

The concepts addressed in our analysis included:

Prioritization steps; we adopted 11 categories of prioritization steps, which we developed for a recent systematic review on prioritization for evidence synthesis [22];

Generation of initial list of topics (descriptive analysis);

Output of the priority setting exercises (descriptive analysis);

Stakeholder involvement; we adopted the categories we developed for a recent systematic review on prioritization for evidence synthesis [22], which is based on the 7Ps framework [23];

Prioritization criteria; we used a common framework of prioritization criteria that we developed for a recent systematic review on prioritization approaches in the development of health practice guidelines [24] (see S5 File).

Results

Fig 1 shows the study flow diagram summarizing the study selection process. Out of the 33,339 identified citations, twelve papers met our inclusion criteria [14–19, 25–30]. We excluded 896 articles based on full text screening for the following reasons: not a paper type of interest (n = 49); not describing a reproducible prioritization exercise (n = 322); not about health practice guidelines (n = 525).

Fig 1

Study flow diagram.

General characteristics of the included studies

Table 1 shows the general characteristics of the twelve included studies. One prioritization exercise was conducted in 1998 while the remaining ones were conducted between 2010 and 2017. Half of the prioritization exercises were implemented at a national level (n = 6) [14, 16, 19, 27, 29, 30], while the rest were implemented at regional (n = 3) [25, 26, 28], provincial (n = 2) [15, 18], or international levels (n = 1) [17]. All of the prioritization exercises focused on prioritizing guideline topics (as opposed to prioritizing questions or outcomes) and addressed clinical topics. While three exercises focused on updating guidelines [14, 15, 26], nine focused on de novo development [16–19, 25, 27–30], and none addressed adaptation of guidelines. Funding sources were mainly professional societies (n = 5) [16, 25, 26, 28, 29] or public funding (n = 4) [15, 18, 19, 30].

Table 1

General characteristics of the exercises for prioritizing guideline topics.

Study	Lead entity	Target audience	Year of prioritization conduct	Scope of prioritization	Topic (specific domain)	Focus of prioritization	Type of guideline development	Support/Funding
Agbassi, 2014 [15]	Program in Evidence-based Care, Clinical Practice Guideline Initiative of the Ontario provincial cancer system	Clinical Practice Guidelines developing groups	2011–2012	Provincial (Ontario)	Clinical (cancer care)	Topics	Update	Funded by the Cancer Care Ontario
Brouwers, 2003 [18]	Practice Guidelines Initiative of Cancer Care Ontario's Program in Evidence-based Care	Clinicians	Not reported1	Provincial (Ontario)	Clinical (role of prophylactic anticonvulsant drugs in brain tumors)	Topics	De novo	“Sponsored by Cancer Care Ontario and Ontario Ministry of Health and Long-Term Care”
Borgonjen, 2015 [16]	Not reported	Dermatological professional organizations	2010	National (Netherlands)	Clinical (dermatological diseases)	Topics	De novo	“Supported by Dutch Society of Dermatology and Venereology”
Becker, 2018 [14]	Not reported	Clinical Practice Guidelines groups	2014–2015	National (Germany)	Clinical (acute perioperative and posttraumatic pain)	Topic (Guideline sections)	Update2	Not funded
Farrell, 2015 [30]	Not reported	Clinicians	2013–2014	National (Canada)	Clinical (medication/drug classes for deprescription)	Topics	De novo	Funded by the Ontario Ministry of Health and Long-Term Care
Jo, 2015 [19]	Korean Academy of Medical Sciences (KAMS) and Korea Centers for Disease Control and Prevention (KCDC)	Clinical Practice Guidelines developing groups (KAMS and KCDC)	2013–2014	National (Korea)	Clinical (chronic diseases)	Topics	De novo	“Supported by Korea Health Promotion Foundation and Korea Centers for Disease Control and Prevention”
Kerr, 2009 [17]	Not reported	Clinicians (epilepsy specialists)	Not reported	International (6 countries; not specified)	Clinical (epilepsy in adults with intellectual disability)	Topics	De novo	Not reported
Loeffen, 2015 [29]	Not reported	International Clinical Practice Guidelines developers	2013	National (Netherlands)	Clinical (supportive care in pediatric cancer)	Topics	De novo	“Supported by Dutch Cancer Society”
Nast, 2019 [28]	European Academy of Dermatology and Venereology and the Division of Evidence-Based Medicine, Charité –Universitätsmedizin Berlin	Clinicians (dermatologists)	2017	Regional (Europe)	Clinical (dermatology and venereology)	Topics	De novo	Funded by the European Academy of Dermatology and Venereology
van der Sanden, 2002 [27]	Not reported	Dutch Dental Association	1998	National (Netherlands)	Clinical (dentistry)	Topics	De novo	Not reported
van der Veer, 2015 [26]	European Renal Best Practice	Clinical Practice Guidelines developing groups	2014	Regional (Europe)	Clinical (vascular access for haemodialysis)	Topics	Update	“Supported by a grant of the European Renal Association-European Dialysis Transplant Association”
van der Veer, 2016 [25]	European Renal Best Practice	Clinicians (Nephrologists and geriatricians)	Not reported	Regional (Europe)	Clinical (chronic kidney disease in older adults)	Topics	De novo	“Financially endorsed by the European Renal Association-European Dialysis Transplant Association”

1 The projected completion date of the guideline was Winter, 2004.

2 A priori, the guideline steering group decided not to update the whole guideline.

We present below our findings summarized as the steps of prioritization, with a focus on two of those steps for which detailed information was available (namely the generation of initial list of topics and prioritization criteria). Then, we review stakeholder involvement in the prioritization exercises. Finally, we review the processes and outputs of the prioritization exercises stratified by whether the prioritization exercise was related to de novo development or updating.

Steps of prioritization

Table 2 outlines the prioritization steps addressed in the twelve included studies across three phases: pre-prioritization, prioritization and post-prioritization phases. Although most of the studies (n = 11) addressed at least one step in the pre-prioritization phase [14–17, 19, 25–30], less than half (n = 5) addressed at least one step during the post-prioritization phase [14, 15, 17, 25, 27]. Four studies covered more than half of the 11 prioritization steps across the three phases [14, 15, 19, 25].

Table 2

Steps addressed when prioritizing guideline topics.

Paper	Pre-prioritization phase			Prioritization phase				Post-prioritization phase
	Development of guiding/ethical principles	Generation of initial list of topics	Collection of technical data to inform discussions	Use of established prioritization methods	Research gap analysis	Use of prioritization criteria	Prioritization/Ranking	Refinement of priorities into guideline topics	Dissemination and implementation of priorities	Revision mechanism	Monitoring and evaluation
% of studies	n = 3 25%	n = 10 83%	n = 2 17%	n = 3 25%	n = 2 17%	n = 8 67%	n = 11 92%	n = 2 17%	n = 3 25%	n = 0 0%	n = 3 25%
De novo development of guidelines
Brouwers, 2003 [18]						✓		N/A
Borgonjen, 2015 [16]		✓2					✓
Farrell, 2015 [30]	✓	✓				✓	✓	N/A
Jo, 2015 [19]		✓	✓	✓	✓	✓	✓
Kerr, 2009 [17]		✓					✓	✓	✓
Loeffen, 2015 [29]		✓				✓	✓
Nast, 2019 [28]		✓			✓		✓
van der Sanden, 2002 [27]		✓					✓				✓
van der Veer, 2016 [25]	✓	✓				✓	✓	✓	✓
Updating of guidelines
Agbassi, 2014 [15]		✓		✓3	N/A	✓	✓	N/A	✓		✓
Becker, 2018 [14]			✓	✓4	N/A	✓	✓	N/A			✓
van der Veer, 2015 [26]	✓	✓			N/A	✓	✓

1 We did not include ‘stakeholder engagement’ as a separate step given that it was included across the three phases. The colored cells denote that stakeholders were engaged in the step.

2 Unclear whether participants were given an initial list and asked to suggest additional ones, or if it was based on suggestions only.

3 The exercise is based on former updating procedures of the Program in Evidence-based Care and procedures of other established guideline developing groups (e.g., Scottish Intercollegiate Guidelines Network and National Institute for Health and Care Excellence).

4 The pilot study followed a previously developed updating procedure (using information from a systematic review on the methods for updating clinical practice guidelines).

Prior to conducting the prioritization exercises, most of the included studies (n = 10) generated initial lists of topics [15–17, 19, 25–30], while only a few studies reported on the development of ethical principles to guide the conduct of the exercise (n = 3) [25, 26, 30], or on the collection of technical data to inform further discussions (n = 2) [14, 19].

Most studies used prioritization criteria (n = 8) [14, 15, 18, 19, 25, 26, 29, 30] and ranked priorities (n = 11) [14–17, 19, 25–30] during the prioritization phase. Out of eight studies, two refined priorities into guideline topics (excluding four studies where this step was not applicable as the exercise started with topics and not broad themes) [17, 25]. Less than half of the studies conducted or reported on a plan for dissemination and implementation (n = 3) [15, 17, 25] or monitoring and evaluation (n = 3) [14, 15, 27]. All of the studies involved stakeholders during various prioritization steps across the three phases, with the majority involving stakeholders in the generation of initial list of topics (n = 8) [16, 17, 25–30], use of criteria (n = 8) [14, 15, 18, 19, 25, 26, 29, 30] and prioritization/ranking of priorities (n = 10) [14–17, 19, 25, 26, 28–30]. Only one study engaged stakeholders in the post-prioritization phase [17].

Generation of initial list of topics

Table 3 shows the steps involved in generating initial list of topics. One frequently used method for generating initial list of topics was seeking input from stakeholders (n = 8) [16, 17, 25–30]. Other methods included reviewing the literature (n = 7) [15, 17, 19, 25–27, 30], referring to the health information system (n = 1) [19] and to previous priority setting exercises (n = 1) [15].

Table 3

Steps involved in generating initial list of topics.

Paper	Literature review				Health information system	Previous priority setting exercises	Stakeholder input	Refinement of the initial list of topics
	Existing trial	Existing systematic review	Existing practice guideline	Other
% of studies	n = 3 25%	n = 2 17%	n = 4 33%	n = 3 25%	n = 1 8%	n = 1 8%	n = 8 67%	n = 4 33%
De novo development of guidelines
Brouwers, 20031 [18]
Borgonjen, 2015 [16]							✓
Farrell, 2015 [30]				✓2			✓
Jo, 2015 [19]			✓	✓3	✓
Kerr, 2009 [17]	✓	✓					✓
Loeffen, 2015 [29]							✓
Nast, 2019 [28]							✓	✓
van der Sanden, 2002 [27]				✓4			✓	✓
van der Veer, 2016 [25]	✓		✓				✓	✓
Updating of guidelines
Agbassi, 2014 [15]			✓			✓
Becker, 2018 [14]
van der Veer, 2015 [26]	✓	✓	✓				✓	✓

1 The guideline topic was identified by the Neuro-oncology disease site group and was then confirmed through surveying practitioners.

2 The type of literature searched was not specified (i.e., reports outlining the prevalence and impact of polypharmacy).

3 Global Burden of Disease (GBD).

4 Original contributions, clinical reports, editorials and letters to the editor (1992–1997), and analysis of discussions of dental peer groups (1989–1998).

Prioritization criteria

Table 4 presents the prioritization criteria that 10 out of the 12 studies reported on. Eight studies used their proposed criteria as part of their prioritization exercises [14, 15, 18, 19, 25, 26, 29, 30], while two studies proposed criteria but did not use them in the exercise [16, 28].

Table 4

Prioritization criteria and the domains they fall under (n = 10).

Item Paper	Disease-related factors					Interest			Practice		Guideline development					Potential impact of the intervention				Implementation considerations
	Health burden	Economic burden	Burden on healthcare system	Equity relevance	Urgency	Health professional level	Consumer level	National level	Practice variation	Uncertainty or controversy about best practice	Absence of guidance	Unsatisfactory guidance	Availability of evidence	Availability of low certainty evidence	Potential for changing existing guidance	Impact on health outcomes	Economic impact	Impact on the healthcare system	Impact on equity/access	Feasibility of intervention implementation	Availability of resources	Acceptability
% papers reporting the criterion	n = 9 90%	n = 3 30%	n = 3 30%	n = 0 0%	n = 0 0%	n = 3 30%	n = 3 30%	n = 1 10%	n = 3 30%	n = 3 30%	n = 2 20%	n = 2 20%	n = 3 30%	n = 2 20%	n = 3 30%	n = 5 50%	n = 2 20%	n = 2 20%	n = 1 10%	n = 2 20%	n = 1 10%	n = 1 10%
De novo development of guidelines
Brouwers, 2003 [18]	✓					✓
Borgonjen, 20152 [16]	✓					✓	✓	✓	✓				✓	✓		✓	✓	✓		✓
Farrell, 2015 [30]							✓				✓		✓	✓		✓	✓	✓		✓		✓
Jo, 2015 [19]	✓	✓
Loeffen, 2015 [29]	✓															✓
Nast, 20192 [28]	✓	✓	✓						✓						✓
van der Veer, 2016 [25]	✓		✓							✓						✓
Updating of guidelines
Agbassi, 2014 [15]	✓											✓	✓		✓
Becker, 2018 [14]	✓								✓	✓	✓	✓			✓						✓
van der Veer, 2015 [26]	✓	✓	✓			✓	✓			✓						✓

1 All prioritization criteria have been worded in a way favoring prioritization (i.e., a favorable assessment of the criterion indicates higher priority).

2 Criteria were proposed but were not used in the prioritization exercise.

The studies included a mean of seven prioritization criteria (range: 3–13), with a total of 70 criteria reported. We attempted to match the 70 criteria to a published framework of 20 guideline prioritization criteria classified into six domains (Table 4 and S5 File). During the matching process, we added two criteria that emerged from the included studies (i.e., availability of low certainty evidence and acceptability). Table 4 shows the classification of the identified prioritization criteria according to the new framework.

The most frequently reported criteria related to the health burden of disease (n = 9) [14–16, 18, 19, 25, 26, 28, 29] and potential impact of the intervention on health outcomes (n = 5) [16, 25, 26, 29, 30]. None of the studies included equity relevance of the disease or urgency as explicit criteria. Eleven (out of the total of 22 criteria listed in the framework) was the highest number of criteria reported by a study [16].

Stakeholder involvement

Table 5 shows the types of stakeholders involved in prioritizing guideline topics and the methods used to engage them. All included studies involved healthcare providers in the prioritization exercises, while four studies (33%) involved researchers [14, 19, 29, 30], and only one study (8%) involved patients (Table 5) [26]. In Loeffen et al., the authors reported not including patients, parents, and caretakers as they planned to understand the needs of the professionals beforehand [29]. Seven studies described stakeholder recruitment methods which ranged from the use of professional networks (e.g., members directory), to searching databases and emailing clinicians [16–18, 25, 26, 28, 30].

Table 5

Types of stakeholders involved in prioritizing guideline topics and the methods of engagement.

Paper	Types of stakeholders													Description of recruitment method	Method(s) of engagement
	Public policymakers	Health care providers	Researchers	Members of the public	Patients and their representatives	Caregivers	Health system payers	Health care managers	Intergovernmental agencies/Research funders	Product makers/Industry	Press & journalists	Non-governmental organizations	Other
% of studies reporting	n = 0 0%	n = 12 100%	n = 4 33%	n = 0 0%	n = 1 8%	n = 0 0%	n = 0 0%		n = 0 0%	n = 0 0%	n = 0 0%	n = 0 0%	n = 0 0%	n = 7 58%
De novo development of guidelines
Brouwers, 2003 [18]		✓												✓	Prioritization: modified Dillman (online survey via email)
Borgonjen, 2015 [16]		✓												✓	Generation and Prioritization: online survey
Farrell, 2015 [30]		✓	✓											✓	Generation: modified Delphi (round 1 of online survey via email)Prioritization: modified Delphi (rounds 2 and 3 of online survey via email)
Jo, 2015 [19]		✓	✓												Prioritization: online survey via email
Kerr, 2009 [17]		✓												✓	Generation: modified Delphi (round 1 of online survey via email)Prioritization: modified Delphi (round 2 of online survey via email)
Loeffen, 2015 [29]		✓	✓												Generation: Delphi (round 1 of online survey via email)Prioritization: Delphi (round 2 of online survey vi email)
Nast, 2019 [28]		✓												✓	Generation: online piloted survey via email (round 1)Prioritization: online piloted survey via email (round 2)
van der Sanden, 2002 [27]		✓													Generation: online survey
van der Veer, 2016 [25]		✓												✓	Generation: open consultation (online piloted survey) Prioritization: open consultation (online piloted survey) & expert consensus meeting (nominal group technique: 2-round voting)
	Updating of guidelines
Agbassi, 20141 [15]		✓													Prioritization: completion of 2 surveys: annual document assessment and document review in 2 iterations (2 assessment and review cycles)
Becker, 2018 [14]		✓	✓												Prioritization: online survey via email and consensus conference
van der Veer, 2015 [26]		✓			✓									✓	Generation: online piloted survey (round 1)Prioritization: online piloted survey (rounds 1 and 2)

1 Following the initiation of a review, review outcomes (endorsement, archive, and update) should be approved by a larger expert panel comprising a multidisciplinary team of clinicians and other stakeholders.

All prioritization exercises surveyed stakeholders (e.g., Delphi approach) as a method of engagement. Other methods included the nominal group technique (n = 1) [25] and consensus conference (n = 1) [14]. Stakeholders were engaged via an online platform (e.g., online surveys, email discussions) in all included studies, with two studies using both online and in-person meetings [14, 25]. The frequency of engagement varied from only once (n = 4), to twice (n = 6), or three times (n = 2).

Prioritization processes and outputs for de novo development (n = 9)

Table 6 describes the processes and outputs of the prioritization exercises. The nine studies that implemented prioritization processes for the de novo development of guidelines followed common steps of reviewing the literature (n = 5) [17, 19, 25, 27, 30] and/or engaging stakeholders (n = 9) [16–19, 25, 27–30], while considering the availability of existing guidelines on the suggested topics (n = 3) [16, 19, 28]. In fact, one study conducted the prioritization exercise regardless of existing guidelines which resulted in prioritizing 20 topics; all of which were covered by existing guidelines [16].

Table 6

Prioritization processes and outputs of the conducted prioritization exercises.

Study	Initial list of priorities	Process (steps starting with initial list and ending with final list of priorities)	Output (final list of priorities)
De novo development of guidelines
Brouwers, 2003 [18]	One suggested topic	→ One topic was identified by the Neuro-oncology disease site group (DSG)→ Members of the DSG conducted a survey of Ontario clinicians to confirm topic selection and determine their support for the topic→ Survey results showed variation in current practice and that practitioners feel that a guideline is necessary→ One topic was confirmed for guideline development	One confirmed topic
Borgonjen, 2015 [16]	157 topics and 10 criteria	→ 157 dermatological topics were selected and ranked by 118 dermatologists as priority topics regardless of existing guidelines via a survey→ 10 criteria were included for ranking by dermatologists who were asked to add any missing criterion→ 20 topics were prioritized based on pooled scores; all of which were covered by existing guidelines; with an overlap between topics 12–20 and a further set of 15 topics and a significant difference in raking between top 5 and other 30 topics.→ 8 criteria were prioritized, and 3 additional criteria were suggested	Ranked list of 20 topics with additional 15 topics (overlapping confidence intervals) Ranked list of 8 criteria
Farrell, 2015 [30]	29 drug/drug classes	→ 29 drug/drug classes were identified by research team and included in round one of survey; with 14 drug/drug classes reaching required consensus level (>70%) and 2 new drug classes added from comments→ In round two, participants were asked to rank the 2 new drug classes, and were asked to re-rank the 14 drugs/drug classes while considering the overall round one results and justifying selection of the top 5 choices→ In round 3, participants were asked to rank the top 5 drug classes using specific criteria→ Top 14 drug classes were identified using the overall mean rank and by profession	Ranked list of 14 drug classes
Jo, 2015 [19]	41 chronic diseases	→ 41 chronic diseases were selected based on:  • Global Burden of Disease, 2012 Health Insurance Statistics Yearbook and ICD-10  • excluding diseases for which guidelines are being developed by KAMS→ Data on burden of disease and a list of available guidelines were provided to the experts for consideration in prioritization→ Analytic hierarchy process and subjective assessment were used for ranking→ Top 20 diseases were derived from each assessment	Two ranked lists of 20 chronic diseases
Kerr, 2009 [17]	30 clinical areas	→ An initial list of 30 clinical areas was derived based on the below steps: • experts were sent a CD-ROM with detailed evidence from Cochrane review (abstracts from review and all interventional trials identified) • each expert was invited to submit 10 areas where "it was felt new clinical guidance statements were necessary"→ Participants were then asked to rank the top 10 areas in another mailing→ A large difference was observed in ranking score after area ranked 11 so this was the cut-off point	Ranked list of 11 clinical areas
Loeffen, 2015 [29]	41 topics	→ 41 topics were suggested by core team then rated by experts in round one of survey→ 10 additional topics were suggested by experts→ 21 topics were excluded from round two (16 topics had mean scores < 2.5 on one of the 3 criteria and 5 topics were not in top 20)→ 30 topics (top 20 topics and the 10 additions) were sent to experts for further rating→ Top 10 topics were identified using the overall mean rank and by profession	Ranked list of 10 topics
Nast, 2019 [28]	265 topics under disease categories	→ 265 topics were suggested by participants of round 1 of the survey→ Suggestions were combined into 35 broader topics to be ranked by participants→ Suggestions well covered by existing guidelines or not specific to decide whether they are covered by existing guidelines were excluded→ Respondents were asked to rank their top 10 topics in round 2 of the survey→ Top 10 topics were generated according to total weighted points	Ranked list of 10 topics
van der Sanden, 2002 [27]	1027 topics	→ 1027 topics were obtained from 3 methods (survey, peer group, and literature) as follow: • Survey: dentists confirming important of guideline development proposed a maximum of 5 topics with justification • Analysis of peer group (8–10 dental practitioners) discussions over 1989–1998 • All national dental journals and periodicals (n = 8) were screened over 1992–1997→ 4th method: linear regression lines of topics obtained from literature→ Two researchers collected data and reclassified topics into 9 groups; topics reported less than 6 times within a method were excluded→ An overall rank was obtained by adding rank numbers assigned to the 4 methods for each topic; lowest value indicated highest priority→ The reliability of a method was tested by the ‘‘item-rest sum correlation”; a correlation of rank positions of one method with the sum of rank positions obtained by the other 3 methods→ 34 topics were prioritized as a result of ranking and reclassification	Ranked list of 34 topics belonging to 9 topic groups
van der Veer, 2016 [25]	48 topics in 6 categories	→ 48 topics in 6 categories were generated from a scoping literature review (813 titles) and views of an international expert panel→ List was refined by the panel into 47 topics in 7 categories via an open online consultation of professionals (asked to select 50% of topics within each category as priority topics and add suggestions)→ Expert consensus meeting with 2-round voting yielded a ranked list of the 46 topics based on overall median and range. One topic was not included in the rating process due to an administrative error	Ranked list of 46 topics in 7 categories
Updating of guidelines
Agbassi, 2014 [15]	151 PEBC guideline documents	→ 151 PEBC guideline documents were assessed in consultation with a clinical expert and facilitated by a methodologist: 37 archived, 33 deferred, 6 special cases and 75 need review→ Documents within review category underwent prioritization into low- (n = 20), medium- (n = 10), high- (n = 18) and urgent-priority (n = 27)→ In order of priority, clinical expert and methodologist reviewed guidelines (via conducting a streamlined systematic review) to determine effect of new evidence on existing recommendation and further action→ Then documents were classified as either endorsed (n = 15), updated (n = 8) or archived (n = 7), while others either required a new version (n = 1), had a review initiated (n = 7) or the process was incomplete (n = 35)→ Review outcomes (endorsement, archive, and update) should be approved by a larger expert panel comprising a multidisciplinary team of clinicians and other stakeholders	Categorization of guideline documents into endorse (n = 15), update (n = 8) or archive (n = 7)
Becker, 2018 [14]	35 guideline sections	→ Limited search yielded 902 abstracts of potentially relevant evidence on the 35 guideline sections→ Further new evidence was identified via an online survey of CGP group members who also rated the sections based on evidence and clinical relevance→ Sections were subdivided in groups with high (15), medium (9), or low (11) need for update based on median scores→ A consensus conference was held to finalize the list of sections with “high” need for update; 7 sections were allocated from low and middle to high need for update and 2 new sections were suggested based on median voting. However, these additions were not presented in the paper.	Ranked list of 15 guideline sections
van der Veer, 2015 [26]	39 topics in 4 categories	→ 39 topics were drafted as a result of literature review and input from expert group→ In survey round 1, participants (patients and clinicians) ranked topics and suggested 3 new ones (42 topics)→ In survey round 2, participants ranked the 42 topics which were listed based on mean (standard deviation) ratings resulting in two ranked top 10 lists (for patients and clinicians)	Ranked list of 42 topics (with two ranked lists)

There was a variation in the types of outputs of the prioritization exercises. Most of the studies prioritized topics (n = 6) [16, 18, 25, 27–29], one prioritized clinical areas (n = 1) [17], one prioritized drug classes (n = 1) [30], and one prioritized chronic diseases (n = 1) [19]. None prioritized questions or outcomes. Most of the studies provided ranked lists of priorities (n = 8) [16, 17, 19, 25, 27–30], while one study had a topic suggested prior to the exercise and then confirmed as a result of prioritization [18]. The numbers of priorities derived from the initial lists varied between the studies (range 1–46).

Prioritization processes and outputs for updating (n = 3)

Studies that implemented prioritization exercises for updating (n = 3) either assessed candidate guideline documents for updating [15] or selected a specific guideline a priori and assessed potential topics or sections to be covered by updating [14, 26].

Agbassi et al. used a stepwise process in which two questionnaires were implemented to prioritize guidelines for updating and to assess the effect of new evidence on existing recommendations [15]. van der Veer et al. consulted clinicians and patients about priority topics to be covered by the update of 2007 vascular access guideline of the European Renal Best Practice [26]. Becker et al. classified guideline sections of a German clinical practice guideline based on evidence and clinical relevance (Table 6) [14]. Two studies used categories (e.g., urgent, high, medium, or low) to reflect the relative need for updating [14, 15]. One study reported a median of 167 days for the time taken to implement the prioritization process (range 18–358 days) [15].

In terms of outputs, two studies provided ranked lists of priorities [14, 26]. One study prioritized 8 out of 151 guideline documents for updating [15], another study prioritized 15 out of 35 guideline sections [14]. The third study generated a list of 42 topics from an initial list of 39 topics to be covered by the updated guideline [26].

Discussion

Summary of findings

We systematically reviewed the literature for prioritization exercises that have been conducted for the de novo development, update or adaptation of health practice guidelines. We identified twelve eligible studies that focused on prioritizing clinical topics and were predominantly conducted for the de novo development of guidelines; none addressed adaptation. The priority setting exercises consisted of several steps that we grouped in three phases: pre-prioritization, prioritization and post-prioritization. The two most commonly used steps were the generation of an initial list of topics (mostly by seeking input from stakeholders or by reviewing the literature) and ranking of priorities. The two least used steps were research gap analysis and having a revision mechanism. Most of the included studies used prioritization criteria as part of the exercises, with the most common criteria being the health burden and potential impact of the intervention on health outcomes. All studies involved stakeholders, particularly healthcare providers, in prioritizing guideline topics. Stakeholders were mainly involved in the generation of initial list of topics, use of criteria and ranking of priorities.

Interpretation of findings

We observed that the generated priority topics were generally broad and non-specific. This might have been due to the fact that the vast majority of the exercises did not describe a step of refinement of the priority topics. It is essential to refine the topics in a way that would enable an easy transition from topics into meaningful questions appropriate for guideline development [31].

In addition, one of the Institute of Medicine (IOM) standards in guideline development is establishing transparency [32]. The IOM emphasizes the need for detailed and publicly accessible guideline development processes including methods for priority setting [32, 33]. Such processes would increase the credibility and the potential uptake of the end results [34]. We found only three studies reporting on the development of ethical principles to guide the conduct of the exercises.

Prioritization should be supported by an effective dissemination plan to ensure that generated priorities inform prospective research and ultimately improve health [35]. The dissemination of priorities to researchers and funders helps in directing research agendas to guideline topics that are most important to stakeholders [36]. Although the number of prioritization exercises has been increasing over time, very few exercises reported on dissemination or implementation strategies. Consistent with findings of earlier reviews on prioritization for health research [35, 37], only three of the included exercises mentioned dissemination.

Furthermore, our reliance on a recently developed framework of prioritization criteria allowed us to categorize 66 out of the 70 criteria in the included studies. The most commonly used criterion was ‘health burden’. The majority of the remaining criteria were used by two or less exercises. For instance, although equity is one of the most frequently reported criteria in the priority setting literature [38], none of the exercises considered the equity relevance of the condition. An equity-oriented approach to priority setting is important for ensuring inclusiveness [38, 39]. While this could reflect a decision by the designers of the exercises to focus on few but relevant criteria, it could also point to the failure of these exercises to be comprehensive in their use of criteria. Indeed, Nast et al. highlighted the need for such exercises to address a wide range of explicit criteria that extend beyond disease-related factors [28].

Overall, the observed variation in the prioritization steps and criteria used in the included prioritization exercises could potentially be explained by the need to tailor the decision on how to conduct a prioritization exercise to the needs of relevant stakeholders and to the available resources, such as time and funding.

A recent systematic review highlighted the opportunity to engage diverse types of stakeholders in prioritizing guideline topics [24]. Incorporating views of various stakeholders in guideline development can potentially reduce a biased selection of topics by few groups and increase transparency [7, 31]. Moreover, considering the needs of different stakeholders may improve the uptake and usability of guidelines [40]. While all exercises included in this review involved health care providers, only four and one respectively engaged researchers and patients. None engaged the other eight types of stakeholders that we assessed. Patient involvement in priority setting for guideline development has been widely supported in the literature [41-44]. It helps direct guidelines toward questions that matter most to patients, expanding beyond the interests of researchers and clinicians [45-47]. However, potential barriers to patient involvement include limited resources (e.g., lack of funds and stakeholder time), slowed down and longer process, and difficulty in identifying appropriate representatives [42, 43, 48]. In addition, guidance on how to engage patients is limited [43]. Despite potential challenges, some of the available methods for engaging patients have been evaluated, and thus can be used to ensure appropriate patient involvement [49, 50]. Moreover, maintaining regular communication with patients or their representatives facilitates meaningful engagement [51].

The online approach to engaging stakeholders was adopted by all studies. Online platforms are considered practical and cost-efficient ways of engaging stakeholders [52]. Other methods that were not frequently used in the prioritization exercises include in-person meetings and workshops. Although not widely used (for practical and financial reasons), those methods might improve interactions and discussions between stakeholders and in turn generate different priorities. In addition, face-to-face meetings are one of the knowledge exchange methods with the greatest impact on policymaking [53]. Furthermore, most stakeholders were engaged through the Delphi survey method, which is a simple consensus tool for obtaining the views of a large group of relevant stakeholders [54] using structured and iterative group interactions [55]. The Delphi method is commonly used in both guideline development and in health research prioritization [56-58], explaining its use in prioritization for guideline development.

The included studies on prioritization for updating conducted the exercises at different time points of the updating process. One exercise was implemented to identify the clinical guidelines in greatest need of update after a surveillance process, while the other two exercises aimed to identify the topics or sections in greatest need of update for a selected guideline.

Strengths and limitations

We used a rigorous and transparent process including a comprehensive search strategy, duplicate and independent selection, and duplicate and independent data extraction [59]. In addition, and by drawing on an extensive body of literature since the 1990s and up to July 2019, this review synthesizes almost three decades of published research on prioritization for guideline development. On the other hand, we built on two recent systematic reviews of prioritization approaches to develop our data extraction and analysis framework (e.g., how to categorize the steps of prioritization, prioritization criteria).

There are limitations to our scoping review process. First, we did not appraise the quality of the included studies. However, this is consistent with the scoping review methodology [21] and no tool has been developed for the critical appraisal of priority setting exercises. Second, we did not search the grey literature, particularly websites of guideline developing organizations, due to time and resource constraints.

Comparison to other reviews

Our work adds to former reviews on the topic, e.g., the review by Garcia et al. which focused on the update of health decision-making tools, one of which was guidelines [20]. Consistent with our findings, Garcia et al. reported variability in the methods used to implement the prioritization exercises for updating. On the other hand, our study presents a more in-depth analysis of relevant characteristics such as the steps and criteria for prioritization exercises. Our list of criteria is consistent with, but a bit more comprehensive than the list by Garcia et al.

Implications for practice

Our findings can assist clinicians, researchers, funders, policymakers, and other stakeholders seeking to develop health practice guidelines in prioritizing topics to be addressed. Given that there are no standard prioritization best practices for guideline development [7], it might be challenging to provide specific guidance on which prioritization exercise to use. However, the decision on whether and how to conduct a prioritization exercise should be tailored to the needs of relevant stakeholders and to the available resources, including time and funding. Furthermore, the detailed lists of identified steps and criteria can serve as a menu of options for guideline developers to select from, as judged appropriate to the context, and through a transparent decision-making process.

Implications for future research

There is a need to develop methods and guidance for prioritization of not only topics, but also for questions and outcomes in guidelines projects. Exploring the same question of this study through the analysis of guideline handbooks would be helpful for that purpose. Further rigorous evaluation research can help with a better understanding of potential facilitators and barriers to prioritization. Moreover, and because all of the included conducted exercises were developed by researchers from high-income countries, future studies can focus on the effectiveness of the exercises in low- and middle-income countries. It is also essential to evaluate the impact of those exercises on resource allocation and on clinical outcomes.

Conclusions

This review identified 12 prioritization exercises that addressed different aspects of priority setting for guideline development and update. The detailed lists of prioritization steps and criteria can serve as a menu of options for guideline developers to select from, as judged appropriate to the context. This review also provided insight into the types of stakeholders involved in the prioritization of health practice guidelines. Engaging diverse stakeholders, particularly patients and their representatives, is essential to align guideline development with the needs and priorities of relevant stakeholders. However, the roles of stakeholders in the prioritization processes need to be further investigated.

PRISMA-ScR checklist.

(DOCX)

Click here for additional data file.

Study protocol.

(DOCX)

Click here for additional data file.

Search strategy.

(DOCX)

Click here for additional data file.

Data extraction variables.

(DOCX)

Click here for additional data file.

Prioritization criteria framework.

(DOCX)

Click here for additional data file.

22 Nov 2019

PONE-D-19-24700

The implementation of prioritization exercises in the development and update of health practice guidelines: a systematic review

PLOS ONE

Dear Elie A. Akl

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please respond to the reviewers feedback and, in particular, kindly address the suggestions provided on the discussion.

We would appreciate receiving your revised manuscript by 23 December 2019. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Vicki Jane Flenady

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

1. As your study does not include an assessment of methodological quality and risk of bias of the included studies, please consider whether a "scoping review" or "evidence mapping" would be a more appropriate descriptor than "systematic review." Please update the title and text to reflect this.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: N/A

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Summary of the research and overall impression

In this manuscript, the authors perform a comprehensive systematic review of studies describing priority setting for guideline development and update, to better understand methodologies in use for this decision-making process. This is a well-constructed, detailed and considered review. Strengths and limitations are well described, and data (including supplementary files) is presented in a clear and detailed manner. The reported findings from 12 eligible studies that there is substantial variation in the methods used to undertake prioritisations exercises whilst not surprising, provides valuable evidence to highlight the importance of explicit and transparent prioritisation processes in this area.

Major issues

None

Minor Issues

1. I am uncertain of the appropriateness of referencing a review which has been submitted but not yet published. Has your previous review on prioritization for evidence synthesis now been published, if so, please reference accordingly? (Reference number 22- Fadlallah R, El-Harakeh A, Bou-Karroum L, Lotfi T, El-Jardali F, Hishi L, et al. Prioritizing topics or questions for evidence syntheses in health: a systematic review. Manuscript submitted for publication. 2019.)

Reviewer #2: This study sought to systematically review the use of prioritisation processes for developing, updating, or adapting clinical practice guidelines in health care. Twelve eligible studies are included and described, of which all focussed on prioritising broad clinical topics. There was variation in the prioritisation exercises on various domains, including stakeholder involvement, steps of prioritisation addressed, and criteria used to prioritise topics.

This is a comprehensive and rigorous review with important implications for future research and guideline development. The rationale, objectives, and potential value of the review are clearly described. In general, the manuscript is very well-written, particularly the introduction, methods, and results. The manuscript and supplementary files are well-organised and appropriately detailed. The authors should be commended on their clear and comprehensive reporting of methods and results. Overall, this review offers several important observations about prioritisation processes, including potential deficiencies, which can assist to improve and standardise such initiatives for researchers and guideline developers.

I have a few comments and suggestions, which focus mainly on the Discussion.

Main comments:

- I felt the 'summary of findings' had omitted some interesting and important results of the review that would be worth briefly summarising in that section of the manuscript. Specifically, the dominant criteria used to prioritise topics (i.e. health burden and impact on health outcomes) and stakeholder involvement (i.e. this this was largely focussed in care providers). The latter is particularly important to mention in the summary given (diverse) stakeholder involvement is critical for maximising relevance and uptake of guidelines, as stated in the introduction and later in the discussion.

- It was striking that only one of the prioritisation exercises engaged patients or their representatives. This is a clear gap in the reviewed prioritisation exercises and it is worth exploring in the discussion the potential reasons why this might be the case. For example, it may be that research compliance issues and/or funding and resource limitations makes patient-engagement more difficult? These sorts of potential barriers (and their possible solutions) could be mentioned in paragraph 3 of the interpretation of findings (p. 30).

- Paragraph 4 of the interpretation of results (p. 30) could be expanded. It is noted that all studies adopted an online approach to engagement, likely for practical and financial reasons. It is worth mentioning the other methodologies available for priority-setting exercises, that have been potentially under-utilised (e.g. workshops and face-to-face meetings). Such methodologies may be more accessible for certain stakeholder types, and may also yield different priorities…

- As a whole, the interpretation of results felt somewhat brief. I felt it could be improved by drawing out a few more of the review's salient findings, for discussion and critical appraisal. For example, only 3 studies reported on the development of ethical principles to guide the conduct of the exercise; only 3 reported a plan for dissemination; and none included equity relevance of the condition. Do the authors see these as important gaps in current practice that should be addressed in future priority setting exercises?

- The authors state that having not searched the grey literature was a limitation of their review. It was not clear to me what was meant by "Nonetheless, these would require a different search strategy and would not reflect real life exercises.". Was the intention to say that prioritisation exercises would not be published in grey literature? Please clarify in the manuscript.

- I think the findings of this review will also be of benefit to researchers who may design and conduct priority setting activities in future. It is worth adding 'researchers' explicitly to the list given in the opening paragraph of the 'implications for practice' (p. 32).

- As the authors state under the implications for practice, "the decision on whether and how to conduct a prioritization exercise should be tailored to the needs of relevant stakeholders and to the available resources, including time and funding" (p. 32). Indeed, this may explain some of the variation in the prioritisation exercises reviewed. It is worth stating this in the discussion.

- I think it is important to (re)state in the conclusion the importance of diverse stakeholder involvement, particularly with regard to patients and their representatives, which the review has highlighted as a significant gap.

Minor comments:

- The % symbol is missing from the data label row in Table 4.

- There is an 's' missing from 'stakeholder' in the second paragraph under Stakeholder involvement (p. 20).

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: PONE-D-19-24700_comments.docx

Click here for additional data file.

14 Jan 2020

14 January 2020

Professor Vicki Jane Flenady

Academic Editor, PLOS ONE

Re: “The implementation of prioritization exercises in the development and update of health practice guidelines: a scoping review”

Dear Dr. Flenady,

Thank you for the opportunity to revise and resubmit our above titled manuscript to PLOS ONE. We thank you and the reviewers for taking the time to review the manuscript. We found the comments and suggestions very constructive and used them to improve it.

Kindly find below a point-by-point response to the comments and a description of the changes made to the manuscript.

Thank you so much and we look forward to the outcome of the peer review process.

Sincerely,

Elie A. Akl, MD, MPH, PhD

AUBMC, Department of Internal Medicine

P.O. Box: 11-0236

Riad-El-Solh Beirut 1107 2020

Beirut – Lebanon

Phone: 00961 1 374374

ea32@aub.edu.lb

The reviewers’ comments and our response are listed below.

Editor: As your study does not include an assessment of methodological quality and risk of bias of the included studies, please consider whether a "scoping review" or "evidence mapping" would be a more appropriate descriptor than "systematic review."

Please update the title and text to reflect this.

Response:

Thank you. After careful revision, we changed the study descriptor to ‘scoping review’ and updated the title and text of the manuscript accordingly.

Reviewer 1: Summary of the research and overall impression

In this manuscript, the authors perform a comprehensive systematic review of studies describing priority setting for guideline development and update, to better understand methodologies in use for this decision-making process. This is a well-constructed, detailed and considered review. Strengths and limitations are well described, and data (including supplementary files) is presented in a clear and detailed manner. The reported findings from 12 eligible studies that there is substantial variation in the methods used to undertake prioritisations exercises whilst not surprising, provides valuable evidence to highlight the importance of explicit and transparent prioritisation processes in this area.

Major issues

None

Response:

We thank the Reviewer for this very positive feedback.

Minor Issues

1. I am uncertain of the appropriateness of referencing a review which has been submitted but not yet published. Has your previous review on prioritization for evidence synthesis now been published, if so, please reference accordingly? (Reference number 22- Fadlallah R, El-Harakeh A, Bou-Karroum L, Lotfi T, El-Jardali F, Hishi L, et al. Prioritizing topics or questions for evidence syntheses in health: a systematic review. Manuscript submitted for publication. 2019.)

Response:

Thank you for this question. The paper was accepted for publication on December 11, 2019, which allows us to include a definitive citation.

Fadlallah R, El-Harakeh A, Bou-Karroum L, Lotfi T, El-Jardali F, Hishi L, et al. A common framework of steps and criteria for prioritizing topics for evidence syntheses: a systematic review. Manuscript submitted for publication. 2019. Journal of Clinical Epidemiology. 2019.

Reviewer #2:

This study sought to systematically review the use of prioritisation processes for developing, updating, or adapting clinical practice guidelines in health care. Twelve eligible studies are included and described, of which all focused on prioritising broad clinical topics. There was variation in the prioritisation exercises on various domains, including stakeholder involvement, steps of prioritisation addressed, and criteria used to prioritise topics.

This is a comprehensive and rigorous review with important implications for future research and guideline development. The rationale, objectives, and potential value of the review are clearly described. In general, the manuscript is very well-written, particularly the introduction, methods, and results. The manuscript and supplementary files are well-organised and appropriately detailed. The authors should be commended on their clear and comprehensive reporting of methods and results. Overall, this review offers several important observations about prioritisation processes, including potential deficiencies, which can assist to improve and standardise such initiatives for researchers and guideline developers.

Response:

We sincerely thank the Reviewer for the positive and constructive assessment of our manuscript.

I have a few comments and suggestions, which focus mainly on the Discussion.

Main comments:

Comment 1:

I felt the 'summary of findings' had omitted some interesting and important results of the review that would be worth briefly summarising in that section of the manuscript. Specifically, the dominant criteria used to prioritise topics (i.e. health burden and impact on health outcomes) and stakeholder involvement (i.e. this was largely focused in care providers). The latter is particularly important to mention in the summary given (diverse) stakeholder involvement is critical for maximizing relevance and uptake of guidelines, as stated in the introduction and later in the discussion.

Response:

Thank you for highlighting this issue. We agree and have amended the ‘summary of findings’ section by adding the following (p. 29):

“The two most commonly used steps were the generation of an initial list of topics (mostly by seeking input from stakeholders or by reviewing the literature) and ranking of priorities. The two least used steps were research gap analysis and having a revision mechanism. Most of the included studies used prioritization criteria as part of the exercises, with the most common criteria being the health burden and potential impact of the intervention on health outcomes. All studies involved stakeholders, particularly healthcare providers, in prioritizing guideline topics. Stakeholders were mainly involved in the generation of initial list of topics, use of criteria and ranking of priorities.”

Comment 2:

It was striking that only one of the prioritisation exercises engaged patients or their representatives. This is a clear gap in the reviewed prioritisation exercises, and it is worth exploring in the discussion the potential reasons why this might be the case. For example, it may be that research compliance issues and/or funding and resource limitations makes patient engagement more difficult? These sorts of potential barriers (and their possible solutions) could be mentioned in paragraph 3 of the interpretation of findings (p. 30).

Response:

We thank the reviewer for bringing this to our attention. Indeed, the involvement of patients in only one exercise was surprising. We now reflect on patient engagement in the discussion section and assess potential reasons for not involving patients in most prioritization exercises, as per the below (p. 30):

Patient involvement in priority setting for guideline development has been widely supported in the literature [41-44]. It helps direct guidelines toward questions that matter most to patients, expanding beyond the interests of researchers and clinicians [45-47]. However, potential barriers to patient involvement include limited resources (e.g., lack of funds and stakeholder time), slowed down and longer process, and difficulty in identifying appropriate representatives [42, 43, 48]. In addition, guidance on how to engage patients is limited [43]. Despite potential challenges, some of the available methods for engaging patients have been evaluated, and thus can be used to ensure appropriate patient involvement [49, 50]. Moreover, maintaining regular communication with patients or their representatives facilitates meaningful engagement [51].

Comment 3:

Paragraph 4 of the interpretation of results (p. 30) could be expanded. It is noted that all studies adopted an online approach to engagement, likely for practical and financial reasons. It is worth mentioning the other methodologies available for priority-setting exercises, that have been potentially under-utilised (e.g. workshops and face-to-face meetings). Such methodologies may be more accessible for certain stakeholder types, and may also yield different priorities…

Response:

We agree with the reviewer and have expanded paragraph four of the ‘interpretation of results’ to examine the value of other methods (workshops and face-to-face meetings). Please see below:

Other methods that were not frequently used in the prioritization exercises include in-person meetings and workshops. Although not widely used (for practical and financial reasons), those methods might improve interactions and discussions between stakeholders and in turn generate different priorities. In addition, face-to-face meetings are one of the knowledge exchange methods with the greatest impact on policymaking [53].

Comment 4:

As a whole, the interpretation of results felt somewhat brief. I felt it could be improved by drawing out a few more of the review’s salient findings, for discussion and critical appraisal. For example, only 3 studies reported on the development of ethical principles to guide the conduct of the exercise; only 3 reported a plan for dissemination; and none included equity relevance of the condition. Do the authors see these as important gaps in current practice that should be addressed in future priority setting exercises?

Response:

Thank you, this has been detailed as follow (p. 30):

In addition, one of the Institute of Medicine (IOM) standards in guideline development is establishing transparency [32]. The IOM emphasizes the need for detailed and publicly accessible guideline development processes including methods for priority setting [32, 33]. Such processes would increase the credibility and the potential uptake of the end results [34]. We found only three studies reporting on the development of ethical principles to guide the conduct of the exercises.

Prioritization should be supported by an effective dissemination plan to ensure that generated priorities inform prospective research and ultimately improve health [35]. The dissemination of priorities to researchers and funders helps in directing research agendas to guideline topics that are most important to stakeholders [36]. Although the number of prioritization exercises has been increasing over time, very few exercises report on dissemination or implementation strategies. Consistent with findings of earlier reviews on prioritization for health research [35, 37], only three of the included exercises mentioned dissemination.

Furthermore, our reliance on a recently developed framework of prioritization criteria allowed us to categorize 66 out of the 70 criteria in the included studies. The most commonly used criterion was ‘health burden’. The majority of the remaining criteria were used by two or less exercises. For instance, although equity is one of the most frequently reported criteria in the priority setting literature [38], none of the exercises considered the equity relevance of the condition. An equity-oriented approach to priority setting is important for ensuring inclusiveness [38, 39]. While this could reflect a decision by the designers of the exercises to focus on few but relevant criteria, it could also point to the failure of these exercises to be comprehensive in their use of criteria.

Comment 5:

The authors state that having not searched the grey literature was a limitation of their review. It was not clear to me what was meant by "Nonetheless, these would require a different search strategy and would not reflect real life exercises.". Was the intention to say that prioritisation exercises would not be published in grey literature? Please clarify in the manuscript.

Response:

Thank you for the comment. Obviously, we were not as clear as we should have been. Our assessment is that unpublished priority setting exercises could be available on websites of organizations that produce guidelines. We felt that the identification of all of these organizations and sifting through their websites would be a very time and resource consuming task that might not return much information. We have clarified this point under the ‘strengths and limitations’ section as follows:

“Second, we did not search the grey literature, particularly websites of guideline developing organizations, due to time and resource constraints.

Comment 6:

I think the findings of this review will also be of benefit to researchers who may design and conduct priority setting activities in future. It is worth adding 'researchers' explicitly to the list given in the opening paragraph of the 'implications for practice' (p. 32).

Response:

We agree that researchers can benefit from the findings of our review in designing prioritization exercises for guidelines. We have made the suggested edit as follows:

Our findings can assist clinicians, researchers, funders, policymakers, and other stakeholders seeking to develop health practice guidelines in prioritizing topics to be addressed.”

Comment 7:

As the authors state under the implications for practice, "the decision on whether and how to conduct a prioritization exercise should be tailored to the needs of relevant stakeholders and to the available resources, including time and funding" (p. 32). Indeed, this may explain some of the variation in the prioritisation exercises reviewed. It is worth stating this in the

discussion.

Response:

Thank you. This has been clarified in the discussion section as follows (p. 30):

“The observed variation in steps and criteria used the included prioritization exercises could potentially be explained by the need to tailor the decision on how to conduct a prioritization exercise to the needs of relevant stakeholders and to the available resources, such as time and funding.”

Comment 8:

I think it is important to (re)state in the conclusion the importance of diverse stakeholder involvement, particularly with regard to patients and their representatives, which the review has highlighted as a significant gap.

Response:

Thank you for bringing this to our attention. We have edited the conclusion as follows (p. 33):

“This review also provided insight into the types of stakeholders involved in the prioritization of health practice guidelines. Engaging diverse stakeholders, particularly patients and their representatives, is essential to align guideline development with the needs and priorities of relevant stakeholders.”

Minor comments:

- The % symbol is missing from the data label row in Table 4.

- There is an 's' missing from 'stakeholder' in the second paragraph under Stakeholder involvement (p. 20).

Response:

Thank you for highlighting this issue. We modified Table 4 as suggested. We also changed 'stakeholder' to ‘stakeholders’ in the second paragraph under the ‘stakeholder involvement’ section (p. 20).

Submitted filename: Response letter_Guideline prioritization exercises_20200114.docx

Click here for additional data file.

4 Feb 2020

The implementation of prioritization exercises in the development and update of health practice guidelines: a scoping review

PONE-D-19-24700R1

Dear Elie Aki

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Vicki Jane Flenady

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

7 Feb 2020

PONE-D-19-24700R1

The implementation of prioritization exercises in the development and update of health practice guidelines: a scoping review

Dear Dr. Akl:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Vicki Jane Flenady

Academic Editor

PLOS ONE