Development of a prioritisation tool for the updating of clinical guideline questions: the UpPriority Tool protocol.

INTRODUCTION: Due to a continuous emergence of new evidence, clinical guidelines (CGs) require regular surveillance of evidence to maintain their trustworthiness. The updating of CGs is resource intensive and time consuming; therefore, updating may include a prioritisation process to efficiently ensure recommendations remain up to date. The objective of our project is to develop a pragmatic tool to prioritise clinical questions for updating within a CG. METHODS AND ANALYSIS: To develop the tool, we will use the results and conclusions of a systematic review of methodological research on prioritisation processes for updating and will adopt a methodological approach we have successfully implemented in a previous experience.We will perform a multistep process including (1) generation of an initial version of the tool, (2) optimisation of the tool (feasibility test of the tool, semistructured interviews, Delphi consensus survey, external review by CG methodologists and users and pilot test of the tool) and (3) approval of the final version of the tool.At each step of the process, we will (1) calculate absolute frequencies and proportions (quantitative data), (2) use content analysis to summarise and draw conclusions (qualitative data) and (3) draft a final report, discuss results and refine the previous versions of the tool. Finally, we will calculate intraclass coefficients with 95% CIs for each item and overall as indicators of agreement among reviewers. ETHICS AND DISSEMINATION: We have obtained a waiver of approval from the Clinical Research Ethics Committee at the Hospital de la Santa Creu i Sant Pau (Barcelona). The results of the study will be published in peer-reviewed journal and communicated to interested stakeholders.The tool could support the standardisation of prioritisation processes for updating CGs and therefore have important implications for a more efficient use of resources in the CG field. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

To develop the tool, we will use the results and conclusions of a systematic review of methodological research on prioritisation processes for updating.

We will adopt a methodological approach we have successfully implemented in a previous experience.

We will collect views from clinical guidelines (CG) developers (semistructured interviews and external reviews), CG methodological experts (Delphi consensus survey) and CG users (semistructured interviews); these will allow us to pool different stakeholders’ opinions about CG updating prioritisation processes.

The principal limitation of the study is that we will not perform a formal validation of the tool.

Introduction

Clinical guidelines (CGs) are ‘statements that include recommendations intended to optimise patient care that are informed by systematic reviews (SRs) of evidence and an assessment of the benefits and harms of alternative care options’.1 Due to a continuous emergence of new evidence,2 3 CGs require regular surveillance of evidence to maintain their trustworthiness.4 5

Several studies have assessed length of time that CGs and their recommendations remain valid.4–8 Based on this evidence, most CG developers have adopted updating strategies based on predetermined time frames.9

An updating strategy involves different processes including the identification of new evidence; the assessment of the impact of new evidence on the current CG recommendations and whether an update is required and the update of the CG if needed.9 10 The updating of CGs is resource intensive and time consuming.11 In the current context of restricted resources, there is a growing interest in approaches that support decision-making for updating CGs.12

We define the prioritisation process for updating of CGs as the methodology used to determine which CGs should be prioritised to ensure that resources are invested in updating the topics that are most relevant to different stakeholders.12 The prioritisation process includes two main stages: (1) assessment of CGs using prioritisation criteria (eg, availability of new evidence, clinical relevance or users’ interest) and (2) classification of CGs in groups according to priority for updating (eg, high, medium or low relevance for updating).12

Different prioritisation processes could be implemented at different time points within an updating strategy. For example, a prioritisation process could be implemented to identify the CGs in greatest need of update (prioritisation across available CGs)13 14 or to identify the clinical questions in greatest need of update within a prioritised CG (prioritisation within a CG).15 16

Until now, there is wide variability and suboptimal reporting of the methods used to develop and implement processes to prioritise updating of CGs.12

Aims and objectives

Primary objective

To develop a pragmatic tool to prioritise clinical questions for updating within a CG.

To identify the most important items required to prioritise clinical questions for updating within a CG.

To describe each item, establish a rating scale of items and provide a guidance on how to rate them.

To develop guidance on how to calculate and present priority scores to support decision-making for updating clinical questions within a CG.

Methods and analysis

To develop the UpPriority Tool, we will use the results and conclusions of a systematic review of methodological research on prioritisation processes for updating12 and will adopt a methodological approach we have successfully implemented in a previous experience.17 We will perform a multistep process including (1) generation of an initial version of the tool, (2) optimisation of the tool (feasibility test of the tool, semistructured interviews, Delphi consensus survey, external review by CG methodologists and users and pilot test of the tool) and (3) approval of the final version of the tool (table 1, figure 1).

Table 1

Characteristics of the multistep development process

	Generation of the initial version	Optimisation of the tool						Approval of the final version
		Feasibility test	Semistructured interviews	Delphi consensus survey	External review with clinical guidelines developers	External review with clinical guidelines users	Pilot test
Objective	To develop the initial version of the tool	To explore the feasibility of the tool	To identify current practices in prioritisation processes for updating CGs	To reach a consensus about the included items of the tool	To assess the usefulness* and understanding of each item of the tool	To assess the usefulness* and understanding of each item of the tool	To explore the interobserver reliability of the final version of the tool	To approve the final version of the tool
Study design	–	Methodological survey	Semistructured interviews	Delphi consensus survey	Survey	Semistructured interviews	Methodological survey	–
Participants	UpSG	CG	CG developers	CG methodological experts from G-I-N Updating Guidelines Working Group	CG developers from G-I-N community	CG users	CG	UpSG
Main outcome	–	Time to apply the tool	Participants’ experiences with prioritisation processes for updating CGs	Items considered important to prioritise clinical questions for updating within a CG	Usefulness* rating for each item of the tool	Participants’ views of prioritisation processes for updating CGs with the tool	Intraclass coefficient with 95% CI	–
Study size	–	Convenience sample	Sampling saturation	20–30 participants	250 organisations and individual members	Sampling saturation	Convenience sample	–

*Usefulness: The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.29

CG, clinical guideline; G-I-N, Guidelines International Network; UpSG, UpPriority Steering Group.

Figure 1

Multistep development process.

Generation of the initial version of the tool

Objective

The objective is to develop the initial version of the tool (items, scoring calculation and summary report).

Method

The UpPriority Steering Group (UpSG) will participate in informal discussion and will approve the initial version of the tool.

Participants

UpSG.

Optimisation of the tool

Feasibility test of the tool

The objective is to explore the feasibility and refine the initial version of the tool.

Study design

Methodological survey.

A CG developed within the Spanish National Health System Clinical Guideline Program, published within the last 2 years and with <50 clinical questions.

Main outcome

Time to apply the tool.

Other variables

Response rate, characteristics of participants and workplace, characteristics of clinical questions, priority scores (single item and overall items) and overall assessment of the tool (table 2).

Table 2

Study variables in multistep development process

	Feasibility test	Semistructured interviews	Delphi consensus survey	External review with clinical guidelines developers	External review with clinical guidelines users	Pilot test
Response rate	X		X	X		X
Characteristics of participants and workplace	X	X	X	X	X	X
Characteristics of clinical questions	X					X
Priority scores	X					X
Current practices in prioritisation processes for updating CGs		X
Assessment of each item		X	X (inclusion and understanding)	X (usefulness and understanding)	X (usefulness and understanding)
Assessment of the scores calculation		X	X	X	X
Assessment of the summary report		X	X	X	X
Overall assessment of the tool	X	X	X	X	X	X

CG, clinical guideline.

Data collection

Two reviewers from the original Guideline Development Group (GDG) and two reviewers from the UpSG will apply the initial version of the tool. We will use online software to design the survey and collect responses (www.digestepiclin.com).

Bias

To minimise non-response bias, the survey will be available online for 1 month; weekly email reminders will be sent to reviewers. To minimise observer bias, two reviewers from outside the UpSG will apply the tool.

Study size

Convenience sample.18

Data analysis

For quantitative data, we will calculate absolute frequencies and proportions. For qualitative data, we will use content analysis to summarise and draw conclusions (atlasti.com).19 Questionnaires with no response in over 20% of the items will be withdrawn. We will draft a final report, discuss results and refine the initial version of the tool with the UpSG.

Semistructured interviews

The objective is to identify current practices in prioritisation processes for updating CGs and to refine the initial version of the tool.

Semistructured interviews (face-to-face, telephone or internet).

CG developers that (1) have experience in CG development and/or updating (defined as having participated in GDG and/or Guideline Updating Group (GUG) at least once in the past year) and (2) are fluent in English or Spanish. We will identify participants with the help of the UpSG. When someone does not respond or cannot participate, another contributor will be recruited.

Participants’ experiences with prioritisation processes for updating CGs.

Characteristics of participants and workplace, current practices in prioritisation processes for updating CGs, assessment of each item, assessment of the scoring calculation, assessment of the summary report and overall assessment of the tool (table 2).

Interviews will be audiotaped and transcribed (each interview will last approximately 1 hour).

To minimise interviewer bias, semistructured interviews will be conducted using an interview guide.

We will recruit participants and collect data until information becomes repetitive and no new information emerges (sampling saturation).20 21

For quantitative data, we will calculate absolute frequencies and proportions. For qualitative data, we will use content analysis to summarise and draw conclusions (atlasti.com).19 We will draft a final report, discuss results and refine the initial version of the tool with the UpSG.

Delphi consensus survey

The objective is to reach a consensus about the included items and refine the initial version of the tool.

Delphi consensus survey.

Before the first Delphi round, we will provide the results of previous methodological research to Delphi panel members.

In the first Delphi round, we will ask participants to rate whether each item should be included in the tool and its clarity using a seven-point Likert scale (1=strongly disagree and 7=strongly agree).22 We will calculate the median score for inclusion of each item and will classify them as (1) excluded (median score of 0–3 points), (2) review, modify and retest (median score of 4–5 points or with substantial comments) and (3) included (median score of 6 to 7 points and without substantial comments).

After each Delphi round, we will provide feedback to Delphi panel members (all responses will be anonymised prior to circulation). We will conduct additional Delphi rounds until consensus for inclusion or exclusion is reached and no more relevant comments were provided (two or three rounds, as needed).

CG methodological experts that (1) have methodological experience in CGs development and/or updating (defined as having participated in a CG technical team at least once in the past year and/or in methodological research) and (2) are fluent in English or Spanish. We will identify participants by contacting professionals associated with the Guidelines International Network (G-I-N) Updating Guidelines Working Group (http://www.g-i-n.net/working-groups/updating-guidelines) or authors of methodological research. Non-responders will not be invited to subsequent rounds.

Items considered important to prioritise clinical questions for updating within a CG.

Other variables (per round)

Characteristics of participants and workplace, assessment of each item (inclusion and understanding), assessment of the scoring calculation, assessment of the summary report and overall assessment of the tool (table 2).

We will use online software to design the survey and collect responses (www.digestepiclin.com).

To minimise selection bias of Delphi panel members, all G-I-N Updating Guidelines Working Group members will be invited to participate. To minimise non-response bias, the survey will be available online for 1 month; weekly email reminders will be sent to reviewers.

Twenty to 30 participants.23

For quantitative data, we will calculate absolute frequencies and proportions. For qualitative data, we will use content analysis to summarise and draw conclusions (atlasti.com).19 Questionnaires with no response in over 20% of the items will be withdrawn. We will draft a final report, discuss results and refine the initial version of the tool with the UpSG.

External review

External review with clinical guidelines developers

Objective

The objective is to assess the usefulness and understanding of each item and refine the initial version of the tool.

Study design

Survey.

Participants

CG developers that (1) have experience in CG development/updating (defined as having participated in GDG and/or GUG at least once in the past year) and (2) are fluent in English or Spanish. We will identify participants by contacting professionals associated with the G-I-N community (http://www.g-i-n.net).

Main outcome

Usefulness rating for each item of the tool.

Other variables

Characteristics of participants and workplace, assessment of each item (usefulness and understanding), assessment of the scoring calculation, assessment of the summary report and overall assessment of the tool (table 2).

Data collection

We will use online software to design the survey and collect responses (www.digestepiclin.com).

Bias

To minimise selection bias of survey participants, all G-I-N members will be invited to participate. To minimise non-response bias, the survey will be available online for 1 month; weekly email reminders will be sent to reviewers. Furthermore, the questionnaire will be pilot tested to improve wording and layout.

Study size

Currently, about 250 organisations and individual members are registered in the G-I-N community (http://www.g-i-n.net/membership/members-around-the-world).

Data analysis

For quantitative data, we will calculate absolute frequencies and proportions. For qualitative data, we will use content analysis to summarise and draw conclusions (atlasti.com).19 Questionnaires with no response in over 20% of the items will be withdrawn. We will draft a final report, discuss results and refine the initial version of the tool with the UpSG.

External review with clinical guidelines users

The objective is to assess the usefulness and understanding of each item and refine the initial version of the tool.

Semistructured interviews (face-to-face, telephone or internet).

CG users (defined as healthcare professionals that use CGs on a regular basis) who are fluent in English or Spanish. We will identify participants with the help of the UpSG. When someone does not respond or cannot participate, a new contributor will be recruited.

Participants’ views of prioritisation processes for updating CGs with the tool.

Characteristics of participants and workplace, assessment of each item (usefulness and understanding), assessment of the scoring calculation, assessment of the summary report and overall assessment of the tool (table 2).

Interviews will be audiotaped and transcribed (each interview will last approximately 1 hour).

To minimise interviewer bias, semistructured interviews will be conducted using an interview guide.

We will recruit participants and collect data until information becomes repetitive and no new information emerges (sampling saturation).20 21

For quantitative data, we will calculate absolute frequencies and proportions. For qualitative data, we will use content analysis to summarise and draw conclusions (atlasti.com).19 We will draft a final report, discuss results and refine the initial version of the tool with the UpSG.

Pilot test of the tool

The objective is to explore the interobserver reliability of the final version of the tool and refine the initial version of the tool.

Methodological survey.

A CG developed within the Spanish National Health System Clinical Guideline Programme, published within the last 2 years and with <50 clinical questions.

Intraclass coefficient (ICC) with 95% CI for each item and overall.

Response rate, characteristics of participants and workplace, characteristics of clinical questions and priority scores (single item) and overall assessment of the tool (table 2).

Two reviewers from the original GDG and two reviewers from the UpSG will apply the initial version of the tool. We will use online software to design the survey and collect responses (www.digestepiclin.com).

To minimise non-response bias, the survey will be available online for 1 month; weekly email reminders will be sent to reviewers. To minimise observer bias, two reviewers from outside the UpSG will apply the tool.

Convenience sample; the results of the pilot test will inform the sample size calculation for a subsequent main study.24

For quantitative data, we will calculate absolute frequencies and proportions. For qualitative data, we will use content analysis to summarise and draw conclusions (atlasti.com).19 Questionnaires with no response in over 20% of the items will be withdrawn. We will calculate the ICC with 95% CI for each item and overall as an indicator of agreement among reviewers. According to the scale proposed by Landis and Koch, the degree of agreement between 0.00 and 0.20 is poor, from 0.21 to 0.40 is fair, from 0.41 to 0.60 is moderate, from 0.61 to 0.80 is substantial and from 0.81 to 1.00 is almost perfect.25 We will draft a final report, discuss results and refine the initial version of the tool with the UpSG.

Approval of the final version of the tool

Objective

The objective is to approve the final version of the tool (items, scoring calculation and summary report).

Method

The UpSG will participate in informal discussion and will approve the final version of the tool.

Participants

UpSG.

Ethics and dissemination

We have obtained a waiver of approval from the Clinical Research Ethics Committee at the Hospital de la Santa Creu i Sant Pau (Barcelona, Spain), since this study will not involve patients or biological samples.

The results of the study will be published in peer-reviewed journal and communicated to interested stakeholders (eg, via international conferences, electronic bulletin or website).

We will develop the UpPriority tool through a comprehensive development process, including the use of previous methodological evidence,12 17 feasibility testing of the tool and engagement of the international CG community (semistructured interviews, Delphi consensus survey and external review) and finally a pilot testing of the tool.

Previous SRs on CG updating strategies found limited evidence on processes that could inform the decision of which CGs should be prioritised for updating.9 10 26 There are, nevertheless, new studies that underscore the relevance of the prioritisation process in CG updating,13 27 coinciding with a growing interest among developers to shift from developing to updating CGs.28

We recently systematically reviewed the available evidence on strategies to prioritise the updating of SRs, health technology assessments and CGs.12 We observed that there is wide variability and suboptimal reporting of the methods used to develop and implement such prioritisation processes. Therefore, developers may have difficulties selecting and implementing a prioritisation method to optimise the updating process of CGs.

Agbassi et al13 implemented an annual step-by-step prioritisation process of CGs for updating.13 The authors reviewed CGs using two questionnaires; the process requires evidence search, evidence review and review approval.13 We will build our proposal on this process while addressing some of its shortcomings. Following a comprehensive development process, we will develop a pragmatic survey based tool that will likely be less resource intensive and time consuming compared with formal approaches (based on step-by-step algorithm that generally includes literature searches). We will also publish detailed and explicit guidance to allow developers to implement the tool in their institutions and to adapt it, if needed, to their specific circumstances.

We expect to develop a pragmatic tool (items, scoring calculation and summary report) that will be applicable to all clinical questions within a CG and should be easy to uptake by CG developers. The UpPriority Tool could support the standardisation of prioritisation processes for updating CGs and therefore have important implications for a more efficient use of resources in the CG field.