Protocol of an interdisciplinary consensus project aiming to develop an AGREE II extension for guidelines in surgery.

INTRODUCTION: Appraisal of Guidelines for Research and Evaluation (AGREE II) is an instrument that informs development, reporting and assessment of clinical practice guidelines. Previous research has demonstrated the need for improvement in methodological and reporting quality of clinical practice guidelines specifically in surgery. We aimed to develop an AGREE II extension document for application in surgical guidelines. METHODS AND ANALYSIS: We have performed a structured literature review and assessment of guidelines in surgery using the AGREE II instrument. In exploratory analyses, we have identified factors associated with guideline quality. We have performed reliability and factor analyses to inform the development of an extension document. We will summarise this information and present it to a Delphi panel of stakeholders. We will perform iterative Delphi rounds and we will summarise the final results to develop the extension instrument in a dedicated consensus conference. ETHICS AND DISSEMINATION: Funding bodies will not be involved in the development of the instrument. Research ethics committee and Health Research Authority approval was waived, since this is a professional staff study only and no duty of care lies with the National Health Service to any of the participants. Conflicts of interest, if any, will be addressed by reassigning functions or replacing participants with relevant conflicts. The results will be disseminated through publication in peer reviewed journals, the funders' websites, social media and direct contact with guideline development organisations and peer-reviewed journals that publish guidelines. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This is the first project to address guideline development and reporting in surgery.

It will combine statistical considerations, conceptual parameters to be derived from qualitative synthesis and a formal Delphi process.

It will involve a panel of stakeholders from a variety of scientific, cultural and geographical backgrounds.

The project will not address specific disciplines of surgery.

Introduction

Research evidence is the primary source to inform medical practice forming the cornerstone of evidence-based medicine.1 An average of 5639 new articles were indexed per month under the subject heading ‘Surgery’ in the National Library of Medicine over the past decade.2 Given this fact, keeping abreast of the latest evidence is a strenous task for healthcare practitioners. Clinical practice guidelines evaluate, summarise and contextualise research evidence into actionable recommendations.3 As such, guidelines have a direct impact on delivery of healthcare and surgical services. It is therefore of paramount importance to ensure the highest quality standards in developing and reporting guidelines.

A great amount of scientific endeavour in the past few years has focused on the quality of scholarly work, including clinical practice guidelines.4 Reporting standards have been developed for virtually all study designs and have been summarised by the Enhancing the Quality and Transparency of health Research (EQUATOR) Network.5 Appraisal of Guidelines for Research and Evaluation (AGREEII) constitutes a framework for developing, appraising and reporting clinical practice guidelines.6 It is endorsed by major international and national agencies, including the WHO and the National Institute for Health and Care Excellence (NICE) in the UK.7 8 AGREE II is a generic tool that applies to all disciplines of medicine, and no modification or extension of the framework has been proposed, described or developed for specific clinical branches such as surgery.

The tool is composed of 23 items organised in seven thematic domains: scope and purpose, stakeholder involvement, rigour of development, clarity of presentation, applicability and editorial independence. It concludes with an overall assessment and a statement of whether the guideline is considered of sufficient quality to be used or recommended in clinical practice (online supplementary appendix).

Need for an AGREE II extension

Our research group has acted as methodological and content coordinators of landmark surgical guidelines and have served as members of surgical guideline development groups.9–14 Even though members of our group, in their role as guideline developers, have made every effort to comply with the highest methodological standards, as indicated by adherence to Grading of Recommendations Assessment, Development and Evaluation (GRADE) and AGREE II methodologies,15 16 we noticed that compliance with all aspects of several parameters of the AGREE II instrument was not possible.

For example, the item ‘The potential resource implications of applying the recommendations have been considered’ may be difficult to be universally addressed. Cost-effectiveness studies are scarce in the surgical literature and relevant evidence typically varies in different settings.17 Since surgical expertise varies across countries and institutions, there is a need for the instrument to consistently apply to different healthcare settings. Surgical interventions are complex, and details on the interventions/comparators are imperative for the target users to be able to assess the external validity of the guidelines. Specialists from different specialties and allied health professionals with a wide range of expertise are involved in the treatment of surgical patients, which makes their involvement in guideline development paramount. We have hypothesised that the original AGREE II document may not be applicable to clinical practice guidelines in surgery, which often represent complex and multifaceted interventions.

Objective

There are a few guideline reporting documents in other fields of medicine18 19; however, a scoping literature review by our group has not identified any document to inform guideline development and reporting in the field of surgery. Our aim was to develop an extension of the AGREE II instrument that is specific for surgery through an evidence-informed and consensus-based approach.

Methods and analysis

We have formed an international multidisciplinary and interdisciplinary collaborative research working group that consists of surgeons, guideline developers, evidence synthesis experts, GRADE methodologists,20 biostatisticians and a lead member of the AGREE collaboration. This is a tripartite project named Guideline Assessment Project (GAP): Filling the GAP in Surgical Guidelines. A summary of the project is outlined in figure 1. The project is a result of a partnership between an international team of surgical research experts and two of the AGREE research team leads (IDF and MB). The AGREE research team is currently under a membership renovation process, and, therefore, neither of the authors can speak on behalf of the entire AGREE group. However, both AGREE research team leads state that AGREE has supported the project from its inception. Furthermore, they have agreed to support dissemination activities by making this new tool available in the AGREE website (https://agreetrust.org).

Figure 1

Development steps of the guideline assessment project with the ultimate objective to develop an AGREE II extension document for surgical guidelines. AGREE II, Appraisal of Guidelines for Research and Evaluation.

This protocol complies with the Guidance for Developers of Health Research Reporting Guidelines.21

Patient and public involvement

No patients were involved in the development of this protocol.

GAP I: literature review and exploratory analyses

We have previously performed a structured review to identify clinical practice guidelines in the field of surgery published over a 10-year period.22 We have assessed the methodological and reporting quality of the selected guidelines using the original AGREE II criteria. Domain scores (calculated by summing up all the scores of the individual items in a domain and by scaling the total as a percentage of the maximum possible score for that domain)16 ranged between 0% and 56%, suggesting generally inadequate and highly variable guideline quality. The median overall score was 4 out of a maximum of 7, and 40% of guidelines were not considered suitable for use based on their quality as assessed using the AGREE II instrument.

In exploratory analyses, we have found guidelines produced by surgical organisations with a high (≥1 guideline per year) output (OR 3.79, 95% CI 1.01 to 12.66), and those produced by surgical organisations with a guideline committee (OR 4.15, 95% CI 1.47 to 11.77) have higher odds of reaching sufficient quality and being recommended for use.22

GAP II: statistically calibrating the AGREE II instrument

The second part of this project was focused on statistical calibration of the AGREE II instrument. We have used quality appraisal data from GAP I and employed a series of statistical methods to explore reliability, internal consistency and unidimensionality of the AGREE II instrument when it is applied in surgical guidelines. We investigated the internal consistency that refers to the extent to which all items of the instrument measure the same hypothetical construct. We explored if and how test items are intercorrelated. Large intercorrelations among test items are indicative of the items measuring the same construct. Using reliability analysis, Kendall’s tau statistics, factor analysis and the item response theory, we explored whether items of each AGREE II domain are intercorrelated and are, therefore, indicators of the same construct. Statistical modelling showed that excluding five items from the original tool (items 1, 2, 5, 7 and 8) and rearranging the remaining items into four domains instead of six would enhance the instrument. We have finally drafted a modified AGREE II document for guidelines in surgery, on the basis of the outcomes of statistical models.

GAP III: AGREE II extension for surgical guidelines

The third part of the project aims to use the information from the previous GAP projects and other published information on the topic to develop the extension document using a structured Delphi process involving relevant stakeholders.

The multidisciplinary Delphi panel will include surgical specialists, journal editors, guideline development bodies, GRADE representatives and patient representatives. Under consideration of the evidence, stakeholders will be asked to provide their input through a Delphi process, which will inform the preparation of an AGREE II extension for surgical guidelines.

Participants

The executive group consists of surgeons (ML-C, SRM, GS, GAA, NKF and SAA), members of surgical quality and research boards (ML-C, NKF and SAA), guideline developers (ML-C, IDF, MB, GS, NKF and SAA), evidence synthesis experts (IDF, GAA, DM and SAA), GRADE methodologists (ML-C and SAA),20 biostatisticians (DM and ST) and two leads of the AGREE Group (IDF and MB). It is further divided into four working groups with distinct functions and responsibilities:

The strategic steering group is responsible for overseeing the project.

The methods group coordinates the methodology of the project.

The evidence review group will review the literature for evidence on candidate new items to be included in the extension document.

The evidence synopsis group will summarise evidence for presentation to Delphi participants.

The group attended a 1-day meeting to discuss the findings of previous work, define the methodology and study design, and identify potential stakeholder groups to comprise the Delphi panel.

Delphi process

The Delphi panel will consist of key stakeholders, including representatives from different surgical disciplines (general surgery, urology, thoracic surgery, vascular surgery and paediatric surgery), guideline developers from different continents and representatives of guideline development organisations. The public will be involved by participation of patient representatives from the European Patients Forum (box 1). We will develop a web-based survey tool to facilitate Delphi exercises. Findings of previous work (GAP I and GAP II) and further evidence that will be identified through a scoping literature review will be summarised and presented to Delphi participants. Summary information will also be available on the project website (https://gap-project.org). Online links to full documents for detailed review of the evidence will be provided.

General surgeon

Urologist

Thoracic surgeon

Vascular surgeon

Paediatric surgeon

Journal editor

National authority representative

NICE representative

Guideline developer/representative from Europe

Guideline developer/representative from North America

Guideline developer/representative from Asia

Guideline developer/representative from middle-income country

Healthcare provider representative

Representative from GRADE

Guideline implementer

Patient representative

WHO representative

European Commission representative

GIN representative

EQUATOR representative

EQUATOR, Enhancing the Quality and Transparency of health Research; GIN: Guidelines International Network; GRADE, Grading of Recommendation Assessment, Development and Evaluation; NICE, National Institute of Health and Care Excellence.

The first round will include open-ended questions to identify candidate items for inclusion in the extension document. Responses will be grouped and summarised by the methods group before the second round is commenced.

The second round will include closed-ended questions in a 5-point Likert scale to assess participants’ opinions and level of agreement on including candidate items or excluding existing items from the extension document. As per protocol, 1/2 indicates strong/moderate disagreement; 3 indicates no opinion; and 4/5 indicates moderate/strong agreement. Candidate items will have been identified through GAP I and GAP II and the scoping literature review. We will discard low-scoring items (ie, those with a median score of 1/2 on the Likert scale) and use the shortlisted items in a third Delphi round. We will repeat the process until an agreement of 80% (4/5 on the Likert scale) is reached among Delphi participants.

The Delphi panel’s contribution will be acknowledged by group authorship in subsequent publications of the extension document, the elaboration document and supporting tools.

Qualitative research synthesis

We will perform qualitative evidence synthesis to identify factors of conceptual importance to the quality of evidence in surgery. The overarching question will be

How do clinical practice guidelines in surgery differ from non-surgical guidelines? Specific thematic questions will be addressed:

Which are the concepts that make surgical guidelines different from guidelines or summary evidence in other medical fields?

Which are potential items that may be of sufficient importance to be included in an AGREE II extension for surgical guidelines?

Which are the items of the original AGREE II that might not be relevant to surgical guidelines?

How should items of the original AGREE II instrument be modified to be more relevant to an AGREE II extension for surgical guidelines?

We will conduct a scoping search of PubMed, Embase and Google Scholar. In keeping with realist review guidelines,23 there will be no restrictions on the types of study design eligible for inclusion. We will consider editorials, letters to the editor, commentaries, opinions and any type of publication that captures the breadth discussions about development of surgical guidelines. Information will be used to identify characteristics that specifically apply to surgical guidelines.

The realist review will aim to develop an explanatory understanding of development and reporting of surgical guidelines, how surgical guidelines differ from non-surgical ones and how AGREE II can be modified to reflect the specific aspects of surgical guidelines. According to the realist synthesis methodology, studies will be assessed based on criteria of relevance (whether they contribute to the development or testing of the initial theories)24 and appropriateness for addressing the research questions.25 26

Studies will be entered into ATLAS.ti and coded to identify the specific features relevant to development and reporting of surgical guidelines. Themes will be discussed by the research team using an iterative and speculative process.26 Adjudication and triangulation will be applied to refine theories which can be used across the studies to understand findings.

Furthermore, we will invite users of social media through the project account on Twitter (@GAProject2) and through communication streams of the sponsoring bodies (Facebook, Twitter and email newsletters) to nominate parameters of importance in the development and reporting of guidelines in surgery, and will group and summarise their responses. Evidence identified from the aforementioned pathways, along with information from GAP I and GAP II, will be summarised and taken into account when developing the extension document.

Consensus meeting

Following the Delphi process, the executive group will meet to discuss the findings and compose the first draft of the extension document. We will present new items that will be identified through the Delphi exercise and the qualitative synthesis, and discuss their plausibility and possible inclusion in the instrument. Similarly, items to be excluded with the respective rationale will be discussed. The group will finalise the extension document by ordering and allocating items into domains.

The executive group will hold a further meeting with the advisory group, which is composed of journal editors and representatives of surgical associations to discuss dissemination and implementation processes of the developed extension instrument.

Pilot testing and assessment of internal validity

The extension instrument will be pilot-tested by two members of the executive group. One member will apply the instrument on the surgical guidelines, which were identified by the structured search process as described in GAP I22 and on additional guidelines that will be identified by extending the search to the present date. A second member will independently follow the same process in a randomly selected sample of 15 guidelines. The biostatistical team will assess the internal validity by applying the statistical models of GAP II. Any difficulties encountered with the use of the instrument will be documented and addressed. Results of the statistical assessment will be appraised against statistical findings of the appraisal of the original AGREE II instrument (GAP II).

AGREE II extension statement

The extension statement along with an explanation and elaboration (E&E) document will be composed by the executive group. The E&E document will detail the use of the extension instrument in developing and reporting a new surgical guideline and appraising an existing surgical guideline.

AGREE II extension checklist

A checklist including the AGREE II extension items will be developed with the aim of this checklist to be used by guideline developers (to summarise development and reporting parameters), guideline users (to appraise quality), peer reviewers and journal editors (to assess adequacy of reporting parameters).27

Feedback and criticism

We will invite constructive feedback on the instrument through the dedicated website (https://gap-project.org), and we will consider comments in letters to the editor and via the social media. An ad hoc team will collect and summarise the feedback received in 3 monthly intervals for the first year after publication, and the executive group will discuss and address this information in web-based meetings.

Monitoring, update and future steps

The executive group will monitor the use of the extension document and appraise its applicability in surgical guidelines for a reasonable period of time after dissemination and will publish their findings. Following consideration of the outcomes, feedback, criticism, suggestions and new evidence in the field, we will discuss the need for an update. The development of further extension instruments for national surgical guidelines and guidelines in distinct surgical or other interventional disciplines will be considered following discussions with key stakeholders.

Implications for practice and research

Clinical practice guidelines directly impact clinical practice and healthcare delivery and, as such, development must follow rigorous methodological and reporting standards. The AGREE II instrument has been designed as a generic tool for development and appraisal of clinical practice guidelines.6 It is not intended to substitute established detailed guidance on guideline development principles, processes and procedures, such as the GRADE approach.15 It has addressed a vital need to summarise and detail essential development steps and reporting parameters for high-quality guidelines. Furthermore, as an appraisal instrument, it may be used by healthcare practitioners, policymakers and other stakeholders to inform decisions regarding the use of an existing guideline.

In addition, AGREE II has been shown to be a valuable tool for assessment of guideline quality in several clinical disciplines and evidence fields.22 28–36 Such summaries alert the scientific community to the need for improving specific aspects of clinical practice guidelines (corresponding to the instrument domains) or the overall quality of guidelines. Our previous research has highlighted the need for improvement of the quality of surgical guidelines.22 An AGREE II extension for surgical guidelines is expected to meet this need.

The outcome of this project will be the first AGREE II extension document. Reporting Tool for Practice Guidelines in Healthcare (RIGHT) is another reporting instrument for clinical practice guidelines.37 We are aware of a planned RIGHT extension for public versions of guidelines and a RIGHT extension for adapted practice guidelines.38 39 An extension document of RIGHT for surgical guidelines would be justified as well. However, in view of the evidenced gap in methodological quality of surgical guidelines,22 we considered more appropriate to elaborate on AGREE II, as it addresses guideline development, reporting and appraisal.

Strengths and limitations

This tripartite project is the first to employ statistical models to inform the validity of an extension, modification or update document on guidelines reporting. We will correlate statistical findings with conceptual considerations of the need for adjustments/extension of the AGREE II instrument. The project methods group has adopted recommendations on developing research reporting guidelines, proposed by a collaborative team who have developed a significant number of such guidelines.21 The holistic approach to developing an extension document for clinical practice guidelines in surgery is reflected in the diverse scientific, cultural and geographical backgrounds of experts in the field involved in the project. Similarly, the Delphi panel will include stakeholders and members from a variety of backgrounds, including clinicians/surgeons, methodologists, guideline developers, policymakers and the public (patient representatives).

A face-to-face meeting of Delphi participants, instead of a full web-based Delphi process, might be more efficacious in developing the extension document allowing direct exchange of opinions, information and ideas. We will encourage a full participation and exchange of information by developing a user-friendly and interaction-allowing web-based platform. Furthermore, we will incentivise participation and engagement of potential Delphi members by proposing group authorship and participation in future associated projects.

Ethics and dissemination

Research Ethics Committee and Health Research Authority approval was waived since this is a professional staff study only and no duty of care lies with the National Health Service to any of the participants. We will request electronic informed consent from Delphi participants and the responses of the Delphi members will be anonymised.

We have obtained conflict of interest forms of all executive group members and will request electronic and/or written informed consent by Delphi participants and members of the advisory group. We will deal with potential conflicts of interest by reassigning functions or replacing participants who pose interest conflict.

We will submit the final paper with the extension document to be considered for publication in the UEG Journal and Surgical Endoscopy, as defined in the respective predevelopment agreements. We will negotiate simultaneous publications in other surgical journals for widest dissemination as recommended by the Guidance for Developers of Health Research Reporting Guidelines.21

We will make the extension document available in a dedicated website with links to the original publications. We will encourage surgical organisations with guideline development activities to use the instrument. We will further pursue dissemination through the websites of the funding bodies and channels of social media of major stakeholers, such as the Guideline International Network, GRADE and EQUATOR.

Through direct contact, we will advise international surgical and guideline development organisations and policymakers to endorse the extension instrument. Furthermore, editors of surgical journals will be advised to provide an extension instrument checklist that authors of clinical practice guidelines should submit, along with the original manuscript.

The GAP III study aims to address the need for improvement of the methodology, reporting and appraisal of surgical guidelines. An extension document specifically designed for clinical practice guidelines in surgery will further improve the value, use and applicability of the AGREE II instrument in the surgical field with the ultimate goal of enhancing patient care, experience and outcomes.