Quality appraisal of clinical practice guidelines on pancreatic cancer: a PRISMA-compliant article.

Clinical practice guidelines (CPGs) play an important role in health care. The guideline development process should be precise and rigorous to ensure that the results are reproducible and not vague. To determine the quality of guidelines, the Appraisal of Guidelines and Research and Evaluation (AGREE) instrument was developed and introduced. The objective of this study is to assess the methodological quality of CPGs on pancreatic cancer. Five databases (included MEDLINE and EMBASE) and guideline websites were searched till April, 2014. The methodological quality of the guidelines was assessed by 4 authors independently using the AGREE II instrument. From 2526 citations, 21 relevant guidelines were included. The overall agreement among reviewers was moderate (intraclass correlation coefficient = 0.86, 95% confidence interval 0.64-0.96). The mean scores were moderate for the domains "scope and purpose" and "clarity of presentation"; however, they were low for the domains "stakeholder involvement" (31.22), "rigor of development", "applicability", and "editorial independence". These domain scores were lower when compared with international levels. There are 5 (23.81%) guidelines that described the systematic methods for searching. Moreover, only 5 (23.81%) guidelines reported that methodological expertise were included in the guideline developing teams. The quality and transparency of the development process and the consistency in the reporting of pancreatic cancer guidelines need to be improved. Many other methodological disadvantages were identified. In the future, pancreatic cancer CPGs should base on the best available evidence rigorously developed and reported. Greater efforts are needed to provide high-quality guidelines that serve as a useful and reliable tool for clinical decision making in this field.

INTRODUCTION

Clinical practice guidelines (CPGs) are defined as “statements that include recommendations intended to optimize patient care that is informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options.”[1] The main role of CPGs is to provide clinicians with explicit recommendations on how to manage health conditions and reduce the use of unnecessary, ineffective, or harmful interventions.[2] The quality of guideline is the primary factor that influences implementation and dissemination of guideline.

It is well known that pancreatic cancer is the fourth largest cancer killer among adults in the United States[3]; furthermore, it is one of the top 10 cancer killers in Europe and industrialized countries.[4,5] The incidence of pancreatic cancer in China has been increasing dramatically during the past several decades.[6] Because of its high mortality and incidence, the treatment of pancreatic cancer became one of the greatest oncological challenges in this century. There were >20 CPGs referred to manage pancreatic cancer and developed over the last 2 decades. However, there is no study for assessing the quality of guidelines on pancreatic cancer. The Appraisal of Guidelines for Research and Evaluation (AGREE) instrument is an appraisal tool and validated instrument of guidelines.[2,6] The aim of the present study was to systematically review the quality of CPGs related to pancreatic cancer all over the world.

MATERIALS AND METHODS

Guideline Searching

We systematically searched electronic databases and guideline websites or databases, including PubMed, Embase, the Chinese National Knowledge Infrastructure, the Chinese Biomedical Literature Database, the Chinese National Knowledge Infrastructure WanFang Database, the Guidelines International Network database (http://www.g-i-n.net/library), the National Guideline Clearinghouse (http://www.guideline.gov), the National Institute for Health and Care Excellence (http://www.nice.org.uk/), the Scottish Intercollegiate Guidelines Network (http://www.sign.ac.uk/), the Canadian Medical Association CPG Infobase (http://www.cma.ca/cpgs/), and the National Comprehensive Cancer Network (http://www.nccn.org). We searched the databases from the date of their inception to April 30, 2014, with no restriction on language. The terms “guideline, guide, guidance, consensus, recommendation, criteria, statement, pancreatic neoplasma, pancreatic tumor, pancreatic cancer, and pancreatic adenocarcinoma” were used for searching the electronic databases.

Selection of Guidelines and Data Extraction

We included the guidelines of pancreatic cancer, which met the eligibility criteria as follows: which met the definition of a guideline as proposed by the IOM,[1] which focused on pancreatic cancer, in which the language was restricted to English and Chinese. The exclusion criteria were as follows: which was an old version of the topic; which was a comprehensive guideline and only mentioned pancreatic cancer; if the guideline published in journal was the same as that indexed in guideline database or website (ie, with the same developers, topic, target population, content), we only included the guideline indexed in guideline database or website. Four authors screened the guidelines independently. Following screening, the 4 authors extracted the data of guidelines through the standard form. To reduce the chance of errors, the 4 authors extracted data separately, checked the entries for consistency, and agreed on a single set of data.

Quality Appraisal

Each guideline was independently evaluated by 4 reviewers according to the AGREE II instrument. The AGREE II instrument is an international, rigorously developed, and validated instrument.[2,6] It consists of 23 key items organized within 6 domains. Each item in a domain is scored from 1 (strongly disagree) to 7 (strongly agree). The score for each domain is obtained by summing all the scores of the individual items in a domain and then standardizing as follows: (obtained score−minimal possible score)/(maximal possible score−minimal possible score). The maximal possible score of each item is 7, which represents that the quality of reporting is exceptional and the guideline meets the full criteria and considerations articulated in the instrument. The minimal possible score of each item is 1 when there is no information about this item reported in the guideline. We initially conducted 2 rounds (a total of 10 guidelines) of pilot test before assessing all of the included guidelines.

Statistical Analysis

We calculated AGREE II domain scores as means and categorical variables with the number of cases and corresponding percentages. Intraclass correlation coefficients (ICCs) were considered to assess the interrater reliability within each domain.[7] We analyzed the data with SAS 9.2 (SAS Institute Inc, Cary, NC, USA).

RESULTS

Literature Search

The systematic literature search yielded 2526 citations. We eliminated 39 duplicates, leaving 2487 citations for title and abstract review. Based on the title and abstract, 2385 citations were subsequently excluded because they were not CPG and related to the management of pancreatic cancer. Finally, a total of 21 guidelines[8-28] were finally included for a full review (Figure 1).

FIGURE 1

Flow of information through the different phases of the literature search. CBM = Chinese Biomedical Literature, CMA = Canadian Medical Association, CNKI = Chinese National Knowledge Infrastructure, CPG = clinical practice guideline, GIN = Guidelines International Network, NCCN = National Comprehensive Cancer Network, NGC = National Guideline Clearinghouse, NICE = National Institute for Health and Care Excellence, SIGN = Scottish Intercollegiate Guidelines Network, WHO = World Health Organization.

Characteristics of Guidelines

A total of 21 CPGs published from 2001 through 2014 were evaluated using the AGREE II with 4 reviewers per guideline. Of the 21 selected CPGs, 5 were from America, 4 from China, the same number as Canada, both of 3 guidelines are from Switzerland and Britain, 2 from Spain, respectively (Table 1). Seventeen of the CPGs focused on regular pancreatic cancer, whereas the other 2 CPGs considered advanced pancreatic cancer; only 1 guideline focuses on periampullary and ampullary carcinomas, and 1 guideline focuses on metastatic pancreatic adenocarcinoma (Table 1). When we mentioned the scope of the CPGs, 4 guideline topics covered pancreatic cancer diagnosis, treatment, and follow-up; 6 guidelines covered diagnosis and treatment; 9 guidelines only pay attention to treatment; 1 to diagnosis; 1 focus on prevention, screening, and treatment. Most of the guidelines cited references (range 0–434, mean 60). The average total page numbers of the guidelines was 17 (range 3–109) (Table 1). Table 2 shows the results for each of the guideline areas after being evaluated using the AGREE II instrument. The overall agreement among reviewers for the evaluation with the AGREE II instrument was moderate (ICC = 0.83, 95% confidence interval 0.66–0.92). Below we describe the appraisal results according to the AGREE II domains.

TABLE 1

Characteristics of Clinical Practice Guidelines for Pancreatic Cancer

TABLE 2

Guideline Score According to Score on Each of the Domains Assessed by the AGREE II Instrument

Appraisal of Guidelines

Scope and Purpose

The score for this domain reflects the degree to which the overall objectives of the guidelines, clinical questions covered, and patients to whom the guidelines were meant to apply were specifically described.[2,29] The mean score for this domain was 51.32%, and 10 of the guidelines (47.62%) scored <50%.

In this part, stakeholder involvement was used to evaluate the degree to which CPGs represent the views of their intended users. It can indicate whether individuals from all associated professional fields were represented, whether the views and preferences of the target population (eg, patients, public) had been involved, and whether those target users of the guidelines were well-defined.[2,29] The mean score was 31.22%, with only 5 of the CPGs (23.81%) scoring >50%, which suggested the poor involvement of stakeholders in guideline development. None of the guidelines involved patients in the development process or was piloted among end users.

Rigor of Development

Here, rigor of development was considered the most important issue. As we can see, it could evaluate the integrity of the developing process, which included the reporting of the search methodology, criteria of evidence selection, methods used to formulate recommendations, and the assessment of risk and benefit; it also included links between evidence and recommendations, external review, and updating mechanisms.[2,29] The mean score for this domain was only 24.40%. Only 3 guidelines (14.29%) scored >50%. Over 80.95 % of the guidelines did not mention any database in their search strategy and did not include a system to evaluate the quality of the evidence or grade the strength of the recommendations. Only 8 guidelines described a procedure for updating the guidelines.

Clarity of Presentation

This is an important issue that could effectively assess the clarity of the guidelines; it could also make sure which recommendation is specific and unambiguous, whether different management options are clearly presented, and whether key recommendations are easily identifiable. Furthermore, it indicated whether these guidelines are supported by tools for their application.[2,29] Overall, the mean score for this domain is the highest (51.72%). However, there are still 9 guidelines scored <50% for this domain.

Applicability

Applicability is an essential issue to evaluate those key factors that are pertinent to guideline implementation. Meanwhile, it assesses the advantages and barriers of guidelines described in the guidelines to their application; it evaluates the possibility of considering the potential resource implications of applying the recommendations, and it also indicated the effect of guidelines in presenting the monitoring or auditing criteria.[2,29] The score on this domain was the lowest (22.32%), and all of the guidelines scored <50%. None of the guidelines discussed cost implications.

Editorial Independence

This domain addresses conflicts of interest, specifically whether the guidelines were editorially independent from the funding body and whether potential conflicts of interest were reported for the members of the guideline development group.[2,29] The mean score for this domain was 29.76%. Seventeen guidelines scored <50%. Eight (38.10%) guidelines did not report whether they received funding or not and failed to report whether or not the views of the funding body influenced the content of the guidelines.

Stratification of CPG Quality

Table 3 presents the means of the domain quality scores from focus of the guideline, year of AGREE II publication (2010), publication type (journal and guideline database), type of development group (individual and medical society), and systematic search or not. There was no difference in 6 domain quality related to year of AGREE II publication. Meanwhile, the scores from CPGs published in guideline database were higher significantly on the 6 domains when compared with CPGs published in journal. The scores from CPGs developed by individuals were lower than that by the medical societies on these 4 domains (scope and purpose, stakeholders, rigor, and applicability). When we mentioned systematic search or not, there were higher scores in 5 domains (scope and purpose, stakeholders, rigor, clarity, and editorial independence).

TABLE 3

Mean (±SD) AGREE II Scores by Subgroups

DISCUSSION

With using a standardized appraisal instrument that was endorsed by the World Health Organization (WHO), we reported the results of the first systematic evaluation of the quality of pancreatic cancer CPGs, and this method was becoming an accepted standard in guidelines development.[30] Overall, the results in Table 4 showed that the quality of the guidelines assessed was low when compared with Alonso-Coello et al[31] who reported the quality of CPGs across a wide range of health care topics published since 1980.

TABLE 4

A Comparison of Domain Scores Between These 21 Clinical Practice Guidelines and International Level (%)

Our study indicated that the quality domains with acceptable scores (>50%) are ‘scope and purpose’ and ‘clarity of presentation.’ Most of the guidelines described their specific and focused clinical questions and target populations well. To further enhance and improve, providing specific information and clear summaries are needed.

As results showed, the domain ‘rigor of development’ had a low mean score (24%). Undoubtedly, there were some reasons leading to the pessimistic result. First of all, in this study, only 4 guidelines were found to perform a well-documented systematic literature search. Second, which is also very important, few of these guidelines were externally reviewed prior to publication. There should be more external reviewing, only with multidisciplinary discussion, the guidelines could be more comprehensive and better. Thirdly, only 38.10% of the guidelines mentioned updates; as a general rule, CPGs should be reassessed for validity every 3 years.[31,32] Another reason for low scores on “rigor of development” is that the method used was poorly reported in the guidelines. Of course, this could be improved by using addenda, which included searching strategies, literature selection process, or evidence tables. Nowadays, hyperlinks to these addenda and methodology sections can make more and more researchers come to be familiar with this new tool for guideline study, and for sure, it can be helpful to improve the condition.

Similarly, the scores in the “stakeholder involvement” domain were low. These low scores reflect the lack of multidisciplinary teams. Only 5 guidelines that reported methodological expertise were included in guideline developing teams. Moreover, none involved patients in the development process or was piloted among end users.

The lowest scores on “applicability” and “editorial independence” were particularly conspicuous. These findings appear to be fairly widespread among the CPGs. The low scores may be the result of guideline development groups considering guideline development and guideline implementation as separate activities.

As we all know, the quality of ‘applicability’ domain also plays a critical role in reflecting the implementation of guidelines. For an effective guideline, it should be advisory on how the recommendations can be implemented. As a key point, professionals with the relevant expertise should be incorporated in the developing group at early stages; only with this, the guideline can be more professional and applicable for any relevant field. Alternatively, before implementing or adapting a particular guideline, users should be made to know about some important issues such as barriers, costs, indicators, or criteria for monitoring, which were closely related with the applying of guidelines.[33]

Finally, the low scores in the ‘editorial independence’ domain may be due to a lack of information about funding sources and conflicts of interests. It would be relatively easy to raise the scores by providing more information on these items. New approaches for dealing with financial and intellectual conflicts of interest are being implemented.[34-36] The developers of CPGs need to pay more attention to these domains during the development process.

According to the analyzing results, unfortunately, there were 6 guidelines that could not be given a specific recommendation. What's more, methods used in recommendation are varying greatly and did not have a standard method to divide levels of evidence and grades of recommendation, which cause the variation of different guidelines. In order to make the consistent, many researchers are focused on the development of an effective tool, which can provide a specific recommendation. Fortunately, the GRADE system was developed as a common, sensible approach to grade quality of evidence and strength of recommendation. Nowadays, there are lots of studies indicating the advantages of GRADE system compared with other systems are reported.[37] In this study, in order to improve the evaluating quality, the GRADE approach was recommended in CPGs.[38,39] In a word, to be an effective guideline, there is a point of great importance to provide advice, which is associated with how the recommendations can be implemented. Of course, the discussion of the potential impact of recommendations on resources should be an essential part of it, and it requires clearly defined criteria derived from the key recommendations.[40]

Based on the analyzing results, it is obvious to find out several strengths in our study. First of all, according to the searching data, it turns out that this study is the first systematic review of guideline quality over a wide range of topics covering the last 20 years; the structured and explicit GRADE approach increased the validity of the findings, which made the evaluation more professional and comprehensive. Second, 4 independent and experienced evaluators achieved a high degree of agreement when they assessed the articles. Thirdly, the quality control of the data extraction in this study was performed by 4 reviewers, which further enhanced the confidence in the results.

However, we realized that there were also several limitations that might bias our study. First, in this study, only those CPGs written in English and Chinese versions were included, and guidelines written entirely in other languages might have been overlooked; the good thing is that more and more researchers in different countries would like to publish their studies in English, hopefully, in the future; with the updating, our study could be more complicated. Second, there are some guidelines published in books, booklets, or government document forms, which we cannot search through the Internet, and this limitation may understate the quality of CPGs. The third is that AGREE II instrument only assessed the reporting of the different items and not the content validity of the recommendations. Lastly, another potential limitation is that the guideline developer could include some of the items listed in AGREE in process, but did not report it.

CPGs are of great importance to the treatment by clinical physicians, so the quality of CPGs would be an essential issue for clinical work. As a consequence, adaptation of high-quality existing guidelines should be a very important job to fulfill perfect clinical practice, and it can be a good option as an alternative to de novo guideline development, which may increase the efficiency of guideline development. Furthermore, the ADAPTE Collaboration has developed a generic adaptation process that aims to foster valid and high-quality adapted guidelines.[29,41] In order to develop a better and more applicable guideline, guideline developers from different specialties should join efforts together, make the topic or condition in a appropriate state, and start sharing their resources and initiatives. Of course, an effective assessing tool will be essential for creating a good guideline, and finally more and more efforts should be spent on forming networks or collaborations to avoid duplication and missing in evaluating the available evidence. Organizations such as WHO, the Guidelines International Network, and the Cochrane Collaboration should play a major role in supporting this work.