Impact and use of reviews and 'overviews of reviews' to inform clinical practice guideline recommendations: protocol for a methods study.

INTRODUCTION: Guidelines are systematically developed recommendations to assist practitioner and patient decisions about treatments for clinical conditions. High quality and comprehensive systematic reviews and 'overviews of systematic reviews' (overviews) represent the best available evidence. Many guideline developers, such as the WHO and the Australian National Health and Medical Research Council, recommend the use of these research syntheses to underpin guideline recommendations. We aim to evaluate the impact and use of systematic reviews with and without pairwise meta-analysis or network meta-analyses (NMAs) and overviews in clinical practice guideline (CPG) recommendations. METHODS AND ANALYSIS: CPGs will be retrieved from Turning Research Into Practice and Epistemonikos (2017-2018). The retrieved citations will be sorted randomly and then screened sequentially by two independent reviewers until 50 CPGs have been identified. We will include CPGs that provide at least two explicit recommendations for the management of any clinical condition. We will assess whether reviews or overviews were cited in a recommendation as part of the development process for guidelines. Data extraction will be done independently by two authors and compared. We will assess the risk of bias by examining how each guideline developed clinical recommendations. We will calculate the number and frequency of citations of reviews with or without pairwise meta-analysis, reviews with NMAs and overviews, and whether they were systematically or non-systematically developed. Results will be described, tabulated and categorised based on review type (reviews or overviews). CPGs reporting the use of the Grading of Recommendations, Assessment, Development and Evaluation approach will be compared with those using a different system, and pharmacological versus non-pharmacological CPGs will be compared. ETHICS AND DISSEMINATION: No ethics approval is required. We will present at the Cochrane Colloquium and the Guidelines International Network conference. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This methods study will be one of only a few studies to evaluate if and how systematic reviews with or without pairwise meta-analysis, systematic reviews with network meta-analyses and ‘overviews of systematic reviews’ (overviews) are incorporated into clinical practice guidelines.

We are using a novel methodology to evaluate recommendations for clinical treatment in a random sample of clinical practice guidelines.

A limitation of our study is the narrow search dates of the test set of clinical practice guidelines.

A further limitation is that clinical practice guidelines and their updates will be excluded if they do not contain a methods section and a full bibliography, which may lead to underestimation or overestimation of the proportion of guideline recommendations using review-level evidence.

Our study is focused on clinical practice guidelines for the management or treatment of any clinical condition. Future studies looking into the use of reviews in screening or diagnostic recommendations would also be useful to determine the quality of recommendations.

Background

Clinical practice guidelines (CPGs) are recommendations developed for specific clinical conditions, targeted at clinicians and are intended to standardise and improve healthcare practice.1 CPGs aid in healthcare decision-making by formulating recommendations on clinical management strategies. Approaches to CPG development vary widely. The steps in CPG development involve defining the aims of the guideline, searching the literature, selecting, critically appraising and synthesising the results of research, and making recommendations.1–4 Many CPG developers, such as the WHO and the Australian National Health and Medical Research Council (NHMRC), recommend the use of systematic reviews and ‘overviews of systematic reviews’ to underpin guideline recommendations.5 6 The NHMRC Guidelines for Guidelines state: ‘Guidelines should ideally be informed by at least one well-conducted systematic review. In some cases, guideline developers may also consider overviews of multiple systematic reviews, or may incorporate individual studies and other sources of evidence where reviews are not available or feasible.’6

Systematic reviews and ‘overviews of systematic reviews’ reduce research waste by using the results of already published research.7 8 Systematic reviews aim to synthesise the results of primary studies of pairwise comparisons on the same topic. Depending on the similarity and variability of the included primary studies, systematic reviews may or may not include a pooled meta-analysis of effect estimates directly comparing two interventions. A systematic reviewer may also decide to conduct a network meta-analysis (NMA) if the aim of the review is to compare two or more interventions using a common comparator, the included studies are similar and the transitivity assumption is upheld.4 Systematic reviews with NMAs compare multiple interventions using both direct comparisons of interventions within clinical trials and indirect comparisons across trials based on a common comparator.9 Overviews of systematic reviews (overviews; also termed umbrella reviews, meta-reviews, or systematic reviews of reviews) aim to primarily search for, retrieve and synthesise the results of multiple systematic reviews.10–12 For topic areas with a large literature base and broad scope, overviews serve as an efficient way to synthesise review-level evidence.13 Well-conducted and reported systematic reviews with or without pairwise meta-analysis, systematic reviews with NMAs and ‘overviews of systematic reviews’ (overviews) represent the best available evidence to inform CPGs.4 14

Guidelines should clearly state the methods used to create the recommendations, use a standard grading system to assess the strength/certainty of the evidence, report potential biases and limitations of the process, and provide frequent updates.15–18 CPGs can use a non-systematic or systematic process to gather, assess and synthesise evidence to inform recommendations. Developers of guidelines can conduct a literature review (using non-systematic methods), a systematic review (using systematic methods with inclusion of all study types (primary studies, systematic reviews, overviews)) or an overview of systematic reviews (using systematic methods with inclusion and synthesis of systematic reviews) (figure 1). Guideline developers can retrieve a combination of evidence for synthesis in recommendations such as: only primary studies, primary studies and systematic reviews, only systematic reviews or systematic reviews in combination with other CPGs or health technology assessments (HTAs).

Figure 1

Legend: Process used to gather, assess and synthesise evidence to inform recommendations (ie, systematic, non-systematic). Clinical practice guidelines can use a non-systematic or systematic process to gather, assess and synthesise evidence to inform the recommendations. Developers of guidelines can conduct a literature review (using non-systematic methods), a systematic review (using systematic methods with inclusion of all study types (primary studies, systematic reviews, overviews)) or an overview of systematic reviews (using systematic methods with inclusion and synthesis of systematic reviews). Using these methods, guideline developers can retrieve only primary studies, primary studies and systematic reviews, only systematic reviews, and/or systematic reviews and clinical practice guidelines/health technology assessment (HTA) reports/overviews of systematic reviews. CPG, clinical practice guideline.

Impact is defined by the National Institute for Health and Care Excellence19 as research that results in a change in understanding arising through dissemination activities or which results in a clear recommendation. Use and citation of reviews to inform CPG recommendations has been studied by various groups.18 20 21 Silagy et al 21 examined the proportion of guideline recommendations on smoking cessation citing and using Cochrane reviews, and concluded that systematic reviews supported the recommendations in 68% of UK, 89% of New Zealand, 98% of US and 100% of Canadian guidelines.21 Bunn and colleagues20 found that there were 722 citations of Cochrane reviews in 248 guidelines.20

The quality and certainty/strength of the evidence in recommendations in CPGs have been evaluated as well. Fanaroff et al 15 found that only 8.5% of recommendations from the American College of Cardiology/American Heart Association guidelines, and 14.3% of recommendations from the European Society of Cardiology guidelines were supported by evidence from multiple clinical trials.17 Additionally, Schumacher et al 17 found that only 8.6% of the recommendations from the American Thoracic Society CPGs were derived from high-quality evidence (ie, a randomised controlled trial or a systematic review with meta-analysis).18

Recommendations in CPGs should be developed systematically using the identified review types. As outlined in the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach for guideline development,22 the body of evidence underpinning a recommendation would be considered conclusive if it has been judged to be of high certainty (ie, of high quality, precise, homogeneous and consistent). Recommendations without review-level evidence may indicate gaps in the evidence base (ie, a lack of adequately-designed relevant studies) or problems with the CPG methodology; namely problems with the search strategy (eg, missing relevant systematic reviews), or eligibility criteria (eg, inclusion of only primary studies). Assessing the evidence underpinning recommendations in CPGs enables knowledge users to determine the trustworthiness of the recommendations. We therefore aim to evaluate if and how systematic reviews and overviews of systematic reviews are incorporated into CPG recommendations.

Methods

We have registered this protocol in the Open Science Framework (OSF) (https://osf.io/rju4f/). The design is a methods study in the knowledge synthesis field, and the study follows systematic review methods guidance for searching, study selection, data extraction and critical appraisal. As this is a methods study, no relevant research reporting checklists exist. Formal ethical approval is not required as primary data will not be collected. The study started in May 2018, and study screening and selection is completed as of May 2019.

Search

CPGs will be retrieved from the Turning Research Into Practice (TRIP) and Epistemonikos databases over a 2-year period (1 January 2017 to 31 December 2018) to limit the number of CPGs screened. In Epistemonikos, we will select the filter for guidelines (called ‘Broad syntheses’) to retrieve CPGs (online supplementary file 1). Epistemonikos includes citations retrieved from the following databases: Cochrane Database of Systematic Reviews, PubMed, Embase, The Cumulative Index to Nursing and Allied Health Literature, PsycINFO, Literatura Latinoamericana y del Caribe en Ciencias de la Salud, Database of Abstracts of Reviews of Effects, the Campbell Collaboration’s online library, the Joanna Briggs Institute (JBI) Database of Systematic Reviews and Implementation Reports, and the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) Evidence Library. As the TRIP database only contains CPGs, we will download all records without restricting study type. TRIP retrieves guidelines from over 289 journal publications and has recently migrated all content from the Agency for Healthcare Research and Quality (AHRQ)’s Clinical Guidelines Clearinghouse (www.guidelines.gov) which was shut down on 16 July 2018 (Jon Brassey, personal communication, 2018).

The references from these sources will be imported into a single EndNote file, de-duplicated and screened at the full text level independently by two authors to identify citations meeting our inclusion criteria. All authors involved in study selection will screen ten studies as a calibration exercise to establish agreement in definitions of eligibility criteria.

Random selection

The retrieved citations will be randomly sorted using Microsoft Excel’s RAND function and screened using a form designed in Microsoft Excel (2013). Screening will start with the lowest random number and continue until 50 guidelines are included. This sample size was chosen to be large enough to include a variety of clinical conditions. Discrepant decisions will be resolved by discussion with a senior author.

Eligibility criteria

Guidelines are defined as systematically developed statements to assist in clinical decision-making about treatment recommendations for clinical conditions.23 24

Pertain to the management or treatment of any clinical condition. CPG recommendations for management may include, for example, recommendations for lifestyle modifications, when to implement or adjust therapy, choice of therapy including treatment combinations and ways to prevent harms associated with therapy.

Produced by a group or organisation (ie, not authored by one person).

Contain at least two explicit recommendations for treatment or management of a condition.

Published between 1 January 2017 and 31 December 2018.

Contain a description of their methodology within the guideline or in supporting documents (eg, inclusion/exclusion criteria, key terms used to search, number of databases searched, number of authors used to select studies, methods used to create recommendations or quality/risk of bias assessment).

Contain a reference list (ie, a bibliography).

Inclusion criteria

If more than one publication from the same organisation or author group is identified, we will include the most recent version of the CPG.

Exclusion criteria

CPGs without recommendations or that focus solely on screening or diagnosis will be excluded. CPGs will also be excluded for the following reasons:

The full text is unavailable.

It is designed for local use (eg, in a single health facility or single regional health service).

It is designed for use with only hospitalised patients or patients in long-term care facilities.

It aims to provide recommendations for patterns of use of medications (eg, guidance about adherence to medications) but not treatment choice.

The eligibility criteria will be piloted by all data extractors (CL, DS, BM, CR, TL, SG) independently on a sample of 10 guidelines retrieved from the search to ensure consistent application. Once the guidelines are screened and included, we will attempt to retrieve any supplementary files, methods documents, published systematic reviews or any other documentation supplementary to the guideline.

Definitions

Systematic review. A review is considered systematic7 8 if it reports:

Question(s) formatted using participants, interventions, comparisons, outcomes and study design (PICOS).

Eligibility criteria for all study types.

Full search strategy for at least one database (ie, keywords reported and a full search strategy reported in an appendix).

Search in the main body of the manuscript (ie, not only in the abstract) using two or more electronic databases.

Process for selecting/screening studies (eg, number of authors; independent process).

An overview of systematic reviews aims to primarily identify, include and synthesise the results of secondary analyses (systematic reviews, guidelines or health technology assessments).10–12

A review with pairwise meta-analysis is a traditional meta-analysis in which the effect estimates of two interventions are compared directly, following a judgement that the included studies are sufficiently similar to warrant pooling.

A review with NMA compares multiple interventions using both direct comparisons of interventions within randomised trials and indirect comparisons across trials based on a common comparator.9

Overviews and reviews with pairwise or NMAs may or may not have used systematic methods.

Data extraction

Data from fifty guidelines will be extracted for evaluation. Each included practice guideline will be examined first to determine whether reviews or overviews of reviews were used and cited in support of one or more of the guideline's recommendations (yes or no for each review type). If yes, we will evaluate all treatments or management recommendations that cite each review type.

We will note whether the review types cited were Cochrane publications. We will also assess whether reviews were cited in sections of the guidelines other than in the recommendation sections.

A data extraction form will be developed in Microsoft Excel (2013). Ten CPGs will be independently extracted by two authors and then discussed to come to consensus about definitions and to calibrate the coding (online supplementary file 2). Full data extraction will be done independently by two authors and compared. Any discrepancies will be discussed, and conflicts will be arbitrated by a senior author.

Data extracted at the guideline level will include: name of the guideline, year of publication, country, the organisations or commissioning agency (publisher), type of publisher (government, medical society, university, other (specify)), aim of the guideline, publishing journal (if applicable), open source/paywall, the date of the last search for evidence to be included in the guideline, funding, declaration of conflicts of interest by developers, stakeholder affiliation with/honoraria from pharmaceutical companies, target population (general population or specific subpopulations such as those identified by age (eg, children and adolescents, adults of any age, older adults), sex/gender or co-morbidities) and scope (pharmacological or non-pharmacological treatment (eg, surgical, medical device)), levels of evidence (type), strength/certainty of evidence (type) and scoring system method (with reference). If the GRADE approach was used to assess the strength/certainty of the evidence of the recommendations within a guideline, we will evaluate how this was done, and if it was done according to the GRADE working group guidelines.25 We will also extract eligibility criteria for included study designs, and whether the review conducted to develop recommendations was published or not.

Outcomes that will be extracted from the guidelines

The primary outcomes of the study calculated as number and proportions are as follows:

Recommendations that use systematic reviews without meta-analysis.

Recommendations that use systematic reviews with pairwise meta-analysis.

Recommendations that use systematic reviews with NMAs.

Recommendations that use overviews of systematic reviews.

The secondary outcomes of the study calculated as number and proportions are as follows:

Reviews that are Cochrane publications.

Guidelines that use GRADE for evaluating certainty/strength of the evidence.

Guidelines that use other assessments for evaluating certainty/strength of the evidence (and type of tool used).

Guidelines using a levels of evidence system and type of system used.

Currency of the guideline (calculated by the time from last search to full publication).

Guidelines reporting any conflicts of interest disclosures by authors.

If a review or overview of review is cited within a recommendation, we will also look for evidence that critical appraisal was conducted, and record which tool was used (eg, Assessing the Methodological Quality of Systematic Reviews [AMSTAR],26 Risk of Bias Assessment Tool for Systematic Reviews [ROBIS]27).

Gaps in evidence supporting a recommendation

If a guideline does not cite a Cochrane publication, we will search the Cochrane Database of Systematic Reviews using the keywords used in the main search strategy of the guideline. We will note whether a Cochrane systematic review or an overview of systematic review could have been identified and used to inform the recommendations by checking the search dates of the CPG.

Cochrane reviews are known for using robust methodology,28–30 and by searching for missed Cochrane evidence, we can evaluate whether a guideline might be missing high-quality evidence. However, Cochrane reviews are prone to biases like any other non-Cochrane review, and should not be considered high quality without an assessment of the risks of bias.

Risk of bias assessment of the review process for informing the guideline recommendations

We will assess risk of bias of the guideline recommendations using the following criteria:

Explicit statement of the questions or objectives reported in terms of PICOS elements.

Eligibility criteria for all study designs reported.

Systematic search strategy reported to retrieve studies (ie, keywords or full search strategy reported in an appendix).

Systematic search conducted (ie, two or more databases searched).

Process reported for selecting/screening studies (eg, number of authors, independent process)

Quality/risk of bias of the review or overview supporting/refuting the recommendation assessed.

Primary studies assessed for risk of bias (quality)

We have adapted these quality items from the ROBIS tool which comprehensively assesses the risk of bias of a systematic review.27 The tool includes items relating to internal validity and classifies them in the following domains: study eligibility criteria, identification and selection of studies, data collection and study appraisal and synthesis and findings. The seven items we are using to assess the recommendations are not comprehensive but are meant to give an indication of whether basic quality guidelines to reduce bias have been followed.

The items will be presented in tables and in graphs. Guidelines reporting all seven items will be deemed as high quality.

Open access

We aim to produce a replicable study by publishing the study protocol, making the data tables publicly accessible and publishing the final manuscript in an open access journal. All data management and study processes will be conducted and recorded in the Open Science Framework.

Data analysis

We will calculate the number and frequencies of citations of systematic reviews and overviews of systematic reviews and their characteristics, found in recommendations from the 50 included guidelines. Results will also be described, tabulated and categorised based on review type (systematic review with and without pairwise and NMA and overviews of systematic reviews).

We plan to calculate the number and proportion of the total number of recommendations supported by the various types of systematic reviews. We will note any differences in frequency of use between the review types, the process of the development of CPGs and, in particular, recommendations within the guideline, the prevalence of quality assessment of the review types, use of up-to-date evidence and methodological issues in CPG development. Additional information will be put into appendices.

To estimate the time that it takes to conduct each guideline, we will calculate the difference between the initial literature search date and publication date using the month and day function in Excel 2013.

If sufficient studies are collected to make meaningful comparisons (≥10), we will compare whether guidelines reporting the use of the GRADE approach differed to those that do not, based on our outcomes, and whether guidelines with different broad category conditions and scope (pharmacological vs non-pharmacological) differ in methodology.

Discussion

Systematic reviews with and without pairwise meta-analysis or NMA in addition to overviews of reviews are important study designs to inform the practice of evidence-based medicine. The use of evidence in the form of systematic reviews is now considered to be an international standard for guideline development,5 6 and other review types, such as ‘overviews of systematic reviews’ and systematic reviews with NMAs often inform the development of clinical guidelines; however, the extent of this practice is unknown. This study aims to identify the frequency of citation of review types and assess the quality of guideline recommendations.

Strengths and limitations

The strengths of our methods include the adoption of systematic and transparent methods, specific and explicit eligibility criteria, broad search strategies using multiple sources, randomised selection of studies, and duplicate and independent processes for study selection and data extraction. A main limitation of our study is the narrow search dates of the test set of CPGs. The date range of guideline publication was chosen to retrieve a contemporary and manageable number of guidelines, as expanding the time interval will retrieve thousands of CPGs.

In addition, when coding guidelines using the data extraction items, substantial judgement will be required. To mitigate the subjectivity of classifying and coding characteristics and methods used in reporting CPG recommendations, all authors will pilot the data extraction form on 10 studies. The piloting results will be discussed to refine the wording of the items, come to consensus about definitions and calibrate the coding. Full data extraction will be done independently by two authors, compared and any discrepancies will be discussed, and conflicts will be arbitrated by a senior author.

A further limitation is that CPGs and their updates will be excluded if they did not contain a methods section and a full bibliography which may lead to underestimation or overestimation of the proportion of guideline recommendations using review-level evidence. Our study is focused on CPGs for the management or treatment of any clinical condition. Future studies looking into the use of reviews in screening or diagnostic recommendations would also be useful to determine the quality of recommendations.

Ethics and dissemination

No ethics approval is required as no human subjects are involved. The findings of this study will be disseminated and presented at the annual Cochrane Colloquium and the Guidelines International Network (GIN) conference. The Cochrane Colloquium is an international gathering to promote methods in the production of high-quality, relevant, accessible systematic reviews and other synthesised research.31 The GIN conference is an international symposium for those who work with guidelines from development and methodology through to implementation and evaluation.32 The results will also be circulated through social media (Twitter, Facebook, ResearchGate), author-affiliated websites and university workshops.