Appraising of the Clinical Practice Guidelines Quality in the Non-Pharmacological Management of Chemotherapy-Induced Febrile Neutropenia; A Review

Background: Febrile neutropenia is a common and serious chemotherapy side effect, is associated with increased mortality, morbidity, and treatment expenditures. Several CPGs (Clinical practice guidelines) have been released for managing chemotherapy-induced febrile neutropenia. The aim of this study is Appraisal of the clinical practice Guidelines quality in the management of chemotherapy-induced febrile neutropenia.
Methods: A review study with a systematic search of the present CPGs for the management of chemotherapy-induced febrile neutropenia. After screening the CPGs based on eligibility criteria, three CPGs were selected and 5 independent reviewers appraised them for methodological quality by using the AGREE II Instrument.
Results: Three CPGs were included; all of them were evidence-based guidelines. The clarity of presentation domain scored the highest and the applicability domain has the lowest score among all domains of AGREE instrument and the rest of domains scored as descending respectively; Scope and purpose, stakeholder involvement, editorial independence, rigor of development. In general, the intra-class correlation coefficient (ICC) scores of all domains were very good according to the Landis and Koch’s scale, except the Applicability domain scored as substantial. Conclusions: This study showed the quality of appraised CPGs. Three domains of these CPGs based on the AGREE instrument scored less than other domains and were in relatively unfavorable status: applicability, rigor of development, editorial independence. Given the importance of these domains in guideline implementation, it is necessary to take actions for reducing these defects. Creative Commons Attribution License

Introduction

Febrile neutropenia (FN) is a common and serious chemotherapy side effect (Cameron, 2009), which is the reduction of the number of neutrophils in almost all patients who receive chemotherapy (Crawford, 2013), and it is being defined as neutrophil counts below 500 neutrophils/mm3 or a neutrophil counts less than 1000 cells/ mm3 with a predicted decrease to below 500 cells/mm3 (Moores, 2007). This complication affect patient’s quality of life (Hashiguchi et al., 2015) and associated with increased mortality, morbidity, and treatment expenditures (Lopez-Pousa et al., 2010). One hundred of the hospital admissions in every Month, are due to FN also it is becoming a cause of death for 1/500 patients. There is evidence that mortality rates of chemotherapy-induced FN are increasing more rapidly than the numbers of new cancer cases (National Chemotherapy Advisory Group, 2009).

It is not surprisingly the auditing of clinical practice indicate there is significant variability in cancer care and there are many ways to improve performance (Dale, 2009) for this reason, we should own local consensus protocols with systematic audit, deep discussion about mortality caused by chemotherapy-induced FN. active communication with healthcare professionals in primary and secondary services for caring patients, is important as to be sure all of the febrile neutropenic patients get treatment continuously, rapidly and safely (Cameron, 2009). CPGs are statements developed with aim to help people in clinical, strategies- related and system -related decisions making (Brouwers et al., 2012, Brouwers et al., 2010d). CPGs should have optimal usability and adaptability- for- purpose, then they require to be up to date, valid and credible (Zhang et al., 2014); moreover, CPGs are important to improve care quality and decrease the variability in daily practice and expenditures (Font-Gonzalez et al., 2016) given CPGs is varied widely in quality (Shaneyfelt et al., 1999; Vigna-Taglianti et al., 2006, Abdelsattar et al., 2015), the quality of methods applied to develop guideline has been caused worry (Burgers et al., 2003; Brouwers et al., 2010f) a strategy need to determine which CPGs must be implemented (NCCMT, 2013) also CPGs should have special the quality indicators; Therefore, these indicators were defined and validated by the AGREE Collaboration (Reames et al., 2013).

The AGREE Collaboration defines the quality of guideline as follows; “the confidence that the potential biases of guideline development have been addressed adequately and that the recommendations are both internally and externally valid, and are feasible for practice”. The aim of the AGREE instrument is to present a framework to appraise guidelines quality, a methodological strategy for developing the guideline, and provide what information and how the information should be reported in CPGs (NCCMT, 2013).

Given the designing of CPGs fill the gap between clinical practice and research and the oncology CPGs are not always resulted of evidence-based research and not fair methodologically (Reames et al., 2013). The purpose of CPGs is the organization of many related raw Data to easier understanding and apply in a clinical environment (Harris, 1997); therefore, CPGs have the potential for constructing policy and even regulation. A great responsibility emerges from this great power (Dans and Dans, 2010). Given the process of guideline development should be rigorous and precision to ensure the results are retrievable and un-ambiguous, the health care professionals need to appraise CPGs critically to understand whether they well- constructed and they are representative of superior Evidence (Harris, 1997; Kremer et al., 2013; Deng et al., 2016).

An appraising of extant CPGs is necessary as the first step toward the development of an international, uniform and coordinated guideline (Kremer et al., 2013). Recently A study has been conducted by Nasr et al., (2018) has appraisedall the extant CPGs in chemotherapy-induced FN for adult patients, they haven’t appraised only non-pharmacological recommendations in their study. Then all over the worlds, there are few recommendations about non-pharmacological management of chemotherapy-induced FN; moreover, little information exit about the quality of CPGs in this area. The aim of this study is Appraisal of the quality of CPGs in the non-pharmacological management of chemotherapy-induced FN.

Materials and Methods

A review study with a systematic search of present CPGs of chemotherapy-induced FN which the AGREE II instrument was used for appraising CPGs of chemotherapy-induced FN. The AGREE II instrument comprised of 23 items categorized in 6 domains (Table 2); domain 1; scope and purpose, domain 2; stakeholder involvement, domain 3; rigor of development, domain 4; clarity of presentation, domain 5; applicability, and domain 6; editorial independence (Brouwers et al., 2010d). In addition, this is an important instrument that puts the developed guideline under the microscope of general scrutiny and causes guideline developers to take accountability for their works (Dans and Dans, 2010).

The AGREE II instrument has numerous uses which one of them is the evaluation of the quality of CPGs which are candidates to implement in clinical practice (Brouwers et al., 2010d). In addition this is an important instrument that puts the developed guideline under the microscope of general scrutiny, and causes guideline developers take accountability for their works (Dans and Dans, 2010).

Search strategy and CPG selection

The experienced researcher (SS) conducted a comprehensive search of sites and databases as mentioned; NCCN (National comprehensive cancer network), NGC (National Guideline Clearinghouse), NICE (National Institute for Clinical Excellence), SIGN (Scottish Intercollegiate Guidelines Network), CCO (Cancer Care Ontario), G-IN (http://www.g-i-n.net/), ONS (Oncology Nursing Society), NHSC (http://www.health.gov.au/), Science Direct, Cochrane Library, Scopus, ProQuest, PubMed, Google Scholar, Google, Yahoo, CINAHL (Cumulative Index to Nursing and Allied Health Literature), and mentioned sites (http://www.health.govt.nz/, http://www.cancerview.ca/, http://www.esmo.org/, www.guideline.gov, http://www.health.govt.nz/, http://www.rch.org.au/, http://www.worldgastroenterology.org/, http://www.cancercare.mb.ca/, http://bestpractice.bmj.com/) for finding eligible guidelines released from observation time to Oct 2017 Also, Iranian databases included; Irandoc, SID, Magiran were used by keywords; “quality, clinical practice guidelines, chemotherapy-induced febrile neutropenia, cancer-related infection” or with a combination of them were applied.

Eligibility criteria

Five independent reviewers appraised records based on title and abstract and using inclusion criteria containing; 1. The target population is adults (≥ 18 years old), 2. FN-related to chemotherapy, 3. EN language for CPGs, 4. As the structured guideline, 4. Introducing itself as “guideline”, 5. CPGs which provided non-pharmacological recommendations including prevention, diagnosis, and management of Neutropenia were selected. All of the included CPGs were evidence-based (the existence of evidence and recommendations) and were original reports based on systematic reviews of related research evidence. We determined whether CPGs were evidence-based CPGs through reporting search strategy in guideline and grading strength of evidence.

Exclusion criteria

Expert opinion articles, literature reviews, systematic reviews, narrative reviews, protocols, clinical pathways, physician guide booklets, patient’s guides, editorials were excluded.

Selection of CPGs

As mentioned, the research team screened CPGs based on eligibility criteria (Figure 1). Finally, the full texts of selected records were assessed by 2 appraisers to ensure their eligibility.

Figure 1

Flowchart of Selecting Included Guidelines

Appraising selected CPGs applying the AGREE II instrument

Then five independent reviewers appraised the methodological quality of screened CPGs by using the AGREE II instrument (the second version, updated 2013) (Brouwers et al., 2010b). Reviewers appraised each item and scored them from 1 (strongly disagree) to 7 (strongly agree) to measure the extent that each of criteria is provided by CPG (Brouwers et al., 2010b; Armstrong et al., 2016; Burls, 2010; Salarvand et al., 2017). Score1 pertains to the lack of information or the concept is reported very weak. Score7 demonstrates the quality of reporting is excellent and all of mentioned criteria and considerations in the users’ manual were addressed. The score between 2 to 6 demonstrates reporting of the item of the AGREE II instrument haven’t completely met criteria and considerations (Brouwers et al., 2010d).

Each domain’s final scores were calculated based on the summation of reviewers’ scores, and maximum possible score and also the difference between the maximum and minimum possible scores for each domain was calculated (Bazzano et al., 2016) The Overall assessment of CPGs (3-recommended, 2-recommended with modifications, 1- not recommended) was determined independently by each reviewer. In this section, the overall quality of the CPG and reviewers’ recommendations for applying it, is evaluated. Domains scores were reported as a percentage. The significant disagreements between the evaluators were resolved by discussion and consensus but the minor disagreements were ignored.

Given the scores of 6 domains are independent and shouldn’t be added as a single quality score. The score for each domain was calculated by this formula: Obtained score – Minimum possible score/ Maximum possible score – Minimum possible score (Brouwers et al., 2010b).

Also, each of the guidelines’ domains was scored above 60 %, we determined its quality score as favorable and median scores under 30 % (unfavorable), median scores between 30– 60 % (relatively unfavorable) were defined (Salarvand et al., 2017).

In addition, a guideline was “strongly recommended” if five to six of its domains were scored as ≥50 %; a guideline is “recommended” if three to four of its domains scored as ≥50 %; a guideline is “weakly recommended” if one to two of its domain scored as ≥50 %; and a guideline is “not recommended” if all domains scored as <50 % (Zhang et al., 2014).

Statistical analysis

Each domain of the AGREE II instrument was analyzed by descriptive statistical analysis. Inter-rater reliability within each domain was assessed by applying intra-class correlation coefficient (ICC) (CI: 95 %) (Table 3). The degree of agreement was categorized according to Landis and Koch’s scale: poor (<0.00), slight (0.00–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80) and very good or almost perfect (0.81–1.00) (Sanclemente et al., 2014). The SPSS software (version 13) was applied for statistical analyses.

Table 3

Intra-class Correlation Coefficient of the AGREE Domains for Mean Rater Scores

Domain	Intraclass correlation coefficient (95 % CI/average)	Cronbach’s alpha	F value	sig
Scope and purpose	0.860 (0.669–0.934)	0. 95	24.2	0.000
Stakeholder involvement	0.801 (0.666–0.850)	0.96	23.3	0.003
Rigor of development	0.816 (0.570–0.874)	0.96	68.8	0.001
Clarity of presentation	0.624 (0.115–0.806)	0.97	29.3	0.005
Applicability	0.888 (0.874–0.896)	1	248.5	0.000
Editorial independence	0.877 (0.859–0.891)	0. 95	137	0.000

Results

Study selection

As presented in Figure, by searching keywords in databases and related sites, 78 documents were found. After two reviewers assessed the relevancy, title, abstract and full text of searched records, remained documents were screened according to eligibility criteria. Three CPGs were eventually included. All of Three guidelines were evidence-based. None of them was a consensus-based or adapted guideline (Table 1).

Table 1

Summary and Characteristics of Cancer Therapy-induced Neutropenia Clinical Practice Guidelines

Guideline Title	Date Released	Country or region	Institute	Update	Type of guideline	Focus of guideline
Neutropenic sepsis: prevention and management of neutropenic sepsis in cancer patients	2012	UK	NICE clinical guideline 151	2012	Evidence-based guideline	the prevention, identification and management of neutropenic sepsis
Management of febrile neutropenia in adult cancer patients	2014	Canada	Cancer Control Alberta (CCA)	(1) 2008(2) 2012(3) 2014	Evidence-based guideline	the management of adult cancer patients with febrile Neutropenia.
Prevention and treatment of cancer-related infections	2017	USA	National Comprehensive Cancer Network (NCCN)	(1) 2012(2) 2013(3) 2014(4) 2015(5) 2016(6) 2017	Evidence-based guideline	Prevention and treatment of cancer-related infections

The results of CPGs appraisal by the AGREE-II instrument

Five reviewers appraised three included CPGs by the AGREE-II instrument. In scope and purpose domain, the median score for the included CPGs was 94.4% (range 60–97.8%) and none of the guidelines scored below 60%. In stakeholder involvement, the median score for the included CPGs was 76.7% (range 23.3–90%) and just one guideline scored below 60%.

In rigor of development, the median score for the included CPGs was 45% (range 33.7%– 80%) and two guidelines scored under 60%. Evaluated CPGs obtained the highest score in the clarity and presentation domain. The median score obtained by the CPGs was 94.4% (range: 78.9- 95.5%) and none scored below 60%. As evaluated, CPGs obtained the lowest score in applicability domain. The median score for the included CPGs was 31.7% (range 30.8%–56.7%) and all of the guidelines scored below 60%. In the editorial independence domain, the median score for the included CPGs was 54.45% (range 46.7%-62.2%) and two guidelines scored below 60%.

In general, all of the guidelines received the following scores in descending order: clarity of presentation (median score, 94.4%), scope and purpose (median score, 94.4%), stakeholder involvement (median score, 76.7%), editorial independence (median score, 54.45%), rigor of development (median score, 45%), and applicability (median score, 31.7%). Also, the Overall Assessment was scored by the median score, 80%.

The guidelines’ Overall quality Assessment

In addition, each appraiser scored the overall assessment of CPGs. One CPG [3] obtained higher scores in all domains than others and was strongly recommended because the majority of domain (above 4 domains) were scored above 50%. This guideline was produced by the National Comprehensive Cancer Network (NCCN). One CPG [1] is recommended because three of its domains were scored above 50%. One CPG [2] was weakly recommended due to two its domain scored as ≥50 % (see Table 2).

Table 2

The Individual AGREE II Domain Results for Each Cancer Therapy-induced Neutropenia Guideline

Guideline Title	Scope and purpose	Stakeholder involvement	Rigour of development	Clarity of presentation	Applicability	Editorial independence	Overall assessment
Neutropenic sepsis: prevention and management of neutropenic sepsis in cancer patients	94.4%	76.7%	33.7%	95.5%	56.7%	46.7%	recommended
Management of febrile neutropenia in adult cancer patients	97.8%	23.3%	45%	94.4%	30.8%	46.7%	weakly recommended
Prevention and treatment of cancer related infections	60%	90%	80%	78.9%	31.7%	62.2%	Strongly recommended
Median domain score	94.4%	76.7%	45%	94.4%	31.7%	54.45%

The inter-rater reliability assessment among five appraisers has been shown in Table 3. In general, the ICC scores of all domains were very good according to the Landis and Koch’s scale, except the applicability domain which scored as substantial. The inter-rater reliability assessment among five appraisers has been shown in Table 3. In general, the ICC scores of all domains were very good according to the Landis and Koch’s scale, except the clarity of presentation domain which scored as substantial. Finally, the appraisers evaluated all of the included CPGs as recommended without or with modifications to their current form.

Discussion

This study showed the appraisal of the quality of CPGs in the prevention and management of chemotherapy-induced FN. Scope and purpose domain (items 1–3) contains “the overall aim of the guideline,” and “the specific health questions, and the target population” (Don-Wauchope et al., 2012, Brouwers M et al., 2010 b). This study indicated the score of this domain in all guidelines was satisfactory (median score, 94.4%), and this domain got the same score as the clarity of presentation domain. As Armstrong, et al. Indicated both of the domains, scope and purpose and clarity of presentation, had the highest scores among the rest of domains (Armstrong et al., 2016) also, in other several studies, scope and purpose domain got the highest score or was reported in favorable status (Salarvand et al., 2017; Sanclemente et al., 2014; Sabharwal et al., 2013; Nasr et al., 2018).

The stakeholder involvement (items 4-6) domain covers evaluation of the extent to which the guideline represents the patients’ views and preferences (Xie et al., 2016) and appraises “to what extent the guideline was developed by the appropriate stakeholders”(Brouwers et al., 2010a; Brouwers et al., 2010e; Brouwers et al., 2010b). It includes the description of the target users and relevant professional group clearly (Xie et al., 2016). This study showed after both of scope and purpose and clarity of presentation domains, stakeholder involvement domain gained a higher score than other domains (median score, 76.7%). Also, Salarvand, et al. reported the same findings in this domain (Salarvand et al., 2017), Sabharwal et al., (2013) ranked the stakeholder involvement domain as a fourth domain that scored less than three domains, including scope and purpose and clarity of presentation and rigor of development.

The rigor of development (Items 7-14) is the core of guideline methodology and mainly focuses on evidence search strategy, grading, briefing, formulating recommendations (Xie et al., 2016). This domain warrants systematic search performance of evidence and describes methods applied to formulate evidence (Brouwers et al., 2010b). In the present study, CPGs didn’t get a satisfactory score on this domain (median score, 45%). Other studies confirm this finding in which the most of the CPGs had serious methodological defects (Reames et al., 2013; Salarvand et al., 2017; Cranney et al., 2002) and the guidelines’ methodological quality need to improve if used in clinical settings (Potting et al., 2009).

The clarity and presentation domain (items 15-17) evaluates presentation and framework (Xie et al., 2016) this domain is comprised of guideline language, structure, and framework and emphasizes the unambiguity, specificity, and clarity of the recommendations (Brouwers et al., 2010f). In the present study, this domain got the highest score all of AGREE II instrument domains (median score, 94.4%). The findings of other studies which evaluated CPGs confirm it got a high or the highest score in this domain (Salarvand et al., 2017; Sanclemente et al., 2014; Birken et al., 2015; Sabharwal et al., 2014) as appraisers could easily see that the clear recommendations have been presented in the CPG (Sabharwal et al., 2014).

Applicability domain (items 18-21) contains “the likely barriers and facilitators to implementation, strategies to improve uptake, guideline monitoring, and resource implications of applying the guideline” (Brouwers et al., 2010b). This domain has the highest effect on guideline implementation (Deng et al., 2016). In the present study, the evaluation of the applicability domain didn’t get a favorable score and scored the lowest level (median score, 31.7%). In all found studies, the applicability domain had a weak score or scored the lowest (Salarvand et al., 2017; Armstrong et al., 2016; Xie et al., 2016; Potting et al., 2009; Sabharwal et al., 2014; Burgers et al., 2004; Nasr et al., 2018). Yagasaki and Komatsu (2011) mentioned that although evidence-based CPGs are important for improving the patient care quality, their implementation and applicability are below expectations.

Editorial independence domain (items: 22-23) covers the funding issues and conflict of interests of all involved members (Xie et al., 2016). The present study showed this domain didn’t get a favorable score (median score, 54.45%). Other studies show all of the evaluated guidelines scored weak in this domain (Armstrong et al., 2016; Potting et al., 2009). Also, Xie, et al., (2016) stated editorial independence character was ignored in many guidelines. In contrast, the Nasr et al., (2018) Reported the editorial independence was the highest ranked domain across all under-surveyed guidelines.

The overall guideline assessment section pertains to the ranking of the overall quality of guideline, whether the guideline is recommended to use in clinical practice (Brouwers et al., 2010b). The overall assessment of CPGs scored at a favorable level by appraisers. But it needs to modify CPGs and improve their quality. Other studies found there is a need to enhance the methodological quality of the CPGs to use them in clinical practice (Salarvand et al., 2017; Potting et al., 2009; Birken et al., 2015). In addition, in the present study, two CPGs were recommended and another one recommended for modifications. Salarvand et al., (2017) reported only one guideline was recommended by appraisers. In the other hand in the present study, according to obtained scores in six domains of all evaluated CPGs, two CPGs were recommended or strongly recommended and one CPG was weakly recommended. Zhang et al., (2014) reported 7 CPGs were strongly recommended or recommended and one CPG was not recommended.

In conclusion, we conducted this study as a part of the adaptation process for our own non-pharmacological practice and for including high-quality CPGs. Overall, Guidelines 3 and 1 obtained a higher mean score than Guideline 2. This study showed the quality of appraised CPGs. As two CPGs were recommended or strongly recommended and one CPG was weakly recommended. Three domains of these CPGs based on the AGREE instrument scored less than other domains and were in relatively unfavorable status: applicability, rigor of development, editorial independence. Given the importance of these domains to guideline implementation, it is necessary to take actions for reducing these defects; Therefore, studies should be conducted to improve guideline development processes, and to recognize the barriers and facilitators to the guideline implementation.

Implications for practice

Enhancing the methodological quality of CPGs, for the prevention and management of chemotherapy-related FN in clinical practice, is necessitated.

Ethical considerations

This study is a part of a Ph.D. thesis, with the Vice-Chancellor in Research Affairs’ confirmation (Code: 395651).

Conflict of interest

None- declared.

Appendix1

The AGREE II Instrument is available at www.agreetrust.org.