Appraising the quality standard underpinning international clinical practice guidelines for the selection and care of vascular access devices: a systematic review of reviews.

OBJECTIVE: Catheter-related bloodstream infections are one of the most important adverse events for patients. Evidence-based practice embraces interventions to prevent and reduce catheter-related bloodstream infections in patients. At present, a growing number of guidelines exist worldwide. The purpose of the study was to assess clinical practice guidelines for peripheral and central venous access device care and prevention of related complications.
DESIGN: Systematic review of clinical practice guidelines: We conducted a search of the literature published from 2005 to 2018 using Medline/PubMed, Embase, CINAHL, Ovid, ScienceDirect, Scopus and Web of Science. We also evaluated grey literature sources and websites of organisations that compiled or produced guidelines. Guideline quality was assessed with the Appraisal of Guidelines for Research and Evaluation, Second Edition tool by three independent reviewers. Cohen's kappa coefficient was used to evaluate the concordance between reviewers.
RESULTS: We included seven guidelines in the evaluation. The concordance between observers was substantial, K=0.6364 (95% CI 0.0247 to 1.2259). We identified seven international guidelines, which scored poorly on crucial domains such as applicability (medium 39%), stakeholder involvement (medium 65%) and methodological rigour (medium 67%). Guidelines by Spanish Health Ministry and UK National Institute for Health and Care Excellence presented the highest quality.
CONCLUSIONS: It is crucial to critically evaluate the validity and reliability of clinical practice guidelines so the best, most context-specific document is selected. Such choice is a necessary prior step to encourage and support health organisations to transfer research results to clinical practice. The gaps identified in our study may explain the suboptimal clinical impact of guidelines. Such low adoption may be mitigated with the use of implementation guides accompanying clinical documents. © Author(s) (or their employer(s)) 2018. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Methodological quality evaluation of vascular access devices clinical guidelines is lacking.

Despite recommendations being based on subjective assessments, evaluators’ concordance using the Appraisal of Guidelines for Research and Evaluation, Second Edition tool was substantial.

We identified low scores on crucial guideline domains such as applicability, rigour and stakeholder involvement.

The gaps identified may explain the suboptimal impact of guidelines in practice and the need for implementation guides accompanying the clinical documents.

Introduction

Vascular access devices (VADs) are the most commonly used invasive devices in hospitals worldwide.1 These devices expose patients to multiple complications related to their insertion, maintenance and management.1 2 Catheter-related bloodstream infections (CRBSIs) are one of the adverse events with worst impact for patients3 in terms of morbidity and mortality.4 CRBSIs account for 40.3% of all bloodstream infections, of which are associated with the use of central venous catheters (CVCs) and peripheral venous catheters (PVCs).5 The incidence of central line-associated bloodstream infections is 5.7 per 1000 catheter-days,6 and 0.1%, 0.5 per 1000 catheter-days in PVCs.7 CRBSIs can prolong length of hospital stay2 5 and present an attributable mortality rate of up to 25%.8 9 The approximate average cost per episode is US$45 000 and resulting in US$2.3 billion of unnecessary expenditure per year in the USA.3

Nowadays, health organisations benefit from the advances in knowledge about service delivery generated by researchers resulting in optimal care for patients and citizens. The aim of such evidence-based practice is to offer care experiences that are informed by the best available scientific knowledge, clinical expertise and user preferences.10 11 To assist the decision-making of health professionals and patients about appropriate healthcare interventions in specific clinical circumstances,12 clinical practice guidelines (CPGs) are developed. These CPGs are based on empirical evidence, explicit in their methods of development, critically reviewed by experts, free of conflicts of interest and with specific and unambiguous recommendations.

CPGs are not however exempt from problems.13 In recent years, the number of these CPGs has grown dramatically and thus the volume of evidence proven to be unmanageable and of variable quality.14 Another concern may refer to the inclusion of studies with statistical yet marginal or insignificant clinical significance.15 Additionally, there is frequent tardiness in the implementation of the recommendations within the CPGs, probably fuelled by perceptions of clinical judgement as the main element in clinical decision-making.16 These facts can ultimately weaken the credibility of CPGs and therefore increase the difficulty of their implementation.

To date, few systematic attempts have been made to compare the quality of CPGs that provide recommendations for the care and prevention of adverse events associated with vascular catheter in adults.17 Therefore, the purpose of this study was to assess the quality of such CPGs using the Appraisal of Guidelines Research and Evaluation, Second Edition (AGREE II) tool18 and analyse methodological factors related to the process of CPGs development on effective knowledge mobilisation.

Methods

Search strategy

Two researchers (IB-M, MAR-C) conducted the search from March to April 2018. Searches were constructed using relevant medical subject headings (MeSH): ‘Complications,’ ‘Catheter-Related Infections,’ ‘Catheter obstruction,’ ‘Phlebitis,’ ‘Extravasation of Diagnostic and Therapeutic Materials,’ ‘Peripheral Catheterization,’ ‘Central Venous Catheterization,’ ‘Nursing care,’ ‘Guideline’ and ‘Evidence-Based Practice’ with variations of keywords terms ‘Adverse events’ and ‘Implementation.’ The following databases and bibliographic resources were searched: Medline/PubMed, Embase, CINAHL, Ovid, ScienceDirect, Scopus and Web of Science. We also explored the websites of organisations that compile or produce guidelines such as Joanna Briggs Institute (JBI, Australia), National Health and Medical Research Council (NHMRC, Australia), Registered Nurses Association of Ontario (RNAO, Canada), The National Institute for Health and Care Excellence (NICE, United Kingdom), National Guideline Clearinghouse (NGC, US), US Centers for Disease Control and Prevention (CDC, US), Scottish Intercollegiate Guidelines Network (SIGN, Scotland) and Library Clinical Practice Guidelines of the National Health Services (GuiaSalud, Spain). In addition, we evaluated grey literature available in professional societies and groups such as the Infusion Nurses Society. The search considered evidence from January 2005 to April 2018 to avoid the inclusion of CPGs with outdated recommendations online supplementary appendix 1.

Eligibility criteria

We used the definition of CPG developed by the Institute of Medicine.12 We selected CPG with recommendations developed systematically and including a grading system related to the quality of evidence or the strength of the recommendations to assist healthcare professionals and patients in making decisions about appropriate healthcare, at the prevention of complications, management and care on CRBSIs associated with VAD in hospital adults. Expert consensus statements and specifics CPGs of other pathologies, such as cancer, radiology or renal were excluded, as its target population was very specific.

Data management and extraction

IB-M and MAR-C independently conducted the selection of studies, with disagreements resolved after discussion with a third researcher (JDP-G). The titles and abstracts obtained were then scrutinised for selection of CPGs. Only full-text studies in English or Spanish were included in the review.

IB-M and MAR-C independently extracted the following information: lead author, developing organisation, country, year of publication, recommendations, quality of evidence and strength of recommendations. IB-M developed a data extraction form based on the domains specified in the AGREE II instrument for the evaluation of CPGs online supplementary appendix 2.18 The extracted data were synthesised and similarities and differences compared.

We reported our findings in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-analyses guidelines (PRISMA).19 The documents to be evaluated were read randomly by the reviewers, to avoid biasing the scores due to familiarity with the previously evaluated CPGs.20

Data analysis

Three reviewers independently assessed the CPGs using the AGREE II instrument.18 This tool use methodological quality criteria to examine the variability of quality, rigour and transparency used during the development of CPGs. IB-M calculated the final quality score according to the protocol of the instrument and the scores provided by the three reviewers for the items within each domain. In addition, a median score and rank was obtained for each domain of the instrument. Cohen’s kappa coefficient was used to evaluate the concordance between reviewers on the item ‘global guide evaluation’. The following labels will be assigned to the corresponding ranges of kappa: Poor (<0.00), Slight (0.00–0.20), Fair (0.21–0.40), Moderate (0.41–0.60), Substantial (0.61–0.80) and almost Perfect (0.81–1.00).21 The Epidat V.4.1 software was used for statistical analysis.22

Patient and public involvement

No patients or members of the public were involved in the development of the research question, nor the design or the conduct of the study, nor the interpretation and writing up of results.

Results

A total of 1447 citations and abstracts were identified through the database search, with eight further CPGs obtained from guideline development organisations. Duplicates were deleted and excluded. The reviewers also discarded six summaries of CPG; a standard of practice from the Infusion Nurses Society; a Best Practice Sheet of Joanna Briggs Institute; two CPGs for the maintenance of intravascular catheters in children; four oncology CPGs; one Anaesthesiology CPG; two infusion CPGs; one radiology CPG; two nutrition CPGs and two renal CPGs; and one older version online supplementary appendix 3. The selection process is summarised in the PRISMA flow diagram in figure 1.

Figure 1

The Preferred Reporting for Systematic Reviews and Meta-analyses diagram representing the systematic literature search. AGREE II, Appraisal of Guidelines for Research and Evaluation, Second Edition.

Seven CPGs were finally included in the synthesis of evidence and evaluated with the AGREE II instrument. The characteristics of the selected CPGs are presented in table 1. The CPGs were developed by National Institute for Health and Care Excellence,23 Registered Nurses Association of Ontario,24 25 Ministry of Health26 (Spain), Centres for Disease Control and Prevention/Healthcare Infection Control Practices Advisory Committee,27 The Society for Healthcare Epidemiology of America/Infectious Diseases Society of America28 and Department of Health of Queensland Government29 (Australia).

Table 1

Characteristics of clinical practice guidelines (CPG)

	CPG	Organisation	Country	Year
1	epic3: National Evidence-Based Guidelines for Preventing Healthcare-Associated Infections in National Health Service Hospitals in England.	National Institute for Health and Care Excellence.	UK	2014
2	CPG on intravenous therapy with non-indwelling devices in adults.	CPG work group on intravenous therapy with non-indwelling devices in adults. Ministry of Health.	Spain	2014
3	Strategies to Prevent Central Line–Associated Bloodstream Infections in Acute Care Hospitals: 2014 Update.	Society for Healthcare Epidemiology of America/Infectious Diseases Society of America.	USA	2014
4	Guideline Peripherally Inserted Central Venous Catheter.	Centre for Healthcare Related Infection Surveillance and Prevention and Tuberculosis Control. Queensland Government.	Australia	2013
5	Guidelines for the Prevention of Intravascular Catheter-related Infections.	Society for Healthcare Epidemiology of America/Infectious Diseases Society of America.	USA	2011
6	Care and Maintenance to reduce Vascular Access Complications.	Registered Nurses’ Association of Ontario (RNAO).	Canada	2008
7	Assessment and Device Selection for Vascular Access.	RNAO.	Canada	2008

The CPGs were published between 2008 and 2014: three from 2014,23 26 28 one from 2013,29 one from 201127 and two from 2008.24 25 There was heterogeneity in the objectives of the different documents. For example, the epic323 aimed to describe clinically effective measures to be used by health professionals in hospital infection prevention; however, in three24 25 27 the specified objective was the provision of recommendations to professionals to help in decision-making about peripheral and central venous access. Recommendations for CVC alone were detailed in one paper,28 with another focusing only on peripherally inserted central catheters.29

The scores for each domain evaluated with the AGREE II tool are presented in table 2. Domain 1 (‘Scope and purpose’) relates to the general scope and purpose of the CPGs, specific clinical questions to be addressed and target population. The median score was 74% (range 37%–100%), with five documents scoring above 70%.

Table 2

Results obtained from clinical practice guideline (CPG) using the Appraisal of Guidelines for Research and Evaluation, Second Edition instrument

CPG (organisation)	Scope (%)	Stakeholder involvement (%)	Rigour (%)	Clarity (%)	Applicability (%)	Independence (%)	Final evaluation (%)	Recommended
1. National Institute for Health and Care Excellence	93	91	87	84	39	94	86	Yes
2. Ministry of Health, Spain	100	89	87	91	81	86	90	Yes
3. Centers for Disease Control and Prevention/Healthcare Infection Control Practices Advisory Committee	74	44	22	82	32	69	48	No
4. Queensland Government	37	25	9	81	14	19	38	No
5. Society for Healthcare Epidemiology of America/Infectious Diseases Society of America	70	49	28	84	39	75	67	Yes, with modification
6. Registered Nurses Association of Ontario (RNAO)	60	65	67	91	88	83	57	Yes, with modification
7. RNAO	74	72	69	91	85	78	62	Yes, with modification
Median (range)	74 (37–100)	65 (25–91)	67 (9–87)	84 (81–91)	39 (14–88)	78 (19–88)	62 (38–90)

The domain 2 (‘Stakeholder involvement’) reflected whether the guideline development process included input and involvement from appropriate stakeholders. The median score obtained was 65% (range 25%–91%), with three guidelines scoring above 70%. Additionally, the item 5 (‘The views and preferences of the target population have been sought’) in the AGREE II instrument received the lowest score.23–26

In terms of ‘Rigour of development’ (domain 3), the median score across the category was 67% (range 9%–87%), with two guidelines scoring above 70%. This section focused on the methods in place for gathering and synthesising evidence during the development or updating of the document. Most guidelines described the process of development, but they did so in varying levels of detail. We found poor description of the process followed to search and review evidence, and only one document described the approach to follow regarding updates of the information.

The domain 4 (‘Clarity of presentation’) centred on how specific and unambiguous the recommendations were, including clearly presented options for the management of the health condition and with easily identifiable key recommendations. This domain was the best evaluated, obtaining a median score of 84% (range 81%–91%), and with all CPGs exceeding the 70% score threshold.

‘Applicability criteria’ (domain 5) referred to facilitators and barriers to the implementation of recommendations within the guidelines, the strategies for implementation of such recommendations and the likely resource implications associated. The median score obtained by the CPGs on this domain was 39% (range 14%–88%) with three above 70%. The CPGs were rated negatively as they did not take into account possible enabling factors for implementation, strategies to improve adoption, resource considerations or availability of monitoring indicators.

Finally, the domain 6 (‘Editorial independence’) aimed to ensure a lack of bias in the development of the guideline. The domain was relatively well scored, with a median across CPGs of 78% (range 19%–88%) and five above 70%. Most CPGs did provide statements regarding funding, potential influence on the recommendations endorsed or relevant conflicts of interest.

Overall assessment

The average score across the CPGs was moderate (median 62%), scores ranged from 38%29 to 90%.26 The best results following the global evaluation of the CPGs and their recommendations for practice of care were obtained by CPGs of the Ministry of Health in Spain,26 and epic3,23 being strongly recommended. In addition, three CPGs24 25 28 have been recommended, which would needed modifications on the domains of rigour and applicability28; process of actualisation24 25 and two CPGs27 29 have not been recommended for use of clinical practice (table 2).

Agreement among reviewers

The agreement among the three independent reviewers is presented in table 3. The overall agreement was substantial (K=0.6364; 95% CI 0.0247 to 1.2259), with perfect intracategory concordance (K=1.0) for guidelines classified as ‘recommendable’. The evaluators demonstrated a moderate agreement (K=0.5556) for those guides classified as ‘recommendable with modifications’ coinciding with a moderate quality of evaluation. However, the agreement was small (K=0.2) when the quality of the documents was low and their use in clinical practice was not recommended.

Table 3

Kappa concordance index among observers for recommendation of CPGs

Concordance index between three observers with three categories
Category	Kappa	95% CI		Statistic Z	P values
Category 1 (recommended)	1	1	1	4.2426	0.0000
Category 2 (recommended, with modifications)	0.5556	−0.2392	1.3199	2.3570	0.0184
Category 3 (not recommended)	0.2	−0.6512	0.9906	0.8485	0.3961
Global kappa	0.6364	0.0247	1.2259	3.6282	0.0003

Discussion

Our study focused on international CPGs aimed at facilitating the management and care of VAD and preventing complications associated on CRBSIs with their use in adults. We compared the quality of existing guides using the AGREE II tool and analysed relevant methodological factors related to the process of guideline development. Many of the CPGs included in this study scored highly in some, but not all, domains. Overall, the quality of the CPGs examined was moderate. More than 70% of these presented specific and well-described recommendations including different options for clinical management, specific objectives, health aspects to be addressed and the target population. The best scores were received by the domains ‘Clarity of presentation’ and ‘Scope and objective’.

However, we identified low scores on other equally crucial domains for effective implementation including ‘stakeholder involvement’, ‘methodological rigour’ and applicability’. These findings are consistent with other studies of similar methodology.20 30 Regarding stakeholder involvement, the guidelines did not report on whether the views of patients or users were sought during the process of guideline development. Such absence may be due to a number of factors including lack of recognition and value of the perceptions of the main actors targeted by the guides.13 CPGs tend to centralise knowledge from experts without the consideration of patient or user involvement in guideline development, assuming that the patients’ care may have a less active role.31–33

In terms of methodological rigour, the evidence suggests that the quality and accuracy of CPG elaboration can be highly variable,34 even when dealing with the same subject. These low scores may be due to excessively rigorous criteria for the inclusion of clinical trials based on organisational or behavioural interventions. CPGs evaluated offer little or no information on the strengths and limitations of the evidence used, or the approach and timing of updates or consideration of emerging evidence.

Finally, a lack of consideration towards applicability criteria may hamper the implementation of guideline recommendations in clinical practice, this aspect is as important as the methodological rigour and stakeholder involvement in effective knowledge mobilisation. Currently, there is a fundamental problem that may be influencing the difficulty of implementing recommendations. CPGs only include studies that provide results about what needs to be done and are collected in the best available evidence. However, the use of realist revisions on what has worked, and for whom could help in optimising the implementation process.35 In the last decade, we have observed the growth in the volume of evidence to be appraised to a point of unmanageability, especially clinical guidelines, obfuscating evidence integration.14 Some CPGs include studies with statistically significant benefits, but which may be marginal in clinical practice.13 Ultimately, the low scores in the domains referred along with the issues cited above may explain the low applicability of guidelines and the reported low adherence of recommendations across many clinical areas and settings.36

Clinical implementation is a complex, multifaceted phenomenon37 which requires a deep understanding of decision-making processes and active strategies within organisations.38 Such strategies ought to incorporate mechanisms to influence how tacit and explicit knowledge is constructed and internalised in routine practice.39 Decision-making is not only achieved through a careful selection of information based on a defined evaluation of possible outcomes40 but depends as much on multiple human factors. Scientific evidence should receive significant, but not necessarily predominant, attention since optimal decisions would require the integration of such best evidence with clinical expertise together with the preferences of users,10 41 leading to the development and use of ‘mindlines’.16 39 42 Further, knowledge transfer must be guided by conceptual models that articulate proposed strategies to effect behavioural change and the factors that may influence adoption of best clinical practice. At present, models of proven efficiency43–47 have been instrumental to translate the results produced by evidence syntheses onto the necessary adherence by professionals, thus facilitating dynamics of knowledge transfer and mobilisation.48 We endorse the need to supplement CPG with implementation guides that facilitate and ensure effective implementation process to fidelity the best available evidence. We endorse the need to supplement CPG with implementation guides that facilitate and ensure effective implementation process to fidelity the best available evidence, including result indicators associated with prevention and control of infections related to VAD.49 50 Likewise, the identification of barriers and constraints at institutional and individual level should be the first step,51 for the inclusion of strategies that promote fidelity52 to recommendations through multicomponent and multimodal interventions53 54 formed by facilitation of evidence,46 55 the use of e-learning in health professionals56 and health information of the users.57 These multimodal interventions may lead to notable improvements in the clinical outcomes of users.2 58

Limitations

This review presents limitations, chiefly related to the search process and inclusion criteria of guidelines. For example, we excluded guidelines not published in English and Spanish. Regarding the quality evaluation of the guidelines, the AGREE II instrument, despite its validity, does not offer guidance to interpret the results. Our criteria for endorsing some guidelines for practice of care were based on the global score across all domains, being a subjective perception to each evaluator. However, we performed a concordance analysis among the evaluators’ findings for the recommendation of the guidelines. Further, AGREE II only assesses the quality of guidelines’ structure and content but does not delve into the practical implications or quality of the recommendations.

Conclusions

Our findings indicate that quality of CPGs reviewed for the prevention of complications, management and care associated with VAD was moderate, being substantial the overall agreement among reviewers. There is a need to incorporate mechanisms of critical evaluation about the validity and reliability of selected guidelines within environments of practice, as a prior and essential requisite to knowledge mobilisation. The gaps identified with low CPG scores in critical domains for knowledge transfer may explain the suboptimal clinical impact of guidelines on healthcare practice. This is evidenced by the partly adherence of healthcare professionals to recommendations.36 59 We endorse the need to supplement clinical practice documents with implementation guides that ensure effective implementation.