Systematic review of clinical practice guidelines in kidney transplantation.

BACKGROUND: Clinical practice guidelines (CPGs) are widely used to inform the development of protocols for clinical management. Previous work has demonstrated that the quality of CPGs varies widely. This systematic review aimed to determine the quality of CPGs in kidney transplantation in the UK.
METHODS: CPGs in kidney transplantation published between 2010 and 2017 were identified through searches of MEDLINE, NHS NICE Evidence, and websites of relevant UK societies. Using the Appraisal of Guidelines for Research and Evaluation (AGREE) II tool, three appraisers rated the quality of CPGs across six domains, the overall quality of each CPG, and whether it should be recommended for future use. Domain scores were calculated, and inter-rater reliability using the intraclass correlation coefficient (ICC) was reported.
RESULTS: Thirteen CPGs met the inclusion criteria. The domain 'clarity of presentation' scored highest, followed closely by 'scope and purpose'. The poorest scoring domains were 'applicability' and 'editorial independence'. Editorial independence also had the widest range of scores. Of the 13 CPGs, one was not recommended for future use, seven were recommended for use with modifications, and five for future use with no need for modification. Mean overall CPG quality was 5 (range 3-6) of a maximum score of 7, and mean inter-rater reliability was substantial with an ICC of 0·71.
CONCLUSION: UK CPGs scored satisfactorily, although with wide variation in how well each domain scored both within and across CPGs. The quality of UK CPGs can still be improved.

Introduction

Clinical practice guidelines (CPGs) are considered important in guiding clinical care and driving improvements in clinical outcomes by defining evidence‐based practice. Concerns regarding the quality of guidelines may contribute to poor uptake1 2. Critical appraisal of CPGs in healthcare is limited, and guidelines assessed so far show variability in their quality3.

Kidney donation and transplantation is a complex area, with centres needing to consider donor and recipient assessment and eligibility, surgical management, medical management, immunosuppressive medication and management of complications. Centres rely heavily on evidence‐based CPGs in the development of local protocols for patient management. It is therefore imperative that CPGs are robustly developed, up‐to‐date and evidence‐based.

A small number of previous studies have examined the quality and consistency of CPGs within specific areas of transplantation, such as living kidney donation4. As part of this study, ten evidence‐based CPGs and consensus statements published between 1996 and 2010 from Australia, the UK, the USA, continental Europe and Canada that focused on the screening and follow‐up of living kidney donors were considered. The review found a difference in recommendations across organizations, with most CPGs lacking methodological rigour. Similarly, a systematic review5 assessing the quality of cancer screening recommendations for solid organ transplant recipients found considerable variability in recommendations and weaknesses in the quality of CPGs.

With around 3000 kidney transplants taking place each year in the UK alone, it is imperative that high‐quality UK CPGs exist in kidney transplantation6. No quality assessment of UK CPGs in kidney transplantation has been undertaken previously. The aim of this systematic review was to perform a critical appraisal of UK CPGs across all aspects of kidney transplantation.

Methods

Identification of clinical practice guidelines

The critical appraisal of UK CPGs was undertaken as part of a larger systematic review project of international CPGs in kidney transplantation, registered with the PROSPERO database of systematic review protocols (PROSPERO ID: CRD42015027356). The review was reported in line with the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) statement7. Relevant CPGs published from 2010 until January 2017 were identified through systematic searches of MEDLINE and the National Health Service (NHS) and National Institute for Health and Care Excellence (NICE) Evidence search platform. In addition, websites of the British Renal Society, British Transplantation Society (BTS), NHS Blood and Transplant, and the Renal Association were searched manually for any CPGs that may have been missed by the systematic literature searches.

Bibliographical searches were conducted on 24 January 2017 and included keywords and Medical Subject Headings (MeSH) terms for kidney transplantation, combined with terms for CPGs. Results were limited to the English language. The full search strategy employed is shown in Appendix  S1 (supporting information). One author reviewed the abstracts of all potentially eligible studies and searched the society websites. Full texts of possible inclusions were checked by two authors against the inclusion criteria.

Inclusion and exclusion criteria and data extraction

All guidelines and consensus statements published by a UK society or other professional organization, where the main aim was to provide guidelines and/or recommendations specific to kidney transplantation, were included. Studies intended for other solid organs (for example liver, lung, heart) and those primarily for kidney disease were included only if they had a chapter specific to kidney transplantation. Draft, unpublished guidelines, conference papers, opinions, discussion papers, reviews and summaries of CPGs where the full guideline was available elsewhere were excluded. Historical versions of guidelines that had subsequently been updated and CPGs published before 2010 were also excluded, as these were considered to be out of date with respect to current clinical practice.

Patient involvement

All CPGs were examined for patient involvement, which was considered to have been met if the guideline explicitly described that a patient representative was included in the working group and contributed to the development of the CPG. Without such a statement, the CPG was classified as not including patient involvement.

The AGREE II instrument

The Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument was considered the most comprehensively validated appraisal tool, having been used in previous healthcare studies to determine the quality of CPGs, and implemented for CPG evaluation and development8 9.

AGREE II consists of 23 items organized into six quality domains: scope and purpose, stakeholder involvement, rigour of development, clarity of presentation, applicability, and editorial independence (Table  1). Each item is scored on a seven‐point Likert scale from 1 (strongly disagree) to 7 (strongly agree). A score of 1 is given when there is no information relevant to the item, or it is very poorly reported, and a score of 7 is given when the reporting is exceptional and the item meets the full criteria. A score is calculated for each of the six domains; this involves summation of the scores for the individual items in that domain and scaling the total as a percentage of the maximum possible score for that domain (range 0–100 per cent).

Table 1

AGREE II domains and items

AGREE II domain	AGREE II item
Scope and purpose	1. The overall objective(s) of the guideline is (are) specifically described
	2. The health question(s) covered by the guideline is (are) specifically described
	3. The population (patients, public, etc.) to whom the guideline is meant to apply is specifically described
Stakeholder involvement	4. The guideline development group includes individuals from all the relevant professional groups
	5. The views and preferences of the target population (patients, public, etc.) have been sought
	6. The target users of the guideline are clearly defined
Rigour of development	7. Systematic methods were used to search for evidence
	8. The criteria for selecting the evidence are clearly described
	9. The strengths and limitations of the body of evidence are clearly described
	10. The methods for formulating the recommendations are clearly described
	11. The health benefits, side effects, and risks have been considered in formulating the recommendations
	12. There is an explicit link between the recommendations and the supporting evidence
	13. The guideline has been externally reviewed by experts prior to its publication
	14. A procedure for updating the guideline is provided
Clarity of presentation	15. The recommendations are specific and unambiguous
	16. The different options for management of the condition or health issue are clearly presented
	17. Key recommendations are easily identifiable
Applicability	18. The guideline describes facilitators and barriers to its application
	19. The guideline provides advice and/or tools on how the recommendations can be put into practice
	20. The potential resource implications of applying the recommendations have been considered
	21. The guideline presents monitoring and/or auditing criteria
Editorial independence	22. The views of the funding body have not influenced the content of the guideline
	23. Competing interests of guideline development group members have been recorded and addressed

In addition to the domain scores, AGREE II consists of two overall assessments. For the first, appraisers rate the quality of the entire CPG on the same seven‐point Likert scale. For the second overall assessment, appraisers decide whether they would consider the CPG appropriate for future use, and answer with either ‘yes’, ‘yes with modifications’ or ‘no’.

Critical appraisal with AGREE II

Each CPG was evaluated by three appraisers (one transplant clinician and two methodologists). All appraisers completed the AGREE II tutorial and practice exercise before commencing critical assessments of CPGs.

To account for discrepancies, a methodology was followed that allowed appraisers to revise their initial ratings on any item where their score differed from that of the other two appraisers by 5 or more points (1 or 2 versus 7, and 1 versus 6)10. The outlying appraiser was informed of the item where the discrepancy occurred and given the opportunity to adjust their score if information was missed or misinterpreted. If the appraiser chose not to adjust their score, the other two appraisers were contacted and given the same opportunity. Both the initial and the adjusted scores were calculated.

Statistical analysis

Domain scores are expressed as percentages with means and ranges. The intraclass correlation coefficient (ICC) was used to measure inter‐rater reliability11. Two‐way mixed model statistical analyses were performed using MedCalc® version 15.11.0 for Windows® (https://www.medcalc.org). The analysis included average measures for absolute agreement; the degree of agreement was measured using the following definitions for ICC12: 0·20 or less, slight agreement; 0·21–0·40, fair agreement; 0·41–0·60, moderate agreement; 0·61–0·80, substantial agreement; 0·81–1·00, almost perfect agreement.

Results

Included studies

Literature searches retrieved 897 records (NHS NICE Evidence, 489; MEDLINE, 407; society websites, 1), of which 13 CPGs were subsequently included in this systematic review13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25. These included six published by the BTS, two by the Renal Association, one by NICE, one by the Royal College of Pathologists in consultation with the Renal Association and National Renal Pathology External Quality Assessment Scheme, and three that had been developed by the BTS in collaboration with at least one of the following organizations: the Renal Association, British Society for Histocompatibility and Immunogenetics and the British Committee for Standards in Haematology. Six were published between 2010 and 2011, and seven from 2013 onwards.

Domain scores

The highest domain score was from the domain ‘clarity of presentation’ (mean 85 (range 65–94) per cent), followed by ‘scope and purpose’ (79 (50–93) per cent) (Table  2). The domains ‘stakeholder involvement’ (55 (33–72) per cent) and ‘rigour of development’ (54 (26–75) per cent) had intermediate scores, and the poorest scoring domains were ‘applicability’ (37 (18–63) per cent) and ‘editorial independence’ (39 (0–83) per cent). The domain editorial independence had the greatest range of scores; clarity of presentation had the smallest range.

Table 2

Domain scores, overall scores and intraclass correlation coefficients for all UK clinical practice guidelines

UK CPGs	Year	Domain
		Scope and purpose (%)	Stakeholder involvement (%)	Rigour of development(%)	Clarity of presentation (%)	Applicability (%)	Editorial independence (%)	Mean overall score	Recommended for future use	ICC
Clinical practice guidelines for the management of atypical haemolytic uraemic syndrome in the United Kingdom16	2010	93	33	26	89	22	0	3	No	0·92
Renal Association clinical practice guideline on the assessment of the potential kidney transplant recipient18	2011	78	33	49	91	38	69	5	Yes	0·81
Renal Association clinical practice guideline on post‐operative care of the kidney transplant recipient15	2011	89	57	54	94	63	56	6	Yes	0·64
Living donor kidney transplantion (3rd edn)19	2011	85	65	58	78	32	8	5	Yes, with modifications	0·78
The prevention and management of CMV disease after solid organ transplantation (3rd edn)25	2011	74	57	51	91	60	39	5	Yes, with modifications	0·72
Single‐port laparoscopic nephrectomy. Interventional procedures guidance23	2011	56	50	58	65	54	36	5	Yes, with modifications	0·18
Transplantation from deceased donors after circulatory death20	2013	76	56	65	94	18	33	5	Yes, with modifications	0·82
Tissue pathway for medical renal biopsies24	2013	87	69	67	80	63	83	5	Yes	0·62
Management of the failing kidney transplant21	2014	78	67	67	87	21	44	5	Yes	0·81
The detection and characterisation of clinically relevant antibodies in allotransplantation14	2014	50	54	41	72	22	6	4	Yes, with modifications	0·56
UK guidelines for living organ donation from prisoners13	2015	87	48	28	80	21	19	3	Yes, with modifications	0·86
Kidney and pancreas transplantation in patients with HIV (2nd edn)22	2015	87	57	66	94	21	50	5	Yes, with modifications	0·85
Guidelines for antibody incompatible transplantation (3rd edn)17	2016	87	72	75	89	49	64	6	Yes	0·68
Mean (range)		79 (50–93)	55 (33–72)	54 (26–75)	85 (65–94)	37 (18–63)	39 (0–83)	5 (3–6)		0·71 (0·18–0·92)

This table is based on original scores. CPGs, clinical practice guidelines; CMV, cytomegalovirus.

Overall scores

The overall score varied from 3 to 6 of 7 across CPGs. Ten of the 13 CPGs scored at least 5, and three CPGs scored 3 or 4. The highest scoring CPGs were rated at 6. Overall, seven CPGs were recommended for future use with modifications, five for use without modifications, and one was not recommended (Table  2).

Individual item scores

The highest to lowest scoring individual items are shown in Table  3. Compliance with AGREE II items ranged from 14 to 100 per cent across all CPGs, with the highest mean compliance being 93 per cent and the lowest 38 per cent. The greatest range in item compliance was for ‘Competing interests of guideline development group members have been recorded and addressed’, followed by ‘The guideline presents monitoring and/or auditing criteria’.

Table 3

AGREE II items listed from highest to lowest scoring

Order	Compliance (%)*	Domain	AGREE II Item
1	93 (76–100)	Clarity of presentation	Key recommendations are easily identifiable
2	89 (67–100)	Clarity of presentation	The recommendations are specific and unambiguous
3	86 (62–100)	Scope and purpose	The overall objective(s) of the guideline is (are) specifically described
4	84 (57–95)	Scope and purpose	The population (patients, public, etc.) to whom the guideline is meant to apply is specifically described
5	81 (38–100)	Rigour of development	There is an explicit link between the recommendations and the supporting evidence
6	79 (62–90)	Rigour of development	The health benefits, side effects, and risks have been considered in formulating the recommendations
7	78 (52–95)	Clarity of presentation	The different options for management of the condition or health issue are clearly presented
8	75 (52–90)	Scope and purpose	The health question(s) covered by the guideline is (are) specifically described
9	74 (33–100)	Stakeholder involvement	The guideline development group includes individuals from all the relevant professional groups
10	73 (19–95)	Rigour of development	The strengths and limitations of the body of evidence are clearly described
11	71 (33–95)	Stakeholder involvement	The target users of the guideline are clearly defined
12	60 (19–81)	Rigour of development	The guideline has been externally reviewed by experts prior to its publication
13	56 (19–100)	Applicability	The guideline presents monitoring and/or auditing criteria
14	55 (24–86)	Rigour of development	The methods for formulating the recommendations are clearly described
15	54 (14–86)	Rigour of development	A procedure for updating the guideline is provided
16	51 (14–100)	Editorial independence	Competing interests of guideline development group members have been recorded and addressed
17	47 (33–57)	Applicability	The guideline describes facilitators and barriers to its application
18	46 (24–86)	Rigour of development	Systematic methods were used to search for evidence
19	45 (14–81)	Editorial independence	The views of the funding body have not influenced the content of the guideline
20	43 (24–71)	Applicability	The guideline provides advice and/or tools on how the recommendations can be put into practice
21	39 (14–71)	Rigour of development	The criteria for selecting the evidence are clearly described
22	39 (19–57)	Stakeholder involvement	The views and preferences of the target population (patients, public, etc.) have been sought
23	38 (19–67)	Applicability	The potential resource implications of applying the recommendations have been considered

Table is based on original scores.

Values are mean (range).

Patient involvement and funding body

No CPG explicitly described in the guideline document that a patient representative was included in the working group and had contributed to the development of the guideline. One CPG included funding details within the guidance document. Funding information was available for the BTS in their Guideline Development Policy26, but did not describe how CPGs produced in collaboration with other organizations were funded. CPGs produced solely by the BTS as well as the CPG published by the Royal College of Pathologists did not receive external funding. Funding information was not located for CPGs developed by, or in collaboration with, the Renal Association, NICE, the British Society for Histocompatibility and Immunogenetics, and the British Committee for Standards in Haematology.

Inter‐rater reliability

The ICC across CPGs ranged from 0·18 to 0·92, indicating a slight to almost perfect level of inter‐rater agreement (Table  2). The mean inter‐rater reliability was substantial at 0·71.

Adjusted scores

Scores on 11 items were adjusted across CPGs. Five of these adjustments were made by a clinician and six by a methodologist. The domain scores for the original and adjusted scores were the same for all domains except ‘applicability’ (decreased from 37 to 36 per cent) and ‘editorial independence’ (decreased from 39 to 36 per cent), which showed minor changes (Table  S1, supporting information). There were no score adjustments across CPGs for the two overall ratings.

Discussion

This systematic review of CPGs in kidney transplantation has demonstrated that, overall, UK CPGs scored satisfactorily. Most, however, still lacked key information, and wide variation existed both within and across CPGs in how well each AGREE II domain scored.

The highest scoring domains of ‘scope and purpose’ and ‘clarity of presentation’ were reported well across CPGs, consistent with previous studies from the USA, UK, Europe, Canada, Australia, Switzerland and international organizations4 5. These domains evaluate the overall aim of the CPG, the specific health questions and target population, as well as the language, structure and format of the CPG. Previous studies27 28 have shown that the simplicity and format of CPGs may determine their usage among professionals, and those CPGs that are easily understood and tested are more likely to be implemented. With this in mind, perhaps guideline developers believe these to be essential components of CPGs and therefore allocate more time and resources to the development and incorporation of these aspects.

The domains ‘stakeholder involvement’ and ‘rigour of development’ were not reported as well among UK CPGs, and could be improved considerably. The rigour of development domain assesses the methodological rigour utilized in the development of CPGs. In this domain, CPGs are examined in the formulation of recommendations, as to whether a voting system, informal consensus or more structured processes such as the Delphi method or Glaser techniques have been employed. This domain also considers the selection of evidence, and whether the strengths and limitations have been assessed. Formal instruments such as the Jadad scale and Grading of Recommendations Assessment, Development and Evaluation (GRADE) method may be utilized. It has been suggested that this domain is the strongest indicator of guideline quality and is closely linked to the confidence placed in guidance29. However, UK CPGs demonstrated considerable variability in how well this domain was addressed. Most lacking among UK CPGs were the undertaking and reporting of systematic searches of the evidence and a clear description of the inclusion and exclusion criteria. Transparency of the search process reassures readers that the best available evidence has been used in the development of CPGs. The quality of UK CPGs in this domain would likely improve by addressing these aspects.

The domain ‘stakeholder involvement’ examined the individuals involved in the development of CPGs to ensure all relevant professional groups were incorporated, including patients and the public. Strong methodological expertise and the inclusion of all relevant clinical professionals as well as patient and public groups have been advocated3 as paramount for developing and implementing guidelines. Patient and public representatives offer valuable social, ethical and organizational insights that might otherwise be overlooked30. An example is ‘patient preferences’, where the desirability of a particular treatment choice may determine the optimal treatment for the individual31. These representatives offer perspectives that may enable protection against flawed practices, which are perhaps incorporated by professionals to control costs or protect special interests32. This analysis demonstrated that most UK CPGs did not consider the views and preferences of patients and public groups in developing recommendations. The quality of UK CPGs could be improved by including patient and public representatives on the guideline development panel, where they have the opportunity to vote and comment on recommendations as well as debate the role of patients in the implementation of CPGs33. Formal consultations, interviews or literature reviews to determine the priority topics, values and preferences of these groups could be used, or draft, unpublished CPGs sent to patients or members of the public for review34.

As in other studies4 5, the ‘applicability’ domain was the poorest scoring domain across UK CPGs. It is concerning that this domain scored inadequately as it considers how the CPG could be used in daily practice, including facilitators and barriers to the application of the CPG, resource implications, and the inclusion of monitoring or auditing criteria. The lowest scoring item was in this domain (The consideration of the potential resource implications of applying the recommendations). Resources have been shown to be critical to the implementation of guidance, and without considering resource implications CPGs are less clinically applicable, less likely to be implemented and less conducive to improving healthcare practices1. Not including practical considerations for the implementation of recommendations in the clinical setting may waste the vast resources invested in developing CPGs when recommendations cannot be translated into practice, or are unaffordable35. The applicability of CPGs should be examined and incorporated to ensure they do not become an unattainable standard for clinical practice.

The domain ‘editorial independence’ considered the influence of the funding body in the formulation of recommendations, and the potential conflicts of interest of guideline developers. This domain also scored poorly, and the greatest variability in compliance to AGREE II was in recording and addressing the competing interests of guideline developers. For CPG users to assess the risk of bias accurately and have confidence in the credibility of the guidance, it is essential for guideline developers to ensure transparency in this area. Other studies have also identified deficiencies in this domain1 3. There may also be a commonality of interactions between authors of CPGs and the pharmaceutical industry, although many of these authors are doubtful that their relationships have influenced recommendations36. Judgements may be affected subconsciously, and it is difficult to tackle these issues when declarations of interest are dependent on self‐reporting and veracity36 37. A formal published process that reports all conflicts of interest and funding will likely improve the methodological rigour of CPGs. These declarations should be located within CPGs, rather than summary documents that can be difficult to locate38.

This systematic review is not without limitations. Searches were constrained to include only relevant UK societies. It may be that other UK CPGs not endorsed by those societies, such as European or international CPGs, are used regularly by the UK kidney transplant community. A strength of this systematic review is the inclusion of three appraisers for each CPG. To account for differences that may have been attributed to clinical versus methodological opinions, this systematic review included both methodological experts and kidney transplant clinicians as appraisers. Inter‐rater reliability demonstrated that reliability was substantial and that no major discrepancies existed between appraisers. This review included the most recent version of UK CPGs, although almost half were published before 2012. Interestingly, the only CPG not recommended for future use was published in 2010.

AGREE II also has limitations. It requires methodological transparency, and does not specifically assess the quality of evidence behind CPG development. CPGs may comprise robust methods, but the scientific content of recommendations could be inaccurate or biased owing to an incomplete collection or inadequate interpretation of the evidence39. Variability in the content of national recommendations for the screening and follow‐up of living kidney donors has been shown, when based on expert opinion or sparse evidence4. Although AGREE II focuses more on methodological quality than on content, clear reporting of methodological aspects allows guideline users to be better equipped to judge the validity of the content. AGREE II considers all domains as equally important; however, the number of items in each domain varies. Items in the ‘editorial independence’ domain will have a greater effect on the overall domain score as it contains only two items, compared with eight in ‘rigour of development’. No specific guidance is given by AGREE II on how to make the overall assessments, and appraisers may have weighed the various agree domains differently when making their judgements. A high score also does not prove that a particular guideline is useful in clinical practice9.

In general, CPGs did not adequately consider the potential resource implications of applying the recommendations, nor did they include the views and preferences of the target population. However, the clarity of presentation across all CPGs was generally well executed. Key recommendations were easily identifiable, specific and unambiguous, and the different options for management of the condition or health issue were clearly presented. Closer adherence to the AGREE II items would likely improve the overall quality of UK CPGs in kidney transplantation.

Disclosure

The authors declare no conflict of interest.

Additional supporting information may be found online in the supporting information tab for this article.

Appendix S1 Search strategy (Word document)

Table S1 Adjusted domain scores, overall scores and intraclass correlation coefficients for all UK clinical practice guidelines (Word document)

Appendix S1 Search strategy

Table S1 Adjusted domain scores, overall scores and intraclass correlation coefficients for all UK clinical practice guidelines

Click here for additional data file.