Abdominal aortic aneurysm clinical practice guidelines: a methodological assessment using the AGREE II instrument.

OBJECTIVES: Abdominal aortic aneurysm (AAA) clinical practice guidelines (CPGs) provide evidence-based information on patient management; however, methodological differences exist in the development of CPGs. This study examines the methodological quality of AAA CPGs using a validated assessment tool.
METHODS: Medline, EMBASE and online CPG databases were searched from 1946 to 31 October 2021. Full-text, English language, evidence-based AAA CPGs were included. Consensus-based CPGs, summaries of CPGs or CPGs which were only available on purchase were excluded. Five reviewers assessed their quality using the Appraisal of Guidelines for Research and Evaluation II instrument. An overall guideline assessment scaled score of ≥80% was considered as the threshold to recommend CPG use in clinical practice.
RESULTS: Seven CPGs were identified. Scores showed good inter-reviewer reliability (intraclass correlation coefficient 0.943, 95% CI 0.915 to 0.964). On average, CPGs performed adequately with mean scaled scores of over 50% in all domains. However, between CPGs, significant methodological heterogeneity was observed in all domains. Four CPGs scored ≥80% (European Society of Cardiology, the Society of Vascular Surgery, the European Society of Vascular Surgery and the National Institute of Health and Care Excellence), supporting their use in clinical practice.
CONCLUSIONS: Four CPGs were considered of adequate methodological quality to recommend their use in clinical practice; nonetheless, these still showed areas for improvement, potentially through performing economic analysis and trial application of recommendations. A structured approach employing validated CPG creation tools should be used to improve rigour of AAA CPGs. Future work should also evaluate recommendation accuracy using validated appraisal tools. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study uses a widely used validated assessment tool to objectively assess the methodological quality of abdominal aortic aneurysm (AAA) clinical practice guidelines (CPGs).

Good inter-reviewer reliability was maintained across the five assessors who independently assessed the AAA CPGs.

A systematic approach was used to identify CPGs for inclusion in this review.

However, limited number of CPGs were available for assessment.

This study also does not comment on the accuracy of recommendations or the robustness of evidence behind them.

Introduction

The management of the abdominal aortic aneurysm (AAA) patient is complex and involves members of the multidisciplinary team, including, but not limited to, vascular scientists/technologists, specialist nurses, emergency physicians, interventional radiologists and vascular surgeons. Multiple management pathways exist, with a spectrum of diagnostic and treatment options available. Therefore, in view of this complexity and to aid clinicians in their management of AAA, multiple clinical practice guidelines (CPGs) have been created by national and international organisations. These are developed employing different methodologies, leading to heterogeneity stemming from evidence grading, recommendation level, topics covered and the professionals who have been involved in their creation. Clinicians, while not legally obligated to use CPGs, will often do so in their practice. Methodological rigour is, therefore, required to provide accurate recommendations for the best care of AAA patients.

The quality of individual CPG methodology may be assessed using objective instruments designed for such a purpose. One such instrument is the Appraisal of Guidelines for Research and Evaluation II (AGREE II) tool.1 While other tools exist, the AGREE II instrument has been the most extensively employed to assess CPG methodological quality and has been applied in many other clinical areas, including orthopaedic surgery and other subject matters in vascular surgery.2 3 In addition, the AGREE II instrument has the benefit of acting as both a scoring system to evaluate CPG methodology as well as providing a framework for CPG development. In depth assessment of the process of CPG guideline development is performed using six domains: (1) scope and purpose, (2) stakeholder involvement, (3) rigour of development, (4) clarity of presentation, (5) applicability and (6) editorial independence; importantly, it does not evaluate the scientific or clinical accuracy of the recommendations.

This study aims to employ the AGREE II instrument to determine the methodological quality of current AAA CPGs to fulfil two main objectives:

Identify AAA CPGs of high methodological quality that may be recommended for use in clinical practice.

Determine areas for improvement in future versions of current AAA CPGs or newly developed CPGs.

Materials and methods

Search strategy

The Medline and EMBASE bibliographic databases were searched from 1946 to 31 October 2021 using the search algorithm outlined below:

(((((practice guideline*) OR clinical practice guideline*) OR recommendation*))

AND

((((management) OR diagnosis) OR assessment) OR treatment))

AND

(((((abdominal aorta) OR abdominal aortic aneurysm*) OR aortic aneurysm*) OR aorto-iliac aneurysm*) OR AAA))

The Guidelines International Network and NHS Evidence Search databases were also searched over the same period using the terms “aorta”, “aortic aneurysm” and “abdominal aortic aneurysm”.

The reference lists of identified CPGs were hand searched to identify further relevant guidelines. Two reviewers (KHMT and SS) independently performed the article search and reviewed the full CPGs to ensure fulfilment of all inclusion criteria. Any conflict between the two reviewers was discussed in person, and any unresolved disagreements were referred to a third reviewer (SO).

Inclusion and exclusion criteria

Full text, English language CPGs including evidence-based recommendations on AAA diagnosis and management were included. CPGs based only on expert consensus, CPG summaries or CPGs which were only available by purchase were excluded.

Agree II assessment

Five reviewers (KHMT, SS, MM, AG and TL) independently assessed the included CPGs. All reviewers were selected based on their experience and extensive involvement in vascular surgery research. A range of clinical experience was also represented by the reviewers chosen, ranging from a newly qualified doctor to a trained consultant vascular surgeon. For each quality statement in the seven domains of the AGREE II instruments (domains and quality statements can be found in table 1), CPGs were rated from 1 (lowest quality) to 7 (highest quality). The criteria for grading scores are provided in the AGREE II instrument for each statement to guide reviewers in their assessment but does not give specific points required for each numerical score.4 Readers should note that the final domain entitled Overall Guideline Assessment contains two statements, the first of which (on overall quality) is rated from 1 to 7 while the second (on recommendation for use) is scored with a ‘yes’, ‘yes with modifications’ or ‘no’.

Table 1

Agree II instrument domains and statements

Domain	Statements
1—Scope and purpose	The overall objective(s) of the guideline is (are) specifically described. The health question(s) covered by the guideline is (are) specifically described. The population (patients, public, etc) to whom the guideline is meant to apply is specifically described.
2—Stakeholder involvement	The guideline development group includes individuals from all relevant professional groups. The views and preferences of the target population (patients, public, etc) have been sought. The target users of the guideline are clearly defined.
3—Rigour of development	Systematic methods were used to search for evidence. The criteria for selecting the evidence are clearly described. The strengths and limitations of the body of evidence are clearly described. The methods for formulating the recommendations are clearly described. The health benefits, side effects and risks have been considered in formulating the recommendations. There is an explicit link between the recommendations and the supporting evidence. The guideline has been externally reviewed by experts prior to its publication. A procedure for updating the guideline is provided.
4—Clarity of presentation	The recommendations are specific and unambiguous. The different options for management of the condition or health issue are clearly presented. Key recommendations are easily identifiable.
5—Applicability	The guideline describes facilitators and barriers to its application. The guideline provides advice and/or tools on how the recommendations can be put into practice. The potential resource implications of applying the recommendations have been considered. The guideline presents monitoring and/or auditing criteria.
6—Editorial Independence	The views of the funding body have not influenced the content of the guideline. Competing interests of guideline development group members have been recorded and addressed.
Overall guideline Assessment	Rate the overall quality of this guideline. I would recommend this guideline for use. (Yes, yes with modifications, no).

Numerical domain scores were added and scaled using a predefined equation to determine the scaled quality score for each domain. The domain scaled quality score for each CPG was determined by taking the raw score total as a percentage of the maximum possible score for that domain using the following equation: [Obtained score - Minimum possible score/Maximum possible score − Minimum possible score] × 100. All assessment and scaling of scores were performed according to the user manual available from the AGREE Research Trust website.4

A two-way mixed model was used to calculate intraclass correlation coefficients (ICCs) as a measure of inter-reviewer reliability. An overall guideline assessment scaled score of ≥80% was required to consider the guideline of adequate quality to recommend use in clinical practice, as previously done in published studies using the AGREE II instrument.5 6 All statistical analyses were performed using SPSS Statistics V.25 (IBM).

Patient and public involvement

No patients were involved in this study.

Results

Selected guidelines

Seven CPGs were identified from the literature search7–13 (figure 1). CPGs were published between 2005 and 2020. Guideline development group members included general practitioners, nurses, emergency physicians, radiologists, cardiothoracic surgeons and vascular surgeons. The CPGs originated from the USA (n=3),7–9 with the remainder coming from Europe (n=3)10–12 and Brazil (n=1).13 Further guideline characteristics are summarised in (online supplemental file 1.

Figure 1

Guideline selection flow chart. AAA, abdominal aortic aneurysm; CPG, clinical practice guideline.

Quality scores

Inter-reviewer reliability varied between domains, ranging from moderate (domains 1 and 6) to good (domains 2, 3, 4, 5 and overall quality) as determined using the ICCs calculated. The overall ICC of all scores showed excellent reliability (ICC 0.943, 95% CI 0.915 to 0.964) (table 2).

Table 2

Intraclass correlation coefficient of scores given to the individual domains

Domain	Intraclass correlation	95% CI		Inter-reviewer reliability
		Lower bound	Upper bound
1	0.679	0.364	0.921	Moderate
2	0.891	0.722	0.977	Good
3	0.797	0.541	0.954	Good
4	0.753	0.469	0.943	Good
5	0.875	0.688	0.973	Good
6	0.709	0.405	0.930	Moderate
Overall Guideline Assessment	0.841	0.620	0.965	Good
All domains	0.943	0.915	0.964	Excellent

Raw and scaled quality scores are summarised in table 3.

Table 3

Individual reviewer, raw total and scaled scores for individual domains

	Domain (min. score, max. score)
Guideline	1 (min. 3, max. 21)	2 (min. 3, max. 21)	3 (min. 8, max. 56)	4 (min. 3, max. 21)	5 (min. 4, max. 28)	6 (min. 2, max. 14)	Overall guideline assessment (min. 1, max. 7)
American College of Cardiology/American Heart Association7
Reviewer 1	14	11	40	18	12	11	5
Reviewer 2	18	14	41	20	19	11	6
Reviewer 3	17	13	46	20	20	11	6
Reviewer 4	15	13	44	20	16	13	6
Reviewer 5	15	8	49	21	10	12	5
Raw total	79	59	220	99	77	58	28
Scaled (%)	71.1	48.9	75.0	93.3	47.5	80.0	76.7
Brazilian Society of Cardiovascular Surgery13
Reviewer 1	11	8	20	13	5	2	3
Reviewer 2	14	9	34	20	9	2	4
Reviewer 3	14	10	30	18	9	2	4
Reviewer 4	12	7	16	12	4	5	2
Reviewer 5	14	8	16	12	4	11	4
Raw total	65	42	116	75	31	22	17
Scaled (%)	55.5	30.0	31.7	66.7	9.2	20.0	40.0
Society of Interventional Radiology8
Reviewer 1	10	6	32	8	10	7	4
Reviewer 2	18	11	40	12	13	9	5
Reviewer 3	15	10	42	14	15	8	5
Reviewer 4	12	8	22	9	11	11	4
Reviewer 5	13	6	22	9	11	9	3
Raw total	68	41	158	52	60	44	21
Scaled (%)	58.9	28.9	49.2	41.1	33.3	56.7	53.3
European Society of Cardiology10
Reviewer 1	15	10	40	17	15	12	6
Reviewer 2	21	15	43	21	22	12	6
Reviewer 3	19	14	46	19	22	12	6
Reviewer 4	12	9	34	21	9	12	5
Reviewer 5	15	10	34	21	9	12	6
Raw total	82	58	197	99	77	60	29
Scaled (%)	74.4	47.8	65.4	93.3	47.5	83.3	80.0
Society for Vascular Surgery9
Reviewer 1	14	10	39	17	21	9	6
Reviewer 2	19	16	49	19	23	12	7
Reviewer 3	17	14	48	21	21	11	6
Reviewer 4	15	16	45	21	16	10	6
Reviewer 5	17	12	45	21	16	11	5
Raw total	82	68	226	99	97	53	30
Scaled (%)	74.4	58.9	77.5	93.3	64.2	71.7	83.3
European Society of Vascular Surgery11
Reviewer 1	13	13	37	16	15	8	6
Reviewer 2	18	19	48	19	23	11	7
Reviewer 3	18	18	50	21	21	11	6
Reviewer 4	17	15	42	21	13	10	6
Reviewer 5	18	14	42	21	13	10	6
Raw total	84	79	219	98	85	50	31
Scaled (%)	76.7	71.1	74.6	92.2	54.2	66.7	86.7
National Institute of Clinical Excellence12
Reviewer 1	18	18	44	19	21	12	7
Reviewer 2	19	20	50	17	25	13	7
Reviewer 3	21	21	51	17	27	13	7
Reviewer 4	21	21	47	20	18	11	7
Reviewer 5	21	21	47	20	18	12	7
Raw total	100	101	239	93	109	61	35
Scaled (%)	94.4	95.6	82.9	86.7	74.2	85.0	100.0
Average scaled scores
Mean±SD (%)	72.2±12.8		65.2±18.4	81.0±20.1	47.1±21.2	66.2±22.7	74.3±20.6

Cells with scaled scores ≤25%, 26%–50%, 51%–75% and ≥76% are coloured with red, orange, yellow, and green respectively. Bolded scaled scores reflect the lowest and highest scores in each domain.

Individual domain performance

Domain 1

Domain 1 focuses on the intended goals of the guideline, including statements that cover the scope of the CPGs or the health questions and target populations considered in the development of the CPGs.

AAA CPGs on average performed well in this domain (mean score 72.2%±12.8%) with all CPGs scoring over 50% in the scaled scores. CPGs that performed better in this domain laid out clear review questions, specific objectives and defined precisely the patient population that the CPG was intended for. For example, the European Society for Vascular Society CPG suggested that their CPG not only apply to patients with AAAs, but also to those with iliac artery aneurysms. Additionally, this CPG defined that the recommendations would include patients with ‘juxtarenal AAA, isolated iliac aneurysms, mycotic and inflammatory aneurysms and concomitant malignant disease’.11 Poorer performing CPGs had either unclear objectives such as ‘critically reviewing the indications and the surgical results in the treatment of several aortic diseases’13 or failed to address any of the statements in their publication.

Domain 2

Domain 2 evaluates whether the CPG was developed by relevant stakeholders, which is essential for multidisciplinary input. It also considers the various aspects of healthcare provision offered by different healthcare professionals and the perspectives of the intended users of the CPG.

While CPGs performed adequately, there was greatest heterogeneity in scores for this domain (mean score 54.5%±23.5%). This was largely due to the variability in team members that may be involved in the management of AAAs. For example, the Society of Interventional Radiology guideline scored poorly due to the sole involvement of radiologists in the CPG development process. While this CPG arguably was targeted at patients undergoing endovascular aneurysm repair, other medical professionals are involved in patients’ care pre-endovascular and postendovascular aneurysm repair.8 In this aspect, the National Institute for Health and Care Excellence (NICE) CPG performed well, involving paramedics, general practitioners, vascular nurses, anaesthetists, radiologists and vascular surgeons in their guideline committee.12 With reference to patient involvement in particular, the European Society for Vascular Society7 did especially well to involve AAA patients in focus groups and obtain feedback on a plain English summary of the recommendations.

Domain 3

Domain 3 contains eight statements which appraise the evidence-based rigour of the literature search methodology, evidence selection and evaluation, as well as the procedure followed in the formulation of recommendations, together with the processes in place to permit guideline updating.

CPGs had acceptable scores in this domain, averaging 65.2%±18.4%. CPGs that performed poorly failed to describe the literature search methodology or the formulation of the methods involved in the development of their recommendations. The CPGs that performed well provided detailed information regarding a systematic evidence search (eg, databases, inclusion/exclusion criteria), recommendation formulation methodology (eg, Delphi consensus), and the avenues for feedback prior to CPG publication. For example, the NICE committee published a draft document which was freely available online, allowing any registered stakeholder to provide their comments on the draft recommendations.12 In addition to systematic reviews answering specific questions on the best modality and optimal frequency for surveillance after endovascular aneurysm repair, the Society for Vascular Surgery CPG also performed an ‘umbrella’ systematic review to synthesise evidence from previous reviews. This CPG also clearly outlined the databases used in their evidence search and provided the full search strategy in their online supplemental material.

Domain 4

Domain 4 evaluates the organisation of the guideline, discussing language and format. This domain includes statements assessing the ambiguity of included recommendations and the ease for CPG users to identify important recommendations.

CPGs performed very well in this domain (mean score 81.0%±20.1%), with four CPGs achieving a scaled score of >90%.7 9–11 These CPGs all provided clear summaries of their recommendations, highlighting important recommendations that should be implemented. Consistent formatting was used to draw readers’ attention to each recommendation, either with the recommendations placed in tables9–11 or clear signposting of recommendations followed by the evidence related to them.7 These CPGs also did well in placing their key recommendations in a summary section at the very start of the document.9 12 CPGs that performed poorly in this domain failed to highlight important recommendations, with recommendation statements embedded within the main text and making it difficult for readers to quickly identify recommendations.8 13

Domain 5

Domain 5 assesses how guideline developers consider the translation of recommendations into clinical practice. These statements include the consideration of barriers and facilitators to implementation, uptake improvement strategies and resources required for implementation.

CPGs fared worst in this domain, with the lowest average scores recorded (mean score 47.1%±21.2%). One method to identify potential resource limitations would be to implement a pilot implementation—this strategy, unfortunately, was not used by any CPG development group in this review. Alternatively, cost–benefit analysis could be very useful in estimating the economic impact of recommendation implementation. Cost–benefit analysis performed in other reviews were considered in the NICE and Society for Vascular Surgery CPGs.9 12 Furthermore, these CPGs also provided potential research and audit questions to improve the current evidence base and service provision.9 12 The lack of pilot implementation programmes or cost–benefit analysis resulted in poorer scores in the Society of Interventional Radiology8 and European Society of Cardiology10 CPGs.

Domain 6

Domain 6 includes statements defining competing interests, both at the individual (declaration of individual CPG development group members’ conflicts of interest) and institutional level (funding bodies’ involvement).

In this domain, CPGs performed adequately and achieved an average scaled score of 66.2%±22.7%. CPGs that performed well provided unequivocal statements that described the conflicts of interest of CPG development group members7 12 and, if any, the involvement of funding agencies. One CPGs failed to include any statement of such a nature,13 which is an easily corrected omission in future versions of this CPGs.

Overall guideline assessment

The Overall Guideline Assessment consists of two components: the overall quality rating of the CPG (rated from 1 to 7), and whether the reviewer would recommend use in clinical practice. Based on the predefined criteria of a scaled score of >80%, four guidelines9–12 were rated of adequate methodological quality for use in clinical practice. These guidelines were from the European Society of Cardiology,10 the Society of Vascular Surgery,9 the European Society of Vascular Surgery,11 and the NICE.12

Discussion

This review has employed the AGREE II instrument in the assessment of the AAA CPGs, which is a widely used and validated methodological assessment tool that also acts as a framework for CPG development. The strength of the assessment was improved by the inclusion of five reviewers at various levels of academic and clinical training, and good inter-reviewer reliability was achieved. It must be noted that this assessment was limited by the small number of CPGs that were available in the literature, which is surprising given the clinical burden that AAAs represent.

AAA-related mortality is considerable, ranked as the 12th–15th cause of death in the USA, UK and various European countries.14 Mortality post-AAA rupture can exceed 80% and contributes to over 44.6 deaths per 100 000 population in the UK.15 It is thus of utmost importance that CPGs detailing recommendations related to the diagnosis and management of this critical condition be of good quality, robust evidence base and easy access. Quality in CPGs stems from methodological rigour, and four CPGs included in this review were considered of adequate methodological quality for use in clinical practice. However, methodological rigour in each domain is independent and therefore, independent efforts are essential to correct the issues identified. This may be assisted by the multiple guideline development frameworks that exist, including the AGREE II instrument, G-I-N Standards,16 or Guidelines 2.0.17 To reduce heterogeneity and improved methodological rigour in CPGs, the international community should agree on a specific framework for use in future CPG development.

While no clear characteristic significantly impacted on the AGREE II scores, a good domain 1 performance appeared to set a higher standard for other domain scores. This may be due to having clear methodology outlined prior to the beginning of the CPG development process—clear objectives, specific clinical questions, specific patient populations on whom to apply the recommendations—which may set a framework for meticulous methodology in other domains. NICE, for example, employs an established methodology that is constant across CPGs in various subject matters, and these methods are codified in a manual that is used by development groups during the development process.18

It must be noted that in some CPGs a limited scope of individuals and specialties were involved in the creation of the guidelines. For example, as discussed above in the results for domain 2, the Society of Interventional Radiology CPG only included interventional radiologists in their CPG development group; consequently, the document focused on specific AAA management options (namely the various types of endovascular repair).8 This is reflected in lower scores in domain 2 as well as the overall scores of the AGREE II instrument. Inclusion of a more diverse panel in the CPG development group, on the other hand, resulted in broader and more holistic approaches to how AAAs should be managed, as seen in the NICE12 and European Society of Vascular Surgery11 CPGs.

Domain 5 was the poorest performing domain in this assessment, with four CPGs achieving a scaled score of <50%.7 9 10 13 This is not an isolated issue pertaining to AAA CPGs —the lack of consideration of the resources required to apply CPG recommendations has been seen in other vascular surgery topics including venous leg ulcers3 and lymphoedema,19 and other fields such as orthopaedics.2 Echoing the conclusions of these reviews, poor performance in this domain is especially concerning as recommendations are futile if not translated into clinical practice and applicable to the target populations. This is additionally important in the management of AAA given the variable costs that arise depending on the modality of treatment (ie, open vs endovascular repair). A recent observational study from the USA showed that while endovascular aneurysm repair was associated with lower admission and fixed costs when compared with open repair, this was outweighed by increases in variable hospitalisation costs associated with the procedure over time.20 It is clear, therefore, that resource allocation must be considered in the formulation of CPG recommendations. Recommendations should reflect the economic situation of the local population and not be universal management ideals that may be unattainable in the specific country or region. It would be ideal that development groups of future CPGs recognise this fact and perform local cost–benefit analyses or pilot implementation of CPGs to identify specific economic barriers unique to their healthcare system or population and adapt the CPG to overcome these issues.

It is important to note that while this study does not intend to examine the accuracy of the recommendations nor the robustness of the evidence behind them, it would nonetheless be remiss of the authors not to discuss issues with the current evidence. Most of the recommendations are based on historic RCTs, including the UK Endovascular Aneurysm Repair (EVAR)-121 and EVAR-222 among others.23–25 Evidence-based recommendations will therefore be limited by old data if trials were performed more than a decade ago. These trials need to be updated, especially considering the constantly evolving technologies increasingly employed in vascular surgery today, although recent registry data does seem to suggest that the findings of these trials currently hold true.26–28 CPGs, as an extension, should therefore also be constantly updated as newer trials are published. Certain groups have proposed using an online electronic wiki platform, allowing CPGs to become ‘living documents’ that can be updated in sections29 as new trial data becomes available.

The assessment of AAA guidelines via the AGREE II instrument has highlighted methodological inadequacies. Development groups of future versions of these CPGs should consider addressing these factors. Furthermore, while this review has provided a detailed methodological assessment of currently available AAA CPGs, readers should note that has not addressed the scientific accuracy of the recommendations.

Conclusion

In this methodological review of current AAA CPGs, four have been deemed adequate for clinical use; nonetheless, all have been shown to have shortfalls in their methodology. Future CPG iterations should consider that rigorous methodology can only be achieved through conscious effort. However, high methodological quality in existing CPG versions may not necessarily result in high-quality future versions. A structured approach is integral to an organised outcome; instruments to provide such a structure and thus boost methodological rigour are widely available and should be implemented by developmental groups to improve confidence in CPG rigour. This will, in turn, support the implementation of good evidence-based recommendations to improve the care of AAA patients internationally.