Comparison of three contemporary risk scores for mortality following elective abdominal aortic aneurysm repair.

OBJECTIVE/
BACKGROUND: A number of contemporary risk prediction models for mortality following elective abdominal aortic aneurysm (AAA) repair have been developed. Before a model is used either in clinical practice or to risk-adjust surgical outcome data it is important that its performance is assessed in external validation studies.
METHODS: The British Aneurysm Repair (BAR) score, Medicare, and Vascular Governance North West (VGNW) models were validated using an independent prospectively collected sample of multicentre clinical audit data. Consecutive, data on 1,124 patients undergoing elective AAA repair at 17 hospitals in the north-west of England and Wales between April 2011 and March 2013 were analysed. The outcome measure was in-hospital mortality. Model calibration (observed to expected ratio with chi-square test, calibration plots, calibration intercept and slope) and discrimination (area under receiver operating characteristic curve [AUC]) were assessed in the overall cohort and procedural subgroups.
RESULTS: The mean age of the population was 74.4 years (SD 7.7); 193 (17.2%) patients were women and the majority of patients (759, 67.5%) underwent endovascular aneurysm repair. All three models demonstrated good calibration in the overall cohort and procedural subgroups. Overall discrimination was excellent for the BAR score (AUC 0.83, 95% confidence interval [CI] 0.76-0.89), and acceptable for the Medicare and VGNW models, with AUCs of 0.78 (95% CI 0.70-0.86) and 0.75 (95% CI 0.65-0.84) respectively. Only the BAR score demonstrated good discrimination in procedural subgroups.
CONCLUSION: All three models demonstrated good calibration and discrimination for the prediction of in-hospital mortality following elective AAA repair and are potentially useful. The BAR score has a number of advantages, which include being developed on the most contemporaneous data, excellent overall discrimination, and good performance in procedural subgroups. Regular model validations and recalibration will be essential.

There are currently no widely used risk prediction models in elective abdominal aortic aneurysm (AAA) repair. This study validates three risk prediction models for in-hospital mortality. Overall, all models demonstrated good performance and are potentially useful. The British Aneurysm Repair (BAR) score, which was developed using the UK National Vascular Database, has a number of advantages over the other models, including being developed on the most contemporaneous data, excellent discriminatory ability overall, and good performance in procedural subgroups. Validated risk prediction models such as the BAR score can facilitate clinical decision making, improve informed consent, and risk-adjust elective AAA repair outcome data.

Introduction

Risk prediction models are an important aspect of contemporary surgical practice. They can be used to provide information on the risks of surgery, to guide clinical decision-making, and to risk-adjust clinical outcome data. Accurate risk prediction models are particularly important for elective abdominal aortic aneurysm (AAA) repair. First, because most patients with AAA are asymptomatic, with an increasing number of patients identified through screening programmes, accurate estimates of procedural risk are important for optimising clinical decision-making. Second, named individual surgeon outcomes following elective AAA repair have recently been published in the UK. Risk-adjusting published surgical outcome data is essential if fair comparisons between surgeons and hospitals are to be made, and inappropriate risk-averse clinical decisions avoided.

A number of risk prediction models have previously been developed for AAA repair. However, unlike in cardiac surgery, where risk prediction models have been widely accepted and utilised in clinical practice, risk models have never been widely used in AAA repair. Reasons for this lack of adoption include doubts about the accuracy and applicability of models to contemporary practice, lack of vigorous model validation, the inability to easily perform model calculations and uncertainty over how such models might be used in clinical practice.

To address the lack of suitable risk models, a number of models have recently been developed, including the British Aneurysm Repair (BAR) score, and the Medicare and Vascular Governance Northwest (VGNW) models.4, 5, 6 All three models have been developed specifically for elective AAA repair and have demonstrated potential accuracy in validation studies performed to date.7, 8, 9 Surgical risk models can lose accuracy over time as practices change and outcomes improve. Therefore, if a model is to be recommended for use in clinical practice, it is important that it has been validated and found to be accurate for contemporary practice. Therefore, the objective of this study was to perform a contemporary prospective validation of the Medicare, VGNW, and BAR risk models.

Materials and Methods

This was a prospective, multicentre study conducted on behalf of VGNW, a peer-led clinical governance programme that audits the results of vascular surgeons across the north-west of England and Wales. Data were collected on consecutive elective AAA repairs performed between April 2011 and March 2013. As only pseudonymous, nonidentifiable data were used for this study, ethical approval was not required. Elective procedures were defined by the timing of surgery rather than the mode of admission. Data collected included patient demographics, comorbidities, preoperative medications, preoperative investigations, procedural details and clinical outcomes.

Patient characteristics were defined as follows: ischaemic heart disease included a history of angina, previous myocardial infarction or previous coronary revascularisation; respiratory disease was defined as the presence of chronic respiratory disease with dyspnoea on exertion or at rest; diabetes included any patient receiving treatment for diabetes (diet controlled, noninsulin- and insulin-dependent). Medications were recorded if they were being taken by the patient on admission for surgery. Abnormal electrocardiogram was recorded if there was evidence of any of the following; atrial fibrillation or any other abnormal rhythm; > 5 ectopic beats/minute; Q or ST/T wave changes. Abnormal laboratory investigations were defined as serum sodium <135 or >145 mmol/L; serum potassium <3.5 or >5.5 mmol/L; urea >7.5 mmol/L; white cell count <3.0 × 109/L or >11.0 × 109/L; and haemoglobin <11 g/dL for women and <13 g/dL for men. Activity and outcome data for the VGNW programme are validated against Hospital Episode Statistic data by the VGNW team, with risk factor data also validated where possible.

Data were cleaned by removing duplicate records, correcting transcriptional discrepancies, and resolving any clinical or temporal conflicts. Missing data were imputed with the sample median for continuous or ordinal variables, and the mode for dichotomous variables. Thoraco-abdominal or isolated iliac aneurysm repairs were excluded from the analysis. The primary outcome measure for the study was in-hospital mortality, defined as death due to any cause during admission for elective AAA repair. Following risk factor imputation, the Medicare, VGNW, and BAR scores were calculated for each record. Risk factors included in each model are shown in Table 1.

Table 1

Risk factors included in the British Aneurysm Repair (BAR) score, Medicare model, and Vascular Governance Northwest (VGNW) model.

Bar score	Medicare model	VGNW model
Open repair	Open repair	Open repair
Age (continuous)	Age (grouped)	Age (continuous)
Female sex	Female	Female
Creatinine >120 μmol/L	Chronic renal disease	Creatinine (continuous)
Cardiac disease	End-stage renal disease	Diabetes
Abnormal ECG	Cardiac failure	Antiplatelet medication
Previous aortic surgery/stent	Vascular disease	Respiratory disease
Abnormal white cell count
Abnormal sodium
AAA diameter (cm)
ASA grade (I–IV)

Note. ECG = electrocardiogram; AAA = abdominal aortic aneurysm; ASA = American Society of Anesthesiologists.

Model performance was assessed using measures of calibration and discrimination in the overall cohort and separately in both procedural and gender subgroups. Discrimination was evaluated by determining the receiver operating characteristic (ROC) curve, which is summarized by the area under the curve (AUC) alongside an approximate 95% confidence interval (CI). An AUC of ≥0.9 is defined as “outstanding”, an AUC of 0.8–0.9 is considered “excellent”, an AUC of 0.7–0.8 represents “acceptable” discrimination, and an AUC of ≤0.5 represents no discrimination.

In the overall cohort and procedural subgroups, model calibration has been summarised by calculating the observed to expected (O:E) ratio and an exact 95% CI. If, on average, the model is well calibrated, then the O:E ratio should be close to 1. If the O:E is above or below 1 this indicates under- and over-prediction respectively. Model calibration has been further assessed in the overall cohort by dividing the cohort into low (bottom 50%), medium (middle 25%), and high-risk (top 25%) groups based on the model's predicted mortality. Model performance statistics were calculated for each risk group. Calibration plots for each model based on these groups were produced showing the mean predicted probability of outcome against the observed proportion of outcomes. Approximate 95% CIs for the observed mortality proportions in each group are shown as error bars in the figures.

The calibration intercepts and slope parameters were also calculated for each model in the overall cohort. The calibration intercept and slope are calculated by fitting a logistic regression model with the dependent variable set as the observed outcome and the independent variable set as the log-odds (“logit”) transformed model prediction. If the model was perfectly calibrated the intercept and slope would equal 0 and 1 respectively. The intercept is a measure of overall model calibration, which is whether predictions agree on average with observed probabilities. A chi-square test (on 2 degrees of freedom) for “unreliability” (null hypothesis: intercept = 0 and slope = 1) was also performed. All p-values <.05 were considered significant. All statistical analyses were performed using R version 3.0.2.

Results

Patient characteristics

Data were available for 1,142 procedures performed by 60 surgeons at 17 different hospitals. Sixteen procedures were excluded as they related to either thoracic or isolated iliac aneurysm repairs, with two further records excluded as there was evidence that the procedure was nonelective. This resulted in a final cohort for analysis of 1,124 elective AAA repairs. The mean age of the population was 74.4 years (SD 7.7); 193 (17.2%) patients were women. The majority of patients (759, 67.5%) underwent endovascular aneurysm repair (EVAR), and most patients (1,037, 92.3%) were asymptomatic. Additional patient characteristics are given in Table 2.

Table 2

Characteristics of the study population.

Risk factor	Frequency (%)	Missing data (%)
Age (years)a	74.4 (7.7)	0.2
Female sex	193 (17.2)	0.0
AAA diameter (cm)a	6.3 (1.2)	6.0
Previous aortic surgery/stent	67 (6.0)	1.3
AAA symptoms	87 (7.7)	6.1
Ischaemic heart disease	357 (31.8)	8.5
Previous myocardial infarction	163 (14.5)	8.5
Cardiac failure	27 (2.4)	9.2
Respiratory disease	199 (17.7)	14.5
Diabetes	162 (14.4)	3.2
Antiplatelet medication	716 (63.7)	0.5
Antihypertensive medication	371 (33.0)	0.5
Statin therapy	778 (69.2)	0.4
Smoking status	–	12.2
Ex-smoker	298 (26.5)	–
Current smoker	246 (21.9)	–
Abnormal ECG	359 (31.9)	5.7
Abnormal sodium	105 (9.3)	11.0
Abnormal potassium	36 (3.2)	10.4
Abnormal urea	308 (27.4)	10.4
Creatinine >120 μmol/L	186 (16.5)	10.1
Creatinine >200 μmol/L	21 (1.9)	10.1
Abnormal WCC	82 (7.3)	9.0
Abnormal haemoglobin	296 (26.3)	9.0
ASA grade	–	11.4
I	56 (5.0)	–
II	434 (38.6)	–
III	604 (53.7)	–
IV	30 (2.7)	–
Open repair	365 (32.5)	0.0

Note. AAA = abdominal aortic aneurysm; ECG = electrocardiogram; WCC = white cell count; ASA = American Society of Anesthesiologists.

Continuous data displayed as mean (SD).

Overall model calibration

There were 32 in-hospital deaths in the study cohort, giving an overall mortality of 2.8%. The Medicare predicted mortality was 2.8%, which was not significantly different from the observed mortality, giving an O:E of 1.02 (95% CI 0.70–1.44, p = .858). The VGNW predicted mortality was 3.2%, giving an O:E ratio of 0.89 (95% CI 0.61–1.25, p = .560) and the BAR score predicted mortality was 2.4%, giving an O:E ratio of 1.19 (95% CI 0.81–1.68, p = .334). Calibration plots for the three models are shown in Fig. 1. The calibration intercepts and slopes also demonstrated good calibration for all three models, with each unreliability statistic being insignificant (Medicare p = .332, VGNW p = .756, BAR p = .581). Model performance in the low-, medium-, and high-risk groups, defined based on the probabilities calculated by each model, is shown in Table 3, Table 4, Table 5.

Figure 1

Calibration plots for low-, medium-, and high-risk groups for the British Aneurysm Repair (BAR), Medicare, and Vascular Governance Northwest (VGNW) risk models. The black dashed line is the line of equality that represents perfect calibration. Vertical lines represent approximate 95% binomial confidence intervals of the observed mortality proportion.

Table 3

Calibration of the Medicare model for low-, medium-, and high-risk groups; groups were defined according to the Medicare model predicted mortality.

	Low risk	Medium risk	High risk
Predicted mortality ranges	0 < x ≤ 2.084	2.084 < x ≤ 3.230	3.230 < x ≤ 32.519
Number of patients, n	611	247	266
EVAR, n	534	120	105
Open AAA repair, n	77	127	161
Observed mortality rate, %	1.1	2.4	7.1
Expected mortality rate, %	1.3	2.8	6.1
O:E ratio, 95% CI	0.85 (0.34–1.76)	0.88 (0.32–1.91)	1.16 (0.70–1.82)
p	0.861	1.000	0.457

Note. EVAR = endovascular aneurysm repair; AAA = abdominal aortic aneurysm; O:E = observed to expected; CI = confidence interval.

Table 4

Calibration of the Vascular Governance Northwest (VGNW) score for low-, medium-, and high-risk groups; groups were defined according to the VGNW score predicted mortality.

	Low risk	Medium risk	High risk
Predicted mortality ranges	0 < x ≤ 2.089	2.089 < x ≤ 3.817	3.817 < x ≤ 62.330
Number of patients, n	562	281	281
EVAR, n	487	184	88
Open repair, n	75	97	193
Observed mortality rate, %	1.1	1.8	7.5
Expected mortality rate, %	1.2	2.8	7.6
O:E ratio (95% CI)	0.88 (0.32–1.92)	0.63 (0.20–1.46)	0.99 (0.61–1.51)
p	1.000	0.375	1.000

Note. EVAR = endovascular aneurysm repair; O:E = observed to expected; CI = confidence interval.

Table 5

Calibration of the British Aneurysm Repair (BAR) score for low-, medium-, and high-risk groups; groups were defined according to the BAR score predicted mortality.

	Low risk	Medium risk	High risk
Predicted mortality ranges	0 < x ≤ 1.245	1.245 < x ≤ 2.540	2.540 < x ≤ 44.335
Number of patients, n	562	281	281
EVAR, n	496	196	67
Open repair, n	66	85	214
Observed mortality rate, %	0.4	1.8	8.9
Expected mortality rate, %	0.7	1.8	6.4
O:E ratio (95% CI)	0.52 (0.06–1.88)	1.00 (0.32–2.33)	1.38 (0.89–2.04)
p	0.603	1.000	0.124

Note. EVAR = endovascular aneurysm repair; O:E = observed to expected; CI = confidence interval.

Overall model discrimination

The BAR score demonstrated excellent discrimination in the overall cohort, with an AUC of 0.83 (95% CI 0.76–0.89). The discriminative ability of the Medicare and VGNW models was acceptable, with AUCs of 0.78 (95% CI 0.70–0.86) and 0.75 (95% CI 0.65–0.84) respectively. The ROC curves for the models in the overall cohort are shown in Fig. 2.

Figure 2

Receiver operating characteristic curves for the British Aneurysm Repair (BAR), Medicare, and Vascular Governance Northwest (VGNW) risk models in the overall cohort. The grey line represents the line of equality.

Model performance in separate open AAA repair and EVAR groups

The in-hospital mortality rates for open AAA repair and EVAR were 6.8% and 0.9% respectively. The predicted mortality in the open AAA repair group was 4.4% (O:E 1.56 [95% CI 1.01–2.30], p = .033), 5.4% (O:E 1.26 [95% CI 0.81–1.86], p = .260), and 4.9% (O:E 1.41 [95% CI 0.91–2.08], p = .095) for the Medicare, VGNW, and BAR scores respectively. In the EVAR group, the predicted mortality was 2.0% (O:E 0.46 [95% CI 0.18–0.94], p = .029), 2.1% (O:E 0.43 [95% CI 0.17–0.89], p = .017), and 1.2% (O:E 0.76 [95% CI 0.31–1.57], p = .619) for the Medicare, VGNW, and BAR scores respectively.

In the open AAA repair subgroup again only the BAR score demonstrated acceptable discrimination, with an AUC of 0.70 (95% CI 0.61–0.78). Both the Medicare and VGNW models demonstrated unacceptable discrimination, with AUCs of 0.68 (95% CI 0.58–0.78) and 0.64 (95%CI 0.53–0.75). In the EVAR subgroup only the BAR score demonstrated acceptable discrimination, with an AUC of 0.75 (95% CI 0.55–0.95). Both the Medicare and VGNW models demonstrated unacceptable discrimination with AUCs of 0.66 (95% CI 0.47–0.85) and 0.56 (95% CI 0.31–0.81) respectively.

Model performance by sex

The in-hospital mortality for men was 2.8% and for women was 3.1%. The predicted mortality for men was 2.4% (O:E 1.14 [95% CI 0.75–1.67], p = .462), 2.6% (O:E 1.08 [95% CI 0.71–1.59], p = 0.682), and 1.9% (O:E 1.47 [95% CI 0.96–2.16], p = .055) for the Medicare, VGNW and BAR scores respectively. For women, the predicted mortality was 4.5% (O:E 0.70 [95% CI 0.26–1.52], p = 0.494), 6.2% (O:E 0.50 [95% CI 0.18–1.09], p = .109), and 4.8% (O:E 0.64 [95% CI 0.24–1.40], p = .408) for the Medicare, VGNW and BAR scores respectively. For men, the BAR score demonstrated excellent discriminatory ability, with an AUC of 0.85 (95% CI 0.78–0.92). Both the Medicare and VGNW models demonstrated acceptable discrimination with AUCs of 0.78 (95% CI 0.69–0.86) and 0.76 (95% CI 0.65–0.87) respectively. For women, the Medicare model demonstrated excellent discriminatory ability with an AUC of 0.88 (95% CI 0.76–0.99). The BAR score and VGNW model both demonstrated acceptable discrimination with AUCs of 0.79 (95% CI 0.67–0.92) and 0.76 (95% CI 0.53–0.99) respectively.

Discussion

We evaluated three contemporary risk prediction models for in-hospital mortality following elective AAA repair, and demonstrated that all three models are potentially useful. The BAR score, developed using data from the UK National Vascular Database demonstrated excellent discriminatory ability overall and retained acceptable discriminatory ability in separate open AAA repair and EVAR cohorts. Although the Medicare and VGNW models demonstrated acceptable discrimination overall for elective AAA repair, discrimination was inadequate for open AAA repair and EVAR separately. All three models demonstrated good calibration in the overall cohort. However, only the BAR score demonstrated acceptable calibration in both procedural subgroups.

As with any registry study, data quality and completeness are inevitable limitations; however, no variable used was missing in >15% of cases. All required variables were available as defined for the calculation of the BAR score and VGNW model; however, for the Medicare model, some risk factors had to be inferred from related data as the risk factors as defined were not collected by the VGNW programme for the study period. A history of vascular disease was presumed for patients identified as taking preoperative antiplatelet medication. This assumption is likely to have led to an overestimation in risk for the Medicare model, as only ∼31% of patients had vascular disease in the Medicare cohort compared with the assumed proportion in this study of 61.7%. Chronic renal disease was assumed for any patient with a creatinine >120 μmol/L and end-stage renal disease assumed for any patient with a creatinine >200 μmol/L. This assumption led to similar proportions of patients being classified as having renal disease in this study as in the cohort used for development of the Medicare model.

This study was based on prospectively collected, validated clinical audit data from multiple centres throughout the north-west of England and Wales. A comprehensive assessment of each model's performance with regard to discrimination, calibration, and clinical validity has been performed. Good model discrimination is important, as this tends to remain stable over time, and poor discriminatory performance is difficult to correct.10, 14 Adequate calibration of a model is, perhaps, more essential for clinical decision-making and risk adjustment of surgical outcome data. Unlike poor discrimination, if a model is poorly calibrated this can potentially be addressed using a variety of methods. Although good calibration was demonstrated overall, both the Medicare and VGNW models demonstrated over-prediction of risk in patients undergoing EVAR, and the Medicare model also under-predicted risk for open AAA repair.

A trend towards under-prediction of risk in high-risk patients was demonstrated by the Medicare model and BAR score, while all three models tended to under-predict risk in men and over-predict risk in women. However, a limitation of this study is that the relatively small numbers of outcomes, particularly for the subgroup analyses, may have led to substantial differences in model performance not being detected. It is necessary that these models be tested on larger datasets in order to confirm their performance in specific subgroups. The number of outcomes available also meant it was not possible to perform an overall Hosmer–Lemeshow test.

Clinical validity of the models is suggested by the risk factors that are included in all three models, including open repair, age, female sex, and renal disease. Cardiac disease is included in both the BAR score and Medicare model, and vascular disease (identified in the VGNW model by use of antiplatelet medication) is present in the Medicare and VGNW models. Diabetes and respiratory disease are exclusive to the VGNW model, while previous aortic surgery or stent, abnormal white cell count, abnormal sodium, AAA diameter (cm), and American Society of Anesthesiologists grade (I–IV) are exclusive to the BAR score. Potential reasons for differences in risk factors between the models include the size of the dataset available for model development, the availability of each risk factor, differences in the time interval of data collection, difference in the populations studied between development datasets and differences in risk factor definitions.

In addition to differences in risk factors included in the models there are also differences in the outcomes the models were designed to predict. The BAR score was developed to predict in-hospital mortality, whereas the VGNW model was designed to predict 30-day mortality. The Medicare model was developed to predict either in-hospital mortality or 30-day mortality. These differences in outcome definitions are unlikely to affect model performance significantly as in this study no deaths occurred after discharge, but within 30 days of the procedure. Clearly, perioperative mortality is an important outcome for patients, surgeons, and healthcare providers, and is also the outcome currently used for publication of elective AAA repair outcome data. However, other important outcomes, such as re-intervention, long-term survival, and aneurysm-related mortality are key indicators of quality and may be used in the publication of surgical outcome data going forward.

Both the VGNW and Medicare models have previously been validated using multicentre and single-centre data and found to perform acceptably with regard to discrimination and calibration.7, 8, 9 This study represents the second successful external validation of the BAR score, with the previous validation performed using data from a randomised clinical trial. All the models studied have been developed within the last 5 years; however, the development datasets cover significantly different time periods, which may account for differences in model performance. The datasets used to develop the VGNW model and Medicare model are relatively out of date with respect to modern vascular surgical practice represented in this cohort. With regard to the type of data used for model development, the BAR score and VGNW model were built using clinical datasets, and the Medicare model was developed using administrative data. It has been suggested that using clinical datasets for risk model development is more appropriate than using administrative datasets.18, 19

Unlike some older models for AAA repair,20, 21, 22 all three models were based on datasets that included exclusively AAA repairs (both open repairs and EVAR). Although the BAR score demonstrates acceptable performance for separate EVAR and open cohorts, it is possible that models designed specifically for either open repair or EVAR may have improved performance for these procedures separately. However, a combined model approach is potentially more suitable for clinical practice as it allows the easy calculation of the risk of both procedures and is more appropriate for risk-adjustment purposes.

Although all three models validated here are potentially useful, in our opinion the BAR score is the most appropriate for informing clinical decision-making and risk-adjusting surgical outcome data in elective AAA repair. Although, overall, a statistically significant superior performance of the BAR score has not been demonstrated, it was the only model to retain acceptable discrimination and calibration in both EVAR and open repair subgroups. The BAR score is also based on the most contemporary data, and can be easily accessed and calculated in <30 seconds using either a website (www.britishaneurysmrepairscore.com) or an App (via the Apple App Store or Google Play).

When using a model in clinical decision-making it is important that the clinician should (i) understand that the risk prediction model does not predict the outcome for an individual patient, but provides an estimate of the risk for a population of patients with similar characteristics undergoing the same procedure; (ii) understand any potential limitations of the model; (iii) know how their own performance may influence the prediction; and (iv) adjust the prediction based on important risk factors not captured in the model. As risk prediction models tend to lose calibration over time, it is vital that models are validated every 2–3 years and recalibrated. This approach allows new risk factors to be incorporated into updated models to potentially further improve model performance. Dynamic modelling is an alternative approach that could be adopted to address calibration, but would require improved data collection and informatics infrastructure.

Funding

This project was partly funded by the National Institute for Health Research Health Technology Assessment (NIHR HTA) programme (project no. 09/91/39) and will be published in full in Health Technology Assessment. The views and opinions expressed herein are those of the authors and do not necessarily reflect those of the HTA programme, NIHR, National Health Service, or Department of Health. The Vascular Governance Northwest programme is funded by the Supra District Audit Fund, via Bury Primary Care Trust.

Conflict of Interest

The authors declare no financial conflicts of interest. S.W.G. and C.N.M. were part of the team that developed the Vascular Governance Northwest and British Aneurysm Repair (BAR) scores. G.L.H. was part of the team that developed the BAR score.