Prediction Models for Clinical Outcome After a Carotid Revascularization Procedure.

Background and Purpose- Prediction models may help physicians to stratify patients with high and low risk for periprocedural complications or long-term stroke risk after carotid artery stenting or carotid endarterectomy. We aimed to evaluate external performance of previously published prediction models for short- and long-term outcome after carotid revascularization in patients with symptomatic carotid artery stenosis. Methods- From a literature review, we selected all prediction models that used only readily available patient characteristics known before procedure initiation. Follow-up data from 2184 carotid artery stenting and 2261 carotid endarterectomy patients from 4 randomized trials (EVA-3S [Endarterectomy Versus Angioplasty in Patients With Symptomatic Severe Carotid Stenosis], SPACE [Stent-Protected Angioplasty Versus Carotid Endarterectomy], ICSS [International Carotid Stenting Study], and CREST [Carotid Revascularization Endarterectomy Versus Stenting Trial]) were used to validate 23 short-term outcome models to estimate stroke or death risk ≤30 days after the procedure and the original outcome measure for which the model was developed. Additionally, we validated 7 long-term outcome models for the original outcome measure. Predictive performance of the models was assessed with C statistics and calibration plots. Results- Stroke or death ≤30 days after the procedure occurred in 158 (7.2%) patients after carotid artery stenting and in 84 (3.7%) patients after carotid endarterectomy. Most models for short-term outcome after carotid artery stenting (n=4) or carotid endarterectomy (n=19) had poor discriminative performance (C statistics ranging from 0.49-0.64) and poor calibration with small absolute risk differences between the lowest and highest risk groups and overestimation of risk in the highest risk groups. Long-term outcome models (n=7) had a slightly better performance with C statistics ranging from 0.59 to 0.67 and reasonable calibration. Conclusions- Current models did not reliably predict outcome after carotid revascularization in a trial population of patients with symptomatic carotid stenosis. In particular, prediction of short-term outcome seemed to be difficult. Further external validation of existing prediction models or development of new prediction models is needed before such models can be used to support treatment decisions in individual patients.

In 2015, around 5.4 million patients worldwide suffered a first-ever ischemic stroke.[1] About 10% to 15% of these strokes are caused by thromboembolism from an atherosclerotic plaque at the carotid bifurcation or the internal carotid artery (ICA).[2] Such strokes can be prevented by surgical excision of the plaque with carotid endarterectomy (CEA) or by carotid artery stenting (CAS).[3] The benefit of carotid revascularization for an individual patient depends on the balance between the long-term risk of vascular complications on medical treatment and the periprocedural risk of these complications. Risk prediction models can be used to predict the absolute risk of periprocedural stroke or death after CEA or CAS in an individual patient.

Over the past decade, many prediction models for outcome after carotid revascularization have been developed. In a recent literature review, we identified 46 prediction models that can be applied to patients with symptomatic or asymptomatic ICA stenosis.[4] Only a few of these prediction models were validated in independent patient populations, although external validation is an essential step in prediction model development that should be performed before a model can be implemented in clinical practice.[5,6]

Therefore, we aimed to assess the external performance of prediction models for short-term and long-term outcome after carotid revascularization that were identified with our literature review. We selected prediction models that used readily available patient characteristics that were known before procedure initiation. We validated these models in patients with symptomatic ICA stenosis enrolled in randomized controlled trials comparing CAS versus CEA.

Methods

Selection of Prediction Models

In the literature review, we identified 46 prediction models for clinical outcome after CAS or CEA. Sixteen models were excluded from the current study; 10 of these models used procedural characteristics (eg, duration of procedure), 3 models predicted risk of myocardial infarction or other cardiac complications only, and 3 models did not contain sufficient information to validate the model (Figure). Details of the literature review and references of the excluded studies are provided in the online-only Data Supplement.

Flowchart of prediction models selected for external validation from previous literature review. Models for short-term outcome predict risk ≤30 d after the procedure; models for long-term outcome predict risk >30 d after the procedure. CAS indicates carotid artery stenting; CEA, carotid endarterectomy; and MI, myocardial infarction.

The 30 models we selected for external validation included 23 short-term outcome models (4 CAS and 19 CEA models) and 7 long-term outcome models (2 CAS, 4 CEA, and 1 combined model). We classified models that predicted in-hospital risk or risk up to 30 days after the procedure as short-term outcome models, whereas models that predicted risk after longer follow-up were classified as long-term outcome models. Tables I and II in the online-only Data Supplement provide an overview of the predictors used in each prediction model, the predicted outcome measure, and a description and key baseline characteristics of the development cohorts. The most often predicted outcome measure was a composite of stroke or death for the short-term outcome models and mortality for the long-term outcome models.

Validation Population

The validation population consisted of patients included in 4 randomized controlled trials that are pooled by the CSTC (Carotid Stenosis Trialists’ Collaboration)[7]: the EVA-3S (Endarterectomy Versus Angioplasty in Patients With Symptomatic Severe Carotid Stenosis) trial,[8] the SPACE (Stent-Protected Angioplasty versus Carotid Endarterectomy) trial,[9] the ICSS (International Carotid Stenting Study),[10] and the CREST (Carotid Revascularization Endarterectomy Versus Stenting Trial).[11] Collectively, these trials randomly allocated 4754 patients with symptomatic carotid artery stenosis of ≥50% according to the North American Symptomatic Carotid Endarterectomy Trial criteria[12] to undergo either CAS or CEA. Data on individual patient characteristics such as demographics, comorbidities, and degree of stenosis were collected at baseline in all 4 trials. Details on study design and methods are described elsewhere.[8-11] The CSTC welcomes other investigators to propose analyses of the pooled data. Such analyses could then be performed in close collaboration with the CSTC Steering Committee.

We excluded patients in whom the randomly allocated procedure was not completed or who suffered a stroke before procedure initiation. This resulted in 2184 patients who underwent CAS and 2261 patients who underwent CEA for analyses.

Outcome Measures

All patients in the validation population were followed up for the occurrence of stroke or death. Stroke was uniformly defined in all 4 trials as the occurrence of acute focal neurological deficit with symptoms that lasted for >24 hours caused by ischemic or hemorrhagic cerebrovascular disturbance.

We used stroke or death risk within 30 days after the procedure as the primary outcome for all short-term outcome models. In addition, we externally validated all short-term outcome models for the original outcome measure that was used in the model development study, if data on this outcome measure were available in our validation population. We externally validated all long-term outcome models for the original outcome measure. Three long-term outcome models predicted death occurring within 5 years after the procedure (Hoke 2012, Conrad 2013, Wallaert 2013); we externally validated these models with data from EVA-3S and ICSS only, because sufficiently long-term follow-up data from these 2 trials were available to the CSTC.

Statistical Analysis

We first matched predictors and outcome measures of each prediction model to variables in the validation population. Proxies were used in case a direct match was unavailable (Description of proxies is available in the online-only Data Supplement). A value of zero was assigned for predictors that were not available in any of the 4 trials (indicated with an asterisk in Table I in the online-only Data Supplement).

For most predictors, the proportion of patients with missing data within each trial was low (<2%), although more missing values occurred for antiplatelet use (49%) and for presence of contralateral ICA occlusion (4%). However, some predictors have not been measured in one or more of the 4 trials and are thus systematically missing in some of the trials (Table III in the online-only Data Supplement). Also, the occurrence of myocardial infarction during follow-up was not available in CREST. Missing data were imputed with the MICE package in R using source trial as predictive variable in the imputation model and creating 10 imputed data sets.

Per model, we applied the original regression formula to calculate the risk of the outcome for each patient in the validation population. In case the original regression formula was unavailable despite contacting the authors, and a risk score was reported in the original publication, we assessed the performance of this risk score.

Predictive performance of the models was analyzed with discrimination and calibration. Discrimination describes the ability of a model to distinguish between someone with and without the outcome and was examined with the C statistic. A C statistic of 0.5 reflects a model with no discriminative ability, and a value of 1.0 indicates perfect discriminative ability. For each prediction model, we calculated the C statistic in each trial separately and performed a random-effects meta-analysis of the results to account for heterogeneity between the 4 trials. Subsequently, we pooled the C statistics of each multiply imputed data set with Rubin’s rules.[13] Calibration indicates whether the predicted risks of the outcome correspond with the observed risks in the validation population. This was examined with calibration plots and calibration slopes. Differences between the incidence of the outcome in the validation population and the development cohorts are known to influence calibration. Therefore, we recalibrated the models for which we applied the original regression formula to the incidence of the outcome in the validation population by adjusting the original intercept (for logistic regression models) or baseline hazard (for Cox regression models).[14,15] Here, we present calibration of the models after recalibration. Calibration plots before recalibration are provided in Figure I in the online-only Data Supplement.

We performed 2 sensitivity analyses. First, we externally validated each prediction model in each of the 4 trials separately. Second, we externally validated each model in complete cases (ie, patients who had data on all variables available that were used by the prediction model; a value of zero was assigned for predictors that were not available in any of the 4 trials). Statistical analyses were performed with SPSS version 22 and R-3.3.2.

Results

Validation Population and Occurrence of Outcome

Stroke or death within 30 days after revascularization occurred in 158 (7.2%) of the patients who underwent CAS and in 84 (3.7%) patients who underwent CEA. Stroke or death within 1 year occurred in 248 (11.4%) patients after CAS and in 163 (7.2%) patients after CEA.

Baseline characteristics of the development and validation populations are presented in Tables II and III in the online-only Data Supplement. In both the development and validation populations, about two-thirds of patients were men and the mean age was 70 years. All patients in the validation population had symptomatic ICA stenosis, whereas only 2 models were specifically developed in symptomatic patients, 8 in asymptomatic patients, and 19 in both symptomatic and asymptomatic patients (Table II in the online-only Data Supplement).

We applied the original regression formula for 22 of the 30 models. Eight models did not report the original regression equation, and the authors were unable to provide additional information on request; therefore, we assessed performance of the risk scores.

CAS Short-Term Outcome Models

C statistics for the 4 short-term outcome models after CAS ranged from 0.55 to 0.64 for our primary outcome in the validation population and were consistently lower than the corresponding C statistics in the development cohorts (Table). The Hawkins 2012 model had the highest C statistic in the validation population. However, the calibration curves of all 4 models deviated from the ideal calibration slope for patients within the highest risk categories (Figure II in the online-only Data Supplement). Both age and history of stroke or symptomatic status were used as predictors in the 2 models with best model performance (Hawkins 2012, Wimmer 2012).

CEA Short-Term Outcome Models

For the 19 short-term outcome models after CEA, C statistics ranged from 0.49 to 0.60 for our primary outcome, which indicates poor discriminative performance (Table). The Bekelis 2013b model had reasonable calibration with a calibration slope (0.88) near the ideal value of 1.0. However, the curve was narrow with an absolute risk difference of only 5% between patients in the lowest and highest risk groups, similar to most other short-term outcome models after CEA. This indicates that these models cannot distinguish between patients with a low or high risk of the outcome (Figure II in the online-only Data Supplement). The Kuhan 2001 and Bennett 2015 models were probably overfitted to the original development population, which is reflected by the rather flat calibration curves with too low predictions for patients in the lowest risk categories and too high predictions for patients in the highest risk categories. The calibration curves of the other models deviated from the ideal calibration curve for predicted risks >5%. Most frequently used predictors in the models with most reasonable performance (Calvillo-King 2010a, Bekelis 2013b) were history of transient ischemic attack or stroke and history of coronary artery disease.

CAS or CEA Long-Term Outcome Models

We externally validated 7 prediction models for long-term outcome after CAS or CEA. C statistics ranged from 0.59 to 0.67 and were clearly higher compared with those of the short-term outcome models (Table). All calibration plots showed increasing observed risk of the outcome with increasing predicted risk (Figure II in the online-only Data Supplement). For the long-term outcome models after CAS, the Hoke 2012 model had the highest C statistic in combination with a reasonable calibration curve; for the long-term outcome models after CEA, the Wallaert 2013 model had best discrimination and calibration. Age, diabetes mellitus, and heart failure were used as predictors in both models with best external performance.

Sensitivity Analyses

External validation in each of the 4 trials separately yielded results comparable to our main analyses, although results were less precise. C statistics and calibration curves varied more between the trials for the prediction models after CEA than for the models after CAS, which may be caused by the low number of outcome events per trial after CEA (data not shown). Performance of the prediction models in a complete case analysis also showed results comparable to our main results (Table IV in the online-only Data Supplement).

Discussion

In this study, we assessed predictive performance of 30 existing prediction models for short-term and long-term outcome after carotid revascularization in a large population of patients with symptomatic ICA stenosis. Most prediction models for short-term outcome after CAS or CEA had poor ability to discriminate between patients who will and will not get an event. In addition, most calibration curves were narrow (ie, absolute risk difference between patients in the lowest and highest risk groups was small), and risk was overestimated in the highest risk categories. The long-term outcome models showed slightly better external performance compared with short-term outcome models, but most of them were originally developed to predict mortality. Hence, these prediction models cannot predict which patients have a higher long-term stroke risk despite CAS or CEA, and therefore, are not specific for outcome prediction after carotid revascularization.

Differences in study population characteristics between the development cohorts and the validation population may have contributed to the poor external performance of most models. Most models were developed in observational cohorts of both symptomatic and asymptomatic patients, whereas they were validated in a trial population of symptomatic patients. Trial populations are probably more homogenous and might have lower mean absolute complication risks than study populations from observational cohort studies because of strict inclusion and exclusion criteria and are therefore less representative for the total patient population undergoing CAS or CEA. Also, the proportion of patients with vascular risk factors or (cardiac) comorbidities might have been different in the validation population, which reduces usefulness of such characteristics to distinguish between someone with and without the outcome. However, most models showed only moderate discrimination in the development cohorts (Table); thus, we did not expect to find a model with excellent predictive performance in the validation population.

Another reason for the poor external performance could be that clinicians have already identified the high-risk patients for CAS or CEA and have avoided treating these patients. Consequently, the risk difference in the remaining eligible patients who undergo the intervention is small, which makes it difficult to identify a high-risk subset. It is mathematically impossible to assess this explanation because data on the outcome of patients that did not undergo CAS or CEA are absent. Nevertheless, our results suggest that among those eligible for CAS or CEA, identification of a high-risk subset is currently impossible.

Missing data in the validation population may be a third reason for the poor external performance. In the validation population, zeros were inserted for predictors that were unavailable. Also, proxies were used if no direct match was available in the validation population. Therefore, not all prediction models could completely be applied to the validation population, which may have influenced predictive performance adversely.

Although external validation is an essential step in prediction model development, only 4 of the 30 included models had been externally validated in an independent patient population before; 2 were externally validated in the same article in which their development was described (Alcocer 2013 and Cheng 2016),[16,17] and the other 2 had each been validated twice in independent validation studies (Rothwell 1999 and Tu 2003).[18-21] Our findings are similar to the findings of these previous studies, except that the discriminative performance of the Tu 2003 and Cheng 2016 models was slightly better in the previous external validations. A calibration plot was reported for only one of the previous external validation studies,[19] whereas we reported calibration plots for each prediction model that we validated.

Our study has several strengths. We performed an extensive external validation study of all existing prediction models for clinical outcome after carotid revascularization that we identified with a literature review. Moreover, all models were validated in the same study population, which allows for direct comparison of their predictive performance. Furthermore, multiple imputation was used to handle missing data, which is preferable to complete case analysis.[22]

Nevertheless, some limitations need to be mentioned. First, our validation population consisted of symptomatic ICA stenosis patients, whereas the majority of the models was developed for both symptomatic and asymptomatic patients. Symptomatic and asymptomatic patients have a different absolute complication risk after CAS or CEA and have different characteristics that may influence this risk (eg, type of most recent ipsilateral ischemic event, time between most recent event and procedure). Moreover, study populations that consist of both symptomatic and asymptomatic patients have more variation in patient characteristics and absolute complication risk compared with our validation population. External validation of prediction models developed for asymptomatic patients or both symptomatic and asymptomatic patients in a population of symptomatic patients may have reduced predictive performance of these models. Second, the amount of systematically missing predictors was considerable in some of the individual trials. However, we performed multiple imputation to account for these missing variables. Moreover, predictive performance in the total population after multiple imputation was comparable to the performance in the complete case analysis. Third, some variables were unavailable in the validation population and were assigned a value of zero for all patients; consequently, these variables could not add to the predictive ability of the models. Last, we could not externally validate 6 prediction models, because insufficient information was reported or insufficient data were available to externally validate these models.

The aim of this study was to assess external performance of prediction models for clinical outcome after carotid revascularization based on readily available patient characteristics. Yet, none of the prediction models showed good discrimination and consistently good calibration in our validation population, whereas calibration is especially important if the aim of the model is to identify patients at different levels of risk. Consequently, we cannot recommend the use of any of these models in clinical practice. External validation of the prediction models in other study populations is demanded, preferably in observational cohorts of patients undergoing CAS or CEA and with sufficient information available on predictors and outcomes. In case external performance of the existing prediction models in other study populations would remain poor, new prediction models for short-term outcome after CAS or CEA need to be developed considering the most important predictors from our previous literature review[4] as candidate predictors. Such models may eventually be combined with long-term outcome models to estimate absolute risk of postprocedural complications after carotid revascularization and optimize decision-making between CAS and CEA in individual patients.

Conclusions

In this external validation study, prediction models with readily available patient characteristics poorly predicted short-term and long-term outcome after carotid revascularization in a trial population of patients with symptomatic carotid stenosis. Further external validation of existing prediction models or development of new prediction models—preferably in a more heterogeneous study population with sufficient information on predictors and outcomes—is needed to accurately estimate risks after CAS or CEA in individual patients with carotid stenosis.

Acknowledgments

Drs Volkers, Greving, Kappelle, and Algra designed the study plan. Drs Volkers, Greving, and Algra undertook the statistical analysis and interpreted the results together with Dr Kappelle. Dr Mas, Dr Ringleb, Dr Bonati, and G. Howard extracted individual patient data from the contributing trials. Dr Volkers wrote the first version of the article. All authors contributed to data interpretation, critical revision of the article, and approved the final version. All authors gave final approval to submit for publication.

Sources of Funding

This study was funded by the Rudolf Magnus Young Talent Fellowship (University Medical Center Utrecht; grant to Dr Greving). Dr Halliday’s research is funded by the National Institute for Health Research, Oxford Biomedical Research Center. Dr Bonati’s research was funded by the Swiss National Science Foundation (grant 32003B_156658). G. Howard and Dr Brott are funded by the National Institute of Health/National Institute of Neurological Disorders and Stroke.

Disclosures

None.

Table.

Discrimination (C Statistic) of Each Prediction Model in the Validation Population and in the Original Model Development Study