Prediction models for clinical outcome after a carotid revascularisation procedure: A systematic review.

INTRODUCTION: Prediction models for clinical outcome after carotid artery stenting or carotid endarterectomy could aid physicians in estimating peri- and postprocedural risks in individual patients. We aimed to identify existing prediction models for short- and long-term outcome after carotid artery stenting or carotid endarterectomy in patients with symptomatic or asymptomatic carotid stenosis, and to summarise their most important predictors and predictive performance. PATIENTS AND METHODS: We performed a systematic literature search for studies that developed a prediction model or risk score published until 22 December 2016. Eligible prediction models had to predict the risk of vascular events with at least one patient characteristic.
RESULTS: We identified 37 studies that developed 46 prediction models. Thirty-four (74%) models were developed in carotid endarterectomy patients; 27 of these (59%) predicted short-term (in-hospital or within 30 days) risk. Most commonly predicted outcome was stroke or death (n = 12; 26%). Age (n = 31; 67%), diabetes mellitus (n = 21; 46%), heart failure (n = 16; 35%), and contralateral carotid stenosis ≥50% or occlusion (n = 16; 35%) were most commonly used as predictors. For 25 models (54%), it was unclear how missing data were handled; a complete case analysis was performed in 15 (33%) of the remaining 21 models. Twenty-eight (61%) models reported the full regression formula or risk score with risk classification. Twenty-one (46%) models were validated internally and 12 (26%) externally. Discriminative performance (c-statistic) ranged from 0.66 to 0.94 for models after carotid artery stenting and from 0.58 to 0.74 for models after carotid endarterectomy. The c-statistic ranged from 0.55 to 0.72 for the external validations. DISCUSSION: Age, diabetes mellitus, heart failure, and contralateral carotid stenosis ≥50% or occlusion were most often used as predictors in all models. Discriminative performance (c-statistic) was higher for prediction models after carotid artery stenting than after carotid endarterectomy.
CONCLUSION: The clinical usefulness of most prediction models for short- or long-term outcome after carotid artery stenting or carotid endarterectomy remains unclear because of incomplete reporting, methodological limitations, and lack of external validation.

Introduction

Extracranial internal carotid artery (ICA) stenosis of ≥50% or occlusion caused 12% of strokes in patients who were admitted to the hospital because of first-ever or recurrent ischaemic stroke.[1] Both carotid artery stenting (CAS) and carotid endarterectomy (CEA) reduce the risk of new ipsilateral stroke in patients with symptomatic ICA stenosis, but carry a periprocedural complication risk. The risk of stroke or death within 30 days after treatment is higher after CAS than after CEA, which is partly explained by the effect of age,[2] but the postprocedural rate of ipsilateral stroke does not differ between both treatments.[3] For patients with asymptomatic ICA stenosis, preferred treatment (carotid revascularisation or best medical treatment) remains unclear.[4]

Prediction of clinical outcome after CAS or CEA could aid physicians to estimate periprocedural and postprocedural risks and to choose the appropriate treatment in individual patients. Our aim was to identify existing prediction models or risk scores for short- or long-term outcome after CAS or CEA in patients with symptomatic or asymptomatic ICA stenosis from the literature, and to summarise their most important predictors and predictive performance. We focused on prediction models that included patient characteristics that are known before a procedure is initiated rather than procedural characteristics, because such models may aid in choosing the appropriate treatment (CAS, CEA, or optimal medical treatment alone).

Methods

Literature search

We performed a literature search in PubMed and Embase to identify articles on prediction models for short- or long-term outcome after CAS or CEA published until 22 December 2016.

We defined short term as models that predicted in-hospital risk of outcome or risk up to 30 days after the procedure; we classified all other models as long-term models. We searched with synonyms and index terms for ‘carotid artery stenting OR carotid endarterectomy’ AND ‘prediction model’ AND ‘vascular events’. Supplementary Table 1 shows the full syntax.

Eligibility criteria

We included all articles that reported the development or external validation of one or more multivariable prediction models for individual risk estimation of clinical outcome in patients with symptomatic or asymptomatic carotid artery stenosis who underwent CAS or CEA. We classified articles as model development studies if the authors reported the development of a prediction model in their objectives or conclusions, or if it was clear from other information in the article that they developed a prediction model for individual risk estimation (e.g. if they presented a risk score). Articles were included if they reported models for predicting periprocedural or postprocedural risk of vascular events (stroke, myocardial information, or death). In addition, the reported models had to include at least one patient characteristic that is known before a procedure is initiated and that can easily be retrieved from a patient’s medical record (e.g. demographics, vascular risk factors, comorbidities).

We excluded studies of patients with carotid artery stenosis who received medical treatment only or who underwent combined procedures for coronary and carotid artery stenosis. In addition, we excluded studies that used laboratory, imaging, or procedural characteristics as predictors in the final prediction model only. However, we did include studies that used both patient characteristics and characteristics of additional examinations or procedural characteristics in the final prediction model.

Screening process

One author (EJV) screened all potentially eligible studies on title and abstract and subsequently reviewed full-text copies of the selected publications. Uncertainty about eligibility assessment was resolved with discussion between two authors (EJV and JPG).

All prognostic factor studies that were identified through our search were screened full text to assess whether the authors developed a prediction model that met our inclusion criteria. References and related citations of the studies selected for critical appraisal were screened for additional relevant model development and external validation studies.

Critical appraisal and data extraction

We designed a standardised form for critical appraisal and data extraction based on items used in the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies.[5] Data on the following items were extracted for each study: study design, data source, recruitment period, characteristics of study population, outcome type, number of study participants and outcome events, modelling method, predictors used in the final model, model performance, and method of internal validation. If a study described multiple prediction models, data were extracted separately for each model. For studies describing an external validation of a prediction model (i.e. assessment of a model’s predictive accuracy in a different study population), we extracted the type of external validation (e.g. temporal, geographical), whether the external validation was performed by the same investigators who developed the model, characteristics of external validation study population, geographical location, number of study participants and outcome events, and model performance.

Data extraction and critical appraisal was performed by one author (EJV); in case of doubt this was resolved with discussion between two authors (EJV, JPG).

Statistical analysis

Results on modelling methods, model presentation, and predictors used in the final models were summarised with descriptive statistics. We did not perform a meta-analysis of predictive performance of the included prediction models, because predictors and predicted outcomes differed between the models.

To get an impression on the average effect size of the individual predictors, we performed a random-effects meta-analysis of all effect estimates (odds/risk/hazard ratios) for predictors that were used in at least 25% of prediction models for short-term outcome after CAS, short-term outcome after CEA or long-term outcome after CAS or CEA. Subsequently, we classified them into strong (≥2.5 or <0.4), moderate (1.5 to <2.5 or 0.4 to <0.7), or weak (1.1 to <1.5 or 0.7 to <0.9) predictors.

All results in this study are reported according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses statement.[6]

Results

Literature search and study selection

We identified 6366 unique studies in PubMed and Embase with our systematic search (Supplementary Figure 1). After screening titles and abstracts, 103 studies remained. Thirty-six model development studies met our inclusion criteria after full-text screening. Screening references and related citations of these studies revealed three additional articles (one model development study and two external validation studies).

The 37 model development studies that we selected for critical appraisal and data extraction described 46 unique prediction models: 11 models in patients who underwent CAS, 34 models in patients who underwent CEA, and one model in both patients who underwent CAS or CEA (Supplementary Table 2). Five of these studies developed more than one prediction model that met our inclusion criteria. Thirty-six models predicted short-term outcome (≤30 days after the procedure), and 10 models predicted long-term outcome (>30 days after the procedure) after CAS or CEA. Table 1 summarises the modelling methods separately for models that predict short- and long-term outcome.

Table 1.

Reported methods of prediction models for short-and long-term outcome.

	Short-term outcome	Long-term outcome	All models
	Total n = 36	Total n = 10	Total n = 46
Data source
Single centre	7 (19%)	5 (50%)	12 (26%)
Multicentre	26 (72%)	5 (50%)	31 (67%)
Trial	1 (3%)	0 (0%)	1 (2%)
Other	2 (6%)	0 (0%)	2 (4%)
Publication year, median (range)	2010 (1993–2016)	2014 (2012–2016)	2012 (1993–2016)
Most commonly predicted outcomes
Stroke or death	12 (33%)	0 (0%)	12 (26%)
Mortality	2 (6%)	7 (70%)	9 (20%)
Stroke, myocardial infarction, or death	6 (17%)	1 (10%)	7 (15%)
Stroke	3 (8%)	0 (0%)	3 (7%)
Other	13 (36%)	2 (20%)	15 (33%)
Most common period of follow-up
Postprocedural in-hospital	8 (22%)	0 (0%)	8 (17%)
Seven days after procedure	2 (6%)	0 (0%)	2 (4%)
30 days after procedure	26 (72%)	0 (0%)	26 (57%)
One year after procedure	0 (0%)	2 (20%)	2 (4%)
Three years after procedure	0 (0%)	3 (30%)	3 (7%)
Five years after procedure	0 (0%)	2 (20%)	2 (4%)
Other	0 (0%)	3 (30%)	3 (7%)
Presentation of prediction model
Regression coefficients	14 (39%)	1 (10%)	15 (33%)
Risk score	2 (6%)	1 (10%)	3 (7%)
Both	20 (56%)	8 (80%)	28 (61%)
Modelling method
Logistic regression	35 (97%)	2 (20%)	37 (80%)
Cox proportional hazards	0 (0%)	8 (80%)	8 (17%)
Other	1 (3%)	0 (0%)	1 (2%)
Shrinkage of predictor weights
Uniform	9 (25%)	3 (30%)	12 (26%)
Other	1 (3%)	0 (0%)	1 (2%)
Any	10 (28%)	3 (30%)	13 (28%)
None	26 (72%)	7 (70%)	33 (72%)
Internally validated models[a]
Bootstrapping	10 (28%)	3 (30%)	13 (28%)
Cross-validation	7 (19%)	2 (20%)	9 (20%)
Split sample	3 (8%)	0 (0%)	3 (7%)
Any	16 (44%)	5 (50%)	21 (46%)
None	20 (56%)	5 (50%)	25 (54%)
Externally validated models
Temporal	4 (11%)	0 (0%)	4 (9%)
Geographical	2 (6%)	1 (10%)	3 (7%)
Fully external	3 (8%)[b]	2 (20%)[c]	5 (11%)
Any	9 (25%)	3 (30%)	12 (26%)
None	27 (75%)	7 (70%)	34 (74%)

Results are presented as numbers (%), unless stated otherwise. Models for short-term outcome predict risk ≤30 days after the procedure; models for long-term outcome predict risk >30 days after the procedure.

Multiple internal validation methods may be used for one model.

Three prediction models were externally validated in a different paper.

Type of external validation unclear for one model.[7]

Data source

Most models were developed with data from multiple institutions (31 models; 67%). For the majority of these 31 models, data from large registries or health care databases were used (26 models; 57%). One model was developed with randomised trial data.

Predicted outcomes

In total, the 46 models predicted 16 different single and composite outcomes. For short-term prediction models, the most commonly predicted outcome was a composite of stroke or death (12 models; 33%); for long-term prediction models this was mortality (seven models; 70%) (Table 1). The other 12 predicted outcomes included myocardial infarction (two models; 4%), and composites of neurological and cardiac complications, such as transient ischaemic attack and congestive heart failure (Supplementary Table 2).

Predictors

In total, the models used 309 predictors (Figure 1, Supplementary Table 3). Median number of predictors per model was 7 (range 2–14). For short-term models, the most common predictors were age (67 versus 56% for CAS versus CEA), diabetes mellitus (33 versus 37%), history of TIA or stroke (22 versus 37%), and symptomatic status (44 versus 19%). Other common predictors for short-term outcome after CAS were anatomical characteristics (e.g. tortuous carotid arterial system or ulceration of carotid plaque on angiogram; 67%) and procedural characteristics (e.g. procedural hemodynamic depression or long duration of procedure; 56%). Other common predictors for short-term outcome after CEA were female sex (22%), ischaemic heart disease (26%), heart failure (41%), coronary artery disease (22%), renal insufficiency or dialysis (22%), and contralateral stenosis ≥ 50% or occlusion (33%). For long-term models, the most common predictors were age (100%), smoking (30%), diabetes mellitus (80%), heart failure (50%), chronic obstructive pulmonary disease or dyspnoea (60%), renal insufficiency or dialysis (50%), symptomatic status (20%), contralateral stenosis ≥ 50% or occlusion (50%), and statin use (30%). Symptomatic status was used as predictor in 11 (34%) of the 32 prediction models that were developed in both symptomatic and asymptomatic patients. Conflicting results regarding the direction of the predictor effect were found for female sex, high blood pressure, prior coronary intervention, ipsilateral carotid stenosis ≥70%, and antiplatelet use, but these predictors were only included in a few prediction models.

Figure 1.

Predictors used per type of prediction model. Models for short-term outcome predict risk ≤30 days after the procedure; models for long-term outcome predict risk >30 days after procedure. CABG: coronary artery bypass graft; CAS: carotid artery stenting; CEA: carotid endarterectomy; COPD: chronic obstructive pulmonary disease; PCI: percutaneous coronary intervention; TIA: transient ischaemic attack.

Supplementary Table 3 shows the pooled effect size for each predictor with classification of its strength. We observed weak to moderate predictors for short-term outcome after CEA and long-term outcomes, but strong predictors for short-term outcome after CAS. Anatomical and procedural characteristics are the strongest predictors for short-term outcome after CAS.

Sample size

The number of patients used to develop the prediction models ranged from 221 to 49,411 (Table 2). Number of events per candidate variable could be calculated for 13 (28%) models; the total number of candidate predictors was unclear for the remainder. Median number of events per candidate variable was 10.6 (range 1.9–52.3).

Table 2.

Results of all developed and externally validated prediction models.

	CAS		CEA		CAS and CEA	All models
	Short-term outcome	Long-term outcome	Short-term outcome	Long-term outcome	Long-term outcome		Results not reported (%)
Developed models	Total n = 9	Total n = 2	Total n = 27	Total n = 7	Total n = 1	Total n = 46	Total n = 46
Number of patients[a]	606 (221–11,122)	460 (317–602)	6553 (741–49,411)	2001 (291–4114)	506	3095 (221–49,411)	0 (0%)
Proportion of events	4.3% (2.4–11.5)	24.5% (13.9–35.0)	3.0% (0.5–6.9)	14.2% (6.7–26.8)	13.8%	4.1% (0.5–35.0)	0 (0%)
Age in years	71.5 (66.7–76.3)	70.8 (70.5–71.0)	70.6 (62.5–74.5)	70.0 (67.3–74.2)	69.0	70.6 (62.5–76.3)	7 (15%)
Proportion of males	63.4% (61.0–74.0)	72.8% (67.6–77.9)	59.1% (39.7–95.4)	60.6% (56.7–78.7)	62.3%	61.0% (39.7–95.4)	3 (7%)
Proportion of symptomatic patients	36.0% (5.1–100)	40.4%	43.9% (0–100)	25.3% (0–84.0)	0	40.9% (0–100)	6 (13%)
c-statistic[b]	0.73 (0.69–0.94)	0.73 (0.66–0.79)	0.68 (0.58–0.74)	0.72 (0.69–0.74)	NR	0.71 (0.58–0.94)	11 (24%)
External validations	Total n = 1	Total n = 2	Total n = 11[c]	Total n = 0	Total n = 1	Total n = 15	Total n = 15
Number of patients	1544	345 (137–552)	1998 (134–71,222)	NA	352	1026 (134–71,222)	0 (0%)
Proportion of events	NR	29.0% (18.9–39.0)	3.2% (0.7–22.1)	NA	NR	3.3% (0.7–39.0)	4 (27%)
c-statistic	0.68	0.68 (0.66–0.69)	0.65 (0.55–0.72)	NA	NR	0.66 (0.55–0.72)	3 (20%)

CAS: carotid artery stenting; CEA: carotid endarterectomy; NA: not applicable; NR: not reported.

Results are presented as median (range), unless stated otherwise. Models for short-term outcome predict risk ≤30 days after the procedure; models for long-term outcome predict risk >30 days after the procedure.

For some models only the total number of procedures was reported.

Optimism-adjusted c-statistic was used in case this was reported.

Three prediction models for short-term outcome after CEA were externally validated twice.

Modelling methods

Most prediction models for short-term outcome were developed with logistic regression analysis (35 models; 97%) and those for long-term outcome with Cox proportional hazards analysis (eight models; 80%). Out of all 46 prediction models, 21 (46%) models were internally and 12 (26%) models were validated at least once externally (Table 1).

In 25 models (54%), the authors did not report how they handled missing predictor or outcome data. In the remaining 21 models, complete case analysis was performed in 15, imputation was performed in three, and a combination of these methods was performed in three models. Twenty-eight (61%) of all models reported the full regression formula with intercept or baseline hazard, or the complete risk score with classification of risk according to this score.

Twenty-one (46%) newly developed models selected candidate predictors for the initial multivariable model based on univariable associations with the outcome. Only 12 (26%) models included all candidate predictors or chose variables based on previous literature or clinical experience.

Model presentation

Supplementary Table 4 shows which performance measures were reported for newly developed models and for external validations of these models. The c-statistic was the most often reported discrimination measure, both for newly developed models and for external validations. The Hosmer–Lemeshow test was the most often reported calibration measure for developed models.

Predictive performance

Newly developed models

Discriminative performance in terms of the c-statistic ranged from 0.58 to 0.94 for all newly developed models (Table 2). Median c-statistic was higher for models that predict short- or long-term outcome after CAS than after CEA. The highest c-statistics were reported for models that predict short-term outcome after CAS and included anatomical and procedural characteristics, and the lowest c-statistics for models that predict short-term outcome after CEA.

External validation studies

In 13 articles, 15 external validations were performed. The majority of the 46 developed models (34 models; 74%) has never been validated externally. Of the 12 models that were validated externally, nine (75%) were validated once, and three models were validated twice (Supplementary Table 2). Out of the 12 externally validated models, nine (75%) were solely externally validated in the same paper in which their development was described, and three (25%) were validated by independent researchers.[8-10]

The number of patients in the 15 external validation cohorts ranged from 134 to 71,222 (Table 2). The c-statistic ranged from 0.55 to 0.72 and was consistently lower than the corresponding c-statistic in the model development cohort, except for the model that was validated in the same study population in which the original model was developed.[9]

Discussion

This systematic review provides an overview of prediction models for short- and long-term outcome that are developed for patients who underwent carotid revascularisation for symptomatic or asymptomatic ICA stenosis. Age, diabetes mellitus, and contralateral carotid stenosis ≥50% or occlusion were most often used as predictors in all models. For short-term outcome models, symptomatic status, anatomical characteristics, and procedural characteristics were most important predictors after CAS; heart failure and history of stroke or TIA were most important predictors after CEA. For long-term outcome models after CAS or CEA, heart failure, COPD or dyspnoea, and renal insufficiency or dialysis were most important predictors. Predictive performance of the prediction models in terms of discrimination was in general poor to moderate. Only a minority of these prediction models was validated internally or externally. Methodological quality was low for the majority of the prediction models due to incomplete presentation and methodological limitations.

From the results of our systematic review, we can draw conclusions on most common predictors and their strength. However, predictors that were less often used in the included prediction models cannot be designated as unimportant, because not all studies included the same candidate predictors when they developed their model.

Discriminative performance of the prediction models differed between models developed in CAS and CEA patients, but was in general better for models developed in CAS patients. Specifically for short-term outcome models after CAS, addition of anatomical and procedural characteristics improved predictive performance; c-statistics were higher in short-term outcome models with anatomical and procedural characteristics after CAS compared with models without such characteristics, whereas this was not the case for short-term outcome models after CEA. Highest c-statistic (0.94) was reported for a risk score that predicted risk of a composite of stroke, death, or myocardial infarction within 30 days after CAS.[11] However, the risk of overfitting was high for this model because of the low number of patients and the lack of internal or external validation.

The prediction models included in our systematic review have several methodological and statistical shortcomings that hamper assessment of clinical usefulness in other study populations. First, most models were developed in large study populations from registries or health care databases; such studies may have a higher risk of bias due to missing outcome data and the lack of consistent definitions of predictors and outcomes. Moreover, patients included in these study populations were selected for CAS or CEA by their treating physicians based on vascular risk factors, comorbidities, and clinical experience. In general, multicentre prospective cohort studies with consistent definitions and more accurate event registration are preferred when developing a new prediction model.[12] Second, 20 (43%) of all prediction models were published more than five years ago (before 2012), while improved medical therapy with antiplatelets or anticoagulants, treatment of vascular risk factors, and healthy lifestyle changes have probably lowered short- and long-term risk of vascular events in carotid stenosis patients in more recent years. Furthermore, information on regression coefficients and model intercepts or baseline hazards was often lacking, which makes it difficult to externally validate the prediction model. Regarding model presentation, most studies reported the Hosmer-Lemeshow test as measure for calibration, whereas calibration is preferably reported graphically with a calibration plot.[13] Also, most studies did not report a measure for precision of the predictive performance, such as a confidence interval. Last, risk of overfitting is high for most prediction models included in our systematic review due to selection of candidate predictors based on statistical testing, lack of shrinkage and internal or external validation, and small sample sizes.

A major strength of our study is that this is the first systematic review of prediction models for short- and long-term outcome after CAS or CEA. In addition, we performed a comprehensive systematic search and used a standardised form for critical appraisal and data extraction of the selected studies. Nevertheless, our systematic review has some limitations. First, we included prediction models that also used procedural characteristics, whereas we aimed to identify models that predict clinical outcome based on preprocedural patient characteristics. However, the models with procedural characteristics also included preprocedural patient characteristics and thus contributed to our analysis of most frequently used predictors. Second, we used a broad outcome (vascular events) which may limit direct comparison of predictive performance of the included prediction models. Yet, our systematic review provides a comprehensive overview of all published prediction models after CAS or CEA. Third, we separately extracted data from each prediction model in case one study described multiple models (e.g. for different outcome types) or in case multiple studies used data from the same source and time period; this may have influenced our analysis of most important predictors and the meta-analysis of predictor strength. Fourth, some studies may not have reported all results in case of poor predictive performance of their developed prediction model (e.g. c-statistic close to 0.5), which is referred to as selective reporting bias. Selective reporting of predictive performance may have caused overestimation of the median c-statistics in our systematic review. Last, we could not retrieve full-text copies of three studies,[14-16] so we may have missed some prediction models.

We found it difficult to evaluate clinical usefulness of most of the prediction models included in our systematic review because of incomplete reporting and lack of validation. A new prediction model should be applied to other study populations to assess generalisability before the model can be used in daily practice.[17] In addition, all but one of the included prediction models predicted outcome after either CAS or CEA, so a common use of one of these models to predict risk after both revascularisation procedures is not possible. Hence, we cannot recommend the use of one specific prediction model for outcome prediction after CAS or CEA from our systematic review. Therefore, we are planning to conduct an external validation study of prediction models included in this systematic review.

In 2015, the TRIPOD statement was introduced, which contains a set of recommendations for the reporting of prediction modelling studies.[13] Use of this statement for the reporting of model development studies may facilitate assessment of methodological quality and clinical usefulness, and may enable external validation of newly developed prediction models by other researchers.

Conclusion

Many prediction models or risk scores for short- and long-term outcome after carotid revascularisation have been developed. Usefulness of most models remains limited because of methodological shortcomings, incomplete presentation, and lack of external validation. External validation of these models is necessary before they can be used to aid physicians in predicting clinical outcome and choosing an appropriate treatment in daily practice.