Clinical prediction model suitable for assessing hospital quality for patients undergoing carotid endarterectomy.

BACKGROUND: Assessing hospital quality in the performance of carotid endarterectomy (CEA) requires appropriate risk adjustment across hospitals with varying case mixes. The aim of this study was to develop and validate a prediction model to assess the risk of in-hospital stroke or death after CEA that could aid in the assessment of hospital quality. METHODS AND
RESULTS: Patients from National Cardiovascular Data Registry (NCDR)'s Carotid Artery Revascularization and Endarterectomy (CARE) Registry undergoing CEA without acute evolving stroke from 2005 to 2013 were included. In-hospital stroke or death was modeled using hierarchical logistic regression with 20 candidate variables and accounting for hospital-level clustering. Internal validation was achieved with bootstrapping; model discrimination and calibration were assessed. A total of 213 (1.7%) primary end point events occurred during 12 889 procedures. Independent predictors of stroke or death included age, prior peripheral artery disease, diabetes mellitus, prior coronary artery disease, having a symptomatic carotid lesion, having a contralateral carotid occlusion, or having New York Heart Association Class III or IV heart failure. The model was well calibrated and demonstrated moderate discriminative ability (c-statistic 0.65). The NCDR CEA score was then developed to support simple, prospective risk quantification in the clinical setting.
CONCLUSIONS: The NCDR CEA score, comprising 7 clinical variables, predicts in-hospital stroke or death after CEA. This model can be used to estimate hospital risk-adjusted outcomes for CEA and to assist with the assessment of hospital quality.

Introduction

Carotid endarterectomy (CEA) reduces stroke risk in both symptomatic and asymptomatic patients with carotid atherosclerosis compared with medical therapy.[1-4] While a number of studies have considered many potential risk factors for the development of adverse events after CEA, only a few have offered overall risk assessments for patients considering CEA.[5-12] These studies have been limited to reporting experience in Medicare populations,[8] in clinical trial populations,[7,9] in asymptomatic patients only,[8,11] in a smaller number of centers in Canada before aggressive lipid‐lowering and antiplatelet therapy was standard,[6] or based on administrative data sets.[12]

The National Cardiovascular Data Registry (NCDR), including the Carotid Artery Revascularization and Endarterectomy (CARE) Registry, provides innovative platforms to assess hospital quality. Sponsored by the American College of Cardiology Foundation in conjunction with the Society for Cardiovascular Angiography and Interventions, Society of Interventional Radiology, American Academy of Neurology, American Association of Neurological Surgeons/Congress of Neurological Surgeons, Society for Vascular Medicine, and Society of Vascular and Interventional Neurology, the CARE Registry was designed to create a national surveillance system to assess prevalence and outcomes of patient undergoing carotid revascularization. As of November 2013, 130 hospitals participate in the registry.[13] Participating hospitals receive quarterly reports with information pertaining to number of procedures performed, patient characteristics, and measures of quality including discharge medication prescriptions and crude rates of complications such as in‐hospital stroke and death. However, because of differences in case mix among hospitals, differences in crude adverse event rates may be a reflection of differences in patient complexity, particularly with respect to clinical features that strongly influence outcomes.

While most attempts to assess hospital procedural quality have relied on administrative claims data,[14] observational registries that include prospectively collected clinical data such as the CARE Registry afford the opportunity to create validated, clinically based prediction models to serve as the basis for risk adjustment for hospital quality reports. In this report, we used the CARE Registry to develop and validate a preprocedural risk model and risk score that will be used to provide feedback on risk‐adjusted outcomes after CEA to participating hospitals to promote quality improvement.

Methods

The CARE Registry enrolls patients with carotid stenosis who have undergone revascularization with either CEA or carotid artery stenting (CAS).[15] Between April 2005 and July 2013, the registry accrued data from 12 889 CEA procedures performed at 60 hospitals. Data elements and definitions are listed at https://www.ncdr.com/webncdr/care/home/datacollection. Detailed quality checks are implemented before incorporation of data into the registry.[16] A waiver of written informed consent and authorization for this study was granted by the Chesapeake Research Review Incorporated Institutional Review Board.

To maximize the applicability of the findings, all except those undergoing CEA for acute evolving stroke were included. The primary outcome of interest was the occurrence of in‐hospital stroke or death. Stroke was defined as a new neurologic deficit persisting for at least 24 hours after the procedure. Outcomes were abstracted by trained data collectors.

Statistical Analyses

We examined broad ranges of variables related to sociodemographic characteristics, cardiovascular history, and neurologic history based on clinical experience and understanding of the literature. We conducted bivariate comparisons of characteristics of patients with or without the outcome of interest (Table 1) by using χ2 tests for dichotomous variables and t tests for continuous variables. We then generated a logistic model for the primary end point conditioned on clinical variables.[17] Variables were selected using backward elimination of 20 variables using a criterion of P<0.05 to remain. Model coefficients were then refit using a hierarchical generalized linear model accounting for clustering by hospital (see supplemental online Appendix).

Table 1.

Baseline Clinical Characteristics

Characteristic	Total	Stroke or Death	No Stroke or Death	P Value
	(N=12 889)	(n=213)	(n=12 676)
Age, y	70.8±10.6	72.9±10.1	70.8±10.6	0.004
Age ≥80 y	2459 (19.1%)	55 (25.8%)	2404 (19.0%)	0.012
Male	7603 (59.0%)	116 (54.5%)	7487 (59.1%)	0.175
White	7603 (59.0%)	116 (54.5%)	7487 (59.1%)	0.175
GFR, mL/min per 1.73 m2	72.4±30.8	70.8±26.0	72.4±30.9	0.477
GFR<30, mL/min per 1.73 m2 or current dialysis	568 (4.4%)	9 (4.2%)	559 (4.4%)	0.896
Current dialysis	216 (1.7%)	5 (2.3%)	211 (1.7%)	0.411
Tobacco use	9307 (72.2%)	149 (70.0%)	9158 (72.2%)	0.459
Hypertension	11 526 (89.4%)	197 (92.5%)	11 329 (89.4%)	0.143
Dyslipidemia	10 502 (81.5%)	166 (77.9%)	10 336 (81.5%)	0.178
Peripheral artery disease	3979 (30.9%)	85 (39.9%)	3894 (30.7%)	0.004
Diabetes mellitus	4518 (35.1%)	91 (42.7%)	4427 (34.9%)	0.018
Chronic lung disease	2826 (21.9%)	60 (28.2%)	2766 (21.8%)	0.026
Home oxygen	404 (14.4%)(N=10 062)	8 (13.3%)(n=153)	396 (14.5%)(n=9909)	0.785
Major surgery planned within 8 wk	471 (3.7%)	24 (11.3%)	447 (3.5%)	<0.001
Type of major surgery				0.002
Cardiac	305 (65.0%)	22 (95.7%)	283 (63.5%)
Vascular	99 (21.1%)	0 (0.0%)	99 (22.2%)
Other	65 (13.9%)	1 (4.3%)	64 (14.3%)
	(N=12 416)	(n=189)	(n=12 227)
Previous neck radiation	115 (0.9%)	5 (2.3%)	110 (0.9%)	0.042
Previous neck surgery	115 (0.9%)	5 (2.3%)	110 (0.9%)	0.042
Previous laryngeal nerve palsy				0.003
No	12 842 (99.7%)	209 (98.1%)	12 633 (99.7%)
Yes, right	25 (0.2%)	2 (0.9%)	23 (0.2%)
Yes, left	14 (0.1%)	2 (0.9%)	12 (0.1%)
Cardiac history
Ischemic heart disease	5486 (42.6%)	105 (49.3%)	5381 (42.5%)	0.046
Two or more coronary arteries with stenosis ≥70% (LAD, LCx, RCA)	2890 (22.7%)	61 (29.3%)	2829 (22.6%)	0.021
Left main coronary stenosis ≥50%	521 (4.1%)	19 (9.2%)	502 (4.0%)	<0.001
MI within 6 wk	181 (1.4%)	10 (4.7%)	171 (1.3%)	<0.001
Angina CCS Angina Class III or IV within 6 wk	306 (2.4%)	15 (7.0%)	291 (2.3%)	<0.001
History of heart failure	1221 (9.5%)	36 (16.9%)	1185 (9.4%)	<0.001
NYHA Class III or IV within 6 wk	306 (2.4%)	15 (7.0%)	291 (2.3%)	<0.001
NYHA Class III/IV or LVEF ≤35%	626 (4.9%)	23 (10.8%)	603 (4.8%)	<0.001
LVEF assessed preprocedure	6816 (52.9%)	139 (65.3%)	6677 (52.7%)	<0.001
LVEF ≤35%	385 (3.0%)	14 (6.6%)	371 (2.9%)	0.002
History of atrial fibrillation/flutter	1439 (11.2%)	32 (15.0%)	1407 (11.1%)	0.072
Moderate to severe aortic stenosis	325 (2.5%)	12 (5.6%)	313 (2.5%)	0.004
Moderate to severe mitral stenosis	78 (0.6%)	3 (1.4%)	75 (0.6%)	0.139
Mechanical aortic or mitral valve	210 (1.6%)	5 (2.3%)	205 (1.6%)	0.402
Permanent pacemaker or ICD	657 (5.1%)	10 (4.7%)	647 (5.1%)	0.786
ASA physical classification grade				<0.001
1	193 (1.5%)	1 (0.5%)	192 (1.5%)
2	1432 (11.2%)	17 (8.0%)	1415 (11.2%)
3	8660 (67.5%)	111 (52.4%)	8549 (67.8%)
4	2511 (19.6%)	81 (38.2%)	2430 (19.3%)
5	33 (0.3%)	2 (0.9%)	31 (0.2%)
Neurologic history and risk factors
Dementia or Alzheimer's disease	307 (2.4%)	11 (5.2%)	296 (2.3%)	0.007
History of seizure or seizure disorder	261 (2.0%)	4 (1.9%)	257 (2.0%)	1.000
Previous carotid revascularization	1979 (15.4%)	34 (16.0%)	1945 (15.3%)	0.805
Prior CEA	1883 (14.6%)	31 (14.6%)	1852 (14.6%)	0.982
Prior CAS	122 (0.9%)	4 (1.9%)	118 (0.9%)	0.143
Prior ipsilateral CAS	20 (0.2%)	0 (0.0%)	20 (0.2%)	1.000
Prior ipsilateral CEA	240 (1.9%)	2 (0.9%)	238 (1.9%)	0.445
Neurologic event(s) preprocedure	5484 (42.6%)	116 (54.5%)	5368 (42.4%)	<0.001
Prior TIA	3353 (26.0%)	67 (31.5%)	3286 (25.9%)	0.068
Prior ischemic stroke	1752 (13.6%)	43 (20.2%)	1709 (13.5%)	0.005
Prior hemorrhage or hemorrhagic stroke	78 (0.6%)	2 (0.9%)	76 (0.6%)	0.370
Procedure information
Target carotid vessel				0.507
Right	6523 (50.6%)	103 (48.4%)	6420 (50.6%)
Left	6366 (49.4%)	110 (51.6%)	6256 (49.4%)
Anesthesia type				0.058
General	11 671 (90.6%)	201 (94.4%)	11 470 (90.5%)
Local	1210 (9.4%)	12 (5.6%)	1198 (9.5%)
Urgent cardiac surgery within 30 d	431 (3.4%)	24 (11.5%)	407 (3.2%)	<0.001
Target lesion symptomatic within 6 mo	4510 (35.0%)	101 (47.4%)	4409 (34.8%)	<0.001
Restenosis in target vessel after prior CAS	228 (1.8%)	2 (0.9%)	226 (1.8%)	0.594
Restenosis in target vessel after prior CEA	228 (1.8%)	2 (0.9%)	226 (1.8%)	0.594
Contralateral carotid artery occlusion	686 (5.3%)	24 (11.3%)	662 (5.2%)	<0.001
Fibromuscular dysplasia of carotid artery	21 (0.2%)	3 (1.4%)	18 (0.1%)	0.005

GFR indicate glomerular filtration rate; LAD, left anterior descending coronary artery; LCx, left circumflex artery; RCA, right coronary artery; MI, myocardial infarction; CCS, Canadian Cardiovascular Society; NYHA, New York Heart Association; LVEF, left ventricular ejection fraction; ICD, implantable cardioverter defibrillator; ASA, American Society of Anesthesiologists; CEA, carotid endarterectomy; CAS, carotid artery stenting; TIA, transient ischemic attack.

The performance of a prediction model within a sample of data may overestimate the true performance of the model within the broader population. The difference between the “apparent performance” and the true performance is known as “optimism.”[18] The extent of optimism of prespecified models can be estimated for similar patient populations using internal validation techniques such as bootstrapping. After fitting the model, we conducted internal validation by refitting the model in 1000 bootstrap samples with replacement. Measures of model performance were corrected for optimism, and the final model was recalibrated based on a “shrinkage” factor derived from the calibration slope.[17] To further assess calibration, we fitted a smoothed line showing the relationship between predicted and observed risk of in‐hospital stroke or death based on the final model.[17] This method has less variability and less potential bias compared with traditional split‐sample validation and k‐fold cross‐validation.[19] To assess variability in risk‐adjusted event rates among hospitals, we estimated the “predicted”‐to‐expected ratio of in‐hospital stroke or death (or the standardized adverse event ratio) for each hospital based on its observed case mix.[20] To support routine clinical use, we developed a risk score based on a points system with weights based on the coefficients in the final clinical model.[21]

Values of P<0.05 (2‐tailed) were considered statistically significant. Statistical analyses were performed using SAS 9.2 (SAS Institute) and R.

Results

There were a total of 213 (1.7%) in‐hospital stroke or death events after 12 889 CEA procedures. Stroke occurred in 179 (1.4%) and death occurred in 52 (0.4%) patients. Of the strokes, 124 (69.2%) were ipsilateral to the revascularized artery. Bivariate analysis (Table 1) demonstrated that individuals who developed in‐hospital stroke or death were older and were more likely to have peripheral artery disease (PAD), significant coronary artery disease, recent myocardial infarction, or congestive heart failure. They were also more likely to have a tracheostomy present or have preexisting laryngeal nerve palsy. Patients who developed in‐hospital stroke or death were more likely to have had a prior ischemic stroke, were more likely to be symptomatic from the target lesion, or were more likely to have a contralateral carotid occlusion.

Multivariable Model

After multivariable regression, 7 variables were retained in the final model (Table 2). These variables included age, PAD, history of diabetes mellitus, history of myocardial infarction within 6 weeks, symptomatic target lesion within 6 months, contralateral carotid occlusion, and a history of New York Heart Association Class III or IV heart failure within 6 weeks. The patient characteristics most strongly associated with in‐hospital stroke or death in the final model were a history of recent myocardial infarction (odds ratio [OR] 2.66, 95% CI 1.89 to 3.74), the presence of a contralateral carotid stenosis (OR 2.27, 95% CI 1.83 to 2.81), for a recent history of New York Heart Association Class III or IV heart failure (OR 2.39, 95% CI 1.78 to 3.19). This model was found to have a c‐statistic of 0.65, demonstrating moderate discrimination. The model was well calibrated (Hosmer‐Lemeshow P=0.48, Figure 1).

Table 2.

Multivariable Predictors of In‐Hospital Stroke or Death, N=12 889

	β‐Coefficient	SE	t Value	P Value	Odds Ratio (95% CI)
Intercept	−6.1609	0.5974	−10.31	<0.0001
Age (per 10 y)	0.2321	0.07026	3.30	0.0010	1.26 (1.18 to 1.35)
Peripheral artery disease	0.3353	0.1386	2.42	0.0156	1.40 (1.22 to 1.61)
Diabetes mellitus	0.3167	0.1349	2.35	0.0189	1.37 (1.20 to 1.57)
MI within 6 weeks	0.9788	0.3408	2.87	0.0041	2.66 (1.89 to 3.74)
Target lesion symptomatic in prior 6 months	0.5374	0.1354	3.97	<0.0001	1.71 (1.49 to 1.96)
Contralateral carotid occlusion	0.8177	0.2143	3.82	0.0001	2.27 (1.83 to 2.81)
NYHA Class III/IV	0.8694	0.2918	2.98	0.0029	2.39 (1.78 to 3.19)

MI, myocardial infarction; NYHA, New York Heart Association.

Figure 1.

Observed vs predicted probability of in‐hospital stroke or death. This calibration plot depicts observed (y‐axis) vs predicted (x‐axis) in‐hospital stroke or death rates for patients undergoing carotid endarterectomy. Differences between observed and predicted event rates were small across all levels of risk (Hosmer‐Lemeshow P=0.52).

Integer‐Based Scoring System

Based on the multivariable model, an integer‐based scoring system (the NCDR CEA score) was built for clinical use (Table 3). Observed rates of death or stroke are shown as a function of the point total on the integer scoring system (Figure 2).

Table 3.

NCDR CEA Risk Score System

Variable	Points
Age, y
<50	0
50 to 59	2
60 to 69	4
70 to 79	6
80 to 89	8
≥90	10
Peripheral artery disease	3
Diabetes mellitus	3
MI within 6 weeks	8
Target lesion symptomatic in previous 6 months	5
Contralateral carotid occlusion	7
NYHA Class III/IV	7

NCDR indicates National Cardiovascular Data Registry; CEA, carotid endarterectomy; MI, myocardial infarction; NYHA, New York Heart Association.

Figure 2.

Stroke or death rates based on cumulative risk score. Observed stroke or death rates are shown as a function of the total number of points on the NCDR CEA risk score. CEA indicates carotid endarterectomy; MACE, combined endpoint of stroke or death; NCDR, National Cardiovascular Data Registry.

Risk‐Standardized Event Rates by Hospital

Using the multivariable model, standardized ratios of in‐hospital stroke or death for each hospital were generated by hospital and were plotted in Figure 3. These ratios varied from 0.64 to 2.10, suggesting differences in CEA outcomes among the registry hospitals not attributable to observable differences in case mix.

Figure 3.

Histogram of hospitals according to risk‐standardized event rates. Risk standardized stroke/death rates indicate in‐hospital stroke or death.

Discussion

Using a large population of patients undergoing CEA in the United States, we developed and validated a risk model and a risk score that predicts in‐hospital stroke and death after CEA. The risk factors we identified have been previously described by other investigators to increase risk of adverse events after CEA, either alone or in combination with other factors.[22-23]

Hospital Quality Assessment

With growing emphasis on public outcome reporting, we believe that risk models based on easily collected clinical factors will have an increasingly important place in the assessment and reporting of health care quality. The NCDR CEA Risk Score includes relevant clinical factors that are likely not available through traditional administrative data sets. Inclusion of these clinical factors in the risk model improves the face validity of these tools for clinicians and underscores the importance of clinical registries in measuring hospital quality. The variability in standardized adverse event ratios observed across participating hospitals provides evidence that such models may be useful to hospitals in efforts to benchmark performance against other hospitals and for lower‐performing hospitals to initiate efforts to improve outcomes.

NCDR CEA Risk Score in the Context of Previous Risk Stratification Models

This risk score was developed in a large data set (>10 000 procedures) of CEA in both symptomatic and asymptomatic patients from a broad range of hospitals across the United States. The patient population was not limited to Medicare beneficiaries or those enrolled in clinical trials. We also note the overall low event rates in this population of patients. This could indicate evolution of treatment of this patient population (including improved background medical therapy, surgical techniques, anesthesia care, etc); it could be related to patient selection in the data set; or it could be related to the self‐reported nature of the data without routine, independent neurologist assessment in every patient. Changes in surgical and perioperative practice, indicated by changing rates of adverse events over time, underscore the need for ongoing surveillance and, when necessary, updating risk prediction algorithms that are intended for clinical practice. Importantly, this risk score was developed accounting for natural clustering in the data by hospital site and can be used for assessments of hospital quality at the site level.

Study Limitations

Several limitations warrant discussion. First, prediction of in‐hospital stroke or death is limited to the in‐hospital period and does not predict long‐term outcomes in these patients as this information is not available in CARE. Second, we did not adjust for all variables that influence CEA‐related risk, as indicated by the moderate discriminative ability of the model. For instance, the stenosis severity of the carotid lesion and the overall level of disability of the patient before the index procedure are not well characterized in CARE and thus have not been captured in the model. However, models with imperfect patient‐level discrimination are still useful for the assessment of hospital quality without introducing bias so long as unobserved variables strongly associated with in‐hospital stroke or death are similarly prevalent across participating hospitals.[20] The stroke end point was not adjudicated beyond reporting by individual sites and patients were not routinely tested by a neurologist in a standardized way postprocedure. The CARE Registry thus could underreport minor strokes, as has occurred in other studies. However, the overall rates of death and stroke in our study at 30 days were similar to those observed in both the CREST (Carotid Revascularization Endarterectomy versus Stenting Trial) trial and other quality improvement programs.[12,24] Finally, this registry was not designed to inform us about patients who were screened but did not have CEA.

Conclusions

We developed and internally validated a risk model to predict in‐hospital stroke or death after CEA using data from the CARE Registry. The model will be used to support hospital comparisons of risk‐adjusted outcomes among the >100 participating hospitals, allowing for accurate benchmarking of hospital performance or quality improvement efforts for CEA. In addition, it may provide valuable information about procedural risk for patients considering this therapy.

Supplemental Statistical Appendix.

Click here for additional data file.