Long-Term Post-CABG Survival: Performance of Clinical Risk Models Versus Actuarial Predictions.

BACKGROUND/AIM: Clinical risk models are commonly used to predict short-term coronary artery bypass grafting (CABG) mortality but are less commonly used to predict long-term mortality. The added value of long-term mortality clinical risk models over traditional actuarial models has not been evaluated. To address this, the predictive performance of a long-term clinical risk model was compared with that of an actuarial model to identify the clinical variable(s) most responsible for any differences observed.
METHODS: Long-term mortality for 1028 CABG patients was estimated using the Hannan New York State clinical risk model and an actuarial model (based on age, gender, and race/ethnicity). Vital status was assessed using the Social Security Death Index. Observed/expected (O/E) ratios were calculated, and the models' predictive performances were compared using a nested c-index approach. Linear regression analyses identified the subgroup of risk factors driving the differences observed.
RESULTS: Mortality rates were 3%, 9%, and 17% at one-, three-, and five years, respectively (median follow-up: five years). The clinical risk model provided more accurate predictions. Greater divergence between model estimates occurred with increasing long-term mortality risk, with baseline renal dysfunction identified as a particularly important driver of these differences.
CONCLUSIONS: Long-term mortality clinical risk models provide enhanced predictive power compared to actuarial models. Using the Hannan risk model, a patient's long-term mortality risk can be accurately assessed and subgroups of higher-risk patients can be identified for enhanced follow-up care. More research appears warranted to refine long-term CABG clinical risk models.

Risk models can provide clinicians with useful information to evaluate a patient's optimal treatment strategy and to facilitate informed discussions about their inherent risk. Historically, statistical risk models have been developed both nationally and regionally to predict short‐term outcomes (e.g., 30‐day operative mortality) following coronary artery bypass graft (CABG) surgery.1, 2 Focused on mortality or morbidity predictions, the published algorithms vary as to the patient risk factors, procedural details, and institutional structures that are incorporated as model‐eligible variables.

Historically, the cardiac care team has used short‐term mortality predictions in assessing the best course of treatment. Although the confidence interval around any point estimate of risk is inherently large, surgeons discussed with each patient their individual probability of surviving CABG surgery or incurring perioperative major complications. Beyond the hospital‐related mortality or morbidity risks calculated, a surgeon counseled patients as to their unique or rare risk factors—not all of which may be adequately represented in the specific model from which their estimated risk of short‐term mortality or morbidity was originally derived.

While a priority was placed in the past upon predicting post‐CABG outcomes either in‐hospital or within 30 days of surgery, the majority of published risk models do not provide useful information to guide long‐term post‐CABG patient management. As advances have been made in perioperative care over the last few decades, operative mortality has decreased; and more attention is now required to improve long‐term survival following surgery.3, 4 Further improvements in long‐term risk models may be useful to aid in the determination of the most appropriate course of perioperative treatment, coordinate post‐discharge risk factor modification or continuity of care strategies with other clinical specialists, and help patients to frame their own expectations and/or implement behavioral modifications to optimize their post‐CABG long‐term mortality risk.

Long‐term mortality models (such as the one published by Wu et al., referred to here as the “Hannan model”) enable the clinician to quickly and easily predict patient mortality several years after CABG surgery at the bedside, without the need for computational software.5, 6 To date, there has not been a comparison of the findings derived from clinical long‐term mortality risk models with actuarial projections commonly used by insurers to assess risk. Actuarial models calculate the probability of long‐term mortality based upon population demographic data alone (such as age, gender, and race/ethnicity), and are a useful comparison because they do not take into account medical comorbidities in assessing risk of mortality. The purpose of this study was to explore if clinical risk models actually provide more accurate mortality predictions than population‐based actuarial models that do not account for medical comorbidities. Thus, the differences in long‐term mortality risk estimates as predicted by a clinical risk model were compared to the projections derived from an actuarial model at one‐, three‐, and five years post‐CABG surgery.

MATERIALS AND METHODS

Data for all patients who underwent an isolated CABG procedure at Stony Brook University Hospital (SBUH) between July 2006 and June 2011 (n = 1,220) was extracted from the SBUH Department of Surgery's Cardiothoracic Surgery quality assurance (QA) program's database. This retrospective cohort analysis was approved as part of the Stony Brook Medicine Department of Surgery project titled the “Surgical Quality Data Use Group” (SQDUG) as part of Stony Brook University's Committee on Human Subjects in Research (CORIHS) approved research protocol #17053‐4.

As part of the original cardiothoracic surgery quality improvement program, the vital status (observed mortality) for all records was assessed (latest death data update performed on October 4, 2013) by searching a commercially available interface to the Social Security Death Index (SSDI) by Social Security Number (SSN).7 As this approach was potentially subject to error (e.g., inaccurate SSN's recorded in hospital records), a subset of 128 patients were also searched by last name and month and year of birth. To evaluate the reliability of SSDI search engine performance, this pilot test search was conducted using two different commercial SSDI.8 Comparison of results between these two different SSDI search approaches revealed a high concordance for the 128 pilot‐test records compared, giving a κ statistic of 0.9375.

Additionally, the time (in days) from date of surgical procedure to date of death was calculated for each record, and three different dummy indicator variables were created to identify one‐, three‐, and five‐year postsurgery deaths. The time (in days) from surgical procedure to last SSDI update (October 4, 2013) was also calculated, with patient subgroups created that had at least one, three, and five years of postsurgery follow‐up time.

All quality assurance records were moved to a separate server and subsequently de‐identified. From the de‐identified “master” database, the records for this SQDUG‐approved analysis were extracted. Based on each record's risk factor profile, the one‐, three‐, and five‐year Hannan mortality estimates were calculated. To do this, the presence or absence of the Hannan risk factors was assessed, with points assigned to each patient's records based on the presence of those risk factors as specified by Wu et al. (2012) and recounted here in Table 1.6 The point score was based on the summation of points for each patient profile, with possible point totals ranging from 0 to 28. Based on the number of points accumulated, profiles were then assigned a corresponding mortality risk, also published by Wu et al., for each of the three different time periods studied.

Table 1

Risk Factors Used in Hannan and Actuarial Risk Models

Risk Factor	Hannan Points	Actuarial Model
Age (years)		★
≤ 50	0
51–59	1
60–69	3
70–79	5
≥ 80	7
Gender (male/female)		★
Race (white/non‐white)		★
Body mass index (kg/m2)
< 18.5	2
18.5–24.99	1
25.0–39.99	0
≥ 40	1
Ejection fraction (%)
< 30	2
30–39	1
≥ 40	0
Hemodynamically unstable or shock	2
Left main coronary artery disease	1
Cerebrovascular disease	1
Peripheral arterial disease	1
Congestive heart failure	1
Malignant ventricular arrhythmia	1
Chronic obstructive pulmonary disease	1
Diabetes mellitus	2
Renal failure
Requiring dialysis	6
Creatinine >2.5 mg/dL	3
No renal failure	0
Previous open heart operations	1

A * signifies that an item was included as a variables in the actuarial model.

For these same time periods, the actuarial projections were also determined for the study sample using data published by the Centers for Disease Control (CDC) in the National Vital Statistics Reports 2011.9 The data from Table 3 of this publication (Number of deaths and death rates, by age, race, and sex: United States, 2009) was used for this purpose. As the data in that table is presented as probabilities of dying in a given year of life rather than cumulative mortality risk, the given probabilities had to be transformed. This was done for each of our SBUH patient records using the equation: where M is the mortality prediction for each time period of interest, t is the time period (i.e., one, three, and five years), P is the patient's probability of dying in a given year, and i represents each year in the time period beginning on the date of surgery.

Using the methods described by DeLong et al., a nested c‐index comparison was used to compare the findings from the Hannan risk model versus the actuarial projections to determine if there was a significant difference in the predictive power of these two models.10, 11 Specifically, this non‐parametric method uses Mann–Whitney U statistics to compare two correlated receiver operating characteristic (ROC) curves. As an alternative method, the Demlar approach (i.e., calculating “net reclassification index” [NRI]) was not necessary to assess model performance, as the Hannan and Actuarial model's risk factors already had a previously documented association with long‐term survival. 12, 13, 14 For the observed outcome of mortality, the model predictions from the Hannan model were compared to the actuarial model's predictions for the one‐, three‐, and five‐year patient follow‐up subgroups, with the latter model being used as the reference standard. Calculations were performed using the ROCCONTRAST statement in SAS©, as outlined in reference 10.

The mortality predictions for the Hannan and actuarial models were then directly compared to the observed mortality extracted from the SSDI. This was done by calculating the observed/expected mortality (O/E) ratio for each Hannan point value, with the goal of better understanding model differences across the patient illness severity. To assess relative model performance, logistic regression was used to estimate the Hannan and Actuarial model‐specific ability to predict time‐specific mortality. Using these logistic models, a c‐index and a Hosmer‐Lemeshow “goodness of fit” P‐value were calculated. In addition to assessing traditional model performance metrics, the Hannan and Actuarial models' clinical calibrations were assessed across low (<5%), medium (5% to 25%), and high (>25%) categories for mortality risk.

A linear regression model was used to further explore which specific risk factors may be driving this difference. Both simple and more complex multiple regression models were created to identify Hannan model risk factors that had a statistically significant impact (and relatively greater effect sizes for the observed associations) on these differences for the one‐, three‐, and five‐year mortality predictions.

To evaluate for a potential bias related to the exclusion of patient records with missing risk data (as required by the Hannan model), two imputation approaches were used. A sensitivity analysis was run using the two different imputed datasets to assure the original study's conclusions (i.e., the primary data set analyzed with complete risk information) were robust. Additionally, the Hannan model predictions were sub‐stratified to evaluate for a potential gender‐related bias. All calculations were performed using SAS© 9.2 (Cary, NC, USA).

RESULTS

A total of 1220 CABG patient records were extracted for this retrospective, observational study's analysis. The completeness of all risk variables (based on the subset of risk variables required to calculate both the Hannan and actuarial mortality projections) was evaluated, and 179 records were excluded from the primary analysis due to missing risk factor data (as required by the Hannan algorithm). Additionally, 13 patients expired post‐CABG prior to discharge, so these records were also removed in accordance with the approach used to develop the Hannan model. The final study sample included 1028 CABG patient records, which represented 84.3% of patients undergoing CABG procedures at SBUH between July 2006 and June 2011.

Total median follow‐up time was five years (interquartile range [IQR] = three years, six years). As of the latest SSDI death update, the total cumulative mortality rate across all study records was approximately 15%. As the Hannan model provides mortality estimates for one, three, and five years after CABG surgery, patient subgroups were created containing those patient records with at least one year of follow‐up (n = 1,028); at least three years of follow‐up (n = 934); and at least five years of follow‐up (n = 526). A description of the prevalence of the risk factors used in the analyses described herein within these three study samples can be found in Supplementary Table 1. Though the Hannan model also provides mortality estimates for seven years after surgery, there were only 60 SBU patients (of 1028; 5.8% within this seven‐year patient cohort). As this was not a sufficient sample size for a meaningful comparison, the seven‐year mortality analysis was not pursued.

The one‐, three‐, and five‐year O/E mortality ratios are plotted in Figure 1 by Hannan point score, with confidence intervals identified for the actuarial model's O/E ratios. No confidence intervals were calculated for the Hannan O/E ratios, however, as the values of these confidence intervals were not published with the point estimates given in the Wu et al. publication.6 For further reference, all observed mortality proportions and models' predicted mortality probabilities are listed in Supplementary Table 2.

Figure 1

O/E ratios for mortality one, three, and five years following CABG surgery, by Hannan point total. The Hannan model's long‐term (i.e., five‐year) mortality predictions appear to be its most accurate. For all time periods studied, the Hannan model better predicted mortality in relatively sicker patients (i.e., those with higher point totals).

Table 2

Performance of Five‐Year Mortality Models by Age Category

Age	Hannan Model O/E Ratio (95%CI)	Actuarial Model O/E Ratio (95%CI)
≤50 years	2.24 (0.58–3.90)	5.69 (1.47–9.90)
51–60 years	0.64 (0.22–1.06)	1.37 (0.48–2.27)
61–70 years	0.96 (0.58–1.34)	1.52 (0.92–2.11)
71–80 years	0.89 (0.56–1.23)	1.24 (0.77–1.70)
≥80 years	1.41 (0.79–2.03)	1.08 (0.61–1.55)

O/E, observed/expected.

For the one‐year follow‐up subgroup (i.e., patients followed for at least one year; mortality within one year of surgical procedure), both models overestimate the risk of mortality for those patients with lower Hannan point scores (i.e., those patients with fewer comorbidities or other risk factors), leading to O/E ratios that are less than 1.0. Not surprisingly, the O/E ratios for the models start to diverge as patients get sicker (increasing Hannan point totals), with the Hannan model having more accurate mortality estimates, and the actuarial model significantly underestimating mortality. In the three‐year subgroup, the Hannan model appears to generally overestimate mortality, while the actuarial model oscillates around 1.0 for lower Hannan point scores and significantly under‐predicts mortality for those patients with higher Hannan scores. Finally, in the five‐year subgroup, the Hannan model does a notably better job of predicting mortality, hovering around an O/E ratio of 1.0 for all point totals. Most of the actuarial model projections for this five‐year time period were significantly different than the Hannan model. However, the actuarial model severely under‐estimated the risk of mortality for patients with higher Hannan point scores.

Comparing overall model performance, the Hannan and actuarial models did not show a significant difference in their relative accuracy for one‐year predicted mortality estimates (AUC = 0.84 [CI = 0.78, 0.91] vs. 0.78 [CI = 0.70, 0.86], nested AUC comparison p = 0.101). For both three‐ and five‐year estimates, however, there were significant differences (Fig. 2 and Supplementary Table 3). The three‐year predictions revealed the Hannan model to more accurately predict the observed mortality as compared to the actuarial model (AUC = 0.76 [CI = 0.71, 0.82] vs. 0.70 [CI = 0.63,0.76], p = 0.008). Similarly, the five‐year predictions revealed the same conclusions (AUC = 0.76 [CI = 0.71, 0.82] vs. 0.70 [CI = 0.64, 0.77], p = 0.021). In general, the Hannan model was relatively well‐calibrated when compared to the Actuarial model, which did not meet the H‐L “goodness of fit” test for model years one and five (H–L P‐value < 0.05). In comparing observed mortality rates across low (<5%), medium (>5% to 25%), and high (>25%) clinical risk categories, the Actuarial model did not have a high degree of clinical calibration as compared to the Hannan model.

Figure 2

ROC curve comparison of Hannan and Actuarial models at one, three, and five years. The Hannan model more accurately predicted the observed three‐ and five‐year mortality than the actuarial model (p = 0.008 and p = 0.02, respectively). The difference in one‐year mortality predictions was not significant (p = 0.10).

A sensitivity analysis using the two different imputed datasets (with risk factor data imputed for the subgroup of records that had been dropped from the original analysis due to incomplete data) evaluated for a potential bias, with no difference in the study findings identified. Linear regression analyses were used to identify those Hannan risk factors that were key drivers for the model differences identified (Supplementary Table 4 and Supplementary Table 5). With the highest impact, preoperative renal failure (defined as requiring dialysis or creatinine >2.5 mg/dL) stood out as being the most important Hannan risk variable associated with the model differences observed. Further, these variables had the highest parameter estimates among those reaching statistical significance in the multiple linear regression analysis, appearing to be even higher than would be expected based on their inherent weighting in the Hannan model.

Additionally, analyses were performed to identify any potential differential benefit of the clinical risk model in older patients, particularly octogenarians, as this patient sub‐group has previously been identified as a high‐risk for CABG surgery.15 O/E ratios were calculated for patients with five‐year follow‐up data. These results were stratified by patient age at the time of CABG surgery (Table 2). While the Hannan model provided better predictions for patients up to age 80, this difference only reached statistical significance for the patient subgroup ≤50 years of age, the age group for which the actuarial model's predictions showed the greatest deviation from the observed mortality. Though not statistically significant, the actuarial model, which inherently weighted age more heavily within its algorithm, provided less accurate predictions for long‐term mortality for this younger patient cohort.

A special analysis was performed to evaluate if the Hannan model was potentially gender biased. As the Hannan model was developed for the entire New York State, it was unclear if it would perform better differentially on men versus women in our study. For the five‐year Hannan model predictions, the c‐indices (with confidence intervals) were calculated for our female patients (c‐index = 0.6487; [CI 0.5246, 0.7727]) and for our male patients (c‐index = 0.8056; [CI 0.7510, 0.8603]). For the five‐year c‐statistic comparisons, there does not appear to be an important Hannan model gender bias. As our single center's small sample size precluded more comprehensive assessments, however, a much larger cardiac surgery database with long‐term follow‐up would be more appropriately used to rigorously evaluate for any gender bias in using the Hannan prediction model.

CONCLUSIONS

For our 1028 SBUH records, this retrospective database analysis demonstrated that the Hannan model better predicted one‐, three‐, and five‐year post‐CABG mortality rates for the subgroup of patients with higher Hannan point scores (and thus, presumably the sicker patient subgroup) than those provided by actuarial models based on age, gender, and race/ethnicity data alone. The Hannan model performed best for predicting five‐year mortality, the longest time period analyzed here, across all point totals. Upon closer review, the Hannan model was seen to be most accurate for risk predictions for the subgroup of patients ≤50 years of age. This supports Wu et al.'s statement that their model is useful “…for informed consent and as an aid in determining treatment choice,”6 but suggests the model's greatest strength may lie in its accuracy in providing long‐term mortality estimates for younger or sicker patients. The Hannan model was not designed to be specific for cardiac‐related deaths only; therefore, no comment can be made on the likely distribution of non‐cardiac versus cardiac‐related causes of death in our patient sample.

When the difference between the Hannan model and actuarial model was assessed, preoperative renal failure had a high degree of association with the differences found between the clinical versus actuarial risk models' estimates. This was somewhat expected given the inherent weighting of renal failure within the Hannan model; however, the parameter estimates for renal failure in the multivariate model appeared to be even higher than that weighting would predict. Given our small sample size, interaction terms (e.g., the interaction between renal failure and different age thresholds) were not able to be included in the model. Among the Hannan model's risk factors evaluated, however, these findings suggest that preoperative renal dysfunction may be an even more important driver of long‐term mortality than previously thought. If confirmed, cardiac surgeons may be advised to place particular emphasis on ensuring their patients receive appropriate post‐discharge specialty care for their preexisting renal dysfunction as an important intervention that may have a particularly strong impact on patients' long‐term survival. Further research endeavors to improve their long‐term survival and accurate modeling of their unique risk profile is indicated.

Chronic renal failure has previously been shown to be an independent risk factor for death, cardiovascular morbidity, and hospitalization within the general population.16 Within the STS national adult cardiac database, the degree of preoperative renal dysfunction has also been directly correlated with short‐term mortality.2 Furthermore, the STS has documented that this important risk factor (as determined by a high serum creatinine level), is prevalent—that more than 75% patients had some degree of preoperative renal dysfunction, including more than 25% whose renal dysfunction was classified as moderate or severe.17 The relationship between post‐CABG renal failure and long‐term survival, particularly after CABG surgery, appears to have poor implications for dialysis patients. One study estimated the post‐CABG survival for dialysis patients as 44% at eight years and 36.2% at 10 years.18 Other publications have documented similarly poor long‐term outcomes for patients undergoing aortic valve replacement surgery with preexisting renal dysfunction.19, 20 This evidence suggests that it may be beneficial to assess the patients' renal functionality prior to and after surgery, to evaluate for optimal use of medications, suitability for operative management, and to consider postdischarge referral to a nephrologist when appropriate.

Given the evidence demonstrating the role of renal failure as potentially modifiable risk factors for post‐operative mortality in CABG patient, those patients (with abnormal renal function) may require extra consideration during the preoperative period (e.g., evaluation of the potential role for medical management versus interventional options); enhancing the informed consent process to assure both the surgeon and patient concur as to the “optimal” approach. Post‐CABG, moreover, there appears to be more of a need than ever for coordinating interdisciplinary, clinical team‐based, collaborative management of patients postdischarge. The establishment and/or utilization of a “heart team” represents one such multidisciplinary approach to improve the quality of postdischarge cardiovascular patient care a Class I–C recommendation in the ACCF/AHA Guidelines for Coronary Artery Bypass Graft Surgery.21, 22 Though the use of a heart team is typically constructed to include the cardiac surgeon and cardiologist, based on these findings it appears that other specialists, most notably nephrologists, may be important additions to consider in building future “heart teams.”

It should be noted that this single‐center study experience might not be broadly applicable to other institutions. Further, the multi‐year mortality information for our patients used in the analysis was only available for a subset of patients in our cardiac surgery QA database's records (due to the limited time available for follow‐up of patients with more recent surgery dates). An appropriate next step would be to perform a similar analysis using a dataset of patients with long‐term (seven‐year) follow‐up data available.

Another notable limitation is this study's reliance on the existing SSDI search engines available to identify observed mortality outcomes. Though commonly used in clinical outcomes research and shown to be >90% sensitive for death when searched by SSN, this still leaves open the possibility for error in reporting the observed mortality.23 This challenge was mitigated, in part, by verifying that there was a high concordance when searching for our patient records using different approaches. Nevertheless, the analyses conducted here are still subject to any errors contained within the SSDI itself.

Finally, it should be noted that this analysis was based upon the published Hannan long‐term mortality models. It does not preclude the possibility that other patient factors not considered for inclusion within the Hannan model may be important predictors of long‐term post‐CABG mortality. The Society of Thoracic Surgeons (STS) long‐term mortality model used for the ASCERT trial, for example, incorporated several different variables including smoking status (i.e., previous or current smoker), immunosuppressive therapy, and preoperative atrial fibrillation, as these other risk factors were identified by STS as important prognostic indicators of long‐term survival.24 Moreover, there is always the possibility that unconventional risk factors such as socioeconomic factors or psychiatric comorbidities (e.g., major depression) may have a profound, but potentially alterable, impact on patient on patient mortality.25, 26, 27

Additional supporting information may be found in the online version of this article at the publisher's website.

Figure S1. Distribution of records across Hannan point scores.

Click here for additional data file.

Figure S2. Comparison of Hannan model and actuarial model's observed mortality rates by clinical risk category.

Click here for additional data file.

Table S1. Summary of Available CABG Data by Major Risk Factors

Table S2. One‐, Three‐, and Five‐Year Mortality Findings: Observed, Hannan Risk Model and Actuarial Model Predictions

Table S3. Nested C‐index Comparison Parameters for One‐, Three‐ and Five‐Year Time Points

Table S4. One‐, Three‐, and Five‐Year Simple Linear Regression Findings: Risk Variable‐Specific Associations with Calculated Difference in Clinical‐Actuarial Risk Estimates

Table S5. One, Three‐, and Five‐Year Multiple Linear Regression Findings: Risk Variable‐Specific Associations with Calculated Difference in Clinical‐Actuarial Risk Estimates

Table S6. Nested C‐Index analysis—Parameters for Hannan and Actuary Models Versus Observed Mortality Comparisons

Table S7. Clinical Risk Calibration—High, Medium, and Low Risk Categories

Click here for additional data file.