Derivation and Validation of a Geriatric-Sensitive Perioperative Cardiac Risk Index.

BACKGROUND: Surgical patients aged 65 and over face a higher risk of cardiac complications from noncardiac surgery. The Revised Cardiac Risk Index (RCRI) and the Gupta Myocardial Infarction or Cardiac Arrest (MICA) calculator are widely used to predict this risk, but they are not specifically designed to predict MICA in geriatric patients. Our hypothesis is that a new geriatric-sensitive index, derived from geriatric data, will capture this population's unique response to risk factors. METHODS AND
RESULTS: The model was developed using the NSQIP (National Surgical Quality Improvement Program) 2013 geriatric cohort (N=584,931) (210,914 age ≥65) and validated on the NSQIP 2012 geriatric cohort (N= 485,426) (172,905 age ≥65). Least Angle Shrinkage and Selection Operator regression was used for initial variable selection. The Geriatric-Sensitive Cardiac Risk Index (GSCRI) was then evaluated in the 2012 data set. The area under the curve (AUC) was compared among the GSCRI, RCRI, and Gupta MICA in the 2012 data set. The GSCRI had an AUC of 0.76 in the validation cohort among geriatric patients. When the Gupta MICA was tested on geriatric patients in the validation cohort, a significant deterioration (≈17%) was noted, as well as a significant underestimation of the risk. The GSCRI AUC of 0.76 in the geriatric subset was significantly greater (P<0.001) than those in the RCRI (AUC=0.63) or Gupta MICA (AUC=0.70) models, outperforming the RCRI and Gupta MICA models in geriatric patients by 13% and 6%, respectively, with a ΔAUC and P-value of 0.13 (P<0.001), and 0.06 (P<0.001).
CONCLUSIONS: The GSCRI is a significantly better predictor of cardiac risk in geriatric patients undergoing noncardiac surgery.

Clinical Perspective

What Is New?

This study provides a new perioperative risk‐prediction tool designed specifically for geriatric patients.

This study demonstrates that currently used risk models of the Lee Revised Cardiac Risk Index and Gupta Myocardial Infarction and Cardiac Arrest have moderate performance in older patients and tend to underestimate the actual cardiac risk in this age group.

This study demonstrates the importance of developing and integrating geriatric‐specific risk models.

What Are the Clinical Implications?

The Geriatric‐Sensitive Perioperative Cardiac Risk Index may be used for perioperative cardiac risk stratification and evaluations for patients 65 years of age or older.

With a significant change in the demographics and an increasing older patient population undergoing surgery, this Geriatric‐Sensitive Perioperative Cardiac Risk Index could be an effective tool for cardiac risk evaluation.

There are 40 million people aged 65 and over living in the United States today.1 Although they account for just 15% of the US population, they receive one third of all inpatient surgeries.1, 2 By 2030, 72 million Americans will be 65 and over, accounting for 20% of the US population and an increasing number of surgeries.1, 3, 4 All inpatient surgery carries a risk of cardiac complications for all adult patients, regardless of age.5, 6, 7 Cardiac arrest after noncardiac surgery is associated with a hospital mortality rate of 65%.8, 9 Myocardial infarction (MI) after noncardiac surgery is associated with a hospital mortality rate of 15% to 25%.10, 11, 12 Nonfatal MI is associated with increased mortality during the first 6 months after surgery.8, 11 Older adults are more prone to MI and cardiac arrest during or after surgery.11 Researchers have developed clinical tools for estimating cardiac risk. The Revised Cardiac Risk Index (RCRI) and Gupta MI and Cardiac Arrest (MICA) calculator are widely used indices to estimate perioperative risk; however, neither tool is specifically designed to assess the risk in geriatric patients. The objectives of this study are to develop and to validate a geriatric‐sensitive cardiac risk index. Our hypothesis is that a new geriatric‐sensitive index, derived specifically from geriatric data, will capture this population's different response to risk factors.13, 14 With a growing geriatric population and a projected increase in noncardiac, usually elective, surgeries4, 15 and the association with substantial cardiac morbidity and mortality,16, 17 it becomes imperative to have accurate estimations of the cardiac risk for geriatric patients.

The aims of this study were (1) to investigate the performance of the RCRI and Gupta MICA perioperative cardiac risk models in a geriatric population, (2) to evaluate the incidence of MICA after noncardiac nonemergency surgery across the age spectrum, (3) to develop a geriatric‐sensitive perioperative cardiac risk index (GSCRI) optimized for use with geriatric patients and sensitive to the clinical and physiologic uniqueness of this population (given that prior hypothesis‐driven objectives were met), and (4) to conduct comparative performance analysis of the GSCRI, RCRI, and Gupta MICA models.

Methods

Participants

We utilized the NSQIP (National Surgical Quality Improvement Program) cohort, a multicenter database of surgical outcomes collected prospectively by trained professionals in a systematic fashion.18 At each center, a certified Surgical Clinical Reviewer collects the data using a variety of methods, including medical chart abstraction. NSQIP developed various mechanisms to ensure data quality, including establishing high interrater reliability and auditing of selected participating sites. In addition, reviewers undergo rigorous training and annual certification to ensure they have the knowledge and resources available to collect high‐quality data.18 NSQIP collected data for over 300 variables, including risk factors for postsurgical cardiac events and 30‐day postsurgery morbidity and mortality outcomes.

In this study NSQIP years 2012 (N=543 885) and 2013 (N=651 940) were used. Participants who had emergency surgery (2012=54 729; 2013=63 980) or cardiac surgery (2012=3730; 2013=3029) were excluded, leaving a sample size of 485 426 (172 905 age ≥65) in the 2012 data and 584 931 (210 914 age ≥65) in the data from 2013. The UCLA Institutional Review Board determined that the analysis of the deidentified data set was exempt from review.

Outcome

The end point of interest is intraoperative/postoperative MICA within 30 days of surgery. Cardiac arrest is defined in the NSQIP

The absence of cardiac rhythm or presence of chaotic cardiac rhythm, intraoperatively or within 30 days following surgery, that results in a cardiac arrest requiring the initiation of CPR, which includes chest compressions. Patients are included who are in a pulseless ventricular tachycardia or fibrillation in which defibrillation is performed and pulseless electrical activity arrests requiring chest compressions. Patients with automatic implantable cardioverter defibrillators that fire although the patient has no loss of consciousness should be excluded.18

Myocardial infarction is defined in the NSQIP as

An acute MI that occurred intraoperatively or within 30 days following surgery as manifested by one of the following: (1) documentation of electrocardiogram changes indicative of acute MI (one or more of the following: ST elevation >1 mm in 2 or more contiguous leads, new left bundle branch block, new q‐wave in 2 or more contiguous leads); (2) new elevation in troponin greater than 3 times the upper level of the reference range in the setting of suspected myocardial ischemia; (3) physician diagnosis of MI.18

The Revised Cardiac Risk Index

The RCRI is a previously published index of postsurgical cardiac risk that uses 6 risk factors of major cardiac complications. These risk factors are high‐risk surgery, history of ischemic heart disease, history of heart failure, history of cerebrovascular disease, diabetes mellitus requiring insulin treatment, and serum creatinine >2.0 mg/dL. The risk factors are binary (present/absent). High‐risk surgery is defined as vascular surgery and open intraperitoneal or intrathoracic procedures. Participants with no risk factors were assigned a predicted probability of 0.4%; those with 1 risk factor are assigned 1.0%, 2 risk factors 2.4%, and 3 or more risk factors 5.4%.8 There were a total of 485 426 (172 905 age ≥65) participants in the NSQIP 2012 who were able to have their RCRI scores computed.

The Gupta MICA Model

The Gupta MICA is a risk score for perioperative cardiac risk developed in the NSQIP 2007 and validated in the NSQIP 2008 data sets.19 The risk score is comprised of 5 items: the participant's functional status, American Society of Anesthesiologists (ASA) classification, creatinine levels, age, and type of surgery that will be performed. There were a total of 479 453 (170 737 age ≥65) participants in the NSQIP 2012 that had their Gupta MICA scores computed.

Statistical Analysis

Analyses were conducted in Stata version 13.1 (College Station, TX) and R version 3.1.0 (R Foundation, Vienna, Austria). Details of the specific analyses and statistical methods are presented in the descriptions of the model building process below. The 2 NSQIP data sets used in this study were assessed separately with the NSQIP 2013 used as a derivation data set and the NSQIP 2012 used as a validation data set.

Candidate Variables for a GSCRI

Candidate variables for use in the GSCRI were chosen from the NSQIP based on the previous literature and other risk indices currently in use. The variables examined were sex, high‐risk surgery, history of congestive heart failure, history of stroke, currently taking insulin, diabetes mellitus status, dialysis, being on medication to control hypertension, smoking status, history of chronic obstructive pulmonary disease, ASA classification, functional status, creatinine level, type of surgery to be performed, dyspnea, high blood urea nitrogen levels, and high‐risk surgery including vascular surgery and open intraperitoneal or intrathoracic procedures. Bivariate logistic regression models showing the odds ratio and 95% confidence intervals for each of the risk factors when included as the sole predictor of MICA are displayed in Table 1. A multivariable model of these risk factors is additionally provided in Table S1.

Table 1

Odds Ratios for Predictors of MICA in Geriatric Patients for Noncardiac Surgeries (NSQIP 2012)

Variable	Group	Age ≥65 y
		OR (95% CI)	P Value
Age	Per 1 y	1.05 (1.04, 1.06)	<0.001
Sex (Ref=Female)	Male	1.55 (1.41, 1.71)	<0.001
High‐risk surgery (Ref=No)	Yes	1.72 (1.56, 1.90)	<0.001
Hx CHF (Ref=No)	Yes	5.49 (4.45, 6.78)	<0.001
Stroke (Ref=No)	Yes	2.91 (2.35, 3.62)	<0.001
Taking insulin (Ref=No)	Yes	2.85 (2.52, 3.23)	<0.001
Diabetes mellitus (Ref=No)	Yes, not insulin dependent	1.43 (1.26, 1.63)	0.001
	Yes, insulin dependent	3.04 (2.68, 3.45)	0.001
Dialysis (Ref=No)	Yes	5.16 (4.29, 6.21)	<0.001
Medications for hypertension (Ref=No)	Yes	2.34 (2.05, 2.66)	<0.001
Smoking status (Ref=Former/Never)	Current	1.48 (1.29, 1.69)	<0.001
Hx COPD (Ref=No)	Yes	2.24 (1.96, 2.55)	<0.001
ASA class (Ref=I)	II	4.20 (1.04, 16.93)	0.044
	III	14.93 (3.73, 59.8)	<0.001
	IV	48.1 (11.9, 192.8)	<0.001
	V	81.13 (14.6, 450)	<0.001
Functional status (Ref=Independent)	Partially dependent	3.02 (2.59, 3.51)	<0.001
	Totally dependent	3.90 (2.88, 5.27)	<0.001
Creatinine category (Ref <1.5)	1.5 to 2.5	2.67 (2.33, 3.06)	<0.001
	>2.5	4.78 (4.05, 5.64)	<0.001
	Missing	0.34 (0.25, 0.45)	<0.001
Surgical category (Ref=Hernia)	Anorectal	3.90 (2.41, 6.32)	<0.001
	Aortic	7.10 (5.05, 9.97)	<0.001
	Bariatric	2.02 (0.96, 4.28)	0.065
	Brain	3.89 (2.26, 6.68)	<0.001
	Breast	0.30 (0.15, 0.59)	0.001
	ENT	1.64 (0.74, 3.62)	0.223
	Foregut/hepatopancreatobiliary	4.19 (3.07, 5.74)	<0.001
	GBAAS/intestinal	4.77 (3.52, 6.45)	<0.001
	Neck	0.50 (0.23, 1.11)	0.087
	Obstetric/gynecologic	1.13 (0.69, 1.85)	0.630
	Orthopedic	2.99 (2.22, 4.02)	<0.001
	Other abdomen	2.94 (1.79, 4.82)	<0.001
	Peripheral vascular	4.93 (3.61, 6.73)	<0.001
	Skin	3.48 (2.20, 5.51)	<0.001
	Spine	2.25 (1.55, 3.27)	<0.001
	Thoracic	4.25 (2.85, 6.34)	<0.001
	Vein	10.9 (7.96, 14.99)	<0.001
	Urology	2.39 (1.71, 3.34)	<0.001
Dyspnea (Ref=No)	Yes	2.10 (1.86, 2.38)	<0.001
BUN (Ref <30)	>30	2.98 (2.65, 3.35)	<0.001
	Missing	0.41 (0.33, 0.50)	<0.001
Laparoscopic surgery (Ref=No)	Yes	0.69 (0.56, 0.84)	<0.001

ASA Class indicates American Society of Anesthesiology Physical Status Classification System Class; BUN, blood urea nitrogen levels; CI, confidence interval; ENT, ear, nose, or throat; GBAAS; Gallbladder, appendix, adrenals or spleen; Hx CHF, history of congestive heart failure; Hx COPD, history of chronic obstructive pulmonary disease; MICA, myocardial infarction or cardiac arrest; NSQIP, National Surgical Quality Improvement Program; OR, odds ratio.

Variable Selection

In order to develop an index that is more specific to geriatric patients, the aforementioned candidate predictors were used in a Least Angle Shrinkage and Selection Operator regression analysis in the NSQIP 2013 derivation data set on the geriatric subset20 implemented in the R package glmnet. Tenfold cross‐validation was used to select the appropriate shrinkage parameter, which was determined to be 0.00001801. Because of the size of the NSQIP data set and known theoretical underpinnings of the candidate variable list, no variable had coefficients shrunk to 0 completely. As a result, in order to develop a parsimonious model, predictors with shrunken coefficients greater than 0.7 were selected for use in the final model in addition to variables with known clinical importance. This value of 0.7 was chosen based on the scree plot of the rank‐ordered shrunken coefficients as the inflection point, after which the difference in coefficient magnitudes between successive variables was negligible. The developed GSCRI will thus represent a hybridization of data‐driven approaches and clinical insight. Table 2 summarizes the variables used in each risk model. Description of variables from NSQIP 2012 can be found in Table S2.

Table 2

Summary of the Variables Used in Each Risk Model

Risk Factors	Existing Indices		Study Index
	RCRI	Gupta	GSCRI
High‐risk surgery	✓
Type of surgery		✓	✓
Heart failure	✓		✓
Ischemic heart disease	✓
Cerebrovascular disease	✓
Stroke history			✓
Diabetes mellitus requiring insulin	✓
Diabetes mellitus status			✓
Creatinine level ≥2.0 mg/dL	✓
Creatinine level ≥1.5 mg/dL		✓	✓
ASA classification		✓	✓
Functional status		✓	✓
Age		✓

ASA classification indicates American Society of Anesthesiology Physical Status Classification System Class; GSCRI, Geriatric‐Sensitive Cardiac Risk Index; Gupta, Gupta Myocardial Infarction and Cardiac Arrest; RCRI, Lee Revised Cardiac Risk Index,

Model Building

The variables identified for inclusion from the Least Angle Shrinkage and Selection Operator model were history of stroke, ASA classification, and type of surgery. Additional clinically relevant variables such as functional status, creatinine level (>1.5), diabetes mellitus status, and a history of congestive heart failure were also selected for the final model. These additional variables have been demonstrated to be important factors in risk prediction for MICA.5, 19 These variables, having been selected to be sensitive to geriatric patients, were then used in a logistic regression model in the NSQIP 2013 derivation data to predict MI and cardiac arrest in the geriatric subset. Table 3 shows the final GSCRI model.

Table 3

Coefficients From Variables Selected by Least Angle Shrinkage and Selection Operator and Hypothesis in Noncardiac Surgery (NSQIP 2013)

Variable	Group	Noncardiac Surgery
		ln (OR) (95% CI)	P Value
Stroke (Ref=No)	Yes	2.08 (1.81, 2.35)	<0.001
ASA class (Ref=I)	II	0.28 (−0.61, 1.17)	0.539
	III	1.34 (0.46, 2.23)	0.002
	IV	2.04 (1.15, 2.93)	<0.001
	V	3.63 (2.52, 4.74)	<0.001
Surgical category (Ref=Hernia)	Anorectal	1.02 (0.59, 1.46)	<0.001
	Aortic	1.32 (1.02, 1.63)	<0.001
	Bariatric	0.31 (−0.34, 0.95)	0.348
	Brain	0.24 (−0.35, 0.83)	0.426
	Breast	−1.14 (−1.74, −0.55)	<0.001
	ENT	0.32 (−0.42, 1.06)	0.393
	Foregut/hepatopancreatobiliary	1.03 (0.75, 1.31)	<0.001
	GBAAS/intestinal	1.13 (0.87, 1.40)	<0.001
	Neck	−0.04 (−0.58, 0.49)	0.869
	Obstetric/gynecologic	0.12 (−0.32, 0.56)	0.587
	Orthopedic	0.47 (0.20, 0.73)	<0.001
	Other abdomen	0.16 (−0.32, 0.63)	0.513
	Peripheral vascular	0.82 (0.55, 1.10)	<0.001
	Skin	0.41 (−0.03, 0.84)	0.065
	Spine	0.42 (0.08, 0.75)	0.014
	Thoracic	1.06 (0.71, 1.41)	<0.001
	Vein	1.35 (1.06, 1.64)	<0.001
	Urology	0.55 (0.26, 0.85)	<0.001
Functional status (Ref=independent)	Partially dependent	0.23 (0.06, 0.41)	0.007
	Totally dependent	0.72 (0.43, 1.01)	<0.001
Creatinine category (Ref <1.5)	>1.5	0.57 (0.45, 0.70)	<0.001
	Missing	−0.41 (−0.68, −0.14)	0.002
Hx CHF (Ref=No)	Yes	0.60 (0.39, 0.80)	<0.001
Diabetes mellitus (Ref=No)	Yes, not insulin dependent	0.09 (−0.04, 0.22)	0.164
	Yes, insulin dependent	0.47 (0.34, 0.60)	<0.001
Constant		−6.79 (−7.70, −5.89)	<0.001

ASA Class indicates American Society of Anesthesiology Physical Status Classification System Class; CI, confidence interval; GBAAS, Gallbladder, appendix, adrenals or spleen; Hx CHF, history of congestive heart failure; NSQIP, National Surgical Quality Improvement Program; OR, odds ratio.

Model Evaluation

For comparison with the previously published risk scores, the coefficients from the 2013 derivation data were then used to predict the risk of MI in the NSQIP 2012 validation data set. There were a total of 485,426 participants in the NSQIP 2012 who were able to have this GSCRI computed. Previous published coefficients of the RCRI and Gupta MICA risk indices were used to predict the risk of postsurgical MI in the NSQIP 2012 data set. The predictive value of the models was used to calculate the area under the receiver operating characteristics curve for the overall sample and within the geriatric (age ≥65) age group and compared between models using the Delong method in the pROC package in R version 3.1.0. Plots of observed versus predicted risk (calibration plots) were used to visually assess the fit of these models in the geriatric group.

Evaluation of Upper Limit for Model Performance

Although we believe in the clinical utility of a parsimonious model for predicting MICA, modern computational algorithms can be used to maximize the predictive ability of the predictors by modeling complex nonlinear and high‐order interactions. Often called machine or statistical learning, these techniques provide greater predictive accuracy, though often at the expense of interpretability. In our study we used 1 such technique, stochastic gradient boosting, in order to provide us with an upper‐limit benchmark for our GSCRI model performance. Using a gradient‐boosted machines (GBM) model21, 22; with the gbm package in R,23 we evaluated the set of predictors from the RCRI, Gupta MICA, and GSCRI in the geriatric subset of the NSQIP 2013 derivation data set. Tenfold cross validation was used to select the interaction depth and number of iterations that minimized the cross‐validated error at a shrinkage rate of 0.01. The optimal model, once selected using cross validation, was then applied to the 2012 validation data set to predict MICA risk. Area under the curve (AUC) was computed from this model and used as our likely upper limit of predictive ability from the set of predictors under evaluation.

Results

In the NSQIP 2012 validation data set the majority of the sample was female (58%) and middle‐aged (age mean=57, SD=16), with few instances of perioperative MI (N=2357, ≈0.5%).

The odds of MICA were 4.8 times greater in those 65 or over (≈1% versus ≈0.2%). Figure 1 shows how the risk of MICA increases nonlinearly with age. Among geriatric patients, the risks of postoperative cardiac arrest and MI were 0.37% and 0.67%, respectively. The risk of death in this group was 1.64%, although these models do not evaluate this risk. Sample characteristics were not substantively different in the NSQIP 2013 derivation data. Clinical characteristics of the derivation and validation cohorts are displayed in Table S3.

Figure 1

Association of actual MICA incidence with age along with the linear and lowess fits. MICA indicates myocardial infarction and cardiac arrest.

Development of the GSCRI

The coefficients from the GSCRI in the NSQIP 2013 data set are displayed in Table 3. All variables were statistically significant predictors of MICA (P<0.05). Figure 2 shows the relative importance of each variable to the GSCRI in the derivation set (NSQIP 2013) as measured by partial χ2 minus the predictor degrees of freedom. ASA Class and surgical category were the most influential variables in the model, while congestive heart failure, diabetes mellitus, and functional status contribute almost equally and were least influential in forming the final index.

Figure 2

Importance of each variable to the GSCRI in the training set as measured by partial χ2 minus the predictor degrees of freedom. ASA Class indicates American Society of Anesthesiology Physical Status Classification System Class; GSCRI, Geriatric‐Sensitive Cardiac Risk Index; Hx CHF, history of congestive heart failure.

Comparison of Risk Scores

AUCs for each of the models can be found in Table 4. The GSCRI had a significantly higher AUC than either the RCRI or Gupta MICA in both the geriatric group (AUC=0.76) and overall sample (AUC=0.83). The Gupta MICA also outperformed the RCRI in the geriatric group (0.70 versus 0.63) and overall sample (0.72 versus 0.68). We additionally found the Gupta MICA model to be poorly calibrated (Figure 3), with an underestimation of risk in the geriatric sample. Although both the RCRI and GSCRI also underestimated the risk, the median difference from the observed risk was only −0.28 and −0.04 percentage points different, respectively, whereas the Gupta MICA was off by −0.73% in the geriatric patients. In Figure 4 we see that the Gupta MICA model severely underestimates the risk in the low‐risk categories and overestimates the risk in high‐risk categories. Given this, we find the Gupta MICA model to be poorly calibrated in the NSQIP 2012 validation data. Although the RCRI also underestimates risk for low‐risk patients, it is well calibrated for the highest‐risk group (2.4% to 5.4% in RCRI). The GSCRI is very well calibrated and tends to only slightly overestimate the risk in the highest predicted risk group. For the performance of risk prediction in geriatric patients for noncardiac surgery groups by age, see Table S4. Additionally, to compare the performance of this new risk score GSCRI with the RCRI as well as the Gupta MICA risk scores, net reclassification index (5% threshold) and integrated discrimination improvement metrics were compared, and the GSCRI provides an improvement over both the RCRI and Gupta MICA (see Table S5).

Table 4

Differential Performance of Risk Prediction in Geriatric Patients for Noncardiac Surgeries (NSQIP 2012)

	Age ≥65 y AUC (95% CI)	Overall AUC (95% CI)
	Δ, P Value	Δ, P Value
RCRI	0.63 (0.62, 0.65)	0.68 (0.67, 0.69)
Gupta MICA	0.70 (0.69, 0.71)	0.72 (0.71. 0.73)
GSCRI	0.76 (0.75, 0.77)	0.83 (0.82, 0.83)
RCRI vs Gupta MICA	0.07, P<0.001	0.04, P<0.001
GSCRI vs RCRI	0.13, P<0.001	0.15, P<0.001
GSCRI vs Gupta MICA	0.06, P<0.001	0.11, P<0.001

AUC indicates area under the curve; CI, confidence interval; Δ, difference between indices; GSCRI, Geriatric‐Sensitive Cardiac Risk Index; Gupta MICA, Gupta Myocardial Infarction and Cardiac Arrest; NSQIP, National Surgical Quality Improvement Program; RCRI, Lee Revised Cardiac Risk Index.

Figure 3

Models' calibration among geriatric patients. Calibration plots compare the 3 studied models among geriatric patients. Gupta MICA model shows underestimation of the MICA risk in geriatric patients. GSCRI indicates Geriatric‐Sensitive Cardiac Risk Index; Gupta MICA, Gupta myocardial infarction and cardiac arrest; NSQIP, National Surgical Quality Improvement Program; RCRI, Lee Revised Cardiac Risk Index.

Figure 4

Alternate calibration plot. The average observed risk is presented for each of the predicted risk categories (defined by the RCRI). GSCRI indicates Geriatric‐Sensitive Cardiac Risk Index; Gupta MICA, Gupta myocardial infarction and cardiac arrest; RCRI, Lee Revised Cardiac Risk Index.

Upper Limit of Model Performance

The GBM model indicated an optimal interaction depth of 3 and 3086 iterations and was developed in the NSQIP 2013 derivation data set. When this model was applied to the 2012 validation data set, the AUC was found to be 0.79, indicating a likely upper limit of performance for these variables on geriatric patients in the NSQIP data sets. The GSCRI, with an AUC of 0.76, approaches this upper limit while maintaining parsimony.

Discussion

This article demonstrates the concept of developing a predictive model in the geriatric population, in contrast to other articles where the model development is across a wider age spectrum. Currently, geriatric patients have low participation in clinical trials and are often excluded due to age‐related comorbidities. When included, the data of older participants are often pooled together with participants of younger ages who have much lower risk, which possibly leads to inaccurate parameter estimation. Developing predictive models on these pooled data that ignore age categories can lead to models that are dominated by variables and coefficients not optimized for performance in geriatric patients and hence provide decreased predictive accuracy and lower sensitivity to certain geriatric characteristics. This holds true especially in the development of predictive models because even minor inaccuracies in the derivation of the parameter estimates could dramatically affect the discrimination and calibration of a model.

In this study our hypothesis of the need for specific geriatric analysis and model derivation proved to be valid, and our efforts culminated in producing the Geriatric‐Sensitive Perioperative Cardiac Risk Index, GSCRI. Our GSCRI has an AUC of 0.76 and outperformed the RCRI and Gupta MICA models by 13% (P<0.001) and 6% (P<0.001) in geriatric patients of the validation cohort, respectively (see Table 4). Although the GSCRI was developed for optimal performance in a geriatric population, we wished to test the GSCRI against the RCRI and Gupta MICA in the overall population as well. We found that the GSCRI has an AUC of 0.83, which outperformed the RCRI and Gupta MICA by 15% (P<0.001) and 11% (P<0.001), respectively (Table 4). When the Gupta MICA was tested with the published coefficients on geriatric patients, a significant deterioration (≈17%) from the previously published performance in the NSQIP 2007 was noted, and a significant underestimation of the risk was also noted, likely resulting from assuming a linearity of age and deriving estimates that are not specific for the geriatric population when conducting the analysis for the Gupta MICA calculator.

The GSCRI model contained 7 variables, and the first 3 variables (stroke, ASA Class, surgical category) were selected using a Least Angle Shrinkage and Selection Operator regression analysis in the NSQIP 2013 data on the geriatric subset. The method selects the most statistically important variables that contribute to the occurrence of the outcome. The other variables (diabetes mellitus, functional status, elevated creatinine >1.5 mg/dL, congestive heart failure) were added to include clinically significant variables that are common across various indices of perioperative cardiac risk.5, 19 Although additional relevant variables could have been added, the increased model complexity would not meaningfully improve the model's predictive ability based on our examination of the upper limit of model performance from the candidate variables. Creating a parsimonious model was essential to ensure the ease of use that physicians working in clinical settings require.

Our data‐driven variable selection method (Least Angle Shrinkage and Selection Operator) selected 3 variables, but we felt that the addition of common clinically important risk factors to the model would improve estimation of the MICA risk in geriatric patients. The inclusion of additional risk factors that are known to increase the surgical cardiac risk, such as congestive heart failure,5 elevated creatinine,5, 19 diabetes mellitus,5, 19 and functional status,19 may be important in generalizing the GSCRI to novel samples outside of NSQIP. In modeling these variables, we chose a creatinine level of 1.5 mg/dL as opposed to 2 mg/dL because geriatric patients often have a decreased glomerular filtration rates with lower serum creatinine levels in comparison with younger patients.24 The performance of the GBM achieved a performance of AUC=0.79, indicating that a complex model containing higher‐order interactions would achieve a performance close to our model (ΔAUC=3%); therefore, we believe our model was able to achieve good performance without the loss of interpretability common to statistical learning algorithms such as GBM.

These models reflect contemporary risk associated with each surgical category; hence, updating these models every few years is imperative to take into account the improved surgical outcomes and decreased complication rates that result from enhanced medical care and improved surgical techniques. This possibly explains why the GSCRI outperformed the other 2 models in nongeriatric patients, as it has the advantage of being tested on a data set only 1 year apart from the derivation data set, whereas the Gupta MICA was developed on a 2007 data set.19 Additionally, the RCRI was not derived to predict the cardiac risk within 30 days of surgery but is aimed solely at predicting the risk during a hospital stay.5 In the modern world, using equations developed so long ago and on unique populations (ie, RCRI) is of questionable value, particularly in an era when curated data sources such as the NSQIP and other large data sets are readily available.

With the growth of the geriatric population and increased awareness of the uniqueness of this growing segment,11, 25 The GSCRI represents a step forward for cardiac risk prediction for geriatric patients. Our study demonstrated the necessity of developing risk models optimized for geriatric patients in order to produce accurate predictions. The GSCRI outperforms the Gupta MICA and RCRI in the AUC by 7% and 13%, respectively.

We believe we might have reached a predictive limit in our ability to predict perioperative risk in geriatric patients in our sample. GBM, an exploratory statistical learning technique, was used to examine the maximal predictive ability for the set of predictors available in the NSQIP 2012 data set. One strength of this technique is the ability to utilize nonlinear and high‐order interactions that provide the maximal predictive accuracy for the outcome given the data set; however, it comes at the expense of interpretability. Even with the complex GBM modeling we could not reach a C‐statistic that was >0.8 in the geriatric patients. This low value is not unexpected, especially given the wide variation in the health status of geriatric patients. We may need to consider other variables for predicting the risk in geriatric patients in order to achieve more accurate predictions; however, we are currently limited by the variables that are available in the NSQIP databases.

The findings of the study were driven by our initial hypothesis, and therefore, the GSCRI represents a hybrid of hypothesis and data‐driven approaches. We hope our novel index may help set a new standard in surgical risk estimation for geriatric patients. To facilitate that purpose, we intend to develop an online calculator to increase the utility of the GSCRI for physicians. Physicians will be able to readily estimate the cardiac surgical risk for a geriatric patient by answering 7 online questions, and the index will produce cardiac risk probability by integrating these answers into the equation of the GSCRI from Table 3. Finally, we would like to stress that the GSCRI should be accompanied by clinical evaluation and comprehensive geriatric assessment to add further insights to the actual risk. Because no risk model can substitute for the clinical judgment of physicians, the GSCRI is meant to be a supplemental tool to aid in the process of perioperative cardiac risk management.

Our study has certain limitations that need to be considered. There is a paucity of geriatric‐specific data available in the data set. Additional variables could provide further prognostication and predictive value that is relevant for this population. Therefore, this model is the first of a series of models that will need to be updated by integrating geriatric‐relevant data in a timely manner. Hence, our future endeavors will focus on integrating and testing the usability of biologic variables such as inflammatory factors and other significant factors including nutritional status, functional status, depression, cognition, and frailty indices.2, 11, 26, 27, 28 In addition, the external validation was conducted using a different data‐set year from the same organization: NSQIP. Additional studies further validating this model using data from different external data sets are warranted. An additional limitation was the finding that functional status had a small impact on the outcome, perhaps because fewer than 1% of the patients were “totally dependent,” but also, perhaps, we need a much better classification system for functional status than the simple 3 categorical variables, a true limitation of the available data.

In conclusion, current risk‐prediction models have poor performance in geriatric patients; therefore, we developed a new index, the GSCRI, to help improve the accuracy of prediction in this unique population. The GSCRI as a new tool is a work in progress, and we expect to continuously update and improve this new index as more data become available.

Sources of Funding

This research was supported by the National Institutes of Health, National Center for Advancing Translational Sciences (NCATS), University of California, Los Angeles (UCLA), Clinical and Translational Science Institute (CTSI) Grant Number UL1TR001881. This material is based on work supported by the office of Academic Affiliations, Department of Veterans Affairs.

Disclosures

None. The contents do not represent the views of the US Department of Veterans Affairs, National Institutes of Health (NIH), or the US Government. The American College of Surgeons National Surgical Quality Improvement Program and the hospitals participating in the ACS NSQIP are the source of the data used herein; they have not verified and are not responsible for the statistical validity of the data analysis or the conclusions derived by the authors.

Table S1. Multivariable Model With Odds Ratios for Predictors of MICA in Age ≥65 for Noncardiac Surgeries (NSQIP 2012)

Table S2. Description of Variables From NSQIP 2012

Table S3. Clinical Characteristics of the Derivation and Validation NSQIP Cohorts

Table S4. Performance of Risk Prediction in Geriatric Patients for Noncardiac Surgeries Groups by Age (NSQIP 2012)

Table S5. NRI and IDI Metrics in NSQIP 2012, Age ≥65

Click here for additional data file.