Derivation and Validation of a Novel Cardiac Intensive Care Unit Admission Risk Score for Mortality.

Background There are no risk scores designed specifically for mortality risk prediction in unselected cardiac intensive care unit (CICU) patients. We sought to develop a novel CICU-specific risk score for prediction of hospital mortality using variables available at the time of CICU admission. Methods and Results A database of CICU patients admitted from January 1, 2007 to April 30, 2018 was divided into derivation and validation cohorts. The top 7 predictors of hospital mortality were identified using stepwise backward regression, then used to develop the Mayo CICU Admission Risk Score (M-CARS), with integer scores ranging from 0 to 10. Discrimination was assessed using area under the receiver-operator curve analysis. Calibration was assessed using the Hosmer-Lemeshow statistic. The derivation cohort included 10 004 patients and the validation cohort included 2634 patients (mean age 67.6 years, 37.7% females). Hospital mortality was 9.2%. Predictor variables included in the M-CARS were cardiac arrest, shock, respiratory failure, Braden skin score, blood urea nitrogen, anion gap and red blood cell distribution width at the time of CICU admission. The M-CARS showed a graded relationship with hospital mortality (odds ratio 1.84 for each 1-point increase in M-CARS, 95% CI 1.78-1.89). In the validation cohort, the M-CARS had an area under the receiver-operator curve of 0.86 for hospital mortality, with good calibration (P=0.21). The 47.1% of patients with M-CARS <2 had hospital mortality of 0.8%, and the 5.2% of patients with M-CARS >6 had hospital mortality of 51.6%. Conclusions Using 7 variables available at the time of CICU admission, the M-CARS can predict hospital mortality in unselected CICU patients with excellent discrimination.

Clinical Perspective

What Is New?

We developed and validated the Mayo CICU Admission Risk Score (M‐CARS), a novel hospital mortality risk score for prognostic application in cardiac intensive care unit patients.

The M‐CARS is a 10‐point integer score incorporating variables readily available early after presentation, including admission diagnoses (shock, cardiac arrest, and respiratory failure), the Braden Skin Score (a proposed measure of frailty), and relevant laboratory values (anion gap, blood urea nitrogen, and red blood cell distribution width).

The M‐CARS provided robust stratification of hospital mortality risk, separating patients into low‐ (score <2), intermediate‐ (score 2–4), and high‐risk (score >4) categories.

What Are the Clinical Implications?

If prospectively validated, the M‐CARS could be used to provide rapid mortality risk stratification at the time of, or potentially before, cardiac intensive care unit admission to facilitate patient triage and recognition of high‐risk cardiac intensive care unit patients.

Low‐risk patients (M‐CARS <2, corresponding to <1% observed hospital mortality) could be considered for either admission to an intermediate care setting or early cardiac intensive care unit dismissal, while the M‐CARS could enable more accurate outcome prognostication and early discussions regarding goals of care in high‐risk patients.

Introduction

Mortality risk stratification in acutely ill patients with cardiac disease dates back to the earliest days of the coronary care unit.1 Diagnosis‐specific risk scores have been validated for patients with acute coronary syndromes (ACS) and heart failure (HF).2 Furthermore, prognostic scoring systems for general intensive care unit (ICU) patients have also been validated.2, 3, 4, 5 Risk stratification facilitates patient triage, optimizes resource allocation, and benchmarks outcomes.6, 7, 8 Development and validation of prediction models were identified in a recent expert consensus review as research priorities for the field of critical care cardiology.8 The need for novel risk scores in the cardiac intensive care unit (CICU) population is emphasized by the increasingly complex and heterogeneous mix of patients with extensive cardiac and noncardiac comorbidities served by the modern CICU.9, 10, 11

To date, no CICU‐specific risk score has been developed for the purposes of mortality risk prediction among unselected CICU patients, and there are no established tools available that utilize information readily available at admission to identify and triage low‐risk patients. Established ICU severity of illness scores such as the Sequential Organ Failure Assessment (SOFA) and Acute Physiology and Chronic Health Evaluation (APACHE) scores have demonstrated very good discrimination for hospital mortality in CICU populations, with area under the receiver‐operator characteristic curve (AUC) values of 0.8 or greater.11, 12, 13, 14 However, these established ICU risk scores have important limitations, including the need to accumulate 24 hours of data after ICU admission to provide optimal risk prediction, as opposed to utilizing information available at the time of ICU admission.3 Although a SOFA score <2 on the first CICU day was associated with a low risk of hospital and postdischarge mortality in a prior study of CICU patients, the need for 24 hours of clinical data precludes the use of the SOFA score for CICU triage guidance. In order to be useful for CICU triage, a risk score would need to be calculated using data available at the time of potential CICU admission.12

We tested the hypothesis that we could derive a simple, parsimonious integer risk score in unselected contemporary CICU patients using no more than 7 variables available at the time of CICU admission that could accurately stratify the risk of hospital mortality with discrimination for hospital mortality comparable to established ICU risk scores that were calculated on the first CICU day.

Methods

Study Design and Participants

The data, analytic methods, and study materials will not be made available to other researchers for purposes of reproducing the results or replicating the procedure. This study was approved by the Institutional Review Board of Mayo Clinic as posing minimal risk to patients, and therefore was performed under a waiver of informed consent. As the derivation cohort for development of the risk score, we used an existing database of consecutive unique adult patients (≥18 years of age) admitted to the CICU at Mayo Clinic Hospital St. Mary's Campus between January 1, 2007 and December 31, 2015, as previously reported.11, 12, 13 Only data from the first CICU admission during this period were considered, and patients admitted before January 1, 2007 or discharged after December 31, 2015 were excluded from this derivation cohort. After development of the risk score (as described below), a validation cohort was constructed by including consecutive unique adult patients (≥18 years of age) admitted to the CICU at Mayo Clinic Hospital St. Mary's Campus between January 1, 2016 and April 30, 2018; only data from the first CICU admission during this time period were considered, but eligible patients from the derivation cohort could be included. Patients admitted before January 1, 2016 or discharged after April 30, 2018 were excluded from this validation cohort. According to Minnesota state law statute 144.295, patients must provide authorization in order to be included in observational research studies; patients who did not provide Minnesota Research Authorization were excluded from the study. The Mayo Clinic CICU is a 16‐bed unit caring for critically ill patients with cardiovascular disease, which does not admit postoperative cardiac surgery patients, patients receiving extracorporeal membrane oxygenator support, or durable mechanical circulatory support devices. Patients are cared for by a team of residents and fellows led by an attending cardiologist, with consultation by a critical care medicine provider for patients with respiratory failure.

Data Source and Definitions

Demographic, vital sign, laboratory, and other clinical and outcome data were extracted electronically from the medical record, including procedures and therapies performed during the CICU and hospital stay.11, 12, 13 The admission value of all vital signs, clinical measurements, and laboratory values was used, defined as either the first value recorded after CICU admission or the value recorded closest to CICU admission. Admission diagnoses were based on all International Classification of Diseases, Ninth Revision (ICD‐9) diagnosis codes recorded on the day of CICU admission and the day before or after CICU admission; these admission diagnoses were not mutually exclusive and the primary admission diagnosis could not be determined. The APACHE‐III score, APACHE‐IV predicted hospital mortality, Day 1 SOFA score, and Oxford Acute Severity of Illness Score (OASIS) were calculated for all patients using data from the first 24 hours of CICU admission, with missing variables imputed as normal as the default.11, 12, 13, 15, 16, 17 The Charlson Comorbidity Index and individual comorbidities were determined based on a previously validated electronic algorithm.18

Statistical Analysis

The primary outcome was all‐cause hospital mortality and the secondary outcome was CICU mortality, based on electronic review of the medical record. To identify predictors of hospital mortality in the derivation cohort, stepwise backward logistic regression was performed until no more than 7 predictor variables remained. Candidate predictor variables included demographics, comorbidities, critical care therapies, laboratory data, diagnoses, and vital signs from the time of CICU admission; only variables with data available in >75% of patients were considered as candidate variables. Each continuous predictor variable from this multivariable model was converted to a categorical variable based on values above or below the optimal cutoff for each variable, defined as the highest value of Youden's J index (sensitivity+specificity−1) on receiver‐operator characteristic (ROC) curve analysis for that variable in the derivation cohort. These categorical variables were re‐entered into a multivariable logistic regression model and the beta coefficients from this final model were used to determine relative weighting (either 1 or 2 points) for abnormal values of each predictor variable by normalizing each coefficient to the lowest coefficient. The assigned points were used to create a novel discrete integer risk score, called the Mayo CICU Admission Risk Score (M‐CARS), ranging from 0 to 10 points. As is customary for prognostic scoring systems, missing variables to calculate the M‐CARS were imputed as normal (0 points).

Groups were compared using Student t test for continuous variables (reported as mean) and Pearson χ2 test for categorical variables (reported as %). Logistic regression was used to calculate odds ratio and AUC (c‐statistic) values for hospital mortality, to determine calibration for prediction of hospital mortality using the Hosmer–Lemeshow statistic (with nonsignificant P values reflecting good calibration) and to determine overall predictive accuracy for hospital mortality using the Brier score (with lower scores reflecting better accuracy). AUC CI for the Mayo CICU Admission Risk Score (M‐CARS) and existing ICU risk scores for discrimination of hospital mortality were calculated via 2000 bootstrap samples, and AUC values were compared using the DeLong test. Statistical analyses were performed using JMP version 13.0 Pro (SAS Institute, Cary, NC) and R version 3.4.2 (https://www.r-project.org/). P<0.05 were considered statistically significant.

Results

Study Population

The derivation cohort included 10 004 patients, after exclusion of 2632 of 12 618 adult admissions between January 1, 2007 and December 31, 2015 (1877 readmissions and 755 without Minnesota Research Authorization; Figure 1A).11, 12, 13 The validation cohort included 2634 patients, after exclusion of 677 of 3311 adult admissions between January 1, 2016 and April 30, 2018 (385 readmissions and 292 without Minnesota Research Authorization; Figure 1B); this included 210 duplicated patients from the derivation cohort. Combined, the derivation and validation cohorts included 12 638 total patient admissions in the overall study population. Patients in the validation cohort were older (68.4 versus 67.4 years, P=0.005) and differed from patients in the derivation cohort in terms of comorbidities, illness severity, admission diagnoses, and CICU therapies (Table 1). In‐hospital death occurred in 908 (9.1%) patients in the derivation cohort, including 570 (5.7%) who died in the CICU; in‐hospital death occurred in 258 (9.8%) patients in the validation cohort, including 155 (5.9%) in the CICU (P>0.1 for comparison of CICU and hospital mortality between cohorts).

Figure 1

Flow diagram demonstrating inclusion and exclusion criteria for the derivation (A) and validation (B) cohorts.

Table 1

Baseline Characteristics of Derivation and Validation Cohorts

Variable	Derivation Cohort (n=10 004)	Validation Cohort (n=2634)	P Value
Baseline characteristics
Age, y	67.4±15.2	68.4±15.1	0.005
Female sex	3746 (37.4%)	1013 (38.5%)	0.34
White race	9236 (92.3%)	2433 (92.4%)	0.94
Comorbidities
CCI	2.4±2.6	2.5±2.7	0.009
Prior MI	1980 (19.8%)	416 (15.8%)	<0.001
Prior heart failure	1953 (19.6%)	672 (25.5)	<0.001
Prior stroke	1229 (12.3%)	295 (11.2%)	0.12
Prior CKD	2031 (20.4%)	621 (23.6%)	<0.001
Prior diabetes mellitus	2837 (28.4%)	795 (30.2%)	0.08
Prior cancer	2135 (21.4%)	538 (20.4%)	0.28
Prior lung disease	1944 (19.5%)	522 (19.8%)	0.70
Prior dialysis	571 (5.7%)	61 (2.3%)	<0.001
Severity of illness scores
APACHE‐III score	61.0±25.3	59.7±23.9	0.012
APACHE‐IV predicted mortality	0.1690	0.1662	0.49
OASIS	25.3±10.3	26.5±10.1	<0.001
Day 1 SOFA score	3.4±3.2	3.7±3.3	<0.001
Max wk 1 SOFA	3.9±3.3	4.2±3.5	<0.001
Mean wk 1 SOFA	3.0±2.6	3.1±2.6	0.024
CICU therapies and procedures
Invasive ventilator	1607 (16.1%)	457 (17.4%)	0.11
Noninvasive ventilator	1489 (14.9%)	491 (18.6%)	<0.001
Vasopressors	2090 (20.9%)	626 (23.8%)	0.001
Inotropes	928 (9.3%)	246 (9.3%)	0.92
RBC transfusion	1173 (11.7%)	243 (9.2%)	<0.001
Dialysis in CICU	487 (4.9%)	120 (4.6%)	0.50
Admission diagnoses
Acute coronary syndrome	4267 (43.1%)	1060 (40.2%)	0.008
Heart failure	4564 (46.1%)	1609 (61.1%)	<0.001
Cardiac arrest	1193 (12.0%)	297 (11.3%)	0.27
Shock	1349 (13.6%)	541 (20.5%)	<0.001
Respiratory failure	2079 (21.0%)	1002 (38.0%)	<0.001

Data presented as number (%) for continuous variables and mean±SD for categorical variables. P value is for the comparison of patients in the derivation and validation cohort using t test for continuous variables and χ2 test for categorical variables. A total of 210 duplicated patients are included in both cohorts. Admission diagnoses are not mutually exclusive. APACHE indicates Acute Physiology and Chronic Health Evaluation; CCI, Charlson Comorbidity Index; CICU, cardiac intensive care unit; CKD, chronic kidney disease; MI, myocardial infarction; OASIS, Oxford Acute Severity of Illness Score; RBC, red blood cell; SOFA, Sequential Organ Failure Assessment.

Multivariable Analysis

The final 7 predictors of hospital mortality in the derivation cohort by stepwise backward regression were admission Braden skin score, admission red blood cell distribution width (RDW), admission blood urea nitrogen (BUN), admission serum anion gap, admission diagnosis of cardiac arrest, admission diagnosis of shock (any type), and admission diagnosis of respiratory failure (Table 2). The AUC value for the final logistic regression model in the validation cohort was 0.90 using Braden skin score, anion gap, RDW, and BUN as continuous variables and 0.89 when these were converted to categorical variables based on the optimal cutoff values.

Table 2

Multivariable Logistic Regression Model for Prediction of Hospital Mortality Using the Final 7 Categorical Variables

Variable	Beta (SE)	OR (95% CI)	P Value
Cardiac arrest	0.560 (0.068)	3.178 (2.550–3.961)	<0.001
Shock	0.582 (0.063)	3.093 (2.522–3.795)	<0.001
Respiratory failure	0.338 (0.064)	2.145 (1.745–2.638)	<0.001
Braden score ≤15	0.481 (0.061)	2.606 (2.135–3.182)	<0.001
Anion gap >14	0.267 (0.062)	1.912 (1.571–2.327)	<0.001
RDW >14.3	0.359 (0.063)	2.145 (1.743–2.639)	<0.001
BUN >23	0.413 (0.063)	2.334 (1.898–2.869)	<0.001

Beta represents the beta coefficient and SE denotes standard error for the logistic regression model. BUN indicates blood urea nitrogen; OR, odds ratio; RDW, red blood cell distribution width.

Development of the Mayo CICU Admission Risk Score

Using the beta coefficients of the final logistic regression model (Table 2) to determine weighting, the M‐CARS (range=0–10) was developed (Table 3): (1) 2 points each were assigned for an admission diagnosis of cardiac arrest or shock and 1 point was assigned for an admission diagnosis of respiratory failure; (2) 2 points were given for a Braden skin score ≤12 and 1 point for a Braden skin score 13–15; (3) 1 point was assigned each for an elevated BUN, anion gap, or RDW. A total of 1380 (13.8%) patients in the derivation cohort and 196 (8.0%) patients in the validation cohort had missing data for at least 1 variable in the M‐CARS (Table 4). All further analyses were performed after missing data for calculating the M‐CARS were imputed as normal (score of 0). The distribution of the M‐CARS differed between the derivation and validation cohorts (P<0.001; Figure 2), with a lower mean value of the M‐CARS in the derivation cohort (2.0 versus 2.7; P<0.001). The prevalence of abnormal values of each variable in the M‐CARS differed between the derivation and validation cohorts, except for elevated BUN and admission diagnosis of cardiac arrest (Table 4); this difference was largest for elevated anion gap (17.8% in the derivation cohort and 53.6% in the validation cohort).

Table 3

Calculating the M‐CARS Based on Points Assigned for Each Risk Factor

Variable	Value	Points Assigned
Admission value of BUN	>23 mg/dL	1
	≤23 mg/dL	0
Admission value of anion gap	>14	1
	≤14	0
Admission Braden skin score	≤12	2
	13–15	1
	>15	0
Admission value of RDW	>14.3	1
	≤14.3	0
Admission diagnosis of cardiac arrest	Yes	2
	No	0
Admission diagnosis of shock	Yes	2
	No	0
Admission diagnosis of respiratory failure	Yes	1
	No	0

Missing data are assumed to be normal (score 0). The score ranges from 0 to 10. BUN indicates blood urea nitrogen; M‐CARS, Mayo CICU Admission Risk Score; RDW, red blood cell distribution width.

Table 4

Mean Value, Prevalence of Abnormal Values, and Prevalence of Missing Data for Each Variable in the M‐CARS

	Derivation Cohort	Validation Cohort	P Value
Prevalence of missing data
Any missing data	1380 (13.8%)	196 (8.0%)	<0.0001
Missing anion gap	1076 (10.8%)	156 (5.9%)	<0.0001
Missing BUN	402 (4.0%)	147 (5.6%)	0.0005
Missing RDW	599 (6.0%)	162 (6.2%)	0.7548
Missing Braden skin score	452 (4.5%)	73 (2.8%)	<0.0001
Missing admission diagnosis	106 (1.1%)	0 (0%)	<0.0001
Mean values for continuous variables
M‐CARS	2.0±2.1	2.7±2.2	<0.0001
Anion gap	11.7±3.6	15.1±3.7	<0.0001
BUN	26.6±18.8	27.8±19.9	0.0104
RDW	14.8±2.2	15.0±2.2	<0.0001
Braden skin score	17.6±3.4	18.2±3.3	<0.0001
Prevalence of abnormal values
Anion gap >14	1594 (17.8%)	1329 (53.6%)	<0.0001
BUN >23	3926 (40.9%)	1053 (42.3%)	0.1895
RDW >14.3	4222 (44.9%)	1312 (53.1%)	<0.0001
Braden score ≤15	2447 (25.6%)	512 (20.1%)	<0.0001
Cardiac arrest	1193 (12.0%)	297 (11.3%)	0.2733
Shock	1349 (13.6%)	541 (20.5%)	<0.0001
Respiratory failure	2079 (21.0%)	1002 (38.0%)	<0.0001

BUN indicates blood urea nitrogen; M‐CARS, Mayo CICU Admission Risk Score; RDW, red blood cell distribution width.

Figure 2

Distribution of the M‐CARS in the derivation and validation cohorts. M‐CARS indicates Mayo CICU Admission Risk Score.

M‐CARS and Hospital Mortality

Across the entire study population, patients who died in the hospital (n=1166, 9.2%) had higher mean M‐CARS values (5.1 versus 1.9, P<0.001) and the odds ratio for hospital mortality for each 1 point increase in the M‐CARS was 1.84 (95% CI 1.78–1.89). The optimal cutoff of the M‐CARS for prediction of hospital mortality was 4, with a sensitivity of 75% and specificity of 73%. There was a graded relationship between the M‐CARS and hospital mortality, as shown in Figure 3 and Figure S1; patients with a M‐CARS <2 or M‐CARS ≥5 had similar mortality in the derivation and validation cohorts, while patients with a M‐CARS of 2, 3, or 4 had lower mortality in the validation cohort (P<0.05).

Figure 3

Observed hospital mortality as a function of the M‐CARS (range 0–10) in the derivation and validation cohorts. M‐CARS indicates Mayo cardiac intensive care unit (CICU) Admission Risk Score.

Patients with M‐CARS <2 (n=5952, 47.1%) had a hospital mortality rate of 0.8% in the entire study population, yielding a 99.2% negative predictive value for hospital death (unadjusted odds ratio 0.04 compared with patients with M‐CARS ≥2, 95% CI 0.03–0.05, P<0.001). Patients with a M‐CARS >6 (n=655, 5.2%) had a hospital mortality rate of 51.6% in the entire study population, yielding a 51.6% positive predictive value for hospital death (unadjusted odds ratio 14.36 compared with patients with M‐CARS ≤6, 95% CI 12.13–17.00, P<0.001). Patients with both a Day 1 SOFA score <2 and a M‐CARS <2 were at very low risk of hospital mortality (0.4%), while patients with both a M‐CARS ≥2 and a Day 1 SOFA ≥2 had substantially higher hospital mortality (18.3%). As demonstrated in Figure 4 and Figure S2, the M‐CARS was a better predictor of hospital mortality among patients with discordant Day 1 SOFA and M‐CARS.

Figure 4

Observed hospital mortality as a function of the M‐CARS and Day 1 SOFA score, in the overall population, derivation, and validation cohorts. CICU indicates cardiac intensive care unit; M‐CARS, Mayo CICU Admission Risk Score; SOFA, Sequential Organ Failure Assessment.

Mayo CICU Admission Risk Score Performance

When missing variables were imputed as normal, AUC values of the M‐CARS for hospital mortality were 0.86 in the validation cohort (Table 5), compared with 0.87 in the derivation cohort (Table S1). After excluding the 210 duplicated patients who were included in the derivation cohort, the AUC value was 0.87 in the validation cohort. In a sensitivity analysis, the AUC values for the M‐CARS for hospital mortality were similar in the 1576 (12.5%) patients with missing data, compared with patients having complete data in the entire population (0.88 versus 0.89). In the entire population, the M‐CARS had similar discrimination for hospital mortality in patients with (n=5327, 42.5%) and without (n=7205, 57.5%) an admission diagnosis of ACS (AUC 0.87 versus 0.87). In the entire population, the M‐CARS had lower discrimination for hospital mortality in patients with (n=6173, 49.3%) than without (n=6359, 50.7%) an admission diagnosis of HF (AUC 0.82 versus 0.91). Among patients without shock or cardiac arrest (n=9756, 77.8%), discrimination for hospital mortality by the M‐CARS remained very good (AUC 0.82).

Table 5

AUC Values, HL Statistic P Values, and Brier Score for the M‐CARS and Established ICU Risk Scores for Discrimination of Hospital Mortality in the Validation Cohort

Risk Score	AUC (95% CI)	HL P Value	Brier Score
APACHE‐III score	0.794 (0.765–0.824)	0.895	0.073
APACHE‐IV predicted mortality	0.795 (0.765–0.825)	<0.001	0.084
Day 1 SOFA score	0.788 (0.758–0.818)	0.622	0.075
OASIS	0.772 (0.742–0.803)	0.906	0.079
M‐CARS	0.864 (0.842–0.886)	0.212	0.069

P<0.05 for comparison of M‐CARS to all of the established ICU risk scores by De Long test. APACHE indicates Acute Physiology and Chronic Health Evaluation; AUC, area under the receiver‐operator characteristic curve; HL, Hosmer–Lemeshow; ICU, intensive care unit; M‐CARS, Mayo CICU Admission Risk Score; OASIS, Oxford Acute Severity of Illness Score; SOFA, Sequential Organ Failure Assessment.

The M‐CARS had a higher AUC value for discrimination of hospital mortality compared with the APACHE‐III, APACHE‐IV, OASIS, and Day 1 SOFA scores in both the validation cohort (Table 5) and the derivation cohort (Table S1). The differences between AUC values for hospital mortality using the M‐CARS compared with existing risk scores were significant in both cohorts (all P<0.05 using the DeLong test). Calibration of the M‐CARS for hospital mortality in the validation cohort was good (P=0.21) using the Hosmer–Lemeshow statistic (Table 5). As demonstrated in the calibration plot (Figure S3), all deviations from ideal prediction represented lower observed mortality in the validation cohort. Overall accuracy of the M‐CARS for mortality was better than the established ICU risk scores in the validation cohort based on a lower value of the BRIER score (Table 5).

Discussion

The M‐CARS is a novel, simple integer risk score that predicts hospital mortality in a large cohort of unselected CICU patients. The M‐CARS uses 7 variables available at the time of CICU admission including relevant admission diagnoses (cardiac arrest, shock, and respiratory failure), a marker of frailty (Braden skin score), and commonly available laboratory parameters (RDW, BUN, anion gap). Patients with a M‐CARS lower than 2 had <1% hospital mortality, while patients with a M‐CARS above 6 had >50% hospital mortality. The M‐CARS displayed very good discrimination for hospital mortality that was superior to established ICU risk scores; in addition, the calibration and overall predictive accuracy of the M‐CARS were better than currently available ICU risk scores. Importantly, the M‐CARS performed well in both the derivation and validation cohorts, despite significant differences in baseline characteristics, severity of illness, admission diagnoses, CICU therapies, and data availability.

The M‐CARS combines several established and novel risk factors to predict hospital mortality in CICU patients. Admission diagnoses have been demonstrated to influence outcomes for patients with similar degrees of physiologic derangement, and cardiac arrest, shock, and respiratory failure were reported as independent risk factors for mortality in a recent CICU study.16, 19, 20, 21 One novel aspect of the M‐CARS is the inclusion of a measure of frailty, which is a known contributor to adverse outcomes among critically ill patients that has not been included in prior ICU risk scores.22 The Braden skin score is a simple bedside nursing tool designed to identify patients at elevated risk of pressure ulcer, which assesses mobility, activity, sensory perception, nutrition, moisture, and friction/shear to provide an evaluation of skin integrity and overall patient status.11 We have previously proposed that the Braden Skin Score reflects frailty, and we previously reported that the Braden skin score had an AUC for 0.80 for hospital mortality in this same CICU population.11 BUN, a biomarker that integrates both renal dysfunction and neurohormonal activation, is a known predictor of mortality in acutely ill patients, particularly in patients with heart failure.23 The anion gap, a surrogate for levels of unmeasured anions such as lactate in the blood, has previously been identified as a predictor of mortality in critically ill patients.24 Lactate levels were available in fewer than one quarter of patients, preventing inclusion of this relevant variable in the M‐CARS. We performed an exploratory analysis by assigning 1 point for lactate >2.7 (the optimal cutoff) in place of anion gap >14 in the M‐CARS, and the AUC for hospital mortality across the entire population was similar (0.88 versus 0.87). The RDW, defined as the range of variation of red blood cell volume and a marker of anisocytosis, was a strong predictor of mortality in this CICU population, consistent with prior studies in patients with acute myocardial infarction or heart failure, as well as acutely ill noncardiac patients.25, 26, 27 Despite being consistently associated with higher mortality in multiple studies, the reasons for the association between an elevated RDW and mortality remain uncertain and are presumably multifactorial. Proposed pathophysiologic mechanisms driving abnormal hematopoiesis with an increase in the red blood cell volume variability (and thus RDW) include inflammatory cytokines, oxidative stress, or neurohormonal activation, which presumably contribute to higher illness severity and an adverse prognosis.25, 26, 27 One important risk factor that was not included in the M‐CARS was age, which was the last variable removed by stepwise regression before yielding the top 7 predictors of mortality that were used to create the M‐CARS (ie, the eighth most significant predictor of mortality). Including age in the initial multivariate model did not substantially improve the AUC (0.889 versus 0.883), so age was not included for simplicity.

The M‐CARS overcomes some potential limitations of currently available ICU risk scores, including the need to use the worst values for each variable during the first 24 hours of ICU admission to optimize mortality risk prediction and the fact that risk score performance may be sensitive to missing data.3, 28 By using data available at the time of CICU admission, the M‐CARS allows mortality risk stratification early during the CICU course without the need to wait a full 24 hours. In this same CICU population, 24‐hour scores such as SOFA, APACHE‐III, APACHE‐IV, and OASIS were previously found to have very good discrimination for hospital mortality, although only OASIS had good calibration.11, 12, 13 The M‐CARS, in addition to demonstrating better discrimination than these established ICU risk scores, has good calibration and better overall accuracy. Despite differences in baseline characteristics reflecting temporal changes in the CICU population, the calibration of the M‐CARS was better in the validation cohort than the derivation cohort; this might reflect the larger size of the derivation cohort, which is known to affect the Hosmer–Lemeshow statistic.29 In addition, missing data are known to affect mortality prediction by ICU risk scores, but appear not to substantially influence risk prediction by the M‐CARS.28 The AUC value for hospital mortality using the M‐CARS in patients with ACS was comparable to that initially reported for the GRACE (Global Registry of Acute Coronary Events) risk score.2 Likewise, the AUC value for hospital mortality using the M‐CARS in HF patients, while lower than for patients without HF, compares favorably to established HF risk scores.4, 5

An important issue with any risk score is its clinical utility and the circumstances under which it is best applied, particularly when the score implies high risk. A major advantage of the GRACE risk score is that it facilitates identification of ACS patients who benefit from early coronary angiography.30 Unlike the GRACE risk score that is specific for ACS, the M‐CARS can be applied more broadly to unselected CICU patients. The M‐CARS can identify high‐risk patients who may benefit from early discussion of prognosis and goals of care. Patients classified as high risk (M‐CARS >6) had an observed hospital mortality rate exceeding 50%, warranting a re‐assessment of the optimal therapeutic strategies for these patients including potential palliative care consultation. Further research will be needed to determine whether the M‐CARS can be used to identify CICU patients who are more or less likely to benefit from specific interventions.

The development and use of prognostic risk scores that identify low‐risk patients at the time of hospital admission has important implications for the guidance of triage and clinical decision‐making. Almost half of the patients in this population could be classified as low risk (M‐CARS <2), with an observed hospital mortality rate <1%. We speculate that many of these low‐risk patients without specific acute critical care needs could have been safely cared for outside of the CICU or targeted for early CICU discharge. This hypothesis will require prospective validation, because this study cannot determine whether the low observed mortality rate in these patients was because of interventions performed during CICU admission. Prior studies have suggested very low risk of hospital mortality among CICU patients in this population with a SOFA score <2 on the first CICU day.12 In this study, a SOFA score <2 identified fewer patients as low‐risk compared with patients with a M‐CARS <2, and the observed hospital mortality was higher among patients with a SOFA score <2 compared with patients with a M‐CARS <2.

Limitations

As with all retrospective, single‐center cohort studies, this study has a number of important limitations that may influence the generalizability of the M‐CARS to other CICU populations. The external applicability of the M‐CARS may be reduced by derivation in a predominantly white tertiary‐care referral CICU population, which likely differs from that served by other institutions with a different demographic mix; furthermore, the logistics and organizational structure of the Mayo Clinic CICU likewise may differ from other institutions. However, the observed hospital mortality and relative prevalence of admissions because of ACS and HF are similar to those of other recent CICU studies; these studies included a higher prevalence of nonwhite patients, which is a notable difference from the Mayo Clinic population that may influence risk prediction.9, 21 Patients receiving extracorporeal membrane oxygenator or durable mechanical circulatory support devices represent important CICU patient subgroups that were not included in this population from which the M‐CARS was derived, and the M‐CARS cannot be assumed to apply to those patients without further validation. The source of admission was not consistently available, and could have potentially influenced risk prediction by the M‐CARS. The external applicability of the M‐CARS remains to be established, although our use of a validation cohort obtained after derivation of the M‐CARS supports the validity of the M‐CARS. The inclusion of 210 patients from the derivation cohort in the validation cohort (accounting for 8% of the validation cohort) could be viewed as a limitation, but the performance of the M‐CARS did not appear to be affected by the presence of these duplicated patients. While missing data did not appear to influence the AUC values for the M‐CARS substantially, missing data are known to influence mortality risk prediction by other ICU risk scores.28 More than 10% of patients were missing data for at least 1 predictor variable and the reasons for missingness were not available. Only variables available in >75% of patients were included as candidate variables in the multivariate models, preventing inclusion of potentially important but less‐frequently measured variables such as serum lactate. The diagnoses of cardiac arrest, shock, and respiratory failure were based on ICD‐9 diagnostic codes, and more specific diagnostic criteria may perform differently for risk prediction. We did not have complete data available to calculate other ICU scores that are derived using data from the time of ICU admission, such as the Mortality Probability Model (MPM)0‐III or Simplified Acute Physiology Score (SAPS)‐3. However, a prior analysis from our institution demonstrated superior mortality prediction by the APACHE‐III and APACHE‐IV scores compared with MPM0‐III and SAPS‐3, implying that the M‐CARS would be expected to have superior discrimination as well.31 Data on resuscitation status were similarly unavailable.

Conclusions

The M‐CARS integrates readily available admission data, including both established and novel risk factors, to predict hospital mortality with superior performance compared with established ICU risk scores used in clinical practice. The M‐CARS can identify patients with low hospital mortality risk for whom CICU admission is either unnecessary or only briefly required, provide graded mortality risk in intermediate‐risk patients, and identify high‐risk patients who are likely to die in the hospital and may benefit from early palliative care discussions. Further research will be needed to validate the M‐CARS in other CICU populations, to determine its ability to predict the risk of other clinically relevant outcomes beyond mortality, and to determine how it can be integrated into clinical decision‐making. We anticipate that the recent development of the multicenter CCCTN (Critical Care Cardiology Trials Network Registry) could offer an opportunity to externally validate the M‐CARS in the future.

Disclosures

None.

Table S1. Area Under the ROC Curve (AUC) Values (95% CI) for the M‐CARS and Established ICU Risk Scores for Discrimination of Hospital Mortality in the Derivation Cohort

Figure S1. Observed hospital mortality as a function of the M‐CARS (range 0–10) in the overall population, derivation, and validation cohorts.

Figure S2. Observed CICU mortality as a function of the M‐CARS and Day 1 SOFA score, in the overall population, derivation, and validation cohorts.

Figure S3. Calibration plot demonstrating observed hospital mortality in the validation cohort (Y axis) as a function of predicted hospital mortality based on the derivation cohort (X axis) using the M‐CARS.

Click here for additional data file.