Validation of a Retrospective Computing Model for Mortality Risk in the Intensive Care Unit.

OBJECTIVE: To compare the predictive performance of Epic Systems Corporation's proprietary intensive care unit (ICU) mortality risk model (IMRM) with that of the Acute Physiology and Chronic Health Evaluation (APACHE) IV score.
METHODS: This is a retrospective cohort study of patients treated from January 1, 2008, through January 1, 2018. This single-center study was performed at Mayo Clinic (Rochester, MN), a tertiary care teaching and referral center. The primary outcome was death in the ICU. Discrimination of each risk model for hospital mortality was assessed by comparing area under the receiver operating characteristic curve (AUROC).
RESULTS: The cohort mostly comprised older patients (median age, 64 years) and men (56.7%). The mortality rate of the cohort was 3.5% (2251 of 63,775 patients). The AUROC for mortality prediction was 89.7% (95% CI, 89.5% to 89.9%) for the IMRM, which was significantly greater than the AUROC of 88.2% (95% CI, 87.9% to 88.4%) for APACHE IV (P<.001).
CONCLUSION: The IMRM was superior to the commonly used APACHE IV score and may be easily integrated into electronic health records at any hospital using Epic software.

Monitoring mortality risk in an intensive care unit (ICU) is important for ensuring safe outcomes for patients and benchmarking an ICU’s overall performance against those of other institutions. Clinical scoring systems may serve as prognostic models for hospital mortality, and a commonly used model is the Acute Physiology and Chronic Health Evaluation (APACHE) IV, which has shown good discrimination and calibration for benchmarking ICU performance.

Epic Systems Corporation has developed a proprietary algorithm that uses 49 variables to calculate mortality risk in the ICU. Designed to be run retrospectively after patients have been discharged, the model may help health care providers discover trends in patient outcomes and, similar to the APACHE IV, may be used as a benchmarking tool. Based on data from 86,509 ICU stays in 4 health care organizations from 2011 through 2017, Epic’s ICU mortality risk model (IMRM) uses logistic regression analysis to predict whether a patient will die after receiving care in the ICU (Epic Systems Corporation, unpublished data, 2018). The derivation cohort included academic, community, and research hospitals with an overall mortality of 10.8%. Additional characteristics of the derivation cohort are proprietary, and were not available to the investigators for evaluation.

Variable selection in development of IMRM started with a set of more than 1200 variables. Zero variance predictors and near-zero variance variables with a frequency ratio of greater than 99/1 were dropped based on analysis of three training data sets to identify and eliminate or redefine variables that were not generally available, with final feature selection performed with a bootstrapped lasso regression. Broadly, the variables included in the final model include age, principal diagnosis code, comorbid conditions, code status, vital signs, urine output, laboratory values, presence of invasive lines, and procedures performed.

The IMRM is available in the latest version of Epic software but has not been externally validated extensively or directly compared with the more commonly used APACHE IV model. Our objective was to compare the predictive performance of the IMRM with that of APACHE IV in an independent cohort of ICU patients. A potential effect would be to integrate a convenient algorithm into an institution’s electronic health records system that can quickly provide valuable patient safety data.

Methods

Study Design

This retrospective single-center cohort study was performed at Mayo Clinic (Rochester, MN), a tertiary care teaching and referral center. The Mayo Clinic Institutional Review Board approved this study, which posed minimal risk to patients. All patients had research authorization on file to permit use of their health records for research purposes.

Patient Selection

A robust patient registry that included APACHE IV scores was readily available for retrospective validation at our institution. Patients in this APACHE IV registry were included in our comparison with the IMRM if they were adults (age ≥18 years); had an ICU admission of at least 4 hours during a hospital stay from January 1, 2008, through January 1, 2018; and had a principal diagnosis code.

Measures

Variables were obtained from the APACHE IV registry and from electronic health records, which were searched using Advanced Cohort Explorer, our institutional search engine for patient health records. The APACHE IV registry contained calculations for predicted hospital mortality, which were based on the following variables: age; body temperature; mean arterial pressure; heart rate; respiratory rate; use of mechanical ventilation; fraction of inspired oxygen; partial pressure of oxygen; partial pressure of carbon dioxide; arterial pH; sodium; urine output; creatinine; urea; blood glucose; albumin; hematocrit; white blood cell count; Glasgow Coma Scale score; presence of chronic health conditions (chronic renal failure, cirrhosis, hepatic failure, metastatic carcinoma, lymphoma, leukemia, myeloma, immunosuppression, or AIDS); pre-ICU length of hospitalization; origin before ICU admission; readmission; emergency surgery; and type of admission diagnosis.

The IMRM was based on the following variables from the first 24 hours in a patient’s ICU stay except as noted: demographic characteristics (age); encounter details (time from hospital admission to ICU admission, do-not-resuscitate code status, and partial code status); urine output; the laboratory values conferring the highest risk categorization if available or the most recent value from the hospital admission (hemoglobin, white blood cell count, platelet count, albumin, blood urea nitrogen, glucose, sodium, total bilirubin, anion gap, arterial blood gas pH, and partial pressure of oxygen in arterial blood); the worst fraction of inspired oxygen; use of an airway; use of an arterial line; medication administration (calcium blockers, diuretics, anticonvulsants, cardiovascular sympathomimetics, antibiotics, antihypertensives, anticoagulants, nonnarcotic analgesics, and general anesthetics); procedure orders (nonviolent restraints and comfort measures); principal procedures during the hospital admission (heart valve procedures, coronary artery bypass, and respiratory intubation and mechanical ventilation); the worst scores (Glasgow Coma Scale and Richmond Agitation-Sedation Scale scores); and the worst vital signs (mean arterial pressure, body temperature, heart rate, respiratory rate, and peripheral capillary oxygen saturation). Procedures, clinical characteristics, and medications recorded within the first 24 hours of ICU stay were used in the IMRM.

The IMRM also included comorbid conditions (liver disease, metastatic cancer, and solid tumor without metastasis) and principal diagnoses (circulatory system disease, digestive system disease, endocrine or immunity disorders, mental illness, and respiratory system disease). Principal diagnoses were based on the definitions provided by the Healthcare Cost and Utilization Project’s Clinical Classifications Software.

After obtaining the values for these variables, we normalized the laboratory values to the reference range. All variables then underwent winsorization, spline transformation, and imputation of median values for missing values. The medians used for imputation were from the original model training populations as IMRM is designed to be able to be scored on patients with missing information. The IMRM calculated mortality risk on the basis of the results of the logistic regression model.

Statistical Analysis

Continuous variables were reported as median (interquartile range), and categorical variables were reported as number (percentage). The IMRM logistic regression model was run in RStudio (RStudio Inc) to simulate the scoring IMRM typically performs in a live EHR environment. The mortality risk predictions from the models were compared with the actual mortality rate by using binary logistic regression in JMP 13 Pro (SAS Institute Inc). Discrimination of each risk model for hospital mortality was assessed by comparing area under the receiver operating characteristic curve (AUROC). With MedCalc (MedCalc Software bvba), the DeLong test was used to compare AUROCs and produce a P value for significance between the two models. A calibration curve was also designed, and Brier scores calculated to compare relative performance of models.

Results

We initially identified 73,630 patients who met the eligibility criteria. Duplicate ICU admissions were excluded. Of the initial 73,630 patients, 63,775 had APACHE IV scores available in the registry. The other 9855 patients were excluded from analysis because they had incomplete raw data, which precluded APACHE IV calculation.

The validation cohort included mostly older patients and slightly more men than women (Table 1). Circulatory, endocrine and immunity, respiratory, and digestive system diseases were highly prevalent. In addition, 8021 of 63,775 patients (12.6%) had undergone a heart valve procedure, and 3882 of 63,775 patients (6.1%) had undergone coronary artery bypass (Table 2).

Table 1

Demographic Characteristics, Encounter Details, Principal Diagnoses, and Comorbid Conditions of the Validation Cohort (N=63,775)a,b

Characteristic	Value
Demographic characteristic
Age, years	64 (51-75)
Male	36,169 (56.7)
Encounter detail
Duration of hospital admission, days	5.3 (3.2-9.1)
Duration of ICU admission, days	1.1 (0.9-2.4)
Time from hospital admission to ICU admission, hours	3.8 (0.02-9.3)
Do-not-resuscitate code status	5,054 (7.9)
Partial code statusc	208 (0.3)
ICU type
Cardiac	8,088 (12.7)
Cardiovascular surgical	11,684 (18.3)
Medical ICU	14,242 (22.3)
Medical-surgical ICU	6,809 (10.7)
Neuro ICU	13,843 (21.7)
Surgical ICU	9,109 (14.2)
Principal diagnosis
Circulatory system disease	33,139 (52.0)
Digestive system disease	20,102 (31.5)
Endocrine, nutritional, metabolic, and immune system disorder	30,543 (47.9)
Mental illness	8,501 (13.3)
Respiratory system disease	30,199 (47.4)
Lower risk postoperative monitoring	8,952 (14.0)
Comorbid condition
Liver disease	3,813 (6.0)
Metastatic cancer	2,456 (3.9)
Solid tumor without metastasis	7,067 (11.1)
Scores
Acute physiology score	42 (30-57)
APACHE III score	55 (40-72)
SOFA score	4 (1-7)

APACHE = Acute Physiology and Chronic Health Evaluation; ICU = intensive care unit; SOFA = sequential organ failure assessment.

Continuous variables are reported as median (interquartile range), and categorical variables are reported as n (%).

Partial code status includes any code status that does not include DNR nor “full code” in the description (eg, “do not intubate” only, and other infrequently used variant code statuses).

Table 2

Procedures Administered Within the First 24 Hours of Intensive Care Unit Stay to the Validation Cohort (N=63,775)

Procedure	n (%)
Airway placement	26,617 (41.7)
Respiratory intubation and mechanical ventilation	25,778 (40.4)
Heart valve procedure	8,021 (12.6)
Arterial line placement	7,581 (11.9)
Nonviolent restraints	6,223 (9.8)
Comfort measures	5,391 (8.5)
Coronary artery bypass	3,882 (6.1)

In the validation cohort, heart rate was near the high end of the reference range (median, 99 beats/min) (Table 3). The median Richmond Agitation-Sedation Scale score was −1, which indicates drowsiness on admission to the ICU. White blood cell count (median, 11,500 cells/μL), glucose (median, 147 mg/dL), and fraction of inspired oxygen (median, 60%) were increased. Arterial blood gas pH (median, 7.33) and partial pressure of oxygen in arterial blood (median, 78 mm Hg) were decreased, which is consistent with the prevalence of respiratory intubation and mechanical ventilation (40.4%). Medication review suggested a high prevalence of antibiotics in 38,609 of 63,775 patients (60.5%), nonnarcotic analgesics in 36,554 of 63,775 patients (57.3%), general anesthetics in 27,598 of 63,775 patients (43.3%), antihypertensives in 23,774 of 63,775 patients (37.3%), cardiovascular sympathomimetics in 22,076 of 63,775 patients (34.6%), and diuretics in 15,570 of 63,775 patients (24.4%) (Table 4).

Table 3

Clinical Characteristics Within the First 24 Hours of Intensive Care Unit Stay of the Validation Cohort (N=63,775)a

Characteristic	Median (Interquartile Range)	Reference Range
Mean arterial pressure, mm Hg	74 (62-87)	70-110
Body temperature, °C	36.6 (36.3-37.7)	36.1-37.2
Heart rate, beats/min	99 (57-117)	60-100
Respiratory rate, breaths/min	16 (16-16)	12-20
Peripheral capillary oxygen saturation, %	90 (87-93)	90-100
Glasgow Coma Scale score	15 (15-15)	NAb
Richmond Agitation-Sedation Scale score	−1 (−3 to 0)	NAc
Urine output, mL/24 hours	1,566 (1,206-1,972)	Variable
Hemoglobin, g/dL	12.3 (8.9-13.9)	11.6-15 for women13.2-16.6 for men
White blood cell count, cells/μL	11,500 (6,600-16,300)	3,400-9,600
Platelet count, cells/μL	164,000 (112,000-223,000)	135,000-371,000
Albumin, g/dL	3.7 (3.1-4.1)	3.5-5.0
Blood urea nitrogen, mg/dL	18 (13-27)	6-24
Glucose, mg/dL	147 (96-210)	70-100
Sodium, mEq/L	139 (133-143)	135-145
Total bilirubin, mg/dL	0.3 (0.2-0.5)	0-1.2
Anion gap, mEq/L	13 (11-16)	7-15
Arterial blood gas, pH	7.33 (7.28-7.39)	7.35-7.45
Partial pressure of oxygen in arterial blood, mm Hg	78 (58-107)	83-108
Fraction of inspired oxygen, %	60 (30-100)	21

NA = not applicable.

The Glasgow Coma Scale ranges from 0 to 15. A lower score indicates more severe brain injury.

The Richmond Agitation-Sedation Scale ranges from −5 to +4. A score of −5 indicates an unarousable patient; 0, a calm and alert patient; and +4, a combative patient.

Table 4

Medications Administered Within the First 24 Hours of ICU Stay to the Validation Cohort (N=63,775)a

Medication and Route	No. (%)
Antibiotics, IV or IM	38,609 (60.5)
Nonnarcotic analgesics, any	36,554 (57.3)
General anesthetics, any	27,598 (43.3)
Antihypertensives	23,774 (37.3)
Cardiovascular sympathomimetics, IV or IM	22,076 (34.6)
Diuretics, IV or IM	15,570 (24.4)
Anticonvulsants
Not IV or IM	12,484 (19.6)
IV or IM	7,650 (12.0)
Calcium blockers, IV or IM	5,291 (8.3)
Anticoagulants, not IV or IM	3,514 (5.5)

ICU = intensive care unit; IM = intramuscular; IV = intravenous.

Calibration of the models can be seen in Figure 1. The Brier score of IMRM was 0.027, compared with 0.059 for APACHE IV. Overall, this was suggestive of better calibration of IMRM.

Figure 1

Calibration curve of Intensive Care Unit Mortality Risk Model (IMRM) versus Acute Physiology and Chronic Health Evaluation (APACHE) IV model for mortality.

The median (interquartile range [IQR]) predicted mortality rate was 1.2% (IQR, 0.5% to 3.2%) for the IMRM and 8.1% (IQR, 3.1% to 19.9%) for the APACHE IV model. The actual mortality rate of the cohort was 3.5% (2251 of 63,775 patients). The AUROC was 89.7% (95% CI, 89.5% to 89.9%) for the IMRM, which was significantly greater than the AUROC of 88.2% (95% CI, 87.9% to 88.4%) for the APACHE IV model (P<.001) (Figure 2).

Figure 2

Receiver operating characteristic curves of Intensive Care Unit Mortality Risk Model (IMRM) and Acute Physiology and Chronic Health Evaluation (APACHE) IV for Intensive Care Unit Mortality Risk.

Discussion

On the basis of its higher AUROC and improved calibration, the IMRM showed improved diagnostic performance and accuracy for predicting mortality than the APACHE IV model. Although the AUROCs of IMRM (89.7%) and APACHE IV (88.2%) were significantly different, the absolute difference of 1.5% may not be considered clinically significant by some practitioners. Nevertheless, the IMRM’s greater AUROC suggests that it can be an alternative, useful prognostic scoring system for ICU patients.

Outcome prediction models are readily available for use in ICUs. In addition to the APACHE IV score, other prognostic models include the Simple Acute Physiology Score and Mortality Probability Models., The APACHE IV score is widely used in the United States, can assist with prognostication and assessing outcome measures, and is used to benchmark overall performance among ICUs. Despite the complexity of this calculation, the APACHE IV score can be easily and automatically calculated by electronic health record systems. Comorbidity data, based on diagnoses and billing information, may have gaps from incomplete charting, but despite this common limitation, both IMRM and APACHE IV scores performed well overall. In the current study, the APACHE IV scores were generated automatically from patient data, whereas the IMRM scores were individually calculated for validation purposes. However, the IMRM score can also be easily and automatically calculated to estimate mortality risk.

The differential performance of these models may be attributed in part to the periods in which they were derived. APACHE IV was calibrated in 2006, and many advances have occurred in ICU care, which may change the significance of specific predictive variables over time. IMRM may be better calibrated to present care by being a newer model. Periodic recalibration has been suggested as a need for predictive models, and the lack of recalibration of the existing APACHE IV model may account for some of the performance differential seen here. The IMRM includes more treatment-based variables to analyze the initial 24 hours (Table 5), possibly reflecting the shift towards earlier and more aggressive interventions in the intervening years being a better surrogate of critical illness than diagnostics.

Table 5

Comparison of Variables in IMRM Versus APACHE IVa

Variable	IMRM	APACHE IV
Age	X	X
Admitting diagnosis	X	X
Time from hospital to intensive care unit admission	X	X
Origin		X
Readmission		X
Emergency surgery	X	X
Code status	X
Urine output	X	X
Hemoglobin	X
Hematocrit		X
White blood cell count	X	X
Platelet count
Albumin	X	X
Creatinine		X
Blood urea nitrogen	X	X
Glucose	X	X
Sodium	X	X
Total bilirubin	X	X
Anion gap
Arterial blood gas pH	X	X
Po2	X	X
Pco2		X
Fio2	X
Airway use	X
Mechanical ventilation		X
Arterial line use	X
Calcium channel blocker administration	X
Diuretic administration	X
Anticonvulsant administration	X
Cardiovascular sympathomimetic administration	X
Antibiotic administration	X
Antihypertensive administration	X
Anticoagulant administration	X
Nonnarcotic analgesic administration	X
General anesthetic administration	X
Use of nonviolent restraints	X
Comfort measures order	X
Glasgow coma scale	X	X
Richmond Agitation Sedation Scale	X
Mean arterial blood pressure	X	X
Body temperature	X	X
Heart rate	X	X
Respiratory rate	X	X
SpO2	X

APACHE = Acute Physiology and Chronic Health Evaluation; Fio2 = fraction of inspired oxygen; IMRM = Intensive Care Unit Mortality Risk Model; SpO2 = peripheral capillary oxygen saturation.

Another weakness regards the handling of missing variables. IMRM uses median imputation to handling missing data points, whereas APACHE IV does not; this discrepant approach leads to a great deal of variation in how missing data points may be weighted. We excluded those who had so few data points available that a meaningful APACHE IV score could not be calculated; often, these were patients who had too brief a stay in the ICU for data to be gathered, either because they were transferred out quickly or they expired. Remaining patients with sufficient data to calculate an APACHE score had any missing values imputed in the IMRM score, or excluded in the APACHE score according to the distinct data models. The impact of missing variables on the accuracy of the score for patients with brief ICU stays with little physiologic data recorded cannot be adequately assessed in our study.

Strengths of our study include its relatively large sample size (N=63,775) and the use of our robust institutional APACHE IV registry, which provided readily available data for retrospective validation. However, the single-center design and the large referral base of this study may affect its generalizability to other community settings in the United States. For example, this cohort comprised patients with medically complex conditions and a high prevalence of circulatory, endocrine, respiratory, and digestive system diseases, which may not be characteristic of patients at other institutions. In addition, despite the large sample size, scores were missing for 9855 patients (13.4%) in the APACHE IV registry and could not be compared with those of the IMRM. We also have a lower standardized mortality ratio in the present cohort. The center at which this was studied has historically had exceptionally low ICU mortality compared to national benchmarks for similar acuity patients, attributed to high compliance with local policies and bundle practices to deliver consistent care. This thus limits the generalizability of this single center experience.

Conclusion

This retrospective cohort study shows the feasibility of using the novel proprietary IMRM to predict the mortality rate in the ICU. The IMRM had a greater AUROC and better calibration than the commonly used APACHE IV score and may provide more accurate estimates of mortality risk. Hospitals that use Epic’s electronic health record system may benefit from the inclusion of an easily integrated and automated mortality risk calculator in patient charts.