The Ability of the Acute Physiology and Chronic Health Evaluation (APACHE) IV Score to Predict Mortality in a Single Tertiary Hospital.

BACKGROUND: The Acute Physiology and Chronic Health Evaluation (APACHE) II model has been widely used in Korea. However, there have been few studies on the APACHE IV model in Korean intensive care units (ICUs). The aim of this study was to compare the ability of APACHE IV and APACHE II in predicting hospital mortality, and to investigate the ability of APACHE IV as a critical care triage criterion.
METHODS: The study was designed as a prospective cohort study. Measurements of discrimination and calibration were performed using the area under the receiver operating characteristic curve (AUROC) and the Hosmer-Lemeshow goodness-of-fit test respectively. We also calculated the standardized mortality ratio (SMR).
RESULTS: The APACHE IV score, the Charlson Comorbidity index (CCI) score, acute respiratory distress syndrome, and unplanned ICU admissions were independently associated with hospital mortality. The calibration, discrimination, and SMR of APACHE IV were good (H = 7.67, P = 0.465; C = 3.42, P = 0.905; AUROC = 0.759; SMR = 1.00). However, the explanatory power of an APACHE IV score >93 alone on hospital mortality was low at 44.1%. The explanatory power was increased to 53.8% when the hospital mortality was predicted using a model that considers APACHE IV >93 scores, medical admission, and risk factors for CCI >3 coincidentally. However, the discriminative ability of the prediction model was unsatisfactory (C index <0.70).
CONCLUSIONS: The APACHE IV presented good discrimination, calibration, and SMR for hospital mortality.

Introduction

Due to population aging and chronicization of diseases, the severity of diseases among patients admitted to intensive care units (ICUs) has been increasing. Treatment costs have also been increasing for these patients. Since ICUs have limited medical resources, an objective patient triage protocol and evaluation of the severity of a condition must be developed to minimize unnecessary use of ICUs and allow for efficient use of the limited healthcare resources.

Prognostic scoring systems have been developed to estimate the in-hospital mortality of ICU patients [1-4]. For this reason, over the last three decades, severity scoring systems, such as the Acute Physiology and Chronic Health Evaluation (APACHE) score, the Simplified Acute Physiology Score (SAPS), and the Mortality Probability Model (MPM), have been attempted to be used as a critical care triage criterion beyond predicting hospital mortality in critically ill patients [5].

The APACHE scoring system developed by Knaus et al. [1] achieved higher calibration than Zimmerman et al. [6] after a series of improvements, suggesting that APACHE IV is more accurate [3,7]. According to recent studies, APACHE IV exhibits satisfactory discriminatory performance both in the United States, where it was first developed, and outside the United States [8-11]. The older APACHE II model has been validated in Korean populations; it exhibits poor calibration and modest discrimination for hospital mortality [12]. Whereas, the performance of the APACHE IV has not been sufficiently examined.

This study aimed to investigate the suitability of APACHE IV severity scores and MPMs in an ICU within a tertiary general hospital, by analyzing the relationships among APACHE IV scores at the time of the admission, the predicted mortality rate, and the actual mortality rate comparing with APACHE II model. This study also verified the usefulness of the APACHE IV score as a standard triage protocol for admission in the ICU.

Materials and Methods

1) Patients

The study was conducted in a 1,200-bed capacity referral hospital with four adults’ ICU: medical, surgical, emergency, and cardiac ICU, respectively. Two dedicated intensivists supervised all of 56 ICU beds in all ICUs on a semi-closed system. All patients were admitted between August 1, 2013 and July 31, 2014. The same type of patients suggested by Zimmerman et al. [6] for the development of APACHE IV were involved in this study. We excluded patients if they were younger than 17 years old or if the primary outcome of hospital mortality was uncertain. We also excluded patients with an ICU stay <48 hours, patients with burns, patients missing an APACHE IV score on day 1 in the ICU, patients who were admitted for simple postcardiovascular intervention monitoring, and patients who gave do-not-resuscitate orders. Three hundred and sixty-four patients were screened during the study period and 318 eligible patients were enrolled for analysis (Figure 1). This study was approved by the institutional review board of Chungnam National University Hospital (No. 2015-06-053), in accordance with the Declaration of Helsinki.

Figure 1.

Flow chart of the study population. Initially, 364 intensive care unit patients were enrolled from August 1, 2013 to July 31, 2014. The following patients were excluded: patients who were being readmitted (n = 2), patients who had missing data (n = 8), pediatric patients (n = 1), duplicated data (n = 7), and patients who lost to follow up (n = 28).

2) Differences between APACHE II and APACHE IV model

APACHE II score is calculated based on 12 physiologic criteria and estimates risk based on data available within the first 24 hours of an ICU stay [13,14]. APACHE IV was designed to assess the severity of illness as well as the prognosis in the ICU and has 17 physiological criteria, adding new variables such as mechanical ventilation, thrombolysis, impact of sedation on Glasgow Coma Scale, rescaled Glasgow Coma Scale, PaO2/ FiO2 ratio and disease-specific subgroups, to the existing APACHE III variables [6]. Disease-specific scoring systems have been developed for several important subgroups treated in the ICU since an APACHE III model [15].

3) Data collection

Two critical care fellows and one trained nurse prospectively collected and reviewed electronic medical records. The electronic medical records provided all of the data required to predict the mortality rate using APACHE II and APACHE IV. Predicted hospital mortalities were calculated using the equations of APACHE II and APACHE IV as follows: logit for APACHE II = –3.517 + (APACHE II) × 0.146. The APACHE IV score and predicted mortality rate calculation on a website (http://www.mecriticalcare.net/icu_scores/apacheIV.php) were used in the present study.

4) Identification of risk factors independent of the APACHE IV score and triage model development

Cox proportional hazards regression was conducted to examine associations with death after adjustment for the APACHE IV score. Hazard ratios (HRs) were used to quantify the relationship between risk factors and death. An ICU triage model, which predicts hospital mortality, was constructed by combining the APACHE IV score and the other risk factors identified above the Cox regression models.

5) Statistical analysis

Descriptive statistics are presented as medians and interquartile ranges, or as numbers with percentages. A univariate logistic regression analysis was performed to evaluate any associations between various risk factors and hospital mortality. Among the variables used in the model, the risk factors with P-values less than 0.05 were selected for multivariate logistic regression analysis.

Discrimination is defined as the power to distinguish between survivors and non-survivors, and this was evaluated by receiver operating characteristic (ROC) analysis [6]. Calibration was defined as agreement between individual probabilities and actual outcomes. It was assessed using the Hosmer-Lemeshow goodness-of-fit C statistic with P-values greater than 0.05 indicating good calibration [7]. The standardized mortality ratio (SMR) was the ratio between the observed and predicted number of deaths. To test for statistical significance, we calculated 95% confidence interval (CI) according to the method described by Hosmer and Lemeshow [6]. We estimated HRs in univariate Cox proportional hazards models with 95% CIs and level of statistical significance. To test the discrimination ability of different combinations of parameters like APACHE IV, Charlson Comorbidity index (CCI), and department, we used the area under the receiver operating characteristic curve (AUROC) by Delong method.

Statistical analysis was performed using SPSS version 21.0 (IBM Corp., Armonk, NY, USA). STATA version 12.0 (StataCorp., College Station, TX, USA) was used for C-statistics and the R and MKmisc version 3.2.2 was used for H-statistics.

Results

1) Baseline characteristics of the population

The baseline characteristics of the patients are shown in Table 1. The total number of patients was 318, 79 of whom were non-survivors, exhibiting a 24.8% mortality rate. Among the mortality factors, age, CCI scores, unplanned ICU admission, use of vasoactive agents, acute respiratory distress syndrome (ARDS), severe sepsis or septic shock, APACHE II scores, and APACHE IV scores exhibited statistically significant differences between the survivors and non-survivors (P < 0.001). The mean age of the non-survivors was 70.7 ± 12.6 years, which was higher than that of the survivors (63.1 ± 15.5 years). The mean CCI score of the non-survivors was 5.4 ± 3.2 points, exceeding that of the survivors (3.4 ± 2.5 points), suggesting that the non-survivors had more comorbidities than the survivors. No statistically significant differences were found for sex, body mass index, and ICU lengths of stay between the survivors and non-survivors.

Table 1.

Patient characteristics, scores, and predicted mortality in each prognostic model

Variable	Total (n = 318)	Survivor (n = 239)	Non-survivor (n = 79)	P-value
Age (yr)		63.1 ± 15.5	70.7 ± 12.6	<0.001
Male sex		131 (54.8)	49 (62.0)	0.262
Body mass index (kg/m2)		22.6 ± 3.9	22.6 ± 4.2	0.948
Charlson Comorbidity index score, age adjusted		3.4 ± 2.5	5.4 ± 3.2	<0.001
Route of admission	315			0.001
Ward		68 (29.2)	39 (50.6)
Emergency room		168 (70.0)	38 (49.4)
Other ICU/hospital		2 (0.9)	0
Admission type	305			0.092
Emergency surgery		30 (13.1)	9 (11.8)
Elective surgery		47 (20.5)	5 (6.6)
No surgery		152 (66.4)	62 (81.6)
Unplanned ICU admission	307	79 (34.6)	47 (59.5)	<0.001
Comorbidities
Heart failure		22 (9.2)	3 (3.8)	0.122
Solid cancer		16 (6.7)	4 (5.1)	0.791
Chronic pulmonary disease		11 (4.6)	8 (10.1)	0.097
Infection-related admission	292	28 (13.0)	16 (20.8)	0.103
Mechanical ventilation (invasive)		74 (31.0)	35 (44.3)	0.030
CRRT		18 (7.5)	12 (15.2)	0.043
Vasoactive agent		52 (21.8)	42 (53.2)	<0.001
Major diagnosis
ARDS		5 (2.1)	19 (24.1)	<0.001
Severe sepsis or septic shock		33 (13.8)	34 (43.0)	<0.001
CPCR survivor		15 (6.3)	10 (12.7)	0.068
APACHE II score		21.0 ± 8.7	29.4 ± 8.8	<0.001
APACHE II predicted mortality		46.0 ± 11.1	56.6 ± 11.0	<0.001
APACHE IV score		71.4 ± 32.6	105.3 ± 37.8	<0.001
APACHE IV predicted mortality		24.5 ± 24.2	50.6 ± 30.1	<0.001
ICU length of stay (d)		11.0 ± 12.8	12.5 ± 12.2	0.367
Hospital length of stay (d)		41.7 ± 47.4	39.0 ± 89.2	0.736

Values are presented as mean ± standard deviation or number (%).

ICU: intensive care unit; CRRT: continuous renal replacement therapy; ARDS: acute respiratory distress syndrome; CPCR: cardiopulmonary cerebral resuscitation; APACHE: Acute Physiology and Chronic Health Evaluation.

2) Performance of the APACHE IV and APACHE II models in the prediction of hospital mortality

Both prognostic models showed reasonable discrimination and calibration (Table 2). The AUROC of the APACHE IV and APACHE II models were 0.759 and 0.752. ROC curves for the two scoring systems are shown in Figure 2. Hosmer-Lemeshow goodness-of-fit test generated P-values >0.05 for both APACHE II and APACHE IV, indicating that the models were comparable. APACHE IV exhibited the same mortality prediction rate as the observed mortality. The APACHE II model exhibited a lower SMR than the APACHE IV model (SMR APACHE IV, 1.000 [95% CI, 0.789 to 1.250]; APACHE II, 0.991 [95% CI, 0.788 to 1.248]).

Table 2.

Discrimination and calibration of APACHE IV and APACHE II

Model	No.[a]	AUROC (95% CI)	Hosmer-Lemeshow goodness-of-fit test				SMR
			C-test	P-value	H-test	P-value
APACHE IV	304	0.759 (0.699–0.819)	3.42	0.905	7.679	0.465	1.000 (0.789–1.250)
APACHE II	304	0.752 (0.692–0.811)	4.55	0.805	7.817	0.452	0.991 (0.788–1.248)

APACHE: Acute Physiology and Chronic Health Evaluation; AUROC: area under the receiver operating characteristic curve; CI: confidence interval; SMR: standardized mortality ratio.

Among a total of 318 patients, 304 have both APACHE II and APACHE IV score data and the others could not be calculated.

Figure 2.

Comparison of the area under the receiver operating characteristic curves of APACHE II and APACHE IV. The areas under the receiver operating characteristic curve were 0.759 and 0.752 in APACHE IV and APACHE II, respectively. APACHE: Acute Physiology and Chronic Health Evaluation.

3) Performance characteristics of different combination of parameters for predicting hospital mortality

The multivariate logistic regression model was used to determine independent risk factors for hospital mortality by using all variables with a P-value <0.05 in the univariate model. Among the variables used in the analysis, the APACHE IV score (OR, 1.023; P < 0.001; 95% CI, 1.012 to 1.034), the CCI (OR, 1.200; P < 0.001; 95% CI, 1.053 to 1.368), ARDS (OR, 13.187; P < 0.001; 95% CI, 3.941 to 44.122) and unplanned ICU admission (OR, 2.239; P = 0.015; 95% CI, 1.169 to 4.287) were associated independently with hospital mortality (Table 3).

Table 3.

Independent risk factors for hospital mortality

Variable	Hospital mortality
	Odds ratio	P-value[a]
APACHE IV score	1.023 (1.012–1.034)	<0.001
Age	1.015 (0.989–1.045)	0.223
CCI score	1.200 (1.053–1.368)	<0.001
Mechanical ventilation	0.534 (0.254–1.120)	0.097
Vasoactive agent	1.697 (0.798–3.610)	0.169
Sepsis	1.609 (0.729–3.550)	0.239
ARDS	13.187 (3.941–44.122)	<0.001
Unplanned ICU admission	2.239 (1.169–4.287)	0.015

Among a total of 318 patients, only 307 were included in the regression analysis due to the variable of “unplanned ICU admission statuses” for 11 patients being unknown.

APACHE: Acute Physiology and Chronic Health Evaluation; CCI: Charlson Comorbidity index; ARDS: acute respiratory distress syndrome; ICU: intensive care unit.

P < 0.05.

With a cutoff score of 93, the APACHE IV score predicted hospital mortality with the highest sensitivity and specificity. The highest sensitivity and specificity were observed for CCI scores greater than 3. Table 4 shows that the HRs for hospital mortality. Medical admission, ARDS, CCI >3, and an APACHE IV >93 score were significant risk factors for hospital mortality.

Table 4.

Predicting value for hospital mortality in patients assessed by Cox proportion hazards models

Risk factor	Hazard ratio (95% CI)	P-value[a]
Multivariate analysis
Surgical department admission	Reference
Medical department admission	3.534 (1.634–7.692)	0.001
Acute respiratory distress syndrome	2.661 (1.526–4.640)	0.001
CCI score >3	1.819 (1.061–3.120)	0.030
APACHE IV score >93	2.140 (1.304–3.510)	0.003

CI: confidence interval; CCI: Charlson Comorbidity index; APACHE: Acute Physiology and Chronic Health Evaluation.

P < 0.05.

The explanatory power of APACHE IV scores >93 in predicting the hospital mortality rate was 44.1%; the model explained 44.1% of the total variance. With regards to models in which medical admission, which is a risk factor for hospital mortality, CCI scores >3, and APACHE IV scores >93 were added, the APACHE IV scores >93 and medical admission model had an explanatory power of 48.9%, and the APACHE IV scores >93, medical admission, and CCL >3 model had an explanatory power of 53.8%. Therefore, including risk factors in the models improved their explanatory power. However, the discriminative ability of the prediction models was not satisfactory to use them in a triage protocol for admission in the ICU, since all three models had a C index lower than 0.7 (Table 5).

Table 5.

Performance characteristics of different combination of parameters for predicting hospital mortality in patients

Model	C-index (95% CI)	P-value
APACHE IV >93	0.680 (0.626–0.731)	Reference
APACHE IV >93 + medical department	0.686 (0.632–0.737)	0.679
APACHE IV >94 + medical department + CCI >3	0.659 (0.604–0.711)	0.372

The area under the receiver operating characteristic curve by Delong method was used.

CI: confidence interval; APACHE: Acute Physiology and Chronic Health Evaluation; CCI: Charlson Comorbidity index.

Discussion

The present study aimed to investigate the suitability of APACHE IV severity scores and MPMs in the ICU of a tertiary general hospital. The study also evaluated the usefulness of APACHE IV scores as a single criterion of a triage protocol for admission in ICUs.

There has been much debate on the need for objective directives to follow for an ICU admission triage, which aims to efficiently provide critically ill patients with resources within ICUs [16-18]. A substantial amount of research has also been conducted on the efficacy of existing physiological scores in predicting mortality in ICUs [19-22]. Previous studies have proposed various triage protocols for ICU admission [23].

Since the APACHE scoring system is based on objective physiological factors, it eliminates the possibility of errors made by the user. It also allows for simultaneous comparison and prospective analyses of patients from different ICUs, and can be applied to a wide diversity of patients. Therefore, it was our hypothesis that the APACHE scoring system would be useful as the admission triage criterion.

This study showed that APACHE IV had good calibration and modest discrimination among the critically ill patients in a single center. Moreover, APACHE IV showed an SMR close to the actual mortality rate. Therefore, the suitability of APACHE IV for evaluating the severity of patients’ conditions and predicting their prognoses was verified in this urban referral hospital. It was also found that APACHE IV makes more accurate predictions of patients’ prognoses compared to APACHE II scores in even single center. This could be explained due to the advance of APACHE IV model using additional factors such as mechanical ventilation support, disease specific subgroup analysis, and the specific reason for ICU admission. Daley et al. [24] pointed out that APACHE II has been widely used for measuring ICU performance but this scoring system is not disease specific [25-29]. The same as that APACHE IV scoring system show more reliable prediction in Asia population were observed in other validation studies [30-33].

In aspect of the discriminative ability of new prediction models, the result exhibited unsatisfactory discrimination to use them in a triage protocol for admission in the ICU, as showing all three models had a C index lower than 0.7. There are some reasons why the new prediction model showed unsatisfactory discrimination as an ICU triage protocol. First, the semi-closed system of the ICUs in which the present study was conducted would have prevented lots of patients with a too-sick-to-benefit status from admitting to ICUs in advance by intensivists [34]. Second, the sample size was smaller than that of previous studies on triage models leading to insufficient statistical analysis [35]. Finally, despite establishing 93 as our cutoff score for APACHE IV, the higher score of a prognostic model has greater explanatory power. Though, the present study was limited to comparing the scores divided into only two ranges (<93 or ≥93).

This study had some strengths. The validity of APACHE IV in surgical and medical ICUs, emergency ICUs, and cardiovascular ICUs was evaluated by using data of the same type of patients suggested by Zimmerman et al. [6] for the development of APACHE IV. Second, the present study is meaningful in that it attempted to verify the validity of a standard prognostic scoring system for few domestic studies. Moreover, the results suggest that the APACHE IV system makes more accurate predictions of patients’ prognoses compared to APACHE II scores even in single centers. Finally, within the scope what we know, there is rare research demonstrating that the APACHE IV model would not be advisable as a single criterion for admission in Korea ICU.

However, the study results may contain selection bias since this study took place in only one institution. We could not obtain information on the survival status of patients who could not be monitored due to loss to follow up. And this study had a smaller sample size of 318 patients compared to the sample sizes used in previous studies [19,20,36].

According to our results, in order to establish objective criteria for admission to ICUs, overall clinical judgment of internal medicine patients, severity of comorbidities, main diagnosis type (ARDS, septic shock), age, likelihood of recovery, opinions of medical professionals, patients’ conditions, and prognoses may be helpful. And multilateral research involving larger patient populations and disease groups is essential.

In conclusion, in this work, the APACHE IV scoring system exhibits satisfactory discrimination and excellent calibration, but the result supposed that it was not appropriate to be used as a single criterion for ICU admission. Further research on determining the ICU admission priority of critically ill patients would be necessary.