Predicting Mortality of Patients With Sepsis: A Comparison of APACHE II and APACHE III Scoring Systems.

BACKGROUND: Acute Physiology, Age and Chronic Health Evaluation (APACHE) II and III scores were developed in 1985 and 1991, respectively, and are used mainly for critically ill patients of all disease categories admitted to the intensive care unit (ICU). They differ in how chronic health status is assessed, in the number of physiologic variables included (12 vs. 17), and in the total score. These two scoring systems have not been compared in predicting hospital mortality in patients with sepsis.
METHODS: We retrospectively identified all septic patients admitted to our 54-bed medical-surgical ICU between June 2009 and February 2014 using the APACHE outcomes database. We calculated correlation coefficients for APACHE II and APACHE III scores in predicting hospital mortality. Receiver-operating characteristic (ROC) curves were also used to assess the mortality predictions.
RESULTS: We identified a total of 2,054 septic patients. Average APACHE II score was 19 ± 7, and average APACHE III score was 68 ± 28. ICU mortality was 11.8% and hospital mortality was 18.3%. Both APACHE II (r = 0.41) and APACHE III scores (r = 0.44) had good correlations with hospital mortality. There was no statistically significant difference between the two correlations (P = 0.1). ROC area under the curve (AUC) was 0.80 (95% confidence interval (CI): 0.78 - 0.82) for APACHE II, and 0.83 (95% CI: 0.81 - 0.85) for APACHE III, suggesting that both scores have very good discriminative powers for predicting hospital mortality.
CONCLUSIONS: This study shows that both APACHE II and APACHE III scores in septic patients were very strong predictors of hospital mortality. APACHE II was as good as APACHE III in predicting hospital mortality in septic patients.

Introduction

Sepsis is a life-threatening condition that affects more than 1 million patients a year in the United States and even more patients around the globe and is one of the leading causes of death. New definitions for sepsis and septic shock (sepsis-3) were published [1]. Sepsis is now defined as life-threatening organ dysfunction caused by a dysregulated host response to infection. Septic shock is a subset of sepsis with circulatory and cellular/metabolic dysfunction associated with a higher risk of mortality. Mortality from septic shock in the intensive care unit (ICU) is estimated to range between 45% and 63% in observational studies [1-7]. Many different scoring systems are used to analyze the severity of disease and predict mortality in patients admitted to the ICU including patients with sepsis, such as the Acute Physiology, Age and Chronic Health Evaluation (APACHE), the simplified acute physiology score (SAPS), and the mortality probability model (MPM) [8-14]. Predictive scoring systems are measures of disease severity that are used to predict outcomes, typically mortality, of patients in the ICU. They are helpful for standardizing research and comparing the quality of patient care across ICUs. Scoring systems are typically developed using prospectively collected data from a large number of patients from several ICUs. The individual components and the point in time for data collection vary among the predictive scoring systems, which can lead to differences in efficacy and prediction validity. No single instrument has been adequately proven to be superior to another in its ability to predict death. When choosing a predictive scoring system, it is important to use a score that was appropriately validated and that accurately predicts the outcomes in the population of interest (i.e. sepsis). Other factors that should be taken into consideration include feasibility, ease of use, and cost. APACHE II and III scores were developed in 1985 and 1991, respectively, and are used mainly for critically ill patients of all disease categories admitted to the ICU. They differ in how chronic health status is assessed, in the number of physiologic variables included (12 vs. 17), and in the total score. Compared with the APACHE II, the APACHE III is more complex and time-consuming. APACHE II and APACHE III have been compared in patients with liver cirrhosis [15], acute head injury [16], patients undergoing surgery [17], patients with respiratory failure [18], and in all ICU patients [19]. These two scoring systems have not been compared in predicting hospital mortality in patients with sepsis.

Methods

We retrospectively identified all septic patients admitted to our 54-bed medical-surgical ICU between June 2009 and February 2014 using the APACHE outcomes database. APACHE outcome database is free, web-based offering from Cerner Corporation that provides users the ability to calculate and report on outcomes data based upon the APACHE predictions available in the public domain. The system may then be used either as an on-line calculator to quickly obtain scores and predictions for individual patients or subsets of patients on an as needed basis, or as a severity-adjusted outcomes measurement system for an individual ICU (or group of ICUs) to assess quality of care and identify opportunities for improvement. The diagnosis of sepsis is based on surviving sepsis campaign (SSC) definition [1]. Data were entered using a software program that included computerized pick lists and automated calculation of physiological means and gradients and error checking. Patient identifiers were removed from the database, and informed consent was waived by our institutional review board. Mean, standard deviation and P values were reported for comparisons. We calculated Pearson correlation coefficient (PCC) for APACHE II and APACHE III scores in predicting hospital mortality. Receiver-operating characteristic (ROC) curves were also used to assess the mortality predictions.

Results

We identified a total of 2,054 septic patients. Average APACHE II score was 19 ± 7, and average APACHE III score was 68 ± 28. ICU mortality was 11.8% and hospital mortality was 18.3%. Both APACHE II (r = 0.41) and APACHE III scores (r = 0.44) had good correlations with hospital mortality. There was no statistically significant difference between the two correlations (P = 0.1). ROC area under the curve (AUC) was 0.80 (95% confidence interval (CI): 0.78 - 0.82) for APACHE II, and 0.83 (95% CI: 0.81 - 0.85) for APACHE III, suggesting that both scores have very good discriminative powers for predicting hospital mortality (Table 1, Fig. 1).

Table 1

Comparison of APACHE II and APACHE III Scoring Systems in Patients With Sepsis

	APACHE II	APACHE III
Number of patients	2,054	2,054
Average score (SD)	19 (7)	68 (28)
Correlation with hospital mortality, r	0.41	0.44
ROC (95% CI)	0.80 (0.78 - 0.82)	0.83 (0.81 - 0.85)

APACHE: Acute Physiology, Age and Chronic Health Evaluation; SD: standard deviation; ROC: receiver-operating characteristic curve; CI: confidence interval.

Figure 1

Receiver operating characteristic curves for APACHE II score and APACHE III score for prediction of hospital mortality.

Discussion

This study shows that both APACHE II and APACHE III scores in septic patients were very strong predictors of hospital mortality. APACHE II was as good as APACHE III in predicting hospital mortality in septic patients. In 200 patients with acute head injury, APACHE II and III were compared to Glasgow coma scale (GCS). APACHE III yielded the best results of correct prediction of functional outcome, but not mortality [16]. In another study, APACHE II was compared to APACHE III in 844 consecutive patients in the surgical ICU. Both scoring systems overestimated mortality, but estimations made by APACHE III were significantly (P < 0.01) higher. The authors concluded that in situations where APACHE II underestimated mortality, APACHE III may be corrective. However, the differences were subtle and may be difficult to detect in smaller studies [17]. This study addressed postoperative patients in a surgical ICU, a patient population that could differ significantly from a sicker septic population like our study. In addition, this included a much smaller number of patients than our study, which could increase possibility of type I error. Another retrospective study compared the capability of the APACHE II and APACHE III scoring systems to predict outcome for patients with respiratory failure in a medical ICU. APACHE III had a higher discriminative power (area 0.7462) than APACHE II (area 0.6856; P < 0.05) for predicting in-hospital mortality [18]. According to the authors, the extent of physiologic derangement in these patients is likely to have been less marked at the time of ICU admission, which may explain why the predicted mortality rate calculated by the APACHE II model was lower than the actual mortality rate (lead-time bias). In addition, patients with isolated respiratory organ failure are different than septic patients who may have multiple organ failures and/or compromise, like our patient population. A multicenter, multinational study comparing the severity of illness in ICU patients showed that APACHE III had a better ROC area than APACHE II. In 12,899 patients, the improvement in the ROC curve area of APACHE III compared with APACHE II was highly significant (P < 0.0001). APACHE II and APACHE III scores were highly correlated (r2 = 0.827) [19]. The area under the ROC curve provides an adequate measure to compare discrimination for each scoring system [20]. Our results demonstrated that APACHE III was not better than APACHE II in predicting hospital mortality, although the discriminative power of APACHE III may be increased by the inclusion of more physiologic variables. The possible reasons behind our results could be multifactorial. For instance, the variables used to assess comorbid conditions in APACHE III and in APACHE II are similar. Even though APACHE III has more independent variables than the APACHE II, the common variables in APACHE II and III systems are mean arterial blood pressure (MBP), respiratory rate, arterial oxygen pressure (PaO2), oxygen gradient between alveoli and artery (PA-aO2), and serum creatinine concentration (for acute renal failure in APACHE III). These variables may account for most prediction of outcome in patients with sepsis. This may explain why APACHE II and APAPCHE III were not different in our study. In addition, many other factors not accounted for in APACHE II or APACHE II can influence the assessment of outcome in septic patients, including case mix, severity of illness, number of progressive organ failures, quality of care, source of ICU admission (emergency room, hospital floor, and another hospital) [21], as well as other potential biases, such as lead-time bias [22].

Our study has several limitations. It is a retrospective and a single center study. However, it encompasses a large cohort of patients, and the outcomes of septic patients in our study are similar to other larger multicenter studies [2-7]. Pearson correlation, used to compare APACHE II and APACHE III in our study, also has its limitations. It assumes a linear relationship which may not be the case in our study. In addition it is unduly affected by extreme values, and a high degree of correlation (like our study) does not necessarily mean a very close relationship. APACHE II and APACHE III systems have their own limitations. Accurate discrimination and calibration are two key characteristics that should be met by all predictive scoring systems. Discrimination describes the accuracy of a given prediction, in our case, its ability to discriminate between survivors and non-survivors. Calibration describes how the score performs over a wide range of predicted mortalities (i.e., the agreement between observed and expected numbers of survivors and non-survivors across all probabilities of death). APACHE systems (II and III) calibration can deteriorate over time largely due to the effects of case-mix and different patient interventions, which often results in an overestimation of mortality, and thus poor discrimination. APACHE systems were developed from, and validated in patients admitted to ICUs across many institutions and therefore may not accurately assess outcomes associated with specific diseases, such as sepsis. In addition, the degree of agreement likely varies substantially by population, therefore point estimates of agreement in terms of generalizability of our findings must be interpreted with caution especially because it is a retrospective single center study. Our study was performed in an academic referral hospital; therefore our results may not be applicable to institutions with different patient populations.

Conclusion

APACHE III is more complex, has more variables, and data collection is more time consuming than APACHE II. This study shows that both APACHE II and APACHE III scores in septic patients were very strong predictors of hospital mortality. APACHE II was as good as APACHE III in predicting hospital mortality in septic patients. APACHE III is more burdensome, dependent upon proprietary software, and is more costly. On the other hand, APACHE II is easier to use, and is cheaper (free on line calculators).