Acute Physiology and Chronic Health Evaluation II and Simplified Acute Physiology Score II in predicting hospital mortality of neurosurgical intensive care unit patients.

We study the predictive power of Acute Physiology and Chronic Health Evaluation II (APACHE II) and Simplified Acute Physiology Score II (SAPS II) in neurosurgical intensive care unit (ICU) patients. Retrospective investigation was conducted on 672 consecutive ICU patients during the last 2 yr. Data were collected during the first 24 hours of admission and analyzed to calculate predicted mortality. Mortality predicted by two systems was compared and, multivariate analyses were then performed for subarachnoid hemorrhage (SAH) and traumatic brain injury (TBI) patients. Observed mortality was 24.8% whereas predicted mortalities were 37.7% and 38.4%, according to APACHE II and SAPS II. Calibration curve was close to the line of perfect prediction. SAPS II was not statistically significant according to a Lemeshow-Hosmer test, but slightly favored by area under the curve (AUC). In SAH patients, SAPS II was an independent predictor for mortality. In TBI patients, both systems had independent prognostic implications. Scoring systems are useful in predicting mortality and measuring performance in neurosurgical ICU setting. TBI patients are more affected by systemic insults than SAH patients, and this discrepancy of predicting mortality in each neurosurgical disease prompts us to develop a more specific scoring system targeted to cerebral dysfunction.

INTRODUCTION

Scoring systems have been continuously developed to predict outcomes in patients with severe illness, to improve resource allocation and to assist in clinical decision-making particularly for intensive care unit (ICU) patients (1-3). Acute physiology and chronic health evaluation II (APACHE II) (4) and simplified acute physiology score II (SAPS II) (5) are two representative systems currently in wide use for measuring the condition of individual ICU patients (6, 7).

In these systems, the reliability of an outcome prediction in a given population depends on the case mix of that population because the underlying disease category has an independent role in hospital stay outcomes in critically ill patients (4, 7). However, these systems have not always proven valid in specific patient populations such as those with septicemia (7), HIV positive serum (8, 9), Pneumocystis carinii pneumonia (10), cardiac diseases (11) or neoplastic diseases (12, 13). Patients who are admitted to the neurosurgical ICU (NICU) are likely in many instances to have higher mortality despite multimodal intensive management, regardless of their neurosurgical diagnosis. There have been some reports on the efficacy of SAPS II in predicting the outcome for patients with subarachnoid hemorrhage (SAH) and on APACHE II in traumatic brain injury (TBI) (14-16).

The purpose of this study is therefore twofold: first, to compare the discriminating capability of APACHE II and SAPS II score to predict mortality in a group of NICU patients; and second, to assess the applicability of APACHE II and SAPS II scores in two specific disease categories, SAH and TBI. In doing so, the impact of systemic or extracerebral organ dysfunction on the outcome of acutely ill NICU patients will be better defined and future direction identified.

MATERIALS AND METHODS

Study population

Records of 705 consecutive patients who were admitted to NICU from July 2003 through June 2005 were retrospectively examined, and 672 of these were included in this study. Thirty-three patients were excluded because of a chronic moribund state at admission (n=9) or incomplete data gathering (n=24). According to the International Classification of Disease, 10th ed. (ICD-10), the main reason for admission was diagnosis of neurosurgical disease at the time of hospital discharge. For patients admitted to the ICU more than once during a hospitalization episode, only data from the first admission were used.

Data collection

This retrospective study involved a careful review of all medical charts including laboratory results. Patient data observed during the first 24 hr of the hospital stay was collected to obtain following variables: neurosurgical diagnosis, temperature (℃), systolic and mean arterial blood pressure (mmHg), heart rate, respiratory rate, PaO2 or FiO2 (mmHg), arterial pH and bicarbonate, serum sodium, potassium, urea and creatinine, urine output, serum white blood cell count, hematocrit, platelet count and bilirubin, age, type of admission, Glasgow Coma Scale (GCS) score, presence of chronic diseases (chronic organ insufficiency) or immuno-compromised state. When a patient died within the first 24 hr of admission, we selected the most perturbed value of each variable during the period between admission and death (4, 5).

For all patients, APACHE II and SAPS II scores were calculated as described in the original literatures, as was the risk of death according to the published logistic equations (4, 5). The associated risks of hospital mortality were derived using data from each patient's ICU stay and predictive equations of the respective scoring system. Severe chronic illnesses included cirrhosis, New York Heart Association class IV heart failure, chronic respiratory failure, end-stage renal disease, and immuno-suppression. Hospital mortality was defined as the number of patients who died during hospital stay, including deaths in ICU.

Statistical analysis

Continuous variables were expressed as mean±standard deviation (SD) and were compared using Standard t-test. Categorical values were expressed in absolute and relative frequencies, and were analyzed using chi-square test with commercially available statistical software (SPSS Ver. 10, Chicago, IL, U.S.A.). All variables with a P value >0.05 were excluded from the final models. Predicted mortality was calculated using logistic regression formulae described in the original articles (4, 5). Standardized mortality ratio (SMR) was obtained by dividing observed mortality by predicted mortality. The 95% confidence interval (CI) for SMR was calculated using observed mortality as a Poisson variable, and dividing its 95% CI by the predicted mortality (17).

Comparison of the two scoring systems for goodness-of-fit and prediction ability was performed by various methods. Calibration (the ability to provide a risk estimate corresponding to observed mortality) was assessed using calibration curves (2) and chi-square statistics as proposed by Lemeshow-Hosmer to test the goodness of fit of the model (18). A receiver operating characteristic (ROC) curve was built for each severity index, and area under the ROC curve (AUC) (19) was used to test the ability of the models to discriminate between patients who survived or patients who did not.

For patients with SAH and TBI, we related hospital death to baseline characteristics and SAPS II and APACHE II scores during the first 24 hr after admission using a logistic regression model that yielded a crude odds ratio (OR). Multivariate analyses were then calculated using a forward selection method. By using AUC of the corresponding ROC, discriminating power was also evaluated. Finally, analyses of individual elements of SAPS II and APACHE II values were entered in a multivariate logistic regression model with a forward selection method. Variables with a P value >0.10 were excluded.

RESULTS

The main features of the study population are shown in Table 1. There were 207 patients with TBI and 159 patients with SAH.

Table 1

Characteristics of 672 patients enrolled

*Includes ischemia or infarction (n=41), vascular malformation (n=17), and otherwise unspecified intracranial bleeding (n=9). †Includes congenital anomaly (n=6), and demyelinating or degenerative disease (n=3).

SAH, subarachnoid hemorrhage; ICH, intracerebral hemorrhage; IVH, intraventricular hemorrhage.

Predicted mortality

Observed mortality during hospital stay was 24.8% (167/672) and that during ICU stay was 21.4% (144/672). Mean APACHE II and SAPS II values were 37.74% (range: 2-39) and 38.39% (range: 15-90), respectively. Both systems were highly correlated (Bravais-Pearson correlation coefficient, 0.86, P<0.01). The mean predicted risk of death for the overall patient population, survivors and non-survivors are listed in Table 2. There was no significant difference of SMR between the two predictive scoring systems (0.66 for APACHE II and 0.65 for SAPS II). Fig. 1 depicts the distributions of predicted risks for the two systems, both of were skewed toward low scores.

Table 2

The mean predicted risk of death for all patients, for the survivors and for the non-survivors

APACHE II, Acute physiology and chronic health evaluation II; SAPS II, simplified acute physiology score II.

Fig. 1

Grouped distributions of predicted risk of hospital death for APACHE II and SAPS II scores

Calibration and discrimination

The calibration curves for APACHE II and SAPS II scores show that both were close to the line of perfect prediction (Fig. 2). Table 3 shows the number of predicted deaths in each scale and the number of observed deaths over probability intervals of 10%. Comparison (as proposed by Lemeshow-Hosmer) between the contingency tables using a homogeneity chi-square test provides a very significant P value for the APACHE II scoring systems (P<0.01) but not for SAPS II (P=0.07) (Table 4).

Fig. 2

Comparison of the calibration curves for APACHE II and SAPS II scores for hospital mortality prediction.

Table 3

Evaluation of the goodness-of-fit of APACHE II and SAPS II models of hospital mortality*

Lemeshow-Hosmer chi-square statistics were 46.16, P<0.01 for APACHE II and 13.09, P=0.07 for SAPS II.

APACHE II, Acute physiology and chronic health evaluation II; SAPS II, simplified acute physiology score II.

Table 4

Comparison of the scoring systems performances to predict ICU and hospital mortality

*Using calibration curves, the ability to provide a risk estimate corresponding to observed mortality was assessed between two systems and it showed equal result. Using Lemeshow-Hosmer method, χ2 statistics was calculated to test the goodness of fit of the model between the contingency tables and it provided a very significant p-value for the APACHE II scoring system (P<0.01) but not for SAPS II (P=0.07). †Using AUC, the ability to discriminate between patients who survived and patients who did not was assessed. Comparison of the AUC revealed a slightly better fit in favor of SAPS II (area, 0.81 vs. 0.79 for APACHE II).

ICU, intensive care unit; APACHE II, Acute physiology and chronic health evaluation II; SAPS II, simplified acute physiology score II; ROC, receiver operating characteristic curve.

Discrimination was assessed by ROC curves. Comparison of the AUC revealed a slightly better fit in favor of SAPS II (area, 0.81 vs. 0.79 for APACHE II) (Fig. 3).

Fig. 3

Discriminative ability of clinical prediction rules (outcome=death) derived from APACHE II and SAPS II scoring systems

Univariate and multivariate predictors for death in SAH patients

In univariate analysis, SAPS II, patients' age, GCS score and Fisher grade showed predictive implications for hospital death, while APACHE II did not. Moreover, SAPS II had a "dose-dependent" relationship to death such that higher scores suggested increased mortality. In APACHE II, only those of the above tertiles showed such relation with death. Multivariate analysis showed similar results, and the AUC was 0.82. Although detailed analysis is not shown, systolic blood pressure, heart rate, PaO2/FiO2, serum potassium, age, and GCS scores were individual factors contributing to the univariate predictors for mortality in SAPS II. In multivariate analysis, systolic blood pressure, PaO2/FiO2, age, and GCS scores were independent predictors of mortality (Table 5).

Table 5

Univariate and multivariate analyses of predictors for hospital death in SAH patients (n=159)

SAH, subarachnoid hemorrhage; APACHE II, acute physiology and chronic health evaluation II; SAPS II, simplified acute physiology score II; GCS, Glasgow coma scale; AUC, area under the curve; ROC, receiver operating characteristic curve.

Univariate and multivariate predictors for death in TBI patients

Univariate analysis showed that APACHE II, SAPS II, sex, GCS score, presence of systemic injury, systolic blood pressure, and PaO2 were predictors for hospital mortality in TBI patients. The main differences from SAH patients were a greater contribution of systemic factors and exclusion of patients' ages. Both SAPS II and APACHE II also showed a "dose-dependent" relationship to death, with higher scores indicating increasing mortality. Multivariate analysis showed similar results, and the AUC of 0.88 was more discriminating than for patients with SAH. Systolic and mean arterial blood pressure, heart rate, PaO2 or FiO2, arterial pH and bicarbonate, serum urea and creatinine, urine output, and GCS score were contributing factors for SAPS II and APAVHE II in univariate analysis. In multivariate analysis, independent prognostic factors were the same as for the univariate results except for the exclusion of heart rate, serum creatinine level, and urine output (Table 6).

Table 6

Univariate and multivariate analyses of predictors for hospital death in TBI patients (n=207)

TBI, traumatic brain injury; APACHE II, acute physiology and chronic health evaluation II; SAPS II, simplified acute physiology score II; GCS, Glasgow coma scale; AUC, area under the curve; ROC, receiver operating characteristic curve.

DISCUSSION

General perspectives of APACHE II and SAPS II

Illness severity scoring systems are becoming more important tools for measuring ICU performance and outcome, allocating resources, triage of patients, and quality assurance. In the future, such scoring systems will play a larger role in financial reimbursement or even accreditation for individual critical care units (20). As stated previously, the APACHE II and SAPS II systems are based on multiple logistic regression equations that describe abnormalities in multiple physiologic variables during the first 24 hr in the ICU, because many deaths occur soon after admission (4, 5). These scores are used to categorize patients in clinical trials and to compare units with a calculation of the probability for hospital death and SMR. This has been assumed to be an indicator of ICU performance where unity implies that observed performance matches expected performance.

These scores have been tested in a wide range of patient populations with different results (21-23). Owing to pre-existing or accompanying cerebral insult, patients admitted to NICU tended to show more unfavorable outcomes compared with non-NICU patients, and this is verified in our previous report (24). In this paper, however, we did not assess the relationship between such scoring systems and individual patient outcomes. This fact prompted us to investigate the discriminative power of SAPS II and APACHE II in predicting the hospital mortality of NICU patients. In both systems, predicted mortality was much higher than actual mortality. This might be attributed to surgical intervention, resuscitation in the emergency room, or altered physiologic factors observed more than 24 hr after admission that were unforeseen, or inherent to the cerebral pathophysiologic process.

Scoring systems in patients with SAH and TBI

In this study, the amount of extravasated blood clot on CT scan (Fisher grade) and the level of consciousness at admission (GCS) are still the most important determinants predicting mortality of SAH patients. However, GCS assessment only accounts for 15/71 (21.1%) in APACHE II score and 15/163 (9.2%) in SAPS II score. Moreover, Fisher grade is not included in the APACHE II and SAPS II scoring systems. Therefore, a separate or complementary measurement scale must be added or prepared when considering this specific condition. Instead, these systems have systemic, extra-cerebral indices of organ dysfunction, which was tailored to average physiologic variables. Age and cardio-pulmonary parameters (systolic blood pressure, PaO2/FiO2) are proven independent predictors for mortality. Myocardial stunning and neurogenic pulmonary edema mediated by systemic catecholamine surge are well-known systemic manifestations following SAH. They present as ischemic heart disease showing ST segment depression, T wave inversion on electrocardiography, or ventilatory dysfunction showing effusion or inflammatory infiltration into the alveoli (25, 26).

According to Claassen et al. (27), hypoxemia, metabolic acidosis, hyperglycemia and cardiovascular instability within 24 hr of admission were independent prognosticators of death or severe disability in SAH patients. It is interesting that physiologic derangements besides the above-mentioned factors and the presence of systemic inflammatory response syndrome (SIRS) have been continuously suggested to have prognostic implications (25). APACHE II and SAPS II scores have all theses factors in their automated calculation tables. We cannot determine exactly why the APACHE II score did not reach statistical significance while the SAPS II score did. Inclusion or exclusion of co-morbidity is deemed a main differential point between two systems.

The ideal ICU scoring system should provide a predictive basis for decision-making in individual patients as well as a comparative assessment of ICU performance. Most scoring systems have been constructed in general ICU populations and were therefore not validated for specific patients or groups. This has been especially true for TBI patients, who are younger and do not have chronic health problems frequently seen in an older patients, resulting in underestimated predicted mortality (28). The main finding in the present study is that patient age was not related to hospital death, whereas TBI patients were more likely to die as the severity of accompanying systemic injury increased. Both APACHE II and SAPS II systems had statistical significance with mortality in a dose-dependent fashion. The impact of GCS score and cardiopulmonary dysfunctions (low blood pressure, low oxygen saturation) were similar to those of SAH patients.

Limitation and future direction

Although these scoring systems have certain advantages, limitations still exist in routine use. First, although these scores were prospectively recorded by medical personnel, a bias due to differences in calculating scores and validating patient-derived parameters cannot be completely excluded. Post-hoc verification on all processes of data interpretation will be necessary. Second, this study was conducted at only one center. The results therefore, reflect the outcome of specific patients in a tertiary care center and may not be generally applicable to all hospitals in all cases. However, the study gives some insight into this issue, at least from a tertiary care perspective. Third, data collection and compilation have been identified as problems with the APACHE II and SAPS II systems (29). Lead time bias, the question of where the patients came from and how long they were in the hospital prior to ICU admission may influence outcome (30). Fourth, the scoring systems are not adequate to make decisions for the management of individual patients due to the relatively high mortality rate predicted in survivors and the low one predicted in non-survivors. APACHE II and SAPS II scores differed significantly in individual patient populations, and these severity scores are not accurate enough to be used in the routine management of these patients. The appropriate allocation of limited resources available must be addressed. However, the decision to withdraw life support must not rely entirely on these scoring systems. Instead, alterations of management planning such as instituting surgical treatment, reinforcing pharmacological or medical intervention or transferring patients to non-NICU, should be considered (14).

In spite of these limitations, we were able to obtain some helpful findings when assessing hospital mortality using APACHE II and SAPS II in NICU patients. First, there was a significant increase in observed mortality when APACHE II or SAPS II scores increased. Both systems, however, overestimated mortality. The SMR was significantly below 1.0 in both scoring groups. An SMR below 1.0 may have at least three different explanations: selection of less severe patients, good clinical performance, or error of the system itself. Second, calibration and discrimination was good for both systems. Correlation between the APACHE II and SAPS II was excellent, but this is not surprising, given the overlap in the variables considered. Score prediction was tested using criteria suitable to evaluate the calibration and discrimination properties of an outcome prediction score. The calibration curves, comparing observed proportions with predicted proportions of hospital death, were virtually identical. The distribution of the calculated probability of hospital death in both APACHE II and SAPS II were both skewed toward the low score values. Third, there was no major difference in predicting hospital mortality according to goodness-of-fit of the model, as shown by the calibration curves. However, when assessed by the Lemeshow-Hosmer method, APACHE II was statistically significant whereas SAPS II was not. Discrimination between survivors and non-survivors appeared to be slightly superior with SAPS II according to the AUC (Fig. 3). To obtain a better discrimination, more research is needed to define new variables based not on expert opinion but rather on statistical models (6, 31). Finally, if a certain variable were included in this system and consecutively checked, evaluation of new therapies, surveillance of resource utilization, and quality assessment of each ICU would be possible, in addition to outcome prediction.

In summary, we conclude that both APACHE II and SAPS II score systems can be used to approximately predict in-hospital mortality of neurosurgical ICU patients, but not to measure performance or to help in definite clinical decision-making. Neither can be relied on to provide prognostic information for an individual patient. There was some discordance between predictive implications in both systems, particularly in the two different disease categories of SAH and TBI patients. Although the ideal scoring system has yet to be developed and no system has ever been demonstrated to be completely reliable, the ongoing improvement of existing systems should no doubt continue.