Comparison of acute physiology and chronic health evaluation II and Glasgow Coma Score in predicting the outcomes of Post Anesthesia Care Unit's patients.

CONTEXT: Acute physiology and chronic health evaluation II (APACHE II) is one of the most general classification systems of disease severity in Intensive Care Units and Glasgow Coma Score (GCS) is one of the most specific ones. AIMS: The aim of the current study was to assess APACHE II and GCS ability in predicting the outcomes (survivors, non-survivors) in the Post Anesthesia Care Unit's (PACU). SETTINGS AND
DESIGN: This was an observational and prospective study of 150 consecutive patients admitted in the PACU during 6-month period.
MATERIALS AND METHODS: Demographic information recorded on a checklist, also information about severity of disease calculated based on APACHE II scoring system in the first admission 24 h and GCS scale. STATISTICAL ANALYSIS USED: Logistic regression, Hosmer-Lemeshow test and receiver operator characteristic (ROC) curves were used in statistical analysis (95% confidence interval).
RESULTS: Data analysis showed a significant statistical difference between outcomes and both APACHE II and Glasgow Coma Score (GCS) (P < 0.0001). The ROC-curve analysis suggested that the predictive ability of GCS is slightly better than APACHE II in this study. For GCS the area under the ROC curve was 86.1% (standard error [SE]: 3.8%), and for APACHE II it was 85.7% (SE: 3.5%), also the Hosmer-Lemeshow statistic revealed better calibration for GCS (χ(2) = 5.177, P = 0.521), than APACHE II (χ(2) = 10.203, P = 0.251).
CONCLUSIONS: The survivors had significantly lower APACHE II and higher GCS compared with non-survivors, also GCS showed more predictive accuracy than APACHE II in prognosticating the outcomes in PACU.

INTRODUCTION

In critically ill-patients, several scoring systems have been introduced and developed over the last three decades. The acute physiology and chronic health evaluation (APACHE) II was released in 1985.[1] It is one of the most widely used scoring systems, have been developed to prognosticate hospital mortality of Intensive Care Unit (ICU) patients.[2] The APACHE II as a general severity scoring system is beneficial in predicting outcomes such as mortality and length of stay (LOS), clinical decision making, comparison of ICU results and monitoring quality of ICU delivered care.[3] In addition, the APACHE II score has been shown to be of use for individual patients in triage.[4] The Glasgow Coma Score (GCS) was developed in 1974.[5] It is a widespread assessment tool to evaluate the level of consciousness. It is categorized as a specific scoring system. Validity and reliability of two models are obtained in several studies.[6789] In spite of that, there are still conflicting data concerning which of the scoring systems is the best predictor tool. External validation is essential before routine application of each model to a group of patients that is different from the group originally used for model development.

Prognostication is an important part of the management of any critically ill patients. There are many studies that have assessed the ability of the APACHE II and GCS to prognosticate the patient's outcomes in ICUs. Donnino et al.,[10] assessed the performance of APACHE II in post-cardiac arrest. In a prospective observational study, a total of 228 subjects were included in the analysis. The mean age was 70 years and 32% of the patients were female. They concluded that APACHE II score is a poor predictor of outcomes at time zero for out-of-hospital cardiac arrest, for in-hospital cardiac arrest it was a modest indicator of illness severity and predictor of mortality/neurologic morbidity. Cardoso and Chiavone,[11] analyzed the performance of APACHE II to predict the mortality rate of discharged patients from the ICU. In a prospective study using the data from 355 patients admitted to the ICU, they noted this tool was shown to be useful for stratifying the patients at greater risk of death after discharge from ICU, also they proposed the professionals, particularly nurses to pay attention to use this system in managing human and technological resources for this group of patients. Cudworth et al.,[12] in a 7-year retrospective study that was undertaken of all traumatic inferior vena cava (IVC) injuries presenting to a tertiary care trauma center, noted that GCS is a significant predictor of mortality in patients with traumatic IVC lesions. In another study Chou et al.,[9] conducted a retrospective analysis of patients with tuberculosis meningitis (TBM) from March 1996 to February 2006; they concluded GCS is an effective predictor of the discharge outcomes of adult patients with TBM.

Most of the previous studies have shown the good discriminative power, but different calibration for APACHE II and GCS.[811131415161718] Differences in the performance of these two scoring systems reinforce that the external validation is essential before routine usage, due to variation in case mix, structure and organization of acute medical care and lifestyles between populations. It is recommended by the researchers that regular re-calibration should be undertaken irrespective of what scoring system is selected, in order to minimize “model fade” and provide clinicians and managers interested in benchmarking a well-validated model to predict mortality. Literature available on this subject in Iranian context is very limited. The aim of this study was to compare the prognostic accuracy of APACHE II and GCS in predicting the outcomes in Post Anesthesia Care Unit (PACU).

MATERIALS AND METHODS

Design

A prospective, observational cohort study of patients who had undergone procedures that require anesthesia, during the period from 7/2011 to 2/2012.

Population

The study population included 150 consecutive patients admitted to the PACU. Excluded from the study population were patients with ICU LOS <24 h as APACHE II cannot be calculated in these patients.

Data collection

Data collection included demographic information (including age and gender), GCS (score before sedation), pre-existing underlying disease, 12 common physiological, laboratory values necessary for computing severity of illness as assessed by APACHE II score. Patients’ privacy maintained by not publishing identifying information. Based on the worst data from the first 24 h after admission to the PACU, a mark adjusting for chronic health problems and a mark for patient age APACHE II was calculated. Validity and reliability of both systems are obtained in many studies.[119202122] APACHE II included of 12 physiologic variables (heart rate, systolic blood pressure, temperature, oxygenation, respiratory rate, arterial pH, serum sodium, potassium and creatinine, hematocrit, white blood cell count and GCS), a chronic health evaluation and age adjustment score. Each variable is weighted from 0 to 4 score, with higher scores denoting an increasing deviation from normal. It is measured during the first 24 h of ICU admission; the maximum score is 71. Points of 25 or less denote less than 50% mortality, while points of 35 or more denote more than 80% mortality.[1] The GCS provides a score in the range 3-15; patients with scores of 3-8 are usually said to be in a coma. The total score is the sum of the scores in three categories including: Eye opening response (4 point), verbal response (5 point) and motor response (6 point).[5] Data were recorded initially on a standardized data collection form and then transferred to SPSS Statistical software version 21 (©Copyright IBM corporation and other(s) 1989-2013).

Intervention

None.

Outcome measures

The primary outcomes for this investigation were survivors and non-survivors.

Data analysis

All analyses were conducted using the SPSS Statistical software version 21 (©Copyright IBM corporation and other(s) 1989-2013). To summarize the study population, simple descriptive statistics were used. Data for continuous variable are presented as means with standard deviations. Categorical data are presented as frequencies with percentages. The association between APACHE II and GCS with outcomes was assessed by logistic regression. APACHE II and GCS were considered to be independent continuous predictor variables. P < 0.05 was considered to be significant. Validation of the APACHE II and GCS were performed using standard tests to measure discrimination and calibration. Discrimination was defined as the power to distinguish between survivors and non-survivors and was assessed by calculating the area under the curve (AUC) of the receiver operator characteristic (ROC) curve. An AUC of 0.5 (a diagonal line) is equivalent to random chance, AUC >0.7 indicates a moderate prognostic model, and AUC >0.8 (a bulbous curve) indicates a good prognostic model.[23] Calibration was defined as an agreement between individual probabilities and actual outcomes, it was assessed using the Hosmer-Lemeshow Goodness-of-Fit test and P > 0.05 was considered as well-calibrated.

RESULTS

A total of 150 patients admitted on PACU were evaluated. The median age of the cohort was 57.41 ± 22.80 years (range 3-97 years), which 91 (60.7%) were men, and 59 (39.3%) were women. The overall mortality for all subjects was 19.3% (21.150). The characteristics of the study samples are shown in Table 1. For the entire cohort of patients, the mean APACHE II, GCS score and length of PACU stay were significantly different between the survivors and non-survivors [Table 2]. Non-survivors showed significantly higher APACHE II scores than survivors. The mean APACHE II score for patients who died was 21.86 ± 6.91 compared with 12.19 ± 5.40 for survivors, P < 0.0001; Also Non-survivors had the mean GCS score 5.14 ± 1.62 compared to 7.97 ± 1.90 for survivors, P < 0.0001. The mean length of PACU stay (5.63 ± 9.04 days) was significantly different between survivors and non-survivors (4.80 ± 8.81; 9.11 ± 9.30 days, respectively).

Table 1

The characteristics of the study samples

Table 2

Comparison of APACHE II, GCS score and length of PACU stay for survivors and nonsurvivors

Discrimination for both scoring system was good. The best Yuden index (sensitivity + specificity-1) was used to determine the best cut-off point for each scoring system. Using a cut-off score 13.5, The APACHE II score predicted hospital mortality with a sensitivity of 96.6%, a specificity of 62.8% and accuracy of 79.7%, with an area under the ROC curve of 0.857 ± 0.035 SE (95%; 0.788-0.925, P < 0.0001). For GCS a cut off score 8.5 showed a sensitivity of 82.8%, a specificity of 82.6% and accuracy of 82.7% [Table 3], also the area under the ROC curve was 0.861 ± 0.038 SE (95%; 0.786-0.937, P < 0.0001). ROC curves were drawn for the APACHE II scoring systems and GCS to assess predictive accuracy [Figure 1].

Table 3

Prediction of outcomes on the first day of PACU admission

Figure 1

Receiver operator characteristic curves for acute physiology and chronic health evaluation (APACHE) II and Glasgow Coma Score (GCS) score. The area under curve is 0.857 for APACHE II and 0.861 for GCS score

By using the Hosmer-Lemeshow Chi-square statistic, the GCS score showed better calibration (χ2 = 5.177, P = 0.521) than APACHE II score (χ2 = 10.203, P = 0.251).

DISCUSSION

Two models for predicting outcomes in the PACU have been evaluated in this study. Both mean APACHE II and GCS scores were significantly higher in non-survivors when compared to survivors (P < 0.0001). The predictive accuracy of mortality models is generally assessed by determining the area under the ROC curve or by calculating the Hosmer-Lemeshow Chi-square statistic. Comparing the efficacy of the predictive power for outcomes of APACHE II and GCS in patients who were admitted in PACU, we measured discrimination and calibration for both prognosticating models. Good capability of discriminating survivors from non-survivors obtained by GCS, with the area under ROC of 0.861, it was slightly better, but very close to 0.857, the value obtained from APACHE II. The cut-off score for GCS and APACHE II in our study were 6.5 and 13.5, retrospectively, and both models showed good overall accuracy. A simple prognostic model, GCS, showed better discriminating power based on AUCs. This difference may arise from case-mix and need for short-term or long-term care. Calibration as assessed by the Hosmer-Lemeshow Chi-square statistic was better for the GCS than APACHE II too (χ2 = 5.177, P = 0.521; χ2 = 10.203, P = 0.251, retrospectively). The GCS is an important component of APACHE II; one can argue that this difference might be explained by suitability of APACHE II in long-term ICU care.

Our results are in agreement with several studies have been cited that higher APACHE II or lower GCS scores were significantly associated with higher mortality or poor prognosis.[89142425262728] One study mentioned APACHE II may not replace the role of GCS in prognosticating the early functional outcomes in ICU, But for prediction of the late mortality, the APACHE II has better accuracy than GCS.[14] Another study cited APACHE II should not replace GCS in assessment of illness severity or prediction of mortality in non-traumatic situation.[8] Zali et al.,[27] pointed to the superiority of APACHE II to GCS in the prediction of mortality in patients with multiple trauma. Because there is a great variation in clinical and other patient characteristics among ICUs, it is doubtful that one scoring system can be validated in all types of populations among different ICUs. Inconsistent with our results, discriminating power of APACHE II and GCS in different studies was from good to excellent (AUC range: 0.76-0.922),[101113141516] for instance: Ho et al.,[18] conducted a retrospective cohort study of 11,107 non-cardiac surgery ICU admissions during 11 years from 1 January 1993 to 31 December 2003, the mean 24-h APACHE II score was 15.4 and the actual hospital mortality was 16.3%; They noted the overall discrimination ability by AUC of 24-h APACHE II was 83.8% (95% confidence interval [CI]: 82.9-84.7). However, in Patel and Grant's study,[29] AUC for the APCHE II (0.672 ± 0.030) was lower than previous studies; they noted this discrepancy could be due to erratic quality of care or differences between the study population and the population in the original studies, also their study had exclusion criteria that surgical and coronary care admissions were excluded.

The results of this study are consistent with the findings of Tsai et al.,[7] study, the best cut-off scores were selected 8 for GCS and 13 for APACHE II compared to 8.5 for GCS and 13.5 for APACHE II in our study; They noted there may be a tendency indicating the superiority of GCS over APACHE model (AUC: GCS 92.2%, 95% CI: 83.4-100.0, P = 0.003; APACHE II 88.4%, 95% CI: 67.2-100.0, P = 0.006) that is similar to our findings too (AUC: GCS 86.1%; APACHE II 85.7%). In two other studies,[814] the cut off score for GCS was 5 (AUC: 88% and 86%) and for APACHE II was 19 and 17, retrospectively (both AUC: 84%). Chou et al.,[9] noted good sensitivity and specificity for both models, like our findings. Similar to our results, in Kim and Kim study,[30] there was a statistically significant difference between GCS and length of ICU stay in two groups of patients. The results of study conducted by Khwannimit and Geater,[16] were unlike ours, their study implemented in Thailand and APACHE II had excellent discrimination (AUC: 0.91, P < 0.001), but poor calibration (Hosmer-Lemeshow statistic was 66.59, P < 0.001), moreover, completing the study, Cerro et al.,[17] cited that APACHE II had no consistent performance for calibration and discrimination so concluded its application in emergency and in-hospital patients is limited. Perhaps these discrepancies can be explained by the fact that a scoring system based on a testing and validation set from one population when transferred to another population without modification will often lose predictive accuracy. Therefore, even if initially the model discriminates well, it is possible that following an improvement or deterioration in quality of care the performance of the model would change and would result in reducing applicability of the severity of illness scoring system to the situation. These problems may be overcome by recalibrating the model frequently to take into account changes in quality of care and improved survival.

The present study has several limitations. First, the study conducted at a single center with a limited sample size: Sample size is known to have a major influence on the measured calibration when using the Hosmer-Lemeshow goodness-of-fit test. Second, as a single-center study, there may be bias with regard to case mix, quality of ICU care and ICU policy. A multi-center study would mitigate the concerns over case mix and benefit from a larger sample size. Discrimination of the models is dependent on both the nature of the population being evaluated and the quality of care being rendered so as to improve the predictive accuracy of these models in an individual ICU such as ours, it may be necessary to customize the models, or perhaps to utilize scoring systems specific for particular disease conditions to estimate mortality.

CONCLUSION

The findings from the present study confirmed that the APACHE II and GCS with good discrimination and calibration are useful tools for the assessment of PACU outcomes. Although both models fit very well to PACU populations, the GCS had better discrimination and calibration; moreover it is simple and economical yet reliable model to predict outcomes in PACU.