Validation of a prognostic score for mortality in elderly patients admitted to Intensive Care Unit.

CONTEXT: The performance of a prognostic score must be evaluated prior to being used. The aim of the present study was to evaluate the predictive ability of hospital mortality of Simplified Acute Physiology Score 3 (SAPS 3) score in elderly patients admitted to Intensive Care Units (ICUs). AIMS: The aim of the present study was to evaluate the SAPS 3 score predictive ability of hospital mortality in elderly patients admitted to ICU. SETTINGS AND
DESIGN: This study was conducted as a prospective cohort, in two mixed ICUs. PATIENTS AND METHODS: Two hundred and eleven elderly patients were included.
INTERVENTIONS: None. We compared the predictive accuracy of SAPS 3 measured at the first hour at ICU and Acute Physiology and Chronic Health Evaluation II (APACHE II) measured with the worst values in the first 24 h at ICU. The patients were followed until hospital discharge. STATISTICAL ANALYSIS USED: Evaluation of discrimination through area under curve receiver operating characteristic (aROC) and calibration by Hosmer-Lemeshow (HL) test.
RESULTS: The median age was 68 years. The hospital mortality rate was 35.54%. The mean value of SAPS 3 was 62.54 ± 12.51 and APACHE II was 17.46 ± 6.77. The mortality predicted by APACHE II was 24.98 ± 19.96 and for standard SAPS 3 equation 41.18 ± 22.34. The discrimination for SAPS 3 model was aROC = 0.68 (0.62-0.75) and to APACHE II aROC = 0.70 (0.63-0.78). Calibration: APACHE II with HL 10.127 P = 0.26, and standard SAPS 3 equation HL 7.204 P = 0.51.
CONCLUSIONS: In this study, the prognostic model of SAPS 3 was not found to be accurate in predicting mortality in geriatric patients requiring ICU admission.

Introduction

With the steady increase in life expectancy of the elderly, the prevalence of pathological conditions that require attention within the Intensive Care Unit (ICU) is also changing, increasing demand for these services.[12]

Admission to the ICU of geriatric patients in critical condition has changed over time, showing a gradual increase. The outcomes of these patients are dependent on several characteristics, including the severity of acute illness. Currently, in developed countries, the elderly constitute up to 50% of all revenue and almost 60% of all days in the ICU.[3] The process of care in an ICU, in addition to diagnosis and treatment offered, must include a prognosis for risk of death during their stay in the ICU or later during hospital stay.

Different severity scoring systems have been widely used in studies with heterogeneous populations to characterize patients in terms of the severity of disease.[4] These studies have used second- and third-generation Mortality Prediction Models (MPM) such as Acute Physiology and Chronic Health Evaluation II (APACHE II), Simplified Acute Physiology Score 2 (SAPS 2), and MPM II, commonly used in the ICU.[5] However, the performance of most recent models of mortality prediction has been evaluated specifically in the elderly population with special conditions, such as oldest surgical patients, with different results.[567] The performance of a prognostic score must be evaluated prior to being used in a particular group of patients.[4567] It has been suggested that the construction, validation, or adaptation of the existing instruments could incorporate factors specific to the geriatric patient.[6]

The SAPS 3 has an appropriate calibration and discriminative performance in general population admitted to an ICU, plus it is easy to apply.[8910] However, its specific performance in a population with particular characteristics, such as aged patients, is limited.[6] Therefore, the primary aim of the present study was to evaluate the SAPS 3 score predictive ability of hospital mortality in elderly patients admitted to medical-surgical ICU, a secondary aim was to compare with APACHE II predictive mortality model.

Patients and Methods

This prospective, observational, and analytical study was conducted during the period February, 2013-October, 2013. It was performed in two mixed ICUs of tertiary level of attention in public institutions.[11]

During the study period, only patients 60 years and over requiring ICU admissions were included, of both sexes with any diagnosis, with an ICU stay at least 24 h. Patients with readmissions or with incomplete records were excluded from the study.

Data were collected by two intensive care physicians through the application of a specific and standardized data collection form which included all components of the SAPS 3 score. In addition, the variables for the prediction model APACHE II were recorded; this model is the most widely used in the vast majority of ICU. The investigators were responsible for data collection, controlling data completeness and quality. We recorded demographic, clinical, and laboratory variables needed to calculate the scores for the SAPS 3 scale and APACHE II, at admission and during the first 24 h after admission to the ICU. Also recorded was the main reason for ICU admission. We used the data from the first 24 h to calculate the APACHE II score, while data from the first hour after admission were used to calculate the SAPS 3 score. In addition to the standard equation of the SAPS 3 score, we also estimated the predicted mortality rates using equations customized for the different geographic regions, including the one for North America (NA) and Central and South America (CSA).[89] Previous functional and premorbid information was not recorded due to the absence or low quality of these data in the medical records. Vital status at hospital discharge was the outcome. Patients were classified based on the reason for ICU admission as medical, scheduled surgical, and emergency surgical. No data were missing for any variable.

The research protocol was reviewed and approved by The National Commission of Scientific Research (registration number R-2013-785-008). All family members of patients signed informed consent.

Statistical analysis

Continuous variables with normal distribution were expressed as mean and standard deviation, and compared with Student's t-test. Continuous variables with a nonnormal distribution were reported as median and interquartile range (IQR) and compared using Mann-Whitney U-test. Categorical variables were presented as absolute numbers (frequency percentages) and analyzed by Chi-square test. Validation of the scoring system was performed using standard tests to measure calibration and discrimination. Discrimination was evaluated by calculating the area under curve receiver operating characteristic (ROC). The area under a ROC curve quantifies the overall ability of the test to discriminate between those individuals who will die and those that will not. A truly useless test has an area of 0.5. A perfect test has an area of 1.00. Hanley-McNeil test was performed to compare the areas under the curve of the two prognostic models, SAPS 3, and APACHE II. Calibration refers to the correlation between the predicted and actual outcome for the entire range of risk and the Hosmer-Lemeshow (HL) goodness-of-fit C statistic was used to test the fit of the logistic model with P > 0.05 in well-fitting models. Standardized mortality rates with respective 95% confidence intervals (CIs) were calculated for each model by dividing observed by predicted mortality rates.

Results

During the study period, 790 patients were admitted to two ICU, of which 216 adults over 60 years were considered eligible for the study, as they had ICU stays of at least of 24 h. We excluded patients with missing data (n = 5); a total of 211 patients were included in the study. The baseline characteristics of the patients are reported in Table 1. The median age of patients was 68 (IQR 63–74) years, predominantly males (n = 124, 58.8%). The most frequent reasons for ICU admission were related with a surgical condition. 21% septic plus surgical procedure, 13% septic nonsurgical, hemorrhagic shock plus surgical procedure 13%, 10% aortic surgery, and 10% craniotomy. The ICU mortality was 12.8% and hospital mortality was 35.54% (n = 75). A median length of stay in the ICU was 5 days (IQR 3–10).

Table 1

Baseline characteristics of the patients

Interpretation of prognostic scores

The mean value of SAPS 3 was 62.54 ± 12.51, and for APACHE II, it was 17.46 ± 6.77. Hospital mortality of elderly patients admitted to ICU was 35.54% (n = 75/211). The hospital mortality predicted by SAPS 3 standard equation was 41.18 ± 22.34; for equations of NA 35.29 ± 17.68 and for CSA equation 51.90 ± 24.75 and for APACHE II was 24.98 ± 19.96. The analyses of the performance of models are presented in Table 2. Discrimination of the APACHE II and SAPS 3 models show a poor performance to predict mortality in older adults with a ROC of 0.70 (CI 95% 0.63–0.78) P < 0.0001 and of 0.68 (CI 95% 0.616–0.75 P < 0.0001), respectively. The performance in the discrimination of the two prognostic models for their areas under the curve was compared but with no statistically significant difference, with P = 0.72 in Hanley-McNeil test.

Table 2

Discrimination, calibration and standardized mortality ratios for the SAPS 3 and APACHE II models

ROC curves are presented in Figure 1. In calibration, both models showed good performance: SAPS 3 standard equation HL 7.20 P = 0.51, SAPS 3 NA equation HL 6.93 P = 0.54 and for SAPS 3 CSA equation HL 6.85 P = 0.55, and for APACHE II with HL C 10.13 P = 0.26 [Figure 2].

Figure 1

Discrimination of the Acute Physiology and Chronic Health Evaluation II and Simplified Acute Physiology Score 3 models

Figure 2

Calibration of both models and the different regional formulae of Simplified Acute Physiology Score 3

The standardized mortality ratio showed that SAPS 3 NA equations have a good performance with an SMR of 0.99 and the standard and CSA equations of SAPS 3 overestimated mortality with an SMR 0.86 and 0.68, respectively, APACHE II model strongly underestimated hospital mortality SMR of 1.42 [Table 2].

Discussion

This study evaluated the performance of the prognostic SAPS 3 model in geriatric medical-surgical population in critical condition, with its standard formula and customized versions for CSA and NA, and compared with the APACHE II prediction, the latter being the model used in our media. In geriatric patients, the performance of other prognostic models for hospital mortality has been evaluated, including previous versions of SAPS and APACHE II.[612131415161718]

Different external validations of the SAPS 3 prediction model for hospital mortality have used younger patients with medical and/or surgical conditions, and even with very specific diagnoses such as cancer, as well as patients undergoing any type of transplant.[18] Some of these validations include persons 60 years and over in their study, and in some of these studies, the average age is around this limit.[18] Meanwhile, the vast majority of the research of the SAPS 3 model has shown a good performance in discrimination, similar or even better than that described in the original paper.[89] In elderly patients, there is still insufficient information to consider the SAPS 3 model as appropriate to predict mortality in this group of patients.[45] The experiences reported are mainly in surgical patients, where this instrument shows good discrimination and calibration.[6181920] However, Hernandez and Palo,[21] evaluated the performance of the SAPS 3 to predict ICU mortality among critically-ill patients of different case mixes admitted to a Philippine private ICU and stratificated the patients on basis of age, showing a good calibration but a low discrimination and a significant overestimation of ICU mortality by standardized mortality ratio in elderly patients. It is necessary to consider that surgical patients have different physiological and functional characteristics than other patients with a medical condition that may influence prognosis.

The performance of the models in our sample was fair to poor. The calibration of the models studied by statistical goodness-of-fit showed that the observed hospital mortality was not different from the expected mortality for both prognostic models in this particular group of patients, for both the SAPS 3 in its original version and the regional versions, and likewise for APACHE II. Other authors have evaluated prognostic models for hospital mortality in geriatric patients; they have shown different results in this respect, such as the case de Rooij et al.,[13] who evaluated the performance of SAPS 2 and its recalibrated version, finding that the original SAPS 2 model was not calibrated to population while the recalibrated version showed good performance. Qiao et al.,[22] meanwhile, found that the APACHE II model functions if properly calibrated to the study group. Sikka et al.[16] did not find proper calibration for APACHE II and SAPS 2 models. These differences in the results found have suggested that the models respond to the potential effect of sample size on the results of the statistical goodness-of-fit, the null hypothesis being more difficult to reject when the sample size decreases.[1821] This situation has been observed in other validations of the SAPS 3 conducted in the general population, which even in small samples have shown the model to have a good calibration and much larger samples have shown a good performance in this respect.[18]

The ability to distinguish patients who die and assigning them higher scores compared with those who live was similar between the APACHE II model and SAPS 3 in its various forms, the latter having a poor performance. The value of the area under the curve of the SAPS 3 of our sample turned out much lower than that reported in other populations that included geriatric patients without performing a subanalysis for this population. The authors that evaluated the APACHE II model in geriatric population or considered these patients in their study subjects, found areas under the curve superior to ours,[131923] although Sikka et al.,[16] found very similar results to those we report.

The present study has some limitations. The first is that it was performed in only two ICUs, which are third-level hospitals, and in population with access to social security, which can affect the external validity of our results for application in other ICU that do not meet these characteristics. Second, the size of the analyzed sample is below the average for other validations. Finally, the power of the analysis of goodness-of-fit HL is dependent on the sample size, since it has been observed that small samples tend to have a good fit while large samples can generate a poor fit.

Conclusions

The geriatric patients admitted to the ICU in critical condition have a high percentage of risk of hospital mortality. The SAPS 3 prediction model of hospital mortality had a regular ability to distinguish patients at risk of hospital death and those not at risk in geriatric critically-ill patients in our cohort. The variant of SAPS formula 3 for the North American region is the only one to match standardized mortality rate calibrated properly in our population. The SAPS 3 model did not show a higher performance than that shown by the APACHE II model. Re-performing calibrations, adjustments and/or adaptations of existing models to predict hospital mortality for use with geriatric patients in critical condition may be necessary.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.