Evaluation and Validation of Four Scoring Systems: the APACHE IV, SAPS III, MPM0 II, and ICMM in Critically Ill Cancer Patients.

BACKGROUND AND AIMS: To evaluate and validate four severity-of-illness scores, acute physiology and chronic health evaluation IV (APACHE IV), simplified acute physiology score III (SAPS III), mortality probability models II at 0 hours (MPM0 II), and ICU cancer mortality model (ICMM), in a prospective cohort of critically ill cancer patients.
MATERIALS AND METHODS: Single-center, prospective observational study performed in a 14-bedded combined medical-surgical ICU of a tertiary care cancer center of India, from July 2014 to November 2015. Score performance was judged by discrimination and calibration, using the area under receiver-operating characteristics (ROC) curve and Hosmer-Lemeshow goodness-of-fit test, respectively.
RESULTS: A total of 431 patients were included in the study. Intensive care unit (ICU) and hospital mortality were 37.4% and 41.1%, respectively. The area under ROC curve for APACHE IV, SAPS III, MPM0 II, and ICMM were 0.73, 0.70, 0.67, and 0.67, respectively. Calibration as calculated by Hosmer-Lemeshow analysis type C statistics for APACHE IV, SAPS III, MPM0 II, and ICMM shows good calibration with Chi-square values of 5.32, 9.285, 9.873, and 9.855 and p values of 0.723, 0.319, 0.274, and 0.275, respectively.
CONCLUSION: All the four models had moderate discrimination and good calibration. However, none of the mortality prediction models could accurately discriminate between survivors and nonsurvivors in our patients. HOW TO CITE THIS ARTICLE: Siddiqui SS, Narkhede AM, Kulkarni AP, Prabu NR, Chaudhari HK, Divatia JV, et al. Evaluation and Validation of Four Scoring Systems: the APACHE IV, SAPS III, MPM0 II, and ICMM in Critically Ill Cancer Patients. Indian J Crit Care Med 2020;24(4):263-269.

INTRODUCTION

The survival of cancer patients has improved, with better understanding of disease and advances in treatment options.[1,2] Cancer patients are admitted to the intensive care unit (ICU) due to complications related to disease, treatment, or unrelated causes. The ICU outcomes of cancer patients have also improved with advances in the management of the critically ill.[3-7] However, in some cancer patients, the prognosis still remains grave, such as those with advanced malignancies, multiple comorbidities, prolonged immunosuppression, and severe complications of either disease or treatment. Therefore, predicting patient outcome is a challenge. The ICU scoring systems and prediction models help to predict mortality, some systems, the length of ICU stay, and organ dysfunction. These systems calculate the severity of illness scores using points allotted for age, comorbidities, underlying disease, clinical, and laboratory variables. Various severity of illness scoring systems are available; some are specific to disease or organ (e.g., Glasgow coma scale or Child Pugh Score) and some are general, e.g., acute physiology and chronic health evaluation (APACHE), simplified acute physiology score (SAPS) and mortality probability model (MPM). General severity of illness scoring systems are being continuously upgraded to improve their accuracy in the world of continuously improving diagnostics and therapeutics.[8] Groeger and colleagues devised a disease-specific scoring system, ICU cancer mortality model (ICMM),[9] for prediction of mortality in critically ill cancer patients by modifying MPM0 II, but both of these systems have their limitations. The newer versions of general severity of illness scoring systems, such as SAPS III[10] and APACHE IV,[11] and their comparison to cancer-specific score ICMM have been evaluated for their utility in cancer patients in only a few studies.[12,13] The ability to predict outcomes is important for resource allocation, prognostication, and comparison of various ICUs. To the best of our knowledge, APACHE IV has not been compared to ICMM in any other prospective study and neither any study comparing the recent versions of general severity of illness scoring systems to ICMM has been published from Indian subcontinent. We, therefore, prospectively evaluated the performance of SAPS III, APACHE IV, MPM0 II, and ICMM in predicting outcomes of critically ill cancer patients.

MATERIALS AND METHODS

Population

Adult (>18 years) critically ill cancer patients.

Type of Study

Prospective observational study.

Setting

Fourteen-bedded mixed medical–surgical ICU of a tertiary care cancer referral center.

Approval for the study and a waiver of consent from institutional ethics committee was obtained. The study was conducted in the 14-beded ICU of tertiary care cancer hospital from July 2014 to November 2015. All adult (>18 years) cancer patients admitted to the ICU were included. Our ICU predominantly admits medical oncology, prolonged, or complicated oncologic surgical patients. The ICU has 10 general beds and 4 isolation beds. Patients from all the medical and surgical oncology services are admitted from wards, emergency room, operation theaters, or other hospitals. Those patients who require post anesthesia care unit for more than 48 hours and require intensive care are also transferred to the ICU. The inclusion and exclusion criteria for the study are given below:

All ICU admissions medical, surgical, or emergency with pathologically proven diagnosis of cancer with stay of >24 hours.

Age >18 years.

The most recent admission was considered in case of multiple hospital admissions.

For patients requiring readmission to the ICU during the same hospital stay, only the first ICU admission was considered.

Bone marrow transplant patients.

Patients with relapse malignancy or metastatic disease with no treatment options available for cancer.

Patients who had been considered cured of their cancer for more than 5 years.

Data Collection

Various demographic, clinical, and laboratory variables needed to calculate the scores being studied were collected at admission within one hour and over the first 24 hours of ICU admission. The worst value in the specified period was collected, and in case of missing value, normal value was assigned.

Other details collected were the type of malignancy, cancer status, cancer-directed treatment(s), ICU admission diagnosis, type of ICU admission, comorbidities, presence of neutropenia (absolute neutrophil count <500 cells/mm3), ICU interventions during the first 24 hours, date and time of ICU admission, date of ICU discharge, source of ICU admission, and ICU and hospital length of stay. The outcomes studied was hospital mortality.

Calculation of Scores and Mortality Prediction

The APACHE IV score was calculated using the online calculator available at http://www.mecriticalcare.net/icu_scores/apacheIV.php,[14] SAPS III score was calculated with the application provided by https://play.google.com/store/apps/details?id=org.lacofi.saps3,[15] and MPM0 II was calculated from online calculator provided by intensivecarenetwork.com on the link http://intensivecarenetwork.com/Calculators/Files/Mpm.html.[16] The ICMM score calculator was designed on Excel sheet and data was entered to get the probability of mortality.

Statistical Analysis

The SPSS software version 20 (SPSS-20; IBM, Chicago, USA) for windows was used for statistical analysis. Data has been presented as mean ± SD, when indicated. Student's t test was used to compare normally distributed continuous variables. The p value less than 0.05 was considered statistically significant.

The ability and accuracy of the models for hospital mortality prediction were determined by examining their discrimination and calibration. Discrimination was tested by the area under the ROC curve. The Hosmer–Lemeshow goodness-of-fit type C statistics was used to evaluate calibration.

Discrimination and calibration describe the overall predictive power of the model. The mortality risk of our population was analyzed. Standardized mortality ratio (SMR) was calculated for each model by dividing the observed mortality with that as expected by the respective model.

RESULTS

Four hundred and thirty-one adult patients were included in the study. The demographic and general characteristics are depicted in Table 1. Comorbidities and reasons for ICU admission are given as graphs (Figs 1 and 2).

Table 1

Demographic characteristics

Characteristics	Patients (n = 431)	Survivors (n = 254)	Nonsurvivors (n = 177)	p value
Age (years), mean ± SD	50.94 ± 15.386	51.48 ± 14.966	50.17 ± 15.982	0.385
Gender, male (%)	255 (59.16)	151 (59.45)	104 (58.76)	0.886
Neutropenia (%)	68 (15.8)	28 (11.02)	40 (22.60)	0.001
Cancer type
Locoregional solid (%)	269 (62.4)	181 (71.26)	88 (49.72)	<0.0001
Hematolymphoid (%)	101 (23.4)	44 (17.32)	57 (32.20)	0.0005
Metastatic solid (%)	61 (14.2)	29 (11.42)	32 (18.08)	0.07
Cancer status
Controlled/remission (%)	175 (40.6)	116 (45.67)	59 (33.33)	0.0137
Newly diagnosed (%)	186 (43.2)	108 (42.52)	78 (44.07)	0.049
Recurrence/progression (%)	70 (16.2)	30 (11.81)	40 (15.75)	0.0043
Type of ICU admission
Medical (%)	335 (77.7)	184 (72.44)	151 (85.31)	0.0024
Scheduled surgical (%)	61 (14.1)	47 (18.50)	14 (7.91)	0.003
Emergency surgical (%)	35 (8.1)	23 (9.06)	12 (6.78)	0.3949
ICU intervention within 24 hours
Mechanical ventilation (%)	344 (79.8)	180 (70.87)	164 (92.66)	<0.001
IMV (%)	250 (58)	133 (52.36)	117 (66.10)	0.004
NIV (%)	46 (10.7)	28 (11.02)	18 (10.17)	0.778
NIV followed by IMV (%)	48 (11.1)	19 (7.48)	29 (16.38)	0.004
Vasopressor (%)	216 (50.1)	107 (42.12)	109 (61.58)	<0.001
Dialysis (%)	19 (4.4)	9 (3.54)	10 (5.65)	0.295
Source of admission
Ward (%)	177 (41.07)	94 (37.01)	83 (46.89)	0.0402
Emergency room (%)	96 (22.27)	60 (23.62)	36 (20.34)	0.4203
Operation theater (%)	62 (14.39)	46 (18.11)	16 (9.04)	0.0083
PACU/other ICU (%)	76 (17.63)	42 (16.54)	34 (19.21)	0.4737
Others (%)	20 (4.64)	12 (4.72)	8 (4.52)	0.9209
Cancer diagnosis
Hematolymphoid (%)	101 (23.43)	44 (17.32)	57 (32.20)	0.0003
ALL	19	6	13	0.0132
AML	29	9	20	0.0016
NHL	38	21	17	0.6301
HL	6	4	2	0.6982
CML	1	0	1	0.2304
Plasma cell neoplasm	8	4	4	0.6042
Solid organ (%)	330 (76.57)	210 (82.68)	120 (67.80)	0.0003
Head and neck	34	22	12	0.4759
Upper GI	54	36	18	0.2168
Lower GI	38	24	14	0.5793
Breast	32	20	12	0.6699
Genitourinary	29	21	8	0.1265
Gynecological	31	19	12	0.7818
Lung	19	6	13	0.0132
Neurological	7	3	4	0.3834
Hepatopancreaticobiliary	62	39	23	0.4922
Sarcoma	22	18	4	0.0251
Thymic neoplasm	2	2	0	0.2367

IMV, invasive mechanical ventilation; NIV, noninvasive ventilation; PACU, post anesthesia care unit; ALL, acute lymphocytic leukemia; AML, acute myeloid leukemia; HL, Hodgkin's lymphoma; NHL, non-Hodgkin's lymphoma; CML, chronic myeloid leukemia; GI, gastrointestinal; SD, standard deviation

Fig. 1

Comorbidities

Fig. 2

Reasons for intensive care unit admission

The most common ICU intervention used within the first 24 hours was invasive mechanical ventilation (IMV). Other ICU interventions were noninvasive ventilation (NIV), vasopressors, and renal replacement therapy. Many patients received multiple ICU interventions. Prior to ICU admission, cancer-directed treatment received by patients included chemotherapy (264, 61.3% patients), surgery (226, 52.4% patients), and radiotherapy (55, 12.8% patients) alone or in combination. Two hundred and forty-three (56.4%) had received chemotherapy and 35 (8.1%) radiotherapy within 6 months prior to ICU admission. Median ICU and hospital LOS was 5 days (IQR 3–9 days) and 14 days (IQR 8–22 days). ICU and hospital mortality were 37.4% and 41.1% respectively.

The ROC curves and calibration plots of various severity of illness scores studied are depicted in Figures 3 and 4, respectively. Mortality prediction and individual system score characteristics are presented in detail in Table 2. The median SOFA score of our cohort at admission was 5 (IQR 4–7). We studied the correlation between MPM0 II and SOFA score at admission. The Pearson's correlation coefficient was 0.303 indicating a moderate correlation between MPM0 II and SOFA score at admission which was statistically significant (two tailed p value < 0.001).

Fig. 3

Receiver–operating characteristics curves

Fig. 4

Calibration curves

Table 2

Characteristics of scoring systems

Scoring systems	Score (mean ± SD)	Area under ROC curve	Goodness of fit		Predicted mortality	SMR
		(95% CI)	χ2	p value
APACHE IV	72.23 ± 24.65	0.73 (0.681–0.778)	5.32	0.723	129.759	1.364
SAPS III	59.62 ± 13.18	0.7 (0.655–0.754)	9.285	0.319	157.836	1.121
MPM0 II	–	0.67 (0.618–0.722)	9.873	0.274	127.421	1.389
ICMM	–	0.67 (0.615–0.718)	9.855	0.275	212.458	0.8333

SD, standard deviation; ROC, receiver–operating characteristic; SMR, standardized mortality ratio; APACHE, acute physiology and chronic health evaluation; SAPS, simplified acute physiology score; MPM, mortality probability model; ICMM, ICU cancer mortality model

DISCUSSION

Intensivist's perception on admitting critically ill cancer patients to the ICU is changing, and in current era, cancer patients comprise 15–20% of total admissions in the general ICUs.[17,18]

In the current study, we found that APACHE IV, SAPS III, ICMM, and MPM0 II scoring systems had moderate discrimination as per the area under the ROC curve and good calibration as calculated by Hosmer and Lemeshow analysis type C statistics in critically ill cancer patients.

When calculating the SMRs of the four systems, it was found that APACHE IV, SAPS III, MPM0 II scores underpredicted mortality; on the other hand, ICMM overpredicted mortality.

The study population of critically ill cancer patient, a mix of patients admitted for postoperative care and those admitted from wards showed that medical patients had relatively high scores and higher mortality as compared to their postsurgical counterparts.[19] Earlier studies[12,13] evaluated the newer versions of SAPS III and APACHE IV in critically ill cancer patients, wherein majority of them were admitted for postoperative care. A multicenter prospective cohort study by Soares et al.[12] evaluating 717 patients for validation of the four prognostic scoring systems including SAPS III and ICMM has found good discrimination with SAPS III and ICMM, with areas under ROC recorded as 0.84 and 0.79, respectively. This study[12] consisted of 53% scheduled surgical cases. On further analysis of their data after excluding scheduled surgical patients, the discrimination worsened. However, in our study, the population of scheduled surgical patients were patients with complicated intraoperative course. Another large retrospective observational cohort study recently published by Xing et al.[13] had observed excellent discrimination with APACHE IV and SAPS III while evaluating 981 patients, of whom 93% patients were scheduled surgical patients. Further analysis excluding the scheduled surgical patients worsened discrimination. In contrast, in our study most patients were medical admissions and only 14.1% was scheduled surgical patients.

The first study evaluating MPM0 II in critically ill cancer patients[9] found area under ROC curve of MPM0 II as 0.625, similar to our study (0.67). Another study evaluating MPM0 II in critically ill cancer patients found area under ROC curve for MPM0 II as 0.729 after excluding their scheduled surgical patients[20] which is similar to our study. The ICMM has been quoted as having good discrimination power in few studies[9,21] while not in others.[7,20] Both MPM and ICMM would be useful to determine ICU admission if accurate, as they are based on variables at or before ICU admission; whereas the other systems include values within 24 hours after ICU admission. Interestingly, we did not find any difference in discrimination between survivors and nonsurvivors by MPM0 II compared to ICMM as evident from similar area under ROC curves in our study.

As in other studies in critically ill cancer patients,[12,20] the general severity of illness scoring systems underpredicted mortality in our patient cohort. However, ICMM overpredicted mortality in our study, which is consistent with the studies by Soares and colleagues[12,17] but not with the study by Berghmans et al.[21]

The patient cohort in the current study adequately represented all types of cancer patients including hematolymphoid malignancies, solid organ malignancies, metastatic solid organ malignancies, neutropenic patients, extremely sick patients with a large proportion having septic shock and multiorgan failure, needing organ support therapies (mechanical ventilation, vasopressor, and dialysis), high severity scores, and consequently higher ICU and hospital mortality.

Calibration for all the models was good in our study. Compared to the original cohort in which the scores were developed, our cohort is relatively smaller; and as the calibration is influenced by the sample size, it may be the reason we observed good calibration.[7,20,22]

Except the ICMM, none of the other models were developed in an exclusive cohort of cancer patients; neither did they include a similar cohort for validation of the model.

Though ICMM was developed and validated in an exclusive cohort of cancer patients, it was developed in early 90s. Since then there has been significant change in the treatment of malignancies and in intensive care. This may be the reason that ICMM did not perform well in our cohort of patients. Similarly MPM0 II is also an old scoring system, because of which it did not perform well in our ICU cohort.

Additionally, the SMR of less than 1 may reflect this improvement in cancer and intensive care management. This finding necessitates the need for upgradation of ICMM or development of new scoring system for critically ill cancer patients. From the observations in our study, none of severity of illness models could perform well. The APACHE IV and SAPS III are relatively new severity of illness scoring systems and have been shown to perform reasonably well in some recent studies and can still be used in general patient cohorts.[23-26] Other methods of predicting outcomes may be worth exploring in such cases. Employing machine-learning applications may be a logical alternative. It has the advantage of employing the prediction-variable dynamicity as opposed to the variables collected within 24 hours of ICU admission or within one hour of ICU admission and outcome prediction. It may also address the differences in regional variation, as it selects and validates the variables from the same populations. Its implementation is easy as the ICUs become rich sources of data, with introduction of electronic medical record at many places.[27] Machine learning tools are exemplified by artificial neural networks (ANNs), Bayesian networks, support vector machines, and decision trees.[28] Of the given methodologies of machine learning tools, ANNs have been widely studied for outcome prediction. It has performed better than or at par with the traditional logistic regression models for mortality prediction in a variety of patients under different settings.[29-31] Another pragmatic approach may be giving limited-time ICU trial to the patients in whom there is prognostic uncertainty. Time-limited ICU trial of 5 days helped identify cancer patients[32] who may improve with aggressive ICU therapies, in a study where unconditional organ support was given to the patients and no treatment limitations were discussed with family for first 5 days into the ICU. Patients who required the aggressive therapies (mechanical ventilation, vasopressor, or renal replacement) after day 3 were found to have 100% mortality.[32]

The strength of our study is that, it is a prospective study, done in the ICU of a tertiary care cancer center, and the study population adequately represents all types of cancer patients. Our patient population predominantly constituted medical admissions which is a relatively unexplored or underexplored area with respect to the recent generation of severity of illness scoring systems. In the cohort of critically ill cancer patients, acute respiratory failure and sepsis/septic shock have been cited as predominant reason for medical admissions to ICU in many studies,[7,12,20,33,34] which is consistent with our findings. However, its limitation is that it is a single-center study.

CONCLUSION

In our patient cohort, we found that APACHE IV, SAPS III, ICMM, and MPM0 II had moderate discrimination and good calibration. The findings of the study may be applicable to ICUs caring for critically ill cancer patients admitted predominantly for medical reasons and may not be applicable to patients admitted for routine postoperative care.