Comparison of the pediatric risk of mortality, pediatric index of mortality, and pediatric index of mortality 2 models in a pediatric intensive care unit in China: A validation study.

This study was designed with the aim of comparing the performances of the pediatric risk of mortality (PRISM), pediatric index of mortality (PIM), and revised version pediatric index of mortality 2 (PIM2) models in a pediatric intensive care unit (PICU) in China.A total of 852 critically ill pediatric patients were recruited in the study between January 1 and December 31, 2014. The variables required to calculate PRISM, PIM, and PIM2 were collected. Mode l performance was evaluated by assessing the calibration and discrimination. Discrimination between death and survival was assessed by calculating the area under the receiver-operating characteristic curve (AUC). Calibration across deciles of risk was evaluated using the Hosmer-Lemeshow goodness-of-fit χ test.Of the 852 patients enrolled in this study, 745 patients survived until the end of the PICU stay (107 patients died, 12.56%). The AUCs (95% confidence intervals, CI) were 0.729 (0.670-0.788) for PRISM, 0.721 (0.667-0.776) for PIM, and 0.726 (0.671-0.781) for PIM2. The Hosmer-Lemeshow test revealed a chi-square of 7.26 (P = 0.51, v = 10) for PRISM, 26.28 (P = 0.0009, v = 10) for PIM, and 10.28 (P = 0.21, v = 10) for PIM2. The standardized mortality rate was 1.14 (95%CI: 0.93-1.36) for PRISM, 1.89 (95%CI: 1.55-2.27) for PIM, and 2.13 (95%CI: 1.75-2.55) for PIM2.The PRISM, PIM, and PIM2 scores demonstrated an acceptable discriminatory performance. With the exception of PIM, the PRISM and PIM2 models had good calibrations.

Introduction

The aim of pediatric intensive care units (PICUs) is to provide quality care for critically ill children. Mortality is the most frequently assessed outcome in modern intensive care units (ICUs) and is considered an important assessment criterion for medical services. However, similar conditions should be taken into account when comparing the quality and efficacy of medical services across different units and countries, such as severity of illness on admission and availability of medical resources. Therefore, 2 alternative methods of mortality prediction – the pediatric risk of mortality (PRISM and PRISM III)[ and the pediatric index of mortality and pediatric index of mortality 2 (PIM and PIM2)[ are used to measure the severity illness and the risk of mortality in PICUs. These models predict the risk of mortality by using the logistic regression modeling to obtain an equation that describes the relationship between predictor variables and the probability of death.

PRISM was developed using data collected from PICUs in the United States and was published in 1988 by Pollack et al.[ It is a commonly used mortality prediction model, initially derived from the physiology stability index.[ PRISM III, a 3rd-generation physiology-based prediction model for mortality, has been available since 1996 and offers an improved predictive capability.[ PRISM has been widely used in both developed and developing countries, and the model has both satisfactory discrimination and calibration.[ However, PRISM III has limitations, chiefly for its considerable price even in developed countries,[ and for this reason, it was not evaluated in this study.

PIM was developed with data collected from both Australia and the United Kingdom between 1994 and 1996. It is a simple model consisting of 8 variables measured at the time of admission to an ICU.[ In 2003, the PIM Study Group published a revised version of PIM in Australia, New Zealand, and the United Kingdom. PIM2[ boasts its improved calibration, safety, and adjustment for various diagnoses when compared to the original version.[

So far, the performance of the PRISM and PIM scoring systems has been compared only in certain specific disease categories or within heterogenic groups of patients from PICUs.[ In China, the performance of these scores has been thoroughly investigated in Chinese neonatal ICUs.[ In 2015, our team has compared the performance of the PRISM and PIM2 models in Chinese PICUs.[ However, only 412 discontinuous critically ill pediatric patients transferred to Hunan Children's Hospital were included, and the purpose of the study was to assess the performance of PRISM and PIM2 scores in predicting mortality in this patient population. Therefore, the present study was designed to compare the performances of PRISM, PIM, and PIM2 for 852 continuous critically ill pediatric patients in a PICU in China.

Materials and methods

Ethics statement

This study was approved by the Ethics Committee of Hunan Children's Hospital with the trial registration number HCHLL-2014004. Informed written consent was obtained from the legal guardians of each child enrolled in this study prior to data collection. All data collected from the participants were fully anonymized.

Study subjects

All patients consecutively admitted to the PICU in Hunan Children's Hospital between January 1 and December 31, 2014 were included in this study. Patients were excluded from this study if they died within 2 hours of admission or if they were discharged within 24 hours of admission. Hunan Children's Hospital is the largest tertiary hospital in China, serving a population of 71 million and a land area of 211,800 km2. The hospital has 1800 beds, and 40 of these are in the PICU. Hunan Province is located in central China and had a population of 222 million children in 2013.

Each patient received a chart recording age, sex, diagnosis, temperature, respiratory rate, heart rate, blood pressure, oxygen saturation, white blood cell count, platelet count, liver enzymes, prothrombin time, partial thromboplastin time, levels of bilirubin, glucose, blood urea, creatinine, sodium, potassium, and calcium, arterial blood gas, Glasgow Coma Score, pupillary reactions to bright light, high or low risk diagnosis, occurrence of mechanical ventilation at any time during the first hour in the ICU, elective admission to the ICU, recovery from surgery or a procedure as the main reason for the ICU admission, and admission following cardiac bypass. Patient outcomes (death or survival) and length of stay in the PICU were also recorded. All patients were classified on admission according to their diagnostic group as follows: central nervous system, respiratory, cardiovascular, hematological, and miscellaneous. The variables for PIM and PIM2 were collected between the 1st contact with a PICU doctor and up to the 1st hour after admission to the PICU; PRISM with the 14 physiologic values can be collected during the 1st 24 hours after admission to the PICU. Respiratory rate, heart rate, blood pressure (systolic/diastolic), PaO2/FiO2, PaCO2, prothrombin time, partial thromboplastin time, total bilirubin, calcium, potassium, glucose, HCO3−, and pupillary reactions are the exact predictor variables for PRISM. Elective admission, underlying condition, response of the pupils to bright light, mechanical ventilation, systolic blood pressure, base excess, and FiO2/PaO2 are the exact predictor variables for PIM. Elective admission, recovery postprocedure, cardiac bypass, high risk diagnosis, low risk diagnosis, no response of the pupils to bright light, mechanical ventilation, systolic blood pressure, base excess, and FiO2∗100/PaO2 are the exact predictor variables for PIM2. The PRISM, PIM, and PIM2 scores were calculated using the formula published in the original articles.[ Data were collected by a team of experienced research nurses, and the treating team was blinded to these scores and predictions. The probability of mortality was only calculated at the end of the study period to avoid bias in patient management.

Estimations of sample size

This is a validation study comparing 3 prognostic scores (PRISM, PIM, and PIM2) in a PICU. We used the sample size calculating formula for a validation study,  , to estimate the sample size. With α = 0.05 (bilateral), β = 0.1, tolerance error δ = 0.05, and the area under the receiver-operating characteristic curve ϑ1 = ϑ2 = 0.85, the estimated sample size was 779.

Statistical analysis

All analyses were performed using the Statistical Package for the Social Sciences (SPSS, Windows version 18.0) and Stata (version 7.0) software packages. Demographics and lengths of hospital and PICU stay were presented using median and interquartile ranges (IQRs). Categorical variables were compared using the chi-square test, while medians of continuous variables were compared using either the Mann–Whitney or Kruskal–Wallis tests. Model performance was evaluated by assessing the calibration and discrimination. Discrimination between death and survival was assessed by calculating the area under the receiver-operating characteristic curve (AUC).[ An AUC > 0.70 was considered an acceptable discriminatory performance.[ Calibration across deciles of risk was evaluated using the Hosmer–Lemeshow goodness-of-fit χ2 test[; this test was calculated as Σ (O − E)2/E, where O is the observed and E the expected number of events in each group of risk. For this test, a P-value >0.05 indicates good calibration. The overall performance of the 3 scoring systems was assessed using a standardized mortality ratio (SMR), which is the ratio of the risk-adjusted observed mortality to the expected mortality derived from the development set where the score was developed. Calibration across the diagnosis was assessed by calculating the SMR[ with a 95% confidence intervals (CIs) assessed using the Fisher exact test. Pearson correlation analysis was applied to the estimated probabilities of death to show the correlations among the PRISM, PIM, and PIM2 scores. Bland–Altman analysis (constructed using the MedCal software version 15.2.2) was used to measure assay agreement among the PRISM, PIM, and PIM2 scores.

Results

A total of 885 patients were enrolled in this study between January 1 and December 31, 2014. However, 13 patients were discharged within 2 hours of admission for unknown reasons, 8 patients died within 2 hours of admission after cardiopulmonary resuscitation, and 12 patients were excluded due to lack of data. The final dataset for PRISM, PIM, and PIM2 analysis consisted of 852 patients with 107 (12.56%) recorded deaths. The median age was 6.5 months (IQR: 2–21), and 745 patients survived at the end of the PICU stay. Age and sex distribution were not significantly different between the survival and death groups (P > 0.05). The majority of the patients had respiratory diseases, nervous system diseases, and miscellaneous conditions. The median duration of hospital stay was 8 days (IQR: 4–15), and the length of hospital stay for the survival group was longer than that for the death group (Z = −7.01, P < 0.05). The PRISM median score was 11 (IQR: 7–14) for all patients, while that for the deaths was higher than that for the survival group (P < 0.05). The median probability of mortality for PRISM, PIM, and PIM2 scores was 6.20% (3.06%–12.25%), 2.54% (1.90%–6.92%), and 2.20% (1.59%–5.78%), respectively, and was significantly higher in the death group than in the survival group for the 3 scores (P < 0.05). Patient characteristics are presented in Table 1.

Table 1

Characteristics of the pediatric patients in PICU.

Table 2 summarizes the performance of the 3 models. Of the 852 patients enrolled in this study, 107 patients (12.56%) died. Expected mortality was 94.14 patients (11.05%) by PRISM, 56.50 (6.63%) by PIM, and 50.20 (5.89%) by PIM2. The SMR for the entire sample was 1.14 (95%CI: 0.93–1.36) for PRISM, 1.89 (95%CI: 1.55–2.27) for PIM, and 2.13 (95%CI: 1.75–2.55) for PIM2 in this study, respectively. The AUC (95% CI) was 0.729 (0.670–0.788) for PRISM, 0.721 (0.667–0.776) for PIM, and 0.726 (0.671–0.781) for PIM2 (Fig. 1).

Table 2

Performance of the models.

Figure 1

Superposition of 3 ROC curves. The area under the ROC curve was 0.729 for PRISM (95% CI 0.670–∼0.788), 0.726 for PIM (95% CI 0.671–0.781), and 0.721 for PIM2 (95% CI 0.667–0.776). Findings were shown to have a good discriminatory performance between survivals and nonsurvivals. PIM = pediatric index of mortality, PIM2 = pediatric index of mortality 2, PRISM = pediatric risk of mortality, ROC = receiver-operating characteristic.

Table 3 shows the model calibration across the various levels of probability of death. The PRISM score under-predicted mortality at 0 to 0.061978 and 0.252372 to 0.999999 predicted probability levels. The PIM and PIM2 2 scores under-predicted mortality at 10 predicted probability all levels. The Hosmer–Lemeshow test revealed a chi-square of 7.26 (P = 0.51, v = 10) for PRISM, 26.28 (P = 0.0009, v = 10) for PIM, and 10.78 (P = 0.21, v = 10) for PIM2.

Table 3

Calibration of scores across 10 different level of probability of death.

The estimated probabilities of death revealed a positive and significant correlation between PRISM and the 2 PIM models with a Pearson correlation coefficient of r = 0.490 (P < 0.001) and 0.477 (P < 0.001), respectively. The correlation coefficient between PIM and PIM2 was 0.938 (P < 0.001) (Table 4). The Bland–Altman plot was used to reveal the differences between the 2 scores. The Bland–Altman plot with linear regression analyses with 95% confidence limits is presented in Figs. 2–4. A significant difference among the PRISM, PIM, and PIM2 was not observed.

Table 4

The correlation relationship among PRISM, PIM, and PIM2.

Figure 2

The Bland–Altman plot for comparison between PRISM and PIM. PIM = pediatric index of mortality, PRISM = pediatric risk of mortality.

Figure 4

The Bland–Altman plot for comparison between PIM and PIM2. PIM = pediatric index of mortality, PIM2 = pediatric index of mortality 2.

The Bland–Altman plot for comparison between PRISM and PIM2. PIM2 = pediatric index of mortality 2, PRISM = pediatric risk of mortality.

Discussion

In this study, we compared the performance of 3 major published mortality predicting tools in PICUs in China. The major findings were as follows: all 3 scoring systems demonstrated acceptable discrimination between death and survival with AUCs > 0.70; the fit between observed and expected outcomes was close for all 3 models according to the SMR in this setting; and predicted results were similar to those observed for PRISM and PIM2 in terms of evaluating calibration by the Hosmer–Lemeshow goodness-of-fit χ2 test (with the exception of PIM which displayed poor calibration). The performances of PRISM, PIM, and PIM2 have thus far been evaluated in only 1 study in China. The study was performed on 243 consecutive neonate admissions in a Chinese neonatal ICU and demonstrated good discrimination and calibration in all 3 models.[ It further showed that discrimination in the 3 models was similar to certain other publications[ but was slightly weaker than that of several studies conducted in both developed[ and developing countries.[ In a study conducted in India, Singhal et al[ demonstrated that PRISM had an acceptable predictive value (AUC = 0.72). However, in a study conducted in The Netherlands, Visser et al[ reported that the AUC values for PIM, PIM2, PIM2-ANZ06, PIM2-ANZ08, and PRISM were >0.8 in 8 PICUs. In a study conducted in Iran, the AUCs for PRISM were reported to be 0.803.[ This discrepancy may be attributed to the variances in the characteristics of the study populations, small sample sizes (particularly for number of recorded deaths), and improvement in the quality of medical care in the past several decades since the development of these predictive mortality models.

In the present study, observed mortality was significantly higher than that predicted (SMR > 1) by the PRISM, PIM, and PIM2. Our findings, however, differ from those of the majority of studies conducted in developing or developed countries, which reported these models to be either under- or over-predicted the mortality rates in their samples.[ For example, studies conducted in both Hong Kong and India have reported SMRs ranging from 0.61 to 3.3 for PIM and from 0.79 to 1.20 for PRISM.[ In Pakistani, a study has demonstrated that the SMRs (95% CI) using the PRISM and PIM2 models were 1.20 (0.94–1.50) and 1.57(1.24–1.59), respectively.[ The authors have attributed this to the variances in patient profiles, medical resources, and quality of intensive care in PICUs. The variation in SMR may be attributed to the threshold for initiating, timing of intensive care and quality of care, and the accuracy of data collection.

In the present study, under-prediction in certain predicted probability levels could be related to the PICU in question. First, the majority of patients enrolled in this study had respiratory disorders. However, patients admitted to the PICU with respiratory dysfunction would have been treated at either the emergency unit or the referring hospital and may therefore have blood gas analysis results within normal limits. And patients with mechanical ventilation have normal blood gases at admission to PICU, which lead to the lower value of FiO2/PaO2 and base excess. Furthermore, the organ dysfunction of referral patients may be improved in the referring hospital. Thus, these patients would have low scores for all 3 models, although their base conditions may not have been resolved and may in fact deteriorate due to underlying illnesses. Second, under-prediction could be attributed to the low number of deaths at each risk interval.[ Third, due to the availability of specialized emergency medical services and transfer centers, patients are often stabilized before admission to the PICU. Low scores across the 3 models would, therefore, reflect a transient improvement in physiological status on PICU admission as a result of intervention either in the emergency room or during transfer.[ Finally, under-prediction of mortality across the 3 models could be a result of actual management of large numbers of severely ill children, where limited manpower and resources may impact the quality of intensive unit care provided.

In the present study, we attempted to validate the PRISM, PIM, and PIM2 models and compared them for both discrimination and calibration. Although still debatable, both discrimination and calibration are important in the validation of any prognostic scoring model.[ The importance of either function is dependent on the objective of prognostic score use.[ If the study aims to distinguish between survival and nonsurvival among patients, then the capacity to discriminate is the most important. However, if the study aims to compare observed and expected mortalities at different intervals of severity, then calibration capacity is of greater importance. Thus, discrimination and calibration are both essential for global evaluation of the scores.

In our study, the correlation between PRISM and the 2 PIM models was r = 0.490 (P < 0.001) and 0.477 (P < 0.001), respectively. A study conducted by Martha et al[ has demonstrated that the correlation coefficient between PRISM and PIM models was r = 0.65 (P < 0.05); the correlation coefficient was lower than that in other developing countries.[

Our study has demonstrated that the performance of the 3 models is similar in terms of their capacity to discriminate between surviving and moribund patients. Nevertheless, PIM exhibited a poor calibration capacity, which was also observed by Bertolini et al.[ The power of calibration was tested in all the scoring systems using the Hosmer–Lemeshow goodness of fit test, and a P > 0.05 was considered a good calibration for the model. Several other studies have also shown good discrimination but poor calibration for PRISM, PIM, and PIM2 models.[ This poor calibration may be attributed to factors such as poorly performing healthcare systems and limited resources in developing countries. A further possibility is the low number of deaths reported at each level, and a study by Pearson et al[ recommends that special care be taken when variance is low within a small series (eg, <20 deaths per unit). Other possible contributors include different case mixes,[ disease patterns,[ and critically ill patients with moderate or severe malnourishment.[

Limitations

A major limitation in this study was the small sample size when compared to the original validation studies. Additionally, all data used in this study were obtained from a single PICU. The small sample size is likely to interfere with the accurate application of the Hosmer–Lemeshow test for goodness of fit. Further confirmation of the results obtained in this study is warranted before the generalized use of these scores in a Chinese hospital setting.

Conclusion

This study has established that PRISM, PIM, and PIM2 scores discriminate between surviving and moribund patients. Both the PRISM and PIM2 models displayed good calibrations, while this was not the case for PIM. However, a positive correlation was observed across the PRISM, PIM, and PIM2 scores. Therefore, the application of PRISM had a good score in PICU in China.