[Performance of the Pediatric Index of Mortality 2 in a pediatric intensive care unit].

OBJECTIVE: To assess the discrimination and calibration of the Pediatric Index of Mortality 2 in patients admitted to a pediatric intensive care unit.
METHODS: The study was conducted with a contemporary cohort from November 2005 to November 2006. Patients aged 29 days to 18 years were included in the study. Patients who died within 12 hours of admission and cases of readmission were excluded from the study. The performance of the Pediatric Index of Mortality 2 was assessed by means of the Hosmer-Lemeshow goodness-of-fit test, the standardized mortality ratio and the area under receiver operating characteristic (ROC) curve with 95% confidence interval. The significance level was established as 5%.
RESULTS: A total of 276 admissions to the pediatric intensive care unit were included in the analysis. The mortality rate was 14.13%, and the efficiency of admission 0.88%. The median age of the sample was 42.22 months, and most participants were male (60.1%). Most admissions were referrals from the emergency department. The mean duration of stay in pediatric intensive care unit was 6.43±5.23 days. Approximately 72.46% of admissions were for clinical reasons and exhibited an association with the outcome death (odds ratio: 2.9; 95%CI: 1.09-7.74; p=0.017). Calibration of the Pediatric Index of Mortality 2 with the chi-square statistic was 12.2686 (p=0.1396) in the Hosmer-Lemeshow goodness-of-fit test, and the standardized mortality ratio was 1.0. The area under the ROC curve assessing model discrimination was 0.778.
CONCLUSION: Pediatric Index of Mortality 2 exhibited satisfactory performance.

INTRODUCTION

Ever since their appearance in Sweden in 1955,( pediatric intensive care units (PICUs) have provided a relevant contribution to the care of severely ill children based on the best use of human resources and high-cost equipment as well as continuous monitoring of patients, thus allowing for early intervention and better therapeutic outcomes, including recovery and prevention or reduction of permanent disabling injuries. However, this modality of care is associated with high financial costs. In the United States, the expenses associated with intensive care represented approximately 1% of the gross domestic product in 1994, i.e., USD 67 billion.(

The results of studies that investigated the demographic profile, mortality, morbidity and average stay in PICUs vary as a function of characteristics particular to various countries and even among different health services in the same country. These divergences not with standing, there is a consensus about the need for studies to improve the understanding of pediatric intensive care outcomes to allow for more adequate allocation of human, technological and financial resources.

For the reasons outlined above, studies have been performed to assess various methods to predict the severity and mortality risk of children, including the Pediatric Risk of Mortality (PRISM),( PRISM III,( Pediatric Index of Mortality (PIM),( PIM2( and PIM3.( The aim of the present study was to investigate the performance of PIM2 at the PICU of the Hospital Infantil Nossa Senhora da Glória (HINSG) by analyzing its calibration and discrimination.

METHODS

HINSG is affiliated with the public hospital network established by the State Department of Health of Espírito Santo, Brazil and is a state and regional reference hospital for specialized high-complexity pediatric care. The six-bed PICU of HINSG is a teaching and care-providing center that admits patients up to age 18 years, with 340 admissions per year on average.

Prospective data collection was performed from November 2005 to November 2006 in a contemporary cohort comprising all patients aged 29 days to 18 years. Patients who died within 12 hours of admission were excluded from the study, as well as cases of readmission and infants aged zero to 28 days. The study was approved by the Research Ethics Committee of HINSG, no. 43/2005, with a waiver of informed consent.

Efficiency of admission was defined as the percentage of patients that exhibited PIM2 >1% at admission. PIM2( includes clinical and laboratory data that are firmly established as part of the intensive care routine within the first hour of admission (blood pressure, use of mechanical ventilation, pupillary light reflex, arterial blood gases, recovery from surgical procedures, high- or low-risk diagnosis and elective admission or not). Thus, no intervention was performed exclusively for research purposes or involved risk for the patients. The PIM2 scoring system is in the public domain, and its authors authorized its use in the present study.

The performance of the PIM2 was assessed through analysis of its calibration and discrimination. Calibration evaluates how well the model classifies patients into low-, medium- and high-risk categories. For that purpose, the Hosmer-Lemeshow goodness-of-fit test was used, which is based on individual PIM2 values, and displays the observed-expected mortality ratios in deciles or in standard risk categories (<1%, 1-4%, 5-15%, 1629%, ≥30%). For this test, p-values >0.05 denote a good fit. The expected total mortality is the sum of the probabilities of death in each category. The standardized mortality ratio (SMR) represents the ratio of observed to expected mortality rates; values <1 imply good performance.(

Model discrimination was inferred from the area under the receiver operating characteristic (ROC) curve plotted using PIM2 values. The ROC curve represents the relationship between the sensitivity and specificity of a given test( and in the present case expresses how well the model distinguishes between patients who lived and those who died. Although for some authors an area under the ROC curve of 0.75 or more is considered clinically useful,( for others, 0.70 is adequate.(

Analysis was performed using Statistical Package for the Social Sciences (SPSS) software version 11.0, EpiInfo™ version 3.3.2 and MedCalc version 9.2.1.0. Descriptive statistics were performed (frequency tables, calculation of means and standard deviation), as well as exploratory hypothesis testing and application of the chi-square test of association. The significance level was established as 5%.

To analyze the performance of the PIM2, the Hosmer-Lemeshow goodness-of-fit test was used,( and the SMR and area under the ROC curve were calculated( together with the corresponding 95% confidence intervals (95%CI).

RESULTS

A total of 333 patients were admitted to the HINSG-PICU during the study period, and 57 were excluded (30 were newborn infants, 18 were readmissions, six died within 12 hours of admission, two were transferred to other hospitals and one was older than 18 years).

Relative to the 276 patients included in the study, the mortality rate was 14.13%. The efficiency of admission was 88%, the median age of the sample was 42.2 months, and most participants were male (60.1% of admissions). None of these demographic variables exhibited statistically significant correlation with an outcome of death (Table 1). With regard to patient origin, 48.9% were referred by the hospital emergency department, and this group exhibited a significant difference in death outcomes compared to the remainder of the sample (p=0.03). The median stay in the PICU was five days, with an average of 6.43±5.23 days. Table 2 describes the interquartile distribution of the studied sample according to age (in months), duration of stay in the PICU (in days) and PIM2 scores. Admissions for clinical reasons corresponded to 72.4% of the total number and exhibited association with an outcome of death (odds ratio- OR: 2.9; 95%CI: 1.09-7.74; p=0.01) (Table 3).

Table 1

Overall characteristics of the sample

Variables	Results
Number of patients/deaths	276/39
Mortality rate	14.13
Efficiency of admission	88
Age (months)	42.22 (5.38-122.25)
Origin
Emergency department	48.91
Hospital wards	25.00
Surgery department	17.03
Other hospitals	9.06
Duration of stay (days)	5 (3-8)
Reasons for admission
Clinical	72.46
Surgical	27.54

Efficiency of admission: number of patients with Pediatric Index of Mortality 2 >1% at admission. Results expressed as number (%), mean±standard deviation, or median (25%-75%).

Table 2

Interquartile distribution of the variables of age, duration of stay and Pediatric Index of Mortality 2 score (N=276)

	Age in months	Duration of stay in days	PIM2
Percentiles
25	5.38	3.00	1.6892
50	42.22	5.00	4.8373
75	122.25	8.00	10.1550

PIM 2 - Pediatric Index of Mortality 2.

Table 3

The variables origin and reason for admission according to outcomes

Variables	Outcomes		p value
	Discharge	Death
Origin
Emergency department	117 (49.37)	18 (46.15)	0.03
Hospital wards	57 (24.05)	12 (30.77)
Surgery department	45 (18.99)	2 (5.13)
Other hospitals	18 (7.59)	7 (17.95)
Admission
Clinical	166 (70.04)	34 (87.18)	0.01
Surgical	71 (29.96)	5 (12.82)
Total	237 (100)	39 (100)

Results expressed as number (%).

Tables 4 and 5 describe the results of the Hosmer-Lemeshow goodness-of-fit test and the SMR values expressed in deciles according to the five standard mortality risk categories described in the literature. The chi-square statistic was 12.26 (p=0.13) across deciles and 1.34 (p=0.71) across the five risk categories. In both cases, the overall SMR was 1.0, although the variation corresponding to the highest and lowest risk scores was overestimated. Figure 1 depicts the comparison between the expected and observed mortality rates per risk category. The area under the ROC curve was 0.77 (95%CI: 0.720.82), as shown in figure 2.

Table 4

Hosmer-Lemeshow goodness-of-fit test across Pediatric Index of Mortality 2 deciles and standardized mortality ratio according to the observed and expected mortality rates

Outcomes
Survival			Death
PIM2	Observed	Expected	Observed	Expected	SMR
1	28	26.112	0	1.888	0
2	27	26.028	1	1.972	0.51
3	27	25.950	1	2.05	0.49
4	26	25.781	2	2.219	0.90
5	25	25.602	3	2.398	1.25
6	25	25.399	3	2.601	1.15
7	23	25.137	5	2.863	1.75
8	25	24.390	3	3.61	0.83
9	17	21.934	11	6.066	1.81
10	14	10.668	10	13.332	0.75
Total	237	237.001	39	38.999	1.00

PIM 2 - Pediatric Index of Mortality 2; SMR - standardized mortality ratio. χ2=12.26; p=0.13.

Table 5

Hosmer-Lemeshow goodness-of-fit test across Pediatric Index of Mortality 2 risk categories and standardized mortality ratio according to the observed and expected mortality rates

Outcomes
Survival			Death
PIM2	Observed	Expected	Observed	Expected	SMR
0 I---1	33	32.22	0	0.78	0
1 I---5	102	102.54	7	6.46	1.08
5 I---15	72	72.11	12	11.89	1.01
15 I---30	19	20.26	10	8.74	1.14
>30	11	9.87	10	11.14	0.90
Total	237	237.00	39	39.01	1.00

PIM 2 - Pediatric Index of Mortality 2; SMR - standardized mortality ratio. χ2=1.34; p=0.71.

Figure 1

Deaths according to risk categories - Pediatric Index of Mortality 2.

Figure 2

Area under the ROC curve of the Pediatric Index of Mortality 2: 0.77 (95% confidence interval: 0.72-0.82).

DISCUSSION

The results of the present study indicate that the performance of the PIM2 was good. In addition, an association was found between mortality and admission for clinical reasons, as well as with origin from the emergency department. The duration of stay in the PICU was similar to those reported in other studies, with a mortality rate of 14.13% in the present study. The present study is part of a master dissertation project investigating the epidemiology of a PICU, in which the nosological features of the patient population and health care resources remained stable after the end of the study. Therefore, it may be concluded that the investigated PICU complies with the current recommendations of the Brazilian Health Surveillance Agency (Agência Nacional de Vigilância Sanitária - ANVISA) for the care of severely ill children. The investigated PICU does not perform therapeutic interventions, such as the use of extracorporeal membrane oxygenation (ECMO), inhaled nitric oxide, high frequency oscillatory ventilation, transplantations and use of the Swan-Ganz catheter.

The actual availability of PICU beds in our milieu results in a high prevalence of admissions of very severely ill individuals. In addition, as our hospital is a regional reference hospital for pediatric subspecialties, approximately 30% of the patients admitted to PICU exhibit comorbidities.

These facts not with standing, the healthcare outcomes based on the PIM2 score are similar to those reported by other centers for the same type of patients. Studies conducted in countries with limited resources found a mortality rate of 32%,( while Briassoulis( reported a mortality rate of 12% in one Greek PICU, and Brady( reported mortality rates varying from 3.2% to10.1% in various PICUs in the United Kingdom.

Of the demographic variables, age has been associated with an outcome of death. Einloft et al.( found a mortality rate of 13.22% among infants younger than one year, with a relative risk (RR) of 1.86 (95%CI: 1.65-2.10; p<0.0001). De Freitas Aragão et al.( found a statistically significant association between death in children admitted to the PICU and age below two years (p=0.007). In the present study, however, no significant differences were found in this regard.

A study conducted by El-Nawawy( that included 406 children admitted to the PICU of a pediatric teaching hospital in Egypt over a period of 13 months found a mortality rate of 38%. Approximately 57.9% of the children admitted to that PICU had been referred from the emergency department, although with no significant difference in outcome. In this context, the features inherent to emergency care of severely ill patients should also be taken into consideration, as interventions are performed in settings and under conditions quite different from those in other hospital departments.( This situation entails several possible determinants, including the type of care afforded, issues related to access to healthcare services, particularly high-complexity issues, organization of the healthcare network and difficulties associated with the regional distribution of specialized care. Therefore, further studies assessing those features are needed.

The association found in the present study between admissions for clinical reasons and an outcome of death agrees with findings reported by other authors. In a study conducted by López-Herce et al.,( 66.5% of admissions were for clinical reasons and associated with a greater mortality rate compared to surgical patients, 6.5±3.7% versus 2.6±3.2%.

The duration of stay of severely ill patients in the ICU varies among the various centers and ranges from 6.8 to 11.6 days on average.( In the present study, the duration of stay of approximately 43% of non-survivors was four to seven days, and approximately 70% of deaths occurred within seven days of admission.

In their study conducted in Greece, Briassoulis et al.( found an increase in the duration of stay in the PICU in 2001 following a period of stability (1997-2000), which they attributed to an increase in the number of admissions of patients with chronic diseases. In contrast, the case-control study conducted by van der Heide et al.( did not find significant differences between the characteristics of long-term patients and the control group.

The Hosmer-Lemeshow goodness-of-fit test yielded a chi-square value of 12.26 and p=0.14 across the analysis in deciles and a chi-square value of 1.34 and p=0.71 across the five risk categories. The overall SMR was 1.0, although the variation corresponding to the strata with the highest and lowest risk scores was overestimated. These results indicate satisfactory calibration.

In a study by Thukral et al.,( the overall SMR was 1.57, and with the Hosmer-Lemeshow goodness-of-fit test, the chi-square value was 7.64 (p=0.47). These authors called attention to the fact that their study population from India exhibited demographic and socioeconomic profiles that were different from those in the country where PIM2 was first formulated, in addition to a greater load of severity of illness being managed with fewer resources and differences in the quality of care, which may have contributed to an underestimation of the mortality risk.

Although the SMR was 0.85 in a study by Eulmesekian et al.(, the Hosmer-Lemeshow goodness-of-fit test did not yield significant results. These authors observed that several researchers have raised concerns about the use of the PIM2. In fact, the original PIM2 formulators recommended caution in the use of the Hosmer-Lemeshow goodness-of-fit test because small and clinically irrelevant differences within a large sample could exhibit the same p-value as large and clinically relevant differences within a small sample.

One further topic deserving of attention concerns the collection of data at admission. At that time, as a function of the support measures established, some children may exhibit stable clinical conditions despite the severity of disease, the progression of which over time may thus contradict predictions associated with low PIM2 scores, as Martha et al.( discussed with regard to the PIM.

Slater et al.( proposed that the variations in PIM2 calibration maybe associated with the following factors: results of patient intervention, changing attitudes as to the indications for commencing and discontinuing life support, changes in the thresholds for admission to intensive care, and the clinical profile of patients. These factors not with standing, these authors indicate a cutoff point of 1.0 for the PIM2 SMR.

The area under the ROC curve found by various authors indicates that the discrimination of the PIM2 is adequate: 0.90 (95%CI=0.89-0.92);( 0.81 (95%CI=0.750.87);( 0.84 (95%CI=0.82-0.86);( 0.97 (95%CI=0.96-0.99).( The variation among these values is due to the above mentioned impact of the profile of each individual center on model performance. In the present study, the area under the ROC curve was 0.778 (95%CI=0.725-0.826), which indicates that the model discrimination was satisfactory in the assessed population.

The PIM2 exhibited mixed results in populations that were different from the one in which it was originally tested. For that reason in 2013, Straney et al. published an updated version of the PIM2 known as the PIM3,( which was based on an international, multicenter, prospective cohort study conducted in 2010 and 2011. That version formulated a novel stratification of the variables corresponding to recovery from surgical procedures (cardiac surgery with bypass, cardiac surgery without bypass and other surgical procedures) as well as the risk categories, which are currently defined as low-, high- and very high-risk. The model discriminatory performance was better in Australia and New Zealand (area under the ROC curve: 0.91 [0.90-0.93]), compared to the United Kingdom and Ireland (area under the ROC curve: 0.85 [0.84-0.86]).

According to the current evidence, the predictive ability of the PIM2 is not affected when it is calculated using data collected four hours after admission to the PICU.( Other authors found that the PIM2 score discriminated well between survivors and deaths in the PICU and emphasized the use of PIM2 in the stratification of interventions.( One study conducted in a mixed ICU raised some concerns as to the use of the PIM2, as there was overestimation of deaths in the highest risk group, more particularly in cardiac-surgical patients.( This adjustment was based on the PIM3.(

The mortality risk prediction indices should be considered as auxiliary tools for the management of healthcare services and critically assessed as a function of each situation, and they must be systematically applied to the analysis of groups of patients and should never influence the management of individual cases. The present study had some limitations. As a function of its cross-sectional and descriptive design, the results entail limitations for the investigation of phenomena. Then, because the study was conducted in a single PICU, the results may not be extrapolated to other health services with different characteristics. Finally, the data collected for analysis are quite old, although the epidemiological and structural profile of the investigated PICU did not exhibit any changes, but the results agree with those reported in more recent studies.

CONCLUSION

The present study showed that the discrimination and calibration of the Pediatric Index of Mortality 2 were adequate. It further found an association between mortality and admission for clinical reasons, as well as with origin from the emergency department. As a function of its operational characteristics, the Pediatric Index of Mortality 2 may be used as an auxiliary tool for the management of intensive care units.