Clinical implications of discrepancies in predicting pediatric mortality between Pediatric Index of Mortality 3 and Pediatric Logistic Organ Dysfunction-2.

BACKGROUND: Pediatric Index of Mortality 3 (PIM 3) and Pediatric Logistic Organ Dysfunction-2 (PELOD-2) are validated tools for predicting mortality in children. Research suggests that these tools may have different predictive performance depending on patient group characteristics. Therefore, we designed this study to identify the factors that make the mortality rates predicted by the tools different.
METHODS: This retrospective study included patients (<18 years) who were admitted to a pediatric intensive care unit from July 2017 to May 2019. After defining the predicted mortality of PIM 3 minus the predicted mortality rate of PELOD-2 as "difference in mortality prediction," the clinical characteristics significantly related to this were analyzed using multivariable regression analysis. Predictive performance was analyzed through the Hosmer-Lemeshow test and area under the receiver operating characteristic curve (AUROC).
RESULTS: In total, 945 patients (median [interquartile range] age, 3.0 [0.0-8.0] years; girls, 44.7%) were analyzed. The Hosmer-Lemeshow test revealed AUROCs of 0.889 (χ2=10.187, P=0.313) and 0.731 (χ2=6.220, P=0.183) of PIM 3 and PELOD-2, respectively. Multivariable linear regression analysis revealed that oxygen saturation, partial pressure of CO2, base excess, platelet counts, and blood urea nitrogen levels were significant factors. Patient condition-related factors such as cardiac bypass surgery, seizures, cardiomyopathy or myocarditis, necrotizing enterocolitis, cardiac arrest, leukemia or lymphoma after the first induction, bone marrow transplantation, and liver failure were significantly related (P<0.001).
CONCLUSIONS: Both tools predicted observed mortality well; however, caution is needed in interpretation as they may show different prediction results in relation to specific clinical characteristics.

INTRODUCTION

Predicting mortality is very important in the process of caring for critically ill patients. Depending on the likelihood of mortality, the urgency of the use of medical resources can be assessed, and the medical condition can be detected and treated early before progression [1-3]. There are various tools for predicting mortality in critically ill pediatric patients, among which Pediatric Index of Mortality 3 (PIM 3) and Pediatric Logistic Organ Dysfunction-2 (PELOD-2) are widely used worldwide [4,5]. The predictive performance and effectiveness of both prediction tools have been proven through several validation studies [6-9].

However, in a retrospective study of children who received bone marrow transplantation, it was reported that there was no significant difference in mortality predicted by PIM (previous version of PIM 3) between survivors and non-survivors [10]. Another study of patients undergoing surgery for congenital heart disease reported a weak relationship between the severity of the patient’s condition and the PELOD (previous version of PELOD-2) score [11]. Since it is important to be aware that the performance of a mortality prediction system may vary according to a specific disease or patient group, we attempted to find studies on PIM 3 and/or PELOD-2, which are the upgraded versions of PIM and PELOD, respectively. However, to the best of our knowledge, none of the available studies exactly fit this purpose. Therefore, we designed this study with the aim of determining whether there are patient group characteristics that influence the mortality predictive performance of PIM 3 and PELOD-2, and if any, we aimed to determine the specific factors that cause the difference in performance between these tools.

MATERIALS AND METHODS

Study Setting

This retrospective observational study was conducted at a 24-bed medical and surgical pediatric intensive care unit (PICU) of a tertiary hospital. Patients under the age of 18 years who were admitted to the PICU from July 2017 to May 2019 were included, and patients with vital signs that were considered non-physiologic were excluded from the analysis. The non-physiologic vital signs were defined as: heart rate (HR) above 300 beats/min or below 30 beats/min, respiratory rate (RR) above 120 breaths/min or below 5 breaths/min, body temperature above 42°C or below 30°C, and oxygen saturation below 30%.

Data Collection and Pre-processing

The following data were collected from the hospital’s electronic health records: age; sex; physical findings such as blood pressure (BP), HR, and RR; clinical findings such as vasoactive-inotropic scores and the use of mechanical ventilation; and laboratory findings such as blood gas analysis results and electrolyte levels. Among these variables, BP, HR, and RR, whose normal ranges change with age [12], were not used in order to avoid age-related bias, but the z-score for each variable was calculated and used for analyses. In the process of calculating the z-score, the “generalized additive models for location scale and shape” and “sitar” package of R software (R Foundation for Statistical Computing, Vienna, Austria) were used [13,14]. The PIM 3 and PELOD-2 scores were calculated using formulas presented in the development studies [4,5]. When calculating the scores of the above tools, the worst one was used within 1 hour of entering the PICU, and the results of the examination within 1 hour of entering the PICU were used based on the examination execution time, not the examination result report time. The process of recording data through the hospital information system was performed by one researcher, and the PIM 3 and PELOD-2 scores were calculated through R coding.

Outcome Measures

The primary outcome in this study was an analysis of factors affecting the mortality prediction performance of PIM 3 and PELOD-2. For this, the value obtained by subtracting the predicted mortality rate of PELOD-2 from the predicted mortality rate of PIM 3 was defined as “difference in mortality prediction”, and related factors were analyzed using multivariable linear regression. The secondary outcome was whether there was a difference from observed mortality in each subgroup; this was obtained by performing subgroup analysis on categorical variables among factors that were significantly related to “difference in mortality prediction” in the multivariable analysis.

Statistical Analysis

To analyze the relationship between the mortality predicted by PIM 3 or PELOD-2 and the observed mortality, the area under the receiver operating characteristic curve (AUROC) and Hosmer-Lemeshow goodness of fit test were used, and the grade was set to 10 steps. Linear regression analyses were used to analyze factors related to the “difference in mortality prediction,” and factors that showed significant results in the univariable analyses were used to create a multivariable linear regression model. The final model was derived using the backward selection method. All statistical analyses were performed using R version 4.0.3 (R Foundation for Statistical Computing, Vienna, Austria), and P<0.05 were considered statistically significant.

Ethics Statements

The need for obtaining ethics approval of the study protocol and written consent from the study participants was waived by the Institutional Review Board of the institution where this study was conducted (H-2004-229-1119).

RESULTS

Baseline Characteristics

During the study period, a total of 1,073 patients were screened, and 945 patients were finally analyzed after applying the inclusion and exclusion criteria (Figure 1). The median (interquartile range) age was 3.0 years (0.0–8.0 years), and 44.7% of the patients were girls. Table 1 provides detailed information regarding the baseline characteristics. The Hosmer-Lemeshow goodness of fit test was conducted to confirm whether the mortality predicted by the severity scoring tools differed from the observed mortality. The results of PIM 3 (AUROC=0.889, χ2=10.187, P=0.313) showed no statistically significant difference from the observed mortality. The results of PELOD-2 were also AUROC=0.731, χ2=6.220, P=0.183, showing that there was no difference from the observed mortality. Both PIM 3 and PELOD-2 showed fair to good predictive performance in predicting the observed mortality (Figure 2).

Figure 1.

Flowchart of the study population.

Table 1.

Demographic and baseline characteristics of the participants

Variable	Values (n=945)
Age (yr)	3.0 (0.0 to 8.0)
Female	422 (44.7)
Length of stay in ICU (hr)	24.2 (19.0 to 77.7)
Underlying disease
Cardiovascular disease	138 (13.3)
Endocrinologic disease	48 (4.6)
Gastrointestinal disease	84 (8.1)
Genetic disease	108 (10.4)
Genitourinary disease	62 (6)
Hemato-oncologic disease	174 (16.8)
Infectious disease	42 (4.1)
Neuromuscular disease	171 (16.5)
Ophthalmologic disease	27 (2.6)
Psychologic disease	38 (3.7)
Respiratory disease	119 (11.5)
Trauma	25 (2.4)
Physical finding
Z-score of SBP by age	0.0 (–0.7 to 0.7)
Z-score of DBP by age	–0.0 (–0.7 to 0.6)
Z-score of MBP by age	0.0 (–0.6 to 0.6)
Z-score of HR by age	0.0 (–0.7 to 0.7)
Z-score of RR by age	0.0 (–0.7 to 0.7)
Body temperature (℃)	36.8 (36.3 to 37.2)
Oxygen saturation (%)	99.0 (95.0 to 100.0)
Glasgow coma scale
Eye	4.0 (4.0 to 4.0)
Verbal	5.0 (5.0 to 5.0)
Motor	6.0 (6.0 to 6.0)
Fixed pupil reflex, both	14 (1.5)
Clinical finding
Vasoactive-inotropic score	0.0 (0.0–3.2)
Mechanical ventilation application	696 (73.7)
Laboratory finding
pH	7.4 (7.3 to 7.4)
Partial pressure of CO2 (mm Hg)	41.0 (36.0 to 47.0)
Total CO2 (mmol/L)	24.0 (21.5 to 27.2)
Base excess (mmol/L)	–0.8 (–3.5 to 1.4)
Leukocyte (×103 cells/μl)	9.4 (5.6 to 13.9)
Platelet (×103 cells/μl)	192.0 (106.0 to 282.0)
Glucose (mg/dl)	115.0 (87.0 to 156.0)
Potassium (mg/dl)	4.0 (3.6 to 4.4)
Lactate (mmol/L)	1.5 (0.9 to 2.5)
BUN (mg/dl)	8.0 (5.0 to 10.0)
Creatinine (mg/dl)	0.3 (0.1 to 0.4)
Bilirubin (mg/dl)	0.6 (0.3 to 1.0)
PT-INR	1.2 (1.1 to 1.3)
aPTT (sec)	34.7 (29.9 to 42.0)
Elective admission to ICU[a]	773 (81.8)
Association between ICU admission and surgery[a]
Not related to surgery	200 (21.2)
Bypass cardiac surgery	209 (22.1)
Non-bypass cardiac surgery	44 (4.7)
Non-cardiac surgery	492 (52.1)
Low-risk diagnosis[a]
None	909 (96.2)
Bronchiolitis	4 (0.4)
Diabetic ketoacidosis	6 (0.6)
Seizure	26 (2.8)
High-risk diagnosis[a]
None	908 (96.1)
Spontaneous cerebral hemorrhage	12 (1.3)
Cardiomyopathy or myocarditis	15 (1.6)
Hypoplastic left heart syndrome	1 (0.1)
Neurodegenerative disorder	3 (0.3)
Necrotizing enterocolitis	6 (0.6)
Very high-risk diagnosis[a]
None	913 (96.6)
Cardiac arrest	15 (1.6)
Severe combined immune deficiency	2 (0.2)
Leukemia or lymphoma after first induction	3 (0.3)
Bone marrow transplant recipient	6 (0.6)
Liver failure	6 (0.6)
Predicted mortality rate by PIM 3	2.0 (0.9 to 2.7)
Predicted mortality rate by PELOD-2	0.9 (0.5 to 1.4)
Observed all-cause ICU mortality	17 (1.8)

Values are presented as median (interquartile range) or number (%).

ICU: intensive care unit; SBP: systolic blood pressure; DBP: diastolic BP; MBP: mean BP; HR: heart rate; RR: respiratory rate; CO2: carbon dioxide; BUN: blood urea nitrogen; PT INR: prothrombin time international normalized ratio; aPTT: activated partial thromboplastin time; PIM 3: pediatric index of mortality 3; PELOD-2: pediatric logistic organ dysfunction-2.

For this classification, the criteria of the PIM 3 calculation formula were used [4].

Figure 2.

The receiver operating characteristic (ROC) curves according to situations of observed mortality and predicted mortality by Pediatric Index of Mortality 3 (PIM 3) and Pediatric Logistic Organ Dysfunction-2 (PELOD-2). P-values were derived using the Hosmer-Lemeshow goodness of fit test. AUC: area under the curve.

Main Outcomes

Based on the multivariable analysis, oxygen saturation (β=–0.065, standard error [SE]=0.012, P<0.001), base excess (β=–0.124; SE=0.024, P<0.001), a diagnosis of seizures (β=–3.598, SE=0.723, P<0.001), and cardiac bypass surgery (β=–2.083, SE=0.264, P<0.001) were associated with a decrease in the “difference in mortality prediction” between the tools. That is, the mortality rate predicted by PELOD-2 tends to increase to a greater extent than the mortality rate predicted by PIM 3 as the factors correspond to the above variables. In contrast, partial pressure of CO2 (β=0.041, SE=0.010, P<0.001); platelet counts (β=0.004, SE=0.001, P<0.001); blood urea nitrogen levels (β=0.045, SE=0.017, P=0.008); diagnoses of cardiomyopathy or myocarditis (β=3.810, SE=0.948, P<0.001), necrotizing enterocolitis (β=4.356, SE=1.356, P<0.001), cardiac arrest (β=20.691, SE=0.813, P<0.001), and leukemia or lymphoma after the first induction (β=9.066, SE=2.163, P<0.001); bone marrow transplantation (β=6.255, SE=1.542, P<0.001); and liver failure (β=5.907, SE=1.257, P<0.001) were associated with an increase in the “difference in mortality prediction” (the more the above factors were met, the higher the predicted mortality rate of PIM 3 was that of PELOD-2) (Table 2).

Table 2.

Demographics and baseline variables on linear regression analysis

Variable	Univariable analysis			Multivariable analysis
	Estimate	SE	P-value	Estimate	SE	P-value
Age (yr)	0.059	0.031	0.055
Sex
Male	Reference
Female	–0.466	0.310	0.133
Physical finding
Z-score of SBP by age	0.247	0.155	0.113
Z-score of HR by age	0.140	0.154	0.364
Z-score of RR by age	0.142	0.155	0.359
Body temperature (℃)	0.075	0.192	0.696
Oxygen saturation (%)	–0.091	0.015	<0.001	–0.065	0.012	<0.001
Glasgow coma scale
Eye	0.700	0.183	<0.001
Verbal	0.407	0.121	0.001
Motor	0.718	0.143	<0.001
Fixed pupil reflex	0.948	1.277	0.458
Clinical finding
Vasoactive-inotropic score	–0.006	0.008	0.410
Mechanical ventilation
No	Reference
Yes	0.672	0.350	0.055
Laboratory finding
pH	–15.690	1.656	<0.001
Partial pressure of CO2 (mm Hg)	0.082	0.014	<0.001	0.041	0.010	<0.001
Total CO2 (mmol/L)	–0.028	0.032	0.372
Base excess (mmol/L)	–0.223	0.035	<0.001	–0.124	0.024	<0.001
Leukocyte (×103 cells/μl)	0.053	0.021	0.013
Platelet (×103 cells/μl)	0.005	0.001	<0.001	0.004	0.001	<0.001
Glucose (mg/dl)	0.000	0.002	0.786
Potassium (mg/dl)	0.266	0.124	0.033
Lactate (mmol/L)	0.435	0.089	<0.001
BUN (mg/dl)	0.081	0.020	<0.001	0.045	0.017	0.008
Creatinine (mg/dl)	0.377	0.295	0.202
Bilirubin (mg/dl)	0.114	0.096	0.234
PT	0.070	0.024	0.004
PT INR	1.588	0.436	<0.001
aPTT (sec)	–0.002	0.009	0.808
Elective admission to ICU[a]
No	Reference
Yes	–2.825	0.389	<0.001
Association between ICU admission and surgery[a]
Not related to surgery	Reference			Reference
Bypass cardiac surgery	–2.786	0.361	<0.001	–2.083	0.264	<0.001
Non-bypass cardiac surgery	1.908	0.730	0.009
Non-cardiac surgery	0.018	0.309	0.955
Low-risk diagnosis[a]
None	Reference			Reference
Bronchiolitis	–3.210	2.375	0.177
Diabetic ketoacidosis	–2.215	1.942	0.254
Seizure	–2.998	0.938	0.001	-3.598	0.723	<0.001
High-risk diagnosis[a]
None	Reference			Reference
Spontaneous cerebral hemorrhage	2.024	1.377	0.142
Cardiomyopathy or myocarditis	3.281	1.230	0.008	3.810	0.948	<0.001
Hypoplastic left heart syndrome	0.654	4.747	0.891
Neurodegenerative disorder	0.934	2.743	0.733
Necrotizing enterocolitis	5.547	1.935	0.004	4.356	1.356	0.001
Very high-risk diagnosis[a]
None	Reference			Reference
Cardiac arrest	23.157	0.978	<0.001	20.691	0.813	<0.001
Severe combined immune deficiency	4.838	3.355	0.150
Leukemia or lymphoma after first induction	6.826	2.734	0.013	9.066	2.163	<0.001
Bone marrow transplant recipient	6.635	1.931	0.001	6.255	1.542	<0.001
Liver failure	5.937	1.933	0.002	5.907	1.257	<0.001

SE: standard error; SBP: systolic blood pressure; HR: heart rate; RR: respiratory rate; CO2: carbon dioxide; BUN: blood urea nitrogen; PT INR: prothrombin time international normalized ratio; aPTT: activated partial thromboplastin time; ICU: intensive care unit.

For this classification, the criteria of the PIM 3 calculation formula were used [4].

Table 3 shows the results of the subgroup analyses based on categorical variables among the factors confirmed to affect “difference in mortality prediction” in the multivariable analysis. Similar to the results of the multivariable analysis, PELOD-2 had a higher predictive mortality rate than PIM 3 in cases of bypass cardiac surgery or seizures, and PIM 3 had a higher predictive mortality rate in other cases. In the Hosmer-Lemeshow goodness of fit test result, only the predicted mortality of PIM 3 could be analyzed in both ‘cardiomyopathy or myocarditis’ and ‘cardiac arrest’ cases, and none of them showed a statistically significant difference; thus, PIM 3 predicted the actual observed mortality well in the subgroup (Table 3).

Table 3.

Subgroup analysis of factors related to the difference in predicted mortality between PIM 3 and PELOD-2

Variable	No. of patients	Observed mortality	PIM 3				PELOD-2
			Predicted mortality rate	AUROC	χ2	P-value	Predicted mortality rate	AUROC	χ2	P-value
Bypass cardiac surgery	209	0	0.7 (0.6–0.9)	NA	NA	NA	0.9 (0.5–1.4)	NA	NA	NA
Seizure	26	0	0.4 (0.2–0.6)	NA	NA	NA	1.1 (0.5–3.5)	NA	NA	NA
Cardiomyopathy or myocarditis	15	1 (0.1)	4.4 (3.0–5.9)	0.929	1.243	0.996	0.9 (0.3–2.2)	NA	NA	NA
Necrotizing enterocolitis	6	0	7.6 (6.5–10.3)	NA	NA	NA	0.7 (0.5–1.4)	NA	NA	NA
Cardiac arrest	15	3 (20.0)	22.0 (18.2–35.1)	0.639	11.809	0.160	3.5 (1.1–3.5)	NA	NA	NA
Leukemia or lymphoma after first induction	3	2 (66.7)	6.2 (3.5–13.8)	NA	NA	NA	1.4 (1.1–1.4)	NA	NA	NA
Bone marrow transplant recipient	6	1 (16.7)	8.6 (6.0–10.1)	NA	NA	NA	0.9 (0.5–2.2)	NA	NA	NA
Liver failure	6	1 (16.7)	9.2 (8.6–9.6)	NA	NA	NA	1.5 (0.5–3.5)	NA	NA	NA

Values are presented as number (%) or median (interquartile range).

PIM 3: Pediatric Index Of Mortality 3; PELOD-2: Pediatric Logistic Organ Dysfunction-2; AUROC: area under the receiver operating characteristic curve; NA: not applicable.

DISCUSSION

We conducted this study to determine whether there may be a difference in the predictive performance between PIM 3 and PELOD-2. Further, we investigated the specific factors that cause the difference. We found that both PIM 3 and PELOD-2 showed good performance in predicting the observed mortality; however, both showed slightly different results in predicting mortality according to the clinical characteristics of the patients.

Previous studies reported that the AUROC range of PIM 3 was 0.75–0.88 [8,15-19]. The AUROC of PELOD-2 was reported to be in the range of 0.75 to 0.94 [7,17-20]. In our results, the AUROC values of PIM 3 and PELOD-2 were 0.889 and 0.731, respectively, which were not significantly different from those in previous studies.

In this study, the “difference in mortality prediction” was affected by several factors, suggesting that the predicted mortality rates of PIM 3 and PELOD-2 may be affected by the characteristics of the patient group. As mentioned earlier, there is no existing study comparing PIM 3 and PELOD-2 according to patient group characteristics; thus, it was impossible to directly compare the results of our study with those in the existing literature. However, we were able to find one published paper suggesting that PELOD-2 scores may be lower in certain patient groups [20]. That study is a prospective observational study of critically ill children who needed plasma transfusion admitted to 101 PICUs in 21 countries. It was reported that the mortality prediction of PELOD-2 showed a fair performance, i.e., an AUROC of 0.76, but a relatively low predictive power compared to previous results, i.e., an AUROC of 0.934. In addition, the study concluded that the predictive power of PELOD-2 may be different in specific subpopulations [21]. Although plasma transfusion itself was not analyzed as a relevant factor in our study, PELOD-2 showed a lower mortality rate than PIM 3 in patients with leukemia, bone marrow transplantation, and liver failure who were expected to require large amounts of plasma transfusion. Of course, these results cannot be directly applied; however, we believe it might be a worthwhile point considering the relevance to existing studies conducted in patients who received plasma transfusions.

In the subgroup analysis results, most observed mortality was closer to the predicted mortality of PIM 3 than the predicted mortality of PELOD-2. However, we did not think that this was a result that meant that PIM 3 was superior to PELOD-2. This is because, while targeting a specific subpopulation, the sample size corresponding to each subgroup was very small (e.g., only three patients in the case of leukemia or lymphoma after first induction). Additionally, since this subgroup analysis is binary based on whether the listed factors are or not, there may be other potential confounding factors in each subgroup. Further, as a result of the Hosmer-Lemeshow test, most of the items were not applicable. The test was designed to compare the predicted mortality rate range (from 0% to 100%) with mortality by dividing it into 10 segments with 10% intervals. Therefore, it was difficult to derive results from the small sample population by dividing it into several subgroups.

This study has several limitations. First, this was a single center study. Thus, there may be differences in the results when the tools are applied in other institutions. However, we attempted to include a sufficiently large number of participants in our study. Second, the timing at which each tool was applied may have been different for individual patients. This is because the calculation definitions of PIM 3 and PELOD-2 are different, and our study was performed within a defined calculation time range. Third, the sample size was relatively insufficient to perform subgroup analysis. Finally, it was thought that clinical management before ICU admission might have been different for each patient, but it is not considered to be the focus of the study itself.

Both PIM 3 and PELOD-2 showed good results in predicting mortality but showed different predictive results depending on the specific clinical characteristics of the patient. Since the study was conducted at a single center and contained a relatively insufficient sample size, it may be difficult to directly apply the results of this study to other institutions. Therefore, it is necessary to supplement these results with multicenter studies including sufficient sample sizes in the future. Moreover, when applying and interpreting the above tools in clinical practice based on these results, it is necessary to consider the characteristics of each individual patient.

▪ The predictive performance of both Pediatric Index of Mortality 3 and Pediatric Logistic Organ Dysfunction-2 is good.

▪ There is a difference in performance between the tools based on patient characteristics and groups.