Performance of the Pediatric Index of Mortality 3 Score in PICUs in Argentina: A Prospective, National Multicenter Study.

OBJECTIVE: To assess the performance of the Pediatric Index of Mortality 3 score in a population of children admitted to PICUs in Argentina.
DESIGN: Prospective, national, multicenter study.
SETTING: Forty-nine PICUs located in Argentina belonging to public and private institutions. PATIENTS: All children between 1 month and 16 years old admitted to the participating PICUs between May 15, 2016, and February 15, 2017.
INTERVENTIONS: None. MEASUREMENT AND MAIN
RESULTS: A total of 6,602 patients were enrolled in the study. The observed mortality was 8% (531/6,602), whereas mortality predicted by Pediatric Index of Mortality 3 was 6.16% (407 deaths). The standardized mortality rate was 1.3 (95% CI, 1.20-1.42). The area under the receiver operating characteristic curve was 0.83 (95% CI, 0.82-0.85). The Hosmer-Lemeshow test showed that the difference between the mortality observed and the mortality predicted by Pediatric Index of Mortality 3 was statistically significant (χ, 135.63; p < 0.001).
CONCLUSIONS: The Pediatric Index of Mortality 3 score adequately discriminated patients who died from those who survived in our population. However, the observed mortality was higher than predicted by the score. The use of an updated instrument such as Pediatric Index of Mortality 3 will allow an actual comparison between pediatric intensive care provided in the country and care provided internationally. This might also allow future planning of pediatric intensive care services in Argentina.

Mortality prognostic scores are valuable tools for assessing the quality of care provided to critically ill patients (1). These are mathematical models, based on the presumption that there is a predictable relationship among severity of illness, evidenced by certain physiologic alterations, patient characteristics (diagnoses or complex chronic conditions [CCCs]), and risk of death (2, 3). It is assumed that the probability of death calculated before the initiation of intensive care treatment is independent of the quality of care received in the PICU. Therefore, these scores might be employed for assessing the results of each institution when compared with other institutions, directly (by comparing adjusted mortality) or indirectly (by comparing the number of actual deaths with the number of deaths predicted by the model) (4).

In the field of intensive care, the standardized mortality ratio (SMR), an indicator that compares the observed number of deaths with the predicted number of deaths in a specific period, is typically employed to assess the performance of PICUs (5). An SMR of 1 indicates a perfect agreement between observed and estimated mortality. If the performance of a PICU is higher or lower than expected, then the ratio will be below or above 1, respectively. However, for this interpretation to be valid, an adequate mortality prediction model is needed. At the same time, the use of prognostic mortality scores is essential for the assessment of quality of care in the PICU (6).

The Ministry of Health of Argentina recommends using the Pediatric Index of Mortality (PIM) 2 as a model for predicting the risk of death in PICUs (7). Similarly, this score is used for adjusting mortality by the Quality Benchmarking Program of the Argentine Society of Intensive Care (SATI-Q) (8, 9).

The performance of PIM2 was assessed in our country and Latin America and showed an adequate capacity for discriminating between nonsurvivors and survivors (10, 11). However, from the moment the model was developed to this day, the quality of intensive care provided around the world has changed dramatically due to advancements in technology and treatments.

In the last few years, modifications in the prediction capacity of the score have been detected in the countries where it was developed, showing an actual mortality below the PIM2 prediction (12, 13). In order to correct these changes observed in calibration, the authors published an updated version called PIM3 in 2013 (14). The performance of this new model is yet to be assessed in Argentina. Similarly, literature related to validations of the score in populations different from the ones it was generated in is scarce (15–17).

As using updated tools for assessing results from PICUs is of vital importance, we designed this research whose main objective was to evaluate the performance of the PIM3 score in a sample of patients admitted to PICUs in Argentina. Employing the PIM2 score as an instrument for predicting mortality, locally validated but developed more than 20 years ago, may condition an overvalued estimation of the results obtained in the ICUs in a country. For this reason, determining if this new version of the model can be used to adjust the risk of death of critically ill children in our country is essential. If so, an updated instrument that allows establishing a comparison between local intensive care and international care will be available.

MATERIALS AND METHODS

We designed a multicenter, observational, prospective, cross-sectional study. PICUs of Argentina were invited to participate through the scientific societies that group pediatric intensivists of the country.

All patients requiring intensive care between 1 month and 16 years old, admitted consecutively in participating PICUs between May 15, 2016, and February 15, 2017, were included. Newborns were excluded because they are usually treated in neonatal ICUs in Argentina. Patients admitted from other PICUs and those referred to other units for continuation of treatment or still hospitalized by March 1, 2017, were excluded from the analyses.

The following data were recorded during each admission: admission diagnosis, age, gender, length of stay in PICU, days of mechanical ventilation, outcome (survival or death) at discharge from PICU, and variables necessary for calculating the PIM3 score. The coefficients for each variable included in the model, the odds ratio, and their respective 95% CI are shown in Supplemental Table 1 (Supplemental Digital Content 1, http://links.lww.com/PCC/A772) (14). The presence of any CCC, as defined by the Feudtner classification (18), at the time of admission was also recorded. For describing participating units, number of beds, type of institution (general, pediatric), and type of management (private, public, or social security) were also recorded.

The instrument employed for data collection was the SATI-Q software (Hardineros Sistemas, Buenos Aires, Argentina), a computing tool provided free of charge to Argentine PICUs that voluntarily participate in the SATI-Q Program. This program is an initiative sponsored by SATI with the purpose of collecting data related to quality standards in intensive care (9). For this study, the software was updated to include the PIM3 score calculation as a new function.

To guarantee the quality of the data, each participating PICU designated a contact in charge of data recording and supervision. For standardization of the PIM3 calculation method, a standard operating procedure manual was created and delivered to each PICU. For training purposes, each person in charge in every unit was asked to calculate the score for five example cases. The results were sent via e-mail and subsequently discussed with the main research team. Each PICU sent the database within the first 2 months of data collection. At that time, the understanding of the protocol and the guidelines for the construction of the score were assessed, and the necessary modifications were implemented. An e-mail address for direct queries was made available for clarifying any doubts that arose during the study period.

Once the data collection period concluded, each participating unit sent the database to the coordination center for analysis. The data were sent encrypted and anonymized, for ensuring the safety of the data and the protection of personal information. The database is registered in the Argentine National Directorate for the Protection of Personal Data.

Ethical Considerations

The ethical and scientific aspects of the protocol were assessed and approved by the Research Ethics Committees of the participating units. In all cases, the need for informed consent for participation was waived because the data collected were considered routine practice in every PICU, the data protection requirements were met, and due to the observational characteristics of the study.

Statistical Analysis

Continuous quantitative variables were expressed as median and interquartile range (IQR) according to their distribution. Continuous discrete variables were expressed as median and range, and categorical variables were expressed as frequencies and percentages. The performance of the PIM3 score was assessed by analyzing its discrimination and calibration in the general population and different subgroups (age, diagnoses at admission, and presence of CCC). Simultaneously, SMR and its 95% CI were analyzed.

The discrimination or ability of the model to differentiate between survivors and nonsurvivors was assessed by calculating the area under the receiver operating characteristic curve (AUC-ROC) and its 95% CI. Calibration or degree of agreement between the number of predicted and observed events was calculated by using the Hosmer-Lemeshow goodness-of-fit test in the general population and stratified by deciles of risk.

In order to analyze the performance of PIM3 in the different subgroups, age groups were classified as follows: 1–11 months/12–59 months/60–119 months/120–191 months. The analyzed diagnostic groups were as follows: 1) cardiac (including postoperative), 2) injury, 3) neurologic, 4) postoperative (noncardiac), 5) respiratory, and 6) miscellaneous (19).

The AUC-ROC and SMR with their corresponding 95% CI were calculated in each age subgroup, diagnostic category, and according to the presence of any CCC.

Sample size was calculated based on the formula N = 10 k/p, where N is the minimum number of cases to be included, “k” is the number of independent variables included in the PIM3 logistic regression model, and “p” is the smallest proportion of expected positive cases in the population (deaths) (20). As PIM3 comprises 13 independent variables and expected proportion of deaths was calculated at 8% according to the latest general pediatric reports of the SATI-Q program (21), a sample size of 1,625 patients was calculated for obtaining a minimum of 130 events.

All statistical analyses were performed using Excel 2010 (Microsoft, Redmond, WA), Access 2010 (Microsoft, Redmond, WA), MedCalc version 16.4 (MedCalc Software bvba, Ostend, Belgium), and STATA IC/11 (StataCorp LLC, College Station, TX).

RESULTS

Fifty-two PICUs agreed to participate in the study. After the data collection period concluded, 49 units sent their records. PICUs characteristics and distribution of admitted patients according to the setting are described in Table (22). The median number of beds per PICU was 10 (range of 3–26). The median number of admitted patients per unit during the study period was 108 (IQR, 53–182); 38 PICUs (77, 5%) had less than or equal to 200 admissions.

During the study period, 7,075 admissions were registered in the participating PICUs; 473 (6.68%) were excluded from the analysis because of exclusion criteria, for being still hospitalized at the end of the study period or due to the existence of missing data necessary for the calculation of PIM3 (Fig. ). A total of 6,602 records were analyzed. Their characteristics are detailed in Table . Respiratory conditions were the main reason for admission in the PICU, and patients less than 1 year old were the predominant age group. Although the sample showed high prevalence of CCC, variability according to reason for admission was evident. Only 4% of patients admitted due to injuries had some CCC versus 85.8% of admissions due to cardiac conditions.

Flow chart of the study population.

Observed mortality in the sample was 8.04% (531/6,602), whereas mortality predicted by PIM3 was 6.17% (407 deaths). SMR was 1.3 (95% CI, 1.2–1.42). The difference between the number of deaths observed and PIM3 predicted deaths was statistically significant (p < 0.001).

The AUC-ROC for the entire cohort was 0.83 (95% CI, 0.82–0.85), showing an adequate performance of the score for discriminating between nonsurvivors and survivors.

Table shows the observed mortality and expected mortality in the different risk deciles, according to the goodness-of-fit test (Hosmer-Lemeshow). In all cases, the observed mortality was higher than PIM3 predicted mortality. The difference was statistically significant in the general population and for most deciles of mortality risk (χ2, 135.63; 8 degrees of freedom; p < 0.001). However, the difference between observed and expected mortality in the highest predicted mortality deciles (> 6.48%) was not statistically significant.

An analysis of discrimination and calibration of the model according to the volume of PICU number of admissions showed that for units with less than or equal to 200 admissions, the AUC-ROC was 0.84 (95% CI, 0.82–0.87) versus 0.82 (95% CI, 0.80–0.85) in units with more than 200 admissions. The SMR was 1.37 (95% CI, 1.21–1.55) and 1.25 (95% CI, 1.1–1.4), respectively.

Analysis by Age Group

The discrimination ability of PIM3 was adequate in all age groups. The observed mortality was higher than the mortality predicted by the score in all groups, especially in patients more than 120 months old. The difference between observed mortality and PIM3 predicted mortality was statistically significant (Table ), except in children less than 1 year old.

Analysis by Diagnostic Group

PIM3 showed adequate discrimination ability in all diagnostic groups. The lowest discrimination ability was observed in patients admitted for respiratory disease, showing an AUC-ROC of 0.70, with a 95% CI lower limit of 0.66.

Regarding the score calibration, the observed mortality was higher than the predicted mortality in all diagnostic groups, except in patients admitted for injuries. The difference between the observed mortality and PIM3 predicted mortality was statistically significant in all diagnostic groups, except in the neurologic category, injury category, and, although borderline, in postoperative admissions (Table ).

Analysis According to the Presence of CCC

PIM3 showed adequate discrimination ability in patients with some CCC when admitted to the PICU and in previously healthy patients. In this group, the observed mortality was higher than the PIM3 predicted mortality; the difference was statistically significant (276 observed deaths vs 179.3 predicted deaths). Although the observed mortality in previously healthy patients was higher than the expected mortality, this difference was not statistically significant (Table ).

DISCUSSION

Mortality risk prediction models in PICUs are usually built in developed countries, based on a population with particular characteristics according to its case mix, available resources, and health system organization. Before being implemented by a country as tools for measuring intensive care quality and individual performance of PICUs, they must be validated in a locally representative sample of patients.

This study was carried out to assess the performance of the PIM3 score in a population of patients admitted to PICUs in Argentina, a medium-to-high income country (according to the World Bank classification), and with health system characteristics different from the countries where the score was developed (23). The obtained results indicate that the score has an adequate capacity for discriminating between survivors and nonsurvivors, in the general population and all the different age and diagnostic subgroups. However, the observed mortality exceeds the mortality predicted by the score.

In general terms, its performance is comparable to PIM2 according to the validation study carried out in Argentina in 2009 and Latin America in 2013, both in terms of discrimination and calibration capacity (10, 11). This study yielded a PIM3 AUC-ROC of 0.83 in the general population: 83% of nonsurvivors showed a higher PIM3 predicted death probability than survivors, compared with 88% of patients in the population in which the score was developed (14). Similarly, previous PIM2 validation studies carried out in Argentina and Latin America showed an adequate discrimination ability, with AUC-ROCs of 0.84 and 0.82, respectively (10, 11).

Instead of 407 predicted deaths, 531 deaths were observed in the sample. The observed mortality was higher than the expected mortality in all intervals of risk probability and in the majority of the analyzed subgroups (age, diagnostic, presence of CCC). Similarly, during the PIM2 validation study carried out in Argentina in 2009, 297 deaths were observed versus 246 predicted deaths. This tends to happen when mortality risk prediction scores are used in populations other than those they were developed in, especially when said populations have different characteristics, in terms of admission pathologies in PICUs, comorbidities, and health system (fragmentation, accessibility, available resources, and social-sanitary conditions).

Regarding the characteristics of the admitted population, we can mention that PIM3 predicted mortality in our population was 6.17%, higher than the mortality predicted in the regions in which the model was developed (4.1% in the United Kingdom/Ireland and 2.8% in Australasia). Only 18.8% of patients in our sample had an elective admission versus 41% in the original population. Likewise, 16% of patients in our population were admitted for postsurgery recovery versus 39.7% of patients in the original sample.

The profile of children admitted in PICU in our population, more severely ill and non elective admissions, may reflect differences in admission criteria in argentine PICUs, or less accessibility to these units. These differences might not be adequately expressed by the model, affecting its performance when used in our area. But, at the same time, the excess of deaths observed might be interpreted as differences in the quality of care provided in our PICUs compared with the units in which PIM3 was developed.

So far, few studies assessing the performance of the model in external populations have been published. In a retrospective study, Wofler et al (15) reported an adequate performance of the score in a sample of Italian PICUs. In this population, AUC-ROC was 0.88 and SMR was 0.98. There were no statistically significant differences between predicted and observed mortality. On the contrary, Lee et al (16) reported an AUC-ROC of 0.77 and SMR of 1.29 in a sample of 1,656 patients from one Korean PICU. The PIM3 score showed higher performance in Italy, possibly as it has a population and health system of characteristics similar to the United Kingdom and Australasia.

The analysis by age group indicated that teenagers showed the greatest difference between observed mortality and model-predicted mortality, reporting an SMR of 1.89. Similarly, Wofler et al (15) reported an SMR of 1.4 for this population in Italian PICUs (15). These groups will likely show characteristics affecting the score performance, such as different CCCs that were not considered as an adjustment variable by PIM3. This challenge in mortality risk prediction for teenage patients is also evidenced by other scores built with different statistical techniques such as the one introduced by Arzeno et al (2), who suggest the need to develop specific scores for this population.

In the analysis performed according to the diagnosis at admission, observed mortality was higher than mortality predicted by PIM3 in all groups, except in patients admitted for injury. This is similar to the results observed in the PIM2 validation carried out in Latin America, in which 22 Argentinian PICUs participated (11). This finding is possibly related to a higher score performance in patients without previous comorbidity because only 25 patients (4%) admitted for injuries had some CCC in our sample. Calibration capacity was inadequate, showing statistically significant differences between expected and observed mortality in all groups, except in patients admitted for neurologic problems and injuries. Paradoxically, calibration in the original sample failed in patients admitted for neurologic problems.

The analysis of the population according to the presence of CCC showed that PIM3 performance in patients with previous comorbidities was inadequate in terms of calibration. Although mortality in patients without CCC was higher than expected (6.99% vs 6.25%), the difference was not statistically significant and SMR was 1.12. In contrast, the SMR for patients with CCC was 1.54. Currently, no studies on the assessment of score performance in this particular group have been published. The PIM3 model considers leukemia, postinduction lymphoma, neurodegenerative diseases, or bone marrow transplant, as adjustment variables of high and very high risk of death, but excludes as risk factors conditions such as HIV or post-liver transplant admissions, which are still associated with higher mortality rates in our country. According to reports from the World Health Organization, the HIV-AIDS mortality rate reached 0.9 per 100,000 inhabitants in Australia versus 8.9 per 100,000 in Argentina (24) by 2012. Similarly, other oncologic or immune-hematologic pathologies, which are not considered in the score as risk factors, may be associated with a higher mortality rate in PICUs in our region. Decreased availability of palliative care and anticipated decisions regarding end-of-life care may influence access to ICUs for patients with low recovery capacity in Argentina. The above mentioned factors might partially explain a better performance of the model in children admitted without comorbidities.

Other conditions that might explain the excess of deaths observed in our population are the characteristics of the Argentinean health system, highly fragmented and not centralized, existing high number of PICUs that treat a small volume of patients (25). In our study, 77% of the units that participated had a low volume of admissions (200 or less). In these units, the SMR was higher than in the units with larger volume of admissions. It is possible that reductions in mortality could be achieved if critical patients were admitted to large PICUs, as proposed by other authors (26).

As a limitation of the study, we can mention that neonatal patients were not included even though the original PIM3 model has included this in their assessments. This is because neonatal (<28 d old) admissions are managed in neonatal ICUs in our country. Another limitation is that not all PICUs in the country were included, as no entity groups them in a mandatory manner. However, units in public and private hospitals, and in general and pediatric hospitals, are represented in the sample. Furthermore, PICUs from all five regions of the country considered by the Ministry of Health took part in the study although the central region provinces showed a clear predominance, which reflects the concentration of the Argentine population in that region (27).

In contrast, the study results could be generalized for PICUs that are members of the SATI-Q pediatric program, as 30 of the 33 units participating in the registry in 2015 were also included in this research (28). This program, sponsored by the SATI since 2005, has the voluntary participation of units located in different provinces of the country. It represents a source of free and publicly available data, which provides information on quality indicators in Argentinean PICUs for benchmarking purposes. A general report on predefined quality indicators, like the SMR resulting from the analysis of the total number of admissions in the participating units, is prepared and published annually.

Understanding the performance of PIM3 in a local representative sample allows us to use the score as a mortality prediction tool for the construction of SMR in each individual PICU and at a national level. The results of this study show that using PIM3 to predict mortality, the actual SMR for PICUs participating in the SATI-Q program is 1.3, instead of the values observed in recent years using PIM2 (21). The performance of each participating PICU can be assessed by comparing their obtained SMRs with this value. This analysis conducted on an annual basis, can detect changes in population characteristics and the score performance, and will allow the comparison of each PICU against a local SMR and the comparison of SMR over time, as it has been performed with the use of PIM2 to date.

We suggest that it would be optimal to switch from PIM2 to PIM3 as the score to predict mortality in Argentine PICUs given that using a nonupdated model, like PIM2, might result in the misconception that care in our units is better than it actually is. On the other hand, using an up-to-date tool to compare care provided in local PICUs with international care for benchmarking purposes is necessary to highlight characteristics in the local care model that could have an impact on our patients’ outcomes.

An objective measurement of the results is necessary to evaluate the impact of the measures aimed at improving the care of the critical pediatric patient, either by improving the detection, the quality of the initial care, the accessibility to the PICUs, or the human and technological resources available in them.

CONCLUSIONS

This study assessed the performance of the PIM3 score in a large sample of patients admitted to PICUs in Argentina. The score showed an adequate ability to discriminate between the population of patients who survive and those who die. Instead, observed mortality was higher than predicted mortality in the general population and the population stratified by age, diagnosis or presence of CCC. The use of an updated instrument such as PIM3 will allow an actual comparison between pediatric intensive care provided in the country and care provided internationally. This might also allow future planning of pediatric intensive care services in Argentina.

ACKNOWLEDGMENTS

We express our most sincere gratitude to all PICU members who participated in the study and for their commitment to the project. We also thank the Argentine Society of Intensive Care who provided the SATI-Q software as a data collection tool for this study and has continuously supported research projects proposed by its members.

TABLE 1.

Characteristics of PICUs and Volume of Admissions

TABLE 2.

Characteristics of the Patient's Population (n = 6,602)

TABLE 3.

Hosmer-Lemeshow Goodness-of-Fit Test for Deciles of Mortality Risk: χ2: 135.63; 8 degree of freedom; p < 0.001

TABLE 4.

Model Fit and Discrimination by Age Groups

TABLE 5.

Model Fit and Discrimination by Admission Diagnostic Groups

TABLE 6.

Model Fit and Discrimination According to the Presence of Chronic Complex Conditions