ICU-free days as a more sensitive primary outcome for clinical trials in critically ill pediatric patients.

BACKGROUND: Our objective was to assess the association between intensive care unit (ICU)-free days and patient outcomes in pediatric prehospital care and to evaluate whether ICU-free days is a more sensitive outcome measure for emergency medical services research in this population.
METHODS: This study used data from a previous pediatric prehospital trial. The original study enrolled patients ≤12 years of age and compared bag-valve-mask-ventilation (BVM) versus endotracheal intubation (ETI) during prehospital resuscitation. For the current study, we defined ICU-free days as 30 minus the number of days in the ICU (range, 0-30 days) and assigned 0 ICU-free days for death within 30 days. We compared ICU-free days between the original study treatment groups (BVM vs ETI) and with the original trial outcomes of survival to hospital discharge and Pediatric Cerebral Performance Category (PCPC).
RESULTS: Median ICU-free days for the BVM group (n = 404) versus ETI group (n = 416) was not statistically different: 0 ICU-free days (interquartile range, 0-10) versus 0 (0-0), P = 0.219. Median ICU-free days were greater for BVM group in 3 subgroups: foreign body aspiration 30 (0-30) versus 0 (0-21), P = 0.028; child maltreatment 0 (0-14.2) versus 0 (0-0), P = 0.004; and respiratory arrest 25 (1-29) versus 7.5 (0-27.7), P = 0.015. In the original trial, neither survival nor PCPC demonstrated differences in all 3 subgroups-survival was greater with BVM for child maltreatment and respiratory arrest and favorable PCPC was greater with BVM for foreign body aspiration. Overall, in the current study, patients with more ICU-free days also had greater survival to hospital discharge and more favorable PCPC scores.
CONCLUSIONS: This initial study of the association between ICU-free days and patient outcomes during prehospital pediatric resuscitation appears to support the use of ICU-free days as a clinical endpoint in this population. ICU-free days may be more sensitive than either mortality or PCPC alone while capturing aspects of both measures.

INTRODUCTION

Background

The design of a clinical trial in critically ill or injured pediatric patients, a highly heterogeneous population, ideally centers on a patient‐centered primary outcome, one that captures both the desirability of each patient's clinical course and their final status. This population may be affected by serious illness with extremely high mortality (eg, sudden infant death syndrome and cardiac arrest), high morbidity but low mortality (eg, severe asthma exacerbations), or both high mortality and morbidity (eg, multiple traumatic injuries). , Thus, a generally applicable outcome measure must capture both mortality and the speed or extent of recovery, the latter reflecting the degree of morbidity associated with the acute illness or injury. , , , ,

Traditionally, outcome measures in cardiac arrest resuscitation research have focused on processes of care (eg, response times and procedure success) and clinical outcomes (eg, return of spontaneous circulation, survival and neurological outcome). , , , , The traditional clinical outcomes are usually binary variables. A common challenge to the design of resuscitation trials is that the reliance on traditional binary outcome can result in large minimum sample sizes, which make the trial logistically challenging or impossible. Even more problematic is applying precious and extensive resources to conduct a clinical trial in which important differences in clinical outcomes are not detected because the main outcome measure lacked sensitivity for the true differences. Investigators have explored alternative outcome measures, such as the length of time a patient requires care in an intensive care unit (ICU) (eg, ICU days or days on mechanical ventilation). , , , However, a shorter ICU stay may result from either a rapid recovery or an early death. , , , , ,

A common solution to the problems with traditional outcome measures is to use treatment‐free days (eg, ICU‐free days and ventilator‐free days), defined as the length of time, within a pre‐specified period, for which the patient is alive and free of the specific treatment being tracked. , , A number of authors have suggested using ICU‐free days as a standard main outcome measure for clinical trials. ICU‐free days assumes that remaining in the ICU is a general indicator of the ongoing severity of illness, with the additional caveat that any patient who dies within the period of observation (eg, 30 or 90 days) is assigned zero ICU‐free days, even if they were transferred out of the ICU and alive for some period during their hospital stay. ,

Importance

Emergency medical services (EMS) research will benefit from a generally applicable outcome measure that captures both the final mortality outcome and the speed of recovery. Moreover, such an outcome measure would likely be a more sensitive measure of treatment effect than mortality or other binary outcomes. A more sensitive main outcome measure, as the basis for a trials sample size calculation, might also lead to briefer, less expensive research, a critical issue in EMS research. Whether the number of ICU‐free days reasonably captures the outcomes of critically ill and injured children, however, is unknown.

Objective of this study

Our objective was to compare ICU‐free days, as an alternative clinical trial outcome, with survival and Pediatric Cerebral Performance Category (PCPC). , , We used data collected from the Pediatric Airway Management Project—a randomized controlled trial of bag‐valve‐mask ventilation (BVM) versus endotracheal intubation (ETI) during prehospital airway management in children. In the current study, we compare ICU‐free days to the traditional outcomes of survival and neurological outcomes, both for the main treatment groups (BVM vs ETI) and patient subgroups.

METHODS

Study design

This study is a secondary analysis of data collected prospectively during a previous randomized clinical trial. We studied an alternative main outcome measure (ICU‐free days), which could be derived from the existing clinical trial data. , This secondary analysis of de‐identified data was determined to be exempt from the requirement for Institutional Review Board (IRB) approval.

Selection of participants and setting

Original study enrollment was conducted from March 15, 1994 to January 1, 1997. Patients were eligible if they were either aged 12 years or younger or estimated to weigh 40 kg or less and if they required airway management based on 1 or more of the following criteria: cardiopulmonary arrest (patient apnea without a palpable pulse); respiratory arrest (patient apnea only, with pulse present); respiratory failure (with respiratory rate >60/min or <12/min) with a non‐purposeful response or no response to pain; complete or severe partial airway obstruction; traumatic cardiopulmonary arrest; traumatic respiratory arrest; closed or open head trauma with a non‐purposeful response or no response to pain; and paramedic assessment that assisted ventilation was necessary.

The original clinical trial was conducted in Los Angeles and Orange Counties, California, which are contiguous metropolitan areas with a well‐established EMS system. At the time of the trial, both counties had 2‐tiered 911 systems consisting of basic and advanced life support units, and base hospitals provided online medical direction for the out‐of‐hospital treatment of critically ill pediatric patients. Paramedics transported critically ill or injured pediatric patients according to the appropriate guidelines for each county. Adult ETI and BVM had been practiced in both counties for over 10 years when the study began, though pediatric ETI was introduced specifically for the study.

Original study intervention

The design of the trial and the trial results have been published previously. , Briefly, paramedics performed airway management for critically ill pediatric patients according to a predetermined protocol. Data were collected by a combination of study form completion (paramedic and ED physician) and structured chart review. Paramedics in the study EMS system received intensive study‐specific education on BVM and ETI prior to study commencement. Patients were assigned by calendar day to receive BVM (odd days) or BVM followed by ETI (even days). Pediatric Magill forceps for foreign body removal could be used on either odd or even days when basic life support maneuvers failed.

Outcomes

In the original clinical trial, survival to discharge from an acute care hospital and neurological status were evaluated retrospectively, by chart review. Neurological status was classified by a 5‐category, ordinal scale based on a modified PCPC: normal or no change from baseline, mild disability, moderate disability, severe disability, or coma or vegetative state. ,

Relevant patient subgroups were defined prior to the collection of study data based on expert consensus, including experts in pediatric emergency care and EMS. Patient subgroups were based on final diagnosis and included sudden infant death syndrome, submersion injury, head injury, multiple traumas, foreign body aspiration, seizure, child maltreatment, cardiopulmonary arrest, respiratory arrest, and reactive airway disease.

For the current study, ICU data collected in the original trial were used to calculate ICU‐free days. We defined the ICU‐free days as 30 minus the number of days in the ICU (range, 0–30 days). For patients who survived and were in the ICU for less than 30 days, the ICU‐free day's outcome measure was obtained by subtracting the length of the ICU stay from 30. Any patient who died at any time before or on day 30 was assigned an outcome of zero ICU‐free days. Patients in the ICU for 30 or more days were also assigned zero ICU‐free days. This approach is reasonable as the neurological status of critically ill patients treated for more than 30 days in the ICU would likely be poor. , , ,

As main outcome measures for clinical trials, survival and related outcomes have significant limitations. Using data from a well‐known pediatric prehospital trial, ICU‐free days compared favorably to survival and to neurologic outcome, with the potential to detect smaller treatment effects.

Analysis

ICU‐free days were summarized using the descriptive statistics median and interquartile ranges (IQR). We compared differences in ICU‐free days for both the main study groups (BVM and ETI) and the patient subgroups using the Wilcoxon rank sum test, with P <0.05 considered statistically significant. Data were analyzed using the R statistical software package (RStudio Inc., Boston, MA, USA). Additional information on the design, implementation, and results of the Pediatric Airway Management Project can be found in previously published articles. ,

RESULTS

Enrollment

The original study enrolled 830 patients, with 410 assigned to the BVM group and the remaining 420 to the ETI group. Ten patients were excluded from this analysis due to incomplete records—6 in the BVM group and 4 in the ETI group. Data for all the other patients in the population were retained and analyzed according to the intention‐to‐treat principle. Therefore, the final sample consisted of 820 patients (404 BVM, 416 ETI).

Of the 820 patients analyzed, 587 died (ETI: 306, BVM: 281) and 233 survived (ETI: 110, BVM 123) during follow‐up. The vast majority of deaths occurred early, with 584 (99%) in the first 30 days in the ICU.

Patient characteristics

The baseline characteristics of enrolled patients are shown in Table 1 as reported in the original trial. As reported in the original trial publication, median age was 1.2 years for the BVM group and 1 year for the ETI group. We found no statistically significant differences between the treatment groups in gender, ethnicity, emergency department disposition, or subgroups categorized by the apparent etiology of the illness or injury. However, a nominally significant difference was noted in the distribution of patients with a final diagnosis of sudden infant death syndrome.

TABLE 1

Patient demographics by pediatric airway management group

	No. (%) of patients
Demographic characteristic	BVM	ETI	P
Age (years) median (IQR)	1.25 (0.25–3.73)	1.0 (0.25–3.33)	0.77
Sex, male	247/403 (61)	236/415 (57)	0.20
Ethnicity			0.87
Hispanic	172 (45)	174 (44)
White	106 (28)	102 (26)
Black	69 (18)	75 (19)
Asian	25 (6)	26 (7)
Other	10 (3)	15 (4)
Patient disposition			0.79
Died	219 (54)	231 (56)
ICU	83 (20)	77 (18)
Transfer	67 (17)	78 (19)
Operating room	14 (3)	16 (4)
Ward or nursery	11 (3)	8 (2)
Home	9 (2)	6 (1)
Patient declared dead without resuscitation in the ED	123/367 (34)	110/369 (30)	0.28
Final diagnosis
Sudden infant death syndrome	59 (14)	82 (19)	0.049
Submersion injury	56 (14)	43 (10)	0.13
Head injury	27 (7)	36 (9)	0.28
Multiple trauma	37 (9)	51 (12)	0.15
Foreign body aspiration	13 (3)	13 (3)	0.95
Status epilepticus	38 (9)	33 (8)	0.47
Child maltreatment	24 (6)	22 (5)	0.70
Cardiopulmonary arrest	293 (71)	303 (72)	0.83
Respiratory arrest	55 (13)	55 (13)	0.89
Reactive airway disease	12 (3)	11 (3)	0.80

Abbreviations: BVM, bag‐valve‐mask ventilation; ETI, endotracheal intubation; IQR, interquartile range; ED, emergency department.

Age data were available for 402 patients in the BVM group and 409 in the ETI group. Gender data were available for 403 patients in the BVM group and 415 in the ETI group. Ethnicity data were available for 382 patients in the BVM group and 392 in the ETI group. Disposition data were available for 403 patients in the BVM group and 416 patients in the ETI group. Final diagnosis data were available for 410 patients in the BVM group and 420 in the ETI group. Adapted with permission from JAMA.2

Main outcome

Median ICU‐free days was 0 (IQR 0–10) for the 404 patients in the BVM group and 0 (0–0) for the 416 in the ETI group (P = 0.219; Table 2). Median ICU‐free days was significantly higher for BVM versus ETI in 3 of 10 subgroups: children with foreign body aspiration, child maltreatment, and respiratory arrest (P = 0.028, 0.004, 0.015, respectively).

TABLE 2

Subgroup ICU‐free days by airway management method

	BVM (n = 404)		ETI (n = 416)
Final diagnosis	Survived to discharge/total patients in group (%) a	ICU‐free days median (IQR) b	Survived to discharge/total patients in group (%) a	ICU‐free days median (IQR) b	P
Sudden infant death syndrome	0/58 (0)	0 (0–0)	0/80 (0)	0 (0–0)	NA
Submersion injury	18/55 (32)	0 (0–10.5)	20/43 (46)	0 (0–28)	0.070
Head injury	8/25 (32)	0 (0–13)	9/36 (25)	0 (0–0)	0.589
Multiple trauma	7/37 (18)	0 (0–0)	12/51 (23)	0 (0–0)	0.326
Foreign body aspiration	9/13 (69)	30 (0–30)	5/13 (38)	0 (0–21)	0.028
Seizure	35/37 (94)	29 (27–29)	26/32 (81)	29 (22.2–30)	0.696
Child maltreatment	10/24 (41)	0 (0–14.2)	1/22 (4)	0 (0–0)	0.004
Cardiopulmonary arrest	24/290 (8)	0 (0–0)	24/301 (7)	0 (0–0)	0.719
Respiratory arrest	46/54 (85)	25(1–29)	33/54 (61)	7.5 (0–27.7)	0.015
Reactive airway disease	6/12 (50)	14 (0–29)	3/10 (30)	0 (0–16.5)	0.220
Overall	123/404 (30)	0 (0–10)	110/416 (26)	0 (0–0)	0.219

Abbreviations: BVM, bag‐valve‐mask ventilation; ETI, endotracheal intubation; IQR, interquartile range.

Number of patients in each subgroup based on available data to determine ICU‐free days.

Data for ICU days are presented by median and IQR. P calculated by Wilcoxon rank‐sum test.

Figure 1 displays the distributions of ICU‐free days, for the overall group and patient subgroups. In the overall and most subgroups, the median ICU‐free days is 0; however, for the seizure subgroup, median and interquartile numbers were both greater than 20, showing the highest number of ICU‐free days regardless of treatment group. The median values in the foreign body aspiration and respiratory arrest group are also higher than those of the other subgroups.

FIGURE 1

Distribution of ICU‐free days among patient subgroups. Each panel shows ICU‐free days by treatment group, either BVM or ETI. The leftmost panel in the top row shows the overall comparison of ICU‐free days between the BVM and ETI treatment groups. The overall comparison of BVM versus ETI was not significant (P = 0.219). Remaining panels show the BVM versus ETI comparison within a subgroup. BVM, bag‐valve‐mask‐ventilation; ETI, endotracheal intubation; SIDS, sudden infant death syndrome

ICU‐free days versus neurological outcomes by treatment group are displayed in Table 3. ICU‐free days by neurological outcome across both treatment groups and all patient subgroups are shown in Figure 2. In general, patients with better neurological outcomes experienced a greater number of ICU‐free days. Median ICU‐free days for the higher PCPS scores (normal or no change from baseline) was 29 (27.2–30) for BVM and 28 (23–29) for the ETI group. When exceptions occurred, they were in patient categories with limited sample size.

TABLE 3

ICU‐free days by neurologic outcome

Neurologic outcome	BVM (n = 404)		ETI (n = 416)
	No. of patients	ICU‐free days median (IQR)	No. of patients	ICU‐free days median (IQR)
Normal or no change from baseline	72	29 (27.2–30)	58	28 (23–29)
Mild disability	20	27 (22.5–29)	27	24.5 (18.2–28)
Moderate disability	6	14.5 (5.7–20.2)	7	19 (6–22.5)
Severe disability	10	13 (0–14.7)	6	21.5 (17.2–25)
Coma/vegetative	15	10 (0–15)	12	8.5 (0.75–16.2)
Death	281	0 (0)	306	0 (0)

Abbreviations: BVM, bag‐valve‐mask ventilation; ETI, endotracheal intubation; IQR, interquartile range.

FIGURE 2

Distribution of ICU‐free days by PCPC category. PCPC, Pediatric Cerebral Performance Category

LIMITATIONS

This study has several important limitations. First, the analyzed dataset is relatively old. Although outcomes for children presenting in cardiac arrest (generally poor outcomes) and with seizures (generally good outcomes) are unlikely to have changed substantially since the original study, advances in pediatric emergency and critical care may have resulted in relatively better outcomes for some of the other groups. However, we believe that the relationship between PCPC and ICU‐free days likely remains valid, because these outcomes would be expected to improve in tandem as care advances. Second, although the comparison of ICU‐free days between patients treated with BVM and ETI yielded a greater number of statistically significant results in patient subgroups than the original outcomes, there is no guarantee that these differences represent true treatment effects within those subgroups. Specifically, we made no correction for multiple comparisons. Moreover, a separate analysis has demonstrated that some of the differences observed in subgroups may represent chance variation. Third, the Pediatric Airway Management Project trial reported a higher ICU mortality rate than most pediatric ICUs currently report; however, 75% of the patients in the trial were in cardiac arrest prior to ICU admission, and the overall survival rates of these patients were similar to those reported in recently published trials of therapeutic hypothermia in this population. , , , Our study suggests that the apparent gain in sensitivity with the proposed ICU‐free days outcome measure might be limited to pediatric populations with high mortality. Last, the ICU‐free day endpoint, as implemented here, assigns the same value, “0,” to a patient who dies within 30 days and to a patient who remains in the ICU for 30 days or longer. Although patients requiring 30 days of ICU care often have poor long‐term outcomes, there may be value in a modification to the approach we used in the current study. One alternative is assigning “−1″ to patients who die before 30 days, and this approach has been used in some settings.

DISCUSSION

In this study, we find evidence supporting the use of 30‐day ICU‐free days as a primary outcome measure for trials that compare therapies for critically ill or injured children in the prehospital setting, using data from the prehospital Pediatric Airway Management Project as the example. The available dataset includes a highly heterogeneous population of critically ill pediatric patients in prospectively defined subgroups with a wide range of outcomes.

An ideal measure of clinical trial outcomes captures clinically important and patient‐centered values, is objective and simple to implement prospectively or retrospectively, and is at least comparable to traditional outcome measures. When assessing the care of critically ill and injured children, the endpoint of ICU‐free days has many of these desirable characteristics, and is a common alternative outcome measure. ICU‐free days also captures factors related to the cost of care and the use of hospital resources. The ICU‐free days outcome measure may allow for more efficient clinical trial implementation and data collection because it is an outcome that is readily available from administrative and electronic health records. It has broader applicability compared with other, disease‐specific outcomes. This is ideal for clinical trials that aim to study interventions, such as prehospital pediatric airway management, that are applied to a wide range of ill and injured patients. Broad inclusion criteria are desirable since specific subgroups of critically ill children are rare, limiting the feasibility of conducting studies targeted to specific populations. Further, the number of ICU‐free days can be objectively derived from health records, with little interpretation, resulting in high interrater reliability. By assigning zero ICU‐free days to patients who die, regardless of whether there is a period during which they are alive and out of the ICU, the potential limitation associated with competing risks in composite endpoints is addressed.29 Otherwise, a patient who dies quickly in the ICU and thus has a short ICU stay would be assigned a favorable outcome identical to a patient who recovers quickly and is discharged home.

We have evaluated ICU‐free days as an alternative outcome using data from a previous controlled trial of BVM versus ETI in the prehospital care of critically ill and injured children. The original study captured both in‐hospital mortality and the neurologic outcomes of enrolled subjects at hospital discharge. Thus, the dataset provides a powerful tool with which to compare the ICU‐free days endpoint to well‐accepted and clinically relevant outcomes in pediatric critical care. ICU‐free days as a clinical outcome was able to capture both survival and neurological outcome utilizing a single outcome. In comparison, the original trial expressed this outcome separately as survival (BVM: 30% vs ETI: 26%) and good neurologic outcome (BVM: 23% vs ETI: 20%). Furthermore, we found that ICU‐free days appeared to be a more sensitive measure of treatment effect, as reflected by a larger number of statistically significant treatment effects in patient subgroups (child maltreatment, foreign body aspiration, and respiratory arrest), than either of the prior endpoints alone. In contrast, the previous trial showed differences in survival for the child maltreatment (BVM: 42% vs ETI: 5%) and respiratory arrest subgroups (BVM: 85% vs ETI: 61%) and in neurologic outcome in the foreign body subgroup (BVM: 69% vs ETI: 23%). This difference with the original trial suggests that the endpoint of ICU‐free days may be a practical, sensitive, and more powerful single measure of treatment effects (ie, allowing for reduced numbers of patients in subgroups while maintaining the power to detect differences in outcome) in evaluating the prehospital care of critically ill children, compared to either a simple mortality endpoint or an endpoint based on the PCPC.

In conclusion, based on a secondary analysis of data from a prospective clinical trial, we found ICU‐free days to be an appropriate primary endpoint for clinical outcome in EMS research in the pediatric population. ICU‐free days may be more sensitive than either mortality or PCPC scores alone in identifying treatment effects in the prehospital care of critically ill and injured infants and children.

AUTHORS CONTRIBUTIONS

All authors contributed to the study concept and design. HC, BW, and RJL wrote the initial versions of the manuscript. BW and RJL contributed statistical expertise. All authors contributed to the critical revision of the manuscript for important intellectual content.

CONFLICTS OF INTEREST

The authors declare no conflicts of interest.

SUPPORTING INFORMATION

Click here for additional data file.