Prognostic relevance and inter-observer reliability of chest-imaging in pediatric ARDS: a pediatric acute respiratory distress incidence and epidemiology (PARDIE) study.

PURPOSE: Definitions of acute respiratory distress syndrome (ARDS) include radiographic criteria, but there are concerns about reliability and prognostic relevance. This study aimed to evaluate the independent relationship between chest imaging and mortality and examine the inter-rater variability of interpretations of chest radiographs (CXR) in pediatric ARDS (PARDS).
METHODS: Prospective, international observational study in children meeting Pediatric Acute Lung Injury Consensus Conference (PALICC) criteria for PARDS, which requires new infiltrate(s) consistent with pulmonary parenchymal disease, without mandating bilateral infiltrates. Mortality analysis focused on the entire cohort, whereas inter-observer variability used a subset of patients with blinded, simultaneous interpretation of CXRs by intensivists and radiologists.
RESULTS: Bilateral infiltrates and four quadrants of alveolar consolidation were associated with mortality on a univariable basis, using CXRs from 708 patients with PARDS. For patients on either invasive (IMV) or non-invasive ventilation (NIV) with PaO2/FiO2 (PF) ratios (or SpO2/FiO2 (SF) ratio equivalent) > 100, neither bilateral infiltrates (OR 1.3 (95% CI 0.68, 2.5), p = 0.43), nor 4 quadrants of alveolar consolidation (OR 1.6 (0.85, 3), p = 0.14) were associated with mortality. For patients with PF ≤ 100, bilateral infiltrates (OR 3.6 (1.4, 9.4), p = 0.01) and four quadrants of consolidation (OR 2.0 (1.14, 3.5), p = 0.02) were associated with higher mortality. A subset of 702 CXRs from 233 patients had simultaneous interpretations. Interobserver agreement for bilateral infiltrates and quadrants was "slight" (kappa 0.31 and 0.33). Subgroup analysis showed agreement did not differ when stratified by PARDS severity but was slightly higher for children with chronic respiratory support (kappa 0.62), NIV at PARDS diagnosis (kappa 0.53), age > 10 years (kappa 0.43) and fluid balance > 40 ml/kg (kappa 0.48).
CONCLUSION: Bilateral infiltrates and quadrants of alveolar consolidation are associated with mortality only for those with PF ratio ≤ 100, although there is high- inter-rater variability in these chest-x ray parameters.

Take-home message

Introduction

Acute respiratory distress syndrome (ARDS) includes non-cardiogenic pulmonary edema caused by damage to the alveolar epithelial–endothelial barrier. These changes result in the clinical hallmarks of ARDS which include radiographic abnormalities [1]. Bilateral opacities are required in the American European Consensus Conference (AECC) and Berlin definitions [2, 3]. The Pediatric Acute Lung Injury Consensus Conference (PALICC) definition for pediatric ARDS (PARDS) eliminated the requirement for bilateral radiographic findings, although evidence of new infiltrate(s) consistent with acute pulmonary parenchymal disease [4, 5] is still required. Rationale for this change included: high inter-observer variability in the interpretation of bilateral infiltrates on chest X-ray (CXR) [6-10], low sensitivity of CXR for alveolar consolidation that often lags behind the degree of hypoxemia evidence that diffuse alveolar damage may neither be homogeneous nor bilateral [11], lack of evidence that bilateral infiltrates contribute to risk stratification in ARDS [12].

We sought to prospectively evaluate components of the rationale to eliminate bilateral infiltrates from the PALICC definition, through the Pediatric Acute Respiratory Distress Syndrome Incidence and Epidemiology (PARDIE) study, which was conducted in 145 pediatric intensive care units (PICUs) in 27 countries [13]. Our primary hypothesis was that bilateral infiltrates are not independently associated with mortality after controlling for oxygenation impairment. Secondary hypotheses included that bilateral infiltrates and quadrants of alveolar consolidation have poor inter-rater reliability between ICU physicians and radiologists, and that comorbidities amongst other factors influence this variability.

Methods

Study design and participants

The present research comprises a pre-specified ancillary study of PARDIE. PARDIE was a prospective, international observational study with the goal of understanding the implications of the new PALICC definition. Children's Hospital Los Angeles (Los Angeles, CA, USA) was the coordinating center, and sites obtained ethical approval from their local institutional review board (IRB) or used a central IRB. Enrollment occurred over 10 distinct weeks during 2016–2017. Detailed PARDIE methods have been previously published [13], and the PARDIE protocol is provided in Appendix in ESM.

Patients were eligible if they met PALICC criteria [4]: (1) hypoxemia within 7 days of a clinical insult; (2) respiratory failure not fully explained by cardiac failure or fluid overload; (3) chest imaging with new infiltrate/s; (4) hypoxemia severity: for noninvasively ventilated subjects, PaO2/FiO2 (PF) ≤ 300 with full-face/oronasal mask with continuous positive airway pressure (CPAP) ≥ 5 cm H2O, while for intubated subjects, oxygenation index (OI) ≥ 4. Noninvasive equivalents SpO2/FiO2 (SF) and oxygenation saturation index (OSI) were allowed for subjects without PaO2 [14-16]. Patients had to be diagnosed within 24 h of enrollment [13].

Subjects were excluded if they were perioperative from a cardiac intervention, had active perinatal lung disease, or met PARDS criteria > 24 h before screening. Site investigators were responsible for assuring completeness and validity of data (Appendix in ESM).

Data collection

Sites participating in this ancillary study collected additional data related to chest imaging obtained for clinical purposes over the first 3 days of PARDS diagnosis. CXRs were independently interpreted by multiple providers without receiving prior educational training, ideally one pediatric intensivist and one radiologist who recorded their interpretations on specific rating forms (Appendix in ESM). If additional chest imaging (computed tomography (CT) or ultrasound) was available, a radiologist completed the rating form for the additional imaging.

Intensivists had to be either an attending physician or a fellow in pediatric critical care, and radiologists were either a general or pediatric radiologist. Years of clinical experience post-fellowship were recorded for all providers. Elements assessed included infiltrates (uni/bilateral) and anatomical quadrants of consolidation (right upper lobe, right middle/lower lobe, left upper lobe, left lower lobe), presence of pleural effusion, pneumothorax, subcutaneous emphysema, pneumomediastinum, pneumoperitoneum, and cardiomegaly. Two yes/no questions were added: "Do you believe the areas of opacification are predominately due to atelectasis?" and "Do you believe the CXR was consistent with congestive heart failure?" Participants were blinded to other clinical information.

These radiographic data were combined with information from PARDIE study which included demographics, diagnostic information, hypoxemia severity, ventilator settings, severity of illness, and outcome [13]. The protocol, rating form, and data management are in Appendix in ESM.

Analysis plan

The main PARDIE dataset was used to test the primary hypothesis that bilateral infiltrates are not independently associated with mortality after controlling for oxygenation impairment, using CXR interpretations from the site intensive care practitioner. Data are presented using categorical “buckets” for oxygenation impairment based on PALICC severity: NIV, mild, moderate, and severe. These criteria use PF/SF ratio for noninvasive ventilation (NIV) and OI/OSI for invasive mechanical ventilation (IMV). Given that NIV patients may have varying degrees of oxygenation impairment, we further analyzed all patients together using PF/SF ratio. SF ratio was converted into PF equivalents using previously published formulae [14]. However, PF ratio was found to have a nonlinear relationship with mortality. As such, PF ratio was categorized into the ranges used in the Berlin definition, for stratified analysis. Logistic regression models were then built within the substrata with PF ≤ 100, to satisfy assumptions of regression. Similar analysis was done for the secondary variable of four quadrants of alveolar consolidation. Sample size was not a priori specified because it was based on recruitment from the parent PARDIE study, which enrolled 708 patients with a mortality rate of 17%. Post hoc power analysis demonstrated we could detect an 8.5% mortality difference with unilateral versus bilateral infiltrates for all PARDS patients, or a 16% difference in those with PF ≤ 100, with an alpha of 0.05 and a power of 0.8 [13].

A specific V.3 dataset was used to test the secondary hypotheses that bilateral infiltrates and quadrants of consolidation have poor inter-rater reliability between ICU physicians and radiologists, and that comorbidities amongst other factors influence this variability. For inter-observer variability, all films had simultaneous interpretations by an intensivist and radiologist. If multiple practitioners (intensivist, radiologist) provided interpretations, we selected the most experienced in each field prioritizing pediatric over general radiologists. Inter-observer variability was calculated using Cohen’s k coefficient. Despite some limitations [9], kappa appropriately accounts for chance-corrected agreement between readers. Values < 0 were interpreted as poor; 0–0.2, slight; 0.21–0.4, fair; 0.41–0.6, moderate; 0.6–0.8, substantial; and 0.8–1 almost perfect agreement [17, 18]. To evaluate factors associated with disagreement in the presence or absence of bilateral infiltrates, demographics and other variables were compared. To aid clinical interpretation and calculation of kappa, continuous variables were categorized using a locally weighted smoothing scatter plot which explored the relationship between the disagreement on bilateral infiltrates and the variable of interest. If a dose response or cutoff point was discerned from the plot, continuous data were correspondingly categorized. Disagreement was then analyzed with kappa statistics. This is necessary because the rates of bilateral infiltrates in ARDS patients are correlated with other variables (i.e., oxygenation impairment). Raters are more likely to agree about bilateral infiltrates by chance alone in subgroups where the rates of bilateral infiltrates are higher. Kappa corrects for this chance-related agreement. Individual kappa values and 95% confidence intervals are presented, but no specific inference testing was performed.

Finally, other images (CT and ultrasound) were matched to a CXR from the same day. When multiple X-ray interpretations were available, the radiologist’s interpretation was preferentially taken, followed by the most experienced intensivist. Categorical variables were compared with Chi-squared or Fischer exact tests. All tests were two-sided with p values less than 0.05 deemed significant. Analyses were performed using SAS (version 9·4; SAS Institute, Cary, NC) and STATA version 15 and verified by two statisticians.

Results

Association between CXR findings and ICU mortality

All 708 patients enrolled in the PARDIE study had a CXR available at PARDS diagnosis, and 17.1% of patients died [13]. Bilateral infiltrates were present on the initial CXR in 523 patients compared with 185 patients with unilateral infiltrates (19.5% vs 10.3% mortality, p = 0.004). Patients were stratified by the initial degree of oxygenation impairment as measured by PALICC grouping, and bilateral infiltrates on initial CXR were associated with higher mortality for those with NIV PARDS (19.1% vs 5.9%, p = 0.03), with a nonsignificant trend for higher mortality in those with severe PARDS based on OI/OSI criteria ( 34% vs 16%, p = 0.07, Table 1a).

Table 1

Mortality associated with the degree of hypoxemia as measured by initial PALICC grouping (NIV, mild, moderate, severe) and/or: (a) presence or absence of bilateral infiltrates; (b) quadrants of alveolar consolidation at the time of PARDS diagnosis

	NIV	Mild	Moderate	Severe
(a) Bilateral infiltrates
p value	0.03	0.45	0.5	0.07
Bilateral	21/110 (19.1%)	21/152 (13.8%)	10/114 (8.8%)	50/147 (34%)
Unilateral	3/51 (5.9%)	7/69 (10.1%)	5/40 (12.5%)	4/25 (16%)
(b) Quadrants
p value	0.004	0.9	0.08	0.03
1	0/38 (0%)	6/51 (11.8%)	2/26 (7.7%)	6/15 (40%)
2	10/57 (17.6%)	8/72 (11.1%)	7/51 (13.7%)	11/47 (23.4%)
3	4/30 (13.3%)	6/42 (14.3%)	5/41 (12.2%)	7/39 (18%)
4	9/33 (27.2%)	8/53 (15.1%)	0/33 (0%)	29/66 (43.9%)
Unknown	1/3 (33.3%)	0/3 (0%)	1/3 (33.3%)	1/5 (20%)

Data presented as number of patients who died/total number of patients (mortality%)

When combining patients on IMV and NIV, mortality and the rate of bilateral infiltrates increased as initial PF ratio decreased, particularly when PF was ≤ 100 (Supplemental Table 1a). There was no association between bilateral infiltrates on the initial film and mortality (OR 1.3 (95% CI 0.68, 2.5), p = 0.43) for patients with PF > 100, regardless of initial mode of ventilation. When the initial PF ≤ 100, bilateral infiltrates were associated with higher mortality for those on NIV (p = 0.0495) with a strong trend for those on IMV (p = 0.06, Fig. 1). Overall mortality was 27.8% for patients with PF ≤ 100 and bilateral infiltrates, compared to 9.8–14.4% for all other combinations of infiltrates and PF ratios (Fig. 2). Limiting the analysis to 277 patients with PF ≤ 100 (combining IMV and NIV), patients with bilateral infiltrates had a higher risk of mortality (OR 3.6 (1.4, 9.4, p = 0.01)), which was not modified by being on NIV (p = 0.75) or the actual PF ratio (p = 0.87).

Fig. 1

Relationship between ICU mortality and chest X-ray (CXR) infiltrates stratified by initial PF ratio (or SF ratio equivalent) and ventilation type (invasive or noninvasive mechanical ventilation) at PARDS diagnosis. 708 subjects from the whole PARDIE cohort had a CXR available (185 unilateral/523 bilateral infiltrates) at PARDS diagnosis. Bars represent number of cases per initial PF ratio and ventilation type. Whiskers represent 1 standard error around the point estimate for mortality (squares). NIV, noninvasive ventilation; IMV, invasive mechanical ventilation; PF: ratio of partial pressure of oxygen (PaO2) to fractional concentration of oxygen (FiO2), PO2/FiO2 ratio

Fig. 2

Subjects and ICU mortality stratified by initial PF ratio at PARDS diagnosis combined with chest X-ray infiltrates (uni/bilateral). 708 subjects from the whole PARDIE cohort had a CXR available (185 unilateral/523 bilateral infiltrates) at PARDS diagnosis. Bars represent number of cases per initial PF ratio. Whiskers represent ± 1 standard error around the point estimate for mortality (squares). Mortality was substantially higher for patients with PF ≤ 100 and bilateral infiltrates, compared to all other combinations of infiltrates and PF ratios. This includes patients on either noninvasive ventilation or invasive mechanical ventilation. PF: ratio of partial pressure of oxygen (PaO2) to fractional concentration of oxygen (FiO2), PO2/FiO2 ratio

Within the first 3 days of PARDS diagnosis, 90 of the 185 patients who did not initially have bilateral infiltrates went on to develop them. Mortality for the 95 patients who did not have bilateral infiltrates was 8.4% versus 18.4% in the 613 patients who did have bilateral infiltrates within the first 3 days of PARDS (p = 0.016). Similar to the initial CXR, there was no association between bilateral infiltrates and mortality amongst patients who always had a PF > 100 within the first 3 days of PARDS diagnosis (7.8% vs 6.3%, p = 0.8), with a nonsignificant trend for higher mortality (25.4% vs 12.9%, p = 0.12) for those with bilateral infiltrates and at least one PF ≤ 100 (Supplemental Fig. 1).

There was an association between the number of quadrants of alveolar consolidation on the initial CXR and mortality. When stratifying by the degree of hypoxemia as measured by initial PALICC grouping, a higher number of quadrants (particularly four) was associated with higher mortality for those with NIV (p = 0.004, Table 1b). There was no clear association with quadrants of consolidation and mortality for those with mild/moderate PARDS. For severe PARDS, either one or four quadrants had close to 40% mortality, while 2 or 3 quadrants had closer to 20% (Table 1b). When combining patients on IMV and NIV, there was no association between four quadrants of consolidation and mortality amongst patients with an initial PF > 100 (16.6% vs 11%, (OR 1.6 (0.85, 3), p = 0.14)), but for those with initial PF ≤ 100, four quadrants of consolidation was associated with higher mortality (33.7% vs 20.2%, OR 2 (1.14, 3.5, p = 0.02)), which was not modified by being on NIV (p = 0.81) or the actual PF ratio (p = 0.83) (Supplemental Fig. 2). When looking at specific quadrants, consolidation of the right upper lobe (RUL) was not associated with mortality, while consolidation in any other quadrant was associated with higher mortality (Supplemental Table 2a). However, there was no clear discernable pattern between specific combinations of quadrants and mortality (Supplemental Table 2b) for those with fewer than four quadrants.

Inter-observer variability

Of the 708 patients enrolled in the PARDIE study, 379 contributed data to the ancillary V.3 study with 1151 CXRs. There were simultaneous interpretations of 702 CXRs from 233 patients (Flowchart, ESM). Demographics, hypoxemia severity, and outcomes are provided in Table 2. A total of 225 films were at PARDS onset, while 173, 159, and 145 on days 1 to 3, respectively. Attending pediatric intensivists performed 586 (83.5%) of critical care interpretations, with a median of 6 (3, 23) years of experience post-fellowship, with the remaining 116 (16.5%) interpreted by fellows. Pediatric radiologists interpreted 691 (98.4%) of radiologist interpretations, with a median of 8 (5, 13) years of experience post-fellowship.

Table 2

Description of the PARDIE V3 cohort

Variable	V3 cohort (n = 233)	V0 cohort not in V3 (n = 475)	Full V0 cohort (n = 708)
Demographicsa (n = 232)
Age (years)	4.3 (0.76–10.9)	3.0 (0.73–9.5)	3.3 (0.73–9.6)
Female (%)	100 (43.1)	174 (36.7)	274 (38.8)
Nonwhite race (%)	100 (43.1)	181 (38.6)	281 (40.1)
Not Hispanic ethnicity (%)	160 (69)	259 (55.1)	419 (59.7)
Season
Spring (%)	64 (27.5)	137 (28.8)	201 (28.4)
Summer (%)	36 (15.4)	52 (11)	88 (12.4)
Fall (%)	50 (21.5)	94 (19.8)	144 (20.3)
Winter (%)	83 (35.6)	192 (40.4)	275 (38.8)
Region
North America (%)	173 (74.2)	275 (57.9)	448 (63.3)
Europe (%)	45 (19.3)	74 (15.6)	119 (16.8)
Central/South America (%)	11 (4.7)	90 (19)	101 (14.3)
Middle East, Asia, Australia and New Zealand (%)	4 (1.7)	36 (7.6)	40 (5.7)
Country income
High income (%)	230 (98.7)	375 (79)	605 (85.5)
Middle income (%)	3 (1.3)	100 (21.1)	103 (14.6)
Comorbidities (%)
Any comorbidity	147 (63.1)	298 (62.7)	445 (62.9)
LV dysfunction	13 (5.6)	27 (5.7)	40 (5.7)
Prematurity	47 (20.2)	84 (17.7)	131 (18.5)
Chronic pulmonary disease	82 (35.1)	115 (24.2)	197 (27.8)
Chronic respiratory support	41 (17.6)	79 (16.6)	120 (17)
At home ventilation	9 (3.9)	19 (4)	28 (4)
Congenital heart disease	31 (13.3)	47 (9.9)	78 (11)
Acquired heart disease	23 (9.9)	33 (7)	56 (7.9)
Neuromuscular disease	32 (13.7)	90 (19)	122 (17.2)
Cancer	17 (7.3)	42 (8.8)	59 (8.3)
Immunosuppression	37 (15.9)	58 (12.2)	95 (13.4)
PARDS risk factor (%)
Pneumonia	144 (61.8)	301 (63.4)	445 (62.9)
Non-pulmonary sepsis	49 (21)	87 (18.3)	136 (19.2)
Other	40 (17.2)	87 (18.3)	127 (17.9)
PALICC categories (%)
NIV	50 (21.5)	111 (23.4)	161 (22.7)
Mild	80 (34.3)	141 (29.7)	221 (31.2)
Moderate	48 (20.6)	106 (22.3)	172 (24.3)
Severe	55 (23.6)	117 (24.6)	172 (24.3)
Severity of illness, probability of death
PIM 3	2.9% (0.5–5.9)	3.3% (0.7–7.9)	3.2% (0.6–6.7)
PRISM IV	3.4% (1.5–8.9)	3.6% (1.6–11.6)	3.5% (1.6–10.3)
Outcomesa (n = 232)
VFD, invasive ventilation	20.8 (11.3–24.2)	20.0 (0.3–24.4)	20.4 (5.5–24.4)
VFD, invasive and NIV	19.1 (6.4–23.5)	18.7 (0–23.4)	18.9 (0–23.4)
Length of invasive MV	6.2 (3.3–12.1)	5.5 (2.6–10.7)	5.9 (2.7–11.1)
Length of invasive MV and NIV	7.4 (4.1–14.1)	6.9 (3.5–12.3)	7 (3.7–13)
Mortality outside ICU
Hospital mortality (%)	35 (15)	96 (20.3)	131 (18.5)
90-day mortality (%)	36 (15.5)	100 (21.1)	136 (19.3)

Comparison to the patients included in V0 but not the in V3 and to the full V0 cohort

LV dysfunction, left ventricular dysfunction; NIV, noninvasive ventilation; inMV, invasive mechanical ventilation; VFD, ventilator-free days. Data presented as median (IQR) or number (%)

aData to calculate LMV missing from one patient

Of the 702 films, radiologists and intensivists agreed on the presence or absence of bilateral infiltrates in 537 (76.5%) cases. Intensivists identified bilateral infiltrates in 81/702 (11.5%) cases when radiologists did not, and radiologists identified bilateral infiltrates in 84/702 (12%) cases when intensivists did not. Correcting for chance-related agreement, inter-observer agreement between interpretations of bilateral infiltrates and quadrants of consolidation was “slight” (kappa 0.31 and 0.33). If each quadrant is considered independently, the greatest agreement is reached in RUL (kappa, k = 0.59, moderate agreement), with the least agreement in the left lower lobe (k = 0.33) (Supplemental Fig. 3). There was relatively more agreement on the presence of pleural effusion (k = 0.5), pneumothorax (k = 0.7), pneumomediastinum (k = 0.75), and subcutaneous emphysema (k = 0.77), but poor agreement on cardiomegaly (k = 0.19) and atelectasis (k = 0.16) (Table 3). Radiologists diagnosed atelectasis in 36.3% of CXRs compared with 20% by intensivists.

Table 3

Inter-observer agreement between pediatric intensivist and radiologist interpretations of chest X-ray findings

	ICU physician	Radiologist	Kappa (95% CI)
RUL [n (%)]	462 (65.8)	427 (60.8)	0.59 (0.53, 0.65)
RML/RLL [n (%)]	570 (81.2)	536 (76.4)	0.45 (0.37, 0.53)
LUL [n (%)]	404 (57.6)	362 (51.6)	0.43 (0.36, 0.49)
LLL [n (%)]	557 (79.3)	592 (84.3)	0.33 (0.24, 0.41)
Bilateral infiltrates [n (%)]	549 (78.2)	552 (78.6)	0.31 (0.22, 0.39)
No infiltrates [n (%)]	9 (1.3)	22 (3.1)	0.18 (− 0.01, 0.37)
Pleural effusion [n (%)]	247 (35.3)	276 (39.4)	0.5 (0.44, 0.57)
Quadrants of consolidation [n (%)]			0.33 (0.28, 0.38)
0/unknown	9 (1.3)	22 (3.1)
1	88 (12.5)	82 (11.7)
2	174 (24.8)	190 (27.1)
3	167 (23.8)	177 (25.2)
4	264 (37.6)	231 (32.9)
Pneumothorax [n (%)]	18 (2.6)	27 (3.9)	0.7 (0.55, 0.86)
Subcutaneous emphysema [n (%)]	28 (4)	33 (4.7)	0.77 (0.66, 0.89)
Pneumomediastinum [n (%)]	16 (2.3)	21 (3)	0.75 (0.59, 0.91)
Cardiomegaly [n (%)]	48 (6.9)	48 (6.9)	0.19 (0.08, 0.31)
Atelectasis [n (%)]	140 (20)	254 (36.3)	0.16 (0.09, 0.23)

Total N = 702 films unless otherwise specified

RUL right upper lobe, RML middle lobe, RLL right left lobe, LUL left upper lobe, LLL left lower lobe

Kappa (k): agreement beyond chance was classified as follows: poor agreement, k < 0.00; slight, k < 0.20; fair, k = 0.21–0.4; moderate, k = 0.41–0.6; substantial, k = 0.61–0.8; almost perfect agreement, k > 0.8 [14, 15]

Factors associated with disagreement

Factors associated with the overall rate of bilateral infiltrates include comorbidities, PARDS severity, ventilator settings, age, and fluid balance. As an example, bilateral infiltrates were present in approximately 72% of CXRs when FiO2 levels were between 0.21 and 0.4, compared with 87% with FiO2 0.8–1. The percent agreement for bilateral infiltrates between radiologists and intensivists is 70.5% for those with FiO2 0.21–0.4, but 85.6% for those with FiO2 0.8–1. While the overall agreement is higher when FiO2 is higher, after correcting for the chance-expected agreement, the kappa values were similar (0.26 vs 0.36). Hence, FiO2 level does not appear to influence inter-observer reliability. Similar findings were seen for other variables (Table 4), and kappa values generally remained in the 0.2–0.4 range. Some notable exceptions included higher agreement for chronic respiratory support (k = 0.62), NIV PARDS (k = 0.51), age > 10 years (k = 0.43), and net fluid balance > 40 ml/kg on the day before the CXR (k = 0.48) (Table 4). Lower agreement was noted for acquired heart disease (k = 0.1) and those with a mean airway pressure between 12 and 15 cm H2O (k = 0.16).

Table 4

Level of agreement to judging bilateral infiltrates as present or absent

	ICU doctor	Radiologist	% Agreement	Kappa (95% CI)
Total CXR bilateral [N (%)]	549/702 (78.2)	552/702 (78.6)	76.5	0.31 (0.22, 0.39)
Total PARDS severity [N (%)]
Missing	31/42(73.8)	29/42 (69.1)	66.7	0.19 (− 0.13, 0.5)
Non-PARDS	106/143 (74.1)	107/143 (74.8)	71.3	0.25 (0.07, 0.42)
NIV	20/35 (57.1)	20/35 (57.1)	77.1	0.53 (0.25, 0.82)
Mild	181/238 (76.1)	190/238 (79.8)	73.5	0.23 (0.09, 0.37)
Moderate	126/149 (84.6)	122/149(81.9)	82.6	0.38 (0.18, 0.57)
Severe	85/95 (89.5)	84/95 (88.4)	86.3	0.31 (0.02, 0.59)
	p < 0.001	p = 0.0024
Age (years)
< 1	152/218 (69.7)	176/218 (80.7)	73.4	0.30 (0.16, 0.43)
1–5	131/165 (79.4)	134/165 (81.2)	75.2	0.22 (0.04, 0.39)
5–10	100/123 (81.3)	95/123 (77.2)	76.4	0.29 (0.09, 0.48)
> 10	162/192 (84.4)	144/192 (75)	81.3	0.43 (0.28, 0.58)
Ventilator settings
Mean airway pressure (cmH20)
Missing	64/89 (71.9)	59/89 (66.3)	74.2	0.4 (0.19, 0.6)
<12	159/221 (71.9)	166/221 (75.1)	71.5	0.27 (0.13, 0.41)
12–15	137/175 (78.3)	148/175 (84.6)	74.3	0.16 (− 0.01, 0.32)
15–20	129/153 (84.3)	121/153 (79.1)	81.7	0.39 (0.21, 0.57)
>20	60/64 (93.7)	58/64 (90.6)	90.6	0.35 (− 0.05, 0.75)
FiO2
0.21–0.39	141/193 (73.1)	138/193 (71.5)	70.5	0.26 (0.12, 0.41)
0.4–0.59	211/277 (76.2)	222/277 (80.1)	75.8	0.29 (0.16, 0.42)
0.6–0.79	110/135 (81.5)	113/135 (83.7)	81.5	0.36 (0.16, 0.56)
0.8–1	82/90 (91.1)	75/90 (83.3)	85.6	0.36 (0.09, 0.63)
Fluid balance (n = 459)
Negative	102/143 (71.3)	110/143 (76.9)	73.4	0.31 (0.15, 0.49)
0–40 ml/kg/d	164/201 (81.6)	152/201 (75.6)	75.1	0.26 (0.11, 0.41)
> 40 ml/kg/d	89/115 (77.4)	96/115 (83.5)	83.5	0.48 (0.28, 0.68)
Comorbidities
LV dysfunction	33/39 (84.6)	33/39 (84.6)	69.2	− 0.18 (− 0.28, − 0.08)
CPD	193/244 (79.1)	187/244 (76.6)	79.5	0.41 (0.27, 0.54)
Congenital heart disease	82/95 (86.3)	78/95 (82.1)	83.2	0.37 (0.12, 0.62)
Acquired heart disease	60/73 (82.2)	57/73 (78.1)	71.2	0.1 (− 0.15, 0.35)
Neuromuscular disease	72/82 (87.8)	61/82 (74.4)	79.3	0.34 (0.11, 0.58)
Cancer	50/59 (84.7)	41/59 (69.5)	74.6	0.3 (0.05, 0.56)
Immune-suppression	106/118 (89.8)	93/118 (78.8)	80.5	0.28 (0.07, 0.49)
Chronic resp. support	89/117 (76.1)	91/117 (77.8)	86.3	0.62 (0.44, 0.79)
Pneumonia	348/431 (80.7)	340/431 (78.9)	76.8	0.28 (0.17, 0.39)
Sepsis	118/154 (76.6)	122/154 (79.2)	76.6	0.32 (0.15, 0.5)
Mechanical ventilation (n = 676)
Invasive	481/606 (79.4)	489/606 (80.7)	76.9	0.28 (0.19, 0.37)
Noninvasive	52/70 (74.3)	48/70 (68.6)	80	0.51 (0.29, 0.73)

Other imaging

There were 32 CTs from 21 subjects and 20 ultrasound images from 15 subjects in the dataset. 31/32 Chest CTs and 15/20 ultrasounds could be matched to a CXR on the same day. Comparisons are shown in Supplemental Table 3. Kappa values for bilateral infiltrates between CT and CXR, and ultrasound and CXR were both 0.56. There were high levels of agreement between ultrasound and CXR for pleural effusion (k = 0.76), although this was lower with CT and CXR (k = 0.18).

Discussion

We have found that radiographic findings related to bilateral infiltrates or quadrants of alveolar consolidation are associated with mortality for PARDS patients with a PF ≤ 100 either on invasive or noninvasive ventilation. There was no clear relationship between CXR and mortality for patients with less severe oxygenation impairment. Mortality for those with PF ≤ 100 and bilateral infiltrates was 28%, compared to 10–15% for all other patients in the cohort. While nearly 87% of patients who meet PALICC criteria were deemed to have bilateral infiltrates within 3 days of PARDS diagnosis, the agreement on which patients these are between intensivists and radiologists is only marginally better than expected by chance alone (kappa 0.31–0.33). Similar findings were seen for quadrants of consolidation. While this study was in pediatric ARDS, we would encourage focused investigation down these lines in adults with ARDS.

Prognostic relevance and inter-observer reliability of chest imaging are important considerations in creating diagnostic criteria for ARDS [2, 3, 19–22]. With regard to our hypothesis that radiographic criteria would not retain prognostic relevance after controlling for oxygenation impairment, we found that bilateral infiltrates or four quadrants of consolidation as gauged by an intensivist were associated with higher mortality for patients with more severe oxygenation impairment (PF ≤ 100). Interestingly, there was no relationship between CXR findings and mortality for those with less severe oxygenation impairment. Moreover, 50% of patients who did not have bilateral infiltrates at PARDS diagnosis went onto develop them within the first 3 days of PARDS. Consistent with the data from the initial film, bilateral infiltrates only appeared to be associated with higher mortality for those with PF ≤ 100, although not statistically significant, likely because only 13% of patients who met PALICC criteria did not have bilateral infiltrates within 3 days of PARDS. Oxygenation impairment and opacification on CXR are both surrogates for loss of end-expiratory lung volume and intrapulmonary shunt, important components of ARDS pathobiology. However, from our data, it appears as if bilateral infiltrates combined with severe oxygenation impairment constitute a PARDS patient who has much higher risk of mortality than any other type of PARDS patient, regardless of mode of ventilation.

Intriguingly, when looking at quadrants of consolidation amongst those with PF ≤ 100, patients with one or four quadrants of consolidation had similar mortality, which was considerably lower than those with two or three quadrants. This observation reinforces that current methods to gauge chest imaging have limitations related to prognostic relevance in ARDS. Perhaps this can be overcome with more explicit methods for density of opacification, as suggested by the RALE score [12]. Unfortunately, we did not gather the elements required to calculate this score in our study.

There is no gold standard to define ARDS in either children or adults. Some argue that histopathology should be the definitive gold standard, but this will skew evaluation of diagnostic criteria to those who die or are sicker. Clinical syndromes as ARDS are defined using a constellation of signs and symptoms which together make it highly likely that the patient has the disease, to apply appropriate treatment strategies or aid in prognosis. Therefore, diagnostic criteria should be objective and reproducible. With regard to our second hypothesis that radiographic criteria have poor inter-observer reliability, we found the kappa values for chance-corrected agreement between intensivists and radiologists on bilateral infiltrates or quadrants of consolidation were in the fair range. While some elements had higher agreement (RUL opacification, pneumothorax), these are not used in ARDS definitions. Our study represents real-world clinical practice, collecting data from adequately experienced radiologists and intensivists. Furthermore, there was no clear pattern that one specialty was more likely to interpret bilateral infiltrates or more quadrants of disease, as the rates were similar, confirming that interpretations of bilateral infiltrates and quadrants of consolidation are not reproducible amongst practitioners who routinely evaluate CXRs. While there was higher “agreement” for those with severe hypoxemia, when adjusting for the fact that these patients are more likely to have bilateral infiltrates, the chance-corrected agreement was only marginally better than chance alone.

These results are consistent with previous investigations, which have found kappa values between 0.27 and 0.3 on the interpretation of bilateral infiltrates amongst ARDS patients [7, 23]. Strategies focused on standardized training interventions or automated processing of digital images [9, 10, 23, 24] have been proposed, but it is debatable whether training improves reliability. Likewise, the agreement can be improved when readers are experts in ARDS, although this has clear limitations for clinical practice [6]. Studying the impact of such interventions is an important area for the future research, but it seems clear that current methods for the interpretation of bilateral infiltrates as part of diagnostic criteria for ARDS are not reproducible between intensivists and radiologists.

We also found that intensivists and radiologists are more likely to agree on the presence or absence of bilateral infiltrates when patients were on chronic respiratory support prior to PARDS, had NIV PARDS, age > 10 years and a positive fluid balance. In contrast, there were higher levels of disagreement for those with some component of cardiac disease. Retaining the bilateral infiltrate requirement in the definition may therefore inadvertently make the diagnosis more likely in certain subgroups, such as older children, and more difficult in others, such as those with cardiac disease.

Given limitations of chest X-rays, some propose using lung ultrasound or chest CT. Indeed, the PALICC definition affords using any chest imaging to diagnose pulmonary parenchymal disease [5]. However, we have confirmed their use is very uncommon. Therefore, despite some potential therapeutic and diagnostic advantages [25-28], lack of widespread use of these modalities is a major limitation.

Our study has limitations. First, this study was not focused on whether the pathobiology of ARDS mandates exclusively bilateral lung involvement. It is important to uncouple this fundamentally pathophysiologic question from our study which surrounds the prognostic relevance and inter-observer reliability of the interpretation of CXRs in ARDS. Our study is not meant to directly address whether ARDS definitions should distinguish “unilateral” from “bilateral” disease. It appears important to focus research on understanding whether the severe hypoxemia group with bilateral infiltrates represents a different phenotype of ARDS, given such a high risk of poor outcome. However, CXRs may be an inaccurate way to do this, and other chest imaging modalities are not used routinely to replace CXRs in ARDS. Second, not all sites participated in the ancillary study. Mortality analysis focused on interpretations for 708 PARDS patients, while the inter-observer analysis was conducted on a smaller subgroup. Nevertheless, simultaneous interpretation of 702 radiographs represents one of the largest studies of inter-rater reliability in either adults or pediatrics [6–10, 24], and the groups were similar (Table 2). Third, practitioners were blinded to clinical information. We cannot predict whether inter-observer agreement would be better if more clinical information was provided, although previous studies have not shown a substantial impact [29]. Fourth, very few CT or US interpretations were available, resulting in small sample size. Fifth, this study uses the term infiltrates based on the PALICC definition, which is increasingly being replaced with “opacifications” in the radiology literature as it is difficult to differentiate atelectasis from consolidation. Indeed, we identified a kappa of 0.16 between radiologists and intensivists regarding atelectasis. Future studies of PARDS should likely embrace “opacifications” instead of infiltrates. Sixth, sample sizes for subgroup analysis were small and likely inadequately powered for potentially clinically meaningful effects. Furthermore, we did not do inference testing to compare kappa values between groups of patients, so these analyses are only descriptive and should be interpreted with caution. Finally, there are many variables which will confound the relationship between bilateral infiltrates and mortality. We only evaluated oxygenation impairment in this manuscript but have previously shown that radiographic criteria do not retain an independent relationship with mortality after controlling for other factors such as immune-suppressed status and ARDS trigger [13, 30].

In conclusion, in this large international study focused on real-world practitioners evaluating chest imaging in ARDS, we confirmed that the pediatric intensivist’s interpretation of bilateral infiltrates or four quadrants of consolidation is associated with mortality only for patients with PF ratio (or equivalent) ≤ 100. This subgroup of patients is particularly high risk for mortality and may deserve special considerations in both clinical care and research. However, there is high inter-rater variability in the interpretation of bilateral infiltrates and quadrants of consolidation on CXRs. From our study, it is not possible to discern whether all patients with “unilateral” versus “bilateral” disease on CXR have the same pathobiology, which may increase the false positive rate of the PALICC definition. Balancing these issues of sensitivity and specificity should be a focus of the future research and definitions of ARDS.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 34077 kb)

Supplementary file2 (DOCX 137 kb)

Supplementary file3 (DOCX 20 kb)

Chest radiograph findings of bilateral infiltrates or four quadrants of consolidation in children with pediatric ARDS are associated with higher mortality only for children with severe oxygenation impairment. However, these radiographic criteria have poor inter-rater reliability.