Validation of Respiratory Rate-Oxygenation Index in Patients With COVID-19-Related Respiratory Failure.

OBJECTIVES: The respiratory rate-oxygenation (ROX) index is a fraction of oxygen saturation, Fio2, and respiratory rate that has been validated to predict receipt of invasive mechanical ventilation in patients receiving high-flow nasal cannula (HFNC). This study aimed to validate ROX in a cohort of inpatients with COVID-19-related respiratory failure.
DESIGN: Retrospective validation of the ROX index. We calculated sensitivity, specificity, positive predictive value, negative predictive value, and 95% CIs of ROX for invasive mechanical ventilation any time during hospitalization.
SETTING: Twenty-one hospitals of Kaiser Permanente Northern California, an integrated healthcare delivery system. PATIENTS: We identified adults with positive severe acute respiratory syndrome coronavirus 2 polymerase chain reaction test within 3 weeks of, or during, hospitalization between February 1, 2020, and December 31, 2020. We calculated ROX at 12 hours after HFNC initiation. We grouped patients as low (≥ 4.88), intermediate (< 4.88 and ≥ 3.85), or high (< 3.85) risk using previously published thresholds.
INTERVENTIONS: None.
MEASUREMENTS AND MAIN RESULTS: We identified 1,847 patients who had no limitation of life support. Of these, 525 (31.7%) received invasive mechanical ventilation any time during hospitalization and 511 died (27.7%). The sensitivity, specificity, positive predictive value, and negative predictive value of 12-hour ROX threshold (< 3.85) predicting invasive mechanical ventilation were 32.3% (95% CI, 28.5-36.3%), 89.8% (95% CI, 88.0-91.4%), 59.4% (95% CI, 53.8-64.9%), and 74.1% (95% CI, 71.8-76.3%), respectively.
CONCLUSIONS: The 12-hour ROX index has a positive predictive value (59.4%) using threshold of less than 3.85 for COVID-19 patients needing invasive mechanical ventilation. Our health system has embedded ROX into the electronic health record to prioritize rounding during periods of inpatient surge.

Prior studies show that patients with acute hypoxemic respiratory failure can benefit from oxygen delivered via high-flow nasal cannula (HFNC) (1). Accordingly, HFNC has been commonly used to treat patients with severe COVID-19 pneumonia (2). Tools to predict which patients with COVID-19 requiring HFNC will require invasive mechanical ventilation are needed to prevent delays in care and allocate resources appropriately. The respiratory rate-oxygenation (ROX) (3, 4) index is a fraction of oxygen saturation divided by Fio2 as the numerator and respiratory rate as the denominator. ROX index has been validated to predict receipt of invasive mechanical ventilation in patients receiving HFNC for any cause before COVID-19 existed (3). A previous prospective validation study showed that ROX index had better performance with increasing time from initiation of HFNC (2-, 6-, and 12-hr) and could be used to identify both low- and high-risk patients (3). Data evaluating performance of ROX in COVID-19 is limited. We aimed to validate ROX in a large cohort of COVID-19 patients receiving HFNC.

METHODS

We identified adults (≥ 18 yr) with a positive severe acute respiratory syndrome coronavirus 2 polymerase chain reaction test within 3 weeks prior to, or during, a hospitalization between February 1, 2020, and December 31, 2020, across 21 hospitals of Kaiser Permanente Northern California, an integrated healthcare delivery system. We narrowed to patients receiving HFNC oxygen. If patients required invasive mechanical ventilation, they had to have greater than or equal to 1 HFNC recording before intubation. We excluded patients with limitation of life support based on code status order before HFNC initiation. We decided not to exclude patients who received noninvasive ventilation before or after HFNC initiation because the goal was to determine accuracy of ROX regardless of whether noninvasive ventilation was delivered.

For each patient, we calculated ROX index, choosing the value closest to, but preceding, the 12-hour timestamp after HFNC initiation (3). We chose the 12-hour timepoint because 1) ROX is more accurate over time (3) and 2) death was shown to increase after 12 hours (3). We grouped patients as low (≥ 4.88), intermediate (< 4.88 and ≥ 3.85), or high (< 3.85) risk according to previous studies (3). We calculated sensitivity, specificity, positive predictive value, negative predictive value, and 95% CIs of ROX for invasive mechanical ventilation any time during hospitalization using two risk thresholds (< 4.88 and < 3.85). We evaluated the percent of patients by risk group who had 1) invasive mechanical ventilation within 24 hours of HFNC initiation, 2) invasive mechanical ventilation anytime during hospitalization, or 3) inpatient death.

To confirm the relationship between ROX risk group and risk of subsequent clinical outcome, we developed a Cox model. Exposure was ROX risk group and outcome was invasive mechanical ventilation or death. We censored patients if discharged alive. We report unadjusted/adjusted hazard ratios (95% CI) with low-risk group as reference. Covariates included age, sex, Comorbidity Point Score, version 2 (5), vitals (heart rate, blood pressure), severity of illness score (Epic Deterioration Index), and laboratory values (blood gas, bicarbonate, troponin, d-dimer). Epic Deterioration Index (range, 0–100) is a validated severity of illness score developed by Epic that is based on clinical data including vital signs, laboratory values, and nursing assessments.

The Kaiser Permanente Northern California Institutional Review Board approved the project (1262727-51), waiving the need for informed consent. We used STATA/SE 14.2 (StataCorp LLC, College Station, TX).

RESULTS

We identified 2,039 patients who received HFNC, including 1,847 who had no limitation of life support. Of these, 657 (35.6%) received subsequent noninvasive ventilation, 525 (31.7%) received invasive mechanical ventilation any time during hospitalization, and 511 died (27.7%). The median time from admission to HFNC initiation was 20.8 hours (interquartile range [IQR], 4.1–56.1 hr). Based on the 12-hour ROX value, 1,176 patients (63.7%) were low risk, 353 (19.1%) were intermediate, and 318 (17.2%) were high (Table 1). Among those receiving invasive mechanical ventilation, median time from HFNC initiation to invasive mechanical ventilation was 83.5 hours (IQR, 27.8–178.9 hr).

TABLE 1.

Baseline Characteristics of Patients Hospitalized for COVID-19 Who Required High-Flow Oxygen and Did Not Have Limitation of Code Status

Variable	Respiratory Rate-Oxygenation Index at 12 hr After Initiation of High-Flow Nasal Cannula
	Low Risk ≥ 4.88, n = 1,176	Intermediate Risk ≥ 3.85 to < 4.88, n = 353	High Risk < 3.85, n = 318
Age	61.6 (15.7)	62.2 (14.2)	60.0 (14.2)
Male	716 (60.9%)	212 (60.1%)	222 (69.8%)
Race
Hispanic	489 (41.6%)	150 (42.5%)	146 (45.9%)
White	338 (26.3%)	104 (26.4%)	85 (23.6%)
Asian	239 (20.3%)	75 (21.3%)	59 (18.5%)
Black	114 (9.7%)	28 (7.9%)	34 (10.7%)
Comorbidity Point Score, version 2[a]	29.5 (35.7)	27.8 (34.2)	23.0 (28.6)
Comorbidities
Diabetes with and without complications	441 (37.5%)	131 (37.1%)	111 (34.9%)
Peripheral vascular disease	396 (33.7%)	125 (35.4%)	94 (29.6%)
Chronic pulmonary disease	306 (26.0%)	80 (22.7%)	64 (20.1%)
Chronic renal insufficiency	236 (20.1%)	59 (16.7%)	43 (13.5%)
Congestive heart disease	106 (9.0%)	31 (8.8%)	19 (6.0%)
Cerebrovascular disease	92 (7.8%)	29 (8.2%)	20 (6.3%)
Malignancy	58 (4.9%)	24 (6.8%)	23 (7.2%)
Dementia	57 (4.9%)	14 (4.0%)	13 (4.1%)
Rheumatologic disease	26 (2.2%)	8 (2.3%)	< 5
Metastatic disease	13 (1.1%)	5 (1.4%)	< 5
Mild liver disease	17 (1.5%)	< 5	< 5
Severe liver disease	11 (0.9%)	< 5	< 5
Peptic ulcer disease	8 (0.7%)	< 5	< 5
Hemiplegia	8 (0.7%)	< 5	< 5
Acquired immunodeficiency disorder	6 (0.5%)	< 5	< 5
Average Epic Deterioration Index[b] on day of initiation of high-flow oxygen	40.6 (9.9)	42.6 (10.1)	48.2 (10.2)
Average oxygen saturation/Fio2 on day of initiation of high-flow oxygen	163 (91)	216 (84)	119 (78)
Average respiratory rate on day of initiation of high-flow oxygen	23 (4)	25 (4)	29 (4)
Admit from home	1,107 (94.1%)	342 (96.9%)	304 (95.6%)

Comorbidity Point Score, version 2 is an externally validated comorbidity index developed by Kaiser Permanente Northern California. It is calculated using all diagnoses incurred by a patient in 12 mo prior to hospitalization (range, 0–1,014). A score of 0–39 correlates to 1-yr mortality of 0.3% and 40–64 correlates to 5.3%.

Epic Deterioration Index is a widely used, validated severity of illness score developed outside of Kaiser Permanente Northern California that is embedded into the electronic health record. It is calculated based on clinical data including vital signs, laboratory values, and nursing assessments (range, 0–100).

Continuous variables are displayed with mean (sd). Categorical variables are displayed with n (%).

The sensitivity, specificity, positive predictive value, and negative predictive value of 12-hour ROX threshold of less than 4.88 predicting invasive mechanical ventilation were 58.1% (95% CI, 54.0–62.2%), 73.8% (95% CI, 71.3–76.2%), 50.7% (95% CI, 46.8–54.5%), and 79.2% (95% CI, 76.7–81.5%), respectively. The corresponding values using 12-hour ROX threshold of less than 3.85 were 32.3% (95% CI, 28.5–36.3%), 89.8% (95% CI, 88.0–91.4%), 59.4% (95% CI, 53.8–64.9%), 74.1% (95% CI, 71.8–76.3%), respectively.

Figure 1 displays percent of patients by risk group who had 1) invasive mechanical ventilation within 24 hours of HFNC initiation, 2) invasive mechanical ventilation anytime during hospitalization, and 3) death. Adverse outcomes increased with ROX risk group. Of the 1,176 low-risk patients, 245 (20.8%) were intubated during the hospitalization; of the 353 intermediate-risk patients, 151 (42.8%) were intubated during the hospitalization; and of the 318 high-risk patients, 189 (59.4%) were intubated during the hospitalization. Similarly, death occurred in 235 (20.0%) of low, 126 (35.7%) of intermediate, and 150 (47.2%) of high risk.

Figure 1.

This figure shows the percent of patients with COVID-19-related respiratory failure by respiratory rate-oxygenation (ROX) risk group at 12 hours who experience various outcomes. The bars indicate the percent of patients who: 1) received mechanical ventilation within 24 hours of high flow cannular initiation, 2) received mechanical ventilation anytime during hospitalization, and 3) died in the hospital.

In survival analysis, the unadjusted hazard ratio was 2.1 (95% CI, 1.7–2.6) for intermediate risk and 2.9 (95% CI, 2.3–3.5) for high risk, both compared with low risk. The adjusted hazard ratio was 2.2 (95% CI, 1.7–2.7) for intermediate risk and 3.0 (95% CI, 2.4–3.7) for high risk, both compared with low risk.

DISCUSSION

We found that 12-hour ROX index has a positive predictive value (59.4% for < 3.85 threshold) for COVID-19 patients needing invasive mechanical ventilation and could risk stratify patients for both early invasive mechanical ventilation and death. This positive predictive value is lower than the original ROX validation cohort (81%) (3), which could be due to viral pneumonia being an infrequent diagnosis in that cohort (~12%). During periods of COVID-19 surge, when hospitals are under tremendous strain, tools like ROX index can guide resource and staff allocation. Our health system has embedded ROX into the electronic health record to identify high-risk patients.

There is no acceptable cutoff to call a positive predictive value adequate. One must interpret the positive predictive value in the context of the clinical scenario and cost/benefit of the intervention. For example, we have shown in previous work in the New England Journal of Medicine that a model implemented with a positive predictive value of only 10% can be associated with a decrease in mortality (6). The positive predictive value of 59.4% in this study correlates to a number needed to evaluate (7) of 1.7 (calculated as 1/positive predictive value), which is excellent to use as an electronic screening tool, especially when it is combined with bedside clinical assessment. Additionally, the intervention of rounding prioritization costs nothing because patients would be rounded on anyway. Although the positive predictive value of ROX index was lower in COVID-19 patients, we felt that it was adequate to embed in the electronic health record because it would be used as a rounding prioritization tool, not a definitive answer about whether to intubate or not. We did not explore changing the model threshold for COVID-19 patients because we preferred embedding one model with one threshold across patients. Otherwise, the model would need to electronically account for diagnoses, which are inaccurate early in hospitalization.

This study fills important gaps in COVID-19 risk stratification. We report on a substantially larger cohort of patients receiving HFNC across multiple centers (8–10), which bolsters confidence in our findings. The range of number of patients and hospitals in previous studies is between 62 and 255 patients and 1–5 hospitals, respectively (8–10). We also examine a highly contemporary cohort, including those treated after the Randomized Evaluation of COVID-19 Therapy (RECOVERY) trial established dexamethasone as standard of care in COVID-19 pneumonia (11). Prior smaller studies included data through June 2020 (8–10). We also performed survival analysis to confirm that intermediate- and high-risk ROX groups have different likelihoods of invasive mechanical ventilation or death compared with low-risk ROX group. Further research is needed to assess ROX at later time points, the predictive capability of each part of the ROX index, and the utility of using noninvasive ventilation in COVID-19.

This study has important limitations. Respiratory rate may not be accurate (12), which can alter ROX values. Factors not accounted for in ROX should guide decisions to intubate, including patients’ work of breathing and secretions. Because this was a validation study, we used the two thresholds that were previously used to develop and validate ROX prior to the COVID-19 pandemic. It may be beneficial in the future to study a stricter threshold. Strengths include its large, population-based sample spanning all waves of the pandemic and evaluating multiple outcomes across ROX risk groups.

CONCLUSIONS

ROX index at 12 hours after initiation of HFNC for COVID-19 has adequate positive predictive value using the stricter threshold of less than 3.85. Our integrated healthcare system has embedded ROX index into the electronic health record to use as a rounding prioritization tool to identify high-risk patients during periods of inpatient surge.

ACKNOWLEDGMENTS

We thank Patricia Kipnis, PhD, for providing statistical consultation.