Validation of an Electronic Phenotyping Algorithm for Patients With Acute Respiratory Failure.

Acute respiratory failure is a common reason for ICU admission and imposes significant strain on patients and the healthcare system. Noninvasive positive-pressure ventilation and high-flow nasal oxygen are increasingly used as an alternative to invasive mechanical ventilation to treat acute respiratory failure. As such, there is a need to accurately cohort patients using large, routinely collected, clinical data to better understand utilization patterns and patient outcomes. The primary objective of this retrospective observational study was to externally validate our computable phenotyping algorithm for patients with acute respiratory failure requiring various sequences of respiratory support in real-world data from a large healthcare delivery network.
DESIGN: This is a cross-sectional observational study to validate our algorithm for phenotyping acute respiratory patients by method of respiratory support. We randomly selected 5% (n = 4,319) from each phenotype for manual validation. We calculated the algorithm performance and generated summary statistics for each phenotype and a priori defined clinical subgroups.
SETTING: Data were extracted from a clinical data warehouse containing electronic health record data from 46 ICUs in the southwest United States. PATIENTS: All adult (≥ 18 yr) patient records requiring any type of oxygen therapy or mechanical ventilation between November 1, 2013, and September 30, 2020, were extracted for the study.
INTERVENTIONS: None.
MEASUREMENTS AND MAIN RESULTS: Micro- and macroaveraged multiclass specificities of the algorithm were 0.902 and 0.896, respectively. Sensitivity and specificity of phenotypes individually were greater than 0.90 for all phenotypes except for those patients extubated from invasive to noninvasive ventilation. We successfully created clinical subgroups of common illnesses requiring ventilatory support and provide high-level comparison of outcomes.
CONCLUSIONS: The electronic phenotyping algorithm is robust and provides a necessary tool for retrospective research for characterizing patients with acute respiratory failure across modalities of respiratory support.

Acute respiratory failure imposes significant strain on the U.S. healthcare system. Hospital admissions for acute respiratory failure doubled between 2001 and 2009, costing more than $54 billion in 2009 alone (1), which burdens patients with long hospital stays, increased morbidity, and high mortality rates. Acute respiratory failure is common in patients requiring admission to the ICU (2), where pneumonia accounts for 50–75% of acute “hypoxemic” respiratory failure admissions (3, 4) and is a leading cause of death (1).

Noninvasive respiratory strategies are frequently used and often effective methods of treating patients with acute respiratory failure to avoid sedation and mechanical ventilation. Noninvasive respiratory strategies, which historically involved positive pressure delivered via a tight-fitting facemask (continuous positive airway pressure and bilevel positive airway pressure), are effective in patients with chronic obstructive pulmonary disease or decompensated acute heart failure (5–8). However, noninvasive positive-pressure ventilation (NIPPV) shows mixed results when used for patients with acute “hypoxemic” respiratory failure, trading improved ventilator associated pneumonia rates, ventilation days, and ICU length-of-stay (9, 10), for high failure rates (11, 12). High-flow nasal oxygen (HFNO) is an alternative to NIPPV that provides flow-dependent physiologic effects similar to NIPPV (13–16). Although both are used with increasing frequency, the need to understand the role and outcomes associated with each strategy are paramount.

Large-scale electronic health record-based datasets have potential to advance this understanding through high-quality observational studies. However, accurate phenotyping of patients based on treatment modality is an obstacle given nonstandardized terminology and lack of mapping to Common Data Model (17). In addition, manual review of large datasets is burdensome, time-consuming, and financially ineffective. We previously developed a computable phenotyping algorithm that seeks to overcome those obstacles (18), and the goal of this study was to validate that algorithm using a large clinical dataset.

MATERIALS AND METHODS

Study Population

This retrospective observational study used clinical data from patients admitted to any hospital in the Banner Health Network, which uses the Cerner Millennium (Cerner Corporation, North Kansas City, MO) electronic health record. All adult (≥ 18 yr) patient records requiring any type of oxygen therapy or mechanical ventilation between November 1, 2013, and September 30, 2020, were extracted for the study. Readmissions and pediatric admissions (< 18 yr) were excluded to facilitate a crude comparison of patient outcomes. Data were deidentified and consisted entirely of structured clinical data for the duration of each hospital stay.

We applied our rule-based phenotyping algorithm (18) to create seven phenotypes (Fig. ) of patients based on the sequence of respiratory support received. The algorithm sequences various types of therapy records and time stamps to determine the appropriate phenotype. The algorithm requires more than one record for each ventilation type to be sequenced and to corroborate the sequence using surrogate records such as medications related to preintubation, intraintubation, and postintubation cares (e.g., rapid sequence intubation medications, neuromuscular blocking agents, and continuous sedative agents) and nurse or respiratory therapy charts. A decision tree model of the phenotyping algorithm is included in the Supplementary Digital Content, Figure 1, http://links.lww.com/CCX/A936.

Phenotypes 0–2 are patients treated with a single therapy (invasive mechanical ventilation [IMV], NIPPV, and HFNO). Phenotypes 3 and 5 are patients treated with either NIPPV or HFNO and subsequent intubation. Phenotypes 4 and 6 are those that were intubated initially and extubated to either NIPPV or HFNO. Patient records with either low-flow or conventional oxygen therapy were excluded.

Data Analysis

The primary objective of this study was to validate the phenotyping algorithm. A randomly selected 5% of records from each phenotype were manually validated by two authors (P.E. and J.M.M.) by examining the sequenced records for each patient to ensure correct classification. A random 5% selection of each individual phenotype was used to avoid underrepresenting certain phenotypes due to large class imbalance and better illustrates algorithm performance on phenotypes with fewer patients. Sensitivity and specificity of each cohort from manual validation as well as the commonly observed causes of misclassification are reported. Multiclass, micro- and macroweighted average specificities, and F1 score are also reported. Microaveraged metrics calculate the performance using the individual true and false positives and negatives from each class, whereas macroaveraged metrics calculate performance for each class individually and average across all classes. A multiclass classification confusion matrix is also included.

The secondary outcome was to generate descriptive statistics for four clinical subgroups: heart failure, chronic obstructive pulmonary disease, asthma, and de novo respiratory failure that include demographics, primary diagnoses, comorbidities, and basic physiologic measurements upon admission to evaluate the utility of the phenotypes generated by the algorithm. Primary diagnoses, comorbidities, and a priori-determined clinical subgroups were selected using International Classification of Diseases, 9th and 10th Versions. A full list of codes can be found in the Supplementary Digital Content, for primary diagnoses (Supplementary Digital Content, Table , http://links.lww.com/CCX/A936), comorbidities (Supplementary Digital Content, Table , http://links.lww.com/CCX/A936), and clinical subgroups (Supplementary Digital Content, Table , http://links.lww.com/CCX/A936). Primary diagnoses were classified into the following categories: 1) cardiovascular; 2) neurologic, psychiatric, and cerebrovascular; 3) gastrointestinal; 4) respiratory; 5) infectious, allergic, and immunologic; 6) trauma; and (7) renal. Comorbidities included heart failure, diabetes, chronic kidney disease, chronic liver disease, chronic obstructive pulmonary disease, and immunosuppression or neoplasm.

Categorical variables are reported as proportions, and continuous variables are reported as median values with interquartile ranges (IQRs). Statistical comparisons of patient-related outcomes were outside the scope of this study, as statistical comparisons will require matching and controlling for confounders.

Data preprocessing and analyses were performed using Python Language Reference Version 2.7.14 (Python Software foundation, Wilmington, DE) and R Version 3.4.3 (R Foundation for Statistical Computing, Vienna, Austria). This work adheres to the STrengthening the Reporting of OBservational studies in Epidemiology reporting guidelines for observational studies and was approved by the institutional review boards of both the University of Arizona (1907780973) and Banner Health (483-20-0018).

RESULTS

Algorithm Performance

There were 183,783 total patients in the dataset. Of those, 16,832 patients were excluded for readmissions and 76,680 for receiving low-flow conventional oxygen therapy. The remaining 88,271 patients were categorized by the phenotyping algorithm (Fig. 1), resulting in 1,894 patients being excluded due to erroneous and inconsistent records, and 86,377 patients were assigned to one of seven phenotypes. Manual validation was performed on 4,319 randomly selected patients (5%) (Tables and ). Multiclass validation (micro and macro)-weighted average specificities were 0.994 and 0.982, respectively, with an average F1-score of 0.989.

Figure 1.

STrengthening the Reporting of OBservational studies in Epidemiology flow diagram of study participants. HFNO = high-flow nasal oxygen, IMV = invasive mechanical ventilation, NIPPV = noninvasive positive pressure ventilation.

The algorithm performed well (specificity and sensitivity greater than 0.90) for all phenotypes. Phenotypes with the highest sensitivity and specificity were phenotypes 1 (NIPPV only), 2 (HFNO only), and 3 (IMV only), with no incorrectly classified patients in the manual validation set. The lowest performing was phenotype 4 (extubation to NIPPV) with a specificity and sensitivity of 0.919 and 1.00, respectively. Typically, incorrect classification was due to erroneous record sequences, whereby it was unclear what the correct phenotype should be. Of the 25 algorithm failures to classify, 76% appeared to be because of incorrect use of terminology, 20% were because of record-keeping errors, and 4% were because of multiple therapy transitions (e.g., HFNO to NIPPV to IMV to HFNO).

Descriptive Statistics

Demographics, severity of illness, comorbidities, and vital signs were generally similar across all IMV, NIPPV, and HFNO (see Table for abbreviated summary statistics; and Supplementary Digital Content, Table 4, http://links.lww.com/CCX/A936, for complete summary statistics). Initial treatment with IMV was the most common therapy (59%), followed by NIPPV (38%) and then HFNO (3%). NIPPV was the most used primary noninvasive strategy. The lower prevalence of HFNO may be due to our dataset dating to 2013, prior to widespread HFNO use that has increased substantially in recent years.

A noninvasive strategy was used postextubation in 6,783 patients (13%) (Supplementary Digital Content, Table 4, http://links.lww.com/CCX/A936). Of the 32,515 total patients that were treated with NIPPV as the first assigned therapy, 15% (n = 5,039) required subsequent intubation, and of the 2,537 patients treated with HFNO as the first assigned therapy, 23% (n = 582) required subsequent intubation. Of the 51,325 patients that required IMV, 4,943 (10%) were extubated to NIPPV, and 1,840 (4%) were extubated to HFNO. Patients first treated with IMV were generally intubated soon after admission (median hours, 10.2 [IQR, 2–36]), whereas median time to NIPPV (47.6 hr [13–110 hr]) and HFNO (50.2 hr [20–114 hr]) were comparatively prolonged (Supplementary Digital Content, Table 4, http://links.lww.com/CCX/A936). More information on the crude (unadjusted) outcomes and clinical subgroups can be found in the Supplementary Digital Content, Tables 5 and 6, http://links.lww.com/CCX/A936.

DISCUSSION

These results confirm successful performance of our computable phenotyping algorithm using a large, clinically available, nonstandard data set. The algorithm successfully classified patients with 100% specificity for the phenotypes, where only a single therapy was required and demonstrated 99.6% and 100% specificity when NIPPV or HFNO, respectively, failed and required intubation. The algorithm showed 95.6% specificity when phenotyping patients extubated to HFNO, but only 91.9% for patients extubated to NIPPV (Table 1).

TABLE 1.

Validation Performance Metrics for Each Phenotype

Phenotype	Specificity	Sensitivity
0: Invasive mechanical ventilation	1.00	0.9893
1: NIPPV	1.00	1.00
2: HFNO	1.00	1.00
3: NIPPV failure	0.9960	0.9960
4: Invasive mechanical ventilation to NIPPV	0.9190	1.00
5: HFNO failure	1.00	1.00
6: Invasive mechanical ventilation to HFNO	0.9565	1.00

HFNO = high-flow nasal oxygen, NIPPV = noninvasive positive pressure ventilation.

Classification performance is highly dependent on the quality of input data and preprocessing due to the rule-based nature of the algorithm and dependence on available concepts within the data. The reduced performance for patients extubated to NIPPV appeared to be due to extended lengths of stay requiring multiple strategies. This resulted in an unclear sequence of records due to inaccurate timestamps and unclear terminology such as “continuous positive airway pressure” for both a noninvasive respiratory strategy and to indicate pressure support ventilation in intubated patients. Although we compensated for this specific term in our algorithm, these input errors and unclear terms can make the sequence of events lead to misclassifications. Records indicating “IMV assessments” for tracheostomy and reintubations also caused a small portion of incorrect classification.

Performance of the algorithm would benefit from standardized terminologies across electronic health record systems (e.g., Systemized Nomenclature of Medicine—Clinical Terms and Current Procedural Terminology). Computable phenotyping algorithms such as this have potential utility for analysis using large datasets where manual classification is not feasible. However, inconsistent terms may degrade algorithms and decrease the validity of research or quality improvement findings depending on the rules used for phenotyping. Standardized terminology would also permit more detailed classification, whereby the rule-based sequencing of records better captures reintubations and complex crossovers (e.g., HFNO to NIPPV to IMV to HFNO). Our algorithm is adaptable and can be operationalized and deployed broadly at other sites. However, the phenotyping process should take local and institutional practices into consideration when porting from one dataset or site to another (19).

Overall, these results show that our phenotyping algorithm is a robust and useful tool for retrospective, observational research that can further correct for confounding variables. Adjusted comparisons are outside the scope of this study but found several crude observations. We found that utilization of NIPPV in our dataset (38% of patients) is consistent with previous reports, where NIPPV is used in roughly 40% of ICU patients (20, 21). NIPPV utilization for patients with acute hypoxemic, or de novo, respiratory failure in our dataset was more than twice that reported in other publications (22, 23). HFNO carried the highest overall ICU mortality at 28.1%, but failure was associated with a lower relative increase in ICU mortality (47%) than failure of NIPPV (72%), with similar ICU and hospital mortalities of 60.4% and 60.7%, respectively. These findings would suggest there are important differences in the phenotype of patients assigned to each treatment, which prohibit comparisons between the two modalities without accounting for these confounders.

There are several limitations to this study. The data used were extracted from an electronic health record-based clinical data warehouse. Hospitals contributing data range in size, geographic location, population density, practice style and staffing, and academic or community focus. As a result, utilization and expertise also vary. This is particularly true when comparing NIPPV with HFNO due to subjectivity regarding first-line therapy assignment or the decision to intubate a patient that failed either therapy. Clinical variations, in a study this size across many institutions with varying degrees of expertise and experience, could lead to inconsistencies in outcomes data. Given these important confounders by indication and the primary objective of validating the algorithm performance, only high-level observations are reported.

Additionally, the availability of HFNO is not uniform, and the utilization has likely changed over time. The nonuniformity over time was exacerbated by widespread use of HFNO in COVID-19 patients and may limit the generalizability of these results. Because HFNO is a newer therapy with a nasal cannula interface, there is potential disparity in treatment assignment based on numerous factors that cannot be accounted for in modeling. Simply adjusting for severity of illness is fraught with error given that the variables that determine severity of illness are highly dependent on treatment assignment when patients are assigned early (before the first 24 hr).

Low data veracity in our dataset can potentially lead to incorrect classification, as seen for the cohort of patients extubated to NIPPV whereby overlapping concepts for NIPPV and IMV mode exist. Similar risks exist for HFNO patients as there are no standard concepts that indicate use of a high-flow nasal cannula system, such as the Optiflow (Fisher and Paykel Healthcare, Aukland, New Zealand) or the Vapotherm (Vapotherm, Exeter, NH). Validation of each phenotype, however, does allow for deeper clinical subgroup analysis. Future use of this algorithm could emulate random therapy assignment case-matching or statistical weighting and supports the need for standardized concepts and terminologies.

CONCLUSION

Our algorithm can reliably phenotype patients based on the respiratory support strategy received, either in isolation or in combination. In this dataset, NIPPV was more commonly used than HFNO, both as a primary therapy and in the postextubation period. Future research can leverage this algorithm for observational comparisons to guide trial development, quality improvement projects, and clinical care.

ACKNOWLEDGMENTS

We thank Don Saner and Mario Arteaga at Banner Health for their support of this work through the clinical data warehouse.

TABLE 2.

Multiclass Confusion Matrix From Manual Validation Illustrating the Number of Patients Actually in Each Phenotype Relative to the Phenotype Assigned by the Algorithm

Algorithmic Phenotype,a n (%)	Actual Phenotypeb
	0	1	2	3	4	5	6
0	2,227 (100)	—	—	—	—	—	—
1	—c	1,374 (100)	—	—	—	—	—
2	—	—	98 (100)	—	—	—	—
3	1 (0.004)	—	—	251 (99.6)	—	—	—
4	19 (0.077)	—	—	1 (0.4)	227 (91.9)	—	—
5	—	—	—	—	—	29 (100)	—
6	4 (0.043)	—	—	—	—	—	88 (95.7)

aPhenotype as determined by the algorithm.

bActual phenotype determined by manual validation.

cNot applicable or zero.

TABLE 3.

Demographics Summary

Parameter	Invasive Mechanical Ventilation			NIPPV		High-Flow Nasal Oxygen
	Cohort			Cohort		Cohort
	0	4	6	1	3	2	5
	IMV	IMV→NIPPV	IMV→HFNO	NIPPV	NIPPV→IMV	HFNO	HFNO→IMV
Patients, n	44,542	4,943	1,840	27,476	5,039	1,955	582
Male sex, %	60	58	62	53	56	53	61
Age, median (IQR)	64 (50–74)	67 (58–75)	63 (51–73)	70 (59–79)	66 (56–75)	71 (59–81)	64 (52–73)
Ethnicity, %
Hispanic or Latino	17	14	23	13	17	16	21
Not Hispanic or Latino	83	86	77	87	83	83	78
Acute Physiology and Chronic Health Evaluation IVa score on ICU admission, median (IQR)	63 (46–86)	63 (49–80)	75 (54–99)	51 (39–63)	68 (53–86)	54 (42–68)	72 (54–96)
Primary diagnosis by organ system, %
Cardiovascular	28	26	20	14	12	10	10
Respiratory, all causes	10	17	11	23	20	19	13
Comorbidities, %
Heart failure	8	5	4	8	4	5	1
Chronic obstructive pulmonary disease	< 1	< 1	< 1	< 1	< 1	< 1	< 1
Vital signs on treatment assignment, median (IQR)
Heart rate	85 (73–99)	83 (72–96)	90 (75–102)	83 (72–96)	87 (74–102)	88 (77–100)	94 (82–106)
Systolic blood pressure	120 (105–136)	121 (108–137)	118 (103–136)	126 (111–143)	118 (104–135)	121 (108–140)	116 (104–132)
Diastolic blood pressure	67 (58–78)	68 (59–77)	64 (57–75)	70 (61–79)	66 (56–76)	68 (59–78)	65 (55–78)
Oxygen Saturation (pulse oximetry)	97 (95–99)	97 (95–99)	97 (95–99)	96 (94–98)	97 (94–99)	95 (92–97)	96 (93–98)
Respiratory rate	18 (16–22)	18 (16–21)	20 (17–24)	18 (17–21)	20 (17–24)	18 (17–22)	22 (18–26)

HFNO = high-flow nasal oxygen, IMV = invasive mechanical ventilation, IQR = interquartile range, NIPPV = noninvasive positive pressure ventilation.