Predictive Algorithms for a Crisis.

The COVID-19 pandemic has been associated with global surges in need for hospital-based and other healthcare resources, as well as shortages in vital items such as ICU beds, mechanical ventilators, and personal protective equipment (Fig. 1) (1). During the initial waves of the COVID-19 pandemic, several countries experienced a shortage of mechanical ventilators (2). Decision-making on triage and rationing of scarce resources during a crisis can lead to critical adjustments to what would otherwise be considered standard of care. For patients who are fully ventilator-dependent, decisions surrounding initiation or termination of mechanical ventilation can acutely mean the difference between life and death (2). Although health system surge capacities (both for acute and routine care needs) can be essential to pandemic preparedness (1), it is also of paramount importance that policies on crisis standards of care be informed by both evidence and ongoing reevaluation.

Figure 1.

Depiction of mechanical ventilator, ICU bed, and other resources.

In this issue of Critical Care Medicine, Keller at al (3) provide key insights regarding the use of a preintubation Sequential Organ Failure Assessment (SOFA) score to predict COVID-19 mortality. In a multicenter, retrospective, large database cohort study of over 15,000 mechanically ventilated COVID-19 patients at 86 U.S. healthcare systems, the authors used electronic health record data to assess the predictive capacity of the SOFA score for inhospital mortality in COVID-19 patients. The authors found that the SOFA score demonstrated poor discriminant accuracy for inhospital mortality in mechanically ventilated patients. The observed area under the receiver operating curve (AUC) for the SOFA score to predict inhospital mortality was 0.66 (95% CI, 0.65–0.67), generally considered poor accuracy (the authors used the cutoffs of less than 0.7 as poor accuracy, 0.7–0.8 as moderate, 0.8–0.9 as good, and greater than 0.9 as excellent) (4). The addition of comorbidities did not substantially improve the predictive model, and age alone performed better for predicting inhospital mortality than the SOFA score. Even when reviewing ventilated patients with COVID-19 who survived hospitalization, the SOFA score poorly predicted those who required long-term acute care.

This study adds notable value for several reasons. The findings add multi-institutional external validity, with a large sample size, to an increasing number of smaller studies questioning the utility of using the SOFA score in this context (5, 6). In a 2021 research letter published in JAMA, Raschke et al (5) conducted a retrospective review of 675 adult patients with COVID pneumonia requiring mechanical ventilation across 18 ICUs in the southwestern United States. Similar to the findings from Keller et al (3), the authors of the JAMA research letter found poor discriminant accuracy for the SOFA score to predict hospital mortality or discharge to hospice (AUC, 0.59; 95% CI, 0.55–0.63). In both studies, the authors appropriately questioned the value of using a SOFA scoring system not designed for this purpose that distributes points across several domains (i.e., neurologic, cardiovascular, pulmonary, renal, hepatic, and coagulation). The combined scientific rigor and multidisciplinary nature of the study by Keller et al (3) add to the potential for this new study to inform important health policy decisions. The authors of the Keller et al (3) study have affiliations spanning both an academic medical institution, the National Institutes of Health Clinical Center, the National Institute of Allergy and Infectious Disease, and the Frederick National Laboratory for Cancer Research. The authors report that, at the time of their study, several U.S. states with crisis standard-of-care guidelines still had ventilator triage algorithms featuring the SOFA score. In this regard, the findings are particularly timely and relevant for reassessment of crisis standards during a time when ventilators may be more available.

Although the study provides valuable findings, it is not without limitations. The authors did not assess outcomes that go beyond hospital discharge. Although inhospital mortality is an important outcome, many other relevant considerations should be taken into account in making these challenging decisions, including resource consumption, quality of life, functional outcomes, long-term sequelae, healthcare disparities, and health equity. As such, the use of SOFA in crisis standards of care may exacerbate racial and ethnic disparities in resource allocation, and a deliberate approach that ensures incorporation of a health equity perspective is key (7, 8). There is added value in looking at these important considerations as primary outcomes, similar to how inhospital mortality was assessed for this particular study. There are also limitations related to how certain factors were adjusted, or not adjusted for, in the current analysis. Although the authors performed an additional sensitivity analysis to account for patients who had “an International Classification of Diseases, 10th Edition Major Operating Room procedure code on the same day as intubation (to account for potential preoperative rather than critical illness-related intubation),” there are many other nonoperating room and/or minor procedures that frequently require endotracheal intubation during a hospitalization, including some of the procedures done by interventional radiologists, cardiologists, gastroenterologists, and other procedural subspecialists. Other large database studies have been able to be more granular regarding diagnostic and therapeutic procedures in their assessment of outcomes (9, 10). The authors also did not adjust for patients who were transferred to another hospital at the end of their index hospitalization, which may have ultimately led to an inhospital mortality (just in a different hospital). Other large database studies looking at mortality have been able to track and account for these types of hospital transfers (11, 12). Although these latter limitations may have been inherent to a combination of the data available to the authors and the selected study design, it is nevertheless encouraging that the authors provided sensitivity analyses spanning several different considerations, all of which were consistent with the primary results.

COVID-19 has generated a renewed focus on the concept of pandemic preparedness as an important topic of discussion at the highest levels of healthcare and beyond (1, 13). It is important that crisis standards of care be informed by a representative group of multidisciplinary experts reviewing the latest evidence and considering a range of important outcomes and lessons learned from past experience. Similar to how patient safety has been referred to as “…[not] a preoccupation of the past […] not a problem that has been solved, but rather an ongoing requirement” (14, 15), preparation for the ever-changing needs of COVID-19, variants, long-term effects, and new developments should be subject to ongoing reappraisal based on evolving evidence.