Stratifying Deterioration Risk by Acuity at Admission Offers Triage Insights for Coronavirus Disease 2019 Patients.

OBJECTIVES: Triaging patients at admission to determine subsequent deterioration risk can be difficult. This is especially true of coronavirus disease 2019 patients, some of whom experience significant physiologic deterioration due to dysregulated immune response following admission. A well-established acuity measure, the Rothman Index, is evaluated for stratification of patients at admission into high or low risk of subsequent deterioration.
DESIGN: Multicenter retrospective study.
SETTING: One academic medical center in Connecticut, and three community hospitals in Connecticut and Maryland. PATIENTS: Three thousand four hundred ninety-nine coronavirus disease 2019 and 14,658 noncoronavirus disease 2019 adult patients admitted to a medical service between January 1, 2020, and September 15, 2020.
INTERVENTIONS: None.
MEASUREMENTS AND MAIN RESULTS: Performance of the Rothman Index at admission to predict in-hospital mortality or ICU utilization for both general medical and coronavirus disease 2019 populations was evaluated using the area under the curve. Precision and recall for mortality prediction were calculated, high- and low-risk thresholds were determined, and patients meeting threshold criteria were characterized. The Rothman Index at admission has good to excellent discriminatory performance for in-hospital mortality in the coronavirus disease 2019 (area under the curve, 0.81-0.84) and noncoronavirus disease 2019 (area under the curve, 0.90-0.92) populations. We show that for a given admission acuity, the risk of deterioration for coronavirus disease 2019 patients is significantly higher than for noncoronavirus disease 2019 patients. At admission, Rothman Index-based thresholds segregate the majority of patients into either high- or low-risk groups; high-risk groups have mortality rates of 34-45% (coronavirus disease 2019) and 17-25% (noncoronavirus disease 2019), whereas low-risk groups have mortality rates of 2-5% (coronavirus disease 2019) and 0.2-0.4% (noncoronavirus disease 2019). Similarly large differences in ICU utilization are also found.
CONCLUSIONS: Acuity level at admission may support rapid and effective risk triage. Notably, in-hospital mortality risk associated with a given acuity at admission is significantly higher for coronavirus disease 2019 patients than for noncoronavirus disease 2019 patients. This insight may help physicians more effectively triage coronavirus disease 2019 patients, guiding level of care decisions and resource allocation.

Hospitalized severe acute respiratory syndrome coronavirus 2 disease (COVID-19) patients are susceptible to a dysregulated immune system response giving rise to significant deterioration and elevated rates of ICU utilization, mechanical ventilation, and in-hospital mortality (1, 2). In regions with high COVID-19 case rates among the general population, hospitals have been overwhelmed by patient volumes (3–5). These surge conditions have resulted in fatigued, overextended clinical staff having to make rapid assessment of patient risk, a difficult task during the best of times, to allocate limited hospital resources, for example, mechanical ventilators and ICU beds. The challenges faced by caregivers under pandemic conditions cannot be overstated (6). Use of preexisting, well-validated acuity models to support triaging of COVID-19 patients has the potential to ease the burden on clinical staff by supporting them in making difficult care decisions in a more effective and objective fashion.

Among COVID-19 patients, reports indicate that age and comorbidity correlate with mortality risk (7, 8). We postulate that both are in essence indirect and inadequate estimates of patient acuity. Age itself is not intrinsically a risk factor but rather a proxy for age-related diseases and infirmities. Comorbidity is a more explicit measure of diseases and infirmities; however, the presence or absence of comorbidities does not directly translate to acuity without a characterization of extent of impairment. Our work evaluates the relationship between impairment and subsequent in-hospital deterioration by explicitly measuring physiologic acuity at admission using the Rothman Index (RI), an extensively validated, commercially available patient condition score (PeraHealth, Charlotte, NC) which is in widespread use by hospitals and health systems across the United States. The RI is a finely graduated acuity score with a large dynamic range, from a maximum of 100 to a minimum of –91, enabling clear distinctions in risk to be made across the acuity spectrum, including in cases of mild impairment. This stands in contrast to other common acuity measures, for example, the National Early Warning Score (NEWS), the Modified Early Warning Score (MEWS), or the Confusion-Urea-Respiratory Rate-Blood Pressure-Age score (CURB-65), which have score ranges of 21, 15, and 6 points, respectively (9–11). Unlike models that are explicitly developed and trained around COVID-19, the RI is suitable for evaluating differences in acuity implications between coronavirus disease (COVID) and non-COVID populations. To calibrate the reader, decreasing RI scores correspond to increasing acuity: 100 implies no physiologic impairment; 65 is typical acuity for patients discharged to skilled nursing facilities (12); 40 is approximately the acuity of patients considered for transfer to an ICU; and 0 corresponds to a 50% likelihood of in-hospital mortality.

RI model features include vital signs, blood oxygen saturation, heart rhythm, eleven different body system nursing assessments, the Braden score, and seven laboratory values. The required data are routinely documented on all hospitalized patients and recorded in the electronic medical record (EMR), necessitating no additional patient assessment or clinical documentation. Software containing the RI algorithm interfaces with the EMR allowing real-time updates using emergency department and inpatient unit data. The model accommodates asynchronous data inputs and is calculated when sufficient inputs (no more than two missing vital signs or nursing assessments) have been received (12). The model development methodology has been published (12–16) and clinically validated as an indicator of patient acuity and early deterioration (17–20).

The RI is unique in its inclusion of the full range of body system nursing assessment data which categorize physical and behavioral evaluations for dozens of factors (e.g., patient stops eating, becomes confused, has trouble walking, develops edema, etc.). Deficits charted by nurses frequently serve as indicators of deterioration earlier than vital sign derangement (13) and allow the RI to perform significantly better as a predictor of adverse outcomes than acuity scores predominantly based on vital signs, such as MEWS (14). Recent work examining the impact of aspirin and anticoagulation on COVID-19 patients identified the utility of the RI in controlling for severity of illness among COVID-19 study populations (21).

Here, we evaluate the ability of the RI model at the time of hospital admission to stratify patients into high or low risk of subsequent severe deterioration with an emphasis on opportunities to support risk stratification in the COVID-19 population.

MATERIALS AND METHODS

Data

We analyzed data from LifeBridge Health System’s Sinai Hospital of Baltimore, in Baltimore, MD, and from three Yale New Haven Health System (YNHHS) hospitals in New Haven, Bridgeport, and Greenwich CT. These include urban and suburban community hospitals and an academic medical center.

Non–COVID-19 data were restricted to a timeframe predating changes to admission and care practices provoked by COVID-19, to reflect a representative sample of non-COVID patients. Data from Sinai Hospital of Baltimore included 2019 non-COVID-19 patients admitted and discharged between January 1, and March 15, 2020, and 355 COVID-19 patients admitted and discharged between March 21, and September 14, 2020. Data from YNHHS included 12,639 non–COVID-19 patients admitted and discharged between January 1, and March 15, 2020 and 3,144 COVID-19 patients admitted and discharged between April 1, and June 15, 2020. At all hospitals, COVID-19 patients were diagnosed based on laboratory tests performed either before or during the hospital admission and identified either by indicators in the patient chart or International Classification of Diseases, 10th Edition (ICD-10) diagnosis code U07.1.

The study population was limited to patients 18 years old or older admitted to a medical service (identified by Medicare Severity Diagnosis Related Groups) and who had RI scores during their stay. More than 97% of adult medical patients at each hospital had RI scores. This study was approved with a waiver of informed consent (number 072003) by the Bridgeport Hospital Institutional Review Board and received exempt status from the LifeBridge Health Institutional Review Board.

Population Characteristics

Mean and median age and length of stay (LOS), sex, rates of mortality, rates of discharge to hospice, ICU utilization, and mechanical ventilation, as well as Charlson Comorbidity Index (CCI) for all patients and the subset who expired, are reported for COVID-19 and non–COVID-19 patients. The proportion of COVID-19 and non–COVID-19 patients presenting with each of the individual CCI comorbidities as well as hypertension and obesity is also reported. All comorbidities were identified based on ICD-10 diagnosis codes from the current visit (22). Differences between COVID-19 and non–COVID-19 populations were evaluated for significance with t tests for equal or unequal variance.

RI at Admission as a Predictor of Risk

To evaluate differences in deterioration risk between COVID-19 and non–COVID-19 patients, we compared RI at the time of admission to an inpatient unit with in-hospital mortality rates for COVID-19 and non–COVID-19 populations. We validated the use of the RI with COVID-19 patients by using a single-variable logistic regression model to compare the first RI score in predicting two independently evaluated endpoints: whether the patient expired in the hospital or spent any time in the ICU. Discriminatory performance was assessed using the receiver operating characteristic area under the curve (AUC) together with evaluation of model calibration.

Determining Operating Thresholds for Risk Stratification

Precision (positive predictive value) and recall (sensitivity) values at different RI threshold values were evaluated, and two threshold values were selected for categorizing patients as either low or high risk for subsequent deterioration.

Opportunities to Augment Level of Care Decisions

In order to evaluate whether RI-based risk stratification could provide new insight to providers making care decisions, we analyzed level of care assignment at admission. For patients meeting the RI high risk criteria at admission, we evaluated whether patients were admitted directly to ICU, transferred to ICU following admission (and if so, how long after admission), or never admitted to an ICU, to determine potential opportunities for improved or more timely location assignment on the basis of admission risk.

RESULTS

Population characteristics comparing the general medical non–COVID-19 patient population with the COVID-19 population at each hospital are given in Table . Statistically significant differences between COVID-19 and non–COVID-19 groups are indicated.

In general, the COVID-19 population is older, more likely to be male, and has longer lengths of stay. They are much more likely to require admission to the ICU, use of a ventilator, and expire in the hospital. However, mean CCI of patients who expire is lower (less acute) for COVID-19 than for the non–COVID-19 population. Preponderant comorbidities include chronic obstructive pulmonary disease, congestive heart failure, diabetes, hypertension, and renal disease, in agreement with other reports (7, 8, 23), whereas in terms of relative occurrence of comorbidities, dementia, diabetes, obesity, and hypertension are proportionally overrepresented in the COVID-19 population at all four sites.

Evaluation of in-hospital mortality as a function of acuity as measured by the RI at admission reveals that COVID-19 patients have a significantly higher mortality risk than non–COVID-19 patients for a given acuity, as shown in Figure . Data volumes at Yale New Haven Hospital supported further separating out respiratory medical patients (diagnoses of pneumonia, influenza, acute upper respiratory infection, and acute respiratory failure), and although their mortality risk is higher than that of nonrespiratory, non–COVID-19 patients, it remains far lower than for COVID-19 patients.

The median time between assignment to an inpatient location and generation of the first RI score for patients at all four hospitals ranged from 1 to 1.3 hours. Data are bucketed by RI decile, and the mean of each decile plotted; scores from 20 to 40 and below 20 are grouped to ensure reasonable sample sizes (sample count details are shown in Supplemental Fig. S-1, http://links.lww.com/CCX/A567).

To validate the performance of the initial RI score as a predictor of subsequent deterioration for both COVID-19 and non–COVID-19 patients, the AUC for discrimination of patients with and without each of two deterioration-related events, in-hospital mortality and need for ICU level care during hospitalization, is calculated (Table ).

For both COVID-19 and non–COVID-19 patients, performance at all hospitals using the initial RI at admission ranges from good to excellent for discriminating in-hospital mortality (AUC of 0.81–0.84 for COVID-19 and 0.90–0.92 for non–COVID-19) and fair to good for predicting need for ICU level of care (AUC of 0.62–0.80 for COVID-19 and 0.74–0.86 for non–COVID-19). The RI model is found to be well calibrated using the Hosmer-Lemeshow goodness of fit test, with p values and calibration curves for COVID-19 and non–COVID-19 populations shown in Supplemental Figure S-2 (http://links.lww.com/CCX/A568). This performance is particularly significant in light of the fact that in many instances, the target outcome may have occurred a considerable time (e.g., a number of days) following patient admission.

Operating Thresholds for Risk Stratification

To determine high- and low-risk RI-thresholds for the COVID-19 and non–COVID-19 populations, precision and recall values of different RI threshold values are evaluated as shown in Figure . To orient the reader, the right-hand most points indicate a threshold of initial RI less than 100, which encompasses all admitted patients, hence providing 100% sensitivity to mortality. Each point moving from right-to-left represents a 10-point reduction in RI threshold value; thresholds capturing 2% or less of the population are omitted. Points with black markers reflect thresholds of RI less than 50 (COVID-19) and RI less than 40 (non–COVID-19). Patients with lower RI scores, and therefore higher acuity at admission, have a higher probability of expiring in the hospital leading to increasing precision at the expense of decreasing recall.

Precision (i.e., PPV) and recall (i.e., sensitivity) for initial Rothman Index thresholds and inpatient mortality for (A) coronavirus disease 2019 (COVID-19) and (B) non–COVID-19 patients. PPV = positive predictive value, Hosp. = hospital.

Operating point thresholds for performance characterization are selected to be initial RI less than 50 to identify high-risk patients and initial RI greater than 70 to identify low-risk patients within the COVID-19 population. For the non–COVID-19 patients, in light of the lower associated risk for a given acuity, a lesser high-risk threshold of initial RI less than 40 is chosen, and the same low-risk threshold of RI greater than 70 is applied. Operating points are a compromise between sensitivity and positive predictive value. Different operating points could be chosen according to a hospital’s patient placement requirements and available resources.

Table details the characteristics of COVID-19 and non–COVID-19 populations flagged on the basis of being below or above the high- and low-risk thresholds, respectively. Fewer than a third of COVID-19 patients meet the high-risk RI criteria; they have a 34–45% mortality rate. Approximately, half the COVID-19 population meet the low risk RI criteria; they have a mortality rate of only 2–5%. Large differences in LOS and rates of hospice discharge, ICU utilization, and mechanical ventilation between the high- and low-risk groups are also evident. Similarly, sharp contrasts in adverse event rates are also seen between the high- and low-risk groups in the non–COVID-19 populations.

To determine if COVID-19 patients deemed high risk based on admission RI were already recognized and treated as such by providers, we use patient location (i.e., ICU vs non-ICU) as a proxy for provider concern. For three of the four hospitals shown in Table 3 with nonnegligible numbers of patients transferred to ICU during hospitalization (i.e., not directly admitted to ICU), the median time from admission to transfer for COVID-19 patients is two to three times longer than for non–COVID-19 medical patients (1.2–2.3 d and 0.7–1.1 d, respectively). Among patients transferred to ICU, the mortality rate of COVID-19 patients is two to three times higher than for non–COVID-19 patients (32–58% and 16–19%, respectively).

TABLE 3.

Characteristics of Patients in High- and Low-Risk Categories

Risk Group	Characteristics	Sinai Hospital of Baltimore		Yale New Haven		Bridgeport		Greenwich
		Non-COVID	COVID	Non-COVID	COVID	Non-COVID	COVID	Non-COVID	COVID
High Risk	Patients in population n (%)	99 (4.9)	92 (25.9)	729 (8.7)	590 (33.8)	270 (9.6)	290 (32)	99 (6.9)	92 (18.7)
	Mortality rate, %	24.5	44.6	17.2	34.6	23.5	33.8	20.2	34.8
	Hospice rate, %	3.1	3.3	9.5	5.4	8.5	8.3	14.1	9.8
	ICU utilization rate, %	76.5	55.4	53.4	47.1	47.1	51.4	34.3	35.9
	Mechanical ventilation rate, %	55.1	40.2	25.7	23.9	21.7	24.1	11.1	23.9
	Direct admit to ICU, %	65.3	34.8	43.8	37.6	31.3	36.2	32.3	20.7
	Direct admit to ICU—expired, %	31.2	68.8	26.0	40.2	27.1	34.3	12.5	26.3
	Transferred to ICU, %	11.2	20.7	9.6	9.7	15.8	15.2	2.0	15.2
	Transferred to ICU—expired, %	18.2	57.9	18.6	47.4	16.3	31.8	50.0	35.7
	No time in ICU, %	23.5	44.6	46.6	52.9	52.9	48.6	65.7	64.1
	No time in ICU—expired, %	8.7	19.5	8.6	28.2	23.6	34.0	23.1	37.3
	Days from admit to ICU transfer, mean, median	1.6, 1.1	3.7, 2.2	1.5, 0.7	5.1, 3.2	2.5, 1.0	3.6, 1.2	0.9, 0.9	4.1, 3.4
	LOS (d), mean, median	8.8, 6.5	12.7, 10.1	8.9, 6.3	13.8, 10.2	7.7, 6.1	11.6, 9.3	6.7, 5.0	10.0, 6.8
	LOS—expired, mean, median	4.0, 2.0	9.4, 6.9	6.6, 4.4	9.8, 6.5	5.1, 3.2	8.2, 4.6	4.7, 3.4	5.8, 5.2
	LOS—hospice, mean, median	3.1, 2.8	22.7, 20.0	7.1, 5.0	9.0, 7.2	7.5, 6.7	6.5, 5.1	2.8, 2.3	9.0, 9.1
Low Risk	Patients in population n (%)	1,490 (73.8)	181 (51)	4,954 (59.1)	665 (38.1)	1,662 (58.9)	387 (42.7)	978 (68.2)	284 (57.8)
	Mortality rate, %	0.4	5.0	0.2	3.3	0.2	2.3	0.3	2.9
	Hospice rate, %	4.6	1.1	0.5	0.6	0.4	0.0	0.7	1.4
	ICU utilization rate, %	4.0	8.8	3.2	18.2	9.6	23.0	6.0	20.8
	Mechanical ventilation rate, %	0.9	6.6	0.5	4.8	0.4	7.0	0.8	9.9
	Direct admit to ICU, %	1.8	2.2	1.7	3.9	7.0	9.3	3.2	6.3
	Direct admit to ICU—expired, %	3.7	50.0	0.0	15.4	0.0	8.3	3.2	5.6
	Transferred to ICU, %	2.1	6.6	1.5	14.3	2.5	13.7	2.9	14.4
	Transferred to ICU—expired, %	6.3	50.0	9.3	15.8	7.1	11.3	7.1	12.2
	No time in ICU, %	96.0	91.1	96.8	81.8	90.4	77.0	94.0	79.2
	No time in ICU—expired, %	0.2	0.6	0.0	0.6	0.1	0.0	0.0	0.9
	Days from admit to ICU transfer, mean, median	3.2, 1.6	3.8, 3.4	2.9, 1.7	3.1, 2.2	1.5, 0.9	3.1, 2.2	1.7, 0.7	5.7, 3.5
	LOS (d), mean, median	6.1, 4.4	8.1, 6.3	4.8, 3.1	9.1, 6.5	4.3, 3.3	8.0, 5.6	3.4, 2.9	7.4, 5.0
	LOS—expired, mean, median	8.7, 6.3	11.2, 9.2	12.6, 9.7	17.4, 15.1	12.2, 13.1	20.3, 17.2	11.6, 10.1	15.9, 14.6
	LOS—hospice, mean, median	9.0, 8.9	26.0, 26.0	10.5, 8.0	12.6, 4.4	5.8, 3.9	0.0, 0.0	2.5, 2.7	11.2, 10.7

COVID = coronavirus disease, LOS = length of stay.

DISCUSSION

A direct measure of acuity is an effective predictor of physiologic susceptibility to decline as shown in the AUC analysis in Table 2. Importantly, we also see that for a given level of acuity at admission, as measured by the RI (Fig. 1), mortality rates for COVID-19 patients are higher than for the non–COVID-19 medical population; this difference holds even when comparing COVID-19 with a subpopulation of non–COVID-19 medical patients with respiratory-related diagnoses. The higher COVID-19 risk can be understood in light of reports that some COVID-19 patients suffer a significant dysregulated immune system response, often several days into their hospital admission (2, 24). Furthermore, as seen in Table 1, we find that the comorbidity burden at admission for those patients who expired in the hospital, as measured by the CCI, was less for COVID-19 patients than for non–COVID-19 patients. These findings imply that a clinician’s sense of how to treat a COVID-19 patient may be led astray by how he or she would treat a non–COVID-19 patient.

TABLE 2.

Area Under the Curve (95% CI) for Rothman Index Discriminatory Performance for Inpatient Mortality, ICU Utilization

Outcome	Sinai Hospital of Baltimore		Yale New Haven		Bridgeport		Greenwich
	Non–COVID-19	COVID-19	Non–COVID-19	COVID-19	Non–COVID-19	COVID-19	Non–COVID-19	COVID-19
Mortality	0.91 (0.87–0.95)	0.82 (0.76–0.82)	0.91 (0.89–0.93)	0.81 (0.78–0.83)	0.92 (0.89–0.95)	0.82 (0.78–0.85)	0.90 (0.87–0.94)	0.84 (0.78–0.89)
ICU utilization	0.86 (0.83–0.89)	0.80 (0.75–0.85)	0.83 (0.82–0.85)	0.69 (0.66–0.72)	0.74 (0.72–0.76)	0.67 (0.63–0.70)	0.75 (0.72–0.79)	0.62 (0.57–0.68)

COVID-19 = coronavirus disease 2019.

Figure 1.

Rothman Index (RI) at admission versus mortality rate for coronavirus disease 2019 (COVID-19) (solid red) and non–COVID-19 (dashed blue) populations (error bars are ± 2 sem) for three hospitals. For the Yale New Haven graph, the COVID-19 population is solid red, non–COVID-19 respiratory is dashed blue, and non–COVID-19 nonrespiratory is dotted green.

TABLE 1.

Characteristics of Coronavirus Disease 2019 and Noncoronavirus Disease 2019 Populations

Hospital	Sinai Hospital of Baltimore		Yale New Haven		Bridgeport		Greenwich
Characteristics	Non-COVID	COVID	Non-COVID	COVID	Non-COVID	COVID	Non-COVID	COVID
Patient count, n	2,019	355	8,383	1,746	2,822	907	1,434	491
Age (yr), mean, median	57.6, 61	63.8, 65c	57.7, 60	65.4, 66c	58.2, 60	64.0, 65c	58.4, 58	62.9, 62c
Length of stay (d), mean, median	6.8, 5.0	10.3, 8.0c	5.8, 3.9	12.0, 8.6c	5.3, 3.9	10.4, 7.6c	4.2, 3.2	8.6, 6.0c
Male, %	38.7	46.2b	43.1	48c	39.0	49.7c	32.2	57.6c
Expired, %	2.5	16.9c	2.3	15.6c	3.0	16c	2.8	11c
Hospice discharges, %	2.4	5.1b	2.4	3.0	2.1	3.6b	2.8	3.9
ICU utilization during stay, %	12.0	23.7c	11.4	30.5c	17.9	38.8c	10.2	26.7c
Mechanical ventilation, %	5.9	16.6c	3.9	12.7c	3.6	16.9c	2.0	14.7c
CCI all patients, mean, median	2.5, 2	2.5, 2	2.6, 2	2.6, 2	2.4, 2	2.3, 2	1.6, 1	1.5, 1
CCI expired patients, mean, median	5.3, 5	3.7, 3a	5.2, 5	3.8, 3c	4.8, 4	3.5, 3c	4.1, 4	2.7, 2b
Comorbidity	% of Patients With Comorbidity
AIDS/HIV	2.1	2.3	0.4	0.6	0.9	0.4c	0.2	0.4
Cancer	6.9	3.7a	12.2	4c	8.3	3.2c	6.6	2.4c
Cerebrovascular disease	13.8	8.7b	7.6	6.8	6.0	4.7	4.8	3.1
Chronic pulmonary disease	27.2	23.7	30.5	27.2b	31.5	26b	19.6	17.1
Congestive heart failure	22.9	20.3	23.1	24.1	25.2	21.8a	17.0	10.4c
Dementia	6.8	19.2c	6.7	21.3c	11.1	22.5c	9.4	12.4
Diabetes with complications	15.2	19.7a	14.8	20.8c	15.6	19.4b	7.3	9.8
Diabetes without complications	22.1	33.5c	19.0	33.9c	19.5	29.7c	11.2	24.4c
Hypertension	35.1	44.5c	29.6	39.6c	28.9	36.4c	24.8	36.3c
Metastatic carcinoma	3.0	1.1a	5.7	1.5c	3.5	1c	2.7	0.8a
Mild liver disease	6.4	4.5	7.3	5.8a	6.3	3.2c	4.5	1.8b
Moderate or severe liver disease	1.9	2.0	2.7	1.5b	2.0	0.9a	1.4	0.6
Myocardial infarction	12.1	10.1	9.4	9.8	9.8	7.9	6.7	6.9
Obesity	16.4	24.8c	14.8	29.8c	17.6	25.5c	10.2	16.3c
Paraplegia and hemiplegia	4.6	2.5	2.3	2.2	2.1	1.8	1.9	1.4
Peptic ulcer disease	1.0	2.0	1.2	0.8	1.5	0.6a	1.1	1.2
Peripheral vascular disease	10.3	6.5a	8.2	5.7c	7.9	6.3	4.0	3.7
Renal disease	19.1	21.1	19.2	24.7c	19.1	23.8a	13.6	13.0
Rheumatic disease	2.9	3.4	3.9	3.7	3.3	2.5	3.7	2.2

CCI = Charlson Comorbidity Index, COVID = coronavirus disease.

ap < 0.05.

bp < 0.01.

cp < 0.001.

Although physicians incorporate many factors into their determination of patient risk, there is a subset of patients for whom risk is not clear-cut. In such instances, reference to an objective acuity-based threshold may help in guiding care decisions. Even in instances when the physician may be uncertain and the acuity-based risk is neither low nor high, but somewhere in the middle (i.e., moderate), an objective score may still be of use, according to circumstances. For example, when beds and resources are scarce (as in a pandemic surge scenario), patients with moderate acuity risk may be treated as lesser risk, and when bed capacity and resources are readily available, a score connoting moderate risk may guide toward closer monitoring or more conservative care decisions out of an abundance of caution.

Given the potential opportunity for acuity model–based risk stratification to provide insight to clinicians, it is important to ask whether the patients retrospectively identified as high risk by the RI were already receiving appropriate and timely care. We note that a large minority of expired COVID-19 patients were never admitted to ICU, although this may have been due to the goals of care for those patients. On the other hand, for patients who were transferred to the ICU, we see that high-risk COVID-19 patients were transferred substantially later in their hospitalization and had a much higher mortality rate than the non–COVID-19 high-risk population. It is possible that if the physician was alerted to the high risk inferred from the initial RI for these COVID-19 patients, there may have been an opportunity to affect outcomes by altering care earlier, whether by more frequent vitals sign measurement, placement in higher level of care, or other means of increased vigilance.

Perhaps equally important, the low-risk group accounted for nearly half of all hospitalized COVID-19 patients and had a substantially lower rate of in-hospital mortality, discharge to hospice, ICU utilization, and mechanical ventilation than the high-risk group. The vast majority (95–97%) of low-risk patients did not expire and had approximately half the LOS of the nonexpiring high-risk population.

Identifying low-risk patients may facilitate decisions to transfer or discharge these patients into lower or nonacute care settings and free hospital and ICU bed capacity for use with higher risk patients. Especially during surge situations, an objective measure may help ensure that resources, supplies, and bed capacity are allocated efficiently. This applies to high-risk patients who may benefit from closer monitoring or more intensive therapies and to low-risk patients, some of whom may not need high levels of care or indeed hospitalization at all.

Robustness, Dynamics, and Practical Considerations

The majority of scores in the public domain which have been evaluated for use with COVID-19 patients, for example, quick Sequential Organ Failure Assessment, NEWS, CURB-65, and their variants (25–28), are based on a limited number of clinical inputs, simple algorithms, and lack a large finely graduated dynamic range. On the other hand, although sophisticated models developed specifically for COVID-19 have proliferated, concerns about their generalizability, robustness, and utility persist, with a recent systematic review unable to recommend a single one for use in practice (29).

Our approach is in essence a middle ground: applying a well-documented, feature-rich acuity score, which nevertheless uses readily available data and has a track record of straightforward implementation at numerous U.S. hospitals. Applying a complex model such as the RI in a simple manner by using cut points avoids problems such as training set artifacts, overfitting, or failure to account for a dynamic care and patient environment, that can arise when model training on a static data set. As our precision recall curves illustrate, alternative cut points could have been chosen. However, the method’s generalizability is shown by similar risk stratification—for identifying both high and low risk patients—using the same cut points at four geographically distinct hospitals with different characteristics: large/small, urban/suburban, academic/community. Even in the face of declining mortality rates, the need for appropriate treatment has not diminished, and the need to know early-on which patients would or would not benefit from more aggressive therapies or warrant closer monitoring remains unaltered.

Limitations

This work was a retrospective analysis of COVID-19 patients, and as such presupposes knowledge of whether a patient was accurately diagnosed with COVID-19. This work does not account for possible coding error nor uncertainty arising from the possibility of false-positive or false-negative diagnostic laboratory tests. Additionally, care of COVID-19 patients evolved rapidly during the time period spanned by our study data, and the use of treatments such as dexamethasone, remdesivir, convalescent plasma, monoclonal antibodies, proning, and anticoagulation therapies changed both during and subsequent to the period of our study data (30–36) as did perspectives on the benefits of early, aggressive intubation (37, 38). During COVID-19 surge conditions, particularly at Yale New Haven Hospital, it may be that some patients were not moved to the ICU due to a more widespread acceptance of palliation, and lack of ICU use may therefore reflect changes in care processes rather than an underrecognition of severity of disease process. Without chart review and more complete contextual data, it is not possible to know what patient goals of care may have been and the extent to which they influenced decisions related to treatment or to receiving care in the ICU. Similarly, we note that identifying high- and low-risk patients is not synonymous with identifying patients who are most or least likely to benefit from treatment which requires a contextual evaluation of individual patient factors and goals. Only via a prospective study can the impact of this acuity score–based risk triage approach be demonstrated.

CONCLUSIONS

A direct measure of acuity at the time of admission is shown to be an effective means for identifying patients at risk of significant deterioration, stratifying the risk of critical deterioration in COVID-19 patients. We found that the relationship of comorbidity burden and acuity to mortality is different for COVID-19 and non–COVID-19 patients. COVID-19 patients experience a significantly higher risk of mortality compared with non–COVID-19 medical (respiratory or nonrespiratory) patients at the same level of acuity. This insight may affect treatment decisions, particularly for those COVID-19 patients presenting with modest levels of acuity, where it is not clear what the appropriate level of care should be. Selected acuity thresholds are shown to be a generalizable and robust means of identifying high- and low-risk COVID-19 patients. Use of such an acuity measure at the time of admission may assist front-line clinicians in aligning level of care decisions with hospital and ICU capacity constraints, ensuring that patients are placed appropriately and limited resources are allocated efficiently.

ACKNOWLEDGMENTS

We extend our thanks to Christine Sullivan and Lynn Lewis of Sinai Hospital of Baltimore for their help to make this work both feasible and useful and to Dr. Robert Fogerty of Yale New Haven Hospital for his thoughtful insights and suggestions. We also want to thank the Yale New Haven Health System, LifeBridge Health, the University of California Irvine Medical Center, and PeraHealth, Inc., for support of the researchers involved in the project.