Identifying patients with symptoms suspicious for COVID-19 at elevated risk of adverse events: The COVAS score.

OBJECTIVE: Develop and validate a risk score using variables available during an Emergency Department (ED) encounter to predict adverse events among patients with suspected COVID-19.
METHODS: A retrospective cohort study of adult visits for suspected COVID-19 between March 1 - April 30, 2020 at 15 EDs in Southern California. The primary outcomes were death or respiratory decompensation within 7-days. We used least absolute shrinkage and selection operator (LASSO) models and logistic regression to derive a risk score. We report metrics for derivation and validation cohorts, and subgroups with pneumonia or COVID-19 diagnoses.
RESULTS: 26,600 ED encounters were included and 1079 experienced an adverse event. Five categories (comorbidities, obesity/BMI ≥ 40, vital signs, age and sex) were included in the final score. The area under the curve (AUC) in the derivation cohort was 0.891 (95% CI, 0.880-0.901); similar performance was observed in the validation cohort (AUC = 0.895, 95% CI, 0.874-0.916). Sensitivity ranging from 100% (Score 0) to 41.7% (Score of ≥15) and specificity from 13.9% (score 0) to 96.8% (score ≥ 15). In the subgroups with pneumonia (n = 3252) the AUCs were 0.780 (derivation, 95% CI 0.759-0.801) and 0.832 (validation, 95% CI 0.794-0.870), while for COVID-19 diagnoses (n = 2059) the AUCs were 0.867 (95% CI 0.843-0.892) and 0.837 (95% CI 0.774-0.899) respectively.
CONCLUSION: Physicians evaluating ED patients with pneumonia, COVID-19, or symptoms suspicious for COVID-19 can apply the COVAS score to assist with decisions to hospitalize or discharge patients during the SARS CoV-2 pandemic.

Introduction

COVID-19 is a pandemic with worldwide impact which has depleted healthcare resources in many areas [1,2]. Due to constraints in the availability of testing [3] and the rapid spread of the disease, data are limited about which patients are at highest risk of clinical decompensation. Presenting symptoms of fever, cough, dyspnea and fatigue have been consistently reported from initial reports from China, Italy, and now the United States [[4], [5], [6], [7], [8], [9], [10]]. Initial reports describe risk factors and outcomes for admitted patients, but lack information about outcomes for patients outside of the hospital. Frontline physicians who make key decisions about disposition would benefit from rules or models to help them decide who requires hospitalization and who can be safely discharged to home. This is particularly challenging for clinicians who do not have SARS-CoV-2 testing results available at the time hospitalization decisions are made. Historically, scores for bacterial pneumonia, such as the pneumonia severity index (PSI) or CURB-65 have shown to improve the quality and efficiency of healthcare by objectively informing physicians about patient risks of death within 30-days [[11], [12], [13], [14], [15]]. Given the unique clinical features of COVID-19, there is a need for a novel risk score using data commonly available at the time of the Emergency Department (ED) encounter, in order to inform clinical decisions regarding the disposition of patients with symptoms suspicious for COVID-19.

Substantial surges of infected patients have saturated limited resources (hospital beds, ventilators, etc.) [2] in certain areas of the United States. As a result, this pandemic has changed the flow and availability of usual routine care, even among areas with low COVID-19 prevalence as a result of preparation for potential surges. Usual acute care is currently hampered by the fear that health care workers have of contracting COVID-19 [16], limited personal protective equipment [17], and healthcare work flow changes [[18], [19], [20]]. These circumstances impact the ability of health professionals to even use some of the diagnostic tests found predictive of adverse COVID-19 outcomes [21] in hospitalized patients [22]. Many of these labs may have limited use in the ED setting due to resource limitations, test availability and long turn-around-times for many results. If a prediction score could be derived with clinical variables readily available at presentation, it would be far more useful for physicians and health systems to evaluate the millions of patients with acute symptoms suspicious for COVID-19.

Our study addresses the need [23] to inform frontline health professionals and health systems about which patients are at greatest risk of serious adverse events (death or respiratory decompensation requiring invasive mechanical ventilation, non-invasive ventilation, high-flow oxygen, or oxygen delivered via face mask) among those presenting with pneumonia or other symptoms suspicious for COVID-19 (fever, cough, dyspnea, fatigue, etc.) during the pandemic. This is especially needed during a time when rapid SARS CoV-2 testing is not available for most ED patients. Therefore, in this study we developed and validated the COVAS risk score using patient characteristics associated with adverse outcomes (comorbidities, obesity/BMI, vital signs, age and sex) [9,10] to predict a patient's risk of a serious adverse event within 7-days of the ED encounter.

Methods

Setting and participants

We conducted a study of all adult ED encounters at 15 community hospitals between March 1, 2020 to April 30, 2020. These dates cover a period of time when substantial community spread of COVID-19 was prevalent in our region [24] but prior to general availability of point-of-care testing results at the time of the ED visit. Study sites were all part of a Southern California integrated health system providing health care to over 1 million ED patients per year (study sites ranging from ≈25,000 to 95,000 ED visits per year). Among all ED visits, approximately 80% are made by health plan members. The integrated system allows for unique data capture of all clinical information included in the electronic health record (EHR) for in-network visits, and for claims data for utilization outside of the integrated system.

ED patient encounters were included for adult (≥ 18 years) patients who had an ICD-10 diagnosis for pneumonia, suspected or confirmed COVID-19 (COVID-19 disease, suspected COVID-19, Exposure to coronavirus or screening for COVID-19) or a symptom suspicious for COVID-19 including cough, fever, fatigue, dyspnea, vomiting, diarrhea, pharyngitis, acute upper respiratory infection, or influenza-like illness (eTable 1). We excluded patients who were not members of the integrated health plan because we do not have accurate comorbidity information for these patients, nor accurate follow-up for outcome information when discharged from the ED. Only the index encounter, or first eligible ED visit, was included in the study sample used to derive and validate the risk score, while subsequent visits were included as outcomes. From the final study sample, we took a stratified random 80% sample for the derivation cohort and reserved the remaining 20% as the validation cohort.

Criterion standard for COVID-19

The criterion standard to diagnose COVID-19 is a positive SARS-CoV-2 polymerase chain reaction (PCR) test. Our study EDs, like most community EDs, have limited access to rapid PCR testing, therefore most PCR test results are not available for at least 12–24 h requiring ED physicians to make a clinical diagnosis of COVID-19 prior to making a decision to hospitalize or discharge the patient. Additionally, patients with COVID-19 report a variety of symptoms that manifest from infection with the virus [[4], [5], [6],8,10], therefore any patient with a COVID-19 diagnosis, pneumonia, or the infectious symptoms attributed to COVID-19 (eTable 1) were included in the derivation and validation of the risk score. The score was then tested among subgroups of patients with an initial ED ICD-10 COVID-19 or pneumonia diagnosis, as well as among patients who had a positive SARS-CoV-2 PCR test within 7-days of the encounter.

Outcome measurements

Our primary outcome was a composite measure of death or respiratory decompensation defined as receipt of mechanical ventilation, non-invasive ventilation, high-flow oxygen or oxygen delivered via face mask, within 7-days (ascertained from mortality files, inpatient flowsheet records and CPT/ICD-10 codes; eTable 2) of ED presentation. Secondary outcomes included positive test results for SARS CoV-2 within 7-days of encounter, hospitalizations (inpatient or observation status), and return ED visits or hospitalizations after discharge from the index ED visit.

Exposures and statistical analysis

To develop a parsimonious prediction model using information available at the point of care in the ED, we collected data about demographic characteristics, vital signs and comorbid conditions. For comorbidities, we collected ICD-10 codes from the Elixhauser Index (diabetes, hypertension, cancer, etc.) [25,26]. We used clinical judgement to identify other candidate variables available at the time of triage for ED visits [[8], [9], [10],16,22]. We did not include laboratory test results, because most patients with suspected COVID-19 did not have such testing, and the receipt of testing likely signals that the patient had been identified as seriously ill by the treating physician.

We included 42 variables in a least absolute shrinkage selection operator (LASSO) regression model to minimize collinearity and to avoid over-fitting. [27,28] The LASSO model with the smallest predicted residual sum of squares was selected based on 10-fold cross validation. We next included all variables from the optimal LASSO regression in a logistic regression model [22]. Those variables which remained significant were included in the final risk score. To simplify the clinical score among common variables, we combined the Elixhauser diabetes and hypertension categories (complicated and uncomplicated) for analysis. We used standard National Institutes of Health (NIH) body mass index (BMI) categories [<18.5 (underweight), 18.5–24 (normal), 25–29 (overweight), 30–34 (obese class I), 35–39 (obese class II), and ≥ 40 (obese class III, or extreme obesity) [29]. Vital sign thresholds were informed based on histograms stratified by deciles, but primarily were informed based on current clinical recommendations and the clinical judgement among co-authors. Age was categorized by decade, but given small sample sizes among extreme older ages, those ages above the highest significant age group were clustered into the highest significant category. We performed 10-fold cross validation to obtain stable parameter estimates for the variables in the logistic regression. Standardized scores were created for each variable in the logistic regression model by dividing each variable coefficient by the smallest coefficient. For ease of use, variables were clustered into 5 categories including comorbidities, obesity/BMI, vital signs, age and sex (COVAS).

To assess discrimination, the area under the curve (AUC) was calculated, and a priori we determined that a final score with an AUC ≥ 0.80 would have meaningful accuracy to guide disposition decisions among suspected COVID-19 patients in the ED. [11,14,30,31] We calculated sensitivity, specificity, positive predictive values (PPV), negative predictive values (NPV) at each threshold value for the COVAS score from ≥1 to ≥15. Positive likelihood ratios (PLR) and negative likelihood ratios (NLR) were also computed for each possible score. To assess model calibration, we evaluated plots of the observed and predicted values and calculated the Brier score. The final score was validated in a separate data set representing 20% of the study sample. We report the same performance metrics in the validation cohort as reported in the derivation sample.

Descriptive tables report the percentage of patients with a 7-day adverse event at each risk score. We also performed subgroup analyses and assessed risk score performance among patients with a pneumonia diagnosis, or among those with a COVID-19 confirmed diagnosis at the index ED encounter. Last, we describe the number of patients hospitalized and discharged stratified by the primary outcome.

All analyses were performed using SAS (version 9.3; SAS Institute, Inc., Cary, NC). All tests of statistical significance were 2-sided with α = 0.05. This study was approved by the [Institution name blinded for review] Institutional Review Board.

Results

The study sample included 26,600 ED patient encounters (21,280 derivation and 5320 validation cohorts) (eFig. 1). In the derivation cohort, the mean age (SD) was 50.7 (19.6), 9053 (42.5%) were men, and 1811 (8.5%) were severely obese (BMI ≥ 40) (Table 1 ). The subgroups included 2620 (12.3%) patients with pneumonia and 4465 (21.0%) patients with a COVID-related diagnosis. ED encounters resulting in an adverse event within 7-days represented 863 (4.1%) patients (177 died and 686 with respiratory decompensation) and 4226 (19.9%) were hospitalized. Among the relatively small number who received a COVID-19 test (N = 6178), 1105 (17.9%) had a confirmed COVID-19 diagnosis within 7-days (eTable 3). The validation cohort represented a comparable sample (eTables 3 and 4).

Table 1

Patient characteristics of derivation cohort. Adverse events were defined as any need for high-flow mask, high-flow nasal cannula, non-invasive ventilation, invasive mechanical ventilation or death within 7-days.

	Total N = 21,280		No Adverse Event N = 20,417		Adverse Event N = 863
	N	%	N	%	N	%
Age
Mean (SD)	50.7 (19.6)		50.1 (19.4)		65.9 (16.6)
Median (IQR)	50 (34, 66)		49 (34, 65)		68 (56, 78)
Age group
<50	10,595	49.8	10,447	51.2	148	17.2
50–59	3439	16.2	3321	16.3	118	13.7
≥60	7246	34.1	6649	32.6	597	69.2
Gender
Female	12,227	57.5	11,864	58.1	363	42.1
Male	9053	42.5	8553	41.9	500	57.9
Race
Asian/PI	2038	9.6	1924	9.4	114	13.2
Black	2834	13.3	2725	13.4	109	12.6
Hispanic	9637	45.3	9279	45.5	358	41.5
White	5953	28.0	5685	27.8	268	31.1
Other	818	3.8	804	3.9	14	1.6
BMI
Underweight <18.5	465	2.2	426	2.1	39	4.5
Normal 18.5–24.9	5163	24.3	4922	24.1	241	27.9
Overweight 25–29.9	6268	29.5	6062	29.7	206	23.9
Obese 30–39.9	6835	32.1	6566	32.2	269	31.2
Severely Obese ≥40	1811	8.5	1705	8.4	106	12.3
Missing	738	3.5	736	3.6	2	0.2
Smoking
Never or Passive	11,823	55.6	11,422	55.9	401	46.5
Quit	4744	22.3	4440	21.8	304	35.2
Active	1071	5.0	1027	5.0	44	5.1
Missing	3642	17.1	3528	17.3	114	13.2
Elixhauser comorbidity score, mean (SD)1	3.1 (3.4)		2.9 (3.2)		7.2 (3.8)
Heart rate, continuous2
Mean (SD)	89.9 (18.8)		89.6 (18.5)		97.8 (23.2)
Median (IQR)	88 (76, 101)		88 (76, 101)		97 (82, 112)
Heart rate, categorical2
<110	17,559	82.5	16,955	83.0	604	70.0
110–120	1809	8.5	1687	8.3	122	14.1
≥121	1318	6.2	1187	5.8	131	15.2
Missing	594	2.8	588	2.9	6	0.7
Systolic BP, continuous2
Mean (SD)	139.2 (22.9)		139.4 (22.6)		134.5 (28.6)
Median (IQR)	137 (124, 152)		138 (124, 152)		132 (116, 150)
Systolic BP, categorical2
>115	17,543	82.4	16,897	82.8	646	74.9
106–115	1766	8.3	1666	8.2	100	11.6
≤105	999	4.7	890	4.4	109	12.6
Missing	972	4.6	964	4.7	8	0.9
Diastolic BP, continuous2
Mean (SD)	81.0 (15.3)		81.2 (15.2)		75.8 (18.0)
Median (IQR)	81 (71, 91)		81 (72, 91)		75 (64, 87)
Diastolic BP, categorical2
≥70	15,971	75.1	15,427	75.6	544	63.0
60–69	2792	13.1	2619	12.8	173	20.1
<60	1545	7.3	1407	6.9	138	16.0
Missing	972	4.6	964	4.7	8	0.9
Oxygen saturation, continuous2
Mean (SD)	97.6 (2.9)		97.7 (2.3)		93.2 (7.7)
Median (IQR)	98 (97, 99)		98 (97, 99)		95 (91, 98)
Oxygen saturation2
≥95	19,057	89.6	18,562	90.9	495	57.4
93–94	867	4.1	769	3.8	98	11.4
≤92	689	3.2	426	2.1	263	30.5
Missing	667	3.1	660	3.2	7	0.8
Respiratory rate, continuous2
Mean (SD)	18.3 (2.9)		18.1 (2.6)		21.5 (6.0)
Median (IQR)	18 (16, 20)		18 (16, 19)		20 (18, 24)
Respiratory rate, categorical2
<20	15,234	71.6	14,868	72.8	366	42.4
20–24	4604	21.6	4280	21.0	324	37.5
≥25	550	2.6	387	1.9	163	18.9
Missing	892	4.2	882	4.3	10	1.2
Fever2	1283	6.0	1137	5.6	146	16.9

Elixhauser comorbidity index based on comorbidities within the year prior.

Vital signs based on presenting measurement.

LASSO regression identified 20 variables that were associated with the primary outcome. These variables were included in a logistic regression model and 15 variables from five categories, were included in the final comorbidity, obesity, vital signs, age and sex (COVAS) score. Electrolyte disorders, cardiac arrhythmias, other neurological disorders, weight loss, congestive heart failure, coagulopathy and diabetes were the seven Elixhauser comorbidities included. Severe obesity (BMI ≥ 40), respiratory rate (20–24 and ≥ 25), oxygen saturation (93–94% and ≤ 92%), systolic blood pressure (≤ 105), fever (temperature ≥ 100.4 F), heart rate (≥ 110), age (50–59 and ≥ 60) and male sex were also included in the score (Table 2 , Figure 1 ). The resulting COVAS score included points ranging from one to seven for each variable, with and an overall possible score from 0 to 34.

Table 2

Multivariate logistic regression in derivation cohort

Parameter	OR (95% CI)	p	Score point value1
Age group (ref ≤50)
50–59	1.67 (1.23–2.26)	0.001	2
≥60	2.02 (1.57–2.61)	<0.0001	3
Male	1.70 (1.43–2.01)	<0.0001	2
BMI (ref = Normal)
Underweight <18.5	0.96 (0.61–1.49)	0.841	0
Overweight 25–29.9	0.80 (0.63–1.01)	0.0569	0
Obese 30–39.9	1.15 (0.92–1.45)	0.2319	0
Severely Obese ≥40	1.81 (1.32–2.48)	0.0002	2
Cardiac Arrhythmia	1.59 (1.32–1.92)	<0.0001	2
Congestive Heart Failure	1.45 (1.18–1.77)	0.0004	1
Coagulopathy	1.53 (1.24–1.89)	<0.0001	1
Diabetes	1.37 (1.14–1.64)	0.0007	1
Fluid and Electrolyte Disorders	2.50 (2.07–3.02)	<0.0001	3
Other Neurological Disorders	1.78 (1.44–2.20)	<0.0001	2
Weight Loss	1.79 (1.43–2.24)	<0.0001	2
Heart rate (ref = 〈110)
110–120	1.44 (1.11–1.86)	0.0059	1
≥121	1.34 (1.02–1.75)	0.0349	1
Systolic BP (ref≥ 115)
106–115	1.26 (0.96–1.65)	0.0961	0
≤105	1.64 (1.25–2.14)	0.0003	2
Oxygen saturation (ref = ≥95)
93–94	1.93 (1.46–2.54)	<0.0001	2
≤92	6.79 (5.43–8.48)	<0.0001	7
Respiratory rate (ref ≤20)
20–24	1.68 (1.39–2.02)	<0.0001	2
≥25	4.08 (3.08–5.41)	<0.0001	5
Fever	2.00 (1.57–2.56)	<0.0001	2

Point value obtained by standardizing the log-scale estimates using the smallest significant estimate (heart rate ≥ 121).

Fig. 1

COVAS Score derived among patients presenting to the ED with symptoms suspicious for COVID-19 during the pandemic (March 1–April 30, 2020).

The AUC in the derivation cohort was 0.891 (95% CI, 0.880–0.901) (Fig. 2 ) with sensitivity ranging from 100% (Score 0) to 41.7% (Score ≥ 15) and specificity from 13.9% (score 0) to 96.8% (score ≥ 15) (eTable 5). A COVAS threshold of ≥5 had 95.1% sensitivity, 56.0% specificity, while a score of ≥11 had 67.8% sensitivity with 89.5% specificity. The Brier score in the derivation cohort was 0.032 demonstrating good calibration (eFigure 2). Similar performance of the COVAS score was observed in the validation cohort (AUC = 0.895, 95% CI, 0.874–0.916; Brier score = 0.031) (eFigures 3 and 4, eTable 6).

Fig. 2

The COVAS score performance for the derivation sample (N = 21,280) among all ED visits with pneumonia or symptoms suspicious for COVID-19 between March 1–April 30, 2020, predicting 7-day death or need for critical respiratory intervention. Subgroups of patients with a COVID-19 diagnosis (N = 4465) and those with pneumonia (N = 2620) are also reported.

In the subgroups of patients who were diagnosed with pneumonia, the COVAS score had an AUC of 0.780 (95% CI 0.759–0.801) in the derivation cohort and 0.832 (95% CI 0.794–0.870) in the validation group. For the ED encounters with a COVID-19 ICD-10 diagnosis, the AUC was 0.856 (95% CI 0.838–0.873) in the derivation sample, and 0.859 (95% CI 0.824–0.893) in the validation group (Figs. 2, eFigure 3). See eTables 6 and 7 for subgroup sensitivities and specificities.

Rates of 7-day adverse events among patients in the derivation cohort were calculated for each COVAS score and ranged from 0% (COVAS score 0) to 35.7% (COVAS score ≥ 15) (Table 3 ). Patients with a COVAS score ≤ 5 had ≤1.5% risk of an adverse event, which may represent a “low-risk” group, and patients with a score ≥ 12 have a ≥ 15% risk representing a “high-risk” cohort. Similar rates were observed for the validation cohort (eTable 8).

Table 3

Adverse events (death or need for critical respiratory intervention) within 7-days of an emergency department visit for symptoms suspicious for COVID-19 from March 1–April 30, 2020, stratified by COVAS score, in the derivation cohort

	Total derivation cohort (N = 21,280)			No adverse event (N = 20,417)		Adverse event (N = 863)
Score	N	Col %	Cumulative Col %	N	Row %	N	Row %
0	2840	13.3	13.3	2840	100.0	0	0.0
1	463	2.2	15.5	461	99.6	2	0.4
2	4052	19.0	34.6	4043	99.8	9	0.2
3	1776	8.3	42.9	1763	99.3	13	0.7
4	2342	11.0	53.9	2324	99.2	18	0.8
5	1969	9.3	63.2	1940	98.5	29	1.5
6	1426	6.7	69.9	1392	97.6	34	2.4
7	1187	5.6	75.4	1155	97.3	32	2.7
8	1021	4.8	80.2	983	96.3	38	3.7
9	785	3.7	83.9	737	93.9	48	6.1
10	689	3.2	87.2	634	92.0	55	8.0
11	582	2.7	89.9	532	91.4	50	8.6
12	460	2.2	92.1	390	84.8	70	15.2
13	374	1.8	93.8	318	85.0	56	15.0
14	306	1.4	95.3	257	84.0	49	16.0
15+	1008	4.7	100.0	648	64.3	360	35.7

Discussion

In this study we derived and validated a risk score (COVAS) to accurately predict death or respiratory decompensation within 7-days among a sample of ED patients with pneumonia or symptoms suspicious for COVID-19. The COVAS score uses variables (comorbidities, obesity/BMI ≥40, vital signs, age and sex) available at the time of acute patient presentation which can inform clinical decisions about which patients may benefit from further diagnostic testing or admission to the hospital. The COVAS score not only performed well in the derivation (AUC 0.891) and validation (AUC 0.895) cohorts, but also in the subgroups of patients with pneumonia (AUC derivation 0.780 and validation 0.832), and most importantly in the subgroup with a COVID-19 related diagnosis at the index ED encounter (AUC derivation 0.856 and validation 0.859).

This study addresses a call for risk scores to guide clinical decisions in the care of patients during the COVID-19 pandemic, using methods that avoid concerns about the quality of previous reports [23]. A strength of our study is the use of a heterogeneous sample which is representative of the types of patients that frontline emergency, urgent care and primary care physicians are currently evaluating, and the 7-day timeline is more relevant than longer periods for acute care decisions. Our study also avoids the current challenges related to the availability of SARS CoV-2 testing, and with the uncertain accuracy of different testing strategies [32], especially at a time when it is challenging to distinguish between COVID-19 and other infections. Targeting an ED patient population adds to the previous reports from hospitalized patient cohorts [8,10,22].

A limitation of this study and the derived COVAS score is the omission of diagnostic laboratory tests which have been associated with COVID-19 severity among hospitalized patients. We acknowledge that including laboratory tests may improve discrimination, but implementation of this approach would require that all patients receive lab tests which may not be necessary nor readily available in a pandemic. Also, the COVAS score may have different results in different patient populations or clinical settings; therefore, future research to validate this score will elucidate its generalizability. However, given the high AUC in our overall sample and subgroups, even a modest reduction in discrimination is likely to be useful. Additionally, the COVAS score does not include social risks that may predict adverse outcomes among at-risk populations, such as those with housing instability or food insecurity [33]. Our study population uses only the initial ED patient encounter for risk assessment, therefore, patients with a return visit may have slightly different risks for an adverse event. Our patient population is also diverse racially and ethnically (45% Hispanic and 13% black) which is a strength of our study, but the COVAS score may have different results in more homogenous populations. Lastly, calculating comorbidities based on ICD codes in the preceding 12 months may pose challenges to physicians calculating the score without electronic assistance. Therefore, we recommend using decision support within an electronic health record to facilitate the accurate calculation of the COVAS score.

The COVAS score discriminates as well, or better, than many scores used in routine clinical practice to inform acute care decisions. Commonly used clinical risk scores like the pneumonia severity index (AUC 0.81), CRB65 (AUC 0.79) and CURB-65 (AUC 0.80), which are routinely used in clinical care [[11], [12], [13]] to predict 30-day mortality [15], are less accurate than the COVAS score for their specific indications. Of note, a more complicated score with many more variables, including a number of lab test results was not as accurate for predicting adverse outcomes among Chinese patients hospitalized with COVID-19 [22].

Since the derivation of the COVAS score, all 15 EDs included in this study implemented the score into routine practice using an automated decision support calculation within the EPIC based electronic health record. From the time of our reported study, the COVAS score was further validated among ED patients who were tested for COVID-19 during a time in Southern California when the incidence and prevalence increased among the study EDs. This additional validation among our study EDs using the same study population in between May and June 2020 resulted in an AUC of 0.822 (95% CI 0.811–0.833), and with much increased capacity to test it performed well among those tested via SARS CoV-2 PCR (n = 18,379, AUC 0.792, 95% CI 0.780–0.805) and was best among those whose COVID-19 test was positive (n = 2091, AUC 0.841, 95% CI 0.812–0.869).

Though the COVAS score requires future research to validate its performance among different patient populations in different health care settings, we believe it provides useful information to frontline physicians to assist with risk stratification and disposition decisions. A challenge for any risk score is choosing the appropriate threshold (sensitivity and specificity) to guide recommendations for patient care, because both the frequency and consequences of adverse events need to be considered. Using information about adverse events in conjunction with COVAS score sensitivity/specificity tables, physicians and systems can choose different thresholds for categorizing patients into low, moderate and high-risk groups. This may inform disposition decisions, in the same way other risk scores have been applied in the ED. [34] One potential application of the COVAS score may be to objectively identify a moderate risk group of patients who may safely avoid hospitalization, but who may benefit from home O2 monitoring and oxygen supplementation.

In summation, among patients visiting an ED for confirmed or suspicious COVID-19 symptoms, the COVAS score, using information available at the time of patient presentation, may help frontline physicians to identify patients who will experience a serious adverse event within 7-days. Future research is needed to further validate this score in other patient populations and clinical settings.

Author contributions

ALS and MKG conceived the study and obtained research funding. BZH collected and analyzed the data. CN and ST assisted with statistical advice. All included authors assisted with the study design and interpretation of results. ALS drafted the manuscript, and all authors contributed substantially to its revision. ALS takes responsibility of the manuscript as a whole.

Funding source

This research was supported by a grant from the Regional Research Committee of Kaiser Permanente Southern California. Grant No.: KP-RRC-20200401