The COVID-19 Worsening Score (COWS)-a predictive bedside tool for critical illness.

OBJECTIVES: To evaluate the accuracy of a new COVID-19 prognostic score based on lung ultrasound (LUS) and previously validated variables in predicting critical illness.
METHODS: We conducted a single-center retrospective cohort development and internal validation study of the COVID-19 Worsening Score (COWS), based on a combination of the previously validated COVID-GRAM score (GRAM) variables and LUS. Adult COVID-19 patients admitted to the emergency department (ED) were enrolled. Ten variables previously identified by GRAM, days from symptom onset, LUS findings, and peripheral oxygen saturation/fraction of inspired oxygen (P/F) ratio were analyzed. LUS score as a single predictor was assessed. We evaluated GRAM model's performance, the impact of adding LUS, and then developed a new model based on the most predictive variables.
RESULTS: Among 274 COVID-19 patients enrolled, 174 developed critical illness. The GRAM score identified 51 patients at high risk of developing critical illness and 132 at low risk. LUS score over 15 (range 0 to 36) was associated with a higher risk ratio of critical illness (RR, 2.05; 95% confidence interval [CI], 1.52-2.77; area under the curve [AUC], 0.63; 95% CI 0.676-0.634). The newly developed COVID-19 Worsening Score relies on five variables to classify high- and low-risk patients with an overall accuracy of 80% and negative predictive value of 93% (95% CI, 87%-98%). Patients scoring more than 0.183 on COWS showed a RR of developing critical illness of 8.07 (95% CI, 4.97-11.1).
CONCLUSIONS: COWS accurately identify patients who are unlikely to need intensive care unit (ICU) admission, preserving resources for the remaining high-risk patients.

INTRODUCTION

By the beginning of 2020, a novel disease called COVID‐19 was recognized and eventually defined as a pandemic by the WHO. The disease‐causing virus, known as SARS‐CoV‐2, with its high tropism for the lower respiratory tract, can produce an infection with a broad spectrum of symptoms ranging from asymptomatic to severe acute respiratory failure, often requiring intensive care unit (ICU) admission.

Since the beginning of the pandemic, many healthcare facilities reorganized entire departments where multidisciplinary teams collaborated to provide care for COVID‐19 patients. Massive effort from the worldwide medical community has been put forth to better understand the pathophysiology of this disease, in order to provide appropriate care, optimize hospital resources, and increase efficiency of workflow. In this context, the availability of an easy‐to‐use standardized scoring system would have been of great help in supporting clinicians with different backgrounds to better identify patients at higher risk of developing a critical illness. Aiming to provide means for a better resource allocation, several prediction models have been developed over the last few months. Vital parameters, comorbidities, and blood test results have been combined to predict disease severity and outcomes for hospitalized COVID‐19 patients. , , , , , , , ,

Among them, Liang et al developed the COVID‐GRAM score, which showed success in the early prediction of critical illness development, defined as admission to the ICU, need for invasive mechanical ventilation (IMV), or death. However, the GRAM score requires ten independent variables, including laboratory results and chest X‐ray and requires online calculations to risk stratify patients. Despite its accuracy, its use could be time‐consuming as not all required parameters are readily available in all settings. In fact, during the first pandemic peak, healthcare facilities experienced an unexpected patient influx to the emergency department (ED) and medical wards with an average of 60 to 80 COVID‐19 patients per hour. Based on this very early Italian experience, such patient influxes made serial radiological imaging unfeasible. For this reason, a less burdensome and rapid prognostic score may be of considerable benefit.

Several of the above‐mentioned prognostic scores integrated radiological data (ie, chest X‐ray or CT scan), but no study has yet investigated the performance of lung ultrasound (LUS) as a prognostic tool in COVID‐19 patients. LUS is available at the patient's bedside, and its reliability and speed as a tool to evaluate acute respiratory disorders in real‐time have been well established. , Moreover, COVID‐19 has a distinctive distribution pattern involving mainly the peripheral and lower regions of the lungs, and presumably this is why LUS demonstrated superior sensitivity to CT scan for pleural and subpleural abnormalities. According to the available literature and contingent need, LUS may play a central role in this pandemic where the risk of healthcare workers’ exposure and patients’ overflow has been a primary concern.

We hypothesized that a new prognostic score, integrating previously validated variables and LUS findings instead of chest radiography, could work as well as the GRAM score for the early identification of COVID‐19 patients developing critical illness. Hence, we firstly tested the GRAM score on our cohort and then developed and internally validated the new COVID‐19 Worsening Score (COWS).

METHODS

Study design

We conducted a single‐center retrospective cohort validation study of the GRAM score and subsequently developed and internally validated a new prognostic score.

The study adhere to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines.

Study population and Setting

The study was conducted in an Italian tertiary Hospital in Turin (San Giovanni Bosco Hospital). All adult patients with a confirmed diagnosis of SARS‐CoV‐2 infection admitted to the ED and thereafter to the medical wards during the epidemic peak between February 26 and May 17 were enrolled. Patients with hospital‐acquired COVID‐19, previous pneumonectomy, or lobar pneumonia on presentation were excluded. SARS‐CoV‐2 infection disease was confirmed by real‐time polymerase chain reaction (RT‐PCR) performed either on nasal swab or on pharyngeal swab. The patients' notes and imaging results were retrieved from electronic medical records, collected in a dedicated COVID‐19 database, and retrospectively analyzed. The City of Turin Ethical Committee approved the study on June 3, 2020 (protocol #82995). The hospital review board waived patients' consent due to the retrospective nature of the study and anonymous data handling and analysis.

Patients characteristics and clinical outcomes

Patient demographic characteristics, comorbidities, presenting symptoms and date of their onset, clinical signs, laboratory test results, and sonographic and radiological findings (chest X‐ray and/or CT) were collected within 48 hours of ED admission. The arterial oxygen partial pressure to fractional‐inspired oxygen (P/F) ratio was also recorded.

The adverse outcome referred to as critical illness in the results section was defined by the occurrence of at least one of the following three events: admission to ICU, need for invasive mechanical ventilation (IMV), or death , due to COVID‐19 within a follow‐up of 30 days postadmission. Supplementary oxygen support or noninvasive ventilation (NIV) was considered favorable outcomes. Need for IMV and ICU admission was decided based on standard of care criteria.

Variables selection

Among the patients’ collected data, we selected the ten variables previously identified in the GRAM score. We chose these ten variables due to their ability to predict the severity of respiratory failure and progression to critical illness. Moreover, P/F ratio on admission and number of days from symptoms onset were included in the analysis. Missing data were further searched in available materials such as handover and notes. In patients that underwent a CT scan, we considered the following findings: the number of pulmonary lobes involved the presence of emphysema and the percentage of well‐aerated lung. These radiological features were predictors of ICU admission or death in COVID‐19 in a previous study.

CT scans (obtained by 64 Slice Discovery HD 750 CT Scanner, General Electric) and chest X‐rays were analyzed by a radiologist with more than ten years of chest imaging experience blinded to patients' outcomes.

The LUS protocol adopted for the study was comprehensive of 6 scanning areas per hemithorax as previously described. Each hemithorax was assessed in one upper and one lower area in the three regions divided by the parasternal, anterior, and posterior axillary lines, respectively. The image focus was placed at the level of the pleural line maintaining the image depth at 8‐12 cm. An already validated aeration score was assigned to each area, and the final LUS score was calculated as the sum of them.

LUS evaluation was performed by 29 clinicians with more than five years of experience in bedside sonographic imaging, and 7 of them subsequently calculated the lung aeration score on all included patients (EB, MC, GL, MC, AG, GF, SS). When in doubt, a second operator (EB) reviewed both imaging and score.

Low‐ to high‐medium frequency (2‐9 MHz) curvilinear probes and three different ultrasound machines were selected for the study (MyLab 5™, MyLab 7™; Esaote, and Sonosite M‐Turbo™ Ultrasound System, Fujifilm).

Statistical analysis

Continuous variables were reported as mean and standard deviation (SD) or median with interquartile range (IQR) as appropriate and categorical variables as numbers and percentages.

Evaluation of the LUS score as a predictor of the adverse evolution of COVID‐19 infection was assessed by univariate level. Restricted cubic splines were modeled to assess the nonlinear effect, and significance was tested by the Wald chi‐square. Significance level was set at 0.05. Finally, the LUS score was dichotomized by the ROC curve analysis. Application of the COVID‐GRAM model on our sample was carried out to evaluate its performance in classifying high‐ and low‐risk patients according to the threshold identified by the ROC curve analysis. The evaluation of the COVID‐GRAM added with the LUS score was then performed.

Aiming to develop a novel and easy to use prognostic score, a selection strategy based on Bayesian model averaging was adopted. The number of comorbidities, LUS score, P/F ratio, dyspnea, and duration of symptoms (days) showed a posterior probability of inclusion greater than 30% and was retained in the final logistic regression model labeled as COWS. Thirty percent was chosen as the cutoff through sensitivity analysis to maximize the bootstrapped predictive accuracy of the selected model.

The performance of the model was assessed in terms of Somers concordance index Dxy (the closer to 1, the better), Brier score (scores closer to zero indicate a better prediction), and calibration slope. An internal validation to correct measures of predictive performance for optimism (over‐fitting) was performed by bootstrapping 500 samples of the data.

To improve the prediction, a shrinkage bootstrap‐based method was applied to re‐estimate regression coefficients. The overall optimism across all models was estimated deriving a shrinkage coefficient equal to the average calibration slope from each of the bootstrap samples. The shrinkage coefficient was applied to the original coefficient to account for over‐fitting. Finally, the intercept was re‐estimated based on the shrunken coefficients to ensure the overall calibration was maintained, producing the final model. All analyses were carried out using R 4.0.0.

RESULTS

Between February 26 and May 17 2020, 274 COVID‐19 patients were admitted to the wards from the ED (Figure 1). Baseline clinical characteristics are summarized in Table 1. One hundred and seventy‐four patients had a final adverse outcome (critical illness), while 100 patients had a favorable outcome (noncritical illness). Complete data for the study analysis, including LUS findings, were available in 143 cases. The mean time between ED admission and outcome was 5.1 days (SD, 5.4; median 3.8; IQR, 1‐7).

FIGURE 1

Diagram of included patients

TABLE 1

Demographics and clinical characteristics of the patients who did and did not develop critical illness

Characteristic	Total (n = 274)	No critical illness (n = 174)	Critical illness (n = 100)	P‐value
Age, mean (SD) [range]	67.7 (14.4) [21‐96]	64.9 (14.5) [21‐96]	72.6 (12.8) [35‐89]	<.000
Gender, male (%)	189 (69.0)	117 (67.2)	72 (72.0)	.412
Days from symptom onset, mean (SD) [range]	5.8 (4.4) [0‐31]	6.4 (4.8) [0‐31]	4.8 (3.5) [0‐14]	.009
Number of comorbidities	(n = 268)	(n = 171)	(n = 97)	<.000
0	71 (24.2)	56 (32.7)	15 (15.5)
1	71 (24.2)	53 (31.0)	18 (18.6)
2	58 (19.8)	31 (18.1)	27 (27.8)
3	38 (13.0)	20 (11.7)	18 (18.6)
4	22 (7.5)	8 (4.7)	14 (14.4)
5+	8 (2.7)	3 (1.8)	5 (5.1)
Malignancy (%)	20 (7.4)	11 (6.4)	9 (9.3)	.387
Dyspnea (%)	139 (51.7)	75 (43.9)	64 (65.3)	.001
Hemoptysis (%)	2 (0.74)	1 (0.58)	1 (1.02)	.597
Unconsciousness (%)	2 (0.74)	—	2 (2.04)	.132
Abnormal chest radiography/CT (%)	201 (82.4)	124 (77.0)	77 (92.8)	.002
LUS score at admission, mean (SD) [range]	(n = 211) 13.4 (7.6) [0‐27]	(n = 146) 12.1 (7.4) [0‐27]	(n = 65) 16.2 (7.3) [0‐27]	<.000
PF ratio at admission, Mean (SD) [range]	(n = 245) 263.9 (94.6) [33‐647]	(n = 164) 297.5 (78.3) [50‐647]	(n = 81) 196 (88.4) [33‐396]	<.001

Performance of GRAM score in this cohort

Necessary data for GRAM score calculation were available in 183 patients. Using the published threshold (40%) for the GRAM score to discriminate between high‐ and low‐risk patients, we identified 51 patients at high risk and 132 at low risk of developing critical illness (Figure 2). When applied to the 143 patients who were integrated in the final analysis, no difference in GRAM score performance was found.

FIGURE 2

GRAM score derived risk groups (on the left) and outcomes (on the right); gray shadows link classification to outcomes and their width is proportional to the number of patients

LUS score as a predictor of the main outcome

LUS images were successfully obtained in 211 patients. LUS aeration score ranged from a minimum of 0 to a maximum of 27, with a mean of 12.1 (SD, 7.4; median, 12) in favorable outcome patients and a mean of 16.2 (SD, 7.3; median, 17) in critically ill patients (P < .001). A higher LUS score was associated with a higher risk ratio (RR) of developing critical illness (RR, 2.05; 95% CI 1.52‐2.77) (Figure 3). A value over 15 (out of 0 to 36) on LUS score demonstrated predictive discrimination between favorable and adverse outcomes (area under the curve [AUC], 0.63; 95% CI 0.676‐0.634).

FIGURE 3

Probability of developing critical illness (Y‐axis) according to increasing values of LUS score (X‐axis)

Performance of GRAM score powered by LUS

As an intermediate analysis, we investigated whether the combination of the dichotomous LUS score with the COVID‐GRAM score could increase the performance of GRAM score alone in predicting adverse outcome. We named this combined score GRAM‐PLUS (GRAM powered by LUS). This calculation was performed in 143 patients based on the available data. The addition of sonographic findings to the GRAM score slightly reduced the number of patients of the low‐risk category as initially established by the GRAM score and raised the RR from 3.0 to 3.18 (Table 2, Figure 4).

TABLE 2

Comparative performance of the three scores (GRAM, GRAM‐PLUS, and COWS) on 143 patients with available data

	High risk			Low risk			RR (95% CI)	P‐value
	Critical illness N (%)	Favorable outcome N (%)	Total	Critical illness N (%)	Favorable outcome N (%)	Total
GRAM score	22 (56.4)	17 (43.6)	39	19 (18.3)	85 (81.7)	104	3.0 (2.02‐4.09)	<.001
GRAM‐PLUS	24 (54.5)	20 (45.5)	44	17 (17.2)	82 (82.8)	99	3.18 (2.07‐4.25)	<.001
COWS	35 (58.3)	25 (41.7)	60	6 (7.2)	77 (92.8)	83	8.07 (4.97‐11.1)	<.001

FIGURE 4

Distribution curves of the patients who developed critically illness (red dots) and those who had favorable outcomes (green dots). X‐axis: linear predictor; Y‐axis: incremental values of GRAM score (upper panel) and GRAM‐PLUS values (lower panel). LUS: lung ultrasound

Of note, when GRAM‐PLUS was validated in our cohort of patients, unconsciousness and hemoptysis were ignored as these signs were absent. The optimism‐adjusted model accuracy index was 0.5193 providing an estimated accuracy of 75.97%.

Performance of COVID‐19 Worsening Score (COWS)

By using the Bayesian averaging model, we selected five predictive variables with their relative coefficients as follow: LUS score greater than 15, the number of comorbidities, days from the symptom onset, dyspnea at presentation, and P/F ratio (Table 3).

TABLE 3

Most predictive variables identified and their effect with 95% confidence intervals

Variable	Effect	95% CI			P‐value
Number of comorbidities	1.688	1.216	—	2.344	.002
LUS score above 15	3.511	1.283	—	9.612	.015
PF ratio	0.218	0.109	—	0.434	<.001
Days from symptom onset	0.595	0.340	—	1.041	.069
Dyspnea	0.308	0.097	—	0.976	.045

COWS ranged from 0 to 1, and the optimal accuracy was identified at a threshold of 0.183. Using this threshold, the same 143 patients were reclassified in 60 high‐risk patients, of whom 35 (58.3%) developed critical illness, and 83 low‐risk patients, of whom 6 (7.2%) developed critical illness (Figure 5). Sensitivity and specificity for critical illness were 0.85 (95% CI, 0.75‐0.96) and 0.75 (95% CI, 0.67‐0.84), respectively. Positive predictive value (PPV) and negative predictive value (NPV) were 0.58 (95% CI, 0.46‐0.71) and 0.93 (95% CI, 0.87‐0.98), respectively. The risk ratio increased to 8.07 (95% CI, 4.97‐11.1) (Table 2). The equation to calculate COWS is shown (below) in Formulas 1 and 2. Finally, we created a nomogram that can be used to calculate COWS manually (Figure 6).

FIGURE 5

Performance of the COWS in classifying high‐ and low‐risk patients. Red dots indicate patients with adverse outcome. Dashed line refers to the COWS threshold

FIGURE 6

Nomogram of COVID‐19 Worsening Score and how to use it

Risk model formula

Formulas , and , . Linear predictor calculation and subsequent COW score calculation. LP = linear predictor; C = number of comorbidities; PF = PF ratio (mmHg); S = days from symptoms onset; L = LUS score; D = dyspnea.

Potential role of CT scan in reclassifying false‐positive and ‐negative patients

After visual inspection of the COWS classification in high‐ and low‐risk patients, we wondered whether second level radiological imaging, such as a thoracic CT scan, might be a viable means to improve the prediction of patients' outcome. Among the 143 patients with complete data, 46 were misclassified by COWS (ie, they turned out to be false‐positive or false‐negative). CT scan data were available only for 59 patients, of whom 55 presented completed data. No statistically significant results were found between the 39 patients incorrectly classified in the high‐risk group compared to the 7 incorrectly classified in the low‐risk group. The number of involved lobes was greater in high‐risk than in the lower risk group (mean 5 vs 2.5; P = .43) as well as the percentage of emphysema (46.7% in the higher risk vs 0% in the lower risk group; P = .485). Percentage of well‐aerated lung was also lower in high‐risk group (75.0% vs 87.5%; P = .229).

DISCUSSION

In this study, we developed and validated a new prognostic bedside score for early identification of COVID‐19‐related critical illness and named it COVID‐19 Worsening Score (COWS). This new score integrated LUS findings and three selected variables of the previously validated COVID‐GRAM score. Since COWS does not require laboratory or radiological results, it enables rapid stratification of patients upon ED arrival. This aspect is critical when considering the large COVID‐19 patient influxes seen worldwide, which occasionally necessitated opening of outdoor tent areas and screening of patients in parking lots.

The overall accuracy of COWS is 80%, which is equal to the GRAM score. However, with a negative predictive value of 93%, COWS better discriminates low‐risk patients than the GRAM score and may thus help in reducing inappropriate ICU admissions and optimizing hospital resources. Moreover, the ability to anticipate clinical worsening could provide benefits to patients, such as shortening the time spent on spontaneous breathing, or on NIV, to prevent patient self‐inflicted lung injury (P‐SILI). ,

COVID‐GRAM score and COWS are not the only recently proposed scores. Several other prognostic scores were developed based on varying mixes of clinical data, laboratory results, and radiological findings. Zhou et al proposed a predictor of disease severity obtained from combining three independent variables: the neutrophils to lymphocytes ratio (N/L), C‐reactive protein (CRP), and D‐dimer values. This product had better predictive performance than single biomarkers as proved by an internal validation study. Several radiological scoring systems were also implemented to assess the severity of the disease and predict patient's outcomes. A chest X‐ray (CXR) scoring system on 18‐point scale, known as Brixia score, was proposed to quantify and monitor the severity of lung abnormalities. The Brixia score when combined with the patient's age and presence of immunosuppression was shown to predict in‐hospital mortality. In a retrospective single‐center study evaluating 1,198 ED COVID‐19 patients, the accuracy of both CXR and computerized CT scan for diagnosis of COVID‐19 was investigated. Sensitivity and specificity of CXR were 0.56 and 0.60, whereas for CT scan these were 0.85 and 0.50, respectively. Despite its low specificity, CT confirmed the diagnosis of COVID‐19 in patients with a false‐negative RT‐PCR as demonstrated in Chen et al study. The lower the number of pulmonary consolidations on the CT, the greater the likelihood of a negative RT‐PCR, suggesting the central role of CT as screening tool when COVID‐19 is strongly suspected. Association between CT findings and patient mortality was also studied. Kunhua et al investigated the combination of CT findings and clinical features in critical versus noncritical COVID‐19 patients. Results indicated that CT could identify patients who needed aggressive treatment and close monitoring. Another retrospective analysis investigated the link between lung aeration at baseline CT with the patient's adverse outcome: The degree of air loss and the presence of 4 or more lung lobes affected by COVID‐19 pneumonia were associated with admission to ICU or death.

Hence, it may be suggested that CT scanning could be the optimal imaging tool in COVID‐19, but it carries the burden of radiation exposure, higher cost, and prolonged equipment cleaning time compared to LUS. Moreover, both CT and LUS are not specific for COVID‐19 pneumonia. Keeping a balance between accuracy and availability, LUS is a tool able to identify early signs of pulmonary lesions of COVID‐19 pneumonia. Even though LUS cannot determine per se whether patients are infected by SARS‐CoV‐2, our results showed that in established COVID‐19 cases, the higher the LUS score, the greater was the risk of developing critical illness. We identified that a LUS score value higher than 15 helps discriminate between favorable and adverse outcomes in our cohort of patients. This result is consistent with previous findings reported by Soummer et al Thus, a score based on sonographic (ie, anatomical), functional, and clinical clues may be the most reliable means to provide a quick evaluation of the patient from complementary points of view.

COWS is based on LUS, P/F ratio, dyspnea, number of disease, and days from symptoms. For this reason, it acts as both a quick bedside tool and a screening test with a high negative predictive value. These two features suggest its usefulness in the context of the rapid evaluation of multiple patients presenting to the ED to avoid inappropriate resource use on low‐risk patients saving costly resources for a minor number of high‐risk patients. To this extent, the use of COWS may help increase appropriateness in the deployment of radiological resources, ventilatory equipment, and ICU admissions. Finally, one of the advantages of COWS compared to the GRAM score may also be its quick repeatability over time.

In the likely event of a second‐wave massive inflow of patients overwhelming hospital resources, patients may be listed according to the calculated predicted risk, in order to help the decision on resource allocation. In particular, stratifying patients by means of COWS may help set the appropriate monitoring level and aid in the difficult process of applying reverse triage criteria for ICU access in extreme conditions.

In the context of a long‐lasting epidemic, where a model of hub‐and‐spoke COVID‐19 hospitals might be used, COWS may speed up the selection of the low‐risk patients who may be safely transferred to spokes, keeping high‐risk patients in the hub center.

This study has several limitations. Firstly, it is a retrospective single‐center study and the sample size was relatively limited, as complete data were available for 143 patients. Moreover, even if the assessment of internal validity suggests potential usefulness of our newly developed score in clinical practice, however, external validation is needed to enhance the generalizability of our findings. A bigger multicenter, prospective research effort would also be advisable for a greater sample size collection. Secondly, despite COWS' ability to identify low‐risk patients, recognition of high‐risk patients remains suboptimal, and further adjustment should be applied. Thirdly, we used GRAM’ variable selection to build our model, instead of starting from all the possible variables collected in our patients. However, this approach is reasonable as the selected variables were variables with plausible clinical relation with the outcome. Fourthly, the P/F may have been calculated on very different FiO2 and with different levels of PEEP (from ZEEP to even 10 cmH2O). Finally, we tried to assess whether a thoracic CT scan might be combined with COWS as a second level examination in selected patients to improve the overall accuracy, but we did not find promising results for this purpose, possibly due to the limited number of observations. Of note, the cross‐sectional area of fat tissue at T7‐T8 vertebral height, assessed in Colombi et al, was not measured in our study due to CT software limitations.

CONCLUSION

COVID‐19 pandemic has severely challenged hospitals' capacity in providing intensive levels of care. After validating the COVID‐GRAM score in our population, we identified a simplified version of the score, by integrating LUS findings, functional, and selected clinical data. The COWS is bedside, quick, and easy to calculate. Its result is able to accurately identify patients who are unlikely to deteriorate or need ICU admission, sparing resources for the minority of COVID‐19 patients with a high‐risk of developing critical illness.

CONFLICT OF INTEREST

MB has participation in EchoNous Inc, Sonosim Inc, Ethos Medical, 410Medical. LV received travel facilities from Cook medical.

AUTHORS CONTRIBUTION

EB and SR contributed equally to concept, drafting article, and revision of the manuscript. MB, AS, DK, TB, and LV contributed in data interpretation and manuscript revision. PB and LC provided the statistical support for data analysis. MMC, MC, PG, AG, GF, SS, MP, and GL supervised data collection and contributed in manuscript revision. MB and LV share the senior authorship and contributed in critical revision and approval of article.