Application of validated severity scores for pneumonia caused by SARS-CoV-2.

OBJECTIVES: Compare the accuracy of PSI, CURB-65, MuLBSTA and COVID-GRAM prognostic scores to predict mortality, the need for invasive mechanical ventilation (IMV) in patients with pneumonia caused by SARS-CoV-2 and assess the coexistence of bacterial respiratory tract infection during admission.
METHODS: Retrospective observational study that included hospitalized adults with pneumonia caused by SARS-CoV-2 from 15/03 to 15/05/2020. We excluded immunocompromised patients, nursing home residents and those admitted in the previous 14 days for another reasons. Analysis of ROC curves was performed, calculating the area under the curve for the different scales, as well as sensitivity, specificity and predictive values.
RESULTS: 208 patients were enrolled, aged 63 ± 17 years, 577% were men. 38 patients were admitted to ICU (235%), of these patients 33 required IMV (868%), with an overall mortality of 125%. Area under the ROC curves for mortality of the scores were: PSI 082 (95% CI 073-091), CURB-65 082 (073-091), MuLBSTA 072 (062-081) and COVID-GRAM 086 (070-1). Area under the curve for needing IMV was: PSI 073 (95% CI 064-082), CURB-65 066 (055-077), MuLBSTA 078 (069-086) and COVID-GRAM 076 (067-085), respectively. Patients with bacterial co-infections of the respiratory tract were 20 (9,6%), the most frequent strains being Pseudomonas aeruginosa and Klebsiella pneumoniae.
CONCLUSIONS: In our study, the COVID-GRAM score was the most accurate to identify patients with higher mortality with pneumonia caused by SARS-CoV-2; however, none of these scores accurately predicts the need for IMV with ICU admission. 10% of patients admitted presented bacterial respiratory co-infection.

Introduction

Pneumonia remains one of the leading causes of death from infection globally in the short and long term. On 11 March 2020, the World Health Organization declared a global pandemic status for SARS-CoV-2, a new virus of the family Coronaviridae which originated in December 2019 in the city of Wuhan. The disease produced by this virus has been called COVID-19, it is a respiratory tract infection with a variable severity clinical spectrum. Approximately 30% of patients develop severe viral pneumonia that requires hospitalization and oxygen therapy,3, 4 and 5% may require admission to an intensive care unit (ICU) due to progression to acute respiratory distress syndrome with potential for complication with sepsis, septic shock, multi-organ failure and death.

The pandemic has required good health organisation and planning in the face of limited resources.6, 7 The clinical assessment can overestimate or underestimate the severity of this pneumonia and lead to hospitalization of mild cases that could be treated at home or to perform insufficient interventions in patients who are at high risk of complications. Validated prognostic scales can overcome these difficulties by objectively and rapidly classifying patients in certain risk categories according to the results that will be obtained in terms of mortality or the need for critical care.8, 9 Spain has recently begun vaccination against SARS-CoV-2 and the evidence on the efficacy of the treatments we apply, although increasing,10, 11, 12 is limited, so in this pandemic situation it is extremely important to find tools that allow us to quickly and correctly classify patients, especially for the early detection of the most serious patients.

In this sense, there are different validated prognostic scales for community-acquired pneumonia whose use is recommended in national and international14, 15 clinical practice guidelines, such as: PSI and CURB-65. These scales have been widely studied in bacterial pneumonia; however, in viral pneumonias the information available is much more limited.18, 19, 20, 21 The MuLBSTA scale was published in 2019. Designed specifically for patients with viral pneumonia, it has better sensitivity and specificity to predict mortality at 30 and 90 days than the CURB-65 scale. Recently Liang et al. published in the JAMA Internal Medicine journal the specific COVID-GRAM prognostic scale for COVID-19, in which they define the primary endpoint as critical COVID-19 illness, composed of: admission to ICU, need for invasive mechanical ventilation (IMV) or death. This scale has an online calculator ( http://118.126.104.170/).

There is a growing number of publications on prognostic scales in COVID-19,24, 25 however, today it is not clear which of them is more accurate in assessing severity in the form of mortality and the need for IMV with admission to the ICU in hospitalized patients. The main objective of our study was to compare the performance of the PSI, CURB-65, MuLBSTA, and COVID-GRAM prognostic scales to predict 30-day mortality in patients admitted with SARS-CoV-2 pneumonia. As a secondary objective, we set out to compare the performance of these scales to predict the need for IMV with admission to the ICU, as well as to assess the existence of respiratory bacterial coinfection during admission.

Material and methods

Design

We conducted a retrospective observational study including patients seen and admitted to our hospital with SARS-CoV-2 pneumonia from 15 March to 15 May 2020, with follow-up until 15 June 2020. The study was carried out in accordance with the Declaration of Helsinki and was approved by the Clinical Research Ethics Committee of the General University Hospital of Alicante.

Patients

Inclusion criteria were: age ≥18 years, positive nucleic acid detection test (RT-PCR, reverse transcriptase polymerase chain reaction) for SARS-CoV-2 and pneumonia by radiological imaging test performed in the emergency department on hospital admission. Patients with a history of bronchiectasis, with active solid organ or haematological tumours, with HIV or another type of immunosuppression, pregnant, institutionalized, and those with hospital admission in the previous 14 days for another reason were excluded. This was so because the PSI and CURB-65 scales used these exclusion criteria in their population in their design and validation. Just to calculate our secondary endpoint, we also excluded patients with baseline withholding of life support; that is, those who, due to their advanced age, frailty, and comorbidities, were not candidates for ICU admission. In our center, all patients were treated based on a medical protocol applied in a multidisciplinary way, following the guidelines of the technical documents provided by the Ministry of Health. Patients discharged from hospital were re-evaluated after 6–8 weeks in the outpatient clinics of Pneumology and Internal Medicine by clinical, laboratory, radiological and respiratory functional assessment.

Outcome variables

The primary endpoint was mortality 30 days after hospital admission. Secondary variables were the need for IMV with admission to the ICU and the existence of respiratory bacterial coinfection, defined as the existence of positive respiratory cultures (sputum, tracheobronchial aspirate or cultures obtained by bronchoscopy), accompanied by clinical changes (fever, oxygenation deterioration), leukocytosis, purulent secretions or the onset of a new or progressive pulmonary infiltrate).

Explanatory variables

Demographic variables, comorbidities, clinical variables, laboratory test results and radiological data were collected on admission, as well as results of respiratory bacteriological cultures obtained during admission. The treatments used, days of hospital stay, need for ICU admission, need for respiratory support (non-invasive and invasive) and survival were also collected.

Regarding the main explanatory variables, the following scales were used: PSI, CURB-65, MuLBSTA22 and COVID-GRAM. The items included in each of the aforementioned scales are listed in Table 1 .

Table 1

Prognostic scales analysed in the study with their different variables and score.

A → PSI		B → CURB-65		C → MuLBSTA		D → COVID-GRAM(http://118.126.104.170/)
Male	Years	Confusion	+1	Multilobar infiltrates	+5	Radiological abnormalities
Female	Years −10	Urea > 42 mg/dL		Lymphocytes≤0.8	+4	Age (years)
Nursing home	+10	Respiratory rate > 30 rpm	+1	Bacterial coinfection	+4	Haemoptysis (yes/no)
Neoplastic disease	+30	Systolic blood pressure < 90 mmHg	+1	Active smoker	+3	Dyspnoea (yes/no)
Liver disease	+20	Diastolic blood pressure < 60 mmHg	+1	Former smoker	+2	Confusion (yes/no)
Congestive heart failure	+10	Age > 65years	+1	Arterial hypertension	+2	Number of comorbidities
Cerebrovascular disease	+10			Age ≥60 years	+2	History of cancer (yes/no)
Renal disease	+10					N/L ratio
Altered mental status	+10					Lactate dehydrogenase, U/L
Respiratory rate ≥30 bpm	+20					Direct bilirubin, μmol/L
Systolic blood pressure <90 mmHg	+20
Temperature < 35 °C or > 40 °C	+15
Heart rate ≥125 bpm	+10
Arterial pH < 7.35	+30
Blood urea nitrogen ≥30 mg/dL	+20
Na < 130 mmo l/L	+20
Glucose ≥250 mg/dL	+10
Packed cell volume < 30%	+10
PaO2 < 60 mmHg	+10
Pleural effusion	+10

N/L: neutrophils/lymphocytes; Na: sodium; PaO2: arterial oxygen pressure.

Statistical analysis

A descriptive analysis was carried out; qualitative variables were expressed as absolute number and percentage, while quantitative variables were expressed as mean ± standard deviation or median (interquartile range), depending on whether the distribution conformed to normality or not. To compare the scales, Receiver Operating Characteristic (ROC) curve analysis was performed, calculating the area under the curve for the different scales with their 95% confidence intervals, as well as sensitivity, specificity, and predictive values of each one to predict the established objectives. An area under the curve ≥0.8 was considered to provide excellent discrimination.

The cut-off points to calculate sensitivity, specificity and predictive values of the scales were the following: PSI, 90 points was used, as from 91 the risk of expected mortality from pneumonia is 8.2–12.5%, and hospitalisation is recommended; CURB-65, one point was used, as from 2 points the risk of expected mortality from pneumonia is more than 6.8% and hospitalisation is recommended; MuLBSTA, 11 points was used, as from 12 points a high risk is established. Regarding the COVID-GRAM scale, the cut-off point with the best ROC curve results was selected. The incidence of respiratory bacterial coinfection was determined. All statistical data analysis was calculated using IBM SPSS Statistics v25 (Armonk, NY) software.

Results

From 15 March to 15 May, 253 patients with SARS-CoV-2 pneumonia were admitted; 45 (17.8%) were excluded for different reasons: 18 because they were institutionalised before admission; 14 patients due to immunosuppression under treatment (5 transplant recipients, 2 with rheumatoid arthritis, one with mixed connective tissue disease, one with polymyalgia rheumatica, one with psoriatic arthritis, one with Crohn's disease, one with pemphigus foliaceus, one with autoimmune hepatitis and one with HIV); 8 patients for active solid organ or haematological tumours (5 solid organ neoplasms and 3 with acute myeloblastic leukaemia); also 2 patients with bronchiectasis, 2 patients who were pregnant and one patient who had been admitted to hospital in the previous 14 days for another reason.

Finally, 208 patients were included, with a mean age of 63 years (±17), 57.7% men. Among the comorbidities, arterial hypertension (47.1%), obesity (34.6%), cardiovascular disease (25.2%), diabetes (22.1%), chronic kidney failure (18.9%) and chronic lung disease stood out (14.9%). The median duration of hospitalization was 9 days (6–13). Of the total, 26 patients (12.5%) died; at the time of study closure, 181 patients had been discharged and only one patient remained hospitalized.

As regards one of the secondary objectives of the study, only to predict the need for IMV with ICU admission, 46 patients out of 208 (22.1%) were excluded due to baseline withholding of life support. Lastly, 162 patients were analysed, with a mean age of 57 years (SD ± 14), 60.5% men. Thirty-eight patients (23.5%) were admitted to ICU; of these, 33 required IMV (86.8%) and 7 (18.4%) of the total ICU patients died. 23 patients were managed with non-invasive respiratory support: 22 with high-flow oxygen therapy and one with CPAP. The median duration of ICU admission was 13 days (8–22).

Mortality predictive ability of the different prognostic scales

Fig. 1 shows the representation of the ROC curves of the different scales in relation to mortality. For the PSI scale, an area of 0.82 (95% CI: 0.73–0.91) was obtained, for CURB-65 of 0.82 (95% CI: 0.73–0.91), for MuLBSTA of 0.72 (95% CI: 0.62–0.81) and for COVID-GRAM of 0.86 (95% CI: 0.70–1; p = 0.001). These results are listed in Table 2 .

Fig. 1

ROC curves of the different prognostic scales to predict mortality at 30 days. A) PSI. B) CURB65. C) MuLBSTA. D) COVID-GRAM.

Table 2

Precision of the prognostic scales for mortality in SARS-CoV-2 pneumonia (n = 208).

Scale	AUC (95% CI)	Sensitivity %	Specificity%	PPV%	NPV %
PSI	0.824 (0.73−0.91)	84.62	73.08	30.98	97.08
CURB-65	0.821 (0.73−0.91)	88.46	54.39	21.70	97.05
MuLBSTA	0.715 (0.62−0.81)	53.85	75.82	24.13	92.00
COVID-GRAM	0.857 (0.70−1)	88.46	73.08	31.94	97.79

AUC: area under the ROC curve; NPV: negative predictive value; PPV: positive predictive value.

The sensitivity to predict mortality of the PSI scale was 84.62%, the CURB-65 scale was 88.46% and the MuLBSTA scale was 53.85%.

To calculate sensitivity, specificity, and predictive values of the COVID-GRAM scale, a 147.5 cut-off point was used, since the best results were obtained with this cut-off point. This scale could be calculated in 159 patients out of the total since direct bilirubin data was missing in the rest. The sensitivity to predict mortality for the COVID-GRAM scale was 88.46%, the specificity was 73.08%, the PPV was 31.94%, while the NPV was 97.79%.

Predictive ability of the need for invasive mechanical ventilation for the different prognostic scales

Fig. 2 shows the representation of the ROC curves of the different scales in relation to the need for IMV. For the PSI scale, an area of 0.73 was obtained (95% CI 0.64–0.82), for CURB-65 0.66 (95% CI 0.55–0.77), for MuLBSTA 0.78 (95% CI 0.69–0.86) and for COVID-GRAM 0.76 (95% CI 0.67–0.85; p = 0.001). These results are listed in Table 3 .

Fig. 2

ROC curves of the different prognostic scales to predict the need for orotracheal intubation. A) PSI. B) CURB65. C) MuLBSTA. D) COVID-GRAM.

Table 3

Precision of the prognostic scales for endotracheal intubation in SARS-CoV-2 pneumonia (n = 162).

Scale	AUC (95% CI)	Sensitivity %	Specificity%	PPV %	NPV %
PSI	0.728 (0.64–0.82)	45.45	85.27	44.12	85.27
CURB-65	0.660 (0.55–0.77)	63.63	65.89	32.30	87.63
MuLBSTA	0.780 (0.69–0.86)	54.54	83.72	46.15	87.80
COVID-GRAM	0.760 (0.67–0.85)	39.39	84.49	39.39	84.49

AUC: area under the ROC curve; NPV: negative predictive value; PPV: positive predictive value.

To calculate the sensitivity, specificity and predictive values of the scales, the same cut-off points used to calculate mortality were maintained. The sensitivity to predict the need for IMV of the PSI scale was 45.45%, 63.63% for the CURB-65 scale and 54.54% for the MuLBSTA scale.

The COVID-GRAM scale could be calculated in 146 patients of those included for this objective, as the direct bilirubin value was missing in the rest. The sensitivity to predict the need for IMV for the COVID-GRAM scale was 39.39%, the specificity was 84.49%, the PPV was 39.39%, while the NPV was 84.49%.

Respiratory bacterial coinfection

Of the patients admitted for SARS-CoV-2 pneumonia, 33 had positive respiratory cultures (15.8%). There were 13 cases of colonisation (39.4%), 8 cases of tracheobronchitis (24.2%) and 12 cases of ventilator-associated pneumonia (36.3%). In the cases of pneumonia associated with mechanical ventilation, most of the microbiological isolates were polymicrobial (66.6%). Therefore, the total number of patients with respiratory bacterial coinfection was 20 (9.6%); of these, 16 were admitted to the ICU (80%).

In cases of respiratory bacterial coinfection, Pseudomonas aeruginosa (30.3%) and Klebsiella pneumoniae (24.2%) were the most frequently isolated pathogens in crops, followed by Enterococcus faecalis (12.1%), Stenotrophomonas maltophila (9%), Enterobacter aerogenes (6%), Rothia mucilaginosa (6%), Klebsiella oxytoca (3%), Escherichia coli (3%), Haemophilus influenzae (3%) and Stahpylococcus aureus (3%).

Discussion

Our study compares different prognostic scales in a cohort of patients with SARS-CoV-2 pneumonia to assess whether they are useful for predicting mortality and the need for IMV with admission to the ICU. As main findings of the study we highlight: 1) the COVID-GRAM scale was the most accurate to identify patients with the highest mortality, it is a simple scale that consists of 10 variables and can be helpful in detecting patients with a worse prognosis; 2) none of these scales accurately predicts the need for IMV with admission to the ICU, since in the area under the ROC curves in all cases <0.8 and have a low sensitivity; 3) 9.6% of the admitted patients had respiratory bacterial coinfection and the most common microorganisms were Pseudomonas aeruginosa and Klebsiella pneumoniae.

In this case, the ROC curve is a graph representing the sensitivity of a scale relative to false positives. An area under the curve ≥0.8 provides excellent discrimination, as it has good sensitivity with few false positives. Adequate sensitivity is essential to keep an eye on patients with poor prognosis. In our study, the COVID-GRAM scale obtained the best results with the best area under the ROC curve and also the best sensitivity, specificity, and predictive values. We believe this is because it is a scale obtained from a series of patients with the same disease, whereas in the other scales, the disease may be caused by different micro-organisms.

The PSI scale consists of 20 variables and attributes a significant weight to age as well as comorbidities. The CURB-65 scale is simpler, since it consists of only 5 items, one of which is the respiratory rate. In patients with SARS-CoV-2 pneumonia we have seen that there is what experts describe as “silent hypoxemia”, in other words, some patients could find themselves with severe hypoxemia without high respiratory rates, so this scale could underestimate the severity of SARS-CoV-2 pneumonia and especially the need for IMV in the ICU. It should not be forgotten that these scales were developed primarily to assess patients in the emergency department, as their main advantage is that they have a high NPV, i.e. when applied to patients with community-acquired pneumonia they can identify low-risk patients who can be safely discharged and managed at home.8, 17 Numerous studies have shown that the PSI and CURB-65 scales have low sensitivity for predicting ICU admission in community-acquired pneumonia, in particular because they assign significant weight to age and comorbidities and low weight to acute respiratory failure. The MuLBSTA scale performs poorly in our study in predicting mortality and the need for IMV, probably because it does not include key clinical and laboratory predictor variables.

The current literature on bacterial coinfection in patients with SARS-CoV-2 infection is scarce.28, 29, 30 The existence of bacterial coinfection has been associated with higher morbidity and mortality. The rate of respiratory bacterial coinfection in our study affected 9.6% of patients, which is in line with other publications, in which coinfection can affect up to 15% of patients.

Our study has some important limitations, such as the fact that it includes cases from a single site, it has a retrospective design, the COVID-GRAM scale could not be calculated in all the patients included as direct bilirubin was not available in some of the laboratory tests and the influence of the treatments on the course of the disease cannot be ruled out. However, the present study also has strengths: all data were collected in a very comprehensive manner, no patient in the study used remdesivir, the treatments prescribed followed the protocol established at the centre, only one patient remained hospitalised at the time of study closure, therefore, the morbidity and mortality data are well estimated and, in addition, this is a series of patients with SARS-CoV-2 pneumonia, in which, in addition to scales such as PSI, CURB-65 or MuLBSTA, the COVID-GRAM scale score, specific for COVID-19 and recently published, has been calculated.

Our study found that the COVID-GRAM classification was the most accurate to identify the patients with the highest mortality. The scale is based on 10 variables, which must be considered on admission to detect patients with a worse prognosis. However, none of the scales accurately predicts the need for IMV with ICU admission in hospitalised patients with SARS-CoV-2 pneumonia. In any case, these prognostic scales are complementary tools to clinical assessment and prospective studies are needed to confirm these findings in order to generalize their applicability. Our study showed that one in 10 patients admitted with SARS-CoV-2 pneumonia had respiratory bacterial co-infection, of which the most commonly found bacteria were Pseudomonas aeruginosa and Klebsiella pneumoniae.

Conflict of interests

The authors declare no conflict of interest.