Chest Radiograph Scoring Alone or Combined with Other Risk Scores for Predicting Outcomes in COVID-19.

Background Radiographic severity may help predict patient deterioration and outcomes from COVID-19 pneumonia. Purpose To assess the reliability and reproducibility of three chest radiograph reporting systems (radiographic assessment of lung edema [RALE], Brixia, and percentage opacification) in patients with proven SARS-CoV-2 infection and examine the ability of these scores to predict adverse outcomes both alone and in conjunction with two clinical scoring systems, National Early Warning Score 2 (NEWS2) and International Severe Acute Respiratory and Emerging Infection Consortium: Coronavirus Clinical Characterization Consortium (ISARIC-4C) mortality. Materials and Methods This retrospective cohort study used routinely collected clinical data of patients with polymerase chain reaction-positive SARS-CoV-2 infection admitted to a single center from February 2020 through July 2020. Initial chest radiographs were scored for RALE, Brixia, and percentage opacification by one of three radiologists. Intra- and interreader agreement were assessed with intraclass correlation coefficients. The rate of admission to the intensive care unit (ICU) or death up to 60 days after scored chest radiograph was estimated. NEWS2 and ISARIC-4C mortality at hospital admission were calculated. Daily risk for admission to ICU or death was modeled with Cox proportional hazards models that incorporated the chest radiograph scores adjusted for NEWS2 or ISARIC-4C mortality. Results Admission chest radiographs of 50 patients (mean age, 74 years ± 16 [standard deviation]; 28 men) were scored by all three radiologists, with good interreader reliability for all scores, as follows: intraclass correlation coefficients were 0.87 for RALE (95% CI: 0.80, 0.92), 0.86 for Brixia (95% CI: 0.76, 0.92), and 0.72 for percentage opacification (95% CI: 0.48, 0.85). Of 751 patients with a chest radiograph, those with greater than 75% opacification had a median time to ICU admission or death of just 1-2 days. Among 628 patients for whom data were available (median age, 76 years [interquartile range, 61-84 years]; 344 men), opacification of 51%-75% increased risk for ICU admission or death by twofold (hazard ratio, 2.2; 95% CI: 1.6, 2.8), and opacification greater than 75% increased ICU risk by fourfold (hazard ratio, 4.0; 95% CI: 3.4, 4.7) compared with opacification of 0%-25%, when adjusted for NEWS2 score. Conclusion Brixia, radiographic assessment of lung edema, and percentage opacification scores all reliably helped predict adverse outcomes in SARS-CoV-2 infection. © RSNA, 2021 Online supplemental material is available for this article. See also the editorial by Little in this issue.

Summary

Brixia, radiographic assessment of lung edema, and percentage opacification produced reliable and reproducible severity scores for chest radiographs in patients with SARS-CoV-2 infection that led to improved accuracy for predicting adverse outcomes when incorporated into National Early Warning Score 2 and International Severe Acute Respiratory and Emerging Infection Consortium: Coronavirus Clinical Characterization Consortium clinical scoring systems.

■ Among 787 patients with confirmed COVID-19, three chest radiograph scores (Brixia, radiographic assessment of lung edema, and percentage opacification) all had good interreader reliability, with intraclass correlation coefficients of 0.87, 0.86, and 0.72, respectively.

■ Radiograph scores helped to predict admission to the intensive care unit (ICU) or death after COVID-19 diagnosis; opacification of 51%–75% (vs 0%–25%) was associated with a 2.2-fold increase in these outcomes among patients eligible for ICU care after adjustment for clinical risk scoring.

Introduction

In December 2019, a strain of coronavirus (SARS-CoV-2) was isolated from bronchoalveolar lavage samples in a cluster of patients with a febrile illness. This illness spread around the world and was declared a pandemic by the World Health Organization on March 11, 2020.

The reference standard for diagnosis of SARS-CoV-2 infection is reverse transcription polymerase chain reaction (RT-PCR) testing because of t he test's high specificity, despite limitations in its sensitivity (1). Chest radiography has limited sensitivity and specificity for identification of patients with SARS-CoV-2 infection (2,3) but can help identify patients with the disease (3–5). Several studies have demonstrated that the severity of lung involvement at chest radiography in SARS-CoV-2 infection is closely correlated with many key outcomes for patients, including intensive care unit (ICU) admission and death (6–20). Several scoring systems for assessing the severity of lung involvement in SARS-CoV-2 infection have been described in the literature. The most widely used and studied systems include Brixia (7,12,21) and radiographic assessment of lung edema (RALE/modified RALE) (6,13,14,16,22). No large studies have compared these systems with percentage opacification.

During the pandemic, severity of illness and risk for need for escalation of care could be predicted by using clinical scoring systems. In England, two examples are National Early Warning Score 2 (NEWS2) and the International Severe Acute Respiratory and Emerging Infection Consortium: Coronavirus Clinical Characterization Consortium (ISARIC-4C) mortality score (23,24). The utility of these scores has been established independently of chest radiology in previously published studies (23–27).

We aimed to assess the reliability, reproducibility, and ability to predict ICU admission or death (adverse outcomes) of three chest radiograph reporting systems (RALE, Brixia, and percentage opacification) in patients with SARS-CoV-2 infection. We also sought to examine the ability of these scores to predict adverse outcomes of SARS-CoV-2 infection, both alone and in conjunction with two clinical scoring systems: NEWS2 and ISARIC-4C mortality.

Materials and Methods

This work was approved by Nottingham University Hospitals Clinical Effectiveness Team (references 20–153C and 20–501C), the Nottingham University Hospitals Caldicott Guardian, Data Protection Impact Assessment (reference 436), and the National Health Service Health Research Authority (reference 282490). The Health Reference Authority confirmed that individual patient consent was not required.

Study Design and Patients

A retrospective, observational cohort study was conducted at Nottingham University Hospitals National Health Service Trust, England. Nottingham University Hospitals serve a population of approximately 2.5 million and has 1500 inpatient hospital beds. By using electronic health records within Nottingham University Hospitals, all patients admitted with RT-PCR–proven SARS-CoV-2 infection and who had undergone chest radiography were selected consecutively between February 2020 and July 2020, as previously described (28). Patients who first underwent chest radiography after ICU admission were excluded because they had already reached a study end point. There were no additional exclusions, and a sample size calculation indicated that more than 500 patients were required for a multivariate prediction model with more than 10 parameters (28). We obtained follow-up data for discharge, subsequent admissions to Nottingham University Hospitals, and death both in and out of the hospital (by the National Health Service Patient Demographics Service).

Chest Radiography

At Nottingham University Hospitals, single-view posteroanterior or anteroposterior chest radiography was routinely performed for patients admitted with proven or suspected SARS-CoV-2 infection. The first chest radiographic examination performed during the admission relating to RT-PCR was selected for analysis. During the early part of the pandemic, reporting was not uniform, and no specific scoring system was used. Thus, all chest radiographs were reassessed by using the RALE (22), Brixia (7,21), and percentage opacification scores (18). Table E1 (online) and Figure 1 describe these scoring systems. Chest radiographs were interpreted by using a Jusha monitor (Nanjing Jusha Commercial and Trading), with a picture communication and archiving system (GE Healthcare). Radiography was performed by using dual-detector system (DigitalDiagnost; Philips), digital system (Ysio Max; Siemens), and the FDR Nano Mobile (Fuji) system.

Figure 1:

Determination of the various scoring systems. (A) Anteroposterior radiograph in 78-year-old woman shows the classic changes of COVID-19 pneumonitis, which consist of opacification in a peripheral and basal distribution (arrows). (B) Radiograph shows calculation of the radiographic assessment of lung edema (RALE) score. The radiograph is divided into four quadrants. Each quadrant is assigned an intensity score and an opacification score. These are multiplied together for each quadrant, and all four scores are added together. The patient has a RALE score of 21. (C) Radiograph shows calculation of the Brixia score. The lungs are divided into six zones, and the degree of opacification is scored as follows: interstitial opacities, interstitial and alveolar opacities (interstitial predominate), and interstitial and alveolar opacities (alveolar predominate), scored as 1, 2, and 3, respectively. The patient has a Brixia score of 11 (1 + 2 + 2 + 1 + 2 + 3). The highest possible Brixia score is 18. (D) Radiograph shows percentage opacification, a simple visual estimate of the total percentage of lung parenchymal opacification.

Implementation of Scoring in the Cohort

Chest radiographs of the first 50 patients chronologically were assessed for all three scoring systems independently by all three consultant radiologists contributing to the study (I.A.Y., Y.H., and E.G., with 16, 15, and 10 years of experience and subspecialty interests in chest, gastrointestinal, and breast radiology, respectively) to assess interreader reliability. The three radiologists measured the time to score images (totaled across multiple chest radiographs) by using a stopwatch. For validation of intrareader reliability, each radiologist performed a second read of each chest radiograph, independent of the first read, at least 1 week later.

Once reliability of scoring was established, the entire cohort of consecutive patients with RT-PCR tests that were positive for SARS-CoV-2 was divided so that each chest radiograph was scored by one of the radiologists. Each radiologist therefore applied all three scores to approximately one-third of the chest radiographs. The radiologists were aware of the RT-PCR status but were blinded to all other clinical information.

Data collection beyond the scoring of chest radiographs has been fully described previously (28). Briefly, data were extracted from electronic records and included demographic characteristics, clinical decisions at eligibility for escalation (an assessment that had no fixed rules and was on the basis of a physician’s assessment of the patient’s frailty, comorbidity, and preferences), ward type (standard inpatient ward or ICU), and all other data needed to calculate ISARIC-4C mortality and NEWS2 (Table E1 [online]). Data were extracted from the date of relevant hospital admission (or of suspicion of SARS-CoV-2 if preceding the date of admission from an emergency department visit) until admission to the ICU, discharge from hospital, or in-hospital death. The time between both earliest suspected diagnosis and earliest confirmed diagnosis and the time of the chest radiograph were also calculated. These data were anonymized before delivery to the research team and were used to calculate NEWS2 and ISARIC-4C mortality.

Statistical Analysis

A P value less than .05 indicated statistical significance where relevant.

One author (T.C.) examined inter- and intraobserver reliability by using intraclass correlation coefficient for the RALE, Brixia, and percentage opacification scores. The intraclass correlation coefficient for interreader agreement was calculated by using a two-way random-effects model for absolute agreement and individual raters. A similar model with mixed effects was used for assessing intrareader agreement.

One author (C.J.C.) performed the cohort analysis. Follow-up started on the date of the reviewed chest radiograph and continued for 60 days, with complete follow-up for death or ICU admission.

Baseline demographic characteristics on the day of chest radiography were tabulated against the worst outcome observed during the follow-up period and were stratified by the initial decision on the eligibility of the patient for escalation because this assessment alters potential patient outcomes.

The median and interquartile range (IQR) for each chest radiograph score were tabulated and categorized (for percentage opacification, 0 = none, 1 ≤ 25%, 2 = 26%–50%, 3 = 51%–75%, 4 = 76%–100%; for RALE scores, 0–12, 13–24, 25–36, and 37–48; for Brixia scores, 0–5, 6–10, 11–15, and 16–18).

Kaplan-Meier survival curves were plotted and stratified by the categories of each score. The 95% CIs were calculated and estimated median survival times were presented where plausible for each category. Differences in overall survival by opacity score quantile were tested by using the log-rank test.

Finally, we assessed the additional predictive contribution of the opacity scores compared with previously published scores. Cox proportional hazard models were fitted by using ISARIC-4C mortality or NEWS2 calculated on the day the chest radiograph was reviewed, and then each opacity score was added to the model. Akaike information criterion was used to compare models, and the Harrell C statistic was used to measure changes in discrimination. Models were stratified by the initial escalation decision. R version 4.0.3 (R Project for Statistical Computing) was used for statistical analysis.

Results

Interrater and Intrarater Reliability and Speed of Reading

The initial 50 selected chest radiographs were obtained in 50 different patients between February 22 and May 27, 2020. The patients ranged in age from 40 to 95 years (mean, 74 years ± 16 [standard deviation]), and 28 of 50 patients (56%) were men. Interrater reliability was good for all three scores, with intraclass correlation coefficients of 0.87 (95% CI: 0.80, 0.92) for RALE score, 0.86 (95% CI: 0.76, 0.92) for Brixia score, and 0.72 (95% CI: 0.48, 0.85) for percentage opacification (Table E2 [online]). Intrarater reliability was also good, with intraclass correlation coefficients of 0.86 (95% CI: 0.75, 0.92) for RALE score, 0.86 (95% CI: 0.75, 0.92) for Brixia score, and 0.88 (95% CI: 0.82, 0.91) for percentage opacification (Table E3 [online]). Across the studies timed by the three readers, the scoring of the studies took a mean of 58 seconds (standard error of the mean, 5.7 seconds) for RALE, 37 seconds for Brixia (standard error of the mean, 3.6 seconds), and 33 seconds for percentage opacification (standard error of the mean, 3.2 seconds).

Description of Cohort for Assessing Performance of Scores in Predicting Severity

A total of 787 consecutive patients were admitted to Nottingham University Hospitals with a positive RT-PCR result for SARS-CoV-2 and a chest radiograph available for review in the study period (Fig 2). For the unadjusted survival analysis, 751 patients (95.4%) had a chest radiograph available before any escalation to the ICU; the median age of this group was 76 years (IQR, 59–84 years), and 419 of 751 (56%) were men. This cohort was stratified by the decision for escalation; 329 patients were eligible for escalation to the ICU and 422 patients were not eligible for escalation. For the further adjusted survival analysis by ISARIC-4C mortality or NEWS2, 628 patients had all the necessary data to calculate the scores (median age, 76 years; IQR, 61–84 years; 344 [57%] were men). The adjusted analysis was also stratified by the decision for escalation, with 266 eligible patients and 362 ineligible patients.

Figure 2:

Flowchart of patients admitted with reviewed chest radiograph (CXR) from February to July 2020 at Nottingham University Hospitals. ICU = intensive care unit, ISARIC-4C = International Severe Acute Respiratory and Emerging Infection Consortium: Coronavirus Clinical Characterization Consortium, NEWS2 = National Early Warning Score 2.

Patients were followed up for 60 days from the date of their initial chest radiographic examination. The median duration between the absolute time of chest radiography and earliest suspected diagnosis was 6 hours 40 minutes (IQR, 72 minutes to 69 hours). The median time between confirmed diagnosis and chest radiograph was 10 hours 43 minutes (IQR, 1 hour 38 minutes to 60 hours).

Patients’ baseline demographic characteristics are shown by their worst recorded outcomes during this follow-up period in Table 1. Among the patients not eligible for escalation who eventually died, the median percentage opacification was 20% (IQR, 4%–45%); among those who survived, the median percentage opacification was lower, at 10% (IQR, 0%–30%). Overall median percentage opacification was higher among patients who were eligible for escalation (median percentage opacification, 20%; IQR, 5%–30%) among those who did not need escalation and 40% among those who were escalated and transferred to the ICU (IQR, 22%–62%) or died (IQR, 10%–65%). There was a similar pattern for the other opacity scores (Table 1).

Table 1:

Baseline Demographic Characteristics by Worst Outcome during 60-Day Follow-up from Time of Earliest Chest Radiographic Examination

Survival Curves

Figure 3 shows crude survival during the 60 days after undergoing chest radiography for the 751 patients with a chest radiograph during the admission before any ICU admission. Kaplan-Meier plots show each percentage opacification score stratified into categories (Fig 3). Each categorized score separated out the patients’ overall survival. Median escalation-free survival was 7.6 days (95% CI: 5.4, 23.7) for percentage opacification of 51%–75% and 2.6 days (95% CI: 1.5, 16.6) for 76%–100% opacity (P < .001, log-rank test). The corresponding median survival times were 7.5 days (95% CI: 5.4, 22.4) for the categorized Brixia score of 11–15 and 1.15 days (95% CI: 0.7, 4.8) for a Brixia score of 16–20 (P < .001, log-rank test). Similarly, median survival was 5.18 days (95% CI: 2.9, 11.4) for a RALE score of 25–36 and 1.21 days (95% CI: 0.6, 4.8) for a RALE score of 36–48 (P < .001, log-rank test).

Figure 3:

Kaplan-Meier survival curves during 60 days after chest radiographic examination (CXR) stratified by quartiles of (A) percentage opacification, (B) Brixia, and (C) radiographic assessment of lung edema (RALE) scores (RALESCR).

Adjusted Survival Analysis

A total of 628 patients had data regarding the required blood tests and observations on the day of the radiographic chest examination to enable the calculation of these measures. Compared with 0%–25% opacification, risk for ICU admission or death was increased by more than twofold (hazard ratio, 2.17; 95% CI: 1.57, 2.77) with 51%–75% opacification and by more than fourfold (hazard ratio, 4.04; 95% CI: 3.4, 4.69) with 76%–100% opacification.

Discrimination increased for percentage opacification, Brixia score, and RALE score. For example, the percentage opacification, Brixia score, and RALE score, adjusted for ISARIC-4C mortality, increased the C statistic from 0.58 to 0.65, 0.65, and 0.66, respectively, and improved the corresponding goodness of fit from an R2 value of 0.06 to 0.13, 0.15, and 0.15, respectively (Table 2).

Table 2:

Cox Proportional Hazard Models Predicting Escalation to Intensive Care Unit or Death within 60 Days for All Patients

This was also observed when the results were stratified by eligibility for escalation in Tables 3 and 4. For example, in patients eligible for escalation, for each score adjusted for ISARIC-4C mortality, the C statistic improved from 0.63 to 0.72, 0.72, and 0.71 respectively, and the corresponding R2 value improved from 0.07 to 0.19, 0.20, and 0.21, respectively. Models and opacity scores had greater discrimination and goodness of fit in patients eligible for escalation than in those who were ineligible; for example, in a comparison of patients ineligible for escalation adjusted for the ISARIC-4C mortality score, the C statistic improved from 0.56 to 0.60, 0.60, and 0.61, respectively, for each score, and the corresponding R2 value improved from 0.03 to 0.07, 0.09, and 0.11, respectively. The highest discrimination was among patients eligible for escalation according to RALE scores and NEWS2 (C statistic, 0.77; R2 = 0.27).

Table 3:

Cox Proportional Hazard Models Predicting Escalation to Intensive Care Unit or Death within 60 Days for Patients Eligible for Escalation

Table 4:

Cox Proportional Hazard Models Predicting Death within 60 Days for Patients Not Eligible for Escalation

Discussion

The identification of patients with SARS-CoV-2 at high risk for deterioration is important for treatment. Such prediction may help triage, enable timely prevention of further deterioration, and aid in resource allocation. This cohort study in a single English city aimed to examine the effect of combining established clinical scores to predict adverse outcomes with a chest radiography severity score to predict key outcomes. We demonstrated that severity on chest radiographs in patients with SARS-CoV-2 can be rapidly and reliably reported with radiographic assessment of lung edema (RALE), Brixia, and percentage opacification scoring systems. All three scores were calculated in less than 1 minute; percentage opacification was the fastest to be calculated. Higher scores were strongly associated with admission to the intensive care unit (ICU) or death up to 60 days after SARS-CoV-2 diagnosis. Opacification on a radiograph of 76%–100% was associated with an overall adjusted risk for ICU admission or death that was three- to fourfold higher than that seen with opacification of 0%–25%. Patients in the highest categories of each score had a median time to ICU admission or death of just 1–2 days. Our data demonstrate that all methods of chest radiograph severity scoring can improve prediction of admission to the ICU or death when added to International Severe Acute Respiratory and Emerging Infection Consortium: Coronavirus Clinical Characterization Consortium (ISARIC-4C) mortality score and National Early Warning Score 2 (NEWS2). In our cohort, the best approach, by discrimination, for predicting outcome on admission with SARS-CoV-2 infection was to combine a chest radiograph severity score with NEWS2 (C statistic, 0.77 for patients eligible for escalation with use of RALE score and NEWS2).

Our cohort had a full 60-day follow-up without loss of any patients to follow-up, stratified by eligibility for escalation of care, in contrast to other published reports (6,7,12–14,16,21,22). Notably, the chest radiograph scores were less discriminative in patients who were not eligible for escalation. This presumably relates to greater contribution of other factors, such as increasing age, frailty, and comorbidity, in this group. Scoring systems, such as NEWS2 (29,30) and ISARIC-4C mortality (31), can help predict the need for escalation in these patients. But neither they nor the closely related ISARIC-4C deterioration score (32) (assessing requirement of ventilatory support or critical care, or death) incorporates a chest radiograph scoring system.

There has been recent interest in the potential of scoring radiographic severity of SARS-CoV-2 pneumonitis changes to predict important patient outcomes. Several studies have reported such data (6–20). Our study focused on percentage opacification, a simple and readily applicable scoring system, and compared it directly to Brixia and RALE scores, which have been studied more extensively. To our knowledge, only one other study has examined a percentage opacification score in relation to patient outcomes (18). One advantage of percentage opacification is that it is quick and straightforward to calculate, as reflected in the speed of its calculation compared with that of Brixia and RALE scores. The latter scores are also more complex to apply in clinical practice because of incorporation of zonal scoring and the need to assess the nature and intensity of the parenchymal opacification. Thus, RALE and Brixia may be more challenging for junior medical staff to apply in practice, and increased complexity may add to existing pressures on staff given the increasingly recognized risks of burnout (33).

Two other studies have also examined the effect of radiologic severity scoring systems in predicting patient outcome. Maroldi et al (12) examined the Brixia score in a cohort of similar size. That study incorporated radiographic scoring for radiographs obtained during the patients’ SARS-CoV-2 admission to determine whether the Brixia score could predict outcomes. The results suggested that the Brixia score can be used to predict mortality. However, the study by Maroldi et al did not stratify by eligibility for escalation, had only in-hospital death as the outcome, and did not compare different scoring systems. Balbi et al (18) examined a smaller cohort of 340 patients from Italy and compared two scores: Brixia and percentage opacification on the admission chest radiograph. The authors concluded that Brixia helps predict mortality, whereas percentage opacification helps predict the need for ventilation. This study categorized the scores into binary variables, did not stratify by eligibility for escalation, and analyzed only in-hospital mortality.

Finally, there has been interest in artificial intelligence in scoring of chest radiograph (6,34) by comparing the performance of artificial intelligence systems to severity scores in predicting patient outcomes. The studied artificial intelligence algorithm incorporates scoring that appears similar to percentage opacification. The results of these artificial intelligence studies suggest a potential future role for artificial intelligence in calculating radiographic severity of SARS-CoV-2 pneumonitis, particularly if it could be combined with the calculation of an early-warning score, such as NEWS2.

Our study had limitations. It was a retrospective single-center study, all patients were RT-PCR positive (clinical diagnoses of SARS-CoV-2 infection were not included), and only admitted patients who underwent chest radiography were studied. These factors may limit the generalizability of our results, particularly to mild cases. In addition, the participating radiologists were not all chest radiologists but were all highly experienced in chest radiograph reporting, especially during the pandemic, thus reflecting real-world practice.

In conclusion, we show that three scoring systems for assessing chest radiograph severity can be calculated quickly for SARS-CoV-2 infection with good reliability and reproducibility. Overall, the higher the score, the worse the outcome for patients with SARS-CoV-2. Incorporation of all three scores into well-described risk prediction models substantially improved the ability to predict adverse outcomes of SARS-CoV-2 infection at hospital admission.