Comparison of Chest CT Grading Systems in Coronavirus Disease 2019 (COVID-19) Pneumonia.

PURPOSE: To compare the performance and interobserver agreement of the COVID-19 Reporting and Data System (CO-RADS), the COVID-19 imaging reporting and data system (COVID-RADS), the RSNA expert consensus statement, and the British Society of Thoracic Imaging (BSTI) guidance statement.
MATERIALS AND METHODS: In this case-control study, total of 100 symptomatic patients suspected of having COVID-19 were included: 50 patients with COVID-19 (59±17 years, 38 men) and 50 patients without COVID-19 (65±24 years, 30 men). Eight radiologists independently scored chest CT images of the cohort according to each reporting system. The area under the receiver operating characteristic curves (AUC) and interobserver agreements were calculated and statistically compared across the systems.
RESULTS: A total of 800 observations were made for each system. The level of suspicion of COVID-19 correlated with the RT-PCR positive rate except for the "negative for pneumonia" classifications in all the systems (Spearman's coefficient: ρ=1.0, P=<.001 for all the systems). Average AUCs were as follows: CO-RADS, 0.84 (95% confidence interval, 0.83-0.85): COVID-RADS, 0.80 (0.78-0.81): the RSNA statement, 0.81 (0.79-0.82): and the BSTI statement, 0.84 (0.812-0.86). Average Cohen's kappa across observers was 0.62 (95% confidence interval, 0.58-0.66), 0.63 (0.58-0.68), 0.63 (0.57-0.69), and 0.61 (0.58-0.64) for CO-RADS, COVID-RADS, the RSNA statement and the BSTI statement, respectively. CO-RADS and the BSTI statement outperformed COVID-RADS and the RSNA statement in diagnostic performance (P=.<.05 for all the comparison).
CONCLUSIONS: CO-RADS, COVID-RADS, the RSNA statement and the BSTI statement provided reasonable performances and interobserver agreements in reporting CT findings of COVID-19. 2020 by the Radiological Society of North America, Inc.

Summary

The CT grading systems of COVID-19 provided a reasonable diagnostic performance with AUCs ranging 0.80-0.84 and good interobserver agreement with Cohen’s kappa values ranging 0.61-0.63, along with positive correlations between RT-PCR positive rates and categories except for the “negative for pneumonia” classifications.

• The level of suspicion of COVID-19 correlated with the RT-PCR positive rate except for the “negative for pneumonia” classifications (Spearman’s coefficient: ρ=1.0, P=<.001 for all the systems).

• Average AUCs were as follows: CO-RADS, 0.84 (95% confidence interval: 0.83–0.85): COVID-RADS, 0.80 (0.78-0.81): the RSNA expert consensus statement, 0.81 (0.79-0.82): and the BSTI guidance statement, 0.84 (0.82-0.86).

• Average Cohen’s kappa in all observers was as follows: CO-RADS, 0.62 (95% confidence interval: 0.58–0.66): COVID-RADS, 0.63 (0.58-0.68): the RSNA expert consensus statement, 0.63 (0.57–0.69): and the BSTI guidance statement, 0.61 (0.58-0.64).

Introduction

During the ongoing pandemic of coronavirus disease 2019 (COVID-19), the prompt diagnosis is crucial to achieve swift and optimal clinical decision making and judge the precaution level necessary on admission to help prevent nosocomial infection in the hospital. Various evidence has documented that early diagnosis and intervention are associated with a better prognosis [1]. The gold standard diagnostic method for COVID-19 is reverse transcription-polymerase chain reaction (RT-PCR) that directly quantifies viral load from a nasopharyngeal swab, sputum, or endotracheal lavage. However, the sensitivity of this method is unclear as false-negative results have been reported in patients with insufficient specimen or those in the initial stage of infection [2]. The turnaround time is also long ranging from a few hours to days. Although chest CT is currently not recommended for routine screening purposes, it provides valuable information serving as a supplementary diagnostic tool of COVID-19 pneumonia especially in circumstances in which RT-PCR tests are not sufficiently available or in patients in whom the possibility of false negative results is suspected, or clinical decisions are required before the PCR test results become available.

With the accumulation of recent publications clarifying the radiological appearance of COVID-19, various attempts have been made to standardize reporting of chest CT for suspected COVID-19. The British Society of Thoracic Imaging (BSTI) proposed Guidance for the Reporting Radiologist as a diagnostic framework of COVID-19 from chest CT and radiograph [3]. The recent RSNA expert consensus statement on reporting advocates a standard nomenclature and imaging classification for COVID-19 pneumonia made up of four categories (typical appearance, indeterminate appearance, atypical appearance, and negative for pneumonia) [4]. A working group of the Dutch Radiological Society devised the COVID-19 Reporting and Data System (CO-RADS) to facilitate the advances in and worldwide dissemination of COVID-19 related information and tools [5]. Another group of researchers devised a different structured reporting system based on a review of 37 published papers on the chest CT findings of COVID-19 entitled the COVID-19 imaging reporting and data system (COVID-RADS) that divides the CT findings into five categories [6].

The published CT grading systems of chest CT findings in COVID-19 patients may facilitate both making the radiological diagnosis and smooth communication among professionals in other fields, and their applicability and validity in the clinical practice was recently reported in several studies [7, 8, 9, 10]. However, no studies have yet directly compared the diagnostic performances and interobserver agreement between them. This prompted us to undertake the present study to validate the performance and interobserver agreement of four sets of CT grading systems, including the COVID-19 Reporting and Data System (CO-RADS), the COVID-19 imaging reporting and data system (COVID-RADS), the RSNA expert consensus statement and the BSTI guidance statement.

Materials and Methods

This study was conducted with the approval of our institutional ethics review board. Written informed consent was waived due to the retrospective nature of the study. The privacy of all patients was protected in full.

Sample Size Calculation and Study Population

Patient backgrounds were standardized by applying the following inclusion criteria: (1) presentation to the outpatient or emergency department of a single institution from January 30 to June 30, 2020, (2) suspected of COVID-19 because of the presence of symptoms suggestive of pneumonia (i.e. fever (>37.5°C) and at least one of the following symptoms; cough, dyspnea, tachypnea, or hypoxemia), (3) having undergone RT-PCR examination, (4) acquisition of chest CT within 5 days of the initial RT-PCR test. Patients were classified as COVID-19 or non-COVID-19 if they tested positive or negative respectively on RT-PCR at least one time. Those who tested negative on the initial RT-PCR but were on a high clinical suspicion of COVID-19 underwent repeat RT-PCR and categorized as COVID-19 positive if repeat RT-PCR tested positive. Those who tested negative on the initial RT-PCR and were not having a high clinical suspicion of COVID-19 or proved to carry other etiologies did not necessarily undergo repeat RT-PCR.

Cases included in a previous publication were excluded from the current study based on the following grounds: (1) the previous publication was used in the process of developing two of the sets of criteria (the RSNA expert consensus statement and COVID-RADS), (2) those included in the previous publication were cases from mass infection cohort under special circumstances, and (3) the purpose of this study was to compare the CT grading systems in usual clinical settings that mostly comprises community-acquired infection with COVID-19. Furthermore, patients with COVID-19 were randomly selected and excluded to adjust the sample size. Fifty patients with COVID-19 and 50 patients without COVID-19 were finally included as case and control subjects, respectively. Flow chart illustrating the patient population was summarized in Figure 1.

Figure 1.

Flow chart illustrating the patient selection.

Clinical Data

Medical records were reviewed for the clinical and imaging findings of patients. The following data were extracted from the medical records: demographic data, presence or absence of smoking history, underlying comorbidities, symptoms and signs, and duration from onset to CT.

Chest CT Acquisition

Non-enhanced chest CT was performed using a 6-row multi-detector CT unit (SOMATOM Emotion 6 scanner; Siemens, Tokyo, Japan) on admission with the following parameters: tube voltage, 130 kVp; effective current 95 mA; collimation, 6×2 mm, helical pitch, 1.4, field of view, 38 cm; matrix size, 512×512. A 1.0-mm gapless section was reconstructed before being reviewed on the picture archiving and communication system monitor.

CT Image Interpretation

The comparison of four sets of CT grading systems were summarized in Table 1. Eight general radiologists (W.G., Y.W., Y.N., R.K., M.K., K.S., H.S., A.F.) served as observers from 5 hospitals in Japan. Four observers had more than 10 years of experience and four less than 10 years. Because none had experience with CO-RADS, COVID-RADS, the RSNA expert consensus statement or the BSTI guidance statement prior to this study, we held a practice review session and consensus meeting to review the statements before the initiation of the experiments. In the practice session, each reader independently scored 40 sample cases of COVID-19, which were not included in this study and recorded the reasons for grading and uncertainty (if any). In the following consensus meeting, we decided on a single consensus grading for each case by majority voting followed by a review of cases of interobserver disagreements and listed unclear/unspecified components in each item of the grading systems (Table 2). Based on these results, we added some supplementary remarks regarding interpretation of criteria and sample CT patterns to facilitate CT grading without confusion or misclassification as follows; (1) added conjunction (and/or) for each item of the criterion, (2) confirmed interpretation of the descriptions regarding laterality (i.e. each category was interpreted as “either unilateral or bilateral” if otherwise specified), (3) confirmed categorization of frequently encountered differential diagnosis of COVID-19 for each criterion (e.g. interstitial pulmonary edema falls into COVID-RADS 1 with only interstitial septal thickening and/or pleural effusion; COVID-RADS 2A when accompanied by peribronchial edema; and COVID-RADS 2B when progressed to pulmonary alveolar edema accompanying GGO and pleural effusion), (4) created sample CT patterns of CO-RADS with regards to its categorization of GGO.

Table 1.

Comparison of the CT grading systems of COVID-19

Table 2.

Summary of reasons for interobserver disagreements of the CT grading systems of COVID-19

Randomization was stratified by the patient’s disease status (patients with COVID-19 vs. patients without COVID-19). All patient information was removed from the data and observers were blinded to all clinical information, including RT-PCR results. Each observer independently scored the four criteria using an original document of each criterion with the above-described additional notes and recorded all data using a spreadsheet prepared in advance (Microsoft, Redmond, WA, USA). Grading was conducted in four separate sessions designated for each criterion, in which the observers provided a single grading for one case (i.e. in the CO-RADS session, the observers provided only CO-RADS gradings, and in the order of CO-RADS session, COVID-RADS session, RSNA session, and BSTI session). The order of the cases was shuffled between sessions.

Statistical Analysis

The AUC was calculated for each sets of criteria for each of the observers. Using the RT-PCR results as the gold standard of COVID-19 diagnosis, the AUC was used to assess the performance of each of the three sets of criteria. Mean AUC across observers and 95% confidence interval (CI) were calculated. Last, for each criterion, the average percentage of patients assigned to each category, including 95% CI, was determined. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated by setting different cut-off points for each criterion.

Interobserver agreement was quantified using three types of kappa values (i.e. Fleiss’ kappa, Cohen’s kappa, and Light’s kappa [10]) calculated across observers. In comparison to the original article, Cohen’s kappa values were obtained by comparing the scores of each observer to the median of the remaining seven observers [5]. Overall agreement was quantified using Fleiss’ kappa and Light’s kappa. The degree of interobserver agreement was considered with the following interval of kappa: 0–0.20 for poor, 0.21–0.40 for fair, 0.41–0.60 for moderate, 0.61–0.80 for good, and 0.81–1.00 for excellent agreement.

Statistical analysis was done using the JMP statistical software program (JMP Pro, version 15.0.0; SAS, Cary, NC, USA), R software (R version 3.6.2, The R Foundation for Statistical Computing, Vienna, Austria). Quantitative variables were expressed as mean ± standard deviation (range) or median and interquartile range based on the normality of data. Categorical variables were presented as the percentage of the total. The comparisons of quantitative variables were evaluated using a non-paired t-test or Mann-Whitney U-test and categorical data using the Pearson χ2 test. The comparisons of AUC and kappa-values were conducted using one-way repeated measures analysis of variance, according to the normal distribution assessed by the Shapiro-Wilk test, and post hoc family-wise error correction for multiple comparisons with paired-t-test. All P values correspond to two-sided tests and the statistical significance level was set at Holm-Bonferroni-corrected P <.05.

Results

Clinical Findings

Demographics and clinical characteristics of the study population are summarized in Table 3. The study population comprised 100 patients, 50 patients with COVID-19 (38 men; mean age, 59 years ±17; range, 18–86) and 50 patients without COVID-19 (30 men; mean age, 65 years ±24, range; 17-100). There was no statistical significance between these groups in age, sex, disease duration, smoking history, or presence of comorbidities.

Table 3.

Characteristics of the patient cohort

Diagnostic performances of the criteria

Eight observers scored 100 patients, making for a total of 800 observations for each criterion. The probability of COVID-19 diagnosis of each category of each set of criteria was summarized in Table 4. The level of suspicion on COVID-19 correlated with the RT-PCR positive rate except for being negative for pneumonia in all the systems (Spearman’s coefficient: ρ = 1.0, P < .001 for all the systems). The diagnostic performance and interobserver agreements of each set of the criterion were summarized in Table 5. Average AUCs with 95% CI for each criterion were as follows; CO-RADS, 0.84 (0.83–0.85): COVID-RADS, 0.80 (0.78–0.81): RSNA grading system, 0.81 (0.79–0.82): and BSTI grading system, 0.84 (0.82-0.86). The AUC values were significantly higher in CO-RADS and BSTI grading system vs. COVID-RADS (vs CO-RADS, P =.0087 and vs BSTI grading system, P =.0033) and RSNA grading system (vs CO-RADS, P =.0097 and vs BSTI grading system, P =.0019). AUC values were not statistically significant in either of the comparisons between CO-RADS and BSTI grading system or COVID-RADS and the RSNA grading system.

Table 4.

RT-PCR positive and negative rates for each category

Table 5.

Comparison of AUC and Cohen's kappa-values between the four criteria

The sensitivity, specificity, PPV, and NPV of each set of criteria was summarized in Table 6. For CO-RADS, the sensitivity, specificity, PPV and NPV was as follows: CO-RADS 5, sensitivity 64.5% (258/400), specificity 89.0% (356/400), PPV 85.4% (258/302), and NPV 71.5% (356/498); CO-RADS 4 or 5, sensitivity 85.5% (342/400), specificity 68.3% (273/400), PPV 72.9% (342/469), and NPV 82.5% (273/331); CO-RADS 3, 4 or 5, sensitivity 91.0% (364/400), specificity 53.8% (215/400), PPV 66.3% (364/549), and NPV 85.7% (215/251). Regarding COVID-RADS, the sensitivity, specificity, PPV and NPV were as follows: COVID-RADS 3, sensitivity 65.5% (262/400), specificity 90.0% (360/400), PPV 86.8% (262/302), NPV 72.3% (360/498); COVID-RADS 2A or 3, sensitivity 69.8% (279/400), specificity 83.0% (332/400), PPV 80.4% (279/347), and NPV 73.3% (332/453); COVID-RADS 2B, 2A, or 3, sensitivity 93.0% (372/400), specificity 29.0% (116/400), PPV 56.7% (372/656), and NPV 80.6% (116/144). The RSNA expert consensus statement achieved the following sensitivity, specificity, PPV and NPV: “typical appearance”, sensitivity 73.5% (294/400), specificity 82.8% (331/400), PPV 81.0% (294/363), and NPV 75.7% (331/437); and “typical appearance” or “indeterminate appearance”, sensitivity 92.0% (368/400), specificity 41.0% (164/400), PPV 60.9% (368/604), and NPV 83.7% (164/196). Finally, the BSTI guidance statement marked the following sensitivity, specificity, PPV and NPV: “CLASSIC COVID-19”, sensitivity 64.5% (258/400), specificity 94.0% (376/400), PPV 91.5% (258/282), and NPV 72.6 (376/518)%; “CLASSIC COVID-19” or “PROBABLE COVID-19”, sensitivity 71.3% (285/400), specificity 87.3% (349/400), PPV 84.8% (285/336), and NPV 75.2% (349/464); “CLASSIC COVID-19”, “PROBABLE COVID-19”, or “INDTERMINATE”, sensitivity 91.3% (365/400), specificity 44.8% (179/400), PPV 62.3% (365/586), and NPV 83.6% (179/214).

Table 6.

Sensitivity, specificity, and positive and negative predictive values of the four criteria.

Interobserver variabilities of the criteria

The interobserver variabilities of COVID-19 diagnosis with 95%CI of each grading system were summarized in Table 5. The overall agreement was good; the average Cohen’s kappa in all observers was 0.62 (95%CI: 0.58–0.66), 0.63 (95%CI: 0.58–0.68), 0.63 (95%CI: 0.57–0.69) and 0.61 (0.58-0.64) for CO-RADS, COVID-RADS, RSNA grading system, and BSTI grading system, respectively, which was not statistically significant from each other. Light’s kappa was 0.56 (95%CI: 0.49-0.63) for CO-RADS, 0.55 (0.49-0.62) for COVID-RADS, 0.55 (0.49-0.62) for RSNA grading system, and 0.54 (0.48-0.62) for BSTI grading system.

Details of all 800 observations by 8 observers were summarized in Supplementary Table 1. Associations between each set of criteria were summarized in Supplementary Table 2-7.

Discussion

We conducted a comparison study of the published CT grading systems CO-RADS and COVID-RADS, and the grading system based on the RSNA expert consensus statement and the BSTI guidance statement. Although the three sets of criteria were effective in diagnosing COVID-19 with a mean AUC of about 0.80, a salient finding was that CO-RADS and the BSTI guidance statement were significantly better in distinguishing COVID-19 from non-COVID-19 etiology than COVID-RADS and the RSNA consensus statement.

All the sets of criteria were effective in terms of interobserver agreements with an average Cohen’s kappa greater than 0.60. For CO-RADS, interobserver agreement was higher in the current study than in the original one (Fleiss’ kappa of 0.56 vs. 0.47) [5]. For the RSNA expert consensus statement, the interobserver agreement was also higher in the current study than in a previous one (Cohen’s kappa of 0.63 vs. 0.5) [10]. We attribute this to our having held a practice session using sample cases and a rigorous consensus meeting to minimize ambiguity and dependence on subjective clinical judgments and facilitate consistent image interpretation. Based on this experience we also added some remarks regarding the interpretation of these criteria to enhance clarity as summarized in Table 2, with some illustrative cases shown in Figure 2-4. As detailed in Table 2, the reasons for dividing the grading common to all the grading systems and hence candidates for possible revision were (1) the presence of more than two predominant patterns, (2) the presence of only small lesions, (3) the presence of co-existing lung disease (i.e. interstitial pneumonia or emphysema). Although some ambiguity in the terminology used may persist (i.e. (half)-rounded shape, small, homogeneous, or extensive) and a little additional modification may be required, further efforts to define the characteristics in ever greater detail may not be worthwhile considering the wide spectrum of radiological presentations and progression of COVID-19.

Figure 2a.

Cases illustrative of downgrading in COVID-RADS despite typical patterns seen in COVID-19. (a) An axial chest CT image of a 72-year-old male with COVID-19 shows peripheral, bilateral multifocal GGOs (arrows) with a background of mild emphysema. The consensus grading was CO-RADS 5, COVID-RADS 3+1 (downgraded to 2B), typical appearance in the RSNA expert consensus statement, and classic COVID-19 in the BSTI guidance statement. (b) An axial chest CT image of a 76-year-old female with COVID-19 shows peripheral, bilateral, multifocal GGOs with left pleural effusion (arrow). The consensus grading was CO-RADS 5, COVID-RADS 3+1 (downgraded to 2B), typical appearance in the RSNA expert consensus statement, and classic COVID-19 in the BSTI guidance statement. (c) An axial chest CT image of a 49-year-old male with COVID-19 shows peripheral, bilateral, multifocal GGOs with mild mediastinal lymph node enlargements (arrows). The consensus grading was CO-RADS 5, COVID-RADS 3+1 (downgraded to 2B), typical appearance in the RSNA expert consensus statement, and classic COVID-19 in the BSTI guidance statement.

Figure 4a.

Cases illustrative of disaccord in gradings between each set of criteria. (a) An axial chest CT image of a 24-year-old male without COVID-19 shows segmental GGOs and consolidation with bronchial wall thickening (dotted circle). The consensus grading was CO-RADS 2, COVID-RADS 3, indeterminate appearance in the RSNA expert consensus statement, indeterminate in the BSTI guidance statement. As illustrated in this case, if observers consider the findings compatible with typical bronchopneumonia, CO-RADS enables classification as a lower grade than do the other three sets of criteria. (b) An axial chest CT image of a 35-year-old female with COVID-19 shows multifocal peribronchovascular segmental centrilobular GGOs with bronchial wall thickening (dotted circle). There were centrilobular GGOs on other images (not shown). The consensus grading was CO-RADS 2, COVID-RADS 2B, atypical appearance in the RSNA expert consensus statement, and indeterminate in the BSTI guidance statement. As illustrated in this case, lesions with centrilobular distribution may be underestimated in CO-RADS and the RSNA expert consensus statement. (c) An axial chest CT image of a 28-year-old male with COVID-19 shows peripheral, multifocal rounded GGOs with visible intralobular lines in the left lower lobe (dotted circle). The consensus grading was CO-RADS 4, COVID-RADS 3, typical appearance in the RSNA expert consensus statement, and classic COVID-19 in the BSTI guidance statement. As illustrated in this case, CO-RADS classifies lesions with unilateral distribution as grade 4 despite their typical appearance seen in COVID-19, while the other three sets of criteria assigning it to the respective highest grade.

Cases illustrative of upgrading encountered in COVID-RADS. (a) An axial chest CT image of a 79-year-old female without COVID-19 shows consolidation, and centrilobular nodules without GGOs in the right lower lobe, representing typical appearances in bronchial pneumonia (dotted circle). There was bronchial wall thickening on other images (not shown). The consensus grading was CO-RADS 2, COVID-RADS 2A+1 (=2B), atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (b) An axial chest CT image of a 52-year-old male without COVID-19 shows segmental consolidation, reflecting typical appearance in lobar pneumonia (dotted circles). There was bilateral pleural effusion in this case (not shown in this figure), and the consensus grading was CO-RADS 2, COVID-RADS 2A+1 (=2B), atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (c) An axial chest CT image of a 23-year-old male without COVID-19 shows diffuse moderate bronchial wall thickening (arrows). The consensus grading was CO-RADS 1, COVID-RADS 2A, negative in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (d) An axial chest CT image of a 79-year-old male without COVID-19 shows bronchial wall thickening (arrow) but is otherwise normal. Bilateral gravity-dependent ground-glass opacities were considered non-pathological changes. The consensus grading was CO-RADS 2, COVID-RADS 2A, atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. As illustrated in this case, COVID-RADS classifies bronchial thickening into higher grades than the other three sets of criteria.

Cases illustrative of upgrading encountered in COVID-RADS. (a) An axial chest CT image of a 79-year-old female without COVID-19 shows consolidation, and centrilobular nodules without GGOs in the right lower lobe, representing typical appearances in bronchial pneumonia (dotted circle). There was bronchial wall thickening on other images (not shown). The consensus grading was CO-RADS 2, COVID-RADS 2A+1 (=2B), atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (b) An axial chest CT image of a 52-year-old male without COVID-19 shows segmental consolidation, reflecting typical appearance in lobar pneumonia (dotted circles). There was bilateral pleural effusion in this case (not shown in this figure), and the consensus grading was CO-RADS 2, COVID-RADS 2A+1 (=2B), atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (c) An axial chest CT image of a 23-year-old male without COVID-19 shows diffuse moderate bronchial wall thickening (arrows). The consensus grading was CO-RADS 1, COVID-RADS 2A, negative in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (d) An axial chest CT image of a 79-year-old male without COVID-19 shows bronchial wall thickening (arrow) but is otherwise normal. Bilateral gravity-dependent ground-glass opacities were considered non-pathological changes. The consensus grading was CO-RADS 2, COVID-RADS 2A, atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. As illustrated in this case, COVID-RADS classifies bronchial thickening into higher grades than the other three sets of criteria.

Cases illustrative of upgrading encountered in COVID-RADS. (a) An axial chest CT image of a 79-year-old female without COVID-19 shows consolidation, and centrilobular nodules without GGOs in the right lower lobe, representing typical appearances in bronchial pneumonia (dotted circle). There was bronchial wall thickening on other images (not shown). The consensus grading was CO-RADS 2, COVID-RADS 2A+1 (=2B), atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (b) An axial chest CT image of a 52-year-old male without COVID-19 shows segmental consolidation, reflecting typical appearance in lobar pneumonia (dotted circles). There was bilateral pleural effusion in this case (not shown in this figure), and the consensus grading was CO-RADS 2, COVID-RADS 2A+1 (=2B), atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (c) An axial chest CT image of a 23-year-old male without COVID-19 shows diffuse moderate bronchial wall thickening (arrows). The consensus grading was CO-RADS 1, COVID-RADS 2A, negative in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (d) An axial chest CT image of a 79-year-old male without COVID-19 shows bronchial wall thickening (arrow) but is otherwise normal. Bilateral gravity-dependent ground-glass opacities were considered non-pathological changes. The consensus grading was CO-RADS 2, COVID-RADS 2A, atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. As illustrated in this case, COVID-RADS classifies bronchial thickening into higher grades than the other three sets of criteria.

Cases illustrative of upgrading encountered in COVID-RADS. (a) An axial chest CT image of a 79-year-old female without COVID-19 shows consolidation, and centrilobular nodules without GGOs in the right lower lobe, representing typical appearances in bronchial pneumonia (dotted circle). There was bronchial wall thickening on other images (not shown). The consensus grading was CO-RADS 2, COVID-RADS 2A+1 (=2B), atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (b) An axial chest CT image of a 52-year-old male without COVID-19 shows segmental consolidation, reflecting typical appearance in lobar pneumonia (dotted circles). There was bilateral pleural effusion in this case (not shown in this figure), and the consensus grading was CO-RADS 2, COVID-RADS 2A+1 (=2B), atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (c) An axial chest CT image of a 23-year-old male without COVID-19 shows diffuse moderate bronchial wall thickening (arrows). The consensus grading was CO-RADS 1, COVID-RADS 2A, negative in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. (d) An axial chest CT image of a 79-year-old male without COVID-19 shows bronchial wall thickening (arrow) but is otherwise normal. Bilateral gravity-dependent ground-glass opacities were considered non-pathological changes. The consensus grading was CO-RADS 2, COVID-RADS 2A, atypical appearance in the RSNA expert consensus statement, and non-COVID in the BSTI guidance statement. As illustrated in this case, COVID-RADS classifies bronchial thickening into higher grades than the other three sets of criteria.

The diagnostic performance of CO-RADS was slightly lower than that noted in the original article (AUC of 0.84 vs. 0.91, this study vs. the CO-RADS original study [5]). This discrepancy was attributed in part to differences in the patient cohorts studied in the original and present works. Reflecting this, as illustrated in Table 4, in this study cohort CO-RADS 1 categories showed a slightly higher RT-PCR positive rate than CO-RADS 2, while the original paper obtained the opposite result with this trend likewise observed in the other two sets of criteria). This may have resulted from differences in the inclusion criteria between the two studies; namely only patients requiring hospitalization were included in the original article, while we adopted broader inclusion criteria, namely all patients irrespective of their observation status in the present one. The difference is easy to understand since patients with COVID-19 has been reported to show different percentages of chest CT positivity in rough parallel with the severity of their symptoms: normal CT findings being observed in 46% of asymptomatic or mildly symptomatic and 21% of symptomatic patients [12]. The results of the current study confirmed that the sensitivity of chest CT is still not sufficient to use as a rule-out-tool. In contrast, the interobserver agreement was higher in the current study (Fleiss’ kappa of 0.56 vs. 0.47, the current study vs. the CO-RADS original study [5]).

COVID-RADS is a relatively simple grading system based on a combination of findings with different levels of suspicion that are stratified according to their frequencies seen in COVID-19. This set of criteria, therefore, differs from the other two sets, in that it does not stratify lesions up to their axial zonal distribution (i.e., central, peripheral, or mixed distribution). However, axial zonal distribution is one of the most conspicuous and specific findings of COVID-19 as documented in previous publications [13-17]. Another limitation of this set of criteria may be related to the designation of multifocal GGO as a “typical finding.” Several other diseases, including viral pneumonia of non-COVID-19 etiology, bronchial pneumonia, nonspecific interstitial pneumonia, acute interstitial pneumonitis (IP), pneumocystis jiroveci pneumonia, and drug-induced pneumonitis may show similar multifocal GGOs. In addition, discussions are needed as to whether a variety of COVID-19 and other non-COVID-19 etiologies should not best be placed in grade 2B. One example is that lesions with typical findings are downgraded to grade 2B when concomitant minor findings exist i.e. small amount of pleural effusion or emphysema, small nodules (e.g., intrapulmonary lymph nodes). However, in our experience, such co-existence is common and has been pointed out elsewhere as well [18-20].

Compared to the other two sets of criteria based on a systematic grading system, the RSNA expert consensus statement and the BSTI guidance statement rely on groups of findings taking types of lesions, numbers, and distribution into consideration together. Based on gestalt imaging interpretation of the CT findings, these systems are easy to understand and put into practice and facilitates communication with physicians in other fields. As detailed in Table 2, one potential limitation of the RSNA consensus statement that may affects its interobserver agreements and diagnostic performance is that it does not address the presence of any co-existing lung diseases. However, as previously reported, COVID-19 pneumonia often mimics acute aggravation of IP or emphysema when superimposed on a background of IP or emphysema, thereby often causing an additional diagnostic burden (Figure 5). In contrast, the BSTI guidance statement downgrades the characteristic CT patterns ("CLASSIC COVID” or “PROBABLE COVID” classifications) when they are accompanied by other cardiopulmonary diseases (e.g. IP) to “indeterminate” classification [3]. This may explain the results of this study in which the RSNA expert consensus statement showed a higher sensitivity than the BSTI guidance statement; and in contrast, the BSTI guidance statement showed a higher specificity than the RSNA consensus statement in diagnosing COVID-19. In comparison to a previous publication on the RSNA grading system, the current study obtained comparable results with a sensitivity of 73.5% (vs. 71.6%), specificity 82.8% (vs.91.6%), and PPV 81.0% (vs. 87.8%) [10]. Previous studies also reported a similar trend observed in the current study in terms of RT-PCR positive rates for each category of the RSNA grading system with the “negative for pneumonia” classification being more frequent than “atypical” classification [9-10].

Figure 5a.

Cases illustrative of diagnostic challenges with severe pre-existing pulmonary abnormalities. (a) An axial chest CT image of a 91-year-old male without COVID-19 shows peripheral multifocal GGOs (dotted circles) in the background of usual interstitial pneumonia and moderate emphysema. There was a definite honeycomb lung destruction on other images (not shown). The consensus grading was CO-RADS 4, COVID-RADS 2B, typical appearance in the RSNA expert consensus statement, and indeterminate in the BSTI guidance statement. (b) An axial chest CT image of a 57-year-old male with COVID-19 shows severe emphysema and airspace expansion with fibrosis. The consensus grading was CO-RADS 1, COVID-RADS 1, negative appearance in the RSNA expert consensus statement, and indeterminate in the BSTI guidance statement.

This study has various limitations. First, because of its retrospective nature, a selection bias may have been introduced. Second, the interpretation of the criteria may have been affected by our addition of supplementary remarks in an attempt to reduce ambiguity. One concern is that it may not have been consistent with the intent of the researchers who originally proposed them. Third, the impact of this study may also have suffered from having been conducted at a single institution. However, we included eight readers with varying degrees of experience who practice at five different institutions so as to represent a broad population of readers. Fourth, only symptomatic patients were included, thereby potentially biasing the sensitivity and specificity calculated in this study. Fifth, some patients with single negative RT-PCR results were deemed negative for COVID-19. Considering the variety in the sensitivity of RT-PCR ranging from 67-98%, false negative cases were not eliminated with a single RT-PCR negative result [21]. Sixth, we did not keep the interval time constant between each session in the interpretation experiment. Ideally, the interval time should have been kept constant between each of these sessions to allow the observers to forget the details of individual cases to avoid bias, but in this study, we decided not do so to facilitate the swiftest possible publication of these findings given the pressing COVID-19 pandemic.

In conclusion, CO-RADS, COVID-RADS, the RSNA expert consensus statement, and the BSTI guidance statement provided reasonable performances and interobserver agreements in reporting the CT findings of COVID-19. Further studies will be needed to further define the clinical implications of these systems in the diagnosis of COVID-19 in a more diverse population.