Performance of Radiologists in Differentiating COVID-19 from Non-COVID-19 Viral Pneumonia at Chest CT.

Background Despite its high sensitivity in diagnosing coronavirus disease 2019 (COVID-19) in a screening population, the chest CT appearance of COVID-19 pneumonia is thought to be nonspecific. Purpose To assess the performance of radiologists in the United States and China in differentiating COVID-19 from viral pneumonia at chest CT. Materials and Methods In this study, 219 patients with positive COVID-19, as determined with reverse-transcription polymerase chain reaction (RT-PCR) and abnormal chest CT findings, were retrospectively identified from seven Chinese hospitals in Hunan Province, China, from January 6 to February 20, 2020. Two hundred five patients with positive respiratory pathogen panel results for viral pneumonia and CT findings consistent with or highly suspicious for pneumonia, according to original radiologic interpretation within 7 days of each other, were identified from Rhode Island Hospital in Providence, RI. Three radiologists from China reviewed all chest CT scans (n = 424) blinded to RT-PCR findings to differentiate COVID-19 from viral pneumonia. A sample of 58 age-matched patients was randomly selected and evaluated by four radiologists from the United States in a similar fashion. Different CT features were recorded and compared between the two groups. Results For all chest CT scans (n = 424), the accuracy of the three radiologists from China in differentiating COVID-19 from non-COVID-19 viral pneumonia was 83% (350 of 424), 80% (338 of 424), and 60% (255 of 424). In the randomly selected sample (n = 58), the sensitivities of three radiologists from China and four radiologists from the United States were 80%, 67%, 97%, 93%, 83%, 73%, and 70%, respectively. The corresponding specificities of the same readers were 100%, 93%, 7%, 100%, 93%, 93%, and 100%, respectively. Compared with non-COVID-19 pneumonia, COVID-19 pneumonia was more likely to have a peripheral distribution (80% vs 57%, P < .001), ground-glass opacity (91% vs 68%, P < .001), fine reticular opacity (56% vs 22%, P < .001), and vascular thickening (59% vs 22%, P < .001), but it was less likely to have a central and peripheral distribution (14% vs 35%, P < .001), pleural effusion (4% vs 39%, P < .001), or lymphadenopathy (3% vs 10%, P = .002). Conclusion Radiologists in China and in the United States distinguished coronavirus disease 2019 from viral pneumonia at chest CT with moderate to high accuracy. © RSNA, 2020 Online supplemental material is available for this article. A translation of this abstract in Farsi is available in the supplement. ترجمه چکیده این مقاله به فارسی، در ضمیمه موجود است.

Summary

Radiologists had high specificity but moderate sensitivity in differentiating COVID-19 from viral pneumonia on chest CT.

■ Three Chinese radiologists had sensitivities of 72%, 72% and 94% and specificities of 94%, 88% and 24% in differentiating 219 COVID-19 from 205 non-COVID-19 pneumonia.

■ Four United States radiologists had sensitivities of 93%, 83%, 73% and 73% and specificities of 100%, 93%, 93% and 100%.

■ The most discriminating features for COVID-19 pneumonia included a peripheral distribution (80% vs. 57%, p<0.001), ground-glass opacity (91% vs. 68%, p<0.001) and vascular thickening (58% vs. 22%, p<0.001).

Introduction

Since the initial outbreak of Coronavirus disease-19 (COVID-19) from Wuhan, China in late December 2019 (1), there have been 87,137 confirmed cases and 2,873 reported deaths distributed across 60 countries as of March 1st 2020 (2, 3). China has had the majority of COVID 19 cases (92%) (3). Patients infected with COVID-19 typically present with fever, cough, dyspnea, and muscle aches while imaging frequently reveals bilateral pneumonia (5).

The standard diagnostic method being used is real-time polymerase chain reaction (RT-PCR) to detect viral nucleotides from specimens obtained by oropharyngeal swab, nasopharyngeal swab, bronchoalveolar lavage, or tracheal aspirate (6). However, recent reports have revealed that RT-PCR has a sensitivity as low as 60-71% for detecting COVID-19 (5, 7, 8), which can possibly be attributed to low viral load present in test specimens or laboratory error (7, 9). These false negatives hinder quarantine efforts, necessitate repeat testing and has the potential to overload the current supply of testing kits and related infrastructure (8). By contrast, chest CT has demonstrated about 56-98% sensitivity in detecting COVID-19 at initial presentation and can be helpful in rectifying false negatives obtained from RT-PCR during early stages of disease development (7, 8). Chest CT of COVID-19 patients reveals areas of consolidation and ground-glass opacity (GGO) with bilateral peripheral involvement in multiple lobes progressing to “crazy-paving” patterns and consolidation. CT signs gradually improve beginning approximately 14 days post-symptom onset (10, 11, 12).

Despite its high sensitivity in diagnosing COVID-19 in a screening population, chest CT had low specificity (25%) in a recent report of 1014 patients with COVID-19 (5). Prior studies have not directly compared chest CT patterns of COVID-19 from viral pneumonia on chest CT. The purpose of this study was to assess the performance of United States (U.S.) and Chinese radiologists in differentiating COVID-19 from viral pneumonia on chest CT.

Materials and Methods

Patient cohort

The institutional review board of all seven hospitals in Hunan Providence, China and Rhode Island Hospital from the U.S. approved this retrospective study and written informed consent was waived. A total of 256 patients with both positive COVID-19 by RT-PCR and chest CT imaging within two weeks were retrospectively identified from 7 Chinese hospitals in Hunan Providence, China from January 6 to February 20, 2020. The RT-PCR results were extracted from the patients’ electronic medical records in the hospital information system (HIS). The RT-PCR assays were performed by using TaqMan One-Step RT-PCR Kits from Shanghai Huirui Biotechnology Co., Ltd or Shanghai BioGerm Medical Biotechnology Co., Ltd, both of which have approved use by the China Food and Drug Administration (CFDA). For patients with multiple RT-PCR assays, positive result on the last performed test was adopted as confirmation of diagnosis. The number of cases included from each hospital is shown in Supplementary Table 1. Among these patients, 37 with negative chest CT were excluded, resulting in a final cohort of 219 patients. The chest CT protocols from the 7 hospitals are shown in Supplementary Table 2.

The radiology search engine MONTAGE (Nuance Communications, Burlington, MA) at Rhode Island Hospital in Providence, RI was used to identify cases that contain the word “pneumonia” in the impression section of the radiology CT reports from 2017 to 2019. The identified CT scans were directly downloaded from the hospital Picture Archiving and Communications Systems (PACS) and non-chest CTs were excluded. Positive results from Respiratory Pathogen Panel (RPP) were used to locate patients of possible viral pneumonia from 2017 to 2019. The tests of ePlex Respiratory Pathogen panel (GenMark Diagnostics, Carlsbad, CA) were performed in the Microbiology Lab of Rhode Island Hospital Pathology Department according to it manufacture protocol (13). A diagram illustrating the initial breakdown of RPP search results is shown in Supplementary Figure 1. The two lists were cross-matched to generate a final list that contained CT chest scans with the word “pneumonia” in the final impression and positive RPP test within 7 days of each other. Then, the impression sections of these CT reports were reviewed by a research assistant (BH) and a radiologist (HXB) board-certified in general diagnostic radiology and interventional radiology with one year of practice experience to identify 205 cases with final CT impression being “consistent with” or “highly suspicious for” pneumonia. Our final cohort consisted of 424 patients. A diagram illustrating patient inclusion and exclusion is shown in Figure 1. A diagram illustrating the final breakdown of RPP results is shown in Figure 2.

Figure 1:

Illustration of study flow RIH: Rhode Island Hospital

Figure 2:

Distribution of viral pathogens based on RPP test in the final cohort. It is an FDA approved assay that simultaneous detects 19 viruses (influenza A virus; influenza A H1 virus; influenza A 2009 H1 virus; influenza A H3 virus; influenza B virus; adenovirus; coronaviruses [HKU1, OC43, NL63, and 229E]; human rhinovirus/enterovirus; human metapneumovirus; parainfluenza viruses 1, 2, 3, and 4; and respiratory syncytial virus [RSV] [RSV subtype A and RSV subtype B]) and 2 bacteria (Mycoplasma pneumoniae and Chlamydia pneumoniae). The ePlex panel (16) has been proven to be a highly sensitive and specific multiplex assay for respiratory pathogen detection. In a multicentric study, the positive percent agreement values (equivalent to sensitivity when a perfect reference method is unavailable) ranged from 85.1% to 95.1% and the negative percent agreement values (equivalent to specificity) ranged from 99.5% to 99.8% when compared to another well-established RP panel from BioFire (Salt Lack City, UT) (17).

Radiologist interpretation

Three Chinese radiologists, who were blinded to RT-PCR results, reviewed all chest CT images and scored each case as COVID-19, pneumonia of other etiology or neither. Fifty-eight cases age-matched between the COVID-19 and non-COVID-19 pneumonia groups were randomly selected from the entire cohort and evaluated in the same way by four U.S. radiologists. All identifying information was removed from the CT studies, which were shuffled and uploaded to 3D slicer for interpretation. Information on the radiologists regarding location of practice, years in practice, cardiothoracic imaging fellowship and COVID-10 specific training experience is shown in Table 1.

Table 1:

Backgrounds of radiologists who reviewed the cases

CT features

Different CT features on the entire cohort were recorded by two Chinese radiologists in consensus. If consensus could not be reached, it was resolved by a senior radiologist (X.Z.) with more than 10 years of chest CT experience.

Statistical analysis

Continuous variables were expressed as medians and ranges, while categorical variables as counts and percentages. Metrics such as sensitivity, specificity, positive predictive value, negative predictive value, and accuracy were calculated to evaluate the diagnostic performance of the radiologists. For the calculations, COVID-19 was considered positive result, while pneumonia of other etiology and neither were considered negative result. Exact binomial 95% confidence intervals were calculated for sensitivity, specificity, PPV, NPV, and accuracy using the epiR package in the R statistical computing language (R version 3.4.2, The R Foundation for Statistical Computing, Vienna, Austria; https://www.r-project.org). P values of <0.05 were considered statistically significant.

Results

Our final cohort consisted of 424 patients, including 205 patients with non-COVID-19 pneumonia from the U.S. and 219 patients with COVID-19 from China. The average number of days between CT and positive RT-PCR test result was 4.1±4.4 days for the COVID-19 group; the average number of days between CT and positive RPP test result was 1.0±1.7 days. Patients with COVID-19 were younger (mean age of 45 vs. 65 years, p<0.001) and less likely to have elevated white blood cell count (29% vs. 55%, p<0.001), but more likely to have reduced lymphocyte count (16% vs. 44%, p<0.001). Clinical characteristics including comorbidities comparing the two groups are shown in Table 2.

Table 2:

Clinical characteristics of the US non-COVID-19 and Chinese COVID-19 patient cohorts

For the entire cohort, the accuracy of the three Chinese radiologists to identify COVID-19 from non-COVID-19 pneumonia was 83% (95 CI: 79-86%), 80% (95% CI: 76-83%), and 60% (95% CI: 55-65%), respectively (Table 3). Sensitivity ranged from 72 to 94%, and specificity showed great variation (24 to 94%) (Table 3).

Table 3:

Performance results of the three Chinese radiologists who reviewed all 424 cases

In a randomly selected, age-matched cohort of 58 patients, the accuracy of four U.S. radiologists to differentiate COVID-19 from non-COVID-19 pneumonia was 97% (95% CI: 88-100%), 88% (95 CI: 77-95%), 83% (95% CI: 71-91%), and 84% (95% CI: 73-93%), respectively (Table 4). Sensitivity ranged from 70 to 93%, and specificity ranged from 93 to 100% (Table 4).

Table 4:

The results of 3 Chinese radiologists and 4 U.S. radiologists who reviewed 58 age-matched cases.

Compared to non-COVID-19 pneumonia, COVID-19 pneumonia was more likely to have a peripheral distribution (80% vs. 57%, p<0.001), ground-glass opacity (91% vs. 68%, p<0.001), fine reticular opacity (56% vs. 22%, p<0.001), vascular thickening (59% vs. 22%, p<0.001) and reverse halo sign (11% vs. 1%, p=0.005), but less likely to have a central+peripheral distribution (14% vs. 35%, p<0.001), air bronchogram (14% vs. 23%, p=0.014), pleural thickening (15 vs. 33%, p<0.001), pleural effusion (4 vs. 39%, p<0.001) and lymphadenopathy (2.7% vs. 10.2%, p<0.001) (Table 5).

Table 5:

CT features comparing COVID-19 and viral pneumonia

Figure 3 demonstrates example cases in which the majority of the radiologists mistook non-COVID-19 pneumonia for COVID-19 infection (A-B) or vice versa (C-H).

Figure 3a:

A. Cases in a 58-patient review process that are commonly misdiagnosed. Example cases where the majority of radiologists mistook non-COVID-19 pneumonia for COVID-19 (A and B) or other vice versa (C-H)

Discussion

COVID-19 is spreading rapidly worldwide while present diagnostic methods for identifying the virus have limitations. RT-PCR testing has low sensitivity early on in the disease course. Although chest CT has high sensitivity, it has low specificity (7, 11). This low specificity may stem from the fact that it is difficult to distinguish COVID-19 findings from findings of other disease on chest CT (5). To optimize patient management, medical care, and disease control it is important to determine the efficacy of chest CT in distinguishing COVID-19 from pneumonia of other etiologies by radiologists. The study conducted herein revealed that radiologists are capable of distinguishing COVID-19 from viral pneumonia on chest CT with high specificity but moderate sensitivity.

Healthcare providers being able to reliably differentiate COVID-19 from other causes of pneumonia on chest CT would benefit diagnostic workup of the disease by compensating for the poor sensitivity of RT-PCR, particularly during early disease stages. Although chest CT has demonstrated high sensitivity relative to RT-PCR testing for COVID-19 diagnosis, it may not reveal distinct patterns for COVID-19 in all cases. This can make it hard to distinguish COVID-19 from other causes of viral pneumonia. For example, influenza and COVID-19 both demonstrate GGO and consolidation on chest CT (14). Introducing the possibility of pathologies with similar chest CT findings to those of COVID-19 ultimately complicates the differential diagnosis. Our study is significant because it demonstrates that radiologists are capable of distinguishing COVID-19 from other etiology of pneumonia on chest CT with high specificity. This suggests that there is potential that if the differential diagnosis is between COVID-19 and non-COVID-19 pneumonia, a negative diagnosis of COVID-19 by radiologists on chest CT may be good enough to exclude patients from having the disease with fairly good certainty. Our analysis of specific cases where radiologists were wrong reveals that the mistakes were made when the COVID-19 chest CT findings are either subtle (likely reflecting early time in the disease process) (Figures 3C and 3D) or when COVID-19 has atypical chest CT findings (Figures 3E, 3F, 3G, and 3H). It is worth noting that non-COVID-19 pneumonia can also have typical appearance of COVID-19 (Figures 3A and 3B). This poses a dilemma because mandated quarantine for all suspected cases can put significant strain on medical infrastructure, healthcare providers, and the lives of patients, but may need to be followed as a necessary precaution due to variation in presentation with timing of disease and atypical findings on chest CT. Future direction includes development of an artificial intelligence classifier that can further augment radiologist performance in combination with clinical information.

Figure 3c:

A. Cases in a 58-patient review process that are commonly misdiagnosed. Example cases where the majority of radiologists mistook non-COVID-19 pneumonia for COVID-19 (A and B) or other vice versa (C-H)

Figure 3d:

A. Cases in a 58-patient review process that are commonly misdiagnosed. Example cases where the majority of radiologists mistook non-COVID-19 pneumonia for COVID-19 (A and B) or other vice versa (C-H)

Figure 3e:

A. Cases in a 58-patient review process that are commonly misdiagnosed. Example cases where the majority of radiologists mistook non-COVID-19 pneumonia for COVID-19 (A and B) or other vice versa (C-H)

Figure 3f:

A. Cases in a 58-patient review process that are commonly misdiagnosed. Example cases where the majority of radiologists mistook non-COVID-19 pneumonia for COVID-19 (A and B) or other vice versa (C-H)

Figure 3g:

A. Cases in a 58-patient review process that are commonly misdiagnosed. Example cases where the majority of radiologists mistook non-COVID-19 pneumonia for COVID-19 (A and B) or other vice versa (C-H)

Figure 3h:

A. Cases in a 58-patient review process that are commonly misdiagnosed. Example cases where the majority of radiologists mistook non-COVID-19 pneumonia for COVID-19 (A and B) or other vice versa (C-H)

Figure 3b:

A. Cases in a 58-patient review process that are commonly misdiagnosed. Example cases where the majority of radiologists mistook non-COVID-19 pneumonia for COVID-19 (A and B) or other vice versa (C-H)

Our study has several limitations. First, the cohort size was small, especially when it comes to cases reviewed by U.S. radiologists. There is selection bias associated with our screening strategy as well. It remains unclear if diagnostic outcomes would improve in a more well-balanced and larger-scale prospective study of similar design. It is also noteworthy that the U.S. radiologists in this study had minimal training specific to diagnosing COVID-19 and that Chinese radiologists practiced in an area with relatively low prevalence of the disease. It is possible that Chinese radiologists working near the epicenter of the disease with a higher degree of experience specific to COVID-19 would have performed significantly better than either group in the present study. In addition, although RPP test and chest CT findings within 7 days of symptom presentation were used to enrich our “pneumonia of other etiology” cohort with viral pneumonia cases, the cause and effect relationship is not 100%. Thus, it is possible that some of the selected patients had mixed viral and bacterial pneumonia or other diseases entirely. Finally, the radiologists were not given clinical information during the evaluation, which could have further improved their performance.

As more research is done, information may be gathered by providers to make this differential more facile to navigate. Until that point, it is recommended that individuals with signs of pneumonia on chest CT be quarantined while RT-PCR testing is performed in conjunction with a thorough medical evaluation including travel history and disease contacts in order to make an accurate COVID-19 diagnosis and prevent disease spread. In conclusion, radiologists had high specificity but moderate sensitivity in distinguishing COVID-19 from viral pneumonia on chest CT.