COVID-19 on Chest Radiographs: A Multireader Evaluation of an Artificial Intelligence System.

Background Chest radiography may play an important role in triage for coronavirus disease 2019 (COVID-19), particularly in low-resource settings. Purpose To evaluate the performance of an artificial intelligence (AI) system for detection of COVID-19 pneumonia on chest radiographs. Materials and Methods An AI system (CAD4COVID-XRay) was trained on 24 678 chest radiographs, including 1540 used only for validation while training. The test set consisted of a set of continuously acquired chest radiographs (n = 454) obtained in patients suspected of having COVID-19 pneumonia between March 4 and April 6, 2020, at one center (223 patients with positive reverse transcription polymerase chain reaction [RT-PCR] results, 231 with negative RT-PCR results). Radiographs were independently analyzed by six readers and by the AI system. Diagnostic performance was analyzed with the receiver operating characteristic curve. Results For the test set, the mean age of patients was 67 years ± 14.4 (standard deviation) (56% male). With RT-PCR test results as the reference standard, the AI system correctly classified chest radiographs as COVID-19 pneumonia with an area under the receiver operating characteristic curve of 0.81. The system significantly outperformed each reader (P < .001 using the McNemar test) at their highest possible sensitivities. At their lowest sensitivities, only one reader significantly outperformed the AI system (P = .04). Conclusion The performance of an artificial intelligence system in the detection of coronavirus disease 2019 on chest radiographs was comparable with that of six independent readers. © RSNA, 2020.

Summary

An artificial intelligence system (CAD4COVID-XRay) can identify characteristics of coronavirus disease 2019 on chest radiographs with performance comparable to that of six readers.

■ An artificial intelligence (AI) system used to evaluate chest radiographs of coronavirus disease 2019 (COVID-19) pneumonia yielded an area under the receiver operating characteristic curve of 0.81 on chest radiographs from 454 patients with reverse transcription polymerase chain reaction (RT-PCR) results.

■ The performance of an AI system in the detection of COVID-19 pneumonia was comparable with that of six independent radiologists, with an operating point of 85% sensitivity and 61% specificity in comparison with RT-PCR assays as the reference standard for the presence or absence of severe acute respiratory syndrome coronavirus 2 viral infection.

Introduction

The diagnostic test for coronavirus disease 2019 (COVID-19) infection is a reverse transcription polymerase chain reaction (RT-PCR) test. However, there has been a severe shortage of test kits worldwide; furthermore, laboratories in most countries have struggled to process the available tests within a reasonable time frame. Although efforts to increase the capacity for RT-PCR testing have been underway, health care workers attempting to triage symptomatic patients have turned to imaging in the form of chest radiography or CT. Imaging is part of triage to assess pulmonary health and route patients to the appropriate parts of the health care system. There are several strategies and flowcharts used to diagnose and rule out COVID-19, and chest radiography and CT have been widely used as part of the initial screening process (1–4).

Although many countries have experienced difficulties in allocating scarce resources throughout the COVID-19 pandemic, countries in the developing world with economic, infrastructural, governmental, and health care problems (resource-constrained settings) are particularly at risk. In these resource-constrained settings, the COVID-19 pandemic could have consequences far more severe than we have seen in industrialized countries. The World Health Organization reported that, as of April 15, outbreaks were confirmed in 45 African countries, describing 10 759 cases with 520 deaths (5). Given the lack of access to medical care and the low availability of RT-PCR tests across the African continent, it is likely that the true numbers are much higher. The strategy in these regions must focus heavily on detection and reduction of transmission through effective isolation and quarantine processes.

Chest radiography is a fast and relatively inexpensive imaging modality that is available in many resource-constrained health care settings. Unfortunately, there is a severe shortage of radiologic expertise in these regions to allow for precise interpretation of such images (6). An artificial intelligence (AI) system may be a helpful tool as an adjunct for radiologists or, in the common case that radiologic expertise is not available, for the medical team (7,8). Previous work in the related task of tuberculosis detection on chest radiographs (9–11) has shown that software can perform at the level of an expert radiologist for tuberculosis identification. In this study, we evaluate the performance of an available (12) AI system for the detection of COVID-19 pneumonia on chest radiographs.

Materials and Methods

Data Acquisition

This study was approved by the institutional review boards of Jeroen Bosch Hospital (‘s Hertogenbosch, the Netherlands), Bernhoven Hospital (Uden, the Netherlands), and Radboud University Medical Center (Nijmegen, the Netherlands). Informed written consent was waived, and data collection and storage were performed in accordance with local guidelines.

AI System for Chest Radiograph Interpretation

CAD4COVID-XRay is a deep learning–based AI system used to detect COVID-19 characteristics on frontal chest radiographs. The software was developed by Thirona (Nijmegen, the Netherlands) and provided for this study. Some authors are employed by Thirona (R.H.H.M.P., A.M., J.M.) or a consultant to Thirona (B.v.G.), and the other authors had control of inclusion of all data and information in this study. CAD4COVID-Xray is based on CAD4TB version 6 software (9), which is a commercial deep learning system for the detection of tuberculosis on chest radiographs. As preprocessing steps, the system uses image normalization (13) and lung segmentation using U-net software (14). This is followed by patch-based analysis using a convolutional neural network and an image-level classification using an ensemble of networks.

The system was retrained, first on a pneumonia data set (15) that was acquired prior to the COVID-19 outbreak. These data are publicly available and have been fully anonymized. It is known to come from one center, but details of the x-ray system or systems are not available. This data set includes 22 184 images, of which 7851 are labeled as normal and 5012 are labeled as depicting pneumonia. The remainder had other abnormalities inconsistent with pneumonia. A validation set of 1500 images (500 per label, equally split between posteroanterior and anteroposterior images) was held out and used to measure performance during the training process. The purpose of retraining using these data was to make the system sensitive and specific to pneumonia in general because large numbers of COVID-19 images are difficult to acquire at present. To fine-tune the system to detect COVID-19 specifically, an additional training set of anonymized chest radiographs was acquired from Bernhoven Hospital that contained 416 images from RT-PCR–positive subjects and 191 images from RT-PCR–negative subjects. These were combined with 96 COVID-19 images from other institutes and public sources and 291 images from Radboud University Medical Center from the pre–COVID-19 era (used to increase numbers of negative samples). This data set of 994 images was used to retrain the system a final time, holding 40 images out for validation (all from Bernhoven Hospital, equally split between positive and negative and posteroanterior and anteroposterior). This data set consisted of all RT-PCR–confirmed data available to us (excluding the test set) with the addition of negative data to balance the class sizes. The system takes approximately 15 seconds to analyze an image on a standard personal computer.

The test set was selected from chest radiographs from the Jeroen Bosch Hospital acquired from individuals suspected of having COVID-19 who presented to the emergency department with respiratory symptoms between March 4 and April 6, 2020. All patients underwent laboratory measurements, chest radiographic imaging, and RT-PCR testing (Thermo Fischer Scientific, Bleiswijk, the Netherlands).

The imaging data included both standard radiographs (posteroanterior and lateral projection) of the chest (Digital Diagnost; Philips, Eindhoven, the Netherlands), of which only the posteroanterior images were selected, as well as the anteroposterior projections obtained with a mobile system (Mobile Diagnost; Philips). Of all 827 frontal images, one image per patient with a RT-PCR result available was selected (n = 555). In instances when multiple chest radiographs were available for a patient, the best-quality image acquired for diagnostic purposes was selected. This selection contained only one image of a minor (aged 4 years), which was included because the AI software is intended to work on patients aged 4 years or older. In total, 87 images that did not display the entire lungs or that were acquired for nondiagnostic purposes, such as checking tube positioning, were excluded. The patient characteristics of the remaining 468 images are detailed in Table 1.

Table 1:

Properties of Training, Validation, and Test Sets

Multireader Study

The test set was scored by six readers (A.J.G.K., chest radiologist with 5 years of experience; M.B.J.M.K., chest radiologist with 20 years of experience; E.T.S., chest radiologist with more than 30 years of experience; S.S., chest radiologist with 6 years of experience; C.M.S., chest radiologist with more than 20 years of experience; M.R., radiologist with 24 years of experience). Readers assigned each image to one of the following categories: 0: normal, no finding; 1: abnormal but no lung opacity consistent with pneumonia; 2: lung opacity consistent with pneumonia (unlikely COVID-19); 3: lung opacity consistent with pneumonia (consistent with COVID-19).

Readers could also mark images as unreadable. All readers assessed the images independently and were fully blinded to other reader opinions, clinical information, and RT-PCR results.

Reader consensus was used to evaluate the AI system against a radiologic reference standard and to provide an overview of the pulmonary abnormalities of the test set from a radiologic viewpoint. To create a consensus among readers, the most frequently chosen score for an image was selected. Where there was a tie of frequencies, the higher score was selected.

Statistical Analyses

Performance of the AI system was assessed by generating a receiver operating characteristic (ROC) curve from the AI system scores. Area under the ROC curve is reported. Similarly, reader performance was evaluated by thresholding at different score levels to generate ROC points.

Confidence intervals (CIs) on the ROC curve and on the reader sensitivity and specificity points were generated by bootstrapping (16).

For each reader sensitivity value, the corresponding specificity and specificity of the AI system at that sensitivity setting are computed. A significant difference is determined by means of the McNemar test. The resulting P values are reported in each case (P < .05 was considered indicative of a significant difference).

Additionally, the performance of the AI system and each reader was measured against a consensus radiologic reference standard of the remaining five readers. To create an ROC curve, the reference standard must be binary. This was achieved by setting the reference standard at 1 for images rated as consistent with COVID-19, and at 0 for images with any other consensus score.

Positive predictive values (PPVs) and negative predictive values (NPVs) were calculated for all readers and for the consensus reading using a reference standard of RT-PCR results. We defined three operating points for the AI system at sensitivities of 60%, 75%, and 85%, respectively, and computed the same metrics.

Results

Any image considered unreadable by any of the readers was excluded from analysis. Of the 468 images, 454 were successfully read by all six readers. Readers were not required to specify reasons for rejection of images; however, when comments were provided, they related to poor image quality caused by weak inspiration or incorrect patient positioning. To provide an overview of the content of the test set from a radiologic point of view, the consensus of all six readers was established for the remaining 454 images. This consensus labeled 117 cases as normal (category 0), 94 cases as containing abnormalities other than pneumonia (category 1), 26 cases as pneumonia not consistent with COVID-19 (category 2), and 217 cases as consistent with COVID-19 pneumonia (category 3). These numbers indicate the diversity of disease in the test set.

The AI system was applied successfully to all 454 cases. Figure 1 shows examples of the AI system heat maps in a patient with positive RT-PCR findings and a patient with negative RT-PCR findings.

Figure 1a:

Top: Images in a 74-year-old man with positive reverse transcription polymerase chain reaction (RT-PCR) test results for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) viral infection. (a) Frontal chest radiograph. (b) Artificial intelligence (AI) system heat map overlaid on a shows pneumonia-related features. The AI system score for this subject is 99.8. Bottom: Images in a 30-year-old man with negative RT-PCR test results for SARS-CoV-2 viral infection. (c) Frontal chest radiograph. (d) AI system heat map overlaid on c. The AI system score for this subject is 0.2.

The ROC results for all six readers and the AI system using RT-PCR results as the reference standard are depicted in Figure 2. The AI system achieved an area under the ROC curve of 0.81. In most regions of the ROC curve, the system performed better than, or at the same level as, the readers. Clusters of points from radiologic readers are seen at sensitivities of approximately 60%, 75%, and 85%. Although the ROC curve indicates specificity at all sensitivity levels, we identified three particular operating points in line with these sensitivities where reader points were clustered. At 60% sensitivity, the AI system had a specificity of 85% (95% CI: 79%, 90%); at 75% sensitivity, the specificity was 78% (95% CI: 66%, 83%), and at a setting of 85% sensitivity, the specificity decreased to 61% (95% CI: 48%, 72%).

Figure 2:

Receiver operating characteristic (ROC) curve for the artificial intelligence (AI) system and points for each reader (point locations are specified in the figure legend). Reference standard is reverse transcription polymerase chain reaction (RT-PCR) test result. The 95% confidence intervals are shown as a shaded area for the ROC curve, and crosshairs indicate each reader point. The AI system operating points discussed in the text are shown at sensitivities of 60%, 75%, and 85%. The test data set has 454 patients (223 with positive RT-PCR results and 231 with negative RT-PCR results). AUC = area under the ROC curve.

Table 2 compares the AI system and reader performance at sensitivity values fixed for the readers’ ROC points. The system outperformed all readers at their highest sensitivity for detection of COVID-19 characteristics. At intermediate sensitivity settings, the system significantly outperformed reader 3, whereas no reader performed significantly better than the system. At the lowest sensitivity setting, only reader 2 outperformed the system (P = .04), whereas the system continued to outperform reader 3 (P = .01).

Table 2:

AI System Specificities at Sensitivities Fixed to Match Reader Performance at Various Score Cutoff Values

We additionally compared each reader and the AI system against the radiologic reference standard set by consensus of the remaining five readers. These results are shown in Figure 3. The area under the ROC curve of the AI system against the radiologic reference standards was generally slightly higher than against the RT-PCR test results (with the exception of the fifth curve in Fig 3). In each plot, the system performance was close to the individual reader, with the exception of reader 2, who achieved slightly better results compared with the consensus of the other five readers.

Figure 3:

Receiver operating characteristic (ROC) curves for the artificial intelligence system and each reader individually. Reference standard in each case is the consensus reading of the remaining five readers. The 95% confidence intervals are shown as a shaded area for the ROC curve. AUC = area under ROC curve.

Results of the analysis of PPVs and NPVs are shown in Table 3. The AI operating points were selected at sensitivities of 60%, 75%, and 85%, coinciding with the observed clusters of points from the radiologic readers at these locations in the ROC curve (Fig 1). At low and intermediate sensitivity operating points, AI has a performance similar to that of the readers (using the related cutoff point for reader scores) in terms of PPV and NPV. On the other hand, at high sensitivity, AI outperformed the six readers in terms of both NPV and PPV.

Table 3:

PPVs and NPVs for Readers, AI System, and Consensus Reading

Discussion

In this study, we evaluated the performance of an artificial intelligence (AI) system to detect abnormalities related to coronavirus disease 2019 (COVID-19) at chest radiography on an independent test set and compared it with radiologist readings. The external test set used to evaluate the AI system was from a hospital system different from that used to train and validate the AI system. The examinations in the test set were representative of the chest radiographs obtained during the peak of the COVID-19 epidemic in the Netherlands and were not selected to exclude other abnormalities. On the basis of reader consensus, 120 of these images had abnormalities not consistent with COVID-19, 117 were completely normal, and the remaining 217 had abnormalities consistent with COVID-19.

The AI system performance for detection of COVID-19 was compared with the performance of six independent readers and was found to be comparable or even better at high-sensitivity operating points. In the clinical setting, the PPV and NPV of AI may be considered more useful, indicating the likelihood of COVID-19 given a positive or negative result from the system (17). Our results show that at a fixed operating point (sensitivity of 75%), the AI system has a PPV of 77% and an NPV of 76%. This result is comparable with performance using the consensus of all six readers (PPV, 72%; NPV, 78%).

The results achieved by the AI system compared with radiologist readings are noteworthy, given the fact that the appearance of COVID-19 pneumonia on chest radiographs can be highly variable, ranging from peripheral opacifications only to diffuse opacifications, which makes differentiation from other diseases challenging (4,18,19). Chest radiographs may be normal initially or may show mild disease. However, Wong et al (19) showed that of all patients with COVID-19 who required hospitalization, 69% had abnormal chest radiograph findings at admission. During hospitalization, 80% showed chest radiograph abnormalities, which were most extensive 10–12 days after symptom onset (19). Frequent findings related to COVID-19 on chest radiographs are ground-glass opacities, diffuse air space disease, bilateral lower lobe consolidations, and peripheral air space opacities and are predominantly dorsobasal in both lungs (4,19). Pleural effusions, lung cavitation, and pneumothorax may occur but are relatively rare (20).

To improve the performance of the AI system for COVID-19 detection, a larger training set of chest radiographs is needed. Improvements also may be obtained by combining chest radiograph analysis with clinical and laboratory findings.

In future work, the role of AI in the care or triage of patients during the COVID-19 pandemic should be investigated, taking all related patient information and the experience level of the health care professionals interpreting the radiographs into account.

Our study had several limitations. First, the test set comes from one institution, which might not be representative of data from other centers. Second, the number of COVID-19 (RT-PCR–positive) images in the training set of the system was relatively small (512 images) relative to the number of labeled pneumonia (non–COVID-19) images (5012 images), and the system evaluated only frontal radiographs. Also, the test set was not ideally suited to test the ability of the AI system to differentiate COVID-19 pneumonia from non–COVID-19 pneumonia because the test set had been obtained during the peak of the pandemic and the number of nonviral pneumonia cases (according to the reader consensus) was relatively small. We used the RT-PCR assay as the reference standard, but RT-PCR has limited sensitivity for COVID-19 infection (71%) (21). This suggests that there may be subjects in our test set with indications of COVID-19 on chest radiographs but with a negative RT-PCR result.

In summary, we evaluated an artificial intelligence (AI) system for detection of coronavirus disease 2019 (COVID-19) characteristics on frontal chest radiographs. The performance of the AI system was comparable to that of the six independent readers. The tool is made available pro bono on the manufacturer’s website so as to be of benefit in public health surveillance and response systems worldwide and may provide support for radiologists and clinicians in chest radiography assessment as part of a COVID-19 triage process.