AI detection of mild COVID-19 pneumonia from chest CT scans.

OBJECTIVES: An artificial intelligence model was adopted to identify mild COVID-19 pneumonia from computed tomography (CT) volumes, and its diagnostic performance was then evaluated.
METHODS: In this retrospective multicenter study, an atrous convolution-based deep learning model was established for the computer-assisted diagnosis of mild COVID-19 pneumonia. The dataset included 2087 chest CT exams collected from four hospitals between 1 January 2019 and 31 May 2020. The true positive rate, true negative rate, receiver operating characteristic curve, area under the curve (AUC) and convolutional feature map were used to evaluate the model.
RESULTS: The proposed deep learning model was trained on 1538 patients and tested on an independent testing cohort of 549 patients. The overall sensitivity was 91.5% (195/213; p < 0.001, 95% CI: 89.2-93.9%), the overall specificity was 90.5% (304/336; p < 0.001, 95% CI: 88.0-92.9%) and the general AUC value was 0.955 (p < 0.001).
CONCLUSIONS: A deep learning model can accurately detect COVID-19 and serve as an important supplement to the COVID-19 reverse transcription-polymerase chain reaction (RT-PCR) test. KEY POINTS: • The implementation of a deep learning model to identify mild COVID-19 pneumonia was confirmed to be effective and feasible. • The strategy of using a binary code instead of the region of interest label to identify mild COVID-19 pneumonia was verified. • This AI model can assist in the early screening of COVID-19 without interfering with normal clinical examinations.

Introduction

Coronavirus disease 2019 (COVID-19) can cause a fatal acute respiratory distress syndrome and has spread worldwide [1-5]. The main diagnostic methods for COVID-19 include the IgG/IgM antibody test, the reverse transcription–polymerase chain reaction (RT-PCR) test and a chest computed tomography (CT) exam [6-9]. At present, the RT-PCR test has been used widely to screen COVID-19 close contacts and presumptive patients [6]. However, the RT-PCR test is not sufficiently sensitive [7], and the test results are influenced by the experimental environment, kit quality and sampling skills in the actual examination [10]. Meanwhile, the limited supply of RT-PCR reagents has also brought about unprecedented challenges in preventing the spread of the disease.

Chest CT scans are a critical component in the diagnosis of suspected COVID-19 patients [6, 11–16]. For example, Tao et al [6] analysed 1014 cases and demonstrated that the sensitivity of a chest CT exam was 88%. Another study of 51 COVID-19 patients showed that the chest CT scan identified 50 cases successfully [7]. Compared with the RT-PCR test, a chest CT scan provides more standard and intuitive information, and it could serve as an important supplement to the RT-PCR test [13, 16]. However, since there are a large number of COVID-19 close contacts who need to be scanned, expanding the use of chest CT for rapid scanning is likely to be constrained by a shortage of radiologists. Automatic analysis of chest CT images using a computer-assisted diagnosis (CAD) model is proposed as a solution to this issue [17-19]. Recently, as one of the core artificial intelligence (AI) technologies, deep learning has achieved significant accuracy in automatic detection of many lung diseases [20-24]. Therefore, studies have begun to explore adopting a deep learning–based medical image analysis model to assist in the diagnosis of COVID-19 [25-29], such as an AI-based COVID-19 detection model [25] and the EfficientNet B4 recognition model [26], and these efforts have made certain progress.

However, the existing models still have shortcomings. First, the existing models for COVID-19 recognition require the extraction of the lung region [25] or suspicious slices [29] from the CT volume, thus greatly reducing the utility of the model. Second, for the severe or critical ill COVID-19 patients, the lesions are relatively obvious and easily recognised by the radiologists or CAD models. However, it has been reported that about 80% of the infected people were COVID-19 cases with mild pneumonia, and have only small lesions in the CT volumes [3]. The existing models have limited ability to detect and locate these small lesions [26, 29]. Third, in clinical examination, there are some mild COVID-19 pneumonia cases with CT findings for whom the initial RT-PCR results were negative. Whether a deep learning model can correctly identify these cases has not yet been reported. Due to the large proportion of the mild COVID-19 pneumonia cases in early screening, AI models which can accomplish the above tasks are needed urgently.

In this study, we developed a deep learning model to identify mild COVID-19 pneumonia. The model adopted the recent atrous convolution-based technology [30-33], which has a strong ability for detail feature extraction. By assigning simple binary labels, the proposed model can largely reduce the manual intervention needed and assist in the rapid screening of presumptive COVID-19 patients.

Materials and methods

Patients

We used chest CT scans from four hospitals, and Table 1 shows the basic information of the hospitals and patients. The study was approved by the ethics committees of all the participating hospitals, and the data were analysed anonymously. The samples included 2946 3D volumetric chest CT exams from 2575 patients with a diagnosis of COVID-19 pneumonia, community-acquired pneumonia (CAP) and non-pneumonia (NP). All the data were collected between 1 January 2019 and 31 May 2020. The inclusion criteria were as follows: (1) CT volumes with a layer thickness ≤ 5 mm; (2) patients with clinical signs of mild pneumonia but no signs of severe pneumonia: SpO2 ≥ 90% on room air, have clinical symptoms (fever, cough, dyspnoea, fast breathing) and CT findings (small patchy shadow, interstitial changes, ground glass opacity and infiltration), or close contacts only have CT findings, and this inclusion criteria is only applicable to the COVID-19 and CAP cases [5]; (3) NP and mild CAP patients treated before 1 December 2019 or NP, mild CAP and mild COVID-19 pneumonia patients after 1 December 2019 who had received an RT-PCR test; (4) patient’s initial CT scan (repeated examinations were removed). After applying the inclusion criteria, 2087 chest CT scans remained, with each CT scan corresponding to a unique patient. In total, 568 (27%) were mild COVID-19 pneumonia cases, 763 (37%) were NP patients and 756 (36%) were mild CAP patients, and the average age of the patients was 47 ± 18 years. Among the 213 COVID-19 cases from the Tongji Hospital, 81 (38%) presented the features of mild pneumonia on CT examination, but the initial RT-PCR test was negative (75 cases were positive in the second test and 6 cases were confirmed positive by the third test). Of the 355 COVID-19 cases from the Huangpi People’s Hospital of Jianghan University, 137 (39%) showed the features of mild pneumonia on CT examination, but the initial RT-PCR test was negative (116 cases were positive in the second test and 21 cases were confirmed positive by the third test).

Table 1

Descriptive statistics for the multi-center chest CT exam data

	Center 1	Center 2	Center 3	Center 4
Total population	591	495	452	549
Female (percentage)	325 (55%)	246 (50%)	237 (52%)	269 (49%)
Age, mean, years (SD)	50 (17)	49 (19)	45 (16)	46 (19)
COVID-19 patients	355	-	-	213
Mild CAP	127	290	165	174
NP	109	205	287	162

Center 1: The Huangpi People’s Hospital of Jianghan University, Wuhan; Center 2: Zhejiang Provincial People’s Hospital; Center 3: Physical Examination Center of Cancer Hospital of the University of Chinese Academy of Sciences (Zhejiang Cancer Hospital); Center 4: Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology; SD standard deviation; CAP community-acquired pneumonia; NP non-pneumonia

Image acquisition

The CT scans of all the enrolled patients were performed on CT scanners manufactured by the General Electric Company (GE) and Siemens. The patients were scanned in a head-first supine position with their arms raised and placed beside their ears. All of the patients underwent CT scanning without the administration of contrast material. The parameters of the CT scans were as follows: tube energy 120 kV; reference tube current 110–260 mAs; pitch 0.81–1.22 mm; each volume contained 93–564 slices with a varying slice thickness from 1 to 5 mm; median CTDIvol 2.9 mGy (IQR 1.8–4.2). To avoid center effect, all samples were reconstructed with the soft reconstruction kernel and 5 mm slice thickness, and the matrix was 512×512.

Image labelling and pre-processing

Instead of pre-extraction of lung shapes or suspicious slices to accomplish the identification task, our dataset was labelled in a binary fashion according to the following rules: samples that were both RT-PCR positive and CT imaging–conformed COVID-19 were labelled as 1; samples that were negative in RT-PCR testing or the cases that occurred before 30 November 2019 were labelled as 0. All identifying patient information was removed to protect privacy.

Algorithms

In previous work, by introducing sparse constrained atrous convolution to the traditional ResNet [31, 34], we built a cross-layer sparse atrous convolution network (CSAC-Net) which performed well in official tests of pattern analysis, statistical modelling, computational learning and visual object classification (PASCAL VOC) dataset1. In the current study, we extended the above model to a three-dimensional version, thereby establishing a 3D CSAC-Net for the CT imaging–based diagnosis of COVID-19 cases with mild pneumonia. The overall structure of the proposed model is shown in Fig. 1, and it included 39 layers of 3D convolution, 10 layers of 3D atrous convolution, 3 layers of 3D max pooling in baseline, 5 layers of 3D max pooling in cross-layer subnets and 3 layers of full connection. As is shown in Fig. 1, a cross-layer 3D atrous convolution structure was designed between each two residual modules. The feature maps from the cross-layer subnets were merged into the output of baseline as the basis of full connection layer. Details of the parameter settings and equations for the network are shown in Supplementary Materials 2. Although we adopted three cross-layers in this paper, for larger-scale data, repeatedly increasing the number of this structure to seek better recognition performance could be considered. We also compared the performance of our 3D CSAC-Net with baseline 3D ResNet [34], support vector machine (SVM) [35] and random forest (RF) [36] methods with common parameter settings (see Supplementary Materials 3).

Fig. 1

Graphical summary of the utilized deep learning method: (a) the training set and binary labels, (b) the general framework of the 3D CSAC-Net, (c) the testing set with new patients and the model output

Model training and testing

All the 1538 (74%) patients from the first three centers were used to train the model, while the 549 (26%) patients from the fourth center were treated as the independent testing cohort to verify the performance. Image enhancement methods were used during training, including 0.9 to 1.1 times random scaling, horizontal reversal, vertical reversal and −15° to 15° random rotation. The size of the training batch was 10, the gradient descent strategy was based on an Adam optimiser and the initial learning rate was 10−3. The final model was trained for 75 epochs under the above super-parameters.

Performance evaluation

The trained deep learning model was evaluated with the 549-patient independent testing set to assess its sensitivity and specificity in identification. We calculated the true positive rate and the true negative rate of the model with different thresholds and used the receiver operating characteristic (ROC) curve with a 95% confidence interval (CI) and the corresponding area under the curve (AUC) to evaluate the model’s performance. Several typical COVID-19, CAP and NP samples were selected to generate heatmaps.

Statistical analysis and software

The software used to develop the DCNN model was based on the Ubuntu 16.04 operating system and included TensorFlow 1.9, Keras 2.1.4 and the open-source programme Python 3.6.5 (The Python Software Foundation). The training was conducted on an Intel Core I7-7740X CPU 4.30 GHz with an NVIDIA GeForce TITAN Xp GPU. All statistical analyses were performed using the Python packages statsmodels, pymc, pylab, sklearn and seaborn.

Results

The detailed distributions of the training set and independent testing set are shown in Table 2. To better evaluate the model, we divided the independent testing set into three subsets. Subset S1 was the mild COVID-19 pneumonia cases, S2 was the NP cases and S3 was the mild CAP cases. The first panel in Fig. 2 shows the ROC curves for the three conditions. The black curve is the overall ROC curve for distinguishing mild COVID-19 pneumonia cases from the rest of the patients (S1 vs S2 + S3). The sensitivity was 91.5% (195/213; p < 0.001, 95% CI: 89.2–93.9%), the overall specificity was 90.5% (304/336; p < 0.001, 95% CI: 88.0–92.9%) and the AUC value was 0.955 (p < 0.001). The yellow curve shows the ROC curve for distinguishing between the mild COVID-19 pneumonia cases and NP patients (S1 vs S2). The sensitivity was 93.0% (198/213; p < 0.001, 95% CI: 90.4–95.6%), the specificity was 93.2% (151/162; p < 0.001, 95% CI: 90.7–95.8%) and the corresponding AUC value was 0.971 (p < 0.001). The green curve shows the ROC for distinguishing the mild COVID-19 pneumonia cases from the mild CAP patients (S1 vs S3). The sensitivity was 90.6% (193/213; p < 0.001, 95% CI: 87.9–93.3%), the specificity was 87.4% (152/174; p < 0.001, 95% CI: 84.0–90.7%) and the AUC value was 0.940 (p <0.001).

Table 2

Data distributions of the training and testing sets

Data	Class	RT-PCR positive (initial)	RT-PCR positive (multiple)	Bacterial culture positive	Bacterial culture negative	Non-infectious lung disease	Healthy lung
Training set (total 1538)	COVID-19	218	137
	Mild CAP			Bacterial pneumonia 325 Bacterial & mycoplasma pneumonia 79	Viral pneumonia 87 Mycoplasma pneumonia 91
	Other					Nodule 200 Other 24	377
Testing set (total 549)	COVID-19	132	81
	Mild CAP			Bacterial pneumonia 90 Bacterial & mycoplasma pneumonia 23	Viral pneumonia 31 Mycoplasma pneumonia 30
	Other					Nodule 59 Other 6	97

RT-PCR positive (initial) the initial result of RT-PCR test was positive, RT-PCR positive (multiple) the initial result of RT-PCR was negative while the positive result was obtained after multiple tests, CAP community-acquired pneumonia

Fig. 2

Receiver operating characteristic (ROC) curves: the first panel is the ROC curve of our model for distinguishing mild COVID-19 pneumonia from both mild CAP and NP cases, where S1 is the 213 mild COVID-19 pneumonia scans, S2 is the 162 NP scans and S3 is the 174 mild CAP scans; the second panel shows a comparison of using the 3D ResNet, RF, SVM, and our method to identify mild COVID-19 pneumonia cases with both positive and negative results in initial RT-PCR test, where IRP means the COVID-19 cases with initial RT-PCR positive results and IRN represents the COVID-19 cases with initial RT-PCR negative results.

The second panel in Fig. 2 shows a comparison between different methods to identify COVID-19 cases in patients with either positive or negative results in the initial RT-PCR test. The sensitivity and specificity of our method for the initial RT-PCR positive COVID-19 cases were 92.4% (122/132; p < 0.001, 95% CI: 90.0–94.8%) and 90.8% (305/336; p < 0.001, 95% CI: 88.2–93.4%), respectively, and the AUC value was 0.957. For the initial RT-PCR negative COVID-19 cases, the sensitivity and specificity were 90.1% (73/81; p < 0.001, 95% CI: 87.2–93.0%) and 90.5% (304/336; p < 0.001, 95% CI: 87.7–93.3%), respectively, and the AUC value was 0.951. And the p value between the identification of cases with positive and negative initial RT-PCR test equals to 3.6 × 10−4 which is less than 0.001, indicating no statistical difference. The ROC curve of the 3D-ResNet, RF and SVM methods is also shown in Fig. 2. The results of the SVM method were slightly better than the RF method, and their respective AUCs were between 0.670 and 0.764, while the performance of deep learning methods was superior to that of traditional methods, with AUC values between 0.910 and 0.957.

Table 3 compares the identification results of the AI model under different thresholds with two junior radiologists (less than 5 years of experience) and two senior radiologists (more than 10 years of experience). As was shown, the recognition performance of the AI model was better than that of manual interpretation, and the model achieved better performance when setting the threshold value to 0.5. Moreover, compared with the average 196 seconds for manual identification, the average processing time of our model for each chest CT volume in the test set was 0.66 s, indicating a high detection efficiency. We also calculated the sensitivity of detection of those COVID-19 cases which obtained a negative result in the initial RT-PCR test.

Table 3

Identification results of the radiologists and AI model with fixed threshold

Thresholds or radiologists	TPR (%)	TPR of multiple RT-PCR (%)	TNR (%)	TP (total 213)	TP of multiple RT-PCR (total 81)	TN (total 336)
Identification results of radiologists
JR1	66.2%	64.2%	75.6%	141	52	254
JR2	63.8%	65.4%	73.2%	136	53	246
SR1	77.5%	76.5%	81.3%	165	62	273
SR2	74.2%	71.6%	80.1%	158	58	269
Identification results of the AI model
0.10	100.0%	100.0%	8.3%	213	81	28
0.20	98.6%	98.8%	30.1%	210	80	101
0.30	98.1%	97.5%	54.2%	209	79	182
0.40	97.2%	97.5%	74.4%	207	79	250
0.50	92.0%	90.1%	90.2%	196	73	303
0.60	72.8%	70.4%	95.8%	155	57	322
0.70	50.7%	51.9%	98.2%	108	42	330
0.80	32.4%	34.6%	99.4%	69	28	334
0.90	9.4%	11.1%	100.0%	20	9	336

JR junior radiologist, SR senior radiologist, TPR true positive rate, TNR ture negative rate, TP true positive cases, TN true negative cases, TPR of multiple RT-PCR TPR of the cases which were negative in initial RT-PCR test and conformed positive by multiple RT-PCR tests, TP of multiple RT-PCR the number of true positive cases which were negative in initial RT-PCR test and conformed positive by multiple RT-PCR tests

Figure 3 shows slices of the original chest CT images and the corresponding heatmaps, and these heatmaps were extracted from the last 3D max pooling layer. Figure 3a to c are the original chest CT slices of the COVID-19 patients. The model recognised mild COVID-19 pneumonia accurately, and many high-weight features were extracted from the lesion areas (Fig. 3d to f). For non-COVID-19 patients, the weights of the feature maps (Fig. 3j to l) were significantly lower, even when some lesions were present. Figure 4 shows several typical misdiagnosed samples. Figure 4a is a misdiagnosed COVID-19 case, and Fig. 4b and c are typical mild CAP patients that were misidentified as COVID-19. As can be seen, these samples were challenging, and by analysing the corresponding heatmaps, we believe that the misdiagnoses of COVID-19 were mainly due to the insignificant lesions, while the false positive sample was caused by the great similarity in convolutional features.

Fig. 3

Comparison of the heatmaps for different types of cases: a to c are the CT slices of three mild pneumonia COVID-19 cases, where a and b obtained negative results in initial RT-PCR (sample a was confirmed positive by the second test, b was confirmed positive by the third test), d to f are the corresponding heatmaps for a to c, g is the slice of a non-pneumonia case, h and i are CT slices for two mild CAP cases, j to l are the corresponding heatmaps for g to i

Fig. 4

Feature heatmaps of some misdiagnosed cases: a is the CT slice for a misdiagnosed case of COVID-19; b and c are two misdiagnosed CAP cases, d to f are the corresponding heatmaps for a to c

Discussion

An effective AI model, 3D CSAC-Net, was developed to identify mild COVID-19 pneumonia and mark the corresponding lesion areas. To the best of our knowledge, this is the first multi-center study that has adopted an AI model which does not require pre-extraction of ROI or suspicious slices to identify mild COVID-19 pneumonia. And it is also the first report on the use of AI to identify mild COVID-19 pneumonia cases with negative results in initial RT-PCR. The experimental results using the independent testing set showed a sensitivity, specificity and AUC value of 91.5%, 90.5% and 0.955, respectively. Since the RT-PCR test is not sufficiently sensitive, using AI to process CT images to screen close contacts or presumptive patients would be an important supplement.

At present, AI models that can accurately detect COVID-19 are urgently needed, and the following problems need to be taken into account: (1) compared with the critical ill COVID-19 patients whose lesions are obvious in chest CT scans and relatively easy to diagnose through IgG/IgM testing [37, 38], most of the infected people have only small lesions in the CT volumes [3, 39], so it is necessary to develop AI models which can accurately identify these small lesions in the early screen; (2) complex pre-processing and manual interventions need to be reduced as much as possible to increase the usability of the model; and (3) in addition to recognising the COVID-19 cases, the feature heatmaps of the AI model should locate the lesions. However, the previous models have limited ability to solve these problems, and the recognition bias is more obvious when the lesions are small (see Fig. 3 in [25], Fig. 5 in [26] and Fig. 4 in [29]). Recently, many efforts have been devoted to these problems [40-43]. And in this study, to address the first and second issues listed above, we designed a 3D deep learning model that uses the simple binary labels to identify mild COVID-19 pneumonia and greatly reduces the manual intervention. The labels are used in the training stage only, and the normalised CT images can be input into the model directly in practical application. To address the third issue, our proposed model uses the recent atrous convolution method, which has superior detailed feature extraction abilities (see Fig. 3d–f), and that is of great help to radiologists.

There are still some limitations to this study. First, although all the cases of COVID-19 included in the study were confirmed via RT-PCR testing, the CAP and NP patient samples from before 30 November 2019 were not tested by RT-PCR. Considering that the virus outbreak was first detected in December 2019, we treated the CAP and NP patients prior to 30 November 2019 as non-COVID-19 patients by default. Second, due to the self-learning and black box properties of deep learning itself, the specific connotations of high-weight regions of convolutional feature maps still need to be explored. Third, to reduce the complexity of integrating the model into computer-assisted diagnostic systems or remote diagnostic systems, our study did not identify the category of CAP. Since many AI models have been proposed to identify different types of CAP [44, 45], we plan to combine our deep learning network with CAP recognition methods and optimise our model further in the future. Fourth, in this study, we reconstructed all data to uniform 5 mm slice thickness to reduce center effect. However, the center effect could not be completely avoided, especially when compared with other models, because these models might be trained and tested on completely different data sets.

In summary, a deep learning model based on atrous convolution was designed to identify mild COVID-19 pneumonia and mark the corresponding lesion areas. A total of 2087 chest CT scans from four hospitals were studied. The experimental results showed that both the sensitivity and specificity were greater than 90%, and the general AUC value was 0.955, indicating clinical applicability. The comparative study also have shown that the performance of our model was superior to the identification of traditional methods and radiologists, especially for those mild COVID-19 pneumonia cases which obtained negative results in the initial RT-PCR test. The proposed model is expected to be an efficient and economic method for assisting in the early screening of COVID-19 without interfering in normal clinical examinations.

Network modules and parameter settings for the proposed 3D CSAC-Net model. (PDF 186 kb)

The list of radiomics features extracted from the CT images. (DOCX 27 kb)