COVID-19 lesion discrimination and localization network based on multi-receptive field attention module on CT images.

Since discovered in Hubei, China in December 2019, Corona Virus Disease 2019 named COVID-19 has lasted more than one year, and the number of new confirmed cases and confirmed deaths is still at a high level. COVID-19 is an infectious disease caused by SARS-CoV-2. Although RT-PCR is considered the gold standard for detection of COVID-19, CT plays an important role in the diagnosis and evaluation of the therapeutic effect of COVID-19. Diagnosis and localization of COVID-19 on CT images using deep learning can provide quantitative auxiliary information for doctors. This article proposes a novel network with multi-receptive field attention module to diagnose COVID-19 on CT images. This attention module includes three parts, a pyramid convolution module (PCM), a multi-receptive field spatial attention block (SAB), and a multi-receptive field channel attention block (CAB). The PCM can improve the diagnostic ability of the network for lesions of different sizes and shapes. The role of SAB and CAB is to focus the features extracted from the network on the lesion area to improve the ability of COVID-19 discrimination and localization. We verify the effectiveness of the proposed method on two datasets. The accuracy rate of 97.12%, specificity of 96.89%, and sensitivity of 97.21% are achieved by the proposed network on DTDB dataset provided by the Beijing Ditan Hospital Capital Medical University. Compared with other state-of-the-art attention modules, the proposed method achieves better result. As for the public COVID-19 SARS-CoV-2 dataset, 95.16% for accuracy, 95.6% for F1-score and 99.01% for AUC are obtained. The proposed network can effectively assist doctors in the diagnosis of COVID-19 CT images.

Introduction

COVID-19 is a highly infectious viral pneumonia caused by 2019-nCoV (also known as SARS-CoV-2) [1], [2]. Since first discovered in Wuhan, Hubei, it has been found in 212 countries around the world. On March 11, 2020 local time, the Director-General of the World Health Organization Dr Tedros Adhanom Ghebreyesus announced that according to the assessment, the WHO believed that the current new crown pneumonia epidemic can be called a pandemic. As of March 1, 2021, COVID-19 has more than 110 million confirmed cases and more than 2.5 million confirmed deaths [3]. With the coming winter, there has been sharp and exponential rises in coronavirus infections across many countries in the world. It grows even faster with severe new variants of coronavirus and has caused huge human and economic losses. Early detection, isolation and treatment are the most effective means to prevent COVID-19 [4].

Reverse transcription polymerase chain reaction (RT-PCR) is considered to be the gold standard for SARS-CoV-2 diagnosis, which is pertain to nucleic acid test [5]. The average test time takes 2–3 h. Because it directly detects the viral nucleic acid in the collected specimens and has strong specificity and relatively high sensitivity, it has become the main detection method at present [6], [7], [8]. It can only detect whether the patient is infected with COVID-19, but cannot describe the specific infection. It is necessary to use common medical imaging for aided diagnosis. The widely used imaging tools include Computer Tomography (CT) and X-Ray. CT images are widely used in medical image processing, such as the treatment of lung-related diseases [9], [10]. CT images also can be used to classify COVID-19 and localize lesions. A large number of studies have shown that CT has an important role in the study of COVID-19 [11], [12], [13].

Deep learning has achieved the best results in image classification, image segmentation as well as medical image processing. For example, [14], [15], [16], [17] have achieved satisfactory results for medical image classification, detection and segmentation for lung cancer. It is of great significance to apply deep learning to COVID-19 lesion discrimination and localization.

In this paper, the deep learning method is used for the auxiliary diagnosis and localization of COVID-19 on CT images, and a new attention module is proposed. The attention mechanism includes modules for extracting multi-receptive field features, spatial attention blocks and channel attention blocks. Experiments show that this method has achieved significant results in the diagnosis and location of COVID-19 lesions. The contribution points of our method are as follows:

Pyramid convolution module (PCM) is proposed to extract the features of multi-receptive field. So that the network can extract the features of different sizes of infection area, which improves the overall network diagnosis and positioning effect.

The multi-receptive field spatial attention block (SAB) and the multi-receptive field channel attention block (CAB) are designed in the proposed network. The channel and spatial attention modules focus the extracted features on the lesion area to improve the accuracy of diagnosis and localization.

The proposed network can well locate the salient lesion area of COVID-19, which can provide auxiliary information for doctors to quantitative analysis for effective treatment.

The rest of this paper is organized as follows. Section 2 introduces some work related to our method. Section 3 describes our proposed method in detail. We introduce the datasets and report the experimental results in Section 4. Finally, we discuss and conclude the paper in Sections 5.

Application of deep learning in the diagnosis of COVID-19

Recently, deep learning has been widely used in many fields of computer vision and has achieved effective results, especially for medical image processing. Image classification is the basic task to use deep learning. Lecun et al. [18] first used convolutional neural networks for image classification. Many classic networks were proposed following, such as VGG [19], Inception [20], [21], [22], ResNet [23], DenseNet [24].

Attention mechanism was first widely used in the direction of natural language processing, and achieved the state-of-the-art results [25]. Computer vision has also introduced this idea, and many studies have achieved satisfactory results. Wang et al. [26] proposed Residual Attention Network (RAN) for image classification, which added an attention module in ResNet. The purpose was to obtain a larger receptive field as an attention map, which was more beneficial to classification. Hu et al. [27] proposed a general attention module SE-Net, which can learn the relationship between each channel and get the weight of different channels for multiply the original feature map. Inspired by this, Park et al. [28] proposed Convolutional Block Attention Module (CBAM). Compared with the SE-Net which only focused on the channel dimension, CBAM combined spatial attention and channel attention to guide features to focus on salient areas respectively. SE-Net and CBAM can be embedded in mainstream basic networks to improve the feature extraction capabilities of the network without significantly increasing the amount of calculation and parameters. In addition, Park et al. [29] also proposed Bottleneck Attention Module (BAM) to construct a hierarchical attention which is similar to a human perception procedure.

In recent years, deep learning has been used for medical image processing and have achieved satisfactory results. Due to the COVID-19 outbreak, many researchers have focused their research on the use of deep learning for study on COVID-19 [30], [31], [32], [33]. Wang et al. [34] proposed COVID-Net to screen suspected patients with coronavirus infection by identifying the obvious signs of the disease on chest X-rays. Compared with VGG-16 [20] and ResNet50 [24], the number of parameters and calculations of COVID-Net was greatly reduced, and the highest 93.3% for accuracy on the COVID-x dataset [34] was obtained. Ouchicha et al. [35] proposed a novel deep learning model named CVDNet to classify COVID-19 infection from normal and other pneumonia cases using chest X-ray images. The proposed model was constructed by using two parallel levels with different kernel sizes to capture local and global features of the inputs. Qian et al. [36] proposed a M3Lung-Sys to multi-class classification of pneumonia on CT images. Slice-level classification network and patient-level classification network were combined to classify CT images in multiple categories. Wang et al. [37] proposed a new joint learning framework to achieve accurate diagnosis of COVID-19 by effectively learning heterogeneous datasets with distributed differences. By redesigning the recently proposed COVID-Net [34], a strong backbone had been built in terms of network architecture and learning strategies. As well as, in order to improve the classification performance of each dataset, the contrast training target was adopted to enhance the domain invariance of semantic embedding. On SARS-CoV-2 dataset [38], the proposed method achieved 90.83% for accuracy, 90.87% for F1-Score and 96.24% for AUC. Harsh et al. [39] proposed a deep transfer learning algorithm to accelerate the detection of COVID-19 cases by using X-ray and CT-Scan images of the chest. On SARS-CoV-2 dataset [38], the proposed method achieved 94.04% for sensitivity and 95.86% for specificity.

Although the above methods have achieved good results, it does not propose a solution to the problem of COVID-19 infection area with different sizes and shapes. This paper proposes a novel attention module that involves multi-receptive field features to achieve more accurate representation for different lesions. The experimental results show that our method can get better results than other state-of-the-art methods and may provide auxiliary discrimination and localization for COVID-19 diagnosis.

Methods

Our method uses VGG16 as the backbone. The overall network is described in Fig. 1. There are five stages and there fully connection layers. Each stage includes two or three convolutional layers and a max pooling layer. Different from VGG16, we replace the convolution layer before each pooling layer with the proposed multi-receptive field attention module, which is shown in the yellow cubes in Fig. 1. The corresponding detailed architecture of the attention module is illustrated in the yellow dotted box. Following, we introduce the proposed attention in sub-sections.

Fig. 1

Architecture of proposed network. (For interpretation of the references to color in this legend, the reader is referred to the web version of this article.)

Multi-receptive field attention module

The architecture of multi-receptive field attention module is shown in the yellow dotted box in Fig. 1. Assuming that the given input feature map is , the multi-receptive field feature map sets are obtained by passing it through a PCM. Then is sent to two attention blocks separately. Firstly, are summed and then sent to the SAB to obtain the spatial attention weight (). Secondly, are concatenated and then sent to the CAB to obtain the channel attention weight . Finally, the and are respectively multiplied with the feature map obtained by concatenating and to obtain the final feature map . The process can be expressed as the following (1), (2), (3): where ⊗ is element-wise multiplication. is obtained feature map after the attention module. The following will introduce the blocks of PCM, SAB and CAB. Fig. 2 shows the detailed of the three blocks.

Fig. 2

Architecture of PCM, SAB and CAB. (a) is the structure of PCM, (b) is the structure of SAB, (c) is the structure of CAB. As illustrated, the outputs of PCM are sent to both SAB and CAB which make the attention blocks contain multi-receptive field information.

. PCM

The PCM aims to transfer multi-receptive field information to the attention blocks. The detailed structure is shown in Fig. 2(a). The input feature map is first passed through a convolutional layer to adjust the number of channels to reduce the number of parameters. The number of channels of the kernel can be set freely. Then the convolutional layers with different kernel sizes is passed through to obtain the multi-receptive field feature map sets . The number of channels of kernels is the same as the number of input channels, and the size of kernels is related to the size of the input feature maps, as shown in Table 1.

Table 1

The size and number of kernels for different input feature maps (based on VGG16).

Input feature maps	Number of kernels	Kernel sizes	Channel
64×112×112	4	3, 9, 15, 21	64
128×56×56	4	3, 7, 11, 15	128
256×28×28	4	3, 5, 7, 9	256
512×14×14	4	1, 3, 5, 7	512
512×7×7	4	1, 3, 5, 7	512

. SAB

The SAB uses the multi-receptive field feature map sets obtained from PCM to extract the spatial attention weight. The architecture is shown in Fig. 2(b). First, element-wise addition is performed on the multi-receptive field feature map sets to obtain the output . Then is passed to a convolutional layer with kernel size for refining features. An average pooling layer in the channel dimension is used to obtain feature map . After a 3 × 3 convolutional layer, the spatial attention weight can be obtained through a sigmoid on . The process can be shown in Eq. 4.where denotes the sigmoid function. represents convolution with kernel size . is average pooling down-sampling operation.

. CAB

Similar to SAB, the CAB uses the obtained from PCM to extract the channel attention weight. The structure of CAB is shown in Fig. 2(c). Multi-receptive field feature map sets is concatenated to get . Then is through an average pooling layer in spatial dimension to obtain . Next，a 1 × 1 convolutional layer is used to change the number of channels and new feature map is obtained. Finally passing a sigmoid function, the channel attention weight is obtained. The process can be shown in Eq. (5).where denotes the sigmoid function. represents convolution with kernel size . means average pooling down-sampling operation.

Loss function

The final result is described in Eq. (3). With the proposed attention module, the network can focus more on the lesion area and finally extract more representative features, which greatly improves the classification effect of the network.

The loss function used in this paper is the cross-entropy loss, which is commonly used in binary classification, as following:Where y is the ground truth label of an input CT image, is the prediction results from the network.

Experiment

We verify the validity of our proposed method on two dataset, our own COVID-19 dataset DTDB and a public datasets SARS-CoV-2 [38].

DTDB is provided by the Beijing Ditan Hospital Capital Medical University, including 40 patients infected with COVID-19 in different periods and 40 people without COVID-19 infection. We extracted 3650 slices containing COVID-19 infection and 3600 slices not containing COVID-19. The dataset was divided into 5800 images for training, 1450 images for validation by using 5-fold cross-validation. We used basic networks with different attention modules to conduct experiments.

SARS-CoV-2 [38] dataset is a publicly available COVID-19 CT dataset, containing 1252 CT scans that are positive for SARS-CoV-2 infection (COVID-19) and 1230 CT scans for patients non-infected by SARS-CoV-2, 2482 CT scans in total. The dataset was divided into 1986 images for training, 496 images for validation by using 5-fold cross-validation. We compared with different state-of-the-art methods to verify the effectiveness of our method.

The proposed network was implemented in Pytorch [40]. All the COVID-19 CT images in our experiments had been resized to . In order to reduce the influence of overfitting caused by limited datasets, we employed several data augmentation operations, including random rotation and random flipping (up-down or left-right in x-y planes). The Adam optimizer was employed for training our model in an end-to-end manner with an initial learning rate of 0.001 and betas of (0.9, 0.999). The learning rate decayed by 0.1 every 30 epochs. The batch size was set to 16 on an NVIDIA GeForce GTX 1070 GPU with 8 GB memory.

On DTDB dataset and SARS-CoV-2 dataset, training on the training set took about 3.6 h and 2.5 h respectively. As for testing a CT slice, it costed an average of 0.058 s on an NVIDIA GeForce GTX 1070 GPU with 8 GB memory.

Evaluation criteria

The metrics employed to quantitatively evaluate classification are accuracy, sensitivity, specificity and F1-score.

Accuracy represents the accuracy rate of classification results, as shown in Eq. (7):

The sensitivity and specificity measure the classifier's ability to identify positive samples and negative samples respectively:

F1 score is a measure of classification problems. It is a harmonic average of the precision and recall, with a maximum of 1 and a minimum of 0, as show in Eq. (12).

Ablation studies

In this section, we performed ablation experiments to verify the effectiveness of the two attention blocks. We used VGG16 as the backbone. The input image was resized to 224 × 224. The kernels information of the PCM was shown in Table 1. The experimental results were shown in Table 2.

Table 2

Experimental results of ablation experiments (best in bold).

Description	Accuracy (%)	Specificity (%)	Sensitivity (%)
VGG16 [20]	93.02	94.32	91.25
VGG16 + SAB	94.76	96.35	93.32
VGG16 + CAB	96.02	96.83	95.13
VGG16 + SAB + CAB	97.12	96.89	97.21

From Table 2, we can see that both SAB and CAB can improve the diagnostic effect of the network. The accuracy improved by 1.74% and 2.03% respectively. When SAB and CAB were combined, the accuracy was further improved, which was 4.10% higher than the basic network. Fig. 3 shows the ROC curve and the confusion matrix of the ablation studies.

Fig. 3

The ROC curve (left) and confusion matrix of ablation studies (right).

Experimental results

For DTDB COVID-19 dataset, we compared with three widely used image classification attention modules, including Residual Attention Network (RAN) [26], SE-net [27], CBAM [28], Table 3 summarized the experimental results.

Table 3

COVID-19 diagnostic results on DTDB COVID-19 dataset.(best in bold).

Description (based onVGG16)	Accuracy (%)	Specificity (%)	Sensitivity (%)
VGG16 [20]	93.02	94.32	91.25
RAN [26]	94.76	94.53	95.10
SE-Net [27]	94.06	95.35	92.56
CBAM [28]	94.27	96.73	91.76
Ours	97.12	96.89	97.21

It can be seen from Table 3 that our attention modules had a better effect than other commonly used attention modules. For accuracy, our method was 2.36% higher than RAN, 3.06% higher than SE-Net, and 2.85% higher than CBAM. For sensitivity and specificity, our method was (2.36%, 2.11%), (1.54%, 4.65%), (0.16%, 5.45%) higher than RAN, SE-Net, and CBAM respectively. Experimental results proved that the attention module we proposed can better recognize COVID-19 in CT images. Fig. 4 shows the ROC curve and the confusion matrix of the experimental results.

Fig. 4

The ROC curve (left) and the confusion matrix of different attention modules (right).

On SARS-CoV-2 dataset [38], we compared with baseline and two state-of-the-art methods in [37], [39]. Table 4 summarized the experimental results.

Table 4

COVID-19 diagnostic results on SARS-CoV-2 dataset. (best in bold).

Description	Accuracy (%)	F1-score (%)	AUC (%)
VGG16 [20]	86.87	87.46	92.16
Wang et al. [37]	90.83	90.87	96.24
Harsh et al. [39]	95	95	–
ours	95.16	95.6	99.01

Experimental results illustrated that the proposed network achieved better result than other methods. Comparing with backbone, the proposed network improved the accuracy of 8.29%, F1-score of 8.14% and AUC of 6.85% respectively. As for [37], we achieved 4.33%, 4.73% and 2.77% improvements in accuracy, F1-score and AUC respectively. Compared to [39], the proposed method reported better results without transfer learning. The experimental results illustrated the effectiveness of the proposed method for COVID-19 discrimination.

Infection visualization

Above we quantitatively analyzed the experimental results of our network and showed the effectiveness of our method. Then we used the Grad-Cam [41] method for visualization and localization, which can verify the reliability of our method.

We randomly selected several test CT images and located the lesion area with Grad-CAM median value greater than 0.7. Fig. 5 shows the results.

Fig. 5

The visualization results of Grad-CAM. Row 1 is the visualization results and row 2 are the corresponding original CT images.

As can be seen from the Fig. 5, our attention focused on the area where the positive samples were located. Our method can better locate the distinguishable COVID-19 lesion parts of CT images to improve the classification ability.

Lesion localization and quantitative analysis

DTDB dataset contains some COVID-19 patients with CT cases at different periods. We used Grad-CAM to locate the lesion area of the COVID-19 patients from infection to cure on CT images. In a case of COVID-19 patients, Fig. 6 visualized CT slices with the largest area of feature map response greater than 0.7 and positive diagnosis of COVID-19. It showed the progress with treatment periods for 4 patients.

Fig. 6

COVID-19 lesion localization in different periods of patients.

From the CT images of patients with COVID-19, Fig. 6 illustrated obviously that the area of the lesion increased faster in the early stage. With the advancement of treatment for some periods, the lesion area decreased significantly, such as the 2, 4, 6, 8 rows in Fig. 6. The qualitative visualization results of the proposed method had the same trend and the high response region of the localization approached the real lesion region, shown in the 1, 3, 5, 7 rows in Fig. 6.

In order to illustrate our conclusion more intuitively, we quantitatively analyzed the weighted normalized maximum area of the cases in different periods of the 4 patients in Fig. 6. is described in Eq. (13).where represents the feature map maximum response region area of the i-th patient j-th period, and represents the obtained probability output from the network for the corresponding slice with COVID-19 positive. represents normalization operation.

We used combined diagram to quantitatively analyze the weighted normalized maximum area (WNM-Area) in Fig. 7. It also reflects a basically consistent infection changing trend that the infection of coronavirus has been serious at early periods and then tend to decrease along with effective treatment. The proposed method can provide a quantitative analysis of infection area in different treatment periods and assist doctors to evaluate the progress of COVID-19 treatment effectively.

Fig. 7

Combined diagram for variation of weighted normalized maximum area in different periods. The blue dotted tend line represents the change trend of the WNM-Area of the patients over time.

Conclusions

This paper presents a convolutional neural network structure for the diagnosis of COVID-19 on CT images. The structure of the network is based on VGG16. The designed attention modules are added in each stage before pooling layer. It is composed of pyramid convolution module (PCM), multi-receptive field spatial attention block (SAB) and multi-receptive field channel attention block (CAB). PCM can improve the diagnosis results of different size for lesion regions. Better classification accuracy is achieved by using SAB and CAB which can make features extracted by the network focus on lesion areas. Experiments performed on our DTDB COVID-19 dataset achieved 97.12% for accuracy, 96.89% for specificity, and 97.21% for sensitivity. Compared with other commonly used attention modules, our methods get better results. As for public COVID-19 SARS-CoV-2 dataset, more satisfactory results are obtained compared with the state-of-the-art methods. The quantitative and visualization results indicate that the features extracted by the proposed network focus on the localization of lesions regions, which is more helpful in COVID-19 diagnosis. The same results are achieved for lesion localization. By locating the lesion area of the same patient in different periods, it can assist doctors to determine the effectiveness of the treatment. The proposed method provides quantitative diagnostic information for doctors, as well as auxiliary lesion location information. Our method has the space for further optimization and also can be transformed to other medical diseases auxiliary diagnosis.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.