MFDNN: multi-channel feature deep neural network algorithm to identify COVID19 chest X-ray images.

The use of chest X-ray images (CXI) to detect Severe Acute Respiratory Syndrome Coronavirus 2 (SARS CoV-2) caused by Coronavirus Disease 2019 (COVID19) is life-saving important for both patients and doctors. This research proposes a multi-channel feature deep neural network (MFDNN) algorithm to screen people infected with COVID19. The algorithm integrates data over-sampling technology and MFDNN model to carry out the training. The oversampling technique reduces the deviation of the prior probability of the MFDNN algorithm on unbalanced data. Multi-channel feature fusion technology improves the efficiency of feature extraction and the accuracy of model diagnosis. In the experiment, Compared with traditional deep learning models (VGG19, GoogLeNet, Resnet50, Desnet201), the MFDNN model obtains an average test accuracy of 93.19% in all data. Furthermore, in each type of screening, the precision, recall, and F1 Score of the MFDNN model are also better than traditional deep learning networks. Furthermore, through ablation experiments, we proved that a multi-channel convolutional neural network (CNN) is superior to single-channel CNN, additional layer and PSN module, and indirectly proved the sufficiency and necessity of each step of the MFDNN classification method. Finally, our experimental code will be placed at https://github.com/panliangrui/covid19.

Introduction

SARS-CoV-2 causes COVID19. Since the first report, it has become a global pandemic, with 180 million confirmed cases and 3.91 million deaths. It is extremely contagious characteristics and delayed vaccination has made developing countries vulnerable to virus attacks. Now the nucleic acid detection mechanism has played an essential role in screening the flow of people. Reverse Transcription Polymerase Chain Reaction (RT-PCR) is the most standard diagnostic technology available [1]. However, its sensitivity is relatively low, and the result is highly dependent on the sample area obtained and heavily reliant on the operator's technique [2]. The important thing is that this method takes time. However, time is a key factor in isolating, preventing and treating people infected with COVID19, which will limit the efficiency of COVID19 screening. With the global spread of COVID19, medical research has found that CXI can identify people infected with COVID19. Therefore, as a supplement to RT-PCR technology, it plays an essential role in detecting and evaluating people infected with COVID19.

Computed tomography (CT), lung ultrasound (LUS), and Chest-X ray radiography are among the most commonly used imaging modalities to identify COVID19 infections [3-5]. It is widely used in large hospitals because of the safety, painlessness, non-invasiveness, clear image, high-density resolution, and apparent morbidity of CXI. In addition, experienced doctors can make real-time diagnoses through CXI. Therefore, CXI is one of the most commonly used and readily available methods to detect COVID19 infections [6]. However, there are many similarities in the CXI characteristics of patients with COVID19 and common pneumonia, which poses a massive challenge to radiologists in diagnosing patients with COVID19.

In recent years, artificial intelligence has prompted tremendous progress in the field of biomedicine, such as medical diagnosis, intelligent image recognition, intelligent health management, intelligent drug development, and medical robots [7-9]. Machine learning-based methods have developed many applications in the accurate analysis of CXI, such as diagnosing and evaluating people infected with COVID19 [10, 11]. Standard machine learning algorithms include linear regression, random forest (RF), K-nearest neighbor (KNN), decision tree (DT), etc. [12, 13]. Abolfazl et al. used dimensionality reduction methods to extract the best features of CXI to build an efficient machine learning classifier, and the classifier distinguishes COVID19 and non-COVID19 cases with high accuracy and sensitivity [6]. Dan et al. used three different machine learning models to predict the deterioration of the patient's condition, compare them with the currently recommended predictors and APACHEII risk prediction scores, and obtain high sensitivity, specificity, and accuracy [14]. Mohamed et al. used the new fractional multi-channel exponential moments (FrMEMs) to extract features from CXI [15], p. 19].

Then, the improved Manta-Ray Foraging Optimization (MRFO) method was used for feature selection, and the KNN method was used to classify the two types of CXI [15]. However, deep learning is the hottest research direction in the field of machine learning. The CXI deep learning method for COVID19 classification has been actively explored. Linda et al. proposed a deep convolutional neural network called COVID-Net to help clinicians improve screening [16]. Ali et al. proposed five models based on pre-trained convolutional neural networks (ResNet50, ResNet101, ResNet152, InceptionV3, and Inception-ResNetV2) to implement four different binary classifications: COVID19, normal (healthy), viral pneumonia and bacterial pneumonia. CXI has achieved a high accuracy rate [17]. Loannis et al. automatically detected CXI based on the transfer learning method of convolutional neural network and achieved 96.78%, 98.66%, and 96.46% accuracy, sensitivity, and specificity [18]. Ezz et al. proposed the COVIDX-Net network based on seven deep convolutional network models with different architectures and obtained F1 scores of 0.89 and 0.91, respectively [19].

Inspired by machine learning and deep learning and the accumulation of previous work experience, we will further explore the impact of experimental data and deep convolutional neural networks on the detection algorithm or detection system in this article. However, in most databases, unbalanced label classes often occur, which will cause the convolutional neural network to be biased to correctly identify image data with many class labels. Multi-channel CNN is usually superior to single-channel CNN in terms of computational efficiency and accuracy [20-22]. A similar method has been proposed due to previous work. However, the previous work mainly focused on optimizing the processing features on multiple channels, and did not discuss fusion, nor did it optimize the features after fusion. Therefore, the experiment selected a multi-channel input single-channel output parallel deep neural network as the front part of the algorithm, additional layer as the main method of feature fusion, and added the features extracted from the multi-channel. As the tail of MFDNN, PSN can perform secondary convolution operations on the basis of feature fusion to improve the efficiency and accuracy of feature extraction. Two similar neural networks map the input to a new space and represent the output in the new space. By calculating the value of loss, the similarity between image features is calculated [23].

Therefore, the main focus of this article is to solve the accuracy of the COVID19 detection algorithm. Around this problem, we will solve the following problems separately: (1) Deal with the imbalance of sample labels. (2) Optimize the feature extraction of the deep neural network algorithm. (3) Evaluate the classification effect of the network algorithm. To achieve this goal, we first analyze the degree of imbalance in the sample data. We found that chest X-ray data of people who were not infected with COVID19 was significantly more than other categories through the data set analysis. To be able to classify the chest radiograph data set more accurately, our main contributions are as follows.

To balance the impact of the unbalanced label data set on model training, when processing CXI, we embed the oversampling method into the model to balance all categories of data.

We propose an MFDNN algorithm based on multi-channel input, single-channel output, and centralized weight sharing. The algorithm model concentrates the feature maps of multi-channel chest radiographs and optimizes the feature extraction process. As the tail of MFDNN, PSN can extract features from the additional layer for the second time.

Finally, the MFDNN is compared with the classic deep neural networks (VGG19, GoogLeNet, Resnet50, Desnet201). The MFDNN model is better than other models in precision, recall, and F1 Score and confusion matrix.

Materials and methods

In this section, we first introduce the flow chart of the MFDNN algorithm. It mainly includes two parts: oversampling and the MFDNN model. The first part is primarily data preprocessing, and the second part primarily uses the MFDNN model for feature extraction and patient diagnosis. Figure 1 shows our proposed algorithm classification process.

Fig. 1

MFDNN algorithm flow chart. It includes two parts, namely data oversampling, feature extraction and classification

Materials

X-rays pass through the chest, and different body parts absorb the rays, and the film will not be exposed or partially exposed. After the film is processed, this part is white, forming an imaging manifestation. As shown in Fig. 2, COVID images have the symptom of "white lung"; a large lung area is white. The lung opacity class is very similar to the normal class. The main difference is the increase in lung texture. Small area texture image blur appears in viral pneumonia images, which may be due to lung inflammation caused by other viruses. Compared with COVID, the contour of both lungs is visible, the transparency is acceptable, and the texture is slightly obvious. The proposed MFDNN model is trained and tested on a public dataset (COVID19 chest X-ray dataset). The data set consists of 3616 COVID19 positive cases, 10,192 normal, 6012 lung opaque (non-COVID lung infection) and 1345 viral pneumonia images [24, 25]. The dataset can be downloaded from the website [25]. Before the experiment, we need to have a preliminary understanding of the CXI data set.

Fig. 2

Chest X-ray medical images of four different label samples (COVID, Lung_Opacity, Normal, Viral Pneumonia)

Oversampling

The classification of imbalanced data sets is still a problematic point for deep neural network classification. Different methods have been used in the literature to deal with unbalanced data [26-28]. The commonly used method is the resampling technique. In addition, the resampling method includes two methods, namely under-sampling and over-sampling. In oversampling, the minority class samples are copied to balance the size of each class in the training data. In undersampling, some majority class samples are removed during the training process to balance the size of each class. Therefore, when the model is trained on balanced data, it should exhibit unbiased behavior. Different types of undersampling methods have been proposed [26-28]. However, it is reported that random oversampling is the simplest method and exhibits similar performance to other complex methods. Therefore, in this article, we use random oversampling to balance the training process and reduce the bias in building the model.

As shown in Fig. 3a, by comparing the characteristics of the database, the chest radiographs of people infected with COVID19 are significantly less than those of normal people. However, the steps of neural networks and human brain extraction are similar. Therefore, when the probability of memorizing normal chest radiographs of the MFDNN model is greater than remembering the chest radiographs of COVID19 infected persons, the model is more likely to recognize the chest radiographs of normal people, which may lead to COVID19 infection. Therefore, the probability of screening by the user is reduced, and the model cannot be applied to the actual detection process. Therefore, the experiment chooses to oversample to generate new samples for a few categories to ensure that the model has the same probability of remembering different CXI. Figure 3b shows the result of oversampling. The sample data size of the minority class is the same as the sample data size of the majority class.

Fig. 3

a The sample size of the original dataset. b The number of samples after oversampling

Multi-channel feature deep neural network

The multi-channel feature deep neural network (MFDNN) algorithm is experimentally designed. The oversampling dataset passes through three identical feature extraction modules. First, the features will be merged in the middle of the frame. Then further feature extraction is performed on the collected image features through the Siamese network, which improves image feature extraction work efficiency. Finally, we will introduce the function of each layer in detail from the feature extraction module.

The image can be seen as a high-dimensional matrix composed of feature vectors. In feature extraction, using a small convolution kernel can reduce the convolution operation's error rate, so the 3 × 3 convolution kernel is selected in the convolution layer, and the step size is 1 for matrix operation. The convolutional layer is defined as:

The feature value at the position (, ) of the kth feature map of the lth layer is . Where is the weight of the lth layer, is the bias of the lth layer, and is the (, ) unknown input block of the lth layer. Thus, 64 convolution kernels simultaneously perform local perception and share parameters on the input image.

Batch Normalization (BN) is widely used training technique in deep networks. A BN layer whitens activations within a mini-batch of N examples for each channel dimension and transforms the whitened activations using affine parameters and , denoting by activations in each channel, BN is expressed as [29]:where

The mean and variance of activations within a mini-batch,

Select the Rectified Linear Unit (ReLU) function as the convolutional layer's activation function to reduce the probability of model overfitting [30]. The ReLU function will make part of the neuron output 0 to enhance the sparsity of the network. Besides, it reduces the interdependence between parameters and alleviates the problem of overfitting. For the MFDNN model, the ReLU function enables each neuron to exert the greatest screening effect, saving a lot of calculations in the whole process, which is defined as:

The maximum pooling method is selected to obtain the maximum value of the feature tiles from the convolutional layer as the output in the pooling layer. The down-sampling convolution kernel size is set to 2 × 2, the stride size is set to 2, and the feature matrix after the pooling operation is filled in a "same" manner to alleviate the excessive sensitivity of the convolution layer to the position. The maximum pooling layer reduces the parameters by reducing the dimension, removing redundant features, simplifying the network's complexity, and other methods to achieve nonlinear feature extraction. The input of the lth layer is mapped to the output through the neuron, which is defined as:

The additional layer is the main part of multi-channel feature fusion. The additional layer serves as an intermediate hub for merging the output from the pooling layer, combining feature weights. Assuming the output is, its effect can be expressed as:

As the input of the global average pooling (GAP) layer, the learned "distribution feature representation" is selectively mapped to the labelled sample space. The activation function of each neuron in the GAP layer generally uses the ReLU function [31]. It can replace the fully connected layer in the traditional structure, thereby reducing the amount of storage required for the large weight matrix of the fully connected layer. Moreover, it also has the features and capabilities of easy fine-tuning of a pre-trained model with a conventional structure. Since the working principle of the fully connected layer involves calculating the inner product of the input vector and the weight of each row, the row size of the weight matrix needs to be the same as the number of input elements [32]. Therefore, as the input changes, we also need to adjust the weight matrix , , to a corresponding size bywhere is the size of the input feature map, , is the index of the output neurons and input feature maps, and is the modified weight matrix [32]. Given the computational complexity in the GAP layer, the dropout layer chooses a 40% random probability to discard some feature weights to reduce the model complexity and prevent overfitting. Finally, it is classified by Softmax, and the output of multiple neurons is mapped to the interval of (0, 1), which is defined as:

We use to represent training samples and to represent the label of samples, and train the neural network by using gradient descent. In this article, the cross-entropy function is used to calculate the Loss of the MFDNN model. For a single example, the cross-entropy loss function can be expressed as:where represents the sth neuron in the output layer corresponding to the sth type, is the indicator function. The weight parameter is continuously updated through the backpropagation loss function. In order to better integrate upsampling and MFDNN, we propose an MFDNN classification algorithm for detecting COVID19 patients.

Results and discussion

Experimental settings

Before oversampling, the data set is divided into training and test data at a ratio of 0.8:0.2. Then, the over-sampled training data is divided into training set and validation set according to the ratio of 0.8:0.2. Before training the MFDNN model, we choose to flip the data and augment the data with translation to expand the training data to avoid over-fitting the model. The experiment is set to 30 epochs, the batch size is set to 32, and the Adam algorithm is used as the optimizer of the model. The initial learning rate is 0.003, and after each epoch, the learning rate will drop by half. Before each epoch training, the training data and verification data will be randomly shuffled. Each model is trained on a single RTX3060.

Results

In this section, we will explain the evaluation indicators used to quantify model classification. To this end, we use an indicator based on a confusion matrix. These indicators include test accuracy, precision, recall, and F1 Score. To evaluate the model, we need to perform a detailed analysis of each category. Therefore, we need to count true positive (TP), false positive (FP), true negative (TN), and false positive (FN) [33].

Test accuracy: the proportion of samples correctly predicted to the total samples

Precision: the ratio of true positive predictions to total positive predictions

Recall: Ratio of true positive to the total observation made by the proposed model

F1 Score: It is the harmonic mean of precision and recall

Confusion matrix: It is the measurement of the performance of the model. It compares the actual and predicted values in the form of True Positive, False Negative, True Negative and False Positive

True Positive (TP): True positive are the forecasts which were at first positive and, additionally, anticipated by the AI model as positive.

False Positive (FP): False positives are the forecasts which were initially negative and anticipated by the AI model as positive.

True Negative (TN): True negatives are the forecasts which were initially negative and anticipated by the AI model as unfavourable.

False Negative (FN): False-negative are the forecasts which were initially positives and anticipated by the model as negative

The experiment first trained five models under the algorithm of MFDNN, namely Densenet201, ResNet50, VGG19, GoogLeNet, and MFDNN. Among them, the accuracy of the MFDNN model is 93.19%. Among them, the COVID category received a Recall of 0.9447 and an F1 score of 0.9358; the Lung_Opacity category received a precision of 0.9144 and an F1 score of 0.9106; the Normal class received a recall of 0.9431 and an F1 score of 0.9389; the Viral Pneumonia category received an F1 score of 0.9504. Table 1 details the test reports of each type of chest radiograph under different models. From the classic deep learning model analysis, for the COVID19 data set, the deeper the network layer, the worse the effect of the model. For example, the test results of the Densenet201 model only get good prediction results in a few categories. GoogLeNet obtains the best results in the classic deep learning network, but compared to the MFDNN model, the traditional deep learning model does not achieve the best test results.

Table 1

Densenet201, ResNet50, VGG19, GoogLeNet, MFDNN classification technical report

		Densenet201	ResNet50	VGG19	GoogLeNet	MFDNN
COVID	Precision	0.9272	0.9473	0.9532	0.9572	0.9369
	Recall	0.8105	0.87	0.7331	0.8976	0.9447
	F1 score	0.8649	0.907	0.8288	0.9264	0.9358
Lung_Opacity	Precision	0.7198	0.801	0.7968	0.854	0.9144
	Recall	0.9359	0.9143	0.8611	0.9002	0.9068
	F1 score	0.8137	0.8539	0.8277	0.8765	0.9106
Normal	Precision	0.941	0.9302	0.8627	0.9266	0.9348
	Recall	0.8057	0.89	0.9001	0.923	0.9431
	F1 score	0.8681	0.9097	0.881	0.9248	0.9389
Viral pneumonia	Precision	0.9007	0.9675	0.956	0.9611	0.9257
	Recall	0.9777	0.8847	0.8885	0.9182	0.9765
	F1 score	0.9376	0.9242	0.921	0.9392	0.9504
Test accuracy		0.8544	0.8932	0.8599	0.9113	0.9319

Bold values indicate that the metric is optimal for that row

Secondly, Fig. 4 describes the confusion matrix of each model prediction test set. This can give us a rough idea of how all images are classified and where most misclassifications occur. It can be seen from the figure that the probability of the prediction error of the Normal class is greater than the probability of the prediction error of the other classes. This shows that the up-sampling method embedded in the algorithm has a positive effect. It makes the model not biased to ignore infected patients during the detection process. When the MFDNN model discriminates four types of samples, the misjudgment rate is relatively low.

Fig. 4

Confusion matrix of Densenet201, ResNet50, VGG19, GoogLeNet, MFDNN model

Ablation experiment

To prove the necessity of steps 6 and 7 in the MFDNN classification model, we designed an ablation experiment to explore the accuracy of multi-channel feature fusion combined with PSN. The first step is to remove the PSN and additional layers, generate a simple CNN, train the model, and record the evaluation index value of the model. The second step is to remove the additional layer, keep the remaining part of the MFDNN model and name it DNN. Again, train the model and record the evaluation index value of the model. The third step is to use only the feature fusion method to generate a multi-channel feature convolutional neural network (MFCNN), train the model and record the evaluation index value of the model. Finally, the fourth step compares all the model detection results with the MFDNN model classification results.

The impact of additional layers

In order to verify the influence of the additional layer, we compared the performance of the three models of CNN, DNN, and MFDNN. As shown in Table 2, we analyzed the performance of CNN and DNN models from accuracy, recall, F1 score and test accuracy. In most indicators of all categories, the evaluation indicators of the MFDNN model are higher than other models. In terms of test accuracy, the MFDNN model is 3.33% higher than the DNN model.

Table 2

CNN, DNN, MFDNN classification technical report

		CNN	DNN	MFDNN
COVID	Precision	0.7538	0.769	0.9369
	Recall	0.9732	0.984	0.9447
	F1 score	0.8496	0.8633	0.9358
Lung_Opacity	Precision	0.8877	0.886	0.9144
	Recall	0.8189	0.868	0.9068
	F1 score	0.8519	0.8769	0.9106
Normal	Precision	0.9367	0.9603	0.9348
	Recall	0.8805	0.8851	0.9431
	F1 score	0.9077	0.9211	0.9389
Viral pneumonia	Precision	0.7472	0.8364	0.9257
	Recall	1	0.9825	0.9765
	F1 score	0.8553	0.9036	0.9504
Test accuracy		0.875	0.8986	0.9319

Bold values indicate that the metric is optimal for that row

Secondly, we have a detailed understanding of the classification effect of the model in different categories through the confusion matrix. By comparing the confusion matrix analysis of the CNN and DNN models in Fig. 5 with the MFDNN model in Fig. 4, we find that the MDNN model has smaller errors than the CNN and DNN models in detecting all categories. Therefore, the additional layer can capture a wider range of information from the image, thereby significantly improving the performance of the model.

Fig. 5

Confusion matrix of CNN, DNN model

Role of PSN

According to the data analysis in Table 3, we found that MFCNN and MFDNN models have a large gap in accuracy, recall and F1 score. The test accuracy of the MFDNN model is 0.0333 higher than that of the MFCNN model. Compared with the confusion matrix of the MFCNN model and the MFDNN model in Fig. 6, we find that the MFCNN model has greater errors than MFDNN in detecting all categories. Because the MFCNN model does not include the PSN module, the MFDNN model includes the PSN module. Therefore, we conclude that the function of PSN is to perform secondary feature extraction based on the extracted features, so that the features can be used more effectively and the accuracy of the model can be improved.

Table 3

MFCNN, MFDNN classification technical report

		MFCNN	MFDNN
COVID	Precision	0.7911	0.9369
	Recall	0.9811	0.9447
	F1 score	0.8759	0.9358
Lung_Opacity	Precision	0.8977	0.9144
	Recall	0.8536	0.9068
	F1 score	0.8751	0.9106
Normal	Precision	0.9465	0.9348
	Recall	0.8972	0.9431
	F1 score	0.9212	0.9389
Viral pneumonia	Precision	0.8513	0.9257
	Recall	0.9745	0.9765
	F1 score	0.9087	0.9504
Test accuracy		0.9	0.9319

Fig. 6

Confusion matrix of MFCNN, MFDNN model

This paper proposes an MFDNN algorithm to screen people infected with COVID19. The algorithm integrates data oversampling technology and a MFDNN model to carry out the training. In the experiment, we used the publicly available CXI database to train the model. First, compared with traditional deep learning models (VGG19, GoogLeNet, ResNet50, Densenet201), the MFDNN model obtains an average test accuracy of 93.19% in all data. Furthermore, in each type of screening, the precision, recall, and F1 Score of the MFDNN model are also better than traditional deep learning networks. Through ablation experiments, we proved that multi-channel CNN is superior to single-channel CNN, additional layer and PSN module, and indirectly proved the sufficiency and necessity of each step of the MFDNN classification method. Secondly, comparing the latest CoroDet model, the MFDNN algorithm is 1.91% higher than the CoroDet model in the four-classification experiment of COVID19 infected persons. However, the limitation of this experiment is mainly in the disadvantages of X-rays. For opaque images of the lungs, RT-PCR is needed to assist in the screening of COVID19 infections.