Image Segmentation of Retinal Blood Vessels Based on Dual-Attention Multiscale Feature Fusion.

Aiming at the problem of insufficient details of retinal blood vessel segmentation in current research methods, this paper proposes a multiscale feature fusion residual network based on dual attention. Specifically, a feature fusion residual module with adaptive calibration weight features is designed, which avoids gradient dispersion and network degradation while effectively extracting image details. The SA module and ECA module are used many times in the backbone feature extraction network to adaptively select the focus position to generate more discriminative feature representations; at the same time, the information of different levels of the network is fused, and long-range and short-range features are used. This method aggregates low-level and high-level feature information, which effectively improves the segmentation performance. The experimental results show that the method in this paper achieves the classification accuracy of 0.9795 and 0.9785 on the STARE and DRIVE datasets, respectively, and the classification performance is better than the current mainstream methods.

1. Introduction

Various ophthalmological diseases and cardiovascular and cerebrovascular diseases will affect retinal blood vessels to varying degrees, such as deformation and hemorrhage. In recent years, retinal vessel segmentation techniques have been applied to the diagnosis of various ophthalmic diseases [1]. For retinal segmentation algorithms, there are roughly two categories, supervised methods and unsupervised methods. Unsupervised methods are rule-based segmentation algorithms such as matched filtering [2, 3], ship tracking [4], and morphological methods [5, 6]. However, these algorithms lack generalization ability, which leads to the formation of false edges. Among the supervised algorithms, methods based on image processing [7] and optimization [8] are used to process retinal images. The optimization algorithm proposed in the literature [9-11] is a good reference idea in the direction of feature extraction.

Zhu et al. [12] proposed an ensemble method for color retinal blood vessel segmentation based on supervised learning. The method uses feature vectors as input datasets, trains weak classifiers through classification and regression trees, and uses iterative training to construct an AdaBoostClassifier for blood vessel segmentation. Upadhyay et al. [13] applied two multiscale methods, namely, local directional wavelet transform and global curvelet transform, which were effectively used for vessel enhancement and segmentation. Wang et al. [14] proposed a hierarchical retinal vessel segmentation method. First, histogram equalization and Gaussian filtering are used to enhance the green channel fundus image, and then, a simple linear iterative clustering method is used to segment the superpixels, and a pixel is randomly selected from each superpixel to represent the entire superpixel as a sample for feature extraction; finally, CNN extracts hierarchical features, and random forest is used as a classifier. In ref [15], retinal images were enhanced using Principal Component Analysis- (PCA-) based grayscale transformation and contrast-limited adaptive histogram equalization (CLAHE) and a new matched filter are designed to segment blood vessels. Reference [16] proposes a genetic algorithm to optimize the parameter adjustment process, and reference [17] proposes a weighted composite model structure. This also provides some new ideas for follow-up research.

In recent years, deep learning algorithms have been successfully applied to retinal blood vessel segmentation tasks because they can adaptively extract features at the abstraction level of images, greatly improving the accuracy of blood vessel segmentation. Xiao et al. [18] improved U-Net network, which incorporates a residually connected convolutional network and a new weighted attention model for retinal vessel segmentation. Soomro et al. [19] proposed a deep convolutional neural network (CNN) for retinal blood vessel segmentation, which successfully improved the segmentation quality of tiny blood vessels. Haft-Javaherian et al. [20] proposed CNN with fully connected layers for segmentation of 3D blood vessels in in vivo volume images obtained by multiphoton microscopy. Long et al. [21] propose a fully convolutional network (FCN) to solve the problem of semantic segmentation, using ground truth as supervision information to train the network for pixel-level prediction, thereby further extending image-level classification to pixel-level classification. Reference [22] proposed a supervised network (CTF-Net) with feature augmentation module (FAM), which can successfully reduce the number of parameters of the model and improve the accuracy of the model. In [23], a cross-connected convolutional neural network (CcNet) was proposed, and the cross-connection of the main path and the secondary path of CcNet fused multilevel features. In [24], a computationally efficient, differentiable loss function (soft-clDice) was proposed for training arbitrary neural segmentation networks. These methods greatly improve the speed and accuracy of retinal vessel segmentation.

However, the above methods still have problems in fine segmentation of retinal vessels. Aiming at this problem, we propose a segmentation method of retinal blood vessels based on dual-attention multiscale feature fusion residual network. The proposed method uses components such as ECA-Net and SA, which effectively enhance the edge and global information processing of feature maps. The experimental results show that the method proposed in this paper has obvious advantages in accuracy, specificity, sensitivity, etc. and is more effective in the processing of retinal blood vessel segmentation details.

The main contributions of this paper are as follows:

We propose a feature fusion residual module including ECA-Net to adaptively calibrate weight features to avoid gradient dispersion and network degradation

We use SA module multiple times in the feature extraction network to extract image features from low dimensional to high dimensional, effectively exploring the feature dependencies of spatial and channel dimensions

Compared with the other five recent fundus vessel segmentation networks, the proposed network shows the best performance on both datasets

The rest of this paper is arranged as follows. Section 2 details the proposed method for fundus vessel segmentation. Section 3 describes the experimental validation and discusses the results. Section 4 summarizes the full text and introduces future research directions.

2. Methods

In this subsection, we first introduce the proposed model; then, we introduce the spatial channel attention network (SANet) and the lightweight attention network (ECA-Net), and finally, we introduce the feature fusion combined with the lightweight attention network residual module.

2.1. Proposed Method

In view of the existing research foundation, to improve the segmentation accuracy of retinal blood vessel images, this paper proposes a fundus blood vessel segmentation strategy based on dual attention feature fusion residual network. The network uses SA and ECA modules many times to adaptively select information that is beneficial to segmentation; the low-level texture, shape, and other features are fused with high-level abstraction level features, which greatly enhances the segmentation performance of the network. The execution flow of the entire network is shown in Figure 1. Throughout the training process, when inputting data with a size of 64 × 64 × 3, the C_B_R, SA, and C_B_R modules are sequentially passed through to obtain a feature map F1 with a size of 64 × 64 × 16. The step size of these two C_B_R is 1, the convolution kernel is 3, and the filters are 32 and 16, respectively. On the one hand, F1 obtains a feature map F4 with a size of 32 × 32 × 64 through an SA module and a C_B_R module with a stride of 2, a convolution kernel of 3, and a filter of 64; On the other hand, F1 passes SA, C_B_R , Block1, and C_B_R modules and obtains a feature map F2 with a size of16 × 16 × 64. The stride of these two C_B_R modules is 2, the convolution kernel is 1, and the filters are 32 and 64, respectively. F2 first obtains a feature map F5 with a size of 32 × 32 × 64 through an ECA module and an upsampling module with a stride of 2. On the other side, F2 obtains a feature map F3 with a size of 32 × 32 × 64 through Block1, C_B_R, and the upsampling module in turn. The step size of the C_B_R module is 1, the convolution kernel is 1, the filters are 64, and the step size of the upsampling module is 4. After that, add the feature maps F4 and F5, and then, perform the Concat operation with F3 to obtain a feature map F6 with a size of 32 × 32 × 128. Then, pass F6 through two C_B_R modules and an upsampling module to get the final output result, where the stride of the two C_B_R modules is 1, the convolution kernel is 1, and the filters are 32 and 3, respectively. The SA, ECA, and Block1 modules are described in detail in the following sections. In the testing process, we get the trained model according to the training process and then infer the input image to get the final predicted image.

Figure 1

The training and testing process of the proposed network.

2.2. Spatial Channel Attention Network (SANet)

Attention mechanisms, which enable neural networks to accurately focus on all relevant elements of the input, have become an important part of improving the performance of deep neural networks. The attention mechanisms widely used in computer vision research mainly include spatial attention and channel attention, which are used to capture pixel-level pairwise relationships and channel dependencies, respectively. In this paper, the Shuffle Attention (SA) module [25] is first used to explore feature dependencies in both spatial and channel dimensions, as shown in Figure 2. This module aggregates all subfeatures, realizes information communication between different subfeatures, and effectively combines spatial attention and channel attention. Let the input feature map be XϵR, where C represents the number of channels, H represents the height of the feature map, and W represents the width of the feature map.

Figure 2

Spatial channel attention network (SA).

SA first divides the input feature maps into G groups along the channel direction, X = [X1, ⋯.., XG], XϵR, where each different subfeature captures different semantic information and then reassigns weight information to each group of submodules. Specifically, each group of submodules is fed into a parallel spatial attention module and a channel attention module, respectively.

For the channel attention module, compared with SEBlock, this paper uses global average pooling (GAP) to embed global information and generate SϵR feature map, which greatly reduces the amount of parameters, which is defined as

Finally, the final channel attention result is output through the gating mechanism and the sigmoid activation function: Where W1ϵRand b1ϵRare used to move and zoom S.

Furthermore, the authors use spatial attention to select meaningful spatial information from feature maps. Different from channel attention, the author first uses group norm for the grouped feature map X to obtain spatial statistics and then uses F(•) to enhance the information representation of the feature map, as shown in Equation (3): where W2ϵR and b2ϵR.

Finally, the channel attention result and the spatial attention result are concatenated through the Concat operation to obtain the weight information of the redistributed feature map.

2.3. Lightweight Attention Network (ECA-Net)

The SA module greatly increases the number of parameters of the network while redistributing the weights of the feature maps. On the other hand, the direct correspondence between channel and attention weights is essential, and proper crosschannel interaction can significantly reduce the model complexity while maintaining performance. In this paper, ECA-Net [26] is used to adaptively allocate the network feature. As shown in Figure 3, the author uses a local crosschannel interaction strategy without dimensionality reduction and selects an adaptive size convolution kernel to ensure the coverage of local crosschannel interaction. Specifically, the input feature map is first subjected to a global average pooling operation; then, a crosschannel one-dimensional convolution operation is performed through a convolution kernel of size K, and finally, the input feature map is multiplied by the sigmoid activation function for output.

Figure 3

Lightweight attention network (ECA-Net).

δ is the sigmoid activation function a represents multiplication, GAP represents global average pooling, K represents the adaptive convolution kernel, and K can be defined as where c is the number of input feature map channels and r, b are adjustable variables; in this paper, r = 2 and b = 1.

2.4. Lightweight Attention Feature Fusion Residual Module (Block1)

Considering that the deep network will cause gradient disappearance, gradient dispersion, etc., this paper designs a residual network with a lightweight attention module, which adaptively calibrates the feature information and integrates the feature information of different levels, as shown in Figure 4. This module consists of modules such as 2D convolution, BN, Relu activation function, and ECA-Net. Specifically, it can be described as

Figure 4

Residual networks with lightweight attention modules.

Among them, x represents the input feature map, y represents the output feature map, R represents the Relu activation function, BN represents batch normalization, and Cov represents the two-dimensional convolution operation. The convolution kernel of the first two-dimensional convolution is 3 × 3, and the convolution kernel of the second two-dimensional convolution is 1 × 1, and the entire module does not change the shape and size of the feature map.

3. Experimental Results and Analysis

In this section, we first introduce the dataset to be used and the preprocessing method of the dataset; then, I introduce the experimental parameter settings and evaluation criteria; and finally, we analyze the experimental results in detail.

3.1. Dataset and Data Preprocessing

The STARE dataset [27] consists of 20 images of retinal fundus vessels of size 700 × 605 pixels, which are part of the Dutch Diabetic Retinopathy Screening Project. Since the validation set and training set are not clearly divided, this paper uses the first 10 images as the training set and the last 10 images as the test set. The DRIVE dataset [28] consists of 40 images of size 565 × 584 pixels, which are divided into datasets and validation sets according to the official division. To prevent overfitting caused by too few datasets, for the two datasets, this paper firstly uses flipping, rotation, and translation to augment the data and then extracts patches with a size of 64 × 64 pixels from the large-resolution images as the final training data.

3.2. Experimental Parameter Settings

The experiments in this paper are based on the PyTorch 3.7 deep learning framework and the Python 3.6.9 compiler, and the GPU used is an RTX3070 with 8 G of video memory. The epoch of the model is set to 100, the batch size is set to 128, the optimizer is set to Adam, the initial learning rate is 0.001, and the exponential decay rate is set to 0.9. The loss function selects the cross-entropy loss function, which is defined as follows: where y is the true label and is the predicted label.

3.3. Evaluation Criteria

To evaluate the performance of the proposed method for fundus retinal image segmentation. This paper uses accuracy, sensitivity, and specificity as evaluation metrics. The accuracy represents the percentage of correctly segmented pixels in the entire image, the sensitivity represents the percentage of correctly segmented blood vessel pixels in the total blood vessel pixels, and the specificity is the percentage of correctly classified background pixels in the total background pixels. The details are shown in the following equation: where TP represents the number of correctly segmented vessel pixels and TN represents the number of correctly segmented background pixels. FP represents the number of incorrect segmentation of background pixels, and FN represents the number of incorrect segmentation of blood vessel pixels.

3.4. Analysis of Results

To verify the effectiveness of the proposed method, this paper compares with other state-of-the-art methods on the STARE and DRIVE datasets, including methods based on Frangi filter and Otsu [29], learning discriminative unary features through a 2D convolutional neural network, an improved dense CRF model [30], DoubleU-Net [31], a conditional deep convolutional generative adversarial network-based approach [32], an attention-fused U-Net network [33]. The specific quantitative comparison results are shown in Table 1. The qualitative comparison results are shown in Figure 5.

Table 1

Evaluation metrics of different methods in DRIVE and STARE datasets.

Methods	DRIVE			STARE
	Acc	Sen	Spe	Acc	Sen	Spe
Meng et.al. [29]	0.9383	0.7871	0.9664	0.8871	0.7372	0.9391
Zhou et.al. [30]	0.9469	0.8078	0.9674	0.9585	0.8065	0.9761
Jiang et.al. [31]	0.9642	0.8201	0.9843	0.9667	0.7991	0.9854
Jiang et.al. [32]	0.9608	0.8274	0.9775	0.9771	0.8538	0.9878
Li et.al. [33]	0.9678	0.7921	0.9810	0.9678	0.8392	0.9823
Proposed	0.9795	0.8258	0.9896	0.9785	0.8368	0.9889

Figure 5

Visualization results of different methods on DRIVE and STARE datasets: (a) original image; (b) ground truth; (c) literature [29]; (d) literature [30]; (e) literature [31]; (f) literature [32]; (g) literature [33]; (h) proposed.

As can be seen from Table 1, the Accuracy (Acc), sensitivity (Sen), and specificity (Spe) values of the proposed method on the DRIVE dataset are 0.9795, 0.8258, and 0.9896, respectively; the overall accuracy is better than other methods, compared with the suboptimal method [33]; the accuracy, sensitivity, and specificity values lead by 1.17%, 3.37%, and 0.86%, respectively, achieving leading classification performance. The accuracy, sensitivity, and specificity values on the STARE dataset are 0.9785, 0.8268, and 0.9886, respectively. Compared with the suboptimal method [32], the accuracy and specificity values lead by 0.14% and 0.11%, respectively, and the sensitivity value lags behind by 1.7%, which also achieves better classification performance. Figure 5 shows the visualization results of different methods on two datasets, where the first three rows are the results of the DRIVE dataset, and the last row is the results of the STARE dataset. The first column is the original fundus retinal image, the second column is the ground-truth label, the third to seventh columns are the visualization results of the literature [29-33], and the last column is the visualization results of the proposed method in this paper. It can be seen from the results that the method proposed in this paper can identify more detailed parts of blood vessels, which verifies the performance of the proposed method.

4. Conclusion

This paper proposes a dual-attention-based multiscale feature fusion residual network for retinal vessel image segmentation [34-37]. The paper first designs a feature fusion residual module including ECA-Net, which effectively extracts image details and solves problems such as gradient dispersion and network degradation; then uses SA and ECA and modules such as feature fusion; adaptively aggregate features that are effective for segmentation; enhance network feature representation; and finally, effectively aggregate features at different stages to improve the segmentation performance of the network. The experimental results show that the image segmentation method proposed in this paper achieves the best classification performance. The accuracy, sensitivity, and specificity values on the DRIVE dataset are 0.9795, 0.8258, and 0.9896, respectively; on the STARE dataset, the accuracy, sensitivity, and specificity values are, respectively, 0.9785, 0.8268, and 0.9886, which fully demonstrates that the method proposed in this paper can effectively capture detailed features such as vessel endings. Since manual labeling is difficult and labor-intensive, we will focus on the application of unsupervised segmentation methods in retinal blood vessel image processing tasks in the future.