Flame Edge Detection Method Based on a Convolutional Neural Network.

In this study, an improved flame edge detector based on convolutional neural network (CNN) was proposed. The proposed method can generate edge graphs and extract edge graphs relatively effectively. Our network architecture was based on VGG16 primarily, the last two max-pooling operators and all full connection layers of the VGG16 network were deleted, and the rest was taken as the basic network. The images output by the five convolution layers were upsampled to the size of the input images and finally fused to the edge image. Error calculation and back propagation of the fusion image and label image are carried out to form a weakly supervised model. Using the open datasets BSDS500 to train the network, the ODS F-measure can reach 0.810. Various experiments were carried out on different flame and fire images, including butane-air flame, oxygen-ethanol flame, energetic material flame, and oxygen-acetylene premixed jet flame, and the infrared thermogram was also verified by our method. The results demonstrate the effectiveness and robustness of the proposed algorithm.

Introduction

Flame temperature is an important parameter in the combustion process.[1] As one of the seven international basic units, it has important physical and chemical significance, improves the accuracy of flame temperature measurement technology research, and has great significance in military applications and industrial fields, such as aviation, aerospace,[2] nuclear explosion,[3] metallurgy,[4] and thermal power generation.[5] However, in a complex temperature field measurement, with high pressure, high spin, high impact, strong noise, and the influence of environmental factors, such as the test system to produce strong coupling interference, the temperature usually cannot be measured.

As an important step in flame image processing, flame edge detection often lays the foundation for flame temperature measurements and three-dimensional temperature field reconstruction. First, the definition of the flame edge can be used to directly calculate the position, shape, size, and other characteristic parameters of the flame. Second, flame edge processing can filter the background noise of the oblique tone, which minimizes the influence of background noise and significantly reduces the amount of data in 3D reconstruction. Third, edge intelligence plays an important role in fire safety and management. It is necessary to report abnormal conditions such as fires in real time during monitoring. In addition, edges can be used to divide a group of flames, which helps in multi-flame monitoring in the industry and improves energy efficiency.

In the last few decades, several techniques have been proposed for identifying fire and flame image edges. Bheemul[6] proposed an effective flame contour extraction technology, which can obtain the complete flame contour by finding the place with the greatest gray variation as the peak point and connecting the peak points successively. However, this technique is only suitable for stable and simple flame images. Toreyin et al.[7] proposed an online learning fire detection method based on the hidden Markov model and the time flicker modeling method based on wavelet, which successfully detected fire in real-time video, and the false alarms issued by earlier methods could be drastically reduced. Qin Jiang[8] proposed an improved canny edge detector based on the adaptive smoothing combination of the geometry of the flame, which was used to detect fire regions in large-space fire videos. Chacon-Murguia and Perez-Vargas[9] detected and analyzed fire information in a video through the analysis of shape regularity and intensity saturation features. Qiu et al.[10] proposed an adaptive edge detection algorithm for flame image processing, which is based on the Sobel edge detection algorithm and can adjust high and low thresholds according to the scene, which has good robustness. Gupta et al.[11] proposed a simple and robust method for flame image edge detection based on local binary mode and threshold technology. The method first acquired the flame contour through the local binary mode and made the flame edge clearly visible by a double threshold adjustment. Gaidhane[12] proposed a flame edge detection method based on the local binary mode, double threshold, and Levenberg–Marquardt optimization techniques. This study primarily used the non-maxima suppression technique to obtain thinner and clearer flame edges.

Although these methods have their own advantages for specific application scenarios, such as fire detection or shape reconstruction in complex backgrounds or for detecting early fires and triggering fire alarms, they all have some limitations. For example, some of the detected flame edges were unclear, discontinuous, or did not match the actual flame shape. To detect the size and shape of the flame, as well as its geometric characteristics, it is necessary to obtain clear, thin, continuous, and as close as possible flame edges. An effective and efficient technique remains a challenge for image processing researchers.

In this study, an edge detection operator was used for flame edge detection, and it was designed based on the convolutional neural network (CNN). Our network architecture was based on VGG16 network primarily, the last two max-pooling operators and all full connection layers of the VGG16 network were deleted, and what remained was our basic network. Our network was trained with the BSDS500 datasets and validated on a variety of flame images. The results show that our method can effectively detect the flame edge.

Overview of Edge Detection Techniques

Edges are an important feature of an image. In general, an image is regarded as the input signal, and the edge of the image refers to discontinuous points or a collection of points with drastic changes in the gray value of the image, which can be represented by the first and second derivatives in mathematics. An image can be represented as a two-dimensional function f(x, y). Its gradient can be defined aswhere g and g are vectors in the x and y directions.

The magnitude of this vector can be expressed as

In general, it can be approximated by an absolute value, ∇f ≈ |g| + |g|, and using an absolute value to approximate avoids the calculation of squares and square roots but still has the property of derivation. In practical calculation, the magnitude of the gradient is usually referred to as ″gradient″.[13]

A basic feature of the gradient vector is that the direction of f maximum rate of change at the coordinates (x, y) is indicated, and the angle at which the maximum rate of change occurs is

Image edge extraction is to use a convolution kernel to solve the gradient of x and y. By obtaining the sum of the absolute values of the two convolution images, the corresponding edge values can be obtained, which is the Sobel edge detector.[14]Figure is the convolution factors of Sobel in the x-direction and the y-direction.

Figure 1

Convolution factor of Sobel in the (a) x-direction and the (b) y-direction.

The Sobel and Prewitt operators[15] are first-order differential operators developed based on the Roberts operator. They have a certain inhibitory effect on noise, but their edge-positioning accuracy is not high.

Because the first-order differential operator can not effectively identify the edge when processing an image with a uniform grayscale, the second-order differential operator is better for edge detection. Second-order differential operators mainly include the Laplacian, LOG, DOG, and Canny operators. Among them, the Canny edge detection algorithm[16] is a multilevel edge detection algorithm developed by John F. Canny in 1986, which is considered to be the optimal algorithm for edge detection. Compared to other edge detection algorithms, the edge recognition accuracy was much higher.

Generally, the purpose of edge detection is to significantly reduce the image data size while preserving the original image attributes. Currently, many algorithms are available for edge detection. Although the Canny algorithm has a long history, it can be said to be a standard algorithm for edge detection and is still widely used in research.

The brief steps of the Canny operator are as follows: Denoise, look for gradients, apply non-maximum suppression to filter out non-edge pixels, apply the double threshold method to determine the possible potential threshold, and use lag technology to find boundaries.

In the last decade, some techniques based on edge detection have been applied to flame images and analyses as described in the Introduction.[6−12] Although significant progress has been made, there are still some limitations to this study. In the era of deep learning, especially with the emergence of the convolutional neural network,[17] CNN has advantages such as powerful ability in automatic learning of advanced representations of natural images, and edge detection using CNN has become a new trend. In 2015, Xie et al.[18] proposed holistically nested edge detection (HED), which was used to detect and extract the edges of natural images. HED has the advantage of increasing the feel field of feature maps and is robust; however, it has the disadvantage of increasing parameters. Because HED is based on VGG-16[19] as the backbone architecture, only the last layer or before the pooling layer is considered as a feature, and the characteristics of the middle layer are ignored. Until ResNet appeared, the network’s rich convolutional features were effectively applied to various visual tasks. In 2017, Liu et al.[20] proposed a full convolutional network to efficiently utilize the richer convolutional feature (RCF) of each convolutional neural network layer, optimize the loss function, and obtain the best edge detection effect of the year.

However, as shown in Figure ((a) is the original figure, (b) is the Sobel operator, (c) is the Prewitt operator, (d) is the LOG operator, (e) is the Laplacian operator, and (f) is the Canny operator), while this algorithm on the edge of the picture is good, the use of the traditional edge detection method for flame edge detection cannot obtain continuous, clear, and thin edges. The complexity of real images, such as gain ideal edges, is not always possible, so temperature measurement, such as the 3D reconstruction task, is complicated.

Figure 2

Results of traditional edge detection methods. (a) Origin images. (b) Sobel method. (c) Prewitt method. (d) LOG method. (e) Laplacian method. (f) Canny method.

Improved Flame Edge Detection

Flame Edge Detection Architecture

In this section, we will introduce the architecture of flame edge detection. As shown in Figure , it is obvious that our architecture is based on the VGG16[19] network and ResNet.[21] VGG16Net comprises 13 convolution layers and three fully connected layers, which are divided into five stages. A pooling layer was connected after each stage. VGG16Net uses a stack of smaller convolution kernels to increase the receptive field and it can reduce the number of parameters of the network; on the other hand, the extracted useful feature by each convolution can be coarser. The starting point of our architecture is based on the VGG16 network primarily.

Figure 3

Our flame edge detection network architecture. The input is an image with arbitrary sizes, and our network outputs an edge possibility map in the same size.

The last stage and fourth pooling layer are reduced in our network. We used the same convolution block as VGG16Net with a kernel size of 3 × 3, with a stride of 1 and padding a pixel at the edge of the image. Each convolution is followed by batch normalization and ReLU as the activation function. For the maximum pooling layer, we used a 3 × 3 kernel with a stride of 2. Owing to many convolution operations, some important edge information will be seriously lost as described in detail in DeepEdge.[22]We added a 1 × 1 convolution[21] block after the third convolution block to average the subsequent convolution output, and the yellow square in the figure can be reflected.

Our network is a multiscale fusion edge detection algorithm, such as HED[18] and RCF.[20] In the main architecture, the size of the side output will decrease after max pooling; we need to upsample the side output to be the same size as the input image. The upsampling process was realized using bilinear interpolation.

The feature output of the first convolution layer is not high in semantics and has more noise, but it contains more location and detail information with a higher resolution.[23] We used the output of the first convolution layer and the output of the pooling layer as the side output, and a total of five side outputs were concatenated and fused together to form the flame edge image.

Compared with HED, our flame edge detection algorithm is mainly different in the following two points: on one hand, HED only considers the output of each layer of VGG network; while ignoring the information contained in the middle layer, our algorithm utilizes the output of all the convolution layers, which enables us to capture more edge information. On the other hand, since each convolution will lose important features after HED performs many convolutions, the 1 × 1 convolution block was added after the third convolution block to carry out average edge connections.

Loss Functions

We denote S = {(X, Y)} as the training set, where X is the input image, n is the total number of images in the datasets, and Y = {y1, y1···y} is the pixel-wise ground true label. The sampled pair from the training set is denoted as (X, Y).

Then, as the model is supervised, to address the class imbalance problem in edge detection, we apply a class-balancing technique as described in HED, which is solved as followswhere β is defined to balance the losses of two classes: edge and non-edge, β = |Y–|/|Y|, and 1 – β = |Y+|/|Y|. |Y–| and |Y+| are the number of edge and non-edge pixels in the ground truth, respectively. where W is the collection of all network parameters, w is the n-corresponding parameter, Pr(y = 1|X; W, w) ∈ [0,1] is the probability of predicting y = 1 at pixel j, and the probability is obtained by applying a standard sigmoid function σ( · ).

A mixed-weight layer was added to the architecture to extract powerful high-level features. We denoted the parameters unique to the main branch as wm and the parameters unique to the side branch as w. We then minimize eq via a standard stochastic gradient descent.where and are the losses of the main branch and side branch and the weight α of the side branch loss is set at different scales on different datasets.

Experiments on Flame Edge Detection

We implemented our network using the publicly available PyTorch library.[24] The entire network was initialized using a pretrained VGG-16 network model. The Datasets in image edge detection include MDBD,[25] NYUD,[26] CID,[27] PASCAL,[28] BSDS300, and BSDS500.[29] In this article, the datasets we used was the BSDS500 (Berkeley segmentation data set). The datasets provided by the Computer Vision Group of Berkeley University can be used for image segmentation and edge detection. It contains 200 training, 100 verification, and 200 test images, each of which is labeled by four to nine annotators. The global learning rate was set to 1 × 10–4 and was divided by 10 after every 1000 iterations. The Adam optimizer was adopted to train the network. We set the batch size to eight for the experiment, and according to the loss–epoch curve in Figure , the x axis represents the number of iterations of training and the y axis represents the loss function of training on the BSDS500 datasets. Our network converges after 4000 epochs.

Figure 4

Loss–epoch curve on BSDS500 datasets.

We evaluated our flame edge detection performance using commonly used evaluation metrics, including average precision (AP), fixed contour threshold (ODS), and per-image best threshold (OIS). The F-measure of ODS is also be considered

The performance of the human eye in edge detection is known as the 0.803 ODS F-measure; the HED is 0.788, and the Canny operator is 0.611. Figure shows the precision/recall curves on the BSDS500 datasets. Recall on the x axis represents the probability of detecting real edge pixels occupying all real edge pixels. The y axis precision represents the probability that edge pixels generated through our network account for real edge pixels. The F-measure is the harmonic mean of precision and recall. The ODS F-measure can be calculated from eq , our flame edge detection method achieves better results than the average human performance, which can reach 0.810.

Figure 5

Precision/recall curves on BSDS500 datasets.

Results and Discussion

After training our flame edge detection method, as described in Section , we used an ethanol flame, butane flame, and a quasi-static simulation of the explosion field designed to evaluate our algorithm. The flame images were taken by the high-sensitivity monochrome camera Dhyana 400BSI V2 camera with an effective area of 13.3 mm × 13.3 mm, pixel size of 6.5 μm × 6.5 μm, resolution of 4 MP(2048(H) × 2048(V)), and frame rate of 74 fps@CameraLink. A narrow band filter is installed in front of the camera with a central wavelength of 768 nm and a bandwidth of 20 nm. We captured the flame image at 40 frames with an exposure time of 15 μs. As shown in Figure , the top row shows the original image of different flames captured by the Dhyana camera. The following row shows the edge map obtained by our flame edge detection method. Figure a shows the butane–air flame. The combustion device is described in the paper.[30]Figure b shows the oxygen–ethanol flame. Figure c is the energetic material flame. Figure d is the oxygen–acetylene premixed flame. Compared to the traditional methods shown in the Figure , it is obvious that our algorithm can successfully detect the clear edge of the flame, something that traditional edge detection methods cannot achieve.

Figure 6

Some results of flame edge detection. Original images (top row) and edge images (following row): (a) butane–air flame; (b) oxygen–ethanol flame; (c) energetic materials flame; (d) oxygen–acetylene premixed jet flame.

After extracting the edge of the oxygen–ethanol flame, as shown in Figure b, we designed an element doping method[30] combined with laser-induced breakdown spectroscopy (LIBS)[31,32] to measure flame temperature. First, we used LIBS to analyze the energy spectrum of the energetic materials and selected K2SO4 as an appropriate dopant. A black-body furnace calibration experiment was conducted based on the Planck black-body radiation law. Relationships between the temperature of the blackbody cavity and image gray value (T–G), radiation brightness, and image gray value (L–G) were calculated. Finally, the temperature field of the oxygen ethanol flame was calculated using the equation[30]

After a clear edge is extracted, all pixels outside the edge are removed based on previous work so that only the effective flame position can be calculated, greatly reducing the amount of calculation. We used standard thermocouples to measure the flame at 10 points and verified our method with a maximum relative error of less than 4%.

We also validated the edge detection of infrared thermography and used DL-700 infrared thermography to collect the medium oxygen–acetylene flame generated by the designed high-temperature simulated fireball device. As shown in Figure a, our device including the main base is evenly distributed around the three root guide rails; it has a step screw next to the stepper motor and drive control, and there is a bracket on the stepper screw, which is connected with a flame spray gun. The premixed acetylene–oxygen is injected into the bracket, which can form a stable spherical flame 30 cm wide and 20 cm high. A DL-700 infrared thermal imager was used for acquisition. The working band of the infrared thermal imager was 8–14 μm, and that of the standard lens was 25 mm with a temperature range of 673.2–2273.2 K. It was placed 3 m away from the flame center for shooting, as shown in Figure . From left to right are the working diagram of the high-temperature simulation fireball device, system acquisition principle diagram, infrared thermal image diagram, and edge diagram. Form Figure d), it can be observed that our edge detection algorithm is also applicable to infrared thermal images.

Figure 7

High temperature fireball simulation device. (a) Working diagram of high temperature simulation device. (b) Infrared calorimeter measurement principle diagram. (c) Infrared thermography captured by the DL-700. (d) Flame edge extraction by our algorithm.

By detecting the edge of the flame clearly, we can get many parameters of the flame, such as the area of the flame calculated by the pixel points surrounded by the edge, the perimeter of flame by edge point can be calculated, and the damage radius of the thermobaric projectile can be analyzed effectively. The background can be removed to significantly reduce the amount of data required for the flame temperature calculation[33] and 3D reconstruction. The furnace flame can be diagnosed by our flame edge detection, and some corresponding measures can be taken to improve production efficiency.

Conclusions

In this study, a flame edge detection method based on the convolutional neural network (CNN) was designed. The algorithm was improved based on HED and combined with the VGG16Net and ResNet. Using the PyTorch framework to build our network and the BSDS500 datasets for training, the F-measure of our network reached 0.810, which is higher than that of the HED algorithm. We also compared it with the traditional edge-detection algorithm. Experimental results show that our algorithm can clearly detect the continuous edge of the flame and extract the edge of the infrared thermal image well. Our flame edge detection method can be used to further calculate the flame area, perimeter, and other related parameters, and it can easily filter out the background information of the flame and reduce the amount of data calculation. We used the proposed algorithm combined with the element doping method to measure the flame temperature; the maximum relative error of temperature measurement is less than 4%. At present, our experimental objects are only targeted at small-sized and medium-sized flames in a laboratory environment, and we will conduct further studies on forest fires and thermobaric bombs. On the other hand, future studies could fruitfully explore this issue further by real-time edge detection of the flame.