Literature DB >> 35639677

A low-light image enhancement method with brightness balance and detail preservation.

Canlin Li¹, Jinjuan Zhu¹, Lihua Bi¹, Weizheng Zhang¹, Yan Liu¹.

Abstract

This paper proposes a new method for low-light image enhancement with balancing image brightness and preserving image details, this method can improve the brightness and contrast of low-light images while maintaining image details. Traditional histogram equalization methods often lead to excessive enhancement and loss of details, thereby resulting in an unclear and unnatural appearance. In this method, the image is processed bidirectionally. On the one hand, the image is processed by double histogram equalization with double automatic platform method based on improved cuckoo search (CS) algorithm, where the image histogram is segmented firstly, and the platform limit is selected according to the histogram statistics and improved CS technology. Then, the sub-histograms are clipped by two platforms and carried out the histogram equalization respectively. Finally, an image with balanced brightness and good contrast can be obtained. On the other hand, the main structure of the image is extracted based on the total variation model, and the image mask with all the texture details is made by removing the main structure of the image. Eventually, the final enhanced image is obtained by adding the mask with texture details to the image with balanced brightness and good contrast. Compared with the existing methods, the proposed algorithm significantly enhances the visual effect of the low-light images, based on human subjective evaluation and objective evaluation indices. Experimental results show that the proposed method in this paper is better than the existing methods.

Entities: Chemical

Mesh：

Year: 2022 PMID： 35639677 PMCID： PMC9154181 DOI： 10.1371/journal.pone.0262478

Source DB: PubMed Journal: PLoS One ISSN： 1932-6203 Impact factor: 3.752

1 Introduction

People often get some poor quality images in the process of image acquisition because of the complexity of the geographical environment and time. For example, the low-light image often has shortcomings of low contrast, uneven brightness and poor image visibility due to the lacking of light source, which is extremely unfavorable for the subsequent processing of images such as feature extraction and image segmentation. However, low-light images are often produced in real life. The general causes of low light can be divided into the following situations roughly: (1) In the night, the images are very dark due to insufficient or no light source; (2) The uneven illumination caused by the shading of buildings leads to the existence of dark regions in the image; (3) There are also in special places, such as underground mines, interiors and other dark places. People are also difficult to capture satisfactory images. It can be seen that this problem has affected the daily life of people including photography, forensics, traffic monitoring and even engineering safety monitoring [1]. Therefore, it has become a challenging and very important task to enhance the low-light image [2]. The purpose of the low-light image enhancement is to improve the brightness and clarity of the image so that people can obtain more useful and accurate information from it. In recent years, this issue has become a hot research topic, many scholars have studied and discussed it. In the past few years, a variety of image enhancement methods have been proposed. For the low-light image enhancement, many classical algorithms have been proposed, such as histogram equalization (HE) [3], gamma correction [4], retinex theory [5], etc. HE algorithms mainly focus on enhancing image contrast. HE method has received considerable attention due to its simple and direct implementation. It mainly remaps the image gray level, so that the histogram of the image follows the uniform distribution [6]. However, the traditional HE methods use the same histogram transformation for the whole image, and do not take into account the characteristics of special images. For example, the enhancement effect of this method is not significant when there are uneven brightness areas in the image. For low illumination image, due to its low dynamic range (low contrast), the image quality will be reduced in the process of histogram equalization and single gray level change. In order to solve this problem, different versions of improved methods have appeared. Considering the difference of the image local contrast, some scholars proposed adaptive histogram equalization (AHE) [7]. AHE algorithm performs histogram enhancement for each pixel by calculating the transformation function of each pixel’s neighborhood. AHE algorithm takes into account the local information of the image and effectively improves the image local contrast. However, its disadvantage is that it enlarges the noise in the homogeneous region of the image while improving the contrast. Therefore, the contrast limited adaptive histogram equalization (CLAHE) algorithm [8] is proposed. Based on AHE algorithm, CLAHE algorithm limits the histogram of each sub block, which can control the noise caused by AHE algorithm and make the image contrast more natural. In addition, in order to overcome the brightness problem, researchers have proposed some methods to maintain the average brightness based on HE algorithm. For example, the double histogram equalization method based on image mean segmentation [9] advocates that the average brightness of the input image can be maintained while the contrast of the enhanced image is low. However, the enhancement effect of this method for low illumination image in extremely dark environment is not ideal, and there is a strong oversaturation phenomenon. Similarly, recursive mean segmentation histogram equalization (RMSHE) [10] and brightness-preserving dynamic histogram equalization (BPDHE) [11] can keep the average brightness of the input image when processing the image output, but they may still not generate images with ideal natural appearance. In 2012, Khan et al. proposed an algorithm based on brightness maintenance, namely weighted average multi-segment histogram equalization (WAMSHE) [12]. Unfortunately, this method often has excessive enhancement when processing some images, due to the pixel ratio of the image segmentation regions is too large. Using the platform-based clipping method to modify the histogram can avoid intensity saturation and improve excessive enhancement. Ooi et al. [13] proposed a platform constrained double histogram equalization (BHEPL) method in 2009. Firstly, the input histogram is segmented according to the average value of the original histogram, and then the sub histogram is clipped by using the platform limit value, and then the traditional processing is performed. In addition, there are quadrant dynamic histogram equalization (QDHE) [14] and dynamic quadrant histogram equalization platform limitation (DQHEPL) [15], DQHEPL is an improvement of BHEPL, which divides all images into four sub images for equalization operation, so it can’t keep the stability of gray level for some images. The histogram-based method has achieved good results in enhancing image contrast, but often overlooked the image details. Gamma correction is also based on the grayscale transformation of the image to achieve the improvement of brightness, and the effect of detail restoration is not outstanding [16]. Retinex theory advocates maintaining the color constancy of the image [17], which can retain the original details of the image, but it is relatively weak in balancing the brightness of the image, and halos and artifacts often appear [18].Classical algorithms based on retinex theory include single-scale retinex (SSR) algorithm [19], multi-scale retinex (MSR) algorithm [20], multi-scale retinex with colour restore (MSRCR) [21], and so on. The low-light image enhancement via illumination map estimation (LIME) method proposed by X Guo et al. [22] is also based on the retinex theory, but this method refine the initial illumination map by imposing a structure prior on it, as the final illumination map, so it can get better enhancement effect. For the enhancement of low-light images, maintaining image details and improving image contrast are theoretically opposed, and it is difficult to keep a balance between the two in the same algorithm. In order to achieve the purpose of improving image contrast and retaining image details at the same time, some new methods have emerged. To achieve the trade-off among detail enhancement, local contrast improvement and maintaining the natural feeling of the image, reference [23] used weighted fusion strategy to weigh the advantages of different technologies. In reference [24], the image is divided into structure layer and texture layer for processing, so as to preserve the image details. Reference [25] uses discrete wavelet transform (DWT) to decompose the image into detail coefficients containing edge information (image details) and approximate coefficients containing illumination information, and the approximate coefficients are separately enhanced to protect image details. In addition, there are some new methods based on intelligent optimization algorithms, such as genetic algorithm (GA) [26], particle swarm optimization (PSO) algorithm [27], artificial bee colony (ABC) algorithm [28] and cuckoo search (CS) algorithm [29]. Reference [30] introduces the method of using GA and PSO algorithm to enhance contrast in spatial domain and frequency domain. Due to the introduction of GA and PSO algorithm, good image results are obtained. The introduction of such optimization algorithms can help to solve many complex problems in image processing. Reference [31] regards the enhancement of gray image contrast as an optimization problem, which is solved by combining PSO and discrete wavelet transform (DWT). By adjusting a new expansion parameter, the target fitness criterion is maximized to enhance the contrast and local details of the image. Reference [32] proposed an adaptive image enhancement method based on non-subsampled contour transformation (NSCT), fuzzy sets and ABC optimization, which helps to improve the low contrast and low clarity of images obtained in practical applications, this method optimizes fuzzy parameters through ABC algorithm, which improves its adaptability. In reference [33], a channel segmentation image enhancement method based on discrete shearlet transform and PSO algorithm is proposed. By using PSO algorithm, image artifacts and unnatural forces caused by high directional coefficients are eliminated, and the brightness component of the image is enhanced. Reference [34] firstly applied the sigmoid function to enhance the image when processing remote sensing images, and then adopted a multi-objective particle swarm optimization algorithm to maximize the amount of information in the image while maintaining image intensity. Experiments show that this method can retain significant details on the original image. Swarm intelligence optimization algorithm has made remarkable achievements in improving the adaptability of image enhancement algorithm and simplifying the complex problems in image enhancement. In summary, in the issue of low-light image enhancement, the fusion strategy is effective in theory to improve the image contrast and preserve the image details at the same time. The existing methods can be roughly divided into two ideas: One is to separate the original image details by transforming the space to achieve the effect of detail preservation. In this method, few people consider that there will be noises in the detail components, and at the same time, there may be noise amplification problems during the inverse transformation process, which resulting in unsatisfactory results. The other is to achieve the goal by weighted fusion of images with different advantages processed by each method, but the adaptability of this model is relatively weak and can’t be applied to different natural images. The results of image processing with different features may vary greatly. The method proposed in this paper is similar to the image fusion strategy, but it does not adopt the weighted fusion method. The purpose of this paper is to improve the image contrast and balance the image brightness while preserving the details of the original image. For improving the contrast of the image and enhancing the brightness of the image, HE method has been very outstanding. In this paper, the double histogram equalization with a double automatic platform (DHEDAPL) method based on improved CS algorithm is used to process the image, we can obtain the image A with balanced brightness and good contrast. In the reference [35], the author proposed the method named bi-histogram equalization using two plateau limits (BHE2PL), but plateau limits is only calculated by the statistics of histograms in this method, a single platform calculation method cannot maximize the advantages of this method. In this paper, we get the range of platform limit through histogram statistics, and use the improved CS algorithm to select optimal value in the range. In view of the aspect of retaining image details, this paper uses the method based on total variation in reference [36], to extract the main structure of the image, and then get the texture detail mask B which contains all the texture details of the image, this method can obtain the image details by extracting the image texture, and has good noise resistance, and at the same time, we don’t need the inverse transformation process, so the problem of noise amplification is avoided. Finally, we add texture detail mask B directly to image A to obtain the final enhanced image C. The structure of the rest of this paper is as follows: The second part introduces the related work in this paper; The third part describes the algorithm framework and implementation steps in detail; The fourth part conducts experiments on the method proposed in this paper, firstly, it verifies the anti-noise performance of the proposed model, then compares the method in this paper with other mainstream algorithms, and analyzes it through two aspects: human subjective evaluation and objective image quality indicators; The last chapter summarizes the work of this paper.

2 Related theories

Through the analysis of the introduction of the previous chapter, we know that in recent years, more and more scholars try to transform image problems into optimization problems. Therefore, the application of intelligent optimization algorithms in image processing is becoming more and more mature, including the PSO algorithm and the CS algorithm are frequently used, both of which have their own advantages, but also have their own limitations. The HE method is simple, has certain advantages in enhancing image contrast and brightness, and is also favored by many scholars. However, the problems of over enhancement and loss of image details of traditional histogram have always been concerned by scholars and urgently need to be solved. This paper designs a method for low-light image enhancement with balancing image brightness and preserving image details, which aims to solve the above problems. In this chapter, we mainly introduce three classical theories: particle swarm optimization algorithm, cuckoo search algorithm and histogram equalization algorithm, it paves the way for the next chapter to introduce the design method of this paper.

2.1 PSO algorithm

The particle swarm optimization (PSO) algorithm was originally an evolutionary algorithm proposed by James Kennedy and Russell Eberhart in 1995 [27]. Its development is based on the study of the social behavior of bird predation. The basic idea of the standard particle swarm optimization algorithm is: assuming that the search space is M-dimensional and there are N particles in the space, the spatial position of the i-th particle can be expressed as and velocity can be expressed as . Defining the historical best advantage of the particle (Personal best) is pbest, denoted as , the global best found by defining the population is gbest, denoted as . The first step of PSO algorithm is to initialize the population particles as random particles (random solutions), and then find the optimal solution iteratively. In each iteration, the particles update their speed and position through two extreme values pbest and gbest. For particles in the m-th (1≤m≤M) dimension, the iterative update formula is as follows: Where the parameter ω in Formula (1) is the inertia weight value, c1 and c2 are learning factors, r1 and r2 are random numbers in the range of [0, 1]. The update of the particle state depends on three aspects: The first is the inheritance of the previous particle speed based on the trust of particle in the previous own state, which belongs to the "inherited part". The second is the learning of itself, which is an extension of itself. The process of the state belongs to the "cognitive part". The third is the learning of the group, which reflects the mutual sharing and cooperation of information among particles and belongs to the "social part". The pseudo code of PSO algorithm is shown in Fig 1.

Fig 1

PSO.

The PSO algorithm is widely introduced into the application of image enhancement field because of its simple principle and easy implementation, but it is easy to fall into the local optimum and the search accuracy is not enough.

2.2 CS algorithm

In 2009, Yang and Deb developed the algorithm based on the parasitic reproduction behavior of cuckoos [29]. The cuckoo search (CS) algorithm mainly simulates the parasitic cuckoo’s nestling and the flight mechanism of certain birds [37] to effectively solve the optimization problem. The cuckoo relies on the nests of other hosts to lay eggs. The host bird treats the eggs as its own eggs. If the host bird recognizes the foreign eggs, the host bird will either throw away the eggs or leave the nest and build a new nest at a new location. Assuming that there is a bird egg in each nest, each bird egg represents a solution, and the new solution is represented by the bird egg. The basic goal of the CS algorithm is to find the best nest to incubate the bird egg by random walk. The three idealized rules to be considered to form a theory of cuckoo’s nest finding are: Each cuckoo lays one egg at a time and then places it in a randomly selected nest. In a randomly selected set of nests, those nests with high-quality eggs will be carried over to the next generation. The number of available host nests is fixed, and the probability that the eggs laid by the cuckoo will be found by the host bird is Pa∈[0, 1]. Based on the above three idealized rules, the cuckoo optimization search uses Formula (3) to update the position of the next generation nest: Where represents the i-th nest-position in the t-th generation, ⊕ represents the point-to-point multiplication, and α represents the step size factor, which is used to control the step size, and the value is usually set to 1. Levy(λ) is a random search path generated by the Levy flight that obeys the parameter λ, and its moving step length obeys the stable distribution of Levy: Where μ obeys the normal distribution, λ is the power coefficient, and λ = 1.5. It can be seen from Formula (4) that the optimization path of the CS algorithm is composed of two parts, namely frequent short jumps and occasional long jumps. This optimization method can make it easier for the algorithm to jump out of the local optimum. The pseudo code of CS algorithm is shown in Fig 2.

Fig 2

CS.

2.3 Histogram equalization

The histogram equalization (HE) algorithm [6] is a method of automatically adjusting image contrast using grayscale transformation, it calculates the gray scale transformation function by the probability density function of gray scale. It is a histogram correction method based on the cumulative distribution function transformation method. For an image X of size M×N, take (i,j) as the spatial coordinates of the pixels in the image, and X(i,j) represents the intensity of pixels in the image. The pixel value range of each coordinate is k = 0,1,…,L−1, where L is the maximum gray level in the image. Then the histogram H associated with the image describing the frequency of the intensity value in the image is defined as: Where n represents the number of occurrences of intensity k in the image. The probability density function p(k) related to H is defined as: Where p(k) represents the probability of intensity k. Then its cumulative density function c(k) is: Where X0 is the lowest intensity within the range for calculating the cumulative density function, the transformation function f(k) associated with standard histogram equalization uses c(k) to map the input image to the dynamic range [X0, X]. For the special case of HE, X0 = 0, X = L−1, the mapping function is as follows: Then the result image Y = {Y(i,j)} produced by the histogram equalization can be expressed as: HE algorithm expands the dynamic range of the histogram, but at the same time, the gray level of the transformed image is reduced, which will cause the loss of some details. For some images with peaks, the processing of HE algorithm can cause unnatural effects of excessive enhancement.

3 The proposed method

3.1 Framework description

Starting from the characteristics of low-light images, the proposed method aims to improve the contrast of low-light images, balance its brightness while retaining its detailed features. In order to obtain an image with good contrast and balanced image brightness, this paper uses the DHEDAPL method based on improved CS algorithm. First, the image histogram is segmented, and then the histogram statistical information and improved CS algorithm are used to determine the limit value of each sub-histogram platform, use the corresponding platform limit to clip each sub-histogram separately, and after obtaining the corrected histogram, perform traditional histogram equalization on the sub-histogram. Here the histogram of the original image is divided into two, there are four platform limits and eight corresponding limit values. The improved CS optimization method is used here, each platform value can be selected within a certain range, and we select the optimal value through the evaluation function. This method is more adaptive than other methods that calculate the platform limit in a fixed way. The image detail texture feature is a kind of global feature, which is different from the color feature. The texture feature is not based on the feature of the pixel, but needs to be statistically calculated in the area containing multiple pixels. An image can use "structure + Texture" to express, but usually extracting the image structure will emphasize the regularity of the texture. We use the method based on the total variation model to extract the main structure of the image. The model does not require texture rules or symmetry, and is general and random. By removing the main structure of the image, we can get the texture that contains all the details of the image. An image can be expressed in the form of "structure + texture", but the regularity of the texture is usually emphasized when extracting the image structure. We use the method based on the total variation model to extract the main structure of the image, which does not require texture rules or symmetrical, it is general and random. By removing the main structure of the image, we can get the texture that contains all the details of the image. Different from the image detail layer extracted by other transformations, the image detail obtained by texture extraction has good noise resistance. Finally, we add the texture to the image processed by DHEDAPL optimized based on improved CS algorithm to obtain the final enhanced image. As shown in Fig 3, it is a main schematic of low-light image enhancement. Next, we will introduce the specific steps of each link in detail.

Fig 3

The flow chart of the image processing method proposed in this article.

3.2 Improved CS algorithm

In the subsection 2.1 and subsection 2.2 of chapter 2, the standard PSO algorithm and CS algorithm have been introduced. At present, these two swarm intelligence optimization algorithms are widely used in various fields and have been verified in the direction of image enhancement, they are all effective optimization methods. However, the individual evaluation of the two algorithms shows certain shortcomings. The most obvious feature of the PSO algorithm is that it is easy to fall into the local optimum. In addition, CS algorithm uses frequent short jumps and occasional long jumps in the optimization process, this optimization method can make it easier to jump out of the local optimum for the algorithm, at the same time, CS algorithm also has the disadvantages of slowing convergence speed and lacking of vitality in later search [38], the reason is that the CS algorithm uses random selection when updating the population, so that the local update can’t quickly find the truly optimal nest position. Inspired by PSO algorithm, it is easier to find the best nest by traversing the population position. Under the framework of the CS algorithm, improved CS algorithm uses the traversal idea of PSO algorithm to design a new optimization search method. This method is easy to jump out of the local optimum and converges quickly. Its algorithm flow is shown in the Fig 4. Then, we conducted a comparative experiment of improved CS algorithm, PSO algorithm, and CS algorithm, the results are shown in Fig 5, It can be seen that the convergence speed of improved CS algorithm is obviously better than that of PSO algorithm and CS algorithm, the particles can still maintain the exploration vitality in the middle and later stage, and the optimization result is also better than that of PSO algorithm and CS algorithm.

Fig 4

Improved CS.

Fig 5

Convergence rate analysis for platform limit parameter optimization using PSO, CS and improved CS.

3.3 Double histogram equalization with double automatic platform

As shown in subsection 3.1, we first perform DHEDAPL processing on the image, in the new equalization method proposed here, the original histogram is first divided to obtain two sub-histograms. Here, four platform limits are used, and each sub-histogram uses two limit values. Different from other methods, the method in this paper is no longer just based on the statistics of the histogram for the selection of the platform limit value, but for each platform limit value according to the histogram statistical information to set the upper and lower bounds, and then use the improved CS algorithm to wandering evaluation within the range, the selection of the platform limit value will be selected within this range adaptively. First, we calculate the average intensity M of the global histogram, and clip the histogram according to the average intensity M (as shown in Fig 6). The calculation formula for M is as follows:

Fig 6

Histogram segmentation.

Where p(k) is the probability of intensity k, its calculation method is shown in Formula (6). After calculating the value of M, the original histogram is divided into obtain two sub-histograms, which are the lower histogram H and the upper histogram H. H contains all the intensity values between [Imin, M], and H contains All intensity values between [M+1, Imax], where Imin represents the minimum intensity value that has appeared at least once in the image, and Imax represents the maximum intensity value that appeared in the image. After segmenting the global histogram, calculating the platform limit L of each sub-histogram is the next task. The classic platform value calculation method is as follows: In the formula, C is a coefficient whose value is between 0 and 1, and Pk is the peak value in the histogram, that is: This article will use the local information obtained from the input histogram to predetermine the range of the platform limit L. Here, the gray scale ratio GC of the sub-histogram is calculated instead of C in the formula, and GC is used as a coefficient to calculate the basic value of each platform limit: Where Pk and Pk are the maximum intensity peaks of the lower sub-histogram and the upper sub-histogram respectively, L and L are the base values of the lower and upper platform for lower sub-histograms, L and L are the base values of the lower and upper platform for upper sub-histogram. The gray scale ratios GC and GC of the lower sub-histogram and the gray scale ratios GC and GC of the upper sub-histogram are defined as: Where M and M are the average intensities of the lower sub-histogram and the upper sub-histogram respectively, λ and λ are the gray scale ratios differences of the lower sub-histogram and the upper sub-histogram respectively. The calculation method is as follows: Where N and N are the total number of pixels in the lower sub-histogram and the upper sub-histogram respectively. At this time, the basic values L, L, L, and L of each platform have been obtained, and finally we set the range of each platform value as follows: Within this range, improved CS is used to optimize the value selection to obtain the final platform limit values , , , and , and then clip the histogram. The application of improved CS in this method will be described in detail in next section. For the value of the lower sub-histogram (Imin The process of modifying the histogram is shown in Fig 7. After the histogram is modified, the sub-histogram can be independently equalized according to Formula (9) to obtain an enhanced image. This process can be expressed by the following formula:

Fig 7

Histogram clipping.

Where I represents the original input image, Γ represents the process of dual-histogram dual-adaptive platform processing, and I represents the output image after our enhancement processing.

3.4 Optimization of platform value with improved CS

As mentioned in the subsection 3.3, the method to obtain the basic range of each platform value has been determined. After obtaining the range, we use improved CS to explore the space range and judge the quality of the platform value through the evaluation function, the platform value optimization process can be divided into the following steps: Step 1: Initialize the number of nests N and dimension D, initialize the nest position X within the basic range of the platform, find probability Pa, the maximum number of iterations T and the fitness value set fitness; Step 2: Traverse the position X(i,:) in the population, use the parameters of the corresponding position as the platform value, perform DHEDAPL enhancement on the image, evaluate the corresponding enhanced image using the evaluation function, store the corresponding fitness value fitness, compare it to get the largest fitness value fitnessmax and save the position of the nest as X(max,:); Step 3: Update the population through Lévy flight to obtain a new population X’, traverse the population position X’(i,:), repeat the operation of step 2 to obtain the corresponding image enhancement result and fitness value , as well as the corresponding maximum fitness value and its position X’(max,:); Step 4: Determine whether to perform population replacement. If , replace the original population with the population updated by Lévy flight, and update the maximum fitness value fitnessmax and position X(max,:) at the same time; Step 5: Determine whether the cuckoo egg is found, if it is found, update the population by Formulas (1) (2), if it is not found, keep the nest position. Step 6: Judge whether the iteration stop condition is reached, if not, then jump to Step2, if the iteration stop condition is met, exit the loop. The last retained position with the maximum fitness value carries the parameters as the final DHEDAPL corresponding platform value. Where the fitness function, as a criterion for judging the quality of the location, is the key to the effectiveness of the final algorithm. The design of the fitness value in this paper takes into account the global and local information of the image. The specific form is as follows: Where I represents the image enhanced with the parameters of the corresponding position; I is the sobel edge image of I; sum(I) is the sum of the intensity values of all pixels of I; n_edge(I) is the number of such edge pixels with an intensity value greater than a certain fixed intensity value; M×N is the size of I, that is, the number of I pixels; H(I) is the entropy value of image I, the larger the value, the more information the image contains, its calculation method is as Formula (48); MSE(I) is the image mean square error, the smaller the value indicates that the degree of image distortion is smaller, its calculation method is as Formula (33). Where g(i,j) is the gray value of the pixels in row i and j column of the input image; h(i,j) is the gray value of the pixels in row i and column j of the enhanced image.

3.5 Image texture extraction

As shown in the framework flow of subsection 3.1, another direction is to extract the texture of the image. An image can usually be decomposed into a "structure-texture" form, it can be shown by Formula (34), reference [39, 40] extracts the main structure of the image by means of total variation regularization, namely Formula (35). Where I is the input, which can be the brightness channel; x is the image pixel index; I is the extracted structure image; data item (I−I)2 is to make the extracted structure similar to the structure in the input image; β is a weight value; is the total variational regularizer, written as: In the two-dimensional anisotropy expression, ∂ and ∂ are partial derivatives in two directions. Through experiments, it is found that this kind of total variation regularizer has limited ability in distinguishing strong structural edges and textures. For different images in the natural world, the texture characteristics are usually not fixed and without certain rules. In order to further highlight the texture and structural elements, a general pixel-level windowed total change metric D(x) and a new window inherent change metric L(x) are introduced here.to form a new regularizer, such as Formula (37): Where y∈R(x), R(x) is rectangular areas centered on pixel x; D(x) and D(x) are the total window changes of pixel x in the directions of h and v; L(x) and L(x) are a new kind of inherent variation window. They are different from D(x) and D(x) in that they do not contain modulus, only the position of the absolute variation changes. The absolute variation from the gradient becomes the total absolute variation. Because the gradient variation has positive and negative values, the inherent variation of this window is almost zero in the undulating and uniform background area, and the variation is very large only at relatively large edges, and they pay more attention to the overall spatial change of the image; ε is a small positive number used to avoid the denominator being 0; g is a weighting function defined according to the spatial affinity, expressed as g is the gaussian kernel function, where σ is the space ratio for controlling window, which also affects the smoothness of the image to a certain extent. In summary, the new structure extraction method is as follows: The difference between Formula (43) and Formula (35) lies in the regularizer. The regularizer of Formula (43) is Formula (37), which adds a new kind of inherent variation window (40) and (41). The function of is the same as before, β is a weight, it is also used to control the smoothness of the image. By adjusting the values of β and σ, the separation amount of image texture can be controlled to a certain extent. Generally, the value range of β is (0, 0.05), and the value range of σ is (0, 6). Through experiments, we find that when β slowly increases, the amount of texture separation of the image also increases, the texture separation is particularly obvious when β is set to 0.02; Similarly, when σ increases, the amount of texture separation will also increase, the separation of texture and structure is relatively clean when σ is set to 3. Considering the two parameters, and as shown in Table 1, we finally decide β = 0.015 and σ = 3. The effect is shown in Fig 8.

Table 1

Index values of images processed with different parameters.

	Input	a	b	c	d	e	f	g	h	i	j	k
PSNR	Inf	36.240	34.760	34.061	33.378	33.355	36.240	35.684	34.882	34.264	33.738	32.826
SSIM	1.000	0.977	0.972	0.970	0.968	0.967	0.977	0.975	0.971	0.969	0.968	0.967
Entropy	7.462	7.426	7.415	7.415	7.408	7.412	7.426	7.414	7.412	7.415	7.420	7.402

Fig 8

Parameter selection experiment.

Finally, when the structure I is extracted, we can get the texture I of the image, namely:

3.6 Image texture addition

At the end of the enhancement framework process in subsection 3.1, we enhanced the image with DHEDAPL optimized based on improved CS to obtain an image I with suitable brightness and good contrast, but its disadvantage is that the image details are not obvious enough. While we strip out the detail information I in the original image through the new total variational regularization method described in subsection 3.5. At this time, we can add I to I, which can enrich the details of the image, and this operation has little effect on the brightness characteristics and contrast characteristics of I at the same time. As shown in Fig 9, the effects of performing separate DHEDAPL processing on an image and adding texture details of the original image after DHEDAPL processing are compared, it can be seen that after the original image is processed by DHEDAPL, the brightness and contrast are improved significantly, but the image details are smoother. On this basis, the original image texture details are added, the overall visual effect of the image is good, and the brightness contrast is suitable for human eyes. View the enlarged details of the image, the image with texture details I contains more details and the texture is clearer. Therefore, the final output image of the method proposed in this paper is represented by the following definition:

Fig 9

Image texture adding effect comparison.

(a) Original image (b) Image processed by DHEDAPL (c) The image processed by DHEDAPL and adding detailed texture.

Where I is the final output image we obtained, I is the image obtained after the original image is processed by DHEDAPL, and I is the texture details of the original image.

Image texture adding effect comparison.

(a) Original image (b) Image processed by DHEDAPL (c) The image processed by DHEDAPL and adding detailed texture.

4 Experiments

In this section, we shown the experimental results of the proposed method for low-light image. All the codes in this method are run in matlabR2019a, and all the experiments are performed on a PC with operating system windows10, 4GB memory and 3.20GHz i5 CPU. The test images selected in this article are all low-light images, which can be divided into three categories. Due to the limited space of the article, the following are selected as the representatives for display. The test images selected in this article are all low-light images, which can be divided into three groups: (1) 20 random images from MIT-Adobe 5K data set [41]; (2) The first 100 images from MIT-Adobe 5K data set; (3) 64 images from DICM data set [42]. The first two groups of images are paired images, and the third group of images are unpaired images. These images all have the characteristics of low brightness, unclear image details and low contrast. In this chapter, we first verify the anti-noise performance of the new total variational model in making image texture mask, and compare it with the unsharp mask method, which is the classical method of making mask [43]. In addition, we have compared the proposed method with other mainstream algorithms. The contrast algorithm includes the adaptive gamma algorithm AGCWD in [44], Al-Ameen proposed the new image illumination enhancement method [45], dual-platform limit-based double histogram equalization (BHE2PL) [33], the biologically inspired multi-exposure fusion framework method (BIMEF) [46], brightness-maintained dynamic histogram equalization (BPDHE) [11], a simple yet effective low-light image enhancement (LIME) method [22], multi-scale retinex with color restoration algorithm (MSRCR) [21], the nature preservation enhancement algorithm (NPEA) in [47] and the simultaneous reflectance and illumination estimation method (SRIE) in [48]. Therefore, there are three sections in this chapter: (1) Anti-noise performance analysis of texture mask made by the new total variational model; (2) Subjective evaluation of the experimental results; (3) Objective evaluation of the experimental results of the proposed method.

4.1 Anti-noise performance analysis

In chapter 3, we have introduced the method in detail, in order to enrich the texture details of the image processed by DHEDAPL, we try to use a new total variational model to extract the image texture information and avoid noise as much as possible. In this section, we verify it through experiments, and analyze the image information contained in the image before and after adding texture information. At the same time, we also compare it with the unsharp mask, which is the classic method [43]. As shown in the Fig 10, we show the changes of the two images before and after adding texture information. Although the images processed by DHEDAPL have been satisfactorily evaluated by human eyes in terms of contrast and brightness, the image details are not clear enough and there are few image details; The contrast of the image with unsharp mask is improved, but not too much image detail is added; After the texture details extracted by the new total variational model are added to the DHEDAPL processed image, we can obviously feel the richness of details, such as several details shown in the figure, there are bees and stamens.

Fig 10

Image texture adding effect comparison.

(a) Image processed by DHEDAPL (b) Image processed by DHEDAPL adding unsharp mask (c) Image processed by DHEDAPL adding texture mask.

(a) Image processed by DHEDAPL (b) Image processed by DHEDAPL adding unsharp mask (c) Image processed by DHEDAPL adding texture mask. In addition, we conducted comparative experiments on 20 randomly selected images in MIT-5K data set. From the data results shown in the Table 1, there are the average values of the four evaluation values of 20 images before and after adding texture information, which are peak signal to noise ratio (PSNR), structural similarity (SSIM), average gradient (AG) and natural image quality evaluator (NIQE) respectively. PSNR and SSIM are full reference metrics. PSNR is the ratio of the energy of the image peak signal to the average energy of noise, which can be used to evaluate the image noise content, the larger the value, the less the image noise and the better the image quality; SSIM is an index to measure the similarity between two images, its value range is (0,1), the closer the value is to 1, the more complete the image structure is and the better the image quality is; AG and NIQE are non-reference metrics, in which the value of AG reflects the gradient detail contained in the image, the larger the value, the more image detail information; The blind image quality evaluation index NIQE is a relatively comprehensive image evaluation index, the smaller its value, the better the image visual feeling. It can be seen from the data in the Table 2, the AG value of the image with unsharp mask and the image with texture mask has been improved compared with the previous images, especially the image with texture mask; But the probability of noise increased by adding unsharp mask is very high. From the average value of PSNR of 20 images, it can be seen that the PSNR value of the image added with unsharp mask decreases, but the PSNR of the image does not decrease after adding the texture information extracted by the new total variational model, even the PSNR value is improved; For the SSIM index, there also has the same evaluation. Moreover, for the NIQE index, the image with texture information extracted by the new total variational model has also received the best evaluation. Therefore, it is enough to show that our model not only increases the image texture information, but also does not introduce too much noise to destroy the image quality and structure, and achieves the purpose of enriching the image details.

Table 2

Average value of four evaluation indexes of 20 random images of MIT-5K.

	DHEDAPL	Unsharp mask	Ours
PSNR	22.2326	21.6499	22.4175
SSIM	0.8301	0.8181	0.8313
AG	6.3646	6.7872	7.2978
NIQE	2.9636	2.9837	2.8562

4.2 Subjective evaluation

The visual perception of an image is one aspect of evaluating image quality, and people’s perception of an image includes various aspects such as color, brightness, and clarity. Different algorithms have different degrees of improvement in various aspects of low-light images, as shown in Figs 11–30, which are the effect of each test image processed by various methods, from these experimental results, our method as a whole has the greatest improvement in image visual effects compared to other algorithms.