Improved L0 Gradient Minimization with L1 Fidelity for Image Smoothing.

Edge-preserving image smoothing is one of the fundamental tasks in the field of computer graphics and computer vision. Recently, L0 gradient minimization (LGM) has been proposed for this purpose. In contrast to the total variation (TV) model which employs the L1 norm of the image gradient, the LGM model adopts the L0 norm and yields much better results for the piecewise constant image. However, as an improvement of the total variation (TV) model, the LGM model also suffers, even more seriously, from the staircasing effect and is not robust to noise. In order to overcome these drawbacks, in this paper, we propose an improvement of the LGM model by prefiltering the image gradient and employing the L1 fidelity. The proposed improved LGM (ILGM) behaves robustly to noise and overcomes the staircasing artifact effectively. Experimental results show that the ILGM is promising as compared with the existing methods.

Introduction

Image smoothing aims at removing the insignificant details and preserving salient structure such as edges, there are many applications of image smoothing in computer graphics and image processing. However, there is always a dilemma for the smoothing algorithms to simultaneously remove details and preserve edges. For example, the linear Gaussian filter can smooth images effectively but blur the edges seriously. The objective of almost all methods focuses on how to get a tradeoff between flattening details and preserving sharp edges and there have been a plenty of works devoted to this task. Bilateral filter is one of such works and it is simple and effective in removing noise-like structures[1]. Soon after its debut in 1998, there has been a flurry of extensions such as [2-6] and its accelerated variants [7-9]. A comprehensive review of the bilateral filter was presented in [10]. Anisotropic diffusion [11] is another category of filter aiming at suppressing noise while preventing salient edges. Since its introduction, there has been a great deal of researches devoted to the theoretical and practical understanding of this model for image smoothing [12-20]. For example, Black et al. proposed the robust anisotropic diffusion model based on robust statistics [12], Weickert proposed the structure tensor model based diffusion [13][14], Sochen et al. proposed the manifold model based diffusion [15], and modified Perona-Malik model based on directional Laplacian was proposed in [16]. The fourth-order partial differential equation models were also proposed, such as [17-20]. Total variation (TV) takes the form of an unconstrained optimization model [21], where the desired image is obtained as the minimizer to a certain functional that contains both regularization and fidelity terms. The TV model employs the L1 norm of the image gradient as the regularization term and behaves more stably than the anisotropic diffusion [11]. Therefore, the TV model was studied and applied widely for image smoothing [22-30]. However, the TV-based approach suffers from staircasing effect. There are also other works aiming at image smoothing, such as the local extremes model by Subr et al.[31], the guided filter model by He et al.[32] and the weighted least squares method [33].

Recently, Xu et al. proposed an improvement of the TV model by replacing the L1 norm of the image gradient with the L0 norm, i.e., the L0 gradient minimization (LGM)[34]. The LGM model performs more effectively than the TV model for piecewise constant images, however, just as the TV model, the LGM model also suffers, even more seriously, from the staircasing effect. Since the LGM model counts the number of non-zero gradients in the result, it is not robust to noise. In order to overcome these drawbacks of the LGM model, an improved LGM (ILGM) mode is proposed by prefiltering the image gradient during the iteration in the LGM model. The L1 fidelity is also adopted since it is more robust than the L2 one when erroneous measurements exist [35][36]. Similar to LGM [34], the alternating minimization (AM) algorithm [37] is employed for the ILGM model by introducing auxiliary variables; the AM algorithm also yields global optimal result for the ILGM model. Experiment and comparison show that the ILGM model outperforms the LGM model on staircase suppression and noise robustness.

The rest of paper is organized as follows. Section 2 reviews the L0 gradient minimization (LGM), Section 3 introduces the improved L0 gradient minimization (ILGM) with L1 fidelity. In section 4, some experimental results are reported and analyzed. Finally, we conclude our work in Section 5.

LGM: L0 Gradient Minimization

The LGM model [34] employs an L0 penalty term to directly measure gradient sparsity and has a strong ability to preserve edges. Let f be the input image and U the smoothed one, the gradient of image U at pixel p is denoted by ∇U = (∂ U ,∂ U ). L0 gradient measure is expressed as: Where #{} denotes the number of pixels of which the gradient is not zero. The term E(U) serves as the regularization term and is combined with a general data constraint term that makes the result U be structurally similar to the input image f. The energy function of the LGM is defined as follows: where λ is a non-negative parameter directly controlling the weight of the regularization term. However, this regularization term is difficult to optimize because the value of E(U) may range from tens to thousands and it is non-convex and non-derivative. To solve the problem, the AM algorithm [37] is employed and two auxiliary variables h and v are introduced to replace the ∂ U and ∂ U respectively. The specific objective function is expressed as

Where ∂ U and ∂ U are replaced by h and v respectively in E(U), and β is a parameter to control the similarity between the auxiliary variables and their corresponding gradients.

Keeping in mind that the energy function for the LGM model is reformulated as Eq (3) by introducing two auxiliary variables h and v. h and v are then estimated from the image gradient (∂ U,∂ U) by exerting an L0 norm constraint. However, when there is false gradient in the image caused by noise or inhomogeneity, h and v will deviate from the correct values. Based on this consideration, we propose to pre-filter the image gradient (∂ U,∂ U) by an edge-preserving filter when estimating h and v. Since the L1 fidelity is more robust to outliers than the L2 one, we also adopt the L1 fidelity. As a result, the proposed ILGM model is formulated as follows,

Where the (h , v ) is estimated from an optimized version (M,N) of the image gradient (∂ U,∂ U) when solving Eq (4). The LGM model is employed to calculate (M,N) from (∂ U,∂ U) as follows,

In order to handle the L1 fidelity, the variable W is introduced to denote the difference between U and f. Then, the functional in Eq (4) is rewritten as follows,

The numerical implementation of Eqs (5) and (6) is identical to the LGM model, so the alternating minimization algorithm [37] is employed by introducing other auxiliary variables to approximate ∇M and ∇N respectively. As for Eq (7), it will be split into three subproblems to alternatively minimize W, (h ,v ) and U by using the AM algorithm[37], which lead to an global optimization procedure.

Subproblem 1: Fixing U, h , v , solve W

With U, h , v fixed, we rewrite problem (7) as

Through the matrix calculus, we can get the unique solver, and the solution can be expressed as, where the convention 0·(0/0) = 0 is followed, and (U − f )/|U − f | denotes the sign function. α is the parameter and set to 0.1 in our experiments.

Subproblem 2: Fixing U and W, solve (h ,v )

Given U and W, we first estimate M and N from (∂ U,∂ U) using Eqs (5) and (6) respectively. Then, the objective function for (h ,v ) reads where E(h , v ) denotes the number of non-zero elements in |h | + |v |. This problem is similar to Subproblem 2 in the numerical implementation of LGM model[34], and the strategy there is employed to obtain (h ,v ). After some manipulation, the estimate of (h , v ) is given by

In this step, we compute for each pixel p the minimum energy value.

Subproblem 3: Fixing W, h and v , solve U

Given h , v , W, the objective function for U reads

Eq (12) is quadratic and thus has a global minimum even by gradient decent. Alternatively, it can be efficiently solved using Fast Fourier transform (FFT), which yields solution where F(·) is the FFT operator and F(·)* denotes the complex conjugate.

The whole algorithm is summarized in Algorithm 1. Parameter β is automatically adapted in iterations starting from a small value β 0, it is multiplied by κ each time.

Algorithm 1

ILGM: Improved L0 Gradient Minimization.

Input: image f, λ, β max, α, and κ

Initialization: U = f, the current parameter β 0 = λ, α = 0.1, κ = 2, β max = 1.0e5, i = 0

While β < β max , do

With U (i), solve for W (i) in Eq (9)

With U (i), solve for M (i) and N (i) in Eqs (5) and (6)

With M (i) and N (i), solve for hM(i), vN(i) in Eq (11)

With hM(i), vN(i) and W (i), solve for U (i+1) in Eq (13)

Update the parameter β = κ ⋅ β, i++

End

Output: The smoothed image U

Experimental Results

In this section, we will demonstrate the performance of the proposed method and make a comparison with several state-of-the-art methods including Bilateral filtering(BLF) [1], Weighted least square method(WLS)[33] (http://www.cs.huji.ac.il/~danix/epd/wlsFilter.m), Total variation (TV)[21] (The minimization of TV is solved using Chambolle's method in Ref.[29], when a color image is encountered, a generalization of Chambolle's method in Ref.[30] is employed.), and the LGM model [34]. Several natural images are adopted as test images, and we focus on the performance of noise robustness, edge preservation, and staircasing effect. To get the best results, the parameters for these methods are manually tuned and presented in the figure captions.

The noisy color image coined by Farbman et al. [33] is employed to test the noise robustness of the proposed ILGM model. The results are presented in Fig 1. Since a prefiltering process is employed for the ILGM, the λ is smaller than that for the original LGM model. However, the proposed ILGM model is not sensitive to λ. For example, Fig 1F, 1G and 1H are the results with different λ, which are almost indistinguishable. Since the numerical implementation or parameters of the associated methods are not presented in [34], the results in Fig 1 are slightly different from those in [34], however, similar results are reported in [36]. Fig 2 shows another example. In this example, the ILGM method also yields a much better result. These observations demonstrate the noise robustness of ILGM method.

Fig 1

Comparison of the ILGM method with the BLF,WLS, TV and LGM methods on a synthetic image.

(a) Noisy image, (b) BLF (σ = 12, σ = 0.5), (c) WLS (λ = 2, α = 3), (d) TV (λ = 3), (e) LGM (λ = 0.3), (f) ILGM (λ = 0.004), (g) ILGM (λ = 0.005) and (h) ILGM (λ = 0.006). Images in 2nd and 4th row are the local close-ups of those in 1st and 3rd row.

Fig 2

Smoothing results of a natural image using the LGM and ILGM.

(a) A natural image, results of (b) LGM (λ = 0.3), and (c) ILGM (λ = 0.005). (d)-(f) are the close-ups of (a)-(c) marked by blue rectangle in (a).

Fig 3 shows a flower image. In this example, we aim at smoothing the texture in the leaves and flower petals, but preserving the sharp edge between the leaf and the petal. The results of the BLF, WLS, TV, LGM, and ILGM models are presented. One can see from Fig 3 that there is still texture in the results of the BLF, WLS and TV models, and the results of the LGM and ILGM models are approximately piecewise constant. However, from the line profiles shown in Fig 3G, it is clear that there are still oscillations in the results of the LGM model. In contrast, the result of the ILGM model is much nearer to piecewise constant but the sharp edges are also preserved.

Fig 3

Results of image smoothing on a flower image.

(a) flower image, results by (b) BLF(σ = 4, σ = 0.2), (c) WLS(λ = 0.4, α = 1.4), (d) TV(λ = 3), (e) LGM(λ = 0.02), and (f) ILGM(λ = 0.0006). (g) Line profiles of the R channel of the 291st column (blue line in (a)).

Figs 4 and 5 show another two examples to demonstrate the edge preservation performance of the associated methods. It is clear from Fig 4 that there is still texture in the results of the BLF, WLS and TV methods, even though the edges in (c) and (d) are somewhat blurred. The LGM model smoothed out the texture clearly, but the edges are merged, see Fig 4E. On the other hand, the ILGM method preserved the sharp edges very well while smoothing out the texture clearly. Fig 5A shows an image of texture. In this image, we also aim at smoothing out the texture while preserving the edges. We first smoothed the image using LGM and ILGM respectively, then detected the edges in the results. The smoothed images and edge detection results are shown in Fig 5. One can conclude that the ILGM model performs much better than the LGM model from this example.

Fig 4

Edge-preserving experiment on a wood image.

(a) The original image, results by (b) BLF (σ = 4, σ = 0.2), (c) WLS (λ = 0.8, α = 2), (d) TV(λ = 4), (e) LGM (λ = 0.3), and (f) ILGM(λ = 0.0025).

Fig 5

Comparison of edge-preserving performance on a natural image.

(a) The natural image, results by (b) LGM (λ = 0.35), and (c) ILGM(λ = 0.0035). (d)-(f) are the edge detection results of (a)-(c), respectively.

It is well-known that the TV model suffers from the staircasing effect [21][23]. Although the LGM model replaces the L1 norm in the TV model with the L0 norm, it also suffers from this notorious effect [34]. However, the proposed ILGM model can conquer this staircasing effect effectively. Fig 6 shows an example of this case. In this example, the wood texture image suffers from inhomogeneity. When smoothing the texture, the LGM model yields staircases due to intensity inhomogeneity, see (b) and the close-up (e). However, the proposed ILGM model yields a much smoother result, see (c) and the close-up (f), and the wood texture is also removed. This example manifests that the ILGM model outperforms the LGM model on staircase suppression.

Fig 6

Comparison of the staircasing effect of the LGM and ILGM models.

(a) The wood image, results by (b) LGM (λ = 0.007), and (c) ILGM (λ = 0.0002). (d)-(f) are the close-ups of (a)-(c),respectively.

Conclusion

Recently, the L0 gradient minimization (LGM) is proposed for image smoothing and performs effectively for piecewise constant images. However, it suffers from staircasing effect and is not robust to noise. In this paper, we presented an improved LGM (ILGM) model by prefiltering the image gradient and employing L1 fidelity. The proposed ILGM model behaves robustly to noise and overcomes the staircase artifact effectively. Experimental results and comparison with several state-of-the-art methods show that the ILGM model is promising. In the future, we will expand our method to find more applications.