Sparse Adaptive Iteratively-Weighted Thresholding Algorithm (SAITA) for Lp-Regularization Using the Multiple Sub-Dictionary Representation.

Both L1/2 and L2/3 are two typical non-convex regularizations of Lp (0<p<1), which can be employed to obtain a sparser solution than the L₁ regularization. Recently, the multiple-state sparse transformation strategy has been developed to exploit the sparsity in L₁ regularization for sparse signal recovery, which combines the iterative reweighted algorithms. To further exploit the sparse structure of signal and image, this paper adopts multiple dictionary sparse transform strategies for the two typical cases p∈{1/2, 2/3} based on an iterative Lp thresholding algorithm and then proposes a sparse adaptive iterative-weighted Lp thresholding algorithm (SAITA). Moreover, a simple yet effective regularization parameter is proposed to weight each sub-dictionary-based Lp regularizer. Simulation results have shown that the proposed SAITA not only performs better than the corresponding L₁ algorithms but can also obtain a better recovery performance and achieve faster convergence than the conventional single-dictionary sparse transform-based Lp case. Moreover, we conduct some applications about sparse image recovery and obtain good results by comparison with relative work.

1. Introduction

Compressed sensing (CS) [1,2] and sparse representation [3,4] have been widely used in the field of wireless communications [5,6,7] and image processing [8,9,10]. CS implies that it is possible to reconstruct the sparse signal/image from incomplete data if some prior knowledge and reconstruction constraints are satisfied. Mathematically, the unconstrained minimization is the optimal model to obtain the sparsest solution : where denotes the zero-norm function to find the number of nonzero elements in ; denotes the down-sampling measurement matrix; and represent the observed vector and the unknown sparse image, respectively; is the regularization parameter to balance between the fidelity of the image and the sparsity; and is a small positive constant, e.g., . Unfortunately, this problem (1) is an NP (non-deterministic) problem, and thus, it is difficult to efficiently solve. When the matrix satisfies some necessary conditions [11], an alternative convex relaxation method are developed using the regularization method as: where denotes the -norm. Then, the NP problem (1) is converted into problem (2), which is a typical a convex optimization problem and can be solved efficiently, such as with the alternating direction method of multipliers (ADMM) [12,13], fast iterative shrinkage-thresholding algorithm (FISTA) [14], Nesterov’s algorithm (NESTA) [15], and approximate message passing (AMP) [16]. However, the method of regularization can only obtain a suboptimal solution and usually requires much more measurements. Theoretical analysis of CS implies that better performance can be obtained by taking advantage of sparser information in many systems, especially in the presence of strong noise interference.

1.1. The Non-Convex Penalties

Many state-of-the-art algorithms have been proposed to improve the performance of the regularization algorithms. The non-convex penalty regularization algorithms are among the most effective algorithms for sparse recovery problems. Research shows that the non-convex penalty-based optimization methods can more closely approximate the sparsest solution over the -norm penalty in problem (2), which requires a weaker incoherent condition and fewer measurement data [17]. There have been many non-convex functions proposed as relaxations of the -norm penalty, such as the smoothly clipped absolute deviation penalty (SCAD) [18], the -norm penalty [17] and the minimax concave penalty (MCP) [19]. By replacing the -norm with the -norm, the non-convex -norm regularization optimization method is described as: where . Unfortunately, when , problem (3) becomes a non-convex, non-smooth, and non-Lipschitz optimization problem. Thus, the -norm optimization is always difficult to efficiently address.

1.2. The Iterative Thresholding Algorithm of Regularization

There are two main classes of algorithms to solve the non-convex -norm optimization problem. One is the iteratively reweighted algorithm [20], and the other is the iterative thresholding algorithm (ITA). As one of the most effective and efficient methods, the ITA has been employed for many sparse recovery optimization problems due to its low computational complexities, including the iterative hard thresholding for regularization [21], the iterative soft thresholding for regularization [22] and the iterative thresholding for regularization [23]. and regularizations are two special and important cases, not only their solutions can be expressed in closed-forms, but also their importance on sparse modeling. Recent studies show that regularizer can be taken as the most representative regularizer [24] and regularization is more effective in image deconvolution problem [25]. Hence, in this paper, we focus on the and regularizations, which is described in Equation (4) and (5):

1.3. New Multiple-State Sparse Transform Based Regularization Algorithm

Recently, some new multiple-state sparse transform based algorithms were proposed to exploit more a priori knowledge of the signal/image by employing some new sparsifying transform strategies. A shearlet-based multiple level sparse representation algorithmic framework was proposed in [26,27] for the unconstrained regularization by adaptively incorporating the iteratively reweighted shrinkage step. To enhance the sparsity, the algorithm [27] is specifically adapted to the sparse structure of the multiscale coefficients based on the ADMM [12,13]. Similarly, considering the fact that the sparsity of a certain signal/image will change under different sparsifying transform dictionaries, a sparsity-induced composite regularization algorithm was proposed for the unconstrained regularization problem (Co-L1) [28]. The novel Co-L1 method is described as: where are different dictionaries, denotes the number of , and represents the dictionary size. The regularization parameters play the roles of weighting the -norm of the sparsifying transform coefficients . Multiplying by the weighting parameter , the regularizer is indeed a composition of multiple dictionary based regularizers. The algorithm [28] can significantly improve the image reconstruction performance over the fast iterative shrinkage-thresholding algorithm (FISTA) [29] by iteratively and adaptively weighting the composite regularizer. We define these new emerged sparsifying transforms as the “multiple-state” transform. The common property of these methods is how to exploit the prior information from the multiple-state sparsifying transform to improve the conditioning of the sparse recovery problems.

In this paper, benefiting from the improvement of the regularization algorithm [24,25,30], and motivated by recent advances in the iterative reweighted algorithms, we propose a new iteratively-weighted algorithm framework for , norm minimization using the multiple-state sparsifying transform, i.e., multiple sub-dictionary sparse representation [28]. The contributions of this paper are summarized as follows. (1) Based on the multiple sub-dictionary sparse representation, we develop a new iteratively-weighted thresholding algorithm, which is called as SAITA-. (2) By comparison with the existing iteratively-reweighted parameter scheme, we propose an updating regularization parameter for weighting the sub-dictionary. (3) regularization and regularization are special and important on sparse modeling, particularly on sparse recovery. However, related studies are rare, this paper also extend the applications to sparse image recovery and Magnetic resonance imaging (MRI) and get good results.

The organization of the rest of the paper is as follows: in Section 2, we propose the multiple sub-dictionary -regularization in the SAITA- algorithm, including the multiple sub-dictionary sparsifying transforms and the iterative reweighted scheme for the SAITA- regularizer. Then, in Section 3, we develop a new norm minimization, iteratively thresholding algorithm SAITA-. To confirm the proposed algorithm, we conduct simulations and applications in image restorations in Section 4. In Section 5, we further validate our proposed algorithm using three applications. Finally, conclusions are given in Section 6.

2. The Proposed Multiple Sub-Dictionary-Based Regularization

The multiple dictionary sparsifying transform method for the regularization optimization problem was proposed in [28], which extends the well-known Lasso problem into a composite regularization problem. Motivated by the composite regularization method for the -norm, this paper employs the multiple sub-dictionary method for the regularization problem.

Suppose is the non-sparse, raw signal. We can obtain the sparse coefficients through an analysis dictionary . The shearlet transform [31] and the wavelet transform [32] are two typical sparsifying transforms. We choose the wavelet transform as the ideal transform because of its effectiveness to compress natural images. The main steps to design the multiple sub-dictionary sparsifying transform based regularization method are:

First, we construct an over-complete sparsifying transform matrix by: and: where denotes the number of sub-dictionaries in the over-complete sparsifying transform matrix , the sub-dictionary matrix is acquired by a collection of row vectors , such as the “dbN” wavelet basis [33], and represents the number of column in . From Equation (8) we can see that each is composed of a set of rows from the over-complete sparsifying transform matrix , and:

By splitting the matrix into several sub-dictionaries , we convert the sparsifying transform to a composition of several with different sparse structures. Intuitively, we can utilize these differences to improve the recovery performance in sparse recovery problems. In this paper, we choose , so , which are orthogonal matrixes.

Then, a new multiple non-convex regularization method is proposed: where is a linearly weighted combination of multiple sub-dictionary based regularizers : the denotes the iterative weighted regularization parameter. Hence, the contribution of each sub-dictionary is controlled adaptively and iteratively with the weighted parameter , and the regularizer will vary across the sub-dictionary index . Intuitively, the variation of each sub-dictionary based regularizer is best weighted by the parameter to improve the sparse recovery problem.

3. The Proposed SAITA- Algorithm

The major disadvantage of the minimization is that it is nonconvex, making it difficult to efficiently solve. In this section, we first introduce the iteratively thresholding representation theory for the conventional algorithm according to the existing series of algorithms in [25,34]. Then, we deduce the SAITA- algorithm combined with the proposed weighted scheme .

3.1. The Relationship of the SAITA- and Conventional Methods

Considering the multiple sub-dictionary regularization problem in Equation (10), when , we have: where the denotes the objective functions. Correspondingly, the conventional single dictionary based analysis -norm minimization problem is as follows:

The proposed SAITA- methods of (12) and the conventional method (13) are nearly identical, and the major difference is how to weight the contribution of the -norm by the regularization parameter [28]. Compared with the conventional method, the SAITA method can exploit more prior knowledge of the sparse signal/image for reconstruction. Figure 1 depicts the relationship between the two methods. In the case of (A), the number of sub-dictionaries is reduced to 1, and the multiple dictionaries . convert into a single . Then, the proposed SAITA- algorithm converts to the conventional single dictionary method [24,25]. In case (B), with the increase of the number , the conventional single dictionary method converts to the proposed SAITA- method.

Figure 1

The relationship between the conventional single dictionary based thresholding method and the proposed SAITA method.

3.2. The Thresholding Representation Theory for SAITA-Lp

According to the relationship between the proposed SAITA- method and the conventional method shown in Figure 1, we first consider the conventional single dictionary analysis problem (13). The first order optimality condition of is described as: in which the operator denotes the gradient. Letting , we have:

Multiplying by any parameter to control the step size and adding in both sides of Equation (15):

Then, we can immediately obtain:

That is:

To this end, when , the resolvent operator [24,25,30] is denoted as: where and are the regularization parameter and the step tunning parameter, respectively. Then: where (e.g., , or ) controls the step size in each iteration.

According to the Equation (20), we immediately imply: in which: where represents the -th iterative solution. The resolvent operator is defined as: where the is the nonlinear function defined by: when ; is the threshold value, and [24]: when ; is the related threshold value, and [25]: where denotes the sign function and

3.3. The Proposed SAITA-Lp Algorithm

As mentioned above, the iteratively weighted parameter plays a key role during the optimization process for the sparse recovery problem. For the proposed SAITA-L method, the iteratively weighted parameter mainly plays two roles. One is the role of controlling the tradeoff of the fidelity and the prior knowledge between the quadratic term and the regularizer, and the other role is controlling the contribution of each regularizer. Unfortunately, it is not clear how to do this because setting an ideal parameter is not straightforward. In [28], a iteratively updating parameter was reduced by applying a Maximization-Minimization algorithm shown as: where controls the sub-dictionary size, is a small constant which prevent the denominator form zero. From Equation (28) we can obtain some useful information about setting a proper regularization parameter. Firstly, the contribution of denominator in Equation (28) is to counterweigh each sub-dictionary based regularizer; Secondly, the numerator control the size of each sub-dictionary. Inspired by the above insights, in this paper, we design the important iteratively weighted parameter as: where controls the sub-dictionary size as showed in [28], is a small constant to tune it that is determined from the following experimental results. Then the parameter plays the role of weighting each -norm regularizer adaptively.

The following are the analytical justifications. (i) When signal is sparser under any dictionary of than others, the value of each regularizer is smaller. Hence, the dictionary of is more appropriate, and on the other hand, a smaller regularizer will be beneficial to minimizing the objective function. Thus, the weight of the regularizer should be enhanced. (ii) When the signal may not be sparse enough under another dictionary , that is, the dictionary of is not ideal, the value of will be larger. The larger will not be helpful to minimizing the objective function; thus the weight of the should be smaller to counterweigh the regularizer.

For the main comparison, the conventional single-dictionary based -regularization method in problem (13) will be considered, and the tradeoff parameter is a fixed constant, which is shown as:

From Equation (30), we find that the conventional single-dictionary based parameter is a constant and will not vary.

Moreover, we employ the forward-backward linear strategy to accelerate the convergence of the proposed algorithm as [14]: and:

The proposed iteratively-weighted SAITA- algorithm can be described in Algorithm 1.

4. Performance Analysis and Discussion

We first conduct some experiments to determine the value of and verify the performance of the proposed , then we evaluate the superiority of the proposed SAITA algorithm compared with the conventional single dictionary analysis iterative thresholding algorithms [24,25] and the Co-L1 [28]. All the experiments in this paper were conducted on a personal computer (2.21 GHz, 16 GB RAM) in a MATLAB (R2014a) platform.

Assuming the clean image , we construct a measurement matrix using the “Spread Spectrum” operator [35], so the measurement image shows: , where n denotes the additive noise. Since the wavelet is known to compress natural raw images very efficiently, we choose the wavelet transform as the sparsifying transform operator. Then, we construct the sparsifying transform matrix by concatenating the undecimated ‘db1’ and ‘db2’ wavelet transform with 2-levels of decomposition. Thus, we can obtain the sub-dictionaries: .

The SNR measurement is adopted to measure the noise level, which is defined as , where and denote the number of and the variance of the white Gaussian noise, respectively. The higher the value of mSNR, the weaker of the noise level is. We evaluate the performance by the popular recovery SNR: , where denotes the estimated sparse image. The higher the value of , the better the performance. We conduct the experiments by utilizing the well-known figure of “Cameraman” with , which is shown in Figure 2a. To reduce the computation time, we choose only a part of the figure, shown in Figure 2b.

Figure 2

(a) the cropped Cameraman image of size . (b) the selected image portion of size .

4.1. The Value Range of in

In Section 4.1, we first determine the value range for in by evaluating the performances with different values of . The results are shown in Figure 3 and Figure 4. From the results, we can find that when , the proposed algorithms perform well and enjoy strong robustness. With the increase of , the performances deteriorate rapidly. Therefore, we estimate that the value of should be to obtain the adaptive weighting. We specially set:

Figure 3

The of the proposed SAITA- algorithm of and . (a) The RSNR of . (b) The RSNR of .

Figure 4

The recovery of the proposed SAITA- algorithm of and . (a) The RSNR versus Sampling Ratio of ; (b) The RSNR versus Sampling Ratio of .

4.2. The Recovery SNR Performances Versus Sampling Ratio

In Section 4.2, we evaluate the robustness of the proposed algorithm by considering the RSNR versus the sampling ratio. Specifically, we set three mSNR levels to . Figure 5 depicts the RSNR versus the sampling ratio. Based on the results, the proposed SAITA- algorithm performs better than the conventional single dictionary based algorithm, and the robustness of the proposed algorithm is good.

Figure 5

The RSNR performances of the proposed SAITA- algorithm and the algorithm with . (a) . (b) .

4.3. The Recovery SNR Performance Versus Measurement SNR

For our third experiment, we investigate the influence of different noise levels on the proposed algorithm and compare the algorithm and Co-L1 [28]. Similarly, we consider three sampling ratio levels of . We evaluate the performance by the RSNR versus the lower mSNR (20 dB~40 dB) of the image, and the results are presented in Figure 6.

Figure 6

(a) The case of for the sampling ratios of the Cameraman images. (b) The case of for the sampling ratios of the cameraman images. The RSNR versus mSNR of the proposed SAITA- algorithm and the algorithm for three low sampling ratios .

From the results, we can find that the proposed SAITA- algorithm can obtain a higher recovery SNR and has better robustness and fidelity than the Co-L1. The robustness and fidelity of the corresponding algorithm will deteriorate with the increase of the signal measurement SNR.

4.4. The Relative Error Performances Versus the Number of Iterations

We study the convergence and the reconstruction error by the relative error performances versus the number of iterations. Choosing the relative error as the second quality measurement, the formula is given:

Considering the proposed SAITA- and the corresponding algorithm from the result shown in Figure 7, when the sampling ratio is 0.2 (shown in (a)), the relative errors of the proposed SAITA- algorithm are significant smaller, and converge faster than the corresponding algorithm. When the sampling ratio is 0.5 (shown in (b)), though the final relative errors are close, the convergence speed of the proposed algorithm is higher (the number of iterations are approximately 15 and 7, respectively). While compared to Co-L1 [28], our proposed SAITA- algorithm can obtain a markedly lower relative error when the sampling ratio is . In addition, one can observe that the relative error is slightly smaller than for , while the convergence rate is faster than the .

Figure 7

(a) The relative error for the lower sampling ratio in the cameraman image. (b) The relative error for the higher sampling ratio of in the cameraman image. The relative errors verse the iteration number of the proposed SAITA- algorithm and the algorithm.

5. Practical Experiments

The proposed algorithms can be applied in many practical applications [36,37,38,39,40,41,42]. In this section, we conduct some typical applications about sparse image recovery and medical imaging to extend the applications of and regularizations and illustrate the excellent robustness and adaptation of the proposed SAITA-, algorithm. The conventional single dictionary analysis iterative thresholding algorithms [24,25] and the Co-L1 [28] as the standard algorithms for comparison.

5.1. Application 1: Image Sparse Restoration

In the first application, the proposed SAITA- algorithm is applied to restoring the “Cameraman” image shown in Figure 2 versus sampling ratio . We use the reduced cameraman image as the objective image. Figure 8a,c show the recovery results using the conventional single dictionary algorithm of and , respectively. Figure 8b,d show the recovery images using the corresponding multiple sub-dictionary algorithm of and , respectively. The experiments show that all the algorithms can recover the images, and the proposed multiple sub-dictionary algorithms significantly outperform the conventional single-dictionary algorithms. In Figure 9, we depict the RSNR of four algorithms vs. different sampling ratios. When and , it can be observed that the proposed SAITA algorithm can obtain a lager RSNR compared with the conventional algorithms. There is no obvious improvement between the two cases of and , and the SAITA- algorithm outperforms the SAITA- algorithm with a weak advantage.

Figure 8

The recovery effects of the proposed SAITA- and the conventional algorithms for the and cameraman image. (a) , ; (b) SAITA- algorithm (), ; (c) , ; and (d) SAITA- algorithm (), .

Figure 9

The RSNR of the SAITA- algorithms vs. the sampling ratio for the cameraman image.

5.2. Application 2: Medical Imaging

In the application 2, we extend the applications of our proposed algorithm to solve typical medical construction problems. We first consider the well-known Shepp-Logan phantom, and then we construct a high-quality dMRI cardiac cine [8,28].

5.2.1. Test 1 for the Shepp-Logan Model

In the test 1, we first consider the well-known Shepp-Logan phantom of with an . We construct the compressed noisy measurement signal by utilizing the “Spread Spectrum” operator as the measurement matrix [35], and we conduct a sparsifying transform with the constructed operator (‘db3’, ).

From the experimental results shown in Figure 10, we find that the proposed SAITA- algorithm can recover the images perfectly, as shown in Figure 10b,d, while the conventional single dictionary algorithms failed to recover the image, which is shown in Figure 10a,c. The proposed SAITA- algorithm of can obtain the best effect compared with the other algorithms, and the next best is the SAITA- algorithm of . In Figure 11, we depict the RSNR of the respective algorithms versus different sampling ratios . When and , it can be observed that the proposed SAITA- algorithms can obtain a larger RSNR than the algorithms with significantly.

Figure 10

The recovery effects of the proposed SAITA- and the corresponding algorithm for the , Shepp-Logan image. (a) , ; (b) SAITA- algorithm (), ; (c) , ; and (d) SAITA- algorithm (), .

Figure 11

The RSNR of the proposed SAITA- algorithm and the conventional algorithm vs. the sampling ratio for the Shepp-Logan image.

5.2.2. Test 2 for Real-World Data (2D MRI)

MRI is a typical medical inverse problem that can be solved well by CS [8]. In this experiment, we apply the proposed algorithm to real-world medical data. We investigate a simplified “dynamic MRI” problem [8] and use the high-quality MRI cardiac cine as the ground truth and select a spatio-temporal slice of [28]. We construct the sparse matrix with a vertical concatenation of ‘db1’ and ‘db2’ orthogonal discrete wavelet bases with two levels of decomposition (‘db1’, ‘db2’, and ). Figure 12 presents the constructed MRI images using the SAITA- algorithm and algorithms. From the experiment results, we can see that the proposed SAITA algorithm can reconstruct the images perfectly, as shown in Figure 12b,d, while the conventional algorithms failed to restore the image in Figure 12a,c. The proposed multiple sub-dictionaries algorithm of obtained the best effect and the corresponding recovery SNR is 23.1872 dB. Next is the proposed algorithm of with recovery SNRs of 21.0189 dB. In Figure 13, we depict the RSNR of four algorithms versus different sampling ratios . When and , it can be observed that the proposed SAITA- algorithms can obtain a lager RSNR compared with the conventional single dictionary algorithms. The results also show that the algorithms of outperform the methods of . That is to say, the regularization can exploit more prior knowledge than regularization in MRI.

Figure 12

The recovery effects of the proposed SAITA- and the corresponding and algorithm for the , 2D MRI image. (a) algorithm, ; (b) SAITA- algorithm, ; (c) algorithm, ; and (d) SAITA- algorithm, .

Figure 13

The RSNR of the proposed SAITA- algorithm and the algorithm versus the sampling ratio for the 2D MRI image.

6. Conclusions

In this paper, we propose a novel adaptive iteratively weighted thresholding algorithm (SAITA-) based on the conventional and thresholding algorithm by incorporating the multiple sub-dictionary sparse representation strategy. We make the following conclusions from the above experiments:

Using the proposed multiple sub-dictionary sparsifying transforms strategy, we construct a multiple sub-dictionary based regularization method to exploit more priori knowledge of images for the sparse image recovery problem. By multiplying by the proposed adaptive weighting parameter , we can gain more control of weighting the contribution of each sub-dictionary based regularizer. Experiments show that the proposed algorithms appear to perform better than the conventional single-dictionary algorithms, especially when the sampling rate is very low (e.g., 0.1~0.3);

Compared with the -norm regularization based work, the nonconvex -norm penalty can more closely approximate the -norm minimization over the -norm, which gives a sparser solution and needs fewer measurement data.

In our experiments, we find that the recovery performances between the and are close, even when the corresponding algorithm can obtain a better performance over the . Hence, a proper need to be selected in practical applications.

Moreover, the proposed SAITA- method also indicates that it is feasible to improve the recovery performance by exploiting the signal inner sparse structure and designing a proper sparse representation dictionary. Thus, it is beneficial to exploit the signal sparse structure with a dictionary learning method, which will be the subject of future work.

The proposed SAITA- algorithm can be extended to other non-convex penalties include smoothly clipped absolute deviation (SCAD) and minimax concave penalty (MCP).