An accelerated proximal augmented Lagrangian method and its application in compressive sensing.

As a first-order method, the augmented Lagrangian method (ALM) is a benchmark solver for linearly constrained convex programming, and in practice some semi-definite proximal terms are often added to its primal variable's subproblem to make it more implementable. In this paper, we propose an accelerated PALM with indefinite proximal regularization (PALM-IPR) for convex programming with linear constraints, which generalizes the proximal terms from semi-definite to indefinite. Under mild assumptions, we establish the worst-case [Formula: see text] convergence rate of PALM-IPR in a non-ergodic sense. Finally, numerical results show that our new method is feasible and efficient for solving compressive sensing.

Introduction

Let denote the set of all real numbers, the Euclidean space of all real vectors with n coordinates. In this paper, we are going to solve the following linearly constrained convex programming: where is a closed proper convex function, , . Throughout, we assume that the solution set of Problem (1) is nonempty. By choosing different objective function , a variety of problems encountered in compressive processing, machine learning and statistics can be cast into Problem (1) (see [1-8] and the references therein). The following are two concrete examples of Problem (1):

• The compressive sensing (CS): where is the sensing matrix; is the observed signal, and the -norm and -norm of the vector x are defined by and , respectively.

• The wavelet-based image processing problem: where is a diagonal matrix whose elements are either 0 (missing pixels) or 1 (known pixels), and is a wavelet dictionary.

Problem (1) can be converted into the following strongly convex programming: where the constant is a penalty parameter. Introducing the Lagrange multiplier to the linear constraints , we get the Lagrangian function associated with Problem (4): which is also the augmented Lagrangian function associated with Problem (1). Then the dual function is denoted by and the dual problem of (4) is Due to the strong convexity of the objective function of Problem (4), is continuously differentiable at any , and , where (see, e.g., Theorem 1 of [9]). Solving the above dual problem by the gradient ascent method, we get a benchmark solver for Problem (1): the augmented Lagrangian method (ALM) [10, 11], which first minimizes the Lagrangian function of Problem (4) with respect to x by fixing to get , and set ; then it updates the Lagrange multiplier λ. Specifically, for given , the kth iteration of PALM for Problem (1) reads where is a relaxation factor. Though ALM plays a fundamental role in the algorithmic development of Problem (1), the cost of solving its first subproblem is often high for general and A. To address this issue, many proximal ALMs [3, 12–14] are developed by adding the proximal term to the x-related subproblem, where is a semi-definite matrix. By setting with , the x-related subproblem reduces to the following form: where . The above subproblem is often simple enough to have a closed-form solution or can be easily solved up to a high precision. The proximal ALM is so instructive, and along this philosophy, a lot of efficient proximal ALM-type methods [3, 15–18] have been proposed. However, a new difficult problem has arisen for the proximal ALM-type methods, which is how to determine the optimal value of the proximal parameter τ. Numerical results indicate that smaller values of τ can often speed up the corresponding proximal ALM-type method [19].

In [20], Fazel et al. pointed out that the proximal matrix G should be as small as possible, while the subproblem related to the primal variable x is still relatively easy to tackle. Furthermore, though any can ensure the global convergence of proximal ALM-type methods, the computation of the norm is often high for some problems in practice, especially for large n. Therefore, it is meaningful to relax the feasible region of the proximal parameter τ. Quite recently some efficient methods with relaxed proximal matrix [21-24] have been developed for Problem (1). Specifically, He et al. [23] proposed a positive-indefinite proximal augmented Lagrangian method, in which the proximal matrix is , where is an arbitrarily positive definite matrix in , and . If we set with , then , which maybe indefinite.

Many research results on the convergence speed of the ALM-type methods have been presented recently. For the classical ALM, He et al. [25] firstly established the worst-case convergence rate in the non-ergodic sense, and further developed a new ALM-type method, which has convergence rate. However, for inequality (3.13) in [25], the authors only proved with , and we cannot ensure . Similar problem occurs in Theorem 2.2 of [26]. In the following, we shall prove that the inequality holds for the method in [26]. Quite recently, by introducing an ingeniously designed sequence , Lu et al. [27] proposed a fast PALM-type method without proximal term, which has convergence rate.

In this paper, based on the study of [27], we are going to further study the augmented Lagrangian method and develop a new fast proximal ALM-type method with indefinite proximal regularization, whose worst-case convergence rate is in a non-ergodic sense. Furthermore, a relaxation factor is attached to the updated formula of our new method, which is often beneficial to speed up convergence in practice.

The rest of this paper is organized as follows. In Section 2, we list some necessary notations. We then give the proximal ALM with indefinite proximal regularization (PALM-IPR) and show its worst-case convergence rate in Section 3. In Section 4, numerical experiments are conducted to illustrate the efficiency of PALM-IPR. Finally, some conclusions are drawn in Section 5.

Preliminaries

In this section, we give some notations used in the following analysis and present two criteria to measure the worst-case convergence rate of PALM type methods. At the end of this section, we prove that the inequality holds for the method in [26].

Throughout, we use the following standard notations. For any two vectors , or denote their inner products. The symbols and represent the -norm and -norm for vector variables, respectively. denotes the n-dimensional identity matrix. If the matrix is symmetric, we use the symbol to denote even if G is indefinite; (resp., ) denotes that the matrix G is positive definite (resp., semi-definite).

The following identity will be used in the following analysis:

Definition 2.1

A point is a Karush-Kuhn-Tucker (KKT) point of Problem (1) if the following conditions are satisfied:

Note that the two conditions of (7) correspond to the dual feasibility and the primal feasibility of Problem (1), respectively. The solution set of KKT system (7), denoted by , is nonempty under the nonempty solution set of (1). By (7) and the property of the convex function , for any , we have Based on (8) and , we have the following proposition.

Proposition 2.1

([28])

Vector is an optimal solution to Problem (1) if and only if there exists such that where .

Now let us review two different criteria to measure the worst-case convergence rate in [28, 29].

(1) In [29], Xu presented the following criterion: where . The second inequality of (10) implies that there must exist at least one with .

(2) In [28], Lin et al. proposed the following criterion: where . Obviously, inequality (11) is motivated by equality (9). Compared with (10), the criterion (11) is more reasonable. Therefore, we shall use (11) to measure the convergence rate of our new method.

Now, we prove that the inequality holds for the iteration method proposed in [26].

Theorem 2.1

For the KKT point of Problem (1), the tuple generated by the method in [26] satisfies

Proof

Since is generated by the following subproblem: then it holds that i.e., This and the convexity of the function yield So, it holds that This completes the proof. □

PALM-IPR and its convergence rate

In this section, we first present the proximal ALM with indefinite proximal regularization (PALM-IPR) for Problem (1) and then prove its convergence rate step by step.

To present our new method, let us define a sequence which satisfies , and Then the sequence has the following properties [27]: and

Algorithm 3.1

(PALM-IPR for Problem (1))

Step 1. Input the parameters , the tolerance . Initialize and . Set .

Step 2. Compute

Step 3. If , then stop; otherwise, choose , and set . Go to Step 2.

To prove the global convergence of PALM-IPR, we need to impose some restrictions on the proximal matrix , which is stated as follows.

Assumption 3.1

The proximal matrix is set as , where and the matrix , .

Remark 3.1

The proximal matrix maybe indefinite. For example, if we set , then , which is indefinite when A is full-column rank.

Using the first-order optimality condition of the subproblem of PALM-IPR, we can deduce the following one-iteration result.

Lemma 3.1

Let be the sequence generated by PALM-IPR. For any , it holds that

From the first-order optimality condition for z-related subproblem (15), we have where . Then, from the convexity of and (21), we have

From (16) and the convexity of again, we thus get where the second inequality follows from (22). Then, by rearranging terms of the above inequality, we arrive at where the second inequality uses (19), and the third inequality comes from identity (6). Dividing both sides of the above inequality by , we get From (17), we have Substituting this into the above inequality and using (12) lead to (20). This completes the proof. □

Lemma 3.2

Let be the sequence generated by PALM-IPR. For any with , it holds that

Adding the term to both sides of (20), we get

Now, let us deal with the term . By (17), we have Substituting the above equality into (24) and using (19), we get (23) immediately. This completes the proof. □

Let us further deal with the term on the right-hand side of (24).

Lemma 3.3

Let be the sequence generated by PALM-IPR. For any with , it holds that

By Assumption 3.1, we have Using the inequality with and , we get Substituting this inequality into the right-hand side of the above equality, we obtain assertion (25) immediately. This completes the proof. □

Then, from (23) and (25), we have where the second inequality comes from (17) and the fact .

Based on (26), we are going to prove the worst-case convergence rate of PALM-IPR in an ergodic sense.

Theorem 3.1

Let be the sequence generated by PALM-IPR. Then

Setting and in (26), we get Summing the above inequality over and by (17), we have where the second inequality follows from and . By equations (65), (68) of [27], we have and Substituting the above two relationships into (27) and using (13), we get Then multiplying two sides of the above inequality by and using (12) lead to where the second inequality follows from (14). The above inequality with results in the conclusion of the theorem. The proof is completed. □

Numerical results

In this section, we apply PALM-IPR to some practical applications and report the numerical results. All the codes were written by Matlab R2010a and conducted on ThinkPad notebook with 2GB of memory.

Problem 4.1

(Quadratic programming)

Firstly, let us test PALM-IPR on equality constrained quadratic programming (ECQP) [29] to validate its stability: We set the problem size to and generate and according to the standard Gaussian distribution. We compare PALM-IPR with the classical ALM with . For PALM-IPR, we set for simplicity. We have tested the experiment sixty times, and the numerical results are listed in Table 1, in which ‘NC’ means the number of convergence; ‘ND’ means the number of divergence and ‘Ratio’ means the ratio of succession. From Table 1, we can see that PALM-IPR performs much more stably than the classical ALM.

Table 1

Comparison of PALM-IPR with ALM

Method	NC	ND	Ratio
PALM-IPR	59	1	98.33%
ALM	36	24	60.00%

Problem 4.2

(Compressive sensing: the linearly constrained minimization problem (2))

Now, let us test PALM-IPR on the compressive sensing to validate its acceleration. For this problem, we firstly elaborate on how to solve subproblem (15) resulting from PALM-IPR. Due to the existence of the augmented term , we cannot get the closed-form solution if we do not attach the proximal term . In this case, we have to solve the subproblem inexactly, which is often time-consuming. Therefore, we choose the proximal matrix in (15) as , in which , and , and subproblem (15) can be written as which is equivalent to and has a closed-form solution as follows: in which . Note that all computations are component-wise.

In this experiment, we set and with , where k is the number of random nonzero elements contained in the original signal. The sensing matrix A is generated by the following Matlab scripts: , , and the nonzero entries of the true signal , whose values are sampled from the standard Gaussian: ; ; , are selected at random. The observed signal b is generated by . In addition, we set , . In this experiment, we set the proximal parameter . Furthermore, the stopping criterion is or the number of iterations exceeds 104, where is the iterate generated by PALM-IPR. Furthermore, all initial points are set as . For comparison, we also give the numerical results of PALM-SDPR [14] and the proximal PALM with positive-indefinite proximal regularization (PALM-PIPR) [21], and the proximal matrix is set to be in PALM-SDPR and in PALM-PIPR, and , . Furthermore, we set in PALM-PIPR. Since the computational load of all three methods are almost the same at each iteration, we only list the number of iterations (‘Iter’), the relative error (‘RelErr’) when three methods achieve the stopping criterion. The numerical results are listed in Table 1, and in the view of statistics, all the results are the average of 10 runs for each pair .

Numerical results in Table 2 indicate that: (1) All methods have succeeded in solving Problem (2) for all the scenarios; (2) The new method PALM-IPR outperforms PALM-SDPR and PALM-PIPR by taking a fewer number of iterations to converge except .

Table 2

Comparison of PALM-IPR with PALM-SDPR and PALM-PIPR

n	a	b	IPR		SDPR		PIPR
			Iter	RelErr	Iter	RelErr	Iter	RelErr
500	0.2	0.2	62.3	0.0488	93.2	0.0473	98.6	0.0491
	0.2	0.1	20.5	0.0482	73.6	0.0440	69.2	0.0443
1,000	0.2	0.2	91.9	0.0492	80.0	0.0476	84.3	0.0484
	0.2	0.1	20.6	0.0462	83.3	0.0469	79.9	0.0450
1,500	0.2	0.2	66.4	0.0493	77.5	0.0482	87.5	0.0485
	0.2	0.1	20.3	0.0471	76.6	0.0459	75.9	0.0453
2,000	0.2	0.2	44.7	0.0492	58.3	0.0491	54.0	0.0488
	0.2	0.1	19.1	0.0474	85.2	0.0481	86.5	0.0459
3,000	0.2	0.2	46.3	0.0488	86.8	0.0480	93.2	0.0477
	0.2	0.1	20.2	0.0474	69.3	0.0472	64.6	0.0496

Conclusions

In this paper, an accelerated augmented proximal Lagrangian method with indefinite proximal regularization (PALM-IPR) for linearly constrained convex programming is proposed. Under mild conditions, we have established the worst-case convergence rate in a non-ergodic sense of PALM-IPR. Two sets of numerical results, which illustrate that PALM-IPR performs better than some state-of-the-art solvers, are given.

Similar to our proposed method, the methods in [26, 30] also have the worst-case convergence rate in a non-ergodic sense, but they often have to solve a difficult subproblem at each iteration, and some inner iteration has to be executed. A prominent characteristic of the methods in [26, 30] is that the parameter β can be any positive constant, but the parameter β in PALM-IPR changes with respect to the iteration counter k, and it can actually go to infinity as . In practice, we often observe that larger β usually induces to slower convergence. Therefore, the method with proximal term, faster convergence rate and constant parameter β deserves further research.