A symmetric version of the generalized alternating direction method of multipliers for two-block separable convex programming.

This paper introduces a symmetric version of the generalized alternating direction method of multipliers for two-block separable convex programming with linear equality constraints, which inherits the superiorities of the classical alternating direction method of multipliers (ADMM), and which extends the feasible set of the relaxation factor α of the generalized ADMM to the infinite interval [Formula: see text]. Under the conditions that the objective function is convex and the solution set is nonempty, we establish the convergence results of the proposed method, including the global convergence, the worst-case [Formula: see text] convergence rate in both the ergodic and the non-ergodic senses, where k denotes the iteration counter. Numerical experiments to decode a sparse signal arising in compressed sensing are included to illustrate the efficiency of the new method.

Introduction

We consider the two-block separable convex programming with linear equality constraints, where the objective function is the sum of two individual functions with decoupled variables: where () are closed proper convex functions; () and , and () are given nonempty closed convex sets. The linear constrained convex problem (1) is a unified framework of many problems arising in real world, including compressed sensing, image restoration, and statistical learning, and so forth (see, for example, [1-3]). An important special case of (1) is the following linear inverse problem: where and are given matrix and vector, is a regularization parameter and is the -norm of a vector x defined as . Then setting , , (2) can be converted into the following two-block separable convex programming: which is a special case of problem (1) with the following specifications:

Existing algorithms

In their seminal work, Glowinski et al. [4] and Gabay et al. [5] independently developed the alternating direction method of multipliers (ADMM), which is an influential first-order method for solving problem (1). ADMM can be regarded as an application of the Douglas-Rachford splitting method (DRSM) [6] to the dual of (1), or a special case of the proximal point algorithm (PPA) [7, 8] in the cyclic sense. We refer to [9] for a more detailed relationship. With any initial vectors , , the iterative scheme of ADMM reads where is the Lagrangian multiplier and is a penalty parameter. The main characteristics of ADMM are that it in full exploits the separable structure of problem (1), and that it updates the variables , , λ in an alternating order by solving a series of low-dimensional sub-problems with only one unknown variable.

In the past few decades, ADMM has received a revived interest, and it has become a research focus in optimization community, especially in the (non)convex optimization. Many efficient ADMM-type methods have been developed, including the proximal ADMM [8, 10], the generalized ADMM [11], the symmetric ADMM [12], the inertial ADMM [13], and some proximal ADMM-type methods [14-18]. Specifically, the proximal ADMM attaches some proximal terms to the sub-problems of ADMM (4). The generalized ADMM updates the variables and λ by including a relaxation factor , and is often advantageous to speed up its performance. The symmetric ADMM updates the Lagrangian multiplier λ twice at each iteration and includes two relaxation factors , . Recent researches of the symmetric ADMM can be found in [12, 15, 18]. The inertial ADMM unifies the basic ideas of the inertial PPA and ADMM, which utilizes the latest two iterates to generate the new iterate, therefore it can be viewed as a multistep method. For the proximal ADMM, the objective functions of its sub-problems are often strongly convex, which are often easier to be solved than those of (4). However, a new challenge has arisen for the proximal ADMM-type methods. It is how to choose a proper proximal matrix. In fact, most proximal ADMM-type methods need to estimate the matrix norm (), which demands lots of calculations, especially for large (). Quite recently, some customized Douglas-Rachford splitting algorithms [19-21], and the proximal ADMM-type methods with indefinite proximal regularization are developed [22, 23], which dissolve the above problem to some extent. All the above mentioned ADMM-type methods are generalizations of the classical ADMM, because they all reduce to the iterative scheme (4) by choosing some special parameters. For more new development of the ADMM-type methods, including the convergence rate, acceleration techniques, its generalization for solving multi-block separable convex programming and nonconvex, nonsmooth programming, we refer to [24-28].

Contributions and organization

We are going to further study the generalized ADMM. Note that the first sub-problem in the generalized ADMM is irrelevant to the relaxation factor α. That is, the updating formula for does not incorporate the relaxation factor α explicitly. Furthermore, is often advantageous for the generalized ADMM [14]. Therefore, in this paper, we are going to propose a new generalized ADMM, whose both sub-problems incorporate the relaxation factor α directly. The new method generalizes the method proposed in [29] by relaxing the feasible set of α from the interval to the infinite interval , and can be viewed as a symmetric version of the generalized ADMM.

The rest of the paper is organized as follows. In Section 2, we summarize some necessary preliminaries and characterize problem (1) by a mixed variational inequality problem. In Section 3, we describe the new symmetric version of the generalized ADMM and establish its convergence results in detail. In Section 4, some compressed sensing experiments are given to illustrate the efficiency of the proposed method. Some conclusions are drawn in Section 5.

Preliminaries

In this section, some necessary preliminaries which are useful for further discussions are presented, and to make our analysis more succinct, some positive definite or positive semi-definite block matrices are defined and their properties are investigated.

For two real matrices , , the Kronecker product of A and B is defined as . Let () denote the standard definition of -norm; in particular, . For any two vectors , or denote their inner product, and for any symmetric matrix , the symbol (resp., ) denotes that G is positive definite (resp., semi-definite). For any and , the G-norm of the vector x is defined as . The effective domain of a closed proper function is defined as , and the symbol denotes the set of all relative interior points of a given nonempty convex set . Furthermore, we use the following notations:

Definition 2.1

[30]

A function is convex if and only if Then, for a convex function , we have the following basic inequality: where denotes the subdifferential of at the point y.

Throughout the paper, we make the following standard assumptions for problem (1).

Assumption 2.1

The functions () are convex.

Assumption 2.2

The matrices () are full-column rank.

Assumption 2.3

The generalized Slater condition holds, i.e., there is a point .

The mixed variational inequality problem

Under Assumption 2.3, it follows from Theorem 3.22 and Theorem 3.23 of [31] that is an optimal solution to problem (1) if and only if there exists a vector such that is a solution of the following KKT system: where is the normal cone of the convex set at the point , which is defined as . Then, for the nonempty convex set and , it follows from [32] (Example 2.123) that , where is the indicator function of the set , and is the subdifferential mappings of at the point .

Lemma 2.1

For any vector , , the relationship is equivalent to and the inequality

Proof

From , we have and there exists such that From the subgradient inequality (5), one has Thus, where the second inequality comes from and .

Conversely, from , , we have which together with implies that From this and Theorem 3.22 of [31], we have . This completes the proof. □

Remark 2.1

Based on (6) and Lemma 2.1, the vector is an optimal solution to problem (1) if and only if there exists a vector such that Moreover, any satisfying (7) is an optimal solution to the dual of problem (1). Obviously, (7) can be written as the following mixed variational inequality problem, denoted by : Find a vector such that where , , and The solution set of , denoted by , is nonempty by Assumption 2.3 and Remark 2.1. It is easy to verify that the linear function is not only monotone but also satisfies the following desired property:

Three matrices and their properties

To present our analysis in a compact way, now let us define some matrices. For any , set and for , set The above defined three matrices M, Q, H satisfy the following properties.

Lemma 2.2

If and (), then the matrix H defined in (11) is positive semi-definite.

Set , which is positive for any . By (11), we have Obviously, the first part is positive semi-definite, and we only need to prove the second part is also positive semi-definite. In fact, it can written as The middle matrix in the above expression can be further written as where ⊗ denotes the matrix Kronecker product. The matrix Kronecker product has a nice property: for any two matrices X and Y, the eigenvalue of equals the product of , where and are the eigenvalues of X and Y, respectively. Therefore, we only need to show the 2-by-2 matrix is positive semi-definite. In fact, Therefore, the matrix H is positive semi-definite. The proof is then complete. □

Lemma 2.3

If and (), then the matrices M, Q, H defined, respectively, in (10), (11) satisfy the following relationships: and

From (10) and (11), we have Then the first assertion is proved. For (13), by some simple manipulations, we obtain We now break up the proof into two cases. First, if , then Therefore, (13) holds. Second, if , then Note that The middle matrix in the above expression can be further written as Since the right-hand side of (15) is also positive semi-definite. Thus, we have Substituting (16) into (14) and by the expression of , we obtain (13). The lemma is proved. □

Algorithm and convergence results

In this section, we first describe the symmetric version of the generalized alternating direction method of multipliers (SGADMM) for formally, and then we prove its global convergence in a contraction perspective and establish its worst-case convergence rate in both the ergodic and the non-ergodic senses step by step, where k denotes the iteration counter.

Algorithm

Algorithm 3.1

SGADMM

Choose the parameters , , (), the tolerance and the initial iterate . Set .

Generate the new iterate by

If then stop and return an approximate solution of ; else set , and goto Step 1.

Remark 3.1

Obviously, the iterative scheme (17) reduces to the generalized ADMM when , and further reduces to (4) when (). That is to say, if the parameters and (), then the classical ADMM is recovered. Since the convergence results of the (proximal) ADMM have been established in the literature [23, 33, 34], in the following, we only consider .

Global convergence

For further analysis, we need to define an auxiliary sequence as follows:

Lemma 3.1

Let and be the two sequences generated by SGADMM. Then and

From the definition of , we get Then (20) is proved. For (21), we have Therefore (21) is also right. This completes the proof. □

Thus, based on (19) and (20), the two sequences and satisfies the following relationship: where M is defined in (10).

The following lemma shows that the stopping criterion (18) of SGADMM is reasonable.

Lemma 3.2

If (), and , then the iterate produced by SGADMM is a solution of .

By invoking the optimality condition of the three sub-problems in (4), we have the following mixed variational inequality problems: for any , Then, adding the above three inequalities and by (20), (21), we get Then by (19), we obtain Then, by (11) (the definition of Q), the above inequality can be rewritten as for any . Therefore, if (), and , then by (20), we have . Then . Thus, we have which together with (23) implies that This indicates that the vector is a solution of . This completes the proof. □

Lemma 3.3

Let and be two sequences generated by SGADMM. Then, for any , we have

Applying the identity with we obtain This together with (12) and (22) implies that Now let us deal with the last term in (25), which can be written as Substituting the above inequality into (25), the assertion of this lemma is proved. □

Theorem 3.1

Let and be two sequences generated by SGADMM. Then, for any , we have

First, combining (23) and (24), we get From the monotonicity of , we have Adding the above two inequalities, we obtain the assertion (26). The proof is completed. □

With the above theorem in hand, we are ready to establish the global convergence of SGADMM for solving .

Theorem 3.2

Let be the sequence generated by SGADMM. If , (), then the corresponding sequence converges to some , which belongs to .

Setting in (26), we have where the second inequality follows from . Thus, we have Summing over , it yields By and the positive semi-definite of H, the above inequality implies that Thus, by the definition of H, we have where is positive definite by . From (27) again, we have which indicates that the sequence is bounded. Thus, and are both bounded. Then is bounded. If , is bounded; otherwise, , that is, is full-column rank, which together with implies that is bounded. In conclusion, is bounded.

Then, from (28) and , the sequence is convergent. Suppose it converges to . Let be a cluster point of and be the corresponding subsequence. On the other hand, by (20) and (28), we have and Thus, Then, taking the limit along the subsequence in (23) and using (29), for any , we obtain which indicates that is a solution of . Then, since in (27) is arbitrary, we can set and conclude that the whole generated sequence converges by (). This completes the proof. □

Convergence rate

Now, we are going to prove the worst-case convergence rate of SGADMM in both the ergodic and the non-ergodic senses.

Theorem 3.3

Let and be the sequences generated by SGADMM, and set Then, for any integer , we have , and

From (17) and the convexity of the set , we have . From (26), we have Summing the above inequality over , we get By the definition of and the convexity of , the assertion (30) follows immediately from the above inequality. This completes the proof. □

The proof of the next two lemmas is referred to those of Lemmas 5.1 and 5.2 in [24]. For completeness, we give the detail proof.

Lemma 3.4

Let be the sequence generated by SGADMM. Then we have

Setting in (23), we have Similarly setting in (23) for , we get Then, adding the above two inequalities and using the monotonicity of the mapping , we get

By (32), we have Using (13), (22) and on both sides of the above inequality, we get Then we get the assertion (31). The proof is completed. □

Lemma 3.5

Let be the sequence generated by SGADMM. Then we have

Setting and in the identity we can derive which completes the proof of the lemma. □

Based on Lemma 3.5, now we establish the worst-case convergence rate of SGADMM in a non-ergodic sense.

Theorem 3.4

Let be the sequence generated by SGADMM. Then, for any and integer , we have

By (27), we get This and (33) imply that Therefore, the assertion of this theorem comes from the above inequality immediately. The proof is completed. □

Remark 3.2

From (34), we see that the larger α is, the smaller , which controls the upper bounds of . Therefore, it seems that larger values of α are more beneficial for speeding up the convergence of SGADMM.

Numerical experiments

In this section, we present some numerical experiments to verify the efficiency of SGADMM for solving compressed sensing. Those numerical experiments are performed in Matlab R2010a on a ThinkPad computer equipped with Windows XP, 997 MHz and 2 GB of memory.

Compressed sensing (CS) is to recover a sparse signal from an undetermined linear system , where (), can be depicted as problem (2).

Obviously, Problem (2) is equivalent to the following two models:

Model 1: Problem (3).

Model 2:

The iterative schemes for (3) and (35)

Since (3) and (35) are both some concrete models of (1), SGADMM are applicable to them. Below, we elaborate on how to derive the closed-form solutions for the sub-problems resulting by SGADMM.

For problem (3), its first two sub-problems resulting by SGADMM are as follows.

• With the given and , the -sub-problem in (17) is (here ) which has the following closed-form solution:

• With the updated , the -sub-problem in (17) is (here with ) and its closed-form solution is given by where, for any , is defined as and should be taken 0 if .

Similarly, for problem (35), its first two sub-problems resulting by SGADMM are as follows.

• With the given and , the -sub-problem in (17) is (here ) and its closed-form solution is given by

• With the updated , the -sub-problem in (17) is (here with ) and its closed-form solution is given by

Obviously, the above two iterative schemes both need to compute and , which is quite time consuming if n is large. However, noting that these two terms are invariant during the iteration process, therefore we need only compute them once before all iterations.

Regarding the penalty parameter β and the constant α in SGADMM, any and can ensure the convergence of SGADMM in theory. There are two traditional methods to determine them in practice. One is the tentative method, which is easy to execute. The other is the self-adaptive adjustment method, which needs much computation. In this experiment, for β and α, we use the tentative method to determine their suitable values. For β, Xiao et al. [35] set for ADMM. Motivated by this choice, we set in our algorithm. As for the parameter α, we have pointed out in Remark 3.2 that larger values of α may be beneficial for our algorithm. Here, we use (3) to do a little experiment to test this. We choose different values of α in the interval . Specifically, we choose . Other data about this experiment are as follows: the proximal parameter τ is set as ; the observed signal y is set as in Matlab; the sensing matrix A and the original signal x are generated by and Then the observed signal y is further set as . The initial points are set as , . In addition, we set the regularization parameter , and the dimensions of the problem are set as , , , where k denotes the number of the non-zeros in the original signal x. To evaluate the quality of the recovered signal, let us define the quantity ‘RelErr’ as follows: where x̃ denotes the recovered signal. The stopping criterion is where denotes the function value of (2) at the iterate .

Numerical results

The numerical results are graphicly shown in Figure 1. Clearly, the numerical results in Figure 1 indicate that Remark 3.2 is reasonable. Both CPU time and number of iterations are descent with respect to α. Then, in the following, we set , which is a moderate choice for α.

Figure 1

Sensitivity test on the parameter .

Now, let us graphically show the recovered results of SGADMM for (3) and (35). The proximal parameter τ is set as for (3), and for (35). The initial points are set as , for (3), and , for (35). Other parameters are set the same as above. Figure 2 reports the numerical results of SGADMM for (3) and (35).

Figure 2

Numerical results of SGADMM for ( ) and ( ). The top: the original signal; the second: the noisy measurement; the bottom two: recovered signal.

The bottom two subplots in Figure 2 indicate that our new method SGADMM can be used to solve (3) and (35).

In the following, we do some numerical comparisons to illustrate the advantage of our new method and to analyze which one is more suitable to compressed sensing (2) between the two models (3) and (35). SGADMM for (3) is denoted by SGADMM1, SGADMM for (35) is denoted by SGADMM2. We also compare SGADMM with the classical ADMM. The numerical results are listed in Table 1, where ‘Time’ denotes the CPU time (in seconds), and ‘Iter’ denotes the number of iterations required for the whole recovering process, , . The numerical results are the average of the numerical results of ten runs with different combinations of γ and σ.

Table 1

Comparison of SGADMM1, SGADMM2 and ADMM

n	γ	σ	SGADMM1			SGADMM2			ADMM
			Time	Iter	RelErr	Time	Iter	RelErr	Time	Iter	RelErr
1,000	0.3	0.2	0.6911	92.4	0.0387	0.9578	266.0	0.0394	0.9812	264.0	0.0393
	0.2	0.2	0.6661	118.6	0.0825	1.3915	421.8	0.0915	1.3603	419.6	0.0915
	0.2	0.1	0.5008	85.3	0.0609	0.4758	139.0	0.0579	0.5008	138.0	0.0539
2,000	0.3	0.2	2.1965	90.0	0.0437	3.6535	267.7	0.0447	3.5412	265.6	0.0447
	0.2	0.2	2.2339	109.6	0.0785	5.2182	431.4	0.0874	5.1543	429.0	0.0874
	0.2	0.1	1.5428	79.9	0.0534	1.7893	142.8	0.0467	1.7613	140.8	0.0513

Discussion

The numerical results in Table 1 indicate that: (1) by the criterion ‘RelErr’, all methods successfully solved all the cases; (2) by the criteria ‘Time’ and ‘Iter’, SGADMM1 performs better than the other two methods. Especially the number of iterations of SGADMM1 is about at most two-thirds of the other two methods. This experiment also indicate that the model (3) is also an effective model for compressed sensing, and sometimes it is more efficient than the model (35), though they are equivalent in theory. In conclusion, by choosing some relaxation factor , SGADMM may be more efficient than the classical ADMM.

Conclusions

In this paper, we have proposed a symmetric version of the generalized ADMM (SGADMM), which generalizes the feasible set of the relaxation factor α from to . Under the same conditions, we have proved the convergence results of the new method. Some numerical results illustrate that it may perform better than the classical ADMM. In the future, we shall study SGADMM with to perfect the theoretical system.