Modified hybrid decomposition of the augmented Lagrangian method with larger step size for three-block separable convex programming.

The Jacobian decomposition and the Gauss-Seidel decomposition of augmented Lagrangian method (ALM) are two popular methods for separable convex programming. However, their convergence is not guaranteed for three-block separable convex programming. In this paper, we present a modified hybrid decomposition of ALM (MHD-ALM) for three-block separable convex programming, which first updates all variables by a hybrid decomposition of ALM, and then corrects the output by a correction step with constant step size α ∈ ( 0 , 2 - 2 ) which is much less restricted than the step sizes in similar methods. Furthermore, we show that 2 - 2 is the optimal upper bound of the constant step size α. The rationality of MHD-ALM is testified by theoretical analysis, including global convergence, ergodic convergence rate, nonergodic convergence rate, and refined ergodic convergence rate. MHD-ALM is applied to solve video background extraction problem, and numerical results indicate that it is numerically reliable and requires less computation.

Introduction

Many problems encountered in applied mathematics area can be formulated as separable convex programming, such as basis pursuit (BP) problem [1-3], video background extraction problem [4-7], image decomposition [8-10], and so on. Thus the solving of separable convex programming plays a fundamental role in applied mathematics and has drawn persistent attention. In the existing literature, several forms of separable convex programming have been investigated [11-15], in which the following three-block separable convex programming rouses more interest: where () are lower semicontinuous proper convex functions, () and , () are nonempty closed convex sets in (). Throughout this paper, we assume that the solution set of problem (1) is nonempty.

The Lagrangian and augmented Lagrangian functions of problem (1) are defined, respectively, as where is the Lagrange multiplier associated with the linear constraints in (1), and is a penalty parameter. Applying the augmented Lagrangian method (ALM) [16] to problem (1), we can obtain the following iterative scheme: Obviously, three variables , , are all involved in the minimization problem of (4), which makes the method often hard to implement. One technique to handle this is to split the subproblem into several small scale subproblems. Based on this, if we split it in a Gauss–Seidel manner and adopt the famous alternating direction method of multiplier (ADMM) [11], we obtain the following iterative scheme: On the other hand, if we split it in a Jacobian manner, we get the following full parallel iterative scheme: Compared with the minimization problem in (4), the scale of the minimization procedures in (5) and (6) is decreased, and they fully utilize the separable property of the objective function of (1), thus the new iterative schemes (5) and (6) gain some solvability. However, their convergence cannot be guaranteed under milder conditions as shown in [12, 17]. To overcome this drawback, several new techniques, such as the regularization method with large proximal parameter [18-23], the prediction-correction method with shrunk step size [12, 13, 24–26], etc., have been developed.

Compared with the regularization method, the prediction-correction method has attracted extensive interest, and during the past decades many scholars have performed studies in this direction. For example, He et al. [24] proposed an ADMM-based contraction type method for solving multi-block separable convex programming, which first generates a temporal iterate by (5), and then corrects it with a Gaussian back substitution procedure. Later, He et al. [12] developed a full Jacobian decomposition of the augmented Lagrangian method for solving multi-block separable convex programming, which first generates a temporal iterate by (6), and then corrects it with a constant step size or varying step size. Different from the above, Han et al. [13] proposed a partial splitting augmented Lagrangian method for solving three-block separable convex programming, which first updates the primal variables , , in a partially-parallel manner, and then corrects , λ with a constant step size. Later, Wang et al. [25] presented a proximal partially-parallel splitting method for solving multi-block separable convex programming, which first updates all primal variables in a partially-parallel manner, and then corrects the output with a constant step size or varying step size. Quite recently, Chang et al. [26] proposed a convergent prediction-correction-based ADMM in which more minimization problems are involved. In conclusion, the above iteration schemes first generate a temporal iterate by (5) or (6) or their variants, and then generate the new iterate by correcting the temporal iterate with varying step size or a constant step size.

Varying step size needs to be dynamically updated at each iteration, which might be computationally demanding for large-scale (1). Hence in this paper, we consider the prediction-correction method with constant step size for solving problem (1). To the best of our knowledge, He et al. [12] first proposed a prediction-correction method with constant step size for solving (1), and they proved that the upper bound of the constant step size is 0.2679. By taking a hybrid splitting of (4) as the prediction step, Wang et al. [25] relaxed the upper bound of the constant step size to 0.3670 and Han et al. [13] further relaxed it to 0.3820. In practice, to enhance the numerical efficiency of the corresponding iteration method, larger values of the step size are preferred as long as the convergence is still guaranteed [26]. In this paper, based on the methods in [12, 13, 25], we propose a modified hybrid decomposition of the augmented Lagrangian method with constant step size, whose upper bound is relaxed to 0.5858.

The rest of this paper is organized as follows. Section 2 lists some notations and basic results. In Sect. 3, we present a modified hybrid decomposition of the augmented Lagrangian method with larger step size for problem (1) and establish its global convergence and refined convergence rate. Furthermore, a simple example is given to illustrate that is the optimal upper bound of the constant step size in MHD-ALM. In Sect. 4, some numerical results are given to demonstrate the numerical advantage of larger step size. Finally, a brief conclusion including some possible future works is drawn in Sect. 5.

Preliminaries

In this section, we give some notations and basic results about the minimization problem (1), which will be used in the forthcoming discussions.

Throughout this paper, we define the following notations: and

Definition 2.1

A tuple is called a saddle point of the Lagrangian function (2) if it satisfies the inequalities

Solving problem (1) is equivalent to finding a saddle point of [26, 27]. Therefore, to solve (1), we only need to solve the two inequalities in (7), which can be written as the following mixed variational inequality: where Because is a linear mapping with skew-symmetric coefficient matrix, it satisfies the following property: The mixed variational inequality (8) is denoted by , whose solution set is denoted by , which is nonempty from the assumption on problem (1).

To solve , He et al. [28] presented the following prototype algorithm:

A prototype algorithm for , denoted by ProAlo:

Prediction: For given , find and Q satisfying where the matrix Q has the property: is positive definite.

Correction: Determine a nonsingular matrix M, a scalar , and generate the new iterate via

Condition 2.1

The matrices Q, M in the ProAlo satisfy that the three matrices , , are positive definite.

Under Condition 2.1, He et al. [28] established the convergence results of ProAlo, including the global convergence, the worst-case convergence rate in ergodic or nonergodic sense, where t is the iteration counter. See Theorems 3.3, 4.2, 4.5 in [28].

To end this section, we give the following lemma which will be used in the subsequent section.

Lemma 2.1

([27])

Let be a closed nonempty convex set, and be two convex functions. If the function is nondifferentiable, the function is differentiable, and the solution set of the problem is nonempty, then if and only if

Algorithm and its convergence

In this section, we give the process of the modified hybrid decomposition of the augmented Lagrangian method (MHD-ALM) for three-block separable convex programming (1) and establish its convergence results, including global convergence, ergodic convergence rate, nonergodic convergence rate, and refined ergodic convergence rate.

Algorithm: MHD-ALM

Let parameters , , tolerance error . Choose an initial point . Set .

Compute the prediction iterate via

If , then stop; otherwise, go to Step 3.

Generate the new iterate by Replace by k, and go to Step 1.

Remark 3.1

Different from the iterative schemes (5) and (6), the iterative scheme (13) first updates the primal variable and then updates the primal variables , in a parallel manner. Furthermore, the feasible set of the step size α in MHD-ALM is extended from in [12], in [13], in [25] to .

The methods in [12, 13, 24–26] and MHD-ALM all fall into the algorithmic framework of prediction-correction methods. The main differences among these methods are: (i) in the prediction step, the methods in [24, 26] update all the primal variables in a sequential order; the method in [12] updates all the primal variables in a parallel manner; the methods in [13, 25] and MHD-ALM update all the primal variables in a partial parallel manner, i.e., they first update and then update , in a parallel manner; (ii) in the correction step, the method in [13] updates , λ; the method in [26] and MHD-ALM update , , λ, and the methods in [12, 24, 25] update all the variables.

The convergence analysis of MHD-ALM needs the following assumption and auxiliary sequence.

Assumption 3.1

The matrices , in problem (1) are both full column rank.

Define an auxiliary sequence as

To prove the convergence results of MHD-ALM, we only need cast it into the ProAlo and ensure the following two conditions hold: (i) the generated sequence satisfying (11), (12); (ii) the resulting matrices Q, M satisfying Condition 2.1 in Sect. 2. We first verify the first condition. Based on Lemma 2.1, we can derive the first order optimality conditions of the subproblems in (13), which are summarized in the following lemma.

Lemma 3.1

Let be the sequence generated by MHD-ALM and be defined as in (15). Then it holds that where the matrix Q is defined by

Proof

Based on Lemma 2.1 and using the notation of in (15), the first order optimality conditions for the three minimization problems in (13) can be summarized as the following inequalities: Furthermore, the definition of the variable in (15) gives Adding the above four inequalities, rearranging terms, and using the definition of the matrix Q, the function , we can get the result (16). This completes the proof. □

Remark 3.2

When , by (15), we get , , . Thus, . This and inequality (16) indicate that Therefore, , and the stopping criterion of MHD-ALM is reasonable.

By the definition of in (15), the updating formula of can be represented as This together with (14), (15) gives where the matrix M is defined as

Now to establish the convergence results of MHD-ALM, we only need to verify that the matrices Q, M satisfy Condition 2.1 in Sect. 2.

Lemma 3.2

Let the matrices Q, M be defined as in (17) and (19). If and Assumption 3.1 hold, then we have

the symmetric matrix is positive definite;

the matrix is symmetric and positive definite;

the matrix is symmetric and positive definite.

(i) From the definition of Q, we have Therefore, for any , we have

If , then , so by (20), we get .

If , then from (20) we get where the first inequality follows from the inequality , and the second inequality comes from Assumption 3.1.

(ii) From the definition of Q, M, we have which is obviously positive definite by Assumption 3.1.

(iii) Similarly, from the definition of Q, M, we have where This together with Assumption 3.1 implies that we only need to prove the matrix is positive definite. In fact, it can be written as where ⊗ denotes the matrix Kronecker product. Thus, we only need to prove the 3 order matrix is positive definite, whose three eigenvalues are , , . Then, solving the following three inequalities simultaneously we get . Therefore, the matrix G is positive definite for any . This completes the proof. □

Lemma 3.2 indicates that the matrices Q, M defined as in (17) and (19) satisfy Condition 2.1 in Sect. 2, and thus we get the following convergence results of MHD-ALM based on Theorems 3.3, 4.2, 4.5 in [28].

Theorem 3.1

(Global convergence)

Let be the sequence generated by MHD-ALM. Then it converges to a vector , which belongs to .

Theorem 3.2

(Ergodic convergence rate)

Let be the sequence generated by MHD-ALM, be the corresponding sequence defined in (15). Set Then, for any integer , we have

Theorem 3.3

(Nonergodic convergence rate)

Let be the sequence generated by MHD-ALM. Then, for any and integer , we have where is a constant.

The term on the right-hand side of (21) is used to measure the ergodic convergence rate of MHD-ALM. However, it is not only independent of the distance between the initial iterate and the solution set but also hard to estimate due to the variable v. Therefore, inequality (21) is not a reasonable criterion to measure the nonergodic convergence rate of MHD-ALM. In the following, we shall give a refined result from the objective function and constraint condition of problem (1), which is more reasonable, accurate, and intuitive.

Lemma 3.3

Let be the sequence generated by MHD-ALM. Then, for any , we have

The proof is similar to that of Lemma 3.1 in [28] and is omitted for brevity of this paper. This completes the proof. □

Theorem 3.4

(Refined ergodic convergence rate)

Let be the sequence generated by MHD-ALM, be the sequence defined in (15). Set Then, for any integer , there exists a constant such that

Choose . Then, for any , we have . From the definition of in (9), we have where the first equation follows from (10). Setting in (22), we get Combining the above two inequalities gives Summing the above inequality from to yields Dividing both sides of the above inequality by t, we get Then it follows from the convexity of () that where , . Since (23) holds for any λ, we can set and consequently, Set and we thus get Since (here denotes the solution set of problem (1)), we have Combining the above two inequalities gives which completes the proof. □

As mentioned in Sect. 1, He et al. [12] used a simple example to show that the iterative scheme (6) may diverge for two-block separable convex programming. If we set , in (1) and MHD-ALM, then MHD-ALM reduces to the method in [12]. In this case, the feasible set of α in [12] is (0,0.3670), the same as that of the method in [25] for three-block separable convex programming. Now we use this example given in [12] to show that: (i) larger values of can enhance the performance of MHD-ALM; (ii) MHD-ALM with may diverge.

Example 3.1

Consider the linear equation

Obviously, the linear equation (24) is a special case of problem (1) with the specifications: , , , , . Due to , , in the following we do not consider the variable . The solution set of the corresponding mixed variation inequalities is For MHD-ALM, we set , the initial point , , and choose The stopping criterion is set as or the number of iterations exceeds 10,000.

The numerical results are graphically shown in Fig. 1, which illustrates that when , the number of iterations is descending with respect to α, while when , the number of iterations increases quickly. Therefore, is optimal for this problem, and larger values of α in its feasible set indeed can enhance the numerical performance of MHD-ALM. Of course, some extreme values, such as the values near the upper bound 0.5858, are not appropriate choices.

Figure 1

Sensitivity test on the step size α

Now, we show that MHD-ALM may diverge for . By some simple manipulations, the iterative scheme of (13) and (14) for problem (24) can be written in the following compact form: where Three eigenvalues of the matrix are Now let us consider the following two cases:

(1) For any , we have Then for , where is the spectral radius of . Hence, the iterative scheme (25) with is divergent for this problem.

(2) For , by eigenvalue decomposition, the matrix can be decomposed as where Thus, by (25), we get from which it holds that Hence, the iterative scheme (25) with is also divergent for this problem. Overall, is the optimal upper bound of the step size α in MHD-ALM.

Now let us consider some special cases and extensions of MHD-ALM:

(1) Problem (1) with , reduces to two-block separable convex programming, which can be solved by MHD-ALM as follows: and Since the iterative scheme (26), (27) is a special case of MHD-ALM, it is convergent for any . Furthermore, by Example 3.1, is the optimal upper bound of the constant step size α in (26), (27).

(2) Similarly, problem (1) with , also reduces to two-block separable convex programming, which can be solved by MHD-ALM as follows: and Following a similar analysis procedure, we can prove that the iterative scheme (28), (29) is convergent for any .

(3) Extending MHD-ALM to solve four-block separable convex programming: we can get the following iterative scheme: and With similar reasoning, we can prove that the iterative scheme (30), (31) is convergent for any .

Numerical results

In this section, we demonstrate the practical efficiency of MHD-ALM by applying it to recover low-rank and sparse components of matrices from incomplete and noisy observation. Furthermore, to give more insight into the behavior of MHD-ALM, we compare it with the full Jacobian decomposition of the augmented Lagrangian method (FJD-ALM) [12] and the proximal partially parallel splitting method with constant step size (PPPSM) [25]. All experiments are performed on a Pentium(R) Dual-Core CPU T4400@2.2 GHz PC with 4 GB of RAM running on 64-bit Windows operating system.

The mathematical model of recovering low-rank and sparse components of matrices from incomplete and noisy observation is [20] where is a given matrix, is a balancing parameter, is a penalty parameter, is the index set of the observable entries of D, and is the projection operator defined by Problem (32) is a concrete model of the generic problem (1), and MHD-ALM is applicable. For this problem, the three minimization problems in (13) all admit closed-form solutions, which can be found in [20].

Simulation example

We generate the synthetic data of (32) in the same way as [5, 20]. Specifically, let , be the low-rank matrix, the sparse matrix, respectively, and rr, spr, and sr represent the ratios of the low-rank ratio of (i.e., ), the number of nonzero entries of (i.e., ), and the observed entries (i.e., ), respectively. The observed part of the matrix D is generated by the following Matlab scripts, in which b is the vectorization of D:

In this experiment, we set , , , the initial iterate , and use the stopping criterion or the number of iterations exceeds 500.

The parameters in the three tested methods are listed as follows:

FJD-ALM: .

PPPSM: , , , .

MHD-ALM: .

In Tables 1 and 2, we report the numerical results of three tested methods, in which the number of iterations (denoted by ‘Iter.’), the elapsed CPU time in seconds (denoted by ‘Time’), the relative error of the recovered low-rank matrix, and the relative error of the recovered sparse matrix are reported when the stopping criterion (33) is satisfied.

Table 1

Numerical comparisons between different algorithms for

rr	spr	sr	Method	Iter.	Time	∥L^k−L^∗∥∥L^∗∥	∥S^k−S^∗∥∥S^∗∥
0.05	0.05	0.9	FJD-ALM	97	7.42	5.7325e−04	9.7830e−05
			PPPSM	133	14.70	2.4034e−04	3.6358e−05
			MHD-ALM	44	5.17	7.5910e−04	1.5025e−04
0.05	0.05	0.6	FJD-ALM	120	11.03	1.6528e−03	1.3102e−04
			PPPSM	181	17.62	5.4329e−04	5.1513e−05
			MHD-ALM	75	6.76	1.7148e−03	1.1343e−04
0.1	0.1	0.9	FJD-ALM	127	11.39	2.2566e−03	2.2374e−04
			PPPSM	175	17.79	5.2215e−04	6.1755e−05
			MHD-ALM	80	6.86	1.5097e−03	1.5994e−04
0.1	0.1	0.8	FJD-ALM	188	17.45	4.2439e−03	2.7536e−04
			PPPSM	222	20.78	1.2697e−03	9.5339e−05
			MHD-ALM	119	10.15	2.8338e−03	1.9242e−04
0.1	0.15	0.9	FJD-ALM	209	20.18	3.9031e−03	2.3423e−04
			PPPSM	222	25.95	1.3444e−03	9.5133e−05
			MHD-ALM	120	12.37	3.2420e−03	2.0704e−04

Table 2

Numerical comparisons between different algorithms for

rr	spr	sr	Method	Iter.	Time	∥L^k−L^∗∥∥L^∗∥	∥S^k−S^∗∥∥S^∗∥
0.05	0.05	0.9	FJD-ALM	105	93.16	2.6808e−04	6.1154e−05
			PPPSM	129	131.08	9.9216e−05	2.3683e−05
			MHD-ALM	46	46.82	3.5053e−04	9.5144e−05
0.05	0.05	0.6	FJD-ALM	139	117.67	7.1439e−04	6.8845e−05
			PPPSM	147	138.08	2.1260e−04	3.4427e−05
			MHD-ALM	57	53.31	5.9883e−04	9.8047e−05
0.1	0.1	0.9	FJD-ALM	119	100.13	9.3082e−04	1.6381e−04
			PPPSM	188	179.91	2.3296e−04	4.2820e−05
			MHD-ALM	61	57.04	8.9591e−04	1.6198e−04
0.1	0.1	0.8	FJD-ALM	133	114.80	1.2760e−03	1.8083e−04
			PPPSM	192	182.46	3.7423e−04	5.2624e−05
			MHD-ALM	76	67.08	1.3497e−03	1.6979e−04
0.1	0.15	0.9	FJD-ALM	139	117.89	1.9819e−03	2.3040e−04
			PPPSM	206	188.47	4.4773e−04	5.7452e−05
			MHD-ALM	84	76.34	1.6396e−03	1.8795e−04

Numerical results in Tables 1 and 2 indicate that: (i) all methods successfully solved all the tested cases; (ii) both MHD-ALM and PPPSM perform better than FJD-ALM, and MHD-ALM performed the best. The reason maybe that FJD-ALM updates all the primal variables in a parallel manner, while PPPSM and MHD-ALM update , based on the newest updated to accelerate the convergence speed. Furthermore, the step size α of MHD-ALM is larger than that of PPPSM, and the latter is larger than that of FJD-ALM. Therefore, larger values of α can enhance the efficiency of the corresponding method.

Application example

In this subsection, we apply the proposed method to solve the video background extraction problem with missing and noisy data [29]. There is a video taken in an airport, which consists of 200 grayscale frames with each frame having pixels. We need to separate its background and foreground. Vectorizing all frames of the video, we get a matrix , and each column represents a frame. Let be the matrix representations of its background and foreground (i.e., the moving objects), respectively. Then the rank of L is equal to one exactly, and S should be sparse with only a small number of nonzero elements. We consider only a fraction entries of D can be observed, whose indices are collected in the index set Ω. Then the background extraction problem with missing and noisy data can be casted as problem (32). In the experiment, the parameters in MHD-ALM are set as , , the parameters in (32) are set as , , and the initial iterate . We use the same stopping criterion as (33) with the tolerance 10−2.

Figure 2 displays the separation results of the 10th and 125th frames of the video with , which indicate that the proposed MHD-ALM successfully separates the background and foreground of the two frames.

Figure 2

The 10th and 125th frames of the clean video and the corresponding corrupted frames with (the top and third lines); the extracted background and foreground by MHD-ALM (the second and fourth lines)

Conclusion

In this paper, a hybrid decomposition of the augmented Lagrangian method is proposed for three-block separable convex programming, whose most important characteristic is that its correction step adopts a constant step size. We showed that the optimal upper bound of the constant step size is . Preliminary numerical results indicate that the proposed method is more efficient than similar methods in the literature.

The following two issues deserve further researching: (i) Due to Condition 2.1 being only a sufficient condition to ensure the convergence of the ProAlo, is 1 the optimal upper bound of α in the iterative scheme (28), (29)? Similarly, is the optimal upper bound of α in the iterative scheme (30), (31)? (ii) If we choose different step sizes for , , λ in the correction step of MHD-ALM, the feasible set of these step sizes needs more discussion.