A hybrid conjugate gradient algorithm for constrained monotone equations with application in compressive sensing.

Combining the projection method of Solodov and Svaiter with the Liu-Storey and Fletcher Reeves conjugate gradient algorithm of Djordjević for unconstrained minimization problems, a hybrid conjugate gradient algorithm is proposed and extended to solve convex constrained nonlinear monotone equations. Under some suitable conditions, the global convergence result of the proposed method is established. Furthermore, the proposed method is applied to solve the ℓ 1 -norm regularized problems to restore sparse signal and image in compressive sensing. Numerical comparisons of the proposed algorithm versus some other conjugate gradient algorithms on a set of benchmark test problems, sparse signal reconstruction and image restoration in compressive sensing show that the proposed scheme is computationally more efficient and robust than the compared schemes.

Introduction

Let C be a non-empty, closed and convex subset of and be a continuous and monotone mapping. By monotonicity, it means for all , the function satisfies In this paper, we are interested in finding solution of the nonlinear monotone equations with convex constraints of the form It has been found that various problems with vast applications in interdisciplinary areas can be elegantly modeled using (2). For instance, the power flow equations [1], compressive sensing [2], the economic equilibrium problem [3]. To this effect, researchers have focused on numerical methods for solving (2). Several algorithms have been proposed for solving (2). For example, Newton method, Quasi-Newton method, Levenberge Marquardt method and a series of their variants, see [4], [5], [6], [7], [8] for an overview of these results. It is not surprising that the mentioned methods are attractive, this is due to their rapid convergence from a sufficiently good initial guess. However, they are not suitable for handling large scale nonlinear systems of equations because at each iteration, the Jacobian matrix or its approximation is needed.

In recent times, influenced by the projection method developed by Solodov and Svaiter [9], some of the first-order optimization methods such as the conjugate gradient (CG) method which are well known for solving large-scale unconstrained optimization problems and characterized by their simplicity and low storage have been extended by several researchers to solve (2) (see [2], [10], [11], [12], [13], [14]). Most of these extensions and newly developed methods are variants of the well known CG-method which is one of the foremost vital numerical methods for unconstrained optimization. Some of the earliest conjugate gradient algorithms includes the Fletcher-Reeves (FR) method [15], the Polak Ribiére-Polyak (PRP) method [16], [17], the Hestenes-Steifel (HS) method [18], the Liu-Storey (LS) [19] method, Dai-Yuan (DY) method [20].

Motivated by the good practical behavior of the LS method and strong convergence of the FR method, Djordjević [21] proposed a hybrid conjugate gradient algorithm for solving unconstrained minimization problem. Numerical experiment indicates that the proposed algorithm is efficient and superior to other conjugate gradient algorithms such as the conjugate gradient descent algorithm which is often referred to as CG_DESCENT [2]. Recently, the CG_DESCENT was extended to solve large-scale nonlinear convex constraint monotone equations by Xiao and Zhu [2]. The method was shown to be efficient in solving monotone equations arising from compressive sensing. Can the method of Djordjević be extended to solve constrained monotone equations inheriting the good practical behavior of the LS method and strong convergence of the FR method? Also, how about the computational performance of the method? The focus of this article is to give a positive answer to these questions.

The main contribution of this paper is to propose, analyse and test a hybrid conjugate gradient algorithm combined with the projection technique of Solodov and Svaiter to solve problem (2). Furthermore, with the reformulation of the -norm problem as a non-smooth monotone equation [22], the proposed algorithm is used to solve sparse signal and image restoration problem. In addition, we show that the proposed algorithm exhibit some appealing properties. For instance, the algorithm exhibit less number of iterations and function evaluations. Under some mild assumptions, the global convergence of the algorithm is established. Numerical experiment indicates that the proposed algorithm is efficient, robust and competitive.

This paper is structured as follows: In section 2, we recall some preliminaries. Next, we give the description of our proposed algorithm. Analysis of its global convergence is given in section 3. Numerical results obtained from testing the new method to solve some benchmark test problems are reported in section 4. Finally, we end this paper with section 6 where we demonstrate application of the proposed method in recovery sparse signal and image restoration.

Preliminaries and algorithm

Given an initial point , an iterative scheme for (2) generally generate a sequence of iterates by where is the step length which is computed by a certain line search and is the search direction usually satisfying with positive constant c. If G is the gradient of a real-valued function , the descent condition means that is a direction of sufficient descent g at . For convenience, we abbreviate as .

To describe our algorithm, we recall the projection map denoted as , which is a mapping from onto the nonempty convex set C, that is which has the well known nonexpansive property In what follows, we assume that G satisfies the following assumptions.

The mapping G is Lipschitz continuous on . That is,

The solution set is nonempty.

In this paper, we propose the search direction based on the method proposed in [21]. Specifically, is determined by where and parameter computed as a convex combination of LS and FR methods. That is, Substituting (8) into (6), we have We select such that the search direction satisfies the famous conjugacy condition. That is, We have rearranging gives, where .

Next, we formally present a hybrid conjugate gradient algorithm as a convex combination of LS and FR method for solving (2). For simplicity, we refer to this algorithm as HLSFR algorithm.

(HLSFR)

Input. Choose any random point , the positive constants: , , . Set .

Step 0. Compute . If , stop. Otherwise, compute the search direction by If , then compute using (8)

If , then compute

If , then compute

If , we set .

Step 1. Let be determined by the following line-search Step 2. Compute the trial point Step 3. If and , stop. Otherwise, compute where Step 4. Set and go to step 1.

It is clear to see that the we propose is similar to that proposed in [21] but with different definition on and . The search direction defined by (6) originated in [23] which was originally used in solving unconstrained optimization problem. Here, the method is extended to solve nonlinear monotone equations with convex constraints.

Let be the search direction generated by (6). Then, always satisfies the sufficient decent condition (4). That is, for all .

From the definition of (6), it is easy to see that (15) holds. □

Convergence analysis

The line search is well defined. That is, for all , there exists a non negative integer m satisfying (12).

We begin by contradiction. Suppose there exist such that (12) is not satisfied for any nonnegative integer m, that is Using the continuity of G and letting yields which contradicts (15). This completes the proof. □

The HLSFR algorithm is well defined.

The first step is to notice that, from the line search (12), if , then does not satisfy (12), that is, Equation (16) combined with (15), we have Since G is a Lipschitz continuous function, then the above inequality is valid. Thus, from (17), This proves Lemma 3.2. □

Suppose G is monotone and Lipchitz continuous on and the sequence is generated by (14) in HLSFR algorithm, then there exists such that

Recall that, from the nonexpansiveness of the projection operator, it holds that for any , The above inequality (21) implies that the sequence is a decreasing sequence. Therefore, the sequence is bounded, that is In addition, we obtain Using the Lipchitz continuity of G, we have Setting proves Lemma 3.3. □

Let and be sequences generated by (13) and (14) respectively under Assumption 1, Assumption 2 using HLSFR Algorithm, then is a descent direction of the function at the point where .

At , the function has a gradient of . By monotonicity property (1), it can be seen that which indicates that the function has a descent direction at the iteration point . □

Suppose Assumption 1, Assumption 2 hold and the sequence and are generated by (13) and (14) respectively in HLSFR algorithm. Then,

is bounded

.

The following theorem establishes the global convergence of HLSFR algorithm.

From (23), we know that the sequence is bounded. From (25), we have Utilizing (19) and (1), we have Combined with (26), it is easy to deduce that Then, we obtain, Hence the sequence is bounded owing to the boundedness of .

From inequality (20), we get which means From the fact that the function G is continuous and the sequence is bounded, we know that the sequence is bounded. Hence, there exist a positive such that and furthermore Hence,

From the nonexpansiveness of the projection operator, we have □

Suppose conditions of Assumption 1, Assumption 2 hold. Then, the sequence generated by (14) in HLSFR method converges globally to a solution of (2).

Suppose (28) does not hold, meaning there exists a constant such that By (15), we know which implies By (6), we have for all . Since (27) hold, it follows that for every there exist such that for every . Choosing and where , it holds that for every . Integrating with (18), (29), (30) and (31), we know that for any k sufficiently large The last inequality yields a contradiction with in Lemma 3.5. Consequently, (28) holds. The proof is completed. □

Numerical results

We present a detail report of the numerical experiment in testing the performance of HLSFR. We compared HLSFR with the CGD, PCG, PDY and ACGD methods in [2], [13], [24], [25] respectively. The mapping G is taken as where the associated initial points for these problems are We made use of the following benchmark test problems in testing the effectiveness and robustness of the methods.

This problem is the Exponential function [26] with constraint set , that is,

Modified Logarithmic function [26] with constraint set , that is,

The Nonsmooth Function [27] with constraint set .

The Strictly convex function [28], with constraint set , that is,

Tridiagonal Exponential function [29] with constraint set , that is,

Nonsmooth function [30] with constraint set .

The Trig exp function [26] with constraint set , that is,

The Penalty 1 function [25] with , that is,

The function with defined by,

The function with defined,

The codes for all methods were written on a windows 10 HP personal computer with 2.40 GHz processor, 8 GB RAM of Intel(R) Core (TM) i3-7100U using Matlab R2019b software. We choose the following parameters: , , in implementing HLSFR. For the other methods, their parameters were set as in their respective papers. Furthermore, for each test problem, the iterations are terminated when the inequality is satisfied. Failure is declared if the inequality is not satisfied after 1000 iterations. A comprehensive results of our numerical experiment are presented in the appendix section. The columns of the presented tables have the following definitions:

IP: denotes the initial points

DIM: denotes the dimension of the problem

NI: represents iterative number

NF: denotes iterative number of function evaluation.

CPU: denotes the CPU time in seconds when the algorithms terminate

NORM: denotes the final norm equation

From Tables 3, 4, 5, 6, 7, 8, 9, 10, 11 and 12, it is not difficult to see that all methods solved all the test problems successfully. However, the HLSFR method highly performs better compared with CGD, PCG, PDY and ACGD in terms of the iteration number and the number of function evaluations.

Table 3

Numerical results for Problem 1.

DIM	IP	HLSFR				CGD				PCG				PDY				ACGD
		NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM
1000	x1	1	3	1.0823	0.00E+00	50	149	0.04673	9.11E-07	19	75	0.092184	4.72E-06	18	72	0.58052	4.26E-07	17	67	0.005714	4.53E-06
	x2	1	3	0.012591	0.00E+00	42	125	0.021608	9.97E-07	18	71	0.035649	5.72E-06	16	64	0.027927	3.45E-07	8	31	0.006222	9.26E-06
	x3	2	6	0.00375	6.28E-16	58	173	0.022573	8.22E-07	30	120	0.04483	8.59E-06	17	68	0.033385	4.70E-07	18	72	0.010303	5.80E-06
	x4	7	21	0.022676	4.00E-07	52	155	0.042289	8.48E-07	19	75	0.013987	5.82E-06	17	68	0.022742	7.21E-07	17	67	0.007245	4.37E-06
	x5	7	21	0.047679	5.23E-07	35	104	0.01894	6.10E-07	18	71	0.022429	9.14E-06	17	68	0.079305	7.15E-07	16	63	0.009693	9.82E-06
	x6	2	6	0.002211	0.00E+00	59	176	0.019966	9.55E-07	34	135	0.027208	7.37E-06	17	68	0.018899	6.95E-07	9	36	0.004608	3.90E-06
	x7	7	21	0.010068	4.00E-07	52	155	0.020642	8.48E-07	19	75	0.015801	5.82E-06	17	68	0.017112	7.21E-07	17	67	0.011732	4.37E-06
	x8	7	21	0.009221	6.60E-07	35	104	0.017981	6.11E-07	18	71	0.014044	9.15E-06	17	68	0.01607	7.16E-07	16	63	0.011277	9.83E-06
	x9	7	21	0.007807	2.16E-07	51	152	0.067866	8.00E-07	18	71	0.007959	9.75E-06	17	68	0.016342	7.32E-07	16	63	0.009217	9.79E-06
5000	x1	1	3	0.004128	0.00E+00	48	143	0.35099	9.34E-07	20	79	0.037975	4.70E-06	18	72	0.091098	9.40E-07	18	71	0.034532	4.03E-06
	x2	1	3	0.063436	0.00E+00	41	122	0.50749	8.34E-07	18	71	0.10733	7.42E-06	16	64	0.051997	7.61E-07	9	36	0.021112	3.89E-06
	x3	2	6	0.012812	6.28E-16	58	173	0.061541	8.22E-07	30	120	0.070734	8.59E-06	17	68	0.061095	4.70E-07	18	72	0.044432	5.80E-06
	x4	8	24	0.016178	5.00E-08	50	149	0.060155	8.63E-07	19	75	0.030924	9.92E-06	18	72	0.068399	5.36E-07	17	67	0.029379	8.99E-06
	x5	8	24	0.021606	5.21E-08	36	107	0.042146	8.19E-07	19	75	0.03437	9.15E-06	18	72	0.062534	5.35E-07	17	67	0.031705	8.79E-06
	x6	2	6	0.006381	0.00E+00	59	176	0.073312	9.55E-07	34	135	0.066134	7.37E-06	17	68	0.05826	6.95E-07	9	36	0.18994	3.90E-06
	x7	8	24	0.023883	5.00E-08	50	149	0.072813	8.63E-07	19	75	0.039391	9.92E-06	18	72	0.064131	5.36E-07	17	67	0.035233	8.99E-06
	x8	8	24	0.025457	5.63E-08	36	107	0.075877	8.19E-07	19	75	0.041779	9.15E-06	18	72	0.058588	5.35E-07	17	67	0.035003	8.80E-06
	x9	7	21	0.025834	5.30E-08	49	146	0.072482	9.52E-07	19	75	0.034482	9.66E-06	18	72	0.062149	5.35E-07	17	67	0.030316	8.81E-06
10000	x1	1	3	0.016629	0.00E+00	47	140	0.10544	9.79E-07	20	79	0.055695	6.64E-06	19	76	0.12181	4.44E-07	18	71	0.056367	5.70E-06
	x2	1	3	0.031943	0.00E+00	40	119	0.080776	8.97E-07	18	71	0.04976	9.50E-06	17	68	0.11546	3.55E-07	9	35	0.051461	5.50E-06
	x3	2	6	0.089531	6.28E-16	58	173	0.1461	8.22E-07	30	120	0.11683	8.59E-06	17	68	0.12369	4.70E-07	18	72	0.058851	5.80E-06
	x4	8	24	0.042885	7.27E-08	49	146	0.1106	8.99E-07	20	79	0.09813	6.17E-06	18	72	0.11517	7.57E-07	18	71	0.058665	5.06E-06
	x5	8	24	0.057914	7.41E-08	37	110	0.072591	6.95E-07	20	79	0.06743	5.79E-06	18	72	0.10486	7.56E-07	18	71	0.055802	4.98E-06
	x6	2	6	0.008709	0.00E+00	59	176	0.14788	9.55E-07	34	135	0.10822	7.37E-06	17	68	0.11873	6.95E-07	9	36	0.044591	3.90E-06
	x7	8	24	0.047393	7.27E-08	49	146	0.10977	8.99E-07	20	79	0.057499	6.17E-06	18	72	0.098904	7.57E-07	18	71	0.055068	5.06E-06
	x8	8	24	0.038672	7.72E-08	37	110	0.076211	6.95E-07	20	79	0.063101	5.79E-06	18	72	0.12217	7.57E-07	18	71	0.060447	4.98E-06
	x9	8	24	0.052426	1.55E-07	47	140	0.20135	8.12E-07	20	79	0.072131	6.16E-06	18	72	0.14623	7.59E-07	18	71	0.061031	4.97E-06
50000	x1	1	3	0.027125	0.00E+00	46	137	0.5439	8.30E-07	21	83	0.2535	6.64E-06	20	80	0.43976	8.84E-07	19	75	0.23021	5.10E-06
	x2	1	3	0.023587	0.00E+00	39	116	0.41363	8.43E-07	19	75	0.23779	8.80E-06	17	68	0.33496	7.93E-07	10	40	0.11264	2.33E-06
	x3	2	6	0.044199	6.28E-16	58	173	0.49551	8.22E-07	30	120	0.46619	8.59E-06	17	68	0.40581	4.70E-07	18	72	0.16705	5.80E-06
	x4	8	24	0.11349	1.68E-07	47	140	0.43289	9.33E-07	21	83	0.261	5.91E-06	19	76	0.41494	5.63E-07	19	76	0.21902	4.48E-06
	x5	8	24	0.091825	1.68E-07	38	113	0.31557	9.33E-07	21	83	0.19767	5.79E-06	19	76	0.38138	5.62E-07	19	76	0.19472	4.46E-06
	x6	2	6	0.038903	0.00E+00	59	176	0.46931	9.55E-07	34	135	0.34305	7.37E-06	17	68	0.33145	6.95E-07	9	36	0.088923	3.90E-06
	x7	8	24	0.11501	1.68E-07	47	140	0.69908	9.33E-07	21	83	0.24347	5.91E-06	19	76	0.40886	5.63E-07	19	76	0.18883	4.48E-06
	x8	8	24	0.21179	1.70E-07	38	113	0.59588	9.33E-07	21	83	0.20193	5.79E-06	19	76	0.40335	5.63E-07	19	76	0.18902	4.46E-06
	x9	8	24	0.11512	4.27E-08	39	116	0.3637	6.65E-07	21	83	0.28083	5.79E-06	19	76	0.52614	5.65E-07	19	76	0.19398	4.46E-06
100000	x1	1	3	0.034912	0.00E+00	45	134	0.77402	9.06E-07	21	83	0.42087	9.39E-06	21	84	0.8809	6.05E-07	19	75	0.46145	7.21E-06
	x2	1	3	0.042479	0.00E+00	39	116	0.59479	7.72E-07	20	79	0.36671	5.52E-06	18	72	0.67054	3.76E-07	10	40	0.24162	3.29E-06
	x3	2	6	0.059236	6.28E-16	58	173	0.76354	8.22E-07	30	120	0.65474	8.59E-06	17	68	0.62133	4.70E-07	18	72	0.43411	5.80E-06
	x4	8	24	0.20832	2.38E-07	46	137	0.64004	9.92E-07	21	83	0.37783	8.27E-06	20	80	0.92749	7.77E-07	19	76	0.37867	6.32E-06
	x5	8	24	0.23556	2.38E-07	39	116	0.65969	7.91E-07	21	83	0.41069	8.19E-06	20	80	0.79866	7.77E-07	19	76	0.36352	6.30E-06
	x6	2	6	0.06482	0.00E+00	59	176	0.80171	9.55E-07	34	135	0.61319	7.37E-06	17	68	0.96643	6.95E-07	9	36	0.21668	3.90E-06
	x7	8	24	0.19377	2.38E-07	46	137	0.71526	9.92E-07	21	83	0.39291	8.27E-06	20	80	0.98442	7.77E-07	19	76	0.41014	6.32E-06
	x8	8	24	0.19487	2.39E-07	39	116	0.57312	7.91E-07	21	83	0.39435	8.19E-06	20	80	0.78821	7.77E-07	19	76	0.39413	6.30E-06
	x9	8	24	0.30738	5.93E-08	41	122	0.71712	9.52E-07	21	83	0.43257	8.21E-06	20	80	0.86416	7.79E-07	19	76	0.42681	6.30E-06

Table 4

Numerical results for Problem 2.

DIM	IP	HLSFR				CGD				PCG				PDY				ACGD
		NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM
1000	x1	8	23	0.018212	2.80E-07	75	223	0.80072	8.04E-07	16	60	0.024105	8.69E-06	17	67	0.014428	7.80E-07	5	14	0.003424	3.60E-08
	x2	7	20	0.010173	7.80E-07	55	163	0.054278	8.99E-07	15	58	0.008816	8.59E-06	13	51	0.014789	7.68E-07	3	8	0.004621	5.17E-07
	x3	14	41	0.026389	8.16E-07	56	166	0.049743	8.94E-07	16	62	0.023068	5.23E-06	14	55	0.008207	3.79E-07	427	1706	0.28339	9.99E-06
	x4	17	50	0.020437	6.68E-07	72	214	0.036264	8.60E-07	19	73	0.015537	6.23E-06	17	67	0.015355	4.48E-07	47	184	0.039464	9.78E-06
	x5	17	50	0.050271	6.68E-07	72	214	0.029064	8.60E-07	19	73	0.013726	6.23E-06	17	67	0.022895	4.48E-07	47	184	0.030139	9.78E-06
	x6	15	43	0.023681	9.98E-07	61	181	0.054219	9.60E-07	18	69	0.013178	6.33E-06	15	59	0.013177	4.21E-07	13	49	0.010699	4.00E-06
	x7	17	50	0.025723	6.68E-07	72	214	0.041652	8.60E-07	19	73	0.01093	6.23E-06	17	67	0.013001	4.48E-07	47	184	0.026531	9.78E-06
	x8	17	50	0.025859	6.68E-07	72	214	0.073851	8.61E-07	19	73	0.011677	6.25E-06	17	67	0.014163	4.48E-07	47	184	0.023996	9.33E-06
	x9	17	50	0.039069	6.39E-07	72	214	0.065443	8.49E-07	19	73	0.011878	6.03E-06	17	67	0.022129	4.48E-07	47	184	0.028836	9.51E-06
5000	x1	8	23	0.032659	6.77E-07	78	232	0.24516	9.05E-07	17	64	0.055484	9.24E-06	18	71	0.069709	5.71E-07	5	14	0.012506	6.26E-09
	x2	8	22	0.037024	7.45E-07	59	175	0.084645	8.02E-07	16	62	0.037537	9.35E-06	14	55	0.042903	5.44E-07	3	8	0.014479	1.75E-07
	x3	14	41	0.050599	8.08E-07	56	166	0.13993	8.87E-07	16	62	0.037155	5.17E-06	14	55	0.040223	3.76E-07	27	106	0.054074	9.55E-06
	x4	18	52	0.050527	9.45E-07	75	223	0.26478	9.71E-07	20	77	0.056123	6.86E-06	17	67	0.075953	9.87E-07	12	44	0.032188	2.83E-06
	x5	18	52	0.14547	9.45E-07	75	223	0.1264	9.71E-07	20	77	0.04529	6.86E-06	17	67	0.064816	9.87E-07	12	44	0.037678	2.83E-06
	x6	15	43	0.050711	9.88E-07	61	181	0.090101	9.52E-07	18	69	0.048865	6.41E-06	15	59	0.052666	4.20E-07	17	65	0.039516	7.74E-06
	x7	18	52	0.069951	9.45E-07	75	223	0.12312	9.71E-07	20	77	0.054758	6.86E-06	17	67	0.078804	9.87E-07	12	44	0.035473	2.83E-06
	x8	18	52	0.11842	9.45E-07	75	223	0.22357	9.72E-07	20	77	0.051195	6.87E-06	17	67	0.052195	9.87E-07	12	44	0.034693	2.83E-06
	x9	18	52	0.10997	9.56E-07	75	223	0.34172	9.71E-07	20	77	0.060697	6.92E-06	17	67	0.05724	9.88E-07	12	44	0.037483	2.82E-06
10000	x1	8	23	0.041183	9.66E-07	80	238	0.26206	8.17E-07	18	68	0.092888	6.50E-06	18	71	0.099253	8.06E-07	5	14	0.020898	3.62E-09
	x2	8	23	0.083979	2.12E-07	60	178	0.16174	9.04E-07	17	66	0.070972	6.60E-06	14	55	0.084051	7.66E-07	3	8	0.015687	1.21E-07
	x3	14	41	0.085061	8.08E-07	56	166	0.13119	8.86E-07	16	62	0.05767	5.17E-06	14	55	0.075646	3.76E-07	35	138	0.13043	9.91E-06
	x4	18	53	0.20063	7.34E-07	77	229	0.22668	8.78E-07	20	77	0.098276	9.69E-06	18	71	0.099754	4.64E-07	12	44	0.052358	3.89E-06
	x5	18	53	0.27686	7.34E-07	77	229	0.2324	8.78E-07	20	77	0.093564	9.69E-06	18	71	0.10881	4.64E-07	12	44	0.064254	3.89E-06
	x6	15	43	0.09867	9.86E-07	61	181	0.16297	9.51E-07	18	69	0.067092	6.42E-06	15	59	0.080194	4.20E-07	17	65	0.087748	8.10E-06
	x7	18	53	0.11891	7.34E-07	77	229	0.34381	8.78E-07	20	77	0.078502	9.69E-06	18	71	0.098208	4.64E-07	12	44	0.054981	3.89E-06
	x8	18	53	0.12076	7.34E-07	77	229	0.36919	8.78E-07	20	77	0.081604	9.69E-06	18	71	0.13537	4.64E-07	12	44	0.074371	3.89E-06
	x9	18	53	0.15044	7.39E-07	77	229	0.35996	8.76E-07	20	77	0.13768	9.68E-06	18	71	0.11891	4.62E-07	12	44	0.057607	3.86E-06
50000	x1	9	25	0.15309	8.71E-07	83	247	1.0802	9.34E-07	19	72	0.3017	7.24E-06	20	80	0.42989	7.70E-07	6	19	0.080077	4.49E-06
	x2	8	23	0.16512	4.78E-07	64	190	0.77041	8.26E-07	18	70	0.29234	7.37E-06	15	59	0.32333	5.78E-07	7	25	0.10219	2.94E-06
	x3	14	41	0.31406	8.07E-07	56	166	0.67991	8.85E-07	16	62	0.28346	5.16E-06	14	55	0.23355	3.75E-07	36	142	0.62329	8.88E-06
	x4	19	56	0.43663	5.77E-07	81	241	1.062	8.03E-07	22	85	0.44496	5.43E-06	19	75	0.39463	3.46E-07	13	47	0.19753	7.28E-06
	x5	19	56	0.38161	5.77E-07	81	241	1.1367	8.03E-07	22	85	0.34541	5.43E-06	19	75	0.48171	3.46E-07	13	47	0.20861	7.28E-06
	x6	15	43	0.30917	9.85E-07	61	181	0.64703	9.50E-07	18	69	0.26254	6.44E-06	15	59	0.27749	4.20E-07	17	65	0.21671	8.33E-06
	x7	19	56	0.3637	5.77E-07	81	241	1.0176	8.03E-07	22	85	0.34772	5.43E-06	19	75	0.38515	3.46E-07	13	47	0.22692	7.28E-06
	x8	19	56	0.36649	5.77E-07	81	241	1.0743	8.03E-07	22	85	0.44399	5.43E-06	19	75	0.46512	3.46E-07	13	47	0.19916	7.28E-06
	x9	19	56	0.63323	5.78E-07	81	241	1.4438	8.06E-07	22	85	0.4979	5.40E-06	19	75	0.38699	3.46E-07	13	47	0.25878	7.27E-06
100000	x1	9	26	0.35624	2.47E-07	85	253	1.9715	8.45E-07	20	76	0.57694	5.13E-06	22	88	1.0154	6.15E-07	6	19	0.15741	6.18E-06
	x2	8	23	0.25119	6.76E-07	65	193	1.4595	9.34E-07	19	74	0.62487	5.22E-06	15	59	0.90318	8.17E-07	7	25	0.24042	4.14E-06
	x3	14	41	0.38415	8.07E-07	56	166	1.1403	8.85E-07	16	62	0.36519	5.16E-06	14	55	0.48722	3.75E-07	36	142	1.1879	9.08E-06
	x4	19	56	0.85542	8.16E-07	82	244	2.0197	9.08E-07	22	85	0.69632	7.67E-06	20	80	1.0938	5.47E-07	13	47	0.42213	8.40E-06
	x5	19	56	0.81838	8.16E-07	82	244	2.2071	9.08E-07	22	85	0.79312	7.67E-06	20	80	0.97473	5.47E-07	13	47	0.41177	8.40E-06
	x6	15	43	0.57124	9.85E-07	61	181	1.3264	9.50E-07	18	69	0.55801	6.44E-06	15	59	0.53508	4.20E-07	17	65	0.51383	8.35E-06
	x7	19	56	0.76892	8.16E-07	82	244	2.1547	9.08E-07	22	85	0.65754	7.67E-06	20	80	0.91264	5.47E-07	13	47	0.39895	8.40E-06
	x8	19	56	0.67393	8.16E-07	82	244	2.1183	9.08E-07	22	85	0.64507	7.67E-06	20	80	0.98593	5.47E-07	13	47	0.41949	8.40E-06
	x9	19	56	0.96709	8.17E-07	82	244	2.5937	9.07E-07	22	85	0.81881	7.66E-06	20	80	0.97391	5.48E-07	13	47	0.53617	8.35E-06

Table 5

Numerical results for Problem 3.

DIM	IP	HLSFR				CGD				PCG				PDY				ACGD
		NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM
1000	x1	14	41	0.0183	4.81E-07	78	233	0.82269	8.87E-07	22	87	0.011525	5.34E-06	17	68	0.011873	7.24E-07	11	43	0.008686	6.30E-06
	x2	12	36	0.023188	7.31E-07	68	203	0.05126	9.14E-07	19	75	0.007828	5.62E-06	15	60	0.013488	4.96E-07	10	39	0.004849	4.44E-06
	x3	F	F	F	F	60	179	0.029411	9.74E-07	16	63	0.008361	7.72E-06	13	52	0.013733	8.83E-07	9	35	0.009361	2.85E-06
	x4	18	54	0.030502	8.81E-07	76	227	0.031992	8.32E-07	21	83	0.013737	6.92E-06	17	68	0.00999	3.88E-07	13	51	0.006173	2.84E-06
	x5	18	54	0.015063	8.81E-07	76	227	0.074951	8.32E-07	21	83	0.012266	6.92E-06	17	68	0.024643	3.88E-07	13	51	0.013775	2.84E-06
	x6	F	F	F	F	64	191	0.04589	8.46E-07	17	67	0.00594	7.68E-06	14	56	0.014141	7.41E-07	F	F	F	F
	x7	18	54	0.030096	8.81E-07	76	227	0.031962	8.32E-07	21	83	0.01521	6.92E-06	17	68	0.013337	3.88E-07	13	51	0.006512	2.84E-06
	x8	18	54	0.033431	8.80E-07	76	227	0.093709	8.33E-07	21	83	0.010497	6.93E-06	17	68	0.012659	3.88E-07	13	51	0.008327	2.85E-06
	x9	18	54	0.020059	8.90E-07	76	227	0.064367	8.34E-07	21	83	0.011911	7.04E-06	17	68	0.013935	3.87E-07	13	51	0.011743	2.79E-06
5000	x1	14	42	0.043669	7.53E-07	82	245	0.20528	8.13E-07	23	91	0.060791	5.98E-06	18	72	0.049102	5.41E-07	12	47	0.026256	3.67E-06
	x2	13	38	0.038964	6.54E-07	72	215	0.169	8.37E-07	20	79	0.038254	6.29E-06	16	64	0.051745	3.74E-07	10	39	0.02449	9.93E-06
	x3	F	F	F	F	60	179	0.092302	9.74E-07	16	63	0.033785	7.72E-06	13	52	0.036974	8.83E-07	9	35	0.024014	2.85E-06
	x4	19	57	0.068222	6.93E-07	79	236	0.13927	9.53E-07	22	87	0.059257	7.76E-06	17	68	0.046376	8.68E-07	13	51	0.029617	6.30E-06
	x5	19	57	0.048611	6.93E-07	79	236	0.14781	9.53E-07	22	87	0.052782	7.76E-06	17	68	0.059893	8.68E-07	13	51	0.027401	6.30E-06
	x6	F	F	F	F	64	191	0.10051	8.46E-07	17	67	0.03014	7.68E-06	14	56	0.04967	7.41E-07	F	F	F	F
	x7	19	57	0.070133	6.93E-07	79	236	0.12815	9.53E-07	22	87	0.046812	7.76E-06	17	68	0.047516	8.68E-07	13	51	0.028182	6.30E-06
	x8	19	57	0.14375	6.93E-07	79	236	0.12428	9.53E-07	22	87	0.064462	7.76E-06	17	68	0.10568	8.68E-07	13	51	0.031773	6.30E-06
	x9	19	57	0.084187	7.01E-07	79	236	0.15892	9.59E-07	22	87	0.064349	7.71E-06	17	68	0.067408	8.70E-07	13	51	0.032564	6.26E-06
10000	x1	15	44	0.13781	4.26E-07	83	248	0.22246	9.20E-07	23	91	0.080213	8.46E-06	18	72	0.093398	7.65E-07	12	47	0.045959	5.18E-06
	x2	13	38	0.061573	9.25E-07	73	218	0.26335	9.47E-07	20	79	0.068726	8.90E-06	16	64	0.070683	5.28E-07	11	43	0.035235	3.65E-06
	x3	F	F	F	F	60	179	0.20574	9.74E-07	16	63	0.086356	7.72E-06	13	52	0.059207	8.83E-07	9	35	0.031756	2.85E-06
	x4	19	57	0.12639	9.80E-07	81	242	0.30108	8.62E-07	23	91	0.083338	5.50E-06	18	72	0.091757	4.11E-07	13	51	0.050319	8.89E-06
	x5	19	57	0.10872	9.80E-07	81	242	0.21114	8.62E-07	23	91	0.074474	5.50E-06	18	72	0.082871	4.11E-07	13	51	0.046841	8.89E-06
	x6	F	F	F	F	64	191	0.16324	8.46E-07	17	67	0.14347	7.68E-06	14	56	0.068477	7.41E-07	F	F	F	F
	x7	19	57	0.11703	9.80E-07	81	242	0.26074	8.62E-07	23	91	0.075044	5.50E-06	18	72	0.089915	4.11E-07	13	51	0.053132	8.89E-06
	x8	19	57	0.19919	9.80E-07	81	242	0.19701	8.63E-07	23	91	0.072729	5.50E-06	18	72	0.088977	4.11E-07	13	51	0.051074	8.90E-06
	x9	19	57	0.13663	9.88E-07	81	242	0.26997	8.62E-07	23	91	0.094142	5.48E-06	18	72	0.14686	4.14E-07	13	51	0.067577	8.93E-06
50000	x1	15	44	0.23977	9.52E-07	87	260	1.8282	8.42E-07	24	95	0.34206	9.48E-06	20	80	0.30151	5.51E-07	13	51	0.14639	3.01E-06
	x2	14	41	0.1817	5.79E-07	77	230	1.3441	8.67E-07	21	83	0.23777	9.97E-06	17	68	0.24967	3.91E-07	11	43	0.17692	8.17E-06
	x3	F	F	F	F	60	179	1.2927	9.74E-07	16	63	0.18235	7.72E-06	13	52	0.20767	8.83E-07	9	35	0.10176	2.85E-06
	x4	20	60	0.2897	7.71E-07	84	251	1.231	9.87E-07	24	95	0.33937	6.16E-06	18	72	0.29045	9.19E-07	14	55	0.19307	5.17E-06
	x5	20	60	0.2563	7.71E-07	84	251	0.98366	9.87E-07	24	95	0.32036	6.16E-06	18	72	0.37354	9.19E-07	14	55	0.19138	5.17E-06
	x6	F	F	F	F	64	191	1.4508	8.46E-07	17	67	0.23677	7.68E-06	14	56	0.20764	7.41E-07	F	F	F	F
	x7	20	60	0.24518	7.71E-07	84	251	1.1827	9.87E-07	24	95	0.3051	6.16E-06	18	72	0.34183	9.19E-07	14	55	0.19257	5.17E-06
	x8	20	60	0.28419	7.71E-07	84	251	1.0601	9.87E-07	24	95	0.32944	6.16E-06	18	72	0.26908	9.19E-07	14	55	0.19161	5.17E-06
	x9	20	60	0.32687	7.73E-07	84	251	1.5334	9.87E-07	24	95	0.47879	6.15E-06	18	72	0.34374	9.17E-07	14	55	0.25619	5.16E-06
100000	x1	15	45	0.3195	9.43E-07	88	263	3.1639	9.53E-07	25	99	0.68617	6.72E-06	21	84	0.64426	4.91E-07	13	51	0.29855	4.26E-06
	x2	14	41	0.34837	8.19E-07	78	233	1.96	9.81E-07	22	87	0.5382	7.06E-06	17	68	0.52521	5.53E-07	12	47	0.24981	3.00E-06
	x3	F	F	F	F	60	179	1.5726	9.74E-07	16	63	0.46066	7.72E-06	13	52	0.38743	8.83E-07	9	35	0.25983	2.85E-06
	x4	21	62	0.41639	6.98E-07	86	257	2.9241	8.94E-07	24	95	0.59391	8.71E-06	20	80	0.77236	4.62E-07	14	55	0.35879	7.30E-06
	x5	21	62	0.54923	6.98E-07	86	257	2.1442	8.94E-07	24	95	0.53777	8.71E-06	20	80	0.59394	4.62E-07	14	55	0.29892	7.30E-06
	x6	F	F	F	F	64	191	1.9524	8.46E-07	17	67	0.39127	7.68E-06	14	56	0.39452	7.41E-07	F	F	F	F
	x7	21	62	0.41948	6.98E-07	86	257	2.2592	8.94E-07	24	95	0.591	8.71E-06	20	80	0.62608	4.62E-07	14	55	0.39671	7.30E-06
	x8	21	62	0.50253	6.98E-07	86	257	2.0441	8.94E-07	24	95	0.57094	8.71E-06	20	80	0.63396	4.62E-07	14	55	0.2969	7.30E-06
	x9	21	62	0.74426	6.99E-07	86	257	2.8709	8.94E-07	24	95	0.75883	8.69E-06	20	80	0.6459	4.62E-07	14	55	0.52403	7.30E-06

Table 6

Numerical results for Problem 4.

DIM	IP	HLSFR				CGD				PCG				PDY				ACGD
		NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM
1000	x1	1	3	0.006154	0.00E+00	76	227	0.42078	8.61E-07	19	75	0.01192	5.73E-06	16	64	0.0095	8.79E-07	17	67	0.011751	5.39E-06
	x2	1	3	0.001905	0.00E+00	68	203	0.026772	8.60E-07	18	71	0.010833	9.93E-06	15	60	0.010303	5.13E-07	10	39	0.007579	3.65E-06
	x3	1	3	0.001996	2.22E-16	59	176	0.033022	9.62E-07	15	60	0.013106	9.35E-06	13	52	0.009199	8.83E-07	10	40	0.010692	4.91E-06
	x4	16	48	0.016692	8.42E-07	74	221	0.031073	8.94E-07	20	79	0.00808	6.89E-06	17	68	0.012899	4.91E-07	18	71	0.011959	6.51E-06
	x5	16	48	0.023724	8.42E-07	74	221	0.03155	8.94E-07	20	79	0.014524	6.89E-06	17	68	0.012656	4.91E-07	18	71	0.008487	6.51E-06
	x6	15	44	0.017326	7.29E-07	62	185	0.035103	9.27E-07	17	67	0.005744	6.62E-06	15	60	0.012038	4.55E-07	15	59	0.008223	6.75E-06
	x7	16	48	0.020707	8.42E-07	74	221	0.022633	8.94E-07	20	79	0.015181	6.89E-06	17	68	0.009832	4.91E-07	18	71	0.014186	6.51E-06
	x8	17	50	0.032877	6.73E-07	74	221	0.024689	8.94E-07	20	79	0.01043	6.91E-06	17	68	0.010587	4.96E-07	18	71	0.009381	6.53E-06
	x9	15	45	0.022213	9.31E-07	74	221	0.044716	9.00E-07	20	79	0.00766	6.77E-06	17	68	0.007763	5.01E-07	18	71	0.010474	6.54E-06
5000	x1	1	3	0.00449	0.00E+00	79	236	0.093652	9.85E-07	20	79	0.027713	6.42E-06	17	68	0.042218	6.59E-07	18	71	0.02803	5.45E-06
	x2	1	3	0.00811	0.00E+00	71	212	0.11582	9.85E-07	20	79	0.026781	5.57E-06	16	64	0.034413	3.86E-07	10	39	0.01728	8.15E-06
	x3	1	3	0.005698	2.22E-16	59	176	0.13671	9.62E-07	15	60	0.019444	9.35E-06	13	52	0.025878	8.83E-07	10	40	0.024215	4.91E-06
	x4	17	51	0.13449	7.31E-07	78	233	0.10456	8.19E-07	21	83	0.036654	7.73E-06	18	72	0.041715	3.69E-07	19	75	0.036873	6.61E-06
	x5	17	51	0.032044	7.31E-07	78	233	0.11744	8.19E-07	21	83	0.032061	7.73E-06	18	72	0.046373	3.69E-07	19	75	0.028516	6.61E-06
	x6	15	44	0.078846	7.29E-07	62	185	0.11264	9.27E-07	17	67	0.026217	6.63E-06	15	60	0.033336	4.55E-07	15	59	0.023959	6.76E-06
	x7	17	51	0.036052	7.31E-07	78	233	0.15566	8.19E-07	21	83	0.029738	7.73E-06	18	72	0.041526	3.69E-07	19	75	0.033766	6.61E-06
	x8	17	51	0.092	7.64E-07	78	233	0.10586	8.19E-07	21	83	0.072791	7.73E-06	18	72	0.061274	3.70E-07	19	75	0.0304	6.62E-06
	x9	17	51	0.042002	6.15E-07	78	233	0.368	8.22E-07	21	83	0.033322	7.70E-06	18	72	0.040994	3.80E-07	19	75	0.032687	6.61E-06
10000	x1	1	3	0.007893	0.00E+00	81	242	0.67484	8.92E-07	20	79	0.051839	9.08E-06	17	68	0.066822	9.32E-07	18	71	0.079653	7.70E-06
	x2	1	3	0.007634	0.00E+00	73	218	0.17572	8.91E-07	20	79	0.074758	7.88E-06	16	64	0.067781	5.46E-07	11	43	0.02922	3.00E-06
	x3	1	3	0.015904	2.22E-16	59	176	0.22518	9.62E-07	15	60	0.033559	9.35E-06	13	52	0.051043	8.83E-07	10	40	0.0272	4.91E-06
	x4	18	53	0.10333	6.69E-07	79	236	0.31815	9.26E-07	22	87	0.050369	5.48E-06	18	72	0.076736	5.22E-07	19	75	0.045885	9.36E-06
	x5	18	53	0.060803	6.69E-07	79	236	0.26202	9.26E-07	22	87	0.050461	5.48E-06	18	72	0.071018	5.22E-07	19	75	0.044874	9.36E-06
	x6	15	44	0.065192	7.29E-07	62	185	0.15996	9.27E-07	17	67	0.039317	6.63E-06	15	60	0.05065	4.55E-07	15	59	0.035413	6.76E-06
	x7	18	53	0.094819	6.69E-07	79	236	0.23701	9.26E-07	22	87	0.062936	5.48E-06	18	72	0.076797	5.22E-07	19	75	0.066942	9.36E-06
	x8	18	53	0.091724	6.84E-07	79	236	0.17905	9.27E-07	22	87	0.078648	5.48E-06	18	72	0.085378	5.23E-07	19	75	0.045675	9.36E-06
	x9	18	53	0.082879	7.19E-07	79	236	0.17998	9.24E-07	22	87	0.067936	5.49E-06	18	72	0.081977	5.28E-07	19	75	0.047638	9.33E-06
50000	x1	1	3	0.01661	0.00E+00	85	254	0.73548	8.17E-07	22	87	0.1758	5.10E-06	19	76	0.23641	9.22E-07	19	75	0.1678	7.79E-06
	x2	1	3	0.015122	0.00E+00	77	230	0.67746	8.16E-07	21	83	0.16936	8.83E-06	17	68	0.19794	4.04E-07	11	43	0.10268	6.70E-06
	x3	1	3	0.015695	2.22E-16	59	176	0.42812	9.62E-07	15	60	0.11693	9.35E-06	13	52	0.15029	8.83E-07	10	40	0.091225	4.91E-06
	x4	18	54	0.26938	8.30E-07	83	248	0.70681	8.49E-07	23	91	0.31199	6.14E-06	19	76	0.28722	6.74E-07	20	79	0.15468	9.47E-06
	x5	18	54	0.22256	8.30E-07	83	248	0.67509	8.49E-07	23	91	0.18519	6.14E-06	19	76	0.2694	6.74E-07	20	79	0.15464	9.47E-06
	x6	15	44	0.25757	7.29E-07	62	185	0.46034	9.27E-07	17	67	0.19891	6.63E-06	15	60	0.1812	4.55E-07	15	59	0.14469	6.76E-06
	x7	18	54	0.2146	8.30E-07	83	248	0.65063	8.49E-07	23	91	0.27739	6.14E-06	19	76	0.24048	6.74E-07	20	79	0.16284	9.47E-06
	x8	18	54	0.14835	8.33E-07	83	248	0.70144	8.49E-07	23	91	0.20178	6.14E-06	19	76	0.31257	6.74E-07	20	79	0.16382	9.47E-06
	x9	18	54	0.16348	8.57E-07	83	248	0.7146	8.49E-07	23	91	0.28552	6.12E-06	19	76	0.23715	6.73E-07	20	79	0.1575	9.51E-06
100000	x1	1	3	0.024111	0.00E+00	86	257	1.3852	9.24E-07	22	87	0.47813	7.21E-06	20	80	0.65758	7.47E-07	20	79	0.36895	4.99E-06
	x2	1	3	0.02247	0.00E+00	78	233	1.17	9.24E-07	22	87	0.34454	6.25E-06	17	68	0.36751	5.71E-07	11	43	0.16275	9.48E-06
	x3	1	3	0.03738	2.22E-16	59	176	0.8849	9.62E-07	15	60	0.28862	9.35E-06	13	52	0.36203	8.83E-07	10	40	0.15124	4.91E-06
	x4	19	56	0.31899	7.52E-07	84	251	1.2631	9.60E-07	23	91	0.44782	8.68E-06	19	76	0.48767	9.54E-07	21	84	0.40317	6.06E-06
	x5	19	56	0.33527	7.52E-07	84	251	1.457	9.60E-07	23	91	0.353	8.68E-06	19	76	0.55845	9.54E-07	21	84	0.31449	6.06E-06
	x6	15	44	0.26252	7.29E-07	62	185	0.96211	9.27E-07	17	67	0.26657	6.63E-06	15	60	0.40234	4.55E-07	15	59	0.21568	6.76E-06
	x7	19	56	0.31033	7.52E-07	84	251	1.2752	9.60E-07	23	91	0.3295	8.68E-06	19	76	0.61574	9.54E-07	21	84	0.41612	6.06E-06
	x8	19	56	0.35272	7.54E-07	84	251	1.239	9.60E-07	23	91	0.39743	8.68E-06	19	76	0.47514	9.54E-07	21	84	0.3471	6.06E-06
	x9	19	56	0.33608	7.50E-07	84	251	1.1977	9.59E-07	23	91	0.41664	8.69E-06	19	76	0.47144	9.54E-07	21	84	0.42077	6.06E-06

Table 7

Numerical results for Problem 5.

DIM	IP	HLSFR				CGD				PCG				PDY				ACGD
		NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM
1000	x1	9	26	0.014308	4.55E-07	81	242	0.43008	8.63E-07	23	91	0.013039	6.09E-06	17	68	0.017566	9.43E-07	12	47	0.006837	5.17E-06
	x2	9	26	0.021676	7.37E-07	83	248	0.046613	8.42E-07	23	91	0.017683	9.28E-06	18	72	0.037333	4.82E-07	12	47	0.013967	7.88E-06
	x3	9	26	0.017498	7.74E-07	83	248	0.044732	8.73E-07	23	91	0.01172	9.63E-06	18	72	0.015285	5.00E-07	12	47	0.01114	8.18E-06
	x4	9	27	0.013582	8.60E-07	82	245	0.073931	8.99E-07	23	91	0.012915	7.93E-06	18	72	0.021834	4.12E-07	12	47	0.011322	6.74E-06
	x5	9	27	0.017552	8.60E-07	82	245	0.087185	8.99E-07	23	91	0.020411	7.93E-06	18	72	0.021438	4.12E-07	12	47	0.007829	6.74E-06
	x6	9	27	0.019158	5.89E-07	83	248	0.04814	8.71E-07	23	91	0.019498	9.61E-06	18	72	0.023522	4.99E-07	12	47	0.012242	8.16E-06
	x7	9	27	0.015932	8.60E-07	82	245	0.052921	8.99E-07	23	91	0.013342	7.93E-06	18	72	0.016194	4.12E-07	12	47	0.01134	6.74E-06
	x8	9	27	0.019722	8.60E-07	82	245	0.04338	8.99E-07	23	91	0.013424	7.92E-06	18	72	0.020161	4.12E-07	12	47	0.013278	6.73E-06
	x9	9	27	0.009656	9.32E-07	82	245	0.063622	9.01E-07	23	91	0.013643	7.93E-06	18	72	0.016972	4.13E-07	12	47	0.00937	6.74E-06
5000	x1	8	24	0.032365	5.92E-07	84	251	0.199	9.89E-07	24	95	0.067194	6.83E-06	18	72	0.075774	7.08E-07	13	51	0.037741	3.01E-06
	x2	8	24	0.032245	5.10E-07	86	257	0.26411	9.65E-07	25	99	0.066083	5.22E-06	19	76	0.088501	3.58E-07	13	51	0.041404	4.59E-06
	x3	9	26	0.04046	6.16E-07	87	260	0.19137	8.01E-07	25	99	0.065242	5.41E-06	19	76	0.093793	3.72E-07	13	51	0.044886	4.77E-06
	x4	10	29	0.041491	5.88E-07	86	257	0.26052	8.24E-07	24	95	0.070926	8.89E-06	18	72	0.07529	9.22E-07	13	51	0.041548	3.92E-06
	x5	10	29	0.040664	5.88E-07	86	257	0.32941	8.24E-07	24	95	0.086487	8.89E-06	18	72	0.091088	9.22E-07	13	51	0.05445	3.92E-06
	x6	9	26	0.050973	8.50E-07	87	260	0.21405	8.01E-07	25	99	0.064145	5.41E-06	19	76	0.11547	3.72E-07	13	51	0.036684	4.77E-06
	x7	10	29	0.039105	5.88E-07	86	257	0.20749	8.24E-07	24	95	0.06261	8.89E-06	18	72	0.08249	9.22E-07	13	51	0.038958	3.92E-06
	x8	10	29	0.04557	5.88E-07	86	257	0.19755	8.24E-07	24	95	0.068231	8.89E-06	18	72	0.085026	9.22E-07	13	51	0.035929	3.92E-06
	x9	10	29	0.036645	4.36E-07	86	257	0.19841	8.24E-07	24	95	0.093715	8.90E-06	18	72	0.085331	9.22E-07	13	51	0.042215	3.93E-06
10000	x1	8	24	0.051154	3.22E-07	86	257	0.40323	8.95E-07	24	95	0.10829	9.66E-06	19	76	0.16951	3.32E-07	13	51	0.070885	4.26E-06
	x2	8	24	0.050852	5.07E-07	88	263	0.38203	8.73E-07	25	99	0.13603	7.38E-06	21	84	0.16399	4.00E-07	13	51	0.065384	6.50E-06
	x3	8	24	0.053192	4.68E-07	88	263	0.35275	9.06E-07	25	99	0.115	7.66E-06	21	84	0.19134	4.15E-07	13	51	0.075127	6.74E-06
	x4	9	27	0.053443	6.58E-07	87	260	0.41228	9.33E-07	25	99	0.11631	6.30E-06	20	80	0.16239	5.88E-07	13	51	0.07051	5.55E-06
	x5	9	27	0.052797	6.58E-07	87	260	0.42477	9.33E-07	25	99	0.13438	6.30E-06	20	80	0.18493	5.88E-07	13	51	0.11233	5.55E-06
	x6	8	24	0.052837	7.94E-07	88	263	0.39233	9.06E-07	25	99	0.1112	7.65E-06	21	84	0.18611	4.15E-07	13	51	0.061267	6.74E-06
	x7	9	27	0.057635	6.58E-07	87	260	0.35563	9.33E-07	25	99	0.12898	6.30E-06	20	80	0.16743	5.88E-07	13	51	0.078467	5.55E-06
	x8	9	27	0.056419	6.58E-07	87	260	0.38846	9.32E-07	25	99	0.13804	6.30E-06	20	80	0.15289	5.88E-07	13	51	0.061632	5.55E-06
	x9	9	27	0.059129	2.70E-07	87	260	0.36412	9.34E-07	25	99	0.12657	6.30E-06	20	80	0.16362	5.88E-07	13	51	0.072629	5.55E-06
50000	x1	8	24	0.13295	6.45E-07	90	269	1.4145	8.20E-07	26	103	0.52217	5.42E-06	22	88	0.79337	3.65E-07	13	51	0.22327	9.53E-06
	x2	8	24	0.15183	9.82E-07	91	272	1.3927	1.00E-06	26	103	0.48147	8.26E-06	24	96	0.77851	7.08E-07	14	55	0.27364	3.78E-06
	x3	9	26	0.18609	4.08E-07	92	275	1.4134	8.30E-07	26	103	0.41064	8.58E-06	24	96	0.8668	7.35E-07	14	55	0.23217	3.92E-06
	x4	9	26	0.17376	7.08E-07	91	272	1.3785	8.54E-07	26	103	0.4928	7.06E-06	23	92	0.77416	7.32E-07	14	55	0.33219	3.23E-06
	x5	9	26	0.18155	7.08E-07	91	272	1.3865	8.54E-07	26	103	0.48041	7.06E-06	23	92	0.65644	7.32E-07	14	55	0.24114	3.23E-06
	x6	9	26	0.1553	4.08E-07	92	275	1.4131	8.30E-07	26	103	0.42987	8.58E-06	24	96	0.78074	7.35E-07	14	55	0.33316	3.92E-06
	x7	9	26	0.19292	7.08E-07	91	272	1.3931	8.54E-07	26	103	0.53661	7.06E-06	23	92	0.70399	7.32E-07	14	55	0.24212	3.23E-06
	x8	9	26	0.15393	7.08E-07	91	272	1.4504	8.54E-07	26	103	0.43061	7.06E-06	23	92	0.70673	7.32E-07	14	55	0.28021	3.23E-06
	x9	9	26	0.14097	5.39E-07	91	272	1.4143	8.54E-07	26	103	0.3998	7.06E-06	23	92	0.73136	7.32E-07	14	55	0.22989	3.23E-06
100000	x1	8	24	0.27797	9.12E-07	91	272	2.9557	9.28E-07	26	103	0.89856	7.67E-06	24	96	1.6424	6.57E-07	14	55	0.60769	3.51E-06
	x2	9	26	0.33938	5.56E-07	93	278	2.9333	9.05E-07	27	107	1.1367	5.86E-06	29	116	2.2829	5.93E-07	14	55	0.53702	5.34E-06
	x3	9	26	0.3373	5.77E-07	93	278	2.8816	9.39E-07	27	107	1.0392	6.08E-06	29	116	2.1516	6.15E-07	14	55	0.59951	5.55E-06
	x4	9	26	0.30358	5.60E-07	92	275	2.9059	9.66E-07	26	103	1.0031	9.98E-06	26	104	1.8211	6.44E-07	14	55	0.54641	4.56E-06
	x5	9	26	0.33247	4.75E-07	92	275	2.9764	9.66E-07	26	103	1.0496	9.98E-06	26	104	1.8442	6.44E-07	14	55	0.60126	4.56E-06
	x6	9	26	0.31425	5.77E-07	93	278	2.917	9.39E-07	27	107	1.0072	6.08E-06	29	116	2.1713	6.15E-07	14	55	0.63201	5.55E-06
	x7	9	26	0.3306	4.75E-07	92	275	3.3714	9.66E-07	26	103	1.0065	9.98E-06	26	104	1.8355	6.44E-07	14	55	0.56667	4.56E-06
	x8	9	26	0.31528	4.75E-07	92	275	2.8726	9.66E-07	26	103	1.0548	9.98E-06	26	104	1.8133	6.44E-07	14	55	0.54508	4.56E-06
	x9	9	26	0.32075	4.76E-07	92	275	3.0658	9.66E-07	26	103	0.95345	9.98E-06	26	104	1.8203	6.44E-07	14	55	0.53629	4.56E-06

Table 8

Numerical results for Problem 6.

DIM	IP	HLSFR				CGD				PCG				PDY				ACGD
		NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM
1000	x1	12	35	0.029049	4.64E-07	38	113	1.5939	6.56E-07	21	83	0.009165	6.52E-06	6	24	0.008392	6.51E-07	11	43	0.005861	3.97E-06
	x2	11	32	0.010305	9.75E-07	37	110	0.022469	9.48E-07	17	67	0.008178	6.98E-06	17	68	0.013304	6.92E-07	10	39	0.012039	2.46E-06
	x3	15	44	0.024444	6.21E-07	38	113	0.022915	7.65E-07	18	71	0.008026	7.24E-06	18	72	0.019236	3.67E-07	10	39	0.007047	4.95E-06
	x4	16	47	0.029762	7.29E-07	37	110	0.022978	6.47E-07	20	79	0.016209	8.69E-06	19	76	0.015778	4.09E-07	11	43	0.00795	3.10E-06
	x5	16	47	0.023724	7.29E-07	37	110	0.035643	6.47E-07	20	79	0.013577	8.69E-06	19	76	0.0169	4.09E-07	11	43	0.007036	3.10E-06
	x6	15	45	0.030054	9.83E-07	38	113	0.026261	7.56E-07	21	83	0.011036	4.87E-06	18	72	0.020828	3.70E-07	12	47	0.007196	7.47E-06
	x7	16	47	0.020398	7.29E-07	37	110	0.037504	6.47E-07	20	79	0.017326	8.69E-06	19	76	0.014674	4.09E-07	11	43	0.008818	3.10E-06
	x8	16	47	0.021973	7.30E-07	37	110	0.025528	6.47E-07	20	79	0.009359	8.68E-06	19	76	0.019374	4.10E-07	11	43	0.007293	3.09E-06
	x9	16	47	0.030921	7.46E-07	37	110	0.060642	6.56E-07	20	79	0.013278	8.78E-06	19	76	0.012923	3.94E-07	11	43	0.010979	3.12E-06
5000	x1	12	36	0.050053	6.79E-07	39	116	0.060036	9.18E-07	22	87	0.04615	7.10E-06	7	28	0.020765	6.10E-08	11	43	0.023684	8.88E-06
	x2	12	35	0.039825	4.65E-07	39	116	0.058258	8.30E-07	18	71	0.031432	7.60E-06	18	72	0.057909	5.59E-07	10	39	0.040338	5.49E-06
	x3	15	44	0.057796	7.99E-07	40	119	0.080503	6.70E-07	19	75	0.042786	7.53E-06	18	72	0.078605	8.22E-07	11	43	0.027138	2.57E-06
	x4	17	50	0.078499	6.25E-07	38	113	0.058221	9.05E-07	21	83	0.04181	9.46E-06	19	76	0.080876	9.15E-07	11	43	0.02473	6.94E-06
	x5	17	50	0.065272	6.25E-07	38	113	0.14559	9.05E-07	21	83	0.042197	9.46E-06	19	76	0.06045	9.15E-07	11	43	0.022927	6.94E-06
	x6	16	47	0.036037	6.68E-07	40	119	0.10331	6.68E-07	21	83	0.046372	4.92E-06	18	72	0.096532	8.22E-07	12	47	0.02725	3.99E-06
	x7	17	50	0.066678	6.25E-07	38	113	0.093066	9.05E-07	21	83	0.036007	9.46E-06	19	76	0.062289	9.15E-07	11	43	0.026667	6.94E-06
	x8	17	50	0.060576	6.25E-07	38	113	0.076496	9.05E-07	21	83	0.053183	9.46E-06	19	76	0.0768	9.16E-07	11	43	0.027605	6.93E-06
	x9	17	50	0.065735	6.26E-07	38	113	0.095071	9.10E-07	21	83	0.050867	9.42E-06	19	76	0.080463	9.21E-07	11	43	0.033035	6.90E-06
10000	x1	12	36	0.094012	9.61E-07	40	119	0.11766	8.12E-07	23	91	0.078114	4.89E-06	7	28	0.048952	8.62E-08	12	47	0.048396	3.03E-06
	x2	12	35	0.064997	6.57E-07	40	119	0.14917	7.34E-07	19	75	0.067982	5.23E-06	18	72	0.11357	7.90E-07	10	39	0.035316	7.77E-06
	x3	16	47	0.082227	5.81E-07	40	119	0.11575	9.48E-07	20	79	0.071007	5.16E-06	19	76	0.11519	4.22E-07	11	43	0.043755	3.61E-06
	x4	17	50	0.091098	8.83E-07	39	116	0.11448	8.00E-07	22	87	0.088341	6.52E-06	20	80	0.10801	4.69E-07	11	43	0.050355	9.81E-06
	x5	17	50	0.089616	8.83E-07	39	116	0.1367	8.00E-07	22	87	0.065983	6.52E-06	20	80	0.11862	4.69E-07	11	43	0.046022	9.81E-06
	x6	16	47	0.14203	9.36E-07	40	119	0.12105	9.46E-07	23	91	0.079529	6.19E-06	19	76	0.10274	4.22E-07	13	51	0.046248	6.38E-06
	x7	17	50	0.095615	8.83E-07	39	116	0.16129	8.00E-07	22	87	0.094775	6.52E-06	20	80	0.121	4.69E-07	11	43	0.072467	9.81E-06
	x8	17	50	0.21555	8.83E-07	39	116	0.12837	8.00E-07	22	87	0.074221	6.52E-06	20	80	0.13616	4.69E-07	11	43	0.039266	9.81E-06
	x9	17	50	0.10375	8.75E-07	39	116	0.3069	7.99E-07	22	87	0.099181	6.53E-06	20	80	0.12777	4.68E-07	11	43	0.04783	9.79E-06
50000	x1	13	38	0.22221	7.00E-07	42	125	1.3933	7.10E-07	24	95	0.31765	5.33E-06	20	80	0.39988	6.95E-07	12	47	0.16422	6.77E-06
	x2	12	36	0.17902	9.62E-07	42	125	0.55424	6.42E-07	20	79	0.2098	5.70E-06	19	76	0.44299	6.42E-07	11	43	0.1288	4.19E-06
	x3	16	47	0.25355	8.83E-07	42	125	0.50726	8.30E-07	21	83	0.33955	5.59E-06	19	76	0.44244	9.44E-07	11	43	0.12579	8.05E-06
	x4	18	53	0.26099	6.54E-07	41	122	0.39556	7.00E-07	23	91	0.23472	7.10E-06	21	84	0.45082	3.80E-07	12	47	0.15312	5.30E-06
	x5	18	53	0.21421	6.54E-07	41	122	0.40101	7.00E-07	23	91	0.31743	7.10E-06	21	84	0.3826	3.80E-07	12	47	0.1681	5.30E-06
	x6	17	50	0.29397	5.53E-07	42	125	0.4652	8.29E-07	21	83	0.28602	5.86E-06	19	76	0.3848	9.44E-07	12	47	0.13501	2.54E-06
	x7	18	53	0.25801	6.54E-07	41	122	0.42786	7.00E-07	23	91	0.25659	7.10E-06	21	84	0.43777	3.80E-07	12	47	0.17412	5.30E-06
	x8	18	53	0.3159	6.54E-07	41	122	0.36268	7.00E-07	23	91	0.24778	7.10E-06	21	84	0.43678	3.80E-07	12	47	0.15937	5.30E-06
	x9	18	53	0.31437	6.55E-07	41	122	0.55496	7.02E-07	23	91	0.37644	7.09E-06	21	84	0.68463	3.81E-07	12	47	0.18109	5.30E-06
100000	x1	13	38	0.46788	9.90E-07	43	128	0.76993	6.28E-07	24	95	0.56137	7.54E-06	20	80	0.83997	9.83E-07	12	47	0.26978	9.58E-06
	x2	13	38	0.33088	4.43E-07	42	125	0.77585	9.08E-07	20	79	0.42556	8.06E-06	20	80	0.7169	7.45E-07	11	43	0.29106	5.93E-06
	x3	16	47	0.42191	8.55E-07	43	128	0.8996	7.34E-07	21	83	0.50609	7.90E-06	20	80	0.79422	9.21E-07	12	47	0.30009	2.74E-06
	x4	18	53	0.43122	9.25E-07	41	122	0.68719	9.90E-07	24	95	0.61866	4.89E-06	20	80	0.83134	6.67E-07	12	47	0.29504	7.49E-06
	x5	18	53	0.46021	9.25E-07	41	122	0.73513	9.90E-07	24	95	0.51049	4.89E-06	20	80	0.87882	6.67E-07	12	47	0.2534	7.49E-06
	x6	17	50	0.55415	7.04E-07	43	128	0.81192	7.34E-07	21	83	0.57768	8.09E-06	20	80	0.8959	9.21E-07	12	47	0.26603	3.24E-06
	x7	18	53	0.51063	9.25E-07	41	122	0.91418	9.90E-07	24	95	0.46009	4.89E-06	20	80	0.74139	6.67E-07	12	47	0.38601	7.49E-06
	x8	18	53	0.4617	9.25E-07	41	122	0.66684	9.90E-07	24	95	0.63668	4.89E-06	20	80	0.94291	6.67E-07	12	47	0.35009	7.49E-06
	x9	18	53	0.72029	9.24E-07	41	122	1.1989	9.92E-07	24	95	0.75387	4.90E-06	20	80	0.80012	6.68E-07	12	47	0.35252	7.48E-06

Table 9

Numerical results for Problem 7.

DIM	IP	HLSFR				CGD				PCG				PDY				ACGD
		NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM
1000	x1	0	0	0.038906	0	0	0	0.78166	0	0	0	0.002616	0	0	0	0.003328	0	0	0	0.000844	0
	x2	12	35	0.033115	4.05E-07	122	365	0.67291	9.16E-07	23	91	0.038102	7.76E-06	36	144	0.16109	6.34E-07	61	243	0.078169	9.62E-06
	x3	12	35	0.0381	7.68E-07	83	248	0.17512	9.01E-07	101	403	0.18844	9.86E-06	36	144	0.26356	6.40E-07	132	527	0.27147	9.23E-06
	x4	13	38	0.039655	4.65E-07	110	329	0.11532	9.44E-07	19	75	0.032265	9.48E-06	36	144	0.18872	6.45E-07	37	147	0.050214	8.52E-06
	x5	13	38	0.024441	3.59E-07	62	185	0.07797	7.88E-07	17	67	0.027818	6.63E-06	24	96	0.11755	5.84E-07	29	115	0.054364	6.75E-06
	x6	14	41	0.04153	4.87E-07	69	206	0.093189	9.49E-07	179	715	0.23112	9.93E-06	34	136	0.18349	7.16E-07	119	475	0.14755	9.02E-06
	x7	13	38	0.052096	4.65E-07	110	329	0.11319	9.44E-07	19	75	0.025705	9.48E-06	36	144	0.1563	6.45E-07	37	147	0.069248	8.52E-06
	x8	13	38	0.031708	3.60E-07	61	182	0.075667	8.88E-07	17	67	0.025078	6.62E-06	22	88	0.14498	4.33E-07	29	115	0.04373	6.84E-06
	x9	17	50	0.049173	3.59E-07	132	395	0.14667	9.21E-07	17	67	0.048771	7.80E-06	32	128	0.16597	2.41E-07	220	879	0.37058	9.69E-06
5000	x1	0	0	0.002789	0	0	0	0.003597	0	0	0	0.007603	0	0	0	0.001808	0	0	0	0.003491	0
	x2	12	35	0.10778	7.57E-07	121	362	0.52844	9.10E-07	20	79	0.12976	8.49E-06	34	136	0.80841	8.36E-07	77	307	0.53458	9.40E-06
	x3	12	36	0.13359	7.78E-07	80	239	0.41913	9.09E-07	100	399	0.69403	9.74E-06	34	136	0.83492	8.69E-07	87	347	0.66276	8.66E-06
	x4	13	38	0.1268	9.52E-07	107	320	0.46613	9.77E-07	19	75	0.12409	5.69E-06	34	136	0.81581	8.49E-07	44	175	0.29489	1.00E-05
	x5	13	38	0.12218	5.01E-07	64	191	0.28878	7.27E-07	18	71	0.14112	5.79E-06	22	88	0.50975	9.54E-07	40	159	0.28649	7.70E-06
	x6	14	41	0.12093	4.83E-07	74	221	0.34861	9.38E-07	611	2443	4.0397	9.79E-06	32	128	0.71834	5.84E-07	116	463	0.90756	9.75E-06
	x7	13	38	0.19395	9.52E-07	107	320	0.50089	9.77E-07	19	75	0.12433	5.69E-06	34	136	0.75098	8.49E-07	44	175	0.28655	1.00E-05
	x8	13	38	0.14336	5.01E-07	64	191	0.33556	7.24E-07	18	71	0.13688	5.79E-06	23	92	0.57897	2.97E-07	31	123	0.24484	7.96E-06
	x9	17	50	0.15414	7.01E-07	106	317	0.48913	9.45E-07	18	71	0.15946	6.36E-06	27	108	0.60206	3.49E-07	42	167	0.33114	9.65E-06
10000	x1	0	0	0.00298	0	0	0	0.002765	0	0	0	0.002035	0	0	0	0.006066	0	0	0	0.002716	0
	x2	12	36	0.2002	5.65E-07	120	359	0.94571	9.61E-07	20	79	0.25766	6.25E-06	34	136	1.4786	6.78E-07	61	243	0.89428	8.97E-06
	x3	13	38	0.16935	3.90E-07	79	236	0.67652	9.34E-07	99	395	1.4137	9.23E-06	34	136	1.477	7.07E-07	135	539	1.8146	9.45E-06
	x4	13	39	0.15451	6.73E-07	107	320	0.91163	9.02E-07	19	75	0.21665	5.57E-06	34	136	1.5588	6.89E-07	23	91	0.28068	6.83E-06
	x5	13	38	0.20324	5.44E-07	65	194	0.50449	7.35E-07	18	71	0.23743	8.53E-06	22	88	1.1074	9.79E-07	26	103	0.36902	7.22E-06
	x6	14	41	0.18092	5.24E-07	76	227	0.6752	9.44E-07	189	755	2.7342	9.94E-06	31	124	1.3468	7.33E-07	131	523	1.709	9.68E-06
	x7	13	39	0.19382	6.73E-07	107	320	0.85944	9.02E-07	19	75	0.21799	5.57E-06	34	136	1.5649	6.89E-07	23	91	0.34965	6.83E-06
	x8	13	38	0.35269	5.45E-07	65	194	0.51017	7.34E-07	18	71	0.25233	8.53E-06	21	84	0.987	7.65E-07	26	103	0.32723	7.24E-06
	x9	17	51	0.27824	8.84E-07	129	386	1.0032	9.30E-07	18	71	0.26393	8.76E-06	27	108	1.2328	2.79E-07	75	299	1.1469	9.17E-06
50000	x1	0	0	0.012584	0	0	0	0.009213	0	0	0	0.009373	0	0	0	0.009031	0	0	0	0.010445	0
	x2	13	38	0.84045	7.66E-07	119	356	4.0401	9.39E-07	19	75	1.0123	9.22E-06	34	136	5.8422	6.35E-07	94	375	4.5319	9.54E-06
	x3	13	38	0.62033	8.66E-07	77	230	2.6023	9.99E-07	96	383	4.4204	8.76E-06	34	136	5.5657	6.70E-07	124	495	5.9186	9.61E-06
	x4	14	41	0.69678	5.16E-07	105	314	3.5696	9.65E-07	19	75	1.0568	8.01E-06	33	132	5.574	6.56E-07	15	59	1.0361	7.21E-06
	x5	13	39	0.63099	4.82E-07	67	200	2.4286	8.48E-07	19	75	1.0308	6.76E-06	23	92	3.962	9.16E-07	29	115	1.5984	8.81E-06
	x6	14	41	0.79236	6.78E-07	77	230	2.7058	9.28E-07	107	427	5.0498	9.28E-06	32	128	5.6124	6.70E-07	178	711	8.4281	9.35E-06
	x7	14	41	0.90717	5.16E-07	105	314	3.5951	9.65E-07	19	75	1.0551	8.01E-06	33	132	5.7912	6.56E-07	15	59	0.85465	7.21E-06
	x8	13	39	0.64677	4.83E-07	67	200	2.4259	8.48E-07	19	75	1.0081	6.76E-06	23	92	3.9011	9.07E-07	29	115	1.5584	8.85E-06
	x9	18	53	1.0288	6.35E-07	121	362	4.0524	9.09E-07	19	75	1.0884	6.26E-06	26	104	4.4559	3.85E-07	115	459	5.6331	9.80E-06
100000	x1	0	0	0.020129	0	0	0	0.044349	0	0	0	0.016332	0	0	0	0.016318	0	0	0	0.02095	0
	x2	13	39	1.4101	4.49E-07	118	353	8.7163	9.89E-07	20	79	1.9685	4.35E-06	33	132	11.752	8.00E-07	127	507	12.9108	9.95E-06
	x3	13	39	1.2614	6.14E-07	77	230	6.1223	9.43E-07	96	383	9.5064	9.73E-06	33	132	11.5596	8.45E-07	183	731	18.0213	9.91E-06
	x4	14	41	1.4097	7.29E-07	105	314	8.316	9.08E-07	20	79	2.1194	3.86E-06	35	140	12.4189	6.03E-07	15	59	1.9333	8.01E-06
	x5	13	39	1.3517	8.61E-07	68	203	5.4894	8.63E-07	19	75	2.0334	9.63E-06	23	92	8.0264	9.84E-07	21	83	2.4448	7.92E-06
	x6	14	41	1.3917	7.81E-07	77	230	5.9139	9.06E-07	108	431	11.1115	8.75E-06	32	128	11.542	5.58E-07	182	727	19.5586	9.37E-06
	x7	14	41	1.7178	7.29E-07	105	314	7.6441	9.08E-07	20	79	2.1252	3.86E-06	35	140	12.3952	6.03E-07	15	59	2.1594	8.01E-06
	x8	13	39	1.3664	8.61E-07	68	203	5.0265	8.63E-07	19	75	2.0399	9.63E-06	24	96	8.698	7.30E-07	21	83	3.0546	7.92E-06
	x9	18	53	1.8446	8.85E-07	121	362	8.769	9.42E-07	19	75	2.0222	8.84E-06	26	104	9.1914	6.07E-07	108	431	12.64	9.99E-06

Table 10

Numerical results for Problem 8.

DIM	IP	HLSFR				CGD				PCG				PDY				ACGD
		NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM
1000	x1	8	20	0.045332	9.80E-07	27	76	0.13731	8.06E-07	9	31	0.15603	7.60E-06	11	42	0.010432	2.67E-07	8	25	0.004228	6.09E-06
	x2	8	20	0.015402	9.80E-07	39	113	0.013511	8.17E-07	9	31	0.011112	7.60E-06	11	42	0.01056	2.67E-07	8	25	0.005784	6.09E-06
	x3	39	113	0.050306	9.61E-07	F	F	F	F	26	99	0.009084	9.77E-06	34	107	0.012388	9.76E-07	213	849	0.091231	1.00E-05
	x4	58	166	0.070369	9.97E-07	F	F	F	F	10	35	0.006974	7.26E-06	11	42	0.008517	2.70E-07	21	81	0.011337	9.91E-06
	x5	58	166	0.0931	9.97E-07	F	F	F	F	10	35	0.006215	7.26E-06	11	42	0.031007	2.70E-07	21	81	0.009868	9.91E-06
	x6	46	134	0.057752	9.80E-07	45	132	0.01524	8.49E-07	9	31	0.005135	7.60E-06	11	41	0.007564	2.67E-07	8	25	0.008325	6.09E-06
	x7	58	166	0.028136	9.97E-07	F	F	F	F	10	35	0.005299	7.26E-06	11	42	0.008569	2.70E-07	21	81	0.016458	9.91E-06
	x8	60	172	0.15433	9.77E-07	44	129	0.014334	9.92E-07	9	31	0.00324	7.60E-06	11	42	0.011897	2.67E-07	8	25	0.007805	6.09E-06
	x9	60	172	0.09293	9.90E-07	73	216	0.019643	9.87E-07	9	31	0.004678	7.60E-06	11	42	0.010021	2.67E-07	8	25	0.006595	6.09E-06
5000	x1	6	15	0.023477	5.40E-07	25	72	0.028521	7.58E-07	7	25	0.014113	1.30E-06	8	31	0.031179	1.59E-07	4	12	0.025081	5.76E-06
	x2	6	15	0.019769	5.40E-07	26	75	0.04814	9.83E-07	7	25	0.016343	1.30E-06	8	31	0.026785	1.59E-07	4	12	0.031653	5.76E-06
	x3	8	22	0.025065	2.74E-07	86	255	0.092447	9.78E-07	17	65	0.033401	8.72E-06	11	34	0.029802	8.53E-07	10	37	0.03622	7.40E-06
	x4	18	52	0.11062	9.23E-07	454	1359	0.59789	9.99E-07	7	25	0.025883	1.42E-06	8	31	0.028639	1.59E-07	64	253	0.15295	9.56E-06
	x5	18	52	0.035124	9.23E-07	F	F	F	F	7	25	0.012562	1.42E-06	8	31	0.044412	1.59E-07	64	253	0.21473	9.56E-06
	x6	13	36	0.032072	8.93E-07	F	F	F	F	7	25	0.013074	1.30E-06	8	30	0.02009	1.59E-07	4	12	0.014082	5.76E-06
	x7	18	52	0.052554	9.23E-07	F	F	F	F	7	25	0.021775	1.42E-06	8	31	0.029879	1.59E-07	64	253	0.30531	9.56E-06
	x8	16	45	0.061545	9.95E-07	F	F	F	F	7	25	0.011686	1.30E-06	8	31	0.026899	1.59E-07	4	12	0.017501	5.76E-06
	x9	16	45	0.03167	9.79E-07	F	F	F	F	7	25	0.016309	1.30E-06	8	31	0.030435	1.59E-07	4	12	0.006705	5.76E-06
10000	x1	8	21	0.03621	3.71E-07	18	51	0.043142	7.45E-07	5	17	0.019474	5.06E-06	12	47	0.17017	7.22E-07	5	17	0.022117	2.19E-06
	x2	8	21	0.034409	3.71E-07	15	42	0.037425	7.64E-07	5	17	0.017798	5.06E-06	11	43	0.086668	7.22E-07	5	17	0.020951	2.19E-06
	x3	15	44	0.098121	9.18E-07	36	106	0.14622	9.67E-07	10	38	0.034521	9.06E-06	8	27	0.039777	7.55E-07	5	18	0.031678	3.45E-06
	x4	8	22	0.045275	8.83E-07	196	585	0.52126	9.96E-07	5	17	0.038537	8.38E-06	11	43	0.080798	7.22E-07	85	337	0.29957	9.75E-06
	x5	8	22	0.027345	8.83E-07	50	147	0.15272	9.44E-07	5	17	0.01883	8.38E-06	11	43	0.084839	7.22E-07	85	337	0.32292	9.75E-06
	x6	8	21	0.043909	3.70E-07	210	627	0.52785	9.77E-07	5	17	0.020962	5.06E-06	11	42	0.077928	7.22E-07	5	17	0.024861	2.19E-06
	x7	8	22	0.038238	8.83E-07	20	57	0.051957	8.78E-07	5	17	0.020192	8.38E-06	11	43	0.079948	7.22E-07	85	337	0.50831	9.75E-06
	x8	9	24	0.043123	7.17E-07	208	621	0.40336	9.88E-07	5	17	0.019995	5.06E-06	11	43	0.083423	7.22E-07	5	17	0.021514	2.19E-06
	x9	9	24	0.12251	6.95E-07	208	621	0.39409	9.94E-07	5	17	0.023803	5.06E-06	11	43	0.10931	7.22E-07	5	17	0.027147	2.19E-06
50000	x1	13	37	0.2028	5.34E-07	9	25	0.10676	4.81E-07	8	30	0.11189	5.15E-06	12	48	0.61588	7.59E-07	5	18	0.095206	2.45E-06
	x2	13	37	0.21638	5.34E-07	9	25	0.082291	5.00E-07	8	30	0.11226	5.15E-06	10	40	0.29855	7.59E-07	5	18	0.18925	2.45E-06
	x3	12	33	0.14161	9.21E-07	F	F	F	F	9	33	0.13984	5.57E-06	12	46	0.25481	7.38E-07	7	25	0.13066	6.08E-06
	x4	10	28	0.11798	8.86E-07	F	F	F	F	8	30	0.087649	5.29E-06	11	44	0.43006	7.59E-07	5	18	0.18536	5.53E-06
	x5	10	28	0.17321	8.86E-07	F	F	F	F	8	30	0.13943	5.29E-06	11	44	0.38657	7.59E-07	5	18	0.092437	5.53E-06
	x6	14	40	0.18836	6.47E-07	117	349	0.91182	9.76E-07	8	30	0.088187	5.15E-06	10	39	0.25857	7.59E-07	5	18	0.09337	2.45E-06
	x7	10	28	0.1297	8.86E-07	F	F	F	F	8	30	0.086955	5.29E-06	11	44	0.44874	7.59E-07	5	18	0.15625	5.53E-06
	x8	12	34	0.19733	7.05E-07	F	F	F	F	8	30	0.14503	5.15E-06	11	44	0.37089	7.59E-07	5	18	0.10386	2.45E-06
	x9	11	32	0.18504	7.94E-07	F	F	F	F	8	30	0.093379	5.15E-06	11	44	0.41393	7.59E-07	5	18	0.078907	2.45E-06
100000	x1	9	25	0.21361	3.85E-07	46	136	0.8254	9.31E-07	6	22	0.12967	6.81E-07	14	56	1.6521	2.19E-07	4	14	0.15074	2.67E-06
	x2	9	25	0.18318	3.85E-07	68	202	1.5356	9.35E-07	6	22	0.16861	6.81E-07	9	36	0.75632	2.19E-07	4	14	0.16524	2.70E-06
	x3	12	34	0.25992	7.10E-07	74	219	1.4336	9.65E-07	7	25	0.17283	1.06E-06	11	42	0.55542	9.91E-07	9	33	0.23368	5.47E-06
	x4	10	28	0.22578	4.70E-07	885	2653	20.7608	9.96E-07	6	22	0.13393	9.73E-07	11	44	1.1324	2.19E-07	8	30	0.28851	8.60E-06
	x5	10	28	0.23293	4.70E-07	887	2659	22.9501	9.99E-07	6	22	0.13266	9.73E-07	11	44	1.2224	2.19E-07	8	30	0.28145	8.60E-06
	x6	12	34	0.30496	8.00E-07	92	274	2.0913	9.82E-07	6	22	0.18112	6.81E-07	9	35	0.45198	2.21E-07	4	14	0.16188	2.70E-06
	x7	10	28	0.20114	4.70E-07	113	337	2.3221	9.65E-07	6	22	0.15632	9.73E-07	11	44	1.0635	2.19E-07	8	30	0.28902	8.60E-06
	x8	10	28	0.2406	5.07E-07	83	247	1.6454	9.53E-07	6	22	0.17352	6.81E-07	11	44	1.176	2.19E-07	4	14	0.13366	2.70E-06
	x9	10	28	0.20577	5.22E-07	121	361	2.7946	9.72E-07	6	22	0.16007	6.81E-07	11	44	1.1805	2.19E-07	4	14	0.17212	2.70E-06

Table 11

Numerical results for Problem 9.

DIM	IP	HLSFR				CGD				PCG				PDY				ACGD
		NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM
1000	x1	8	24	0.005028	5.64E-07	23	68	0.13191	4.46E-07	9	35	0.003296	5.48E-06	11	44	0.006284	4.01E-07	9	35	0.003833	2.48E-06
	x2	8	23	0.008476	8.64E-07	21	62	0.006679	9.28E-07	9	35	0.002772	2.15E-06	11	44	0.008552	1.57E-07	8	31	0.00506	6.41E-06
	x3	8	24	0.005569	3.08E-07	22	65	0.006972	5.61E-07	9	35	0.003386	3.00E-06	11	44	0.010073	2.19E-07	8	31	0.005208	8.93E-06
	x4	8	24	0.0066	2.82E-07	22	65	0.011645	5.14E-07	9	35	0.004007	2.74E-06	11	44	0.008651	2.01E-07	8	31	0.008581	8.18E-06
	x5	8	24	0.009222	2.82E-07	22	65	0.020597	5.14E-07	9	35	0.005052	2.74E-06	11	44	0.007554	2.01E-07	8	31	0.009496	8.18E-06
	x6	8	24	0.006655	3.04E-07	22	65	0.00662	5.53E-07	9	35	0.002685	2.96E-06	11	44	0.009785	2.16E-07	8	31	0.003949	8.81E-06
	x7	8	24	0.005654	2.82E-07	22	65	0.007176	5.14E-07	9	35	0.002847	2.74E-06	11	44	0.006838	2.01E-07	8	31	0.004304	8.18E-06
	x8	8	24	0.013108	2.83E-07	22	65	0.007833	5.15E-07	9	35	0.004705	2.75E-06	11	44	0.009929	2.01E-07	8	31	0.00625	8.19E-06
	x9	8	24	0.008784	2.86E-07	22	65	0.010065	5.19E-07	9	35	0.003325	2.79E-06	11	44	0.005355	1.98E-07	8	31	0.004954	8.01E-06
5000	x1	9	26	0.0176	4.91E-07	23	68	0.039317	9.97E-07	10	39	0.01228	2.07E-06	11	44	0.024799	8.97E-07	9	35	0.02265	5.55E-06
	x2	8	24	0.02893	4.95E-07	22	65	0.03288	9.01E-07	9	35	0.012397	4.81E-06	11	44	0.020903	3.52E-07	9	35	0.2018	2.18E-06
	x3	8	24	0.074084	6.90E-07	23	68	0.023742	5.45E-07	9	35	0.012145	6.71E-06	11	44	0.031888	4.90E-07	9	35	0.032921	3.03E-06
	x4	8	24	0.021806	6.32E-07	23	68	0.022984	4.99E-07	9	35	0.01087	6.14E-06	11	44	0.022776	4.49E-07	9	35	0.020182	2.78E-06
	x5	8	24	0.024323	6.32E-07	23	68	0.024503	4.99E-07	9	35	0.011485	6.14E-06	11	44	0.031036	4.49E-07	9	35	0.046506	2.78E-06
	x6	8	24	0.013959	6.87E-07	23	68	0.064772	5.43E-07	9	35	0.013674	6.68E-06	11	44	0.021317	4.89E-07	9	35	0.018303	3.02E-06
	x7	8	24	0.016761	6.32E-07	23	68	0.036919	4.99E-07	9	35	0.017811	6.14E-06	11	44	0.031354	4.49E-07	9	35	0.023681	2.78E-06
	x8	8	24	0.016687	6.32E-07	23	68	0.047002	5.00E-07	9	35	0.014039	6.14E-06	11	44	0.14942	4.49E-07	9	35	0.013313	2.78E-06
	x9	8	24	0.020212	6.27E-07	23	68	0.022977	5.04E-07	9	35	0.017948	6.10E-06	11	44	0.025546	4.48E-07	9	35	0.017248	2.78E-06
10000	x1	9	26	0.078489	6.94E-07	24	71	0.12993	6.13E-07	10	39	0.020771	2.93E-06	12	48	0.05529	2.51E-07	9	35	0.023868	7.85E-06
	x2	8	24	0.020379	7.00E-07	23	68	0.12961	5.53E-07	9	35	0.03616	6.80E-06	11	44	0.045465	4.97E-07	9	35	0.02255	3.08E-06
	x3	8	24	0.03029	9.76E-07	23	68	0.052404	7.71E-07	9	35	0.020414	9.48E-06	11	44	0.042797	6.93E-07	9	35	0.032979	4.29E-06
	x4	8	24	0.027569	8.93E-07	23	68	0.074305	7.06E-07	9	35	0.01837	8.68E-06	11	44	0.044082	6.35E-07	9	35	0.020341	3.93E-06
	x5	8	24	0.025422	8.93E-07	23	68	0.13022	7.06E-07	9	35	0.075188	8.68E-06	11	44	0.04152	6.35E-07	9	35	0.023121	3.93E-06
	x6	8	24	0.02649	9.74E-07	23	68	0.073075	7.70E-07	9	35	0.01947	9.47E-06	11	44	0.045968	6.92E-07	9	35	0.026716	4.28E-06
	x7	8	24	0.024928	8.93E-07	23	68	0.064781	7.06E-07	9	35	0.027659	8.68E-06	11	44	0.04163	6.35E-07	9	35	0.022106	3.93E-06
	x8	8	24	0.030918	8.93E-07	23	68	0.16717	7.06E-07	9	35	0.023928	8.69E-06	11	44	0.052877	6.35E-07	9	35	0.023894	3.93E-06
	x9	8	24	0.029869	8.95E-07	23	68	0.080044	7.05E-07	9	35	0.027406	8.78E-06	11	44	0.053556	6.35E-07	9	35	0.022737	3.96E-06
50000	x1	9	27	0.10307	3.98E-07	25	74	0.1506	5.95E-07	10	39	0.067349	6.54E-06	14	56	0.21988	2.30E-07	10	39	0.11373	2.67E-06
	x2	9	26	0.088932	6.09E-07	24	71	0.17024	5.37E-07	10	39	0.086555	2.57E-06	12	48	0.17181	2.20E-07	9	35	0.065306	6.88E-06
	x3	9	26	0.095518	8.49E-07	24	71	0.15343	7.49E-07	10	39	0.070467	3.58E-06	12	48	0.15512	3.07E-07	9	35	0.10325	9.60E-06
	x4	9	26	0.07146	7.77E-07	24	71	0.15237	6.86E-07	10	39	0.094863	3.28E-06	12	48	0.18387	2.81E-07	9	35	0.071763	8.79E-06
	x5	9	26	0.088237	7.77E-07	24	71	0.15077	6.86E-07	10	39	0.11394	3.28E-06	12	48	0.16862	2.81E-07	9	35	0.068275	8.79E-06
	x6	9	26	0.076803	8.48E-07	24	71	0.29171	7.49E-07	10	39	0.081884	3.58E-06	12	48	0.1473	3.07E-07	9	35	0.11086	9.59E-06
	x7	9	26	0.078565	7.77E-07	24	71	0.28604	6.86E-07	10	39	0.067458	3.28E-06	12	48	0.17418	2.81E-07	9	35	0.076509	8.79E-06
	x8	9	26	0.086736	7.77E-07	24	71	0.15309	6.86E-07	10	39	0.072279	3.28E-06	12	48	0.21846	2.81E-07	9	35	0.081581	8.79E-06
	x9	9	26	0.097146	7.75E-07	24	71	0.32932	6.89E-07	10	39	0.1241	3.29E-06	12	48	0.15526	2.81E-07	9	35	0.06398	8.80E-06
100000	x1	9	27	0.11399	5.62E-07	25	74	0.6106	8.41E-07	10	39	0.17139	9.25E-06	14	56	0.44534	3.25E-07	10	39	0.18755	3.77E-06
	x2	9	26	0.15248	8.61E-07	24	71	0.28022	7.60E-07	10	39	0.18473	3.63E-06	12	48	0.2881	3.12E-07	9	35	0.12208	9.74E-06
	x3	9	27	0.18602	3.08E-07	25	74	0.31581	4.60E-07	10	39	0.16699	5.06E-06	12	48	0.29146	4.35E-07	10	39	0.16077	2.06E-06
	x4	9	27	0.1428	2.82E-07	24	71	0.48131	9.70E-07	10	39	0.12892	4.63E-06	12	48	0.3706	3.98E-07	10	39	0.18411	1.89E-06
	x5	9	27	0.17049	2.82E-07	24	71	0.40206	9.70E-07	10	39	0.15327	4.63E-06	12	48	0.33931	3.98E-07	10	39	0.13712	1.89E-06
	x6	9	27	0.16253	3.08E-07	25	74	0.27723	4.60E-07	10	39	0.20672	5.06E-06	12	48	0.3074	4.34E-07	10	39	0.13456	2.06E-06
	x7	9	27	0.14438	2.82E-07	24	71	0.26333	9.70E-07	10	39	0.15905	4.63E-06	12	48	0.44484	3.98E-07	10	39	0.13312	1.89E-06
	x8	9	27	0.14597	2.82E-07	24	71	0.24372	9.70E-07	10	39	0.15671	4.63E-06	12	48	0.35288	3.98E-07	10	39	0.19196	1.89E-06
	x9	9	27	0.13731	2.81E-07	24	71	0.27596	9.69E-07	10	39	0.14325	4.64E-06	12	48	0.31933	3.98E-07	10	39	0.13545	1.89E-06

Table 12

Numerical results for Problem 10.

DIM	IP	HLSFR				CGD				PCG				PDY				ACGD
		NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM	NI	NF	CPU	NORM
1000	x1	1	3	0.003912	0	166	497	0.69036	9.13E-07	31	123	0.25366	6.24E-06	13	52	0.51624	3.30E-07	1	4	0.12136	0
	x2	1	3	0.004081	0	152	455	0.067494	9.17E-07	9	35	0.078956	4.99E-06	13	52	0.28982	2.20E-07	9	35	0.13693	2.37E-06
	x3	1	3	0.005348	2.22E-16	134	401	0.076169	9.21E-07	24	96	0.053115	8.77E-06	12	48	0.63759	4.01E-07	14	56	0.059274	6.22E-06
	x4	16	47	0.040778	7.31E-07	165	494	0.066906	9.31E-07	30	119	0.026624	8.24E-06	16	64	0.050373	6.16E-07	17	67	0.011662	7.71E-06
	x5	16	47	0.031295	7.31E-07	165	494	0.064173	9.31E-07	30	119	0.080555	8.24E-06	16	64	0.17673	6.16E-07	17	67	0.35772	7.71E-06
	x6	13	39	0.031615	7.52E-07	141	422	0.060847	9.25E-07	25	99	0.033508	7.79E-06	17	68	0.036626	8.54E-07	14	55	0.011219	9.07E-06
	x7	16	47	0.02627	7.31E-07	165	494	0.15999	9.31E-07	30	119	0.024998	8.24E-06	16	64	0.029012	6.16E-07	17	67	0.016175	7.71E-06
	x8	16	47	0.02149	7.31E-07	165	494	0.12262	9.32E-07	30	119	0.016311	8.24E-06	16	64	0.037983	7.56E-07	17	67	0.060734	7.73E-06
	x9	16	47	0.040906	6.92E-07	165	494	0.068262	9.35E-07	30	119	0.01445	8.26E-06	16	64	0.030959	7.35E-07	17	67	0.038495	7.76E-06
5000	x1	1	3	0.010604	0	173	518	0.27385	9.76E-07	32	127	0.088049	8.31E-06	13	52	0.11491	7.38E-07	1	4	0.008497	0
	x2	1	3	0.006858	0	159	476	0.25698	9.80E-07	10	39	0.029317	2.23E-06	13	52	0.093039	4.93E-07	9	35	0.026642	5.30E-06
	x3	1	3	0.006665	2.22E-16	134	401	0.19158	9.21E-07	24	96	0.082569	8.77E-06	12	48	0.067394	4.01E-07	14	56	0.058619	6.22E-06
	x4	16	48	0.046446	8.02E-07	172	515	0.26576	9.97E-07	32	127	0.084467	6.53E-06	17	68	0.14568	3.10E-07	18	71	0.044645	6.79E-06
	x5	16	48	0.056196	8.02E-07	172	515	0.39559	9.97E-07	32	127	0.14004	6.53E-06	17	68	0.11512	3.10E-07	18	71	0.096528	6.79E-06
	x6	13	39	0.085966	7.52E-07	141	422	0.29676	9.26E-07	25	99	0.056093	7.80E-06	17	68	0.10168	4.89E-07	14	55	0.037525	9.08E-06
	x7	16	48	0.054039	8.02E-07	172	515	0.26396	9.97E-07	32	127	0.074037	6.53E-06	17	68	0.1111	3.10E-07	18	71	0.042014	6.79E-06
	x8	16	48	0.059202	8.02E-07	172	515	0.27002	9.97E-07	32	127	0.091706	6.53E-06	17	68	0.13293	3.22E-07	18	71	0.097957	6.80E-06
	x9	16	48	0.052719	8.25E-07	172	515	0.35188	9.89E-07	32	127	0.096698	6.52E-06	17	68	0.17819	2.95E-07	18	71	0.052604	6.81E-06
10000	x1	1	3	0.019663	0	177	530	0.46763	9.06E-07	33	131	0.12698	7.00E-06	14	56	0.32629	2.26E-07	1	4	0.011638	0
	x2	1	3	0.014307	0	163	488	0.43325	9.10E-07	10	39	0.075608	3.15E-06	13	52	0.12745	6.97E-07	9	35	0.03964	7.49E-06
	x3	1	3	0.011506	2.22E-16	134	401	0.34974	9.21E-07	24	96	0.094225	8.77E-06	12	48	0.13733	4.01E-07	14	56	0.052292	6.22E-06
	x4	17	50	0.067805	6.93E-07	176	527	0.43952	9.25E-07	32	127	0.15461	9.24E-06	17	68	0.17408	4.43E-07	18	71	0.091337	9.61E-06
	x5	17	50	0.094204	6.93E-07	176	527	0.5584	9.25E-07	32	127	0.12309	9.24E-06	17	68	0.2895	4.43E-07	18	71	0.07598	9.61E-06
	x6	13	39	0.067243	7.52E-07	141	422	0.37855	9.26E-07	25	99	0.098141	7.80E-06	17	68	0.21341	4.50E-07	14	55	0.071792	9.08E-06
	x7	17	50	0.086087	6.93E-07	176	527	0.45244	9.25E-07	32	127	0.12887	9.24E-06	17	68	0.22261	4.43E-07	18	71	0.084721	9.61E-06
	x8	17	50	0.10848	6.93E-07	176	527	0.44368	9.25E-07	32	127	0.14797	9.24E-06	17	68	0.18262	4.50E-07	18	71	0.07619	9.61E-06
	x9	17	50	0.084791	7.39E-07	176	527	0.66094	9.24E-07	32	127	0.1238	9.25E-06	17	68	0.19609	4.36E-07	18	71	0.0846	9.64E-06
50000	x1	1	3	0.040787	0	184	551	2.0394	9.69E-07	34	135	0.59763	9.32E-06	16	64	0.87143	3.29E-07	1	4	0.037983	0
	x2	1	3	0.039382	0	170	509	1.9099	9.73E-07	10	39	0.12867	7.04E-06	14	56	0.60424	3.78E-07	10	39	0.21072	3.00E-06
	x3	1	3	0.035945	2.22E-16	134	401	1.2734	9.21E-07	24	96	0.31854	8.77E-06	12	48	0.36476	4.01E-07	14	56	0.24291	6.22E-06
	x4	17	51	0.26368	7.61E-07	183	548	1.8698	9.89E-07	34	135	0.56398	7.32E-06	16	64	0.68032	5.90E-07	19	75	0.25298	8.44E-06
	x5	17	51	0.23584	7.61E-07	183	548	1.8812	9.89E-07	34	135	0.54906	7.32E-06	16	64	0.67644	5.90E-07	19	75	0.28726	8.44E-06
	x6	13	39	0.20824	7.52E-07	141	422	1.6009	9.26E-07	25	99	0.42261	7.80E-06	17	68	0.62999	4.20E-07	14	55	0.18619	9.08E-06
	x7	17	51	0.25547	7.61E-07	183	548	1.8631	9.89E-07	34	135	0.55708	7.32E-06	16	64	0.65903	5.90E-07	19	75	0.31361	8.44E-06
	x8	17	51	0.24578	7.61E-07	183	548	3.6494	9.89E-07	34	135	0.44942	7.32E-06	16	64	0.68492	5.81E-07	19	75	0.31555	8.44E-06
	x9	17	51	0.27934	7.54E-07	183	548	2.0495	9.90E-07	34	135	0.54755	7.34E-06	16	64	0.59958	5.81E-07	19	76	0.3564	8.42E-06
100000	x1	1	3	0.06954	0	187	560	3.8112	9.99E-07	35	139	1.3182	7.85E-06	16	64	1.6176	4.65E-07	1	4	0.096628	0
	x2	1	3	0.060857	0	174	521	3.8221	9.03E-07	10	39	0.30533	9.96E-06	14	56	1.1566	5.34E-07	10	39	0.34704	4.24E-06
	x3	1	3	0.067384	2.22E-16	134	401	2.7412	9.21E-07	24	96	0.6021	8.77E-06	12	48	0.78582	4.01E-07	14	56	0.62844	6.22E-06
	x4	18	53	0.4831	6.58E-07	187	560	3.8213	9.18E-07	35	139	1.0063	6.17E-06	16	64	1.2757	8.35E-07	20	79	0.79438	4.68E-06
	x5	18	53	0.46322	6.58E-07	187	560	3.835	9.18E-07	35	139	1.256	6.17E-06	16	64	1.5281	8.35E-07	20	80	0.73412	4.68E-06
	x6	13	39	0.33844	7.52E-07	141	422	2.912	9.26E-07	25	99	0.71975	7.80E-06	17	68	1.408	4.17E-07	14	55	0.5084	9.08E-06
	x7	18	53	0.47895	6.58E-07	187	560	4.0575	9.18E-07	35	139	1.242	6.17E-06	16	64	1.3442	8.35E-07	20	80	0.7918	4.68E-06
	x8	18	53	0.48391	6.58E-07	187	560	4.1165	9.18E-07	35	139	1.0061	6.17E-06	16	64	1.4101	8.22E-07	20	80	0.72111	4.68E-06
	x9	18	54	0.49485	3.22E-07	187	560	3.9494	9.19E-07	35	139	1.1207	6.16E-06	16	64	1.3269	8.21E-07	20	80	0.81733	4.68E-06

To visualize the efficiency of HLSFR algorithms, we adopt the Dolan and More [31] performance profile. Figs. 1, 2 and 3 illustrates the performance profile of the five algorithms, where the performance indices are the total number of iterations, iterative number of function evaluations and CPU time of Tables 3, 4, 5, 6, 7, 8, 9, 10, 11 and 12 in the appendix section. It can be seen that the HLSFR algorithm is the best solver with probability of solving 60% and 68% of the test problems with the least number of iterations and function evaluation respectively.

Figure 1

Performance profiles for the number of iterations.

Figure 2

Performance profiles for number of function evaluations.

Figure 3

Performance profiles for CPU running time.

Application in compressive sensing

General description

Digital image processing plays an important role in medical sciences, biological engineering and other areas of science and engineering [32], [33], [34]. Let be a linear operator and be a sparse original signal. For a given observation that satisfies It is necessary to reconstruct the original signal from the linear system . However, the framework is typically ill-conditioned and grant infinite solutions. In this case, it is typical to look for the sparsest one among all solutions provided that b is gained from a profoundly sparse signal. As a rule, the Basis Pursuit denoising problem is appropriate where τ is a positive parameter.

In what follows, we give a short overview of the reformulation of (32) into a convex quadratic program by Figueiredo in [35]. Consider any vector x such that . The vector x can be rewritten as where and , for all with . Subsequently, we represent the -norm of a vector as , where is an n-dimensional vector with all element one. Hence, (32) can be rewritten as Moreover, from [35], with no difficulty, (33) can be rewritten as where , , , .

Obviously, H is a positive semi-definite matrix, which implies that equation (34) is a convex quadratic programming problem. Quite recently, equation (34) was translated into a linearly inequality problem by Xiao and Zhu [2] which is equivalent to where is said to be continuous and monotone, see [22], [36]. Therefore, (34) can be effectively solved using the HLSFR method.

Numerical results

In this subsection, our main focus is utilizing HLSFR Algorithm in the restoration of one dimensional sparse signal and image restoration. We begin the experiment with the restoration of a one dimensional sparse signal from its limited measurement with additive noise. Similar to [2], [37], [38], the quality of restoration is measured by using their mean squared error (MSE) defined by where is the original signal and x is the restored signal. The parameters for HLSFR were set as follows: , and .

The goal of our experiment is to recover a sparse signal of length n from k observations with . Due to the capacity restrictions of the PC, we select a small size signal with signal length of 1029 and sampling measurement of 512. The original signal contains 128 randomly non-zero elements. Furthermore, during experiment, a random Gaussian matrix A using the Matlab command is generated. In the test, the measurement b is computed by where δ is the Gaussian noise distributed as .

To evaluate the performance of HLSFR, we test it against similar algorithms which were specially designed to solve monotone nonlinear equations with convex constraints and reconstructing sparse signal in compressive sensing. These algorithms include: CGD [2], PCG [24] and IPBDF [39]. For fairness in comparing the algorithms, iteration process of all algorithms started at and terminated when where is the objective function and denotes the function value at . See Fig. 4 for the numerical results consisting of the original sparse signal, the measurement and the reconstructed signal by each algorithm. Moreover, in Fig. 5, we give a visual illustration of the performance of each method relative to their convergence behavior from the view of merit function values and relative error as the iteration numbers and computing time increases.

Figure 4

Reconstruction of sparse signal. From the top to the bottom is the original signal (First plot), the measurement (Second plot), and the reconstructed signals by CGD (Third plot), PCG (Fourth plot) and HLSFR (Fifth plot).

Figure 5

Comparison results of HLSFR, CGD, PCG and IPDBF algorithm. The x-axes represent the number of iterations (top left and bottom left) and the CPU time in seconds (top right and bottom right). The y-axes represent the MSE (top left and top right) and the function values (bottom left and right).

Comparing the four algorithms in Fig. 4, it is not difficult to see that the original signal was recovered by the four algorithms. However, HLSFR won in decoding sparse signal in compressive sensing. This is reflected by its lesser number of iterations, computing time and lesser MSE. To further illustrate the efficiency HLSFR, we repeated the experiment on 10 different noise samples. Each time the experiment is run, HLSFR proves to be more efficient than the CGD, PCG and IPDBF in terms of iteration numbers and CPU time and most importantly, MSE. See summary in Table 1.

Table 1

The experimental results of compressed sensing problem via CGD, PCG, IPDBF and HLSFR method.

	CGD			PCG			IPBDF			HLSFR
	CPU	ITER	MSE	CPU	ITER	MSE	CPU	ITER	MSE	CPU	ITER	MSE
	4.48	557	1.36E-03	9.30	164	3.45E-03	9.30	1189	2.85E-03	1.27	155	1.18E-03
	2.00	211	2.63E-03	9.27	179	1.44E-03	9.27	1186	1.32E-03	1.34	165	3.71E-04
	4.50	584	1.18E-03	9.91	175	3.33E-03	9.91	1305	2.29E-03	1.53	189	1.02E-03
	3.36	429	9.00E-04	9.28	197	1.31E-03	9.28	1202	1.35E-03	1.34	169	4.85E-04
	3.63	467	1.16E-03	9.81	267	1.51E-03	9.81	1329	1.76E-03	1.39	171	7.29E-04
	4.22	529	9.89E-04	10.36	156	3.01E-03	10.36	1287	1.72E-03	1.39	169	6.07E-04
	3.55	462	1.43E-03	8.11	217	1.94E-03	8.11	1153	2.47E-03	1.03	166	1.27E-03
	4.17	501	1.36E-03	9.44	164	2.69E-03	9.44	1204	2.20E-03	1.17	181	1.04E-03
	1.92	221	4.18E-03	9.03	169	3.31E-03	9.03	1113	3.01E-03	3.31	182	1.39E-03
	1.91	248	2.92E-03	10.72	212	1.86E-03	10.27	1429	1.68E-03	1.61	201	4.13E-04
Avg	3.37	420.9	1.81E-03	9.52	190	2.38E-03	9.48	1240	2.07E-03	1.54	174.8	8.52E-04

Next, we illustrate the performance of HLSFR algorithm in image restoration. In this experiment, a matrix A (partial DWT matrix) whose k rows are randomly selected from the DWT matrix. This type of matrix A requires no storage and helps in speeding up the matrix-vector multiplications involving A and . The test images we considered are personal images with color which were taken with a digital camera. These images include: TP1, TP2, TP3 and TP4. All test images are of the size except for TP1 which is .

The quality of image restoration is determined by signal-to-ratio (SNR) and the peak signal-to-noise ratios (PSNR). For the image restoration experiment, the chosen parameters for HLSFR are . See Fig. 6 for the original, blurred, and restored images by each algorithm.

Figure 6

Restoration of TP1 (Top), TP2 (Top-middle), TP3 (Bottom-middle) and TP4 (Bottom). From the left is the blurred with noise image, followed by the reconstructed images by CGD, IPDBF and HLSFR.

Furthermore, five Gaussian blur kernel were utilized in testing the efficiency of the methods. Their numerical performance is reported in the table that follows where denotes that the test problem i which is solved by a Gaussian blur kernel with standard deviation σ.

From Table 2, it can be observed that, under the five Gaussian blue kernel, the quality of the restored images by HLSFR is much better than that of CGD, IPBDF. This is reflected by smaller value of the ObjFunc and MSE. Similarly, larger SNR, SSIM and PSNR indicate that the restored images from the blurred images by HLSFR are much more closer to the original one than the recovered ones by CGD and IPDBF in most cases. The MATLAB implementation of the SSIM index can be obtained at http://www.cns.nyu.edu/~lcv/ssim/.

Table 2

Efficiency Comparison for restoration between CGD, IPDBF and HLSFR under different Gaussian blur Kernels.

IMAGE	CGD						IPDBF						HLSFR
	ObjFun	MSE	SNR	SSIM	PSNR	TIME	ObjFun	MSE	SNR	SSIM	PSNR	TIME	ObjFun	MSE	SNR	SSIM	PSNR	TIME
TP1(4)	1.01E+06	1.60E+05	-0.99	0.02	5.14	3.53	1.01E+06	1.15E+05	0.38	0.02	6.51	5.89	1.01E+06	1.11E+05	0.55	0.02	6.68	26.03
TP1(1)	2.66E+05	2.26E+04	7.46	0.12	13.59	2.81	2.65E+05	1.81E+04	8.42	0.14	14.55	4.67	2.65E+05	1.90E+04	8.2	0.13	14.34	25.94
TP1(0.1)	1.18E+04	2.65E+03	16.71	0.77	22.84	4.72	1.18E+04	2.59E+03	16.81	0.78	22.95	16.14	1.18E+04	2.58E+03	16.83	0.78	22.97	68.36
TP1(0.25)	7.33E+04	6.53E+03	12.87	0.35	19.01	3.86	7.33E+04	5.77E+03	13.41	0.39	19.55	9.56	7.33E+04	5.64E+03	13.52	0.39	19.65	42.81
TP1(6.25)	1.57E+06	3.36E+05	-4.15	0.01	1.98	3.58	1.57E+06	2.62E+05	-3.1	0.01	3.03	8.14	1.57E+06	2.56E+05	-2.98	0.01	3.15	39.63
TP2(4)	1.25E+06	2.06E+05	-1.58	0.11	5	3.97	1.25E+06	1.42E+05	-0.09	0.13	6.49	6.41	1.25E+06	1.39E+05	0.03	0.13	6.61	30.05
TP2(1)	3.24E+05	2.87E+04	6.82	0.35	13.4	3.23	3.24E+05	2.35E+04	7.7	0.38	14.28	5.44	3.23E+05	2.46E+04	7.5	0.37	14.08	30.05
TP2(0.1)	1.25E+04	5.05E+03	14.24	0.81	20.82	5.58	1.25E+04	4.93E+03	14.37	0.82	20.95	15.17	1.25E+04	4.91E+03	14.4	0.82	20.98	67.03
TP2(0.25)	8.81E+04	9.57E+03	11.53	0.56	18.11	4.67	8.81E+04	8.52E+03	12.03	0.58	18.61	11.61	8.81E+04	8.44E+03	12.08	0.59	18.66	48.84
TP2(6.25)	1.93E+06	4.47E+05	-4.83	0.06	1.75	4.28	1.93E+06	3.41E+05	-3.69	0.06	2.89	11	1.93E+06	3.31E+05	-3.59	0.06	2.99	54.53
TP3(4)	1.26E+06	1.99E+05	1.08	0.06	5.24	4.05	1.26E+06	1.42E+05	2.42	0.08	6.58	6.34	1.26E+06	1.38E+05	2.5	0.08	6.66	30.55
TP3(1)	3.30E+05	2.80E+04	9.37	0.25	13.52	3.38	3.30E+05	2.17E+04	10.44	0.28	14.59	5.14	3.28E+05	2.36E+04	10.12	0.28	14.28	31.22
TP3(0.1)	1.64E+04	3.46E+03	18.46	0.78	22.62	5.81	1.64E+04	3.36E+03	18.59	0.79	22.74	14.64	1.64E+04	3.34E+03	18.62	0.79	22.77	67.38
TP3(0.25)	9.20E+04	8.10E+03	14.72	0.46	18.87	4.66	9.19E+04	7.21E+03	15.29	0.48	19.44	11.39	9.20E+04	7.03E+03	15.36	0.49	19.52	49.52
TP3(6.25)	1.93E+06	4.72E+05	-2.74	0.03	1.42	4.95	1.93E+06	3.81E+05	-1.86	0.03	2.3	12.11	1.93E+06	3.74E+05	-1.75	0.03	2.4	57.45
TP4(4)	1.26E+06	2.00E+05	-0.19	0.05	5.1	4.31	1.25E+06	1.49E+05	1.14	0.06	6.42	7.42	1.26E+06	1.44E+05	1.3	0.06	6.58	32.02
TP4(1)	3.30E+05	3.25E+04	7.8	0.19	13.08	4.17	3.30E+05	2.65E+04	8.71	0.21	13.99	5.16	3.29E+05	2.84E+04	8.41	0.21	13.69	36.25
TP4(0.1)	1.71E+04	1.02E+04	13.24	0.63	18.52	4.16	1.70E+04	9.99E+03	13.33	0.64	18.61	9.63	1.71E+04	9.88E+03	13.36	0.64	18.65	42.03
TP4(0.25)	9.33E+04	1.41E+04	11.67	0.37	16.96	4.64	9.32E+04	1.31E+04	12.03	0.39	17.31	10.11	9.33E+04	1.29E+04	12.08	0.39	17.36	38.2
TP4(6.25)	1.93E+06	4.72E+05	-3.76	0.02	1.52	4.53	1.93E+06	3.85E+05	-2.83	0.03	2.45	11.52	1.93E+06	3.75E+05	-2.72	0.03	2.56	55.34

Conclusions

We have presented a hybrid conjugate gradient projection method for solving convex constrained nonlinear equations. The algorithm is a convex combination of two conjugate gradient algorithm for solving unconstrained optimization problem [15], [19]. Under some appropriate conditions, the global convergence of the method is established. Results from numerical experiment show that our method is practical, effective and out performs the CGD, PCG, PDY and ACGD for some given convex constraint benchmark test problems with dimension ranging from 5000 to 100,000 and different initial points. Furthermore, one major contribution of this article is the utilization of the proposed algorithm in solving the -norm regularized problem in compressive sensing. Computational results from reconstructing sparse signal and blurred images have shown that the proposed method is competitive with the compared ones.

Declarations

Author contribution statement

Abdulkarim Hassan Ibrahim: Conceived and designed the experiments; Wrote the paper.

Poom Kumam: Contributed reagents, materials, analysis tools or data.

Auwal Bala Abubakar: Performed the experiments; Wrote the paper.

Wachirapong Jirakitpuwapat: Performed the experiments.

Jamilu Abubakar: Analyzed and interpreted the data; Wrote the paper.

Funding statement

This work was supported by Theoretical and Computational Science (TaCS) Center under Computational and Applied Science for Smart research Innovation Cluster (CLASSIC), Faculty of Science, KMUTT. Abdulkarim Hassan Ibrahim was supported by the Petchra Pra Jom Klao Doctoral Scholarship, Academic for Ph.D. Program at KMUTT (Grant No. 16/2561).

Competing interest statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.