Robust Variable Selection and Estimation Based on Kernel Modal Regression.

Model-free variable selection has attracted increasing interest recently due to its flexibility in algorithmic design and outstanding performance in real-world applications. However, most of the existing statistical methods are formulated under the mean square error (MSE) criterion, and susceptible to non-Gaussian noise and outliers. As the MSE criterion requires the data to satisfy Gaussian noise condition, it potentially hampers the effectiveness of model-free methods in complex circumstances. To circumvent this issue, we present a new model-free variable selection algorithm by integrating kernel modal regression and gradient-based variable identification together. The derived modal regression estimator is related closely to information theoretic learning under the maximum correntropy criterion, and assures algorithmic robustness to complex noise by replacing learning of the conditional mean with the conditional mode. The gradient information of estimator offers a model-free metric to screen the key variables. In theory, we investigate the theoretical foundations of our new model on generalization-bound and variable selection consistency. In applications, the effectiveness of the proposed method is verified by data experiments.

1. Introduction

Variable selection has attracted increasing attention in the machine learning community due to the massive requirements of high-dimensional data mining. Under different motivations, many variable selection methods have been constructed and shown promising performance in various applications. From the viewpoint of hypothesis function space, there are mainly two types of variable selection approaches with respect to linear assumption and nonlinear additive assumption, respectively. For the linear model assumption, variable selection algorithms are usually formulated based on the least-squares empirical risk and the sparsity-induced regularization, which include Least Absolute Shrinkage and Selection Operator (Lasso) [1], Group Lasso [2] and Elastic net [3] as special examples. For the nonlinear additive model assumption, various additive models have been developed to relax the linear restriction on regression function [4,5]. It is well known that additive models enjoy the flexibility and interpretability of their representation and can remedy the curse of dimensionality of high-dimensional nonparametric regression [6,7,8]. Typical examples of additive models include Sparse Additive Models (SpAM) [9], Component Selection and Smoothing Operator (COSSO) [10] and Group Sparse Additive Models (GroupSpAM) [11]. Most of the above approaches are formulated under Tikhonov regularization scheme with special hypothesis function space (e.g., linear function space, nonlinear function space with additive structure).

More recently, some works have been made in [12,13,14,15] to alleviate the restriction on the hypothesis function space, which just require that the regression function belongs to a reproducing kernel Hilbert space (RKHS). In contrast to the traditional structure assumption on regression function, these methods identify the important variable via the gradient of kernel-based estimator. There are two strategies to improve the model flexibility through the gradient information of predictor. One follows the learning gradient methods in [13,14,16], where the functional gradient is used to construct the loss function for forming the empirical risk. Under this strategy, two model-free variable selection methods are presented by combining the error metric associated with the gradient information of estimator and the coefficient-based -regularizer in [13] and -regularizer in [14], respectively. In particular, the variable selection consistency is also established based on the properties of RKHS and mild parameter conditions (e.g., the regularization parameter, the width of kernel). The other follows the structural sparsity issue in [15,17], where the functional gradient is employed to construct the sparsity-induced regularization term. Rosasco et al. in [17] proposes a least-squares regularization scheme with nonparametric sparsity, which can be solved by an iterative procedure associated with the theory of RKHS and proximal methods. Magda et al. [15] introduces a nonparametric structured sparsity by considering two regularizers based on partial derivatives and offers its optimization with the alternating direction method of multiples (ADMM) [18]. Moreover, to further improve the computation feasibility, a three-step variable selection algorithm is developed in [12] with the help of the three building blocks: kernel ridge regression, functional gradient in RKHS, and a hard threshold. Meanwhile, the effectiveness of the proposed algorithm in [12] is supported by theoretical guarantees on variable selection consistency and empirical verification on simulated data.

Despite the aforementioned methods showing promising performance for identifying the active variables, all of them rely heavily on the least-squares loss under the MSE criterion, which is sensitive to non-Gaussian noise [19,20], e.g., the heavy-tailed noise, the skewed noise, and outliers. In essence, learning methods under MSE aim to find an approximator to the conditional mean based on empirical observations. When the data are contaminated by a complex noise without zero mean, the mean-based estimator is difficult to reveal with the intrinsic regression function. This motivates us to formulate a new variable selection strategy in terms of other criterion with respect to different statistical metric (e.g., the conditional mode). Following the research line in [12,19], we consider a new robust variable selection method by integrating the issues of modal regression (for estimating the conditional mode function) and variable screening based on functional derivatives. To the best of our knowledge, this is the first paper to address robust model-free variable selection.

Statistical models for learning the conditional mode can be traced back to [21,22], which include the local modal regression in [23,24] and the global modal regression in [25,26,27]. Recently, the idea of modal regression has been successfully incorporated into machine learning methods from theoretical analysis [19] and application-oriented studies (e.g., cognitive impairment prediction [20] and cluster estimation [28]). Particularly, Feng et al. [19] considers a learning theory approach to modal regression and illustrates some relations between modal regression and learning under the maximum correntropy criterion [29,30,31]. In addition, Wang et al. [20] formulates a regularized modal regression (RMR) under modal regression criterion (MRC), and establishes its theoretical characteristics on generalization ability, robustness, and sparsity. It is natural to extend the RMR under linear regression assumption to general model-free variable selection setting.

Inspired by recent works in [12,19], we propose a new robust gradient-based variable selection method (RGVS) by integrating the RMR in RKHS and the model-free strategy for variable screening. Here, the kernel-based RMR is used to construct the robust estimator, which can reveal the truly conditional mode, even when facing data with non-Gaussian noise and outliers. Moreover, we evaluate the information quantity of each input variable by computing the corresponding gradient of estimator. Finally, a hard threshold is used to identify the truly active variables after offering the empirical norm of each gradient associated with hypothesis function. The above three steps assure the robustness and flexibility of our new approach.

To better highlight the novelty of RGVS, we present Table 1 to illustrate its relation with other related methods, e.g., linear models (Lasso [1], RMR [20]), additive models (SpAM [9], COSSO [10]), and General variable selection Method (GM) [12].

Table 1

Properties of different regression algorithms.

	Lasso [1]	RMR [20]	SpAM [9]	COSSO [10]	GM [12]	Ours
Learning criterion	MSE	MRC	MSE	MSE	MSE	MRC
Model assumption	linear	linear	additive	additive	model-free	model-free

Our main contributions can be summarized as follows.

We formulate a new RGVS method by integrating the RMR in RKHS and the model-free strategy for variable screening. This algorithm can be implemented via the half-quadratic optimization [32]. To our knowledge, this algorithm is the first one for robust model-free variable selection.

In theory, the proposed method enjoys statistical consistency on regression estimator under much general conditions on data noise and hypothesis space. In particular, the learning rate with polynomial decay is obtained, which is faster than in [20] for linear RMR. It should be noted that our work is established under the MRC, while all previous model-free methods are formulated under the MSE criterion. In addition, variable selection consistency is obtained for our approach under a self-calibration condition.

In application, the proposed RGVS shows the empirical effectiveness on both simulated and real-world data sets. In particular, our approach can achieve much better performance than the model-free algorithm in [12] for complex noise data, e.g., containing Chi-square noise, Exponential noise, and Student noise. Experimental results together with theoretical analysis support the effectiveness of our approach.

The rest of this paper is organized as follows. After recalling the preliminaries of modal regression, we formulate the RGVS algorithm in Section 2. Then, theoretical analysis, optimization algorithm, and empirical evaluation are provided from Section 3 to Section 5 respectively. Finally, we conclude this paper in Section 6.

2. Gradient-Based Variable Selection in Modal Regression

Let and be a compact input space and an output space, respectively. We consider the following data-generating setting where , and is a random noise. For the feasibility of theoretical analysis, we denote the intrinsic distribution of generated in (1) as . Let be empirical observations drawn independently according to the unknown distribution . Unlike sparse methods with certain model assumption (e.g., Lasso [1], SpAM [9]), the gradient-based sparse algorithms [12,13] mainly aim at screening out the informative variables according to the gradient information of intrinsic function. For input vector , the variable information is characterized by the gradient function . Clearly, implies that the j-th variable is uninformative [12,17]. Considering an -norm measure on the partial derivatives, we denote the true active set as where and is the marginal distribution of .

Indeed, all the gradient-based variable selection algorithms [12,13,17] are constructed under Tikhonov regularization scheme in RKHS [33,34]. The RKHS associated with the Mercer kernel K is the closure of the linear span of . Such a Mercer kernel is a symmetric and positive semi-definite function. Denote as the inner product in , the reproducing properties of RKHS means .

2.1. Gradient-Based Variable Selection Based on Kernel Least-Squares Regression

In this subsection, we recall the gradient-based variable selection algorithm in [12] associated with least-squares error metric. When the noise in (1) satisfies (i.e., Gaussian noise), the regression function equals to the conditional mean, which can be represented by

Here denotes the conditional distribution of Y given x. Theoretically, the regression function in (3) is the minimizer of expected least-squares risk

As is unknown in practice, we cannot get directly by minimizing over certain hypothesis space. Given training samples , the empirical risk with respect to the expected risk is denoted by

The gradient-based variable selection algorithm in [12] depends on the estimator defined as below: where is the regularization parameter and is the kernel-norm of f. The properties of RKHS [33] assure that where and . Denote and , the closed-form solution is

Following Lemma 1 in [12], for any , we have , where

After imposing the empirical norm on , i.e., we get the estimated active set where is a pre-configured constant for variable selection.

The general variable selection method has shown some theoretical advantages in [12], e.g., the representation flexibility and the computation feasibility. However, the gradient-based method [12] may result in a degraded performance for real-world data without the zero-mean noise condition. Inspired by the modal regression [19,35] to learn the conditional mode, we propose a new robust gradient-based variable selection method under much general noise condition.

2.2. Robust Gradient-Based Variable Selection Based on Kernel Modal Regression

Unlike the traditional zero-mean noise assumption [12,17], the modal regression requires that the conditional mode of random noise is zero for any , i.e., where is the conditional density of conditioned on X. In fact, this assumption imposes no restrictions on conditional mean, and can include the heavy-tailed noise, the skewed noise, and outliers. Then, we can verify that the mode-regression function where denotes the conditional density of Y conditioned on . It is worth noting that is assumed to be unique and existing here. As shown in [19,20], in (6) is the maximizer of the MRC over all measurable functions, which is defined as

The maximizer of is difficult to be obtained since both and are unknown. Fortunately, Theorem 5 of [19] has proved that , where is the density function of at 0 and which can be easily approximated by the kernel density method [20]. With the help of modal kernel for the density estimation, we can formulate an empirical kernel density estimator at 0

Setting , we get the corresponding expected version

In addition, the modal regression also can be interpreted by minimizing a mode-induced error metric [19]. When for any , the mode-induced loss can be defined as which is related closely with the correntropy-induced loss in [19,36]. Given training samples , we can formulate the RMR in RKHS as where is a turning parameter that controls the complexity of the hypothesis space, and is the kernel-norm of .

Denote , and . From the representer theorem of kernel methods, we can deduce that with

From Lemma 1 in [12], we know that for any and ,

The empirical measure on gradient function is

Then, the identified active set can be written as where is a positive threshold selected under the sample-adaptive tuning framework [37].

3. Generalization-Bound and Variable Selection Consistency

This section establishes the theoretical guarantees on generalization ability and variable selection for the proposed RGVS. Firstly, we introduce some necessary assumptions.

The representing function ϕ associated with modal kernel

Observe that some smoothing kernels meet Assumption 1, such as Gaussian kernel and Logistic kernel, etc.

The conditional density function

Assumption 2 has been used in [19,20], which assures upper bound on together with Assumption 1.

Let

Assumption 3 has been used extensively in learning theory literatures, see, e.g., [38,39,40,41,42,43,44]. In particular, the Gaussian kernel belongs to .

Our error analysis begins with the following inequality in [19], where the relationship between and is provided.

Under Assumptions 1–2, there holds for any measurable function f:

This indicates us to bound the excess risk via estimating . To be specific, we further make an error decomposition as follows.

Under Assumptions 1–3, there holds

According to the definition of in (9), we have

Then, we can deduce that

This together with Lemma 1 yields the desired result. □

Observe that characterizes the divergence between the data-free risk and the empirical risk . To establish its uniform estimation, we need to give the upper bound of firstly.

According to the definition of , we have

Then,

For

Lemma 3 tells us that with for any , where . This motivates us to measure the capacity of through the empirical covering number [45].

Suppose that where with

Next, we introduce a concentration inequality established in [46].

Let where

Under Assumptions 1–3, taking with confidence at least

Denote a function-based random variable set by

Under Assumption 1, for any , we have

Combining the above inequality and the properties of empirical covering number [40,41], we have where is defined in (13).

According to Assumption 1, there exists . Furthermore, we get where and are the bounded constants.

Recalling (14) and (15), we know Lemma 4 holds true for with , , , and . That is to say, for any and , with confidence

Combining Lemma 2 and (16) with , we have with confidence at least where is positive constants independently of .

Setting and , we have and . Putting these selected parameters into (17), we get the desired estimation. □

Theorem 1 provides the upper bound to the excess risk of under the MRC, which extends the previous ERM-based analysis in [19] to the regularized learning scheme. In addition, we can further bound after imposing Assumption 3 in [19].

Let the conditions of Theorem 1 be true. Assume that with confidence at least

The learning rate derived in Corollary 1 is faster than for the linear regularized modal regression [20]. Meanwhile, it should be noted that some kernel functions meet , e.g., Gaussian kernel, Sigmoid kernel, and Logistic kernel.

Since the proposed RGVS employs the non-convex mode-induced loss, our variable selection analysis is completely different from kernel method with least-squares loss [12]. Here, we introduce the following self-calibration inequality, which addresses that a weak convergence on risk implies a strong convergence in kernel-norm under certain conditions.

For any given σ and

Assumption 4 characterizes the concentration of our estimator near with the kernel-norm metric. Indeed, the current restriction is related to Assumption 4 in [12], Theorem 2.7 in [47] for quartile regression, and the so-called RNI condition in [48,49] as well.

In addition, the following condition is required, which implies that the gradient function associated with truly informative variables is separated well from zero. Similar assumptions can also be found in [12,50]. For simplicity, we denote .

There exists some constant

Let Assumptions 1–5 be true. For any given

As shown in [12], by direct computation, there holds where denotes the Hilbert-Schmidt operator on , , , and is a positive constant. The concentration inequality for kernel operator in [17] states that with confidence .

Meanwhile, with the similar proof of Theorem 1, we can deduce that with confidence at least , where is a positive constant. Setting and , we further get with confidence . This excess risk estimation together with Assumption 4 implies that with confidence , where is a positive constant.

Combining (18)–(20), we have with confidence where is a constant independently of .

Now we turn to investigate the relationship between in (12) and in (2). Firstly, we suppose there exists some but . That is to say . By Assumption 5 with , we have which contradicts with (21). This implies that with confidence .

Secondly, we suppose there exists some but . This means and . Then which contradicts with (21) with confidence . Therefore, the desired property follows by combining these two results. □

Theorem 2 demonstrates that the identified variables are consistent with truly informative variables with probability 1 as . This result guarantees the variable selection performance of our approach, provided that the active variables have enough gradient signal. In the future, it is necessary to further investigate the self-calibration assumption for RMR in RKHS.

When choosing Gaussian kernel as the modal kernel, the modal regression is consistent with regression under the maximum correntropy criterion (MCC) [36]. In terms of the breakdown point theory, Theorem 24 in [19] established the robustness characterization of kernel regression under MCC and Theorem 3 in [36] provided robust analysis for RMR. These results imply the robustness of our approach.

4. Optimization Algorithm

With the help of half-quadratic (HQ) optimization [32], the maximization problem (9) can be transformed into a weighted least-squares problem, and then get the estimator via the ADMM [18]. Indeed, the kernel-based RMR (9) can be implemented directly by the optimization strategy in [36,51] for Gaussian kernel-based modal representation, and in [20] for Epanechnikov kernel-based modal representation. For completeness, we provide the optimization steps of (9) associated with Logistic kernel-based density estimation.

Consider a convex function

As illustrated in [52], a convex function and its convex conjugate function satisfy

According to the Logistic-based representation and (22), we have

Applying (23) into (10), we can obtain the augmented objective function where , and is the auxiliary vector. Then the maximization problem (24) can be solved by the following iterative optimization algorithm.

According to Theorem 1 in [20], we have . Then, for a fixed , can be updated by . While b is settled down, update via

For and , the problem (25) can be rewritten as where is an operator that transforms the vector into a diagonal matrix. By setting , we have

When is obtained from (26), we can calculate the gradient-based measure by (11) directly. Then we apply a pre-specified threshold to identify the truly active set . Here, the threshold is selected by the stability-based criterion [37], which include two steps as below. Firstly, the training samples are randomly divided into two subsets, and the identified active variable sets and are obtained under given for the k-th splitting of training samples. Then, the threshold is updated by maximizing the Cohen kappa statistical measure .

The optimization steps of RGVS are summarized in Algorithm 1.

5. Empirical Assessments

This section assesses the empirical performance of our proposed method on simulated and real-world datasets. Three variable selection methods are introduced as the baselines, which include Least Absolute Shrinkage and Selection Operator (Lasso) [1], Sparse Additive Models (SpAM) [9], and General Variable Selection Method (GM) [12].

In all experiments, the RKHS associated with Gaussian kernel is employed as the hypothesis function space. For simplicity, we denote and as the proposed RGVS method with Gaussian modal kernel and Logistic modal kernel, respectively. In the simulated experiments, we generate three datasets (with identical sample size) independently as the training set, the validation set, and the testing set, respectively. The hyper-parameters are tuned via grid research on validation set, and the corresponding grids are displayed as follows: the regularization parameter : ; the bandwidth and h: ; the threshold : }.

5.1. Simulated Data

Now we evaluate our approach on two synthetic data used in [12,13]. The first example is a simple additive function and the second one is a function that includes interaction terms.

We generate the p-dimension input

This example follows the way of Example 1 to generate data. The differences are that

For each evaluation, we consider training set with different size and dimension . To make sure the results are reliable, each evaluation is repeated 50 times. Since the truly informative variables are usually unknown in practice, we evaluate the algorithmic performance according to the average squares error(ASE) defined as . To better evaluate the algorithmic performance, we also adopt some metrics used in [12,13] to measure the quality of model regression, e.g., Cp (correct-fitting), SIZE (the average number of selected variables), TP (the average number of the selected true informative variables), FP (the average number of the selected uninformative variables), Up (under-fitting probability), Op (over-fitting probability). The detail result is summarized in Table 2 and Table 3. To further support the competitive performance of the proposed method, we also provide the experimental results on ASE in Figure 1 and Cp in Figure 2 with and . Figure 1 and Figure 2 show that our method has always performed well with different n.

Table 2

The averaged performance on simulated data in Example 1 (left) and Example 2 (right).

Noise	(n,p)	Method	SIZE	TP	FP	Up	Op	Cp	ASE	SIZE	TP	FP	Up	Op	Cp	ASE
	(100, 150)	Lasso	3.92	3.92	0.00	0.36	0.00	0.64	1.369	4.40	4.28	0.12	0.44	0.12	0.44	5.112
	n<p	SpAM	4.12	3.92	0.20	0.08	0.16	0.76	1.075	5.02	4.98	0.04	0.04	0.04	0.92	1.611
		GM	4.12	3.88	0.24	0.12	0.16	0.72	1.123	5.14	4.98	0.16	0.04	0.12	0.84	1.775
		RGVS_Gau	4.00	3.92	0.08	0.08	0.04	0.88	1.003	5.12	5.00	0.12	0.00	0.08	0.92	1.565
		RGVS_Log	3.84	3.80	0.04	0.20	0.04	0.76	1.131	5.12	4.92	0.20	0.08	0.16	0.76	1.914
N(0,1)	(150, 150)	Lasso	4.20	3.92	0.04	0.16	0.12	0.72	1.245	4.48	4.28	0.20	0.40	0.16	0.44	4.794
Gaussian Noise	n=p	SpAM	4.00	4.00	0.00	0.00	0.00	1.00	0.804	5.00	5.00	0.00	0.00	0.00	1.00	1.612
		GM	3.96	3.92	0.04	0.08	0.04	0.88	1.011	5.04	5.00	0.04	0.00	0.04	0.96	1.627
		RGVS_Gau	3.96	3.96	0.00	0.04	0.00	0.96	0.899	5.00	5.00	0.00	0.00	0.00	1.00	1.500
		RGVS_Log	3.96	3.80	0.04	0.08	0.04	0.88	1.083	5.02	5.00	0.02	0.00	0.03	0.97	1.622
	(200, 150)	Lasso	4.00	3.92	0.08	0.20	0.00	0.80	1.252	4.52	4.52	0.00	0.40	0.00	0.60	2.507
	n>p	SpAM	4.00	4.00	0.00	0.00	0.00	1.00	0.829	5.04	5.00	0.04	0.00	0.02	0.98	1.497
		GM	3.96	3.96	0.00	0.04	0.00	0.96	1.012	5.06	5.00	0.06	0.00	0.04	0.96	1.528
		RGVS_Gau	4.00	4.00	0.00	0.00	0.00	1.00	0.813	5.00	5.00	0.00	0.00	0.00	1.00	1.485
		RGVS_Log	3.96	3.96	0.00	0.04	0.00	0.96	0.915	5.00	5.00	0.00	0.00	0.00	1.00	1.453
	(100, 150)	Lasso	3.72	3.64	0.08	0.36	0.04	0.60	5.854	4.32	3.72	0.60	0.68	0.12	0.20	6.888
	n<p	SpAM	4.24	3.76	0.48	0.24	0.28	0.48	4.342	5.52	5.00	0.52	0.00	0.40	0.60	4.328
		GM	4.24	3.80	0.44	0.20	0.36	0.44	4.012	4.48	4.44	0.04	0.30	0.05	0.65	3.611
		RGVS_Gau	4.16	3.96	0.20	0.04	0.20	0.76	2.937	5.08	4.92	0.16	0.06	0.16	0.78	3.486
		RGVS_Log	4.18	3.90	0.28	0.16	0.12	0.72	2.681	5.08	4.84	0.24	0.08	0.18	0.74	3.968
X^2(2)	(150, 150)	Lasso	5.32	3.84	1.48	0.16	0.48	0.36	5.392	5.16	4.16	1.00	0.60	0.16	0.24	4.503
Chi-square Noise	n=p	SpAM	4.04	3.96	0.08	0.04	0.08	0.88	2.765	5.32	5.00	0.32	0.00	0.24	0.76	3.748
		GM	4.00	3.88	0.12	0.12	0.08	0.80	2.873	4.98	4.92	0.06	0.05	0.05	0.90	4.173
		RGVS_Gau	3.96	3.92	0.04	0.08	0.04	0.88	2.809	5.02	5.00	0.02	0.00	0.02	0.98	2.929
		RGVS_Log	4.08	3.96	0.12	0.04	0.12	0.84	2.097	5.04	5.00	0.04	0.00	0.04	0.96	3.519
	(200, 150)	Lasso	4.24	4.00	0.24	0.00	0.28	0.72	5.805	4.36	4.32	0.04	0.52	0.04	0.44	3.754
	n>p	SpAM	4.08	4.00	0.08	0.00	0.08	0.92	2.463	5.04	5.00	0.04	0.00	0.04	0.96	3.634
		GM	4.04	4.00	0.04	0.00	0.04	0.96	2.523	5.18	5.00	0.18	0.00	0.20	0.80	3.816
		RGVS_Gau	3.96	3.96	0.00	0.04	0.00	0.96	2.449	5.00	5.00	0.00	0.00	0.00	1.00	2.989
		RGVS_Log	3.96	3.96	0.00	0.04	0.00	0.96	1.738	5.00	5.00	0.00	0.00	0.00	1.00	3.457
	(100, 150)	Lasso	3.46	3.46	0.00	0.60	0.00	0.40	4.631	4.64	4.00	0.64	0.60	0.20	0.20	4.567
	n<p	SpAM	4.28	3.88	0.40	0.12	0.28	0.60	4.599	5.84	5.00	0.84	0.00	0.44	0.56	4.224
		GM	4.20	3.64	0.56	0.36	0.36	0.28	3.941	5.36	4.68	0.68	0.32	0.28	0.40	4.528
		RGVS_Gau	4.20	3.88	0.32	0.12	0.20	0.68	3.274	5.06	4.82	0.24	0.14	0.14	0.72	3.907
		RGVS_Log	3.96	3.80	0.16	0.20	0.16	0.64	2.775	5.12	4.92	0.20	0.04	0.16	0.80	3.667
E(2)	(150, 150)	Lasso	5.30	3.66	1.64	0.20	0.44	0.36	4.747	5.64	4.16	1.48	0.48	0.28	0.24	4.786
Exponential Noise	n=p	SpAM	4.04	3.96	0.08	0.04	0.08	0.88	3.403	5.28	5.00	0.28	0.16	0.00	0.84	4.969
		GM	4.08	3.96	0.12	0.04	0.12	0.84	3.177	4.98	4.92	0.06	0.08	0.04	0.88	4.129
		RGVS_Gau	4.00	4.00	0.00	0.00	0.00	1.00	2.724	5.02	4.98	0.04	0.02	0.04	0.94	2.964
		RGVS_Log	4.00	4.00	0.00	0.00	0.00	1.00	2.643	5.00	4.96	0.04	0.04	0.04	0.92	3.918
	(200, 150)	Lasso	3.80	3.80	0.00	0.20	0.00	0.80	4.291	4.68	4.60	0.08	0.28	0.08	0.64	3.669
	n>p	SpAM	4.00	4.00	0.00	0.00	0.00	1.00	2.988	5.24	5.00	0.24	0.00	0.20	0.80	4.808
		GM	3.96	3.96	0.00	0.04	0.00	0.96	3.016	4.98	4.98	0.00	0.04	0.00	0.96	3.878
		RGVS_Gau	4.00	4.00	0.00	0.00	0.00	1.00	2.884	5.00	5.00	0.00	0.00	0.00	1.00	3.041
		RGVS_Log	3.96	3.92	0.04	0.09	0.00	0.91	3.113	4.96	4.96	0.00	0.04	0.00	0.96	3.771
	(100, 150)	Lasso	4.92	3.80	1.12	0.28	0.32	0.40	2.301	6.52	3.92	2.60	0.64	0.20	0.16	6.971
	n<p	SpAM	4.90	3.80	1.1	0.24	0.20	0.56	1.698	7.92	4.72	3.20	0.24	0.44	0.32	4.658
		GM	5.00	3.64	1.36	0.32	0.32	0.36	1.551	5.68	4.32	1.32	0.40	0.32	0.28	3.561
		RGVS_Gau	4.14	3.94	0.20	0.05	0.10	0.85	0.822	5.00	4.84	0.16	0.08	0.16	0.76	2.308
		RGVS_Log	4.14	3.88	0.26	0.12	0.16	0.72	1.208	4.96	4.80	0.16	0.16	0.12	0.72	2.339
t(2)	(150, 150)	Lasso	5.08	3.72	1.36	0.24	0.40	0.36	1.793	6.32	3.80	2.52	0.68	0.20	0.12	6.020
Student Noise	n=p	SpAM	4.30	4.00	0.30	0.00	0.32	0.68	0.955	5.44	5.00	0.44	0.00	0.28	0.72	2.739
		GM	4.04	3.80	0.24	0.16	0.16	0.68	1.046	5.56	4.60	0.96	0.28	0.08	0.64	2.557
		RGVS_Gau	4.00	4.00	0.00	0.00	0.00	1.00	0.757	4.98	4.98	0.00	0.08	0.00	0.92	1.716
		RGVS_Log	3.92	3.88	0.04	0.12	0.04	0.84	1.169	4.96	4.96	0.00	0.04	0.00	0.96	1.723
	(200, 150)	Lasso	5.00	3.92	1.08	0.32	0.20	0.48	1.262	5.44	4.36	1.08	0.44	0.28	0.28	2.976
	n>p	SpAM	4.10	4.00	0.10	0.00	0.27	0.73	1.060	5.64	5.00	0.64	0.00	0.28	0.72	2.427
		GM	4.00	3.96	0.04	0.04	0.04	0.92	1.011	5.20	4.72	0.48	0.20	0.04	0.76	2.350
		RGVS_Gau	4.04	4.00	0.04	0.00	0.10	0.90	0.681	5.00	5.00	0.00	0.00	0.00	1.00	1.517
		RGVS_Log	4.04	4.00	0.04	0.00	0.04	0.96	0.884	4.96	4.96	0.00	0.04	0.00	0.96	1.672

Table 3

The averaged performance with simulated data in Example 1.

Noise	(n,p)	Method	SIZE	TP	FP	Up	Op	Cp	ASE
	(300, 500)	Lasso	1.98	1.98	0.00	1.00	0.00	0.00	1.98
	n<p	GM	4.04	4.00	0.04	0.00	0.04	0.96	0.80
		RGVS_Gau	4.06	4.00	0.06	0.00	0.06	0.94	0.63
		RGVS_Log	4.14	3.98	0.16	0.01	0.03	0.96	0.88
N(0,1)	(500, 500)	Lasso	1.92	1.92	0.00	1.00	0.00	0.00	1.35
Gaussian Noise	n=p	GM	4.06	4.00	0.06	0.00	0.06	0.94	0.78
		RGVS_Gau	4.02	4.00	0.02	0.00	0.02	0.98	0.59
		RGVS_Log	4.04	4.00	0.04	0.00	0.02	0.98	0.74
	(700, 500)	Lasso	1.88	1.88	0.00	1.00	0.00	0.00	1.55
	n>p	GM	4.04	4.00	0.04	0.00	0.04	0.96	0.77
		RGVS_Gau	4.02	4.00	0.02	0.00	0.02	0.98	0.62
		RGVS_Log	4.00	4.00	0.00	0.00	0.00	1.00	0.73
	(300, 500)	Lasso	1.80	1.80	0.00	1.00	0.00	0.00	4.45
	n<p	GM	4.18	4.00	0.18	0.00	0.14	0.86	2.92
		RGVS_Gau	4.09	4.00	0.09	0.00	0.11	0.89	2.39
		RGVS_Log	4.06	3.88	0.18	0.12	0.14	0.74	1.95
X^2(2)	(500, 500)	Lasso	1.74	1.74	0.00	1.00	0.00	0.00	4.62
Chi-square Noise	n=p	GM	4.14	4.00	0.14	0.00	0.14	0.86	3.01
		RGVS_Gau	4.08	4.00	0.08	0.00	0.06	0.94	2.22
		RGVS_Log	4.04	3.98	0.06	0.02	0.06	0.92	1.82
	(700, 500)	Lasso	1.86	1.86	0.00	1.00	0.00	0.00	4.37
	n>p	GM	4.28	4.00	0.28	0.00	0.24	0.76	2.96
		RGVS_Gau	4.02	4.00	0.02	0.00	0.02	0.98	2.13
		RGVS_Log	4.02	4.00	0.02	0.00	0.02	0.98	1.72
	(300, 500)	Lasso	2.04	2.04	0.00	1.00	0.00	0.00	4.25
	n<p	GM	3.94	3.87	0.07	0.13	0.05	0.82	3.14
		RGVS_Gau	4.02	4.00	0.02	0.00	0.02	0.98	2.36
		RGVS_Log	3.98	3.94	0.04	0.06	0.02	0.92	1.92
E(2)	(500, 500)	Lasso	1.94	1.94	0.00	1.00	0.00	0.00	4.34
Exponential Noise	n=p	GM	4.12	4.00	0.12	0.00	0.10	0.90	2.35
		RGVS_Gau	3.99	3.96	0.03	0.04	0.03	0.93	2.37
		RGVS_Log	4.02	4.00	0.02	0.00	0.02	0.98	1.71
	(700, 500)	Lasso	1.90	1.90	0.00	1.00	0.00	0.00	4.67
	n>p	GM	4.08	4.00	0.08	0.00	0.06	0.94	2.33
		RGVS_Gau	3.99	3.99	0.00	0.01	0.00	0.99	1.74
		RGVS_Log	4.05	4.00	0.05	0.00	0.05	0.95	1.92
	(300, 500)	Lasso	1.96	1.96	0.00	1.00	0.00	0.00	4.63
	n<p	GM	3.50	3.46	0.04	0.24	0.04	0.72	2.48
		RGVS_Gau	4.14	3.94	0.20	0.06	0.10	0.84	0.82
		RGVS_Log	4.00	3.98	0.02	0.02	0.00	0.98	0.90
t(2)	(500, 500)	Lasso	1.76	1.76	0.00	0.98	0.00	0.02	3.83
Student Noise	n=p	GM	4.30	4.00	0.30	0.00	0.16	0.84	1.96
		RGVS_Gau	4.00	4.00	0.00	0.00	0.00	1.00	0.76
		RGVS_Log	4.02	4.00	0.02	0.00	0.01	0.99	0.75
	(700, 500)	Lasso	1.96	1.96	0.00	0.96	0.00	0.04	2.46
	n>p	GM	4.06	4.00	0.06	0.00	0.04	0.96	1.95
		RGVS_Gau	4.04	4.00	0.04	0.00	0.06	0.94	0.68
		RGVS_Log	4.00	4.00	0.00	0.00	0.00	1.00	0.74

Figure 1

The average squares error (ASE) vs. the sample size n under different noise (A and B represent Example 1. and Example 2 respectively).

Figure 2

The correct-fitting probability (Cp) vs. the sample size n under different noise (A and B represent Example 1. and Example 2 respectively).

Empirical evaluations on simulated examples verify the promising performance of RGVS on variable selection and regression estimation, even for data with non-Gaussian noises (e.g., the Chi-square noise , the Student noise , and the Exponential noise ). Meanwhile, GM and RGVS have similar performance under the Gaussian noise setting, which is consistent with our motivation for algorithmic design.

5.2. Real-World Data

We now evaluate our RGVS on Auto-Mpg and Requirements of buildings, which are all collected from UCI. Since the variable number is very limited for the current datasets, 100 irrelative variables are added, which are generated from the distribution of .

Auto-Mpg data describes the mile per gallon of automobile (MPG). It contains 398 samples and 7 variables, including Cylinders, Displacement, Horsepower, Weight, Acceleration, Model year, and Origin. The second real data sets is obtained to assess the heating load and cooling load requirements of buildings which contains 768 samples and 8 input variables, including Relative Compactness, Surface Area, Wall Area, Roof Area, Overall Height, Orientation, Glazing Area, and Glazing Area Distribution. In particular, it has two response variables (heating load and cooling load).

Now, we use the 5-fold cross validation to tune the hyper-parameters and employ the relative sum of the squared errors (RSSE) to measure learning performance. Here , where is the estimator of f and denotes the average value of f on the test set . Experimental results are reported in Table 4 and Table 5.

Table 4

Learning performance on Auto-Mpg.

Variable	CyL	DISP	HPOWER	WEIG	ACCELER	YEAR	ORIGN	RSSE(std)
Lasso	-	-	-	✔	-	✔	-	0.5918(0.3762)
SpAM	✔	✔	-	✔	-	-	-	0.2754(0.0191)
GM	✔	✔	✔	✔	-	✔	✔	0.2547(0.0313)
RGVS_Gau	✔	✔	✔	✔	-	✔	-	0.1425(0.0277)
RGVS_Log	✔	✔	✔	✔	-	✔	✔	0.1379(0.0183)

Table 5

Learning performance on Heating Load (UP) and Cooling Load (DOWN).

Variable	RC	SA	WA	RA	OH	ORIENT	GA	GAD	RSSE(std)
Lasso	-	-	-	-	✔	-	✔	-	0.1739(0.0801)
SpAM	-	-	-	✔	✔	-	-	-	0.1684(0.0045)
GM	✔	✔	✔	✔	✔	-	-	-	0.1244(0.0383)
RGVS_Gau	-	✔	✔	✔	✔	-	✔	-	0.0935(0.0099)
RGVS_Log	-	-	✔	✔	✔	-	✔	-	0.1110(0.0066)
Lasso	-	-	✔	-	✔	-	✔	-	0.2119(0.0926)
SpAM	-	-	-	✔	✔	-	-	-	0.1910(0.0131)
GM	✔	✔	✔	✔	✔	-	-	-	0.1515(0.0120)
RGVS_Gau	-	✔	✔	✔	✔	-	✔	-	0.1339(0.0116)
RGVS_Log	✔	✔	✔	✔	✔	-	-	-	0.1368(0.0077)

As shown in Table 4, our method identifies similar variables as GM, but can achieve the smaller RSSE. At same time, SpAM and Lasso tend to select less variables than GM and RGVS, which may discard the truly informative variable for regression estimation. Table 5 shows RGVS has better performance for both the Heating Load data and the Cooling Load data. All these empirical evaluations validate the effectiveness of our learning strategy consistently.

6. Conclusions

This paper proposes a new RGVS method rooted in kernel modal regression. The main advantages of RGVS are its flexibility on mimicking the decision function and adaptivity on screening the truly active variables. The proposed approach is evaluated by the theoretical analysis on the generalization error and variable selection, and by the empirical results on data experiments. In theory, our method can achieve the polynomial decay rate with . In applications, our model has shown the competitive performance for data with non-Gaussian noises.