Regularized Variance Estimation and Variance Stabilization of High Dimensional Data.

Among the problems posed by high-dimensional datasets (so called p ≫ n paradigm) are that variable-specific estimators of variances are not reliable and tests statistics have low powers, both due to a lack of degrees of freedom. In addition, variance is observed to be a function of the mean. We introduce a non-parametric adaptive regularization procedure that uses the information contained in the mean to jointly generate local shrinkage estimators of the mean and variance. Regularized t-like statistics derived from these shrinkage estimators have significant more statistical power than their standard sample counterparts, regular common-value shrinkage estimators, or when the information contained in the sample mean is ignored. These estimators feature interesting properties of variance stabilization and normalization that can be used for preprocessing high-dimensional multivariate data.