Semi-Parametric Estimation Using Bernstein Polynomial and a Finite Gaussian Mixture Model.

The central focus of this paper is upon the alleviation of the boundary problem when the probability density function has a bounded support. Mixtures of beta densities have led to different methods of density estimation for data assumed to have compact support. Among these methods, we mention Bernstein polynomials which leads to an improvement of edge properties for the density function estimator. In this paper, we set forward a shrinkage method using the Bernstein polynomial and a finite Gaussian mixture model to construct a semi-parametric density estimator, which improves the approximation at the edges. Some asymptotic properties of the proposed approach are investigated, such as its probability convergence and its asymptotic normality. In order to evaluate the performance of the proposed estimator, a simulation study and some real data sets were carried out.

1. Introduction

Density estimation is a widely adopted tool for multiple tasks in statistical inference, machine learning, visualization and exploratory data analysis. Existing density estimation algorithms can be categorized into either parametric, semi-parametric, or non-parametric approaches. In the non-parametric framework, several methods have been set forward for the smooth estimation of density and distribution functions. The most popular one, called kernel method, was introduced by [1]. The advances were carried out by [2] to estimate a density function. The reader is recommended to consult the paper [3] for an introduction to several kernel smoothing techniques. However, kernel methods display estimation problems at the edges, when we have a random variable X with density function f supported on a compact interval. Moreover, if is a sample with the same density f, it is well known, in non-parametric kernel density estimation, that the bias of the standard kernel density estimator is of a larger order near the boundary than that in the interior, where K is a kernel (that is, a positive function satisfying ) and () is a bandwidth (that is, a sequence of positive real numbers that goes to zero). Let us now suppose that f has two continuous derivatives everywhere and that, as , and . Let for . Near the boundary, the expression of the mean and the variance are indicated as and These bias phenomena are called boundary bias. Numerous authors have elaborated methods for reducing these phenomena, such as data reflection [4], boundary kernels [5,6,7], local linear estimator [8,9], use of beta and gamma kernels [10,11] and bias reduction [12,13]. For a smooth estimator of a density function f with finite known support, there have been several methods, such as Vitale’s method [14], which is based on Bernstein polynomials and expressed as where is the empirical distribution function and is the Bernstein polynomial. This estimator was investigated in the literatures [15,16,17,18] and, more recently, by [12,19,20].

Within the parametric framework, it is noteworthy that the Gaussian mixture model can be used to estimate any density function, without any problem of estimation on the edge. This refers to the fact that the set of all normal mixture densities is dense in the set of all density functions under the metric [21]. The investigation of mixture models stands for a full field in modern statistics. It is a probabilistic model introduced by [22] to illustrate the presence of subpopulations within an overall population. It has been developed so far by various authors, such as [23]. It is used for data classification and it provides efficient approaches of model-based clustering. The authors of [24] demonstrated that, when a Gaussian mixture model is used to estimate a density non-parametrically, the density estimator that uses the Bayesian information criterion (BIC) of [25] to select the number of components in the mixture is consistent [26].

However, we obtain the non-parametric kernel estimate of a density if we fit a mixture of n components in equal proportions , where n is the size of the observed sample. As a matter of fact, it can be inferred that mixture models occupy an interesting niche between parametric and non-parametric approaches to statistical estimation.

More recently, in the parametric context, [27] proposed a parametric model using Bernstein polynomials with positive coefficients to estimate the unknown density function f; this estimator is defined as follows: where , for , (, , ) and are the estimators of the parameters , obtained by the Expectation Maximization (EM) algorithm as follows: with for The proposed method gives a consistent estimator in distance under some conditions.

The problem at the edge does not arise for the parametric model. For this reason, the basic idea of this work is to consider a shrinkage method using Bernstein (Vitale’s estimator) and Gaussian mixture estimators, to construct a shrinkage density estimator, in order to improve the approximation at the edge. A shrinkage estimator is a convex combination between estimators [28]. Basically, this implies that a naive or raw estimate is improved by combining it with other information.

The remainder of this paper is organized as follows: In the next section, we recall some intrinsic properties of the classical EM algorithm in the context of the Gaussian mixture parameter estimation. In Section 3, we introduce a new semi-parametric estimation approach based on the shrinkage method using Bernstein polynomials and Gaussian mixture densities. In Section 4, the consistency of the proposed estimator is exhibited, as well as its asymptotic normality.

Section 5 highlights a simulation study that compares the performance of the proposed approach with the Bernstein estimator, the standard Gaussian kernel estimator and Guan’s estimator. The closing Section 6 crowns the whole work, wraps the conclusion and provides new perspectives for future work.

2. Background

The Gaussian Mixture Model and Em Algorithm

Let us consider , a sequence of independent and identically distributed (i.i.d.) with common Gaussian mixture density defined by where satisfies and

Finally, for each observed data point , we associate a component label vector in order to manage the data clustering. This random vector is defined such that if the considered observation is drawn form the component of the mixture and otherwise. Consequently, is distributed as a multivariate Bernoulli distribution with vector parameters as follows: The EM algorithm is a popular tool in statistical estimation problems involving incomplete data or problems which can be posed in a similar form, such as the mixture parameters estimation [23,29]. In the EM framework, corresponds to the complete data and stand for the hidden data. Hence, the complete-data log-likelyhood is expressed by The two steps of the EM algorithm, after l iterations, are the following: We repeat these two steps until where is a fixed threshold of convergence. The convergence properties of the EM algorithm have been investigated by [29] and by [30]. Relying upon Jensen’s inequality, it can be noticed that, as increases, the log-likelihood function also increases [29]. Consequently, the EM algorithm converges within a finite iteration number and gives the parameters’ maximum likelihood estimates. Therefore, under some conditions and according to [29], we have In what follows, .

E-step: The conditional expectation of the complete-data log-likelyhood given the observed data, using the current fit , is defined by

The posterior probability that belongs to the component of the mixture at the iteration, is expressed as

Finally, we obtain

M-step: It consists of a global maximization of with respect to .

The updated estimates are stated by

3. Proposed Approach

The proposed semi-parametric approach rests upon the shrinkage combination between the Gaussian mixture model and the Bernstein density estimators using the EM algorithm for the parameter estimations. The literature on shrinkage estimation is enormous. From this perspective, it is noteworthy to mention the most relevant contributions. The authors of [28] were the first to introduce the classic shrinkage estimator. The authors of [31] provided theory for the analysis of risk. Oman [32,33] developed estimators which shrink Gaussian density estimators towards linear subspaces. An in-depth investigation of shrinkage theory is displayed in Chapter 5 of [34].

The proposed semi-parametric approach based upon estimating the density function f relies on the same principle of Stein’s works and there are two aspects along this line. The first setting is non-parametric in the sense that we do not assume any parametric form of the density. The non-parametric setting is very important as it allows us to perform statistical inference without making any assumption on the parametric form of the true density f. The second setting is to consider the Gaussian mixture model as a parametric estimator of the unknown density f.

In what follows, we consider a sequence of i.i.d. random variables having a common unknown density function f supported on . We here develop a shrinkage method to estimate the density function, which is divided into the following three steps:

We consider the Bernstein estimator of the density function f, which is defined as

In view of (13), we consider the Gaussian mixture density as an estimator of the density function f, given by where , and are estimated by the EM algorithm defined in (13).

We consider the shrinkage density estimator form defined by and we use the EM algorithm to estimate the parameter of the proposed model.

By the same way as considered in Section 2, the two steps of the EM algorithm, after t iterations, are denoted in terms of the following: The estimation of is obtained from by iterating the EM algorithm until convergence.

E-step: The conditional expectation of the complete-data log-likelihood given the observed data, using the current , is provided by where is a discrete random vector, following a multivariate Bernoulli distribution with vector parameters . Using Bayes’s formula, we obtain the posterior probability in the iteration denoted by and

M-step: It consists of a global maximization of with respect to .

The updated estimate of is indicated by

Therefore, the proposed estimator of the density function f is defined by

Basically, it is a shrinkage estimator that shrinks the Bernstein estimator towards the Gaussian mixture density by a specified amount of . If , the estimator reduces to the Bernstein estimator .

4. Convergence

In this section, we derive some asymptotic properties of the proposed estimator when the sample size tends to infinity. First, we assume that and K are fixed. The following proposition gives the probability convergence of the proposed estimator .

(Probability convergence). If where

The proof of Proposition 1 necessitates the following technical Lemma.

Let where

The proof of this lemma is reported in [35].

First, using Lemma 1 and following the same steps as the proof of Theorem 4.4 in [35], we prove that   and Then, according to Slutsky’s Theorem, we obtain Second, based on Theorem 3.1 in [16], we obtain In addition, referring to (18) and (19) and grounded on the application of Slutsky’s Theorem, we conclude the proof.   □

According to [21], the density is a close approximation to the mixture density . Thus, the estimator provides an approximation to the true density .

To study the asymptotic normality of the estimator given by (17), we set forward the following assumptions in [36].

For almost and for all , the partial derivatives , and of the density g exist and satisfy that and are bounded, respectively, by , and , where and are integrable and , satisfies

The Fisher information matrix is positively defined at .

(Normality asymptotic). Under the regularity conditions–, if where

Using Theorem 3.2 in [16], we obtain Thus, According to Theorem 3.1 in [36], we obtain Using the delta method, we obtain where is the Jacobian matrix of and . Since if , then, using Slutsky’s Theorem, we conclude the proof. □

The following corollary is a consequence of the previous proposition which gives an asymptotic confidence interval of the density f, for a risk .

The where

In the next section, we study the performance of the proposed estimator in estimating different distributions by comparing it to the performances of the Bernstein estimator and of the Gaussian kernel estimator.

5. Numerical Studies

5.1. Comparison Study

In this section, we investigate the performance of the proposed estimator given in (17), through estimating different densities by comparing it to the performance of the Bernstein estimator defined in (2), the standard Gaussian kernel estimator defined in (1) and the Guan’s estimator defined in (3). We apply the Bernstein estimator when the sample is concentrated on the interval . For this purpose, we need to make some suitable transformations in the different cases that are listed as follows: If the support is infinite, say, , we can consider as the finite support of f, where and are the minimum and the maximum order, respectively. We choose a and b such that and are of , , and , where F is the distribution function [27]. Then, we can use the transformed sample, which transforms to the interval mentioned in the case 1.

Let us suppose that X is concentrated on a finite support ; then, we work with the sample values , where .

For the density functions concentrated on , we can use the transformed sample , which transforms the range to the interval .

For the support , we can use the transformed sample , which transforms the range to the interval .

In the simulation study, three sample sizes were considered, , , and , as well as the following density functions:

The beta mixture density ;

The beta mixture density ;

The normal mixture density ;

The chi-squared density.

The gamma mixture density ;

The gamma mixture density .

Our sample was decomposed into a learning sample of a size of 2/3 of the considered sample, on which the various statistical methods were constructed, and a second sample of a size of 1/3 of the considered sample, on which the predictive performance of the three methods were tested. For each density function f and sample size n, we computed the integrated squared error (), the integrated absolute error () and the Kullback–Leibler divergence () of the estimator over trials. where is the estimator computed from the sample and

Indeed, it is advised to consider a learning sample bigger than a testing sample. In this work, our sample was decomposed into a learning sample of a size of 2/3 of the considered sample, on which the various statistical methods were constructed, and a second sample of a size of 1/3 of the considered sample, on which the predictive performances of the three methods were tested. Each run of the proposed estimator performed the following steps:

We first generated a random sample of size n from the models’ density .

We then split the generated data into a training set of a size of of the considered sample and a test set of a size of of the considered sample.

We applied the proposed estimator, using the observed data only from the training set, in order to estimate the density function.

The test set was then used to compute the estimation errors , and .

To select the optimal parameter K, we used the Gap Statistics algorithm [37]. We considered a Monte Carlo experiment to select the optimal choice of the degree m of the Bernstein polynomial and the bandwidth h of the kernel estimator, for each point . We determined the parameters m (for ) and h (for with ), which minimized the , which was approximated by the .

We considered random samples of sizes , and .

Departing from Table 1, Table 2 and Table 3 and Figure 1, we deduce the following:

Table 1

Average for trials of Bernstein estimator, standard Gaussian kernel estimator and the proposed estimator , for , and . The bold values indicate the smallest values of .

Density	n	Proposed	Bernstein	Kernel	Guan’s
		Estimator	Estimator	Estimator	Estimator
	50	0.092242	0.096684	0.197497	0.140323
(a)	100	0.091364	0.092242	0.174251	0.089129
	200	0.075532	0.079299	0.148143	0.086305
	50	1.157827	1.215347	0.530446	0.816906
(b)	100	0.235402	0.306704	0.482152	0.276573
	200	0.199786	0.289870	0.474805	0.255716
	50	0.001423	1.808252	2.222369	1.035522
(c)	100	0.000384	1.410606	1.641689	1.014602
	200	0.000227	1.348292	1.077352	0.994346
	50	0.525192	2.812448	4.936701	0.589465
(d)	100	0.492752	2.483141	2.331765	0.579595
	200	0.162917	0.898103	1.154646	0.507362
	50	2.180849	2.231986	2.424656	1.084340
(e)	100	2.050098	2.133496	2.295932	0.835670
	200	2.042379	2.086204	2.053453	0.717715
	50	0.313388	0.896995	1.397111	0.663889
(f)	100	0.253988	0.656400	0.762742	0.516530
	200	0.186290	0.577408	0.417980	0.472094

Table 2

Average for trials of Bernstein estimator, standard Gaussian kernel estimator and the proposed estimator , for , and . The bold values indicate the smallest values of .

Density	n	Proposed	Bernstein	Kernel	Guan’s
		Estimator	Estimator	Estimator	Estimator
	50	0.250241	0.250241	0.391072	0.251641
(a)	100	0.196423	0.207109	0.367361	0.232399
	200	0.180536	0.191673	0.348499	0.214117
	50	0.855008	0.823416	0.621137	0.747562
(b)	100	0.417735	0.457722	0.669027	0.438088
	200	0.386280	0.455983	0.583057	0.423595
	50	0.035161	0.971720	1.238839	0.948451
(c)	100	0.019331	0.960044	1.157838	0.944000
	200	0.013233	0.923259	0.953675	0.931293
	50	0.669269	0.807667	1.942776	0.648617
(d)	100	0.657815	1.486974	1.415036	0.642986
	200	0.351205	1.573886	0.881866	0.637942
	50	0.830317	0.834203	1.499828	0.678362
(e)	100	0.706770	0.713225	1.374609	0.645112
	200	0.681767	0.692357	1.225502	0.620315
	50	0.453948	0.640596	1.068483	0.608421
(f)	100	0.414676	0.714844	0.782939	0.584572
	200	0.383248	0.883320	0.516527	0.584320

Table 3

Average for trials of Bernstein estimator, standard Gaussian kernel estimator and the proposed estimator , for , and . The bold values indicate the smallest values of .

Density	n	Proposed	Bernstein	Kernel	Guan’s
		Estimator	Estimator	Estimator	Estimator
	50	0.025048	0.025048	0.289818	0.066817
(a)	100	0.012086	0.015023	0.081830	0.058541
	200	0.003256	0.001788	0.060468	0.029419
	50	1.088053	1.120246	0.575298	0.667079
(b)	100	0.284795	0.325933	0.381406	0.659105
	200	0.255697	0.310759	0.150096	0.654338
	50	2.689781	3.871172	4.732324	3.360316
(c)	100	0.011844	3.591156	4.196295	3.356093
	200	0.009426	3.505050	3.169852	3.318666
	50	0.976450	0.870222	4.702359	0.878995
(d)	100	0.960633	1.572251	3.031783	0.763044
	200	0.281862	1.584022	1.537355	0.742696
	50	1.549035	1.560172	5.538142	1.799133
(e)	100	1.207084	1.217043	3.498273	1.557894
	200	1.153420	1.169645	1.322299	1.387843
	50	0.528337	1.052789	1.952294	0.651422
(f)	100	0.292017	0.805962	1.191070	0.537625
	200	0.062893	0.589790	0.679154	0.339442

Figure 1

Quantitative comparison between the proposed estimator and Guan’s estimator of for (left) and (right).

The results displayed in Table 1, Table 2 and Table 3 show that the , and decreased as the sample size increased.

Using the proposed estimator, we obtained better results than those given by the other estimators in a large part of the cases.

The Figure 2 and 3 give a better sense of where the error is located.

Figure 2

Quantitative comparison among the mean squared error of the kernel estimator, the Bernstein estimator, the Guan’s estimator and the proposed estimator of for .

For the case (e) of the gamma mixture, the average and of Guan’s estimator (1.3) were smaller than those obtained by the proposed density estimator (3.4) and the Bernstein estimator (1.2). However, in all the other cases, using an appropriate choice of the degree m, the average and of the proposed density estimator (3.4) were smaller than what achieved by the kernel estimator (1.1), the Bernstein estimator (1.2) and Guan’s estimator (1.3), even when the sample size was large for same cases.

When we changed the parameters of the gamma mixture density in the sense that we had a smaller bias, our estimator was more competitive than the other approaches and we obtained better results.

Almost in all considered cases, the average of the density estimator (17) was smaller than that obtained by the Bernstein estimator defined in (2), that of the kernel estimator defined in (1) and that of Guan’s estimator.

In the considered distribution , by choosing the appropriate m, the curve of the proposed distribution estimator (3.4) was closer to the true distribution than that of Guan’s estimator (1.3), even when the sample size was very large.

Referring to Figure 2 and Figure 3, we infer the following:

Figure 3

Quantitative comparison among the mean squared error of the kernel estimator, the Bernstein estimator, the Guan’s estimator and the proposed estimator of for .

None of the estimators for the gamma mixture density had good approximations near . However, the of the proposed estimator was closer to zero than that of the Bernstein estimator and the kernel estimator, especially near the edge .

Guan’s estimator and the kernel estimator for the normal mixture density had good approximations near . However, the of the proposed estimator was closer to zero than that of the other estimators, especially near the two edges.

Therefore, we note that, for difficult distributions that diverge at the boundaries, the proposed method would fail, but not as badly as the standard methods without shrinkage. In addition, the performed simulations revealed that, on average, the proposed approach could lead to satisfactory estimates near the boundaries, better than the classical Bernstein estimator.

5.2. Real Dataset

5.2.1. COVID-19 Data

In this subsection, we consider the COVID-19 data displayed in the INED website https://dc-covid.site.ined.fr/fr/donnees/france/ (accessed on 16 February 2022). These data concern the numbers of deaths due to COVID-19 in France (daily) from 21 March 2021, for 454 days. These data are such that and . Then, it is convenient to assume that the density of the numbers of deaths is defined on the interval and transform the data into the interval unit. The Monte Carlo procedure was performed and resulted in for the standard kernel estimator defined in (1.1), for the Bernstein estimator defined in (1.2), the proposed estimator, and for Guan’s estimator. These estimators are exhibited in Figure 1 (right panel) along with a histogram of the data. All the estimators are smooth and seem to capture the pattern highlighted by the histogram. We record that the proposed estimator outperformed the other estimators near the boundaries.

5.2.2. Tuna Data

The last example concerns the tuna data reported in [38]. The data are derived from an aerial line transect survey of Southern Bluefin Tuna in the Great Australian Bight. An aircraft with two spotters on board flew randomly over allocated line transects. These data correspond to the perpendicular sighting distances (in miles) of 64 detected tuna schools to the transect lines. The survey was conducted in summer when tuna data tend to stay on the surface. The data are such that and . The Monte Carlo procedure was performed and resulted in for the standard kernel estimator defined in (1), for the Bernstein estimator defined in (2) and the proposed estimator, and for Guan’s estimator. These estimators are illustrated in Figure 4 (left panel) along with a histogram of the data. All the estimators are smooth and seem to capture the pattern highlighted by the histogram. We assert that the proposed estimator outperformed the other estimator, especially near the boundaries.

Figure 4

Qualitative comparison among the kernel estimator defined in (1), the Bernstein estimator defined in (2), Guan’s estimator (3) and the proposed density estimator (17) of Tuna data (left) and of COVID-19 data (right).

6. Conclusions

In this paper, we propose a shrinkage estimator of a density function based on the Bernstein density estimator and using a finite Gaussian mixture density. This method rests on three steps. The first step consists of considering the Bernstein estimator . The second relies upon the Gaussian Mixture density as an estimator of the unknown density f. The last step consists of considering the shrinkage form and EM algorithm in order to estimate the parameter . The asymptotic properties of this estimator were established. Afterwards, we demonstrate the effectiveness of the proposed method using some simulated and real data. We clarify how it can lead to very satisfactory estimates near the boundaries and in terms of , and . Eventually, we would simply assert that our research work is a step that may be taken further, extended and built upon as it lays the ground and paves the way for future works to elaborate a semi-parametric regression estimator using the shrinkage method. We also plan to work on the case where is a random variable. Another future research direction would be to extend our findings to the setting of serially dependent observations.