A Generalized Relative (α, β)-Entropy: Geometric Properties and Applications to Robust Statistical Inference.

Entropy and relative entropy measures play a crucial role in mathematical information theory. The relative entropies are also widely used in statistics under the name of divergence measures which link these two fields of science through the minimum divergence principle. Divergence measures are popular among statisticians as many of the corresponding minimum divergence methods lead to robust inference in the presence of outliers in the observed data; examples include the ϕ -divergence, the density power divergence, the logarithmic density power divergence and the recently developed family of logarithmic super divergence (LSD). In this paper, we will present an alternative information theoretic formulation of the LSD measures as a two-parameter generalization of the relative α -entropy, which we refer to as the general ( α , β ) -entropy. We explore its relation with various other entropies and divergences, which also generates a two-parameter extension of Renyi entropy measure as a by-product. This paper is primarily focused on the geometric properties of the relative ( α , β ) -entropy or the LSD measures; we prove their continuity and convexity in both the arguments along with an extended Pythagorean relation under a power-transformation of the domain space. We also derive a set of sufficient conditions under which the forward and the reverse projections of the relative ( α , β ) -entropy exist and are unique. Finally, we briefly discuss the potential applications of the relative ( α , β ) -entropy or the LSD measures in statistical inference, in particular, for robust parameter estimation and hypothesis testing. Our results on the reverse projection of the relative ( α , β ) -entropy establish, for the first time, the existence and uniqueness of the minimum LSD estimators. Numerical illustrations are also provided for the problem of estimating the binomial parameter.

1. Introduction

Decision making under uncertainty is the backbone of modern information science. The works of C. E. Shannon and the development of his famous entropy measure [1,2,3] represent the early mathematical foundations of information theory. The Shannon entropy and the corresponding relative entropy, commonly known as the Kullback-Leibler divergence (KLD), has helped to link information theory simultaneously with probability [4,5,6,7,8] and statistics [9,10,11,12,13]. If P and Q are two probability measures on a measurable space and have absolutely continuous densities p and q, respectively, with respect to a common dominating -finite measure , then the Shannon entropy of P is defined as and the KLD measure between P and Q is given by In statistics, the minimization of the KLD measure produces the most likely approximation as given by the maximum likelihood principle; the latter, in turn, has a direct equivalence to the (Shannon) entropy maximization criterion in information theory. For example, if is finite and is the counting measure, it is easy to see that , where U is the uniform measure on . Minimization of this relative entropy, or equivalently maximization of the Shannon entropy, with respect to P within a suitable convex set , generates the most probable distribution for an independent identically distributed finite source having true marginal probability in with non-informative (uniform) prior probability of guessing [14,15]. In general, with a finite source, denotes the penalty in expected compressed length if the compressor assumes a mismatched probability Q [16,17]. The corresponding general minimizer of given Q, namely its forward projection, and other geometric properties of are well studied in the literature; see [18,19,20,21,22,23,24,25,26,27,28,29] among others.

Although the maximum entropy or the minimum divergence criterion based on the classical Shannon entropy and the KLD measure is still widely used in major (probabilistic) decision making problems in information science and statistics [30,31,32,33,34,35,36,37,38,39,40,41,42,43], there also exist many different useful generalizations of these quantities to address eminent issues in quantum statistical physics, complex codings, statistical robustness and many other topics of interest. For example, if we consider the standardized cumulant of compression length in place of the expected compression length in Shannon’s theory, the optimum distribution turns out to be the maximizer of a generalization of the Shannon entropy [44,45] which is given by provided , the complete vector space of functions for which the -th power of their absolute values are -integrable. This general entropy functional is popular by the name Renyi entropy of order [46] and covers many important entropy measures like Hartley entropy at (for finite source), Shannon entropy at , collision entropy at and the min-entropy at . The corresponding Renyi divergence measure is given by whenever and coincides with the classical KLD measure at . The Renyi entropy and the Renyi divergence are widely used in recent complex physical and statistical problems; see, for example, [47,48,49,50,51,52,53,54,55,56]. Other non-logarithmic extensions of Shannon entropy include the classical f-entropies [57], the Tsallis entropy [58] as well as the more recent generalized -entropy [59,60] among many others; the corresponding divergences and the minimum divergence criteria are widely used in critical information theoretic and statistical problems; see [57,59,60,61,62,63,64,65,66,67,68,69,70] for details.

We have noted that there is a direct information theoretic connection of KLD to the Shannon entropy under mismatched guessing by minimizing the expected compressed length. However, such a connection does not exist between the Renyi entropy and the Renyi divergence as recently noted by [17,71]. Herein, it has been shown that, for a finite source with marginal distribution P and a (prior) mismatched compressor distribution Q, the penalty in the normalized cumulant of compression length is not ; rather it is given by where and are defined by The new quantity also gives a measure of discrimination (i.e., is a divergence) between the probability distributions P and Q and coincides with the KLD at . This functional is referred to as the relative -entropy in the terminology of [72] and has the simpler form The geometric properties of this relative -entropy along with its forward and reverse projections have been studied recently [16,73]; see Section pan class="Gene">2.1 for some details. This quantity had, however, already been proposed earlier as a statistical divergence, although for only, by [74] while developing a robust estimation procedure following the generalized method-of-moments approach of [75]. Later authors referred to the divergence proposed in [74] as the logarithmic density power divergence (LDPD) measure. The advantages of the minimum LDPD estimator in terms of robustness against outliers in data have been studied by, among other, [66,74]. Fujisawa [76], Fujisawa and Eguchi [77] have also used the same divergence measure with in different statistical problems and have referred to it as the -divergence. Note that, the formulation in (6) extends the definition of the divergence over the region as well.

Motivated by the substantial advantages of the minimum LDPD inference in terms of statistical robustness against outlying observations, Maji et al. [78,79] have recently developed a two-parameter generalization of the LDPD family, namely the logarithmic super divergence (LSD) family, given by This rich superfamily of divergences contain many important divergence measures including the LDPD at and the Kullback-Leibler divergence at ; this family also contains a transformation of Renyi divergence at which has been referred to as the logarithmic power-divergence family by [80]. As shown in [78,79], the statistical inference based on some of the new members of this LSD family, outside the existing ones including the LDPD, provide much better trade-off between the robustness and efficiency of the corresponding minimum divergence estimators.

The statistical benefits of the LSD family over the LDpan class="Chemical">PD family raise a natural question: is it possible to translate this robustness advantage of the LSD family of divergences to the information theoretic context, through the development of a corresponding generalization of the relative -entropy in (6)? In this paper, we partly answer this question by defining an independent information theoretic generalization of the relative -entropy measure coinciding with the LSD measure. We will refer to this new generalized relative entropy measure as the “Relative ” and study its properties for different values of and . In particular, this new formulation will extend the scope of the LSD measure for as well and generate several interesting new divergence and entropy measures. We also study the geometric properties of all members of the relative -entropy family, or equivalently the LSD measures, including their continuity in both the arguments and a pan class="Chemical">Pythagorean-type relation. The related forward projection problem, i.e., the minimization of the relative -entropy in its first argument, is also studied extensively.

In summary, the main objective of the present paper is to study the geometric properties of the LSD measure through the new information theoretic or entropic formulation (or the relative -entropy). Our results indeed generalize the properties of the relative -entropy from [16,73]. The specific and significant contributions of the paper can be summarized as follows.

We present a two parameter extension of the relative -entropy measure in (6) motivated by the logarithmic S-divergence measures. These divergence measures are known to generate more robust statistical inference compared to the LDPD measures related to the relative -entropy.

In the new formulation of the relative -entropy, the LSD measures are linked with several important information theoretic divergences and entropy measures like the ones named after Renyi. A new divergence family is discovered corresponding to case (properly standardized) for the finite measure cases.

As a by-product of our new formulation, we get a new two-parameter generalization of the Renyi entropy measure, which we refer to as the Generalized Renyi entropy (GRE). This opens up a new area of research to examine the detailed properties of GRE and its use in complex problems in statistical physics and information theory. In this paper, we show that this new GRE satisfies the basic entropic characteristics, i.e., it is zero when the argument probability is degenerate and is maximum when the probability is uniform.

Here we provide a detailed geometric analysis of the robust LSD measure, or equivalently the relative -entropy in our new formulation. In particular, we show their continuity or lower semi-continuity with respect to the first argument depending on the values of the tuning parameters and . Also, its lower semi-continuity with respect to the second argument is proved.

We also study the convexity of the LSD measures (or the relative -entropies) with respect to its argument densities. The relative -entropy (i.e, the relative -entropy at ) is known to be quasi-convex [16] only in its first argument. Here, we will show that, for general and , the relative -entropies are not quasi-convex on the space of densities, but they are always quasi-convex with respect to both the arguments on a suitably (power) transformed space of densities. Such convexity results in the second argument were unavailable in the literature even for the relative -entropy, which we will introduce in this paper through a transformation of space.

Like the relative -entropy, but unlike the relative entropy in (2), our new relative -entropy also does not satisfy the data processing inequalities. However, we prove an extended Pythagorean relation for the relative -entropy which makes it reasonable to treat them as “squared distances” and talk about their projections.

The forward projection of a relative entropy or a suitable divergence, i.e., their minimization with respect to the first argument, is very important for both statistical physics and information theory. This is indeed equivalent to the maximum entropy principle and is also related to the Gibbs conditioning principle. In this paper, we will examine the conditions under which such a forward projection of the relative -entropy (or, LSD) exists and is unique.

Finally, for completeness, we briefly present the application of the LSD measure or the relative -entropy measure in robust statistical inference in the spirit of [78,79] but now with extended range of tuning parameters. It uses the reverse projection principle; a result on the existence of the minimum LSD functional is first presented with the new formulation of this paper. Numerical illustrations are provided for the binomial model, where we additionally study their properties for the extended tuning parameter range as well as for some new divergence families (related to ). Brief indications of the potential use of these divergences in testing of statistical hypotheses are also provided.

Although we are primarily discussing the logarithmic entropies like the Renyi entropy and its generalizations in this paper, it is important to point out that non-logarithmic entropies including the f-entropy and the Tsallis entropy are also very useful in several applications with real systems. Recently, several complex physical and social systems have been observed to follow the theory developed from such non-logarithmic, non-additive entropies instead of the classical additive Shannon entropy. In particular, the Tsallis entropy has led to the development of the nonextensive statistical mechanics [61,64] to solve several critical issues in modern physics. Important areas of application include, but certainly are not limited to, the motion of cold atoms in dissipative optical lattices [81,82], the magnetic field fluctuations in the solar wind and related q-triplet [83], the distribution of velocity in driven dissipative dusty plasma [84], spin glass relaxation [85], the interaction of trapped ion with a classical buffer gas [86], different high energy collisional experiments [87,88,89], derivation of the black hole entropy [90], along with water engineering [63], text mining [65] and many others. Therefore, it is also important to investigate the possible generalizations and manipulations of such non-logarithmic entropies both from mathematical and application point of view. However, as our primary interest here is in logarithmic entropies, we have, to keep the focus clear, otherwise avoided the description and development of non-logarithmic entropies in this paper.

Although there are many applications of extended and general non-additive entropy and divergence measures, there are also some criticisms of these non-additive measures that should be kept in mind. It is of course possible to employ such quantities simply as new descriptors of the complexity of systems, but at the same time, it is known that the minimization of a generalized divergence (or maximization of the corresponding entropy) under constraints in order to determine an optimal probability assignment leads to inconsistencies for information measures other than the Kullback-Leibler divergence. See, for instance [91,92,93,94,95,96], among others. So, one needs to be very careful in discriminating the application of the newly introduced entropies and divergence measures for the purposes of inference under given information, from the ones where it is used as a measure of complexity. In this respect, we would like to emphasize that, the main advantage of our two-parameter extended family of LSD or relative -entropy measures in parametric statistical inference is in their strong robustness property against possible contamination (generally manifested through outliers) in the sample data. The classical additive Shannon entropy and Kullback-Leibler divergence produce non-robust inference even under a small proportion of data contamination, but the extremely high robustness of the LSD has been investigated in detail, with both theoretical and empirical justifications, by [78,79]; in this respect, we will present some numerical illustrations in Section 5.2. Another important issue could be to decide whether to stop at the two-parameter level for information measures or to extend it to three-parameters, four-parameters, etc. It is not an easy question to answer. However, we have seen that many members of the two-parameter family of LSD measures generate highly robust inference along with a desirable trade-off between efficiency under pure data and robustness under contaminated data. Therefore a two-parameter system appears to work well in practice. Since it is a known principle that one “should not multiply entities beyond necessity”, we will, for the sake of parsimony, restrict ourselves to the second level of generalization for robust statistical inference, at least until there is further convincing evidence that the next higher level of generalization can produce a significant improvement.

2. The Relative -Entropy Measure

2.1. Definition: An Extension of the Relative -Entropy

In order to motivate the development of our generalized relative -entropy measure, let us first briefly describe an alternative formulation of the relative -entropy following [16]. Consider the mathematical set-up of Section 1 with and assume that the space is equipped with the norm and the corresponding metric for . Then, the relative -entropy between two distributions P and Q is obtained as a function of the Cressie-Read power divergence measure [97], defined below in (11), between the escort measures and defined in (5). Note that the disparity family or the -divergence family [18,98,99,100,101,102,103] between P and Q is defined as for a continuous convex function on satisfying and with the usual convention . We consider the -function given by with the convention that, for any , if and if . The corresponding -divergence has the form which is just a positive multiple of the Cressie-Read power divergence with the multiplicative constant being ; when this constant is present, the case leads to the KLD measure in a limiting sense. Note that, our -function in (10) differs slightly from the one used by [16] in that we use in place of there; this is to make the divergence in (11) non-negative for all ([16] considered only ) which will be needed to define our generalized relative entropy. Then, given an , [16,17] set and show that the relative -entropy of P with respect to Q can be obtained as It is straightforward to see that the above formulation (12) coincides with the definition given in (6). We often suppress the superscript whenever the underlying measure is clear from the context; in most applications in information theory and statistics it is either counting measure or the Lebesgue measure depending on whether the distribution is discrete or continuous.

We can now change the tuning parameters in the formulation given by (12) suitably as to arrive at the more general form of the LSD family in (7). For this purpose, let us fix , and assume that are the -densities of P and Q, respectively. Instead of considering the re-parametrization as above, we now consider the two-parameter re-parametrization . Note that, the feasible range of , in order to make , now clearly depends on through ; whenever we have and if we need . We have already taken care of this dependence through the modified function defined in (10) which ensures that is non-negative for all . So we can again use the relation as in (12), after suitable standardization due to the additional parameter , to define a new generalized relative entropy measure as given in the following definition.

Given any The cases

A straightforward simplification gives a simpler form of this new relative -entropy which coincides with the LSD measure as follows. Note that, it coincides with the relative -entropy at the choice . For the limiting cases, it leads to the forms By the divergence property of , all the relative -entropies are non-negative and valid statistical divergences. Note that, in view of (14), the formulation (13) extends the scope of LSD measure, defined in (7), for .

For any and , for all probability measures P and Q, whenever it is defined. Further, if and only in .

Also, it is important to identify the cases where the relative -entropy is not finitely defined, which can be obtained from the definition and convention related to divergence; these are summarized in the following proposition.

For any

pan class="Chemical">P is not absolutely continuous with respect to Q and , in which case .

P is mutually singular to Q and , in which case also .

and , in which case also is undefined.

The above two propositions completely characterize the values and existence of our new relative -entropy measure. In the next subsection, we will now explore its relation with other existing entropies and divergence measures; along the way we will get some new ones as by-products of our generalized relative entropy formulation.

2.2. Relations with Different Existing or New Entropies and Divergences

The relative -entropy measures form a large family containing several existing relative entropies and divergences. Its relation with some popular ones are summarized in the following proposition; the proof is straightforward from definitions and hence omitted.

For

, the KLD measure.

, the relative α-entropy.

, a scaled Renyi divergence, which also coincides with the logarithmic power divergence measure of [80].

, where and are as defined in (5).

Note that, items 3 and 4 in pan class="Chemical">Proposition 3 indicate a possible extension of the Renyi divergence measure over negative values of the tuning parameter β as follows: Note that this modified Renyi divergence also coincides with the KLD measure at

However, not all the members of the family of relative -entropies are distinct or symmetric. For example, for any . The following proposition characterizes all such identities.

For , and distributions pan class="Chemical">P, Q, the relative -entropy is symmetric if and only if . In general, we have for any .

Recall that the KLD measure is linked to the Shannon entropy and the relative -entropy is linked with the Renyi entropy when the prior mismatched probability is uniform over the finite space. To derive such a relation for our general relative -entropy, let us assume and let U denote the uniform probability measure on . Then, we get where the functional is given in Definition 2 below and coincides with the Renyi entropy at . Thus, it can be used to define a two-parameter generalization of the Renyi entropy as follows.

For any probability measure P over a measurable space Note that, at

The GRE is a new entropy to the best of our knowledge, and does not belong to the general class of entropy functionals as given in [104] which covers many existing entropies (including most, if not all, classical entropies). The following property of the functional is easy to verify and justifies its use as a new entropy functional. To keep the focus of the present paper clear on the relative -entropy, further properties of the GRE will be explored in our future work.

For any probability measure P over a finite measure space

if pan class="Chemical">P is degenerate at a point in Ω (no uncertainty).

if P is uniform over

Consider distributions With the new entropy measure, GRE, the entropy of the normal distribution Interestingly, the GRE of a normal distribution is effectively the same as its Shannon entropy or Renyi entropy up to an additive constant. However, similar characteristic does not hold between the relative entropy (KLD) and relative whereas the general relative Note that the relative

Consider the exponential distribution P having density A simple calculation leads to the following form of the our new GRE measure of the exponential distribution P. Once again, the new GRE is effectively the same as the Shannon entropy or the Renyi entropy, up to an additive constant, for the exponential distribution as well.

Further, if for

Before concluding this section, we study the nature of our relative -entropy as . For this purpose, we restrict ourselves to the case of finite measure spaces with . It is again straightforward to note that for any and any distributions P and Q on . However, if we take the limit after scaling the relative entropy measure by we get a non-degenerate divergence measure as follows. for , and These interesting relative entropy measures again define a subfamily of valid statistical divergences, from its construction. The particular member at is linked to the LDPD (or the -divergence) with tuning parameter and can be thought of as a logarithmic extension of the famous Itakura–Saito divergence [105] given by This Itakura–Saito-divergence has been successfully applied to non-negative matrix factorization in different applications [106] which can be extended by using the new divergence family in future works.

3. Geometry of the Relative -Entropy

3.1. Continuity

We start the exploration of the geometric properties of the relative -entropy with its continuity over the functional space . In the following, we interchangeably use the notation and to denote and , respectively. Our results generalize the corresponding properties of the relative -entropy from [16,73] to our relative -entropy or equivalent LSD measure.

For a given , consider the function from to . This function is lower semi-continuous in for any , . Additionally, it is continuous in when and the relative entropy is finitely defined.

First let us consider any and take in . Then, . Also, and hence a general version of the dominated convergence theorem yields in . Thus, we get Further, following ([107], Lemma 1), we know that the function is lower semi-continuous in for any and any probability measure on . Taking , we get from (21) that in . Therefore, the above lower semi-continuity result along with (9) implies that

Now, note that the function is continuous and increasing on for and on for . Thus, combining (22) with the definition of the relative -entropy in (13), we get that i.e., the function is lower semi-continuous.

Finally, consider the case . Note that the dual space of is since . Also, for , we have , the dual space of the Banach space . Therefore, the function defined by is a bounded linear functional and hence continuous. Now, take in so that as . Therefore, in implying in . Hence, by the continuity of T on , we get However, from (14), we get This proves the continuity of in its first argument when . ☐

Whenever Ω is finite (discrete) equipped with the counting measure μ, all integrals in the definition of become finite sums and any limit can be taken inside these finite sums. Thus, whenever defined finitely, the function is always continuous in this case.

For a general infinite space Ω, the function is not necessarily continuous for the cases . This can be seen by using the same counterexample as given in Remark 3 of [16]. However, it is yet to be verified if this function can be continuous for cases.

For a given , consider the function from to . This function is lower semi-continuous in for any and .

Fix an and , which in turn fixes a . Note that, the relative -entropy measure can be re-expressed from (13) as Now, consider a sequence in and proceed as in the proof of Proposition 5 using ([107], Lemma 1) to obtain

Now, whenever with or with , we get from (25) and (26) that In all other cases, we consider the function as in the proof of Proposition 5. This function is continuous and increasing whenever the corresponding relative entropy is finitely defined for all tuning parameter values; on for and on for . Hence, again combining (26) with (25) through the function , we conclude that Therefore, the function is also lower semi-continuous. ☐

As in Remark 2, whenever Ω is finite (discrete) and is equipped with the counting measure μ, the function is continuous in for any fixed , and .

3.2. Convexity

It has been shown in [16] that the relative -entropy (i.e., ) is neither convex nor bi-convex, but it is quasi-convex in p. For general , however, the relative -entropy is not even quasi-convex in ; rather it is quasi-convex on the -power transformed space of densities, , as described in the following theorem. Note that, for , . Here we define the lower level set and its power-transformed set , for any and .

For any given , and , the sets are convex for all . Therefore, the function is quasi-convex on .

Note that, at , our theorem coincides with pan class="Chemical">Proposition 5 of [16]; so we will prove the result for the case . Fix , a real , , and . Then . For , we consider with . We need to show that , i.e., .

Now, from (14), we have Since , we have For any , we get Now, using the extended Minkowski’s inequalities from Lemma 1, given below, along with (31) and noting that , we get that Therefore, by (29) and the fact that is increasing in u, we finally get . This proves the result for .

The case can be proved in a similar manner and is left as an exercise to the readers. ☐

Fix

It follows by using the Jensen’s inequality and the convexity of the function . ☐

Next, note in view of pan class="Chemical">Proposition 4 that, for any , . Using this result along with the above theorem, we also get the quasi-convexity of the relative -entropy in q over a different power transformed space of densities. This leads to the following theorem.

For any given , and , the function is quasi-convex on . In particular, for the choice , the function is quasi-convex on .

Note that, at , the coincides with the KLD measure (or relative entropy) which is quasi-convex in both the arguments p and q on .

3.3. Extended Pythagorean Relation

Motivated by the quasi-convexity of on , we now present a Pythagorean-type result for the general relative -entropy over the power-transformed space. It generalizes the corresponding result for relative -entropy [16]; the proof is similar to that in [16] with necessary modifications due to the transformation of the domain space.

Fix an

Suppose

Suppose

Let to be the probability measure having -density for . Also note that, with , we have Thus, (34) is equivalent to the statement and we have where and . Now consider the two implications separately.

Only if statement: Now, let us assume that for all . Then, we get for all . Letting , we get that In order to find the derivative of , we first note that and hence Further, using a simple modification of the techniques in the proof of ([16], Theorem 9), it is easy to verify that the derivative of with respect to exists and is given by Hence we get Therefore, the derivative of exists and is given by . Therefore, using (40), we get that which implies (38) after substituting the values from (41) and (42).

If statement: Now, let us assume that (34)—or equivalently (38)—holds true. Further, as in the derivation of (38), we can start from the trivial statement to deduce Now, multiply (38) by and (44) by , and add to get In view of (37), this implies that This proves the if statement of Part (i) completing the proof. ☐

Note that the if statement follows directly from Part (i).

To prove the only if statement, we first show that and are finite since is finite. For this purpose, we note that by the definition of and hence . Therefore, we get Integration with respect to and using (29), we get where c is a constant. Similarly one can also show that .

Therefore, we can apply Part (i) to conclude that These relations imply that The proof of the above results proceed in a manner analogous to the proof of (38). Now, if either of the inequalities in (46) is strict, the corresponding inequality in (47) or (48) will also be strict. Then, multiplying (47) and (48) by and , respectively, and adding them we get (44) with a strict inequality (in place of an equality), which is a contradiction. Hence, both inequalities in (46) must be equalities implying (35) and (36). This completes the proof. ☐

Note that, at , the above theorem coincides with Theorem 9 of [16]. However, for general as well, the above extended pan class="Chemical">Pythagorean relation for the relative -entropy suggests that it behaves “like" a squared distance (although with a non-linear space transformation). So, one can meaningfully define its projection on to a suitable set which we will explore in the following sections.

4. The Forward Projection of Relative -Entropy

The forward projection, i.e., minimization with respect to the first argument given a fixed second argument, leads to the important maximum entropy principle of information theory; it also relates to the Gibbs conditioning principle from statistical physics [16]. Let us now formally define and study the forward projection of the relative -entropy. Let denote the set of probability measure on and let the set of corresponding -densities be denoted by .

Fix Note that we must assume that, so that the above relative -entropy is finitely defined for .

We first prove the uniqueness of the forward -projection from the Pythagorean property, whenever it exists. The following theorem describe the connection of the forward -projection with Pythagorean relation; the proof is same as that of ([16], Theorem 10) using Theorem 4 and hence omitted for brevity.

Consider the set Further, if

Consider the set such that is convex and fix . If a forward -projection of q on exists, it must be unique a.s..

Suppose and are two forward -projection of q on . Then, by definition, . Applying Theorem 5 with and , we get Hence or by non-negativity of relative entropy, which further implies that a.s. by Proposition 1. ☐

Next we will show the existence of the forward -projection under suitable conditions. We need to use an extended Apollonius Theorem for the -divergence measure used in the definition (13) of the relative -entropy. Such a result is proved in [16] for the special case ; the following lemma extends it for the general case .

Fix Let but the inequality gets reversed if

By (37), we get Then the Lemma follows by an application of the extended Minkowski’s inequalities (32) and (33) from Lemma 1. ☐

We now present the sufficient conditions for the existence of the forward -projection in the following theorem.

Fix and with and . Given any set for which is convex and closed and for some , a forward -projection of q on always exists (and it is unique by Corollary 1).

We prove it separately for the cases and , extending the arguments from [16]. The case can be obtained from these two cases by standard limiting arguments and hence omitted for brevity.

The Case :

Consider a sequence such that for each n and as . Then, by Lemma 2 applied to and with , we get where is defined by Note that, since is convex, and so . Also, using the non-negativity of divergence, (53) leads to Taking limit as , one can see that and hence . Thus, as by non-negativity. This along with a generalization of Pinker’s inequality for -divergence ([100], Theorem 1) gives whenever (which is true since ); here denotes the total variation norm. Now, by triangle inequality Thus, is Cauchy in and hence converges to some , i.e., and g is a probability density with respect to since each is so. Also, (57) implies that in -measure and hence in by an application of generalized dominated convergence theorem.

Next, as in the proof of ([16], Theorem 8), we can show that is bounded and hence for some , possibly working with a subsequence if needed. Thus we have in . However, since is closed, we have is closed and hence for some . Further, since , we must have and hence . Since and , Proposition 5 implies that where the second equality follows by continuity of the function , definitions of sequence and (13). Hence, we must have , i.e., is a forward -projection of q on .

The Case :

Note that, in this case, we must have , since . Then, using (29), we can see that where and Now, since and hence is closed, one can show that is also closed; see, e.g., the proof of ([16], Theorem 8). Next, we will show that is also convex. For take and for some and , and take any . Note that where However, by convexity of , and also by the extended Minkowski inequality (33). Therefore, and hence is convex.

Finally, since , is a reflexive Banach space and hence the closed and convex is also closed in the weak topology. So, the unit ball is compact in the weak topology by the Banach-Alaoglu theorem and hence its closed subset is also weakly compact. However, since g belongs to the dual space of , the linear functional is continuous in weak topology and also increasing in s. Hence its supremum over is attained at and some , which is the required forward -projection. ☐

Before concluding this section, we will present one example of the forward -projection onto a transformed-linear family of distributions.

Fix Let us denote the corresponding μ-density set by

Then, with The proof follows by extending the arguments of the proof of ([

Note that, at the special case , is a linear family of distributions and the above example coincides with ([16], Theorem 11) on the forward projection of relative α-entropy on . However, it is still an open question to derive the forward -projection on .

5. Statistical Applications: The Minimum Relative Entropy Inference

5.1. The Reverse Projection and Parametric Estimation

As in the case of the forward projection of a relative entropy measure, we can also define the reverse projection by minimizing it with respect to the second argument over a convex set keeping the first argument fixed. More formally, we use the following definition.

Fix

We can get sufficient conditions for the existence and uniqueness of the reverse -projection directly from Theorem 6 and the fact that ; this is presented in the following theorem.

Fix and with and . Given any set for which is convex and closed and for some , a reverse -projection of p on exists and is unique.

The reverse projection is mostly used in statistical inference where we fix the first argument of a relative entropy measure (or divergence measure) at the empirical data distribution and minimize the relative entropy with respect to the model family of distributions in its second argument. The resulting estimator, commonly known as the minimum distance or minimum divergence estimator, yields the reverse projection of the observed data distribution on the family of model distributions with respect to the relative entropy or divergence under consideration. This approach was initially studied by [9,10,11,12,13] to obtain the popular maximum likelihood estimator as the reverse projection with respect to the relative entropy in (2). More recently, this approach has become widely popular, but with more general relative entropies or divergence measures, to obtain robust estimators against possible contamination in the observed data. Let us describe it more rigorously in the following for our relative -entropy.

Suppose we have independent and identically distributed data from a true distribution G having density g with respect to some common dominating measure . We model g by a parametric model family of -densities , where it is assumed that both g and have the same support independent of . Our objective is to infer about the unknown parameter . In minimum divergence inference, an estimator of is obtained by minimizing the divergence measure between (an estimate of) g and with respect to . Maji et al. [78] have considered the LSD (or equivalently the relative -entropy) as the divergence under consideration and defined the corresponding minimum divergence functional at G, say , through the relation whenever the minimum exists. We will refer to as the minimum relative -entropy (MRE) functional, or the minimum LSD functional in the language of [78,79]. Note that, if , i.e., for some , then we must have . If , we call as the “best fitting parameter" value, since is the closest model element to g in the LSD sense. In fact, for , is nothing but the reverse -projection of the true density g on the model family , which exists and is unique under the sufficient conditions of Theorem 7. Therefore, under identifiability of the model family we get the existence and uniqueness of the MRE functional, which is presented in the following corollary. Although this estimator was first introduced by [78] in terms of the LSD, the results concerning the existence of the estimate were not provided.

Consider the above parametric estimation problem with

Suppose for some . Then the unique MRE functional is given by .

Suppose . If is convex and closed and for some , the MRE functional exists and is unique.

Further, under standard differentiability assumptions, we can obtain the estimating equation of the MRE functional as given by where . It is important to note that, at , the MRE functional coincides with the maximum likelihood functional since , the KLD measure. Based on the estimating Equation (61), Maji et al. [78] extensively studied the theoretical robustness properties of the MRE functional against gross-error contamination in data through the higher order influence function analysis. The classical first order influence function was seen to be inadequate for this purpose; it becomes independent of at the model but the real-life performance of the MRE functional critically depends on both and [78,79] as we will also see in Section 5.2.

In practice, however, the true data generating density is not known and so we need to use some empirical estimate in place of g and the resulting value of the MRE functional is called the minimum relative -entropy estimator (MREE) or the minimum LSD estimator in the terminology of [78,79]. Note that, when the data are discrete and is the counting measure, one can use a simple estimate of g given by the relative frequencies , where is the indicator function of the event A; the corresponding MREE is then obtained by solving (61) with replaced by and integrals replaced by sums over the discrete support. Asymptotic properties of this MREE under discrete models are well-studied by [78,79] for the tuning parameters and ; the same line of argument can be used to extend them also for the cases in a straightforward manner.

However, in case of continuous data, there is no such simple estimator available to use in place of g unless . When , the estimating Equation (61) depends on g through the terms and ; so we can simply use the empirical distribution function in place of G and solve the resulting equation to obtain the corresponding MREE. However, for , we must use a non-parametric kernel estimator of g in (61) to obtain the MREE under continuous models; this leads to complications including bandwidth selection while deriving the asymptotics of the resulting MREE. One possible approach to avoid such complications is to use the smoothed model technique, which has been applied in [108] for the case of minimum -divergence estimators. Another alternative approach has been discussed in [109,110]. However, the detailed analyses of the MREE under the continuous model, in either of the above approaches, are yet to be studied so far.

5.2. Numerical Illustration: Binomial Model

Let us now present numerical illustrations under the common binomial model to study the finite sample performance of the MREEs. Along with the known properties of the MREE at (i.e., the minimum LSD estimators with from [78,79]), here we will additionally explore their properties in case of and for the new divergences related to .

Suppose are random observations from a true density g having support for some positive integer m. We model g by the Binomial() densities for and . Here an estimate of g is given by the relative frequency . For any and , the relative -entropy between and is given by which can be minimized with respect to to obtain the corresponding MREE of . Note that, it is also the solution of the estimating Equation (61) with replaced by the relative frequency . However, in this example, and hence the MREE estimating equation simplifies to We can numerically solve the above estimating equation over , or equivalently over the transformed parameter , to obtain the corresponding MREE (i.e., the minimum LSD estimator).

We simulate random sample of size n from a binomial population with true parameter with and numerically compute the MREE. Repeating this exercise 1000 times, we can obtain an empirical estimate of the bias and the mean squared error (MSE) of the MREE of (since is very small in magnitude). Table 1 and Table 2 present these values for sample sizes and different values of tuning parameters and ; their existences are guaranteed by Corollary 2. Note that the choice gives the maximum likelihood estimator whereas only yields the minimum LDPD estimator with parameter . Next, in order to study the robustness, we contaminate 10% of each sample by random observations from a distant binomial distribution with parameters and and repeat the above simulation exercise; the resulting bias and MSE for the contaminated samples are given in Table 3 and Table 4. Our observations from these tables can be summarized as follows.

Table 1

Bias of the MREE for different , and sample sizes n under pure data.

β	α
	0.3	0.5	0.7	0.9	1	1.1	1.3	1.5	1.7	2
n=20
0.1	−0.210	−0.416	−0.397	−0.311	−0.277	−0.227	−0.130	0.021	0.024	0.122
0.3	2.218	−0.273	−0.229	−0.160	−0.141	−0.115	−0.096	−0.068	−0.036	0.034
0.5	−0.127	0.001	−0.125	−0.088	−0.082	−0.069	−0.058	−0.042	−0.032	−0.019
0.7	−0.093	−0.110	−0.010	−0.046	−0.044	−0.029	−0.023	−0.031	−0.023	−0.020
0.9	−0.066	−0.056	−0.028	−0.001	−0.015	−0.002	0.008	0.000	−0.006	−0.013
1	−0.041	−0.045	−0.017	0.005	−0.002	0.011	0.014	0.012	0.008	−0.003
1.3	−0.035	−0.013	0.023	0.036	0.030	0.039	0.088	0.039	0.035	0.021
1.5	−0.003	0.012	0.048	0.053	0.047	0.058	0.053	0.170	0.048	0.035
1.7	0.012	0.028	0.058	0.067	0.061	0.070	0.070	0.058	0.269	0.045
2	0.008	0.049	0.078	0.084	0.078	0.086	0.087	0.078	0.069	0.444
n=50
0.1	−0.085	−0.301	−0.254	−0.183	−0.156	−0.106	−0.002	0.114	0.292	0.245
0.3	1.829	−0.176	−0.150	−0.078	−0.066	−0.042	−0.045	−0.014	0.005	0.030
0.5	−0.056	0.099	−0.054	−0.037	−0.033	−0.026	−0.019	−0.009	−0.007	−0.005
0.7	−0.009	−0.059	0.035	−0.012	−0.013	−0.005	−0.002	−0.009	−0.002	0.006
0.9	−0.031	−0.031	−0.009	0.012	0.002	0.013	0.021	0.015	0.008	0.004
1	0.014	−0.023	0.000	0.011	0.009	0.019	0.022	0.020	0.018	0.004
1.3	0.002	−0.004	0.022	0.034	0.027	0.030	0.084	0.034	0.035	0.028
1.5	0.009	0.023	0.038	0.044	0.037	0.042	0.034	0.174	0.040	0.032
1.7	0.028	0.029	0.049	0.054	0.047	0.050	0.047	0.036	0.277	0.039
2	0.040	0.051	0.065	0.068	0.059	0.063	0.060	0.051	0.041	0.464
n=100
0.1	−0.028	−0.216	−0.175	−0.113	−0.103	−0.063	0.036	0.169	0.452	0.349
0.3	1.874	−0.135	−0.125	−0.052	−0.044	−0.022	−0.038	−0.023	0.009	0.024
0.5	−0.002	0.146	−0.034	−0.026	−0.025	−0.021	−0.019	-0.001	−0.008	−0.009
0.7	0.000	−0.042	0.045	−0.009	−0.013	−0.009	0.000	−0.009	−0.008	−0.001
0.9	0.007	−0.025	−0.015	0.001	−0.004	0.005	0.009	0.013	−0.001	−0.003
1	0.014	−0.010	−0.007	−0.001	−0.001	0.005	0.009	0.014	0.010	0.009
1.3	0.036	0.010	0.006	0.015	0.010	0.010	0.065	0.012	0.019	0.014
1.5	0.041	0.023	0.018	0.022	0.017	0.018	0.006	0.158	0.016	0.015
1.7	0.052	0.027	0.028	0.032	0.024	0.025	0.016	0.009	0.267	0.019
2	0.056	0.043	0.042	0.043	0.033	0.034	0.023	0.020	0.013	0.454

Table 2

MSE of the MREE for different , and sample sizes n under pure data.

β	α
	0.3	0.5	0.7	0.9	1	1.1	1.3	1.5	1.7	2
n=20
0.1	0.347	0.251	0.222	0.145	0.122	0.106	0.098	0.242	0.206	0.240
0.3	7.506	0.147	0.100	0.069	0.063	0.059	0.059	0.062	0.098	0.169
0.5	0.238	0.076	0.067	0.051	0.049	0.047	0.050	0.055	0.064	0.101
0.7	0.177	0.091	0.056	0.045	0.044	0.043	0.045	0.055	0.056	0.071
0.9	0.163	0.085	0.061	0.045	0.042	0.043	0.047	0.053	0.058	0.064
1	0.171	0.085	0.064	0.045	0.042	0.045	0.048	0.053	0.058	0.063
1.3	0.148	0.082	0.065	0.052	0.046	0.046	0.061	0.055	0.058	0.065
1.5	0.146	0.085	0.069	0.056	0.050	0.050	0.051	0.087	0.061	0.065
1.7	0.150	0.085	0.070	0.060	0.053	0.055	0.055	0.056	0.134	0.066
2	0.132	0.091	0.076	0.065	0.059	0.060	0.060	0.060	0.061	0.265
n=50
0.1	0.334	0.170	0.118	0.066	0.044	0.037	0.067	0.195	0.401	0.275
0.3	5.050	0.093	0.051	0.026	0.021	0.020	0.024	0.027	0.035	0.050
0.5	0.196	0.059	0.030	0.018	0.017	0.018	0.021	0.026	0.030	0.037
0.7	0.191	0.053	0.031	0.018	0.016	0.017	0.023	0.025	0.028	0.035
0.9	0.131	0.050	0.029	0.019	0.016	0.018	0.022	0.025	0.028	0.029
1	0.154	0.044	0.031	0.018	0.017	0.020	0.022	0.024	0.027	0.031
1.3	0.112	0.046	0.029	0.023	0.018	0.018	0.033	0.028	0.029	0.031
1.5	0.108	0.049	0.033	0.024	0.020	0.022	0.022	0.059	0.031	0.031
1.7	0.119	0.049	0.036	0.026	0.022	0.023	0.025	0.025	0.108	0.033
2	0.108	0.053	0.040	0.030	0.025	0.026	0.028	0.029	0.028	0.249
n=100
0.1	0.295	0.139	0.085	0.038	0.022	0.022	0.068	0.201	0.583	0.403
0.3	4.770	0.075	0.039	0.016	0.011	0.011	0.017	0.019	0.023	0.035
0.5	0.189	0.061	0.022	0.011	0.009	0.012	0.016	0.017	0.022	0.023
0.7	0.141	0.038	0.024	0.010	0.009	0.010	0.014	0.017	0.018	0.021
0.9	0.123	0.035	0.021	0.011	0.009	0.011	0.012	0.015	0.019	0.021
1	0.122	0.036	0.019	0.010	0.009	0.011	0.013	0.016	0.017	0.020
1.3	0.114	0.035	0.019	0.012	0.009	0.010	0.021	0.016	0.017	0.019
1.5	0.105	0.037	0.019	0.012	0.010	0.011	0.012	0.045	0.017	0.020
1.7	0.097	0.034	0.021	0.014	0.011	0.012	0.014	0.014	0.092	0.020
2	0.088	0.039	0.023	0.016	0.012	0.013	0.013	0.016	0.016	0.227

Table 3

Bias of the MREE for different , and sample sizes n under contaminated data.

β	α
	0.3	0.5	0.7	0.9	1	1.1	1.3	1.5	1.7	2
n=20
0.1	−0.104	−0.382	−0.340	−0.243	−0.131	−0.071	0.090	0.188	0.295	0.379
0.3	3.287	−0.157	−0.187	−0.135	−0.113	−0.091	−0.045	0.013	0.107	0.237
0.5	2.691	1.483	−0.024	−0.067	−0.069	−0.043	−0.031	−0.010	−0.003	0.051
0.7	3.004	2.546	1.168	0.036	−0.017	−0.008	0.003	0.006	0.005	0.010
0.9	3.133	2.889	2.319	0.917	0.222	0.058	0.019	0.023	0.017	0.022
1	3.183	2.986	2.558	1.619	0.805	0.214	0.039	0.030	0.031	0.019
1.3	3.239	3.121	2.902	2.550	2.262	1.872	0.613	0.077	0.049	0.040
1.5	3.255	3.170	3.012	2.775	2.606	2.396	1.676	0.571	0.069	0.051
1.7	3.271	3.194	3.071	2.903	2.790	2.661	2.256	1.489	0.578	0.057
2	3.289	3.216	3.122	3.012	2.942	2.865	2.649	2.305	1.690	0.682
n=50
0.1	0.384	−0.170	−0.189	−0.132	−0.054	0.024	0.104	0.171	0.261	0.382
0.3	3.549	0.000	−0.122	−0.086	−0.077	−0.053	−0.023	0.029	0.054	0.118
0.5	2.875	1.771	0.040	−0.048	−0.048	−0.029	−0.013	−0.015	−0.017	0.003
0.7	3.091	2.698	1.294	0.048	−0.010	−0.014	−0.001	0.004	0.001	−0.005
0.9	3.205	2.945	2.379	0.939	0.226	0.045	0.009	0.013	0.012	0.013
1	3.240	3.011	2.612	1.609	0.793	0.196	0.018	0.014	0.021	0.012
1.3	3.316	3.171	2.925	2.548	2.239	1.819	0.554	0.034	0.020	0.020
1.5	3.346	3.223	3.034	2.780	2.596	2.363	1.589	0.502	0.035	0.022
1.7	3.362	3.254	3.100	2.916	2.791	2.643	2.199	1.383	0.518	0.025
2	3.373	3.281	3.162	3.035	2.955	2.865	2.622	2.236	1.575	0.650
n=100
0.1	0.610	−0.138	−0.105	−0.031	0.002	0.040	0.117	0.184	0.270	0.381
0.3	3.906	0.136	−0.071	−0.050	−0.052	−0.028	−0.028	−0.008	0.023	0.066
0.5	2.927	1.934	0.101	−0.034	−0.027	−0.016	0.006	0.000	−0.003	−0.008
0.7	3.122	2.761	1.348	0.066	0.004	−0.007	0.007	0.011	0.012	0.000
0.9	3.241	2.955	2.406	0.958	0.238	0.047	0.004	0.014	0.022	0.017
1	3.289	3.045	2.651	1.622	0.798	0.202	0.010	0.011	0.016	0.023
1.3	3.362	3.204	2.944	2.567	2.245	1.812	0.533	0.028	0.015	0.022
1.5	3.384	3.269	3.058	2.802	2.610	2.369	1.567	0.485	0.027	0.018
1.7	3.405	3.305	3.133	2.940	2.811	2.658	2.196	1.357	0.504	0.018
2	3.421	3.327	3.204	3.065	2.980	2.886	2.633	2.234	1.541	0.637

Table 4

MSE of the MREE for different , and sample sizes n under contaminated data.

β	α
	0.3	0.5	0.7	0.9	1	1.1	1.3	1.5	1.7	2
n=20
0.1	0.403	0.248	0.465	0.576	1.025	1.093	1.613	1.565	1.626	1.591
0.3	12.595	0.142	0.103	0.075	0.192	0.188	0.362	0.590	1.016	1.537
0.5	7.443	2.268	0.088	0.062	0.058	0.059	0.065	0.189	0.241	0.527
0.7	9.209	6.645	1.410	0.069	0.056	0.058	0.063	0.068	0.119	0.208
0.9	9.982	8.493	5.512	0.882	0.119	0.068	0.065	0.069	0.075	0.090
1	10.292	9.072	6.672	2.692	0.693	0.117	0.068	0.070	0.076	0.087
1.3	10.664	9.916	8.574	6.641	5.240	3.610	0.430	0.079	0.079	0.089
1.5	10.778	10.229	9.238	7.850	6.940	5.883	2.917	0.389	0.079	0.087
1.7	10.884	10.379	9.599	8.582	7.942	7.234	5.235	2.326	0.403	0.087
2	11.004	10.515	9.915	9.233	8.814	8.369	7.177	5.472	2.998	0.547
n=50
0.1	1.552	0.815	0.741	0.703	0.966	1.190	1.129	1.224	1.165	1.210
0.3	14.969	0.105	0.047	0.030	0.078	0.075	0.280	0.559	0.566	0.881
0.5	8.345	3.190	0.049	0.025	0.021	0.022	0.025	0.029	0.035	0.184
0.7	9.634	7.335	1.694	0.031	0.020	0.022	0.027	0.029	0.033	0.039
0.9	10.353	8.723	5.712	0.898	0.077	0.027	0.028	0.030	0.032	0.039
1	10.578	9.126	6.871	2.619	0.645	0.067	0.027	0.030	0.033	0.039
1.3	11.069	10.129	8.608	6.548	5.064	3.359	0.329	0.033	0.034	0.038
1.5	11.263	10.457	9.268	7.787	6.801	5.648	2.576	0.279	0.032	0.038
1.7	11.371	10.655	9.676	8.567	7.854	7.051	4.908	1.968	0.298	0.037
2	11.449	10.833	10.060	9.275	8.793	8.276	6.947	5.079	2.560	0.461
n=100
0.1	2.102	0.399	0.808	0.945	0.924	0.929	0.891	1.012	1.233	1.120
0.3	17.185	0.141	0.033	0.018	0.013	0.014	0.018	0.142	0.258	0.453
0.5	8.624	3.768	0.056	0.015	0.011	0.015	0.017	0.018	0.022	0.028
0.7	9.809	7.646	1.828	0.024	0.011	0.013	0.018	0.019	0.020	0.023
0.9	10.559	8.764	5.812	0.927	0.070	0.018	0.017	0.020	0.021	0.023
1	10.870	9.312	7.058	2.648	0.645	0.057	0.017	0.019	0.021	0.023
1.3	11.342	10.306	8.691	6.619	5.068	3.312	0.297	0.020	0.020	0.023
1.5	11.494	10.727	9.379	7.880	6.845	5.646	2.484	0.251	0.021	0.021
1.7	11.632	10.960	9.848	8.675	7.932	7.101	4.866	1.873	0.272	0.022
2	11.739	11.102	10.297	9.422	8.910	8.363	6.973	5.040	2.420	0.430

Under pure data with no contamination, the maximum likelihood estimator (the MREE at ) has the least bias and MSE as expected, which further decrease as sample size increases.

As we move away from and in either direction, the MSEs of the corresponding MREEs under pure data increase slightly; but as long as the tuning parameters remain within a reasonable window of the point and neither component is very close to zero, this loss in efficiency is not very significant.

When or approaches zero, the MREEs become somewhat unstable generating comparatively larger MSE values. This is probably due to the presence of inliers under the discrete binomial model. Note that, the relative -entropy measures with are not finitely defined for the binomial model if there is just only one empty cell present in the data.

Under contamination, the bias and MSE of the maximum likelihood estimator increase significantly but many MREEs remains stable. In particular, the MREEs with and the MREEs with close to zero are non-robust against data contamination. Many of the remaining members of the MREE family provide significantly improved robust estimators.

In the entire simulation, the combination appears to provide the most stable results. In Table 4, the best results are available along a tubular region which moves from the top left-hand to the bottom right-hand of the table subject to the conditions that and none of them are very close to zero.

Based on our numerical experiments, the optimum range of values of , providing the most robust minimum relative -estimators are , and , . Note that this range includes the estimators based on the logarithmic power divergence measure as well as the new LSD measures with .

Many of the MREEs, which belong to the optimum range mentioned in the last item and are close to the combination , generally also provide the best trade-off between efficiency under pure data and robustness under contaminated data.

In summary, many MREEs provide highly robust estimators under data contamination along with only a very small loss in efficiency under pure data. These numerical findings about the finite sample behavior of the MREEs under the binomial model and the corresponding optimum range of tuning parameters, for the subclass with , are consistent with the findings of [78,79] who used a Poisson model. Additionally, our illustrations shed lights on the properties of the MREEs at as well and show that some MREEs in this range, e.g., at and , also yield optimum estimators in terms of the dual goal of high robustness and high efficiency.

5.3. Application to Testing Statistical Hypothesis

We end the paper with a very brief indication on the potential of the relative -entropy or the LSD measure in statistical hypothesis testing problems. The minimum possible value of the relative entropy or divergence measure between the data and the null distribution indicates the amount of departure from null and hence can be used to develop a statistical testing procedure.

Consider the parametric estimation set-up as in Section 5.1 with and fix a parameter value . Suppose we want to test the simple null hypothesis in the one sample case given by

Maji et al. [78] have developed the LSD-based test statistics for the above testing problem as given by where is the MREE with parameters and . [78,79] have also developed the LSD-based test for a simple two-sample problem where two independent samples of sizes and are given from true densities , respectively and we want to test for the homogeneity of the two samples trough the hypothesis The proposed test statistics for this two-sample problem has the form where and are the MREEs of and , respectively, obtained from the two samples separately Note that, at , both the test statistics in (63) and (64) become asymptotically equivalent to the corresponding likelihood ratio tests under the respective null hypothesis. Maji et al. [78,79] have studied the asymptotic properties of these two tests, which have asymptotic null distributions as linear combinations of chi-square distributions. They have also numerically illustrated the benefits of these LSD or relative -entropy-based tests, although with tuning parameters only, to achieve robust inference against possible contamination in the sample data.

The same approach can also be used to develop robust tests for more complex hypothesis testing problems based on the relative -entropy or the LSD measures, now with parameters , and also using the new divergences . For example, consider the above one sample set-up and a subset and let we are interested in testing the composite hypothesis with similar motivation from (63) and (64), we can construct relative entropy or LSD-based test statistics for testing the above composite hypothesis as given by where is the restricted MREE with parameters and obtained by minimizing the relative entropy over and is the corresponding unrestricted MREE obtained by minimizing over . It will surely be of significant interest to study the asymptotic and robustness properties of this relative entropy-based test for the above composite hypothesis under one sample or even more general hypotheses with two or more samples. However, considering the length of the present paper, which is primarily focused on the geometric properties of entropies and relative entropies, we have deferred the detailed analyses of such MREE-based hypothesis testing procedures in a future report.

6. Conclusions

We have explored the geometric properties of the LSD measures through a new information theoretic formulation when we develop this divergence measure as a natural extension of the relative -entropy; we refer to it as the two-parameter relative -entropy. It is shown to be always lower semicontinuous in both the arguments, but is continuous in its first argument only if . We also proved that the relative -entropy is quasi-convex in both its arguments after a suitable (different) transformation of the domain space and derive an extended Pythagorean relation under these transformations. Along with the study of its forward and reverse projections, statistical applications are also discussed.

It is worthwhile to note that the information theoretic divergences can also be used to define new measures of robustness and efficiency of a parameter estimate; one can then obtain the optimum robust estimator, along Hampel’s infinitesimal principle, to achieve the best trade-off between these divergence-based summary measures [111,112,113]. In particular, the LDPD measure, a prominent member of our LSD or relative -entropy family, has been used by [113] who have illustrated important theoretical properties including different types of equivariance of the resulting optimum estimators besides their strong robustness properties. A similar approach can also be used with our general relative -entropies to develop estimators with enhanced optimality properties, establishing a better robustness-efficiency trade-off. The present work opens up several interesting problems to be solved in future research as already noted throughout the paper. In particular, we recall that the relative -entropy has an interpretation from the problem of guessing under source uncertainty [17,71]. As an extension of relative -entropy, a similar information theoretic interpretation of the relative -entropy (i.e., the LSD) is expected and its proper interpretation will be a useful development. Additionally, we have obtained a new extension of the Renyi entropy as a by-product and detailed study of this new entropy measure and its potential applications may lead to a new aspect of the mathematical information theory. Also, statistical applications of these measures need to be studied thoroughly specially for the continuous models, where the complications of a kernel density estimator is unavoidable, and for testing complex composite hypotheses from one or more samples. We hope to pursue some of these interesting extensions in future.