On the Jensen-Shannon Symmetrization of Distances Relying on Abstract Means.

The Jensen-Shannon divergence is a renowned bounded symmetrization of the unbounded Kullback-Leibler divergence which measures the total Kullback-Leibler divergence to the average mixture distribution. However, the Jensen-Shannon divergence between Gaussian distributions is not available in closed form. To bypass this problem, we present a generalization of the Jensen-Shannon (JS) divergence using abstract means which yields closed-form expressions when the mean is chosen according to the parametric family of distributions. More generally, we define the JS-symmetrizations of any distance using parameter mixtures derived from abstract means. In particular, we first show that the geometric mean is well-suited for exponential families, and report two closed-form formula for (i) the geometric Jensen-Shannon divergence between probability densities of the same exponential family; and (ii) the geometric JS-symmetrization of the reverse Kullback-Leibler divergence between probability densities of the same exponential family. As a second illustrating example, we show that the harmonic mean is well-suited for the scale Cauchy distributions, and report a closed-form formula for the harmonic Jensen-Shannon divergence between scale Cauchy distributions. Applications to clustering with respect to these novel Jensen-Shannon divergences are touched upon.

1. Introduction and Motivations

1.1. Kullback–Leibler Divergence and Its Symmetrizations

Let be a measurable space [1] where denotes the sample space and the -algebra of measurable events. Consider a positive measure (usually the Lebesgue measure with Borel -algebra or the counting measure with power set -algebra ). Denote by the set of probability distributions.

The Kullback–Leibler Divergence [2] (KLD) is the most fundamental distance [2] between probability distributions, defined by:where p and q denote the Radon–Nikodym derivatives of probability measures P and Q with respect to (with ). The KLD expression between P and Q in Equation (1) is independent of the dominating measure . Table A1 summarizes the various distances and their notations used in this paper.

Table A1

Summary of Distances and Their Notations.

Weighted mean	M_α, α∈(0,1)
Arithmetic mean	A_α(x,y)=(1−α)x+αy
Geometric mean	G_α(x,y)=x^1−αy^α
Harmonic mean	H_α(x,y)=xy(1−α)y+αx
Power mean	Pαp(x,y)=((1−α)x^p+αy^p)^1p,p∈R∖{0}, lim_p→0Pαp=G
Quasi-arithmetic mean	Mαf(x,y)=f^−1((1−α)f(x)+αf(y)), f strictly monotonous
M-mixture	ZαM(p,q)=∫_t∈XM_α(p(t),q(t))dμ(t)with ZαM(p,q)=∫_t∈XM_α(p(t),q(t))dμ(t)
Statistical distance	D(p:q)
Dual/reverse distance D^*	D^*(p:q):=D(q:p)
Kullback-Leibler divergence	KL(p:q)=∫p(x)logp(x)q(x)dμ(x)
reverse Kullback-Leibler divergence	KL^*(p:q)=KL(q:p)=∫q(x)logq(x)p(x)dμ(x)
Jeffreys divergence	J(p;q)=KL(p:q)+KL(q:p)=∫(p(x)−q(x))logp(x)q(x)dμ(x)
Resistor divergence	1R(p;q)=121KL(p:q)+1KL(q:p). R(p;q)=2J(p;q)KL(p:q)KL(q:p)
skew K-divergence	K_α(p:q)=∫p(x)logp(x)(1−α)p(x)+αq(x)dμ(x)
Jensen-Shannon divergence	JS(p,q)=12KLp:p+q2+KLq:p+q2
skew Bhattacharrya divergence	B_α(p:q)=−log∫_Xp(x)^1−αq(x)^αdμ(x)
Hellinger distance	D_H(p,q)=1−∫_Xp(x)q(x)dμ(x)
α-divergences	I_α(p:q)=∫αp(x)+(1−α)q(x)−p(x)^αq(x)^1−αdμ(x),α∉{0,1} I_α(p:q)=A_α(q:p)−G_α(q:p)
Mahalanobis distance	D_Q(p:q)=(p−q)^⊤Q(p−q) for a positive-definite matrix Q≻0
f-divergence	I_f(p:q)=∫p(x)fq(x)p(x)dμ(x), with f(1)=f^′(1)=0f strictly convex at 1
reverse f-divergence	If*(p:q)=∫q(x)fp(x)q(x)dμ(x)=I_f^⋄(p:q)for f^⋄(u)=uf(1u)
J-symmetrized f-divergence	J_f(p;q)=12(I_f(p:q)+I_f(q:p))
JS-symmetrized f-divergence	Ifα(p;q):=(1−α)I_f(p:(pq)_α)+αI_f(q:(pq)_α)=I_fαJS(p:q)for fαJS(u):=(1−α)f(αu+1−α)+αfα+1−αu
Parameter distance
Bregman divergence	B_F(θ:θ^′):=F(θ)−F(θ^′)−〈θ−θ^′,∇F(θ^′)〉
skew Jeffreys-Bregman divergence	SFα=(1−α)B_F(θ:θ^′)+αB_F(θ^′:θ)
skew Jensen divergence	JFα(θ:θ^′):=(F(θ)F(θ^′))_α−F((θθ^′)_α)
Jensen-Bregman divergence	JB_F(θ;θ^′)=12B_Fθ:θ+θ^′2+B_Fθ^′:θ+θ^′2=J_F(θ;θ^′).
Generalized Jensen-Shannon divergences
skew J-symmetrization	JDα(p:q):=(1−α)Dp:q+αDq:p
skew JS-symmetrization	JSDα(p:q):=(1−α)Dp:(1−α)p+αq+αDq:(1−α)p+αq
skew M-Jensen-Shannon divergence	JS^M_α(p:q):=(1−α)KLp:(pq)αM+αKLq:(pq)αM
skew M-JS-symmetrization	JSDM_α(p:q):=(1−α)Dp:(pq)αM+αDq:(pq)αM
N-Jeffreys divergence	J^N_β(p:q):=N_β(KLp:q,KLq:p)
N-J D divergence	JDN_β(p:q)=N_β(D(p:q),D(q:p))
skew (M,N)-D JS divergence	JSDM_α,N_β(p:q):=N_βDp:(pq)αM,Dq:(pq)αM

The KLD is also called the relative entropy [2] because it can be written as the difference of the cross-entropy minus the entropy:where denotes the cross-entropy [2]:and denotes the Shannon entropy [2]. Although the formula of the Shannon entropy in Equation (4) unifies both the discrete case and the continuous case of probability distributions, the behavior of entropy in the discrete case and the continuous case is very different: When , Equation (4) yields the discrete Shannon entropy which is always positive and upper bounded by . When , Equation (4) defines the Shannon differential entropy which may be negative and unbounded [2] (e.g., the differential entropy of the Gaussian distribution is ). See also [3] for further important differences between the discrete case and the continuous case.

In general, the KLD is an asymmetric distance (i.e., , hence the argument separator notation using the delimiter ‘:’) In information theory [2], it is customary to use the double bar notation ‘‖’ instead of the comma ‘,’ notation to avoid confusion with joint random variables. The reverse KL divergence or dual KL divergence is:

In general, the reverse distance or dual distance for a distance D is written as:

One way to symmetrize the KLD is to consider the Jeffreys Divergence [4] (JD, Sir Harold Jeffreys (1891–1989) was a British statistician.):

However, this symmetric distance is not upper bounded, and its sensitivity can raise numerical issues in applications. Here, we used the optional argument separator notation ‘;’ to emphasize that the distance is symmetric but not necessarily a metric distance. This notation matches the notational convention of the mutual information if two joint random variables in information theory [2].

The symmetrization of the KLD may also be obtained using the harmonic mean instead of the arithmetic mean, yielding the resistor average distance [5] :

Another famous symmetrization of the KLD is the Jensen–Shannon Divergence [6] (JSD) defined by:

This distance can be interpreted as the total divergence to the average distribution (see Equation (10)). The JSD can be rewritten as a Jensen divergence (or Burbea–Rao divergence [7]) for the negentropy generator (called Shannon information):

An important property of the Jensen–Shannon divergence compared to the Jeffreys divergence is that this distance is always bounded:

This follows from the fact that

Finally, the square root of the JSD (i.e., ) yields a metric distance satisfying the triangular inequality [8,9]. The JSD has found applications in many fields such as bioinformatics [10] and social sciences [11], just to name a few. Recently, the JSD has gained attention in the deep learning community with the Generative Adversarial Networks (GANs) [12]. In computer vision and pattern recognition, one often relies on information-theoretic techniques to perform registration and recognition tasks. For example, in [13], the authors use a mixture of Principal Axes Registrations (mPAR) whose parameters are estimated by minimizing the KLD between the considered two-point distributions. In [14], the authors parameterize both shapes and deformations using Gaussian Mixture Models (GMMs) to perform non-rigid shape registration. The lack of closed-form formula for the KLD between GMMs [15] spurred the use of other statistical distances which admit a closed-form expression for GMMs. For example, in [16], shape registration is performed by using the Jensen-Rényi divergence between GMMs. See also [17] for other information-theoretic divergences that admit closed-form formula for some statistical mixtures extending GMMs.

In information geometry [18], the KLD, JD and JSD are invariant divergences which satisfy the property of information monotonicity [18]. The class of (separable) distances satisfying the information monotonicity are exhaustively characterized as Csiszár’s f-divergences [19]. A f-divergence is defined for a convex generator function f strictly convex at 1 (with ) by:

The Jeffreys and Jensen–Shannon f-generators are:

1.2. Statistical Distances and Parameter Divergences

In information and probability theory, the term “divergence” informally means a statistical distance [2]. However in information geometry [18], a divergence has a stricter meaning of being a smooth parametric distance (called a contrast function in [20]) from which a dual geometric structure can be derived [21,22].

Consider parametric distributions belonging to a parametric family of distributions (e.g., Gaussian family or Cauchy family), where denotes the parameter space. Then a statistical distance D between distributions and amount to an equivalent parameter distance:

For example, the KLD between two distributions belonging to the same exponential family (e.g., Gaussian family) amount to a reverse Bregman divergence for the cumulant generator F of the exponential family [23]:

A Bregman divergence is defined for a strictly convex and differentiable generator F as:where is an inner product (usually the Euclidean dot product for vector parameters).

Similar to the interpretation of the Jensen–Shannon divergence (statistical divergence) as a Jensen divergence for the negentropy generator, the Jensen–Bregman divergence [7] (parametric divergence JBD) amounts to a Jensen divergence for a strictly convex generator :

Let us introduce the notation to denote the linear interpolation (LERP) of the parameters. Then we have more generally that the skew Jensen–Bregman divergence amounts to a skew Jensen divergence :

1.3. J-Symmetrization and -Symmetrization of Distances

For any arbitrary distance , we can define its skew J-symmetrization for by:and its JS-symmetrization by:

Usually, , and for notational brevity, we drop the superscript: . The Jeffreys divergence is twice the J-symmetrization of the KLD, and the Jensen–Shannon divergence is the -symmetrization of the KLD.

The J-symmetrization of a f-divergence is obtained by taking the generator where is the conjugate generator:

The -symmetrization of a f-divergence with is obtained by taking the generator

We check that we have:

A family of symmetric distances unifying the Jeffreys divergence with the Jensen–Shannon divergence was proposed in [24]. Finally, let us mention that once we have symmetrized a distance D, we may also metrize this symmetric distance by choosing (when it exists) the largest exponent such that becomes a metric distance [8,25,26,27,28].

1.4. Contributions and Paper Outline

The paper is organized as follows:

Section 2 reports the special case of mixture families in information geometry [18] for which the Jensen–Shannon divergence can be expressed as a Bregman divergence (Theorem 1), and highlight the lack of closed-form formula when considering exponential families. This fact precisely motivated this work.

Section 3 introduces the generalized Jensen–Shannon divergences using statistical mixtures derived from abstract weighted means (Definitions 2 and 5), presents the JS-symmetrization of statistical distances, and report a sufficient condition to get bounded JS-symmetrizations (Property 1).

In Section 4.1, we consider the calculation of the geometric JSD between members of the same exponential family (Theorem 2) and instantiate the formula for the multivariate Gaussian distributions (Corollary 1). We discuss about applications for k-means clustering in Section 4.1.2. In Section 4.2, we illustrate the method with another example that calculates in closed form the harmonic JSD between scale Cauchy distributions (Theorem 4).

Finally, we wrap up and conclude this work in Section 5.

2. Jensen–Shannon Divergence in Mixture and Exponential Families

We are interested to calculate the JSD between densities belonging to parametric families of distributions.

A trivial example is when and are categorical distributions: The average distribution is a again categorical distribution, and the JSD is expressed plainly as:

Another example is when and both belong to the same mixture family [18] :for linearly independent component distributions . We have [29]:where is a Bregman divergence defined in Equation (20) obtained for the convex negentropy generator [29] . The proof that is a strictly convex function is not trivial [30].

The mixture families include the family of categorical distributions over a finite alphabet (the D-dimensional probability simplex) since those categorical distributions form a mixture family with . Beware that mixture families impose to prescribe the component distributions. Therefore, a density of a mixture family is a special case of statistical mixtures (e.g., GMMs) with prescribed component distributions.

The mathematical identity of Equation (35) that does not yield a practical formula since is usually not itself available in closed form. Worse, the Bregman generator can be non-analytic [31]. Nevertheless, this identity is useful for computing the right-sided Bregman centroid (left KL centroid of mixtures) since this centroid is equivalent to the center of mass, and independent of the Bregman generator [29].

Since the mixture of mixtures is also a mixture, specifically it follows that we get a closed-form expression for the JSD between mixtures belonging to .

The Jensen–Shannon divergence between two distributions

This amounts to calculate the Jensen divergence: where

Now, consider distributions and belonging to the same exponential family [18] :where denotes the natural parameter space. We have [18]:where F denotes the log-normalizer or cumulant function of the exponential family [18].

However, does not belong to in general, except for the case of the categorical/multinomial family which is both an exponential family and a mixture family [18].

For example, the mixture of two Gaussian distributions with distinct components is not a Gaussian distribution. Thus, it is not obvious to get a closed-form expression for the JSD in that case. This limitation precisely motivated the introduction of generalized JSDs defined in the next section.

Notice that in [32,33], it is shown how to express or approximate the f-divergences using expansions of power pseudo-distances. These power chi distances can all be expressed in closed form when dealing with isotropic Gaussians. This result holds for the JSD since the JSD is a f-divergence [33].

3. Generalized Jensen–Shannon Divergences

We first define abstract means M, and then generic statistical M-mixtures from which generalized Jensen–Shannon divergences are built thereof.

Definitions

Consider an abstract mean [34] M. That is, a continuous bivariate function on an interval that satisfies the following in-betweenness property:

Using the unique dyadic expansion of real numbers, we can always build a corresponding weighted mean (with ) following the construction reported in [34] (page 3) such that and . In the remainder, we consider .

Examples of common weighted means are:

the arithmetic mean ,

the geometric mean , and

the harmonic mean .

These means can be unified using the concept of quasi-arithmetic means [34] (also called Kolmogorov–Nagumo means):where h is a strictly monotonous function. For example, the geometric mean is obtained as for the generator . Rényi used the concept of quasi-arithmetic means instead of the arithmetic mean to define axiomatically the Rényi entropy [35] of order in information theory [2].

For any abstract weighted mean, we can build a statistical mixture called a M-mixture as follows:

The-interpolation whereis the normalizer function (or scaling factor) ensuring that

The A-mixture (‘A’ standing for the arithmetic mean) represents the usual statistical mixture [36] (with ). The G-mixture of two distributions and (’G’ standing for the geometric mean G) is an exponential family of order [37] 1:

The two-component M-mixture can be generalized to a k-component M-mixture with , the -dimensional standard simplex:where .

For a given pair of distributions p and q, the set describes a path in the space of probability density functions. This density interpolation scheme was investigated for quasi-arithmetic weighted means in [38,39,40]. In [41], the authors study the Fisher information matrix for the -mixture models (using -power means).

We call the -weighted M-mixture, thus extending the notion of -mixtures [42] obtained for power means . Notice that abstract means have also been used to generalize Bregman divergences using the concept of [43].

Let us state a first generalization of the Jensen–Shannon divergence:

For a mean M, the skew M-Jensen–Shannon divergence (for

When , we recover the ordinary Jensen–Shannon divergence since (and ).

We can extend the definition to the -symmetrization of any distance:

For a mean M and a distance D, the skew M-

By notation, we have . That is, the arithmetic JS-symmetrization of the KLD is the JSD.

Let us define the -skew K-divergence [6,44] as where . Then the Jensen–Shannon divergence and the Jeffreys divergence can be rewritten [24] as since . Then . Similarly, we can define the generalized skew K-divergence:

The success of the JSD compared to the JD in applications is partially due to the fact that the JSD is upper bounded by . So, one question to ask is whether those generalized JSDs are upper bounded or not?

To report a sufficient condition, let us first introduce the dominance relationship between means: We say that a mean M dominates a mean N when for all , see [34]. In that case we write concisely . For example, the Arithmetic-Geometric-Harmonic (AGH) inequality states that .

Consider the term

When mean dominates the arithmetic mean , we have and

Notice that (when is the arithmetic mean), and we recover the fact that the -skew Jensen–Shannon divergence is upper bounded by (e.g., when ).

We summarize the result in the following property:

The M-JSD is upper bounded by

Let us observe that dominance of means can be used to define distances: For example, the celebrated -divergences can be interpreted as a difference of two means, the arithmetic mean and the geometry mean:

We can also define the generalized Jeffreys divergence as follows:

For a mean N, the skew N-Jeffreys divergence (for

This definition includes the (scaled) resistor average distance [5] , obtained for the harmonic mean for the KLD with skew parameter :

In [5], the factor is omitted to keep the spirit of the original Jeffreys divergence.

We can further extend this definition for any arbitrary divergence D as follows:

The skew

We now show how to choose the abstract mean according to the parametric family of distributions to obtain some closed-form formula for some statistical distances.

4. Some Closed-Form Formula for the M-Jensen–Shannon Divergences

Our motivation to introduce these novel families of M-Jensen–Shannon divergences is to obtain closed-form formula when probability densities belong to some given parametric families . We shall illustrate the principle of the method to choose the right abstract mean for the considered parametric family, and report corresponding formula for the following two case studies:

The geometric G-Jensen–Shannon divergence for the exponential families (Section 4.1), and

the harmonic H-Jensen–Shannon divergence for the family of Cauchy scale distributions (Section 4.2).

Recall that the arithmetic A-Jensen–Shannon divergence is well-suited for mixture families (Theorem 1).

4.1. The Geometric G-Jensen–Shannon Divergence

Consider an exponential family [37] with log-normalizer F:and natural parameter space

The log-normalizer (a log-Laplace function also called log-partition or cumulant function) is a real analytic convex function.

We seek for a mean M such that the weighted M-mixture density of two densities and of the same exponential family yields another density of that exponential family (e.g., ). When considering exponential families, choose the weighted geometric mean for the abstract mean : , for . Indeed, it is well-known that the normalized weighted product of distributions belonging to the same exponential family also belongs to this exponential family [45]: where the normalization factor is for the skew Jensen divergence defined by:

Notice that since the natural parameter space is convex, the distribution (since ).

Thus, it follows that we have:

This allows us to conclude that the G-Jensen–Shannon divergence admits the following closed-form expression between densities belonging to the same exponential family:

Please note that since and , it follows that .

The dual divergence [46] (with respect to the reference argument) or reverse divergence of a divergence D is defined by swapping the calling arguments: .

Thus, if we defined the Jensen–Shannon divergence for the dual KL divergence then we obtain:

Please note that .

In general, the JS-symmetrization for the reverse KL divergence is where and . Since (arithmetic-geometric inequality), it follows that .

The α-skew G-Jensen–Shannon divergence

4.1.1. Case Study: The Multivariate Gaussian Family

Consider the exponential family [18,37] of multivariate Gaussian distributions [47,48,49]

The multivariate Gaussian family is also called the multivariate normal family in the literature, or MVN family for short.

Let denote the composite (vector,matrix) parameter of an MVN. The d-dimensional MVN density is given by where denotes the matrix determinant. The natural parameters are also expressed using both a vector parameter and a matrix parameter in a compound object . By defining the following compound inner product on a composite (vector,matrix) object where denotes the matrix trace, we rewrite the MVN density of Equation (83) in the canonical form of an exponential family [37]:where is the compound natural parameter and is the compound sufficient statistic. The function is the strictly convex and continuously differentiable log-normalizer defined by:

The log-normalizer can be expressed using the ordinary parameters, , as:

The moment/expectation parameters [18,49] are

We report the conversion formula between the three types of coordinate systems (namely the ordinary parameter , the natural parameter and the moment parameter ) as follows:

The dual Legendre convex conjugate [18,49] is and .

We check the Fenchel-Young equality when and :

The Kullback–Leibler divergence between two d-dimensional Gaussians distributions and (with ) is

We check that since and . Notice that when , we have that is half the squared Mahalanobis distance for the precision matrix (a positive-definite matrix: ), where the Mahalanobis distance is defined for any positive matrix as follows:

The Kullback–Leibler divergence between two probability densities of the same exponential families amount to a Bregman divergence [18]:where the Bregman divergence is defined by with . Define the canonical divergence [18] since . We have .

Now, observe that when . In particular, this holds for the multivariate normal family. Thus, we have the following proposition.

For the MVN family, we have with the scaling normalization factor:

More generally, we have for a k-dimensional weight vector belonging to the -dimensional standard simplex:where .

Finally, we state the formulas for the G-JS divergence between MVNs for the KL and reverse KL, respectively:

The skew G-Jensen–Shannon divergence where (matrix harmonic barycenter) and

Notice that the α-skew Bhattacharyya distance [7]:between two members of the same exponential family amounts to a -skew Jensen divergence between the corresponding natural parameters:

A simple proof follows from the fact that

Therefore, we have with . Thus, it follows that

The JS-symmetrization of the reverse Kullback–Leibler divergence between densities of the same exponential family amount to calculate a Jensen/Burbea–Rao divergence between the corresponding natural parameters.

4.1.2. Applications to k-Means Clustering

Let denote a point set, and denote a set of k (cluster) centers. The generalized k-means objective [23] with respect to a distance D is defined by:

By defining the distance of a point to a set of points, we can rewrite compactly the objective function as . Denote by the minimum objective loss for a set of clusters: . It is NP-hard [50] to compute when and the dimension . The most common heuristic is Lloyd’s batched k-means [23] that yields a local minimum.

The performance of the probabilistic k-means++ initialization [51] has been extended to arbitrary distances in [52] as follows:

(Generalized k-means++ performance, [53]). Let and

Then the generalized k-means++ seeding guarantees with high probability a configuration C of cluster centers such that:

To bound the constants and , we rewrite the generalized Jensen–Shannon divergences using quadratic form expressions: That is, using a squared Mahalanobis distance:for a positive-definite matrix . Since the Bregman divergence can be interpreted as the tail of a first-order Taylor expansion, we have:for (open convex). Similarly, the Jensen divergence can be interpreted as a Jensen–Bregman divergence, and thus we have for . More precisely, for a prescribed point set , we have , where denotes the closed convex hull. We can therefore upper bound and using the ratio . See [54] for further details.

A centroid for a set of parameters is defined as the minimizer of the functional

In particular, the symmetrized Bregman centroids have been studied in [55] (for ), and the Jensen centroids (for ) have been investigated in [7] using the convex-concave iterative procedure.

4.2. The Harmonic Jensen–Shannon Divergence (H-)

The principle to get closed-form formula for generalized Jensen–Shannon divergences between distributions belonging to a parametric family consists of finding an abstract mean M such that the M-mixture belongs to the family . In particular, when is a convex domain, we seek a mean M such that with .

Let us consider the weighted harmonic mean [34] (induced by the harmonic mean) H:

The harmonic mean is a quasi-arithmetic mean obtained for the monotone (decreasing) function (or equivalently for the increasing monotone function ).

This harmonic mean is well-suited for the scale family of Cauchy probability distributions (also called Lorentzian distributions):where denotes the scale and the standard Cauchy distribution.

Using the computer algebra system Maxima (http://maxima.sourceforge.net/) we find that (see Appendix B) where the normalizing coefficient is since we have .

The H-Jensen–Shannon symmetrization of a distance D between distributions writes as:where denote the weighted harmonic mean. When D is available in closed form for distributions belonging to the scale Cauchy distributions, so is .

For example, consider the KL divergence formula between two scale Cauchy distributions:where A and G denote the arithmetic and geometric means, respectively. The formula initially reported in [56] has been corrected by the authors. Since (and ), it follows that . Notice that the KL divergence is symmetric for Cauchy scale distributions. We note in passing that for exponential families, the KL divergence is symmetric only for the location Gaussian family (since the only symmetric Bregman divergences are the squared Mahalanobis distances [57]). The cross-entropy between scale Cauchy distributions is , and the differential entropy is .

Then the H-JS divergence between and is:

We check that when , we have .

The harmonic Jensen–Shannon divergence between two scale Cauchy distributions

Let us report some numerical examples: Consider and , we find that . When and , we find that .

Notice that KL formula is scale-invariant, and this property holds for any scale family:

The Kullback–Leibler divergence between two distributions

A direct proof follows from a change of variable in the KL integral with and . Please note that although the KLD between scale Cauchy distributions is symmetric, it is not the case for all scale families: For example, the Rayleigh distributions form a scale family with the KLD amounting to compute a Bregman asymmetric Itakura–Saito divergence between parameters [37].

Instead of the KLD, we can choose the total variation distance for which a formula has been reported in [38] between two Cauchy distributions. Notice that the Cauchy distributions are alpha-stable distributions for and q Gaussian distributions for ([58], p. 104). A closed-form formula for the divergence between two q-Gaussians is given in [58] when . The definite integral is available in closed form for Cauchy distributions. When , we have .

We refer to [38] for yet other illustrative examples considering the family of Pearson type VII distributions and central multivariate t-distributions which use the power means (quasi-arithmetic means induced by for ) for defining mixtures.

Table 1 summarizes the various examples introduced in the paper.

Table 1

Summary of the weighted means M chosen according to the parametric family in order to ensure that the family is closed under M-mixturing: .

JS^M_α	Mean M	Parametric Family	ZαM(p:q)
JS^A_α	arithmetic A	mixture family	ZαM(θ_1:θ_2)=1
JS^G_α	geometric G	exponential family	ZαG(θ_1:θ_2)=exp(−JFα(θ_1:θ_2))
JS^H_α	harmonic H	Cauchy scale family	ZαH(θ_1:θ_2)=θ_1θ_2(θ_1θ_2)_α(θ_1θ_2)_1−α

4.3. The M-Jensen–Shannon Matrix Distances

In this section, we consider distances between matrices which play an important role in quantum computing [59,60]. We refer to [61] for the matrix Jensen–Bregman logdet divergence. The Hellinger distance can be interpreted as the difference of an arithmetic mean A and a geometric mean G:

Notice that since , we have . The scaled and squared Hellinger distance is an -divergence for . Recall that the -divergence can be interpreted as the difference of a weighted arithmetic minus a weighted geometry mean.

In general, if a mean dominates a mean , we may define the distance as

When considering matrices [62], there is not a unique definition of a geometric matrix mean, and thus we have different notions of matrix Hellinger distances [62], some of them are divergences (smooth distances defining a dualistic structure in information geometry).

We define the matrix M-Jensen–Shannon divergence for a matrix divergence [63,64] D as follows:

For example, we can choose the von Neumann matrix divergence [63]:or the LogDet matrix divergence [63]:where square matrices and have dimension d.

5. Conclusions and Perspectives

We introduced a generalization of the celebrated Jensen–Shannon divergence [6], termed the , based on M-mixtures derived from abstract means M. This new family of divergences includes the ordinary Jensen–Shannon divergence when both M and N are set to the arithmetic mean. We reported closed-form expressions of the M Jensen–Shannon divergences for mixture families and exponential families in information geometry by choosing the arithmetic and geometric weighted mean, respectively. The -skewed geometric Jensen–Shannon divergence (G-Jensen–Shannon divergence) between densities and of the same exponential family with cumulant function F is

Here, we used the bracket notation to emphasize that the statistical distance is between densities, and the parenthesis notation to emphasize that the distance is between parameters. We also have . We also show how to get a closed-form formula for the harmonic Jensen–Shannon divergence of Cauchy scale distributions by taking harmonic mixtures.

For an arbitrary distance D, we define the skew N-Jeffreys symmetrization:and the skew :

A Java™ source code for computing the geometric Jensen–Shannon divergence between multivariate Gaussian distributions is available online at https://franknielsen.github.io/M-JS/.