On f-Divergences: Integral Representations, Local Behavior, and Inequalities.

This paper is focused on f-divergences, consisting of three main contributions. The first one introduces integral representations of a general f-divergence by means of the relative information spectrum. The second part provides a new approach for the derivation of f-divergence inequalities, and it exemplifies their utility in the setup of Bayesian binary hypothesis testing. The last part of this paper further studies the local behavior of f-divergences.

1. Introduction

Probability theory, information theory, learning theory, statistical signal processing and other related disciplines, greatly benefit from non-negative measures of dissimilarity (a.k.a. divergence measures) between pairs of probability measures defined on the same measurable space (see, e.g., [1,2,3,4,5,6,7]). An axiomatic characterization of information measures, including divergence measures, was provided by Csiszár [8]. Many useful divergence measures belong to the set of f-divergences, independently introduced by Ali and Silvey [9], Csiszár [10,11,12,13], and Morimoto [14] in the early sixties. The family of f-divergences generalizes the relative entropy (a.k.a. the Kullback- Leibler divergence) while also satisfying the data processing inequality among other pleasing properties (see, e.g., [3] and references therein).

Integral representations of f-divergences serve to study properties of these information measures, and they are also used to establish relations among these divergences. An integral representation of f-divergences, expressed by means of the DeGroot statistical information, was provided in [3] with a simplified proof in [15]. The importance of this integral representation stems from the operational meaning of the DeGroot statistical information [16], which is strongly linked to Bayesian binary hypothesis testing. Some earlier specialized versions of this integral representation were introduced in [17,18,19,20,21], and a variation of it also appears in [22] Section 5.B. Implications of the integral representation of f-divergences, by means of the DeGroot statistical information, include an alternative proof of the data processing inequality, and a study of conditions for the sufficiency or -deficiency of observation channels [3,15].

Since many distance measures of interest fall under the paradigm of an f-divergence [23], bounds among f-divergences are very useful in many instances such as the analysis of rates of convergence and concentration of measure bounds, hypothesis testing, testing goodness of fit, minimax risk in estimation and modeling, strong data processing inequalities and contraction coefficients, etc. Earlier studies developed systematic approaches to obtain f-divergence inequalities while dealing with pairs of probability measures defined on arbitrary alphabets. A list of some notable existing f-divergence inequalities is provided, e.g., in [22] Section 1 and [23] Section 3. State-of-the-art techniques which serve to derive bounds among f-divergences include:

Moment inequalities which rely on log-convexity arguments ([22] Section 5.D, [24,25,26,27,28]);

Inequalities which rely on a characterization of the exact locus of the joint range of f-divergences [29];

f-divergence inequalities via functional domination ([22] Section 3, [30,31,32]);

Sharp f-divergence inequalities by using numerical tools for maximizing or minimizing an f-divergence subject to a finite number of constraints on other f-divergences [33];

Inequalities which rely on powers of f-divergences defining a distance [34,35,36,37];

Vajda and Pinsker-type inequalities for f-divergences ([4,10,13,22] Sections 6–7, [38,39]);

Bounds among f-divergences when the relative information is bounded ([22] Sections 4–5, [40,41,42,43,44,45,46,47]), and reverse Pinsker inequalities ([22] Section 6, [40,48]);

Inequalities which rely on the minimum of an f-divergence for a given total variation distance and related bounds [4,33,37,38,49,50,51,52,53];

Bounds among f-divergences (or functions of f-divergences such as the Rényi divergence) via integral representations of these divergence measures [22] Section 8;

Inequalities which rely on variational representations of f-divergences (e.g., [54] Section 2).

Following earlier studies of the local behavior of f-divergences and their asymptotic properties (see related results by Csiszár and Shields [55] Theorem 4.1, Pardo and Vajda [56] Section 3, and Sason and Vérdu [22] Section 3.F), it is known that the local behavior of f-divergences scales, such as the chi-square divergence (up to a scaling factor which depends on f) provided that the first distribution approaches the reference measure in a certain strong sense. The study of the local behavior of f-divergences is an important aspect of their properties, and we further study it in this work.

This paper considers properties of f-divergences, while first introducing in Section 2 the basic definitions and notation needed, and in particular the various measures of dissimilarity between probability measures used throughout this paper. The presentation of our new results is then structured as follows:

Section 3 is focused on the derivation of new integral representations of f-divergences, expressed as a function of the relative information spectrum of the pair of probability measures, and the convex function f. The novelty of Section 3 is in the unified approach which leads to integral representations of f-divergences by means of the relative information spectrum, where the latter cumulative distribution function plays an important role in information theory and statistical decision theory (see, e.g., [7,54]). Particular integral representations of the type of results introduced in Section 3 have been recently derived by Sason and Verdú in a case-by-case basis for some f-divergences (see [22] Theorems 13 and 32), while lacking the approach which is developed in Section 3 for general f-divergences. In essence, an f-divergence is expressed in Section 3 as an inner product of a simple function of the relative information spectrum (depending only on the probability measures P and Q), and a non-negative weight function which only depends on f. This kind of representation, followed by a generalized result, serves to provide new integral representations of various useful f-divergences. This also enables in Section 3 to characterize the interplay between the DeGroot statistical information (or between another useful family of f-divergence, named the divergence with ) and the relative information spectrum.

Section 4 provides a new approach for the derivation of f-divergence inequalities, where an arbitrary f-divergence is lower bounded by means of the divergence [57] or the DeGroot statistical information [16]. The approach used in Section 4 yields several generalizations of the Bretagnole-Huber inequality [58], which provides a closed-form and simple upper bound on the total variation distance as a function of the relative entropy; the Bretagnole-Huber inequality has been proved to be useful, e.g., in the context of lower bounding the minimax risk in non-parametric estimation (see, e.g., [5] pp. 89–90, 94), and in the problem of density estimation (see, e.g., [6] Section 1.6). Although Vajda’s tight lower bound in [59] is slightly tighter everywhere than the Bretagnole-Huber inequality, our motivation for the generalization of the latter bound is justified later in this paper. The utility of the new inequalities is exemplified in the setup of Bayesian binary hypothesis testing.

Section 5 finally derives new results on the local behavior of f-divergences, i.e., the characterization of their scaling when the pair of probability measures are sufficiently close to each other. The starting point of our analysis in Section 5 relies on the analysis in [56] Section 3, regarding the asymptotic properties of f-divergences.

The reading of Section 3, Section 4 and Section 5 can be done in any order since the analysis in these sections is independent.

2. Preliminaries and Notation

We assume throughout that the probability measures P and Q are defined on a common measurable space , and denotes that P is absolutely continuous with respect to Q, namely there is no event such that .

The relative information provided byaccording to, where, is given by

More generally, even if, let R be an arbitrary dominating probability measure such that(e.g.,); irrespectively of the choice of R, the relative information is defined to be

The following asymmetry property follows from (

The relative information spectrum is the cumulative distribution function

The relative entropy is the expected valued of the relative information when it is distributed according to P:

Throughout this paper, denotes the set of convex functions with . Hence, the function is in ; if , then for all ; and if , then . We next provide a general definition for the family of f-divergences (see [3] p. 4398).

(f-divergence [9,10,12]). Let P and Q be probability measures, let μ be a dominating measure of P and Q (i.e., ; e.g., ), and let and . The f-divergence from P to Q is given, independently of μ, by where

We rely in this paper on the following properties of f-divergences:

Let. The following conditions are equivalent:

there exists a constantsuch that

Let, and letbe the conjugate function, given byfor. Then,; , and for every pair of probability measures,

By an analytic extension of in (12) at , let

Note that the convexity of implies that . In continuation to Definition 3, we get with the convention in (16) that , We refer in this paper to the following f-divergences:

Relative entropy: with

Jeffrey’s divergence [: with

Hellinger divergence of order [2] Definition 2.10: with

Some of the significance of the Hellinger divergence stems from the following facts:

The analytic extension of at yields

The chi-squared divergence [61] is the second order Hellinger divergence (see, e.g., [62] p. 48), i.e.,

Note that, due to Proposition 1, where can be defined as

The squared Hellinger distance (see, e.g., [62] p. 47), denoted by , satisfies the identity

The Bhattacharyya distance [63], denoted by , satisfies

The Rényi divergence of order is a one-to-one transformation of the Hellinger divergence of the same order [11] (14):

The Alpha-divergence of order , as it is defined in [64] and ([65] (4)), is a generalized relative entropy which (up to a scaling factor) is equal to the Hellinger divergence of the same order . More explicitly, where denotes the Alpha-divergence of order . Note, however, that the Beta and Gamma-divergences in [65], as well as the generalized divergences in [66,67], are not f-divergences in general.

divergence for [2] (2.31), and the total variation distance: The function results in

Specifically, for , let and the total variation distance is expressed as an f-divergence:

Triangular Discrimination [: with

Note that

Lin’s measure [68] (4.1): for . This measure can be expressed by the following f-divergence: with

The special case of (41) with gives the Jensen-Shannon divergence (a.k.a. capacitory discrimination):

divergence [57] p. 2314: For , with and , and where (46) follows from the Neyman-Pearson lemma. The divergence can be identified as an f-divergence: with where . The following relation to the total variation distance holds:

DeGroot statistical information [3,16]: For , with

The following relation to the total variation distance holds: and the DeGroot statistical information and the divergence are related as follows [22] (384):

3. New Integral Representations of -Divergences

The main result in this section provides new integral representations of f-divergences as a function of the relative information spectrum (see Definition 2). The reader is referred to other integral representations (see [15] Section 2, [4] Section 5, [22] Section 5.B, and references therein), expressing a general f-divergence by means of the DeGroot statistical information or the divergence.

Letbe a strictly convex function at 1. Letbe defined aswheredenotes the right-hand derivative of f at 1 (due to the convexity of f on, it exists and it is finite). Then, the function g is non-negative, it is strictly monotonically decreasing on, and it is strictly monotonically increasing onwith.

For any function , let be given by and let be the conjugate function, as given in (12). The function g in (54) can be expressed in the form as it is next verified. For , we get from (12) and (55), and the substitution for yields (56) in view of (54).

By assumption, is strictly convex at 1, and therefore these properties are inherited to . Since also , it follows from [3] Theorem 3 that both and are non-negative on , and they are also strictly monotonically decreasing on . Hence, from (12), it follows that the function is strictly monotonically increasing on . Finally, the claimed properties of the function g follow from (56), and in view of the fact that the function is non-negative with , strictly monotonically decreasing on and strictly monotonically increasing on . ☐

Letbe a strictly convex function at 1, and letbe as in (54). Let and let and be the two inverse functions of g. Then,

In view of Lemma 1, it follows that is strictly monotonically increasing and is strictly monotonically decreasing with .

Let , and let . Then, we have where (61) relies on Proposition 1; (62) relies on Proposition 2; (64) follows from (3); (65) follows from (56); (66) holds by the definition of the random variable V; (67) holds since, in view of Lemma 1, , and for any non-negative random variable Z; (68) holds in view of the monotonicity properties of g in Lemma 1, the definition of a and b in (58) and (59), and by expressing the event as a union of two disjoint events; (69) holds again by the monotonicity properties of g in Lemma 1, and by the definition of its two inverse functions and as above; in (67)–(69) we are free to substitute > by ≥, and < by ≤; finally, (70) holds by the definition of the relative information spectrum in (4). ☐

The functionin (54) is invariant to the mapping , for , with an arbitrary . This invariance of g (and, hence, also the invariance of its inverse functions and ) is well expected in view of Proposition 1 and Lemma 2.

For the chi-squared divergence in (26), letting f be as in (27), it follows from (54) that which yields, from (58) and ( . Calculation of the two inverse functions of g, as defined in Lemma 2, yields the following closed-form expression:

Substituting (72) into (60) provides an integral representation of .

Let . Then, we have where (74) holds by (4); (75) follows from (3); (76) holds by the substitution ; (77) holds since , and finally (78) holds since . ☐

Unlike Example 1, in general, the inverse functionsandin Lemma 2 are not expressible in closed form, motivating our next integral representation in Theorem 1.

The following theorem provides our main result in this section.

The following integral representations of an f-divergence, by means of the relative information spectrum, hold:

Let

be differentiable on;

be the non-negative weight function given, for, by

the function be given by

Then,

More generally, for an arbitrary, letbe a modified real-valued function defined as

Then,

We start by proving the special integral representation in (81), and then extend our proof to the general representation in (83).

We first assume an additional requirement that f is strictly convex at 1. In view of Lemma 2,

Since by assumption is differentiable on and strictly convex at 1, the function g in (54) is differentiable on . In view of (84) and (85), substituting in (60) for implies that where is given by for , where (88) follows from (54). Due to the monotonicity properties of g in Lemma 1, (87) implies that for , and for . Hence, the weight function in (79) satisfies

The combination of (80), (86) and (89) gives the required result in (81).

We now extend the result in (81) when is differentiable on , but not necessarily strictly convex at 1. To that end, let be defined as

This implies that is differentiable on , and it is also strictly convex at 1. In view of the proof of (81) when f is strict convexity of f at 1, the application of this result to the function s in (90) yields

In view of (6), (22), (23), (25) and (90), from (79), (89), (90) and the convexity and differentiability of , it follows that the weight function satisfies for . Furthermore, by applying the result in (81) to the chi-squared divergence in (25) whose corresponding function for is strictly convex at 1, we obtain

Finally, the combination of (91)–(94), yields ; this asserts that (81) also holds by relaxing the condition that f is strictly convex at 1.

In view of (80)–(82), in order to prove (83) for an arbitrary , it is required to prove the identity

Equality (95) can be verified by Lemma 3: by rearranging terms in (95), we get the identity in (73) (since ). ☐

Due to the convexity of f, the absolute value in the right side of (79) is only needed for (see (88) and (89)). Also, since .

The weight functiononly depends on f, and the functiononly depends on the pair of probability measures P and Q. In view of Proposition 1, it follows that, for, the equalityholds onif and only if (11) is satisfied with an arbitrary constant . It is indeed easy to verify that (11) yields on .

An equivalent way to writein (80) is where . Hence, the function is monotonically increasing in , and it is monotonically decreasing in ; note that this function is in general discontinuous at 1 unless . If , then

Note that if, thenis zero everywhere, which is consistent with the fact that.

In the proof of Theorem 1-(1), the relaxation of the condition of strict convexity at 1 for a differentiable function (32), the function

Theorem 1-(1) with

Theorem 1 yields integral representations for various f-divergences and related measures; some of these representations were previously derived by Sason and Verdú in [22] in a case by case basis, without the unified approach of Theorem 1. We next provide such integral representations. Note that, for some f-divergences, the function is not differentiable on ; hence, Theorem 1 is not necessarily directly applicable.

The following integral representations hold as a function of the relative information spectrum:

Relative entropy [

Hellinger divergence of order

In particular, the chi-squared divergence, squared Hellinger distance and Bhattacharyya distance satisfywhere (100) appears in [

Rényi divergence [

In particular, the following identities hold for the total variation distance:where (105) appears in [

DeGroot statistical information:

Triangular discrimination:

Lin’s measure: For where

Jeffrey’s divergence:

See Appendix A. ☐

An application of (112) yields the following interplay between the divergence and the relative information spectrum.

Let

Then,

the sets

for

We start by proving the first item. By our assumption, is continuous on . Hence, it follows from (112) that is continuously differentiable in ; furthermore, (45) implies that is monotonically decreasing in , which yields .

We next prove the second and third items together. Let and . From (112), for , which yields (115). Due to the continuity of , it follows that the set determines the relative information spectrum on .

To prove (116), we have where (120) holds by switching P and Q in (46); (121) holds since ; (122) holds by switching P and Q in (115) (correspondingly, also and are switched); (123) holds since ; (124) holds by the assumption that has no probability masses, which implies that the sign < can be replaced with ≤ at the term in the right side of (123). Finally, (116) readily follows from (120)–(124), which implies that the set determines on .

Equalities (117) and (117) finally follows by letting , respectively, on both sides of (115) and (116). ☐

A similar application of (107) yields an interplay between DeGroot statistical information and the relative information spectrum.

Let

Then,

and

the sets

for for and

By relaxing the condition in Theorems 3 and 4 where determines

In view of Theorems 1, 3 and 4 and Remark 8, we get the following result.

Let (131) and (132), respectively, determines

Corollary 1 is supported by the integral representation of where

4. New -Divergence Inequalities

Various approaches for the derivation of f-divergence inequalities were studied in the literature (see Section 1 for references). This section suggests a new approach, leading to a lower bound on an arbitrary f-divergence by means of the divergence of an arbitrary order (see (45)) or the DeGroot statistical information (see (50)). This approach leads to generalizations of the Bretagnole-Huber inequality [58], whose generalizations are later motivated in this section. The utility of the f-divergence inequalities in this section is exemplified in the setup of Bayesian binary hypothesis testing.

In the following, we provide the first main result in this section for the derivation of new f-divergence inequalities by means of the divergence. Generalizing the total variation distance, the divergence in (45)–(47) is an f-divergence whose utility in information theory has been exemplified in [17] Chapter 3, [54],[57] p. 2314 and [69]; the properties of this measure were studied in [22] Section 7 and [54] Section 2.B.

Let

Let and be the densities of P and Q with respect to a dominating measure . Then, for an arbitrary , where (139) follows from the convexity of and by invoking Jensen’s inequality.

Setting with gives and where (146) follows from (143) by setting . Substituting (143) and (146) into the right side of (139) gives (135). ☐

An application of Theorem 5 gives the following lower bounds on the Hellinger and Rényi divergences with arbitrary positive orders, expressed as a function of the divergence with an arbitrary order .

For all and

Inequality (147), for , follows from Theorem 5 and (22); for , it holds in view of Theorem 5, and equalities (17) and (24). Inequality (148), for , follows from (30) and (147); for , it holds in view of (24), (147) and since . ☐

Specialization of Corollary 2 for in (147) and in (148) gives the following result.

For

From [

is a tight lower bound on the chi-squared divergence as a function of the total variation distance. In view of (49), we compare (151) with the specialized version of (149) when (151), as a result of the use of Jensen’s inequality in the proof of Theorem 5; however, it is interesting to examine how much we loose in the tightness of this specialized bound with (49), the substitution of (149) gives

and, it can be easily verified that

if (152) is at most twice smaller than the tight lower bound in the right side of (151);

if (151).

Setting

Inequality (153) forms a counterpart to Pinsker’s inequality: proved by Csiszár [12] and Kullback [70], with Kemperman [71] independently a bit later. As upper bounds on the total variation distance, (154) outperforms (153) if nats, and (153) outperforms (154) for larger values of .

In [

The lower bound in the right side of (155) is asymptotically tight in the sense that it tends to ∞ if (153), on the other hand, is equivalent to

Although it can be verified numerically that the lower bound on the relative entropy in (155) is everywhere slightly tighter than the lower bound in (156) (for (156) is appealing since it provides a closed-form simple upper bound on (153)), whereas such a closed-form simple upper bound cannot be obtained from (155). In fact, by the substitution (155), we get the inequality (155) is equivalent to the following upper bound on the total variation distance as a function of the relative entropy: where W in the right side of (157) denotes the principal real branch of the Lambert W function. The difference between the upper bounds in (153) and (157) can be verified to be marginal if (153) is clearly more simple and amenable to analysis.

The Bretagnole-Huber inequality in (153) is proved to be useful in the context of lower bounding the minimax risk (see, e.g., [

In [22] Section 7.C, Sason and Verdú generalized Pinsker’s inequality by providing an upper bound on the divergence, for , as a function of the relative entropy. In view of (49) and the optimality of the constant in Pinsker’s inequality (154), it follows that the minimum achievable is quadratic in for small values of . It has been proved in [22] Section 7.C that this situation ceases to be the case for , in which case it is possible to upper bound as a constant times where this constant tends to infinity as we let . We next cite the result in [22] Theorem 30, extending (154) by means of the divergence for , and compare it numerically to the bound in (150).

([ where the supremum is over where

As an immediate consequence of (159), it follows that which forms a straight-line bound on the divergence as a function of the relative entropy for . Similarly to the comparison of the Bretagnole-Huber inequality (153) and Pinsker’s inequality (154), we exemplify numerically that the extension of Pinsker’s inequality to the divergence in (162) forms a counterpart to the generalized version of the Bretagnole-Huber inequality in (150).

Figure 1 plots an upper bound on the divergence, for , as a function of the relative entropy (or, alternatively, a lower bound on the relative entropy as a function of the divergence). The upper bound on for , as a function of , is composed of the following two components:

Figure 1

Upper bounds on the divergence, for , as a function of the relative entropy (the curvy and straight lines follow from (150) and (162), respectively).

the straight-line bound, which refers to the right side of (162), is tighter than the bound in the right side of (150) if the relative entropy is below a certain value that is denoted by in nats (it depends on );

the curvy line, which refers to the bound in the right side of (150), is tighter than the straight-line bound in the right side of (162) for larger values of the relative entropy.

It is supported by Figure 1 that is positive and monotonically increasing, and ; e.g., it can be verified that , , , and (see Figure 1).

Bayesian Binary Hypothesis Testing

The DeGroot statistical information [16] has the following meaning: consider two hypotheses and , and let and with . Let P and Q be probability measures, and consider an observation Y where , and . Suppose that one wishes to decide which hypothesis is more likely given the observation Y. The operational meaning of the DeGroot statistical information, denoted by , is that this measure is equal to the minimal difference between the a-priori error probability (without side information) and a posteriori error probability (given the observation Y). This measure was later identified as an f-divergence by Liese and Vajda [3] (see (50) here).

The DeGroot statistical information satisfies the following upper bound as a function of the chi-squared divergence: and the following bounds as a function of the relative entropy:

where (160);

The first bound in (163) holds by combining (53) and (149); the second bound in (164) follows from (162) and (53) for , and it follows from (52) and (154) when ; finally, the third bound in (165) follows from (150) and (53). ☐

The bound in (164) forms an extension of Pinsker’s inequality (154) when (52), the bound in (165) is specialized to the Bretagnole-Huber inequality in (153) by letting .

Numerical evidence shows that none of the bounds in (163)–(165) supersedes the others.

The upper bounds on (163) and (165) are asymptotically tight when we let which implies that also (163) and (165) are specialized to a-priori error probability, is also equal to the DeGroot statistical information since the a-posterior error probability tends to zero in the considered extreme case where P and Q are sufficiently far from each other, so that

Due to the one-to-one correspondence between the (53), which shows that the two measures are related by a multiplicative scaling factor, the numerical results shown in (164) and (165); i.e., for (164) is tighter than the second bound in (165) for small values of the relative entropy, whereas (165) becomes tighter than (164) for larger values of the relative entropy.

Let (12). Then,

for

for

Inequalities (167) and (168) follow by combining (135) and (53). ☐

We end this section by exemplifying the utility of the bounds in Theorem 7.

Let where

Without any loss of generality, let

In this example, we compare the simple closed-form bounds on (163)–(165) with its exact value

To simplify the right side of (174), let where for

Hence, from (174)–(176),

To exemplify the utility of the bounds in Theorem 7, suppose that μ and λ are close, and we wish to obtain a guarantee on how small (163)–(165) are, respectively, equal to (175)), and the calculation of the right side of (178) appears to be sensitive to the selected parameters in this setting.

5. Local Behavior of f-Divergences

This section studies the local behavior of f-divergences; the starting point relies on [56] Section 3 which studies the asymptotic properties of f-divergences. The reader is also referred to a related study in [22] Section 4.F.

Let

the sequence where

Then

The result in (180) follows from [56] Theorem 3, even without the additional restriction in [56] Section 3 which would require that the second derivatives of f and g are locally Lipschitz at a neighborhood of 1. More explicitly, in view of the analysis in [56] p. 1863, we get by relaxing the latter restriction that (cf. [56] (31)) where as we let , and also

By our assumption, due to the continuity of and at 1, it follows from (181) and (182) that which yields (180) (recall that, by assumption, ). ☐

Since f and g in Lemma 4 are assumed to have continuous second derivatives at 1, the left and right derivatives of the weight function (79) at 1 satisfy, in view of Remark 3,

Hence, the limit in the right side of (180) is equal to

Let and be the densities of P and Q with respect to an arbitrary probability measure such that . Then, ☐

The result in Lemma 5, for the chi-squared divergence, is generalized to the identity

for all (33)). The special case of

The result in Lemma 5 can be generalized as follows: let

with

If (192), and (191) is specialized to (186). However, if (191) scales linearly in λ if

We next state the main result in this section.

Let

P and Q be probability measures defined on a measurable space

Then,

Let be a sequence in , which tends to zero. Define the sequence of probability measures

Note that implies that for all . Since it follows from (194) that

Consequently, (183) implies that where in (197) is an arbitrary sequence which tends to zero. Hence, it follows from (197) and (200) that and, by combining (186) and (201), we get

We next prove the result for the limit in the right side of (195). Let be the conjugate function of f, which is given in (12). By the assumption that f has a second continuous derivative, so is and it is easy to verify that the second derivatives of f and coincide at 1. Hence, from (13) and (202), ☐

Although an f-divergence is in general not symmetric, in the sense that the equality (195) stems from the fact that the second derivatives of f and

Under the conditions in Theorem 8, it follows from (196) that where (206) relies on L’Hôpital’s rule. The convexity of

The following result refers to the local behavior of Rényi divergences of an arbitrary non-negative order.

Under the condition in (194), for every

Let . In view of (23) and Theorem 8, it follows that the local behavior of the Hellinger divergence of order satisfies

The result now follows from (30), which implies that

The result in (208) and (209), for , follows by combining the equalities in (210)–(214).

Finally, the result in (208) and (209) for follows from its validity for all , and also due to the property where is monotonically increasing in (see [73] Theorem 3). ☐