Stochastic Order and Generalized Weighted Mean Invariance.

In this paper, we present order invariance theoretical results for weighted quasi-arithmetic means of a monotonic series of numbers. The quasi-arithmetic mean, or Kolmogorov-Nagumo mean, generalizes the classical mean and appears in many disciplines, from information theory to physics, from economics to traffic flow. Stochastic orders are defined on weights (or equivalently, discrete probability distributions). They were introduced to study risk in economics and decision theory, and recently have found utility in Monte Carlo techniques and in image processing. We show in this paper that, if two distributions of weights are ordered under first stochastic order, then for any monotonic series of numbers their weighted quasi-arithmetic means share the same order. This means for instance that arithmetic and harmonic mean for two different distributions of weights always have to be aligned if the weights are stochastically ordered, this is, either both means increase or both decrease. We explore the invariance properties when convex (concave) functions define both the quasi-arithmetic mean and the series of numbers, we show its relationship with increasing concave order and increasing convex order, and we observe the important role played by a new defined mirror property of stochastic orders. We also give some applications to entropy and cross-entropy and present an example of multiple importance sampling Monte Carlo technique that illustrates the usefulness and transversality of our approach. Invariance theorems are useful when a system is represented by a set of quasi-arithmetic means and we want to change the distribution of weights so that all means evolve in the same direction.

1. Introduction and Motivation

Stochastic orders [1,2], are orders defined in probability theory and statistics, to quantify the concept of one random variable being bigger or smaller than another one. Discrete probability distributions, also called probability mass functions (pmf), are sequences of n-tuples of non-negative values that add up to 1, and can be thus interpreted in several ways, for instance, as weights in computation of moments of the discrete random variable described by the pmf, or as equivalence classes of compositional data [3]. Stochastic orders have found application in decision and risk theory [4], and in economics in general, among many other fields [2]. Some stochastic orders have been defined based on order invariance: two pmf’s are ordered when the arithmetic means of any increasing sequence of real numbers weighted with the corresponding pmf’s are ordered in the same direction. This raises the question whether this invariance might also hold for other kind of means beyond the arithmetic mean.

The quasi-arithmetic means, also called Kolmogorov or Kolmogorov–Nagumo means, are ubiquitous in many branches of science [5]. They have the expression , where is a real-valued strictly monotonous function, a sequence of reals, and a set of weights with . This family of means comprises the usual means: arithmetic , , harmonic , , power mean , . For a long time, economists have discussed the best mean for a problem [6]. Harmonic mean is used for the price earning ratio, and power means are used to represent the aggregate labor demand and its corresponding wage [7], and the constant elasticity of substitution (CES) [8]. Yoshida [9,10] has studied the invariance under quasi-arithmetic means with function increasing and for utility functions. In information theory, Alfred Rényi [11] defined axiomatically the entropy of a probability distribution as a Kolmogorov mean of the information conveyed by result k with probability , and recently, Americo et al. [12] defined conditional entropy based on quasi-arithmetic means. In physics, the equivalent spring constant of springs combined in series is obtained as the harmonic mean of the individual spring constants, and in parallel, as their arithmetic mean [13], while the equivalent resistance of resistors combined in parallel is obtained by the harmonic mean of the individual resistances, and in series by their arithmetic mean [14]. In traffic flow [15], arithmetic and harmonic mean of speed distribution are used. In [16] both geometric and harmonic mean are used in addition to arithmetic mean to improve noise source maps.

In our recent work on inequalities for generalized, quasi-arithmetic weighted means [17], we found some invariance properties, that depended on the particular relationship considered between the sequences of weights. These relationships between weights define first stochastic order, and likelihood ratio order. Their application to multiple importance sampling (MIS), a Monte Carlo technique, has been presented in [18], its application to cross entropy in [19], and in [20] applications to image processing, traffic flow and income distribution have been shown. In [21], the invariance results on products of distributions of independent scalar r.v.’s [22] was generalized to any joint distribution of a 2-dimensional r.v.

In this paper, we show that the order invariance is a necessary and sufficient condition for first stochastic order, and that it holds under any quasi-arithmetic mean. We also study invariance under the second stochastic order, likelihood ratio, hazard-rate, and increasing convex stochastic orders. The fact that the invariance results hold for both increasing and decreasing monotonic functions allows us to use both utilities and liabilities, represented by efficiencies and expected error, respectively, to look for an optimal solution, where in liabilities we look for minimum expected error, while in utilities for maximum efficiency.

The rest of the paper is organized as follows. In Section 2, we introduce the stochastic order; in Section 3, the arithmetic mean and its relationship with stochastic order. In Section 4, we present the invariance theorems; in Section 5, we discuss the invariance for concave (convex) functions; in Section 6, its application to stochastic orders; in Section 7, we present an example based on the linear combination of Monte Carlo estimators. Finally, conclusions and future work are given in Section 8.

2. Stochastic Orders

Stochastic orders are pre-orders (i.e, binary relationships holding symmetric and transitive properties) defined on probability distributions with finite support. Note that equivalently, one can think of sequences (i.e., ordered sets) of non-negative weights/values that sum up to one. Observe that any sequence of M positive numbers such that can be considered a probability distribution. It can be seen too as an element of the (M-1)-Simplex, i.e., . While several interpretations hold and hence increase the range of applicability, in the remaining of this paper, we will talk of sequence without any loss of generality.

We use the symbols first ) and whenever last ). The case with both

(Toy example). Given sequences

(Mirror property). One desirable property of stochastic orders is such that when reversing the ordering of the sequence elements, the stochastic order should be reversed as well, i.e., if mirror property.

(Mirror property). We say that a stochastic order has the mirror property if

Observe that the simple orders defined before, , , do not hold this property, but holds it. We will see in Section 6 that usual stochastic orders do hold the mirror property. However, an order that is insensitive to the permutation of the elements of a sequence, like majorization or Lorentz order, does not hold the mirror property.

3. Quasi-Arithmetic Mean

Stochastic orders are usually defined by invariance to arithmetic mean [1,2] (see Section 6), and we want to investigate in this paper invariance to more general means. We define here the kind of means we are interested in.

(Quasi-arithmetic or Kolmogorov or Kolmogorov–Nagumo mean). A quasi-arithmetic weighted mean (or Kolmogorov mean)

Examples of such mean are arithmetic weighted mean (), harmonic weighted mean (), geometric weighted mean () and in general, the weighted power means ().

Given a distribution , Shannon entropy, , and Rényi entropy, , can be considered as quasi-arithmetic means of the sequence with weights : Shannon entropy with (arithmetic mean or expected value), and Rényi entropy with [11]. Tsallis entropy, , can be considered the weighted arithmetic mean of the sequence with weights , where is the q-logarithm function [23]. Without loss of generality, we consider from now on that where is a real valued function.

Consider the sequences of M positive weights

(a) for

(b) for

(a’) for

(b’) for

(c) the following inequalities hold:

(d) the following inequalities hold: If

Indirect or partial proofs can be found in [17,20]. We provide a complete proof in the Appendix A. □

Observe that in Lemma 1, it is sufficient to consider the monotonicity of the sequences

Given a mean

(a) for all increasing functions

(b) for all increasing functions

(a’) for all decreasing functions

(b’) for all decreasing functions

(c) Condition (c) of Lemma 1 holds

It is a direct consequence of Lemma 1 and the definition of quasi-arithmetic mean, observing that the inverse of a strictly monotonic increasing (respectively decreasing) function is also increasing (respectively decreasing). □

(Invariance). Given two distributions

(a) for all increasing functions

(b) for all increasing functions

(a’) for all decreasing functions

(b’) for all decreasing functions

It is a direct consequence of the observation that conditions (c) and (d) in Lemma 1 do not depend on any particular function considered, and thus the order of inequalities does not change with the mean considered as long as and are kept fixed. □

4. Invariance

The following properties relate stochastic order with the quasi-arithmetic mean. Let be the set of monotonous functions, the set of increasing functions, the set of decreasing functions.

(preserve mean order property). We say that a stochastic order preserves mean order for a given mean

(preserve inverse mean order property). We say that a stochastic order preserves inverse mean order for a given mean

Theorem 2 together with the preserve mean order properties allows us to state the following invariance property:

(preserve mean order invariance). Given a stochastic order that preserves mean order (respectively preserves inverse mean order) for a given mean and for

Observe that from Lemma 1 and Theorems 1 and 3, we have that a necessary and sufficient condition for an order to preserve mean order for (or preserve inverse mean order for ) is the holding of Equation (6), independently of the mean considered. We will see in Section 6 that this corresponds to first stochastic dominance order.

5. Concavity and Convexity

Let us consider now , the set of all increasing concave functions, , the set of all increasing convex functions, , the set of all decreasing concave functions, and , the set of all decreasing convex functions. The following theorem relates the preserve mean order properties with the mirror property.

If an order holds the mirror property, then holding preserve mean order for

Suppose and decreasing and concave (respectively convex). Then, by the mirror property , and by the hypothesis of the theorem where , and because if is decreasing and concave (respectively convex) then is increasing and concave (respectively convex). □

The following result is necessary to prove Lemma 2.

Given

Consider the quasi-arithmetic mean with Kolmogorov function (with inverse ). When is increasing, is decreasing and viceversa. When is convex, is concave and viceversa. We have

□

The following Lemma is needed to prove how the invariance properties for one mean extends to other means.

Given two distributions and Then, for each line in

Additionally, for each line in

The proof of lines 1–8 in Table 1 is in the Appendix B. Lines 1’-8’ in Table 1 are a direct consequence of Theorem 5 applied to lines 1–8. □

Table 1

For each line, for filling the conditions in first and second column, then Equations (20) and (21) hold for filling the conditions in columns three and four. By changing from increasing to decreasing, the reverse of Equations (20) and (21) hold. ICX: convex and increasing, ICV: concave and increasing, DCX: convex and decreasing, DCV: concave and decreasing.

	g	f	g^🟉	f^🟉
1	ICX	ICV	ICV	ICV
1’	"	"	DCX	ICV
2	DCX	ICX	DCV	ICX
2’	"	"	ICX	ICX
3	ICV	ICX	ICX	ICX
3’	"	"	DCV	ICX
4	DCV	ICV	DCX	ICV
4’	"	"	ICV	ICV
5	ICX	ICV	DCV	ICX
5’	"	"	ICX	ICX
6	DCX	ICX	ICV	ICV
6’	"	"	DCX	ICV
7	ICV	ICX	DCX	ICV
7’	"	"	ICV	ICV
8	DCV	ICV	ICX	ICX
8’	"	"	DCV	ICX

Given a quasi-arithmetic mean

It is enough to apply Lemma 2, taking into account that if Equation (20) holds then Equation (21) holds too by the mirror property. □

Consider the weighted arithmetic mean

(a) If order is preserved for mean

(b) If order is preserved for mean

Arithmetic mean is a quasi-arithmetic mean with increasing function , which is both concave-increasing and convex-increasing, and thus Table 1 collapses to Table 2.

Table 2

For each line, , filling the conditions in second column and Equations (20) and (21) holding, then Equations (20) and (21) hold too for filling the conditions in columns three and four. ICX: convex and increasing, ICV: concave and increasing, DCX: convex and decreasing, DCV: concave and decreasing.

f	g^🟉	f^🟉
ICV	ICV	ICV
ICV	DCX	ICV
ICX	ICX	ICX
ICX	DCV	ICX
ICV	DCV	ICX
ICV	ICX	ICX
ICX	DCX	ICV
ICX	ICV	ICV

□

Functions , for are convex-increasing and with are convex-decreasing over , convex-increasing over , while , , for are concave-increasing over . Affine functions are both concave and convex over . If is convex, is concave and viceversa, and the composition of concave-increasing and concave is concave, and convex-increasing and convex is convex. We will see in the next section that preserving the order for mean for is defined as second-order stochastic dominance or increasing concave order, and preserving the order for mean for is defined as increase convex order stochastic dominance. Both orders hold the mirror property and thus Corollary 1 applies to both.

6. Application to Stochastic Orders and Cross-Entropy

6.1. First-Order Stochastic Dominance

Originating in the economics risk literature [24], first-order stochastic dominance [1,2] FSD, between two probability distributions, , , , is defined as:

Remember that the expected value is the arithmetic weighted mean.

A necessary and sufficient condition for first-order stochastic dominance is defined by the condition of Lemma 1, which is equivalent to condition of Lemma 1, thus , i.e., the mirror property holds. From the definition of FSD, and from Theorem 3, we can redefine FSD, as there exists a mean such that, for any increasing function , Equation (16) holds. This is, the definition of FSD is independent of the mean considered, while the original definition relies on the expected value (arithmetic mean). The mean considered can be arithmetic, harmonic, geometric or any other quasi-arithmetic mean.

Let us consider now a strictly monotonous function , and define a generalized cross-entropy . Observe that it is a quasi-arithmetic mean, and for , we get the cross-entropy . Other functions that generalize cross entropy have been defined in the context of training deep neural networks [25]. We can state the following theorem:

Given distributions

Observe first that is a decreasing sequence. From the hypothesis, condition of Lemma 1 holds, thus we can then apply Theorem 1 to the mean with function and for decreasing. □

The following result relates the entropies of two distributions with the first stochastic order.

Given

The Kullback–Leibler distance is always positive [26], , and thus we have that . Applying Theorem 7 for , □

Second-order stochastic dominance between two probability distributions,

6.2. Second-Order Stochastic Dominance and Increase Convex Ordering

Trivially Let us consider the cumulative distribution function, , and the survival function . A necessary and sufficient condition for second-order stochastic dominance [1,2] is the following: or equivalently,

Let us see first that , i.e., SSD holds the mirror property. Effectively, define , and the survival function . However, , , , , and substituting in Equation (22) or Equation (23), we obtain the desired result.

From Corollary 1, second-order stochastic dominance preserves mean order for all means defined by a concave-increasing/convex-decreasing (respectively concave-decreasing/convex-increasing) function , and for the set of all concave-increasing functions (respectively convex-increasing functions ). For instance, order is preserved for the geometric mean, or any mean with , and for , or for or any mean with with , as the harmonic mean, for . In particular, we can state the following theorem about cross-entropy of two distributions,

Given distributions

We have that , where stands for geometric mean. The geometric mean is a quasi-arithmetic mean with function , concave-increasing. Using Definition 6 and applying Corollary 1(a), we obtain the inequality and we apply the function to both members of this inequality. □

When we consider , a convex instead of a concave function, we talk of increasing convex order, ICX. This is,

Second-order stochastic dominance between two probability distributions,

is greater in increasing convex order than , , if and only if the following inequalities hold: or equivalently,

Trivially Mirror property is also immediate. From Corollary 1, second-order stochastic dominance preserves mean order for all means defined by a convex-increasing/concave-decreasing (respectively convex-decreasing/concave-increasing) function and for the set of all convex-increasing functions (respectively concave-increasing functions ). For instance, any mean with , as the weighted quadratic mean with , or , as the harmonic mean with , or the mean with with . In particular, we can state the following result,

Given distributions

Consider the quasi-arithmetic mean with , convex-decreasing. We have that . Using Definition 7 and applying Corollary 1(b), we have that , and we apply to each member of the inequality the function . □

Likelihood ratio dominance, LR,

6.3. Likelihood Ratio Dominance

As , we have that LR holds the mirror property.

It can be shown that LR order implies FSD order,

and then LR order holds Theorems 3 and 7 and Equation (16) holds for any mean ,

Likelihood-ratio order implies first stochastic order, i.e.,

There are indirect proofs in the literature [1,17]. A direct proof is in the Appendix C. □

As the condition for LR order, Equation (26), is easy to check, this order comes very handy in proving sufficient condition for FSD order. Additionally, for the uniform distribution and any increasing distribution , we have that , while for any decreasing distribution we have that .

Consider Shannon entropy, , and Rényi entropy of a distribution , that as seen in Section 3 are quasi-arithmetic means of the sequence with weights , and with and , respectively. Without loss of generality, we can consider in increasing order, then the sequence will be decreasing. We have that , and then we can apply Theorem 3, and obtain for Shannon entropy where is the arithmetic mean of , and for Rényi entropy, observing that , For Tsallis entropy, which can be considered the weighted arithmetic mean of the sequence with weights , as for in increasing order the sequence will be decreasing (the derivative of is positive), we have similarly to Shannon entropy,

We can also state the following result for the generalized cross entropy . We say that is comonotonic [27,28] with when, for all , .

Given distributions

Without loss of generality, we can reorder so that both are increasing. We know that and thus . From the definition of , preserve mean order holds for increasing functions and for arithmetic mean, thus we can then apply Theorem 3 for decreasing. □

From Theorem 12, when are comonotonic, then

A probability distribution

6.4. Hazard Rate

Observe that this can be written as or

If we consider now the sequences written in inverse order, i.e., , it is clear that if then , and from , we get which means , and HR order holds the mirror property.

It can be shown that and then HR order holds Theorems 3 and 7, and Equation (16) holds for any mean .

7. Example: Linear Combination of Monte Carlo Techniques

When we want to estimate an integral using MIS (multiple importance sampling) Monte Carlo methods, we have several choices of techniques, each of them with a given pdf , , which provide the primary estimators . If for , we have that , then the technique is unbiased. We are interested in optimal ways of combining the techniques. One option is linearly combining the different estimators , , with weights and sampling proportions , with . If all techniques are unbiased, the resulting combination is also unbiased. The variance is given by where are the variances for the primary estimators of each technique and V is the variance for the primary MIS estimator.

The optimal combination of weights, i.e. the one that leads to minimum variance, has been studied in [18,29,30].

The variance value will depend on two sets of weights, and , but there are cases where we can reduce it to a single set of weights. We present the following examples, where variances are taken for :

when , then , the weighted arithmetic mean () of .

when the sampling proportions are fixed, the optimal variance is given by , the weighted harmonic mean () of .

when weights are fixed, the optimal variance is given by , which is the weighted power mean of with exponent ().

Observe that the variance in these three cases is a quasi-arithmetic mean.

Let us now order in increasing order, and let us take , decreasing, and . We have that and thus, by Theorem 11, , and we can apply Theorem 2 with to the quasi-arithmetic means above. Thus, the variance is less when taking sampling proportions or coefficients decreasing in than for equal sampling or equal weighting.

The same would be the case when considering a different measure of error, whenever the error of the combined techniques can also be expressed as a weighted quasi-arithmetic mean of the values of this measure for all techniques. For instance, suppose we use as measure of error the standard deviation, , , the above cases become

when , then , the weighted root mean square or quadratic mean of , or weighted power mean with ().

when the weights are fixed, the optimal standard deviation is given by , the power mean of with ().

when the sampling proportions are fixed, the optimal standard deviation is given by , the arithmetic mean of ().

For more details see [18].

Liabilities vs. Utilities

From the above example, we can study the relationship between utilities, which we try to maximize, and liabilities, that we try to minimize. As a liability can always be considered as the inverse of a utility, we can see from Theorem 1 that establishing invariance properties on the order between means of utilities is equivalent to doing this with liabilities, except that the order is inverted. We can define for instance the efficiency as the inverse of the variance, and obtain it as a weighted mean of individual technique’s efficiencies, . Consider, for instance, the first case in the example above, where . We have where denotes the weighted harmonic mean.

Considering now the second case or weighted arithmetic mean. For the third case, which is the weighted power mean with .

8. Conclusions and Future Work

We have presented in this paper the relationship between stochastic orders and quasi-arithmetic means. We have proved several ordering invariance theorems, that show that given two distributions under a certain stochastic order, the ordering of the means is preserved for any quasi-arithmetic mean we might consider, this is, not only for the arithmetic mean (or expected value). We have shown how the results apply to first order, second order, likelihood ratio, hazard-rate, and increasing convex stochastic orders, and its application to cross-entropy. We have also presented an application example based on the linear combination of Monte Carlo estimators, and shown that the invariance allows costs or liabilities to be considered as the symmetric case of utilities.

In the future, we want to generalize our results to spatial weight matrices [31]. The rows in a spatial weight matrix are weights that give the influence of n entities over each other. Different weighted means such as arithmetic, harmonic, or geometric [32] can be used to compute this influence. We can thus apply our invariance results to each row. We will also investigate which of our results for Shannon entropy extend to Tsallis entropy too. Both Shannon and Tsallis entropy are weighted arithmetic means [23], and given monotonic, both and are monotonic too. Finally, we will investigate the invariance of the different stochastic orders under the operations of compositional data [3].

Table A1

Different possible combinations where the concavity/convexity of can be predicted for and increasing. ICX: convex and increasing, ICV: concave and increasing, DCX: convex and decreasing, DCV: concave and decreasing.

	g	g^−1	g^🟉	f^🟉	h=g^🟉∘f^🟉	f=g^−1∘h
1	ICX	ICV	ICV	ICV	ICV	ICV
2	DCX	DCX	DCV	ICX	DCV	ICX
3	ICV	ICX	ICX	ICX	ICX	ICX
4	DCV	DCV	DCX	ICV	DCX	ICV

Table A2

Different possible combinations where the concavity/convexity of can be predicted and is increasing when is increasing and decreasing (or viceversa) when is increasing. ICX: convex and increasing, ICV: concave and increasing, DCX: convex and decreasing, DCV: concave and decreasing.

	g	g^−1	g^🟉	f^🟉	f^🟉(m)	h=g^🟉∘f^🟉∘m	f=g^−1∘h
5	ICX	ICV	DCV	ICX	DCX	ICV	ICV
6	DCX	DCX	ICV	ICV	DCV	DCV	ICX
7	ICV	ICX	DCX	ICV	DCV	ICX	ICX
8	DCV	DCV	ICX	ICX	DCX	DCX	ICV