Exact Partial Information Decompositions for Gaussian Systems Based on Dependency Constraints.

The Partial Information Decomposition, introduced by Williams P. L. et al. (2010), provides a theoretical framework to characterize and quantify the structure of multivariate information sharing. A new method ( I dep ) has recently been proposed by James R. G. et al. (2017) for computing a two-predictor partial information decomposition over discrete spaces. A lattice of maximum entropy probability models is constructed based on marginal dependency constraints, and the unique information that a particular predictor has about the target is defined as the minimum increase in joint predictor-target mutual information when that particular predictor-target marginal dependency is constrained. Here, we apply the I dep approach to Gaussian systems, for which the marginally constrained maximum entropy models are Gaussian graphical models. Closed form solutions for the I dep PID are derived for both univariate and multivariate Gaussian systems. Numerical and graphical illustrations are provided, together with practical and theoretical comparisons of the I dep PID with the minimum mutual information partial information decomposition ( I mmi ), which was discussed by Barrett A. B. (2015). The results obtained using I dep appear to be more intuitive than those given with other methods, such as I mmi , in which the redundant and unique information components are constrained to depend only on the predictor-target marginal distributions. In particular, it is proved that the I mmi method generally produces larger estimates of redundancy and synergy than does the I dep method. In discussion of the practical examples, the PIDs are complemented by the use of tests of deviance for the comparison of Gaussian graphical models.

1. Introduction

The Partial Information Decomposition (PID) [1] provides a theoretical framework to characterize and quantify the structure of multivariate information sharing. That is, given a target variable Y, and a number of predictor variables the PID attempts to describe the mutual information between the target and predictors in terms of that which is unique to each predictor, as well as that which is shared (redundant) or synergistic between subsets of predictors. However, while the PID framework provides a theoretical structure for this sharing, practical applications require measures to quantify the different terms. Although a number of different candidate measures have been proposed, this remains an open area of research [2,3,4,5,6,7,8].

In James et al. [2] recently proposed a measure based on dependency constraints, denoted , which quantifies the unique information conveyed by a single predictor. In the case of two predictors, this is sufficient to obtain all four terms of the full PID; for higher order systems some terms remain indeterminate. For larger systems, there are a number of noted concerns with the PID approach. For three predictors, it has been shown that the proposed axioms and lattice cannot result in a non-negative decomposition. A specific counter example has been demonstrated [9], and an alternative view based on an intuitive interpretation of the relationship between PID and secret sharing schemes also demonstrated the same issue [10]. Despite this, systems with two predictors can still be of theoretical and practical interest, so we focus here on that specific case [2,5,11].

The measure was derived and presented for discrete systems [2]. However, there are many applications in which continuous variables might be subjected to the same analysis, and the PID approach has been considered for Gaussian systems [6,12]. is derived from considering dependency constraints imposed within a lattice of maximum entropy probability models. Here, we apply the same logic to derive in the case of continuous Gaussian variables. In this case, the maximum entropy probability models are Gaussian graphical models [13,14,15], also termed covariance selection models [16]. We provide closed form expressions for the two predictor PID, for both univariate and multivariate continuous Gaussian predictors and target. Code implementing these measures is provided as the Supplementary Materials.

First, we provide a brief review of the PID (Section 1.1) and the discrete measure (Section 1.2). In Section 2, we derive for univariate Gaussian variables, and in Section 3 extend to multivariate Gaussian variables.

1.1. The Partial Information Decomposition

The partial information decomposition was introduced in [1] as a method to decompose mutual information in a multivariate system in terms of redundancies and synergies within and between subsets of predictors. Formally, the PID is developed as the Mobiüs inversion of a shared information measure over the lattice of antichains of predictor variables. We refer the reader to [1] for the full details.

In this manuscript, we focus on the case of two predictors, , and a target Y. In this case, the mutual information is decomposed into four terms:

red, the information about Y that is shared, common or redundant between and ,

unq0, the information about Y that is available only from ,

unq1, the information about Y that is available only from ,

syn, the information about Y that is only available when and are observed together.

These terms satisfy the following intuitive relationships:

Given the existence of these three constraints in terms of classical mutual information values, there is only one degree of freedom left to specify the bivariate PID. With any of the four terms quantified, the remaining three can be easily calculated. The initial formulation of [1] was based on quantifying redundancy, and deriving the other quantities, but others have focussed on quantifying unique information or synergy directly.

1.1.1. The Partial Information Decomposition for Gaussian Variables

The original definition of the PID and most of the subsequent work referenced above focussed on discrete variables. However, there are many applications where continuous-valued Gaussian variables are interesting subjects for information theoretic analysis. For example, simplified model systems [17,18] or empirical data analysis [19,20]. In [12], all discrete PID measures available at the time were considered and their principles applied to multivariate Gaussian systems, where one univariate component of the Gaussian is denoted the target. It was shown [12] that for a univariate target, if red, unq0 and unq1 depend only on the predictor-target marginal , distributions, then there is a unique non-negative PID for which the redundancy is given by the minimum mutual information (MMI). Several proposed discrete PID measures fall into this class [1,3,4,5,21], so for Gaussian systems these approaches are all equivalent and equal to the MMI PID. The full bivariate MMI PID is defined as follows:

The MMI PID takes the redundancy component to be the minimum of the two mutual informations between the target and the predictors. Hence, one of the unique information components will always be zero. The MMI PID has been used also with Gaussian systems involving multivariate time series [12,22]; for an alternative approach, see [23].

Another recently proposed measure, , exploits the additivity of local or pointwise entropy to calculate the common change in surprisal provided by multiple predictors. By considering the signs of the local predictor-target information values, and the sign of the set theoretic intersection provided by local co-information, it is possible to sum up only pointwise terms that unambiguously correspond to redundant or overlapping information. is calculated on the maximum entropy distribution subject to pairwise marginal constraints (i.e., including the distribution). It therefore does not satisfy the Barrett conditions, and is not equivalent to the MMI PID. It also does not provide a non-negative PID, even in the two predictor case, since it is possible for one predictor to provide a unique negative contribution at the pointwise level (since pointwise mutual information is not non-negative). While there is no closed form expression for for Gaussians, it has been implemented using Monte Carlo methods [6]. [2] is also not invariant to changes in the predictor-predictor marginal distribution, and therefore does not reduce to MMI in the Gaussian case either.

1.2. Unique Information via Dependency Constraints

In [2] a method is proposed to quantify the unique information conveyed by a predictor variable. They start from a lattice of maximum entropy models subject to marginal constraints, where the lattice structure comes from the hierarchy of marginal constraints. This lattice is illustrated in Figure 1.

Figure 1

A dependency lattice of models (based on [2]). Edges coloured green (b, d, i, k) correspond to adding the constraint to the model immediately below. Edges coloured red (c, f, h, j) correspond to adding the constraint to the model immediately below.

For example, represents the maximum entropy distribution, having probability density function (p.d.f.) , under the constraints that the univariate marginals match exactly the univariate marginals of the original distribution, which has p.d.f. . That is: , , . represents the maximum entropy distribution subject to the constraints . represents the maximum entropy distribution subject to the constraints , and so on. The lattice structure arises from the higher order constraints enforcing corresponding lower order constraints, so that for example imposing a bivariate marginal constraint such as means also that the lower order constraints and also hold. Note that in [2] there is an additional model for the full distribution including third order interactions. We focus here on Gaussian systems which are fully determined by their first and second order moments, and so do not feature any triple-wise interactions. Therefore, model does not appear in our lattice for Gaussian systems.

For the sake of brevity in the sequel, rather than speaking of imposing a constraint of the form , for example, we will speak of ‘adding the constraint ’.

The colored edges correspond to adding a pairwise marginal constraint. Blue edges represent the constraint , i.e., preserving the pairwise dependency between and . Green and red labelled edges correspond to the addition of the and the dependencies respectively. For each model we calculate the mutual information between predictors and target under that model, . The unique information in is then obtained as the minimum change in along all the green edges due to the addition of the constraint to the model below. Similarly, the unique information in can be obtained as the minimum change in along all the red edges due to the addition of the constraint to the model below. Therefore, for example, the edge value d is equal to and the edge value f is equal to

If the edge labels in Figure 1 represent the change in mutual information along that edge, then the PID is given by: or:

It is shown in [2] that this approach is consistent; the same PID results from either of the two forms above. They also show that the resulting PID satisfies the core axioms of symmetry, self-redundancy, monotonicity, local positivity and the identity axiom [2].

2. An PID for Univariate Gaussian Predictors and Target

Since we will find in Section 2.2 that the required maximum entropy distributions that are described in Section 1.2 are Gaussian graphical models we begin with a brief discussion of such models.

2.1. Gaussian Graphical Models

The independence graph for a probability distribution on three univariate random variables, has three vertices and three possible edges, as described in Table 1. Let .

Table 1

Graphical models and independences for the probability distribution of . The vertices for random variables, are denoted by , respectively. Edges are denoted by pairs of vertices, such as . In the column of independences, for example, indicates that and Y are conditionally independent given (based on [13], p. 61).

Model	Independences	Edge Set	Diagram	Description
G_1	1⊥⊥2|0,0⊥⊥2|1	{}		Mutual independence
	1⊥⊥2|0
G_2	2⊥⊥0|1,2⊥⊥1|0	{(0,1)}		Independent subsets
G_3	1⊥⊥0|2,1⊥⊥2|0	{(0,2)}		Independent subsets
G_4	0⊥⊥1|2,0⊥⊥2|1	{(1,2)}		Independent subsets
G_5	1⊥⊥2|0	{(0,1),(0,2)}		One independence
G_6	0⊥⊥2|1	{(0,1),(1,2)}		One independence
G_7	0⊥⊥1|2	{(0,2),(1,2)}		One independence
G_8	None	{(0,1),(0,2),(1,2)}		Complete interdependence

Graphical models represent the conditional independences present in a probability distribution, as described in Table 1.

Suppose that has a multivariate Gaussian distribution with mean vector , positive definite covariance matrix and p.d.f. . There is no loss of generality in assuming that each component of has mean zero and variance equal to 1 [12]. If we let the covariance (correlation) between and be p, between and Y be q and between and Y be r, then the covariance (correlation) matrix for is and we require that are each less than 1, and to ensure positive definiteness we require also that

Conditional independences are specified by setting certain off-diagonal entries to zero in the inverse covariance matrix, or concentration matrix, ([13], p. 164). Given our assumptions about the covariance matrix of Z, this concentration matrix is where

We now illustrate using these Gaussian graphical models how conditional independence constraints also impose constraints on marginal distributions of the type required, and we use the Gaussian graphical models and to do so.

Since is multivariate Gaussian and has a zero mean vector, the distribution of is specified via its covariance matrix . Hence, fitting any of the Gaussian graphical models involves estimating the relevant covariance matrix by taking the conditional independence constraints into account. Let and be the covariance and concentration matrices of the fitted model .

We begin with the saturated model which has a fully connected graph and no constraints of conditional independence. Therefore, there is no need to set any entries of the concentration matrix K to zero, and so = . That is: model is equal to the given model for .

Now consider model . In this model there is no edge between and Y and so and Y are conditionally independent given . This conditional independence is enforced by ensuring that the [1, 3] and [3, 1] entries in are zero. The other elements in remain to be determined. Therefore has the form

Given the form of , has the form where is to be determined. Notice that only the [1, 3] and [3, 1] entries in have been changed from the given covariance matrix , since the [1, 3] and [3, 1] entries of have been set to zero. An exact solution is possible. The inverse of is

Since the [1, 3] entry in must be zero, we obtain the solution that and so the estimated covariance matrix for model is

The estimated covariance matrices for the other models can be obtained exactly using a similar argument.

Model contains the marginal distributions of , , Y, and . It is important to note that these marginal distributions are exactly the same as in the given multivariate Gaussian distribution for , which has covariance matrix . To see this we use a standard result on the marginal distribution of a sub-vector of a multivariate Gaussian distribution [24], p. 63.

The covariance matrix of the marginal distribution is equal to the upper-left 2 by 2 sub-matrix of , which is also equal to the same sub-matrix in in (16). This means that this marginal distribution in model is equal to the corresponding marginal distribution in the distribution of Z. The covariance matrix of the marginal distribution is equal to the lower-right 2 by 2 sub-matrix of , which is also equal to the same sub-matrix in in (16), and so the marginal distribution in model matches the corresponding marginal distribution in the distribution of Z. Using similar arguments, such equality is also true for the other marginal distributions in model .

Looking at (17), we see that setting to [1, 3] of K entry to zero gives . Therefore, simply imposing this conditional independence constraint also gives the required estimated covariance matrix .

It is generally true ([13], p.176) that applying the conditional independence constraints is sufficient and it also leads to the marginal distributions in the fitted model being exactly the same as the corresponding marginal distributions in the given distribution of . For example, in (19) we see that the only elements in that are altered are the [1, 3] and [3, 1] entries and these entries corresponds exactly to the zero [1, 3] and [3, 1] entries in . That is: the location of zeroes in determines which entries in will be changed; the remaining entries of are unaltered and therefore this fixes the required marginal distributions. Therefore, in Section 2.2, we will determine the required maximum entropy solutions by simply applying the necessary conditional independence constraints together with the other required constraints.

We may express the combination of the constraints on marginal distributions and the constraints imposed by conditional independences as follows [16]. For model , the th entry of is given by where is the edge set for model (see Table 1). For model , the conditional independences are imposed by setting the th entry of to zero whenever

Before moving on to derive the maximum entropy distributions, we consider the conditional independence constraints in model . In model we see from Table 1 that this model has no edge between and and none between and Y. Hence, and are conditionally independent given Y and also and Y are conditionally independent given . Hence, in K in (17) we set the [1, 2] and [2, 3] (and the [2, 1] and [3, 2]) entries to zero to enforce these conditional independences. That is: and . Taken together these equations give that and , and so the estimated covariance matrix for model is

We also note that model in Figure 1 also possesses the same conditional independences as . This is true for all of the maximum entropy models , and so when finding the nature of these models in the next section we apply in each case the conditional independence constraints satisfied by the graphical model .

2.2. Maximum Entropy Distributions

We are given the distribution of Z which is multivariate Gaussian with zero mean vector and covariance matrix in (16), and has p.d.f. For each of the models , we will determine the p.d.f. of the maximum entropy solution subject to the constraints and the separate constraint for model as well as the conditional independence constraints given in Table 2.

Table 2

Conditional independence constraints satisfied by the Gaussian graphical models that are applied when determining the maximum entropy models .

U_1:p=qr,q=pr,r=pq
U_2:q=pr,r=pq	U_3:p=qr,r=pq	U_4:p=qr,q=pr
U_5:r=pq	U_6:q=pr	U_7:p=qr

We begin with model . As shown in the previous section, the estimated covariance matrix for model , , is equal to the covariance matrix of , . By a well-known result [25], the solution is that is multivariate Gaussian with mean vector zero and covariance matrix, . That is: is equal to the given distribution of .

For model , the conditional independence constraint is and so

Hence, using a similar argument to that for , the maximum entropy solution for model is multivariate Gaussian with zero mean vector and covariance matrix , and so is equal to the model .

In model , the conditional independence constraints are and so and .

Therefore, and the maximum entropy solution for is multivariate Gaussian with zero mean vector and covariance matrix , and so is equal to . The derivations for the other maximum entropy models are similar, and we state the results in Proposition 1.

The distributions of maximum entropy, , subject to the constraints ( and the conditional independence constraints in Table 2, are trivariate Gaussian graphical models having mean vector and with the covariance matrices given above in Table 3.

Table 3

Covariance matrices, with corresponding concentration matrices, for the Gaussian graphical models which were derived as maximum entropy probability models in Proposition 1.

Model	Σ^_i	K_i^
U_1:X_0,X_1,Y	100010001	100010001
U_2:X_0X_1,Y	1p0p10001	11−p^21−p0−p10001−p^2
U_3:X_0Y,X_1	10q010q01	11−q^210−q01−q^20−q01
U_4:X_1Y,X_0	10001r0r1	11−r^21−r^20001−r0−r1
U_5:X_0X_1,X_0Y	1pqp1pqqpq1	1(1−p^2)(1−q^2)1−p^2q^2(q^2−1)p(p^2−1)q(q^2−1)p1−q^20(p^2−1)q01−p^2
U_6:X_0X_1,X_1Y	1pprp1rprr1	1(1−p^2)(1−r^2)1−r^2(r^2−1)p0(r^2−1)p1−p^2r^2(p^2−1)r0(p^2−1)r1−p^2
U_7:X_0Y,X_1Y	1qrqqr1rqr1	1(1−q^2)(1−r^2)1−r^20(r^2−1)q01−q^2(q^2−1)r(r^2−1)q(q^2−1)r1−q^2r^2
U_8:X_0X_1,X_0Y,X_1Y	1pqp1rqr1	1|Σ_Z|1−r^2qr−ppr−qqr−p1−q^2pq−rpr−qpq−r1−p^2

The estimated covariance matrices in Table 3 were inverted to give the corresponding concentration matrices, which are also given in Table 3. They indicate by the location of the zeroes that the conditional independences have been appropriately applied in the derivation of the results in Proposition 1.

It is important to check that the relevant bivariate and univariate marginal distributions are the same in all of the models in which a particular constraint has been added For example, the constraint is present in models The marginal bivariate distribution has zero mean vector and so is determined by the upper-left 2 by 2 sub-matrix of the estimated covariance matrices, ([24], p. 63). Inspection of Table 3 shows that this sub-matrix is equal to in all four models. Thus, the bivariate distribution of is the same in all four models in which this dependency constraint is fitted. It is also the same as in the original distribution, which has covariance matrix in (16). Further examination of Table 3 shows equivalent results for the and bivariate marginal distributions. The univariate term Y is present in all eight models. The univariate distribution of Y has mean zero and so is determined by the [3, 3] element of the estimated covariance matrices ([24], p. 63). Looking at the column, we see that the variance of Y is equal to 1 in all eight models, and so the marginal distribution of Y is the same in all eight models. In particular, this is true in the original distribution, which has covariance matrix in (16).

2.3. Mutual Information

Some required results involving mutual information will now be stated. They will be used to find expressions for the total mutual information of each model and also in constructing the and PIDs.

Application of (27) with the covariance matrices given in Table 3 gives the following expressions for the total mutual information for the maximum entropy models derived in Proposition 1.

2.4. The PID for Univariate Gaussian Predictors and Target

The PID for Gaussian predictors and target will now be constructed; for details, see (8)–(11) in Section 1.2.

Using the results in Table 4 together with the dependency lattice in Figure 2, we may write down expressions for all the required edge values, and they are given in Table 5.

Table 4

Expressions for the predictors-target mutual information for the eight models in the dependency lattice of Figure 2, as described in Table 3.

U_8:12log1−p^21−p^2−q^2−r^2+2pqr	U_4:I(X_1;Y)
U_7:12log1−q^2r^2(1−q^2)(1−r^2)	U_3:I(X_0;Y)
U_6:I(X_1;Y)=12log11−r^2	U_2:0
U_5:I(X_0;Y)=12log11−q^2	U_1:0

Figure 2

A dependency lattice of models (based on [2]). Edges coloured green (b, d, i, k) correspond to adding the term to the model immediately below. Edges coloured red (c, f, h, j) correspond to adding the term to the model immediately below. The two relevant sub-lattices are shown here.

Table 5

Expression for the edge values in the dependency lattice in Figure 2 that are used to determine the unique informations.

b=I(X_0;Y)=12log11−q^2	c=I(X_1;Y)=12log11−r^2
d=I(X_0;Y)	f=I(X_1;Y)
i=12log1−q^2r^2(1−q^2)(1−r^2)−I(X_1;Y)	h=12log1−q^2r^2(1−q^2)(1−r^2)−I(X_0;Y)
k=12log1−p^21−p^2−q^2−r^2+2pqr−I(X_1;Y)	j=12log1−p^21−p^2−q^2−r^2+2pqr−I(X_0;Y)

By making use of the edge values given in Table 5 together with (8)–(11) from Section 1.2 the PID can be constructed. We now state some results for the PID for univariate Gaussian predictors, , and target, Y, with proofs given in Appendix A.

For two univariate Gaussian predictors,

The PID possesses consistency as well as the core axioms of non-negativity, self-redundancy, monotonicity, symmetry and identity.

When unq0 is equal to b or d, the the redundancy component is zero.

When unq0 is equal to i, the redundancy and both unique informations are constant with respect to the correlation between the two predictors.

When the correlations between each predictor and the target are both non-zero, then unq0 is equal to either i or to k.

When unq0 is equal to k, the synergy component is zero.

The redundancy component in the PID is greater than or equal to the redundancy component in the PID with equality if, and only if, at least one of the following conditions holds: (i) either predictor and the target are independent; (ii) either predictor is conditionally independent of the target given the other predictor.

The synergy component in the PID is greater than or equal to the synergy component in the PID with equality if, and only if, at least one of the following conditions holds: (i) either predictor and the target are independent; (ii) either predictor is conditionally independent of the target given the other predictor.

The and PIDs are identical when either and Y are conditionally independent given or and Y are conditionally independent given , and in particular they are identical for models . In model the synergy component of is zero.

The PID is defined in (4)–(7) in Section 1.1. We now consider examples of the PID as well as comparisons between the and PIDs in the following subsections.

We consider the PID when

2.5. Some Examples

When , we see from Table 4 that and , so unq0 = 0, and since the redundancy component is also zero. The unique information, unq1, and the synergy component, syn, are equal to respectively. The PID is exactly the same as the PID.

We consider the PID when

When , we see from Table 5 that and also that because since It follows that , and that the synergy component is equal to

Since , from (29), the redundancy component is zero, as is unq1. The PID is exactly the same as the PID.

We consider the PID when

Under the stated conditions, it is easy to show that and and so the minimum edge value is attained at i. Using the results in Table 5 and (29)–(31), we may write down the PID as follows.

For this situation, the PID takes two different forms, depending on whether or not . Neither form is the same as the PID.

Compare the and PIDs when , and .

The PIDs are given in the following table.

There is a stark contrast between the two PIDs in this system. Since , the PID has two zero unique informations, whereas has equal values for the uniques but they are quite large. The PID gives much larger values for the redundancy and synergy components than does the PID. In order to explore the differences between these PIDs, 50 random samples were generated from a multivariate normal distribution having correlations The sample estimates of were and the sample PIDs are

We now apply tests of deviance in order to test model within the saturated model . The null hypothesis being tested is that model is true (see Appendix E). The results of applying tests of deviance ([13], p. 185), in which each of models is tested against the saturated model , produced approximate p values that were close to zero () for all but model which had a p value of . This suggests that none of the models provides an adequate fit to the data and so model provides the best description. The results of testing and within model gave strong evidence to suggest that the interaction terms and are required to describe the data, and that each term makes a significant contribution in addition to the presence of the other term. Therefore, one would expect to find fairly sizeable unique components in a PID, and so the PID seems to provide a more sensible answer in this example. One would also expect synergy to be present, and both PIDs have a large, positive synergy component.

Prediction of grip strength

Some data concerning the prediction of grip strength from physical measurements was collected from 84 male students at Glasgow University. Let Y be the grip strength, be the bicep circumference and the forearm circumference. The following correlations between each pair of variables were calculated: and PIDs applied with the following results.

The and PIDs are very similar, and the curious fact that unq0 in is equal to the synergy in is no accident, It is easy to show this connection theoretically by examining the results in (30)–(31) and Table 5; that is, the sum of unq0 and syn in the PID or the sum of unq1 and syn in the PID is equal to the synergy value in the PID. This happens because the PID must have a zero unique component.

These PIDs indicate that there is almost no synergy among the three variables, which makes sense because the value of is close to zero, and this suggests that and Y are conditionally independent given . On the other hand, is 0.1427 which suggests that and Y are not conditionally independent given , and so both terms and are of relevance in explaining the data, which is the case in model This model has and therefore no synergy and also a zero unique value in relation to . The results of applying tests of deviance ([13], p. 185), in which each of models is tested within the saturated model , show that the approximate p values are close to zero () for all models except and . The p value for the model is , while the p value for the test of against is approximately 0.45. Thus, there is strong evidence to reject all the models except model and this suggests that model provides a good fit to data, and this alternative viewpoint provides support for the form of both PIDs.

2.6. Graphical Illustrations

We present some graphical illustrations of the PID and compare it to the PID; see Section 1.1.1 and Section 1.2 for definitions of these PIDs.

Since , both the unique informations are zero in Figure 3a. The redundancy component is constant, while the synergy component decreases towards zero. In Figure 3b, we observe change-point behaviour of when . For the unique components of are equal, constant and positive. The redundancy component is also constant and positive with a lower value than the corresponding component in the PID. The synergy component decreases towards zero and reaches this value when . The synergy is lower that the corresponding synergy for all values of p.

Figure 3

The & PID components are plotted for a range of values of the correlation (p) between the two predictors. Two combinations of the correlations () between each predictor and the target are displayed. The total mutual information is also shown as a dashed black curve.

At , the synergy “switches off” in the PID, and stays “off” for larger values of p, and then the unique and redundancy components are free to change. In the range the redundancy increases and takes up all the mutual information when , while the unique informations decrease towards zero. The and profiles show different features in this case. The “regime switching” in the PID is interesting. As mentioned in Proposition 2, the minimum edge value occurs with unq0 = i or k. When unq0 = k the synergy must be equal to zero, whereas when unq0 = i the synergy is positive and the values of the unique informations and the redundancy are constant. Regions of zero synergy in the PID are explored in Figure 5.

In Figure 4a,b, there are clear differences in the PID profiles between the two methods. The synergy component switches off at and is zero thereafter. For , both the uniques are much larger than those of , which are zero, and has a larger redundancy component. For , the redundancy component in increases to take up all of the mutual information, while the unique information components decrease towards zero. In contrast to this, in the PID the redundancy and unique components remain at their constant values while the synergy continues to decrease towards zero.

Figure 4

In (a,b), the & PIDs are plotted for a range of values of the correlation (p) between the two predictors. One combination of the correlations () between each predictor and the target are displayed. In (c,d), the & PID are plotted for a range of allowable values of q, where q is equal to r, for . The total mutual information is also shown as a dashed black curve.

The PIDs are plotted for increasing values of in Figure 4c,d when . The and profiles are quite different. As q increases, the uniques remain at zero, while the uniques rise gradually. Both the redundancy and synergy profiles rise more quickly than their counterparts, probably because both their uniques are zero. In the PID, the synergy switches on at and it is noticeable than all the components can change simultaneously as q increases.

One of the characteristics noticed in Figure 3 and Figure 4 is the ’switching behaviour’ of the PID in that there are kinks in the plots of the PIDs against the correlation between the predictors, p: the synergy component abruptly becomes equal to zero at certain values of p, and there are other values of p at which the synergy moves from being zero to being positive.

In Proposition 2, it is explained for the PID that when both predictor-target correlations are non-zero the minimum edge value occurs at edge value i or k. When the synergy moves from zero to a positive value, this means that the minimum edge value has changed from being k to being equal to i, and vice-versa. For a given value of p, one can explore the regions in space at which such transitions take place. In Figure 5, this region of zero synergy is shown, given four different values of p. The boundary of each of the regions is where the synergy component changes from positive synergy to zero synergy, or vice-versa.

Figure 5

Regions in space in which the synergy component in the PID is equal to zero, plotted for four different values of p. Also, the determinant of is positive.

The plots in Figure 5 show that synergy is non-zero (positive) whenever q and r are of opposite sign. When the predictor-predictor correlation, p, is 0.05 there is also positive synergy for large regions, defined by , when q and r have the same sign. As p increases the regions of zero synergy change shape, initially increasing in area and then declining as p becomes quite large (). As p is increased further the zero-synergy bands narrow and so zero synergy will only be found when q and r are close to being equal.

When p is negative, the corresponding plots are identical to those with positive p but rotated counter clockwise by about the point Hence, synergy is present when q and r have the same sign. When q and r have opposite signs, there is also positive synergy for regions defined by

The case of is of interest and there are no non-zero admissible values of q and r (where the covariance matrix is positive definite) where the synergy is equal to zero. Hence the system will have synergy in this case unless or . This can be seen from the synergy expression in Example 3.

3. Multivariate Continuous Predictors and Target

We now extend the results developed in Section 2 and consider the case where the three continuous variables become random vectors , of dimensions respectively, with mean vectors equal to a zero vector of lengths , respectively, and covariance matrices equal to an identity matrix of the respective sizes The fact that there is no loss of generality in making these assumptions will be explained in Section 3.4. We stack these random vectors into the random vector , where has dimension and assume that has a multivariate Gaussian distribution with p.d.f. mean vector and covariance matrix given by where the matrices are of size , respectively, and are the cross-covariance (correlation) matrices between the three pairings of the three vectors and so defined on , where

3.1. Properties of the Matrices P, Q, R, and the Inverse Matrix of

We require some matrix results, which will be proved in Appendix B.

Suppose that a symmetric matrix M is partitioned as where the diagonal blocks

When the covariance matrix (32) is positive definite then the following matrices are also positive definite, and hence nonsingular:

Also, the determinant of each of these matrices is positive and bounded above by unity, and it is equal to unity if, and only if, the matrix involved is the zero matrix. Furthermore,

With these results in place, we now present the inverse of , which is equal to the concentration matrix K. It was determined by solving simultaneous equations for block matrices and we omit the details. It is where

The various inverses used in (35)–(40) are valid for the following reasons. The matrix is positive definite, and so its inverse is also positive definite. By Lemma 1, are positive definite and so invertible, which means in turn that their inverses are invertible. From Lemma 2, we have that the matrices are invertible. Therefore the sub-matrices in the inverse of in (35)–(40) are well-defined.

3.2. Block Gaussian Graphical Models

As in Section 2.1 , we will consider graphical models to express the conditional independences in the probability distribution for , although each graph will still have three vertices, with each vertex representing one of the random vectors, Each graph can be thought of as a block independence graph. This means that only dependences between pairs of vectors will be represented, while there will be no dependences among the variables within each of the three random vectors, since they are mutually independent. The models which express conditional dependences have the same format as in Table 1 in Section 2.1 and we use the same notation again here, the only difference being to express in a bold font. We term these models ’block graphical models’ since we are treating each random vector as the block containing a number of mutually independent random variables. Here is an illustration of such a model:

The model in Figure 6 is the block version of model from Table 1 and denoted as A product term, such as , encapsulates correlations between each random variable in and each random variable in . For example, in Figure 6 there are 12 correlations between the elements of and , and 6 correlations between the elements of and . Using the block notation provides some simplicity, for otherwise one would be required to write expressions such as for the set of constraints within each block interaction term. The block graphical models in the multivariate version of the dependency lattice are given in Figure 7. We now define the conditional independence constraints for the block versions of model in Table 1 and determine some of their estimated covariance matrices. The block version of model has no conditional independences. Hence, no block zeroes are imposed on the concentration matrix and the estimated covariance matrix for this model is , which means that model is equal to the given distribution of .

Figure 6

An illustration of a block graphical model for the random vectors , and . contains four random variables, while has three and has two. This model expresses the conditional independence of and given . In this model, the bivariate marginals and as well as lower-order marginals, are fixed.

Figure 7

A dependency lattice of block graphical models. Edges coloured green (b, d, i, k) correspond to adding the set of constraints within to the model immediately below. Edges coloured red (c, f, h, j) correspond to adding the set of constraints within to the model immediately below. The two relevant sub-lattices are shown here.

Consider the block version of model in Table 1. In , and are conditionally independent given and so we apply the constraint in the concentration matrix K in (34). From (35), this block constraint is

Given the results stated at the end of Section 3.1, we can pre-multiply by and post-multiply by in (41) to obtain the required block constraint as that is Hence, the estimated covariance matrix for block model is

Applying a similar argument to the expressions for V and W in (36) and (37), it can be shown that for the conditions for conditional independence between and given in model the block constraint is , while for the conditional independence between and given in model the block constraint is Hence, the estimated covariance matrices for block models and are

In block model both of the conditional independences defining block models and are present. Therefore, the conditional independence constraints are and . Combining them gives which may be written as . By Lemma 2, we may pre-multiply by the inverse of to obtain , which in turn implies that . Hence the conditional independence constraints for block model are and so the estimated covariance matrix is

The other estimated covariance matrices can be derived in a similar fashion. As in Section 2.1, it is the case that applying the conditional independence constraints also ensures that the required marginal distributions in each of the block graphical models are equal to the corresponding marginal distributions in the given distribution of . We also note, in particular, that model in Figure 7 has the same conditional independences as those present in block graphical model , and this is true for each of the maximum entropy models , and so when finding the form of these models in the next section we apply in each case the conditional independence constraints satisfied by the block graphical model .

3.3. Maximum Entropy Distributions

We are given the distribution of Z which is multivariate Gaussian with zero mean vector and covariance matrix in (32), and has p.d.f. For each of the models , we will determine the p.d.f. of the maximum entropy solution subject to the constraints and the separate constraint for model as well as the conditional independence constraints given in Table 6.

Table 6

Conditional independence constraints satisfied by the block Gaussian graphical models that are applied when determining the maximum entropy models .

M_1:P=QR^T,Q=PR,R=P^TQ
M_2:Q=PR,R=P^TQ	M_3:P=QR^T,R=P^TQ	M_4:P=QR^T,Q=PR
M_5:R=P^TQ	M_6:Q=PR	M_7:P=QR^T

For model , the estimated covariance matrix and so the maximum entropy distribution is equal to block graphical model , which is equal to the given distribution of . Similarly, the maximum entropy model is equal to the block graphical model , which is multivariate Gaussian with zero mean vector and covariance matrix defined in (42), and so on. Hence we can state our results in Proposition 3.

The distributions of maximum entropy, , subject to the constraints (45)–(46) and the conditional independence constraints in Table 6 are block Gaussian graphical models having mean vector and with the covariance matrices given below in Table 7.

Table 7

Covariance matrices for the Gaussian block graphical models in Proposition 3.

Model	Σ^_i	Model	Σ^_i
M_1:X_0,X_1,Y	I_n_0000I_n_1000I_n_2	M_5:X_0X_1,X_0Y	I_n_0PQP^TI_n_1P^TQQ^TQ^TPI_n_2
M_2:X_0X_1,Y	I_n_0P0P^TI_n_1000I_n_2	M_6:X_0X_1,X_1Y	I_n_0PPRP^TI_n_1RR^TP^TR^TI_n_2
M_3:X_0Y,X_1	I_n_00Q0I_n_10Q^T0I_n_2	M_7:X_0Y,X_1Y	I_n_0QR^TQRQ^TI_n_1RQ^TR^TI_n_2
M_4:X_1Y,X_0	I_n_0000I_n_1R0R^TI_n_2	M_8:X_0X_1,X_0Y,X_1Y	I_n_0PQP^TI_n_1RQ^TR^TI_n_2

We can now check by inspecting the entries in Table 7 that particular marginal distributions involving two blocks, such as , are the same in all of the models and also equal to the marginal distribution in the given distribution. For example, the block interaction term is present in models . The distribution of is multivariate normal with mean vector equal to a zero vector and covariance matrix given by the bottom right 2 by 2 block matrix in ([24], p. 63). For each of these four models, we can see by inspection that this covariance matrix is and so the ( marginal distribution is the same in all of these four models in which this particular block interaction term has been fitted. Since is equal to the given distribution of , it follows that this marginal distribution is the same as in the given distribution. Similar checks can be made regarding the other marginal distributions involving two blocks to find that a similar conclusion applies also to them. We can also check that the single-block terms, such as have the same distribution. This term has been fitted in all eight models. Since the mean vector of is a zero vector, its distribution is determined by its covariance matrix. This is given by the central [2, 2] sub-matrix in . Inspection of the fitted covariance matrices in Table 7 reveals that the relevant matrix is the same in all eight models. Hence this block marginal distribution is fixed in the eight maximum entropy distributions.

3.4. Mutual Information

It was claimed in Section 3 that there is no loss of generality in assuming that the mean vectors of are a zero vector and that their covariance matrices are an identity matrix, of the required sizes. We will now demonstrate this, by calculating the mutual information using the general form of covariance matrix (which is partitioned conformably to in (32)): where

Since (i = 0, 1, 2) is positive definite (by Lemma 1) it has a positive definite square root ([26], pp. 405–406). Therefore, using standard properties of determinants,

From [24], we can state the following marginal distributions. and are multivariate normal with covariance matrices respectively. The formula for the entropy of a multivariate Gaussian distribution is required. For a k-dimensional random variable following a Gaussian distribution with mean vector and covariance matrix , the entropy in the distribution of W is [25]

Using the formula for entropy in (52), and using a similar argument to that which produced (49), we may write

Therefore, applying a version of (27) with vector arguments, and using (32), the total mutual information is given by using the fact that (Lemma 2)

Therefore, we have demonstrated that the mutual information between the predictors and the target does not depend on the either the mean vectors or the covariance matrices of the individual random vectors , and . Using (52), the distributional results (50)–(51) and similar arguments to that leading to (49), we state formulae for the other required mutual informations.

Expressions for the total mutual information, , for models are provided in Table 8.

Table 8

Expressions for the predictors-target mutual information for the eight models in the dependency lattice in Figure 7, that are stated in Table 7.

M_8:12logI_n_1−P^TPΣ_Z	M_4:I(X_1;Y)
M_7:12logI_n_1−RQ^TQR^TI_n_2−Q^TQI_n_2−R^TR	M_3:I(X_0;Y)
M_6:I(X_1;Y)=12log1I_n_2−R^TR	M_2:0
M_5:I(X_0;Y)=12log1I_n_2−Q^TQ	M_1:0

3.5. The PID for Multivariate Gaussian Predictors and Targets

Using the expressions for the total mutual information between predictors and the target in Table 8, formulae for the edge values that are used in the construction of the PID, are given in Table 9. They are computed by subtracting the mutual informations of the relevant models in Figure 7; for example, k is computing by subtracting the mutual information in model from that in model .

Table 9

Expression for the edge values in the dependency lattice in Figure 7 that are used to determine the unique informations.

b=d=I(X_0;Y)=12log1I_n_2−Q^TQ	c=f=I(X_1;Y)=12log1I_n_2−R^TR
i=12logI_n_1−RQ^TQR^TI_n_2−Q^TQI_n_2−R^TR−I(X_1;Y)	h=12logI_n_1−RQ^TQR^TI_n_2−Q^TQI_n_2−R^TR−I(X_0;Y)
k=12logI_n_1−P^TPΣ_Z−I(X_1;Y)	j=12logI_n_1−P^TPΣ_Z−I(X_0;Y)

Given the edge values in Table 9, we can form the PID for multivariate Gaussian predictors and targets.

We now state some results for this PID. Proofs are given in Appendix C.

For two multivariate Gaussian predictors,

This PID possesses consistency as well as the core axioms of non-negativity, self-redundancy, monotonicity, symmetry and identity.

When unq0 is equal to b or d, the the redundancy component is zero.

When unq0 is equal to i, the redundancy and both unique informations are constant with respect to the correlation matrix P between the two predictors, .

When neither predictor and the target are independent, then unq0 is equal to either i or to k.

When unq0 is equal to k, the synergy component is zero.

The redundancy component in the PID is greater than or equal to the redundancy component in the PID with equality if, and only, if at least one of the following conditions holds: (i) either predictor and the target are independent; (ii) either predictor is conditionally independent of the target given the other predictor.

The synergy component in the PID is greater than or equal to the synergy component in the PID with equality if, and only, if at least one of the following conditions holds: (i) either predictor and the target are independent; (ii) either predictor is conditionally independent of the target given the other predictor.

The and PIDs are identical when either and are conditionally independent given or and are conditionally independent given , and in particular they are identical for models . In model the synergy component of is zero.

3.6. Examples and Illustrations

The multivariate version of the PID was implemented using the edge coefficients in Table 9 together with the PID rules in (61)–(62). The matrices, , were given an equi-correlation structure in which all the entries were equal within each matrix: where denote here the constant correlations with each matrix and denotes an n-dimensional vector whose entries are each equal to unity.

Taking , respectively, the covariance (correlation) matrix was computed and plots produced of the PIDs as displayed below. The covariance matrix is positive definite only for limited ranges of . The PID was computed using the formulae in Section 2.5, but replacing by their vector counterparts , respectively.

Figure 8 shows some plots of the multivariate and PIDs as a function of p, for particular values of q and r. These plots display similar characteristics to those shown in Figure 3, Section 2.6. Some further plots are displayed in Figure 9. This time the PIDs are shown for increasing values of , for two values of p. Again, these plots have similar characteristics to those considered in Figure 4, Section 2.6.

Figure 8

The and PID components are plotted for a range of values of the correlation (p) between the two predictors. Two combinations of the correlations () between each predictor and the target are displayed. The total mutual information is also shown as a dashed black curve.

Figure 9

The and PID components are plotted for a range of values of the correlation (q) between the predictor and the target . Two combinations of the correlations () between each predictor and the target are displayed. The total mutual information is also shown as a dashed black curve.

Values of were chosen, ensuring that the covariance matrix was positive definite, and the equi-correlation structure defined in (49) was used. The PID results are presented in the table above.

We see rather different compositions of PID components in the three examples as well as some differences between the two methods. For the first system, both methods have a very large value for the synergy component, with having larger values for the unique informations than but lower redundancy. The two methods produce fairly similar PIDs for the second system, although there are some differences of about 0.09 bit in all of the components. The third system has strong differences between the and PIDs. has large values for the two unique components along with small values for redundancy and synergy, whereas has large values for redundancy and synergy and very small values for the uniques.

In these examples, the dimensions of the predictors and target have an impact on the resulting PIDs as well as the correlations.

Prediction of calcium contents

The multivariate PID and the PID were applied using data ([27], p. 145) on 73 women involving one set of predictors (Age, Weight, Height), another set of two predictors (Diameter of os calcis, Diameter of radius and ulna), and target (Calcium content of heel and forearm). The following results were obtained.

Both PIDs indicate the presence of synergy and a large component of unique information due to the variables in . The PID shows more unique information but less redundancy and synergy than . To explore matters further, deviance tests ([13], p. 185) were performed which compared models against the saturated model . In all seven tests the p-values are very small indeed (), indicating that there is very strong evidence to reject models and that model provides the best explanation of the data. The test of against provided very strong evidence in favour including the term in model , and so there is little surprise that both PIDs have a large value for unq0. The test of model against model gave very strong evidence that should also be included in model . On this occasion, this has not led to a large value for unq1, so perhaps these two terms combine to produce synergy and redundancy.

The PIDs were also computed with the same and but taking to be another set of four predictors (Surface area, Strength of forearm, Strength of leg, Area of os calcis). The following results were obtained.

In this case, the and PIDs are very similar, with the main component being due to unique information due to the variables in . The PID indicates zero synergy and almost zero unique information due to the variables in Again, deviance tests were performed. In six of the seven tests the approximate p-values are very small indeed (less than ). The exception is model for which the deviance test has an approximate p-value of 0.98, indicating that model provides the simplest explanation for the data and an extremely good fit to the data. In model , it is expected that there will be zero synergy as well as a zero unique component due to the variables in , and this matches quite well the information produced in the PIDs.

When working with real or simulated data it is important to use the correct correlation matrix. In order to use the results given in Table 9 and (61)–(62) it is essential that the input covariance matrix has the structure of , as given in (32). The computational approach used here is described in Appendix D.

4. Discussion

We have applied the method to obtain bivariate partial information decompositions for Gaussian systems with both univariate and multivariate predictors and targets. We give closed form solutions for all PID terms in both these cases, to allow easy computation of a PID from a covariance or correlation matrix. The main properties enjoyed by for Gaussian systems are the same as those defined in [2]. The characteristics of the PIDs for Gaussian system have been illustrated by graphical exploration as well as numerical examples.

Given that the method employs a lattice of probability models, Gaussian graphical models, it seems natural when attempting to understand the form of a particular PID to consider formal statistical tests in order to determine which of the models in the lattice best fits the data. Therefore, deviance tests have been used for this purpose. They provide a useful complementary approach, as demonstrated in the examples considered.

There are now three approaches to the PID for Gaussian systems, [12], [6] and [2] as developed here. While they may agree in some cases, these methods are in general all distinct. For and the redundancy and unique information values are not invariant to the predictor-predictor marginal distribution (here p or P), and so they are not equivalent to . Here, we proved that redundancy and synergy measured with are never less than those measured with , and are equal in specific circumstances regarding marginal and conditional independence (see Propositions 2 and 4 (f, g)). We note that if the full system or data matches any one of the models (or ) then a conditional independence condition is met. This forces one unique component and the synergy component to be equal to zero, and and are identical. Therefore, in practice, if any of these models provide an acceptable fit to the data, then the and PIDs are likely to be quantitatively very similar. If models (or ) provide a better fit to the data, then all four components can be non-zero. By considering the perspective of multivariate linear regression based on the conditional distribution of given and , as in [21] for the univariate case, synergy is expected to be present in model , and and can diverge. This is also the case with model , although here the synergy component of is zero. A more thorough comparison of the behaviour of these different measures across families of Gaussian systems could help to illustrate their different interpretations and perhaps shed light on the different approaches to the PID in the discrete case.

As noted, while the PID has the property that the redundancy component does not depend on the correlation between the predictors this is not true in general for the PIDs. When there is positive synergy in the PIDs it is the case that redundancy and unique information are invariant to predictor-predictor dependence (p), but when synergy is zero this does not hold (see e.g., Figure 3). However, as shown, the redundancy and synergy terms are always less than or equal to the corresponding terms. From considering the arguments related to the best fitting models, it seems that in some cases the approach may overstate redundancy. Further, for it is by definition not possible for two predictors to both carry unique information. Considering the properties of Gaussian systems and simple noisy additive linear systems this seems unintuitive: if the predictors are independent or anti-correlated but with fixed correlation with the target it seems more natural that, across samples, they each provide a positive unique information contribution to an estimate of the target (see e.g., Example 4). Similarly, it also seems intuitive that in a Gaussian setting the amount of information shared between two predictors with fixed target correlation should increase as the correlation between the predictors increases (Figure 3d). Further, one would imagine, in general that it should be possible for two variables to carry the same amount of information, but for that information to be different.

Both of these considerations suggest that the dependence on the predictor-predictor marginals in both and seems to be more natural for Gaussian systems. The invariance to the predictor- predictor marginals was a foundational assumption in the derivation of the method presented in [5], and was based on a decision theoretic operationalization of unique information. However, a game theoretic extension of this approach in [6] suggests that this invariance is not a natural requirement for a measure of shared information. In addition, for Gaussian systems there are existing classical variance-based approaches to the problem, such as commonality analysis [28,29], based on semi-partial correlation, or path analysis [30], which could provide another perspective on the problem. Systematically comparing these methods is an interesting area for future work.

The PIDs presented provide a non-negative decomposition of a joint predictor-target mutual information for Gaussian systems. This could have broad applications, from an exploratory statistical tool, to analysis of complex systems and networks. Gaussian models or approximations have been used for computing information-theoretic statistics from experimental data in neuroscience and neuroimaging [19,20]. For example, in neuroimaging there are often statistical effects of a stimulus observed in multiple recorded responses (for example different brain regions, or different temporal offsets from stimulation). Methods such as the PID can provide a practical tool to relate two such modulations and so give insight into whether they are likely to reflect the same or different brain processes. Similarly, if multiple stimulus features or aspects are presented in an experiment, the PID can be applied to quantify how much of the neural response is commonly predicted from both stimulus features, uniquely available from each or synergistically available only from the combination.

The method of [2] is a very general one and it could be applied to systems other than discrete systems [2] or the Gaussian PIDs developed here. For example, the method could be used with other types of graphical model, such as mixed discrete-continuous systems [13,15] based on the CG model, and also in multivariate autoregressive modelling of time series data [12,31,32] using graphical models [33,34,35]. We look forward to engaging in further exploration of the potential of the method.