A Definition of Conditional Probability with Non-Stochastic Information.

The current definition of a conditional probability enables one to update probabilities only on the basis of stochastic information. This paper provides a definition for conditional probability with non-stochastic information. The definition is derived by a set of axioms, where the information is connected to the outcome of interest via a loss function. An illustration is presented.

1. Introduction

The theory of conditional probability is a well-established mathematical theory that provides a procedure to update probabilities by taking new information into account. The new work in this paper is motivated by the fact that such a procedure is available only if the information which is used to update the probability concerns stochastic events, that is, events to which a probability is assigned. In other words, such information needs to be already included in the probability model. The statistical implications are most pertinent to Bayesian inference where non-stochastic information, obtained by experts for example, is often employed.

To set the scene, denotes the set of possible outcomes of an experiment, which we assume is finite, is a probability mass function on , and I is an arbitrary piece of information about the unknown outcome of the experiment, i.e., information for which one would consider revising . We argue that there must be a formal way to derive the updated probability mass function, which we write as . For example, might represent a horse race and a bookmaker has set p. It starts to rain, so “it is raining”, and the bookmaker revises p to . One imagines that the bookmaker does so from experience and uses no formal rule to implement the change. This lack of a formal rule then questions the extent of the relevance of subjective probability if one can appeal to no formal mechanism for revising belief distributions with arbitrary, yet relevant, information. Recall that the formal Bayes conditional probability rule requires the specification of for all possible pieces of information I that one could receive–to us, at least, this is an impossible task.

It is impossible to set the probabilities a priori for all possible I, yet this is required for the validation of the mathematical implementation of the Bayes rule. However, setting a loss function once I has been received, is, we argue, a totally viable task. This would establish the loss, on some scale, e.g., monetary, if the elementary event occurs once I has been received. Our formal updating rule relies on the construction of such .

I is a piece of information for which it is possible to construct a loss function on that is valued into and is not identically infinite. So, is the loss that the experimenter assigns to the outcome in when holding the piece of information I. We look at an illustration involving such later on in the paper. Once I has been received, the probability of updating is P, where for every .

For example, if I is the information “event has occurred” and then our formal rule will coincide with the usual definition of conditional probability.

When the standard definition of conditional probability does not apply, e.g., I is not such a statement or B is not a subset of , for example, an alternative definition based on a mathematical decision theoretic framework can be used. When information received is non-stochastic, but relevant to an outcome of interest, we cannot use a probability distribution and so we need an alternative way to connect the information I with the outcome of interest . We do so using loss functions and a set of axioms.

This paper provides a definition for conditional probability on the basis of I. Reference [1] addressed this issue considering the minimization of a cumulative loss function which involved g-divergences. In this paper, instead, the definition of conditional probability is solely based on a set of axioms. This idea was developed by reference [2] who defined a framework for general Bayesian inference and discussed its application to important statistical problems. The present paper also addresses the issue of calibrating the conditional probability with non-stochastic information.

1.1. Relationship to the Literature

In the literature, alternative definitions of conditional probability, such as the Jeffrey’s Rule of conditioning, are given where new information is not put in terms of the occurrence of an event included in the model. These definitions rely on the assumption that information can be given in the form of a constraint on the probability. Constraints considered are of the type where g is a measurable real function on , and is the updated probability function. The idea is to minimize the Kullback–Leibler divergence subject to the constraint (2), which represents the information I. This problem can be solved using Lagrange multipliers.

For more detail about conditionalization based upon constraints on the conditional distribution, see references [3,4,5,6,7,8]. Our approach is different as we can deal with more types of information. On the other hand, our definition can encompass potentially arbitrary information about the outcome; all we need is to construct a loss function for each in once I has been received.

1.2. Motivation

The need for a definition of conditional probability outside of the usual set-up is the avoidance of paradoxes where one knows event B has occured yet is not a subset of . Paradoxes arise when B is not deemed to be part of the stochastic model yet is subsequently forced to be so.

Such difficulties arise in different puzzles, such as, for instance, Freund’s puzzle of the two aces, introduced by reference [9]. For other puzzles about conditional probability, see, for instance, reference Gardner [10].

These puzzles have been widely used to discuss the concept of conditional probability. According to reference [11], such a concept is justifiable only on the basis of “a set of rules that tell, at each step, what can happen next”, which he calls “protocol”. For conditioning on an event B, he asserts that “we are assuming B is in your probability model, i.e., in the field of events to which you assign probabilities. So it is implicit in the principle of total evidence that your probability model should include a model for what you learn”. On the other hand, Hutchison [12,13] emphasizes that the updating process needs to take into account the circumstances under which the truth of I was conveyed. Also, Bar–Hillel and Falk [14] claims that knowing how knowledge was obtained is “a crucial ingredient to select the appropriate model”. These authors present different views about the concept of conditionalization, but all agree on the fact that there would not be a problem if it was known how the information I had become available and therefore, one could build a model including I.

The concept of conditional probability distributions is certainly appropriate as a procedure to update probabilities on the basis of any new information that has already been included in the probability model. However, it can be difficult to construct a model that considers all possible relevant information that could become available in the future. Therefore, a problem arises when one obtains some new and possibly unexpected information and wants to use it to update a probability distribution. Indeed, it does not seem appropriate to assess the probability of something which has been already observed. Our basic assumption is that the information I can be connected to the outcome of interest via a loss function . In this way, it is possible to update the probability P, even if I is some new unexpected information that was not included in the probabilistic framework.

Clearly, we are dealing with information about outcomes of interest which take many forms. It is well-known that Bayesian inference can rely on expert opinions which often take the form of non-stochastic information. In particular, our framework allows coherent updates of probabilities involving such information, and hence, the practical relevance of our framework to Bayesian inference.

2. Results

This section reports the current definition of conditional probability and presents and motivates our definition for conditional probability with non-stochastic information.

2.1. The New Definition

If represents prior beliefs about , then we argue that a valid and coherent update of on the basis of I is the posterior, , where and is a positive constant.

Looking at this form, it seems that we are proposing the equivalent of a Bayes rule with

This is correct only if the piece of information I is an element of a known space of possible outcomes. Indeed, only in such a case would it be possible to assess a probability distribution in such a space.

In our proposal, a much less complex framework is considered, since it is required to just set the loss once I has been received. Our axioms which lead to (3) do not regard (4) as the probability.

We shall now go into the details of how some natural assumptions imply (3). The following axioms are considered:

The posterior form is given by for some function (). Since and are all that are available and set, it is clear that the update must solely depend on these functions. We are, hence, asking for the unique which provides the update for all , i.e., invariant. This is a reasonable requirement since how one updates should not depend on .

If the piece of information (I) is made of two independent pieces of information, and , in the sense that , for every in , then the posterior probability satisfies the following: This ensures we end up with as the same object whether we update with together or one after the other.

If for every and some , then where is the restriction of to B, is the information that B certainly occurs or has occured, and is p restricted and normalized to B, i.e., . In other words, outcomes with infinite loss are disregarded in the updating.

Lower evidence (larger loss) for a state should yield smaller posterior probabilities under the same prior. So, if for some , for and for , then

If , then . That is, if the observation provides no information about , since the loss function is a constant, then the posterior is the same as the prior.

As a consequence of these, we have the following theorem:

For for some .

In order to prove Theorem 1, the following Lemma is useful.

Let

If .

for any set ,for every ω in B. This means that whether we update the prior probability .

Combination of axioms 2 and 5 yield axiom 6. Indeed, one just needs to consider (5) the case where the loss function or is constant.

Combining conditions 3 and 5, one obtains that updating with the loss (1) corresponding to information yields the conditional probability that is obtained by restricting and normalizing the prior to B. Now, consider and let be any piece of information associated with a given loss function (l). The sum of the two corresponding losses is infinite on and coincides with l on B, which is the loss considered in condition 3. Therefore, based on condition 2, condition 7 is obtained. ☐

In the two points case, say , the prior probability () is determined by a real number (z) in the unit interval (i.e., ), and the loss takes only two values, say and (i.e., , . According to condition 6, we can replace the pair with . Therefore, the posterior probability is a function of and z, say . In order to proceed, it is convenient to think in terms of odds rather than probabilities. So, if , we can consider (i.e., ). We can do the same with the posterior (). The posterior odds, , are a function of the loss difference (l) and the prior odds (t), which we denote by .

To deal with the odds, (5) becomes where h is a replication of l with a possible different I. Moreover, with a constant loss, the posterior is equal to the prior (condition 5), i.e., for every .

At this stage, we consider a prior with three mass points, say . The prior is given by (i.e., and ), with , for . The loss can be zero at one point without loss of generality (condition 6), so it takes values into the set . Let us consider the updating rule, , for priors with just two mass points. We can use this to update the conditional probability of given , i.e., . To this aim, we update the prior with masses and considering just the loss values, , i.e., disregarding the point with its loss (). In other words, we aim to obtain the right-hand-side of (8) and apply condition 7 given by Lemma 1. denotes the odds corresponding to the conditional probability of given , that is . is updated on the basis of the loss values () and zero, i.e., with . Similarly, we define and . We update on the basis of and , i.e., with and with . Clearly, , and this factorization of conditional odds has to hold also after updating, i.e., where . Formally, this identity is a consequence of (8), i.e., updating the conditional probability is the same as conditioning the updated probability. Since (11) must hold for every , and for every , for every and . If , and say , recalling (10), for every and every real l. Letting , we find that for every and every . A combination of (9) and (12) yields for every which, in turn, implies by monotonicity that for some . Indeed, according to condition 4, is a monotone, not an increasing function of l. This implies that must be positive. Hence, for every and every . In this way, we are basically done with the two points case.

Let us now consider the general finite case. We want to update the prior with mass points at given by , where are non-negative and their sum is less than or equal to one, and it is convenient to set . In terms of odds, the prior is given by the vector (being , or equivalently, , ), where and . Here, it is convenient to set . Moreover, we consider an valued function which provides the vector of the updated odds as being and . Now the question is how we could recover from , where the latter gives the updating rule for the two points case. Recall the notation used for the conditional odds, i.e., , is the odds corresponding to the conditional probability of given for a distinct . We can see that . According to (8), this identity will also have to be satisfied by the updated odds. Since we update with , we must have which by (13) becomes:

, (14) becomes and the updated probability of is

In this way, we have shown how to extend our coherent updating rule from the two points case to the general finite case.  ☐

The proof implies the existence of , and this is no surprise since loss functions are only defined up to scalar factors. Hence, the setting of is a calibration issue and to set this, we need more information that is not associated with any specific information (I). So, is set to be the loss function that contains initial beliefs at the outset before any further information (I) is obtained, and this is defined to be on the same scale as any . We regard , where is the prior information used to set p.

The prior expected loss is then defined by

Now, entropy is also regarded as an expected loss based on the self-information loss function −. Hence, we also have expected loss/entropy which is measured as and so calibration is achieved by matching these expected losses, so that Finally, in this section, we note that it is quite straightfoward to extend the uniqueness argument to all countably infinite , which replaces the uniqueness argument for all . However, we need more work to extend separate uniqueness to the general .

2.2. An Illustration

The loss function l is chosen by the decision-maker on the basis of the available information. Such information sometimes happens to be stochastic, i.e., belonging to a set B of outcomes to which a probability is assigned. If this is the case, one should update the probability by means of the usual conditional probability. It is tantamount to use the loss function (1) in (3). If the available information is not stochastic, then one can resort to the approach described in the present paper to properly assess the loss function l. A simple and very concrete example is now presented. Consider a horse race in which six horses participate. In order to decide how to bet, one assesses the probability for each horse to win. demotes the probability that horse number j wins for . In this example, the elements of are the numbers corresponding to the horses participating in the race, of which one will win.

Before the race begins, it starts raining. Since conditions have changed, the probabilities need to be updated. It is problematic to pursue this aim by resorting to the current definition of conditional probability. In fact, this requires knowledge of the probability that it rains and that the horse number j wins. As an alternative, one could calculate the conditional probabilities of a win for each horse by applying Bayes’ theorem, which requires the probability that it rains given the win of horse j. However, it is raining and the race has not yet been run!

It is therefore appropriate to resort to the definition of a conditional probability distribution given in this paper. So, is the prior probabilty that horse j wins and is the probability that “it is raining”. be the loss function, i.e., the loss incurred to the bookmaker, that is, the governor of the probabilities, if horse j wins. To elaborate, such an could be the assessed loss if the bookmaker puts all the takings received or a fixed amount on horse j to win.

The bookmaker then, on the same scale, assesses for each j which adjusts the loss . Obviously, a higher score will be given to those horses whose ability to run is more affected by the rain. In this way, one can use the ideas for setting to get . The updated probability that the j-th horse wins is revised to for .

3. Discussion

We have established a framework in which we can update probabilities in the light of general, i.e., non-stochastic, information. Given that we cannot connect the information and the outcome of interest via a probability model, we do so through a loss function. Minimizing a cumulative loss function involving the information on one side and the probability distribution on the other yields the updated probability distribution. When the information is stochastic, we employ the self information loss function; the solution then reverts to the standard definition of conditional probability.

We believe the framework has direct implications for Bayesian inference where “word of mouth” information from experts is often obtained. It is interesting to note that our framework allows this information to update p at any point. Usually, the non-stochastic information I comes before the stochastic information given by an event B included in the probability framework, but in our approach, we can have B before I.