A puzzle of proportions: Two popular Bayesian tests can yield dramatically different conclusions.

Testing the equality of two proportions is a common procedure in science, especially in medicine and public health. In these domains, it is crucial to be able to quantify evidence for the absence of a treatment effect. Bayesian hypothesis testing by means of the Bayes factor provides one avenue to do so, requiring the specification of prior distributions for parameters. The most popular analysis approach views the comparison of proportions from a contingency table perspective, assigning prior distributions directly to the two proportions. Another, less popular approach views the problem from a logistic regression perspective, assigning prior distributions to logit-transformed parameters. Reanalyzing 39 null results from the New England Journal of Medicine with both approaches, we find that they can lead to markedly different conclusions, especially when the observed proportions are at the extremes (ie, very low or very high). We explain these stark differences and provide recommendations for researchers interested in testing the equality of two proportions and users of Bayes factors more generally. The test that assigns prior distributions to logit-transformed parameters creates prior dependence between the two proportions and yields weaker evidence when the observations are at the extremes. When comparing two proportions, we argue that this test should become the new default.

INTRODUCTION

Researchers frequently wish to test whether two populations differ. In medicine and public health, for example, the resulting statistical analysis frequently concerns testing whether or not two proportions differ. Examples include testing whether a vaccine decreases the number of infections compared to a control, whether sexual minorities are more prone to suicide compared to their heterosexual counterparts, or whether tightly or less‐tightly controlling hypertension leads to fewer miscarriages in pregnant women.

In these applications, it is crucial to be able to discriminate between evidence of absence and absence of evidence. For example, Magee et al conducted a trial to investigate the effect of a tight (target diastolic blood pressure, 85 mm Hg) or a less‐tight (target diastolic blood pressure, 100 mm Hg) control of hypertension in pregnant woman on, among other outcomes, pregnancy loss. They found no significant difference between the two conditions, with 15 out of 493 women in the less‐tight control condition and 13 out of 488 in the tight control condition having lost their child, yielding an estimated odds ratio of 1.14 (95% CI: [0.53, 2.45]). How confident are we that there is indeed no difference between the two conditions rather than the data being inconclusive? Bayesian statistics provides a principled way of quantifying evidence via the Bayes factor, , , thus providing one avenue to discriminate between evidence of absence and absence of evidence.

The Bayes factor quantifies how well one hypothesis predicts the data compared to another. Using the test of equality between two proportions as an example, let denote the combined data from the two groups. We have: where the sample sizes are assumed fixed and under we have that while under we have that . By quantifying relative predictive performance, the Bayes factor tells us how we should update our prior beliefs about relative to after observing the data: A Bayes factor of, say, 15 means that the data are 15 times more likely under one hypothesis compared to the other. While there exist verbal guidelines that may aid in the interpretation of the Bayes factor (eg, Bayes factors in the range from 1 to 3 constitute weak evidence, those in the range from 3 to 10 constitute moderate evidence, and values larger than 10 constitute strong evidence) , , the Bayes factor should be understood as a continuous measure of evidence.

While the Bayes factor does not depend on the prior probability of hypotheses, it does depend crucially on the prior over parameters in the models instantiating and , which becomes apparent when expanding: where and indicate the respective prior distributions.

There exist two main Bayes factor approaches for testing the equality of two proportions. The more popular one comes from the analysis of contingency tables, and assigns independent beta distributions directly to . , We call this approach the “Independent Beta” (IB) approach. The second approach is less widely used, and assigns a prior to the average log odds and the log odds ratio . , We call this approach the “logit transformation” (LT) approach. In this paper, we show that these two approaches can yield markedly different results. This is especially the case when the observed proportions are at the extremes (ie, very low or very high), as is the case for a large number of applications including the three examples mentioned above. Consider the study by Magee et al again. The IB approach yields a Bayes factor of 12.30 in favor of , while the LT approach yields a mere 1.17. In other words, under the IB approach the data are about 12 times more likely under the hypothesis that tightly or less‐tightly controlling hypertension have the same effect on miscarriages compared to a hypothesis assuming a difference. Under the LT approach, however, the data are about equally likely under both hypotheses, which constitutes equivocal evidence. The answer to the question “is observing two equally small proportions strong or weak evidence for the null hypothesis?” depends, therefore, crucially—and nontrivially—on the prior setup.

This article is structured as follows. In Section 2, we outline these two ways of testing the equality of two proportions in more detail. In Section 3, we highlight the occasionally stark differences of the two approaches by reanalyzing 39 statistical tests reported in the New England Journal of Medicine and explain why these differences occur. In Section 4, we end by reviewing the implications of the prior setup and what users of Bayes factors should be mindful of when testing the equality of two population parameters. We argue that the LT approach should become the default when testing the equality of two proportions because it (a) induces prior dependence between proportions which almost always are, in fact, dependent, and (b) yields a sensibly milder assessment of the evidence compared to the IB approach when the observations are at the extremes.

TWO WAYS OF TESTING THE EQUALITY OF TWO PROPORTIONS

In this section, we outline two ways of testing the equality of two proportions. In Section 2.1 we describe the Independent Beta approach, and in Section 2.2 we describe the logit transformation approach.

The independent beta (IB) approach

In order to nest the null hypothesis under the alternative hypothesis, we introduce the difference parameter and the grand mean . Using this parameterization, we have that:

The hypotheses are then specified as:

Under this approach, we assign independent priors to and . Figure 1 visualizes the joint prior distribution (top) under for (left) and (right). Increasing values of a implies that the joint prior mass is more concentrated around . The bottom panels visualize the conditional prior distribution of given that we know that . Knowing the value of does not change our prior about , which follows from the assumption of prior independence.

FIGURE 1

Top: Joint prior distribution assigned to under (left) and (right). Bottom: Conditional prior distribution of given that

The IB Bayes factor is available in analytic form. , , In the literature on contingency tables, our setup corresponds to an independent multinomial sampling scheme where the row (or column) sums of the contingency table (here and ) are fixed; for an extension to other sampling schemes and more than two groups, see References 12 and 13. Gunel and Dickey suggest as a default value. Note that values lead to an undefined prior density for and , which implies model selection inconsistency in case is indeed 1 or 0. Hence these values should be avoided on principle grounds. Because the beta prior is conjugate for the binomial likelihood, the posterior distributions of and are again (independent) beta distributions.

The logit transformation (LT) approach

The test proposed by Kass and Vaidyanathan and implemented by Gronau et al does not assign a prior directly to , but applies a logit transformation and assigns priors to the transformed parameters (, ). Specifically, we write: where is a grand mean and is the difference in log odds (ie, the log odds ratio):

While this is a more involved reparameterization than in the IB approach above, another way to formulate this setup is by writing: which readers familiar with logistic regression may recognize. Using this setup, we test the hypotheses:

In contrast to above, we now assign priors to and rather than to and directly. In particular, under both hypotheses we assume with . Under we assume . The top left panel in Figure 2 visualizes the implied joint prior distribution on under for . The prior mass is concentrated along the diagonal, which indicates that and are dependent. The bottom left panel illustrates this fact: if we know that , then the prior on shifts toward this value. Setting removes the prior dependency, as the right column in Figure 2 shows. For values of , and become anti‐correlated and hence observing a small value of results in a prior that puts more mass on large values for . Such an inverse relation is undesirable in almost all empirical applications, and so values are therefore to be avoided. Gronau et al developed software to compute the Bayes factor using this prior specification, first proposed by Kass and Vaidyanathan, suggesting as a default value.

FIGURE 2

Top: Joint prior assigned to for (left) and (right). Bottom: Conditional prior distribution of given that . In both cases we assume

Comparison of priors

A direct comparison of the two prior specifications may be helpful to get further intuition for their differences. While the IB approach does not assign a prior distribution to or explicitly, assigning a prior to and induces a prior distribution on these quantities. Conversely, the LT approach assigns a prior to and and this induces a prior on and . The induced prior distributions under both approaches are nonstandard (see Appendix A), but their densities can be calculated numerically.

The top left panel in Figure 3 shows the prior distribution assigned to by the LT approach for (shaded blue) and (striped blue) and the IB approach for (shaded red) and (striped red) under . Similarly, the top right panel shows the prior distribution assigned to for the two approaches and prior parameter values. The (default) IB approach assigns comparatively more mass to large values of and , which in practice means that it expects larger differences between the sample proportions. The bottom panel shows the marginal priors for and , where we find that the LT approach assigns comparatively less mass to extreme values. The LT approach cannot result in a uniform distribution on the proportions under because of the Gaussian prior on . If it instead would assign a (standard) logistic prior to (which has fatter tails), the prior on the proportions would be uniform; see a related discussion in Appendix B. In the next section, we discuss a somewhat surprising difference between these two tests.

FIGURE 3

Top: Prior distributions assigned to (left) and (right) under the LT (blue, vertical lines) and IB approach (red, horizontal lines), respectively. Bottom: Marginal prior distribution of and under the two approaches. The density that is filled and has the highest peak corresponds to

PRACTICAL IMPLICATIONS OF THE PRIOR SETUP

To see the implications of the two different prior specifications in practice, in Section 3.1 we reanalyze 39 null results published in the New England Journal of Medicine, previously analyzed by Hoekstra et al using the IB approach. We then explain why these difference occur in Section 3.2. The data and code to reproduce all analyses and figures is available from https://github.com/fdabl/Proportion‐Puzzle.

Reanalysis of New England Journal of Medicine studies

Hoekstra et al considered all 207 articles published in the New England Journal of Medicine in 2015. The abstract of 45 of these articles contained a claim about the absence or nonsignificance of an effect for a primary outcome measure, and 37 of those allowed for a comparison of proportions, reporting 43 null results in total. We focus on those results that can be reanalyzed using a test between two proportions, which results in a total of 39 tests from 32 articles. The top left panel in Figure 4 contrasts Bayes factors in favor of computed using the IB approach (rectangles) across with Bayes factors computed using the LT approach (circles) across . In virtually all cases and across specifications, the Bayes factor in favor of is higher under the IB approach, and this difference is frequently substantial. As the parameter a is increased under the IB approach, the expected difference between the two groups is smaller (see top left panel in Figure 3). Therefore, the predictions under become more similar to the predictions under , and the Bayes factor decreases. Conversely, as is increased under the LT approach, the expected difference between the two group increases, and the Bayes factor in favor of increases.

FIGURE 4

Top: Bayes factors using the IB (rectangles) and LT approach (circles) in favor of across studies reported in Hoekstra et al (left) or for simulated equal proportions with (right) for values and with . Bottom: Joint prior distribution of under the IB approach with (left) and under the LT approach with and (right). Black dots and blue rectangles indicate the maximum likelihood estimates of the proportions in the studies analyzed by Hoekstra et al.

The top left panel in Figure 4 shows that, for some studies, the IB and LT Bayes factors cannot be brought into the vicinity of each other by changing the prior parameters in the way specified above, while for other studies the Bayes factors do overlap substantially. The studies without overlap are those indexed as 1 to 12. They are shown as blue rectangles in the bottom panels in Figure 4, which shows the joint prior density for the IB (left) and the LT approach (right) with the symbols indicating the maximum likelihood estimates for the individual studies. Note that the larger the proportions in the bottom panels, the more likely are the two Bayes factors to overlap in the top left panel.

While Hoekstra et al are reassured by the fact that their Bayesian reanalysis (using the IB approach) yields strong evidence in favor of on average (given that all studies reported a nonsignificant P‐value), using the LT approach yields a more uncertain picture. While the median Bayes factor across the studies under the IB approach is 12.30, the median Bayes factor under the LT approach is only 4.79. This difference is driven by the extremes, where the two approaches suggest substantially different conclusions. We have already seen one example in the introduction concerning hypertension and pregnancy loss (analysis identifier 3). Another example is given by Joura et al, which compared the efficacy of the 9vHPV against the qHPV vaccine for preventing, among others, cervical, vulvar, and varginal disease and persistent human papillomavirus related infections in women. Comparing different outcome measures in women who were HPV‐negative and HPV‐positive at baseline, the IB approach yields Bayes factors consistently around 100 while the LT approach never shows Bayes factors larger than 10 (see analysis identifiers 1, 2, 5, and 6). The IB approach thus strongly suggests that the two vaccines have a similar efficacy, while the LT approach suggests that more data is needed to reach a firm conclusion.

The initially diverging and then converging pattern of the Bayes factors can be seen neatly in the top right panel in Figure 4, which shows the Bayes factors under the null hypothesis for increasing values of . Note that the pattern is symmetric, so that for the same holds, just mirrored. At the extremes , the IB approach yields a Bayes factor in favor of of 50.75, while the LT Bayes factor gives a mere 1.40. The Bayes factor decreases as under the IB approach, but increases under the LT approach. Consequently, the difference between the Bayes factors becomes less pronounced as we move to more central proportions, which matches the observation in the empirical analysis shown in the top left panel in Figure 4. For , the IB Bayes factor yields 5.70 and the LT Bayes factor yields 3.67.

In summary, we have found that the two approaches give a very different answer to the question “is observing two equally small proportions strong or weak evidence in favor of the null hypothesis?”. The IB Bayes factor suggests that the evidence is often orders of magnitude larger than the LT Bayes factor.

Explaining the difference

Why does the IB Bayes factor yield so much stronger evidence than the LT Bayes factor for when the proportions are small and of roughly equal size? And why does the IB Bayes factor decrease as the proportions get closer to while the LT Bayes factor increases? To answer these questions, we need to zoom in on the differences in the respective prior specifications. The two approaches differ in two key ways. First, while the IB approach assigns independent priors to and , the LT approach assigns dependent priors to them. Second, the LT approach employs a logit transformation while the IB approach does not. Appendix D shows that it is not the prior dependence that underlies the difference between the two approaches. Instead, as we will see below, it is the logit transformation.

The fact that the IB Bayes factor is larger than the LT Bayes factor even when the latter approach expects a larger difference in the proportions (eg, a larger or , compare with in Figure 3) means that focusing only on the difference parameter is not sufficient to explain the difference between the Bayes factors. Instead, we turn to a sequential predictive perspective. From such a perspective, we may first use the prior to predict the data from group one, update the prior to a posterior, and then predict the data from group two. One can rewrite the marginal likelihoods to make this sequential perspective apparent, see Appendix E. This perspective shows that there is a crucial difference in the predictions that the CT and LT approach make under . Under both approaches there is a common parameter under , and the prior assigned to gets updated by data from the first group, after which predictions about the second group are made. Under the alternative hypothesis, however, the IB approach implies that observing data from group one does not update our beliefs about likely values of , and so data from group one cannot inform the subsequent prediction about group two. Under the LT approach, such information sharing does take place. This difference is shown in the top panels in Figure 5, which visualizes the joint prior distribution for and under the IB (left) and LT (right) approaches. We see that under the IB approach, learning about does not influence our predictions about likely differences between the two groups. In contrast, under the LT approach we find that when we learn that is either small or large, we expect small differences between the proportions compared to when we learn that is about half. This explains why the LT Bayes factor in favor of increases as we move from extreme values of toward values around , as shown in the top right panel in Figure 4: at the extremes, expects smaller differences between and , and hence it more closely resembles , leading to a more equivocal Bayes factor. Moving toward more central values, expects larger differences and hence outpredicts it by a larger margin.

FIGURE 5

Top: Joint prior distribution for under the IB approach with (left) and under the LT approach with and (right). Bottom: Joint prior distribution for under the two approaches

While prior dependence is necessary to change one's beliefs about likely differences between the proportions upon learning , the fact that the expected difference decreases in the LT approach is due to the logit transformation. When the two rates are small, say and , the log odds difference is large, in this case . The prior on in the LT approach renders such large log odds differences to be unlikely, which means that a smaller rate difference is expected. Conversely, if the rates are somewhere in the center, say and , then the log odds difference is small, in this case . The prior on in the LT approach views such small log odds differences as likely; in fact, much larger ones are possible. Therefore, the LT approach expects larger rate differences when the rates are in the center. On the log odds scale, the expected difference does not change markedly as varies, as the bottom right panel in Figure 5 shows.

In contrast to the LT approach, the IB approach does not automatically reduce the expectations about likely differences as we learn that takes on extreme values. Taking slices at particular values for in the top left panel in Figure 5 would show uniform distributions ranging from to . Instead of reducing the size of an expected difference at the extremes, the IB approach actually amplifies it. This occurs because of the boundedness of the rate parameters. In particular, if , then the only values for which, say, are . Differences in absolute magnitude between the two rates of up to 0.10 if are therefore assigned a probability of 0.10. If, however, , the values of for which are . This means that differences in absolute magnitude between the two rates of up to 0.10 if are assigned a probability of 0.20, or twice as much as in the case that . Another way to see this is to note that the expected difference in the case that is , while the expected difference in the case that is ; more generally, the expected difference increases linearly as we move to the extremes, as can be seen in the top left panel in Figure 5. The bottom left panel in Figure 5 shows an associated and pronounced increase in the expected log odds difference as moves toward extreme values.

DISCUSSION

Being able to discriminate between evidence of absence and absence of evidence is key in medicine and public health. The Bayes factor, which pits the predictive performance of two hypotheses against each other, is a method for doing so. Reanalyzing 39 null results published in the New England Journal of Medicine, we have found that the strength of evidence depends crucially on the prior specification. Comparing an approach that assigns independent beta distributions directly to the rate parameters (IB) to one that employs a logit transformation (LT), we have found that the former approach suggests evidence that is orders of magnitudes larger than the latter approach when observing small proportions of roughly equal size. Consider again the effect of tightly or less‐tightly controlling hypertension in pregnant women. The IB approach suggests a Bayes factor about ten times larger in favor of the null hypothesis than the LT approach. Similarly, the Bayes factor in favor of the hypothesis that two vaccines against the human papillomavirus are equally effective is up to 38 times larger under the IB approach, depending on the outcome measure. While the original statistical analysis by the authors is more involved (eg, adjusting for various covariates), these differences are striking. As we have seen, they occur because of the boundedness of the parameter space, which leads to the expectations of large differences at the extremes under the IB approach. In contrast, the LT approach expects smaller differences at the extremes.

Although the two Bayes factors frequently differed markedly in size, they always both provided evidence in favor of in the cases we studied in this paper. This need not be the case, however. For example, a large‐scale investigation into the effect of aspirin found that 26 out of who received a placebo suffered a fatal heart attack, but only 10 out of who received aspirin did. The IB Bayes factor yields strong evidence in favor of () while the LT Bayes factor finds moderate evidence in favor of (). This is again owed to the fact that the IB approach expects larger differences than the LR approach; a small effect size is more likely under than under in the IB approach simply because under unrealistically large effect sizes are assumed to be plausible.

Our results have a number of implications. First, researchers should think carefully about their prior setup. This not only implies thinking about the parameter values for the priors, but also the nature of the priors and what they imply for the data. As we have seen, one may be fooled by assessing how sensitive the Bayes factor is by varying only prior parameters conditional on the model. For all studies we reanalyzed that had proportions at the extremes, varying the prior parameters of the IB approach still resulted in substantial evidence in favor of the null hypothesis. However, changing the prior setup to the LT approach, we found markedly reduced evidence. If there are sensible alternative parameterizations, it may be prudent to explore how sensitive the results are to these. In particular, our exposition suggests that assessing the predictions that follow from the model can help get a better intuition of its assumptions. In our case, this made clear that the IB approach expects larger rate differences at the boundaries of the parameter space compared to the center, while the LT approach expects the reverse. We were struck by the initially puzzling contrast between these two tests, given that the task—comparing two proportions—seemed so simple. This suggests that even more caution needs to be applied when using Bayes factors for comparing models that are invariably more complex. In the same spirit of sensitivity analysis, it is prudent to compare inferences based on the Bayes factor with inferences based on the posterior distributions of the parameters. While testing precise hypotheses based on credible intervals is arguably inappropriate since it makes inference conditional on , ignoring , , alarm bells should go off when they show stark differences to the Bayes factor results, as in our case (compare Figure 4 and C1). Researchers who rely only on the posterior distribution for inference or for trial design are unaffected by our results.

FIGURE C1

Shows the posterior mean and 95% credible interval of the log odds difference (top) and the rate difference (bottom) under the IB approach (rectangles, ) and the LT approach (circles, )

The second implication of our work is that the LT approach appears to be better suited for testing the equality of two proportions than the currently more popular IB approach. First, assuming prior dependence strikes us as a more sensible approach especially for medicine and public health. This is because it is generally unlikely that gaining information about one group does not influence our beliefs about the other group. For example, if a particular treatment for cancer yields a success rate, surely this should inform our expectations for the success rate of a comparison treatment. Incorporating this dependency results in more cautious predictions, which reduces the eagerness with which the standard IB approach suggests strong evidence in favor of (see also Appendix D). While we have focused on testing the sharp null hypothesis , the difference between dependent and independent priors is important also when testing other hypotheses. Howard discusses the difference between a dependent and independent prior in the context of comparing against , and similarly recommends dependent priors, as independent priors lead to a test that is not “sufficiently cautious”.

Most importantly, however, is the fact that the LT approach avoids the issues associated with a bounded parameter space. In particular, the IB approach is not sensitive to the fact that the implications of a rate difference depends on the actual values of the rates. For example, and and and imply the same rate difference, but while the former represents a 100% increase, the latter represents only a 10% increase. Yet the IB approach assigns more prior mass to large rate differences at the extremes of the parameter space compared to its center. In contrast, the reverse holds for the LT approach because of the logit transformation. This strikes us as more sensible. With the recent development of accessible software that makes the LT approach easily available, , we hope that the use of the LT approach for testing the equality of two proportions becomes more widespread.

To sum up, we have seen an initially puzzling divergence in evidence when reanalyzing 39 null results from the New England Journal of New Medicine with two different Bayes factor tests. We have explained this divergence and sided with the approach that employs a logit transformation and assumes prior dependence between the two proportions. We suggest that this approach, rather than the one motivated from a contingency table perspective, should be routinely applied when testing the equality of two proportions. Our journey should also act as a cautionary tale for users of Bayes factors, stressing the importance of assessing the predictions of one's models, conducting thorough sensitivity analyses, and never relying on just a single quantity. We have demonstrated the usefulness of this in the simple case of testing the equality of two proportions. Most applications will arguably be more complex, however, increasing the potential for puzzles and stressing the need for a holistic evidence assessment.

AUTHOR CONTRIBUTIONS

Eric‐Jan Wagenmakers proposed the study. Karoline Huth, Quentin F. Gronau, Alexander Etz, and Eric‐Jan Wagenmakers worked on the problem initially. After an interim period, Fabian Dablander and Eric‐Jan Wagenmakers resumed the work and solved the puzzle. Fabian Dablander and Karoline Huth analyzed the data. Fabian Dablander wrote the manuscript. All authors read, provided comments, and approved the submitted version of the paper.

CONFLICTS OF INTEREST

The authors declare that there were no conflicts of interest.