Bounding the Plausibility of Physical Theories in a Device-Independent Setting via Hypothesis Testing.

The device-independent approach to physics is one where conclusions about physical systems (and hence of Nature) are drawn directly and solely from the observed correlations between measurement outcomes. This operational approach to physics arose as a byproduct of Bell's seminal work to distinguish, via a Bell test, quantum correlations from the set of correlations allowed by local-hidden-variable theories. In practice, since one can only perform a finite number of experimental trials, deciding whether an empirical observation is compatible with some class of physical theories will have to be carried out via the task of hypothesis testing. In this paper, we show that the prediction-based-ratio method-initially developed for performing a hypothesis test of local-hidden-variable theories-can equally well be applied to test many other classes of physical theories, such as those constrained only by the nonsignaling principle, and those that are constrained to produce any of the outer approximation to the quantum set of correlations due to Navascués-Pironio-Acín. We numerically simulate Bell tests using hypothetical nonlocal sources of correlations to illustrate the applicability of the method in both the independent and identically distributed (i.i.d.) scenario and the non-i.i.d. scenario. As a further application, we demonstrate how this method allows us to unveil an apparent violation of the nonsignaling conditions in certain experimental data collected in a Bell test. This, in turn, highlights the importance of the randomization of measurement settings, as well as a consistency check of the nonsignaling conditions in a Bell test.

1. Introduction

In physics, the terminology “device-independent” apparently made its first appearance in Ref. [1] where the authors drew a connection between the celebrated discovery by Bell [2] and the vibrant field of quantum cryptography [3]. As of today, device-independent quantum information has become a well-established research area where Bell-inequality-violating correlations find applications not only in the distribution of secret keys [4,5,6] (see also Ref. [7]), but also in the generation of random bits [8,9,10], as well as in the assessment of uncharacterized devices (see, e.g., Refs. [11,12,13,14,15,16,17]). For a comprehensive review, see Refs. [18,19].

A device-independent approach to physics, however, could be traced back, for example, to the work of Bell [2]. There, he showed that any local-hidden-variable (LHV) theory [20] must be incompatible with certain quantum predictions. The proof is “device-independent” in the sense that one needs no further assumption about the nature of the theory (including the detailed functioning of any devices that one may use to test the theory). Rather, the proof relies on a common ingredient of operational physical theories—correlations between measurement outcomes, i.e., the probability of getting particular measurement outcomes conditioned on certain measurement choices being made—to manifest the incompatibility.

By now, this incompatibility has been verified in various loophole-free Bell tests, such as those reported in Refs. [21,22,23,24,25]. Importantly, any real experiments must involve only a finite number of experimental trials. Statistical fluctuations must thus be carefully taken into account in order to draw any conclusion against a hypothetical theory, such as an LHV theory. For example, using the observed relative frequencies as a naïve estimator of the underlying correlations would generically (see, e.g., Refs. [26,27]) lead to a violation of the nonsignaling conditions [28,29]. Since the assumption of nonsignaling is a prerequisite for any Bell tests, it is only natural that a Bell test of LHV theories must also be accompanied by the corresponding test of this assumption [22,23,24,25,30] (see also Refs. [31,32,33]).

The effects of statistical fluctuations in a Bell test were (in fact, still are) often reported in terms of the number of standard deviations the estimated Bell violation exceeds the corresponding local bound (see, e.g., Refs. [34,35,36,37,38,39,40,41,42]). However, there are several problems with such a statement (see Refs. [19,43] for detailed discussions). Alternatively, as a common practice in hypothesis testing, one could also present the p-value according to a certain null hypothesis (e.g., the hypothesis that a LHV theory holds true). The corresponding p-value then describes the probability that the statistical model (associated with the null hypothesis) produces some quantity (e.g., the amount of Bell-inequality violation) at least as extreme as that observed.

A pioneering work in this regard is that due to Gill [44] where he presented a p-value upper bound according to the hypothesis of a LHV theory based on the violation of the Clauser-Horne-Shimony-Holt (CHSH) [45] Bell inequality. A few years later, a systematic method that works directly on the observed data (without relying on any predetermined Bell inequality)—by the name of the prediction-based-ratio method—was developed by one of the present authors and coworkers [43] (see also Ref. [46]). This method was designed for computing a p-value upper bound—based on the data collected in a Bell test—according to LHV theories. As we shall show in this work, essentially the same method can be applied for the hypothesis testing of some other nonlocal physical theories, thus allowing us to bound the plausibility of physical theories beyond LHV theories.

Indeed, since the pioneering work by Popescu and Rohrlich [28], there has been an ongoing effort (see, e.g., Refs. [47,48,49,50]) to find well-motivated physical [51,52] or information-theoretic [53,54,55,56] principles to recover precisely the set of quantum correlations. Unfortunately, none of these has succeeded. Rather, they each define a set of correlations that outer approximates the quantum set [57]. In other words, they also contain correlations that are more nonlocal than that allowed by quantum theory. For example, the so-called “almost-quantum” [50] set of correlations is one such superset of the quantum set, yet satisfying essentially all the proposed principle known to date. In the rest of this work, it suffices to think of this set as a fairly good outer approximation to the quantum set of correlations.

In this work, we show that the prediction-based-ratio method can be applied to test any physical theory that is constrained to produce correlations that is amenable to a semidefinite programming [58] characterization. In particular, it can be applied to test any physical theory that is constrained to produce nonsignaling [28] correlations, or any theory that respects macroscopic locality [51] or which gives rise to the almost-quantum [50] set of correlations etc.

2. Methods

2.1. Preliminaries

For a complete description of the prediction-based-ratio method and a comparison of its strength against the martingale-based method [44], we refer the reader to Ref. [43]. Here, we merely recall the necessary ingredients of the prediction-based-ratio method and show how it can be used to achieve the purpose of bounding the plausibility of physical theories based on the data collected in a Bell test, with minimal assumptions. Making this possibility evident and demonstrating how well it works in practice are the main contributions of the present work.

For simplicity, the following discussions are based on a Bell test that involves two parties (Alice and Bob) who are each allowed to perform one of two measurements randomly selected at each trial, each produces one of two possible outcomes. Generalization to other Bell scenarios will be evident. To this end, let us denote the measurement choice (input) of Alice (Bob) by x (y) and the corresponding measurement outcome (output) by a (b), where . The extent to which the distant measurement outcomes are correlated is then succinctly summarized by the collection of joint conditional probability distributions .

In an LHV theory, the outcome probability distributions can be produced with the help of some LHV (distributed according to ) via the local response functions satisfying and such that [2]:

Hereafter, we refer to any that can be decomposed in the above manner as a (Bell-) local correlation and denote the set of such correlations as .

In contrast, if Alice and Bob conduct the experiment by performing local measurements on some shared quantum state , quantum theory predicts setting-dependent outcome distributions for all of the form:where and denote, respectively, the local positive-operator-value-measure element associated with the a-th outcome of Alice’s x-th measurement and the b-th outcome of Bob’s y-th measurement. Accordingly, we refer to any that can be written in the form of Equation (2) as a quantum correlation and the set of such correlations as .

Importantly, both local and quantum correlations satisfy the nonsignaling conditions [29]:where and are marginal probability distributions of . Should (any of) these conditions be violated in a way that is independent of spatial separation, Alice and Bob would be able to communicate faster-than-light [28] via the choice of measurement . We shall denote the set of satisfying Equation (3) as . It is known that , , and are convex sets and that they satisfy the strict inclusion relations (see, e.g., Ref. [19] and references therein).

A few other convex sets of correlations are worth mentioning for the purpose of subsequent discussions. To this end, note that the problem of deciding if a given is in is generally a difficult problem. However, the characterization of can, in principle, be achieved by solving a converging hierarchy of semidefinite programs [58] due to Nacascués, Pironio, and Acín (NPA) [59,60] (see also Ref. [61,62]). The lowest level outer approximation of in this hierarchy, often denoted by , happens to be exactly the set of correlations that is characterized by the physical principle of macroscopic locality [51]. A finer outer approximation of corresponding to the lowest-level hierarchy of Ref. [62], which we denote by , is known in the literature as the almost-quantum set [50], as it appears to satisfy all the physical principles that have been proposed to characterize . In Section 3, we use and as examples to illustrate how the prediction-based-ratio method can be adapted to test physical theories that are constrained to produce correlations from these sets.

2.2. Finite Statistics and the Prediction-Based-Ratio Method

Coming back to an actual Bell test, let be the total number of experimental trials carried out during the course of the experiment. During each experimental trial, x and y are to be chosen randomly according to some fixed probability distribution (This distribution may be varied from one trial to another but for simplicity of discussion, we consider in this work only the case where this is fixed once and for all before the experiment begins). From the data collected in a Bell test, a naïve (but very commonly-adopted) way to estimate the correlation between measurement outcomes is to compute the relative frequencies that each combination of outcomes occurs given the choice of measurement , i.e., where is the total number of trials the events corresponding to are registered and is the number of times the particular combination of measurement settings is chosen. By definition, .

If the experimental trials are independent and identically distributed (i.i.d.) corresponding to a fixed state with fixed measurement strategies , then in the asymptotic limit, where here would satisfy Equation (2). In this limit, the amount of statistical evidence in the data against a particular hypothesis can be quantified by the Kullback-Leibler (KL) divergence [63] (also known as the relative entropy) from to , see Refs. [64,65] for a detailed explanation with quantum experiments. We remark that the KL divergence is directly related with the Fisher information metric and so it measures the distinguishability of a distribution from its neighborhood. This provides a motivation for using the KL divergence as a measure of statistical evidence.

In the (original) prediction-based-ratio method of Ref. [43] (see also Ref. [66]), the hypothesis of interest is that the experimental data can be produced using an LHV theory, in other words, that the underlying correlation . For convenience, we shall refer to this hypothesis as . In this case, given and , the relevant KL divergence from to reads as

As the objective function in Equation (5) is strictly convex in and the feasible set is convex, the minimizer of the above optimization problem—which we shall denote by —is unique (see, e.g., Ref. [27]). It follows from the results presented in Ref. [43] that this unique minimizer can be used to construct a Bell inequality: where the non-negative coefficients of the Bell inequality are defined via the ratios

This Bell inequality is the key ingredient of the prediction-based-ratio method and is ideally suited for performing a hypothesis test of .

To understand the method, we introduce the random variables X and Y to denote the random inputs and the variables A and B to denote the random outputs of Alice and Bob at a trial. The ability to select measurement settings randomly, in particular, is an indispensable prerequisite of the prediction-based-ratio method, or more generally, a proper Bell test (see, e.g., Ref. [20]). We further denote the possible values of inputs and outputs by the respective lower-case letters. Then we can think of the ratio R in Equation (6) as a non-negative function of the inputs and outputs at each experimental trial such that its expectation according to an arbitrary with the fixed input distribution satisfies

Equation (7) is an alternative way of expressing the Bell inequality of Equation (6). A real experiment necessarily involves only a finite number of experimental trials in time order. Here, we have split the experimental data into two sets: the data from the first trials as the training data and the data from the remaining trials as the hypothesis-testing data. In practice, we first construct the function R using the training data and then perform a hypothesis test with the test data. Since the ratio R is determined before the hypothesis test based on the prediction according to the training data, R is called a prediction-based ratio.

Given a prediction-based ratio and a finite number of test data, we can quantify the evidence against the hypothesis by a p-value. For concreteness, suppose that the actual measurements chosen at the i-th test trial are , and the corresponding measurement outcomes observed are , . Then the value of the prediction-based ratio at the i-th test trial is , abbreviated as . We introduce a test static T as the product of the possible values of the prediction-based ratio at all test trials, so the observed value of the test statistic is . If we denote by the total number of counts registered for the input-output combination in the test data, then t can be expressed also as

According to Ref. [43], the p-value, which is defined as the maximum probability according to the hypothesis of obtaining a value of T at least as high as t actually observed in the experiment, is bounded by

The smaller the p-value, the stronger the evidence against the hypothesis is, in other words, the less plausible LHV theories are. It is worth noting that the p-value bound computed in this manner remains valid even if the experimental trials are not i.i.d., while when the experimental trials are i.i.d., the p-value bound is asymptotically optimal (or tight) [43].

2.3. Generalization for Hypothesis Testing Beyond LHV Theories

The following two simple observations, which allow one to apply the prediction-based-ratio method to test physical theories beyond those described by LHV, are where our novel contribution enters. Firstly, we make the observation that in the above arguments leading to the p-value bound of Equation (9), the actual hypothesis only enters at Equation (6) via the set of correlations compatible with the hypothesis . In particular, if we are to consider the hypothesis that the data observed is produced by a physical theory H (e.g., a nonsignaling theory), then we merely have to replace by the (convex) set of correlations (e.g., ) associated with H in the optimization problem of Equation (5). The method then allows us to bound the plausibility of the hypothesis via the p-value bound in Equation (9) with the possible values of the prediction-based ratio given by where is the unique minimizer of the optimization problem:

Although Equation (8), Equation (9) and Equation (10) together provide us, in principle, a recipe to test the plausibility of a general physical theory H, its implementation depends on the nature of the set of correlations associated with the hypothesis. Indeed, a crucial part of the procedure is to solve the optimization problem of Equation (11) for the convex set of correlations compatible with H, which is generally far from trivial. If is a convex polytope, such as and , or the set of correlations associated with the models considered in Refs. [67,68]), it is known [43] that Equation (11) can indeed be solved numerically.

Our second observation is that for the convex sets of correlations that are amenable to a semidefinite programming characterization, such as those considered in Refs. [59,62,69,70], Equation (11) is an instance of a conic program [58] that can be efficiently solved using a freely available solver, such as PENLAB [71]. To see this, one first notes that, apart from the constant factor , the optimization of Equation (11) is essentially the same as that considered in Ref. [27]. A straightforward adaptation of the argument presented in Appendix D 2 of Ref. [27] would then allow us to complete the aforementioned observation. The data observed in a Bell test can thus be used to test not only , but also and even the hypothesis that the observation is compatible with Born’s rule, cf. Equation (2), via outer approximations of (such as and ).

A remark is now in order. In order to avoid so-called p-value hacking, it is essential that the test data used in the computation of the test statistic T is not used to determine , and hence the values of the prediction-based ratio R in Equation (10). In this work, for simplicity we use the first trials of an experiment as the training data for estimating and further constructing a prediction-based ratio R that is applied for all test trials. In principle, we can use different training data for different test trials. For example, we can define the training data for a test trial as the data from all trials performed before this test trial, and then we can adapt the construction of the prediction-based ratio for each individual test trial. We refer to Ref. [43] for more details on the adaptability of the prediction-based ratio.

3. Results

To illustrate how well the prediction-based-ratio method works in identifying data that are not even explicable by some nonlocal physical theories, such as quantum theory, we now consider a few examples of applications of the method. As above, we restrict our attention to a bipartite Bell test, where each party performs two binary-outcome measurements randomly selected at each trial. Throughout this section, we assume that the input distribution is uniform, specifically for all combinations of . In Section 3.2 and Section 3.3 we study the behaviour of numerically simulated Bell tests based on hypothetical sources of correlations described in Section 3.1, while in Section 3.4, we analyze the real experimental data reported in Ref. [72].

3.1. Modeling a Bell Test

For our numerical simulations, we consider a that resembles a nonlocal source targeted at in various actual Bell experiments [35,36,37,72,73]:where , is the Popescu-Rohrlich (PR) correlation [28] with and for all is the white-noise distribution. In Equation (12), the real parameter v can be seen as the weight associated with in the convex mixture. Importantly, the nonlocal source represented by such a mixture can (in principle) be produced by performing appropriate local measurements on a maximally entangled two-qubit state if and only if (see, e.g., Refs. [27,57]). In particular, when —corresponding to an ideal nonlocal source—the mixture gives rise to the maximal quantum violation of the CHSH [45] Bell inequality.

To mimic an experimental scenario with noise (something unavoidable in practice), we shall introduce a slight perturbation to the ideal source of Equation (12). Specifically, we require the measurement outcomes observed at each trial in the simulated Bell test to be governed by the nonlocal source , where is the weight associated with the noise term . Moreover, for the purpose of illustrating the effectiveness of the method in identifying non-quantum-compatible data, we set . In our simulations, we set and . However, as long as the given mixture lies outside (and hence also outside ), the actual choices of and are irrelevant. The only impact that these choices may have is the number of trials needed to falsify the hypothesis with the same level of confidence. Inspired by the experiments of Ref. [72] where ∼, we set in our simulations . Note also that instead of , we can equally well choose another set of correlations that admits a semidefinite programming characterization, such as those described in Refs. [59,62].

“The observed data is compatible with a physical theory that is constrained to produce only the almost-quantum set of correlations.”

Since we are interested to model a nonlocal source that obeys the nonsignaling conditions of Equation (3), there is no loss in generality by considering . To this end, let be the j-th extreme point of the nonsignaling polytope [29], then we may write where is the weight associated with in the convex decomposition of . We may thus write the nonlocal source of interest as:

Finally, to simulate the raw data obtained in an N-trial Bell test for any given input distribution and correlation , we make use of the MATLAB toolbox Lightspeed developed by Minka [74].

3.2. Simulations of Bell Tests with an i.i.d. Nonlocal Source

Let us begin with the case of i.i.d. trials, corresponding to a source of correlation that remains unchanged throughout the experiment, and where the inputs at each trial are independent of the inputs of the previous trials. To this end, we first sample the weights uniformly from the interval and renormalize them such that . With our choice of and , it is easy to find such a randomly generated correlation that lies outside . (Verifying that any given is (not) in can be carried out by solving a semidefinite program. Specifically, for any given correlation , if the maximal white-noise visibility such that is smaller than 1, then , and hence outside , otherwise .) For convenience, we denote by the specific set of employed in our simulation of 500 Bell tests, each with trials. In Figure 1, we summarize the steps involved in our analysis of the numerically simulated data using the prediction-based-ratio method. The resulting p-value upper bounds are summarized in Table 1.

Figure 1

Flowchart summarizing the steps involved in our application of the prediction-based-ratio method on the simulated data of a single Bell test. In the first step, we separate the data into two sets, with the data collected from the first trials serving as the training data while the rest is used for the actual hypothesis testing. Specifically, the training data is used to compute the relative frequencies and to minimize the KL divergence with respect to the set of correlations associated, respectively, with the hypothesis of and . The correlation that minimizes gives rise to a Bell-like inequality with coefficients . The remaining data is then used to compute where . Finally, a p-value bound according to the hypothesis is obtained by computing .

Table 1

Summary of frequency distributions of the p-value upper bounds obtained from 500 numerically simulated Bell tests, each consists of trials and assumes the same i.i.d. nonlocal source of Equation (13) that lies outside . The second and third row give, respectively, the frequency distributions according to the hypothesis associated with (nonsignaling) and (almost-quantum). For these hypotheses, the smallest p-value upper bound found among these 500 Bell tests are, respectively, 0.14 and . The second to the fifth column give, respectively, the fraction of simulated Bell tests having a p-value upper bound (for each hypothesis) that satisfies the given (increasing) threshold (e.g.,  for the second column). Similarly, in the last column, we give the fraction of instances where the p-value upper bound obtained is trivial, i.e., exactly equals to 1. The smaller the p-value upper bound, the less likely it is that a physical theory associated with the hypothesis produces the observed data. Thus, the larger the value in the second (to the fourth) column, the less likely it is that the assumed physical theory holds true. In contrast, the larger the value in the rightmost column, the weaker the empirical evidence against the assumed theory is.

p-Value Bound	≤10^−10	≤10^−4	≤10^−2	≤10^−1	Trivial
NS	0	0	0	0	97%
Q˜	58%	85%	90%	93%	5.8%

As expected, despite statistical fluctuations, the data does not suggest any obvious evidence against the nonsignaling hypothesis. In fact, among the 500 p-value bounds obtained, 97% of them are trivial (i.e., equal to unity), while the smallest non-trivial p-value bound obtained is approximately 0.14. On the contrary, for the hypothesis test of the almost-quantum set of correlations, more than half of the simulated Bell tests give a p-value upper bound that is less than . Although there are also 5.8% of these simulated Bell tests that give a trivial p-value bound according to the almost-quantum hypothesis, we see that the method generally works very well in falsifying this hypothesis. In fact, a separate calculation (not shown in the table) shows that when we increase to , all the 500 p-value upper bounds obtained according to the almost-quantum hypothesis are less than or equal to .

3.3. Simulations of Bell tests with a non-i.i.d. Nonlocal Source

In a real experiment, the assumption that the experimental trials are i.i.d is often far from justifiable, as that would require, for example, that the experimental setup remain as it is over the entire course of the experiment. As a result, we also consider here the case where the source that generates the data actually varies from one trial to another. To this end, for the i-th trial of the Bell test, we simulate according to the conditional outcome distributions:where labels the single nonsignaling extreme point used to mix with at this trial, cf. Equation (13) with if but vanishes otherwise. Moreover, to facilitate a comparison with the i.i.d. case, before the i-th trial, we randomly pick according to the probability where is exactly the probability employed in the simulation of Section 3.2. With this choice, the outcome distributions governed by the nonlocal source of Equation (14) (for the i-th trial) averages to that of Equation (13) when the number of trials . Again, we follow the steps summarized in Figure 1 to compute the relevant p-value upper bounds using the prediction-based-ratio method. The resulting p-value upper bounds are summarized in Table 2.

Table 2

Summary of frequency distributions of the p-value upper bounds obtained from 500 numerically simulated Bell tests. Each of these Bell tests involves trials and each trial assumes a varying source of Equation (14). For the hypothesis of and , associated with (second row) and (third row), respectively, the smallest p-value upper bound found among these 500 instances are 0.21 and . The significance of each column follows that described in the caption of Table 1.

p-Value Bound	≤10^−10	≤10^−4	≤10^−2	≤10^−1	Trivial
NS	0	0	0	0	97%
Q˜	17	59%	69%	72	24%

As with the i.i.d. case, for these 500 simulated Bell tests, our application of the prediction-based-ratio method does not lead to any obvious evidence against the nonsignaling hypothesis . However, for the hypothesis associated with the almost-quantum set , our results (last row of Table 2) give more than half of the p-value upper bounds that are less than (accordingly, 17% if we set the cutoff at ). Although there are 24% of these instances where the returned p-value upper bound for the same hypothesis is trivial, we see that, as with the i.i.d. case, the method remains very effective in showing that the observed data cannot be entirely accounted for using a theory that is constrained to produce only almost-quantum correlations. In addition, as with the i.i.d. case, our separate calculation shows that the effectiveness of this method can be substantially improved when we increase to : all the 500 p-value upper bounds obtained according to the almost-quantum hypothesis become less than or equal to .

3.4. Application to Some Real Experimental Data

Armed with the experience gained in the above analyses, let us now analyze the experimental results presented in Figure 3 of Ref. [72] using the prediction-based-ratio method. One of the goals of Ref. [72] was to experimentally approach the boundary of the quantum set of correlations in the two-dimensional subspace spanned by the two Bell parameters:where is the correlator. To this end, the Bell parameter for 180 uniformly-spaced values of were estimated by performing the measurements presented in Appendix A of Ref. [72] on a two-qubit maximally entangled state.

Unfortunately, only the total counts for each combination of input-output (rather than the time sequences of raw data) given the value of are available [75]. Therefore, in analogy with the analyses presented above, we use the relative frequencies obtained for as the training data to derive a prediction-based ratio (which corresponds to a Bell-like inequality) for the hypothesis test using the data associated with (for the case of , the hypothesis test uses the data associated with ). The analysis therefore essentially follows the steps outlined in Figure 1, but with the computation of t carried out using Equation (8) instead, since we do not have the time sequences of raw data. Moreover, to apply the prediction-based-ratio method, we assume, as with the numerical experiments reported earlier that the input distributions are uniform, i.e., for all combinations of . A summary of the p-value upper bounds obtained from these 180 Bell tests is given in Table 3.

Table 3

Summary of frequency distributions of the p-value upper bounds obtained from the 180 Bell tests of Ref. [72] according to the hypothesis of and (associated, respectively, with , the second row, and , the third row) under the assumption that the measurement settings were randomly chosen according to a uniform distribution. The significance of each column follows that described in the caption of Table 1.

p-Value Bound	≤10^−10	≤10^−4	≤10^−2	≤10^−1	Trivial
NS	38%	45%	48%	51%	48%
Q˜	35%	44%	47%	49%	49%

For both hypotheses, approximately half of the p-value upper bounds obtained are trivial. At the same time, about the same fraction of the p-value bounds obtained are less than (with the majority of them being less than ). In fact, the smallest of the p-value upper bounds are remarkably small: for the hypothesis of nonsignaling and for the hypothesis of almost-quantum . These results strongly suggest that under the assumption that the measurement settings were randomly chosen according to a uniform input distribution, it is extremely unlikely that a physical theory associated with each of these hypotheses can produce the observed relative frequencies.

These conclusions that the observed data are incompatible with the fundamental principle of nonsignaling or with quantum theory (via the almost-quantum hypothesis), however, turn out to be flawed, as it was brought to our attention [75] that during the course of the experiment, the measurement bases were not at all randomized—the measurements were carried out in blocks using the same combination of before moving to another. Why should this pose a problem? In the extreme scenario, if the measurement settings were fully correlated to some local hidden variable, it is known that the the resulting correlation between measurement outcomes can violate the nonsignaling conditions of Equation (3), see, e.g., Ref. [76]. Consequently, it is not surprising that in the prediction-based-ratio method (as well as any other methods employed for the statistical analysis of a Bell test), the measurement inputs during the i-th trial, as discussed in Section 2, ought to be randomly chosen.

4. Discussion

As discussed in the last section, the conclusion that “the experimental data of Ref. [72] show a violation of the nonsignaling principle" based on an erroneous application of the prediction-based-ratio method is unfounded. The results are nonetheless thought-provoking. For example, suppose for now that we had access to the raw data for all trials. Since the analysis was flawed because of the nonrandomnization of measurement settings, one can imagine that—under the assumption that the trials are exchangeable—we first artificially randomize the hypothesis-testing trials to simulate the randomization of measurement settings in the experiment. Should we then expect to obtain p-value bounds with fundamentally different features? The answer is negative. The reason is that in our crude application of the method, only the number of counts for each input-output combination matters, see Equation (8). In particular, the actual trials in which a particular combination of appears are irrelevant in such an analysis.

So, if one holds the view that the nonsignaling principle cannot be flawed, then one must come to the conclusion that “should the measurement choices be randomized, it would be impossible to register the same number of counts for each input-output combination”. A plausible cause for this is that the experimental setup suffered from some systematic drift during the course of the experiment, which is exactly a manifestation that the experimental trials are not i.i.d. It might then appear that a hypothesis test of the nonsignaling principle is hopeless in such a scenario. However, as mentioned above, the prediction-based-ratio method is applicable even for non-i.i.d. experimental trials. Indeed, as we illustrate in Section 3.3 (see, specifically Table 2), such fluctuations have not lead to any false positive in the sense of giving very small p-value upper bound according to the nonsignaling hypothesis.

More generally, as the above example of Section 3.4 illustrates, an unexpectedly small p-value upper bound according to the nonsignaling hypothesis may be a consequence that certain premises needed to perform a sensible Bell test are violated. In other words, an apparent violation as such does not necessarily pose a problem to any physical principle, such as the nonsignaling principle that is rooted in the theory of relativity. However, as nonlocal correlations also find applications in device-independent quantum information processing [18,19], it is important to carry out such consistency checks alongside the violation of a Bell inequality before one applies the estimated nonlocal correlation in any such protocols.

Of course, an unexpectedly small p-value upper bound according to the nonsignaling hypothesis could also be a consequence of mere statistical fluctuation. Indeed, our results in Section 3.2 and Section 3.3 show that when a null hypothesis indeed holds true, it can still happen that one obtains a relatively small p-value upper bound (of the order of ) even after a large number of trials (). However, as explained in Appendix 1 of Ref. [43], if a null hypothesis is correct, the probability of obtaining a p-value upper bound smaller than q with the prediction-based-ratio method is no larger than q. Indeed, in each of these instances, p-value upper bounds that are less than 10 occur way less than 50 times among the 500 simulated experiments. In any case, this means that even though the prediction-based-ratio method already gets rids of the often unjustifiable i.i.d. assumption involved in such an analysis, the interpretation of the significance of a small p-value upper bound must still be carried out with care, as advised, for example, in Refs. [77,78,79].

5. Conclusions

In this work, we revisited the prediction-based-ratio method developed [43]—in the context of a Bell test—for performing hypothesis tests of LHV theories. We showed that with the two observations presented in Section 2.3, the method can equally well be applied to perform hypothesis tests of other physical theories, specifically those that are constrained to produce correlations amenable to a semidefinite programming characterization. Prime examples of such theories include those that obey the principle of nonsignaling [28], those that satisfy the principle of macroscopic locality [51], the so-called v-causal models [67], as well as physical theories that are constrained to produce the almost-quantum set [50] or any other outer approximations [59,62,69] of the quantum set of correlations.

To illustrate the effectiveness of the method, we first numerically simulated 500 Bell tests using a hypothetical source of correlations that lies somewhat outside the almost-quantum set of correlations. We then applied the method to obtain a p-value upper bound according to both the almost-quantum hypothesis and the nonsignaling hypothesis for the simulated data obtained in each of these Bell tests. In the majority () of these 500 instances, the p-value upper bound according to the almost-quantum hypothesis is less than . Since a p-value upper bound quantifies the evidence against the assumed (almost-quantum) theory given the observed data, these results show that in most of these simulated Bell tests, the data is unlikely to be explicable by the assumed theory. In a similar manner, we numerically simulated another 500 Bell tests using a hypothetical source that varies from one trial to another. Again, the method remained very effective (giving a p-value upper bound that is less than for 69% of the instances) in identifying the incompatibility between the observed data and the assumed (almost-quantum) theory in such a non-i.i.d. scenario.

Finally, we applied the prediction-based-ratio method to the experimental data of Ref. [72]. To this end, we assumed that the measurement settings were randomly chosen with uniform distributions. An application of the method under this assumption again led to very small p-value upper bounds () for more than 40% of the 180 Bell tests analyzed—not only for the almost-quantum hypothesis, but also for the nonsignaling hypothesis. Such a violation of the nonsignaling conditions, however, is apparent, as we learned after the analysis that the measurement settings were not randomized during the course of the experiments, thereby invalidating one of the basic assumptions needed in the application of the prediction-based-ratio method. Nonetheless, as we remarked in the Discussion section, the analysis nevertheless unveils that the possibility of using the prediction-based-ratio method to identify a situation where a certain premise is needed to perform a proper Bell test, such as the randomization of settings, is invalidated.

Note added: While preparing this manuscript, we became aware of the work of Smania et al. [80], which also discussed, among others, the implication of not randomizing the settings in a Bell test, and its relevance in quantitative applications.