A P-value model for theoretical power analysis and its applications in multiple testing procedures.

BACKGROUND: Power analysis is a critical aspect of the design of experiments to detect an effect of a given size. When multiple hypotheses are tested simultaneously, multiplicity adjustments to p-values should be taken into account in power analysis. There are a limited number of studies on power analysis in multiple testing procedures. For some methods, the theoretical analysis is difficult and extensive numerical simulations are often needed, while other methods oversimplify the information under the alternative hypothesis. To this end, this paper aims to develop a new statistical model for power analysis in multiple testing procedures.
METHODS: We propose a step-function-based p-value model under the alternative hypothesis, which is simple enough to perform power analysis without simulations, but not too simple to lose the information from the alternative hypothesis. The first step is to transform distributions of different test statistics (e.g., t, chi-square or F) to distributions of corresponding p-values. We then use a step function to approximate each of the p-value's distributions by matching the mean and variance. Lastly, the step-function-based p-value model can be used for theoretical power analysis.
RESULTS: The proposed model is applied to problems in multiple testing procedures. We first show how the most powerful critical constants can be chosen using the step-function-based p-value model. Our model is then applied to the field of multiple testing procedures to explain the assumption of monotonicity of the critical constants. Lastly, we apply our model to a behavioral weight loss and maintenance study to select the optimal critical constants.
CONCLUSIONS: The proposed model is easy to implement and preserves the information from the alternative hypothesis.

Background

Power analysis is a key technique in the experimental design to reveal an effect of a given size. Traditional power calculation usually assumes a single hypothesis test, but it is quite common for researchers to test several hypotheses simultaneously. Clinical trials often require two or more hypotheses to be tested, and studies which involve comparing treatments using multiple outcome measures happen frequently in medical research [8]. The development of high-throughput biology leads to a dramatic increase in the number of hypothesis tests in genomics [20]. However, there are a limited number of studies on power analysis in multiple testing procedures. For scientific studies with multiple hypotheses, in order to correctly control the false positives, multiplicity adjustments to p-values should be taken into account in power analysis. A consequence of multiplicity adjustments is the loss of power [21] and the change in sample size requirements [16].

The p-value is a tail probability given the null hypothesis is true. Under the null hypothesis, the p-value is a uniformly distributed random variable between 0 and 1. If the null hypothesis is false, the p-value’s distribution depends on the alternative hypothesis, which usually satisfies the inequality

i.e., the random variable P is less than a standard uniform random variable in the stochastic order, where the random variable P is the probability of rejecting the null hypothesis when the alternative hypothesis is true.

In order to perform a p-value based power analysis, certain distribution models are needed to describe the behavior of p-values under the alternative hypothesis. In general there are two approaches. The p-value models based on the original test statistics [17] or based on copulas [27] are usually with complex expressions, and further calculations or evaluations require integrations. So the theoretical analysis is difficult and numerical simulations are often needed. The p-value models based on Dirac function [10, 24] are over-simplified, which limits their application areas.

In this paper, we propose the step-function-based p-value models under the alternative hypothesis, which are simple enough to perform theoretical power analysis, but not too simple to lose the information from the alternative hypothesis. Two applications in multiple testing procedures are shown and one application in weight-loss treatment is given.

Methods

A widely used p-value model assumes that the statistic under the null hypothesis follows N(0,12), and the statistics under the alternative hypothesis follows N(δ,12). The p-values are calculated based on one-sided test [17].

The density function of the normal-distribution-based p-value model is

with mean and variance

Alternatively, we propose a step-function-based p-value model under the alternative hypothesis, which has a density function

with mean and variance where 0≤g≤1≤f. The parameter (f,g) indicates the deviation of the random variable P under the step-function-based p-value model from a standard uniform random variable.

For comparison between the normal-distribution-based p-value model and the step-function-based p-value model, the parameters (f,g) of the step-function-based p-value model were chosen to match the means and the variances for the normal-distribution-based p-value model with parameter δ, as shown in Table 1. For the simplified step-function-based p-value model with parameter f, means are matched with the normal model.

Table 1

Normal-distribution-based p-value model and step-function-based p-value model

Normal			Step-function			Simplified step-function
δ	Mean	SD	f	g	(1−g)/(f−g)	f	1/f
0.2	0.444	0.286	1.249	0.795	0.451	1.127	0.888
0.5	0.362	0.274	1.728	0.555	0.380	1.382	0.724
1	0.240	0.236	2.937	0.288	0.269	2.086	0.480
2	0.079	0.130	9.061	0.059	0.105	6.357	0.157
3	0.017	0.049	37.143	0.007	0.027	29.503	0.034

In addition, a simplified step-function-based p-value model with a single parameter f∈[1,+∞) is achieved when assuming g=0. The corresponding density function is

with mean and variance For the simplified step-function-based p-value model, a larger parameter f corresponds to a larger effect size. As a special case, when f=1, the distribution is uniform [0,1]. The probability density functions and the cumulative distribution functions of the normal-distribution-based and the step-function-based p-value models are compared in Fig. 1. The step-function-based p-value models serve an approximation to the p-value models based on the original test statistics.

Fig. 1

Normal-distribution-based and step-function-based p-value models (δ=0.5)

Based on the univariate model, the corresponding multivariate p-value model has a density function

where different h (·)’s may have different parameters f ’s and g ’s. In the following sections, the simplified step-function-based p-value model is applied to two problems in multiple testing procedures.

Results

Application: optimal choices of the critical constants

Assume n hypotheses with p-values . Sort p-values as p (1)≤⋯≤p (, and H (1),⋯,H ( are the corresponding null hypotheses. Consider testing the global null hypothesis under the control of type I error [15]

The global test compares p ( with its corresponding critical constant c α for every i=1,⋯,n. If for some i’s, p (≤c α, then the global null hypothesis is rejected. Simes [25] proposed a test with c =(n−i+1)/n, and other choices of c ’s were proposed by Rom [22], Cai and Sarkar [7], Gou and Tamhane [11].

Among different choices of critical constants c ’s, people usually run simulations [19] or rely on numerical calculations [14] to make power comparison in order to choose suitable sets of critical constants. Besides the existing computationally intensive methods, the step-function-based p-value model is an alternative choice to theoretically calculate the powers and make comparisons between different multiple testing procedures.

In this section we first show how the most powerful critical constants can be chosen using our proposed method for a global test with two hypotheses, and then we can apply the successive recursion process to calculate the power for a global test with n hypotheses. For a global test with two single hypotheses, the control of type I error under independence requires where c 1≥c 2. This equality is equivalent to

From (6) we get , and the derivative , which is less than zero. So c 2 decreases when c 1 increases.

By using the simplified step-function-based p-value model, without loss of generality, we assume 1≤f 1≤f 2, the probability of rejecting the global hypothesis is

When the global hypothesis contains two single hypotheses (n=2), there are two configurations of true and non-true null hypotheses where the global null hypothesis is false: (1) one true null hypothesis (n 0=1) and one false null hypothesis (m=1), and (2) two false null hypotheses (m=2).

First, assume that one null hypothesis is true and the other is false, say, f 1=1 and f 2=f, then the power is

When , from (6) it follows that c 1(c 1−2c 2)α=1−2c 2, therefore hence power increases when c 2 decreases (c 1 increases).

When , from (6) we get , consequently

Note that f α c 1≥1, so power increases when c 1 decreases (c 2 increases).

We calculate the maximal power for different f’s. When we assume the alternative hypothesis has a small effect size, where , the maximal power is achieved when c 2=0 and . When we assume the alternative hypothesis has a moderate effect size, where , the maximal power is achieved when c 1=1/(f α) and c 2=(f 2 α−1)/(2α f(f−1)), so when f increases, we can follow the strategy to decrease c 1 (increase c 2) to achieve the maximal power. When we assume the alternative hypothesis has a large effect size, where , the maximal power is achieved when we choose a test with c 2≥1/(f α), so when f is large enough, different tests have similar power.

Second, assume that both null hypothesis are false with the same effect size, say, f 1=f 2=f, then the power is

When , from (6) we get c 1(c 1−2c 2)α=1−2c 2, then power increases when c 2 decreases (c 1 increases).

We calculate the maximal power for different f values. When we assume both hypotheses are false and with a small effect size, where , the maximal power is achieved when c 2=0 and . When we assume both false hypotheses have a big effect size, where , the maximal power is achieved when the test satisfies c 1≥1/(f α). By taking both and into account, it follows that and c 2=0 is the uniformly best choice when both null hypotheses are false.

In general, for a global test with n single hypotheses, where m of them are true significances (false null hypotheses), define the probabilities as where Pr indicates the probability for n hypotheses, where m is the number of the true significances, and n 0=n−m is the number of the true nulls.

Since we have the recurrence relation for i=n

Similarly, for general i, since

we have the general recurrence relation for i

Finner and Roters [9], Cai and Sarkar [7], and Gou and Tamhane [11] defined a special case of the probability B for m=0 to calculate the type I error under the global null hypothesis. They proved a recurrence relationship among B ’s. We generalize this result to B ’s and have this recurrence relationship (10) under the simplified step-function-based p-value model for power analysis.

By starting from

and using the recurrence relation (10), the power is calculated by

Note that and the control of type I error is satisfied if

When f is specified and the set of critical constants is given, the exact power can be calculated by using (11). Since only arithmetic calculations are needed, the power can be computed very fast.

For theoretical analysis, we consider the situation where f is not too small.

The largest possible c is achieved by using the set of critical constants which satisfies c 1=c 2=⋯=c =c, and c =⋯=c =0. The control of type I error requires that

the largest possible c can be solved from (12). If we only take the leading term, we have an approximate solution

So when the maximal power is achieved when the test satisfies c ≥1/(f α).

Note that when m is relatively large (e.g., more than n/2), the bound is small, and the global tests with large c 1,⋯,c and small c ,⋯,c tend to have large power. Similar observations were reported by Gou and Tamhane [11] based on simulations.

Note that in this application power is simply the probability of rejecting where at least one H is false. For testing multiple hypothesis, powers can be of different types: individual, average, disjunctive, and conjunctive, and the appropriate power concept is determined on a case-by-case basis [4]. These power definitions can also be used in the proposed method by using the step-function-based p-value model.

Power analysis can be complex when multiple hierarchical objectives are involved. Alosh and Huque [1] discussed the power for testing hierarchically ordered endpoints. The step-function-based p-value model can be applied to various clinical trials, e.g., group sequential designs [18], graphical procedures [5, 6].

Application: monotonicity of the critical constants

For multiple test procedures, critical constants are often required to satisfy [7, 12]

This requirement is called the monotonicity assumption of critical constants.

Suppose that we have a set of critical constants , and , so the monotonicity assumption is not satisfied. Note that

So if a test with critical constants controls type I error below α, then another test with critical constants , which satisfies the monotonicity assumption, also controls type I error below α, and has the same power with the previous test which does not satisfy the monotonicity assumption. Hence, only the set of critical constants which satisfies the monotonicity assumption needs to be considered.

Many multiple tests have critical constants which satisfy a strict monotonicity assumption [11, 22, 25]

Some multiple tests satisfy the monotonicity assumption (13), but do not satisfy the strict monotonicity assumption (14) [3, 26]. In general, these tests are not as powerful as the tests which satisfy the strict monotonicity assumption (14) [11]. Our step-function-based p-value models can explain that this assumption is necessary because the corresponding tests are generally more powerful than other tests which do not satisfy this assumption.

For multiple tests with two single hypotheses, by using the simplified step-function-based p-value model, we have several observations: (1) when there is one true null hypothesis and one false null hypothesis, only if , the test which does not satisfy (14) can be more powerful than or as powerful as all the tests which satisfy (14), (2) when there are two false null hypotheses, the test does not satisfy (14) is less powerful than some of the tests which satisfy (14) for all f values. In general, on the parameter space of the effect size (under simplified step-function-based p-value model, the effect size is a function of parameter f), the tests which do not satisfy (14) have more power than all other tests which satisfy (14) only at a zero measure subspace of the effect size. This fact explains that usually people prefer multiple tests which satisfy the strict monotonicity property (14), because these tests are generally more powerful than the tests which do not satisfy (14).

A worked example

Annesi et al. [2] evaluated behavioral weight-loss treatments. They recruited 110 women whose BMI’s are between 30 and 40 kg/m2, and randomly assigned the participants to a comparison treatment with a print manual and telephone follow-ups, or an experimental treatment of the coach approach exercise-support protocol. The self-efficacy for controlled eating (SE-eating) is one of the psychological predictors of behavioral changes. Annesi et al. [2] reported that during the weight-loss phase (month 0-6), the SE-eating increases in the experimental group were significantly greater than the increases in the comparison group with t=2.88, and there was no significant between-group difference during the weight-loss maintenance phase (month 6-24) with t=−0.48.

The increases of the self-efficacy for controlled eating were evaluated both during the weight-loss phase and during the weight-loss maintenance phase, so the multiplicity adjustment is advised to apply. To choose the optimal multiplicity correction based on the estimated effect size from the pilot study, we recommend our proposed step-function-based p-value model because it is easy to implement and preserves the information from the alternative hypothesis. If we take the weight-loss study by Annesi et al. [2] as a pilot study, we have the information that the standardized SE-eating increase during the weight-loss phase is normally distributed with mean δ=2.88 and variance 1, and the increase during the weight-loss maintenance phase is normally distributed with mean δ=0 and variance 1. To match the mean for the normal-distribution-based p-value model with parameter δ=2.88, the parameter f of the simplified step-function-based p-value model is 23.979. From (7) and by using the significance level α=0.05, the maximal power is (1+f 2 α)/(2f)=62 %, and the optimal choice of critical constants is (c 1,c 2)=(1/(f α),(f 2 α−1)/(2α f(f−1)))=(0.8341,0.5036). So the larger p-value is compared with 0.8341α and the smaller p-value is compared with 0.5036α, and if any p-value is less than the corresponding critical value, the global null hypothesis will be rejected.

The step-function-based p-value models for power analysis simplify the theoretical analysis that is difficult in many situations. At the same time, information of loss remains at an acceptable level. Finner and Gontscharuk [10] and Sarkar et al. [24] used a tool called the Dirac–uniform configuration for power analysis, where all p-values under the false null hypotheses follow a Dirac distribution with point mass at 0. When the Dirac-uniform configuration is applied to Annesi et al.’s [2] study, the information of δ is lost, and any choice of positive critical constants (c 1,c 2) will result a claim of significance. Under the p-value model based on Dirac function, all choices of critical constants have the same power, and the optimal choice is unable to be located.

The Dirac-uniform configuration is too brief to include necessary information to choose the optimal critical constants. Hung et al. [17] discussed a p-value model based on normal distribution. By using Annesi et al.’s [2] research as a pilot study, one test statistic is N(δ,12), and the other test statistic is N(0,12). The probability of rejecting the global null hypothesis is

The optimal choice of critical constants is followed by solving

This optimization problem has no explicit solution. When δ=2.88 and α=0.05, the optimal solution is (c 1,c 2)=(1.0076,0.4998). While the step-function based p-value model and the normal-distribution based p-value model produce similar choices of critical constants (c 1,c 2), the model based on step function has an explicit solution of critical constants and requires little computational effort.

Discussion and conclusions

We have given a step-function-based p-value model and its simplified version. These p-value models are simple and concise to perform theoretical power analysis. In addition, different test statistics, for example, t, chi-square or F, can be transformed to the p-value scale. These models can be applied to the field of multiple testing procedures to explain the assumption of monotonicity of the critical constants. We also use these p-value models to choose suitable sets of critical constants with more power. In this paper, we consider the independent p-values for the multivariate cases. Dependence structures can be brought into these p-value models, like Sarkar et al. [23] or Gou and Tamhane [13], and we will report the dependent multivariate p-value models in a separate paper. Finally, there are many applications of the step-function-based p-value models, and multiple testing procedure is an example in point.