Statistical notes for clinical researchers: Sample size calculation 2. Comparison of two independent proportions.

Principles and methods of sample size determination for the comparison of two independent means were introduced in the previous article.1 In this article we will discuss sample size determination for the comparison of two independent proportions. While the same principles about power, α error, and β error can be applied as well, the selection of an adequate test is needed for the comparison of two independent proportions, depending on the condition whether the sample size is large enough or not.

Selection of two-sample test for proportions

The comparison of two independent proportions occurs when we perform an experiment with a binary outcome (e.g., success or failure) obtained from two groups (e.g., control vs. experiment groups). Let's call the sample sizes and proportions of success in the control and experimental groups as n1, n2, p1, and p2, respectively. When the sample size of an experiment is large enough (e.g., n1p1 > 5, and n2p2 > 5), we can assume the sampling distribution of proportions as approximately normal based on the central limit theorem. Therefore, when sample size is large enough, we can apply z test approximation in comparing two independent proportions and the same principle can be applied in the required sample size calculation.

The comparison of two proportions can be expressed as a 2 × 2 contingency table. When we test the association of two binary variables (two groups and two outcomes), we usually apply the Pearson chi-square test. When the sample size is large enough, the result of chi-square test and z test approximation are equally reliable and show the same result with an identical p value. If the condition satisfies for the application of the Pearson chi-square test, use of z test approximation in sample size calculation is justified.

However, if there is one or more cells with expected frequencies smaller than 5, statistical packages display the warning message that the Pearson chi-square test results cannot be reliable. In those cases, we need to apply the Fisher's exact test instead of the Pearson chi-square test. The Fisher's exact test is applicable when the sample size is small and we could expect that there may be one or more small cells (< 5). Therefore, different sample size calculation procedure following the Fisher's exact test will be applied when the sample size is small.

Calculation of sample size

1. Z test approximation for large sample test

When we suppose the sample size is large enough, following sample size calculation procedure can be performed based on z test approximation. Let's assume the following conditions: proportion of control group (p) = 0.2, proportion of experimental group (p) = 0.3. If we set α error level (two-sided) = 0.05 (corresponding zα/2 = 1.96), power level = 0.8 (corresponding zβ = 0.84), and allocation ratio ( n1 / n2= κ) = 1, the sample size is calculated as approximately 291 subjects per group using the following formula.2

The calculation procedure can be accomplished using the software G*Power. G*Power displays a similar result of sample size, 294 (Example 1).

2. Fisher's exact test for small samples test

When we expect the sample size is small and there may be one or more small cells (< 5), following sample size calculation procedure can be performed based on the Fisher's exact test. Let's assume the following conditions: proportion of control group (p1) = 0.05, and proportion of experimental group (p2) = 0.3. We set α error level (two-sided) = 0.05 (corresponding zα/2 = 1.96), power level = 0.8 (corresponding zβ = 0.84), and allocation ratio (n1/n2 = κ) = 1. Because there is no definite formula related to the case, we need to use a software for sample size determination. Table 1 shows exemplary results applying various levels of proportions, α error, and power.

Table 1

Results of sample size determination for Fisher's exact test (two-sided)

		α = 0.05		α = 0.01
p1	p2	Power = 0.8	Power = 0.9	Power = 0.8	Power = 0.9
0.05	0.30	39	51	56	68
	0.40	24	31	35	42
	0.50	17	21	23	28
0.10	0.30	69	89	98	123
	0.40	36	47	51	64
	0.50	23	29	33	40

Sample size determination procedure using G*Power

An appropriate sample size for the comparison of proportions can be obtained by performing the following steps using G*Power, a free software.

1. Z test approximation for two independent proportions (Large sample test)

Step 1: Select statistical test types

Menu: Tests - Proportions - Two Independent Groups: Inequality, z-Test

Step 2: Select one-sided or two-sided test

Step 3: Select α error level

Step 4: Select power level

Step 5: Select allocation ratio

Step 6: Calculation of sample size

Menu: Calculate

Example 1) Two tailed test Proportions:

2. Fisher exact test for two independent proportions (Small sample test)

Step 1: Select statistical test types

Menu: Tests - Proportions - Two Independent Groups: Inequality, Fisher's exact test

Step 2 - Step 5 (the same as example 1)

Step 6: Calculation of sample size

Menu: Calculate

Example 2) Two tailed test Proportions: p