Seriously misleading results using inverse of Freeman-Tukey double arcsine transformation in meta-analysis of single proportions.

Standard generic inverse variance methods for the combination of single proportions are based on transformed proportions using the logit, arcsine, and Freeman-Tukey double arcsine transformations. Generalized linear mixed models are another more elaborate approach. Irrespective of the approach, meta-analysis results are typically back-transformed to the original scale in order to ease interpretation. Whereas the back-transformation of meta-analysis results is straightforward for most transformations, this is not the case for the Freeman-Tukey double arcsine transformation, albeit possible. In this case study with five studies, we demonstrate how seriously misleading the back-transformation of the Freeman-Tukey double arcsine transformation can be. We conclude that this transformation should only be used with special caution for the meta-analysis of single proportions due to potential problems with the back-transformation. Generalized linear mixed models seem to be a promising alternative.

INTRODUCTION

A key application of meta‐analytical methods is the pooling of proportions, such as prevalence of a specific infection or disease.(1, 2, 3, 4) Classic fixed‐effect and random‐effects meta‐analysis methods5 are typically used to combine single proportions. In order to use these methods, proportions are generally transformed using either the log,6 logit,7 arcsine,8 or Freeman‐Tukey double arcsine9 transformations. These transformations are implemented for pure mathematical reasons, eg,  variance stabilization (details on the transformations are given in Appendix A and summarized in Table A1). For pooling, the transformed proportions and corresponding standard errors are used in the generic inverse variance method.5 An alternative yet more elaborate approach based on the logit transformation are generalized linear mixed models (GLMMs),10 which account for the binomial structure of the data and thus avoid the generic inverse variance method. Irrespective of the meta‐analysis method and transformation, results are usually presented on the original probability scale after using the corresponding back‐transformation.

Table A1

Definition and properties of prevalence transformations with number of events a and total sample size n

		Approximate
Transformation	Estimate	Variance	Comments
log6	log(a/n)	1a−1n	Infinite estimate and variance for zero events
logit7	loga/n1−a/n	1a+1n−a	Infinite estimate and variance for zero or all events
arcsine8	arcsina/n	14n	Variance stabilizing; defined for zero events
Double arcsine9	0.5arcsina/(n+1)+	14n+2	Outperforms arcsine for small prevalences;
	arcsin(a+1)/(n+1)		sample size needed in back‐transformation11

Whereas the back‐transformation of meta‐analysis results is straightforward for the log, logit, and arcsine transformations, this is not the case for the Freeman‐Tukey double arcsine transformation, albeit possible.11 In order to calculate the inverse of the Freeman‐Tukey double arcsine transformation, a single sample size has to be specified. Accordingly, for a single study, a one to one relation exists between transformation and its inverse, however, in a meta‐analysis with different sample sizes the value of the back‐transformation depends on the specified sample size. Typically, the harmonic mean of sample sizes is used in the back‐transformation.11

CASE STUDY: META‐ANALYSIS ON PREVALENCE OF HEPATITIC C VIRUS INFECTIONS

We report results of meta‐analyses with five studies estimating the prevalence of hepatitis C virus (HCV) infections in the general population of Nepal, which constitute a subset of an unpublished dataset with 28 studies.12 This unpublished dataset comprises testing for a total of 972 123 individuals among whom 3696 were HCV antibody positive. The prevalence across studies ranged from 0% to 18.4% with a median of 0.5%. We restrict ourselves to the five‐study subset for didactic reasons; the same issues encountered in this subset also exist in the full dataset.

We conducted classic meta‐analyses using the arcsine, Freeman‐Tukey double arcsine, and logit transformations, respectively. Furthermore, we fitted GLMMs implicitly using the logit transformation. Details on the statistical methods are provided in Appendix A. We used R function metaprop() from R package meta 13 (see Supporting Information). Results are summarized in Table 1.

Table 1

Estimates and 95% confidence intervals of HCV prevalence meta‐analyses using arcsine, Freeman‐Tukey double arcsine, and logit transformations, respectively

Transformation	Transformed	HCV Infections
(Meta‐analysis Model)	Proportion	per 1000 Observations
Arcsine (fixed)	0.044 (0.042 to 0.046)	1.94 (1.77 to 2.13)
Double arcsine (fixed)	0.044 (0.042 to 0.046)	0.00 (0.00 to 0.00)
Logit (fixed)	−6.231 (−6.323 to −6.139)	1.96 (1.79 to 2.15)
GLMM (fixed)	−6.238 (−6.330 to −6.147)	1.95 (1.78 to 2.14)
Arcsine (random, τ^=0.0003)	0.044 (0.042 to 0.046)	1.94 (1.76 to 2.13)
Double arcsine (random, τ^=0.0020)	0.044 (0.041 to 0.048)	0.00 (0.00 to 0.00)
Logit (random, τ^=1.1758)	−5.451 (−6.649 to −4.254)	4.27 (1.29 to 14.01)
GLMM (random, τ^=0.0000)	−6.238 (−6.330 to −6.147)	1.95 (1.78 to 2.14)

Note. GLMM (fixed) = logistic regression; GLMM (random) = random intercept logistic regression; between‐study variance estimate .

Abbreviations: GLMM, generalized linear mixed model; HCV, hepatitis C virus.

Under the fixed‐effect model, results depicted as transformed proportions (middle column in Table 1) are very similar for the two methods using the arcsine and logit transformations, respectively. Whereas the random‐effects estimates are also very similar with a slightly smaller confidence interval for the arcsine transformation, the results for the two logit methods are rather different due to a very different estimate for the between‐study variance.

For easier interpretation, results are back‐transformed to the original scale. Due to the small prevalences, we express results as HCV infections per 1000 observations. In Table 1 (right column), the results using the inverse of the Freeman‐Tukey double arcsine transformation based on the harmonic mean of 85 are highly irregular with HCV prevalences and confidence limits exactly equal to zero. Under the fixed‐effect model, all of the other three methods show very similar results. Conversely, under the random‐effects model, results for the classic meta‐analysis method using the logit transformation are very different from the other results.

Looking at Figure 1, we see that the meta‐analysis estimators are reasonable summaries of transformed prevalences. On the other hand, back‐transformed meta‐analysis results are clearly off the mark in Figure 2 with meta‐analysis estimators smaller than all individual study results. Note that the back‐transformation works as expected for individual study results, eg, the prevalence is 1/29 = 0.03448 for study 26, which corresponds to 34.48 HCV infections per 1000 observations.

Figure 1

Forest plot of hepatitis C virus (HCV) meta‐analysis with Freeman‐Tukey double arcsine transformation and without back‐transformation of results. PFT, Freeman‐Tukey double arcsine transformed proportion

Figure 2

Forest plot of hepatitis C virus (HCV) meta‐analysis with Freeman‐Tukey double arcsine transformation and back‐transformation according to Miller11

The harmonic mean of 85 is obviously the wrong choice in this meta‐analysis with sample sizes ranging from 29 to more than 200 000. Figure 3 shows the influence of sample size on meta‐analysis results (see also Table A2). For sample sizes between 10 and around 120, results are exactly zero for the back‐transformation of the Freeman‐Tukey double arcsine transformation. The number of HCV infections per 1000 observations then steeply increases up to a sample size of 500 when the effect of sample size starts to slowly level out.

Figure 3

Influence of sample size on results of hepatitis C virus (HCV) meta‐analysis using inverse of Freeman‐Tukey double arcsine transformation according to Miller11

Table A2

Estimated number of HCV infections per 1000 observations for additional sample sizes in fixed‐effect and random‐effects meta‐analyses using the back‐transformation of the Freeman‐Tukey double arcsine method

	HCV Infections per 1000 Observations
Sample Size	Fixed Effect	Random Effects	Mean
85	0.000	0.000	Harmonic
500	1.083	1.097
1000	1.486	1.500
1254	1.575	1.590	Geometric
10 000	1.902	1.917
46 892	1.941	1.956	Arithmetic
100 000	1.947	1.962
1 000 000	1.951	1.966

As noted earlier, the results of the random‐effects model are very different for the two logit methods due to different between‐study variance estimates. This discrepancy can be explained by looking at the confidence intervals of individual studies in the corresponding forest plots (Figures 4 and 5). Confidence intervals, based on the normal approximation, are much narrower for the two smallest studies in the classic random‐effects meta‐analysis (Figure 4) than the confidence intervals, based on the Clopper‐Pearson method taking the binomial distribution into account,(14, 15) in the GLMM meta‐analysis (Figure 5). Apparently, in these two small studies with only 1 HCV infection and less than 50 observations, the assumption of a normally distributed logit transformed proportion is not fulfilled. With increasing numbers of infections and sample sizes, approximate and Clopper‐Pearson confidence intervals get closer to each other. Obviously, the very narrow confidence intervals of the two smallest studies result in an inflated between‐study variance estimate leading to a larger estimate for the pooled mean HCV prevalence and a much wider confidence interval for the pooled mean HCV prevalence.

Figure 4

Forest plot of hepatitis C virus (HCV) meta‐analysis using classic method and logit transformation. Confidence intervals for individual studies are based on normal approximation for logit transformed proportions

Figure 5

Forest plot of hepatitis C virus (HCV) meta‐analysis using generalized linear mixed model. Confidence intervals for individual studies are based on Clopper‐Pearson method(14, 15)

DISCUSSION

Our case study shows that meta‐analysis results based on the back‐transformation of the Freeman‐Tukey double arcsine transformation11 can be very misleading and even smaller than all individual study results. We observe similar undesirable results in a meta‐analysis using the complete dataset with 28 studies. To our knowledge, this is the first publication reporting such an anomaly and erratic results.

In our view, the main reason for this unexpected behaviour is the very extreme pattern of sample sizes that range from 29 to more than 200 000. The harmonic mean of 85 is much smaller than 3 of the 5 sample sizes. For such highly skewed sample sizes, the harmonic mean is by definition rather small, which may result in nonsensical back‐transformed probabilities.

In order to prevent misleading conclusions for the Freeman‐Tukey double arcsine transformation, several sample sizes could be used to evaluate the sensitivity of meta‐analysis results; however, this may lead to diverging meta‐analysis estimates. In our example, using the arithmetic or geometric mean in the back‐transformation (see Table A2) would result in random‐effects estimates of 1.96 and 1.59 HCV infections per 1000 observations, respectively. Here, results for the harmonic mean are obviously wrong; however, it is rather unclear whether to rely on the results for the arithmetic or geometric mean. All other transformations (arcsine, logit, and log) do not have this intrinsic problem in the presentation of meta‐analysis results.

Overall, the arcsine transformation appears to be the best classic method for the meta‐analysis of single proportions. However, as application of GLMMs for meta‐analysis is nowadays straightforward due to its implementation in common software, there is neither a real reason nor a clear advantage for using an approximate method. Accordingly, we support the viewpoint of  previous works,(10, 16, 17, 18)  recommending the use of GLMMs for the meta‐analysis of single proportions. From our perspective, the only disadvantage of a GLMM is that individual study weights are not available, which we consider as a minor drawback; analysts seeing this differently should use the arcsine transformation.

Our recommendation is purportedly in contrast to advice by Barendregt et al1 promoting the use of the Freeman‐Tukey double arcsine transformation over the logit transformation. However, this publication only considered these transformations under the classic meta‐analysis model. We agree with Barendregt et al1 that the use of the logit transformation is problematic in inverse variance meta‐analyses with small event numbers or sample sizes; this is also visible in our example. These problems with the logit transformation under the classic meta‐analysis do not translate to GLMMs. The classic meta‐analysis model assumes that treatment estimates of individual studies follow a normal distribution that is obviously critical in studies with small numbers of events and observations. The arcsine and Freeman‐Tukey double arcsine transformation are less affected by this normality assumption than the logit transformation. However, GLMMs taking into account the binomial structure of the data are not affected by this problem at all.(10, 16)

CONCLUSIONS

Our case study shows that the Freeman‐Tukey double arcsine transformation should only be used with special caution for the meta‐analysis of single proportions due to potential problems in the back‐transformation of meta‐analysis results. In our view, a sensitivity analysis using other sample sizes is mandatory for this transformation. GLMMs seem to be a promising alternative which is nowadays available in common meta‐analysis software.

CONFLICT OF INTEREST

The author reported no conflict of interest.

Supporting info item

Click here for additional data file.

Supporting info item

Click here for additional data file.