John P A Ioannidis1, Thomas A Trikalinos, Elias Zintzaras. 1. Clinical Trials and Evidence-based Medicine Unit, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina 45110, Greece. jioannid@cc.uoi.gr
Abstract
OBJECTIVES: Meta-analyses are routinely evaluated for the presence of large between-study heterogeneity. We examined whether it is also important to probe whether there is extreme between-study homogeneity. STUDY DESIGN: We used heterogeneity tests with left-sided statistical significance for inference and developed a Monte Carlo simulation test for testing extreme homogeneity in risk ratios across studies, using the empiric distribution of the summary risk ratio and heterogeneity statistic. A left-sided P=0.01 threshold was set for claiming extreme homogeneity to minimize type I error. RESULTS: Among 11,803 meta-analyses with binary contrasts from the Cochrane Library, 143 (1.21%) had left-sided P-value <0.01 for the asymptotic Q statistic and 1,004 (8.50%) had left-sided P-value <0.10. The frequency of extreme between-study homogeneity did not depend on the number of studies in the meta-analyses. We identified examples where extreme between-study homogeneity (left-sided P-value <0.01) could result from various possibilities beyond chance. These included inappropriate statistical inference (asymptotic vs. Monte Carlo), use of a specific effect metric, correlated data or stratification using strong predictors of outcome, and biases and potential fraud. CONCLUSION: Extreme between-study homogeneity may provide useful insights about a meta-analysis and its constituent studies.
OBJECTIVES: Meta-analyses are routinely evaluated for the presence of large between-study heterogeneity. We examined whether it is also important to probe whether there is extreme between-study homogeneity. STUDY DESIGN: We used heterogeneity tests with left-sided statistical significance for inference and developed a Monte Carlo simulation test for testing extreme homogeneity in risk ratios across studies, using the empiric distribution of the summary risk ratio and heterogeneity statistic. A left-sided P=0.01 threshold was set for claiming extreme homogeneity to minimize type I error. RESULTS: Among 11,803 meta-analyses with binary contrasts from the Cochrane Library, 143 (1.21%) had left-sided P-value <0.01 for the asymptotic Q statistic and 1,004 (8.50%) had left-sided P-value <0.10. The frequency of extreme between-study homogeneity did not depend on the number of studies in the meta-analyses. We identified examples where extreme between-study homogeneity (left-sided P-value <0.01) could result from various possibilities beyond chance. These included inappropriate statistical inference (asymptotic vs. Monte Carlo), use of a specific effect metric, correlated data or stratification using strong predictors of outcome, and biases and potential fraud. CONCLUSION: Extreme between-study homogeneity may provide useful insights about a meta-analysis and its constituent studies.
Authors: Elias Zintzaras; Katrin Uhlig; George N Koukoulis; Afroditi A Papathanasiou; Ioannis Stefanidis Journal: J Hum Genet Date: 2007-09-06 Impact factor: 3.172
Authors: Jørn Wetterslev; Christian S Meyhoff; Lars N Jørgensen; Christian Gluud; Jane Lindschou; Lars S Rasmussen Journal: Cochrane Database Syst Rev Date: 2015-06-25