John P A Ioannidis1. 1. Clinical and Molecular Epidemiology Unit, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, and Biomedical Research Institute, Foundation for Research and Technology–Hellas, Ioannina, Greece. jioannid@stanford.edu
Abstract
CONTEXT: Many studies report volume abnormalities in diverse brain structures in patients with various mental health conditions. OBJECTIVE: To evaluate whether there is evidence for an excess number of statistically significant results in studies of brain volume abnormalities that suggest the presence of bias in the literature. DATA SOURCES: PubMed (articles published from January 2006 to December 2009). STUDY SELECTION: Recent meta-analyses of brain volume abnormalities in participants with various mental health conditions vs control participants with 6 or more data sets included, excluding voxel-based morphometry. DATA EXTRACTION: Standardized effect sizes were extracted in each data set, and it was noted whether the results were "positive" (P < .05) or not. For each data set in each meta-analysis, I estimated the power to detect at α = .05 an effect equal to the summary effect of the respective meta-analysis. The sum of the power estimates gives the number of expected positive data sets. The expected number of positive data sets can then be compared against the observed number. DATA SYNTHESIS: From 8 articles, 41 meta-analyses with 461 data sets were evaluated (median, 10 data sets per meta-analysis) pertaining to 7 conditions. Twenty-one of the 41 meta-analyses had found statistically significant associations, and 142 of 461 (31%) data sets had positive results. Even if the summary effect sizes of the meta-analyses were unbiased, the expected number of positive results would have been only 78.5 compared with the observed number of 142 (P < .001). CONCLUSION: There are too many studies with statistically significant results in the literature on brain volume abnormalities. This pattern suggests strong biases in the literature, with selective outcome reporting and selective analyses reporting being possible explanations.
CONTEXT: Many studies report volume abnormalities in diverse brain structures in patients with various mental health conditions. OBJECTIVE: To evaluate whether there is evidence for an excess number of statistically significant results in studies of brain volume abnormalities that suggest the presence of bias in the literature. DATA SOURCES: PubMed (articles published from January 2006 to December 2009). STUDY SELECTION: Recent meta-analyses of brain volume abnormalities in participants with various mental health conditions vs control participants with 6 or more data sets included, excluding voxel-based morphometry. DATA EXTRACTION: Standardized effect sizes were extracted in each data set, and it was noted whether the results were "positive" (P < .05) or not. For each data set in each meta-analysis, I estimated the power to detect at α = .05 an effect equal to the summary effect of the respective meta-analysis. The sum of the power estimates gives the number of expected positive data sets. The expected number of positive data sets can then be compared against the observed number. DATA SYNTHESIS: From 8 articles, 41 meta-analyses with 461 data sets were evaluated (median, 10 data sets per meta-analysis) pertaining to 7 conditions. Twenty-one of the 41 meta-analyses had found statistically significant associations, and 142 of 461 (31%) data sets had positive results. Even if the summary effect sizes of the meta-analyses were unbiased, the expected number of positive results would have been only 78.5 compared with the observed number of 142 (P < .001). CONCLUSION: There are too many studies with statistically significant results in the literature on brain volume abnormalities. This pattern suggests strong biases in the literature, with selective outcome reporting and selective analyses reporting being possible explanations.
Authors: Cathy Davies; Andrea Cipriani; John P A Ioannidis; Joaquim Radua; Daniel Stahl; Umberto Provenzani; Philip McGuire; Paolo Fusar-Poli Journal: World Psychiatry Date: 2018-06 Impact factor: 49.548
Authors: Katarzyna Jednoróg; Artur Marchewka; Irene Altarelli; Ana Karla Monzalvo Lopez; Muna van Ermingen-Marbach; Marion Grande; Anna Grabowska; Stefan Heim; Franck Ramus Journal: Hum Brain Mapp Date: 2015-01-17 Impact factor: 5.038
Authors: Margaret Quinn; Maureen McHugo; Kristan Armstrong; Neil Woodward; Jennifer Blackford; Stephan Heckers Journal: Psychiatry Res Neuroimaging Date: 2018-08-03 Impact factor: 2.376
Authors: Katherine S Button; John P A Ioannidis; Claire Mokrysz; Brian A Nosek; Jonathan Flint; Emma S J Robinson; Marcus R Munafò Journal: Nat Rev Neurosci Date: 2013-04-10 Impact factor: 34.870
Authors: Ardesheer Talati; Spiro P Pantazatos; Franklin R Schneier; Myrna M Weissman; Joy Hirsch Journal: Biol Psychiatry Date: 2012-06-29 Impact factor: 13.382
Authors: Sander V Haijma; Neeltje Van Haren; Wiepke Cahn; P Cédric M P Koolschijn; Hilleke E Hulshoff Pol; René S Kahn Journal: Schizophr Bull Date: 2012-10-05 Impact factor: 9.306