Scott M Bartell1,2,3. 1. Program in Public Health, Susan and Henry Samueli College of Health Sciences, University of California Irvine, 2032 Anteater Instruction & Research Building, Irvine, CA, 92697-3957, USA. sbartell@uci.edu. 2. Department of Statistics, Donald Bren School of Information and Computer Sciences, University of California Irvine, Irvine, CA, USA. sbartell@uci.edu. 3. Department of Epidemiology, School of Medicine, Susan and Henry Samueli College of Health Sciences, University of California Irvine, Irvine, CA, USA. sbartell@uci.edu.
Abstract
PURPOSE OF REVIEW: In recent years, investigators in a variety of fields have reported that most published findings can not be replicated. This review evaluates the factors contributing to lack of reproducibility, implications for environmental epidemiology, and strategies for mitigation. RECENT FINDINGS: Although publication bias and other types of selective reporting may contribute substantially to irreproducible results, underpowered analyses and low prevalence of true associations likely explain most failures to replicate novel scientific results. Epidemiologists can counter these risks by ensuring that analyses are well-powered or precise, focusing on scientifically justified hypotheses, strictly controlling type I error rates, emphasizing estimation over statistical significance, avoiding practices that introduce bias, or employing bias analysis and triangulation. Avoidance of p values has no effect on reproducibility if confidence intervals excluding the null are emphasized in a similar manner. Increased attention to exposure mixtures and susceptible subpopulations, and wider use of omics technologies, will likely decrease the proportion of investigated associations that are true associations, requiring greater caution in study design, analysis, and interpretation. Though well intentioned, these recent trends in environmental epidemiology will likely decrease reproducibility if no effective actions are taken to mitigate the risk of spurious findings.
PURPOSE OF REVIEW: In recent years, investigators in a variety of fields have reported that most published findings can not be replicated. This review evaluates the factors contributing to lack of reproducibility, implications for environmental epidemiology, and strategies for mitigation. RECENT FINDINGS: Although publication bias and other types of selective reporting may contribute substantially to irreproducible results, underpowered analyses and low prevalence of true associations likely explain most failures to replicate novel scientific results. Epidemiologists can counter these risks by ensuring that analyses are well-powered or precise, focusing on scientifically justified hypotheses, strictly controlling type I error rates, emphasizing estimation over statistical significance, avoiding practices that introduce bias, or employing bias analysis and triangulation. Avoidance of p values has no effect on reproducibility if confidence intervals excluding the null are emphasized in a similar manner. Increased attention to exposure mixtures and susceptible subpopulations, and wider use of omics technologies, will likely decrease the proportion of investigated associations that are true associations, requiring greater caution in study design, analysis, and interpretation. Though well intentioned, these recent trends in environmental epidemiology will likely decrease reproducibility if no effective actions are taken to mitigate the risk of spurious findings.
Keywords:
False discovery rate; False positive; Family-wise error rate; Hypothesis testing; Reliability; Reproducibility; Type I error; p value
Authors: Timothy L Lash; Matthew P Fox; Richard F MacLehose; George Maldonado; Lawrence C McCandless; Sander Greenland Journal: Int J Epidemiol Date: 2014-07-30 Impact factor: 7.196
Authors: Judy S LaKind; Michael Goodman; Susan L Makris; Donald R Mattison Journal: J Toxicol Environ Health B Crit Rev Date: 2015 Impact factor: 6.393