Simon Klau1,2, Sabine Hoffmann1,3, Chirag J Patel4, John Pa Ioannidis5,6,7,8,9, Anne-Laure Boulesteix1,3. 1. Institute for Medical Information Processing, Biometry, and Epidemiology, Ludwig-Maximilians-Universität München, Munich, Germany. 2. Leibniz Institute for Prevention Research and Epidemiology-BIPS, Bremen, Germany. 3. LMU Open Science Center, Ludwig-Maximilians-Universität München, Munich, Germany. 4. Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA. 5. Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA, USA. 6. Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, USA. 7. Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, USA. 8. Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA, USA. 9. Department of Medicine, Stanford University School of Medicine, Stanford, CA, USA.
Abstract
BACKGROUND: The results of studies on observational associations may vary depending on the study design and analysis choices as well as due to measurement error. It is important to understand the relative contribution of different factors towards generating variable results, including low sample sizes, researchers' flexibility in model choices, and measurement error in variables of interest and adjustment variables. METHODS: We define sampling, model and measurement uncertainty, and extend the concept of vibration of effects in order to study these three types of uncertainty in a common framework. In a practical application, we examine these types of uncertainty in a Cox model using data from the National Health and Nutrition Examination Survey. In addition, we analyse the behaviour of sampling, model and measurement uncertainty for varying sample sizes in a simulation study. RESULTS: All types of uncertainty are associated with a potentially large variability in effect estimates. Measurement error in the variable of interest attenuates the true effect in most cases, but can occasionally lead to overestimation. When we consider measurement error in both the variable of interest and adjustment variables, the vibration of effects are even less predictable as both systematic under- and over-estimation of the true effect can be observed. The results on simulated data show that measurement and model vibration remain non-negligible even for large sample sizes. CONCLUSION: Sampling, model and measurement uncertainty can have important consequences for the stability of observational associations. We recommend systematically studying and reporting these types of uncertainty, and comparing them in a common framework.
BACKGROUND: The results of studies on observational associations may vary depending on the study design and analysis choices as well as due to measurement error. It is important to understand the relative contribution of different factors towards generating variable results, including low sample sizes, researchers' flexibility in model choices, and measurement error in variables of interest and adjustment variables. METHODS: We define sampling, model and measurement uncertainty, and extend the concept of vibration of effects in order to study these three types of uncertainty in a common framework. In a practical application, we examine these types of uncertainty in a Cox model using data from the National Health and Nutrition Examination Survey. In addition, we analyse the behaviour of sampling, model and measurement uncertainty for varying sample sizes in a simulation study. RESULTS: All types of uncertainty are associated with a potentially large variability in effect estimates. Measurement error in the variable of interest attenuates the true effect in most cases, but can occasionally lead to overestimation. When we consider measurement error in both the variable of interest and adjustment variables, the vibration of effects are even less predictable as both systematic under- and over-estimation of the true effect can be observed. The results on simulated data show that measurement and model vibration remain non-negligible even for large sample sizes. CONCLUSION: Sampling, model and measurement uncertainty can have important consequences for the stability of observational associations. We recommend systematically studying and reporting these types of uncertainty, and comparing them in a common framework.
Authors: N E Day; M Y Wong; S Bingham; K T Khaw; R Luben; K B Michels; A Welch; N J Wareham Journal: Int J Epidemiol Date: 2004-08-27 Impact factor: 7.196
Authors: Stylianos Serghiou; Chirag J Patel; Yan Yu Tan; Peter Koay; John P A Ioannidis Journal: J Clin Epidemiol Date: 2015-09-28 Impact factor: 6.437