Chirag J Patel1, Belinda Burford2, John P A Ioannidis3. 1. Department of Biomedical Informatics, Harvard Medical School, 10 Shattuck St., Room 314A, Boston, MA 02115, USA. 2. Melbourne School of Population and Global Health, Level 4, 207 Bouverie St., The University of Melbourne, Victoria 3010, Australia. 3. Department of Biomedical Informatics, Harvard Medical School, 10 Shattuck St., Room 314A, Boston, MA 02115, USA; Department of Medicine, Stanford Prevention Research Center, Stanford University School of Medicine, Medical School Office Building, Room X306, 1265 Welch Rd, Stanford, CA 94305, USA; Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA 94305, USA; Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA 94305, USA; Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA 94305, USA. Electronic address: jioannid@stanford.edu.
Abstract
OBJECTIVES: Model specification-what adjusting variables are analytically modeled-may influence results of observational associations. We present a standardized approach to quantify the variability of results obtained with choices of adjustments called the "vibration of effects" (VoE). STUDY DESIGN AND SETTING: We estimated the VoE for 417 clinical, environmental, and physiological variables in association with all-cause mortality using National Health and Nutrition Examination Survey data. We selected 13 variables as adjustment covariates and computed 8,192 Cox models for each of 417 variables' associations with all-cause mortality. RESULTS: We present the VoE by assessing the variance of the effect size and in the -log10(P-value) obtained by different combinations of adjustments. We present whether there are multimodality patterns in effect sizes and P-values and the trajectory of results with increasing adjustments. For 31% of the 417 variables, we observed a Janus effect, with the effect being in opposite direction in the 99th versus the 1st percentile of analyses. For example, the vitamin E variant α-tocopherol had a VoE that indicated higher and lower risk for mortality. CONCLUSION: Estimating VoE offers empirical estimates of associations are under different model specifications. When VoE is large, claims for observational associations should be very cautious.
OBJECTIVES: Model specification-what adjusting variables are analytically modeled-may influence results of observational associations. We present a standardized approach to quantify the variability of results obtained with choices of adjustments called the "vibration of effects" (VoE). STUDY DESIGN AND SETTING: We estimated the VoE for 417 clinical, environmental, and physiological variables in association with all-cause mortality using National Health and Nutrition Examination Survey data. We selected 13 variables as adjustment covariates and computed 8,192 Cox models for each of 417 variables' associations with all-cause mortality. RESULTS: We present the VoE by assessing the variance of the effect size and in the -log10(P-value) obtained by different combinations of adjustments. We present whether there are multimodality patterns in effect sizes and P-values and the trajectory of results with increasing adjustments. For 31% of the 417 variables, we observed a Janus effect, with the effect being in opposite direction in the 99th versus the 1st percentile of analyses. For example, the vitamin E variant α-tocopherol had a VoE that indicated higher and lower risk for mortality. CONCLUSION: Estimating VoE offers empirical estimates of associations are under different model specifications. When VoE is large, claims for observational associations should be very cautious.
Authors: Anastassios G Pittas; Mei Chung; Thomas Trikalinos; Joanna Mitri; Michael Brendel; Kamal Patel; Alice H Lichtenstein; Joseph Lau; Ethan M Balk Journal: Ann Intern Med Date: 2010-03-02 Impact factor: 25.391
Authors: Margaret E Wright; Karla A Lawson; Stephanie J Weinstein; Pirjo Pietinen; Philip R Taylor; Jarmo Virtamo; Demetrius Albanes Journal: Am J Clin Nutr Date: 2006-11 Impact factor: 7.045
Authors: Chirag J Patel; David H Rehkopf; John T Leppert; Walter M Bortz; Mark R Cullen; Glenn M Chertow; John Pa Ioannidis Journal: Int J Epidemiol Date: 2013-12-16 Impact factor: 7.196
Authors: Paolo Boffetta; Joseph K McLaughlin; Carlo La Vecchia; Robert E Tarone; Loren Lipworth; William J Blot Journal: J Natl Cancer Inst Date: 2008-07-08 Impact factor: 13.506
Authors: Sinan Guloksuz; Bart P F Rutten; Lotta-Katrin Pries; Margreet Ten Have; Ron de Graaf; Saskia van Dorsselaer; Boris Klingenberg; Jim van Os; John P A Ioannidis Journal: Schizophr Bull Date: 2018-10-17 Impact factor: 9.306
Authors: Arjun K Manrai; Chirag J Patel; Nils Gehlenborg; Nicholas P Tatonetti; John P A Ioannidis; Isaac S Kohane Journal: Pac Symp Biocomput Date: 2016
Authors: Aaron R Caldwell; Andrew D Vigotsky; Matthew S Tenan; Rémi Radel; David T Mellor; Andreas Kreutzer; Ian M Lahart; John P Mills; Matthieu P Boisgontier Journal: Sports Med Date: 2020-03 Impact factor: 11.136
Authors: Ryan D Ross; Xu Shi; Megan E V Caram; Pheobe A Tsao; Paul Lin; Amy Bohnert; Min Zhang; Bhramar Mukherjee Journal: Health Serv Outcomes Res Methodol Date: 2020-10-20
Authors: Stylianos Serghiou; Chirag J Patel; Yan Yu Tan; Peter Koay; John P A Ioannidis Journal: J Clin Epidemiol Date: 2015-09-28 Impact factor: 6.437
Authors: Lingzhi Chu; John P A Ioannidis; Alex C Egilman; Vasilis Vasiliou; Joseph S Ross; Joshua D Wallach Journal: Int J Epidemiol Date: 2020-04-01 Impact factor: 7.196