Georgia Salanti1, John P A Ioannidis. 1. The Clinical Trials and Evidence-Based Medicine Unit, Department of Hygiene and Epidemiology, University of Ioannina, School of Medicine, Ioannina, Greece.
Abstract
OBJECTIVE: Meta-analyses of observational studies often get spuriously precise results. We aimed to factor this skepticism in meta-analysis calculations. STUDY DESIGN AND SETTING: We developed a simple sensitivity analysis starting from the assumption that any single observational study cannot give us more than a maximum certainty c% (called credibility ceiling) that an effect is in a particular direction and not in the other. Each study included in meta-analysis is adjusted for different credibility ceilings c and the consistency of the conclusion examined. We applied the method in three meta-analyses of observational studies with nominally statistically significant summary effects (mortality with teaching versus nonteaching health care; risk of non-Hodgkin's lymphoma with hair dyes; mortality with omega-3 fatty acids). RESULTS: Between-study heterogeneity I(2) estimates dropped from 36%-72% without a ceiling effect to 0% with ceilings of 9%, 4%, and 4% in the three meta-analyses, respectively. Nominal statistical significance was lost with ceilings of 10%, 8%, and 11%, respectively. The likelihood ratios suggested that even with minimal ceiling effects, there was no strong support for the credibility of each of these three associations. CONCLUSIONS: Consideration of credibility ceilings allows conservative interpretation of observational evidence and can be applied routinely to meta-analyses of observational studies.
OBJECTIVE: Meta-analyses of observational studies often get spuriously precise results. We aimed to factor this skepticism in meta-analysis calculations. STUDY DESIGN AND SETTING: We developed a simple sensitivity analysis starting from the assumption that any single observational study cannot give us more than a maximum certainty c% (called credibility ceiling) that an effect is in a particular direction and not in the other. Each study included in meta-analysis is adjusted for different credibility ceilings c and the consistency of the conclusion examined. We applied the method in three meta-analyses of observational studies with nominally statistically significant summary effects (mortality with teaching versus nonteaching health care; risk of non-Hodgkin's lymphoma with hair dyes; mortality with omega-3 fatty acids). RESULTS: Between-study heterogeneity I(2) estimates dropped from 36%-72% without a ceiling effect to 0% with ceilings of 9%, 4%, and 4% in the three meta-analyses, respectively. Nominal statistical significance was lost with ceilings of 10%, 8%, and 11%, respectively. The likelihood ratios suggested that even with minimal ceiling effects, there was no strong support for the credibility of each of these three associations. CONCLUSIONS: Consideration of credibility ceilings allows conservative interpretation of observational evidence and can be applied routinely to meta-analyses of observational studies.
Authors: Tai Lim Kim; Gwang Hun Jeong; Jae Won Yang; Keum Hwa Lee; Andreas Kronbichler; Hans J van der Vliet; Giuseppe Grosso; Fabio Galvano; Dagfinn Aune; Jong Yeob Kim; Nicola Veronese; Brendon Stubbs; Marco Solmi; Ai Koyanagi; Sung Hwi Hong; Elena Dragioti; Eunyoung Cho; Leandro F M de Rezende; Edward L Giovannucci; Jae Il Shin; Gabriele Gamerith Journal: Adv Nutr Date: 2020-11-16 Impact factor: 8.701
Authors: Maria Kyrgiou; Ilkka Kalliala; Georgios Markozannes; Marc J Gunter; Evangelos Paraskevaidis; Hani Gabra; Pierre Martin-Hirsch; Konstantinos K Tsilidis Journal: BMJ Date: 2017-02-28
Authors: Myrela O Machado; Nicola Veronese; Marcos Sanches; Brendon Stubbs; Ai Koyanagi; Trevor Thompson; Ioanna Tzoulaki; Marco Solmi; Davy Vancampfort; Felipe B Schuch; Michael Maes; Giovanni A Fava; John P A Ioannidis; André F Carvalho Journal: BMC Med Date: 2018-07-20 Impact factor: 8.775
Authors: Jong Yeob Kim; Johanna Steingroever; Keum Hwa Lee; Jun Oh; Min Jae Choi; Jiwon Lee; Nicholas G Larkins; Franz Schaefer; Sung Hwi Hong; Gwang Hun Jeong; Jae Il Shin; Andreas Kronbichler Journal: J Clin Med Date: 2020-02-01 Impact factor: 4.241