Iztok Hozo1, Benjamin Djulbegovic2, Austin J Parish3, John P A Ioannidis4. 1. Department of Mathematics, Indiana University Northwest, Gary, IN. 2. Beckman Research Institute, City of Hope Institution, Duarte, CA; Department of Computational and Quantitative Medicine, City of Hope Institution, Duarte, CA; Division of Health Analytics; Evidence-based Medicine and Comparative Effectiveness Research, 1500 East Duarte Road, Duarte, CA. Electronic address: bdjulbegovic@coh.org. 3. Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA; Department of Emergency Medicine, Lincoln Medical Center, Bronx, New York. 4. Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA; Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Stanford, California; Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, California; Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, California, Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA.
Abstract
OBJECTIVE: To analyze distribution of "dramatic", large treatment effects. STUDY DESIGN & SETTING: Pareto distribution modeling of previously reported cohorts of 3,486 randomized trials (RCTs) that enrolled 1,532,459 patients and 730 non-randomized studies (NRS) enrolling 1,650,658 patients. RESULTS: We calculated the Pareto α parameter, which determines the tail of the distribution for various starting points of distribution [odds ratiomin (ORmin)]. In default analysis using all data at ORmin ≥1, Pareto distribution fit well to the treatment effects of RCTs favoring the new treatments (P = 0.21, Kolmogorov-Smirnov test) with best α = 2.32. For NRS, Pareto fit for ORmin ≥2 with best α = 1.91. For RCTs, theoretical 99th percentile OR was 32.7. The actual 99th percentile OR was 25; which converted into relative risk (RR) = 7.1. The maximum observed effect size was OR = 121 (RR = 11.45). For NRS, theoretical 99th percentile was OR = 315. The actual 99th percentile OR was 294 (RR = 13). The maximum observed effect size was OR = 1473 (RR = 66). CONCLUSIONS: The effects sizes observed in RCTs and NRS considerably overlap. Large effects are rare and there is no clear threshold for dramatic effects that would obviate future RCTs.
OBJECTIVE: To analyze distribution of "dramatic", large treatment effects. STUDY DESIGN & SETTING: Pareto distribution modeling of previously reported cohorts of 3,486 randomized trials (RCTs) that enrolled 1,532,459 patients and 730 non-randomized studies (NRS) enrolling 1,650,658 patients. RESULTS: We calculated the Pareto α parameter, which determines the tail of the distribution for various starting points of distribution [odds ratiomin (ORmin)]. In default analysis using all data at ORmin ≥1, Pareto distribution fit well to the treatment effects of RCTs favoring the new treatments (P = 0.21, Kolmogorov-Smirnov test) with best α = 2.32. For NRS, Pareto fit for ORmin ≥2 with best α = 1.91. For RCTs, theoretical 99th percentile OR was 32.7. The actual 99th percentile OR was 25; which converted into relative risk (RR) = 7.1. The maximum observed effect size was OR = 121 (RR = 11.45). For NRS, theoretical 99th percentile was OR = 315. The actual 99th percentile OR was 294 (RR = 13). The maximum observed effect size was OR = 1473 (RR = 66). CONCLUSIONS: The effects sizes observed in RCTs and NRS considerably overlap. Large effects are rare and there is no clear threshold for dramatic effects that would obviate future RCTs.
Authors: Benjamin Djulbegovic; Ambuj Kumar; Heloisa P Soares; Iztok Hozo; Gerold Bepler; Mike Clarke; Charles L Bennett Journal: Arch Intern Med Date: 2008-03-24
Authors: Marianne Razavi; Paul Glasziou; Farina A Klocksieben; John P A Ioannidis; Iain Chalmers; Benjamin Djulbegovic Journal: JAMA Netw Open Date: 2019-09-04
Authors: Benjamin Djulbegovic; Ambuj Kumar; Branko Miladinovic; Tea Reljic; Sanja Galeb; Asmita Mhaskar; Rahul Mhaskar; Iztok Hozo; Dongsheng Tu; Heather A Stanton; Christopher M Booth; Ralph M Meyer Journal: PLoS One Date: 2013-03-21 Impact factor: 3.240
Authors: Tim Mathes; Nina-Kristin Mann; Petra Thürmann; Andreas Sönnichsen; Dawid Pieper Journal: BMC Med Res Methodol Date: 2022-08-30 Impact factor: 4.612