Hannah Ewald1, John P A Ioannidis2, Aviv Ladanie3, Kimberly Mc Cord3, Heiner C Bucher4, Lars G Hemkens5. 1. Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, University of Basel, 4031 Basel, Switzerland; Swiss Tropical and Public Health Institute, University of Basel, 4051 Basel, Switzerland; University Medical Library, University of Basel, Basel, Switzerland. 2. Department of Medicine, Stanford Prevention Research Center, Stanford University School of Medicine, Stanford, CA 94305, USA; Meta-Research Innovation Center at Stanford (METRICS), Stanford School of Medicine, Palo Alto, CA 94304, USA; Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA 94305, USA; Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA 94305, USA; Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA 94305, USA. 3. Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, University of Basel, 4031 Basel, Switzerland; Swiss Tropical and Public Health Institute, University of Basel, 4051 Basel, Switzerland. 4. Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, University of Basel, 4031 Basel, Switzerland. 5. Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, University of Basel, 4031 Basel, Switzerland. Electronic address: Lars.Hemkens@usb.ch.
Abstract
OBJECTIVES: To evaluate how estimated treatment effects agree between nonrandomized studies using causal modeling with marginal structural models (MSM-studies) and randomized trials (RCTs). STUDY DESIGN: Meta-epidemiological study. SETTING: MSM-studies providing effect estimates on any healthcare outcome of any treatment were eligible. We systematically sought RCTs on the same clinical question and compared the direction of treatment effects, effect sizes, and confidence intervals. RESULTS: The main analysis included 19 MSM-studies (1,039,570 patients) and 141 RCTs (120,669 patients). MSM-studies indicated effect estimates in the opposite direction from RCTs for eight clinical questions (42%), and their 95% CI (confidence interval) did not include the RCT estimate in nine clinical questions (47%). The effect estimates deviated 1.58-fold between the study designs (median absolute deviation OR [odds ratio] 1.58; IQR [interquartile range] 1.37 to 2.16). Overall, we found no systematic disagreement regarding benefit or harm but confidence intervals were wide (summary ratio of odds ratios [sROR] 1.04; 95% CI 0.88 to 1.23). The subset of MSM-studies focusing on healthcare decision-making tended to overestimate experimental treatment benefits (sROR 1.44; 95% CI 0.99 to 2.09). CONCLUSION: Nonrandomized studies using causal modeling with MSM may give different answers than RCTs. Caution is still required when nonrandomized "real world" evidence is used for healthcare decisions.
OBJECTIVES: To evaluate how estimated treatment effects agree between nonrandomized studies using causal modeling with marginal structural models (MSM-studies) and randomized trials (RCTs). STUDY DESIGN: Meta-epidemiological study. SETTING: MSM-studies providing effect estimates on any healthcare outcome of any treatment were eligible. We systematically sought RCTs on the same clinical question and compared the direction of treatment effects, effect sizes, and confidence intervals. RESULTS: The main analysis included 19 MSM-studies (1,039,570 patients) and 141 RCTs (120,669 patients). MSM-studies indicated effect estimates in the opposite direction from RCTs for eight clinical questions (42%), and their 95% CI (confidence interval) did not include the RCT estimate in nine clinical questions (47%). The effect estimates deviated 1.58-fold between the study designs (median absolute deviation OR [odds ratio] 1.58; IQR [interquartile range] 1.37 to 2.16). Overall, we found no systematic disagreement regarding benefit or harm but confidence intervals were wide (summary ratio of odds ratios [sROR] 1.04; 95% CI 0.88 to 1.23). The subset of MSM-studies focusing on healthcare decision-making tended to overestimate experimental treatment benefits (sROR 1.44; 95% CI 0.99 to 2.09). CONCLUSION: Nonrandomized studies using causal modeling with MSM may give different answers than RCTs. Caution is still required when nonrandomized "real world" evidence is used for healthcare decisions.
Authors: Lucia C Petito; Xabier García-Albéniz; Roger W Logan; Nadia Howlader; Angela B Mariotto; Issa J Dahabreh; Miguel A Hernán Journal: JAMA Netw Open Date: 2020-03-02