Ryoko Susukida1, Rosa M Crum1,2,3, Cyrus Ebnesajjad1,4, Elizabeth A Stuart1,5,6, Ramin Mojtabai1,3. 1. Department of Mental Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA. 2. Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA. 3. Department of Psychiatry and Behavioral Sciences, Johns Hopkins University School of Medicine, Baltimore, MD, USA. 4. The Fred Hutchinson Cancer Research Center, Seattle, WA, USA. 5. Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA. 6. Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA.
Abstract
AIMS: To compare randomized controlled trial (RCT) sample treatment effects with the population effects of substance use disorder (SUD) treatment. DESIGN: Statistical weighting was used to re-compute the effects from 10 RCTs such that the participants in the trials had characteristics that resembled those of patients in the target populations. SETTINGS: Multi-site RCTs and usual SUD treatment settings in the United States. PARTICIPANTS: A total of 3592 patients in 10 RCTs and 1 602 226 patients from usual SUD treatment settings between 2001 and 2009. MEASUREMENTS: Three outcomes of SUD treatment were examined: retention, urine toxicology and abstinence. We weighted the RCT sample treatment effects using propensity scores representing the conditional probability of participating in RCTs. FINDINGS: Weighting the samples changed the significance of estimated sample treatment effects. Most commonly, positive effects of trials became statistically non-significant after weighting (three trials for retention and urine toxicology and one trial for abstinence); also, non-significant effects became significantly positive (one trial for abstinence) and significantly negative effects became non-significant (two trials for abstinence). There was suggestive evidence of treatment effect heterogeneity in subgroups that are under- or over-represented in the trials, some of which were consistent with the differences in average treatment effects between weighted and unweighted results. CONCLUSIONS: The findings of randomized controlled trials (RCTs) for substance use disorder treatment do not appear to be directly generalizable to target populations when the RCT samples do not reflect adequately the target populations and there is treatment effect heterogeneity across patient subgroups.
RCT Entities:
AIMS: To compare randomized controlled trial (RCT) sample treatment effects with the population effects of substance use disorder (SUD) treatment. DESIGN: Statistical weighting was used to re-compute the effects from 10 RCTs such that the participants in the trials had characteristics that resembled those of patients in the target populations. SETTINGS: Multi-site RCTs and usual SUD treatment settings in the United States. PARTICIPANTS: A total of 3592 patients in 10 RCTs and 1 602 226 patients from usual SUD treatment settings between 2001 and 2009. MEASUREMENTS: Three outcomes of SUD treatment were examined: retention, urine toxicology and abstinence. We weighted the RCT sample treatment effects using propensity scores representing the conditional probability of participating in RCTs. FINDINGS: Weighting the samples changed the significance of estimated sample treatment effects. Most commonly, positive effects of trials became statistically non-significant after weighting (three trials for retention and urine toxicology and one trial for abstinence); also, non-significant effects became significantly positive (one trial for abstinence) and significantly negative effects became non-significant (two trials for abstinence). There was suggestive evidence of treatment effect heterogeneity in subgroups that are under- or over-represented in the trials, some of which were consistent with the differences in average treatment effects between weighted and unweighted results. CONCLUSIONS: The findings of randomized controlled trials (RCTs) for substance use disorder treatment do not appear to be directly generalizable to target populations when the RCT samples do not reflect adequately the target populations and there is treatment effect heterogeneity across patient subgroups.
Keywords:
Generalizability; National Institute of Drug Abuse Clinical Trials Network; propensity score weighting; randomized controlled trials; substance use disorder treatment; treatment effect heterogeneity
Authors: Elizabeth A Stuart; Stephen R Cole; Catherine P Bradshaw; Philip J Leaf Journal: J R Stat Soc Ser A Stat Soc Date: 2001-04-01 Impact factor: 2.483
Authors: Kathleen M Carroll; Samuel A Ball; Charla Nich; Steve Martino; Tami L Frankforter; Christiane Farentinos; Lynn E Kunkel; Susan K Mikulich-Gilbertson; Jon Morgenstern; Jeanne L Obert; Doug Polcin; Ned Snead; George E Woody Journal: Drug Alcohol Depend Date: 2005-09-28 Impact factor: 4.492
Authors: Nicolas Hoertel; Yann Le Strat; Carlos Blanco; Pierre Lavaud; Caroline Dubertret Journal: Depress Anxiety Date: 2012-04-11 Impact factor: 6.505
Authors: Carlos Blanco; Mark Olfson; Renee D Goodwin; Elizabeth Ogburn; Michael R Liebowitz; Edward V Nunes; Deborah S Hasin Journal: J Clin Psychiatry Date: 2008-08 Impact factor: 4.384
Authors: Brittany N Rudd; Briana S Last; Courtney Gregor; Kamilah Jackson; Steven Berkowitz; Arturo Zinny; Hilary E Kratz; Lauren Cliggitt; Danielle R Adams; Lucia M Walsh; Rinad S Beidas Journal: Am J Community Psychol Date: 2019-08-20
Authors: Benjamin Ackerman; Ian Schmid; Kara E Rudolph; Marissa J Seamans; Ryoko Susukida; Ramin Mojtabai; Elizabeth A Stuart Journal: Addict Behav Date: 2018-10-25 Impact factor: 3.913
Authors: Tiarnan D Keenan; Susan Vitale; Elvira Agrón; Amitha Domalpally; Andrew N Antoszyk; Michael J Elman; Traci E Clemons; Emily Y Chew Journal: Ophthalmol Retina Date: 2019-06-11
Authors: Noa Krawczyk; Ramin Mojtabai; Elizabeth A Stuart; Michael Fingerhood; Deborah Agus; B Casey Lyons; Jonathan P Weiner; Brendan Saloner Journal: Addiction Date: 2020-02-24 Impact factor: 6.526
Authors: Michael A Webster-Clark; Hanna K Sanoff; Til Stürmer; Sharon Peacock Hinton; Jennifer L Lund Journal: Epidemiology Date: 2019-01 Impact factor: 4.822
Authors: Zhaoyi Chen; Hansi Zhang; Yi Guo; Thomas J George; Mattia Prosperi; William R Hogan; Zhe He; Elizabeth A Shenkman; Fei Wang; Jiang Bian Journal: NPJ Digit Med Date: 2021-05-14
Authors: Stephan Ehrhardt; Anton P Porsteinsson; Cynthia A Munro; Paul B Rosenberg; Bruce G Pollock; Davangere P Devanand; Jacobo Mintzer; Tarek K Rajji; Zahinoor Ismail; Lon S Schneider; Sheriza N Baksh; Lea T Drye; Dimitri Avramopoulos; David M Shade; Constantine G Lyketsos Journal: Alzheimers Dement Date: 2019-10-03 Impact factor: 16.655