Darryl Abrams1, Sydney B Montesi2, Sarah K L Moore3, Daniel K Manson3, Kaitlin M Klipper1, Meredith A Case3, Daniel Brodie1, Jeremy R Beitler1. 1. Center for Acute Respiratory Failure and Division of Pulmonary, Allergy, and Critical Care Medicine, Columbia University College of Physicians and Surgeons and New York-Presbyterian Hospital, New York, NY. 2. Division of Pulmonary and Critical Care Medicine, Massachusetts General Hospital and Harvard Medical School, Boston, MA. 3. Department of Medicine, Columbia University College of Physicians and Surgeons and New York-Presbyterian Hospital, New York, NY.
Abstract
OBJECTIVES: Recurring issues in clinical trial design may bias results toward the null, yielding findings inconclusive for treatment effects. This study evaluated for powering bias among high-impact critical care trials and the associated risk of masking clinically important treatment effects. DESIGN, SETTING, AND PATIENTS: Secondary analysis of multicenter randomized trials of critically ill adults in which mortality was the main endpoint. Trials were eligible for inclusion if published between 2008 and 2018 in leading journals. Analyses evaluated for accuracy of estimated control group mortality, adaptive sample size strategy, plausibility of predicted treatment effect, and results relative to the minimal clinically important difference. The main outcome was the mortality risk difference at the study-specific follow-up interval. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Of 101 included trials, 12 met statistical significance for their main endpoint, five for increased intervention-associated mortality. Most trials (77.3%) overestimated control group mortality in power calculations (observed minus predicted difference, -6.7% ± 9.8%; p < 0.01). Due to this misestimation of control group mortality, in 14 trials, the intervention would have had to prevent at least half of all deaths to achieve the hypothesized treatment effect. Seven trials prespecified adaptive sample size strategies that might have mitigated this issue. The observed risk difference for mortality fell within 5% of predicted in 20 trials, of which 16 did not reach statistical significance. Half of trials (47.0%) were powered for an absolute risk reduction greater than or equal to 10%, but this effect size was observed in only three trials with a statistically significant treatment benefit. Most trials (67.3%) could not exclude clinically important treatment benefit or harm. CONCLUSIONS: The design of most high-impact critical care trials biased results toward the null by overestimating control group mortality and powering for unrealistic treatment effects. Clinically important treatment effects often cannot be excluded.
OBJECTIVES: Recurring issues in clinical trial design may bias results toward the null, yielding findings inconclusive for treatment effects. This study evaluated for powering bias among high-impact critical care trials and the associated risk of masking clinically important treatment effects. DESIGN, SETTING, AND PATIENTS: Secondary analysis of multicenter randomized trials of critically ill adults in which mortality was the main endpoint. Trials were eligible for inclusion if published between 2008 and 2018 in leading journals. Analyses evaluated for accuracy of estimated control group mortality, adaptive sample size strategy, plausibility of predicted treatment effect, and results relative to the minimal clinically important difference. The main outcome was the mortality risk difference at the study-specific follow-up interval. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Of 101 included trials, 12 met statistical significance for their main endpoint, five for increased intervention-associated mortality. Most trials (77.3%) overestimated control group mortality in power calculations (observed minus predicted difference, -6.7% ± 9.8%; p < 0.01). Due to this misestimation of control group mortality, in 14 trials, the intervention would have had to prevent at least half of all deaths to achieve the hypothesized treatment effect. Seven trials prespecified adaptive sample size strategies that might have mitigated this issue. The observed risk difference for mortality fell within 5% of predicted in 20 trials, of which 16 did not reach statistical significance. Half of trials (47.0%) were powered for an absolute risk reduction greater than or equal to 10%, but this effect size was observed in only three trials with a statistically significant treatment benefit. Most trials (67.3%) could not exclude clinically important treatment benefit or harm. CONCLUSIONS: The design of most high-impact critical care trials biased results toward the null by overestimating control group mortality and powering for unrealistic treatment effects. Clinically important treatment effects often cannot be excluded.
Authors: E Rivers; B Nguyen; S Havstad; J Ressler; A Muzzin; B Knoblich; E Peterson; M Tomlanovich Journal: N Engl J Med Date: 2001-11-08 Impact factor: 91.245
Authors: Roland M du Bois; Derek Weycker; Carlo Albera; Williamson Z Bradford; Ulrich Costabel; Alex Kartashov; Lisa Lancaster; Paul W Noble; Steven A Sahn; Javier Szwarcberg; Michiel Thomeer; Dominique Valeyre; Talmadge E King Journal: Am J Respir Crit Care Med Date: 2010-12-03 Impact factor: 21.405
Authors: Christopher W Seymour; Jason N Kennedy; Shu Wang; Chung-Chou H Chang; Corrine F Elliott; Zhongying Xu; Scott Berry; Gilles Clermont; Gregory Cooper; Hernando Gomez; David T Huang; John A Kellum; Qi Mi; Steven M Opal; Victor Talisa; Tom van der Poll; Shyam Visweswaran; Yoram Vodovotz; Jeremy C Weiss; Donald M Yealy; Sachin Yende; Derek C Angus Journal: JAMA Date: 2019-05-28 Impact factor: 56.272
Authors: Michael O Harhay; Jason Wagner; Sarah J Ratcliffe; Rachel S Bronheim; Anand Gopal; Sydney Green; Elizabeth Cooney; Mark E Mikkelsen; Meeta Prasad Kerlin; Dylan S Small; Scott D Halpern Journal: Am J Respir Crit Care Med Date: 2014-06-15 Impact factor: 21.405
Authors: Amelie Elsäßer; Jan Regnstrom; Thorsten Vetter; Franz Koenig; Robert James Hemmings; Martina Greco; Marisa Papaluca-Amati; Martin Posch Journal: Trials Date: 2014-10-02 Impact factor: 2.279
Authors: Anders Granholm; Waleed Alhazzani; Lennie P G Derde; Derek C Angus; Fernando G Zampieri; Naomi E Hammond; Rob Mac Sweeney; Sheila N Myatra; Elie Azoulay; Kathryn Rowan; Paul J Young; Anders Perner; Morten Hylander Møller Journal: Intensive Care Med Date: 2021-12-02 Impact factor: 41.787
Authors: Daniel M Benjamin; Spencer P Hey; Amanda MacPherson; Yasmina Hachem; Kara S Smith; Sean X Zhang; Sandy Wong; Samantha Dolter; David R Mandel; Jonathan Kimmelman Journal: PLoS One Date: 2022-02-08 Impact factor: 3.240
Authors: Miguel Á Ibarra-Estrada; Eduardo Mireles-Cabodevila; Yessica García-Salas; José A López-Pulgarín; Quetzalcóatl Chávez-Peña; Roxana García-Salcido; Julio C Mijangos-Méndez; Guadalupe Aguirre-Avalos Journal: Crit Care Med Date: 2022-04-01 Impact factor: 9.296
Authors: Fernando G Zampieri; Bruno R da Costa; Suvi T Vaara; François Lamontagne; Bram Rochwerg; Alistair D Nichol; Shay McGuinness; Danny F McAuley; Marlies Ostermann; Ron Wald; Sean M Bagshaw Journal: Crit Care Date: 2022-08-25 Impact factor: 19.334
Authors: Christopher J Yarnell; Darryl Abrams; Matthew R Baldwin; Daniel Brodie; Eddy Fan; Niall D Ferguson; May Hua; Purnema Madahar; Danny F McAuley; Laveena Munshi; Gavin D Perkins; Gordon Rubenfeld; Arthur S Slutsky; Hannah Wunsch; Robert A Fowler; George Tomlinson; Jeremy R Beitler; Ewan C Goligher Journal: Lancet Respir Med Date: 2020-11-20 Impact factor: 30.700