Emily M Lord1, Isabelle R Weir1, Ludovic Trinquart2. 1. Boston University School of Public Health, Department of Biostatistics, 801 Massachusetts Avenue, Boston, MA 02118, USA. 2. Boston University School of Public Health, Department of Biostatistics, 801 Massachusetts Avenue, Boston, MA 02118, USA. Electronic address: ludovic@bu.edu.
Abstract
OBJECTIVE: Statistical significance drives interpretation of randomized controlled trials (RCTs). We examined the type S error risk-claiming a new drug is falsely beneficial-and exaggeration ratio-how estimated effects differ from true effects-to re-emphasize direction and magnitude of treatment effects. STUDY DESIGN AND SETTING: We systematically reviewed RCTs supporting Food and Drug Administration (FDA) approval of cancer drugs between 2007 and 2016. We extracted data for overall survival (OS), progression-free survival (PFS), and response outcomes from FDA reviews. We estimated type S error risks and exaggeration ratios by considering replicated RCTs of equal size and a range of true effects. RESULTS: We analyzed 42 RCTs for 39 approved drugs. Across 38 RCTs reporting OS, the median type S error risk was 0.00% (Q1-Q3, 0.00-0.01%) and 3.56% (0.40-6.74%), for true hazard ratios of 0.7 and 0.9, respectively, indicating confidence in effect direction. The corresponding exaggeration ratios were 1.09 (1.01-1.11) and 1.30 (1.13-1.42), indicating median overestimations of 9% and 30%. Similar results held for PFS and response outcomes. CONCLUSIONS: The type S error risk and exaggeration ratio provide additional insights into the replicability of RCTs. Our analyses also quantify the winner's curse, in which pivotal RCTs tend toward overoptimism.
OBJECTIVE: Statistical significance drives interpretation of randomized controlled trials (RCTs). We examined the type S error risk-claiming a new drug is falsely beneficial-and exaggeration ratio-how estimated effects differ from true effects-to re-emphasize direction and magnitude of treatment effects. STUDY DESIGN AND SETTING: We systematically reviewed RCTs supporting Food and Drug Administration (FDA) approval of cancer drugs between 2007 and 2016. We extracted data for overall survival (OS), progression-free survival (PFS), and response outcomes from FDA reviews. We estimated type S error risks and exaggeration ratios by considering replicated RCTs of equal size and a range of true effects. RESULTS: We analyzed 42 RCTs for 39 approved drugs. Across 38 RCTs reporting OS, the median type S error risk was 0.00% (Q1-Q3, 0.00-0.01%) and 3.56% (0.40-6.74%), for true hazard ratios of 0.7 and 0.9, respectively, indicating confidence in effect direction. The corresponding exaggeration ratios were 1.09 (1.01-1.11) and 1.30 (1.13-1.42), indicating median overestimations of 9% and 30%. Similar results held for PFS and response outcomes. CONCLUSIONS: The type S error risk and exaggeration ratio provide additional insights into the replicability of RCTs. Our analyses also quantify the winner's curse, in which pivotal RCTs tend toward overoptimism.
Authors: Katherine S Button; John P A Ioannidis; Claire Mokrysz; Brian A Nosek; Jonathan Flint; Emma S J Robinson; Marcus R Munafò Journal: Nat Rev Neurosci Date: 2013-04-10 Impact factor: 34.870
Authors: Benjamin Djulbegovic; Ambuj Kumar; Paul P Glasziou; Rafael Perera; Tea Reljic; Louise Dent; James Raftery; Marit Johansen; Gian Luca Di Tanna; Branko Miladinovic; Heloisa P Soares; Gunn E Vist; Iain Chalmers Journal: Cochrane Database Syst Rev Date: 2012-10-17
Authors: Lucia Sobrin; Yinxi Yu; Samuel Han; Gayatri Susarla; John H Kempen; Rebecca A Hubbard; Brian L VanderBeek Journal: Graefes Arch Clin Exp Ophthalmol Date: 2021-05-29 Impact factor: 3.535