Stephen D Walter1, Lehana Thabane2, Matthias Briel3. 1. Department of Health Research Methodology, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada. Electronic address: walter@mcmaster.ca. 2. Department of Health Research Methodology, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada. 3. Department of Health Research Methodology, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada; Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel and University of Basel, Basel, Switzerland.
Abstract
OBJECTIVES: The fragility of clinical trial findings has been previously defined as the number of changes in outcomes that are required to change their statistical significance. We show that reliance on statistical significance alone provides only a limited and potentially misleading perspective, and an enhanced approach is developed. METHODS: Clinical importance of trial results and their quantitative stability are incorporated into an enhanced framework to assess fragility. RESULTS: Examples show that the small data changes required to affect statistical significance may actually be unlikely to occur. Recognizing this limitation, and because statistical significance conveys no information about the treatment effect size, our approach additionally takes into account the clinical importance of the results and their quantitative stability. The interpretation of studies with various combinations of these features is described. CONCLUSION: The concept of fragility should include clinical importance of trial findings and their quantitative stability, as well as statistical significance. Study results should be declared as stable only if they are statistically significant and quantitatively stable, but they can be either clinically important or unimportant; otherwise, the findings should be declared as unstable, or fragile.
OBJECTIVES: The fragility of clinical trial findings has been previously defined as the number of changes in outcomes that are required to change their statistical significance. We show that reliance on statistical significance alone provides only a limited and potentially misleading perspective, and an enhanced approach is developed. METHODS: Clinical importance of trial results and their quantitative stability are incorporated into an enhanced framework to assess fragility. RESULTS: Examples show that the small data changes required to affect statistical significance may actually be unlikely to occur. Recognizing this limitation, and because statistical significance conveys no information about the treatment effect size, our approach additionally takes into account the clinical importance of the results and their quantitative stability. The interpretation of studies with various combinations of these features is described. CONCLUSION: The concept of fragility should include clinical importance of trial findings and their quantitative stability, as well as statistical significance. Study results should be declared as stable only if they are statistically significant and quantitatively stable, but they can be either clinically important or unimportant; otherwise, the findings should be declared as unstable, or fragile.
Authors: Benjamin R Baer; Mario Gaudino; Mary Charlson; Stephen E Fremes; Martin T Wells Journal: Proc Natl Acad Sci U S A Date: 2021-12-07 Impact factor: 12.779
Authors: Benjamin R Baer; Stephen E Fremes; Mario Gaudino; Mary Charlson; Martin T Wells Journal: BMC Med Res Methodol Date: 2021-11-20 Impact factor: 4.615
Authors: N Bryce Robinson; Stephen Fremes; Irbaz Hameed; Mohamed Rahouma; Viola Weidenmann; Michelle Demetres; Mahmoud Morsi; Giovanni Soletti; Antonino Di Franco; Marco A Zenati; Shahzad G Raja; David Moher; Faisal Bakaeen; Joanna Chikwe; Deepak L Bhatt; Paul Kurlansky; Leonard N Girardi; Mario Gaudino Journal: JAMA Netw Open Date: 2021-06-01