Michael Walsh1, Sadeesh K Srinathan2, Daniel F McAuley3, Marko Mrkobrada4, Oren Levine5, Christine Ribic6, Amber O Molnar7, Neil D Dattani8, Andrew Burke4, Gordon Guyatt6, Lehana Thabane9, Stephen D Walter6, Janice Pogue10, P J Devereaux11. 1. Department of Clinical Epidemiology and Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario, Canada, L8S4L8; Department of Medicine, McMaster University, 1280 Main Street West, Hamilton, Ontario, Canada, L8S4L8; Population Health Research Institute, Hamilton Health Sciences and McMaster University, 237 Barton St East, Hamilton, Ontario, Canada, L8L2X2. Electronic address: walshm@phri.ca. 2. Department of Surgery, University of Manitoba, Health Sciences Centre, GE611 Sherbrooke St., Winnipeg, Manitoba, Canada, R3A1R9. 3. Centre for Infection and Immunity, Queen's University of Belfast, Health Sciences Building, 97 Lisburn Road, Belfast, BT97BL, UK; Regional Intensive Care Unit, Royal Victoria Hospital, Victoria Hospital, 274 Grosvenor Road, Belfast, BT126BA, UK. 4. Department of Medicine, Western University, London Health Sciences Centre, University Hospital, 339 Windemere Road, London, Ontario, Canada, N6A 5A5. 5. Department of Medicine, McMaster University, 1280 Main Street West, Hamilton, Ontario, Canada, L8S4L8. 6. Department of Clinical Epidemiology and Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario, Canada, L8S4L8; Department of Medicine, McMaster University, 1280 Main Street West, Hamilton, Ontario, Canada, L8S4L8. 7. Department of Medicine, University of Ottawa, Ottawa Hospital, Riverside Campus, 1967 Riverside Drive, Ottawa, Ontario, Canada, K1H7W9. 8. Faculty of Medicine, University of Toronto, Medical Sciences Building, 1 Kings College Circle, Toronto, Ontario, Canada, M5S1A8. 9. Department of Clinical Epidemiology and Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario, Canada, L8S4L8. 10. Department of Clinical Epidemiology and Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario, Canada, L8S4L8; Population Health Research Institute, Hamilton Health Sciences and McMaster University, 237 Barton St East, Hamilton, Ontario, Canada, L8L2X2. 11. Department of Clinical Epidemiology and Biostatistics, McMaster University, 1280 Main Street West, Hamilton, Ontario, Canada, L8S4L8; Department of Medicine, McMaster University, 1280 Main Street West, Hamilton, Ontario, Canada, L8S4L8; Population Health Research Institute, Hamilton Health Sciences and McMaster University, 237 Barton St East, Hamilton, Ontario, Canada, L8L2X2.
Abstract
OBJECTIVES: A P-value <0.05 is one metric used to evaluate the results of a randomized controlled trial (RCT). We wondered how often statistically significant results in RCTs may be lost with small changes in the numbers of outcomes. STUDY DESIGN AND SETTING: A review of RCTs in high-impact medical journals that reported a statistically significant result for at least one dichotomous or time-to-event outcome in the abstract. In the group with the smallest number of events, we changed the status of patients without an event to an event until the P-value exceeded 0.05. We labeled this number the Fragility Index; smaller numbers indicated a more fragile result. RESULTS: The 399 eligible trials had a median sample size of 682 patients (range: 15-112,604) and a median of 112 events (range: 8-5,142); 53% reported a P-value <0.01. The median Fragility Index was 8 (range: 0-109); 25% had a Fragility Index of 3 or less. In 53% of trials, the Fragility Index was less than the number of patients lost to follow-up. CONCLUSION: The statistically significant results of many RCTs hinge on small numbers of events. The Fragility Index complements the P-value and helps identify less robust results.
OBJECTIVES: A P-value <0.05 is one metric used to evaluate the results of a randomized controlled trial (RCT). We wondered how often statistically significant results in RCTs may be lost with small changes in the numbers of outcomes. STUDY DESIGN AND SETTING: A review of RCTs in high-impact medical journals that reported a statistically significant result for at least one dichotomous or time-to-event outcome in the abstract. In the group with the smallest number of events, we changed the status of patients without an event to an event until the P-value exceeded 0.05. We labeled this number the Fragility Index; smaller numbers indicated a more fragile result. RESULTS: The 399 eligible trials had a median sample size of 682 patients (range: 15-112,604) and a median of 112 events (range: 8-5,142); 53% reported a P-value <0.01. The median Fragility Index was 8 (range: 0-109); 25% had a Fragility Index of 3 or less. In 53% of trials, the Fragility Index was less than the number of patients lost to follow-up. CONCLUSION: The statistically significant results of many RCTs hinge on small numbers of events. The Fragility Index complements the P-value and helps identify less robust results.
Authors: Mario Gaudino; Irbaz Hameed; Mohamed Rahouma; Faiza M Khan; Derrick Y Tam; Giuseppe Biondi-Zoccai; Michelle Demetres; Mary E Charlson; Marc Ruel; Filippo Crea; Volkmar Falk; Leonard N Girardi; Stephen Fremes; Joanna Chikwe Journal: JAMA Intern Med Date: 2020-07-01 Impact factor: 21.873
Authors: Amit X Garg; Matthew T V Chan; Meaghan S Cuerden; P J Devereaux; Seyed Hesameddin Abbasi; Ainslie Hildebrand; François Lamontagne; Andre Lamy; Nicolas Noiseux; Chirag R Parikh; Vlado Perkovic; Mackenzie Quantz; Antoine Rochon; Alistair Royse; Daniel I Sessler; Pallav J Shah; Jessica M Sontrop; Georgios I Tagarakis; Kevin H Teoh; Jessica Vincent; Michael Walsh; Jean-Pierre Yared; Salim Yusuf; Richard P Whitlock Journal: CMAJ Date: 2019-03-04 Impact factor: 8.262
Authors: Ravi S Keshari; Robert Silasi; Narcis I Popescu; Constantin Georgescu; Hala Chaaban; Cristina Lupu; Owen J T McCarty; Charles T Esmon; Florea Lupu Journal: J Thromb Haemost Date: 2019-10-16 Impact factor: 5.824