Carl Shen1, Isabel Shamsudeen2, Forough Farrokhyar3, Kourosh Sabri4. 1. Division of Ophthalmology, Department of Surgery, McMaster University, Hamilton, Ontario, Canada. Electronic address: carl.shen@medportal.ca. 2. Division of Ophthalmology, Department of Surgery, McMaster University, Hamilton, Ontario, Canada. 3. Departments of Surgery & Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada. 4. Division of Ophthalmology, Department of Surgery, McMaster University, Hamilton, Ontario, Canada; McMaster Paediatric Eye Research Group, Department of Surgery, McMaster University, Hamilton, Ontario, Canada; McMaster Pediatric Surgery Research Collaborative, Department of Surgery, McMaster University, Hamilton, Ontario, Canada.
Abstract
PURPOSE: Evidence-based medicine is guided by our interpretation of randomized controlled trials (RCTs) that address important clinical questions. Evaluation of the robustness of statistically significant outcomes adds a crucial element to the global assessment of trial findings. The purpose of this systematic review was to determine the robustness of ophthalmology RCTs through application of the Fragility Index (FI), a novel metric of the robustness of statistically significant outcomes. DESIGN: Systematic review. METHODS: A literature search (MEDLINE) was performed for all RCTs published in top ophthalmology journals and ophthalmology-related RCTs published in high-impact journals in the past 10 years. Two reviewers independently screened 1811 identified articles for inclusion if they (1) were a human ophthalmology-related trial, (2) had a 1:1 prospective study design, and (3) reported a statistically significant dichotomous outcome in the abstract. All relevant data, including outcome, P value, number of patients in each group, number of events in each group, number of patients lost to follow-up, and trial characteristics, were extracted. The FI of each RCT was calculated and multivariate regression applied to determine predictive factors. RESULTS: The 156 trials had a median sample size of 91.5 (range, 13-2593) patients/eyes, and a median of 28 (range, 4-2217) events. The median FI of the included trials was 2 (range, 0-48), meaning that if 2 non-events were switched to events in the treatment group, the result would lose its statistical significance. A quarter of all trials had an FI of 1 or less, and 75% of trials had an FI of 6 or less. The FI was less than the number of missing data points in 52.6% of trials. Predictive factors for FI by multivariate regression included smaller P value (P < 0.001), larger sample size (P = 0.001), larger number of events (P = 0.011), and journal impact factor (P = 0.029). CONCLUSIONS: In ophthalmology trials, statistically significant dichotomous results are often fragile, meaning that a difference of only a couple of events can change the statistical significance. An application of the FI in RCTs may aid in the interpretation of results and assessment of quality of evidence.
PURPOSE: Evidence-based medicine is guided by our interpretation of randomized controlled trials (RCTs) that address important clinical questions. Evaluation of the robustness of statistically significant outcomes adds a crucial element to the global assessment of trial findings. The purpose of this systematic review was to determine the robustness of ophthalmology RCTs through application of the Fragility Index (FI), a novel metric of the robustness of statistically significant outcomes. DESIGN: Systematic review. METHODS: A literature search (MEDLINE) was performed for all RCTs published in top ophthalmology journals and ophthalmology-related RCTs published in high-impact journals in the past 10 years. Two reviewers independently screened 1811 identified articles for inclusion if they (1) were a human ophthalmology-related trial, (2) had a 1:1 prospective study design, and (3) reported a statistically significant dichotomous outcome in the abstract. All relevant data, including outcome, P value, number of patients in each group, number of events in each group, number of patients lost to follow-up, and trial characteristics, were extracted. The FI of each RCT was calculated and multivariate regression applied to determine predictive factors. RESULTS: The 156 trials had a median sample size of 91.5 (range, 13-2593) patients/eyes, and a median of 28 (range, 4-2217) events. The median FI of the included trials was 2 (range, 0-48), meaning that if 2 non-events were switched to events in the treatment group, the result would lose its statistical significance. A quarter of all trials had an FI of 1 or less, and 75% of trials had an FI of 6 or less. The FI was less than the number of missing data points in 52.6% of trials. Predictive factors for FI by multivariate regression included smaller P value (P < 0.001), larger sample size (P = 0.001), larger number of events (P = 0.011), and journal impact factor (P = 0.029). CONCLUSIONS: In ophthalmology trials, statistically significant dichotomous results are often fragile, meaning that a difference of only a couple of events can change the statistical significance. An application of the FI in RCTs may aid in the interpretation of results and assessment of quality of evidence.
Authors: Benjamin R Baer; Stephen E Fremes; Mario Gaudino; Mary Charlson; Martin T Wells Journal: BMC Med Res Methodol Date: 2021-11-20 Impact factor: 4.615
Authors: Robert L Parisien; David P Trofa; Michaela O'Connor; Brock Knapp; Emily J Curry; Paul Tornetta; T Sean Lynch; Xinning Li Journal: JB JS Open Access Date: 2021-10-21