Kenneth A Frank1, Qinyun Lin2, Spiro Maroulis3, Anna S Mueller4, Ran Xu5, Joshua M Rosenberg6, Christopher S Hayter3, Ramy A Mahmoud7, Marynia Kolak2, Thomas Dietz8, Lixin Zhang9. 1. Measurement and Quantitative Methods, Education; Agriculture and Natural Resources, Michigan State University, East Lansing, MI. Electronic address: kenfrank@msu.edu. 2. Center for Spatial Data Science, University of Chicago, Chicago IL. 3. School of Public Affairs, Arizona State University, Phoenix, AZ. 4. Department of Sociology, Indiana University, Bloomington, IN. 5. Department of Allied Health Sciences, University of Connecticut, Storrs, CT. 6. Education, Health and Human Sciences, University of Tennessee, Knoxville. 7. Optinose, Inc. Yardley, PA. 8. Environmental Science and Policy, Sociology, Animal Studies, Michigan State University, East Lansing, MI. 9. Epidemiology and Biostatistics, Microbiology and Molecular Genetics, Michigan State University, East Lansing, MI.
Abstract
OBJECTIVES: We apply a general case replacement framework for quantifying the robustness of causal inferences to characterize the uncertainty of findings from clinical trials. STUDY DESIGN AND SETTING: We express the robustness of inferences as the amount of data that must be replaced to change the conclusion and relate this to the fragility of trial results used for dichotomous outcomes. We illustrate our approach in the context of an RCT of hydroxychloroquine on pneumonia in COVID-19 patients and a cumulative meta-analysis of the effect of antihypertensive treatments on stroke. RESULTS: We developed the Robustness of an Inference to Replacement (RIR), which quantifies how many treatment cases with positive outcomes would have to be replaced with hypothetical patients who did not receive a treatment to change an inference. The RIR addresses known limitations of the Fragility Index by accounting for the observed rates of outcomes. It can be used for varying thresholds for inference, including clinical importance. CONCLUSION: Because the RIR expresses uncertainty in terms of patient experiences, it is more relatable to stakeholders than P-values alone. It helps identify when results are statistically significant, but conclusions are not robust, while considering the rareness of events in the underlying data.
OBJECTIVES: We apply a general case replacement framework for quantifying the robustness of causal inferences to characterize the uncertainty of findings from clinical trials. STUDY DESIGN AND SETTING: We express the robustness of inferences as the amount of data that must be replaced to change the conclusion and relate this to the fragility of trial results used for dichotomous outcomes. We illustrate our approach in the context of an RCT of hydroxychloroquine on pneumonia in COVID-19 patients and a cumulative meta-analysis of the effect of antihypertensive treatments on stroke. RESULTS: We developed the Robustness of an Inference to Replacement (RIR), which quantifies how many treatment cases with positive outcomes would have to be replaced with hypothetical patients who did not receive a treatment to change an inference. The RIR addresses known limitations of the Fragility Index by accounting for the observed rates of outcomes. It can be used for varying thresholds for inference, including clinical importance. CONCLUSION: Because the RIR expresses uncertainty in terms of patient experiences, it is more relatable to stakeholders than P-values alone. It helps identify when results are statistically significant, but conclusions are not robust, while considering the rareness of events in the underlying data.
Authors: Benjamin R Baer; Stephen E Fremes; Mario Gaudino; Mary Charlson; Martin T Wells Journal: BMC Med Res Methodol Date: 2021-11-20 Impact factor: 4.615