Benjamin Djulbegovic1,2,3, Marianne Razavi1,2,3, Iztok Hozo4. 1. Department of Supportive Care Medicine, City of Hope National Medical Centre, Duarte, California, USA. 2. Department of Haematology, City of Hope National Medical Centre, Duarte, California, USA. 3. Comparative Effectiveness Research and Evidence-Based Medicine, City of Hope National Medical Centre, Duarte, California, USA. 4. Department of Mathematics, Indiana University, Gary, Indiana, USA.
Abstract
RATIONALE, AIMS AND OBJECTIVES: New therapies are increasingly approved by regulatory agencies such as the Food and Drug Administration (FDA) and the European Medicines Agency (EMA) based on testing in non-randomized clinical trials. These treatments have typically displayed "dramatic effects" (ie, effects that are considered large enough to obviate the combined effects of biases and random errors that may affect the study results). The agencies, however, have not identified how large these effects should be to avoid the need for further testing in randomized controlled trials (RCTs). We investigated the effect size that would circumvent the need for further RCTs testing by the regulatory agencies. We hypothesized that the approval of therapeutic interventions by regulators is based on heuristic decision making whose accuracy can be best characterized by the application of signal detection theory (SDT). METHODS: We merged the EMA and FDA database of approvals based on non-RCT comparisons. We excluded duplicate entries between the two databases. We included a total of 134 approvals of drugs and devices based on non-RCTs. We integrated Weber-Fechner law of psychophysics and recognition heuristics within SDT to provide descriptive explanations of the decisions made by the FDA and EMA to approve new treatments based on non-randomized studies without requiring further testing in RCTs. RESULTS: Our findings suggest that when the difference between novel treatments and the historical control is at least one logarithm (base 10) of magnitude, the veracity of testing in non-RCTs seems to be established. CONCLUSION: Drug developers and practitioners alike can use the change in one logarithm of effect size as a benchmark to decide if further testing in RCTs should be pursued, or as a guide to interpreting the results reported in non-randomized studies. However, further research would be useful to better characterize the threshold of effect size above which testing in RCTs is not needed.
RATIONALE, AIMS AND OBJECTIVES: New therapies are increasingly approved by regulatory agencies such as the Food and Drug Administration (FDA) and the European Medicines Agency (EMA) based on testing in non-randomized clinical trials. These treatments have typically displayed "dramatic effects" (ie, effects that are considered large enough to obviate the combined effects of biases and random errors that may affect the study results). The agencies, however, have not identified how large these effects should be to avoid the need for further testing in randomized controlled trials (RCTs). We investigated the effect size that would circumvent the need for further RCTs testing by the regulatory agencies. We hypothesized that the approval of therapeutic interventions by regulators is based on heuristic decision making whose accuracy can be best characterized by the application of signal detection theory (SDT). METHODS: We merged the EMA and FDA database of approvals based on non-RCT comparisons. We excluded duplicate entries between the two databases. We included a total of 134 approvals of drugs and devices based on non-RCTs. We integrated Weber-Fechner law of psychophysics and recognition heuristics within SDT to provide descriptive explanations of the decisions made by the FDA and EMA to approve new treatments based on non-randomized studies without requiring further testing in RCTs. RESULTS: Our findings suggest that when the difference between novel treatments and the historical control is at least one logarithm (base 10) of magnitude, the veracity of testing in non-RCTs seems to be established. CONCLUSION: Drug developers and practitioners alike can use the change in one logarithm of effect size as a benchmark to decide if further testing in RCTs should be pursued, or as a guide to interpreting the results reported in non-randomized studies. However, further research would be useful to better characterize the threshold of effect size above which testing in RCTs is not needed.