Pitfalls in Reporting Sample Size Calculation Across Randomized Controlled Trials Involving Ivermectin for the treatment of COVID-19.

Since the outbreak of coronavirus disease 2019 (COVID-19), morbidity and mortality continue to increase worldwide.[1] Several drugs have been developed/repurposed for the treatment of COVID-19; ideally, a drug that is inexpensive and widely available would be accessible for a large group of individuals.[2-4] Ivermectin seems to fulfill the aforementioned criteria and is currently being investigated as a repurposed drug candidate for the treatment of COVID-19.[5]

In evidence-based medicine, randomized controlled trials (RCTs) are the “gold standard” for evaluating the efficacy about therapeutic interventions. Commendably, several RCTs have been performed to evaluate the efficacy of ivermectin in the treatment of COVID-19. RCTs that are performed based on standards of high methodological quality are considered to be at the top of the hierarchy of evidence. Unfortunately, vast variations in the methodological quality of RCTs have been observed.[6] During the COVID-19 pandemic, there are temptations to bypass the standards required for the conduct of RCTs, as observed for the RCTs of hydroxychloroquine for the treatment of COVID-19.[7] This pandemic has brought suffering and death to populations everywhere, and we turn to RCTs for a solution to end the pandemic. Nevertheless, it seems that lessons have not been learned where defects in the performance of RCTs involving hydroxychloroquine, especially in terms of sample size requirements, had been repeated during the design and conduct of RCTs of ivermectin for the treatment of COVID-19.

A key item in the design of RCTs is sample size determination, which should be based on the relative difference in the event rate/proportion between the experimental group and the control group without interventions (normal population), as well as the statistical significance level and the desired statistical power.[8] The P value (alpha value), which has been conventionally set at the 0.05 level in clinical research, defines the accepted risk of type I error (ie, the risk of falsely detecting an effect). On the other hand, the probability of committing a so-called type II error (ie, the ability to find a statistically significant result when it exists) is given by the beta value. The statistical power is the probability of avoiding type II error (ie, 1 minus the beta value) and is typically set to 80% or more for clinical trials. Therefore, when the sample size, and hence the statistical power, is inadequate, there is a higher likelihood that a true difference may remain undetected, compared to when the statistical power is adequate. Mortality benefit is one of the most, if not most, important indicators in the clinical trials that would lead to the widespread recommendation of the tested interventions in clinical practice guidelines. Importantly, mortality involves no subjectivity in its ascertainment. It is a common occurrence in patients with COVID-19, especially those of severe disease, and is of broad importance to patients and the medical community. Thus far, we observed that the published RCTs investigating ivermectin use in COVID-19 had been underpowered to detect mortality benefits.

In the recent systematic review and meta-analysis[9] on the effect of ivermectin toward the risk of mortality in patients with COVID-19, the authors included 17 RCTs, of which 9 RCTs reported a nonzero control mortality events, and were being pooled in their meta-analysis. Across all the 9 trials (Table 1) with nonzero control mortality events, the investigators of each trial either did not report a priori sample size calculation or did not consider the mortality outcome during a priori sample size calculation (ie, mortality not considered as the primary outcome). Therefore, the precision of the mortality estimates should be questioned in these trials because they may have suffered from insufficient statistical power due to insufficient sample sizes. Since the a priori sample size calculation based on estimated difference in mortality outcome was not available in the 9 trials, we could not determine whether the statistical power was sufficient in each trial.

The RECOVERY trial, which investigated the effect of dexamethasone as compared with usual care in patients with COVID-19, reported a mortality estimate of 0.83 (95% confidence interval 0.75–0.93), which subsequently led to the recommendation of dexamethasone by the World Health Organization (WHO) for patients with severe COVID-19.[10] Indeed, this was the only therapeutic agent thus far recommended by the WHO. We considered this mortality estimate as the benchmark and thus used the same estimate (0.83) reported in the RECOVERY trial for dexamethasone to calculate the sample size required to achieve at least 80% power and 90% power, respectively, across the ivermectin trials. While a mortality estimate of 0.85 has been considered an optimal measure of acceptable efficacy for therapeutic intervention in patients with COVID-19, we used a more conservative estimate (0.83) based on the findings from the RECOVERY trial on dexamethasone.[10,11] We noticed that none of the trials have a sufficient sample size to detect a proportional reduction of mortality event by 17% (relative risk of 0.83) between ivermectin and control. The power of these trials ranged merely between 5% and 10% (Table 1). Therefore, the mortality estimates produced from these ivermectin trials might not be precise; in other words, these trials are considered to be confident of only 5%–10% to detect significant mortality benefits.

Table 1.

Proportion of mortality, reporting of power calculation, actual sample size, sample sizes needed to obtain 80% or 90% statistical power, and actual power in RCTs involving the use of ivermectin for the treatment of COVID-19.

Items	Actual proportion of mortality (control arm) (%)	Presence of significant mortality benefits	Power calculation	Actual sample size		Sample size needed to achieve relative risk of 0.83*				Actual power (%)
				Intervention	Control	≥80% power		≥90% power
						Intervention	Control	Intervention	Control
Elgazzar et al (retracted)	12.0	Yes	NR	200	200	3686	3686	4936	4936	10
Lopez-Medina et al[12]	0.5	No	NR	200	198	98,964	98,964	132,509	132,509	5
Mahmud et al[13]	1.7	No	NR	183	180	28,785	28,785	38,542	38,542	6
Ravikirti et al[14]	7.0	No	NR	55	57	6645	6645	8897	8897	6
Galan et al[15]	21.7	No	NR	52	115	1388	2776	1851	3702	8
Gonzalez et al[16]	16.2	No	NR	36	37	2612	2612	3498	3498	6
Hashim et al[17]	8.6	No	NR	70	70	5324	5324	7128	7128	6
Okumuş et al[18]	30.0	No	NR	30	30	1201	1201	1608	1608	7
Niaee et al[19]	18.3	Yes	NR	120	60	3356	1678	4514	2257	9

Two-sided significance level at 0.05.

NR, not reported.

From a statistical point of view, it is acceptable for the WHO to call for more adequately powered RCTs before the issue of widespread recommendation for ivermectin to be used in patients with COVID-19. This again signifies the importance of sample size estimation before performing clinical trials, especially during the COVID-19 pandemic, where we require quick but also high-quality evidence to inform clinical practice. Perhaps the design of RECOVERY trials should be used as a reference for the future design of randomized trials of ivermectin for the treatment of COVID-19. The RECOVERY Collaborative Group considered mortality outcome in their sample size calculation for every randomized trial that they reported thus far; the sample size estimation was based on a power of at least 90% at a 2-sided P value of 0.01 to detect a proportional mortality reduction of 20% in most trials. Even across the trials that fail to detect significant mortality benefits (Table 2), the actual power of each of the trials was more than 80%, and indeed at least 90% in most trials.

Table 2.

Proportion of mortality, reporting of power calculation, sample sizes needed, actual sample size, and actual power in RECOVERY trials which failed to detect significant mortality benefits.

RECOVERY trial	Actual proportion of mortality (control arm) (%)	Predetermined relative risk	Two-sided significance level	Deisred power (%)	Ratio of control to intervention	Sample size needed		Actual sample size		Actual power (%)
						Intervention	Control	Intervention	Control
Aspirin[20]	17.2	0.875	0.01	90	1:1	3289	3289	7351	7541	99
Azithromycin[21]	19.2	0.800	0.01	90	2:1	2183	4366	2582	5181	95
Hydroxychloroquine[22]	25.0	0.800	0.01	90	2:1	1568	3136	1561	3155	90
Colchicine[23]	20.8	0.800	0.01	90	1:1	2615	2615	5610	5730	99
Convalescent plasma[24]	24.1	0.800	0.01	90	1:1	2174	2174	5795	5763	99
Lopinavir-ritonavir[25]	22.4	0.800	0.01	90	2:1	1804	3608	1616	3424	86

We are delighted by the news that Oxford University has launched a clinical trial to test the efficacy of ivermectin in patients with COVID-19. However, we urge that the conduct of RCTs of COVID-19 in the future, regardless of ivermectin or other therapeutic interventions, should be adequately powered to detect mortality benefits. Taxpayers' money, often used to fund the clinical trials, should not be wasted further for trials with inconclusive results.