Ivermectin for COVID-19: Addressing Potential Bias and Medical Fraud.

Ivermectin has become a controversial potential medicine for coronavirus disease 2019. Some early studies suggested clinical benefits in treatment of infection. However, the body of evidence includes studies of varying quality. Furthermore, some trials have now been identified as potentially fraudulent. We present a subgroup meta-analysis to assess the effects of stratifying by trial quality on the overall results. The stratification is based on the Cochrane Risk of Bias measures and raw data analysis where possible. The results suggest that the significant effect of ivermectin on survival was dependent on largely poor-quality studies. According to the potentially fraudulent study (risk ratio [RR], 0.08; 95% CI, 0.02-0.35), ivermectin improved survival ~12 times more in comparison with low-risk studies (RR, 0.96; 95% CI, 0.56-1.66). This highlights the need for rigorous quality assessments, for authors to share patient-level data, and for efforts to avoid publication bias for registered studies. These steps are vital to facilitate accurate conclusions on clinical treatments.

In June 2020, ivermectin, a Food and Drug Administration (FDA)–approved antiparasitic drug, was shown to have antiviral effects against severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in vitro [1]. Following this, ~86 clinical trials investigating ivermectin for coronavirus disease 2019 (COVID-19) have been registered globally. In late 2020, clinical trials began reporting very compelling clinical benefits for ivermectin in the treatment of COVID-19. From late 2020 onwards, multiple groups produced meta-analyses that reported that ivermectin had a significant effect on survival, hospitalizations, clinical recovery, and viral clearance [2, 3]. Our meta-analysis was first presented in January 2021 and published in July 2021 [4]. It suggested that ivermectin resulted in a significant 56% improvement in survival, favorable clinical recovery, and reduced hospitalizations. Such optimistic results from multiple meta-analyses have escalated public interest in using ivermectin for the treatment and prevention of COVID-19, despite the World Health Organization (WHO) only recommending its use within clinical trials [5, 6].

However, as with all meta-analyses, a key limitation is the quality and completeness of the available evidence. During our original assessment of studies, standardized Cochrane Risk of Bias measures (RoB 2) had classified several studies as “high risk of bias” [7]. A study by Niaee et al. from Iran that reported a randomized methodology was found to have significant differences in baseline characteristics across treatment arms [8]. This suggests that the participants were not randomized appropriately, which could bias the outcomes. Second, a study by Okumus et al. from Turkey did not provide any information on allocation concealment, and it was unclear if the participants or investigators were blinded, which risks introducing observation bias [9]. Lastly, a study by Hashim et al. from Iraq provided insufficient details on the randomization process, had a lack of clarity on participants who were analyzed, and involved unblinded assessment of a subjective outcome [10].

Furthermore, some studies were then identified to be potentially fraudulent. For example, on July 15, 2021, a study by Elgazzar et al. from Egypt was retracted from the preprint server Research Square due to “ethical concerns” [11]. It has been reported that the data for ~79 participants were duplicates, some deaths were recorded on dates before the trial had started, and instances of plagiarism were identified in the text [12]. Similarly, a study conducted in Lebanon by Raad et al. was also identified to have duplicate data for multiple participants when the patient-level database was analyzed in September 2021 [13]. Before these inconsistencies were identified, the Elgazzar and Raad studies had been included in multiple meta-analyses, which suggested significant benefits for ivermectin in the treatment of COVID-19 [2, 3]. In our original meta-analysis, the Raad study accounted for 11.8% of the effect of ivermectin on hospitalization, and the Elgazzar study accounted for 12.6% of the effect of ivermectin on survival [4].

These instances suggest that the data available to the support the use of ivermectin for COVID-19 are not reliable. In July 2021, after the potentially fraudulent studies were identified, we retracted our published meta-analysis and began working on an updated analysis assessing the effects of stratifying by trial quality on the overall results. Clinical trials evaluating ivermectin for the treatment of COVID-19 had been identified by systematic searching of 8 databases. An in-depth evaluation of study quality was conducted, in addition to the standard Cochrane RoB 2 and CONSORT checklist [7]. First, we evaluated trials based on the effectiveness of their randomization process by comparing baseline characteristics across treatment arms using the chi-square test. Second, randomization dates were checked to ensure that patients were randomized into the treatment arms on similar dates. Third, checks were conducted to evaluate if recruitment to treatment arms was balanced at each investigational center. Furthermore, we analyzed patient-level databases, where available, to check for any evidence of duplicate participants and unexpected homogeneity or heterogeneity. From this, a meta-analysis was conducted with subgroups of clinical trials at different risk of bias levels.

Looking at the key survival outcome, the analysis includes 12 studies with a total of 2628 participants (Table 1) [4]. This included 4 studies at a low risk of bias, 4 with some concerns, 3 at a high risk of bias, and 1 potentially fraudulent study. The analysis demonstrates that on including all 12 studies, ivermectin results in a significant 51% increase in survival (95% CI, 0.28–0.86; P = .01) (Figure 1; Supplementary Figure 1). On excluding the potentially fraudulent Elgazzar study, ivermectin results in a borderline significant 38% increase in survival (95% CI, 0.39–0.99; P = .05) (Figure 1; Supplementary Figure 2). On excluding the high risk of bias studies, ivermectin results in a nonsignificant 10% increase in survival (95% CI, 0.57–1.42; P = .66) (Figure 1; Supplementary Figure 3). Lastly, on excluding studies with some concerns of bias, ivermectin results in a nonsignificant 4% increase in survival (95% CI, 0.56–1.66; P = .90) (Figure 1; Supplementary Figure 4). These observations demonstrate that the significant effect of ivermectin on survival was dependent on the inclusion of studies with a high risk of bias or potential medical fraud.

Table 1.

Studies Included in the Survival Analysis [4]

Study	Risk of Bias Level	Sample Size	Ivermectin Arm	Control Arm
Lopez-Medina et al.	Low risk	398	0/200	1/198
Fonseca et al.	Low risk	168	12/53	25/115
Zoni et al.	Low risk	501	4/250	3/251
Kirti et al.	Low risk	112	0/55	4/57
Rezai et al.	Some concerns	69	1/35	0/34
Abd-Elsalam et al.	Some concerns	164	3/82	4/82
Gonzalez et al.	Some concerns	73	5/36	6/37
Mahmud et al.	Some concerns	363	0/183	3/180
Niaee et al.	High risk	180	4/120	11/60
Hashim et al.	High risk	140	2/70	6/70
Okumus et al.	High risk	60	6/30	9/30
Elgazzar et al.	Apparent fraud	400	2/200	24/200
Total		2628	39/1314	96/1314

Figure 1.

Survival effects of ivermectin.

There are added challenges with clinical trials investigating COVID-19 treatments. In a rapid response to COVID-19, many small-scale studies have been conducted globally for potential agents. However, not all trials have reported findings. An example is a trial for nitazoxanide in Brazil with 600 participants that was completed in October 2020 but has not reported any results yet [14]. Publication bias impacts meta-analyses, with positive and significant results more likely to be reported. Therefore, we believe that by including publication bias, it is even less likely that ivermectin will show significant benefits for COVID-19 treatment. Some nonrandomized trials may also be overinterpreted. For example, in a nonrandomized retrospective cohort study, remdesivir demonstrated an improvement in clinical recovery and reduced mortality risk by 62% [15]. However, when evaluated in the WHO’s randomized placebo-controlled SOLIDARITY trial, remdesivir had little to no effect on mortality [16]. Any initial promising findings from a small number of sources need to be interpreted with caution, studied further, and considered within the wider body of evidence.

The results from this analysis highlight the need for rigorous quality assessments when evaluating clinical trials of drugs for COVID-19. Existing and widely used risk of bias assessment tools are not enough. These tools provide a systematic framework for identifying potential key sources of bias in a trial’s internal methodology but work on the fundamental assumption that a published study is reporting accurate and complete findings. They allow reviewers to make judgments on the assumption that basic standard procedure is followed, the data are real, and no information is being intentionally hidden.

With cases of potential medical fraud now identified, it is essential that access to patient-level databases be provided. If authors fail to provide these data, the study should be considered with a higher index of suspicion. Additionally, it should be mandatory that all registered trials report their findings. We understand that these are substantial changes to established procedures. However, the failure to recognize the potentially fraudulent studies, which led to multiple meta-analyses suggesting significant benefits of ivermectin for COVID-19, indicates that the tools currently used to evaluate the quality of clinical trials are insufficient. These events warrant our stringent recommendations.

Click here for additional data file.