COVID-19 coronavirus research has overall low methodological quality thus far: case in point for chloroquine/hydroxychloroquine.

OBJECTIVES/
BACKGROUND AND OBJECTIVES: Prior epidemics of high-mortality human coronaviruses, such as the acute respiratory syndrome coronavirus (SARS-CoV or SARS-1) in 2003, have driven the characterization of compounds that could be possibly active against the currently emerging novel coronavirus SARS-CoV-2 (COVID-19). Presently, no approved treatment or prophylaxis is available for COVID-19. We comment on the existing COVID-19 research methodologies in general and the published reporting. Given the media attention and claims of effectiveness, we chose chloroquine and hydroxychloroquine, in combination with azithromycin, as an area of COVID-19 research to examine. METHODS/STUDY DESIGN AND
SETTING: MEDLINE and EMBASE electronic databases were searched from 2019 to present (April 3rd, 2020) using a mix of keywords such as COVID-19 and chloroquine and hydroxychloroquine. We also searched the largest clinical medicine preprint repository, medRxiv.org.
RESULTS: We found 6 studies, 3 randomized control trials and 3 observational studies, focusing on chloroquine and hydroxychloroquine (with azithromycin). We critically appraised the evidence.
CONCLUSION: We found that the COVID-19 research methodology is very poor in the area of chloroquine/hydroxychloroquine research. In screening the literature, we observed the same across COVID-19 research in relation to potential treatments. The reporting is very poor and sparse, and patient-important outcomes needed to discern decision-making priorities are not reported. We do understand the barriers to perform rigorous research in health care settings overwhelmed by a novel deadly disease. However, this emergency pandemic situation does not transform flawed methods and data into credible results. The adequately powered, comparative, and robust clinical research that is needed for optimal evidence-informed decision-making remains absent in COVID-19.

Clinical decision-makers must be informed by the best, most trustworthy, highest-quality, robust evidence. This translates into how much confidence we can have in the research findings and thus be optimally informed for decision-making.

The estimates of effect in clinical research depend on the underlying research methodology.

COVID-19 disease is presenting global health systems, clinicians, and patients grave challenges.

No treatment or prophylaxis currently exists for COVID-19.

The overall body of COVID-19 research is very flawed methodologically and underpinned mainly by uncontrolled confounded evidence.

An examination of hydroxychloroquine–azithromycin research findings due to the recent media focus revealed very-low-quality methodology underpins the research. Without a focus on the results, the very serious decisions being made by societies about this drug (and combination) as well as other drugs in COVID-19 research is hampered by the very poor research methodolgies.

Vast amounts of time and resources are being allocated to COVID-19 research, and being potentially squandered.

Flawed methodology and suboptimal reporting of research findings could lead to biased estimates of effect (over-estimates or under-estimates).

This could lead to treatment decisions that are not optimal based on biased estimates which could potentially harm the patient.

This article provides specific suggestions for improving on the COVID-19 research methods and reporting across the breadth of COVID-19 research, with a focus on the issues that researchers must consider in their methodology and reporting if we are to have confidence in the estimates of effect (their findings).

Importantly, a failure to consider harms in research could be detrimental to the patient. A drug may be relatively safe for one medical condition but unsafe for another and as such, must be carefully examined. This article focuses on the potential harms when therapeutic agents such as hydroxychloroquine and azithromycin are being considered.

Research thus far on finding an optimal therapeutic agent(s) for COVID-19 could be hampered by methodologically flawed research.

COVID-19 researchers must immediately and acutely focus on improving their methodology and reporting. Journal editors and the peer-review process must work to safeguard against sub-optimal research being published.

Background

No current treatment or prophylaxis has proven to be effective in coronavirus disease 2019 (COVID-19), and patients receive either symptomatic treatment for milder presentations or more advanced life-support strategies in moderate to severe cases (appropriate oxygenation strategies that could include mechanical ventilation and ECMO within a hospital ICU setting). As the global community eagerly awaits credible scientific solutions for this pandemic, researchers and scientists are under much pressure to identify effective therapeutic and preventive strategies for COVID-19.

This commentary focuses on the quality of current COVID-19 research. We used published clinical studies on chloroquine/hydroxychloroquine as an example to demonstrate some of the methodological concerns around research currently conducted in the field.

In recent weeks, academic journals and public media published and disseminated information on the use of quinine derivatives (i.e., chloroquine and hydroxychloroquine) [1,2] for the treatment of COVID-19. Chloroquine has long been and still is used to prevent and treat malaria, whereas hydroxychloroquine was first used to prevent and treat malaria and is currently used to treat rheumatoid arthritis, some symptoms of systemic lupus erythematosus, juvenile idiopathic arthritis, and other autoimmune diseases.

Although ongoing clinical trials are testing the efficacy and safety of several treatments for COVID-19, including chloroquine and hydroxychloroquine, there is not yet credible evidence from clinical trials on the efficacy and safety of those agents in COVID-19. Most of the data released or published thus far on chloroquine/hydroxychloroquine, and COVID-19 research in general, are imprecise and at high risk of biased estimates of effect. Here lies our concern.

This pandemic emergency is fraught with tremendous uncertainty about the evidence on treatment or prophylaxis. There are many unknowns, and the massive demand for evidence on the treatment of a novel disease such as COVID-19 may be unintentionally affecting studies’ design and conduct. Furthermore, it may inadvertently affect the peer-review and publication process, leading to significant methodology gaps and overall lower quality evidence on COVID-19. These gaps lead to less-informative studies, loss of precious time, and valuable resources. Therefore, current research should balance feasibility and efficiency against methodological rigor and carefully address methodology gaps as much as possible. To support universal clinical decision-making and minimize harm, the research community should focus on conducting and publishing trust-worthy evidence. Hypothesis generating studies are welcome and essential but should be explicitly framed as such, acknowledging their limitation, and not be used for critical decisions, at the national or global level.

Furthermore, experimental studies aiming to confirm or refute these observations should be supported. Until there is a certainty, patients deserve, for themselves and for the sake of future patients, to be randomized in ethical and well-designed clinical trials. Our concerns on the state of the chloroquine and hydroxychloroquine research thus far expand to the quality of research methodology we see spanning across COVID-19 research. We sought to draw attention to this.

What clinical evidence does exist for chloroquine and hydroxychloroquine? Not much

To optimally inform our appraisal of the quality of COVID-19 research, we searched the MEDLINE and EMBASE electronic databases to identify clinical studies on the use of these agents in COVID-19 (2019 to April 3, 2020), finding 564 initial citations that were screened for eligibility, yielding six clinical studies judged eligible [[3], [4], [5], [6], [7], [8]]. As part of this database searching, we also examined the largest clinical medicine preprint repository, medRxiv.org, on a daily basis.

Based on the quality of the evidence to date, can a clinical practice guideline issue any recommendations on the use of chloroquine or hydroxychloroquine alone or combination with other treatments? The case has not been made either way for these quinine derivatives. The body of evidence thus far has been from both in vitro studies [2,9,10] and clinical studies [[3], [4], [5], [6], [7], [8]] of suboptimal methodological quality. The emerging clinical studies such as case series, cohort studies, and randomized controlled trials (RCTs), are all inconclusive; while some may suggest some benefit, the entire body of evidence does not reach the level of certainty and confidence that is required to justify the use of quinine derivatives as a treatment for COVID-19 [[3], [4], [5], [6], [7], [8]]. As exciting as some of these results may appear, the underlying research methodologies are often flawed, and as such, the reported results cannot be trusted. A most recent French prospective study [8] on the use of chloroquine in combination with azithromycin completely contradicts a prior French study asserting the benefit of the combination [6,7]. When critically appraised using the appropriate risk of bias tools, all studies [[3], [4], [5], [6], [7], [8]] have been classified as high risk of biased estimates of effect [11,12] (Table 1 ).

Table 1

Human hydroxychloroquine (HCQ) COVID-19 studies published as of April 6, 2020 [[3], [4], [5], [6], [7], [8]]

Study author, yr, study design, location (reference #)	Sample size; mean/median age; % male	Intervention; comparator	Reported outcomes	Critical appraisal
RCTs
Chen, 2020, RCT, China [3]	30; 15 HCQ, 15 control; 48.5 mean; 70%	HCQ 400 mg per day for 5 days plus SoC, control received SoC	Negative conversion rate	Small sample size, small events, unclear reporting of methods, unclear/absent randomization, concealment, blinding, suboptimal outcomes, sparse reporting on methods
Chen, 2020, RCT, China [4]	62; 31 HCQ; 31 control; mean 44.7 (SD 15.3); 46.8%	5-day HCQ (400 mg/d), control received SoC	Time to clinical recovery (TTCR), clinical characteristics, and radiological results, adverse events	RCT, small sample size, small number of events, unclear reporting of methods, suboptimal methods, suboptimal outcomes, sparse reporting on methods
Huang, 2020, RCT, China [5]	22; 44.0 mean (36.5 to 57.5); 59.1%	twice-daily oral of 500 mg Chloroquine (n = 10) vs. 400/100 mg Lopinavir/Ritonavir (n = 12) for 10 days	Disease progression by RT-PCR, lung pathology with CT, fever, respiratory rate, oxygen saturation and adverse events	RCT, small sample size, small events, unclear reporting of methods, unclear/absent randomization, concealment, blinding, suboptimal outcomes, sparse reporting on methods
Observational studies
Gautret, 2020, open-label nonrandomized observational study, France [6]	42; 26 HCQ, 16 control; 45.1 ± 22.0 (mean/SD); 41.7%	HCQ 600 mg daily 6 d n = 26 (AZ added depending on clinical presentation), control n = 16 (6 lost in f/up due to cessation of treatment, 1 died, 3 to ICU)	Virologic cure, length of hospital stay, mortality, adverse events	Observational, small sample, > 20% attrition in intervention arm, control group taken from different care center, unclear accounting of patients lost/removed from analysis, heterogenous allocation of cotreatments, decisions based on clinician judgment, unadjusted analysis, sparse reporting on methods; considered hypothesis generating
Gautret, 2020, case-series observational, France [7]	80; 52.5 median, 52.5%	200 mg of HCQ three times per day for 10 days combined with AZ (500 mg on D1 followed by 250 mg per day for the next 4 days)	Need for oxygen therapy; transfer to the ICU after at least 3 days of treatment, contagiousness (PCR and culture) and length of stay ID ward	Observational study, no control arm; small sample size, small number of events, unadjusted analysis, no matching, stratification, restriction, sparse reporting on methods; considered hypothesis generating
Molina, 2020, consecutive case-series observational, France [8]	11; 58.7 mean, 64%	HCQ 600 mg/d for 10 days and AZ 500 mg Day 1 and 250 mg days 2 to 5	Virologic cure (positive tests)	Observational study, no control arm; small sample size, small number of events, unadjusted analysis; considered hypothesis generating

Abbreviations: AZ, azithromycin; CT, computed tomography scan; HCQ, hydroxychloroquine; ICU, intensive care unit; ID, infectious disease; RCT, randomized controlled trial; RT-PCR, reverse transcription polymerase chain reaction.

Overall, the methodologies of the published studies are not robust, and the results are tempered largely by selection bias and residual confounding bias. At the design level, most studies lack the randomization, concealment of the generated sequence, and blinding/masking needed to generate sound evidence, when they are RCTs and not observational uncontrolled single-arm case series. At the analysis level, they lack the standard steps taken to minimize confounding such as prospective design, statistical adjustment for prognostic factors, (propensity) matching, or stratification [13]. Even with procedural and statistical controls, the evidence emerging from the COVID-19 research in general and using hydroxychloroquine research thus far as the example, cannot optimally achieve prognostic balance as would a large sample-sized RCT with optimal methodology. Our examination has found the reporting to be very sparse and lacking of the explicitness that is warranted, and the patient-important outcomes needed for decision-making are often not studied or not reported. We do understand the urgency to identify effective treatments, as well as the barriers to perform rigorous research in health care settings overwhelmed by an unprecedented workload and a novel deadly disease. However, these unprecedented and unfortunate circumstances do not transform flawed data into sound results. The adequately powered, comparative, rigorous effectiveness research that is needed for optimal evidence-informed decision-making remains absent in COVID-19 research. Researchers need to prioritize minimizing bias by randomizing a large enough number of patients and masking the treatment allocation as much as it is feasible and fully accounting for all the patients enrolled in the study following the intention-to-treat analysis principle (Table 2 ). These critical components of high-quality, trustworthy research are required to generate high confidence that the estimates of effect reflect the true effects.

Table 2

Possible remedies to improve study methodology

Methodological concerns	Possible solutions or suggestions
Small sample size, small events	Enroll larger numbers of patients, involving multiple centers or adopt harmonized protocols to allow subsequent pooled analysis/meta-analysis; needed for COVID-19 but applicable to all research
Sparse reporting on observational study methods	Conform to the STROBE guidance for reporting observational studies [14]
Sparse reporting on RCT methods	Conform to the CONSORT guidelines for reporting RCTs [15]
Unclear/absent randomization, concealment, blinding in RCT	Allocate individuals to intervention or control therapies based on a random process, in which the patient and the clinical providers are blind to treatment allocation
Confounding bias in observational studies	Use of statistical methods to adjust for confounding factors; propensity score matching, stratification [13]
Data attrition (patients lost to follow-up)	Minimize missing data; use of intention-to-treat analysis, with appropriate handling of missing data; consider complete case analysis as sensitivity analysis

Abbreviations: CONSORT, Consolidated Standards of Reporting Trials; RCT, randomized controlled trial; STROBE, STrengthening the Reporting of OBservational studies in Epidemiology.

At this time, more than ever, the trustworthy, high-quality, robust, comparative evidence from ethical RCTs is urgently needed to assess patient-important outcomes, including mortality, morbidity, need for life support, safety, and toxicity, informing on the safe use of chloroquine or hydroxychloroquine (with and without azithromycin) in people with COVID-19. This goes for all therapeutics under present examination for COVID-19 and with an acute focus on possible harms to the patient.

Again, we make this clarion call across the breadth of COVID-19 research, and across all research in general. Of course, these studies will need to be fast, and even better, conducted in a flexible framework (such as adaptive trials) able to accommodate the adding of and switching to different treatments as soon the ones under study are proven ineffective or more promising alternatives are suggested. Consideration of master protocols (harmonization of efforts) and adaptive trial designs become very important [16]. The World Health Organization (WHO) has advocated for randomized multicenter adaptive clinical trials to evaluate the efficacy and safety of investigational therapeutic agents in combination with standard of care for the treatment of hospitalized patients with novel coronavirus disease (COVID-19) [17]. Researchers aligning their methodology to WHO's master protocol will surely improve quality of the COVID research [16,17]. In addition, global research groups conducting clinical trials would help in disseminating their results by adhering to the CONSORT (Consolidated Standards of Reporting Trials) checklist for optimal clinical trial reporting [15] and the STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) statement that outlines the guidance for optimal reporting of observational studies [14]. This is critically important and especially in this COVID-19 urgency when very serious national and global public health decisions are being made based on what information is shared (or not shared). Evidence exists to show that clinical trial results are biased when the trials utilize inferior methodology or report findings without satisfactory description of the methods used. Failing to conceal allocation of the generated sequence has been linked to an exaggeration of the effectiveness of 30% or greater [15]. An improvement of methodologies and reporting will also allow for the systematic review and meta-analytical pooling of study evidence once studies are similar enough across the methodologies and range of patients, interventions, and comparators. This potential summarization where it is possible, will translate into improved precision of estimates of effect and enhanced confidence in the estimates for decision making.

The urgency of the COVID-19 pandemic situation would also make it appropriate to be creative and move beyond the classical modalities and boundaries of academic research: what if a mobile app was made available by a respectable institution to allow randomizing any small number of consenting patients, collecting a small set of relevant covariates (age, sex, days since diagnosis, relevant comorbidities), by any doctor willing to participate in a chloroquine or hydroxychloroquine trial wherever the treatment is available for compassionate prescription (i.e., most of the world)? What if mortality in the two groups, masked as being treatment and control, was posted on a website every 100 patients reaching the outcome (recovered or dead) to transparently show if equipoise persists? What if the data set at 1,000 patients, or every 1,000 patients if needed, was made publicly available for highly skilled statisticians to propose their interpretation? We would get 1,000 patients every few days, and we would be receiving clinically sound results faster than any traditional study framework. Of course, this flexibility may not warrant publication in a top tier journal but could save thousands of lives. We need to use the most optimal methodology and not compromise on rigor but be willing to think outside of the box.

The outcomes being reported in the COVID-19 research thus far are informative but are not ideally patient-important that could help in patient and clinician decision-making. In addition, research question gaps are glaring and future methodologically strong comparative research of quinines for COVID-19 (as well as other drug treatments) should assess the following: 1) the net benefit ratio of chloroquine or hydroxychloroquine alone or in combination with other interventions; 2) specific subgroups classified by age, stage and severity of illness, and other potential effect modifiers; and 3) optimal dosing and timing of dosing. We do believe there is still full equipoise justifying the continued investigation of the role of chloroquine/hydroxychloroquine in studies in patients hospitalized with COVID-19. Caution is urged regarding large scale uptake of the treatment as open-label use of the drug.

Why extreme caution is urged in using chloroquine, hydroxychloroquine, or any drug? We could be doing more harm than good

Primum non nocere—first, do no harm. Potential adverse effects, toxicities, and medication interactions must remain key considerations when using any drug, and chloroquine or hydroxychloroquine, in combination with azithromycin, are no exception. Indeed, the evidence suggests that both drugs (independently) prolong the QT interval leading predisposing patients to serious arrhythmias [[18], [19], [20], [21]]. This is of particular concern when coadministering macrolides (azithromycin) [22], also known to affect cardiac electrophysiology. Indeed, the use of doxycycline [22] has been proposed in azithromycin's place. The U.S. Food and Drug Administration has even warned the public that azithromycin (Zithromax or Zmax) can potentially cause irregular changes in the electrical activity of the heart and could lead to a potentially fatal irregular heart rhythm [23]. In this regard, researchers recently looked at 84 COVID-19 infected patients who were administered a hydroxychloroquine/azithromycin combination [24]. They found that the QTc was prolonged maximally after 3–4 days from the beginning of treatment, and in 25 patients, the QTc increased more than 40 ms. They also found that in 9 patients (11%), the QTc increased to >500 ms, indicative of a high-risk group for arrhythmia.

This issue of potential harmful effects and the urgent need for high-quality, methodologically robust studies also comes from a recent prepublication (not yet peer-reviewed) of in vitro activity of hydroxychloroquine or chloroquine in combination with metformin (used in treatment of type 2 diabetes to lower blood sugar in humans) in mice [25]. Research studies reported that when hydroxychloroquine or chloroquine was combined with metformin as a possible anticancer drug, 30–40% of all mice died. Here is our concern, for we are referring to drugs that have a manageable safety profile as antimalarials, and have been used for other conditions such as lupus, but do seem to impact the cardiac electrophysiology in some manner. The same goes for azithromycin.

Until more high-quality evidence is available and particularly that excludes harm with these drugs in COVID-19 patients, caution is urged especially with combination therapy outside RCTs. There must also be acute pharmacovigilance and monitoring of adverse drug reactions with regard to these and other drugs in COVID-19 patients given that so much in unknown.

Some countries have chloroquine or hydroxychloroquine readily available, in some cases as an over-the-counter medicine as they are required to treat common conditions. National authorities should be mindful of these situations and take measures to govern the use of these medicines and prevent dangerous self-medication. On the equity side, there is a concern of massive purchasing, including for both personal use and research, leading to possible shortage of supply. This can lead to treatment shortages for malaria and autoimmune diseases where there is confirmed benefit.

The use of existing drug treatments such as chloroquine and hydroxychloroquine outside of current guidelines and recommendations may result in adverse effects, including serious illness and death, affect patients with other diseases who may benefit from its use, and hinder the ability to conduct clinical trials if there are high demands to sue these agents by clinicians and patients. Toxic results from in vitro studies may not translate to toxicity in humans, but we caution that care must be exercised in extrapolating in vitro results before establishing clinical efficacy and safety [26]. In making our argument about the very poor research methods and reporting, we felt we should comment on the preliminary very troubling indications that COVID-19 related deaths cluster in areas with high levels of poverty and underprivileged populations. If this early observation gets validated, this may signify that COVID-19 is yet another disease exhibiting aspects, at least in part, of poverty and disparity [27]. For example, recent reports of the African-American communities in the USA tragically suffering greater burden of death from COVID-19, is likely due to a combination of depressed socio-economic conditions, health inequities (lack of access), and higher rates of co-morbidities that are partially associated with lower socioeconomic conditions to begin with that have been endemic for decades in these communities. It is possible that poverty and disparity, when coupled to greater comorbidity and a strained healthcare setting, within the context of COVID-19 infection, leads to more severe disease progression and outcomes. This must be urgently studied [28,29].

As health systems struggle to develop care with equity, justice, fairness, and compassion, such potentially unequal force of mortality and morbidity underpins our urgent clarion call to improve the quality of COVID-19 data, projections, research methodology, and published reporting. These are very complex and serious issues facing nations’ populations especially for health care and resource allocation planning that require sound, trustworthy data, evidence, robust analysis, and underlying research methodologies. In summary, although the massive effort in generating and disseminating evidence globally in response to COVID-19 has to be applauded, the accumulated body of evidence thus far can at best be considered hypothesis generating due to the methodological flaws across COVID-19 clinical research. The methodological quality of ongoing and planned clinical research has to be urgently upgraded if the vast amount of time and resources are not to be squandered in this emergency. Until availability of sound results from one or more COVID-19 clinical trial(s) showing a favorable risk-benefit, caution is urged in considering indiscriminate use of these two drugs, alone or in association. Public health authorities are urged to prioritize resources on those interventions that are currently recommended as standard of care and ensure that empirical clinical use or research on these medications does not endanger adequate supply for the patients that need it for conditions in which the efficacy is already known.