What cannabis can learn from Covid: Hydroxychloroquine research suggests the next step for medical cannabis research.

Covid and cannabis

Because of the massive scale of the Covid-19 pandemic, Covid treatment research is subject to intense politicization, frequent media scrutiny, and continued public interest. As thoroughly described in a recent JAMA Viewpoint Article (Califf, Hernandez, & Landray, 2020), public scrutiny into drug development research has the potential to introduce a new set of incentives into the research process, which can, in turn, disrupt science-based regulation and the delivery of evidence-based treatments. These dangers became abundantly apparent through the US experience with hydroxychloroquine. When influencers and politicians began to endorse hydroxychloroquine as a treatment for Covid based upon early observational and preclinical studies, many in the public, including patients, physicians, and policy-makers, were quick to embrace hydroxychloroquine as an effective treatment, even though observational and preclinical studies are incapable of causally proving a drug's safety or efficacy. This unearned enthusiasm for hydroxychloroquine led to shortages for those who required the drug for approved indications (Jakhar & Kaur, 2020) and even cases of poisonings (Erickson, Chai, & Boyer, 2020). Another observational study (which was later retracted) subsequently found a positive association between hydroxychloroquine use and mortality as well as other adverse events, which may have made recruitment for hydroxychloroquine randomized controlled trials more challenging (Califf et al., 2020). Concerningly, the cacophony of contradictory observational and preclinical evidence presented in the media led some members of the public to adopt a dogmatic attachment to the drug's effectiveness or ineffectiveness in line with their political identity (Fuhrer & Cova, 2020). Since hydroxychloroquine was first suggested as a possible Covid treatment, a large-scale RCT, similar to what would be required for FDA drug approval, along with five smaller RCTs have all failed to find that hydroxychloroquine is an effective treatment for Covid. The authors of the large-scale RCT stated on June 5th, “this result should change medical practice worldwide and demonstrates the importance of large, randomised trials to inform decisions about both the efficacy and the safety of treatments” (RECOVERY Investigators, 2020). Despite this causal evidence, many in the public still believe that hydroxychloroquine is an effective treatment (PolitiFact, 2020), detracting from other potentially effective preventive measures and treatments and fueling conspiratorial theories about pharmaceutical interventions overall (Sattui et al., 2020).

Medical cannabis research shares many characteristics with hydroxychloroquine research. Owing to the political and social history of cannabis, the safety and efficacy of medical cannabis and cannabis-derived products is a political, as well as scientific, discourse. Many patients, physicians, and policy-makers want cannabis to be a safe and effective medication and are willing to endorse cannabis’ safety and efficacy with little supporting evidence (Robledo & Jankovic, 2017). Media outlets frequently and widely cover the results of cannabis research, and like with hydroxychloroquine, many in the public are primed to accept favorable findings, regardless of their methodologies, as truth. Because observational and preclinical studies generally take less time and cost less money than large-scale RCTs, interested parties, particularly “Big Marijuana” companies, are able to sponsor dozens of non-causal studies and publicize their findings, providing more ammunition to their political allies (Caputi, 2020). The dissonance between positive observational trial results and federal cannabis prohibition have caused many in the public to form their own conclusions about the underlying motives for cannabis policy (Castañeda, 2020). Some become distrustful of the actors and systems instituting prohibition, including policy-makers, pharmaceutical regulators, and the pharmaceutical industry.

Because their situations are similar, medical marijuana researchers can potentially learn some lessons from the experience of hydroxychloroquine researchers. Perhaps none is more important than the notion that researchers and regulators should only accept results from large-scale RCTs as evidence of a drug's safety and efficacy regardless of political pressure or competing findings from other forms of research. Much of the harms related to hydroxychloroquine could have been averted if physicians and researchers insisted on proof of safety and efficacy from large-scale RCTs and if the FDA had imposed greater restrictions on use related to non-approved indications. Similarly, an insistence on large-scale RCTs to confirm the safety and efficacy of cannabis and cannabis-derived products, as well as stricter regulatory controls on unsubstantiated health claims made by marijuana marketers, could avert potential public health harms related to inappropriate medical cannabis use. Further, to the extent that cannabis and cannabis-derived products are truly safe and effective for certain conditions, large-scale RCTs can confirm these benefits and give policy-makers, physicians, and patients the confidence to allocate appropriate treatment.

Why large-scale RCTs matter

To the layperson, other forms of research can appear to have equivalent or even greater value compared to large-scale RCTs (Murphy, 2005). This is particularly true for observational studies. Observational studies can have thousands more participants than even large RCTs. They often use complex-sounding statistical techniques, like propensity score matching or growth models, while RCTs are statistically straightforward. Observational studies involve “real users” as opposed to clinical study test subjects. While some are aware of the concept of confounding, many can be appeased by adjustment for confounders in the analytical rather than design phase of the study.

Despite their veneer of credibility, observational studies have no causal interpretations and, instead, can easily provide biased effect estimates. Large-scale RCTs are the only method that can reliably provide causal estimates of an effect (Ioannidis, 2013; Little & Rubin, 2000; Rubin, 1978).

Consider, for example, a treatment that has no effect. The treatment is tested in 100 different trials, each with 1000 participants. The relationship between the treatment and the outcome of interest is confounded by 20 variables (10 numerical, 10 binomial), which could be, for example, gender, race, age, height, weight, and blood pressure. For simplicity, assume that each confounder is randomly distributed around 0.4 (40% probability for binomial variables) for those not receiving the treatment and 0.6 (60%) for those receiving the treatment. The effect size of each confounder on the outcome ranges between 0.5 and 5 (scaled by the number of confounders), with an equal probability of either increasing or decreasing the outcome. To avoid overfitting, I include a random error centered at 0 with a standard deviation of 10.

Observational researchers rarely know all potential confounders or even have access to data on all known confounders. Assume, then, that the researcher knows and collects data on, on average, 30% (6) of the confounders, and adjusts for all of them. Many researchers would consider an observational study with six confounders “well-controlled”, and yet it is reasonable that 10 continuous and 10 dichotomous variables confound a given relationship. However, if we simulate this circumstance, approximately 85 of the 100 trials would produce an estimate significantly different from 0, even though the treatment truly has no effect (Fig. 1 ). Each of these 85 trials may be publishable in separate peer-reviewed journal publications, but none of them would be accurate. Indeed, if the research or publication process is biased in one direction, it may appear that the literature consistently shows a relationship in that direction.

Fig. 1

This figure shows the results of 100 simulated studies, each with 1000 participants. The true effect size of the treatment is 0. There are 20 confounders (10 continuous, 10 binomial). For each trial, confounders are randomly assigned such that untreated individuals have a mean of 0.4 and treated individuals have a mean of 0.6. The effect size of each confounder on the outcome ranges from 0.5 to 5 in either direction. The left panel shows ordinary least squares (OLS) effect size (ES) estimates (scaled by the standard deviation of the outcome) if, on average, 30% of confounders are known and adjusted for, analogous to an observational study. The right panel shows OLS estimate effects if all confounders are adjusted for, analogous to a large-scale RCT. Estimates are green if their 95%CI includes the true effect size (0) and red if not. Simulations are ordered by their effect size in the left panel.

Large-scale RCTs eliminate the dangers emerging from unknown confounders. Because participants are randomized to receive either the treatment or a control and because the sample size of both groups are large, all third variables, including known and unknown confounders, balance between groups.1 In other terms, it is not possible for potential outcomes to correlate with the treatment when the treatment is randomly assigned. In the above example, this is analogous to adjusting for all 20 known and unknown confounders. If we simulate that case, approximately 95 of the 100 trials produce results consistent with the treatment's true effect size.

Other study designs have related problems. Small-scale RCTs, for example, do not necessarily balance confounders or potential outcomes; without help from the law of large numbers, the different treatment arms can, by chance, be correlated with known and unknown confounders and potential outcomes (Zhong, 2009). Many initial safety trials do not have a comparison group altogether, and so the effect can be confounded by time or disease progression. Animal models and preclinical studies frequently fail to produce comparable results in humans due to the immeasurable number of confounding biological systems (Wendler & Wehling, 2010). For these reasons, large-scale RCTs are almost always required for drug approval by regulatory bodies in developed countries around the world. It should be noted that, in the United States, the 21st Century Cures Act has allowed for some flexibility in the study design and statistical analyses of trials used to test new medical devices and drugs’ safety and efficacy (e.g., using Bayesian analysis of clinical trials) (Food & Drug Administration, 2020b; Pallmann et al., 2018). Even in these cases, however, FDA guidance on Bayesian analysis affirms the importance and necessity of random assignment to treatment and a sufficient sample in late-stage investigational new drug trials (Food & Drug Administration, 2020b).

It is true that, strictly speaking, large-scale RCTs are not the only way to establish causal evidence (Craig et al., 2012; Little & Rubin, 2000). For example, natural experiments, which exploit random or quasi-random assignment occurring in the real world, can have many of the same benefits as RCTs (e.g., confounders are balanced among treatment groups) and potentially better generalizability. However, true natural experiments, particularly for the use of pharmaceutical products, are rare. Further, when a natural experiment is found, one needs to be convinced that assignment to the treatment is truly random, or at least orthogonal to potential outcomes conditional on adjusting for observed confounders, before accepting the results as causal. In virtually all cases, that argument requires at least a small leap of faith (Rosenzweig & Wolpin, 2000). Consequently, large-scale RCTs are the only study design that can reliably produce causal evidence.

Large-scale RCTs have another key benefit over observational, early clinical, or natural experiment designs: it is challenging for researchers to intentionally bias their studies to find favorable results. For both pharmaceutical and marijuana research, researchers often have a considerable interest – financial, ideological, or otherwise – in producing findings that suggest the drugs they test are safe and effective. Dishonest researchers may, for example, selectively choose which confounders to include in their models in order to find a spurious but statistically significant result (Caputi, 2016). Large-scale RCTs essentially remove the option for researchers to act in this way. Potential outcomes are balanced through the randomization procedure, and so the researcher merely has to perform some simple and straightforward analytics in order to assess whether the drug had an effect or not. She cannot purposely introduce bias into her model by omitting a confounder. Simply put, with large-scale RCTs, there is little room for dishonest researchers to play statistical games with their data.

It should be noted that not all large-scale RCTs are properly formulated or conducted to produce clinically meaningful results, and the mere presence of a study that brands itself as a large-scale RCT is insufficient to determine whether a drug is safe and effective or not (Ioannidis, 2018). For example, in trials that are inappropriately conducted, randomized groups may differ in post-randomization experiences or randomization may not be properly generated at all. Many unregistered RCTs give undue attention to underpowered, post hoc subgroup analyses. Those with for-profit sponsors may be analyzed and publicized in a way that is beneficial to the sponsor's goals. Therefore, the presence of a large-scale RCT should be seen as a necessary but not sufficient condition to determine whether a drug is safe and effective, and regulators should still be responsible for overseeing the validity of trials.

Implications for medical cannabis research

The vast majority of influential evidence regarding the safety and efficacy of medical cannabis is observational, with virtually all the rest coming from preclinical or early clinical trials. Researchers have highlighted several examples where medical marijuana companies have used weak research to convey unsubstantiated health claims or advocate for commercially advantageous policy (Caputi, 2020). For example, ecological studies have often been used to market cannabis and cannabis-derived products as a solution to the opioid epidemic (Hall et al., 2018; Humphreys & Hall, 2020; Humphreys & Saitz, 2019; Shover, Davis, Gordon, & Humphreys, 2019), while observational survey and clinical studies have been used to inappropriately market that cannabis products are effective for the treatment of diseases like HIV/AIDS (Aphria, 2017) and Parkinson's Disease (NORML, 2020). A recent study found that 90% of medical claims on popular websites were not based upon appropriate cause-and-effect evidence (Boatwright & Sperry, 2020). Only two cannabis-derived compounds (Epidiolex and Sativex) for four indications (Lennox-Gastaut syndrome, Dravet syndrome, Tuberous Sclerosis Complex, and neuropathic pain in multiple sclerosis) have received sufficient support from large-scale RCTs (Devinsky et al., 2018; Thiele et al., 2018) to be approved by pharmaceutical regulators (Food & Drug Administration, 2020a); no other compound or indications have received such supporting causal evidence. While 33 US states have legalized medical cannabis for conditions such as post-traumatic stress disorder, depression, chronic pain, and glaucoma, there is no evidence from large-scale RCTs to support any cannabis-derived compound's safety or efficacy for most any of these conditions (National Academies of Sciences, 2017). For example, many have speculated on the therapeutic effects of cannabidiol for pain and anxiety disorders, and cannabidiol and other cannabis-derived products are frequently recommended for these conditions. However, even though a large-scale RCT of cannabidiol for these conditions would be feasible and relatively straightforward, results from no such trials have been published (Boyaji et al., 2020). The field should be committed to evaluating the efficacy of cannabis for these indications in large-scale RCTs.

For some time, it could be argued that reliance on observational studies was a necessity of cannabis research. Regulatory restrictions on cannabis research made it challenging for researchers to get approval to conduct a large-scale RCT, and institutional investors willing to finance the trials were rare. Even in these restrictive circumstances, some pharmaceutical companies managed to conduct large-scale RCTs of cannabis-derived compounds. In recent years, however, many of those restrictions have been lifted. The 2018 Hemp Act allows for research on hemp-derived CBD in the US, and Canada's Cannabis Act legalized cannabis nationwide, including for research purposes. There are several highly profitable “Big Marijuana” companies with valuations over $1 billion USD that could, if willing, finance these large-scale RCT. Despite these changes in circumstance, the field has stagnated in observational research.

A central (but not the only) reason underlying the field's stagnancy is regulatory leniency surrounding medical marijuana marketing. Because medical marijuana companies have been free to advertise and market the health effects of their products without substantiating evidence, they have little incentive to invest in expensive large-scale RCTs (Caputi, 2020). Further, a naturally curious public and generally supportive media have given marijuana companies a vehicle to tilt public opinion without clinical evidence. Researcher and regulatory insistence on large-scale RCTs could compel Big Marijuana companies to invest in robust research that could move the field forward.

The US experience with hydroxychloroquine should remind cannabis researchers of the importance of large-scale RCTs and convince the field to move from a primarily observational discourse to a primarily clinical discourse. This is likely easier said than done for the same reason that enforcing rigorous science related to hydroxychloroquine has proven difficult: cannabis research is not a dispassionate field, which has added a new set of incentives to the drug research process. Many in the public and, indeed, many researchers have preconceived notions about the safety and efficacy of cannabis, and there is no shortage of powerful political and financial interests willing to incentivize studies that reinforce those notions through the most efficient means possible – in this case, observational studies rather than large-scale RCTs. Indeed, Big Marijuana companies, the companies that should be financing large-scale trials, routinely publicize and sponsor weak research to advance their marketing efforts rather than investing in large-scale RCTs that could meaningfully advance the field (Caputi, 2020). Large-scale RCTs are expensive and time-consuming (DiMasi, Grabowski, & Hansen, 2016); with the same resources required to complete one large-scale RCT, a researcher may be able to produce perhaps a dozen observational studies. Because the public often mistakenly assigns large-scale RCTs and observational studies the same value, the incentives inherently favor observational research. However, if researchers and journal editors recognize the external forces informing the medical cannabis debate, admit to and publicize the limitations of observational research, and insist on prioritizing large-scale RCTs, we may uncover the true risks and value of medical cannabis.

Data availability

The data for this paper is all simulated. The code to produce and analyze this data is available at https://github.com/tlcaputi/observational-simulations.

Declarations of Interest

The author reports an equity interest in Data Science Solutions LLC, a public health consulting firm that received a research grant from the National Consumers League, a 501(c)(3) consumer protection non-profit organization, for research related to medical cannabis regulation.