The significant cost of systematic reviews and meta-analyses: A call for greater involvement of machine learning to assess the promise of clinical trials.

BACKGROUND: More than 90% of clinical-trial compounds fail to demonstrate sufficient efficacy and safety. To help alleviate this issue, systematic literature review and meta-analysis (SLR), which synthesize current evidence for a research question, can be applied to preclinical evidence to identify the most promising therapeutics. However, these methods remain time-consuming and labor-intensive. Here, we introduce an economic formula to estimate the expense of SLR for academic institutions and pharmaceutical companies.
METHODS: We estimate the manual effort involved in SLR by quantifying the amount of labor required and the total associated labor cost. We begin with an empirical estimation and derive a formula that quantifies and describes the cost.
RESULTS: The formula estimated that each SLR costs approximately $141,194.80. We found that on average, the ten largest pharmaceutical companies publish 118.71 and the ten major academic institutions publish 132.16 SLRs per year. On average, the total cost of all SLRs per year to each academic institution amounts to $18,660,304.77 and for each pharmaceutical company is $16,761,234.71. DISCUSSION: It appears that SLR is an important, but costly mechanisms to assess the totality of evidence.
CONCLUSIONS: With the increase in the number of publications, the significant time and cost of SLR may pose a barrier to their consistent application to assess the promise of clinical trials thoroughly. We call on investigators and developers to develop automated solutions to help with the assessment of preclinical evidence particularly. The formula we introduce provides a cost baseline against which the efficiency of automation can be measured.

Introduction

Testing preclinical observations in humans poses a critical stumbling block in developing new clinical interventions that benefit patients. More than 90% of the compounds that enter clinical trials fail to demonstrate sufficient efficacy and safety to gain regulatory approval [[1], [2], [3], [4]]. Recently, this issue has raised calls for thorough evaluations of “whether an experimental treatment is promising enough to warrant testing on people” [5]. Systematic literature review and meta-analysis (SLR) methods are one mechanism to synthesize the totality of the current evidence and assess the promise of clinical trials [[6], [7], [8], [9], [10]].

Although not without flaws [11], SLR methods can capture relevant data to summarize different studies and evaluate their efficacy [12,13]. These methods also apply to preclinical evidence identifying weaknesses in animal studies to propose better mechanisms to determine the most promising therapeutic targets and drugs [14,15]. However, despite the usefulness of SLR, these research methods remain time-consuming and labor-intensive [16,17]. In this communication, we quantify their significant expense of SLR to both academic institutions and pharmaceutical companies by applying a new economic formula. We argue for better solutions to reduce the cost and conclude with suggestions of how machine learning might provide the necessary infrastructure and resources to expedite SLR methods.

The purpose of this paper is to introduce a formula for calculating the hidden cost of performing systematic reviews and meta-analyses and to highlight what the cost can amount to for both research institutions and the pharmaceutical industry.

Methods

An economic formula for quantifying the cost of SLR methods

Previous work quantified the labor effort involved in SLRs, but less attention has been paid to the actual dollar costs involved. The estimated labor effort to produce a single SLR ranges from a minimum of 6 months when an investigator devotes 10–20 h per week [18] to, on average, a total of 16 months involving five co-authors [19], or 1–2 years for completing a Cochrane review [20].

The formula we introduce here consists of four main input parameters (Equation (1)).

The formula estimates the total number of SLRs, both published and unpublished via two parameters, (Npub) and (Pr(unpub)) respectively, and quantifies how much it costs to produce each review via the time (Ehrs) and cost-per-person (Cperson). The inclusion of unpublished SLRs in this calculation is critical. Recent work analyzed this issue as it relates to “network meta-analyses,” one of the most rigorous and advanced forms of this type of analysis [21]. The authors found that 44% (76/174) of network meta-analyses done by (or on behalf of) pharmaceutical companies were never published. While this may not hold for academically focused research institutions, it imposes a substantial cost for pharmaceutical companies. Combining the terms of the formula allows us to estimate the annual costs of producing all of the SLRs at an institution.

Taking an average of the labor effort values above, we estimate it takes one scientist 1.72 years (see Appendix) with an average labor cost of $82,090 per year for a scientist [22]. To calculate the expense for a years' worth of SLRs, we estimate the total number of SLRs published at major academically-focused research institutions [23] and the largest pharmaceutical companies by revenue [24]. To do this, we measured the number of SLRs published in the past five years according to PubMed queries for the main three types of comparative studies (“meta-analysis,” “systematic review” and “comparative effectiveness”) for each of the institutions and companies (Table 1). The limitation of this methodology is that if only one of the authors is company-affiliated, the cost per company should be scaled accordingly. We limited our search to the last five years to account for the temporal effects associated with the more recent popularity in these types of publications.

Table 1

The number of comparative studies performed by the top 10 NIH-funded research institutions and the top 10 largest pharmaceutical companies by revenue. Example of PubMed query: (“meta-analysis"[pt] or “systematic review” or “comparative effectiveness”) and (“Johnson and Johnson”[affiliation] OR “Johnson & Johnson”[affiliation]).

Average	NIH-funded research institution	Number of articles in the last 5 years	Company	Number of articles in the last 5 years
	Johns Hopkins University	1362	J&J	90
	University of Michigan	751	Roche	421
	University of Pittsburgh	568	Pfizer	638
	Washington University in St. Louis	540	Novartis	575
	Stanford University	655	Sanofi	263
	University of California, San Francisco	217	GSK	364
	University of Pennsylvania	752	Merck	392
	Massachusetts General Hospital	752	AbbVie	146
	Brigham and Women's Hospital	836	Bayer	232
	University of California San Diego	175	Abbot	203
5 year average	660.80		332.40
1 year average	132.16		66.48

Results

We found that on average, each major academic institution will publish 132.16 and each pharmaceutical company will publish 66.48 SLRs per year (Table 1). According to our formula, each single SLR costs $141,194.80 ($82,090 X 1.72). Given that 44% of meta-analyses in the pharmaceutical industry go unpublished (i.e., Pr(unbpub) is 0.44), we estimate the pharmaceutical industry publishes 118.71 studies, on average, per year. Therefore, the total cost of all SLRs per year for each pharmaceutical company averages $16,761,234.71, and each academic institution averages $18,660,304.77 ($141,194.80 X 132.16), as we assume all studies are published in academia (i.e., Pr(unpub) is 0)).

Discussion

The purpose of this paper is to introduce a formula for calculating the hidden cost of performing SLRs and to highlight what the cost can amount to for both research institutions and the pharmaceutical industry. SLRs are important but costly mechanisms to develop research hypotheses and answer research questions. We argue that automation (e.g., machine learning, artificial intelligence) could significantly lower the cost of SLRs, ensuring these important efforts will not be abandoned due to their expense. The formula we present here provides a cost baseline against which the efficiency of machine learning can be measured. Current efforts to scale the SLR review process manually include the Cochrane group [10] that addresses the scalability of effort by leveraging tens of thousands of volunteers. While estimable, essentially this distributes the workload and associated cost across a large number of volunteers, spreading it out to become more manageable. We argue that, as the number of questions to answer and the size of the literature both increase, this leaves room for automated methods to fill the need.

Considering our formula, we can explicitly tie machine-learning to specific parameters in the formula, which allows us to understand their effect on the overall effort (and cost) of SLRs. Machine learning will affect how people are paid, as tasks become automated away, but that is challenging to forecast – therefore, we will focus our analysis on the hours it saves (Ehrs) rather than the cost-per-person (Cperson). In a simple argument, one can imagine researchers leveraging tools to automatically screen papers [25] for quality, doing in seconds what used to take hours. As another example, following an approach similar to Ref. [26], authors replicated a systematic review in days, when the original work took months [27] - a significant time savings. If these tools are repurposed for preclinical evidence, and not just clinical, the costs associated with Ehrs would be dramatically reduced, allowing SLR to scale with the number of questions researchers could ask.

Limitations

There are limitations to our approach. The cost estimates we present here are based on data from the top 10 NIH-funded research institutions and the 10 largest pharmaceutical companies by revenue. As a result, the cost of systematic reviews and meta-analyses for institutions with less NIH funding or revenue would be less. However, the main purpose of the paper is to introduce a formula for calculating the hidden cost of performing SLRs and to highlight what the cost can amount to. Additionally, automation such as machine learning methods are not perfect – but they improve over time, and we are optimistic they will eventually reach human performance. Automation also cannot yet address more profound issues, such as identifying preclinical studies that are “plagued by poor design, implementation and reporting” [[28], [29], [30]]. But this is not a detriment. Instead, this is an opportunity for researchers and developers to make significant advances in artificial intelligence research. The rise in automation for SLR-related tasks is the only scalable way to understand the increasing volumes of literature [31]. The human capacity, on the other hand, to read and understand the growing body of preclinical evidence is largely set.

Conclusion

The cost of SLRs are significant and may pose a barrier to their consistent application to thoroughly assess the promise of clinical trials. We need better approaches and tools that enable more researchers with limited budgets and time constraints to take advantage of SLR methods. The goal is to make the available evidence more accessible and to assist in the detection of insufficient evidence before the approval and activation of clinical trials. Therefore, we call on investigators and developers to contribute to the development of automated solutions to particularly help with the assessment of preclinical evidence. Such tools could provide investigators, ethical review boards (IRBs), and efforts such as the national SMART IRB Reliance Initiative [31] with an efficient technical solution to better harness the totality of evidence and to focus the investment in clinical trials on those with sufficient evidence.

Funding

This work was supported by Evid Science, a for-profit corporation.

Conflicts of interest

Co-author Matthew Michelson is the CEO of Evid Science, which funded this research. Co-author Katja Reuter does not report any financial or other conflict of interest.