Assessing the Completeness of Reporting in Preclinical Oncolytic Virus Therapy Studies.

Irreproducibility of preclinical findings could be a significant barrier to the "bench-to-bedside" development of oncolytic viruses (OVs). A contributing factor is the incomplete and non-transparent reporting of study methodology and design. Using the NIH Principles and Guidelines for Reporting Preclinical Research, a core set of seven recommendations, we evaluated the completeness of reporting of preclinical OV studies. We also developed an evidence map identifying the current trends in OV research. A systematic search of MEDLINE and Embase identified all relevant articles published over an 18 month period. We screened 1,554 articles, and 236 met our a priori-defined inclusion criteria. Adenovirus (43%) was the most commonly used viral platform. Frequently investigated cancers included colorectal (14%), skin (12%), and breast (11%). Xenograft implantation (61%) in mice (96%) was the most common animal model. The use of preclinical reporting guidelines was listed in 0.4% of articles. Biological and technical replicates were completely reported in 1% of studies, statistics in 49%, randomization in 1%, blinding in 2%, sample size estimation in 0%, and inclusion/exclusion criteria in 0%. Overall, completeness of reporting in the preclinical OV therapy literature is poor. This may hinder efforts to interpret, replicate, and ultimately translate promising preclinical OV findings.

Introduction

Incomplete and non-transparent reporting of animal experiments contributes to the irreproducibility of basic science studies.2, 3 Such irreproducibility may help explain high attrition rates in drug development. To improve the level of reporting rigor and improve reproducibility, the NIH has identified essential items to be reported for all preclinical experimental animal research. This core set of preclinical reporting guidelines (NIH-PRG) lists items that should, at a minimum, be included in a preclinical publication. To properly evaluate, interpret, and reproduce an experiment’s findings, readers need a clear understanding and appreciation for how the experiment was conceived, executed, and analyzed. This information is necessary to judge the validity of findings. Incomplete and poor reporting does not allow for an accurate appraisal of the experiment’s value, and important findings (positive or negative) could be missed. As demonstrated in clinical research, the quality and rigor of research improves when the reporting of key methodological items improves.7, 8

Cancer immunotherapy is a rapidly growing field. Oncolytic, or “cancer-killing,” viruses (OVs) comprise a promising therapeutic platform that can elicit anti-cancer immune responses.10, 11, 12, 13 Despite the potential benefits of OV therapy and a multitude of preclinical candidates, it should be noted that there is currently only one OV approved for human use by the U.S. Food and Drug Administration and European Medicines Agency (Imlygic for advanced melanoma, approved in 2015). The current state of OV preclinical study design and reporting is unknown, as is whether this could explain in part the relatively small number of approved candidates, despite the first reported studies of the modern OV era being published nearly 30 years ago.

Our primary objectives were to review and assess the rigor in the design and completeness of reporting of preclinical in vivo OV therapy studies by applying the recommendations of the NIH-PRG. Furthermore, our study sought to produce an evidence map highlighting the recent global state of OV research and identify commonly studied viruses, cancers, and animal models. To address this, we used rigorous methods commonly used for systematic reviews, including developing an a priori protocol, a systematic search strategy to identify eligible articles, and screening of articles and extraction of data performed independently and in duplicate. Our appraisal of basic experimental design concepts and key reporting elements aims to increase transparency in preclinical OV research and to improve “bench-to-bedside” translation for OV immunotherapies.

Results

Study Selection

Our search identified 1,554 records. Title and abstract screening excluded 1,244 records, with a subsequent full-text screening excluding an additional 74 records. Study screening and selection are summarized in Figure 1. Two-hundred thirty-six articles were included in our review; the full list and individual data of the included studies can be found on Open Science Framework (OSF; https://osf.io/j2dwm/).

Figure 1

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Flow Diagram for Study Selection

Epidemiology of Studies

Articles came from 20 different countries, based on the corresponding author’s residency at the time of publication (Figure 2). Most common were the United States (n = 72, 31%) and China (n = 63, 27%). The articles were published in 2016 (n = 169, 72%) and the first 6 months of 2017 (n = 67, 28%) in 85 journals. The three most commonly acknowledged sources of research funding were government bodies (n = 196, 43%), academic institutions (n = 107, 24%), and foundations/charities (n = 94, 21%; total sources of funding N = 455). The most commonly reported outcome was tumor size/burden (n = 204, 32%), followed by animal survival (n = 116, 18%), viral infectivity (n = 109, 17%), and host anti-tumor response (n = 94, 15%; total number of reported outcomes N = 630). Seventy-six (32%) articles had titles that clearly indicated that they contained preclinical experiments. The full results for study design and publication characteristics can be found in Table S1.

Figure 2

Country of Publication

Information based on the corresponding author’s residency at the time of the included article publication (image created using Tableau Software, Seattle, WA, USA).

In total, 26 different viral platforms were used. The most common was adenovirus (n = 104, 43%) followed by vaccinia virus (n = 34, 14%) and herpes simplex virus (n = 27, 11%; N = 241). The viral treatment most commonly used was intratumoral injection (n = 176, 68%) of an actively replicating virus (n = 182, 77%) in a monotherapy regimen (n = 152, 59%).

Cancerous cell lines from 23 distinct types of tissue were used (N = 293). The most common were colorectal (n = 42, 14%), skin (n = 35, 12%), breast (n = 33, 11%), liver (n = 28, 10%), brain (n = 25, 9%), and lung (n = 25, 9%). This corresponds with the incidence of these cancers in humans. The majority of cancer models used heterotopic (n = 170, 66%; N = 259), xenograft (n = 153, 61%; N = 252), and implant (n = 235, 99%; N = 238) methods.

The animal models were based on four species, with the majority of experiments using mice (n = 231, 96%; N = 241). In total, 26,044 animals were used; 25,516 of these were mice. Explicit ethics approval for animal use was referenced in the majority of articles (n = 172, 73%). The housing and handling of animals was described in 104 articles (44%). The full results for viral, cancer, and animal model data can be found in Table S2.

Assessment of Reporting

Completeness of reporting against each of the seven NIH-PRG domains was assessed. Each section is labeled by the domains it addresses, and a brief summary of the items recommended by the NIH-PRG is provided to guide and orient the reader. The full results for our reporting assessment can be found in Figure 3 and Table S3.

Figure 3

Reporting Assessment Results

Completeness of reporting across all included studies (N = 236) against the deconstructed NIH preclinical reporting guidelines (NIH-PRG) where n is the number of times each item was reported. Green and red correspond to an item being reported or not reported, respectively.

The Listing of Reporting Guidelines

The use of community-based nomenclature and reporting standards are encouraged in the NIH-PRG. One article (0.4%) listed the use of reporting guidelines (Animal Research: Reporting In Vivo Experiments [ARRIVE]) during study design and manuscript preparation.

Measurement Techniques

Authors are required to report on whether their findings were substantiated under a range of conditions (e.g., different viral dosages and different models). This was reported in 225 articles (95%). Furthermore, the number of subjects used and measurements performed should be sufficiently described, such that a distinction between biological and technical replicates can be made. One-hundred five articles (44%) reported the number of times a measurement was repeated for at least one outcome (e.g., tumor size was measured with calipers three times at each time point). Two articles (1%) reported the number of times a measurement was repeated for all experimental outcomes. Similarly, the number of animals used to measure each outcome was reported in 25 articles (17%).

Statistics and Sample Size Estimation

Statistical tests used, the exact value of n, and defined measures of central tendency and dispersion must be included. Statistical tests used were listed in 225 (95%) articles, the exact value of n in 148 (63%) articles, and measures of central tendency and dispersion in 203 (86%) and 206 (87%) articles, respectively. All four items were reported on in 116 articles (49%).

To determine an appropriate number of animals, an a priori power calculation is required. The primary outcome, sample size used, and the rationale for the sample size must be reported. A primary outcome was explicitly stated in 12 articles (5%). A sample size calculation was listed in 10 articles (4%); of these, 4 articles (2%) included the method of calculation (e.g., software package). No articles reported all items related to sample size estimation.

Randomization and Blinding

To reduce selection bias, the randomization of animals and the method of randomization must be reported. This is only applicable when a study includes multiple experimental arms, which occurred in 234 articles (99%). Random group assignment was stated in 103 articles (44%); of these, the method of randomization (e.g., use of a software package) was described in 2 articles (1%).

In addition, to reduce performance and selection bias, the NIH-PRG recommends that authors report whether experimenters were blinded to group assignment and outcome assessment, respectively. Blinding of personnel conducting experiments was listed in 7 articles (3%) and blinding of personnel assessing outcomes was listed in 15 articles (6%). Complete blinding (i.e., reporting of all items) was listed in 5 articles (2%).

Inclusion and Exclusion Criteria

Criteria for the exclusion of any data (e.g., animals, results) must be reported to minimize selection bias. Furthermore, authors are expected to include information regarding the total number of animals originally procured for the study. The total number of animals used was included in 7 articles (3%). The exclusion of any data, or lack thereof, was listed in 11 articles (5%). To address selective outcome reporting, the experimental design (as described in the Materials and Methods section) was compared to the reported results; 228 articles (97%) listed all the findings. Furthermore, the NIH-PRG requires any pilot or preliminary experiments, as well as any null or negative results to be reported. Pilot studies were reported in 27 articles (11%). Null or negative results were explicitly described in 96 articles (41%). No articles reported on all of the items recommended in the inclusion/exclusion criteria domain.

Discussion

This review was a comprehensive appraisal of the recent preclinical in vivo OV immunotherapy literature. Included studies were published in nearly 100 journals and originated from 20 countries. Twenty-six different viral platforms (and hundreds of genetically modified strains) were used. Twenty-three cancer models (and hundreds of cell lines) were investigated.

Our reporting assessment indicated that basic components of experimental design, such as randomization and blinding, as well as key elements of study methodology, such as sample size estimation and a priori-defined inclusion/exclusion criteria, are poorly reported. Of the 21 items identified from the NIH-PRG recommendations, the vast majority were absent across included studies. The statistics domain was partially reported, with 49% of studies reporting all items. Across the other six NIH-PRG domains complete reporting was 2%, at most. Collectively, these results demonstrate that the quality of reporting in the preclinical in vivo OV therapy literature is poor and the publication of the NIH-PRG has not yet had a substantive influence.

Numerous guidelines for thorough in vivo preclinical study design and reporting exist. This includes the extensive ARRIVE preclinical reporting guidelines that were published in 2010 (and included all elements of the NIH-PRG). Those guidelines were endorsed by hundreds of journals (including many in our sample), learned societies, and funders; however, adherence has been poor.20, 21, 22 In response, the simplified NIH-PRG were created in order to focus on seven key items to report; our study demonstrates that adherence to even these core reporting elements continues to be poor. Several studies have demonstrated common trends of poor reporting. Similar to our study, Macleod et al. showed that reporting both on randomization and the method of randomization was quite low (1.8%) in in vivo studies published by the Nature publication group; interestingly, after mandatory completion of a reporting checklist was instituted, the rate increased to 11.2% in these journals. In other studies, randomization was reported between 0% and 16% of included articles, depending on the journal and topic.20, 22 Comparable results have also been found for blinding, with Macleod et al. finding 4% of articles in their sample reporting on blinding of either experimenter or outcome assessors (which increased to 22.8% after institution of a mandatory checklist). Baker et al. found approximately 20% of journals reported on blinding in their sample of studies. It is important to note that, even if a study did not employ randomization, blinding, sample size calculations, and other factors, the final publication could still easily adhere to the guidelines by transparently reporting that these methods were not used.

The Importance of Methodological Rigor and Transparent Reporting

The importance of rigorous study design and transparent reporting needs to be encouraged. Omission of key concepts in the preclinical setting has been associated with the exaggeration of effect sizes, type I (i.e., false positive) and type II (i.e., false negative) errors, and overestimation of the potential for successful bench-to-bedside translation.24, 25, 26 Incorporating methodological rigor into study design and reporting has been shown to reduce bias in the clinical setting.27, 28, 29, 30 Accordingly, many commentators argue that fundamental concepts of clinical trial methodology, such as randomization and blinding, should be incorporated into preclinical experiments to increase study validity. These practices allow other researchers to evaluate the risk of bias in study reports: randomization addressing selection bias and blinding addressing performance bias. Within the cancer field, the importance of these design elements and reporting is evidenced by recent preclinical systematic reviews and meta-analyses of sunitinib and sorafenib. These studies suggested that poor design and incomplete reporting of preclinical studies likely contributed to overestimated effect sizes in animal experiments relative to subsequent clinical studies in human patients. Within this context, our assessment of the OV literature is an important one that should be considered as efforts to clinically translate OV therapies continue. Within the present study, important methods to reduce bias, such as randomization and blinding were reported in only a minority of studies. It is unclear whether these items are not considered and are therefore omitted from study design or they are performed and simply not reported—a push toward complete reporting would shed light on this.

Strengths and Limitations

The choice of the NIH-PRG for our reporting tool and the method through which the recommendations were deconstructed and items operationalized into a checklist are supported by the expertise of the multidisciplinary research team, including basic scientists, who performed this study. Portions of this checklist were previously used by members of our group. Furthermore, to ensure that the relevant study characteristics were correctly collected, our team included several OV experts (C.I., J.S.D., R.A., N.E.-S., R.K., and B.A.K.).

Regarding potential limitations, the validity of included studies and their risk of bias was not evaluated; this limitation reflects reporting being the primary focus of our study. If risk of bias was evaluated, we would anticipate that most studies would be deemed “unclear risk of bias” in a majority of domains, due to a lack of reporting. Second, it is possible that researchers are aware of reporting guidelines and rigorous study design, but interpreted items differently or simply did not report measures that were performed. Third, authors from countries outside of the United States and those who do not receive funding from the NIH may be unaware of these guidelines. Finally, a number of articles (n = 29, 12%) were published in Oncotarget, a journal that was recently delisted in MEDLINE. However, when comparing the level of reporting in Oncotarget to all other articles, we found no meaningful differences (Table S4).

Future Steps

We believe the impetus for change will need to come through enforcement by external sources, such as funding agencies and journals (editorial boards, in particular).21, 23 Examples include mandatory checklists at the time of submission or requirement that the study be preregistered on an open-source database prior to study commencement (similar to clinical trials).

Our study demonstrates that researchers as a community are either not yet aware of reporting guidelines or have not yet appreciated their significance. That is, they may be aware of some of these concepts (e.g., randomization and blinding), but may not appreciate the consequence that less rigorous design decisions have on results (e.g., exaggerating effect size). Here, we posit a few explanations. First, a contributing factor may be the lack of available formal training opportunities for basic scientists concerning these design and reporting elements. Clear guidelines on how these methods should be conducted and reported need to be established. Nonetheless, even in the absence of implementation of these methods, authors should be able to report that they were not used. Second, the impracticality of implementing these measures in some situations should be considered. For example, a single experimenter cannot be blinded to group allocation or it may be impossible to fully blind experiments (e.g., blinding may not be possible in animals treated with OVs, as they develop clearly visible signs associated with the virus). Again, in these situations, a lack of blinding could still be reported, for transparency. Third, studies adhering to a more rigorous design may require more resources (e.g., extra personnel to blind appropriately), access to which may vary widely among research groups and implementation of which may not be rewarded or compensated for by funding agencies at present. This, however, still does not interfere with transparent reporting regarding the application of these design elements. Last, it is possible that the lack of uptake of reporting guidelines, by other researchers and journal editors discourages any change of practices common in the laboratory. Without tangible incentives, the time required for researchers to implement more rigorous study methodologies and fully report studies may be outweighed by pressures to publish rapidly.

Conclusions

This review is an extensive reporting assessment against an important set of published guidelines in a growing field of cancer research. Given the critical need for new cancer treatments, efficient allocation of resources, and reproducible experiments, it is crucial that preclinical in vivo OV research be conducted and reported in a transparent, methodical, and rigorous way. The items investigated here are fundamental to clinical research and have demonstrated benefits in preclinical research. Future work will need to determine how these guidelines and their underlying principles should be applied to preclinical research.

Materials and Methods

Protocol

Using the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocol (PRISMA-P) statement as a guide, a study protocol was constructed and deposited on OSF (OSF; https://osf.io/j2dwm/). During completion of our review, there were no deviations from the protocol. The protocol was publicly posted concurrent with the commencement of data extraction. Although not a systematic review, the PRISMA statement was used in the drafting of this manuscript.

Eligibility Criteria

We included all preclinical in vivo studies that used an OV therapy (prophylactic or interventional) to treat cancer, published between January 2016 and June 2017 (18 months). There were no limitations on viral platform, cancer model, animal model, comparisons, outcomes, or experimental design. Only original research, in the form of full-text English publications, was included. Abstracts, letters, reviews, and commentaries were excluded. Any in vitro, ex vivo, or clinical experiments were excluded.

Search Strategy and Article Screening

In consultation with an information specialist, a systematic search strategy was developed (OSF; https://osf.io/j2dwm/), and searches of MEDLINE and Embase were performed. Titles that met our eligibility criteria were identified and uploaded to an audit-ready, cloud-based software (Distiller SR; Evidence Partners, Ottawa, ON, Canada). Two independent reviewers performed the process of study selection. Title and abstract screenings were performed independently and in duplicate using an accelerated screening method (one reviewer required to include, two reviewers required to exclude). A subsequent full-text screening was performed in duplicate by two independent reviewers (consensus required to exclude). Any conflicts were resolved through discussion with a third team member.

Data Extraction

Full-text articles which met our inclusion criteria were retrieved and uploaded to Distiller SR. All extractors (N.L.W., G.J.L., J.L.M., and I.C.) were affiliated with the Ottawa Hospital Research Institute. To ensure an adequate level of reviewer agreement, all extractors participated in a pilot training exercise. Each extractor was provided with a training document and independently extracted five articles. Feedback from a core group member (M.M.L.) was given, and a further five articles were independently extracted. At this point, the training exercise was completed, as inter-rater agreement met our pre-defined level of 80%. Two trained members of the extractor team independently extracted data from each article in duplicate. Data were collected concerning common study characteristics (e.g., country of corresponding author, journal, funding source, virus administered, cancer studied, and animal model used) and the level of reporting against the NIH-PRG domains (e.g., randomization and blinding). Conflicts were resolved through consensus discussions between extractors or with a senior investigator when needed. Online methods and supplemental information were retrieved whenever referenced.

Reporting Quality Assessment of Included Studies

The NIH-PRG recommendations are grouped into seven domains: (1) use of community-based reporting standards, (2) distinction between biological and technical replicates, (3) statistics, (4) sample size estimation, (5) randomization, (6) blinding, and (7) inclusion/exclusion criteria. Each of these domains contains a multifaceted recommendation that addresses multiple points. To assess the level of reporting, each domain was deconstructed into unidimensional items. These items were operationalized into 21 “yes” or “no” questions, each addressing a single point. (The full list of questions is contained in our protocol.) For example, the domain of randomization was deconstructed into questions regarding whether random group assignment was reported and whether the method of randomization was described (Figure 4). All items mentioned in the NIH-PRG were considered relevant to our reporting assessment. This 21-item checklist served as our reporting assessment tool.

Figure 4

Constructing the Reporting Checklist

The NIH Preclinical Reporting Guidelines (NIH-PRG) domain of randomization was deconstructed into two unidimensional items, and each was operationalized into a “yes” or “no” question.

Data Synthesis

For each item addressed by the NIH-PRG recommendations, the level of reporting was expressed as the total number of times each item was reported (n) across all studies (N). This descriptive statistic, frequency count, is expressed nominally (n out of N) and as a percentage (n/N). No formal statistical analysis was performed.

Author Contributions

Conceptualization and methodology: M.M.L., J.P., K.D.C., J.-S.D., R.A., J.K., C.I., and D.A.F.; validation: N.E.-S., R.K., and B.A.K.; formal analysis: M.M.L., N.L.W., and D.A.F.; investigation: M.M.L., N.L.W., G.J.L., J.L.M., I.C., J.P., K.D.C., J.-S.D., R.A., J.K., N.E.S., R.K., B.A.K., C.I., and D.A.F.; resources: M.M.L. and D.A.F.; data curation: N.L.W., G.J.L., J.L.M., and I.C.; writing–original draft preparation: M.M.L., N.L.W., and D.A.F.; writing–review and editing: all authors; and funding acquisition: M.M.L. and D.A.F.

Conflicts of Interest

The authors declare no competing interests.