Embracing Transparency Through Data Sharing.

Low rates of reproducibility and translatability of data from nonclinical research have been reported. Major causes of irreproducibility include oversights in study design, failure to characterize reagents and protocols, a lack of access to detailed methods and data, and an absence of universally accepted and applied standards and guidelines. Specific areas of concern include uncharacterized antibodies and cell lines, the use of inappropriate sampling and testing protocols, a lack of transparency and access to raw data, and deficiencies in the translatability of findings to the clinic from studies using animal models of disease. All stakeholders-academia, industry, funding agencies, regulators, nonprofit entities, and publishers-are encouraged to play active roles in addressing these challenges by formulating and promoting access to best practices and standard operating procedures and validating data collaboratively at each step of the biomedical research life cycle.

Introduction

Researchers and funding entities rely on the reproducibility of published discoveries to create new lines of research and to translate research findings into therapeutic or other applications.[1] The scientific community, however, has expressed ongoing concerns regarding the lack of reproducibility and translatability of published biomedical research data.[2,3] Data that are not reliable and robust can lead to assumptions that undermine and nullify the validity of subsequent research[2] and create distrust among funding agencies, the research community, and the general public.

A second area of concern, data reuse, relates to how data are shared and/or cited.[4] Encouragingly, the volume of open-access data and the rate of data dissemination have increased and can lead to new discoveries.[5] Potential issues in data sharing include country- and/or agency-specific policies on open data, how to ensure compliance with those policies, and the existence and adherence to standards necessary to produce reusable data. The key for data reusability is the annotation of the data, also called metadata, in a way that clarifies how the data have been produced and what the data exactly are. Producing reusable files requires effort from the researchers and should be encouraged/rewarded. This is especially relevant in the case of big data in system science. Hence, the balance between developing standards that can endure in evolving (and therefore complex) fields and minimizing the burden of compliance is delicate and is key for the adoption of any standard.[6]

Initiatives within organizations such as the US National Institutes of Health (NIH)[7] and the UK Wellcome Trust,[8] as well as efforts by some scientific journals, are aimed at improving the reproducibility of experimental data[9] and promoting access to data in the open-access literature[10] and code from computational analyses.[11] In particular, considerable attention is being paid to antibodies and animal models.[12] It has proven difficult to translate findings from animal models to achieve a better understanding of human disease. The challenge should be attributed not only to interspecies differences but also to a lack of rigor in study design, model validation, or the usage of (partially) inappropriate animal models, resulting in low reproducibility of studies.

This article is based on an exhibitor-hosted program session presented at the American College of Toxicology’s 38th Annual Meeting in November 2017. Topics covered during the meeting included important considerations for enhancing the reproducibility and translatability of animal models, the importance of specimen preparation and antibody validation in improving the reproducibility of data collected from animal models, and potential avenues to sharing research transparently.

Reproducibility 2020: Progress and Priorities

Global Biological Standards Institute (GBSI) is a nonprofit entity founded in 2013 to improve the quality of preclinical research by advancing best practices and standards. Stakeholders include academic institutions, regulatory and funding agencies, industry, policy and professional organizations, and entities engaged in developing standards and promoting quality. To promote the reproducibility of research findings, GBSI supports standards initiatives in cell line authentication, antibody validation, and laboratory automation.

Multiple published analyses have reported low rates of reproducibility in preclinical studies, and the cost of US research that cannot be replicated has been estimated at $28 billion annually (Figure 1). The major causes of irreproducibility have been attributed to errors and inadequacies in study design, reagents and reference materials, laboratory protocols, and data analysis.[13] The core drivers contributing to irreproducibility are reporting and publication bias, underpowered studies, a lack of open access to methods and data, and a lack of clearly defined standards and guidelines in areas, such as reagent validation.[13] Growing concerns in the scientific community regarding funds spent on research that cannot be reproduced have prompted the NIH to suggest actions designed to promote reproducibility. The proposed interventions include training in good experimental design, checklists for reviewers of grant applications, and an online forum, PubMed Commons, which was developed as a pilot project to discuss published articles.[3,7] PubMed Commons, however, was recently discontinued after 5 years of existence due to a lack of usage, with comments submitted on only 6,000 of the 28 million articles indexed in PubMed. It will be important to follow alternatives to PubMed Commons that will allow scientists to share their comments, either on open peer-review platforms or, possibly, on other forums.

Figure 1.

Categories of errors contributing to irreproducibility in preclinical research. Adapted from Freedman et al.[13]

Antibodies are one class of reagents that contribute substantially to irreproducibility of research results. Typically, these widely used protein-binding reagents are validated and characterized rigorously in clinical studies but rarely in preclinical research, owing to a combination of a lack of standards, user apathy, and inconsistent quality assurance and control practices in manufacturing.[14,15] Even when antibodies have been validated and characterized, they may be misused by researchers with poor training in laboratory methods, resulting in unreliable and irreproducible findings. The GBSI is working to develop a qualitative scorecard for assessing antibody performance in techniques, such as Western blotting, immunohistochemistry, immunoprecipitation, and enzyme-linked immunosorbent assay. The intent of the scorecard is to provide a first-of-its-kind system to assess antibody performance using a point structure to reward the additional validation steps performed as well as the antibody’s specificity and sensitivity to its intended target.

Another problem area affecting reproducibility is that variability in collection protocols and storage conditions of biospecimens can affect testing outcomes. Ideally, investigators would follow validated standard operating procedures (SOPs). However, there is currently a lack of SOPs, and specimens are often collected without using an SOP at all. When SOPs are used, they may not be adhered to consistently or made available to other research groups. Results may not be comparable where different protocols were used. Repositories such as BioSpecimen Commons (https://biospecimencommons.org/) offer researchers a database of SOPs for collection and analysis of a variety of biospecimen sources and types. Researchers can share and compare methods to ensure the use of appropriate samples and sampling techniques.

To improve reproducibility in nonclinical research, and by extension, the quality of scientific data, all stakeholders must play active roles. Funders should require prescreening of the proposed study design, reagent validation, cell line authentication, and open access to the protocols used, raw data, and publications. Research institutions should implement training programs for graduate students and postdoctoral researchers while holding their laboratory directors accountable for appropriate supervision. Journals should require detailed methods and more stringent reporting guidelines in research articles. The industry should provide validation data, including the results of internal replication procedures. Scientific societies should establish area-specific standards and competencies. All stakeholders should be involved in training and in post-publication review. These collective efforts can improve research quality and enhance reproducibility.[12]

Reproducibility in Basic and Preclinical Research: Enhancing the Quality and Translatability of Animal Models

Two types of research utilize animal models: fundamental research, which includes hypothesis-driven mechanistic work that defines new concepts or identifies potential drug candidates, and nonclinical research, which includes proof-of-concept studies and safety/toxicity studies of new drugs or devices. Interspecies differences are a challenge in generalizing the findings of nonclinical research, and low reproducibility of studies with animal models of disease renders the findings less applicable. The reasons for low reproducibility include underpowered studies, poor experimental design, confirmation bias resulting from a lack of randomization and blinding, neglecting to account for sex-specific effects, inappropriate statistical analysis, and pressure to publish only positive results.[16] Reproducibility, however, is fundamental to maintaining funder and stakeholder trust and to justifying the funding of studies that may lead to drug development. Practices that result in poor reproducibility may lead to time and money spent on research that cannot be reproduced. Another unintended consequence is subjecting patients to clinical trials of agents that may not be effective. Figure 2 displays a schematic showing the 13 common steps of the research life cycle, each of which offers opportunities to improve the reproducibility of biomedical research.

Figure 2.

Common steps in the research life cycle. Adapted from Freedman et al.[12] PI indicates the principal investigator.

Research in animals has long been a cornerstone of nonclinical investigation and is based on shared disease mechanisms and pathways. Nonetheless, modeling human disease in animals presents researchers with multiple challenges, including interindividual variability among humans owing to genetics, environmental contributions, age, diet, lifestyle, microbiome, and health status. Another challenge to translatability of nonclinical research findings is within-species differences in model animals.[17] Using inbred strains can minimize this variability, cut down on the number of animals required to detect differences between exposure groups, and maximize statistical power, as increased genetic diversity requires a larger sample to maintain power. However, maintaining genetic stability in colonies of inbred animals can become difficult over time. Translatability from animal models to humans can be done, but it has limitations. Minimizing bias to increase internal validity (ie, within an experiment, in a specific laboratory) can result in specific and reductionist experimental systems, which in turn lowers the external validity (ie, across laboratory and experiments) and generalizability of the findings.[17]

Importantly, before addressing the details of study design and conduct summarized below, it is of foremost importance to ensure that the animal models used are validated and predictive to address the clinical questions at hand (ie, that they are fit-for-purpose).[18] One cannot expect that all clinical criteria are replicated perfectly in a single model, and gaining a detailed understanding of the strengths and limits of a model is crucial.

Reproducibility in nonclinical research involving animals can then be enhanced by accurate record-keeping, the cornerstone of Good Laboratory Practices; maintaining genetic stability in the experimental animal colony; using precise definitions and standard nomenclature in protocols and publications; critically reviewing the experimental design, power analysis, variables, metrics, and data analysis; applying appropriate statistical methods; and reporting complete and transparent findings. Improved reproducibility will extend the utility and predictive value of animal models of disease.[17]

INTERVALS: A Data- and Results-Sharing Platform Can Improve Transparency in Industry-Funded and Conducted Research

Sharing data in a way that enables reanalysis and reuse is not yet done in a systematic manner, despite the fact that it would clearly benefit the scientific community and society in general, especially when it relates to the scientific assessment of consumer products, including drugs. When the quality of science is questioned due to the affiliation of the scientists, the funding source, or the research topic, sharing data and results transparently is even more critical. For example, tobacco harm reduction is a controversial topic that would benefit greatly from the disclosure of methods, data, and results. Tobacco harm reduction at the population level depends in part on the availability of modified risk tobacco products (MRTPs) and their acceptance and use by smokers who do are unable or unwilling to quit smoking cigarettes.[19] Candidate MRTPs must undergo a scientific assessment process that includes comparison of their biological impact with that of cigarettes in nonclinical and clinical studies.

INTERVALS (https://www.intervals.science/) is an online platform developed by Philip Morris International Research & Development to demonstrate the scientific rigor, thoroughness, and precision required to assess the toxicology of candidate and potential MRTPs while enabling reuse of data sets, encouraging external verification of testing strategies, and informing the scientific community.[20]

In order to address reproducibility concerns, INTERVALS was built using the latest standards in data sharing and reproducible research, including FAIR (Findable, Accessible, Interoperable and Reusable) data principles, to gather detailed information on the design and conduct of studies. This should enable easy review of methods and results as well as reuse of data and the generation of new hypotheses.

The INTERVALS platform allows researchers to find relevant information on studies, (more) detailed protocols, and, most importantly, interoperable data files in a single platform to allow independent reanalysis of key findings, meta-analyses, and efficient data reuse. The INTERVALS toolbox also includes direct links to computational resources, such as AeroSolved (http://www.aerosolved.com/), a computational fluid dynamics code developed to simulate the generation, transport, evolution, and deposition of complex aerosol mixtures.

INTERVALS also encourages communication by enabling constructive feedback on studies and protocols and will foster education by providing reference texts and media on diverse topics relevant to tobacco harm reduction and toxicology assessment of inhaled consumer products. With additional developments, INTERVALS is set to become a public repository for nonclinical and clinical data obtained during investigations into the toxicological effects of acute exposure and the course of disease progression following chronic exposure.[20] INTERVALS can serve as a hub for a community of scientists from industry, academia, nonprofit organizations, foundations, regulatory bodies, and publishers with a common interest in harm reduction (Figure 3).

Figure 3.

INTERVALS, a platform to enable transparency in tobacco harm reduction research.

Certainly as important as sharing data and results, the verification of methods used to generate them, as well as in-depth peer review of study design, results obtained, and interpretation thereof, are highly important. The sbv IMPROVER project (https://www.sbvimprover.com/)[21] was built to apply a crowdsourcing strategy to verify scientific processes and results in systems biology research in an industrial context. It has already proven useful for the collaborative discovery of signatures for specific diseases[22] or exposure status[23] and to investigate issues of general interest, including the translatability of rodent data to humans.[24] To date, the sbv IMPROVER crowdsourcing collaborations have included more than 600 scientists and have led to the successful completion of 4 challenges and 3 datathons as well as the publication of 15 manuscripts in peer-reviewed journals. To complement this challenge-based benchmarking of methods, in-depth peer review of studies, managed by a third party, are conducted. Panels of experts organized in consecutive peer-review panels are given access to study reports and/or publications as well as to the INTERVALS platform to get the best possible overview of the research. They can anonymously and independently (via web) give their unbiased opinion on study design choices, study conduct appropriateness, quality of data, and objectivity of results interpretation.

Summary

Improving the reproducibility of research findings should be possible with the adoption of validated reagents, rigorous study design and analysis, adherence to SOPs, verification of methods with tools such as sbv IMPROVER, and transparency in sharing data and methods with platforms such as INTERVALS. Achieving high levels of reproducibility will require the cooperation and adoption of best practices by all stakeholders: researchers, academic institutions, funding entities, journals, industry, regulatory agencies, and nonprofit and professional groups. The establishment and adherence to validation and verification standards would optimize the use of research funds and facilitate the leveraging of research findings to meet therapeutic targets. This may be accomplished with a collaborative and transparent approach in industry-funded and conducted research to enable evidence-based decision making.