The reproducibility of biomedical research: Sleepers awake!

There is increasing concern about the reliability of biomedical research, with recent articles suggesting that up to 85% of research funding is wasted. This article argues that an important reason for this is the inappropriate use of molecular techniques, particularly in the field of RNA biomarkers, coupled with a tendency to exaggerate the importance of research findings.

Time to listen!

Most researchers are aware that the translation into clinical practice of even the most promising findings of basic research is slow and rare [1] and that the clinical effectiveness of drugs cannot be predicted reliably from early research-based evidence [2]. A string of five articles and an editorial printed in the Lancet in early 2014 addressed this issue, reaching the staggering conclusion that 85% of biomedical funding is wasted on research that is ill conceived, poorly executed, inappropriately analysed, inadequately reported, side-tracked by bias and stifled by red tape.

The first article considers research priorities and questions their relevance for human health in the context of a worldwide investment in biomedical research of around US$240 billion [3]. The second highlights the usual absence of detailed written protocols for and poor documentation of research and comments on the systematic preference for quantity rather than quality, and novelty rather than reliability [4]. The third article discusses the modern approach to the regulation, governance, and management of biomedical research and emphasises how inefficient management can easily compromise the interests of patients and public [5]. The fourth points out that a large percentage of health research-associated protocols, full study reports and participant-level datasets are rarely available, that there is selective reporting of methods and results and that this leads to the introduction of bias, detrimentally effects the care of patients and wastes huge amounts of research funding [6]. The final article re-emphasises the absolute requirement for accurate, exhaustive and transparent reporting and notes that whilst reporting guidelines are important, they are all much less adopted and adhered to than they should be [7]. Finally, the editorial discusses the consistent and colossal failure of initially promising research findings to translate into improvements in health in light of the many, economic, political, social and cultural factors that influence researchers, funders, regulators, institutions and companies [8]. The authors make the revolutionary suggestion that rather than using journal impact factors to assess academics, it might be more constructive to gauge the methodological rigour, transparency of reporting and reproducibility of results of their output, which would of course facilitate the publication of more reliable and biologically relevant data.

Time to act!

Clarion calls don’t come much louder, insistent or with more authority. The authors’ principal recommendations involve proposals to improve every aspect of research project procedures relating to standardisation, governance and the research environment infrastructure:

Enhance translational research outcomes by increasing clarity from researchers, funders and regulators with regards to research proposal justification and transparency of protocols [3].

Make research results more reliable and relevant by ensuring public availability of protocols, analysis plans, raw data and providing continuous training and professional development coupled to reward incentives [4].

Streamline management of biomedical research to make it fit for purpose by harmonising regulations and processes affecting research, together with appropriate research designs and monitoring [5].

Make information more widely available by adopting performance metrics for distribution of research protocols and results, development of standards for data sharing practices and introducing legally enforced policies on information sharing [6].

Ensure that research reports address the needs of research users by encouraging better and more complete reporting and investing in the development and maintenance of an infrastructure for reporting, reviewing and archiving [7].

Is anyone listening?

These are of course very laudable aims and all make perfect sense, yet it is hard not to cast a weary eye over previous such propositions, generally published by the same cohort of authors in similarly prominent high impact factor journals. It really is striking just for how long there have been reports about the poor quality of research methodology, inadequate implementation of research methods and use of inappropriate analysis procedures as well as lack of transparency of reporting. All have failed to stir researchers, funders, regulators, institutions or companies into action.

For example, an earlier, shorter, but similar narrative of the problems facing applied research expressed surprise at the level of waste caused by issues such as poorly designed research studies pursuing the wrong questions, then either delaying or failing to publish in peer-reviewed journals and finally reporting outcomes selectively [9]. The authors’ list of recommendations to address these issues was rather similar to the ones published five years later, and is echoed by several complementary analyses of the challenges facing – omics research with regards to sample size and bias [10], validation practices [11] and design, conduct, and analysis of such studies [12]. Whether these expositions have had much impact is anyone's guess, but the fact that there is an unremitting reiteration of the same issues suggests not. Remarkably in this age of supposed austerity, neither politicians nor the public seem to care that the majority of research effort is wasted. It is shocking that whilst primary health care budgets are being trimmed, patient waiting times are increasing and the medical infrastructure is crumbling, no media investigation scrutinises the huge waste that is so evident to researchers themselves and, if stopped, would bring huge benefits to “our health, our comfort and our grasp of truth” [12]. It is truly extraordinary that whilst most researchers are aware of the fundamental flaws afflicting biomedical research, very few individuals, institutions, journal editors or funding agencies are prepared to put their heads above the parapet of what sometimes looks suspiciously like conspiratorial silence.

Lack of reproducibility

There has been a view for some time that the major problem no longer is the validity of expression measurements, but rather, the reliability of inferences from the data [13]. This conveniently disregards the fact that the results from thousands of these studies remain in the scientific literature and are likely to confuse current opinions and confound future studies. Furthermore, whilst inappropriate data analysis is of course a major problem, the widespread disregard for and dismissal of technical variability continues to be at the heart of much of the reproducibility endemic in biomedical research. This is most obviously demonstrated by the fact that independent researchers cannot reproduce the vast majority of peer-reviewed published work.

Whilst repeatability, the successful replication of results and conclusions from one study in the same laboratory, is important, reproducibility, defined as the independent confirmation of results and conclusions from a study, is the bedrock on which the scientific principle is based. Sadly, it has become clear that the results reported by most publications in the biomedical field do not fulfil that basic criterion. In a recent investigation at least half of researchers surveyed had the personal experience of being unable to reproduce published data and many were unable to resolve the issue even when helped by original authors [14]. In another investigation 11/12 (92%) studies could not be reproduced when the methods outlined in the original papers were replicated exactly, 26/38 (68%) when the methods were adapted and 2/2 (100%) when different methods were used [15]. Incidentally, it is unclear whether the authors analysed 67 or 57 projects, as the numbers from one of their figures add up to 67, whereas those from another add up to 57. A third study could not reproduce 47/53 (89%) of findings even when the laboratories that conducted the original experiments were consulted about their methods and, in some cases, experiment were repeated in the original laboratory [16]. Tellingly, all studies for which findings could be reproduced provided complete information on reagents and authors had paid close attention to controls and investigator bias. An examination of 271 in vivo studies using rodent models or non-human primates concluded that only 60% of the articles included information about the number and characteristics of the animals and only 70% provided detailed descriptions of the statistical analyses [17]. In a similar vein, 54% of all resources referred to in over 200 recent publications in the fields of Neuroscience, Developmental Biology, Immunology, Cell and Molecular Biology and General Biology could not be adequately identified, regardless of domain, journal impact factor, or reporting requirements [18]. Yet another investigation set out to assess the study design, statistical analyses, and reporting of 156 papers published in 2008 on cerebrovascular research. Few papers were hypothesis-driven or properly designed, many did not include adequate information on methods and results and were characterised by poor statistical details [19]. All these events lead to the inevitable conclusion that there is a spectacular lack of attention to detail by researchers, reviewers and editors.

Technical issues

The viewpoint that there is no longer a problem with the validity of expression measurements is also challenged by the clear evidence that molecular techniques can be unfit for purpose. The problems associated with real-time quantitative PCR (qPCR) have been extensively aired [20], [21], [22], [23], [24], [25], [26], [27], [28], but the emphasis on qPCR has concealed the complications associated with the reverse transcription (RT) step, which converts RNA into cDNA. RT is a basic and essential molecular technique that feeds into a number of other methods that have become the mainstay of applications in modern biology, diagnostics, medicine, forensics and biotechnology. This reaction is carried out by a choice of different enzymes with different properties that are not just enzyme-specific but are also dependent on the advanced buffers supplied with the enzymes. There was early recognition that a consideration of mRNA structure is essential prior to any investigation into gene expression [29] and that reverse transcriptases (RTases) differ in their ability to transcribe through the extensive secondary structure in mRNA [30], [31]. This led to the realisation that both technical and biological variability had to be considered when analysing results from RT-qPCR experiments and that the variability of the RT step presented an important impediment to reliable data interpretation [32]. Subsequent reports showed that the method of cDNA priming affects both accuracy and reproducibility of RT-qPCR experiments [33], [34] and that reactions primed by target-specific primers are linear over a wider range than similar reactions primed by random primers [35]. The first empirical evidence for high variability being an inherent property of the RT step was provided in 2004 with two rather important, but woefully overlooked publications. The first showed that RT-qPCR gene expression measurements are comparable only when the same priming strategy, reaction conditions and RNA concentration are used in all experiments [36]. The second reported RT- and target-dependent cDNA yields that differed by up to 100-fold [37], something that also applies to digital RT-PCR [38]. The paper's judicious recommendation that assays should be run at least as duplicates starting with the reverse transcription reaction continues to be ignored, even though the soundness of that advice was demonstrated by a follow-up publication [39]. Yet another paper reported gene-related factors as the primary determinants of RT variability and called for an evaluation of the RT robustness of control genes in RT-qPCR normalisation [40]. A recent paper has demonstrated that the fold changes reported by most RNA biomarkers are well within the range of variability observed when multiple RT reactions are carried out on the same template [41]. The conclusions from these papers are rather stark and place question marks behind many of the results reported not just in the biomedical literature but in the scientific literature as a whole. Unfortunately, recommendations for improvement are universally ignored and a large number of publications that use the RT step are characterised by inadequate and non-transparent reporting and conclusions that are based on expression changes well within experimental variability.

MIQE

The “minimum information for publication of qPCR experiments” (MIQE) guidelines constitute a list of recommendations that target the reliability of qPCR results, with the aim of helping to ensure the integrity of the scientific literature, promote consistency between laboratories, and increase experimental transparency [25]. They are the most widely cited of such initiatives, which include MIAME [42] and MISFISHIE [43] targeted at various molecular techniques. MIQE is based on an earlier proposal aimed at improving the quality of RT-qPCR experiments in 2002 [44] and covers every aspect important to the qPCR assay itself, as well as issues relating to pre- and post-assay parameters. By 2009 it was pretty obvious that many publications using PCR-based methods were seriously flawed and that this impaired the readers’ ability to evaluate critically the quality of the published results. The actual trigger for the publication of the MIQE guidelines was provided by the after effects of a paper that used RT-qPCR to detect measles virus in children with autism [45]. These results were never reproduced independently [46], [47], [48] nor repeated by the authors themselves [49]. An analysis of the raw data underlying the original publication revealed a series of mistakes, inappropriate analysis methods and misinterpretations that completely invalidated any of the paper's findings [50], [51]. Despite the presentation of detailed evidence at the major autism trials in 2007 in the USA, three initial judgements and three subsequent appeals judgements confirming that the underlying data were unreliable, this paper has never been retracted and its false conclusions remains unrestricted in the scientific literature.

The number of citations of the original MIQE publication in the peer-reviewed literature has just passed 3000, it has been followed by several editorials on their implementation [26], [52], [53] and has encouraged the publication of the MIQE guidelines for digital PCR [54]. All major instrument and reagent manufacturers incorporate the guidelines into training their field application specialists, actively encourage their implementation, support and organise world-wide workshops and publish effective guides to help their realisation [55], [56]. Unfortunately, researchers themselves are far less enthusiastic and in addition to the thousands of peer-reviewed papers that are already published, thousands more are published every year without regard for the MIQE guidelines and report results and conclusions that are unlikely to be correct [57], [58]. Recent surveys of qPCR-based publications found that the problems addressed by the MIQE guidelines continue to persist [28], [59] and an investigation focusing on normalisation procedures in RT-qPCR-associated colorectal cancer biomarkers concluded that virtually none of the studies could be reliably assessed [60].

Biomarkers

The identification of clinically useful biomarkers has been one of the most active areas of research, with tens of thousands of papers claiming to have identified a wide range of diagnostic, prognostic or predictive biomarkers of human disease [8]. However, conclusions are often based on large-scale measurements of biological processes that “drown” any understanding in a “sea of measurements” [61] and inapplicable statistics [62]. This headlong stampede has only occasionally been punctuated by thoughtful, insightful and hence ignored counsels for cautions such as that provided by one of the earliest discussions of biomarkers that also included a checklist for evaluating their reliability prior to use in a clinical context [63].

The starting point for most biomarkers continues to be the analysis of microarray results, even though there is plenty of evidence that should council caution. One study that deserves to be considered more widely highlights the discordance between microarray results and those from non-microarray-based clinical and biological data [64]. Others have identified poor study design and patient selection, reliance on p-value as well as technical and analytical issues as obstacles in the identification of practically useful biomarkers, and include recommendations for marker reporting [65], [66]. A fairly recent report comments that “the most spectacular published signatures predictive for chemotherapy response are based on unreliable data” [67]. Another analysis is more optimistic in that it suggests that “reproducible [microarray] data can be achieved and clinical standards are beginning to emerge”, but adds the caveat that there is a need for establishing a suitable workflow to “correctly sample a tumour, isolate RNA and process this for microarray analysis, evaluate the data, and communicate findings in a consistent and timely fashion” [68].

The need for easy and complete public data access [7] and data checklists [69] is highlighted by the results of studies that reanalyse such data and cannot reproduce the original analyses. An analysis of 46 microarray studies addressing major cancer outcomes revealed that they were characterised by selective data analysis and discussion of results [70], whereas the analyses from only two of 18 microarray-based gene expression profiling studies could be reproduced [71]. Suboptimal design and inadequate assay information are typical features of cancer prognostic marker studies [72] and when combined with the strong reporting bias characteristic of many articles suggests that this literature may be largely unreliable [73]. The problem is as serious today as it was five, ten and 15 years ago. A recent microarray study found that the human genomic response to acute inflammatory stresses is not reproduced in current mouse models [74], a finding that challenges the use of mice as essential animal models in biomedical research. However, a reanalysis of the data not only found that human and murine gene expression patterns associated with inflammatory conditions are congruent, but also identified a number of pathways commonly regulated in humans and mice [75]. Incidentally, analysis-related discordance is just as apparent in other biomedical areas of research: 35% of published reanalyses of data from randomised clinical trials (RCTs) led to changes in findings that implied conclusions different from those of the original article about the types and number of patients who should be treated [76]. In addition, not only are many publications describing RCT outcomes inconsistent with their protocol, but also there is clear evidence for publication as well as outcome reporting bias. Studies reporting significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported [77]. If outcomes of RCTs are to be clinically useful, there is an obvious need to publish all results and outcomes, regardless of their significance.

Next generation sequencing (NGS), which allows hypothesis-neutral analysis with accuracy and a wide dynamic range and has started to supersede microarrays, is not exempt from these challenges. The variable lengths of RNA obtained after fragmentation prior to sequencing can be a source of bias, resulting in incorrect ranking of differentially expressed genes [78] and the technical variability of transcriptome sequencing experiments is sufficiently high so that RNA abundance estimates can substantially disagree even when coverage levels are high [79]. Quality assessment of sequencing data also appears to be somewhat less than complete, as witnessed by the surprise expressed by one group of authors when they noticed that standard DNA alignment algorithms assume that sequences are accurate and do not allow for the systematic incorporation of quality parameters into sequence alignments [80]. Unless prompt action is taken, it is easy to see how NGS data will become as unreliable as those obtained from qPCR and microarray experiments, except that there will be vastly more of them and troubleshooting will be proportionately more difficult.

Hype and retractions

The peer-reviewed literature is widely perceived as comprising articles that have been carefully evaluated by experts in the field for methodological consistency, completeness and soundness as well as for the logical consistency of scientific claims based on experimental results. Whilst it is acknowledged that not every conclusion is correct and that subsequent new data can lead to a re-evaluation of those findings, the combination of rigorous peer-review and publication in reputable journals bestows on peer-reviewed papers the appearance of abidance, authenticity and authority.

Whatever the veracity of this perception, the route by which research findings are disseminated to a wider public is considerably less assured. An examination of press releases by high-profile medical journals found that these do not routinely highlight study limitations, disclose the role of industry funding and present findings perceived to be newsworthy in a way that may exaggerate their importance [81]. The somewhat unhelpful advice of Nature Genetics for a scientist who finds his work being thus glorified is to reduce the hype by explaining the implications of the work clearly “without using our field's rich jargon” [82]. Unselective press coverage is also apparent, as demonstrated by a study that followed up the work presented in 147 conference abstracts and reported in 252 news stories [83]. After three years 25% of abstracts remained unpublished, 25% were published in low impact factor journals and 50% in journals with high impact factors. Thirty-nine publications chosen for front-page coverage followed the same pattern, suggesting that media attention is undiscerning and not focused on quality research. Most of these reports would have been preliminary findings with tentative validity and indiscriminate press coverage is likely to create a misplaced impression of confidence and validity and is detrimental to engendering trust in research or individuals engaged in research. Interestingly, sometimes even researchers themselves cannot agree on the relevance of their own published work. It should not be unreasonable to expect that all authors contributing to a research publication agree on the interpretation of results and any conclusions that are put forward. Remarkably, that is not necessarily the case as revealed by the results of a small study of ten papers that appeared in The Lancet [84]. These indicated that the discussions did not reflect the opinions of all contributing authors and that the authors frequently disagreed about the importance and implications of their own findings and where their research was heading.

Sometimes, it is necessary for a published article to be retracted. For example, a high profile article that used publicly available data to identify signatures from microarray profiles of human cancer cell lines with known in vitro drug sensitivity or resistance to predict in vivo chemotherapeutic responses [85] had to be withdrawn [86] after clinical trials had started because independent reanalysis of the data showed unintentional mislabelling and indexing errors that resulted in none of the original results being reproducible [87]. Retractions make up a miniscule percentage of all publications (871/>17 million between 1950 and 2007) [88], although the number of articles retracted per year has increased by between 11- and 19-fold from 2001 to 2010 [89]. Editors of high impact factor journals often use another device, “expressions of concern”, prior to retraction to question published data, with six journals accounting for 40% of such notifications [89]. Ironically, there is disagreement in the literature about the effects and subsequent impacts of retracted publications. It is not clear whether there is a correlation between journal impact factor and number of retractions, with one analysis showing none [90] and another reporting a strong correlation [91]. Certainly our own data show very clearly that there is a negative correlation between transparency of reporting and journal impact factor [28].

The most common reason for retraction is the uncovering of a scientific mistake, although there is increasing evidence of fabrication and data plagiarism [92], [93], [94]. Neither cause is usually cited as a reason for retraction [95], even though these papers represent a calculated effort to deceive [96]. Hence it is rather disconcerting that retractions may not be an effective means of removing fraudulent results from the literature [97]. An analysis of the impact of retracted articles in the biomedical literature from 1966 to 1997 found that even after publication of the retraction most of these articles continue to be cited as valid work with no mention of the retraction [92], with most citations implicitly or explicitly positive [98] even up to 24 years post retraction [99]. A more recent publication also found that a retraction did not affect the paper's citations rate within 12 months of retraction [100] and an investigation of the effect of rebuttals on seven high-profile original articles found that original articles were cited much more frequently than the rebuttals, with no reduced effect on annual citation numbers [101]. In contrast, other reports found that retraction significantly reduced subsequent citation [102], [103] and that this reduction extends to the authors’ other published work [104]. Remarkably, an unexpected 18% of authors self-cite retracted work post retraction [why would they do that?], but only 10% of those authors also cite the retraction notice [105]. Around 1% of the NIH budget between 1992 and 2012 (US$58 million) was spent on papers retracted due to misconduct, with each article accounting for a mean of $392,582 in direct costs [106]. Not surprisingly, there have been calls to institute more aggressive means of notification to the scientific community [107], yet the topic is rarely mentioned and different publishers differ in their retraction policies, which can make it not immediately obvious that a paper has been retracted [108]. Although retrieval of a publication in PubMed also retrieves the retraction statement, authors often rely on other authors’ citation to that work and so inadvertently enhance that articles legitimacy.

Biological relevance

I would like to conclude with a brief mention of another reason for the low ratio of translation of research promise to real-life biomedical application. Much biomedical research is performed in a clinical and methodological vacuum [109] and research findings are frequently extrapolated from disease models that have little resemblance to the corresponding human condition [110]. Even though their unsuitability is occasionally recognised [111], [112], [113], [114], [115], [116], [117], [118], there is a need to acknowledge more universally and transparently the limitations in the clinical translation of non-human models in general to man [119]. No matter how transparent, well-designed, analysed and reported, if research results are obtained from inappropriate models they can never be translated into a valid human disease context.

Conclusions

Late in the day, editors of some of the major scientific journals have come to realise that they have been seriously negligent in their approach to overseeing the publication of reliable research results and have started to confront their responsibilities [120], [121], [122]. A series of similarly worded editorials appeared in Nature Structural Biology [123], Nature Genetics [124], Nature Neuroscience [125], Nature Immunology [126], Nature Medicine [127], Nature Methods [128], Nature Cell Biology [129] and announced measures to “reduce irreproducibility”. These are a welcome, if long overdue acknowledgement of the journals’ acquiescence to such vast quantities of inadequate reporting of experimental detail in much of the peer-reviewed literature.

However, the proof is in the pudding and an analysis of ten random papers published in late 2014 in Nature-titled journals [130], [131], [132] reveals the astonishing fact that there is actually less technical information provided than in the past (Table 1). Every single publication uses a single, unvalidated reference gene for normalisation (all old favourites), all use the 2−ΔΔCq method with no attempt to calculate PCR efficiency and reporting of RNA quality control measures is non-existent. Yet for the ΔΔCq calculation to be valid, the amplification efficiencies of target and reference must be approximately equal [140]; if not the PCR must be optimised [141]. What hope of ever changing anything if journal editors choose to ignore their own guidelines?

Table 1

A search was carried out on the Nature web site (www.nature.com) for the term “reverse transcription real time PCR” and ten papers published in the second half of 2014 were selected at random. Materials and methods and supplementary information was examined for information on RT-qPCR protocols.

Reference	[130]	[131]	[132]	[133]	[134]	[135]	[136]	[137]	[138]	[139]
RNA integrity	No	No	No	No	No	No	No	No	No	No
RNA purity	No	No	No	No	No	No	No	No	No	No
RT conditions	No	No	No	No	No	No	No	No	No	No
PCR efficiency	No	No	No	No	No	No	No	No	No	No
Primer sequence	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Reference genes	β-Actin	18S rRNA	18S rRNA	GAPDH	Rpl32	GAPDH	Actin	HPRT	GAPDH	GAPDH
Validated	No	No	No	No	No	No	No	No	No	No

In contrast, research published in Biomolecular Detection and Quantification (BDQ) will have to pass a careful double blind peer-review process that will examine the validity of its molecular study design, measurement, data analysis and reporting. Our aim is to establish a reputation for unbiased publishing of studies that adhere to best practice guidelines and promote excellence in molecular measurement and its data analysis. Subject areas are deliberately broad to allow BDQ to serve as a repository for sharing key findings across what may otherwise be disparate specialties. It is hoped that BDQ will attract a wide range of critical, thoughtful, well-designed and excellent publications to help provide the impetus towards a more transparent and meaningful scientific literature.

Finally, the question “Modern biomedical research: an internally self-consistent universe with little contact with medical reality?” [110] should now be extended to include the addition “and based on an unacceptable level of technical proficiency”. It will be a long time before the many contradictions apparent in every area of the life sciences are addressed, never mind corrected [142]. But who is going to lead this effort, and how many vested interests will conspire to retain the status quo?

Statement

The investigation on the measles virus/autism publication was carried out for the MMR vaccine litigation trial at the High Court of Justice in London and at the US Vaccine court. The author acted as an expert witness and was compensated by the solicitors acting for the principal defendants SmithKline Beecham Plc and Smith Kline & French Laboratories Ltd., Merck & Co Inc. and Sanofi Pasteur MSD Ltd. The author also acted as an expert witness for the US Department of Justice and was paid.