Repeat after me: Replication in clinical neuroimaging is critical.

Background1

NeuroImage: Clinical is a journal that specifically deals with novel insights relating to the use of neuroimaging to improve our understanding, diagnosis and treatment of brain-related disorders. The numerous papers already submitted have confirmed that the importance of this enterprise is recognised by the wider clinical community. We hope that the journal will continue to grow and come to be a valued outlet for the best and most innovative developments in the neuroimaging of disease.

But the enterprise brings with it substantial tension. By encouraging studies that report novel and exciting insights, there is a danger that we may come to overlook a fundamental requirement to consolidate, scrutinise, test and, if necessary, rethink or discard these insights. It is an embarrassing, though by no means unique, failing in the neuroimaging field that the quest for the new and exciting has frequently over-shadowed the more humdrum side of science—the need to replicate and re-evaluate. This failing is particularly indefensible in clinical research where a key part of the goal must surely be to translate the insights generated by initial experiments into credible, practically useful advances in patient management: prognostics, diagnostics, treatment and monitoring.

To put things simply, if the frequently elegant experiments and fascinating results of small scale clinically-orientated studies are ultimately to influence clinical practice, they need to be replicated and extended on a larger scale. At present, with most journals placing a greater dividend on novelty, the incentive for researchers to do this may seem too low to encourage the work. We would like to play our own part in remedying this by actively encouraging the submission of replications. We are therefore planning to create a new category of paper (“Replication Study”). We recognise that there is enormous complexity attached to replication within science (see, for example, the November 2012 issue of Perspectives on Psychological Science) and we acknowledge that, given the complexity of neuroimaging, we run the risk of stifling the overall effort if we are too prescriptive about what authors of replication studies must do or not do and how they should or should not state their claims. Rather, in launching this new category, we put forward below a series of general considerations and questions for the author wishing to submit a replication paper. These considerations will be kept under review and we welcome comments.

The science of replication: what papers do we want to publish?

We envision two general strategies for replication studies loosely referred to as internal or external replication. Internal replication entails the reporting, perhaps in a single manuscript, of several experiments (particularly in independent samples) with each experiment replicating and extending the preceding one. One example of this approach is found in the Quarterly Journal of Experimental Psychology (pub Taylor and Francis). This within-paper/study design is a particularly powerful but somewhat under-used approach in neuroimaging. It would be greatly welcome at NeuroImage: Clinical.

External replication is the more complicated and, perhaps, challenging enterprise of performing an experiment in a different laboratory, with new samples. These complications are further aggravated in neuroimaging, where different scanners, data formats, image processing pipelines and statistical inferences are adopted. Thus, a second study is never identical to the first, and so there is always a question of what constitutes sufficient agreement to constitute replication. There is clearly a need for methodological innovations to facilitate comparisons across laboratories. In the meantime, these complexities do not diminish the importance of attempting such external replications and we hope that the knowledge that such work will be welcome at NeuroImage: Clinical will encourage scientists to embark upon it. A number of papers have outlined and developed taxonomies of replication (Burman et al., 2010, Drozar, 2010, Morrison et al., 2010) and journals have used these structures to prescribe ways in which authors might wish to set up, analyse and discuss their findings. To ensure methodological consistency, some, for example, have emphasised the need for close collaboration, or at the very least discussion between the authors of the replication study and those of the original report.

While we do not wish to set a rigid guideline, we do ask that, in preparing a replication paper for NeuroImage: Clinical, reviewers consider one or more of the following general questions.

If a finding has been successfully replicated, in what ways does this replication move the field forward? Has the replication been achieved in a larger group, making the finding more credible? Does it generalise the finding? Does it add new insights that may assist in drawing this finding towards having an impact on the clinic?

If the original finding has not been replicated, why not? Has the original protocol been followed satisfactorily, i.e. to a sufficient degree that the non-replication calls the original finding or its interpretation into question? Does the study go further and actually offer an alternative explanation for the original finding, one that is more fully tested and explored in the newer study? There are several reasons why a finding may not replicate and not all of these arise from an erroneous experimental procedure or conclusion on the part of the original authors.

Are the authors of the replication independent of those who presented the original finding? We suggest that a positive replication may carry more weight if it is conducted in a different lab by different experimenters. (Whether a failure to replicate carries more weight if carried out by the original authors is perhaps another matter).

In essence, we would ask authors considering such a submission to think about whether the original paper/finding is of sufficient weight or importance that its replication or non-replication will itself be of interest and impact and whether, if given further credence by its replication, it might have more immediate implications for clinical management. Conversely, for a failure to replicate, the question is whether this seriously challenges the paper's current or future impact on research and practice? It would be helpful if authors submitting such studies touched on these points in their cover letter.

The place of meta-analysis

Perhaps implicit in the above discussion is the view that a finding is either “real” or not and that if it is, it should be reproducible. We recognise that this dichotomy is a simplistic one: a finding is not necessarily void if it doesn't replicate nor is it necessarily valid if it does. Neuroimaging studies require complex statistical analyses at the level of a single group, and even the single subject. The patterns emerging across multiple studies are likely to require correspondingly complex analyses for their validation. For this reason, meta-analyses of clinical neuroimaging findings will play a large part in taking the field forward and in clarifying the nature and cause of variability, a variability which, of course, may prove central to moving the field towards patient management. After all, while we usually study groups, we diagnose and treat individuals.

A word on editorial process

Peer review

We are mindful that, when subjecting a replication study to the peer review process, the authors of the original paper may be deemed to have a conflict of interest. If the follow-up study reports a successful replication, then they may be well-disposed towards it and, if it does not, the opposite may be true. Yet, despite this possible conflict, the authors of the original study may be in the best position to offer comments and criticisms. We propose therefore that one of the authors (preferably the senior or corresponding author) should be invited to participate in the peer review process but that their view will be balanced by two independent reviewers. That is, the aim will be to recruit three rather than the usual two peer reviewers in order to provide as informed and balanced a review process as possible.

Publication

Should a replication study be considered suitable for publication, we will aim, where appropriate, to provide an opportunity for the authors of the original paper to write a brief response. Of course, the publication of such a response would depend upon it being provided sufficiently promptly.

Final comments

Our hope is that you will join us in this endeavour to strengthen the links between basic and clinical imaging science, and ultimately play our part in allowing exciting, albeit fragile insights to translate into genuine clinical improvements.