An open science peer review oath.

One of the foundations of the scientific method is to be able to reproduce experiments and corroborate the results of research that has been done before. However, with the increasing complexities of new technologies and techniques, coupled with the specialisation of experiments, reproducing research findings has become a growing challenge. Clearly, scientific methods must be conveyed succinctly, and with clarity and rigour, in order for research to be reproducible. Here, we propose steps to help increase the transparency of the scientific method and the reproducibility of research results: specifically, we introduce a peer-review oath and accompanying manifesto. These have been designed to offer guidelines to enable reviewers (with the minimum friction or bias) to follow and apply open science principles, and support the ideas of transparency, reproducibility and ultimately greater societal impact. Introducing the oath and manifesto at the stage of peer review will help to check that the research being published includes everything that other researchers would need to successfully repeat the work. Peer review is the lynchpin of the publishing system: encouraging the community to consciously (and conscientiously) uphold these principles should help to improve published papers, increase confidence in the reproducibility of the work and, ultimately, provide strategic benefits to authors and their institutions.

Open science and the future of peer-review

Science builds on itself. New knowledge is gained in the context of the enlightenment of earlier discoveries and, for the foundations to remain solid each discovery must be accurate and reliable. But science in the real world is messy, and advances haltingly and piecewise. Often, prior information is incomplete, so conclusions drawn need revising in the light of new, later evidence. This means that self-reference and self-checking are cornerstones of the scientific method; and, having reported our experiments, it is vital that readers are able to repeat them and understand how we reached our conclusions. Various studies have shown that, for a study to be successfully reproduced, information presented in the literature is often inadequate and the underlying data not readily available [1] - a significant drawback. Several commentators have concluded that weaknesses in the way that research investigations are currently conducted, and how their results are disseminated via article publication have become detrimental to the scientific process [2– 7]. As the technological sophistication of science increases, and the equipment used becomes more specialised, the data generated are harder to represent in traditional media, and reporting how experiments were performed so that independent researchers can repeat them becomes progressively harder.

Open science is a movement that seeks to ensure that the results and the data of scientific research are, and continue to be, available to all. One way in which reproducibility issues can be tackled is through the use of open-science and open-data practices [8, 9]. As attendees of the AllBio: Open Science & Reproducibility Best Practice Workshop, we discussed how the problem of keeping science transparent and reproducible in an increasingly technology-driven, and specialised, domain could be addressed.

One route, at the heart of scientific endeavour, is through the peer-review process. Peer review is an important gatekeeper and key component of scientific discourse. Before research findings can be formally accepted, they must be evaluated and commented upon by other domain experts, who provide advice about the quality or validity of the work to journal editors and/or readers (whether via pre-publication peer review, open peer review or post-publication peer-review systems). Importantly, peer review happens at a personal, rather than institutional, level and is carried out by individuals; it is therefore an ideal mechanism for engaging most researchers, given that all scientists peer review or are peer reviewed.

Peer review oaths

The peer-review process is not infallible [10– 12]; its weaknesses are many and varied, including the time available to perform thorough reviews, reviewers’ expertise, journals’ perception of relevance/interest/impact, and so on. Arguably, one of the most significant problems – certainly the one that generates most friction – is that reviewers can safely dispense self-serving and biased critiques, fully protected by the mask of anonymity. In some instances, anonymity may be the only pragmatic option, but in the interest of objectivity and constructive discourse it shouldn’t be the cornerstone of the peer-review process. Scientists have become sufficiently frustrated by these issues to devise ad hoc solutions to help safeguard the quality of reviews, and allow reviewers to affirm that they will review in an ethical and professional way, and encourage clearer review processes. This has led to the articulation of various reviewer’s oaths ( e.g., [13– 15]); it is these that inspired our discussions and prompted this article.

An outcome of our debate was the formulation of an addition to these oaths that helps codify the role of reviewers, encouraging and promoting best practices to ensure that the science they review is as open and reproducible as possible. The amendment includes guidelines not only on how to review professionally, but also on how to support transparent, reproducible and responsible research, while optimising its societal impact and maximizing its visibility.

The open-science peer-review oath (which can be found on figshare for reuse [16]) we therefore propose is as follows ( Box 1).

Principle 1: I will sign my name to my review

Principle 2: I will review with integrity

Principle 3: I will treat the review as a discourse with you; in particular, I will provide constructive criticism

Principle 4: I will be an ambassador for the practice of open science

Towards ‘open science friendly’ reviews

The declaration in your review

Peer-review oaths tend to be short declarations that reviewers make at the start of their written comments, typically dictating the terms by which they will conduct their reviews. The novelty in our amendment is the addition of a new declaration, which we recommend including in review preambles:

I will be an ambassador for open science

For it to be successful requires reviewers to follow the spirit of the statement throughout their reviews, which may involve a little additional work. The statement itself is derived from the rationale outlined below. It is for reviewers to decide how to implement this on a case-by-case basis – it is case specific, but nevertheless provides a framework for nurturing useful and hopefully fruitful discussions.

Guidelines for open science reviewers

Here is how we envisage the review process from an open-science reviewer's point of view:

I will work with you to help improve your research, as I believe that peer review should be an open, supportive and collaborative process. I will therefore sign my review and state my identity.

I will, before I review, ensure that I have everything (raw data, detailed methods/protocols, etc.) necessary to make a complete, independent assessment.

I will encourage the application, and, where necessary, provide guidance for any open-science best practices relevant to my field that would help support transparency, reproducibility, re-use and integrity of your research.

I will check that any data and software code are consistent with the text, that any digital object identifiers and accession numbers are correct and correctly cited, and that any models presented are archived, referenced and accessible.

I will check that the data and any software code and support documentation that underpin the published concept are made available in a manner that provides long-term unrestricted access.

I will advise, where necessary, on how to achieve better transparency and availability (in terms of materials and methodology, data and code access, versioning, algorithms, software parameters and standards), with the understanding that adherence to best open-science practices may require further effort from you.

I will support others in writing open reviews, where it is appropriate for me to do so.

I will decline to review if I am not an appropriate reviewer (whether because of my expertise or because of my relationship with the author(s)). In doing so, I will provide journal editors with an honest appraisal of these issues, and will openly explain how I reached my decision so that alternative reviewers may be found.

Conclusion

We believe that combining the open-science principle with some of the other key peer-review principles will help the scientific community to repeat published experiments, and to reach the same, or similar, conclusions.

The authors have done a nice job responding to my critique of the original version, and the resulting revised version is much easier to read. Reading the other reviews, and the comments, I can see that this article is sparking a good amount of discussion on an important topic. Clearly this will not be the last word on the subject, and that is a good thing.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

The manuscript has been improved compared to the first version. Despite the authors laudable intentions, I am still unconvinced that this will prove terribly useful to the scientific community, but I do not see any reason why this paper should not be indexed and referenced.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

The revised paper is much improved in terms of its (now appropriate and not tedious) length and clarity of statements. I still see a problem with the proposed absolute requirement (listed as the first principle) of disclosing the reviewer's identity. While noble in principle, I think it is unrealistic at this time; if so, compromising the first principle would appear to make the subsequent ones partly irrelevant.

On the other hand, the authors clearly explained their decision to avoid addressing one specific issue of the reviewing process - frequent requests for tedious and time consuming experiments that do not contribute substantially to the main point of the paper, often precluding publication. As an experimental scientist, however, I do see this point as critical and think that its resolution would  contribute more to the fairness of review than disclosing the reviewer's identity.

Finally, I am still surprised that the related work by Bienz and Weston is not discussed and/or cited.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

The principles outlined here are important, and this piece is certainly timely. I have two suggestions to improve the manuscript.  First, as currently constituted, the manuscript is a bit repetitive, with a large text box recapitulating what is also laid out in much the same format in the body of the manuscript. Perhaps each of the major points could be expounded upon in the text.

Second, while it is hard to argue with the specific tenets enumerated here, there are quite a lot of them. I guess my feeling is that for ideas like this to be more widely adopted that it would be helpful if the core principles could be boiled down to a more manageable size (which would be ideally presented in a text box).

I applaud the authors for doing their part to increase openness in scientific publishing, something I agree is very much needed.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Thanks for your helpful and candid report Larry. We agree the manuscript was a little repetitive and so we have made it much more succinct, by concentrating on the open science principle we want to champion. We have also condensed the oath into 4 principles with an accompanying rationale and made it available on FigShare, which should make it much easier to follow and reuse.

I think this article makes many good points, and I also agree with the other critiques. However, I do see danger in full transparency. The problem is that significance of study can be  subjective and used in a biased way to sway readers towards a view of greater importance, in order to achieve a good standing with the authors.  At this point in time, it is important for editors of other non-transparent journals to remind reviewers of the key aspects of this oath – to critically read the paper and make constructive comments as best as they can.  Editors need to use their power to delete inappropriate reviews or reviewers.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Thanks Suzanne for your helpful and candid report. We have now satisfied the other reviews to make the oath clearer and simpler to use. We take your point that perceived impact can indeed sway readers and we certainly agree that as well as the scientific community practicing open science and reviewing openly,  journals and journal editors need to play a role in ensuring that reviewers are doing a job that ensures integrity - we believe they can use the principles of the oath to do this.

This manuscript was written by participants of a workshop entitled "AllBio: Open Science & Reproducibility Best Practice Workshop" which took place at the TGAC in Norwich UK in September 2014. The alleged purpose of the oath and manifesto proposed in this manuscript is to make "transparency, reproducibility and citizen-scientist engagement the default parameters for performing sound research."

Those goals are clearly highly laudable, and I completely agree that the process of scientific refereeing would greatly improve by going 'open'. I do, however, feel very ill at ease about acting as a referee for this particular paper, for the following three main reasons:

This is not a scientific manuscript, and it neither contains data, nor reviews a scientific topic. I am thus left wondering why such a manuscript should need to be peer reviewed. After reading it, I certainly cannot  conclude that it is scientifically sound. The best I can conclude is that it is not scientifically unsound.

Although this manuscript is very short, I must say that I found it rather difficult to read because of its structure which contains many redundancies and of the fuzziness of its purpose. One aspect that particularly bothers me is the unquestioned assumption that open refereeing will improve reproducibility. The first thing the authors should clarify is what they mean by reproducibility since there are at least two types that I can think of:

The first one concerns data reproducibility (or robustness). In other words will similar data, leading to the same conclusions, be obtained if the experiment is repeated (on a different day and/or with different samples and/or in a different place and/or by different people etc…). The second type of reproducibility relates to the capacity of other scientists to reproduce an experiment described in a published manuscript. I suspect that when they refer to 'science reproducibility', the authors refer to the latter type, although the first type is the most important one in my eyes.

The authors duly acknowledge that they have been inspired by reviewer's oaths previously proposed by others (refs 13-15), and if I am being completely honest, I do not find that the set of 17 rules they propose represent a significant improvement on those previously proposed oaths. Because I do not want to plagiarise Jonathan Eisen, I will simply suggest that the authors should very seriously consider following the many suggestions he has made on his blog to improve this paper ( http://icis.ucdavis.edu/?p=505 ), and especially the idea of reversing the Oath and the Manifesto.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Thanks Etienne for your very helpful and candid report.  We agree that the article needed to be made clearer in order to convey its intentions to the reader and we have taken measures to do this by clearly elucidating the open science principle. We have also made it clear about what we mean by reproducibility, and how the robustness of research should mean that similar data should be able to produce the same conclusions. We have also simplified the oath on the advice given to us from you and the other reviewers to make it easier to read and reuse.

Overall, I believe this is a laudable proposal for a code of practice in academic peer review. In one sense it would be nice if such a code was unnecessary; after all, most of the practices outlined here should form part of any graduate training in science. In practice, of course, we know that the reality of peer review often falls short of achieving its aims. Here the authors outline 5 key principles and 17 key practices, which - if adhered to - would likely result in a more transparent and effective peer review mechanism.

While I am supportive of this initiative, I do have some concerns about the way it is presented and I also wonder how adhering to it could be incentivised. I will outline below some suggestions for possible improvement: Regardless of the above concerns, I applaud this much-needed call for greater transparency in the peer review process.

The authors begin by outlining the 17 key practices and only at the end do they group them together within governing principles. I think the overall structure of the paper would be clearer if this order were reversed - beginning with the five principles, explaining the key practices in each case that serve them, before returning to the code of practice or "oath" as the authors call it.

Some of the individual practices in the oath are somewhat ambiguous to me, and others seem to overlap. For instance, what exactly does it mean for a reviewer to "state their limits"? This is elaborated later under principle 2 but I would recommend clearer descriptions at all times. Other practices seem very similar, or least belong to common subsets of behaviour; e.g. practice 7 and practice 10; practice 2 and 6; practices 13, 14, 15 and 16 are all very similar and not clearly distinguished. With some careful attention I suspect the number of practices here could be halved, which would improve readability and likely uptake.

The language expanding on some of the core principles is perhaps little purple in places, e.g. " I will use the majority of ‘doves’ to balance the ‘hawks’ in my review by sharing the content." I'm not entirely sure what this means, but if it refers to the intention to balance sharp critical feedback with constructive suggestions or positive feedback, this could perhaps we said in a more straightforward way. The authors should bear in mind that not all readers will have English as a first language, and so more direct and less metaphorical phrasing could widen the appeal.

Two concerns with signed reviews need to be addressed. The first is the potential negative consequences felt by junior (non-tenured) scientists or minorities, who could, at least in theory, face severe repercussions for criticizing the work of senior/powerful colleagues who sit on editorial boards or grant panels. In my mind this conflict has never been properly addressed; while I believe it is completely reasonable for tenured scientists to be open and accountable in their reviews (I always sign mine), it is questionable whether this should be required uniformly across science. The second concern with open reviewing is the potential legal backlash of scientists being sued by litigious authors who feel aggrieved by a reviewer's published comments. We are seeing this already on PubPeer and elsewhere. This raises the question of who assumes legal responsibility for the content of a signed review. In the case of F1000Research, for instance, does it or will it provide legal indemnity to reviewers who choose to unmask themselves?

Finally, I would prompt the authors of this paper to consider, and ideally speculate on, ways their oath (or code of practice as I would call it) might be implemented and incentivised in science. Could it, for instance, be worked into the next REF in some way? How could this be achieved and what challenges would need to be overcome? How does this initiative relate to other emerging group-led initiatives, such as the Agenda for Open Research (https://agendaforopenresearch.org/)? There authors have no shortage of good intentions but as we know, there lies a world of groupthink and social inertia between good intentions and good practices.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Thanks Chris for your very helpful and candid report. Like all of the reviewers have suggested we have simplified the oath and made it much clearer and simpler to use. Yes, we agree that some of the language used was difficult for non-native English speakers and hence we have made it clearer to read in places that may of caused confusion. We also agree with your point about new/junior reviewers who may feel exposed by openly reviewing someone more senior than them and have acknowledged that in the guidelines. Here at F1000Research, we have implemented a system of co-authoring referee reports to help support junior researchers who feel that on their own they may be subject to repercussions from the authors. With regards to the legality of an open review on F1000Research, reviewers submit their report according to our terms and conditions which clearly state the report will be published under a CC BY license, and that the report must not be defamatory. However as a publisher we are also conscious about each review that is published on the site and we have system in place to ensure that nothing libel would be intentionally published. More generally, openness and transparency in signing a report means that they are on the whole much more civil and are of a constructive nature that encourages dialogue. We believe the peer review process should be a collaborative process and hence that is the environment we try to provide. Here is a letter of response from Rebecca Lawrence, Managing Director of F1000Research, recently published in response to the recent libel story from the anonymous review on PubPeer: . Finally, just after we published the review we became aware of the Agenda for Open Research who have similar goals that align with us and they were kind enough to link to the oath in their guideline for reviewers. We fully support their initiative, and we will seek opportunities to collaborate where possible. You are right, it would be great to push to get open reviewing to be recognised as an official part of the REF (and other funding bodies) and acknowledged as a measurable scientific output. Here at F1000Research we are taking steps to make this happen; currently we mint each report with a DOI and we are heavily involved in the t in collaboration with and .

This article addresses a very important issue of peer review. Although, many of us tend to regard it in a way of "if it ain't broke, don't fix it", we also are often frustrated with the process and wish it were more fair, transparent and to the point (rather than requesting numerous experiments that are only tangential to the paper's main message). Thus, constructive approach to potential improvement of the peer review system is always welcome and should be encouraged. With this in mind, I support, in principle, publication of this manuscript. On the other hand, the way that the authors present their ideas is not perfect. Here is why:

First they make a huge emphasis in their introductory remarks on experimental reproducibility and societal impact. However, their "oath" does not address these issues specifically and constructively; instead, it offers declarative statements which are mostly trivial and already represent part and parcel of today's peer review process (e.g., "I will comment on how well you have achieved transparency, in terms of materials and methodology, data and code access, versioning, algorithms, software parameters and standards, such that your experiments can be repeated independently").

Second, the authors do not address clearly one very important aspect of potential improvement of the review process which is "To request further experiments only as a last resort, and only if they are essential to validate the conclusions of the paper. No experiments extending the study beyond its conclusions, or with unreasonable cost or time implications, should be proposed. An estimate of time required for the additional work should be provided." This is a quote from Mariann Bienz and Kathy Weston who also addressed the issue of peer reviewing ( http://elifesciences.org/elife-news/a-reviewers-charter).

Third, the "oath" is unreasonably long. Again, I suggest to look at the post by Bienz and Weston, which unlike the present paper, is much clearer, laconic and to the point.

Fourth, the authors quite nonchalantly suggest to overturn the cornerstone of the present review system - its anonymity. What is the chance for that? The reason for such a step should be well reasoned and substantiated.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Thanks Vitaly for your very helpful and candid report. We have now explained what we hope to achieve from the oath, how we think it could address the issue of reproducibility and how the oath could have a positive societal impact in the scientific community. We believe at the researcher level there is a unique opportunity to help spread open science practices when reviewing articles and help improve what the community as a whole believes should be available in a paper in order for it to be reproduced. We agree with your point about not asking for additional experiments, however we have decided to concentrate fully on elucidating the requirements of the open science principle specifically and not the other principles. As with the other reviewers and comments we have taken all of your advice and shortened the oath to make it simpler for reuse. We also agree with your point about anonymity and we have discussed this in the manuscript on why we should move towards open peer review.