Some Consequences of Refusing to Participate in Peer Review.

The proliferation of journals has had an unexpected side effect: it is now difficult to find qualified reviewers willing to devote the time necessary for assessing journal contributions. Although it is difficult to find data, most scientists involved in the academic world have their inboxes deluged with a cornucopia of invitations to submit to new journals, speak at conferences (as a keynote, for sure!), and review articles. New online journals (such as ours) strive to publish the finest and most relevant work for our readers while at the same time maintaining rapid "turn around times" for authors. This requires that a complex system function flawlessly. But, there are some aspects of the system that are creaky; some are broken entirely.

The proliferation of journals has had an unexpected side effect: it is now difficult to find qualified reviewers willing to devote the time necessary for assessing journal contributions. Although it is difficult to find data, most scientists involved in the academic world have their inboxes deluged with a cornucopia of invitations to submit to new journals, speak at conferences (as a keynote, for sure!), and review articles. New online journals (such as ours) strive to publish the finest and most relevant work for our readers while at the same time maintaining rapid “turn around times” for authors. This requires that a complex system function flawlessly. But, there are some aspects of the system that are creaky; some are broken entirely.

The unspoken truth of the academic journal is that its sustenance is the unpaid work of its authors, reviewers, and editors. From the authors’ perspective, publishing in reputable journals is the currency by which they pay for academic advancement. As professors, we require ourselves to contribute to the growth of knowledge in our discipline and it is by this contribution that we are judged. As grant seekers, our work is often judged by the quality (and sadly, the amount) of work contributed to the literature. Although all academic institutions profess to judge their faculty by the tripartite standard of research, education, and service, it is mostly publications and grants that compel promotion and tenure.

One of the major forces regulating the emergence of competitors in any industry is the barrier to entry. In the case of new journals, the major barrier is publication cost. Online publishing has reduced but not eliminated this cost. With demand for publishing being high, it was only a matter of time before the journal world became very crowded indeed. Additional pressures from granting agencies to publish in accessible journals coupled with faculty members’ need to have national and international recognition for promotion and tenure guaranteed that the number of journals would increase rapidly.

It is impossible to quantify the number of scientific journals in the world. It is not even possible to quantify the number of peer-reviewed journals. In 2007, Larsen and Von Ins [1] estimated the number of scientific publications at about 24000. As of November 2014, the National Library of Medicine currently indexes 5642 journals for Medline [2]. A careful study of journals in the narrow field of nanomedicine has documented a dramatic increase in the number of journals (Fig. 1) [3]. Of course this field is notable for its rapid advances and significance. But, the inflection point correlates roughly to the emergence of online journals.

Fig. 1.

Proliferation of nano journals. The green line shows the number of new nanotechnology journals launched each year from 1985 to the present; the blue line is a cumulative tally of all nanotechnology journals (including those that have ceased publication); the red line includes only current journals (defined as journals that have published at least one issue dated ‘2009’ or ‘2010’ as of 30 October 2010). Only journals that publish original research papers are included; newsletters, book series, magazines and conference series publications have been excluded. The purple line shows the number of journals in the ‘nanoscience and nanotechnology’ category of the JCR. Source: Grieneisen & Zhang 2010 [3].

The rise of the “predatory open access” journal has been well documented [4]. Less well recognized is the problem of finding competent peer reviewers. Peer review is the pillar on which science stands, for better or for worse. It is a complex and time-consuming process whose economic rationale is questionable. For our Journal, the process works rhythmically. A manuscript comes in over the transom or is solicited. One of the Editors-in-Chief assigns it to an Associate Editor who then picks no fewer than two reviewers for each manuscript. Reviews are received and analyzed by the Associate Editor who has domain expertise in the area of the manuscript. The Associate Editor makes a recommendation to the Editors-in-Chief who make the final decision regarding disposition. This whole process should be done efficiently, balancing a thoughtful review with the needs of the authors for a prompt decision. And all this is done anonymously.

From a transaction point of view, of course no money changes hands. The currency of the system is provided by the value placed on it by the community. If you review my article, I have to be willing to review yours. And if I am appointed to an editorial board or asked to review an article, it serves as recognition by the community that I possess certain expertise and this can be leveraged for academic “perks” like promotion, tenure, and maybe even 0.025% of a secretary’s time (we can dream!) Thus, review activities require either enlightened self-interest or altruism. In the era of constrained resources then, review activities have to be balanced against revenue-generating opportunities, leisure, family time, eating, and unspeakable necessities.

Where does this beautiful system come a cropper? In an infinite number of unpredictable ways. The Associate Editor is busy, he or she does not respond timely to the review request. The reviews returned are not of sufficient quality to compel a decision. The Editor-in-Chief is otherwise occupied. The server is down. The manuscript’s presentation does not support publication. But, in my experience, the greatest difficulty is finding qualified reviewers. It is not rare to have 10 refusals for every acceptance. In articles requiring narrow expertise, the ratio is sometimes even less favorable.

That having been said, does peer review work? The increasing number of retractions that we have all observed seems to suggest that there is one or more flaws in the system. Retraction Watch is a blog that monitors scientific irregularities in publication [5]. Its importance has recently been recognized by a $400000 two-year grant from the MacArthur Foundation [6]. It is tempting to think that increased scrutiny has caused this increase but Steen and colleagues found only a modest effect of observation [7]. A now notorious sting operation performed by Science Magazine [8] netted a painfully high acceptance rate for an obviously bogus paper. Some respected publishing houses that have embraced the open-access model were caught up. Perfunctory reading of the article would have immediately disqualified it, suggesting that no or trivial review was performed.

As a journal editor, I live in fear of publishing a bad paper or having to retract it (Fig. 2). Sometimes, though, the deception is so skilled that it escapes into publication. Other times, it takes years to establish bad science and a paper stays in the public space doing severe damage. This occurred in the Lancet piece linking autism with MMR vaccine. Twelve years elapsed before the paper was fully retracted [9]. During that interval, the spurious link between autism and MMR became enshrined in many belief systems with the result that there is a decline in some childhood immunization consequent to vaccine refusal by their parents [10].

Fig. 2.

Papers published and retracted per year since 1973. Note that the multipliers are different. For the sake of simplicity,error here includes all infractions except fraud (e.g., scientific error, plagiarism, duplication, other). Apparent declines in recent years must be interpreted with caution as additional papers may be retracted in the future, thereby reversing this decline. Source: Steen et al. 2013 doi:10.1371/journal.pone.0068397.g001 [7].

Several papers have been published suggesting a ranking method for reviewers [11], but the proliferation of journals and the tendency to milk several publications from each experiment will continue to thwart foolproof peer review. Our Journal and many others use plagiarism software and it has blocked several papers from being submitted to peer review.

Journal review is a subjective but necessary process. It has been complicated by the proliferation of new journals (ours included!) But the success of new journals (ours included!) supports the need for new avenues of information dissemination. In order to maintain the integrity of journals, the community has to support the review process. Too busy? Suggest a trusted, young colleague and help him or her with the review process. You teach them how to write grants, right? Ameliorating the scientific community is a social responsibility. Our failure to rise to the challenge promotes the proliferation of bad science—the result of which is embarrassing at the least and societal catastrophe at the most. The scientific publishing world is far from perfect; increasing its health by improving the peer review system is to everyone’s benefit.