How many manuscripts should I peer review per year?

Peer review provides the foundation for the scholarly publishing system. The conventional peer review system consists of using authors of articles as reviewers for other colleagues' manuscripts in a collaborative-basis system. However, authors complain about a theoretical overwhelming number of invitations to peer review. It seems that authors feel that they are invited to review many more manuscripts than they should when taking into account their participation in the scholarly publishing system. The high number of scientific journals and the existence of predatory journals were reported as potential causes of this excessive number of reviews required. In this editorial, we demonstrate that the number of reviewers required to publish a given number of articles depends exclusively on the journals' rejection rate and the number of reviewers intended per manuscript. Several initiatives to overcome the peer review crises are suggested. Copyright: © Pharmacy Practice and the Authors.

Peer review provides the foundation for the scholarly publishing system. Despite the pessimistic conclusion in Jefferson et al.’s abstract – “At present, little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research” –, the two studies included in their systematic review, which aimed to assess “the effects of peer review on study report quality,” clearly demonstrate the positive effects of peer review on the methodological quality and the value of the articles reviewed.1,2,3

Alternative methods for peer review have been studied, even utilizing randomized controlled trial designs, but testing their impact on the quality of the articles in a real-life environment “would be costly, time-consuming and sometimes not feasible”.4 At the end of the day, the conventional peer review system was reported to be one of the most efficient systems in Kovanis et al.’s analysis.4 In fact, an experience of post-publication review already exists and has exposed the risks associated with the system: Social media is a perfect example of a non-reviewed publishing system, which incontrovertibly has led to a high prevalence of fake news. Facebook’s adoption of fact-checking programs – nothing more than a post-publication review system – demonstrated the limitations of any post-publication peer review.5 This is a lesson we should learn before introducing post-publication review as a common practice in scientific publishing in substitution of traditional pre-publication peer review.6,7

So, if peer review seems to be a good system to improve article quality, why is the system permanently under criticism? Let’s be honest: We are in a rush to publish our papers. Sometimes because they are part of a master’s or doctoral dissertation, other times because we need to add a line to our CVs. Scientific articles live forever and should not follow the popular saying concerning newspapers: “Today’s News, Tomorrow’s Fish Wrap”.

When authors complain about publication delay and the tardiness of the peer review process, we would rather provide figures, as we usually do in science. Many studies evaluated the publication process times in different biomedical areas and geographic regions, reporting acceptance lag (i.e., time from submission date to acceptance date) of usually over 100 days.8,9,10,11,12,13,14 Pharmacy Practice reported a first response time after peer review comments of 92 days (SE=5.7) in 2018.15 We are happy to announce that Pharmacy Practice first response time for original research articles accepted decreased to 80 days (SE=3.8) in 2019, with an acceptance lag of 124 days (SE=5.0).

As editors of a scientific journal, we have to ask authors who complain about the long publication process times: Do you think we intentionally extend the article’s processing time? Don’t you think that we would prefer to quickly make a decision as to whether to accept or reject the hundreds/thousands of articles we receive? To accept an article, the editor of a peer reviewed journal needs a number of peer reviewer comments supporting the quality of the manuscript. However, to reject a paper, two options exist: desk rejection or rejection supported by peer reviewers’ comments. A desk rejection is the negative decision made exclusively by the editor or the editorial board prior to any external peer review process. Considering the principles of a peer reviewed journal, desk rejection should only apply when the manuscript received is outside of the scope of the journal or the study suffers from methodological flaws beyond any possible repair. Although commonly used, desk rejection subverts the concept of a peer review system.16

Interestingly, authors also complain about the excessive number of manuscripts they are invited to review. Some of them write ironic commentaries about why they decline invitations to review based on personal events.17 Pharmacy Practice has started an in-depth analysis of its peer review selection process, with the aim of identifying differential characteristics of the accepters and decliners. Apart from the “I’m buried in reviews” argument and individuals who simply do not respond to the invitation email, other explanations for declining to serve as peer reviewers were as follows:

I’m at the end of the semester

I’m about to go on vacation

I’m on vacation

I’ve just returned from vacation

I’m at the beginning of the semester

So, if in the six-month period of a semester we exclude these four or five month vacation-related periods, not a lot of availability to review remains, especially if we add leaves of absence, sabbaticals, and conference abroad attendance justifications.

As scientists, and before killing the traditional (a.k.a. conventional) peer review system, let us make some calculations to explore what should be the real burden of the system for authors invited to review other’s manuscripts. This is to say, let us calculate the number of reviewers required per article published, using the conventional peer review system (following Kovanis et al.’s terminology), and considering that a manuscript, if rejected, is submitted to a different journal with the same rejection rate. The first journal received A articles and assigned R reviewers to each article, resulting in A*R total reviewers assigned. With a T rejection rate, that first journal will publish A*(1-T) articles. The remaining A*T articles will be submitted to a second journal that will assign the R reviewers to each article, resulting in a total of R*A*T reviewers, publishing (A*T)*(1-T) articles and rejecting A*T*T articles that will be submitted to a third journal. So, the total number of reviewers assigned to the initial A articles after a series of N journals will be:

And the number of articles published will be:

So, the total number of required peer reviewers per published article will be:

In fact, the number of reviewers per article published depends only on two variables: the number of peer reviewers assigned per manuscript and the journal’s rejection rate. The latter is expected to have an inverse (negative) correlation with the “climbing upwards” number of existing journals alleged by Rohn.18 Thus, with a commonly used number of three reviewers assigned to each manuscript received, a journal with an 80% rejection rate will need 15 reviewers to complete the task in order to publish one article.18 Figure 1 provides the shape of the series with two to five reviewers assigned per manuscript received.

Figure 1

Total number of reviewers required per article published as a function of a journal’s rejection rate

Colored lines represent the number of reviewers assigned per manuscript received

In plain language, to keep the scholarly peer reviewing publishing wheel spinning, the authors of each article published in a journal with an 80% rejection rate should review 15 manuscripts; and if the same research team published five articles in a given year, they should have reviewed 75 manuscripts. Considering an average of five authors per article, each author, in theory, should have to review fifteen manuscripts per every article that they publish. This does not seem to be an unreasonable number of manuscripts to review, but is higher than many researchers do. As a rule of thumb, in the case of an 80% rejection rate journal with three reviewers assigned per manuscript, the number of manuscripts each researcher should review per year is:

So, what makes authors perceive that they are overwhelmed with the number of invitations they receive to act as peer reviewers? The answer is quite obvious: to maintain the quality of the peer review system and avoid the overwhelming feeling, every author has to serve as a peer reviewer. When one author declines an invitation to review, another author will be invited, and so on. Reviewing three manuscripts per article published is not a hard job, but reviewing 15 manuscripts per article published, which could result in 75 reviews a year if you publish five articles, may be overwhelming. However, this is not a system problem, but a neglect of duty from the other four co-authors who should be sharing the task.

In 2019, Pharmacy Practice sent out 891 invitations to act as a peer reviewer, with 36 returned as undeliverable emails. From the remaining 855 invitations, 13 (1.5%) colleagues declared that the topic of the manuscript was outside of their expertise, 4 (0.5%) declared that they had a conflict of interest, 209 (24.4%) declined because they were busy, and 411 (48.1%) ignored the invitation altogether and did not reply to the email. Additionally, 7 individuals who had accepted the review never completed the task (12 reviews were ‘in progress’ at the time this editorial was written).

Can we solve this peer review crisis? Yes, we can. Before killing the system, we can try some of the many possible solutions. First and foremost, conducting an educational effort to raise awareness among authors of scientific articles that all should act as peer reviewers, not only the lead or the corresponding authors. Then, a practicality that some journals are implementing, email addresses of all the authors should be available. At the end of the day, per authorship requirements, all authors are responsible for the entire content of the article published. A second potential solution is to compensate reviewers for their time. The job of peer reviewers was traditionally associated with generosity and collegiality, or even just as a moral obligation. Compensating the review effort is still an unsolved issue.17,19 Third, we should accept that peer reviewers, when they perform a good review, contributed to the final version of the article more so than many of the individuals listed in the acknowledgements section. Unfortunately, journals, indexers, academic institutions and funding bodies are not considering these contributions as curricular merits. Three years ago, Pharmacy Practice started a new practice of including all peer reviewers of the past year as part of collective author in the first editorial of the new year. Thus, their names are searchable in PubMed using the [IR] field descriptor.15,20 Finally, a more complete and fair method of recognizing the contribution of a reviewer to the final version of the article, would be to list them in the article, which would require open peer reviews. Journals and indexers can organize systems to provide public recognition to open reviewers, but more educational efforts are required to change the mind of those defending the old-fashioned blind and double blind peer review processes.21,22 More drastic solutions may exist, but hopefully they will not be necessary.

Pharmacy Practice 2018 peer reviewers

Three reviews:

Margarida Castel-Branco, University of Coimbra, Portugal

Filipa A. Costa, ISCSEM, Portugal

Derek Stewart, Qatar University, Qatar

Two reviews:

Maria Cordina, University of Malta, Malta

Jack Collins, University of Sydney, Australia

Paul Dillon, Royal College of Surgeons, Ireland, Ireland

Sofia Kälvemark Sporrong, University of Copenhagen, Denmark

Damian Świeczkowski, Medical University of Gdansk, Poland

Van D. Tran, RUDN University, Russia

One review

Qalab Abbas, Aga Khan University Hospital, Pakistan

Ali A. Al-Jumaili, University of Iowa, United States

Abdelmajid H. Alnatsheh, Parkview Regional Medical Center, United States

Moawia Altabakha, Ajman University, United Arab Emirates

Wasem Alsabbagh, University of Waterloo, Canada

Chioma Amadi, City University of New York, United States

Johanna Aponte-González, Colombia National University, Colombia

Alejandro Arana, RTI Health Solutions, Spain

Ronen Arbel, Sapir College, Israel

Zubin Austin, University of Toronto, Canada

Minyon Avent, University of Queensland, Australia

Asnakew A. Ayele, University of Gondar, Ethiopia

David Balayssac, CHU Clermont-Ferrand, France

Claudio Barbaranelli, Sapienza University of Rome, Italy

Ben J. Basger, University of Sydney, Australia

Charlotte Bekker, Radboud University Medical Center, Netherlands

Durga Bista, Kathmandu University, Nepal

Aline F. Bonetti, Federal University of Parana, Brazil

Helena H. Borba, Federal University of Parana, Brazil

Marcel L. Bouvy, Utrecht University, Netherlands

Cecilia Brata, University of Surabaya, Indonesia

Rachele S. Britt, Beth Israel Deaconess Medical Center, United States

Lea Brühwiler, Patientensicherheit Schweiz, Switzerland

Sarah Brown, Cardiff Metropolitan University, United Kingdom

Josipa Bukic, University of Split, Croatia

Paul W. Bush, Duke University Hospital, United States

Ana C. Cabral, University of Coimbra, Portugal

Barry L. Carter, University of Iowa, United States

Kimberly L. Carter, University of Pennsylvania Health System, United States

Manuel J. Carvajal, Nova Southeastern University, United States

Afonso M. Cavaco, University of Lisbon, Portugal

Huan Keat Chan, Hospital Sultanah Bahiyah, Malaysia

Tyler Chanas, Vidant Medical Center, United States

Timothy F. Chen, University of Sydney, Australia

Bernadette Chevalier, University of Alberta, Canada

Allison M. Chung, Auburn University, United States

Mariann D. Churchwell, University of Toledo, United States

Richard Cooper, University of Sheffield, United Kingdom

Erika Cretton-Scott, Samford University, United States

Petra Czarniak, Curtin University, Australia

Ryan G. D’Angelo, University of the Sciences, United States

Rhian Deslandes, Cardiff University, United Kingdom

Shane P. Desselle, Touro University, United States

Parastou Donyai, University of Reading, United Kingdom

Aaron Drovandi, James Cook University, Australia

Julie Dunne, Dublin Institute of Technology, Ireland

Abubaker Elbur, Imam Abdulrahman Bin Faisal University, Saudi Arabia

Paul Forsyth, NHS Greater Glasgow & Clyde, United Kingdom

Victoria Garcia Cardenas, University of Technology Sydney, Australia

Miguel A. Gastelurrutia, University of Granada, Spain

Maria C. Gaudiano, Italian National Institute of Health, Italy

Natalie Gauld, University of Auckland, New Zealand

Chris M. Gildea, Saint Joseph Health System, United States

Ainhoa Gomez-Lumbreras, University Hospital Vall d’Hebron, Spain

Brian Godman, Karolinska Institute, Sweden

Jason R. Goldsmith, University of Pennsylvania, United States

Diego Gómez-Ceballos, Funiber, Colombia

Jean-Venable R. Goode, Virginia Commonwealth University, United States

Elisabeth Grey, University of Bath, United Kingdom

Olga Grintsova, Pharmacy of Detmold Post, Germany

Gerusa C. Halila, Federal University of Parana, Brazil

Nicola J. Hall, University of Sunderland, United Kingdom

Tora Hammar, Linnaeus University, Sweden

Drayton A. Hammond, Rush University, United States

Furqan K. Hashmi, University of Punjab, Pakistan

Mohamed A. Hassali, University of Science Malaysia, Malaysia

Andi Hermansyah, Airlangga University, Indonesia

Ludwig Höllein, University of Wuerzburg, Germany

Nejc Horvat, University of Ljubljana, Slovenia

Yen-Ming Huang, University of Wisconsin-Madison, United States

Klejda Hudhra, University of Medicine Tirana, Albania

Inas R. Ibrahim, Uruk University, Iraq

Katia Iskandar, Lebanese International University, Lebanon

Sherine Ismail, King Saud Bin Abdulaziz University, Saudi Arabia

Kristin K. Janke, University of Minnesota, United States

Kelsey L. Japs, VA Palo Alto, United States

Jennie B. Jarrett, University of Illinois at Chicago, United States

Jean-Pierre Jourdan, CHU de Caen Normandie, France

Maram G. Katoue, Kuwait University, Kuwait

Margaret Kay, University of Queensland, Australia

Clark D. Kebodeaux, University of Kentucky, United States

Thomas G. Kempen, Uppsala University, Sweden

Jennifer Kirwin, Northeastern University, United States

Nathalie Lahoud, Lebanese University, Lebanon

Anna Laven, Heinrich-Heine-University, Germany

Anandi V. Law, Western University of Health Sciences, United States

Miranda G. Law, Howard University, United States

Sukhyang Lee, Ajou University, South Korea

Leticia Leonart, Federal University of Parana, Brazil

Michelle D. Liedtke, University of Oklahoma, United States

Phei Ching Lim, Hospital Pulau Pinang, Malaysia

Amanda Wei Yin Lim, National Institutes of Health, Malaysia

Chung-Ying Lin, Hong Kong Polytechnic University, China

José Julián López, Universidad Nacional de Colombia, Colombia

Rosa C. Lucchetta, Federal University of Parana, Brazil

Karen Luetsch, University of Queensland, Australia

Elyse A. MacDonald, University of Utah Health Care, United States

Katie MacLure, Robert Gordon University, United Kingdom

Kurt Mahan, Presbyterian Healthcare Services, United States

Mark J. Makowsky, University of Alberta, Canada

Márcia Malfará, University of São Paulo, Brazil

Bejoy P. Maniara, James J. Peters VA Medical Center, United States

Brahm Marjadi, Western Sydney University, Australia

Gary R. Matzke, Virginia Commonwealth University, United States

Christopher McCoy, Beth Israel Deaconess Medical Center, United States

Tressa McNorris, Roseman University of Health Sciences, United States

Angelita C. Melo, Federal University of São João Del-Rei, Brazil

Zahra Mirshafiei Langaria, Shahid Beheshti University of Medical Sciences, Iran

Norazlina Mohamed, University Kebangsaan Malaysia, Malaysia

Jean Moon, University of Minnesota, United States

Michelle Murphy, Cooper University Hospital, United States

Sagir Mustapha, Ahmadu Bello University, Nigeria

Joseph Nathan, CVS Health, United States

Sujin Nitadpakorn, Chulalongkorn University, Thailand

Lucas M. Okumura, Clinical Hospital of Porto Alegre, Brazil

Edmund N. Ossai, Ebonyi State University, Nigeria

Courtney Pagels, Sanford Medical Center Fargo, United States

Subish Palaian, Ajman University, United Arab Emirates

Bridget Paravattil, Qatar University, Qatar

Nilesh Patel, University of Reading, United Kingdom

Guenka Petrova, Medical University Sofia, Bulgaria

Daphne Philbert, University Utrecht, Netherlands

Ann M. Philbrick, University of Minnesota, United States

Jill M. Plevinsky, Rosalind Franklin University, United States

Eng Whui Poh, Southern Australia Health, Australia

Bobby Presley, University of Surabaya, Indonesia

Urszula Religioni, Medical University of Warsaw, Poland

Oleksa G. Rewa, University of Alberta, Canada

Jadranka V. Rodriguez, University of Zagreb, Croatia

Sónia Romano, Centre for Health Evaluation & Research, Portugal

Olaf Rose, impac2t, Germany

Paula Rossignoli, Parana Health Secretariat, Brazil

Janelle F. Ruisinger, University of Kansas, United States

Hala Sacre, Lebanese Pharmacists Association, Lebanon

Wada A. Sadiq, Bayero University, Nigeria

Teresa M. Salgado, Virginia Commonwealth University, United States

Martina Salib, Royal Prince Alfred Hospital, Australia

Shane Scahill, University of Auckland, New Zealand

Terri Schindel, Edmonton Clinic Health Academy, Canada

Hanna Seidling, University of Heidelberg, Germany

Marguerite Sendall, Queensland University of Technology, Australia

Benjamin Seng, Duke-NUS Medical School, Singapore

Ana Seselja Perisin, University of Split, Croatia

Adji P. Setiadi, University of Surabaya, Indonesia

Amy Shaver, University at Buffalo, United States

Olayinka O. Shiyanbola, University of Wisconsin-Madison, United States

Tin Fei Sim, Curtin University, Australia

Bilge Sozen-Sahne, Hacettepe University, Turkey

Sidney Stohs, Creighton University, United States

Ieva Stupans, University of New England, Australia

André-Marie Tchouatieu, Medicines for Malaria Venture, Switzerland

Roberta Teixeira, National Institute of Cardiology, Brazil

Fitsum S. Teni, Addis Ababa University, Ethiopia

Fernanda S. Tonin, Federal University of Parana, Brazil

Jessica S. Triboletti, Butler University, United States

J. W. Foppe van Mil, Van Mil Consultancy, Netherlands

Tineshwaran Velvanathan, National University of Malaysia, Malaysia

Tara B. Vlasimsky, Denver Health Medical Center, United States

Helen Vosper, Robert Gordon University, United Kingdom

Sandy Vrignaud, University Hospital Center of Angers, France

Jennifer Walters, VCU Health, United States

Cheri K. Walker, Southwestern Oklahoma State University, United States

Geoffrey C Wall, Drake University, United States

Jocelyn A. Watkins, University of Warwick, United Kingdom

Mayyada Wazaify, University of Jordan, Jordan

Tommy Westerlund, Malmö University, Sweden

Sara A. Wettergreen, University of North Texas, United States

James S. Wheeler, University of Tennessee, United States

Kyle J. Wilby, University of Otago, New Zealand

Charlene Williams, University of North Carolina, United States

Aris Widayati, University Sanata Dharma, Indonesia

Matthew J. Witry, University of Iowa, United States

Seth E. Wolpin, University of Washington, United States

David Wright, University of East Anglia, United Kingdom

Nancy Yunker, Virginia Commonwealth University, United States

Ismaeel Yunusa, Massachusetts College of Pharmacy and Health Sciences, United States