Quality of the Reviews Submitted by Attendees of a Workshop on Peer Review.

OBJECTIVE: The objective of the study was to study the methodological quality and error detection of the review by the participants of a peer review workshop.
METHODS: All participants of the workshop were invited to peer review a randomized controlled trial. The manuscript was E-mailed to them after introducing eight deliberate errors to it. Specific instructions and a deadline were provided. All the reviews were analyzed using review quality instrument (RQI). Furthermore, the rate and the type of errors identified were recorded.
RESULTS: Of 25 participants, 16 (64%) returned the reviews. The mean total score on RQI was 4.12 (standard deviation 0.70, 95% confidence interval 3.74-4.50); the items which most reviewers did not discuss where the importance of research question and originality of the paper. The number of errors correctly identified varied from 0 to 6 (median 3), the most common being a wrong conclusion (87.5%), randomization procedure (50%), written informed consent (50%), ethics committee approval (42.8%), and masking (31.2%). Only 5 (31.2%) gave an overall recommendation on whether the manuscript should be accepted or not.
CONCLUSIONS: Major errors were readily identified by the reviewers; however, the need for training was felt in some areas in which the review quality was modest.

INTRODUCTION

External peer review is a method used across the biomedical journals to help the editor to detect flaws and select and improve submitted manuscripts.[1] Although it is used worldwide, the effect of peer review is still uncertain as found in a systematic review.[2] With a rapid growth in research and increased specializations, there is an exponential surge in the number of manuscripts submitted for publication which overburdens qualified reviewers. As a result, most of the peer review is actually carried out by amateur reviewers, leading to wide variability in the quality of peer review. Fortunately, the final decision to accept or reject the manuscript lies with the editor. Indeed, good peer review improves the chances of acceptance and helps to maintain high publishing standards.[3] The reviewer characteristics that correlated with higher review quality were younger age and association with academic institutions.[45]

Peer review is not immune to bias. It may be related to the content of manuscript, confirmation to beliefs held by reviewer, acceptance of positive results while rejecting negative ones, and those related to conflict of interest.[6] However, in the absence of any alternate method, peer review, although imperfect, is still considered as necessary to maintain the quality of the published manuscripts. It provides a unique opportunity for not only acknowledging the strengths in the manuscript but also possible detection of errors or flaws and subsequent correction before its publication. Several strategies have been used to enhance the peer review quality, but they may not be always effective. For example, double blinding of peer review increased the work of editorial staff without having any positive effect of review quality.[7] It has been found that lack of training of peer reviewers can affect the quality of published manuscripts.[8] Open peer reviews and peer review training workshops are options to improve the quality of peer reviews.[910]

Studies have revealed variable rates of error detection during peer review.[1] Indeed, the editors’ rating of individual reviewers correlated with the reviewers’ ability to report flaws in the manuscript.[11] Previous studies have shown that reviewers have missed deliberate errors introduced in the manuscripts.[12131415] Training of the peer reviewers improved the rate of error detection in Schroter et al.[15] study. The current study was planned in conjunction with a peer review workshop to identify the quality of the reviews the participants did before the workshop. The objective was to study the methodological quality and error detection of the review by the participants of a peer review workshop.

METHODS

This was an observational study carried out during a peer review workshop conducted by the authors at Kochi, Kerala, in South India, as a part of capacity building exercise organized by Kerala Journal of Psychiatry in 2015. The study sample consisted of participants who attended the workshop. All of them were qualified psychiatrists either in private practice or associated with teaching hospitals as faculty. They were either already doing peer review or were identified as potential peer reviewers for Kerala Journal of Psychiatry.

Before the workshop, all the participants registered for it were invited to perform peer review of a manuscript. An open-access, published “randomized controlled trial (RCT)” was chosen as manuscripts on RCTs have been found to provide more structure for review as compared to other research papers.[15] To maintain anonymity, the names of the authors and the details of study location were removed from the manuscript. In addition, eight deliberate errors were introduced in the manuscript. The errors were in the areas of hypothesis, randomization method, allocation concealment, masking, study site, IRB approval, informed consent, and wrong conclusion. The manuscript was E-mailed to all the participants with a request to return them 1 week before the date of workshop.

All the available reviews were analyzed by the first author, SKP, using review quality instrument (RQI) version 3.2.[9] RQI has 8-item that are scored on a 5-point Likert scale with higher scores indicating higher quality (1 = poor and 5 = excellent). It assesses the comments of the reviewer on five aspects of a manuscript (importance of the research question, originality of the paper, strengths and weaknesses of the method, presentation, interpretation of results) and two aspects of the review (constructiveness and substantiation of comments). In addition, the rate and type of errors identified by the participants were also recorded.

Data obtained were analyzed using Statistical Package for the Social Sciences (SPSS) for Windows, version 16.0, Chicago, SPSS Inc. Bar graph with error bars showing 95% confidence intervals (CIs) was used to summarize the RQI scores.

RESULTS

Out of 25 participants, 16 (64%) returned the reviews on time. The mean total RQI score was 4.12 (standard deviation [SD] 0.70, 95% CI 3.74–4.50). The RQI scores for each item are summarized in Figure 1. The reviews were courteous (mean 3.5, SD 0.7), commented on presentation (mean 3, SD 0.7), were constructive (mean 3, SD 0.7), and were well substantiated (mean 3, SD 0.9). However, originality (mean 1, SD 0) and importance (mean 1.2, SD 0.8) of research question were given least importance.

Figure 1

Mean review quality instrument scores of participants (n = 16)

Figure 2 shows the errors identified by the reviewers. Number of errors in the manuscript that were correctly identified varied from 0 to 6 (median 3); 10 (62.5%) identified at least three or more errors. The errors not identified varied from 2 to 8 (median 5); 12 (75%) did not identify five or more errors. The most common error identified was “wrong conclusion,” by 14 (87.5%) participants, followed by “randomization method” and “informed consent” by 8 (50%) each. The errors that were identified least were “hypothesis” (6.2%) and “allocation concealment” (12.5%). However, an overall recommendation was given by only 5 (31.2%) reviewers.

Figure 2

Errors identified by the reviewers (n = 16)

DISCUSSION

The mean total RQI scores of the participants was 4.12 (SD 0.7), which is higher in comparison to the previous studies.[5910] It has been found that assigning reviewers with age <40 years, from a top academic institution, well known to the editor, and those blinded to the identity of the manuscript's authors, the probability of getting a good review was 87%.[4] In the study, although we have not examined these characteristics, it is likely that those participating in the workshop for peer reviewers may be motivated to produce a good review, which explains the higher RQI scores. In this study, most reviewers did not discuss the importance of research question and none discussed originality of the paper, in contrast to previous reports.[910] Indeed, the research question has been considered as the single most important component of the study.[16] To arrive at a research question, one has to be familiar with the current knowledge and the lack of information in a particular area, mostly after a systematic review of literature.[17] Furthermore, most research is incremental, i.e., would expand our knowledge in a particular area in small steps. Thus, identifying the originality of the research is a challenge for the reviewers, and would be based on the gaps in the knowledge as ascertained by the systematic review.

It is the responsibility of the reviewers to comment on the paper in a polite and constructive manner, even if the final recommendation is rejection. It was reassuring to find that the tone of the peer review was courteous in 94% (15/16) of the reviews, and none of them was overtly abusive. It has been reported that peer reviews are more courteous when they are signed in an open review as compared to unsigned reviews.[18] A “down-to-earth” approach has been suggested by Roberts et al.[19] for reviewing a manuscript and recommend to provide constructive feedback for all the manuscripts.

An average of 37% (3 out of 8) of the deliberate errors were identified by reviewers, which is similar to the previous studies which found that more than two-third did not identify errors.[1315] Most common errors detected were: wrong conclusion (87.5%), randomization procedure (50%), and written informed consent (50%); similar to findings of Schroter et al.[15] where reviewers readily identified errors in sampling and randomization. Errors that were not identified include “no hypothesis,” “allocation concealment,” whereas, wrong conclusions were not identified by majority in previous reports.[1315] Only 5 (31.2%) gave overall recommendation whether the manuscript should be accepted or not, much less than previous reports. It is suggested that an overall recommendation is given by all the peer reviewers, whether to accept as it is or with modifications or to reject the manuscript, as it aids the editor in making a decision.

Limitations of the current study include small sample and participants from a peer review workshop, thus limiting generalizability. In conclusion, major errors were readily identified by the reviewers. However, the review quality was modest in some areas which require further improvement. The authors need to be aware of guidelines on how to perform peer review of RCTs such as consolidated standards of reporting trials statement and checklists.[20] Furthermore, reviewers may be suggested to look for the major and minor flaws in a study using standard reporting guidelines.[21] Training workshops may be conducted to raise awareness about recent reporting guidelines.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.