A protocol of a cross-sectional study evaluating an online tool for early career peer reviewers assessing reports of randomised controlled trials.

INTRODUCTION: Systematic reviews evaluating the impact of interventions to improve the quality of peer review for biomedical publications highlighted that interventions were limited and have little impact. This study aims to compare the accuracy of early career peer reviewers who use an innovative online tool to the usual peer reviewer process in evaluating the completeness of reporting and switched primary outcomes in completed reports. METHODS AND ANALYSIS: This is a cross-sectional study of individual two-arm parallel-group randomised controlled trials (RCTs) published in the BioMed Central series medical journals, BMJ, BMJ Open and Annals of Emergency Medicine and indexed with the publication type 'Randomised Controlled Trial'. First, we will develop an online tool and training module based (a) on the Consolidated Standards of Reporting Trials (CONSORT) 2010 checklist and the Explanation and Elaboration document that would be dedicated to junior peer reviewers for assessing the completeness of reporting of key items and (b) the Centre for Evidence-Based Medicine Outcome Monitoring Project process used to identify switched outcomes in completed reports of the primary results of RCTs when initially submitted. Then, we will compare the performance of early career peer reviewers who use the online tool to the usual peer review process in identifying inadequate reporting and switched outcomes in completed reports of RCTs at initial journal submission. The primary outcome will be the mean number of items accurately classified per manuscript. The secondary outcomes will be the mean number of items accurately classified per manuscript for the CONSORT items and the sensitivity, specificity and likelihood ratio to detect the item as adequately reported and to identify a switch in outcomes. We aim to include 120 RCTs and 120 early career peer reviewers. ETHICS AND DISSEMINATION: The research protocol was approved by the ethics committee of the INSERM Institutional Review Board (21 January 2016). The study is based on voluntary participation and informed written consent. TRIAL REGISTRATION NUMBER: NCT03119376. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2017. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

To the best of our knowledge, this will be the first study that will assess the efficacy of using an innovative online tool to help early career peer reviewers to assess the quality of reporting of randomised controlled trials.

A randomised selection of trials from a large panel of biomedical journals.

The absence of background for choosing the a priori effect size.

Introduction

The peer review process is a cornerstone of biomedical research publication.1 The editorial peer review process is described as journal editors relying on the views of independent experts in making decisions on, for example, the publication of submitted manuscripts or presentation of reports at meetings.2 The peer review system is considered the best method for helping scientific editors decide on the acceptability of a manuscript for publication.3 Nevertheless, the effectiveness of the system has been questioned. In 2010, a report commissioned by the UK House of Commons showed that the cost to the UK Higher Education Institutions in terms of staff time was £110 to £165 million per year for peer review and up to £30 million per year for the work done by editors and editorial boards.4 Worldwide, peer review is estimated to cost £1.9 billion annually and accounts for about one-quarter of the overall costs of scholarly publishing and distribution.5 The human resources were estimated to be about 15 million hours by 2010.6

Furthermore, the peer review system may fail to identify important flaws in the quality of submitted manuscripts and published articles.7 8 A recent study by Hopewell et al 9 showed that peer reviewers failed to detect important deficiencies in the reporting of methods and results of randomised trials. There is also considerable indirect evidence of the failure of peer review. Systematic reviews of clinical questions highlight a high prevalence of key information not being provided in peer-reviewed articles. Moreover, there is evidence to suggest that peer reviewers are not able to detect fraud,10 mistakes11 12 or spin.13 14 Systematic reviews evaluating the impact of interventions to improve the quality of peer review for biomedical publications highlighted few randomised controlled trials (RCTs) and that interventions such as training have little or no impact.15 16

How the peer review process is currently organised can explain the difficulties encountered. A number of tasks are expected from peer reviewers when assessing reports of an RCT.17 They have to assess the completeness of reporting, whether outcomes were switched from that reported in the trial registry, the choice of the comparator and intervention, the relevance of outcomes, the risk of bias, the external validity, the presence of spin, the originality, importance and relevance of the RCT to the journal’s readers, etc. It may be unrealistic to expect a peer reviewer to complete all of these tasks.

Furthermore, the assessment of the completeness of reporting should be a prerequisite in the peer review process, because lack of transparency is a barrier to the assessment of important aspects of a trial such as the risk of bias18 and reproducibility.19 Another essential task of peer review is the identification of switched outcomes (ie, a discrepancy between the prespecified outcomes reported in the trial registry or protocol and outcomes reported in the manuscript). To avoid reporting bias(es),20 all prespecified outcomes should be reported in the manuscript, and authors should report any changes to trial outcomes after the trial started, with reasons.21 The Centre for Evidence-Based Medicine Outcome Monitoring Project (COMPare) assessed RCTs published in the five general medical journals with the highest impact factor. They compared the outcomes published to the outcomes recorded in trial registries or protocols. Among the 67 RCTs examined in July 2016 (www.COMPare-Trials.org), only nine had no discrepancies; 354 outcomes were not reported and 357 new outcomes were ‘silently’ added.18

Despite being essential, the assessment of the completeness of the reporting and the identification of switched outcomes are not appropriately performed.9 22 These tasks do not require a high level of expertise and could be performed by researchers early on in their career. Early career peer reviewers could use a simple online tool based on the CONSORT 2010 checklist and Elaboration and Explanation publication20 for reporting parallel-group RCTs developed according to the same principles used for the CobWeb tool.23

CobWeb is a writing aid tool based on CONSORT 2010. It was developed to provide guidance to authors when writing a manuscript of an RCT. The content of the tool was based on the CONSORT 2010 checklist and the Explanation and Elaboration document. For each domain, the tool comprised the corresponding CONSORT checklist item(s), bullet points with the key elements that need to be reported extracted from the Explanation and Elaboration document, as well as (an) example(s) of good reporting.

The peer reviewer tool will use the existing CobWeb elements reworded to be appropriate for peer reviewers and for the items not considered in CobWeb, we will extract the bullet points from the Explanation and Elaboration document of the CONSORT 2010 following the same process.21

The tool will feature bullet points eliciting the meaning of each checklist item. This tool will allow for standardising the completion of this task, be an opportunity to train early career peer reviewers to become more expert peer reviewers and be used to provide standardised and personalised feedback to authors based on the CONSORT items and bullet points; the authors would receive a request to report each item (with associated bullet points) that the early career peer reviewer identified as being inadequately reported.

The objectives of this project are as follows:

Develop an online tool and training module based (a) on the CONSORT 2010 checklist and the Explanation and Elaboration document modified for early career peer reviewers (ie, early stage researchers: masters students, PhD students, residents involved in clinical research during their study and clinicians involved in clinical research who never peer reviewed a manuscript) for assessing the completeness of reporting of 10 key items and (b) the COMPare process used to identify switched in primary outcomes in completed reports of RCTs.

Compare the performance of early career peer reviewers who use the peer reviewer tool with usual peer reviewers in identifying inadequate reporting and switched outcomes in completed reports of RCTs.

The planning, conduct, analysis and reporting of the study were discussed by the study’s scientific committee.

Methods and analysis

Study design

We approached the issue of peer reviewing as if it were a diagnostic test aimed at detecting incomplete reporting. Before evaluating this ‘diagnostic test’ in an RCT, we aimed to evaluate its accuracy in a cross-sectional study. For this purpose, we defined a gold standard (ie, assessment by systematic reviewers) and two diagnostic tests to be compared: usual peer reviewers and early career peer reviewers trained and using the online tool.

This is a cross-sectional diagnostic study comparing the accuracy of early career peer reviewers who use the tool to that of usual peer reviewers when evaluating the completeness of reporting and a switched in primary outcome(s) in completed reports of RCTs at the first submission (figure 1). All procedures in the study will be consistent with ethical practice. This study received ethical approval from the Institutional Review Board of the INSERM ethics committee (IRB 17–335). The protocol was written following the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) guidelines.

Figure 1

The study design.

The tool: COBPeer

Principles of the tool

The online peer review tool provides guidance to early career peer reviewers when they are assessing the completeness of reporting of RCTs based on the relevant CONSORT 2010 checklist items, switched in primary outcomes from trial registry records based on the process used in COMPare and the associated CONSORT 2010 Elaboration and Explanation document.

Before using the tool, the early career peer reviewer will undergo an online training exercise consisting of evaluating extracts of manuscripts for each CONSORT item and corresponding trial registration records with specific feedback according to their answers.

The content of the tool and training will be tested and validated by the scientific committee of the study.

Completeness of the reporting of RCTs section

To assess the completeness of reporting of RCTs, the tool will be developed according to the same principles used for developing the CobWeb tool.23

We will focus on the CONSORT items that are least often reported (ie, reported in less than 50% of the reports assessed as shown in the study published by Hopewell et al 9) and are necessary to conduct a meta-analysis (ie, items of the risk of bias tool, outcomes data, safety data).

The selected items are:

five items in the methods section (outcomes (6a), randomisation/sequence generation (8a), allocation concealment mechanism (9), blinding (11 a, 11b));

four items in the results section (participant flow (13a, 13b), outcomes and estimation (17a), harms (19));

one item (23) dedicated to trial registration (reporting registration number)

We decided to focus on these items because they are essential for systematic reviewers when evaluating the risk of bias and recording the outcome data.

Of note, we relied on the CONSORT extension for harm24 to develop the bullet point for item 19 and on the SPIRIT guidelines explanatory document25 to develop the bullet point for item 6a.

An item will be considered adequately reported if all bullet points related to this item are indicated as being reported. For example, item 11 ‘Blinding’ will be considered adequately reported if the authors reported all the information below:

How the blinding was performed (eg, used of placebo, intervention by physician unaware of the study).

The similarities of the characteristics of the interventions (eg, appearance, taste, smell, method of administration).

The content of the tool is available in table 1.

Table 1

The most important items of the Consolidated Standards of Reporting Trials (CONSORT) 2010 statement and bullet points that early career peer reviewers will assess

CONSORT Item	CONSORT item
Outcomes	Item 6a. Completely defined prespecified primary outcome measures, including how and when they were assessed i
		Yes	No
	Was the primary outcome(s) clearly identify (eg, the primary/main outcome was pain)?
	If no go to next section If yes answer the following questions, Please check if the author clearly report for the primary outcome(s): The variable of interest (eg, pain, all-cause mortality) How the outcome was assessed (eg, VAS, Beck Depression Inventory score, pain scale) The analysis metric (eg, change from baseline, final value, time to event) The summary measure for each study group (eg, mean, proportion with score>2) Time point of interest for analysis (eg, 3 months)*NA if survival analysis Who assessed the outcome (eg, the patient, doctor, nurse, caregiver, other)
Randomisation Sequence generation	Item 8a. Method used to generate the random allocation sequence
	Did the author report: The method of sequence generation (eg, a random number table or computerised random number generator, or other)	Yes	No
Allocation concealment	Item 9. Mechanism used to implement the random allocation sequence (eg, sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned
	Did the author report: How the care provider enrolling patients was blinded to the next assignment in the sequence. Possible methods can rely on For centralised or ‘third-party’ assignment (ie, use of a central telephone randomisation system, automated assignment system) Having a third party prepare the randomisation list and hide the allocation assignment in advance via numbered identical bottles or sequentially numbered, sealed, opaque envelopes If the mechanism of the random allocation sequence is completely described but the sequence is not adequately concealed, please tick yes	Yes	No
Blinding	Was the study blinded yes/no If yes go to 11a If no go to 13a Item 11a. If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how Item 11b. If relevant, description of the similarity of interventions
	Did the author report: Who (ie, participants, healthcare providers, data collectors, outcome adjudicators, and data analysts) was blinded to treatment assignments? How was performed the blinding? (eg, used of placebo, intervention by physician unaware of the study) The similarities of the characteristics of the interventions (eg, appearance, taste, smell, method of administration)*NA	Yes	No
Participant flow	Item 13a. For each group, the numbers of participants who were randomly assigned, received intended treatment and were analysed for the primary outcome Item 13b. For each group, losses and exclusions after randomisation, together with reasons
	Did the authors report a flow chart	Yes	No
	Did the author report in the flow chart or in the text: Number of participants randomised in each group Number of participants who received intended treatment in each group Number of participants did not receive the allocated treatment with reasons in each group Number of participants lost to follow-up with reasons in each group Number of participants who discontinued intervention with reasons in each group Number of participants analysed for the primary outcome in each group Number excluded from analysis with reasons in each group	Yes	No
	Outcomes and estimation	Item 17a. For each primary outcome, results for each group, and the estimated effect size and its precision (such as 95% CI)
	Did the author report for primary outcome: (answer yes if it is true for all primary outcomes) Result in each group (mean (SD) or nb of event/n) Difference in estimated effect between groups (eg, OR, risk ratio (RR), risk difference (RD), HR, difference in median survival time, mean difference (MD)) Precision for difference between groups (eg, 95% CI)	Yes	No
Harms	Item 19. All important harms or unintended effects in each group
	Did the author report: How harms-related information was collected (eg, mode of data collection, timing, attribution methods) For each group, participant withdrawals due to harm Results in each group for each harms type with denominator (mean (SD) or nb of event/n)	Yes	No
Registration	Item 23. Registration number and name of registry
	Did the author report: The registration number	Yes	No
Consistency between data registered and reported
Did authors report the same primary outcome in the register and manuscript (same variable, same metric, same time point) or was the primary outcome added, deleted, changed
If the study was not registered, please tick the box □
If the study was registered, report  □ The link to the online registry:  □ Date of the registration:  □ Date of the start of the study:
Was the primary outcome(s) reported in the register or manuscript not sufficiently described to identify switch in outcomes?		Yes	No
Did you identified any outcome(s) reported by the authors as primary outcome(s) while not registered as such?		Yes	No
Did you identified any outcome(s) registered as primary outcome but not reported as such in the manuscript?		Yes	Noi
Did you identified any change in terms of time frame, metric or other information between the primary outcome(s) registered and reported in the manuscript?		Yes	No
If yes, please list the discrepancies:
–
–
Did the authors justify the switched outcome(s) in the manuscript?		Yes	No	NA

Switched outcomes section

The switched outcomes section will be based on the process used in COMPare that was simplified. The early career peer reviewer will have to:

find the trial’s registry entry, by using http://google.com/;

copy the link to the online registry;

copy the date of the first registration;

copy the date of the beginning of the study;

copy the prespecified original primary outcome(s) from the registry;

compare the primary outcome(s) reported in the manuscript to the primary outcome(s) reported in the registry and determine whether there was a switch in outcome (ie, primary outcome added, deleted or changed);

indicate if authors justified the switched outcome(s) in the manuscript.

Overall the RCT manuscript will be classified as

no switch in outcome;

presence of switch in outcomes (at least one outcome was switched);

unable to assess.

Of note, the early career peer reviewer will not search for the protocol or assess the protocol as was done in COMPare because of time constraints.

Online training module

The online training module will start with a short reminder about the importance of the completeness of reporting and the assessment of switched outcomes from the trial register. For each item, the tool will provide bullet points of the item with explanations. Then, early career reviewers will assess passages from RCTs that are adequately or inadequately reported by using the tool. For each item assessed, participants will receive personalised feedback with a detailed explanation of the reported and missing bullet points.

Participants

Inclusion criteria

Early career peer reviewers are defined as early stage researchers: master students, PhD students, residents involved in clinical research during their study and clinicians who have never reviewed a manuscript.

Exclusion criteria

Participants will be excluded if they have already peer reviewed one RCT.

Setting and recruitment

We will use a large strategy of participants’ recruitment. Participants will be identified from:

editors of biomedical journals. We will contact specifically contact editors of journals interested in research in peer review field (eg, BMC Medicine, Trials, BMC Peer Review and Research Integrity);

learnt societies (eg, European Society of Emergency Medicine);

network of international students (eg, Students 4 Best Evidence);

University Paris Descartes and Paris Diderot students.

We will send a standardised email to their contacts for promoting our study. The email will invite people to participate in an online training course on peer review. They will be informed of the inclusion criteria and about the different steps of the study (to complete an online training module and then peer review one manuscript randomly selected from our sample). If they agree to take part, they will then log onto the system.

Blinding

Evaluators (systematic reviewers) will be blinded to the peer reviewers’ assessment, early career peer reviewers’ assessments and the content of the tool. They will have access to only one manuscript to assess. They will be instructed to base their assessment on only the content of the manuscript.

The evaluation of peer reviewers was blinded as performed before the study.

Early career peer reviewers will be blinded to the aim of the study, systematic reviewers’ assessments and peer reviewers’ comments. Systematic reviewers’ assessments will not be accessible. Peer reviewers’ comments in theory are accessible for articles published in BioMed Central series medical journals, BMJ and BMJ Open; however, the journal name, authors’ names and study title will be masked to early career peer reviewers.

Manuscript selection

We searched the US National Library of Medicine’s PubMed database to identify all primary reports of two-arm parallel-group RCTs with an individual randomisation unit, whatever the treatment evaluated, that were published in the BioMed Central series medical journals, BMJ, BMJ Open and Annals of Emergency Medicine and indexed with the publication type ‘Randomised Controlled Trial’. We included journals from BioMed Central series medical journals publishing at least 5 RCT reports per year between 1 January 2015 and 13 December 2016. We retrieved 1600 citations. One researcher screened all titles and abstracts and included all primary reports of RCTs assessing interventions in human participants who were randomly allocated to intervention groups. We excluded cluster RCTs, cross-over trials, equivalence and non-inferiority trials, feasibility studies, cost-effectiveness studies, phase I trials, study protocols, non-randomised controlled trials, secondary publications or analyses, pilot studies, systematic reviews, methodology studies and early phase studies. The screening process identified 222 eligible reports, among which we randomly selected 120 reports of RCTs. For each RCT, we retrieved the submitted manuscript and the first-round peer reviewers’ comments (available online or requested from editors/publishers). Citations for 17 RCTs with documents not available were replaced by 17 other randomly selected RCTs. Overall, we identified 120 manuscripts from 24 different journals.

Interventions

Gold standard

The gold standard will be the assessment of the completeness of reporting by systematic reviewers of the 10 CONSORT items for the first manuscript submitted, (ie, before any changes were made as part of the peer review process). This assessment corresponds to the assessment of the risk of bias (random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, selective reporting of outcomes) and the extraction of efficacy and harm outcome data. Pairs of systematic reviewers will independently extract data from eligible reports; any differences between reviewers will be resolved by discussion, with the involvement of an arbitrator if necessary. Reviewers involved in the data extraction will have expertise in the conduct of systematic reviews and will assess the completeness of reporting from the systematic-reviewer perspective. They will not have access to the tool to avoid being influenced by the tool. Consequently, the gold standard will rate items as inadequately reported only if the reporting is a real barrier to the conduct of a systematic review. This situation will ensure that the assessment of the early career peer reviewer who use the tool is identifying inadequate reporting that is not relevant.

The systematic reviewers will also systematically compare the outcomes reported in the manuscript and the outcomes reported in the registry and will document any discrepancies.

Overall, for the first manuscript, before any changes were made as part of the peer review process, each item will be classified as adequately reported (yes/no) for the gold standard.

Online tool used by early career peer reviewers

If early career peer reviewers agree to participate, they will start the training module. If they successfully complete the training module, they will then download one randomly selected manuscript and assess at the first version submitted at the first round by using the online tool. They will be asked to perform this assessment in 30 min maximum.

There will be one early career peer reviewer per manuscript.

Usual peer reviewer

The usual peer review process assessment of the completeness of reporting and switched outcomes will be assessed from the peer reviewers’ reports. Two researchers will read all the peer review reports for the first round. The researchers will determine whether the peer reviewers raised some concerns about the completeness of reporting of the 10 CONSORT items considered and identified a switch in outcomes between the manuscript and the register. The assessment of all peer review reports for each manuscript will be combined (ie, the item will be rated as incompletely reported if at least one peer reviewer rated it as such). The two researchers will be blinded to the gold standard assessment.

Outcomes

Main outcome measure

The primary outcome will be the mean number of items accurately classified per manuscript initially submitted to the journal.

For the completeness of the reporting of RCTs section; 10 CONSORT items will be considered: Outcomes (6a), Randomisation/sequence generation (8a), Allocation concealment mechanism (9), Blinding (11 a, 11b), Participant flow (13a, 13b), Outcomes and estimation (17a), Harms (19) and one item (23) dedicated to trial registration.

Each item will be classified as ‘adequately reported’ (yes/no).

For the switched outcomes section, the manuscript will be classified as:

no switch in primary outcome;

presence of switch in primary outcomes (at least one outcome was switched);

unable to assess.

Secondary outcome measure

The secondary outcomes will be the mean number of items accurately classified per manuscript for the 10 CONSORT items and the sensitivity, specificity and likelihood ratio to detect the item as adequately reported and to identify switch in outcomes.

Because the gold standard may be less accurate than the tool used by early career peer reviewers in detecting inadequate reporting, we will systematically submit apparent false-positive results to experts identified by the scientific committee, who will indicate whether they believe the item is adequately reported (table 2).

Table 2

Planned tabulations

		Gold standard
		Item completely reported	Item incompletely reported
Early career peer reviewers	Item completely reported	True positive	False positive
	Item incompletely reported	False negative	True negative
Usual peer reviewers	Item completely reported	True positive	False positive
	Item incompletely reported	False negative	True negative

Item adequately classified=True positive+true negative.

The primary outcome will be the mean number of items accurately classified per manuscript across all the trial reports that are assessed.

Statistics

Sample size calculation

We will allow for detecting an effect size of 0.3 for the mean number of items accurately classified per manuscript with a power of 90% and a two-sided alpha-level of 5%.

Each manuscript will be assessed by one early career peer reviewer; each early career peer reviewer will assess one manuscript. So, we will randomly select 120 reports of RCTs to be assessed in the study. We do not have any data on the mean (SD) number of items accurately classified by usual peer reviewers. For each manuscript, we will have a single assessment related to the usual peer review process that will combine the first assessment of all peer reviewers for that manuscript.

Statistical analysis

The primary outcome will be the mean number of items accurately classified per manuscript by usual peer reviewers and by early career peer reviewers, the gold standard being the systematic reviewers’ assessment. We will compare the mean number of items accurately classified per manuscript by early career peer reviewers versus usual peer reviewers. This will be performed using a paired t-test. The sensitivity and specificity comparing the early career peer reviewers and the usual peer-reviewers’ assessment will be determined by a test for pair proportion.

Data management and monitoring

Data will be entered on at a secure, password-protected, website. As part of the data protection, confidentiality and anonymity requirements, all participants will be assigned a unique individual identifier. The database will be managed by the main investigators (CA and BI). Access will be limited to individuals deemed appropriate. No interim analyses will be performed.

Ethics and dissemination

Authorisation by the Commission Nationale de l’Informatique et des Libertés (file number 2021376) whose purpose is to protect participants’ personal data and the institutional review board of the INSERM ethics committee (IRB 17–335) (see Appendix 1 in the online supplementary file 1) was obtained and the study protocol was registered at ClinicalTrials.gov (http://clinicaltrials.gov, NCT03119376). Any changes or additions to the protocol will be registered on ClinicalTrials.gov. Participation will be voluntary. All participants will provide informed consent and may withdraw at any time (see Appendix 2 in the online supplementary file 1). Following their participation, they will receive one Continued Medical Education credit.

The results will be presented at international congresses, published in a peer-reviewed journal and summarised on ClinicalTrials.gov. In addition, the findings will be disseminated via an open access website so that they can be used by the wider community. Study results will also be disseminated directly to participants.

Any changes or additions to the protocol will be registered on Clinicaltrials.gov. Authorship of papers arising from this study will be based on contributions to the study including intellectual content as recommended by the International Committee of Medical Journal Editors.

The data set will be the property of the scientific committee. The results will be presented at international congresses, published in a peer-reviewed journal and summarised on ClinicalTrials.gov

Discussion

Several researchers have highlighted that the quality of reporting of peer-reviewed and editor-approved clinical research is poor.26–28 However, because biomedical journals remains the best method for disseminating biomedical knowledge, it is fundamental to assess the implementation of novel interventions and must be assessed in different actors of the publication system to improve its efficiency and the quality of reporting. In this study, we limited our research to interventions to improve quality of reporting through peer review. To the best of our knowledge, this will be the first study of an innovative online tool to help early career peer reviewers to assess the quality of reporting of RCTs.