State of reporting of primary biomedical research: a scoping review protocol.

INTRODUCTION: Incomplete or inconsistent reporting remains a major concern in the biomedical literature. Incomplete or inconsistent reporting may yield the published findings unreliable, irreproducible or sometimes misleading. In this study based on evidence from systematic reviews and surveys that have evaluated the reporting issues in primary biomedical studies, we aim to conduct a scoping review with focuses on (1) the state-of-the-art extent of adherence to the emerging reporting guidelines in primary biomedical research, (2) the inconsistency between protocols or registrations and full reports and (3) the disagreement between abstracts and full-text articles. METHODS AND ANALYSES: We will use a comprehensive search strategy to retrieve all available and eligible systematic reviews and surveys in the literature. We will search the following electronic databases: Web of Science, Excerpta Medica Database (EMBASE), MEDLINE and Cumulative Index to Nursing and Allied Health Literature (CINAHL). Our outcomes are levels of adherence to reporting guidelines, levels of consistency between protocols or registrations and full reports and the agreement between abstracts and full reports, all of which will be expressed as percentages, quality scores or categorised rating (such as high, medium and low). No pooled analyses will be performed quantitatively given the heterogeneity of the included systematic reviews and surveys. Likewise, factors associated with improved completeness and consistency of reporting will be summarised qualitatively. The quality of the included systematic reviews will be evaluated using AMSTAR (a measurement tool to assess systematic reviews). ETHICS AND DISSEMINATION: All findings will be published in peer-reviewed journals and relevant conferences. These results may advance our understanding of the extent of incomplete and inconsistent reporting, factors related to improved completeness and consistency of reporting and potential recommendations for various stakeholders in the biomedical community. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/.

In this scoping review, we will assess the consistency and completeness of reporting in the biomedical literature with regards to adherence to reporting guidelines, consistency between protocols or registrations and full reports and agreement between abstracts and full-text articles.

Results from our study will advance our understanding of the extent of incomplete and inconsistent reporting, factors related to improved completeness and consistency of reporting and potential recommendations for various stakeholders in the biomedical community.

A potential limitation may be the small number of eligible studies for this scoping review.

Introduction

Primary research is generally defined as the empirical research studies with collection of original primary data.1 The current reporting in primary biomedical research remains an issue of concern in the literature.2 For instance, it is widely recognised that incomplete reporting is pervasive in biomedical research, leading to potential waste of resources, sceptical interpretation of findings and even scientific misconduct.2 One study showed that over 50% of research findings were not sufficiently or completely reported to make them usable or replicable, which represented a substantial waste of resources and efforts.3 Likewise, it is difficult to make an informed judgement about the risk of bias and credibility of findings in a study due to its incomplete reporting and lack of linkage to protocol or registration.4 Moreover, incomplete reporting can result in unnecessary exposure or harm to patients and lead to imprecise or biased treatment effect estimates to inform decision-making.2 5 To improve transparent and complete reporting in biomedical research, reporting guidelines have been developed and widely adopted by more and more journals. The EQUATOR (Enhancing Quality and Transparency of Health Research) network provides support for the dissemination of such guidelines including the CONSORT (Consolidated Standards of Reporting Trials) for clinical trials, STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) for observational studies, PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) for systematic reviews, STARD (Standards for Reporting Diagnostic accuracy studies) for diagnostic or prognostic studies and ARRIVE (Animal Research: Reporting In Vivo Experiments) for animal studies, among others.6 Evidence has shown that application of guidelines is associated with improved standards of reporting, and looking for missing items from guidelines of submissions in the peer review process can enhance the quality of peer reviews and the finalised publications.7–10 Despite the usefulness of reporting guidelines, adherence to such guidelines in the biomedical research remains unsatisfactorily low.4 11 12

Beyond poor adherence to reporting guidelines, inconsistent or biased reporting between protocols or registrations and fully published articles has also raised significant concerns.13–16 For instance, one study comparing protocols and full reports in clinical trials found that approximately two-thirds of full reports had at least planned primary outcome modified, introduced or omitted.17 Similarly, another study focusing on trials funded by Canadian Institutes of Health Research reported that 40% of the trials had a difference in primary outcomes between protocols and full reports.18 Furthermore, abstracts, as the generally most read and accessed section of a publication, were found to be distorted or overly-optimistic presentations of results than were shown in full reports.19 Discrepancy between abstracts and full reports deserves more intensive attention and stringent examination in biomedical research because (1) abstracts are usually prepared with the least care; (2) readers draw conclusions about a study mainly depending on abstracts and (3) audiences may make their decisions only based on abstracts especially when full reports are not accessible.4 19 20

Even though there is increasing evidence on incomplete and inconsistent reporting in different fields of biomedicine and for different guidelines, there is no overarching summary of the evidence with regards to (1) the state-of-the-art extent of adherence to the emerging reporting guidelines in primary biomedical research, (2) the inconsistency between protocols or registrations and full reports or (3) the disagreement between abstracts and full-text articles. Therefore, we aim to conduct a scoping review to explore the current state of incomplete and inconsistent reporting in primary biomedical research and to investigate factors associated with improved completeness and consistency of reporting, based on evidence from systematic reviews and surveys. While the existing systematic reviews and surveys generally evaluate a specific research area or a group of journals or diseases with quantitative syntheses conducted, our scoping review will differ from them in mapping literature and addressing the state of reporting in the overall primary biomedical community, comprehensively summarising the heterogeneous evidence with a qualitative description reported, and assessing evidence gaps and providing recommendations for future research.21 22

Methods

In this scoping review, we will use a systematic and comprehensive approach to retrieve all available and eligible systematic reviews and surveys in the literature.23 Our study will be conducted and reported based on the PRISMA guideline.24 However, no risk-of-bias assessment in individual studies or quantitative synthesis will be performed because they are not relevant to this scoping review.

Our results will be presented in three parts including (1) current adherence to reporting guidelines; (2) inconsistency between protocols or registrations and full reports and (3) discrepancy between abstracts and full reports. The outline of this scoping review is shown in figure 1. We also provide a summary table for these three parts (table 1). For the first part, we will build on previous work on adherence to reporting guidelines which was limited to six guidelines for human studies and up to 2012.11 Our previous work will be expanded, updated and included in this scoping review.

Figure 1

Flow diagram showing the outline of this scoping review.

Table 1

Summary of key factors for the three parts (guideline adherence, inconsistency between protocols/registrations and full reports and inconsistency between abstracts and full reports) included in the scoping review

Key factor	Guideline adherence	Inconsistency between protocols or registrations and full reports	Inconsistency between abstracts and full reports
Primary objective	Current state of reporting in primary biomedical research
Secondary objective	Factor associated with improved completeness or consistency of reporting
Outcome	Level of guideline adherence	Level of (in)consistent reporting
Comparator reference	Reporting guidelines	Protocols or registrations	Full reports
Main data collected	Adherence to the items listed in guidelines	Inconsistent reporting on study-validity-related factors*
Data analysis	Qualitative description summarised

*Study-validity-related factors including research questions, study designs, study populations or sample sizes, interventions or exposures, time duration, comparators, statistical plan, result presentations and interpretations and conclusions or recommendations.

Study eligibility

Systematic reviews that include primary studies and evaluate incomplete or inconsistent reporting with a focus on adherence to guidelines, comparison between protocols or registrations and full reports or consistency between abstracts and full reports, will be eligible. For the purposes of this review, an eligible systematic review will be defined as study with predetermined objectives, eligibility criteria, at least one electronic database searched, data extraction and at least one study included. All the surveys that include primary studies and focus on specific research questions in primary biomedical research will be eligible for inclusion in this scoping review.

Adherence to reporting guidelines:

We will include systematic review and surveys of the following guidelines: CONSORT, PRISMA, STROBE, STARD, ARRIVE, QUOROM (Quality of Reporting of Meta-analysis), TREND (Transparent Reporting of Evaluations with Non-randomized Designs), MOOSE (Meta-analysis Of Observational Studies in Epidemiology), CARE (Case Report), SRQR (Standards for Reporting Qualitative Research), COREQ (Consolidated criteria for Reporting Qualitative research), TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis), SQUIRE (Standards for QUality Improvement Reporting Excellence), CHEERES (Consolidated Health Economic Evaluation Reporting Standards), SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) and REMARK (Reporting Recommendations for Tumour Marker Prognostic Studies). Systematic reviews that do not evaluate adherence to any of the aforementioned guidelines will not be included in our study.

Consistency between protocols/registration and full reports:

Systematic reviews or surveys from all fields of biomedical research will be eligible if they included a study objective of comparing protocols or registrations with full reports and provided data on such comparison.

Agreement between abstracts and full reports:

Systematic reviews or surveys from all fields of biomedical research will be eligible if they included a study objective of comparing abstracts with full reports and provided data on such comparison.

Furthermore, to expand the extent of this scoping review, we will also include systematic reviews or surveys that specifically investigated the incomplete or inconsistent reporting for study subgroups for all the three parts above (figure 1).

Exclusion criteria

For all the three parts, the systematic reviews or surveys will be excluded if (a) their objectives are not incomplete or inconsistent reporting, (b) they do not focus on primary biomedical research studies, (c) they only publish editorials, abstracts, letters or commentaries without full-length texts, (d) they are duplicates of the included systematic reviews or surveys, or (e) they do not provide data on incomplete or inconsistent reporting.

Search strategy

We will search the electronic databases including Web of Science, Excerpta Medica Database (EMBASE), MEDLINE and Cumulative Index to Nursing and Allied Health Literature (CINAHL), for relevant studies. The search will be limited between January 1996 and 30 September 2016 given that the CONSORT (Consolidated Standards of Reporting Trials) statement was the first reporting guideline in biomedical research and developed in 1996.25 The search strategy will be designed with the assistance of an experienced librarian. Key descriptors that include terms for systematic reviews or surveys, reporting, and guidelines or adherence or inconsistency or registrations or protocols or abstracts will be used for the search, for instance, (Systematic reviews OR surveys OR reviews) AND (quality of reporting OR completeness of reporting OR selective reporting OR consistency of reporting OR biased reporting OR subgroup) AND ((QUOROM OR TREND OR MOOSE OR CONSORT OR STROBE OR PRISMA OR CARE OR SRQR OR COREQ OR STARD OR TRIPOD OR SQUIRE OR CHEERES OR ARRIVE OR SPIRIT OR REMARK) OR (Adherence OR Consistency OR Protocol OR Registration OR Abstract)). Online supplemental table S1 shows the detailed search terms used in this scoping review.

Ovid search terms modified for MEDLINE, EMBASE, CINAHL and Web of Science (from January 1996 and September 30th 2016)

Study selection

Titles and abstracts retrieved will be first screened for eligibility before full texts are thoroughly examined. Reasons will be documented for excluded studies when assessing full texts. All the reference lists from the included systematic reviews or surveys will be also reviewed to retrieve additional relevant studies. We will limit the search to English language because of the lack of resources for translation of other languages. All the search processes will be performed by two reviewers (YJ and IN) independently. Disagreement will be addressed by consensus after discussion, and a third reviewer (GL) will be consulted if no consensus is reached. The kappa statistic will be used to quantify the level of agreement previous to their consensus between the two reviewers (YJ and IN).26

Outcomes

In this scoping review, our primary outcomes include levels of adherence to reporting guidelines, levels of consistency between protocols or registrations and full reports and the agreement between abstracts and full reports, all of which are expressed as percentages, quality scores or categorised rating (such as high, medium, low).

Specifically, for the first part, incomplete reporting will be assessed by the levels of adherence to reporting guidelines and their checklists when available, for example, for the CONSORT guideline, the percentage of adopting the guideline and the rates/scores of adhering to the components (title and abstract, introduction, methods, results, discussion and other information) among the included primary studies in the systematic review or survey will be our outcomes of interest. Levels of consistency between protocols or registrations and full reports and between abstracts and full reports will be evaluated by the agreement on the study-validity-related factors including research questions, study designs, study samples, interventions or exposures, outcome measures, time duration, comparators, statistical plan, result presentations and interpretations and conclusions or recommendations. For instance, some studies may investigate the changes in the study-validity-related factors from the prespecified protocols that are identified in full reports; the percentages of such changes will be our outcomes collected.

Our secondary outcomes are the factors associated with improved completeness and consistency of reporting as reported from the included systematic reviews and surveys.

Data collection

Two reviewers (YJ and IN) will independently collect data from the included systematic reviews or surveys using data extraction forms. The data extraction forms will be piloted and modified before its final version to be used. Specifically, we will extract the data as shown below:

basic characteristics: authors, publication year, journal in which the study is published, field of study, study region, number of primary studies included, number of study samples (including animals and participants) and reporting guideline (or its extension or modification) assessed in the systematic review or survey;

for the adherence to guidelines, we will gather the reported adherence to the items specified in the corresponding guideline; for the consistency between protocols or registrations and full reports, and the agreement between abstracts and full reports, data extracted include (dis)concordance for research question, study population or sample size, intervention (or exposure), comparator, outcome, time duration, study design, statistical plan, result presentations and interpretations, conclusion and other information specifically evaluated in the systematic review or survey;

outcome measures presented as levels of adherence to reporting guidelines or levels of consistency will be collected for all the relevant items if provided;

factors that are found to be related to improved completeness and consistency of reporting in the individual systematic reviews or survey;

authors' overall conclusion in the systematic review or survey.

Any disagreement will be resolved by the two reviewers' discussion and consensus. In addition, we will contact the authors of included systematic reviews to collect essential and relevant data if necessary.

Data analysis

The levels of adherence to guidelines and the levels of consistency will be described using medians and IQRs across all the included studies.

The general characteristics of included studies, levels of adherence to reporting guidelines or levels of consistency between protocols or registrations and full reports and between abstracts and full reports, factors related to improved completeness and consistency of reporting and conclusions in the included studies will be summarised and discussed in our review. No pooled analyses or quantitative syntheses will be performed given the heterogeneity of the included systematic reviews and surveys. Likewise, factors associated with improved completeness and consistency of reporting will be summarised qualitatively.

Quality assessment of included studies

We will evaluate the quality of all the included systematic reviews, using the AMSTAR (a measurement tool to assess systematic reviews) criteria.27 The R(evised)-AMSTAR will not be used in our study, given its limited application and unknown measurement properties.28 However, some items of AMSTAR may not be applicable to all the included systematic reviews. For instance, the item 9 ‘were the methods used to combine the findings of studies appropriate’ (because not all the systematic reviews used a pooled estimate) is not relevant to some included studies, thereby being omitted from the quality evaluation. Likewise, we will not assess quality of the included surveys due to lack of relevant assessment tools or guidelines.

Discussion

Incomplete or inconsistent reporting remains a major concern in the biomedical literature including preclinical studies, diagnostic research, qualitative studies, economic studies, clinical trials and observational studies, among others.2 4 When the reporting is incomplete or inconsistent, the apparent methodological quality of published findings may not reveal the actual quality of the study as evaluated from the protocol or registration or abstracts, yielding the published findings unreliable, irreproducible or sometimes misleading.3 4 17 29 In this scoping review, we will assess the completeness of reporting in the literature and adherence to reporting guidelines, consistency between protocols or registrations and full reports and agreement between abstracts and full-text articles. We will present our results as three parts, where the first part of adherence to reporting guidelines is an updated and expanded research based on our previous work.11 In contrast, for the other parts of inconsistency between protocols or registrations and full reports and discrepancy between abstracts and full reports, no study summarising all the best current evidence in multidisciplines is available. Unlike the individual systematic review and survey that reports confirmatory point estimates in a specific area or disease or in a group of journals,13 16 17 30 our scoping review will show the general mapping for the state of reporting in the overall primary biomedical research. With the evidence gaps explored in this scoping review, findings may advance our understanding of the extent of incomplete and inconsistent reporting, factors related to improved completeness and consistency of reporting and potential recommendations for various stakeholders in the biomedical community. All findings will be published in peer-reviewed journals electronically and in print.