Quality of reporting for randomised clinical trials published in Latin American and Spanish journals: A protocol for a systematic survey of three clinical specialities.

INTRODUCTION: Quality of reporting refers to how published articles communicate how the research was done and what was found. Gaps and imprecisions of reporting hamper the assessment of the methodological quality and internal and external validity. The CONsolidated Standards of Reporting Trials (CONSORT) are a set of evidence-based recommendations of the minimum elements to be included in the reporting of randomised controlled trials (RCTs) to ensure a complete and transparent account of what was done, how it was done and what was found. Few studies have been conducted on the impact of CONSORT on RCTs published in Latin American and Spanish journals. We aim to assess the reporting quality of RCTs of three clinical specialities published in Spanish and Latin American journals, as well as to assess changes over time and associations of quality with journal and country indicators. METHODS AND ANALYSIS: We will conduct a systematic survey of all RCTs published in Spanish-language journals in three clinical fields (dentistry, neurology and geriatrics) from 1990 to 2018. We will include RCTs from previous work that has identified all RCTs on these medical fields published in Spain and Latin America. We will update this work via handsearching of relevant journals. Assessment of quality of reporting will be conducted independently and in duplicate using the CONSORT 2010 Statement. We will also extract journal and country indicators. We will conduct descriptive statistics and secondary analyses considering the year, country, and journal of publication, among others. ETHICS AND DISSEMINATION: The Universidad de Santiago de Chile's ethics committee approved the protocol. We will disseminate the results of this work in peer-reviewed scientific journals and conference proceedings. We expect to raise awareness among researchers, journal editors and funders on the importance of training in reporting guidelines and using them from the inception of RCT protocols. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

We will build on and update previous work that has comprehensively identified randomised controlled trials (RCTs) on dentistry, neurology and geriatrics published in Spain and Latin America.

With the full set of RCTs, we will be able to assess the quality of RCT reporting.

We will extract data following a set of items derived from the CONsolidated Standards of Reporting Trials (CONSORT) 2010 checklist.

Assessment with the CONSORT tool can be reviewer dependent.

A robust quality control system is needed to resolve differences between reviewers regarding CONSORT compliance.

Introduction

Each day, healthcare professionals are faced with clinical questions that call into play their decision-making skills regarding, for example, the best treatment or management strategy for patients. To make better clinical decisions, they must search for the best available evidence and undertake a critical analysis of what they find while also taking into account patients’ preferences and values.1 In this context, randomised controlled trials (RCTs) provide the best evidence regarding the efficacy of interventions and are the primary input for systematic reviews, which, in turn, are used to inform clinical practice guidelines or health policies. As RCTs are used for clinical decision-making processes, they must be reported with the highest standards regarding design, conduct, analysis and presentation of results.2 As such, there is a growing interest in determining the quality of RCTs, which involves mainly two dimensions: methodological quality and quality of reporting.

Methodological quality refers to designing RCTs to minimise bias from the inception of the study. Biases in RCTs are systematic deviations of the results or the inferences made from an RCT,3 4 resulting in underestimation or overestimation of the true effect of the interventions.5 Many scales, systems and classifications have been developed to assess the risk of bias of RCTs. One of the most widely used tools for this purpose is the Cochrane tool for assessing risk of bias in randomised trials, which was developed in 2005 and covered the main sources of bias in clinical trials: selection bias, performance bias, detection bias, attrition bias, reporting bias and other bias.6 A new update has recently been released.7

Nevertheless, methodological quality cannot be separated in importance from reporting quality. Quality of reporting refers to how published articles communicate how the research was done and what was found. Gaps and imprecisions of reporting hamper the assessment of the methodological quality and internal and external validity.8 Reporting guidelines were developed to address this problem by improving transparency, accuracy and completeness of the health research literature.2 The first CONsolidated Standards of Reporting Trials (CONSORT) Statement was published in 1996 to improve the quality of reporting of RCTs. The most current CONSORT version, released in 2010,9 has been translated into several languages, including Spanish.10

The CONSORT reporting guideline is a set of evidence-based recommendations of the minimum elements that should be included in the reporting of RCTs to ensure a complete and transparent account of what was done, how it was done and what was found. It consists of a 25-item checklist distributed in six domains: title and abstract, introduction, methods, results, discussion and other information.11

After the first statement was released, there was a gradual uptake of endorsement in editorial policies of high-impact medical journals,12 13 which subsequently expanded to over 500 journals.14 While endorsement helps, it may not be enough if journals do not enforce its use during peer review. Secondary research on the impact of the CONSORT guideline has found evidence that its adoption improves the quality of reporting of RCTs.15–17

Methodological quality has also improved during the past years. A study that assessed the risk of bias of over 20 000 RCTs included in Cochrane reviews published between 2011 and 2014 concluded that inadequate methods (resulting in bias) and poor reporting (precluding proper assessment of the risk of bias) has decreased over time, especially in high-impact journals.18 According to this study, in journals without an impact factor—of which 52% are not in English—this problem is pervasive and has not shown a decreasing trend.

Few studies have been conducted on the impact of reporting guidelines on RCTs published in Latin American and Spanish journals. A descriptive study that evaluated compliance with RCT registration and use of reporting guidelines in a random sample of 101 RCTs published in 56 journals from Latin America and the Caribbean found that only 3% acknowledged adherence to the CONSORT statement.19 Another study that systematically hand searched RCTs published in Spanish-language dermatology journals between 2007 and 2012 concluded that most of the identified articles had a high risk of bias mainly due to gaps in reporting on important methods items.20 A third study on RCTs of assisted reproductive techniques published in Spanish-language obstetrics and gynaecology journals reported similar findings.21

Recognising the need to gather the whole body of published RCTs in Spanish-language journals, the Iberoamerican Cochrane Network embarked on an effort to identify all biomedical journals published in Spain and Latin America. They found 1498 biomedical journals that published original research, most of which did not have an impact factor (97%), and only 4.1% were indexed in MEDLINE.22 Hence, the retrieval of these clinical trials using electronic searches in databases is not feasible, potentially limiting their inclusion in systematic reviews, possibly resulting in an avoidable waste of research evidence.23 These considerations led to the development of an ongoing project called BADERI (Database of Iberoamerican Clinical Trials and Journals, by its initials in Spanish), officially launched in October 2015, to identify through hand searching all the RCTs published in these journals.24 Once identified, references to the RCTs are included in CENTRAL (Cochrane Collaboration Central Register of Controlled Trials), the Cochrane RCT global repository, thus giving visibility to this evidence. Handsearching of RCTs in obstetrics and gynaecology, dermatology, physiotherapy, dentistry, orthopaedics and ophthalmology have been completed,21 22 25–27 with several others underway. We will follow up on this previous work in agreement with Cochrane Iberoamérica and include in our research project any clinical specialities that are already underway and where resources were lacking to finish the task.

Contrasting with the many studies assessing methodological or reporting quality of RCTs indexed in MEDLINE,17 18 28 only a few studies assessed the quality and compliance with reporting guidelines of RCTs conducted in Latin America and the Caribbean but published elsewhere.19 29 A more comprehensive understanding of the status of RCT-conducting as they are published in the journals of Spain and Latin America will help editors and funders undertake actions to strengthen the methodological competencies of researchers based in these countries. The systematic assessment of reporting quality we aim to undertake will contribute to this endeavour.

Objective

The primary aim of our study is to examine the reporting quality of RCTs of three clinical specialities and areas (dentistry, neurology and geriatrics) published in Spanish and Latin American journals. A secondary aim will be to assess changes over time and associations of quality with journal and country indicators.

By clinical field (dentistry, neurology and geriatrics), to characterise RCTs published in Spanish-language journals of Spain and Latin America, included in BADERI.

By clinical field, to determine compliance with the CONSORT guideline.

By clinical field, to establish associations between the quality of reporting and the year of the introduction of the latest CONSORT statement (2010), controlling by country variables, journal metrics and year of publication.

Methods

Study design

We will conduct a systematic survey of all RCTs published in Spanish-language journals in three clinical fields (dentistry, neurology and geriatrics) from 1990 to 2018.

Data source

We will use the RCTs identified via hand searching of Spanish and Latin American journals of dentistry,26 neurology30 and geriatrics31 32 as registered in the BADERI database.

We will update these studies by hand searching the most recently published issues of the journals included in these studies. The methods for handsearching are published elsewhere (21, 24–27). We will also search for new journals in these medical fields that could have emerged after 2015, following a previously published methodology.24

Population

We will study the whole population of RCTs for each clinical field, as provided by BADERI, meaning that no sample size calculation will be needed.

Eligibility criteria

We will include randomised or quasi-randomised clinical trials with a recoverable full text published between 1990 and 2018. We will exclude articles not addressing the clinical specialities of interest. We will also exclude articles that report conference proceedings, are pilots or feasibility studies, conduct secondary analysis on RCTs, are translations of RCTs published elsewhere, letters and editorials.

Data extraction

We will extract data on 23 CONSORT items plus four additional items (see table 1), and journal variables such as journal name, country of publication (the country where the journal that published the paper is located), impact factor (if applicable). We will also obtain country socioeconomic data such as per capita gross domestic product (GDP), per cent of GDP invested in research and development, the number of journals indexed in the Web of Science. We will use the World Bank and the Web of Science Journal Citation Report as data sources for these variables.

Table 1

The instrument used to extract data on consort variables (items)

CONSORT item	#	Description and definition	Scoring
Title	1a	Identification as a randomised trial in the title	1=word random appears in the title0=no word ‘random’ appears
Abstract	1b	Structured summary of trial design, methods, results, and conclusions (for specific guidance see CONSORT for abstracts)	1=structured abstract0=no structured abstract
Trial design	3a	Description of trial design (such as parallel, factorial) including allocation ratio (ex-split-mouth)	1=well-described design0=not well-described design
Participants	4a	Eligibility criteria for participants	1=specified0=not specified
	4b	Settings (A) and locations (B) where the data were collected	2=A and B1=A or B0=not specified
Interventions	5	The interventions for each group (A) with sufficient details to allow replication, including how and when they were actually administered (B) (ie, ‘usual care’ for control group not enough)]	2=A and B1=A, but only one group with details0=only A or only one group without details
Outcomes	6a	Completely defined prespecified primary (A) and secondary outcome (B) measures, including how and when they were assessed (C)	2=A and B and C1=A or B (no distinction)+C0=A or B, no C
Sample size	7a	How sample size was determined	1=specified0=not specified
Sequence generation	8a	The method used to generate the random allocation sequence	1=specified0=not specified
Allocation concealment mechanism	9	The mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned	1=steps for concealment specified0=concealment not specified
Implementation	10	Who generated the random allocation sequence (A), who enrolled participants (B) and who assigned participants to interventions (C)	2=A and B and C1= (A and B) or (A and C)0=A missing
Blinding	11a	If done, who was blinded (A) after assignment to interventions (eg, participants, care providers, those assessing outcomes) and how (B)	2= (A and B) OR reason why the study is open-label1=declares who is blinded but no details as how0=declares the study blind but no who nor how
Statistical methods	12a	Statistical methods used to compare groups for primary and secondary outcomes	2=states full stats method for each outcome1=states stats methods for primary outcome0=states stats methods vaguely
Participant flow (a diagram is strongly recommended)	13a	For each group, the numbers of participants who were randomly assigned (A), received intended treatment (B) and were analysed for the primary outcome (C)	2=A and B and C (narrative in text OR complete flow diagram)1=A or B or C missing (only one missing)0=only one reported or no info at all
	13b	For each group, losses (A) and exclusions (B) after randomisation, together with reasons (C)	2=A and B and C1=A or B or C missing (only one missing)0=only one reported, or no info at all
Recruitment	14a	Dates defining the periods of recruitment (A) and follow-up (B)	2=A and B reported1=A or B reported0=none reported
Baseline data	15	A table showing baseline demographic and clinical characteristics for each group	1= ‘table 1’ present0= ‘table 1’ not present
Numbers analysed	16	For each group, number of participants (denominator) included in each analysis (A) and whether the analysis was by original assigned groups (B) (ITT or per protocol)	2=A and B1=A or B0=not stated
Outcomes and estimation	17a	For each primary and secondary outcome, results for each group (A), and the estimated effect size (B) and the precision (CI) (C) (only for primary outcome)	2=A and B and C1=A or B or C missing (only one missing)0=only one reported or none
Harms	19	All important harms or unintended effects in each group	1=harms described0=harms not described
Registration	23	Registration number and name of trial registry	1=present0=absent
Protocol	24	Where the full trial protocol can be accessed, if available	1=present0=absent
Funding	25	Sources of funding and other support (such as the supply of drugs)	1=present0=absent
Additional item	AIa	Total number of patients randomised	Annotate sample size
Additional item	AIb	Conflict of interest statement	1=present0=absent
Additional item	AIc	Ethics review	1=present0=absent
Additional item	AId	Language of article	Spanish/English/Portuguese

CONSORT, CONsolidated Standards of Reporting Trials; ITT, Intention to treat.

Each item may be measured as a binary outcome (reports yes/reports no) or with three ordinal categories (full reporting, partial reporting, no reporting). We will exclude any item regarding sections Introduction and Discussion because of the inherent subjectivity in the appraisal of these items (items 2a, 2b, 20, 21 and 22). To avoid issues of transforming total scores from non-applicability of items, we will disregard items 3b, 4b, 6b, 7b, 8b, 11b, 12b, 14b, 17b, and 18. The resulting score will range from 0 to 34 on the CONSORT items. We will also extract further data on the total number of patients recruited, conflict of interest statement (present/absent), ethics review (present/absent), and language of publication (Spanish, English, Portuguese), and year and country of publication.

Different research teams will extract data for the three clinical fields. The senior investigators will be responsible for training, calibration and resolution of discrepancies during the data extraction and analysis process. Teams will be composed of medical students in different years of undergraduate and graduate training.

Training and calibration will consist of passing the data extraction sheet with the predefined CONSORT 2010 items to a small sample of RCTs. Each data extractor will be trained in the data extraction, and a pilot run-in will be done to fine-tune the process. During the pilot period, each data extractor will be asked to review from five to eight papers. Agreement measures will be calculated, and when there are 10% or fewer items with discrepancies, the pilot will be concluded. If over 10% discrepancy persists, the pilot will be extended with further calibration and training. This process will be reiterated with each group of data extractors for each clinical field. We will use the CONSORT explanatory paper as a guide.33

Two reviewers will extract for each article independently and in parallel. Responses will be submitted to a data analyser who will record agreement/disagreement and input the responses into the data processing software. Discrepancies will be communicated to each pair of data extractors who will discuss the findings and reach a consensus; if no consensus is possible, the senior statistical investigator will act as referee.

Additionally, the senior reviewers will randomly select 10% of the articles with no discrepancies to verify agreement with the scoring system defined for this study. This quality control process will be conducted in predefined periods during the data extraction phase to ensure a minimum of errors in the data collection of CONSORT adherence.

Outcomes

The primary outcome will be the total score of the 23 predefined per protocol CONSORT 2010 items for each RCT (score range 0–34) and will assess the quality of reporting for each article in overall terms. Comparisons over time (before CONSORT 2010 and after CONSORT 2010) will be made using this overall score.

The secondary outcome will be the total score for each one of the 23 predefined per protocol CONSORT 2010, that is, the frequency of compliance per item across articles. For this purpose, those items that were categorised at three levels on the total sum score (primary outcome) will be transformed into binary variables thus facilitating the comparison between items. Three-level items (grade 2, 1 or 0) will be transformed according to the following rule: grade 2 will be collapsed into grade 1, grade 1 will be collapsed into grade 0, grade 0 will remain unchanged. This will result in a downgrade of partial adherence to non-adherence. This outcome will allow us to describe which items are reported poorly and if there is improvement over time regarding specific items. Additional per protocol items will also be included in the final analysis.

Statistical analysis

Since we are including the whole population of RCTs for each clinical field, no statistical inference techniques will be performed. A descriptive analysis will be done on the RCTs selected for this study. Summary statistics, figures and tables will be provided.

To assess the impact of the 2010 CONSORT guideline, a comparison (absolute difference of scores) will be computed between average scores observed on articles published before 2010 and articles published from 2011 onwards.

A secondary analysis will describe each of the 23 CONSORT items included in this study to determine compliance and to explore the items that most contribute to non-compliance. Charts with percentages of studies complying with each item will be provided, by year and by periods (pre-CONSORT and post-CONSORT 2010). We will explore changes for the primary and secondary outcomes on the quality of RCTs according to sample size, country of publication, funding, and language, and according to whether RCTs were published in Spanish journals or Latin American journals.

Statistical analysis will be performed with the R package statistical software (R Foundation for Statistical Computing, Vienna, Austria; 2019).

Ethics and dissemination

This protocol was approved by the Institutional Ethics Committee of Universidad de Santiago de Chile, Ethics Report No 524, dated 10 August 2018.

The results of this work will be disseminated in peer-reviewed scientific journals and conference proceedings. We expect to raise awareness among researchers, journal editors and funders on the importance of training in reporting guidelines and using them from the inception of RCT protocols.

Patient and public involvement

No patients or public were involved in the development of this protocol. However, we expect our results will positively impact patient and public interest by providing information on the reporting of RCTs.

Discussion

CONSORT and other reporting guidelines are widely used in clinical and epidemiological research. While there is plentiful research on how these guidelines have impacted research published in the English language34–43 not enough attention has been paid to compliance with these guidelines of RCTs published in Spanish. We hypothesise that compliance with CONSORT in Spain and, especially, in Latin America, lags that observed in English-speaking countries and of journals indexed in MEDLINE, as shown in previous research.18 This important gap in reporting quality affects the overall impact of research generated from Latin America. While it is unclear the extent to which reporting quality may be used as a proxy for methodological quality, we will be able to contrast our results against previous work that has looked into methodological quality for geriatrics.31 32

Because of the nature of the population of journals that we selected for our study, most of which are included only in databases with incomplete information, or available only in print version in university libraries, it is not possible to retrospectively know which journals endorsed and enforced CONSORT by year. Conversely, we will be able to detect any improvement in RCTs reporting but whether this stems from researchers’ initiative or due to journal requirements, will not be known from our study.

We recognise that our study is planning a substantial reduction in the number of CONSORT items to be checked for compliance and the addition of a few others, which may make it more difficult to compare with previously published studies also assessing CONSORT compliance of RCTs. That said, in our literature search we did not find studies that used the same score scale or same number of items. Wilson et al, for example, used a truncated version of the CONSORT checklist giving priority to the methodological items,39 as did Hopewell et al.44 Loguercio et al used a third category (‘item fully present’, ‘item partially present’ and ‘item absent’) in their assessment of CONSORT compliance.45 In a nutshell, there is no core CONSORT compliance extraction form short of using the CONSORT checklist without any modifications, but in our view this would diminish the full appraisal of reporting quality as we aim to assess in our study.

We will publish the results of this three-staged study in peer-reviewed biomedical journals of each one of the three clinical fields that will be explored, preferentially in journals publishing in Spanish. We aim to raise awareness among researchers, journal editors, and funders on the importance of training in reporting guidelines and using them from the inception of the study protocol.