Do surgery journals insist on reporting by CONSORT and PRISMA? A follow-up survey of 'instructions to authors'.

AIMS: Guidance has been published on how best to report randomised controlled trials (Consolidated Standards of Reporting Trials - CONSORT) and systematic reviews (Preferred Reporting Items for Systematic Reviews and Meta-analysis - PRISMA). In 2011, we reported a low rate of enforcement by surgery journals for submitted manuscripts to conform to these guidelines. The aim of this follow-up study is to establish whether there has been any improvement.
METHODS: We studied the 134 surgery journals indexed in the Journal Citation Report. The 'Instructions to Authors' were scrutinised for inclusion of the following guidance: CONSORT, PRISMA, clinical trial registration and systematic review registration.
RESULTS: Compared to 2011, there has been an improvement in the endorsement of reporting guidance in journals' 'Instructions to Authors' in 2014, as follows: trial registration (42% vs 33%), CONSORT (42% vs 30%) and PRISMA (19% vs 10%, all p < 0.001). As in 2011, journals with a higher impact were more likely to adopt trial registration (p < 0.001), CONSORT (p < 0.001) and PRISMA (p = 0.002). Journals with editorial offices in the UK were more likely to endorse guidance compared to those outside the UK (p < 0.05). Only one journal mentioned registration for systematic reviews.
CONCLUSIONS: Surgery journals are presently more likely to require submitted manuscripts to follow published reporting guidance compared to three years ago. However, overall concordance rates are still low, and an improvement is required to help enhance the quality of reporting - and ultimately the conduct - of randomised control trials and systematic reviews in surgery.

Introduction

Reporting guidelines exist to encourage transparency and improve the standards of reporting in evidence-based medicine. Universal implementation of such guidelines would go some way towards addressing the suboptimal reporting of randomised controlled trials (RCTs), systemic reviews and meta-analyses in surgery.

Randomised controlled trials (RCTs) are an indispensable tool for high quality assessment of causality and, aside from systematic reviews, remains the gold standard for assessing treatment effects. Trial registries exist to ensure clinical transparency, prevent the duplication of research and to enable regulation. The failure to register RCTs can be regarded as a breach of scientific and ethical conduct [1]. The registration of clinical trials is now essential in some countries, it is strongly recommended by most official bodies and should be included in published reports [1]. In 2010, to address problems of inadequacy in the objective evaluation of RCTs, the Consolidated Standards of Reporting Trials (CONSORT) statement was revised [2-4]. This 25-item checklist and flow diagram facilitates good reporting and manuscript development in interventional studies [3].

In 1999, the Quality of Reporting of Meta-analyses (QUOROM) statement was published. This included a 27-item checklist to encourage good reporting of systematic reviews [5]. QUOROM later gained support from many surgery and medicine journal editors including the Cochrane Collaboration [6]. A decade later, the QUOROM Statement was revised to incorporate recent methodological advances, and renamed PRISMA (Preferred Reporting Items of Systematic reviews and Meta-Analyses) [7]. In 2011, a prospective registry of systematic reviews in health and social care – PROSPERO – was created by the Centre for Reviews and Dissemination (CRD) [8]. This is an international database of prospectively-registered systematic reviews to help address problems with transparency, reporting bias and the duplication of reports. It is hoped that the complementary use of PRISMA and PROSPERO in the conduct and publication of systematic reviews and meta-analyses will ensure good reporting in the medical literature.

It is believed that enforcement of these guidelines will lead to an improvement in the quality of reporting, and thus the conduct, of RCTs and systematic reviews in medical research [9]. In 2011, we looked at the “Instructions to Authors” of a group of journals dedicated to surgical research, to see how frequently trial registration and adherence to guidelines was required [9]. We found that most journals did not mention, let alone insist upon, this guidance. The purpose of the current study is to ascertain whether surgery journals more frequently require manuscript submissions to adhere to these guidelines, compared to three years ago.

Methods

In our previous report, we studied all journals listed under “Surgery” in the science edition of the 2009 Journal Citation Reports (Thomson Reuters, New York, US). We used a similar approach in this report, (http://apps.webofknowledge.com, last accessed 14th July, 2014). We only included journals which actively publish original research. In 2011, this resulted in a list of 136 surgery journals. However, of these, two no longer exist (one discontinued, the other incorporated into another publication), so 134 journals were available for follow-up analysis (Fig. 1, below).

Fig. 1

List of journals included in the study.

An internet-based investigation was performed by accessing the online instructions for authors for each surgery journal. Information regarding a journal's enforcement of PRISMA, CONSORT and trial registration was entered into a custom-designed spreadsheet (Excel 2010, Microsoft, Redmond) in March 2011, and then repeated in July 2014. PROSPERO did not exist when the original study was conducted, so a paired analysis could not be undertaken in this case. The journal's impact factor was recorded. Each journal was assigned a geographical zone based upon the location of their editorial office, and this was categorised into four distinct groups: UK, Europe (excluding the UK), North America (US and Canada) and other (the remaining countries).

For each of the guidelines mentioned above, and for each journal, the level of enforcement was documented as compulsory (an essential criteria for manuscript acceptance), recommended (usage encouraged, but not mandatory) or unclear (guidelines were mentioned but the necessity of their inclusion in the manuscript remained ambiguous). Data were collected independently by two authors (T.S., P.K.), with Yes/No discrepancies noted in 0.7% of items. This equated to a good intra-observer reliability (Cohen's kappa, κ = 0.992). All inconsistencies were reviewed by a third author (S.S.) and resolved after subsequent discussion with all authors.

Impact factor (IF) was analysed as a continuous variable, and compared to reporting frequency using logistic regression [10]. The associations between geographical region and frequency of guideline recommendations by journals were calculated using Pearson's chi-squared test (with three degrees of freedom), with subgroup analyses calculated using post hoc z-tests for proportions. To look for evidence of an improvement between the years 2011 and 2014 of paired frequencies, McNemar's test was used. All statistical analyses were performed using SPSS (v18, IBM, Chicago).

Results

As previously mentioned, 136 surgery journals were included in our original study in 2011 [9]. In our most recent search, two journals were found to no longer be in existence, therefore we were unable to provide follow-up data and they were excluded from our analysis. This left 134 journals for analysis with a range of impact factor from 0.1 to 6.3.

The overall distribution of data with regards to impact factor and geographical location is represented in Tables 1 and 2. Impact factors were similar in the 2011 and 2014 analysis showing that 65% and 62% of journals had an impact factor of ≤2, respectively. As noted in our previous report, the majority (55%) of surgery journals' editorial offices were located in Northern America.

Currently, guidelines for CONSORT are endorsed by 56 (42%) of surgery journals. Of these, this was an absolute requirement of 44 (33%) journals (Fig. 2, below), and most journals requiring CONSORT provided a web reference (91%). Journals with a higher impact factor are more likely to endorse the use of CONSORT (p < 0.001). A significant difference was found between geographical location and endorsement of CONSORT (p < 0.001), with the UK-based journals being the most likely to enforce this guidance (92%, p < 0.05). On the other hand, only 22% of European journals (excluding the UK) endorsed the use of CONSORT. Over the last three years, overall endorsement of CONSORT has improved by 40% (p < 0.001, Table 3).

Fig. 2

Percentage levels of endorsement of each guideline, given that the guideline has been mentioned in the instructions to authors. The labels above each bar indicate the absolute numbers. Systematic review registration has not been included, as only one journal mentioned this, and it was only recommended in this case.

Table 3

Change in the frequency of endorsement of guidelines between 2011 and 2014.

	2011	2014	p value
	(n = 134)	(n = 134)
CONSORT	40(30%)	56(42%)	p < 0.001
PRISMA	13(10%)	26(19%)	p < 0.001
Trial registration	44(33%)	56(42%)	p < 0.001

P values highlighted in bold are statistically significant.

Clinical trial registration was mentioned by 56 (42%) journals; of these it was an absolute requirement in 91%. Web links to trial registration databases were provided by most of these journals (77%). Journals with a higher impact factor were more likely to enforce trial registration (p < 0.001). No association between a journal's geographical location and enforcement of trial registration was identified (p = 0.067). In the last three years, there has been a 27% increase in the rate of enforcement of trial registration (p < 0.001).

We observed that PRISMA was enforced by 26 (19%) surgery journals, with 21 (81%) of these providing a web reference. Surprisingly, only one journal stated PRISMA as an absolute requirement for manuscript submission. Journals with a higher impact factor are more likely to encourage use of PRISMA (p = 0.002). Again, a significant difference in geographical location and endorsement of PRISMA has been observed (p = 0.002), with the best concordance found in UK-based journals (p < 0.05). Compared to our initial analysis in 2011, the overall endorsement of PRISMA has increased by 100% (p < 0.001). Only one journal (<1%) mentioned systematic review registration (by PROSPERO) in their instructions to authors; furthermore, this was a recommendation not a requirement. No correlation between impact factor or geographical location and endorsement of review registration was identified.

Discussion

In this study, we analysed the “Instructions for Authors” of indexed surgery journals to determine the current rate of endorsement of published guidelines for systematic reviews (PRIMSA, review registration) and RCTs (CONSORT, trial registration) in the surgery literature. In addition, we compared these results to our previous study performed in 2011 to see if there has been an improvement [9]. Overall, the endorsement of these guidelines has increased significantly since 2011, although rates of endorsement are still far below ideal. We found that journals with a higher impact factor were more likely to enforce reporting guidelines. Of the guidelines analysed, CONSORT and trial registration were most likely to be endorsed. During the study period, a significant increase in the endorsement of CONSORT, PRISMA and trial registration in journals' instructions to authors was noted (p < 0.001). Systematic review registration was the least widely-used guideline; this may in part be because it is the most recent addition of the guidelines studied.

There is growing body of evidence suggesting that manuscripts published in surgery fail to report RCTs according to the CONSORT statement [11-14]. In particular, studies show that articles comply on average with 49–55% of the CONSORT items [11-13]. Notably, these studies revealed substandard reporting for methods of randomisation, blinding, trial registration and funding. The specific inclusion of trial registration identifiers in published RCTs has also been scrutinised in the surgery literature. Lee et al. identified that only 10% of articles specified a trial registry number in published RCTs pertaining to trauma surgery [12]. In addition, this study also identified that journals with a higher impact factor are more likely to publish reports who follow CONSORT guidelines [12]. Furthermore, a study of high impact orthopaedic journals reporting systematic reviews has shown that on average only 68% of PRISMA statement items are included in their articles [15]. Together, these reports suggest that the application of guidelines dedicated to improving the quality of systematic reviews and randomised controlled trials in surgery is poor. Only with the support of journals can we hope for this suboptimal reporting to improve.

The issue of suboptimal endorsement in journals' editorial policies has been found in non-surgical specialities. For example, similar studies in paediatric and psychiatric journals found that endorsement of CONSORT occurred in 20% and 23% of cases, respectively [16,17], although this level was greater (37%) in higher-impact medical journals [18]. In a study of general medical journals, clinical trial registration was recommended by 51% (vs. 42% in this study) [19]. In line with our results, endorsement is more frequent in journals with a higher impact factor [16].

There are limitations to this study. Firstly, not all surgery journals are indexed in the Journal Citation Report and thus there is a risk of selection bias. Moreover, many high quality surgery articles are published in medical journals and these would not be included in our analysis. The 2014 Journal Citation Report now identifies 199 surgery journals in publication. However, these additional journals were not appraised because we had no previous data to compare them with. We do not know the impact of this exclusion from our study because it is not known if the age of a journal impacts its likeliness to adhere to reporting guidelines. There may be a feeling that generic reporting guidelines may not be entirely appropriate for use in surgical trials (for example, due to difficulties in blinding), and hence the lack of a journal's requirement for following guidelines could be an intended omission. However, this should not preclude a report conforming to the other appropriate items in the guidelines. Furthermore, “Instructions to Authors” guidelines may not accurately reflect the editorial policies of the journal, and these were not investigated. However, for purposes of transparency, we would recommend journals report consistent guidelines in their editorial policies and instructions for authors. Finally, despite its known weaknesses, our assessment of a journal's profile was made according to impact factor [20].

We report that guidelines intended to standardise and enhance the clarity of reporting in systematic reviews and randomised controlled trials are not widely enforced by surgery journals, although there has been some improvement over the last three years. It is vital that good publication practice is supported in the medicine and surgery literature to allow sound decision-making in healthcare. Overriding responsibility remains within the remit of the investigators to conduct high-quality research. However, this should be encouraged by journals and editors through their editorial and peer-review policies, and made explicit in their instructions to authors.

Conflict of interests

None.

Financial support

None.

Ethical approval

None.

Consent

None.

Author contribution

Tanya Smith: data collection, analysis, interpretation, drafting manuscript.

Priyantha Kulatilake: data collection, analysis, interpretation, drafting manuscript.

Lucy Brown: data collection, interpretation, drafting manuscript.

James Wigley: study concept and design, interpretation, critical revision of manuscript.

Waseem Hameed, study design, interpretation, critical revision of manuscript.

Saran Shantikumar, study concept and design, data collection, analysis, interpretation, critical revision of manuscript.

Guarantor

Saran Shantikumar.

Table 1

Association between impact factor and endorsement of guidelines, given as n (%).

	Impact factor				p value
	0–1	1–2	2–3	>3
	(n = 33, 25%)	(n = 50, 37%)	(n = 29, 22%)	(n = 22, 16%)
CONSORT	7(21%)	18(36%)	15(52%)	16(73%)	P < 0.001
PRISMA	3(9%)	8(16%)	7(24%)	9(41%)	p = 0.002
Trial registration	8(24%)	17(34%)	12(41%)	19(86%)	p < 0.001
Review registration	0(0%)	0(0%)	1(3%)	0(0%)	p = 0.748

P values highlighted in bold are statistically significant.

Table 2

Association between geographical location and endorsement of guidelines, given as n (%).

	Geographical location				p value
	North America	Europe (excl. UK)	UK	Other
	(n = 74, 55%)	(n = 36, 27%)	(n = 13, 10%)	(n = 11, 8%)
CONSORT	30(41%)	8(22%)	12(92%)	6(55%)	p < 0.001
PRISMA	17(23%)	3(8%)	6(46%)	1(9%)	p = 0.002
Trial registration	31(42%)	10(28%)	8(62%)	7(64%)	p = 0.067
Review registration	1(1%)	0(0%)	0(0%)	0(0%)	p = 0.845

P values highlighted in bold are statistically significant.