Poor adherence of randomised trials in surgery to CONSORT guidelines for non-pharmacological treatments (NPT): a cross-sectional study.

OBJECTIVE: To systematically assess adherence of randomised trials in surgery to Consolidated Standards of Reporting Trials (CONSORT) guidelines for non-pharmacological treatments (NPT). Surgical trials are considered more difficult to design and execute than pharmacological trials. Furthermore, the original CONSORT statement does not address some aspects that are vital to the transparent reporting of surgical trials. The CONSORT-NPT extension was designed to address these issues but adherence in medical and surgical journals has not been assessed.
DESIGN: Cross-sectional study. SAMPLE: We identified eight general medical and eight surgical journals, indexed in PubMed and published in 2011, with the highest impact factors in their respective categories. MAIN OUTCOMES: Adherence to CONSORT statement and CONSORT-NPT extension items.
RESULTS: We identified 54 surgical trials (22 published in medical journals and 32 in surgical journals). There were eight items for which there was less than 30% overall compliance (seven were specific to the CONSORT-NPT extension). These seven items are related to: a full description of the care providers, centres and blinding status in the abstract (n=7/54, 13%), eligibility criteria for centres performing the interventions (n=13/54, 24%), how adherence of care providers with the protocol was assessed or enhanced (n=7/54, 13%), how clustering by care providers or centres was addressed as it relates to sample size (n=3/54, 6%), how care providers were allocated to each group (n=9/54, 17%), how clustering by care providers or centres was addressed as it relates to statistical methods (n=2/54, 4%), a description of care providers (case volume, qualification, expertise, etc) and centres (volume) in each group (n=0/54, 0%).
CONCLUSIONS: Adherence of surgical trials to CONSORT-NPT extension items is much poorer than to the standard CONSORT statement. Adherence also appears to be superior in general medical journals compared with surgical journals. Raising awareness and conducting qualitative research to identify areas for specific intervention will be important going forward.

This study is the first to assess surgical trials reported in general medical and surgical journals for adherence to the Consolidated Standards of Reporting Trials (CONSORT)-non-pharmacological treatments extension.

However, the final cross-sectional sample was small with only 54 trials. This precluded a detailed statistical analysis.

Furthermore, the cross-sectional nature of the study precluded us from determining whether there had been improvement over time.

Introduction

Randomised controlled trials (RCTs) are designed to determine the association between efficacy of a treatment and clinical outcome. In this regard, they are considered as the gold standard of healthcare evidence and the resulting conclusions can significantly affect clinical practice.1 It is therefore imperative that trials are well designed and correctly executed. However, it is equally important that trials are fully and transparently reported to allow proper critical appraisal by the scientific community.

Key information is often missing from published trials2 3 and there may be a correlation between incomplete reporting and poor trial methodology.4–6 Such missing information can include items as crucial as sample size, details of randomisation, blinding and the choice of primary outcome. In response to this problem, the Consolidated Standards of Reporting Trials (CONSORT) statement was launched in 1996 and aimed to provide a checklist of essential items that authors should report when publishing their study.7 The CONSORT statement was updated in 2001 and recently in 2010 and is now endorsed by more than 600 leading medical journals.8 9 While the CONSORT statement has been credited with improving the reporting standards of RCTs,10 many recent studies have highlighted remaining deficiencies in medical11–14 and surgical literature.15–18

Surgical trials are often considered more difficult to design and execute than pharmacological trials.19 Furthermore, the original CONSORT statement does not address some aspects that are vital to the transparent reporting of surgical trials such as difficulty in blinding patients and outcome assessors, variation in surgical technique and experience of operators. In 2008, an extension to the CONSORT statement was published providing specific recommendations for the reporting of RCTs of non-pharmacological treatment (CONSORT-NPT).20 Examples of added items include specifying the eligibility criteria for centres performing the intervention and how care providers are allocated to each trial group.

The aim of this study was to analyse the quality of reporting of RCTs in surgery published in medical and surgical journals based on the reporting criteria included in the 2010 CONSORT statement and CONSORT-NPT extension.

Methods

Search strategy

We identified eight general medical and eight surgical journals with the highest ISI impact factors from the ‘Medicine, General and Internal’ and ‘Surgery’ categories, respectively, of the 2011 Journal Citation Reports provided by Thomson Reuters.21 All 16 journals (see online supplementary appendix S1) are indexed on PubMed and a search was then conducted to identify reports of RCTs published in these 16 journals. The search (see online supplementary appendix S2) combined the ‘Cochrane Highly Sensitive Search Strategy’ for identifying randomised trials22 with the publication year 2011 and the journal name (conducted in March 2012). Also, the terms ‘surgery OR surgical OR surgeon’ were added when searching eight general medical journals to restrict results to RCTs in surgery. The search was conducted independently for each journal. All titles and abstracts retrieved from the search were assessed for eligibility by the authors (MN, MM, DH, WT and FC) such that each record was reviewed independently by at least two authors. Studies in which it was not clear whether the inclusion criteria had been met were reviewed in full text and discrepancies were resolved by consensus. All journals included in our sample are published in English.

Inclusion and exclusion criteria

We defined a randomised trial as a prospective study assessing healthcare interventions in human participants who were randomly allocated to study groups. Studies were considered eligible for inclusion if they were: (1) reports of a randomised controlled trial, (2) published in 2011 (either print or online e-publication during 2011) and (3) the primary aim of the study was considered as an interventional therapy. For the purposes of this study, an interventional therapy was defined as a therapy involving (1) some element of invasion or trauma to the body and 2) the requirement for operator skill to achieve a successful requirement and with the exception of an intervention being used purely to deliver a pharmacological treatment (ie, catheter delivered drug; see online supplementary appendix S2). We excluded reports where (1) one of the trial arms did not contain an interventional therapy as defined above, (2) a drug was the primary intervention, even in a surgical population (eg, chemotherapy for ovarian cancer) or (3) the RCT had been previously published and the current report was merely a follow-up or subgroup analysis using the same cohort of patients.

Data extraction

We created a modified version of the CONSORT checklist which contained all of the 2008 CONSORT-NPT checklist items and all of the standard 2010 CONSORT checklist items. The resulting checklist had a total of 42 items (see online supplementary appendix S3). Two authors (DH and WT) independently assessed each of the eligible reports against this checklist. Reports were also scored by the same authors for trial quality using the extended version of the Linde Internal Validity Scale (ELIVS) (see table 3, online supplementary appendix S2). The ELIVS scoring system used in this study was developed from initial work by Jadad et al23 and Linde et al.24 It measures the following quality domains: treatment allocation, randomisation method, allocation concealment, postrandomisation baseline comparison, blinding, handling and reporting of withdrawals and intention to treat analysis. Any discrepancies were resolved by consensus. Interobserver analysis was assessed by calculating the Cohen's κ score (score 0.74 based on disagreement of 268/2268 points). Extraction of data from studies was carried out in Microsoft Excel (2010, Microsoft Corporation, Redmond, Washington, USA) using a prepiloted form that was tested on two randomly selected studies from 2010.

Table 3

Results from survey of corresponding authors

Question number	Topic	Yes	No	Can't remember/not sure
1	Currently aware of NPT	11 (65%)	5 (35%)	0
2	Aware of NPT at submission	6 (35%)	10 (65%)	0
3	NPT mentioned by editorial staff	2 (12%)	10 (59%)	5 (29%)
4	NPT mentioned by peer-reviewers	1 (6%)	11 (71%)	4 (24%)
5	NPT endorsement by journal would affect submission choice	5 (29%)	10 (65%)	1 (6%)

CONSORT, Consolidated Standards of Reporting Trials; NPT, non-pharmacological treatment.

For each report, we also extracted the following data: the number of authors, the continent where the study was conducted, multicentre status, number of study participants and reporting of ethics review and conflict of interest. For each journal included, we obtained the ISI 2011 impact factor and whether or not the journal endorsed (eg, recommended or required) the CONSORT statement and CONSORT-NPT extension (information obtained in 2012).

Author survey

We also emailed the corresponding author for each included report, with five questions relating to the CONSORT-NPT extension in April 2013. The questions were: (1) Are you currently aware of the 2008 CONSORT-NPT extension? (2) Were you aware of the 2008 CONSORT-NPT extension at the time of submission? (3) Did the journal editorial staff mention the CONSORT-NPT extension to you during the editorial process (other than the instructions for authors on the journal website)? (4) Did the journal peer-reviewers mention the CONSORT-NPT extension to you during the review process? (5) Would your choice of journal for submission be affected by whether or not the journal mentions the CONSORT-NPT extension in their online instructions for authors? Each answer could be reported as ‘yes’, ‘no’ or ‘cannot remember/unsure’.

Outcomes and statistical analysis

Our primary outcome measure was adherence measured as the proportion of articles reporting each individual CONSORT and CONSORT-NPT checklist item. We also compared any differences in adherence between reports published in general medical journals with those published in surgical journals. All analyses were performed using STATA statistical software V.12.1 (College Station, Texas, USA).

Results

Our initial PubMed searches identified 831 possible reports, of which 771 were excluded as ineligible based on the information reported in the title and abstract. Sixty full-text articles were retrieved for further assessment, of which six were excluded because they were reports of previously published trials. This left 54 RCTs with a combined total of 16 338 patients from 11 journals that met the inclusion criteria (summarised in figure 1).

Figure 1

Flow diagram of study selection.

The baseline characteristics of the included trials are shown in table 1. The medical journals had a tendency towards a higher number of patients and a larger number of authors as well as a greater proportion of multicentre, higher quality (as measured by the ELIVS scale) trials. The requirement for CONSORT adherence was variable between the medical and surgical journals. Overall, only around half of the articles were published in a journal that required (26/54 studies; 48%) CONSORT adherence (table 1). The percentage of articles published in a journal that mentioned CONSORT in the instructions to peer reviewers (9/54 studies; 27%) was lower.

Table 1

Baseline characteristics of included studies

	Overall	Medical journals	Surgical journals
	(n=54)	(n=22)	(n=32)
Trial characteristics
Number of patients, median (IQR)	177 (110–410)	363 (195–757)	129 (71–177)
Number of authors, median (IQR)	9 (6–12)	12 (9–17)	7 (6–11)
Impact factor, median (IQR)	7.5 (4.5–30.0)	33.6 (30.0–53.5)	4.6 (4.4–7.5)
Multicentre trials, n (%)	28 (52)	20 (91)	8 (25)
Ethics review, n (%)	54 (100)	22 (100)	32 (100)
COI declared, n (%)	47 (87)	22 (100)	25 (78)
ELIVS Quality score, mean (SD)	5.1 (1.5)	5.8 (1.4)	4.5 (1.3)
Journal CONSORT endorsement
CONSORT required in ITA, n (%)	26 (48)	13 (59)	13 (41)
CONSORT recommended in ITA, n (%)	28 (52)	9 (41)	19 (59)
CONSORT mentioned in ITPR, n (%)	9 (27)	6 (27)	3 (10)

COI, conflict of interest; ELIVS, Extended Linde Internal Validity Scale; ITA, instructions to authors; ITPA, instructions to peer-reviewers.

Adherence of trials to the modified CONSORT checklist was variable ranging from 0% to 100% for each of the individual 42 checklist items (table 2). The highest scoring trials satisfied 36 of 42 items while the lowest scoring trial satisfied only 18 items (median 27, IQR 23–31). There were eight items for which there was less than 30% overall compliance (indicated with an asterisk in table 2). Of these eight items, seven were specific to the CONSORT-NPT extension. These seven items related to the following topics: a full description of the care providers, centres and blinding status in the abstract (item 1b; adherence 13%), eligibility criteria for centres performing the interventions (item 4b; adherence 24%), how adherence of care providers with the protocol was assessed or enhanced (item 5c; adherence 13%), how clustering by care providers or centres was addressed as it relates to sample size (item 7a; adherence 6%), how care providers were allocated to each group (item 8b; adherence 17%), how clustering by care providers or centres was addressed as it relates to statistical methods (item 12b; adherence 4%), a description of care providers (case volume, qualification, expertise, etc) and centres (volume) in each group (item 15b; adherence 0%). The non-CONSORT-NPT item with less than 30% adherence related to the presentation of absolute and relative effect sizes for binary outcomes (item 17b; adherence 28%).

Table 2

Adherence of studies to modified CONSORT-NPT checklist

CONSORT number	Point	Overall adherence, n (%)	Medical adherence, n (%)	Surgical adherence, n (%)
1a	Title and abstract	46 (85)	17 (77)	29 (91)
1b		7 (13)*	7 (32)	0
2a	Background and objectives	54 (100)	22 (100)	32 (100)
2b		54 (100)	22 (100)	32 (100)
3a	Trial design	22 (41)	14 (64)	8 (25)
3b		54 (100)	22 (100)	32 (100)
4a	Participants	54 (100)	22 (100)	32 (100)
4b		13 (24)*	7 (32)	6 (19)
5	Interventions	48 (89)	19 (86)	29 (91)
5a		39 (72)	20 (91)	19 (59)
5b		50 (93)	20 (91)	30 (94)
5c		7 (13)*	6 (27)	1 (3)
6a	Outcomes	48 (89)	22 (100)	26 (81)
6b		54 (100)	22 (100)	32 (100)
7a	Sample size	3 (6)*	2 (9)	1 (3)
7b		52 (96)	22 (100)	30 (94)
8a	Randomisation sequence generation	30 (56)	12 (55)	18 (56)
8b		9 (17)*	2 (9)	7 (22)
9	Allocation concealment mechanism	28 (52)	17 (77)	11 (34)
10	Randomisation implementation	18 (33)	13 (59)	5 (16)
11a	Blinding	29 (54)	16 (73)	13 (41)
11b		30 (56)	16 (73)	14 (44)
12a	Statistical methods	54 (100)	22 (100)	32 (100)
12b		2 (4)*	0	2 (6)
13a	Participant flow	0*	0	0
13b		51 (94)	22 (100)	29 (91)
New	Implementation of intervention	38 (70)	12 (55)	26 (81)
14a	Recruitment	48 (89)	22 (100)	26 (81)
14b		51 (94)	22 (100)	29 (91)
15a	Baseline data	52 (96)	22 (100)	30 (94)
15b		0*	0	0
16	Numbers analysed	52 (96)	22 (100)	30 (94)
17a	Outcomes and estimation	21 (39)	18 (82)	3 (9)
17b		15 (28)*	14 (64)	1 (3)
18	Ancillary analyses	42 (78)	20 (91)	22 (29)
19	Harms	51 (94)	21 (95)	30 (94)
20	Limitations	35 (65)	19 (86)	16 (50)
21	Generalisability	24 (44)	14 (64)	10 (31)
22	Interpretation	51 (94)	22 (100)	29 (91)
23	Registration	49 (91)	22 (100)	27 (84)
24	Protocol	25 (46)	18 (82)	7 (22)
25	Funding	40 (74)	22 (100)	18 (56)

Headings in italics are covered within the CONSORT-NPT extension. Non-italics are exclusive to CONSORT 2010 statement. Further details available in online supplementary appendix S3.

We did not compare the different adherence rates in a statistically formal way between the trials published in general medical and surgical journals as originally planned owing to the small sample size and the large number of hypotheses that could potentially be tested (all 42 checklist items). General medical journals tended to report better adherence to checklist items than surgical journals.

We contacted the lead author for each of the 54 reports to ask about their awareness of CONSORT-NPT. Only 17 authors replied (31% response rate) and so we were not able to perform formal quantitative analysis on the survey results. On the basis of the replies we received, approximately one-third of respondents were aware of CONSORT-NPT at the time of submission, while two-thirds are aware of its existence now (table 3). Given the time lapse between manuscript submission and our short survey, about a quarter of respondents were unable to remember whether journal editors and peer-reviewers had mentioned CONSORT-NPT during the review process. Finally, one-third of respondents agreed that their choice of journal for submission would be affected by whether or not CONSORT-NPT was mentioned in the instructions for authors section of the journal (we did not ascertain the direction of this preference).

Discussion

Summary of main findings

In this study, we included 54 reports of surgical RCTs published in 2011 from a cross-sectional sample of 11 medical and surgical journals. We assessed these reports for their adherence to a combined CONSORT and CONSORT-NPT checklist with two main findings. First, reporting adherence of surgical RCTs to the CONSORT-NPT extension was much poorer than adherence to the main CONSORT checklist. Second, general medical journals were broadly superior in their NPT reporting as compared with surgical journals. To our knowledge, this is the first study to demonstrate this difference between journal types for the CONSORT-NPT extension and one of only a few studies to document NPT adherence.

Comparison with the literature

The findings from our study are in agreement with the existing literature on CONSORT adherence. A recent systematic review of 53 studies found that reporting has remained suboptimal despite the CONSORT statement having been active in various iterations since 1996.25 However, the authors suggest that journal endorsement does appear to have had a positive impact on adherence. One review that included a comparison between surgical RCTs published in medical and surgical journals also found that adherence to CONSORT items was significantly superior in medical journals.16 Both of these articles assessed adherence only to the standard CONSORT statement. A recent study that assessed adherence specifically to the CONSORT-NPT extension checklist before (2004) and after (2010) the checklist was launched found little improvement in NPT-specific items (although these have had less time for absorption by the community than the standard CONSORT statement).17 These were reported in less than 50% of trials during 2010. The adherence rates in our study for similar NPT items were even lower.

Limitations

Our study has several limitations. First, we included only studies published in high-impact English language journals indexed in PubMed in 2011. In combination with our strict inclusion criteria on what constituted a surgical intervention, this led to a small sample of only 54 RCTs. We were consequently unable to perform a detailed statistical analysis on individual checklist items. The decision to limit the cross-sectional sample to 1 year was made on pragmatic grounds owing to the very lengthy process of scoring the RCTs against the checklist items. It was also for this reason that we restricted the number of journals we searched to the top eight impact factor journals within each specialty.

A second limitation pertains to the cross-sectional nature of our study. We are unable to suggest whether any progress is being made in adherence to the NPT extension. As previously described, an interrupted before–after study found only moderate improvement between 2004 and 2010.17 Our author survey suggested that there had been an increase in the proportion of authors who were aware of the existence of the CONSORT-NPT extension between the time of submission and now. Whether this translates to improved adherence at the current time is unknown. A third limitation includes the fact that we did not assess study protocols for adherence to CONSORT criteria. Some authors may have included additional study details within the protocol.

A final limitation concerns the author survey. We were restricted in our ability to analyse these data by the poor response rate and the significant time lag between submission of the RCTs (c. 2010) and distribution of the survey to the corresponding authors in 2013.

Implications for authors and journals

The CONSORT-NPT items with the poorest adherence were predominantly related to details on the implementation of the intervention, the providers (surgeons) and the centres. The CONSORT-NPT extension was specifically created to encourage reporting of these intervention-specific items given their importance in generalising a trial intervention to non-trial populations. As an example, two early symptomatic carotid surgery trials, NASCET and ACAS,26 27 had restrictive criteria for selecting which surgeons and centres were permitted to perform the intervention. Consequently, one large national cohort study that followed on from these trials did not see as large an improvement in patient outcomes.28 The study pointed out that less than 4% of all US hospitals providing carotid endarterectomy were included in NASCET and indeed that Medicare patients treated at trial hospitals had a lower risk of dying than at other hospitals.

We might anecdotally expect CONSORT-NPT items to be more vigorously enforced by surgical journals. This is on the basis that surgeons (who would likely form a greater component of the journal's editorial board and peer reviewers) would be more familiar with the multiple elements of the intervention and the importance of these elements in their own practice. Naturally, therefore, they might be keener to see these reported more thoroughly in manuscripts reporting RCTs. Our results appear to suggest the opposite in that general medical journals displayed superior NPT adherence. It is difficult to ascertain whether this finding is confounded by the much larger impact factor of the general medical journals in our sample and the potential for a self-fulfilling prophecy (ie, better reported trials opt preferentially to try and publish in the high-impact medical journals rather than such RCTs being well reported as a prime result of enforcement by the medical journal). Overall, the wealth of potential confounders makes it difficult to conclude why medical journals displayed superior adherence.

The literature on CONSORT adherence failures is extensive and is developing similarly for the CONSORT-NPT extension.25 Now that the problem has been well documented, the focus will likely shift towards identifying actionable areas for intervention. In the first instance, we suggest that qualitative interviews and focus groups with stakeholders at surgical trials departments will be important. Identifying the precise barriers to adherence will better inform the community on how best to improve reporting and where the greatest impact can be had. For example, does the problem lie with restrictive word counts, lack of time, lack of enforcement or simply just lack of awareness? How far would journals be prepared to go with enforcement? Would this be a viable option for smaller impact journals, perhaps fearful of driving authors away by enforcing reporting guidelines too rigidly?

These are all important questions that could be further elucidated by qualitative research in this field. New guidance on surgical RCT methodology29 and calls for greater investment in surgical research30 should be combined with a greater awareness of the CONSORT-NPT extension. Reporting standards, like trial design, are not static but need to adapt to the changing research landscape. CONSORT, therefore, needs to respond to proposals for new reporting standards such as those proposed by the IDEAL Collaboration29 in future NPT extensions. Notably, the All Trials movement pushing for transparency of pharmaceutical trials has garnered much attention from the public and press over recent months.31 This momentum has added weight to the growing call for thorough reporting to be considered a core duty of clinical researchers rather than just a desirable trait.

Conclusion

The findings from this cross-sectional review of surgical trials suggest that adherence to CONSORT-NPT extension items is much poorer than to the standard CONSORT statement. Adherence also appears to be superior in general medical journals compared with surgical journals. A combination of more qualitative research to identify the areas for specific intervention and a continuing effort to raise awareness of the CONSORT-NPT extension among stakeholders will be important going forward.