The quality of surgical versus non-surgical randomized controlled trials.

BACKGROUND: Randomized controlled trials (RCTs) of surgical interventions are often more difficult to conduct, and their quality may lag behind other trials.
OBJECTIVES: To compare quality domains in trials of surgical interventions, to a previously reported control sample of general medical trials from December 2006. DATA SOURCES: MEDLINE, EMBASE and CENTRAL were searched in May 2009. STUDY ELIGIBILITY: English language, full text RCTs, performed on humans that compared a surgical intervention to any other intervention. STUDY APPRAISAL AND SYNTHESIS: Records were ordered according to their date of publication. The most recently published trials were assessed first, until the required sample of 400 trials was obtained. The search and data collection were piloted by three authors, and was thereafter collected by one author. Data was checked.
RESULTS: Although most quality domains were poorly reported, surgical trials compared favourably to general medical trials. Surgical trials were 24% more likely to have an adequate method of random sequence generation, and 71% more likely to have an adequate method of allocation concealment. However, blinding was 40% less likely to be adequate in surgical trials, and sources of funding were 33% less likely to be reported. LIMITATIONS: A single author collected most data. Data was checked and this resulted in few changes.
CONCLUSIONS: Reporting of most quality domains in surgical intervention trials was better than general medical trials. Blinding was less likely due to the difficulty in conducting sham surgical trials, and reporting of sources of support should improve with adherence to reporting guidelines.

Introduction

It has been suggested that the quality of surgical randomized controlled trials (RCTs) is generally poorer than other specialties [1]. Surgical trials are faced with particular challenges. Where surgical interventions are tested, blinding is difficult [2], patients and/or surgeons usually prefer one treatment, and surgical interventions are difficult to standardize [3]. In one subject area, osteoarthritis, trials with pharmacologic interventions had higher quality scores than non-pharmacologic trials [4]. It is unclear if this finding is generalizable to other clinical areas, as a direct comparison has not been done. We compared the quality of surgical trials to a previously published cross sectional sample of non-surgical trials [5] in a mixture of general medical specialties.

Methods

Study sampling

We searched MEDLINE, EMBASE and CENTRAL for surgical trials using a highly sensitive RCT filter used by the Cochrane Collaboration, combined with a surgery filter. The syntax of the search strategy is available online. We included English language, full text RCTs, performed on humans that compared a surgical intervention to any other intervention (Table 1).

Table 1

Inclusion criteria for surgical trials.

• Randomized controlled trial. We defined this as a study where participants are randomly assigned to one of two or more treatment arms of a clinical trial.

• Published as a full text article. Studies published as abstracts or conference proceedings were excluded.

• Published in the English language.

• The primary publication of an investigation, where multiple publications from one investigation were found. This was defined as the first (earliest) publication from an investigation, or the publication where the methods of the trial were described in full.

• Conducted on humans (not cadavers).

• Compared a surgical intervention to any other intervention. We defined a surgical intervention as any procedure that requires surgical training and is usually performed by a surgeon of any subspecialty recognized by the Royal Australasian College of Surgeons. This included cardiothoracic, general (encompassing upper/lower gastrointestinal and transplant), neurosurgery, otolaryngology, paediatric, plastic, vascular, orthopaedic and urology surgery.

We used a previously published quality assessment of general medical trials from December 2006 as a control group [5]. In order to retrieve a sufficient sample of surgical trials for comparison, we executed the search in May 2009. Assessment of trials proceeded in reverse chronological order until the required sample of 400 trials was included. Titles/abstracts were screened first, before an assessment of full text articles. Study identification was piloted for consistency by two authors using the first 1000 records. This process resulted in almost perfect agreement (kappa statistic = 0.85). Further screening was done by one author, and then checked by a second author.

Data extraction

We extracted the same quality domains as Hopewell et al., [5] including “randomization” in the title of the publication, specification of primary outcomes, sample size calculation, random sequence generation, allocation concealment, blinding of patients, caregivers, and/or outcome assessors, handling of attrition, and source of funding. We contacted the authors to ensure our definitions were uniform (personal email communication with Sally Hopewell, 17th September 2012).

Three authors piloted the data collection proforma. This process resulted in a high level of agreement in the classification of domains (kappa statistic 0.75). Following the pilot phase, data collection was performed by one author, and then checked by a second author. One data point discrepancy was discussed and resolved, and arbitration by a third author was not required.

Data analysis

Quality domains were assessed as adequate or inadequate/unclear, and the proportion of adequately reported trials was calculated. For each domain, the proportion of adequately reported surgical trials was compared to the proportion adequately reported according to Hopewell et al. [5] A risk ratio (RR) was calculated along with its 95% confidence interval (CI), where a RR > 1 favoured surgical RCTs (see Fig. 1).

Fig. 1

Forest plot depicting comparison of adequate scientific quality domains for surgical RCTs vs. general medical RCTs. RR = Risk Ratio. 95% CI = 95% Confidence Interval.

Results

12,674 records were assessed based on title and/or abstract, and 1015 records based on their full text. 400 RCTs were included. Surgical trials were published over a period of eight months from August 2008 to May 2009.

Most scientific quality domains were poorly reported (Table 2). 35% of trials did not specify a primary outcome, and more than half (58%) did not report a sample size calculation. Most trials (57%) did not report an adequate method of allocation concealment.

Table 2

Reporting of surgical randomized controlled trials.

Quality characteristic	n (%) (Total N = 400)
“Randomized” in title	222 (56)
Specification of primary outcome
Adequate	261 (65)
Unclear	139 (35)
Sample size calculation
Adequate	167 (42)
Unclear	233 (58)
Generation of random sequence
Adequate	168 (42)
Computer generated	111 (66)
Random number table	29 (17)
Cards	11 (7)
Lots/lottery	10 (6)
Coin flip	4 (2)
Other	3 (2)
Unclear	232 (58)
Allocation concealment
Adequate	173 (43)
Envelopes	128 (74)
“Sealed”	102 (80)
“Opaque”	30 (23)
“Numbered”	37 (29)
Central/third party allocation	45 (26)
Unclear	227 (57)
Blinding
Any blinding	140 (35)
Patient	67 (17)
Carer	32 (8)
Outcome assessor	123 (31)
Primary outcome blinded	106 (27)
Unclear	260 (65)
Handling of attrition
Intention to treat	107 (27)
Follow up only	256 (64)
Inadequate	37 (9)
Source of funding
Reported	165 (41)
Full industry	10 (6)
Partial industry	64 (39)
Non-industry	55 (33)
No external source	36 (22)
Unreported	235 (59)

The reporting of surgical trials compared favourably to general medical trials. In surgical trials, the term “randomization” was 67% more likely to be used in the title (RR = 1.67, 95%CI 1.45–1.92), and were 24% more likely to describe their primary outcome (RR = 1.24, 95%CI 1.12–1.38), 24% more likely to report an adequate method of random sequence generation (RR = 1.24, 95%CI 1.06–1.45), and 71% more likely to report an adequate method of allocation concealment (RR = 1.71, 95%CI 1.43–2.04). Blinding (RR = 0.60, 95%CI 0.51–0.69) and source of funding (RR = 0.67, 95%CI 0.59–0.76) were less likely to be reported in surgical RCTs.

Discussion

We found that scientific quality domains were often inadequately reported in surgical trials, but reporting compared well with previous assessments of RCTs in other specialities. It is possible that (given surgical RCTs are often more challenging to perform) authors of surgical RCTs are more aware of the importance of methodological safeguards. Blinding was less common, and this reflects the difficulty of blinding a surgical trial in the absence of a sham intervention, which was rare. Sources of trial support were also less likely to be reported in surgical RCTs. Recent controversy regarding conflicts of interest [6], and adherence to reporting guidelines is expected to result in improvements.

Sources of funding

Dr. Sam Adie was supported by scholarship grants from the National Health and Medical Research Council of Australia (630761) (Biomedical Postgraduate Scholarship), and the Royal Australasian College of Surgeons (165214) (Sir Roy McCaughey Research Fellowship). The funders had no role in the design, data collection or analysis of this study.

Declaration of conflicts of interest

The authors declare no conflicts of interest.

Authors' contributions

Sam Adie was involved in the literature search, study design, data collection, data analysis, data interpretation, and the preparation of the manuscript.

Ian Harris was involved in the study design, data interpretation, and the preparation of the manuscript.

Justine Naylor was involved in the data collection, data interpretation, and the preparation of the manuscript.

Rajat Mittal was involved in the data collection, data interpretation, and the preparation of the manuscript.

Ethics committee approval

Ethics approval was gained from the South Western Sydney Local Health District, Human Research Ethics Committee, approval number QA2010/013.