Quality of abstracts of randomized control trials in five top pain journals: A systematic survey.

BACKGROUND: The reporting quality of abstracts of randomized control trials (RCTs) is inadequate despite the publication of consolidated standards of reporting trials extension for abstracts (CONSORT-A). We compared the reporting quality of abstracts in pain journals before and after the publication of CONSORT-A.
METHODS: We searched MEDLINE in April-2016 for RCTs published in five pain journals: Pain, Pain Physician, European Journal of Pain, Clinical Journal of Pain and Pain Practice for pre- and post-CONSORT-A period (2005-2007 and 2013-2015). Data were extracted in duplicate from 250 abstracts for compliance with CONSORT-A, and for items known to affect reporting quality: journal endorsement of CONSORT, number of trial centers, sample-size, type of intervention, industry-sponsorship and significance of results. The primary outcome was mean number of items reported and the secondary outcome was the reporting of each item. We used logistic regression and Poisson regression for analyses.
RESULTS: Most trials were single centric (76%), had sample size <100 (63%), involved pharmacological intervention (59%) and were non-industry funded (70%). The mean number of items reported was better for 2013-2015 (mean difference 0.94; 95% confidence-interval [CI]: 0.50-1.38, p < 0.001). Post-CONSORT-A, trials were more likely to report as randomized in the title (odds ratio (OR) 2.69; 95% CI 1.61-4.49), describe eligibility criteria and settings (OR 2.47; 95% CI 1.35-4.54), provide effect size and precision for primary outcome (OR 2.47; 95% CI 1.19-5.16), inform harms (OR 1.80; 95% CI 1.05-3.07) and report trial registration (OR 5.13; 95% CI 1.44-18.32). Post-CONSORT-A period (incident rate ratio (IRR) 1.15; 95% CI 1.07-1.24), endorsement of CONSORT statement by the journal (IRR 1.08; 95% CI 1.02-1.14), multi-centric studies (IRR 1.14; 95% CI 1.08-1.20), and studies with pharmacological interventions (IRR 1.07; 95% CI 1.02-1.13) were significantly associated with reporting of more items.
CONCLUSIONS: Abstract reporting for trials in pain literature was better in the post-CONSORT-A period, but there is room for improvement.

Introduction

Pain journals are increasingly publishing RCTs over the last few years. Abstracts of randomized controlled trials (RCTs) are often the first and the only source read by busy physicians [1]. This could be due to non-availability of full-text from lack of paid subscription, non-English language of articles or most commonly, time constraint. Therefore, it is necessary that important details about the trial are transparently and completely provided to the readers to make accurate judgment regarding suitability of applying the trial findings to their patients. Previous studies have demonstrated that reporting quality is poor among general medical journals [2] and adherence to the consolidated standards of reporting trials extension for abstracts (CONSORT-A) has resulted in some improvements in reporting [5], [10]. The CONSORT-A is a 17-item checklist that the authors are expected to adhere to while reporting the abstracts of trials [6]. The quality of reporting of abstracts of RCTs in pain journals and the impact of CONSORT-A on the reporting quality is currently not known. The purpose of this study is to inform pain practitioners and researchers on the current quality of reporting of abstracts in pain journals and how reporting of abstracts of RCTs actually need to be done. To achieve this purpose, we compared the completeness of reporting of abstracts before and after the publication of CONSORT-A in five top pain journals and secondarily, explored the factors that might possibly influence the quality of reporting.

Methods

We conducted a thorough search of Medline in April 2016 for abstracts of RCTs published in top five pain journals as determined by the journal citation index report of the Thomson Reuters 2014 [7] for the period 01-01-2005 to 31-12-2007 (pre-CONSORT-A) and 01-01-2013 to 31-12-2015 (post-CONSORT-A). The journals included in the study were Pain, Pain Physician, European Journal of Pain, Clinical Journal of Pain and Pain Practice. Our search strategy and other aspects about the methods are reported in detail in our protocol [12]. Briefly, we included abstracts of RCTs if they were reports of RCTs, published in English language and involved human subjects. We excluded articles if the abstract was not available, if the abstract was published as a conference proceeding, if the trial was still recruiting patients and if it was a duplicate publication. Ethical approval was not obtained as this study was only a systematic survey of the published literature. The sample size for this study was determined based on our hypothesis that there will be significant improvement in the mean number of items reported after the publication of CONSORT-A. The details regarding sample size estimation is described in our protocol [12].

We extracted data regarding adherence to each of the 17 items of the CONSORT-A for both the study periods. Additionally, we obtained information regarding journal endorsement of CONSORT statement, number of centers included in the trial, sample size of the study, type of therapeutic intervention, industry sponsorship and significance of results for the primary outcome from the article full-text to explore and explain the quality of reporting. Both screening of titles and abstracts and full-text review were done independently and in pairs by four reviewers. The agreement between reviewers for inclusion of abstracts was assessed using kappa statistic. Any disagreement was resolved through consensus and if consensus could not be reached, through arbitration by a third author. A pilot exercise was performed before formal data extraction with 10% of the abstracts to improve clarity regarding eligibility criteria and to increase consistency among reviewers.

Statistical analyses

The characteristics of the included trials were analyzed using descriptive statistics and reported as mean (standard deviation [SD]) or median (first quartile, third quartile) for continuous variables depending on the data distribution and count (percent) for categorical variables. We describe the count (percent) of articles reporting each item by period of publication (2005–2007 vs. 2013–2015). The mean (median) number of items (0–17) reported for each study period and the unadjusted and adjusted mean (median) differences were compared using a two-sample t-test and generalized estimation equations (GEE), respectively and reported with 95% confidence intervals (CIs) and p values. Similarly, the compliance with each of the 17 items of the CONSORT-A for 2005–2007 period were compared with 2013–2015 period using Chi-squared tests and GEE test was used to analyze the data adjusting for confounders. The unadjusted ORs, and 95% CI are reported. Finally, the incidence rate ratios (IRRs) for reporting items for 2013–2015 period was compared with 2005–2007 period using GEE assuming a Poisson distribution and an unstructured correlation matrix. Adjusted IRRs, 95% CIs and p-values are reported. The threshold for statistical significance was set at alpha = 0.05.

For the multivariable analysis using GEE, adjustments were made for 1) journal endorsement of the CONSORT statement, 2) number of trial centers [multiple centers versus single center], 3) type of intervention [pharmacological versus non-pharmacological], 4) sample size [≤ 100 versus > 100], 5) statistical significance for primary outcome of the trial [statistically significant versus not significant] and 6) funding status [industry funded versus others] with journal as a grouping factor. Data was analyzed using Statistical Package for Social Sciences (SPSS) Version 24.0 (SPSS, Inc., 2009, Chicago, Illinois, USA).

Results

We retrieved 953 abstracts from our search, 430 in the pre-CONSORT-A period and 523 in the post-CONSORT-A period. We excluded 536 ineligible abstracts; retaining 417 (146 in pre-CONSORT-A and 271 in post-CONSORT-A period) for inclusion. Based on our sample size estimation as described in our protocol [12], we needed 125 abstracts for each period. Hence, we randomly selected 125 abstracts from each period for analysis. The flow diagram demonstrating details of the study process is shown in PRISMA Flow diagram (Fig. 1). We achieved a very high agreement for inclusion of articles between the reviewers; kappa = 0.94 (95% CI = 0.91, 0.96), p < 0.001. Table 1 provides information on the pre-defined study characteristics and description of articles by study period. The mean number of items reported were 6.12 (1.59) and 7.06 (1.93) for pre- and post-CONSORT-A periods respectively. The unadjusted difference by a two-sample t-test was 0.94 (95% CI: 0.50–1.38, p < 0.001) and the adjusted difference by GEE was 0.89 (95% CI: 0.47–1.31, p < 0.001).

Fig. 1

PRISMA Flow diagram showing study selection procedure.

Table 1

Distribution of articles by study period and pre-selected characteristics.

Characteristic	Study period		Total (n = 250)n (%)
	2005–2007 (n = 125)n (%)	2013–2015 (n = 125)n (%)
Journal
Pain	43 (34.4)	25 (20.0)	68 (27.2)
Pain Physician	03 (2.4)	25 (20.0)	28 (11.2)
European journal of pain	24 (19.2)	25 (20.0)	49 (19.6)
Clinical journal of pain	49 (39.2)	25 (20.0)	74 (29.6)
Pain practice	06 (4.8)	25 (20.0)	31 (12.4)
Site(s) of study
Single	98 (78.4)	92 (73.6)	190 (76.0)
Multiple	27 (21.6)	33 (26.4)	60 (24.0)
Sample Size
1–s100	84 (67.2)	73 (58.4)	157 (62.8)
>100	41 (32.8)	52 (41.6)	93 (37.2)
Intervention
Pharmacological	73 (58.4)	75 (60.0)	148 (59.2)
Others	52 (41.6)	50 (40.0)	102 (40.8)
Funding
Industry	46 (36.8)	28 (22.4)	74 (29.6)
Non-industry	79 (63.2)	97 (77.6)	176 (70.4)

We observed statistically significant improvements in completeness of reporting during the post-CONSORT-A period for five of the seventeen items, compared to the pre-CONSORT-A period. More abstracts in the 2013–2015 periods identified the trial as randomized in their titles (OR 2.69; 95% CI 1.61–4.49), reported eligibility for participants and details regarding settings better (OR 2.47; 95% CI 1.35–4.54), provided effect size for primary outcome and it's precision (OR 2.47; 95% CI 1.19–5.16), reported adverse events (OR 1.80; 95% CI 1.05–3.07) and informed about trial registration number and name of the registry (OR 5.13; 95% CI 1.44–18.32) (Table 2). The rest of the items were similarly reported during both the study periods.

Table 2

Odds ratio for compliance with 17 items checklist before and after publication of CONSORT statement for abstracts.

Item name	Item details	Pre-CONSORT periodN (%)	Post-CONSORT periodN (%)	Univariate analysisOR; (95% CI)
Title	Identifies the study as randomized	53 (42.4)	83 (66.4)	2.69 (95% CI: 1.61–4.49)
Authors	Contact details for corresponding author provided	124 (99.2)	122 (97.6)	0.33 (95% CI: 0.03–3.20)
Trial design	Describes trial design (e.g. parallel, cluster)	37 (29.6)	45 (36.0)	1.34 (95% CI: 0.79–2.27)
Methods
Participants	Provides eligibility criteria for participants and details of settings	20 (16.0)	40 (32.0)	2.47 (95% CI: 1.35–4.54)
Interventions	Informs interventions for each group	124 (99.2)	124 (99.2)	1.00 (95% CI: 0.06–16.17)
Objective	Reports specific objective or hypothesis	119 (95.2)	122 (97.6)	2.05 (95% CI: 0.50–8.39)
Outcome	Clearly defines primary outcome	56 (44.8)	63 (50.4)	1.25 (95% CI: 0.76–2.06)
Randomization	Informs how participants were allocated to interventions	1 (0.8)	3 (2.4)	3.05 (95% CI: 0.31–29.72)
Blinding (masking)	Provides details on whether there was blinding and who was blinded	6 (4.8)	12 (9.6)	2.11 (95% CI: 0.77–5.80)
Results
Numbers randomized	Informs about number of participants randomized to each group	44 (35.2)	47 (37.6)	1.11 (95% CI: 0.66–1.86)
Recruitment	Reports about trial status (e.g. completed or interim analyses)	3 (2.4)	3 (2.4)	1.00 (95% CI: 0.20–5.05)
Numbers analyzed	Informs about number of participants analyzed in each group	11 (8.8)	8 (6.4)	0.71 (95% CI: 0.28–1.83)
Outcome	Provides effect size for primary outcome and it's precision	12 (9.6)	26 (20.8)	2.47 (95% CI: 1.19–5.16)
Harms	Reports about adverse events	33 (26.4)	49 (39.2)	1.80 (95% CI: 1.05–3.07)
Conclusions	Interprets and summarizes results	119 (95.2)	122 (97.6)	2.05 (95% CI: 0.50–8.39)
Trial registration	Provides registration number and name of trial registry	3 (2.4)	14 (11.2)	5.13 (95% CI: 1.44–18.32)
Funding	Informs about source of funding	0 (0.0)	0 (0.0)	Not estimable

After GEE, post-CONSORT-A period (IRR 1.15; 95% CI 1.07–1.24; p < 0.001), endorsement of CONSORT statement by the journal (IRR 1.08; 95% CI 1.02–1.14; p = 0.005), multi-centric studies (IRR 1.14; 95% CI 1.08–1.20; p < 0.001), and studies with pharmacological interventions (IRR 1.07; 95% CI 1.02–1.13; p = 0.014) were significantly associated with reporting of more items. Sample size, statistical significance of the primary outcome and funding status were not associated with number of items reported (Table 3).

Table 3

Adjusted incidence rate ratios for total number of items reported from CONSORT extension for abstracts.

Factors	Adjusted incidencerate ratio [IRR] (95% CI)	P value
Period of publication
Pre-CONSORT	1	<0.001
Post-CONSORT	1.15 (95% CI: 1.07–1.24)
Endorsement of CONSORT by journal
No	1	0.005
Yes	1.08 (95% CI: 1.02–1.14)
Study site(s)
Single	1	<0.001
Multiple	1.14 (95% CI: 1.08–1.20)
Intervention
Non-pharmacological	1	0.014
Pharmacological	1.07 (95% CI: 1.02–1.13)
Sample size
1-100	1	0.546
>100	1.03 (95% CI: 0.94–1.12)
Results of trial
Statistically insignificant	1	0.071
Statistically significant	1.11 (95% CI: 0.99–1.25)
Funding status
Non-funded	1	0.223
Industry funded	1.03 (95% CI: 0.98–1.08)

Discussion

In this study, we observed that the overall reporting quality of abstracts in pain journals was poor and the improvement in the number of items of the CONSORT-A reported was also marginal (6.12 ± 1.59 and 7.06 ± 1.93 for pre- and post-CONSORT-A periods, respectively). We observed improvement for only five of the seventeen items of CONSORT-A for the period 2013–2015 compared to 2005–2007 in our study, while for other items there was no difference between the two time periods studied. In a study evaluating the reporting quality in four high impact factor medical journals two years after the publication of CONSORT-A, poor adherence to CONSORT-A components (9–99%) was documented [5]. Given the prolonged time period (seven years) since publication of CONSORT-A, we expected significant improvement in number of items reported in our study. However, our findings suggest that awareness about CONSORT-A among all stakeholders; readers, authors, reviewers and journal editors is inadequate and consequently, adherence to CONSORT-A remains poor contributing to incomplete reporting of abstracts in pain journals.

The quality of reporting of abstracts in pain journals also fares poorly in comparison with other medical journals. The mean number of items reported in 2012, four years after publication of CONSORT-A, was higher than 2007 in top five general medical journals (9.1 vs. 12.1; p < 0.001) [10]. This is in contrast to our findings for top five pain journals (6.1 vs. 7.1; p < 0.001).

Based on earlier evidence, we had a-priori hypothesized that certain characteristics will influence better reporting. Previous studies have demonstrated that RCTs from journals that endorse CONSORT statement [14], multi-centric studies [11], studies with larger sample size [3], studies involving pharmacological intervention [13], studies reporting significant results for their primary outcome [8] and industry sponsored studies [8] are more compliant with the CONSORT guidelines and therefore better reported. However, in this study, we observed that only factors such as post-CONSORT-A period of publication, endorsement of CONSORT statement by the journal, multi-centric studies and studies with pharmacological interventions were significantly associated with reporting of more items in the abstract.

In terms of improvement in quality of reporting since publication of CONSORT-A, our findings about pain journals are better (5.5% improvement in the mean items reported) than that reported for major anesthesia journals, where a mere 2.4% improvement between pre- and post-CONSORT period was observed [4]. However, this improvement is small compared to an 18% improvement observed in high impact factor general medical journals [10] in 2012. Considering that more years have passed since the publication of CONSORT-A, our current findings reflects poor reporting of abstracts in pain journals. Given the above, it is desirable that the editors of various pain journals implement the use of CONSORT-A in the editorial process and ensure that the authors complete this checklist as a part of their manuscript submission. Secondly, the journal should encourage reviewers to evaluate the abstracts of pain trials with regards to quality based on the compliance to the CONSORT-A. Lastly, authors should voluntarily report all elements of the CONSORT-A to ensure transparent and complete reporting of various aspects of their study. These measures are likely to improve the quality and completeness of reporting of abstracts of pain trials in the coming years.

This study has relevance to all those involved in pain management and research. Pain journals are increasingly publishing RCTs to disseminate high quality evidence. Uniform and complete reporting of various aspects of the study design, methods and results not only help the clinicians, but also patients who seek treatment or wish to volunteer for research, to interpret the abstract accurately and make well-informed decisions. Previous studies have shown that incomplete information leads to suboptimal application of findings from RCTs [9] and therefore better reporting is likely to result in increased utilization of new found evidence from well conducted trials.

There are certain limitations of this study. Firstly, we included only top five pain journals based on their impact factors which may not be representative of all pain journals. However, it is more likely that pain journals with lower impact factors are less likely to have better reporting quality than their more popular counterparts. Secondly, improvement in the quality of reporting could be due to reasons other than adherence to CONSORT-A. These factors include better conduct of the study itself resulting in better reporting, mandatory requirements by funding agencies, and increased author awareness. It is important to note that poor reporting in the abstract does not always mean poor conduct of study as frequently, word restriction for abstracts and requirement of a particular structure of abstract by journals might contribute to incomplete reporting. Thirdly, the univariate analyses in this study were purely exploratory in nature and therefore not adjusted for multiple testing. Lastly, we adjusted for certain factors known to contribute to reporting quality during our analysis but there could possibly be other factors that might have influenced reporting quality which we did not consider.

Conclusions

The overall quality of reporting of abstracts in pain journals remains poor despite the availability of guidelines for reporting of abstracts of trials. Compared to the 2005–2007 (pre-CONSORT-A) period, marginal improvement in reporting quality of abstracts of RCTs was observed during 2013–2015 (post-CONSORT-A) period in the top five pain journals we studied. Apart from post-CONSORT-A period, endorsement of CONSORT statement by the journal, multicentric trials and pharmacological RCTs were associated with more number of items being reported. All stakeholders, namely; readers, researchers, reviewers and journal editors need to work together to effectively implement CONSORT-A guidelines to improve the reporting quality of abstracts in pain journals.

Significance

This study demonstrates that the reporting quality of abstracts of randomized controlled trials remains poor in top pain journals despite several years of CONSORT guidelines for abstracts.

Author contributions

LT was responsible for the conception of the review. RC was involved in the search strategy. SK and LT were involved in the designing of the review. SK, SB, LPFA and MW were involved in review and data extraction. YJ and MP performed the data analyses. SK was involved in writing the initial draft, SB, MW and LPFA contributed to improvements in the manuscript and LM and LT critically revised the final manuscript. All authors discussed the results and contributed to the manuscript.

Funding

No funding was obtained for this study.

Conflict of interests

None of the authors have any conflict of interest to declare.