Reporting quality of randomised controlled trial abstracts among high-impact general medical journals: a review and analysis.

OBJECTIVE: The aim of this study was to assess adherence to the Consolidated Standards of Reporting Trials (CONSORT) for Abstracts by five high-impact general medical journals and to assess whether the quality of reporting was homogeneous across these journals.
DESIGN: This is a descriptive, cross-sectional study.
SETTING: Randomised controlled trial (RCT) abstracts in five high-impact general medical journals. PARTICIPANTS: We used up to 100 RCT abstracts published between 2011 and 2014 from each of the following journals: The New England Journal of Medicine (NEJM), the Annals of Internal Medicine (Annals IM), The Lancet, the British Medical Journal (The BMJ) and the Journal of the American Medical Association (JAMA). MAIN OUTCOME: The primary outcome was per cent overall adherence to the 19-item CONSORT for Abstracts checklist. Secondary outcomes included per cent adherence in checklist subcategories and assessing homogeneity of reporting quality across the individual journals.
RESULTS: Search results yielded 466 abstracts, 3 of which were later excluded as they were not RCTs. Analysis was performed on 463 abstracts (97 from NEJM, 66 from Annals IM, 100 from The Lancet, 100 from The BMJ, 100 from JAMA). Analysis of all scored items showed an overall adherence of 67% (95% CI 66% to 68%) to the CONSORT for Abstracts checklist. The Lancet had the highest overall adherence rate (78%; 95% CI 76% to 80%), whereas NEJM had the lowest (55%; 95% CI 53% to 57%). Adherence rates to 8 of the checklist items differed by >25% between journals.
CONCLUSIONS: Among the five highest impact general medical journals, there is variable and incomplete adherence to the CONSORT for Abstracts reporting checklist of randomised trials, with substantial differences between individual journals. Lack of adherence to the CONSORT for Abstracts reporting checklist by high-impact medical journals impedes critical appraisal of important studies. We recommend diligent assessment of adherence to reporting guidelines by authors, reviewers and editors to promote transparency and unbiased reporting of abstracts. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/

Data were gathered through an objective extraction process.

Our study benefitted from a large sample size.

Reviewers were blinded to the journal and randomly assigned articles to score.

Articles were from 2011 to 2014, providing researchers and editors adequate time to fully implement the checklist guidelines published in 2008.

Our study conclusions may not be applicable to journals not included in our analysis.

Background

Randomised controlled trials (RCTs) are considered the gold standard for evidence for interventions, but assessing trial validity is dependent on the quality and transparency of the study report.1 2 This requires the inclusion of key study information in abstracts and manuscripts so that readers can properly assess the validity and generalisability of each study and apply the findings to their patient population. Responsibility extends to peer reviewers and medical journal editors who must verify that the information needed to evaluate study quality is reported in abstracts and manuscripts, and guidelines exist to ensure that the essential elements are reported in the manuscript and the abstract.3–6 The abstract may be the only read portion of the study, and without clear reporting, this could lead to misinterpretation and poor patient outcomes.7 One study found that as many as 63% of practising internal medicine physicians relied solely on the abstract of general internal medicine or general medical journals. This was consistent for physicians with and without formal epidemiological training.7

The Consolidated Standards of Reporting Trials (CONSORT) Statement was initially developed in 1996 to improve reporting of RCTs in journals.3–6 8 These guidelines underwent subsequent modification in 2001, 2006 and 2010 to improve the manuscript reporting process.8 In 2008, the CONSORT for Abstracts checklist was created as an extension of the original CONSORT Statement in order to improve the reporting of RCT abstracts in journals and conference proceedings thereby allowing readers to quickly assess validity and applicability of a trial.4

Editors of high-impact journals have endorsed the use of the CONSORT guidelines to facilitate transparent and unbiased reporting of trial results.9 10 Articles published in high-impact journals are commonly cited in the medical literature and frequently reported in the lay press;11 accordingly, the importance of complete and unbiased reporting in these journals is paramount. However, two previous studies examining high-impact medical journals highlighted the lack of adherence to reporting guidelines in RCT abstracts.12 13 A number of other studies have also assessed the quality of reporting of abstracts of RCTs using various methodologies, but many of these had factors that limited their overall rigour, such as small sample sizes, limited blinding and a lack of an evaluation of inter-rater agreement.14–21 We conducted this study to rigorously assess the adherence of high-impact, high-visibility general medical journals to the reporting quality standards set forth in the CONSORT for Abstracts checklist. The primary outcome was per cent overall adherence to the 19-item CONSORT for Abstracts checklist. Secondary outcomes included per cent adherence in checklist subcategories and assessing homogeneity of reporting quality across the individual journals.

Methods

Search strategy and study selection

We conducted a descriptive, cross-sectional study of RCT abstracts in five journals with the highest impact factors in 2014.22 We included abstracts published between 2011 and 2014 in The New England Journal of Medicine (NEJM), the Annals of Internal Medicine (Annals IM), The Lancet, the British Medical Journal (The BMJ), and the Journal of the American Medical Association (JAMA) that reported the main results of parallel-group RCTs. We excluded observational or cohort studies, interim analyses, economic analyses of RCTs, post-trial follow-up studies, subgroup and secondary analyses of previously reported RCTs, editorials, letters and news reports. An author (KD) not involved in the abstract scoring applied the search strategy (figure 1) on 1 December 2014 to identify up to 100 of the most recent RCTs published in each of the top five general medical journals that met the eligibility criteria. Abstract selection for a particular journal started with those published in 2014, proceeded backwards in time and stopped when 100 abstracts for that journal that met eligibility criteria had been identified or the search for the year 2011 was completed, whichever came first. Abstracts were stored in EndNote X7 (Thomas Reuters, Philadelphia, Pennsylvania, USA). An author (DC) not involved in abstract scoring or data analysis imported abstracts from Endnote X7 into Excel (Microsoft, Redmond, Washington, USA). A customised Excel macro was used to remove PubMed Identification code, journal name, author names and journal-specific subheadings to ensure blinding of reviewers to journal. The same author (DC) maintained the key linking journal number (J1–J5) with journal name. In order to avoid bias, the authors remained blinded to journal identification, through data analysis and synthesis.

Figure 1

MEDLINE search strategy.

A priori sample size calculation using a β of 0.2 and an α of 0.05 indicated that a minimum of 58 studies per journal would be needed to detect a difference in adherence between any two journals of 25% or greater using the two-sample proportions Pearson's χ2 test. The difference of 25% was chosen by the authors to be the minimum difference they felt would be meaningful between any two journals.

Checklist development, application and inter-rater agreement

We determined compliance to each aspect of the CONSORT for Abstracts checklist through the use of a 19-item checklist (figure 2). This checklist was developed in an iterative manner by the authors by expanding the published CONSORT for Abstracts checklist to allow for evaluation of each component of the recommendations.4 Discrepancies between authors over application of checklist items were resolved by consulting the published explanation of the CONSORT for Abstracts checklist4 and by adding instructions and examples to the checklist items as shown in figure 3. Prior to scoring of study abstracts, raw inter-rater agreement for each item on the 19-item checklist was evaluated through a test run of 32 RCT abstracts published prior to 2011 and scored independently by three physician authors (MA, MH and RW) with graduate-level training in epidemiology and critical appraisal. We chose raw per cent agreement as a measure of inter-rater reliability for simplicity as well as the known difficulties with chance-corrected measures of agreement such as Cohen's κ, which can produce misleadingly low values in the setting of high-per cent agreement.15 23–25 After ensuring adequate inter-rater agreement with this sample, each study abstract was scored by a single author (MA, MH or RW).

Figure 2

CONSORT for Abstracts checklist.

Figure 3

Flow diagram of the study.

Data extraction and analysis

Study abstracts were randomly ordered using a computer-generated sequence in Excel (Microsoft) and divided among the three physician authors for review. Each item on the checklist was scored dichotomously (figure 3). The proportion of abstracts adherent to each checklist item was calculated for the entire sample and for the abstracts published in each journal. χ2 test of homogeneity using a significance level of 0.05 was utilized to test the null hypothesis that the proportion of abstracts adherent to checklist items was homogeneous across journals.26

Descriptive analysis was performed using STATA, V.13 (StataCorp, College Station, Texas, USA). The proportion of adherence was determined across all journals and checklist items, for all checklist items by individual journal and for all journals by individual checklist item. This represented the average of the adherence rates for all checklist items, weighted by the number of abstracts scored for each item. The number of abstracts scored differed only for the item of blinding, composed of two parts: items 10 and 11. Item 10 looked at generic blinding, whereas item 11 looked for detailed descriptions. If the study was described as blinded or masked to group assignment, the abstract was then rated as adherent if it was stated who was blinded or masked. This was concordant with the CONSORT for Abstracts checklist. If the study was not described as blinded or masked to group assignment, the abstract was not scored for blinding.

The raters (MA, MH and RW) were all trained clinicians with expertise in clinical epidemiology and critical appraisal.

Results

Journal characteristics

Individual journal characteristics are listed in table 1.

Table 1

Journal characteristics

Journal	Impact factor*	Use of CONSORT endorsed	Overall adherence† (%)
NEJM	54.42	Yes‡	55
The Annals IM	16.104	Yes‡	70
The Lancet	39.207	Yes	78
The BMJ	16.378	Yes	65
JAMA	35.289	Yes‡	63

*As determined by ISI Impact Factor 2014.

†Overall adherence to checklist items per journal based on our study findings.

‡No explicit mention of CONSORT for Abstracts in instructions to authors.

NEJM, The New England Journal of Medicine; Annals IM, the Annals of Internal Medicine; The BMJ, the British Medical Journal; JAMA, the Journal of the American Medical Association.

Study characteristics

PubMed search results yielded 466 study abstracts from the top 5 general medical journals (100 from NEJM, 66 from Annals IM, 100 from The Lancet, 100 from The BMJ and 100 from JAMA), 3 of which were later excluded during abstract scoring (NEJM) because they were not RCTs (figure 3). Of note, Annals IM had fewer RCTs (n=66) published during the study timeframe than the other journals in our study. Mean agreement among the 3 reviewers for checklist items was 84% in the pre-study run-in.

Assessment of reporting quality of the CONSORT for Abstracts checklist items

Overall, adherence to the CONSORT for Abstract checklist among journals varied (table 2). Analysis of all scored items showed an overall adherence of 67% (95% CI 66% to 68%). Adherence was lowest for the reporting of allocation concealment and random sequence generation. Conformity to the checklist was highest for reporting clear interpretations of the trial, stating trial objectives clearly, and including trial registration data. Adherence rates to 8 of the checklist items differed by >25% between journals. The Lancet had the highest overall adherence rate (78%; 95% CI 76% to 80%), whereas NEJM had the lowest (55%; 95% CI 53% to 57%).

Table 2

Adherence by checklist item

Variable	Observations	Mean	95% CI
Title	463	0.79	0.75 to 0.83
Trial design	463	0.51*	0.44 to 0.55
Eligibility criteria	463	0.76*	0.72 to 0.80
Study setting	463	0.58*	0.54 to 0.63
Intervention	463	0.87	0.83 to 0.90
Objective	463	0.96*	0.93 to 0.97
Primary outcome	463	0.91	0.88 to 0.93
Random sequence generation	463	0.19	0.15 to 0.22
Allocation concealment	463	0.08	0.06 to 0.11
Blinding	228	0.60	0.53 to 0.66
Number of randomised per group	463	0.61*	0.56 to 0.65
Number of analysed per group	463	0.44*	0.39 to 0.48
Outcome stated	463	0.83	0.79 to 0.86
Effect size/precision	463	0.81	0.78 to 0.85
Harms or side effects	463	0.50*	0.45 to 0.54
Interpretation stated	463	0.99	0.98 to 1.00
Trial registration	463	0.96	0.94 to 0.98
Source of funding	463	0.53*	0.48 to 0.57
Overall adherence	8099	0.67	0.66 to 0.68

*χ2 test for homogeneity among journals was statistically significant (p<0.001).

When comparing compliance among the individual journals by checklist items, NEJM lagged behind in more categories (five) than all other journals combined, but led the other journals in reporting of harms. The BMJ and JAMA lagged behind the other journals in reporting the funding source of the study. Overall adherence rates displayed substantial heterogeneity among journals. The Lancet had the highest overall adherence rate, whereas NEJM had the lowest (table 3).

Table 3

Adherence to checklist items by individual journal

Variable	NEJM	Annals IM	The Lancet	The BMJ	JAMA
Title (%)	9*	95	97	100	98
Trial design	30*	70	61	59	39
Eligibility criteria	57	77	83	84	78
Study setting	18*	74	52	85	67
Intervention	78	88	92	88	87
Objective	81	100	99	100	99
Primary outcome	90	86	95	90	90
Random sequence generation	0	21	62†	7	3
Allocation concealment	0	17	22	3	0
Blinding	24*	85	80	73	35*
Number of randomised per group	34*	53	77	66	69
Number of analysed per group	32	33	65	43	41
Outcome stated	85	73	88	74	93
Effect size/precision	77	85	80	81	85
Harms or side effects	69†	38	66	28	43
Interpretation stated	98	100	100	98	100
Trial registration	97	89	99	95	97
Source of funding	98	76	94	3*	2*
Overall adherence	55	70	78	65	63

*Journals lagged the majority by 25% or more (p<0.001) for the category.

†Journals led the majority by 25% or more (p<0.001) for the category.

Discussion

This descriptive, cross-sectional analysis examining adherence to the CONSORT for Abstracts checklist in the 5 highest impact general medical journals from 2011 to 2014 showed that the overall adherence was 67%, with markedly lower adherence to individual checklist items (down to 0% for some items in some journals) and substantial variability across journals. Reporting of allocation concealment and random sequence generation in the abstract text was uncommon (<25%) across all journals except The Lancet. The Lancet showed the highest rate of overall adherence and NEJM the lowest (78% and 55%, respectively), though these differences did not meet our prespecified criteria for a meaningful difference of 25%.

Similar to prior work, we found that incomplete adherence to abstract reporting guidelines persists,12 13 27–38 particularly on domains known to influence study results (eg, allocation concealment and blinding).12 13 15 Our study improved on previous studies as a large descriptive, cross-sectional study with a larger sample (n=463) of recently published abstracts in high-impact journals where results receive the highest attention among the clinical research and practising community. This allowed us to show with statistical significance the heterogeneity across the journals. By examining abstracts published between 2011 and 2014, this study provides an updated view of the state of adherence to reporting guidelines since Ghimire et al,12 which evaluated abstracts published in 2010. Although comparison with Ghimire's work suggests modest improvement in areas such as blinding, our study showed that adherence is still suboptimal (<60% for many items). There is also wide variation in reporting between journals, indicating an opportunity for standardising abstract reporting across the medical literature. Hopewell et al4 examined the impact of editors' implementation of the CONSORT for Abstracts checklist, and their results suggested that effective application of the guidelines led to improved reporting of RCT abstracts. They showed that active implementation of the guidelines led to immediate improvement in the mean number of reported checklist items. The call for improvement in the reporting of RCTs and their abstracts is not new.12 28–38 Indeed, many high-impact journals have endorsed the use of the CONSORT Statement and the CONSORT for Abstracts checklist.9 Various iterations have been created in an effort to improve reporting across a variety of research venues.36

Journals whose editors endorse and enforce the checklist show evidence of improved abstract reporting.13 The five high-impact general medical journals examined in our study have endorsed the use of CONSORT as well as the CONSORT for Abstracts checklist, either through their instructions to authors, inclusion on the CONSORT website or both (table 1), although some were more explicit than others. Suboptimal adherence would seem to imply that what is lacking is enforcement. It would be expected that journals of the calibre featured in this study should be better able to enforce guidelines given a presumably more robust editorial staff and a more rigorous copy editing process. Their failure to enforce the guidelines would suggest that further steps are necessary to maintain adherence. One suggestion would be improved communication of expectations by making the CONSORT for Abstracts checklist a more obvious requirement. Also, our inter-rater agreement on our run-in data was only 84%, which may indicate a need to make the CONSORT checklist less ambiguous for authors, peer reviewers and editors in order to achieve improved adherence.

Our study had several potential limitations. First, our study’s conclusions may lack applicability to journals not included in our analysis. Second, our study did not include analysis of temporal trends within our timeframe. If such trends exist, it may be misleading to identify one journal as lagging behind the others if the rates of improvement also differed between journals. Third, we considered all checklist items to be of equal importance, but experts may differ on the relative importance of each item. Finally, our inter-rater agreement averaged only 84%.

Our study's strengths included robust and reproducible data extraction, blinding reviewers to the journal, randomly assigning articles to reviewers and inclusion of studies from a time period (2011–2014) that provided adequate opportunity for implementation of the CONSORT for Abstracts since its publication in 2008.

In conclusion, the CONSORT for Abstracts is a valuable tool for improving transparency of reporting of clinical trial results. However, our findings indicate a need for systematic editorial and reviewing processes to improve adherence to these guidelines and the transparency of abstract reporting in high-impact medical journals. If we are going to realise the full potential of the CONSORT for Abstracts checklist in improving the quality of abstract reporting, it is critical that editors, peer reviewers and authors commit to its conscientious application.