Using the CONSORT statement to evaluate the completeness of reporting of addiction randomised trials: a cross-sectional review.

OBJECTIVES: Evaluate the completeness of reporting of addiction randomised controlled trials (RCTs) using the Consolidated Standards of Reporting Trials (CONSORT) statement.
SETTING: Not applicable. PARTICIPANTS: RCTs identified using a PubMed search of 15 addiction journals and a 5-year cross-section. OUTCOME MEASURES: Completeness of reporting.
RESULTS: Our analysis of 394 addiction RCTs found that the mean number of CONSORT items reported was 19.2 (SD 5.2), out of a possible 31. Twelve items were reported in <50% of RCTs; similarly, 12 items were reported in >75% of RCTs. Journal endorsement of CONSORT was found to improve the number of CONSORT items reported.
CONCLUSIONS: Poor reporting quality may prohibit readers from critically appraising the methodological quality of addiction trials. We recommend journal endorsement of CONSORT since our study and those previous have shown that CONSORT endorsement improves the quality of reporting. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Application of robust methodology, as recommended by the Cochrane collaboration.

15 addiction journals included over a 5-year period.

Cross-sectional design, limited to addiction journals, reduces generalisability of our findings.

Introduction

The completeness and clarity of reporting research studies is essential for readers to fully appreciate and evaluate a study’s methodological rigour. Complete reporting is also necessary to determine the applicability of findings to patient care. Sims et al 1 likened poor reporting to blinding readers when important methodological details or results are omitted from published reports. Moher et al argue that, “inadequate reporting borders on unethical practice when biased results receive false credibility”.2 Previous studies have found that clinical trial interventions are insufficiently reported to permit replication or to allow physicians to enact the intervention in the clinical setting.3 4 Others have found that the poor reporting of systematic reviews does not even permit the initial searches to be replicated.5 Thus, across the clinical research spectrum, reporting is variable, but often suboptimal and in need of improvement.

To address reporting deficiencies, researchers have developed reporting guidelines which provide best-practice guidance to study authors on reporting pertinent information for various study designs. The Consolidated Standards of Reporting Trials (CONSORT) statement6 is an evidence-based set of 25 items that provides specific guidance for reporting randomised trials and has an accompanying flow diagram to document the flow of participants throughout a trial. CONSORT has been widely adopted by 585 journals; over 50% of core medical journals listed in the Abridged Index Medicus on PubMed currently endorse or require CONSORT.7 We found only four addiction journals listed as endorsers on the CONSORT website.

In this study, we evaluate the completeness of reporting of addiction clinical trials, an area of study in which little is known about reporting practices. We used the CONSORT statement as the basis for this investigation, as CONSORT is widely recognised as the authoritative source for trial reporting. Results from this investigation will assist in identifying areas well reported within addiction trials and areas where improvements are needed. We also evaluate whether particular trials characteristics are associated with more complete reporting.

Methods

We conducted a cross-sectional study of published addiction clinical trials; therefore, our study was not subject to Institutional Review Board oversight as it did not meet the regulatory definition of human subjects research. For purposes of reporting, we followed the reporting guidelines for meta-epidemiological studies8 and, when relevant, the Preferred Reporting Items for Systematic Reviews and Meta-analyses guidelines.9

Bibliographic databases searches and journal selection

One investigator (MV) searched PubMed (which includes the MEDLINE collection) on 22 June 2018. This search was conducted to identify clinical trials published between 1 January 2013 and 31 December 2017 using PubMed’s Clinical Trial[ptyp] filter. This filter has been shown to maximise sensitivity to ensure relevant studies are not excluded.10 Journals listed in the addiction category of Google Scholar metrics were selected based on their h5-index. Beginning with the journal with the highest h5-index, we conducted PubMed searches to see whether each journal had published at least 10 clinical trials. We continued this process until 15 journals were selected.

We deployed the final search string as follows: ((((((((((((((“Addiction (Abingdon, England)“[Journal]) OR (“Drug and alcohol dependence’”[Journal])) OR (“Nicotine & tobacco research : official journal of the Society for Research on Nicotine and Tobacco”[Journal])) OR “Addictive behaviors”[Journal]) OR (“Alcoholism, clinical and experimental research”[Journal])) OR “Psychology of addictive behaviors : journal of the Society of Psychologists in Addictive Behaviors”[Journal]) OR “Addiction biology”[Journal]) OR (“Journal of studies on alcohol and drugs”[Journal])) OR “The International journal on drug policy”[Journal]) OR (“Drug and alcohol review”[Journal])) OR (“Alcohol and alcoholism (Oxford, Oxfordshire)“[Journal])) OR “Journal of substance abuse treatment”[Journal]) OR “Alcohol (Fayetteville, N.Y.)“[Journal]) OR “The American journal on addictions”[Journal]) OR “Substance use & misuse”[Journal] AND (Clinical Trial[ptyp] AND (“2013/01/01”[PDat] : “2017/12/31”[PDat])).

Screening records for eligibility

To be eligible for inclusion, a study must have reported the use of a randomised clinical trial design and address one of the following related to drugs, alcohol or tobacco: (1) addiction prevention, (2) stabilisation following excessive use of a substance (drugs, alcohol or tobacco), (3) relapse prevention and (4) recovery maintenance. For purposes of this study, the National Institutes of Health definition of clinical trial was used to determine inclusion, which involves the prospective placement of participants to an experimental condition using randomisation methods and testing the effects of an intervention.11 We eliminated other study types, including observational study designs (eg, case–control and cohort studies), systematic reviews and meta-analyses, and case reports. We also excluded letters to the editor, other editorials, commentaries and perspectives articles.

Two investigators (SJ and HW) screened all studies for eligibility in an independent, blinded fashion which is consistent with our previous investigations.12–14 We used Rayyan, an online systematic review application, to screen PubMed records for eligibility with the blinding feature turned on. After the initial screening process was completed, the two investigators held a consensus meeting to review the screening decisions and resolve disagreements by discussion.

Data extraction and scoring

Two investigators (SJ and HW) performed blinded, double data extraction. As with screening, a consensus meeting was held after completion of the data extraction process to review and resolve discrepancies. The following items were extracted from each article: journal, year of publication and funding source. We next evaluated each item of the CONSORT Statement, which can be found in table 1. For each included journal, we manually reviewed the Instructions for Authors page (or equivalent) to determine if CONSORT was endorsed.

Table 1

Included journals (ordered by Google Scholar ranking) and mean adherence to CONSORT items (n=31)

Journal	Mean (SD)
Addiction (n=65)	24.8 (3.2)
Drug and Alcohol Dependence (n=73)	19.9 (4.2)
Nicotine & Tobacco Research (n=61)	18.8 (4.4)
Addictive Behaviors (n=37)	15.4 (4.8)
Alcoholism, Clinical and Experimental Research (n=32)	19.7 (5.7)
Psychology of Addictive Behaviors (n=19)	13.7 (4.5)
The International Journal on Drug Policy (n=1)	Mean not calculated
Addiction Biology (n=3)	Mean not calculated
Journal of Substance Abuse Treatment (n=52)	18.3 (3.7)
Alcohol and Alcoholism (n=15)	19.9 (4.4)
Journal of Studies on Alcohol and Drugs (n=12)	15.6 (4.6)
Drug and Alcohol Review (n=4)	Mean not calculated
The American Journal on Addictions (n=13)	16.8 (4.4)
Substance Use & Misuse (n=7)	Mean not calculated

We planned a multiple regression analysis to investigate the association between funding source, journal and journal endorsement of CONSORT on individual trial CONSORT scores. This regression analysis was thwarted because of a large predominance of public funding and collinearity. Therefore, we conducted an independent sample t-test to compare the mean CONSORT score for trials published in CONSORT-endorsing journals and non–CONSORT-endorsing journals. We further conducted a one-way ANOVA, with Bonferroni adjustments, to compare trials related to drug, alcohol, tobacco or mixed (eg, co-occurring alcohol and tobacco) addictions. All analyses were conducted using Stata 15.1.

Results

Our database search returned 1546 records, of which 394 RCTs were eventually included (figure 1). A full list of included RCTs can be found online (https://osf.io/cy5j3/). The 394 RCTs were most often published in Drug and Alcohol Dependence (n=73), Addiction (n=65), and Nicotine and Tobacco Research (n=61). Included RCTs were most often funded by public sources (eg, government) (n=315).

Figure 1

Flow diagram of included and excluded studies. RCT, (drug, alcohol, and tobacco) randomised controlled trial.

CONSORT compliance

Figure 2 presents a histogram that summarises the distribution of trials obtaining particular CONSORT compliance scores. The mean number of CONSORT items reported was 19.2 (SD 5.2), out of a possible 31. The adherence to CONSORT for all included trials, stratified by journal, is shown in table 1. Twelve items were reported in <50% of RCTs (table 2), including such items as where a protocol can be accessed (item 24) and sample size estimations (item 7a). Similarly, 12 items were reported in >75% of RCTs, including important items like sources of funding (and role of funders) (item 25), eligibility criteria (item 4a), and a balanced interpretation of harms and benefits (item 22).

Figure 2

Histogram of trial adherence to CONSORT.

Table 2

Adherence to each CONSORT item

CONSORT Item	n/N	%
1a. Identification as a randomised trial in the title?	222/394	56.3%
1b. Structured summary of trial design, methods, results and conclusions? (See CONSORT for Abstracts)	312/394	79.2%
2a. Scientific background and explanation of rationale?	392/394	99.5%
2b. Specific objectives or hypotheses?	389/394	98.7%
3a. Description of trial design (such as parallel, factorial) including allocation ratio?	110/394	27.9%
4a. Eligibility criteria for participants?	391/394	99.2%
4b. Settings and locations where the data were collected?	346/394	87.8%
5. The interventions for each group with sufficient details to allow replication, including how and when they were actually administered?	392/394	99.5%
6a. Completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed?	212/394	53.8%
7a. How sample size was determined?	154/394	39.1%
8a. Method used to generate the random allocation sequence?	249/394	63.2%
8b. Type of randomisation; details of any restriction (such as blocking and block size)?	212/394	53.8%
9. Mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned?	126/394	32.0%
10. Who generated the random allocation sequence, who enrolled participants and who assigned participants to interventions?	89/394	22.6%
11a. If done, who was blinded after assignment to interventions (eg, participants, care providers, those CONSORT 2010 checklist Page 2 assessing outcomes) and how?	128/394	32.5%
12a. Statistical methods used to compare groups for primary and secondary outcomes?	226/394	57.4%
12b. Methods for additional analyses, such as subgroup analyses and adjusted analyses?	158/394	40.1%
13a. For each group, the numbers of participants who were randomly assigned, received intended treatment and were analysed for the primary outcome?	303/394	76.9%
13b. For each group, losses and exclusions after randomisation, together with reasons?	282/394	71.6%
14a. Dates defining the periods of recruitment and follow-up?	171/394	43.4%
15. A table showing baseline demographic and clinical characteristics for each group?	342/394	86.8%
16. For each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups?	302/394	76.6%
17a. For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% CI)?	192/394	48.7%
18. Results of any other analyses performed, including subgroup analyses and adjusted analyses, distinguishing pre-specified from exploratory?	152/394	38.6%
19. All important harms or unintended effects in each group? (See CONSORT for Harms)	133/394	33.8%
20. Trial limitations, addressing sources of potential bias, imprecision and, if relevant, multiplicity of analyses?	361/394	91.6%
21. Generalisability (external validity, applicability) of the trial findings?	237/394	60.2%
22. Interpretation consistent with results, balancing benefits and harms, and considering other relevant evidence?	393/394	99.7%
23. Registration number and name of trial registry?	176/394	44.7%
24. Where the full trial protocol can be accessed, if available?	35/394	8.9%
25. Sources of funding and other support (such as supply of drugs), role of funders?	386/394	98.0%

Our pre-planned multiple regression investigating the association between journal, consort endorsement and funding source on adherence to CONSORT was thwarted because of the large disparities in funding source group sizes (public: n=315/394, 79.9%) and collinearity of journal adherence to CONSORT as a predictor. Therefore, we conducted an independent sample t-test comparing the mean CONSORT adherence for trial published in CONSORT-endorsing journals and trial published in non–CONSORT-endorsing journals. The mean difference between the two groups was −4.5 (95% CI −5.49 to 3.55) items, indicating that trials published in CONSORT-endorsing journals adhere to significantly more items than other trials. On comparison of RCTs related to drug, alcohol, tobacco or mixed addictions, we found that drug dependence RCTs (n=111) had the highest mean CONSORT score (20.0, SD 4.7) and alcohol dependence RCTs (n=117) had the lowest mean CONSORT score (18.2, SD 5.6). The mean difference between these two cohorts was 1.9 CONSORT items and was statistically significant (p=0.04). No other mean differences were significant.

Discussion

In this investigation of trial reporting, 12 CONSORT items were reported less than 50% of the time in RCTs published in addiction journals. Previously, it has been shown that low-quality studies may be incorporated into meta-analyses, thus biasing downstream treatment effects.15 Further, bias associated with key trial characteristics, such as allocation concealment, has been shown to exaggerate trial summary effects.16–18 Additional forms of bias, such as selective outcome reporting bias,19–22 are prevalent across biomedicine. Consequently, poor reporting quality may render readers, who are likely aware of at least one form of bias prevalent in RCTs, incapable of critically appraising the validity of addiction RCT results. However, our study also showed that journal adherence to CONSORT was associated with better reporting of RCT items. It is possible that this result is confounded by journal impact factor, but we are reassured of the effect of CONSORT endorsement by previous studies. A systematic review of 53 published studies found that overall, reporting quality in RCTs is suboptimal but that journal endorsement of CONSORT is an intervention that has proven benefit.23 Namely, journal endorsement of CONSORT greatly improved the reporting of allocation concealment, scientific rationale for the trial, sample size estimations and method of sequence generation.

Other than our study, evaluations of the completeness of reporting of RCTs in addiction science have been limited. Our study found that mean CONSORT adherence was approximately two-thirds of included CONSORT items and that journal endorsement of CONSORT resulted in higher mean CONSORT adherence by included trials. One previous study24 investigated the completeness of reporting of 127 alcohol treatment outcome RCTs. Trials published in Addiction, Alcohol and Alcoholism, Drug and Alcohol Dependence, and Journal of Consulting and Clinical Psychology—all CONSORT-endorsing journals—were compared with trials published in Alcoholism: Clinical and Experimental Research, Journal of Studies on Alcohol and Drugs, Journal of Substance Abuse Treatment, and Psychology of Addictive Behaviors—non-endorsing journals. Authors reported that improvements in trial reporting over time were noted in both groups; however, endorsing journals experienced improvements over time for reporting random assignment, masking, participant flow and statistics. In contrast, the trend over time for non-endorsing journals was not statistically significant for any of these item subgroups. Results from this study formed the basis for a policy change at Alcohol: Clinical and Experimental Research 25 that began requiring clinical trialists to adhere to CONSORT for trial reporting. Two narrative reviews26 27 accompanied the editorial and discussed the importance of improved trial reporting and design for alcohol use disorders. A 2019 investigation of the Instructions for Authors sections of 88 addiction journals found that less than a quarter of the journals endorsed adherence to various reporting guidelines, with CONSORT endorsement being highest at only 14.8% of journals.28 In response to these findings, these authors expressed, “there is an urgent need to improve the author instructions segment of addiction science journals so that the process of research dissemination can occur more effectively”.28

In our study, trials published in Addiction—a CONSORT-endorsing journal—received the highest composite scores on overall reporting. We speculate two possibilities here. First, Addiction provides explicit directions for research reporting in its instructions to authors. Multiple reporting guidelines are mentioned by name. The EQUATOR Network, the international establishment devoted to the advancement of improving study reporting, is also referenced. Previous studies have confirmed that when journals provide detailed guidance to authors, quality of research reporting is improved.29 30 Second, Addiction encourages authors to use Penelope (www.penelope.ai), a tool created by the EQUATOR Network, to perform an automated inspection of a manuscript on reporting compliance with various reporting guidelines. Penelope generates a report to authors that assesses structure, declarations, statistics, referencing and other common reporting errors prior to manuscript submission to the journal. While we are unaware of any published studies that evaluate Penelope, we surmise that its simplicity of use and quick feedback may prompt investigators to make alterations to their manuscripts prior to journal submission. Empirical evaluations on Penelope are recommended.

While in this discussion we focus on the issues of trial reporting at large, our results confirm that specific items are particularly problematic. The CONSORT explanation and elaboration document outlines in detail the rationale and importance for each item.31 Many items relate to reporting methodological information, such as randomisation (items 8a, 8b, 9, 10) and blinding (item 11a). None of the randomisation or blinding items were reported at a high rate, with the most reported item relating to the method of randomisation (8a) and the least reported item relating to who was blinded (11a). Other items relate to the availability of published protocols (item 24) or trial registration numbers (item 23) that can be used to inspect the possibility of biases such as selective outcome reporting or questionable trial alterations. These items were also poorly reported, especially item 24 regarding protocols. Only 44.7% (176/394) of included RCTs provided a registration number, while 8.9% (35/394) directed readers to a protocol.

Our study is subject to strengths and limitations. Regarding strengths, we applied gold standard systematic review methodology recommended by the Cochrane Collaboration32 for study screening and data extraction—both were done in a blinded, duplicate fashion. Furthermore, we included a large number of journals relative to other investigations that restricted their samples to five or so journals. We also included a larger sample of trials than similar investigations across other clinical disciplines. Taken together, these strengths lend credibility to the validity of our data and, thus, the robustness of our conclusions. Regarding limitations, our study design is cross-sectional. Our results should be interpreted descriptively, and caution should be taken when generalising our findings outside the scope of our sample. Additionally, we only looked at articles published in addiction journals, which does not completely encompass all addiction trials published. This may have led to an underestimation of CONSORT adherence as other trials may have been published in journals with stricter reporting requirements. It is also possible that confounding factors may influence our results rather than CONSORT endorsement. We did not look particularly at funding source, and funders—such as the National Institutes of Health—may have their own particular reporting requirements outside of CONSORT that influenced results.33 Some CONSORT items are subjective and may be interpreted differently than we interpreted them. While we applied the greatest standardisation possible, this subjectivity should be carefully considered when interpreting results from our study.

In conclusion, our study found inconsistencies in the completeness of reporting of RCTs published in addiction journals. To ensure that all trial evidence generated for the prevention, treatment or management of addiction can be critically appraised by all stakeholders, we recommend all addiction journals require trial authors to consult the CONSORT checklist prior to submission. Turner et al’s23 Cochrane review found no evidence that journal endorsement hinders the completeness of RCT reporting. Further, the authors of this review argue that journals are not sending clear messages to authors and that the fidelity of endorsement of reporting guidelines by journals has been weak. Explicit guidance and follow-up from addiction journals may, thus, lead to the publication of RCTs which are better reported, better interpreted and better implemented in the clinical setting.