Assessment of adherence to the statistical components of consolidated standards of reporting trials statement for quality of reports on randomized controlled trials from five pharmacology journals.

BACKGROUND: The Consolidated Standards of Reporting Trials (CONSORT) statement is a device to standardize reporting and improve the quality of controlled trials. However, little attention is paid to the statistical components in the CONSORT checklist. The present study evaluates the randomized controlled trials [RCTs] published in five high impact pharmacology journals with respect to its statistical methods.
METHODS: Randomized Controlled Trials [RCTs] published in the years 2013 & 2014 in five pharmacology journals with high impact factor, The Journal of Clinical Pharmacology (JCP), British Journal of Clinical Pharmacology (BJCP), European Journal of Clinical Pharmacology (EJCP), Journal of Pharmacology & Pharmacotherapeutics (JPP) and Indian Journal of Pharmacology (IJP) were assessed for adherence to the statistical components of CONSORT statement.
RESULTS: Of the 174 RCTs analysed, 103 described the method of sample size calculation. Of the five journals, maximum reports in JCP (34/50) and minimum in IJP (13/31) adhered to the CONSORT checklist [item 7a-sample size calculation]. Most reports mentioned the statistical methods used for analysis of data. (171/174) as per the checklist [item 12=statistical methods used]. Analysis of variance (ANOVA) was the most commonly used test (88/174). The software used for statistical analysis was mentioned in 111 RCTs and SPSS was used more frequently (58/111). The exact p value was stated in 108 reports. Certain errors in statistical analysis were also noted (40/174).
CONCLUSION: These findings show inconsistencies and non- adherence to the statistical components of the CONSORT statement especially with respect to sample size calculation. Special attention must be paid to the statistical accuracy of the reports.

INTRODUCTION

Reports of randomized controlled trials (RCTs) are the “gold standard” by which decisions about treatment efficacy and effectiveness are made by health professionals and others.[1] Since RCTs, more than any other methodology, can have a powerful and immediate impact on patient care, the published report should be of the highest possible standards.[2] Ideally, accurate and complete reporting concerning the design, conduct, analysis, and generalizability of the trial should be conveyed.[3] Adequate reporting is essential because the readers need and deserve to know the quality of methods to make informed judgments regarding the validity of a clinical trial.[1234] Evidence indicates that reports of low-quality RCTs, compared with reports of higher-quality ones, overestimate the effectiveness of interventions by about 30% across a variety of health care conditions.[567] The scientific world worries, of course, that sloppy reporting reflects sloppy methods, and that with sloppy methods come biased results.[8]

A flurry of activity to address the defect and standardize reporting of RCTs culminated in the most prominent guideline, the consolidated standards of reporting trials (CONSORT) in 1996 which has been revised twice, in 2001 and 2010.[678] The objective of the CONSORT is to provide a guideline for authors to standardize and improve the reporting of trials.[9] Preliminary appraisals suggest that the use of CONSORT items is associated with improvements in the quality of reports published.[10] However, little attention is paid to the statistical components in the CONSORT checklist. Adherence to the CONSORT requires continual appraisal for improving the precision and transparency of published trials. The aim of this study was to assess the adherence to the statistical components of CONSORT 2010 statement for quality of reports on RCTs from five pharmacology journals.

METHODS

Data sources

We selected five pharmacology journals in descending order of impact factor as per the Thomas Reuters impact factor list as follows: The Journal of Clinical Pharmacology (JCP), British Journal of Clinical Pharmacology (BJCP), European Journal of Clinical Pharmacology (EJCP), Journal of Pharmacology and Pharmacotherapeutics (JPP), and Indian Journal of Pharmacology (IJP).

We conducted a MEDLINE/PubMed search to identify all RCTs published between January 2013 and December 2014 with the following search strategy: “Journal of Clinical Pharmacology” (Jour) OR “British Journal of Clinical Pharmacology” (Jour) OR “European Journal of Clinical Pharmacology” (Jour) OR “Journal of Pharmacology and Pharmacotherapeutics” (Jour) OR “Indian Journal of Pharmacology” (Jour) AND RCT (ptyp) AND MEDLINE (sb) AND (“2013/01/01 [PDAT]”: “2014/12/31[PDAT]”).

Study selection

RCTs of preventive and therapeutic interventions were selected. We included reports in which the allocation of participants to interventions was described as random, randomly allocated, randomized or randomization. Other study designs such as observational studies, economic analyses on RCTs, quasi-randomized trials, cluster randomized trials, diagnostic or screening tests, follow-up studies of previously reported RCTs, editorials, reviews, case reports, and letters were excluded.

Data extraction

The reviewers underwent training in evaluating RCTs using the CONSORT 2010 statement with special reference to its statistical components. One reviewer extracted data from all included papers. A second reviewer independently checked a random sample of 30% of the papers. Discrepancies were resolved where possible by discussion, and the sample results were compared with the full results using Kappa scores.

The following items were included in the checklist: Name, issue and volume of the journal, method of sample size calculation (item 7a of CONSORT checklist), alpha and beta error in calculation, percentage of drop-outs considered for sample size determination, statistical methods used to compare primary and secondary outcomes (item 12a of CONSORT checklist), statistical software used for analysis, and use of exact P value.

Data analysis

Data for descriptive statistics were described as frequencies and percentages. The data were analyzed using Microsoft Excel version 2013.

RESULTS

Among the 174 RCTs included for the study, 28.7% (50/174) were published in JCP followed by 23% (40/174) from BJCP, 19.5% (34/174) from EJCP, 17.8% (31/174) from IJP, and 10.9% (19/174) from JPP.

Of the 174 reports in the study, 103 (59.2%) adhered to item 7a of the CONSORT checklist and reported the method of sample size determination. A journal-wise distribution is given in Table 1.

Table 1

Journal-wise distribution of reports mentioning method of sample size determination

The method used for sample size determination was most commonly based on the primary outcome of the study in question 72.8% (75/103) followed by previous studies 21.4% (22/103) and very few reports 0.05% (6/103) used feasibility as a method. A detailed distribution is given in Table 2.

Table 2

Methods of sample size determination used in individual journals

The assumption of alpha and beta error was mentioned in 40% (70/174) reports. The alpha error was assumed at 5% in all the reports. Out of these 70 reports, the beta error was assumed as 80% in 48 of 70 reports and 90% in 22 of 70 reports.

28.7% (50/174) reports mentioned the drop-out rate for calculation of sample size. Out of these 50 reports, the drop-out rate was assumed to be 25% in 23 reports and 20% in 27 reports.

Most reports 98.3% (171/174) mentioned the test used for statistical analysis. Two articles in IJP and one article in BJCP did not mention the test used for statistical analysis in the methodology section in spite of using a test for statistical analysis.

The most commonly used tests for statistical analysis have been outlined in Table 3.

Table 3

Tests used for statistical analysis

Analysis of variance

The software used for statistical analysis was mentioned in 63.8% (111/174) reports. Out of these 111 reports, Statistical Package for the Social Sciences (SPSS) was used most frequently 52.3% (58/111) followed by same-area sales 27.9% (31/111) then GraphPad 17.1% (19/111) and others 2.7% (3/111).

The actual P value was reported in 62.0% (108/174) reports.

DISCUSSION

In this study, we have observed that the majority of RCTs reported in high impact factor pharmacology journals did not adhere completely to the statistical components of CONSORT guidelines. A study by Ayatolahi et al.[1112] reported that 6.2% reports mentioned the method of sample size calculation. In a study done by Nojomi et al.[7] only 30% of evaluated articles reported the method of sample size calculation. The method of sample size determination (item 7a of CONSORT checklist) was mentioned in 59.2% reports in our study. This indicated an increase in adherence to the CONSORT checklist of RCTs in our study compared to the previous studies. To the best of our knowledge, there are no studies evaluating only the statistical components of the CONSORT statement. The method of sample size calculation was most commonly based on the primary outcome of the study followed by previous studies and feasibility. Most RCTs should mention the drop-out rate for the calculation of sample size. However, in our study we found that only 28.7% articles reported the drop-out rate.

The reporting of statistical methods used to compare groups for primary and secondary outcomes was 98.2%. Similar results (96.3%) were obtained in a study done by Nojomi et al.[7] The most common test used for statistical analysis was analysis of variance followed by Mann–Whitney U-test and then Student's t-test and others. None of the studies have reported the most commonly used software for statistical analysis. In our study, SPSS Software was used most frequently.

We have assessed RCTs published only over the last 1-year from five pharmacology journals. We reviewed RCTs published in high impact factor pharmacology journals that have universal acceptance in the pharmacology research community. To the best of our knowledge, no studies have specifically evaluated the adherence to the statistical components of the CONSORT statement.

CONCLUSIONS

Thus, the finding from this study shows that the quality of reports on RCTs from five pharmacology journals did not adhere completely to the statistical components of the CONSORT 2010 statement. CONSORT is an evolving guideline not considered to be an absolute standard.[13] However, improvements in the quality of RCT reports can be expected by adherence to the existing standards and guidelines.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.