The critical appraisal of randomized controlled trials published in an Indian journal to assess the quality of reporting: A retrospective, cross-sectional study.

BACKGROUND: Although randomized controlled trials (RCTs) are the highest levels of evidence, they might not necessarily be of good quality. Hence, RCTs should always be appraised critically. Critical appraisal is the corroboration of evidence by methodically studying its validity, reliability, and applicability.
OBJECTIVE: The primary objective of this study was to do a critical appraisal of the RCTs published in Indian Journal of Pharmacology (IJP) from 2011 to 2016. The secondary objective was to scrutinize how adequately the published RCTs adhere to the Consolidated Standards of Reporting Trials (CONSORT) declaration.
MATERIALS AND METHODS: The present study included all RCTs published as full-text articles in IJP from January 2011 to December 2016. The identified RCTs were critically appraised using the critical appraisal checklist based on CONSORT 2010 guidelines and its extensions.
RESULTS: According to this analysis, 75% (95% confidence interval [CI]: 0.56-0.87) of the articles had given details about the sample size calculation. Nearly 89.29% (95% CI: 0.72-0.96) of the articles described the method for generating random allocation sequence, but only 35.71% (95% CI: 0.20-0.54) of the articles described allocation concealment method. Almost 35.71% (95% CI: 0.20-0.54) of the trials reported results as per the principle of the intention to treat (ITT). Nearly 21.43% (95% CI: 0.10-0.39) of the studies reported CIs in the present study.
CONCLUSION: Allocation concealment method, analysis of the data based on the ITT principle, and reporting CIs were found to be underreported in this study. There should be more emphasis on reporting of allocation concealment, ITT analysis, and CI. Copyright:

INTRODUCTION

One of the most important skills a physician needs in the era of evidence-based medicine is the skill to scrutinize scientific publication critically. Critical appraisal is a systematic way of reading, comprehending, elucidating, and pinpointing the limitations of and determining the adequacy of the results of scientific publications. Critical appraisal weighs up how valid, reliable, and valuable the research will be. It requires thoroughly scrutinizing research evidence to evaluate its validity, results, relevance, impact, and applicability before coming to any conclusion. Critical appraisal encourages sound decision-making based on the best available evidence. It helps us determine how accurate a piece of research is (validity), how genuine the result is (reliability), and how pertinent it is to our patient (applicability).[1]

In general, randomized controlled trials (RCTs) provide authentic results that could apprise future research or clinical practice.[23] However, trials carried out with insufficient methodological approaches are associated with inflated treatment effects.[2] There is ample evidence in the public domain to prove that the standard of reporting of published RCTs is not optimum.[24] The Consolidated Standards of Reporting Trials (CONSORT) statement helps in the comprehensive and unambiguous reporting of trials and facilitates their critical appraisal and analysis.[4] Clinicians must rely on the scientific publications to keep themselves up-to-date on recent developments on new therapies as well as for new information on old therapies. RCTs might not be necessarily of good quality, hence they should always be appraised critically.[1] Moreover, very less information is available about the reporting quality of RCTs published in Indian journals.

Objective

The primary objective of this study was to do a critical appraisal of the RCTs published in Indian Journal of Pharmacology (IJP) from 2011 to 2016. The secondary objective was to scrutinize how adequately the published RCTs adhere to the CONSORT statement.

MATERIALS AND METHODS

Study design

This was a cross-sectional and retrospective study. It was based on the critical appraisal and assessment of reporting quality of published RCTs in IJP as per the CONSORT statement. This research relied exclusively on information freely available in the public realm, hence ethical sanction was not required.

Study selection

The present study included all RCTs published as full-text articles in IJP from January 2011 to December 2016.

Eligibility criteria

Inclusion criteria

Studies were included if they were described within the paper as a RCT or claimed to use random assignment for participants. Studies were only eligible if they were controlled trials with two comparators.

Exclusion criteria

Animal experiments, systematic review/meta-analyses, pharmacoeconomic studies, drug safety studies, pharmacokinetic/pharmacodynamic studies, drug utilization studies, and cross-sectional studies were excluded. Cohort studies and case series were also excluded. RCTs published as short communication and letters to the editor were not included in the present analysis because of brief information and word limitation.

Data extraction

The eligible articles were identified by screening of titles, abstracts, and methodology.

Assessment of reporting quality

The identified RCTs were critically appraised using the CONSORT statement. A critical appraisal checklist was prepared based on the CONSORT 2010 guiding principle and its extensions [Table 1].[5] The critical appraisal of a RCTs published in IJP with reference to its methodology was done to assess its validity, reliability, and applicability.

Table 1

Critical appraisal checklist for randomized controlled trials

Criteria	Yes	No	NA	95% CI
Title
Identification as a randomized trial in the title	21 (75)	7 (25)	-	0.75 (0.56-0.87)
Abstract
Structured summary of trial design, methods, results, and conclusions	28 (100)	0	-	1 (0.87-1)
Introduction
Scientific background and explanation of rationale	28 (100)	0	-	1 (0.87-1)
Specific objectives or hypotheses	28 (100)	0	-	1 (0.87-1)
Materials and methods
Description of trial design (such as parallel and factorial) including allocation ratio	28 (100)	0	-	1 (0.87-1)
Eligibility criteria for participants	28 (100)	0	-	1 (0.87-1)
Settings and locations where the data were collected	25 (89.29)	3 (10.71)	-	0.89 (0.72-0.96)
The interventions for each group with sufficient details to allow replication, including how and when they were actually administered	28 (100)	0	-	1 (0.87-1)
Completely defined prespecified primary and secondary outcome measures, including how and when they were assessed	27 (96.43)	1 (3.57)	-	0.96 (0.82-0.99)
How sample size was determined	21 (75)	7 (25)	-	0.75 (0.56-0.87)
Method used to generate the random allocation sequence	25 (89.29)	3 (10.71)	-	0.89 (0.72-0.96)
Mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned	10 (35.71)	18 (64.29)	-	0.35 (0.20-0.54)
If blinding done, who was blinded after assignment to interventions (e.g., participants, care providers, and those assessing outcomes) and how	16 (57.14)	12 (42.86)	-	0.57 (0.39-0.73)
Statistical methods used to compare groups for primary and secondary outcomes	28 (100)	0	-	1 (0.87-1)
Results
Participant flow diagram	10 (35.71)	18 (64.29)	-	0.35 (0.20-0.54)
For each group, the number of participants who were randomly assigned, received intended treatment, and were analyzed for the primary outcome	26 (92.86)	2 (7.14)	-	0.92 (0.77-0.98)
For each group, losses and exclusions after randomization, together with reasons	26 (92.86)	2 (7.14)	-	0.92 (0.77-0.98)
Dates defining the periods of recruitment and follow-up	16 (57.14)	12 (42.86)	-	0.57 (0.39-0.73)
A table showing baseline demographic and clinical characteristics for each group	20 (71.43)	8 (28.57)	-	0.71 (0.52-0.84)
“Intention-to-treat” analysis	10 (35.71)	18 (64.29)	-	0.35 (0.20-0.54)
For each primary and secondary outcome, results for each group and the estimated effect size	28 (100)	0	-	1 (0.87-1)
Precision of effect size (such as 95%CI)	6 (21.43)	22 (78.57)	-	0.21 (0.10-0.39)
All important harms or unintended effects in each group	28 (100)	0	-	1 (0.87-1)
Discussion			-
Trial limitations, addressing sources of potential bias, imprecision, and, if relevant, multiplicity of analyses	19 (67.86)	9 (32.14)	-	0.67 (0.49-0.82)
Generalizability (external validity and applicability) of the trial findings	Not analyzed	Not analyzed	-
Interpretation consistent with results, balancing benefits and harms, and considering other relevant evidence	28 (100)	0	-	1 (0.87-1)
Other information
Registration number and name of trial registry	9 (32.14)	19 (67.86)	-	0.32 (0.17-0.50)
Where the full-trial protocol can be accessed, if available	0	28 (100)	-	0 (0-0.12)
Sources of funding and other support (such as supply of drugs), role of funders	8 (28.57)	20 (71.43)	-	0.28 (0.15-0.47)

NA=Not applicable, CI=Confidence interval

Statistical analysis

Descriptive statistics was utilized in this study. The lower and upper bounds of the 95% confidence interval (CI) for the proportions were calculated. The data were analyzed by means of the SPSS (Statistical Package for the Social Sciences), version 16; IBM Corporation, Chicago, Illinois, USA.

RESULTS

A total of 1102 articles published from January 2011 to December 2016 in the IJP were screened for eligibility. The process used to select potentially relevant studies for inclusion in our study is depicted in Figure 1. From the total of 1102 articles, 596 were omitted after screening of titles and abstracts. A total of 506 studies underwent further evaluation. Of these, 478were excluded after full-text review.

Figure 1

Flow diagram of citations through the retrieval and the screening process

Of the 28 included articles, 21 (75%, 95% CI: 0.56–0.87) mentioned “randomization” in the title. In 28 (100%, 95% CI: 0.87–1) articles, the abstract was structured; these articles described the scientific rationale, gave details of specific objectives or hypotheses, gave a description of trial design, and gave details of eligibility criteria for the participants. Twenty-five (89.29%, 95% CI: 0.72–0.96) articles gave details about setting and locations. Twenty-seven (96.43%, 95% CI: 0.82–0.99) articles defined outcome measures.

Twenty-one (75%, 95% CI: 0.56–0.87) articles mentioned how sample size was determined. Twenty-five (89.29%, 95% CI: 0.72–0.96) articles mentioned about the method for random allocation sequence generation. Ten (35.71%, 95% CI: 0.20–0.54) articles mentioned about allocation concealment method. Sixteen (57.14%, 95% CI: 0.39–0.73) articles were blinded studies. Twenty-eight (100%, 95% CI: 0.87–1) studies described statistical methods for outcome assessment.

Ten (35.71%, 95% CI: 0.20–0.54) articles gave participant flow diagram. Twenty-six (92.86%, 95% CI: 0.77–0.98) articles gave details for each group, including the number of randomly assigned participants. Sixteen (57.14%, 95% CI: 0.39–0.73) studies mentioned the dates of recruitment and follow-up. Twenty (71.43%, 95% CI: 0.52–0.84) studies gave a table displaying the baseline characteristics of each group. “Intention-to-treat (ITT)” analysis was done in ten (35.71%, 95% CI: 0.20–0.54) studies. The appraised effect size was mentioned in 28 (100%, 95% CI: 0.87–1) articles, but the precision of effect size (CI) was mentioned in only six (21.43%, 95% CI: 0.10–0.39) studies. Adverse effects in each group were mentioned in 28 (100%, 95% CI: 0.87–1) studies.

Trial limitation was addressed in 19 (67.86%, 95% CI: 0.49–0.82) studies. Generalizability could not be evaluated because it was challenging to objectively evaluate them. Registration number and name of trial registry were mentioned in nine (32.14%, 95% CI: 0.17–0.50) studies. Sources of funding were mentioned in eight (28.57%, 95% CI: 0.15–0.47) studies [Table 1].

DISCUSSION

The three keys of critical appraisal are validity, reliability, and applicability. When assessing validity, both internal and external validity should be assessed. When critiquing a study design and its subsequent findings, evaluation of the internal validity of the research paper is required. The aptness of the study results to support an association between the intervention and the outcome is internal validity. When assessing internal validity, emphasis is given to reporting of items such as randomization method, concealment of allocation, sample size calculation, baseline data reporting, estimated effect size precision, and ITT analysis reporting.[6]

In this study, it was found that 75% of the articles provide details about the sample size calculation. In a survey by Zhang et al., 2016, no trials reported about adequate sample size calculation. In the same study by Zhang et al., 2016, with another set of journals, only 12.5% of the trials described the sample size calculation.[7] In a study by Shi et al., 2014, only 0.4% of the trials reported adequate sample size.[8] Nearly 75%of the articles were identified as randomized in title in this study. In a study by Choi et al., 2014, only 10.7% of the articles mentioned “randomization” in their title.[3] In a study by Shi et al., 2014, only 0.3% of the trials reported “randomized control trial” in the title.[8] Almost 89.29% of the articles described method for generating random allocation sequence, but only 35.71% of the articles described allocation concealment methods in this study. In a study by Pratoomsoot et al., 2015, 47.89% of the RCTs reported the methods for generating the random sequence allocation, and 29.58% of the study implemented a random allocation sequence.[9] In a study by Zhai et al., 2015, the allocation sequence generation was described in 35.4% of the trials and concealment of allocation was described in 28.5% of the trials.[10] In a study by Zhang et al., 2016, 33.4% of the trials reported about generation of allocation sequence, but only 3.7% of the trials reported about concealment of allocation.[7] In a review of Dias et al., 2006, 49% of the trials were found not to have provided details of the randomization method and of all the trials assessed, only 10% had described adequate forms of allocation concealment.[11] In a survey by Chen et al., 2014, randomization and allocation concealment were reported in 67% and 49% of the RCTs, respectively.[12]

Only 35.71% of the trials have reported results as per the principles of ITT concept in this study. Aberrations from this principle can lead to biased results. In a study by Zhai et al., 2015, 23.8% of the trials used ITT analysis.[10] In a survey by Chen et al., 2014, 44% of the trials reported ITT analysis.[12] In the study by Partsinevelou and Zintzaras, 2009, 87.5% of the trials did not give any data about “ITT” analysis.[6] In this study, 57.14% of the studies reported adequate blinding and 71.43% of the studies reported baseline data. In a study by Pratoomsoot et al., 2015, approximately 31% of the RCTs clearly reported about blinding.[9] In a study by Zhai et al., 2015, blinding was reported in 57.0% of the trials.[10] Blinding and baseline characteristics were reported in 51% and 97% of the RCTs, respectively, in a study by Chen et al., 2014.[12] External validity means generalizability (representative of the study sample). As there is no validated scale to measure external validity, it was challenging to objectively evaluate them.[13]

Reliability of the result refers that if the study were conducted again, the result would be the same. It is usually interpreted as the accuracy of measurement. It discourses the treatment effect magnitude and the estimation of the treatment effect precision. The size of the treatment effect is often expressed as the average difference between groups on some objective outcome measures. CIs are the most common measures of precision. There are certain weaknesses of P values: not truly compatible with hypothesis testing, never meant to be the sole indicator of significance, and no consideration of effect size. There are many factors which influence P values such as sample size, chance, effect size, and statistical power. Comparatively more information on the precision of the inference is provided by the 95% CI.[14] Though all the studies reported P values in the study, only 21.43% of the studies reported CIs. Similarly, in an analysis of orthopedic research by Vavken et al., 2009, only 22% of the studies reported CIs.[15]

Limitation of this research is that RCTs from only one Indian journal (IJP) were included. Another limitation was small sample size. RCTs' quality was not evaluated directly as the data were not corroborated from the corresponding authors. This study focused on the methodological specifics, which may differ from the quality of the actual study. Assessing clinical trial protocols along with contacting the investigators for more data may ameliorate quality appraisal.

CONCLUSION

This study demonstrated that underreported items from the CONSORT statement included allocation concealment method and analysis of the data based on the “ITT” principle. Apart from that, reporting of CIs was found to be less. There should be stress on allocation concealment, ITT analysis, and reporting of the CI to augment the validity of the trials. The results of this study would aid in improved observance to the CONSORT statement. Meticulous reporting of trials will help clinicians to endorse a new therapy or modify therapies currently in practice.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.