Assessing the quality of randomized controlled urological trials conducted by korean medical institutions.

PURPOSE: To assess the quality of randomized controlled urological trials conducted by Korean medical institutions.
MATERIALS AND METHODS: Quality assessment was conducted by using the Jadad scale; in addition, the van Tulder scale and the Cochrane Collaboration risk of bias tool were used as individual indices. All assessments were performed by two reviewers. If the outcomes differed, the two reviewers and a third reviewer adjusted the discrepancy in the results through discussion. Starting from 1986, a quality analysis of randomized controlled trials (RCTs) was conducted in 1-year and 5-year units. The quality assessment was conducted by subject, type of intervention, presence of double blinding, presence of funding, and review by an Institutional Review Board (IRB).
RESULTS: Whereas the number of RCTs published has gradually increased, there was no significant difference in the quality of the RCTs according to publication year. Drug studies, double-blind studies, studies with funding, and studies reviewed by IRBs had higher quality scores and a higher percentage of high-quality RCTs than did other studies. Thirty-six RCTs were published in journals included in the Science Citation Index and 20 RCTs were published in journals included in the Science Citation Index Expanded. The largest number of RCTs (32.32%) were published by the Korean Journal of Urology.
CONCLUSIONS: A quantitative increase was observed in RCTs over time, but no qualitative improvement in the RCTs was observed. It seems necessary to put effort into the quality improvement of RCTs at the design stage.

INTRODUCTION

Evidence-based medicine (EBM) was introduced by the EBM Working Group in 1992 and is defined as the conscientious, judicious, and explicit use of the best evidence in making decisions for the care of individual patients [1]. Randomized controlled trials (RCTs) have been acknowledged as important standards for EBM. The RCT is a study design that can minimize bias and produce the most dependable data among study methods [2]. However, even the RCT cannot eliminate all bias, which can occur at the designing, conducting, or reporting phase and can yield misleading results [3]. Peer review is an essential method for preventing such results from being used as the basis of clinical applications [4]. Moreover, objective methodological quality assessment of published studies is essential because these studies can influence the quality of medical care received by patients [5]. The methodological quality of an article can represent its overall quality and should therefore be assessed at the design, conduct, and analysis steps [6,7]. In addition, any erroneous data can be identified throughout the assessment process, thereby eliminating the erroneous data from clinical care and saving medical expenses [8]. When meta-analyses or systematic reviews are conducted with the use of data from low-quality RCTs, the reliability of the conclusions drawn can be impaired [3].

Several methods can be used to assess the methodological quality of clinical trials, such as scales, individual markers, and checklists. The scales method is easy to apply to interstudy comparisons and is more applicable to performing quantitative assessments on the quality of a clinical trial than are the other methods. Randomization, blinding, and dropout are the factors of scale that directly relate to reducing bias. The Jadad quality assessment scale (Jadad scale) is a representative quality assessment tool consisting of these three items [9]. The Jadad scale has been widely used because of its simple questionnaire and capacity to make assessment easy, but it does not include an assessment item for concealment of allocation. However, there is an individual marker method that assesses allocation concealment, which is a way to randomize the allocation sequence to avoid any selection bias in the allocation of patients for treatment [10]. The van Tulder scale and the Cochrane Collaboration risk of bias tool (CCRBT) include an assessment item for concealment of allocation.

Some recent studies have examined issues in the quality assessment of RCTs. For example, Lee et al. [11] analyzed all RCTs published in the Korean Journal of Urology (Korean J Urol). The present study assessed the quality of urological RCTs that were conducted in Korea by using the Jadad scale, the van Tulder scale, and the CCRBT. The results of the present study can be used to assess the past and present status of urology research in Korea. Moreover, the results of this study can suggest guidelines for future studies and thus improve the medical practice of urology in Korea.

MATERIALS AND METHODS

1. Cohort study

Original articles that described urological studies conducted in Korean institutions were manually searched.

2. Selection of RCTs

Two reviewers independently searched all reports of RCTs conducted in Korean institutions by using the PubMed Medline database and KoreaMed. The reviewers used search limits and searched for terms such as "random," "randomized," and "randomly." All articles searched were selected as RCTs by the Methods section of the articles. The third reviewer made a final selection by adjusting the data collected.

3. Assessment of the quality of RCTs

Quality assessment was conducted by using the Jadad scale; in addition, the van Tulder scale and the CCRBT were used as individual indices. All assessments were performed by two reviewers. If there were different outcomes, the two reviewers and a third reviewer adjusted the discrepancy in the results through discussion. Starting from 1986, a quality analysis of RCTs was conducted in 1-year and 5-year units. The quality assessment was conducted by subject, type of intervention, presence of double blinding, presence of funding, and review by an Institutional Review Board (IRB).

1) Jadad scale

The Jadad scale is also known as the Oxford quality scoring system and assesses RCT-related literature. It is composed of five points in total: two in relation to randomization, two in relation to blinding, and one in relation to the dropout rate [9]. When the report includes only general comments with no detailed description of randomization and blinding, one point in each category is given. One point is added when there is a detailed description of the appropriate method. However, when the description method is inappropriate, one point is deducted. When the specified number and reasons for dropouts by each subject group are provided, one point is given. Even if there are no dropouts, this should be stated. If the total is ≥3 points, the study is considered to be high quality, but if the total is ≤2 points, the study is considered to be low quality. However, if it was not possible for the design of the study to be double-blinded, the study considered to be high quality if the total score is ≥2 points.

2) van Tulder scale

The van Tulder scale is designed to make assessments of 11 components including randomization, allocation concealment, baseline characteristics, patient blinding, caregiver blinding, observer blinding, cointervention, compliance, dropout rate, end point assessment time point, and intention-to-treat analysis [12]. The assessment method is the selection of "yes," "no," or "don't know" for each item. If ≥5 items are satisfied (≥5 points), the quality of the report is deemed to be high.

3) CCRBT

The CCRBT assesses the quality of RCTs by using six classifications: sequence generation, allocation concealment, blinding, incomplete outcome data, selective outcome reporting, and other potential threats to validity. The assessment indicates "yes," "no," or "unclear" for each domain, designating a low, high, and unclear risk of bias, respectively. In cases in which the first three questions are answered with "yes" and when no important concerns related to the last three domains are identified, the study is classified as having a low risk of bias [13]. If ≤2 domains are answered with "unclear" or "no," the study is classified as having a moderate risk of bias. If ≥3 domains are answered with "unclear" or "no," the study is classified as having a high risk of bias.

4. Statistical analysis

The one-way analysis of variance test was used to compare and analyze the respective scores obtained by each assessment tool, and a chi-square test was used to compare and analyze the ratio of high-quality articles and the quality assessment outcomes from the CCRBT. The quality assessment of RCTs according to intervention type, double blinding, funding, and IRB review was analyzed by Student's t-test. IBM SPSS ver. 18.0 (IBM Co., Armonk, NY, USA) was used for all statistical analyses, and a p-value of <0.05 was considered statistically significant.

RESULTS

1. Quantitative variation of RCTs over time

Ninety-nine urological RCTs were published between 1986 and 2011 (Fig. 1). Among these, 2 articles (2.02%) were published from 1986 to 1990, 6 articles (6.06%) from 1991 to 1995, 12 articles (12.12%) from 1996 to 1997, 14 articles (14.14%) from 2001 to 2005, 52 articles (52.53%) from 2006 to 2010, and 13 articles (13.13%) in 2011 (Table 1).

FIG. 1

Flow sheet.

TABLE 1

Characteristics of RCTs according to publication year

Values are presented as number (%).

RCT, randomized controlled trial; IRB, Institutional Review Board.

a:Chi-square test.

2. Qualitative variation in RCTs over time

The results of the quality assessment are presented in 5-year units starting in 1986. The mean Jadad scale for RCTs from 1986 to 1990 was 1.50±0.71, and the scores increased to 2.38±0.96 in 2011 (p=0.045) (Table 1). However, no statistical difference was seen in the quality assessment of RCTs according to publication year with the use of 1-year units (p=0.192) (Fig. 2). There were no high-quality articles published from 1986 to 1990, but this number increased to 7 of 13 RCTs (53.85%) in 2011 (p=0.344) (Table 2).

FIG. 2

Quality assessment of randomized controlled trials according to publication year by use of 1-year units. One-way analysis of variance. a:Chi-square test.

TABLE 2

Quality assessment of RCTs according to publication year

Values are presented as mean±standard deviation or number (%).

RCT, randomized controlled trial.

One-way analysis of variance.

a:Chi-square test.

2) van Tulder assessment scale

There was no statistical difference in the quality assessment of RCTs according to publication year (5-year units, p=0.974; 1-year units, p=0.811). The percentage of high-quality articles showed no statistical difference according to publication year (p=0.904) (Table 2).

There were no low risk of bias articles among the RCTs published up to 2011 by the CCRBT assessment. There was no statistical difference according to publication year by the CCRBT (5-year units, p=0.252; 1-year units, p=0.632).

3. Analysis of RCT quality by medical subject

Among the RCT articles presented during the 26 years (1986 to 2011) included in this study, 20 articles were about erectile dysfunction, 15 were about benign prostatic hyperplasia, 12 were about prostate cancer, 8 were about stone disease, 8 were about overactive bladder, 7 were about premature ejaculation, 6 were about chronic pelvic pain syndrome, 4 were about stress urinary incontinence, 3 were about bladder cancer, and 16 were about other subjects. There were no statistically significant differences observed among groups by the CCRBT assessment. However, both in the Jadad scale assessment and in the van Tulder scale assessment, the quality of RCTs and the percentage of high-quality RCTs differed significantly among subjects (Table 3).

TABLE 3

Characteristics of RCTs according to subjects

Values are presented as mean±standard deviation or number (%).

RCT, randomized controlled trial; CPPS, chronic pelvic pain syndrome.

One-way analysis of variance.

a:Chi-square test.

4. Analysis of factors related to the quality of the articles

Drug studies, double-blind studies, studies with funding, and studies reviewed by IRBs had higher quality scores and a higher percentage of high-quality RCTs than did other studies (Table 4). The differences were statistically significant.

TABLE 4

Factors associated with quality of RCTs

Values are presented as mean±standard deviation or number (%).

RCT, randomized controlled trial; IRB, Institutional Review Board.

Student's t-test.

a:Chi-square test.

5. Publication journal and institution of study

Thirty-six RCTs were published in journals included in the Science Citation Index and 20 RCTs were published in journals included in the Science Citation Index Expanded. The largest number of RCTs (32.32%) was published by Korean J Urol (Fig. 3). Sixteen RCTs were conducted at Seoul National University (Fig. 4).

FIG. 3

Journals in which articles were published. SCI, science citation idex; SCIE, Science Citation Index Expanded.

FIG. 4

Medical institution of the corresponding author.

DISCUSSION

In the present study, the quality of published urological RCTs conducted in Korea from 1986 to 2011 was assessed. Whereas the number of RCTs published has gradually increased, there was no significant difference in the quality of RCTs according to publication year. The small number of methodological descriptions of concealment of allocation and the lack of double-blind studies were the greatest factors in preventing studies from receiving high-quality assessments.

There have been few qualitative analyses of RCTs. Uetani et al. [2] analyzed medical RCTs conducted in Japan by using consolidated standards for reporting of trials (CONSORT) statements. The CONSORT statement was announced in 1996 with an aim to decrease the number of poorly conducted RCTs by detailing appropriate standards. Since the release of the CONSORT statement, various journals and organizations have played a leading role in enhancing the quality of RCTs [14]. Uetani et al. [2] showed that of 98 RCTs conducted in Japan from January to March 2004 that had been published, only 11 were in accordance with the CONSORT statement. However, because the CONSORT statement is not a quality assessment tool, it cannot be used to quantify the qualitative analysis.

Various types of methodological quality assessment tools for RCTs are available, including the Campell, Chalmers, CCRBT, Jadad, Moher, Newell's, and van Tulder methods. However, assessment of the quality of trials remains controversial, and no consensus exists on which tools are highly accurate and valid [15]. The present study was able to overcome such limitations through the use of three different tools. These three tools are representative assessment tools and are commonly used both nationwide and worldwide. The Jadad scale is a simple and easy method of performance assessment, but does not include assessment items for concealment of allocation. Therefore, additional analyses were performed by using the van Tulder scale and the CCRBT to supplement the Jadad scale. Patient, caregiver, and observer blinding as well as allocation concealment are considered in the van Tulder scale. Also, the term for intention-to-treat analysis is included in van Tulder scale. Although the CCRBT has a limitation in that it cannot quantify the quality of RCTs, it can be used to perform more objective analysis of RCTs because of its more detailed interpretation of each classification.

Kim et al. [16] analyzed the quality of RTCs published in five academic journals (the Korean Journal of Internal Medicine, the Journal of the Korean Surgical Society, the Korean Journal of Obstetrics and Gynecology, the Korean Journal of Pediatrics, and the Korean Journal of Family Medicine) by using the Jadad scale. They found that the number of RCTs with a Jadad score of more than 2 points increased in the 1990s compared with the 1980s. In addition, Chung et al. [5] analyzed RCTs published in the Korean Journal of Family Medicine from 1980 to 2005 and reported that the number of RCTs increased according to publication year. This increase in the number of RCTs may have been a result of the growing influence of EBM [17]. In the present study, as in these previous reports, the number of RCTs was found to have increased over time. However, the quality of the RCTs did not significantly change according to publication year.

Lee et al. [11] previously analyzed the quality of RCTs published in the Korean J Urol. According to their study, there were 28 RCTs out of 3,156 original articles presented from 1991 to 2010. The mean Jadad scale score was 1.75 and 8 RCT articles were determined to be high-quality articles. They found that the number of RCT articles and their quality improved over time. Moreover, they suggested that the descriptions of allocation concealment in the RCTs published in the Korean J Urol needed improvement. In the present study, among the urological RCTs conducted in Korea, only three articles had proper descriptions of allocation concealment. Hewitt et al. [18] showed that 46% of RCTs published in four different medical journals (the British Medical Journal, the Journal of the American Medical Association, the Lancet, and the New England Journal of Medicine) had inappropriate descriptions of allocation concealment. According to Schulz and Grimes [19], not incorporating allocation concealment into a study can impair the effects of randomization and blinding. The omission of allocation concealment could distort the results of the intervention by more than 40%. The quality of Korean urology reports would be improved if clinical studies included adequate blinding and allocation concealment.

Double blinding can be applied to drug studies more easily than to nondrug studies, such as those investigating surgery. Therefore, studies investigating drugs were scored as higher quality in comparison with studies not investigating drugs. In the present study, double-blind studies and drug studies were of better quality than other studies.

The present study confirmed that studies that were under IRB review were of better quality than RCTs that were not. To the best of our knowledge, no previously published study has analyzed the association of IRB review and the quality of articles. IRB review partially serves to assess the feasibility of the study design and the performance of the study protocol. The system by which RCT study protocols are validated and approved by IRBs has played a major role in raising the quality of articles, because IRB approval is considered an international quality standard. Clifford et al. [20] reported that there was no association between the funding source and the quality of the article. However, in the present study, funded studies were of better quality than were those that were not funded. This result may indicate that receiving financial support helps investigators to design well-organized studies and perform orderly research, resulting in higher quality articles. Moreover, the results of the present study indicated that most RCTs were published by selected universities and most RCTs were published in Korean J Urol.

This study had at least one limitation. It is possible that some RCTs were not considered in the present study. The extraction of RCTs and their quality assessments were based on the subjective judgment of the researchers. Therefore, two medical doctors independently extracted the RCTs. The assessments were performed independently by two reviewers and the outcomes were adjusted to ensure objectivity and reliability.

CONCLUSIONS

For the first time, the quality of urological RCTs conducted in Korea was assessed. The number of RCTs conducted by Korean medical institutions has increased over time. However, no qualitative improvement of RCTs was observed over time. In present study, areas for improvement were identified, and the findings of this study may contribute significantly to the overall quality of urological medical research in Korea.