Quality Assessment and Relevant Clinical Impact of Randomized Controlled Trials of Varicocele: Next Step to Good-Quality Randomized Controlled Trial of Varicocele Treatment.

PURPOSE: To assess the quality of randomized controlled trials (RCTs) on varicocele published from 1979 to 2017.
MATERIALS AND METHODS: We searched for original RCT on varicocele published between 1979 and 2017. Jadad scale, van Tulder scale, and Cochrane Collaboration Risk of Bias Tool were used to analyze RCT quality over time. Effects on RCT quality including funding source, Institutional Review Board (IRB) approval, and intervention were assessed. Treatment parameters of varicocele were also analyzed.
RESULTS: Blinding and allocation concealment were described in 25.9% and 9.4% of RCT, respectively. Both tended to increase and a sharp dip in allocation concealment was observed in 2010-2017. Jadad scores increased steadily from 1979 to 2017 (1.28±0.59 to 2.19±1.10, p<0.01). Van Tulder scores tended to increase from 1979 to 2017 (4.21±0.94 to 5.58±1.58, p<0.01). RCTs with funding statements had higher Jadad (Yes vs. No, 3.25±0.50 vs. 1.70±0.97; p<0.01) and van Tulder (Yes vs. No, 7.25±1.26 vs. 4.81±1.26; p<0.01) scores than unfunded RCTs. IRB approval and intervention were associated with better quality.
CONCLUSIONS: The number of RCTs on varicocele increased from 1979 to 2017. Also, quality improved over time with increasing IRB approval, funding, and multicenter trial. Most RCTs on varicocele reported the use of surgical treatment. RCTs of surgical treatments have limitations to satisfy the condition of RCT to conduct, but their quality has improved over time.

INTRODUCTION

Evidence-based medicine (EBM) involves the application of medical treatment that derives high-quality results through a systematic process that combines clinical evidence [1]. With the emphasis on EBM in the medical field, randomized controlled trial (RCT), in which evidence is most highly valued, are becoming important. This is because the RCT is the most reliable method for evaluating the effectiveness of medical treatment by minimizing bias [2]. For this reason, the number of RCT is rapidly increasing worldwide, while the conditions and criteria for RCT are high [3,4]. To improve RCT quality, it is necessary to evaluate objective methodological study quality [5]. This is because bias can occur if the basic elements of the RCT, from design to performance, are not properly implemented [6]. The International Committee of Medical Journal Editors (ICMJE) recommends the CONSORT statement as a guideline for improving RCT quality [7,8]. Although the CONSORT statement may help investigators conduct RCTs easily and appropriately, it is a just guideline, not a quality evaluator [9]. RCT methodological quality evaluation methods include scales, checklists, and individual markers, among others. Of them, scales have the advantage of better enabling quantitative assessments of clinical trial quality than other methods [8].

The Jadad scale, van Tulder scale (VTS), and Cochrane Collaboration Risk of Bias Tool (CCRBT) are representative quality assessment tools for the scale. The Jadad scale is a tool for evaluating randomization, double-blinding, and dropout to reduce bias [10]. Although the Jadad scale is simple and easy to evaluate, it does not include allocation concealment, an indicator of individual markers, so it is difficult to avoid selection bias in assigning patients to treatment. The VTS and CCRBT contain evaluation items for allocation concealment, which has the advantage of evaluating selection bias.

A varicocele is a state in which the pampiniform plexus, a vein in the scrotal sac, is palpably dilated secondarily due to retrograde blood flow of the vein to the testis [11]. Varicocele occurs in 10% to 15% of men and causes primary infertility in approximately 35% to 50% and secondary infertility in 81% [12]. There are many ways to treat varicocele: inguinal varicocelectomy, subinguinal microscopic varicocelectomy, percutaneous embolization, and laparoscopic varicocele ligation. However, controversy persists about the ideal treatment methods [13,14].

Erectile dysfunction, premature ejaculation, varicocele, and etc. are representative diseases that have been conducted a lot of randomized controlled studies in the andrology field. For this reason, we conducted our study.

No studies to date have evaluated varicocele RCT quality. Therefore, we suggest the direction of further studies by evaluating varicocele RCT quality using the three representative RCT quality assessment tools: the Jadad scale, VTS, and CCRBT.

MATERIALS AND METHODS

1. Subjects

The subjects of the analysis were RCTs identified based on a search of “varicocele” in PubMed and MEDLINE. Original RCTs of varicocele were searched from 1979 to 2017 (Supplement Table 1). An RCT quality analysis was performed by dividing the studies into three periods (early, 1979–2000; mid, 2001–2009; and late, 2010–2017) considering the CONSORT statement was first published in 2001 and revised in 2010 considering application term.

2. RCT selection

Two reviewers (WJS, HJP) independently searched PubMed and MEDLINE. We extracted and validated articles using the search terms “randomized,” “randomization,” “randomly,” “varicocele”, and “varicocelectomy.” Differences in extraction results between reviewers were subjected to data adjustment (Fig. 1).

Fig. 1

Flow diagrams. RCT: randomized controlled trial.

3. Evaluation method using quality assessment tools

Two reviewers (JHC, KSK) separately analyzed the RCTs using the Jadad scale, VTS, and CCRBT. When differences occurred, one other reviewer settled them.

1) Jadad scale

The Jadad scale, also known as the Oxford quality scoring system, consists of three items. It consists of two scores for randomization, two scores for blinding, and one score for dropout. Regarding randomization, one point is given if the article mentions randomization, while another point is given if it includes the appropriate randomization method and 1 point is subtracted for inappropriate randomization for a total of 0–2 points. Regarding blinding, 1 point is given when double blinding is mentioned, and an additional 1 point is given when it includes an appropriate blinding method and 1 point is subtracted for inappropriate blinding for a maximum of 2 points. If the article mentioned dropout, 1 point was given. The quality evaluation is assessed by a total score of 5 points, with 0–2 points indicating low quality and 3–5 points indicating high quality [15].

2) van Tulder scale

The VTS evaluates RCT quality using the following 11 items: appropriateness of the randomization, concealment of treatment assignment, similarity of baseline characteristics, blinding of the patients, blinding of the treatment provider, observer blinding, co-intervention, compliance, dropout/failure rate, end-point assessment time point, and intention-to-treat analysis. Each item is assessed as Yes, No, or Do Not Know; if the total score is 5 or higher, the RCT is considered of high quality [16].

3) Cochrane Collaboration Risk of Bias Tool

Cochrane coalitions are divided into sequence generation, allocation concealment, blinding, incomplete outcome data, selective outcome reporting, and other potential threats to validity. Each item is answered with Yes, No, or Unclear, and the reviewers assess the answers Yes, No, or Unclear of each item according to the detailed criteria. Yes indicates high quality, No indicates low quality, and Unclear indicates insufficient information. A study is classified as having a low risk of bias when the first three questions are answered as Yes and no important matters related to the last three domains are identified, as having a moderate risk of bias when Unclear or No is answered in ≤2 domains, and as having a high risk of bias when Unclear or No is answered in ≥3 domains [17].

4. Quality assessment of RCTs using other factors

We qualitatively analyzed the RCTs according to the presence or absence of Institutional Review Board (IRB) approval, funding, intervention, multicenter design, and allocation. We also assessed the articles against the treatment measures that we wanted to see in each study. The treatment methods for varicocele and numerical distribution of the journals that published RCTs on varicocele were confirmed.

5. Statistical analysis methods

One-way analysis of variance was used to compare differences in scores among the assessment tools. The comparisons of percentages of high-quality RCTs and the qualitative assessment by CCRBT were analyzed using the chi-square test. In addition, Student's t-test was used to compare RCT quality according to the presence or absence of IRB approval, funding, blinding, and intervention. SPSS version 22.0 was used for the statistical analysis, and values of p<0.05 were considered statistically significant.

6. Ethics statement

This study is based on analysis of previous published articles. We did not approved IRB.

RESULTS

1. Quantitative change in RCTs over time

From 1979 to 2017, 85 original RCTs were published on varicocele. Of them, 29 were published in the early period, 30 were published in the mid period, and 26 were published in the late period. From the early stage to the late stage, studies including blinding, concealed allocation, funding, IRB approval, and multicenter design tended to increase (Table 1).

Table 1

Characteristics of RCTs by publication year with quality assessment of RCTs

Factor		Early (1979–2000)	Mid (2001–2009)	Late (2010–2017)	Total	p-value
Original articles		20	24	41	85
Blinding		2 (10.0)	7 (29.2)	13 (31.7)	22 (25.9)	0.09
Concealment of allocation		0 (0.0)	1 (4.2)	7 (17.1)	8 (9.4)	0.02
Multicenter		0 (0.0)	1 (4.2)	6 (14.6)	7 (8.2)	0.04
Funding		0 (0.0)	0 (0.0)	4 (9.8)	4 (4.7)	0.06
IRB		0 (0.0)	9 (37.5)	35 (85.4)	44 (51.8)	<0.01
Intervention		19 (95.0)	23 (95.8)	41 (100.0)	83 (97.6)	0.19
Jadad scale
	Score	1.20±0.52	1.63±0.92	2.15±1.09	1.78±1.00	<0.01
	High quality	1 (5.0)	5 (20.8)	17 (41.5)	23 (27.1)	<0.01a
VTS
	Score	4.05±0.95	4.88±1.08	5.39±1.46	4.93±1.35	<0.01
	High quality	5 (25.0)	12 (50.0)	28 (68.3)	45 (52.9)	<0.01a
CCRBT
	High risk	18 (90.0)	17 (70.8)	26 (63.4)	61 (71.8)	0.02a
	Moderate risk	2 (10.0)	7 (29.2)	11 (26.8)	20 (23.5)
	Low risk	0 (0.0)	0 (0.0)	4 (9.8)	4 (4.7)

Values are presented as number only, number (%), or mean±standard deviation.

RCT: randomized controlled trial, IRB: Institutional Review Board, VTS: van Tulder scale, CCRBT: Cochrane Collaboration Risk of Bias Tool.

Analysis of variance (ANOVA) test and achi-square test were used.

2. Qualitative changes in RCTs over time

1) Jadad quality assessment scale

From 1979 to 2017, the average Jadad scale score of the RCTs was 1.78±1.00: 1.20±0.52 in early the early period, 1.63±0.92 in the mid period, and 2.15±1.90 in the late period, showing a steady increase in scores (p<0.01). In addition, the number of high-quality studies was 1 (5.0%) in the early period, 5 (20.8%) in the mid period, and 17 (41.5%) in the late period (p<0.01). The total number of high quality RCTs was 23 (27.1%) (Table 1).

The average scores in the early, mid, and late periods increased to 4.05±0.95, 4.88±1.08, and 5.39±1.46, respectively (p<0.01). The number of high-quality studies was 5 (25.0%), 12 (50.0%), and 28 (68.3%) in the early, mid, and late periods, respectively (p<0.01), while the total number of high-quality studies was 45 (52.9%; p<0.01).

The numbers of low risk of bias RCTs on the CCRBT assessment were 0 (0.0%), 0 (0.0%), and 4 (9.8%) in the early, mid, and late period, respectively (p<0.01; Table 1).

3. Analysis of factors related to study quality

When analyzing Jaded scale and VTS scores, the number of high-quality RCTs with funding and IRB approval was significantly higher. On the CCRBT analysis, the number of RCTs with a low risk of bias was also significantly higher as well when funding and IRB approval were performed (Table 2).

Table 2

Factors associated with quality of RCTs

Factor		No. of RCTs	Jadad scale		van Tulder scale		Cochrane Collaboration Risk of Bias Tool
			Score	High quality	Score	High quality	High risk	Moderate risk	Low risk
Funding source
	Yes	4 (4.7)	3.25±0.50	4 (100.0)	7.25±1.26	4 (100.0)	0 (0.0)	3 (75.0)	1 (25.0)
	No	81 (95.3)	1.70±0.97	19 (23.5)	4.81±1.26	41 (50.6)	61 (75.3)	17 (21.0)	3 (3.7)
	p-value		<0.01	<0.01a	<0.01	0.05a	<0.01a
Reviewed by IRB
	Yes	44 (51.8)	2.23±1.03	19 (43.2)	5.61±1.37	33 (75.0)	23 (52.3)	17 (38.6)	4 (9.1)
	No	41 (48.2)	1.29±0.87	4 (9.8)	4.20±0.87	12 (29.3)	38 (92.7)	3 (7.3)	0 (0.0)
	p-value		<0.01	<0.01a	<0.01	<0.01a	<0.01a
	Allocation
	Yes	8 (9.4)	2.88±0.99	6 (75.0)	7.25±1.04	8 (100.0)	4 (50.0)	3 (37.5)	1 (12.5)
	No	77 (90.6)	1.66±0.94	17 (22.1)	4.69±1.14	37 (48.1)	0 (0.0)	17 (22.1)	60 (77.9)
	p-value		<0.01	<0.01a	<0.01	<0.01a	<0.01a
Intervention
	Yes	83 (97.6)	1.80±1.01	23 (27.7)	4.96±1.35	45 (54.2)	59 (71.1)	20 (24.1)	4 (4.8)
	No	2 (2.4)	1.00±0.00	0 (0.0)	3.50±0.71	0 (0.0)	2 (100.0)	0 (0.0)	0 (0.0)
	p-value		0.27	0.38a	0.13	0.13a	0.67a
Multicenter
	Yes	7 (8.2)	1.29±0.49	0 (0.0)	4.43±1.27	3 (42.9)	6 (85.7)	1 (14.3)	0 (0.0)
	No (single center)	78 (91.8)	1.82±1.03	23 (29.5)	4.97±1.36	42 (53.8)	55 (70.5)	19 (24.4)	4 (5.1)
	p-value		0.18	0.09a	0.31	0.58a	0.66a

Values are presented as number (%) or mean±standard deviation.

RCT: randomized clinical trial, IRB: Institutional Review Board.

Student's t-test and achi-square test were used.

4. Assessment of RCT quality according to varicocele treatment parameters

With regard to varicocele treatment distribution, RCT studies on sperm parameters, treatment feasibility, and other variables were published, in that order (Table 3). In the analysis of RCT quality according to varicocele treatment subjects, the mean Jadad scale scores were 1.57±0.89, 1.38±0.81, and 2.85±0.99 for sperm parameter, treatment feasibility, and other, respectively (p<0.01), while the numbers of high-quality RCTs were 7 (16.7%), 2 (9.5%), and 10 (76.9%), respectively (p<0.01). The VTS scores for sperm parameter, treatment feasibility, and other were 4.64±1.19, 4.48±1.33, and 6.15±1.35, respectively. There were also 19 (45.2%), 7 (33.3%), and 11 (84.6%) high-quality RCTs for sperm parameter, treatment feasibility, and other, respectively as assessed using the VTS (p=0.01). The numbers of RCTs assessed by CCBRT as having a low risk of bias for sperm parameter, treatment feasibility, and other were 0 (0.0%), 1 (4.8%), and 2 (15.4%), respectively (p<0.01; Table 3).

Table 3

Assessment of RCT quality according to treatment parameters of varicocele

Region	Jadad scale		van Tulder scale		Cochrane Collaboration Risk of Bias Tool
	Score	High quality	Score	High quality	High risk	Moderate risk	Low risk
Treatment feasibility (n=21)	1.38±0.81	2 (9.5)	4.48±1.33	7 (33.3)	17 (81.0)	3 (14.3)	1 (4.8)
Postoperation pain (n=3)	3.00±0.00	3 (100.0)	6.67±0.58	3 (100.0)	0 (0.0)	2 (66.7)	1 (33.3)
Sperm parameter (n=42)	1.57±0.89	7 (16.7)	4.64±1.19	19 (45.2)	36 (85.7)	6 (14.3)	0 (0.0)
Testis function (n=3)	1.33±0.58	0 (0.0)	4.67±0.58	2 (66.7)	3 (100.0)	0 (0.0)	0 (0.0)
Pregnancy (n=3)	2.00±1.00	1 (33.3)	5.33±0.58	3 (100.0)	2 (66.7)	1 (33.3)	0 (0.0)
Other (n=13)	2.85±0.99	10 (76.9)	6.15±1.35	11 (84.6)	3 (23.1)	8 (61.5)	2 (15.4)
p-value	<0.01	<0.01	<0.01	0.01	<0.01
Total (n=85)	1.78±1.00	23 (27.1)	4.93±1.35	45 (52.9)	61 (71.8)	20 (23.5)	4 (4.7)

Values are presented as mean±standard deviation or number (%).

RCT: randomized clinical trial.

Analysis of variance (ANOVA) and chi-square test were used.

5. Distribution of RCTs of varicocele treatments

RCTs on varicocele have reported treatment results, and we have summarized the treatment types used. The most studied treatment methods accounted for 27.4%, followed by open varicocelectomy (23.5%), other methods (16.5%), laparoscopic varicocelectomy (11.8%), and embolization (9.4%) (Fig. 2A). Journals with publications on RCTs related to varicocele included Urology, Journal of Urology, and Andrologia (Fig. 2B).

Fig. 2

Distribution of (A) randomized control trial of treatment and (B) journals publishing randomized control trial in varicocele. JLAST: Journal of Laparoendoscopic & Advanced Surgical Techniques, Int J Surg: International Journal of Surgery, Urol J: Urology Journal, J Androl: Journal of Andrology, Int J Androl: International Journal of Andrology, Asian J Androl: Asian Jornal of Andrology, Can J Urol: Canadian Journal of Urology, Eur Urol: European Urology, Fertil Steril: Fertility and Sterility, Hum Reprod: Human Reproduction, BJU Int: British Journal of Urology International, J Urol: Jornal of Urology.

DISCUSSION

In the quality assessment of RCTs conducted on varicocele revealed that the quality steadily improved from 2007 to 2012 using the Jadad scale, VTS, and CCRBT, and the number of RCTs also steadily improved. RCT quality was higher in cases of IRB approval and funding. Many RCTs on varicocele were associated with sperm parameters, and the most frequently RCTs on varicocele were published in Urology. The number of RCTs increased in most international journals. This increase in RCTs is considered the reason for the growing importance of EBM. Scales et al [18] reported that the number and percentage of RCTs increased over time in journals such as The Journal of Urology, Urology, European Urology, and BJU International from 1996 to 2004. Jo et al [19] found that the number of RCTs increased over time by analyzing RCTs published in the Journal of Sexual Medicine from 2004 to 2012. This study revealed that the number of RCTs published on varicocele has been increasing steadily. Distribution of RCTs of varicocele treatments showed the most studied treatment methods accounted for followed by comparing treatment, open varicocelectomy, and etc. Treatment of varicocele have complication of recurrence. This probably made number one distribution of RCTs of varicocele treatments with comparing treatment.

As the value of EBM in clinical medicine increases, a systematic and scientific approach to RCTs is emphasized since quality is important [20]. In these studies, the CONSORT statement has been used as a guideline for conducting valid RCTs. In addition, in recent years, when authors submit their RCTs to journals, they are supposed to ensure that RCTs are conducted in accordance with the CONSORT statement by completing tasks in a checklist before submitting their manuscripts for publication. A quality assessment is important because it plays an important role in determining whether a study is actually acceptable or further research is necessary. In addition, a quality assessment evaluates the bias occurring in the study process, validity of the study conclusions, and need for further studies. Thus, RCT quality assessment should be conducted [21,22]. In the past, there were limitations in analyzing RCT quality, and only statistical elements were conducted [5,9]. There are many tools for quantitatively assessing RCT quality such as Campell, Moher, Chalmers, Jadad, VTS, Newell's, and CCRBT, but the most accurate and valid tool has yet to be identified [5]. In this study, we used three tools, the Jadad scale, VTS, and CCRBT, which are able to comprehensively analyze the various elements of the CONSORT statement to assess RCT quality. The Jadad scale is simple and easy to evaluate a tool for randomization, double-blinding, and dropout to reduce bias. The VTS and CCRBT contain evaluation items for allocation concealment, which has the advantage of evaluating selection bias which is limited by the Jadad scale. Allocation concealment helps to prevent selection bias during the assignment of patients to the treatment arms.

An RCT quality assessment using these tools was recently conducted in several places. Kim et al [23] analyzed RCT quality in the International Journal of Impotence Research using the Jadad scale, VTS, and CCRBT. Decreases were observed in the mean Jadad scale scores of the RCTs and in the percentage of high-quality RCTs over time. On the VTS, the scale score and the high quality ratio remained constant; on the CCRBT, the ratio of studies with a low risk of bias was on the rise. Overall, efforts to improve quality are needed in the future. Lee et al [24] analyzed the RCTs published in Neurourology and Urodynamics from 1993 to 2012 using the Jadad scale, VTS, and CCRBT and found that the number of RCTs increased but the quality did not improve.

The number of RCTs on varicocele published from 1979 to 2017 has steadily increased. On the Jadad scale, RCT quality improved continuously from the early to late period. On the VTS, the percentage of high-quality RCTs gradually increased from the early to late periods. On the CCRBT, the percentage of studies with a low risk of bias was 0% in the early and mid periods but increased in the late period. This means that the RCT studies on varicocele show gradual quality improvements because of increasing the conditions of CONSORT statement such as double blinding, randomization, concealment of allocation, and etc. When Clifford et al [25] analyzed the association between quality and funding source in 100 RCTs that were published by five high-impact peer-reviewed general medical journals with high impact factors, they reported that they were irrelevant. In contrast, Kim et al [23] reported that RCTs that were financially supported tended to be larger-scale studies that had better organized research designs; thus, the number of high-quality RCTs was higher among those that received funding. In this study, the presence of funding influenced RCT quality, that is, the funded studies had higher quality scores than the non-funded studies. This implies that funding improved the quality of RCTs on varicocele.

In recent years, studies have been published on the relationship between qualitative RCT studies and IRB approval. IRB approval is a step that is recognized for the validity of design and implementation in the planning phase of a study; it is considered an international standard. Due to the importance of IRB approval, almost all journals now request IRB approval for clinical trials. Study quality is better in cases in which IRB approval is obtained [26]. Bridoux et al [27] reported that when RCT studies are approved by an IRB, they tend to be of higher quality. In this study, the quality differences based on IRB approval were also investigated. Of the RCTs on varicocele, the number of studies obtained IRB approvals increased over time, and RCTs with IRB approval had higher quality scores on the Jadad scale and VTS than those without IRB approval. The percentage of high-quality RCTs was also high. IRB approval played a major role in improving varicocele RTC quality. A valid study plan for obtaining IRB approval in an RCT plays a key role in improving the percentage of high-quality studies. Schulz and Grimes [28] reported that, when the allocation was not properly concealed, randomization in clinical research may be compromised, and even with initial randomization, the effect of the intervention could be distorted by as much as 40% [28]. Hewitt et al [29] assessed RCT quality published in the New England Journal of Medicine, BMJ, JAMA, and Lancet and found that 46% of studies inadequately concealed the allocation. Of the RCTs on varicocele, 8 (9.4%) concealed the allocation. RCTs studies that concealed the allocation showed high-quality results on the Jadad scale and VTS. A low percentage of RCT studies concealed the allocation, a major component in improving study quality. Thus, RCT studies including allocation should be performed in the future.

With regard to study limitations, no representative assessment tool is available to qualitatively analyze RCTs; moreover, a single tool that can evaluate all items listed in the CONSORT statement does not exist. The major limitation of RCT for diseases that require surgical or interventional treatment for varicocele is difficult to design with double-blinding, there is no placebo, and there is no same evaluation tools, which is the limitation of the dataset. With definite tool of evaluating quality of RCT could show us restriction. However, here we tried to compensate for these limitations by using the three tools that were the most widely used to evaluate RCT quality. As another limitation of this study, because the research and evaluation were performed manually, the researchers' subjective opinions could have affected the evaluation process. Our results include studies from 40 years ago. It would not be nice to do a quality assessment using research from 40 years ago. The study of the early period does not mean much.

This study is of significance because no studies have evaluated varicocele RCT quality. Suggestions are needed to direct better further studies. Also, varicocele is a disease that causes scrotum pain and infertility, and recently, in treatment of infertility is incresing.

CONCLUSIONS

The number of published RCTs on varicocele increased from 1979 to 2017. RCT quality improved over time with increases in blinding, IRB approval, funding, multicenter design, and allocation concealment. Most RCTs on varicocele focused on surgical treatments, and although they have limitations to satisfy the condition of RCT to conduct, their quality has improved over time.