Reporting Quality Assessment of Randomized Controlled Trials Published in Nephrology Urology Monthly Journal.

CONTEXT: Randomized controlled trials (RCTs) are important tools for evidence-based health care decisions. It is, therefore, important that they be conducted and reported with the highest possible standards. The aim of this study was to evaluate the reporting quality of the RCTs published in nephrology urology monthly journal and to examine whether there was a change over time in the reporting quality. EVIDENCE ACQUISITION: The quality of each report was assessed using the Consolidated Standards of Reporting Trials (CONSORT) 2010 Statement checklist and a 5-point quality assessment instrument, i.e. the Jadad scale.
RESULTS: Eighteen (14 Iranian and 4 non-Iranian) RCTs were published from 2012 to 2014 on topics including renal stone (16.6%), hemodialysis and transplantation (38.8%), and prostate conditions (11.1%). Interventions comprised surgery, drugs, and teaching method in 7 (38 %), 10 (55%), and 1 (5%) of them, respectively. According to the CONSORT checklist, the weakest reported items were registration number, identification as a randomized trial in the title, and settings and locations where the data were collected. The mean Jadad score of the reports was 2.72 ± 1.36 (54% of their maximum possible total score). According to the Jadad and CONSORT scales, there was an increase in the quality of reporting from 2012 to 2014.
CONCLUSIONS: This assessment shows low reporting quality scores in reports. Training courses for researchers, using standard reporting tools (e.g. CONSORT 2010 Statement checklist), and consultation with methodologists can improve the quality of published RCTs.

1. Context

1.1. Quality Assessment of Randomized Controlled Trials

In randomized controlled trials (RCTs), the participants are allocated to two groups of intervention and control for the comparison of some outcomes between them (1). RCTs are considered the most valuable method to assess the efficacy of treatments whether or not the result of a comparison is statistically significant. Thus, it is important that they be conducted and reported with the highest possible quality so as to enable the readers to judge which results are internally valid and bias-free. Moreover, it is significant to differentiate between assessing the quality of a trial and the quality of its reporting. The quality of a trial is defined as the confidence that the design, conduct, and analysis of the trial have minimized or avoided biases in its treatment comparisons. This definition focuses on design quality. The quality of a report can be defined as the provision of information about the design, conduct, and analysis of the trial. A biased but well-reported trial can receive a high score of quality. Inversely, a well-conducted but weakly reported trial can receive a low score of quality (2).

Assessing the quality of RCTs is a relatively important development and is usually performed via three tools of component, checklist, and scale. The component tool evaluates some aspects of a trial, whereas checklists and scales involve lists of items for the assessment of its quality. Scales provide a numeric score of quality which can be formally included into a systematic review study (2).

Meta-analyses of RCTs are being published increasingly (2, 3); there is, therefore, great interest in the quality assessment of the RCTs in such analyses (4-9). If safety and efficacy results of a meta-analysis are significantly affected by the quality of the original trials, then its results may be less meaningful if quality is not assessed formally (10).

There has also been a rise in the number of journals publishing RCTs. The past decade (i.e. 2000 - 2014) has witnessed an increase in the number of Iranian RCTs published in PubMed journals. Searching PubMed using the “randomized controlled trial” keyword in the [Title/Abstract] reveals an increase over time, i.e. from 15 records between 2000 and 2005 to 92 records between 2006 and 2010 and 373 records between 2011 and 2014. Consequently, it is important for the authors, reviewers, and editors of journals to pay special attention to reporting quality assessment.

1.2. Nephrology Urology Monthly Journal

Nephrology Urology Monthly (NUM) is a clinical open-access Iranian journal indexed in PubMed with an average number of published articles per year of 63. NUM is an authoritative clinical source devoted to selected compilations of the latest worldwide and interdisciplinary research and reviews in the field of basic and clinical urology and nephrology. The journal’s main focus is on the efficacy in improving clinically relevant outcomes such as mortality, morbidity, and quality of life. NUM accepts all kinds of manuscripts and other scientific communications, including original manuscripts, meta-analyses and reviews, health economic papers, debates, and consensus statements of clinical relevance of nephrological and urological fields. The impact factor and rejection rate of this journal in 2012 were 0.3 and 32%, respectively.

1.3. Objectives

Our primary focus was to evaluate the reporting quality of the RCTs published in NUM. As a secondary aim, we examined whether there was a change over time in the reporting quality.

2. Evidence Acquisition

This study is a journal-based assessment. The inclusion criterion was trials on humans with control groups published from 2012 to 2014 in NUM. We extracted descriptive information such as the year and location of the study, number and gender of patients, condition under investigation (diseases), outcomes, intervention and comparison groups, and ethical approval.

We also completed a comprehensive quality assessment of each report using the Consolidated Standards of Reporting Trials (CONSORT) and Jadad methods. The CONSORT, first conducted in 1996, comprises randomization, allocation concealment, sample size, statistical analysis, blinding, and primary and secondary outcomes (1). The objective of the CONSORT is to provide guidance to authors and reviewers about how to improve the quality of reporting. The CONSORT has been revised and published as the CONSORT 2010 statement checklist. This checklist contains multiple modified items listed separately (25 items and 37 sub-items) about title, abstract, introduction, methods, results, discussion, and additional information. Each item is reported as “yes” if the author has reported it. The CONSORT 2010 Statement and its website (www.consort-statement.org) are helpful in enhancing the reporting quality of RCTs. In the present assessment, we employed only 7 important items of the 37 sub-items of the CONSORT 2010 Statement: 1a) title and abstract: identification as a randomized trial in the title; 4a) methods (participants): eligibility criteria for the participants; 4b) methods (participants): Settings and Locations where the data were collected; 5) methods (interventions): interventions for each group with sufficient details to allow replication, including how and when they were actually administered; 6a) methods (outcomes): completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed; 17a) results (outcomes and estimation): for each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval); and 23) Registration Number and the name of the trial registry.

The Jadad scale, comprised of 5 items of randomization, method of randomization, blinding, method of blinding, and dropouts and withdrawals, is used to assess quality. Each question contains a “yes” or “no” response option. In total, 5 points can be awarded, with higher points indicating superior quality. Although this scale is conducted primarily to assess the quality of the reports on pain studies, it has been used in other areas as well (8).

In the present study, two reviewers conducted all the assessments. We performed a prior training to evaluate the quality of the RCTs via the two methods. We compared the number of the checklist criteria (7 items of 37 sub-items) that were reported appropriately as specified in the CONSORT 2010 Statement checklist. We also assessed separately the items and the total quality score obtained from the Jadad scale. The mean of the number of appropriate reported items according to the CONSORT 2010 Statement checklist and the mean Jadad score over time were compared.

3. Results

Our database searching on PubMed identified 188 published articles from 2012 to 2014 in NUM. Of these, 170 non-trial or non-human trials were excluded and 22 trials remained. Four reports had no control group and failed to meet our eligibility criteria. Finally, 18 RCTs were selected (Figure 1). The descriptive and quality assessment information on the RCTs is depicted in Tables 1 and 2.

Figure 1.

Tracking and Enrollment of the Randomized Controlled Trials

Table 1.

Quality Assessment via the Jadad Score for the Randomized Controlled Trials Published in Nephrology Urology Monthly From 2012 to 2014 [a]

Reference	Randomization		Blinding		Dropouts/Withdrawals	Total Score
	Randomized	Method of Randomization	Blinding	Method of Blinding
(11)	0, NR	0, NR	0, NR	0, NR	1	1
(12)	1	0, NR	0, NR	0, NR	0, NA	1
(13)	1	0, NR	1	1	1	4
(14)	1	1	0, NR	0, NR	1	3
(15)	1	0, NR	0, NR	0, NR	0, NA	1
(16) b	1	0, NR	1	1	0, NA	3
(17)	1	0, NR	0, NR	0, NR	1	2
(18) b	1	0, NR	1	1	1	3
(19)	1	0, NR	1	0, NR	0, NA	2
(20)	1	0, NR	1	1	1	4
(21)	1	0, NR	0, NR	0, NR	1	2
(22)	1	0, NR	0, NR	0, NR	1	2
(23)	1	0, NR	0, NR	0, NR	1	2
(24) b	1	1	1	1	1	5
(25) b	1	0, NR	0, NR	0, NR	1	2
(26)	1	0, NR	1	1	1	5
(27)	1	1	1	1	1	5
(27)	1	0, NR	0, NR	0, NR	1	2

a Abbreviations: NA: not appropriate; and NR, not reported.

b First author or corresponding author was affiliated to Baqiyatallah university of medical sciences.

Table 2.

Randomized Controlled Trials Published in Nephrology Urology Monthly From 2012 to 2014 [a]

Reference	Year	Country	Area	Sex	Sample Size	Interventions	Comparison Group	Outcome	Ethical Approval	Quality Score (Jadad)	CONSORT 2010 Statement Checklist Important Items No. [b]
											1a	4a	4b	5	6a	23	17a	No.Yes Items
(11)	2012	Egypt	Infertility	M	20 + 20	Loupe-assisted sub-inguinal varicocelectomy	Sub-inguinal varicocelectomy	Sperm parameters	N	1	N	Y	N	Y	Y	N	Y	4
(12)	2012	India	Analgesics for prostate biopsy	M	20 + 20 + 20	Diclofenac patch, periprostatic nerve block	No analgesic	Pain scores	N	1	N	Y	Y	Y	Y	N	Y	5
(13)	2012	Iran	Helicobacter pylori eradication in hemodialysis patients	MF	20 + 17	Omeprazole, amoxicillin, clarithromycin	Omeprazole, amoxicillin, azithromycin	UBT and the HBsAg test	N	4	Y	Y	Y	Y	Y	Y	Y	7
(14)	2012	Iran	Renal stone	MF	20 + 20	Prone supine percutaneous nephrolithotomy	supine percutaneous nephrolithotomy	Electrolyte, hemodynamic and metabolic changes	N	3	Y	Y	N	Y	Y	N	Y	5
(15)	2012	Iran	Renal stone	MF	32 + 30	Transurethral lithotripsy	Shock-wave lithotripsy	Renal stone management success	N	1	N	Y	N	Y	Y	N	Y	4
(16) c	2012	Iran	Renal transplantation complications	MF	112 + 101	Cyclosporine	Cyclosporine and calcium channel blockers	Gingival Index of McGaw and others, and Packet Index of Turesky–Gilmore–Glickman	Y	3	N	Y	N	Y	Y	N	Y	4
(17)	2012	Iran	Teaching hypertension	-	28 + 23	Cooperation lecture	Planned lecture	Long-term learning quality	N	2	N	Y	Y	NA	NA	N	NA	2
(18) c	2013	Iran	Hyperparathyroidism .	MF	37 + 39	250 mg vitamin C	Placebo saline	Serum PTH	Y	3	Y	Y	Y	Y	Y	Y	Y	7
(19)	2013	Iran	Benign prostatic hyperplasia	M	100+ 100	Modified transurethral resection of the prostate	Standard Transurethral resection of the prostate	Symptom scoring, post-micturating residual volume, uroflowmetry urine examination, bacterial count and assessment for late complications, International Index of Erectile Function and quality of life, uroflowmetry test	N	3	N	Y	Y	Y	Y	N	Y	5
(20)	2013	Iran	Dialysis complications	MF	90 + 90	80 mg/day Aspirin	Placebo	Catheter efficacy	N	4	N	Y	Y	Y	NA	N	Y	4
(21)	2013	Iran	Dialysis		30 + 30	Side-to-side (STS) anastomosis	End-to-side (ETS) anastomosis	Arteriovenous fistulae efficacy	N	2	Y	Y	Y	Y	Y	N	NA	5
(22)	2013	Iran	Functional iron deficiency in patients under hemodialysis	MF	20 + 20	Intravenous iron	Intravenous ascorbic acid	Hb and iron metabolism indices	Y	2	Y	Y	Y	Y	Y	Y	N	6
(23)	2014	Iran	Voiding dysfunction	MF	42 + 42	Midazolam	-	Disorders of the urinary tract and voiding dysfunction	N	2	Y	N	Y	Y	NA	Y	Y	5
(24) c	2014	Iran	Hemodialysis	MF	55 + 55 + 31	Vitamin C supplementation	1: Saline / 2: no intervention	CRP level	N	5	Y	Y	Y	Y	Y	Y	N	6
(25) c	2014	Iran	Urethral stricture	F	86	On-demand dilatation	Intermittent dilatation	Effectiveness of urethral stricture treatment	N	2	Y	Y	Y	Y	Y	Y	N	6
(26)	2014	Iran	Renal stone	MF	52 + 50	Tamsulosin	Placebo	Success rate of ureteroscopic lithotripsy	Y	5	Y	Y	Y	Y	Y	Y	N	6
(27)	2014	Iran	Diabetic nephropathy	MF	30 + 30	Spironolactone + placebo	Spironolactone + losartan	Diabetic nephropathy treatment success rate	Y	5	Y	Y	Y	Y	Y	Y	Y	7
(28)	2014	Malaysia	Hypovitaminosis D	MF	25 + 25	Oral calcitriol + calcium carbonate	Calcium carbonate alone	Renal function	Y	2	Y	Y	NA	Y	Y	Y	Y	6

a Abbreviations: NA, not appropriate; N, no; and Y, yes.

b Mentioned items are as followings: 1a, title and abstract; identification as a randomized trial in the title; 4a, methods (participants); eligibility criteria for the participants; 4b, methods (participants); settings and locations where the data were collected; 5, methods (interventions); interventions for each group with sufficient details to allow replication, including how and when they were actually administered; 6a, methods (outcomes); completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed; 17a, results (outcomes and estimation); for each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval); and 23, registration number and name of trial registry.

c First author or corresponding author was affiliated to Baqiyatallah university of medical sciences.

Fifteen (83%) reports were submitted from Iran. In 11 (61%) studies, the first or the corresponding author was affiliated to Baqiyatallah university of medical sciences. There were 12 (66%) reports involving both male and female genders. In 7 (38.8%) reports, hemodialysis and renal transplantation patients were the target group. The measured outcome in 6 (33%) reports was lab data, and the intervention methods were surgery and drug therapy in 7 (38.8%) and 10 (55.5%) reports, respectively (Table 3).

Table 3.

Randomized Controlled Trials Published in Nephrology Urology Monthly[a]

Status	Values [a]
Country
Iran [b]	4 (22.22)
Iran	11 (61.11)
Other	3 (16.67)
Outcome
Lab	6 (33.33)
Other (pain scores, success rate, quality of life, etc.)	1 (5.56)
Gender, male and female	12 (66.67)
Interventions
Surgery	7 (38.89)
Drug	10 (55.56)
Teaching method	1 (5.56)
Patients
Infertility	1 (5.56)
Prostate	2 (11.11)
Renal stone	3 (16.67)
Hemodialysis and renal transplantation	7 (38.89)
Voiding dysfunction	1 (5.56)
Teaching hypertension	1 (5.56)
Urethral stricture	1 (5.56)
Diabetic nephropathy	1 (5.56)
Hypovitaminosis D	1 (5.56)
Consort statement items c
1a	11 (61.11)
4a	17 (94.44)
4b	12 (66.67)
5	17 (94.44)
6a	15 (83.33)
17a	16 (88.89)
23	5 (27.78)
Jadad score
Randomization	17 (94.4)
Method of randomization	3 (16.6)
Blinding	8 (44.4)
Method of blinding	7 (38.8)
Dropout	14 (77.7)

a Data are presented as No. (%).

b First or corresponding author was affiliated to Baqiyatallah university of medical sciences.

c Mentioned items are as followings: 1a, title and abstract; identification as a randomized trial in the title; 4a, methods (participants); eligibility criteria for the participants; 4b, methods (participants); settings and locations where the data were collected; 5, methods (interventions); interventions for each group with sufficient details to allow replication, including how and when they were actually administered; 6a, methods (outcomes); completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed; 17a, results (outcomes and estimation); for each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval); and 23, registration number and name of trial registry.

The items of 1a, 4a, 4b, 5, 6a, and 17a and 23 items of the CONSORT 2010 Statement checklist were reported in 61.1%, 94.4%, 66.6%, 94.4%, 83.3%, 88.8%, and 27.7% of the RCTs, correspondingly (Table 3). Accordingly, the best reports were related to the eligibility criteria for the participants and the pre-specified primary and secondary outcome measures. We defined the number of the appropriate reported items as a total score for this checklist. Three studies obtained full points (7 items). The quality of reporting (by CONSORT 2010 Statement) was 4.4, 5.4, and 6 in the years 2012, 2013, and 2014, respectively, showing a trend of improvement over time. The quality of reporting in the RCTs performed by Baqiyatallah university of medical sciences, other Iranian universities, and other countries was 3.25, 3, and 1.2, respectively. Thus, the highest reporting quality in the RCTs was achieved by Baqiyatallah university of medical sciences (Table 4).

Table 4.

Randomized Controlled Trials Published in Nephrology Urology Monthly According to the Year of Publication

	Year				Country
	2012	2013	2014	Total	Iran [a]	Iran	Other
n	7	5	6	18
Jadad scale b	2.7	2.8	3.5	2.72	3.25	3	1.3
CONSORT 2010 Statement c	4.42	5.4	6	5.27	5.75	5.09	5

a First or corresponding author was affiliated to Baqiyatallah university of medical science.

b Mean of score.

cMean of the number of appropriate reported items.

The mean score of the Jadad scale was 2.72 ± 1.36 (54% of the maximum possible total score). The items of randomization, method of randomization, blinding, method of blinding, and dropouts/withdrawals of the Jadad scale were reported in 94.4%, 16.6%, 44.4%, 38.8%, and 77.7% of the RCTs, respectively. This score was 2.7, 2.8, and 3.5 in the years 2012, 2013, and 2014, respectively, which shows an increase in the quality of reporting over time (Table 4).

4. Discussion

Some potential biases in RCTs may never be minimized. For example, double blinding is questionable ethically and scientifically during surgical trials (3). One way to improve the reporting quality of RCTs is to use the CONSORT 2010 Statement. The journals that adhere to this checklist in their publication of RCTs have higher reporting quality than the ones that do not (8).

Proper reporting is required to generate unbiased comparison groups in controlled trials. Nonetheless, the reports of Iranian researchers in some journals usually provide inadequate or unacceptable information on critical points. Indeed, there are weaknesses in the reporting quality of the RCTs published by Iranian researchers in both Persian and English languages. A previous research suggested, however, that the quality of the non-English RCTs was similar to that in the English ones (9).

In 2012, (29), utilizing the CONSORT 2010 Statement, assessed 314 RCTs indexed in PubMed with affiliation to Tehran university of medical sciences and Iran university of medical sciences and showed that only the intervention used in the two groups was presented completely (100%) in the abstracts. In our assessment, the best report was related to defining eligibility criteria for the participants and defining intervention for each group. The other items in the Amanollahi et al. (29) study regarding the method of randomization, method of blinding, identification as a randomized trial in the title (item 1a), eligibility criteria for the participants (item 4a), and settings and locations where the data were collected (item 4b) were reported weakly and seen in 5.4%, 50.3%, 37.6%, 66.4%, and 19.4% of the reports, respectively. These percentages are comparable to those in our assessment. We showed that the Method of Randomization (16.6%), Eligibility criteria (94.4%), and Settings and Locations where the data were collected (66.6%) were reported with higher quality in the RCTs included in our study.

In 2013, Ghujazadeh et al. (30), drawing on the CONSORT 2010 Statement, assessed the reporting quality of 141 RCTs published by Iranian researchers in obstetrics and gynecology level-1 journals and showed that the weaknesses were chiefly in the Methods and Material, where out of 17 items, sample size determination, method of randomization, details of any kind of randomization (e.g. categorization and block formation), and blinding method accounted for the most notable shortcomings. The authors found that the method of randomization and method of blinding were reported in 39% and 50.4% of the RCTs; these percentages are higher than those in the present study. In 2014, Faizi et al. (31) in the quality assessment of the RCTs on applied psychotherapy for chronic pains in Iran showed that the mean score of Jadad was 1.53 ± 1.37, while this score had a higher mean in our assessment. The authors reported that the items (5 items) of the Jadad score were appropriately reported in 41.2%, 64.7%, 11.8%, 5.9%, and 29.4% of the RCTs, respectively, while these percentages in our study were 94.4%, 16.6%, 44.4%, 38.8%, and 77.7% of the RCTs, correspondingly, which indicates a higher quality of reports in our assessment with the exception of the method of randomization item.

Moher et al. (3) evaluated the reporting quality of RCTs on pediatric alternative medicine and showed that the studies achieved approximately 40% of their maximum possible total Jadad score. This result in our study was 54.4%.

According to the SCImago Journal Rank (SJR) indicator (http://www.scimagojr.com/journalsearch.php?q=19700201343&tip=sid&clean=0), our findings demonstrated that the rank of NUM in 2013 was higher than that in 2012 (SJR2012 = 0.118 vs. SJR2013 = 0.126). This shows that the reporting quality score can be considered a supplement to the ranking indices.

In general, meticulous reporting is required to generate an unbiased comparison of groups in RCTs. Our reporting quality assessment of the RCTs published in NUM, however, revealed low quality scores. It is possible that the Iranian investigators in this field have conducted few RCTs and therefore, not very experienced. Training courses for researchers, utilizing necessary reporting standard tools such as the CONSORT 2010 Statement by the editors of medical journals, and consultation with methodologists can improve the quality of RCTs.