Assessing number and quality of urology open access journals: 2011 to 2018.

BACKGROUND/AIMS: There is clear evidence that publishing research in an open access (OA) journal or as an OA model is associated with higher impact, in terms of number of reads and citation rates. The development of OA journals and their quality are poorly studied in the field of urology. In this study, we aim to assess the number of OA journals, their quality in terms of CiteScore, percent cited and quartiles, and their scholarly production during the period from 2011 to 2018.
METHODS: We obtained data about journals from www.scopus.com, and we filtered the list for urology journals. We obtained data for all Scopus indexed journals during the period from 2011 to 2018. For each journal, we extracted the following indices: CiteScore, Citations, scholarly output, and SCImago quartiles. We analyzed the difference in quality indices between OA and non-OA urology journals.
RESULTS: Urology journals have increased from 66 journals in 2011 to 99 journals in 2018. The number of OA urology journals has increased from only 10 (15.2%) journals in 2011 to 33 (33.3%) journals in 2018. The number of quartile 1 (the top 25%) journals has increased from only 1 journal in 2011 to 5 journals in 2018. Non-OA urology journals had significantly higher CiteScore compared with OA journals till the year 2015, after which the mean difference in CiteScore became smaller with insignificant p-value.
CONCLUSION: Number and quality of OA journals in the field of urology have increased throughout the last few years. Despite this increase, non-OA urology journals still have higher quality and output.

Introduction

The field of urology is among the rapidly developing fields, particularly in terms of technologic developments, where the last years were associated with several developments in robotic surgery, imaging, as well as its nanotechnology applications among others. One of the remarkable developments in urology field is its research output. Previous studies have pointed to an increase in the number of publications,23 as well as the quality and impact of publications in different urology fields.45 A study that assessed the trend of impact factor of urology journals between 2000 and 2006 found an increase in the impact factor throughout the studied years. Yet, none of the previous studies assessed the number or impact of open access (OA) journals in urology.

The overall proportion of OA literature has increased from 27% in 2006 to 50% in 2010, the main source of which are OA journals. There is clear evidence that publishing research in an OA journal or as an OA model is associated with higher impact, through number of reads and citation rates. Moreover, OA journals lead to better dissemination of knowledge with the benefit of more citations to the authors. The development of OA journals and their quality are poorly studied in the field of urology. Using data from Scopus database, we aim to assess the number of OA urology journals, their quality in terms of CiteScore, percent cited, and quartiles, and their scholarly production during the years from 2011 to 2018.

Materials and methods

We obtained data about journals from www.scopus.com, where the list of journals and their corresponding indices are openly available for download. We obtained data for all Scopus-indexed journals during the period from 2011 to 2018. We filtered the list for urology journals. OA journals covered by Scopus are indicated as OA if the journal is listed in the directory of open access journals (DOAJ) and/or the directory of open access scholarly resources (ROAD).

Variables

For each journal, we extracted the following variables:

CiteScore: Measures average citations received per document published in the serial.

Citation count: Citations received in 1 year (e.g., 2017) for the documents published in the previous 3 years (e.g., 2014–2016).

Scholarly output: Summation of documents published in the serial title (e.g., 2017) during the 3 years before the year of the metric (e.g., 2014–2016).

SCImago quartiles: Quartile 1 (Q1) = 99th–75th CiteScore percentile. Q2 = 74th–50th CiteScore percentile. Q3 = 49th–25th CiteScore percentile. Q4 = 24th–0 CiteScore percentile.

Statistical analysis

We used SPSS version 22.0 (Chicago, IL) in our analysis. We used mean ± SD to describe continuous variables (i.e., journal indices). We used count (frequency) to describe other nominal variables (i.e., OA journals). We used independent sample t test to analyze the mean difference in CiteScore and scholarly output with OA status in each year. All underlying assumptions were met, unless otherwise indicated. We adopted a p value of 0.05 as the significance threshold.

Results

Number of urology journals has increased from 66 journals in 2011 to 99 journals in 2018 (Fig. 1). This increase was also associated with an increase in the quality metrics, that the mean CiteScore increased from 1.09 ± 1.03 in 2011 to 1.37 ± 1.12 in 2018 (Table 1).

Figure 1

Number of OA and non-OA urology journals in the years 2011–2018. OA = open access.

Table 1

Citation and output indices for urology journals from 2011 to 2018.

	CiteScore		Percentile		Citation count		Scholar output		SJR		SNIP		Percent cited
Year	Mean	SD	Mean	SD	Mean	SD	Mean	SD	Mean	SD	Mean	SD	Mean	SD
2011	1.09	1.03	49.48	29.13	820.20	1,729.49	474.05	690.62	0.60	0.64	0.61	0.52	38.65	25.00
2012	1.04	1.07	49.68	29.02	753.81	1,678.91	429.20	681.95	0.57	0.70	0.59	0.56	37.81	25.45
2013	1.07	1.08	49.54	29.02	736.85	1,663.16	422.05	686.90	0.63	0.86	0.64	0.60	38.91	25.52
2014	1.01	1.04	49.45	29.10	677.86	1,586.41	405.91	677.61	0.58	0.84	0.65	0.64	38.43	25.68
2015	1.05	1.03	49.44	29.06	683.80	1,543.75	414.01	643.03	0.60	0.88	0.61	0.61	38.89	24.40
2016	1.13	1.02	49.54	29.01	662.05	1,446.25	403.68	590.24	0.65	0.87	0.64	0.60	41.16	23.38
2017	1.33	1.10	49.54	29.05	702.81	1,498.22	393.29	531.16	0.69	0.94	0.71	0.62	45.68	23.68
2018	1.37	1.12	50.10	28.70	684.56	1,433.17	387.69	530.01	0.69	0.99	0.72	0.64	46.69	23.57

SJR = SCImago journal rank, SNIP = source normalized impact per paper.

The number of OA urology journals has increased from only 10 (15.2%) journals in 2011 to 33 (33.3%) journals in 2018 (Fig. 1). This was associated with an increase in quality of these OA journals, as the number of Q1 journals has increased from only 1 journal in 2011 to 5 journals in 2018, and the CiteScore has increased from 0.75 ± 0.75 in 2011 to 1.10 ± 0.73 in 2018. On the other hand, we observed a decrease in mean scholarly output of OA journals from 236 publications per journal in 2011 to around 199 publications per journal in 2018 (Table 2).

Table 2

Citation metrics and scholarly output for urology OA journals from 2011 to 2018.

Year	Number of OA journals	Number of Q1 OA journals	CiteScore for OA journals, mean ± SD	Percent cited for OA journals, mean ± SD	Citation count, mean ± SD	Scholarly output, mean ± SD	Rank, mean ± SD
2011	10	1	0.75 ± 0.75	33.5 ± 24.3	158.5 ± 171.3	236.3 ± 226.1	37.9 ± 16.6
2012	15	1	0.61 ± 0.60	31.53 ± 21.85	121.8 ± 182.6	188.6 ± 214.2	43.87 ± 16.75
2013	16	2	0.6 ± 0.75	28.88 ± 24.0	147.31 ± 231.83	200.88 ± 196.17	49.25 ± 21.26
2014	19	3	0.58 ± 0.62	29.68 ± 21.77	151.74 ± 253.80	184.47 ± 175.30	53.21 ± 21.19
2015	20	3	0.73 ± 0.65	34.75 ± 21.26	175.15 ± 262.58	184.15 ± 149.16	52.5 ± 21.81
2016	26	4	0.88 ± 0.64	40.08 ± 20.32	173.23 ± 232.86	190.54 ± 148.66	53.73 ± 23.75
2017	26	6	1.13 ± 0.71	46.19 ± 20.07	217.27 ± 270.80	199.11 ± 155.77	51.46 ± 24.33
2018	33	5	1.10 ± 0.73	45.12 ± 21.18	210.36 ± 224.55	198.94 ± 144.64	55.61 ± 24.01

Upon comparing CiteScore quality index between OA and non-OA urology journals, we found that non-OA journals had significantly higher CiteScore compared with OA journals till 2015, after which the mean difference in CiteScore became smaller with insignificant p value (Table 3). Scholarly output for non-OA journals is significantly higher than OA journals’ output in all over the period from 2011 to 2018 (Table 4).

Table 3

CiteScore citation metrics for urology OA versus non-OA journals from 2011 to 2018.

Year	CiteScore for non-OA journals	CiteScore for OA journals	Mean difference	95% CI	p
2011	1.15 ± 1.07	0.75 ± 0.75	0.40	−0.19–0.98	0.169
2012	1.15 ± 1.13	0.61 ± 0.60	0.54	0.12–0.97	0.014
2013	1.17 ± 1.13	0.63 ± 0.75	0.54	0.07–1.02	0.027
2014	1.13 ± 1.11	0.58 ± 0.62	0.55	0.16–0.94	0.006
2015	1.15 ± 1.10	0.73 ± 0.65	0.42	0.03–0.81	0.036
2016	1.21 ± 1.12	0.88 ± 0.64	0.33	−0.34–0.69	0.075
2017	1.40 ± 1.21	1.13 ± 0.71	0.27	−0.13–0.67	0.177
2018	1.50 ± 1.25	1.10 ± 0.73	0.41	−0.06–0.88	0.086

Table 4

Scholarly output in terms of mean publications per year for urology OA versus non-OA journals from 2011 to 2018.

	Output for non-OA journals	Output for OA journals	Mean difference	95% CI	p
2011	516.5 ± 737.0	236.3 ± 226.1	280.2	35.44–525.0	0.026
2012	489.35 ± 744.32	188.6 ± 214.2	300.75	79.70–521.80	0.008
2013	477.34 ± 753.0	200.88 ± 196.17	276.47	65.2–487.8	0.011
2014	466.88 ± 749.56	151.74 ± 253.80	282.41	85.99–478.83	0.005
2015	478.76 ± 711.51	184.15 ± 149.16	294.61	114.12–475.1	0.002
2016	480.65 ± 667.48	190.54 ± 148.66	290.11	123.37–456.85	0.001
2017	464.39 ± 599.27	199.11 ± 155.77	265.28	111.50–419.06	0.001
2018	482.06 ± 621.36	198.94 ± 144.64	283.12	65.04–501.20	0.011

Discussion

We observed an increase in both number and quality of urology journals during the period from 2011 to 2018. The proportion of OA journals has also increased from 15.2% in 2011 to 33.3% in 2018, this was also associated with an increase in the quality of OA journals, as indicated by CiteScore measure. Although the scholarly output and the citation metrics of non-OA journals are generally higher than OA journals, this difference has decreased over the years, becoming nonstatistically significant since 2015.

The impact of OA status on number of citations received and the citation metrics of a journal was thoroughly studied. A study that analyzed several citation indices between OA and non-OA medical journals found that OA journals had higher CiteScore compared with non-OA journals, although the latter had higher scholarly output. This was inconsistent with our findings about urology journals, where non-OA journals had generally higher CiteScore compared with OA journals, although the difference between them has decreased throughout the studied years. Another study on medical journals that assessed the predicting factors associated with citations found that journal's current citation index was a predictor of future citations. In a study that assessed predictors of citations in 4 non-OA urology journals, its main finding was that study design was the main predictor of future citations, where randomized controlled trials were expected to have higher number of citations compared with other study designs.

In 2009, Hennessey et al published an article analyzing the top 100 cited articles in urology, they only included 15 journals, none of which was an OA journal. In 2013, Nason et al published an update for the list with 16 journals, and none of these was an OA journal as well. We believe the main reason behind these observations was that most of these articles were old articles, published in as early as 1956. Another measure of the impact of an article is its altmetric score, which measures the impact of journal articles by tracking social media, Wikipedia, public policy documents, blogs, and mainstream news activity. In a study that assessed the top 100 urology articles in terms of altmetric score, articles published in OA journals were among the top 10. Recently, several changes are being taken in the landscape of regulations urging publishing in OA journals, especially for the US National Institute of Health and the UK Wellcome Trust funding bodies.1516 In a study that analyzed various citations and output indicator of the OA journal “Korean Journal of Urology,” the author found an increase in the number of citations, which was associated with changing the journal official language to English and changing the journal into an OA journal.

Readers of this and other bibliometric articles should bear in mind several limitations. Although we used rigorous indices to reflect the strength of journals (i.e., CiteScore and citations), we didn’t use the more popular “impact factor” index, as it is a product of other databases. Moreover, there are different bibliometric databases that can be used for such an analysis, including Scopus (which was used in the present study), PubMed, and Web of Science. A study that compared bibliometric analysis using PubMed, Web of Science, and Scopus found that PubMed database was only devoted to biomedical sciences and did not provide advanced analytic tools. For Web of Science, although it provided a user-friendly interface, its restricted inclusion criteria for journals may hamper the generalizability of the results. Bibliometric analysis using Scopus database includes a wide range of high-quality journals with advanced analytic tools and comparative indicators, which is why we chose it for our current analysis.

Number and quality of OA journals in the field of urology have increased throughout the last few years. Despite this increase, non-OA urology journals still have higher quality and output. Funders and initiatives that promote OA publishing should further stress on publishing in OA journals, as we believe this will enhance their quality and impact and further enhance the development of the field of urology.

Acknowledgments

None.

Statement of ethics

The study did not use patient's data, and was waived from ethical approval.

Conflict of interest statement

The authors declare that they have no financial conflict of interest with regard to the content of this report.

Funding source

None.

Author contributions

Research Conception: SAA.

Data collection: SAA.

Literature review: ZA, MM, MH, MO, OO.

Data analysis: SAA, ZA.

Manuscript drafting: ZA, MM, MH, MO, OO.

Manuscript revision: SAA.

Approval of final version: SAA, ZA, MM, MH, MO, OO.