A comparison of scientometric data and publication policies of ophthalmology journals.

Purpose: This retrospective database analysis study aims to present the scientometric data of journals publishing in the field of ophthalmology and to compare the scientometric data of ophthalmology journals according to the open access (OA) publishing policies.
Methods: The scientometric data of 48 journals were obtained from Clarivate Analytics InCites and Scimago Journal & Country Rank websites. Journal impact factor (JIF), Eigenfactor score (ES), scientific journal ranking (SJR), and Hirsch index (HI) were included. The OA publishing policies were separated into full OA with publishing fees, full OA without fees, and hybrid OA. The fees were stated as US dollars (USD).
Results: Four scientometric indexes had strong positive correlations; the highest correlation coefficients were observed between the SJR and JIF (R = 0.906) and the SJR and HI (R = 0.798). However, journals in the first quartile according to JIF were in the second and third quartiles according to the SJR and HI and in the fourth quartile in the ES. The OA articles published in hybrid journals received a median of 1.17-fold (0.15-2.71) more citations. Only HI was higher in hybrid OA; other scientometric indexes were similar with full OA journals. Full OA journals charged a median of 1525 USD lower than hybrid journals.
Conclusion: Full OA model in ophthalmology journals does not have a positive effect on the scientometric indexes. In hybrid OA journals, choosing to publish OA may increase citations, but it would be more accurate to evaluate this on a journal basis.

The medical publishing sector is currently growing, and digital publishing has an important place in this field. Therefore, there is a need for scientists to be able to compare and rank the quality of publications of medical journals or institutions. These types of value measurements are made using scientometric indexes.[1] These data are used for various purposes such as selecting journals to subscribe to or to submit an article to, for calculating the impact of published authors, for academic motivation, and for allocating grants. However, there is no consensus on which of the scientometric indexes used in the ranking of journals is ideal.

All the scientometric indexes evaluate the number of citations in different ways. Although there are many scientometric indexes, the annual Journal Citation Reports (JCR) journal impact factor (JIF) based on Science Citation Index Expanded (SCIE) data is the most commonly used index[2] and has received important criticisms in literature.[3] As an alternative to JIF, scientometric indexes such as the Hirsch index (HI) are used in many databases.[4] Taking the number of citations into consideration as much as the source of citations, more recent alternatives use calculation models measuring the mean prestige per article. As the Eigenfactor score (ES) of the SCIE database, this model is included as the scientific journal ranking (SJR) in the Scopus database.[5]

Another aspect of the scientometric measurement of an article is that it is an important target in respect of increasing the scientific effectiveness to be able to reach the target mass. With the spread of digital publishing, medical journals have transitioned to the open access (OA) publishing model to meet this demand.[6] OA allows free access to scholarly research online, removing the barriers of price and permission for readers.[7] Journals that publish OA function as a service provider for the wide distribution of science in addition to peer reviews. OA journals have been observed to achieve this aim by taking more citations than subscription-based journals.[8]

OA model is implemented in several ways, with the most commonly recognized forms being “gold,” “green,” “hybrid,” “gratis,” and “libre” OA, with differences in publishing licenses and charges.[7] Gold OA scholarly research is available for free fully. Journals may choose to publish all research with gold OA, or choose a hybrid form, which allows authors to choose.[7] The gold OA publication model incurs an economic burden for scientists, as in addition to the submission fee and pages charge, there is an extra OA article-processing charge (APC).[9] This economic burden creates a significant barrier between scientists and medical journals.

The number of citations of a published article and the time for citations to come may be different in every branch of science. Therefore, it has been recommended that evaluation and comparison of scientometric indexes should be made specific to the scientific field. Although various studies have compared scientometric indexes in other areas of medicine,[101112] and there have been publications of the effect of coronavirus disease 2019 (COVID-19) articles on the scientometric data of ophthalmology journals recently,[1314] comparisons of scientometric data in ophthalmology journals, specifically the comparisons of scientometric data regarding different OA publications models are limited.

Although OA models have differences in publishing licenses and charges, from an author’s perspective, there are two main questions when choosing a journal to submit a scholarly article to: (1) Does the journal publish OA in a compulsory way (models either publish full OA, employ hybrid OA, or publish subscription only) and (2) Does the author pay any APC? (models either charge authors for publishing or are free of charge). Therefore, in this study, we aimed to present and compare the scientometric data of journals publishing in the field of ophthalmology with different models of OA publishing: journals that publish all articles OA (full OA) (with APC or free of charge) and journals publishing OA on request (hybrid OA).

Methods

This retrospective scientometric study was conducted in accordance with the Declaration of Helsinki. The study did not involve the human or animal participants; hence, the study approval was not obtained from an ınstitutional review board of an ethical committee. A total of 60 journals in the SCIE database publishing scientific articles in the field of ophthalmology were initially included for evaluation in this study. Of these, 12 were excluded; three were not in English, two had incomplete data in the SCIE or Scopus database,[1516] four accepted publications in a field other than clinical ophthalmology, and three journals only evaluated invited authors. Thus, the study was conducted with the data of 48 journals.

The scientometric and bibliometric data were obtained in January 2021 from data on the Clarivate Analytics InCites[15] and Scimago Journal & Country Rank[16] websites. The latest data available on these websites were for the year 2019. By examining the Clarivate Analytics InCites database, the JIF and ES of journals, articles published in the last 5 years and the number of citations of these articles, and articles published OA and “subscription only” and the number of citations for these articles were evaluated.[15] From the starting point of these data, the ratio was calculated for OA publication in the journals included for evaluation (OA-published articles/total number of articles) and the ratio of citations obtained by OA articles in journals with a hybrid OA policy to the ratio of OA articles (number of citations of OA articles: total number of citations/number of OA articles: total number of articles). The SJR score and HI, presented as scientometric data in the Scimago Journal & Country Rank database, were included in the study.[16] Quarterly slices of the journals were calculated again according to the four scientometric indexes included in the evaluation (JIF, ES, SJR, HI).

All the data about the publications of the journals were obtained from the journal and publishers’ websites. From the examination of the journal website, the types of articles published in the journal were evaluated separately (review, original article, case report, others) and the journals were separated into two main categories: (1) journals publishing all types of articles (original research, review, case report, other types of papers) and (2) journals publishing articles other than case reports. The OA publishing policies of the journals were separated into three categories: (1) full OA with publishing fees, (2) full OA without fees, and (3) hybrid OA.

Data related to the fees required from the authors, the timing of payment (before or after acceptance), the amount, and the purpose of the fee charged (submission fees, page fees, or OA APC fees) were obtained from the journal websites and from public access data of the publishers. The fees requested for color printing were not included in the analysis. The fees were stated as US dollars (USD).

Statistical analysis

Data obtained in the study were analyzed statistically using SPSS for Windows version 23.0 software. Continuous variables were presented as median (minimum–maximum) values and categorical variables as number (n) and percentage (%). In the analysis of categorical variables, the Chi-square test was used, and Spearman’s rho analysis was applied to determine correlations between categorical data. The Mann–Whitney U-test was applied for continuous variables. A value of P < 0.05 was accepted as statistically significant in all the analyses.

Results

We evaluated a total of 48 journals in the SCIE database which publish scholarly articles in the field of ophthalmology. The bibliometric and scientometric data and the publishing policies of the journals are summarized in Table 1. Of the journals evaluated in the study, 25 published all types of articles and 23 published all article types except case reports. Thirty-seven journals employed a hybrid OA model and 10 journals had full OA publication policy (seven with charges, three free of charge). The authors were not charged fees in one journal with a subscription model. The median OA article rate was 5.37% (minimum 0.53%–maximum 100%) and the citations of these articles were 7.64% (minimum 0.21%–maximum 100%) of the citations of all the articles. The OA citation to OA article ratio was median 1.0 (minimum 0.15–maximum 2.71). Overall, 44 journals charged OA fees: seven full OA journals requested compulsory publishing fees and 37 journals employed hybrid OA and charged for OA on demand. The median cost of OA publishing was 3290 USD (minimum 900 USD–maximum 5000 USD). All journals requested payments after acceptance, and there were no journals that requested submission fees or pages charge [Table 1].

Table 1

The bibliometric data, scientometric data, and OA publishing policies of the ophthalmology journals

Type of article
All article types			25 (52.1%)
All article types except case reports			23 (47.9%)
Bibliometric data	Number of citable articles	2015	121 (24-930)
		2016	121 (17-820)
		2017	147 (14-651)
		2018	139 (14-672)
		2019	145 (12-532)
	Total number of citations	2015	1915 (537-47,257)
		2016	2317 (624-50,425)
		2017	2473 (778-52,637)
		2018	2811 (857-52,234)
		2019	2930 (896-52,362)
	Percentage of citable OA articles		5.37 (0.53-100)
	Percentage of citations to OA articles		7.64 (0.21-100)
	Citation/article ratio		1.0 (0.15-2.71)
Scientometric data	JIF		2.11 (0.61-12.33)
	ES		0.00453 (0.00107-0.05311)
	SJR		0.92 (0.37-4.49)
	HI		62 (16-229)
Publishing policy	Full OA		10 (20.8%)
		With charge	7 (14.6%)
		Free of charge	3 (6.3%)
	Hybrid OA		37 (77.1%)
	Subscription		1 (2.1%)
Publishing fee	Mandatory		7 (14.6%)
	Optional		37 (77.1%)
	Free		4 (8.3%)
Timing of payment	Before acceptance	0 (0%)
	After acceptance		44 (91.6%)
OA charge (n=44)		(US dollar)	3290 (900-5000)

ES=Eigenfactor score, HI=Hirsch index, JIF=journal impact factor, OA=open access, SJR=scientific journal ranking

Four scientometric indexes had strong positive correlations; the highest correlation coefficients were observed between the SJR and JIF (R: 0.906) and the SJR and HI (R: 0.798) [Table 2]. The quartile distributions of the journals in the four different scientometric indexes are presented visually in Table 3, with JIF as the reference. The journal ranked first according to JIF and the SJR, 22 according to the ES, and 25 according to HI. The journals in the first quartile according to JIF were in the second and third quartiles according to the SJR and HI, and could drop as low as the fourth quartile in the ES. Of the OA journals, only one full OA journal with fees was in the first quartile according to JIF, SJR, and ES and two full OA journals with fees were in the first quartile in HI. Three journals publishing OA free of charge were not in the first quartile in the four scientometric indexes [Table 3].

Table 2

Spearman correlation coefficients of the scientometric data of ophthalmology journals

	JIF	ES	SJR	HI
JIF	-	-	-	-
ES	0.589	-	-	-
SJR	0.906	0.663	-	-
HI	0.752	0.692	0.798	-

ES=Eigenfactor score, HI=Hirsch index, JIF=journal impact factor, SJR=scientific journal ranking

Table 3

Quartile distribution according to the scientometric data of ophthalmology journals

JIF	ES	SJR	HI
#1	#8a	#1	#2
#2	#2	#2	#8a
#3	#5	#3	#3
#4	#3	#5	#5
#5	#31c	#4	#16
#6	#7	#6	#7
#7	#6	#7	#14
#8a	#23a	#8a	#4
#9	#9	#9	#10
#10	#10	#13	#6
#11	#21	#11	#21
#12	#14	#15	#23a
#13	#20	#20	#20
#14	#19	#21	#13
#15	#16	#14	#19
#16	#36a	#12	#35
#17	#37a	#19	#22b
#18	#30	#10	#26
#19	#26	#23a	#9
#20	#35	#26	#11
#21	#13	#16	#30
#22b	#1	#28	#12
#23a	#40a	#24a	#39
#24a	#12	#31c	#15
#25	#4	#17	#1
#26	#22b	#35	#34
#27a	#44b	#22b	#28
#28	#15	#43	#47
#29	#24a	#34	#31c
#30	#32	#36a	#33
#31c	#39	#27a	#43
#32	#41	#30	#25
#33	#47	#25	#18
#34	#25	#32	#27a
#35	#33	#37a	#32
#36a	#29	#29	#29
#37a	#38	#39	#38
#38	#34	#33	#44b
#39	#17	#47	#17
#40a	#45	#45	#45
#41	#42	#42	#46
#42	#18	#18	#41
#43	#28	#40a	#37a
#44b	#27a	#41	#42
#45	#43	#38	#36a
#46	#11	#44b	#48b
#47	#48b	#46	#40a
#48b	#46	#48b	#24a

ES=Eigenfactor score, HI=Hirsch index, JIF=journal impact factor, SJR=scientific journal ranking. JIF is colored on the basis of quartile slices, starting from light gray as the first quartile to dark gray as the fourth quartile. aOA with charge, bOA free of charge, csubscription model, no superscript letter: hybrid OA

On comparing the bibliometric and scientometric data of the ophthalmology journals with full OA and hybrid OA publishing policies, no statistically significant difference was found in the types of articles published and the bibliometric data. In 37 journals with a hybrid OA publishing policy, the median OA article rate was 3.97% (minimum 0.53%–maximum 17.2%), and the citations of these articles were 5.79% (minimum 0.21%–maximum 31.36%) of the citations of all the articles. In hybrid OA journals, the OA citation to OA article ratio was median 1.17 (minimum 0.15–maximum 2.71). The OA citation to OA article ratio was similar between full OA and hybrid OA journals (P = 0.415, Mann–Whitney U-test). JIF, ES, and SJR were similar in the scientometric data (P = 0.09, 0.78, and 0.105, respectively, Mann–Whitney U-test), and HI was higher in hybrid OA journals (P = 0.025, Mann–Whitney U-test). On comparing the APCs, full OA journals charged a median 1525 USD lower than hybrid OA journals (3375 US D vs. 1850 USD, P = 0.001, Mann–Whitney U-test) [Table 4 and Fig. 1].

Table 4

The bibliometric data, scientometric data, and OA publishing policies of hybrid OA and full OA ophthalmology journals

	Hybrid OA (n=37)	Full OA (n=10)
Types of articles
All article types	21 (56.8%)	4 (40%)
All article types except case reports	16 (43.2%)	6 (60%)
Bibliometric data
Number of citable articles
2015	115 (24-336)	185 (56-930)
2016	109 (17-322)	211 (59-820)
2017	129 (14-307)	255 (65-651)
2018	129 (14-317)	280 (62-672)
2019	118 (12-328)	268 (58-532)
Total number of citations
2015	2267 (649-31,586)	920 (537-47,257)
2016	2446 (624-36,581)	1342 (718-50,452)
2017	2621 (778-37,957)	1928 (900-52,637)
2018	2829 (857-38,634)	2410 (947-52,234)
2019	2968 (1022-38,634)	2786 (896-52,362)
Percentage of citable OA articles	3.97 (0.53-17.2)	100
Percentage of citations to OA articles	5.79 (0.21-31.36)	100
Citation/article ratio	1.17 (0.15-2.71)	1
Scientometric data
JIF	2.39 (0.98-12.33)	1.7 (0.61-3.47)
ES	0.00449 (0.001070-0.04542)	0.004575 (0.00128-0.05311)
SJR	1.07 (0.43-4.49)	0.82 (0.37-1.79)
HI*	71 (38-229)	43 (16-209)
OA charge (USD)
n (37 vs. 7)**	3375 (2390-5000)	1850 (900-2390)

ES=Eigenfactor score, HI=Hirsch index, JIF=journal impact factor, OA=open access, SJR=scientific journal ranking. *P<0.05, Mann-Whitney U-test. **P<0.001, Mann-Whitney U-test

Figure 1

A comparison of scientometric data and OA fees of all journals included in the study, full OA journals, and hybrid OA journals. All data are presented as median and interquartile ranges. *Hybrid OA versus full OA, P < 0.05, Mann–Whitney U-test. **Hybrid OA versus full OA, P < 0.001, Mann–Whitney U-test OA = open access

Discussion

Scientometric indexes are widely referred to by those making decisions, but most existing indexes have been severely criticized for various reasons. The criticisms of JIF, such as being field specific and being open to manipulation by several editorial policies, are predominant.[1718] It has also been determined that three-quarters of the articles published in the top-level medical journals have fewer citations from JIF index of the journal in question.[3] The ES and SJR indexes, which consider both the number of citations and the citation prestige, have been more recently accepted than JIF and HI and have not been included in comparative evaluations with other indexes in every branch of science.[419] One of the aims of this study was to comparatively evaluate the data obtained by determining the scientometric data of ophthalmology journals.

Results obtained from our data showed strong correlations between the scientometric indexes of journals published in ophthalmology, and the strongest correlation was found between the SJR and JIF. Similar studies of hematology, and clinical allergy and ımmunology journals conducted previously have reported similar correlations as in the current study.[1112] In another study that compared the scientometric indexes of journals in the fields of radiology, nuclear medicine, and medical ımaging, higher correlation coefficients than those of the current study were obtained.[20] In a study that examined journals in the field of sleep science, the journal found to have the highest JIF value was observed to be a review journal, and JIF was almost twice as high as the next nearest journal. However, when evaluated according to HI, the two journals were ranked equal first.[4] In contrast, another study conducted of journals published in anatomy and morphology fields reported that the correlatio ns between JIF, ES, and SJR were weak and the journal ranked first according to JIF was third according to the SJR and 20th according to the ES.[5]

Studies related to hematology, and clinical allergy and ımmunology journals have shown that despite a strong correlation between different scientometric indexes, as seen in the current study, there was variability in the comparisons of the point rankings of the indexes of these journals.[1112] The results obtained in the current study suggest that the comparisons of the scientometric index are directly specific to the scientific branch publishing the journal in question and the structure of the scientometric indexes. According to the results of the present study, the correlation between the SJR and JIF was strong enough to be used interchangeably for ophthalmology journals. Roldan-Valadez et al.[1] recommended that in the selection of journals with JIF ≥1 as a preliminary scale, the SJR and ES scientometric indexes should be used together. Furthermore, the San Francisco Declaration on Research Assessment (DORA) stated that using journal-based scientometric indexes might not be appropriate, and especially JIF index should not be used.[21] However, researchers frequently use JIF as a sole factor for journal choice due to its widespread recognition. Our results show that if an ophthalmology researcher is planning on using scientometric data to understand the performance of an ophthalmology journal, rather than using JIF alone, it can be recommended that using either JIF or SJR, with either the ES or HI would provide a better understanding of scientometric performance, since these scientometric data are based on different calculation methods and different citation databases. It must also be kept in mind that the results obtained in this study were specific to journals published in ophthalmology, and similar studies are required for other scientific areas.

Researchers today use the digitalized world to retrieve peer-reviewed data, as they can have fast er access to a wider variety of journals and articles, and thus, it strengthens the scientific impact of research. The paywall to access online research remains a significant barrier, and the OA publishing model of medical journals emerged to meet this need.[22] As a result of this altered paradigm, the way authors evaluate journals and the method of allocating the financial resources required for scientific research can change the behavior of researchers. The OA model is currently accepted as an important factor in journal selection,[22] and over time, there has been a significant amount of economic support given to the OA model.[6] It has been claimed that the OA publishing model receives more citations than the subscription model, but discussions on this area in literature are still ongoing. A study based on the evaluation of articles reported that articles published in OA journals received 1.3-fold more citations.[23] Similarly, in psychiatry journals, OA publishing was characterized by an increase in citations.[8] In another study that evaluated scientometric indexes and OA fees in journals published in the field of surgery, there was an extremely weak correlation between the scientometric indexes and OA fees.[24] In the current study, the majority of the ophthalmology journals used the hybrid OA publishing model. Although the median attributable OA article percentage in these journals was low (5.37%), this value was determined to be similar to the value in both hematology journals (5.94%) and clinical allergy and immunology journals (3.26%).[1112] Only 10 journals had a full OA publishing policy, and seven of these applied the full OA model with a mandatory OA APC charge. Similar to the results reported previously for hematology, and clinical allergy and ımmunology journals, no significant difference was determined in the scientometric data favoring the full OA model.[1112] However, in another study that examined the citation rate of scientific articles in the field of ophthalmology according to the OA publishing policy, this policy was seen not to affect the number of citations.[25] In contrast, in the current study, the OA articles published in journals with a hybrid OA publishing policy were seen to receive a median of 1.17-fold more citations compared to all the articles. However, this range was very broad, from 0.15- to 2.71-fold. Consequently, when the full OA journals were compared with the hybrid OA journals, despite a positive contribution of the OA publishing policy to the scientometric indexes, it was seen that OA publishing in hybrid journals could result in more citations. Nevertheless, these results must be evaluated as specific to the field of ophthalmology.

The median cost of OA publishing is 3290 USD, and this requires a significant economic evaluation for scientists publishing in OA ophthalmology journals. Full OA journals are less expensive than hybrid OA journals, with a median cost of 1525 USD. This difference has been reported as a median of 900 USD in hematology journals and 2895 USD in clinical allergy and immunology journals.[1112] In addition to economic evaluation, journals with full OA and hybrid OA publishing policies should also be evaluated considering the scientometric indexes. A difference in favor of hybrid OA journals was observed in HI, and the other scientometric indexes were similar. In hybrid OA journals, the OA articles received a median of 1.17-fold citations than non-OA articles. However, due to the wide range of this data, it should be evaluated on a journal-by-journal basis, which is available in the relevant databases. If an article is to be published OA, it would be appropriate to select after evaluating the APC fees and the citation rate of OA articles specific to the journal.

There are several strengths of this study. We have included four different scientometric data with different calculation methods, from two different databases with different citation data. Moreover, we also determined the effect of OA publishing policies on the effect of scientometric data. However, there are some limitations to this study as well. The data were primarily evaluated on a journal basis, not on an article basis. There is a need for future article-based studies of OA in this field, as they could show different results to those obtained in this study. Another limitation was that bibliographic and scientometric data were obtained from the databases, and publishing fees were obtained from journal websites. It must be considered that there could be total or partial fee discounts for articles from undeveloped or developing countries. Journals may even feel that it is appropriate to publish OA without any charge at the request of some authors.

Conclusion

The advantages and disadvantages of how the scientometric data have been calculated and the extent of the correlations must be considered when evaluating scientometric data in ophthalmology. Our results suggest that either JIF or SJR, and either the ES or HI, could be used together rather than JIF alone. Moreover, scientometric data were similar between full and hybrid OA journals; full OA publishing in ophthalmology had no positive effect. However, in some hybrid OA journals, the OA-published articles received more citations; it would be more accurate to evaluate hybrid OA journals for OA publication, considering the OA APC fees and increase in citation counts on a journal-by-journal basis.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.