Evaluation of Scientific Outputs of Kashan University of Medical Sciences in Scopus Citation Database based on Scopus, ResearchGate, and Mendeley Scientometric Measures.

INTRODUCTION: It is essential to evaluate the impact of scientific publications through citation analysis in citation indexes. In addition, scientometric measures of social media also should be assessed. These measures include how many times the publications were read, viewed, and downloaded. The present study aimed to assess the scientific output of scholars at Kashan University of Medical Sciences by the end of March 2014 based on scientometric measures of Scopus, ResearchGate, and Mendeley.
METHODS: A survey method was used to study the articles published in Scopus journals by scholars at Kashan University of Medical Sciences by the end of March 2014. The required data were collected from Scopus, ResearchGate, and Mendeley. The data were analyzed with descriptive statistics. Also, the Spearman correlation was used between the number of views of articles in ResearchGate with citation number of the articles in Scopus and reading frequency of the articles in Mendeley with citation number in Scopus were examined using the Spearman correlation in SPSS 16.
RESULTS: Five-hundred and thirty-three articles were indexed in the Scopus Citation Database by the end of March 2014. Collectively, those articles were cited 1,315 times. The articles were covered by ResearchGate (74%) more than Mendeley (44%). In addition, 98% of the articles indexed in ResearchGate and 92% of the articles indexed in Mendeley were viewed at least once. The results showed that there was a positive correlation between the number of views of the articles in ResearchGate and Mendeley and the number of citations of the articles in Scopus.
CONCLUSION: Coverage and the number of visitors were higher in ResearchGate than in Mendeley. The increase in the number of views of articles in ResearchGate and Mendeley also increased the number of citations of the papers. Social networks, such as ResearchGate and Mendeley, also can be used as tools for the evaluation of academics and scholars based on the scientific research they have conducted.

1. Introduction

Medical research has an important role in maintaining societal health. For this reason, governments, charities, and private companies have made large investments in this sector (1). Research in this area is both costly and highly sensitive. For this purpose, studies in this area should be performed with high precision and confidence levels for both cost-benefit and cost-effectiveness purposes. Scientific articles also should be evaluated to ensure the credibility of the studies. Quantitative evaluation of scientific findings from research activities helps authorities and planners, allowing them to benefit from human and financial resources with less cost, which can optimize the economic and social structure of the country (2). The scientometric method is a quantitative evaluation method that properly balances budgets, economic costs, and increased research efficiency (3). In this approach, various evaluation methods are used, and they deliver credible research outputs and knowledge production. Reviewing the articles indexed in valid databases is the most efficient method of assessing scientific output and the overall credibility of studies. In these types of studies, quantitative measurement of scientific production may, to some extent, determine both the frequency and process of studies of any country, institution, discipline, and individual (4). However, this method is not as precise as other methods, and it has many deficiencies. For example, publications that are not cited also may be useful (5), because many non-writer experts, e.g., practitioners, Master’s students (6), public people (7), and instructors, read research papers (8) and use research publications in such activities as teaching (9) or business (10). For this reason, complementary methods have been added to the evaluation method in recent years, among which the concepts related to measures in scientific journals such as “read,” “usage,” and “download” (7) can be noted. Media or social networks provide tools that can provide the data used for this evaluation. The tool can provide a broader image of scientific impact due to increasing popularity among people (11). Bookmarking is one feature of these networks, which refers to the number of usages of a paper or document. The number of bookmarkings of an article or document shows how many times that article or document was used (12). A by-product of the academic usage of a social website is that it determines the latest statistics on the popularity of articles on social websites. This feature creates new online indexes for evaluating the effect of articles and a new category called “Altmetrics,” which is a new approach “for assessing and tracking the scientific impact of a social website” (13).

As mentioned earlier, one method cannot be used for the evaluation of research output and the assessment of its effectiveness; however, a combination of various methods may deliver useful data. Many studies have investigated the research production of different scholars in the field of medical sciences. Some studies have examined the scientific production related to such specific subjects as HIV (14), stems cells (15), and traditional medicine (16). Some studies have also investigated the research production of scholars in various universities of Medical Sciences, such as Isfahan University of Medical Sciences (2), Gilan University of Medical Sciences (17), and Iran University of Medical Sciences (18). In most of these studies, scientific products indexed in the citation databases of Web of Science (19), Scopus (20), or both databases (21) have been examined. The authors did not found a study that investigated scientific outputs at the same time on citation databases and social networks, such as ResearchGate and Mendeley. For this reason, the authors decided to conduct a study of the scientific output of scholars at Kashan University of Medical Sciences. The present study aimed to evaluate their scientific output by the end of March 2014 based on scientometric measures of Scopus, ResearchGate, and Mendeley. The specific objectives of the study were:

To determine the number of indexed articles of scholars at Kashan University of Medical Sciences in the Scopus Citation Index in terms of year, type of article, journal, and citation.

To determine the number of citations received by indexed articles of scholars at Kashan University of Medical Sciences in the Scopus Citation Index in terms of year and type of paper.

To determine the coverage of social networks of ResearchGate and Mendeley on articles from Kashan University of Medical Sciences indexed in the Scopus Citation Index.

To determine the number of views of articles of scholars at Kashan University of Medical Sciences in ResearchGate.

To determine the number of articles of scholars at Kashan University of Medical Sciences with the highest frequency of views and downloads in ResearchGate.

To determine the frequency with which the articles of scholars at Kashan University of Medical Sciences in Mendeley were read.

To determine the articles of scholars at Kashan University of Medical Sciences with the highest reading frequency in Mendeley.

To determine the correlation between the number of views of articles of scholars at Kashan University of Medical Sciences in ResearchGate and Mendeley Social Networks with citations to the articles in Scopus.

2. Material and Methods

2.1. Study setting

A survey method was used in this study. No sampling procedure was conducted in this study, and all of the articles published in the Scopus Citation Database by the scholars at Kashan University of Medical Sciences by the end of March 2014 were investigated.

2.2. Data Collection

The required data were collected from the Scopus Citation Index and two Social Networks, i.e., ResearchGate and Mendeley. Scopus was selected due to its relative homogeneity regarding all disciplines and delivery of better results. This is unlike the Science Citation Databases, which focus only on science and social science. It should be stated that the two social networks are academic and scientific and provide social bookmarking and evaluation services (22, 30). The authors followed two major steps to extract the required data as follows:

The first step: data extraction from Scopus articles

The name, Kashan University of Medical Sciences, was logged into the search section based on organizational affiliation in Scopus 2014 in order to extract the titles. All articles were reviewed due to the diversity of the writing styles of the authors and the need to determine the authors who were affiliated with the University. In addition, information on each article was saved in an SPSS file to be analyzed in the final stage. The extracted information included title, authors, authors’ affiliations with Kashan University of Medical Sciences, year of publication, type of article, name of the journal, and the number of citations of each article.

The second step: bookmark data extraction from ResearchGate and Mendeley

The titles of articles affiliated with Kashan University of Medical Sciences that were indexed in the Scopus Citation Databases were searched in two social networks, i.e., ResearchGate and Mendeley. In addition, bookmarking data on each article were recovered from the two social networks. In the case of availability of papers, reading frequency and full texts of every article were added to the data extracted in the first stage.

2.3. Statistical analysis

For statistical analysis of the data, we used descriptive statistic for objectives (1–8) and, for the eighth objective, we used Spearman’s correlative test in SPSS version 16 (SPSS, Inc., Chicago, Illinois, USA).

3. Results

3.1. First Objective: To determine the number of indexed articles of scholars at Kashan University of Medical Sciences in the Scopus Citation Index in terms of year, type of article, journal, and citation

The findings showed that 533 articles from 1960 to 2014 were indexed in the Scopus Citation Database in which at least one author was affiliated with Kashan University of Medical Sciences. In addition, 2,297 authors contributed to these articles, 1,263 of whom were affiliated with Kashan University of Medical Sciences. In fact, an average of four authors were involved per article. Moreover, 35.4% of the articles were prepared by scholars at Kashan University of Medical Sciences, and 65.4% of the articles were developed in collaboration with academics of other universities and research centers. Later assessments indicated that 157 authors had one paper. The greatest number of publications produced by one faculty member was 31 articles. Considering the type of articles, the results showed that 92.7% were research papers, and 7.3% were other types of articles. A review of the journals showed that 533 articles were published in 248 journals.

3.2. The second objective: To determine the number of citations received by indexed articles of scholars at Kashan University of Medical Sciences in the Scopus Citation Index in terms of year and type of paper

It was found out that 533 articles were cited 1,315 times. The studies showed that 277 articles were not cited. Approximately 20% of the citations were to seven articles, with 90, 35, 31, 28, 25, 24, and 22 citations (Table 1). Table 2 shows the frequency distribution of citations of the articles based on the year of publication. As shown in Table 2, only 21.2% of the articles were published prior to 2008, and 78.8% of the articles were published after 2008. However, 80% of the citations belonged to the articles published prior to 2011. Only 20% of the citations were to the articles published between 2011 and 2014. Table 3 shows the frequency distribution of then citations of articles based on the types of articles. As shown in Table 3, 95.8% of the articles were research papers, and 3.4% were review articles. Due to the ratio of research articles compared to review articles, review articles were cited more than research articles.

Table 1

Most-cited articles in Scopus

Title of article	Time cited	Type of article	Year of publication
Change of conduction velocity by regional myelination yields constant latency irrespective of distance between thalamus and cortex	90	Research	2003
Factors influencing survival after in-hospital cardiopulmonary resuscitation	35	Research	2005
Survey of echinococcosis and hydatidosis in Kashan region, Central Iran	31	Research	2006
A model for empowerment of nursing in Iran	28	Review	2005
Human homology and candidate genes for the Dominant megacolon locus, a mouse model of hirschsprung disease	25	Research	1997
Childhood cutaneous leishmaniasis: report of 117 cases from Iran	24	Research	2006
Involvement of thromboxane A2 (TXA2) in the early stage of oleic acid-induced lung injury and the preventive effect of ozagrel, a TXA2 synthase inhibitor, in guinea-pigs	22	Research	2004

Table 2

Frequency of citations by year of publication

Year of Publication	Articles		Time cited		Cumulative frequency
	n	%	n	%
1997	1	0.2	25	1.9	1.9
1999	1	0.2	3	0.2	2.1
2000	3	0.6	20	1.5	3.6
2001	1	0.2	12	0.9	4.6
2002	1	0.2	6	0.5	5
2003	8	1.5	107	8.1	13.2
2004	5	0.9	56	4.3	17.4
2005	16	3	135	10.3	27.7
2006	60	5.6	172	13.1	40.8
2007	22	4.1	110	8.4	49.1
2008	25	4.7	92	7	56.1
2009	49	9.2	186	14.1	70.3
2010	42	7.9	124	9.4	79.7
2011	79	14.8	142	10.8	90.5
2012	99	18.6	94	7.1	97.6
2013	135	25.3	41	2.4	100
2014	16	3	0	0	100
Total	533	100	1315	100

Table 3

Frequency distribution based on the type of paper

Type of article	Articles		Time cited
	n	%	n	%
Research Article	494	92.7	1260	95.8
Review Article	14	2.6	45	3.4
Letter	12	2.3	9	0.7
Other	13	2.5	1	0.1
Total	533	100	1315	100

3.3. The third objective: To determine the coverage of social networks of ResearchGate and Mendeley on articles from the Kashan University of Medical Sciences indexed in the Scopus Citation Index

It was found out that 395 articles were indexed in ResearchGate among 533 articles indexed in Scopus. Full texts of 214 articles were available among 395 indexed articles. In addition, 395 indexed articles in ResearchGate were viewed 20,799 times and downloaded 2,324 times. Moreover, 234 articles were indexed in Mendeley among 533 studied articles. Full texts of 36 articles were available. All of these articles were viewed 901 times in this social network (Table 4). Approximately 48% of indexed articles in Scopus were cited at least once. The average number of citations per article was five. The articles were covered by ResearchGate (74%) more than Mendeley (44%). Furthermore, 98% of the articles indexed in ResearchGate and 92% of the articles indexed in Mendeley were viewed at least once. As observed in Table 4, any article bookmarked in ResearchGate was viewed 54 times on average. In contrast, any article bookmarked in Mendeley was viewed four times.

Table 4

Scopus and social networks coverage summary of the literature review

Network	Indexed documents (n)	All events (citations & view)	Non-zero incidents; n (%)	Average number of non-zero incidents	Max	Min
Scopus	533	1315	256 (48.03%)	5.13	90	0
ResearchGate	395 (73.69%)	20799	387 (97.97%)	53.74	269	0
Mendeley	234 (43.65%)	901	216 (92.3%)	4.17	61	0

3.4. The fourth objective: To determine the number of views of articles of scholars at Kashan University of Medical Sciences in ResearchGate

Examining the frequency of viewing the articles in ResearchGate showed that 54% of the articles were viewed between “one to 50 times,” 32% of the articles were viewed between “50 to 100 times,” and 11% of the articles were viewed “more than 100 times.” Only 2% of the articles were never viewed. Table 5 shows the three articles that had the highest number of views.

Table 5

Most viewed articles in ResearchGate

Title of article	Year of Publication	Times Visited	Type of article
A comparative study of the therapeutic effects of the Zataria multiflora vaginal cream and metronidazole vaginal gel on bacterial vaginosis	2008	269	Research Article
Occupational exposure to manganese-containing welding fumes and pulmonary function indices among natural gas transmission pipeline welders	2012	244	Research Article
Ischiorectal block with bupivacaine for post hemorrhoidectomy pain	2012	181	Research Article

3.5. The fifth objective: To determine the number of articles of scholars at Kashan University of Medical Sciences with the highest frequency of views and downloads in ResearchGate

Maximum number of views in ResearchGate was 269 times. The results showed that five of the 29 articles (20%) that were viewed the most in ResearchGate were review articles. All of articles were downloaded 2,324 times. One article was downloaded 239 times which had highest rate of downloads. On average, any bookmarked article was downloaded six times in Mendeley. Table 6 shows that five articles (20%) were downloaded the most often. Table 6 shows that an article entitled “Gastrointestinal Parasites of Stray Cats in Kashan, Iran” had the highest number of downloads (239 times, 10% of the downloading frequency). Also, two of the five articles that had the highest number of downloads in ResearchGate (20%) were review articles.

Table 6

Most-downloaded articles in ResearchGate

Title of article	Year of Publication	Type of article	Times Downloaded
Gastrointestinal parasites of stray cats in Kashan, Iran	2009	Research Article	239
In vitro and in vivo investigations on bone regeneration potential of laminated hydroxyapatite/gelatin nanocomposite scaffold along with DBM	2012	Research Article	82
Long term potentiation as a mechanism for learning and memory	2009	Review Article	67
Radio-protective role of antioxidant agents	2012	Review Article	67
A multicenter study of clinical and laboratory findings of palindromic rheumatism in Iran	2012	Research Article	66

3.6. The sixth objective: To determine the reading frequency of articles of scholarsat Kashan University of Medical Sciences in Mendeley

Examining the reading frequency of the articles in Mendeley showed that 86% of the articles were read between “1 to 10 times,” 5% of the articles was read between “11 and 20 times,” and 2% of the articles were read “more than 20 times.”

3.7. The seventh objective: To determine the articles of scholars at Kashan University of Medical Sciences with the highest reading frequency in Mendeley

Studied articles in Mendeley were viewed 901 times. Table 7 shows that 20% of the articles had the highest number of views in Mendeley. The highest number of views in Mendeley was 61 times. Two of the five articles maximally viewed (20%) in Mendeley were review articles.

Table 7

Most viewed articles in Mendeley

Title of article	Year of Publication	Times Downloaded	Type of article
Change of conduction velocity by regional myelination yields constant latency irrespective of distance between thalamus and cortex	2003	61	Research Article
Differences in integrin expression and signaling within human breast cancer cells	2011	48	Research Article
Exercise in the metabolic syndrome	2012	30	Review Article
A model for empowerment of nursing in Iran	2005	24	Review Article
Undergraduate nursing students’ compatibility with the nursing profession	2005	20	Research Article

3.8 The eighth objective: To determine the correlation between the number of views of articles of scholars at Kashan University of Medical Sciences in ResearchGate and Mendeley Social Networks with citations to the articles in Scopus

The correlation between the number of views of articles in ResearchGate with citation number of the articles in Scopus and reading frequency of the articles in Mendeley with citation number in Scopus were examined using the Spearman correlation. The results showed a significant relationship between the variables with 99% confidence level and an error level less than 1%. In this regard, the correlation coefficient between two variables of the number of views of articles in ResearchGate with citation number of the articles in Scopus was 0.310, which showed a positive relationship between the two variables. It can be concluded that increased number of views of articles in ResearchGate increased citations to the articles. The correlation coefficient between the two variables of reading frequency of the articles in Mendeley with citation number of the articles in Scopus was equal to 0.247, which indicated a positive relationship between the two variables. Thus, the increased number of views of articles in Mendeley increased the citations to the articles. However, the correlation between the numbers of views of articles in ResearchGate was associated with higher citations of reading frequency of the articles in Mendeley with the number of citations to the articles.

4. Discussion

The results showed an increase in publication of articles in recent years. In terms of article type, 92.7% covered “original research,” 2.6% encompassed “review articles,” and 4.7% was allocated to other types of articles. Investigating scientific products of Shahre-kord University of Medical Sciences showed that 74.6% of scientific production in the Web of Science consisted of research papers (14). Most scientific products of patient rights in WOS covered original articles (66.10%). It can be observed that the nature of scientific production in the field of medical science significantly was the research format, i.e., medical science produced mostly research papers. Examining the citations to articles showed that the highest number of citations to an article was 90 times. In addition, 48% of the articles indexed in Scopus were cited at least once. Examining 1,613 articles in Nature and Science Journals in 2007 showed that all the articles were viewed at least once and a maximum of 1000 times in the Science Citation Index (32). Sotoudeh, Mazarei and Mirzabeygi investigated articles of 83 journals in the fields of Librarianship and Information Science and showed that 22.17% of the articles were cited at least once in WOS, and 22.6% of these papers were bookmarked in CiteULike (26). Coverage of articles and the number of bookmarking were higher in ResearchGate than in Mendeley. In addition, access to full text of the articles was higher in ResearchGate than in Mendeley. This showed the popularity of ResearchGate as a research scientific social network among the community of Medical Sciences. A study entitled “Does ResearchGate show researchers’ performance?” achieved the same results. This was because the correlation between all indices was reported on average or high levels (27). In most studies, the results showed excellent coverage of Mendeley on articles of most scientific fields (28, 25). Thelwall and Wilson (2015) reviewed the articles in the field of Medical Science indexed in Scopus in 2009 and showed that 89% of the papers were saved in Mendeley, 78% of the articles were at least read once in Mendeley, and 73% of the articles were cited at least once in Scopus (28). Mohammadi et al. (2015) studied the papers in four areas of clinical medicine, engineering and technology, social sciences, physics and chemistry, which were indexed in WOS in 2008. They showed that 41% of the articles were stored in Mendeley (25). Another study showed that 80% of articles of Public Library of Science were covered by Mendeley, 31% were covered by CiteULike, and 10% of the articles were covered by Delicious (11). In addition, 355 studied articles were viewed 20,799 times in ResearchGate and 901 times in Mendeley. Furthermore, 98% of the articles indexed in ResearchGate and 92% of the articles indexed in Mendeley were viewed at least once. Any bookmarked articles in ResearchGate were viewed 54 times on average. Any bookmarked article in Mendeley was viewed four times on average. The results showed that mean viewing number of the articles in ResearchGate was 10 times the mean number of citations to papers. These findings indicated that the articles were read by many people but were rarely cited. Therefore, viewing number of the articles shows how many times the articles were used in other contexts, such as teaching and business. Mohammadi et al. (2015) reported that the mean reading number of the articles in the field of social sciences was twice the mean number of citation to the articles in Mendeley. This was reversed in case of articles in the fields of chemistry, physics, and clinical medicine. The mean reading number of the articles was equal to the mean number of citation to the articles in the field of engineering and technology (25). Another study showed that the number of bookmarked works was higher than the mean number of citations in Mendeley (3.32% bookmarking vs. 2.17% citations). The highest number of bookmarked works was observed in the field of Health and Biomedical Sciences (13.6%) (29). Mohammadi and Thelwal (2015) showed that the mean number of saved articles in all disciplines, except psychology, was higher than the mean number of citations to articles in Mendeley (24). These results are not in line with those obtained by Sotoude, Mazarei and Mirzabeygi (2015). They showed that the number of bookmarked articles in CiteULike was negligible compared to the number of citation to the articles in WOS (26). Although only 2.6% of the studied papers were review articles, these article covered part of 20% of the most visited and downloaded articles. This suggests the importance of review articles. Review articles are an important category of medical papers, which are mostly read and associated with high credibility due to importance and integrity. Editors are mostly interested in such articles because a review article written by a well-known author is widely welcomed, read more, and referred more, which increases the impact factor of the journal. Readers are also interested in this type of articles as the perfect solution to meet the increasing spread of knowledge and acquire up-to-date information.

The results indicated that the correlation coefficient between the two variables of viewing number of the articles in ResearchGate with citations to the article in Scopus and the correlation coefficient between two variables of reading frequency of the articles in Mendeley with the number of citations to articles in Scopus were positive and significant. It can be concluded that increased number of viewed articles in ResearchGate and Mendeley increased the number of citations to articles. Bar-Ilanet al. (2012) showed that 81.6% of works of 57 scholars in field of Scientometrics indexed in Scopus were covered by Mendeley. The results showed that the correlation coefficient between the number of citations to articles in Scopus and reading frequency of the articles in Mandeley was equal to 0.448 (33). Mohammadi and Thelwal (2013) showed a significant and positive correlation between the number of bookmarked articles in Mendeley and number of citations to articles in studied fields (22). Li and Thelwal (2012) also showed a significant correlation between citations and altmetrics bookmarking index in Mendeley for some genetic articles published in 2008 (34). Thelwall and Wilson (2015) showed a significant and positive correlation between frequency of bookmarking in Mendeley and the number of citations in Scopus in the field of medical sciences (28). Although the correlation between altmetrics measures and the number of citations in citation indexes does not prove that the former leads to the latter, the study of this issue is the first step to rational evaluation of an altmetrics measure. A positive correlation indicates that the two are not completely apart from each other and have common similarities. A significant correlation between bookmarking and citation is not an evidence of causality but probably indicates that some readers have cited the papers. The significant correlation between the two variables shows that bookmarking and citation reflect similar aspects of impact of an article. However, the correlation between the two variables was not strong enough to conclude that the number of citations and bookmarking reflects the same image of significant impact of research. It seems that frequency of bookmarking shows reading frequency of an article, which combines the two impacts of research and broad expertise.

5. Conclusions

The findings showed better coverage and more number of visitors in ResearchGate than Mendeley. This may be due to the fact that most ResearchGate users are working in the fields of medical sciences and biology. High viewing number of articles in ResearchGate and Mendeley suggests the high impact of the two social networks in increasing the visibility of scientific works. It is recommended that medical scholars use this network as a means of “self-archiving” and “finding information.” Also, the Ministry of Health, universities, research institutions, and research centers should be aware of the importance of attendance and membership of scholars, faculty members, and even students in social networks, particularly higher education students. This probably will be used as a criterion for scientific – research assessment of scholars and academics in science policy-making.