A 5-year scientometric analysis of research centers affiliated to Tehran University of Medical Sciences.

BACKGROUND: Since Tehran University of Medical Sciences (TUMS) has the oldest and highest number of research centers among all Iranian medical universities, this study was conducted to evaluate scientific output of research centers affiliated to Tehran University of Medical Sciences (TUMS) using scientometric indices and the affecting factors. Moreover, a number of scientometric indicators were introduced.
METHODS: This cross-sectional study was performed to evaluate a 5-year scientific performance of research centers of TUMS. Data were collected through questionnaires, annual evaluation reports of the Ministry of Health, and also from Scopus database. We used appropriate measures of central tendency and variation for descriptive analyses. Moreover, uni-and multi-variable linear regression were used to evaluate the effect of independent factors on the scientific output of the centers.
RESULTS: The medians of the numbers of papers and books during a 5-year period were 150.5 and 2.5 respectively. The median of the "articles per researcher" was 19.1. Based on multiple linear regression, younger age centers (p=0.001), having a separate budget line (p=0.016), and number of research personnel (p<0.001) had a direct significant correlation with the number of articles while real properties had a reverse significant correlation with it (p=0.004).
CONCLUSION: The results can help policy makers and research managers to allocate sufficient resources to improve current situation of the centers. Newly adopted and effective scientometric indices are is suggested to be used to evaluate scientific outputs and functions of these centers.

Introduction

Recently, high priority is being given to regular evaluation of scientific outputs of research groups, institutions, universities, and research institutes and centers to know about their current status (1, 2). Scientometric analysis of scientific publications has become a challenging and inevitable necessity for many research institutes and organizations (1).

Scientometric results are often used for decision making on budget,and appointment and promotion of the researchers (3). Moreover, these studies can provide policy makers and planners with important evidence of the results and effects of research programs (4). Scientometric indicators are also used for grading research centers (5, 6) and universities (7, 8).

So far, scientometric studies have been mostly performed on the scientific products of the research’s fields (9-14), laboratory groups (15), universities and scientific institutes (6, 8,16), and even countries. Although few studies have evaluated the scientific output of research centers and institutes (2, 17), they have not compared the centers with each other. Due to the confidentiality of the findings of such studies, only certain results are reported. Different studies have employed different scientometric indicators (18). Though few/no studies have used…indicators to report research centers status and performance.

In Iran, the research performance of medical universities and their affiliated research centers has been annually evaluated under the supervision of the deputy for research of Iranian Ministry of Health and Medical Education (MOHME) since 2000. Paper publication, which is a major part of knowledge production, is one of the most important aspects of these evaluations (19). Based on the objectives of these evaluations, modernand important scientometric indicators are not used and the performance of the research centers is not analyzed. Tehran University of Medical Sciences (TUMS) is the oldest center of modern higher education in Iran and has ranked first in the annual evaluations of research outputs of medical universities for twelve consecutive years. Based on the SCImago Institutions Rankings, TUMS ranked 306 among 4851 institutions in 2014. It has the largest number of students, faculty members, and research centers among all Iranian medical universities. Since scientometric indicators provide a comprehensive insight to the scientific output of research centers, this study was designed and conducted to analyze the performance of TUMS’ research centers and identify the factors affecting the their scientific output through scientometric indices.

Methods

This cross-sectional scientometricstudy was conducted on the research centers of TUMS whose approval dated back to 3 years before the study implementationand included in the annual evaluation reports of MOHME. Thirty-six eligible centers were included in the study.

Data were collected using the following method:

a) An author-made questionnaire which was completed in the centers: The questionnaire had four sections. Section one included basic data, such as the mandate of the center, years of activity, and budget. Section two addressed real and non-real properties of the centers. Section three and four were about partner institutions and research fields, respectively. For validity and reliability issues, the questionnaire was piloted in three research centers (Addiction, AIDS, and Mental Health Research Centers) and necessary modifications were made accordingly‏.

‏ b) Data of scientific products of research centers including, papers (published in journals indexed in valid databases, and non-indexed papers published in foreign scientific journals and domestic scientific research journals), books and patents registered in the name of the research centers between 2007 and 2011 were derived from the reports of the annual evaluations of MOHME through the website of the MOHME Deputy for Research‏.

c) To calculate hybrid scientometric indicators, the number of each centers’ papers (original articles, brief articles, reviews, editorials, and letters to the editor) indexed in Scopus database and the number of their citations from the time of establishment to the year 2012 were collected. To make sure of including all affiliations of research centers, ISI Web of Science was used in addition to Scopus, as well‏.

d) Research personnel performance (i.e. their participation in scientific media) was derived from Scopus database. The 5-year performance (2007-2011) of each research center was based on scientific activities of researchers with the affiliation of that center‏.

The following scientometric indicators were calculated after data collection: a) h-index, as the most common and popular scientometric indicator, which is used to evaluate the quality and impact of the scientific products of researchers (20) and research institutes (21), b) g-index, introduced by Egghe to improve h index, which in contrast to h-index increases with citations instead ofthe threshold and gives more weight to publications that have a better quality and are more frequently cited (21), c) hg-index, the geometric mean of h and g index, which increases their advantages and reduces their errors (1), d) A-index which shows mean citations in the h-core (22), e) the R-index measures the h-core’s citation intensity (22), f) e-index which was introduced by Zhang and includes excess citations to papers in h-core (22), g) impact index which shows high impact papers of the center relative to all papers published by the same center (16), h) citation per paper, which makes it possible to compare researchers and research institutes with different research background (24), and i) the ratio of the number of papers to the number of researchers. It should be mentioned that to observe confidentiality regarding the information of the centers, the name of the research centers is not reported and their determinants were reported using random numbers assigned to each center.

Using SPSS 20, data were first analyzed descriptively and the trend of the publications during the 5-year was shown as a line chart. Factors affecting the scientific productions of the research centers between 2007 and 2012 were also studied: the age of the center (years from establishment to 2012), mean of funds allocated to research contracts in each center, type of budget (having or not having a separate budget line), the mandate of the center (clinical/basic science), the area of the center (square meter), real property (the center was categorized as low, intermediate, or high based on having a clinic, inpatient ward, library, laboratory, workshop, conference hall and salon, and type of ownership (complete, mutual, no ownership)), non-real property (the center was categorized as low, intermediate, or high based on having workshop, laboratory and audiovisual equipment), research fields (the environments in which the research center implements its research activities), partner organizations (foreign or domestic institutes, centers, or universities with which the research center had financial, research, educational, or intellectual ties), number of joint projects with partner organizations, and the number of research council sessions. Univariate linear regression followed by multiple linear regressions were used to determine the factors affecting the scientific productions of the research centers. P values less than 0.05 were considered statistically significant.

Results

Of 36 eligible centers, 30 agreed to participate in the study (response rate=83.3%). The major scientific output of the research centers was papers (n=4867), of which 94.4% (n=4593) were indexed in ISI Web of Science, Medline/PubMed, Biological Abstracts, Scopus, Embase, and Chemical Abstracts. Moreover, 58.04% (n=2825) of the papers were indexed in ISI Web of Science. The median number of the 5-year papers of the evaluated research centers was 150.5 and the median number of 5-year indexed papers was 142. In addition, we evaluated the papers indexed in Scopus and their citations since the establishment of each center to the end of 2012 and the results showed that the median number of the papers was 127, the median citation was 515, and the median number of non-cited papers was 38.50 (Table 1-3).

Table 1

The 5- yearsscientific productions of the research centers, from the reports of the MOHME annual evaluations

Research centers’ ID	patents	books	indexed papers *	papers
3	1	9	383	405
7	1	5	346	395
2	0	2	392	393
27	1	1	304	308
14	11	0	206	238
29	0	4	165	182
28	0	6	175	180
20	1	6	174	178
8	0	0	154	176
22	2	0	164	164
16	0	4	146	162
23	0	4	139	161
21	0	5	152	160
11	0	2	152	154
24	0	6	145	151
5	1	2	146	150
13	0	1	139	142
30	0	5	131	136
19	1	2	123	132
9	0	5	124	125
17	0	0	98	116
18	0	8	98	108
1	0	0	106	106
15	0	6	85	91
12	0	2	73	75
26	0	0	73	73
10	0	3	71	72
4	0	0	70	70
6	7	0	46	49
25	0	9	13	15
total	26	97	4593	4867

Table 3

Descriptive statistics of the indexed papers in Scopus and their citations

	median	Mean	IQR	SD	Max	Min
Papers	127	147.5	103.0	119.90	632	32
Citations	515	850.9	103.0	1221.10	6629	73
Non-cited papers	38.50	49.7	41.0	34.50	142	7
Papers with at least one citation	79.50	97.7	78.0	96.40	517	12

The total number of published papers and the total number of indexed papers had an increasing trend until 2009 but there was a decrease in 2010 and 2011. Moreover, there was a decrease in the publication of books in 2011 (Fig. 1 ).

Fig.1

The trend of the total papers and indexed papers

The median number of published books was 2.5 with standard deviation (SD) of 2.8, and the median number of patents was zero with a SD of 2.3 (only 9 centers had patents) during 2007-2011. Moreover, the median of the number of papers to the number of researchers between 2007 and 2011 was 2.26 (Table 2).

Table 2

descriptive statistics of the 5- years scientific productions

	Median	Mean	IQR	SD	Max‏.	Min‏.
5‏-year papers	150.5	162.2	76.0	97.90	405	15
Indexed 5-year papers	142	153.1	73.0	92.30	392	13
Books	2.5	3.2	5.2	2.80	9	0
Patents	0	0.8	1.0	2.30	11	0
Papers per Author	2.9	2.2	1.0	1.90	8.9	0.8

The scientometric indicators were calculated based on papers indexed in Scopus. The median citation per paper was 3.7; h-index which shows the quality and impact of scientific productions had a median of 11. The median g-index, hg-index, and A-index were 17.5, 14.3, and 22.9 respectively and the median R-index, e-index, and impact index were 16.5, 11.2, and 1.6, respectively (Table 4 and 5).

Table 4

The hybrid scientometricsindicators for each research centers

Researchcenters’ ID	C/P *	Impact index	h-index	A-index	R-index	e-index	hg-index	g-index
2	10.5	2.8	37	60.5	47.3	29.5	43.9	52
3	6.1	3.1	34	52.6	42.3	25.1	39.5	46
28	10.1	2.9	24	48.4	34.1	24.2	29.8	37
24	8.0	3.0	23	46.8	32.8	23.4	28.7	36
8	5.5	2.6	22	50.1	33.2	24.9	28.5	37
22	12.6	3.5	21	43.6	30.2	21.8	25.9	32
17	8.6	2.6	19	27.7	22.9	12.9	21.8	25
29	6.2	1.9	16	41.7	25.8	20.3	21.5	29
15	5.8	2.2	15	23.9	18.9	11.5	17.3	20
13	9.7	2.3	14	46.4	25.5	21.3	19.8	28
5	5.3	2.0	14	27.1	19.5	13.5	17.1	21
19	3.4	1.8	13	26.0	18.4	13.0	16.1	20
16	3.6	1.7	12	22.0	16.2	10.9	14.3	17
21	3.3	1.6	12	21.0	15.9	10.4	14.3	17
7	2.4	1.2	11	25.5	16.7	12.6	14.1	18
26	6.0	1.8	11	31.1	18.5	14.9	14.8	20
14	2.1	1.1	10	21.9	14.8	10.9	12.6	16
18	8.4	1.9	10	43.6	20.9	18.3	14.8	22
27	3.4	1.6	10	15.3	12.3	7.3	11.4	13
30	2.9	1.5	9	16.6	12.2	8.2	10.8	13
9	2.6	1.1	8	21.2	13.0	10.2	10.6	14
20	2.6	1.3	8	15.7	11.2	7.9	9.8	12
4	3.8	1.6	8	15.1	11.0	7.5	9.8	12
1	2.2	1.0	8	15.4	11.1	7.7	9.8	12
23	2.2	1.2	7	12.9	9.5	6.4	8.4	10
6	4.2	1.5	7	18.6	11.4	9.0	9.2	12
12	1.9	1.2	6	11.5	8.3	5.7	7.3	9
10	2.7	1.2	6	13.0	8.8	6.5	7.3	9
11	1.9	0.9	5	11.4	7.5	5.6	6.3	8
25	2.3	1.0	4	15.0	7.7	6.6	5.6	8

*Citation per paper

Table 5

Evaluation of the 5-year scientific productions’ determinants with univariate linear regression

Determinants	B	Sig.
Number of researchers	-1.80	<0.001*
Number of Foreign partner organizations	16.54	0.007*
Type of budget (separate budget line)	-65.47	0.106
Number of Total partner organizations	2.41	0.169
Age of the center	5.88	0.267
Number of Domestic partner organizations	1.79	0.411
Non-real property (low, intermediate, high)	18.10	0.418
Number of research council sessions	0.93	0.440
mean budget of the research contracts	3.14	0.481
Number of Research fields	-1.99	0.649
Real property (low, intermediate, high)	-8.74	0.706
Number of joint projects	-0.59	0.752
the mandate of the center (clinical)	-7.85	0.840
Area of the center (square meter)	0.003	0.899

Of the 30 centers, 22 (73.4%) did not have a separate budget line and 20 centers (66.7%) had clinical activity. Regarding real property, 43.3% (n=13), 33.4% (n=10), and 23.3% (n=7) of the centers were high, medium, and low, respectively. As for non-real property, 33.4% (n=10), 33.3% (n=10), and 33.3% (n=10) of the centers were high, medium, and low, respectively. The mean±SD age of the centers was 9.8±3.5 years, and the mean±SD number of researchers was 64.3±46.5. The mean±SD number of partner organizations was 7.7±10.4 and the mean±SD number of the sessions of the research council was 33.7±19.5.

The variables of budget (having or not having a separate line), the mandate of the center (clinical/ basic science), number of research personnel, real property (high, medium, low), non-real property (high, medium, low), area, age, mean budget of the research contracts, number of foreign and domestic partner organizations, number of joint projects, number of research fields, and the number of the sessions of research council were investigated with a univariate linear regression model and the results showed that the number of research personnel (R2=0.72, p<0.01) and the number of foreign partner organizations (R2=0.25, p=0.007) had a significant direct relationship with the publication of papers (Table 5).

Multiple linear regression was used to assess the effect of the variables. Based on the multiple model, younger age of the center (p=0.001), number of research personnel (p<0.001), and having a separate budget line (p=0.016) had a significant direct correlation while real property (p=0.004); number of joint projects (p=0.001) had a significant inverse correlation with the number of published papers (Table 6).

Table 6

Evaluation of the 5-year scientific productions’ determinants with multivariate linear regression

Predictor variables	Beta	Sig.
Number of researchers	0.95	<0.001
Age of the center	-0.27	0.001
Number of joint projects	-2.01	0.001
Real property (low, intermediate, high)	-0.31	0.004
Type of budget (separate budget line)	-0.16	0.016

Discussion

In this study, we used scientometric indicators to evaluate the 5-year performance of the research centers of TUMS and identify the factors affecting their performance.

Our study showed that the median number of the papers in the 5 years was 150.5 and the median number of papers indexed in ISI Web of Science was 74. The total number of published papers and the total number of indexed papers of the evaluated centers had an increasing trend until 2009 but decreased slightly in 2010 and 2011, which could be due to the special attention of TUMS to research activities and publication of scientific papers in recent years. Moreover, the results showed that the majority of the papers (58.04%) were indexed in ISI Web of Science, indicating that attention was paid to quality of the papers besides the increase in their number. In Iran, studies performed by Sotoudeh (25) and Hayati (26) showed that the number of the Iranian papers published in international journals was the highest in recent years. Moreover, a study by Eskrootchi (27) showed that Iranian scientific papers had an increasing trend during 30 years. No similar investigation has been performed on evaluation of scientific output of several research centers but a study on the 36-year scientific products of Pasteur Institute of Iran in the Web of Science revealed that 823 papers were published by the institute until the end of 2009, indicating the high speed of science production between 2005 and 2009 (28). Publication of papers in different scientific fields has had an increasing trend in recent decades worldwide (29-34). The mean citation was 850.9 with a standard deviation of 1221.1 in the present study, the median of the citations per paper was 3.7, and the mean number of papers without citation was 49.7. In a study by Bazrafshan, the total number of citations to the papers published by Pasteur Institute of Iran until the end of 2009 was 4397 and the indicator of citations per paper was 5.2 (28), indicating that the scientific impact of the papers of Pasteur Institute was more than the evaluated research centers.

Hybrid metrics are a subset of qualitative indicators which are calculated to express both the productivity and effectiveness in a numeral (35). In scientometric studies, h-index is usually calculated for researchers, research fields and groups, universities, and even countries, and a limited number of studies have used h-based indicators worldwide. For example, in the 10 best medical universities of the US, the mean h index was 52.5 between 2000 and 2002, 50 between 2003 and 2005, 35.5 between 2006 and 2008, and 15.5 between 2009 and 2011 (36). In a study by Wang in National Taiwan University, h index of the groups of optics, pharmacology, oncology, and general and internal medicine was 25, 35, 42, and 25, respectively (37). Hendrix evaluated medical universities of the US in terms of scientometric indicators and reported that the median impact index, an h-based indicator, was 3.16 (16) while the h index and g-index of TUMS were 46 and 66 until 2010, respectively (38). On the other hand, a study by Bornmann et al (39) showed that hybrid metrics in post doc researchers of biomedicine were lower; the median of h, g, A, and r indices were 2, 3, 23.5, and 8.09, respectively. Moreover, the scientific performance of the research centers and institutes were not evaluated using these indices.

Advanced scientometric indicators were used in the present study. The median of h index, as the most common scientometric indicator, was 11 and the median of g, e, A, hg, indices and impact index were 17.5, 11.2, 22.9, 14.3, 16.5, and 1.6, respectively. Since these indices are affected by both the number of papers and citations, research centers should not only focus on the quantity of the papers but should regard their quality, as well.

In our research, the factors affecting the paper publication of research centers were also evaluated. The research centers in our study were divided to clinical and basic sciences. Based on the results of multiple logistic regression, there was no significant difference between paper publication and the mandate of the center. Moreover, there was an inverse significant correlation between the age of the centers and the number of published papers. The age of the research centers and their scientific performance showed an inverse negative correlation; in line with a study by Bonaccorsi (40). The reason could be that the younger centers find themselves in competition with older centers and try more. Moreover, younger centers may have better research and publication capabilities. However, a scientometric study performed by Díaz-Morán et al in different psychiatry and legal medicine in the University of Barcelona showed that the h-index of researchers in newly founded groups was lower (11).

In recent years, the share of the governmental and non-governmental research budgets has increased substantially in Iran (41). In our research, although the relationship between the mean budget of the research contracts and the number of papers in 5 years was not significant, there was a significant correlation between the number of papers and budget line. Having a separate budget line showed a significant linear relationship with paper publication. Moreover, findings showed a significant positive relationship between the number of research personnel and the number of papers in 5 years. Hendrix also reported that the annual budget and number of faculty members had a significant positive effect on the number of papers published by US medical schools and their citations, and also stated that budget affected the quantity of research more than its quality (16). Moreover, Rey-Rocha et al found that organized and integrated research teams had higher productivity (42).

In our study, the average of the number of papers in 5 years to the number of researchers was 2.9, while Hendrix reported that the average of the number of papers to the number of faculty members in US medical schools was 10.2 (37) and Bonaccorsi et al found that the average of the number of patients to the researchers of Italian National Research Council (CNR) was 5.7 (40), indicating a lower rate of paper production per researcher in our study.

The results showed that the area and non-real properties of research centers like laboratory and audiovisual equipment had no significant relationship with the number of papers while the correlation of real properties with the number of papers was significant and inverse. Bonaccorsi also found a negative significant correlation between the dimensions of the research center and its scientific production, i.e. smaller research institutes had more scientific production (40). However, Lissoni et al found that an optimal work environment had a great effect on scientific productivity (13).

In our study, the correlation of the number of papers with foreign and domestic partner organizations, number of research fields, and number of research council sessions was not significant while a significant negative relationship was observed between the number of papers and the number of joint projects. The reason could be that the results of a number of joint projects were not published in the form of a paper. On the other hand, since partner organizations may be scientifically weak, unknown, or invalid, joint projects may have little effect on the quantity and quality of the scientific products of the research centers while a study by Lissoni et al showed that partnership in big or international projects had a strong positive effect on the quality and quantity of publications (13). Moreover, Zhou et al performed a study on scientific publications in China and found that international collaborations in China increased substantially in a period of seven years which could positively affect the quality and quantity of scientific products (43).

Conclusion

The results of our study showed the increasing trend of the publication of papers in TUMS research centers. Continuation of this increasing trend depends on different factors such as improving research infrastructures like the number of specialized researchers, increasing the budget of research activities and projects, and holding educational courses on scientific writing. It should be mentioned that although Tehran University of Medical Sciences has many prestigious research centers that are active in different areas but the results of this study cannot be generalized to all research centers in Iran.

Since scientometric studies are important to evaluate the quality and effectiveness of the scientific productions of research centers, the results of this study can be employed for improving the current situations, optimal allocation of the resources, persuasion of strong research centers, and supporting weaker centers. Therefore, it is necessary that managers and policy makers pay more attention to the results and findings of this studies at university and national levels and make grounds for scientometric studies through providing access to credible databases such as ISI Web of Science and Scopus. Advanced indicators should be employed to evaluate the scientific output and performance of the research centers in order to provide the grounds for improving the quality and quantity of their performance.

It is also suggested that researchers in the field of scientometrics use hybrid metrics in addition to scientometric and productivity indicators to better evaluate the quality and impact of the scientific output of researchers, research centers, scientific groups, and universities.

Acknowledgements

This study was part of a registered thesis conducted in TUMS (registration number: 20332-102-04-91). The authors would like to extend their gratitude to research centers who participated in this study and all the people who helped us during this research.

Conflict of interest

The researchers were students and faculty members of Tehran University of Medical Sciences and the project was partly funded by the Vice-Chancellor for Education and partly by the Vice-Chancellor for Research. It should be mentioned that at the time of the study, Dr. Rahimi-Movaghar and Dr. Ghalichi each worked in one of the evaluated centers and Dr. Rahimi-Movaghar was in charge of TUMS Research Centers Coordination Office. The rest of the research team had no conflicts of interest with the centers.