Effect of university policies on research productions: a scientometric study.

BACKGROUND: This study aimed to assess the effect of new package of interventions on scientific productions rate in Shahid Beheshti University of Medical Sciences.
METHODS: Through a health system research, we extracted policies from the strategic plan of the university and 10 interventions were developed to increase the scientific productions in terms of quality, quantity and commercialization and to develop infrastructure for research in health service provision and education. For evaluating the effectiveness of interventions, citation and publication indicators for individuals and schools were analyzed using descriptive statistics and t-test. They were extracted from Scopus and ISI web of knowledge during period of 1/1/2009 to 30/5/2012.
RESULTS: There was an increasing trend in scientific productions from 2009 to mid-2012. We found 60 percent of total scientific productions of the university were published during last 3.5 years. During this 3.5 years, 10 more percentile of faculty members involved in research. Schools of pharmacy, Medicine and Health had the highest scientific products. Mean for h-index was 1.5 (SD=2.49) in ISI and 1.9 (SD=2.89) in Scopus database (p<0.001).
CONCLUSION: Effective policies and interventions lead to 46% increase in scientific productions from 2009 to 2010 and 56% increase from 2010 to 2011.

Introduction

As the era of sciences extended, research organizations increasingly perceived the need for scientometrics indices for comparison of researchers and also trend analysis. Hirsch introduced h-index as one of the most acceptable indices in 2005 (1). It is not limited only to the researchers; Braun et al introduced a Hirsch-type index for journals which was defined as “h published papers, each of them has at least h citations”; they suggested it as a supplement for impact factor (2, 3). Recently researchers recommended to use h-index to distinguish “hot topics” from “old topics” (4, 5).

Iranian Ministry of Health and Medical Education (MOHME) developed a comprehensive scientific national map (including values, goals, priorities, policies and strategies). In research era, sub-secretary for research and technology of Iranian MOHME developed a plan based on this map and national development plans in 2007; in this plan duties of each 42 universities of medical sciences have been clearly defined to attain the goals. They had to train human resources, prepare suitable setting and equipment, and facilitated research process. These universities also developed their strategic plan based on these duties. Although there have had strategic plans and related interventions in research system of the universities, but their effect on performance and scientific production of the universities remained unclear. As we did not find any published evidence on effect of research system interventions on scientific productions of medical universities, managers can use this information for evidence based policy making and management in the university (6).

We conducted this study in order to assess the effect of these new policies and related interventions on scientometrics indices of Shahid Beheshti University of Medical Sciences (SBUMS) and its faculties during period of 1/1/2009 to 30/5/2012. This university is the second ranked university in research among 42 universities of medical sciences in Iran and we assessed all the faculty members in this study (7).

Methods

SBUMS has more than 1200 faculty members, 10 schools, and 48 research centers. It prepares health services for more than 80% of citizens of metropolitan of Tehran, the capital of Iran by this time. We designed a package of interventions based on the policies in sub-secretary for research and technology of SBUMS; then assessed its effect on scientometrics indices.

Policies

Strategic plan of SBUMS contains policies like: to expand applied and fundamental Health system researches; to extend national and international relationship in research; to develop context for scientific production using new technologies; to support researchers; to improve efficacy and effectiveness of research; to absorb and extend resources; to improve research products in terms of quality, quantity and commercialization; to develop infrastructure for research in health service provision and education.

Interventions

Based on strategic planning, following interventions were implemented:

1) There was a gap in the process of conducting scientific productions from theories or questions of faculty members. We needed some intermediate personnel to do the administrative process of research; they should facilitate the process of developing proposals from an idea. In order to train human resources, we designed an educational program for research development and management and presented it for researchers and research centers. Duration of this course was 300 hours in 12 educational blocks. This course empowered the manpower to design, manage and finalize a research proposal, manage its accomplishment and even wiring articles drafts. We named these personnel as research workers. 2) Access to full text articles of scientific journals was a major concern; Sub-secretary for research subscribed to major sources of scientific product like ISI web of knowledge, Scopus, Oxford, ProQuest, and Ovid. To be more enriched, back files of 800 journals were bought too. All the faculty members and researchers could use this database at the university website and through other related IPs. 3) In order to develop research setting and facilities we established a comprehensive central laboratory. New technologies like advanced PCR were employed in this laboratory. 4) Other intervention for setting a baseline for research was developing a central animal lab. In this lab new technologies like MRI (Magnetic Resonance Imaging) for animals were used. 5) First incubator for pharmaceutical sciences established to communicate university to industry. 6) We facilitate administrative processes of research using web-based software; Submitting proposals, progress reports, result of ethics committee and other administrative affairs were done through this software. 7) To conduct researches toward goals of organization, research priorities for schools and sub-secretaries of the university in two separate projects were defined (8, 9). In these research projects the recommended method of council on health research for development was used (10); all the stakeholders were defined in different relevant organization and also in community level. In next step and stakeholder analysis was done. The priorities extracted through brain storming technique and then finalized using Delphi technique. These projects revealed the fields of research priorities in the university in different categories of basic sciences, clinical sciences and health system research. The University encouraged researchers to use these priorities by allocating resources only for them. 8) A supportive funding organization (Alborz Persian-Drug Fund) established to connect non-academic to academic organizations. This funding agency supports the researches that lead to a considerable product for health market. On the other hand a “clear room” established to attract financial resources for research from MOHME. 9) Extending the research centers from 20 to 48 research centers and organize them in context of 15 research school, each of them contain at least three subject related research center. These research schools were included in five relevant research institutes, each of them cover at least three research schools. This process is ongoing and new research centers are adding every year. It decreased the financial burden of over-head costs of staffing and maintenances.10) The 10th intervention defined as special financial support for faculties that disseminated the results of their projects as articles. These bonuses were defined based on journals’ scientific credit, Impact Factor and indexing. More bonuses were paid for PubMed and ISI web of knowledge indexed journals as the first degree journals; they paid more for journals with higher impact factors. This bonus was besides the grunts they received for the projects. On the other side faculty members’ articles- at least one English language article in PubMed or ISI journals- may facilitate annual promotional benefit. Qualified faculty members in Iran receive an employment benefit each year by which their salary increases about five percent. This benefit was only based on their educational activities before that. By this new rule they encouraged to be more active in research fields.

Outcomes assessment

We considered two years as latent period, from 2007 to 2009, for interventions to take effect and outcomes revealed. For evaluating the effectiveness of interventions, research team considered descriptive statistics of publications in each year, total publications during last three years, row number of citations, total number of citations during last three years, h-index for individuals and means of publications in each year and last three years, number of citations, total number of citations during last three years for schools. They were extracted from Scopus and ISI web of knowledge during period of 1/1/2009 to 30/5/2012. Final list of faculty members based on their schools, scientific rank and employment situation requested from human resource office of the university. In order to prevent biases one expert faculty member in search who had history of working in two sub-secretaries for education and for research and technology of the university extracted these indices. He Searched data bases considering that Iranian names and family names may be written in different formats in English and sometimes one person has different profiles in these databases. Using specific protocols of Scopus and ISI web of knowledge and general rules of search such as using asterisk “*”, conjunctions -“OR”, “AND”, NOT”-, quotations “”,dashes and other relevant techniques for covering different writing formats all the faculty members of the university were assessed on scientometrics indices.

These indices were summarized based on schools. The trend of row number of scientific production extracted and presented in table 4. The indices were presented as descriptive statistics and were compared using t-student test when appropriate. Statistical analyses were performed using SPSS software, version 16.0 (SPSS Inc, Chicago, USA).

This study was approved by the Research Ethics Committee of SBUMS and was carried out in accordance with the Declaration of Helsinki (1989) of the World Medical Association.

Results

Totally 1203 faculty members were evaluated. Among these faculty members 128 (10.7%) were professors, 286 (23.8%) were associate professors, 626 (52.2%) were assistant professor, 156 (13.1%) were instructors and for the other 7, there was no scientific rank available in database.

There was totally 4605 article in Scopus since 1985 submitted by affiliation of Shahid Beheshti University of Medical Sciences, among them almost 60% of scientific productions were published from 2009 to mid-2012. The trend was increasing as about 420 (10%) of them were published in 2009, 613 (13%) in 2010, 957 (21%) in 2011 and 792 (17%) in first half of 2012.

Total articles and citations, their means based on years and also databases were presented in Table 1.

Table 1

means for scientometrics indices of faculty members of Shahid Beheshti University of Medical Sciences in Scopus and ISI during 2009-mid 2012

		Scopus		ISI
		Articles	Citations	Articles	Citations
Mid 2012	Total	844	6486	635	4119
	Mean (SD)	0.7 (2.08)	5.4 (19.80)	0.5 (1.49)	3.4 (13.04)
2011	Total	1924	13283	1556	9018
	Mean (SD)	1.6 (4.27)	11.1 (37.65)	1.3 (3.42)	7.5 (26.84)
2010	Total	1717	10077	1279	6857
	Mean (SD)	1.4 (3.62)	8.4 (30.52)	1.1 (2.72)	5.7 (21.98)
2009	Total	1494	7222	1154	5050
	Mean (SD)	1.2 (3.15)	6 (23.34)	1 (2.63)	4.2 (18.68)
2009-mid2012	Total	5135	30582	3986	30925
	Mean (SD)	4.3 (10.40)	25.4 (90.56)	3.3 (8.18)	17.4 (66.62)
Total	Total	10698	54300	8295	35909
	Mean (SD)	8.81 (22)	45.1 (176.44)	6.9 (18.07)	29.9 (124.25)
h-index : Mean (SD)	1.9 (2.89)	1.5 (2.49)

In Scopus database, 80% of faculty members had at least one publication. After 2009, one more percentile involved in research and had publication in this database. H-index for 60% of them was one or more. In ISI web of knowledge 70% of faculty members had at least one indexed publication. After 2010, ten more percent of faculty members involved in research and had articles indexed in this database. There was the same situation for citations in both databases. Scientometrics indices are presented as percentile groups of faculty members for Scopus in Table 2 and for ISI in Table 3.

Table 2

Percentiles for scientometrics indices of faculty members of Shahid Beheshti University of Medical Sciences in Scopus during 2009-mid 2012

	Percentiles
	10	20	30	40	50	60	70	80	90	95	97
Total articles	0	0	1	1	2	4	7	12	22	34	45
Total citations	0	0	0	0	2	7	17	38	100	222	335
Articles mid-2012	0	0	0	0	0	0	1	1	2	3	4
Articles 2011	0	0	0	0	0	1	1	2	4	7	9
Articles 2010	0	0	0	0	0	1	1	2	4	6.9	8
Articles 2009	0	0	0	0	0	0	1	2	4	5.8	8
Citations mid-2012	0	0	0	0	0	1	3	5	13	24	36.8
Citations 2011	0	0	0	0	1	2	5	10	26	54.6	78.9
Citations 2010	0	0	0	0	0	1.4	4	8	19	42	64.6
Citations 2009	0	0	0	0	0	0	2	5	14	32	48.9
H-index	0	0	0	0	1	1	2	3	5	8	9

Table 3

Percentiles for scientometrics indices of faculty members of Shahid Beheshti University of Medical Sciences in ISI during 2009-mid 2012

	Percentiles
	10	20	30	40	50	60	70	80	90	95	97
Total articles	0	0	0	1	2	3	5	9	19	28.9	38.9
Total citations	0	0	0	0	1	4	9	19	64	152.6	242.9
Articles mid-2012	0	0	0	0	0	0	0	1	2	3	4
Articles 2011	0	0	0	0	0	1	1	2	4	6	8
Articles 2010	0	0	0	0	0	0	1	1	3	5	7.9
Articles 2009	0	0	0	0	0	0	1	1	3	4	6
Citations mid-2012	0	0	0	0	0	0	1	3	8	16.9	24.9
Citations 2011	0	0	0	0	0	1	3	6	18	39.9	57.9
Citations 2010	0	0	0	0	0	1	2	4	12	32	45
Citations 2009	0	0	0	0	0	0	1	2.8	9	24	34
H-index	0	0	0	0	1	1	2	2	4	6	8.9

Comparing the databases showed that means for h-index is statistically significant different in ISI and Scopus (p<0.001) (Table 1).

Table 4 shows the scientific publications during 2009-2011 in Scopus and ISI data bases divided by schools. Comparing means based on year showed that there was no statistically significant difference between means in all schools in all selected years (p>0.05 for all) except for clinical departments of school of medicine in which in year 2010 (p=0.015) and 2009 (p=0.001) means for articles in Scopus was statistically significant higher than ISI; but in 2011 the means had no difference (p=0.098).

Table 4

Mean (Standard Deviation) for Articles of Schools of Shahid Beheshti university of Medical Sciences in Scopus and ISI databases during 2009-2011

	Number of faculty members	2009		2010		2011		h-index
Schools		Scopus	ISI	Scopus	ISI	Scopus	ISI	Scopus	ISI
School of Medicine (Faculties of clinical disciplines )	569	1.1 (1.83)	0.7 (1.46)	1.3 (2.29)	0.95 (1.86)	1.3 (2.47)	1 (2.02)	1.64 (2.29)	1.3 (2.08)
School of Medicine (Faculties of basic Sciences disciplines)	103	1.8 (3.35)	1.62 (3.30)	1.7 (2.69)	1.54 (2.60)	2.2 (3.29)	2 (3.34)	2.9 (3.26)	2.4 (2.89)
School of Dentistry	122	0.42 (0.88)	0.36(0.87)	0.45 (1.13)	0.38 (0.99)	0.62 (1.33)	0.55 (1.23)	1 (2.02)	0.89 (1.66)
School of Pharmacy	40	2.1 (2.44)	1.9 (2.30)	2 (2.46)	1.8 (2.29)	2.2 (2.33)	1.7 (1.83)	5 (3.73)	3.9 (3.53)
School of Rehabilitation	38	0.13 (0.41)	0.13 (0.34)	0.32 (0.62)	0.29 (0.70)	0.42 (0.79)	0.26 (0.50)	0.42 (1.20)	0.37 (1.10)
School of Paramedical Sciences	62	0.35 (0.93)	0.23 (96)	0.60 (1.34)	0.39 (1.22)	0.68 (1.59)	0.48 (1.21)	0.90 (1.54)	0.66 (1.28)
School of Public Health	22	1.1 (1.71)	0.6 (1.18)	1.2 (2.89)	0.9 (1.91)	1.7 (2.97)	1.4 (3.26)	2.1 (3.21)	1.2 (2.46)
School of Health, safety & environment	21	0.1 (0.48)	0.1 (0.44)	0.1 (0.30)	0.1 (0.22)	0.4 (0.74)	0.2 (0.54)	0.6 (1.63)	0.5(1.54)
School of Nutrition	35	0.7 (1.38)	0.9 (2.13)	2.3 (4.1)	1.6 (3.13)	2.6 (4.69)	2.2 (3.86)	2.5 (3.88)	1.8 (3.45)
School of Nursery and Midwifery	54	0.1 (0.53)	0.1 (0.52)	0.2 ( 0.53)	0.1 (0.33)	0.4 (1.23)	0.2 (0.82)	0.4 (1.08)	0.2 (0.71)

Discussion

In this study we found scientific productions of faculty members in SBMUS have increased following new policies and interventions. Scientific productions during last 3.5 years exceed those in 24 years before the study. There was continuous growth during few years of the study. Considering that indexing articles in Scopus and ISI took mean time of 2-3 months, we expect another rapid growth in 2012 as we had 792 articles only in first half of 2012.

There is an increasing trend of published articles in the country. In order to control the probable bias and showing the effect of our interventions the national research data considered as control group. If we compare these two trends. Country publication trend, as Saboury reported, showed 12% increase from 2009 to 2010 and 32% from 2010 to 2011 in ISI database. Our study showed 46% increase from 2009 to 2010 and 56% from 2010 to 2011.This high differences support the effectiveness of our interventions (11, 12). This finding supports the causal effect of our interventions on increasing trend of scientific productions.

H-index used for assessment of articles citations in this study. This index is best known index for evaluating researchers function but there are some weak points. As Bornmann discussed h-index is suitable for young researchers because old researchers if even had no new publication their h-index will increase by the time (13); on the other side, if they had high h-index, new publication can’t increase their h-index in short time. To overcome this issue we also assessed row number of publications during last 3.5 years divided by year. Batista et al found that the percent of researcher with h-index>10 defers between disciplines; they conclude it is difficult to compare scientist of deferent disciplines (14). H-index is insensitive to un-cited or less-cited articles; on the other hand if a paper has higher citations than h (even double or triple of h-index) the h-index remains constant. They proposed a new g-index to cover this weak point but it is not used generally yet (15). Sangwal also noted this weak point and focused on “conceptualization of tapered Hirsch index hT, circular citation area radius R and citation acceleration a” (16).

Braun mentioned lack of standards for different disciplines and sub-disciplines and number of authors in each article are main concern in evaluating h-index. Faculties in more specialized disciplines have lower h-index in general and others in general or interdisciplinary eras like cancer, Diabetes or nutrition have higher h-index. We also assessed the h-index and scientific productions in different schools and found that their values were different. Schools of Medicine, Pharmacy, and Nutrition had the higher means compare to the other Schools; Post-graduate disciplines in master or PhD degrees and residency programs present in these schools more than the others and this may explain the differences. School of Health also had higher mean score; department of epidemiology may affect the scientometrics indices of this school. All the authors had the same weight in h-index; this was another challenging point of h-index in ours and also Braun study (2).

Analysis of data showed that there were some other points for intervention. One of them is considerable number of faculty members that had no publication indexed in these two databases. In next Step University authorities should develop policies and plans to involve them in qualified researches. Before that the cause of their disaffiliation should be cleared. Next studies could be focused on the effect of each intervention separately.

The means of articles in ISI and Scopus were statistically different. H-index can be variable based on data base as Bar-ilain compared three main databases of ISI, Scopus and Google-Scholar (17). We found in our study that we assessed only articles indexed in Scopus and ISI; but many faculties have articles in Persian -or in English- and most of them did not indexed in these two databases: so this doesn’t show all research efforts of faculty members. Rip also noted that scientometrics is limited to a main data base of ISI and the other data bases have not the same characteristic (18). It might be difficult for them to write in English as Van leeuwen et al, mentioned (19). On the other hand, editing the articles and the higher fees for the publication process in some journals may push them to publish their articles in journals of other databases which are free and open access. To overcome this concern university has had some interventions like financial supports but they can be more focused and facilitated.

Although there was a clear methodology but we had some limitations in practice. We could not assess the pure effect of every intervention as they were used as a package. The other important limitation of this study was different spelling of Persian names in English. For Example family name of ”mousavi” which is a common in Iran can be written by 27 another spellings of “moosavi, mosavi, musavi, mousavie, moosavie, mosavie, musavie, mousavei, moosavei, mosavei, musavei, mousavii, moosavii, mosavii, musavii, mousavy, moosavy, mosavy, musavy, mousavey, moosavey, mosavey, musave, mousavee, moosavee, mosavee, musavee”. When we searched this name using asterisk, there were many more names with additional suffixes especially in ISI, and make it difficult for judgment. Considering this point there may be some under-estimation in counting publications and H-index. Van Raan also confirmed this under-estimation in non-English countries and in articles with high number of authors (20).

Scientific writing should prepare solutions for health problems. Writing for writing is not the case. The main concern in research is paying attention to applied researches (21). Researches should be ordered by industry or other users or by a third party for them. Unfortunately this process is not matured in developing countries. A considerable portion of their researches are copies of developing countries researches without any native modification. New policies should consider whether studies are designed based on appropriate questions and focus on quality of researches in next studies.

Conclusion

There is increasing trend in scientific production of SBMUS in both Scopus and ISI data bases. It could be mostly due to the effect interventions in research system during last five years. As Iran is one of the third high ranked in Middle East in terms of scientometrics (22), it is important to monitor progress of scientific production by yearly interval.