Health research evaluation and its role on knowledge production.

BACKGROUND: Knowledge production and evaluation are two important functions of health research system (HRS). In this article, we aimed to reveal the correlation between evaluation of health research organizations and health knowledge production promotion.
METHODS: A comprehensive evaluation system was developed to evaluate the academic performance of national medical science universities on an annual basis. It assess following domains; stewardship, capacity building and knowledge production. Measurable indicators for each domain were assigned, a 'research profile' for each department was provided. In this study, we compared the results of annually national Health Research System evaluation findings during 2005-2008.
RESULTS: The number of scientific articles has been increased from 4672 to 8816 during 2005 to 2008. It is mentionable that, the number of articles which has been published in indexed data bases has risen too. This fact could be related to directed policy for more international publication of scientific articles from Iran. The proportion of total articles to the number of academic members was 1.14 in 2008, comparing to 0.84 in 2005. It means that this proportion have increased about twice (0.7 Vs 0.45) during mentioned time. Moreover, other scientific products such as authored books based on domestic researches and cited articles in textbooks have increased according to special attention to knowledge production by policy makers.
CONCLUSION: We conclude that Health System Research evaluation could be used as a mean for implementing policies and promoting knowledge production.

Introduction

Knowledge management is defined by World Health Organization (WHO) as, “a set of principles, tools and practices that enable people to create knowledge, and to share, translate and apply what they know to create value and improve effectiveness” (1).In recent decades, knowledge managers have focused on developing and enriching knowledge production strategies (2). In this regard, Health Research System (HRS) has a prominent responsibility in scaling up knowledge production (3).

HRS has certain special functions that help filling the gap between Research and Development (R&D), especially in developing countries (2, 4). One of these main functions is Monitoring and Evaluation (5). This function of the HRS is an operational component of stewardship and is useful as a tool for promoting the other function of HRS such as creating and applying research. HRS evaluation could be used for expanding policies, emerging trends in health research, knowledge generation, and technology development and making recommendations for future national policies (6).

Following the stated mission of the World Health Organization (WHO) as, “generation of appropriate knowledge and its utilization towards improved health”, the Department of Research and Technology of the MOHME in the Islamic Republic of Iran was tasked with the responsibility for designing and implementing the HRS functions and policies especially monitoring and evaluation (7). The main focus of this evaluation system was knowledge production to promote the academic performance of the researchers with an emphasis on promoting international scientific publications. Evaluation of research activities in terms of knowledge production was performed based on stakeholders agreed criteria.

The present paper aims to assess the contribution of the HRS evaluation on promotion of HRS performance and health knowledge production.

Materials and Methods

During last decade, the Department of Research and Technology of the Ministry of Health and Medical Education (MOHME) of I.R. Iran (as the main policy making organization in biomedical and health related research) concentrated on knowledge management promotion. Scientific activity is traditionally measured by the production of scientific publications, i.e. the number of articles published in scientific journals. Besides one decisive criterion in the realization of research policy is the number or volume of valid scientific research studies conducted by specialists in different fields (4).

A comprehensive evaluation system was developed to evaluate the academic performance of national medical science universities on an annual basis. Through a collaborative effort the key HRS authorities developed certain measures. A pilot study was conducted to establish the applicability of the evaluation measures adopted. In each year, the evaluation procedures were reviewed and renewed. Based on certain policies some measures were changed in line with stakeholders’ views and the objective results obtained. “Stewardship”, “Capacity Building”, and “Knowledge Production” were three domains that considered in evaluation process. (Table 1) shows the Medical Sciences University Evaluation Form.

Table 1:

Medical Sciences University Evaluation Form

Domain	Subject	Measure	Score
	Strategic Planning	Existence of strategic program, report of annual program evaluation	0–100
Stewardship	Priority Setting	Existence of research priorities, Percentage of research projects in line with research priorities	0–100
	Ethics committee Activities	Percentage of referred proposals to ethic committee	0–100
Capacity Building	International Congress	Held international congress confirmed bycabinet	Each case 100
	National congress	Held national congress confirmed by Health ministry	0–100
	Research related Workshops	Held required workshop based on Research Council confirmation	0–150
	Scientific Awards	Get award from Razi or Kharazmi festivals	First 100Second 70Third 50Student award 50
	Book authoring	Books which at least 1% of their references are authors self citations	Each case 25
	Citation to articles in textbooks	Citation to articles in credible textbooks	Each case 15–25
	Innovations and patents	Practical innovations and patents reported in scientific journals	National cases: 2–10 International cases: 5–100
	Paper presentation in national and international congresses	Paper presentation in oral or poster form, in national and international congresses and papers indexed in ISI proceedings	Presented paper in national congress: Each case 0.5Presented paper in Internationalcongress -Oral presentation 3 -Poster presentation 0.5 -ISI Abstract Meeting:2-3 -ISI Proceeding:3-4 -Invited Speaker 6
Knowledge on Production	Indexed articles in ISI/Web of Science and PubMed/Medline	Indexed articles in PubMed/Medline and ISI /Web of Science by considering the impact factor with two coefficients as the quality criteria of a journal.	Each article indexed in ISI/WOS :25+2(Impact Factor) Each article indexed in PubMed/Medline: 25
	Indexed articles in other international indexing databases(such as Scopus and Chemical Abstract and etc.)	Indexed articles in other international indexing databases ; Scopus and Chemical Abstract, Biological Abstract, embase, Index copernicous, IMEMRO	Each case 10–15
	Nonindexed published articles in national and international scientific journals	Non-indexed published articles in journals witch have accredited by the Iranian Ministry of Science, Research and Technology (MSRT) and the Ministry of Health and Medical Education (MOHME) and credible foreign journals	Each case 5

According to indexing type, article type and the place of publication, all documents were classified and cleaned in order to avoid double counting. The articles were divided into 5 types including of original research article, review article, brief report or short communication, case report and letter to the editor.

The evaluation was carried out by a team consisting of research experts at the office of undersecretary for research. All team members applied the developed measures to Health Research System in 40 universities of medical sciences annually. The measures were assessed during a meeting with the representative of each university. The evaluation forms were filled out based on controlling presented official documents.

Results

Figure 1 shows the results of the 2005–2008 annual HRS evaluation report. The data presents the total number of published articles and medical researchers during mentioned time period. The total number of published articles has increased from 4672 to 8816 during mentioned period. It is noticeable that during the same period total number of medical researchers has increased 1.2%. The results were analyzed according to cited articles in widely recognized data bases. Figure 2 provides detailed information about the growth in the total number of cited articles.

Fig. 1:

Annual academic and research paper publication in 2005–2008

Fig. 2:

Articles published during 2006–2008 period

The number of indexed articles in reliable and widely recognized data bases has risen by 127%. Although articles sited in ISI/PubMed have decreased in 2008, the number of published articles in Iranian journals indexed in ISI/PubMed has increased by 36% (from 654 papers in 2007 to 893 in 2008). Noticeably, the proportion of ISI Web of Science/PubMed indexed articles per medical researcher has increased from 0.09 to 0.33.

Retrieved data from 2005 to 2008 show that the trend of published articles cited in other recognized data bases including Scopus, EMBASE, Chemical Abstract, Index Medicus, EMRO and Index Copernicus was ascending. It is noticeable that the number of articles published in Iranian journals indexed in mentioned databases had increased by 14%. Also, the above Figure shows that the total number of non-indexed articles had a descending trend. A closer look at the data reveals that almost, 85% of articles were original, 7% case reports, 3% short communication papers, 3% letter to the editor and 2% review articles in 2008.

In 2008, national medical universities were ranked by the scores they received on different knowledge production axes. Table 2 shows the top ten ranking Iranian medical sciences universities in terms of their scores on knowledge production.

Table 2:

Knowledge production scores of top Ten Iranian Universities based on the 2008 evaluation results

NO.	Medical Science University	2005			2008
		Academic Member	Total Articles	Knowledge Production Score	Academic Member	Total Articles	Knowledge Production Score
1	Tehran	1236	1156	18178	1265	2346	43682
2	Shahid Beheshty	1049	656	9474	1115	1426	25711
3	Isfahan	637	382	5976	611	806	13778
4	Shiraz	538	299	5373	656	746	13494
5	Tabriz	553	269	3797	607	647	11269
6	Mashhad	584	320	4085	629	562	9422
7	Iran	721	245	3353	696	529	8043
8	Baghiatallah	183	185	2583	199	368	5979
9	Ahvaz	420	129	1678	472	317	5209
10	Mazandaran	209	187	2529	201	222	4063

The comparisons show that in terms of knowledge production, Tehran University of Medical Science stands in the first place. This university had 2346 scientific articles in 2008 that covered 26.6% of total national biomedical articles. Sitting in the second place is Shahid Beheshti University of Medical Science having published 1426 articles, which comprises 16% of total articles related to this university and Isfahan University of Medical Sciences University as the third rank had 806 scientific articles in 2008 that have increase about 110% growth in publication. It is mentionable that these top ten universities covered 72% of our country biomedical articles.

Findings of this study show that the proportion of total articles per research project was 0.84 in 2005 comparing with 1.14 in 2008. Moreover the proportion of total articles per medical researcher has increased about twice (0.7 Vs 0.45). In terms of knowledge production, the data collected in 2008 shows that 14,357 articles were presented in scientific colloquiums (9,938 in national congresses and 4,419 in international congresses) compared to the 6585 in 2005.

Furthermore, other scientific products such as published books, published scientific articles witch cited by credible textbooks have increased according to special attention to knowledge production by policy makers.

Discussion

The National Iranian Plan for Health Innovation and Science Development aims to increase the capacity to create and facilitate knowledge sharing and dissemination as two main policies (8). The Department of Research and Technology in the Ministry of Health and Medical Education has been developing systematic plans to realize these policy objectives in medical sciences universities. Furthermore, supporting national programs for knowledge management, promoting knowledge generation and establishing knowledge pattern are national level strategies recommended by WHO (2).

Health research has been broadly defined as “the generation of new knowledge using the scientific method to identify and deal with health problems” (9). Generated Knowledge will enable research stakeholders to improve implementation of Health research in order to promote health outcomes and health equity (3, 10).

Significant increase in number of scientific published papers has improved Iran’s scientific ranking word wide (11, 12). According to ESI (Essential Science Indicators), among a five-year period (2005–2009) Iran stands in the 19th rank for scientific output (13). This promotion could be as a result of the systematic approach to research management which enables research stakeholders to improve health outcomes and health equity (3, 10). Increased attention of health research policy makers to research projects in terms of quality as well as quantity, could contribute to raising the ratio of articles per projects (14).

According to the findings of this study the number of cited articles in ISI/PubMed has increased 1.3 times from 2005 to 2008. This fact could be as a result of revising research polices which offered more credit to the quality rather than sheer quantity. The number of Iranian biomedical journals has increased from 8 to 155 (1979 to 2010) (15). The number of Iranian medical journals that are indexed in scientific database increased from 53 in 2000 to 141 in 2008 (16). Increased accessibility to scientific data bases, advances in information and communication technology (ICT) and indexing Iranian journals in credible databases have promoted Iranian publication output (17).

The proportion of articles to academic member among, Tehran, Shahid Beheshti and Isfahan medical sciences universities were respectively 1.8 and 1.2 and 1.3.Ascending trend of this proportion among top universities leads to achievement of the comprehensive Medical Plan objectives.

In conclusion we believe that planned policies could be safely applied through HRS evaluation criteria. Therefore national policies may flow from the past, present and future by efficient use of available resources.

Ethical considerations

Ethical issues (Including plagiarism, Informed Consent, misconduct, data fabrication and/or falsification, double publication and/or submission, redundancy, etc) have been completely observed by the authors.