Peer Review Research Evaluation in Iran: Strengths, Weaknesses and Suggestions.

BACKGROUND: Peer based evaluation is a qualitative assessment done in different fields and levels. The aim of this study was to express the results of peer review evaluation in selected Iranian clinical research centers.
METHODS: Four main domains consist of Leadership and governance, Structure, Knowledge products and Impact in thirty Iranian clinical research centers were evaluated based on peer review in 2019. Strengths and weak points with peer's suggestions were extracted based on qualitative analysis.
RESULTS: Governance and impact domains have been more weak points than others. Equipment, facilities, physical space and human resource have been desirable in many research centers, and also there were some good developments in research publication. The most important suggestion was pay more attention to technology in planning, infra-structure and impact levels.
CONCLUSION: Review missions of clinical research centers with more emphasis on health impact is necessary to clinical improvement.

Introduction

Research evaluation is a process to improve the effectiveness and efficiency of research policies; it goes with conducting an at a path model in the relationships between policy makers and researchers, which make a large autonomy at academic level. Thus, the justification of this process emerged from the need to control in order to receive main objectives, such as knowledge production based on research priorities and so on (1). In United Kingdom, evaluation was implemented by government at national level as a mechanism for resource allocation (2), but it may be done at the institutional level to stimulate learning processes by local policy makers(3). Moreover, evaluation procedures may be influenced by two models; centralized (top down) or decentralized (bottom up) which in most cases this process is a mixed initiatives (4). The purpose of evaluation can be focused on specific function or whole activities. It may be done in output, outcome or impact level (5).

In Italy, three-year research assessment exercise in main disciplinary sectors such as chemistry, biology, humanities and economics began by the Italian Committee for the Evaluation of Research in 2004. In this evaluation system, R&D performance of public research structures was judged by peer reviewers (6).

In 2001, Iranian Ministry of Health and Medical Education was established the research evaluation system (RES) of medical universities and research centers. This system mostly included output indicators which determined based on key stakeholders’ opinions. Annually, these indices were modified and developed based on feedback received from researchers, policy makers and other stakeholders and finally in 2015, in parallel with the quantitative evaluation, the qualitative assessment of the selected research centers with four main domains including leadership, structure, knowledge production and impact were added to RES.

The aim of this study was to express the results of peer review evaluation of selected research centers and this paper only presents the peer review results for clinical cases.

Methods

Setting

The current study was implemented in Iran. In Iran, there are 803 Medical Research Centers (MRC) in clinical and biomedical fields. Based on budget line, all of MRCs divided to two groups including dependent to university budget and independent. The number of MRCs with independent budget line is 51 (30 and 21 clinical and biomedical research centers respectively).

All of MRCs with more than one year research activities have been evaluated by Ministry of Health and Medical Education (MOHME) and also some of them based on special criteria were assessed through peer review evaluation.

Design

This study was implemented in 2019. All of clinical research centers with independent budget line and also, the first research rank among clinical centers based on quantitative annual evaluations were selected for this project. This method was a qualitative; and conducted based on experts’ opinion in each field. A number of national and international experts came to research center and reviewed all of the past five years research documents, visited the physical space and facilities and finally based on special instruction, completed the evaluation forms and expressed strengths, weaknesses and their suggestions to improve research function of MRC.

For each RC, at least two national and one international reviewers with associate professor or higher scientific degree in the special field of the center were selected. Moreover, domestic reviewers should not be affiliated to research center.

Two training workshops were carried out for reviewers and members of research centers to get familiar with instruction. The duration of each peer review was considered between one to three days.

Indicators

Four main domains were evaluated by peer reviewers as follow:

- Leadership and governance

- Structure

- Knowledge products

- Impact

Each domain has two or more axes. Table 1 shows main domains, related axes and topics covered. After gathering all of descriptive reports, the meaning units were identified based on open coding and by the induction method, all of themes and subthemes were identified. In order to increase the validity and reliability of the data, the content of reports was confirmed by the reviewers at the end of each session. The triangulation technique was also used.

Table 1:

Main domains, related axes and topics covered in Iranian clinical research centers based on peer review -2019

Domain	Axis	Topics covered
Governance and leadership	Strategic planning	The existence of a five year strategic plan that includes action plan, research priorities, appraisal and updates
	National and international collaborations	The number, importance and achievement of domestic and foreign cooperation in the past five years
Structure	Infra-structure	Five top projects, future major projects, new techniques, different educational resources
	Efficiency	Budget, manpower, physical space, equipment
Research products	Knowledge production & dissemination Technology	Five Top published articles, number of citation, Subject compliance with the priorities, Foreign patent registration, product production, launch of new methods
Research impact	In three levels : Community, Service provider, Policy makers	Document based any change in each level

In this study, all ethical issues such as confidentiality, trustworthiness, etc. were observed by the referees and monitored by the national team.

Results

Thirty clinical research centers were participated. The main research fields of these MRCs consist of cardiology, endocrinology, cancer and gastroenterology. Based on content analysis, 213 open codes, 4 main domain, 7 themes and 21 sub-themes were extracted.

In this part, based on each main domain, their axes, strengths, weakness and suggestion for each axis were presented as follow:

Governance and leadership

This domain has two axes consist of:

Strategic planning

Strengths: based on peers’ opinions, the majority of research centers in this axis have been no definite strength point and only a few of them succeeded in correctly developing the strategic plan and their action plans.

Weaknesses: the majority of research centers had main weak points in this axis. Majority of them didn’t have any research line and map. Research priorities in many centers were not set in the right way and few did not set at all. Other challenges faced by many of these research centers were the scattered work or dispersion, low depth research activities, lack of program updates and monitoring. According to most peer opinions, low commitment to the program is the dominant feature in almost all of MRCs.

“No one has taken the strategic plan seriously in the centers” (peer reviewer)

Suggestion: one of the most important suggestions was establishing legal requirements for the strategic plan formulation and commitment to its implementation by Ministry of Health and Medical Education. This coercion can be done in a variety of ways, such as budget allocation, performance monitoring, incentives and so on. Stakeholder participation in designing the plan, attention to upstream documents in Prioritization and specify the research lines were the main suggestions in this part.

National and international collaboration

Strengths: almost less than half of research centers had broad international communications with great organizations. Send out student abroad and joint project implementation is the most form of international collaboration in these MRCs.

Weaknesses: A number of MRCs didn’t have any cooperation with domestic and foreign partners. Each MRC has all of their required equipment and there isn’t any sharing between laboratory devices, it means that they interested in working alone without any cooperation with each other and it leads to waste of resources, duplication and so on.

Suggestion: Strengthening inter-agency cooperation, interdisciplinary communication, increased team working, enhancing international cooperation were peers suggestion to develop this item. Mission identification based on priorities and task division among RCs can cause more resource synergism and goal achievement.

Structure

This domain has two axes consist of:

Infrastructure

Strengths: In most research centers, educational resources were appropriate and training was provided to researchers through holding various workshops, attending conferences and so on. Top research projects in some of the MRCs had scientific value and the majority of them were national level.

Weaknesses: the most important weakness in some of research centers was low attention to create appropriate infrastructure for technological activities. Based on opinions referees, the important issue of technology has been neglected in many research centers. Therefore, neither the necessary infrastructure nor the technology manpower had been embedded in some research centers.

Suggestion: the most important suggestion in this field was to revise the research centers ‘missions toward technology, capacity building of researchers, equipping research centers on new technologies, establishing knowledge based companies and attraction appropriate funds .

Efficiency

Strengths: based on peers’ opinions, one of the most important strength in this section was the efficient manpower in most research center and the majority foreign grant attraction has been made through these outstanding researchers. The physical space has two contradictory modes. In some centers it was very convenient and in some cases, it was quite limited and there was only one meeting room!

Advanced research equipment in some of RCs was as good point.

Weaknesses: Inappropriate attraction of researchers with research center missions was one of the weak points in some RCs. It means that, in some cases, the recruitment of researchers had taken place before setting the priorities or verifying the missions and the expertise of the researchers is not consistent with the projects. National cooperation and domestic grant attraction were another weak point. Sporadic activities in some of RCs were very bold.

Suggestion: according to expert views, developments of national cooperation, formulation of main missions based on priorities and encourage researchers to team working, avoiding duplication, sharing the facilities and equipment can improve efficiency and reduce extra and vain costs.

Research products

This domain has two axes consist of:

Knowledge production & dissemination

Strengths: articles publication and efforts to improve their quality were as strengths points in some RCs. Increase number of high quality papers (the first quartile quality in each field: Q1) and citations are two witnesses to this claim. Weaknesses: The variety of published articles and their mismatch with the priorities and missions and also increased citation to self-publication (more than 20%) were two main weaknesses. Suggestion: Eliminate the weaknesses of knowledge production such as self-citation more than 20%, quantity improvement, dispersion in publication and so on.

Technology

Strengths: In a small number of centers, there were limited technology products such as new drugs and so on.

Weaknesses: technology products deficit was one of the most important weakness in target group.

Suggestion: the most important suggestions were pay more attention to patent registry and filing, Production of strategic products and in general, the focus on research commercialization.

Impact

This domain consists of three levels; health service users, health service provider and policy makers. This domain is the most important part of evaluation and has the highest score in clinical field.

Strengths: There was no particular strength in this area. Globally, most research centers focus on producing articles rather than producing products; and much less attention has been paid to the impact by them.

Weaknesses: Most of the weaknesses were related to this domain. Almost, there was not pay attention to the changes caused by research products. Economic impact, currency savings, reducing drug complications, facilitating service are the examples of impacts scrutinized very little.

Suggestion: Establishment of knowledge translation and exchange (KTE) unit in research centers, training of KTE principles to researchers, special attention to impact domain by politicians in such a way that this criterion has been considered for budget allocation in research centers were some examples of peers’ suggestions.

Discussion

Among the areas evaluated in clinical research centers, governance and impact domains have been more weak points than others. Equipment, facilities, physical space and human resource have been desirable in many research centers, and also there are some good developments in research publication. Evaluated RCs have had fairly good input, roughly moderate process which needs to be modified and higher output in article publishing and inappropriate in outcome and socioeconomic impact. There is relationship between inequality and socioeconomic growth status (7). Therefore, it is necessary that the missions of research centers with pay more attention on socioeconomic impact will be determined. Obviously, stakeholder participation in developing the strategic plans, setting priorities is very important (8, 9).

In Iran, in recent years, a lot of efforts have been made in knowledge production such as establishing the research centers (10), building capacity of researchers (11), enhancing the number and credibility of Iranian medical science journals (12) promoted Iran’s scientific status in the region (13, 14) and now it’s time to focus more on technology and product development. Establishing knowledge -based companies and technology development centers, supporting technology projects and encouraging researchers to carry out applied researches may be useful in structural reform (15).

The present study has some strengths points and barriers. Peer based qualitative evaluation is one of the most important evaluation method (16) implemented for the first time in Iran (17). Considering three important domains in research including governance, structure and impact instead of counting the number of articles is strength of this study.

One of the main limitations of this study is that the results cannot be completely generalized to other centers, since most of these centers have independent budget line, proper human and physical resources and equipment. Moreover, the referees’ considerations in presenting some points of view are also other limitations of this method.

Conclusion

Review missions of clinical research centers with more emphasis on health impact is necessary to clinical research improvement in Iran.

Ethical considerations

Ethical issues (Including plagiarism, informed consent, misconduct, data fabrication and/or falsification, double publication and/or submission, redundancy, etc.) have been completely observed by the authors.