The Evaluation of Hospital Performance in Iran: A Systematic Review Article.

BACKGROUND: This research aimed to systematically study and outline the methods of hospital performance evaluation used in Iran.
METHODS: In this systematic review, all Persian and English-language articles published in the Iranian and non-Iranian scientific journals indexed from Sep 2004 to Sep 2014 were studied. For finding the related articles, the researchers searched the Iranian electronic databases, including SID, IranMedex, IranDoc, Magiran, as well as the non-Iranian electronic databases, including Medline, Embase, Scopus, and Google Scholar. For reviewing the selected articles, a data extraction form, developed by the researchers was used.
RESULTS: The entire review process led to the selection of 51 articles. The publication of articles on the hospital performance evaluation in Iran has increased considerably in the recent years. Besides, among these 51 articles, 38 articles (74.51%) had been published in Persian language and 13 articles (25.49%) in English language. Eight models were recognized as evaluation model for Iranian hospitals. Totally, in 15 studies, the data envelopment analysis model had been used to evaluate the hospital performance.
CONCLUSION: Using a combination of model to integrate indicators in the hospital evaluation process is inevitable. Therefore, the Ministry of Health and Medical Education should use a set of indicators such as the balanced scorecard in the process of hospital evaluation and accreditation and encourage the hospital managers to use them.

Introduction

Healthcare is a major concern in many countries (1). The complexity of today’s health care organizations, their costs, specialization, and the importance of efficiency and effectiveness of services are among the factors encouraging the health care organizations to change their performance evaluation processes and to apply the organizational improvement models (2). Undoubtedly, evaluation is one of the broadest and most controversial issues in the management field (3). Performance evaluation or performance assessment is one of the most important managerial tasks (4).

Evaluation is the formal and regulated determination of effectiveness, efficiency and acceptability of a planned action to fulfill certain goals (5). Performance is defined as achieving the desired objectives (6). Performance evaluation refers to a set of actions and activities carried out in order to increase the optimal use of resources to achieve the goals efficiently and effectively. The measurement system should be able to compare the performance within the organization, as well as the performance among the similar organizations. The existence and application of a suitable model to evaluate the performance of hospitals can lead to the increases in the responsiveness and patients’ satisfaction and the improvement of service quality (7).

Although several models have been designed to evaluate the hospital performance, most of them either, have limited application or evaluate different dimensions of the performance. Some of these models have focused more on the structural elements or inputs, some of them on the process evaluation and others on the results, and there have been few hospital performance evaluation systems included the balanced evaluation of the inputs, processes and outputs (8). Therefore, various models have been used in different studies. Some of the challenges in the design of a hospital performance evaluation system are the identification of performance evaluation objectives, the evaluation of different dimensions of the hospital performance, and the participation of stakeholders in the design and development of the performance evaluation system (7). It is not surprising that hospitals, as major consumers of the health system budgets and funds, are paid special attention by the researchers and policymakers (9).

Iranian Ministry of Health and Medical Education (MOHME) is responsible for providing most of the secondary health services. Sixty seven percent of hospitals in Iran affiliated to The MOHME, fourteen percent affiliated to the private and non-governmental sector, and nineteen percent of hospitals affiliated to other entities such as insurance organizations and other providers (10). Official method for evaluating the hospital performance is the accreditation standards of the Ministry of Health (11). Accreditation standards provide a framework used as a common model for evaluating health care throughout the world (12).

Although the design of performance evaluation system has become a necessity, the existence of several models to evaluate the performance of hospitals indicates that measurement, evaluation and improvement of the hospital performance, contrary to their appearances, are not easy (7). The use of quantitative methods of evaluating performance such as cost-benefit and cost-efficiency analyses and some indicators such as effectiveness, efficiency, productivity, etc. does not meet the needs of hospitals. In order to evaluate the hospital performance, there is a need for comprehensive models, which can evaluate the performance of hospitals continuously and systematically in all fields and wards according to the main criteria of performance (13).

This research aimed to systematically study and outline the methods of hospital performance evaluation used in Iran. Many studies have been conducted in order to evaluate the performance of hospitals, however, given the large number of models and methods in this area, it seems that there is no clear agreement on which model is better to be used.

Methods

This study aimed to systematically review the articles published about the hospital performance evaluation in Iran. Accordingly, the Persian-language articles published in the Iranian scientific journals, as well as the English-language articles published in the journals of inside and outside of Iran were searched. The researchers used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), which is a standard guideline for systematic reviews and contains 27 items (14). This study sought to describe which models were used to evaluate the hospital performance in Iran.

Search strategies: In order to find studies published electronically until Sep 2014, the articles published in the local and international journals and the theses available in the database were used as follows:

The keywords of “Assessment”, “Evaluation”, “Efficiency”, “Hospital”, “Performance”, “Iran”, and their combinations, as well as their Persian equivalence were used to search for Persian and English-languages articles in the Iranian and international databases.

The search was limited to all Persian-language articles published in the Iranian scientific journals and the related English-language articles published in the Iranian and non-Iranian journals indexed in the selected databases from Sep 2004 to Jun 2014. This search was carried out from Mar to July 2014.

For finding the related articles, the researchers searched the Iranian electronic databases, including SID, IranMedex, IranDoc, Magiran, as well as the non-Iranian electronic databases, including Medline, Embase, Scopus, and Google Scholar. Five inclusion criteria were applied: 1) the report included hospital performance evaluation; 2) the unit of analysis was the hospital; 3) the data required for analysis were available (by access to the full text of the publication or by request from the author); 4) the study’s observations were limited to hospitals within the boundaries of Iran; 5) the report was published in Persian or English. First, the article titles were studied and the duplicates were removed. Then, the remaining articles were carefully studied by the researchers. All Persian and English-language articles published in the Iranian and non-Iranian scientific journals whose full texts were available were chosen and unrelated articles were removed.

The exclusion criteria were: the lack of access to the full texts of articles, letters to the editor, articles with the same title and topic published in both Persian and English languages, articles related to the evaluation of some wards of a hospital, articles whose study population and samples had not been determined or the validity and reliability of their data collection tools had not been explained.

For reviewing the selected articles, a data extraction form, developed by the researchers according to the aim of study, was used. This form included sections for writing the characteristics of each selected article, including the authors’ name, the year of publication, the aim of studies, number of hospitals included in the study, the types of studies, materials and methods, data related to the models of the hospital performance evaluation used, and the results of the articles. At this stage, two researchers involved in the selection of contents and data extraction. The main characteristics of the selected studies and their results have been summarized according to the following variables:

- Types of studies: The classification of studies was as follows: interventional (experimental) studies, descriptive cross-sectional studies, descriptive-analytical studies, qualitative studies, mixed method (qualitative and quantitative) studies, review articles, and systematic reviews.

- The number of hospitals evaluated in each study.

- The language of study (Persian or English language).

- The year of publication.

Results

Seventy-four articles from 733 Persian articles and 28 articles from 309 English ones were selected. At this stage, 70 articles (51 articles in Persian and 19 articles in English) were included in the study among which 5 articles were excluded because there was not any access to their full texts and 7 articles were excluded because they had been published by two languages (English as well as Persian). Overall, 58 articles remained. After excluding articles, which did not meet the inclusion criteria, the articles which had more complete data and were more relevant to the aim of study were selected and their full texts were given to two independent judges and experts in the hospital performance evaluation. The differences between them were explained in the three-member committee, in the presence of project manager and supervisor, and the related decisions were made. Eventually, 51 articles were selected. The flow chart of literature review and data extraction has been shown in Appendix I.

The main characteristics of these selected articles, according to the aim and variables of the study, have been presented in Tables 1 to 2. The results showed that 10 articles (19.6%) had been published from 2004 to 2009 and 41 articles (80.4%) had been published from 2009 to 2014 (Table 1). Among these published articles, there were 26 descriptive studies (50.98%), 19 descriptive-analytical studies (37.2%), 2 review articles (3.92%), 2 descriptive and cross-sectional studies (3.92%), 1 systematic review (1.96%), and 1 mixed method (qualitative and quantitative) studies (1.96%) (Table 2).

Table 1:

The number of selected articles by the year of publication

Year of Publication	2004	2005	2006	2007	2008	2009	2010	2011	2012	2013	2014	Total
Frequency	1	0	1	1	2	5	3	9	13	11	5	51
(%)	(1.96)	(0)	(1.96)	(1.96)	(3.92)	(9.8)	(5.88)	(17.65)	(25.49)	(21.57)	(9.81)	(100)

Table 2:

The characteristics of the 51 selected articles classified by the aim and type of variables

Evaluation	Models	Survey	BSC	EFQM Model	Baldrige Model	Pabon Lasso	DEA	Accreditation	Ratio Analysis	Hybrid Models	Total
Variables											Frequency	%
Type of Studies	Descriptive			7	5	9	1	2		2	26	50.98
	Systematic Review						1				1	1.96
	Mixed Method		1								1	1.96
	Descriptive-Comparative		2								2	3.92
	Descriptive-Analytical					1	13		1	4	19	37.26
	Review	1	1								2	3.92
Number of Hospitals Evaluated	NA	1	4								5	9.8
	1			6	5					1	12	23.53
	2			1		1					2	3.92
	3–10					3				2	5	9.81
	10–20					3	8			2	13	25.5
	More than 20					3	7	2	1	1	14	27.45
Language	Persian	1	3	6	4	6	12	2		4	38	74.51
	English		1	1	1	4	3		1	2	13	25.49
Year of Publication	2004–2009	1	1	1	1	1	5	1		1	12	23.53
	2009–2014		3	6	4	9	10	1	1	5	39	76.47
	Total	1	4	7	5	10	15	2	1	6	51	100

The models of hospital performance evaluation used in Iranian hospitals could be explained as:

The standards of MOHME:

The Pabon Lasso (PL) model:

The Data Envelopment Analysis (DEA):

The EFQM Excellence Model:

The Malcolm Baldrige Model:

The Balanced Scorecard (BSC):

The Ratio Analysis (RA):

Hybrid methods:

More details are presented in Table 3 and 4.

Table 3:

The models of hospital performance evaluation used by the selected articles

Reference	Evaluation model	Objectives	The results of hospital performance evaluation
(9); (15).	Accreditation standards	Study of the hospital performance evaluation based on the accreditation standards	Emphasizing the lack of any interests for evaluating organization and paying attention to the performance indicators such as Bed Occupancy Rate (BOR), average length of stay (LoS) and Bed Turnover Rate (BTR) in the annual hospital evaluation, instead of focusing only on the availability and structural indicators (9). The lack of attention to the financial dimension was considered as the most important weakness (15).
(8); (16–24).	Pabon Lasso (PL) Model	Hospital performance evaluation using three indicators, including BOR, BTR, and Average LOS	Most hospitals had low performance in terms of their BOR, BTR, or both and the researchers had suggested the development of outpatient services, the transfer of a number of beds to other hospitals, and the prevention of developing and expanding hospitals as good and proper strategies for increasing the studied hospitals’ productivity and efficiency.
(25–39)	Data Envelopment Analysis (DEA)	Study of the hospitals’ technical efficiency using DEA	The results showed that the potential for improving technical efficiency in the studied hospitals was equal to three to seven percent. The hospital services followed the constant returns to scale. Furthermore, the surplus factors of production, especially nursing staff, were evident in the studied hospitals. Therefore, the researchers suggested removing surplus manpower.
(40–46)	EFQM Excellence Model	Hospital performance evaluation or self-assessment, and determining the areas which need improvement	Studied hospitals had been reported as poor to moderate based on the EFQM Model. The results showed the strengths and weaknesses in 9 areas of the model. Using this model, because of its systematic approach, attention to the organizational processes, and being result-oriented, had been recommended.
(2); (13); (47–49).	Malcolm Baldrige Model	Hospital performance evaluation or self-assessment, and determining the areas which need improvement	The performance of studied hospitals had been reported as poor to moderate based on the Baldrige Model. In all five selected studies, the evaluation had been carried out only in one hospital. Using this model provided the opportunities for comparing the performance of different hospitals inside and outside of the country.
(4); (6); (50–51).	BSC	Providing a model for hospital performance evaluation using the BSC	Several dimensions of patients, internal processes, financial dimension, employees’ learning and growth (48), and clinical dimension (4) have been identified and the related indicators have been determined.
			A model offered with six dimensions, including 1) hospital objectives, 2) hospital inputs, 3) structures and systems, 4) processes, 5) the outputs, and 6) performance outcomes (5).
(52)	Ratio Analysis	Comparing the performance of Iranian hospitals	Paying attention to these four indicators (BTR, bed turnover interval rate, average LoS and BOR), along with the mortality rate, and using the combination of different performance evaluation models have been recommended for a better description of the image of the hospital performance.

Table 4:

Hybrid models used for hospital performance evaluation by the studied articles

Reference	Evaluation model	Objectives	Results
(53)	A combination of the fuzzy AHP and the BSC model (FAHP-BSC)	Using the fuzzy AHP method to weigh the dimensions of the BSC and their indicators	Paying attention to the weight of and priority for each of dimensions and their indicators in the future planning and decision-making is essential.
(54–55)	A combination of the DEA and the PL model	Measuring and evaluating the efficiency of 18 general hospitals using two separate models	According to the PL model, 44.5% of the studied hospitals were efficient, while according to the DEA, 61% of studied hospitals were efficient.
		Evaluating the performance and efficiency of 23 hospitals using two separate models	There was not complete compatibility between the results of two models. It is suggested that judgment on the performance of a hospital based on separate indicators was not logical, and the use of models including several factors was necessary and more realistic.
(56)	A combination of the BSC, DEA and SERVQUAL	Determining the relative efficiency of 13 public hospitals using the BSC, DEA and SERVQUAL	The mean of studied hospitals’ relative efficiency was 0.945. The score of SERVQUAL model was also considered as an output. The researchers emphasized that the combination of the BSC and DEA could reduce the disadvantages of each of the two models and strengthen the advantages of each ones.
(57)	A combination of the DEA and Analytical Hierarchy Process (AHP)	Evaluating the relative efficiency of all Qom hospitals using the DEA and AHP methods	According to the method of constant returns to scale, three hospitals were efficient and five hospitals were inefficient. However, according to the method of variable returns to scale, four hospitals were efficient and four hospitals were inefficient. The most hospitals had not worked efficiently.
(58)	A combination of the MCDM and ratio analysis methods	Measuring the efficiency of five hospitals using the MCDM and RA methods	The RA is unable to provide a final conclusion about the efficiency and performance or ranking of a hospital. In contrast, the MCDM methods can specify the final ranking of a hospital and determine the key indicators for evaluating each type of efficiency through normalizing the data.

Discussion

Growing trend in published articles can indicate the importance of hospital performance evaluation and the efforts made to meet the need of hospitals to have a suitable model for evaluating their performance. However, the Iranian hospitals do not have good quality and efficiency yet. This can be due to the evaluation and accreditation standards and items (12). In addition, the difficulties is in determining a clear and precise strategy, the lack of employees’ knowledge of the processes of and reasons for evaluating hospitals, the lack of systemic views in the experts, the insufficient attention to or overemphasis on some parts of the evaluation process according to the personal preferences, the existence of interests for evaluating organization, not real accreditation degrees in some cases, not getting proper feedback about the results (3), and much emphasis on documentation. In addition, hospitals are suffering from shortages of manpower, lack of proper equipment, lack of positive attitudes in the hospital managers, and lack of required and adequate skills and knowledge in the field of accreditation (11).

ريالelating the performance indicators such as BOR, average LoS and BTR and also paying attention to the financial indicators are one of the priorities of the hospital evaluation system in Iran (9, 15). These weaknesses have encouraged the researchers to use other models of hospital performance evaluation.

Several studies used the PL model (8, 16–24). However, the recommendations made based on the PL model are beyond the indicators used. Some of these recommendations include to transfer a number of beds to other hospitals (20), stop the increases in the number of hospitals and their expansion due to their inefficiency and focus on the hospitals’ efficiency (16), develop the outpatient services (22), and develop and implement strategies by the policymakers for improving the efficiency of and resource allocation in the hospitals (16, 17, 23).

These recommendations cannot persuade policymakers due to the limitations in the indicators used in the analyses, not considering some issues such as the geographical location, quality of services and socio-political factors. Other factors such as management style also have effects on the improvement of hospital performance not evaluated by these indicators (18). Therefore, paying more careful attention to the indicators of managers’ competence and management style, defining the key performance indicators, making continuous improvements in the hospitals’ performance, using the evaluation results for planning and policy making, and making efforts to increase the utilization of hospital resources has also been suggested (24).

Most of the studied articles (15 of 51), had used the DEA model. According to the results, the potential for improving technical efficiency in the studied hospital was equal to three to ten percent (26, 27, 32–35, 37, 38) although the potential of 17% had also been reported (38). In addition, the surplus factors of production, especially nursing staff, were evident in the studied hospitals. Therefore, the researchers had suggested removing surplus manpower (26, 27, 32–34, 36–38). Besides, taking some measures such as improving the quality and quantity of services, increasing the financial resources, and carrying out the continuous evaluation of the performance have been recommended (34). The possibility of evaluating the performance of a large number of hospitals is also one of the advantages of using this model. Moreover, DEA has some limitations such as the methodological problems, validity and reliability limitations, and the lack of attention to the quality (25). The recommendation to reduce the human resources, due to the hospitals’ problems associated with the lack of manpower, can cause the DEA results are less applicable. In addition, other factors such as mismanagement can be the main cause of inefficiency in hospitals (about 71%) (39). Therefore, the role of management should be emphasized.

In 7 of the 51 selected studies reviewed, the EFQM had been used. The Baldrige Model had also been used in five selected studies to evaluate the performance of hospitals. In these studies, the performance of studied hospitals had been reported as poor to moderate. These models can be used for identifying the strengths and weaknesses of hospitals areas of the models. Using these models had been recommended because of their systematic approach, paying attention to the organizational processes-based management, and being result-oriented (40–46). The model is examined in several hospitals (1, 59, 60). Although, there is empirical evidence that focusing on the content addressed in the EFQM Criteria leads hospitals to performance improvement but it has a long journey to become the most important standard of hospitals (33, 48, 49). This model has often been used in one hospital and it seems that the differences in their evaluation processes prevent the possibility of comparing these models with each other, although, the most important standard of hospitals is Joint Commission Accreditation Standards (61).

Four to five dimensions of patients, internal processes, financial dimension, employees’ learning and growth, and clinical dimension have been identified and the related indicators have been determined in the BSC model (4, 50, 51). It seems that the BSC can be useful in evaluating hospitals because of the multiple criteria used in this model. However, if there is not any specific supportive policy on and management’s commitment to its use and implementation, there will not also be any possibility of applying this model. One of the limitations of the RA method is the lack of attention to the quality so that the researchers had also suggested paying attention to the quality and using the combination of different performance evaluation models for a better description of the image of the hospital performance (52).

In 6 of the 51 selected studies, the hybrid methods had been used for evaluating hospital performance. The results of using the DEA and PL models are also contradictory, so that in the Mehrtak et al. the larger number of studied hospitals was efficient using the DEA (54). However, in the Marnani et al. the larger number of studied hospitals was inefficient according to the results of DEA model (55). According to these contradictions, the researchers have suggested that use of models including several indicators is necessary and more realistic. Asadi et al. used a combination of the BSC, DEA and SERVQUAL, and tried to reduce the disadvantages of each models and strengthen the advantages of each ones (56). This model is too complex because the hospitals do not have the complete experience of implementing the BSC yet to integrate it with the other models.

The major limitation of this study was access to the required data and information because the search in the Persian databases was difficult due to the problems with the computer language. The access to the full-text of English articles was more limited because of restrictions on the payment through the banking system due to sanctions and, consequently, the lack of access to some databases.

Conclusion

Overall, the process of evaluating the performance of hospitals in recent years has attracted more attention so that the trend of publishing articles in this area has been growing. The models used to evaluate the hospital performance have been the accreditation standards, the PL model, the DEA, the EFQM Model, the Baldrige Model, the BSC, and the RA. In addition, in some cases, an attempt has been made to use a combination of these models. Some of the hybrid models include the combination of the BSC and fuzzy AHP models (FAHP-BSC), the DEA and PL models, the BSC and DEA and SERVQUAL, the DEA and AHP, as well as the MCDM and RA methods.

The current system of hospital performance evaluation is not good and sufficient and has some weaknesses. Using a combination of models to integrate indicators in the hospital evaluation process is inevitable. Therefore, the MOHME should use a set of indicators such as the BSC in the process of hospital evaluation and accreditation and encourage the hospital managers to use them.

Ethical considerations

Ethical issues (Including plagiarism, informed consent, misconduct, data fabrication and/or falsification, double publication and/or submission, redundancy, etc.) have been completely observed by the authors.