Identification of the most appropriate variables for measuring the efficiency of Iranian public hospitals: Using Delphi technique.

CONTEXT: Selecting variables is a fundamental step in evaluating comparative efficiency because the results of measuring efficiency depend on the used variables. AIMS: The aim of this study is to provide a comprehensive set of input and output variables for measuring efficiency with an emphasis on application in general hospitals in Iran.
MATERIALS AND METHODS: This study comprised a literature review followed by a Delphi survey process. After extracting the variables from the literature review in order to reach consensus on them and identify the native variables, the researchers used the Delphi technique in three rounds. Thirty Iranian hospital managers, in Alborz, Saveh, Qazvin, Qom, and Hamadan universities, participated in this study. For analysis, the interquartile range (IQR) and median were used. IQR was used to assess the agreement of Delphi panel members.
RESULTS: After literature review, nine indicators were identified as input variables and 11 indicators were identified as output variables. After the proposed changes by Delphi members, 24 input variables and 24 output variables were identified to measure hospital efficacy. Finally, ten variables were selected as inputs and ten variables were selected as outputs to measure the performance of public hospitals in Iran by using the consensus of the members in the Delphi panel.
CONCLUSIONS: This study proposes a framework for selecting the most appropriate variables for measuring the hospital efficiency with an emphasis on nonparametric methods. Choosing variables to measure hospital efficiency requires infrastructure such as an intelligent information system.

Introduction

International organizations consider economic growth through efficiency as one of the most important economic goals of the countries. Therefore, organizations’ efforts have always been to achieve maximized outcomes and output with spending minimal cost and resources.[123] In this regard, international health organizations, such as the World Health Organization, are particularly interested in assessing the performance of health systems.[45] Hence, one of the main goals of the countries is to improve the quality and efficiency of the health system and the optimal use of resources.[6] Hospitals, as the most important consumer units of the resources in the health-care sector, spend the highest costs and health budgets. This declares the importance of evaluating the effectiveness of these complex institutions.[791011] Accordingly, several experimental studies have examined the strategic importance of efficiency in hospitals. Small and large private hospitals achieve higher levels of efficiency rather than public hospitals.[12] In fact, the weakness in hospital management leads to waste of money, human resources, buildings, and equipment. By preventing or reducing the waste of resources, they can be used to provide more services, or to develop the accessibility and also improve the quality of hospital services.[13] On the other hand, inefficiency in the utilizing hospital resources is widespread in developing countries.[14] In Iran, despite the increase in the share of the health system in gross domestic product, the use of hospital resources was ineffective after implementation of the Health System Reform Plan in 2014.[4]

As a matter of fact, hospital is an organization that simultaneously faces multiple inputs and outputs, and it is not easy to assess its efficiency.[15] The methods of assessing efficiency are as follows:

Parametric methods estimate Frontier production functions with econometrics approach and calculate technical efficiencies of individual firms in an industry. However, nonparametric method uses the linear programming model to determine the best performance in a sample and then, the efficiency is measured based on the differences in the observed amounts. Data envelopment analysis (DEA) is one of the most common nonparametric methods.[412161718] Regarding the natural characteristics of hospital's services (its multidimensions), DEA is a special nonparametric method and a powerful tool for measuring hospital productivity.[8101119]

Despite the various methods for measuring the performance of health-care organizations, such as balanced score card, legal inspections, third-party assessments, and statistical indicators, there is no agreement on using the appropriate measurement method for the performance of these organizations.[8] The study of the literature reveals the diversity of techniques and the variables used in measuring hospital efficiency in developing countries. The selection of input and output variables is an essential step in comparative performance evaluation because the results of performance measurement methods depend on the variables used. While a wide range of variables have been used to measure the efficiency of hospitals in developed countries, few efforts have been made in developing countries to provide a comprehensive framework for selecting the most appropriate variables.[3]

Therefore, according to the importance of selecting the most suitable variables in hospital efficiency studies and the lack of a specific framework for presenting these variables, this study is conducted with the propose of identifying and providing input and output variables for measuring the efficiency of hospitals with the emphasis on its application in general hospitals in Iran.

Materials and Methods

After initial extraction and structured the input and output variables to measure hospital efficiency with the goal of reaching consensus and agreement on and identifying indigenous variables, the researchers used a comprehensive overview and Delphi technique in three stages. In addition, in this research, Delphi technique was used for two reasons including geographic dispersion and impossibility of running face-to-face interviews.

In this research, the modified Delphi technique was used, and several modifications were made in various questions (open/closed) and their (qualitative/quantitative) analysis method. According to the scope of the research which was limited to hospitals of the Ministry of Health in Iran, 50% (thirty persons) of the managers of Iran's medical sciences hospitals (based on the classification of education hub in the Ministry of Health), including universities of Iran, Alborz, Saveh, Qazvin, Qom, and Hamadan medical sciences members of the Delphi panel, were formed.

Table 1 shows the profile of the Delphi panel members. The entire Delphi process was conducted during the course of training of hospital managers in Iran University of Medical Sciences.

Table 1

Panel members’ characteristics

Characteristics	Number (percentage of frequency)
Gender
Female	3 (10)
Male	27 (90)
Management background (years)
10-3	8 (26)
18-10	13 (43)
26-18	9 (31)
Degree
Bachelor	14 (46)
MA	11 (36)
P.H.D	4 (18)

The selection of the members of the Delphi panel was based on their experience and knowledge about hospital management and improving the hospital efficiency, their willingness to participate in the study, and the availability of this group for researchers. In order to have a high accountability rate in starting the Delphi study, the research's information and consent forms to participate in the Delphi study were submitted to the members.

First round

In order to determine the input and output variables to assess the efficiency of hospitals, the first review of the literature was done. The review of the texts was done according to the research's subject and purpose through method.

For this purpose, the review was conducted in the period of 1999–2018 in databases including the Web of Science, Google Scholar, PubMed, Scopus, and ProQuest. The used keywords included hospital, data envelopment analysis, efficiency, technical efficiency, scale efficiency, Malmquist Index, and productivity.

The used studies included experimental investigations about measuring the efficiency of hospitals and their input and output variables. The studies which entered into the present research were supposed to have the following criteria: (1) being published in English language from January 1999 to January 2018, (2) input and output variables are defined for assessing the efficiency of hospitals, (3) DEA method has been used to measure the efficiency.

Hospital efficiency assessment variables were prepared based on a review of the literature. Then, these variables were investigated by a researcher and a health economist in terms of relevance to Iran's ground conditions. Subsequently, all the variables identified by the researcher, in the form of a list, were submitted to the members of the Delphi panel, and they were requested to indicate their input and output variables for measuring the efficiency in Iranian hospitals if they were not included in the list. Data from the first stage were analyzed using quantitative content analysis. The type of content analysis was conceptual, in which the concept was selected and the number of repetition was counted. Each variable was reconsidered if it seemed to be unclear or needed to be combined to other variable.

Second round

All the variables identified in the first stage were given to the members of the Delphi panel in a questionnaire format for scoring each of the variables based on their importance. The rating range was from 1 (unimportant) to 5 (very important). In addition, along with each variable, there was a space for expressing free comments of contributors. At the end of the second round, a report was presented to the panel members, in which the frequencies of the selected responses (distribution of scores) as well as the median scores and interquartile range (IQR) of each variable were presented. According to the researchers, achieving a median of 4 or higher, in the 5-point Likert spectrum, was a criterion for agreement between panel members. This method (definition of a median as an agreement) is one of the most common criteria for reaching consensus in the Delphi approach.[20] Finally, based on the comments and scoring of the group, modifications were made to the variables.

Third round

In this step, each member of the Delphi panel received a questionnaire including questions and ratings created by the researcher in the previous stages in order to be reviewed. In fact, the third phase provided an opportunity for the participants to review their information and judgment more clearly.[21] Hence, in the third stage, a structured and graded questionnaire designed by Likert scale was given to the experts. The method of analysis was the calculation of IQR and median. IQR was used in order to assess the success of the experts’ agreement.[2022] Furthermore, median was calculated for each item so that the agreement for the importance of each item was scored.[23]

Results

The findings of the first round were as follows:

After researching selected information bases and finding the resulted articles by using DEA, the repeated articles were removed and 35 full-text archives in English language were selected. Twenty potential indexes were recognized to be used in the mentioned studies. There were nine and ten indexes related to, in turn, input variables and output variables. Table 2 summarizes the results of this finding.

Table 2

Results of literature review

Basic classification	Description	Variable
Input variables	Variables that are considered in assessing the efficiency of hospitals as production inputs	Number of active beds
		Number of full-time employees
		Number of professional staff
		Number of managerial staff
		Number of general practitioners
		Number of specialist physicians
		Number of nurses
		Number of paramedics
		Other personnel
Output variables	Variables that are considered in measuring the efficiency of hospitals as production outputs	Number of patients discharged
		Number of surgeries
		Number of laboratory tests
		Number of outpatient admission
		Number of inpatient admission
		Bed turnover
		Bed occupancy rate
		Average length of stay
		The income-expenditure ratio

Three rounds of Delphi were done. In the first, second, and third rounds, 26 (86%), 21 (70%), and 15 (50%) people out of thirty possible respondents responded to the designed questionnaire, respectively.

In the first round, the managers announced their view points about maintaining, changing, combining, or removing the extracted variables derived from previous studies. After performing the Delphi members’ suggested changes, 24 input and output variables for assessing hospital efficiency were recognized. As indicated in Table 3 qualitative variables in addition to quantitative ones were suggested by hospital managers with regard to input variables. Geographical condition and the culture of services received were two examples of qualitative variables regarding output variables; hospital-related infections and patients’ mortality were known as negative outputs. Few of them represented the most efficient performance in hospital.

Table 3

First Delphi findings

Input variables	Frequency	Median	Interquartile range	Output variables	Frequency	Median	Interquartile range
Number of beds	21	5	1	Number of hospital discharge	21	4	0
Number of intensive beds	21	5	1	Number of outpatient visits	21	4	1
Number of general beds	21	4	1	Number of surgeries	21	5	1
Number of full-time employees	21	4	1	Number of laboratory tests	21	4	1
Number of professional nurses	21	5	1	Number of outpatient admissions	21	4	0
Number of managerial staff	21	4	1	Number of inpatient admissions	21	5	1
Number of office support staff	21	4	1	Bed occupancy rate	21	5	1
Number of general practitioners	21	4	2	Bed turnover	21	5	1
Number of specialist physicians	21	5	1	Average length of stay	21	5	1
Number of administrative staff	21	4	1	Hospital income	21	5	1
Number of service employees	21	4	2	The income-expenditure ratio	21	5	0
Grants assigned to the hospital by the university	21	5	0	The mortality rate after 24 h per 1000 discharged patients	20	4	1
Number of clinics	21	4	1	Patient satisfaction	21	5	1
Conditions and geographical situation	21	4	1	Depreciation of the building	21	4	0
Culture of the recipient community	20	4	0	Depreciation of the equipment	20	4	0
Number of outpatient service units	20	4	0	Number of cooked meals	21	4	1
Value of capital assets of the hospital	21	4	1	Number of graphs taken from patients	21	4	1
Number of paraclinical units	21	4	0	Outpatient rate	21	4	0
Tariffs for health-care services	18	4	1	Number of emergency patients’ admission	21	4	1
The price of energy	20	4	0	Average time for appointment of patients in emergency ward	21	4	1
Number of paramedical staff	21	4	0	Insurance deductions	21	5	1
Specialized equipment	21	4	1	The rate of hospital infections	20	5	0
Infrastructure of the hospital	21	4	2	Patient re-admission rate	21	4	1
Number of wards	21	4		Financial ratios (liquidity, activity, etc.)	21	5	1

The findings of the second round were as follows:

After finalizing variables in first round questionnaires Delphi, the questionnaire to measure the importance of each variable was shared with panel members. As shown in Table 2, 5 variables (about 20%), and 19 variables (about 80%) of input variables obtained the median 5 and 4 respectively. Regarding output variables, 11 variables (45%) with a median of 5 as the highly important variables and 13 variables (55%) with a median of 4 as important variables were recognized. In addition to the importance assessment, deciding about the agreement on suggested indices was known as the other finding of the second Delphi findings; in other words, out of the 24 input variables, 6 variables with very high agreement (IQR = 0), 15 variables with average agreement (IQR = 1), and 3 variables with two agreement (IQR = 2) were recognized.

Meanwhile, regarding output variables, 7 variables with high agreement and 17 variables with average agreement were recognized. Those variables with median 4 and over and IQR of 1 and <1 were inserted in the third Delphi round in order to final decision-making and agreement.

The findings of the third round were as follows:

In this round, a two-part questionnaire was given to the members. These two parts included, in turn, a report of scoring system for the second round's variables and selected variables achieved from the second round. After finishing the third round and result analysis finally, 10 variables as inputs and 10 variables as outputs were selected to measure the performance of public hospitals in Iran using the consensus views of the participants in the Delphi panel. Table 4 shows Final variables of hospital efficiency measurement.

Table 4

Final variables of hospital efficiency measurement

Input variables	Output variables
Number of beds	Number of outpatient admissions
Number of intensive beds	Number of inpatient admissions
Number of full-time employees	Bed occupancy rate
Number of specialist physicians	Bed turnover
Number of clinics	Average length of stay
Conditions and geographical situation	The mortality rate after 24 h per 1000 discharged patients
Value of capital assets of the hospital	Patient satisfaction
Tariffs for health-care services	Number of emergency patients’ admission
Specialized equipment	Average time for appointment of patients in emergency ward
Hospital infrastructure	Patient re-admission rate

Discussion

Developing countries such as Iran have considered the process of measuring hospitals’ efficiency in recent years. Meanwhile, the dependency of comparing the results of hospitals efficiency on selected variables leads to clarify the role of variable selection in measuring efficiency;[2425] However, few studies have provided a framework for selecting performance variables in the hospital. This has led to the use of studies from a variety of variables and, ultimately, to provide ambiguous and different solutions to the causes of various measurements. The present study was conducted to recognize domestic variables for measuring efficiency in Iranian hospitals. Hence, Delphi method was used by thirty managers of the hospitals affiliated to Iran University of Medical Sciences including Iran, Alborz, Saveh, Ghazvin, Ghom, and Hamadan University of Medical Sciences. Delphi research was done in three rounds to achieve alliance. Finally, 10 variables as inputs and 10 variables as outputs were selected for assessing efficiency in Iranian general hospitals as input and output variables.

The validity of these selected variables is important from some mythological dimensions. First, the members of Delphi members were experienced hospital managers who were oriented completely toward all the aspects of hospital assessment.

Furthermore, the selection of variables was done based on the viewpoints of people who were present completely in all the Delphi rounds. In addition, providing report about the experts’ viewpoints before every round is known as the strong aspect of the present investigation. However, the managers’ disorientation to the assessment models of hospital efficiency and the necessity of regarding the pros and cons of each kind of models constituted the limitations of the present study.

Based on Pabon–Lasso model, only three performance indices including bed turnover, bed occupancy rate, and average length of stay were highly regarded in assessing hospitals’ efficiency.[26] Hence, this model cannot satisfy the possibility of assessment of different performance dimensions of hospitals and compare them with each other.

In addition, in parametric methods such as Frontier production function, the assessment of hospital efficiency depends on the type of production function for evaluating the efficiency of the units.[27] Exert of assumption is the limitations of this method for measuring performance. However, in nonparametric methods such as data envelopment analysis independent of any hypothesis, the decision unit's efficiency is measured in comparison with other similar units.

It should be noted that selecting input and output variables is one of the most important aspects of this method.

In most of the studies related to assessing hospitals’ efficiency performed by DEA, a series of specific variables were used as input and output variables. A systematic review conducted in Iran demonstrated that the number of humane forces including doctors and nurses and also hospital beds was used as the input variable. Furthermore, the number of surgery cases, admitted patients, outpatients, bed occupancy rate, and average length of stay were used as output variables in the mentioned investigations.[28] Afzali et al. provided a framework to select the most suitable variables to assess efficiency in Iranian hospitals. The findings of this study showed that there is little conceptual transparency for the selection of variables, and the variables that have been selected so far limited. In this study, it has been suggested that, in addition to traditional variables, developmental variables should also be used to obtain a full spectrum of performance and quality of care. Therefore, we should have a more reliable tool for measuring the efficiency of the hospitals.[3] In the present study, members of the Delphi panel have proposed a combination of variables. However, the number of proposed variables is much higher. Based on a general rule in the Data Envelopment Analysis Method, the total number of units to be evaluated should be three times more than or equal to the total input and output variables. Therefore, a series of variables utilized in most of the studies must be used. They should consist of the number of doctors and nurses as input variables and also bed occupancy rate and number of discharged patients as output variables. According to the present study's results and the limitations of DEA method in using all of these variables, it is possible to assess hospitals’ efficiency based on different groups of input and output variables by making scenarios. This leads to the analysis of the sensitivity of efficiency scores among various hospitals according to selected input and output variables.

In the present study, some indices such as patient readmission and mortality rates were suggested as hospitals’ output variables. It should be noted that the lower index reflects the good performance and efficiency of a hospital. The DEA measures the efficiency of hospitals by using mathematical optimization approaches.

Caballer-Tarazona et al. studied efficiency analysis in three hospital units in Valencia for making suitable guidelines. They utilized DEA nonparametric approach for assessing efficiency.

In this study, the input variables included the number of physicians, the number of beds (the number of beds used for each ward) and the output variables included the number of counseling sessions, the number of successful counseling sessions, and the number of surgical interventions. In this study, it is recommended to use qualitative variables such as satisfaction of consumers as output variables In this study, it has recommended to use qualitative variables such as consumer satisfaction as output and synthetic variables that summarize various variables as input variables.[6] The results of this study showed that not only quantitative indicators such as bed occupancy and the average length of stay should be considered for performance measurement, but also quality indicators such as patient satisfaction as one of the most important outcomes of the hospital should be considered. Nevertheless, assessing efficiency in quantitative fields is less difficult than that in qualitative ones. In addition, the absence of reliable and common criteria among different hospitals to assess qualitative indices is known as the limitation of using these sorts of variables.

In general, various factors influence the selection of variables and assessment of the efficiency of hospitals in addition to the possible results and suggestions.

In this regard, the DEA method has been used in most of the performance measurement studies in Iran's hospitals. Furthermore, the emphasis has been on inputs rather than on outcomes because managers have more control over inputs than outputs. Another limitation on the selection of input and output variables in Iran is the limitation of information systems that do not provide accurate data on the variables required.[29]

Conclusions

It seems that hospital managers take into consideration quantitative indices in addition to qualitative ones for assessing hospital efficiency Of course, it should be considered that the use of a reliable and commonly used tool between hospitals to assess the quality variables for entering these variables is necessary for assessing the efficiency of the prerequisites. Health system managers and policymakers should note that the limitation of the use of resources and inputs in their health system should focus more on the efficiency and effectiveness of service sector departments and organizations. The use of performance measurement tools based on measurable and reliable variables can be used as a tool for evaluating hospital managers. Furthermore, the results can be utilized as the guidelines by hospital managers to assess and promote hospitals’ efficiency. Up-to-date and integrative information systems are one of the main substructures of efficiency assessment. The selection of suitable variables for assessing hospital efficiency needs appropriate substructures. In addition, managers’ perceptions of managerial insights and knowledge can be effective in choosing appropriate variable and measuring efficiency in hospitals.

Financial support and sponsorship

The project was funded by the Health Management and Economics Research Center at Iran University of Medical Sciences. Ethics code is IR.IUMS.REC1396.32011.

Conflicts of interest

All authors have no conflicts of interest to declare.