Efficiency measurement of health care organizations: What models are used?

Background: Literature abounds with various techniques for efficiency measurement of health care organizations (HCOs), which should be used cautiously and appropriately. The present study aimed at discovering the rules regulating the interplay among the number of inputs, outputs, and decision- making units (DMUs) and identifying all methods used for the measurement of Iranian HCOs and critically appraising all DEA studies on Iranian HCOs in their application of such rules.
Methods: The present study employed a systematic search of all studies related to efficiency measurement of Iranian HCOs. A search was conducted in different databases such as PubMed and Scopus between 2001 and 2015 to identify the studies related to the measurement in health care. The retrieved studies passed through a multi-stage (title, abstract, body) filtering process. Data extraction table for each study was completed and included method, number of inputs and outputs, DMUs, and their efficiency score.
Results: Various methods were found for efficiency measurement. Overall, 122 studies were retrieved, of which 73 had exclusively employed DEA technique for measuring the efficiency of HCOs in Iran, and 23 with hybrid models (including DEA). Only 6 studies had explicitly used the rules of thumb.
Conclusion: The number of inputs, outputs, and DMUs should be cautiously selected in DEA like techniques, as their proportionality can directly affect the discriminatory power of the technique. The given literature seemed to be, to a large extent, unsuccessful in attending to such proportionality. This study collected a list of key rules (of thumb) on the interplay of inputs, outputs, and DMUs, which could be considered by most researchers keen to apply DEA technique.

↑ What is “already known” in this topic:

There is a well-established association between the number of inputs, outputs, and DMUs selected and efficiency scores

→ What this article adds:

This study collected a list of key rules (of thumb) on the interplay of inputs, outputs, and DMUs, which could be considered by most researchers keen to apply DEA technique.

Introduction

Efficiency simply means how optimal an organization makes use of its inputs to produce outputs (1), that is, the most goods or services out of the least resources. More broadly, it is conceived as the ability to do things right (2). Measuring and comparing the efficiency of decision- making units (an efficiency related term for organization, DMU hereafter) is useful for cutting costs and improving resource management. Thus, in recent years many studies have addressed efficiency measurements (3).

Efficiency is a broad concept, with varying types including technical, allocative, economic, scale, and management. Technical (productive) efficiency (TE) is the difference between the actual output/input ratio and the ideal ratio, made up of scale and pure (managerial) efficiency (4). Scale efficiency implies that the production share of each firm is optimal when the firm produces (good, service) at the minimum point of average cost. Management efficiency refers to using correct and optimal methods for management. Allocative (price) efficiency occurs, where the price equals the marginal cost of the resources used up in the production (5). However, in practice, it is defined as choosing a combination of health care interventions, which besides minimizing the cost of producing each service, maximizes cost-effectiveness (6). Economic efficiency is technical multiplied by allocative efficiency, broadly pointing towards productivity, performance, quality, profit, and the reduction of total workforce employed and costs (7).

Efficiency could be measured easily by dividing its outputs by inputs. However, given the complexity of DMUs resulted from their numerous, and often various, objectives and multiple inputs and outputs, the measurement is complicated and challenging. Similar to the diversity of efficiency concept, there are fairly various approaches for measuring efficiency. Generally, efficiency assessment can be done by various methods including ratio analysis, least-squares regression (LSR), total factor productivity (TFP), stochastic frontier analysis (SFA), and Envelopment Analysis (DEA) (8-10). Specifically, they could be classified into parametric and nonparametric methods. Data (DEA) is a known nonparametric method for the analysis of technical efficiency. Using a set of data, an empirical efficiency frontier as a best practice frontier is created, which envelopes all other observations of DMUs. It is a linear programming technique that has no parameter to estimate. There is no restriction on the number of DMUs addressed as well as on the output and input variables (11).

A firm using 2 inputs of X1, X2 and producing output Y, uses a production function to determine how much output Y should be produced (Fig. 1). The production function represents the maximum output that an organization can attain with the given combinations of X1and X2. As displayed in the figure, Points 1 and 3, which are nearest to the origin and X1 and X2 axes would technically be efficient points because they could produce a unit of output by lesser inputs. A straight line between the 2 given points shows the efficiency frontier. A line parallel to X2 axis and another line parallel to the X1 axis can generate efficient frontiers (Fig. 1). Technical efficiency of x1 and x2 combination is as follows:

Fig. 1

Efficiency frontier in input-output map

Technical efficiency ranges from 0 to 1. A DMU on the efficiency frontier will receive a score of 1 and those not on the efficiency frontier line will be between 1 and 0. The frontier has been static, but it can be dynamic, and its status may change over time by technology improvements.

DMUs produce outputs using inputs in a given process. A limitation of DEA technique is that the result highly relies on the numbers of the inputs and outputs and their relationships, which should be selected cautiously, as any change in the number of those variables could affect the efficiency scores. This is a challenge for users of this technique because when the proposed rules regarding the numbers of inputs and outputs in proportion to the number of DMUs are not followed, some inefficient units may be wrongly considered efficient. Additional number of inputs and outputs or a strong correlation between the inputs or output themselves might also decrease DEA discriminatory power. Therefore, these issues should be in a way addressed before DEA analysis (12). For example, in the case of preexisting correlation, the variables with the lowest co-correlation and the strongest relationship with the commensurate inputs or outputs should be selected. As to the excessive number of inputs and outputs, some rule of thumbs, discussed in details later, should be followed, even though these have always been challenging for the researchers. DEA is also a common method used in estimating the efficiency of health care organizations such as hospitals.

Many studies have been conducted to investigate the productivity and efficiency of health care services (13-15). Given the abovementioned challenges, the results of these studies should be cautiously applied, particularly in developing countries. In DEA, the setting, assumptions, type, and the number of variables could affect the results. However, there is no universally accepted criterion or gold standard to compare the studies that measure the efficiency of health care units. Most of the studies on the efficiency measurement of Iranian health care organizations (HCOs) have not pointed to any clear rule on the proportion between the number of inputs, outputs, and DMUs. For example, in a systematic review of relevant literature, Jahangiri (16) argues that the theoretical aspects of DEA and its applicability should be considered for valid decisions. In addition, large numbers of inputs and outputs in proportion with the number of DMUs may diminish the discriminatory power of DEA (17).

The current study is of threefold objectives; firstly, to identify all models and methods used for the efficiency measurement of HCOs; secondly, to discover the rules regulating the relationship between the numbers of inputs and outputs in relation to the numbers of DMUs; and finally, to critically appraise all the Iranian context related DEA studied with respect to their application of the rules corresponding to the number of inputs and outputs and DMUs.

Methods

This review study has employed a systematic search of all studies related to efficiency measurement of HCOs in Iran. The term of ‘efficiency’ was the only keyword used for searching efficiency measurement literature, and other similar words such as productivity, performance, and efficacy were not included. Inclusion criteria for selecting the articles of study are as follow:

Studies related to efficiency measurement in health care

Indication of any model or method of efficiency measurement by the articles

Articles in both English and Persian Languages

Time period between 2001 and 2015

Search strategy was built using Boolean operators (AND, OR, NOT), in which the title and abstract of articles were searched separately in 2 languages. The search strategy and databases are illustrated in Table 1.

Table 1

Search strategy and databases searched

General search strategy	Iranian databases	International databases
*((((health*[Title/Abstract]) OR hospital[ Title/Abstract])) AND Iran[Title/Abstract]) AND efficiency[Title/Abstract]	SID, Magiran, Iranmedex	Scopus, Web of Science, Pubmed, Springer, Science Direct

Data extraction

Study identification: A 3-step filter was employed to select the final articles based on their title, abstract, and whole body considering preset inclusion criteria. One investigator (BR) screened the title and abstract of the articles identified. A small percentage of the articles (25%) were randomly rechecked by a second reviewer (EJ) and a third reviewer became involved in case of any disagreement (SE). Given the large percentage of the final articles using DEA for efficiency measurement, this study focused on identifying the rules related to the relationship among inputs, outputs, and DMUs in such a method. Therefore, the articles were critically appraised mostly with respect to addressing the rules regarding the relationship among inputs, outputs, and DMUs. These rules were identified drawing on the existing referral books in health care efficiency measurement by DEA (11,18-21).

Data extraction table for each study was completed including the title of study, year, method, number of inputs and outputs, DMUs, and their efficiency score. Finally, the findings were reported in a descriptive way.

Results

1. Efficiency measurement techniques/models

In the initial search, we retrieved 1833 studies. As Diagram 1 exhibits, 181 studies were excluded because of duplication. Title and abstract screening led to the exclusion of 1493 articles, collectively, most of which were in Persian. Out of 188 articles, 36 were removed considering the main inclusion criteria, especially any use of efficiency measurement method or model for HCOs. One study was also excluded due to lack of access to its full text. A total of 122 papers reached to final stage of paper identification and assessment (Fig. 2), of which 32 were in English and 90 in Persian. Overall, 73 studies exclusively used DEA, 17 applied Pabon Lasso, 5 used stochastic frontier analysis, 1 employed Malmquist index, and 1 study used a researcher-built model. The remaining 25 papers had adopted hybrid approaches (Table 2).

Fig. 2

Table 2

Number of studies in each model

Type	Efficiency measurement methods	No. of studies
Single model	DEA	73
	Pabon lasso	17
	Stochastic frontier analysis (SFA)	5
	Malmquist index	1
	Researcher-built model	1
Mixed models	DEA and SFA	6
	DEA and Malmquist index	6
	DEA and AHP	4
	DEA and Pabon Lasso	2
	DEA, Pabon Lasso, Malmquist index	2
	DEA, Balanced Scorecard and Servqual	1
	DEA and Goal Programming	1
	DEA and Bargaining game	1
	SFA and Balanced Scorecard	1
	Ratio Method and multi-criteria decision-making method	1

Studies included in the systematic search

There have been a large number of studies on the efficiency of the public and private sector organizations worldwide. In Iran, more than 120 studies were published on the efficiency of HCOs during 2001 and 2015, most of which had applied DEA. As demonstrated in Fig. 3, most of these researches were published during 2008 and 2012.

Fig. 3

Number of efficiency studies during 2001 and 2015

Various organizations were under study in these papers (Fig. 4). Most of the papers (76%) assessed efficiency in hospitals, 9% compared efficiency in health units and health system between Iran and other countries and also among the province; 8% measured efficiency in health centers; 4% in medical universities, schools, and departments; 2% assessed the efficiency of the health insurance organizations; and finally, 1% assessed the performance of health managers (Fig. 4).

Fig. 4

Numbers of studies related to various organizations in the health sector

2. Rules on the relationships among the number of inputs, outputs, and corresponding DMUs

In DEA, the appropriateness of inputs, outputs, and DMUs number is a key determinant, which directly influences the validity of the results. There are several rules of thumbs in the literature regulating such a relationship. Our review found 5 main rules regarding the number of inputs and outputs in relation to the number of DMUs in health care organizations including (M is the number of inputs and S the number of outputs).

R1: The number of DMU ≥ max {(m × s) or 3 (m + s)}

R2: The number of DMU ≥ 3 (m + s)

R3: The number of fully efficient DMUs (FEDMUs) ≤ 1/3 DMUs

R4: The number of DMUs ≥ 2(s + m)

R5: The number of DMUs ≥ (m×s)

3. Appraisal of the studies with respect to the rules compliance

Overall, 96 studies had employed DEA technique to measure the efficiency of HCOs in Iran during 2001 and 2015, of which, 23 had hybrid models (Table 2). Only 6 studies had explicitly used the aforementioned rules out of 96, among which, 4 observed the R2, 1 study complied with R4, and finally 1 study applied R5. However, further analysis revealed that out of 96 studies, 44 had ignored the R2 completely and 48 had not clearly R2 indicated but followed it accidentally.

Discussion

The number of studies on efficiency measurement of HCOs in Iran has increased remarkably from 2001 to 2015, and the majority used the frontier approaches such as DEA and SFA. Almost 60% of studies applied DEA model given its clear advantages (22-27). Contrary to other similar techniques, DEA is able to consider multi-output and input processes, which are inherent in health care (28-30). Moreover, as DEA does not mostly address profit maximization or cost minimization (31,32), it is more applicable in not-for-profit organizations such as public hospitals. It is also able to combine inputs and outputs with various natures to generate efficiency score (31). Hospitals were at the forefront of efficiency measurement studies (94% of studies) because of their massive financial transactions in health sector and their large share of overall health care costs (33), requiring strict efficiency enhancement and cost containment. Not need to mention the fact that they are multiproduct organizations.

From 1978 that Charnes et al. (34) introduced DEA, the number of studies in this area have increased considerably. Despite these benefits, DEA suffers from some disadvantages (35-38). For instance, it chooses the frontier from given DMUs and the comparison is made inside the reference set. Thus, if the frontier is not of a high efficiency, all results might be invalid, without rendering a real picture of the efficiency status. Therefore, the input/output-oriented approaches present different efficiency results. In response, the mixed methods of efficiency measurement are introduced, moving towards combining DEA with SFA or Malmquist index. As each model has its own advantages and disadvantages, using hybrid models may overcome the disadvantages and strengthen the advantages in making a comprehensive model with more robustness.

In addition, 5 main rules were found influencing the validity of DEA results, which can allegedly enhance the discriminatory power of DEA models and improve the quality of results (39). Overall, the more the variety of inputs and outputs, the more comprehensive efficiency measurement results could be achieved. However, the discriminatory power of DEA increases if the number of DMUs rises in line with the number and variety of inputs and outputs (40). On the other hand, along with a growth in the number and variety of inputs and outputs and resulting heterogeneity, corresponding exogenous effects might appear and affect the results (12). In fact, a larger data set requires more accurate and complicated calculations.

Some studies have suggested that the small number of DMUs in DEA compared to that of inputs and outputs could lead to bias, while the high number of DMUs, if it is in proportion with the number of inputs and outputs, could shift the efficiency frontier towards a production (a real) frontier (41-44). Therefore, drawing on rules of thumb, the selection of inputs, outputs, and DMUs should be done cautiously, in a way that less important variables could be ignored or merged. As in the other measurement techniques such as regression, there is a degree of freedom for selecting variables; DEA is not an exception and its applicants are advised to use these rules. These rules are highly cited in the literature (11,45-47). Especially, R2 is a commonly suggested rule on the interplay of inputs, outputs, and DMUs (called ‘gold standard’ hereafter) in most studies measuring the efficiency of HCOs (17,48-50).

A tiny percentage of the studies were found to be observing the rules on the interplay of inputs, outputs, and DMUs, specifically the ‘gold standard’. It seems either there is no overall consensus in the usefulness of these rules for generating accurate efficiency measurement results or they are ignorant of the rules. As most of the related literature published especially, in developed countries, have in a way implied to these rules, the latter appears more likely, though not certain.

This review also revealed that most of the studies measure the technical efficiency and very few looked at other types such as allocative, economic, scale, or management efficiency. The reason is that these types of efficiencies require price of inputs, which are hardly accessible, or valid in HCOs.

Conclusion

HCOs are complicated systems, thus, efficiency measurement should be conducted based on transparent principles. The number of inputs, outputs, and DMUs should be cautiously selected as their proportionality can directly affect the discriminatory power of DEA technique. No attention to the rules regulating such proportionality might lead to many efficient DMUs in DEA analysis, when they are not de facto. Despite a strong tendency of many researchers to DEA, it appeared not to be used correctly in most of the literature investigated, conveying a wrong message to the managers and policymakers of their HCOs’ performance status.

This study mainly investigated that whether a right number of inputs and outputs was considered in proportion to sample size (the number of DMUs), and vis-à-vis by Iranian DEA related literature published during 2001 and 2015. It has further identified, as a key contribution, a list of rules on the interplay of inputs, outputs, and DMUs, which could be considered by most researchers keen to apply DEA technique. Another study could be initiated to systematically appraise all related literature worldwide.

Conflict of Interests

The authors declare that they have no competing interests.