Program evaluation: considerations of effectiveness, efficiency and equity.

Health programs should be effective in reaching their objectives and should do so through efficient use of scarce resources. Moreover, the benefits should be equitably distributed among the targeted beneficiaries. The individual desirability of these attributes of program success is unquestioned, but their mutual compatibility should be. Narrowly based programs might be very efficient, whereas expansion into more difficult areas could be costly. Moreover, the difficult areas are likely to comprise disadvantaged groups that are inequitably served at present.The paper argues that measures of effectiveness, efficiency and equity need to be more sharply defined, so that trade-offs in program emphasis can be objectively assessed. Methods of analysis that make the trade-offs clear are pre-sented and applied to real data taken from a family planning study in India.

INTRODUCTION

Long-neglected underserved, largely rural, populations of the world are receiving increasing attention as the Health for All ideal is pursued through programs of primary health care intended to expand service coverage and improve health.1–3 The aim to reach larger numbers with services that are both effective and affordable breaks down into three distinct components requiring attention in program evaluation4. There is first the fundamental concern for program effectiveness, i.e. the magnitude of benefits achieved. The second concern relates to efficiency, i.e., the amount of scarce resources needed to produce the benefits. Finally, considerations of equity relate to the distribution of benefits in the target population in relation to individual needs5.

Objectives of effectiveness, efficiency, and equity are often considered together as if they were mutually reinforcing, or at least mutually compatible. Such assumptions must be questioned, however, in at least two regards.

First, whereas a small, narrowly targeted program may be very efficient, by its very nature it benefits relatively few individuals. Because a wider distribution of benefits can be achieved only at increasing marginal (incremental) cost, a more effective program would be less efficient overall. To illustrate this, consider the following reported experience from three immunization strategies6.

Immunization and other programs catering to those who present themselves spontaneously at a service facility can provide the service quite inexpensively, but, depending upon utilization patterns, may reach only a fraction of the target population. In contrast, outreach programs could extend benefits to those not otherwise covered, but would do so at relatively high unit cost.

The second illustration of possible incompat-ibility concerns efficiency, or cost-effectiveness, and equity. It seems reasonable to expect that the high cost of equitably serving exceptionally resistant or hard-to-reach members of the population may require considerations of efficiency to be sacrificed. On the other hand, the needs of the underserved may be so great service impact outweighs the additional costs incurred. Assessment of the additional resources required to achieve equity is the main focus of the discussion that follows.

It is evident that untested assumptions must give way to critical appraisal of the trade-offs. Of course, the appraisal requires clear definition of terms in a way that permits quantitative comparison of the measures of interest. Measures of cost-effectiveness are well-established but even here we shall note certain ambiguities that remain to be resolved. The operational meaning of equity remains ill-defined. This paper first proposes a set of measures for assessing cost-effectiveness and equity. By means of a hypothetical example we then describe methods of analysis of the trade-offs involved. Finally, the methods are applied to real data taken from a family planning study in India.

EVALUATIVE MEASURES

For evaluation purposes, programs are characterized by: inputs required (IP), usually in terms of cost; intermediate outputs achieved (OP), for example service coverage; and ultimate effect or outcome (OC), for example reduced mortality. For the purpose of dealing with issues in concrete terms, we consider a hypothetical program of measles immunization. In a certain relatively privileged community (ADV), we suppose that virtually every non-immunized child gets measles and about 4 percent die from it. Immunization coverage can be achieved for 2 Riyals per child. A second community (DIS) includes disadvantaged ethnic minorities with limited educational and job experience. Children from this community suffer a 6 percent mortality rate from measles, and the additional outreach effort necessary to gain community support leads to a cost of 4 Riyals per immunization. This information is summarized in Table 1.

Table 1

Basic Information: Hypothetical case of Measles Imrnunization Efficiency

One view of cost-effectiveness relates coverage to resources required, i.e. OP/IP. Thus, in Community ADV 500 immunizations can be achieved per 1,000 Riyals, whereas only 250 immunizations can be attained from the same resources in Community DIS. In these terms, resources can be used more cost-effectively in Community ADV.

In other cases cost-effectiveness measures relate outcome (OC) to input (IP). This leads to statements of cost per death averted, IP/OC, or alternatively specification of outcome per unit cost, OC/IP. Using the latter measure in the measles case, we find that 20 deaths can be averted per 1,000 Riyals in ADV, compared to 15 in DIS. Again, the program in ADV would be considered more cost-effective, though the advantage is not as pronounced as in the OP/IP comparison.

Since principal interest centers on outcome, the measure OC/IP is preferred, where available. Even more useful is the separation of this measure into its two components:

(1) OC / IP= OP /IP × OC / OP.

The first term on the right carries the traditional connotation of efficiency, since it relates output to input. The second term measures effect per unit of service, which might be designated impact. Then, in words (1) describes.

(2) Cost - Effectiveness = Efficiency × Impact

Unfortunately, these terms have been used to mean different things by various authors. Perhaps it would be better to adopt a new set of value-free terminology. Whatever terms we employ, however, it seems useful to recognize the three inter-related measures in Equation (1). The advantage is especially apparent when considerations of equity are introduced. To illustrate, in the immunization example (Table 1) we find that services can be provided more efficiently in ADV, but they have greater impact in DIS. The latter effect favoring DIS is not sufficient, however, to overcome the efficiency of ADV, so that overall the program is more cost-effective in ADV.

Assessment of inequities presupposes the presence of one or more subgroups with unmet needs that exceed those of more favored groups in the population. It is necessary, therefore, to add one more evaluative measure (ND) that makes explicit the level of need. This should be expressed in the same units as outcome which, after all, is an expression of need reduction. In the illustrative case need is expressed in terms of avertable deaths from measles in the target populations.

SOURCES OF INEQUITY

Conditions of disadvantage in service coverage or health status are exhibited in many ways. Information on differences among geopolitical units, such as districts, is widely available and can serve as the basis for policies on resource allocation. Urban-rural differences represent another geographic basis for inequity. Socio-economic or cultural factors are also important but may be more difficult to isolate. The importance of maternal education, for example, is increasingly recognized as a risk factor in the well-being of children as well as mothers.7 Whatever the sources of inequity, it is necessary in practice that they be identified along lines on which resources are, or might be, allocated in an effort to redress the disparities.

MEASURES OF INEQUITY

Of more direct relevance for our purposes are the measures of inequity used to distinguish groups deemed to be of interest. In some cases input measures are employed. For example, geographical differences in ratios of health personnel, facilities or hospital beds per population are frequently cited as indications of inequity. Related measures of access arc also common. Thus we find measures of the proportion of the population having reasonable access to safe water or residing within one hour travel time to a health unit8. Inequities in charges and payment for services are another concern regarding inputs that are receiving more attention as various cost recovery schemes seek ways to ensure that those unable to meet the charges will not be excluded from care.9 User charges for drugs are being promoted under the Bamako Initiative,10 and one study has analyzed the equity ramifications from the use of drug profits to support preventive services coverage among various socioeconomic groups.11 Musgrove cites evidence that the poor tend to be subsidized to a greater extent for health care than for education12.

Consideration has likewise been given to inequities in output. For example, information on eight indicators of coverage have been obtained and analyzed from forty developing countries8. The indicators include such measures as receipt of antenatal care, delivery by qualified birth attendant, immunization and growth monitoring. The authors also examined activity-to-population ratios for consultations and hospital admissions, as well as certain input indicators such as access to safe water and to health units. Comparisons among the indicators revealed that urban-rural inequalities are generally greater for sanitary facilities than for safe water, and the latter are greater than inequality in access to health units. Correspondingly, urban-rural differences in health unit consultations tend to be less than for hospitalizations. Finally, coverage with qualified attendants at delivery tends to be more inequitable than for antenatal care.

Comparative measures of health status, or need, are common, as evidenced by frequent reference to regional or socioeconomic differences in mortality or fertility. Measures of outcome, that is reduction in need, are less common because of the extended period typically required to produce these changes and the difficulty of attributing them to specific program effects. Nevertheless, outcome remains the focus of ultimate interest for most programs.

COMPARATIVE INDICATORS

It is fair to ask whether equity demands that two groups achieve equivalent status or whether it is sufficient that each exceeds a specified minimum. Most often in practice comparative rates are cited without reference to acceptable minima. Sometimes more sophisticated odds ratios or measures of relative risk are used. For example, the observation may be made that the risk of death in infancy is twice as high in one population group compared to another13.

For comparison of several groups, Lorenz curves have been drawn to depict departures from equality of circumstances. If physicians were uniformly distributed throughout the population, for example, districts with 30 percent of the population would have 30 percent of the physicians, 60 percent of the population would be associated with 60 percent of the physicians, and in general a straight-line diagonal relationship would hold throughout the range from 0 to 100 percent on both the population and resource axes. If certain districts with 30 percent of the population had 50 percent of the physicians, however, and districts with another 30 percent of the population had only 20 percent of the physicians, the Lorenz curve depicting this relationship would bend below the diagonal and the relative area between the Lorenz curve and the diagonal could be calculated as the Gini coefficient. The coefficient will vary from 0 (perfect equality) to 1. To illustrate, regional inequalities in Peru in 1982 produced a Gini coefficient of 0.51 for physicians, 0.38 for nurses, and 0.34 for hospital beds12.

For curative services, Musgrove has suggested that all aspects of equity can be assessed in terms of four probabilities12:

P(S) the unconditional probability of needing medical care;

P (T / S) the conditional probability of receiving treatment, given the need for it;

P (C / S) the conditional probability of being cured by treatment; and

P (R / T’) the conditional probability o recovering without treatment.

COST-EFFECTIVENESS AND EQUITY

Clearly, no single measure of equity can be declared definitive. It is equally clear that an assessment of equity should have two desirable features. First, it should be possible to compare the equity implications of a program strategy with its cost-effectiveness so that trade-offs can be appraised rationally. Second, the analysis should ideally be comprehensive in encompassing the full range of considerations of comparative need, input, output and outcome. We return to the immunization example to describe a method of appraisal that incorporates these two desirable features. In assessing possible strategies, we suppose that Community ADV has a target population of 8,000 children, Community DIS has 10,000 children, and there is an available budget of 20,000 Riyals, which is insufficient to reach all members of both target populations. How might the budget be allocated between communities?

COST-EFFECTIVENESS

Because the OC/IP ratio is more favorable in ADV, the most cost-effective strategy would utilize 16,000 Riyals to immunize the entire target population in ADV and to utilize the remaining 4,000 Riyals to reach 1,000 children in DIS. As shown in Table 2, the total number of deaths averted would be 380. Such an approach may be unrealistic and politically unacceptable, but it represents an extreme position as a basis for comparison.

Table 2

Comparison of Imrnunization Strategies

EQUAL INPUT PER CAPITA

As already indicated, equity can be assessed with respect to input, output, outcome or need. Perhaps the simplest strategy would be to equalize input per capita in the two communities. Considering the population ratio of 8 : 10, the strategy would allocate 8/18 of the budget, or 8,889 Riyals to ADV and the remaining 11,111 Riyals to DIS. Because immunization is twice as costly in DIS as in ADV, the coverage attainable in ADV would be twice that in DIS (56 percent compared to 28 percent). A total of 345 deaths would be averted, somewhat fewer than with the most cost-effective strategy because of the diversion of resources to DIS, where they are less productive. The expected results are summarized in Table 2.

EQUAL OUTPUT PER CAPITA

A second seemingly equitable strategy would achieve equal levels of coverage (k) in both populations. To determine the level of coverage achievable for 20,000 Riyals, considering differences in immunization cost, we have

(2.00) (8,000) k + (4.00) (10,000) k = 20,000

k = 0.357.

A 35.7 percent immunization rate translates into 114 deaths averted in ADV, 214 in DIS and 328 overall.

EQUAL OUTCOME PER CAPITA

Another view of equity would achieve similar reductions in the measles mortality rates in the two communities. Algebraically, this means

OCa / 8 = OCd / 10,

where subscripts a and d refer to ADV and DIS respectively. For purposes of budgetary allocation, the equation must be translated into input terms. Making use of the OC/IP ratios in Table 1, we have

OCa / IPa = 20

OCa / 20IPa;

and

OCd / IPd = 15

OCd /15 IPd

The original equation

OCd /8 = OCd/10

then becomes

20IPa /8 = 15IPd/10

IPa = 0.6IPd.

Thus, the budget for Community ADV should be 60 percent that of Community DIS, or 7,500 Riyals. This allocation is expected to avert 150 deaths in ADV, 188 in DIS, and 338 overall.

EQUAL RESIDUAL NEED PER CAPITA

While the preceding perspective on equity is attractive in that it focuses on the outcome of ultimate interest, deaths averted, it disregards the higher mortality rate in Community DIS.

The initial twenty point difference (60 versus 40 per thousand) would be maintained (41 versus 21). In fact, the disparity would become proportionately greater under this strategy. This leads to the most stringent definition of equity: equalization of need in the two groups. If funds are not available to eliminate measles mortality altogether, the death rates in the two groups should at least be made similar. Algebraically, this means

(NDa – OCa) / 8 = (NDd – OCd) / 10

From Table 1 we observe that

NDa = 320 and

NDd = 600;

therefore, we have

(320 – OCa ) / 8 = (600 – OCd) /10

Again converting the outcome terms to inputs according to Table 1 relationships, we have

(320 – 20IPa) / 8=(600–151P;d) / 10

Rearranging terms yields

1.5IPd – 2.51IPa = 60 – 40 = 20.

We note, in addition, the constraint IPd + IPa = 20.

It follows that 1.5IPd – 2.5IPa = IPd + IPa

or

IPd =7IPa.

Therefore, 17,500 Riyals is to be allocated to DIS, and 2,500 Riyals goes to ADV.

COMPARISON OF STRATEGIES

Table 2 shows that each of the “equitable” strategies would avert considerably fewer deaths than the most cost-effective approach. The relative trade-off depends, of course, upon how stringently equity is defined. The “equal input per capita” criterion is least stringent and would result in 345 deaths averted, which is 9 percent below the maximum possible, 380. In contrast, the strategy to equalize death rates in the two populations would avert 18 percent fewer deaths than is possible with the most cost-effective approach.

Even greater differences are revealed among strategies with respect to coverage. All of the ADV children would be vaccinated under the most cost-effective strategy; coverage would drop to 56 percent under the “equal input” approach; and it would reach only 16 percent if the most stringent equity criterion were applied. The corresponding coverage levels in DIS would be 10, 28 and 44 percent.

The magnitude of the differences depends, of course, upon the specific conditions assumed in the hypothetical example. Moreover, we should note that the artificial illustration is unrealistic, in that it has assumed constant marginal costs and benefits. The assumption that it is no more costly to go from 80 to 90 percent coverage than it is to increase coverage from 30 to 40 percent must be tested in practice. Nevertheless, the example has served to highlight certain principles. The application of the methods to real data will be more informative and in addition will force us to grapple with the practical problems of measurement. We turn next to such an application.

PRACTICAL APPLICATION: THE NARANGWAL PROJECT

The evaluative framework we have outlined was used in the assessment of findings from the Narangwal Research Project conducted in Punjab State, India during the period 1969 - 1974 and designed to test the effects of alternative combinations of health and family planning services. Although the research findings are no longer current and have been analyzed extensively elsewhere14, the Project results remain relevant because of the extensive and careful measurement made of both inputs and outputs. As a research undertaking, the measurement of project effects was crucial in line with the research objectives. It was recognized, however, that regardless of the magnitude of these effects, the findings would be of little practical value unless the services proved to be affordable to the government and capable of wider application. Since a range of health, nutrition and family planning services was offered, it was essential that costs of the individual components be established and the most cost-effective combinations determined. Methods of functional analysis developed earlier1516 were employed for this purpose. Only the family planning component will be considered here.

The Narangwal data are also useful because of the presence of two main caste groups. The Jar (high) caste was traditionally advantaged whereas the Ramdasias were thought to be underserved and were the special focus of project services. Thus, questions of equity as well as effectiveness were consciously addressed, although the former was not defined in advance.

INPUTS

Personnel inputs, the single largest element of cost, were determined through work sampling methods of functional analysis. Other costs were derived similarly. For example, space charges were assigned according to which personnel used the space (e.g., a consulting room) and how those personnel allocated their time among service components. In this way, costs were calculated for each service activity and these costs were distributed between castes according to the volume of each category of services received. The results are summarized in Table 4.

Table 4

Ratios for cost-effectiveness and equity calculations by caste

OUTPUTS

Outputs were measured in terms of family planning acceptance rates. By the end of the project, acceptance rates for the two castes were nearly the same: 45.9 percent for high caste couples and 47.0 percent for the low caste. Pre-project use of modern methods had reached 20.6 percent for the high caste and 12.8 percent for low caste couples. Following usual evaluation practice, the project can only take credit for the incremental effects, 25.3 and 34.2 percent respectively. These outputs are shown in Table 4 as rates per thousand target population.

OUTCOMES

Impact on fertility was the ultimate effect of interest, and this depends upon a number of factors, including method effectiveness and duration of use. In an attempt to establish a similar measure of outcome that would still provide a satisfactory basis for assessing cost-effectiveness and equity, we followed the approach illustrated in Table 3 for the study population of both castes.

Table 3

Illustrative calculation of outcome from Narangwal project results

Of the 3,541 members of the target population 1,027, or 29 percent, were current users of some method at the end of the study. A separate analysis had shown that non-users experienced 407 pregnancies per 1,000 couple years of experience. In contrast, those reporting use of the condom had 268 pregnancies per 1,000 couple years of experience. The condom was considered to have averted 139 (407 - 268) pregnancies and therefore to be 139/407, or 34.2 percent effective. Thus, according to Table 3 the 398 condom users represented the equivalent of 136 fully protected users. Effective use rates were similarly calculated for other methods. Presumably sterilization was fully effective, but a record of pregnancies was found for a few cases that reported sterilization, and the effectiveness rate therefore reflected such reporting errors.

The 1,027 actual users were the equivalent of 700 fully protected users. Again, however, the project could not accept full credit for the protection achieved, since 233 of the users had been acceptors of modern methods during the pre-project period. Details of prior use were not available, but considering the limited range of available methods and inadequacies of follow-up, the relative effectiveness for those users was considered to be 30 percent. Thus the 233 users would have produced the equivalent of 70 fully protected users. The net gain equivalent to 630 fully protected couples from 3,541 eligibles represents a net use-effectiveness of 178 per thousand eligibles, which is the measure of outcome used. The corresponding measures derived for the two castes is shown in Table 4.

FINDINGS

The separation of cost-effectiveness into its two components in Table 4 produces unexpected results. Determinations of efficiency (OP/IP) favor the low caste group, contrary to prior assumptions regarding their resistance to family planning acceptance. The impact (OC/OP) of acceptance on this group is less than that for the Jar (high) caste, also contrary to expectations. Despite these reversals from expectation, the overall measure of cost-effectiveness (OC/IP) favors the high caste, as anticipated, thereby bringing cost-effectiveness and equity considerations into conflict.

Further investigation revealed that low caste couples had about as much pre-project experi-ence with family planning as high caste couples. Because the former mainly used ineffective tradi-tional methods, however, they did not get counted as “acceptors”. In fact, however, they were highly receptive to the notion of family limitation from the beginning. As in the past, however, low caste project acceptors tended to rely on less effective methods, notably the con-dom. Thus their level of impact did not reach that of the high caste group. The separate ap-praisal of efficiency and impact suggested that the main attention in the disadvantaged low caste population need not be focused on gaining acceptance of family planning but upon contin-ued and effective use after acceptance.

Considering the edge in cost-effectiveness in favor of the advantaged, what are the implications for equity and to what extent did the project, in fact, achieve equity between the castes? Results of this analysis are summarized in Table 5.

Table 5

Ramifications of different resource allocation strategies

The study population was 58.7 percent high caste. Therefore, Table 5 is based upon a hypothetical population of 10,000 eligible couples, consisting of 5,870 Jats and 4,130 Ramdasias. From Table 4 we note that over the period of approximately three years covered by the data, the project spent $1,828 per thousand eligibles. Table 5 then examines alternative ways of allocating $18,280 to the population under consideration.

The cost-effective extreme would allocate as much as necessary to fully meet the need of the Jats, (that is to reach the equivalent of 300 fully protected couples per thousand), and to allocate residual funds to the low caste group. Other more equitable possibilities explored include the same criteria considered earlier, namely equalization of input, output, outcome, or need per capita. The allocation actually employed is also shown for comparison.

In this case, the several options differ little in their overall outcome, but they differ markedly in the results produced in the separate caste groups. According to criteria chosen for resource allocation, the level of effective protection per 1,000 couples could range from 175 to 245 for Ramdasias and 245 to 300 for Jats. Although a definition of equity was not established for the project in advance, in fact it came close to meeting the most stringent criterion, equalization of need.

CONCLUSIONS

Although the notion of cost-effectiveness is well-established, complete uniformity in application of the measure remains elusive. We have focused on the association between outcome and input. An important advantage of this definition is that it permits separate analysis of the efficiency and impact components. Data from the Narangwal project indicated that each component can be very informative in its own right apart from their composite effect. Because the combined effects of efficiency and impact can lead to a measure of cost-effectiveness that is not entirely compatible with the notion of equity, it is important that the latter he defined clearly. At present the term is considerably more ambiguous than the meaning of cost-effectiveness. In illustrating the use of various reasonable criteria, we have seen that markedly different results can be attained, depending upon which meaning is attached to the concept of equity.

In practice, the derivation of appropriate numerical values for use in the analysis can be difficult. The difficulty arises first in determining the net effect directly attributable to the project in question. Moreover, interest in a specified ultimate outcome must be weighed against the desire for simplicity and a reasonably prompt appraisal of outcomes. An acceptable compromise reached in the Narangwal project was described.