Assessing the population health impact of market interventions to improve access to antiretroviral treatment.

Despite extraordinary global progress in increasing coverage of antiretroviral treatment (ART), the majority of people needing ART currently are not receiving treatment. Both the number of people needing ART and the average ART price per patient-year are expected to increase in coming years, which will dramatically raise funding needs for ART. Several international organizations are using interventions in ART markets to decrease ART price or to improve ART quality, delivery and innovation, with the ultimate goal of improving population health. These organizations need to select those market interventions that are most likely to substantially affect population health outcomes (ex ante assessment) and to evaluate whether implemented interventions have improved health outcomes (ex post assessment). We develop a framework to structure ex ante and ex post assessment of the population health impact of market interventions, which is transmitted through effects in markets and health systems. Ex ante assessment should include evaluation of the safety and efficacy of the ART products whose markets will be affected by the intervention; theoretical consideration of the mechanisms through which the intervention will affect population health; and predictive modelling to estimate the potential population health impact of the intervention. For ex post assessment, analysts need to consider which outcomes to estimate empirically and which to model based on empirical findings and understanding of the economic and biological mechanisms along the causal pathway from market intervention to population health. We discuss methods for ex post assessment and analyse assessment issues (unintended intervention effects, interaction effects between different interventions, and assessment impartiality and cost). We offer seven recommendations for ex ante and ex post assessment of population health impact of market interventions.

We present a framework to structure the assessment of population health impact of antiretroviral treatment market interventions, both for funding decisions and for empirical evaluation of impact.

Market interventions achieve population health impact through effects on markets (e.g. on product quantity or quality, speed of delivery, or research investment) and health systems (e.g. coverage, efficacy, effectiveness, or scope of treatment).

We describe assessment methods and their limitations and seven practical recommendations are made for assessment of population health impact of market interventions.

Introduction

In the 2010 report Towards Universal Access, the World Health Organization (WHO) notes a remarkable worldwide increase in the number of people receiving antiretroviral treatment (ART), from an estimated 700 000 in 2004 to more than 5 million in 2009 (WHO/UNICEF/UNAIDS 2010). Despite this extraordinary progress, about two-thirds of people, who according to the 2010 WHO guidelines needed ART in 2009, did not receive it (WHO/UNICEF/UNAIDS 2010). Moreover, the total number of people needing ART is expected to increase substantially in coming years, both because current ART is effective in decreasing mortality in HIV-infected individuals (Herbst ; Bärnighausen ) and because the new WHO ART guidelines recommend ART initiation at an earlier HIV disease stage (WHO 2010). Finally, for first-line ART the new WHO guidelines recommend antiretroviral drug combinations with more expensive drugs than the previously recommended combinations (WHO 2010), implying higher treatment costs per patient-year. For these reasons, the future financing required to achieve universal coverage will be vast and is unlikely to be met. Even at the current level of ART need, available funding falls far short of funding need. UNAIDS estimated a gap of US$6.5 billion between the funding ‘available for AIDS from all sources’ and the ‘UNAIDS estimate of resources needed’ in 2008 (Kates ), and the Global Fund to Fight AIDS, Tuberculosis and Malaria predicts that its future funding will ‘fall short of the estimated resources needed to meet demand from developing countries seeking to further scale up their disease programs’ (The Global Fund to Fight AIDS, Tuberculosis and Malaria 2010). It is thus important to consider how limited financial resources for ART can be used most efficiently.

In service provision, changes in the structure of ART programmes (Bärnighausen 2007), task shifting (Lehmann ; Shumbusho ; Zachariah ; Bärnighausen ), and technology changes can improve efficiency. In the supply of products required for treatment, market interventions can increase the quantity, quality, reliability, delivery, or rate of innovation of ART products, and thus affect the efficiency of resource use. Examples of ART market interventions include:

Financial incentives to improve investment in the development of ART-related products (such as advance market commitments, priority review vouchers, or innovation inducement prices);

Product prequalification by international organizations to ensure donor funds are used to purchase quality medicines in an accelerated time frame;

Patent pools, which allow more efficient licensing of patented antiretroviral medicines (ARV) for generic production in return for royalty payments;

Equitable access licensing, i.e. licenses for new ARV allowing generic manufacturers to produce the licensed products for markets in developing countries, thus reducing the ARV prices in these markets;

Purchasing of ART products from multiple producers to keep markets competitive;

Pooling of orders to increase the efficiency of price negotiations and balance negotiation power on the demand and supply sides;

Third-party price negotiation;

Technology transfers to develop and produce medicines;

Training of Ministry of Health (MoH) staff and ART programme managers in supply chain management to ensure uninterrupted availability of ARV in all treatment sites and to increase the efficiency of ARV supply chains.

These market interventions are described in a number of publications (Herrling 2008; Hecht ; Waning ; GHT Coalation 2010; Noehrenberg 2010; Waning ; Waning ). While the intended direct effect of most market interventions is to increase the value of ARV products that can be obtained for a given budget, most organizations that have used, or plan to use, market interventions—including UNITAID, the Clinton Health Access Initiative (CHAI), the United States President's Emergency Plan for AIDS Relief (PEPFAR), and the Global Fund to Fight AIDS, Tuberculosis (TB) and Malaria—ultimately aim to improve population health in the developing world. These organizations are increasingly asked to produce evidence that their actions improve not only market conditions but also population health (Feachem and Sabot 2006; Feachem and Sabot 2007). The organizations thus face the following two tasks: first, to select those market interventions proposed for funding that are most likely to substantially affect population health and, second, to evaluate whether implemented interventions have led to population health improvements. The first task demands assessment before a market intervention has been implemented, while the second calls for assessment after intervention implementation. In the following, we refer to assessment before intervention implementation as ex ante assessment and assessment after implementation as ex post assessment. The distinction between the data and analytic requirements for these two different types of assessment has not always been recognized.

Below, we first describe a framework that can be used to analyse the pathways from market interventions to population health. We then consider the content of ex ante assessment, describe different approaches to ex post assessment, and discuss five issues in the assessment of market interventions (unintended effects, interaction effects, and the time horizon, impartiality and costs of assessment). Finally, we offer seven recommendations for the evaluation of population health impact of market interventions.

The pathways from market interventions to population health outcomes can be divided into three steps (Figure 1). First, an ART market intervention should have market effects. Intended market effects include ARV price reduction, product quality improvement and assurance, improved product delivery, and advances in innovation. Second, these market effects can entail health system effects, such as increased coverage of the population needing ART, improved ART efficacy or effectiveness, or expanded scope of treatable diseases and conditions that ART patients suffer from. Finally, health systems effects can translate into changes in the morbidity and mortality of populations needing HIV care and ART. Distinguishing between these three steps is useful in identifying the pathways through which market interventions can affect population health and in assessing how to quantify population health impact.

Figure 1

Pathways from market interventions to population health impact

Ex ante assessment

For organizations aiming to improve population health through market interventions, it is essential to compare estimates of potential health impact in selecting proposals for funding. Of course, in such ex ante assessments effects cannot be measured, since the intervention has yet to take place. However, three types of assessment are possible at this stage. First, a due-diligence evaluation of the safety and efficacy of existing products can be undertaken to ensure that a market intervention does not harm patients and, at least in optimal circumstances, will improve the health of individual patients. In most cases, safety and efficacy assessment can be based on the results of published randomized controlled trials and national market approvals.

Second, the precise mechanisms through which a project is thought to affect population health can be described qualitatively, using existing evidence to support each step of the hypothesized pathways (using, for instance, the structure proposed in Figure 1). Such a description will serve to establish the plausibility of the population health impact related to a particular market intervention. Estimates of market effect should be conducted in a manner consistent with economic theory, such as theories of optimal procurement (Naegelen 2002; Chen ) or oligopolistic competition (Vives 1999). For example, an intervention that decreases the price of an ART product should be expected to increase the quantity of the product sold. Estimates of impact on population health, on the other hand, should be in agreement with biological theory and clinical evidence. For example, increases in ART adherence should improve treatment effectiveness and reduce HIV-related morbidity and mortality.

Third, the quantitative market effects, health system effects and population health impact can be predicted in modelling exercises, using assumptions about the size of the effect of market interventions on markets, health systems, and morbidity and mortality, based on the empirical literature or expert opinion. For instance, structural equation models have been used to estimate the market effects of patent protection (Chaudhuri ). Past models of the impact of HIV interventions on population health outcomes have included models that are calibrated using one parameter (Granich ; Hecht ) or multiple parameters (Stover ; Lima ), as well as system dynamics models that reflect the underlying mechanisms of the interventions (Bärnighausen ; Bärnighausen and Bloom 2009). In some circumstances, such extensive modelling exercises may be possible to forecast intervention health impacts; however, in many other situations, quantitative predictions of health impact may be based on less resource-intense modelling, such as simple translations of expected market results into health outcomes using published estimates of treatment effectiveness.

Ex post assessment

In contrast to ex ante assessment, ex post assessment can, at least in principle, be based on observed outcomes, allowing evaluation of intervention effect and impact. The difficulty in measuring outcomes and attributing them to particular market interventions will commonly increase from left to right along the pathway shown in Figure 1. One reason for this progression is that the effect of other factors on outcomes is likely to increase along the pathway. For instance, education, nutrition and the physical environment are likely to be important determinants of population health, such as morbidity and mortality in people needing ART, in addition to the level of treatment coverage. The same factors, however, are unlikely to produce market effects, such as the quality of an ART product. We would expect that it will be more difficult to adequately control for the effect of confounding factors on health systems and health, and hence to attribute these outcomes to market interventions, than to control for the influence of confounding factors on markets. Moreover, the length of lag times between market intervention and outcomes increases along the pathway. While a market intervention may lead to increased coverage with certain ARV within weeks, the impact of this health system effect on population health may not be observable for several months (e.g. mortality declines in people needing ART) or years (e.g. changes in the rate of development of drug-resistant virus). Hence, it will be more difficult to control for secular changes in the evaluation of intervention effects on population health than on markets or health systems.

Finally, the difficulty of measuring results increases along the pathway. Short-run market effects, such as prices and quantity, can commonly be observed with good accuracy in existing central databases (Waning ). In contrast, health outcomes, such as cause-specific mortality, are rarely measured accurately in developing countries. For instance, in most countries in sub-Saharan Africa vital registration systems record only small proportions of all deaths (Mathers ) and, among the recorded deaths, commonly misreport the causes (Groenewald ). Similarly, information on ART regimens used, retention in care and adherence are difficult to obtain.

For ex post assessment of population health impact, it may thus be substantially easier to measure market or health system effects rather than population health impact. To assess health impact, when only market or health system effects are observed, a translation step is required to predict changes in health outcomes from changes in factors closer to the intervention along the causal pathway to health (Figure 1). For such ‘translations’, estimates of mortality or morbidity effects are necessary; in many cases, such estimates will be available in the published literature.

Choice of ex post assessment design

Limitations of randomized controlled trials

The organizations that fund and implement market interventions also need to agree on how to measure effects. Randomized controlled trials (RCTs) are commonly hailed as the gold standard for evaluation. However, RCTs suffer from a number of limitations, both in general (Deaton 2009) and in the evaluation of ART market interventions in particular. First, in the case of many types of market interventions, RCTs will be impossible to conduct, because the interventions commonly function through central mechanisms benefitting all market actors equally, thus ruling out random assignment of actors to intervention and control groups. For instance, the use of a patent pool that allows companies in developing countries to manufacture an antiretroviral drug generically before patent expiration cannot be randomly assigned to some countries and not to others.

Second, even if market interventions could theoretically be assigned randomly, the very mechanisms by which these interventions function will make it likely that market actors will not accept the assignment and will attempt to circumvent it. As the actors are likely to more vigorously attempt to circumvent assignment the higher the pay-off from circumvention, the deviations from assignment are likely to lead to biased estimates of intervention effects (Deaton 2009). For example, if a market intervention increases the quality of an antiretroviral drug in an intervention country (e.g. by ensuring that counterfeit drugs are screened out), consumers of the drug in a control country may attempt to buy the drug from the intervention country, and they are expected to do so more vigorously the more they will gain from a better-quality product (e.g. the more severely they suffer from HIV disease).

Third, random assignment may not be politically feasible, in particular if an intervention is believed to be effective. For instance, a training programme of provincial Ministry of Health managers that aims to improve their capacity to negotiate ARV prices may be perceived as beneficial before its effectiveness has been established. The provincial managers may thus oppose a randomized trial of the effectiveness of the training programme. Fourth, RCTs have been criticized as tools to evaluate development and global health interventions, because their findings are often unlikely to be transferable to settings other than the trial setting and they do not provide insights into the mechanism leading to the effect (Biglan ; Deaton 2009). Finally, RCTs are usually more expensive than alternative evaluation approaches. Thus, with few exceptions, RCTs are unlikely to be either a feasible or meaningful approach to evaluate the impact of interventions in antiretroviral markets on population health.

Approaches to evaluate intervention impact based on observational data

A range of evaluation methods has been developed in econometrics and the evaluation sciences to infer causality from observational data. Table 1 provides an overview of a number of commonly used methods to evaluate the effect of interventions on outcomes (interrupted time series, regression discontinuity, and difference-in-difference, matching, regression and instrumental-variable approaches). The table also explains limitations of each approach. Table 2 provides examples of how these evaluation methods could be used to assess the population health impact of market interventions.

Table 1

Examples of methods for evaluating the effect of interventions on outcomes

Evaluation method	Description	Limitations
Interrupted time series (ITS)	ITS uses the observed time trajectory of an outcome before an intervention to forecast the future trajectory of the outcome in the absence of the intervention. It is a time series approach estimating the effect δ in the following equation	The evolution over time of the outcome before the intervention may not be a good counterfactual for how the outcome would have evolved had the intervention not taken place.
	where
	is the development of an outcome over time, represented by function , is a variable indicating whether an intervention is in effect in time period t, and is the error term.
Regression discontinuity	Intervention assignment is a discontinuous function of a variable Zi. The effect of the intervention on the outcome Yi can be estimated as δ in the following equation	The extrapolation of data observed below the threshold to values above the threshold may not be a good counterfactual for how the outcome would have evolved had the intervention not taken place.
	where
	Ti is 1 if Zi is above a certain threshold value (and the intervention is thus in effect) and 0 if it is below the threshold value (and the intervention is not in effect) and is the error term.
Difference-in-difference	Outcomes are observed for a group that receives the intervention both in the period before and the period after the intervention, and for a non-randomly assigned control group during the same time periods. The effect of the intervention can then be estimated as in the following equation	Difference-in-difference estimation assumes that the outcome in the intervention and in the control group follow the same trend over time. If the outcome follows a different trend in the two groups, the effect estimate will be biased. It is also necessary for difference-in-difference estimation to have identified a group unaffected by the intervention (which may be difficult if markets are connected) and to have measured the outcome before the time of the intervention in both the intervention and the control group (which requires planning in advance of the intervention implementation).
	where if observation i belongs to the intervention group and zero otherwise, in the period when treatment occurs and zero otherwise, and is the error term.
Matching	Matching balances the distribution of observed variables in those who receive the intervention and those who do not, so that the difference in the observed outcome between the two groups can be attributed solely to the effect of the intervention.	Unobserved variables may affect the outcome, leading to biased effect estimates.
Regression	Regression can be used as an adjustment technique to estimate the effect of an intervention on an outcome δ, controlling for common dependence of the intervention and the outcome on other variables Xi. The effect δ can be estimated in the following equation	The set of control variables may not be sufficient to achieve an unbiased estimate of the effect of the intervention on the outcome (omitted variable bias). If the outcome affects the intervention (reverse causality), the effect size estimate will be biased.
	where is the intervention variable, represents other variables, is a vector of coefficients and is the error term.
Instrumental variables	If a variable exists that predicts the non-random assignment or non-random intensity of an intervention, but does not affect the outcome of interest (except by way of the intervention), the association between this instrumental variable and the intervention variable can be used to estimate the causal effect of the intervention on the outcome.	The hypothesis that a variable does not independently affect the outcome cannot be tested and is often hard to defend. If the instrumental variable is only weakly associated with the intervention variable, effect estimation can be severely biased. It is also necessary for instrumental-variable estimation to have identified a group unaffected by the intervention, which is difficult in interconnected markets.

Table 2

Examples of approaches to evaluate the population health impact of ART market interventions

Market intervention	Mechanism for intended and unintended effects	Example evaluation approach
Pooled procurement of antiretroviral medicines across several antiretroviral treatment programmes	Intended effect: Increased market power on the demand side, reducing the price of the antiretroviral drug and thus the quantity demanded (market effect), which increases coverage (health system effect) and decreases mortality in HIV-infected people.	Difference-in-difference analysis comparing population health outcomes before and after the start of pooled procurement in geographical areas with programmes participating in this intervention to areas without participating programmes.
	Potential unintended effect: Increased market power on the demand side may lead to consolidation of suppliers, increasing the risk of collusion and price increases.
Financial support for the development of a ‘Mother and Baby Pack’ to increase access to prevention of mother-to-child transmission of HIV (PMTCT)	Intended effect: The financial support leads to the development of a locally appropriate ‘Mother and Baby Pack’ and increases PMTCT coverage.	Interrupted time series analysis of HIV-related child mortality.
	Potential unintended effect: The initial success of the pack decreases the incentives for governments to invest in systems for PMTCT, leading to worse population health outcomes, as new and improved PMTCT technologies become available that cannot be delivered as a pack.
Financing ready-to-use therapeutic foods (RUTF) for paediatric ART patients	Intended effects: Financing of RUTF increases the quantity of RUTF consumed (market effect), which in turn improves the overall effect of paediatric ART (health system effect), improving morbidity and mortality in children on ART.	Use of random shocks to the RUTF supply chain (e.g. rainfall impeding transport) as an instrumental variable to estimate the effect of RUTF on health outcomes in children on ART.
	Potential unintended effects: Supply of internationally produced RUTF through donor funding may reduce the incentives to develop locally appropriate products and crowd out local industry producing RUTF.
Training of district health managers in supply chain management in those districts in a country that fall below a certain poverty threshold (i.e. the most vulnerable ones)	Intended effects: Improved supply chain management skills lead to improved delivery of antiretroviral drugs to patients (market effect), increasing the effectiveness of treatment (health system effect) and decreasing morbidity and mortality in patients on ART.	Regression discontinuity analysis of mortality of ART patients as a function of the district poverty level.
	Potential unintended effects: Improved skills of district health managers may improve their value to pharmaceutical companies who increasingly hire them, leaving public health systems depleted of management skills.

Assessment issues

Unintended effects

Market interventions that are successful in achieving a particular intended outcome may have unintended effects that run counter to the goal of improving population health. For instance, interventions that successfully decrease the price of paediatric ARV and consequently improve the health of HIV-infected children may also reduce the incentives for pharmaceutical companies to invest in the development of new and better paediatric ARV, precluding potential future health gains. See Table 2 for further examples of unintended intervention effects.

In both the ex ante and ex post assessment of the population health impact of market interventions, researchers and organizations funding or implementing market interventions need to carefully consider possible unintended consequences. Economic theory may offer guidance in ex ante assessment. For example, based on economic theory we would expect limitations to international patent protections of drugs for certain diseases to lead to price reductions of existing drugs (Waning et al. 2010), but also to reduce investment in the development of new drugs for the same diseases (Ngués 1993; Reichman 2009; Hemphill 2010; Honig and Lalonde 2010). In contrast, interventions that reduce the fixed costs of supplier entry into the market for drugs for a particular disease may lower the price of some drugs, without reducing the economic incentives to invest in the development of new drugs for the same disease.

If, despite potential unintended consequences, a market intervention is implemented, the plans for ex post assessment should include data collection to ensure that analyses can determine the extent to which such effects will indeed have occurred. Many of the unintended consequences may be more difficult to observe directly or to measure, because they are medium- to long-run effects. For example, measuring decreased investment in new treatments as a response to a market intervention generally requires verifiable information from firms or a sufficiently long lag to observe the outcomes of such investment.

Interaction effects

It is possible that different ARV market interventions interact in affecting markets and population health. For instance, interventions reducing ARV prices in certain countries may not improve population health because information on increases in demand following a price reduction is not transmitted successfully through ARV supply chains. It is unlikely that organizations implementing one or a few market interventions can adequately predict or evaluate all possible intervention interactions. Organizations funding market interventions, on the other hand, are well positioned to consider how different market interventions enhance or impede each other's effects. These organizations should thus invest in forecasting interaction effects in ex ante assessment, and ensure that such effects are adequately taken into account in ex post analysis. In these assessments, it should also be considered in how far some of the unintended effects of one intervention can be mitigated by other interventions. For instance, worries about the effect of ARV price-reducing interventions on investment in ARV development may be lessened if innovative financing mechanisms are simultaneously put into place that increase funding flows for ART research and development (Hecht ).

Time horizons of assessment

In many situations, we would expect different effects of market interventions to play out at different times relative to the implementation of the intervention. Price reductions may be observable in the short run following an intervention, while effects on investment decisions may only become observable after a considerable lag time. In assessing intervention impact ex post, researchers thus need to decide on the appropriate time horizon for evaluation. In presenting the results of short-run evaluations, studies need to emphasize that certain longer-run effects may not yet have become observable; and in order to consider evidence on intervention effects as conclusive, in many cases evaluation of long-run outcomes may be necessary.

The fact that some intervention effects only occur in the long run, however, does not eliminate the need for short-term assessments of intervention effects. First, the funding cycles of organizations investing in market interventions may require assessments of interventions results before some long-run effects will have manifested themselves. Second, if an intervention produces undesirable outcomes in the short run, funding or implementing organizations may decide to change or stop it, independent of long-term results. For instance, if an intervention that was expected to increase ART coverage in the short run is found instead to reduce coverage, the organizations funding and implementing the intervention may decide to stop it, independent of any potential long-run impact on population health. Third, while the scope of intervention outcomes that can be observed expands with increasing time horizon of observation, attribution of causal relationships between outcome and intervention becomes increasingly difficult. Thus, short-run assessments of market intervention impacts should usually be required.

Impartiality of assessment

While it seems appropriate to ask organizations applying for funding to implement market interventions to forecast the market effects and population health impacts of the intervention in ex ante assessment, ex post assessment of intervention success may only be considered impartial if the assessor is independent from the implementing organization. Moreover, while the data an assessor uses may only be available within the implementing organization, it will be essential for credible evaluation results that these data can be easily verified, for instance, by independent auditors. In planning interventions in ARV markets, funding agencies should thus discuss with implementing organizations which data should be collected for evaluation and how the data should be collected. For instance, for data on ART coverage anonymized extractions of clinic records may be required; and data on ART supply chain interventions may need to be collected and verified with independent audits in the organizations that receive the products transported through the supply chain.

Costs of assessment

Assessment, both ex ante and ex post, is costly. Data on intervention implementation and outcomes need to be collected, entered, checked and analysed. The results of the analyses need to be interpreted and presented. We would expect the costs of ex ante assessment to decrease with the level of previous knowledge about the population health impact of an intervention and to increase with the complexity of models needed to adequately forecast intervention impact. The costs of ex post assessments will depend on the:

Extent to which data on intervention impact are collected through routine systems (such as market reporting mechanisms or vital registration systems) or need to be collected specifically for the assessment;

Assessment design (such as controlled trials vs observations studies);

Time horizon of the assessment;

Assessment frequency.

Funders of market interventions need to decide how much money they are willing to invest in intervention assessment. While these costs would normally constitute only a small proportion of total intervention costs, in certain cases very costly assessments may be justified. High costs of an ex post assessment (for instance, for a RCT) may be justified if a market intervention has never been evaluated in terms of its efficacy in improving population health. In contrast, once intervention efficacy has been established in one particular case, it may no longer be necessary to invest in costly efficacy evaluation; rather it may suffice to evaluate the effect of the intervention on health systems, such as changes in coverage with ART. Funders should also consider the potential synergies in pooling assessment resources across implementing organizations. Such synergies could arise in the joint use of analysts and in pooling data on the same type of intervention across organizations in order to increase the power of studies in detecting significant effects on markets and population health.

Discussion

International organizations intervening in markets to improve population health need to assess the potential impact of their actions before implementation, monitor impact during implementation and document the effects after implementation. Assessments are needed at all stages to select the most promising interventions, continue effective interventions, discontinue ineffective ones and demonstrate to stakeholders and the public that interventions fulfill their function. Market interventions pose a number of specific evaluation challenges. For one, they are relatively far removed from their final goals, working their way to health impact through effects on markets and health systems. Moreover, markets are dynamic in nature, and numerous factors other than the intervention affect market and health outcomes. Finally, in most cases it will be impossible to evaluate market interventions in an RCT, because randomization will not be feasible for technical, political or ethical reasons.

The assessment of health impact ex ante should always include a due-diligence evaluation of the safety and efficacy of a product whose market will be affected by the intervention, and a description of the hypothesized mechanisms via which the intervention will impact population health. Quantitative assessment of intervention impact can only be obtained through predictive modelling. For interventions that have precedents that were rigorously evaluated, it may be appropriate to request from applicants quantitative predictions of population health impact. For interventions that have never been implemented, on the other hand, quantitative prediction of population health impact may not be possible with any useful degree of precision.

In this context, it is important to note that while we have maintained the distinction between ex ante assessment (i.e. before a market intervention has been implemented) and ex post assessment (i.e. after the intervention has been implemented) throughout the paper, in reality this distinction will often not be clear-cut. A first reason is that the relationship between the decision to implement an intervention, the actual implementation and assessment of intervention outcomes is not linear but cyclical, as ex post assessment of an intervention will feed into ex ante assessment to support the decision on whether an intervention should be continued in its current form, continued with adjustments, or discontinued. A second reason is that even if one particular intervention has never been implemented, making any ex post assessment impossible, related interventions may have been implemented, whose ex post assessment could be used in the ex ante assessment of the novel intervention. Related interventions comprise interventions that differ to some extent in content (e.g. a different approach to train health workers in supply chain management), context (e.g. implementation in a different country), or scale (e.g. a previous pilot study) from the novel intervention. A third reason is that two interventions can never be the precisely the same if they are implemented during different time periods, because the social, economic and political contexts change over time. As such, evidence from past interventions can never be thought to perfectly predict future intervention success and in ex ante assessments we will always need to consider the applicability of ex post assessments, even if the past interventions do not differ from the future ones in content, geographic reach, or scale.

These caveats on the distinction between ex ante and ex post intervention are important to keep in mind, when considering our recommendations below. Furthermore, while we focus our recommendations on the evaluation of market interventions, some may be applicable to other types of interventions, such as interventions in health systems and services.

Recommendation 1:

The decision makers selecting market interventions for funding should routinely require applicants to provide: For some types of interventions, it may be appropriate to demand that applicants provide a quantitative prediction of population health impact.

Descriptions of hypothesized causal mechanisms leading from intervention to health impact;

Assessment of the quality of the evidence that the intervention will achieve its intended effect;

Assessment of the risk that an intervention will lead to unintended consequences.

For ex post assessment of population health impact, stakeholders need to take several decisions. First, they need to decide at which point along the causal chain from intervention to health impact to observe outcomes. Second, they need to decide which evaluation approach to use in analysing the observed data. In general, it will be easier to measure market and health system variables than health outcomes, because the former are commonly collected routinely and are available in central databases, while the latter will often require additional data collection. In addition, it will often be easier to attribute changes in market and health system outcomes to an intervention compared with changes in health outcomes, because the former will depend less on other intervening variables and will react more immediately to the intervention than the latter. Measuring only market or health system outcomes, however, implies that causal analyses will be confined to the pathway from intervention to these effects, whereas the impact on health needs to be estimated in predictive models. Those models are necessarily based on a range of uncertain assumptions, including estimates of intervention health effects. Model-derived results may thus be less convincing to stakeholders and more difficult to publish than effect estimates based directly on observed population health outcomes.

Recommendation 2:

Applicants should be required to demonstrate how their intervention could be assessed during and after the intervention. Given the trade-offs involved in the decision on which outcomes to measure and which to model, it is important that all stakeholders agree on an evaluation design before the start of the intervention.

Different evaluation approaches have different data needs. For instance, interrupted time series analysis of the effect of market interventions on health systems outcomes may be possible with aggregated data (e.g. by district) and with limited information on variables other than the outcome and the timing of the intervention. In contrast, estimation of effect size based on matching will require data at the level of the unit at which the intervention is assigned. Moreover, matching estimation will in general perform better when more variables are available on which to match. Difference-in-difference estimation necessarily requires measurement of baseline values before the start of the intervention, while regression discontinuity analysis can be conducted when only data from periods after intervention start are available. Instrumental variable estimation will commonly require collection of data on variables that are not routinely available in health systems and ART programmes, such as data on weather conditions, geographical distance between market actors, or a country's cultural history. Data that do not arise in the organizations implementing an intervention should be collected by independent entities, such as consultancy companies hired by funders or academic research teams. Data that can only be collected within the organizations implementing an intervention need to be independently verified to ensure that the data are not distorted to meet criteria of intervention success.

Recommendation 3:

Given the varying data collection needs of different evaluation approaches and the need to collect baseline data in the case of some approaches, it is crucial that monitoring and evaluation are built into the intervention from the beginning. To ensure that assessment results are credible, data used in the assessment need to be independently verifiable and the assessment itself should be conducted by individuals or organizations that are independent from the organizations funding or implementing the intervention.

It is important to distinguish between monitoring and ex post evaluation. In this article, we discuss approaches to evaluate the impact of ART market interventions to establish as far as possible the causal effect of the intervention on population health outcomes. Such evaluation serves the purpose of improving our understanding of the functioning of market interventions and legitimizes future decisions on the continuation of interventions. Monitoring, on the other hand, serves managerial purposes, ensuring that the intervention is implemented as intended, that intervention resources are used correctly and without waste, and that unintended intervention effects are quickly detected. Unlike evaluation, which is undertaken at certain discrete time points after the start of an intervention, monitoring is an ongoing exercise, which needs to continue even if the effectiveness of an intervention has been proven.

Recommendation 4:

Stakeholders need to decide when to evaluate the effect of an ART market intervention. Separately from an ex post assessment plan, they need to agree on a monitoring approach—ex post assessment cannot fulfill the managerial functions of monitoring.

Evaluation is a costly exercise. It may require specific data collection, data input and quality control, as well as analysts’ time to compute estimates of intervention effect size and significance. On the one hand, it may thus seem wasteful to repeat evaluating an intervention, after its causal effect on population health has been firmly established. However, it is possible that effectiveness is a function of intervention stage (e.g. if patients utilizing a new ART product immediately after market introduction differ systematically from patients utilizing the product at later points in time), intervention scale (e.g. if intervention resources are used more efficiently the more people the intervention reaches), or secular trends (e.g. if the structure of companies producing ART products changes over time).

While both ex ante and ex post assessments of the population health impact of market interventions are important, the costs of assessment should be reasonable in comparison to the costs of intervention implementation. Funders of market interventions should devise strategic long-term plans for intervention assessment including explicit considerations of assessment costs. It may be possible to reduce the costs of assessment by pooling assessment resources across organizations.

Recommendation 5:

Decision makers need to assess the added value of repeating ex post assessments at later stages of an intervention. The decision will likely depend on theoretical considerations of how far intervention effects could differ when assessed at a different point in time.

Recommendation 6:

The costs of assessment need to be appropriate.

Organizations implementing individual interventions may not be aware of all potential unintended effects of an intervention, and they may not have knowledge of interventions implemented by other organizations. Funders of market interventions should thus invest in background documents and guidelines on unintended effects of interventions. Such documents can serve as a starting point for evaluating these types of effects in assessments of the population health impact of market interventions. Moreover, they should systematically collect information on the activities of other funders to avoid duplication, facilitate collaboration and allow evaluation of interventions across funding institutions.

Recommendation 7:

Assessors need to investigate unintended effects of interventions and interaction effects between different interventions. Such assessments should be systematically supported by funders of market interventions.

In sum, both ex ante and ex post assessments of the population health impact of intervening in ART markets require careful planning. Stakeholders need to decide which impacts to assess, what data to collect, which analytical approaches to use, and when and how often to evaluate impact. These decisions will depend on the type of intervention, stakeholders’ expectations of the strength of the evidence and the impartiality of the assessor, data availability, limitations of existing analytic methods and the cost of assessment.

Funding

TB was supported by Grant 1R01-HD058482-01 from the National Institute of Child Health and Human Development (NICHD), NIH. TB, MK and BW received financial support for this research project from UNITAID. The funding agencies had no role in design of the study, or in the preparation, review or approval of the manuscript.

Conflict of interest

We declare that we have no competing interests.