Methodological Challenges to Economic Evaluations of Vaccines: Is a Common Approach Still Possible?

Economic evaluation of vaccination is a key tool to inform effective spending on vaccines. However, many evaluations have been criticised for failing to capture features of vaccines which are relevant to decision makers. These include broader societal benefits (such as improved educational achievement, economic growth and political stability), reduced health disparities, medical innovation, reduced hospital beds pressures, greater peace of mind and synergies in economic benefits with non-vaccine interventions. Also, the fiscal implications of vaccination programmes are not always made explicit. Alternative methodological frameworks have been proposed to better capture these benefits. However, any broadening of the methodology for economic evaluation must also involve evaluations of non-vaccine interventions, and hence may not always benefit vaccines given a fixed health-care budget. The scope of an economic evaluation must consider the budget from which vaccines are funded, and the decision-maker's stated aims for that spending to achieve.

Key Points for Decision Makers

Introduction

Vaccines are undoubtedly one of the global health success stories of the past century, credited with saving 700,000 lives in the USA in 1994–2013 [1], and 7 million lives in the poorest countries in 2000–2014 [2]. Spending on vaccination has increased dramatically. This has been driven both by expansion in vaccine coverage worldwide, as well as the introduction of new (but currently expensive) vaccines such as human papillomavirus, pneumococcal conjugate and meningococcal B vaccines. The cost to fully immunise a child according to the WHO’s recommended schedule was estimated to have risen from US$0.67 in 2001 to US$45.59 in 2014 [3]. Funding vaccination in 94 of the world’s poorest countries during the “Decade of Vaccines” (2011–2020) is projected to cost over US$50 billion, but to save 25 million lives [4].

With this increased funding has come the need to ensure that money is well spent. Economic evaluation of vaccines has emerged as a key tool to inform effective spending on vaccination. The number of economic evaluations of vaccines has risen sharply over the last 20 years [5]. Both national immunisation technical advisory groups [6] and major global health funders [7, 8] consider economic evidence before recommending a new vaccine. However, the use of economic evaluation has not been without controversy. Initial recommendations against introducing human papillomavirus vaccination in Australia [9] and meningococcal B vaccination in the UK [10] elicited criticism and were eventually reversed following re-evaluations of the underlying economic models (accompanied by negotiations with manufacturers which may have achieved vaccine price reductions). In contrast, in Thailand human papillomavirus vaccination has not been funded to this day following economic analyses suggesting that it was not cost effective at current high prices. Instead the cervical screening programme was scaled up as it was found to be a more cost-effective cancer prevention strategy [11].

A principal critique is that most economic evaluations do not capture salient features of vaccines which are relevant to decision makers. As a result, the full benefits of vaccination are not adequately incorporated. This paper discusses the challenges to standard economic evaluation of vaccines that have been raised, the extent to which proposed methodological changes may alter conclusions about current vaccines and some suggested principles for responding to these challenges.

Challenges to “Standard” Economic Evaluation Methodology

The most common approach to economic evaluations of vaccines (and health-care interventions more generally) is cost-utility analysis. This aims to inform allocation of a fixed budget in order to maximise health, measured in terms of generic population health measures such as quality-adjusted life years (QALYs) or disability-adjusted life years (DALYs). This is done by comparing the incremental cost effectiveness of a new intervention (such as a vaccination programme) to a fixed threshold. The threshold in principle represents the health foregone by reductions in the spending on other interventions in order to fund the new intervention.

Cost-utility analysis methods are described in guidelines from the WHO [12] and the US Panel on Cost-Effectiveness in Health and Medicine [13]. They are the preferred form of economic evaluation in the majority of national guidelines for pharmacoeconomic evaluations, which state a preference for this issue [14]. The perspective of such analyses may be limited to the health-care sector (as advocated by the UK’s National Institute for Health and Care Excellence [15]), or can extend to all costs and effects borne by society (as advocated by the WHO [12] and the US Panel on Cost-Effectiveness in Health and Medicine [13]).

A number of challenges to this “standard” approach to economic evaluation have been made with regards to vaccination:

Type of Analysis

Laxminarayan and co-workers suggest that benefit-cost analysis, using value of statistical life measures to monetise health gains, is a more comprehensive framework than cost-utility analysis, as it also captures the non-health benefits of vaccines [16]. According to their proposal, the value of a life could be estimated by examining either revealed preferences (studies of compensatory payments for altered risk of death such as wage-risk trade-offs, i.e. premiums paid for work in dangerous occupations) or stated preferences (such as willingness-to-pay studies for specific vaccines). Both these methods should in principle capture the value people place on both health and non-health benefits of increased life expectancy. This methodology has been used to value the benefit of vaccination in developing countries in the “Decade of Vaccines” [17].

Fiscal Impact

Fiscal implications of vaccination programmes are not always clear in traditional cost-effectiveness analyses because costs are aggregated over long time horizons, subject to discounting and presented in economic rather than financial terms. Alternative approaches such as budget impact and return on investment analyses can facilitate accurate budgeting and directly address constraints of health-care budget holders [18]. Explicit budget optimization has been suggested as an alternative to cost-effectiveness analysis particularly in countries where there is little fiscal space to fund new interventions [19, 20].

Distribution of Health

It has been suggested that the role of economic evaluation of vaccines should not simply be to maximise health, but to ensure equitable distribution of health (and related benefits such as protection from catastrophic health expenditure associated with a serious infectious disease) [21]. These dimensions have received greater attention following the United Nations General Assembly’s adoption of Universal Health Coverage as a target; this seeks to ensure that everyone can obtain the health services they need without having to suffer financial hardship [22].

Broader Epidemiological Outcomes

Many vaccines have “herd” or indirect benefits on non-vaccinated individuals by preventing onward transmission in vaccinated people [23, 24]. Vaccination may also reduce the need to prescribe antimicrobial drugs and hence reduce antimicrobial resistance. Also, vaccines often prevent a range of secondary outcomes besides the primary outcome being targeted. For example, human papillomavirus vaccines prevent a range of non-cervical cancers and warts in both men and women [25].

Broader Social Welfare Benefits

It has been suggested that vaccines bring about “wider societal benefits” [26] by preventing disease, such as enabling people (and their families) to contribute more resources to society. Beyond productivity losses, there may be an even broader set of benefits such as improved educational outcomes, reduced birth rates (because families may decide to have fewer children when there is a greater chance of each child surviving to adulthood), increased household savings, macroeconomic growth and political stability [27-29].

Supporting Innovation

Vaccine manufacturers have suggested that high-income countries should incentivise the risky investment needed to fund research in particularly innovative vaccines, by allowing producers to appropriate most of the economic surplus due to vaccine development prior to patent expiry [10, 28].

Societal Preferences

People may place greater value on QALYs gained from preventing an episode of severe or life-threatening illness rather than an equivalent number of QALYs gained from preventing many episodes of mild illness [10].

Health Service Pressures

Infections like rotavirus and influenza may cause seasonal outbreaks during winter in temperate countries when health-care services are already stretched [30]. The true opportunity cost of health-care utilisation during these times may hence be underestimated.

More Favourable Methodological Assumptions

Vaccines have features not shared by most other health interventions, such as potentially long time lags between the time of intervention and time of its effects (disease prevention) [24]. Hence there have been suggestions that current methodological guidelines around issues such as discounting [31] disadvantage vaccines. Vaccines also provide peace of mind to recipients and their caregivers who are afraid of disease, which is not captured through health measures such as QALYs and DALYs but may be captured using techniques such as willingness-to-pay [32, 33].

Integrated Analyses with Other Interventions

Many vaccines should arguably not be evaluated as single interventions, because they have knock-on effects on the cost effectiveness of other (non-vaccine) interventions. For instance, human papillomavirus vaccination can reduce the number of cervical screens a woman needs in her lifetime and hence bring considerable cost savings [34]. Malaria vaccines have been suggested to synergise with bed nets to produce larger health gains than the sum total of benefits from each intervention on its own [35].

Would Changing Current Methodology Affect Economic Evaluations of Vaccines?

Table 1 shows commonly used vaccines, the results of recent reviews of relevant economic evaluations of vaccines, and proposed changes to economic evaluation methodology that are particularly relevant to the vaccines. Most vaccines are already clearly seen to be good value for money using standard cost-utility analysis methodology. The traditional Expanded Programme on Immunization (EPI) vaccines (measles, diphtheria, tetanus, pertussis, polio and tuberculosis) are clearly cost effective in every country and may even be cost-saving even from a health systems perspective [36]. More recent vaccines such as rotavirus [37], pneumococcal conjugate [38] and human papillomavirus [39] vaccines are also clearly cost effective at the prices at which they are offered to Gavi, the Vaccine Alliance, to middle-income countries that have succeeded in negotiating competitive prices (such as members of the Pan-American Health Organization [40]), and to high-income countries with large public sector purchasers that negotiate prices through competitive tenders.

Table 1

Cost effectiveness of different vaccines (based on recent reviews), and additional benefits of those vaccines not usually included in economic evaluations

Vaccine (target group)	Cost effectiveness using standard methodsa			Proposed benefits not currently included
	High-income countries	Low- and middle-income countries	Reasons if not cost effective
Traditional vaccines (measles, diphtheria, pertussis, tetanus, polio, tuberculosis) (infant)	Clearly cost effective, probably cost saving. Cost/DALY of US$7–438 (2001 values) excluding cost savings due to reduced health care use [36]		No studies found suggesting the traditional vaccine package as a whole is not cost-effective	Non-specific mortality prevention [53], improved educational achievement [54]
Haemophilus influenzae type b (infant)	Cost effective or cost saving in most settings in a systematic review. Not cost effective in a few studies which assumed low disease incidence and/or treatment costs [55]		Uncertainty about vaccine prices and disease incidence [55]	Productivity gains due to reduced sequelae and changed household decisions [56]
Human papillomavirus (young adolescent females)	Cost effective in all studies found in a systematic review [46]	Cost effective in most countries in a global modelling study [39]; cost effective in all but one study (for Thailand) found in a systematic review [57]	A few studies suggest vaccination not cost effective if prices are very high [11]	Paid and unpaid work by patients, changes to household behaviour [25], reduced frequency of cervical screening [34]
Meningococcal Group B (infant)	Cost effective in the UK if the vaccine price is low and assumptions are favourable to the vaccine [10]	Not examined	High vaccine price, uncertainty about protection [10]	Preference for preventing severe over mild diseases [10]
Pneumococcal conjugate (infant)	Cost effective, especially for higher-valency vaccines, if societal costs and/or herd protection is included [45, 58, 59]	Cost effective if societal costs are included [38]	Serotype replacement (for low-valency vaccines) [45]	(Not specifically studied)
Rotavirus (infant)	Cost-effective at low prices [60]	Cost effective [37, 60]	Initial high price of the vaccine in high-income countries [60]	Burden on caregivers [61], mild episodes not leading to health-care attendance [61], reduced hospital bed pressure [30], household financial risk protection [21, 62]
Varicella (early childhood)	Cost effective or cost saving if either societal perspective is taken or without hypothesised increase in shingles [43]	Not examined	Hypothesised increase in shingles (herpes zoster) as a result of decreased varicella exposure [43]	Sick-leave compensation [43]

aCost-effectiveness conclusions are based on views surveying literature across a range of settings and methodological assumptions. A vaccine is considered cost effective if its incremental cost-effectiveness ratio is below a country’s gross domestic product (GDP) per capita, unless an alternative threshold is suggested

Vaccination programmes for which cost-utility analyses do not clearly support vaccine introduction generally fall into several categories: (1) some vaccines targeted at particular risk groups (male human papillomavirus [41] and maternal pertussis vaccination [42]), (2) some vaccines for which the long-term ecological consequences are uncertain and may even be negative (varicella [43] and 7-valent pneumococcal conjugate [44] vaccination) and (3) some vaccines against diseases which are extremely rare, even though they may have severe outcomes (meningococcal group B [10] vaccination). In these situations, the outcomes of economic evaluations often depend on the exact assumptions and parameter values used; incorporating additional benefits may make vaccination appear more likely to be cost effective.

Considering the Decision Maker’s Objectives

Many of the proposed broader outcomes of vaccines, particularly epidemiological outcomes such as indirect protection and protection against secondary endpoints, are already routinely incorporated into existing economic evaluations, particularly those set in high-income countries [10, 43, 45, 46]. Other outcomes, such as those relating to broader societal benefits, innovation and hospital bed pressures, are less commonly incorporated, even though there is experimental and observational evidence linking some vaccines to improved educational attainment and reduced health disparities [29]. Evidence supporting extrapolation of these benefits to even wider effects such as household economic behaviour, macroeconomic growth or national stability is weaker. However, there appears to be no a priori reason for excluding vaccine benefits which are grounded in evidence-based causal pathways.

However, economic evaluation methodology needs to take into account the decision maker’s needs. The ultimate aim of an evaluation is to increase the probability of a decision that is aligned with the decision maker’s stated goals about what to achieve with a given budget [47]. The problem for vaccines is that the “decision maker” is often not obvious, because there may be a separation between the funder of a vaccination programme (such as Gavi), the consumer (country governments who implement a vaccination programme) and the beneficiary (the population of the country, including people who are not vaccinated but who benefit through herd protection).

In all situations, economic evaluation of vaccines must be cognisant of the budget from which vaccines are funded. These evaluations are conducted for a range of reasons: advocacy (making the case for securing internal and/or external financial resources for a vaccine) [27], selecting the right mix of interventions to optimise the health-care budget [19], informing tender negotiations between purchasers and vaccine manufacturers [48], and ex-post appraising the value for money of past decisions [49]. Regardless of the purpose of the analysis, the underlying methodology and assumptions should arguably be standardised as long as vaccines are being funded out of the same budget and with the same decision maker. This is to ensure the most efficient outcomes are achieved from the decision to purchase the vaccine right to final price negotiations, in line with the stated goal of the decision maker.

If vaccines are funded from a health-care budget, then the optimal use of that budget is ultimately a question of productive rather than allocative efficiency, i.e. using the budget so as to maximise production of the outcome(s) desired by the budget holder [50]. For example, in the UK, health (measured in terms of QALYs) is the maximand for most health economic evaluations following the National Institute for Health and Care Excellence’s reference case [15], in part because there is no obvious way that savings to other government departments or to the private sector could accrue back to the health budget. In such evaluations, the shadow price of health (represented by the cost-effectiveness threshold) should arguably be derived from the budgetary constraint, rather than based on human capital arguments about the intrinsic economic value of a healthy life year, or even individual willingness-to-pay measures such as the value of statistical life. In settings where the shadow price of health is unknown, and the number of spending options are relatively limited, budget optimisation provides an alternative analytical framework. This is mathematically equivalent to cost-utility analysis when applied over the entire budget, but it could also be used to optimise selection of a subset of potential interventions in a particular disease area [19, 20].

When vaccines are funded from fixed health-care budgets, increased investment in vaccines normally involves disinvestment from other items in the budget. Hence, any change to the methodology used for evaluating vaccines needs to involve broadening of the standard “reference case” used for all economic evaluations. It is important to avoid the appearance of changing methodology purely to support the case of a particular vaccine. This kind of “special pleading” ultimately benefits nobody, because it reduces the confidence that decision makers have in the results of economic evaluations when they do not see evidence that evaluations have discriminatory power in identifying poor investments as well as good ones. As Beutels and co-workers said in their commentary on funding vaccines, “we do not plead for a special case, but for a level playing field” with other interventions [24], by ensuring that unique features of vaccines are appropriately captured in current frameworks rather than by using separate rules for vaccines.

An overall change in economic evaluation methodology may not always be to the advantage of vaccines if the health budget is fixed. For instance, childhood vaccination may indeed have broader benefits outside health in the strict sense, in terms of child development, household financial security and economic development. However, the evidence linking other childhood interventions such as malaria prevention, deworming or sanitation with some of these benefits may be stronger than that for vaccines [29]. Similarly, increased peace of mind from protection against infectious diseases has been mentioned as a benefit of vaccination. However, non-vaccine health-care spending arguably produces comparable or greater peace of mind benefits. Examples of this are being reassured about shorter waiting times for hospital admission (following investment in hospital bed capacity) or availability of end-of-life care (following investment in such care).

Alternatively, the decision maker may use the health-care budget to optimise non-health benefits as well as health, as is the case with most pharmacoeconomic guidelines [14]. The scope of these broader benefits is set by the decision maker, but if a societal perspective is taken then there is no reason to exclude any benefit that improves welfare [13]. Two options for this are a cost-utility analysis with a societal perspective, and a cost-consequences analysis [51]. A cost-consequences analysis allows presentation of outcomes that are difficult to monetise, and recognises that monetary benefits outside the health-care sector should not be treated in the same way as cost savings to the health-care provider. An example of this is the recently introduced extended cost-effectiveness analysis, which presents indicators of equity as well as efficiency [21]. However, we must be aware that taking a wider perspective under a fixed health-care budget will actually reduce the total health improvement, because there will be situations in which an intervention will be displaced by one that is less efficient in improving health, when the latter intervention brings non-health benefits of greater value.

If vaccination is being funded from a wider budget that includes non-health spending, then a benefit-cost or return on investment framework encompassing all the broader consumption benefits of vaccines regardless of sector would appear to ensure a level playing field, since this is the methodology typically used to value non-health investments. Still, there are practical considerations. In most countries, the allocation of spending between different ministerial budgets (such as health and education) is a high-level political decision. Human capital arguments about the value of increased investment in health care generally or even vaccination specifically (such as those made by the recent Lancet Commission on Investing in Health [52]) may have a role in informing such allocations. However, there are unlikely to be technical resources or indeed even the political mandate for microeconomic evaluations of individual programmes (such as the introduction of specific vaccines) to directly inform high-level allocations of the entire national budget.

Lastly, when a vaccination programme is being funded in part or whole by an external government or organisation (such as Gavi), the objective of spending is for the global good rather than the social welfare of any particular population. From a global perspective, there appears to be no justification for placing a different value on a life saved in (for example) the UK and in Mali. However, for such indifference to be translated into the global distribution of health-care spending would require rich countries to accept much larger transfers from health-care spending in their countries to poorer countries than we are likely to see. Hence a pragmatic way forward given limited aid budgets is to again consider the productive efficiency question of maximising certain welfare aims within the budget of donor countries or organisations. For a completely altruistic donor, these may be aligned with the aims of local health-care providers or the local population as a whole. However, often vaccine funding is provided in pursuit of externalities such as disease eradication, reduced antimicrobial resistance, reduced inter-country health inequalities and greater political stability (and hence greater global security and trade). If such outcomes are desired and can be quantified, then they are appropriate to include in economic evaluations in this context.

Table 2 summarises the appropriate analysis to use depending on the objective and outcomes to include in different situations.

Table 2

Appropriate analysis to use and outcomes to include in economic evaluations of vaccines based on the budget holder and its priorities. “Welfare” refers to all utility that individuals derive from consumption, including utility from improved health

Budget	Maximand		Analysis	Broader economic outcomes included?
Health care	Health	→	Cost-utility (health care perspective) or budget optimisation	No
Health care	Welfare	→	Cost-utility (societal perspective) or cost-consequences	Yes (depending on decision maker)
Government	Welfare	→	Benefit-cost	Yes (depending on decision maker)
External donor	Health	→	Cost-utility (health care perspective) or budget optimisation	No
External donor	Health + externalities benefitting the global community	→	Depends on decision maker	Yes (depending on decision maker)

Conclusion

Current economic evaluations exclude many of the wider benefits of vaccines. However, this may be by design of the economic evaluation methodology being used. Any changes to the methodology must apply equally to all health interventions, which may not always favour greater funding for a particular vaccine. The aim of any modification should ultimately be to ensure equal treatment for vaccines rather than special preference.

Economic evaluations of vaccines usually fail to capture all the societal benefits of vaccination.

Broadening the benefits considered must also involve evaluations of non-vaccine interventions and hence may not always benefit vaccines given a fixed health-care budget.

The scope of an evaluation must consider the budget from which vaccines are funded, and the decision-maker’s stated aims for that spending to achieve.