Avoiding Trouble Ahead: Lessons Learned and Suggestions for Economic Evaluations of COVID-19 Vaccines.

With vaccines for coronavirus disease 2019 (COVID-19) being introduced in countries across the world, policy makers are facing many practical considerations about how best to implement a vaccination programme. The supply of vaccines is insufficient for the global population, so decisions must be made as to which groups are prioritised for any vaccination and when. Furthermore, the aims of vaccination programmes will differ between countries, with some prioritising economic benefits that could stem from the relaxation of non-pharmaceutical interventions and others seeking simply to reduce the number of COVID-19 cases or deaths. This paper aims to share the experiences and lessons learned from conducting economic evaluations in Singapore and Thailand on hypothetical COVID-19 vaccines to provide a basis for other countries to develop their own contextualised economic evaluations, with particular focus on the key uncertainties, technical challenges, and characteristics that modellers should consider in partnership with key stakeholders. Which vaccines, vaccination strategies, and policy responses are most economically beneficial remains uncertain. It is therefore important for all governments to conduct their own analyses to inform local policy responses to COVID-19, including the implementation of COVID-19 vaccines in both the short and the long run. It is essential that such studies are designed, and ideally conducted, before vaccines are introduced so that policy decisions and implementation procedures are not delayed.

Key Points for Decision Makers

Rationale

It is no secret that the supply of coronavirus disease 2019 (COVID-19) vaccines is insufficient for the global population. The World Health Organization (WHO) target production capacity of COVID-19 vaccines is 2 billion doses in 2021 [1]. Depending on whether immunisation requires one or two doses, this target level of production could result in 1–2 billion immunised individuals, up to 25% of the global population [2]. Policy makers should consider the aim of any COVID-19 vaccination programme before implementation. Countries face different baseline situations and political priorities, which highlights the need for local analyses to ensure policy is informed by context-specific evidence [3]. For example, some governments must decide how many people (and what types) should be vaccinated so that future large outbreaks of COVID-19 can be avoided and non-pharmaceutical interventions (NPIs), such as social distancing measures and international travel bans, can be eased to allow the economy to return to pre-COVID-19 standard practices. As an example of differing priorities, some countries may wish to use COVID-19 vaccines primarily to decrease the number of deaths, whereas other countries may wish to maximise economic productivity. These differing objectives could result in the need to vaccinate different subpopulations (e.g. vulnerable and elderly populations vs. working age populations). Thus, governments need to transparently and fairly consider which population groups should be prioritised for vaccination and for subsequent rounds of vaccination.

Many countries have already begun their vaccination campaigns, with a variety of different vaccines in use and more in development [4]. Most countries have thus far prioritised healthcare workers and elderly groups for vaccination, though others (such as Indonesia, Thailand, and Singapore) have prioritised other subpopulations [5], and Tanzania and Madagascar have rejected vaccines altogether [6, 7]. Younger groups in Indonesia and migrant workers in Thailand and Singapore have been identified as key drivers of transmission and have been prioritised for vaccination [8, 9]. As of March 2021, 30 COVID-19 vaccines were in human clinical studies and 86 vaccines were at the preclinical stage, with seven recently approved for early or limited use [10]. The current vaccines differ substantially, both in their effects and in their cold-storage requirements, and it is likely that vaccines in development will differ further [11]. As such, governments face ongoing decisions about the most appropriate vaccine(s), with respect to their country’s profile, the outbreak situation, and vaccine prices. Also, high-income countries such as Singapore, and upper-middle-income countries such as Thailand, will need to pay the full cost of COVID-19 vaccines for their own population. Countries should be well-prepared for short- and long-term procurement procedures, including pricing negotiations, to ensure that the significant investment in COVID-19 vaccines justifies the opportunity costs from displaced investment in the health sector or across other public programmes.

This paper aims to inform future economic evaluations of COVID-19 vaccines based on the experiences of Thai and Singaporean teams currently conducting economic evaluations of hypothetical COVID-19 vaccines. It also highlights novel methodological challenges in conducting an economic evaluation of these vaccines, because the COVID-19 pandemic substantially affects both healthcare systems and the wider economy at local, regional, and global levels. The methodological discussion points raised in this article can provide insights for technical teams in other countries, helping them to initiate timely economic assessments to inform contextualised COVID-19 vaccination policies.

In Thailand, the project team planned at least two rounds of stakeholder consultation, with participants including policy makers from the Public Health Emergency Operations Centres (EOCs), the Thai Ministry of Public Health, the National Health Security Office, the National Vaccine Institute, the Centre of Excellence in Vaccine Research and Development, the Bank of Thailand, the Ministry of Finance, the Office of the Prime Minister, and various academics [41, 42]. The first meeting was conducted to refine the policy and research questions to investigate, define the study scope, and agree upon the methodological approach, assumptions used, and study timelines. The second meeting will be conducted towards the end of the project to verify study results and clarify policy recommendations. These stakeholder meetings are also important to ensure the quality and transparency of the study. In Singapore, focus group discussions were conducted with stakeholders, including business owners, business consultants, business senior management, healthcare professionals, and representatives of the general population. At the time of submission of this article, three meetings had been conducted and several more were planned.

Why Is an Economic Evaluation Necessary, and Is It Feasible?

Economic evaluations of COVID-19 vaccines can answer a broader set of questions beyond simply comparing the cost effectiveness of using vaccines to a cost-effectiveness threshold. For example, while considering a country’s COVID-19 prevalence, demography, economic, and health infrastructure, they can help to clarify

the most desirable COVID-19 vaccine characteristics for prevention and control from an economic perspective (e.g. single dose/double dose, alternative types of vaccine efficacy, storage requirements);

which subpopulation groups (stratified by age, occupation, geographical location, or other characteristics) should be prioritised, and in what order, given limited vaccine supplies;

the best value for money combination of NPIs and COVID-19 vaccines for controlling COVID-19 in a particular setting;

the economically justifiable price for a vaccine given its supply chain and delivery costs and booster dose requirements; and

whether distributing single-dose vaccinations to a larger population and delaying delivery of a secondary dose is preferable to providing two doses to a smaller population more quickly.

Furthermore, even among cost-saving interventions, which COVID-19 vaccines may be expected to be in many settings, economic evaluations can still be used to understand which of the available interventions is the most cost effective. Given the variety of vaccines currently available and the scale of investment required, set against the importance of other health conditions (such as non-communicable diseases), selecting the most cost-effective intervention remains an important issue.

The feasibility of developing a robust economic evaluation of COVID-19 vaccines is in question for several reasons. First, there are many uncertainties about the biology and natural history of COVID-19, and these present unique challenges for conducting economic evaluations for COVID-19 vaccines. As an example, there is no clear explanation for the variety of case fatality rates and virus severity in different settings (although age and ethnicity have been suggested as potential factors [12, 13]), and the duration of natural immunity occurring from infection and the impact of new variants is not yet well-understood [14]. These examples will affect the perceived value of COVID-19 vaccines to policy makers in different settings.

Second, considerable uncertainty remains regarding the efficacy, administrative requirements, cost, supply availability, supply chain requirements, and delivery platforms, among others, of the currently available and future vaccines.

Third, political and economic pressures mean that rejecting a COVID-19 vaccine is not an option for many governments. Therefore, unlike traditional economic evaluations, which may inform the acceptance or rejection of a vaccine, an economic evaluation of COVID-19 vaccines will need to address different policy questions, described further below. Furthermore, Thailand’s experiences show that using cost-effectiveness evidence to inform price negotiations with manufacturers is more effective than negotiating without the support of such evidence [15].

Fourth, vaccine availability is sporadic in many countries, and this is critical because the local prevalence of the disease can change dramatically in a matter of weeks. This means that the vaccine needs of countries may also change by the time supplies arrive; it is possible that most of the population may already have been infected and developed immunity by the time vaccines are widely available. A published cost-effectiveness analysis of H1N1 vaccination demonstrated that timing of vaccination relative to the peak of the pandemic influenced the cost effectiveness of the vaccination strategy [16].

Addressing Methodological and Policy-Making Concerns

Although Thailand and Singapore have methodological guidelines for conducting economic evaluations of healthcare interventions, stakeholder engagement meetings conducted by teams in both countries revealed that certain aspects of these guidelines were not appropriate for evaluations of COVID-19 vaccines, and adjustments were required. The types of considerations and issues raised during these discussions are detailed in the following sections. The issues raised during these stakeholder engagement meetings highlighted the importance of these activities in formulating the economic evaluation methods to ensure evaluations are fit for informing policy-making decisions.

Evaluation Scope

Defining the type of policy questions and scope of the analysis are key steps for understanding what type of economic evaluation methods are appropriate, necessary, and sufficient. The exact nature of the local policy- and decision-making context will vary between countries. Factors such as the interests of various stakeholders, the precise policy question to be addressed, and the types of resources that need to be allocated will inform whether cost-effectiveness analyses or cost-benefit analyses (or both) should be conducted for this evaluation. If the policy question considers how the healthcare budget or resources should be allocated to maximise health, then cost-effectiveness analysis is usually most appropriate; this could include which populations it would be most cost effective to vaccinate given a fixed number of COVID-19 vaccine doses. However, if the question considers how best to set the budget devoted to combatting COVID-19 (including reallocations of non-health government spending and adjustments to regulations or taxes), then cost-benefit analysis may be appropriate [17]. Both approaches have advantages and disadvantages, and the suitability of each approach is partially dependent on the availability of data. Equally, some policy questions may not require a model to be addressed where it is infeasible to do so. In some countries, such as New Zealand, multi-criteria decision analyses (MCDAs) are considered by decision makers, and the outputs of an economic evaluation may be incorporated as one of the criteria [43]. Suggestions have already been made on how MCDAs may be applied to inform COVID-19 vaccine allocation decision making [44]. Both the Singapore and the Thailand team opted to conduct cost-effectiveness analyses.

Study Perspective

The impact of COVID-19 has not been limited to the healthcare sector; the entire spectrum of economic activities has been disrupted, with many countries implementing lockdowns and closures of national borders that have dramatically changed work processes and consumer behaviour. These measures have had knock-on effects, with many countries experiencing rising unemployment, recessions, and the issuance of large fiscal stimulus packages in response. As such, policy makers are unlikely to consider analyses that cover only a healthcare provider perspective appropriate [18]. The Singapore study will adopt both health system and societal perspectives in their analyses, whereas the Thailand study will adopt a societal perspective only, in line with the Thai health technology assessment (HTA) guidelines [19]. It should be noted that the Agency for Care Effectiveness in Singapore recommends that the health system perspective only be adopted. However, the Second Panel on Cost-Effectiveness in Health and Medicine recommended the adoption of at least two perspectives [20].

Locally Relevant PICO Definitions

The population, intervention, comparator, and outcomes (PICO) framework is used in evidence-based medicine for framing and answering a clinical or health-related question and has since been used in the design of economic evaluations in healthcare.

Population

In the context of insufficient supply of COVID-19 vaccines for the entire population of any country, modellers should conduct scenario analyses for various key subpopulations of interest. These may include groups at high risk for severe health outcomes (e.g., the elderly, patients with non-communicable diseases), groups with high COVID-19 prevalence or transmission dependent on defined NPIs (e.g., those aged 20–39 years in Thailand, foreign workers living in dormitories in Singapore, or healthcare workers and high-contact occupations [e.g., workers at border control or sea ports]), young to middle-age adults only (in case vaccines have not been tested in or are not safe for elderly or juvenile populations), geographically defined subpopulations, and the population as a whole. To an extent, the ability to model and define different subpopulations will be dependent on the data available to capture their characteristics and distinctions from other groups. While vaccines are usually prioritised for those who are most at risk of developing severe health outcomes, we may want to re-think the appropriate target group for COVID-19 vaccination as there are both health and economic considerations. For example, some countries may prioritise prevention of the health system becoming overwhelmed, and other countries may prioritise restoring economic activities, although everyone hopes to achieve both. At the individual level, some will be eager to be vaccinated and others may hesitate or adopt a wait-and-see attitude. Hence, even if an individual is in a high-priority group, they may not want to be vaccinated.

Interventions and Comparators (Policy Measures)

The key intervention of interest is a COVID-19 vaccination programme; however, the precise manifestation of COVID-19 vaccines, future and present, in terms of efficacy and profile is uncertain. Various scenarios with different combinations of vaccine characteristics should be explored to better understand how individual characteristics affect the results, and to allow quick comparison with new vaccines as they receive regulatory approval. It is also worth considering whether interventions exist that could complement the effectiveness of vaccines, such as antibody testing prior to vaccination, which could maximise the effect of a vaccination policy by ensuring that vaccines are prioritised for individuals who test negative for antibodies. Alternatively, places that implement routine swab tests for workers at high risk of exposure to COVID-19 may not need vaccines as urgently. The intervention may also consider other measures such as the relaxation of certain NPIs in parallel with the vaccination programme implementation (Fig. 1). As vaccines may allow certain aspects of life to return to pre-COVID-19 norms, this issue is of particular importance for countries where the vaccine supply will be insufficient to achieve herd immunity. Examples of the types of restrictions that may be eased as vaccination programmes progress include international travel bans and certain social distancing measures, as these measures have had a significant impact on business sectors and thus have a high opportunity cost under a societal perspective analysis. In Singapore and Thailand, the main comparator for the evaluations will be the current policy measures that are in place to control COVID-19, but a variety of scenarios will be compared against each other.

Fig. 1

Possible combinations of policy interventions explored in an economic evaluation

Outcomes

Evaluations should consider several outcomes to provide a breadth of insight about the impact of COVID-19 vaccines to understand the full value of vaccination. If conducting a cost–benefit analysis, all outcomes need to be converted to a monetary value. The national HTA guidelines in both Singapore and Thailand express a preference for cost-utility analyses [21]. The most appropriate outcome may vary according to the decision problem. For analyses that aim to identify the most preferable combination of NPIs and vaccination interventions, COVID-19 deaths and cases may be preferable outcome measures; however, for decisions limited to health system resource allocation decisions, quality-adjusted life-years (QALYs) or disability-adjusted life-years (DALYs) may be preferred. Examples of outcomes that could be included in an economic analysis are as follows:

The number of COVID-19 cases (in different groups, as well as the overall population). Despite uncertainty about the importance of this measure in terms of population health impact, it is frequently cited in the media and, as such, is deemed an important indicator for the public and decision makers, and so should be included in analyses.

The number of severe cases: Severe cases are particularly burdensome to health systems and use scarce resources such as ventilators.

The number of COVID-19 deaths.

The number of excess deaths due to the knock-on impact of COVID-19 on other healthcare services if treatment capacity is exceeded or service provision is restricted.

DALYs or QALYs. Several studies have now been conducted to estimate utility weights for COVID-19 (although the long-term effects of COVID-19 remain unclear) [22-24].

The outcomes included in the Thailand and Singapore economic evaluations are costs (direct medical and non-medical and indirect costs), QALYs, COVID-19 deaths, and the number of COVID-19 cases, stratified by stages of severity.

Costs

With COVID-19 vaccines now available in the marketplace globally and purchase prices varying substantially, scenario and threshold analyses of vaccine prices should be included in any analyses. Vaccine delivery costs have also been estimated in many countries, and these costs should be considered, as should those associated with the vaccination delivery modality (such as facility- or mobile-based delivery services) [25]. There are other challenging hurdles to overcome with respect to costing, such as the decision about which types of costs to include in the model beyond the direct and indirect medical costs included in standard-practice HTAs. Evaluations may attempt to consider costs saved through reduced service provision for treatments for other diseases or reduced presentation at healthcare centres due to COVID-19 restrictions or social distancing measures that discourage people from travelling to health centres. Furthermore, as previously mentioned, the knock-on impact of COVID-19 on other sectors means that there are non-healthcare-related costs to consider within an economic evaluation from a societal perspective, e.g., the impact on the stock market index [26]. However, there is reason to be cautious about the scale of impact that a vaccination may have on the economic performance of a given country (e.g., it is unlikely that vaccinations will result in an instant recovery that recoups the lost stock market value sustained thus far) and instead may just act as protection against further economic losses. The opportunity costs of certain NPIs should also be included in the analyses, where possible, for example, restrictions such as international travel bans faced by the tourism industry or the impact of school closures for social distancing measures. Figure 2 illustrates the economic impact due to COVID-19 that can be divided into two parts: costs directly related to the pandemic and other costs that can be assigned to the policy response. The research teams from Singapore and Thailand are developing an approach to estimate the indirect costs of NPIs. Decisions regarding the types of costs to include should be related back to the policy decisions under evaluation: allocative decisions regarding health system budgets need to understand the costs of clinical services beyond COVID-19, and macroeconomic costs may need to be included for decisions regarding the implementation of NPIs in addition to healthcare costs as the impact.

Fig. 2

Framework for considering economic impact of COVID-19 pandemic

Effectiveness

Effectiveness data used in the evaluation should be identified from a systematic literature review, with the highest-quality data prioritised (e.g., data obtained from well-designed randomised control trials) that are suitable to the local context. However, in the current context of the pandemic, most data related to the effectiveness of vaccines and NPIs are likely to be real-world data, e.g., observational studies. In that regards, care should be taken to collect, analyse, and interpret real-world data. The REALISE (REAL World Evidence In ASia for HEalth Technology Assessment in Reimbursement) working group recently published a guidance document to serve exactly these purposes [27]. It should be noted that, although many NPIs have not been subjected to clinical studies, several papers have estimated the impacts of various NPIs [28-30]. The WHO target product profile outlines the minimum acceptable and preferred characteristics of COVID-19 vaccines [31]. These characteristics encompass different types of vaccine efficacy, including susceptibility, severity, and transmission. Although many clinical trials of vaccines have been designed to measure reductions in susceptibility or severity, transmission remains an important vaccine characteristic that should be incorporated in scenario analyses. Plausible scenarios and combinations of COVID-19 vaccine efficacy are used in the economic evaluations in Thailand and Singapore, informed by available data from clinical trials for COVID-19 vaccines already in use.

Modelling

Modellers face several key methodological challenges when conducting economic evaluations related to COVID-19. We have attempted to outline key uncertainties specific to COVID-19 vaccines, rather than general methodological challenges associated with the modelling of COVID-19 transmission and cases specifically as these have been documented extensively elsewhere (such as in the COVID-19 Multi-Model Comparison Collaboration Technical Group report) [32, 33], though it is still important that modellers consider these issues in their economic evaluations. For example, key models have differed significantly in their approaches to modelling COVID-19 transmission and the way in which they have implemented contact patterns as well as the types of interventions they consider. Furthermore, few models consider the indirect effects of COVID-19 on other diseases or the wider economic impact of COVID-19 [32], but this is necessary for a holistic understanding of the impact of COVID-19 vaccination programmes. Assumptions also differ about how vaccines work, how many vaccines will be available in the future, what scenarios of vaccination are explored, and the approach for assessing the optimal outcome for vaccination [34-36]. Different models may be needed for different settings for several reasons, such as differing demographics, COVID-19 transmission history, types of NPIs in place, and the approach for assessing optimal vaccination strategies.

Modelling teams must also weigh up the advantages and disadvantages of developing de novo models or adapting existing models; de novo models may take considerably more time to develop and test, yet results may be needed quickly to inform policy decisions. In Thailand, development of a model with a simple and easy-to-use interface was deemed important by stakeholders to improve transparency and confidence in the model results. Conversely, in Singapore, an age-stratified susceptible, exposed, infectious, and recovered (SEIR) compartmental model was desired and had not yet been developed (there were agent-based and non-age-stratified SEIR models); the model also needed to consider the population living in the foreign workers dormitories, as this was an important transmission location for Singapore. The model structure is similar to that of those used in other countries but parameterised and adapted for the Singapore context.

Uncertainty

Effectively considering uncertainty in economic evaluations of this type is paramount, and we have reiterated this throughout the paper. As is standard in economic evaluations for health technologies, individual parameter uncertainty should be considered through deterministic sensitivity analyses, and scenario analyses can be helpful for identifying and ranking the most important areas of uncertainty (in terms of their effects on outcomes and thus decision making), but it is also necessary to analyse the combined parameter uncertainty in the model through probabilistic sensitivity analyses. Furthermore, combinations of key scenarios should be explored to examine structural uncertainties, such as varying scenarios of vaccine availability, vaccinated subpopulations, geographical locations of vaccination (especially in countries with international land borders), waning immunity, and study time horizons [37].

For reference, key characteristics of the Thailand and Singapore economic evaluations are available in the electronic supplementary material.

Considerations for Local Analyses

It is important for all governments in all countries to conduct their own analyses to inform local policy responses to COVID-19, including the implementation of COVID-19 vaccines. It is preferable that such studies are designed, and ideally conducted, before vaccines are available or in use in a country, so that policy decisions and implementation procedures do not become delayed. In many cases, these contextualised studies are unlikely to take place in practice, because of the complex nature of COVID-19 modelling, limited in-country modelling capacity, and lack of data. The first step countries should take is to ensure that they develop a framework for evaluating and implementing vaccines, which can be used to evaluate current vaccines using the available data and can incorporate new data or vaccines as they arrive. The authors of this manuscript believe that all countries with capacity to do so should perform local analyses, and countries without capacity should seek international support. Conducting an economic evaluation for COVID-19 vaccines should be seen as part of a responsible and thorough preparation for future pricing negotiations (with vaccine manufacturers or a COVID-19 Vaccines Global Access [COVAX] facility) and coverage decisions.

Economic evaluations of this type, where the characteristics of an intervention are unclear, are sometimes called ‘early’ HTA. Early HTA can be used to inform clinical trial design, including the selection of appropriate surrogate outcomes and efficient sample sizes, and has been used by regulators and payers in Europe [38]. A benefit of early HTA is that by informing evidence-generation activities and by testing a range of hypothetical scenarios (with varied vaccine characteristics, implementation, and economic considerations), it can streamline and accelerate the eventual decision-making process for current and future COVID-19 vaccines. These benefits are not trivial; a local economic evaluation highlights the major sources of uncertainty or evidence gaps in a country’s evaluations in different scenarios. If conducted far enough in advance, these evaluations will provide country teams with the opportunity to mitigate uncertainty where possible through further research and to increase the robustness (and thus usefulness for policy making) of their economic evaluation results. A pertinent issue for countries with a low prevalence of COVID-19 is whether they should primarily use vaccines to reduce transmission into the country. Given the uncertainty regarding the impact of vaccination on transmission, an economic evaluation could be used to demonstrate the need for evidence generation on this topic, for example through a value-of-information analysis and through scenario analyses of different combinations of vaccine efficacy [39].

Throughout the economic evaluation, it is important to ensure that a range of stakeholders are involved and have the opportunity to provide input and feedback on the evaluation methods, as recommended in the collaborative modelling and implementation framework detailed in the COVID-19 Multi-Model Comparison Collaboration Policy Group report [40]. The stakeholders involved should be diverse enough to reflect the scope of the economic evaluation. For example, if the economic evaluation is being conducted from a societal perspective, then the stakeholders must represent groups from across society affected by the decision problem. As the effects of COVID-19 are far reaching and impactful (economically and in terms of health outcomes), stakeholder engagement should involve representatives from ministries other than health, the business sector, and the public. Ensuring this buy-in from stakeholders can help minimise opposition to the conclusions and recommendations of the studies.

Development of effective COVID-19 vaccines is only one step on the pathway to delivery of a successful vaccination programme. Economic evaluations should be used to help maximise the public benefit from a COVID-19 vaccination programme.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 26 kb)

Economic evaluations can have an important role in generating evidence to inform and optimise coronavirus disease 2019 (COVID-19) vaccination strategies.

This manuscript details the key methodological and procedural learnings from economic evaluations that have been conducted in Thailand and Singapore.

These learnings can help inform and improve economic evaluations of COVID-19 vaccination programmes in other countries.