Challenges of evaluating and modelling vaccination in emerging infectious diseases.

Outbreaks of emerging pathogens pose unique methodological and practical challenges for the design, implementation, and evaluation of vaccine efficacy trials. Lessons learned from COVID-19 highlight the need for innovative and flexible study design and application to quickly identify promising candidate vaccines. Trial design strategies should be tailored to the dynamics of the specific pathogen, location of the outbreak, and vaccine prototypes, within the regional socioeconomic constraints. Mathematical and statistical models can assist investigators in designing infectious disease clinical trials. We introduce key challenges for planning, evaluating, and modelling vaccine efficacy trials for emerging pathogens.

Introduction

Assessing vaccine efficacy for an emerging pathogen with unknown pandemic potential that may cause high morbidity and mortality presents a number of unique challenges. There may be limited or no data on transmission routes, pathology, viral reservoirs, clinical outcomes, level of pre-existing immunity, the pathogen’s seasonality, and/or epidemiology such as the COVID-19 pandemic. Identification and characterisation of the pathogen can take a long time, requiring laboratory equipment that might not be available, particularly for outbreaks in low-resource settings. The pathogen may mutate and adapt to new environments, causing inconsistent symptoms and outcomes.

Vaccination can help reduce the burden of emerging infectious diseases, but there are unique challenges of developing a vaccine in a pandemic setting. Vaccine development and approval typically take 10–15 years, which underscores the need for solutions to hasten the licensing process (Han, 2015). The average cost of advancing at least one new vaccine candidate for an infectious disease from preclinical trials to the end of a Phase 2a trial is $319–469 million, which includes the cumulative cost of failed vaccine candidates (Gouglas et al., 2018). The prolonged time between pathogen identification, vaccine licensure, and mass vaccine distribution may enable a highly transmissible pathogen to spread globally.

For both emerging and re-emerging pathogens, such as those on the World Health Organization’s list of Blueprint priority diseases (Mehand et al., 2018), it is difficult to predict when and where cases will occur, how many people will develop disease, and how long the outbreak will last. Furthermore, these prerequisites may be undermined by lack of surveillance data, which may hinder local and international responses to allow allocation of resources. It may therefore be challenging to identify high-risk populations that would provide the requisite number of cases for a vaccine trial. Unlike planning for endemic diseases, interventions for outbreaks must be agile to evaluate the protective efficacy of vaccines before the outbreak progresses or subsides in the area of the vaccine trials.

Other challenges in rapid vaccine development include understanding optimal dosage, correlates of protection, cross-reactivity with other closely related pathogens, short- and long-term side effects, antibody-dependent enhancement of disease, and absence of experimental animal disease models. Live attenuated vaccines are likely too risky for a pathogen with high pandemic potential, but inactivated vaccines may not have high enough efficacy to slow the outbreak. This problem, however, was obviated in the COVID-19 pandemic whereby the mRNA and viral vector/mRNA constructs were used in a rapid development cycle. Easily transmitted pathogens that cause severe disease may also require laboratories with high levels of biosafety precautions for vaccine production.

Emergency use authorization streamlines the regulatory process to full United States Food and Drug Administration (U.S. FDA) approval and subsequent licensure. However, once a vaccine is approved for emergency use, mass production to meet global demand for vaccination worldwide is challenging. There may be disparities in access between high and low resource settings. Cold chain distribution is challenging even in non-pandemic settings, particularly for vaccines that require ultra-low temperature freezers. Political pressure to hasten the vaccine authorisation process could lead to release of a potentially ineffective or harmful vaccine. Expedited vaccine trials may also lead to higher vaccine hesitancy, perhaps due in part to relatively short follow-up duration.

Information accrued during an abbreviated planning phase might be augmented by mathematical and statistical models. These can help investigators to subsequently design large-scale phase III vaccine trials by capturing dynamics such as trial duration, population size, the scale of the epidemic, the locations with the highest predicted incidence during the vaccine trial, and anticipated costs. However, the value of simulations is only as good as the surveillance data underlying the model assumptions. Here we consider some of the challenges in designing, evaluating, and modelling vaccine efficacy trials for emerging pathogens.

The challenge of unpredictable incidence and waning transmission

A unique challenge for investigators planning vaccine efficacy trials against emerging infectious diseases is the unpredictability of the disease. To accrue enough primary endpoints to reliably assess protective efficacy, the trial must be placed in a location with active transmission. While vaccine efficacy trials against endemic diseases may need to be very large or run for several years, they can rely on a relatively stable background incidence level of disease transmission such that eventually they will hit their target of sufficient cases to have adequate statistical power for the hypothesis test. In contrast, sites with reliable incidence for an emerging infectious disease may not exist, and the probability of future transmission in any pre-specified location may be so low that a standard trial is infeasible.

Several novel strategies are available to make planning these trials more successful, primarily by incorporating flexible and adaptive features that allow the trial to evolve along with an outbreak (Dean et al., 2019). A notable example is the ring vaccination trial design used to assess the rVSV-ZEBOV vaccine during the West African Ebola epidemic (Ebola ça Suffit Ring Vaccination Trial Consortium, 2015, Henao-Restrepo et al., 2017). As new Ebola virus disease cases were identified, their contacts and contacts of contacts formed a ring that could be randomised individually or as a cluster to vaccine vs. placebo. This design takes vaccination to where transmission is occurring. For ring vaccination to work as either a containment strategy or a trial design, mobile teams must be able to move quickly, and the vaccine must be fast-acting to provide immune protection quickly. Efficacy studies are no longer feasible for Ebola since there are licensed vaccines, and effectiveness studies, measures of how well the vaccine performs in the real world, are challenging in rapidly evolving outbreaks. Therefore, stepped wedge cluster designs can be used leveraging roll out of limited supply or test negative control designs. When outbreaks are sporadic, investigators may choose to vaccinate cohorts in areas where Ebola is likely to recur and then pool data when cases emerge. Alternative approaches are needed for multi-dose vaccines, so that the vaccine has a chance to achieve substantial protection before participants are at risk of infection. This suggests the role of an intermediate design; like a standard trial, much of the preparatory work is done in advance, including enhancing surveillance in a target area, yet recruitment only proceeds in sub-sites with evidence of transmission. The properties of such a design are to be explored further to prepare for future pandemics.

In addition to uncertainty in where outbreaks will start, investigators also do not know how long outbreaks will last. This can lead to questions about whether it is worthwhile to initiate a trial, since incidence could wane before the trial accrues enough data. Underpowered results are difficult to interpret, yet can impact policy-making and jeopardise the conduct of future confirmatory trials. One solution is to implement master protocols so that separate outbreaks can contribute to the same analysis (Dean et al., 2020a). Other potential approaches include merging trials after their initiation, as was done for several trials of Oxford/AstraZeneca’s COVID-19 vaccine (Voysey et al., 2021). A key challenge is figuring out how to make collaborations across locations or research teams more feasible, as investigators are best able to address these key scientific questions when working together. The concept of prospective meta-analysis (PMA) deserves consideration in this context, e.g., as described on the website for the Cochrane PMA Methods Group (The Cochrane Collaboration, 2021).

The challenge of integrating models into design decisions

Mathematical models of infectious diseases can enhance the speed and probability of successful conduct of vaccine efficacy trials, which are very resource intensive and can take a few years to complete (Halloran et al., 2017). Models can be used to select countries and sites to include in the trial because they are predicted to have high infection rates, to predict trial endpoint rates for sample size calculation, and predict when key milestones will be reached (e.g., interim analyses for Emergency Use Authorization [EUA] applications) (Madewell et al., 2021). Model accuracy to predict rates and timing of infection incidence is important to meet these purposes.

A challenge in using models to guide efficacy testing of vaccines is which model, or type of model, to use (Kretzschmar et al., 2021). Early in the COVID-19 pandemic, several models were created that provided a wide range of epidemic estimates. The United States Centers for Disease Control and Prevention (U.S. CDC) created an averaging ensemble approach of many models in the public domain and portrayed these on its website (Centers for Disease Control and Prevention, 2021). For vaccine development teams, there is a choice whether to adopt the average model or to evaluate the accuracy of the individual models and only work with those that appear to have the best predictive accuracy (Dean et al., 2020b).

Models are generally validated by comparing a prediction made in the past with a current “gold standard” of accuracy. A challenge is what “gold standard” to use if the population-based infection incidence is not available (Metcalf et al., 2015). Some options for gold standards are positive cases from testing sites, mortality data or seroprevalence surveys, which are then converted into predictions for trial endpoints, such as incidence of symptomatic or moderate-to-severe infections. The translation of surveillance data into symptomatic infection incidence is prone to erroneous or over-simplifying assumptions and validation checks are needed. In addition, each type of surveillance data has its limitations in accuracy. Mortality data are subject to varying processes for cause of death determination, and some national authorities may suppress mortality data for political reasons. The number of positive cases at testing centres is influenced by test availability and accuracy of tests. Such source data inaccuracies need to be accounted for. Population-based cross-sectional serosurveys may be used to measure prevalence, which is the equivalent of cumulative incidence, and models of disease infection can estimate cumulative incidence/prevalence; however, serosurveys require resources and time to conduct.

In the absence of site-level epidemiologic data, models often extrapolate key parameters, such as age-structured contact rates, seroprevalence, case-fatality rates, and public health measures (e.g., social distancing, lockdowns) from other locations, which could be at different stages of the epidemic. These factors, as well as the reproductive number, are highly dynamic over time and by location, reflective of numerous social, geographical, political, and other factors, and need to be updated regularly in the model (Bertozzi et al., 2020). Models may not accurately reflect the speed of roll-out at any given site. A region predicted as a future hot-spot via a rise in cases could also be targeted for additional mitigation measures (e.g., vector control) that affect its risk profile. These data may be further confounded by comorbidities, occupation, socioeconomic status, and access to healthcare and treatment. Vaccination rollout strategies may vary from country to country, including which groups are prioritised, the rate of vaccination, and which vaccines are authorised. These variables coupled with limited vaccine trial data make it challenging to quantify the relevant aspects of vaccine safety and performance for specific population subgroups. This would require methods that allow heterogeneities in the population to be taken into account.

Factors specific to the pathogen or host that may be unknown early in the pandemic could necessitate model refinement over time, including time from symptoms to transmission, latent period, infectious period, protection against reinfection, waning immunity, and asymptomatic transmission. Continual model calibration and sensitivity analyses are required to evaluate how well the model output matches new observed data.

Early vaccine trial data can provide insight into vaccine efficacy against symptomatic disease, but it may be months before there are any data regarding vaccine efficacy for other endpoints such as infection and transmission. Models may therefore use historical vaccination data for other closely related pathogens for parameterisation.

The challenge of quickly implementing trials

When an emerging infectious disease produces an outbreak, even if a vaccine candidate is available for evaluation, several logistical challenges exist. First, a protocol for a field study must be prepared. Then the countries, or subnational units, where the outbreak is occurring must give Institutional Review Board approval. The necessary ability to conduct a field trial including knowledge of the local human populations must be available. If preparation is not made before outbreaks, then it may be that the outbreak is over before the trial even begins. The Ebola outbreak in West Africa 2014–2016 illustrated these challenges. The PREVAIL trial in Liberia (Kennedy et al., 2016) and STRIVE trial in Sierra Leone (Widdowson et al., 2016) were unable to evaluate Ebola vaccine efficacy because transmission had essentially ended by the time of implementation. In contrast, the ring vaccination trial in Guinea was able to evaluate an Ebola vaccine candidate despite declining transmission because the design took the trial to where the transmission was occurring (Henao-Restrepo et al., 2017, Henao-Restrepo et al., 2015). Thus, pre-preparation as well as innovative study design can help overcome some of the challenges of quickly implementing a trial.

The challenge of improving data collection, recruitment, retention

Emerging infectious diseases, particularly very contagious ones such as COVID-19, pose challenges for conventional randomised controlled trials (RCTs). Conventional RCTs often have frequent clinic visits, complex assessment schedules, highly selected participant populations, and concentration of sites in academic centres. In resource limited settings, with a backdrop of the outbreak and in regions where clinical research has never been performed, data collection, and quality oversight can be challenging. However, the COVID-19 pandemic has innovated the approach to RCTs, with lessons for future outbreaks. Because of social distancing and travel restrictions, unnecessary contacts need to be reduced and risks minimised. The periodic lockdowns create an environment that demands flexibility to adapt. The urgency and economy of the trials require RCTs with tens of thousands of participants with minimal loss to follow-up. For example, Pfizer enroled 43,000 participants in its Phase III efficacy trial for SARS-CoV-2 (Polack et al., 2020). Rather than a highly selected population, a wide range of participants representing the most vulnerable are needed in the trials, such as older adults, people with comorbidities, and ethnic and religious minorities. To achieve a diverse population by age, the COVID-19 vaccine trials conducted stratified enrolment by age group and large efforts were devoted to recruiting ethnic/racial minorities into the trials.

Minimising disruption to routine life is crucial for recruitment and retention in clinical trials (Fogel, 2018). One approach that responds to this need is the Decentralized Clinical Trial (DCT). DCTs are defined as those executed through telemedicine and mobile/local healthcare providers, using procedures that vary from the traditional clinical trial model (e.g., the investigational medical product is shipped directly to the trial participant). For example, CVS Clinical Trials Services partnered with biopharmaceutical stakeholders to design decentralised approaches for Phase III/IV trials (CVS Health, 2021). DCTs are participant-centred trials (Clinical Trials Transformation Initiative, 2018). Why are DCTs particularly appropriate for emerging infectious diseases? DCTs fit as pragmatic randomised trials in health care delivery framework as outlined in the U.S. FDA Real Word Evidence framework (U.S. Food & Drug Administration, 2019). Current technology provides platforms for data integration, validated tools for data collection, e-consent, and digital engagement. The participant-centred focus allows engagement in the study protocol and communications. Technology also allows digital outreach and screening, faster trial participant recruitment, e-diary for signs and symptoms, and addressing health disparities by more diverse recruitment. It may be possible in the future to develop research infrastructure that links existing research cohorts, or readiness cohorts, through these technologies which have potential to accrue large numbers of participants thereby enabling even more rapid response. In addition, digital tools have also been used to follow trial subjects over longer follow up periods for safety surveillance from patient electronic health records and wearable data of patient function and physiology (Dhruva et al., 2020).

The challenge of maintaining a placebo arm

The value of concurrent controls cannot be underestimated particularly given a setting of a dynamic and unstable disease. Concurrent controls enable differentiation of patient outcomes resulting from the vaccine from outcomes caused by other confounding factors. The control group demonstrates what would have happened to the vaccinated group had they not been vaccinated. Concurrent controls are selected from the same source population as the vaccinated group over the same period and should be similar to the vaccinated group with respect to all other factors that could affect the outcome.

Placebo-controlled trials are the most efficient and reliable path to generate evidence of vaccine safety and efficacy. Yet in a setting where multiple vaccine candidates are being developed in parallel, some may receive EUA or full approval before others, as in the case of COVID-19. The availability of an alternative vaccine is a complicating factor. With an authorised vaccine available to the public, according to local eligibility guidelines, a natural question is why someone who is otherwise eligible for vaccination would choose to continue in a randomised trial where they might have received placebo (Rubin, 2021).

There may be populations where access to authorised vaccines is still limited, so placebo-controlled trials would be considered acceptable. One way to make enrolment more attractive is to change the allocation ratio in the RCTs. Pfizer-BioNTech, Moderna, and Johnson & Johnson trials used 1:1 allocation, so participants had a 50% chance of getting the vaccine (Baden et al., 2021, Polack et al., 2020, Sadoff et al., 2021). The Oxford/AstraZeneca phase III trial professed 2:1 allocation in favour of getting the investigational vaccine (AstraZeneca, 2021). Another option would be to allow placebo group participants subsequent access to vaccines through public availability under EUA approvals. A further alternative is a delayed vaccination comparator where ultimately all participants are vaccinated, which has its own attendant strengths and weaknesses (Nason, 2016). This strategy was used in the Guinea Ebola ring vaccination trial (Henao-Restrepo et al., 2017).

Non-inferiority trials are another option for evaluating a new vaccine in which participants are randomised to an experimental vaccine or to a vaccine with an established record of safety and efficacy. The investigational vaccine may have greater efficacy relative to the existing vaccine, or have other advantages such as lower costs, ease of storage and transport, and fewer required doses. Non-inferiority trials would likely be prohibitively large to accrue enough events but may enable reliable evaluation of efficacy (Fleming et al., 2021). Identification of immune correlates of protection from vaccines could facilitate development of new and modified vaccines (World Health Organization, 2013).

A related challenge is continuing to maintain placebo-control within a trial for a vaccine that has already established efficacy. Maintaining the placebo arm allows us to generate comparator-controlled data on long-term safety and duration of protection, as well as information regarding vaccine efficacy against severe disease or for specific subpopulations (e.g., defined by age or comorbidities) (Krause et al., 2020). For assessing waning of protection, an alternative strategy is blinded crossover where those who had been randomised to receive placebo would receive active vaccine and those who had been randomised to receive active vaccine would receive placebo injections (Follmann et al., 2020). There are logistical challenges, however, of having to bring back tens of thousands of people for a second round of shots. This design also does not enable the assessment of long-term safety.

The challenge of evaluating vaccine effects on infectiousness

Estimating the effect of vaccination in reducing the infectiousness to other individuals is crucial for understanding indirect population-level protection, for many policy decisions, and to parameterise the models used to explore these decisions. Here we denote the vaccine efficacy for infectiousness () as 1 minus the ratio of the transmission rate from a vaccinated infected person to the transmission rate from an unvaccinated infected person; thus, conditions on (assumed) exposure to infection. It is further possible to stratify on the vaccination status of the individuals being exposed (Halloran et al., 1997).

An ideal study design to estimate would be to randomise individuals to vaccination and then recruit the contacts of these randomised individuals (e.g., household members, dormitory co-residents). When one of the randomised individuals becomes infected, one could observe the proportion in the unit who become infected. One could estimate by comparing the secondary attack rates (SAR) from vaccinated infected individuals with SARs from unvaccinated infected individuals. For example, was estimated for pertussis vaccination using households in Niakhar, Senegal (Halloran et al., 2003, Préziosi and Halloran, 2003).

Many challenges exist in conducting studies to evaluate . One is ascertaining infection versus disease in the primary (index) cases. For instance, in the pertussis vaccination study, the index case as well as the secondary cases were ascertained on disease. In this situation, the analysis was restricted to the effect of reducing transmission of a clinical disease outcome. If a large proportion of the infections are asymptomatic, this would not necessarily provide an accurate estimate of the effect of vaccination on reducing person-to-person transmission. It would be better to be able to ascertain infections in a timely manner in both primary and secondary infections. If the index case is ascertained on disease, it then would be possible to do targeted, active ascertainment of infection in those who have been exposed. However, the would still not measure the effect of vaccination on reducing infectiousness from pre-symptomatic or asymptomatic infection.

In the case of viral respiratory infections, another option for estimating the reduction in potential infectiousness is to measure viral load in the nasopharynx as a surrogate of . For example, Levine-Tiefenbrun et al. demonstrated decreased SARS-CoV-2 viral load among infections following vaccination (Levine-Tiefenbrun et al., 2021). They compared the cycle threshold (Ct) values from PCR testing in vaccinated and unvaccinated infected individuals. The Ct value is defined as the number of cycles for the fluorescent signal to cross the background level (threshold). They estimated that vaccination reduced the viral load by about a factor of four for infections occurring 12–28 days after the first dose.

A challenge of using viral load as a surrogate for infectiousness is that it might not have been validated as being closely related to infectiousness. One can also calculate viral load in various ways, including peak viral load, area under the curve viral load, duration of viral load, viral load at a particular time point, among others.

The challenge of estimating indirect, total, and overall effects

When a vaccine is introduced into a population, vaccinated individuals develop an immune response, which reduces the probability of infection or disease (direct effects). Concomitantly, there is potential for indirect effects of vaccination, also known as spillover or herd effects, which are the effects of vaccination in reducing infection or disease among unvaccinated individuals in the same population as vaccinated individuals (Fine et al., 2011, Halloran et al., 1991). Vaccines may provide indirect protection by reducing transmission via reduction in duration of the infectious period, pathogen load, and/or symptoms that promote spreading. Vaccines may additionally provide indirect protection at a group level by reducing the opportunities for infection transmission by reducing the number of social contacts that can transmit infection. An example of indirect effects is a reduction in incidence among groups not targeted for vaccination, such as children or older adults. Evidence of indirect effects has been demonstrated for several vaccines, including pneumococcal disease (Whitney et al., 2003), Haemophilius influenzae b (Adegbola et al., 2005), influenza (Arinaminpathy et al., 2017), rotavirus (Van Effelterre et al., 2010), hepatitis A (Samandari et al., 2004), and others. Indirect effects may even be greater than direct effects (Pradas-Velasco et al., 2008).

The total effects of vaccination represent the combined effect of being vaccinated and being in a population with a vaccination programme, or the combination of direct and indirect effects (Halloran et al., 1991). As indirect effects are almost always positive, the total effect is usually greater than the direct effect (Shim and Galvani, 2012).

The overall effects of vaccination are the difference, or relative reduction, in average outcomes in the entire population where some individuals are vaccinated compared to the average outcomes of the entire population that did not receive vaccination (Halloran et al., 1991). The overall effect of vaccination is therefore of great interest for public health considerations.

Individually randomised controlled trials may substantially underestimate total vaccine effects by ignoring population-level effects. Cluster randomised controlled trials (cRCTs), however, allow estimation of indirect, total, and overall effects (Halloran et al., 1999). Total effects are the primary measure of vaccine efficacy in cRCTs as vaccinated individuals are both directly protected from vaccination and indirectly protected from other vaccinated individuals in the same cluster (World Health Organization, 2020). Indirect effects are estimated in cRCTs from the difference in the degree of protection that unvaccinated individuals gain in vaccinated clusters compared to unvaccinated individuals in unvaccinated clusters (Halloran et al., 2010). Once a vaccination programme begins in a population, it may be difficult to disentangle direct and indirect effects.

cRCTs generally assume that disease transmission occurs within cluster without transmission from outside clusters or migration of individuals between clusters (Halloran et al., 2010). Contamination between clusters receiving vaccination and those not receiving vaccination may dilute the indirect, total, or overall effects of the vaccination programme by increasing the similarity of clinical outcomes between trial arms (Halloran et al., 2010). Clusters should therefore be stable and discrete, precluding transmission across clusters. “The fried egg design” may be used to control for contamination resulting from contact between clusters (Hayes and Moulton, 2017). In this design, the entire cluster is vaccinated, but only the central area or yolk is assessed to control for contamination between clusters.

Evaluating indirect, total, and overall effects of vaccination depends on the specific epidemiological setting and is influenced by factors specific to the pathogen (virulence, duration of infectivity, asymptomatic infection), vaccine (effectiveness, duration of protection, antibody response), and population (population density, vaccination coverage, allocation to different groups, social mixing patterns, natural immunity, hygiene conditions) (Halloran et al., 2010). These variables that could affect clinical outcomes differ between settings, thereby making it difficult to extrapolate indirect, total, and overall effects to other settings. Indirect effects are also difficult to measure at sites of extreme coverage (e.g., everyone or no one gets vaccinated). Although the indirect effects of vaccination generally augment vaccine efficacy, challenges arise when indirect effects are detrimental (e.g., transmission-blocking malaria vaccines or immune-enhanced exacerbation of disease) (Halloran et al., 1989).

Another challenge arises when surveillance is not specific for the illness of interest or there may be asymptomatic infections, both of which may necessitate additional laboratory testing. Nonspecific case definitions can attenuate population-level effects (Halloran et al., 2010).

Measuring population-level vaccine effects on disease incidence and mortality requires collecting and analysing data after vaccine introduction into a population and may need to be extended over several seasons. Halloran and Hudgens consider estimating population-level effects of vaccination using large, routinely collected datasets such as health insurance claims, electronic health records, and registries (Halloran and Hudgens, 2018). One such example for bacterial disease vaccines is the U.S. CDC’s Active Bacterial Core (Whitney et al., 2003). Ideally, large cRCTs are needed with high vaccination coverage and many clusters in both vaccination and placebo arms. These trials may be expensive and not generalisable to vaccination campaigns. Consequently, mathematical modelling may be helpful to better quantify population-level effects. Models may account for pathogen transmission characteristics given different levels of vaccination coverage, vaccine effectiveness, adverse effects, and vaccine hesitancy (Jordan et al., 2006). The decision regarding the introduction of a new vaccine may be based in part on the level of indirect protection afforded by the new vaccine.

The challenge of interpreting vaccine efficacy

Vaccines are described by their efficacy, which is measured as the proportionate reduction in infection or disease among the vaccinated cohort compared to the unvaccinated cohort. A vaccine with 100% efficacy would result in zero incidence among vaccinated individuals, whereas a vaccine efficacy of 70% represents a 70% reduction of infection or clinical disease from the expected total if not given vaccination. Although vaccine efficacy is typically described by a single number, there are alternative measures of vaccine efficacy – efficacy against infection, transmission, disease, and severe disease – and these can vary for a single vaccine. In most trials, the primary endpoint is laboratory-confirmed symptomatic disease, such as for COVID-19 (Shapiro et al., 2021). Efficacy against severe disease, hospitalisation, and death are often of greatest clinical relevance (Dean and Madewell, 2021). In general, vaccines that prevent mild symptomatic disease tend to have a similar or stronger protective effect against severe disease. Understanding vaccine efficacy against susceptibility to infection with or without symptoms is determined by testing individuals regardless of symptoms for the pathogen of interest (e.g., RT-PCR) or a serological response to that pathogen. Even if a vaccine does not prevent infection, it may reduce the likelihood of symptomatic disease, progression to severe disease, or transmission to others by boosting the immune response. Another endpoint is efficacy against infectiousness which measures how well the vaccine reduces the ability of the individual to transmit the disease to others. The vaccine may reduce duration of the infectious period, pathogen load, and/or symptoms that promote spreading like vomiting, diarrhoea, coughing, or sneezing. Vaccine efficacy endpoints are prequels to subsequent emergency use authorisations and can influence public interest in vaccination.

The real meaning of measures of efficacy of COVID-19 vaccines, as obtained from randomised double blinded placebo-controlled trials is still an open topic for debate. This is attested by recent points of views expressed (Lipsitch and Dean, 2020, Olliaro, 2021). Three elements that populate this debate are the mechanisms of action by which a vaccine can manifest its effect, the heterogeneity of the immune response of individuals upon vaccination, and the experimental conditions under which a vaccine is being tested (the history of past and current contacts with infectious individuals that challenge the protection conferred by the vaccine). Our list of elements to consider is not exhaustive but provides a first approximation to the challenges in interpreting the epidemiological meaning of efficacy of a vaccine from an individual and population perspective.

Fig. 1 provides a starting point for our scrutiny of the concept of vaccine efficacy. We closely follow the discussion in Halloran et al., 1999, Struchiner and Halloran, 2007, Struchiner et al., 1994. The diagram depicts a schematic description of the sequence of pathogenic processes leading to the endpoint of interest that was identified as being the target of evaluation. The first dashed rectangle (A) represents vaccination (V) and covariate levels (X) of each participant in the trial population. Vaccine allocation and the role of randomisation are important design considerations at this point. In field trials, vaccinated and unvaccinated individuals are exposed to an infectious source by natural means (dashed rectangle (B)). Status E (exposure) is not known or difficult to assess. Rates and describe the instantaneous probability of being challenged by an infectious inoculum and are functions of time and other social and environmental factors. Once challenged by an inoculum, vaccinated and unvaccinated individuals might progress up to the endpoint of interest at rates and , respectively (dashed rectangle C). Both rates are dependent on time and social, environmental, and biological covariates. Vaccine efficacy is then reported as based on total (i.e., compound) rates and which depict the transition from the susceptible vaccinated and unvaccinated states to the endpoint of interest. This measure, however, does not translate into statements such as “The rationale is that if 95% of people are protected from disease after two doses …” (Olliaro, 2021). The latter statement, however, reveals an ideal concept with which our cognition feels comfortable, although unavailable to us unless we can overcome the methodological challenges identified next.

Fig. 1

Evaluation of vaccine efficacy.

The rationale expressed by The Lancet editorial (Olliaro, 2021) implies the need to estimate the proportion of people protected from disease by the vaccine. Therefore, alternative sources of protection such as adherence to social distancing behaviour and natural resistance to infection due to inherited genetic makeup, for example, need to be controlled for by means of a placebo group. Most importantly, both groups, vaccinated and placebo, should share the same challenging experience to the virus under a known exposure history. Being compound rates, and are functions of and . Therefore, the latter two rates representing background infectiousness (challenge) enter the definition of and render its interpretation more difficult. Notice that the contributions of and do not cancel out when we take the ratio . The complex interaction between and the force of infection () has also been noticed by Kaslow (2021). We could argue then that is closer to the cognitive ideal of a measure of efficacy that is independent of the experimental settings through background infectiousness. Under the assumption of time invariant rates we have (see for example (Morrison, 1979)): and

Therefore, the relationship between the two measures of vaccine efficacy is

The second equality is obtained by assuming as is the case in randomised trials. Also notice that, in principle, we could directly estimate and from the placebo arm once proper data characterising exposure to infection and onset of symptoms, which are routinely collected in the trials, become available. Access to these data allows for breaking down into and .

Although seems to be an improvement over , its interpretation is still subject to further scrutiny. From the previous expression one can also estimate but its relationship to the proportion of people actually protected from disease due to vaccination still requires further clarification. The rate describes failure of the vaccine in blocking infection and/or eliminating symptoms. Our inability to disentangle these mechanisms can have far reaching implications in understanding patterns of infection transmission in the population after vaccination. In addition, this rate is a summary measure of, possibly, heterogeneous individual protections conferred by the vaccine. Distinct distributions of protection can lead to the same value of indicating that the former distributions are non-identifiable from alone.

It is also challenging to compare estimates between vaccine trials during a pandemic. The vaccines differ in terms of dosage, storage, and timing between doses. Trials may vary in terms of sample size, eligibility criteria, clinical endpoints, duration, time period of vaccine efficacy assessment, location, infectiousness of evolving viral variants, and time of year (Rapaka et al., 2021). Point estimates of may therefore transmit limited information and should be accompanied by measures of uncertainty. For example, Pfizer/BioNTech and Moderna finished enroling participants in COVID-19 vaccine Phase III trials by October, 2020, whereas Johnson & Johnson finished enrolment in December, 2020 when cases, hospitalisations, and prevalence of new SARS-CoV-2 variants were higher in many countries. Some trials may test for SARS-CoV-2 infection only in symptomatic participants whereas others test participants regardless of symptoms. Definition of symptomatic infection may also vary between trials. Trial data may be presented via peer-reviewed publications, preprints, press releases, and others and may vary in terms of comprehensiveness and quality of data presented.

In summary, the interpretation of measures of vaccine efficacy, obtained from randomised placebo-controlled field trials, as the “proportion of people protected” by the vaccine requires the input of additional information, such as: disentangling of the contributions of infection blocking and disease modifying as the mechanisms of action of the vaccine; the distribution of protection conferred by the vaccine (e.g., all-or-nothing and leaky); the reconstruction of the history of infections before and after vaccination including number of infections, interval between infectiousness, and strain composition. Furthermore, the “proportion of people protected” by the vaccine is not an accurate definition unless all the people are identical in every way. These challenges can only be overcome by further development of laboratory and statistical methodology.

The challenge of continuously evaluating a vaccine when a pathogen is evolving

The challenge of continually evaluating a vaccine when a pathogen is evolving is being illustrated by the pandemic of SARS-CoV-2 and vaccine trials and implementation. The part of the virus expressed (or encoded) by most of the initial vaccine candidates corresponds to the spike protein in the wild-type SARS-CoV-2 variant from Wuhan, though the antigens and epitopes derived from the spike protein differ across the different vaccine constructs. Less than a year into the pandemic other evolutionary variants are arising that have mutations in the spike protein. A biological challenge is to determine whether immunity induced by the vaccines induces neutralising immunity to the new variants. In principle, this can be achieved by a number of different tests (Plotkin, 2010, World Health Organization, 2013). In principle, also, new vaccines can be produced that have RNA that produces pieces of the new variants’ spike proteins in them. If a vaccine has been approved for use, whether emergency or licensed, then regulatory challenges arise under what conditions can the new vaccine containing the variant be approved for use without having to conduct Phase III trials again (U.S. Food & Drug Administration, 2021). Immunogenicity markers can be helpful in bridging between original vaccine and variant-modification vaccines, as has been done for haemophilus influenzae vaccines in the past. Another logistical challenge is to step up manufacturing of the new vaccine as well as to determine geographically where initial administration should be prioritised. Some manufacturers rely on technological transfer of sterile manufacturing processes to regional vaccine manufacturers to provide adequate supplies globally in a short time period.

Current challenges and future directions

We described a number of innovative vaccine trial designs for emerging pathogens with pandemic potential. Although COVID-19 presented a test case for innovations, many of the trials used for developing vaccines for COVID-19 were fairly traditional in design, but noteworthy in the speed of execution of the trials. Many challenges of vaccination are exemplified directly from experience with COVID-19, which may be relevant to future pandemics. Likewise, Ebola presented challenges. Large-scale manufacturing and distribution of vaccines are major challenges, and the access to vaccine quantities varies greatly between countries. A paucity of supply means that decisions need to be made about whom to vaccinate first—vulnerable populations or those most likely to transmit the pathogen assuming the vaccine prevents infectiousness. Certain groups may be at higher risk of exposure (e.g., homeless, incarcerated, refugees) or outcomes (older adults, individuals with comorbidities, pregnant women). A modelling study demonstrated that an effective transmission-blocking vaccine prioritised to adults 20–49 years was optimal at reducing overall incidence of SARS-CoV-2, whereas a scenario that prioritised adults > 60 years was better at reducing mortality and years of life lost (Bubar et al., 2021). Optimal strategies identified in other studies include allocating vaccines based on the number of social contacts and total social proximity time (Chen et al., 2021), targeting essential workers early in the pandemic (Mulberry et al., 2021), and prioritising individuals by county of residence or communities of low socioeconomic status (Chapman et al., 2021). Another consideration is vaccinating certain occupational workers such as teachers to minimise social and economic disruption. As a greater proportion of the population gets vaccinated, strategies may shift to utilising more pop-up and mobile clinics to reach people living in rural and underserved communities. Certain groups such as children and pregnant women may need to wait longer as more information about vaccine characteristics becomes available.

For two-dose vaccines, another consideration is whether to allow individuals to complete the series or vaccinate as many people as possible with initial doses while delaying boosters. Half doses are another possibility, which could help get more vaccines to the public faster. These considerations depend in part on the durability of protection after the first dose, whether the vaccine reduces infectiousness, and whether partial vaccine-induced protection may select for more variants (Matrajt et al., 2015).

Vaccination strategies may shift over time from pandemic to endemic. The pathogen may continue to circulate worldwide or in specific regions for years, which is dependent on factors including herd immunity and pathogen evolution. Initial vaccines may not be as effective against emerging variants, which may necessitate booster doses designed to provide protection against these. Furthermore, it takes time to establish duration of immunity following vaccination.

Overcoming vaccine hesitancy is becoming increasingly more difficult. Reasons for COVID-19 vaccine hesitancy in a large representative sample of U.S. adults included concern about side effects, belief that vaccination is unnecessary, and distrust of the government (King et al., 2021). Anti-vaccination conspiracy theories may play some role by spreading false information about vaccine side effects and understating the risk of COVID-19. Exposure to online misinformation is associated with lower intention to get vaccinated to protect oneself and others (Loomba et al., 2021). Strategies to overcome vaccine hesitancy include providing clear and concise evidence-based information about the pathogen, vaccine, and side effects. Full U.S. FDA approval of vaccines could improve public confidence regarding vaccine safety, which was demonstrated for 2009 H1N1 influenza (Quinn et al., 2009).

Evaluating the impact of vaccination is challenging and may not always be quantifiable. In addition to direct health benefits (e.g., reduction in morbidity and mortality), there may also be economic (e.g., fewer healthcare costs) and social (e.g., strengthening healthcare infrastructure) benefits (Rodrigues and Plotkin, 2020). The challenges outlined herein underscore the importance of dynamic modelling for the design of vaccine trials for a pathogen with pandemic potential, accounting for heterogeneous populations and rapidly evolving understanding of the pathogen and its epidemiological characteristics.

CRediT authorship contribution statement

ZJM: Writing – original draft, Writing – review & editing. NED: Conceptualization, Writing – original draft, Writing – review & editing, Supervision, Project administration, Funding acquisition. JAB: Conceptualization, Writing – original draft, Writing – review & editing. PMC: Conceptualization, Writing – original draft, Writing – review & editing. KJD: Conceptualization, Writing – original draft, Writing – review & editing. CJS: Conceptualization, Writing – original draft, Writing – review & editing, Supervision, Project administration. MEH: Conceptualization, Writing – original draft, Writing – review & editing, Supervision, Project administration.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: The work was funded by National Institutes of Health grants R01 AI139761 (Natalie E. Dean, M. Elizabeth Halloran, Zachary J. Madewell) and R37 AI032042 (M. Elizabeth Halloran). The content is solely the responsibility of the authors and does not necessarily represent the views of NIH. Claudio J. Struchiner was partially funded by CNPq and FAPERJ. Jesse A. Berlin and Paul M. Coplan are full-time employees of and shareholders in Johnson & Johnson. Kourtney J. Davis is a full-time employee of and shareholder in Janssen.