Evaluating the Effectiveness of Vaccines Using a Regression Discontinuity Design.

The regression discontinuity design (RDD), first proposed in the educational psychology literature and popularized in econometrics in the 1960s, has only recently been applied to epidemiologic research. A critical aim of infectious disease epidemiologists and global health researchers is to evaluate disease prevention and control strategies, including the impact of vaccines and vaccination programs. RDDs have very rarely been used in this context. This quasi-experimental approach using observational data is designed to quantify the effect of an intervention when eligibility for the intervention is based on a defined cutoff such as age or grade in school, making it ideally suited to estimating vaccine effects given that many vaccination programs and mass-vaccination campaigns define eligibility in this way. Here, we describe key features of RDDs in general, then specific scenarios, with examples, to illustrate that RDDs are an important tool for advancing our understanding of vaccine effects. We argue that epidemiologic researchers should consider RDDs when evaluating interventions designed to prevent and control diseases. This approach can address a wide range of research questions, especially those for which randomized clinical trials would present major challenges or be infeasible. Finally, we propose specific ways in which RDDs could advance future vaccine research.

A critical aim of infectious disease epidemiologists and global health researchers is to evaluate disease prevention and control strategies and quantify reductions in risks following implementation. In particular, vaccines and vaccination programs have been one of the most successful public health interventions ever developed, yet challenges remain for assessing the effectiveness of new vaccines and the use of existing vaccines in novel settings or among new target groups. Epidemiologic studies evaluating vaccines are essential to quantify vaccine efficacy and effectiveness against disease-specific endpoints and to identify optimal strategies for preventing infectious diseases. Individually randomized clinical trials are typically required for the licensure of vaccines and provide important evidence needed to evaluate vaccine efficacy and safety. However, challenges often arise in extrapolating efficacy results from randomized clinical trials to different epidemiologic contexts. Vaccine efficacy and effectiveness in diverse epidemiologic contexts might differ according to myriad factors including, among others, demographic characteristics of the at-risk population, the transmission intensity of the pathogen, and the existing distribution of immunity among the target population. For example, vaccine-induced immunity can interact with preexisting natural immunity in populations; the same vaccine can have different effectiveness in populations with different levels immunity (1, 2). Thus, evaluating vaccines introduced into new contexts is crucial.

The regression discontinuity design (RDD) provides an opportunity for causal inference using observational data to evaluate vaccines and to produce robust estimates of vaccine effects (3, 4). Vaccine policies often assign individuals to treatment or to the status quo based on an arbitrary cutoff of a continuous variable such as age. The RDD is a quasi-experimental design first proposed in 1960 (5) to evaluate interventions where eligibility is based on an arbitrary cutoff. It assumes that, in a small neighborhood around the cutoff, treatment assignment is ignorable, so that locally the potential outcome is independent of assignment as in a randomized study (3, 6, 7). Therefore, the RDD can be used to estimate the local treatment effect (around a small neighborhood) at the cutoff that defines eligibility for the treatment. Recent advances in econometrics and statistics have shed light on the conditions under which RDDs produce good estimates (8). One identifying assumption of the RDD is that the assignment covariate, such as age, is continuous around the cutoff. Another is that the relationship between the assignment covariate, such as age, and the potential outcomes under the two possible assignments is continuous at the cutoff (3). Although developed for continuous outcomes, it has been shown that RDDs can be extended to dichotomous and time-to-event outcomes (3), as might be used in vaccine studies. If the assumptions inherent to an RDD are satisfied, the design has strong internal validity and minimizes concern about selection bias.

Here, we argue that RDDs have potential to add to the existing repertoire of methods for evaluating vaccines, especially newly introduced vaccines. A 2018 review of quasi-experimental designs for evaluating vaccines mentioned the RDD just briefly (9). A 2015 review of primary research found that only one of more than 30 publications applying RDDs to research questions in medicine, epidemiology, and public health evaluated the impact of a vaccine (10). Since then, only a few other examples relevant to vaccine evaluation, described below, have been published, highlighting the need to consider this approach more broadly.

The RDD approach is particularly well-suited to evaluating vaccines when a vaccination program is designed to target a specific group defined by a continuous variable with a clear cutoff (for example, as noted, a group defined by age) that establishes eligibility for the vaccine, and this condition must be met. This is often the case when a vaccine is introduced as part of routine administration or a mass-vaccination campaign. Human papillomavirus (HPV) vaccine introduction is a recent example of a licensed vaccine with distinct cutoffs that determine vaccination eligibility. HPV vaccine is recommended routinely for adolescents aged 11–12 years in the United States and those aged 9–13 years or in grades 4–8 in Canada. Smith et al. used administrative health data to demonstrate the impact of HPV vaccine introduction on cervical dysplasia and anogenital warts (11) and on indicators of sexual behavior (12) in the 2 years after vaccine introduction among girls in grade 8 (and eligible for the vaccine) compared with those in grade 8 during the 2 years prior to vaccine introduction. They used birthdate, a continuous variable available in their database, as a proxy for grade. Many vaccines within many national and subnational vaccination programs worldwide have been introduced or recommended based on age cutoffs, so there are many potential opportunities for using an RDD to evaluate vaccines. An advantage of the RDD is that it allows evaluation based on concurrent, or nearly concurrent, outcomes in the comparison groups, so temporal trends would be similar in both vaccinated and unvaccinated groups.

Another context in which RDDs present an opportunity to evaluate vaccines is when the incidence of disease is quite low or when there is a long time lag until the development of the outcome. When incidence is low, it is infeasible to conduct large-scale clinical trials sufficiently powered to evaluate the impact of a new vaccine on disease endpoints. If a newly introduced vaccine is recommended for a specific age group, then those just above the cutoff and those below the cutoff could be followed forward in time, similar to the HPV vaccine example. However, the follow-up might be much longer than 2 years. Temporal trends affecting those on either side of the cutoff would likely be similar for both groups even in long-term follow-up.

Sample size is often an issue with RDDs, but such an approach is feasible even for rare diseases, if the outcome of interest can be assessed using routinely collected surveillance data. For instance, new meningococcal vaccines are typically evaluated based on immunogenicity alone given the low incidence of meningococcal disease caused by any single serogroup. In 2015, the United Kingdom became the first country to routinely recommend meningococcal B (MenB) vaccination as part of their infant vaccination program, and eligibility for the vaccine is based on birthdate. The change in incidence of MenB disease has been evaluated using a cohort study design (13). Many questions remain about MenB vaccines, including whether they might also protect against other serogroups causing disease. So far, investigating this question has been limited to immunologic studies based on surrogate endpoints (14). Similarly, a retrospective case-control study in New Zealand (15) and an ecological analysis in Norway (16) suggest that MenB vaccines, designed to protect against Neisseria meningitidis, might also contribute to declines in Neisseria gonorrhoeae; however, evidence is mixed. An RDD could be designed in a setting where MenB vaccines are recommended for teenagers (as in the United States) and at a time when uptake is sufficiently high to robustly evaluate the potential cross-protection offered by meningococcal vaccines, and build upon early evidence, demonstrating a novel application of RDD in vaccine research.

In the historic 1954 Salk polio vaccine trial, involving more than 1.8 million children (17, 18), the initial design called for students in grade 2 to be offered vaccine and compared with students in grades 1 and 3 who remained unvaccinated, thus resembling an RDD. However, the lack of blinding and the potential for bias resulted in the design being changed more than midway through recruitment to a design individually randomizing students in all 3 grades to vaccine or placebo (19). The change to a randomized trial was possible because the vaccine was not yet licensed and not yet recommended for use. An RDD design would be a feasible alternative to a randomized clinical trial in similar situations when randomization is not possible or where it is unethical, as is often the case once a vaccine is licensed.

One issue in using RDDs for evaluating vaccines is the potential for indirect effects, also called spillover effects, whereby those in the control group might benefit from the reduction in risk resulting from vaccinating the treatment group (20, 21). Generally, the RDD would estimate the effect of vaccination in those eligible compared with those who are ineligible for vaccination. The analogy to an individually randomized trial would yield an estimate of the direct effect, the reduction in risk of the outcome among the treatment group compared with the control group. (The definition of these terms differs from that in mediation analysis (1, 22).) Estimates of direct effects based on the difference estimators could be affected by indirect effects because incidence among those who are ineligible for the vaccine might be affected by the vaccinated individuals. When evaluating newly introduced vaccines in new settings, the indirect effects might be expected to be low. Sävje et al. (23) showed that if one erroneously assumes there are no indirect effects in a setting with limited or even moderate indirect effects, standard estimators are likely close to the direct effect if the sample is sufficiently large. Aronow et al. (24) showed that under a local randomization assumption, the difference in means estimator of an RDD as applied to subjects near the cutoff is unbiased for a regime-specific effect. This is equivalent to the average direct effect (25) for the subpopulation of subjects within the window near the threshold. Thus, if the indirect effects are low for a newly introduced vaccine, the RDD should give a reliable estimate. A collection of RDDs might eventually be used to estimate how the local treatment effect varies with level of vaccine coverage in different populations, but this would require more methodological research.

While RDDs estimate direct effects of vaccines, if overall effects are of interest, interrupted time-series designs are often used, especially when new policies are implemented very rapidly. An example is the introduction of rotavirus vaccination in Ghana in April 2012 (26). Moscoe et al. (10) pointed out that the interrupted time series can be interpreted as a subtype of the RDD, where calendar time is the assignment variable, and the cutoff occurs when the new policy is implemented.

As long as a cutoff defines eligibility, and individuals are essentially distributed at random around that cutoff, RDDs can be used to evaluate a vaccination policy on any outcome of interest. For example, Helleringer et al. (27) examined the effects of a mass-vaccination campaign on utilization of routine vaccination using RDD by comparing infants born just before the mass-vaccination campaign with infants born just after the campaign, who were, by definition, not exposed to the campaign.

RD cannot guarantee unbiasedness. Efforts need to be undertaken to systematically evaluate outcomes in the groups on both sides of the cutoff and assess whether assumptions are met (28). The two identifiability assumptions mentioned above can be checked by testing whether the density of the data is continuous around the threshold and whether the baseline covariates are balanced at the threshold, as one checks in a randomized trial. Smith et al. (29) suggested further to check whether a discontinuity in the probability of exposure at the cutoff exists and that the value of the variable determining the cutoff was not manipulated. Despite its limitations, the RDD can assess the “local causal treatment effect ‘at the threshold’” (3, p. 731; 4, p. 740) for a vaccination program in real time under real-world conditions without a randomized experiment, without withholding vaccination from any eligible group, and by taking advantage of the natural experiment created by implementation of the vaccination program. It should be used more frequently.