Clinical Endpoints for Evaluating Efficacy in COVID-19 Vaccine Trials.

Several vaccine candidates to protect against severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection or coronavirus disease 2019 (COVID-19) have entered or will soon enter large-scale, phase 3, placebo-controlled randomized clinical trials. To facilitate harmonized evaluation and comparison of the efficacy of these vaccines, a general set of clinical endpoints is proposed, along with considerations to guide the selection of the primary endpoints on the basis of clinical and statistical reasoning. The plausibility that vaccine protection against symptomatic COVID-19 could be accompanied by a shift toward more SARS-CoV-2 infections that are asymptomatic is highlighted, as well as the potential implications of such a shift.

Widespread use of safe and durably effective vaccines for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), especially in combination with multiple concomitant prevention strategies (1), would curtail the coronavirus disease 2019 (COVID-19) pandemic. Several vaccine candidates have entered or will soon enter phase 3 clinical trial testing (2–4). Regulatory approval of a SARS-CoV-2 vaccine will require demonstration of safety and clinical benefit in a placebo-controlled efficacy trial. Guidance from the U.S. Food and Drug Administration (FDA) recommends minimal phase 3 success criteria for approval of a vaccine: an estimated reduction in the primary endpoint of at least 50% in the vaccine group versus the placebo group, with the 95% CI providing assurance of at least a 30% reduction (5)—a benchmark consistent with the World Health Organization's Solidarity Vaccines Trial design (6). The FDA guidance also indicates that acceptable primary endpoints for approval could include SARS-CoV-2 infection, symptomatic infection (COVID-19), severe COVID-19 (5), or some combination of these.

We address 4 salient issues on study endpoints in COVID-19 vaccine efficacy trials. First, we propose a general set of clinical endpoints to facilitate a harmonized evaluation and comparison of the efficacy of vaccine candidates, overall and across relevant subgroups. Second, we consider the pros and cons of various endpoints for use as primary endpoints. Third, we recommend adequate follow-up of all participants to enable enhanced sensitivity regarding effects on severe COVID-19 as well as assessment of the longer-term vaccine effect on the set of endpoints, including an assessment of durability of protection. Fourth, we recommend including asymptomatic infection as a study endpoint, given that vaccine protection against COVID-19 could be accompanied by a shift toward more asymptomatic SARS-CoV-2 infections, a plausible outcome if the vaccine does not confer sterilizing immunity.

Set of Clinical Endpoints to Facilitate Harmonized Vaccine Efficacy Evaluation and Comparison

The interconnections and definitions of our proposed set of study endpoints in COVID-19 vaccine efficacy trials are described in Figure 1 (top and middle, respectively). Two of the endpoints—virologically confirmed symptomatic SARS-CoV-2 infection regardless of the severity of symptoms (COVID-19) and virologically confirmed SARS-CoV-2 infection with symptoms classified as severe (severe COVID-19)—will likely be universally used because they fit standard endpoints used in virtually all vaccine efficacy trials (7). Coronavirus disease 2019 is ascertained at presentation, whereas severe COVID-19 is ascertained at presentation and through a fixed schedule of postdiagnosis follow-up (Figure 1, bottom), which continues through the resolution of all symptoms, ensuring that all severe COVID-19 endpoints are distinguished from nonsevere COVID-19 endpoints. For the COVID-19 endpoint, per FDA guidance, outcomes can include fever or chills, cough, shortness of breath or difficulty breathing, fatigue, muscle or body aches, headache, new loss of taste or smell, sore throat, congestion or runny nose, nausea or vomiting, and diarrhea. Corresponding outcomes for severe COVID-19 include clinical signs at rest indicative of severe illness; respiratory failure; evidence of shock (on the basis of specific blood pressure thresholds); clinically significant acute renal, hepatic, or neurologic dysfunction; admission to an intensive care unit; and death. The FDA guidance acknowledges the need for adaptation of symptom-based case definitions for pediatric patients and for persons with respiratory comorbid conditions.

Figure 1.

Clinical endpoint relationships, definitions, and example sampling scheme for diagnosed COVID-19 cases.

BOD = burden of disease; COVID-19 = coronavirus disease 2019; NAAT = nucleic acid amplification test; PCR = polymerase chain reaction; SARS-CoV-2 = severe acute respiratory syndrome coronavirus 2. Top. Structural relationships among study endpoints in a COVID-19 vaccine efficacy trial. Middle. Study endpoint definitions. Bottom. Example schedule of disease severity monitoring and virologic sampling for COVID-19 cases, with data or sample collection beginning at COVID-19 diagnosis and extending past COVID-19 diagnosis, in a setting where frequent follow-up of confirmed cases can be assured. Participants diagnosed with virologically confirmed, symptomatic SARS-CoV-2 infection (COVID-19) enter a postdiagnosis sampling schedule to monitor viral load and COVID-19–related symptoms (types, severity levels, and durations). This follow-up continues through resolution of all symptoms, enabling distinction of the nonsevere and severe COVID-19 endpoints.

* Seroconversion is assessed via a validated assay that distinguishes natural vs. vaccine-induced SARS-CoV-2 antibodies.

† Alternatively, the asymptomatic infection endpoint can also include an RNA PCR positive test result obtained through testing regardless of symptoms (e.g., as a requirement for travel, return to school or work, or elective medical procedures) and follow-up to confirm that the participant remains asymptomatic.

‡ Timed to be as close to day 28 after symptom onset as possible.

Serologic assays to accurately detect anti–SARS-CoV-2 nucleocapsid antibodies, which would be elicited by naturally acquired infection but not by SARS-CoV-2 spike protein–based vaccination, have been developed and validated (8), providing the technology required to enable evaluation of efficacy against the infection endpoint. The nesting of endpoints and their partitioning into mutually exclusive and exhaustive categories aid in the interpretation of results (Figure 1 , top). Every infection endpoint is either a COVID-19 endpoint or an asymptomatic infection endpoint, and a harmonized analysis of these 3 endpoints can assess the overall vaccine effect on infection and the proportion of this effect on each component endpoint. Similarly, every COVID-19 endpoint is either a nonsevere or severe COVID-19 endpoint, and a harmonized analysis of these 3 endpoints can elucidate the proportion of the vaccine effect on each component endpoint.

An additional endpoint includes all COVID-19 cases and quantitatively differentiates severe from nonsevere COVID-19, which we call the burden of disease (BOD) endpoint (9). To understand the difference between the BOD and COVID-19 endpoints, consider that vaccine efficacy is commonly expressed as a relative reduction (vs. placebo) in the risk for becoming an endpoint “case.” This is mathematically equivalent to assigning a binary score to each trial participant based on their case status and expressing vaccine efficacy as the relative reduction in the mean endpoint score. The COVID-19 endpoint is scored as 0 for no disease and 1 for disease. The BOD endpoint score extends this by using 0 for no COVID-19, 1 for nonsevere COVID-19, and 2 for severe COVID-19.

Given the anticipated heterogeneity among phase 3 trial participants, consistent with established practices in clinical trials (10), it would be important to explore whether vaccine efficacy is generally consistent across subgroups with differing levels of pretrial risk for SARS-CoV-2 infection or COVID-19. Accordingly, for each of the core endpoints described in Figure 1, we advocate reporting point estimates and 95% CIs for vaccine efficacy for prespecified subgroups defined by factors that include sex assigned at birth; age; geographic location; race/ethnicity; and presence or absence of preexisting health problems, such as heart or lung conditions, severe obesity, or diabetes. Because the phase 3 trial designs are generally powered for assessing overall but not subgroup-specific vaccine efficacy, such summaries should be interpreted as hypothesis-generating rather than hypothesis-confirming. Moreover, for specific population subgroups of interest, such as women who become pregnant while participating in a phase 3 trial, the number of endpoint cases will likely be too small to support reliable assessments of vaccine efficacy from a single trial; meta-analyses of relevant phase 3 trials and postapproval studies may be warranted.

Pros and Cons of Different Endpoints for Use as the Primary Endpoints

From both a public health perspective and an individual perspective, prevention of severe COVID-19 is perhaps the most important clinical benefit expected from an effective vaccine. There is precedent (for example, dengue [11], influenza [12-14], pertussis [15], pneumococcal bacteremia [16], rotavirus [17], and varicella [18]) that many vaccines confer greater efficacy against severe disease than milder disease. However, severe COVID-19 constitutes a relatively small portion of COVID-19 cases, and incidence varies widely by age, underlying risk, and ethnicity (19–21), implying that statistical power to demonstrate adequate vaccine efficacy against the severe COVID-19 endpoint may be lower than that for an endpoint that includes reduction in nonsevere COVID-19. For that reason, the broader-encompassing endpoint of COVID-19 symptomatic disease is deemed an appropriate primary endpoint and has been selected as such for all 6 ongoing phase 3 trials (4) and for the Solidarity Vaccines Trial. Moreover, there is consensus to assess severe COVID-19 as a key secondary endpoint.

Given that detection of safety problems with vaccines is critically important, the statistical analysis plans of the trials use 2-sided 95% CIs for vaccine efficacy for each study endpoint, so the data analyses can detect evidence for a higher rate of any endpoint in the vaccine versus the placebo group. Vaccine efficacy is 1 minus the relative risk (vaccine to placebo) of an endpoint, such that data analysis results (for example, Figure 2) can be equivalently reported as point estimates and 95% CIs for the relative risk for each endpoint, with evidence of vaccine harm deriving from an inference of a relative risk greater than 1. Given the precedent that vaccines can increase the risk for severe disease in certain populations (22–25), analysis of the severe COVID-19 endpoint is of particular interest. In addition to the standard analysis that includes all randomly assigned participants and thus provides a valid answer with high confidence, supportive analyses that compare rates of the severe COVID-19 endpoint between vaccine and placebo COVID-19 endpoint cases are also recommended. These latter analyses compare groups that were not randomly assigned and thus are susceptible to potential postrandomization selection bias (26); hence, the data analysis should adjust for baseline prognostic factors of severe COVID-19 and include sensitivity analyses to assess the robustness of results to potential postrandomization selection bias.

Figure 2.

Hypothetical example of results of a COVID-19 vaccine efficacy trial with 2:1 (vaccine–placebo ratio) randomization, with the analysis done for 147 total COVID-19 cases.

BOD = burden of disease; COVID-19 = coronavirus disease 2019; SARS-CoV-2 = severe acute respiratory syndrome coronavirus 2. Top. Number of uninfected and infected participants in each group, along with breakdown by endpoint for infected trial participants. Bottom. Vaccine efficacy point estimates and 95% CIs against 6 clinical endpoints. The black, dashed vertical line in the forest plot marks the lower 95% confidence bound of 30% given in guidance from the U.S. Food and Drug Administration.

The BOD endpoint, which encodes severe disease as being worse than nonsevere disease, may be viewed by some as more informative than the COVID-19 endpoint. Moreover, it can provide increased statistical power if the vaccine provides greater protection against severe than nonsevere COVID-19. Table 1 shows this for a 2:1 (vaccine to placebo ratio) randomization trial design, with 147 cases required to ensure 90% power for rejecting a null hypothesis of vaccine efficacy against COVID-19 being at most 30% assuming the true vaccine efficacy is at least 60% and the simple conditional binomial method described in the Supplement (available at Annals.org) is used for analysis. (A corresponding 1:1 randomization design could also be considered, with 160 COVID-19 cases required to ensure 90% power.) The results show that if vaccine efficacy against COVID-19 and severe COVID-19 is at least 60%, then both the COVID-19 and BOD endpoints provide about 90% or better power to meet FDA success criteria. However, if vaccine efficacy against COVID-19 is slightly lower at 55%, power decreases to 74% for the COVID-19 endpoint, whereas power for the BOD endpoint is between 75% and 92% if the vaccine confers a 60% to 90% relative reduction in severe COVID-19. Of note, if vaccine efficacy against severe COVID-19 is at or below 30%, BOD will be notably less powerful than COVID-19. A potential option is to use BOD and COVID-19 as dual primary endpoints with a statistical adjustment for associated multiple testing. Although the power of this strategy can be about as good as or notably better than that achieved with COVID-19 as the sole primary endpoint across all of the scenarios in Table 1, the clinical relevance of statistical success for a vaccine associated with moderate protection against nonsevere COVID-19 but poor protection against severe COVID-19 is unclear. Statistical details on methods used for the illustrative power calculations and for the multiplicity adjustment that leverages the correlation between the COVID-19 and BOD endpoints are provided in the Supplement.

Table 1. Comparison of Statistical Power for Different Efficacy Endpoints*

We believe that the BOD endpoint adheres to regulatory guidelines for suitability as a primary or key secondary endpoint for phase 3 trials, including clinical relevance in upweighting severe disease endpoints and sensitivity to detect a meaningful intervention effect validly assessed on the basis of the randomization principle (27). However, the BOD endpoint has 3 potential limitations. First, unlike for the COVID-19 endpoint, vaccine efficacy against BOD cannot be expressed as a percentage reduction in the risk for becoming an endpoint “case,” hampering interpretability for some. Second, consensus is lacking about the best way to score the degree of severity of COVID-19 events; specifically, there is a question about how each unit increment in the score relates to the corresponding increment in clinical significance. A third potential limitation, shared with the severe COVID-19 endpoint, is that follow-up after COVID-19 diagnosis is needed to ascertain some severe COVID-19 endpoints, whereas COVID-19 can be more rapidly and easily assessed, as defined at presentation. Regardless of whether BOD is used as a primary or key secondary endpoint, a separate efficacy evaluation for each of the COVID-19 incidence and severity components embedded in the composite BOD endpoint is necessary for clinical interpretability.

Given that asymptomatic infection is relatively common (28), infection cases would be expected to occur at a notably higher frequency than symptomatic disease. However, although prevention from acquiring SARS-CoV-2 infection (whether that means true sterilizing immunity or absence of detectable antibodies to the invading virus) has its advocates, this is considered a high bar to achieve (29, 30). A vaccine that protects against disease and confers limited protection against infection would still provide great clinical benefit and overall utility.

Importance of Longer-Term Follow-up of All Participants

We recommend longer-term follow-up of all participants (that is, after occurrence of 147 COVID-19 endpoints for a 2:1 randomization trial design) in a double-blind manner, if feasible, for at least a year after randomization or until deployment of a vaccine that has been proven effective in each participant's geographic region (6) for the following reasons. First, as described earlier, severe COVID-19 constitutes a relatively small portion of COVID-19 cases (19–21), suggesting that an efficacy evaluation for severe COVID-19 will likely be underpowered in an analysis based on 147 COVID-19 events. The data shown in Table 1 confirm this expectation. Thus, longer-term follow-up would enhance the sensitivity of detection of vaccine effects on severe COVID-19. Second, if a candidate vaccine is proven to be safe and found to have efficacy against 1 or more clinical endpoints, effective planning of mass immunization campaigns and strategies will require knowledge of the duration of such protection. Waning vaccine efficacy against a clinical endpoint has been documented in randomized controlled trials of several vaccines, including the RTS,S/AS01 malaria vaccine (31) and 2 similar killed whole-cell oral cholera vaccines (32). Waning vaccine efficacy against a clinical endpoint has also been suggested by observational case–control studies of several vaccines, including the influenza vaccine (33). In addition to improving insights about durability of effects, extended follow-up provides needed insights about whether a vaccine could make COVID-19 more hazardous, referred to as disease enhancement. Thus, longer-term follow-up as defined earlier would also provide important information on duration of the vaccine effect on each of the endpoints defined in Figure 1.

Inclusion of Asymptomatic Infections as a Study Endpoint

Because each infection endpoint is also either an asymptomatic infection endpoint or a COVID-19 endpoint (Figure 1), vaccine efficacy levels for infection and for COVID-19, together with an assumption of the expected percentage of infections in the placebo group that will be asymptomatic, determine the vaccine efficacy for asymptomatic infection. Table 2 shows this mathematical calculation if 40% of the infections in placebo recipients are expected to be asymptomatic (28). Under this expectation, if a vaccine reduces SARS-CoV-2 infections by a modest amount, say 20%, but reduces symptomatic infections by an impressive 70%, the net result will be a 55% increase in asymptomatic infections. In such a scenario, although the vaccine would likely meet success criteria if COVID-19 (or BOD) were the sole primary endpoint and would provide direct clinical benefit to individual vaccine recipients, it may have different population-level effects on the number of secondary transmissions (34). If vaccine recipients acquiring asymptomatic infection have low viral infectivity and are weak transmitters, then the vaccine in the aforementioned scenario would likely confer public health benefit on transmission. However, viral shedding data (35, 36) and epidemiologic and modeling studies (37) suggest that unvaccinated asymptomatic persons can transmit the virus (38, 39), raising questions about the transmission potential of vaccine recipients who acquire asymptomatic infection. A vaccine that essentially converts symptomatic to asymptomatic infections, without also decreasing viral shedding and transmission potential, may paradoxically increase transmissions given that viral testing and isolation are presumably less likely in asymptomatic persons. Conversely, if the vaccine decreases both symptomatic infections and transmission potential, then the population benefits may be magnified. Acknowledging the tenuous link in extrapolating results from nonhuman primate studies to humans, we note that challenge studies of rhesus macaques vaccinated against SARS-CoV-2 (vs. unvaccinated controls) offer evidence that either possibility may occur ([40] and [41], respectively). Designing phase 3 trials to evaluate a vaccine effect on asymptomatic infections, which requires collecting blood from all participants periodically for serology testing or swabs for virology testing (along with incorporating symptom-triggered testing for virologically confirmed SARS-CoV-2 infection to ensure that any presymptomatic persons who have no symptoms at the time of a positive virologic test result, but eventually do develop symptoms, will be correctly classified as having a COVID-19 endpoint rather than an asymptomatic infection endpoint) will be an important way to carefully explore this issue.

Table 2. Interconnection Among VE Against SARS-CoV-2 Infection, Symptomatic SARS-CoV-2 Infection (COVID-19), and Asymptomatic SARS-CoV-2 Infection

Figure 2 shows recommended forest plot reporting of vaccine efficacy results for the proposed core study endpoints for a hypothetical phase 3 clinical trial of a COVID-19 vaccine (see the Supplement for calculation details). Here, success criteria would be met if COVID-19, severe COVID-19, or BOD were the sole primary endpoint. Vaccine efficacy is low for preventing infection. The negative vaccine efficacy against asymptomatic infection can be reasonably interpreted as the vaccine having shifted infections that would have been symptomatic without vaccination to asymptomatic infections. Although an alternative interpretation that the vaccine at least partially caused more asymptomatic infections that would not have occurred without vaccination cannot be ruled out, it would be deemed less biologically plausible given the promising vaccine efficacy results for the COVID-19, severe COVID-19, and BOD endpoints.

The fact that a vaccine with low efficacy against acquisition of SARS-CoV-2 infection and moderate to high efficacy against symptomatic infection could result in a net increase in asymptomatic infections highlights the need to understand the effect of asymptomatic infections on transmission of COVID-19 (42). The contribution of asymptomatic carriers to transmission dynamics has been modeled for many infectious diseases, including influenza (43) and Bordetella pertussis (causative agent of whooping cough). For the latter, modeling analyses have considered the plausible situation where vaccination with the acellular B pertussis vaccine prevents symptomatic disease but does not prevent asymptomatic transmission, and concluded that such a scenario could explain the observed increase in B pertussis incidence during the past few decades (44). For SARS-CoV-2, modeling of the effect of a vaccine that reduces or prevents COVID-19 but increases asymptomatic infections entails biological issues (for example, the relative rates of asymptomatic vs. symptomatic infection, the relative transmissibility of asymptomatic vs. symptomatic infection, the relative vaccine effect on secondary transmission for asymptomatic vs. symptomatic infection, how this relative transmissibility differs in unvaccinated vs. previously vaccinated infected persons, and how well this relative transmissibility can be measured by a viral load putative surrogate marker) and behavioral issues (for example, the rapidity and completeness of isolation after COVID-19 diagnosis, the coverage of SARS-CoV-2 testing in communities after vaccine approval, and the coverage of vaccination). Under very high rates of vaccine coverage, increased numbers of asymptomatic infections may be innocuous because any excess transmissions will tend to be toward vaccinated persons who are well protected against severe disease. In contrast, in communities with low vaccination coverage, excess asymptomatic infections with transmission potential could increase the number of severe COVID-19 cases because the excess transmissions will tend to be to unvaccinated persons who are fully vulnerable to severe COVID-19. We anticipate that recommendations about how a vaccine should be used in communities will likely be informed not only by the vaccine efficacy against infection or disease estimated in the clinical trials but also by estimates of community effect predicted by transmission models. Duration of vaccine efficacy against the core set of study endpoints will be a key parameter influencing the community impact.

Conclusion

A primary endpoint should be clinically meaningful (27), sensitive, and specific. A simple designation, such as reducing COVID-19 disease severity, seems uncomplicated, but it masks potential interpretability and misclassification concerns. The COVID-19, severe COVID-19, and BOD endpoints depend on sets of prescribed symptoms, and myriad definitions could be used to specify these sets. Although the FDA recommends defining the COVID-19 endpoint as virologically confirmed SARS-CoV-2 infection accompanied by 1 or more of 11 symptoms (5), trialists have latitude to select particular symptoms and severities needed to trigger virologic testing. It is important to define a common COVID-19 endpoint that can be used consistently across trials, both for interpretation of results and for facilitation of meta-analyses of trials.

Because several candidate COVID-19 vaccines are entering phase 3 testing, it is important to adopt a standard set of clinical endpoints for vaccine efficacy evaluation across all of the trials to provide uniform, comprehensive evaluations of benefit and risk and to support pooling data for analyses of immunologic surrogate endpoints. We have proposed a core set of primary and secondary clinical endpoints. The designation of endpoints as primary versus secondary may differ by vaccine candidate because some may have a greater preevaluation probability of blocking SARS-CoV-2 acquisition, which could justify including infection as a primary endpoint. Regardless, we propose that COVID-19 and severe COVID-19 should be important standalone clinical endpoints to assess in every vaccine efficacy trial, with adequate follow-up of all participants to accumulate meaningful data for assessment of longer-term protection against both endpoints, especially severe COVID-19, which needs greater endpoint counts for a reliable quantification of vaccine efficacy. Moreover, we have noted reasons for considering the BOD endpoint as a (dual) primary or key secondary endpoint if the vaccine candidate is expected to work best against severe COVID-19 and at an intermediate level against nonsevere COVID-19. In such scenarios, the BOD endpoint will provide improved power relative to COVID-19, although this gain may be offset by a potential detriment in clinical interpretability associated with this composite endpoint. Finally, given that a desired, vaccine-induced decrease in the incidence of symptomatic SARS-CoV-2 infections may be accompanied by a shift toward more asymptomatic infections, we recommend including the ability to evaluate vaccine efficacy against the asymptomatic infection endpoint in trial designs.

Click here for additional data file.