Assurance in vaccine efficacy clinical trial design based on immunological responses.

The assurance of a future clinical trial is a key quantitative tool for decision-making in drug development. It is derived from prior knowledge (Bayesian approach) about the clinical endpoint of interest, typically from previous clinical trials. In this paper, we examine assurance in the specific context of vaccine development, where early development (Phase 2) is often based on immunological endpoints (e.g., antibody levels), while the confirmatory trial (Phase 3) is based on the clinical endpoint (very large sample sizes and long follow-up). Our proposal is to use the Phase 2 vaccine efficacy predicted by the immunological endpoint (using a model estimated from epidemiological studies) as prior information for the calculation of the assurance.

INTRODUCTION

Decision‐making is a key problem in drug development, and it has been shown that statistical power is not sufficient to guarantee the success of any clinical trial as it does not take into account the uncertainty of the prespecified (vaccine) effect.

Spiegelhalter and Freedman (1986) proposed to take into account this uncertainty using a hybrid “classical‐Bayesian” approach called “Bayesian predictive power” (Spiegelhalter et al., 2004, Section 6.5). Their proposed approach can be thought of as a weighted average of power across the range of possible values of effect sizes. For this reason, this probability has also been referred to as the “expected power” or “average power” (Gillett, 1994). It is called as well “Probability of Success” or “Predictive Probability of Success.” More recent work on this topic can be found in Carroll (2013) and Ibrahim et al. (2015). In this paper, we will refer to this concept as assurance (O'Hagan et al., 2005).

The prior distribution of an unknown effect can be estimated from a Phase 2 study (O'Hagan et al., 2005). However, in vaccine development, this is not always possible as early‐phase clinical trials (dose selection, dose ranging, coadministration with other vaccines, etc.) are often based on the response observed by an immunological assay, while the clinical endpoint (in this paper, we will consider as an example the infection rate) is not known before the later phases of the development (large Phase 3 trials). In such cases, Phase 2 trials do not provide a direct estimation of the Vaccine Efficacy (VE) on the clinical endpoint where and are the probabilities of infection among vaccinated and unvaccinated individuals, respectively. It is therefore not possible to use standard formulas (such as O'Hagan et al., 2005) to compute the assurance of the Phase 3 trial.

Ideally, the immunological endpoint used for early development is a “surrogate endpoint” (which is called “correlate of protection” in vaccines, WHO, 2013). Qin et al. (2007) proposed the following definitions: (a correlate of risk (CoR) is an “immunological measurement that correlates with the rate or level of a study end point used to measure VE in a defined population,” and a surrogate endpoint is a “CoR that reliably predicts a vaccine's level of protective efficacy on the basis of contrasts in the vaccinated and unvaccinated groups' immunological measurements.” Many statistical methods have been published for the validation of surrogate endpoints: methods based on a single VE trial (such as Prentice criteria, Prentice, 1989; and the principal stratification, Follmann, 2006; Gilbert et al., 2008) and methods based on multiple VE trials (such as meta‐analysis, Buyse et al., 2000). The meta‐analytical approach is considered the most appropriate approach because a relationship between endpoints estimated from a single trial may be insufficient to support predictions across trials (Baker & Kramer, 2003; Burzykowski et al., 2005; Buyse et al., 2000). There is an intense recent discussion of the use and validation of surrogate endpoints in clinical literature (more than 4000 PubMed citations of “surrogate endpoint”). As a recent example in vaccines, Callegaro and Tibaldi (2019) considered the validation of surrogate endpoints in case of large VE.

If the relationship between endpoints is available (even if the surrogate endpoint is not validated for regulatory purpose), then it is possible to compute the assurance using the surrogate endpoint predicted VE as prior. For example, Saint–Hilary et al. (2019) proposed to compute the assurance using the meta‐analytical models.

Unfortunately, for vaccine development (as we said previously), the data necessary to estimate the relationship between the two endpoints (single or multiple VE trials) are usually not available.

In this manuscript, we propose to compute the assurance using as prior the VE predicted by a CoR model (Chan et al., 2000; Dunning, 2006; Dunning et al., 2015) on a Phase 2 immunological study. The CoR model, proposed by the seminal work of Dunning, is a model relating the probability of infection with the value of the immunological assay and can be estimated from epidemiological studies or known from the literature (see, for example, Storsaeter et al., 1998).

The CoR model (relationship between endpoints in the control group) can be used to predict the VE under the assumption that the immunological endpoint is a “statistical surrogate” (Prentice, 1989; Qin et al., 2007). We provide as well two extensions of the CoR model (where additional parameters come from prior elicitation of experts in the field; Dallow et al., 2018) that can be used to compute the assurance when the surrogate is not fully mediating the vaccine efficacy.

STATISTICAL METHODS

Notations and definitions

In this section, we introduce the basic notations and definitions that will be used throughout the paper. For easy of notation, we are not using the hat operator to denote the estimated values.

Immunological Phase 2 trial

Let us consider a Phase 2 trial based on the immunological endpoint (denoted by the humoral or cellular immune readout value in the log10 scale for th subject). A classical measure of immunological vaccine effect is the geometric mean ratio (GMR), which is defined as: where and represent the sample sizes of the vaccine group (V) and the control group (C) group in the Phase 2 trial, respectively; and are the arithmetic means of the immunological readouts. We stress again that the infection rate is not observed in immunological Phase 2 studies (small sample size and short follow‐up).

Phase 3 VE trial

In a classical Phase 3 VE trial, the clinical endpoint of the study ( if the th subject is infected; otherwise) is compared between a vaccine group (V) and a control group (C). The VE is estimated by one minus the relative risk (RR) as follows: where and represent the sample sizes of the V and C groups of the Phase 3 study, respectively.

A classical null hypothesis of a Phase 3 VE trial is rejected when the lower limit of the % confidence interval of the Vaccine Efficacy (VE) is above the superiority margin . As a measure of vaccine effect, we use the log of the relative risk: = and assume asymptotic normality with variance (Katz et al., 1978) where and represent the expected number of infections in the C and in the V group (under the null hypothesis) in the Phase 3 trial, respectively.

Correlate of risk model

We denote by CoR model a statistical model relating the probability of infection with the value of the immunological assay (Dunning, 2006; Dunning et al., 2015). For simplicity, we consider the CoR logistic model We assume that the joint distribution of the parameters of the CoR model is known. As previously mentioned, these can be estimated from epidemiological studies measuring the clinical and the immunological endpoint, or can be retrieved from the literature (see, for example, Storsaeter et al., 1998).

Predicted VE

If the immunological endpoint is a “statistical surrogate” (Prentice, 1989; Qin et al., 2007), then the CoR model is expected to predict (without bias) the VE. The predicted VE (Chan et al., 2000; Dunning, 2006) is then given by where is the logit of the predicted probability of infection for the th subject with immunological endpoint . The variance of the predicted VE can be estimated by bootstrap (Chan et al., 2000). We derived the asymptotic formula as well, which can be used for the logistic and the scaled logit model (Dunning, 2006) (see the Appendix). This formula can be applied when the parameters (and their covariance matrix) are retrieved from the literature, and the individual data used to estimate the CoR model are not available.

Assurance

Statistical power is defined as the probability of rejecting the null hypothesis conditioned to the true prespecified treatment effect. In practice, however, this true effect is not known with certainty, but it has been estimated from previous trials. To incorporate the uncertainties of this observed treatment effect (prior distribution), Bayesian insurance has been proposed as an alternative to conventional statistical power. Following the notations from O'Hagan et al. (2005), we denote as the event of rejecting the null hypothesis in the Phase 3 trial. The conventional definition of power is , where is the vaccine effect. The assurance is the expected power as follows where is the prior distribution of the vaccine effect. We assume that the prior is normally distributed , which can be derived from a Phase 2 trial ().

For Phase 3 VE trials (as described above), the assurance (O'Hagan et al., 2005) is given by where is the “known” variance of the Phase 3 trial, is quantile of the normal distribution, and represents the superiority margin.

Assurance based on immunological predicted VE

Using the CoR model, it is possible to predict the in a Phase 2 immunological study: where is the logit of the predicted probability of infection given the immunological endpoint of the th subject of the study.

The predicted in the Phase 2 trial can be used as a prior of the and the assurance can be computed using Equation (2) with and .

Assurance based on simulations

It is possible to compute the assurance by simulating Phase 3 trials using the CoR model and the distribution of the immunological data estimated in Phase 2. The simulation is performed by repeating times the following four steps: (1) simulation of the individual Phase 3 immunological endpoint () by treatment group from the normal distribution estimated in the Phase 2 trial ( and ); (2) simulation of the CoR parameters ) from the multivariate normal distribution ; (3) simulation of the individual Phase 3 clinical endpoint () using the CoR model with parameters and immunological data () simulated in steps (1) and (2), respectively; (4) computation of the VE (VE, ) and the corresponding confidence interval.

The assurance is the proportion of simulated VE with the lower bound of the confidence interval being above the superiority margin .

Extended CoR model

In the previous sections, we assumed that the immunological variable is a “statistical surrogate.” According to the Prentice framework (Prentice, 1989), the model estimated in epidemiological studies (or in the control group) should allow both the prediction of the risk in the C group and in the V group. However, since we do not know if is a statistical surrogate, it is possible (for different biological reasons) that the model is not capable predicting the risk in the V group. As the model generating the vaccinated clinical data is unknown, we consider an extension of the CoR model including two additional unknown parameters where is the vaccine effect not explained by and is the interaction between the vaccine effect and . The covariance matrix is given by which can be rewritten as where the 2 2 matrices and are unknown. The additional parameters are useful for considering different scenarios. For example, means that is a statistical surrogate endpoint (Prentice criteria met). means that the surrogate endpoint explains only a proportion of the vaccine efficacy (Freedman et al., 1992), while the interaction term means that the vaccine induced immunological endpoint is more (or less) protective than the natural immunological endpoint.

The predicted VE is given by The choice of (and the corresponding matrices , ) can be derived from a prior elicitation of experts in the field (see, for example, Dallow et al., 2018).

Simplified extended CoR model

The extended model described above is flexible, but it can be challenging to elicitate all the additional parameters. For this reason, we consider a simplified version of the model means that is a statistical surrogate endpoint (Prentice criteria met), while () means that the vaccine‐induced surrogate endpoint is more (less) protective than the naturally acquired surrogate endpoint.

If we make the assumption that , then the assurance is given by where and are the prior parameters estimated from the Phase 2 under the assumption that is a statistical surrogate.

RESULTS

For illustration purposes, we consider a hypothetical case study where the CoR model is estimated from a large epidemiological study and data from a small immunological Phase 2 study are available.

We assumed that the parameters of the CoR model were , with and with elements , , and .

Regarding the immunological Phase 2 study, we assume that ; the estimated mean and variance of the immunological endpoint by treatment group were given by and , respectively.

The sample size of Phase 3 vaccine trial was approximately 12,000 ( with randomization ratio 1:1) for a disease with = 1%, 90% power (assuming VE of 50%) and two sided of 5% (the null hypothesis is rejected if the lower limit of the % confidence interval of the VE is above VE).

CoR model ( is a “statistical surrogate”)

Using the CoR model, we estimated the Phase 2 predicted and its variance (see the Appendix for the calculation in “Variance of the Predicted VE (Delta Method)”).

The assurance was then calculated using Equation (2). With the assumption that and the Phase 2 prediction as prior , an assurance value of (based on Equation (2)) was obtained. A similar result () was obtained by computing the assurance by simulations.

Extended CoR model

The assurance value of was obtained by making the strong assumption that Prentice criteria are met. In the following, we relax this assumption by using the unknown parameter of Equation (3). These parameters are unknown and can be derived from prior elicitation. For simplicity, we assume that and that . For illustration, we considered three different scenarios.

In scenario A, we assume that , and we consider two different covariance matrices: (scenario A1—no uncertainty about the fact that is a “statistical surrogate”) and (scenario A2‐–uncertainty about the fact that is a “statistical surrogate”).

In scenario B, we assume that explains only a portion of the vaccine effect (). Note that this scenario corresponds to a surrogate endpoint explaining about 50% of the vaccine efficacy.

In scenario C, explains 100% of the vaccine effect (), but the vaccine induced immunological endpoint is less protective than the natural immunological endpoint ().

Table 1 shows how the predicted VE and the assurance values changes as a function of the unknown parameters .

TABLE 1

Extended CoR model: predicted VE (VE) and assurance () as a function of the unknown parameters ()

Scenario	μ_η	Σ_η,η	VE_pred,2(η)	γ(η)
A1	(0,0)	I_2*0.001	0.45	0.75
A2	(0,0)	Σ/3	0.38	0.66
B	(−0.5,0)	Σ/3	0.63	0.89
C	(0,0.035)	I_2*0.001	0.37	0.62

We can see from Table 1 that the unknown parameters play a major role on the computation of the assurance. The first row corresponds to the case where is a statistical surrogate, and we can see that the assurance value is close to the value obtained using the CoR model. The second row corresponds to the case where we have some uncertainty about the fact that is a statistical surrogate. As the uncertainty increases the value of the assurance decreases. The third row shows the situation where explains only a portion of the VE. In this case, since is equal to −0.5, the vaccine is effective, whatever the value on the surrogate endpoint. Therefore, the predictive probability of success in Phase 3 is high, independently of the value on the surrogate endpoint. The last row shows how the assurance value decreases when the vaccine‐induced immunological endpoint is less protective than the naturally acquired immunological endpoint.

Simplified extended CoR model

Figure 1 shows the assurance calculated using the simplified extended CoR model (Equation (4)) under the assumption that as a function of and . We can see that when the surrogate is a statistical surrogate ( and ), the assurance value is about 0.75. This value is closer to 50% (maximum uncertainty) when there is more uncertainty about the unknown parameter (larger values of ). Furthermore, we can see how the assurance depends on . As expected, the assurance increases (decreases) when the vaccine‐induced surrogate endpoint is more (less) protective than the naturally acquired surrogate endpoint.

FIGURE 1

Case study: assurance as a function of and (simplified extended CoR model)

CONCLUSIONS

The assurance is the expectation of the power, averaged over the prior distribution for the unknown true vaccine effect. The prior distribution of the parameter of interest could be considered as the vaccine effect estimated in an existing Phase 2 trial (O'Hagan et al., 2005). Our work is dedicated to describing how to compute the assurance for vaccines, where early development (Phase 2) is often based on immunological endpoints (e.g., antibody levels) and the confirmatory trial (Phase 3) is based on the clinical endpoint (e.g., infections). Our proposal is to use the Phase 2 vaccine efficacy predicted by the immunological endpoint as prior information. The prediction is done by using the CoR model that links the clinical and the immunological endpoint (Chan et al., 2000; Dunning, 2006; Dunning et al., 2015). Parameters of this model can be estimated using data from epidemiological studies (often conducted in early development) or retrieved from the literature (see, for example, Storsaeter et al., 1998).

The prior distribution of the VE is derived by making the assumption that the immunological endpoint is a “statistical surrogate” (Prentice, 1989; Qin et al., 2007). We provided as well two different extensions of the CoR model that can be used to derive the prior of the VE when the surrogate is not fully mediating the vaccine effect.

We derived a closed formula to compute the variance of the predicted VE when the CoR model is logistic or scaled logistic.

The proposed approach is similar in spirit to the work of Saint‐Hilary and colleagues (Saint–Hilary et al., 2019), where they compute the probability of success (assurance) using a surrogate endpoint. The main difference is that their prediction is based on meta‐analytical models (trial‐level), while our proposal is based on a particular individual‐level model (CoR model) that is estimated from epidemiological studies (where there are no vaccinated subjects). This choice was due to the fact that VE data are rarely available for early vaccine development. The limitation of this choice is that prediction based on individual data (single trial setting) may not be valid at the trial level (Baker & Kramer, 2003; Burzykowski et al., 2005; Buyse et al., 2000). Furthermore, the information elicited from experts to extend the CoR model may be wrong.

For simplicity, in this paper, we ignored the between‐trial variability. This additional variability (known or elicitated from experts) can be included in the CoR model. If multiple epidemiological studies are available, a random study effect should be included in the CoR model of Equation (1).

We assumed that the population of the epidemiological study is the same as the population in the control group of the randomized (small immunological Phase 2 and large efficacy Phase 3) studies. Even if the populations are the same, it is possible that there are some discrepancies between the epidemiological and the randomized immunological data. This problem could be mitigated by including covariates (such as age, sex, etc.) in the CoR model.

In conclusion, in this paper, we described some solutions to compute the assurance for vaccines where early phase clinical trials are often based on immunological data.

AUTHOR CONTRIBUTIONS

All authors contributed equally.

FINANCIAL DISCLOSURE

GlaxoSmithKline Biologicals SA was the funding source for all costs associated to the development and the publication of the present manuscript.

CONFLICT OF INTEREST

AC, TZ and FT are employees of and holds shares in the GSK group of companies.

OPEN RESEARCH BADGES

This article has earned an Open Data badge for making publicly available the digitally‐shareable data necessary to reproduce the reported results. The data is available in the Supporting Information section.

This article has earned an open data badge “Reproducible Research” for making publicly available the code necessary to reproduce the reported results. The results reported in this article could fully be reproduced.