Interim estimates in null models of COVID-19 vaccine effectiveness.

Recently released interim numbers from advanced vaccine candidate clinical trials suggest that a COVID-19 vaccine effectiveness (VE) of >90% is achievable. However, SARS-CoV-2 transmission dynamics are highly heterogeneous and exhibit localized bursts of transmission, which may lead to sharp localized peaks in the number of new cases, often followed by longer periods of low incidence. Here we show that, for interim estimates of VE, these characteristic bursts in SARS-CoV-2 infection may introduce a strong positive bias in VE. Specifically, we generate null models of vaccine effectiveness, i.e., random models with bursts that over longer periods converge to zero VE but that for interim periods frequently produce apparent VE near 100%. As an example, by following the relevant clinical trial protocol, we can reproduce recently reported interim outcomes from an ongoing phase 3 clinical trial of an RNA-based vaccine candidate. Thus, to avoid potential random biases in VE, it is suggested that interim estimates on COVID-19 VE should control for the intrinsic inhomogeneity in both SARS-CoV-2 infection dynamics and reported cases.

In the international race for vaccines against COVID-19 significant progress has been claimed recently, with Pfizer Inc. (New York, NY, USA) reporting an interim analysis from their current phase 3 clinical trials of an RNA-based vaccine candidate (Pfizer, 2020, Polack et al., 2020). From this placebo-controlled, randomized and observer-blind study, vaccine effectiveness (VE) of >90% has been reported based on a preliminary number of 94 confirmed cases of symptomatic SARS-CoV-2 infections accrued over 104 days (between July 27 and November 8, 2020). While these and other interim numbers (Callaway, 2020) are encouraging for potential outcomes from these controlled clinical trials, long-term and thus more realistic VE estimates, will depend on SARS-CoV-2 transmission features that are intrinsically more difficult to control over the short-term.

One such prominent feature is the heterogeneity in intervals between consecutive SARS-CoV-2 infections, which over time leads to highly localized and seemingly random clusters (or bursts) of recorded cases, followed by longer periods of relative inactivity (Adam et al., 2020). This heterogeneity is potentially due to underlying superspreading events that are stratified from highly localized (e.g., household) to less localized levels (such as entire communities, see Liu et al., 2020) and that appear to be driven mainly by symptomatic transmission (Kumar et al., 2021). Statistical evidence for SARS-CoV-2 transmission heterogeneity has been given in the number distribution of secondary cases during superspreading events (Wong and Collins, 2020), as well as in serial interval distributions (Du et al., 2020). SARS-CoV-2 serial intervals often follow a log-normal distribution in which the measured mean μ can be smaller (Du et al., 2020) or larger (Nishiura et al., 2020) than the standard deviation σ. This observation is relevant because it allows a classification according to the burstiness parameter B  = (σ–μ)/(σ + μ), known for various complex dynamical systems (Goh and Barabási, 2008). These systems are characterized by intermittent, heterogeneous time series whenever 0 ≤ B  < 1; for the opposite range, –1 <  B  < 0, the dynamics are fundamentally ordered in time, more homogeneous and predictable. The log-normal distribution, which theoretically covers the entire range of B between the 2 extremes –1 and 1, was proposed to be general and found in the statistics of many complex dynamical systems that exhibit burstiness, including the dynamics of viral infections (Goh and Barabási, 2008).

To test if actual SARS-CoV-2 cases exhibit burstiness, we analyzed the distribution of time intervals in globally recorded SARS-CoV-2 cases from public data repositories that monitor the current pandemic (Supplementary material, Figure S1A and Methods). The analysis indicated that these intervals follow a log-normal distribution and that burstiness is present with a mean μ = 1.2 days, a standard deviation of σ = 3.9 days, and positive B = 0.5. A direct consequence of this observation is that if, within a background population, SARS-CoV-2 infection dynamics are dominated by bursts (B > 0), any random subgroup would also exhibit a similar heterogeneity level. We verified this prediction by generating a series of random group samples from the original set of cases at decreasing sample size, followed by estimating the resulting degree of heterogeneity B. The data showed that the originally observed high level of burstiness (B = 0.5) was robust and did not decrease after such subsampling (Figure S1B). This evidence, in turn, suggests that burstiness itself could become a significant source of random bias for interim estimates of VE. Specifically, as the log-normal distribution implies, one can construct an elementary class of null VE models with burstiness, i.e., statistical control models that over time converge to zero VE, but which at shorter interim times frequently produce apparent VE values close to 100%.

To see this effect of burstiness, consider 2 placebo (no vaccine given) randomized groups with an equal number N 1 = N 2 = N of initially healthy (no COVID-19) individuals, and let a be the yearly SARS-CoV-2 attack rate such that the expected total number of cases after one year becomes C max = 2aN. Then use the log-normal distribution to generate a random sequence of new cases for each group such that after 1 year the accumulated number of cases in each group becomes C max/2, thus over time leading to zero VE. Finally, for all number of days t before one year, calculate the interim vaccine effectiveness, VE(t) = 1 – C 2(t)/C 1(t), where C 1(t) and C 2(t) are the accumulated case numbers in group 1 and 2, respectively. Figure S2 gives a representative output of this procedure with input values a = 1.3% and N = 21999, as per Pfizer’s published clinical trial protocol (Supplementary material Document S1). For this model realization, a conservative value of the burstiness parameter was set at B = 0.4, which in the absolute range of B was reduced by one-tenth from the originally observed high degree (B = 0.5). In the resulting model run, between days 48 and 90, the interim VE reaches above 90%, with 94 or more recorded cases that accumulate in several random bursts. Direct model sampling with the above parameter settings suggests that this vaccine test requirement (VE > 90%, and C 1(t) + C 2(t) ≥ 94) is already met with a relative frequency of 2% (see Supplementary material, Methods section), which further increases monotonically as B increases. In contrast, for an equivalent model but with a symmetrically opposite, low degree of burstiness, B = − 0.4, cumulative case numbers increase steadily, large interim fluctuations in VE do not occur, and overall VE remains close to zero (Figure S3). Thus, as an interim effect, high levels of COVID-19 VE can be produced by random bursts alone, i.e. without any immunization background.

These results suggest that advanced COVID-19 vaccine candidate clinical trials should explicitly address the potential inhomogeneity in recorded SARS-CoV-2 cases when releasing VE data. In addition to the burstiness that characterizes SARS-CoV-2 transmission, recording every positive case during an advanced clinical trial is also a random and variable process that depends on several external stochastic factors (Figure S4). This process can extend over many days (Figure S4), from a trial participant’s symptom onset to a standard clinical test result received and documented after a variable turnaround time (Chwe et al., 2020). It is then, for example, difficult to bring in line these stochastic factors, which would only further amplify the inhomogeneity in recorded case numbers, with the “steadily accumulating cases” in the placebo group of Pfizer’s phase 3 clinical trial that remarkably show no significant heterogeneity during nearly the first 100 days (see, Figure 13 in Supplementary material Document S2, and Figure 3 in Polack et al., 2020). Thus, in contrast to our observations (B > 0), Pfizer’s phase 3 clinical trial data (Polack et al., 2020; Document S2) points to the absence of bursts in accrued COVID-19 cases (B < 0). Consequently, to avoid artificially inflated VE in the more likely situation where case number heterogeneity does occur, cumulative incidence numbers should be released after trial periods long enough to ensure that additional clusters of cases (bursts) do not cause strong fluctuations in VE estimates. In the context of ongoing phase 4 confirmatory trials, such additional control might also help ensure that high levels of COVID-19 VE observed over shorter periods persist after extended periods that would ultimately be necessary to stop a pandemic.

Statements

The author (AML) declares that there are no conflicts of interest.

The author (AML) declares that he received no specific funding for this work.

The author (AML) declares that no ethical approval was required for this work.