Projected resurgence of COVID-19 in the United States in July-December 2021 resulting from the increased transmissibility of the Delta variant and faltering vaccination.

In Spring 2021, the highly transmissible SARS-CoV-2 Delta variant began to cause increases in cases, hospitalizations, and deaths in parts of the United States. At the time, with slowed vaccination uptake, this novel variant was expected to increase the risk of pandemic resurgence in the US in summer and fall 2021. As part of the COVID-19 Scenario Modeling Hub, an ensemble of nine mechanistic models produced 6-month scenario projections for July-December 2021 for the United States. These projections estimated substantial resurgences of COVID-19 across the US resulting from the more transmissible Delta variant, projected to occur across most of the US, coinciding with school and business reopening. The scenarios revealed that reaching higher vaccine coverage in July-December 2021 reduced the size and duration of the projected resurgence substantially, with the expected impacts was largely concentrated in a subset of states with lower vaccination coverage. Despite accurate projection of COVID-19 surges occurring and timing, the magnitude was substantially underestimated 2021 by the models compared with the of the reported cases, hospitalizations, and deaths occurring during July-December, highlighting the continued challenges to predict the evolving COVID-19 pandemic. Vaccination uptake remains critical to limiting transmission and disease, particularly in states with lower vaccination coverage. Higher vaccination goals at the onset of the surge of the new variant were estimated to avert over 1.5 million cases and 21,000 deaths, although may have had even greater impacts, considering the underestimated resurgence magnitude from the model.

Introduction

The rapid development, scale-up, and deployment of COVID-19 vaccines in the United States (US) has been one of the biggest public health successes in the US during this pandemic, with reported cases in a nadir in June 2021 (Centers for Disease Control and Prevention, 2021b), despite increased testing capacities. With this success, non-pharmaceutical interventions (NPIs) were lifted, including mask mandates, in almost every jurisdiction across the US in Spring 2021. However, the emergence of novel variants with increased transmissibility and /or immune escape, particularly the Delta and Omicron variants, has continually raised concern about the potential timing and magnitude of the subsequent resurgence, and the ability to mitigate it through increased uptake of vaccination.

Established in December 2020, the COVID-19 Scenario Modeling Hub is an effort to apply a multiple-model approach to produce six-month projections of the state and national trajectories of cases, hospitalizations, and deaths in the US under defined scenarios (Borchering et al., 2021). Scenarios from projection rounds have focused on control measures, vaccination availability and uptake, emerging variants, and waning immunity (COVID-19 Scenario Modeling Hub, 2020). Projections are released in a timely manner to guide policy decisions and data are made publicly available on a website (COVID-19 Scenario Modeling Hub, 2020).

Here, we detail results from the seventh round of projections, in which increased transmissibility variants were incorporated into projections to assess the potential impact of the Delta variant. In all scenarios, resurgences across the US were projected, with the largest resurgences occurring for scenarios with the highest variant transmissibility (60% increase over the Alpha variant, which most closely resembles estimates for the Delta variant). Corresponding increases in hospitalizations and deaths were also projected. In scenarios with higher vaccine coverage, the size and duration of this resurgence was notably smaller. Cases were projected to increase in early July 2021 at the national level and peak in mid to late September 2021. Corresponding increases in hospitalizations and deaths were also projected. The resurgence was projected to be geographically heterogeneous; although most states were projected to experience some degree of rebound, those having higher vaccine coverage were projected to experience less severe increases in incidence relative to prior observed peaks. However, while the timing of these projected resurges was relatively accurate compared to what has since been reported, the magnitude of reported cases, hospitalization, and deaths far surpassed what was projected. Here, we describe our experience with multi-model projections of the Delta variant to highlight both the value of scenario-based projections for planning, but also the challenges to understand and predict the constantly evolving COVID-19 pandemic.

Results

In the two scenarios with high Delta variant transmissibility (60% more transmissible than Alpha), we projected a national wave of cases to continue to grow over the summer and peak in mid- to late September 2021. In the scenario that assumes lower vaccination coverage among eligible individuals (70%) and higher variant transmissibility (the most pessimistic scenario), this resurgence was projected to peak at 414,000 weekly cases (95% projection interval (PI): 140,000–1,525,000) and 5900 weekly deaths (95% PI: 900–30,000) nationally. Overall, this scenario projected 7,554,000 (95% PI: 3,294,000–28,399,000) cumulative cases and 96,000 (95% PI: 27,000–476,000) cumulative deaths during July 4, 2021–Jan 1, 2022 (Figure 1).

Figure 1.

Historical data and weekly ensemble projections of reported numbers of COVID-19 cases.

(A) Hospitalizations (B) and deaths (C) under four scenarios representing different levels of vaccination and Delta variant transmissibility increase — United States, October, 2020–December, 2021.Projections are ensemble estimates of 9 models projecting four 6-month scenarios with 95% prediction intervals (the grey shading encompasses the prediction intervals from all four scenarios). Projections used empirical data from up to July 3, 2021, to calibrate models (black filled dots). The vertical lines indicate the beginning of each projection, with only data available prior to that point used to fit the projections. Observations available after the projection start are displayed as open dots.

Solid color lines and shaded areas represent median projections and 95% prediction intervals of reported cases for the entire US from each of the 9 models for each of the four scenarios. Projections used empirical data from up to July 3, 2021, to calibrate models (black filled dots). The vertical lines indicate the beginning of each projection, with only data available prior to that point used to fit the projections. Observations available after the projection start are displayed as open dots.

Solid color lines and shaded areas represent median projections and 95% prediction intervals of reported hospitalizations for the entire US from each of the 9 models for each of the four scenarios. Projections used empirical data from up to July 3, 2021, to calibrate models (black filled dots). The vertical lines indicate the beginning of each projection, with only data available prior to that point used to fit the projections. Observations available after the projection start are displayed as open dots.

Solid color lines and shaded areas represent median projections and 95% prediction intervals of reported deaths for the entire US from each of the 9 models for each of the four scenarios. Projections used empirical data from up to July 3, 2021, to calibrate models (black filled dots). The vertical lines indicate the beginning of each projection, with only data available prior to that point used to fit the projections. Observations available after the projection start are displayed as open dots.

Historical data and weekly individual projections of reported numbers of COVID-19 hospitalizations under four scenarios representing different levels of vaccination and Delta variant transmissibility increase — United States, October 2020–December 2021.

With higher variant transmissibility, increasing national vaccination coverage was projected to temper the fall wave slightly and cause it to drop more quickly, but not prevent it. With an increase in national vaccine coverage to 80% by January 1, 2022, the ensemble projected 65,000 (16%) fewer cases and 1300 (21%) fewer deaths per week at the peak, and 1,525,000 (20%) and 21,000 (22%) fewer cumulative cases and deaths, respectively, during July 4, 2021–January 1, 2022, when compared to the scenario where vaccination saturated at 70% nationally (Figure 1).

The projected national resurgence in COVID-19 cases in the higher transmissibility variant scenarios was composed of highly heterogeneous state-level resurgences. The ten states with the largest projected increases in incidence relative to their winter 2020–21 peak were, in descending order, Louisiana, Hawaii, Nevada, Arkansas, Florida, Missouri, Georgia, Alabama, Alaska, and Arizona. The ensemble estimates projected these states to experience median peak levels of weekly incident cases that were 18–69% (95% PI: 1%–541%) of their (smoothed) winter peak, although this was exceeded in many states. The 10 states with the lowest projected resurgences were, in ascending order, Massachusetts, Rhode Island, Vermont, Maine, Minnesota, Pennsylvania, North Dakota, Wisconsin, Tennessee, and South Dakota. The 10 states with the smallest projected resurgence had a median first-dose vaccine coverage of 70% among the eligible population (ages 12+) on July 3, 2021, compared to 56% in the ten states with the highest projected resurgence. We find a high negative correlation (Pearson’s r=–0.66, Figure 2) between projected cumulative deaths per population and vaccination coverage on July 3, 2021 (NCIRD, 2021). In all states, even those with low overall vaccination coverage, at least 76% of people 65+had received at least one dose of the vaccine, which was expected to have a major effect in limiting mortality from Delta (Centers for Disease Control and Prevention, 2021b).

Figure 2.

Projected cumulative cases and mortality in the most pessimistic scenario (low vaccination, high variant transmissibility) and current vaccination coverage by state — United States, July 4, 2021–January 1, 2022.

(A) Correlation between cumulative projected cases per 10,000 population during the 6-month period and proportion of the eligible population vaccinated with at least one COVID-19 vaccine dose by July 3, 2021, by state. Circle sizes represent population size. Single dose coverage was used as data reporting were most reliable for the first dose at the time of this analysis; yet second dose coverage is highly correlated with first dose coverage (Pearson rho = 0.92 on July 3, 2021, p<10–15). (B) Cumulative projected cases per 10,000 population during the 6-month period, by state. (C) Correlation between cumulative projected deaths per 10,000 population during the 6-month period and proportion of the eligible population vaccinated with at least one COVID-19 vaccine dose by July 3, 2021, by state. Circle sizes represent population size. (D) Cumulative projected deaths per 10,000 population during the 6-month period, by state.

The impact of vaccination was already being observed early in the Delta wave: in the 10 states with the largest projected resurgence there was a 9% reduction in the observed case fatality ratio (CFR) comparing August–December 2020 and January–July 2021; in the 10 states with the smallest projected resurgence a 21% reduction in CFR was observed. During the projection period, we projected CFR reductions of 15% and 14%, as compared to August-December 2020. Lower transmissibility variant scenarios projected significantly reduced resurgence, projecting cumulative national cases of only 9% and 13% compared to the winter 2020–21 peak. Similarly, in the previous projection round (Round 6), which similar to the 7th round except it assumed only a 20% transmissibility increase from a novel variant, resurgence was expected to produce only 8% of the cases reported during the winter 2020–21 peak nationally (COVID-19 Scenario Modeling Hub, 2020).

Weekly case observations exceeded our 95% projection interval in the first 9 weeks of the projection period in all scenarios, including those assuming 60% increased transmissibility of the Delta variant (Figure 1). Our projections also tended to underestimate hospitalizations and deaths in the ascending phase of the Delta wave, although to a lesser degree. We compared weekly incident and cumulative cases during the first four weeks after the projection date (July 4–31, 2021). The total median projected number of cases underestimated the observed cases overall during this 4-week period (1,256,000 observed vs 516,000 projected); however, we find a strong correlation between ranking of observed and projected total cases per 100,000 during the first four weeks of the projection period, at the state level (Spearman’s ⍴=0.87, Figure 3). Seven of the ten states with greatest projected incidence rank in the ten worst observed incidence states. Hence while projections did not capture the full scope of the rise in incidence due to the Delta variant, these projections reflected the expected severity ranking among state projections well.

Figure 3.

Comparison of the median projected and observed state-level total COVID-19 case incidences occurring during July 4–31, 2021, United States.

Comparison is based on ranking of incidence per capita in 50 states + DC (Spearman’s rank correlation = 0.867). The grey solid line represents perfect agreement between ranks (y=x), which overlays a regression line fitted to the data (dashed line) and 95% confidence intervals (grey shaded area).

The two high variant transmissibility scenarios, which most closely resembled the characteristics of the circulating Delta variant, projected the timing of the Delta resurgence, projecting deaths to increase simultaneously with reported cases and peak one week after reported cases (Figure 1). However, all scenarios substantially underestimated the magnitude of the Delta wave for all outcomes. Among the two high variant transmissibility scenarios, at the national level the peak cases were under-projected by 70% (95% PI: −25–91%) and 64% (95% PI: −49–88%) (349,183 and 413,733 vs 1.15 M peak reported weekly cases), though the 95% projection interval did capture the reported magnitude of the peak (Figure 1). Similarly, hospitalizations and deaths were also under-projected by 47% (95% PI: −184–84%) and 36% (95% PI: −230–82%) (46,000 and 56,000 vs 87,000 peak reported hospitalizations) and 67% (95% PI: −77–93%) and 59% (95% PI: −120–93%) (4700 and 6000 vs 14,000 peak reported deaths), respectively; both also captured the reported magnitude within projection intervals.

Discussion

Prevalence of the SARS-CoV-2 Delta variant rose quickly in the US between May and June 2021, with the variant achieving dominance by late June 2021, and accounting for over 90% of all SARS-CoV-2 infections for an extended period from late July to early December 2021 (Centers for Disease Control and Prevention, 2021b). This variant prompted concerns about the scale of the COVID-19 resurgence in the US in the summer and fall of 2021, especially in the midst of decreased NPIs and slowing vaccination rates. Projections combining insights from multiple models suggested sizable resurgences of COVID-19 across the US, assuming growth of a variant that is 60% more transmissible than the Alpha variant (an assumption aligned with most estimates of the relative transmissibility of the Delta variant) (Allen et al., 2022). In scenarios with higher vaccination coverage, the magnitude of the resurgence in cases and deaths was substantially lower than in the lower coverage scenarios. Efforts to increase vaccination rates are critical and will save lives before and during future resurgences. At the outset of the Delta wave on July 1, 2021, only 13 states and Washington, D.C. had accomplished President Biden’s goal for vaccination coverage among eligible populations at or above 70%.

The rapid case growth observed in July 2021 in multiple US states was surprising, tracking with or above the projections from our worst-case scenario. Scenarios were designed at the end of June 2021 based on information available at that time about the transmissibility of the Delta variant, vaccine effectiveness, and vaccine coverage; projections were generated on or before July 4, 2021. Our data should not be understood as forecasts, but as projections conditional on the scenarios and assumptions; several reasons could explain why case growth was faster than expected, including key epidemiological and behavioral aspects that may have affected the disease dynamics. Possible mechanisms driving the underestimation of the observed Summer 2021 resurgence may include inaccurate assumptions about the transmissibility and severity of Delta (including changes in serial interval or severity of infection relative to other variants), the effectiveness of the vaccines against infection and transmission, waning of natural and vaccine-derived immunity, changes in testing practices, and the interaction of these factors with NPIs and behavior change (Li et al., 2021; Elliott et al., 2021; Puranik et al., 2021). Assumptions regarding these factors were left to the discretion of the teams and therefore vary across models. Critically, eight of the nine teams did not include waning of natural or vaccine-derived immunity, an assumption that we now know is incorrect. (Higdon et al., 2021) Subsequent rounds of projection have focused on different scenarios of waning immunity. Yet in this round, absence of waning resulted in a much smaller population susceptible to infection with Delta, thus reducing the overall potential magnitude that could be projected by these models. Use of NPIs has been substantially reduced across the US, with a lapse in mask mandates in most states. Although modeled use of NPIs was left to the discretion of individual modeling teams and did not vary between scenarios, results of prior rounds underscore the effectiveness of NPIs, in combination with increasing vaccination, to moderate the spread of a highly transmissible variant (Borchering et al., 2021). The extent to which NPIs may be necessary will vary across states, as those states with high levels of vaccination coverage or natural immunity may be at lower risk for an increase in cases.

The impact of the resurgence on severe disease and mortality was expected to vary substantially across states; states with younger populations and higher vaccination coverage among older and high-risk populations were expected to experience a relatively lower burden of severe disease, even with resurgences in cases. Several states (for example, South Dakota, North Dakota) with low vaccination coverage were not projected to experience major resurgences, likely because of high naturally acquired immunity. In addition, as the projected resurgence continued throughout the summer and into the beginning of the school year, efforts to promote vaccination among eligible school-aged children and college students and to maintain key prevention strategies in schools (for example, mask-wearing among the unvaccinated, physical distancing, screening programs) likely helped reduce risks with a safe return to in-person instruction (Centers for Disease Control and Prevention, 2021a). Observed increases in new vaccinations, particularly among young age groups and in jurisdictions most severely impacted by the Delta variant, was a positive step in this direction (Centers for Disease Control and Prevention, 2021b).

The findings in this report are subject to several limitations. First, considerable uncertainty is inherent to long-term projections. This has been repeatedly illustrated throughout the COVID-19 pandemic, with rapid changes in behavior, deployment of vaccines and boosters, and the emergence of novel variants, each of which has the capacity to drastically shift the epidemic trajectories. Uncertainty may arise from three main sources: specification of the scenarios (for example, uncertainty in transmissibility); errors in the structure or assumptions of individual models given a specific scenario (for example, variations in assumptions about vaccination uptake); and inaccurate calibration based on incomplete or biased data (for example, reporting backlogs). None of the four scenarios considered here were likely to precisely reflect the future reality over a 6-month period. As a case in point the emergence of the Omicron variant in December 2021, at the end of our projection period, could not have been predicted when scenarios were designed in June 2021 (Borchering et al., 2021). Similarly, a resurgence in Delta variant incidence was observed in mid-fall 2021, possibly due to changes in behavior and waning immunity, and is not captured in scenarios or model projections. Further, for a given scenario, there is notable variation among individual model projections with regard to both the timing and the magnitude of the resurgence (Figure 1—figure supplements 1–3). Variation likely reflects differences in model structure, projected vaccine coverage, projected variant growth, and importance of seasonal effects. Some of these variations reflect true scientific uncertainty, making ensemble projections particularly useful to integrate uncertainty between and within individual models. In addition, these scenarios do not specify considerations of Delta infecting previously immune individuals due to moderate antigenic changes, the waning of existing immunity, increases in NPIs, or vaccination among children aged <12 years starting in November 2021, all of which were expected to be important drivers of dynamics in the subsequent months. In the same vein, model estimates are dependent on assumptions about vaccine hesitancy, which are informed in part by large-scale surveys of vaccine sentiments (Carnegie Mellon University Delphi Group, 2021; Estimates of Vaccine Hesitancy for COVID-19, 2021). These surveys may underestimate vaccine hesitancy, as coverage estimates among survey respondents are substantially higher than measured among the overall US population. Additionally, there are limitations to individual component models, although these concerns are tempered by analyzing ensembles of the nine different models. Overall, a full evaluation of our projections and sources of uncertainty is particularly difficult in a scenario context and is beyond the scope of this paper. However, it is worth noting that in this particular round of projection, the relationship between projection accuracy and time horizon is not straightforward (e.g. refer to Figure 1 for a visual assessment of coverage).

Figure 1—figure supplement 1.

Historical data and weekly individual projections of reported numbers of COVID-19 cases under four scenarios representing different levels of vaccination and Delta variant transmissibility increase — United States, October 2020–December 2021.

Solid color lines and shaded areas represent median projections and 95% prediction intervals of reported cases for the entire US from each of the 9 models for each of the four scenarios. Projections used empirical data from up to July 3, 2021, to calibrate models (black filled dots). The vertical lines indicate the beginning of each projection, with only data available prior to that point used to fit the projections. Observations available after the projection start are displayed as open dots.

Figure 1—figure supplement 3.

Historical data and weekly individual projections of reported numbers of COVID-19 deaths under four scenarios representing different levels of vaccination and Delta variant transmissibility increase — United States, October 2020–December 2021.

Solid color lines and shaded areas represent median projections and 95% prediction intervals of reported deaths for the entire US from each of the 9 models for each of the four scenarios. Projections used empirical data from up to July 3, 2021, to calibrate models (black filled dots). The vertical lines indicate the beginning of each projection, with only data available prior to that point used to fit the projections. Observations available after the projection start are displayed as open dots.

Conclusions

The emergence and introduction of more transmissible SARS-CoV-2 variants like the Delta variant was projected to lead to a substantial resurgence of COVID-19 in the US, which was observed in every state across the country. The high variant transmissibility scenarios, which more accurately represented the characteristics of the Delta variant, both in transmissibility and in current case trajectories, projected a significant national resurgence with substantial variation in magnitude across states. Resurgences were expected to be more pronounced in low-vaccination jurisdictions. The projections indicated that even with substantial vaccination coverage, the increased transmissibility of new variants like Delta can continue to challenge our ability to control this pandemic. Renewed efforts to increase vaccination coverage are critical to limiting transmission and disease, particularly in states with low natural immunity and lower current vaccination, in addition to re-instituting control measures like indoor masking when needed. Projections of Delta resurgence presented in this paper were made publicly available in early July 2021 (COVID-19 Scenario Modeling Hub, 2020), 2 months ahead of the peak of the Delta wave, providing actionable results. There is a trade-off between releasing projections in a timely manner to guide decisions, and projection accuracy and uncertainty that improve with incorporation of recent information. While these projections dramatically underestimated the magnitude of the Delta resurgence, demonstrating the challenges to predict this continually evolving pandemic, they did provide value in projecting the timing and emphasizing the importance of vaccination. Multi-model ensemble efforts such as the COVID-19 Scenario Modeling Hub are particularly well-suited to provide disease projections to inform the pandemic response under changing epidemiological and behavioral situations.

Materials and methods

The COVID-19 Scenario Modeling Hub (COVID-19 Scenario Modeling Hub, 2020) convened nine modeling teams in an open call to provide six-month (July 3, 2021-January 1, 2022) COVID-19 projections in the US using data available through July 3, 2021. Each team developed a model to project weekly reported cases, hospitalizations, and deaths, both nationally and by jurisdiction (50 states and the District of Columbia), for four different epidemiological scenarios. Models were calibrated against data from the Johns Hopkins Center for Systems Science and Engineering Coronavirus Resource Center and federal databases (Coronavirus Resource Center, 2020; US Department of Health and Human Services, 2020). The four scenarios included low and high vaccination hesitancy levels, assuming national vaccination coverage saturation at 80% and 70%, respectively, based on hesitancy surveys (Table 1) (Carnegie Mellon University Delphi Group, 2021; Estimates of Vaccine Hesitancy for COVID-19, 2021). Participating teams accounted for vaccination rates by state, age, and risk-groups (for example, older adults and health care workers). Specified vaccine efficacy levels were constant across the scenarios and were based on protection against clinical disease in randomized clinical trials and effectiveness studies; parameters for effectiveness against infection, transmission, and progression to severe outcomes (for example, death) were left to be specified by each team (COVID-19 Scenario Modeling Hub, 2020). When the scenarios were designed in late June 2021, little information was available on vaccine efficacy specific to the Delta variant and on waning immunity. For details on individual model assumptions, see Supplementary file 1.

Table 1.

COVID-19 projection scenarios* — United States, July 4, 2021–January 1, 2022.

Scenarios defined for projection of COVID-19 cases, hospitalizations, and deaths for the sixth round of projections through the COVID-19 Scenario Modeling Hub§.

	Low impact variant;(low transmissibility increase)	High impact variant;(high transmissibility increase)
High vaccination;(low hesitancy)	Vaccination: Coverage saturates at 80% nationally among the vaccine-eligible population* by December 31, 2021† VE is 50%/90% for Pfizer/Moderna against currently circulating variants (1st /2nd dose) and 60% for J&J (1 dose) J&J no longer used†* Variant: 40% increased transmissibility as compared with Alpha for Delta variant. Initial prevalence estimated at state-level by teams.	Vaccination: Coverage saturates at 80% nationally among the vaccine-eligible population* by December 31, 2021† VE is 35%/85% for Pfizer/Moderna against currently circulating variants (1st /2nd dose) and 60% for J&J (1 dose) J&J no longer used†* Variant: 60% increased transmissibility as compared with Alpha for Delta variant. Initial prevalence estimated at state-level by teams.
Low vaccination;(high hesitancy)	Vaccination: Coverage saturates at 70% nationally among the vaccine-eligible population* by December 31, 2021† VE is 50%/90% for Pfizer/Moderna against currently circulating variants (1st /2nd dose) and 60% for J&J (1 dose) J&J no longer used†* Variant: 40% increased transmissibility as compared with Alpha for Delta variant. Initial prevalence estimated at state-level by teams.	Vaccination: Coverage saturates at 70% nationally among the vaccine-eligible population* by December 31, 2021† VE is 35%/85% for Pfizer/Moderna against currently circulating variants (1st /2nd dose) and 60% for J&J (1 dose) J&J no longer used ‡ Variant: 60% increased transmissibility as compared with Alpha for Delta variant. Initial prevalence estimated at state-level by teams.

The Vaccine-eligible population is presumed to be individuals aged 12 years and older through the end of the projection period.

Vaccine hesitancy expected to cause vaccination coverage to slow and eventually saturate at some level below 100%. The saturation levels provided in these scenarios are National reference points to guide defining hesitancy, though the speed of that saturation and heterogeneity between states (or other geospatial scales) and/or age groups are at the discretion of the modeling team (COVID-19 Scenario Modeling Hub, 2020). The high vaccination 80% saturation is defined using the current estimates from the Delphi group (updated from Round 6) (Carnegie Mellon University Delphi Group, 2021). The low saturation estimate of 70% is the lowest county-level estimate from the US Census Bureau’s Pulse Survey from May 26-June 7, 2021 data (Estimates of Vaccine Hesitancy for COVID-19, 2021).

To simplify the models and future projections of vaccine administration, it was assumed continued administration of the Johnson & Johnson (J&J) vaccine would not occur on or after the projection date (after July 4, 2021) due to the limited amount administered previously in the US (as of August 4, 2021 approximately 4 million doses delivered since April 13, 2021 compared to 153 million for Pfizer and Moderna) (Centers for Disease Control and Prevention, 2021b).

COVID-19 Scenario Modeling Hub: https://covid19scenariomodelinghub.org/.

Scenarios assumed one of two levels of increased transmissibility for the Delta variant: 40% (low) or 60% (high) more transmissible than the Alpha variant. Increases in new variant prevalence over time were determined by each modeling team and were estimated at the state level.

Individual models differed substantially in structure and design; see Supplementary file 1 and the COVID-19 Scenario Modeling Hub GitHub website for more details (COVID-19 Scenario Modeling Hub, 2021). Individual modeling teams provided probabilistic projections of incident and cumulative epidemic trajectory for each week of the projection period, with 23 quantiles requested (0.01, 0.025, 0.05, every 5%–0.95, 0.975, and 0.99). These individual projections were combined into an ensemble for each scenario, outcome, week, and location using an equally-weighted linear opinion pool method across teams that trimmed the highest and lowest model at each point and quantile (Stone, 1961; Jose et al., 2014). Point estimates provided here are the median of the ensemble.

For any given pair of scenarios, averted cases and deaths were calculated as the difference (and ratio) between the median point estimates of the ensemble for the two scenarios. To provide a relative measure of resurgence in each state, we compared the intensity of the projected outbreak in the next six months to the size of the winter 2020–2021 outbreak – a period of high hospital burden in many jurisdictions. Specifically, projected resurgences were assessed by taking the ratio of the peak projected median incidence in a given location over the projection period (July 3, 2021-January 1, 2022) to the highest incidence experienced during the winter 2020–2021 period (defined as October 1, 2020–February 28, 2021) for the same location. Winter 2020–21 peaks were identified as the seven-day average centered around the day with the highest incident cases from smoothed curves generated through a penalized cubic spline Poisson regression model fit to the incident cases.

Details on the data used by each model can be found in Supplementary file 1, with further details found on the COVID-19 Scenario Modeling Hub GitHub repository website (https://github.com/midas-network/covid19-scenario-modeling-hub; DOI: 10.5281/zenodo.6584489) (COVID-19 Scenario Modeling Hub, 2021). All model output data and ensembled estimates are publicly available on the GitHub repository. All code used to generate numbers and figures reported in this manuscript are publicly available via the GitHub repository. Code required for ensembling model outputs can be made available upon request. Figure, code, and data are available through the open-source MIT license.

Disclaimer: The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention or the National Institutes of Health. Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the US Government.

In this paper, the authors presented the joint efforts of nine modeling teams to provide a six-month projection of the COVID-19 pandemic across the US, in view of the circulation of the more transmissible Delta variant. The results represented a timely assessment of the risk of COVID-19 resurgence in Summer 2021 when it was conducted in July 2021, and will be of historical interest as an example of modeling efforts to inform real-time decision making during the COVID-19 pandemic. This paper will be of high interest to public health specialists, forecast modelers, and members of the general public interested in the evolution of the COVID-19 pandemic and the impact of public health interventions in the USA.

Our editorial process produces two outputs: i) public reviews designed to be posted alongside the preprint for the benefit of readers; ii) feedback on the manuscript for the authors, including requests for revisions, shown below. We also include an acceptance summary that explains what the editors found interesting or important about the work.

Decision letter after peer review:

Thank you for submitting your article "Projected resurgence of COVID-19 in the United States in July-December 2021 resulting from the increased transmissibility of the Δ variant and faltering vaccination" for consideration by eLife. Your article has been reviewed by 3 peer reviewers, one of whom is a member of our Board of Reviewing Editors, and the evaluation has been overseen by a Senior Editor. The following individual involved in review of your submission has agreed to reveal their identity: Jodie McVernon (Reviewer #3).

As is customary in eLife, the reviewers have discussed their critiques with one another. What follows below is the Reviewing Editor's edited compilation of the essential and ancillary points provided by reviewers in their critiques and in their interaction post-review. Please submit a revised version that addresses these concerns directly. Although we expect that you will address these comments in your response letter, we also need to see the corresponding revision clearly marked in the text of the manuscript. Some of the reviewers' comments may seem to be simple queries or challenges that do not prompt revisions to the text. Please keep in mind, however, that readers may have the same perspective as the reviewers. Therefore, it is essential that you attempt to amend or expand the text to clarify the narrative accordingly.

Essential revisions:

1. Provide a table describing the basic features of the models included in this analysis. Particular parameters that should be described include vaccine efficacy against infection and assumptions regarding natural immunity to reinfection.

2. Provide an analysis showing individual model predictions for a scenario in order to assess inter-model variability of predictions.

3. Provide further discussion regarding what assumptions of the models may have led to the underestimation of the number of cases in the Summer/Fall, and the features of the models that were more closely able to reproduce the observed resurgence.

4. Comment on the timeliness of predictions and the time horizon over which predictions are likely to be valid.

5. Discussion on whether heterogeneity below the state level could have contributed to the overestimation of vaccination effectiveness by the models.

6. Discussion on the impact of waning vaccine immunity, whether this was accounted for by the models, and whether this will be accounted for in future rounds of predictions.

Reviewer #1 (Recommendations for the authors):

– The current first section "overview" serves as both introduction, and summary of results. I think the manuscript would benefit from a more traditional introduction, where the objectives of the COVID-19 Scenario Modeling Hub are detailed, results from previous rounds are summarized, and the objectives of the current round (6? 7? I am not very clear on this) are detailed. The main results could be summarized in the discussion.

– There is very little information about the models which contributing predictions to this analysis. This does not allow the reader to fully assess the underlying methods. I would generally expect in any multiple model analysis to have a table describing basic features of the models, in the same way that a systematic review/meta-analysis would describe the features of the underlying studies. Key features to describe would be the type of model structure, design, and basic underlying assumptions likely to vary between models such as effectiveness against infection, transmission, and progression to severe outcomes, demographic stratifications included etc.

– The main value of combining different model predictions is to capture and explore uncertainty due to differences between models. Please consider including a figure of cross-model comparison of individual model predictions for a given scenario. Some discussion regarding differences between models leading to differences in predictions would also be warranted.

– I liked the comparison between model predictions and observed data for July. I would have liked to see some discussion relating to how this information is likely to influence future rounds of modelling and predictions, as it suggests that some of the assumptions of the models do not reflect the current epidemiology on the ground.

Reviewer #2 (Recommendations for the authors):

1) About the timeliness of the projections: I received the review invitation on 28 September 2021. Could the authors comment on the timeliness of projections? For example, if six-month projections were not possible, would it be more practical to provide projections every two months?

2) About model assumptions and Table 1: It is not clear to me what vaccine efficacy meant here in Table 1. Vaccine efficacy against infection, symptomatic infection, hospitalization, or death of COVID-19? Since the vaccine uptake by 3 July 2021 was the most important factor in the projections of cumulative cases and deaths in Figure 2, what were the assumptions in the nine models about immunity from vaccination and natural infections from the previous waves? Could the authors summarize the most important assumptions for each of the nine models?

3) About source of uncertainties: There were substantial uncertainties in the model projections in Figure 1 (i.e., the 95% CI of projections could cover nearly all the possible actual outcomes). Could the authors identify the major sources of uncertainties of each of the nine models?

Although all the data and codes are publicly available, it will be great if the authors could add a table to summarize the major similarities and differences of the nine models, given the high uncertainties in projections across different models in Figure 1.

Reviewer #3 (Recommendations for the authors):

This ensemble modelling exercise is incredibly useful, but could be even further enhanced by more of a discussion of difference, as indicated above. It will be very important to include more considerations of immune waning and boosting in future rounds.

Will future rounds incorporate assumptions about the duration of protection following the primary series, particularly as boosters come online and will also need to be incorporated? Could waning be a partial explanation for the overestimation of vaccine impacts here?

Naturally acquired immunity is mentioned but is natural boosting of immunised individuals built into any of these models?

Essential revisions:

1. Provide a table describing the basic features of the models included in this analysis. Particular parameters that should be described include vaccine efficacy against infection and assumptions regarding natural immunity to reinfection.

We have added Supplementary File 1, a table that provides a summary of the assumptions for each of the nine contributing models. We have added the following sentence to the Methods (line 101, pg 6):

“For details on individual model assumptions, see Supplementary File 1.”

2. Provide an analysis showing individual model predictions for a scenario in order to assess inter-model variability of predictions.

We have added Figure4-supplemental figures 1-3, which display the individual model projections, demonstrating the inter-model variability of predictions. We have added a reference to these figures on line (230-231). Further, our new supplemental tables describing individual model assumptions will help readers better understand some of the drivers of inter-model variability (Supplementary File 1).

Although we comment on some of the drivers of this variability in the text (see 4th paragraph of Discussion, pg 11), we believe that a formal analysis of individual model predictions is beyond the scope of this paper. Differences between models are many and include variation in model structure, parameters, and calibration; this understanding the drivers of model variation is not straightforward. Overall, there are many dimensions of uncertainty in the epidemiologic situation, and it is likely that no individual model appropriately captures all these aspects of uncertainty. By combining the models together into an ensemble, we aim to better capture the range of this uncertainty and therefore to provide projections that are more useful for public health decision making.

3. Provide further discussion regarding what assumptions of the models may have led to the underestimation of the number of cases in the Summer/Fall, and the features of the models that were more closely able to reproduce the observed resurgence.

We have added additional text to further detail why projections underestimate the observed resurgences.

In the discussion (pg 10), we now state:

“Possible mechanisms driving the underestimation of the observed Summer 2021 resurgence may include inaccurate assumptions about the transmissibility and severity of Delta (including changes in serial interval or severity of infection relative to other variants), the effectiveness of the vaccines against infection and transmission, heterogeneity in vaccine coverage at the county level or below, waning of natural and vaccine-derived immunity, changes in testing practices, and the interaction of these factors with NPIs and behavior change.13–15 Assumptions regarding these factors were left to the discretion of the teams and therefore vary across models.”

No individual model was able to successfully reproduce the observed resurgences at the state-level. Those models that tracked with the observed resurgence at the national level for certain scenarios (particularly USC) did not consistently do so at the state-level.

4. Comment on the timeliness of predictions and the time horizon over which predictions are likely to be valid.

These should not be considered predictions, but rather projections of the epidemic trajectory under a predefined set of assumptions (the “scenario”). It is likely that no scenario will perfectly align with reality over the projection period and hence a simple comparison between observations and projections (as in forecast) is not satisfactory. As a case in point, the Omicron variant emerged at the end of our projection period and was not part of the scenarios that were designed in June 2021 (round 7) and are discussed in this paper.

We note that a full evaluation of our scenario projection efforts is underway and includes various stages of the pandemic (not just the Δ wave). We find that accuracy, measured by coverage or other scoring statistics, does not solely depend on time horizon, but also on the stage of the pandemic and on scenario design. In this round of projection particularly, coverage degrades rapidly over the first 9 weeks of the projection period but recovers in subsequent weeks (as can be seen in Figure 1). A full evaluation is beyond the scope of this paper, and in the meantime, we show a figure comparing state-level projections and observations over the first four weeks of projections (Figure 3).

As regards timeliness, our round 7 projections were made publicly available in mid-July 2021, as COVID19 incidence had just started rising, with the peak of the Δ wave occurring in the first week of September 2021. Hence our projections afforded reasonable time for action.

We have added two sentences in conclusion to highlight the timeliness of our projections (pg 13) and note the inherent difficulties in evaluating scenario-based projections and changes in accuracy by time horizon (pg 12). We also comment that projections are a valid tool that can be used to demonstrate the anticipated impact of changes to the epidemiologic situation – in this case, the relative importance of vaccine coverage and the introduction of a new variant (conclusion).

5. Discussion on whether heterogeneity below the state level could have contributed to the overestimation of vaccination effectiveness by the models.

Teams only submit projections at the state and national level, however many of the contributing models are county-level (Columbia, MOBS, UVA-adaptive, JHU APL). Heterogeneity in vaccination coverage at the county-level (or below) not captured by the models due either to spatial structure or lack of county-level data may very well have contributed to the underestimation of the observed resurgences.

We have added a comment to this effect in discussion pg 10.

6. Discussion on the impact of waning vaccine immunity, whether this was accounted for by the models, and whether this will be accounted for in future rounds of predictions.

Waning of both natural and vaccine-derived immunity was left to the discretion of the individual modeling teams. Only one of the 9 teams included waning immunity in their models. At the time of this round of projections, very little was known about the amount and characteristics of SARS-CoV-2 immunity waning. Subsequent SMH rounds do incorporate waning assumptions; these, along with all rounds of SMH projections, can be accessed on the SMH website (see round 13 in particular, https://covid19scenariomodelinghub.org).

We acknowledge this is now likely a limitation of our projections and have added the following to the Discussion section of this paper to discuss this:

“Critically, eight of the nine teams did not include waning of natural or vaccine-derived immunity, an assumption that we now know is incorrect.16 Absence of waning resulted in a much smaller population susceptible to infection with Delta, thus reducing the overall potential magnitude that could be projected by these models.”

“In addition, these scenarios do not specify considerations of reinfections or breakthrough infections with the Δ variant, the waning of existing immunity, changes in NPIs and behavior, or vaccination among children aged <12 years starting in November 2021, all of which were expected to be important drivers of dynamics in the subsequent months.”

Reviewer #1 (Recommendations for the authors):

– The current first section "overview" serves as both introduction, and summary of results. I think the manuscript would benefit from a more traditional introduction, where the objectives of the COVID-19 Scenario Modeling Hub are detailed, results from previous rounds are summarized, and the objectives of the current round (6? 7? I am not very clear on this) are detailed. The main results could be summarized in the discussion.

We have modified the overview paragraph to have a bit more traditional feel, splitting it into paragraphs and adding the following paragraph about the COVID-19 Scenario Modeling Hub:

“Established in December 2020, the COVID-19 Scenario Modeling Hub is an effort to apply a multiple-model approach to produce six-month projections of the state and national trajectories of cases, hospitalizations, and deaths in the US under defined scenarios.2 Scenarios from projection rounds have focused on control measures, vaccination availability and uptake, and emerging variants.3”

We did leave a summary of the main findings in the “Overview”, as we felt this was appropriate for this short report, but we reduced this to a briefer summary to mirror a more traditional research article format as suggested. Additionally, to make it clear that this paper is focused on Round 7, we have removed all references to Round 6, focusing on Round 7 only.

– There is very little information about the models which contributing predictions to this analysis. This does not allow the reader to fully assess the underlying methods. I would generally expect in any multiple model analysis to have a table describing basic features of the models, in the same way that a systematic review/meta-analysis would describe the features of the underlying studies. Key features to describe would be the type of model structure, design, and basic underlying assumptions likely to vary between models such as effectiveness against infection, transmission, and progression to severe outcomes, demographic stratifications included etc.

We have added a supplemental table (Supplementary File 1) which provide an overview of individual model structure and assumptions.

– The main value of combining different model predictions is to capture and explore uncertainty due to differences between models. Please consider including a figure of cross-model comparison of individual model predictions for a given scenario. Some discussion regarding differences between models leading to differences in predictions would also be warranted.

We have added Figure 1-supplemental figures 1-3 which show the results of the nine individual models. Additionally, we have added a supplemental table (Supplementary File 1) which provide an overview of individual model structure and assumptions. We provide the following commentary in the Discussion section:

“Further, for a given scenario, there is notable variation among individual model projections with regards to both the timing and the magnitude of the resurgence (Figure 1-supplemental figure 1-3). Variation likely reflects differences in model structure, projected vaccine coverage, projected variant growth, and importance of seasonal effects.”

– I liked the comparison between model predictions and observed data for July. I would have liked to see some discussion relating to how this information is likely to influence future rounds of modelling and predictions, as it suggests that some of the assumptions of the models do not reflect the current epidemiology on the ground.

We update the parameters specified in the scenarios each round to align with the most up to date evidence. For example, we have incorporated waning into later rounds (we have just completed round 13 focused on waning). Further, in later rounds, we paid particular attention to estimates of VE specific to the Delta variant that became available later in the year.

Reviewer #2 (Recommendations for the authors):

1) About the timeliness of the projections: I received the review invitation on 28 September 2021. Could the authors comment on the timeliness of projections? For example, if six-month projections were not possible, would it be more practical to provide projections every two months?

This is a good point and we now comment on the timeless of our projections (conclusion p 13):

“Projections of Delta resurgence presented in this paper were made publicly available in early July 2021 3, two months ahead of the peak of the Delta wave, providing actionable results. There is a trade-off between releasing projections in a timely manner to guide decisions, and projection accuracy and uncertainty that improve with incorporation of recent information.”

We acknowledge there was a longer lag to turn this work into a publication.

We do acknowledge that as the epidemiological situation changes over the projection period, none of the scenarios will be expected to align with reality. However, the goal of these projections is not to predict the future but rather to demonstrate the anticipated impact of changes in vaccination coverage or the introduction of a more transmissible variant. While on occasion, we have provided predictions for shorter horizons (eg two closely spaced rounds of 3-month projections for Omicron, rounds 10 and 11, 17 days apart), these are more appropriate for emergency situations. Overall, we believe that a longer projection period better conveys the long-term impacts of changes in disease dynamics.

2) About model assumptions and Table 1: It is not clear to me what vaccine efficacy meant here in Table 1. Vaccine efficacy against infection, symptomatic infection, hospitalization, or death of COVID-19? Since the vaccine uptake by 3 July 2021 was the most important factor in the projections of cumulative cases and deaths in Figure 2, what were the assumptions in the nine models about immunity from vaccination and natural infections from the previous waves? Could the authors summarize the most important assumptions for each of the nine models?

The vaccine effectiveness specified in the scenarios is against symptomatic disease. We have added a note to table 1 clarifying this point. Further, we have added a supplemental table (Supplementary File 1) which provide an overview of individual model structure and assumptions.

3) About source of uncertainties: There were substantial uncertainties in the model projections in Figure 1 (i.e., the 95% CI of projections could cover nearly all the possible actual outcomes). Could the authors identify the major sources of uncertainties of each of the nine models?

We note in the text several key drivers of uncertainty:

“Uncertainty may arise from three main sources: specification of the scenarios (e.g., uncertainty in transmissibility); errors in the structure or assumptions of individual models given a specific scenario (e.g., variations in assumptions about vaccination uptake); and inaccurate calibration based on incomplete or biased data (e.g., reporting backlogs). None of the four scenarios considered here are likely to precisely reflect the future reality over a six-month period.2”

Uncertainty in the ensemble, which is obtained by a linear opinion pool approach, is driven both by differences between the trajectories of individual models, and uncertainty on the trajectory of individual models. We do feel that because sources of uncertainty are so varied, a discussion of model-specific results and uncertainty is beyond the scope of this paper. The reader can also refer to new tables in the supplement that describe the major assumptions from each model (Supplementary File 1).

Although all the data and codes are publicly available, it will be great if the authors could add a table to summarize the major similarities and differences of the nine models, given the high uncertainties in projections across different models in Figure 1.

We have added a table to the supplement with the main assumptions of each of the contributing models.

Reviewer #3 (Recommendations for the authors):

This ensemble modelling exercise is incredibly useful, but could be even further enhanced by more of a discussion of difference, as indicated above. It will be very important to include more considerations of immune waning and boosting in future rounds.

We have strengthened our discussion of sources of model differences and provided a supplementary table of model assumptions. Subsequent SMH rounds have focused on different scenarios of waning and immune escape, which we now point to in the revised discussion. Results can be found at https://covid19scenariomodelinghub.org/index.html.

Will future rounds incorporate assumptions about the duration of protection following the primary series, particularly as boosters come online and will also need to be incorporated? Could waning be a partial explanation for the overestimation of vaccine impacts here?

This is certainly a possibility for future rounds. Round 8, 11 and round 13 incorporated waning immunity and future rounds may further refine this (round 13 is publicly available here https://covid19scenariomodelinghub.org/index.html). It is possible that improperly accounting for waning contributed to the underestimation of cases during the projection period, which we now mention.

Naturally acquired immunity is mentioned but is natural boosting of immunised individuals built into any of these models?

This was left to the discretion of the teams.