The effect of a proof-of-vaccination requirement, incentive payments, and employer-based mandates on COVID-19 vaccination rates in New York City: a synthetic-control analysis.

BACKGROUND: COVID-19 vaccines have been available to all adults in the USA since April, 2021, but many adults remain unvaccinated. We aimed to assess the joint effect of a proof-of-vaccination requirement, incentive payments, and employer-based mandates on rates of adult vaccination in New York City (NYC).
METHODS: We constructed a synthetic control group for NYC composed of other counties in the core of large, metropolitan areas in the USA. The vaccination outcomes for NYC were compared against those of the synthetic control group from July 26, 2021, to Nov 1, 2021, to determine the differential effects of the policies. Analyses were conducted on county-level vaccination data reported by the Centers for Disease Control and Prevention. The synthetic control group was constructed by matching on county-level preintervention vaccination outcomes, partisanship, economic attributes, demographics, and metropolitan area population. Statistical inference was conducted using placebo tests for non-treated counties.
FINDINGS: The synthetic control group resembled NYC across attributes used in the matching process. The cumulative adult vaccination rate for NYC (in adults aged 18 years or older who received at least one dose of an authorised COVID-19 vaccine) increased from 72·5% to 89·4% (+16·9 percentage points [pp]) during the intervention period, compared with an increase from 72·5% to 83·2% (+10·7 pp) for the synthetic control group, a difference of 6·2 pp (95% CI 1·4-10·7), or 410 201 people (90 966-706 532). Daily vaccinations for NYC were consistently higher than those in the synthetic control group, a pattern that started shortly after the start of the intervention period.
INTERPRETATION: The combination of a proof-of-vaccination requirement, incentive payments, and vaccine mandates increased vaccination rates among adults in NYC compared with jurisdictions that did not use the same measures. Whether the impact of these measures occurred by inducing more people to get vaccinated, or by accelerating vaccinations that would have occurred later, the increase in vaccination rates likely averted illness and death. FUNDING: None.

Introduction

High rates of COVID-19 vaccination remain far from universal in many high-income countries despite widespread availability of vaccines in most of these countries. In the USA, data from the Centers for Disease Control and Prevention (CDC) show that 66·7% of people aged 18 years or older had received at least one dose of an authorised COVID-19 vaccine as of July 1, 2021. By June 1, 2022, 89·3% of people aged 18 years or older had received at least one dose of vaccine. However, these averages obscure wide geographical variation across the USA in this age group, from 69·7% (in Wyoming) to at least 95% in several states (eg, New York, New Jersey, and Pennsylvania) on June 1, 2022.

For many diseases that are preventable by vaccines, high levels of adult immunity are achieved by policies that promote vaccination during childhood, such as school entry vaccination requirements. Few vaccines that are specifically targeted for adults receive high levels of uptake. For example, only 50·2% of adults in the USA received an influenza vaccine during the 2020–21 influenza season, and only 70·3% of adults aged 65 years or older had received a pneumococcal vaccine as of 2020.4, 5 Looking at these data, it is clear that different strategies might be needed to achieve high levels of COVID-19 vaccination in adults.

Policy makers who are seeking to increase rates of COVID-19 vaccination in adults would benefit from an analysis of real-world data for the effect of different policies on rates of vaccination. However, research on proof-of-vaccination requirements and payments for COVID-19 vaccination has thus far produced inconsistent results.6, 7, 8, 9

Evidence before this study

Several countries and subnational jurisdictions have implemented policies to increase rates of COVID-19 vaccination. We searched PubMed, medRxiv, RePEc, and NBER for studies published in English up to Feb 16, 2022, that assessed the effect of COVID-19 proof-of-vaccination requirements, mandates, payments, or lotteries on vaccination rates using the following search terms: ((“COVID-19” OR “SARS-CoV-2”) AND (“vaccine” OR “vaccination”) AND (“passport” OR “certificate” OR “mandate” OR “lottery” OR “payment” OR “incentive” OR “green pass” OR “EU digital COVID certificate”)). Research on proof-of-vaccination requirements, incentive payments, and lotteries has shown mixed results. We were unable to identify any empirical studies on policies for increasing COVID-19 vaccination that analysed the joint effect of several policies.

Added value of this study

Our analysis showed that a combination of policies, including a proof-of-vaccination requirement, financial incentives, and employer-based vaccine mandates, can increase rates of adult COVID-19 vaccination. To our knowledge, this study is the first rigorous empirical analysis of a multipronged programme to increase rates of COVID-19 vaccination.

Implications of all the available evidence

Our analysis showed that structured, multipronged government policies can increase adult COVID-19 vaccination rates. Similar policies could increase first-dose vaccination rates in jurisdictions that have not yet implemented such policies or could be used to increase booster dose uptake.

From July 26, 2021, to Nov 1, 2021, New York City (NYC) implemented several measures to increase rates of vaccination in adults. Adult residents in NYC who received a first dose of an authorised COVID-19 vaccine at a site run by the City of New York were eligible for a US$100 incentive payment. In addition, visitors to indoor dining, entertainment, and fitness centers were required to show proof of vaccination (termed the Key to NYC). Lastly, employer-based mandates were required for city government employees, health-care workers (state initiative), and public-school staff (table 1 , appendix p 2). In our study, we aimed to estimate the joint effect of these measures on the rate of adult COVID-19 vaccination.

Table 1

Timeline of New York City (NYC) policies to increase vaccination, by date of announcement

	Effective date	Policy	Population affected	Opt-out	Dose
July 26, 2021	Aug 16, 2021	Mandate	Some NYC government employees*	Testing	Primary course
July 26, 2021	Sept 13, 2021	Mandate	NYC government employees	Testing	Primary course
July 28, 2021	July 28, 2021	$100 incentive	NYC residents	NA	First dose
Aug 3, 2021	Aug 17, 2021†	Proof-of-vaccination	Visitors to covered locations‡	No	First dose
Aug 16, 2021	Sept 27, 2021	Mandate	Health-care workers	No	First dose
Aug 23, 2021	Sept 27, 2021	Mandate	Department of Education employees	No	First dose
Aug 26, 2021	Aug 26, 2021	Outreach§	Residents of 20 neighbourhoods	NA	Primary course
Sept 20, 2021	Sept 20, 2021	Counselling§	Medicaid and Medicare Advantage	NA	First dose
Oct 20, 2021	Oct 20, 2021	$500 incentive	NYC government employees	NA	First dose
Oct 20, 2021	Oct 29, 2021	Mandate	NYC government employees	No	Primary course

NA=not applicable.

The mandate that was effective from Aug 16, 2021, and applied to NYC government employees and contractors in residential and congregate care settings.

Enforcement of the proof-of-vaccination requirement did not begin until Sept 13, 2021.

The proof-of-vaccination requirement applied to employees and patrons (aged 12 years or older) of indoor areas of entertainment and recreational settings, indoor food services, and indoor gyms and fitness settings.

The Vaccine Equity Partner Engagement programme (outreach) and Vaccine Outreach and Counseling programme (counselling) were not explicitly analysed as part of the intervention because both consisted of outreach and engagement, while the other policies consisted of incentives or mandates that were targeted towards individuals. Results are robust to the exclusion of counties known to have implemented policies similar to the Vaccine Outreach and Counseling programme (appendix pp 18–20). We did not test robustness to the exclusion of counties with programmes that were similar to the Vaccine Equity Partner Engagement programme out of concern that we would be unable to successfully identify all counties in the donor pool with similar policies.

Methods

Study design

Our analysis was conducted using vaccination data from April 26 to Nov 1, 2021. Nov 1 was chosen as the end of the analysis period to coincide with the effective date of the last of several policies that NYC implemented in quick succession (table 1, appendix p 2) and to pre-date the Nov 3, 2021, emergency use authorisation for tozinameran (BNT162b2, Pfizer-BioNTech) for children aged between five years and 11 years, which might have indirectly affected adult vaccination rates. We discarded vaccination data from before April 26, 2021, because earlier vaccination data might reflect supply or administration capacity constraints rather than demand. The results of the main analysis were robust to the choice of the start of the preintervention period (appendix p 12).

We split the full preintervention period into a model training period (April 26 to May 23, 2021) and a model validation period (May 24 to July 25, 2021). The cutoff point between the two periods followed from the model specification: the model matched on 4 weeks of preintervention vaccination data, and May 24, 2021, was 4 weeks after April 26. Matching was done using vaccination data from June 28 to July 25, 2021 (termed the matching period) and parameters that were estimated from the training and validation periods. Statistical inference was done using data from the validation and intervention periods. Because data were not collected specifically for this study and the study did not use identifiable personal data, specific ethical approval and participant consent were not required.

Outcomes

The main outcome was the 7 day average of the share of adults who received the first dose of an authorised COVID-19 vaccine on each day. Person-level COVID-19 vaccine administration data were reported to the CDC through jurisdictional immunisation information systems, vaccine administration management systems, or direct data submissions to the COVID-19 Data Clearing House. Data for COVID-19 vaccinations that were administered in NYC in particular were reported to the CDC from the Citywide Immunization Registry, a NYC-run database of birth and immunisation records. The data from the CDC were validated, deduplicated, and published by report date and county of residence. We also did additional data cleaning to remove distortions from reporting adjustments and retroactive reporting (appendix p 3). Vaccination rates were calculated relative to the 2019 American Community Survey 1-year population estimates.

Additional analyses were done on the 7-day averages of the share of people who were 65 years or older who received a first dose of vaccine on each day and the share of adults who completed a primary course on each day. Both series came from the same dataset as the primary outcome and were cleaned analogously.

Synthetic control group selection

We assessed the effect of NYC's vaccination policies on the rates of adult vaccination using the synthetic control method.13, 14 We constructed a control group for NYC as a weighted average of core counties in large metropolitan areas. County weights were chosen so that the control group resembled NYC with regard to preintervention vaccination outcomes and other attributes that are associated with variation in vaccination outcomes (eg, race and ethnicity, partisanship, and age distribution). Vaccination outcomes from the intervention period for the control group were interpreted as the vaccination outcomes that would have prevailed in NYC without the policy intervention. The effect of NYC's policies was identified as the difference in intervention-period vaccination outcomes between NYC and the control group.

The control group was selected in two steps. First, weights were selected for each attribute to capture the relative power of each attribute to predict vaccination outcomes. For any candidate set of attribute weights, a candidate control group could be selected that best replicated NYC on the weighted set of attributes. Attribute weights were selected using vaccination data from the training period to minimise the root mean squared prediction error (RMSPE) in vaccinations between NYC and the control group associated with each candidate set of attribute weights over the validation period. Second, the control group was chosen as the weighted average of counties that best matched NYC over the matching period, as measured by the weighted set of attributes.

We selected the control group from a pool of counties (termed the donor pool) in the core of metropolitan statistical areas with at least 500 000 residents (appendix pp 4–6). We excluded any county that implemented at least one policy that was similar to the policies implemented in NYC before Nov 1, 2021, any county in a state that implemented such a policy, and any county whose largest principal city implemented such a policy. We also excluded counties with 14-day-or-longer gaps in vaccination data and counties in states where the county of residence was known for fewer than 75% of individuals who had completed a primary course. No remaining counties had shorter gaps in vaccination data, so no interpolation was necessary. Data for state, county, and city policies were collected from the Kaiser Family Foundation, US News & World Report, the National Academy for State Health Policy, Ballotpedia, and Google.16, 17, 18, 19 After exclusions, the control pool contained 44 counties in 19 states (appendix pp 5–6).

County weights for the synthetic control group were constructed using preintervention vaccination outcomes and a set of time-invariant attributes, including: (1) the metropolitan area population; (2) partisanship, measured as the share of non-third-party votes that were received by the Democratic candidate in the 2020 presidential election; (3) race and ethnicity; (4) age (percentage of people aged between 16 and 59 years and percentage of people aged 60 years or older); (5) sex; (6) educational achievement among adults aged 25 years or older; (7) median household income; (8) poverty rate; and (9) unemployment rate.12, 20, 21 Attributes were selected a priori based on a literature review of determinants of vaccine hesitancy in high-income countries. Data were not missing for any of these attributes for NYC or for the counties included in the donor pool. Attributes with greater predictive power were weighted more heavily in the construction of county weights (appendix p 7).

Statistical analysis

Statistical inference was conducted using placebo tests. A separate control group was selected for each member of the donor pool. We did not expect to observe differences in vaccination outcomes between any donor pool county and its corresponding control group during the intervention period because no intervention occurred. We compared the intervention period gap in vaccination outcomes for NYC against the distribution of gaps for donor pool counties to determine whether the observed gap for NYC was unlikely to have occurred by chance. Intervention period gaps in vaccination outcomes (which were summarised as the RMSPE during the intervention period) were normalised by the preintervention fit of each control group (which were measured by validation period RMSPE) because counties with a poorer preintervention fit would be expected to have larger intervention-period differences in vaccination outcomes in the absence of the intervention. If each county had the same probability of receiving the intervention, the impact in NYC could be interpreted as statistically significant if the intervention-period-to-preintervention-period ratios of RMSPEs in NYC were larger than the ratios of RMSPEs calculated for at least 95% of units in the donor pool, including NYC. Lastly, we constructed a 95% CI as the set of all constant-in-time intervention effects that, if observed, would not be rejected by the ratios of RMSPEs inference procedure using a threshold for statistical significance of p=0·05.

Analyses were conducted in Stata software (version 17). Model training and estimation of the effect of the interventions were done using the Synth package (Cambridge, MA, USA; Redmond, WA, USA). Statistical inference was conducted using the Synth_Runner package (College Park, MD, USA) and the function SCM-cs_v9 (New Haven, CT, USA).24, 25

Role of the funding source

This work was supported by the regular operating funds of the NYC Mayor's Office of Management and Budget, the NYC Department of Health and Mental Hygiene, and the NYC Health and Hospitals Corporation. NYC vaccination data were collected by the NYC Department of Health and Mental Hygiene in the regular course of business. The funders had no role in the analysis, interpretation, or writing of the manuscript. The NYC Mayor's Office of Management and Budget approved the submission of the manuscript. EC and MC had full access to the data. All authors took responsibility to submit the manuscript for publication.

Results

The control group used in the main analysis consisted of Hudson County, New Jersey (weight: 68·7% of participants in the control group, largest city: Jersey City); Wayne County, Michigan (weight: 23·9%, largest city: Detroit); and Shelby County, Tennessee (weight: 7·4%, largest city: Memphis). NYC resembled the control group across characteristics used in the matching process (table 2 ). Focusing on the attributes that are most predictive of vaccination outcomes (appendix p 7), compared with the control group, NYC had a slightly lower percentage of Hispanic people, more non-Hispanic Asian people and non-Hispanic Black people, and its residents were more likely to have voted for the Democratic candidate in the 2020 presidential election, were older, and had lower incomes. Additionally, the share of adults in NYC who received a first dose of vaccine during the 4 weeks before the intervention period was 0·2 percentage points (pp) higher than the share of adults in the control group who received a first dose of vaccine during the same period (3·4 pp vs 3·2 pp), equating to a difference of 13 786 vaccinations (227 695 vs 213 909) when the control group's vaccination rate was applied to NYC's adult population.

Table 2

Characteristics of New York City (NYC) group and the synthetic control group

	NYC (SE)*	Synthetic control (SE)
Race and ethnicity
Hispanic or Latinx	29·1% (NA)†	31·3% (NA)
Non-Hispanic Asian	14·3% (0·1)	11·6% (0·1)
Non-Hispanic Black	21·7% (0·1)	20·2% (0·1)
Age group
16–59 years	60·5% (0·1)	61·9% (0·2)
≥60	21·1% (0·1)	18·8% (0·2)
Sex
Male	47·7% (0·0)	49·2% (0·1)
Female	52·3% (0·0)	50·8% (0·1)
Educational attainment
High school graduate or higher	83·2% (0·2)	86·4% (0·4)
Bachelor's degree or higher	39·2% (0·2)	39·4% (0·6)
Economic attributes
Median household income‡	$69 407 (681)	$70 164 (1286)
Poverty rate	16·0% (0·2)	15·5% (0·5)
Unemployment rate	5·2% (0·1)	4·2% (0·2)
Metropolitan area population
≥1 million	100% (NA)§	100% (NA)
500 000–999 999	0% (NA)	0% (NA)
Partisanship
Democratic vote share in 2020	77·0% (NA)	71·8% (NA)
Vaccination
At least one dose as of July 25, 2021	72·5% (NA)	73·2% (NA)
Received first dose between June 28, 2021, and July 25, 2021	3·4% (NA)	3·2% (NA)

Educational attainment is reported for adults aged ≥25 years. Democratic vote share was calculated as the share of all non-third-party votes earned by the Democratic candidate in the 2020 presidential election. Vaccination outcomes were calculated for adults. NA=not applicable.

SEs for NYC for most outcomes were calculated from the margins of error (90% CI) that were reported by the Census Bureau. SEs for the synthetic control group were estimated from the means and margins of error reported for each county and do not account for covariance across counties. SEs for the share of residents aged 16–59 years were estimated from the point estimates and SEs for the share of the population aged ≥16 years and ≥60 years.

No SEs are provided for the percent of the population that are Hispanic because the share was controlled to the official population estimate.

Estimates for median household income are reported with a SE rather than an IQR because the American Community Survey 1-year estimate data tables did not include the 25th percentile and 75th percentile of household income for counties.

Metropolitan area population, 2020 Democratic vote share, and vaccination outcomes were observed without sampling error.

Variation in vaccination rates might also reflect variation in perceived risk of COVID-19 transmission. We used case rates reported by the CDC as a proxy for perceived risk. NYC's reported rate of COVID-19 transmission was similar to that of the control group (in terms of trend and absolute magnitude) throughout the preintervention and intervention periods (appendix p 8). This resemblance provides evidence to suggest that the difference in vaccination outcomes during the intervention period was not primarily driven by a discontinuous shift in relative transmission rates between the preintervention period and the intervention period.

Figure 1 shows the 7-day average of the share of adults who were reported as receiving a first dose of an authorised COVID-19 vaccine on each date in NYC and its synthetic control group. The vaccination rate for NYC was similar to that of the control group before the intervention. Over the validation period, which was used as the preintervention period for statistical inference, the cumulative sum of the daily vaccination outcome was only 1·2% higher in NYC than in the control group (11·1 pp vs 10·9 pp), equating to a difference of 8606 vaccinations (732 420 in NYC vs 723 813 in the control group) when the control group's vaccination rate was applied to NYC's adult population. At the start of the intervention period, we observed a large increase in the number of vaccinations in NYC relative to the control trajectory. The vaccination rate in NYC exceeded that of the control group on all but 2 days during the intervention period, except for a dip in NYC vaccination rates observed in early October that was attributable to a reporting lag in CDC data.

Figure 1

Daily COVID-19 vaccinations (first dose) for adults in New York City (NYC) and the synthetic control group, 2021

Data are from the Centers for Disease Control and Prevention. The validation period starts at the y-axis. The vertical dashed line indicates the start of the matching period. The vertical solid line indicates the start of the intervention period. The mid-Oct drop and spike in NYC is caused by a lag in reporting.

We assessed the statistical significance of the result by comparing the gap in vaccinations between NYC and its synthetic control group against the distribution of gaps between each donor pool county and its corresponding synthetic control group. Preintervention gaps for most control counties were similar in size to those observed for NYC. However, the intervention-period vaccination gap for NYC was large relative to the distribution of gaps among donor pool counties (appendix p 9). More precisely, the ratio of RMSPEs for NYC was larger than the ratios of RMSPEs that were estimated for all control counties (appendix p 10). If one were to assign the intervention at random among the 45 donor pool counties (including NYC), the probability of obtaining a ratio of RMSPEs as large as that observed in NYC would be 0·022.

We applied the control group's daily adult vaccination rate after July 25, 2021 to NYC's cumulative adult vaccination rate as of July 25, 2021, to derive a counterfactual cumulative vaccination rate for NYC for July 26, 2021 to Nov 1, 2021 (figure 2 ). The cumulative adult vaccination rate for NYC exceeded the cumulative adult vaccination rate of the counterfactual every day following July 30, 2021. The cumulative gap grew to 6·2 pp (95% CI 1·4–10·7]), or 410 201 people (90 966–706 532]), by Nov 1, 2021. Adjusting this impact estimate to account for the preintervention (validation-period) difference in vaccinations lowered the cumulative gap slightly, to 6·1 pp, or 401 741 people (appendix p 11).

Figure 2

Cumulative percent of adults with at least one dose of a COVID-19 vaccine in New York City and the synthetic control group

Data are from the Centers for Disease Control and Prevention. The vertical solid line indicates the start of the intervention period.

We selected a separate control group using age-specific attributes and vaccination outcomes to estimate the policies’ effect on receipt of first vaccine doses for people aged 65 years or older (appendix pp 23–25). NYC's vaccination rate was similar to that of the control group during the intervention period with a few brief, offsetting exceptions (appendix p 22). The cumulative gap in vaccinations through Nov 1, 2021, was 3967 people (95% CI [–145 581 to 156 951]) or 0·3 pp ([–11·4 pp to 12·3 pp).

Lastly, we selected a third control group to estimate the effect of the policies on the share of adults who had completed a primary course, meaning that they received at least two doses of tozinameran (BNT162b2, Pfizer–BioNTech) or elasomeran (mRNA-1273, Moderna), or at least one dose of Ad.26.COV2.S (Janssen; appendix p 23). A separate synthetic control group was selected for this outcome to match on preintervention trends in vaccination series completion. The vaccination series completion rate for NYC was similar to that of the control group closely through the end of August, 2021 (figure 3 ). NYC's vaccination series completion rate more clearly exceeded that of the control group thereafter, excluding a dip in early October that was attributable to a reporting lag in CDC data. The cumulative gap through Nov 1, 2021, was 270 401 people (95% CI –193 125 to 714 129) or 4·1 pp (–2·9 pp to 10·8 pp]), which was smaller than the gap observed in the main analysis.

Figure 3

Daily COVID-19 vaccination series completion for adults in New York City and the synthetic control group

Data are from the Centers for Disease Control and Prevention. The validation period starts at the y-axis. The vertical dashed line indicates the start of the matching period. The vertical solid line indicates the start of the intervention period. The vertical dotted line indicates the first date on which someone who received the first dose of a two-dose series at the start of the intervention period could have become eligible for a second dose of vaccine.

Discussion

The proof-of-vaccination requirement, incentive payments, and employer-based mandates that were implemented in NYC were associated with an additional 410 201 adults receiving at least one dose of an authorised COVID-19 vaccine, corresponding to a 6·2 pp increase in the vaccination rate. The policies were also followed by a lagged and attenuated increase in the share of adults who completed a primary course, suggesting that many, but not all, New Yorkers who received their first dose in response to the policies chose to receive a second dose of vaccine. No effect was observed for adults aged 65 years or older. However, a smaller effect was expected for this age cohort because fewer older adults remained unvaccinated at the time of programme implementation and because employer-based mandates were less relevant for this age cohort than for younger adults.

An incremental increase in COVID-19 vaccinations, such as that observed in NYC, has important health benefits. The proximal benefits of COVID-19 vaccination for vaccinated individuals are well established.27, 28, 29 More distally, the increased number of vaccinations resulting from the interventions might have improved health outcomes for others by reducing community transmission. For example, one modelling analysis that incorporated this effect on community transmission found that NYC's vaccination campaign as a whole averted 47 895 deaths (95% CI 44 234–51 579) and 303 495 hospitalisations (281 642–324 432) through Jan 31, 2022 (Moghadas SM et al, unpublished). As previous modelling exercises have shown, some impact on health outcomes holds even if the policies moved vaccinations forward in time rather than increasing the maximum number of people vaccinated. Some NYC residents who otherwise might have received a vaccine after an infection instead got vaccinated beforehand, improving their expected health outcomes. On an aggregate level, a forward shift in vaccinations could slow transmission, so that individuals who might have been infected before their vaccination date instead got infected afterward, improving their expected health outcomes.

Our analysis adds to an existing literature on the effect of policies designed to increase demand for COVID-19 vaccinations. Previous studies have focused on individual interventions and have found mixed results.6, 7, 8, 9 Conversely, our analysis estimated the joint effect of a programme of interventions. Analyses of individual policies can help policy makers to decide whether to implement any one candidate intervention; this analysis shows what can be achieved by a structured and multipronged programme.

The impact of NYC's policies on rates of vaccination is best understood in the context of NYC's broader vaccination campaign. Starting before the intervention period, NYC worked to make vaccination universally accessible through mass vaccination sites, community-based points of access (eg, pharmacies), and an in-home vaccination programme. Additionally, NYC sought to address vaccine hesitancy and persuade individuals to get vaccinated through a broad public marketing campaign. The policies studied in this analysis were targeted at individuals for whom earlier outreach and access programmes were insufficient. First, NYC introduced a financial incentive to induce vaccinations among individuals who were open to vaccination but who needed some external stimulus to do so. The incentive was set to $100 (which represents 0·1% of the median household income in NYC) to be meaningful, but not coercive. Second, NYC added a proof-of-vaccination requirement for particular indoor spaces so that individuals would choose to get vaccinated to access these amenities. The proof-of-vaccination requirement applied to restaurants, bars, gyms, and theatres because visits to these locations were discretionary and because restaurants, bars, and gyms are associated with a high transmission risk. Lastly, NYC added employer-based mandates with no testing opt-out to reach remaining unvaccinated individuals.

The composition of the portfolio of policies implemented in NYC warrants close attention. By design, the policies targeted distinct, but overlapping, populations. Each component policy might have induced some vaccinations among individuals who would not have chosen to get vaccinated under any of the other policies in isolation. For example, people who were not covered by employer-based mandates might have sought vaccination to access indoor dining. Additionally, we hypothesise that the policies might have acted synergistically on some residents, generating vaccinations that might not have occurred if only some of the policies had been implemented. For example, some residents might have chosen to get vaccinated to receive $100 and to access indoor dining, but the same residents might not have sought vaccination to receive $100 alone or to access indoor dining alone.

We identified three major limitations to the design of this study. First, we could not control for any time-varying impact of differences in messaging, outreach, or vaccine delivery infrastructure on rates of vaccination. NYC and the control counties each took some actions to make vaccines accessible and to persuade individuals to get vaccinated. The impact estimate might be biased if changes in messaging, outreach, or vaccine delivery infrastructure between the validation period and the intervention period affected the number of vaccinations that occurred during the intervention period in NYC or the control group. Differences in messaging, outreach, or vaccine delivery infrastructure that remained constant throughout the preintervention and intervention periods might also have had a time-varying effect on the daily difference in vaccinations between NYC and the control group. Three programmes implemented in NYC are the most likely sources of a discrepancy between the estimated impact and the true impact of the proof-of-vaccination requirement, incentive payments, and employer-based mandates: (1) a $100 referral incentive for community-based organisations (the NYC vaccine referral bonus), launched June 14, 2021; (2) a $35 million programme to reimburse primary care physicians for proactive vaccination counselling conversations with participants in some Medicaid and Medicare Advantage programmes (the Vaccine Outreach and Counseling program), launched Sept 20, 2021; and (3) vaccine outreach by community health workers in underserved neighborhoods (the Vaccine Equity Partner Engagement project), launched Aug 26, 2021. Our impact estimate reflects the differential effect of these programmes during the intervention period.

Second, there might have been a confounding factor that explains both the robustness of NYC's vaccination policies relative to those implemented in the control group and either: (1) the comparatively high vaccination rate in NYC during the intervention period or, (2) the high vaccination rates in control counties during the matching period. For example, the higher level of media attention in NYC could have resulted in both more aggressive policy choices and a more acute impact of heightened SARS-CoV-2 transmission on vaccinations. Replication in other contexts could alleviate concerns of confounding.

Lastly, the inclusion of primary series completion and receipt of first doses by adults aged 65 years or older as secondary outcomes raised the risk of observing a false positive due to multiplicity. This concern was moot, as neither exploratory analysis resulted in a statistically significant outcome, even before controlling for multiple comparisons.

We also identified three limitations that were specific to the selected control group. First, the small number of counties included in the control group heightened the risk that an idiosyncratic shock in any control county could bias the results. To mitigate this risk, we replicated the analysis three times, omitting each of the control counties from the donor pool in sequence (appendix pp 13–14). All replications produced similar impact estimates.

Second, Hudson County and Shelby County were affected by less stringent versions of some of the policies implemented in NYC. Hudson County was covered by statewide vaccine-or-test mandates for state employees, health-care workers, and school staff; all were introduced during the intervention period. Shelby County implemented a vaccine-or-test mandate for county government employees, also during the intervention period. If these policies were successful, our impact estimate is biased downward.

Lastly, the effect of NYC's policies could be underestimated if the proof-of-vaccination or workplace mandates had spillover effects in Hudson County. Spillover might have had only a small effect because most NYC government employees are required to live in-state and because Hudson County residents can access restaurants, bars, and fitness centres outside of NYC.

Despite these limitations, our study provides empirical evidence that a combination of vaccine policies, including a proof-of-vaccination requirement, financial incentives, and employer-based mandates, can increase adult COVID-19 vaccination rates and should be considered in jurisdictions that are struggling to reach high rates of adult vaccination. Other jurisdictions could find it helpful to sequence interventions so that earlier incentives lay the groundwork for later mandates and to design a portfolio of interventions with at least one relevant intervention for every subpopulation of unvaccinated people.

Data sharing

All data used in the analysis are publicly available and can be downloaded from links provided in the reference list.

Declaration of interests

DAC, TL, and JKV were involved in formulating the studied policies as the commissioner of the NYC Department of Health and Mental Hygiene (DAC), the senior vice president for Ambulatory Care and Population Health at NYC Health and Hospitals (TL), and the New York City's Mayor's senior advisor for public health (JKV). All other authors declare no competing interests.