Frames that matter: Increasing the willingness to get the Covid-19 vaccines.

Herd immunity against Covid-19 demands a high rate of vaccination, which may be challenging to meet given vaccine hesitancy in the U.S. How can Americans' willingness to get vaccinated be increased? Using a survey experiment, we apply seven framing treatments to a representative sample of 1642 U.S. residents that test ways to increase willingness: expert and political figure endorsements, demonstrations of receiving the vaccine, information about the vaccine's approval process, and information underscoring the pandemic's devastating economic impact. We find the approval process and the economy treatments increase the odds of higher vaccination willingness by 42% (p = 0.068) and 46% (p = 0.060), respectively. Additionally, we find suggestive evidence that the effectiveness of a message depends on whether a respondent finds the message/messenger credible. The study advances the understanding of vaccine hesitancy by demonstrating effective public health messaging strategy can facilitate greater acceptance of vaccination.

Introduction

The toll that the Covid-19 pandemic has taken on countries around the world is staggering. In the United States alone, the number of deaths surpassed 500 thousand by March 2021. Fortunately, the rapid development of highly effective vaccines (Polack et al., 2020; Baden et al., 2020) offers hope for achieving herd immunity and containing the disease's spread (Haas et al., 2021; Tande et al., 2021). Yet, herd immunity can be achieved only if a large majority of a country's population gets vaccinated; for example, Kwok et al. (2020, e32) estimate that figure at 69.6% for the U.S. However, surveys taken before and after the commencement of the distribution of vaccines (see Funk and Tyson, 2020; Saad, 2021; Kaiser Family Foundation, 2021) suggest that a sizeable portion of the eligible American population is vaccine hesitant, which “refers to [a] delay in acceptance or refusal of vaccination despite availability of vaccination services” (MacDonald, 2015, 4161).

A key question for American public policy is how to boost willingness to get vaccinated, so that herd immunity becomes achievable more quickly—or at all. What approaches could potentially boost an individual's positive inclination toward the vaccine? We start with the hypothesis that intentions to be vaccinated can be increased through a selection of messengers and messaging, which creatively reorient how individuals evaluate the situation, invoke trust in vaccination, and emphasize the salience of the issue, thus increasing the intention to take the vaccine in a timely manner (Wood and Schulman, 2021).

To better understand vaccination intention, we conducted a survey experiment on a representative sample of 1642 American residents in February of 2021. The survey included seven treatments designed to measure the relationship between different messengers/messaging and the intention to receive a vaccine. Survey experiments randomly vary a framing message across subject groups in order to isolate an independent treatment effect on respondent preferences (Tversky and Kahneman, 1981; Druckman, 2001). These frames act as priming tools to illicit a directional response from subjects (Zaller, 1992; Chong and Druckman, 2007), including on important factors related to vaccination hesitancy (Nyhan et al., 2014; Chen and Stevens, 2017). Specifically, our treatments consider the pro-vaccination message or the demonstration of vaccination coming from a public figure, knowledge about the scientific process underlying the vaccine's approval, and information about the negative economic repercussions of the pandemic.

Our dependent variable (DV) is the individual's willingness to take a Covid-19 vaccine based on the following question: “If access is not an issue, are you willing to get the coronavirus (Covid-19) vaccine?” Answers could be “yes, whenever available”, “no, I am not willing”, and “maybe, but not immediately”. For those that answer “maybe”, we subsequently inquire about when they would feel comfortable being vaccinated. Our analysis, then, essentially focuses on the individual's willingness to be vaccinated, which is of course not the same as the individual actually getting vaccinated (see Dai et al. Forthcoming). Nonetheless, willingness constitutes the necessary first step in convincing individuals to receive the coronavirus vaccine (see Bokemper et al., 2021; Kerr et al., 2021; Kreps and Kriner, 2021). In a novel difference from existing studies, the DV in our study allows us to measure the preferred timing of vaccination across respondents. In addition, to better understand attitudes about vaccination, we collected a rich array of socio-demographic information, social and political views, and opinions on health matters.

Our findings suggest that, relative to the control group, respondents' willingness to be vaccinated increases most robustly under two different messaging treatments: when individuals are informed about the Food and Drug Administration (FDA)'s vaccine approval process and when individuals are reminded about the pandemic's economic consequences. The effects of public figures' embrace of vaccination are not as strong, although we find some evidence for their efficacy (see also Bokemper et al., 2021 on public health experts and Bokemper et al., 2021 and Kreps and Kriner, 2021 on messaging by politicians).

Moreover, these findings, which are robust under multiple specification checks, suggest centering a public health communication strategy on messaging content—which boosts information about vaccine safety, as well as the negative impact of the pandemic—may be more effective than relying on specific public messengers' endorsements of the vaccine. Further, our analysis shows other frames, such as endorsements and demonstrations, also have the potential to positively boost willingness, especially if the respondent correctly perceives the intended message. Finally, beyond known correlates, we demonstrate other variables matter for willingness, such as media consumption (which we think proxies information on the pandemic and vaccines), concern about getting sick, and the rate of mortality from Covid-19 in the individual's locality. Overall, the study illuminates potential pathways to reduce vaccine hesitancy.

Survey experiment design

Our survey experiment design, which is further detailed in the Appendix, prioritized two goals. First, we aimed to construct a novel temporal outcome measure that could help ascertain when individuals might be willing to take the vaccine (if not immediately) with nuance beyond what extant surveys show (Funk and Tyson, 2020; Saad, 2021; Kaiser Family Foundation, 2021). As Table 1 shows, our seven-level outcome variable integrates time: an individual willing to be vaccinated immediately receives a 6, the highest quantitative score for the DV; whereas, an individual never willing to be vaccinated gets a 0 (zero), with all other levels staggered in-between. While the aforementioned surveys broadly illuminate what percentage of the population may be willing to be vaccinated, our study advances existing knowledge by examining how quickly some of the hesitant populations may be willing to take the vaccine.

Table 1

Survey experiment conditions and response options.

Group	Independent Variable: Question Vignette
Control Template	As you know, the Covid-19 (coronavirus) pandemic is impacting the United States: about 24 million people have been infected with this virus, and over 400 thousand people have died from it. There are now highly effective Covid-19 vaccines. If access is not an issue, are you willing to get the coronavirus (Covid-19) vaccine?
Actor Demonstration Template	As you know, the Covid-19 (coronavirus) pandemic is impacting the United States: about 24 million people have been infected with this virus, and over 400 thousand people have died from it. There are now highly effective Covid-19 vaccines. The picture below shows[insert actor title, actor name], receiving the vaccine in December 2020. If access is not an issue, are you willing get the coronavirus (Covid-19) vaccine?
Treatment1	“a critical care nurse, Sandra Lindsay”
Treatment2	“the Director of the National Institute of Allergy and Infectious Diseases, Dr. Anthony S. Fauci”
Actor Textual Endorsement Template	As you know, the Covid-19 (coronavirus) pandemic is impacting the United States: about 24 million people have been infected with this virus, and over 400 thousand people have died from it. There are now highly effective Covid-19 vaccines. [Insert actor title and actor name]has publicly endorsed getting vaccinated against Covid-19. If access is not an issue, are you willing get the coronavirus (Covid-19) vaccine?
Treatment3	“The 46th President of the United States, Joseph R. Biden,”
Treatment4	“The 45th President of the United States, Donald J. Trump,”
Treatment5	“Director of the National Institute of Allergy and Infectious Diseases Dr. Anthony S. Fauci”
Substantive Textual Message Template	As you know, the Covid-19 (coronavirus) pandemic is impacting the United States: about 24 million people have been infected with this virus, and over 400 thousand people have died from it. There are now highly effective Covid-19 vaccines. [Insert substantive frame] If access is not an issue, are you willing get the coronavirus (Covid-19) vaccine?
Treatment6	“The Food and Drug Administration's (FDA) 23-member panel of medical experts including physicians, statisticians, chemists, pharmacologists and other scientists—which evaluates new vaccines before they are released to the public—recently approved Covid-19 vaccines for public use.”
Treatment7	“The negative economic impact of the pandemic is similar to the worst recessions this country has experienced—widespread unemployment, business closures, and food and housing insecurity.”
Dependent Variable: Response Options (Coded Numerical Value)
“Yes, whenever available” (6)
“Maybe, but not immediately” [Trigger follow-up question:] “Starting from today, when would you be most willing to get the coronavirus (Covid-19) vaccine?”
“Within the month” (5)
“Between 1 and 3 months” (4)
“Over 3 months–6 months” (3)
“Over 6 months to 1 year” (2)
“Over 1 year” (1)
“No, I am not willing” (0)

Second, we wanted to scientifically test a variety of messages that could increase intention for vaccination uptake in the American adult population. As Table 1 shows, the control condition provided respondents with contextual background, including statistics on the prevalence of the pandemic and the development of highly effective vaccines, then asked respondents whether (and when) they intend to get vaccinated. Our average “yes” rate of 57.1% in the control is comparable to other surveys conducted during a similar time period (see Table 2 ) (KFF 2021). The information contained in the control group provides a stringent test of our framing treatments, since the treatments vary the vignettes beyond this baseline of well-known information to better assess how willingness for the vaccination can be further increased.

Table 2

Demographics and vaccination willingness in the experimental groups.

		U.S. Adult Population		Control Group		Seven Treatment Groups
		N	%	n	%	n Range	% Range
Totala		255.2 M	100%	205	100%	201–208	100%
Agea	18–34	76.2 M	29.8%	63	30.7%	58–64	28.3%–31.3%
	35-49 (35–50)b	62.1 M	24.3%	67	32.7%	67–71	32.5%–35.3%
	50-64 (51–65)b	62.9 M	24.7%	49	23.9%	35–45	16.8%–21.6%
	65+ (66+)b	54.1 M	21.2%	26	12.7%	28–39	13.9%–18.8%
Gendera	Female	130.9 M	51.3%	105	51.2%	104–109	50.0%–53.7%
	Male	124.4 M	48.7%	100	48.8%	93–102	46.3%–49.0%
	Other	N/A	N/A	0	0.0%	0–2	0.0%–1.0%
Racea	White	160.6 M	62.9%	127	62.0%	122–133	60.7%–64.9%
	Black	31.1 M	12.2%	23	11.2%	22–27	10.7%–13.0%
	Hispanic	41.9 M	16.4%	37	18.0%	33–38	15.9%–18.6%
	Asian	15.2 M	6.0%	13	6.3%	11–12	5.3%–5.9%
	Mixed	4.1 M	1.6%	5	2.4%	1–4	0.5%–2.0%
	Other	2.3 M	1.0%	0	0.0%	0–4	0.0%–2.0%
Party IDc	Democrat	N/A	32%	65	31.7%	63–67	31.7%–33.3%
	Independent/Other	N/A	41%	78	38.1%	72–81	37.1%–39.2%
	Republican	N/A	26%	62	30.2%	58–64	27.4%–30.9%
Vaccination Willingnessd	Yes	N/A	55%	117	57.1%	110–132	52.9%–65.7%
	Maybe	N/A	22%	47	22.9%	42–60	20.9%–29.3%
	No	N/A	22%	41	20.0%	23–45	11.1%–21.6%

Notes: Full survey n = 1,642, MOE ± 3.18% (99% confidence), fielded February 9–15, 2021.

Adult, age, gender, and race population source: U.S. Census Bureau, Population Division. 2020. Annual Estimates of the Resident Population by Sex, Age, Race, and Hispanic Origin for the United States: July 1, 2019 (NC-EST2019-ASR6H).

Census age breaks deviate from survey age breaks; the latter is in parentheses where applicable.

Self-identified partisan identity source: Gallup. 2021. Party Affiliation, fielded February 3–18. https://news.gallup.com/poll/15370/party-affiliation.aspx. Survey data does not include raw counts or decimal place.

Vaccination willingness source: Kaiser Family Foundation, 2021. KFF Health Tracking Poll/KFF COVID-19 Vaccine Monitor February 2021. n = 1,874, MOE ± 3% (99% confidence), fielded February 15–23. http://files.kff.org/attachment/Topline-KFF-COVID-19-Vaccine-Monitor-February-2021.pdf. KFF aggregate “only if required” into the “no” category. Survey results do not include raw counts or decimal place.

All seven randomized treatment conditions within the survey were designed to simulate real-world messages that increase the respondent's knowledge by: showing images of different non-partisan medical professionals getting vaccinated (Treatments 1 and 2) or endorsing vaccination (Treatment 5); messages from political leaders across the U.S. political divide (Treatments 3 and 4); substantive framing about the FDA's approval process of the vaccine (Treatment 6); and details about economic conditions during the pandemic (Treatment 7).

These treatments naturally intersect with other comparable studies, but also differ from them in important ways. Like other studies (Bokemper et al., 2021; Kreps and Kriner, 2021), our choice of survey design reckons with the fact that the American population is largely polarized and distrustful of governmental actors, institutions, and policies relative to other advanced democracies (Citrin and Stoker, 2018). Particularly, T3 and T4 aim to assess whether partisanship can be leveraged to increase vaccination propensity by triggering “in-group” sentiment through ideology (Tajfel and Turner, 1986). Throughout, we also take into account factors that relate to political trust, which has been shown to correlate with vaccination sentiment (Lazarus et al., 2021; Kreps and Kriner, 2021; Kerr et al., 2021).

Similar to other studies, we also examine public health experts like Dr. Fauci (Bokemper et al., 2021), but we also advance upon these analyses. Particularly, T1 presents Nurse Sandra Lindsay—a Black woman who was the first individual to receive a vaccine in the U.S. outside of clinical trials—getting vaccinated. Our primary goal by including Nurse Lindsay was to have reference to a public health expert who was widely publicized, but not a career federal employee like Dr. Fauci (our T2 and T5). We recognize that Nurse Lindsay could potentially channel group representation to increase trust in vaccine safety and efficacy within the African American community (Mansbridge, 1999; Tate, 2003), but testing this hypothesis was not our primary goal. Also, by including both a Dr. Fauci demonstration (T2) and an endorsement (T5), we aimed to analyze potential visual versus textual effects.

Importantly, two treatments—regarding the FDA approval process and explaining pandemic-related economic consequences—are constructed as valence appeals meant to solicit favorable reactions across different respondents. The approval process treatment is intended to increase trust in the vaccine's safety, while the economy treatment hypothetically crystalizes threats to economic livelihood and cues a desire for normalcy. Given the salience of vaccine safety in encouraging individuals to be vaccinated, studies similar to ours have grappled with how to elicit confidence in the vaccines (Kerr et al., 2021), but to our knowledge, we are the first to examine the specific FDA treatment. We are also not aware of any other study that uses information about the effect of the pandemic on the economy as a treatment.

As emphasized earlier, experimental framing research relies on randomly varying messages across subject groups to isolate an independent treatment effect on preferences. However, the very mechanism undergirding the enterprise could be its undoing if other survey questions prime respondent reactions to the experimental conditions (McFarland, 1981; Iyengar and Kinder, 1987; Gaines et al., 2007). To minimize this risk, we required the randomized experimental treatment to be the first question a respondent receives, followed by other randomized questions. This approach privileges the validity of the experimental conditions. Furthermore, our key control variables, such as socio-demographic indicators, have low risk, if any, of being susceptible to priming, since they are largely descriptive (age, race, gender) and/or deeply held (partisan affiliation). This method provides the greatest security within the survey experimental design by guaranteeing the treatments are not influenced by other questions (Gaines et al., 2007), thereby ensuring we capture respondents’ real-world sentiments (Sniderman, 2011, 109–110).

Before fielding the survey, we sought and received approval from [Name Redacted for Review]'s Institutional Review Board. We then contracted with Qualtrics to access their diverse national pool of respondents. Qualtrics fielded the survey from February 9th to 14th and yielded a representative sample of 1642 American adults. To guarantee representativeness, our sample used a quota system on race, gender, partisan affiliation, and age within each of the eight experimental groups. This kind of within-treatment representativeness is a more rigorous approach to ensure internal validity (between the reference groups) and external validity (for generalizability) than survey-wide representativeness.

Table 2 details this within-group balance by comparing our sample to the demographic breakdown of the U.S. adult population. The sample is well-balanced: groups reach near parity on all metrics, except the control group skews younger, more Republican, and less Black. That said, given the random assignment into the experiment's control and treatment groups, our results are reasonably generalizable. Finally, the table also presents existing metrics on vaccination willingness for the U.S. adult population in comparison to the values we observe in our control and treatment groups.

The full codebook and an additional robustness check for representativeness are presented in Appendix Tables A1 and A2, respectively. Using G*Power (Erdfelder et al., 1996), our study is estimated to capture an effect size of 0.3 (standard deviation) between control (n = 205) and treatment group (lowest n = 201) at the α = 0.05 level (two-tailed; power = 0.85). According to G*Power, this is considered a small effect size, suggesting our study is attuned to both large and acute statistical differences. We used STATA/IC 15.0 in our statistical analyses and the data (as a.csv file) and code (as.dta and.do files) are available at: https://github.com/akayaor1/Covid-19VaccineWillingnessProject/.

Results and discussion

To analyze treatment effects, Fig. 1 presents the t-tests between the control group (x̅ = 4.078, 95% CI = 3.738–4.419) and each of the treatment groups. Neither demonstration, by Nurse Lindsay (T1: x̅ = 4.412, 95% CI = 4.096–4.727, p = 0.157) nor by Dr. Fauci (T2: x̅ = 4.412, 95% CI = 3.757–4.409, p = 0.984), significantly change willingness to be vaccinated. While the Nurse Lindsey treatment mean is larger than that of the Dr. Fauci treatment, it is still not statistically significantly different from the control mean. Similarly, all three textual endorsement means (T3: x̅ = 4.239, 95% CI = 3.907–4.571, p = 0.505; T4: x̅ = 4.272, 95% CI = 3.943–4.601, p = 0.420; and T5: x̅ = 3.928, 95% CI = 3.587–4.268, p = 0.539) are not statistically different from the control mean. The insignificance of both Fauci treatments (T2 and T5) means there is inadequate evidence to contrast a show of demonstration versus a message of approval by the same expert here. It is nonetheless telling that the demonstration had no apparent impact on willingness.

Fig. 1

Vaccination willingness and treatment effects (n = 1642).

In contrast, the FDA approval process (T6) is associated with an increase in the mean vaccination willingness from 4.078 in the control to 4.548 (95% CI = 4.254–4.842, p = 0.040). Relative to the control, a description of America's economic woes during the pandemic (T7) shows higher pro-vaccine sentiment with a mean of 4.512 (95% CI = 4.197–4.828, p = 0.066). T6 and T7 likely increase willingness in different ways—the former provides reassuring information that a relatively large group of experts have weighed in and have converged on the decision, while the latter plausibly invokes both empathy and fear. In this light, frames that increase substantive knowledge on salient matters appear to boost vaccination willingness.

We next run a series of ordered logit models, which allow us to examine other correlates of willingness and account for any heterogeneity across treatment groups that our aforementioned quotas may not entirely eliminate. We use ordered logit because our DV is an ordinal variable with interval levels that hierarchically order the data. These results are presented in Table 3 , with the coefficients as odds ratios (OR) and robust standard errors (SE) in parentheses. An odds ratio of higher than 1 indicates greater odds (for willingness), while less than 1 means lower odds. While the experimental analysis is conducive for causal assessments, Models 2 through 5 are better understood as providing evidence of correlation between variables and odds of greater vaccination willingness.

Table 3

Ordered logit models predicting vaccination willingness.

Variables	Model 1: Experimental Treatments	Model 2: Socio-Demographics	Model 3: Political Views/Engagement	Model 4: Media Exposure	Model 5: Personal Health Status	Model 6: Local Covid-19 Situation
T1 Nurse Lindsay (Demo)	1.286(0.250)	1.243(0.256)	1.188(0.256)	1.198(0.260)	1.094(0.244)	1.084(0.243)
T2 Dr. Fauci (Demo)	0.997(0.189)	0.922(0.178)	0.887(0.174)	0.878(0.174)	0.937(0.196)	0.934(0.197)
T3 President Biden (Text)	1.172(0.235)	1.130(0.231)	1.114(0.227)	1.138(0.235)	1.007(0.217)	1.019(0.222)
T4 President Trump (Text)	1.171(0.231)	1.140(0.232)	1.269(0.262)	1.270(0.264)	1.163(0.253)	1.151(0.253)
T5 Dr. Fauci (Text)	0.871(0.167)	0.906(0.179)	0.975(0.199)	0.987(0.202)	1.110(0.233)	1.112(0.236)
T6 FDA Approval (Text)	1.420*(0.273)	1.383(0.280)	1.403*(0.285)	1.352(0.278)	1.092(0.236)	1.085(0.236)
T7 Economy (Text)	1.461*(0.295)	1.455*(0.304)	1.443*(0.309)	1.403(0.303)	1.333(0.303)	1.312(0.300)
Age Cohort		1.179***(0.048)	1.305***(0.058)	1.249***(0.057)	1.146***(0.057)	1.138***(0.057)
Black		1.199(0.193)	0.607***(0.109)	0.574***(0.104)	0.572***(0.104)	0.577***(0.106)
Hispanic		1.554***(0.215)	1.015(0.155)	0.992(0.154)	0.895(0.144)	0.860(0.140)
Asian		1.251(0.286)	0.823(0.197)	0.850(0.208)	0.709(0.203)	0.668(0.195)
Mixed Race		0.781(0.316)	0.470*(0.204)	0.474*(0.207)	0.453(0.236)	0.479(0.266)
Other Race		0.331**(0.162)	0.373**(0.167)	0.390**(0.173)	0.380**(0.146)	0.374***(0.137)
Female		0.659***(0.068)	0.694***(0.076)	0.701***(0.077)	0.635***(0.073)	0.637***(0.074)
Non-Binary		0.971(0.780)	0.751(0.485)	0.694(0.470)	1.486(1.285)	1.491(1.305)
Education Level		1.160***(0.045)	1.115***(0.047)	1.093**(0.047)	1.067(0.047)	1.056(0.047)
Income Level		1.111***(0.026)	1.113***(0.028)	1.106***(0.028)	1.110***(0.030)	1.105***(0.030)
Religious		0.900(0.094)	0.938(0.104)	0.912(0.101)	0.831(0.097)	0.836(0.098)
Party Likert			0.808***(0.022)	0.808***(0.022)	0.847***(0.024)	0.858***(0.024)
Trust in Government			1.674***(0.096)	1.618***(0.094)	1.457***(0.090)	1.449***(0.090)
Government Services Index			2.298***(0.594)	1.815**(0.470)	1.580(0.454)	1.597(0.461)
Media Consumption Index				3.154***(0.806)	2.606***(0.708)	2.455***(0.669)
Self-Identified Health Level					1.111*(0.068)	1.116*(0.069)
Receive Flu Shot					3.608***(0.415)	3.635***(0.420)
Serious Health Conditions					1.027(0.138)	1.051(0.143)
Concern About Getting Sick					1.613***(0.101)	1.617***(0.102)
Know Covid Patient					0.889(0.105)	0.894(0.106)
Covid Incidence per 100 People by County						0.966(0.022)
ln(Covid Deaths by County)						1.081**(−0.036)
n	1,642	1,642	1,642	1,642	1,642	1,640a
Model χ2	12.5	142.6	317.5	326.1	452.2	467.2
df	7	18	21	22	27	29
Loglikelihood	−2,134	−2,066	−1,964	−1,953	−1,847	−1,841
Pseudo R2	0.003	0.035	0.082	0.088	0.137	0.139
Odds ratio coefficients with robust standard errors in parentheses
*** p < 0.01, ** p < 0.05, * p < 0.1

Notes: a Some smaller counties in Utah report pooled Covid-19 metrics, leading to two less respondent observations.

The results in Model 1 present our primary findings. For respondents in the FDA condition, the odds of having a higher vaccination willingness are 1.42 times—or 42%—higher than that of respondents who were in the control group (OR = 1.420, SE = 0.273, p = 0.068). Similarly, those in the economy condition are 46% more likely to have a higher willingness value (OR = 1.461, SE = 0.295, p = 0.060).

Model 2 focuses on sociodemographic variables. Consistent with existing research (Ruiz and Bell, 2021), age increases the odds of having a greater vaccine willingness by 17.9% (OR = 1.179, SE = 0.0.048, p < 0.001), while respondents with higher educational attainment (OR = 1.160, SE = 0.045, p < 0.001) and higher income (OR = 1.111, SE = 0.026, p < 0.001) also have a higher likelihood of greater vaccination willingness. Moreover, in this model, Hispanics are 55.4% (OR = 1.554, SE = 0.215, p = 0.002) more likely to report greater willingness to vaccinate than whites, but this finding loses significance with additional control variables in subsequent models. Finally, consistent with existing research (Kreps and Kriner, 2021, 3256; Kreps et al., 2021, 4), women's odds of greater willingness to vaccinate were 34% lower compared to the male reference group (OR = 0.659, SE = 0.068, p < 0.001), and this pattern holds across all model specifications. With the added demographic controls, the economy treatment maintains significance (OR = 1.455, SE = 0.304, p = 0.073), while FDA approval falls out of conventional levels (OR = 1.383, SE = 0.280, p = 0.109).

Model 3 shows that sociopolitical views are also significantly associated with vaccination willingness. The more one leans Republican on a seven-point Likert scale, the less likely they are to hold a greater willingness by 19.2% (OR = 0.808, SE = 0.022, p < 0.001). This mirrors findings from similar studies constructed at different points in time (Kreps and Kriner, 2021, 3255). Table A3 in the Appendix substitutes a measure of ideology for party, showing conservative respondents have lower odds of greater vaccination willingness (OR = 0.781, SE = 0.027, p < 0.001). A non-surprising finding—but formerly insignificant in Model 2—is the lower likelihood of Black individuals, relative to Whites, by 31% toward (higher) vaccine willingness (OR = 0.607, SE = 0.109, p = 0.006). This is consistent with previous research that finds due to unethical experimentation (Washington, 2006), discrimination in the healthcare system (Chapman et al., 2013), and a general lack of trust in the political system (Woko et al., 2020), this population displays greater hesitancy toward vaccination.

Additionally, Model 3 explores the impact of general political trust and interactions with government. Recent research shows trust in government remains an important predictor of various attitudes, including vaccination sentiment (Lazarus et al., 2021). Our findings in Model 3 confirm the importance of generalized political trust: subjects higher in political trust have a 67% greater odds of higher willingness values (OR = 1.674, SE = 0.096, p < 0.001). Finally, an index of eight interactions with the government (labelled as government services index), including renewing a driver's license or calling 311, is initially positive and significant (OR = 2.298, SE = 0.594. p = 0.013) in Model 3, but the p-values for the variable decline in subsequent models (Model 4: p = 0.021; Model 5: p = 0.112; Model 6: p = 0.104). Once again, the economy treatment continues to hold (OR = 1.455, SE = 0.304, p = 0.073), and FDA regains significance (OR = 1.455, SE = 0.304, p = 0.073) in Model 3 when accounting for demographic and political covariates.

The media index in Model 4 is not found in prior studies related to this topic, but we included it to assess exposure to information, which existing work suggests as boosting inclinations toward taking the flu vaccine (Chen and Stoecker, 2020). While the notion of fake news suggests that more information is not necessarily good, we find the respondents with the maximum media intake across a battery of four news sources—newspaper, radio, television and internet—are over three times as likely to report higher vaccination willingness (OR = 3.154, SE = 0.806, p < 0.001). Plausibly, the greater information level about the pandemic's devastating health and societal effects may trigger both individualistic and societal concerns. With the inclusion of this media variable, both FDA (OR = 1.352, SE = 0.278, p = 0.142) and economy (OR = 1.403, SE = 0.303, p = 0.116) treatments fall out of conventional levels of significance in Models 4 and beyond. Plausibly, the treatments may not reveal any new information for respondents that are already well-informed about vaccine safety and the pandemic's economic consequences.

Model 5 illuminates how an individual's considerations about their own health impact their inclination toward receiving the vaccine. We find that individuals who have a habit of receiving the flu shot are over three and a half times more likely to indicate a higher willingness to receive the Covid-19 vaccine (OR = 3.608, SE = 0.415, p < 0.001), consistent with extant research (Ruiz and Bell, 2021). Additionally, individuals who express concern about contracting Covid-19 have 61% higher odds to hold a greater willingness value (OR = 1.613, SE = 0.101, p < 0.001). Moreover, individuals with a higher rating of their personal health have higher odds of greater willingness (OR = 1.111, SE = 0.068, p = 0.087). Consistent with current research (Kreps and Kriner, 2021), knowing a Covid-19 patient (OR = 0.889, SE = 0.105, p = 0.321) is not correlated with the DV. The same is true for the variable that captures serious health conditions, including severe allergies (OR = 0.981, SE = 0.133, p = 0.889). These apparent non-relationships are plausible given the heterogeneity of mechanisms at play: knowing a Covid-19 patient may expose a respondent to firsthand experience with severe illness or facilitate them downplaying the disease if the patient anecdotally had a mild case. Severe health conditions make personal health salient and may spur one to get vaccinated or it may prevent them from such a path if they believe the vaccine will, for example, trigger their allergies.

Finally, in Model 6, we examine how the societal impacts of Covid-19 relate to an individual's odds of greater vaccination willingness. One obvious place to start is to test the relationship between local death toll and vaccination sentiment. Since the variable for county-level Covid-19 deaths produces a skewed distribution, we use the natural log of the variable to handle skewness and outliers. Model 6 indicates that an increase in log deaths is associated with about 8% higher odds that the respondent will have a greater willingness to vaccinate (OR = 1.081, SE = 0.036, p < 0.019). For ease of interpretation, we also computed the predicted probability of deaths in relation to likelihood of greater vaccination willingness (not shown in table). Holding all other variables constant at their means, going from approximately one standard deviation below the mean value for logged Covid-19 deaths (6.14) to roughly one deviation above the mean (from 4 to 8), on average, increases the share of respondents willing to get vaccinated immediately by 7 points, from 57.17% (95% CI: 51.35%–62.98%) to 64.59% (95% CI: 61.47%–67.71%). Finally, the decreased odds based on incidence rate of Covid-19 in the county are just shy of significance (OR = 0.966, SE = 0.022, p = 0.118). Both of these indicators suggest further research is needed to understand the relationship between Covid-19 impact and individual opinion, such as on vaccination.

Since the experiment is fully randomized, the addition of new variables in Models 2–5 does not change the substantive results from Fig. 1 and Model 1. We think some of the loss of treatment significance in Models 4–6 has more to do with the importance of the variables we add to the estimations than the lack of statistical power in our estimations. As Table A4 in the Appendix shows, when we include media exposure and personal health considerations in separate estimations, these covariates are highly significant and lead the treatments to lose significance. We also explore whether the lack of significance in some variables could be due to our sample sizes: simulations on G*Power (Erdfelder et al., 1996) suggest a sample size of 1642 and α = 0.05 level (two-tailed) can detect odds ratios of 1.2 at a conventional threshold for sufficient power (0.83). While this suggests an adequate sample size for most of the variables in our models—including the initial estimates for the FDA and economy treatments—it is still possible that relatively small but statistically significant effects have failed to materialize in these models.

In cases of heavy multiple hypothesis testing (i.e., many explanatory variables and/or outcomes), it may be important to employ a False Discovery Rate (FDR) approach. FDR techniques probabilistically assume Type I errors (incorrectly rejecting the null hypothesis) grow with the number of variables in a model. But, this approach also increases the chance of encountering a Type II error (failing to reject the null when the null is false). Given the stakes of the public health crisis and to provide a conservative estimate of associations, we employ FDR to account for potential Type 1 errors.

Specifically, we employ the Benjamini et al. (2006) FDR approach on Model 6 in Table 3 to ensure none of the significant results are false positives. This approach takes the expected proportion of errors at a given alpha and iteratively deduces potential errors by accounting for the number of variables that are confirmed—the remaining less significant variables are more likely to be false positives. The technique produces q-values, which are essentially error-adjusted p-values. The q-values for the 29-variable Model 6 (shown in Table A5 in the Appendix) maintain the same results as before with three differences: the statistical significance of age and “other” racial identity drop from p < 0.01 to q < 0.05, while one's self-assessment of health status falls just outside of conventional levels of significance (from p = 0.076 to q = 0.129).

For more fine-grained analysis, in Tables A6 to A12 in the Appendix, we also investigate heterogenous treatment effects. While our study did not intend to analyze specific effects on sub-groups, it nonetheless makes sense—within the limitations of our data—to assess whether the treatments affected the hesitant groups differentially. Based on aforementioned existing research and our results, we identify three key hesitant groups: Republicans (differentiated as both Republican identifiers/leaners and Trump voters) in comparison to Democrats/Biden voters, African-Americans, and women. For some of these groups (such as Black respondents), our sample sizes are too small to reach conclusions, as small sample sizes could mean false negatives (in turn, the low-n constraint means the significant findings can generally be trusted). With this drawback in mind, we find the FDA approval process still stands out as the most robust treatment to positively boost willingness, including for African-Americans (Control: n = 23, x̅ = 3.696, 95% CI = 2.571–4.821; T6: n = 25, x̅ = 4.76, 95% CI = 3.985–5.535; p for difference = 0.115) and Republicans (Control: n = 72, x̅ = 3.236, 95% CI = 2.623–3.849; T6: n = 80, x̅ = 3.963, 95% CI = 3.418–4.507; p for difference = 0.080).

The takeaway from these models is one of caution: while interventions like the ones we present can be effective in boosting willingness, individual exposure to information (our media variable) and health considerations—be they individual or about local circumstances—are shown here to have a greater impact on vaccine willingness.

Finally, we probe our consistent finding that demonstrations and messaging by specific individuals do not affect the intention to get vaccinated. We hypothesize that individuals are more likely to increase their intention to be vaccinated if they are persuaded the messenger genuinely supports vaccination. To explore this, we utilize the manipulation check question we asked immediately following the treatment to assess whether respondents understood the intended message. This form of “internal analysis” allows us to assess whether respondents comprehended the treatment, and subsequently analyze the sub-sample of respondents who did (Wilson et al., 2010, 66).

The manipulation check question template reads: “Considering what you just read about [treatment actor], how would you describe [the treatment actor's] attitude toward the Covid-19 vaccine?” The correct answer across conditions is “[treatment actor] approves of the vaccine.” Due to the nature of the vignette, the economy treatment check asks whether the pandemic positively or negatively impacted the economy, with the correct answer being negative. This particular type of assessment is known as a “factual manipulation check”, since there is a correct and an incorrect answer (Chandler et al., 2014, 121). Heretofore, we have examined the full sample. While it is imperative for researchers to always focus on the full sample, examining those that passed the manipulation check can provide extensions to the core analysis (Montgomery et al., 2018; Aronow et al., 2019, 572–573). Hence, we now turn our focus to only those that passed the manipulation check.

We begin by hypothesizing about why people might fail the manipulation check. Two primary reasons come to mind: one, the respondent did not fully comprehend or pay attention to the treatment; two, the respondent did not believe the message being communicated, either out of personally-held distrust or a perception that the messenger was not genuine. The latter explanation suggests that for framing effects to work, the messengers need to be perceived as credible endorsers to overcome an individual's existing views (Lodge and Taber, 2013). We examine both options, but it is important to note at the onset that in the absence of another experiment to more directly observe “perceived credibility” of the message, our discussions are suggestive at best.

Our survey includes a measure for the duration (in seconds) each respondent took to answer the entire survey, which we use to examine the correlation between the respondent's speed of going through the survey—which could proxy lack of attention in general—and passing the manipulation check. We find a very weak correlation between duration and passing each manipulation check by experimental group (Pearson's r range across groups = −0.125–0.070) These numbers suggest, as best as we can tell, it is not attention per se that causes failure in the manipulation check.

If it is lack of comprehension (i.e., option 1), our most difficult treatments could reasonably have the highest attrition rate (i.e., failing the manipulation check). Since the demonstrations literally show Nurse Lindsay and Dr. Fauci receiving vaccinations, we would expect relatively low and comparable levels of attrition in these two treatments. However, this turns out not to be the case. As Fig. 2 visually depicts, 61 subjects failed the manipulation check (52 “cannot tell” and 9 “disapprove”) in the Nurse Lindsay treatment (T1), while that number is only 34 (28 “cannot tell” and 5 “disapprove”) in the Fauci demonstration (T2).

Fig. 2

Passage and failure of the manipulation check by experimental group.

Given the demonstrations are easy to understand and the lack of meaningful correlation between the demonstrations and the speed of answering the survey, it is conceivable that the manipulation check proxies how credible/convincing the respondents find the specific messenger/message. Indeed, out of our groups, T4 (Trump endorsement) had the largest attrition with 80 (38.8%) of the 206 group members getting the manipulation check wrong (24 “disapprove” and 56 “cannot tell”). This was unsurprising to us because of Trump's oscillation on public health issues and the notable role of anti-vaccination groups within his political constituency (Oliver and Wood, 2018). Further, T4 consists of 33 Democrats, 13 Republicans, and 34 independents or third-party members, or alternatively, 36 Biden voters, 11 Trump voters, 7 voters in support of other candidates, and 26 non-voters, suggesting a partisan skew in failing this condition. In contrast, our relatively more interpretive treatments—the FDA and economy ones—had lower attrition rates: 37 and 14 manipulation check failures, respectively.

All this said, we recognize that there could be a third option: individuals, who are already inclined to get the vaccine could be more likely to pass the manipulation check. Similarly, it could be that the respondent interprets the message as more credible when they have an a priori inclination for vaccination. And finally, it is possible people understand the direction of the treatment, but ex post facto fail the manipulation check to be consistent with their stance on the issue, thereby minimizing cognitive dissonance. Despite randomization, we are not able to arbitrate these possibilities in the absence of the aforementioned research design centered on the mechanisms of respondent information processing. Nonetheless, it is important to recognize the differences in the sample of individuals who passed the manipulation check to gain a fuller sense of possible treatment effects.

Indeed, examining only the sub-sample of those that passed the manipulation check, the means tests in Fig. 3 indicate more of the treatments could boost the respondents’ willingness to be vaccinated. Specifically, economy (T7: x̅ = 4.572, 95% CI = 4.251–4.894, p = 0.039), Nurse Lindsay (T1: x̅ = 4.741, 95% CI = 4.392–5.091, p = 0.010), and FDA approval (T6: x̅ = 4.883, 95% CI = 4.583–5.183, p < 0.001) treatments are all positively significant. Further, the treatments communicating endorsements by President Biden (T3: x̅ = 4.464, 95% CI = 4.116–4.811) and former President Trump (T4: x̅ = 4.524, 95% CI = 4.108–4.940) reach near conventional levels of statistical significance with p-values of 0.120 and 0.106, respectively. Conversely, neither of the Fauci conditions (T2: x̅ = 4.187, 95% CI = 3.835–4.539, p = 0.662; T5: x̅ = 4.213, 95% CI = 3.829–4.596, p = 0.605) are significant. This is in contrast to studies conducted earlier (September 2020), which found Dr. Fauci messaging increased willingness (Bokemper et al., 2021). In short, message recipients generally respond more favorably when they process the intention behind the message and/or find the message credible.

Fig. 3

Treatment effects based on passage of the manipulation check (n = 1342).

To better understand the magnitude of the treatment effects on this group, in addition to other explanatory variables, we again run ordered logit models on the manipulation check sub-sample (full results available in Table A13 of the Appendix). The results are striking: Nurse Lindsay (T1: OR = 1.755, SE = 0.386, p = 0.011), Biden (T3: OR = 1.458, SE = 0.309, p = 0.076), Trump (T4: OR = 1.539, SE = 0.369, p = 0.072), FDA approval (T6: OR = 2.006, SE = 0.424, p < 0.001), and the economy (T7: OR = 1.531, SE = 0.311, p = 0.036) are all statistically significant and increase the likelihood of higher vaccination willingness by a minimum of 1.458 times—or 45.8%. The most successful frame leads respondents in the FDA condition to now be twice as likely to hold a higher willingness relative to the control group. Importantly, the significance of three treatments (T4: OR = 1.717, SE = 0.424, p = 0.028; T6: OR = 1.649, SE = 0.387, p = 0.033; and T7: OR = 1.450, SE = 0.318, p = 0.090) holds in Model 4 with media consumption, while T4 now maintains significance in the final comprehensive Model 6 (OR = 1.644, SE = 0.431, p = 0.058). For those that process the manipulation check correctly, the treatment effects are more durable to covariate inclusion.

While we are unable to definitively pinpoint a specific mechanism, it is clear that respondents that correctly interpreted the directionality of the treatment were, on balance, more willing toward vaccination. We interpret these results to mean that when the messenger is perceived to be sincere, framing effects can be even more powerful. Hence, with some qualification, we consider this a promising finding since it means a wider array of communication approaches, including endorsements, could reduce vaccine hesitancy.

Conclusions and future research

This article contributes to a burgeoning literature on the determinants of vaccine acceptance and how to boost it through messaging and behavioral nudges (Bokemper et al., 2021; Dai et al. Kerr et al., 2021; Kreps et al., 2021; Kreps and Kriner, 2021; Schwarzinger et al., 2021). Although our study naturally intersects with existing analyses in certain dimensions, such as how perceptions of vaccine safety affect the willingness to get vaccinated, it offers a unique survey experiment with original results. In addition to our dependent variable capturing the timeline for intention to be vaccinated, our treatments focus on not just public figure endorsements, but also information about the vaccine approval process and the pandemic's effects on the economy. Having multiple studies (conducted at different points in time and in different countries) applying different techniques to decrease vaccine hesitancy is critical for public health efforts, as extant work—including this study—suggests strategies to boost individual willingness to be vaccinated are crucial first steps in vaccine uptake, and thus herd immunity. We contribute to efforts to understand which strategies are more (or less) likely to be effective, and on whom.

This study shows, first, that an explanation about the FDA's vaccine approval process and an emphasis on the economic toll of the pandemic have the most robust effect on the respondent's willingness to take the vaccine. In particular, the FDA treatment increases the likelihood of greater willingness to be vaccinated by between 42% (full sample) and 100% (manipulation check sub-sample) compared to the control group. Importantly, the FDA treatment appears to work across different hesitant groups. Moreover, the economy treatment has the most persistent effects across the differently specified ordered logit models.

Second, we find that other frames, such as politician endorsements or demonstration of vaccination, could also work, but they remain contingent upon the individual's processing of the stimulus and/or perception of the message's sincerity. Third, this study uniquely documents how fear of catching Covid-19, the level of media consumption, and Covid-19-related mortality in the individual's general area also matter. As such, public campaigns could be crafted to localities and inform the public about the issues of their specific area.

There are, of course, limitations to this study and those of its kind. As noted at the onset, while we measure willingness to be vaccinated as the initial step in public health campaigns for vaccination, the actual vaccination uptake is what ultimately matters for herd immunity. Indeed, existing literature suggests between one-third and half of individuals that state an intention to receive the seasonal influenza vaccine actually followed through with getting vaccinated (Bronchetti et al., 2015, 277–278). This said, the present moment may be associated with greater follow-through, given the pervasiveness of Covid-19, the associated mortalities, and the deep socio-economic effects of the pandemic. Future studies would benefit from devising a wholly contained research design relying on a panel data approach with multiple observations across time per respondent, including verified (as opposed to self-reported) uptake of vaccination. Such a methodological approach allows for monitoring how well intention to vaccinate translates into demonstrated action.

Crucially, motivations—including intentions—are a key precondition for action (Brewer et al., 2017, 158), especially in a U.S. public health regime that generally relies on voluntary compliance and minimal government enforcement. Therefore, measuring one's disposition toward vaccination and identifying persuasive appeals through framing experiments are necessary scholastic endeavors (ibid, 161; Chen and Stevens, 2017, 1071) that may contribute to successful governmental public health initiatives (Benartzi et al., 2017).

To be sure, while persuasive messages may alter intention to vaccinate, it remains unclear how long this intention lasts immediately following the survey. Existing meta-analyses of persistent (or decaying) post-treatment effects through experimental surveys are mixed, but nonetheless find treatments that educate the respondent with new, actionable information are comparatively better at leading to attitudinal or behavioral change than treatments that make preexisting knowledge more accessible or salient (Coppock, 2017). We believe the FDA and economy treatments belong in this knowledge boosting category of treatment.

This study's focus on individual-level intention to vaccinate through an experimental survey yields several successful frames which increase articulated willingness. Information-based communications, of the type tested in this study, appear to present potentially powerful tools to overcome hesitancy in the race to achieve herd immunity.

Credit author statement

Author order is alphabetical and each author contributed equally to this work.

Funding

Lang Center for Civic & Social Responsibility, (SMD, AK, EBM).

Department of Political Science and Provost's Office, (AK).

Department of Economics and Provost's Office, (EBM).

Data and materials availability

All data are available in the main text or the supplementary materials.

Declaration of competing interest

Authors declare that they have no competing interests.