Experimental pre-tests of public health communications on the COVID-19 vaccine: A null finding for medical endorsement, risk and altruism.

OBJECTIVE: Vaccination campaigns against COVID-19 will only be successful if enough people want to take the vaccine. We tested a government communications intervention to encourage uptake.
DESIGN: A pre-registered randomised controlled trial.
METHODS: A large, nationally representative sample were randomly assigned to see one of eight posters. The posters varied by image (general practitioner or two hospital doctors) and message (control with public health guidance not related to vaccination, endorsement of the vaccine from the pictured doctor, endorsement with information about COVID-19 risk, endorsement with information about risk and appeal to get vaccinated to protect friends and family). The posters were presented as part of a larger study. The main outcomes were intention to be vaccinated and how soon people would be willing to be vaccinated.
RESULTS: The posters induced different reactions indicating that participants had engaged with them. The hospital image was generally preferred to the GP image. Perhaps critically, all intervention messages were trusted less than a control message which did not mention the vaccine (Control Poster Mean = 5.65, SE = 0.09 vs. Poster M Mean = 5.18, SE = 0.09, p <.001; vs. Poster M + R Mean = 5.11, SE = 0.09, p <.001; vs. Poster M + R + F Mean = 5.33, SE = 0.09, p =.01). There were no effects of poster type on intention to take the vaccine or how soon people were willing to take it.
CONCLUSION: Although the intervention messages were based on the strongest correlates of vaccine hesitancy identified by contemporaneous surveys, none was effective. More recent research suggests that focusing on the risk of COVID-19 may be less effective than focusing on the benefits of vaccination. Null findings can be as important as positive findings for designing public health campaigns. This study informed government communications about the COVID-19 vaccine.

Introduction

Vaccination is a key protective measure against COVID-19 [1]. While the majority of the population in most countries have either taken or intend to accept a COVID-19 vaccine, there are still significant minorities who are hesitant or resistant to it [2]. These individuals remain at higher risk from infection and illness. If new variants emerge, high uptake of additional rounds of vaccination may be necessary. The latest United Nations recommendation is to focus on communication with vaccine hesitant individuals – those who are not sure whether they want to take the vaccine or not - and to find means of spreading trusted, factual information about the vaccine to this group [3]. Finding the most effective means of communicating with these individuals is critical for global public health.

Women, younger adults, those with children and minority ethnicity groups are more hesitant about the COVID-19 vaccine [4], [5], [6], [7], [8], [9], [10], [11], [12]. Some studies have found that those with lower education and income are also more hesitant, but these associations are less consistent [8], [9], [10], [11], [13], [14]. However, sociodemographic differences are small compared to the effects of psychological differences on hesitancy [12]. Across all sociodemographic groups, people are more likely to be hesitant if they believe that the illness caused by COVID-19 would not be severe or that they are not susceptible to catching COVID-19 [4], [7], [8]. Those who mistrust scientific information are also more likely to fall into a hesitant group [5], [9], [10], [12], [15], [16], [17]. On the other hand, those who have a positive association with healthcare and healthcare professionals are less likely to be hesitant [12].

These factors give some hints about what might form effective communications with hesitant individuals. Some recommendations have been made to try to increase uptake of the COVID-19 vaccine. One recommendation is that healthcare providers endorse the vaccine [3], [11], [18], [19]. Another is to focus on altruism and concern for one’s family, which have both been associated with greater acceptance of the vaccine [5], [6], [20]. As perceived severity of COVID-19 is a strong predictor of acceptance, highlighting the risk of COVID-19 over the risk of the vaccine is also recommended [17], [20].

These recommendations are based on available survey evidence, with few direct tests of different types of messages on intention to take the vaccine. The present paper describes a pre-registered randomised controlled trial that tested the effects of different public health messages on COVID-19 vaccine intentions. Motivated by the recommendations from the survey evidence, this study assessed whether posters highlighting medical endorsement of the vaccine, risk of COVID-19 or vaccination to protect friends and family would influence intention to take the vaccine, compared to a control poster with the general COVID-19 public health advice (see logic model in Fig. 1 ). We also tested whether the image on a poster mattered. We manipulated whether the poster image was of a general practitioner/family doctor (GP) or of hospital doctors. As hospitals are generally associated with more severe illnesses than GP surgeries, and as greater perceived risk of COVID-19 is associated with higher intention to be vaccinated, we theorised that the hospital doctor image may induce a perception of greater severity of disease than the GP image and therefore higher intention be vaccinated. We made the following hypotheses, which were pre-registered on Open Science Framework (OSF) prior to data collection1 :

Fig. 1

Logic model showing experiment design, hypotheses and expected output.

Hypothesis 1: Posters showing a hospital doctor giving a message about COVID-19 vaccination will be associated with stronger intention to be vaccinated than posters showing a GP giving the same message.

Hypothesis 2: Posters that highlight medical endorsement of the COVID-19 vaccine, risk of COVID-19 or vaccination to protect friends and family will be associated with stronger intention to be vaccinated compared to a control poster.

Hypothesis 3: A poster that combines all three messages will have a stronger effect on intention to be vaccinated than a poster that only has medical endorsement of the vaccine or a poster with medical endorsement of the vaccine and information about the risk of COVID-19.

Hypothesis 4: The three posters will induce different reactions in participants compared to the control poster on scales of trust, efficacy, optimism and liking.

The study employed a nationally representative online sample in the Republic of Ireland. It was carried out in January 2021, while the Republic of Ireland was experiencing a third wave of COVID-19 cases. The vaccination programme had begun but was not yet available to most people. At that time only 3.9% of the population had taken any dose of the vaccine and only 0.1% were fully vaccinated [22]. Contemporaneous surveys found that 70% of people who had not yet received a COVID-19 vaccine said they intended to take one, while only 4% said they definitely would not take one [23]. Thus, approximately 1 in 4 people stated openly that they had not yet decided, although some uncertainty may have remained also among those willing to state an intention.

Methods

Sample

A sample of 1600 adults aged 18+ was recruited by a market research agency. The sample was nationally representative based on age, gender, region of residence and socio-economic status. Data collection took place between 21st and 27th January 2021, a few weeks after the vaccination programme had begun but before most people had been offered a vaccine. Participants were invited to take part in a 20-minute online survey about the COVID-19 vaccine. The sample size was selected to allow for sufficient power to test for between-subject differences following exposure to one of eight posters. Participants were paid €4 upon completion of the experiment. The study was approved under the institution’s ethical review policy and informed consent was obtained from all participants.

Survey experiment

The intervention was part of a larger survey on intentions and attitudes towards COVID-19 vaccination. For reasons of space, the results of the larger survey are reported in a separate paper [24]. To control for any possible contamination of the intervention by survey questions, we counterbalanced whether participants got the poster intervention before or after the other survey questions.

The poster intervention had a 2 × 4 stepped design that manipulated the type of image and the message shown on the poster. Two images were used, one of a general practitioner/family doctor (GP) sitting at a desk and the other of a hospital scene with a person on a ventilator. The message content had four levels with either a control message about COVID-19 hygiene measures that did not mention the vaccine (C), endorsement of the COVID-19 vaccine from the medical professional in the image (M), the medical endorsement message and a message highlighting the risk of dying from COVID-19 (M + R) or the medical endorsement and risk messages in addition to a call to protect friends and family by getting vaccinated (M + R + F) (Fig. 2 ). Each participant was randomly assigned to see one poster on screen for a minimum of 10 s, after which they could click to continue. To ensure that participants had paid attention to the poster, they were subsequently asked to select its contents from a list of options. Following this, participants were asked for their opinions of the poster on a series of 7-point numeric response scales. They were asked how effective they thought the poster was (from 1 = ‘very ineffective’ to 7 = ‘very effective’), how optimistic it made them feel (from 1 = ‘very pessimistic’ to 7 = ‘very optimistic’), how much they trusted it (from 1 = ‘not at all’ to 7 = ‘a lot’) and how much they liked it (from 1 = ‘not at all’ to 7 = ‘a lot’). Participants then completed a series of questions assessing demographic characteristics including age, gender, education, employment, and nationality. In the middle of these questions, participants were asked to rate how likely they were to take the COVID-19 vaccine when offered, using a scale ranging from 1 (extremely unlikely) to 7 (extremely likely). They were also asked when they would take the vaccine once it was available to them with the options “as soon as it is available”, “a few weeks after”, “a few months after”, “a year or more after” or “never”.

Fig. 2

Four of the eight posters shown to participants. Note. Participants saw one of four messages: (a) control, (b) medical endorsement (M), (c) medical endorsement + risk (M + R), (d) medical endorsement + risk + protect friends and family (M + R + F). Each message was paired with either the GP image (e.g., a and c) or the hospital image (e.g., b and d).

The pre-registration documents, dataset and analysis syntax are available on OSF.2

Results

Of the 1600 participants, 10 indicated that they were unable to take the vaccine for medical or religious reasons. We removed these participants from further analysis. Demographic characteristics of the sample are shown in Table 1 .

Table 1

Demographic characteristics of the whole sample, broken down by the message type they were exposed to.

	All	Control	Medical Endorsement (M)	Medical Endorsement + Risk (M + R)	Medical Endorsement + Risk + Collective Action (M + R + F)
Gender (Female)	796 (50.1%)	196 (48.8%)	206 (51.9%)	201 (50.6%)	193 (49.1%)
Age
<30	244 (15.3%)	55 (13.7%)	72 (18.1%)	59 (14.9%)	58 (14.8%)
30–39	309 (19.4%)	69 (17.2%)	86 (21.7%)	79 (19.9%)	74 (18.8%)
40–49	296 (18.6%)	77 (19.2%)	69 (17.4%)	73 (18.4%)	77 (19.6%)
50–59	289 (18.2%)	75 (18.7%)	51 (12.9%)	90 (22.7%)	73 (18.6%)
60–69	305 (19.2%)	79 (19.7%)	80 (20.2%)	63 (15.9%)	83 (21.1%)
70+	147 (9.3%)	47 (11.7%)	39 (9.8%)	33 (8.3%)	28 (7.1%)
Region
Connacht/Ulster	317 (19.9%)	79 (19.7%)	78 (19.7%)	75 (18.9%)	85 (21.6%)
Leinster – Dublin	430 (27.0%)	91 (22.6%)	104 (26.2%)	118 (29.7%)	117 (29.8%)
Leinster – Outside of Dublin	418 (26.3%)	117 (29.1%)	113 (28.5%)	106 (26.7%)	82 (20.9%)
Munster	425 (26.7%)	115 (28.6%)	102 (25.7%)	98 (24.7%)	109 (27.7%)
Employed	863 (54.3%)	219 (54.5%)	206 (51.9%)	217 (54.7%)	220 (56.0%)
Locality (Urban)	989 (62.2%)	235 (58.5%)	258 (65.0%)	248 (62.5%)	248 (63.1%)
Nationality (Irish)	1376 (86.5%)	349 (86.8%)	345 (86.9%)	345 (86.9%)	336 (85.5%)
Education (Degree+)	671 (42.2%)	152 (37.8%)	181 (45.6%)	167 (42.1%)	171 (43.5%)
Reduced work due to restrictions	114 (7.2%)	24 (6.0%)	30 (7.6%)	28 (7.1%)	32 (8.1%)
Children	870 (54.7%)	249 (61.9%)	218 (54.9%)	203 (51.1%)	199 (50.6%)
Usually get the flu vaccine
Never	846 (53.2%)	216 (53.7%)	203 (51.1%)	210 (52.9%)	216 (55.0%)
Some years	223 (14%)	56 (13.9%)	55 (13.9%)	56 (14.1%)	56 (14.3%)
Yes – most years	521 (32.8%)	130 (32.3%)	139 (35.0%)	131 (33.0%)	121 (30.8%)

Note. There were no statistically significant differences between groups except for children. The control group had a higher proportion of participants with children than the other three groups.

When asked to identify the content of the poster they had seen as an attention check, 87% of respondents got it correct. This left a total sample size of 1,382. There were no differences in responses by whether participants saw the posters early or later in the study.

Our dependent variables were not normally distributed. We therefore undertook bivariate tests using non-parametric Mann-Whitney U tests and multivariate analysis using ordinal logistic regression, ensuring that models passed diagnostic tests of the underlying proportional odds assumption.

Reaction to posters

We first checked for differences in reactions to the two images across all message types (Fig. 3 ). The hospital image was perceived as more effective and trustworthy, but less optimistic than the GP image (ps < 0.001). There was no difference in how much people liked one image compared to the other (p =.234). Recall that this is a between-subjects design – participants rated only one poster.

Fig. 3

Effects of poster image type on reactions to the poster. Note. Error bars are standard errors.

We then checked for differences in the same reactions by the type of message on the poster (Fig. 4 ), using ordinal logistic regressions. Surprisingly, the control message was seen as more trustworthy than all three intervention messages (Poster M p <.001; Poster M + R p <.001; Poster M + R + F p =.01). Two of the three intervention messages were viewed as slightly more optimistic than the control message (Poster M p <.001; Poster M + R p = 0.35; Poster M + R + F p =.005). None of the intervention messages were perceived as more effective or were liked more than the control message (Efficacy: Poster M p =.813; Poster M + R p = 0.19; Poster M + R + F p =.360; Liking: Poster M p =.447; Poster M + R p = 0.08; Poster M + R + F p =.812).

Fig. 4

Effects of the message content on reactions to the poster. Note. Error bars are standard errors.

We then looked at the interaction between image and message type on these reactions (Fig. 5 ). We carried out ordinal logistic regressions with an interaction between image type and message. Where the proportional odds assumption did not hold, we used a generalised ordinal logistic regression instead [25].

Fig. 5

Interaction between image and message type on reactions to posters. Note. Error bars are standard errors.

The medical endorsement message was perceived as more effective and liked more when paired with the hospital image than the GP image (Table 2 ). The hospital image was perceived as more optimistic when paired with any intervention messages compared to the control. There was no interaction between message and image on ratings of trust.

Table 2

Ordinal logistic regression showing effect of image and message on perceived efficacy, trust, optimism and liking.

	Perceived Efficacy	Trust	Optimism	Like
	B (SE)	B (SE)	B (SE)	B (SE)
GP image (ref. Hospital image)	−0.35 (0.19)	−0.33 (0.20)	1.33 (0.19)***	0.19 (0.19)
Message (ref. Control)
M	0.33 (0.19)	−0.29 (0.20)	1.37 (0.19)***	0.19 (0.19)
M + R	0.24 (0.19)	−0.56 (0.20)**	0.72 (0.20)***	0.01 (0.19)
M + R + F	0.09 (0.19)	−0.36 (0.20)	0.94 (0.19)***	0.02 (0.19)
Image * Message Interaction
GP + M	−0.60 (0.27)*	−0.44 (0.27)	−1.23 (0.27)***	−0.61 (0.27)*
GP + M + R	−0.09 (0.27)	−0.11 (0.28)	−1.16 (0.27)***	−0.49 (0.27)
GP + M + R + F	0.07 (0.27)	0.02 (0.28)	−1.09 (0.27)***	−0.10 (0.27)
/cut1	−3.17 (0.18)	−3.41 (0.18)	−1.92 (0.16)	−2.52 (0.16)
/cut2	−2.54 (0.16)	−2.94 (0.17)	−1.18 (0.14)	−1.91 (0.15)
/cut3	−1.92 (0.15)	−2.41 (0.16)	−0.42 (0.14)	−1.38 (0.14)
/cut4	−1.04 (0.14)	−1.56 (0.15)	0.99 (0.14)	−0.20 (0.14)
/cut5	−0.15 (0.14)	−0.74 (0.15)	1.82 (0.15)	0.56 (0.14)
/cut6	0.82 (0.14)	0.23 (0.14)	2.79 (0.16)	1.56 (0.14)

*p <.05, **p <.01, ***p <.001.

Note. The assumption of proportional odds for the entire model was not met for the effect of image and message on perceived efficacy. We ran the model with an ordinal logistic regression, a generalised ordered logistic regression (Williams, 2016) and a linear regression. All models returned the same pattern of results. The generalised ordered logistic regression showed that the effect was mainly for higher ratings of perceived efficacy. For reasons of space, we have reported the ordinal logistic regression here.

In summary, the poster images and messages induced different reactions in participants, although differences were not large. One surprising finding was that the vaccine messages were all trusted less than the control message.

Effect of posters on vaccine intention

We then assessed whether either image or message type influenced participants’ intention to take the vaccine or the timeframe in which they would be willing to take it. Mean intention was 6.03 (SD = 1.66, range = 1–7). Most people (74%) said they would take the vaccine as soon as it was available, 9% said they would wait a few weeks, 6% said they would wait a few months, 5% said they would wait for a year or more and a further 5% said they would never take it. We checked for the effects of the posters on these variables without controls first and then added sociodemographic characteristics.

We found no effects of image type, message type or the interaction between image and message type on either intention to take the vaccine or on the timeframe in which participants were willing to take it (Table 3 ). Older people were more likely to say they would take the vaccine and that they would take it sooner rather than later. Men were more likely to say they would take it sooner rather than later when offered. People with children and those in Black, Asian and minority ethnicities (BAME) were less likely to say they would take it and more likely to delay taking it if they would. There were no effects of employment or education on intention or timeframe. There were no interaction effects between any sociodemographic characteristic and poster message suggesting that the posters were not effective for specific subgroups.

Table 3

Ordinal logistic regression showing effect of image and message on intention to take the vaccine.

	Intention to take the vaccine			Timeframe for taking vaccine
	Model 1	Model 2	Model 3	Model 1	Model 2	Model 3
	B (SE)	B (SE)	B (SE)	B (SE)	B (SE)	B (SE)
GP image (ref. Hospital image)	0.21 (0.22)	0.03 (0.11)	0.18 (0.22)	−0.18 (0.25)	0.03 (0.12)	−0.15 (0.25)
Message (ref. Control)
M	0.25 (0.21)	0.11 (0.16)	0.27 (0.22)	−0.13 (0.24)	−0.06 (0.18)	−0.18 (0.25)
M + R	0.11 (0.21)	0.08 (0.16)	0.14 (0.22)	−0.05 (0.24)	0.07 (0.18)	−0.06 (0.25)
M + R + F	0.21 (0.21)	0.13 (0.16)	0.19 (0.22)	−0.20 (0.25)	−0.08 (0.18)	−0.17 (0.25)
Image * Message Interaction
GP + M	−0.34 (0.30)		−0.32 (0.31)	0.23 (0.34)		0.23 (0.35)
GP + M + R	−0.14 (0.31)		−0.13 (0.31)	0.25 (0.34)		0.27 (0.35)
GP + M + R + F	−0.18 (0.31)		−0.13 (0.31)	0.24 (0.35)		0.19 (0.36)
Age (ref. < 40)
40–59		0.19 (0.14)	0.20 (0.14)		−0.38 (0.15)*	−0.38 (0.15)*
60+		1.10 (0.17)***	1.10 (0.17)***		−1.08 (0.20)***	−1.08 (0.20)***
Male (ref. Female)		0.19 (0.11)	0.18 (0.11)		−0.37 (0.13)**	−0.37 (0.13)**
Has a child		−0.30 (0.12)*	−0.29 (0.12)*		0.35 (0.14)*	0.35 (0.14)*
Employed		−0.14 (0.12)	−0.14 (0.12)		0.09 (0.13)	0.09 (0.13)
Degree +		0.17 (0.12)	0.17 (0.12)		−0.16 (0.13)	−0.16 (0.13)
BAME		−0.75 (0.24)**	−0.76 (0.24)**		0.86 (0.25)**	0.87 (0.25)**
/cut1	−2.83 (0.19)	−2.73 (0.21)	−2.66 (0.23)	0.97 (0.17)	0.71 (0.20)	0.62 (0.23)
/cut2	−2.35 (0.17)	−2.25 (0.20)	−2.18 (0.22)	1.53 (0.17)	1.29 (0.20)	1.21 (0.23)
/cut3	−2.04 (0.17)	−1.93 (0.19)	−1.86 (0.21)	2.04 (0.18)	1.82 (0.21)	1.73 (0.23)
/cut4	−1.57 (0.16)	−1.45 (0.19)	−1.38 (0.21)	2.77 (0.20)	2.56 (0.23)	2.48 (0.25)
/cut5	−1.04 (0.15)	−0.90 (0.18)	−0.83 (0.20)
/cut6	−0.37 (0.15)	−0.20 (0.18)	−0.13 (0.20)

*p <.05, **p <.01, ***p <.001.

BAME = Black, Asian and minority ethnicities.

Note. The assumption of proportional odds for the entire model was not met when sociodemographic controls were included for intention to take the vaccine, but no variable alone was at fault. We ran the model with an ordinal logistic regression, a generalised ordered logistic regression, and a linear regression (Williams, 2016). All models returned the same result and so we have reported the ordinal logistic regression here. Higher scores on the timeframe for taking the vaccine model mean the participant preferred to wait longer before taking the vaccine.

Suppressor effect of trust?

As the intervention posters reduced trust compared to the control poster, we carried out a supplementary exploratory analysis in which we included reactions (effectiveness, trust, optimism, liking) to the posters as covariates in the models on intention to take the vaccine and timeframe for getting it (Table 4 ). The effect of poster message on intention became statistically significant (Poster M p <.05, Poster M + R p <.01, Poster M + R + F p <.01) and the effect of GP image became statistically significant (GP p <.05). We checked specifications that included each reaction individually. Trust was the variable that, when introduced to the model, led the poster type coefficients to increase and become statistically significant. While noting that this analysis is exploratory and was not pre-registered, the possible implication is that our null result was due to people’s distrust of the poster acting as a suppressor [26]. This is supported by the correlation coefficients between the reactions to the posters and intention to be vaccinated. Trust had the strongest relationship with intention to be vaccinated compared to the other reactions (Table 5 ). The intervention posters may have increased intention to be vaccinated relative to the control had they not reduced trust. Similarly, the GP image was trusted less than the hospital image but, had it not been, it may have been more effective than the hospital image. This is contrary to the prediction we made in H1. It is not clear what led to the reduced trust for the GP image and so it is difficult to speculate on what may have caused this effect.

Table 4

Ordinal logistic regression showing effect of trust on the relationship between poster type and intention to take the vaccine.

	Intention to take the vaccine	Timeframe for taking vaccine
	Model 4	Model 4
	B (SE)	B (SE)
GP image (ref. Hospital image)	0.28 (0.13)*	−0.18 (0.14)
Message (ref. Control)
M	0.41 (0.17)*	−0.34 (0.20)
M + R	0.44 (0.17)**	−0.26 (0.19)
M + R + F	0.46 (0.17)**	−0.39 (0.20)ǂ
Age (ref. < 40)
40–59	−0.08 (0.15)	−0.08 (0.17)
60+	0.56 (0.19)**	−0.49 (0.21)*
Male (ref. Female)	0.20 (0.12)	−0.43 (0.14)**
Has a child	−0.23 (0.13)	0.28 (0.15)
Employed	−0.13 (0.13)	0.06 (0.14)
Degree +	0.22 (0.12)	−0.17 (0.14)
BAME	−0.93 (0.27)***	1.13 (0.28)***
Effectiveness	−0.12 (0.05)*	0.09 (0.06)
Trust	0.82 (0.06)***	−0.71 (0.06)***
Optimism	0.13 (0.05)*	−0.15 (0.06)*
Like	−0.07 (0.06)	0.07 (0.07)
/cut1	0.63 (0.29)	−2.90 (0.32)
/cut2	1.27 (0.29)	−2.18 (0.32)
/cut3	1.69 (0.29)	−1.53 (0.32)
/cut4	2.32 (0.29)	−0.58 (0.32)
/cut5	3.03 (0.30)
/cut6	3.95 (0.31)

ǂp =.05, *p <.05, **p <.01, ***p <.001.

Note. The assumption of proportional odds for the entire model was not met for intention to take the vaccine. We ran the model with an ordinal logistic regression, a generalised ordered logistic regression, and a linear regression (Williams, 2016). All models returned the same result and so we have reported the ordinal logistic regression here. Higher scores on the timeframe for taking the vaccine model mean the participant preferred to wait longer before taking the vaccine.

Table 5

Correlations between the reactions to the posters and intention to take the vaccine.

	1	2	3	4	5
1. Intention to take the vaccine	1
2. Effectiveness	0.30	1
3. Trust	0.48	0.67	1
4. Optimism	0.33	0.51	0.54	1
5. Like	0.34	0.70	0.70	0.68	1

Discussion

Eight posters testing messages with medical endorsement of the COVID-19 vaccine, relative risk of COVID-19 and protection for friends and family were not effective at reducing vaccine hesitancy in a large, nationally representative sample in Ireland. The posters induced different reactions on ratings of trust and optimism suggesting that participants engaged with them, but they did not have follow-through effects on intention to take the vaccine.

There are some reasons why we might have found no effect. All three intervention messages about the vaccine were viewed as less trustworthy than the control poster. Previous work has shown that trust in science predicts vaccine acceptance and we and others had hypothesised that endorsement from a medical professional may help to overcome concerns about the vaccine [5], [9], [10], [12], [15], [16], [17]. People who trusted the poster they saw were more likely to intend to take the vaccine. This pattern is consistent with trust acting as a suppressor and underpinning the null result. Another question is what might have driven this lower level of trust in the intervention messages. It is possible that some participants may have thought that the risk information we showed (15 in 1000 people die from COVID-19) was a smaller risk than they had previously assumed and so trusted the information less. However, we see reduced trust even for posters that give a medical endorsement of the COVID-19 vaccine without any information about risk. We used medical endorsement as the foundation for our intervention posters on the basis that this might help to foster trust. Our results suggest that the trust barrier for vaccination may be difficult to overcome using medical endorsement alone, given that this induced less trust than endorsement of other public health measures, such as hand washing and social distancing. Of course, a personal endorsement from a known trusted medical professional may have a stronger influence than a poster, but the aim of this study was to assess content that could be used in a public health information campaign.

This reduced trust for the posters about vaccination occurred in a country with, at the time, the 6th highest rating of trust for government and 10th highest rating of trust in the healthcare system in the European Union. In a comparative survey carried out in February/March 2021, Ireland gave a mean rating of 4.8 out of 10 to trust in government compared to an EU average of 3.9 and a mean rating of 6.5 for trust in the healthcare system compared to an EU average of 5.9 [27]. A recent analysis comparing 177 countries found that greater trust in government predicted greater vaccine uptake [28]. By March 2022, Ireland had achieved vaccine uptake of 94.6% in those aged 18 and older compared to a cumulative total of 83% in the rest of the EU [22], implying that many people who were initially hesitant ultimately decided to take the vaccine. The next section discusses two possible reasons for reduced trust in the vaccination posters and the null effect: lack of knowledge and perceived benefits.

The diagnostic study conducted simultaneously with this intervention study may shed more light on reasons for hesitancy that were not known at the time and were not included in our poster designs. The study deployed a range of tasks (ranking risks and benefits, choice tasks, multiple choice knowledge questions) to isolate individual perceptions and beliefs about COVID-19 vaccines, then tested which most strongly influenced the vaccine decision [24]. Across multiple types of response, vaccine hesitant or resistant individuals differed from those willing to take the vaccine not so much by seeing greater risks to vaccination, but by failing to perceive benefits. They were also less knowledgeable about all aspects of the vaccine and its development process, which influenced their perceptions of risks and benefits. It is possible that messages about the risks of not getting vaccinated are not as powerful as messages about the benefits of being vaccinated would have been. It is also possible that lack of knowledge about the vaccine and lack of perceived benefits may have resulted in reduced trust of posters about the vaccine. We based our poster designs on the strongest correlates of vaccine hesitancy identified by contemporaneous surveys and qualitative studies. These included the finding that hesitant individuals commonly listed the risks of vaccination as a reason for hesitancy, that they were less trusting of science and that they perceived the threat of COVID-19 to be less severe than vaccine accepting individuals [4], [6], [7], [8], [13], [29], [30]. Coupled with these were hypotheses that endorsement from trusted health professionals may be more influential than endorsements from scientists or the government [5], [9], [10], [12], [15], [16], [17]. The method in the diagnostic study that highlighted the lack of knowledge and lack of perceived benefits differed from regular surveys and qualitative report techniques that ask hesitant individuals to state why they are hesitant or to select from a list of reasons [e.g. [7], [13], [29]]. A problem with these more standard techniques is that people cannot generate or select reasons that they are unaware of, such as misunderstandings or lack of knowledge; respondents don’t know what they don’t know. Because our poster designs were based on such survey findings, they may well have failed to address the primary determinants of hesitancy.

The strength of this study was its large, nationally representative sample of the population. This allowed us to test four message types paired with two images. There are also limitations, of which we highlight two. First, posters may not be the most powerful means of reducing vaccine hesitancy; a different medium with the same messages might be effective. This is particularly relevant for the medical endorsement, which came not from a healthcare professional that the individual knew but from a stranger. Second, it is possible that we encountered a ceiling effect due to high levels of vaccine acceptance. Awareness of this possibility is why, in addition to the substantial sample, we opted for a 7-point scale to capture varying levels of acceptance and a second outcome variable that measured how soon people were willing to take the vaccine. The mean response on the scale was 6.03 with a standard deviation of 1.66 and responses covered all 7 points. This variation was sufficient to capture multiple statistically significant differences in vaccine hesitancy by socioeconomic group. For example, a post-hoc comparison of effect sizes and standard errors (Table 3) shows that the estimated coefficient on membership of the BAME group was over five times greater than the standard errors of the coefficients for the main treatment groups. The study was therefore adequately powered to detect meaningful effects.

The null finding in this study is informative for COVID-19 vaccination campaigns. We found reduced trust for messages about the vaccine compared to a message about other public health measures to prevent the spread of COVID-19. The finding arose in a country that has comparatively high trust in government and in the healthcare system. We do not know the reasons for this reduced trust, but lack of knowledge about the vaccine and a lack of awareness of the benefits may have played a role. Focussing on the relative risk of COVID-19 compared to vaccination against COVID-19 or focussing on pure medical endorsement may be insufficient to encourage hesitant individuals without first addressing gaps in knowledge and awareness. Focussing instead on the benefits of the vaccine and bridging knowledge gaps about the development process and efficacy of the vaccine may be more effective routes. Previous work on vaccine hesitancy in diseases other than COVID-19 has found that highlighting benefits can reduce perceived risks and influence intention to be vaccinated [31]. Recent work on COVID-19 vaccination found that messages about COVID-19 vaccines that were gain-framed, loss-framed or altruistic were all effective when compared to a no-message control at influencing intention to be vaccinated [32]. The loss-framed message was the most powerful but, importantly, all three mentioned a specific benefit of vaccination (production of antibodies so that the individual/vulnerable people would be protected against COVID-19). The messages in our ineffective posters did not mention a specific benefit.

Although we found no effect of our intervention, a null finding to inform what is not effective is arguably just as useful as a positive finding. The pandemic is global, with national vaccine campaigns proceeding at different paces and with minorities of less willing individuals of different sizes in different countries. This research was commissioned by the Department of Health in the Government of Ireland to pre-test public health messages that would be used in the vaccination campaign. Using experimental methods to pre-test communication materials can be a useful way of finding out what is most likely to be effective and, just as importantly for policy, what is not.

Ethical statement

The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.