Using moral foundations in government communication to reduce vaccine hesitancy.

Having a vaccine available does not necessarily imply that it will be used. Indeed, uptake rates for existing vaccines against infectious diseases have been fluctuating in recent years. Literature suggests that vaccine hesitancy may be grounded in deeply rooted intuitions or values, which can be modelled using Moral Foundations Theory (MFT). We examine the respective prominence of the MFT dimensions in government communication regarding childhood vaccinations and explore its effect on parents' vaccine hesitancy. We measure the MFT dimension loading of the vaccination information brochures from the Dutch National Institute for Public Health and the Environment (RIVM) between 2011-2019 and connect this information with the electronic national immunisation register to investigate if the use of moral foundations in government communication has a measurable effect on vaccination uptake. We find the largest positive effect for the dimensions Authority/Subversion and Liberty/Oppression and suggestive evidence in favour of a small positive effect for Purity/Degradation. Conversely, Loyalty/Betrayal actually has a negative effect on vaccination rates. For the dimension Harm/Care, we find no significant effect. While Purity/Degradation and Harm/Care appear to be the two most frequently used moral foundations by RIVM, these dimensions have in fact no or only a minor effect on parents' vaccine hesitancy. Reducing the use of these moral foundations may be the first step towards optimising government communication in this context. Instead, formulations activating the moral foundations Authority/Subversion and Liberty/Oppression appear to have positive effects on vaccination uptake.

Introduction

One of the main societal motivations for childhood vaccinations is to extinguish infectious diseases, such as Measles, Tetanus and Polio, by achieving herd immunisation [1]. Having a vaccination available, however, does not automatically mean that it will be used [2-5]. Advancing the uptake of the vaccine among the public is one of the main challenges for policymakers [6]. In the context of the current COVID-19 pandemic with its widespread vaccination programmes, the issue of vaccine hesitancy has gained new relevance [7]. Understanding factors influencing vaccination acceptance rates can be of crucial importance for the success of this and other vaccination programmes [8, 9]. Thomson and Watson [10] suggest vaccination adoption to be an additively separable function of access and acceptance and coin the axiom “vaccine adoption = access+acceptance” [10]. By using data from the Netherlands, a country in which childhood vaccinations are free of charge and readily accessible for everyone, we hold the aspect of access constant to investigate acceptance-driven parental choice in vaccine adoption rates. Prior studies have mainly focused on the aspect of parental beliefs about childhood vaccinations to explain vaccination uptake intentions (i.e. [11-15]). These are often collected as self-stated vaccination intention through surveys, which may not necessarily translate into actual vaccination uptake (see, e.g. [16]). In this study, by contrast, we employ vaccine adoption rates from governmental care records, which has been characterised as the “gold standard” [17] to measure actual behaviour as opposed to self-stated vaccine hesitancy. Prior evidence-based studies on vaccine hesitancy have focused predominantly on raising knowledge and awareness [18]. Most interventions demonstrate only short-term effects or even a decrease in intention to vaccinate, though [19, 20]. This resistance towards educational interventions suggests that attitudes towards vaccines may instead be rooted in deeper intuitions and emotions.

Moral Foundations Theory was developed to identify “intuitive ethics” that guide people’s behaviour [21]. It consists of six dimensions: Care/Harm, Fairness/Cheating, Loyalty/Betrayal, Authority/Subversion, Purity/Degradation and Liberty/Oppression [22]. We discuss the concept of Moral Foundations Theory in more detail in S1 Appendix. MFT is a relatively new theory, developed at first to explain the dimensions of moral judgement, which has been shown to be an intuitive process [22]. Since then, moral foundations have been applied to explain general political ideologies [23], or specific political stances, for example on death penalty, abortion, gun control and immigration [24], but also on attitudes towards climate change [25], and leadership moralisation [26]. Clifford and Jerit [27] show how the use of moral rhetoric by political elites in the debate about stem cell research affects public attitudes. In this study we investigate if government communication has a measurable effect on parents’ choice to vaccinate their child and most importantly, which moral dimensions appear to trigger these effects.

Amin et al. [28] have been the first to apply MFT in the context of vaccination uptake. They conduct a survey using MFT to explain how deeply rooted intuitions, or “values”, influence vaccine hesitancy. They find that “medium vaccine hesitant parents” are twice as likely to emphasise the moral foundation Purity/Degradation and “very vaccine hesitant parents” are twice as likely to emphasise Purity/Degradation and Liberty/Oppression. The authors suggest that government communication about childhood vaccinations might be more effective in reaching these vaccine hesitant parents if it focuses on Purity/Degradation and Liberty/Oppression. To date, there exists no evidence on whether triggering specific moral foundations translates into measurable change in behaviour, though. Our study contributes at filling this knowledge gap by linking the MFT dimensions used in vaccine related government communication towards parents with actual governmental care records on vaccination uptake.

We use the Moral Foundations Dictionary (MFD) [23, 29, 30], a validated and established dictionary providing information on the words for each foundation, to analyse all vaccination information brochures directed at parents by the Dutch National Institute for Public Health and the Environment (RIVM) between 2011 and 2019. In the Netherlands, care providers hand out these brochures during one of various mandatory consultation visits to all parents with children in the appropriate age group for a given vaccination. We find that RIVM has mainly used the moral foundations Purity/Degradation and Harm/Care in their communication towards parents, while the dimensions Authority/Subversion and Fairness/Cheating have found the least use. In fact, Fairness/Cheating has not been used at all in RIVM’s brochures. We then connect this information on MFT use in the vaccination brochures with vaccination rates from the electronic national immunisation register (Præventis). We find robust evidence for a favourable effect in the moral foundations Authority/Subversion and Liberty/Oppression for reducing parents’ vaccine hesitancy. By contrast, the use of the moral foundation Loyalty/Betrayal in the brochures appears to translate into a negative effect on vaccination rates.

This article is structured as follows: First we introduce our methods. We briefly outline the Dutch Immunisation Programme (NIP), including details on the use of vaccination brochures, before describing our data collection method. Then in Section Results we discuss our results. More concretely, we analyse the use of MFT dimensions in government issued vaccination brochures and then use this information to examine the relationship between the MFT dimensions and vaccination uptake rates. Finally, we will discuss the implications of our results and the associated limitations of our research design in Section Discussion and Conclusion.

Methods

We study childhood vaccination adoption in the Netherlands as a function of moral rhetoric in government communication. First, in Subsection The Dutch National Immunisation Programme, we give a very brief overview of the Dutch National Immunisation Programme (NIP). The first step in our analysis is to code all RIVM brochures on childhood vaccinations targeted at parents between 2011 and 2019. We describe this step in Subsection RIVM Childhood Vaccination Brochures and Moral Foundations. Finally, we use this information—and other control factors—as independent variables to explain the vaccination uptake of various vaccinations within the Dutch NIP during the period 2011–2019. We provide a brief discussion of the national immunisation register and the underlying data in Subsection Immunisation Coverage and Vaccine Hesitancy.

The Dutch National Immunisation Programme

The Dutch National Immunisation Programme (NIP) started in 1957 by offering vaccinations for Diphtheria, Tetanus and Polio [31-35]; more vaccinations have been added since. We include the vaccination schedule of the Dutch NIP applicable for the period 2011–2019 in Fig 1, including an indication of the age at which a child should receive certain vaccinations. The Figure also includes a legend with abbreviations which we use in this article.

Fig 1

Vaccination schedule 2011–2019.

Source: Rijksinstituut voor Volksgezondheid en Milieu [36], with permission.

Vaccinations have been a ‘hot topic’ in Dutch society in recent years [35]. There have been debates about vaccine safety, the non-mandatory status of vaccinations, the admission of non-vaccinated children to kindergarten and a need for improvement of government communication to increase the immunisation coverage [37]. While the immunisation coverage in the Netherlands has been high, historically, RIVM has documented a 2–3% decrease since 2014 [35]. In the Netherlands, immunisation coverage has reached a level below 95%, a percentage that the WHO uses as a threshold for the successful achievement of herd immunisation [38].

RIVM childhood vaccination brochures and moral foundations

To measure the use of moral foundations in government communication, we analyse brochures on childhood vaccinations which are provided by the National Institute for Public Health and the Environment (RIVM) to all parents. These brochures constitute the main communication channel for RIVM to provide parents with information about the Dutch NIP. Children ought to receive one or two vaccinations at a time during various points in their youth (see Fig 1). All vaccinations mentioned in the Dutch NIP are on a voluntary basis and free of charge [35]. Hence, it is the parents who freely decide whether they want to vaccinate their children. However, the government, represented by RIVM, does promote vaccinating children by providing parents with information brochures about the upcoming vaccination and the Dutch NIP. In chronological order, this process of information provision and parents’ choice proceeds as follows:

RIVM writes brochures containing information about childhood vaccinations mentioned in the Dutch NIP.

A child approaches a certain age at which, according to the Dutch NIP, he or she should be vaccinated (i.e.: at birth, before 3 months of age, before 4 years of age…).

Parents go to see the midwife (before giving birth), the consultation clinic (small children) or the Municipal Public Health Services, GGD (9 years old and adolescent girls). These visits are mandatory in the Netherlands.

During these visits, parents receive the brochures about the vaccinations mentioned in the Dutch NIP that correspond to the child’s age. A parent receives a total of four to five brochures (depending on the child’s sex) over the time span of parenting a child.

Parents go home, read the brochure and choose whether or not they want to vaccinate their child.

Parents receive an invitation letter/call to vaccinate their child at a location near them.

Parents do or do not vaccinate their child.

RIVM uses different brochures for parents of children of different age groups, and these brochures refer to the specific vaccinations that are linked to that age category. In other words, a given brochure refers to the vaccinations which the child, according to the vaccination schedule of the Dutch NIP, should receive shortly. To illustrate: the brochure for parents of toddlers refers to the vaccination DTaP-IPV and the brochure for schoolchildren refers to the DTP-IPV and MMR vaccinations. In Table 1, we present an overview of the vaccinations mentioned in the Dutch NIP and the corresponding brochures that contain information about these vaccinations (original Dutch names).

Table 1

Vaccinations and corresponding brochures.

Vaccination	Corresponding brochure (original Dutch title)
Newborns
DTaP-IPV newborns	Folder babies
Hib	Folder babies
HBVa	Folder babies
PCV	Folder babies
MenC	Folder peuters
MMR newborns	Folder peuters
Toddlers
DTaP-IPV toddlers	Folder kinderen 4 jaar
Schoolchildren
DT-IPV schoolchildren	Folder kinderen 9 jaar
MMR schoolchildren	Folder kinderen 9 jaar
Adolescent girls
HPV	Folder HPV

As RIVM does not publish a new brochure every year, we always use the latest edition of a brochure that RIVM has published, which corresponds to the actual use by RIVM. To illustrate: if RIVM published a new brochure in 2012 but no new brochure in 2013, it uses the 2012 brochure for 2013. We present an overview of these brochures and the associated vaccinations in Table 2.

Table 2

Overview of the brochures per year and the vaccinations mentioned in each brochure.

Brochures per year (original Dutch name)	Vaccinations mentioned in brochure
2011
Folder baby’s van 2, 3, 4 en 11 maanden 2011	DTaP-IPV newborns, Hib, HBVa, PCV
Folder kinderen 4 jaar 2011	DTaP-IPV toddlers
Folder peuters 14 maanden 2011	MenC, MMR newborns
2012
Folder baby’s 2, 3, 4 en 11 maanden 2012	DTaP-IPV newborns, Hib, HBVa, PCV
Folder kinderen 4 jaar 2012	DTaP-IPV toddlers
Folder kinderen 9 jaar 2012	DT-IPV, MMR schoolchildren
Folder peuters 14 maanden 2012	MenC, MMR newborns
2013
Folder baby’s van 6–9 weken, 3, 4 en 11 maanden 2013	DTaP-IPV newborns, Hib, HBVa, PCV
Folder extra BMR baby’s	MMR
van 6–14 maanden 2013
Folder HPV 2013	HPV
Folder kinderen 9 jaar 2013	DT-IPV, MMR schoolchildren
2014
Folder HPV 2014	HPV
2015
Folder baby’s 6–9 weken 2015	DTaP-IPV newborns, Hib, HBVa, PCV
Folder HPV 2015	HPV
Folder kinderen 4 jaar 2015	DTaP-IPV toddlers
Folder kinderen 9 jaar 2015	DT-IPV, MMR schoolchildren
Folder peuters 14 maanden 2015	MenC, MMR newborns
2016
Folder HPV 2016	HPV
2018
Folder HPV 2018	HPV
Vaccinaties voor kinderen van 4 jaar 2018	DTaP-IPV toddlers
Vaccinaties voor kinderen van 9 jaar 2018	DT-IPV, MMR schoolchildren
2019
Vaccinaties voor kinderen van 9 jaar 2019	DT-IPV, MMR schoolchildren

Note that for most vaccinations, our data covers the years 2011–2019, with four exceptions. 1) The data for the DT-IPV schoolchildren vaccination only covers the years 2012–2019, because RIVM first created a brochure for this vaccination in 2012. 2) The same also holds for MMR schoolchildren. This brochure has been first published and distributed together with the one for DT-IPV schoolchildren, hence also for this vaccination, the data covers the years 2012–2019. 3) The data for the HPV vaccination for adolescent girls only covers the years 2013–2019, because the relevant brochure was only introduced in 2013 [31]. 4) Finally, the data for the HBVa vaccination only covers 2015–2019 because of a new regulation and implementation issues abound. Before 2011, only risk groups were vaccinated against HBVa. From 2011 onward, however, the HBVa vaccination was implemented for the entire cohort [31]. The effects of this new regulation only manifest from 2014 onward, though, as 2011 is the cohort for the immunisation coverage of HBVa in 2014. We exclude 2014, the first year, because the immunisation coverage in 2014 was only 51.4% before ‘stabilising’ between 2015 and 2019 around 90+% (see S1 Table). It is highly unlikely that the increase in immunisation coverage from 2014 to 2015 is caused by the use of moral foundations in brochures. It is more likely that the low immunisation coverage in 2014 has to do with start-up issues.

We then use the Moral Foundations Dictionary (MFD) [23, 29], which is a list of words and word stems developed for word analysis purposes, to measure the use of moral foundations in these RIVM brochures. For each moral foundation, the MFD includes a list of words and word stems relating to that specific moral foundation (We will include an overview of the MFD in an online repository). As the original MFD does not contain the sixth moral foundation ‘Liberty/Oppression’, we use the list of indicating words for Liberty/Oppression created by Teernstra et al. [30], who analyse tweets on the five original moral foundations and the sixth foundation Liberty/Oppression. Their list is the most established tool in the field for analysing texts on the moral foundation Liberty/Oppression (We include this list in the online S1 Appendix). The MFD and the list of indicating words for Liberty/Oppression are originally written in English. As the brochures from RIVM are written in Dutch, we translate the English Moral Foundations Dictionary and the list of indicating words for the foundation Liberty/Oppression to Dutch. Similar to the method outlined by Harzing [39], both authors who are fluent/native Dutch speakers, independently translate the original English MFD words and word stems into Dutch and resolve any differences in the separately generated translations by discussion. We include this ‘Dutch MFD’ in Supporting Information S1 File and example sentences for each MFT dimension in Table A3 in S1 Appendix. While we remain within the same language family branch (Germanic) for this translation from English to Dutch, residual differences in meaning could potentially account for part of our obtained results. This is a potential limitation which our study shares with a large class of research in a cross cultural context that relies on translating instruments [40, 41].

To measure the use of moral foundations, we scan the brochures from RIVM for words included in the Dutch MFD using the software Linguistic Inquiry Word Count Program (LIWC), which was also used by Clifford and Jerit [27] and Teernstra et al. [30]. The software analyses texts for words indicated by the user of the programme and produces a measure of the extent to which each of the moral foundations is used in the brochure. Specifically, LIWC produces variables expressed in word counts. These variables refer to percentages of words relating to a specific moral foundation in a document, in this case a brochure. For example, if for a hypothetical Brochure A, LIWC indicates a value of 1.58 for Liberty/Oppression, this means that 1.58% of the words in Brochure A refer to the moral foundation Liberty/Oppression. Fig 2 provides a visual overview of the LIWC word counts per MFT dimension (An overview of the output of LIWC including the use of moral foundations in the different brochures is included in the S1 Appendix).

Fig 2

Overview of the dimension loading of the RIVM brochures per year, jittered with 5% noise.

Immunisation coverage and vaccine hesitancy

We use the immunisation coverage (the proportion of the cohort vaccinated in percentages) of children to measure parents’ vaccination acceptance. Immunisation coverage is inversely related to vaccine hesitancy, such that a higher immunisation coverage among children indicates lower vaccine hesitancy among parents. As opposed to self-stated vaccine hesitancy, the immunisation coverage represents an actual change in behaviour towards vaccinations, revealing parents’ true vaccination acceptance. This change in behaviour is the essential goal of government communication: reducing vaccine hesitancy such that it actually leads to a higher immunisation coverage for childhood vaccinations.

We employ publicly available data on the immunisation coverage for the different vaccinations of interest provided by Statistics Netherlands (CBS) [42]. An overview of the national immunisation coverage per vaccination is included in S1 Table. The data provides information about the immunisation coverage of all vaccinations mentioned in the Dutch NIP over the years. The data originates from ‘Præventis’, which is an electronic national immunisation register, developed in 2005 [43]. Præventis is linked to the Dutch population register. It records and validates administered vaccinations at the individual level. Because of its link to the Dutch population register, it is safe to characterise the data to be very close or even equal to the actual population. We use this data at the national aggregate level, as well as at the level of the 25 municipal public health regions (GGD-regio’s, see [44]) and the 355 municipalities [45] in the Netherlands.

Results

We first present results on the use of moral foundations dimensions in Subsection The Use of Moral Foundations. We analyse if certain moral dimensions are used more frequently than others in the brochures of RIVM. Then, in Subsection The Effect of the Use of Moral Foundations on Vaccine Hesitancy, we use the dimension loading of the RIVM brochures to investigate if the use of moral foundations has a measurable effect on vaccination uptake.

The use of moral foundations

We use the MFT dimension loading produced by the LIWC-coded word count to understand which moral foundations RIVM uses in its communication towards parents. Fig 2 provides a graphical overview of the dimension loading of each brochure in a jittered scatter plot illustrating the LIWC word count (i.e. the percentage of words per MFT dimension) as a function of the brochure’s year of use. Each colour represents a different MFT dimension, so each published brochure is represented in the graph by six dots of different colour, representing one word count indication per dimension per brochure. Consider, for example, 2016. For this year we only have one new brochure, ‘Folder HPV 2016’. Accordingly, Fig 2 depicts six different coloured data points for 2016. This initial eyeballing of the data gives a first impression of the heterogeneity of MFT dimensions used in RIVM’s brochures. It appears that Harm/Care and Purity/Degradation are used more frequently than other dimensions. Equally, Fairness/Cheating appears to consistently score very low for the word count. In fact, this dimension has never been used in the brochures. In the following we quantify this heterogeneity. As the data is not normally distributed we employ the rank-based non-parametric Kruskal-Wallis test [46] to identify structural patterns in the use of MFT.

This test can indicate whether there is a significant difference in the use of moral foundation dimensions in RIVM’s brochures. As it does not indicate the directionality of an eventual difference between the categories, we run a post-hoc Dunn’s test (pairwise comparisons) with Benjamini-Hochberg correction for multiple hypotheses testing to see which moral foundations differ significantly from each other [47]. This test compares each category (a category is a moral foundation dimension) to all other categories. The Dunn’s test produces Z-test statistics for each pairwise comparison and the p-value [47].

The Kruskal-Wallis test indicates a significant difference in the use of moral foundations in RIVM’s brochures across the six categories of moral foundations with χ2(5) = 112.583, p = 0.0001 for α = 0.05. Table 3 provides a list of the rank sum for each of the MFT dimensions, ordered by most to least prominent.

Table 3

Output for the Kruskal-Wallis test.

Moral Foundations	N	Rank Sum
Purity/Degradation	22	2428.5
Harm/Care	22	2403.5
Liberty/Oppression	22	1681.5
Loyalty/Betrayal	22	1061.0
Authority/Subversion	22	840.5
Fairness/Cheating	22	363.0

Note that the MFT dimension Fairness/Cheating has in fact never been used in any of the brochures. The value for each of the observations was 0. This means that RIVM has never used words or sentences that relate to fairness, justice and trustworthiness. An example of such a sentence would be: “This brochure contains fair and unprejudiced information about vaccinations.” Kennedy et al. [48] investigate the relationship between language usage and the five MFT dimensions by analysing Facebook status updates. Also in their study, Fairness/Cheating was the most difficult dimension to trace in language.

We use a post-hoc Dunn’s test with Benjamini-Hochberg correction to describe which of the pairwise comparisons between the MFT dimension loadings can be described as significantly different from each other [47]. The results of the Dunn’s test are displayed in Table 4. Most results were significant (for α = 0.05). The results which were not significant are underlined.

Table 4

Dunn’s pairwise comparison of use of moral foundations by moral foundations (Benjamini-Hochberg).

Underlined results are not significant at 5% confidence level.

Column mean—row mean z test statistic (p-value)	Harm/Care	Fairness/Cheating	Loyalty/Betrayal	Authority/Subversion	Purity/Degradation
Fairness/Cheating	8.102478
	0.0000
Loyalty/Betrayal	5.330839	-2.771639
	0.0000	0.0038
Authority/Subversion	6.206407	-1.896071	0.875568
	0.0000	0.0334	0.2042
Purity/Degradation	-0.099271	-8.201749	-5.430110	-6.305678
	0.4605	0.0000	0.0000	0.0000
Liberty/Oppression	2.866939	-5.235539	-2.463900	-3.339468	2.966210
	0.0031	0.0000	0.0086	0.0008	0.0025

The results of the post-hoc Dunn’s test show that the use of moral foundations in RIVM’s brochures is significantly different for almost all moral foundations dimensions at a p-value of p < 0.05. The only two individual categories that do not differ significantly are:

Harm/Care compared to Purity/Degradation

Loyalty/Betrayal compared to Authority/Subversion

The effect of the use of moral foundations on vaccine hesitancy

Using OLS with time-demeaning fixed effects, we investigate the relationship between parents’ vaccine hesitancy, expressed by the immunisation coverage (data from Præventis and [42]), and the use of moral foundations in government communication (the output of LIWC’s analysis of the brochures from RIVM). For this, we aim at explaining the immunisation coverage of the vaccinations mentioned in the Dutch National Immunisation Programme (NIP) (dependent variable / Y) by the use of moral foundations in government communication (independent variables / X’s) and controls.

As parents receive a given brochure only once (when their child is of a specific age), the use of moral foundations in the brochures that were used in the past does not affect the vaccine hesitancy of parents that, according to the Dutch NIP, should vaccinate their children the next year. In other words, the treated population is made up of different individuals each year, maintaining high temporal independence for our analysis.

Our data set contains the immunisation coverage (Y) for each vaccination i in region j and each year t, as well as the output from LIWC listing the use of moral foundations in the brochures from RIVM (X’s). As we have no observations for Fairness/Cheating, this dimension will not be part of the ensuing analysis. The data set can be accessed through the S1 File and example sentences for each MFT dimension in Table A3 in S1 Appendix and it is publicly available via [42]. Vaccination i can be any of the type as in Table 1; region j can either be referring to data on the national level (in this case we have j = 1 only), regional (with j ∈ {1, …, 25}), or municipal level (j ∈ {1, …, 355}). As discussed above, RIVM does not create a new brochure for each vaccination every year. For the years in which RIVM did not create a new brochure, we use the last brochure they published. This results in a complete data set, in which for each vaccination there is data on the use of moral foundations for every year. Next to vaccination rates for each vaccination type on a national aggregate, Præventis also provides this information for each of the 25 municipal public health regions (GGD-regio’s, see [44]) and 355 municipalities [45] in the Netherlands. We complement an overall analysis using data aggregated at the national level, with more refined data at the regional and municipal level to demonstrate the sensitivity of our results. We control for population size (more accurately: The number of children in a given birth cohort, which should receive the specific vaccination in question, according to the Dutch NIP), which varies both by panel (i.e. region and vaccination type) and over time. Furthermore, we include time as control variable (2011 = 1…2019 = 9). The general model looks as follows: where

To assess the robustness of our results, we add four additional controls to the model above. 1) As proxy for the degree of difficulty of the language used, we include a count of words with six or more letters. 2) As theoretically irrelevant structural composition element, we include the share of punctuation. 3) To control for words referring to issues of general morality, we include a category from the MFD [29] which refers to generally morally loaded words that do not pertain to a specific MFT dimension. 4) We account for the different length of brochures by including the absolute word count for each brochure.

Table 5 presents the results of the fixed-effects regressions, commensurable to Model 1. Regressions (1) and (2) use annual vaccination rates for the vaccinations of the Dutch NIP (as outlined in Subsection The Dutch National Immunisation Programme), aggregated at the national level; Regressions (3) and (4) are at the level of the 25 municipal health regions; and Regressions (5) and (6) use data at the level of all Dutch municipalities. While the uneven regressions represent results for the main model (Model 1), the even regressions include additional controls, as discussed above to assess the robustness of our results. Each combination of vaccination i and region j constitutes a separate panel. Within the Netherlands there exists some degree of regional variation as to the general vaccination acceptance, and each type of vaccination also displays some heterogeneity in acceptance rate. By demeaning these fixed effects, we are able to better isolate the true betas without the region- or vaccination specific fixed effects.

Table 5

Results of OLS fixed-effects models regressing the vaccination rate for vaccinations of the Dutch NIP on the MFT dimension loading from the associated information brochures and controls.

We analyse this for data reported at the national, regional and municipal level.

	(1)	(2)	(3)	(4)	(5)	(6)
	National		Regional		Municipal
VARIABLES	Vaccination Rate
Purity/Degradation	0.013	0.781	-0.055	0.553	0.144	0.809***
	(0.95)	(1.23)	(0.47)	(0.36)	(0.15)	(0.13)
Harm/Care	0.687	-1.789**	0.425**	-2.048***	0.301***	-2.008***
	(0.65)	(0.77)	(0.19)	(0.22)	(0.08)	(0.11)
Liberty/Oppression	4.098**	3.890**	4.303***	3.657***	4.296***	3.641***
	(1.69)	(1.77)	(0.64)	(0.70)	(0.22)	(0.23)
Loyalty/Betrayal	-6.993	-14.513***	-6.142***	-14.345***	-5.848***	-13.836***
	(4.40)	(3.98)	(1.30)	(1.28)	(0.46)	(0.52)
Authority/Subversion	22.452***	42.691***	22.568***	44.422***	23.047***	43.730***
	(4.28)	(7.22)	(2.56)	(3.49)	(0.97)	(1.54)
Time	-0.460***	-0.770***	-0.564***	-0.824***	-0.561***	-0.782***
	(0.14)	(0.14)	(0.04)	(0.04)	(0.01)	(0.02)
Population	0.000	0.000	0.001*	0.000	0.002***	0.001***
	(0.00)	(0.00)	(0.00)	(0.00)	(0.00)	(0.00)
Constant	67.829***	67.175***	82.718***	75.669***	86.999***	76.333***
	(14.08)	(18.87)	(3.40)	(3.42)	(0.53)	(1.11)
Controls	No	Yes	No	Yes	No	Yes
Number of observations	82	82	2050	2050	28946	28946
Number of panels	10	10	250	250	3530	3530
Within model R-squared	0.587	0.735	0.488	0.611	0.266	0.330
Between model R-squared	0.498	0.958	0.169	0.851	0.402	0.590
Overall R-squared	0.259	0.845	0.056	0.644	0.171	0.413

* p < 0.10,

** p < 0.05,

*** p < 0.01

Clustered standard errors in parentheses.

For three of the six moral foundations we find a robust significant effect on the immunisation coverage. Two of these have a positive effect on the immunisation coverage. These are respectively Authority/Subversion and Liberty/Oppression. Loyalty/Betrayal, by contrast has a negative effect on the immunisation coverage. The moral foundation Purity/Degradation displays a small positive effect which is significant in Regression (6) only. For this dimension, no significant effect emerges in the absence of our additional controls and for data at the national or regional level. The coefficient for Harm/Care is fairly volatile between the regressions and varies between about -2 and 0.7 percentage points. As the moral foundation Fairness/Cheating was never used in the brochures, it is discarded from the analysis.

Our results indicate a small but robust negative time trend (β ≤ − 0.46, p < 0.001), confirming the observation about the Dutch NIP discussed in Subsection The Dutch National Immunisation Programme of a slowly decreasing immunisation coverage over recent years [35]. Our regression output suggests a decrease of overall immunisation rates at a magnitude of about 0.46–0.74 percentage points annually.

Main result 1. The use of moral foundations in government communication affects vaccine hesitancy.

Overall speaking, the use of moral foundations in the RIVM’s brochures does seem to affect the immunisation coverage (p < 0.0001). As displayed in Table 5, the models account for 26.6–73.5% of the variance within the independent variables. Amin et al. [28] find that vaccine hesitant individuals emphasise the moral foundations Purity/Degradation and Liberty/Oppression. Accordingly, one would expect that government communication about vaccinations ought to focus on these two moral foundations to reduce vaccine hesitancy. We test this theory in our study. We do acknowledge that the issue of vaccine hesitancy constitutes a multi-faceted phenomenon that is influenced by a myriad of different factors [3, 49–51]. Still, the underlying R2 value may be considered high enough to draw informed implications in our context and certainly represents a promising new perspective towards reducing vaccine hesitancy.

Main result 2. The use of the moral foundations Authority/Subversion and Liberty/ Oppression in government communication is associated with less vaccine hesitancy. The use of the moral foundation Purity/Degradation has a weak negative effect on vaccine hesitancy.

We find the strongest effect on the immunisation coverage for the moral foundation Authority/Subversion (β≥22.452, p < 0.001). This means that a one percentage point increase in the use of this dimension in RIVM’s brochures would in expectation lead to an increase in vaccination rate of more than 22 percentage points. For Liberty/Oppression we also find a positive effect on the immunisation coverage (β ≥ 3.641, p < 0.02). This means that the use of Authority/Subversion and Liberty/Oppression may decrease vaccine hesitancy. Regressions (1a), (3a) and (5a) in Table A4 in S1 Appendix present an alternative regression method for which we use the absolute word count for the MFT dimension loading, as opposed to the percentage dimension loading in Table 5. This analysis indicates an effect of about 1 percentage point per signal word for Authority/Subversion and just below 0.3 percentage points for Liberty/Oppression. This means we find a robust positive effect on the vaccination rate of about respectively 1 and 0.3 percentage points per signal word.

To put the size of our coefficients into context, consider for example the last two years from our dataset. Between these two years, the overall average dimension loading for Authority/Subversion across all brochures has slightly increased from 0.115 in 2018 to 0.129 in 2019. With a median national cohort size in the Netherlands of about 184,000 children and an approximated lower bound effect of 22 percentage points, this translates into an expected increase of about ((0.129 − 0.115) ⋅ 0.22 ⋅ 184,000 ≈) 567 vaccinated children between 2018 and 2019. Note that this represents only the approximated isolated effect from this dimension, which only corresponds to one of the many aspects that explain the overall change in vaccination coverage.

The moral foundation Purity/Degradation displays a significant positive effect on immunisation coverage (β = 0.809, p < 0.001) only in Regression (6). We interpret this as some evidence for a small positive effect, albeit at a lower level of robustness. In Table A4 in S1 Appendix, where this dimension has significantly positive estimates in Regressions (3a) and (5a), the estimated effect translates into an increase of about 0.1 percentage points per signal word.

Main result 3. The use of the moral foundation Loyalty/Betrayal in government communication is associated with more vaccine hesitancy.

Our regressions discover a fairly robust effect for Loyalty/Betrayal (β ≤ − 5.848, p < 0.001, except for Regression (1), with p = 0.117). Most interestingly, this delivers evidence for a negative effect of the use of this moral foundation on the immunisation coverage. This means that the use of Loyalty/Betrayal in government communication may increase vaccine hesitancy by about 6–15 percentage points for every percentage point increase of this moral foundation in the brochures. When translated into a per-word measure, Table A4 in S1 Appendix indicates an effect of about −0.7 percentage points per signal word.

Main result 4. We find no robust significant effect for the moral foundation Harm/Care in government communication on vaccine hesitancy.

Our analysis finds no robust significant effect for the use of the moral foundation Harm/Care on the immunisation coverage (−2.048 ≤ β ≤ 0.687, p > 0.0001). This suggests no reliably discernable effect of this moral foundation in the brochures. We cannot draw a reliable conclusion on its effect on the immunisation coverage and vaccine hesitancy.

As the dimension Fairness/Cheating has never been used in RIVM’s brochures, this factor is not included in our analysis. Hence, we cannot assess this dimension’s effect on vaccine hesitancy, unfortunately.

Table 6 summarises our main results for each MFT dimension. We apply the same order as in Table 3, i.e. from most to least frequently used dimension. Most interestingly, the two dimensions that have been used most prominently in vaccination-related communication in the Netherlands appear to have either no effect on vaccination levels (Harm/Care) or only a minor, non-robust positive effect (Purity/Degradation).

Table 6

Summary of the findings.

MFT Dimension	Effect of the Moral Foundation on vaccine hesitancy
Harm/Care	No robust effect on vaccine hesitancy
Purity/Degradation	Weak evidence for a small positive effect
Liberty/Oppression	Decreases vaccine hesitancy
Loyalty/Betrayal	Increases vaccine hesitancy
Authority/Subversion	Decreases vaccine hesitancy
Fairness/Cheating	No data

Discussion and conclusion

In this study, we investigate whether the use of moral foundations in government communication translates into a measurable effect on the parental choice to vaccinate their child. For this we use the Moral Foundations Dictionary [23, 29] to analyse the moral foundation loading of the vaccination information brochures from the Dutch National Institute for Public Health and the Environment (RIVM) between 2011–2019 and connect the resulting moral foundation loading with the electronic national immunisation register.

We find robust evidence for a positive relationship between the use of the moral foundations Authority/Subversion and Liberty/Oppression in government communication and vaccination uptake. Additionally, our analysis finds plausible evidence for a weak positive effect on vaccination uptake by the use of Purity/Degradation. Prior research has identified the MFT dimensions Purity/Degradation and Liberty/Oppression as most important for both medium and highly vaccine hesitant parents [28]. This raises the question of why we find such a strong effect on vaccination uptake by the dimension Authority/Subversion, if vaccine hesitant people do not specifically rely on this moral foundation? Although our study cannot provide a definitive answer and more research is needed to investigate this matter, a possible explanation may be attributed to the fact that Authority/Subversion has some resemblance with Liberty/Oppression. The moral foundation Authority/Subversion concerns individuals’ handling of dominance and forcing beneficial relationships in hierarchies and navigating/altering behaviour in hierarchies [22, 52]. This moral foundation is currently triggered by leaders and modern institutions such as legal courts and law enforcement. The moral foundation Liberty/Oppression concerns individuals’ autonomy and control over their own matters to keep tyrants, bullies and alpha males from becoming too powerful [22]. Future research may broaden our understanding of why these moral foundations in particular reduce vaccine hesitancy—and why the others do not. A somewhat speculative interpretation could be the following: imagine someone who is concerned about oppression by the government or big pharmaceutical companies and believes that these parties force vaccinations upon the population for their own interest. It is easy to imagine that this individual is more easily influenced by a sentence that appeals to the moral foundation Liberty/Oppression, like “Vaccinating your children is your own choice. The government cannot force you to do so” than by a sentence that emphasises that vaccinations do not cause much physical harm (which would appeal to the Harm/Care foundation).

While Haidt [22, Chapter 8] describes the dimension Liberty/Oppression as operating in tension with Authority/Subversion, there are clear parallels between the two. Both have a strong focus on the way in which an individual relates to institutions or persons who have a position of power/dominance towards the individual. Both were, and still are, triggered by a sort of (attempted) dominance (see, i.e. [53, 54] for a discussion on liberalism and the other moral intuitions). The relationship of these two moral foundations may explain why specifically these two dimensions decrease vaccine hesitancy. Certainly, more research is needed to confirm or repudiate this possible explanation.

The very same “why-question” extends to the effect of Loyalty/Betrayal. We find a robust negative effect on vaccine uptake from usage of Loyalty/Betrayal in government communication. Interestingly, this is not the first time that an attempt/intervention mechanism to decrease vaccine hesitancy is found to be counterproductive. Nyhan et al. [55], for example, conduct a survey experiment in which parents receive information about the MMR vaccine. None of the applied interventions lead to an increase in vaccination intention, some even lead to a reduction and an increase in misperceptions. For future research, it may be interesting to investigate why appealing to the moral foundation Loyalty/Betrayal has an adverse effect on parents’ vaccine hesitancy.

Translating the MFD to another language entails the possibility that remaining subtle differences in meaning may not be captured in its entirety by the translation (for a discussion of this issue for cross-cultural research, see [40, 41]). Next to this, we see three limitations of our study design. The most important limitation being the fact that we cannot control for whether parents read the brochures they were given. In fact, they might throw away the brochures before reading them. They might have also read specific parts only, which obfuscates the moral foundations that they have in fact be exposed to. However, this assumption is hard to check in any research design other than an experimental one. Future research may randomise (parts of) the brochures parents receive.

Secondly, other than official government communication, there are websites, blogs and in the Netherlands even a foundation that disseminate alternative information about childhood vaccinations (Stichting Vaccinvrij, or ‘Foundation Vaccination Free,’ see stichtingvaccinvrij.nl). Parents—specifically parents that may already be endowed with a proclivity to not vaccinate—may mistrust government communication (see [8, 9]) and rely on these alternative sources of information instead. The effect of this alternative information, and the use of moral foundations in this information, has not been considered in this article. Again, the direct effect of information sources may be measured most accurately using a controlled experimental research design. However, studying the effect of alternative information sources takes a different approach than studying the effect of using moral foundations in the context of government communication, which is the focus of our study.

Thirdly, events might have occurred that have had an influence on parents’ vaccine hesitancy. Some examples of these events are outbreaks of diseases, an increase in reports on (alleged) adverse effects following immunisation and bad publicity in the media. The effects of these kind of events have not been taken into account in this study. We argue that these types of events should asymptotically cancel out in a wide data-set as the one we employ. Alternative approaches could be to extend the time span of the study, or considering multiple cases other than the Netherlands. In this sense, our study constitutes a part of the building block towards a better general understanding of the use of MFT dimensions in government communication.

Our results show that in the period between 2011 and 2019, the Dutch government has mainly used the moral foundations Harm/Care and Purity/Degradation in their communication towards parents, with Liberty/Oppression in third place. We find that the two most frequently used moral foundations have in fact no or only a minor effect on vaccination uptake. Hence, an obvious policy recommendation would be to reduce the use of these moral foundations, if government wants to optimise the effectiveness of its communication. The Dutch government, as well as other governments, should instead employ formulations triggering the moral foundations Authority/Subversion and Liberty/Oppression. Examples are readily available in the Moral Foundations Dictionary and can be translated to whichever language of choice (see for example Matsuo et al. [56]). Examples of words appealing to Authority/Subversion are obey, duty, respect, control, refuse and oppose. Examples of words appealing to Liberty/Oppression are freedom, choice, right, decision and force.

In the same vein, governments should try and avoid the use of Loyalty/Betrayal, as this moral foundation may increase vaccine hesitancy. This means that governments should avoid the use of words such as united, community, solidarity, enemy and betrayal. Concerning the use of the moral foundation Authority/Subversion by the Dutch government, in particular, there is room for improvement. While our results show that this dimension has the biggest effect on reducing vaccine hesitancy, it has been one of the least-used MFT dimensions. Lastly, the effect of the use of the moral foundation Fairness/Cheating could not be determined as this moral foundation was never used. It may be interesting for governments to experiment with the use of this moral foundation and to track the effects on parents’ vaccine hesitancy.

Our results may be of interest for the current communication on the COVID-19 vaccine. Of course, the usual caveat applies for extrapolating results into a different context. Still, many of the characteristics of our study also apply for the context of COVID-19 vaccination acceptance. What is equal is that we study communication from a public authority (a government organisation) directed at adult citizens on the topic of accepting a non-compulsory vaccination. The two most obvious differences between the COVID-19 context and the childhood vaccinations from the Dutch NIP are certainly the urgency from being in a dreadful pandemic situation and the decision to accept a vaccination for oneself versus accepting vaccinations for one’s children.

Equally, the relevance of our results extends beyond our country of investigation. As before, a degree of caution applies when extrapolating results beyond the context of investigation, especially towards environments that are (culturally and politically) very different. The Netherlands is a developed democratic country with a comparatively high vaccination rate. General health care metrics (i.e. health care expenditures in % of GDP, number of physicians per 10,000 individuals, etc.) of the Netherlands are comparable to other OECD countries [57]. In terms of the Dutch culture, the Hofstede 6D model [58, 59] characterises the Netherlands as low in power distance and masculinity, average to high on uncertainty avoidance, high on indulgence, very high on individualism, and have a “pragmatic orientation” in the dimension of long term orientation. Betsch et al. [60] conduct a cross-cultural experiment on vaccination acceptance in the context of herd immunity. They find that countries that score higher on Hofstede’s individualism dimension have a lower vaccination intention, but a higher responsiveness to communication on the topic of herd immunity. The influence of cultural dimensions on vaccination acceptance or vaccination communication appears to be an understudied branch of literature, which may constitute a promising avenue for future research.

Insights from this study could also help target government communication towards specific groups in the population. For example, Graham et al. [23] show that liberals and conservatives rely on different moral foundations. Tailoring the language of government communication towards its target audience may significantly improve the efficacy at which the intended message gets conveyed. Future research could involve tailoring brochures, or other types of government communication towards certain demographics and assess the effect of this measure.

List of abbreviations (See Fig 1 and S1 Table for an overview of abbreviations in the context of the Dutch National Immunisation Programme)

LIWC  Linguistic Inquiry Word Count Program

MFD   Moral Foundations Dictionary [23, 29]

MFT   Moral Foundations Theory

NIP  National Immunisation Programme

RIVM  Dutch National Institute for Public Health and the Environment

Discussion on Moral Foundations Theory and results from an analysis using absolute effects.

(ZIP)

Click here for additional data file.

Overview immunisation coverage.

Immunisation coverage in the Netherlands for the vaccinations of the National Immunisation Programme.

(ZIP)

Click here for additional data file.

Dutch Moral Foundations Dictionary.

Translation of the original Moral Foundations Dictionary.

(XLSX)

Click here for additional data file.

8 Jun 2021

PONE-D-21-07601

Using moral foundations in government communication to reduce vaccine hesitancy

PLOS ONE

Dear Dr. Heine,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Both reviewers provided important comments (see below). Please read the comments and incorporate those in the revision.

Please submit your revised manuscript by Jul 21 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Kazutoshi Sasahara

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: GENERAL COMMENTS:

*Introduction section*

What is “intentions”?

The authors mention the term “intentions” on page 2 as the concept that has been focused on previous studies, but it is not clearly stated what is it and why it is not suitable for this kind of research. I assume that the concept of “intensions” represents one’s self-stated vaccine hesitancy from the sentences in the same paragraph, but I am not sure.

Explanation of the MFT. The explanation of the MFT can be included in the manuscript itself (not in Appendix) because the flow of the manuscript should not be interrupted. I am not sure why the authors explain it in the separated section.

Relationship between concepts

One of my major concerns is that how morality and vaccine hesitancy are related is not clearly discussed in the Introduction section, which is connected to the aim of this study. The authors describe the MFT and the past research on the MFT and vaccine uptake, but, in the first place, how morality itself and vaccine uptake/hesitancy are related and why the authors should conduct research on vaccine hesitancy with the framework of morality are not argued. The authors may need to explain morality before the MFT to clarify the importance of their study.

Purpose of the study

The last paragraph on page 3 seems to describe the purpose of the present study, but what will be revealed by this study and the importance of this study in the academic field are not elaborated enough. Particularly this paragraph confused me because all the sentences are written in present tense, and also because what will be investigated and the results are presented in the same paragraph. Further, the authors need hypothesis; or, isn’t it a hypothesis-testing type study?

How the purpose is related to the paradigm

Related to the purpose of the study, the authors need to justify the appropriateness of the analyses to accomplish what they want to know. Why do the authors look at the language use? I would like to know more about the rationale.

*Methods section*

I like to see some example words from the MFD. Also, the Dutch version of the MFD needs to be described so that a reader understands it (e.g., How was the original version translated? Is the translation method a prevalent way to be used in past research involving non-English speaking people?).

*Results section*

The authors should present their results more carefully from regression analyses, such as “A significantly predicted B.” The expression used by the authors (e.g., “leads to”) do not efficiently convey what was observed. Also, it is not clear why the authors cited Amin et al. in the Main Result 1 section because Amin et al. found the reverse causal relationship compared to the current study. Please let me know if I am reading wrong. Regarding Main Result 4, I am not sure why the authors mention the Harm/Care foundation specifically. I like to see the rationale in the introduction section.

Results and interpretations

I see the results from the analyses and their interpretations are sometimes mixed up both in style- and concept-wise. For example, the descriptions in lines 93-96 should be in the results section. For another example, although you state that immunization coverage is inversely related to vaccine hesitancy (lines 187-188), it does not automatically mean that “the use of moral foundations in government communication affects vaccine hesitancy (lines 322-323).” What was found in the regression analyses was that the use of moral-related language significantly predicted the vaccine rate. I sometimes have hard time following the authors’ logic—what was FOUND as results and what was SUGGESTED from the results? Further, when the regression is performed, the authors may want to clearly state the independent variables and dependent variables.

Minor issue1: I am not sure if your style is fine in your field, but Methods and Results should be basically written in past tense.

Minor issue 2: I am not sure how the authors use the term “effect size” (e.g., line 348). I was wondering if they calculated Cohens’ d etc.

*Discussion section*

The authors state “future research may investigate the question of why these moral foundations in particular reduce vaccine hesitancy,” but isn’t it the purpose of the current study? It is related to the purpose of this study—the authors need to justify why this particular study was designed this way. It is not just that the variables were randomly chosen or the authors unexpectedly found the associations between them. Further, the authors discuss the possible practical applications of the results from this study, but I like to see how this study can contribute theoretically.

Minor issue: It is easier to understand if the discussion section starts with the summary of the study and results.

*Other*

Tone of the manuscript

The tone of the manuscript (especially results and discussion) seems a bit assertive. I’d caution against the causal language used in some sentences throughout the manuscript, such as “…as this moral foundation increases vaccine hesitancy…(line 473-). Again, it was just that the use of morality-related language predicted/was associated with vaccine rate (positively or negatively).

What the “use of moral foundations” means

The other of my major concerns is the language used in the manuscript. The meaning of the expression, “use of moral foundations,” sounds vague. It is the frequency of the morality-related word. I like to see how the language use and vaccine ratio are conceptually connected.

SPECIFIC COMMENTS:

Page 11: I like to see how N = 22 was calculated; I like to see the number of the broachers for each year.

Tables: The decimals should be aligned.

Page 20, Line 429: Please elaborate how it is counterproductive.

Page 20, Line 451: What is the “underlying study”?

Page 21, Line 469: Please provide some examples of the MFD in different languages.

Appendix (Page 233, Line 528): The description of the pluralist approach is confusing. Does pluralism demur the evolutionary explanation?

Reviewer #2: The paper studies the relationship between the moral dimensions and the parental hesitancy towards vaccination. It is a relevant topic given the current pandemic situation. Few comments are as follows:

Major comments

1. Please describe the process used to translate the English MFD dictionary to Dutch. Moral foundations may vary with culture and usage of the English-translated dictionary may not be able to capture the socio-cultural aspects of a particular region and may potentially bias the results. How do you ensure that English dictionary can be generalised well to the Dutch language? Please refer to [1] to learn more about the translation methodology.

2. The authors have used a moral foundation dictionary (MFD) and word count based methodology to capture the moral rhetoric used in the brochures. MFD created by a small group of experts with limited amounts of stem words and lemmas may not be valid for diverse contexts and populations [2]. The authors are advised to look at the applicability of the extended MFD [2] for their work. Also, many machine learning and word embedding based methods have been shown to better capture the moral rhetoric than word count based method. Authors should consider using such methods ( for example [3,4]) .

3. As mentioned by the authors the NIP started in 1957, please mention the reasons to select the data from 2011-2019 only.

4. In equation (1), moral dimensions i.e. HC, PD etc have been shown as varying with regions. Are there different brochures corresponding to the different regions? Kindly clarify. It seems there are only 22 brochures in total from 2011 to 2019. The immunisation coverage data is available at the high spatial granularity, while the same is not true for moral dimensions data. How do you ensure that the OLS model is not overfitting the data?

5. How do you account for the different lengths of brochures in Table 5? What was the motivation to use absolute word count?

6. Fairness dimension was not detected in the brochures and it is also the most difficult dimension to trace in language [5]. Authors may include this.

Minor comments

7. Kindly include some example statements from the vaccination brochures corresponding to different moral dimensions.

8 . The overall writing can be improved significantly. Few examples

(a)Page 4, line 73, 76, Subsection has been mentioned without referring to one.

(b) Page 16, line 330, “We test this theorem in our study”. Usage of the word “theorem” is incorrect.

(c) Please avoid vague terms such as “some” for results where statistical data can be provided e.g. Table 6 ”some evidence”

9. Table 10 has been referred to in the text after Table 2. Please put them in a sequential order.

10. n Figure 2, for better readability, please include 2012, 2014, etc on x-axis. Also, a descriptive caption can be provided which may help in better understanding of the figure.

[1] Matsuo, Akiko, et al. "Development and validation of the japanese moral foundations dictionary." PloS one 14.3 (2019): e0213343.

[2] Hopp, Frederic R., et al. "The extended Moral Foundations Dictionary (eMFD): Development and applications of a crowd-sourced approach to extracting moral intuitions from text." Behavior Research Methods 53.1 (2021): 232-246.

[3] Sagi, Eyal, and Morteza Dehghani. "Measuring moral rhetoric in text." Social science computer review 32.2 (2014): 132-144

[4] Araque, Oscar, Lorenzo Gatti, and Kyriaki Kalimeri. "MoralStrength: Exploiting a moral lexicon and embedding similarity for moral foundations prediction." Knowledge-based systems 191 (2020): 105184

[5] Kennedy, Brendan, et al. "Moral concerns are differentially observable in language." Cognition 212 (2021): 104696

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

10 Aug 2021

Reviewer 1

Thank you for your helpful feedback on our manuscript! Please find in the following a brief overview of how we addressed each of your comments.

Comment 1 *Introduction section*

What is “intentions”?

The authors mention the term “intentions” on page 2 as the concept that has been focused on previous studies, but it is not clearly stated what is it and why it is not suitable for this kind of research. I assume that the concept of “intensions” represents one’s self-stated vaccine hesitancy from the sentences in the same paragraph, but I am not sure.

Response 1

We add a sentence to clarify this on page 2. Indeed “intentions” refers to the self-stated intention to get vaccinated. As some individuals may end up not getting a vaccination despite their stated intention, this indication is somewhat noisy. Accordingly, we make the argument that governmental records, by contrast, promise to deliver a more accurate measure [1].

Comment 2 Explanation of the MFT.

The explanation of the MFT can be included in the manuscript itself (not in Appendix) because the flow of the manuscript should not be interrupted. I am not sure why the authors explain it in the separated section.

Response 2

We would suppose that adding the fairly elaborated explanation of MFT in the Introduction may unnecessarily bloat it to a point that it loses focus. We believe there are three alternatives: 1) Introducing MFT briefly in the Introduction and referring the reader to a detailed appendix (this is the alternative we have opted for). 2) Transferring the text from

the appendix into the introduction. This would make the introduction about seven pages long, which does not seem to be a desirable option. 3) Taking some more of the content from the Appendix into the Introduction. This options seems more viable than option 2, yet it may create a situation in which neither the appendix nor the somewhat more elaborate explanation in the Introduction remain as a solid self-sustained piece that adds value to the text. We respectfully believe that readers who are not at all familiar with MFT would profit from a thorough introduction of the concept, as presented in the appendix, while readers who do know MFT would not be served with a lengthy overview of the concept of MFT in the introduction.

Comment 3 Relationship between concepts

One of my major concerns is that how morality and vaccine hesitancy are related is not clearly discussed in the Introduction section, which is connected to the aim of this study. The authors describe the MFT and the past research on the MFT and vaccine uptake, but, in the first place, how morality itself and vaccine uptake/hesitancy are related and why the authors should conduct research on vaccine hesitancy with the framework of morality are not argued. The authors may need to explain morality before the MFT to clarify the importance of their study.

Response 3

Thank you for pointing out the lack of explicit link between morality/intuitions and vaccine hesitancy in our introduction. We now make this more explicit towards the end of page 2.

Comment 4 Purpose of the study

The last paragraph on page 3 seems to describe the purpose of the present study, but what will be revealed by this study and the importance of this study in the academic field are not elaborated enough. Particularly this paragraph confused me because all the sentences are written in present tense, and also because what will be investigated and the results are presented in the same paragraph. Further, the authors need hypothesis; or, isn’t it a hypothesis-testing type study?

Response 4

The paragraph starting “We use the...” provides a very brief outline of our method and our results. In the paragraph before (starting “Amin et. al...”) we establish our article’s contribution to the literature and which knowledge gap we address. In particular, our study is the first to connect the MFT dimensions used in vaccine related government communication towards parents with actual governmental care records on vaccination uptake. By doing so we explore whether triggering specific moral foundations translates into measurable change in behaviour and vaccination uptake. As for the exploratory character of our study we decided not to work with pre-formulated hypotheses other than the benchmarks derived from Amin et al.’s study.

Comment 5 How the purpose is related to the paradigm Related to the purpose of the study, the authors need to justify the appropriateness of the analyses to accomplish what they want to know. Why do the authors look at the language

use? I would like to know more about the rationale.

Response 5

We hope that the elaborations added to the Introduction in the context discussed above have clarified the link between the study’s purpose and the applied methods. In short, prior research suggests that vaccine hesitancy may be rooted in fundamental intuitions and emotions. MFT was developed to identify these intuitive ethics that guide people’s behaviour. To measure MFT, researchers usually look at some form of communication and then assess the language employed to identify which dimensions are triggered more prominently. Our study positions itself in this tradition, extending the context towards revealed behaviour and vaccination acceptance.

Comment 6 *Methods section*

I like to see some example words from the MFD. Also, the Dutch version of the MFD needs to be described so that a reader understands it (e.g., How was the original version translated? Is the translation method a prevalent way to be used in past research involving non-English speaking people?).

Response 6

Thank you for pointing out that the original manuscript has been too brief on this aspect. We now add the following explanation on page 9: “Similar to the method outlined by Harzing [2], both authors who are fluent/native Dutch speakers, independently translate the original English MFD words and word stems into Dutch and resolve any differences in the separately generated translations by discussion.” We include the Dutch MFD as supporting information, which will also become available to the reader in the Online Appendix. On page 29 we add a table with example sentences in their original and the translated form for each dimension. On page 9 we add a reference to the table in the main text. Further, we include some example words from the MFD on page 21 and an example sentence on page 16. The complete MFD can be accessed via https://moralfoundations.org/wp-content/ uploads/files/downloads/moral%20foundations%20dictionary.dic.

Comment 7 *Results section*

The authors should present their results more carefully from regression analyses, such as “A significantly predicted B.” The expression used by the authors (e.g., “leads to”) do not efficiently convey what was observed. Also, it is not clear why the authors cited Amin et al. in the Main Result 1 section because Amin et al. found the reverse causal relationship compared to the current study. Please let me know if I am reading wrong. Regarding Main Result 4, I am not sure why the authors mention the Harm/Care foundation specifically. I like to see the rationale in the introduction section.

Minor issue1: I am not sure if your style is fine in your field, but Methods and Results should be basically written in past tense.

Minor issue 2: I am not sure how the authors use the term “effect size” (e.g., line 348). I was wondering if they calculated Cohens’ d etc.

Response 7

Thank you for pointing this out. We rephrased the main results towards more careful language. The study by Amin et al. is most closely related to our study. Accordingly we find a comparison between their results and ours most insightful. In our view, the fact that some of our results show a different directionality than in Amin et al. deserves considerable attention in our article. We think the explicit reference to Amin et al.’s results is in the interest of transparency here. We discuss all MFT dimensions in the main results. The fact that we find no significant effect for a particular MFT dimension conveys an interesting insight too. In a well-powered study, ignoring non-significant results in the body of scientific evidence would feed the publication bias.

In our field it is common to employ present tense throughout the article, except when the context explicitly calls for pas tense, like for example “At one point, Keynesian (1932) logic was predominant, but theories beginning with Lucas overthrew it.” If the editor or journal has a specific preference, we are happy to revise this at an eventual typesetting phase. In the particular OLS fixed effects regression we employ, we can simply use the regression coefficient to make a statement about the size of the underlying effect. To avoid confusion with Cohen’s f2, which is often used to measure the overall effect size for a multiple regression, we apply reformulations where applicable and make sure that we are talking about the effect of A on B, not the overall effect size for the multiple regression.

Comment 8 *Discussion section*

The authors state “future research may investigate the question of why these moral foundations in particular reduce vaccine hesitancy,” but isn’t it the purpose of the current study? It is related to the purpose of this study—the authors need to justify why this particular study was designed this way. It is not just that the variables were randomly chosen or the authors unexpectedly found the associations between them. Further, the authors discuss the possible practical applications of the results from this study, but I like to see how this study can contribute theoretically.

Minor issue: It is easier to understand if the discussion section starts with the summary of the study and results.

Response 8

Thank you for pointing this out to us. The formulation we have chosen here may have been a bit too modest. It is not that we offer no explanation for the effects found through our analysis. In the discussion we offer an interpretation of which channels contribute to the observed effect and our interpretation of the underlying driving force. We change the formulation on page 19 to properly reflect that.

In the Introduction (specifically on page 3), we formulate the contribution of our study to the scientific debate. Reiterated very briefly, this would be to investigate whether the use of moral foundations in government communication translates into measurable change in behaviour, here specifically in the context of vaccination acceptance. Theoretical or empirical studies focussing on explaining the social and psychological chain of causality, however, would require a research design that allows zooming in on the motivational channels to accept a vaccination. This could be achieved through a survey design or the use of carefully selected micro-type (i.e. census) data.

Thank you for your suggestion to open the Discussion and Conclusion with a brief summary of the study and its purpose. We now add a brief paragraph to accommodate this.

Comment 9 *Other*

Tone of the manuscript

The tone of the manuscript (especially results and discussion) seems a bit assertive. I’d caution against the causal language used in some sentences throughout the manuscript, such as “...as this moral foundation increases vaccine hesitancy...” (line 473-). Again, it was just that the use of morality-related language predicted/was associated with vaccine rate (positively or negatively).

Response 9

Thank you for pointing this out to us. We have revised the tone of many of the formulations in this vein, toning down the assertive character of the language in the Results and Discussion Sections.

Comment 10 What the “use of moral foundations” means

The other of my major concerns is the language used in the manuscript. The meaning of the expression, “use of moral foundations,” sounds vague. It is the frequency of the moralityrelated word. I like to see how the language use and vaccine ratio are conceptually connected.

Response 10

Indeed “use of moral foundations” refers to the frequency of morality-related words in a given text, but also to which specific dimension gets triggered. If referred to “the frequency of morality-related words” instead, we would give the false impression that we would only look at a morality-related “bag of words” and sell short the detail of our analysis. We hope that the explicit link between intuition and vaccine acceptance on page 2 creates a more coherent picture of the conceptual connection between moral language and vaccine hesitancy.

Comment 11 SPECIFIC COMMENTS:

Page 11: I like to see how N = 22 was calculated; I like to see the number of the broachers

for each year.

Tables: The decimals should be aligned.

Page 20, Line 429: Please elaborate how it is counterproductive.

Page 20, Line 451: What is the “underlying study”?

Page 21, Line 469: Please provide some examples of the MFD in different languages. Response 11 For the information on a precise breakdown of brochures per year we kindly refer to Table 2. All vaccination brochures and the respective year of publication are listed there. We fixed the decimal alignment in Table 4. We add some brief elaboration on the point of some interventions turning out counterproductive. The underlying study is our study. We change the wording accordingly to avoid confusion. Lastly, we provide an example for a translation of the MFD into Japanese by Matsuo et al. [3].

Comment 12 Appendix (Page 233, Line 528): The description of the pluralist approach is confusing. Does pluralism demur the evolutionary explanation?

Response 12

Thank you for discovering this mistake. The term “demur” is incorrect here. The intention of the sentence on line 528 was, in other words, to express that pluralists object the monist perspective and argue that “evolutionary thinking encourages pluralism”. We replace the term “demur” by “argue”.

Reviewer 2

Thank you for your helpful feedback on our manuscript! Please find in the following a brief overview of how we addressed each of your comments.

Comment 1 Please describe the process used to translate the English MFD dictionary to Dutch. Moral foundations may vary with culture and usage of the English-translated dictionary may not be able to capture the socio-cultural aspects of a particular region and may potentially bias the results. How do you ensure that English dictionary can be generalised well to the Dutch language? Please refer to [3] to learn more about the translation methodology.

Response 1

Thank you for pointing out that the original manuscript has been too brief on this aspect. We now add the following explanation on page 9: “Similar to the method outlined by Harzing [2], both authors who are fluent/native Dutch speakers, independently translate the original English MFD words and word stems into Dutch and resolve any differences in the separately generated translations by discussion.”

In contrast to our translation method, Matsuo et al. [3] transform the English word stems into a subset of words. More concretely, we understand that for each word stem (they employ the example “justifi*”), Matsuo et al. [3] use a website to find the 11 most common words using a particular word stem (here: “justifiable”, “justifiableness”, etc.) and discard all other realisations. We respectfully believe that keeping the word stems allows us to cover a larger domain of words that may fall into the category. By the nature of Matsuo et al. [3]’s technique, by contrast, a certain number of potential words do not make the cut for the 11 most common words for a given word stem. The linguistic distance between English and Japanese probably requires this approach (as Japanese, for example, lacks the concept of punctuating words by blank spaces), but in our case we remain in the same language family with a very similar word and sentence structure.

Another difference between the approach by Matsuo et al. [3] and our study is that while we translate the words and word stems by human translators, Matsuo et al. [3] use an online dictionary and web scraping. We think both methods are theoretically feasible and valid.

Comment 2 The authors have used a moral foundation dictionary (MFD) and word count based methodology to capture the moral rhetoric used in the brochures. MFD created by a small group of experts with limited amounts of stem words and lemmas may not be valid for diverse contexts and populations [4]. The authors are advised to look at the applicability of the extended MFD [4] for their work. Also, many machine learning and word embedding based methods have been shown to better capture the moral rhetoric than word count based method. Authors should consider using such methods (for example [5, 6]).

Response 2

The eMFD [4], published last month, is certainly a promising new tool for research on MFT. The method we employ in our study, i.e. the MFD-based word count method, has established itself among a long stream of literature, i.e. [7, 8, 9, 10]. We believe the eMFD has opened a promising avenue towards an alternative approach for analysing the use of moral language in text. However, considering that the eMFD contains a different list of words than the MFD, the novelty of the eMFD and the established character of our text analysis method, we believe that utilising the eMFD for our research question would be a different paper and lies beyond the scope of this study.

Newly developed machine learning and word embedding methods have been described as superior to the keyword-based methods like LIWC when measuring the rhetoric related to specific topics in a text. For example, when analysing a text on the war on drugs, a given text may also discuss money laundering or other criminal activities. Methods as developed by Sagi and Dehghani [5] are able to measure the moral dimensions of each topic, rather than the text as a whole. Sagi and Dehghani [5] argue: “While keyword-based methods like LIWC can be used to measure the moral rhetoric over entire documents, we are interested in measuring the rhetoric related to specific topics.” The purpose of our study, however, is to analyse the moral rhetoric in government-issued vaccination brochures. These brochures are of interest to us in their entirety, for why we think it is safe to utilise the LIWC word count based method in our study.

On a different note, both the eMFD, as well as the machine learning approaches are currently limited to the five original

MFT dimensions and do not cover the sixth dimension Liberty/Oppression. In our study we are able to also include the sixth dimension into the analysis.

Comment 3 As mentioned by the authors the NIP started in 1957, please mention the reasons to select the data from 2011-2019 only.

Response 3

RIVM has started using standardised brochures to parents in 2011.

Comment 4 In equation (1), moral dimensions i.e. HC, PD etc have been shown as varying with regions. Are there different brochures corresponding to the different regions? Kindly clarify. It seems there are only 22 brochures in total from 2011 to 2019. The immunisation coverage data is available at the high spatial granularity, while the same is not true for moral dimensions data. How do you ensure that the OLS model is not overfitting the data?

Response 4

Thank you for noticing this notation mistake and apologies for the confusion this may have caused! Indeed, there are no specific brochures per region, so the “j”-indices are incorrect for the MFT dimensions in Equation (1). We will drop them from the manuscript. Concerning the issue of overfitting, we understand this problem occurs when a regression has too many dependent variables compared to the number of observations. To our knowledge, the inverse (a comparably small amount of dependent variables to describe a rich set of independent variables) does not constitute an overfitting problem. We believe the issue of dummy variables to be a good counter-example, in which only the absence or presence of a phenomenon is expressed on the part of the dependent variables. This popular regression technique finds widespread applicability in research while by its very nature the independent variables can only ever have two different values, zero or one. The MFT dimensions which we employ, by contrast, contain a much richer domain and bring about ample sufficient variation to the set of dependent variables.

Comment 5 How do you account for the different lengths of brochures in Table 5? What was the motivation to use absolute word count?

Response 5

Thank you for suggesting to control for the length of brochures. We have added the absolute word count per brochure as additional fourth control in Regressions (2), (4) and (6). The directionality of the results remains largely unchanged. We update the discussion of the results accordingly. The motivation for also conducting an analysis using an absolute word count measure in Table 9 was to provide a more transparent and accessible idea of the actual impact of our results. From a public health perspective, percent differences can be small but meaningful across a population. The absolute word count allows us to interpret the magnitude of our estimates as effect per morally relevant signal word.

Comment 6 Fairness dimension was not detected in the brochures and it is also the most difficult dimension to trace in language [11]. Authors may include this.

Response 6

Thank you for pointing out this interesting study. We agree that this finding is very relevant for us in the context of the traceability of Fairness/Cheating and we gladly add a reference to it on page 12 of the manuscript.

Comment 7 Kindly include some example statements from the vaccination brochures corresponding to different moral dimensions.

Response 7

Thank you for this idea which helps make our study more transparent and accessible. On page 29 we add a table with example sentences in their original and the translated form for each dimension. On page 9 we add a reference to the table in the main text.

Comment 8 The overall writing can be improved significantly. Few examples

(a) Page 4, line 73, 76, Subsection has been mentioned without referring to one.

(b) Page 16, line 330, “We test this theorem in our study”. Usage of the word “theorem” is incorrect.

(c) Please avoid vague terms such as “some” for results where statistical data can be provided e.g. Table 6 “some evidence”

Response 8

Thank you for pointing out the missing reference link to the respective subsections on page 4. On page 16 we change the term “theorem” to “theory”. Table 6 serves as a rough overview of the findings. The text and the regression tables present a much more concrete quantification of the results. We adjust “some evidence” to “weak evidence” as it may better represent the general character of the results for this particular MFT dimension.

Comment 9 Table 10 has been referred to in the text after Table 2. Please put them in a sequential order.

Response 9

Table 10 is in the appendix while Table 2 is part of the main text. As by the journal typesetting guidelines, the numbering of the tables is in the order at which they appear in the text, followed by tables that appear in the appendix. If the editor or journal has a specific preference, we are happy to revise this at an eventual typesetting phase.

Comment 10 n Figure 2, for better readability, please include 2012, 2014, etc on x-axis. Also, a descriptive caption can be provided which may help in better understanding of the figure.

Response 10

We added the years in the graph. Please find the caption for Figure 2 on page 11. PLOS ONE typesetting guidelines require authors to put figures after the manuscript and without caption. The caption is included in the manuscript text at the position where the figure would eventually be.

References

1. Rodewald L, Maes E, Stevenson J, Lyons B, Stokley S, Szilagyi P. Immunization performance measurement in a changing immunization environment. Pediatrics. 1999;103(Supplement 1):889–897.

2. Harzing AW. Does the use of English-language questionnaires in cross-national research obscure national differences? International Journal of Cross Cultural Management. 2005;5(2):213–224.

3. Matsuo A, Sasahara K, Taguchi Y, Karasawa M. Development and validation of the Japanese Moral Foundations Dictionary. PLOS ONE. 2019;14(3):e0213343.

4. Hopp FR, Fisher JT, Cornell D, Huskey R,Weber R. The extended Moral Foundations Dictionary (eMFD): Development and applications of a crowd-sourced approach to extracting moral intuitions from text. Behavior Research Methods. 2021;53(1):232–246.

5. Sagi E, Dehghani M. Measuring moral rhetoric in text. Social Science Computer Review. 2014;32(2):132–144.

6. Araque O, Gatti L, Kalimeri K. MoralStrength: Exploiting a moral lexicon and embedding similarity for moral foundations prediction. Knowledge-Based Systems. 2020;191:105184.

7. Long JA, Eveland Jr WP. Entertainment Use and Political Ideology: Linking Worldviews to Media Content. Communication Research. 2021;48(4):479–500.

8. Wheeler MA, McGrath MJ, Haslam N. Twentieth century morality: The rise and fall of moral concepts from 1900 to 2007. PLoS one. 2019;14(2):e0212267.

9. Mooijman M, Hoover J, Lin Y, Ji H, Dehghani M. Moralization in social networks and the emergence of violence during protests. Nature human behaviour. 2018;2(6):389– 396.

10. Clifford S, Jerit J. How words do the work of politics: Moral foundations theory and the debate over stem cell research. The Journal of Politics. 2013;75(3):659–671.

11. Kennedy B, Atari M, Davani AM, Hoover J, Omrani A, Graham J, et al. Moral concerns are differentially observable in language. Cognition. 2021;212:104696.

Submitted filename: Response_Letter1.pdf

Click here for additional data file.

15 Sep 2021

PONE-D-21-07601R1Using moral foundations in government communication to reduce vaccine hesitancyPLOS ONE

Dear Dr. Heine,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Oct 30 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Kazutoshi Sasahara

Academic Editor

PLOS ONE

Journal Requirements:

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

Additional Editor Comments (if provided):

Both reviewers agreed that the MS was improved. However, they also think that the MS needs several minor revisions. Please read their comments and address them properly.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: I am happy with the revision and only have some minor suggestions/comments as follows:

-In the subsection “The Dutch National Immunisation Programme (p. 5)”, the authors state “Our results confirm this trend, providing… (L. 103)”. Is it from the current study? If so, is it appropriate to show a part of the results here in the method section?

-Now that the authors clarify how they made the Dutch version of the MFD, I suggest the authors mention the translation issue as a limitation.

-In the Discussion and Conclusion section, the authors argue a possible difference between Authority and Liberty foundations. I would like to see some articles discussing the relationships between those (or other) foundations (if any). It can help readers to deeply speculate the results from the current study and moral foundations as a whole for future research.

-Thank you for showing how manuscripts are structured in your field (Response 7). It helps me.

Reviewer #2: Table 2 lists 26 brochures. Please highlight the brochures which were not considered for analysis in the table itself to avoid the confusion.

The paper needs serious proof-reading. There are several grammatical mistakes and other errors. e.g. the reference to subsections are missing at many places : pg 17- line 338, pg 10- line 222, page 10-line 224.

As mentioned in pg 17, line 335, fairness/cheating has been discarded from analysis. FC should also be removed from Equation 1.

Since MF dimensions do not vary spatially. I do not see the point of analysing it at national, regional and municipal levels. It is rather surprising that the coeffs for MF dimensions vary significantly at the different spatial resolutions since the MF dimensions and other control variables such as degree of difficulty (which are again derived from linguistic analysis of brochures) and time remain constant across the regions. Is it only modelling the variations in population sizes? Loyalty/betrayal effect became significant at regional and municipal level but not at the national level, however, the values of MF dimensions remain the same.

OLS with time-demeaning fixed effects would suggest that the moral dimensions remain constant over time. Is this a valid assumption?

Authors should not make strong claims regarding the effect of MF dimensions on vaccine hesitancy on the basis of only 22 sample points and there could be many other factors affecting the hesitancy which have not been considered for the analysis. Also, consider that the moral foundations are not completely independent of each other which is also evident from the MF dictionary words.

For line and page numbers, please refer to manuscript with track changes

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

11 Oct 2021

Reviewer 1

Thank you for your kind interest in our study and for the helpful feedback throughout the review process! Please find in the following a brief overview of how we addressed each of your comments.

Comment 1: In the subsection ``The Dutch National Immunisation Programme'' (p. 5), the authors state ``Our results confirm this trend, providing...'' (L. 103). Is it from the current study? If so, is it appropriate to show a part of the results here in the method section?

Response 1

We agree that this sentence is somewhat premature at this particular position in the manuscript. We present our empirical results concerning the time trend for vaccination acceptance in the Results Section on page 17, which we agree is a better position for this discussion. We propose to delete the sentence you refer to on page 5.

Comment 2

Now that the authors clarify how they made the Dutch version of the MFD, I suggest the authors mention the translation issue as a limitation.

Response 2

We now mention this as limitation both in the Methods Section and in the Discussion and Conclusion.

Comment 3

In the Discussion and Conclusion section, the authors argue a possible difference between Authority and Liberty foundations. I would like to see some articles discussing the relationships between those (or other) foundations (if any). It can help readers to deeply speculate the results from the current study and moral foundations as a whole for future research.

Response 3

Thank you for proposing this valuable addition to the manuscript, which we gladly include.

Comment 4

Thank you for showing how manuscripts are structured in your field (Response 7). It helps me.

Response 4

We are glad we could clarify this issue. We would like to reiterate that we are happy to adapt this editorial aspect of our article in an eventual typesetting phase, if the editor feels this would better fit PLOS ONE's publication culture.

Reviewer 2

Thank you for your kind interest in our study and for the helpful feedback throughout the review process! Please find in the following a brief overview of how we addressed each of your comments.

Comment 1

Table 2 lists 26 brochures. Please highlight the brochures which were not considered for analysis in the table itself to avoid the confusion.

Response 1

We now recognise how the representation of Table 2 seems to have caused confusion. Thank you for reiterating this issue to bring it to our attention again. As some brochures have comparably long names (i.e. ``Folder baby's van 2, 3, 4 en 11 maanden 2011''), there is a line break in the particular table cell. We now apply alternating light-grey row colours to tell different brochures from mere line breaks.

Comment 2

The paper needs serious proof-reading. There are several grammatical mistakes and other errors. e.g. the reference to subsections are missing at many places : pg 17- line 338, pg 10- line 222, page 10-line 224.

Response 2

We have performed another round of proofreading and adjusted a number of formulations which we hope elevates the language use within our manuscript to an acceptable level. We apologise for the dead links/references to some of the sections and subsections which we caused by changing the associated command structure within our typesetting software. We hope this issue has been resolved now.

Comment 3

As mentioned in pg 17, line 335, fairness/cheating has been discarded from analysis. FC should also be removed from Equation 1.

Response 3

Thank you for pointing this out to us. We agree that this dimension may be discarded from Equation 1. For consistency sake we also dropped it from our regression outputs in Tables 5 and 10.

Comment 4

Since MF dimensions do not vary spatially. I do not see the point of analysing it at national, regional and municipal levels. It is rather surprising that the coeffs for MF dimensions vary significantly at the different spatial resolutions since the MF dimensions and other control variables such as degree of difficulty (which are again derived from linguistic analysis of brochures) and time remain constant across the regions. Is it only modelling the variations in population sizes? Loyalty/betrayal effect became significant at regional and municipal level but not at the national level, however, the values of MF dimensions remain the same.

Response 4

The purpose of executing the analysis at different spatial levels is to assess the robustness of the results. Concretely, we are interested in demonstrating the results' sensitivity to different ways of measuring the same thing. We add a brief clarification on page 14.

Comment 5

OLS with time-demeaning fixed effects would suggest that the moral dimensions remain constant over time. Is this a valid assumption?

Response 5

The fixed effects panel is not the moral dimensions, which indeed vary over time. Our panel is the type of vaccination in a region. For data on the national level, this would be one of ten vaccinations (i.e. DTaP-IPV newborns, Hib, etc.). For the regional level this would be a convex combination of the ten vaccination types and 25 regions, resulting in 250 panels (equivalent for municipal data). More concretely (and ignoring the control variables for brevity sake) we try and estimate \\(y_{ijt}=\\alpha+\\boldsymbol{x}_{it}\\boldsymbol{\\beta}+\\nu_{ij}+\\epsilon_{ijt}\\), where $y_{ijt}$ is the vaccination rate for vaccination $i$ in region $j$ for time $t$. Further, let $\\boldsymbol{x}_{it}$ be the set of Moral Foundations scores for $i$ in $t$. $\\nu_{ij}$ then is the vaccination and region specific error term; it differs between units, but for any particular unit, its value is constant. Within the Netherlands there exists some degree of regional variation as to the general vaccination acceptance, and each type of vaccination also displays some heterogeneity in vaccination rate. By demeaning these fixed effects, we are able to better isolate the true $\\boldsymbol{\\beta}$ without the region- or vaccination-specific fixed effects. We add a clarifying sentence on page 15. (Kindly refer to the ``Response to Reviewers'' pdf for a proper display of the regression symbols.)

Comment 6

Authors should not make strong claims regarding the effect of MF dimensions on vaccine hesitancy on the basis of only 22 sample points and there could be many other factors affecting the hesitancy which have not been considered for the analysis. Also, consider that the moral foundations are not completely independent of each other which is also evident from the MF dictionary words.

Response 6

The first part of this comment seems akin to comment 4 from review round 1 which discusses the issue of overfitting. The subjects in our study have no influence on which version of the brochures they receive, i.e. there is no self-selection by subjects and exposure resembles random assignment. In that sense, we would categorise our research design as a natural experiment in that clusters of subjects have been exposed to different (experimental/control) conditions determined by factors outside the control of us investigators. Among the long tradition of natural experiments, Angrist and Evans [1], for example estimate the effect of family size on the mother's labour market outcomes. Sargent et al. [2] employ a temporary smoking ban in all public spaces in Helena (Montana) to investigate its effect on the rate of heart attacks at the only local hospital. We hope that these examples from a long tradition of literature may convince the reviewer that the 22 different treatment-like realisations in our study actually constitute a comparably fine granulation when comparing with other natural experiments.

This not withstanding, we do acknowledge that vaccine hesitancy is a multi-faceted phenomenon (see, e.g. page 17) and that many different factors play a role in determining the decision whether or not to accept a vaccine. Still, we believe that the power of our study is strong enough to contribute to increasing our understanding of this decision process and some of the aspects it is significantly impacted by. In the Discussion and Conclusion Section we discuss the relationship between the MFT dimensions.

References

1. Angrist JD, Evans WN. Children and Their Parents’ Labor Supply: Evidence from Exogenous Variation in Family Size. The American Economic Review. 1998;88(3):450–477.

2. Sargent RP, Shepard RM, Glantz SA. Reduced incidence of admissions for myocardial infarction associated with public smoking ban: before and after study. BMJ. 2004;328(7446):977–980. doi:10.1136/bmj.38055.715683.55.

Submitted filename: Response_Letter2.pdf

Click here for additional data file.

20 Oct 2021

Using moral foundations in government communication to reduce vaccine hesitancy

PONE-D-21-07601R2

Dear Dr. Heine,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Kazutoshi Sasahara

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Now both reviewers think that all the comments are properly address. Thank you for your careful revision.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

Reviewer #2: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

28 Oct 2021

PONE-D-21-07601R2

Using moral foundations in government communication to reduce vaccine hesitancy

Dear Dr. Heine:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Kazutoshi Sasahara

Academic Editor

PLOS ONE