Development and Validation of the Multidimensional COVID-19 Vaccine Hesitancy Scale.

The aim of the study was to develop and validate the Multidimensional Covid-19 Vaccine Hesitancy Scale (CoVaH), a self-report measure to assess the beliefs and attitudes beneath vaccination hesitancy and reasons for vaccine refusal in the context of Covid-19. A sample of 1503 Hungarian respondents filled out the scale. Exploratory and confirmatory factor analysis was conducted to identify latent constructs underlying participants' responses. Findings show a robust three-factor solution for the 15-item CoVaH with high factor loadings on each factor: skepticism, risk perception and fear of Covid-19 vaccine. The CoVaH displayed very good fit indices (KMO = .94, RMSEA = 0.049, CFI = .983) and internal consistencies (α values > .89) and was found to have proper convergent, concurrent and discriminant validity in identifying Covid-19 vaccine hesitancy in the general population. The new scale adds to the literature through the identification of the fear of COVID-19 vaccines, as a newly highlighted explanatory variable of COVID-19 vaccine hesitancy, besides the other formerly identified components. The scale, available in English and Hungarian, allows the assessment of vaccine uptake hesitancy and has the potential to help targeted interventions, considering individual factors that interfere with vaccination acceptance.

Introduction

At the onset of the Covid-19 crisis, the state of emergency was declared, entire sectors of activity were closed, telework or work at home was generalized and territories were quarantined in many countries all over the world. Several months later, in subsequent waves of the pandemic, these measures are reapplied over and over again. The ongoing Covid-19 pandemic has fast and prolonged consequences worldwide in healthcare, social and economic life.[1] In the context of economic policy uncertainties and socio-economic challenges,[2,3] global efforts „to flatten the curve” of COVID-19 infection cases are imperious. As a last resort, Covid-19 vaccines have the potential to overcome the continued danger caused by the infectious disease.

In less than one year after the Covid-19 global pandemic outbreak, several vaccines were developed and soon these have been made available. Still, as Covid-19 vaccine uptake remained a personal choice for many, despite government or workforce pressure, mass vaccination is out of reach in many countries, and COVID-19 vaccine hesitancy is putting herd immunity on hold.[4,5] In order to encourage public vaccination, factors that interfere with vaccination acceptance, should be taken into account.[6] The challenges of the global COVID-19 immunization program are connected to vaccine hesitancy in many countries. High levels of Covid-19 vaccine hesitancy led to lower vaccine uptake rates than expected in many countries,[4,5] and therefore the endeavor of authorities of gaining control over the pandemic is put at risk continuously, along with the chance for fast development of herd immunity. The planning of pro vaccination campaigns might need revision based on reasons of vaccine uptake delay or refusal.

Vaccine hesitancy is defined in multiple ways in the literature; an active debate is still ongoing. One definition puts vaccine hesitancy on a behavior continuum, which comprises the possibility of total refusal of vaccine intake on one side and the acceptance of vaccine intake on the other.[7] People can be very hesitant and therefore they end up deciding on permanently declining the vaccine, or can be less hesitant and ultimately, after careful scrutiny, decide upon the uptake of the vaccine. The final decision of vaccine uptake or refusal depends not on the information itself, but on the way how it is interpreted. Personal information processing is influenced by past experiences, knowledge, media influence, perceived risk and perceived benefits of vaccines, etc.[7]

Vaccine hesitancy is also defined as the tendency to delay the decision of taking or refusing the vaccine despite availability of it.[8,9] Larson et al.[9] in a systematic review highlighted a matrix of associative variables, which were grouped in three categories: (1) contextual variables (e.g. communication and media environment, pharmaceutical industry influences), (2) individual- and group- related variables (experience with past vaccination, beliefs, attitudes about health and prevention) and (3) vaccine-specific variables (scientific evidence of risks/benefits, vaccination schedule). All these add a very high complexity to the assessment and to the planning process of interventions aimed at lowering vaccine hesitancy in different populations. The vaccine hesitancy is a very volatile phenomenon, defined within the interplay of many factors, and further clarification is still needed on variables that play the most definitive role in it.

Vaccine hesitancy is often triggered by the lack of risk perception regarding the infectious agent′s effects on health.[4,10] The perceived risk of Covid-19 does not always ensure the vaccine uptake even if there are available vaccines.[8] Risk perceptions may lead to uncertainty toward new vaccines.[11,12] Socioeconomic variables and ethnic identity seem to significantly impact the Covid-19 vaccine hesitancy.[13] The challenges concerning Covid-19 vaccine uptake relate to psychological factors as well, besides development, testing and logistical issues.[14]

Fear of injections can also explain vaccine hesitancy.[15] COVID-19 vaccine hesitancy is associated with fear of adverse effects of vaccines.[9,16] Fearing the safety of the vaccine makes the hesitancy stronger,[17] the newness of the vaccine associates concerns and hesitancy.[10,18] Vaccine hesitancy was correlated with factors like vaccine risks and lack of confidence in the beneficial effects of vaccines in three distinct cultural environments.[19,20] Misinformation and conspiracy beliefs predict vaccine hesitancy through putting more emphasis on side effects and undesirable social control.[10,21] The negative influence of media contribute to low rates of immunization even in well-developed countries.[22]

Former findings highlight that negative information processing and negative framing of vaccines often shadow the beneficial effects of vaccines.[19] One of the possible explanations is concerned with evolutionary gains and the chance of survival. Negative information has a bigger impact over human decisions, information processing, memory and information processing, than positive information.[23] For example, prevention behavior in the context of COVID-19 was predicted by a tendency to focus on negative information in many generations.[24] Similarly, in the matter of vaccine uptake, negative information seems to elevate the chance of refusal or avoidance of the vaccines.[19,25] Vaccine hesitancy is associated with negative information processing and this can explain the overestimation of negative effects of vaccination and the underestimation of beneficial ones.

In line with these, the COVID-19 pandemic induced fear in the people.[26,27] Fear associated negative biases (e. g. negative framing, focus on negative information) were evinced in the literature in health related[19,23,25] and COVID-19 related information processing.[10,20] The fear relates to negative cognitive bias, which can make the vaccine risk seem greater, and skepticism toward the benefits of the vaccines appears.[26,28]

Furthermore, the misinterpretation of mortality-related events can add skepticism to the vaccination decision-making, which most likely will fuel immunization refusal. The more present the negative bias is in processing of mortality and negative emotion related events, the stronger the skepticism about the vaccine′s beneficial effects will be.[25]

Validated Covid-19 vaccine hesitancy assessment instruments

The ambiguous circumstances surrounding the Covid-19 outbreak, the misinformation, the anti-vaccine propaganda in the digital media, the lack of data on the Covid-19 vaccine′s long-term side-effects and the worldwide conspiracy theories make the general negativity focus more likely in the vaccine-hesitant population. To this day, Covid-19 Vaccine Hesitancy is assessed through factors measuring conspiracy, risks and lack of confidence. There are two validated scales published in the literature. The first is the Oxford Covid-19 Vaccine Hesitancy Scale,[10] a newly developed scale which comprises 7 items and measures vaccine hesitancy on a one-dimensional scale, and focuses mostly on the link between hesitancy and vaccine conspiracy beliefs. The second scale is the Vaccine Hesitancy Scale for COVID-19 Vaccination.[20] This covers two factors: “Lack of confidence” and “Risks” and was adapted for HIV patients. The 10-item scale assesses vaccine risk perception and vaccine confidence and was adapted for COVID-19 from the original Vaccine Hesitancy Scale.[19]

By the time of the present study′s data collection, the opportunity of registering for a Covid-19 vaccine was available all over Europe, however, mass vaccination (vaccination extended beyond priority groups) had just begun. Due to the large initial waiting lists, scheduling for the first or the second dose was far from optimal. People without chronic illnesses, outside risk job domains and younger generations still had to wait for their turn. Consequently, the data sample was heterogeneous in respect of COVID-19 vaccine uptake, not only due to individual vaccination hesitancy and undecidedness, but also because of the national waiting list management policy. Our data collection timing was optimal for the phenomenon under investigation. The territory of participant recruitment was legitimate, because the vaccination rate in these countries were and remained less optimal than in many other European countries, even in October 2021.[29]

Aim of study

The aim of our study was to build and validate a complex, multi-dimensional COVID-19 Vaccine Hesitancy Scale (CoVaH) that will help in the identification of possible explanatory factors of this complex phenomenon. The scale comprises items concerned with vaccine risk (adverse effects), fear (linked to individual emotional and physiological reactions) and lack of confidence/ skepticism in the vaccine’s beneficial effect on health and community. The assessment of such a volatile phenomenon like vaccine hesitancy needs a multifactorial approach. The possible threats of the COVID-19 pandemic can justify the addition of the fear factor, which was not considered in former COVID-19 vaccine hesitancy scales.[10,20]

Materials and methods

Participants

A total of N = 1503 adults completed the survey. This sample size is suitable for the statistical analyses required for scale validation (e. g. exploratory and confirmatory factor analysis).[30] The sample consisted of participants from Hungary and ethnic Hungarians mainly from Romania, with 4.5% from other European countries (e.g., Germany, England, Netherlands etc.). Mean age of the participants was 36.9, female participants made up 87% of the sample. The sample was heterogeneous in terms of educational level. Less than 20% were formerly diagnosed with Covid-19, and nearly one third of the sample refused the idea of getting vaccinated, while nearly 10% of participants have already completed the Covid-19 vaccine series (e.g., 2 dose series of Pfizer, 1 dose series of Johnson & Johnson). Approx. every fifth participant declared having a chronic disease. Descriptive statistics for sample characteristics are provided in Table 1.

Table 1.

Sociodemographics and medical characteristics of the sample (N = 1503)

Participant characteristics		N (%)
Age, Range (M± SD)		18–77 (36.88 ± 12.68)
Gender	Male	196 (13.0)
	Female	1307 (87.0)
Educational level	Grade 8 or less	3 (.2)
	Professional school/Grade 10	11 (.7)
	High school without baccalaureate	42 (2.8)
	Baccalaureate	449 (29.9)
	College, university	636 (42.3)
	Master’s degree	300 (20.0)
	PhD	46 (3.1)
	Other	16 (1.1)
Country	Ro	611 (40.7)
	Hu	824 (54.8)
	Other	68 (4.5)
Chronic disease	No	1192 (79.3)
	Yes	311 (20.7)
Flu vaccine past	No	1172 (78.0)
	Yes	331 (22.0)
Optional vaccine past	No	456 (30.3)
	Yes	1047 (69.7)
Diagnosed Covid-19	No	1073 (71.4)
	Yes	273 (18.2)
	Not sure	157 (10.4)
Vaccine intention	Do not consider	459 (30.5)
	Undecided	204 (13.6)
	Vaccinated	434 (28.9)
	Other	406 (27)
Vaccine type	Pfizer	430 (28.6)
	AstraZeneca	125 (8.3)
	Moderna	60 (4.0)
	Sinopharm	45 (3.0)
	Sputnik	70 (4.7)
	Johnson-Johnson	2 (.1)
	No answer	95 (6.3)
	Was not vaccinated	676 (45.0)

Values represent frequency and percentage, unless indicated otherwise.

Scale development and description

We proposed the measurement of COVID-19 hesitancy, covering vaccine-specific (e.g., lack of confidence/skepticism in beneficial effect; dangers of adverse effects), contextual (e. g. Covid-19 circumstances), individual- and group- related variables (e.g., fear of vaccine, skepticism/lack of confidence in individual and community benefits), in concordance with the results reported by Larson et al. in their systematic review.[9]

Considering evolutionary arguments in the context of Covid-19 pandemic for the stronger power of negative information processing in human decisions and acting,[23] fear related items were added to the scale.

The possible negative bias in human health-related information processing was scientifically highlighted in former studies in association with vaccine hesitancy.[10,19,23,25] Based on these, we selected especially COVID-19 Vaccine Hesitancy explanatory factors that can be linked to negative bias or negative interpretation of information. Besides fear, vaccine risk and vaccine skepticism were selected in concordance with other similar scales from the literature, as presented below.

The initial COVID-19 vaccine hesitancy scale included 32 items, inspired by preliminary validated scales. The preliminary scale covered three dimensions of vaccine hesitancy: vaccine fear, vaccine lack of confidence/skepticism and vaccine risk.

In total fourteen items were selected from two scales which measure fear (Fear of Covid-19 scale;[31] Surgical Fear questionnaire[32]), for example: the item “I am most afraid of Corona” was reformulated into “I am afraid of Covid-19 vaccines.”

In order to measure lack of confidence and skepticism regarding the COVID-19 vaccines, eleven items were selected and rewritten from two validated scales. One vaccine hesitancy scale was developed to measure differences in parental hesitancy about childhood vaccination.[19] The other scale measures skepticism toward emerging infectious diseases.[33] Sample item: “Childhood vaccines are important for my child’s health” was reformulated as “Covid-19 vaccines are important for my health..” Reverse scored items were used as well.

The survey assessed the Covid-19 vaccine related risk perception too, with seven items. These items were selected and rephrased from the vaccine skepticism scale developed by LaCour and Davis.[25] The scale measures diverse facets of vaccine skepticism. Items were linked to the potential risks and adverse effects of vaccines. Sample item ”Vaccines can lead to allergies.” rephrased to “Covid-19 vaccines can lead to severe allergic reactions (anaphylactic shock).”

This research sought to validate items on a Likert scale, because this type of scale is more feasible. Responses were rated on a 5-point Likert scale ranging from 1 (Strongly Disagree) to 5 (Strongly Agree). Greater scores indicate higher Covid-19 vaccine related hesitancy. For the final version of the CoVaH, see Appendix 1.

Procedure and data analysis

Online self-reported survey data was collected from 1st of May 2021 until 10th of June 2021.A structured survey was used for recording the sociodemographic information, medical history and vaccine intake of the participants. Participants were recruited on social media platforms via convenience sampling. They completed an initial 32 item version of Multidimensional Covid-19 Vaccine Hesitancy Scale and were inquired about vaccine intention, while sociodemographic and health data was also obtained.

Database was checked and no outliers were detected. Sample characteristics were reported using percentages for categorical variables and mean and standard deviation for continuous variables.

The Kaiser-Meyer-Olkin (KMO) test measure was calculated to check for sampling adequacy (MSA) and the Bartlett’s test of sphericity for sufficiently large item correlations for PCA.[34]

Confirmatory Factor Analysis (CFA) followed the initial Exploratory Factor Analysis (EFA) in the validation of the final version of the scale. For the identification of domains and the retention of items the following criteria was considered a priori:[34] for multicollinearity Kaiser’s criteria of eigenvalues > 1.0, items with factor loading > 0.4; intercorrelations were checked.

The Maximum Likelihood (ML) estimation procedure, the most frequently used fitting function for structural equation models was used. For a good model fit, the criteria for the Likelihood Ratio Test are supposed to be insignificant, however at larger sample sizes (N > 400) other fit indices are recommended over the highly sensitive chi-square statistic and the normed chi-square.[35,36] Due to the restrictiveness of the Model Chi-Square, researchers have sought alternative indices to assess model fit.[37] Therefore, guidelines for interpreting the model fit indices were as follows for absolute fit indices:

Chi-squared-the “original” measure of fit, a good model fit would provide an insignificant result, but it is sensitive to sample size,[35] due to the restrictiveness of the Model Chi-Square, researchers have sought alternative indices to assess model fit,[37] GFI-Goodness of fit index, values of .90 indicate well-fitting models and values of .95 or greater excellent fitting models,[38,39] RMSEA-Root Mean Square Error of Approximation appraises an excellent fit of the model by a value that is lower than .05;[40] for relative (incremental) fit indices: CFI-Comparative Fit Index, TLI-Tucker–Lewis Index, AGFI- Adjusted goodness of fit index, NFI-Normed fit index values greater than .95 are excellent,[38,39,41,42] IFI -Bollen’s incremental fit index over .90 is a good fit, .95 or greater indicate excellent fit,[43] and for the parsimonious fit indices χ2/df the values should be less than 5.0.[44]

For reliability estimates, Cronbach alpha values were provided.

Sensitivity, specificity, and positive predictive value for the detection of vaccination intention/status were determined through plotting receiver operating characteristics (ROC) curves. Pearson correlation and independent samples t test analyses were conducted to evaluate the construct and criterion validity of the scale and independent samples t test and one-way ANOVA (and the more robust Welch statistics for unequal distributions) were computed to check the sociodemographic and health status differences in reported vaccine hesitancy. Conservative Games-Howell post hoc analyses were used to control for sample size effects. Alpha level was set at p ≤ .05. Effect sizes (Hedges’s g for independent samples t test groups with uneven sample size and Cohen’s f for one-way ANOVA) were calculated.

Except for the CFA (AMOS), all statistical calculations were performed in SPSS version 23.0.

Results

Structure, model fit and internal consistency

Exploratory factor analysis

The factorability of the 32 items of the Multi-dimensional Covid-19 Vaccine Hesitancy Scale was examined. Four interpretable factors were identified via Exploratory Factor Analysis (EFA). Initially, five items were excluded because they either loaded similarly on two factors, or had weak factor loadings. Based on their semantic content, factors were labeled as skepticism (Covid-19 vaccine skepticism), risk (hesitancy due to risks of Covid-19 vaccines), fear-physiological (fear of physiological side effects), fear-cognitive (fear of getting vaccinated). Ultimately, the fear-cognitive factor (5 items) was dropped due to unacceptable internal consistency, as no substantial increases in alpha could have been achieved by eliminating more items. A three-factor solution was then accepted and further 7 items eliminated based on their factor loadings and inter item correlations in Confirmatory Factor Analysis (CFA).

Overall, a total number of 17 items of the initial 32 items of the original scale were excluded from the final analysis. The 15-item scale was subjected to principal component analysis (PCA) using orthogonal rotation (varimax). The Kaiser-Meyer-Olkin (KMO) test measure of sampling adequacy (MSA) was excellent KMO = .94 and the Bartlett’s test of sphericity was χ2(105) = 18306.32 (p < .001), indicating sufficiently large item correlations for PCA. Three components had eigenvalues greater than 1 and they explained 74.3% of the variance of vaccine hesitancy. All communalities were above .5 confirming that items shared some common variance with other items. All items in this analysis had primary loadings over .6. Table 2 presents the factor loadings after rotation.

Table 2.

Summary of exploratory factor analysis for the Multidimensional COVID-19 Vaccine Hesitancy Scale/CoVaH (N = 1503)

Items		Component
		1. Skepticism	2. Risk	3.Fear
1.	A Covid-19 védőoltások beadatása jó mód arra, hogy megvédjem magam a Covid-19 betegségtől (R).(Getting Covid-19 vaccines is a good means to protect myself from the Covid-19 disease (R).)	.851	.383	.167
2.	A Covid-19 elleni védőoltások fontosak az egészségem szempontjából (R).(Covid-19 vaccines are important for my health (R).)	.850	.351	.147
3.	A védőoltás fontos a közösségem tagjainak egészsége szempontjából (R).(To be vaccinated is important for the health of others in my community (R).)	.837	.364	.177
4.	A Covid-19 elleni védőoltások hatékonyak (R).(Covid-19 vaccines are effective (R).)	.798	.417	.185
5.	Általában azt teszem, amit az orvosom javasol a Covid-19 védőoltásokkal kapcsolatban (R).(Generally, I do what my doctor or health care provider recommends about Covid −19 vaccines (R).)	.766	.206	.109
6.	A Covid-19 védőoltások súlyos allergiás reakciókat okozhatnak (anafilaxiás sokk).(Covid-19 vaccines can lead to severe allergic reactions (anaphylactic shock).)	.264	.783	.218
7.	A Covid-19 védőoltások kiválthatják azt a betegséget, amely ellen épp védelmet kellene nyújtaniuk.(Covid-19 vaccinations can cause the very illness they are designed to prevent.)	.379	.755	.207
8.	A Covid-19 elleni védőoltások vérrögöket és komplikációkat (pl. trombózis, embólia) okozhatnak.(Covid-19 vaccinations can cause blood clots and complications (ex. thrombosis, embolism).)	.238	.752	.220
9.	A Covid-19 védőoltások túlterhelik az immunrendszert.(Covid-19 vaccines overwhelm the immune system.)	.326	.734	.241
10.	Egyes újonnan megjelenő betegségeket (pl Covid-19-et is), stratégiailag hoztak létre és terjesztettek azért, hogy védőoltásokat dolgozhassanak ki.(Certain emerging diseases, like Covid-19 were strategically created and spread by the government in order to have the opportunity to develop vaccines.)	.271	.652	.146
11.	Óvatosan kell kezelnünk a Covid-19 védőoltással kapcsolatos ajánlásokat.(We have to be cautious about the recommendations that are given to the general public regarding the Covid-19 vaccine uptake.)	.357	.647	.167
12.	Kiráz a hideg (libabőrös leszek), amikor arra gondolok, hogy beoltanak valamelyik Covid-19 elleni védőoltással.(I have chills (goosebumps) when I think about being vaccinated with one of the Covid-19 vaccines.)	.197	.191	.869
13.	Felfordul a gyomrom (hányingerem, gyomoridegem van), amikor arra gondolok, hogy beoltanak valamelyik Covid-19 elleni védőoltással.(I have an upset stomach (nausea, butterflies) when I think about being vaccinated with one of the Covid-19 vaccines.)	.204	.204	.845
14.	A szívem gyorsabban ver (pulzusom megnő) amikor arra gondolok, hogy beoltanak valamelyik Covid-19 védőoltással.(My heart beats rapidly (change in heart rate) when I think about being vaccinated with one of the Covid-19 vaccines.)	.147	.251	.818
15.	Alvászavarom van, mert aggódom, hogy beoltanak az egyik Covid-19 elleni védőoltással(I have sleeping difficulties because I’m worrying about being vaccinated with one of the Covid-19 vaccines.)	.036	.155	.816
Eigenvalues		7.994	2.081	1.067
% of variance		53.3	13.9	7.1
Cumulative %		53.3	67.2	74.3
α subscales		.944	.895	.891
α total scale		.937

Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. Rotation converged in 5 iterations. 1,2,3,4,5 is reversed items (R).

Based on the mean of the items with their primary loadings on each factor, composite scores were provided for each of the three factors. Higher scores indicated greater Covid-19 vaccine hesitancy on each subscale.

Confirmatory factor analysis

Confirmatory Factor Analysis (CFA) was conducted for the three-factor structure model (see Figure 1).

Figure 1.

Factor structure of the Multidimensional COVID-19 Vaccine Hesitancy Scale (CoVaH) .

The normed chi-square, χ2/df = 4.65 is indicative of an acceptable model fit in this case. The model fit indices of the Multidimensional COVID-19 Vaccine Hesitancy Scale are presented in Table 3.

Table 3.

Model fit indices of Multidimensional COVID-19 Vaccine Hesitancy Scale

	χ2	df	p	χ2/df	GFI	AGFI	NFI	IFI	TLI	CFI	RMSEA
3 factor structure/ 15 items	404.863	87	.000	4.654	.964	.950	.978	.983	.979	.983	.049

χ2 = Chi-square statistic; GFI = Goodness of fit index; AGFI = Adjusted goodness of fit index; NFI = Normed fit index; IFI = Bollen’s incremental fit index; TLI = Tucker–Lewis index; CFI = Comparative fit index; RMSEA = Root mean square of error approximation; N = 1503.

Reliability/internal consistency

The subscales showed very good internal consistency: Cronbach’s alpha for 5-item skepticism α = .94, 6-item risk α = .89, 4-item fear α = .89, and the internal validity of the total scale is also excellent, α = .93.

Construct validity (convergent validity)

All three subscales showed high correlation with the total score on the full scale (skepticism r(1501) = .893, p ≤ .01; risk r(1501) = .917, p ≤ .01; fear r(1501) = .678, p ≤ .01) and the subscales were positively correlated with each other (skepticism with risk r(1501) = .724, p ≤ .01; skepticism with fear r(1501) = .412, p ≤ .01; risk with fear r(1501) = .510, p ≤ .01).

Criterion validity (concurrent validity)

Group differences were calculated for concurrent validity (see Table 4). Participants who do not consider taking any Covid-19 vaccine reported greater vaccine hesitancy than participants who are undecided (consider getting vaccinated, but did not yet take any steps toward it) and the participants who have already completed the vaccination series for any Covid-19 vaccine, these differences were all significant. Games-Howell′s post hoc analysis revealed intergroup differences among all three groups, all ps < .001. All Cohen’s f effect sizes were large.

Table 4.

Group differences for criterion (concurrent) and validity

		Vaccine hesitancy	Skepticism	Risk	Fear
Vaccine intention	Do not consider	53.29 (10.12)	22.34 (3.21)	22.54 (5.48)	8.40 (5.09)
	Undecided	39.97 (9.56)	15.92 (4.50)	17.98 (5.27)	6.09 (3.42)
	Vaccinated	26.04 (8.67)	9.36 (4.39)	12.08 (4.85)	4.59 (1.93)
	df1, df2	3,691.90	3,664.26	3,691.17	3,647.86
	F/FWelch	691.76	1037.35	351.17	79.32
	p	≤.001	≤.001	≤.001	≤.001
	Cohen’s f	3.99	2.91	2.07	0.92
Flu vaccine past	No	39.32 (15.20)	15.72 (6.89)	17.22 (6.94)	6.37 (4.08)
	Yes	29.93 (12.13)	11.15 (5.64)	13.68 (5.80)	5.09 (2.83)
	df	651.37	635.55	622.42	755.87
	t/tWelch	11.71	12.37	9.34	6.49
	p	≤.001	≤.001	≤.001	≤.001
	Hedges’s g	0.64	0.69	0.53	0.33
Optional vaccine past	No	46.03 (14.30)	18.78 (6.16)	19.87 (6.51)	7.38 (4.86)
	Yes	33.43 (13.77)	12.95 (6.44)	14.95 (6.47)	5.52 (3.20)
	df	1501	1501	1501	633.04
	t/tWelch	16.12	16.33	13.51	7.46
	p	≤.001	≤.001	≤.001	≤.001
	Hedges’s g	0.90	0.92	0.76	0.49
Gender	Male	35.13 (14.23)	14.72 (7.00)	15.38 (7.08)	5.02 (2.60)
	Female	37.57 (15.19)	14.72 (6.89)	16.60 (6.82)	6.25 (4.01)
	df	1501	1501	1501	351.59
	t/tWelch	2.11	0.01	2.31	5.683*
	p	<.05	.992	<.05	≤.001
	Hedges’s g	0.16	0.00	0.18	0.32
Country	Romania	35.55 (13.69)	14.02 (6.46)	15.79 (6.37)	5.73 (3.33)
	Hungary	38.19 (15.81)	15.02 (7.10)	16.85 (7.09)	6.31 (4.19)
	df	1398.93	1374.58	1381.07	1426.05
	t/tWelch	3.37*	2.78*	2.96*	2.88*
	p	≤.001	≤.01	≤.01	≤.01
	Hedges’s g	0.18	0.15	0.16	0.15
Cronic disease	No	37.81 (14.93)	15.12 (6.84)	16.58 (6.79)	6.12 (3.88)
	Yes	35.09 (15.51)	16.18 (6.91)	15.92 (7.16)	6.00 (3.87)
	df	1501	1501	465.89	1501
	t/tWelch	2.83	4.43	1.469	0.482
	p	≤.001	≤.001	.142	.630
	Hedges’s g	0.18	0.28	−0.01	−0.031
Diagnosed Covid-19	No	37.06 (15.46)	14.62 (7.04)	16.32 (6.96)	6.11 (3.95)
	Yes	36.24 (14.21)	14.33 (6.64)	16.32 (6.83)	5.58 (3.33)
	df	450.123	440.936	1344	484.55
	t/tWelch	0.833	0.631	0.001	2.25*
	p	.405	.529	1.000	<.05
	Hedges’s g	−0.05	−0.04	0.00	0.14

*Welch statistic.

Participants who refused the flu vaccines in the past reported greater vaccine hesitancy than participants who did not refuse the flu vaccines in the past; in the case of all variables the results are significant; effect sizes are medium, except for the fear subscale which holds a small effect size.

Participants who refused any other optional vaccine in the past reported greater vaccine hesitancy than participants who had not refused the optional vaccines in the past. These differences were also significant, with large and medium effect sizes.

Discriminant validity

ROC analysis (Receiver operating characteristic) was performed in order to determine the best cutoff scores (highest true positive rate together with the lowest false positive rate) of the instrument, with proper sensitivity and specificity. The self-reported vaccine intention was taken into account (unvaccinated/vaccinated).

The Area Under Curve (AUC) for the CoVaH total was .97 (95% CI: .96-.98), p ≤ 0.001, which is an excellent value. The cutoff score was as follows: scores above 38.50 will screen positive in 93% of the unvaccinated population, which means very good sensitivity; specificity is 90.3% and it has 9.7% false negative probability.

The AUC for the Skepticism subscale was .97 (95% CI: .96-.98), p ≤ .001, which is also an excellent value. The cutoff score was as follows: scores above 16.50 will screen positive in 93.2% of unvaccinated population, which means very good sensitivity; specificity is 92.4% and it has 7.6% false negative probability.

For the Risk subscale the AUC was .91 (95% CI: .89-.93), p ≤ .001, which is a very good value. The cutoff score was as follows: scores above 17.50 will screen positive in 81.3% of unvaccinated population; specificity is 84.8% and it has 15.2% false negative probability, both are good values.

In the case of Fear subscale, the AUC was .74 (95% CI: .70-.77), p ≤ .001, which is not excellent, but has a fair value since 0.6 is spotted. The sensitivity of this subscale is weak, scores above 5.50 will screen positive only in 52.7% of unvaccinated population; specificity is 89.6% and it has 10.4% false negative probability, which is a good value.

This reveals that the total instrument and its subscales, with the exception of Fear subscale, can significantly distinguish between unvaccinated and vaccinated populations.

Group differences in reported vaccine hesitancy based on sociodemographics and health status

There were significant differences in reported vaccine hesitancy for gender in the case of total vaccine hesitancy score and for two subdimensions, namely risk and fear, with higher values for women on both the total score and the above mentioned subscales, however, all effect sizes were small. No difference was found between men and women regarding skepticism.

We found differences based on countries for all dimensions, vaccine hesitancy total, skepticism, risk and fear, with significantly higher scores for participants from Hungary compared to participants from Romania; all effect sizes were small.

We also found significant differences in reported vaccine hesitancy based on health status. Participants who reported the presence of a chronic disease scored significantly lower on the vaccine hesitancy total and skepticism, with small effect sizes. In the case of risk and fear subscales no differences were detected.

Participants formerly diagnosed with Covid-19 scored lower only on fear as compared to participants without any Covid-19 diagnosis, however the effect size was small. On the total vaccine hesitancy score and the subscales of skepticism and risk there were no significant differences between the two groups.

Discussion

In this study we aimed at elaborating a valid multidimensional scale for the measurement of Covid-19 vaccine hesitancy. In the process of development of the COVID-19 Vaccine Hesitancy Scale (CoVaH) we considered former highlights from the literature connected specifically to COVID-19 vaccine hesitancy and other vaccine hesitancy studies. Some of these highlighted the role of skepticism and vaccine risk in vaccine hesitancy,[10,18-20] others dealt with media and online misinformation,[22] fear of needles,[15] vaccine risks and lack of confidence in benefits of vaccination,[20] conspiracy theories, mistrust,[10] and also uncertainty in newer vaccines.[11,12]

CoVaH comprised three factors (vaccine risk, skepticism and vaccine fear) in COVID-19 Vaccine Hesitancy. Fear related items were added to the scale based on evolutionary arguments regarding human reactions in threatening circumstances.[23]

The results of the study revealed an adequate structural model in CoVaH. The final, validated version of the scale was established based on several widely recognized and recommended inclusion and exclusion criteria in statistical analysis. In this way, EFA and CFA revealed a robust three-factor structure for the 15 retained items, with skepticism (skepticism about vaccines and their benefits), risk (hesitancy due to the risks of vaccines) and fear (fear of getting vaccinated) sub-scales. The high internal consistency of the subscales justifies this multidimensional approach.

Results show that CoVaH has adequate convergent and concurrent validity and was related to other behaviors such as Covid-19 vaccine uptake, flu vaccine uptake and other past optional vaccine uptake. It is of interest to consider the difference in Covid-19 vaccine hesitancy based on former vaccination history (e.g., flu vaccine, optional vaccines beyond national immunization programmes), to outline a greater hesitancy among those who refused flu vaccines and did not opt for any optional vaccines in the past. This can suggest a continuity and stability in behavior regarding vaccine hesitancy, indicating a personal pattern, which can be justified by information processing styles, knowledge and cognitive biases.[26,28]

Based on the vaccination status/intention, medium to large differences in vaccine hesitancy were demonstrated. Moreover, good discriminant properties of the scale were revealed, indicating that CoVaH can accurately detect vaccine uptake intention/ predict vaccination status, being able to differentiate in more than 90% of the cases the unvaccinated from the vaccinated. All three subscales have good or very good values for differentiating between those vaccinated and unvaccinated, and can be used separately as well.

The scale was found to be slightly sensitive to sociodemographic and health status variables, namely women, participants from Hungary and participants without any chronic disease expressed higher vaccine hesitancy, though differences had small effect sizes. Sensitivity to sociodemographic variables was found in former studies as well.[6,10,19] This is in line with Larson’s vaccine hesitancy matrix that, besides the contextual and group factors, highlights the individual determinants of hesitancy.[9]

Given the complexity of vaccine hesitancy, strategies of addressing this phenomenon should take into account the reasons for hesitancy. Our results are in concordance with other findings, which highlighted the reaction of fear in COVID-19 vaccine delay decisions[26] and other vaccine uptake as well.[10,15,18] For rational health related decisions, strategies targeting the control of fear, skepticism and the attitude toward vaccine risks is advisable.

Conclusions

In conclusion, we recommend the use of CoVaH for the measurement of Covid-19 vaccine hesitancy and the prediction of vaccination intention, since it is a valid and reliable tool that also permits the identification of vaccine uptake refusals and can help possible interventions which target COVID −19 vaccine hesitancy.

In terms of possible explanatory variables of vaccine hesitancy, COVID-19 vaccine fear is the new component identified in the COVID-19 vaccine hesitancy structural model, compared to former scales.

Our data suggest that fear as an emotional reaction should be considered in vaccine hesitancy, besides skepticism and vaccine risk.

The strengths of this study include the large sample size. Additionally, it is important to note that the timing of data collection was optimal for investigating the phenomenon of Covid-19 vaccine hesitancy. Furthermore, the CoVaH scale is the first three-factor structured scale, which measures the Covid-19 vaccines hesitancy, and adds to the literature with a more diverse model of vaccine hesitancy and with a very good overall specificity and sensitivity to differentiate unvaccinated people from vaccinated ones. The development and the validation of this instrument relies on a robust methodology and on a representative sample, and all data suggest that this is a useful and valid instrument for professionals who would like to assess COVID-19 vaccine hesitancy.

Despite these contributions, this research has some limitations. First, we would like to note that social desirability was not measured in our population, which could bias the results. The data was collected online, from a convenience sample, in which males were underrepresented, and that can affect generalizability. Another limitation of the research is the lack of comparison to other multidimensional scales on Covid-19 Vaccine Hesitancy, because, at the moment of data collection, the literature lacked any.

We recommend this tool to be applied in subsequent longitudinal studies, to further estimate the predictive validity of vaccine hesitancy, using the vaccination status as a criterion. Future studies should focus on test-retest methodology, validation in other cultures (e. g. more developed countries) and on testing CoVaH predictive validity compared to other multidimensional COVID-19 vaccine hesitancy scales.

Click here for additional data file.