Development and validation of a COVID-19 vaccine hesitancy scale for adults in the United States.

COVID-19 vaccines have been rapidly developed. However, widespread uptake remains a hurdle to a successful pandemic response. A simple, user-friendly survey to measure vaccine hesitancy may facilitate development of interventions aimed at maximizing vaccination. We developed a novel 10-item instrument designed to measure COVID-19 vaccine hesitancy in adults in the United States. We recruited 232 participants through Amazon's Mechanical Turk, an online crowdsourcing platform. The internal consistency (Cronbach's α = 0.89) and temporal stability (r = 0.87; p < 0.001) of our survey was strong. Lower hesitancy (high scores) was associated with higher trust in physicians (r = 0.58; p < 0.001), and higher hesitancy (low scores) was reported with higher belief in conspiracies (r = -0.68; p < 0.001). The correlation between low hesitancy and reported intent to receive (or history of receiving) at least one dose of the COVID-19 vaccine was moderate-strong (r = 0.68; p < 0.001).

Introduction

Coronavirus-19 (COVID-19) vaccines have been rapidly developed. Overcoming the COVID-19 pandemic will require very high inoculation rates [1]. A validated, user-friendly survey may help inform efforts directed towards countering COVID-19 vaccine hesitancy.

Although a validated instrument to measure general vaccine hesitancy (VHS) in adults exists, it does not specifically pertain to vaccination against COVID-19 [2]. Several factors are associated with decreased willingness to obtain a COVID-19 vaccine, including general mistrust in vaccines, concerns about unforeseen effects, protection duration, Food and Drug Administration (FDA) emergency use authorization, and conspiracy beliefs [3], [4], [5], [6]. Increased willingness to get vaccinated is associated with domestic vaccine production, endorsements from government organizations, and higher trust in doctors [3], [7].

We developed and validated a simple survey designed to measure adults’ hesitancy to receive the COVID-19 vaccine.

Materials and methods

Wake Forest University Health Sciences Institutional Review Board approval (#IRB00072726) was obtained in 2021 for all phases of this study.

Phase I: Prototype questionnaire development

Twelve prototype questions related to vaccine hesitancy were developed through a MEDLINE (PubMed) literature review using the linked keywords “covid”, “vaccine”, and “hesitancy”. A United States survey identified multiple factors associated with accepting a COVID-19 vaccine, which included both generic and COVID-19 specific vaccination concerns [3]. A nine-item validated scale designed to measure vaccine hesitancy in adults by Luyten and colleagues was identified for coverage of general concerns [2]. We developed 6 items covering factors specific to the COVID-19 vaccine: emergency use authorization process, origin of production, government endorsement, protection duration, unknown risks, and mild side-effects [3]. Three questions from Luyten and colleagues’ nine-item instrument were excluded before phase I data collection due to redundancy with COVID-19 specific items.

The items were grouped into three domains for analysis: general vaccine hesitancy (GVHS), COVID-19 specific concerns (CVHS), and overall hesitancy to obtain the COVID-19 vaccine (overall CVHS). Each item was scored on a 5-point Likert scale (1 – strongly disagree, 5 – strongly agree). Scoring was reverse-keyed for negatively phrased items. High scores correspond to low hesitancy.

Phase I: Pilot data collection

The first phase of development involved pilot data collection in which 50 subjects were recruited from Amazon Mechanical Turk (MTurk), a validated online crowdsourcing platform used extensively in psychosocial research [8]. Phase I participants were primarily caucasian (72 %) non-hispanic (88 %), females (54 %) with a mean age of 38 years, holding a bachelor’s degree or higher (64 %), from the midwest United States (38 %). All subjects were provided a factsheet which explained details regarding their study participation; consent was assumed based on voluntary participation. Subjects were directed to an online survey, which included a 12-item prototype questionnaire (comprised of 6 GVHS items and 6 CVHS items) and two validated survey instruments: Trust in Physician Scale (TIPS) and Generic Conspiracist Beliefs Scale (GCBS) [9], [10]. The TIPS measures levels of trust in physicians and assesses agreement with 11 statements on 5-point Likert scales [9]. The GCBS measures tendency to believe in conspiracies and is comprised of 11 statements; participants rate belief in each statement on a 5-point Likert scale [10]. Sociodemographic data were collected. Subjects were required to be 18 years of age or older and have an MTurk account. Subjects who provided inaccurate responses to attention check questions or failed to complete the entire survey were excluded from analysis.

Phase II: Instrument modification

Exploratory factor analysis (using the R factanal function with varimax rotation) was conducted to evaluate how strongly items loaded onto each factor; items which loaded poorly (<0.50) were removed. A scree plot was used to identify the number of factors with acceptable eigenvalues (>1).

Phase III: Instrument validation

The same methods used in phase I were used to recruit subjects; however, participants received the finalized 10-item COVID-19 vaccine hesitancy instrument. A total of 260 participants were recruited from MTurk, which included a subset of 21 participants who completed the survey again roughly-three weeks after initial administration (i.e. test–retest). Repeat participants’ initial responses were included in the final analysis.

Phase III: Statistical analysis

Confirmatory factor analysis was performed on the finalized 10-item set to verify the latent structure that resulted from the exploratory factor analysis conducted in phase II. Pearson’s correlation coefficients were calculated for test–retest measurements and for examining the relationship between our 10-item scale, TIPS, and GCBS. Point-biserial correlation coefficients were calculated and two-tailed t-tests were performed for known-group analysis when necessary. Cronbach’s alpha calculations were performed to assess internal consistency. Item-total correlations and communalities were also calculated. Cut-point analysis was conducted to identify a score cut-off to distinguish hesitant vs non-hesitant individuals, and the following three criteria were considered at all possible cut-off points: Youden’s J statistic (sensitivity + sensitivity – 1), shortest distance to the perfect point (0,1) on the receiver operating characteristic (ROC) curve, and sensitivity/specificity equality (the smallest absolute difference between sensitivity and specificity). Statistical analyses were performed using R (version 4.0.3), SAS (version 9.4), and Microsoft Excel.

Results & discussion

Descriptive statistics

Subjects (n = 260) were recruited between March 22, 2021 and April 15, 2021. Twenty-eight subjects were excluded from analysis (21 failed attention check questions; 7 incomplete surveys). A sample of 232 subjects, primarily Caucasian (78 %), democratic (53 %), males (56 %), mean age 39.1 ± 12.4 years (range 21–89 years), holding a Bachelor’s degree or higher (76 %), with children at home (56 %), residing in urban areas (45 %), located in the southern region of the United States (36 %), without a history of skin disease (68 %), and a reported intention to receive (or history of receiving) at least one dose of the COVID-19 vaccine (69 %) met inclusion criteria (Table 1 ).

Table 1

Sociodemographic data of survey respondents.

Parameter	Value
Age, years
Mean (SD)	39.1 (12.4)
Median	36 (21–89)
Sex
Female	97 (42 %)
Male	131 (56 %)
Decline to answer	4 (2 %)
Race
American Indian or Alaska Native	4 (2 %)
Asian	17 (7 %)
Black or African American	16 (7 %)
Caucasian	180 (78 %)
Native Hawaiian or Other Pacific Islander	1 (0 %)
More than one race	8 (3 %)
Unknown or not reported	6 (3 %)
Ethnicity
Not Hispanic	204 (88 %)
Hispanic	22 (9 %)
Unknown, not reported	6 (3 %)
Education
Less than or some high school	1 (0 %)
High school or GED	20 (9 %)
Associate degree or some college	35 (15 %)
Bachelor’s degree	138 (59 %)
Graduate school	38 (16 %)
Income
<$15,000 per year	12 (5 %)
$15,000 - $24,999 per year	18 (8 %)
$25,000 - $34,999 per year	36 (16 %)
$35,000 - $49,999 per year	41 (18 %)
$50,000 - $74,999 per year	64 (28 %)
$75,000 - $99,999 per year	27 (12 %)
$100,000 - $149,999 per year	23 (10 %)
$150,000 - $199,999 per year	3 (1 %)
>$200,000 per year	8 (3 %)
Children at home
Yes	131 (56 %)
No	101 (44 %)
Political party affiliation
Democratic	124 (53 %)
Republican	54 (23 %)
Independent	44 (19 %)
Other	10 (4 %)
Region of United States
Northeast	58 (25 %)
Midwest	58 (25 %)
South	83 (36 %)
West	33 (14 %)
Area of residence
Rural	48 (21 %)
Urban	104 (45 %)
Suburban	80 (34 %)
History of skin disease
Yes	74 (32 %)
No	158 (68 %)
Intent to receive (or history of receiving) at least one dose of the COVID-19 vaccine
Yes	159 (69 %)
No	73 (31 %)

Consistent with prior studies, greater hesitancy was associated with African-American race (r = 0.21; p = 0.001), having children at home (r = 0.24; p < 0.001), living in rural areas (r = 0.40; p < 0.001), residing in the northeastern United States (r = 0.21; p = 0.002), and identifying as a Republican Party supporter (r = 0.39; p < 0.001) [11]. Higher education levels were correlated with lower levels of COVID-19 vaccine hesitancy (r = 0.20; p < 0.001) [12]. These findings support known-group validity.

Although older individuals are less hesitant to receive the COVID-19 vaccine based on prior studies, no associations with age were detected in our sample (p = 0.076) [12].

Internal structure and reliability

In phase II, scree plot evaluation revealed two factors with acceptable eigenvalues (>1), 5.12 and 2.76, which accounted for 0.43 and 0.23 of the variance, respectively. Two COVID-19 specific items were removed from the 12-item prototype due to poor factor loading (<0.50).

In phase III, confirmatory factor analysis of the finalized 10-item set revealed a well-fitting model with a two-factor structure (Comparative Fit Index (CFI) = 0.99, Tucker-Lewis Index (TLI) = 0.98, root mean square error of approximation (RMSEA) = 0.05, and standardized root mean square residual value (SRMR) = 0.04). All factor loadings were significant (p < 0.001) (Table 2 ). Covariances between GVHS and CVHS were also significant (p < 0.001), supporting a correlation between the two factors. The correlation between GVHS and CVHS was further evidenced by repeating confirmatory factor analysis under the assumption of orthogonal (uncorrelated) factors, which resulted in a worse-fitting model (CFI = 0.96, TFI = 0.95, RMSEA = 0.083, SRMR = 0.191). Thus, a two-factor structure (GVHS, CVHS), consisting of 6 GVHS items and 4 CVHS items, showed the best psychometric properties; utilizing the 10-item overall CVHS score (sum of the 6-item GVHS and 4-item CVHS scores) is the most appropriate method of measuring hesitancy to receive the COVID-19 vaccine.

Table 2

Factor loading pattern and communalities of finalized 10-item scale.

Item	Factor loading onto general vaccine hesitancy	Factor loading onto COVID-19 specific vaccine hesitancy	Communalities
Vaccines are important for my health	0.86		0.75
Vaccines are effective	0.81		0.66
Being vaccinated is important for the health of others in my community	0.80		0.65
The information I receive about vaccines from the vaccine program is reliable and trustworthy	0.82		0.67
Getting vaccines is a good way to protect myself from disease	0.84		0.70
Generally, I do what my doctor or health care provider recommends about vaccines	0.75		0.56
I question the safety and effectiveness of the COVID-19 vaccine, because it went through an emergency use authorization process (accelerated FDA approval).*		0.81	0.66
I am reluctant to get the COVID-19 vaccine, because it offers only one year of immunity.*		0.80	0.65
I prefer to wait to get the COVID-19 vaccine, because there might be unknown risks associated with it.*		0.79	0.63
I would not get the COVID-19 vaccine if I knew I would experience even mild side effects.*		0.73	0.53
*negatively phrased item; responses were reverse-keyed

Communalities for the finalized 10-item set ranged from 0.53 to 0.75. Item-total correlations for the finalized 10-item set ranged from 0.56 to 0.78 (Table 3 ). Internal consistency was strong for the overall 10 item-scale (Cronbach’s α = 0.89), the 4 CVHS items (Cronbach’s α = 0.86), and the 6 GVHS items (Cronbach’s α = 0.92).

Table 3

Item-total correlation values for the finalized 10-item scale.

Item	Item-total correlation
Vaccines are important for my health	0.78
Vaccines are effective	0.72
Being vaccinated is important for the health of others in my community	0.76
The information I receive about vaccines from the vaccine program is reliable and trustworthy	0.74
Getting vaccines is a good way to protect myself from disease	0.75
Generally, I do what my doctor or health care provider recommends about vaccines	0.69
I question the safety and effectiveness of the COVID-19 vaccine, because it went through an emergency use authorization process (accelerated FDA approval).*	0.63
I am reluctant to get the COVID-19 vaccine, because it offers only one year of immunity.*	0.58
I prefer to wait to get the COVID-19 vaccine, because there might be unknown risks associated with it.*	0.56
I would not get the COVID-19 vaccine if I knew I would experience even mild side effects.*	0.61
*negatively phrased item; responses were reverse-keyed

Construct validity

As hypothesized, our 10-item overall CVHS scale displayed lower hesitancy with higher trust in physicians (r = 0.58; p < 0.001) and higher hesitancy with higher beliefs in conspiracies (r = -0.68; p < 0.001), supporting construct validity.[4], [7] The 6 GVHS items and 4 CVHS items also demonstrated lower hesitancy with higher trust in physicians (r = 0.55, r = 0.40, respectively; p < 0.001) and higher hesitancy with higher beliefs in conspiracies (r = -0.42, r = -0.74, respectively; p < 0.001). The correlation between low hesitancy and reported intent to receive (or history of receiving) at least one dose of the COVID-19 vaccine was moderate-strong (r = 0.68; p < 0.001). Divergent validity was demonstrated by non-significant associations (p > 0.05) with a history of skin disease and sex.

Temporal stability

Perceptions of the COVID-19 vaccine may be rapidly changing and willingness to receive a vaccine may be influenced by media coverage [13]. The test–retest temporal stability of the 4 CVHS items (r = 0.69; p < 0.001) suggests public attitudes towards COVID-19 vaccination may be malleable (hesitancy decreased over time in our sample of 21 test–retest participants). However, the 6 GVHS items (r = 0.92; p < 0.001) and our 10-item overall CVHS (r = 0.87; p < 0.001) demonstrated strong temporal stability. TIPS also displayed strong temporal stability (r = 0.82; p < 0.001). GCBS exhibited questionable stability (r = 0.53; p = 0.003); however, if the four outliers (i.e., test–retest differences outside the 25th and 75th quartiles by a margin>1.5 times the interquartile range) are removed, the temporal stability of GCBS is very strong (r = 0.97, p < 0.001).

Cut-point analysis: Defining hesitant vs non-hesitant

Youden J values were calculated for all possible cut-off points. The highest Youden J values, 0.48 and 0.46, corresponded to cut-off scores of 38 and 34, respectively. Distances to the ideal (i.e., perfect) point (0, 1) for cut-off scores of 38 and 34 were 0.47 and 0.38 (the smallest distance of all recorded scores), respectively. A cut-off of 38 (sensitivity = 53.5 %, specificity = 94.5 %, PPV = 95.5 %, NPV = 48.3 %) yielded greater differences in sensitivity and specificity (i.e., less equality) than a cut-off of 34 (0.41 vs 0.02). Thus, a score ≥ 34 on our 10-item scale (maximum score 50) was determined to be the ideal cut-point for defining a non-hesitant individual (sensitivity = 72.3 %, specificity = 74.0 %, PPV = 85.8 %, NPV = 55.1 %). Using this cut-point definition, 98 participants (42.2 %) in our sample were considered hesitant.

Limitations

Web-based surveys are convenient and cost-effective methods of collecting high-quality data from diverse geographic regions; however, this study has limitations. Recruited subjects were self-selected and more highly educated than the general population (75 % vs 32 % with a Bachelor’s degree or higher) [14]. Our sample had limited racial diversity (>70 % caucasian, >85 % non-hispanic) and hesitancy differs among demographics [15]. The results may not be broadly generalizable due to our sample size and our survey was only available to participants with internet access and an MTurk account. Additionally, a 10-item survey is unlikely to capture every factor that contributes to COVID-19 vaccine hesitancy. The variety of vaccine manufacturers and changing governmental regulations may also influence how vaccine hesitancy translates to the actual decision to receive the vaccine.

Conclusion

Broad acceptance and uptake of a COVID-19 vaccine are necessary for a successful pandemic response. The decrease in hesitancy over time in our test–retest sample of 21 subjects is consistent with the growing body of data on the safety and efficacy of the COVID-19 vaccine. However, the strong temporal stability suggests that hesitancy is not being fully addressed. Our validated tool can be used to facilitate the development of novel approaches to reduce vaccine hesitancy in adults in the United States. Future studies involving larger sample sizes comprised of multiple countries and languages are needed to further validate our scale.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Steven Feldman has received research, speaking and/or consulting support from a variety of companies including Galderma, GSK/Stiefel, Almirall, Leo Pharma, Boehringer Ingelheim, Mylan, Celgene, Pfizer, Valeant, Abbvie, Samsung, Janssen, Lilly, Menlo, Merck, Novartis, Regeneron, Sanofi, Novan, Qurient, National Biological Corporation, Caremark, Advance Medical, Sun Pharma, Suncare Research, Informa, UpToDate and National Psoriasis Foundation. He is founder and majority owner of www.DrScore.com and founder and part owner of Causa Research, a company dedicated to enhancing patients’ adherence to treatment. The remaining authors have no conflicts to disclose. More details regarding specifics can be found below: Abbvie---Grant Support, Speaker, Consultant. Alvotech---Consultant. Advance Medical---Consultant. Almirall---Consultant. Arcutis ---Consultant. Arena---Consultant. BMS---Consultant. Caremark---Consultant. Causa Technologies---Founder, Stock holder, Chief Technology Officer. Dermavant---Consultant. Amgen, Celgene---Grant Support, Consultant, Speaker. Informa---Royalties. Galderma Laboratories, L.P.---Grant Support, Consultant. Gerson Lehrman Group---Consultant. Guidepoint Global---Consulting. Helsinn---Consulting. Janssen---Grant Support, Speaker, consultant. Kikaku---Consultant. Leo Pharma Inc.---Consultant, speaker. Lilly---Consultant, speaker, grant. Medical Quality Enhancement Corporation---Stock (majority owner). Merck & Co., Inc.---Consultant. Mylan---Consultant. Novartis Pharmaceuticals Corporation---Grant support, Consultant, Speaker. Ortho Dermatology---Consultant, Speaker. Pfizer Inc.---Consultant, Grant support, speaker. Regeneron---Consulting, grant support, speaker. Sanofi---Consultant, Grant support, Speaker. Sienna---Consulting. Sun Pharma---Consulting, Speaker. Suncare Research---Consultant. UpToDate---Royalties. Xenoport---Consulting. Xlibris---Royalties.