A Rasch analysis assessing the reliability and validity of the Arizona CoVHORT COVID-19 vaccine questionnaire.

BACKGROUND: Despite the widespread availability of COVID-19 vaccines in the United States, many that have chosen not to be vaccinated have done so because of vaccine hesitancy. This highlights the need for tools that accurately capture the knowledge, attitudes, and beliefs towards COVID-19 vaccines, and provide steps toward improving vaccine acceptance.
METHODS: Participants of the Arizona CoVHORT (COVID-19 Cohort) received a one-time, electronic based cross-sectional questionnaire intended to capture underlying motivations regarding vaccination, as well as hesitations that may prevent people from getting vaccinated. Rasch analysis was conducted among 4703 CoVHORT participants who had completed the vaccine questionnaire to assess questionnaire reliability and validity. Response categories were grouped to optimize scale functioning and to ensure independent probabilities of participant endorsement.
RESULTS: A total of 4703 CoVHORT participants completed the questionnaire, of whom 68% were female, and who had a mean age of 48 years. Participants were primarily White (90%), highly educated (63% with a college degree or above, with most respondents (45%) having an income of more than $75,000 per annum. The results indicated the questionnaire has good reliability and construct validity for assessing attitudes and beliefs about the COVID-19 vaccines. In-fit mean-squares for included items ranged from 0.61 to 1.72 and outfit mean-squares ranged from 0.56 to 1.75, and correlation coefficients ranged from 0.25 to 0.75. The person-item map indicated normal distribution of logit scores measuring perceptions about COVID-19 vaccinations.
CONCLUSIONS: The CoVHORT vaccine questionnaire demonstrated satisfactory reliability and construct validity in assessing attitudes and beliefs about COVID-19 vaccines. Overall results provide a starting point for a reliable and valid tool to assess knowledge and perceptions about COVID-19 vaccination, ultimately providing public health professionals with an instrument to assess the factors that are associated with vaccine acceptance or hesitancy.

Introduction

The COVID-19 pandemic has resulted in over 190 million infections and 4 million deaths since the novel SARS-CoV-2 virus first emerged in December 2019 (Data, 2021Data). Mitigating behaviors, such as stay-at-home orders, quarantine and isolation have been implemented to slow the transmission of SARS-CoV-2 but have not been enough to end the pandemic. These efforts, while necessary to prevent transmission, resulted in large disruptions to daily life, including the closure of schools and businesses, leading to economic and social disruption. Despite these mitigating efforts, widespread global vaccination will be necessary to contain the pandemic (Bell, 2020).

By December 2020, two different vaccines, manufactured by Moderna and Pfizer, received Emergency Use Authorization (EUA) in the United States (US), allowing for vaccination outside of clinical trials (Administration, 2021a, 2021b). As of February 2021, one additional manufacturer, Johnson and Johnson's Janssen, gained EUA which provided a one-dose vaccination option (Administration, 2021c). Despite these three options, COVID-19 vaccination rates have begun to slow across the country prior to achieving herd immunity, with the US falling short of the goal of having 70% of the adult population vaccinated by July 4, 2021. Possible explanations for slowed vaccine uptake include misinformation regarding the vaccine and its development, historical medical and government mistrust, and issues related to accessing the vaccine (Boulware et al., 2003; Khubchandani et al., 2021).

In identifying and addressing the underlying causes of vaccine hesitancy and misinformation and determining how to improve vaccine acceptance, it was important to create a tool that accurately captures attitudes, perceptions, and beliefs about the COVID-19 vaccines. To address this gap, the Arizona CoVHORT, a longitudinal COVID-19 study, disseminated a cross-sectional vaccine questionnaire among participants in March 2021. The vaccine questionnaire intended to capture underlying motivations regarding vaccination, as well as hesitations that may prevent people from getting vaccinated. The Arizona CoVHORT was established in May of 2020 by an interdisciplinary team of researchers at The University of Arizona and Arizona State University (Catalfamo et al., 2021). CoVHORT aims to capture the long-term effects of SARS-CoV-2 infection in infected individuals and the effects of the COVID-19 pandemic on cases and non-cases. Questionnaire items covered a wide range of topics including physical and mental health, lifestyle changes, maternal and fetal outcomes, chronic disease history and behavioral risk factors (Catalfamo et al., 2021). Detailed information on the study design of the CoVHORT has been published elsewhere (Catalfamo et al., 2021).

This paper aims to assess the reliability and validity of the questionnaire used to measure the attitudes, perceptions, and beliefs of the Arizona CoVHORT participants concerning the COVID-19 vaccine during the start of distribution in the US. Determining the reliability and validity of this questionnaire will allow other researchers to use the instrument to measure and compare vaccine attitudes, perceptions, and beliefs across different populations.

Methods

Rasch analysis was conducted on a cross-sectional sample of 4703 CoVHORT participants who completed the vaccine questionnaire. To meet the assumptions of Rasch analysis, we selected a subset of the vaccine questionnaire items for reliability and validity analyses. All items selected for Rasch analyses were focused on measuring participants’ perspectives on vaccine uptake, specifically focused on reasons that they would or would not receive the vaccine.

CoVHORT participants began to receive the questionnaire, using the electronic data capture and management tool REDCap™ (Nashville, TN, US), beginning March 2, 2021. All enrollees at the time, over the age of 17, received an initial questionnaire invitation sent via email. Three reminder emails were sent every three days to encourage participants to complete the questionnaire. Participant consent was obtained as part of enrollment into the CoVHORT study using the REDCap system as well. This study has been approved by the University of Arizona's Humans Subjects Protection Program (Protocol #2003521636).

Data were exported from the REDCap™ (Nashville, TN) database and into Microsoft Excel version 16.0 to be cleaned and exported into WINSTEPS (SWREG, Minnetonka, MN). To determine instrument validity and scale functioning, Rasch analysis was conducted, a psychometric analytic method that analyzed participant responses to the vaccine questionnaire. Several items from the larger vaccine questionnaire which were most conducive to Rasch analysis were selected. Due to the polytomous response structure, we used a rating-scale Rasch model to analyze the data (Cronbach & Meehl, 1955; Downing, 2002; Nunnally & Bernstein, 1994; Wright & Stone, 1979). Each item that was assessed was designed to capture the beliefs about the COVID-19 vaccines held within the Arizona CoVHORT participant population. Initially 15 questions were selected from the vaccine questionnaire which asked questions on a 5-point Likert scale, 12 of which had response items ranging from completely disagree to completely agree. The other 3 questions were excluded because they were measured on a different scale. Misfitting items were removed because they did not fit the model.

Unidimensionality, or the extent to which the scale assesses one construct, was determined using principal component analysis (PCA) of the residuals (De Battisti et al., 2005). Mean squares and z-standardized residual fit statistics to describe model fit, dimensionality, and local independence (Linacre, 2014). According to recommendations from the literature, good model fit was defined as mean square values less than 2.0, and participant responses and questionnaire item scores between 0.7 and 1.3 (Wright, 1994). We also considered logit scores between −2 and +2 as optimal for an assessment tool (Linacre, 1999). In keeping with the assumptions of Rasch analysis, participants and items with poor fit statistics were excluded from the analysis to optimize the performance of the scale in representing beliefs about COVID-19 vaccination in the CoVHORT study population.

To determine the validity of our response scale, participant responses were analyzed to determine the presence of equal representation of each response option and the independence of the scale responses. Using guidelines from previous literature, it was determined that scale performance can be measured based on: the number of observations within each scaled response, mean squared values ranging from 0.6 to 1.4, and category probability curves (Linacre, 1999).The scale data was evaluated based on the following criteria: 10 or more observations per category, regular observation distribution, average measures advance monotonically with category, outfit mean-squares were greater than 2, low MNSQ were identified, the Andrich threshold was examined. Andrich measures were at least 1.4 logits and less than 5 logits (Linacre, 1999).

In WINSTEPS, an overall logit score indicating the participant's location of the unidimensional construct of beliefs and perceptions about the COVID-19 vaccines was computed for each person and item (Smith, 1991). We also sought to identify gaps in item difficulty, questionnaire content, and overall validity of the questionnaire. To determine the presence of significant gaps in item difficulty, content and validity, the z-test was utilized, and identified additional items that needed to be added along the continuum of the latent construct to fully represent beliefs about COVID-19 vaccines among the study sample.

Finally, person and item reliabilities were assessed using the Rasch model. Person reliability indicates the reproducibility of the person's ranking if the same sample responds to another set of items and ranges from 0.5 to 0.914. Item reliability was measured using Cronbach's alpha. To evaluate the reliability and validity of the items selected from the vaccine questionnaire, items were selected that would meet the assumptions of the Rasch model.

Results

As shown in Table 1, with a response rate of 72.4%, a total of 4703 CoVHORT participants completed the questionnaire. Of those who completed the questionnaire, more than half (68.2%) were female, and who had a combined mean age of 48 years. Participants were primarily white (90.2%), highly educated (63.9% with a college degree or above), with most respondents (45.4%) having a household income of more than $75,000 per annum.

Table 1

Demographic characteristics of 4703 CoVHORT vaccine survey participants.

	Mean ± SD
Age	48.5 ± 15.3
	N	%
Vaccination Status
Vaccinated against COVID-19	3720	79.5
Unvaccinated against COVID-19	962	20.5
Gender
Male	1429	30.4
Female	3223	68.6
Non-binary and Transgender	48	1.0
Race
American Indian or Alaska Native Alone	37	0.8
Asian	108	2.3
Black/African American	41	0.9
White	4263	91.0
Mixed Race	166	3.5
Ethnicity
Non-Hispanic	4057	87.1
Hispanic or Latino	564	12.1
Highest Level of Education Completed
Grades 1-8	9	0.2
Grades 9-12	89	2.4
College: 1 year- 3 years	617	16.5
College: 4 years of more	1206	32.2
Post-graduate Education	1815	48.4
Income
Less than $10,000	57	1.5
$10,000 to less than $20,000	76	2.0
$20,000 to less than $25,000	67	1.8
$25,000 to less than $35,000	126	3.4
$35,000 to less than $50,000	319	8.5
$50,000 to less than $75,000	595	15.9
More than $75,000	2143	57.2

Psychometric analysis

Fit indices for the items included in the Rasch model are presented in Table 2. The observed raw variance explained by persons was 18.6% and the expected raw variance was 18.7%. The unexplained variance in the first contrast had an Eigenvalue of 2.21, which is above the recommended threshold of 2. The observed unexplained variance in the 1st contrast was 14.0%, which was below the expected variance of 22.1%. To further investigate one-dimensionality we examined item fit. These results indicated that the most misfitting item was the item measuring social networks (item 2), which had an infit mean square value of 1.72 and an outfit mean square value of 1.75. Both values were slightly above the recommended thresholds for mean square values, but not so high as to warrant additional action. The correlation coefficient for this item was 0.25, indicating a positive correlation with the latent construct. All items had correlation coefficients ranging from 0.25 to 0.75, all above the acceptable level of 0.20. Additionally, none of the items were negatively correlated with the latent construct of beliefs and perceptions of the COVID-19 vaccines. Based on this evidence, we concluded that the assumption of one-dimensionality was met. Items that were reverse coded and other misfitting items were removed because they did not fit the model and 10 of the 15 questions were included in the final analysis.

Table 2

Fit indices and item calibrations of the rasch analyzed modified perceptions and beliefs about COVID-19 vaccines in misfit order.

#	Item Description	Estimate	Model SE	Infit		Outfit
				MNSQ	ZSTD	MNSQ	ZSTD
2	My social network will judge me negatively if I do not get the vaccine	−1.15	0.03	1.72	9.90	1.75	9.90
3	My vaccination status will influence my employment or access to opportunities and resources	−0.82	0.03	1.53	9.90	1.54	9.90
1	I feel getting COVID-19 is less risky than the COVID-19 vaccines	1.70	0.04	1.43	9.90	1.49	9.35
4	I do not think the vaccine will last long enough to be worthwhile	0.21	0.03	.97	−1.58	1.02	1.03
5	I believe the government information on COVID-19 vaccines is not reliable	0.31	0.03	.91	−4.85	.91	−3.99
6	I am concerned about short term side effects of the vaccine	−0.78	0.03	.82	−9.76	.83	−9.20
7	I am worried the vaccines were made too fast to be safe	0.10	0.03	.72	−9.90	.70	−9.90
8	There are other things in the vaccine that will harm me	1.08	0.03	.68	−9.90	.56	−9.90
9	I am worried about possible side effects from getting the vaccine	−0.63	0.03	.64	−9.90	.63	−9.90
10	I am concerned about the safety of the vaccine	−0.02	0.03	.61	−9.90	.58	−9.90

SE: standard error; MNSQ: mean square; ZSTD: z standard.

Based on the results of the Rasch model, we examined the probability curves to determine whether each category had an independent probability of response. The probability curve indicated that while categories 1 (strongly disagree) and 5 (strongly agree) had independent probabilities of participant response, the middle categories - slightly disagree, neither agree nor disagree, and slightly agree-overlapped significantly. As a result, the middle categories were grouped together to optimize scale functioning and to ensure independent probabilities of participant endorsement. This category was called “ambivalent” since participant responses indicated no strong preference to the questions. We first ensured that each new response category had at least 10 observations. With that assumption met, mean square infit and outfit were examined to ensure that each category fell within the recommended thresholds. Each category met this assumption. Finally, we examined the probability curves to ensure that each category had an independent peak. As shown in Fig. 1, the probability curves illustrated independent probabilities for each category, indicating that three-item response categories are optimal for measuring participant beliefs about the COVID-19 vaccines.

Fig. 1

Category probability curve for participants' perception of the COVID-19 vaccines on a 3-point Likert scale. Peaks are separated and nearly similar, indicating satisfactory likelihood of each response option. The curve on the left represents the “Strongly Disagree” category, the middle curve represents the merged “Ambivalent” category, and the curve on the right represents the “Strongly Agree” category.

After refining the items that were included to ensure one-dimensionality and item fit, the person-item map was assessed (Fig. 2). Although this distribution was much wider than it was in the initial analysis, the map indicated that the participants were normally distributed in terms of person-ability. In this case, person-ability indicates the positive or negative perceptions participants have toward COVID-19 vaccines. Additionally, this map also indicated that each item measured a different level of person-ability. Interestingly, the items included in the final analysis were all more positive than the average person's perception toward the vaccines, with a large proportion of the participants having perceptions about the vaccines below the item that was easiest to endorse. This indicates that while participants had favorable attitudes, perceptions, and beliefs about the COVID-19 vaccines, the questions that assessed these perceptions may have been difficult to agree with.

Fig. 2

Person-Item Map for participants' ability to endorse seeking of COVID-19 vaccines. Each “#” represents 43 people. Each “.” represents 1–42 people. “M” represents the mean perception toward vaccines on the left, and mean item difficulty on the right. “S” and “T” represent one and two standard deviations from the mean.

Additionally, the final map shown in Fig. 2 highlighted significant gaps in the items' ability to measure participant perceptions toward the vaccines. These gaps were between the question that asked about social networks and the vaccine's influence on their employment. Gaps were also present between the questions that assessed perceptions about side effects and vaccine safety, questions that assessed belief in governmental recommendations and ingredients in the vaccines, and between the question about vaccine ingredients and the question that asked about the risks associated with receiving the vaccine. These gaps indicate that additional questions that are easier to understand and endorse need to be asked to adequately measure beliefs about the COVID-19 vaccines in this sample.

Discussion

This study assessed the validity and reliability of the Arizona CoVHORT vaccine questionnaire, measuring attitudes, perceptions, and beliefs of the COVID-19 vaccines. To our knowledge, this is the first study evaluating a questionnaire specifically designed to measure attitudes, beliefs, and perceptions about the COVID-19 vaccines. We evaluated data from 4703 CoVHORT participants. Our study found that the questionnaire administered to CoVHORT participants demonstrated high reliability and validity. We also found that some improvements could be made to optimize scale functioning, including reducing response categories from five categories to three, and adding easier questions that matched participant perceptions about vaccines. Additionally, filling in the significant gaps in measuring the latent construct of beliefs and perceptions about the COVID-19 vaccines would strengthen the scale's ability to measure the range of perceptions about vaccines.

In this study, we used the rating scale model to determine participants' attitudes, perceptions, and beliefs about the COVID-19 vaccines. To meet the assumptions of Rasch modeling, we included a subset of the larger CoVHORT vaccine questionnaire; restricting the included questions to those that assessed a similar latent construct and were assessed on a rating scale. Therefore, some items excluded from the analysis may have been represented in the questionnaire which remained unanalyzed in the Rasch model. Additionally, to optimize scale functioning, we removed items that were reverse coded. These items were excluded from analysis as the literature suggests that reverse worded items measure a different dimension, which was consistent with the findings of this study.

The findings of this research are of key importance to further elucidate COVID-19 vaccine hesitancy in the United States, as they demonstrate that the vaccine questionnaire administered to CoVHORT scores well for validity and reliability of key questionnaire items. Instruments that accurately capture vaccine hesitancy are critical as COVID-19 vaccination in the United States has slowed considerably, and solutions for targeted messaging and interventions based on an understanding of why Americans are refusing vaccination are urgently needed. These strategies will vary by population characteristics. For example, Malik et al. found that there are significant disparities in COVID-19 vaccine uptake by race/ethnicity, income, and educational levels (Malik et al., 2020). However, they also found that knowledge about vaccines was positively associated with favorable perceptions about the vaccines (Malik et al., 2020). The theme of COVID-19 vaccine literacy and trust in vaccine development was also a key finding in a study conducted on vaccine uptake in Italy (Palamenghi et al., 2020). These researchers also highlighted that societal trust in the process of biomedical research is central toward improving perceptions about and uptake of COVID-19 vaccines (Palamenghi et al., 2020). The development of a reliable and valid instrument for assessing hesitancy will therefore provide answers as to the best strategies for improving COVID-19 vaccine uptake.

Strengths and limitations

The present study has several strengths, including its large sample size (N = 4703), its association with a cohort study (i.e., Arizona CoVHORT), and the Rasch analysis strongly supporting unidimensionality, indicating that the present questionnaire a strong metric for assessing perceptions and beliefs about the COVID-19 vaccines. However, there are some notable limitations. The study sample was predominantly White, which limited our statistical power to detect differences in perceptions and beliefs about the vaccines by race/ethnicity. The present study also indicated that respondents generally had higher income and higher levels of education than the general population, which may represent selection bias in the Arizona CoVHORT, as those individuals willing to participate in a population-based cohort may present a non-random sampling of the public and thus influence the results of the present study.

Additionally, the person-item map highlights that participants' logit scores were normally distributed and that participants with more favorable perceptions about the COVID-19 vaccines also had a greater ability to endorse the questions that measured their perceptions and beliefs about the vaccines. As such, there were no questions that participants with poorer perceptions about the vaccines could endorse with ease. This disconnect indicates that researchers and public health professionals creating the questionnaire know significantly more about COVID-19 and its prevention than the average participant in the Arizona CoVHORT and have more positive perceptions about the vaccines. Therefore, research is needed to translate scientific knowledge into the public realm in this area especially because increased knowledge about COVID-19 was associated with more favorable attitudes toward the vaccines. Finally, due to the complex nature of the CoVHORT vaccine survey, we were unable to assess the reliability and validity of the entire questionnaire tool in this study. Future psychometric analysis of the questionnaire should test reliability and validity of other questionnaire item dimensions, particularly by focusing on each of the questionnaire's response pathways that might affect the reliability and validity of the questionnaire.

Conclusion

The Arizona CoVHORT vaccine questionnaire demonstrated satisfactory reliability and construct validity in assessing attitudes and beliefs about the COVID-19 vaccines. This study provides insights into the perceptions about the COVID-19 vaccines held by the Arizona CoVHORT population and provides valuable next steps into the ways that public health professionals can continue to promote vaccine uptake moving forward. Our findings show that CoVHORT participants experienced the most difficulty endorsing the question about what would lead to greater risk given the choice between infection with COVID-19 and getting vaccinated against it. This finding provides public health practitioners with next steps in scientific communication by suggesting that the public needs to be better educated on the risk of infection and how that risk compares to that of vaccinating against future infection. This step could prove valuable in minimizing vaccine hesitancy and increasing public confidence in vaccines for COVID-19, as well as other vaccine-preventable diseases.

Our study identified potential areas of improvement of the vaccine questionnaire to optimize scale functioning in future iterations of the ongoing CoVHORT study. These methods can also be used to optimize other questionnaires pertaining to vaccine uptake in the United States, enabling researchers to construct a more representative understanding of perceptions about the COVID-19 vaccines in the United States. Conducting research on the motivations and hesitations related to vaccination is crucial as vaccination is a key tool in achieving herd immunity, decreasing morbidity and mortality, and ending the COVID-19 pandemic. These results provide a starting point for understanding how Arizona residents perceive the COVID-19 vaccines and points to steps that public health professionals can take to address vaccine hesitancy among this population.

Author contributions

Magdiel A. Habila: Conceptualization, Methodology, Software, Formal Analysis, Writing- Original Draft.

Dora Y. Valencia: Conceptualization, Methodology, Software, Writing- Original Draft.

Sana M. Khan: Writing- Original Draft, Writing- Review & Editing.

Kelly M. Heslin: Data Curation, Writing- Review & Editing.

Joshua Hoskinson: Writing- Original Draft, Writing- Review & Editing.

Kacey C. Ernst: Writing- Review & Editing, Resources.

Kristen Pogreba-Brown: Writing- Review & Editing, Resources.

Elizabeth T. Jacobs: Conceptualization, Writing- Review & Editing.

Felina M. Cordova-Marks: Writing- Review & Editing.

Terri Warholak: Conceptualization, Methodology, Validation, Writing- Review & Editing, Supervision.

Financial disclosure

The authors have no financial interests to disclose.

Funding

None.

Declarations of interest

None.

Ethical statement

The Arizona CoVHORT research procedures were reviewed and approved by the University of Arizona Human Subjects Protection Program.

Declaration of competing interest

The authors have no conflicts of interest to disclose.