Social media study of public opinions on potential COVID-19 vaccines: informing dissent, disparities, and dissemination.

Background The current development of vaccines for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is unprecedented. Little is known, however, about the nuanced public opinions on the vaccines on social media. Methods We adopted a human-guided machine learning framework using more than six million tweets from almost two million unique Twitter users to capture public opinions on the vaccines for SARS-CoV-2, classifying them into three groups: pro-vaccine, vaccine-hesitant, and anti-vaccine. After feature inference and opinion mining, 10,945 unique Twitter users were included in the study population. Multinomial logistic regression and counterfactual analysis were conducted. Results Socioeconomically disadvantaged groups were more likely to hold polarized opinions on coronavirus disease 2019 (COVID-19) vaccines, either pro-vaccine ( B = 0.40 , SE = 0.08 , P < 0.001 , OR = 1.49 ; 95 % CI = 1.26 -- 1.75 ) or anti-vaccine ( B = 0.52 , SE = 0.06 , P < 0.001 , OR = 1.69 ; 95 % CI = 1.49 -- 1.91 ). People who have the worst personal pandemic experience were more likely to hold the anti-vaccine opinion ( B = - 0.18 , SE = 0.04 , P < 0.001 , OR = 0.84 ; 95 % CI = 0.77 -- 0.90 ). The United States public is most concerned about the safety, effectiveness, and political issues regarding vaccines for COVID-19, and improving personal pandemic experience increases the vaccine acceptance level. Conclusion Opinion on COVID-19 vaccine uptake varies across people of different characteristics.

Introduction

Researchers suggest that the transmission of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) will quickly rebound if interventions (e.g., quarantine and social distancing) are relaxed [1]. Vaccination has greatly reduced the burden of many infectious diseases throughout history, and developing SARS-CoV-2 vaccines that can be used globally is, therefore, a priority for ending the pandemic [2]. Nevertheless, as scientists and medical experts around the world are developing and testing coronavirus disease 2019 (COVID-19) vaccines, the United States public is now divided over whether or not to obtain COVID-19 vaccines. According to a recent Pew Research Center study (https://www.pewresearch.org/science/2020/09/17/u-s-public-now-divided-over-whether-to-get-covid-19-vaccine/), in May, 2020, 71% of United States adults indicated that they would definitely or probably obtain a vaccine to prevent COVID-19 if it were available. The percentage dropped sharply, however, to 51% in September, 2020. The survey shows that the United States public is concerned about the safety and effectiveness of possible vaccines, and the rapid pace of the approval process.

Previous studies show that the sharing of public concerns about vaccines might lead to delaying or not getting vaccination [3], which could compromise global COVID-19 vaccine distribution strategies. This phenomenon is termed “vaccine hesitancy” [4] which is a complex issue driven by a variety of context-specific factors [5]. Researchers have investigated public opinions on existing vaccines for vaccine-preventable diseases like MMR [6], [7], HPV [8], [9] and H1N1 [10]. Hesitancy and opinions can vary, however, according to the vaccine involved [11].

Lazarus et al. [12] and Feleszko et al. [13] have investigated the potential acceptance of a COVID-19 vaccine using survey methods, yet little is known about the scope and causes of public opinions on COVID-19 vaccines on social media platforms. Although the survey data of a traditional design can lead to detecting causality, it is labor-intensive and expensive [14], thus, being difficult to deploy surveys at a large scale without introducing social desirability biases [15] and in a timely manner, compared to social media data [14]. In addition, due to the passive nature of collecting social media data, observing social media data can potentially capture a different (and unperturbed) view of human behaviors [16]. To the best of our knowledge, there is no other study that has tracked and understood the public opinion regarding COVID-19 vaccines using social media data.

Meanwhile, the development and testing of COVID-19 vaccines has drawn great attention and response on social media platforms like Twitter and Reddit that allow fast sharing of health information [17], [18], [19] and are found to play a major role in disseminating information about vaccinations [20], [21], [22], [23], [24]. Public attitudes towards the vaccines, therefore, can be reflected by analyzing comments and posts in social media [25], [26].

In the current study, we adopt a human-guided machine learning framework based on state-of-the-art transformer language models to capture individual opinions on COVID-19 vaccines, and categorize these opinions into three groups: pro-vaccine, vaccine-hesitant, anti-vaccine. We use more than 40,000 rigorously selected tweets (out of over six million tweets collected using keywords) posted by over 20,000 distinct Twitter users ranging from September to November of 2020. We aggregate the tweets to reflect the state-level and the national attitudes towards COVID-19 vaccines. To characterize the opinion groups, we extract and infer individual-level features such as demographics, social capital, income, religious status, family status, political affiliations, and geo-locations. Lazarus et al. [12] suggested that personal experience such as COVID-19 sickness in the people and their family, and the external perception such as cases and mortality per million of a nation’s population are associated with the vaccine acceptance level. To quantitatively measure and confirm these two effects, we extract the sentiment of personal pandemic experience and non-pandemic experience for each Twitter user. We collect the number of COVID-19 daily confirmed cases from the data repository maintained by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU) to measure the county-level pandemic severity perception. In our study, we hypothesize that:

Hypothesis 1: There will be differences in demographics, social capital, income, religious status, family status, political affiliations and geo-locations among opinion groups.

Hypothesis 2: The personal pandemic experience will have an impact on shaping the attitude towards potential COVID-19 vaccines.

Hypothesis 3: The county-level pandemic severity perception will have an impact on shaping the attitude towards potential COVID-19 vaccines.

We conduct multinomial logistic regression and find that there are differences in demographics, social capital, income, religious status, political affiliations and geo-locations among the opinion groups. People who have the worst personal pandemic experience are more likely to hold anti-vaccine opinion. In addition, people who have the worst pandemic severity perception are more likely to be vaccine-hesitant. We further show that the individual-level features can be used to anticipate whether this person is in favor of the potential COVID-19 vaccines - or not - over time. By incorporating the individual-level features and additional factor indicators, and by conducting counterfactual analyses, we find that the United States public is most concerned about the safety, effectiveness, and political issues with regard to potential vaccines for COVID-19 and improving personal pandemic experience increases the vaccine acceptance level.

Materials and methods

The Methods section is structured as follows. We describe the datasets we use in Methods M1 and how we infer or extract features in Methods M2. We describe our strategy for opinion mining and the standard of labelling in Methods M3. In Methods M4, we discuss the experimental procedures.

M1 DataSets

Twitter

We use the Tweepy API [27] to collect the related tweets which are publicly available. The search keywords and hashtags are COVID-19 vaccine-related or vaccine-related, including “vaccine”, “COVID-19 vaccine”, “COVID vaccine”, “COVID19 vaccine”, “vaccinated”, “immunization”, “covidvaccine”, “#vaccine” and “covid19 vaccine”. It is noteworthy that the capitalization of non-hastag keywords does not matter in the Tweepy query. Slang and misspellings of the related keywords are also included which are composed of “vacinne”, “vacine”, “antivax” and “anti vax”. In the end, 6,314,327 tweets (including retweets) from September 28 to November 4, 2020 posted by 1,874,468 unique Twitter users are collected. To collect as many related text as possible, both COVID-19 vaccine-related and vaccine-related search keywords are used. However, the tweets collected using the vaccine-related search keywords are not necessarily related to COVID-19 vaccines. For example, MMR vaccine-related or HPV vaccine-related tweets might be crawled as well. In addition, the data collection is carried out during the flu shot season, resulting in collecting many influenza shot-related tweets. We apply a keyword-based search in tweets to remove all the tweets containing MMR, autism, HPV, tuberculosis, tetanus, hepatitis B, flu shot or flu vaccine (4.0% removed).

The tweet content and other Twitter profile information are used to extract or predict demographics, user-level features like the number of followers, income, religious status, family status, political affiliations, geo-locations, sentiment about the COVID-19-related experience and non-COVID-related experience. To infer the sentiment, we use Tweepy API to collect the publicly available tweets posted by each user for the last three months. For example, if the tweet containing the search keywords or hashtags was posted on October 1, 2020, then all the publicly available tweets posted by this Twitter user from July 1 to October 1, 2020 are collected as well. It should be noted that only the last 3,200 tweets can be collected per the Tweepy API limitations.

The preprocessing pipeline is shown in Figure 1 . First, the features of the Twitter users are inferred or extracted. To better understand the relationships between all characteristics, we choose to only keep the users of which we can infer all the features except for sentiment. Next, we achieve the mining of opinions via a human-guided machine learning framework. 25,407 unique users with all the features except for the sentiment scores are used to study the temporal and spatial patterns of the opinions. 10,945 of them with sentiment scores are further included in the characterization study and counterfactual analyses.

Figure 1

The diagram of data preprocessing procedures.

Johns Hopkins University CSSE

We extract the number of COVID-19 daily confirmed cases from the data repository maintained by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU) [28]. The median relative change of the number of daily confirmed cases of the last three months at the county level is calculated to measure the county-level pandemic severity perception.

M2 feature inference

Demographics

Following the methods of Lyu et al. [29], we use the Face++ API [30] to infer the gender and age information of the users using their profile images. The invalid image urls and images with multiple or zero faces are excluded. The gender and age information of the remaining users (i.e., there is only one intelligible face in the profile image) is inferred. Since our study focuses on the opinions of United States adults, the users who are younger than 18 are removed. Face++ can achieve a good accuracy of the gender and age inference of Twitter data [31].

User-level features

Seven user-level features are crawled by Tweepy API as well which include the number of followers, friends, listed memberships, favourites, statuses, the number of months since the user account was created, and the verified status. Moreover, we normalize the number of followers, friends, listed memberships, favourites, and statuses by the number of months since the user account was created.

Geo-locations

For Twitter, we choose to resolve the geo-locations using users’ profiles. Similar to Lyu et al. [29], the locations with noise are excluded, and the rest are classified into urban, suburban, or rural.

Income

Following the method of Preoţiuc-Pietro et al. [32], we use a supervised ensemble model to predict the income of Twitter users. The ensemble model includes Gradient Boost Decision Tree (GBDT), Random Forest, Logistic Regression, and XGboost. We use the income datasets of Twitter users [32] to train our model(s). The features include age, days of Twitter history, the number of followers, friends, listed memberships, favourites, and sentiment score calculated by Vader [33]. We categorize income into three classes (low, medium, high) based on the income levels of Kochhar [34] and turn regression problems into classification problems. The accuracy of is .

Religious status

We assign each user a Boolean value for whether she/he is religious based on the tweets and the description in the profile [35].

Family status

By applying regular expression search, we identify users who show evidence that they are either fathers or mothers [35].

Political affiliations

The political attribute is labelled based on whether this Twitter user followed the Twitter accounts of the top political leaders. The incumbent president (Joe Biden. Joe Biden was the presidential candidate when the data were collected.) and the former president (Donald Trump) are included in the analysis. Due to limitation of Twitter API, only about half of Donald Trump’s follower ID was crawled.

Sentiment

In our study, we intend to infer the sentiment of personal pandemic experience and non-pandemic experience. First, we use keyword search methods to classify the three-month historical tweets into COVID-related and non-COVID-related. If a tweet does not contain any of the keywords: “corona”, “covid”, “covid19”, “coronavirus”, “chinese virus”, “china virus”, “wuhan virus”,“wfh”, “work from home”, “pandemic”, “epidemic”, “herd immunity”, “quarantine”, “lockdown”, “mortality”, “morbidity”, “social distancing”, “mask”, “social distance”, “respirator”, “state of emergency”, “ventilator”, “isolation”, “fatality”, “community spread”, “vaccine”, “vaccinated”, “vaccination”, “panic buying”, “hoard”, it is categorized as non-COVID-related. The example tweets are

“ I can not wait to take the last name of my husband! I feel so good to solidify our union by taking his name. I also cringe a little bit at the whole keep the maiden name on social media thing some girls do...I’m more leave-and-cleave type.” and

“ what a distinguished day that was.”

The remaining tweets are categorized as COVID-19-related. The example tweets are

“I am the type of person who does half an hour of meditation and yoga from my peloton app before going to bed to read some chapters of my book and be fast asleep before 11pm. Quarantine changed me.” and

“ Oooorr...I can wear a mask, get on an plane, in a limited space, with NO social distancing, with people from hundreds of different households, ALL going to various destinations, and then take my mask OFF to eat/drink once I’m in my seat ”.

For each Twitter user, the tweets of the two categories are concatenated, respectively. Next, a normalized, weighted composite score is calculated to measure the sentiment of the tweet content using Vader [33]. The score is between 1 (most extreme negative) and +1 (most extreme positive). Vader outperforms individual human raters when assessing the sentiment of tweets [33].

M3 opinion mining

To capture the opinions expressed through text by Twitter users, we adopt a human-guided machine learning framework inspired by Sadilek et al. [36]. The text are classified into four categories: (1) pro-vaccine, (2) vaccine-hesitant, (3) anti-vaccine, and (4) irrelevant.

Tweets might be retweeted for multiple times. We observe that there are 6,703 non-unique tweets in the initial batch of over 90,000 tweets. These non-unique tweets, combined with their retweets constitute 62.9% of all tweets. As a result, the tweets are divided into two groups - the unique-tweet group and the non-unique-tweet group. 430 non-unique tweets which have been retweeted for at least 20 times are included in the non-unique-tweet group. These tweets and their retweets constitute 41.5% of all tweets. The rest are included in the unique-tweet group. All the tweets of the non-unique-tweet group are manually annotated. However, only a subgroup of the unique-tweet group are manually annotated. The state-of-the-art transformer-based language model [37], trained with the subgroup, is used to make estimates of the rest of the unique-tweet group.

Human-guided machine learning framework

We annotate the opinions of the tweets as pro-vaccine, vaccine-hesitant, or anti-vaccine using a human-guided machine learning framework to strike the best balance between automation and accuracy. In total, we stream over six million publicly available tweets from Twitter using Tweepy API between September 28 to November 4, 2020 with search keywords that are vaccine-related or COVID-19 vaccine-related. Unlike Tomeny, Vargo, and El-Toukhy [26], a majority of the tweets crawled with the search keywords in our study is irrelevant to the actual individual opinions about the vaccines for COVID-19, which causes a challenging class imbalance problem that may not only slow down the annotation process but also hinder the performance of automated classifiers [38]. To address this problem, we adopt a human-guided machine learning framework [36] based on the state-of-the-art transformer language model to label the opinions of the tweets. After extracting or inferring the features of these tweets and their authors, we only keep the ones with all the required informative features available.

We initialize the human-guided machine learning framework by sampling 2,000 unique tweets from the corpus with 244,049 tweets. Three researchers independently read each tweet and make a judgement whether this tweet is irrelevant, pro-vaccine, vaccine-hesitant, or anti-vaccine. Table 1 describes the labeling scheme for each opinion category. We label each tweet as one of the categories as long as it matches one of the descriptions of that category. The label of the tweet is assigned with the consensus votes from three researchers. If three researchers vote entirely differently, the senior researcher determines the label of this tweet after discussing with the other two researchers. The Fleiss’ Kappa score of the three researchers is 0.52. The corpus of the initial 2,000 labelled tweets is fed to the XLNet model [37]. The four-class classification model is trained and validated on an external validation set with 400 annotated tweets. The distribution of the four categories is balanced. We then construct another binary classification model that is trained with only two classes of data. The data for and are almost the same except for the label of the output variable. For , one class includes all the irrelevant tweets of the data for and the other includes all the relevant tweets that are pro-vaccine, vaccine-hesitant, or anti-vaccine in the data for . After training, is used to make estimates for a corpus of 4,500 unlabeled tweets sampled from regarding whether they are irrelevant or relevant. 90% of a new batch of corpus is composed of the top 10% of the most likely relevant tweets. The other 10% of the new batch is sampled uniformly at random to increase diversity. This new batch of corpus of 500 tweets is annotated by the three researchers as aforementioned and is added to the corpus . is trained with the updated and validated again. This whole process is considered as one iteration. For each iteration, the three researchers annotate a new batch of corpus of 500 tweets.

Table 1

Labeling scheme for Tweets

Category	Description
Pro-vaccine	i. Claiming that they would take the vaccine once it is available
	ii. Advocating and supporting vaccine/vaccine-associated entities like vaccine experiment trials
	iii. Believing that the vaccine will be the solution to the pandemic
Vaccine-hesitant	i. Claiming that they would like to take the vaccine after the vaccine is proven safe/effective
	ii. Claiming that they would wait for a while and see whether a vaccine is truly safe/effective
	if there is one
	iii. Showing worries about the effectiveness of a rushed vaccine
Anti-vaccine	i. Promoting/arguing in favor of conspiracy theory about vaccine/vaccine-associated entities
	ii. Believing that an effective vaccine would not be invented quickly and help overcome
	the pandemic
	iii. Believing that a covid-19 vaccine is dangerous for whatever reasons and would not take it
	even though the commenters claim that they are not anti-vaccine
Irrelevant	i. Vaccine News. No written opinion from the commenters
	ii. Including vaccine and the commenters opinions, but the focus is something else
	(i.e., insurance, politics, personal life experience, economics, emotional complaints, etc.)
	iii. Comments/questions on vaccines/vaccine-associated entities but with unclear meanings

This framework actively searches for relevant tweets to increase the sizes of the relevant datasets. Figure 2 shows the percentages of the different opinion groups of the original and the final after five iterations. In each iteration, humans guide the machine to learn the irrelevant, pro-vaccine, vaccine-hesitant, and anti-vaccine tweets by updating the training set. Figure 3 shows the performance of of each iteration. As a result, the framework allows us to label the opinions of the tweets and build the model more efficiently.

Figure 2

Distributions of different categories of the original and final training corpora.

Figure 3

Performance of of each iteration.

Tweets preprocessing

We adopt a tweet preprocessing pipeline from Baziotis et al. [39] which can transform the specific text often used in Twitter to special tokens. For example, if the original tweet is

“Scientists develop a COVID vaccine that could initiate a 10-times stronger immune response ”

After preprocessing, the tweet becomes

“scientists develop a covid vaccine that could initiate a - times stronger immune response ”.

Performance of the XLNet model

Table 2 summarizes the performance of the final four-class XLNet model on the external validation set with 400 samples. The final accuracy is 0.63 and the Cohen’s Kappa score is 0.5, which indicates a good agreement.

Table 2

Performance of the four-class XLNet model

Class	Precision	Recall	F1-score
Irrelevant	0.45	0.84	0.59
Pro-vaccine	0.78	0.52	0.62
Vaccine-hesitant	0.77	0.54	0.64
Anti-vaccine	0.79	0.61	0.69
Overall	0.70	0.63	0.63

M4 analysis details

Statistical analysis

To understand what opinion (i.e., pro-vaccine, vaccine-hesitant, and anti-vaccine) the people () would hold based on the their demographics, social capital, income, religious status, family status, political affiliations, geo-location, sentiment about COVID-19-related experience and non-COVID-related experience, and relative change of the number of daily confirmed cases at the county level, we conduct multinomial logistic regression, selecting vaccine-hesitant group as the reference category.

Counterfactual analyses

Following Chang et al. [40], we intend to estimate the impact of communication-related strategies by constructing a hypothetical machine learning model that reflects the expected effect. To assess the potential outcomes of the communication-related strategies, we build the machine learning model using the real data, and apply the constructed model to the hypothetical data.

The data ranging from September 28 to October 21, 2020 are used to train a support vector machine (SVM) which makes predictions about the opinion group of the data of the latest two weeks (October 22 - November 4, 2020). The real percentage of pro-vaccine users and the prediction percentage are plotted in Figure 4 . The real percentage falls within in one standard deviation of the predicted percentage, indicating a good simulation performance.

Figure 4

Counterfactual analyses illustrate the importance of politics, safety and effectiveness factor indicators, and personal pandemic experience.

We further analyze the relationship between the opinions and the topics of the tweets using the Latent dirichlet allocation (LDA) topic modelling [41] with 10 topics as shown in Figure 5 . The coherence score is 0.31. In the word cloud of each topic, top 30 keywords are plotted. As we can see from the figure, people are most concerned about the safety and effectiveness of the vaccine which is consistent with the Pew Research Center survey (https://www.pewresearch.org/science/2020/09/17/u-s-public-now-divided-over-whether-to-get-covid-19-vaccine/). Some politics-related keywords like “administration”, “white house”, and the names of political figures like “Trump” and “Kamala” are presented as well. To label the factor indicators, we narrow down the 10 topics to two major ones: “safety and effectiveness” and “politics”, and use keyword search methods. The keywords for the safety and effectiveness include “safe”, “effective”, and “efficacy”. The keywords for the politics include “administration”, “politics”, “politician”, “political” and the names of Donald Trump, Mike Pence, Joe Biden and Kamala Harris. Each tweet is labelled 1 if it contains the related keywords, and 0 if it does not. Table 3 shows the descriptive statistics of these two variables. The basic settings for the counterfactual classifiers are the same as . We analyze one factor at a time. We train the classifier with the basic variables and the factor indicator with the real value. The basic variables include user demographics, Twitter usage patterns, sentiment of the pandemic and non-pandemic experience, income, religious status, family status, political affiliation, as well as the population density. The prediction is plotted in orange in Figure 4. Then we change the value of the factor indicator which was originally 1 into 0, keeping other variables constant. The trained classifier is applied to the hypothetical data, and the prediction is plotted in green.

Figure 5

10 topics extracted from the tweets with the top 30 keywords.

Table 3

Descriptive statistics of the factor indicators

Variables	Number	Mean	SD	Min	Max
Politics	10,945	0.2512	0.4337	0	1
Safety and effectiveness	10,945	0.1801	0.3843	0	1

Results

Characterization of different opinion groups

The proportions of the different opinion groups of the United States public change over time as shown in Figure 6 a, which roughly correspond to the major pandemic-related events. Figure 6b shows the number of Twitter users. Overall, 57.65% (6,218 of 25,407) are pro-vaccine, 19.30% (2,469 of 25,407) are vaccine-hesitant, and the rest are anti-vaccine. By aggregating people at the state level, we estimate the opinions about the potential COVID-19 vaccines of each state as shown in Figure 7 . The Southeast of the United States shows a relatively lower acceptance level, so does the cluster of Ohio, Indiana and Kentucky.

Figure 6

(a) The proportions of the opinion groups from September 28 to November 4, 2020. (b) Number of Twitter users from September 28 to November 4, 2020. The data of October 5, 2020 are missing due to a data collection issue.

Figure 7

State-level public opinions about potential COVID-19 vaccines. The numbers in parentheses stand for the sizes of the study populations.

After performing the Granger Causality Test with a one-day lag, we find that (Figure 8 ), in Nevada, Tennessee and Washington, the percentage of the pro-vaccine people deviates the most from the national average (). The percentage of the pro-vaccine group of Washington is above the national average during the most of the time, while the acceptance level of Nevada is relatively lower than the national average. More drastic changes are observed for the acceptance level of Tennessee.

Figure 8

The percentages of the pro-vaccine groups of the national average (US), Nevada (NV), Tennessee (TN), and Washington (WA).

Descriptive statistics and bi-variate correlations of the variables of the multinomial logistic regression are shown in Table 4 . Table 5 summarizes the results of the multinomial logistic regression. The Chi-square test shows that the variables significantly predict the opinion on potential COVID-19 vaccines: , McFadden’s pseudo , which supports our hypotheses. Next, we show the predictive effects of these variables with paired comparisons.

Table 4

Descriptive statistics and the bi-variate correlations

Variables	Mean	SD	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19
1. Gender (0 = male, 1 = female)	0.46	0.50
2. Age (years)	39.89	14.69	−.08^**
3. Verified (0 = no, 1 = yes)	0.04	0.20	−.00	.03^**
4. Twitter history (months)	91.30	43.47	−.02	.03^**	.09^**
5. # Followers	1.60	1.63	.04^**	−.01	.37^**	−.08^**
6. # Friends	1.95	1.25	.04^**	−.00	.09^**	−.29^**	.68^**
7. # Listed memberships	−1.62	0.93	−.00	.08^**	.49^**	.22^**	.69^**	.31^**
8. # Favorites	4.17	1.95	.12^**	−.14^**	−.03^**	−.21^**	.38^**	.47^**	.05^**
9. # Statuses	4.09	1.43	.02^*	−.11^**	.07^**	−.09^**	.53^**	.45^**	.29^**	.58^**
10. Higher-Income (0 = no, 1 = yes)	0.00	0.05	−.00	−.03^**	.03^**	−.01	.05^**	.02^*	.04^**	.02^**	.01
11. Lower-Income (0 = no, 1 = yes)	0.76	0.43	−.00	−.43^**	−.04^**	.05^**	−.15^**	−.18^**	−.12^**	−.13^**	−.09^**	−.10^**
12. Religious (0 = no, 1 = yes)	0.04	0.19	.01	.07^**	−.03^**	−.03^**	.03^**	.07^**	−.03^**	.02	.02	−.00	−.06^**
13. Having kids (0 = no, 1= yes)	0.12	0.32	.09^**	.09^**	.03^**	.02^*	.04^**	.05^**	.04^**	.01	−.05^**	−.01	−.05^**	.09^**
14. Following Trump (0 = no, 1 = yes)	0.11	0.31	−.05^**	.06^**	−.03^**	−.17^**	−.05^**	.04^**	−.12^**	−.01	−.05^**	−.00	−.05^**	.06^**	.04^**
15. Following Biden (0 = no, 1 = yes)	0.17	0.38	.09^**	.07^**	.06^**	.10^**	.05^**	.19^**	.06^**	.09^**	.03^**	.00	−.06^**	−.03^**	.07^**	.01
16. Rural (0 = no, 1 = yes)	0.19	0.40	−.02	.09^**	−.05^**	−.04^**	−.07^**	−.02	−.08^**	−.01	−.04^**	.00	−.05^**	.06^**	.04^**	.05^**	−.00
17. Suburban (0 = no, 1 = yes)	0.14	0.35	−.01	.07^**	−.01	−.02^*	−.05^**	−.03^**	−.05^**	−.03^**	−.02^**	−.02	−.01	.03^**	.04^**	.03^**	−.00	−.20^**
18. Pandemic experience (sentiment)	0.06	0.80	.03^**	−.04^**	.10^**	.03^**	.09^**	−.00	.15^**	−.08^**	−.08^**	.02^*	.04^**	−.03^**	−.00	−.06^**	.01	−.03^**	.00
19. Non-pandemic experience (sentiment)	0.62	0.75	.07^**	−.06^**	.07^**	.04^**	.14^**	.09^**	.14^**	.06^**	.01	.02	.02	.00	.03^**	−.06^**	.01	−.03^**	−.02^*	.27^**
20. Pandemic severity perception (relative change of # daily confirmed cases)	0.01	0.00	−.01	.03^**	-.03^**	−.02^*	−.05^*	−.01	−.07^**	−.00	−.04^**	.01	−.03^**	.06^**	.05^**	.04^**	−.01	.27^**	.12^**	−.01	.01

Note: * . ** . # The numbers of followers, friends, listed memberships, favorites, statuses are normalized by the months of Twitter history and log-transformed.

Table 5

Multinomial logistic regression outputs for the opinion on potential COVID-19 vaccines against demographics and other variables of interest

Predictor	Anti-vaccine			Pro-vaccine
	B	SE	OR (95% CI)	B	SE	OR (95% CI)
Intercept	−1.82^***	0.26		0.79^***	0.20
Age (years)	0.00	0.00	1.00 (1.00, 1.01)	0.01^***	0.00	1.01 (1.01, 1.02)
Twitter history (months)	0.00	0.00	1.00 (1.00, 1.00)	−0.003^***	0.001	0.997 (0.996, 0.999)
# Followers	0.28^***	0.04	1.32 (1.22, 1.42)	0.08^**	0.03	1.09 (1.02, 1.16)
# Friends	−0.18^***	0.04	0.83 (0.77, 0.90)	−0.07^*	0.03	0.93 (0.88, 0.99)
# Listed memberships	−0.63^***	0.06	0.53 (0.47, 0.60)	0.10^*	0.05	1.10 (1.01, 1.20)
# Favorites	0.04^*	0.03	1.12 (1.06, 1.19)	0.04^*	0.02	1.04 (1.01, 1.08)
# Statuses	0.11^***	0.03	1.12 (1.06, 1.19)	−0.06^*	0.02	0.94 (0.90, 0.99)
Pandemic experience (sentiment)	−0.18^***	0.04	0.84 (0.77, 0.90)	0.21^***	0.03	1.24 (1.16, 1.32)
Non-pandemic experience (sentiment)	−0.04	0.04	0.96 (0.89, 1.04)	0.13^***	0.03	1.14 (1.07, 1.22)
Pandemic severity perception (relative change of # daily confirmed cases)	−14.99	8.13	0.00 (0.00, 2.58)	−22.68^***	6.59	0.00 (0.00, 0.00)
Female	−0.25^***	0.06	0.78 (0.69, 0.88)	−0.47^***	0.05	0.63 (0.57, 0.69)
Verified user	−0.61^*	0.27	0.54 (0.32, 0.91)	−0.16	0.14	0.85 (0.65, 1.12)
Higher-income	−170.67	5.00e+36	0.00 (0.00, Inf)	0.47	0.43	1.60 (0.70, 3.68)
Lower-income	0.40^***	0.08	1.49 (1.26, 1.75)	0.52^***	0.06	1.69 (1.49, 1.91)
Religious	0.74^***	0.17	2.10 (1.52, 2.91)	0.37^*	0.15	1.45 (1.07, 1.95)
Having kids	−0.11	0.10	0.90 (0.74, 1.09)	0.00	0.08	1.00 (0.86, 1.15)
Following Trump	0.41^***	0.10	1.51 (1.26, 1.83)	0.06	0.08	1.06 (0.90, 1.25)
Following Biden	−1.22^***	0.10	0.29 (0.24, 0.36)	−0.34^***	0.06	0.71 (0.63, 0.80)
Rural	0.17^*	0.08	1.19 (1.01, 1.39)	0.07	0.07	1.07 (0.94, 1.22)
Suburban	0.18^*	0.09	1.20 (1.01, 1.43)	0.11	0.07	1.12 (0.97, 1.29)
Chi-square	1,340.94^***
df	40
−2 log likelihood	20,171.90
McFadden’s pseudo R^2	0.06
Sample size	10,945

Note: * . ** . *** . The vaccine hesitant group is selected as the reference category. # The numbers of followers, friends, listed memberships, favorites, statuses are normalized by the months of Twitter history and log-transformed.

Women are more likely to hold hesitant opinions

Gender is statistically significant (). Women are likely to hold hesitant opinions rather than polarized opinions (i.e., pro-vaccine, anti-vaccine). Specifically, comparing the anti-vaccine group and vaccine-hesitant group, we find that women are less likely to be anti-vaccine (). Comparing the pro-vaccine group and vaccine-hesitant group, we find that women are also less likely to be pro-vaccine ().

Older people tend to be pro-vaccine

Age is statistically significant (). Comparing the anti-vaccine group and vaccine-hesitant group, we do not find significant evidence that older people are more anti-vaccine. However, comparing the pro-vaccine group and vaccine-hesitant group, we find that people who are one year older are 1.01 () times more likely to be pro-vaccine instead of vaccine-hesitant, which echoes the findings of Lazarus et al. [12]. One potential explanation is that the risk of dying with COVID-19 increases with age [42], and the benefits of not getting infected with COVID-19 outweigh the risk of getting vaccinated.

Different patterns of Twitter usage

A verified Twitter account must represent or other wise be associated with a prominently recognized individual or brand [43]. In our study, verified status is statistically significant (). Comparing the anti-vaccine group and vaccine-hesitant group, we find verified users are less likely to be anti-vaccine (), however, comparing the pro-vaccine group and vaccine-hesitant group, we do not find significant differences.

Months of Twitter history is statistically significant (). Comparing the anti-vaccine group and vaccine-hesitant group, we do not find significant differences, however, comparing the pro-vaccine group and vaccine-hesitant group, we find if the months of Twitter history were to increase by one month, it is 0.997 () less likely to be pro-vaccine than vaccine-hesitant.

After normalizing the number of followers, friends, listed memberships, favorites, and statuses with the number of months of Twitter history, we still find that the social capital is statistically significant. Specifically, there are significant differences in terms of followers counts (), friends counts (), listed memberships counts (), favorites counts (), statuses counts ().

Comparing the anti-vaccine group and vaccine-hesitant group, if the log-scale followers count were to increase by one unit, it is 1.32 () times more likely to be anti-vaccine. If the log-scale friends count were to increase by one unit, it is less likely to be anti-vaccine (). If the log-scale listed memberships count were to increase by one unit, it is less likely to be anti-vaccine (). If the log-scale favorites count were to increase by one unit, it is 1.04 () times more likely to be anti-vaccine. If the log-scale statuses count were to increase by one unit, it is 1.12 () times more likely to be anti-vaccine.

Comparing the pro-vaccine group and vaccine-hesitant group, if the log-scale followers count were to increase by one unit, it is 1.09 () times more likely to be pro-vaccine. If the log-scale friends count were to increase by one unit, it is less likely to be pro-vaccine (). If the log-scale listed memberships count were to increase by one unit, it is 1.11 () times more likely to be pro-vaccine. If the log-scale favorites count were to increase by one unit, it is 1.04 () times more likely to be pro-vaccine. If the log-scale statuses count were to increase by one unit, it is less likely to be pro-vaccine ().

Twitter users who have more followers or fewer friends, or give more favourites are more likely to hold polarized opinion. The larger listed memberships count is, the more likely the Twitter user is pro-vaccine. Twitter users who post more statuses tend to be anti-vaccine.

The lower-income group is more likely to hold polarized opinions

Income is statistically significant (). Comparing the anti-vaccine group and vaccine-hesitant group, we find that the lower-income group is 1.49 () times more likely to be anti-vaccine than the medium-income group, however, the difference between the higher-income group and medium-income group is not significant. Comparing the pro-vaccine group and vaccine-hesitant group, we find that lower-income group is 1.69 () times more likely to be pro-vaccine than medium-income group. The difference between the higher-income group and medium-income group is not significant. Inconsistent with Lazarus et al. [12] that the higher the income is, the more likely people are pro-vaccine, we find the effect of income more nuanced. Lower-income people tend to be polarized.

Religious people are more likely to be polarized

Religious status is statistically significant (). Comparing the anti-vaccine group and vaccine-hesitant group, we find that religious people are more likely to be anti-vaccine than non-religious people (). Comparing the pro-vaccine group and vaccine-hesitant group, we find that religious people are also more likely to be pro-vaccine than non-religious people (). This is in line with Larson et al. [5] that the effect of religious status is complicated.

Political diversion indicates a divided opinion about the potential COVID-19 vaccines

Following Donald Trump is statistically significant (). Comparing the anti-vaccine group and vaccine-hesitant group, we find that the Twitter users who follow Donald Trump are 1.51 () times more like to be anti-vaccine than the Twitter users who do not. Comparing the pro-vaccine group and vaccine-hesitant group, following Donald Trump is not significant.

Following Joe Biden is statistically significant (). Comparing the anti-vaccine group and vaccine-hesitant group, we find that the Twitter users who follow Joe Biden are less like to be anti-vaccine than the Twitter users who do not (). Comparing the pro-vaccine group and vaccine-hesitant group, we find that the Twitter users who follow Joe Biden are also less likely to be pro-vaccine than the Twitter users who do not ().

Twitter users who follow Donald Trump tend to be anti-vaccine, while those who follow Joe Biden tend to be vaccine-hesitant.

People living in suburban or rural areas are more likely to be anti-vaccine

Although the population density of the area is not statistically significant across three opinion categories, we still find differences between the anti-vaccine group and vaccine-hesitant group. People living in suburban areas are 1.20 () times more likely to be anti-vaccine than people living in urban areas. People living in rural areas are 1.20 () times more likely to be anti-vaccine than people living in urban areas.

Most of the results are consistent with Hypothesis 1. There are significant differences in demographics, social capital, income, religious status, political affiliations and geo-locations among opinion groups, however, we do not find significant difference in family status.

Personal experience with COVID-19 and the county-level pandemic severity perception shape the opinion

The sentiment score of personal experience with COVID-19 is statistically significant (). Comparing the anti-vaccine group and vaccine-hesitant group, we find that if the sentiment score of personal experience with COVID-19 were to increase by one unit (i.e., the sentiment became more positive), the person would be less likely to hold anti-vaccine opinion (). Comparing the pro-vaccine group and vaccine-hesitant group, we find if the sentiment score of personal experience with COVID-19 were to increase by one unit (i.e., the sentiment became more positive), the person would be 1.24 times more likely to hold pro-vaccine opinion (), which is consistent with Hypothesis 2.

The sentiment score of non-COVID-related personal experience is overall statistically significant (), but comparing the anti-vaccine group and vaccine-hesitant group, we find no significant difference. However, comparing the pro-vaccine group and vaccine-hesitant group, we find if the sentiment score of non-COVID-related personal experience were to increase by one unit (i.e., the sentiment became more positive), the person would be more likely to hold pro-vaccine opinion ().

The county-level pandemic severity perceptions are overall statistically significant (), supporting Hypothesis 3, but we find no significant difference comparing the anti-vaccine group and vaccine-hesitant group. However, comparing the pro-vaccine group and vaccine-hesitant group, if the relative change of the number of daily confirmed cases at the county level were to increased by one unit, the person would be less likely to be pro-vaccine ().

At the individual level, the personal pandemic experience is a strong predictor of the opinion about COVID-19 vaccines. People who have the worst personal pandemic experience are more likely to hold anti-vaccine opinion. However, the non-pandemic experience is not a strong predictor of anti-vaccine opinion. At the county level, people who have the worst pandemic severity perception (i.e., the relative change of the number of daily confirmed cases is the largest) are more likely to be vaccine-hesitant.

Counterfactual analyses

Removing the safety and effectiveness factors reduces the vaccine acceptance level. However, removing the politics factor increases it

Figure 4 shows the results of counterfactual analyses of factor indicators. Using counterfactual analysis by turning the factor indicator of safety and effectiveness into 0, there is a clear decrease (4.42% on average) of the percentage of the pro-vaccine people. However, by turning the factor indicator of politics into 0, there is a clear increase (22.65% on average) of the percentage of the pro-vaccine people. This indicates that people are most concerned about the relationship between the politics and the potential COVID-19 vaccines, which is also mirrored by the news report [44].

Improving personal pandemic experience increases the vaccine acceptance level

Figure 4 shows the results of counterfactual analyses of different sentiment levels of personal pandemic experience. By increasing the sentiment scores with a factor of 50%, the percentage of the pro-vaccine people increases by 6.39%. However, by reducing the sentiment scores of a factor of 50%, the percentage of the pro-vaccine people decreases by 2.82%.

Robustness verification

The multinomial logistic regression and counterfactual analyses are conducted based on the opinion derived from the social media data. By adopting the human-guided machine learning framework, the dataset is composed of human-annotated data and machine-inferred data. Although the Cohen’s Kappa score of the machine and human annotators reaches 0.5 after five iterations, the results of the analyses could potentially change given the seemingly limited prediction performance. Therefore, we additionally verify the robustness of the findings of multinomial logistic regression and counterfactual analyses by examining two different combinations of the human-annotated data and machine-inferred data:

Only human-annotated data (n = 2,939) are used to conduct the multinomial logistic regression and counterfactual analyses

Human-annotated data (n = 2,939) and 50% machine-inferred data (randomly sampled, n = 4,003) are used to conduct the multinomial logistic regression and counterfactual analyses

The results of the two robustness verification experiments are shown in Appendices. Both of them are basically consistent with the reported results in Table 5 and Figure 4. The findings do not fundamentally change, confirming that the opinion computationally derived from the social media data are reliable for our study.

Discussion

We conduct multinomial logistic regression to investigate the scope and causes of public opinions on vaccines and test three hypotheses. The current study shows the hypothesized effects of most of the characteristics in predicting the odds of being pro-vaccine or anti-vaccine against vaccine-hesitant. The findings suggest that women are more vaccine-hesitant, which is consistent with the Reuters/Ipsos survey (https://www.reuters.com/business/healthcare-pharmaceuticals/more-women-than-men-us-nervous-about-fast-rollout-covid-vaccine-thats-problem-2020-12-11/), and older people tend to be pro-vaccine. With respect to social capital, people who have more followers or fewer friends, or give more favorites, are more likely to hold polarized opinions. Verified status, months of Twitter history, listed memberships counts and statuses counts are statistically significant as well. We also show that the lower-income group is more likely to hold polarized opinions. This is inconsistent with the finding by Lazarus et al. [12]. Moreover, religious people tend to hold polarized opinions. As for political affiliations, Twitter users who follow Donald Trump are more likely to be anti-vaccine rather than vaccine-hesitant, while those who follow Joe Biden tend to be vaccine-hesitant rather than anti-vaccine or pro-vaccine. In addition, we find people who live in rural or suburban areas tend to be anti-vaccine. However, we do not find the hypothesized predictive effect of family status on the opinion about vaccines.

Furthermore, the current study shows the hypothesized predictive effects of the personal pandemic experience and the county-level pandemic severity perception. In particular, personal experience with COVID-19 is a strong predictor of anti-vaccine opinion. The more negative the experience is, the more negative the opinion on vaccines is. People are more likely to be vaccine-hesitant if their pandemic severity perceptions are worse.

Our current study has limitations. The public opinions of some (less populated) states cannot be reflected due to the inadequate data. The findings could be further validated in other populations. In addition, the performance of the XLNet model of this study could be improved so that the proposed human-guided machine learning framework could identify the public opinions on the COVID-19 vaccines more accurately. Specifically, future work could address the class imbalance issue by annotating more tweets or augmenting the training data via back translation (i.e. translating English tweets to another language and then translating them back to English). Despite the limitations, our study broadly captures the public opinions on the potential vaccines for COVID-19 on Twitter. The final F1-score of our study is 0.63, indicating a good performance, which is achieved via the human-guided machine learning framework that is composed of both the state-of-the-art transformer-based language model and the large annotated dataset. By aggregating the opinions, we find a lower acceptance level in the Southeast part of the United States. The changes of the proportions of different opinion groups correspond roughly to the major pandemic-related events. We show the hypothesized predictive effects of the characteristics of the people in predicting pro-vaccine, vaccine-hesitant, and anti-vaccine group. Using counterfactual analyses, we find that people are most concerned about the safety, effectiveness and politics regarding potential COVID-19 vaccines, and improving personal experience with COVID-19 increases the vaccine acceptance level.

Our results can guide and support policymakers making more effective distribution policies and strategies. First, more efforts of dissemination should be spent on the socioeconomically disadvantaged groups who are exposed to potentially higher risks [40], [45], [46] and already possess more polarized attitudes towards the vaccines. Second, messaging for the vaccines is extremely important because the vaccine acceptance level can be increased by removing the politics factor. Third, safety and effectiveness issues need to be well addressed because the acceptance level is reduced by removing this factor. Finally, improving personal pandemic experience may increase the vaccine acceptance level as well and thus all helpful measures should be integrated to maximize the vaccine acceptance. In the future, by combining social media data and more traditional survey data, we hope to acquire deeper insights into the public opinions on potential COVID-19 vaccines and thus inform more effective vaccine dissemination policies and strategies.

Conflicts of interest statement

The authors declare that there are no conflicts of interest.

Funding

This research was supported in part by a University of Rochester Research Award, and National Institutes of Health (Grant No. RF1AG063811-01S2).

Author contributions

Hanjia Lyu, Timothy D. Dye and Jiebo Luo conceived and designed the analysis. Hanjia Lyu collected the data. Hanjia Lyu, Junda Wang, Viet Duong contributed to data analysis. All authors wrote the paper. Hanjia Lyu and Jiebo Luo made critical revision.