Analyzing Iranian opinions toward COVID-19 vaccination.

Objectives: The aim of this study was to assess Iranian tweets in order to: (1) analyze Iranian views toward COVID-19-vaccination; (2) compare Iranian views toward homegrown and imported COVID-19-vaccines; (3) present an effective model for sentiment analysis tasks regarding critical issues such as COVID-19-vaccination. Design and methods: Persian tweets mentioning homegrown and imported vaccines were retrieved between April 1 and and September 30, 2021. The sentiments of retrieved tweets were identified using a deep-learning sentiment-analysis model. A sarcasm detection model, based on a random forest classifier, was used to identify sarcastic tweets and thus minimize misclassification. Finally, Iranian views toward COVID-19 vaccination were investigated.
Results: Subtle differences were found in the number of positive sentiments toward homegrown and imported vaccines, with the latter having dominant positive polarity. Negative sentiments regarding homegrown and imported vaccines increased in some months. No significant differences were observed between the percentages of overall positive and negative opinions toward vaccination.
Conclusion: It is worrisome that negative sentiments toward homegrown and imported vaccines increased in some months in Iran. Health organizations can focus on Twitter in order to promote positive messaging toward COVID-19 vaccination. Sarcasm detection enabled the identification of tweets that ironically stated positive sentiments toward vaccination, thus improving the accuracy of the sentiment analysis results. Our sentiment analysis-sarcasm detection model is a reliable tool for mitigating classification problems.

Introduction

COVID-19 is an infectious disease caused by the SARS-CoV-2 virus (World Health Organization, 2020). The development of vaccines against COVID-19 has been a global aim since the World Health Organization declared the pandemic (Marcec and Likic, 2021). Attaining a level of herd immunity by vaccination remains complicated because public opinions toward vaccines can change based on different events, and even vary between different COVID-19 vaccines (Chen Lyu et al., 2021).

Since Iran's government has aimed to end the pandemic by effective vaccination, it has supported Iranian scientists in developing COVIran Barekat as a homegrown vaccine (Abdoli et al., 2021). Moreover, several imported vaccines are currently being used in Iran, including Sputnik Light, Janssen, Pfizer/BioNTech, AstraZeneca/Oxford, Moderna, and Sinopharm (McGill University, 2021). Comparing homegrown and imported vaccines is a hot topic in Iran, with Iranians engaged in comparing vaccines in order to decide which one they should get. They also use social media, such as Twitter, to express their views online, with COVID-19 being a popular subject since January 2020 (Sattar and Arifuzzaman, 2021). Hence, Twitter allows health organizations to track public perceptions of COVID-19 vaccination, thus helping them to develop plans to increase the uptake of COVID-19 vaccines and end the pandemic. On this basis, our study assessed Iranian tweets to: (1) analyze Iranian views toward COVID-19 vaccination; (2) compare Iranian views toward homegrown and imported COVID-19 vaccines; and (3) present an effective model for sentiment analysis (SA) tasks on critical issues such as vaccination.

To identify public opinions toward COVID-19 vaccination, it was first necessary to assign polarities of ‘positive’, ‘negative’, and ‘neutral’ to each retrieved tweet, using a pre-trained Persian SA model. Deep-learning classifiers based on CNN-LSTM (Convolutional Neural Network — Long Short-Term Memory) hybrid model were used for sentiment classification, which had demonstrated high accuracy in previous works (Bokaee Nezhad and Deihimi, 2019). Sarcasm in tweets can cause unreliable determination of sentiments; hence, detecting it can improve SA results (Schifanella et al., 2016). Due to the nature of their language, Persian speakers often use sarcasm, with Twitter involving many sarcastic tweets (Golazizian, et al., 2020). In our study, sarcasm detection enabled the identification of several tweets that ironically stated a positive sentiment toward vaccination. Thus, in order to increase the accuracy of the SA model and minimize potential misclassifications, a pre-trained sarcasm detection model was used to modify each tweet's sentiment label. A random forest classifier was used for sarcasm detection. Subsequently, the first vaccine-related dataset in Persian for analyzing Iranian opinion toward COVID-19 vaccination was created.

To the best of our knowledge, our study is the first attempt to analyze public concerns regarding COVID-19 vaccines in Iran. Fig. 1 shows the workflow for the suggested methodology.

Fig. 1

Flowchart of the proposed model.

Materials and methods

Data acquisition

Fig. 1 provides an overview of the proposed model architecture. A Python programming language library called ‘Tweepy’ was connected to the Twitter academic API in order to collect related tweets. Separate searches were conducted on Twitter regarding imported and homegrown vaccines. The search keywords for the homegrown vaccine were "واکسن برکت، واکسن داخلی" (Barekat vaccine, homegrown vaccine). Accordingly, the keywords for retrieving tweets on foreign vaccines were واکسن آسترازنکا، واکسن آکسفورد، واکسن فایزر، واکسن مدرنا، واکسن سینوفارم، واکسن خارجی"" (AstraZeneca vaccine, Oxford vaccine, Pfizer vaccine, Moderna vaccine, Sinopharm, foreign vaccine). Subsequently, all Persian tweets relating to our keywords, posted between April 1 and September 30, 2021, were retrieved.

Data preprocessing

In this step, non-Persian tweets, URLs, retweets, mentions, and some special characters such as ‘^ % # - +’ were removed from the retrieved tweets using the ‘re’ Python module. This stage also included two sub-processes: (1) Conversion of emoticons to their equivalent words, using the Emoji Dictionary proposed by Bokaee Nezhad and Deihimi (2020). (2) Conversion of Persian slang and proverbs to their direct meanings using the Proverb Dictionary proposed by Bokaee Nezhad and Deihimi (2020), which can also help classifiers detect sarcasm with more accuracy (Bouazizi and Ohtsuki, 2016). This step was performed manually by five Persian linguists, who checked all retrieved tweets and converted each slang phrase or proverb to its related direct meaning. Next, the Persian text preprocessing toolkit Hazm was applied; this performs various tasks, including normalization, space correction, tokenization, stemming, and tagging of different elements of speech.

Persian datasets

After preprocessing, the results of the imported vaccine searches were combined to develop our first dataset: Foreign-Vaccine. Similarly, our second dataset, Homegrown-Vaccine, was then created. Of the 803 278 tweets collected from Twitter, 400 839 (49.9%) were allocated to the Homegrown-Vaccine dataset and 402 439 (50.09%) to the Foreign-Vaccine dataset.

Vaccine sentiment analysis

A pre-trained hybrid deep-learning model was used to assign three polarity scores (‘positive’, ‘negative’, and ‘neutral’) to each tweet. The model, trained on a Persian database, consisted of 11 616 tweets. It was based on CNN-LSTM architecture and proved the effectiveness of using deep-learning classifiers on Persian datasets. Within this architecture, CNN was used as a feature extractor for LSTM on textual input data. The proposed model also used Word2vec for word embedding.

Using the model, tweets in both datasets were allocated positive (+1), negative (−1), and neutral (0) labels. Fig. 2 illustrates the percentage of tweets in each sentiment class for both datasets. As shown, before using the sarcasm detection model, positive sentiments towards foreign vaccines accounted for 46% of tweets (n = 185 121), followed by negative sentiments (42%; n = 169 024) and neutral sentiments (12%; n = 48 292). At this stage, there was a slight difference between positive and negative sentiment scores toward foreign vaccines. With regard to the homegrown vaccine, positive sentiments accounted for 44% of tweets (n = 176 369), followed by negative sentiments (36%; n = 144 302) and neutral sentiments (20%; n = 80 167). As shown, the prevalence of neutral views on the homegrown vaccine was 8% higher than the same sentiment for foreign vaccines.

Fig. 2

Percentages of sentiment polarities toward imported and homegrown vaccines before using the sarcasm detection model.

Vaccine sarcasm detection

Sarcasm can make it challenging to identify the sentiment of a tweet correctly. Hence, our study needed to consider the possibility of sarcasm in tweets. Sarcasm detection is a significant step for SA, and enabled us to detect several tweets that ironically stated a positive sentiment toward vaccination. A pre-trained machine-learning model for Persian sarcasm detection proposed by Bokaee Nezhad and Deihimi (2019) was used to detect sarcastic tweets. The model was trained on 10 023 Persian tweets. Based on the model, three sets of features were extracted to cover different types of Persian sarcasm: (1) Deep-Polarity-Feature, (2) POS-Feature, and (3) Punctuation-Feature. Consequently, all the retrieved tweets could be represented by feature vectors. Each feature is explained below.

Deep-Polarity-Feature

This feature focused on sentence-level inconsistency. For example, in the retrieved tweet

“من واقعا افتخار می کنم که واکسن داخلی داریم حتما میخوان مجبورمون کنند از این *** بزنیم!” (I'm so proud to have a homegrown vaccine! They will force us to get this F** damn without any doubt!). There is a sentence-level inconsistency between the first and second parts of the tweet: ‘I'm so proud to have a homegrown vaccine!’ being a positive sentiment, and ‘they will force us to get this *** without any doubt!’ being a negative sentiment). Hence, the sarcasm detection model considered each tweet with more than 12 tokens as a Multiple-Sentence-Tweet. Each Multiple-Sentence-Tweet was divided into two parts. Thus, for this example, we have:

Part 1: I'm so proud to have a homegrown vaccine!

Part 2: They will force us to get this *** without any doubt!

As mentioned previously, if there was any sentiment inconsistency between the first and second parts of a tweet, it might have been sarcastic. Hence, in this stage, the sarcasm detection model used the CNN-LSTM model again to detect the sentiment of each part of Multiple-Sentence-Tweets separately. Consequently, two new binary features, dpf1 (deep-polarity-feature 1) and dpf2 (deep-polarity-feature 2) were introduced. Dpf1 was activated if the first and second parts of the Multiple-Sentence-Tweet did not have the same sentiment. Dpf2 was activated if the first or second part of the Multiple-Sentence-Tweet did not have the same sentiment as the whole Multiple-Sentence-Tweet.

POS-Feature

This step was conducted to prepare another binary feature named POS-feature. Based on the proposed approach, two POS patterns appeared in most sarcastic Persian tweets:

pronoun+adverb+adjective+verb, e.g. خدایا من چقدر خوشبختم که قراره واکسن ایرانی بزنم!! (OMG! How lucky I am to receive the homegrown vaccine!!)

noun+adjective+adjective+adjective, e.g. وای یه واکسن خطرناک مشکوک خارجی دیگه! چه باحال! (How Cool! Another dangerous, suspect, imported vaccine!)

In this part, the model checked all POS tags in tweets. If any tweets contained one of these patterns, the POS-feature was activated.

Punctuation-Feature

Many studies have shown that punctuation has a remarkable impact on text classification. Sarcasm not only affects the choice of words and meaning, but also leads to particular styles of punctuation or repetition of words, in order to express certain feelings, such as hatred, wonder, or exaggeration. Consequently, the sarcasm detection model counted the number of repeated sequences of exclamation marks and question marks separately. In addition, it counted the number of repetitive characters, such asههههه andخخخخخ (sarcastic punctuation appeared in several of the sarcastic tweets). Next, the model considered two new binary features, Low-punc-feature and High-punc feature. For this stage, five Persian linguistics investigated several sarcastic tweets to find the optimum number for Punctuation-feature activation. Hence, these features were activated as follows:

The Low-punc-feature was activated if the number of ? or ! or repetitive characters was < 3.

The High-punc-feature was activated if the number of ? or ! or repetitive characters was ≥ 3.

Consider the following tweet: “آره خوشحالم که حق انتخاب ندارم و باید واکسن چینی بزنم!!!!!” (Yeah! I am happy to have no rights and should get the Chinese vaccine!!!!!). The model detected the sarcasm in this tweet by discovering a repeated exclamation mark (the High-punc-feature was activated).

After all sets of features were extracted, the model used the random forest classifier to detect sarcastic tweets. Since we used the pre-trained sarcasm detection model, the random forest classifier did not require any input of initial training data. Hence, it was able to label each tweet from both the homegrown-vaccine and foreign-vaccine datasets with either ‘sarcastic’ (0) or ‘not sarcastic’ (1).

After using the sarcasm detection model on both datasets, our methodology could be applied. Suppose a tweet was classified as ‘positive’ (+1) during the SA stage, but was classified as ‘sarcastic’ (0) during the sarcasm detection stage. In this case, the tweet's sentiment was reversed to ‘negative’. In other words, the sentiment of a tweet was changed when the tweet was classified as ‘positive’ and ‘sarcastic’ simultaneously. Let us consider the following tweet: "من چقدر خوشحالم خدایا مرسی بالاخره بعد از مدت ها انتظار امروز فایزرم رو زدم و احتمالا با عارضه های شیکش تا شب بمیرم !!" (Thank God, I am blissfully happy since I finally had my Pfizer vaccine today, and I am likely to die tonight from its interesting side-effect!!). The SA model classified the tweet as ‘positive’, while the sarcasm detection model labeled it ‘sarcastic’. As a result, the tweet's sentiment was reversed, and classified as ‘negative’. Table 1 shows some cases of sarcasm in both datasets, which misled the SA model.

Table 1

Retrieved tweets containing sarcasm, with their sentiment polarities.

Label	Tweet
Positive	خیلی خوبه بعد از عمری واکسن به ما هم رسید تازه اگر این همون واکسن خارجی خوبه باشه که نمیکشتمون! ☹(That's nice, after ages we finally received the vaccine, I wish it were that excellent foreign vaccine which could not harm us! ☹)
PositivePositive	خدا رو شکر تزریق واکسن به همسنای من شروع شده چقدر خوبه که قراره موش آزمایشگاهی دانشمدامون باشیم(Thank God, it's the turn of my peers to get the vaccine, that's interesting to become our scientists’ laboratory mice!!!)باشه، اصلا برکت عالیه ولی میشه توروخداااااااااااااااااا انقدر برکت برکت نکنید؟؟؟!!!Ok, Barkat is wonderful! But would you pleasssssssse not say Barekat???!!

It was assumed that our SA + sarcasm detection model was finalized, and able to be used in critical classification tasks like assessing sentiments for COVID-19 vaccination. To verify this, we tested its performance on our datasets first, and then compared this with the performance of the SA model alone to ensure that our datasets were well presented, and that labeling by the sarcasm detection model was accurate.

Since we had two datasets in this study — homegrown-vaccine and foreign-vaccine — our model was tested by using cross-validation on each dataset separately, and then considering the average performance metrics achieved for both datasets. To do so, for each dataset, 90% of all labeled data were randomly selected as the training set, and the remainder as the test set. This study used the scikit-learn library, which implements cross-validation by using the KFold scikit-learn class. The results are presented in Table 2.

Table 2

Testing the results of the sentiment analysis model with and without sarcasm detection.

Test	Accuracy average	Precision average	Recall average
k-5 sentiment analysis model	0.733	0.730	0.984
k-5 sentiment analysis model + sarcasm model	0.812	0.826	0.911
k-10 sentiment analysis model	0.760	0.762	0.981
k-10 sentiment analysis model + sarcasm model	0.793	0.821	0.909
k-15 sentiment analysis model	0.720	0.732	0.972
k-15 sentiment analysis model + sarcasm model	0.791	0.823	0.904

Fig. 3 shows the percentage of tweets in each sentiment class after using the sarcasm detection model. Positive sentiments towards foreign vaccines accounted for 43% of tweets (n = 173 048), followed by negative sentiments at 45% and neutral sentiments at 12%. The results revealed that positive views towards foreign vaccines decreased by about 3% after deploying the sarcasm detection model. On the other hand, Fig. 3 indicates that positive sentiments towards the homegrown vaccine accounted for 40% of the tweets (n = 160 335), followed by negative sentiments at 40% and neutral sentiments at 20%. Thus, after deploying the sarcasm detection model, the number of positive and negative opinions towards the homegrown vaccine had become the same, with positive sentiments decreasing by about 4% in this process.

Fig. 3

Percentages of sentiment polarities toward imported and homegrown vaccines after using the sarcasm detection model.

Results and discussion

After first testing the effectiveness of sarcasm detection, to see whether it can be a reliable method for improving SA results, analysis was carried out on Iranian sentiments toward COVID-19 vaccines.

The effectiveness of sarcasm detection

Our results were examined with and without the sarcasm detection model to demonstrate that the SA model in sensitive topics such as vaccine opinions needs to be improved by integrating tools such as sarcasm detection. Moreover, vaccination in Iran involves some public obligations; in such situations, native people tend to use sarcastic assertions to convey their opposition.(Karim, et al., 2021) Therefore, to prove that sarcasm detection on this occasion was a reliable method for improving the accuracy of SA, several tests were applied. These involved a k-fold cross-validation method to estimate the accuracy of the model. As shown in Table 2, sarcasm detection improved the accuracy and precision of SA. The accuracy of the SA model when incorporating the sarcasm detection model was consistently above 79% in every fold, which confirmed that the sarcasm detection model could reliably improve the performance of SA. However, the recall values declined with each fold, which meant that the correct sentiment prediction also decreased. One reason was that we only considered sarcasm with negative meaning, while, in reality, it is reasonable to have sarcasm with positive connotations(Yunitasari, et al., 2019). Nevertheless, by applying the sarcasm detection model to SA, the two separate datasets were generated with more accurate positive, negative, and neutral labels.

Overall sentiment toward COVID-19 vaccines

The frequency of tweets regarding COVID-19 vaccines collected over 6 months is presented in Fig. 4 for each vaccine group. As shown, there was a spike in the number of foreign vaccines tweets during the second week of April, with approximately 17 000 tweets. A possible explanation for this could be the declaration banning the import of UK and US COVID-19 vaccines in Iran during those times. Another notable rise in the weekly number of tweets was observed in the second and third weeks of August. This coincided with Iran's government officially granting permission for the importation of AstraZeneca/Oxford, Pfizer/BioNTech, and Moderna vaccines.

Fig. 4

The frequencies of the collected tweets regarding imported and homegrown COVID-19 vaccines over a 6-month period.

Fig. 4 also reveals a rise in the number of homegrown vaccine tweets in the first and second weeks of August, with about 17 100 tweets. At the time, the government had been required to submit the Barekat vaccine application to WHO for evaluation. Public awareness of this increased people's hesitation towards this homegrown vaccine. Another significant increase relating to the Barekat vaccine was observed in the second and third weeks of May, when Iran's government officially declared the successful development of the Barekat vaccine.

Negative sentiments toward COVID-19 vaccines

The distribution of negative sentiments toward COVID-19 vaccines is illustrated in Fig. 5. As shown, there was no statistically significant change in the negative sentiment of tweets towards foreign vaccines from April to late July. However, publicity on the Pfizer vaccine's side-effects in Iran led to a notable rise in negative tweets between late August and September. This upward trend could also be linked with the accounts of certain tweeters, who carried out negative campaigns against particular groups of vaccines(Yousefinaghani, et al., 2021).

Fig. 5

The distributions of negative sentiments toward imported and homegrown COVID-19 vaccines.

In contrast, negative sentiments toward the homegrown Barekat vaccine increased dramatically at the beginning of April (Fig. 5). As mentioned previously, a ban on the import of UK and US COVID-19 vaccines had been announced during that period. Such reports could have correlated with spikes in negative sentiments toward Barekat vaccine. However, the negative sentiments remained steady until late August. In fact, between late August and September, increases in negative sentiments toward the homegrown and foreign vaccines coincided. A possible explanation could relate to reports claiming that Iran's government was planning to mandate the use of Barekat following reports of side-effects associated with the Pfizer vaccine. Hundreds of people opposed the Barekat vaccine mandate. However, since no mandate occurred, the negative sentiments toward Barekat decreased in late September.

Fig. 6 shows the number of tweets with positive, negative, and neutral sentiments per month for each dataset. Positive tweets regarding foreign vaccines were the dominant sentiments for approximately 4 months of the study period. In comparison, the frequencies of positive and negative sentiments toward the homegrown vaccine were almost the same. By analyzing sentiments between both groups of COVID-19 vaccines, our results indicated that while the negative sentiments relating to foreign vaccines increased, the positive sentiments regarding them decreased no more than 10% for the first 4 months of the study period. However, negative sentiments toward foreign vaccines saw their first peak in late August and September (increasing by approximately 15%). At the same time, the positive sentiments regarding them decreased dramatically, by about 25%. A possible explanation is that these vaccines were initially perceived as ‘safe’ among the Iranian people until August. In late August, the announcement of side-effects associated with some foreign vaccines significantly decreased the positive sentiments toward them. At the same time, a 12% decrease in the positive sentiments towards the homegrown vaccine was observed.

Fig. 6

The distributions of opinions toward imported and homegrown COVID-19 vaccines over a 6-month period.

Interestingly, in early April, the announcement of a ban on UK and US vaccines led to a rise in positive sentiments toward foreign vaccines, while also causing a decrease in positive sentiments toward the homegrown vaccine. In addition, there was no significant difference in the number of neutral tweets regarding each vaccine group during the study period.

Conclusion

Our study analyzed the sentiments of 803 278 Persian tweets concerning COVID-19 vaccines retrieved between April 1 and September 30, 2021. A deep-learning model was used for SA and a machine-learning model for sarcasm detection in order to classify vaccine-related tweets more accurately. It was concluded that sarcasm detection enabled us to detect several tweets that ironically stated a positive sentiment toward vaccination. Thus, it improved the accuracy of the SA results, and our SA-Sarcasm Detection model can be considered a reliable tool for further classification problems.

Our results also indicated a subtle difference in the number of positive sentiments toward the homegrown and foreign vaccines, with the latter having a dominant positive polarity. Indeed, sentiments regarding vaccination remained positively stable throughout the first 4 months of the study period. However, there was a slight decrease in the desire to take the vaccine when reports on certain side effects increased in early August and September, with a worrying increase in negative sentiments regarding both homegrown and imported vaccines. No significant differences were observed between the percentages of overall positive and negative opinions toward vaccination amongst the Iranian people.

Additionally, it was concluded that the issue of a mandatory homegrown vaccine led directly to negative sentiments toward it, while reports of an import ban on foreign vaccines caused positive sentiments toward them and further negative sentiments toward the homegrown vaccine.

Since public healthcare agencies aim to increase the uptake of COVID-19 vaccines in order to end the pandemic, they can perhaps focus on social media, such as Twitter, to promote positive messaging toward vaccination.

One of our study's limitations was that the collected tweets covered just a short period of vaccine availability. Further work could focus on vaccine-related tweets after September 2021, when most people were actively receiving vaccines. Furthermore, this study did not explore the attitudes of Twitter users towards each vaccine separately. Our aim now is to identify more vaccine sentiments and to compare their progression by time, post-engagement metrics such as retweets, favorites, and replies, and account characteristics in order to enhance our work.