Exploring content of misinformation about HPV vaccine on twitter.

Although social media can be a source of guidance about HPV vaccination for parents, the information may not always be complete or accurate. We conducted a retrospective content analysis to identify content and frequencies of occurrence of disinformation and misinformation about HPV vaccine posted on Twitter between December 15, 2019, through March 31, 2020, among 3876 unique, English language #HPV Tweets, excluding retweets. We found that 24% of Tweets contained disinformation or misinformation, and the remaining 76% contained support/education. The most prevalent categories of disinformation/misinformation were (1) adverse health effects (59%), (2) mandatory vaccination (19%), and (3) inefficacy of the vaccine (14%). Among the adverse health effects Tweets, non-specific harm/injury (51%) and death (23%) were most frequent. Disinformation/misinformation Tweets vs. supportive Tweets had 5.44 (95% CI 5.33-5.56) times the incidence rate of retweet. In conclusion, almost one-quarter of #HPV Tweets contained disinformation or misinformation about the HPV vaccine and these tweets received higher audience engagement including likes and retweets. Implications for vaccine hesitancy are discussed.

Introduction

The human papillomavirus (HPV) are a group of viruses that infect around 14 million men and women each year and can lead to six types of cancers, including cervical, penile, and anal (National center for immunization and respiratory diseases (NCIRD), 2019). The centers for disease control and prevention (CDC) recommend adolescents ages nine and older routinely be vaccinated against HPV to protect against infections (NCIRD, 2019). In 2019, 54.2% of adolescents ages 13–17 in the United States (U.S.) were up to date (UTD) on their HPV vaccination series, including 56.8% of girls and 51.8% of boys, despite Healthy People 2020 setting an uptake target of 80% (Elam-Evans et al., 2020; Immunization and Infectious Diseases, 2020). Furthermore, compared to vaccination rates of Tdap (90.2%) and MMR (91.9%), the HPV vaccination rate continues to remain far lower among adolescents (Elam-Evans et al., 2020).

A major contributing factor to lower rates of HPV vaccination is vaccine hesitancy, defined as the reluctance or refusal to vaccinate despite availability of vaccines (MacDonald & SAGE, 2015). The reasons why people choose not to vaccinate are complex (Gilkey et al., 2016); however, complacency, inconvenient access, lack of confidence, and safety concerns are key underlying factors (MacDonald & SAGE, 2015). Parents experience many barriers that reduce the likelihood of HPV vaccination, including concerns about vaccine safety, lack of necessity, no requirement for school, no provider recommendation, and lack of vaccine information (Kornides et al., 2018a, 2018b). Of these barriers, high-quality provider recommendations are most important for improving vaccination rates (Kornides et al., 2018a, 2018b). Given that a strong provider recommendation addressing parents’ concerns about HPV vaccine is the most effective way to increase uptake among all adolescents across age, racial/ethnic groups, and urban versus rural locations, robust interventions that prevent hesitancy and enhance providers’ messages are needed to improve coverage in the U.S. (Cunningham-Erves et al., 2019; Thompson et al., 2012). However, provider messages are not the only HPV vaccine messaging to which parents are exposed.

Social media use is particularly popular among parents, with 96% reporting any use in the past 30 days (Bryan et al., 2020). Among parents of adolescents ages 12–18 who use social media, 42% report reading about or discussing child health and development on social media forums (Bryan et al., 2020). Although social media can be a source of guidance about HPV vaccination, the information may not always be complete or accurate—i.e., consistent with prevailing scientific consensus (Ortiz et al, 2019). This type of incomplete or inaccurate information is known as misinformation, defined as the inadvertent spread of false or incomplete beliefs without intent to cause harm (Wang et al., 2019). In contrast, disinformation is content that has been created and shared with intentional deception on the part of an industry or organization with a financial incentive (Zee, 2009). Misinformation often arises as this disinformation content is shared by well-meaning individuals. It can be difficult for content consumers to identify when the disinformation has been posted with intent to deceive. As many as 41% of parents have been exposed to disinformation and/or misinformation pertaining to vaccine safety, medical and pharmaceutical skepticism, and personal freedom on social media forums (Gunaratne et al., 2019). The most common types of negative HPV vaccine posts on social media include inaccurate claims, conspiracy theories, concerns about vaccine safety and personal freedoms, and lack of vaccine efficacy; all are forms of both disinformation and/or misinformation (Ortiz et al., 2019).

Among parents who use the internet as a source of information, fewer are likely to agree to immunize their children (Berreth, 2011; Jones et al., 2012), while frequenting WebMD.com, MayoClinic.com, and government or other medically endorsed sites most frequently (Berreth, 2011). Specifically, studies have found that exposure to negative HPV vaccine posts on social media result in an increase in parents expressing negative opinions and a reduction in their intention to get the HPV vaccine (Dunn et al., 2015; Nan & Madden, 2012). These posts increase parents’ vaccine hesitancy and thus decrease children’s HPV vaccination rates (Argyris et al., 2021). Those who believe that health-related misinformation on HPV vaccine and cancer is accurate are more likely to believe misinformation about other topics as well, threatening to cause a ripple effect among all types of health information online (Scherer et al., 2021).

The contemporary anti-vaccination movement has utilized social media platforms to spread both disinformation and misinformation to fuel widespread vaccine hesitancy through discredited research linking vaccines to harmful side effects and inadequate safety testing on the part of pharmaceutical companies (Gunaratne et al., 2019). When being exposed to HPV vaccine disinformation and/or misinformation on social media, users are more likely to remember the harms than the benefits, resulting potentially in greater HPV vaccine refusal, lower HPV vaccination rates, and greater likelihood to share that content on social media (Dunn et al., 2015, 2017; Margolis et al., 2019). Identifying the content of disinformation and misinformation will allow us to find more effective, targeted, corrective interventions to address hesitancy resulting from this disinformation and misinformation. Without such knowledge, appropriate messaging to address disinformation and misinformation would be blind.

The interventions developed by prior studies are known as simple corrective debunking. For example, one intervention involved exposure to fact-checking labels on misinformation from trusted universities and institutions, which generated greater positive attitudes towards vaccines (Zhang et al., 2021). Another study developed a social media campaign to increase parental awareness of HPV vaccination, unveiling best practices for effectively responding to HPV vaccination misinformation (Sundstrom et al., 2021). However, simple, fact-based correction of misinformation is ineffective at best, and at worst may inadvertently reinforce the misinformed belief (Cappella et al., 2015). In contrast to simple correctives, correctives which present a counterargument to the misinformation had a stronger debunking effect and resulted in weaker persistence of belief in the misinformation (Chan et al., 2017). These findings suggest that to successfully address negative content about HPV vaccine, an approach is needed that constructs a counter message to replace the debunked beliefs. To successfully implement such an approach, a thorough review of the existing disinformation and misinformation, tactics used, as well as its audience reach, is needed. Building on a prior study (Massey et al., 2020), we sought to characterize the content and frequency of disinformation/misinformation about the HPV vaccine, using Twitter instead of Instagram, with the intention of identifying potential targets for corrective intervention, as these messages have become an emerging parental barrier surrounding vaccine hesitancy.

Given the gaps in the literature, the objectives of this study were to identify the content of HPV vaccine disinformation/misinformation on Twitter and estimate its frequency of occurrence within a specific time interval. Our objective for this study was not to differentiate the posters’ intent, so we will hereafter call all disinformation/misinformation posts “misinformation.” Within the misinformation posts, we evaluated use of gender, personal narratives, and reference to age group. To evaluate the reach of the misinformation, we assessed measures of audience engagement, including retweet count, reply count, and favorites. Overall, our goal was to analyze the content of misinformation and the types of misinformation generating the most audience engagement in order to identify potential targets for corrective intervention.

Methods

We conducted a retrospective content analysis of text-based Tweets posted on the online social media platform, Twitter. We characterized posts relative to content, demographic, confidence, and engagement. The study was approved as exempt by the University of Pennsylvania IRB.

Search strategy

The search was done through the retrieval of Tweets posted between December 15, 2019, through March 31, 2020, from the Twitter application program interface (API) stream. We applied an English language filter, a filter to exclude retweets, and used the hashtag ‘#HPV’ to perform a keyword search. With this approach, we retrieved 5342 Tweets containing information about the disease and its corresponding vaccination. Posts eligible for inclusion had to contain English text; links to external sites and text contained within images in the Tweets were not evaluated. We excluded retweets (n = 0) that passed our filter, and non-English language Tweets (n = 1). We randomly selected 4,258 tweets (80%) out of the remaining 5,341 for manual annotation to efficiently use time and resources to complete the annotation.

Coding approach

Prepopulated information

All coding was completed using REDCap electronic data capture tools hosted at the University of BLINDED FOR REVIEW [11]. We built a Python (v. 3.8.1) program to automatically extract the following data from each Tweet to a CSV file: unique Tweet ID, user ID, postdate (month/day/year/hour/second), text, number of retweets, number of replies, and number of favorites the Tweet received. We uploaded the CSV data to REDCap for further manual annotation by the four annotators. When the annotator opened a record for a Tweet on REDCap, the page was prepopulated with a study ID, the text of the Tweet, and the date that the Tweet was posted. The level of engagement for each Tweet was also prepopulated within the record, which included the number of retweets, replies, and favorites received by each Tweet.

We used an iterative process to develop the codebook. We first obtained a separate sample of 4000 Tweets posted prior to the study start date of December 15, 2019. Three annotators (MLK, LS, EP) conducted independent open coding of an initial sample of 25 randomly selected Tweets from the development set to identify themes and subthemes. We compared themes and subthemes to create an initial codebook. Two additional annotators (AM, LV) joined for the second round of coding, using the proposed codebook, to code an additional 53 Tweets from the development set. We compared codes and further refined the codebook. Using the agreed upon categories, four of the five annotators (LS, EP, AM, LV) coded a set of 152 Tweets, and interrater reliability was ensured by a fifth reviewer (MLK) who crosschecked the initial dichotomous coding and calculated Fleiss Kappa scores. Amongst the four annotators, a Kappa score of 0.91 was achieved, indicating almost perfect agreement between individual annotators. After the process of developing the codebook and determining interrater reliability was complete, each of the four annotators independently coded posts from the study set of 4258 Tweets.

Topics

Concern

Tweets were first coded by whether they raised a concern about the HPV vaccine. Example Tweets in this category included “the gardasil/hpv vaccine destroyed our lives. #gardasillkilledmyson.” Posts that were marked as raising a concern were then further analyzed in greater detail by type of concern (see below). During the open coding process, we determined that almost all posts that raised concerns about HPV vaccine contained some degree of misinformation. However, to allow for legitimate concerns, we created a category for concerns commonly found in the literature, such as a lack of provider recommendation or that the vaccine is not required for school. Annotators could also flag the post for further discussion if the concern seemed legitimate and did not contain misinformation.

Support

Posts that did not raise a concern were distinguished by whether they supported or provided educational information about the virus or corresponding vaccination practices. We considered a non-concern Tweet to be supportive or educational if it provided information about HPV infection, its transmission, its treatment, prevention, or where to get vaccinated or tested. Educational information also included content stated in colloquial or graphic terms and could be stated in the form of a question. Examples of supportive Tweets included “Everyone knows I’m a theatre nurse, recently we operated on a straight man under 25 yrs, for HPV lesion in his mouth. Parents -vaccinate your children, they will be sexually active some day. #HPV #VACCINATE,” and “WomensHealth Reality: #HPV is the most common sexually transmitted infection in the U.S. and can lead to cancer. The vaccine helps lower your risk! #MythsvsReality.”

Confidence

For each Tweet, the annotator reported how confident they were about the category/categories they selected by choosing (1) high confidence, (2) medium confidence, or (3) low confidence.

Type of concern

When an annotator determined that a post raised a concern about the HPV vaccine, they then further sub-categorized the specific area(s) of the concern, including health, safety, pharma, government, common concerns, mandatory, ineffective, vague, and other.

Health concerns were defined as posts deeming the vaccine to be unsafe or stated health-related adverse side effects and reactions. If a post was assigned this sub-category, annotators then selected the specific health condition(s) contained within the text from a list of 25 health conditions identified during the open coding process. Some subcategories of health concerns included death, autoimmune diseases, cancer, “risk or harm.” If the health condition in the post was not included in the subcategory list, the annotator had the option to specify the health concern using free text. Example Tweets within these subcategories included “#STUDY: A Case of Fatal #Myocarditis in a 17 Year OLD Girl 1 Week after Receiving #HPV #Vaccine. Death is a reported side effect of #HPVvaccine #Gardasil listed in the vaccine package insert. #LearnTheRisk #VaccineInjury #VaccineDeath #WakeUP,“ and “The #Gardasil controversy: As reports of adverse effects increase, cervical #cancer rates RISE in #HPV-vaccinated age groups.”

Safety concerns included inadequate or falsified safety monitoring during either the pre- or post-licensure period. This subcategory consisted of posts stating that VAERS reports of harms were ignored, mentioned contaminants in the vaccine, expressed concern about people receiving too many vaccines, or claimed that the vaccine is too new or novel to be considered safe. Examples of Tweets in this category include, “Researchers analysing #HPV vaccination clinical trials find they may have overestimated the efficacy,” and “Did you know that #Hepatitis A vaccine was used as "placebo" control for testing #HPV vaccine Cervarix? It's like testing whether cookies can cause obesity, and use cupcakes as the control group.”

Pharma concerns included Tweets that expressed deception or money-making incentives on the part of pharmaceutical companies. Any mention of specific pharmaceutical company names in negative posts fell within this grouping. Examples include: “Did you know #HPV related cancer is among the rarest and #Gardasil #vaccine does NOT target the forms of HPV that may mutate but target a pair that resolve w normal immune system? Gardasil is most profitable vaccine in history & Merck has immunity for serious injury & fatalities,” and “Amid questions about the safety of the #HPV #vaccine #Gardasil one of the lead researchers for the #Merck drug is speaking out about its risks, benefits and aggressive marketing #vaccines #SCAM.”

Government concerns included Tweets that expressed fear of government or group conspiracy, along with money-making incentives of these groups. This category also encompassed mentions of specific politicians and lawmaking bodies. Example Tweets in this category include, “Look up #BillGates and #eugenics. This is the lefts biggest goal. Depopulation through #HPV vaccine sterilization, planned parenthood, 1 in 36 autism rates through vaccination, it is endless their desire to purge the earth of humans,” and “Why would Governor and BOTH Senators of Virginia want to restrict firearm access for citizens, #violate #CONSTITUTION due to $1 Billion Merck $MRK #HPV #VAXX factory? Hmmm?”.

Common concerns included topics frequently identified in the medical literature as common concerns parents have about the HPV vaccine [11, 14]. These concerns include that the child is too young for vaccination against a sexually transmitted disease/or is not sexually active, that the vaccine was not recommended by a physician or required by a child’s school, or that there is a need more information about the vaccine before the parent is able to make a decision.

Mandatory concerns included posts by users that expressed concern that the vaccine being forced upon them or stated that mandatory vaccination was a violation of parental rights. Example Tweets include: “Chicago teen given #HPV #vaccine from nurse practitioner in mobile van without parental consent or knowledge,” and “If we were really worried about the human papilloma virus, we'd be pushing condoms more for both male & female sexual intercourse. But vaccination with something like #Gardasil isn't about health it's about profit. #HPV #WrapItUp #NoMandates #PreventionOverVaccination.”

Inefficacy concerns were those posts that stated the vaccine did not prevent HPV-associated cancers or HPV infection. Examples include, “It's never too late to question HPV vaccine marketing memes. One hypothesis circulating is that HPV vaccines might act as a type of "booster" which speeds up the development of cervical cancer in some women. #cdnhealth #hpv” and “Merck's own #STUDY: #Gardasil INCREASES Cervical Lesions in Women Infected w/ #HPV by 44.6%. #HPVvaccine may EXACERBATE the very disease it's supposed to prevent. Still, girls & young women are vaccinated without pre-screening for HPVs.”

Vague concerns included those that were general or nonspecific and did not fall within any of the previously listed subcategories of concern. For example, one post that fell within this category stated, “I just wonder about all the girls that received the hpv vaccine in Ireland @REGRET_ie”.

Concern-related information

For all posts that the annotator determined to raise a concern about the HPV vaccine, they coded for the mention of gender or age group, if the concern was presented as a personal narrative/story, and if the personal narrative included a vaccine-related injury to a child.

Gender

Posts that expressed concern about the HPV vaccine were also categorized by whether or not gender was mentioned. Options included that the post mentioned: (1) girls/women, (2) boys/men, (3) both, or (4) neither, doesn’t mention gender. Posts were not annotated for nonbinary gender.

Age

Annotators specified if the concern posts referenced age/age range of the subject(s). Annotators selected from the following options: (1) children, girls or boys, (2) adults, men or women, (3) both, or (4) none of the above: doesn’t mention age.

Personal narrative/story

For Tweets that raised concern about the HPV vaccine, annotators determined if the poster told a personal story, which could be their own or that of someone else. This category included mentions of individuals’ names, hashtags with names of individuals (ex. “#RIPColton”), and case studies referring to individuals. In this section, the annotator selected yes or no.

Injury to child

If the annotator selected yes to answer the personal narrative question, they were then asked if the story included injury to a child (age under < 18). Here, the annotator could select: (1) yes, the story included injury to a child (2) no, or (3) uncertain/age not stated.

Additional information

For all Tweets, the annotator could provide any additional information in a free textbox.

Flagging

For all Tweets, the annotator could flag the post for further discussion with the group of annotators. Annotators flagged Tweets that they had trouble categorizing or included a trend in the type of concern it posed that had not previously been captured in the survey. If the annotator selected to flag a Tweet, they noted the reason for flagging in a free textbox.

Audience engagement

The Python program automatically extracted the following measures for each Tweet: Retweet count, Reply count, and Favorite count. We created a summary measure of audience engagement by summing the three measures together. In addition to reporting the summary measure, we report Retweet count separately in the results as it has the widest audience reach.

Analysis

Data cleaning

Among the 4824 posts by 2127 unique users, 699 contained neither support nor concern and were excluded from further analysis. Among the unique users, n = 1231 posted only one Tweet, and n = 896 posted two or more Tweets. We further excluded n = 249 posts that were duplicates by user ID, text, and postdate. We retained posts that were duplicated by user ID and text but not postdate (n = 218). The majority of these posts (75%) were only reposted once. Among repeat posts, concern posts were more likely to be reposted than supportive posts (16% vs. 2%, p < 0.001).

Missing and outliers

We did not have missing data for the information extracted by the Python program (audience engagement, post text, postdate, etc.). Among the concern posts, 1% (n = 10) were missing a combination of gender, age, and/or personal narrative data. These posts were excluded from the analyses on these variables. The audience engagement data were severely right skewed. We examined the top 5 outliers for each measure and evaluated the original post on Twitter to confirm the source data. All of the metrics were consistent with what the Python program had extracted. As information and misinformation on social media is mainly spread through posters with higher levels of audience engagement, “going viral” is considered a goal of social media. We elected to keep the outliers as valid data points. After data cleaning, the final sample size was n = 3876 posts by 1780 unique users (n = 3710 unique text posts).

Descriptive analysis

We examined the prevalence of the two main categories (concern vs. support), as well as for the subcategories of concern. We calculated the mean (SD) of the summed audience engagement measure (Retweet count + Reply count + Favorites count) and assessed the audience engagement metrics across categorical variables.

Inferential analysis

Our outcome variable of interest was audience engagement. We conducted separate analyses of retweet count and the summed audience engagement measure. We conducted a negative binomial regression by building a GLM model in Stata version 15 with a negative binomial family and log link to allow for excess number of zeros in the audience engagement metrics (i.e., over-dispersed count data), and we adjusted the standard errors for clustering by poster ID to assess for incidence rate ratio (IRR) on audience engagement outcome between post-category, as well as subcategories. Among the misinformation posts, we controlled for gender, age, and personal narrative in a multivariable model.

Sensitivity analyses

We conducted sensitivity analyses for by creating indicator variables for each month that the Tweets were posted and including them in the final model. We found no difference in the results for the final model by including study month and do not present the findings below. We also assessed for differences in the proportion of supportive versus misinformation Tweets posted by month of the study (Table 1).

Table 1

Number (Percent) of support versus misinformation #HPV tweets by study month

Month	Total N	Support n (%)	Misinformation n (%)
Total	3876	2945 (76.0)	931 (24.0)
December 15–31, 2019	468	301 (64.3)	167 (35.7)
January 1–31, 2020	2198	1718 (78.2)	480 (21.8)
February 1–29, 2020	1019	783 (76.8)	236 (23.2)
March 1–31, 2020	191	143 (74.9)	48 (25.1)

Results

Frequency of misinformation

Nearly a quarter (n = 931) of HPV related-content on Twitter over a three-and-a-half-month period contained concerns including misinformation about the HPV vaccine in this retrospective content analysis, whereas 76% (n = 2945) contained supportive information about HPV disease transmission, treatment, or vaccination. Among the supportive posts, 60% (n = 1758) mentioned the HPV vaccine. Table 1 shows the number of Tweets on #hpv vaccine overall by study month, and by supportive versus misinformation.

Overall, the 3876 Tweets had a mean retweet count of 9.12 (SD 134.1), a median of 1, interquartile range (IQR) of 4, a mean audience engagement of 28.65 (SD 542.5), median of 3, and IQR of 12. Misinformation posts had higher retweet counts as compared to supportive posts [mean 24.00 (SD 267.3), median 3 (IQR 16) vs. mean 4.41 (SD 31.4) and median 1 (IQR 2), p < 0.0001] and [mean 62.54 (SD 1062.7) and median 8 (IQR 31) vs. mean 16.36 (SD 172.8) and median 3 (IQR 8), p < 0.0001], respectively. Misinformation posts were 5.44 (95% CI 2.50–11.84) times as likely to be retweeted as support posts (see Table 2).

Table 2

Incidence rate ratio of audience engagement for support versus misinformation #HPV tweets

	IRR (95% CI)
Model 1 retweet count
Support tweets	Referent
Misinformation tweets	5.44 (2.50–11.84)
Model 2 audience engagement a
Support	Referent
Misinformation	4.41 (3.32–5.86)

aAudience engagement = sum of Retweet count, reply and favorite counts

Types of misinformation

Among the subcategories misinformation Tweets, Tweets containing health-related concerns (n = 931) were the most frequently occurring (Table 3.) The top three categories of misinformation were health-related (58%, n = 539/931), concerns over mandatory vaccination (19%, n = 180/931), and misinformation stating that the vaccine is ineffective (14%, n = 127/931). Approximately 15% (n = 134/931) of Tweets contained misinformation about pharmaceutical companies and/or the government combined. Only 2% (n = 17/931) of Tweets contained concerns commonly identified in the literature (lack of provider recommendation, child too young, child not sexually active, vaccine not required for school, etc.).

Table 3

Categories of concern-related tweets around HPV vaccine

Type of concern	Definition and example tweet	Number of tweets (%) * N = 931
Health	Vaccine is not safe: health related adverse side effects/ serious reactions Example: “the gardasil/hpv vaccine destroyed our lives. #gardasillkilledmyson”	539 (58)
Mandatory	Forced vaccination, violation of parental rights Example: “this is a person who wants to mandate medical procedures for your 9 year old without your consent. #gardasisdangerous #parentscalltheshots”	180 (19)
Ineffective	Vaccine does not prevent hpv-associated cancer or hpv infection Example: “#hpv vaccination is not reducing #cervicalcancer rates. To the contrary research shows that it is making matters worse.”	127 (14)
Other safety	Inadequate or falsified safety monitoring “merck’s own #study: #gardasil increases cervical lesions in women by 44%”	106 (11)
Government	Government or group conspiracy, or money-making/profit incentive “governor announced significant investment by merck in the county. #gardasil”	69 (8)
Big pharma	Deception or money-making incentive on part of pharmaceutical companies Example: “when you blindly follow the system, thinking that bigpharma has your best interests, you might just ruin the future of your children. #gardasilkills”	65 (8)
Vague/general	Nonspecific concern (not covered by one of the other categories) “professor condemns #hpvvaccine after winning $270 k federal grant to study it.”	64 (8)
Other	Other specific concern “parents, it is imperative you research everything a doctor/nurse tells you when it comes to #hpv vaccine. Doctors are paid a bonus for every shot they give. It’s the bread and butter of a practice!”	40 (5)
Concerns in the literature	Child too young, child not sexually active, no provider recommendation, poster needs more information, not required by school	16 (2)

*Categories were not mutually exclusive. Tweets could be assigned to multiple categories. Percent not equal to 100

Most of the misinformation posts did not mention gender (73%, n = 670/921). Approximately 20% (n = 183) mentioned females, 5% (n = 51) mentioned males, and 2% (n = 17) mentioned both females and males. Tweets that mentioned any gender were less likely to be retweeted [IRR 0.71 (95% CI 0.69–0.73)] and had lower rates of audience engagement [IRR 0.47 (95% CI 0.46–0.48)] as compared to tweets that did not mention gender.

Approximately 63% of the misinformation Tweets (n = 580/921) did not mention age. Tweets that mentioned children (31%, n = 283/921) were 4.4 times as likely to be retweeted as compared to Tweets that did not mention age (95% CI 4.24–4.49). Similarly, Tweets about children had 6.4 times the rate of audience engagement overall (95% CI 6.39–6.64) vs. Tweets that didn’t mention age. Tweets that included adults had 2.17 (95% CI 2.06–2.30) times the retweet count and 2.11 (95% CI 2.03–2.19) times the audience engagement as compared to Tweets that did not mention age.

Personal narratives

While most misinformation Tweets did not include personal narratives (17%, n = 156/931), those Tweets that included personal narratives had higher levels of audience engagement as compared to those that did not. Tweets that did not include a personal narrative had a mean retweet count of 14.2 (SD 40) vs. 71.7 (SD 646.6) for those Tweets that did. Tweets that included a personal narrative vs. those that did not have an IRR of 5.05 (95% CI 4.90–5.17) for retweets and 7.92 (7.79–8.04) for audience engagement. Retweets and overall audience engagement further increased when the personal narratives were about children. Compared to Tweets without a personal narrative, Tweets that included a personal narrative about children had a retweet IRR of 6.93 (95% CI 6.56–7.33) and an audience engagement IRR of 11.94 (95% CI 11.50–12.40).

Multivariable model

In our multivariable models comparing subcategories of misinformation for the separate outcomes of Retweet count and Audience Engagement and controlling for the use of gender, personal story, and mention of age, Tweets containing concerns about vaccine mandates (vs not) had the highest rates of retweets and engagement, respectively (see Table 4). Tweets containing misinformation in the “other category”, misinformation that stated the vaccine is ineffective, and adverse health-related misinformation also had higher rates of retweets and engagement vs. if the Tweet did not contain these types of misinformation. Certain types of misinformation were less likely to be retweeted; Tweets containing misinformation about Pharma [IRR 0.68 (95% CI 0.63–0.74)], common concerns from the literature [IRR 0.42 (95% CI 0.31–0.56)], and government [IRR 0.40 (95% 0.37–0.43)] were all less likely to be retweeted than if the post did not contain this type of misinformation.

Table 4

HPV vaccine disinformation/misinformation subcategory predictors of retweets and combined audience engagement

Multivariable model	Retweet incidence rate ratio (95% CI)	Engagementa incidence rate ratio (95% CI)
Mandate	10.9 (10.6–11.3)	19.4 (19.0–19.8)
Other	3.7 (3.4–3.9)	6.03 (5.7–6.4)
Ineffective	3.2 (3.1–3.4)	3.8 (3.7–3.9)
Health	3.1 (2.9–3.2)	3.2 (3.1–3.2)
Safety	2.1 (2.0–2.2)	2.5 (2.4–2.6)
Vague	1.5 (1.4–1.6)	1.7 (1.6–1.8)
Pharma	0.68 (0.63–0.74)	0.79 (0.75–0.83)
Common	0.42 (0.31–0.56)	0.50 (0.42–0.59)
Government	0.40 (0.37–0.43)	0.41 (0.39–0.43)

Separate negative binomial model for each subcategory predictor controls for clustering by poster ID, gender, personal story, and age

aAudience engagement = sum of retweet count, reply and favorite counts

Adverse health-related misinformation

Among the subcategories of adverse health-related misinformation (n = 539), non-specific health harm was most frequently occurring (51% of Tweets), followed by the vaccine causing death (23%) and the vaccine causing cancer (17%) (Fig. 1).

Fig. 1

Health misinformation subcategories. Harm NOS health-related harm, not otherwise specified. Chart excludes categories with less than 5%

Discussion

In our review of HPV-related tweets posted in a three and a half-month period, nearly a quarter contained misinformation about the HPV vaccine, with the most prevalent category of misinformation pertaining to adverse health effects from vaccination. We found that misinformation posts had higher rates of audience engagement, including likes, retweets, and replies, representing greater audience reach as compared to supportive posts. This is consistent with most existing research on disinformation and misinformation engagement. Out of a sample of HPV vaccine posts from Facebook collected by Luisi (2020), anti-HPV vaccine messages had significantly greater audience engagement than posts with neutral or pro-HPV vaccine messages. Topics associated with mainstream news events, issues, safety concerns, and conspiracies displayed high correlations in online HPV vaccine reach with both males and females, while positive topics like advocacy and reporting evidence had weaker correlations with online HPV vaccine reach (Dunn et al., 2017). Similarly, on Instagram, although there were more pro-vaccine posts than anti-vaccine, there was a greater average of likes for anti-vaccine posts than pro-vaccine (Basch & MacLean, 2019; Massey et al., 2020), reinforcing the idea that disinformation/misinformation garners greater viewer engagement. However, one previous study—and one of the more dated ones in terms of Twitter research—of HPV-related Tweets from 2014–2015 found no difference in the mean number of retweets between positive and negative Tweets (Massey et al., 2016).

The higher audience reach that misinformation achieves is concerning, as many parents report using social media as a source of information. A 2020 study found that almost all parents reported some social media use within the last 30 days (Bryan et al., 2020). Among the types of platforms used by those parents reporting social media use, 74% use Facebook, 25% use Instagram, and 23% use Twitter. Among social media-using parents, 59% report they have come across useful information specific to parenting within the last 30 days when browsing other social media content (Duggan et al., 2015). As social media engagement increases, there is heightened awareness of the HPV vaccine (Rosen et al., 2020). Our finding that anti-vaccine audience engagement is higher than pro-vaccine is consistent with previous studies (Basch & MacLean, 2019; Luisi, 2020; Massey et al., 2020), and the exposure is subsequently exposing parents to further misinformation on the HPV vaccine.

Although we identified eight broad categories of HPV vaccine-related misinformation, the majority of the types of concern containing misinformation fell into three categories: adverse health effects, mandatory vaccination, and the inefficacy of the vaccine. Of these three, adverse health-related misinformation was overwhelmingly the most common, with 60% of the misinformation posts containing some health-related misinformation. The greater portion of health-related vaccine misinformation is consistent with existing research. Suarez-Lledo and Alvarez-Galvez (2021) cited studies evaluating Twitter messages with the prevalence of health misinformation posts ranging from 0 to 87% (Chary et al., 2017; Krauss et al., 2015). Scientific and academic evidence are mainly used and misused within the online vaccine-hesitant community, with the National Institute of Health being the most hyperlinked source within that community (Getman et al., 2018). Health-related words were used more in anti-vaccination online comments than pro-vaccination comments (Faasse et al., 2016). In Kang et al. (2017), significant topics found within negative social media messages included health topics like CDC, vaccine industry, autism, flu shots, doctors, vaccine ingredients, mandatory vaccines, and pharmaceutical companies. Another study coded anti-vaccination comments on Facebook and found that “educational material” posts were most common (Hoffman et al., 2019). This included links to research articles, quotes from healthcare professionals against vaccines, and the sentiment that parents, not doctors, need to educate themselves and the public, all health-related misinformation. This suggests that counter-messages targeting the health-related risks of the vaccine itself on the health of the child would be particularly salient in future intervention work.

We found the misinformation posts were frequently vague and nonspecific to a gender, age, or specific health condition. This may increase their saliency to a broader audience. The majority of misinformation posts did not mention either gender, and posts that did mention a gender were less likely to be retweeted or score as high on the summed audience engagement metric. However, a previous study has found that HPV tweets with attached images showcase female faces the most with majority being ethnically white, displaying another form of misinformation through inaccuracy of the actual burden of disease (Lama et al., 2018). In another analysis, 150 HPV Instagram posts contained photos that were more likely to feature women (29%) than men (9%) (Basch & MacLean, 2019).

We were particularly interested in the use of personal narratives/stories to convey misinformation messages. The majority of misinformation posts did not use a personal narrative. However, the almost 20% that did use a narrative format were more likely to be retweeted and engaged with as compared to those that did not. Identification with characters increases emotional responses to health messages (Davin, 2000), as no data could alter one’s own experiences (Rodriguez, 2016), offering a psychological reason for the higher engagement. For some vaccine-hesitant individuals, personal experience is even more persuasive than scientific data due to the greater profoundness of the experience over scientific findings (Rodriguez, 2016). Personal narratives pertaining to children were particularly impactful. In personal narratives, avoiding vaccination was deemed the best way to protect children and to be an accountable parent (Kata, 2010). The higher engagement among personal narratives aligned with prior studies. For example, Massey et al. (2020) analyzed HPV vaccine posts on Instagram, finding that personal narrative posts garnered a greater average of likes than information/resource posts, with a higher percentage of personal narrative posts being anti-vaccine than pro-vaccine to display how an individual was negatively impacted by the vaccine (Massey et al., 2020; Teoh, 2019). With these personal narratives in misinformation posts, Margolis et al. (2019) found that a third of the parents in their survey who have only heard stories about harm were more likely to delay HPV vaccination. These stories incorporated details regarding those who were harmed after getting the HPV vaccine, including long-lasting and mild side effects, temporary serious harm, and death (Margolis et al., 2019). There is a trend of greater perceived risk of vaccination and lower intention of getting vaccinated when presented with narratives reporting greater vaccine adverse compared to “base-rate” information (Haase et al., 2020). In any case, one conclusion from ours and others’ research is that statistical, fact-based information may not garner as much engagement in these social media spaces and with already vaccine hesitant parents, and interventions which seek to counter this misinformation may also need to employ narrative tactics to address vaccine hesitancy.

Finally, among the health-related misinformation, the most mentioned were non-specific adverse health effects. We hypothesize that keeping the threat vague and general is more psychologically threatening to the audience. This can firstly be attributed to risk amplification—amplifying the perception of a real or fake risk event, from vague to specific. When viewers subjectively perceive the HPV vaccine as a risk event, they tend to amplify the risks with a negative tone, resulting in greater audience engagement (Luisi, 2021). A second reason could be related to the structure of the message itself. A study of antivaccine content found characteristics of analytical thinking, such as factual, logical structure and fewer anxiety-ridden words, despite the message not being scientifically based (Faasse et al., 2016). These factors that can increase the believability of vague and general threats. Finally, variables in the audience’s environment may impact their likelihood of believing vague and general threats. For example, research done on rumors, similarly defined like general threats as information arising in the context of ambiguity yet relevant to a situation, shows that uncertainty, outcome-relevant involvement, lack of control, anxiety, and belief impact how believable general threats are (DiFonzo & Prashant, 2007). Twitter is considered an informal mechanism for spreading rumors (Abdullah et al., 2015), such that misinformation with weak sources can easily circulate and impact and amplify beliefs (Chen & Sakamoto, 2013).

Inoculation theory-informed interventions may be a particularly valuable way to address misinformation ambiguity, as the theoretical framework suggests that individuals can be “pre-warned” that they might be exposed to certain types of messages meant to lead them astray (Chan et al., 2017). Inoculation theory holds that individuals’ attitudes may be inoculated to resist subsequent persuasive attacks in a manner similar to a vaccine inoculating the immune system against a disease (Compton et al., 2016). Inoculation theory operates on the premise that warning an individual of an impending persuasive attack and providing them with an argument to counteract it will inoculate the individual to potential attitude changes when faced with the attack. This inoculation occurs through ‘prebunking’ or the exposure to persuasive messaging which directly presents both a weakened version of the misinformation and refutations of both the misinformation and its source (Compton et al., 2016). With prebunking, a forewarning is provided about future misinformation before the misinformation narrative has been accepted by the individual (Lewandowsky et al., 2012). While inoculation interventions tend to be brief and are most typically delivered in the form of messaages that present the warning and counterargument, the protective effects of the inoculation can range from weeks to months, depending on the exposure and behavior (Banas & Rains, 2010). Inoculation theory has been found effective in preventing beliefs in misinformation, including vaccine-related misinformation (Maertens et al., 2021; Roozenbeek & van der Linden, 2019; Vivion et al., 2022). For non-specific health misinformation, an intervention could confer resistance to the effect of this type of information by exposing parents to the dangers of ambiguous misinformation and providing examples of what that information looks like and why it is bad. Targeting parents early would be key in these situations, as the theory suggests pre-exposure to potential misinformation—and building up counterarguments and resistance—is key.

Our study, taken within the context of the COVID-19 pandemic and ensuing vaccine hesitancy crises surrounding the COVID-19 vaccines, indicates there is an urgent need to combat HPV vaccine misinformation by preventing belief in it before it leads to hesitancy. Vaccine hesitancy has been strongly associated with vaccine intent and vaccination behaviors, including refusal or delay of vaccination (Edwards et al., 2016). For example, in a recent study of 2020 parents using the Vaccine Hesitancy Scale for the adolescent HPV vaccine, 69% of non-hesitant parents had begun the HPV vaccine series for their child vs. 16% of hesitant parents, p < 0.0001, representing 2.7 (95% 2.5, 3.0) times the odds of refusal/non-vaccination among hesitant parents vs. non-hesitant parents (Helmkamp et al., 2021). Health care providers report regularly encountering parents with hesitancy about adolescent vaccines, especially controversial adolescent vaccines such as the HPV and COVID-19 vaccines, suggesting more proactive strategies are needed to combat misinformation (McRee et al., 2014).

Misinformation evolves rapidly and requires a nimble counter-messaging approach, whether inoculative (prebunking) or post-hoc. For inoculation messages to be successfully implemented, they must be used on emerging misinformation before the beliefs have been widely adopted. Social listening using machine learning and natural language processing computational methods would allow for large-scale data surveillance from multiple social media platforms to understand any changing and emerging concerns and attitudes about vaccines and identify nascent misinformation. In a study about COVID-19 misinformation on TikTok, misleading shares and ‘likes’ that spread false information (using #Coronavirus) were the highest during the first two months of COVID-19 pandemic (Southwick et al., 2021). A parallel study that mined Reddit data also revealed uncertainty and misinformation generation and propagation just before and during the initial rollout of vaccines and launch of health behavior campaigns (Stokes et al., 2020). These findings underscore the necessity of collecting and mining multiple social media platforms to identify and develop effective inoculative messages against HPV vaccine misinformation as well as for other vaccines, especially during anticipated times with higher postings when antivaxxers are more likely to post content in response to provaccine contact, such as Cervical Cancer Awareness month as seen in our study.

We note several limitations to our study. (1) We restricted our analysis to Twitter data. Twitter is not the most commonly used social media platform among parents. However, previous research, as well as our own currently ongoing work, indicate that the vaccine misinformation content is remarkably consistent across platforms (Ashfield & Donelle, 2020). (2) It was beyond the scope of our study resources to differentiate between misinformation and disinformation posts. To do so would require investigating the poster in more depth, to determine the motives and potential profit incentives. However, misinformation most often arises from the spread from disinformation. We also did not collect information or control for the size of the poster’s account, which could act as a potential confounder if the number of followers is differentially associated with misinformation vs. supportive posts, as more followers will naturally increase the audience engagement metrics. We hypothesize that the number of account followers is not differentially associated with the independent variables, but future studies should assess this more thoroughly. (3) To use our available resources efficiently, we limited our search term to “hpv” and the time period to three and a half months thus further limiting the scope of and generalizability of our study. We intend to build upon our findings in a larger, broader longitudinal study of misinformation about HPV vaccine across multiple social media platforms using multiple search terms. (4) We conducted our study in January, which is cervical cancer awareness month in the U.S. We included all posts pertaining to HPV education, awareness, screening, treatment, and prevention in the support category, and cervical cancer awareness month increased the number of these posts. We made this decision as we deemed any content that raised awareness about HPV in general to be supportive of the HPV vaccine. If the analysis were to be restricted to content only supportive of the HPV vaccine (60%), the proportion of supportive to misinformation posts would be much smaller. However, as seen in our breakdown of posts by month, the proportion of supportive to misinformation posts did not change substantially between January and February or March. (5) We controlled for gender, personal narrative, and mention of age in our inferential analyses as confounders, as these were more likely to be mentioned in misinformation than supportive posts and were associated with the audience engagement metric outcomes. It’s possible that these variables serve as mediators between misinformation and audience engagement, enhancing the effect the misinformation message has audience engagement. In such case, controlling for them will have reduced the effect estimate, resulting in an underestimate of the true association between misinformation about the HPV vaccine and audience engagement metrics. A future study examining the role of these communication tactics as potential mediators would elucidate this relationship.

Conclusion

In this review of Tweets related to HPV over a three and a half-month period, we found that nearly a quarter contained misinformation, with adverse health-related misinformation being most common. Misinformation posts were more likely to be engaged with by the Twitter audience as compared to supportive posts. Given that many parents rely on social media for health-related information, future interventions, such as those that prebunk misinformation to prevent the harmful effects on vaccination acceptance are urgently needed.