A comprehensive analysis of Italian web pages mentioning squalene-based influenza vaccine adjuvants reveals a high prevalence of misinformation.

Squalene-based adjuvants have been included in influenza vaccines since 1997. Despite several advantages of adjuvanted seasonal and pandemic influenza vaccines, laypeople's perception of such formulations may be hesitant or even negative under certain circumstances. Moreover, in Italian, the term "squalene" has the same root as such common words as "shark" (squalo), "squalid" and "squalidness" that tend to have negative connotations. This study aimed to quantitatively and qualitatively analyze a representative sample of Italian web pages mentioning squalene-based adjuvants used in influenza vaccines. Every effort was made to limit the subjectivity of judgments. Eighty-four unique web pages were assessed. A high prevalence (47.6%) of pages with negative or ambiguous attitudes toward squalene-based adjuvants was established. Compared with web pages reporting balanced information on squalene-based adjuvants, those categorized as negative/ambiguous had significantly lower odds of belonging to a professional institution [adjusted odds ratio (aOR) = 0.12, p = .004], and significantly higher odds of containing pictures (aOR = 1.91, p = .034) and being more readable (aOR = 1.34, p = .006). Some differences in wording between positive/neutral and negative/ambiguous web pages were also observed. The most common scientifically unsound claims concerned safety issues and, in particular, claims linking squalene-based adjuvants to the Gulf War Syndrome and autoimmune disorders. Italian users searching the web for information on vaccine adjuvants have a high likelihood of finding unbalanced and misleading material. Information provided by institutional websites should be not only evidence-based but also carefully targeted towards laypeople. Conversely, authors writing for non-institutional websites should avoid sensationalism and provide their readers with more balanced information.

Introduction

Let us consider the following Italian word family: squalo, squallido and squallore. While the first word means “shark”, the other two both have negative referential meaning (cf. “squalid” and “squalidness”). This word family derived from the Latin adjective squālidus, derived in turn from the verb squālēre that means “to be dirty/rough”. While the English “squalid” and “squalidness” may be considered as learned terms, the Italian counterparts are rather common. The term “squalene” also comes from the same Latin root since this natural compound was first isolated from the shark liver oil. The denotative(i.e. literal) meaning of squalene may be described as follows: a natural unsaturated 30-carbon polyprenyl compound playing the key role in cholesterol biosynthesis., On the other hand, because the term “squalene” is unfamiliar to most laypeople, its connotative (i.e. emotionally charged) meaning would likely be negative through associative analogies with the Italian words squalo, squallido and squallore.

Nowadays, squalene has various uses. Mostly, it is widely employed by the cosmetic industry because it has properties which make skin smooth and elastic. Squalene may also be considered a potential natural chemopreventive substance; indeed, in the Mediterranean diet – the advantages of which are internationally recognized – daily intake of squalene may reach up to 400 mg, mainly due to the intake of olive oil. By contrast, in other countries, such as the US, its intake is only around 30 mg. Squalene and shark liver oil are also popular dietary supplements. And finally, squalene-oil-in-water emulsions (e.g. MF59®, AS03®, AF03®) have widely been used as vaccine adjuvants since 1997. The major benefits of including squalene-based adjuvants in vaccine formulation are: enhanced immunogenicity; more rapid antibody responses; more persistent antibody responses; enhanced heterologous antibody responses; and the possibility of antigen dosage sparing, which is crucial during emergencies such as influenza pandemics. Seasonal influenza vaccines adjuvanted with MF59® have been widely used to immunize elderly individuals for the last 20 years. Both MF59® and AS03® were components of the 2009 H1N1 pandemic vaccines used in all age groups.

Despite the aforementioned advantages of squalene-based adjuvants, their widespread use generated various scientifically unsound claims and myths (these are described in the Methods section). In Italy, which was the first country where MF59®-adjuvanted influenza vaccine was authorized, such myths were somewhat reinforced by the etymology of the word “squalene” described earlier. A further challenge occurred in late 2014 when the Italian Medicines Agency suspended two batches of the MF59®-adjuvanted seasonal influenza vaccine as a precautionary measure following three post-vaccination deaths. Although, following an investigation, the vaccine complied with all existing standards and no causal association was found, this event gave rise to several consequences including unprecedented clamor in both traditional media and internet, “epidemic of panic”, general increase in mistrust in vaccines and vaccine hesitancy, (consequent?) decrease in the influenza vaccination coverage rate,, as well as a “reawakening” of early myths around squalene and adjuvants. Although this false alarm evolved in a short time frame, its repercussions were long-term and persist even until today. Indeed, both traditional and new media are powerful means of amplifying inaccurate scientific information, undermining confidence in vaccines and reducing vaccination coverage. Probably the best-known example of this phenomenon is provided by the Wakefield case: the widespread media coverage of an alleged link between measles-mumps-rubella (MMR) vaccination and autism led to a dramatic drop in the uptake of this vaccine and an increase in the number of measles cases reported. In the United Kingdom, it took more than ten years to restore the MMR coverage rate to the pre-Wakefield level.

Because the internet is one of the main sources of health-related information, it is widely used by anti-vaccination activists to disseminate misleading information on immunization, and the fact that anti-vaccination websites are highly prevalent on the Italian web and may be highly ranked by the common search engines, we hypothesized there to be a high frequency of web-pages critical to squalene and squalene-based adjuvants among Italian online resources.

The major goal of the present study was to quantitatively and qualitatively analyze a representative sample of squalene-related Italian web pages.

Results

Of 2,550 web pages screened, 2,280 were removed as duplicates. Of 270 texts assessed, 141 met the inclusion criteria (5 and 123 web pages had no reference to influenza and squalene in influenza vaccines, respectively, while one page was not accessible). Subsequently, 2, 7, 33 and 15 web pages were excluded meeting exclusion criteria 1–4, respectively. In total, 84 web pages were included in the analysis.

Forty of 84 web pages [47.6% (95% CI: 36.6–58.8%)] were categorized as negative or ambiguous (Neg/Amb), while the remaining 44 [52.4% (95% CI: 41.2–63.4%)] as positive or neutral (Pos/Neu). Table 1 reports the main characteristics of the included web pages, broken down by the general tone. The top-level domain “.it” largely prevailed among both Pos/Neu and Neg/Amb web pages; no statistically significant difference (χ2(3) = 1.92, p = .59) between the distributions of top-level domains among Pos/Neu and Neg/Amb web pages was established. By contrast, many more (5.8-fold difference; χ2(1) = 12.02, p<.001) Pos/Neu web pages belonged to the category of institutional web sources; the effect size was medium (φ = 0.38). In comparison with Pos/Neu web pages, more than double (χ2(1) = 12.50, p<.001) of those categorized as Neg/Amb had at least one picture with a medium effect size (φ = 0.39). Analogously, the Mann-Whitney test highlighted that the distribution in the absolute number of pictures between Pos/Neu and Neg/Amb was not identical (p <. 001) with a medium effect size (r = 0.38). On the basis of the main topic, pictures were categorized into 6 classes, namely images representing the vaccination process, syringes, vaccine packaging, the influenza virus, sharks/monsters, other.

Table 1.

Characteristics of the web pages analyzed, by general tone.

Variable	Level	Positive/neutral (N = 44)	Negative/ambiguous (N = 40)
Top-level domain, % (95% CI)	.it	72.7 (57.2–85.0)	60.0 (43.3–75.1)
	.com	6.8 (1.4–18.7)	10.0 (2.8–23.7)
	.org	11.4 (3.8–24.6)	12.5 (4.2–26.8)
	Other	9.1 (2.5–21.7)	17.5 (7.3–32.8)
Website source, % (95% CI)	Institutional	43.2 (28.3–59.0)	7.5 (1.6–20.4)
	Non-institutional	56.8 (41.0–71.7)	92.5 (79.6–98.4)
Web pages with at least 1 picture, % (95% CI)	—	34.1 (20.5–50.0)	75.0 (58.8–87.3)
Number of pictures, median (range)	—	0 (0–3)	1 (0–8)
GulpEASE index, mean (SD)	—	42.9 (3.7)	46.4 (3.6)
Common words, % (SD)	—	73.6 (5.3)	76.3 (4.7)

The distribution of single picture categories broken down by general tone is reported in Table 2; Neg/Amb pages presented more frequently images of syringes, viruses and sharks/monsters. However, no statistically significant difference between Pos/Neu and Neg/Amb web pages emerged (Fisher's exact test: p = .19).

Table 2.

Categories of images, by general tone.

Category	Positive/neutral, N	Negative/ambiguous, N
Vaccination	7	6
Syringe	2	10
Vaccine	3	6
Virus	0	4
Sharks/monsters	0	3
Other	9	29

Only 3 web pages had video material (one video each): two videos contained interviews with anti-vaccination activists, while the third had an interview with a Hollywood star on vaccine-related topics. Due to the paucity of videos, these were not analyzed quantitatively.

With regard to the readability properties, Pos/Neu web pages could be judged to be more (|t|(82) = 4.33, p <. 001) difficult to read than Neg/Amb pages with a large effect size of |d| = 0.95 (95% CI: 0.49–1.40). Indeed, all 11 web pages with a GulpEase index < 40 were Pos/Neu (Fisher's exact test: p <. 001). The frequency of commonly used words was also higher (|t|(82) = 2.45, p = .017) in the Neg/Amb category web pages with a medium effect size (|d| = 0.53).

The adjusted logistic regression model showed that web pages belonging to the Neg/Amb category had significantly lower odds of having an institution as a source, and significantly higher odds of having at least one picture and being more readable, as measured by the GulpEase index (Table 3).

Table 3.

Multivariable logistic regression to predict the general tone of web pages analyzed.

Variable	aOR	95% CI	p
Website source (institutional vs non-institutional)	0.12	0.03–0.50	.004
N of pictures (1-unit increase)	1.91	1.05–3.48	.034
GulpEASE index (1-unit increase)	1.34	1.09–1.65	.006
% common words (1-unit increase)	1.01	0.88–1.16	.89

The terms vaccin*, squalene, influenza and adjuvant* were the most common words among both Pos/Neu and Neg/Amb web pages. However, relative to the most frequent word vaccin*, Neg/Amb pages used the root squalene about twice as much as Pos/Neu pages (ratios squalene to vaccin* of 0.41 and 0.23 for Neg/Amb and Pos/Neu web pages, respectively). Other particular features of Neg/Amb web pages included (i) higher weights for the words gulf, war, syndrome, soldier, adverse, minister, government; (ii) higher usage of registered trademarks(e.g. Novartis, Fluad®) and (iii) equal weight for the words health and death (Fig. 1).

Figure 1.

Tag clouds with 50 or more common words found among positive/neutral (A) and negative/ambiguous (B) web pages.

The most prevalent claims reported in Neg/Amb web pages were those relating squalene or squalene-based adjuvants to the Gulf War Syndrome (52.5%) and autoimmune pathologies (47.5%). About a third (35.0%) of web pages dealt with conspiracy theories around squalene and/or squalene-based adjuvants. Other less common claims included safety concerns (27.5%), direct link to deaths (20.0%), and efficacy concerns (5.0%) (Table 4). In addition, one web page claimed that squalene causes infertility, while another linked the use of squalene with autism.

Table 4.

Claims around squalene and/or squalene-based adjuvants found among negative/ambiguous web pages.

Myth	%	95% CI
Squalene is a toxic/unnatural substance	22.5	10.8–38.5
Efficacy of influenza vaccines with squalene-based adjuvants is not demonstrated	5.0	0.6–16.9
Too little/no safety data on squalene-based adjuvants are available; there are more risks than benefits regarding squalene-based adjuvants	27.5	14.6–43.9
Squalene and/or squalene-based adjuvants are/may be directly linked to deaths	20.0	9.1–35.6
Squalene and/or squalene-based adjuvants (may) cause autoimmune disorders	47.5	31.5–63.9
Squalene and/or squalene-based adjuvants (may) cause the so-called Gulf War Syndrome	52.5	36.1–68.5
Conspiracy theories around squalene and/or squalene-based adjuvants	35.0	20.6–51.7

Discussion

The main finding of the present study is that web pages critical to squalene-based adjuvants are highly prevalent on the Italian web: a typical internet user has an equal chance of finding reliable and unreliable information on squalene-based adjuvants. As shown by Google Trends, in the past 5 years web searches around squalene peaked in the period from 23 November to 6 December 2014 (results not shown) and thus correspond exactly to the time in which two batches of MF59®-adjuvanted vaccine were blocked in Italy as a precautionary measure. Considering Italy's relatively large current market share of MF59®-adjuvanted vaccine, the inadequate information on squalene may have contributed to the subsequently observed decrease in vaccination coverage. Proof of this may be found in the fact that, in a single year the vaccination coverage among the Italian elderly population decreased by 12.3% (influenza vaccination coverage among elderly Italians of 55.4% and 48.6% in the seasons 2013–14 and 2014–15, respectively). As we noted above, such situations undoubtedly have long-term consequences: from influenza season 2014–15 to 2015–16 the vaccination coverage increased by only 1.3 percentage points (somewhat like “easy to lose, hard to regain”).

Undoubtedly, unreliable information obtained from the internet, which is the main source of health-related information, contributed to the observed drop in vaccine uptake. Returning to the notorious Wakefield case, it has been shown that sensational news reports (both traditional and online) linking the MMR vaccine to autism, which often recounted the emotional and dramatic stories of parents, aroused greater interest than those which simply provided factual information from scientific studies. More recently, a situation somewhat similar to the Italian “Fluad case” arose in China; after some infant deaths had allegedly been linked to vaccination against hepatitis B, the vaccine in question was recalled by the authorities. Although no causal association was ever established, many parents refused to have their children vaccinated. Again, the primary source of parental information on the issue was the internet. Indeed, it was experimentally documented that people exposed to negative online information on a vaccine express less intention to be immunized.

We then established that the website source, level of readability, and number of images are all independent predictors of the general tone of squalene-related web pages. Non-institutional websites showed an 8.6-fold increase in belonging to the category of Neg/Amb pages. This finding was expected and it is in line with previous Italian research. In particular, all institutional websites analyzed by Poscia et al. were pro-vaccination, while Tozzi et al. have found that web pages from governmental agencies and universities had the highest level of credibility concerning human papillomavirus immunization.

Pos/Neu web pages had on average longer words and sentences that reduced their readability. Indeed, more than a quarter of them could even be judged to be difficult to read for people with a high school diploma, while no such pages were found in the Neg/Amb category. A similar pattern has been observed by Japanese researchers: in comparison with anti-influenza immunization online messages, those pro-influenza immunization were harder to read. More generally, the readability of squalene-related web pages is close to those of informed consent forms written in Italian (mean GulpEase index of 45.7).

Neg/Amb web pages combined different content forms more frequently. In particular, they used a relatively high number of images that could improve the intelligibility of the text since the denotative meaning of the term “squalene” is rather technical. According to a theory developed by Boles, words like “squalene” have a high level of concreteness (it is a “thing”) but low familiarity (extremely low frequency of word usage and exposure) and imageability (i.e. hard to visualize). Accompanying the text with pictures will undoubtedly improve imageability. In cases where the referential meaning of squalene was not given (and even if given this would not be fully comprehended by most users) or was twisted, a text containing emotionally charged images (such as needles and sharks with bloody teeth, as frequently found among Neg/Amb pages) may generate or reinforce negative connotations of squalene, adjuvants, and influenza vaccines in general among people unfamiliar to the topic. Notably, Kata has reported that images of scared needles are a frequent graphical attribute used by the anti-vaccination movement on the web.

From the point of view of word frequency, it emerged that Pos/Neu web pages had a relatively higher focus on vaccination itself and aspects relative to health. By contrast, Neg/Amb web pages exploited more the unfamiliar word “squalene”, politically-related words (conspiracy theories), names of major vaccine manufacturers and other registered trademarks (could be ascribable to the so-called “Big Pharma conspiracy theory)” and accentuated fatal risks (a high usage of the word “death”). The observed differences in quantitative language led us to analyze Neg/Amb pages in a more detailed way.

Despite the fact that: the etiology of Gulf War Syndrome/Illness remains largely uncertain to epidemiologists (it even has no universally adopted definition); most people have anti-squalene antibodies; and no Gulf War veterans received squalene-adjuvanted vaccine,, the thesis that squalene is a causative agent of the syndrome and other autoimmune disorders is highly prevalent. Indeed, 60% of Neg/Amb web pages made such claims. Other safety concerns were also frequently mentioned. To analyze and discuss the observed phenomena, it is useful to look at squalene-based adjuvants as “one more unnatural vaccine ingredient” and then make some parallels with preservatives and other additives used by the food industry. Consumers are very concerned about food additives, poorly informed about preservatives, colorants, and artificial sweeteners, and have difficulties in understanding the subject of food chemicals. Dickson-Spillmann et al. have shown that the risk perception of food chemicals was positively correlated with preference for natural food. By analogy, anti-vaccination activists are frequent supporters of herbalism, veganism, homeopathy, naturopathy and similar practices.

This study has both strengths and limitations. There are basically two strengths: one, the methodology of the study accurately mimics the typical online browsing and search behavior of members of the general public, and therefore, the web pages analyzed are highly representative of real life scenarios, and two, the multidimensional vision of the study outcome. In particular, throughout the Introduction to Discussion sections of this manuscript, we have highlighted issues around squalene-based adjuvants from the perspectives of linguistics, vaccinology and immunology, and public health. The main drawback is the subjective nature of our judgements. Obviously, the authors of this manuscript are experts in vaccinology; however, every effort was made to ensure that the investigators' personal attitudes towards the web pages analyzed did not bias the outcomes and conclusions of the study (e.g. by simplifying and standardizing the rules of classification, etc.). Undoubtedly, a 4-category classification of the general tone of the web page would have allowed us to provide a more detailed description of the information available. In this regard, the use of automatic analyzers of tone and emotion (enabling the variable to be expressed on a continuous or at least multi-category scale) may unveil other important associations, and thus should be considered in future research.

Conclusions

This study makes a comprehensive and multidiscipline analysis of the information available online to Italian internet users (about 75% of the Italian population) concerning squalene-based vaccine adjuvants. Negative, unbalanced, and misleading information was very common; this is potentially dangerous from a Public Health point of view. Unproven claims that influenza vaccines enhanced with squalene-based adjuvants may cause Gulf War Syndrome, autoimmune disorders, and even death, most likely originate from mass media reports, which are later reposted and/or re-elaborated to some degree by vaccine-critical websites, whose presence is high on the Italian web. In future, the mass media must be very cautious in choosing between legitimate concerns and sensationalism, in order to avoid such false alarms and the propagation of misinformation.

Undoubtedly, influenza vaccination must be an informed individual choice. In the web 2.0 era this choice is highly influenced by online resources. Indeed, more than 80% of US healthcare practitioners claim to have had at least one patient who brought internet-acquired health information to a visit in order to ask the physician's opinion on the matter in question. Considering the well-known phenomenon of patient-doctor information asymmetry, in cases where patients bring web-acquired material on squalene-based adjuvants, the role of physician becomes crucial in informing the patient about all benefits and risks of such vaccine formulations. Such provision of information by physicians will undoubtedly help to reassure those hesitant to be vaccinated. Governmental public health agencies and healthcare professionals need to provide reliable, easily accessible and user-friendly information on any given health technology. For instance, given the wide availability of readability formulas in common word processors, governmental public health agencies and healthcare professionals should make greater use of such tools in order to deliver material that is not only evidence-based but also well-targeted to its potential readers. Indeed, it has been demonstrated that an accurate revision of an informed consent form may on average increase the GulpEase index by 84%. Analogously, considering the niche nature of the topic of squalene-based adjuvants used in influenza vaccines, balanced evidence-based texts should be accompanied by neutral/positive and self-explanatory images, animations or, better still, video testimonials with relevant stakeholders or celebrities (indeed, testimonials of anti-vaccination celebrities are rather common among websites critical of immunization). Moreover, in our previous experience with a healthcare prevention-oriented mobile application and a sister website,, we realized that expert-based opinion on the content, usability and other quality attributes may differ from the expectations of lay users. Active collaboration between institutions and potential users, patient organizations, etc. would therefore help to deliver consumer-focused messages.

By contrast, authors contributing to non-institutional websites should communicate science in a more ethical way. Indeed, according to the Society of Professional Journalists' code of ethics, a journalist should “be accurate and fair”, “verify information before releasing it”, “take special care not to misrepresent or oversimplify… a story”, “…update and correct information”, “balance the public's need for information against potential harm”, “consider the long-term implications”, etc.

In our opinion, given that a growing body of literature suggests that the internet is a main driver of vaccination-related decision-making and the fact that we are living in the “post-truth era”, future studies should shift from purely observational to implementation research that is able to efficiently inform public policies.

To conclude, despite thousands of deaths avoided and millions of Euro spared during the last 20 years of squalene-adjuvanted influenza vaccine use, misinformation on this naturally occurring substance with an “unhappy” etymological origin makes it a victim of its name.

Methods

Search strategy and eligibility criteria

A set of queries containing a squalene-related plus influenza- and/or vaccine-related terms was created (LA and DA); this considered various semantically close terms, grammatical numbers, spellings and suggestions made by the Google autocomplete service. In all, 85 queries were created. No explicit Boolean operators (such as “AND” or “OR”) were used since most laypeople and healthcare professionals (HCPs) do not use/are unable to use these functions. We were aware that such a search strategy would have a low specificity and be more time-consuming, with queries being very similar and therefore having a large amount of duplicate results. We, however, opted for this methodology in order to align with the web searching behavior of the typical user and thus identify as much highly ranked squalene-related web pages as possible.

Google was used to locate web pages, because Google is the most widely used search engine in Italy with a market share of approximately 95%.

For each query the first 30 search results (first 3 pages) were consulted since most users do not go beyond this point., In total, 2,550 search results were screened. The search was performed over 4 consecutive days (from 9 to 12 January 2017).

Once duplicates were removed, the selected web pages were screened in order to identify those potentially eligible for analysis. The following inclusion criteria were applied: (i) page availability on the search day; (ii) pages written in Italian; (iii) influenza vaccines as one of the main topics of the web page; (iv) any reference to squalene in influenza vaccines (and not in cosmetics, foods etc.). The exclusion criteria were: (i) no textual information (e.g. only pictures); (ii) small amount of text (<150 words); (iii) highly technical material (e.g. articles published in scientific journals, operating protocols, manuals) targeting HCPs and scientists; (iv) forums, chatrooms and similar.

Analysis of web pages and text mining

Each included web page was analyzed by two investigators (LA and DA) each working independently. All web pages included were static (i.e. content did not change) during the period of retrieval. The general tone of an article, which was the main outcome of the present study, was a priori thought to be categorized as either positive (those approving of squalene-based adjuvants), neutral (those that neither approve nor disapprove of squalene-based adjuvants), negative (those disapproving of squalene-based adjuvants), or ambiguous (those containing both approving and disapproving messages). The definition of each category was discussed and comprehended by the two raters. However, in the phase of content analysis, distinguishing between the positive and neutral tones was still deemed subjective (Cohen's κ 0.60), therefore these two categories were combined for the purpose of analysis. Moreover, most web pages that could be potentially categorized as “ambiguous” reported some scientifically sound facts on squalene/squalene-based adjuvants, but at the same time raised suspicions regarding safety issues of such adjuvants (Cohen's κ 0.55). Following discussion inside the research team and after consultation with external experts, it was agreed that any uncertainty regarding squalene/squalene-based adjuvant safety issues would negatively affect laypeople's perception, and therefore, the “negative” and “ambiguous” categories should be combined. The re-categorization allowed us to have a perfect interrater agreement. A similar categorization (pro/neutral vs adverse) of websites on influenza vaccination has been previously reported. The applied dichotomous rule (i.e. Pos/Neu and Neg/Amb) in the web page categorization undoubtedly allowed us to increase the classification accuracy.

The top-level domain of each web page included for analysis was extracted and categorized as follows: .it, .com, .org, and “other”. Furthermore, web pages were dichotomized by website source into the following categories: (1) governmental agencies/institutions/universities/hospitals/local health units/medical centers/HCP societies (henceforth referred to as “institutional source”) and (2) online newspapers/news sources/informational portals/users' generated content (e.g. blogs) (henceforth referred to as “non-institutional source”). We assumed that the first category, being composed of institutions and therefore more “official”, would present more reliable information on squalene/squalene-based adjuvants. Indeed, it has been shown that the quality of Italian vaccination-related websites belonging to the government or professional associations is on average higher than that of blogs and anti-vaccination movement websites.

To assess the readability properties of the selected squalene-related online resources, the GulpEase index – specifically designed for the Italian language – was applied. The index considers the average number of characters per word, the average number of words per sentence, and is expressed as 89+[(300 × NSentence)–(10 × NLetter)/NWord]; a higher score implies greater readability. It is usually considered that texts with an index <40 would be difficult to read for people with a high school-level educational background. For the automatic readability assessment, which was carried out using Microsoft Word software, each selected article was preliminarily pre-treated in order to avoid misleading results. In particular, since identification of the sentence end performed by the automatic algorithm is triggered by the presence of punctuation marks relative to the end sentence (e.g. period, exclamation mark, question mark), such punctuation marks (eventually found in the middle of sentences, numbers, abbreviations) were deleted. In the same way, phrases without the end of sentence punctuation were also removed from the score calculation.

Other than text, content forms including images and video also quantified and were qualitatively described.

In the qualitative part of the analysis, the lexicon of Pos/Neu and Neg/Amb web pages was compared by constructing and visually inspecting tag clouds with 50 or more common words. Subsequently, we analyzed the content of Neg/Amb web pages in a more detailed way. In the pilot study, we retrieved the first 30 web pages using the search terms “squalene” and “influenza vaccines” in order to establish main scientifically unsound information on the squalene-based adjuvants in influenza vaccines. Six common myths were identified. Such flawed statements together with evidence-based information belying these myths are reported in Table 5.,, Moreover, some conspiracy theories, which are common among anti-vaccination activists, around squalene and/or squalene-based adjuvants were also present. In particular, in the pilot study we identified two such statements: (i) “politicians are personally immunized with non-adjuvanted vaccines, while laypeople are immunized with vaccines containing squalene” and (ii) “the only reason to include squalene in vaccines is for profit, by sparing the amount of antigen and selling adjuvanted vaccines for a higher price.”

Table 5.

Myths and facts about squalene and squalene-based adjuvants in influenza vaccines.

Myths	Facts	Ref.
Squalene is a toxic/unnatural substance	Squalene is a fully biocompatible and biodegradable substance, being the direct predecessor of cholesterol. About 1 g of squalene is synthetized daily in the human liver and is also introduced through the normal diet. A considerable amount of squalene can be found in human sebum and it is largely used by the cosmetic industry.	54,55
Efficacy of influenza vaccines with squalene-based adjuvants is not demonstrated	Several meta-analyses have shown that influenza vaccines containing squalene-based adjuvants (such as MF59®) are both immunogenic and efficacious in preventing influenza and its complications.	56-58
Too little/no safety data on squalene-based adjuvants are available; there are more risks than benefits regarding squalene-based adjuvants	Tens of clinical and observational studies as well as systematic reviews/meta-analyses have shown an acceptable safety profile of MF59®-adjuvanted influenza vaccine. Although in comparison with non-adjuvanted vaccines, there is some increase in the frequency of solicited adverse events (particularly local reactions), most of which are mild. On the other hand, the frequency of unsolicited and serious adverse events tend to be lower among recipients of MF59®-adjuvanted vaccine than among those receiving conventional influenza vaccines. Moreover, to date, MF59®-adjuvanted vaccine is licensed in about 30 countries and more than 100 million doses were administered worldwide from 1997 to date.	56,59
Squalene and/or squalene-based adjuvants are/may be directly linked to deaths	This myth probably comes from the so-called “Fluad case” when during the 2014–15 season 3 deaths occurred within 48 hours from vaccination with MF59®-adjuvanted influenza vaccine. As a result two batches of the vaccine were recalled as a precautionary measure. No causal link was later established. Moreover, it has been calculated that up to 20 Italian people die every day within 48 hours of vaccination by pure chance.	10,11
Squalene and/or squalene-based adjuvants (may) cause autoimmune disorders	In a large (N = 27,998) safety database, no significant difference in terms of adverse events of potential autoimmune origin between people immunized with MF59®-adjuvanted vaccine and those immunized with unadjuvanted vaccines has been found.	59
Squalene and/or squalene-based adjuvants (may) cause the so-called Gulf War Syndrome	In 2000, American researchers reported data on a high frequency of the presence of anti-squalene antibodies among veterans with the so-called Gulf War Syndrome. The methodology of that study was criticized and, according to the US Federal Government, vaccines administered to military personnel in that period of time did not contain squalene. Anti-squalene antibodies are very frequently detectable (at low titers) among people never immunized with vaccines containing squalene. No casual association between positivity to squalene and the development of GWS has been established. The MF59®-adjuvanted vaccine does not induce anti-squalene antibodies nor increase the quantity of pre-existing anti-squalene antibodies.	31-34,60,61

It should be noted that we did not analyze anything other than squalene-related information (i.e. general immunization-related claims made by anti-vaccination activists were not analyzed, nor were quality aspects of the selected web pages since these topics had been already extensively studied in both Italian and international contexts). Individuals interested in the common claims made by anti-vaccination websites may wish to study the work of Kata et al. The quality of Italian vaccination-related websites is assessed in work by Tafuri et al., while the quality of information concerning influenza prevention on Italian and English websites is described by Maki et al.

Statistical analysis

Approximately normally distributed variables were expressed as means with standard deviations (SDs), while ordinal as medians with ranges. Categorical variables were expressed as proportions with 95% confidence intervals (CIs). Differences in approximately normally distributed variables were evaluated by means of the t test, while those ordinal by means of the Mann-Whitney U test. Categorical variables were compared by means of χ2 or Fisher's exact test (whichever test was most appropriate). The effect size for normally distributed data was measured by means of Cohen's d, that for the Mann-Whitney U test – as r = z/√N, while that for 2 × 2 contingency tables – as φ = √χ2/N. Cohen's d was interpreted as small (0.2), medium (0.5), and large (0.8), while r and φ – as small (0.1), medium (0.3), and large (0.5).

To establish a statistical association between the main study outcome (general tone of a web page, where Neg/Amb is 1) and independent variables of interest, a multivariable logistic regression model was constructed. Variables showing an association with the general tone at α < 0.25 in the univariable analysis were included in the multivariable model.

All analyses were performed in the R environment.