A readability comparison of anti- versus pro-influenza vaccination online messages in Japan.

Historically, anti-vaccination sentiment has existed in many populations. Mass media plays a large role in disseminating and sensationalizing vaccine objections, especially via the medium of the Internet. Based on studies of processing fluency, we assumed that anti-influenza vaccination online messages to be more readable and more fluently processed than pro-influenza vaccination online messages, which may consequently sway the opinions of some audiences. The aim of this study was to compare readability of anti- and pro-influenza vaccination online messages in Japan using a measure of readability. Web searches were conducted at the end of August 2016 using two major Japanese search engines (Google.jp and Yahoo!.jp). The included websites were classified as "anti", "pro", or "both" depending on the claims, and "health professional" or "non-health professional" depending on the writers' expertise. Readability was determined using a validated measure of Japanese readability (the Japanese sentence difficulty discrimination system). Readability of "health professional" websites was compared with that of "non-health professional" websites, and readability of "anti" websites was compared with that of "pro" websites, using the t-test. From a total of 145 websites, the online messages written by non-health professionals were significantly easier to read than those written by health professionals (p = 0.002, Cohen's d = 0.54). Anti-influenza vaccination messages were significantly easier to read than pro-influenza vaccination messages (p < 0.001, Cohen's d = 0.74). When health professionals prepare pro-influenza vaccination materials for publication online, we recommend they check for readability using readability assessment tools and improve the text for easy reading if necessary.

Introduction

Influenza vaccinations are the most effective measure for reducing susceptibility and potentially serious influenza-related complications (World Health Organization, 2014). However, influenza immunization rates remain less than optimal in Japan (Nobuhara et al., 2014). Although receiving influenza vaccination is recommended by the Japanese government under the Preventive Vaccinations Law, influenza vaccination is not given at school or in the workplace. Individuals who seek influenza vaccination visit at doctors and pay one's own expense (excluding people over 65 years of age and disabled persons).

Additionally, anti-vaccination sentiment, which includes doubt, fear, and opposition to vaccination, has been in existence (Blume, 2006, Dubé et al., 2015); it had a demonstrable impact on vaccination policies, individuals, and community health (Poland and Jacobson, 2001). Anti-vaccination messages are especially more common on the Internet than in other forms of media (Davies et al., 2002). Vaccination is one of the topics that individuals consult the internet for information and/or advice. (Betsch et al., 2012). These individuals can easily encounter anti-vaccination online messages because search engines list not only pro-vaccination messages but also anti-vaccination messages (Betsch et al., 2012, Davies et al., 2002, Jones et al., 2012). Influenza vaccination in Japan is no exception. Anti-influenza vaccination activists, who are mostly self-proclaimed specialists lacking specialized knowledge and lay people, propagate on the internet that influenza vaccine has little or no efficacy but a high risk of side effects, and that influenza is not a serious disease for which preventive intervention is required (Hirota and Kaji, 2008).

Internet usage statistics show approximately 91% of Japanese, 92% of the British, and 87% of Americans are regular users of the Internet (Internet World Stats, 2001). Online health information seeking is becoming a recurrent activity in people's everyday lives (Fox and Duggan, 2013, Hesse et al., 2005, Lustria, 2007). Of the internet users, 70% say the information they encounter online influences their treatment decisions (Fox and Rainie, 2000). Further, over half (52%) of users believe “almost all” or “most” information on health websites is credible (Fox and Rainie, 2000). If these naïve individuals encounter anti-vaccination online messages, they may accept them and decide not to receive influenza vaccination. Thus, anti-vaccination online messages may increase the number of people not seeking influenza vaccination based on misleading online messages (Hesse et al., 2005).

In the present study, we focus on the readability of influenza vaccination online messages on the assumption that it plays an important role in their acceptability. “Readability” is defined as “the determination by systemic formulae of the reading comprehension level a person must have to understand written materials” (Albright et al., 1996). Text that is “readable” makes information more accessible and useful by improving comprehension, learning, and retention (Klare, 1963). Further, easy-to-read text is more liked and trusted (Schwarz, 2004, Alter and Oppenheimer, 2009), and generates a more favorable reader attitude (Claypool et al., 2015) than difficult-to-read text, according to studies of processing fluency. Processing fluency refers to the ease or difficulty with which new, external information can be processed (Schwarz and Clore, 2007). Human judgment is influenced by not only the content of thoughts but also the metacognitive experience of processing those thoughts (Flavell, 1979, Tversky and Kahneman, 1973). Processing fluency is one metacognitive cue that plays an important role in human judgment (Alter and Oppenheimer, 2009, Schwarz and Clore, 2007). Claypool et al. (2015) argued in their review of fluency studies that fluently processed stimuli make attitudes more favorable.

For example, Rennekamp (2012) created more readable and less readable versions of a financial report by manipulating such factors as sentence length, simplicity of terms, and ease of syntax. After reading the fluent or dysfluent report, participants answered that they were more likely to feel they could trust a fluent report than dysfluent one. Similarly, if vaccine-related online messages are easy to read with simple words, syntaxes, and short sentences, readers may trust them more, and may have more favorable attitude to them. The reverse also applies.

Regrettably, studies indicate that health-care professionals often use jargon that are unfamiliar to health-care users (Byrne and Edeani, 1984, Castro et al., 2007, Ley, 1998), and that health information is often written at readability levels that are too high for the majority of the intended recipients (Rudd et al., 2000); such information is assumed to be dysfluently processed. In such cases, more information may cause target subjects to feel distrust and disfavor rather than empowered, especially for those who have limited health literacy. The Healthy People 2010 report reminds public health communicators to pay special attention to health literacy, defined as “the degree to which individuals have the capacity to obtain, process, and understand basic health information” (U.S. Department of Health and Human Services, 2000). In the field of health literacy, accessibility and appropriateness of health communication have generally been discussed in terms of readability (Nielsen-Bohlman et al., 2004). Thus, paying attention to readability of health information is crucial.

We aimed to compare readability of anti- and pro-influenza vaccination online messages in Japan using a measure of readability. We proposed two hypotheses.

Influenza vaccination related online messages written by non-health professionals are easier to read than those written by health professionals according to a measure of readability.

Anti-influenza vaccination online messages are easier to read than pro-influenza vaccination online messages according to a measure of readability.

Methods

Material collection

Web searches were conducted at the end of August 2016 using the search formula in Japanese input into Google.jp and Yahoo!.jp (the Japanese version of the search engine); influenza AND (vaccine OR inoculation); influenza AND (vaccine OR inoculation) AND (danger OR dangerous); (objection OR effect); (necessary OR unnecessary); (efficacious OR inefficacious); (approval OR disapproval); (receive OR “not receive”); (benefit OR risk). The terms such as “danger”, “dangerous”, “objection”, “unnecessary” and “inefficacious” were included in these formulae for gathering anti-influenza vaccination online messages because only the term “influenza AND (vaccine OR inoculation)” did not gather a sufficient number of anti-messages for examination, and because individuals who doubt about the necessity of influenza vaccination seem to search online using those terms. Google and Yahoo! were chosen because they are the most popular search engines in Japan, accounting for approximately 66% and 29% of all Internet searches respectably (StatCounter Global Stats, 1999). Top 100 results were reviewed for each search formula.

After excluding duplication, results were included for analysis if they did not meet any of the following exclusion criteria: (1) bulletin board system or listserv or newsgroup pages or twitter; (2) pages solely containing brief notices about other website content; (3) video results; (4) non-Japanese websites; (5) inactive links; (6) online messages exclusively about veterinary vaccination; (7) online messages exclusively about vaccination other than influenza (e.g., combination vaccines for children, cervical cancer); (8) online messages without any claims of anti- or pro-vaccination (e.g., cites exclusively about expenses, side reactions and cautions). The URLs of the included materials were recorded in a Microsoft Excel 2016 spreadsheet.

Material classification

The included materials were independently classified as “pro”, “anti” or “both” by two raters: one of the authors (T.O.) and a trained rater. Each was blinded to the other's ratings. Materials that recommended readers receive influenza vaccination were classified as “pro,” and materials that opposed/objected were classified as “anti.” Materials that referred to both claims of pro- and anti-HPV vaccination, but did not state their own assertion, were classified as “both” (see Appendix for coding guidelines). This classification generated six categories: identical classifications between raters (pro-pro, both-both, and anti-anti) and disagreeing classifications (pro-both, anti-both, and pro-anti). Then, pro-pro and pro-both materials were labeled as “pro”; anti-anti and anti-both as “anti”; and both-both as “both.” Materials of seemingly disagreeing classifications, pro-anti, were set aside for discussion on agreement by the raters.

“Pro” and “anti” materials were classified into seven categories depending on the authors' professional expertise: “pro” materials written by individual physicians, nurses, pharmacists, or researchers; “pro” materials written by organizations such as the Ministry of Health, Labour and Welfare of Japan, research centers, pharmaceutical companies; “pro” materials written by authors of news sites; “pro” materials written by laypeople; “anti” materials written by individual physicians or pharmacists; “anti” materials written by authors of news sites; and “anti” materials written by laypeople and alternative therapists.

Then, these seven categories were classified as “health professional” or “non-health professional” depending on the authors' health expertise. We classified materials written by individual physicians, nurses etc. as “health professional”. We also classified materials written by organizations such as the Ministry of Health, Labour and Welfare of Japan, research centers, pharmaceutical companies as “health professional” because those materials were usually under editorial supervision by health professionals such as physicians. We classified materials written by authors of news sites as “non-health professional” because the authors were not medical journalists but writers of general topics. We combined materials written by alternative therapists and laypeople, and classified them as “non-health professional” because we found only two materials by alternative therapists and they were self-proclaimed massager without any qualifications.

This classification generated four categories: “pro” materials written by health professionals; “pro” materials written by non-health professionals; “anti” materials written by health professionals; and “anti” materials written by non-health professionals.

Readability assessment

The materials were edited by removing any URLs and quotations written in English before readability assessment to ensure accurate measurement of Japanese readability. When the included materials comprised multiple pages and included topics other than influenza vaccination, only the relevant pages were assessed by measures of readability.

A validated measure of Japanese readability, which was named “the Japanese sentence difficulty discrimination system” (JSDDS) (Lee and Hasebe, 2013) was used in the present study. The JSDDS is the most authentic validated measure of Japanese readability. The JSDDS calculates Japanese readability on the basis of average length of sentences, difficulty level of words, and proportion of grammatical parts of speech and types of characters per sentences (Lee, 2011). Scores range from 0.5 to 6.4. The higher the score is, the relatively easier the text is to read; 5.5–6.4, very easy; 4.5–5.4, easy; 3.5–4.4, neutral; 2.5–3.4, a little difficult; 1.5–2.4, difficult; 0.5–1.4, very difficult. Score determination was performed using the JSDDS online tool (Lee and Hasebe, 2013).

Statistical analysis

The kappa statistic was used to measure interrater agreement. Distributions and mean readability scores by categories were calculated. Readability of “health professional” materials was compared with that of “non-health professional” materials, and readability of “anti” materials was compared with that of “pro” materials, using the t-test. The size of the effect was estimated by using Cohen's d (Cohen, 1988). Cohen's d is calculated as the difference between 2 means divided by the pooled standard deviation. An effect size of 0.20 is considered small, 0.50 medium and 0.80 or above large (Durlak, 2009, Sun et al., 2010).

Additionally, because online health information seekers usually examine the first 10 search results (Eysenbach and Kohler, 2002), readability of “anti” materials were compared with that of “pro” materials on higher search results, i.e., the materials that were included from the first 10, 20, and 30 search results of all search formulae. P-values were set at 0.05 for all statistical tests. All statistical analyses were performed using SPSS version 21.0 (Chicago, IL, USA).

Results

In total, 145 materials were included. Most were websites or blogs except for two independent Facebook pages. In the present study, websites, blogs, and Facebook pages were labeled collectively as “websites”.

The numbers of materials classified as per their claims were as follows: pro-pro, 87; pro-both, 3; both-both, 9; anti-both, 2; and anti-anti, 44. A weighted kappa coefficient of 0.96 indicated almost perfect interrater agreement.

Up to 46 websites (31.8%) propagated anti-influenza vaccination messages, 90 websites (62%) propagated pro-influenza vaccination messages, and nine websites (6.1%) were “both” (see Table 1). Of pro-influenza vaccination websites, 49 websites (33.8%) were written by health professionals and 41 websites (28.3%) were written by non-health professionals. Of anti-influenza vaccination websites, 42 websites (29%) were written by non-health professionals and four websites (2.8%) were written by health professionals.

Table 1

Distribution of websites by category.

Category	n (%)
Pro-HPV vaccination	90 (62.0)
Anti-HPV vaccination	46 (31.8)
Both	9 (6.1)
Pro by health professionals	49 (33.8)
Physician etc.	31 (21.4)
Organization	18 (12.4)
Pro by non-health professionals	41 (28.3)
News site	17 (11.7)
Laypeople	24 (16.6)
Anti by health professionals
Physician etc.	4 (2.8)
Anti by non-health professionals	42 (29.0)
News site	4 (2.8)
Laypeople	38 (26.2)

The mean readability of 145 total results was 2.93 (SD = 0.55). According to the JSDDS, the number of “very easy” and “very difficult” material was zero in both pro- and anti-influenza vaccination websites (Table 2). One anti-vaccination website was “easy”. “Neutral” materials were 6 (6.7%) in pro- and 11 (23.9%) in anti- vaccination websites. “Little difficult” materials were about 60% both in pro- and anti-vaccination websites. “Difficult” materials were 27 (30.0%) in pro- and 6 (13.0%) in anti-vaccination websites.

Table 2

Distribution of websites by readability level.

	n (%)
	Pro	Anti	Both
Very easy	0	0	0
Easy	0	1 (2.2)	0
Neutral	6 (6.7)	11 (23.9)	1 (11.1)
Little difficult	57 (63.3)	28 (60.9)	7 (77.8)
Difficult	27 (30.0)	6 (13.0)	1 (11.1)
Very difficult	0	0	0

Table 3 shows mean readability comparing “health professional” websites with “non-health professional” websites. Of pro-influenza vaccination websites and total 145 websites, the online messages written by non-health professionals were significantly easier to read than the messages written or editorially supervised by health professionals (2.91 VS 2.69, p = 0.035, Cohen's d = 0.44; 3.04 VS 2.75, p = 0.002, Cohen's d = 0.54). We found no significant difference of readability between materials by non-health professionals and by health professionals in “anti” and “both” categories (3.16 VS 3.46, p = 0.339; 3.06 VS 2.63, p = 0.427).

Table 3

Mean readability of websites: health professional vs non-health professional.

	Non-health professional		Health professional
	M	SD	M	SD	p	d
Anti	n = 42		n = 4
	3.16	0.54	3.46	0.91	0.339	0.51
Pro	n = 41		n = 49
	2.91	0.42	2.69	0.55	0.035⁎	0.44
Both	n = 8		n = 1
	3.06	0.48	2.63	–	0.427	0.9
Total	n = 91		n = 54
	3.04	0.50	2.75	0.60	0.002⁎	0.54

p < 0.05.

Table 4 shows mean readability comparing anti-influenza vaccination websites with pro-influenza vaccination websites. Anti-influenza vaccination messages were significantly easier to read than pro-influenza vaccination messages in the websites written by health professionals, by non-health professionals, and total 136 websites (3.46 VS 2.69, p = 0.015, Cohen's d = 0.31; 3.16 VS 2.91, p = 0.002, Cohen's d = 0.51; 3.19 VS 2.79, p < 0.001, Cohen's d = 0.74).

Table 4

Mean readability of websites: anti-vaccination vs pro-vaccination.

	Anti		Pro
	M	SD	M	SD	p	d
Health professional	n = 4		n = 49
	3.46	0.91	2.69	0.55	0.015⁎	0.31
Non-health professional	n = 42		n = 41
	3.16	0.54	2.91	0.42	0.002⁎	0.51
Total	n = 46		n = 90
	3.19	0.58	2.79	0.51	0.000⁎	0.74

p < 0.05.

Table 5 shows mean readability of websites on higher search results. When the materials that were included from the first 10, 20, and 30 search results of all search formulae were measured, anti-influenza vaccination messages were significantly easier to read than pro-influenza vaccination messages (3.53 VS 2.79, p = 0.004, Cohen's d = 1.57; 3.33 VS 2.87, p = 0.003, Cohen's d = 0.92; 3.25 VS 2.89, p = 0.009, Cohen's d = 0.7).

Table 5

Mean readability of websites for the higher search results.

	Anti		Pro
Search results	M	SD	M	SD	p	d
The first 10	n = 6		n = 15
	3.53	0.57	2.79	0.43	0.004⁎	1.57
The first 20	n = 18		n = 34
	3.33	0.49	2.87	0.51	0.003⁎	0.92
The first 30	n = 23		n = 42
	3.25	0.48	2.89	0.52	0.009⁎	0.7

p < 0.05.

Discussion

In the present study, we compared readability of anti- and pro-influenza vaccination online messages in Japan using a measure of readability. We found more difficult to read materials in pro- than anti- influenza vaccination websites in combination with “little difficult” and “difficult” to read materials according to the JSDDS (93.3% VS 73.9%). Although different scales and criteria were used, this result indicates that we found more difficult to read pro-vaccination websites than the study by Sak et al. (2016) (readability score equal or above the 10th grade levels; 79.2% of pro- VS 76.2% of anti-vaccination sites). We also found that the influenza vaccination-related online messages written by non-health professionals were significantly easier to read than the messages written or editorially supervised by health professionals in pro-vaccination websites and total websites examined. Additionally, we found that anti-influenza vaccination online messages were significantly easier to read than pro-influenza vaccination messages regardless of writers' health expertise. Thus, our hypotheses were partially supported by the study results.

In more details, medium sized effects were presented in comparison between the readability of materials by non-health professionals and professionals in pro-vaccination websites and total websites examined. Previous studies indicate that health professionals often use jargons (Byrne and Edeani, 1984, Castro et al., 2007, Ley, 1998) and that the materials written by health professionals are often difficult to read for the intended recipients (Rudd et al., 2000). Such tendencies of health professionals may have contributed to our finding in the present study.

In anti-vaccination websites, the materials by health professionals were easier to read than ones by non-health professionals. This finding was converse to our hypothesis. However, this difference was not statistically significant. Further, the number of materials by health professionals was small; only four. These four websites may have been easy to read by chance.

In comparison the readability between pro- and anti-influenza vaccination websites, medium sized effects were presented in materials by health professionals, non-health professionals and total materials examined. The reason that pro-vaccination websites were more difficult to read may have been because they explained the efficacy of influenza vaccination using technical concepts such as herd immunity and statistics. Accordingly, these sentences may have tended to be longer with more difficult words and more complex syntaxes.

Conversely, most of anti-vaccination websites were written by laypeople. These lay people may have used less jargons. Additionally, they may have explained the unnecessity of influenza vaccination by simpler concepts; e.g., “I have not received vaccination for years. But I have not catch the flu.” “Even if I/you catch the flu, sleep will cure.” These concepts may have tended to be expressed with shorter sentences, simpler words and syntaxes. Whatever the reasons are, there is a concern that those easy-to-read anti-influenza vaccination websites are more accessible and liked due to their experienced fluent processing (Klare, 1963, Alter and Oppenheimer, 2009, Claypool et al., 2015) than difficult-to-read pro-vaccination websites.

Online-health-information seekers refer to content in listed order from highest search results (Eysenbach and Kohler, 2002). In the present study, difference of readability between anti- and pro-vaccination messages was significant on higher search results (the first 10, 20, and 30). This is important because the easiness to read and the experienced fluent processing of anti-influenza vaccination messages may influence judgement of online-health-information seekers from the beginning of their web searches.

We recognize that the present study has limitations. First, there are major limitations to using measures of readability. These tools measure only word difficulty and sentence length and complications. They do not assess the impact of factors such as font size, color, white space, and illustrations, which may influence processing fluency. However, readability is an important factor to make health information more accessible and useful (Klare, 1963, Nielsen-Bohlman et al., 2004). Therefore, assessing readability of vaccination-related online messages is significant.

Second, although an extensive number of websites (n = 145) were selected for analysis, it was not feasible to examine the universe of sites for reasons of availability, access, and time. This is especially difficult consideration when studying internet web pages because new websites are created each day, while old websites become inaccessible.

Third, only Japanese-language websites were included in the study. To generalize the results of this study to other countries, replications of this study are needed on websites that are written in languages other than Japanese. Fourth, although we classified materials written by medical organizations (e.g., pharmaceutical company) as “health professionals”, some of them may not have been written or supervised by health professionals. Finally, future study needs to investigate whether readability of vaccine-related online messages influences attitude, intention, and behavior of readers in reality.

Conclusions

Paying attention to readability of influenza vaccination related materials is crucial. When health professionals prepare pro-influenza vaccination materials for publication online, we recommend they check message readability using the measures of readability. The JSDDS (Lee and Hasebe, 2013) shows the difficulty level for each word that is used in texts by colored lettering on a screen. Users can improve the texts and make them easier to read by rewriting difficult words into easy words. Additionally, long sentences are found at a glance, displaying all sentences one by one in parallel. Users can improve the texts by shortening the long sentences. Other commonly used readability analysis tools such as the Flesch–Kincaid Grade Level, Flesch Reading Ease scale, Simple Measure of Gobbledygook, and Gunning fog index can also be used to revise readability (Ley and Florio, 1996, Beaunoyer et al., 2016). It is important to write and disseminate easy to read pro-influenza vaccination messages on the Internet for promoting influenza vaccination.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of interests

None declared.

Pro	• The website concludes that individuals should receive influenza vaccination. • Even if the conclusion is not stated, it is obvious that the author of the website recommends individuals to receive influenza vaccination.
Anti	• The website concludes that individuals should not receive influenza vaccination or that influenza vaccination is not necessary. • Even if the conclusion is not stated, it is obvious that the author of the website asserts that individuals should not receive influenza vaccination or that influenza vaccination is not necessary.
Both	• The website includes both assertions of pro- and anti-influenza vaccination (e.g., benefits and risks, necessity and unnecessity). • The website does not state their own assertion or does leave a decision for receiving influenza vaccination to readers.