Communications in the time of a pandemic: the readability of documents for public consumption.

OBJECTIVE: Government communications in a crisis can influence public health outcomes. This research aimed to investigate if written communications of the most commonly sought sources of COVID-19 information available on the internet have readability levels commensurate with those of the general public.
METHODS: Online documents from the World Health Organization (WHO), and the governments of Australia, the UK and the US were assessed for readability using an online instrument that calculated scores for the Flesch Reading Ease Score, the SMOG Index and the Readability Consensus Grade Level.
RESULTS: Similar to the previous research, most documents assessed had a readability standard that was at or above the recommended grade level, and as such inaccessible to substantial portions of the general public. A one-way ANOVA with post hoc tests revealed significant differences among the data, with Australian documents significantly more difficult to read than those from the UK and US.
CONCLUSIONS: Government departments need to consider their audience and monitor readability of the documents they produce to ensure that readers can understand them. Implications for public health: Health communications need to be written at a level appropriate for the targeted population in order to be fit for purpose.

The public expects government communication and intervention in public health issues. Efforts are often made for these interventions to be delivered to ‘hard‐to‐reach’ populations and for health education resources to be assessed for suitability for the target market. , Government public health communications seek to promote optimal health outcomes for often diverse populations.

This paper examines the readability of a sample of Australian, UK and US Government documents as well as those produced by the World Health Organization (WHO) in relation to the COVID‐19 pandemic. For communications about COVID‐19 to effectively reach their target audiences, they must be accessible to diverse populations with varying literacy levels, and therefore examining the readability level of such communications is important.

The literacy levels in the populations of Australia, the UK, and the US are noted, and previous research on the readability of health‐related materials is presented. As the majority of information sought is now accessible through the internet, internet usage is reported. The research method is outlined, followed by the data analyses processes. Results and discussion lead to a conclusion with recommendations for future public health communications.

Readability in the context of communications

Readability is described as “the quality of being easy and enjoyable to read” and readability formulae are described by McLaughlin (1969) as mathematical equations that provide “a measure of the difficulty experienced by people reading a given text, and a measure of the linguistic characteristics of that text”.

In the context of the COVID‐19 pandemic, during which information rapidly evolved, maintaining current and accurate communications with the public was important, and written communications played an important role in a potentially multi‐modal suite of initiatives. Recent American research reported that internet sources about COVID‐19 were not easy to read. The implications were that information that was not understood may lead to no information or inaccurate information being transmitted among vulnerable groups, resulting in increases in behaviours that could lead to unfavourable health outcomes. Written communications form part of a linear and uni‐directional communication process, which is vulnerable to misunderstanding, as no feedback is provided to the sender.

Literacy levels across nationalities

It cannot be assumed that the populations of Australia, the UK and the US have sufficient adult literacy skill attainment to comprehend complex health messages. Internationally many individuals have literacy levels below The (PIAAC) Level Three, the standard required for broad participation in work, education and training, and society. PIAAC data indicate that approximately 40% of the population of Australia, the UK and the US aged 16 to 65 years have difficulty with literacy, scoring below this level. Accordingly, attention must be given to producing health communications at a readability level that allows informed knowledge to be distributed, received and understood by the general population, especially during a crisis.

Lower levels of literacy are more evident in vulnerable groups of low socioeconomic status, which has a relationship with cognitive abilities that persists throughout the lifespan. Older age groups and vulnerable populations are reported as being at greater risk of serious illness with COVID‐19, , and also generally have lower levels of literacy.

Adult reading habits and abilities

Declines in adults’ reading habits have been noted even among tertiary students. Adult readers require connection to and perceived importance or relevance of reading material. Australian data from 2006 revealed an average reading for leisure time of 76 minutes per day, and women may read for longer than men. In a report on 2006 data, 43.7% of Australian respondents indicated reading daily and daily reading for pleasure reduces with age; indeed, declines in reading frequency and engagement over the lifespan begin prior to adulthood, while students are still at school. Recent Australian research investigated reading for pleasure in an adolescent population and reported that the percentage of daily readers reduced with age, with 50% of 15–16‐year‐olds not reading daily. Motivation to read is partly based on topic interest and comprehension of the material. , These data support the contention that to be read and understood, public health documents need to be relevant and accessible to readers.

The readability of internet health resources

As this research was conducted on internet‐based sources, it is relevant to consider internet usage in Australia, the UK and the US. In Australia, internet usage has increased since 2004 and 86% of households in 2017 had access. The proportion of internet users accessing health information increased from 22% in 2015 to 46% in 2018. An increasing amount of health information is available on the internet. In the UK in 2019, 91% of adults reported they had recently used the internet. Increasing numbers of older adults are accessing the internet, with increases from 2011 when 52% of users were aged over 65 years to 83% in 2019. Data from the US indicate that 82% of households have an internet subscription and this number has been consistently increasing.

Readability research on health‐related information has typically employed internet sources, and despite the standard of grade six in the US and year eight in Australia being suggested as the most appropriate level to promote comprehension across the population, information is not commonly produced at these grades. The proposed level of readability for the general population in the UK is 9 years of age (year four), which is the equivalent of year five in Australia and grade five in the US. Research on health communications report that most documents designed for public consumption are not easily read, , , with only two (10%) of the 20 websites reviewed in the easy range for readability. The readability of 18 medical questionnaires indicated that most of the instruments required a reading level above grade eight. Lack of comprehension may result in incorrect responses and subsequently may affect the treatment provided.

However, there have been attempts to improve the readability of health education materials, with reported improvements in a range of patient education materials for urology conditions, and readability scores for orthopaedic materials that match the general readability of the population. A number of other researchers have expressed concerns regarding the readability of health information documents.

The current research

This research is a desk analysis of COVID‐19 written information readily available through the internet. The data were documents available on the websites of The World Health Organization, the Australian Government, West Australian State Government, and government‐produced materials of the UK and US.

The research question is: How well do the written communications of the most commonly sought sources of COVID‐19 information available on the internet meet the readability requirements of the general public?

Method

This research has adopted a case study approach to the written online communications that have been produced in relation to COVID‐19. The focus of the research is on the readability of communications that are aimed at the general public. Documents available on the internet were the source data. A boundary of time has also been placed on the research. The first COVID‐19 case in Australia was recognised on 25 January 2020. Data collection occurred between May and June 2020. Documents did not need to be published during that period to warrant inclusion, and therefore some documents that were published earlier but were still accessible at the time of data collection were included in the research. A strength and limitation of this research is the rapid change in health communications advice to the public over this tumultuous period resulting in changes to the information available. This rapid turnover means that some of the documents included in this research may be changed or superseded and no longer accessible.

Instrument used for analysis

Analysis was conducted through a website that provides scores from seven different readability indices and an average readability score. This website includes scores from Flesch Reading Ease Score, Gunning Fog, Flesch‐Kincaid Grade Level, The Coleman‐Liau Index, The SMOG Index, Automated Readability Index, Linsear Write Formula, and Readability Consensus Grade Level. This latter measure provides an average of the first seven scores. Each of these instruments uses different formulae for calculating their readability scores.

This research focused on three commonly used scores: Flesch Reading Ease Score, The SMOG Index, and Readability Consensus Grade Level. A brief description of these measures and the rationale for their inclusion is provided below. While these instruments were originally developed to analyse printed documents, recent research has used them for analysis of internet‐based sources. , The reliability of online tools for assessing readability has been tested and the website employed in this research has been cited in peer‐reviewed publications. ,

Flesch Reading Ease Score

Developed in the 1940s by Rudolph Flesch, this readability calculation is based on average sentence length (number of words) and average word length (number of syllables). The resultant score ranges from 0 to 100 with a low score indicating greater reading difficulty. The formula is cited as [(0.39 × ASL) + (11.8 × ASW) – 15.59]. A document considered accessible to the general public would score 60 or more. This instrument has been criticised for its simplicity, which does not take comprehension into account. As shown in Table 1, the Flesch Reading Ease Scores are related to grade levels within the US education system. The US school grade levels cited are equivalent to those in the Australian education system, and UK schooling is one year ahead of Australian/US schooling. For example, an Australian student in year 5 would be in grade 5 in the US, but year 6 in the UK. The Flesch Reading Ease Score is commonly used in health literature readability research.

Table 1

Flesch Reading Ease Scores with US education level and USDHHS readability.

Flesch Reading Ease Score	US education level	USDHHS a readability rating
0–29 30–49 50–59	College graduate College 10th–12th Grade	Difficult
60–69 70–79	8th–9th Grade 7th Grade	Average
80–89 90–100	6th Grade 5th Grade	Easy

Note:

a: United States Department of Health and Human Services – these categories are consistently referred to in a number of research papers (for example, Edmunds et al., 2014 ).

USDHHS readability rating

The SMOG Index

The Simple Measure of Gobbledygook (SMOG) was developed in 1969. This formula may offer advantages over the Flesch Reading Ease Score as it more accurately assesses likely comprehension of the material being tested. The SMOG was designed to measure complete comprehension; whereas, other readability formula only measure partial comprehension. To calculate a reading grade in SMOG, one counts the number of words with three or more syllables across three ten‐sentence samples. Then one calculates the square root of that total and adds three to the result. Like the Flesch Reading Ease Score, the SMOG has been used in readability research in the health industry. , The SMOG Index has been employed in this research as it is recommended by the Cochrane Collaboration.

The Readability Consensus Grade Level

This measure is based on the average results from seven well known and used instruments (detailed above). The Readability Consensus Grade Level provides an easy‐to‐understand measure and has been employed in previous research. , , The use of multiple measures for assessing readability and averaging them is supported in the literature. Each instrument has strengths and weaknesses. The Flesch Reading Ease Score is the least conservative and SMOG the most conservative at scoring. , This is due to the SMOG being based on 100% comprehension. Each of the measures that are calculated in the Readability Consensus Grade Level has different formulae. As detailed above, the Flesch Reading Ease Scale formula is RE = 206.835 – (1.015 × ASL) – (84.6 × ASW), where RE = Readability Ease; ASL = Average Sentence Length (i.e. the number of words divided by the number of sentences) and ASW = Average number of syllables per word (i.e. the number of syllables divided by the number of words). The SMOG has extensive instructions that result in the counting of words with more than three syllables and the calculation of a score that provides a SMOG grade = 3 + Square Root of Polysyllable Count. Clearly, there are other parameters in a document that are not measured by these formulae.

Table 2 shows an explanation of the grade levels in the Readability Consensus Grade Level.

Table 2

Readability consensus explanations.

Grade Level	Readability	Age of Grade Level
8	Standard/average	12 – 14 years (7th & 8th grade)
10/ 11	Difficult to read	14 – 15 years (9th & 10th grade)
12	Fairly difficult to read	17 – 18 years (12th grade)
13	Difficult to read	18 – 19 years (college level entry)
14	Difficult to read	21 – 22 years (college level)
16 / 26	Very difficult to read	College graduate

Note:

a:This table only includes the explanation for the Grade Levels that were located in the documents assessed.

Procedure

Since the focus of this research is on health communications created for consumption by the general public, the researchers employed a Google search for “coronavirus” and a separate search for “COVID‐19”. The top three searches on Google related to the Australian Government (Federal) websites, the World Health Organization website, and the West Australian State Government website. The latter site would have appeared as a result of the location of the researchers in Western Australia. As COVID‐19 is an international pandemic, the researchers explored similar sites in the UK and the US as part of the research. Accordingly, the UK Government website and US site specifically for COVID‐19 were included.

Documents and information targeted to the general public were examined. Sampling was purposive. Documents were identified, downloaded and – if necessary – copied to a Word document. They were then analysed, and the scores were entered into a Statistical Package for the Social Sciences (SPSS) file. Headers and footers were deleted; however, all other aspects of the document were left intact. The website employed to calculate the readability required a minimum of 100 words and a maximum of 3000 words; however, in most cases, the full document was employed as they were less than the maximum words. Documents varied in size and addressed a range of topics in relation to COVID‐19. Where a long document was located, only the first page was employed as data. Links on front page documents to other documents were followed; however, if these second‐level documents included further links documents at this ‘third level’ were not accessed. The rationale for this was that if the previous page was difficult to read, then it was unlikely that a reader would proceed further.

Sample and sample size

A total of 52 documents were accessed and reviewed. Data were extracted from the websites of the World Health Organization, the Australian Government, the Western Australian State Government, the UK Government and the US Government.

Data analysis

As previously detailed, analysis was conducted through a website that provides scores for readability. The readability of documents is presented in grade levels (generally based on grade levels from the US and equivalent to year levels in Australia, as previously described).

Results

Results were examined for four groups of documents: WHO, Australia, UK, and US. Descriptive data for each of the three scores are shown in Table 3. Although there are statistical limitations on the use of comparative statistics with small groups, a one‐way ANOVA was conducted for each measure to indicate any group differences. For all measures, the ANOVA was significant with Flesch Reading Ease Score, F(df 3, 48) = 7.32, p = 0.000; SMOG, F(df 3,48) = 6.33, p = 0.001; Readability Consensus Grade Level, F(df 3,48) = 3.291, p = 0.028. Dunnett's T3 post hoc tests indicated that significant differences were present between Australia and the UK, and Australia and the US, across all three measures.

Table 3

Range and Mean Readability scores across all documents reviewed.

		Flesch Reading Ease Score	The SMOG Index	Readability Consensus Grade Level a
WHO (n = 10)	Range Mean	27.7–63 45.6	6.8–15.2 10.4	8–18 12
The Australian Government [incl. WA] (n= 22)	Range Mean	3.3–55.9 35.4	9.8–18.9 12.8	10–26 14
UK Government (n = 10)	Range Mean	46–58.4 53.7	9.3–11.7 10.3	10–13 11
USA COVID‐19 (n=10)	Range Mean	32.6–64.5 53.1	7.3–13.4 9.5	8–16 11
All documents (n=52)	Range Mean	3.3–63 44.3	6.8–18.9 11.2	8–26 13

Note:

a: This readability consensus is based on seven different scales and its aspects are shown in Table 2. Decimal places have been excluded, rounded to whole figures.

Readability Consensus Grade Level

If assessed using the USSDHHS classifications (see Table 1), only two documents were considered average using the Flesch Reading Ease Scores (4%), and four documents using the SMOG Index (8%). No document was classified as easy to read.

Discussion

This research was designed to investigate the readability of written COVID‐19 communications available on the internet aimed at consumption by the general public. Online calculations were employed to provide scores for Flesch Reading Ease Scores, the SMOG Index and Readability Consensus Grade Level.

Based on their readability scores, only two of the 52 documents analysed were considered to be accessible to the Australian and US general population, and none to the UK community. Both documents were from the WHO website, and therefore none of the government sources was accessible. All other documents were above this standard and categorised as difficult to read. Employing the USDHHS categorisation, 4% of the documents assessed with the Flesch Reading Ease Score were ‘average’ in their readability and none were easy to read. Slightly different results were found for the SMOG Index with 8% of documents being categorised as ‘average’. Differences between these readability scores have been identified in other research. , Although the data suggested the UK documents were a little easier to read according to the mean Flesch Ease Reading Score (53.7), documents were all well above the standard of grade five as set by the UK Government.

These results are similar to previous research findings for readability research relating to health documents, with some reporting no ‘easy’ reads, and 4% ‘average’ reads. Other researchers indicated that the materials that they assessed were in the difficult range. SMOG analyses revealed no documents in the ‘easy to read’ category, 1% in the average category, 10% in the difficult category and 89% required a reading grade greater than grade 12. Another paper indicated that five from 70 (7%) documents assessed were in the ‘easy to read’ category.

The statistical analysis of the documents through a one‐way ANOVA with a post hoc Dunnett's T3 test found significant differences in the mean scores of Australia compared to both the UK and the US. The Australian scores indicated significantly more difficult readability. However, the Australian data were not significantly different from WHO data; and WHO data were not significantly different to either the UK or the US data.

This research, based within a communications theory framework, recognises that readability is only part of the whole communication remitted, and individuals with low levels of literacy may receive information of potentially varying quality from other potentially multi‐modal sources such as television, radio and social media. As such, key implications of our findings include a need for further future research that explores how multi‐modal texts (not just written texts) may communicate health messages, and how different textual features employ graphic, video and audio elements to facilitate the transmission of these messages. Particular attention should be given to ease of consumption of these text types in low‐literacy communities. As an extension relevant to this readability work, future research could also focus on how degree of readability and associated reading comprehension influences how received messages are operationalised and incorporated into attitudes and practices in target groups.

Limitations of the research

This research considered written words that are not the only source of information for the public who may access other audio and visual communications. However, despite these available communications, written information may be more effective at maintaining accuracy over verbal message transmission, unless that verbal message is provided by accessible audio and/or video and is from a known and reliable source.

The limitations of instruments such as the Flesch Reading Ease Score have been acknowledged in the academic literature, with suggestions that more complex instruments that take account of visual representations should be developed. Additionally, a focus on understanding the content is required, an aspect that can be missing from the formulae employed in current research. The use of visuals was employed by the WHO and the West Australian State Government on their website to support the written material. The analysis of visual representations was outside the scope of this research.

While there have been efforts in some fields to improve health communication readability, findings in this paper suggest that attention must be given to enhancing the readability of health communications produced by governments during a pandemic. Despite the considerable presence in the media by government officials who have presented verbal information regularly to the public, the level of the written information, which is expected to be more in‐depth and a more permanent source of information, is not accessible to a large proportion of the populations across the three countries. Readability levels can have a detrimental impact on a population's understanding of a rapidly evolving situation. As approximately 40% of each of the populations have literacy levels below those considered necessary to function in today's society, these data suggest that governments have failed to meet the needs of a large number of their people, particularly the vulnerable within communities, putting their health outcomes at risk.

Recommendations and implications for public health

On the basis of the results of this research, we recommend the following.

Government departments and other organisations responsible for the communication of public health measures should familiarise themselves with the diverse reading abilities of their populations. This means that familiarity with readability tools is required, and the employment of literacy experts to assist them in the communication of health messages in emergency situations such as pandemics is appropriate. Documents aimed specifically at vulnerable communities within the population must consider the literacy abilities of these groups. Documents posted on the internet should be assessed for readability and presented ideally at a grade 6 (US), year 6 (Australia) or year 5 (UK) level to increase accessibility across the population. Finally, feedback on readability should be sought from vulnerable populations so that the transmission of information is no longer uni‐directional, and the efficacy of communication can be evaluated to enhance the readability of future communications.

Conclusion

The results found in this research indicate that government departments responsible for public health information do not currently achieve the goal of making the communications available to the majority of the population, and that this is an international issue with similar results from Australia, the UK and the US. To enhance the accessibility of health‐related communications to vulnerable populations, greater efforts need to be made to enhance the readability of these documents so that they are fit for purpose.