Assessing the reading level of online resources on COVID-19.

Dear Editor,

The National Academy of Medicine defines health literacy as the degree to which individuals have the capacity to obtain, process, and understand basic health information needed to make appropriate health decisions. Most patients use the Internet as their initial source of health information (Swoboda et al. 2018). Currently, there is an established body of misinformation surrounding coronavirus disease 2019 (COVID-19) (Ioannidis 2020). Therefore, it is paramount that online health resources for COVID-19 are at an appropriate reading level; we aimed to assess their readability.

In a cache-cleared and location-disabled web browser, “coronavirus information” was entered in three popular search engines (Google, Bing, Yahoo!) on March 24, 2020, using five geographically representative virtual private network (VPN) locations in the United States and Canada (San Francisco, Miami, New York, Vancouver, and Toronto). Only websites in the first three search engine pages of each unique VPN location were included, as people are unlikely to proceed past this point (Eysenbach and Kohler 2002). Public health, governmental (including regional and national), and foundational webpages which pertained to COVID-19 information for the general public were included. Webpages under 100 words, non-English articles, photos, videos, advertisements, news articles, and webpages not aimed at the general public (e.g., for healthcare professionals or businesses) were excluded. Each webpage was assessed by two independent reviewers. Any differences in data extraction were resolved by re-evaluation until consensus.

The American Medical Association (AMA) and National Institutes of Health (NIH) have both recommended that reading levels of online health material should be at or below a sixth-grade level to be comprehensible to the general public (Fahimuddin et al. 2019). Thus, five scores were used to estimate the grade level appropriate for reading each website: Flesch-Kincaid Grade Level (FKGL), Automated Readability Index (AMI), SMOG Index (SMOGI), Coleman-Liau Index (CLI), and Gunning Fog Score (GFS). The general grade level (GGL) was calculated by averaging these five scores. Flesch Reading Ease Score (FRES) applies a scale ranging from 0 to 100 with higher scores indicating easier readability (Abu-Heija et al. 2019). Last, websites were categorized based on reading level: a GGL < 7 was considered easy; between 7 and 10, intermediate; and > 10, hard.

Of 428 webpages, 371 were excluded (330 were duplicates, 11 were < 100 words, 19 were not for the general public, and 11 were news articles). The mean GGL of the remaining 57 included webpages was 10.8. By stratifying readability of webpages with GGL, 70% were considered hard and 30% intermediate (Table 1). Based on FRES, 82% were considered hard to read. None of the webpages were considered easy to read based on either GGL or FRES.

Table 1

Readability of the 57 included webpages

Reading level scores	Mean ± SD (min–max)Number of total websites (n = 57)
GFS	12.6 ± 2.3 (8.6–19)
Easy (< 7)	0
Intermediate (7–10)	7
Hard (> 10)	50
FKGL	10.6 ± 2.2 (6.8–16.1)
Easy (< 7)	1
Intermediate (7–10)	22
Hard (> 10)	34
CLI	10.8 ± 1.6 (8.0–16.0)
Easy (< 7)	0
Intermediate (7–10)	10
Hard (> 10)	47
SMOGI	9.6 ± 1.6 (6.7–13.6)
Easy (< 7)	3
Intermediate (7–10)	33
Hard (> 10)	21
ARI	10.6 ± 2.6 (5.9–17.9)
Easy (< 7)	5
Intermediate (7–10)	18
Hard (> 10)	32
aGGL	10.8 ± 1.9 (7.3–15.6)
Easy (<7)	0
Intermediate (7–10)	17
Hard (> 10)	40
bFRES	51.5 ± 9.7 (23.6–71.5)
Easy (80–100)	0
Intermediate (60–79)	10
Hard (0–59)	47

aThe general grade level (GGL) was calculated based on the average scores of the Gunning Fog Score (GFS), Flesch-Kincaid Grade Level (FKGL), Coleman-Liau Index (CLI), SMOG Index (SMOGI), and Automated Readability Index (ARI)

bFlesch Reading Ease Score (FRES) applies a scale ranging from 0 to 100, with higher scores indicating easier readability

In conclusion, our results show that the readability of online resources for COVID-19 exceeds thresholds previously set by the AMA and NIH. Readability may be improved by limiting sentences to 8–10 words and using simple words, instead of medical terminology (Boztas et al. 2017). We recommend that both government and health information websites re-evaluate their online resources to match an appropriate reading level for the general public.