Assessment of the readability and quality of online patient education materials for the medical treatment of open-angle glaucoma.

Objective: Patient adherence to glaucoma medications is poor, and is linked to low literacy levels. Patients commonly use the internet to access health information, and it is recommended that patient information is written at an 11-year-old reading level. The aim of this study is to assess the readability and quality of online patient education materials for the medical management of open angle glaucoma. Methods and analysis: The top 10 relevant Google searches for nine glaucoma medications (timolol, brimonidine, apraclonidine, dorzolamide, latanoprost, bimatoprost, travoprost, tafluprost and brinzolamide) and three generic searches were analysed for readability and accountability. Readability was assessed using Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI) and Simple Measure of Gobbledygook Index (SMOG). Webpages were classified by source and assessed using Journal of the American Medical Association (JAMA) benchmarks of accountability.
Results: 111 articles were included in the analysis. Mean readability scores were: FRES 55.5 (95% CI 53.4 to 57.5); FKGL 9.7 (95% CI 9.3 to 10.0); GFI 12 (95% CI 11.6 to 12.4) and SMOG 9.3 (95% CI 8.9 to 9.6). One-way analysis of variance demonstrated no significant difference in readability score between source type. 9% of the webpages satisfied all 4 JAMA benchmarks. Pearson's correlation coefficient showed a correlation between the FRES and accountability score (r=0.19, p=0.045).
Conclusion: The majority of online patient education materials for the medical treatment of glaucoma are written at a level too difficult for the general population and fail to meet accountability standards. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Glaucoma is a common preventable cause of blindness and adherence to topical medication is known to be poor. One factor contributing to poor medication adherence is low levels of health literacy among patients. Patient education materials relating to glaucoma treatment are readily available on the internet, and should be written in a way which is accessible to patients whom have a range of health literacy levels. Other research within ophthalmology and other specialities has found that often patient education materials are written at a level too difficult for patients to understand.

The majority of online patient education materials for the medical treatment of glaucoma are written at a level which far exceeds the recommended reading age in the UK and the USA (11 years old).

Patient-orientated websites such as charities and health information sites should consider how readable their information is when creating patient education materials. If the information is made more accessible to patients with lower levels of health literacy, this may help empower patients by giving them a better understanding of their health conditions. Through failing to consider health literacy, these patient focused sites may be compounding health inequalities.

Introduction

Glaucoma

Glaucoma is the second leading cause of irreversible vision loss worldwide.1 The most common type of glaucoma in the UK is primary open-angle glaucoma. It is a chronic asymptomatic disease which affects 10% of people above the age of 75.2 The majority of patients are initially treated medically with eye-drops, the aim of which is to reduce intraocular pressure.3 Sustained patient adherence is critical to prevent disease progression.4

The proportion of glaucoma patients who are non-compliant to medical therapy has been reported to be as high as 50%.5–7 Numerous reasons for non-compliance to glaucoma medications have been cited which include medication cost, tolerability, regimen-related factors and poor health literacy.8 9

Health literacy

Health literacy is defined by the WHO as the achievement of a level of knowledge, personal skills and confidence to take action to improve personal and community health by changing personal lifestyles and living conditions, and is critical to health empowerment.10

Literacy is the capacity to process written and verbal information. Patients with poor health literacy levels are less likely to understand and remember medication advice, and may take medications at the wrong frequency or dose and not understand the adverse effects.11 12 A study in the USA found a positive correlation between low health literacy levels and poor adherence to glaucoma medication. Of the 197 subjects, only 48% could read at or above a 9th grade level. Subsequently, in order to improve compliance, the study recommended adapting the language used in ophthalmic patient education materials (PEMs) to ensure the material is accessible to more patients.13

An American Medical Association Council of Scientific Affairs report concluded that patients with low health literacy have poorer health outcomes and less understanding of their medical conditions and treatment.14 In 2015, Public Health England and the Institute of Health Equity published a report which showed that up to 61% of the working age population find it difficult to understand health and well-being information.15 Furthermore, 16% of adults in England have the lowest level of proficiency in literacy, which is at or below the literacy levels for a 5–7 year old.16 The average literacy level in the UK is considered to be similar to that in the USA, which is equivalent to a 13 year old (eighth grade student in the USA).17

Readability

Readability of written text is an objective measure of the reading skills one must possess to understand the material, and is measured in terms of US ‘grade levels’.18 The readability of text has a significant impact on the comprehension of the material by the patient.19 Despite this, health literacy and readability of online PEMs is often overlooked.20

Health Education England advises that written patient information material should be written at a level that can be understood by an 11 year old.21 Similarly the American Medical Association recommends writing at a US grade level of 6 or less, which is equivalent to a 11–12 year old.22

Previous research has shown that online health information can have an important role in influencing patients decision making regarding their health.23 Four out of five UK adults have access to the internet at home and a recent survey showed 88% of internet users search for health related information online.24 25 However, no quality standard exists for these webpages providing health information, which may lead to misinformation or inaccuracies.26 A tool to assess the quality of webpages providing health information was created by the Journal of the American Medical Association (JAMA).27 They set out four benchmarks to assess the accountability of a webpage, which include authorship, attributions, disclosure and currency.

The readability of online PEMs for ophthalmological conditions such as age related macular degeneration, diabetic retinopathy and intravitreal injections has been reviewed in the literature, and has consistently found that materials are written above the recommended level.26 28–31 The aim of this study is to assess the readability and quality of online PEMs relating to the medical management of open angle glaucoma.

Materials and methods

A series of searches using the search engine Google were conducted in November 2021. Articles were included if they were written in English, and contained patient orientated information. The chosen search terms included either a medication trade name and brand name(s) or a generic search term. The medications included in the search were those recommended by National Institute for Health and Care Excellence for the treatment of open-angle glaucoma.32 The trade and brand names of each medication were included. The nine medications in the search were timolol (Blocadren, Timol), brimonidine (Alphagan), apraclonidine (Iopidine), dorzolamide (Trusopt), latanoprost (Xalatan, Xelpros, Monoprost), bimatoprost (Lumigan, Latisse), travoprost (Travatan), tafluprost (Zioptan, Saflutan, Taflotan, Tapros) and brinzolamide (Azopt). In addition to these medications, three generic search terms ‘glaucoma medication’, ‘glaucoma drops’ and ‘glaucoma treatment’ were also included. To gain the most relevant webpages the search terms used included ‘open angle glaucoma patient information’ as the prefix.

Duplicate websites, resources for healthcare professionals, patient forums, reviews of medications and academic research was excluded. If articles contained information relating to laser or surgical treatment for glaucoma, this was excluded from the readability assessment, but any general information about glaucoma was retained. The methodology used is consistent with other readability articles within ophthalmology and other medical and surgical specialties.30 33–36

The top 10 relevant PEMs were selected for each of the 9 medications listed above and the 3 generic searches. This had potential to generate 120 PEMs. Once duplicates were excluded, nine PEMs were obtained for tafluprost, two for ‘glaucoma treatment’. Ten PEMs were obtained for all other search terms. This provided an overall total of 111 PEMs.

Patient and public involvement

Patients and the public were not involved in this research.

Webpage by source

Webpages were classified into ‘charities’, ‘non-clinical health information’, and ‘clinical health providers’. For the purposes of this study, a charity webpage was one affiliated to a registered charity. Non-clinical health information is any webpage providing patient information, which does not provide clinical services, and clinical health providers is any webpage which provides patient information and clinical services.

Readability assessment tools

Readability was assessed using an online readability software.37 The following four scores were used to assess readability: Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI) and Simple Measure of Gobbledygook Index (SMOG). Readability is calculated by applying a mathematical formula to a passage of text (see table 1). The tools consider different factors such as average number of words per sentence and/or number of syllables per word. The interpretation of each readability tool is described in table 2.

Table 1

Readability tools and their formulas

Assessment tool	Formula
Flesch Reading Ease Score	206.835 – (1.015 × average no of words per sentence) – (84.6 × average no of syllables per word)
Flesch-Kincaid Grade Level	(0.39 × average no of words per sentence) + (11.8 × average no of syllables per word) – 15.59
Gunning Fog Index	0.4 (average no of words per sentence +(100 x average no of 3+syllable words)
Simple Measure of Gobbledygook Index	3 + √Polysyllabic count

Table 2

Readability tools and their interpretation

Assessment tool	Interpretation	Standard
Flesch Reading Ease Score	Ranges from 0 to 1000: Unreadable<30: University graduate level60–70: Understood by 13–15 years old90–100: Very easy to read, understood easily by 11-year-old student (50,51)	90–100, which is easily understood by an 11-year-old student
Flesch-Kincaid Grade Level	Indicates no of years of education required to understand the text	Grades were considered to be at the recommended level (grade 6) if they were ≤6.9
Gunning Fog Index	Ranges from 6 to 176: Reading age of 11–12 years old17: University-level graduate(52)	At the recommended level if 6–6.9
Simple Measure of Gobbledygook Index	Indicates no of years of education required to understand the text	Grades were considered to be at the recommended level (grade 6) if they were ≤6.9

These specific readability tools were chosen as they are commonly used to assess the readability of text, particularly in healthcare settings, and a combination of scores improves the accuracy of the results.38 All webpages had copyright notices, adverts, images, author names, disclaimers and reference lists removed prior to assessing readability.

JAMA benchmarks

To assess the accountability of online PEMs, JAMA benchmarks were applied.27 To comply with the JAMA accountability criteria, each PEM had to include the author, their affiliations and credentials (authorship), include a reference list (attribution), declare any sponsorship, advertising and a disclaimer (disclosure) and the date of the last review and/or update of the page (currency). The webpage was then given a score which ranges from 0 (no criteria fulfilled) to 4 (all 4 criteria fulfilled). The null hypothesis is that there is no association between how readable a webpage is and its accountability score.

Statistical analysis

Statistical analysis was completed with IBM SPSS V.27 software for Mac OSX. FRES, FKGL, SMOG, GFI scores and their respective mean values, SD and 95% CIs were calculated for each of the 12 search items. The data was normally distributed. The distributions of the study variables were amenable to parametric analysis, and mean values were compared using one-way analysis of variance.

To assess the degree of association between accountability and readability scores, Pearson correlation coefficient was used. Statistical significance was set at p<0.05, all p values represented were two sided.

Results

Of the 111 PEMs obtained, the average length was 1411 words, and the average number of words per sentence was 15.3.

The FRES ranged from 15.2 (very difficult) to 79 (fairly easy), with an average of 55.5 (SD=10.8; 95% CI 53.4 to 57.5). The FKGL score ranged from 5.3 to 17.3, with an average of 9.7 (SD=2.1; 95% CI 9.3 to 10.0). The GFI score ranged from 7.7 to 18.5, with an average of 12 (SD=2.1; 95% CI 11.6 to 12.4). The SMOG score ranged from 5.7 to 15.6, with an average of 9.3 (SD=1.7; 95% CI 8.9 to 9.6).

When categorising the websites by source there were: 12 (11%) charities, 25 (22%) clinical health providers and 74 (67%) non-clinical health information websites. The readability scores did not differ significantly between source type: FRES (p=0.28), FKGL (p=0.08), SMOG (p=0.14), GFI (p=0.14) (see table 3).

Table 3

Mean (SD) readability index scores of patient education materials, classified by source

Readability tool	Charities (n=12)	Clinical health providers (n=25)	Non-clinical health information (n=74)	Total (average across all sources, n=111)
FRES	50.7 (16.3)	56.0 (8.1)	56 (10.5)	55.5 (10.8)
FKGL	10.9 (2.8)	9.6 (1.6)	9.4 (2.1)	9.7 (2.1)
GFI	13.0 (2.2)	12.2 (1.7)	11.8 (2.16)	12.0 (2.1)
SMOG	10.1 (2.5)	9.3 (1.4)	9.1 (1.6)	9.3 (1.7)

FKGL, Flesch-Kincaid Grade Level; FRES, Flesch Reading Ease Score; GFI, Gunning Fog Index; SMOG, Simple Measure of Gobbledygook Index.

Out of all 111 search results, 10 (9%) met the requirements to satisfy all 4 JAMA benchmarks of accountability (see table 4).

Table 4

Patient education materials and JAMA accountability

JAMA accountability	No (%) (n=111)
0	16 (14)
1	13 (12)
2	42 (38)
3	30 (27)
4	10 (9)

JAMA, Journal of the American Medical Association.

Using Pearson correlation coefficient, only FRES was found to have a significant correlation with the accountability score (r=0.19, p=0.045). No significant correlation was found between the accountability score and FKGL (r=−0.096, p=0.315), GFI (r=−0.084, p=0.382) and SMOG (r=−0.118, p=0.217).

Discussion

The internet has become a vital source of health information for patients in recent years and it can be a useful tool to help supplement verbal clinical advice. However, a large proportion is written at a higher reading level than recommended.39

Research studying online browsing behaviours analysed 5 million Google searches and found that only 0.78% of Google searchers clicked on something from the second page.40 Therefore, in this study the top 10 results (that satisfied the study inclusion criteria) from a Google search were chosen, as it was deemed unlikely that patients would look beyond these results for their information.

Our study showed that the majority of websites containing information about the medical treatment of glaucoma were classified as being too difficult to read for the majority of the population. On average, the FRES determined the articles to be ‘fairly difficult to read’ (55.5). This is the equivalent to reading the TIME magazine (FRES score 50).41 All of the articles had a FRES score which was above the recommended 6th grade level (11-year-old reading level), and 24% were written at a level suitable for a college graduate or higher.

On average the FKGL score was at the level of a 9th grade student (9.7), which is equivalent to a 14–15 year old. Ninety four articles (84.7%) had an FKGL score that was above the recommended reading level.

The readability of text may also be complicated by use of complex words, and the GFI takes this into account. The average GFI score was 12, which is at the education level of a high school senior and is above the recommended level of 6th grade student. According to the GFI, 111 (100%) of the articles were written at a level above the recommended range and equates to reading the Wall Street Journal.42 Similarly, the SMOG showed on average the articles were written at a 9th US grade level, with 108 (97.2%) of articles being written above the recommended range.

Our findings are consistent with previous studies within ophthalmology, and other surgical specialties.20 43 44 A systematic review conducted by Williams et al focusing on online PEMs in ophthalmology found that the materials are consistently written at a level that is too high for many patients to understand.28 Across the 13 studies included, the median FKGL score represented an 11th grade study level, which is higher than the results obtained in our study (9th grade).

In our study, websites were classified by source and analysed to see whether the readability scores varied between them. The readability scores FRES, FKGL, GFI and SMOG did not significantly differ by type of website in this study. However, given that the majority of webpages were from non-clinical health information sites (67%) compared with charities (11%), a larger sample size may be required to interpret the results accurately.

There is concern about the accuracy of online health information, and a significant proportion of the material currently available is unreliable.45 When considering the accountability of the webpages, only 9% satisfied all four JAMA benchmarks of accountability (see table 4). The most common webpage to satisfy all four benchmarks was one that provides patient information on multiple glaucoma medications (www.patient.co.uk). The most common benchmark to be satisfied was disclosure (66.7%) whereas the least common benchmark to be satisfied was attribution (24.3%).

The association between the FRES score and JAMA accountability score (r=0.19, p=0.045) indicate that more readable PEMs were also more likely to meet the accountability criteria. However, there was no significant association between the other readability scores and JAMA accountability.

Limitations of this study include the assumptions made by the use of readability tools. Although the number of syllables per word, or the length of sentences affect the readability of text, they do not take into account the context or meaning of the words used.46 Furthermore, images, tables, diagrams and the general layout and design of the page impacts the readability of the text, but is not considered when using such scores.

Furthermore, short monosyllabic but technical medical terminology may have been used within the articles and the tools could then underestimate the reading level required to comprehend the text. Similarly longer polysyllabic words may not necessarily be more difficult to understand. For example, medication names (brinzolamide) may be repeated multiple times within the article, however, once initially understood within the text should not be difficult to interpret.

The literature reports that FKGL and FRES underestimate the readability score compared with SMOG. SMOG has been found to be more accurate in assessing readability of healthcare resources than FKGL and FRES, and this should be considered when interpreting results.47 FKGL and FRES take into account different variables compared with SMOG, and therefore, these tools may be limited when used in isolation. Of note though, in this study FKGL, FRES and SMOG estimated the readability to be at a 9th grade level, demonstrating consistency among the scoring systems.

The top 10 results from each Google search was selected, including only websites that were aimed at patients. It is possible that patient browsing behaviours differ from the study inclusion criteria. They may, for example, choose to read information aimed at healthcare professionals, or use forums where information is shared between the general public. We used the most popular search engine within our study, Google, however, other search engines may also be used by patients. We recognise that the results are reflective of searches completed at one point in time (November 2021), and given the rapidly evolving nature of online information the results may not reflect a patients browsing experience following this time period.

Within our study three generic search terms: glaucoma medication, glaucoma treatment and glaucoma drops were used. These were chosen as they were deemed to reflect patient behaviour when conducting a Google search. However, we recognise that these may not be an accurate representation of the search terms patients use. Future studies could be based on data demonstrating patients online browsing behaviour, for example, through patient surveys. This would give us a better understanding of how patients search for and choose health information online.

This study has shown that PEMs relating to the medical treatment of glaucoma are written above the recommended reading level of 11 years old. This includes all source types including charities, clinical health providers and non-clinical health information sites. There is a need for all types of organisations to consider health literacy when creating online content aimed at providing health information to patients. Simple measures to improve the readability of materials include using short simple words, avoiding the use of jargon and writing short sentences.48 Online supplemental material such as the Centers for Disease Control ‘Simply Put It’ guide helps to create easy to understand material.49

People with limited health literacy are known to less successfully manage long term health conditions, and people with low financial and social resources are more likely to have low health literacy. By improving health literacy it can help improve patients adherence to medical instruction and empower them to effectively manage long-term conditions.15

Our study indicates there is a need for greater awareness of readability when publishing online resources, and the readability of current glaucoma literature online could be compounding health inequalities.