Assessment of patient education materials for age-related macular degeneration.

PURPOSE: Age-related macular degeneration (AMD) is a leading cause of vision loss. It is helpful for patients living with AMD to understand the prognosis, risk factors and management of their condition. Online education materials are a popular and promising channel for conveying this knowledge to patients with AMD. However, the quality of these materials-particularly with respect to qualities such as 'understandability' and 'actionability'-is not yet known. This study assessed a collection of online materials about AMD based on these qualities of 'understandability' and 'actionability'.
METHODS: Online education materials about AMD were sourced through Google from six English-speaking nations: Australia, New Zealand, USA, UK, Ireland and Canada. Three Australian/New Zealand trained and registered optometrists participated in the grading of the 'understandability' and 'actionability' of online education materials using the Patient Education Materials Assessment Tool (PEMAT).
RESULTS: This study analysed a total of 75 online materials. The mean 'understandability' score was 74% (range: 38%-94%). The 'understandability' PEMAT criterion U11 (calling for a summary of the key points) scored most poorly across all materials. The mean 'actionability' score was 49% (range: 0%-83%). The 'actionability' PEMAT criterion A26 (using 'visual aids' to make instructions easier to act on) scored most poorly across all materials.
CONCLUSION: Most education materials about AMD are easy to understand, but difficult to act on, because of a lack of meaningful visual aids. We propose future enhancements to AMD education materials-including the use of summaries, visual aids and a habit tracker-to help patients with AMD improve their understanding of disease prognosis, risk factors and eye assessment schedule requirements.

Online education materials about age‐related macular degeneration empower patients to improve their understanding of the disease and help patient adherence to follow up and management.

Current age‐related macular degeneration material is easy to understand but hard to action.

Areas of improvement for age‐related macular degeneration education materials include using summaries, pictures/illustrations to make instructions easier to action and a habit tracker customised to the patient's management plan.

BACKGROUND

Age‐related macular degeneration (AMD) is a leading cause of visual impairment and blindness worldwide. AMD substantially impacts patients' quality of life and emotional well‐being. Patients' behaviours can influence their AMD progression. Poor health behaviours, such as smoking and poor diet, are associated with a higher risk of AMD development and progression.

Education materials about AMD alert patients to the importance of modifiable risk factors (e.g., smoking cessation and a healthy diet), thus empowering patients to make lifestyle changes to reduce the risk of AMD development and progression. Such education materials also draw attention to non‐modifiable risk factors such as age, so that patients appreciate the preventative benefit of routine eye examinations. The ability of education materials to deliver these benefits depends on their quality, typically characterised as readability (i.e., reading level), suitability, accountability (referring to attribution of authorship and sources) and the ability of materials to influence patients' medical decisions. However, these indicators notably do not measure patient comprehension and also do not consider whether education materials are written with a clear call to action.

While education materials have traditionally been printed (e.g., articles, diagrams, brochures and posters), online materials are increasingly prevalent due to their low cost of production and updating. Patients typically search for information on the Internet about a condition, symptoms and treatment. Some patients gain confidence in their medical decision‐making by using online resources. Patients with AMD generally trust the quality of the online resources. However, online resources for ophthalmic information, on average, meet only 26% of criteria for informative content quality due to a lack of information regarding AMD diagnostic procedures, prevention and prognosis.

Previous studies about online AMD‐related materials have assessed the materials using the Flesch–Kincaid Grade Level readability formula and the Suitability Assessment of Materials assessment, with the accuracy and accountability of the materials based on the JAMA benchmark criteria, and the reliability and quality of online AMD educational videos based on the DISCERN criteria. However, they do not assess whether the materials are easy to understand or can identify clear calls to action for patients with AMD. This study analysed the ‘understandability’ and ‘actionability’ of printable education materials available online for patients with AMD. Based on this analysis, the study identifies areas of possible improvement for future education materials.

METHODS

Data collection using web search

A series of patient education materials were collected by entering the search term ‘age‐related macular degeneration’ into the Google search engine during the 2‐month period of January and February 2021. To ensure that previous searches would not impact the search results, the Microsoft Edge web browser was used in ‘InPrivate’ mode. A virtual private network service was used to find the search results from six developed English‐speaking nations : Australia, New Zealand, USA, UK, Ireland and Canada. Eligible materials had to meet the following inclusion criteria: (1) source appears to be designed for people with AMD and (2) the source is written in English. An education material from a website was defined as all content related to AMD featuring across one or multiple webpages within the same website domain; otherwise, webpages within the same website unrelated to AMD were not analysed. If websites contained a separate PDF document, these were evaluated separately. Education materials were excluded if they were targeted towards professionals or required a paywall to access. Due to the possibility of content modification from the authors of the website, a screenshot of the webpage was taken to ensure the graders evaluated the same material.

Data analysis using patient education materials assessment tool (PEMAT)

This study evaluated a series of patient education materials for ‘understandability’ and ‘actionability’ using the Patient Education Materials Assessment Tool (PEMAT). The PEMAT is a validated instrument designed to identify education materials that are suitable (i.e., not too complex) for patients with low levels of health literacy. The PEMAT has been used to assess education materials in other areas of health care, such as in diabetes, hypertension and online information about heart failure, with moderate to excellent inter‐rater reliability. However, the PEMAT has not been used in eye care or ocular disease contexts.

There are different variants of the PEMAT instrument; the specific variant that this study used is the ‘PEMAT‐P’, for printable materials (which includes webpages and PDF documents, but not—for example—video clips). PEMAT‐P consists of 17 criteria to assess materials' ‘understandability’ (i.e., whether patients are able to comprehend materials) and seven criteria to measure materials' ‘actionability’ (i.e., whether patients can identify what to do). These criteria check for the presence of specific features such as active voice (criterion U5), common everyday language (criterion U3), logical sequencing of information (criterion U10), helpful next actions (criterion A20), etc.

PEMAT‐P specifically refers to the concept of a visual aid, referring to an illustration or a diagram. PEMAT‐P (and the PEMAT instrument more broadly) was developed for health communications in general, and not specifically for an eye care‐related health domain (e.g., AMD) in which the term visual aid more conventionally refers to visual aid equipment to assist patients with low vision. We retain Shoemaker et al.’s term visual aid in order to stay faithful to their instrument; however, in this paper, we indicate the term in quotation marks (i.e., ‘visual aid’) to clarify that we refer to the PEMAT meaning of the term rather than the eye care meaning of the term.

Scoring models

According to the PEMAT guidelines, if a PEMAT criterion is graded ‘true’, then it is assigned a score of ‘1’ (encoded in our dataset as ‘T’, see Figure 1); if the criterion is graded ‘false’, then it is assigned a score of ‘0’ (encoded ‘F’, see Figure 1) and if the criterion is graded as ‘not applicable’ (N/A), then it is not counted in the final score. The final ‘understandability’ and ‘actionability’ scores for each source are expressed as a percentage: calculated by the total number of points, divided by the total possible points. Each final score can then be classified as either ‘pass’ or ‘fail’ based on the ≥70% cut‐off score set by the PEMAT guidelines. This part of the scoring process is depicted in Figure 1 below as the ‘horizontal’ score per source.

FIGURE 1

Summary of how per‐source and per‐criterion scores are generated. ‘Understandability’ and ‘actionability’ 70% cut‐off score defined by Shoemaker et al

To identify areas of future improvement, we applied a modified version of the PEMAT extended scoring method proposed by Kang and Lee. In contrast to Kang and Lee's tally‐based count of sources for each criterion, the percentage of satisfactorily performing sources that satisfied each criterion was calculated. In the absence of specific success rates for each criterion, a cut‐off score of 50% based on the pedagogical practice of ‘a pass mark of 50%’ was applied to develop suggestions for future education materials. Figure 1 shows the different pass or fail cut‐off scores used in this study. This part of the scoring process is depicted in Figure 1 as the ‘vertical’ score per source.

Operationalisation

Three Australian/New Zealand trained and registered optometrists participated in the assessment in this study. Two optometrists (EW, KW; see Acknowledgements) independently assessed the materials and recorded the scores in a spreadsheet. If both graders provided the same answer (either ‘T’, ‘F’ or ‘N/A’), then it was considered a match. Otherwise, any other combination (e.g., grader 1 answers ‘F’ and grader 2 answers ‘N/A’) was considered a mismatch. The third optometrist (SH) was involved in a discussion with the other two optometrists to achieve consensus about the scores. The average ‘horizontal’ score and range of scores across all 75 materials for ‘understandability’ and ‘actionability’ were also calculated to reflect overall performance of materials in the field.

RESULTS

The first three results pages returned from the search engine across the six nation searches were evaluated. A total of 92 unique sources were screened for inclusion. Of these, 17 sources were excluded due to a target audience of professionals (n = 12); paywall or subscription needed for access (n = 4) and website broken (n = 1). Finally, 75 unique sources were analysed: 69 were websites screenshots and six were downloadable PDF documents. Figure 2 provides examples of overall high and low scoring materials according to the PEMAT‐P criteria.

FIGURE 2

Example screenshots of high scoring educational materials (source number: 57 and 23) and low scoring educational materials (source number: 22 and 53)

PEMAT‐P ‘understandability’ scores

The mean PEMAT‐P score (i.e., ‘horizontal’ score per source, see Figure 1) of the printable materials for ‘understandability’ was 74% (range: 38%–94%). Of the 75 sources analysed, 44 sources passed the cut‐off score of ≥70%.

The percentage of materials, from a total of 75 sources that satisfied each PEMAT‐P ‘understandability’ criterion, are shown in Figure 3 (diagram adapted from Kang and Lee ). The 50% ‘satisfactory sources’ score (i.e., ‘vertical’ score), marked with a dotted line, shows which criteria passed or failed. The weakest performing areas with less than 50% of satisfactory sources analysed were: (1) criterion U11—not including a summary of key points; (2) criterion U17—not using clear titles or captions for ‘visual aids’ and (3) criterion U15—absence of ‘visual aids’ such as pictures or diagrams to help make information easier to understand.

FIGURE 3

Comparison of ‘Satisfactory Sources’ scores between the various ‘understandability’ criteria of the Patient Education Materials Assessment Tool for printable materials (PEMAT‐P). Adapted with permission from Kang and Lee

PEMAT‐P ‘actionability’ scores

The mean PEMAT‐P score (i.e., ‘horizontal’ score per source) of the printable materials ‘actionability’ was 49% (range: 0%–83%). Of the 75 sources analysed, 13 sources passed the cut‐off score of ≥70%, that is, 62 sources scored <70% on ‘actionability’.

The percentage of materials, from a total of 75 sources that satisfied each PEMAT‐P ‘actionability’ criterion, are shown in Figure 4 (diagram adapted from Kang and Lee ). The 50% ‘satisfactory sources’ score (i.e., ‘vertical’ score), marked with a dotted line, shows which criteria passed or failed. The weakest performing areas with less than 50% of satisfactory sources analysed were: (1) criterion A26—not providing ‘visual aids’ such as pictures or diagrams for actionable instruction; (2) criterion A22—not breaking down actionable items into manageable explicit steps; and (3) criterion A23—not providing a ‘tangible tool’ to help patients take action. Criterion A24—material provides simple instructions or examples of how to perform calculations, was excluded as all graders scored all education materials as ‘N/A’, and hence, this criterion was not considered in the final scoring.

FIGURE 4

Comparison of ‘Satisfactory Sources’ scores between the various ‘actionability’ criteria of the Patient Education Materials Assessment Tool for printable materials (PEMAT‐P). Adapted with permission from Kang and Lee16

DISCUSSION

Interpretation of results

This study evaluated the understandability and actionability of AMD education materials designed for those with the disease, and identified areas of future improvement. Most online education materials evaluated in this study had a good ‘understandability’ score, mean of 74%, and a suboptimal ‘actionability’ score, mean of 49%. A similar trend has been found in prior studies evaluating online education materials using the PEMAT‐P, where the ‘understandability’ score is higher than the ‘actionability’ score. Table 1 compares these PEMAT‐P scores for AMD education materials with those from other healthcare domains. As seen in Table 1, the scores for AMD education materials are comparable with (if not better than) those from other healthcare domains (e.g., hypertension, diabetes, heart failure).

TABLE 1

Scores for age‐related macular degeneration (AMD) materials compared with materials for other examples of healthcare domains

Healthcare domain	PEMAT‐P ‘understandability’ score	PEMAT‐P ‘actionability’ score	Reported by
Age‐related macular degeneration (AMD)	75%	49%	This study.
Hypertension	76%	53%	Ab Hamid et al. 17
Diabetes	62%	39%	Lipari et al. 22
Heart failure	56%	35%	Cajita et al. 18

Abbreviations: PEMAT‐P, Patient Education Materials Assessment Tool for printable materials.

However, when interpreting any such PEMAT‐P results—whether for a specific source of education materials (e.g., in Figure 2) or at various levels of aggregation (e.g., Figures 3 and 4 and Table 1)—it is important to be aware that the PEMAT‐P uses binary classification (i.e., ‘true’ or ‘false’) resulting in a forced choice during grading, which may inadvertently skew the results. This effect may be compounded in cases where some PEMAT‐P criteria can be marked as ‘N/A’, excluding those criteria from both the numerator and the denominator when calculating PEMAT‐P scores (as was the case, for example, for 20 of the 75 sources in relation to criterion U16—‘visual aids’ reinforce rather than distract from the content). The PEMAT‐P was also developed and validated (e.g., internal consistency, inter‐rater reliability) for health communications in general. The tool does not cater for the needs of special populations, for example, patients with vision impairment due to AMD or other causes.

Suggestions for future education materials

Based on the 50% ‘satisfactory sources’ score cut‐off, we suggest three areas of improvement: (1) summaries, (2) ‘visual aids’ and (3) a habit tracker as a ‘tangible tool’. Only 14% of education materials included a summary. Summaries are useful to highlight key points, especially for long education materials, and should be used more often in education materials. According to the PEMAT‐P guidelines, summaries are not needed for short print materials with two or fewer paragraphs and less than one page in length. However, some websites are designed such that, instead of placing all information onto a single page, the website is subdivided into multiple webpages connected with hypertext links. After entering the website from the initial search page, patients may not click or scroll through multiple webpages to find the necessary key information. Figure 5 shows an example of a summary box developed by the lead author that describes smoking cessation as the main modifiable risk factor in AMD.

FIGURE 5

Proposed summary explaining the importance of quitting smoking

Reflected in the PEMAT‐P criteria, ‘visual aids’ may promote both ‘understandability’ (criterion U15) and ‘actionability’ (criterion A26). Only 47% of the sources included ‘visual aids’ (criterion U15). ‘Visual aids’ were assigned one point if the picture is inserted related to AMD, but no points were deducted if the picture was misleading. Misleading pictures (i.e., pictures not truly representative of what early to intermediate AMD stages look like) were present in the education materials analysed. Vision loss in AMD education materials is typically depicted as a central, circular grey or black patch superimposed over a scene, and 76% of patients with early or intermediate AMD reported those scenes to be unrealistic. Only 43% of the sources used clear titles or captions with ‘visual aids’ (criterion U17). To improve ‘understandability’ using ‘visual aids’, we propose (in Figure 6) a ‘visual aid’ with more realistic pictures of what AMD patients have described as blur and distortion, accompanied with clear captions. Figure 6 is an example of what a streetlight would look like for various individuals depending on the stage of AMD: people with healthy eyes or early AMD would see a normal view of a streetlight; intermediate AMD is represented with slightly wavy lines and slight blur in the central vision affecting the surrounding buildings and advanced AMD is represented with blurred central vision of the streetlight and buildings.

FIGURE 6

Example of what a streetlight may look like for different stages of age‐related macular degeneration (AMD)

Only 9% of sources included ‘visual aids’ to help make instructions easier to act on (criterion A26)—the weakest PEMAT‐P criterion. Most education materials only listed modifiable risk factors without using ‘visual aids’. Such a ‘visual aid’ could be developed to show the composition of a diet that may be beneficial for patients with AMD, using similar techniques as the Finnish Heart Association and Finnish Diabetes Association's Baltic Sea Diet Pyramid or the Mediterranean Diet Foundation's Modern Mediterranean Diet Pyramid.

Another issue contributing to poor ‘actionability’ is the absence of a component that breaks down actions into explicit steps (criterion U22). More details should be provided when discussing AMD interventions. Instead of simply listing broad categories of foods to eat (e.g., ‘eat dark green leafy vegetables’), it may be more helpful to describe the specific number of times a week that specific foods within those categories should be consumed (e.g., ‘eat kale or spinach every day’) and the benefits of this behaviour. The explicit action steps to take can be incorporated into a ‘tangible tool’ (criterion U23), such as a checklist or a recipe card. We propose a habit tracker as a ‘tangible tool’ for patients with AMD. This habit tracker is essentially a checklist designed in such a way that recurring actions (e.g., those relating to diet and exercise) are presented appropriately. The design of this habit tracker would be customised to the patient's stage of AMD progression, and other relevant management plans such as smoking cessation or taking supplements. A prototype is shown in Figure 7.

FIGURE 7

Prototype of a habit tracker for patients with age‐related macular degeneration (AMD)

Limitations and future directions

The scope of web search results in this study was limited to the first three pages, and only for English‐language materials. Although valid because patients rarely venture beyond even the first page of the results, , future research may seek to also assess the ‘understandability’ and ‘actionability’ of AMD education materials written in languages other than English.

Secondly, this study analysed patient education materials exclusively using the PEMAT‐P instrument, and complements existing work, for example, by Fortuna et al. based on the Flesch–Kincaid Grade Level readability formula and the Suitability Assessment of Materials assessment, and by Kloosterboer et al. based on the JAMA benchmark criteria. Future research exploring underlying social conditions and psychological factors influencing ‘actionability’, considering frameworks such as the Health Belief Model, the Elaboration Likelihood Model of attitude change or the Heuristic‐Systematic Model of persuasion are also important. Exploration of topics such as health and information literacy will also be addressed in future studies.

CONCLUSION

Most education materials about AMD are easy to understand, but difficult to act on because of a lack of meaningful visual aids. We propose future improvements to AMD education materials including the use of summaries, visual aids and a habit tracker. These improvements may empower patients to improve their understanding of disease prognosis, risk factors and eye assessment schedule requirements to optimise eye care delivery.

AUTHOR CONTRIBUTIONS

Elisa Wang: Conceptualization (lead); data curation (lead); formal analysis (lead); investigation (lead); methodology (equal); project administration (lead); resources (lead); validation (supporting); visualization (lead); writing – original draft (lead); writing – review and editing (supporting). Michael Kalloniatis: Formal analysis (supporting); funding acquisition (lead); supervision (supporting); validation (supporting); visualization (equal); writing – original draft (supporting); writing – review and editing (equal). Angelica Ly: Conceptualization (supporting); data curation (supporting); formal analysis (supporting); investigation (supporting); methodology (equal); project administration (supporting); resources (supporting); supervision (lead); validation (lead); visualization (equal); writing – original draft (supporting); writing – review and editing (lead).

CONFLICT OF INTEREST

The authors declare that they have no competing interests.