What is the quality of drug safety information for patients: An analysis of REMS educational materials.

BACKGROUND: Poor-quality patient drug information has been identified as a major cause of preventable medication errors in the United States. The US Food and Drug Administration (FDA) has the authority to require marketing authorization holders of medicinal products to implement risk evaluation and mitigation strategies (REMS) to ensure that the benefits of a drug or biological product outweigh its risks. Aside from medication guides, no research has been conducted to assess the quality of patient-targeted REMS materials, including whether, and to what extent, patients find these materials understandable and actionable.
PURPOSE: To describe the readability, understandability, and actionability of patient educational materials in currently approved REMS programs, and to highlight opportunities for improving both the quality and effectiveness of these important drug safety tools.
METHODS: Seventy-seven REMS programs were identified from the FDA REMS database. We excluded medication guides (MGs) from our analysis because of the fact that there is a mandatory MG template. Based on this, we identified a total of 27 (non-MG) REMS patient materials on the FDA REMS website for analysis purposes. The materials were tested for readability using the Lexile Measure, the Gunning Fog Index, and Flesch Kincaid and then assessed using the Patient Education Materials Assessment Tool for printable materials, for understandability and actionability.
RESULTS: Twenty-three of 77 (30%) REMS programs used educational materials to communicate serious risks to patients, yielding a total of 27 REMS patient materials for analysis. The median readability score for these materials was at a ninth-grade reading level or higher. While most (89%) of these patient education materials met established criteria for being understandable, less than half (49%) were deemed actionable. DISCUSSION: Currently approved REMS patient materials fell short in terms of recommended reading level, and over half did not meet recommended standards for actionability. Developers of these materials should apply plain language principles when design these materials to improve their readability and to assess both understandability and actionability in order to increase the effectiveness when distributed to patients.

The goal of REMS patient materials (RPMs) is to communicate specific drug risks and safe drug‐use practices that the patient should follow in order to minimize the likelihood that the risk(s) will occur.

While most (89%) of these RPMs met established PEMAT criteria for being understandable, less than half (49%) were deemed actionable. The median readability score for these materials was at a ninth‐grade reading level or higher.

Current approved RPMs can be improved by improving both their readability and actionability.

INTRODUCTION

Providing health and safety information that is “accurate, accessible and actionable” is a stated objective of the US Department of Health and Human Services' National Action Plan to Improve Health Literacy.1, 2 Nowhere, arguably, is this objective more critical than in the context of prescription medication information. Indeed, poor‐quality patient drug information—that is, material that is written at too high a reading level is poorly designed and overly complicated—has been identified as a major cause of preventable medication errors in the United States.3, 4 The importance of high‐quality prescription drug information is further underscored by the fact that an estimated 80 million adults in America have limited health literacy.5 Health literacy status is strongly associated with an individual's ability to read, understand, and interpret written drug information and to perform medication safe use practices, all critical elements in assisting them to safely and appropriately use their prescription drugs.6, 7, 8, 9, 10, 11

While patients in the United States can receive information regarding their prescription drugs in a variety of ways, the main approved source is via the product label, including patient package inserts and medication guides (MGs).12 Beyond the product label, patients can also receive educational materials developed under the aegis of a risk evaluation and mitigation strategy (REMS), a type of drug safety program that the United States Food and Drug Administration (FDA) can impose on drug marketing authorization holders to ensure that the benefits of a drug or biological product outweigh its risks.13, 14, 15 Risk evaluation and mitigation strategy programs are designed to prevent or minimize the likelihood of occurrence of certain serious product‐related risks (eg, teratogenicity and posterior multifocal leukoencephalopathy). If FDA determines that a REMS is necessary, the agency may require one or more REMS elements. Such elements can include a MG, a patient package insert, a communication plan, or elements to assure safe use. The latter represents the most complex type of REMS programs and may include one or more of the following components: health care provider training and/or certification, documentation of safe use conditions, restrictions on product distribution, patient monitoring, or a patient registry.14, 15

The purpose of risk evaluation and mitigation strategy patient materials (RPMs) is to convey information regarding specific drug risks associated with the use of the medication, symptoms to watch for, and safe drug‐use practices that the patient should follow in order to minimize the likelihood that the risk(s) will occur. To date, RPMs have been predominately print‐based and have assumed a range of formats, including patient brochures, patient safety guides, safety information cards, and, in certain instances, MGs. Notably, with the exception of MGs, there are no published guidelines on how RPMs should be developed. However, there is a requirement that all RPMs be evaluated for effectiveness in the postmarketing context.14

Substantial evidence indicates that most written prescription drug information for patients is too complex.7, 8, 16, 17, 18 To date, studies examining the quality of REMS patient materials have focused exclusively on 1 type of material: the MG. Results have shown that MGs are difficult to read7, 19 and have mixed effectiveness in terms of their ability to increase patient knowledge about drug risks.20, 21 Little is known, however, about the effectiveness of a wide range of other types of RPMs including patient brochures, patient guides, and safety information cards. This study sought to address this gap via a comprehensive review and analysis of all RPMs available on the FDA REMS website as of March 2017, excluding MGs that have standard design requirements and hence are not readily modifiable.22 Counseling tools for healthcare providers were also not included because these materials are directed towards prescribers.23 We assessed key features of these RPMs, including their length, readability level, understandability, and actionability; all factors shown to affect the complexity, and hence, effectiveness of patient health information.24

METHODS

RPMs were identified via a review of the FDA's rems@fda.gov, a publicly available, FDA‐sponsored website, which provides the most definitive source of information regarding currently approved REMS programs. The website provides access to descriptions of the REMS programs as well as electronic copies of the approved REMS tools. In March 2017, 2 reviewers (H.W.C. and M.Y.S.) reviewed each of REMS programs on the FDA website to identify REMS patient materials. All types of REMS were included in the review, with the exception of MG only REMS. Figure 1 describes the total number of REMS programs included in the analysis and those that were excluded.

Figure 1

Total number of risk evaluation and mitigation strategy (REMS) programs included in the analysis. FDA, Food and Drug Administration [Colour figure can be viewed at wileyonlinelibrary.com]

Reading level analysis

Readability is a quantitative estimate of reading difficulty as measured by word and sentence difficulty. The reading level of each of the RPMs was assessing using 3 validated readability tools: The Lexile Measure, the Gunning Fogg Index, and the Flesch Kincaid.25, 26, 27 While each of these readability assessments are formulated differently, all examine the number of words per sentence and syllables per word. As there is no gold standard for readability assessments, and each measure has its own limitations, we took the average of these 3 commonly used measures to obtain a more robust estimate of readability level. This approach has been used in the literature evaluating readability of written health materials.28, 29 To prepare the text for analysis, portable document format of the written information was converted into word and text files. Many of the materials assessed included nontraditional text such as headings and bulleted phrases. As measures of reading level often include sentence length (number of words between periods) as one of the components in the score calculation, periods were inserted after each heading and at the end of each bulleted phrase to ensure that they would be treated as short sentences in the analysis. In addition, drug names were removed from the materials in the analysis as the repetition of drug names can inflate the reading level.

Lexile scores were calculated using the Lexile Analyzer, which assigns a score to each document.25 Scores range from 0 to 2000, and each score corresponds to student grade‐level reading norms. Each grade level contains a range of Lexile scores, and most Lexile scores are normal for multiple grade levels. To determine Lexile grade level in this analysis, we first determined all the grade levels in which each documents Lexile score corresponded. For each document, we then averaged the lowest and highest grade level to determine a final Lexile grade level. The Gunning Fog Index scores are equivalent to grade level. In this analysis, Gunning Fog grade‐level estimates were determined by averaging the score from 2 online analyzers (A.R. and H.W.C.).30, 31 Flesch Kincaid grade levels were determined using an established formula.27 Total words, total sentences, and average syllables per word were obtained from Text Content Analysis Tool on usingenglish.com website and entered into the formula.27, 30

Understandability and actionability analysis

The understandability and actionability of the RPMs were assessed using the Patient Education Materials Assessment Tools (PEMAT), a psychometrically validated instrument developed specifically for patient‐targeted educational materials.32 Understandability refers to how well a text can be interpreted by consumers from all backgrounds. It includes text difficulty, formatting, organization, and the quality or clarity of the messages being communicated. Actionability refers to the degree to which the reader knows how to act on the messages being communicated. The PEMAT has 2 versions: one for print and a second for audiovisual materials. The version for print materials (PEMAT‐P) consists of 24 items on 2 domains: understandability (17 items representing 6 topics) and actionability (7 items). The 6 items of understandability include: (1) content, (2) word choice and style, (3) use of numbers, (4) organization, (5) layout and design, and (6) use of visual aids. Similarly, the 7 actionability items assess whether the material: (1) clearly identified one or more actions for the patient to take; (2) addressed the user directly; (3) broke actions down into manageable steps; (4) provided a tangible tool to assist the patient in taking action; (5) provided simple instructions regarding how to perform calculations; (6) explained how to use the charts, graphs, tables, or diagram to take actions; and 7) used visual aids whenever possible.

According to the PEMAT instructions, each scale item is scored as either “Disagree” (0 points) or “Agree” (1 point), unless the item is deemed to be “not applicable” (N/A), in which case it is left unscored. All scale points are then summed and divided by the number of scored items for each scale (excluding the items that were scored N/A) and this value is then converted into a percentage. The calculation of the total possible percentage point excludes the items that were scored as N/A both in the numerator and denominator; therefore, items scored as N/A do not lower the overall scores for the understandability and actionability scales. The total score for each scale can range between 0% and 100%, with a higher percentage indicating that the material has higher understandability and/or actionability. For example, a material that received an understandability score of 90% would be deemed to be more understandable than a material that received an understandability score of 60%. A score of 70% or below is the threshold for determining whether a material is poorly understandable or actionable.29 Two reviewers (H.C. and M.Y.S.) independently analyzed and scored each RPM using the PEMAT‐P. Coauthors H.W.C. and M.Y.S. reviewed each of the RPMs and assigned PEMAT scores independently. Each set of scores was then compared and accepted if they were identical. In instances where there was a disagreement, A.R. reviewed the RPM and provided the tie‐breaker score.

Statistical analysis

All data were analyzed using R version 3.4.0. Interrater reliability for PEMAT scores was calculated using Cohen's kappa.33 Agreement levels were classified as follows: poor (0), slight (0.01‐0.20), fair (0.21‐0.40), moderate (0.41‐0.60), substantial (0.61‐0.80), or almost perfect (0.81‐1.0).34

RESULTS

As of March 31, 2017, there were 77 approved REMS programs available on the REMS@fda.gov website. The earliest REMS program in the REMS@fda.gov was approved in May of 2008. Of the 77 approved REMS programs, 45 [58%] included “Elements to Assure Safe Use (ETASU), 14 [18%] included only a Communication Plan,” and 18 [24%] included only the “Medication Guide” REMS element. Figure 1 shows the total number of REMS programs included in the analysis.

Twenty‐three of the 77 (30%) approved REMS programs featured patient‐targeted educational materials. These 23 REMS programs addressed a range of therapeutic areas including cardiovascular disease (CV), pulmonary/respiratory diseases, neurology, endocrinology, hematology, oncology, dermatology, urology, and bone. Table 1 lists the REMS programs included in the analysis, the REMS elements, RPMs, dates when the REMS program were approved and updated, and the therapeutic areas that they covered.

Table 1

REMS with RPMs included in study analysis (as of March 31, 2017)

	REMS Program	REMS Elements	REMS Patient Materials (RPMs) Title and Date Last Updated	Date REMS Approved	Date Last Updated	Therapeutic Area
1.	Adempas	MG, ETASU implementation system	Adempas guide for females who can get pregnant	10/8/2013	1/17/2017	Cardiovascular (CV/pulmonary/respiratory diseases)
2.	Alosetron	ETASU	Alosetron patient education sheet	5/4/2015	11/22/2016	Gastroenterology
3.	Aveed	ETASU, implementation system	What you need to know about Aveed treatment: A patient guide	3/5/2014	12/9/2016	Endocrinology
4.	Gattex	Communication plan, ETASU	Gattex patient caregiver counseling guide	12/21/2012	5/27/2016	Gastroenterology
5.	Juxtapid	ETASU, implementation system	Juxtapid patient guide	12/21/2012	1/3/2017	Cardiology/vascular diseases
6.	Lemtrada	MG, ETASU, implementation system	Lemtrada: A patient guide	11/14/2014	4/5/2016	Neurology
			Lemtrada infusion reactions A patient guide
			Lemtrada patient safety information card
7.	Letairis	MG, ETASU, implementation system	Letairis program guide for females who can get pregnant	5/29/2009	12/15/2016	Cardiovascular (CV/pulmonary/respiratory diseases)
8.	Lotronex	ETASU	Lotronex patient education sheet	9/2/2010	4/29/2016	Gastroenterology
9.	Natpara	ETASU, implementation system	Natpara patient brochure	1/23/2015	9/29/2016	Hematology
10.	Opsumit	MG, ETASU, implementation system	Opsumit guide for females who can get pregnant	10/18/2013	2/22/2016	Cardiovascular (CV/pulmonary/respiratory diseases)
11.	Pomalyst	MG, ETASU, implementation system	Pomalyst patient guide	2/8/2013	4/22/2016	Hematology/oncology
12.	Probuphine	MG, ETASU, implementation system	Probuphine patient guide last update:	5/26/2016	6/14/2016	Psychiatry/neurology
13.	Prolia	MG, communication plan	Prolia REMS patient brochure	6/1/2010	5/21/2015	Bone
14.	Qysmia	MG, ETASU, implementation system	Qsymia risk of birth defects with Qsymia patient brochure	7/17/2012	9/26/2014	Endocrinology
15.	Revlimid	ETASU, implementation system	Revlimid patient guide	8/3/2010	4/22/2016	Hematology/oncology
16.	Sabril	ETASU, implementation system	Sabril: What you need to know about SABRIL treatment: A Patient_Guide	8/29/2009	6/27/2016	Neurology
17.	Soliris	MG, ETASU	Soliris: Patient safety brochure important safety information for patients	6/4/2010	1/13/2017	Hematology
			Soliris patient safety card
18.	Thalomid	ETASU, implementation system	THAL patient guide	8/3/2010	4/22/2016	Dermatology, hematology oncology
19.	Tracleer	MG, ETASU, implementation system	Tracleer REMS guide for patients	8/7/2009	12/16/2016	Cardiovascular (CV/pulmonary/respiratory diseases)
20.	Xiaflex	ETASU, implementation system	Xiaflex patient guide	2/2/2010	11/28/2016	Urology
21.	Xyrem	MG, ETASU, implementation system	Xyrem patient quick start guide	2/27/2015	7/15/2015	Neurology
22.	Zinbryta	MG, ETASU, implementation system	Zinbryta patient guide	5/27/2016	5/27/2016	Neurology
			Zinbryta patient wallet card
23.	Zydelig	Communication plan	Zydelig patient safety information card	7/23/2014	1/4/2017	Hematology

Abbreviations: ETASU, elements to assure safe use; MG, medication guide; REMS, risk evaluation and mitigation strategy; RPM, risk evaluation and mitigation strategy patient materials.

A total of 27 RPMs were analyzed: 20 REMS programs had 1 RPM each, 2 REMS had 2 RPMs each, and 1 REMS program had 3 RPMs. The RPMs assumed a variety of formats, including patient brochures, patient guides, educational sheets, wallet cards, and patient safety information cards. The length of the materials ranged from 1 to 25 pages, with the median page length of 2 (interquartile range, 2‐8.5) (Table 2). There was no apparent association between the length of the RPMs and the number of risks. For example, RPMs to address risk of serious birth defects ranged in length from 2 to 11 pages long.

Table 2

List of RPMs, risks addressed, average PEMAT scores, average grade level, and page length

RPMs	Risks Addressed	Average Understand‐Ability (%)	Average Action‐Ability (%)	Average Grade Level of Material	Page Length
Adempas for females who can get pregnant	1. Serious birth defects	65.5	50.0	8	8
Alosetron patient education sheet	1. Constipation 2. Ischemic colitis	73.5	60.0	10	2
Aveed what you need to know about Aveed treatment: A patient guide	1. Pulmonary oil microembolism (POME) 2. Severe allergic reaction	74.0	86.0	8	2
Gattex patient caregiver counseling guide	1. Cancer	74.0	45.0	7	2
Juxtapid patient guide	1. Liver problems	61.5	60.0	10	4
Lemtrada A patient guide	1. Infusion reactions 2.Autoimmune conditions 3.Malignancies	88.5	71.5	13	12
Lemtrada infusion reactions A patient guide	1. Serious infusion reactions	83.0	50.0	10	2
Lemtrada patient safety information card	1. Autoimmune conditions 2. Infusion reactions 3. Malignancies	87.0	40.0	10	2
Letairis program guide for females who can get pregnant	1. Serious birth defects	82.5	83.0	8	8
Lotronex patient education sheet	1. Constipation 2. Ischemic colitis	71.5	60.0	12	2
Natpara patient brochure	1. Bone cancer	71.0	20.0	8	1
Opsumit guide for females who can get pregnant	1. Serious birth defects	78.0	83.0	7	8
Pomalyst patient guide	1. Severe life‐threatening birth defects	89.0	83.0	8	11
Probuphine patient guide	1. Complications of migration, protrusion, expulsion, and nerve damage associated with the insertion and removal of implants 2. Accidental overdose, misuse and abuse	87.5	70.0	9	2
Prolia REMS patient brochure	1. Low calcium levels 2. Severe jaw bone problems 3. Unusual thigh fractures 4. Serious infections 5. Skin problems	87.0	80.0	9	2
Qsymia risk of birth defects with Qsymia patient brochure	1. Birth defects	87.5	83.0	12	2
Revlimid patient guide	1. Birth defects 2. Low white blood cells and low platelets 3. Blood clots	86.0	83.0	10	11
Sabril what you need to know about SABRIL treatment A patient guide	1. Permanent vision damage	88.0	60.0	10	2
Soliris patient safety brochure important safety information for patients	. Meningococcal infection	89.5	60.0	11	10
Soliris patient safety card	1. Meningococcal infection	90.5	55.0	10	2
THAL patient guide	1. Birth defects 2. Blood clots	90.5	83.0	10	10
Tracleer REMS guide for patients	1. Liver damage 2. Serious birth defects	85.5	83.0	9	11
Xiaflex patient guide	1. Penile fracture	90.5	90.0	10	2
Xyrem patient quick start guide	1. Significant CNS and respiratory depression 2. Contraindication of use of XYREM with sedative hypnotics and alcohol 3. The potential for abuse, misuse, and overdose associated with XYREM 4. The safe use, handling, and storage of XYREM	87.0	65.0	11	25
Zinbryta patient guide	1. Liver problems 2. Immune system problems	84.0	83.0	9	8
Zinbryta patient wallet card	1. Liver problems 2. Immune system problems	69.5	60.0	9	2
Zydelig patient safety information card	1. Hepatotoxicity 2. Severe diarrhea or colitis 3. Pneumonitis 4. Infections 5. Intestinal perforation	89.0	60.0	9	2
Mean		81.89	66.91	9.60	5.74
Standard error		1.65	3.30	0.27	1.05
Median		86	65	9.67	2

Abbreviations: PEMAT, Patient Education Materials Assessment Tool; RPM, risk evaluation and mitigation strategy patient materials.

Readability

The median readability score corresponded to a 9th‐grade reading level (interquartile range, 8‐10; range, 6‐13). Two‐thirds of REMS patient materials (67%) were assessed to be at a 9th‐grade reading level or higher. Three of the 27 REMS patient materials (11%) attained a readability score of 12th grade or above. Figure 2 shows percent distribution of median readability scores for REMS patient materials by grade level.

Figure 2

Distribution of readability scores for REMS patient materials by grade level

Understandability

The summary of understandability of the patient materials is provided in Table 2. Figure 3 shows the distribution of PEMAT understandability and actionability scores for RPMs.

Figure 3

Boxplots representing the median, upper/lower quartiles, and range (upper/lower bounds) of understandability and actionability score for REMS patient materials (RPMs)

Content

Seventeen of the 27 RPMs (63%) made their purpose completely evident. All of the RPMs avoided including information or content that distracted from the material's purpose.

Word choice and style

All RPMs used common, everyday language and the active voice. However, 3 RPMs (11%) used medical terms that were not defined in such a way as to be easily comprehensible for patients, (eg, homozygous familial hypercholesterolemia, heterozygous familial hypercholesterolemia, pulmonary oil microembolism, and meningococcal infections).

Use of numbers

Nineteen of the RPMs (70%) included no numbers or statistical information of any kind. When numbers were used, 6 of 8 RPMs (75%) had numbers appearing in the material in clear and easy way to understand. None of the RPMs required the user to perform calculations.

Organization

Twenty‐five of the RPMs (93%) “chunked” information in short sections and included informative headers. Similarly, the majority of RPMs (93%) presented information in a logical sequence. Four of the RPMs do not require a summary because these materials were very short. Of the remaining 22 RPMs, the majority (67%) lacked a summary.

Layout and design

All RPMs incorporated visual cues to help draw the patients' attention to key points in the material. Examples of visual cues that were used included arrows, boxes, bullets, bolding, different color font, larger font, and highlighting. Of the 27 RPMs, only 2 RPMs were in black and white, and the rest used a limited palate of colors (2‐5 colors).

Use of visual aids

Only 10 (37%) of the RPMs used visual aids. Seven visual aids were judged as serving to reinforce, rather than distract from the content, and all 8 included illustrations and photographs that were clear and uncluttered. However, only 6 of the 8 RPMs that used visual aids also used clear titles or captions. Twelve (44%) of the RPMs used simple tables with short and clear row and column headings.

Overall, the median understandability score was 85% (interquartile range, 74%‐88%; range, 62%‐90%). Twenty‐four of the 27 RPMs (89%) met or exceeded the 70% threshold for understandability. Figure 4 shows percentage of RPMs with understandability and actionability scores falling above or below the acceptance threshold of 70%.

Figure 4

Percentage of REMS patient materials with understandability and actionability scores falling above or below the acceptance threshold of 70%

Actionability

All RPMs clearly identified at least 1 action the user could take, 26 of 27 addressed the reader directly when describing actions (96%), and 25 of 28 broke actions down into manageable, explicit steps (93%). Only 13 of 27 materials (48%) provide a tangible tool such as checklist to assist the user in taking action. Only 1 of 27 RPMs (4%) provided simple instructions or examples of how to perform calculations; the rest were judged to be “not applicable” because the RPMs have no calculations. Eleven RPMs explains how to use the charts, graphs, tables, or diagrams to take action. Visual aids were rarely used (5 of 27, 19%) to make it easier to act on the instructions.

Overall, the median actionability score was 64% (interquartile range, 60%‐83%; range, 20%‐90%). Thirteen of the 27 REMS patient materials (48%) met or exceeded the 70% threshold for actionability. Table 3 shows aggregate score of RPMs based on individual PEMAT‐P items.

Table 3

Aggregate score of REMS patient materials based on PEMAT‐P items (n = 27)

	All Materials n = 27 (%)
PEMAT Item	Agree	Disagree	Not Applicable
Understandability
Topic 1: Content
1. The material makes its purpose completely evident.	17 (63)	10 (37)	0 (0)
2. The material does not include information or content that distracts from its purpose.	27 (100)	0 (0)	0 (0)
Topic 2: Word Choice & Style
3. The material uses common, everyday language.	27 (100)	0 (0)	0 (0)
4. Medical terms are used only to familiarize audience with the terms. When used, medical terms are defined.	24 (89)	3 (11)	0 (0)
5. The material uses the active voice.	27 (100)	0 (0)	0 (0)
Topic 3: Use of numbers
6. Numbers appearing in the material are clear and easy to understand.	6 (22)	2 (7)	19 (70)
7. The material does not expect the user to perform calculations.	27 (100)	0 (0)	0 (0)
Topic 4: Organization
8. The material breaks or “chunks” information into short sections.	25 (93)	0 (0)	2 (7)
9. The material's sections have informative headers.	25 (93)	0 (0)	2 (7)
10. The material presents information in a logical sequence.	27 (100)	0 (0)	0 (0)
11. The material provides a summary.	5 (19)	18 (67)	4 (15)
Topic 5: Layout & Design
12. The material uses visual cues (eg, arrows, boxes, bullets, bold, larger font, highlighting) to draw attention to key points.	27 (100)	0 (0)	0 (0)
Topic 6: Use of visual aids
15. The material uses visual aids whenever they could make content more easily understood (eg, illustration of healthy portion size).	10 (37)	17 (63)	0 (0)
16. The material's visual aids reinforce rather than distract from the content.	7 (26)	1 (4)	19 (70)
17. The material's visual aids have clear titles or captions.	6 (22)	2 (7)	19 (70)
18. The material uses illustrations and photographs that are clear and uncluttered.	8 (30)	0 (0)	19 (70)
19. The material uses simple tables with short and clear row and column headings.	12 (44)	0 (0)	15 (56)
Actionability
20. The material clearly identifies at least one action the user can take.	27 (100)	0 (0)	0 (0)
21. The material addresses the user directly when describing actions.	26 (96)	1 (4)	0 (0)
22. The material breaks down any action into manageable, explicit steps.	25 (93)	2 (7)	0 (0)
23. The material provides a tangible tool (eg, menu planners, checklists) whenever it could help the user take action.	13 (48)	14 (52)	0 (0)
24. The material provides simple instructions or examples of how to perform calculations.	1 (4)	0 (0)	26 (96)
25. The material explains how to use the charts, graphs, tables, or diagrams to take actions.	11 (41)	1 (4)	15 (56)
26. The material uses visual aids whenever they could make it easier to act on the instructions.	5 (19)	22 (81)	0 (0)

Abbreviations: PEMAT‐P, Patient Education Materials Assessment Tool for printable materials; REMS, risk evaluation and mitigation strategy.

Interrater reliability of the PEMAT

Interrater reliability was substantial (κ = 0.73).

DISCUSSION

This study sought to assess the quality of currently approved REMS educational materials for patients. Specifically, we assessed the readability, understandability, and actionability of these materials using well‐validated, state‐of‐the‐art instruments. To our knowledge, this is the first study of its kind to do so. Our study results add to the existing literature examining the quality and effectiveness of other types of patient education materials.35, 36 Consistent with these other studies, our results highlighted important shortcomings in the quality of these materials. Specifically, while we found that the majority of patient‐targeted REMS materials were meeting recommended standards in terms of understandability (89% met or exceeded the 70% threshold for understandability), they failed to do so in terms of readability and actionability.

Increasing readability typically leads to improvements in understanding.37 However, our data indicated that patient education materials with higher grade‐level readability could still be understandable. Readability score is a quantitative estimate of reading difficulty measured by word and sentence complexity. Using individual words and sentences that are easy for patients to read is a necessary but not sufficient condition for ensuring that they can understand and use educational materials.38, 39 Understandability is a broader and more qualitative construct that encompasses the content of the material, word choice and style, use of numbers, organization, layout and design, and incorporation of visual aids. For individuals with low literacy, visual design and appeal are particularly important40 and increase the likelihood that the health information will be read.41 In sum, readability and understandability are complementary and therefore should be assessed separately.

Our results highlight opportunities for improving the quality of REMS patient materials. Many patients experience difficulty understanding health materials written in technical language, especially those with low health literacy skills. The National Cancer Institute recommends that materials for low health literacy audiences address elements beyond just readability, including, for example, writing style, vocabulary, typography, layout, graphics, and color.39, 42 Other recommendations for low health literate audiences include limiting the length of the educational materials and ensuring that they are written at a fifth‐ to eighth‐grade reading level.39 Risk evaluation and mitigation strategy programs featuring more than 1 type of patient educational material should consider scaling back to a single document or tool. Similarly, the use of medical terminology should be kept to a minimum, and if used, definitions should be provided.43 Summaries and visual aids can also enhance the understandability of these materials and increase the likelihood that they are consistent with best practice recommendations.44

These recommendations are supported by previous research that has shown that patients, when confronted with multiple types of prescription drug information, can become confused or overwhelmed and, as a result, may not read any of the provided materials.45, 46 Further, limiting information to only serious and actionable risks has been shown to improve overall risk recall and recognition.47 Simple representations of risks in REMS materials can also help health care providers communicate safety information to patients more effectively and, potentially, may make REMS consultations more efficient.48 Lastly, while most of the RPMs in this study omitted numeric and statistical information of any kind, presenting data using absolute risk and frequencies has been shown to improve the accuracy of interpretation by patients.49, 50, 51

Several limitations of this research are worth noting. First, reading level measures are designed for use with text that is formatted traditionally, such as books, or newspaper and journal articles. As a result, existing readability instruments are unable to accommodate nontraditional text (e.g., nonsentences such as headings and bullets) such as was commonly found in the RPMs we reviewed. The appropriate method in readability determination is to include nonsentences in the analysis. Since bullet points and headings usually consist of a few words, analyzing the text without modifying nonsentences may artificially lower the readability scores.52 Second, the 70% cutoff for the PEMAT scale scores for understandability and actionability has limited empirical basis.32 We are not aware on any consensus of any standardize acceptable scores for PEMAT. Third, there are other aspects of good design for print materials, such as the inclusion of color, the use of 11 point serif font type, and tailoring information to reflect key characteristics of the reader, which were not systematically assessed in this analysis.53

CONCLUSIONS

In conclusion, while most REMS educational materials for patients were found to be understandable, they had important shortcomings in regard to their readability and actionability. When patients receive well‐designed drug safety information, they are more likely to read it, remember it, and act on it. Higher quality drug safety information for patients benefits everyone including patient with low health literacy. Specifically, patients benefit in terms of improved personal safety when using medicines, and the healthcare system and society as a whole benefit in terms of lower drug‐related mortality and morbidity, and reduced healthcare utilization and associated costs. Incorporating best practice standards into the development of patient‐targeted drug safety information, such as described in this study, represents a practical and highly feasible approach to improving the quality of these important educational materials.

ETHICS STATEMENT

The authors state that no ethical approval was needed.

CONFLICT OF INTEREST

The views expressed in this publication represent those of the authors and do not necessarily represent the views or practices of the authors' employer or any other third party.