Worth the paper they are printed on? Findings from an independent evaluation of the understandability of patient information leaflets for antiseizure medications.

OBJECTIVE: The Patient Information Leaflet (PIL) is an authoritative document that all people with epilepsy in the EU receive when prescribed antiseizure medication (ASM). We undertook the first independent, comprehensive assessment to determine how understandable they are. Regulators state that when patients are asked comprehension questions about them, ≥80% should answer correctly. Also, recommended is that PILs have a maximum reading requirement of US grade 8.
METHODS: Study 1: We obtained 140 current ASM PILs written in English. "Readability" was assessed using four tests, with and without adjustment for influence of familiar, polysyllabic words. A total of 179 online materials on epilepsy were also assessed. Study 2: Two PILs from Study 1 were randomly selected (Pregabalin Focus; Inovelon) and shown to 35 people from the UK epilepsy population. Their comprehension was assessed. Study 3: To understand whether the student population provides an accessible alternative population for future examination of ASM PILs, Study 3 was completed, using the same methods as Study 2, except that participants were 262 UK university students.
RESULTS: Study 1: No PIL had a reading level of grade 8. Median was grade 11. Adjusting for context, the PILs were still at grade 10.5. PILs for branded ASMs were most readable. PILs were no more readable than (unregulated) online materials. Study 2: Users struggled to comprehend the PILs' key messages. The eight questions asked about pregabalin were typically answered correctly by 54%. For Inovelon, it was 62%. Study 3: Most student participants comprehended the PILs' key messages. The questions about Inovelon were answered correctly by 90%; for pregabalin it was 86%. SIGNIFICANCE: This is the first independent and comprehensive examination of ASM PILs. It found that PILs being used fail to meet recommendations and regulatory requirements and risk not being understandable to a substantial proportion of users. In finding that people from the epilepsy population differ markedly in comprehension of PILs compared to students, this study highlights the importance of completing user testing with the target population.

The PIL, as the only document all PWE in the European Union prescribed ASMs routinely receive, could be key to self‐management

No independent evidence is available on the understandability of ASM PILs

We found none of the 140 PILs for ASMs used in the UK met the recommended maximum for reading age

When PWE were shown two of the PILs, they struggled to comprehend their key messages on how to safely and effectively use the related ASM

Only two of 16 comprehension questions asked of PWE were answered by a sufficient proportion to satisfy the threshold regulators recommend

INTRODUCTION

People with epilepsy (PWE) and their significant others assume substantial responsibility for the management of epilepsy. To make informed decisions, they need understandable information. The Patient Information Leaflet (PIL), which has accompanied medicines in the EU since 1999, forms an authoritative document all people receive about their antiseizure medication (ASM). How understandable are they?

Most economically developed nations include a significant minority of people with low literacy levels. In England, 15% of adults have a literacy level at or below that of an 11‐year‐old. A further 28% have a reading age of a 12–14‐year‐old. When information exceeds someone's literacy level, there is the potential for misunderstandings. It has been recommended written materials have a maximum required reading age of ~13 years (US grade 8).

We systematically searched for studies examining ASM PILs. Five , , , , were identified (File S1). Only one examined European PILs. Conducted by Wong, it focused on the PILs for 12 branded ASMs written in English. The length of the sentences used and the complexity of the words within them were quantified. Based on this assessment and comparison to reference data, Wong concluded the PILs should be relatively understandable to UK adults, with the text being classified as representing "plain English." However, published in 1998 and focusing on only branded ASMs, Wong's study tells us little about the understandability of current PILs.

Within the EU, the European Medicines Agency (EMA), and national competent authorities, approve PILs before use. They state PILs should be designed and worded so a maximum number of people can understand them. Since 2005, manufacturers have also legally been required to engage with users to develop their PILs.

A debate is occurring as to whether the EMA's processes are sufficient. , It would be appropriate for the epilepsy community to contribute. In England alone, in the 12 months from December 1, 2020, there were >30 million ASM prescriptions.

What evidence is needed on ASM PILs?

Different methods are available to determine how understandable ASM PILs are.

Readability

One way to gauge how understandable ASM PILs are is by using automated readability tests. These use different text characteristics to estimate "reading ease." Word length is used as a proxy for semantic difficulty, and sentence length indicates syntactic complexity. A numeric value is assigned to the text to indicate its "readability."

A standard application of readability tests to ASM PILs would provide evidence on their ease of use in a common format and allow comparison to PILs for other medications. It would be helpful, however, to also apply them while adjusting for context. This is because for many readability tests, the more polysyllabic words present within a document, the higher the judged required reading age. The challenge is that many polysyllabic words within a PIL (e.g., convulsion, levetiracetam) may be uniquely familiar to the target audience and so be poor predictors of readability. Without adjusting for this, a test might artificially inflate required reading age.

Regulators have made a PIL template available to manufacturers and stipulate standard headings. Some commentators contend this inadvertently reduces the readability of PILs. , In assessing ASM PILs, it would thus be insightful to compare their readability to materials on epilepsy written in English for PWE, whose presentation is less regulated. Online materials meet these criteria.

Literal comprehension

A second way to assess the PILs would be by determining their comprehensibility. Readability does not guarantee comprehension, as factors beyond text characteristics affect it (e.g., prior knowledge, interest, how information is presented).

No published evidence on the comprehensibility of ASM PILs is available. It could be obtained by completing so‐called "user testing." User testing (as per the Australian‐Sless method ) involves a PIL being given to ~20 individuals from the target population. They are asked questions to assess the PIL's ability to ensure people can find and understand information pertinent to the medicine's safe and effective use. Regulators cite it as a way manufacturers can demonstrate their PIL is ready for use, stating that a PIL should be iteratively refined and retested until each question is answered correctly by ≥80% of users.

User testing is resource intensive, as there is a need to recruit people from the target population. When they have a stigmatizing condition, this can be challenging. Funding for researchers undertaking user testing is also not forthcoming. To position the epilepsy community to independently check ASM PILs in the future, it would be helpful to understand whether PILs could be tested with populations more accessible to academic investigators. One that they can recruit from in large numbers is the student population. To be a suitable alternative for testing, the student population's pattern of comprehension results would need to be broadly indicative of those of the epilepsy population.

Objectives of current study

Given the information gaps identified, we conducted a series of studies that sought to:

Describe the readability of current PILs for ASMs prescribed in the UK (Study 1);

Explore factors associated with their readability (Study 1);

Compare the readability of PILs to online epilepsy materials (Study 1);

Complete user testing of a sample of ASM PILs to understand their comprehensibility to persons from the epilepsy population (Study 2); and

Complete user testing of the same ASM PILs, but with a larger sample of persons from the student population (Study 3).

MATERIALS AND METHODS

Study 1: Readability of PILs

Design

Study 1 was a cross‐sectional assessment of PILs for ASMs.

Materials

Patient information leaflets

In the UK, 27 active ingredients are approved for epilepsy (Table 1). The PILs for the 148 medications containing them were obtained on October 6, 2020 from the Electronic Medicines Compendium; 140 (94.6%) were in a format that permitted testing. File S2 lists them.

TABLE 1

Patient Information Leaflets for antiseizure medications that were tested and their readability score by their active ingredient

Active ingredient	F‐K grade level, median (IQR)		FORCAST grade level, median (IQR)		SMOG grade level, median (IQR)		Median grade level, median (IQR)		FRE, median (IQR)
	Application		Application		Application		Application		Application
	Standard	Adjusted for context	Standard	Adjusted for context	Standard	Adjusted for context	Standard	Adjusted for context	Standard	Adjusted for context
Acetazolamide	9.5	7.9	10.7	10.3	9.5	7.9	10.7	10.3	53.0	63.0
Brivaracetam	7.8 (7.7)	6.5 (6.4)	10.4 (10.3)	9.8 (9.8)	7.8 (7.7)	6.5 (6.4)	10.4 (10.3)	9.8 (9.8)	62.5 (61.0)	70.5 (70.0)
Carbamazepine	7.8 (7.7)	6.5 (6.4)	10.3 (10.3)	9.8 (9.8)	7.8 (7.7)	6.5 (6.4)	10.3 (10.3)	9.8 (9.8)	62.5 (61.0)	70.5 (70.0)
Cannabidiol	9.4	8.0	10.4	10.2	9.4	8.0	10.4	10.2	56.0	65.0
Clobazam	8.1 (8.1–8.8)	7.1 (7.0–7.5)	11.0 (10.9–11.1)	10.4 (10.4–10.6)	8.1 (8.1–8.8)	7.1 (7.0–7.5)	11.0 (10.9–11.1)	10.4 (10.4–10.6)	57.0 (54.0–58.0)	63.0 (61.5–64.0)
Clonazepam	7.6 (7.6–7.6)	6.6 (6.6)	10.6 (10.6)	10.1 (10.1)	7.6 (7.6–7.6)	6.6 (6.6)	10.6 (10.6)	10.1 (10.1)	62.0 (61.0)	68.0 (67.0)
Eslicarbazepine acetate	7.6 (7.6–7.6)	6.6 (6.6)	10.6 (10.6)	10.1 (10.1)	7.6 (7.6–7.6)	6.6 (6.6)	10.6 (10.6)	10.1 (10.1)	62.0 (61.0)	68.0 (67.0)
Ethosuximide	7.6 (7.6–7.6)	6.6 (6.6)	10.6 (10.6)	10.1 (10.1)	7.6 (7.6–7.6)	6.6 (6.6)	10.6 (10.6)	10.1 (10.1)	62.0 (61.0)	68.0 (67.0)
Gabapentin	10.1 (10.1–10.5)	8.4 (8.4–8.6)	11.3 (11.2–11.4)	10.7 (10.6–10.7)	10.1 (10.1–10.5)	8.4 (8.4–8.6)	11.3 (11.2–11.4)	10.7 (10.6–10.7)	48.0 (46.0–49.0)	58.0 (57.3–59.0)
Lacosamide	10.1 (10.1–10.5)	8.4 (8.4–8.6)	11.3 (11.2–11.4)	10.7 (10.6–10.7)	10.1 (10.1–10.5)	8.4 (8.4–8.6)	11.3 (11.2–11.4)	10.7 (10.6–10.7)	48.0 (46.0–49.0)	58.0 (57.2–59.0)
Lamotrigine	10.3 (10.0–10.4)	8.3 (8.2–8.3)	11.1 (10.9–11.1)	10.3 (10.3–10.4)	10.3 (10.0–10.4)	8.3 (8.2–8.3)	11.1 (10.9–11.1)	10.3 (10.3–10.4)	49.0 (48.7–50.2)	61.0 (60.7–61.0)
Levetiracetam	11.1 (10.8–11.5)	8.8 (8.7–8.9)	11.5 (11.3–11.5)	10.6 (10.6–10.8)	11.1 (10.8–11.5)	8.8 (8.7–8.9)	11.5 (11.3–11.5)	10.6 (10.6–10.8)	41.0 (38.5–43.5)	56.0 (55.0–57.0)
Oxcarbazepine	10.2 (10.0)	8.7 (8.3)	11.2 (11.1)	10.5 (10.5)	10.2 (10.0)	8.7 (8.3)	11.2 (11.1)	10.5 (10.5)	49.0 (45.0)	58.0 (56.0)
Perampanel	9.2 (9.2–9.2)	7.7 (7.7)	10.8 (10.8)	10.3 (10.3)	9.2 (9.2–9.2)	7.7 (7.7)	10.8 (10.8)	10.3 (10.3)	55.0 (55.0–55.0)	63.5 (63.0)
Phenobarbital	11.2 (9.9)	9.2 (8.7)	11.8 (11.6)	11.2 (11.2)	11.2 (9.9)	9.2 (8.7)	11.8 (11.6)	11.2 (11.2)	39.0 (39.0)	52.0 (49.0)
Phenytoin	11.4 (11.1–11.8)	9.7 (9.5–10.2)	11.5 (11.3–11.6)	10.9 (10.7–11.0)	11.4 (11.1–11.8)	9.7 (9.5–10.2)	11.5 (11.3–11.6)	10.9 (10.7–11.0)	41.5 (40.2–42.7)	51.5 (48.7–52.7)
Piracetam	7.5 (7.5)	6.2 (6.2)	10.8 (10.8)	10.1 (10.1–10.1)	7.5 (7.5)	6.2 (6.2)	10.8 (10.8)	10.1 (10.1–10.1)	61.0 (57.0)	69.0 (67.0)
Pregabalin	10.1 (9.8–10.3)	7.9 (7.8–8.1)	11.6 (11.5–11.7)	10.8 (10.7–10.8)	10.1 (9.8–10.3)	7.9 (7.8–8.1)	11.6 (11.5–11.7)	10.8 (10.7–10.8)	46.0 (45.0–49.0)	59.0 (58.0–60.0)
Primidone	11.2	10.1	11.6	11.1	11.2	10.1	11.6	11.1	39.0	45.0
Rufinamide	8.6 (8.6–8.6)	7.3 (7.3)	10.6 (10.6–10.6)	10.1 (10.1–10.1)	8.6 (8.6–8.6)	7.3 (7.3)	10.6 (10.6–10.6)	10.1 (10.1–10.1)	59.0 (59.0–59.0)	66.0 (66.0–66.0)
Sodium valproate ± valproic acid	9.3 (9.1–9.6)	7.5 (7.4–7.7)	11.1 (10.9–11.3)	10.3 (10.3–10.4)	9.3 (9.1–9.6)	7.5 (7.4–7.7)	11.1 (10.9–11.3)	10.3 (10.3–10.4)	53.0 (52.0–55.0)	64.0 (63.0–64.0)
Stiripentol	9.2 (9.1)	7.8 (7.7)	10.7 (10.7)	10.2 (10.2)	9.2 (9.1)	7.8 (7.7)	10.7 (10.7)	10.2 (10.2)	53.0 (52.0)	61.0 (60.0)
Tiagabine	–	–	–	–	–	–	–	–	–	–
Topiramate	8.8 (8.6–8.9)	7.0 (6.9–7.2)	11.2 (11.1–11.4)	10.6 (10.5–10.7)	8.8 (8.6–8.9)	7.0 (6.9–7.2)	11.2 (11.1–11.4)	10.6 (10.5–10.7)	54.0 (53.0–54.2)	64.0 (62.7–65.0)
Valproic acid	9.5	8.1	11.1	10.5	9.5	8.1	11.1	10.5	52.0	61.0
Vigabatrin	8.5 (8.3)	7.6 (7.3)	11.0 (10.6)	10.7 (10.2)	8.5 (8.3)	7.6 (7.3)	11.0 (10.6)	10.7 (10.2)	57.0 (57.0)	62.0 (62.0)
Zonisamide	8.8 (8.5–9.0)	7.1 (7.0–7.2)	10.5 (10.5–10.6)	10.0 (9.9–10.0)	8.8 (8.5–9.0)	7.1 (7.0–7.2)	10.5 (10.5–10.6)	10.0 (9.9–10.0)	57.0 (55.5–58.5)	67.0 (66.5–67.0)
Overall	9.9 (8.9–10.5)	8.0 (7.4–8.7)	11.3 (10.9–11.5)	10.6 (10.3–10.7)	9.9 (8.9–10.5)	8.0 (7.4–8.7)	11.3 (10.9–11.5)	10.6 (10.3–10.7)	50.0 (45.0–55.0)	60.0 (57.0–64.0)

Note: Median grade score is calculated on the basis of F‐K, FORCAST, and SMOG.

Abbreviations: F‐K, Flesch–Kincaid reading grade score; FORCAST, FORCAST reading grade score; FRE, Flesch Reading Ease score (0–100, higher scores indicate easier to read); IQR, interquartile range; PIL, Patient Information Leaflet; SMOG, Simple Measure of Gobbledygook reading grade score. Where IQR is not present or limited to one number this indicates a lack of range in scores around the median. Often this was beacuse there were only a small number of PILs in the sample with this active ingredient.

Online materials to compare to PILs

A representative sample of 179 online epilepsy materials encountered by PWE was compiled (median word count = 1179, interquartile range [IQR] = 695–1916). File S3 lists them and their identification. In brief, five internet searches were completed on October 6, 2020 using search terms PWE use (i.e., “epilepsy symptoms”, “what is epilepsy”, “epilepsy seizures”, “epilepsy UK” and “epilepsy medication”). Two reviewers independently screened the first 100 results from each search to identify eligible materials.

Preparation of materials for testing

Individual Microsoft Word versions of the PILs and online materials were created. No pictures, symbols, copyright notices, citations, advertisements, or internet addresses were included.

In line with standard practice, the PIL versions included text from five of the six sections that form a PIL in the UK and EU: "1. What X is and what it is used for"; "2. What you need to know before you X"; "3. How to X"; "4. Possible side effects"; and "5. How to store X." We excluded Section 6 ("Contents of the pack and other information"), because patients do not rate the information contained within it as particularly important. ,

The Word versions of the online materials included only text from the landing page.

Tests

The following established tests that consider different text characteristics and estimate years of US schooling required were used: Simple Measure of Gobbledygook (SMOG), Flesch–Kincaid (F‐K), and FORCAST. As per previous studies, , a composite score for each document was formed based on its median score on the tests. It can be broa converted to UK reading age by adding 5.

To provide a measure of readability on a continuous scale, the Flesch Reading Ease (FRE) test was also used. It ranges from 0 to 100; higher scores indicate greater readability. File S4 details the formulae of each test.

Tests were completed using Readability Studio Professional Edition (v2019). They were first run in the standard way and then while adjusting for context (see below).

Adjustments for context

A list of words was compiled for exclusion from consideration by the testing software. It comprised the n = 59 words that create the generic and branded names of the ASMs, n = 78 key epilepsy terms, and n = 1464 adverse event terms. File S5 details them and the rationale. The testing software was also instructed to exclude proper nouns and to treat all numerals as monosyllabic words.

Analysis

As the readability data were not normally distributed (Shapiro–Wilk, p < .01), analyses were completed using nonparametric tests (Mann–Whitney, Wilcoxon signed rank test, Spearman rank test). Central tendency is described according to the median and IQR. The proportion of PILs satisfying the recommended reading rade 8 level is described.

Factors explored for their association with PIL readability were: time since the ASMs focused on had been authorized for use and time since the PIL examined had been revised ; whether the ASM was branded or generic ; and extent to which the ASM was prescribed, with PILs for the three most commonly prescribed ASMs in England (lamotrigine, levetiracetam, valproate) being compared to the others. These analyses were completed using data from when the readability tests were adjusted.

For the main analyses, alpha was set at .05. When exploring factors associated with readability, alpha was Bonferroni adjusted (p < .006).

Analyses were conducted using SPSS (v27).

Study 2: User testing with people from the epilepsy population

An anonymous, cross‐sectional online survey was run using Qualtrics. To minimize participant burden, we tested comprehension of two PILs. Order of presentation was randomized (1:1).

Recruitment

As is standard for user testing, a sample of ~n = 20 users was sought, while recognizing a need to account for potential missing data.

Between November 2021 and February 2022, a participant advertisement was distributed using different social media platforms by UK epilepsy user groups (see Acknowledgments). Table 2 shows the eligibility criteria and approvals.

TABLE 2

Participant inclusion and exclusion criteria

Study 2: epilepsy population	Study 3: student population
Aged ≥16 years (no upper limit)	Aged ≥16 years (no upper limit)
Lives in the UK	Lives in the UK
Able to provide informed consent	Able to provide informed consent
Able to independently read and write in English	Able to independently read and write in English
Self‐report a clinical diagnosis of epilepsy (any syndrome or seizure type) OR be close family member or friend (significant other) to someone with epilepsy
Ineligible: Severe current psychiatric disorders (e.g., acute psychosis) Terminal medical illness	Ineligible: Severe current psychiatric disorders (e.g., acute psychosis Terminal medical illness

Approval was provided by the University of Liverpool's Health and Life Sciences Research Ethics Committee (Reference: 7766). All participants provided informed consent.

To select the two PILs, we stratified the PILs assessed within Study 1 by their adjusted FRE score. We then randomly selected one PIL from the top quartile (namely, Inovelon film‐coated tablets) and one PIL from the bottom quartile (Pregabalin Focus; Table 3).

TABLE 3

Details of PILs selected for user testing (Studies 2 and 3)

Characteristics	PIL 1	PIL 2
	Pregabalin Focus, 20‐mg/ml oral solution	Inovelon, 100‐, 200‐, 400‐mg film‐coated tablets
Active ingredient	Pregabalin	Rufinamide
Branded/generic	Generic	Branded
Word count	25 48	1825
Flesch Reading Ease score (context adjusted)	56; bottom quartile	66; top quartile
Median reading grade score (context adjusted)	11	10
Authorization holder	Focus Pharmaceuticals	Eisai
Version date	February 2019	May 2020

Abbreviation: PIL, Patient Information Leaflet.

Survey content

Participants were asked brief questions about demographics and epilepsy profile (or that of a person with epilepsy they knew). For each PIL, the participant was then asked eight comprehension questions (Table 4).

TABLE 4

Questions asked of participants about the different antiseizure medication leaflets to assess comprehension and the scores of the sample

Patient Information Leaflet	Epilepsy sample, n = 24				Student sample, n = 237
	Correct, % (%, 95% CI)	Rank	Interrater reliability a		Correct, % (95% CI)	Rank	Interrater reliability a
			PABAK (BI; PI)	Agreement			PABAK (BI; PI)	Agreement
Pregabalin Focus, 20‐mg/ml oral solution
1. Imagine you are already taking one antiseizure medication. It is not working well. Your doctor therefore also prescribes you pregabalin. Should you stop the other antiseizure medication? Please explain your answer.	66.7 (46.3–87.0)	4	.71 (BI: −.08; PI: 0)	85.7	48.1% (41.7–54.5)	8	.90 (BI: 0; PI: .04)	95.0
2. Imagine you have a family member who is taking pregabalin. They are thinking about taking the oral contraceptive pill. Explain if they can do this?	58.3 (37.1–79.6)	5	.94 (BI: .03; PI: −.06)	97.1	78.1 (72.8–83.4)	7	.95 (BI: .02; PI: −.55)	97.5
3. Imagine that as well as taking pregabalin you were also taking oxycodone for pain. Explain what may happen if you took these two medicines at the same time.	25.0 (6.3–43.7)	8	.94 (BI: .02; PI: −.51)	97.1	81.0% (75.9–86.0)	6	.90 (BI: .03; PI: −.58)	95.5
4. Imagine that you took too much pregabalin. Explain what you should do.	79.2 (61.6–96.7)	2–3	1 (BI: 0; PI: .37)	100	95.4% (92.7–98.1)	1	1 (BI: 0; PI: −.90)	100.0
5. Imagine that you forgot to take your pregabalin when you were meant to take it. Explain what you should do.	79.2 (61.6–96.7)	2–3	1 (BI: 0; PI: .37)	100	92.8 (89.5–96.1)	2	.98 (BI: −.01; PI: −.85)	99.2
6. Imagine your family member has been prescribed pregabalin. Do they need to take food before they take it?	50.0 (28.4–71.6)	6	1 (BI: 0; PI: 0: −.31)	100	82.3 (77.4–87.2)	5	.94 (BI: −.01; PI: −.64)	97.5
7. Explain how you should store the pregabalin medicine.	41.7 (20.4–62.9)	7	.89 (BI: .06; PI: −.20)	94.3	90.7 (86.9–94.4)	4	.72 (BI: −.14; PI: −.76)	86.0
8. Imagine that after you start taking pregabalin you experience some mood changes or distressing thoughts. Explain what you should do.	83.3 (67.3–99.4)	1	1 (BI: 1; PI: .43)	100	91.1 (87.5–94.8)	3	.98 (BI: 0; PI: −.83)	99.2
Inovelon, 100‐, 200‐, 400‐mg film‐coated tablets
1. What is the name of the medicine that Inovelon contains?	41.7 (20.4–62.9)	7	1 (BI: 0; PI: −.43)	100	91.1 (87.5–94.8)	4	.96 (BI: −.01; PI: −.93)	98.1
2. Imagine you have a friend who has a family history of electrical disturbance of the heart. What might happen if they take this medication?	33.32 (12.9–53.7)	8	.94 (BI: .03; PI: −.46)	97.1	78.9 (73.7–84.1)	7	.97 (BI: −.01; PI: −.57)	98.4
3. Imagine that you have a family member who is breast‐feeding. Explain if this person can take Inovelon.	54.2 (32.7–75.7)	4–5	.77 (BI: .06; PI −.09)	88.6	95.4 (92.7–98.1)	3	.96 (BI: .99; PI: .01)	99.6
4. Imagine that you took too much Inovelon medicine. Explain what you should do.	79.2 (61.6–96.7)	2–3	1 (BI: 0; PI: .37)	100	97.0 (94.8–99.2)	1	1 (BI: 0; PI: −.94)	100.0
5. Imagine that you forgot to take the Inovelon medicine when you were meant to take it. Explain what you should do.	83.3 (67.3–99.4)	1	1 (BI: 0; PI: .42)	100	90.7 (86.9–94.4)	5	.91 (BI: −.02; PI: −.84)	95.3
6. Imagine your family member has been prescribed Inovelon. Do they need to eat food before they take it?	54.2 (32.7–75.7)	4–5	1 (BI: 0; PI: −.02)	100	89.0 (85.0–93.0)	6	.98 (BI: −.01; PI: −.76)	98.8
7. Imagine that when you go to take the Inovelon you notice that the appearance of the tablets has changed, what should you do?	50.0 (28.4–71.6)	6	.94 (BI: .03; PI: −.17)	97.1	70.9 (65.1–76.7)	8	.91 (BI: −.05; PI: −.40)	95.3
8. Imagine that after you start taking the Inovelon medicine, you start to get a rash. Explain what you should do.	79.2 (61.6–96.7)	2–3	.94 (BI: −.02; PI: .40)	97.1	96.2 (93.8–98.7)	2	.96 (BI: .01; PI: −.94)	98.1

Abbreviations: BI, bias index; CI, confidence interval; PABAK, prevalence‐adjusted bias‐adjusted kappa; PI, prevalence index.

Interrater reliability calculated using comprehension data from all participants: completers and noncompleters. A PABAK value of .81–1.00 was considered to indicate almost perfect agreement, .61–.80 substantial agreement, .41–.60 moderate agreement, .21–.40 fair agreement, and .00–.20 slight agreement. PI can range from −1 to +1 (0 indicates equal probability), whereas BI ranges from 0 to 1 (0 indicates equal marginal proportions and so no bias).

PILs remained available to participants when answering the questions, and no time restrictions were applied. Participants typed their answers to the questions within free‐text boxes.

In developing the comprehension questions, regulatory guidance , was followed. Most were framed as scenarios, asking participants in an open‐ended way what the correct course of action was. Some requested the person to imagine finding themselves in a certain situation, others asked them to imagine someone they knew found themselves in the situation. This approach is consistent with guidance and has been used before. It also permitted the same set of questions to be used with all participants regardless of their characteristics (e.g., questions regarding female birth control and breastfeeding could be asked of all). Questions were phrased differently from the relevant text of the PILs, and the order of the topics asked about differed from the PIL. Face validity was confirmed by a consultant neurologist.

Responses to the comprehension questions were coded as correct or incorrect by two independent raters based on criteria established a priori. Any discrepancies were resolved through discussion. Raters were trained undergraduate psychology students (N.C., S.H.). To understand interrater reliability, percentage agreement and the prevalence‐adjusted bias‐adjusted kappa (PABAK) were calculated.

The primary analysis focused on participants who completed comprehension questions for both PILs. As comprehension scores were not normally distributed (Shapiro–Wilk, p < .01), central tendency is described according to the median and IQR. The proportion of participants providing a correct response to each question is reported, along with 95% confidence interval (CI). Questions for each PIL were ranked according to the proportion of correct responses elicited.

To understand how participants answering comprehension questions for two PILs (completers) compared to those completing them for only one PIL (noncompleters), the total comprehension scores the two groups achieved on their first allocated PIL was calculated.

Analyses were conducted using SPSS (v27), StatsDirect3 was used for CIs, and PABAK was determined using the calculator at https://labplantvirol.com/kappa/online/calculator.html.

Study 3: User testing with student population

An anonymous, cross‐sectional online survey similar to that used for Study 2 was employed.

A sample size calculation was completed. It was informed by Biggs et al.’s estimate that 83% of children without epilepsy can potentially answer comprehension questions correctly having read an ASM PIL. This, together with a required confidence level of 95% and precision of ±5%, indicated 214 participants with complete data were required.

Between January and February 2022, participant advertisements were sent by email to students at the University of Liverpool within the schools of engineering, geography, management, and health and life sciences. Table 2 shows the eligibility criteria.

Approval was provided by the University of Liverpool's Health and Life Sciences Research Ethics Committee (Reference: 7766 Amend). All participants provided informed consent.

Materials, survey content, and analysis

Materials, survey content, and analysis were the same as for Study 2. The only difference was that the comprehension scores of completers and noncompleters were formally compared (Mann–Whitney, alpha = .05.).

RESULTS

Characteristics of PILS

Of the 140 PILs, 79 (56.4%) were for generic ASMs. The median authorization date for the ASMs focused on by the PILs was October 3, 2011 (IQR = December 31, 2005 to December 14, 2015); 106 (75.7%) of the ASMs had been authorized after October 2005. The median date on which the PILs examined had last been revised was November 1, 2019 (IQR = April 1, 2019 to March 1, 2020).

The PILs had a median word count of 2439.5 (IQR = 2116–2958.8), of which 17.5% of the words (IQR = 15.8–19.4) were polysyllabic. Sentences within the PILs had a median length of 14.1 words (IQR = 12.9–14.9).

Readability of PILs

According to standard test approach

No PIL had a reading grade score at or below grade 8. The estimated median required reading grade of the documents was 11.2 (IQR = 10.9–11.5), equivalent to a UK reading age of ~16 years (Table 1). The median FRE score of the PILs was 50 (IQR = 45–55).

Scores on F‐K, SMOG, and FORCAST were all significantly correlated with one another in the expected direction (r = .629–.969, all p < .001).

When adjusting for context

The adjustments reduced the proportion of polysyllabic words within the PILs by a median of 3.6% to 14.3% (IQR = 12.5–15) and led to the median reading grade requirement of the PLS being reduced to 10.5 (IQR = 10.2–10.7, z = −10.296, p < .001). FRE scores also significantly improved to 60 (IQR = 57–64, z = 10.282, p < .001). Nevertheless, only one (.7%) PIL had a reading grade at or below 8.

Factors associated with PIL readability

Time since the ASM was authorized and time since the PIL examined had last been revised were not significantly correlated with required reading grade (r = .04 to −.17) or FRE score (r  = −.05 to .06). Moreover, PILs authorized before and after October 2005 did not significantly differ.

Compared to PILs for generic ASMs, PILs for branded ASMs had a significantly lower required reading grade (median = 10.3, IQR = 10.2–10.6 vs. median = 10.6, IQR = 10.3–10.8; U = 1689, p < .008) and higher FRE score (median = 62, IQR = 59–64 vs. median = 59, IQR = 57–63; U = 3150.500, p < .006). PILs for branded and generic ASMs were similar in word count (U = 2690, p > .05), but branded PILs included a smaller proportion of polysyllabic words (13.5 vs. 14.6%; U = 1724.000, p < .006).

The required reading grade for the PILs for the most prescribed ASM ingredients was not statistically different from that of PILs for the ASMs with another ingredient (p = .27). They did have a slightly worse FRE score (median = 58, IQR = 56–63), but this was not significant at the Bonferroni‐corrected level (median = 60, IQR = 58–64.5; U = 1453, p = .01).

Comparison of the readability of PILs with online epilepsy materials

No statistically significant differences were found to exist between PILs and online materials (all p > .05). Their unadjusted median required reading grade was 11.1 (IQR = 10.5–11.7), their FRE was 51 (IQR = 44–58), and four (2.2%) items had a reading grade at or below 8.

Characteristics of participants

Thirty‐five participants from the epilepsy population were recruited. Complete responses to the comprehension questions were provided by 24 (68.6%). It took them a median of 26 min to complete the survey (IQR = 10.1–37.8).

Their median age was 42 years (range = 36–45), most (n = 22, 91.7%) were female, and most (n = 21, 87.5%) took part because they had epilepsy (Table 5). In terms of education, the highest attainment for 12 (50.0%) participants was a basic school certificate (typically completed at the age of 16 years in the UK), one (4.2%) had completed an advanced school certificate (aged 18 years in the UK), four (16.7%) had completed a university degree, and five (20.8%) had completed a postgraduate degree. For two (8.3%) participants, the education level was not clear.

TABLE 5

Characteristics of participant samples for Studies 2 and 3

Factors	Epilepsy sample, n = 24	Student sample, n = 237
Age, years
Median (IQR)	42 (36–45)	20 (19–22)
Sex, n (%)
Male	2 (8.3)	78 (32.9)
Female	22 (91.7)	157 (66.2)
Prefer not to say	0	2 (.8)
Main language
English	23 (95.8)	213 (89.9)
Other	1 (4.2)	24 (10.1)
Relationship with epilepsy
I have epilepsy	21 (87.5)	7 (3.0)
Significant other to someone with epilepsy	3 (12.5)	44 (18.6)
No relationship	0	186 (78.4)
Have you achieved, or are you currently studying for, a qualification at degree level or above?
Yes	9 (37.5)	234 (98.7)
No	15 (62.5)	3 (1.3) a
How often do you have problems learning about medical conditions because of difficulty understanding written information? b
Limited health literacy [score = 1–4]	17 (70.8)	156 (65.8)
Adequate health literacy [score = 5]	7 (29.2)	81 (34.2)
Experience with any of ASMs focused on by PILs
No	21 (87.5)	231 (97.5)
Yes	3 (12.5)	6 (2.5)
Seizures [any type] in prior 12 months c
Median (IQR)	7 (2–10)	–
No	5 (20.8)	–
Yes	19 (79.2)	–

Note: Date are given as n (%) unless otherwise indicated.

Abbreviation: IQR, interquartile range.

These n = 3 participants were at the time of the survey studying at the university for a Foundation Certificate. This is not a university degree, but rather a course to prepare some international students for a subsequent undergraduate degree course.

Health literacy is measured using validated question, “How often do have problems learning about medical conditions because of difficulty understanding written information?” Responses were recorded on a Likert scale from 1 = all of the time, 2 = most of the time, 3 = some of the time, 4 = a little of the time, or 5 = none of the time. A score of 1–4 was categorized as having limited health literacy and score of 5 as adequate health literacy.

Seizure frequency measured according to Thapar et al’s scale, which asks “How many attacks have you had in the last 12 months?” The patient can choose from the following ordinal categories: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or more.

Comprehension

Interrater reliability

Rater agreement was excellent (Table 4). For the Inovelon PIL, raters agreed between 88.6% and 100% of the time (PABAK = .77–1). For the pregabalin PIL, raters agreed between 85.7% and 100% of the time (PABAK = .71–1).

Participant comprehension

Completers versus noncompleters

The median number of correct answers that completers (5, IQR = 2.5–6) and noncompleters (5, IQR = 4–6) achieved for their first allocated PIL was similar.

Pregabalin

The median proportion of participants providing correct responses to the individual questions was 54.2% (range = 25.0%–83.3%). Only one question (Number 8) satisfied the regulators' ≥80% threshold (Table 4). Question 3 elicited the least correct responses.

Inovelon

The median proportion of participants providing correct responses to the individual questions was 62.5% (range = 33.3%–83.3%). One question (Number 5) satisfied the ≥80% threshold. Question 2 elicited the least correct answers.

Two hundred sixty‐two participants were recruited; 237 (90.5%) provided complete responses to the comprehension questions. Median age was 20 years (IQR = 19–22), 66.2% were female, and 24 (10.1%) reported English was not their main language. Seven (3.0%) reported having an epilepsy diagnosis (Table 5). They took a median of 17.5 min to complete the survey (IQR = 13.8–23.4).

Agreement between raters was excellent (PABAK = .72–1; Table 4).

The median number of correct answers that completers (7.0, IQR = 6–8) and noncompleters (7, IQR = 6–8) gave for their first allocated PIL did not significantly differ (U = 2589.5, p > .05).

The median proportion providing correct responses to the individual questions was 86.5% (range = 48.1%–95.4%; Table 4). Six had ≥80% of participants providing correct responses to them. The question eliciting the least correct responses was Question 1.

The median proportion of participants providing correct responses to the individual questions was 90.9% (range = 70.9%–97.0%). Six had ≥80% of participants providing correct responses to them. The question eliciting the least correct responses was Question 7.

DISCUSSION

Main findings

Our comprehensive assessment suggests ASM PILs available in the UK may not be understood by a substantial proportion of the epilepsy population.

We assessed 140 PILs using readability tests. None had a reading age requirement at or below the recommended grade 8 level. Most were grade 11, similar to PILs for other medications. , Based on literacy level data, ~40% of the general adult population in the UK might struggle with the PILs. It could be worse in the epilepsy population, because it is at higher risk of low literacy.

We were cognizant that readability tests might, when applied in a standard way, not offer an accurate assessment. However, even after adjusting for this, the PILs still had too high a reading level (grade 10.5).

Despite all the regulations, templates, and guidelines in place to support PIL development, they performed no better than online materials on epilepsy. By some measures, the latter were marginally better.

There was some evidence that PILs for branded ASMs were more readable than those for generics. However, even branded PILs were written at too high a level (grade 10.3).

To our knowledge, this is first time a difference between branded and generic PILs in Europe has been reported. The practical relevance of the difference is unclear. It is nonetheless concerning. Generic ASMs are commonly prescribed in the UK, and there is momentum to use them more. Why the difference occurred is unknown. It is the case that applications for authorization for generic and branded medications in the EU can be submitted and reviewed slightly differently. This might be relevant.

Although readability tests are helpful, how a document performs with its intended user is the most important test. For Study 2, we recruited 34 people from the epilepsy population and presented them with two PILs. Only two of the 16 questions had sufficient people answering them correctly to meet the ≥80% threshold cited by regulators.

The size of the sample we used for Study 3 was in line with that recommended. Nevertheless, it does lack precision. Thus, it is helpful to consider the CIs for the estimates. Even if the upper bounds of the intervals are used, half of the comprehension questions still fail to satisfy the ≥80% threshold.

The consequences of a person failing to understand a PIL will be context dependent. PILs are also only one way that patients can obtain information about their medications. Deficiencies in the understandability of PILs could, for instance, be mitigated by any counsel the patient receives from their care provider(s). Nevertheless, it is concerning that the questions eliciting the most incorrect responses in Study 2 related to safety warnings released for the two ASMs, namely, potential consequences of taking Inovelon if one has a pre‐existing heart condition and the risks of taking pregabalin with oxycodone.

Only a small number of studies have assessed how well users of other medications comprehend materials written for them about their medication, and variability in the methods used prevents direct comparisons. Nevertheless, the studies do indicate ASM PILs are not unique in their failure to ensure patients consistently comprehend core messages. , , Another important finding from some of these other studies is that they showed how PILs can be successfully modified and patient comprehension improved.

Findings in relation to regulations

Criticisms of PILs are not new. However, most studies from Europe have focused on PILs developed before the 2005 requirement of manufacturers to demonstrate engagement with users. Most of the PILs we examined had been authorized after 2005. Why then did they perform so poorly?

Were the two PILs we considered outliers? This is unlikely. We randomly selected them and included one from the quartile with the best readability score from Study 1.

A second possibility is our participants were unrepresentative. Our participants did report poor seizure control. However, they had characteristics that should have made comprehending the PILs easier. They were more educated than would be expected (37.5% were working toward/had achieved a university degree compared to 27.1% in England ), and ~12% reported some familiarity with one or more of the ASMs focused on by the PILs.

Third, might the way we conducted the user testing differ from the approach used by manufacturers? This is hard to know. The evidence manufacturers submit to regulators is not publicly available.

If we assume manufacturers all use the Australian‐Sless method, then it is true that some differences existed in how we conducted the user testing. However, these should not account for the PILs performing so poorly.

One difference was (partly because of COVID‐19) that we assessed comprehension via a survey, rather than by face‐to‐face interview. The approach has been used before. , Might it, however, have meant people were less likely to be scored as having given a correct response (e.g., answers could not be explored)? Our findings suggest not, because the answers people typed were clear enough for two raters to consistently agree on their correctness.

People viewed the PILs electronically rather than as paper documents. Could this have made the PILs less easy to comprehend? This is possible. However, PILs are used by people in this form, and doing so can allow them to overcome complaints about paper PILs (e.g., zooming in to increase text size, using word search function).

Finally, the assessment process we used with users was abbreviated. We asked users eight questions regarding each PIL. The Australian‐Sless method involves users being asked more (~15). Half of these typically ask the person to show where specific information in the PIL is; the other half assesses the person's comprehension of that information. Regulators state ≥80% of participants should be able to both find information and answer related comprehension questions. To minimize participant burden, we only assessed participants' comprehension (i.e., we did not ask participants to show where the information was or award marks for this). This difference should not explain why PILs performed so poorly in our study, because we simply described the proportion of participants giving correct answers to the different questions and the number satisfying or exceeding the 80% threshold cited by regulators.

Implications

Our findings have relevance for both the UK and the EU. All the PILs examined had been approved while the UK was an EU member. Moreover, the processes the UK uses now that it has left the EU remain similar.

One interpretation of our findings is that more regulation and guidance on PIL development is required. We contend there is a need to first determine how well current regulations on involving users are being adhered to by manufacturers and enforced by regulators. User involvement should be meaningful, not a "tick‐box" exercise. Regulators could clarify the situation by including within the Public Assessment Reports they publish detailed evidence on what user engagement manufacturers did. In the meantime, the identified limitations of PILs highlight the importance of pharmacists and other care providers providing comprehensive medication counseling when dispensing any new ASM.

It would be helpful if the epilepsy community could periodically complete independent evaluations of ASMs. Funding for such work is limited. We explored the utility of completing user testing with the student population. Although the student population was straightforward to recruit and assess, its comprehension scores were not indicative of those of the epilepsy population. At least 80% of the student sample answered 12 of the 16 questions correctly. Moreover, the questions they struggled with most differed. Alternative ways to support independent assessments of ASM PILs warrant consideration.

Strengths and limitations

Our identification of PILs was systematic, the assessment comprehensive, and reporting transparent. The online materials we compared the PILs to were systematically identified and representative. , As shown by our systematic literature search, we are presenting the first published evidence on user testing of ASM PILs with the epilepsy population.

A potential weakness of our study is that the PILs are reflective of those available at one point in time. Some may have since been updated and understandability improved. This seems unlikely, because no substantive changes to how PILs are approved have been introduced. Also, we did not find time since authorization or revision to be related to readability in Study 1.

CONCLUSIONS

PILs are a mandatory document all people prescribed ASMs in the EU receive. Our independent and comprehensive examination of them suggests those being used in the UK may not be understandable to a sizeable proportion of the epilepsy population.

AUTHOR CONTRIBUTIONS

Adam J. Noble (Senior Lecturer in Health Services Research) was chief investigator; conceived of the study; led its design; supervised and coordinated the study and the analysis and interpretation of the data; and wrote the final report. Niamh Coleman and Sara Haddad (Research Assistants) contributed to the running of and recruitment for Studies 2 and 3 and scoring of comprehension responses. They also assisted with the systematic review of the literature. Anthony G. Marson (Professor of Neurology and Consultant Neurologist) contributed to the design of Studies 2 and 3 and the interpretation of results, and reviewed the final report.

CONFLICT OF INTEREST

None of the authors has any conflict of interest to disclose.

File S1

Click here for additional data file.

File S2

Click here for additional data file.

File S3

Click here for additional data file.

File S4

Click here for additional data file.

File S5

Click here for additional data file.