Improving health literacy with mumps, measles and rubella (MMR) vaccination: comparison of the readability of MMR patient-facing literature and MMR scientific abstracts.

Background: Historically, there have been many factors that have influenced mumps, measles and rubella (MMR) vaccine uptake, including media bias, social/economic determinants, parental education level, deprivation and concerns over vaccine safety. Readability metrics through online tools are now emerging as a means for healthcare professionals to determine the readability of patient-facing vaccine information. The aim of this study was to examine the readability of patient-facing materials describing MMR vaccination, through employment of nine readability and text parameter metrics, and to compare these with MMR vaccination literature for healthcare professionals and scientific abstracts relating to MMR vaccination. Materials and methods: The subscription-based online Readable program (readable.com) was used to determine nine readability indices using various readability formulae: Established readability metrics (n = 5) (Flesch-Kinkaid Grade Level, Gunning Fog Index, SMOG Index, Flesch Reading Ease and New Dale-Chall Score), as well as Text parameters (n = 4) (sentence count, word count, number of words per sentence, number of syllables per word) with 47 MMR vaccination texts [patient-facing literature (n = 22); healthcare professional-focused literature (n = 8); scientific abstracts (n = 17)].
Results: Patient-facing vaccination literature had a Flesch Reading Ease score of 58.4 and a Flesch-Kincaid Grade Level of 8.1, in comparison with poorer readability scores for healthcare professional literature of 30.7 and 12.6, respectively. MMR scientific abstracts had the poorest readability (24.0 and 14.8, respectively). Sentence structure was also considered, where better readability metrics were correlated with significantly lower number of words per sentence and less syllables per word.
Conclusion: Use of these readability tools enables the author to ensure their research is more readable to the lay audience. Patient co-production initiatives would help to ensure that not only can the target audience read the literature, but that they understand the content. Increased patient-centric focus groups would give better insights into reasons for MMR-associated vaccine hesitation and vaccine refusal.

Background

There are many factors that can influence mumps, measles and rubella (MMR) vaccine uptake, including media bias, social and economic determinants such as parental education level, deprivation level of the area and concerns over the safety of the vaccine, as described previously. It has been suggested that perception of susceptibility to disease also has an impact on parental decision-making, leading to either non-vaccination or failure to complete the vaccination schedule as evidenced in a report by MacDonald et al., which concluded access to a regular healthcare practitioner, arrangements for childcare and parents who have been transient at some stage during the vaccination programme to be factors.

Health literacy, which may be impacted by parental education level, can be defined as the capacity of the individual to acquire, digest and comprehend basic information and services needed to make suitable decisions regarding their health. In the age of digital public access to many information sources, aimed at both the general public and the healthcare professional, the general public may become overwhelmed with the volume and complexity of the material available. According to Flesch, public information sources are only as good as the individual’s ability to grasp the content. It is imperative, therefore, that these materials are written in a manner that is easily digestible to facilitate understanding, therefore helping guide the decision-maker to an informed conclusion.

Readability scoring is used to quantify the difficulty of written language, giving an estimation of how accessible to the population any given piece of text is. The more general audience the text is aimed at, the more important the readability factor of the literature becomes. Flesch Reading Ease (FRE), Flesch–Kinkaid Grade Level (FKGL), Simple Measure Of Gobbledygook Index (SMOG), the Gunning Fog Index and the New Dale-Chall Score are examples of the formulae used to determine the readability score.

It was therefore the aim of this study to examine the readability of patient-facing materials describing MMR vaccination, through employment of nine readability and text parameter metrics, and to compare these with MMR vaccination literature for healthcare professionals, as well as with scientific abstracts relating to MMR vaccination.

Methods

Retrieval of MMR literature and examination of readability indices using readability tools

Open-access literature (n = 47) related to MMR vaccination aimed at the general public and healthcare professionals in the United Kingdom, Northern Ireland (NI), the Republic of Ireland (ROI), the United States, Australia and Canada, and various public health sources were searched and retrieved. All information was sourced using PubMed, MEDLINE, Google Scholar National Health Service (NHS) websites and governmental websites from the various geographical regions. Keywords used in the search for relevant material included ‘vaccine uptake’, ‘vaccine hesitancy’, ‘vaccine safety information’, ‘vaccine information for parents’, ‘incidence’, ‘correlation’, and ‘measles, mumps and rubella’.

Determination of readability metrics

The subscription-based online Readable program (readable.com) was used to determine nine readability indices using various readability formulae for the following: Established readability metrics (n = 5) (FKGL, Gunning Fog Index, SMOG Index, FRE and New Dale-Chall Score), as well as Text parameters (n = 4) (sentence count, word count, number of words per sentence, number of syllables per word). These readability metrics were selected for investigation as they are usually quoted by most readability papers. Each piece of literature was duly retrieved and converted to a text-only format in a Microsoft Word document, with all images and non-essential text such as headers or hyperlinks removed before being subjected to readability analysis.

Statistical analyses

A Shapiro–Wilk test, performed using SPSS, was applied to check that the data conformed to a normal distribution to ensure all data were normally distributed before performing two-tailed, unpaired t tests. For all tests, p < 0.05 (5%) was considered statistically significant.

Ethical statement

This study did not involve any human subjects or animals and therefore did not require ethical approval. All data presented in this study were taken from information in the public domain.

Results

Analysis of 47 MMR vaccination texts was performed, including (1) patient-facing literature (n = 22), healthcare professional–focused literature (n = 8) and scientific abstracts (n = 17), and scored using various readability parameters, as shown in Tables 1–3. All data sets were found to be normally distributed, and for comparison, two-samples unpaired t tests were performed and showed statistically significant differences for all nine readability metrics (combined) (p = 0.007) when comparing patient-facing and healthcare professional materials, as well as (p = 0.0002) when comparing patient-facing materials with scientific abstracts. When specifically examining patient-facing materials and healthcare professional materials, as well as patient-facing materials and scientific abstracts for both the FKGL and the FRE, all comparisons were statistically different (p < 0.0001). For patient-facing MMR resources, the mean FKGL, FRE, SMOG and Gunning Fog scores were 8.1, 58.4, 10.6 and 9.9, respectively. As sentence structure is linked to readability formulae, sentence structure was also considered, where better readability metrics were correlated with significantly lower number of words per sentence and less syllables per word. For an overview of this relationship between sentence structure and readability, please refer to Table 4.

Table 1.

Readability grading of MMR patient-focused materials (n = 22).

SOURCE	Readability metrics						Text metrics
	Flesch–Kinkaid Grade Level(target is 8)	Gunning Fog Index(target is 8)	SMOG Index(target is 8)	FLESCH Reading Ease(target 60 or higher)	New Dale-Chall Score(target is 8)	Sentence count	Word count	Words per sentence	Syllables per word
MMR MANUFACTURER UK PIL	6.9	8.5	9.6	59	6.1	250	2043	8.2	1.7
MMR MANUFACTURER ROI PIL – (PRIORIX)	6.9	8.5	9.5	58.2	5.4	221	1600	7.2	1.7
MMR MANUFACTURER ROI PIL (MMRvaxPRO)	6.8	8.4	9.6	59.8	6.1	258	2079	8.1	1.7
MMR MANUFACTURER USA PATIENT VIS (MERCK)	7.9	9.6	10.5	59.9	6.3	76	926	12.2	1.6
MMR MANUFACTURER USA PATIENT VIS (PROQUADMMRV)	7.9	9.6	10.6	60.1	6.2	75	923	12.3	1.6
MMR MANUFACTURER CANADA PATIENT INFORMATION	6.2	7.7	8.9	62.5	5.7	209	1432	6	1.6
https://www.nhs.uk/conditions/vaccinations/why-vaccination-is-safe-and-important/	7.5	9	10.1	60.9	5.5	91	1020	11.2	1.6
https://www.nhs.uk/conditions/vaccinations/mmr-vaccine/	6.9	9	9.8	63.8	6.2	75	782	10.4	1.6
Measles, mumps and rubella (MMR)| nidirect	5.4	7.1	8.5	68.7	6.1	58	403	6.9	1.6
The Truth About the MMR Vaccine (healthline.com)	7.4	9.2	10.4	59.9	6.5	98	1004	10.2	1.6
MMR MANUFACTURER AUSTRALIA consumer medicine information	9	10.3	11.3	52.5	6.2	199	2446	12.3	1.7
https://www.cdc.gov/vaccines/parents/why-vaccinate/vaccine-decision.html	8.7	10.6	11.3	55.5	5.8	107	1403	13.1	1.6
https://www.cdc.gov/vaccines/parents/tools/parents-guide/downloads/parents-guide-part3.pdf	8.6	10.8	11.1	60.9	4.9	108	1655	15.3	1.5
OXFORD RESEARCH GROUP – MMR VACCINE	9.2	10.6	11.4	57.4	5.9	86	1388	16.1	1.6
MMR for all: general leaflet – GOV.UK (www.gov.uk)	8.1	10.2	10.3	66.7	5.2	139	2315	16.7	1.5
What to expect after vaccinations – leaflet for parents and carers (publishing.service.gov.uk)	8.9	11	11	61.3	4.7	39	652	16.7	1.5
Measles, mumps and rubella – HSE.ie	8.2	10.3	10.8	60.9	5.9	109	1530	14	1.6
Vaccine safety, concerns and possible side effects – Canada.ca	8.6	10.2	10.9	51.6	6.4	134	1394	10.4	1.7
Vaccines for children: Deciding to vaccinate –Canada.ca	9.4	11.6	11.9	51.2	6.1	79	1031	13.1	1.7
Measles mumps rubella (MMR) vaccine (healthywa.wa.gov.au)	8.4	9.9	10.6	59.7	5.9	84	1081	12.9	1.6
Immunisation for children| Australian GovernmentDepartment of Health	8.8	10.3	10.7	48.6	5.9	114	1065	9.3	1.8
OXFORD RESEARCH GROUP–VACCINE SIDE	12.2	14.4	13.6	46.4	5.9	30	657	21.9	1.6
Mean	8.1	9.9	10.6	58.4	5.9	120.0	1310.4	12.0	1.6
SD	1.4	1.5	1.1	5.4	0.5	64.4	537.0	3.7	0.1

MMR, mumps, measles and rubella; ROI, Republic of Ireland; SMOG, Simple Measure Of Gobbledygook.

Table 2.

Readability grading of healthcare professional–focused materials (n = 8).

SOURCE	Readability metrics						Text metrics
	Flesch–Kinkaid Grade Level	Gunning Fog Index	SMOG Index	Flesch Reading Ease	New Dale- Chall Score	Sentence count	Word count	Words per sentence	Syllables per word
WHO – Talking-with-parents_EN_WHO_WEB.pdf	11.5	13.9	13.2	48.4	5.5	114	2353	20.6	1.6
MMR MANUFACTURER CANADA PROFESSIONAL MATERIAL	10.3	11.8	11.3	34.6	7.9	715	5601	7.8	1.9
MMR MANUFACTURER UK SmPC	11.4	12.9	12.9	33.7	7.9	397	4578	11.5	1.9
MMR MANUFACTURER ROI SmPC (MMRvaxPRO)	11.3	12.7	12.7	33.9	7.9	404	4614	11.4	1.9
MMR MANUFACTURER USA PACKAGE INSERT (MERCK)	10.5	10	11.6	31.4	8.8	396	3559	8.1	2
MMR MANUFACTURER ROI SmPC – (PRIORIX)	14.2	14.5	15.3	26.4	7.9	85	1632	19.2	1.9
MMR MANUFACTURER AUSTRALIA PRODUCT INFORMATION	15	15.8	17	25.1	8.4	299	5426	23.7	1.9
MMR MANUFACTURER USA PACKAGE INSERT (PROQUAD MMRV)	16.7	18	17.8	12.2	8.8	58	1343	23.2	2.1
Mean	12.6	13.7	14.0	30.7	7.9	308.5	3638	15.7	1.9
SD	2.2	2.3	2.3	9.6	1.0	206.3	1574.6	6.3	0.1

MMR, mumps, measles and rubella; ROI, Republic of Ireland; SMOG, Simple Measure Of Gobbledygook; WHO, World Health Organization.

Table 3.

Readability grading of MMR vaccination scientific abstracts (n = 17).

SOURCE	Readability metric						Text metric
	Flesch–Kinkaid Grade Level	Gunning Fog Index	SMOG Index	Flesch Reading Ease	New Dale-Chall Score	Sentence count	Word Count	Words per sentence	Syllables per word
https://doi.org/10.1086/378691	11.1	14.3	13.3	51	5.6	7	139	19.9	1.6
10.1001/jamapediatrics.2017.1965	13	15.6	16.1	39.5	8.8	17	427	25.1	1.7
https://doi.org/10.3390/vaccines8010057	13	15.8	15.8	35.5	8.2	10	208	20.8	1.8
10.1080/21645515.2018.1554971	13.2	15.7	15.2	33.9	9.6	12	246	20.5	1.8
https://doi.org/10.1016/S0140-6736(86)91044-5	13.4	16.8	15.4	33.1	8	5	96	19.2	1.8
https://doi.org/10.1016/j.jpeds.2009.10.015	14	15.8	14.4	22.2	8.8	13	198	15.2	2
doi:10.1093/cid/ciz135.	14	17.9	19.1	17.5	9.5	6	140	23.3	2.1
https://doi.org/10.1016/j.vaccine.2017.07.029	14.2	16.8	16.4	31.4	8.6	7	149	21.3	1.8
https://doi.org/10.1136/bmj.j3862	14.6	17.9	16.6	29.4	9	15	328	21.9	1.8
https://doi.org/10.1093/cid/ciz135	14.9	18.9	22.5	15.3	9.9	4	138	34.5	2.1
https://doi.org/10.1080/10810730.2018.147353 3	15	18	16.3	23.7	8.8	6	123	20.5	1.9
10.1590/0074-02760180517	15.4	15.3	15.4	18.2	9.2	11	206	18.7	2
10.1146/annurev-virology-092818-015515	16.4	20.5	18.2	16.8	8.2	7	155	22.1	2
https://doi.org/10.1016/j.vaccine.2010.04.052	16.5	19.5	18.8	8.9	9.8	7	162	23.1	2.1
10.1093/ofid/ofaa211	17.2	19.2	18.8	11.3	8.4	6	155	25.8	2
https://doi.org/10.1016/S0264-410X(03)00271-8	18.4	22.5	19.4	8.7	9.6	8	204	25.6	2
https://doi.org/10.4161/hv.22065	17.7	20.5	18.1	11.9	9.4	11	272	24.7	2
Mean	14.8	17.7	17.1	24.0	8.8	8.9	196.8	22.5	1.9
SD	1.9	2.2	2.2	11.7	1.0	3.6	80.9	4.0	0.1

MMR, mumps, measles and rubella; SMOG, Simple Measure Of Gobbledygook.

Table 4.

A selection of four readability scoring systems and their relationship with sentence structure.

Readability index name	Summary	Formula
Simple Measure of Gobbledygook (SMOG)	Estimates comprehension in terms of years of education needed. The first, middle and last 10 sentences should be sampled to calculate SMOG. It involves counting words with 3+ syllables	3 + square root √ [number of polysyllabic words × (30 ÷ number of sentences)]
The Gunning Fog Index(GFI)	Similar to SMOG except it estimates first-time readability comprehension in terms of years of education needed	0.4 × [(words ÷ sentences) + 100 × (complex words ÷ total words)]
Flesch–Kincaid Grade Score(FKG)	Usually used in education and assesses readability while weighting heavily on syllable count	0.39 (total words/total sentences) + 11.8 (total syllables/total words) − 15.59
Flesch Reading Ease Score(FRES)	English text on a scale of 100, with a FRES higher score indicating easier comprehension	[206.835 − (1.015 × (total words ÷ total sentences)) − (84.6 × (total syllables ÷ total words))]

Discussion

To examine whether the readability of publicly available MMR vaccination information sources could have a bearing on the decision to vaccinate, an analysis was carried out on a selection of patient-focused literature, healthcare professional–focused literature and scientific abstracts (Tables 1–3).

For patient-facing material, the FKGL score achieved the target grade level of 8.1, Gunning Fog and SMOG indices showed the material to be above the target level by 2 and 3 grades, respectively, and New Dale-Chall scoring placed patient information at 5.9, well below the target grade.

As stated in a report by Nutbeam, an individual’s reading capability is usually dependant on their level of education, and FRE is used to evaluate the level of education needed to read a specific text and is scored usually between 0 and 100, with a lower score being considered more difficult. An acceptable score for public consumption is deemed to be between 60 and 70 to reach the education level of 13- to 15-year-olds. FKGL produces a reading grade level which is based on both word and sentence length, with difficulty increasing in line with grade level, and a score of 8 is considered to be accessible for the majority of the population, as it is suitable for those aged 13–14 years. The Gunning Fog Index produces a grade level between 0 and 20 reflecting the years spent in formal education (from US grade 4 to college level) required to comprehend the text.

Public literature should produce a score of 8 which is suitable for a US grade 8 reading level (age 13–14 years). An acceptable score is 8, which is that of a US grade 8 reading level. The New Dale-Chall scoring index is based on a list of familiar words, and as with the previous readability tests, the score translates to a specific reading grade level, and a New Dale-Chall score of 8 is acceptable as suitable for the general public.

FKGL and FRE have been found to underestimate the grade reading level by 2–3 grades, while SMOG is considered best suited for health applications as it is consistent, more robust and assumes 100% comprehension.

The above readability formulae were chosen for this study as according to Ley and Florio, they are the most commonly used formulae in healthcare to evaluate the readability of information widely accessible to the general public.

Comparisons drawn between UK and non-UK government patient-facing material indicated UK sources to have significantly lower readability scores. This indicates that although the patient-facing material was found to be above the target grade level, the UK governmental sources have a superior readability score than non-UK governmental sources. In a 2021 study by Clarke et al., UK patient resources used for aiding decision-making in the treatment of latent tuberculosis infection were also found to have significantly lower readability than the non-UK counterparts.

For the majority of readability formulae used to test the material, it was found that on average all patient-facing materials are above the target level; nevertheless, readability formulae have their limitations and have been criticised as they are seen as only quick analytical measures to target information for public consumption, assessing only the reading level, not the patients’ understanding of the text. Readability formulae also do not consider background knowledge of the topic. The medical nature of the materials being evaluated also raises readability scoring due to the complexity of the text and scientific vernacular required. With the exception of the SMOG Index, the majority of the readability formulae have been devised for general reading and are not optimal for use with medical or scientific-focused material. Large and complicated medical terms can push the readability score over the target threshold, leaving room for improvement, and in the future, it may be possible to design more bespoke formulae aimed at medical literature.

Going forward, it would be recommended that all patient-facing materials be analysed using both the FKGL and the FRE indices, with the SMOG Index as the gold standard, with a recommended reading grade level of between US sixth and eighth grades. This target grade level has been arbitrarily set previously by health institutions, including the National Work Group on Cancer and Health, the American Medical Association and the National Institutes of Health, while the Centers for Disease Control and Prevention (CDC) recommends the readability to be lower than eighth-grade level.

Study limitations

There are several limitations with using readability formulae to assess how readable narrative text is in patient-facing literature. Grade reading level recommendations are arbitrary and not based on strong evidence. We advocate that the FRGL target should be between sixth and eighth grades to improve readability; however, we appreciate that this may be difficult to accomplish in real-world scenarios, given that readability metrics struggled to accomplish favourable outcomes at eighth grade. Nevertheless, this lack of achieving our targets should not be a deterrent from attempting to improve readability by reducing the grade level and should remain a target, and solutions are sought to help achieve this readability aspiration. Another way to help achieve this would be to focus more on supporting developers to create accessible, health literate information, as well as evaluate or improve the health literacy demands of health information, such as the Patient Education Material Assessment tool. Indeed, some institutions (such as the Agency for Healthcare Research and Quality [AHQR]) already recommend a grade reading level of 4–6 for easy-to-read materials (www.ahrq.gov/talkingquality/resources/writing/tip6.htm). Furthermore, it is unknown whether changes in grade reading levels actually result in any meaningful change in outcomes such as improved knowledge/comprehension.

In conclusion, the readability of vaccine information sources varied across the different types of materials available. Material oriented towards the general public, such as NHS information leaflets or web pages, had an improved readability score than information aimed at the healthcare professionals, which in turn was lower than the information found in scientific abstracts. It can be noted, however, that information sources aimed at the general public, on average, have a readability score above the target level, and this should be addressed by aiming for a desirable reading FKGL of between sixth and eighth grades, to accommodate a wider population. Material available from health facilities could also be produced at different reading age levels, and the local knowledge of the general practitioner or healthcare provider could ensure that the most appropriate material for the individual was available.