Readability of COVID-19 vaccine information for the general public.

BACKGROUND: More than 130 million individuals in the United States have now received at least one dose of a COVID-19 vaccine. Currently, all adults in the Unites States now have access to one of three COVID-19 vaccines. As part of the vaccination procedure, Emergency Use Authorization (EUA) fact sheets, which contain information regarding the vaccine, are provided. The purpose of this study was to analyze the ease of reading (i.e., readability) of the EUA-approved fact sheets for the vaccines currently available in the United States, the V-Safe adverse event survey script, and the Centers for Disease Control and Prevention (CDC) website information on COVID-19 vaccines designed for the general public in the United States.
METHODS: We acquired the Pfizer, Moderna, and Janssen EUA fact sheets, as well as the V-Safe survey script and the CDC website information regarding COVID-19 vaccines. These documents were analyzed for their complexity regarding the following readability factors: average length of paragraphs, sentences, and words; font size and style; use of passive voice; the Gunning-Fog index; the Flesch Reading Ease index; and the Flesch-Kincaid Grade Level index.
RESULTS: Only the V-Safe adverse-event survey script met readability standards for adequate comprehension. The mean readability scores of the EUA fact sheets and the CDC website were as follows: Flesch Reading Ease score (44.35 avg); Flesch-Kincaid Grade Level (10.48 avg); and Gunning-Fog index (11.8 avg).These scores indicate that at least a 10th-grade level education would be required to understand these reading materials.
CONCLUSION: The average person in the United States would have difficulty understanding the information provided in the EUA fact sheets and CDC COVID-19 vaccine website documents; however, the V-Safe survey was written at an adequate reading level. To ensure that the general public fully understands information regarding COVID-19 vaccines, greater care and effort should be given to the development of simplified information material.

Introduction

The 2019 coronavirus disease (COVID-19) has resulted in a global pandemic and the subsequent development of vaccines. Vaccine recipients are encouraged to read several vaccine information documents before they receive a vaccine, including reviewing the Centers for Disease Control and Prevention (CDC) website on COVID-19 vaccines and the vaccine-specific emergency-use authorization (EUA) fact sheet for the Pfizer, Moderna, or Janssen vaccine they are receiving. Subjects are also encouraged to sign up for an adverse-event tracking survey called V-Safe after they receive their vaccination. In addition, the CDC website on COVID-19 and the EUA fact sheets contain basic vaccine information for the general public, including potential benefits, risks, side effects, and necessary precautions. The purpose of this study was to analyze the readability of the EUA fact sheets, the CDC website on COVID-19 vaccine questions, and the V-Safe adverse-event survey script for vaccines in the United States to determine if they meet the guidelines for adequate public readability.

Vaccine recipients should fully comprehend vaccine information reading material in order to weigh the risks and benefits of vaccination and to adequately understand the possible risks and benefits of receiving the vaccine. Because these documents are the primary method of communicating vaccine information to the public, the readability of these materials is critical. Several readability factors may hamper understanding of reading material, including the subject’s education level and the readability of the document. Font size, layout, charts, pictures, vocabulary, and word length also influence readability [1].

Determining the readability of a document is necessary to make sure that the text is written at an adequate comprehension level for the general public with differing reading skills [2]. A National Work Group on Literacy and Health study demonstrated that approximately 25% of U.S. citizens have very low-level reading skills and are unable to comprehend a bus schedule or medication and cleaning-product labels [3]. However, it is vital that all individuals be able to understand the necessary information before receiving a vaccine. The National Adult Literacy Agency suggests that reading materials for the general population should have a 7th-grade readability level, which is the average reading level for adults in the United States [4]. Many experts advise using plain wording, active voice, short sentences, and present tense to achieve this grade level of reading [5].

Methods

We acquired the following vaccine information documents: vaccine-specific COVID-19 vaccine EUA fact sheets, which are given to COVID-19- vaccine recipients in the United States, from Pfizer, Moderna, and Janssen; the full V-Safe adverse-event survey script; and the text related to COVID-19 vaccines found on the CDC website for members of the general public (specifically, the following links: Questions and Answers; Does it Work?; Is it safe?; Are there side effects?; and What if I am pregnant or breastfeeding?) [6] These documents were selected ands reviewed because they were accessible to the public. Although the length of the V-Safe script varies based on the symptoms being reported, we reviewed the entire script because it is possible that some people could view the script in full.

To increase accuracy and reduce bias, two study authors independently analyzed the readability and formatting metrics of each document using Microsoft Word readability tools and calculators. We used several formatting metrics, including total length, sentence length, font size and style, and percentage of passive sentences, in addition to three standard readability formulas to analyze the reading difficulty of each document. The Flesch Reading Ease score is a widely used and trusted readability formula that predicts ease of reading on a scale from 1 to 100, with higher scores indicating easier readability and 70 being the level of an average U.S. reader [5].

The Flesch-Kincaid Grade Level score is a readability index that determines the ease of readability on a scale from 1 to 12. This score corresponds to the education-grade level necessary to comprehend a passage, [5] but scores below 1 or above 12 are obtainable. For comprehension within the U.S. general public, a 7th-grade reading level is advised, but an 8th-grade reading level is often acceptable and lower levels of 5th-grade or 6th-grade are necessary when targeting audiences with low levels of literacy [7]. Table 1 compares the Flesch-Kincaid Grade Level score and Flesch Reading Ease score.

Table 1

Flesch Reading Ease Score Comparison [5].

Style	Flesch Reading Ease Score	Average Sentence Length in Words	Average No. of Syll. Per 100 Words	Type of Magazine	Estimated School Grade Completed	Estimated Percent of US Adults
Very Easy	90–100	8 or less	123 or less	Comics	4th grade	93
Easy	80–90	11	131	Pulp Fiction	5th grade	91
Fairly Easy	70–80	14	139	Slick Fiction	6th grade	88
Standard	60–70	17	147	Digests	7th or 8th grade	83
Fairly Difficult	50–60	21	155	Quality	Some high school	54
Difficult	30–50	25	167	Academic	High school to college	33
Very Difficult	0–30	29 or more	192 or more	Scientific	College	4.5

The Gunning-Fog index is a readability index that scores text from 1 to 20. Scores are derived from the number of complex or polysyllabic words in a sentence and the number of sentences in a roughly 100-word passage. A more cohesive score is obtained by averaging the Gunning-Fog index score for three random 100-word passages throughout a document. An index of 7 is necessary to achieve near-universal comprehension in the United States [8]. Our scores ranged from 6 to 17 and corresponded to 6 = 6th grade to 17 = college graduate.

Results

We reviewed the COVID-19 vaccine fact sheets for the Pfizer, Moderna, and Janssen vaccines available under EUA in the US, the complete V-Safe adverse event survey script, and the CDC general public website on information about COVID-19 vaccines. No significant differences were detected in the Microsoft scoring metrics between the two independent study authors who analyzed readability.

Document length ranged from 5 to 18 pages (8.25 avg). The number of words (including headers) ranged from 1,662 to 5,564 (3,086.8 avg). Average words per page ranged from 175.9 to 334.2 (294.8 avg), and average words per paragraph ranged from 7.0 to 25.1 (17.76 avg). Font size ranged from 11 to 16 points. All documents used sans serif fonts. The average word length ranged from 4.9 to 5.4 characters (5.18 avg). Average sentence length ranged from 9.8 to 18.7 words (14.4 avg). See Table 2 .

Table 2

Online COVID-19 Vaccine Document Statistics.

	Total Pages	Total Paragraphs	Total Words	Words/ Page	Words/ Paragraph	Font Style	Font Size	Word Length (Characters)	Sentence Length (Words)
V-Safe Survey Script	28	624	4,924	175.9	7.0	Calibri	11	4.9	9.8
Pfizer EUA Fact Sheet	5	101	1,671	334.2	16.5	Arial	12	5.4	14.1
Moderna EUA Fact Sheet	5	83	1,662	332.4	20.0	Arial	12	5.2	15
Janssen EUA Fact Sheet	5	80	1,613	322.6	20.2	Arial	12	5.3	14.4
CDC COVID-19 Vaccine Website	18	222	5,564	309.1	25.1	Arial	14–16	5.1	18.7
Recommended	<15	None	None	None	None	Arial/Calibri	≥14	None	12–17

The percent of passive sentences ranged from 0.2% to 21.1% (15.04% avg). The Flesch Reading Ease Score ranged from 40.9 to 66.8 (48.84 avg). The Flesch-Kincaid Grade Level ranged from 6.3 to 11.6 (9.76 avg). The Gunning-Fog Score ranged from 8.03 to 12.43 (11.19 avg). See Table 3 .

Table 3

Online COVID-19 Vaccine Document Readability Scores.

	% Passive	Flesch Reading Ease	Flesch-Kincaid Grade Level	Gunning-Fog Index
V-Safe Survey Script	0.2%	66.8	6.3	8.77
Pfizer EUA Fact Sheet	16.9%	40.9	10.7	10.48
Moderna EUA Fact Sheet	18.7%	46.2	10.2	10.78
Janssen EUA Fact Sheet	18.3%	46.1	10	10.77
CDC COVID-19 Vaccine Website	21.1%	44.2	11.6	15.17
Recommended	0%	≥70	≤7	≤7

Discussion

This study focuses on the vaccine information readability for the COVID-19 vaccines currently distributed in the United States. The importance of readability regarding public health information has been articulated by others relating to COVID-19 [9], [10], [11]. Our analysis found that only the V-Safe adverse-response survey script met most of the recommended readability levels for average vaccine recipients. However, the V-Safe study did not achieve an adequate Gunning-Fog index score, indicating that readability could be further improved by using sentences that are less complex. None of the other information sources we reviewed met current readability standards.

Readability studies demonstrate that the general public in the United States is less likely to understand long and complex documents, [12] and that health information documents should be shorter than 15 pages [13]. The vaccine information documents in this study had an acceptable average of 8.25 page length, but the range extended to 18 pages. Note that we excluded the V-Safe script as an outlier with 28 pages. High word density, or too many words in a sentence, also raises the difficulty level of a text; therefore, it is advised that sentences are a maximum of 12 – 17 words long [14]. Although the average sentence length (14.4 words) in our study was acceptable, the range extended to 18.7 words.

Font style and size also play roles in analyzing the readability of a text. Sans serif fonts (i.e., Arial and Calibri) are easier to read than serif fonts (i.e., Times New Roman). Also, smaller fonts such as size 12 are harder to read than larger fonts such as size 14 [15]. All documents used an appropriate font style, but the size 12 font found in every document was too small. Many passive-voice sentences, or sentences with the subject receiving the action of the verb, also decrease readability [5]. The average percentage of passive-voice sentences in our study was fairly low, but the range extended up to 21%.

Because nearly half of adults in the United States read at a 7th-grade level, [7] vaccine information materials provided for the currently COVID-19 vaccines are too difficult to understand for many adult readers. In addition to simplified readability, instructional graphics should be used to increase comprehension levels. Other multimedia, including pictures, videos, colors, slideshows, and charts, were used on the CDC website and the V-Safe survey but not in the EUA vaccine fact sheets and may increase comprehension. In addition, the responsive nature of the V-Safe survey exposes the individual to very small amounts of text at any one time, making it easier to read and comprehend. Further study is required of cultural and educational barriers to determine the most useful methods of patient education. Personal discussion between vaccine recipients and healthcare professionals may be another effective way to improve readability understanding [16].

Readability could be significantly improved through the use of readability formulas, simplified writing techniques, and awareness of reading levels [17]. Formulas such as the Flesch-Kincaid Grade Level, Flesch Reading Ease Score, and the Gunning-Fog Index allow for readability analysis and aid in the development of easily understood vaccine information documents. Changing other physical aspects of the text can also help to improve readability [1]. Additionally, the importance of readability when considering the illiterate, elderly, and disadvantaged communities widens the scope of this issue.

The limitations of our study include the potential inability of readability statistics to predict true comprehension due to jargon, syntax, or other linguistic features [18], [19]. This study also does not consider whether some vaccine recipients desire additional information beyond what is offered [20]. The strengths of our study include comparisons between widely used and easily accessible online material regarding COVID-19 vaccines. This study also uses standardized, reliable, and widely used metrics to assess readability.

In conclusion, our study demonstrates that a significant portion of the U.S. general public is unable to comprehend the available vaccine information in the documents we studied. The V-Safe script is an exception because it is well written for the average U.S. vaccine recipient. However, the vaccine EUA fact sheets and CDC website fail to meet acceptable readability standards. Other strategies, such as multimedia formatting with visual and audio effects and personal discussion between patients and healthcare professionals, may improve the general public’s understanding of COVID-19 vaccines. Our results indicate potential communication barriers between health professionals and the public regarding the COVID-19 vaccines. More research measuring comprehension in target populations and improved communication strategies is necessary. Given the importance of these documents to inform and build trust within the community regarding COVID-19 vaccines, greater effort must be applied to improve the readability of these information documents.

Disclosures

Dr. Poland is the chair of a Safety Evaluation Committee for novel investigational vaccine trials being conducted by Merck Research Laboratories. Dr. Poland provides consultative advice on vaccine development to AiZtech; Atria; AstraZeneca UK Limited; Contec, Inc.; 3D Communications; Eli Lilly and Company; Emergent Biosolutions; Exelixis, Inc.; ExpertConnect; Genevant Sciences, Inc.; GlaxoSmithKline; Janssen Global Services, LLC; Janssen Research & Development, LLC; Medicago USA; Medscape, LLC; Merck & Co. Inc.; Moderna; Regeneron Pharmaceuticals, Inc.; Sanofi Pasteur SA; Syneos Health and Vyriad. These activities have been reviewed by the Mayo Clinic Conflict of Interest Review Board and are conducted in compliance with Mayo Clinic Conflict of Interest policies. Dr. Poland holds patents related to vaccinia, influenza, and measles peptide vaccines. Dr. Poland has received grant funding from ICW Ventures for preclinical studies on a peptide-based COVID-19 vaccine. This research has been reviewed by the Mayo Clinic Conflict of Interest Review Board and was conducted in compliance with Mayo Clinic Conflict of Interest policies.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.