Accessibility Evaluation of COVID-19 Vaccine Registration Websites across the United States.

This exploratory study investigated the web accessibility of 54 official COVID-19 vaccine registration websites in the US and their concordance with the WCAG 2.0 and 2.1 guidelines. We employed AChecker, WAVE, and SortSite web accessibility evaluation tools to conduct automated analyses of these websites. The results showed suboptimal compliance with WCAG 2.0 and 2.1 guidelines, as determined using the AChecker, WAVE, and SortSite tools. These shortcomings in compliance may pose difficulties to users with disabilities as they access information on the websites. Based on our findings, we offered recommendations for states and other authorities to improve the accessibility of their websites to ensure that users with disabilities can independently schedule vaccination appointments.

INTRODUCTION

Vaccinations against the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) are the most potent weapon against the novel coronavirus (COVID-19) pandemic, which has claimed millions of lives worldwide, disrupted societies, and wrought great economic damage. The US Food and Drug Administration (FDA) has issued an emergency use authorization for the use of the Pfizer-BioNTech COVID-19 prevention vaccine for those aged 16 years and above on December 11, 2020, followed by an emergency use authorization of Moderna’s COVID-19 prevention vaccine for those aged 18 years and above on December 18, 2020. Both authorized mRNA vaccines require 2 doses for the most protection, given 3 weeks apart for Pfizer-BioNTech and 1 month apart for Moderna. Accordingly, the same vaccine type must be used for both shots.

COVID-19 prevention vaccination began on December 14, 2020. States have created dedicated websites to facilitate vaccine registration and to disseminate information on vaccine type, eligibility, where and when to obtain the vaccine, the purpose of the vaccine, and the vaccine’s potential side effects to the public. For COVID-19 vaccine registration, individuals can schedule an appointment through a vaccine management system provided by state governments, counties, or local health departments near to an area where an individual resides. The COVID-19 vaccine management system refers to a web-based portal designed to coordinate and arrange COVID-19 vaccine scheduling and administration, with which individuals can check for vaccine availability and schedule an appointment for the first and second dose. In some cases, individuals also have the opportunity to sign up to get notified on the latest updates concerning the COVID-19 vaccine.

With the above in mind, it is important to implement inclusive practices to eliminate any potential barriers to website access to address the different needs of the population. In the US, there are several laws and legislative measures that have emphasized and articulated on implementing web accessibility standards on government websites, such as Section 508 of the Rehabilitation Act Amendments and Section 225 of the Telecommunications Act of 1996. Web accessibility refers to the practice of making websites usable for people with disabilities so that they can understand, navigate, and interact with the websites. Accessibility includes a range of disabilities, including physical, visual, auditory, cognitive, language, learning, speech, and neurological disabilities. Websites that exhibit accessibility problems may pose barriers for vaccine access among people with disabilities, contravening the goal of equity in public health access. Accessible websites are even more important during the COVID-19 pandemic because mitigation restrictions likely reduce the social support available to people with disabilities. These considerations warrant investigating the accessibility of such vaccine registration websites. Particularly, to the best of our knowledge, there are no studies on web accessibility of US state and territory COVID-19 vaccination management websites.

The goal of this exploratory study is to examine the accessibility of 54 official state and territory COVID-19 vaccine registration websites in the US. We employed open-source tools to conduct automated analyses of these websites. Based on our findings, we offer recommendations for states and other authorities to improve the accessibility of their websites to ensure that users with disabilities can independently schedule vaccination appointments.

BACKGROUND

Web accessibility standards

The World Wide Web Consortium (W3C) released the Web Content Accessibility Guidelines (WCAG) 2.0 and 2.1, in 2008 and 2018, respectively. The WCAG 2.1 encompasses 4 principles, 13 guidelines, and 78 success criteria (ie, 61 success criteria from WCAG 2.0 plus 17 added success criteria concerning mobile devices as well as visual and cognitive impairments). Hence, if content is revealed to be compliant with WCAG 2.1, this implies that it is also complying with WCAG 2.0 standards-backward compatibility. The 13 guidelines are divided into 4 principles that are summarized by the acronym POUR: Perceivable (information and user interface components must be presentable to users in ways they can perceive); Operable (user interface components and navigation must be operable); Understandable (information and the operation of the user interface must be understandable); and Robust (content must be robust enough that it can be interpreted reliably by a wide variety of user agents, including assistive technologies).

WCAG 2.0 and 2.1 are evaluated using three compliance levels: Level AAA (highest conformance), Level AA (intermediate conformance), and level A (lowest conformance). Each guideline has a set of testable success criteria that are at 3 levels of conformance, which determines the extent to which a website follows the accessibility guidelines.

Materials and methods

We used an exploratory quantitative methodology aimed at examining the status of web accessibility compliance of COVID-19 vaccine registration websites in the US as outlined by WCAG 2.0 and 2.1. The analyses were carried out in the period spanning February 16, 2021 to March 2, 2021. The accessibility assessment went through 2 stages. First, we performed an automated accessibility assessment of the home pages of US state and territory COVID-19 vaccine management websites on the basis of WCAG 2.0 and 2.1 standards since it signifies the initial connection with web users. The rationale behind this choice has been articulated in previous research (ie, if the home page was inaccessible to the users, the other pages within the website will also be inaccessible)., Second, we automatically evaluated the accessibility of entire websites for states and territories for which its home pages passed the first stage assessment with zero reported violations.

Website selection

We used a total of 54 official COVID-19 vaccine registration and management websites provided by the online portal of the CDC (https://www.cdc.gov/coronavirus/2019-ncov/vaccines/index.html). The portal provides a list of US vaccine websites under “how do I get a vaccine?” that is organized alphabetically by states and territories and linked to their individual web pages. The unit of assessment for this study was the state or territory website (URLs) provided on the CDC vaccine web page. The 54 selected websites included all 50 states official websites, plus 4 territory-level vaccine websites.

Web accessibility evaluation tools

We employed automatic accessibility testing tools to assess websites’ accessibility in accordance with the WCAG 2.0 and 2.1 standards by using WAVE, AChecker, and SortSite evaluation tools. Each automated tool has its own algorithms which vary in accuracy. Using multiple automated tools helps overcome some of the limitations of automated evaluation which ultimately enhances the reliability of the evaluation.

WAVE is an online free tool launched in 2001 by WebAIM. It provides color-coded icons to enable testers to check for possible accessibility problems on a web page: red (signifying accessibility errors and contrast errors), yellow (signifying alerts that may cause accessibility problems where human judgment is needed), green (signifying features that improve when implemented properly), blue (signifying structural, navigational, or semantic elements), and purple (signifying the use of Accessible Rich Internet Applications [ARIA] within a web page). There are several studies that have evaluated web accessibility of health-related websites using the WAVE as an automated evaluation tool (eg,,) In this article, WAVE was used to assess the web accessibility of the selected websites against the WCAG 2.1 standard.

A Checker is a reliable, free online tool used by researchers to assess website accessibility., In fact, AChecker has been found to be the most common tool used for WCAG 2.0 verification. It outlines 3 levels of problems: “known” (certain accessibility barriers that should be resolved), “likely” (probable barriers that require human decision), and “potential” (AChecker unable to identify problems, requiring a human check).

Unlike WAVE and AChecker, SortSite tool is capable of automatically assessing an entire website against WCAG 2.1 standards. It generates a summary report with the number of pages that have accessibility issues across the 3 compliance levels.

RESULTS

AChecker

We utilized Achecker to assess the web accessibility based on WCAG 2.0 guidelines. A table summarizing the analysis results of known problems (KP), likely problems (LP), and potential problems (PP) in accordance with WCAG 2.0 at the 3 conformance levels: A, AA, and AAA is located in Supplementary Appendix A.

The outcome of the web accessibility analysis was classified into 3 categories: passed, conditional, and failed. The outcome was determined by the type of problems and their quantity per web page. If there were no reported violations, the web page passed the examination. The web page was given a conditional pass if it had zero KPs and at least 1 LP and/or PP. If a web page had at least 1 KP, it failed the examination, since it signified crucial issues that necessitated immediate attention of the web developer. Only 5 state-level web pages (9.25% of 54; ie, Arkansas, Delaware, Idaho, North Carolina, and Virginia) passed the Achecker assessment by having 0 problems across all WCAG 2.0 levels.

The average number of KPs per web page was 2.22 (SD = 3.76) at level A, 12.98 (SD = 29.24) at level AA, and 21.20 (SD = 64.80) at level AAA. Figure 1 depicts some examples of common accessibility problems detected by Achecker. A bad gateway incident was encountered for the Washington state web page when the URL was entered in the Achecker tool; thus, an HTML file upload method was used for this web page to run the Achecker analysis. All other sampled web pages were analyzed using URLs. The most frequently identified accessibility KPs in accordance with the POUR principles are summarized in Table 1.

Figure 1.

Examples of common accessibility problems as per AChecker.

Table 1.

Known problems as per WCAG 2.0 POUR principles

Known problems	WCAG 2.0 (Level A)	WCAG 2.0 (Level AA)	WCAG 2.0 (Level AAA)
	no., (%)	no., (%)	no., (%)
Principle 1:  Perceivable
1.1 Text alternatives: Provide text alternatives for any nontext content.	59 (49.16)	99 (14.12)	115 (10.04)
1.2 Time-based media: Provide alternatives for time-based media.	0	0	0
1.3 Adaptable: Create content that can be presented in different ways without losing information or structure.	21 (17.5)	25 (3.56)	27 (2.35)
1.4 Distinguishable: Make it easier for users to see and hear content including separating foreground from background.	0	497 (70.8)	926 (80.9)
Principle 2:  Operable
2.1 Keyboard accessible: Make all functionalities available from a keyboard.	0	0	0
2.2 Enough time: Provide users enough time to read and use content.	0	0	0
2.3 Seizures: Do not design content in a way that is known to cause seizures.	0	0	0
2.4 Navigable: Provide ways to help users navigate, find content, and determine where they are.	14 (11.67)	53 (7.56)	49 (4.27)
Principle 3:  Understandable
3.1 Readable: Make text content readable and understandable.	4 (3.33)	4 (0.57)	4 (0.34)
3.2 Predictable: Make web pages appear and operate in predictable ways.	0	0	0
3.3 Input Assistance: Help users avoid and correct mistakes.	13 (10.83)	14 (1.99)	15 (1.31)
Principle 4:  Robust
4.1 Compatible: Maximize compatibility with current and future user agents, including assistive technologies.	9 (7.5)	9 (1.28)	9 (0.0001)
Total	120 (100)	701 (100)	1145 (100)

WAVE

Results of assessing accessibility using the WAVE checker are detailed in Table 2 in Supplementary Appendix A. The results show significant variation among state websites in the number of accessibility errors that require fixing. There are websites that show 0 errors (eg, Wyoming, Mississippi, and Ohio) and states that show high error counts (eg, Kentucky: 58 errors and Virginia: 11 errors). We categorized the errors that require fixing using the POUR framework (see Table 2). Figure 2 represents some examples of common errors identified by WAVE.

Table 2.

Accessibility errors categorized by POUR framework

Errors	Count (%)
Principle 1: Perceivable
1.1 Text alternatives: Provide text alternatives for any non-text content.	53 (14.48)
1.2 Time-based media: Provide alternatives for time-based media.	0
1.3 Adaptable: Create content that can be presented in different ways without losing information or structure.	93 (25.41)
1.4 Distinguishable: Make it easier for users to see and hear content including separating foreground from background.	0
Principle 2:  Operable
2.1 Keyboard accessible: Make all functionalities available from a keyboard.	13 (3.55)
2.2 Enough time: Provide users enough time to read and use content.	0
2.3 Seizures: Do not design content in a way that is known to cause seizures.	0
2.4 Navigable: Provide ways to help users navigate, find content, and determine where they are.	109 (29.78)
Principle 3:  Understandable
3.1 Readable: Make text content readable and understandable.	2
3.2 Predictable: Make web pages appear and operate in predictable ways.	0
3.3 Input Assistance: Help users avoid and correct mistakes.	18 (4.92)
Principle 4:  Robust
4.1 Compatible: Maximize compatibility with current and future user agents, including assistive technologies.	78 (21.31)
Total	366 (100)

Figure 2.

Examples of common accessibility errors as per WAVE.

SortSite

An entire website accessibility assessment was run on 22 websites for which its home pages passed the WAVE and/or AChecker tests with 0 reported issues. Results of assessing the accessibility of these websites using the automatic SortSite checker are detailed in Table 3 in Supplementary Appendix A. Results show that only 2 websites (ie, California and Montana) met the WCAG 2.1 standards entirely with 0 reported issues. One site (ie, Mississippi) could not be scanned by SortSite because the start page was blocked. The remaining scanned websites contained nonconformities with WCAG 2.1 standards. While some of the websites had high number of pages with accessibility issues (eg, Washington: 502 pages, Wyoming: 285 pages), others had only a few pages with accessibility issues (eg, Ohio with 3 pages only).

DISCUSSION

To the best of our knowledge, this is the first research study to assess accessibility of COVID-19 vaccine registration websites in the US. Given that government-sponsored websites are expected to present reliable and trusted health information, people including those with disabilities, are more likely to use these websites than commercial ones. Thus, compliance with the standards of the W3C is crucial in bridging the gap between digitally underserved users (ie, individuals with physical impairment) and information–affluent users. A recent cross-country study on public health website accessibility found that most of the investigated public health websites (N = 25) were not accessible enough to people with impairments; in fact, only 3 websites passed the AChecker test with 0 violations. Similarly, our results revealed that individuals with disabilities suffer from inequitable access because of suboptimal compliance with WCAG 2.0 and 2.1 guidelines, as determined using the AChecker, WAVE, and SortSite tools. This implies that states should prioritize accessibility when designing their websites. Although a few home pages showed full compliance with WCAG 2.0 across levels A, AA, and AAA, the preponderance of the investigated home pages are encumbered with several issues that need to be addressed to ensure that users with disabilities can successfully navigate them. Moreover, only 2 websites showed full conformities with WCAG 2.1 standards among those considered for an entire website accessibility evaluation.

We offer the following recommendations based on the reported accessibility violations identified and analyzed using WAVE and AChecker.

Guideline 1.1. on text alternatives under the perceivable principle was mostly violated at conformance level A, with 59 reported incidences of KP in all the investigated home pages, as registered by AChecker. This finding implies that more attention should be given to incorporating text alternatives for any nontext content in a way that allows modification into other forms based on individual needs. These modifications cover braille, large print, speech, symbols, or simpler language. For instance, individuals with visual impairment would not be able to see the information accompanying a figure; thus, a text alternative is provided to enable them to shift the information modes from text to speech.

In accordance with WCAG 2.0 levels AA and AAA, the issue primarily highlighted by AChecker was adherence to Guideline 1.4, regarding distinguishability under the perceivable principle. This guideline recommends the provision of an easy-to-perceive default presentation to accommodate individuals with visual and hearing impairments as they face difficulties in separating foregrounds from backgrounds. There were 497 (70.89%) reported violations at level AA and 926 (80.87%) reported violations at level AAA. These findings suggest that the majority of the investigated home pages are deficient in the usage of color, contrast, audio components, text resizing, text spacing, and/or text images.

Unlike AChecker, the WAVE tool found that Guideline 2.4 on navigability under the operable principle was mostly violated, with 29.78% errors identified. This result implies that more attention should be paid to the provision of methods that assist users in navigating the web pages. These measures include the provision of page titles, section heading for content organization, labels, and purposes of links among others. Although these problems were the most frequently occurring, there are other issues that require attention, including adaptability, readability, input assistance, and compatibility.

In conjunction with automated web accessibility evaluation tools, user testing is highly encouraged, with initiatives that involve people with disabilities as they are the ones who can provide parameters regarding their capabilities. It is important to integrate assistive technologies during web accessibility evaluation.

Automated and human/manual evaluation processes should be continuously carried out because websites are typically subject to changes/updates. A regular evaluation is favorable because it ensures that any emerging issues are identified and then resolved.

Design, development, and maintenance phases of websites should all incorporate accessibility.

The provision of training and accessibility manuals, as well as the enhancement of awareness among web developers, web masters, and designers, is encouraged to help ensure web accessibility.

The current research is not without limitations. First, we limited our sample to official state and territory websites. There are other entities—including counties, cities, and private health service providers—that facilitate vaccine registration. Future research may also assess those websites for compliance with accessibility. Second, the web accessibility analysis was restricted to the use of WAVE, AChecker, and SortSite. Other automated tools may cover different aspects of accessibility standards. Third, our results were derived from automated accessibility evaluation and may, therefore, have missed certain accessibility problems that people with disabilities may experience in real life. Nevertheless, such automated testing tools highlighted some major accessibility errors that need resolution. A longitudinal assessment of the vaccine websites is recommended for future studies to examine web accessibility improvement over time.

CONCLUSION

This study highlighted the need to ensure the compliance of official COVID-19 vaccine registration websites with accessibility guidelines, such as those issued by the W3C. The results showed that the websites are encumbered with certain accessibility problems, as identified using WCAG 2.0 and 2.1 standards. These accessibility shortcomings may pose difficulties to users with disabilities as they try to register for vaccines. Our recommendations should help states and other relevant entities design websites that adhere to accessibility standards in furtherance of equitable public health access.

FUNDING

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

AUTHOR CONTRIBUTIONS

SA and WC drafted the main manuscript text. SA and WC conducted the data analyses. All authors read and approved the final manuscript.

DATA AVAILABILITY STATEMENT

The data underlying this article are available in the article and in its online supplementary material.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

None declared.

Click here for additional data file.