COVID-19 vaccine website accessibility dashboard.

BACKGROUND: The COVID-19 pandemic has exacerbated historical inequities for people with disabilities including barriers in accessing online information and healthcare appointment websites. These barriers were brought to the foreground during the vaccine rollout and registration process.
OBJECTIVE: This cross-sectional study aimed to examine accessibility of U.S. state and territory COVID-19 information and registration centralized websites.
METHODS: The Johns Hopkins Disability Health Research Center created a COVID-19 Vaccine Dashboard compiling COVID-19 information and vaccine registration web pages from 56 states and territories in the United States (U.S.) reviewed between March 30 through April 5, 2021 and analyzed accessibility using WAVE Web Accessibility Evaluation Tool (WAVE). WAVE identifies website accessibility barriers, including insufficient contrast, alternative text, unlabeled buttons, total number of errors, and error density. Web pages were ranked and grouped into three groups by number of errors, creating comparisons between states on accessibility barriers for people with disabilities.
RESULTS: All 56 U.S states and territories had COVID-19 information web pages and 29 states had centralized state vaccine registration web pages. Total errors, error density, and alert data were utilized to generate accessibility scores for each web page, the median score was 259 (range = 14 to 536 and IQR = 237) for information pages, and 146 (range = 10 to 281 and IQR = 105) for registration pages.
CONCLUSIONS: These results highlight barriers people with disabilities may encounter when accessing information and registering for the COVID-19 vaccine, which underscore inequities in the pandemic response for the disability community and elevate the need to prioritize accessibility of public health information.

The COVID-19 pandemic has disproportionately impacted the disability community, compounding already existing barriers that individuals with disabilities face in accessing healthcare, as well as public health information.1, 2, 3, 4, 5 These gaps in access create inequities in accessing COVID-19 testing, treatment, and vaccination. , People with disabilities are more likely to have underlying conditions that impact the immune system and make them exponentially more susceptible to severe and negative COVID-19 outcomes than the general population. , This may result in some individuals with disabilities requiring personal care assistance, and the disruption of caregiving and support services due to the need for social distancing may also increase the risk for health complications unrelated to COVID-19. In particular, individuals with disabilities have a multitude of unique obstacles within the health care system including, but not limited to, difficulties with transportation and barriers in accessing information that have been exacerbated by COVID-19 regulations such as the social distancing mandates. People with disabilities are also more likely than people without disabilities to live in nursing homes or congregate care settings, which have been associated with much higher risk of COVID-19 exposure.2, 3, 4 In addition, for certain disabilities (such as sensory disabilities), there can be differential societal implications of social measures taken to combat the spread of COVID-19. For example, someone who is Blind cannot tell if the people around them are wearing a mask or maintaining social distancing. For these combinations of reasons, people with disabilities are more at risk and differentially impacted by the pandemic, and therefore prioritized for COVID-19 vaccine allocation.

The disability community has historically faced barriers to accessing information.1, 2, 3, 4, 5 The impact of these gaps in access have been elevated during the pandemic. , Inaccessibility of COVID-19 information poses significant barriers, as individuals with disabilities may face longer delays in obtaining information about COVID-19 and receiving COVID-19 vaccine, creating inequities in health and daily life. Gaps in accessibility are compounded by the COVID-19 vaccine registration websites, potentially resulting in greater challenges to register for vaccination for people with disabilities.7, 8, 9, 10 For example, one study analyzing data collected from U.S. state health department websites reported multiple website accessibility barriers including lack of mobile access to COVID-19 vaccine websites, absence of non-English language options, poor readability, and failure to disclose critical information regarding web-based scheduling or availability. While this article is focused on U.S. COVID-19 website, gaps in website accessibility are widespread. One Australian study assessed a variety of WCAG violations, such as low background contrast or lack of alternative or descriptive text, on local ACT COVID-19 information websites while a Canadian article evaluated the potential impact between WCAG score errors in Ontario public health websites and the ease in which those with disabilities had access to COVID-19 vaccine appointments. ,

This project focused on website accessibility, as under the Americans with Disabilities Act (ADA) and the Rehabilitation Act of 1973, states have a responsibility to provide accessible information to people with disabilities. The ADA mandates against discrimination on the basis of disability. Although Title III of the ADA focuses on “public accommodations and commercial facilities”, internet websites are not explicitly mentioned. The Department of Justice has taken the position to consider websites public domain, and as a result, website accessibility is protected under Title III of the ADA. Therefore, under the ADA, state COVID-19 information and registration websites need to be accessible to individuals with disabilities. In addition to the ADA, Section 504 of the Rehabilitation Act of 1973 requires all recipients of federal government aid to ensure their activities, programs, and services are accessible to people with disabilities, including websites. In 2018, Section 508 of the Rehabilitation Act adopted the internationally accepted Website Content Accessibility Guidelines (WCAG) which legally requires federal agencies (or parties doing business with a federal agency) to ensure all pages of their websites are accessible to individuals with disabilities.

This project aimed to examine and quantify the accessibility of U.S. COVID-19 state vaccine information and registration websites and disseminate this information via the Johns Hopkins Disability Health Research Center COVID-19 Vaccine Dashboard. Although website errors and accessibility have been reliably measured by the Web Accessibility Evaluation Tool (WAVE), which has been used to assess website accessibility in many settings15, 16, 17 including a study on two local Australian government COVID-19 websites, this project aimed to track the accessibility of COVID-19 vaccine information across the U.S.

Methods

Data collection

This cross-sectional study utilized data on website accessibility metrics determined from state/territory public health department COVID-19 vaccine web pages across all 50 U.S. states, the District of Columbia, and 5 territories collected by searching the internet for the most up-to-date web page Uniform Resource Locator (URL). As the data was publicly available and did not involve human subjects, IRB approval was not required. Web pages were monitored weekly to ensure that each URL was still active and to update the states and territories that created new web pages. The entire website was not scored. Two types of web pages were analyzed: COVID-19 information and COVID-19 vaccine registration. COVID-19 information web pages were determined based on containing state/territory-wide guidelines on business regulations, social distancing policies, vaccine eligibility, etc. Our analysis focused on the main COVID-19 state information webpage or landing page. As a result, no additional webpages were studied (only one COVID-19 information webpage URL was analyzed for each U.S. state or territory). A registration web page was determined based on containing a centralized portal for residents to register for COVID-19 vaccine appointments. Our analysis was limited to single state/territory-level vaccine pre-registration or registration government web pages and did not include data from states or territories without a centralized state-level registration vaccine portal. For example, states or territories that simply listed the contact information of each locale's health department or displayed an interactive map of which providers and commercial pharmacies had vaccines available were not captured. In addition, states or territories that listed multiple providers, each offering independent pathways for registration, were not included. These web pages were excluded, despite being centralized (i.e., hosted by state- and territory-wide departments of health), because these web pages only served as intermediary information hubs between the resident and vaccine provider. During data collection, state COVID-19 vaccine websites were split evenly between two primary coders, G.J and D.H. Each person had 4 days to collect the data, after which checks were completed, graphs created, and the data was published in the Johns Hopkins COVID-19 Vaccine Dashboard each Wednesday. Data from reviews conducted between March 30 until April 5, 2021 are included in these analyses.

Assessment of web page accessibility

The accessibility scores and rankings for COVID-19 web pages were generated from automatic testing data provided by WAVE, a web accessibility evaluation tool administered by WebAIM at Utah State University. WAVE is a commonly utilized accessibility testing tool that has been in development since 2001. WAVE analyzes patterns in web site codes to identify accessibility barriers such as insufficient contrast, alternative text, unlabeled buttons, total number of errors, and error density. A free, online version of WAVE is publicly available for download at https://wave.webaim.org/. Browser extensions for Chrome and Firefox are also available. The WAVE API was utilized for automated testing of the COVID-19 web pages.

The number of detected accessibility errors, the density of those errors on the page (errors divided by number of page elements), and the number of likely/potential accessibility issues “(“Alerts” within WAVE) were considered when generating the rankings. Automatically detectable accessibility issues do not represent all accessibility issues, but typically indicate barriers for users with disabilities and non-compliance with accessibility guidelines and best practices. Accessibility scores were generated for each web page by rank ordering the 56 COVID-19 information pages (and 29 vaccine registration pages) on number of errors, error density, and likely/potential issues. Weightings were then applied to each of the rank orders (ties were treated equally) to calculate an overall access score—with error ranks being assigned a score of 6, error density ranks being assigned a score of 3, and likely/potential issues assigned a score of 1. This access score is not necessarily a good measure of overall accessibility but provides a normalized metric for comparisons across the web pages. The dashboard presented pages ordered or ranked by the overall access score.

For information web pages, WAVE was run automatically, while for registration web pages, accessibility scores were calculated by navigating to the first web page of the registration site where user-specific information (e.g., name, age, etc.) was entered. Rather than capturing the entire website, the first relevant information or registration page was considered reflective of all web pages of each site. WAVE was run automatically if the URL was linked to the correct web page of the vaccine portal. However, WAVE was run manually (using a browser extension) on the next page of the registration portal if account creation or CAPTCHA (completely automated public turing test to tell computers and humans apart) was required on the first page before entering user-specific data for 9 of the 20 COVID-19 vaccine registration web pages (leaving 20 of the 29 being scores using automated WAVE methods). In manual assessments, testers captured total errors, error density, and alert data—the same data collected via the automated process for pages that could be directly accessed by WAVE. However, automated WAVE data captured more in-depth analyses of the types of errors, such as contrast errors, empty links/buttons, images without alternative text, and unlabeled form inputs. If a state or territory required the creation of a new account to register for vaccination, the account creation page was used to assess accessibility: no new accounts were created by the researchers during this process.

Dashboard

These data were used to create the Johns Hopkins Disability Health Research Center COVID-19 Vaccine Dashboard. The dashboard was made publicly available, focused on ensuring accessibility of information, and was updated weekly to both enhance accessibility to vaccine registration URLs and highlight important considerations about mitigating disparities in COVID-19 information access for people with disabilities. The initial dashboard release focused on COVID-19 vaccine prioritization for people with disabilities, but was expanded to also examine state-level COVID-19 vaccine information and registration website accessibility. These analyses present a cross-sectional snapshot of the accessibility data for the week of March 30 to April 5, 2021. This week was selected as it is the “baseline” when all state website accessibility data began to be tracked.

Statistical analysis

Accessibility rankings were assigned based on an inverse relationship with accessibility score, as higher accessibility rankings corresponded to lower frequency of errors (i.e., lower accessibility score). Contrast errors, error density, total errors, accessibility scores, rankings, and URLs of COVID-19 vaccine information and registration web pages were tabulated. Additional properties of registration web pages—account requirement, pre-registration status, and CAPTCHA implementation—were also tabulated. For both information and registration web pages, the mean, standard deviation, median, range, and interquartile range (IQR) were calculated and tabulated for accessibility scores, total errors, error density, insufficient contrast errors, empty links/buttons, images without alternative text, and unlabeled form inputs. Only those registration web pages that did not preclude automatic WAVE testing due to CAPTCHA or account creation before accessing the main page were included in the table for combined statistics. Accessibility scores for information and registration web pages in the states and territories that hosted both types were plotted as bar graphs. Information and registration web pages were divided into three groups based on their accessibility scores to compare and contrast centralized web pages in various U.S. states and territories. The top third contained the fewest accessibility errors while the bottom third contained the most accessibility errors. Maps color-coded by group were created for easy visualization and comparison of information and registration web page accessibility. The correlation between state information and registration pages was calculated using Pearson's correlation coefficient (and a paired t-test to obtain the p-value) for the subset of 29 states/territories that had centralized registration portals. Statistical analyses were complete using R statistical software (version 3.6.3; The R Foundation).

Results

Accessibility scores were calculated for 56 U.S. states and territories with COVID-19 information web pages (Table 1 ), and the 29 states that had a centralized (i.e., state/territory-wide) COVID-19 registration web page as of April 5, 2021 (Table 2 ). Total accessibility scores for each state and territory information and registration web pages were examined (Fig. 1 a, b, c). The within-state correlation between information and registration page accessibility scores was 0.304 (p < 0.001). States with information and registration web pages that were both in the top third of states with the fewest accessibility errors were California, New Jersey, Maine, and Ohio. States (and one territory) with both types of web pages in the bottom third with the most accessibility errors were New York, Nebraska, Northern Mariana Islands, Florida, Idaho, New Mexico, and Illinois. The most common issues were: (1) text with insufficient contrast to the background (which is more difficult to read, particularly for users with certain visual disabilities), (2) empty links and buttons that would be announced by screen readers but lack descriptions, (3) images without descriptive alternative text for screen reader users, and (4) form inputs not properly labeled with descriptive text (Table 3 ). Other accessibility issue types, such as if a page code did not define the natural language of the document or if headings or ARIA code were used incorrectly, were also considered in the accessibility scores.

Table 1

COVID-19 information web page accessibility ranking by United States state/territory (data as of April 5, 2021).

Rank	State/territory	Score	Total errors	Error density	Contrast errors	URL
1	Minnesota (MN)	14	0	0.00%	0	https://www.health.state.mn.us/diseases/coronavirus/vaccine/plan.html
2	Kansas (KA)	20	0	0.00%	0	https://www.kansasvaccine.gov/
3	Louisiana (LA)	31	0	0.00%	0	https://ldh.la.gov/covidvaccine/
4	California (CA)	45	0	0.00%	0	https://covid19.ca.gov/vaccines/
5	Arkansas (AR)	53	0	0.00%	0	https://www.healthy.arkansas.gov/programs-services/topics/covid-19-vaccination-plan
6	Washington (WA)	54	0	0.00%	0	https://www.doh.wa.gov/Emergencies/COVID19/vaccine
7	Maryland (MD)	55	0	0.00%	0	https://covidlink.maryland.gov/content/vaccine/
8	New Jersey (NJ)	75	1	0.12%	0	https://covid19.nj.gov/faqs/nj-information/slowing-the-spread/who-is-eligible-for-vaccination-in-new-jersey-who-is-included-in-the-vaccination-phases
9	Maine (ME)	96	1	0.36%	0	https://www.maine.gov/covid19/vaccines
10	New Hampshire (NH)	97	1	0.16%	0	https://www.vaccines.nh.gov/
10	Vermont (VT)	97	1	0.15%	1	https://www.healthvermont.gov/covid-19/vaccine
12	North Dakota (ND)	141	3	0.18%	1	https://www.health.nd.gov/covid-19-vaccine-information
13	Rhode Island (RI)	145	2	0.20%	0	https://covid.ri.gov/vaccination
14	Ohio (OH)	151	3	0.40%	2	https://coronavirus.ohio.gov/wps/portal/gov/covid-19/covid-19-vaccination-program
15	Georgia (GA)	179	5	0.38%	5	https://dph.georgia.gov/covid-vaccine
15	Michigan (MI)	179	3	0.51%	1	https://www.michigan.gov/coronavirus/0,9753,7-406-98178_103214---,00.html
17	Connecticut (CT)	188	3	0.97%	3	https://portal.ct.gov/Coronavirus/COVID-19-Vaccination---Phases
18	U.S. Virgin Islands (VI)	202	4	0.93%	4	https://www.covid19usvi.com/vaccines
19	Pennsylvania (PA)	205	5	0.46%	0	https://www.health.pa.gov/topics/disease/coronavirus/Vaccine/Pages/Vaccine.aspx
20	Indiana (IN)	207	5	0.15%	2	https://www.coronavirus.in.gov/vaccine/index.htm
20	Montana (MT)	207	3	1.10%	3	https://dphhs.mt.gov/covid19vaccine
22	Texas (TX)	212	4	0.70%	1	https://www.dshs.state.tx.us/coronavirus/immunize/vaccine.aspx
23	Wisconsin (WI)	232	5	0.43%	2	https://www.dhs.wisconsin.gov/covid-19/vaccine-about.htm
24	Alabama (AL)	234	4	1.10%	1	https://www.alabamapublichealth.gov/covid19vaccine/index.html
24	Nevada (NV)	234	5	0.85%	3	https://www.immunizenevada.org/nv-covid-fighter
26	Puerto Rico (PR)	235	5	1.14%	5	https://www.vacunatepr.com/covid-19
27	Guam (GU)	246	5	0.96%	3	https://dphss.guam.gov/covid-19-vaccinations/
28	Oregon (OR)	247	5	0.54%	1	https://covidvaccine.oregon.gov/
29	Kentucky (KY)	271	7	0.38%	5	https://govstatus.egov.com/ky-covid-vaccine
30	North Carolina (NC)	286	6	0.65%	6	https://covid19.ncdhhs.gov/vaccines
31	Colorado (CO)	287	7	0.39%	2	https://covid19.colorado.gov/for-coloradans/vaccine/vaccine-for-coloradans
32	Utah (UT)	304	7	0.51%	5	https://coronavirus.utah.gov/vaccine-distribution/
33	Massachusetts (MA)	305	9	0.87%	6	https://www.mass.gov/info-details/massachusetts-covid-19-vaccination-phases
34	Tennessee (TN)	324	7	1.24%	3	https://www.tn.gov/health/cedep/ncov/covid-19-vaccine-information.html
35	Oklahoma (OK)	334	6	3.80%	0	https://vaccinate.oklahoma.gov/
36	South Carolina (SC)	344	11	0.60%	6	https://scdhec.gov/covid19/covid-19-vaccine
37	West Virginia (WV)	359	11	2.38%	10	https://dhhr.wv.gov/COVID-19/Pages/Vaccine.aspx
38	Alaska (AL)	364	11	1.72%	10	http://dhss.alaska.gov/dph/epi/id/pages/COVID-19/vaccine.aspx
39	New York (NY)	379	12	2.17%	8	https://covid19vaccine.health.ny.gov/
40	Nebraska (NE)	380	14	0.96%	6	http://dhhs.ne.gov/Pages/COVID-19-Vaccine-Information.aspx
40	Northern Mariana Islands (MP)	380	15	2.09%	12	https://www.vaccinatecnmi.com/
42	Wyoming (WI)	407	17	2.24%	17	https://health.wyo.gov/publichealth/immunization/wyoming-covid-19-vaccine-information/
43	Delaware (DE)	416	16	2.57%	13	https://coronavirus.delaware.gov/vaccine/
44	Florida (FL)	417	15	2.63%	15	https://floridahealthcovid19.gov/covid-19-vaccines-in-florida/
45	Iowa (IA)	419	22	1.30%	14	https://coronavirus.iowa.gov/pages/vaccineinformation
46	Missouri (MO)	427	17	2.56%	17	https://covidvaccine.mo.gov/
47	Hawaii (HI)	433	19	1.87%	17	https://hawaiicovid19.com/vaccine/
48	South Dakota (SD)	450	14	4.47%	8	https://doh.sd.gov/COVID/Vaccine/
49	District of Columbia (DC)	456	17	2.49%	13	https://coronavirus.dc.gov/vaccine
50	Idaho (ID)	459	18	2.38%	16	https://coronavirus.idaho.gov/covid-19-vaccine/
51	New Mexico (NM)	468	20	2.49%	20	https://cv.nmhealth.org/covid-vaccine/
52	American Samoa (AS)	504	33	2.71%	30	https://www.facebook.com/asdoh.hotline/
53	Virginia (VA)	512	51	3.30%	42	https://www.vdh.virginia.gov/covid-19-vaccine/
54	Illinois (IL)	523	110	9.67%	104	https://www.dph.illinois.gov/covid19/vaccine-faq
55	Arizona (AZ)	530	56	5.62%	29	https://azdhs.gov/
56	Mississippi (MS)	536	81	3.03%	77	https://msdh.ms.gov/msdhsite/_static/14,0,420,976.html

Data as of April 5, 2021.

Table 2

COVID-19 vaccine registration web page accessibility ranking by United States state/territory.

Rank	State/territory	Score	Total errors	Error density	Requires account creation/log-in	Pre-registration	Has CAPTCHAa	CAPTCHAa audio option	URL
1	Nevada (NV)	10	0	0.00%	No	No	No	No	https://vax4nv.nv.gov/patient/s/
2	California (CA)	14	0	0.00%	No	No	No	No	https://myturn.ca.gov/screening
2	Maine (ME)	14	0	0.00%	No	No	No	No	https://vaccinateme.maine.gov/screening
2	Massachusetts (MA)	14	0	0.00%	No	Yes	No	No	https://vaccinesignup.mass.gov/#/
5	Indiana (IN)	17	0	0.00%	No	No	No	No	https://vaccine.coronavirus.in.gov/
6	Virginia (VA)	20	0	0.00%	No	Yes	No	No	https://vax.preregister.virginia.gov/#/
7	Alabama (AL)	22	0	0.00%	No	No	No	No	https://govstatus.egov.com/vaccine-eligibility-form
8	District of Columbia (DC)	99	1	0.23%	No	Yes	No	No	https://v51r5a21s.dc.gov/cvq_dc/
8	New Jersey (NJ)	99	1	0.15%	No	Yes	Yes	Yes	https://covidvaccine.nj.gov/
10	Ohio (OH)	110	1	1.12%	No	No	No	No	https://gettheshot.coronavirus.ohio.gov/screening
11	Maryland (MD)	111	1	0.83%	Yes	Yes	No	No	https://onestop.md.gov/users/sign_up?registration_context=%2Fpreregistration
12	Tennessee (TN)	130	2	0.36%	No	No	Yes	Yes	https://vaccinate.tn.gov
13	Arizona (AZ)	131	2	1.38%	Yes	No	No	No	https://podvaccine.azdhs.gov/signup
14	Oklahoma (OK)	139	2	1.00%	No	No	Yes	Yes	https://vaccinate.oklahoma.gov/en-US/
15	Delaware (DE)	146	2	1.56%	No	No	Yes	No	https://vaccinerequest.delaware.gov/s/de-vms-screening?language=en_US
15	Vermont (VT)	146	2	2.44%	Yes	No	No	No	https://vermont.force.com/events/s/selfregistration
17	Missouri (MO)	154	2	1.79%	No	No	No	No	https://modhss.iad1.qualtrics.com/jfe/form/SV_231d5TxZxkGedCt
18	Minnesota (MN)	166	3	0.13%	No	No	No	No	https://vaccineconnector.mn.gov/covid-19%20vaccine/
19	West Virginia (WV)	175	4	0.35%	Yes	No	No	No	https://member.everbridge.net/747122446041089/new
20	Wisconsin (WI)	182	3	0.73%	No	No	Yes	Yes	https://vaccinate.wi.gov/en-US/
21	New York (NY)	200	4	1.44%	No	No	Yes	No	https://am-i-eligible.covid19vaccine.health.ny.gov/Public/prescreener
22	Idaho (ID)	204	5	0.62%	No	Yes	No	No	https://covidvaccine.idaho.gov/register
23	Illinois (IL)	221	7	2.95%	No	No	No	No	https://covidvaccination.dph.illinois.gov/questionnaire
24	Nebraska (NE)	223	3	3.49%	No	No	Yes	Yes	https://vaccinate.ne.gov/en-US/
25	Georgia (GA)	229	8	1.43%	No	Yes	No	No	https://myvaccinegeorgia.com/en/site/ALL
26	Northern Mariana Islands (MP)	233	17	0.70%	No	Yes	No	No	https://www.vaccinatecnmi.com/covid-19/registration/
27	New Hampshire (NH)	250	12	3.20%	No	No	No	No	https://sonh-community.force.com/providers/s/
28	Florida (FL)	274	32	8.99%	No	Yes	No	No	https://myvaccine.fl.gov/#/RegistrationForm
29	New Mexico (NM)	281	57	3.44%	No	No	Yes	No	https://cvvaccine.nmhealth.org

Abbreviations: CAPTCHA=Completely automated public turing test to tell computers and humans apart

Data as of April 5, 2021.

Fig. 1

United States COVID-19 information and vaccine registration web page accessibility (data as of April 5, 2021). (a) COVID-19 information and (b) COVID-19 vaccine registration pages ranked in three groups by number of accessibility errors. (c) COVID-19 information and COVID-19 vaccine registration web page accessibility scores for the 29 states and territories with both types.

Table 3

Statistics for common COVID-19 information web page accessibility errors across the United States.

Issue type	Number of issues detected (% of all errors)	Mean issue per web page (SD)
Insufficient text contrast	549 (78.2%)	9.80 (18.06)
Empty links and buttons	55 (7.8%)	0.98 (1.69)
Missing alternative text	46 (6.6%)	0.82 (2.91)
Missing input labels	18 (2.6%)	0.34 (0.72)

COVID-19 information web page accessibility scores for all 56 states and territories ranged from 14 (Minnesota) to 536 (Mississippi) with a median score of 259 (Table 4 ). Errors ranged from zero (Minnesota, Kansas, Louisiana, California, Arkansas, Washington, and Maryland) to 110 (Illinois) with a median value of 5.50. Mississippi (the state with the lowest accessibility score) had 81 total errors. Error density ranged from 0% to 9.67% with a median value of 0.90%. Insufficient contrast errors ranged from 0 to 104 with a median of 3.50. Errors with empty links and buttons had a range of 0–8 with a median value of 0. Images without alternative text ranged from 0 to 21 with a median of 0, and errors with form inputs not labeled had a median value of 0 with a range of 0–3.

Table 4

Statistics for COVID-19 information web page accessibility data across the United States.

Accessibility	Mean (SD)	Median (range)	Interquartile range
Accessibility scores	274.20 (151.37)	259 (14, 536)	172, 409.25
Total errors	12.54 (19.72)	5.50 (0, 110)	3, 15
Error density, %	1.41 (1.68)	0.90 (0, 9.67)	0.38, 2.28
Insufficient contrast errors	9.80 (18.06)	3.50 (0, 104)	1, 12.25
Empty links/buttons	0.98 (1.69)	0 (0, 8)	0, 1
Images without alternative text	0.82 (2.91)	0 (0, 21)	0, 1
Form inputs not labeled	0.34 (0.72)	0 (0, 3)	0, 0

Abbreviations: SD = standard deviation

Data from 50 US States, the District of Columbia, and 5 U.S. territories, as of April 5, 2021.

Of the 29 states and territories that hosted both types of websites, accessibility scores from the 29 centralized COVID-19 information web pages ranged from 14 to 530 with a median score of 324. Errors ranged from zero to 110 with a median of 7 and a mean of 14.66 (SD 22.66). Error density had a median of 0.01 (range of 0–0.10).

The accessibility scores for state COVID-19 vaccine registration web pages ranged from 10 (Nevada) to 281 (New Mexico), with a median score of 146 (Table 5 ). Error density ranged from 0 (Nevada, California, Maine, Massachusetts, Indiana, Indiana, Virginia, and Alabama) to 0.09 (Florida) with a median percentage of 0.01. Seven of the 29 states had zero total errors (Nevada, California, Maine, Massachusetts, Indiana, Virginia, and Alabama) and New Mexico (the state with the lowest accessibility ranking) had 57 total errors. Errors for the 29 COVID-19 vaccine registration web pages had a median of 2 (Table 5). Four states (Maryland, Arizona, Vermont, and West Virginia) required account creation for access and eleven states (Massachusetts, Virginia, District of Columbia, New Jersey, Maryland, Idaho, Georgia, Northern Mariana Islands, and Florida) were open for pre-registration of the vaccine (Table 2). There were eight state COVID-19 registration web pages with CAPTCHA (New Jersey, Tennessee, Oklahoma, Delaware, Wisconsin, New York, Nebraska, and New Mexico), of which five (New Jersey, Tennessee, Oklahoma, Wisconsin, and Nebraska) had an audio option for completion, which enhances accessibility for individuals with visual impairments (Table 2).

Table 5

Statistics for registration web page accessibility scores, total errors, error density, contrast errors, empty links/buttons, images without alternative text, and unlabeled form inputs.

All 29 registration web pages	Mean (SD)	Median (range)	Interquartile range
Accessibility scores	138.41 (85.52)	146 (10, 281)	99, 204
Total errors	5.90 (11.84)	2 (0, 57)	1, 4)
Error density, %	1.77 (2.06)	1.10 (0, 9.67)	0.36, 2.49
20 registration web pages	Mean (SD)	Median (range)	IQR
Insufficient contrast errors	0.95 (1.82)	0 (0, 6)	0, 1
Empty links/buttons	0.35 (0.81)	0 (0, 3)	0, 0
Images without alternative text	0.45 (0.83)	0 (0, 3)	0, 1
Form inputs not labeled	2.40 (6.44)	0.50 (0, 29)	0, 1.25

Abbreviations: SD = standard deviation

Data from 20 of the 29 registration web pages that have details on types of errors (as the remainder were extracted manually using the WAVE browser plug-in), as of April 5, 2021.

Of the 29 registration web pages, 20 were assessed using automatic WAVE testing while the other 9 (California, Maine, New Jersey, Tennessee, Oklahoma, Wisconsin, Idaho, Nebraska, and New Mexico) exhibited barriers that made manual extraction necessary (Table 5). For these 20 states, insufficient contrast errors ranged from 0 to 6 with a median of 0. The median value of errors regarding empty links and buttons was 0 with a range of 0–3. Web pages containing images without alternative text ranged from 0 to 3 with a median of 0, and errors with form inputs not labeled ranged from 0 to 29 with a median value of 0.50.

Discussion

By creating a dashboard comparing COVID-19 information and registration website accessibility, we found inequities in access to this critical information in the U.S. These results highlight important gaps in the opportunity to access COVID-19 information and vaccines for people with disabilities. The mean number of errors detected across these web pages (12.50 per page) was lower than the mean accessibility errors (51.40 per page) detected on a sample of 1 million web pages from across the internet. This lower mean number of accessibility errors may due to the fact this dashboard only examined state government websites, which generally have fewer error when compared to private webpages. Even though the accessibility of COVID-19 web pages scored better than most web pages, the importance of the information on these pages demands that any accessibility barriers be addressed, as disparities in COVID-19 outcomes and vaccination rates among the disability community may be compounded by these gaps in information access. Website errors and accessibility have been reliably measured by the Web Accessibility Evaluation Tool (WAVE) in many settings,15, 16, 17 including a study on two local Australian government COVID-19 websites, but not to the scale of encompassing all territory-wide websites.

The majority of state rankings did not have consistent accessibility rankings across COVID-19 vaccine information and registration websites. Only a few states had within-state consistency in these rankings, such as California where the COVID-19 vaccine information website was ranked 4th, and the COVID-19 vaccine registration ranked 2nd. The reasons for the within-state inconsistency in accessibility ranks are unknown. It is possible there are within-state variations in the allocation of resources and personnel focusing on website accessibility that may contribute to these differences. It is also possible that COVID-19 vaccine information websites may have been created earlier than registration websites, allowing more time for states to improve information website accessibility based on the public's feedback. States without centralized vaccine registration websites also had discrepancies between information and registration rankings, as no ranking was possible for registration websites in these cases.

It is important to consider how the disability community is impacted by barriers in accessing COVID-19 vaccine information. For instance, CAPTCHA usually requires visual input from the user, creating difficulties for individuals who are blind or are visually impaired. CAPTCHA may therefore create additional delays in or prevent vaccine registration. Delays in vaccine registration that may have resulted from inaccessible websites may result in greater risk for people with disabilities to contract COVID-19, a disease which has been proven to have disproportionally negative outcomes for this population group. Acknowledging barriers for website accessibility, the federal government has recently devoted substantial funding towards reducing barriers in accessing COVID-19 vaccines for the people with disabilities by providing access to direct phone numbers for vaccine registration and providing transportation to vaccine appointments. Information on these efforts, however, is primarily available via the state COVID-19 websites. While this critical funding will likely go far towards closing gaps in accessing vaccines, there remains limited resources and efforts towards ensuring COVID-19 information and registration is actually accessible on state public health web pages. It is important for web designers to prioritize accessibility and include disability advocates and people with disabilities at the outset of website design. Only by partnering with the disability community can web designers create public health websites which are accessible and informative for everyone. While website accessibility is an important contributing factor in barriers to COVID-19 vaccination uptake, this is only one component of the COVID-19 inequities impacting the disability community. Further data is needed to identify gaps in accessibility of all aspects of COVID-19 vaccine distribution, including transportation to the COVID-19 vaccine appointment and accessibility at the vaccine clinic site.

Despite its potential impact, the limitations of this project must be considered. While this project and resulting Dashboard focus on website accessibility, there remains important work in collecting data documenting and identifying gaps in accessibility of the physical vaccine locations and quantifying vaccine rates among people with disabilities to determine inequities. Additionally, the accessibility score/ranking system is primarily designed around people with visual disabilities, and there are barriers to online accessibility for people with other types of disabilities that were not captured. These may include, for example, video captioning for people with hearing loss, or plain language summaries for people with intellectual, executive and cognitive disabilities. Further, some websites collect more information per page (i.e., long format) while other websites collect less information per page but include multiple pages (i.e., wide format). A website with a wide format may have fewer total errors per page but may have a higher percentage of errors with respect to how much information is on a single page. Information web pages were more often in long format, likely accounting for their notably higher accessibility scores compared to those for registration web pages (Table 3, Table 4). To mitigate this, the accessibility scores included error density—the number of errors by web page elements (i.e., how dense errors are within the page content and functionality). Our method measured accessibility scores for the first page of the registration website, which is indicative of patterns in subsequent pages; however, users may abandon the process if significant accessibility barriers are encountered on subsequent pages, and future research should focus on developing a metric for capturing accessibility data across many pages of the same website rather than just the first page. Some COVID-19 vaccine registration web pages required pre-registration, including entering driver's license credentialing, or creating an account. This limitation may have created bias in our sample as we did not gain access to websites requiring pre-registration to avoid entering false information and burdening the system. While WAVE was used in this project, it is only one type of method to evaluate website accessibility. This tool analyzes patterns in web site code and design that align with accessibility compliance failures with a very high level of reliability. However, WAVE, like all automated tools, cannot fully assess all aspects of end user accessibility, and the issues it does analyze are primarily focused on users with visual disabilities. Analyzing plain language or assessing video captioning presence/quality or testing for keyboard accessibility issues would necessitate manual testing that was not conducted as part of this study. Additional aspects of accessibility, such as options for end users to register for a vaccine or get additional supports via phone, were not considered, primarily because web sites are a primary mechanism for accessing such information and functionality. Despite these limitations, the issues detected by WAVE nearly always align with negative impact on users with disabilities and thus provide a useful measure of accessibility. In addition, the COVID-19 vaccine roll-out is rapidly evolving, and some states or territories may have adjusted their information and vaccine registration portals since initial data collection.

Conclusion

This project highlights the barriers that people with disabilities may have encountered when accessing state information and registration COVID-19 vaccine websites in the U.S during the spring of 2021. Our results underscore the addressable, yet persistent inequities in the pandemic response for the disability community. The accessibility of public health information must be prioritized and supported in order to ensure that the disability community is no longer left behind.