Evaluating the accessibility of public health websites: An exploratory cross-country study.

Public health websites are regarded as official references that citizens of any country rely on for domestic and individual health affairs. For people with disabilities, public health resources are often of greater importance; they additionally provide disability context-specific information. However, to leverage the benefits of such resources for the widest demographic groups, Web accessibility requirements should be met at an acceptable level (e.g., WCAG 2.0, Level AA). This study evaluates the accessibility of a number of public health websites from 25 countries. The choice of the selected websites is determined by the extent of the COVID-19 outbreak in the corresponding countries and their rank as of late April, 2020. Ultimately, this study aims at shedding light on the current situation of accessibility to health information and pinpointing the aspects where accessibility to information falls short in public health websites. Using different evaluation tools, the overall results show that the vast majority of public health websites, of a number of different countries, still have many critical accessibility barriers, especially with regards to the perception of information and operability of the interface items. The findings of this study suggest a need for major efforts toward ensuring accessible public health resources in most of the evaluated websites. As this pattern has repeatedly occurred in many relevant studies in different parts of the world, legislation along with educating Web developers regarding Web accessibility requirements and universal design principles become an urgent necessity.

Introduction

Web-based public (i.e., government) information resources have largely contributed to the procurement of quality services and access to reliable information. These different-purpose resources have taken over many of the traditional non-digital practices toward the attainment of public services and information [1]. This has significantly expanded, especially within the major reliance on the Web and the prevalence of ubiquitous technologies for accessing Web resources, as observed nowadays. Public health websites (e.g., ministries of health) are considered among the indispensable pillars of any governmental information resources [2, 3]. They are regarded to have a higher authority over any other source in providing health information [3, 4].

On the one hand, these official resources provide reliable information concerning individual, domestic and global health affairs. Appropriate health counseling and medical advice on the prevention and control of diseases (e.g., HIV/AIDS and Cancer) and relevant subjects (e.g., COVID-19 pandemic) are usually communicated through public health resources [5], keeping citizens aware and knowledgeable against arbitrary and false resources. This is especially important as the literature ascertains that individuals of various age groups search different information resources to draw health care decisions for themselves and their close relatives [6-8], thereby risking their health when the information comes from unreliable resources.

On the other hand, comprehensive information regarding different health matters (e.g., vaccination, immunization, maternity, childcare, chronic diseases, medical facilities, mental and physical heath, medications, drugs and addiction, insurances, updates of the public health situation, etc.) provided through these portals are especially vital for the amelioration of the general public health situation and the reduction of mortality rates, targeting all age groups from infants to the elderly. This is further prolonged on the individual level; these portals are used by citizen for obtaining individual services such as issuance and renewal of health cards, access to and management of medical records, and many more.

While the previous key purposes are sufficient to highlight the significance of public health websites, there is yet one more critical role they play. These resources provide key disability-related information for people with disabilities, constituting 15% of the overall world population [9, 10]. The information is essential for these groups’ wellbeing, with a larger magnitude during unusual circumstances as they are likely to have fragile general health [11]. According to the report in [12], people who live with impairments are more likely to encounter unique risks during exceptional times such as epidemics and pandemics; they are more prone to experiencing severe illness during an outbreak [12]. The same report highlights a critical need for a thorough consideration for those demographic groups in terms of the facilitation of timely information, services, and programmatic responses via public health portals and official channels. This further emphasizes the observed importance of public health websites.

However, for people who are disabled to take full advantage of public health websites, these resources should meet Web accessibility requirements. Web accessibility refers to developing websites and digital content (i.e., text, images, audio, video, or multimedia) in a manner such that diverse users can use them, including people who are disabled [13]. In a detailed view, Web accessibility entails that any user should be able to perceive, understand, navigate, interact and contribute to the Web, with no access barriers of any kind [13]. To measure the extent to which a website is accessible (i.e., conforms to accessibility requirements), different standard Web accessibility guidelines such as WCAG have emerged in the last 3 decades. Accessibility guidelines (i.e., accessibility metrics) outline a set of standards and technical requirements for making any digital content accessible for diverse users.

Therefore, since (1) public health websites are significant in the increasing information and services they provide for the entire population, (2) the world witnesses a rise in the numbers of people who are disabled [11], (3) there exists a number of well-defined accessibility standards on the international level to guide the development of accessible information resources, and (4) facilitating accessible public health websites has the potential to impact citizens’ health-related decisions, making public health websites accessible and inclusive for all users must be ensured in every country.

The goal of this study is to examine the accessibility of a number of public health websites from 25 countries across four continents: Asia, North America, South America, and Europe. This study ultimately aims at empowering people with disabilities via pinpointing the aspects that can be improved while facilitating comparable access to information resources provided in public health websites in many countries across the world. The choice of the selected countries stems from their worldwide rank in terms of the COVID-19 outbreak extent as of late April, 2020. The evaluation approach used in this study is automatic and is conducted using Web accessibility evaluation tools according to WCAG 2.0 and WCAG 2.1 standards. The tools used in this study are: AChecker [14], WAVE [15], W3C HTML Validator [16] and W3C CSS Validator [17].

The rest of the paper is organized as follows: the next section provides a background on Web accessibility guidelines and international legislation for protecting the rights of people with disabilities, along with Web accessibility evaluation approaches. Then, an overview of literature and related work is provided. Next comes the method of this study. The results are then presented, followed by a relevant discussion of the findings and the conclusion.

Background

Users of the Web are diverse; they exhibit different needs reflecting their abilities in how they perceive, understand, navigate, and interact with Web content. For example, how a user may access textual information differs from a user who is visually impaired to a user who is sighted. This suggests that some users are impaired as opposed to others who for us are commonly known to be mainstream and for whom most services and information resources are dominantly designed.

There are many conditions characterizing an impaired user, such as blindness, visual impairments, deafness, hearing impairments, motor impairments, cognitive impairments, and seizure disorder. Besides these conditions, there is what is known as “situational impairment” or “temporary impairment.” Situational impairments result from temporary health, environmental, attentional, or physical factors [18]. We are all prone to experience many of these situational factors that in certain contexts may limit our capabilities in interacting with the surroundings, including computers and ubiquitous devices.

All impairment cases necessitate using one or more of alternative sensory channels, assistive technologies and aids, certain adjustments, or controls in order to access information. This brings information accessibility into mind. Information accessibility refers to the facilitation of access to, navigation, interaction with, and creation (i.e., contribution) of information, regardless of the user’s abilities or the environmental circumstances [19]. In other words, information (e.g., Web content) should be appropriately developed to be compatible with the aforementioned alternatives so that it can be utilized by diverse users. Therefore, a considerable proportion of users would be in a severe disadvantage dealing with Web information resources when their access needs are neglected. And hence comes the role of legislation and standards pertaining to enforcing and ensuring information accessibility.

Legislation

The growing role of the Web as a vital source of information dissemination and acquisition, along with the increasing number of the voices advocating human rights led to the establishment of legislation mandating equality with respect to Web accessibility. The right for equal access to digital information resources is profoundly recognized by the UN Convention on the Rights of People with Disabilities (CRPD). After its official adoption in 2006, CRPD was ratified by 181 countries across the world, and it entered into force in May 2008 [20]. Article 9 of CRPD calls on countries to “take appropriate measures to ensure to persons with disabilities access, on an equal basis with others, to the physical environment, to transportation, to information and communications, including information and communications technologies and systems” [21]. The direct reference to ensuring accessible information and communication technologies (ICTs) within CRPD suggests importance and priority, entailing that no one should be deprived from the access to the countless information resources the Web offers.

There are many laws and legislative measures for promoting the status of the rights of people with disabilities and mandating comparable access to information resources. For example, Section 508 of the American Rehabilitation Act emphasizes that individuals who are disabled must "have access to and use of information and data that is comparable to the access to and use of the information and data by such members of the public who are not individuals with disabilities" [22]. In Australia, it is mandatory that all online information and services be accessible according to the Australian Disability Discrimination Act (DDA) [23]. Although some still lag behind, several countries have similar policies that mandate information accessibility; for example, Austria, Belgium, Brazil, Canada, China, Denmark, Finland, France, Germany, Hong Kong, India, Ireland, Italy, Japan, Korea, Luxemburg, Netherlands, New Zealand, Norway, Portugal, Singapore, Spain, Sweden, Switzerland, Taiwan, Thailand and UK [24]. More details on the legislation enacted in these countries can be found in [24] and [25].

Nowadays, neglecting to provide accessible information to accommodate the needs of diverse users may expose service providers to different lawsuits and serious legal actions. The US witnessed a number of 262, 814, 2314, and 2235 lawsuits filed alleging website inaccessibility under the American with Disabilities Act (ADA) in the years 2016, 2017, 2018, and 2019, respectively [26].

Web accessibility standards

Web accessibility refers to developing websites in a manner that every user can use them, including people with disabilities [27]. This implies that any user should be able to perceive, understand, navigate, interact with, and contribute to the Web, with no access barriers to its content (i.e., text, images, audio, video, or multimedia) [13]. Hence, emerged accessibility guidelines. Accessibility guidelines (i.e., accessibility metrics) outline a set of technical requirements for making any digital content accessible for diverse users. These guidelines detail requirements for removing all access barriers that could prohibit users from accessing, perceiving, using, or interacting with Web content.

Among the most recognized Web accessibility standards worldwide is the Web Content Accessibility Guidelines (WCAG) established by The World Wide Web Consortium (W3C) [13]. Thus far, there are three releases of WCAG: WCAG 1.0 released in 1999, WCAG 2.0 released in 2008, and WCAG 2.1 released in 2018. In every release, WCAG expands in the features and coverage of technical standards, technology platforms, and the demographic groups it targets. This is due to the nature of technology and content; both are in a continual evolution.

The currently valid working standards are WCAG 2.0 and WCAG 2.1. Both standards outline accessibility requirements through a set of guidelines revolving around four fundamental principles: perceivable, operable, understandable, and robust (POUR) [27]. The POUR principles can be interpreted as: any Web content must be (1) totally perceivable using different senses and ways users select (i.e., Perceivable), (2) easy to operate with different user interaction modalities (i.e., Operable), (3) simple to understand for the widest possible audience (i.e., Understandable), and (4) robust for use under different platforms and using various assistive technologies (i.e., Robust).

In WCAG 2.0 and WCAG 2.1, there is a total of 12 and 13 guidelines, respectively. For each guideline, there are up to three conformance levels: Level A (i.e., the lowest), Level AA (i.e., intermediate), and Level AAA (i.e., the highest). Level A is the easiest to achieve and fulfill, as it represents the basic and minimum accessibility requirements that developers should aim at conforming to the levels beyond. In contrast, Level AAA is the most difficult to achieve as it demands a higher level of requirements that compliance with which is sometimes nontrivial to attain. In between comes Level AA; a website should conform to WCAG 2.0 Level AA in order to be accessible for all major groups of users with disabilities [26].

A guideline does not need to have all conformance levels; for example, some guidelines have Level A and Level AAA without Level AA. The conformance level is determined by some testable success criteria. WCAG 2.1 covers all requirements (i.e., 61 success criteria) in WCAG 2.0, with an additional set of 17 success criteria pertaining to mobile devices, visual impairments, and cognitive impairments. This means that any WCAG 2.1 compliant content is also compliant with WCAG 2.0 (i.e., backward compatibility).

Accessibility evaluation

There are two major directions for evaluating (i.e., testing) the extent to which a bpage conforms to standard Web accessibility guidelines: (1) automatic evaluation, and (2) manual evaluation [28]. While each direction has its own strengths and limitations, both directions can be combined for the optimization of the evaluation results.

Automatic evaluation

This type of evaluation is carried out using Web accessibility evaluation tools. These tools can be either standalone software programs or online services whose function is to inspect whether the content items on a webpage meet predefined accessibility requirements or not. There are many tools for evaluating Web accessibility; for example, WAVE, AChecker, TAW, Total Validator, APrompt, and EvalAccess [29, 30]. The choice of a particular tool is determined by its accuracy, exhaustion of the guidelines it covers, and the conformance levels it supports (e.g., Level A, Level AA, and Level AAA). It is highly encouraged to use multiple automatic Web accessibility evaluation tools to take advantage of the merits of each, substitute shortcomings of one another, and thus achieve more sound results [31].

Manual evaluation

From what the name suggests, this direction revolves around employing the human factor in order to evaluate conformance with Web accessibility standards. There are two types of manual evaluation:

Expert

In this approach, accessibility domain experts are recruited to inspect Web accessibility of given webpages. This requires manual inspection of each content item to whether it conforms to accessibility guidelines or not. This direction produces more detailed, precise, and exhaustive results than auto-evaluation tools, considering the usability aspect as well [32]. However, this approach takes time, and recruiting accessibility domain experts is not a cost-effective option [28].

User

In this approach, end users are recruited to test the accessibility of a webpage, not according to any guidelines, but according to their perspectives on how the webpage satisfies their access needs, and whether the content can be easily accessed or not. End users do not have to have any technical accessibility expertise [32]. Rather, they belong to demographic groups affected by the lack of Web accessibility, and thus they are familiar with actual user accessibility needs.

Related literature

Tackling the issue of accessibility to Web-based health information resources has interested many researchers. These resources, as argued before, play a significant role influencing citizens’ health decisions and awareness on how to prevent or control serious health cases. In this section, an overview is provided on a number of relevant studies that address Web accessibility problems in different health information resources.

Brobst [33] evaluated 20 US federal health care websites using both automatic and expert evaluation. The outcomes of the evaluation revealed that only one third of the websites provided an acceptable level of accessibility features, while the other two thirds failed to provide an adequate level of accessibility support. The major accessibility issues identified by Brobst [33] were the lack of alternative descriptions and the limited access to embedded interfaces.

Yi [34] conducted an automatic and expert accessibility evaluation on a number of Korean government health care websites. The study highlighted critical accessibility problems such as non-compliance with keyboard, color contrast ratios, inappropriate alternative descriptions, mismatches in the default language display and markup, content sequences, and bypass blocks.

Martins et al. [35] conducted an evaluation study targeting a number of Iberian health care institutions websites. The final set contained 697 websites evaluated using the ACCESSWEB tool against WCAG 2.0. The results revealed a large number of accessibility errors in the majority of the evaluated websites with an average of 460, 93, and 167 errors at WCAG 2.0 Levels A, AA, and AAA, respectively. The findings of the study revealed that none of the evaluated websites conformed with WCAG 2.0, and hence none was totally accessible. Moreover, the study found no difference between Spain and Portugal in terms of the number of errors found in the corresponding evaluated websites. The study suggested a need for a substantial work in order to overcome the critical accessibility problems detected, thereby, pushing the status of Iberian health care websites in terms of accessibility support.

Kuzma et al. [36] evaluated the websites of 160 hospitals belonging to 16 different countries across 4 continents. The evaluation discovered a large number of accessibility problems in the evaluated bsites. Of the 160 websites, only two were fully Level A compliant. Relatively, Asian hospital websites exhibited a higher degree of problems than the rest of websites. The study discovered a total of 10,832 problems in Level A, and 4,830 problems in Level AAA. Geographically, websites belonging to hospitals located in Asia had the highest number of problems in Level A, followed by the websites belonging to hospitals located in America, Africa, and the EU, respectively. For Level AAA, the highest number of problems was detected in the websites belonging to hospitals located in America, followed by the websites belonging to hospitals located in the EU, Asia, and Africa, respectively. The same study revealed a lack of compliance with the minimum Web accessibility requirements among the vast majority of the sampled hospital websites belonging to 16 different countries, calling for urgent actions to protect the users disadvantaged by the notable low accessibility adherence. The study also highlighted inadequacy in the efforts toward making hospital websites accessible, despite the existence of local legislation in many of the countries to which the evaluated websites belonged. This emphasizes the argument of legislation and standards alone do not guarantee accessibility in real contexts [37].

Kaur et al. [38] targeted 280 hospital websites in metropolitan cities of India for Web accessibility evaluation. The evaluation was automatic using the TAW tool alone. The evaluation results revealed 22,491 errors in total, with an average of 80.32 errors per website, an observed minimum of 3 errors, and an observed maximum of 840 errors. 31% of the sample websites had more than the average number of errors. The study asserts that total compliance with WCAG 2.0 continues to be significantly low among the evaluated hospital websites. The top 5 recurring errors were: Non-text content, info and relationships, link purpose (in context), language of a page not identified, and parsing. Of the 280 evaluated websites, only 28 (i.e., 10%) were fully screen-reader compatible.

Several relatively old studies (i.e., some of them prior the release of WCAG 2.0) tackled the same problem (i.e., accessibility to health care websites), such as [2, 39–42], and [43]. All of these studies revealed serious accessibility problems and very low adherence to accessibility legislation and standards; most evaluated health websites did not exhibit the acceptable compliance with WCAG.

Methods

Website selection

This study aimed at selecting a sample among public health websites in the world taking into account two considerations: (1) the sample should be as random as possible to cover geographically spread countries, and (2) the sample should reflect websites that are expected to have high traffic at the time of the analysis. Therefore, the choice was toward selecting the public health websites of the top 25 countries affected by the COVID-19 pandemic as of late April, 2020. The data of the outbreak ranking were obtained from [44] in late April, 2020. The evaluation was conducted on the local language version of every website. This roots from the desire to inspect to what extent the website is accessible to the people who use it. As one country (i.e., Canada) has two official languages, English and French, both versions of the website were involved in the analysis. Only the landing webpage of every website was used in the evaluation. This is because a landing page is deemed an entry point for any website [45, 46]. Thus, it gives an overview of how the entire website content is developed in terms of accessibility features and support. In addition, landing webpages are often the most important for the website visitors, from which navigation commences. During the entire analysis duration, which spanned nearly over an entire month, one country’s website (i.e., Iran) remained under maintenance. Thus, that website was excluded, leaving 25 website landing webpages in the study, belonging to 24 different countries. Table 1 shows the countries listed according to the COVID-19 outbreak rank, the public health authority in each country, the continent in which each country is located, and the population of each country as obtained from [47].

Table 1

Public Health Websites Targeted in The Study, Listed According to The Corresponding Country’s Rank in Terms of COVID-19 Outbreak as of Late April, 2020

	Country	Public Health Main Authority	Continent	Population
1	USA	Department of Health & Human Services	North America	332,6 M
2	Spain	Ministry of Health, Consumption and Social Welfare	Europe	50.0 M
3	Italy	Ministry of Health	Europe	62.4 M
4	UK	Department of Health and Social Care	Europe	65.8 M
5	France	Ministry of Solidarity and Health	Europe	67.8 M
6	Germany	Federal Ministry of Health	Europe	80.2 M
7	Turkey	Ministry of Health	Asia	82.0 M
8	Russia	Ministry of Healthcare	Asia/Europe	141.7 M
9	Brazil	Ministry of Health	South America	211.7 M
10	China	National Health Commission	Asia	1.4 B
11	Canada	Health Canada	North America	37.7 M
12	Belgium	Public Health, Food Chain Safety and Environment	Europe	11.7 M
13	Netherlands	Ministry of Health, Welfare and Sport	Europe	17.3 M
14	Peru	Ministry of Health	South America	31.9 M
15	India	Ministry of Health and Family Welfare	Asia	1.3 B
16	Switzerland	Federal Office of Public Health	Europe	8.4 M
17	Portugal	Directorate-General for Health	Europe	10.3 M
18	Ecuador	Ministry of Public Health	South America	16.9 M
19	Saudi Arabia	Ministry of Health	Asia	34.2 M
20	Sweden	Ministry of Health and Social Affairs	Europe	10.2 M
21	Ireland	Department of Health	Europe	5.2 M
22	Mexico	Government of Mexico, Health Secretary	North America	128.6 M
23	Singapore	Ministry of Health	Asia	6.2 M
24	Pakistan	Ministry of National Health Services Regulation and Coordination	Asia	233.5 M
			Total	4.36 B

Of all targeted countries, three are located in North America, three are located in South America, six are located in Asia, eleven are located in Europe, and one country (i.e., Russia) is located in both Europe and Asia. The total population of all countries whose public health websites are examined exceeded 4 billion. This is an important indicator of the expected number of disabled users of these websites. Moreover, the traffic on these websites was looked up using Amazon Alexa Web traffic analysis tool [48] in early May, 2020. The data revealed that among the top 10 searched keywords in every website, the median number of COVID-19-related terms of all websites was 6, with an average of 5.1. The median of the overall traffic of those keywords among the top 10 keywords was 58.81%, with an average of 54.79%. This suggests a high reliance on these websites with respect to accessing reliable information during unusual circumstances.

Evaluation tools

The evaluation was conducted using the following automatic accessibility testing tools: AChecker [14], WAVE [15], W3C HTML Validator [16], and W3C CSS Validator [17]. The use of multiple automatic tools is encouraged by Vigo et al. [31] as they highlighted some limitations and variances of automatic tools in terms of accuracy and comprehensiveness in detecting real accessibility issues when compared to manual evaluations. Some tools could substitute deficiencies of other tools, making the evaluation more reliable.

The AChecker tool is used to test conformance with WCAG 2.0 Levels A, AA and AAA. The results of this tool identify three categories of accessibility problems at each level: known, likely, and potential. Known problems are firmly considered as accessibility barriers that must be resolved. Likely problems are considered probable barriers but not as firmly as known problems, and thus a human intervention is needed to judge them. Potential problems are undetected by the tool and a human intervention is inevitably needed to judge them and confirm if they are actually present or not. In this study, the known problems reported by AChecker are manually examined to find out which POUR principle each known problem particularly violates, and which success criterion each known problem fails.

The WAVE tool is utilized to check conformance against WCAG 2.1 standard, Sect. 508 standard, and some additional requirements. The WAVE tool evaluation produces a report detailing errors, alerts, features, contrast errors, structural elements issues, and the use of HTML5 and ARIA within a webpage, according to WCAG 2.1, Sect. 508, and other commonly reported accessibility requirements in practice, not falling under any standard.

In this study, the W3C HTML Validator and the W3C CSS Validator are used to validate HTML and CSS codes. Errors in an HTML code would result in an unpredictable misbehavior of assistive technologies when dealing with the webpage, sometimes this leads to a full incompatibility and thus hindering access to the webpage. In contrast, a properly formed HTML code improves how the webpage is rendered across multiple platforms, and how it is handled by assistive technologies. CSS code is used in conjunction with HTML to adjust the appearance characteristics of the webpage elements, such as positions, colors, fonts, alignments, styling, and formatting. Any errors in the CSS code would harm the resulting style and layout of the webpage, preventing interface elements from rendering and functionality. Hence, valid HTML and CSS codes yield portability, cross-platform functionality, and better accessibility for users of assistive technologies. Therefore, it is crucial to ensure that a webpage follows proper technical standards with regards to the underlying HTML and CSS codes.

Evaluation procedure

A total of 25 landing webpages belonging to 24 different public health websites were evaluated using AChecker, WAVE, W3C HTML Validator, and W3C CSS Validator. Three levels of conformance testing (i.e., corresponding WCAG 2.0 Levels A, AA, and AAA) were conducted using AChecker. The WAVE tool was used to implicitly evaluate conformance with  Section 508 and WCAG 2.1 Levels A and AA. Markup validation was inspected using the W3C HTML Validator and the W3C CSS Validator. The testing was done using Google Chrome running under Microsoft Windows 10 (64 bit). The detected errors in AChecker and WAVE were then manually analyzed. The overall results from all tools were finally combined to rank the webpages according to the average number of detected issues in all tools.

Results

AChecker

In this section, the analysis results of the AChecker testing to the 25 webpages against WCAG 2.0 are presented. The known, likely, and potential errors at A, AA, and AAA conformance levels of WCAG 2.0 are shown in Table 2.

Table 2

AChecker Evaluation Results for All WCAG 2.0 Levels: A, AA and AAA

		Known			Likely			Potential
Country	Test Result	A	AA	AAA	A	AA	AAA	A	AA	AAA
USA	F	1	1	1	0	0	0	380	406	421
Spain	F	22	24	24	2	4	4	588	600	647
Italy	P	0	0	0	3	3	3	985	1034	1102
UK	P	0	0	0	0	1	1	390	442	447
France	F	28	133	109	7	7	7	1928	1968	1977
Germany	F	37	168	258	20	20	20	1587	1752	1757
Turkey	-	BG	BG	BG	BG	BG	BG	BG	BG	BG
Russia	F	7	42	42	0	1	1	1687	1705	1710
Brazil	F	0	30	50	1	1	1	426	447	453
China	P	0	0	0	0	0	0	0	0	0
Canada en	F	4	4	4	1	1	1	352	396	401
Canada fr	F	4	4	4	1	1	1	351	395	400
Belgium	F	12	12	12	0	0	0	389	432	441
Netherlands	F	0	1	1	0	0	0	105	129	134
Peru	F	2	36	36	4	4	4	374	431	436
India	F	7	43	549	0	0	0	958	968	988
Switzerland	F	6	6	6	0	0	0	470	529	541
Portugal	F	72	168	157	1	1	1	1862	1881	1894
Ecuador	F	3	28	28	1	1	1	465	483	489
Saudi Arabia	F	31	41	41	0	0	0	674	682	698
Sweden	F	5	52	20	0	0	0	695	717	725
Ireland	F	22	161	161	0	0	0	319	350	370
Mexico	F	1	1	1	0	0	0	124	137	143
Singapore	F	33	74	74	0	0	0	427	476	499
Pakistan	F	32	81	83	0	0	0	327	370	388
	Total	329	1110	1661	41	45	45	15,863	16,730	17,061
	Min	0	0	0	0	0	0	0	0	0
	Max	72	168	549	20	20	20	1928	1968	1977
	Average	13.7	46.3	69.2	1.7	1.9	1.9	660.9	697.1	710.9
	Median	5.5	29	26	0	0.5	0.5	426.5	461.5	471

BG bad gateway, P pass, F fail

As displayed in Table 2, only 3 webpages (i.e., from Italy, UK, and China) passed the AChecker test (i.e., obtaining zero errors at all WCAG 2.0 levels). Excluding the passing webpages, the rest of webpages failed with an average number of known errors of (15.7 at Level A), (52.9 at Level AA), and (79.1 at Level AAA). This shows that the number of known errors increases as the conformance level becomes higher. This implies that the overall conformance with WCAG 2.0 becomes worse as the level gets higher.

The maximum number of known errors detected by AChecker at WCAG 2.0 Levels A, AA, and AAA were 72, 168, and 549, respectively. At Level A, only 5 webpages (i.e., from Italy, UK, China, Netherlands, and Brazil) were completely compliant. At Level AA and Level AAA, only 3 webpages (i.e., belonging to Italy, UK, and China) were completely compliant.

The average number of known errors per webpage was 13.7 at the basic level (i.e., WCAG 2.0 Level A), 46.3 at the intermediate level (i.e., WCAG 2.0 Level AA), and 69.2 at the advanced level (i.e., WCAG 2.0 Level AAA). A mere look at the median and the average number of known errors (i.e., 29 and 46.3, respectively) at Level AA indicates serious accessibility problems among the evaluated webpages. Half of the webpages had more than 29 errors at Level AA and 26 errors at Level AAA. The number of webpages that had a number of errors exceeding the observed average number of known errors at each level was not minor. Among all webpages, 8 had more errors than the observed average number of known errors at Level A, and 7 webpages had more errors than the observed average number of known errors at both Level AA and Level AAA.

The detected known errors at Level A and Level AA were then manually examined. An overview and classification of the types of the known errors at Level A and Level AA according to the POUR principles are presented in Tables 3 and 4, respectively. Tables 3 and 4 show the webpages ranked according to the number of known errors in Level A and Level AA, respectively.

Table 3

The Types of The Known Level A Errors According to WCAG 2.0 POUR Principles per Webpage

Country	Website rank	P	O	U	R	Total
Italy	1	0	0	0	0	0
UK	1	0	0	0	0	0
Brazil	1	0	0	0	0	0
China	1	0	0	0	0	0
Netherlands	1	0	0	0	0	0
USA	2	0	1	0	0	1
Mexico	2	0	1	0	0	1
Peru	3	0	1	0	1	2
Ecuador	4	3	0	0	0	3
Canada en	5	4	0	0	0	4
Canada fr	5	4	0	0	0	4
Sweden	6	4	0	0	1	5
Switzerland	7	0	2	4	0	6
Russia	8	4	1	2	0	7
India	8	4	1	2	0	7
Belgium	9	12	0	0	0	12
Spain	10	22	0	0	0	22
Ireland	10	20	0	1	1	22
France	11	0	27	0	1	28
Saudi Arabia	12	24	2	5	0	31
Pakistan	13	11	5	16	0	32
Singapore	14	31	0	2	0	33
Germany	15	37	0	0	0	37
Portugal	16	11	53	7	1	72
Turkey	-	BG	BG	BG	BG	-
	Total	191	94	39	5	329

P perceivable, O operable, U understandable, R robust

BG bad gateway

Table 4

The Types of The Known Level AA Errors According to WCAG 2.0 POUR Principles per Webpage

Country	Website ranking	P	O	U	R	TOTAL
Italy	1	0	0	0	0	0
UK	1	0	0	0	0	0
China	1	0	0	0	0	0
USA	2	1	0	0	0	1
Netherlands	2	0	1	0	0	1
Mexico	2	0	1	0	0	1
Canada en	3	4	0	0	0	4
Canada fr	3	4	0	0	0	4
Switzerland	4	0	2	4	0	6
Belgium	5	12	0	0	0	12
Spain	6	24	0	0	0	24
Ecuador	7	28	0	0	0	28
Brazil	8	30	0	0	0	30
Peru	9	32	3	0	1	36
Saudi Arabia	10	34	2	5	0	41
Russia	11	39	1	2	0	42
India	11	39	2	2	0	43
Sweden	12	51	0	0	1	52
Singapore	13	71	1	2	0	74
Pakistan	14	58	7	16	0	81
France	15	81	51	0	1	133
Ireland	16	158	0	2	1	161
Portugal	17	98	54	14	2	168
Germany	18	168	0	0	0	168
Turkey	-	BG	BG	BG	BG	-
	Total	932	125	47	6	1110

P perceivable, O operable, U understandable, R robust

BG bad gateway

Further analysis of the known errors at Level A and Level AA is presented in Tables 5, 6, respectively. In each table, the known errors are classified according to the POUR principles and the relevant failed success criteria.

Table 5

A List of All Known Problems Detected by AChecker at Level A, Sorted from The Most Recurring to The Least

Category	Guideline	Success criteria	Websites affected	Error frequency	Error percentage %
Perceivable	1.1	Img element missing alt attribute Success Criteria 1.1.1 Non-text Content	9	85	25.84
Operable	2.4	Anchor contains no text Success Criteria 2.4.4 Link Purpose (In Context)	6	81	24.62
Perceivable	1.1	Image used as anchor is missing valid Alt text Success Criteria 1.1.1 Non-text Content	7	76	23.10
Understandable	3.3	Label text is empty Success Criteria 3.3.2 Labels or Instructions	5	18	5.47
Perceivable	1.3	input element, type of "text", missing an associated label Success Criteria 1.3.1 Info and Relationships	5	11	3.34
Perceivable	1.3	Input element, type of "text", has no text in label Success Criteria 1.3.1 Info and Relationships	5	11	3.34
Understandable	3.1	Document language not identified Success Criteria 3.1.1 Language of Page	4	10	3.04
Understandable	3.1	Document has invalid language code Success Criteria 3.1.1 Language of Page	4	10	3.04
Robust	4.1	Id attribute is not unique Success Criteria 4.1.1 Parsing	4	5	1.52
Operable	2.4	title element is empty Success Criteria 2.4.2 Page Titled	1	5	1.52
Operable	2.1	onmousedown event missing onkeydown event Success Criteria 2.1.1 Keyboard	0	3	0.91
Perceivable	1.1	input element has alt attribute Success Criteria 1.1.1 Non-text Content	1	2	0.61
Operable	2.4	Document missing title element Success Criteria 2.4.2 Page Titled	1	2	0.61
Perceivable	1.3	textarea element missing an associated label Success Criteria 1.3.1 Info and Relationships	2	2	0.61
Perceivable	1.1	embed element missing noembed element Success Criteria 1.1.1 Non-text Content	1	1	0.30
Perceivable	1.3	Input element, type of "password", missing an associated label Success Criteria 1.3.1 Info and Relationships	1	1	0.30
Perceivable	1.3	Input element, type of "checkbox", missing an associated label Success Criteria 1.3.1 Info and Relationships	1	1	0.30
Perceivable	1.3	Input element, type of "checkbox", has no text in label Success Criteria 1.3.1 Info and Relationships	1	1	0.30
Perceivable	1.3	Input element, type of "password", has no text in label Success Criteria 1.3.1 Info and Relationships	1	1	0.30
Operable	2.1	Script not keyboard accessible—onmouseout missing onblur Success Criteria 2.1.1 Keyboard	1	1	0.30
Operable	2.1	Onmouseover event handler missing onfocus event handler Success Criteria 2.1.1 Keyboard	1	1	0.30
Understandable	3.1	Right to left reading order not marked or marked incorrectly Success Criteria 3.1.1 Language of Page	1	1	0.30
			Total	329	100

Table 6

A List of All Known Problems Detected by AChecker at Level AA, Sorted from The Most Recurring to The Least

Category	Guideline	Success criteria	Websites affected	Error frequency	Error percentage %
Perceivable	1.4	i (italic) element used Success Criteria 1.4.4 Resize text (AA)	13	415	37.39
Perceivable	1.1	Img element missing alt attribute Success Criteria 1.1.1 Non-text Content	9	253	22.79
Operable	2.4	Anchor contains no text Success Criteria 2.4.4 Link Purpose (In Context)	6	108	9.73
Perceivable	1.4	The contrast between the color of selected link text and its background is not sufficient to meet WCAG2.0 Level AA Success Criteria 1.4.3 Contrast (Minimum) (AA)	2	90	8.11
Perceivable	1.1	Image used as anchor is missing valid Alt text Success Criteria 1.1.1 Non-text Content	7	76	6.85
Perceivable	1.4	The contrast between the color of text and its background for the element is not sufficient to meet WCAG2.0 Level AA Success Criteria 1.4.3 Contrast (Minimum) (AA)	4	61	5.50
Understandable	3.3	Label text is empty Success Criteria 3.3.2 Labels or Instructions	5	26	2.34
Perceivable	1.3	Input element, type of "text", missing an associated label Success Criteria 1.3.1 Info and Relationships	5	14	1.26
Perceivable	1.3	Input element, type of "text", has no text in label Success Criteria 1.3.1 Info and Relationships	5	11	0.99
Understandable	3.1	Document language not identified Success Criteria 3.1.1 Language of Page	4	10	0.90
Understandable	3.1	Document has invalid language code Success Criteria 3.1.1 Language of Page	4	10	0.90
Robust	4.1	Id attribute is not unique Success Criteria 4.1.1 Parsing	4	6	0.54
Operable	2.4	title element is empty Success Criteria 2.4.2 Page Titled	1	5	0.45
Operable	2.4	Header nesting—header following h2 is incorrect Success Criteria 2.4.6 Headings and Labels (AA)	3	4	0.36
Operable	2.1	onmousedown event missing onkeydown event Success Criteria 2.1.1 Keyboard	0	3	0.27
Perceivable	1.3	input element, type of "checkbox", has no text in label Success Criteria 1.3.1 Info and Relationships	1	3	0.27
Perceivable	1.3	Textarea element missing an associated label Success Criteria 1.3.1 Info and Relationships	2	3	0.27
Perceivable	1.1	input element has alt attribute Success Criteria 1.1.1 Non-text Content	1	2	0.18
Operable	2.4	Document missing title element Success Criteria 2.4.2 Page Titled	1	2	0.18
Operable	2.4	Header nesting—header following h1 is incorrect Success Criteria 2.4.6 Headings and Labels (AA)	1	1	0.09
Perceivable	1.1	Embed element missing noembed element Success Criteria 1.1.1 Non-text Content	1	1	0.09
Perceivable	1.3	Input element, type of "password", missing an associated label Success Criteria 1.3.1 Info and Relationships	1	1	0.09
Perceivable	1.3	Input element, type of "checkbox", missing an associated label Success Criteria 1.3.1 Info and Relationships	1	1	0.09
Perceivable	1.3	Input element, type of "password", has no text in label Success Criteria 1.3.1 Info and Relationships	1	1	0.09
Operable	2.1	Script not keyboard accessible—onmouseout missing onblur Success Criteria 2.1.1 Keyboard	1	1	0.09
Operable	2.1	Onmouseover event handler missing onfocus event handler Success Criteria 2.1.1 Keyboard	1	1	0.09
Understandable	3.1	Right to left reading order not marked or marked incorrectly Success Criteria 3.1.1 Language of Page	1	1	0.09
			Total	1110	100

The results in Table 3 show that the category with the maximum number of known errors at Level A was perceivable; nearly 58% of the errors fell under this category, followed by operable under which nearly 29% of the known errors fell, leaving 12% of the known errors to understandable and 1% to robust. This suggests that the majority of problems are critical and relevant to how the content is rendered, posing serious barriers.

The same pattern of error types is found again in Level AA as shown in Table 4. The results show that nearly 84% of the known errors fell under the perceivable category, 11% of the known errors fell under the operable category, leaving 4% of the known errors to the understandable category and 1% to the robust category. Again, this emphasizes that the majority of problems are critical, posing serious barriers against access to public health information.

The known errors detected at Level A are further analyzed in Table 5 in terms of the POUR principles, the failed success criteria, the popularity of the error, and the number of affected webpages. As shown in Table 5, among the top 5 popular errors, 3 fell under the perceivable category. The top 5 popular errors were: “img element missing alt attribute,” “Anchor contains no text,” “Image used as anchor is missing valid Alt text,” “Label text is empty,” and “input element, type of text, missing an associated label.” All these relevant errors indicate the lack of descriptive alternative text for many of the webpage items, a very common mistake developers are likely to make. This particular problem affects users who are blind or visually impaired, as lacking alternative descriptive text hinders perceiving the content.

The known errors detected at Level AA are further analyzed in Table 6 in terms of the POUR principles, the failed success criteria, the popularity of the error, and the number of affected webpages. Among the top 5 errors, 4 fell under the perceivable category. The top 5 errors were: “i (italic) element used,” “img element missing alt attribute,” “Anchor contains no text,” “The contrast between the color of selected link text and its background is not sufficient to meet WCAG2.0 Level AA,” and “Image used as anchor is missing valid Alt text.” The most recurring error detected at Level AA suggests that fonts should be selected with care and accessibility in mind, as italicized fonts can be hard to read for many users, not only those who are visually impaired. Thus, rendering textual information should comply with WCAG 2.0 requirements. Again, the top 5 recurring errors suggest the need to ensure providing descriptive alternative text for all items on a webpage. In addition, color contrast requirements ought to be followed as recommended by the WCAG 2.0 standard. For standard text, 4.5:1 contrast ratio at minimum is acceptable, and a contrast ratio of 3:1 at least is acceptable for large text.

During the analysis conducted using AChecker, one bad gateway occurrence was encountered for the webpage belonging to Turkey, and thus the details of the corresponding webpage are left blank in all tables. The two webpages belonging to Health Canada exhibited identical results, indicating that the developers of these webpages maintain one template for both versions of the website (i.e., English and French).

WAVE and W3C Markup validators

This section provides the results of the evaluation conducted using the WAVE tool, the W3C HTML Validator, and the W3C CSS Validator. There are six accessibility categories that the WAVE tool evaluates: (1) Errors, (2) Alerts, (3) Structural elements, (4) Contrast, (5) Features, and (6) HTML5 and ARIA.

Table 7 shows the results obtained by the WAVE tool, identified by the six aforementioned categories. One bad gateway occurrence belonging to Russia was encountered while analyzing the webpages. As seen in Table 7, only 4 webpages out of the 24 evaluated webpages (i.e., nearly 17%) passed the test with no errors at all. These webpages belong to USA, Italy, UK, and the Netherlands. The total number of detected errors among all webpages was 456, with an average of 19 errors, a median of 12 errors, and a maximum number of 136 errors.

Table 7

The Results of WAVE Tool, W3C HTML and CSS Validators

	WAVE Tool					W3C Validators
Country	Error	Alerts	Structural Elements	Contrast	Features	HTML5 & Aria	HTML Val	CSS Val
USA	0	7	35	1	46	22	19	32
Spain	24	49	72	7	116	99	0	5
Italy	0	301	57	36	89	5	1	0
UK	0	8	83	0	20	43	7	15
France	28	33	307	2	23	188	210	25
Germany	14	36	236	5	145	176	16	7
Turkey	18	16	36	4	10	10	8	0
Russia	BG	BG	BG	BG	BG	BG	2	64
Brazil	7	29	56	1	8	0	16	65
China	41	100	61	65	7	0	Cannot be checked	Precondition Failed
Canada en	3	6	95	1	26	927	0	22
Canada fr	3	6	95	1	22	925	0	22
Belgium	13	7	74	3	14	53	39	12
Netherlands	0	3	30	2	8	0	0	140
Peru	24	37	77	1	7	103	145	5
India	9	254	40	12	28	26	5	2
Switzerland	18	139	389	0	27	172	29	14
Portugal	136	262	71	36	58	14	41	24
Ecuador	4	35	42	0	10	25	23	24
Saudi Arabia	11	117	45	63	129	308	11	5
Sweden	5	14	43	4	101	31	22	4
Ireland	22	3	37	0	0	7	48	4
Mexico	3	10	58	0	20	2	9	25
Singapore	38	33	47	13	3	121	51	9
Pakistan	35	42	44	11	13	17	75	40
Total	456	1547	2130	268	930	3274	777	565

BG bad gateway

Of the 24 webpages, 8 webpages (i.e., one third of the webpages) had more errors than the observed average number of errors (i.e., 19). Half of the webpages had more than 12 errors in total. Relatively, this is not a promising result as it suggests that the number of detected errors in 50% of the evaluated webpages is not minor.

To get more insights, Table 8 provides an analysis of the errors detected by the WAVE tool. The errors are classified according to the WCAG POUR categories and the failed success criteria, then ranked according to their frequency. Another column shows the number of webpages affected by each error.

Table 8

The WAVE Tool Error Analysis

Error Type	Category	Guideline	Success Criteria	Websites affected	Frequency	Error %
Empty link	Operable	2.4	2.4.4 Link Purpose	12	195	42.76
Missing alternative text	Perceivable	1.1	1.1.1 Non-text Content	12	79	17.32
Linked image missing alternative text	Perceivable Operable	1.1 2.4	1.1.1 Non-text Content 2.4.4 Link Purpose	9	65	14.25
Broken ARIA reference	Perceivable Robust	1.3 4.1	1.3.1 Info and Relationships 4.1.2 Name, Role, Value	4	37	8.11
Empty button	Perceivable Operable	1.1 2.4	1.1.1 Non-text Content 2.4.4 Link Purpose	10	26	5.70
Missing form label	Perceivable	1.3	1.3.1 Info and Relationships	8	21	4.61
Broken ARIA menu	Robust	4.1	4.1.2 Name, Role, Value	2	18	3.95
Document language missing	Understandable	3.1	3.1.1 Language of Page	5	5	1.10
Empty heading	Perceivable	1.3	1.3.1 Info and Relationships	3	5	1.10
Marquee	Operable	2.2	2.2.2 Pause, Stop, Hide	1	2	0.44
Image map area missing alternative text	Perceivable Operable	1.1 2.4	1.1.1 Non-text Content 2.4.4 Link Purpose	1	1	0.22
Image map missing alternative text	Perceivable	1.1	1.1.1 Non-text Content	1	1	0.22
Missing or uninformative page title	Operable	2.4	2.4.2 Page Titled	1	1	0.22
				Total	456	100

Many errors were mapped to two different POUR categories. The operable type errors were the most popular among the rest. The errors were classified as follows: 290 errors under the operable category, 235 errors under the perceivable category, 55 errors under the robust category, and 5 errors under the understandable category.

As shown in Table 8, the top 5 most popular errors constituting over 88% of the overall errors were: “Empty link,” “Missing alternative text,” “Linked image missing alternative text,” “Broken ARIA reference,” and “Empty button.” Most of these problems are of the perceivable and operable types.

The most recurring (i.e., popular) error was “Empty link” (i.e., the given link does not contain any text), constituting nearly 42.8% of the overall errors with a total of 12 webpages (i.e., 50% of the webpages) affected by it. These findings suggest that this error is dominant and very likely to occur. The second most popular error was “Missing alternative text” (i.e., the non-text item does not have an alternative descriptive text), constituting nearly 17.3% of the overall errors with a total of 12 webpages (i.e., 50% of the webpages) affected by it. The third most popular error was “Linked image missing alternative text” (i.e., the image does not have a descriptive text of its content), constituting nearly 14.3% of the overall errors with a total of 9 webpages (i.e., 37.5% of the webpages) affected by it. The fourth most popular error was “Broken ARIA reference” (i.e., an ARIA item that points to a target that does not exit), constituting nearly 8.1% of the overall errors with a total of 4 webpages affected by it. The fifth most popular error was “Empty button” (i.e., the button does not have any associated descriptive text), constituting nearly 5.7% of the overall errors with a total of 10 webpages (i.e., 42% of the webpages) affected by it.

Of the top 5 recurring errors, 4 errors relate to content items missing alternative and descriptive text. This is an obvious indicator that developers usually do not give much attention to providing alternative text descriptions for the interface items such as buttons, links, and images. For people who are blind or visually impaired, this poses serious accessibility barriers.

With regards to color contrast, the total number of detected problems was 268, with an average of 11 problems, a median of 2.5, and a maximum of 65. The results in Table 7 show that only 4 webpages did not exhibit any problems of this type; the webpages belong to UK, Switzerland, Mexico, and Ireland. The webpage belonging to China had the maximum number (i.e., 65) of color contrast problems. Generally speaking, since the median value was 2.5 (i.e., half of the webpages have none to 2 contrast issues), the results are promising in terms of contrast problems.

The WAVE tool also highlights the use of HTML5 and ARIA in the tested webpage. Both HTML5 and ARIA are used to develop robust websites providing accessible features and support for people who use assistive technologies. As shown in Table 7, the vast majority (i.e., 87.5%) of webpages employ HTML5 and ARIA elements. Only 3 webpages, belonging to Brazil, China, and Netherlands, did not use HTML5 and ARIA.

For HTML and CSS validation, the results are shown in the last two columns in Table 7. In total, HTML parsing errors were found to be 777, while CSS parsing errors were 565. Only 4 webpages passed the HTML Validator test with no parsing errors. These webpages belong to: Spain, Netherlands, and Canada (i.e., the English and French versions). Interestingly, none of these webpages passed the CSS Validator test. The only two webpages that passed the CSS Validator test were: Italy and Turkey.

Since both HTML and CSS strongly relate to each other, a total value combing the number of parsing errors detected in both tools (i.e., W3C HTML Validator and W3C CSS Validator) is calculated for each webpage. The webpages are then ranked according to this score as shown in Fig. 1.

Fig. 1

The list of webpages ranked according to the total number of Markup parsing errors in both tools: W3C HTML Validator and W3C CSS Validator

The total number of parsing errors detected by both tools was 1342, with an average of 56, a median of 45, a minimum of 1, and a maximum of 235. At the top of the rank Italy comes first with 1 error, Spain comes second with 5 errors, and India comes third with 7 errors, while at the end of the list comes France with 235 errors. A total of 9 webpages (i.e., over a third) had more errors than the average number of total errors (i.e., 56) in both tools. Again, this is not promising as both the average number of parsing errors and the number of webpages that had more errors than that average are not minor.

During the test, the webpage belonging to China could not be checked in either tools for a precondition failure problem as indicated by the CSS Validator, and an unknown problem as indicated by the HTML Validator. The results found for both versions of Health Canada webpage were similar, indicating a unified template use in both.

Discussion

The public health websites evaluated in this study were selected based on the worldwide rank of the COVID-19 outbreak in the corresponding countries. The websites of the top 25 mostly affected countries as of late April, 2020 were chosen. The purpose of the evaluation was to shed light on the situation of supporting Web accessibility in public health websites belonging to different countries, and thus to pinpoint the greatest concerns impacting Web accessibility to this vital segment of online information resources.

The results obtained from the analysis conducted using the AChecker tool revealed that the performance, in terms of compliance to WCAG 2.0 guidelines, varied among the evaluated websites. While few webpages exhibited full compliance with all WCAG 2.0 Levels (i.e., A, AA, and AAA), many lagged behind, with two webpages getting extreme results.

In particular, the results showed that only 3 webpages passed the AChecker test with no errors detected at all WCAG 2.0 levels. This means that the vast majority of webpages have accessibility problems at different degrees. This is also evident in that (1) only 5 webpages had no errors at WCAG 2.0 Level A, which is deemed as the very minimum conformance level that any website should meet, and (2) only 3 webpages had no errors at WCAG 2.0 Level AA, which is the conformance level needed by any website to be regarded as accessible for the majority of users.

When the detected errors were further analyzed, the results showed that all POUR principles were violated at different scales; the most popular type of errors in Level A and Level AA fell in the perceivable category, followed by the operable, understandable, and robust categories, respectively. As most recurring errors were of the perceivable type, which entails that the information is not presented in suitable ways that the user can perceive, this suggests that there are critical barriers to accessing the information in many public health websites across the world.

Interestingly, no single error was encountered by all webpages at any level. At WCAG 2.0 Level AA, the error encountered the most (i.e., by 13 webpages) was “i (italic) element used” failing the success criterion 1.4.4 Resize text, with 415 occurrences, constituting nearly 37% of the overall errors detected at Level AA. This error, which is of the perceivable type, was the most frequent error as well. Users who are blind or visually impaired are the most affected by this error.

The second most common error at WCAG 2.0 Level AA, in terms of popularity and the number of affected webpages, was “img element missing alt attribute” failing the success criterion 1.1.1 non-text content, with 253 occurrences in 9 different webpages, constituting nearly 23% of the overall errors detected at Level AA. This error falls under the perceivable category and affects users who are blind and visually impaired. Remarkably, both of these errors affect users who are blind or visually impaired, suggesting that this demographic group is the mostly disadvantaged.

At WCAG 2.0 Level A, the most popular error was “img element missing alt attribute” failing the success criterion 1.1.1 non-text content, with 85 occurrences in 9 different webpages, constituting nearly 26% of the overall errors detected at Level A. This error falls under the perceivable category and affects users who are blind and visually impaired.

The second most popular error at WCAG 2.0 Level A was “Anchor contains no text” failing the success criterion 2.4.4 Link Purpose (In Context), with 81 occurrences in 6 different webpages, constituting nearly 25% of the overall errors detected at Level A. This error, which falls under the operable category, means that the given link has no anchor text to identify its destination.

As can be seen from the obtained results, most recurring errors at both Level A and Level AA were of the perceivable type; this entails that the information is not presented in suitable ways for all users to perceive. These findings suggest that there are critical barriers to accessing the information in many public health websites.

According to the analysis results obtained using the WAVE tool, the performance varied among the evaluated websites. While few webpages exhibited no errors, many lagged behind, with one webpage at least getting relatively extreme results. This, to a large extent, repeats the findings obtained using AChecker tool.

The WAVE tool detected a total of 456 errors, with an average of 19 errors per webpage, a median of 12 errors, and a maximum number of 136 errors. In a close outcome to AChecker, only 4 webpages out of the 24 evaluated webpages (i.e., nearly 17%) passed the test with no errors at all. 2 of these webpages (i.e., Italy and UK) passed the AChecker test already. 8 webpages (i.e., one third of the webpages) had more errors than the observed average number of errors (i.e., 19). This is not a promising result as contrast errors were not even taken into consideration. If contrast errors were taken into account, no single webpage would pass the test (i.e., the minimum detected number of contrast errors was 1).

The analysis of the detected errors revealed that the most popular errors fell under the operable and perceivable categories. This is also obvious in the top 5 recurring errors accounting for 88% of the overall detected errors, most of which are of the perceivable and operable types. Again, this suggests critical accessibility barriers and calls on for more efforts toward facilitating equal access to public health websites.

In total, 4 out of the top 5 recurring errors relate to missing alternative and descriptive text. This particular finding confirms the findings obtained previously using AChecker. Obviously, this error indicates that developers usually do not give much attention to providing alternative text descriptions for the interface items such as buttons, links, and images. For people who are blind or visually impaired, such content poses serious accessibility barriers. The high number of contrast problems, along with other problems, suggest that awareness of accessibility requirements or adhering to those requirements continue to be inadequate.

A closer look at the errors showed that the most common error in the operable category was “Empty link” constituting around 43% of the total number of detected errors and affecting 12 webpages, while “Missing alternative text” was the most common error in the perceivable category, constituting around 17% of the total number of detected errors and affecting 12 webpages.

The markup validation revealed many parsing problems in the underlying HTML and CSS codes of the evaluated webpages. This implies some incompatibility issues with assistive technologies. No single webpage passed both tools test. Only 3 webpages passed the HTML validation test, while only 2 webpages passed the CSS validation test. However, a general look considering the average number of parsing errors in both tools showed that the results are more promising than other aspects of accessibility detected by the previous testing tools.

In general, the results obtained from various tools prove that Web accessibility requirements are still being overlooked by many developers around the world, as seen the Table 9 and Table 10.

Table 9

Summary of The Results Obtained from The Used Tools: AChecker, WAVE, HTML Validator, and CSS Validator

Tool	Total Errors	Max	Min	Average	Median	Pass
AChecker WCAG 2.0, Level A	329	72	0	13.7	5.5	Italy, UK, China, Brazil, Netherlands
AChecker WCAG 2.0, Level AA	1110	168	0	46.3	29	Italy, UK, China
AChecker WCAG 2.0, Level AAA	1661	549	0	69.2	26	Italy, UK, China
WAVE	456	136	0	19	12	Italy, UK, USA, Netherlands
W3C HTML Validator	777	210	0	32.4	16	Spain, Canada, Netherlands
W3C CSS Validator	565	140	0	23.5	14.5	Italy, Turkey

Table 10

The Final Rank of The Evaluated Websites According to The Number of Detected Problems via AChecker at WCAG 2.0 Level AA, WAVE, and W3C Markup Validators

Country	Rank	AChecker AA Errors	WAVE Errors	HTML Validators	CSS Validators	Total	Average
Italy	1	0	0	1	0	1	0.25
UK	2	0	0	7	15	22	5.5
Canada en	3	4	3	0	22	29	7.25
Canada fr	3	4	3	0	22	29	7.25
Mexico	4	1	3	9	25	38	9.5
USA	5	1	0	19	32	52	13
Spain	6	24	24	0	5	53	13.25
India	7	43	9	5	2	59	14.75
Switzerland	8	6	18	29	14	67	16.75
Saudi Arabia	9	41	11	11	5	68	17
Belgium	10	12	13	39	12	76	19
Ecuador	11	28	4	23	24	79	19.75
Sweden	12	52	5	22	4	83	20.75
Brazil	13	30	7	16	65	118	29.5
Netherlands	14	1	0	0	140	141	35.25
Singapore	15	74	38	51	9	172	43
Germany	16	168	14	16	7	205	51.25
Peru	17	36	24	145	5	210	52.5
Pakistan	18	81	35	75	40	231	57.75
Ireland	19	161	22	48	4	235	58.75
Portugal	20	168	136	41	24	369	92.25
France	21	133	28	210	25	396	99
Turkey	X	BG	18	8	0	X	X
China	X	0	41	X	X	X	X
Russia	X	42	X	2	64	X	X
					Average	124.23	31.06

X missing partial results due to certain problems in the corresponding public health webpage

BG bad gateway

After combining the most significant results obtained from all tools, only one website was close to fully conform to accessibility standards, with less than an average of 1 error in the selected categories of the used tools, as seen in Table 10. In Table 10, only significant categories were taken into account: AChecker Level AA, WAVE Errors, W3C HTML Validator, and W3C CSS Validator. This is rooted in that WCAG 2.0 Level AA being the desired conformance level that any website should comply with as to satisfy the needs of the majority of diverse users. WAVE errors are significant since they are firmly considered critical. The W3C Validators are also critical to inspect the compatibility with assistive technologies. The combined results ascertain the low adherence to Web accessibility guidelines in the vast majority of the evaluated public health websites.

When it comes to continents, we see that the final list combining results from all tools includes 11 countries from Europe (i.e., Italy, UK, Spain, Switzerland, Belgium, Sweden, Netherlands, Germany, Ireland, Portugal, and France), 4 countries from Asia (i.e., India, Saudi Arabia, Singapore, and Pakistan), 3 countries from North America (i.e., Canada, Mexico, and USA), and 3 countries from South America (i.e., Ecuador, Brazil, and Peru). A summary of the results with respect to continents is shown in Table 11.

Table 11

Summary of The Combined Results per Continent

Continent	Average	Median	Max	Min	Top
Europe	37.45	20.75	99	0.25	Italy
North America	9.91	9.5	13	7.25	Canada
South America	33.92	29.5	52.5	19.75	Ecuador
Asia	33.13	30	57.75	14.75	India

According to Table 11, the continent with the lowest average of combined errors was North America, followed by Asia, South America, and Europe, respectively. All webpages belonging to countries in North America were ranked among the top 5 in the list: 3, 4, and 5. The obtained results of these webpages were close, with superiority to Canada with 7.25 average number of combined results.

Interestingly, while the top 2 webpages in the entire list belong to European countries, Europe was ranked the lowest (i.e., last) with an average number of combined results of 37.45. This is explained by the variety of results belonging to European webpages, as many of which got unfavorable results as shown in Table 10. The worst 3 results in the overall list belonged to European webpages, as shown in Table 10.

Second in rank comes Asia with 33.13 average number of combined results. None of the Asian webpages was among the top 5 in the overall list. The best among these webpages was the one belonging to India, with 14.75 average number of combined results.

The third-ranked continent was South America with 33.92 average number of combined results. None of the South American webpages was among the top 5 in the overall list. The best among these webpages was the one belonging to Ecuador, with 19.75 average number of combined results.

Finally, the findings of this study highlight the following: (1) there is an evident lack of conformance to Web accessibility guidelines in the vast majority of evaluated public health websites; (2) accessibility standards are often overlooked even in public websites across many countries in the world; (3) there is an urgent need to educate Web developers of the existing accessibility standards and legislation; (4) it is time accessibility, inclusive design, and awareness of the actual needs of diverse users are taught in high school and higher education curricula in order to establish a profound knowledge pertaining accessible information resources; (5) governments should ensure their official information portals are accessible to set examples for other sectors; and (6) legislation along with continual assessments of information resources are both essential to maintain sufficient level of Web accessibility.

Conclusion

The increasing number of people who are disabled, along with the many legislative measures for protecting human rights and the existence of well-defined international accessibility standards make it obligatory for any website to comply with Web accessibility guidelines. This study sought to evaluate the extent to which public health websites in different countries conform to WCAG 2.0, pinpoint the aspects where Web accessibility in public health websites needs to be considerably thought of, and ultimately raise awareness of and advocate the right of diverse users to have an equal access to Web content.

The findings of this study show that the vast majority of the evaluated public health websites did not conform to WCAG 2.0 Level AA, meaning that many users are in a severe disadvantage and suggesting an urgent need for major efforts to fix access barriers for the affected demographic groups. When combining the results of all used tools, only one website had an average number of accessibility problems less than 5.

Although all WCAG POUR principles were violated in the majority of evaluated websites, many of the detected accessibility downfalls pertained dominantly to the perception of information and the operability of the interface items. This indicates serious barriers depriving a wide range of users with disabilities from accessing critical health resources. A closer look at the detected errors revealed violations to many of the basic accessibility conformance level (i.e., WCAG 2.0 Level A). More specifically, some of the top recurring errors were relevant to the lack of alternative descriptive text of the interface items such as links and images. In addition, most websites exhibited potential incompatibility issues with assistive technologies.

These findings suggest that many Web developers are probably not aware of the details of accessibility guidelines, or that these guidelines and corresponding mandating legislations are often overlooked while building websites. In either case, awareness, education, and training of Web accessibility requirements for diverse users, along with law enforcement should be urgently fulfilled across the world.