Accessibility of university websites worldwide: a systematic literature review.

The identity and institutional image of universities are presented to the world through their websites. On their websites, universities publish their academic offerings, their mission, their vision, their academic objectives, their achievements, their regulations, their news and all their university work. Hence, the importance of university websites is accessible. The accessibility of university websites has been evaluated several times in the past, but there is no work that has summarized all the evaluations performed to provide a general overview of the situation. Therefore, in this research we have performed a systematic literature review (SLR) to consolidate, analyze, synthesize and interpret the accessibility results of university websites published in 42 papers that have been selected for this study. The methodology used in this SLR was that proposed in Kitchenham's guidelines, which includes three stages: planning the review, conducting the review and reporting the review. The results present the analysis and synthesis of the evaluations of 9,140 universities in 67 countries. Of these, 38,416 web pages, 91,421 YouTube videos and 28,395 PDF documents were evaluated. Manual methods, methods with automatic tools and the combination of both methods were used for the evaluation. Most websites were evaluated using the ISO/IEC 40500:2012 and Section 508 standards. The accessibility guidelines most commonly violated in the evaluations were: adaptable, compatible, distinguishable, input assistance, keyboard accessible, navigable, predictable, readable and text alternatives. In conclusion, the university websites, YouTube videos and PDF documents analyzed in the 42 papers present important accessibility problems. The main contribution of this SLR is the consolidation of the results of the 42 studies selected to determine the findings and trends in the accessibility of university websites around the world.

Introduction

The SARS-COV-2 pandemic has changed and revolutionized the world, making human beings interdependent on technological equipment whether for work, education, health, acquisition of essential supplies, among others. Also, large-scale national efforts to implement technology in all the processes that humans perform are emerging and evolving rapidly. In addition, electronic equipment, the Internet and the Web play an important role in this process of change. People with disabilities are no strangers to this transformation, but in many cases they must face new barriers rather than experience benefits.

In the Convention on the Rights of Persons with Disabilities (CRPD) of the United Nations [1], access to information and communication, including the Web, is defined as a basic human right. In the Article 21—Freedom of expression and opinion, and access to information, it is stated that governments should urge “private entities that provide services to the general public, including through the Internet, to provide information and services in accessible and usable formats for persons with disabilities” [1].

Tim Berners-Lee, inventor of the World Wide Web and Director of the World Wide Web Consortium (W3C) [2], states that “The power of the Web is in its universality. Access by everyone regardless of disability is an essential aspect” [3]. The W3C’s Web Accessibility Initiative (WAI) stipulates that “it is essential that the Web be accessible to provide equitable and equal access to people with diverse capabilities and not exclude people from using its products and services” [4]. The W3C has created the Web Content Accessibility Guidelines (WCAG) to make websites, electronic documents, PDFs, videos and other resources accessible to people with and without disabilities. It should be noted that WCAG 2.0 was declared as the international standard ISO/IEC 40500:2012 [5, 6]. Therefore, web accessibility applied to university websites will enable people with disabilities to use the Web, including people with “blindness or low vision, deafness or hearing loss, movement limitations, speech disabilities, photosensitivity and combinations thereof, and some adaptations for learning disabilities and cognitive limitations” [7].

In the 2011 World Disability Report, the World Health Organization (WHO) considers that “more than a billion people are estimated to live with some form of disability, or about 15 % of the world’s population (based on 2010 global population estimates). This is higher than previous WHO estimates, which date from the 1970s and suggested around 10 %” [8, pp. 7]. Worldwide, the increase of people with disabilities is notorious according to the statistics of WHO, which allows to affirm that millions of people with disabilities are studying at universities right now. The admission of people with and without disabilities in regular classrooms generates obligations that university institutions must comply with, for example, the accessibility of their web platforms and their educational resources.

Article 24 Education of the CRPD [1] requires the inclusion and participation of all persons in the educational environment. Education is an acquired right that every human being has. Schools, colleges, institutes and universities have been forced to close their facilities, adapting them to online education due to the SARS-COV-2 pandemic [9]. This change has brought with it many barriers that can influence the teaching–learning process of students. These barriers may be reflected in the mastery of technology by teachers, parents, students and the accessibility of the platforms used for the teaching and learning process. The evaluation of the accessibility of university websites has been carried out in numerous research studies to determine their compliance with WCAG, accessibility laws and standards. However, there is no study that analyzes all the work done to give an overview of their situation.

This review examines a dataset resulting from the evaluation results of the accessibility of university websites. The objective of this systematic literature review (SLR) is to synthesize the accessibility results of university websites: to make an analysis, synthesis and interpretation of the results published in 42 selected papers, to determine the web accessibility standards, evaluation methods used and the results obtained. The SLR methodology has been used to guide the research process [10].

Before carrying out this SLR, we made sure that no other similar research study exists [11]. To corroborate this, an exhaustive search for SLRs was carried out to determine the need for it. An SLR establishes a solid base to improve the knowledge, promotes the development of the theory, closes the areas that have been studied in excess and reveals the areas that need to be investigated [12]. Therefore, an SLR is important in determining the application of WCAG and accessibility barriers on university websites worldwide.

This SLR includes the following sections. Section 2 provides the background needed to understand the WCAG, its principles, guidelines, checkpoints or success criteria and their conformance levels. In Sect. 3, the methodology that will help to achieve the goals of the SLR is detailed. In Sect. 4, the results of the research questions and sub-questions are analyzed, synthesized and interpreted. In Sect. 5, the discussion highlights the most important findings of this SLR in an orderly and logical manner and identifies trends and gaps. In Sect. 6, the limitations of this study are presented. Finally, Section 7 presents the conclusions and future work.

Background

This section is necessary to interpret the results obtained from the SLR. It describes the recommendations of WCAG 1.0, 2.0 and 2.1, with their principles or priorities, guidelines, success criteria or checkpoints and conformance levels. The recommendations of the WCAG guide developers in building more accessible websites for people with and without disabilities [13]. In addition, the WCAG also explains how to make accessible the content that will be published on the websites (PDF documents, videos and so on). These guidelines are also used to assess the level of accessibility of websites.

Web content accessibility guidelines (WCAG) 1.0

WCAG 1.0 [14] was published in May 1999 by the W3C, with the aim of providing solutions to web accessibility problems. WCAG 1.0 includes 3 priorities, 14 guidelines, 65 checkpoints and 3 conformance levels, as shown in Fig. 1. Each checkpoint has a priority level according to its impact on accessibility. In addition, each checkpoint has a conformity level that allows to define the conformity level of a website.

Fig. 1

Priorities, checkpoints and conformance levels WCAG 1.0

Web content accessibility guidelines (WCAG) 2.0 and 2.1

The WCAG 2.0 [15], published in December 2008, includes 61 success criteria that are organized under 12 guidelines and 4 principles: Perceptible (P), Operable (O), Understandable (U) and Robust (R).

The WCAG 2.1 [7], published in June 2018, includes the WCAG 2.0 principles, guidelines, success criteria and conformance levels, plus 17 new success criteria and a new guideline. For a website to fully comply with the recommendations of WCAG 2.1, it must meet all the success criteria of WCAG 2.1 and conformance levels. Therefore, when a website is compliant with WCAG 2.1, the website is also compliant with WCAG 2.0. Figure 2 presents the principles, guidelines and conformance levels of WCAG 2.0 and 2.1. In addition, the 17 new WCAG 2.1 criteria have been highlighted in blue. The W3C is working on WCAG 2.2, and a draft was published in August 2020 [16].

Fig. 2

Principles, guidelines and conformance levels of WCAG 2.0 and 2.1 [17]

Conformance levels

WCAG 1.0, 2.0 and 2.1 have three conformance levels [7]:

A: “(the minimum level of conformance), the web page satisfies all the level A success criteria”;

AA: “the web page satisfies all the level A and level AA success criteria”;

AAA: “the web page satisfies all the level A, level AA and level AAA success criteria.”

Website accessibility conformance evaluation Methodology (WCAG-EM)

The procedure for evaluating websites using WCAG 2.0 developed by the W3C/WAI is the Website Accessibility Conformance Evaluation Methodology (WCAG-EM) [18]. This methodology guides evaluators to use good practices to avoid common mistakes and achieve more reliable results. The steps of this methodology can be seen in Fig. 3.

Fig. 3

Evaluation Procedure: WCAG-EM 1.0 [18]

Evaluation methods

Evaluation methods define the procedures, evaluation tools, end users and experts who assist in the evaluation of university websites. In an SLR carried out in 2019 [19], on web accessibility evaluation methods, it was determined that the main methods used are: “1) automatic tools, 2) evaluation by experts and 3) user tests are the most widely used techniques according to the literature.”

A list of tools for reviewing the accessibility of web content, either manually or automatically, is available on a website maintained by the W3C [20]. These tools are online services or software programs (AChecker, TAW, WAVE and others) that assist in checking whether web content is WCAG compliant. In addition, on this website we can find detailed information about each evaluation tool and its access link. Also, this website provides a set of filters to help the user find the evaluation tools that best suit their needs. Developers can use free or paid tools for web accessibility evaluation, or a combination of both. However, WebAIM [21] states that “no automated evaluation tool can tell you if your site is accessible, or even compliant. Human testing is always necessary because accessibility is about the human experience.”

On the other hand, evaluations with real users [22] are informal or formal experiments that are carried out with people who navigate a website in a normal way and their behavior is observed by the evaluators. After these experiments, the evaluators determine the accessibility problems based on what has been observed, user opinions, interviews, questionnaires and so on.

An objective evaluation of web accessibility requires more than the simple use of automatic tools for the evaluation of websites [23]. Effective evaluations are conducted with end users and web accessibility experts.

Methodology

An SLR makes a synthesis of the information published in different scientific databases in an orderly, precise and analytical way about a particular issue. The SLR methodology aims to guide the review process on a topic of interest to determine its research progress and to find new areas of research. This research adopts the Kitchenham’s guidelines, which include the following steps: “Planning Review, Conducting Review and Reporting Review” [24]. Figure 4 presents the flowchart of the SLR methodology used to determine accessibility findings and trends on university websites.

Fig. 4

Flowchart of the SLR methodology

Planning the systematic literature review

The aim of this stage is to determine the need for an SLR and to establish a review protocol. To determine the need for an SLR, an exhaustive search of SLRs in different scientific databases is performed.

Identification of the need for a systematic literature review

Using the following publications [24-27] as a reference, a search string was created to find similar SLRs on the accessibility of university websites and to determine whether the proposed SLR of this study will contribute to filling any gaps. Two equivalent search strings were created, one for the Web of Science database and one for Scopus database:The search with these strings yielded two literature reviews that are summarized and analyzed below: In summary, the first SLR identifies accessibility problems of websites of Saudi governments and universities. The second SLR analyzes the empirical results of the evaluation of accessibility of educational websites. Our SLR presented in this paper, unlike the other two, aims to determine the process of evaluating the accessibility of university websites worldwide. To this end, we start by identifying the continents and countries where accessibility research has been carried out on university websites (RQ1 [RQ1.1, RQ1.2]). Then, the screening process applied for the selection of the websites or other resources that have been evaluated is determined (RQ2 [RQ2.1, RQ2.2], RQ3 [RQ3.1, RQ3.2, RQ3.3]). The accessibility standards, accessibility laws, versions of the WCAG and levels of conformance used in the evaluations are defined (RQ4, RQ5, RQ6 [RQ6.1, RQ6.2, RQ6.3]). The methods, tools, end users or experts that help in the evaluation are defined (RQ7, RQ8, RQ9). The most common errors found in the results and their compliance with the levels of accessibility of the university websites are analyzed in the chosen studies (RQ10, RQ11).

Web of Science: TI=((“web accessibility” OR accessibilit* OR WCAG) AND (universit* OR “higher education” OR education*) AND (web* OR portal) AND (“systematic literature review” OR “literature review” OR “systematic review”));

Scopus: TITLE((“web accessibility” OR accessibilit* OR WCAG) AND (universit* OR “higher education” OR education*) AND (web* OR portal) AND (“systematic literature review” OR “literature review” OR “systematic review”)).

In 2017, an SLR [28] was conducted to identify web accessibility issues on the websites of Saudi universities and governments. The search was conducted on different databases from 2009 to 2017, analyzing a total of 123 articles. In the results, the authors revealed that web accessibility is a universal problem and that many countries in the world, among them Saudi Arabia, face web accessibility issues.

In 2020, an SLR [29] was conducted on the empirical results of the evaluation of accessibility of educational websites. The search was conducted on different databases from January 2009 to October 2019, analyzing a total of 25 papers. This SLR raised 10 research questions. The first three questions carried out the bibliometric analysis of the selected papers, and the remaining seven questions carried out the literature review. The authors in the results determined that 80 % of the analyzed studies make use of automatic evaluation tools, 8 % real users and 12 % a combination of experts, automatic tools and real users. The authors concluded that the educational websites analyzed in the SLR have accessibility problems.

Development of a review protocol

This research aims to compile papers published up to March 2021 on the accessibility of university websites in three scientific databases. To this end, a review protocol is developed, defining the research questions, the search strategy, the inclusion and exclusion criteria and quality assessment of the selected papers, which are analyzed and detailed below.

Research questions

The research questions are classified into four main groups: Eleven research questions were formulated, some with several sub-questions to achieve the objective of this SLR.

What was evaluated?

What standards were used for the evaluation?

What methods were used for the evaluation?

What results were obtained?

The research questions and sub-questions, objectives and expected results formulated in Table 1 will answer the first research question (RQ1 [RQ1.1, RQ1.2], RQ2 [RQ2.1, RQ2.2], RQ3 [RQ3.1, RQ3.2, RQ3.3]);

Table 1

Research questions related to the coverage of the evaluated universities—What was evaluated?

No.	Research question	Objectives	Expected results
RQ1	Where has the accessibility of the universities been analyzed the most?	Identify continents and countries where university websites have been evaluated.	Continents and countries.
RQ1.1	In which continents has the accessibility of university websites been analyzed the most?	Determine the continents where university websites have been evaluated.	Africa, America, Asia, Europe and Oceania.
RQ1.2	In which countries has the accessibility of university websites been analyzed the most?	Determine the countries where university websites have been evaluated.	Angola, Bolivia, Canada, Cameroon, China, France, Italy, Japan, Portugal, Spain, Turkey, UK, and so on.
RQ2	How have been the universities selected for analysis and how many universities have been analyzed?	Determine how universities have been selected for analysis and how many universities have been analyzed.	Methods of selection and number of universities evaluated.
RQ2.1	What methods have been used for the selection of university websites to be evaluated?	Determine the methods of selection of university websites.	All universities, Sampling, Randomly selected, and so on.
RQ2.2	How many university websites have been evaluated?	Determine how many university websites have been evaluated.	Number of university websites evaluated.
RQ3	What type of pages, how many, and what other resources have been evaluated on university websites?	Determine the type of pages, how many, and what other resources have been evaluated on university websites.	Type of web pages, number of web pages and other resources.
RQ3.1	What type of web pages have been evaluated?	Determine what web pages have been evaluated on university websites.	Homepage, contact form, enrollment form, and so on.
RQ3.2	How many web pages have been evaluated?	Determine how many web pages have been evaluated.	Number of web pages evaluated.
RQ3.3	What other resources have been evaluated?	Determine what other resources have been evaluated on university websites.	Videos, PDF, PowerPoint, and so on.

The research questions and sub-questions, objectives and expected results formulated in Table 2 will answer the second research question (RQ4, RQ5, RQ6 [RQ6.1, RQ6.2, RQ6.3]);

Table 2

Research questions related to the standards, laws and methods applied in the selection—What standards were used for the evaluation?

No.	Research question	Objectives	Expected results
RQ4	What are the web accessibility standards used to assess university websites?	Determine the web accessibility standard used to assess university websites.	ISO/IEC 40500:2012, Section 508, and so on.
RQ5	What are the accessibility laws mentioned?	Determine the accessibility laws mentioned in the selected papers.	SI 5568, Stanca Act, and so on.
RQ6	How are the WCAG used to assess university websites?	Determine the versions of the WCAG, the conformance levels and the WCAG-EM methodology used to assess university websites.	WCAG, conformance levels and WCAG-EM methodology.
RQ6.1	What are the WCAG versions used to assess university websites?	Determine the WCAG used to assess university websites.	WCAG 1.0, WCAG 2.0, WCAG 2.1, WCAG 2.2
RQ6.2	What are the conformance levels used to assess university websites?	Determine the conformance levels used to assess university websites.	A, AA, AAA
RQ6.3	Is WCAG-EM used to assess university websites?	Determine whether WCAG-EM methodology is used to assess university websites.	YES or NO

The research questions, objectives and expected results formulated in Table 3 will answer the third research question (RQ7, RQ8, RQ9);

Table 3

Research questions related to the methods, tools and types of users who have evaluated the accessibility—What methods were used for the evaluation?

No.	Research question	Objectives	Expected results
RQ7	What are the methods used to assess university websites?	Determine the methods used to evaluate the accessibility of university websites.	Automatic evaluation, Manual evaluation, Manual and automatic evaluation.
RQ8	What are the automatic tools used to assess university websites?	Identify the automatic evaluation tools that have helped to evaluate university websites.	AChecker, TAW, TENON, WAVE and others.
RQ9	Which experts and users helped assess university websites?	Identity the experts and real users who helped to evaluate the accessibility of university websites.	Blind users, deaf users, researchers, students, teachers, and so on.

The research questions, objectives and expected results formulated in Table 4 will answer the fourth research question (RQ10, RQ11). These questions can be answered only if the accessibility evaluation considered the WCAG.

Table 4

Research questions related to accessibility errors and conformance levels—What results were obtained?

No.	Research question	Objectives	Expected results
RQ10	What are the main errors found on university websites?	Determine the main accessibility errors that have been found on university websites by priority or principle.	Text alternatives, language, tables structure, navigation, contents of forms, keyboard interaction, and so on.
RQ11	What is the accessibility compliant status of university websites?	Determine the accessibility compliant status of the websites of the analyzed universities.	Compliant or Non-compliant.

The scope in an investigation allows to delimit the causes of the facts or phenomena being studied. Once the research questions and sub-questions were defined, the scope of the research was determined using the PICOC method proposed by Petticrew and Roberts [30]:In the results of this SLR, we answer the research questions and sub-questions defined in Tables 1, 2, 3, 4, through the analysis, synthesis and interpretation of the results found in the selected papers. The main contribution of this SLR is the consolidation of the results of the selected papers to determine findings and trends in the subject matter under investigation.

P: Population. Published papers on web accessibility;

I: Intervention. University websites;

C: Comparison. Comparison between accessibility standards, accessibility evaluation methods and the methods used for the selection of university websites to be evaluated;

O: Outcomes. Generate social responsibility in the administrators and developers of university websites;

C: Context. University environments.

A search string is a key piece in an SLR for item selection, as it delimits the scope and coverage of the investigation. The keywords and their replacement (substitution) terms for the search scope have been determined according to the research questions and sub-questions defined in Tables 1, 2, 3, 4. These keywords and their replacement terms used in the search scope of this SLR can be seen in Table 5.

Table 5

Search scope

Scope	Replacement terms
Context	(website* OR “web site” OR “web sites” OR web OR portal) AND
Accessibility	(“web accessibility” OR accessibilit*) AND
Education	(universit* OR “higher education” OR education*) AND
Research type	(eval* OR anal*)

The Boolean operators used in the search string are OR to join the replacement terms, so that the results show at least one of these terms, AND to combine the main parts of the search string where the results show all the search terms. These Boolean operators have been combined several times to create a custom search string. Moreover, the wildcard (*) has been applied to represent both the plural and singular of each keyword or replacement term and also to search for keywords containing certain characters. Finally, double quotes have been used to search for exact phrases. For the search in the scientific databases Web of Science, Scopus and IEEE Xplore, we use a specific search string for each database. The specific search strings used in each scientific database are listed below:The title, the abstract and the keywords describe in a concrete way what will be found in the content of a paper. For this reason, three specific search strings have been created to search the keywords and replacement terms defined in Table 5, in the titles and abstracts of the articles published in the scientific databases.

Web of Science: ((TI=“web accessibility” OR TI= accessibilit*) AND (TI=universit* OR TI=“higher education” OR TI= education*) AND (AB=website* OR AB= “web site” OR AB=“web sites” OR AB=web OR AB= portal) AND (AB=eval* OR AB=anal*));

Scopus: (TITLE ((“web accessibility” OR accessibilit*)) AND TITLE((universit* OR “higher education” OR education)) AND TITLE-ABS ((eval* OR anal*)) AND TITLE-ABS((website* OR “web site” OR “web sites” OR web OR portal)));

IEEE Xplore: ((“Document Title” :“web accessibility” OR “Document Title” :accessibilit*) AND (“Document Title” :universit* OR “Document Title” :“higher education” OR “Document Title” :education*) AND (“Abstract” :website* OR “Abstract” :“web site” OR “Abstract” :“web sites” OR “Abstract” :web OR “Abstract” :portal) AND (“Abstract” :eval* OR “Abstract” :anal*)).

The selection process of the studies plays a very important role in the results of the SLR. Therefore, all studies found with the search strings were evaluated to determine whether they should be included in this research. Papers that did not meet all inclusion criteria were excluded from the review. Papers that met at least one of the exclusion criteria were excluded. The year of publication of the papers was not taken into account as a parameter of inclusion and exclusion. The inclusion criteria used in this SLR are presented below:The exclusion criterion was used to discard papers that met the following conditions:

I1. Papers published in a journal AND;

I2. Papers written in English AND;

I3. Papers that at least mention the accessibility of university websites in the abstract.

E1. Papers that are secondary research (e.g., an SLR) OR;

E2. Duplicated papers OR;

E3. Papers that are not research papers (e.g., letter to the editor, erratum).

Quality assessment (QA) is intended to contribute to the selection of papers through a set of questions that must be answered to guide the research [24]. The indexing of the journals in which the works have been published (Scimago Journal & Country Rank (SJR) and Journal Citation Reports (JCR)) has also been incorporated into the quality questions. Seven questions have been defined to measure the quality of each paper. Each question has a score of 1; therefore, the maximum total score is 7. Table 6 shows the QA questions that were applied to each paper.

Table 6

Quality assessment checklist [31]

No.	Quality assessment question	Expected results
Q.A.1	Is web accessibility detailed in the paper?	YES->1/NO->0
Q.A.2	Is the web accessibility evaluation method specified in the paper?	YES->1/NO->0
Q.A.3	Are the empirical results of the web accessibility evaluation shown?	YES->1/NO->0
Q.A.4	Does the paper discuss any findings of web accessibility evaluation?	YES->1/NO->0
Q.A.5	Are common web accessibility errors described in the results?	YES->1/NO->0
Q.A.6	Is the paper published in a journal indexed in SJR?	(+1) if the paper is indexed in a Q1 journal, (+0.75) if the paper is indexed in a Q2 journal, (+0.5) if the paper is indexed in a Q3 journal, (+0.25) if the paper is indexed in a Q4 journal, (+0) if it is not in the ranking.
Q.A.7	Is the paper published in a journal indexed in JCR?	(+1) if the paper is indexed in a Q1 journal, (+0.75) if the paper is indexed in a Q2 journal, (+0.5) if the paper is indexed in a Q3 journal, (+0.25) if the paper is indexed in a Q4 journal, (+0) if it is not in the ranking.

Conducting the systematic literature review

Identification of research

An SLR allows the analysis of literature on specific research topics. To develop an SLR, it is necessary to determine the search terms and to define the scientific databases where the search will be carried out. In a study conducted in 2019 [32], the search quality of PubMed, Google Scholar and other 26 academic search databases was evaluated; the results showed that Google Scholar is not suitable as a primary search resource. For this reason, the most notable scientific databases in the field of research were selected for this SLR, such as Web of Science, Scopus and IEEE Xplore Digital Library. We estimate that these databases are sufficient because there are even duplicate papers in the search results, i.e., the same paper appears in several databases at the same time, which shows that the coverage of these scientific databases is very high. This collection of databases of bibliographic references was selected using the following criteria:

They collect references from scientific publications that disseminate scientific knowledge;

They index high-quality papers that are peer-reviewed;

They allow customized searches using query operators.

Selection of studies

The selection process covered papers indexed in the scientific databases until March 2021. In the Scopus scientific database 104 papers were found, in Web of Science 99 papers and in the IEEE Xplore 16 papers. Of the 219 papers, the selection was made applying the inclusion and exclusion criteria as follows: 98 papers were discarded because they were not published in a journal (I1); 35 papers were discarded because they were not written in the English language (I2); 19 papers were discarded because the abstract did not describe university websites (I3); 2 papers were discarded because they were the result of an SLR (E1); 23 papers were discarded because they were duplicated (E2), and finally, no papers were discarded because they were all research papers (E3). In summary, of the 219 papers found, 177 were discarded, leaving 42 papers selected for this SLR. The flowchart of the study selection can be seen in Fig. 5.

Fig. 5

Flowchart of studies selection

Quality assessment

To achieve the objective of the SLR, papers must comply with the quality assessment parameters [33] defined in Table 6. After evaluating the papers, the sum of the results obtained from each paper was made. Then, the values obtained were normalized to standardize the results. To this end, the minimum–maximum normalization formula [34] was used, which calculates the final values on a scale of 0 to 1. Below is the formula used to calculate this value (1):where the min(Score) has a value of 0, the max(Score) has a value of 7, and the Score takes the value of the sum of the QA values of each paper. Papers with a normalized score of less than 0.70 were excluded from the SLR.

Reporting the systematic literature review

This stage aims to answer the research questions and sub-questions presented in Tables 1, 2, 3, 4. The following section presents the literature review answering questions RQ1 [RQ1.1, RQ1.2], RQ2 [RQ2.1, RQ2.2], RQ3 [RQ3.1, RQ3.2, RQ3.3], RQ4, RQ5, RQ6 [RQ6.1, RQ6.2, RQ6.3], RQ7, RQ8, RQ9, RQ10 and RQ11, making a summary and synthesis of the data collected from the results of the selected studies.

Results

In this section, we first present the results of the QA of the selected papers. Secondly, we answer each research question by summarizing and discussing the results of the selected papers.

Table 7 shows the 42 papers yielded by the search and selection process. Each paper includes the reference, the publication year and the name of the journal where it was published. In addition, the QA results for each one of the selected papers can also be seen in Table 7. This table is sorted by publication year. To standardize the sum of the QA values, a normalization column was created in which the final compliance value of each paper is calculated between 0 and 1. After the calculation, it was possible to see that several items scored 0.71, but were included in the SLR. These articles were not published in journals indexed in SJR (QA6) and JCR (QA7).

Table 7

Results of the quality assessment of the selected papers, sorted by publication year

Paper	Year	Journal name	Quality assessment								Normalization
			Q.A.1	Q.A.2	Q.A.3	Q.A.4	Q.A.5	Q.A.6	Q.A.7	Score
[35]	2002	Interacting with Computers (IC)	1	1	1	1	1	0.75	0.75	6.50	0.92
[36]	2002	Reference and User Services Quarterly (RUSQ)	1	1	1	1	1	1.00	0.75	6.75	0.96
[37]	2003	Information Technology and Disabilities (ITD)	1	1	1	1	1	0.25	0.00	5.25	0.75
[38]	2005	Internet Research	1	1	1	1	1	0.75	0.00	5.75	0.82
[39]	2007	Library Hi Tech	1	1	1	1	1	0.75	0.00	5.75	0.82
[40]	2007	Journal of Special Education Technology (JSET)	1	1	1	1	1	0.75	0.00	5.75	0.82
[41]	2008	Internet and Higher Education (IHE)	1	1	1	1	1	0.75	0.00	5.75	0.82
[42]	2010	Disability and Rehabilitation: Assistive Technology (DRAT)	1	1	1	1	1	0.75	0.00	5.75	0.82
[43]	2011	Disability and rehabilitation (DR)	1	1	1	1	1	1.00	1.00	7.00	1.00
[44]	2011	Universal Access in the Information Society (UAIS)	1	1	1	1	1	0.50	0.00	5.50	0.78
[45]	2013	Information Technology and Disabilities (ITD)	1	1	1	1	1	0.25	0.00	5.25	0.75
[46]	2013	Universal Access in the Information Society (UAIS)	1	1	1	1	1	0.75	0.25	6.00	0.85
[47]	2014	International Education Studies (IES)	1	1	1	1	1	0.50	0.00	5.50	0.78
[48]	2015	International Journal of Emerging Technologies in Learning (IJET)	1	1	1	1	1	0.50	0.00	5.50	0.78
[49]	2016	Universal Access in the Information Society (UAIS)	1	1	1	1	1	0.75	0.50	6.25	0.89
[50]	2016	Journal of King Saud University - Computer and Information Sciences (JKSUCIS)	1	1	1	1	1	0.75	0.00	5.75	0.82
[51]	2016	International Journal of Advanced Computer Science and Applications (IJACSA)	1	1	1	1	1	0.25	0.00	5.25	0.75
[52]	2016	Journal of Information and Communication Technology-Malaysia (JICT)	1	1	1	1	1	0.50	0.00	5.50	0.78
[53]	2017	International Journal of Online Pedagogy and Course Design (IJOPCD)	1	1	1	1	1	0.00	0.00	5.00	0.71
[54]	2017	Journal of Information and Communication Technology (JICT)	1	1	1	1	1	0.75	0.00	5.75	0.82
[55]	2017	Procedia Computer Science (PCS)	1	1	1	1	1	0.00	0.00	5.00	0.71
[56]	2017	Universal Access in the Information Society (UAIS)	1	1	1	1	1	0.75	0.50	6.25	0.89
[57]	2017	Journal of Computing in Higher Education (JCHE)	1	1	1	1	1	1.00	1.00	7.00	1.00
[58]	2018	International Journal of High Performance Computing and Networking (IJHPCN)	1	1	1	1	1	0.75	0.00	5.75	0.82
[59]	2018	IOP Conference Series: Materials Science and Engineering (IOP)	1	1	1	1	1	0.00	0.00	5.00	0.71
[60]	2018	Universal Access in the Information Society (UAIS)	1	1	1	1	1	0.75	0.25	6.00	0.85
[61]	2018	IEEE Access	1	1	1	1	1	1.00	1.00	7.00	1.00
[62]	2018	Universal Access in the Information Society (UAIS)	1	1	1	1	1	0.75	0.25	6.00	0.85
[63]	2018	Universal Access in the Information Society (UAIS)	1	1	1	1	1	0.75	0.25	6.00	0.85
[64]	2018	Journal of Accessibility and Design for All (JADA)	1	1	1	1	1	0.25	0.00	5.25	0.75
[65]	2019	Universal Access in the Information Society (UAIS)	1	1	1	1	1	0.75	0.75	6.50	0.92
[66]	2019	Journal of King Saud University - Computer and Information Sciences (JKSUCIS)	1	1	1	1	1	0.75	0.00	5.75	0.82
[67]	2019	TechTrends	1	1	1	1	1	0.75	0.00	5.75	0.82
[68]	2019	IEEE Access	1	1	1	1	1	1.00	1.00	7.00	1.00
[69]	2019	International Journal of Computer Science and Network Security (IJCSNS)	1	1	1	1	1	0.00	0.00	5.00	0.71
[70]	2019	Universal Access in the Information Society (UAIS)	1	1	1	1	1	0.75	0.75	6.50	0.92
[71]	2020	Information (Switzerland) (IS)	1	1	1	1	1	0.50	0.00	5.50	0.78
[72]	2020	Data in Brief (DB)	1	1	1	1	1	0.25	0.00	5.25	0.75
[73]	2020	Universal Access in the Information Society (UAIS)	1	1	1	1	1	0.75	0.75	6.50	0.92
[74]	2020	Universal Access in the Information Society (UAIS)	1	1	1	1	1	0.75	0.75	6.50	0.92
[75]	2020	Universal Access in the Information Society (UAIS)	1	1	1	1	1	0.75	0.75	6.50	0.92
[76]	2020	IEEE Access	1	1	1	1	1	1.00	1.00	7.00	1.00

Systematic literature review

RQ1. Where has the accessibility of the universities been analyzed the most?

To answer this research question, two research sub-questions have been answered. These questions identify the continents and countries where university websites were analyzed in the selected papers.

According to the results, we can see that 47.6 % of the selected papers analyze the accessibility of university websites in Asia, 33.3 % in America, 9.5 % in Europe and 2.4 % in Africa. However, there are 3 papers evaluating university websites on more than one continent (6.12 %). The number of papers per continent or continents can be seen in Fig. 6 (see Table 15, in “Appendix” A, for full data).

Fig. 6

Number of papers per continent

Table 15

Data collected for RQ1, RQ4, RQ5

Paper	RQ1		RQ4	RQ5
	RQ1.1 Continent	RQ1.2 Country	Accessibility standard	Accessibility laws
[35]	Europe	UK	WCAG 1.0	.
[36]	America	USA	WCAG 1.0	.
[37]	America	USA	WCAG 1.0, Section 508	.
[38]	America	USA	WCAG 1.0, Section 508	.
[39]	America	Canada, USA	WCAG 1.0, Section 508	.
[40]	America	USA	WCAG 1.0, Section 508	.
[41]	America	USA	WCAG 1.0, Section 508	.
[42]	America	USA	WCAG 1.0, Section 508	.
[43]	Europe	Portugal	WCAG 1.0	.
[44]	Asia	Turkey	ISO/IEC 40500:2012 (WCAG 2.0)	.
[45]	America	USA	WCAG 1.0, Section 508
[46]	America	USA	WCAG 1.0	.
[47]	Europe	Spain	ISO/IEC 40500:2012 (WCAG 2.0)	.
[48]	Asia	Jordan	ISO/IEC 40500:2012 (WCAG 2.0)	.
[49]	Asia	Turkey	ISO/IEC 40500:2012 (WCAG 2.0)	.
[50]	Asia	India	ISO/IEC 40500:2012 (WCAG 2.0)	.
[51]	Asia	Jordan	ISO/IEC 40500:2012 (WCAG 2.0)	.
[52]	Asia	Malaysia	ISO/IEC 40500:2012 (WCAG 2.0), Section 508	.
[53]	America	USA	ISO/IEC 40500:2012 (WCAG 2.0), Section 508	.
[54]	Asia	Palestinian	ISO/IEC 40500:2012 (WCAG 2.0)	.
[55]	Asia	Turkey	ISO/IEC 40500:2012 (WCAG 2.0)	.
[56]	Asia	Kyrgyz Republic	WCAG 1.0	.
[57]	America	USA	WCAG 1.0, Section 508	.
[58]	Asia	Jordan	ISO/IEC 40500:2012 (WCAG 2.0)	.
[59]	Asia	Indonesia	ISO/IEC 40500:2012 (WCAG 2.0)	.
[60]	Asia	Azerbaijan, Kazakhstan, Kyrgyz Republic, Turkey	ISO/IEC 40500:2012 (WCAG 2.0)	.
[61]	America	Antigua Barbuda, Argentina, Aruba, Bolivia, Brazil, Chile, Colombia, Costa Rica, Cuba, Dominica, Ecuador, El Salvador, Guatemala, Haiti, Honduras, Jamaica, Mexico, Nicaragua, Panama, Paraguay, Peru, Puerto Rico, Dominican Republic, Trinidad and Tobago, Uruguay, Venezuela	ISO/IEC 40500:2012 (WCAG 2.0)	.
[62]	Africa	South Africa	ISO/IEC 40500:2012 (WCAG 2.0)	.
[63]	Asia	Israel	ISO/IEC 40500:2012 (WCAG 2.0)	SI 5568
[64]	Asia	Iran	ISO/IEC 40500:2012 (WCAG 2.0)	.
[65]	Europe	Portugal	ISO/IEC 40500:2012 (WCAG 2.0)	.
[66]	Asia	India	ISO/IEC 40500:2012 (WCAG 2.0)	.
[67]	America	USA	ISO/IEC 40500:2012 (WCAG 2.0), Section 508	.
[68]	America, Asia	Argentina, Brazil, Colombia, Chile, China, Mexico, Puerto Rico, USA	WCAG 2.1	.
[69]	Asia	Pakistan	ISO/IEC 40500:2012 (WCAG 2.0)	.
[70]	America, Europe	Chile, Mexico, Spain	ISO/IEC 40500:2012 (WCAG 2.0)	.
[71]	Asia	Saudi Arabia	ISO/IEC 40500:2012 (WCAG 2.0)	.
[72]	America	Antigua Barbuda, Argentina, Aruba, Bolivia, Brazil, Chile, Colombia, Costa Rica, Cuba, Dominica, Ecuador, El Salvador, Guatemala, Haiti, Honduras, Jamaica, Mexico, Nicaragua, Panama, Paraguay, Peru, Puerto Rico, Dominican Republic, Trinidad and Tobago, Uruguay, Venezuela	WCAG 2.1	.
[73]	Europe	Italy	WCAG 2.1	Stanca Act
[74]	Asia	Turkey	ISO/IEC 40500:2012 (WCAG 2.0)	.
[75]	Asia	Kuwait	ISO/IEC 40500:2012 (WCAG 2.0)	.
[76]	America, Africa, Asia, Europe, Oceania	Argentina, Australia, Belgium, Brazil, Canada, China, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, India, Iran, Israel, Italy, Japan, Luxembourg, Mexico, Netherlands, New Zealand, Nigeria, Norway, Portugal, Romania, Russia, Saudi Arabia, South Africa, South Korea, Spain, Sweden, Thailand, Turkey, UK, USA	WCAG 2.1	.

“.....” means that the paper does not identify the accessibility standard or law used

A map has been developed to visually present the scope of web accessibility evaluation in the world (see Fig. 7). The largest number of countries per continent that have evaluated university websites is:

Fig. 7

Map of the universities evaluated by country and continent

America: 7,865 universities in 28 countries;

Asia: 1,003 universities in 18 countries;

Europe: 235 universities in 18 countries;

Africa: 31 universities in 2 countries;

Oceania: 6 universities in 2 countries.

A total of 9,140 university websites worldwide have been evaluated. In summary, the scope of the evaluation is 5 continents and 68 countries. In Fig. 8, we can see the number of university websites evaluated by country and continent (see Table 15, in “Appendix” A, for full data).

Fig. 8

Number of university websites evaluated per country and continent (There is a gap in the scale for the sake of clarity)

RQ2. How have been the universities selected for analysis and how many universities have been analyzed?

To answer this research question, two research sub-questions have been posed. They analyze the number of universities and the methods used for the selection of the web pages of the universities evaluated in the selected papers.

The mode of selection of the universities has been classified into three types: sampling, randomly selected and all universities. When the authors refer to all universities, they are considering those universities that belong to a certain classification or country. In the 42 papers analyzed, 83.33 % of the papers selected universities by sampling, 11.90 % by random selection and 4.77 % all universities. The results can be seen in Fig. 9 (see Table 16, in “Appendix” A, for full data).

Fig. 9

Methods used for the selection of the websites of the universities to be evaluated

Table 16

Data collected for RQ2 and RQ3

Paper	RQ2		RQ3
	RQ2.1 Selection method	RQ2.2 # Universities	RQ3.1 Type web pages	RQ3.2 # web pages	RQ3.3 others resources
[35]	Sampling	11	Homepages	11	.
[36]	Sampling	188	Homepages	188	.
[37]	Sampling	102 (public)	University homepage, Search page, List of university colleges, departments, and/or degree programs, Campus directory of faculty, staff and/or students, Admissions homepage, Course listings, Academic calendar, Employment homepage, Job listings, Campus map, Library homepage	1,013	.
[38]	Sampling	45 (45 universities) (22 government)	Homepages	45	.
[39]	Sampling	56 (49 USA) (7 Canada)	Homepages	56	.
[40]	Sampling	50	Homepages	50	.
[41]	Sampling	6	Homepages	6	.
[42]	Sampling	127	Homepages	127	.
[43]	All universities	64 (public)	Homepages	64	.
[44]	Randomly selected	10 (6 public) (4 private)	Homepages	10	.
[45]	Randomly selected	3,251	Homepages	31,701	28,395 PDFs
[46]	Sampling	51 (University Departments of Special Education)	Homepages	51	.
[47]	Sampling	21 (educational web portals)	.	42	.
[48]	Sampling	6 (3 public) (3 private)	.	6	.
[49]	Randomly selected	10 (6 public) (4 private)	Homepages	10	.
[50]	Sampling	302 (central and public)	Homepages	302	.
[51]	Sampling	36 (9 public) (27 private)	Homepages, registration web page	72	.
[52]	Sampling	20 (public)	Homepages	20	.
[53]	Sampling	24 (public and private)	Homepages	24	.
[54]	Sampling	15	.	15	.
[55]	Sampling	38	Homepages	38	.
[56]	Sampling	42 (28 public) (14 private)	Homepages	42	.
[57]	Sampling	3,141	Homepages	3,141	.
[58]	Sampling	27 (accredited Jordanian)	Homepages	27	.
[59]	Sampling	13 (Ministry of Research, Technology and Higher Education)	Homepages	13	.
[60]	Sampling	60 (Webometrics)	Homepages	60	.
[61]	Sampling	348	Homepages	348	.
[62]	Sampling	26	Homepages	26	.
[63]	Sampling	1	Library Homepage	1	.
[64]	Sampling	50 (Ministry of Health of Iran)	Homepages	50	.
[65]	Sampling	59 (19 polytechnic) (40 universities)	Homepages	59	.
[66]	Sampling	44	Homepages	44	.
[67]	Sampling	24	Homepages	24	.
[68]	Sampling	40	Homepages	40	.
[69]	Randomly selected	4	Homepages	4	.
[70]	Sampling	45 (Webometrics)	Homepages	45	.
[71]	Sampling	6	Homepages	6	.
[72]	Randomly selected	348	Homepages	348	.
[73]	Sampling	67 (public)	Homepages	67	.
[74]	All universities	179 (110 public) (69 private)	Homepages	179	.
[75]	Sampling	41	Homepages	41	.
[76]	Sampling	142 (Shanghai Ranking)	.	.	91,421 YouTube videos

“.....” means that the paper does not mention the type of web pages evaluated or the number of web pages or other resources

The comparison between the WCAG and the methods used in the selection of university websites is briefly presented below:As a result of the comparison, the trend in the use of “sampling” for the selection of university websites in the three versions of the WCAG is evident. However, there are also significantly fewer who use “randomly selected” to determine the university websites to be evaluated.

WCAG 1.0. The selection of university websites for web accessibility evaluation was made using the following methods: sampling (11 papers), randomly selected (1 paper), all universities (1 paper);

WCAG 2.0. The methods for selecting university websites for web accessibility evaluation are as follows: sampling (21 papers), randomly selected (3 papers), all universities (1 paper);

WCAG 2.1. The selection of university websites for web accessibility evaluation was made using the following methods: sampling (3 papers), randomly selected (1 paper).

A total of 9,140 university websites have been evaluated in the 42 selected papers. The paper with the highest number of universities evaluated is [45], which analyzes 3,251 universities. The average number of universities evaluated worldwide is 217.62. In Table 8, we can see the descriptive statistical summary of the universities evaluated per paper (see Table 16, in “Appendix” A, for full data).

Table 8

Descriptive statistical summary of the university websites evaluated per paper

Measure	Number	Description
Min.	1.00	This is the minimum number of university websites that have been analyzed.
Median	43.00	This is the central number of the ordered dataset of the university websites.
Mean	217.62	This is the average number of university websites analyzed.
Max.	3,251.00	This is the maximum number of university websites that have been analyzed.

RQ3 What type of pages, how many and what other resources have been evaluated on university websites?

To answer this research question, three research sub-questions have been posed. They analyze the types of web pages, how many web pages and what other resources have been evaluated on the websites of the universities of the selected papers.

Of the 42 papers analyzed, 37 papers evaluate the home pages of the university websites. The papers [47, 48, 54, 76] do not refer in their content to the web pages that have been evaluated in the universities. The papers [37] and [63] evaluated the library homepage. The results are presented in Fig. 10 (see Table 16, in “Appendix” A, for full data).

Fig. 10

Types of web pages evaluated at universities

A total of 38,416 web pages have been evaluated in the 42 selected papers. The paper with the highest number of evaluated web pages is [45], which analyzes 31,701 web pages. The accessibility data of the web pages were collected through a semi-automatic procedure developed in PHP and using the Google Custom Search API (Google). In Table 9, we can see the descriptive statistical summary of the web pages evaluated per paper (see Table 16, in “Appendix” A, for full data).

Table 9

Descriptive statistical summary of the web pages evaluated per paper

Measure	Number	Description
Min.	1.00	This is the minimum number of web pages evaluated.
Median	44.00	This is the central number of the ordered dataset of the web pages.
Mean	937.00	This is the average number of web pages evaluated.
Max.	31,701.00	This is the maximum number of web pages evaluated.

Other resources, such as videos and PDF documents, have been evaluated on university websites. A total of 91,421 YouTube videos have been evaluated in the paper [76] and 28,395 PDF documents in the paper [45] (see Table 16, in “Appendix” A, for full data).

RQ4. What are the web accessibility standards used to assess university websites?

The standards used in evaluating the accessibility of university websites are ISO/IEC 40500:2012 [6], Section 508 [77] and a combination of the two standards. Section 508 and WCAG 1.0 were used for the evaluation of university websites in 8 papers, ISO/IEC 40500:2012 in 22 papers and the combination of the two standards ISO/IEC 40500:2012 and Section 508 in 3 papers. WCAG 1.0 is used for the evaluation of university websites in 5 papers and WCAG 2.1 in 4 papers. Figure 11 shows the results (see Table 15, in “Appendix” A, for full data).

Fig. 11

Number of papers by web accessibility standard used in the evaluation of university websites

WCAG 2.1 has been used in papers [66, 72, 73, 76] to evaluate the accessibility of university websites. Taking into account that WCAG 2.1 adopts all the principles, guidelines and success criteria of WCAG 2.0, we could say that these papers also use ISO/IEC 40500:2012. Therefore, from this point of view, the total number of papers that use ISO/IEC 40500:2012 is 29 and this is the web accessibility standard most used in the evaluations.

RQ5. What are the accessibility laws mentioned?

The web accessibility laws and policies [78] have been taken as a reference to answer this question, although this is not an exhaustive or definitive list. After reviewing the 42 selected papers, the articles that make use of an accessibility law are cited:Therefore, only 4.76 % of the analyzed papers make use of a web accessibility law in their evaluation. The remaining 40 papers do not mention any web accessibility law (see Table 15, in “Appendix” A, for full data).

[63]—“SI 5568, Israeli web accessibility guidelines” ;

[73]—“Stanca Act, Italian accessibility legislation.”

RQ6. How are the WCAG used to assess university websites?

To answer this research question, three research sub-questions have been answered. They determine the conformance levels and the versions of the WCAG used for the evaluation of university websites. In addition, it is verified whether the WCAG-EM methodology is used in the evaluation of the websites of the universities analyzed in the selected papers.

Considering the official versions of the WCAG published by the W3C, it has been determined that WCAG 1.0 has been used to evaluate university websites in 13 papers, WCAG 2.0 in 25 papers and WCAG 2.1 in 4 papers. Figure 12 shows the trend in the use of the different versions of WCAG over time (see Table 17, in “Appendix” A, for full data).

Fig. 12

Number of papers per version of the WCAG and year of publication

Table 17

Data collected for RQ6

	RQ6
Paper	RQ6.1 WCAG 1.0			WCAG 2.0 or 2.1				RQ6.2 Conformance level			RQ6.3 methodology
	P 1	P 2	P 3	P	O	U	R	A	AA	AAA	WCAG-EM
[35]	✓	✓	✓	✗	✗	✗	✗	✗	✗	✗	✗
[36]	✓	✗	✗	✗	✗	✗	✗	✗	✗	✗	✗
[37]	✓	✗	✗	✗	✗	✗	✗	✓	✗	✗	✗
[38]	✓	✓	✓	✗	✗	✗	✗	✓	✓	✓	✗
[39]	✓	✗	✗	✗	✗	✗	✗	✗	✗	✗	✗
[40]	✓	✓	✓	✗	✗	✗	✗	✗	✗	✗	✗
[41]	✓	✓	✓	✗	✗	✗	✗	✗	✗	✓	✗
[42]	✓	✗	✗	✗	✗	✗	✗	✗	✗	✗	✗
[43]	✓	✓	✓	✗	✗	✗	✗	✗	✗	✗	✗
[44]	✗	✗	✗	✓	✓	✓	✓	✓	✗	✗	✗
[45]	✓	✓	✓	✗	✗	✗	✗	✗	✗	✗	✗
[46]	✓	✓	✓	✗	✗	✗	✗	✗	✗	✗	✗
[47]	✗	✗	✗	✓	✓	✓	✓	✓	✓	✓	✗
[48]	✗	✗	✗	✓	✓	✓	✓	✓	✗	✗	✗
[49]	✗	✗	✗	✓	✓	✓	✓	✓	✗	✗	✗
[50]	✗	✗	✗	✓	✓	✓	✓	✓	✓	✓	✗
[51]	✗	✗	✗	✓	✓	✓	✓	✓	✓	✓	✗
[52]	✗	✗	✗	✓	✓	✓	✓	✓	✓	✓	✗
[53]	✗	✗	✗	✓	✓	✓	✓	✗	✓	✗	✗
[54]	✗	✗	✗	✓	✓	✓	✓	✓	✓	✗	✗
[55]	✗	✗	✗	✓	✓	✓	✓	✗	✓	✗	✗
[56]	✓	✓	✓	✗	✗	✗	✗	✓	✓	✓	✗
[57]	✓	✗	✗	✗	✗	✗	✗	✗	✗	✗	✗
[58]	✗	✗	✗	✓	✓	✓	✓	✓	✗	✗	✗
[59]	✗	✗	✗	✓	✓	✓	✓	✓	✗	✗	✗
[60]	✗	✗	✗	✓	✓	✓	✓	✓	✓	✓	✗
[61]	✗	✗	✗	✓	✓	✓	✓	✓	✗	✗	✗
[62]	✗	✗	✗	✓	✓	✓	✓	✓	✓	✓	✗
[63]	✗	✗	✗	✓	✓	✓	✓	✓	✓	✗	✗
[64]	✗	✗	✗	✓	✓	✓	✓	✓	✓	✗	✗
[65]	✗	✗	✗	✓	✓	✓	✓	✓	✓	✓	✗
[66]	✗	✗	✗	✓	✓	✓	✓	✗	✓	✗	✗
[67]	✗	✗	✗	✓	✓	✓	✓	✓	✓	✗	✗
[68]	✗	✗	✗	✓	✓	✓	✓	✗	✓	✗	✗
[69]	✗	✗	✗	✓	✓	✓	✓	✓	✓	✓	✗
[70]	✗	✗	✗	✓	✓	✓	✓	✓	✓	✗	✗
[71]	✗	✗	✗	✓	✓	✓	✓	✓	✗	✗	✗
[72]	✗	✗	✗	✓	✓	✓	✓	✓	✗	✗	✗
[73]	✗	✗	✗	✓	✓	✓	✓	✗	✓	✗	✗
[74]	✗	✗	✗	✓	✓	✓	✓	✓	✗	✗	✗
[75]	✗	✗	✗	✓	✓	✓	✓	✓	✓	✓	✗
[76]	✗	✗	✗	✓	✓	✓	✓	✓	✓	✓	✗

Of the 42 selected papers, all make use of one of the three versions of WCAG. In 10 papers, university websites are evaluated with conformance level A, 5 papers with conformance level AA and 1 paper with conformance level AAA. In addition, the websites of 5 papers were evaluated with A and AA conformance levels and 12 papers with A, AA and AAA conformance levels. It should be noted that 9 papers [35, 36, 39, 40, 42, 43, 45, 46, 57] do not stipulate the conformance levels used for the evaluation of university websites. The papers by conformance levels can be seen in Fig. 13 (see Table 17, in “Appendix” A, for full data).

Fig. 13

Number of papers by conformance levels

All analyzed papers use some of the steps of the WCAG-EM Methodology; however, none of them stipulates the whole use of this methodology in their manuscript. Therefore, none of the papers makes a strict use of the WCAG-EM methodology in the evaluation of university websites (see Table 17, in “Appendix” A, for full data).

RQ7. What are the methods used to assess university websites?

Taking into account the web accessibility evaluation methods defined in Section II Background, they have been classified into three types: 1) automatic evaluation tools (35 papers), 2) manual evaluation (end users and experts, 4 papers) and 3) manual and automatic evaluation (combination of the two methods, 3 papers). Figure 14 presents the evaluation methods used in the selected papers (see Table 18, in “Appendix” A, for full data).

Fig. 14

Evaluation methods used in the selected papers

Table 18

Data collected for RQ7, RQ8, RQ9

Paper	RQ7	RQ8	RQ9
	Evaluation method	Evaluation tools	Real users	Experts
[35]	Automatic	Bobby, W3C HTML Validate Tool	.	.
[36]	Automatic	Bobby	.	.
[37]	Automatic and Manual	Bobby, Internet Explorer, JAWS	.	2 experts
[38]	Automatic	Bobby	.	.
[39]	Automatic	Bobby	.	.
[40]	Automatic	Bobby	.	.
[41]	Automatic	Bobby	.	.
[42]	Manual	Internet Explorer 7.0 (IE), JAWS, Web Accessibility Toolbar (WAT)	.	Researchers
[43]	Automatic	Bobby, eXaminator, HERA	.	.
[44]	Automatic	AChecker, SortSite, Web Accessibility Checker	.	.
[45]	Manual	Procedure developed in PHP and utilizing Google’s Custom Search API (Google)	.	.
[46]	Automatic	AChecker, Bobby	.	.
[47]	Automatic	TAW	.	.
[48]	Automatic	WAVE	.	.
[49]	Automatic	AChecker, SortSite, Web Accessibility Checker	.	.
[50]	Automatic	AChecker, WAVE, Web Page Analyzer	.	.
[51]	Automatic	AChecker, CynthiaSays, Functional Accessibility Evaluator (FAE), HERA, TAW, WAVE, W3C Markup Validation	.	.
[52]	Automatic	AChecker, WAVE	.	.
[53]	Automatic	AChecker	.	.
[54]	Automatic and Manual	CynthiaSays	16 Blind users	.
[55]	Automatic	EIII Page Checker, TAW, WAVE	.	.
[56]	Automatic	EvalAccess 2.0	.	.
[57]	Manual	Dedicated LAMP (Linux, Apache, MySQL, PHP) server	.	.
[58]	Automatic	AccessMonitor, EIII Page Checker, WAVE	.	.
[59]	Automatic	TAW	.	.
[60]	Automatic	AChecker	.	.
[61]	Automatic	WAVE	.	.
[62]	Automatic	AChecker, TAW	.	.
[63]	Automatic	WAVE	.	.
[64]	Automatic	AChecker, Functional Accessibility Evaluator (FAE)	.	.
[65]	Automatic	AChecker, aXe, WAVE	.	.
[66]	Automatic	aXe, TAW	.	.
[67]	Automatic	AChecker	.	.
[68]	Automatic and Manual	UX Check Tool	.	2 experts
[69]	Automatic	PowerMapper, WAVE	.	.
[70]	Automatic	TAW, WAVE	.	.
[71]	Automatic	SiteImprove, WAVE	.	.
[72]	Automatic	WAVE	.	.
[73]	Automatic	AChecker	.	.
[74]	Automatic	AChecker	.	.
[75]	Automatic	AChecker, Total Validator, WAVE	.	.
[76]	Manual	.	.	Researchers

“.....” means that no evaluation tools or real users or experts are used in the paper

RQ8. What are the automatic tools used to assess university websites?

Considering the web accessibility evaluation tools published on the W3C website [20], it has been determined that 38 selected papers use automatic evaluation tools in their evaluation. Therefore, 90.47 % of university websites are evaluated using automatic tools. Figure 15 shows the number of papers per automatic evaluation tool (see Table 18, in “Appendix” A, for full data).

Fig. 15

Number of papers by automatic evaluation tools

The automatic tools most used in the evaluation of university websites are: AChecker (15 papers), WAVE (14 papers), Bobby (9 papers) and TAW (7 papers). By making an analysis, it is determined that:Bobby, launched in 1996 [79], is one of the first tools for web accessibility evaluation, which had several versions one more advanced than another and lost its continuity in 2005. The other accessibility evaluation tools have adapted to the changes in WCAG. In summary, the most widely used tool with WCAG 1.0 for automatic evaluation of university websites is Bobby. With the publication of WCAG 2.0, university websites were evaluated with the automatic tools AChecker, TAW and WAVE. Finally, with the publication of WCAG 2.1, the automatic tool that trends in use over time is WAVE. Figure 16 shows the trend of Bobby, AChecker, TAW and WAVE over time.

Fig. 16

Trend of the most widely used automatic evaluation tools in the evaluation of the accessibility of university websites over time

Bobby. This tool was used to automatically evaluate the accessibility of university websites with the WCAG 1.0 in papers published between 2002 and 2013;

AChecker. This tool was used to automatically evaluate the accessibility of university websites with the WCAG 1.0 and WCAG 2.0 in papers published between 2011 and 2020;

TAW. This tool was used to automatically evaluate the accessibility of university websites with the WCAG 2.0 in papers published between 2014 and 2019;

WAVE. This tool was used to automatically evaluate the accessibility of university websites with WCAG 2.0 and WCAG 2.1 in papers published between 2015 and 2020.

RQ9. Which experts and users helped assess university websites?

The papers that have used experts or real users for the evaluation of university websites are the following (see Table 18, in “Appendix” A, for full data):

One paper [37] presented an automatic evaluation using Bobby and a manual evaluation using JAWS, Internet Explorer and the help of two experts from university websites;

One paper [42] presented a manual evaluation of university websites using the Internet Explorer 7.0, Web Accessibility Toolbar (WAT), JAWS and the expert knowledge of the authors of the document;

One paper [54] presented an automatic evaluation using CynthiaSays and a manual evaluation with 16 blind users of the websites of the analyzed universities;

One paper [68] presented a heuristic review of university websites using the UX Check Tool and a manual evaluation with two experts in web accessibility;

One paper [76] presented the manual evaluation of the YouTube videos published on university websites with the expert knowledge of the authors of the paper on web accessibility.

RQ10. What are the main errors found on university websites?

The main accessibility errors by checkpoint and priority were identified in the papers that perform the evaluation with WCAG 1.0. The most common errors found by checkpoint in the selected papers are: lack of text alternatives (1.1), missing language (4.3), tables used for layout or with complex structure (5.5) and poor navigation (12.1, 13.1). The highest percentage of errors is in Priority 1 with 40 %, Priority 2 with 35 % and Priority 3 with 25 %. The errors by priority, checkpoint and number of papers can be seen in Table 10 (see Table 19, in “Appendix” A, for full data).

Table 10

Errors by priority, checkpoints and number of papers WCAG 1.0

Priority	Checkpoints	Number of papers
	1.1	12
	1.4	1
Priority 1	4.1	1
	8.1	2
	12.1	2
	14.1	1
	3.1	1
	3.4	1
	4.3	3
	6.4	2
	6.5	1
Priority 2	7.2	1
	7.4	1
	7.5	1
	10.2	2
	12.2	1
	13.1	3
	1.5	1
	5.5	3
	5.6	1
	9.5	1
Priority 3	10.4	1
	10.5	2
	13.5	1
	13.8	1
	13.9	1

Table 19

Data collected for RQ10 and RQ11

Paper	RQ10		RQ11
	Accessibility guidelines	Errors from tables20 and 21	Compliance level
[35]	WCAG 1.0	1.1, 13.1, 13.5, 13.9	N/C
[36]	WCAG 1.0	1.1	N/C
[37]	WCAG 1.0	1.1	N/C
[38]	WCAG 1.0	1.1, 1.5, 3.1, 3.4, 4.3, 5.5, 6.4, 6.5, 7.2, 7.4, 7.5, 10.4, 10.5, 12.1, 12.2, 12.3, 12.4, 13.1	N/C
[39]	WCAG 1.0	1.1	N/C
[40]	WCAG 1.0	1.1	N/C
[41]	WCAG 1.0	1.1, 1.4, 5.5, 9.5, 12.2, 13.1, 13.8	N/C
[42]	WCAG 1.0	1.1, 8.1, 12.1	N/C
[43]	WCAG 1.0	.	N/C
[44]	WCAG 2.0	1.1.1, 1.4.1, 1.4.4, 2.1.1	N/C
[45]	WCAG 1.0	1.1, 4.1, 4.3, 5.6, 10.2	N/C
[46]	WCAG 1.0	1.1, 4.3, 5.5, 6.4, 10.2, 10.5, 13.1	N/C
[47]	WCAG 2.0	1.1.1, 1.3.1, 1.4.4, 2.1.1, 2.1.3, 2.2.1, 2.2.2, 2.2.4, 2.4.2, 2.4.4, 2.4.9, 2.4.10, 3.1.1, 3.2.5, 3.3.2, 4.1.1, 4.1.2	N/C
[48]	WCAG 2.0	1.1.1, 1.3.1, 2.4.2, 2.4.4	N/C
[49]	WCAG 2.0	1.1.1, 1.4.1, 1.4.4, 2.1.1	N/C
[50]	WCAG 2.0	1.1.1, 1.3.1, 1.3.3, 1.4.1, 2.3.1, 2.4.1, 2.4.4, 3.1.1, 3.2.1, 3.2.3, 3.3.2, 3.3.4, 3.3.5, 4.1.1, 4.1.2	N/C
[51]	WCAG 2.0	.	N/C
[52]	WCAG 2.0	1.3.1, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.8, 2.1.1, 2.1.2, 2.1.3, 2.2.1, 2.2.2, 2.2.3, 2.2.4, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 3.1.1, 3.1.2, 3.3.1, 3.3.2, 3.3.3, 3.3.4, 3.3.5, 3.3.6, 4.1.1, 4.1.2	N/C
[53]	WCAG 2.0	.	N/C
[54]	WCAG 2.0	1.3.1	N/C
[55]	WCAG 2.0	1.4.3	N/C
[56]	WCAG 1.0	1.1, 8.1	N/C
[57]	WCAG 1.0	1.1, 5.1, 12.1, 14.1	N/C
[58]	WCAG 2.0	.	N/C
[59]	WCAG 2.0	1.1.1, 1.3.1, 2.4.2, 2.4.4, 3.1.1, 3.2.2, 3.3.2, 4.1.1, 4.1.2	N/C
[60]	WCAG 2.0	1.1.1, 1.3.1, 1.4.1, 1.4.4, 1.4.6, 2.1.1, 2.2.2, 2.4.2, 2.4.4, 2.4.6, 3.1.1, 3.3.2, 4.1.1	N/C
[61]	WCAG 2.0	1.1.1, 2.4.4	N/C
[62]	WCAG 2.0	1.1.1, 1.3.1, 1.4.1, 1.4.4, 1.4.6, 2.1.1, 2.1.3, 2.2.2, 2.4.4, 2.4.6, 2.4.9, 2.4.10, 3.1.1, 3.2.2, 3.3.2, 4.1.1, 4.1.2	N/C
[63]	WCAG 2.0	1.1.1, 1.3.1, 1.3.2, 1.4.3, 2.1.1, 2.2.1, 2.2.2, 2.4.1, 2.4.2, 3.1.1	N/C
[64]	WCAG 2.0	.	N/C
[65]	WCAG 2.0	1.1.1, 1.3.1, 1.4.1, 1.4.3, 1.4.4, 1.4.5, 2.4.6, 2.4.9	N/C
[66]	WCAG 2.0	1.1.1, 1.2.2, 1.2.5, 1.4.1, 2.4.6	N/C
[67]	WCAG 2.0	1.4.1, 1.4.3, 1.4.4	N/C
[68]	WCAG 2.1	1.1.1, 1.3.1, 1.3.2, 1.4.1, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 2.1.1, 2.1.3, 2.2.1, 2.2.2, 2.2.4, 2.4.1, 2.4.10, 3.2.5, 3.3.1, 3.3.2, 3.3.3, 4.1.2	N/C
[69]	WCAG 2.0	1.1.1, 1.3.1, 1.4.3, 2.1.1, 2.2.1, 2.2.2, 2.4.1, 2.4.2, 2.4.4, 2.4.6, 3.1.1, 3.3.2, 4.1.2	N/C
[70]	WCAG 2.0	1.1.1, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.4.7, 2.4.8, 2.4.9, 2.4.10, 3.1.1, 3.1.2, 3.1.3, 3.1.4, 3.1.5, 3.1.6, 3.2.1, 3.2.2, 3.2.3, 3.2.4, 3.2.5, 3.3.1, 3.3.2, 3.3.3, 3.3.4, 3.3.5, 3.3.6, 4.1.1, 4.1.2	N/C
[71]	WCAG 2.0	1.1.1, 1.3.1, 2.4.4, 2.4.7, 3.3.2, 4.1.2	N/C
[72]	WCAG 2.1	1.1.1	N/C
[73]	WCAG 2.1	.	N/C
[74]	WCAG 2.0	1.1.1, 1.3.1, 1.4.1, 1.4.3, 1.4.4, 1.4.6, 2.1.1, 2.1.3, 2.2.1, 2.2.2, 2.4.1, 2.4.4, 2.4.6, 2.4.9, 2.4.10, 3.1.1, 3.2.2, 3.3.2, 4.1.1, 4.1.2	N/C
[75]	WCAG 2.0	1.1.1, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.4.7, 2.4.8, 2.4.9, 2.4.10, 3.1.1, 3.1.2, 3.1.3, 3.1.4, 3.1.5, 3.1.6, 3.2.1, 3.2.2, 3.2.3, 3.2.4, 3.2.5, 4.1.1, 4.1.2	N/C
[76]	WCAG 2.1	1.2.2, 1.2.3, 1.2.5, 1.2.6, 1.2.7, 1.2.8, 1.4.2, 1.4.3, 1.4.4, 2.1.1, 2.2.2	N/C

“N/C” means Non-Compliant. “.....” means that no accessibility errors are present in the paper

In the papers that perform the evaluation with WCAG 2.0, the most common errors have been identified by principle and success criteria. The percentage of errors by principle was Operable with 36 %, Perceivable with 32 %, Understandable with 24 % and Robust with 8 %. The errors by principle, success criteria, conformance levels and number of papers can be seen in Table 11 (see Table 19, in “Appendix” A, for full data).

Table 11

Errors by principle, conformance level, success criteria and number of papers WCAG 2.0

Principle	Success criteria	Conformance level	Number of papers
Perceivable	1.1.1	A	17
	1.2.2	A	1
	1.2.5	AA	1
	1.3.1	A	16
	1.3.2	A	2
	1.3.3	A	3
	1.3.4	AA	2
	1.3.5	AA	2
	1.3.6	AAA	2
	1.4.1	A	10
	1.4.2	A	1
	1.4.3	AA	7
	1.4.4	AA	9
	1.4.5	AA	2
	1.4.6	AAA	4
	2.1.1	A	9
	2.1.2	A	1
	2.1.3	AAA	4
	2.2.1	A	5
	2.2.2	A	7
	2.2.3	AAA	1
	2.2.4	AAA	2
	2.3.1	A	1
Operable	2.4.1	A	7
	2.4.2	A	9
	2.4.3	A	3
	2.4.4	A	13
	2.4.5	AA	3
	2.4.6	AA	9
	2.4.7	AA	3
	2.4.8	AAA	2
	2.4.9	AAA	6
	2.4.10	AAA	5
	3.1.1	A	11
	3.1.2	AA	3
	3.1.3	AAA	2
	3.1.4	AAA	2
	3.1.5	AAA	2
	3.1.6	AAA	2
	3.2.1	A	3
	3.2.2	A	5
Understandable	3.2.3	A	3
	3.2.4	AA	2
	3.2.5	AAA	3
	3.3.1	A	3
	3.3.2	A	10
	3.3.3	AA	2
	3.3.4	AA	3
	3.3.5	AAA	3
	3.3.6	AAA	2
Robust	4.1.1	A	9
	4.1.2	A	10

According to the results, the errors have been classified by guideline. These errors are presented in the content of the websites as follows:In the papers that perform the evaluation with WCAG 2.1, the most common errors have been identified by principle and success criteria. The percentage of errors by principle was: Perceivable with 56 %, Operable with 28 %, Understandable with 13 % and Robust with 3 %. The most common errors classified by guideline are:The errors by principle, success criteria, conformity levels and number of papers can be seen in Table 12 (see Table 19, in “Appendix” A, for full data):

Table 12

Errors by principle, conformance level, success criteria and number of papers WCAG 2.1

Principle	Success criteria	Conformance level	Number of papers
Perceivable	1.1.1	A	2
	1.2.2	A	1
	1.2.3	A	1
	1.2.5	AA	1
	1.2.6	AAA	1
	1.2.7	AAA	1
	1.2.8	AAA	1
	1.3.1	A	1
	1.3.2	A	1
	1.4.1	A	1
	1.4.2	A	1
	1.4.3	AA	2
	1.4.4	AA	2
	1.4.5	AA	1
	1.4.6	AAA	1
	2.1.1	A	2
	2.1.3	AAA	1
	2.2.1	A	1
Operable	2.2.2	A	2
	2.2.4	AAA	1
	2.4.1	A	1
	2.4.10	AAA	1
	3.2.5	AAA	1
Understandable	3.3.1	A	1
	3.3.2	A	1
	3.3.3	AA	1
Robust	4.1.2	A	1

Guideline 1.1—Text alternatives Non-text content has no alternative text;

Guideline 1.3—Adaptable The content is not presented in different ways on the website without losing its structure and information;

Guideline 1.4—Distinguishable The content cannot be seen and heard;

Guideline 2.1—Keyboard accessibility Websites do not allow interaction using the keyboard;

Guideline 2.4—Navigable The website does not include help that allows the user to find the content;

Guideline 3.1—Readable The content of the websites is not clear, concrete and concise;

Guideline 3.2—Predictable Websites do not function in a predictable manner;

Guideline 3.3—Input Assistance The websites do not have a guide for filling in the information on the forms;

Guideline 4.1—Compatible The websites do not allow the use of assistive technology tools.

Guideline 1.1—Text alternatives Non-text content has no alternative text;

Guideline 1.2—Time-based Media There are no time-based alternatives in the media;

Guideline 1.3—Adaptable The content is not presented in different ways on the website without losing its structure and information;

Guideline 1.4—Distinguishable The content cannot be seen and heard;

Guideline 2.1—Keyboard accessibility Websites do not allow interaction using the keyboard;

Guideline 2.2—Enough Time Users are not given enough time to use and read the content;

Guideline 2.4—Navigable The website does not include help that allows the user to find the content;

Guideline 3.2—Predictable Websites do not function in a predictable manner;

Guideline 3.3—Input Assistance The websites do not have a guide for filling in the information on the forms;

Guideline 4.1—Compatible The websites do not allow the use of assistive technology tools.

RQ11. What is the accessibility compliant status of university websites?

The websites of 9,140 universities have been evaluated on the selected papers using automatic and manual methods. However, none of the websites of the universities that were analyzed were found to be compliant with web accessibility standards or with the WCAG (see Table 19, in “Appendix” A, for full data).

Discussion

The discussion is divided into two parts. In the first part, a bibliometric analysis of the selected papers is made. In the second part, we discuss the results found in the analyzed papers.

The bibliometric analysis begins by determining the trend of publication of research results on the evaluation of the accessibility of university websites over time. As important data, an increase has been observed in recent years, from 2016 to 2020, in the research of web accessibility. This trend can be seen in Fig. 17 (see Table 14, in “Appendix” A, for full data).

Fig. 17

Number of papers published per year

Table 14

Bibliometric data extraction

Paper	Source name	JCR	SJR	Year	Month
[35]	Interacting with Computers (IC)	Q2	Q2	2002	July
[36]	Reference and User Services Quarterly (RUSQ)	Q2	Q1	2002	December
[37]	Information Technology and Disabilities (ITD)	✗	Q4	2003	December
[38]	Internet Research	✗	Q2	2005	July
[39]	Library Hi Tech	✗	Q2	2007	July
[40]	Journal of Special Education Technology (JSET)	✗	Q2	2007	December
[41]	Internet and Higher Education (IHE)	✗	Q1	2008	June
[42]	Disability and Rehabilitation: Assistive Technology (DRAT)	✗	Q2	2010	March
[43]	Disability and rehabilitation (DR)	Q1	Q1	2011	March
[44]	Universal Access in the Information Society (UAIS)	✗	Q3	2011	March
[45]	Information Technology and Disabilities (ITD)	✗	Q4	2013	April
[46]	Universal Access in the Information Society (UAIS)	Q4	Q2	2013	June
[47]	International Education Studies (IES)	✗	Q3	2014	May
[48]	International Journal of Emerging Technologies in Learning (IJET)	✗	Q3	2015	August
[49]	Universal Access in the Information Society (UAIS)	Q3	Q2	2016	April
[50]	Journal of King Saud University - Computer and Information Sciences (JKSUCIS)	✗	Q2	2016	June
[51]	International Journal of Advanced Computer Science and Applications (IJACSA)	✗	Q4	2016	July
[52]	Journal of Information and Communication Technology-Malaysia (JICT)	✗	Q3	2016	December
[53]	International Journal of Online Pedagogy and Course Design (IJOPCD)	✗	✗	2017	January
[54]	Journal of Information and Communication Technology (JICT)	✗	Q2	2017	June
[55]	Procedia Computer Science (PCS)	✗	✗	2017	August
[56]	Universal Access in the Information Society (UAIS)	Q3	Q2	2017	November
[57]	Journal of Computing in Higher Education (JCHE)	Q1	Q1	2017	December
[58]	International Journal of High Performance Computing and Networking (IJHPCN)	✗	Q2	2018	January
[59]	IOP Conference Series: Materials Science and Engineering (IOP)	✗	✗	2018	February
[60]	Universal Access in the Information Society (UAIS)	Q4	Q2	2018	May
[61]	IEEE Access	Q1	Q1	2018	June
[62]	Universal Access in the Information Society (UAIS)	Q4	Q2	2018	August
[63]	Universal Access in the Information Society (UAIS)	Q4	Q2	2018	August
[64]	Journal of Accessibility and Design for All (JADA)	✗	Q4	2018	November
[65]	Universal Access in the Information Society (UAIS)	Q2	Q2	2019	April
[66]	Journal of King Saud University - Computer and Information Sciences (JKSUCIS)	✗	Q2	2019	April
[67]	TechTrends	✗	Q2	2019	August
[68]	IEEE Access	Q1	Q1	2019	September
[69]	International Journal of Computer Science and Network Security (IJCSNS)	✗	✗	2019	December
[70]	Universal Access in the Information Society (UAIS)	Q2	Q2	2019	December
[71]	Information (Switzerland) (IS)	✗	Q3	2020	January
[72]	Data in Brief (DB)	✗	Q4	2020	February
[73]	Universal Access in the Information Society (UAIS)	Q2	Q2	2020	February
[74]	Universal Access in the Information Society (UAIS)	Q2	Q2	2020	April
[75]	Universal Access in the Information Society (UAIS)	Q2	Q2	2020	April
[76]	IEEE Access	Q1	Q1	2020	June

“✗” means the journal is not indexed in SJR or JCR

The papers analyzed in this SLR have been published in 27 journals. In Tables 7 or 14, we can consult the name and the acronym of each journal. The UAIS is the journal with the largest number of papers, which is considered normal, since the UAIS “addresses the accessibility, usability, and, ultimately, acceptability of Information Society Technologies by anyone, anywhere, at anytime, and through any media and device” [80]. Figure 18 shows the number of papers published per journal (see Table 14, in “Appendix” A, for full data).

Fig. 18

Number of papers published per journal

The indexation of the journals where the papers have been published is detailed below (see Table 14, in “Appendix” A, for full data):Figure 19 shows graphically the number of papers published in journals indexed in SJR and JCR. The ranking quartiles of the journals indexed in SJR and JCR were determined by the year of publication of the papers. Papers published in 2020 were assigned the 2019 quartiles. Journals that do not have a quartile have been placed in the figure as QX (see Table 14, in “Appendix” A, for full data).

Fig. 19

Number of papers published in SJR and JCR

18 papers in SJR and JCR;

20 papers in SJR;

4 papers are not indexed in SJR and JCR.

After the data were extracted from the selected papers, they were analyzed and interpreted in the results of this SLR. With these data, it has been possible to answer the following questions: Considering that the selected papers come from different points in time, a time trend of their results has been made. Of the 42 selected papers, 11 were conducted in the USA. Therefore, we have made a timeline of the accessibility results in US university websites (shown in Fig. 20), analyzing the results over time. It can be seen that accessibility problems persist over time. This may be due to the new versions of the WCAG, which are becoming more and more comprehensive and evolving over time. In addition, it is quite difficult to compare the results from different studies because there does not exist a baseline and each study applies different methods and tools.

Fig. 20

Results over time of the evaluation of university websites in the USA

What was evaluated?

In answering this question, we determined that the web pages evaluated on the university websites are: university homepage, search page, list of university colleges, departments, and/or degree programs, campus directory of faculty, admissions homepage, staff and/or students, course listings, employment homepage, academic calendar, job listings, campus map and library homepage. In addition, PDF documents and YouTube videos published on university websites have been evaluated.

What standards were used for the evaluation?

The web accessibility standards used in the evaluation of university websites are: WCAG 1.0, WCAG 2.0 (ISO/IEC 40500:2012), WCAG 2.1 and Section 508.

What methods were used for the evaluation?

The evaluation methods used in the selected papers are: manual evaluation, evaluation with automatic tools and the combination of both methods.

What results were obtained?

The results showed that the web pages, PDF documents and YouTube videos evaluated in the selected papers were not accessible when they were evaluated.

We also analyzed the results over time of the evaluation of university websites in countries with more than one article among those selected. Table 13 presents the results. In these papers, accessibility problems also persist over time.

Table 13

Results over time of the evaluation of university websites

Continent	Country	Year	Paper	Results
America	Antigua Barbuda, Argentina, Aruba, Bolivia, Brazil, Chile, Colombia, Costa Rica, Cuba, Dominica, Ecuador, El Salvador,	2018	[61]	The results showed that the universities’ websites have frequent problems related to the lack of alternative image text. It was found that the university websites included in the present study violate web accessibility requirements based on the WCAG 2.0.
	Guatemala, Haiti, Honduras, Jamaica, Mexico, Nicaragua, Panama, Paraguay, Peru, Puerto Rico, Dominican Republic, Trinidad and Tobago, Uruguay, Venezuela	2020	[72]	The results showed that university websites have frequent problems related to the lack of alternative text linked to images. Some of the university websites included in this dataset were found to violate web accessibility requirements based on the WCAG 2.0 and 2.1.
Asia	India	2016	[50]	The results of the statistical classification and accessibility report of the websites showed that a number of improvements are needed to make them more accessible and usable in terms of WCAG 2.0.
		2019	[66]	The results indicated the main accessibility barriers exposed by these websites in terms of metrics such as number of problems, warnings and success criteria violation status.
Asia	Jordan	2015	[48]	The results showed that accessibility errors of universities websites in Jordan, and Arab region exceed the ones in UK by 13 times, and 5 times consequently (comparison).
		2016	[51]	The results showed a significant number of weaknesses in most of the universities. Furthermore, a variation of web accessibility standards was found when the websites were measured using different accessibility tools.
		2018	[58]	The results indicated a fairly strong positive correlation between the different accessibility metrics, while a fairly weak negative correlation is observed between one of the accessibility metrics and Alexa ranks; a fairly strong positive correlation is also observed between Alexa ranks and Webometrics university rankings.
Asia	Turkey	2011	[44]	The results indicated that all university homepages show some accessibility problems.
		2016	[49]	The goal of the audit was to determine whether the accessibility of the website has increased or improved over the past five-year period. The results of the second study showed that, overall, accessibility levels have decreased slightly.
		2017	[55]	The results indicated that none of the websites evaluated are error-free and most of them do not reach an acceptable level of web accessibility compliance.
		2020	[74]	The results showed that 110 state university websites and 69 private university websites, only 10 state university websites and 4 private university websites achieved conformance level A. The majority did not meet the WCAG 2.0 accessibility criteria.
Europe	Portugal	2011	[43]	The main results of this study were as follows: schools overall accessibility of the webpage were not acceptable and only 12.5 % sites had information regarding supporting services to disabled students. The data collected identified an overall web accessibility improvement during the 2007-2008, followed by a stabilization of the results.
		2019	[65]	The results showed that the main violations occur in image contrasts, image alt text and buttons, and links without visible text, which mostly represent a lack of accessibility components. In addition, we recommend providing alternative text for images, as well as other explanatory text.

The website of a university is the most common introduction for many people interested in the university. This is why universities publish their academic offers, regulations, university projection, achievements, among other things on their websites, in order to make their services and their work as a university known to the community. However, for university websites to reach a wider audience, they must comply with web accessibility laws and regulations. Unfortunately, according to the results there seems to be a lack of knowledge or interest in the laws that should be applied, because in many countries policies and laws on web accessibility already exist [78].

Universities have always been at the forefront of technological change. However, according to the papers analyzed, it can be seen that there are isolated efforts in some universities to incorporate the WCAG into their websites.

Also, as an interesting fact, we have been able to verify that only papers [45, 76] have dealt with the evaluation of other resources that exist on the web pages, and the rest have only focused on the web pages. In order for university websites to comply with WCAG, accessibility laws or standards, their web pages and their content must be accessible.

It should be noted that university websites do not appear to be created with web accessibility in mind. This may be due to the lack of knowledge of the website developers. However, universities have a duty to make their websites accessible, to provide universal access to their content. Article 9 of the CRPD [1] stipulates that governments should “promote access for persons with disabilities to new information and communications technologies and systems, including the Internet” and “promote the design, development, production and distribution of accessible information and communications technologies and systems at an early stage” . To this end, it is necessary that the developers of university websites are trained and apply the WCAG in their web pages and in the content that is published.

Limitations of the study

An SLR can be affected by a series of limitations. One of these is bias in data collection by the authors. For this reason, this SLR starts by defining customized search strings with the key terms of this research. These search strings were applied in the extraction of data from each of the scientific databases. In addition, to reduce this bias, we have applied inclusion, exclusion and quality criteria in the selection of the papers. It is worth mentioning that all the authors were involved in the whole SLR process to give more strength to their results.

Another limitation found is that the authors of some of the selected papers did not contrast the errors found with the WCAG success criteria. Therefore, the authors of this SLR contrasted these errors with the WCAG success criteria under their knowledge. In addition, some papers do not detail the errors found in the evaluation.

Also, this SLR does not include papers not published in journals such as conferences, editorials, prefaces, discussion, comments, tutorial summaries, workshop summaries and panels. Therefore, “grey” literature is not considered in this SLR.

Another limitation is time, as WCAG has been evolving from WCAG 1.0 [14], WCAG 2.0 [15], WCAG 2.1 [7], to draft WCAG 2.2 [16] and first public working draft WCAG 3.0 [81], and each of them seeks a wider scope.

Conclusions and future work

The aim of this SLR was to examine the accessibility of university websites as reported in 42 selected papers. The 42 papers analyzed evaluate 9,140 universities distributed in 5 continents and 68 countries. In summary, the results obtained are the following:The results allow us to conclude that university websites show some similar types of violations over time with the WCAG success criteria. Another important finding is that we see a change from WCAG 1.0 to 2.0 and then the recent appearance of 2.1, its evolution in the use for the evaluation of the accessibility of university websites in the world. Also, the W3C lists 154 tools [20] for the evaluation of web accessibility in an automatic way, while 21 tools have been used in the papers analyzed in this SLR. It can be observed that there is a great concentration in a few tools (AChecker, Bobby, TAW and WAVE) that are the preferred ones, although there are many more.

83.33 % of the universities were selected through a sampling;

38,416 web pages, 91,421 YouTube Videos and 28,395 PDF documents were evaluated;

88.10 % of the papers only evaluated the homepage of the websites;

The standards used were ISO/IEC40500:2012, Section 508 and the combination of both;

The evaluation methods used were manual methods, methods with automatic tools and the combination of the two;

90.47 % of the university websites were evaluated with automatic tools;

The most common errors are presented in the following guidelines: Text Alternatives, Adaptive, Distinguishable, Keyboard Accessibility, Navigable, Readable, Predictable, Input Assistance and Compatible.

The global education system has been disrupted by the SARS-COV-2 pandemic. Most institutions had to adapt to online education when their facilities closed. Teachers, students and administrative staff had to go through this transition, but for many people with and without disabilities the situation became very difficult. For this reason, the technological tools for teaching–learning have been adapted according to the needs of the virtual world and the experience of teachers, without measuring their barriers. In order to reduce these barriers, it is necessary to evaluate the acceptance and benefits of technological tools before their application. In addition, the websites of educational institutions must comply with accessibility standards in their portals. Universal access on the web benefits people with and without disabilities.

The developers of university websites must take into account the concept of accessibility from the beginning when creating or redesigning a website. In addition, there are resources that can help achieve accessibility of a website such as templates, plugins, libraries and others. Also, websites in their development phase should be tested with real users and experts.

In addition, this work offers a review of accessibility on university websites that may be useful for others to understand this concept. In future work, a multivocal literature review [82] will be carried out on the accessibility of the contents published on university websites. In addition, it will be reviewed if there are software development architectures that implement accessibility in a website. Also, as future work, it is proposed to make a comparison of the accessibility of public and private university websites and the acceptance of Learning Management Systems (LMS) by people with disabilities.

Table 20

Errors per priorities and checkpoints WCAG 1.0

Priorities and checkpoints WCAG 1.0 [14]	Papers
Priority 1
1.1 Provide a text equivalent for every non-text element (e.g., via “alt,” “longdesc” or in element content). This includes: images, graphical representations of text (including symbols), image map regions, animations (e.g., animated GIFs), applets and programmatic objects, ascii art, frames, scripts, images used as list bullets, spacers, graphical buttons, sounds (played with or without user interaction), stand-alone audio files, audio tracks of video, and video.	[35–42, 45, 46, 56, 57]
1.4 For any time-based multimedia presentation (e.g., a movie or animation), synchronize equivalent alternatives (e.g., captions or auditory descriptions of the visual track) with the presentation.	[41]
4.1 Clearly identify changes in the natural language of a document’s text and any text equivalents (e.g., captions).	[45]
5.1 For data tables, identify row and column headers.	[57]
8.1 Make programmatic elements such as scripts and applets directly accessible or compatible with assistive technologies.	[42, 56]
12.1 Title each frame to facilitate frame identification and navigation.	[38, 42, 57]
14.1 Use the clearest and simplest language appropriate for a site’s content.	[57]
Priority 2
3.1 When an appropriate markup language exists, use markup rather than images to convey information.	[38]
3.4 Use relative rather than absolute units in markup language attribute values and style sheet property values.	[38]
6.4 For scripts and applets, ensure that event handlers are input device-independent.	[38, 46]
6.5 Ensure that dynamic content is accessible or provide an alternative presentation or page.	[38]
7.2 Until user agents allow users to control blinking, avoid causing content to blink (i.e., change presentation at a regular rate, such as turning on and off).	[38]
7.4 Until user agents provide the ability to stop the refresh, do not create periodically auto-refreshing pages.	[38]
10.2 Until user agents support explicit associations between labels and form controls, for all form controls with implicitly associated labels, ensure that the label is properly positioned.	[45, 46]
12.2 Describe the purpose of frames and how frames relate to each other if it is not obvious by frame titles alone.	[38, 41]
12.3 Divide large blocks of information into more manageable groups where natural and appropriate.	[38]
12.4 Associate labels explicitly with their controls.	[38]
13.1 Clearly identify the target of each link.	[35, 38, 41, 46]
Priority 3
1.5 Until user agents render text equivalents for client-side image map links, provide redundant text links for each active region of a client-side image map.	[38]
4.3 Identify the primary natural language of a document.	[38, 45, 46]
5.5 Provide summaries for tables.	[38, 41, 46]
5.6 Provide abbreviations for header labels.	[45]
9.5 Provide keyboard shortcuts to important links (including those in client-side image maps), form controls, and groups of form controls.	[41]
10.4 Until user agents handle empty controls correctly, include default, place-holding characters in edit boxes and text areas.	[38]
10.5 Until user agents (including assistive technologies) render adjacent links distinctly, include non-link, printable characters (surrounded by spaces) between adjacent links.	[38, 46]
13.5 Provide navigation bars to highlight and give access to the navigation mechanism.	[35]
13.8 Place distinguishing information at the beginning of headings, paragraphs, lists, etc.	[41]
13.9 Provide information about document collections (i.e., documents comprising multiple pages.).	[35]

Table 21

Errors per principles, guidelines and success criteria WCAG 2.1

Principles, guidelines and success criteria WCAG 2.1[17]	Level	Papers
1 Principle: Perceivable
Guideline 1.1 Text Alternatives
1.1.1 Non-text Content	A	[44, 47–50, 59–63, 65, 66, 68–72, 74, 75]
Guideline 1.2 Time-based Media: Provide alternatives for time-based media
1.2.2 Captions (Prerecorded)	A	[66, 76]
1.2.3 Audio Description or Media Alternative (Prerecorded)	A	[76]
1.2.5 Audio Description (Prerecorded)	AA	[66, 76]
1.2.6 Sign Language (Prerecorded)	AAA	[76]
1.2.7 Extended Audio Description (Prerecorded)	AAA	[76]
1.2.8 Media Alternative (Prerecorded)	AAA	[76]
Guideline 1.3 Adaptable: Create content that can be presented in different ways (for example simpler layout) without losing information or structure
1.3.1 Info and Relationship	A	[47, 48, 50, 52, 54, 59, 60, 62, 63, 65, 68–71, 74, 75]
1.3.2 Meaningful Sequence	A	[63, 68, 70, 75]
1.3.3 Sensory Characteristics	A	[50, 70, 75]
1.3.4 Orientation	AA	[70, 75]
1.3.5 Identify Input Purpose	AA	[70, 75]
1.3.6 Identify Purpose	AAA	[70, 75]
Guideline 1.4 Distinguishable: Make it easier for users to see and hear content including separating foreground from background
1.4.1 Use of Color	A	[44, 49, 50, 52, 60, 62, 65–68, 74]
1.4.2 Audio Control	A	[52, 76]
1.4.3 Contrast (Minimum)	AA	[52, 55, 63, 65, 67–69, 74, 76]
1.4.4 Resize Text	AA	[44, 47, 49, 52, 60, 62, 65, 67, 68, 74, 76]
1.4.5 Images of Text	AA	[52, 65, 68]
1.4.6 Contrast (Enhanced)	AAA	[52, 60, 62, 68, 74]
1.4.8 Visual Presentation	AAA	[52]
2 Principle: Operable
Guideline 2.1 Keyboard Accessible: Make all functionality available from a keyboard
2.1.1 Keyboard	A	[44, 47, 49, 52, 60, 62, 63, 68, 69, 74, 76]
2.1.2 No Keyboard Trap	A	[52]
2.1.3 Keyboard (No Exception)	AAA	[47, 52, 62, 68, 74]
Guideline 2.2 Enough Time: Provide users enough time to read and use content
2.2.1 Timing Adjustable	A	[47, 52, 63, 68, 69, 74]
2.2.2 Pause, Stop, Hide	A	[47, 52, 60, 62, 63, 68, 69, 74, 76]
2.2.3 No Timing	AAA	[52]
2.2.4 Interruptions	AAA	[47, 52, 68]
Guideline 2.3 Seizures: Do not design content in a way that is known to cause seizures
2.3.1 Three Flashes or Below Threshold	A	[50]
Guideline 2.4 Navigable: Provide ways to help users navigate, find content and determine where they are
2.4.1 Bypass Blocks	A	[50, 52, 63, 68–70, 74, 75]
2.4.2 Page Titled	A	[47, 48, 52, 59, 60, 63, 69, 70, 75]
2.4.3 Focus Order	A	[52, 70, 75]
2.4.4 Link Purpose (In Context)	A	[47, 48, 50, 52, 59–62, 69–71, 74, 75]
2.4.5 Multiple Ways	AA	[52, 70, 75]
2.4.6 Headings and Labels	AA	[52, 60, 62, 65, 66, 69, 70, 74, 75]
2.4.7 Focus Visible	AA	[70, 71, 75]
2.4.8 Location	AAA	[70, 75]
2.4.9 Link Purpose (Link only)	AAA	[47, 62, 65, 70, 74, 75]
2.4.10 Section Headings	AAA	[47, 62, 68, 70, 74, 75]
3 Principle: Understandable
Guideline 3.1 Readable: Make text content readable and understandable
3.1.1 Language of Page	A	[47, 50, 52, 59, 60, 62, 63, 69, 70, 74, 75]
3.1.2 Language of Parts	AA	[52, 70, 75]
3.1.3 Unusual Words	AAA	[70, 75]
3.1.4 Abbreviations	AAA	[70, 75]
3.1.5 Reading Level	AAA	[70, 75]
3.1.6 Pronunciation	AAA	[70, 75]
Guideline 3.2 Predictable: Make web pages appear and operate in predictable ways
3.2.1 On Focus	A	[50, 70, 75]
3.2.2 On Input	A	[59, 62, 70, 74, 75]
3.2.3 Consistent Navigation	A	[50, 70, 75]
3.2.4 Consistent Identification	AA	[70, 75]
3.2.5 Change on Request	AAA	[47, 68, 70, 75]
Guideline 3.3 Input Assistance: Help users avoid and correct mistakes
3.3.1 Error Identification	A	[52, 68, 70, 74]
3.3.2 Labels or Instructions	A	[47, 50, 52, 59, 60, 62, 68–71, 74]
3.3.3 Error Suggestion	AA	[52, 68, 70]
3.3.4 Error Prevention (Legal, Financial, Data)	AA	[50, 52, 70]
3.3.5 Help	AAA	[50, 52, 70]
3.3.6 Error Prevention (All)	AAA	[52, 70]
4 Principle: Robust
Guideline 4.1 Compatible: Maximize compatibility with current and future user agents, including assistive technologies
4.1.1 Parsing	A	[47, 50, 52, 59, 60, 62, 70, 74, 75]
4.1.2 Name, Role, Value	A	[47, 50, 52, 59, 62, 68–71, 74, 75]