Accessibility evaluation of university hospital websites in Turkey.

Hospital web pages serve as an interactive communication resource to meet the needs of patients, healthcare professionals and other stakeholders. The aim of this study is to present the accessibility analysis of 58 university hospital websites in Turkey. For this purpose, websites of the selected university hospitals were analyzed using two different online automated testing tools. The results showed that university hospital websites in Turkey had low compliance levels according to the WCAG 2.0 guidelines. Most of the websites did not even meet the minimum requirements for compliance level A. In addition, almost all of the websites had broken links and about a third of the websites had problems with accessing from mobile devices. Moreover, some important clues that draw attention to the accessibility problems of websites are also discussed in this study. Hence, the results of this study provide feedback to developers and administrators to improve the accessibility of these websites.

Introduction

Up-to-date statistics on Internet usage for 2021 show that European countries have the highest Internet penetration rates in the population after North America with 87.1% [1]. This impressive growth of Information and Communication Technology (ICT) has also made significant advances in the health sector. The Internet has become an important source of information for patients [2]. The Internet and especially hospital websites are at the top of the reference sources for health information in the world [3]. In this context, hospital websites serve as an important source of information for patients and all corresponding stakeholders. Hospital websites mainly provide an interactive service for patients, their families, citizens and doctors. Using hospital websites, stakeholders can access the health services offered by these institutions, access treatment methods, make online appointments, and access doctors' information. The COVID-19 pandemic has clearly shown that the successful survival of healthcare institutions is highly related to the positive perception created in the minds of their target audiences (patients, patient relatives, employees, suppliers, etc.) [4]. Studies indicate that the well-designed and easy-to-use websites of healthcare institutions make a positive impression on patients [4].

World Health Organization (WHO) data show that 15% of the world population lives with some kind of disability and this number is increasing with the aging of the population [5]. In 2006, the United Nations Assembly signed the Convention on the Rights of Persons with Disabilities, which aims to support the rights of the disabled [6]. This contract covers many areas such as Internet access, education and health. Therefore, equal access to information by all individuals constitutes the basis of universality. To ensure the universality of websites, the World Wide Web Consortium (W3C) has proposed various guidelines such as the Web Content Accessibility Guidelines (WCAG 1.0 and 2.0). Based on these guidelines, it is possible to analyze how the product or service meets the needs of users with disabilities.

With the widespread use of websites, studies on website quality evaluation have greatly increased in various application areas, like government, education, and health. The main purposes of these studies are to conduct tests on websites using different methods such as expert assessment, automated tools, or user experience, and to evaluate them in terms of accessibility, usability, and security. A tool-based evaluation is one of the most frequently used methods for this purpose since it can be easily employed and supports manual testing.

Accessible hospital websites should be developed considering the skills, characteristics, educational background, and disabilities of people. In this context, the main purpose of this study is to examine the websites of university hospitals in Turkey in terms of accessibility. This study obtains findings about the quality of service offered to users of hospital web pages according to accessibility criteria and tries to eliminate the gaps in the literature. The findings of this study provide suggestions for hospitals to increase their quality of websites.

This paper is organized as follows: in the following section, we present related studies in the literature. In the third section, we describe the methodology used to evaluate the accessibility of hospital websites. In the fourth section, the evaluation results are presented. The fifth section discusses the analysis of the results, provides useful suggestions, and addresses limitations of the study. In the last section, conclusions are drawn and directions are provided for further research.

Literature review

Accessibility evaluations of hospital websites are presented examined in a number of studies through using different criteria. In these studies, different techniques (e.g., expert assessment, user experience, and automated tools) were employed.

Of the pioneers in this field, Zaphiris and Kurniawan [7], analyzed different health-related websites (governmental, organizational, educational, and commercial) for accessibility and usability with the automated testing tools Bobby and LIFT. While government websites have the highest compliance with the Website Content Accessibility Guidelines, it has been found that accessibility approval is not only related to usability ratings on commercial websites. In following years, BobbyTM verification software was used in a study in which the compliance levels of healthcare websites in Canada was analyzed according to the WCAG 1.0 guidelines [8]. The results showed that the evaluated web pages contained errors according to the WCAG 1.0 Priority 1 level. Mancini et al. [9] found that more than 75 percent of evaluated healthcare websites do not meet the World Wide Web Consortium (W3C) requirements. Moreover, this study compared Spanish, American and British hospital websites in terms of readability and accessibility, analysis and contributed significantly to the regulation of hospital website accessibility in Italy." Website readability evaluation was performed according to the Flesch Index, while web accessibility evaluation was achieved with a TAW automated tool. The analysis showed that the readability indexes tended to be below 60 (standard readability) and only 10 of the 32 websites met the accessibility criteria [10].

A study carried out by Brobst [11] evaluated 20 healthcare websites in the USA with both automated testing tool and expert evaluation. The results of the analysis revealed that two-thirds of the evaluated websites do not provide sufficient accessibility. Kuzma et al. [12] tested the accessibility of 160 hospital websites in 16 countries. The evaluation was made using the TAW tool and it was revealed that there were many accessibility problems in the evaluated websites. While the highest number of accessibility problems were seen on hospital websites in Asia, only 2 of all hospital websites evaluated were Level A compliant. Furthermore, Salarvand et al. [13] examined the quality of the websites of public hospitals in Tehran, Iran. Websites were tested with 3 instruments: a localized checklist of 112 items divided into five sections, Google page rank and Alexa traffic rank. The results of this cross-sectional analysis showed that there was no statistically significant relationship between website evaluation scores with Google page rank, Alexa global traffic rank, and Alexa traffic rank in Iran. All the reviewed websites were in the weak category based on their quality score. In another study conducted in Iran [14], an analysis was performed according to the webometric rankings of 93 hospitals. As a result of the analysis, it was determined that there are important relationships among the size, visibility, and richness of hospital websites. Martins et al. [15] tested the accessibility of health care websites in Spain and Portugal using the AccessWeb software tool. The findings of the study show that none of the websites evaluated were compatible with WCAG 2.0. Also, there was no difference between the accessibility errors of the websites in Spain and Portugal. Kaur et al. [16] evaluated 280 hospital websites in India in terms of accessibility using the TAW tool. The evaluation results showed that the overall compliance of the websites with WCAG 2.0 was significantly lower. Further, Acosta-Vargas et al. [17] performed an accessibility analysis of 22 hospital websites according to Webometrics ranking. Websites were tested for ease of access according to WCAG 2.0 standards using the Web Accessibility Assessment Tool (WAVE) and Tenon automated testing tools. The most neglected control points in the findings are non-textual content, information, link purpose, name, function, and value. In a recent study conducted in Turkey [4], 252 public and private hospitals web sites were analyzed by using software testing tools. It was found that private hospitals received the highest score among the sites evaluated according to 22 different criteria. The findings show that the majority of hospital websites are technologically well-designed, but the websites are not up-to-date.

In a more recent study, Yi [18] conducted an automated and expert review analysis of the accessibility of Korean government healthcare websites. The most common errors reported in the study were keyboard operation errors, focus order errors, jump block errors, page header errors, and the absence of proper link text. In another study, Król and Zdonek examined the quality of the websites of 91 infectious disease hospitals in Poland in their study [19]. This work consisted of 3 stages: testing with automated tools, statistical analysis, and content analysis. Web site performance was examined with Google PageSpeed Insights (PSI), SEO feature values were tested with the Blink Audit Tool, accessibility was verified with the WAVE Web Accessibility Evaluation Tool, and mobile friendliness was tested with the Mobile Compatibility Test (Batch Test Tool). As a result of the analysis, it was determined that many hospital websites use a content management system. The findings show that website performance optimization and search engine optimization issues were neglected.

With the increase in digital health services, it has become important to provide users with an inclusive service. This study explains the methods of the analysis for the accessibility of university hospitals websites in Turkey. Then, we identify accessibility problems and suggest some changes that the developers of the hospital websites can make.

Methodology

Data

This study provides an accessibility analysis of the websites of university hospitals in Turkey. In this context, the list of universities having a medical school was obtained from the Turkish Higher Education Program Atlas [20]. Accordingly, a total of 110 hospitals were identified for evaluation. The websites of this sample were initially investigated based on the following issues:

30 university hospitals that do not have an own website, such as Agri Ibrahim Cecen University Training and Research Hospital, were excluded from the study;

20 university hospitals using a common template belonging to the Ministry of Health of the Republic of Turkey (such as Bakircay University Cigli Training and Research Hospital) were not included in the study;

It was determined that there are hospitals with different names, serving in different places such as Baskent University and Medipol University. The website belonging to the location accepted as the center was included in the study, since the hospitals with the specified conditions are connected to a single center, each has its own website, and these websites are managed through the institution they are affiliated with.

After considering the above issues and excluding a set of university hospitals, the remaining 58 web pages were therefore evaluated as reported in this research. The Appendix lists the websites selected for evaluation.

Tools and techniques

The TAW tool, which is frequently used in different studies and proven to be effective, was used in the accessibility evaluation of hospital websites [21]. TAW is an online tool that analyzes website accessibility, available at http://www.tawdis.net/. This tool allows automatic testing of web accessibility according to WCAG 2.0 guidelines and provides users with a report as a result of the analysis. The report includes a brief summary of problems, warnings and controls that require manual testing as presented in Fig. 1, as well as a detailed report. The TAW tool, used by many researchers such as Karaim and Inal [22] and Ochoa and Crovi [23], has been used and validated in accessibility tests of websites in different domains.

Fig. 1

The user interface of TAW tool

The Dead Link Checker tool was used to test broken links that negatively affect user experience and damage SEO values [24]. The tool was previously used by Bilal et al. [25] to evaluate broken links for government website. Another important criterion affecting web accessibility is mobile-friendliness. For this purpose, the Google Mobile-Friendly Test was used to evaluate website responsiveness on mobile devices [26]. This tool was previously used by Verkijika [27] to evaluate South African university websites.

Results

Accessibility analysis

Website accessibility is important in ensuring that websites are accessible and usable for everyone. In this section, we present the accessibility results of university hospitals analyzed using the TAW automated online testing tool. Using TAW, after analyzing 56 university hospital websites (2 websites could not be tested due to time out problems), accessibility issues were identified that did not meet the requirements for level AAA conformance. The distribution of accessibility errors according to four principles, namely perceivable, operable, understandable, and robust, is presented in Fig. 2. Looking at the results, the principles with the most errors were perceivable and operable, accounting for two-thirds of those errors. The understandable principle had the least error rate with 3%.

Fig. 2

Distribution of accessibility errors according to four basic principles

Figure 3 shows the distribution of accessibility errors by university type. Nearly two-thirds of the accessibility errors of the perceivable principle belonged to private university hospitals. Likewise, more than two-thirds of the errors of the understandable principle belonged to private university hospitals. On the other hand, with similar rates, the websites of state university hospitals had the highest number of errors for the operable and robust.

Fig. 3

Distribution of errors according to principles by university type

Table 1 provides an overview of the distribution of accessibility errors by success criteria. The “overall” column shows the total number of errors detected for that success criterion and the percentage of websites where the success criterion has been violated. On the other hand, in the “university type” column, the percentage of websites that violate the success criteria according to state and private university hospitals, and the average number of errors are indicated.

Table 1

Distribution of success criteria violations

Success criteria	Level	Overall		University type
				State(n = 41)		Private(n = 15)
		%	Total	%	Avg	%	Avg
1.1.1—Non-text Content	A	96.43	1405	95.12	16.66	100	48.13
1.3.1—Info and Relationships	A	98.21	1216	97.56	9.27	100	55.73
2.1.3—Keyboard (No Exception)	AAA	30.36	82	26.83	1.83	40	0.47
2.2.1—Timing Adjustable	A	1.79	1	2.44	0.02	0	0
2.2.2—Pause, Stop, Hide	AAA	3.57	2	2.44	0.02	6.67	0.07
2.2.4—Interruptions	AAA	3.57	2	4.88	0.05	0	0
2.4.2—Page Titled	A	3.57	2	2.44	0.02	6.67	0.07
2.4.4—Link Purpose (In Context)	A	98.21	1470	97.56	21.78	100	38.47
2.4.9—Link Purpose (Link Only)	AAA	71.43	640	65.85	9.05	86.67	17.93
2.4.10—Section Headings	AAA	94.64	469	95.12	6.68	93.33	13.00
3.1.1—Language of Page	A	41.07	23	46.34	0.46	26.67	0.27
3.2.2—On Input	A	33.93	21	29.27	0.34	46.67	0.47
3.3.2—Labels or Instructions	A	60.71	204	51.22	1.24	86.67	10.20
4.1.1—Parsing	A	89.29	1920	87.80	28.88	93.33	49.07
4.1.2—Name, Role, Value	A	69.64	233	58.54	1.61	100	11.13

An overview of the results shows that the number of errors for conformance level A was the greatest. The high number of conformance level A errors indicates that many websites do not meet the minimum accessibility requirements. The most frequently violated success criteria were 1.3.1—Info and Relationships and 2.4.4—Link Purpose (In Context), which were violated by almost all websites. These errors were followed by 1.1.1—Non-text Content, 2.4.10—Section Headings and 4.1.1—Parsing. The most accessibility errors belonged to 4.1.1—Parsing criteria, followed by 2.4.4—Link Purpose (In Context) and 1.1.1—Non-text Content. The least violated success criterion by the websites in general was 2.2.1—Timing Adjustable, which was violated by only one website.

Looking at the university type detail, private university hospital websites had more accessibility errors. The criterion with the highest average number of accessibility errors for private university hospital websites was 1.3.1—Info and Relationships. Likewise, for 1.1.1—Non-text Content criterion, while the number of accessibility errors was 16 on average in public university hospitals, which was almost three times higher in private university hospitals. The highest average error rate in state universities belonged to the 4.1.1—Parsing criterion, and this rate was also higher in private university hospitals for this criterion. On the other hand, 2.2.4—Interruption is the success criterion met by almost all the evaluated websites and no problem was detected.

Broken link analysis

The number of broken links of hospital websites was evaluated with the Deadlink Checker online test tool. According to the evaluation results, one-third of the websites (52 websites were evaluated) had more than 5 broken links. Table 2 summarizes the broken link analysis results.

Table 2

Number of broken links

University type	Average of URLs checked	Broken links
		Min	Avg	Max
State	161.79	0	4.97	24
Private	209.43	1	4.86	16

Among all hospital websites evaluated, the highest number of broken links was 24. Also, the rate of hospital websites with no broken links was about 8% and all of them belonged to state university hospital websites. The average number of broken links for state and private university hospital websites was nearly the same. However, when compared with the number of checked links, it can be said that private university hospital websites have less broken links on average.

Mobile-friendly analysis

The Google Mobile-friendly test tool was used for testing the mobile responsiveness of the websites, and 55 websites could be tested. In Fig. 4, the mobile friendly test results of the websites are presented. The majority (about two-thirds) of hospital websites pass the mobile-friendly test. These results were similar to the analysis findings of Kurt for Turkish universities [28].

Fig. 4

Result of the mobile-friendly test

Table 3 shows mobile-friendliness results by university type. According to the findings, two-thirds of state university hospital websites can be accessed via mobile devices. On the other hand, this rate was almost three quarters in private university hospital websites.

Table 3

Mobile-friendly results by university type

University type	Pass	Fail
State (n = 40)	24 (%60)	16 (%40)
Private (n = 15)	11 (%73.33)	4 (%26.67)

Discussion and conclusion

In this study, 58 university hospital web pages in Turkey were examined in terms of accessibility, with automated test tools. Online automated testing tools were used in the analysis of hospital websites: TAW tool for accessibility errors, Deadlink Checker for broken link analysis, Google Mobile-Friendly Test for mobile responsiveness analysis.

The analysis was performed according to the WCAG 2.0 guidelines, and the results showed that most accessibility errors were related to conformance level A. This error rate indicates that many websites do not meet the key success criteria for accessibility. The operable principle is one that web developers and administrators should consider, followed by the perceivable principle. Almost all websites had accessibility errors related to the following three success criteria: 1.1.1—Non-text Content, 1.3.1—Info and Relationships, and 2.4.4—Link Purpose (In Context). This was not surprising as similar findings were obtained in the accessibility analysis results of university and government websites in the literature [27-31]. When comparing the university type, the accessibility error rate of the private university hospital websites was higher than the state university hospital websites.

This study also analyzed the number of broken links on university hospital websites in Turkey. The findings showed that almost all websites had at least one broken link. The number of websites with five or more broken links accounted for two-thirds of all websites. An overview of the broken links shows that on average, private university hospital websites outperformed state universities. However, we still recommend that developers and site administrators regularly check all websites for broken links and fix any broken links detected.

Finally, hospital websites were evaluated in terms of mobile responsiveness. The results showed that about a third of websites are not mobile-friendly. This indicates that hospital stakeholders may experience accessibility problems while using information and services through websites. This is a concern that needs attention given that mobile devices generate more than 70% of website traffic in Turkey [32].

It is an indisputable fact that hospital websites are an important source of information and services for their stakeholders. This study presented the results of an accessibility evaluation of university hospital websites in Turkey. To the best of our knowledge, there is no published study analyzing the accessibility of university hospital websites. This study contributes to the growing literature by filling this gap. The results of this study generally show that many of the websites had accessibility errors, violating the WCAG 2.0 rules, and that developers and site administrators do not properly consider accessibility guidelines. Our findings related to web accessibility of university hospitals also support previous studies that examine accessibility of Turkish university websites [28, 29, 33].

From a practical standpoint, the results of the study help raise awareness on the importance of equal access to information and provide guidance for researchers, developers, and site administrators. Developers of hospital websites ought to make their websites accessible and user-centered. Future work could extend the present study by using a large number of university hospitals from different countries to reveal the complete picture. In addition, the findings of this study could be supported by extensive user testing and contribute to a better understanding of accessibility issues.

Table 4

List of universities and their hospital websites analyzed

University	Type	Website URL
Afyonkarahisar University of Health Sciences	State	http://www.hastane.afsu.edu.tr
Akdeniz University	State	http://www.hastane.akdeniz.edu.tr
Ankara University	State	http://hastane.ankara.edu.tr
Ataturk University	State	https://hastane.atauni.edu.tr
Aydin Adnan Menderes University	State	https://hastane.adu.edu.tr
Balikesir University	State	http://hastane.balikesir.edu.tr
Uludag University	State	https://suam.uludag.edu.tr
Canakkale 18 Mart University	State	https://hastane.comu.edu.tr
Cukurova University	State	https://balcali.cu.edu.tr
Dicle University	State	http://hastane.dicle.edu.tr
Dokuz Eylul University	State	https://hastane.deu.edu.tr
Duzce University	State	https://hastane.duzce.edu.tr
Ege University	State	https://egehastane.ege.edu.tr
Erciyes University	State	https://hastaneler.erciyes.edu.tr
Eskisehir Osmangazi University	State	https://www.hastane.ogu.edu.tr
Firat University	State	http://ftm.firat.edu.tr
Gazi University	State	https://hastane.gazi.edu.tr
Gaziantep University	State	http://hastanetip.gantep.edu.tr
Hacettepe University	State	http://www.hastane.hacettepe.edu.tr
Harran University	State	http://hastane.harran.edu.tr
Hatay Mustafa Kemal University	State	https://hastane.mku.edu.tr
Inonu University	State	https://totm.inonu.edu.tr
Istanbul University	State	https://hastane-istanbultip.istanbul.edu.tr
Istanbul Cerrahpasa University	State	https://hastanecerrahpasa.istanbulc.edu.tr
Kafkas University	State	https://hastane.kafkas.edu.tr
Kahramanmaras Sutcu Imam University	State	https://hastane.ksu.edu.tr
Karadeniz Teknik University	State	https://www.ktu.edu.tr/farabi
Kirikkale University	State	https://hastane.kku.edu.tr/net
Kocaeli University	State	http://hastane.kocaeli.edu.tr
Manisa Celal Bayar University	State	https://hastane.mcbu.edu.tr
Mersin University	State	http://hastane.mersin.edu.tr
Necmettin Erbakan University	State	http://www.meramtip.com.tr
Ondokuz Mayis University	State	https://hastane.omu.edu.tr/tr
Pamukkale University	State	https://hastane.pau.edu.tr
Selcuk University	State	http://www.hastane.selcuk.edu.tr
Sivas Cumhuriyet University	State	https://hastaneler.cumhuriyet.edu.tr
Suleyman Demirel University	State	https://hastane.sdu.edu.tr
Namik Kemal University	State	http://hastane.nku.edu.tr
Gaziosmanpasa University	State	https://hastane.gop.edu.tr
Trakya University	State	https://tuh.trakya.edu.tr
Van Yuzuncu Yil University	State	http://hastane.yyu.edu.tr
Yozgat Bozok University	State	https://hastane.bozok.edu.tr
Zonguldak Bulent Ecevit University	State	https://hastane.beun.edu.tr
Acibadem Mehmet Ali Aydinlar University	Private	https://www.acibadem.com.tr
Medipol University	Private	https://medipol.com.tr
Baskent University	Private	https://ankara.baskenthastaneleri.com
Beykent University	Private	https://www.ozelbeykenthastanesi.com
Bezmialem Vakif University	Private	https://bezmialemhastanesi.com
Biruni University	Private	https://www.birunihastanesi.com.tr
Atlas University	Private	https://medicinehospital.com.tr
Okan University	Private	https://www.okanhastanesi.com.tr
Istanbul Health and Technology University	Private	https://www.istun.edu.tr
Yeni Yuzyıl University	Private	https://www.gophastanesi.com.tr
Koc University	Private	https://www.kuh.ku.edu.tr
Maltepe University	Private	https://maltepehastanesi.com.tr
SANKO University	Private	https://sankotip.com
TOBB University of Economics and Technology	Private	https://hastane.etu.edu.tr
Yeditepe University	Private	http://www.yeditepehastanesi.com.tr