Evaluation models and criteria of the quality of hospital websites: a systematic review study.

INTRODUCTION: Hospital websites are important tools in establishing communication and exchanging information between patients and staff, and thus should enjoy an acceptable level of quality. The aim of this study was to identify proper models and criteria to evaluate the quality of hospital websites.
METHODS: This research was a systematic review study. The international databases such as Science Direct, Google Scholar, PubMed, Proquest, Ovid, Elsevier, Springer, and EBSCO together with regional database such as Magiran, Scientific Information Database, Persian Journal Citation Report (PJCR) and IranMedex were searched. Suitable keywords including website, evaluation, and quality of website were used. Full text papers related to the research were included. The criteria and sub criteria of the evaluation of website quality were extracted and classified.
RESULTS: To evaluate the quality of the websites, various models and criteria were presented. The WEB-Q-IM, Mile, Minerva, Seruni Luci, and Web-Qual models were the designed models. The criteria of accessibility, content and apparent features of the websites, the design procedure, the graphics applied in the website, and the page's attractions have been mentioned in the majority of studies.
CONCLUSION: The criteria of accessibility, content, design method, security, and confidentiality of personal information are the essential criteria in the evaluation of all websites. It is suggested that the ease of use, graphics, attractiveness and other apparent properties of websites are considered as the user-friendliness sub criteria. Further, the criteria of speed and accessibility of the website should be considered as sub criterion of efficiency. When determining the evaluation criteria of the quality of websites, attention to major differences in the specific features of any website is essential.

1. Introduction

Since quality has an abstract concept, presenting a practical method for determining it is not easy. However, this is the conceptual quality that can be felt, perceived, and judged and when a product or good does not have quality, lack of quality can easily be sensed. Therefore, to improve the reception and use of a product or service, the way quality is determined should be defined, and its evaluation and control procedure should be understood (1), so that during the evaluation process, it is ensured that the delegated tasks and responsibilities are done well (2). One of the services that has had a considerable development in recent years is the World Wide Web, providing the world with a phenomenon called websites. Websites involve connection points and communication of users with electronic information and considering nowadays all institutes try to present their services to different classes, the World Wide Web has been successful in meeting this need in generating information and making them accessible with an ever-increasing speed (3). In the health sector, many hospitals have designed individual websites for themselves presenting various services. Given the sensitivity of hospitals, the services they are seeking to introduce, in response to the increasing development of hospital websites to allow the access of various people to these websites, the issue of website quality has arisen. This is because the richness of websites and the number of referrals to them, contribute to improving the status of the hospital in the specialized area and in offering services to patients. If the performance of a hospital website is weaker than the expected level, redesigning the website or substantial enhancement of the volume and quality of electronic journal may be required (3). To evaluate the quality of websites, various models and criteria have been introduced. The first one is the ISO-9126 Model whose criteria are usability, reliability, efficiency, and functionality. The Mile Model, focusing on the usability of the website, measures the quality of websites based on content, services, navigation, cognitive features of the interface, aesthetics, and technology. And finally, the Minerva Model, proposed to evaluate the quality of cultural websites (1, 4, 5). Unfortunately, despite the various models presented for evaluating websites, determination of proper models and criteria for evaluating the quality of hospital websites has been understudied. Thus, this research tried to identify the criteria required for a hospital website by investigating the evaluation models of the quality of websites. It is hoped that the results of this research are found useful in presenting a model for evaluation of the quality of hospital websites to promote their quality.

2. Material and Methods

This research was a systematic review. The studies either in English or Persian, that have been related to evaluation of website quality and whose title and abstract included keywords such as evaluation, website quality, and website model, were included in this research.

2.1. The search strategy

The international database including Science Direct, Google Scholar, PubMed, Proquest, Ovid, Elsevier, Springer, EBSCO, and regional databases including Magiran, Scientific Information Database and Persian Journal Citation Report (PJCR), were searched using the keywords of website, evaluation, and website quality, according to Table 1. All papers, regardless of the methodology including systematic review, experimental, and cross-sectional studies that were related to the research subject and were accessible in their full text were selected. The models were then reviewed and out of the selected papers, the criteria and sub criteria of website quality were extracted and then classified.

Table 1

Search Key Words

No.	Search Key Words
1	Website and quality
2	Hospital website and quality
3	Website and evaluation
4	Website and model of quality
5	Evaluation and website quality
6	Hospital and website and quality
7	Quality of website and model
8	Evaluation of website and quality
9	Evaluation of website and model
10	Model and hospital and website
11	Model and evaluation and website
12	Model and website and quality
13	Hospital and website and quality
14	Evaluation and quality and website
15	Evaluation and quality and website and hospital
16	Model and website quality
17	Model and hospital website and quality
18	Evaluation and hospital and website
19	Hospital’s website and quality
20	Evaluation and model and quality of website

2.2. The inclusion and exclusion criteria

The papers that were in line with the subject whose full text version was accessible, were included, while the papers that had no conceptual relationship with the subject or their full text was not available, were excluded from the study. The results extracted and investigated in this research were the criteria required for evaluating the quality of hospital websites. The investigation was carried out comparatively.

2.3. Quality assessment

To evaluate the papers qualitatively, the extracted articles were studied by two relevant experts (Figure 1). Following specialist discussion and opinion exchange, 22 papers were selected (Table 2).

Figure 1

The Qualitative Evaluation of the Papers

Table 2

The List of Reviewed Studies

Ref. no.	The Type of study	Country	Title
5	Descriptive	Italy	Towards a model of quality for websites
7	Descriptive	Iran	Analysis of the structure and content of elected educational hospital websites in the country in 2007.
8	Descriptive	Iran	Evaluation of the website quality of libraries of Iranian universities to present proposals to promote their quality
9	Descriptive	Iran	Core websites of the Islamic World universities
10	Descriptive	Iran	Evaluation of Iranian web content in terms of electronic government
11	Descriptive	Iran	A qualitative framework for evaluating medical/hospital websites
12	Navigational	Iran	Status of Iranian websites according to general criteria of evaluation
13	Review	Korea	Designing a model website in the field of industrial groups
14	Descriptive	Britain	Longitudinal trend in academic web links
15	Analytical	Spain	The website evaluation index: Analysis of Spanish universities
16	Analytical	Thailand	Government in the digital era: concept, practice, and development
17	Descriptive	USA	Evaluation of the website quality in information recovery about genetically modified food on the web
18	Cross-sectional	Korea	Evaluation of health information sites on the Internet in Korea
19	Descriptive	USA	Investigation of the factors related to website success in the area of electronic business
20	Descriptive	Cyprus	Electronic government in Cyprus, training workshop of electronic government
21	Experimental	Spain	Quantitative investigation of electronic banking websites
22	Experimental	Kuwait	The experimental test of the relationship between website quality and integrity of investment with website clients
23	Review	USA	The key dimensions of business websites to consumers
24	Descriptive	New Zealand	Important factors of success in the online auction of websites
25	Analytical	Australia	Electronic purchase: analysis of the theory of applications and enjoyment
26	Descriptive	USA	Clothes retailers: the dimensions of website quality and satisfaction
27	Descriptive	Norway	Website access of electronic government

3. Results

3.1. Results resulted from the identified models

To evaluate the quality of hospital websites, various models had been presented, the most famous of which were:

3.1.1. Web-Q-I-M Model

The quality criteria in this model had been developed based on ISO-9126 model, including usability, reliability, efficiency, and functionality. The specific feature of this model was determination of a certain domain of evaluation and conductance of the evaluation step by step (1, 6).

3.1.2. Mile Model

This model focuses on usability of the website and measures the quality by combining the inspection by both an evaluator and users (1). The specified criteria in this model are:

Content: Content means the quality of the information inside the website, and its communicative level.

Services: Services are all the functionalities that the website allows its users to do.

Navigation: This criterion encompasses 1) various means of access by users to a certain part of the information in the website and 2) the logical structure of information to move across one part of information to another.

Cognitive features of the interface: It takes the perception and feeling of the user into consideration and deals with how the users memorize the structure of the website.

Aesthetics: It addresses the graphic design and template of the website, the type of font, color, size, images, and other graphic features in a website.

Technology: It represents the adaption of the website for proper function in a variety of explorers, the security level of the host server, and the interaction between the website and remote database (1, 5).

3.1.3. Minerva Model

This model was suggested for evaluation of the quality of cultural websites (seminaries, archives, libraries, and other cultural institutions) (1). In this model, quality has been defined based on accessibility and usability. The aim of quality indices in this model is to present indices for evaluation of the quality of websites and support the design and development of the cultural websites (5). This model involves the following axes:

Clarity: the clarity of objectives, mission, and the identity of the website for users.

Effectiveness: the content of the website is related to the validity of the website content and presentation of sound information to users.

Maintenance: it focuses on the technical and content support, fresh content, and improvement of the website’s technical functions.

Accessibility: It represents easy access of all users to the website,(for example, access for blind users, those with relative vision, or users with hearing problems), the possibility of using a variety of technologies to present information to users, and the functionality with a variety of explorers, operating systems, and devices.

User-centeredness: This means that the website should meet the needs of users. Further, the users should regard the website as useful, attractive, and easy to use.

Responsibility: It refers to the ability of the website owners in responding to the questions of users, helping users to participate in generating content and answering the questions posed in a forum.

Multilingualism: The website should allow users to use it in several languages. Language is an important barrier in access to a website. Therefore, the multilinguality should be taken into account in websites.

Interactivity: It refers to the interaction of a website with other websites. If a website has been based on standard technologies, techniques, and models, interaction and adaption with other websites and institutions would be easy.

Managed: It denotes the legal issues related to protecting copyright and privacy (1, 6).

3.1.4. Seruni Lucci Model

This model involves entity (who), content (what), services (why), location (where), maintenance (when), usability (how), and possibility (by what means) which are defined as follow:

Entity: includes the sub features of identification of characteristics.

Content: deals with coverage and accuracy of information.

Services: refers to functions and control.

Location: involves usability, accessibility, navigation, and understandability.

Maintenance: includes the sub features of correction and adaption.

Usability: represents accessibility and understandability.

Facilities: encompasses resources, information and communication technology (5)

3.1.5. Web-Qual Model

This instrument has 12 dimensions including information suited to users, interaction, reliability, responsibility time, appearance, design, guesswork, innovativeness, stream, coherent connections, business process, and the ability for substitution (5) (Table 3).

Table 3

The Criteria Used in the Evaluation Models of Quality of Websites

Models:	Web-Qual	Web-Q-I-M	Serruni Lucci	Minerva
Criterion:	User-suited InformationInteractionReliabilityResponsibility TimeDesignGuessworkAppearanceInnovativenessStreamCoherent CommunicationBusiness ProcessAbility for Substitution	UsabilityEfficiencyReliabilityFunctionalityContentNavigation	ContentServicesNavigationAestheticsTechnologyCognitive Features of Interface	ClarityEffectivenessmaintenanceAccessibilityUser-CenterednessResponsibilityMultilingualityInteractionPreserved

3.2. The results obtained from the reviewed studies

Results indicated that different dimensions of website evaluation were considered in various studies. Some of them focused on the structure, the homepage content, page content, and subject content (other links) of websites for evaluation purposes (7, 11, 13, 15, 19–21, 23–25). Other studies mentioned characteristics of audience, website objective, loading, and structural stability among websites as important factors in the success of a website (8, 12, 14, 26). Also quality, size, language, background, and inclusiveness as the determinants of the final success of a website were referred (9, 12). Another study found that attention to the update rate of pages (the duration between the last update and the date the website is used) is essential (10). A technical framework presented seven criteria including: content quality, design quality, organization quality, user-friendliness, functional quality, service quality, and website quality from a technical point of view (11). Moreover the evaluation criteria as the accessibility and updating were mentioned (10, 12, 15–18, 22, 27). Furthermore, entertainment and fun, and website speed was presented as evaluation criteria for websites (13–15). Interactivity was mentioned (12, 15, 16, 24, 25). Another study stated the security, and confidentiality of personal information as the most important factors affecting the willingness of customers to purchase from a website (23–25) as well as multilinguality (27) (Table 4).

Table 4

The Findings Obtained from the Reviewed Studies

Ref. no.	Findings of the Research
5	This includes the features of content, design, and typical features of electronic government websites.
7	He discovered the significance of website accessibility by referring to electronic government websites.
8	The presented model includes correctness, presentation, navigation, content, and interaction.
19	The characteristics of audience and objective of the website, loading, and stability of the structure had a good status across the studied websites.
10	Presentation of evaluation index for analyzing commercial, educational, and nonprofit organizations including accessibility, speed, navigation, and content.
11	The final success of the website is contingent upon factors including quality, size, language, background, inclusion, etc. and thus one cannot mention one or two factors as the only reasons of the success of a website.
12	The identified six important factors in the success of online websites including design and content, consumer education, security, customer support, online communities, and market situation.
13	The results indicated that accessibility was the only feature important for the quality of websites in recovering information.
14	Understanding ease-of-use and utility of electronic purchase websites has a direct and significant effect on the success of websites.
15	A review on the websites of the school board of the Central Eastern University of India, where the surfing cities, accuracy, and validity gained the highest rank. In contrast, the content of these websites including the aspects of employment and retention, interactive aspects, novelty, and contribution to scientific advancement obtained the lowest ranks
16	The portal of an electronic government should be comprehensive, interactive, inclusive, convenient, accessible, secure, customizable, multifunctional, flexible, and developed by the electronic government system.
17	In general, presence of Iranian websites in the Internet was not regular. It is not based on well-documented and specified planning and policy and, as for the information gates of the country; they do not contain a significant volume.
18	Represented the flexibility of this profile of web evaluation and was determinant of the strong and weak points of the investigated websites
19	There were major differences in the design of websites across various groups
20	They found that the content of information, website design, security, and confidentiality of personal information are among the most important factors in influencing the tendency of clients to purchase through Internet.
21	Over 50% of websites did not have updating or date of production, developer, or valid authors and links to other sites.
22	They introduced the dimensions of website quality already presented by various individuals and sources.
23	A variety of news, scientific, commercial, and weblog websites were investigated. It was found that the highest content was related to news websites.
24	They presented a qualitative framework consisting of seven criteria: content quality, design quality, organization quality, user-friendliness, functional quality, quality of services.
25	The websites of Iranian educational hospitals were of medium level in terms of structure and very poor in terms of content and sub-links (thematic content).
26	They identified four factors essential to the success of the web in electronic business. They were information, quality of services, use of systems, fun and entertainment, and the quality of system design.
27	He categorizes the quality of services into four groups of technical, general and specific content, and apparent quality. He believes that these dimensions influence the attitudes of users to the website and attitude of users in turn affects their tendency toward purchasing through the Internet.

4. Discussion

One of the most useful applications of the web is the web portals or website. The aim of this study was to identify the models for evaluating the quality of hospital websites. The results indicated that a great variety and diversity exists among the criteria presented for evaluating the quality of websites. On the other hand, many of these criteria can be considered as the sub criteria of a more general criterion. A study investigated the factors associated with the success of websites in the area of electronic business in the US. The study introduced four factors of web success in electronic business as information, the quality of services, use of systems, entertainment and fun, and the quality of system design (19). Another study experimentally examined the relationship between the quality of websites and integration of investment of website customers in Kuwait. The researcher categorized the quality of services in four aspects of technical, general and specific content, and the apparent quality. He believes that these dimensions influence the attitude of website users, where the positive attitude of users in turn affects their tendency to buy through the Internet (22). A paper called ‘The Government in the Digital Era: Concept, practice, and development in Thailand’, stated 10 characteristics of an electronic government, arguing that an electronic government should be comprehensive, interactive, inclusive, convenient, accessible, secure, customizable, multifunctional, and flexible (16). In Spain experimentally and quantitatively investigated websites of electronic banking with the aim of finding an index for web evaluation, reported that the evaluation index for analyzing the website of commercial, educational, and nonprofit organizations is speed, navigation, and content (21). A paper called ‘The Qualitative Framework of Evaluation of Hospital/medical Websites’ presented a qualitative framework consisting of seven criteria: the quality of content, design, organization, user-friendliness, performance, services, and technical quality (11). A research entitled ‘Electronic Government in Cyprus’ proposed a framework for evaluating the development of electronic government. This framework comprised content, design, and typical features of electronic government websites (20). Results of a research introduced six important factors for success of websites as design, content, consumer education, security, customer support, online communities, and the market situation (24). A model for the quality of websites with the criteria of correctness, presentation, navigation, content, and interaction was presented in Thailand (5). The final success of websites, associated with factors such as quality, size, language, background, inclusion, etc. and it was stated that one cannot mention one or two factors as the only reasons for the success of a website (9). The result of investigation of the selected studies revealed that the criteria of accessibility, content, and apparent features of websites from user perspective such as design, the graphics used in the website, and attractiveness of the page have been mentioned in the majority of studies. The results also showed that the important point in determining the criteria for evaluating the quality of websites is attention to the major differences in the special features of any website; features considered during the design and selection of content for groups with various aims, so that a website becomes successful in responding to an objective for which it has been designed. These features are especially important during the evaluation of websites, since neglecting these features causes many websites to fail to reach the aims for which they have been designed, thus requiring revision. The results also indicated that the features of the information content, website design, security, and confidentiality of personal information are among the most important factors affecting the tendency of clients to use a website, thus paying attention to it in the design of any website and any evaluation model as a common criterion is essential. Determination of a model with criteria specific to special websites is also recommended.

5. Conclusions

Minimum quality criteria for a website are usability, efficiency, ease of use, user friendliness, service, reliability and interaction. The criteria of accessibility, content, design, security, and confidentially of personal information, as the essential criteria, should be taken into account when evaluating a website. It is suggested that the criteria such as ease of use, playing a role in bringing about satisfaction of users, the graphics used in the website, attractiveness of the website, and other apparent characteristics of websites are considered as sub criteria for the criterion of user-friendliness. It is also proposed that the criteria of speed and accessibility of websites, which indicate the efficiency of a website, are included in all models of website evaluation as the sub criterion of efficiency.