Towards a Comprehensive Quality Evaluation Model for Hospital Websites.

BACKGROUND: Hospital websites are important sources for patients to access health information. AIM: The purpose of this study was to develop the quality evaluation model for hospital websites.
METHODS: The quantitative study was conducted through the modified Delphi method in 2014-2015. The population of the study includes 10 experts were chosen by targeting non-randomized method. A questionnaire was prepared based on the prototype that was developed by research papers and related models. The validity of the questionnaires was confirmed by face validity and CVI and CVR estimation. Reliability was obtained by split-half method (α = 0.8). Experts' opinions were collected through interview. Then, their frequency and percentage were determined. Items with options completely agree and agree over 75% was approved, items below 50% were removed, and items 50%-75% were removed after three interviews repetitions.
RESULTS: Most of the experts agreed about the pleasant and harmonious colors, the readable and consistent fonts (100%). The least frequency was allocated to correct grammar and words, support for multilingual and rapidly changing of displaying pages in the website with a frequency of 2 (20%).
CONCLUSION: The minimum qualitative criteria for a website are usability, efficiency, user friendly, service, reliability and interaction.

INTRODUCTION

Hospital portal is a website or web service that provides required information to patients and allows them to get the news, looking for other people and talk them, build a society and interact with other network resources (1). For Many years, information hospital websites have been used as a suitable system to exchange the information between patient, hospital and medical staff (2) and to notify hospitals activities and services (3). Statistics show that almost 75 percent of American and European patients and healthcare professionals use hospital websites to obtain information (4), but low quality websites provide information that is not useful to users and even then users may not visit other sites and the main purpose of website design is not realized. Conducted studies have shown the need for evaluations. As a research that was done on 8 educational websites, hospital websites are structurally on an average level, and were extremely weak in terms of content and following links (thematic content) (2). Another study has shown low number of English web pages, technical and managerial problems and lack of update the websites (5). While the existence of this report has made websites evaluation inevitable. But quality evaluation of websites requires appropriate evaluation criteria. Unfortunately, many of these criteria are not easily measurable (6), do not take into account all aspects of the site and focused on evaluating the usability and accessibility of web sites or evaluation by people with a background in information technology is required (3). focus on purpose and audience criteria, validity, accuracy, objectivity, currency, reliability, coverage, format, presentation special features (7) Competence and credibility of the author and publisher, scope of coverage, content, accuracy, purpose and audience, interaction, accessibility and user-friendly, updating of content, (8) as well as features such as orientation, commitment, use of meta tags and availability were considered as the needed main criteria of websites (9, 10).

In one study, accessibility was listed as the only features for quality of website in data recovery (11). Various models have been proposed to evaluate the commercial websites (2). As studies recommend the use of modern technologies (12). So far, there is no study about the model and quality evaluation criteria of hospital websites and a comprehensive model has not been provided in this regard. This study aims to examine the quality evaluation models in available websites and seek the views of experts on the developed model to evaluate hospital websites. Hoped the research results are helpful to evaluate and improve the quality of hospital websites.

METHODS

This is a quantitative study and was conducted based on the modified Delphi method. The environment of the study was medical science universities of Iran. The study society included experts in health information management, medical informatics and IT professionals in hospital websites departments. 10 professionals who have at least a master’s degree and 2 years work experience in the field of medical websites were selected by non-probability sampling. The first prototype was established through reading relevant studies and available models (9). The opinions of experts were gathered through a questionnaire to evaluate first prototype. The questions of the questionnaire were criteria and sub criteria of the proposed model based on five-item Likert scale (completely agree, agree no idea, disagree and completely disagree).

Validity of questionnaires was approved through face validity and CVI and CVR estimation and obtaining the opinions of four experts (Faculty member of Management and Information Technology and Informing Department). To determine the content validity ratio (CVR), expert opinions in each criterion based on the three Likert including “1” = Required, “2” = useful but not necessary, “3” = not necessary were collected and content validity was calculated by the following formula and variables with content validity higher than 0.99 were retained based on the law’s table (13).

CVR = [ne-(N/2)]/ (N/2);

N = the number of experts who chose the necessary option;

N = the total number of experts.

To determine the content validity index (CVI), the first prototype of experts’ opinions in each criterion was obtained based on the relevance, clarity and simplicity of each of the areas included in the initial questionnaire score from 1 to 4:

1 = not necessary.

2 = essential, but needs further review.

3 = essential, but need to be revised shortly.

4 = absolutely necessary.

The average content validity index of initial questionnaire was calculated. Scores higher than 0.79 were appropriate variables, variables among 0.70 to 0.79 required to reform and variables less than 0.70 were unacceptable (14). It should be noted that during the compilation of the final model, CVI and CVR was measured in all stages. The reliability of the questionnaire was 0.80 using Cronbach’s alpha coefficient and half-split method. The prepared questionnaire with the initial model and explanation was delivered by experts by electronic post or personally. Experts’ opinions were accounted by descriptive statistics in form of number and percentage. Criteria and sub-criteria with completely agree and agree score higher 75% were considered in the final model, criteria and sub-criteria with completely agree and agree score lower 50% have been removed off the model and items with completely agree and agree score between 50% - 75% have been surveyed by three repetitions. Cases where did not acquire agreements after three repetitions excluded from the final model.

RESULTS

A Quantitative research was conducted using modified Delphi technique to develop the quality evaluation model of hospital websites in 2014-2015:

A. CVR and CVI results of preliminary model of quality evaluation for hospital websites

After determining CVR of the first prototype, CVR value for the sub criteria of “propaganda” and “impartiality (objectivity)” was zero and therefore were omitted from the first prototype. The rest of the criteria and sub-criteria were retained in the Final model.

After determining the CVI of the first suggested model, the scores of two sub criteria of “newsgroups” and “Customize” were less than 0.7, therefore were excluded from the first suggested model.

Also item “Frequently Asked Questions” was transferred to interaction criteria and “audience statistic” was transferred to ease of use and user friendly criteria.

B. The results of the Modified Delphi method

The results of the modified Delphi method showed that all criteria and sub criteria in the first phase could gain the agreement above 75%. Therefore, all items were accounted in the final model. Most experts accepted the colors used on the website, fonts, updated information, understandability of menus, useful educational information, useful medical consult and to communicate with a frequency of 10 (100%) and the lowest frequency was allocated to correct spelling and grammar, support for multilingual and fast switching of displaying pages with a frequency of 2 (20%) (Table 1).

Table 1.

Experts’ Opinions on the Proposed Model of Quality Evaluation for Hospital Websites

Criterion	Sub criteria	Completely agree	Agree	No idea	Disagree	Completely disagree	Status
Usability	Correct spelling and grammar	2 (% 20)	8 (%80)	-	-	-	Approved
	Support for multilingual websites	2 (% 20)	6 (%60)	-	2 (% 20)	-	Approved
	Support a variety of web browsers	4 (%40)	3 (%40)	1 (%10)	1 (%10)	-	Approved
	Diversity to display information websites	8 (%80)	2 (% 20)	-		-	Approved
	The appropriateness of Web content	9 (%90)	-	-	1 (%10)	-	Approved
	Pleasant and harmonious use of colors on the website	10 (%100)	-	-	-	-	Approved
	Readable and consistent fonts	10 (%100)	-	-	-	-	Approved
	The availability of Site Map (menu bar) on each page	8 (%80)	2 (% 20)	-	-	-	Approved
	Time interval of information updates	10 (%100)	-	-	-	-	Approved
User-friendly	Ease of website use	7 (%70)	3 (%30)	-	-	-	Approved
	The need for training and previous experience for website	7 (%70)	3 (%30)	-	-	-	Approved
	Ease Search the websites r-frie	2 (% 20)	7 (%70)	1 (%10)	-	-	Approved
	Interesting websites graphics	9 (%90)	-	1 (%10)	-	-	Approved
	Statistics daily visits of the websites	8 (%80)	1 (%10)	1 (%10)	-	-	Approved
Performance	Websites loading speed	8 (%80)	-	1 (%10)	-	1 (%10)	Approved
	Rapid change of pages displayed on the websites manc	2 (% 20)	6 (%60)	1 (%10)	-	1 (%10)	Approved
	The availability of websites	5 (%50)	5 (%50)	-	-	-	Approved
	Understandability of available menus on the website for at least literate	10 (%100)	-	-	-	-	Approved
Services	Usefulness of the information provided on the website for the general population or patients referred to the websites	10 (%100)	-	-	-	-	Approved
	Search capabilities websites	7 (%70)	2 (% 20)	1 (%10)	-	-	Approved
	User training tips (help)	4 (%40)	3 (%30)	1 (%10)	2 (% 20)	-	Approved
	The usefulness of the page "Frequently Asked Questions"	9 (%90)	1 (%10)	-	-	-	Approved
Validity	Clear and attractive logo and website (hospital) is located in place.	9 (%90)	1 (%10)	-	-	-	Approved
	ValidityFull introduction hospitals (History, year established, geographic location, etc.)	9 (%90)	1 (%10)	-	-	-	Approved
	Introducing wards and outpatient and emergency department in hospital websites	9 (%90)	1 (%10)	-	-	-	Approved
	Introducing the facilities completely in the web-sites	9 (%90)	1 (%10)	-	-	-	Approved
Interaction	Satisfactory of internet settlement of outpatients and bedridden in the website	9 (%90)	-	-	1 (%10)	-	Approved
	useful and satisfactory facilities such as inpatient or outpatient reception on the website	9 (%90)	1 (%10)	-	-	-	Approved
	Satisfactory Internet settlement on the website ction	10 (%100)	-	-	-	-	Approved
	InteraSatisfying reception for radiology department	9 (%90)	-	-	1 (%10)	-	Approved
	Satisfying reception for laboratory department	9 (%90)	-	-	1 (%10)	-	Approved
	Satisfying reception for ultrasound department	9 (%90)	-	-	1 (%10)	-	Approved
	To communicate with the author or webmaster	10 (%100)	-	-	-	-	Approved

C. Developing the Final Model of the Quality Evaluation for Hospital Websites (QEHW):

First Criterion: Usability

Effective Website is so that users wish to re-visit it and make use of its services and includes the following criteria:

Timely: To update information, the new and recent information published on the site.

Multilanguage: Multi-language information on hospital websites.

Variety of presentation: present information in various forms (text, video, audio) to allow users to select and download the appropriate format.

Reliability: reliability and the accuracy of the content available on the website.

Appropriateness: appropriateness of the website structure with operational objectives in terms of design, color and shape.

Color: preferred use of bright colors and the use of more than 4 colors on one page. Use bright colors for the background and not to use different and uncoordinated colors on the website.

Text: proportional font and size of text, except in the title or header. Do not fill the website with long texts.

Sounds, images, videos: The small size of the images, sounds, or video used in the website to download page rapidly.

Site map: existence of website map on every page with the link at the bottom of page.

Second Criterion: Efficiency

Includes following criteria:

Speed: High speed of loading and processing on the website.

Responsiveness: The ability of a website to answer questions and meet users’ needs.

Accessibility: to find websites easily through search engines or external websites.

Understandability: To do a task and use of services easily, especially by referring users first.

Third Criterion : Ease Of Use And User Friendly

The attractiveness, ease of use and meet the needs of users with the following criteria:

Ease of use: easy to understand, use, implementation, finding information and navigating websites.

User interface: legible and tangibility of information for users with different educational website.

Graphic: The graphics and user-friendly of a­ website to increase user understanding of information in the website.

Attractiveness: Create a pleasant way for users visiting again website.

Statistics of website visitors

Fourth Criterion: Services

Educational information: Get information for health care and personal hygiene.

b. Search: Ability to search key words and definitive information on websites with the internal search tool.

c. Training user tips: clear guidelines to use of parts, services and forms, especially for low-literacy users and the elderly.

Fifth Criterion: Validity

Websites address suffix:

Logo: clear, readable and interesting logo on all pages of the websites.

Exclusive information of hospital: introducing various hospital departments and facilities, price list and cost of equipment used in various sectors, and...

Sixth Criterion : Interaction

Exchange of information between the user and the website to use website services and facilities include:

Medical consult: free medical consult services for patients

Internet reception: online reception for bedridden patients and outpatients

Internet settlement: Internet settlement of patient billing

Radiology and ultrasound, laboratory and library and database of telemedicine: get results by visiting the hospitals websites and entering a tracking code through a central database.

Page of “Frequently Asked Questions”: Existence questions that are frequently asked on the “Frequently Asked Questions” and show the answers. ()

DISCUSSION

The results of research that aims to develop quality evaluation model for hospital websites showed the developed model with six criteria and sub criteria is suitable to evaluate hospital websites. In the Usability Criterion, pleasant and harmonious the colors, readable and consistent fonts and the interval time of updating the web site achieved the highest agreement of experts with a frequency of 10 (100%). In the user friendly criteria: the highest percentage was related to graphics with a frequency of 9 (90%). In the performance criteria: Most of the agreements were allocated to the understandability of the menus with a frequency of 10. (100%) in the service criteria: the highest percentage of agreement was relating to the logo, the introduction of hospital, introduction of wards and introduction of hospital equipment with a frequency of 9 (90%). In the interaction criterion: the highest percentage was relating to medical consult and administrator’s ability to communicate with a frequency of 10 (100%)

Fang conducted a study in Thailand (2002) and introduced ten features of e-government, including comprehensive, interactive, inclusive, easily accessible, secure, customizable, versatility and flexibility (15). According to Oreste (2005) study in Thailand in providing quality models for a website, correctly, display, navigation, content and interaction were considered as important criteria in the quality of the website (16). Rafe and Monfaredzadeh (2011) provided a qualitative framework consists of 7 criteria to evaluate the quality of websites hospital included: quality of content, quality of design, quality of organization, user-friendly, performance, quality of service and the quality of the websites from a technical point (17). GonzáLez (2004) conducted an empirical study aimed at developing an evaluation index of e-banking websites in Spain and stated that criteria such as availability, speed, navigation and content of the website are required to analyze business, educational and non-profit organizations (18). McInerney and Bird (2005) in America reported that the criterion to evaluate the quality of a good website is accessed .(11)Lin and Joyce (2004) studies in New Zealand introduced six factors, including consumer education, security, customer support, online communities and market position as the most important factors to succeed online auctions (19). Danesh et al (2011) introduced Factors such as size, language, age, including as reasons for the success of a website (20). Kokkinaki (2005) in Cyprus introduced the framework of e-government evaluation such as content, design and common features of e-government Websites (21). Liu and colleagues (2000) in America examined the factors associated with success in the field of e-commerce website and introduced information quality, service quality, use of the system, recreation and system design (22). In Kuwait, Aladwani (2006) conducted an empirical study of the relationship between the quality and integrity of enterprise customer’s web websites and divided the appropriate quality of services in four categories such as Technical, total and detailed content and appearance. He believes that these aspects could affect the attitude of website users (23). The results of observed Quality evaluation models in websites showed that several measures have been effective in the quality of the website, as web QEM offered 6 criteria including usability, reliability, performance, functionality, content and navigation (24). Ministerial Network for Valorizing Activities in Digitization (MINERVA) introduced nine criteria such as clarity, effectiveness, maintainability, availability, user-centric, responsiveness, and multi-language and safety management (25). Cicerone loci introduced seven criteria such as availability, content, service, location, maintenance, usability and feasibility as qualitative criteria in order to evaluate websites (26). Miles model introduced six criteria such as content, services, orientation; cognitive features interface (client), aesthetics and technology as criteria for assessing the quality of a website (24, 27). Web Qual model introduced twelve criteria such as Information tailored to work, interactivity, reliability, response time, designing, speculative, appearance, innovation, flow, integrated communications, business processes and the replacement (16). Comparison of recognized models of website quality evaluation showed that the usability features in both web-QEM, and Cicerone loci model have been used. Efficiency features and performance only was introduced in web-QEM model and other models have not used it.

Service and content features have been used in both Miles and Cicerone loci models. Navigation, aesthetics, technology and user interface features have been used in Miles model. Maintenance features has been used in the MINERVA Cicerone loci model. Clarity, effective, user-centric, responsiveness, multilingual, interaction, managed and protected features have been used in the MINERVA model. Also, possibility, existence and location features have been used only in Cicerone loci model and have not been mentioned in other models. Although some measures presented in this study are consistent with these studies, but some of the criteria and sub-criteria are not consistent with the results such as Sung Eon, Shaw and Schneider (2003) in Korea, which examined sample website design in the industry field and introduced standards of business performance, credit, reliability, content, attractiveness, systematic structure and surfing criteria to design a pattern in the field of industry groups (28).Ranganathan and Ganapathy (2002) in America examined the key dimensions of business websites and found that the information content, web design, security and confidentiality of personal information, including the most important factors that affect the willingness of customers to purchase via the web (29). Lim (2002) in Australia, introduced factors related to electronic shopping such as usefulness, ease of use, enjoyment and safety (30) Mateus et al. (2001) in Spain introduced the criteria of content, speed, availability and facility tour as evaluation indicators (8). This difference can be attributed to this reason that in our study, hospital sites have been considered, but in most studies the properties to evaluate the quality of commercial websites were investigated. The strength of this study is review articles and given models and integrate it with the experts. So that, after reading all models and studies and comparison of criteria and sub criteria, the first model was provided and then was established by collecting experts’ opinions. The weakness of this study can be cited to ignore the comments of others. Users’ comments can be useful for improving the quality of web sites, it is recommended that a separate study to collect user feedback and in the complete model to be considered.

CONCLUSIONS

Use and evaluate hospital information websites are needed as a proper system to exchange information between patient and medical staff and to notify hospital activities and essential services. Minimum quality criteria for a website are usability, efficiency, ease of use, user friendly, service, reliability and interaction.