Evaluating the usability of a national health information system with heuristic method.

BACKGROUND: Hospital Statistics and Information System is one of the most important health information systems in Iran used in all hospitals in this country. Usability problems can reduce the speed and precision of users when interacting with this system. This study aimed to identify the usability problems of a national health system called "AVAB".
MATERIALS AND METHODS: This descriptive cross-sectional study was conducted in 2020, and three experts evaluated the usability of this system independently by the heuristic evaluation method. Nielsen's usability principles were used to identify usability problems and to classify their severity.
RESULTS: A total of 86 unique problems were identified. The highest number of problems were related to the two principles of "help and documentation" and "match between system and the real world" with 23 and 11 usability problems, respectively. The lowest number of problems were related to the two principles of "visibility of system status" and "help users recognize, diagnose, and recover from errors," each with three problems. 58.1% of the identified problems were in the group of major and catastrophic problems.
CONCLUSIONS: With the help of heuristic evaluation method, a significant number of usability problems of Hospital Statistics and Information System were identified. Most of the identified problems were major and catastrophic, and it is necessary to solve these problems by the designers and developers of this system. Copyright:

Introduction

Health information systems in common are computer systems used to collect, store, process, retrieve, and transmit clinical, financial, and administrative information. Information systems are widely used to provide safe, effective, efficient, and timely services.[12] Despite these advantages, users still have problems with the interaction of health information systems, which makes it difficult to adopt these systems fully.[34] With significant changes in giving services based on information communication technology, the evaluation will discover underlying problems affecting human–computer interactions.[5] Usability issues are an obstacle to the efficient use and acceptance of information systems.[6] Usability is defined as the efficient, effective, and satisfactory use of a product by users.[6] Furthermore, usability problems can lead to a lack of user acceptance of the system and its absolute rejection a few months after its launch.[7] Various methods can be applied to evaluate the usability of information systems. Popular methods that can be used to evaluate such systems include the following: cognitive walkthrough, focus group, heuristic evaluation, think aloud method, and adaptation of standard questionnaires (such as system usability scale, end-user usability questionnaire, and NASA Task Load Index questionnaire).[8] Hence, one of the most common methods of usability assessment is heuristic evaluation.[9] Heuristic evaluation efficiently is an inspection approach that can be utilized to recognize chief usability difficulties.[10] Nielsen first introduced this method, in which several principles were employed to identify usability problems and their severity.[11] In this method, each evaluator independently evaluates the user interface of the system based on predetermined principles, and any violations of the principles considered usability problems.[12] According to studies, heuristic evaluation possesses several advantages, including using this method saves time and money, and is fast because assessed by experts. In this evaluation method, the elements in a system's user interface can be compared with the accepted and standard principles. This evaluation method is very simple and has high efficiency.[131415] Heuristic evaluation is performed by 3–5 evaluators and can identify 74%–87% of problems.[1016] Many studies have used the heuristic evaluation method to evaluate the usability of health-care systems; for example, Admission and Medical Records Subsystems of Hospital Information Systems[17] and Laboratory and Radiology Information Systems[18] were evaluated by the heuristic method.

Hospital Statistics and Information System (AVAB in Persian) is a web-based system used to collect, standardize, and update statistics and information of Iran's hospitals and has been launched online. This system has been designed and implemented experimentally to collect statistics and information on structural resources and performance information of the country's hospitals since 2011. “AVAB” is a web-based system and records the latest information on hospital IDs and hospital beds, statistics of medical wards, human resources, medical equipment, and hospital physical space. “AVAB” has hundreds of users, including the Ministry of Health, doctors, nurses, medical equipment experts, health center managers, health information managers, health-care providers, and paramedics throughout Iran. Therefore, considering the high use of Hospital Statistics and Information System throughout the country and its vital role in providing health services, its usability should be carefully evaluated before its widespread use. To the best of our knowledge, no study has been conducted in Iran to evaluate the usability of “AVAB” system. Therefore, this study aims to use the heuristic evaluation method to identify the usability problems of “AVAB” system.

Materials and Methods

Study design and setting

The present study was a descriptive cross-sectional conducted in 2020 in Iran. “AVAB” is a national health information system launched by the Ministry of Health of Iran in 2011 and is used to collect all governmental and nongovernmental medical centers’ hospital statistics. The users of this system are from all hospitals throughout Iran. Since the users of this system are people with numerous disciplines who may be unfamiliar with information systems, it is very significant to evaluate this system's usability. Therefore, in this study, the heuristic evaluation method was used to determine the usability problems of “AVAB” system user interface. This study was carried out at an academic hospital with 220 beds affiliated with Hamadan University of Medical Sciences.

Study participants and sampling

Three experts participated in the evaluation of “AVAB” system. The first evaluator was a Ph.D. student in Medical Informatics (MI) and had more than 4 years of experience with “AVAB” system. The second evaluator was a Ph.D. student in MI who was familiar with different information systems and various methods of evaluating information systems. The third evaluator was a Master of Health Information Technology and had more than 6 years of experience working with “AVAB” system and had evaluated information systems more than twice. All evaluators were familiar with “AVAB” system and participated in one or more usability evaluations of different information systems.

Data collection tool and technique

In this study, the heuristic evaluation method was used. The heuristic evaluation focuses on user interface design evaluation. Nielsen's heuristics are very common in this method and are used by many evaluators. In the Nielsen heuristics evaluation method, ten principles are evaluated [Table 1].

Table 1

Nielsen’s usability heuristics

Principle (heuristic)

Visibility of system status

Match between system and the real world

User control and freedom

Consistency and standards

Error prevention

Recognition rather than recall

Flexibility and efficiency of use

Aesthetic and minimalist design

Help users recognize, diagnose, and recover from errors

Help and documentation

The heuristic evaluation of the user interface was performed in two phases: navigation and analysis.

Navigation phase

At this stage, users became familiar with the user interface structure of “AVAB” system.

Analysis phase

This stage of the evaluation was performed in three main steps:

Step 1: In this step, each expert independently compared the system interface with Jakob Nielsen's principles and entered the problems in the data collection form.

Step 2: In this step, three lists completed by the evaluators were reviewed. Problems that were common to the lists were removed, and a single list of all problems was created. Any disagreement between the evaluators was resolved through discussion and negotiation.

Step 3: In this step, the severity of the problems was determined based on three criteria: frequency of exposure to risk, impact of the problem on the user experience, and persistence of the problem. Each problem has severity range (0 = no problem, 1 = cosmetic, 2 = minor 3 = major, and 4 = catastrophe); the average severity of each problem was obtained by dividing the sum of the severity to the number of evaluators who identified the problems. Finally, the average severity of the problems was calculated and reported based on Table 2.

Table 2

Nielsen’s severity rating scale for usability problems

Problem	Severity	Description
No problem	0	I don’t agree that this is a usability problem at all
Cosmetic	1	Need not be fixed unless extra time is available on project
Minor	2	Fixing this should be given low priority
Major	3	Important to fix so should be given high priority
Catastrophe	4	Imperative to fix this before product can be released

Ethical consideration

The study was approved by the researcher's institute review board. The approval code number was ID: IR.UMSHA.REC.1398.377. Since “AVAB” is a national system and has all hospitals and staff information, it was tried to maintain the principle of confidentiality during the evaluation and reporting of the results.

Results

In this research, three evaluators conducted the heuristic evaluation for “AVAB” web-based system. However, in the analysis phase, various usability-oriented violations of the mentioned system were extracted by checking Nielsen's principles on it and entered the problems in the data collection form. As such, the evaluation led to the identification of 200 problems, which, after eliminating the common problems, we reached 86 unique problems. The highest frequency of problems belonged to principle 10, “help and documentation,” principle 2, “match between system and the real world” and principle 4, “consistency and standards.” Table 3 shows the frequency of determined usability problems based on their severity and the violated heuristics. The highest average severity of problems was observed in principle “help and documentation” with a score of 3.7 and the principle “flexibility and efficiency of use” with an average severity of 3.4; the severity of these two principles is in the realm of catastrophe problems. Figure 1 presents all the principles of evaluation along with the frequency of their problems. Besides, Figure 2 demonstrates the average severity for each violated problem based on principles of heuristic evaluation. Of the total identified problems, 32 (37.20%) cases were catastrophe problems, and 25 (29.06%) cases were major problems; the detailed results are shown in Figure 3. Eight examples of major and catastrophic problems are shown in Figure 4.

Table 3

Determined usability problems per violated heuristics and severity

Principles of heuristic evaluation	Average severity	Severity				Number of problems	Example
		Cosmetic	Minor	Major	Catastrophe
1. Visibility of system status	2.1	2	0	0	1	3	Present feedback to the user as quickly as possible (ideally, immediately)
2. Match between system and the real world	2.1	3	4	3	1	11	Buttons are not consistently labeled
3. User control and freedom	3.2	0	1	2	4	7	There are not “undo” and “redo” functions
4. Consistency and standards	1.9	3	4	2	1	10	The system does not follow the standards for the function of function keys (F1-F12)
5. Error prevention	1.9	1	2	2	0	5	Error messages do not notify the user of the severity of the error
6. Recognition rather than recall	3.2	0	2	2	4	8	The system does not prevent user errors at any time
7. Flexibility and efficiency of use	3.4	0	2	3	4	9	The function keys (F1-F12) are not arranged in logical groups.
8. Aesthetic and minimalist design	2.5	1	3	1	2	7	The “find next” and “find previous” shortcut keys are not used to search the database
9. Help users recognize, diagnose, and recover from errors	3.1	0	1	0	2	3	Icons are no more visible than the environment in which they are placed
10. Help and documentation	3.7	0	0	10	13	23	Where necessary, the system does not provide any help
Total	2.71	10	19	25	32	86	-

Figure 1

Frequency of usability Heuristic violations

Figure 2

Average severity for each violated problem

Figure 3

Frequency of Heuristic problems by severity rating

Figure 4

Eight examples of major and catastrophic problems. P1: The icons used are not in line with our culture. P2: Do not apply suitable combo buttons or drop-down buttons for select gender. P3: Although the system is designed in Persian, the numbers are in English. P4: The icon for closing the page is very small and does not have an interesting color. P5: Numbers must be left-aligned. P6: The field has no title. P7: Although the amount of salary should be a number, you can also enter a letter. P8: Combo button or drop-down button in some items like national code should not be considered. (AVAB is a national system in the Persian language that was translated into English due to the journal's language.)

Discussion

In the heuristic evaluation of “AVAB” system, a total of 86 unique problems were identified. One of the important results of this study was that more than half of the identified problems were catastrophic and major problems. Furthermore, the mean severity score of the identified problems was 2.71, which was in the category of major problems. Based on the results, the principle of “help and documentation” with a mean severity score of 3.7 and 23 problems was rated as catastrophic. In this system, the principle of “help and documentation” was not considered at all and did not provide any help to users and they were forced to use the paper-based instructions provided by the Ministry of Health. In a study conducted by Nabovati et al. in the evaluation of the radiology and laboratory information system, this principle was rated with an average of 4 in the group of catastrophic problems.[18] Failure to follow this principle by health system designers may cause many difficulties for users, and they may be confused. Without help components, users will not receive any additional explanatory information if they have any ambiguity interacting with the system, making it difficult for users to continue working with the system. It should be noted that these problems can be easily solved and provide great help to users; therefore, it is suggested that the designers of health systems take action to solve these problems.

The principle of “flexibility and efficiency of use” with a mean severity score of 3.4 and 9 problems was one of the less considered items in our study. One of the points mentioned in this principle is the lack of definition of function keys (F1-F12) for commands that are utilized frequently by “AVAB” users. Another point mentioned in this principle is that the shortcut keys “find next” and “find previous” were not used to search the system. Consequently, in a study conducted by Sadeqi Jabali et al. to evaluate the admission and medical record information system, the least number of violations (2 problems) was related to the principle “flexibility and efficiency of use.”[19] Usability evaluation of a national health information system by Rangraz Jeddi et al. showed only two problems with this principle.[20] The results of these two studies did not match with the results of our study. Therefore, according to the results of this study, it is recommended to define functional keys for the convenience of working with this system, so that users do not have to use the mouse too much to perform repetitive tasks, and solving this problem can increase users’ speed and accuracy in working with the system.

In terms of the number of problems identified, the two principles “help and documentation” and “match between system and the real world” had the highest number of problems with 23 and 11, respectively. In the study by Rangraz Jeddi et al., these two principles had the least frequency with one problem and did not agree with this study's results.[20] This principle can be improved by rationally arranging the menu options and choosing the color according to the usual expectations for color codes.

Another identified problem with a mean severity score of 3.2 was the principle of “recognition rather than recall.” This study's results are consistent with several studies that can be referred to Joshi et al.'s study, which accounted for 62% of the problems related to this principle.[21] Furthermore, this study's results were consistent with Chan et al. and Khajouei et al.;[2223] items that the designers ignored in this principle were the lack of distinction between active menus and inactive menus, and the menus that allowed the user to select options were not clear. To comply with the principle of “recognition rather than recall,” developers must design systems so that users require minimal mental and physical activity to perform actions. By following these principles, users do not need to memorize much information to carry out their tasks.

Since “AVAB” is an information system used throughout Iran and solving its usability problems is very important, the results of this study can be used by the Ministry of Health.

Limitation and recommendation

In our research, we faced some strengths and limitation points. Nielsen questionnaire has ten broad principles that evaluate various aspects of an information-oriented system. Compared to other data collection methods like observation or interview, a structured questionnaire facilitated collecting data, and evaluators can represent more problems and violations with the aforementioned health information system. The limitation of this study was that, because the Ministry of Health supported this health information system, access to this system required coordination with the relevant organization. Future studies suggest distinguishing the causes of usability problems and difficulties utilizing other usability evaluation techniques such as “think aloud” and “cognitive walkthrough.”

Conclusions

Nowadays, health information systems are widely developed and used in the health industry and settings. That's why, in this study, we decided to evaluate the usability status of a national health system called “AVAB” based on Nielsen's Ten Principles. The usability problems identified in “AVAB” system were in the category of major problems; however, many of these usability gaps can be easily corrected by adhering to system design principles and standards. According to the results of this study, usability problems such as unavailability of help and documentation, user control and freedom, recognition rather than recall, flexibility and efficiency of use, have been the most significant problems that lead to the user not interacting effectively with the system. As it turns out, improving the usability of health information systems is a critical step; this is achieved when we follow existing design principles and standards from the early phased of the systems development life cycle.

Financial support and sponsorship

This work was supported by a grant from Hamadan University of Medical Sciences Research Council with the ID 9805153699.

Conflicts of interest

There are no conflicts of interest.