Usability Evaluation of Three Admission and Medical Records Subsystems Integrated into Nationwide Hospital Information Systems: Heuristic Evaluation.

INTRODUCTION: Usability is one of the quality criteria for information systems and its weakness is one of the main barriers to the adoption of these systems. The purpose of this study was to evaluate the usability of admission and medical records module of three widely used hospital information systems (HISs).
METHODS: In this descriptive study the usability of admission and medical records module of three HISs (HIS1, HIS2, and HIS3) was evaluated using heuristic evaluation method. For each HIS, three expert users of the same system assessed the user interface independently, completed a usability evaluation checklist, and rated severity of each identified problem. The checklist was based on Nielsen's heuristics. For each HIS, three heuristics that have the highest and lowest problem rates and greatest severity of problems were categorized into three separate groups. The results were analyzed using descriptive statistics.
RESULTS: Although HIS1 and HIS2 were used in more hospitals than HIS3, the results showed that the usability problem rates of them were significantly higher than HIS3. The heuristics of "help and documentation", "flexibility and efficiency of use", and "visibility of system status" in the three HISs were categorized into the "highest rate of problems", "lowest rate of problems", and "highest severity of problems" groups, respectively. The heuristics of "diagnose and recover from errors", "error prevention", and "help and documentation" in HIS1 and HIS2 were categorized into the "highest rate of problems" group.
CONCLUSIONS: The results of this study and previous studies show that the most common usability problems with HISs are related to heuristics of "help and documentation", "error prevention", and "help users recognize, diagnose and recover from errors." Also, the large number of hospitals using one HIS does not demonstrate its high usability to others.

INTRODUCTION

Hospital information system (HIS) is one of the most important and most widely used information systems in the health care (1-3). The use of HIS in hospitals has many advantages, including the automating clinical and administrative processes, facilitating and speeding up information exchange between hospital departments, generating accurate and timely reports, improving performance of healthcare providers, increasing patient satisfaction, and reducing costs (1-7). But despite the many benefits, HISs may face difficulties that prevent their successful implementation (3, 8), so their quality needs to be continuously evaluated.

Usability in one of the criteria for the quality of information systems which is evaluated through the assessment of their user interface (UI). Problems with the UI can lead to increased user errors, decreased user satisfaction, and reduced efficiency level, and are an important barrier for the adoption of information systems (8-10). Therefore, identifying UI problems seems to be necessary through the usability evaluation. The Agency for Health Care Research and Quality (AHRQ) has commissioned studies on the usability evaluation of electronic health record systems (EHR) to enhance effectiveness and efficiency (11).

So far, the usability of various healthcare information systems has been evaluated in many studies, which often reported high usability problems and suggested that these types of problems have negative consequences on interaction of users with systems (9, 12-14). In a study evaluating the usability of an electronic medication administration record system, 60 usability problems were identified, with a significant number of major and catastrophic problems. It was reported that these problems can decrease users’ efficiency, effectiveness and satisfaction (15). Another study evaluated the usability of the UI of a laboratory and radiology module of an HIS. The results of that study showed that despite the extensive use of that system in many hospitals, its UI had a significant number of problems with severity of major and catastrophe types, which can delay physicians’ access to the results of laboratory tests and radiology (16). Recently, a study evaluated the data entry module of an electronic patient record system and the results showed that 40% of the navigational actions of experts had deviated from predefined next system action, mainly due to the lack of compliance between system design and user expectations (14).

One of the main modules of each HIS is the admission and medical records module, which is used for performing numerous processes such as admitting patients, controlling hospital beds, planning diagnostic and therapeutic interventions, registering information on patient transmissions from hospitals, and discharge (4). Since most of the processes in the medical records department are performed through HIS, the problems with the UI of the admission and medical records module can have negative effects on the users’ performance, and make it difficult to provide services to patients. Therefore, it seems necessary that the UI problems of this subsystem are identified through the usability evaluation and measures are taken in order to resolve them.

According to our knowledge, no studies have evaluated and compared the usability of the admission and medical records modules of multiple HISs. Therefore, the purpose of this study was to evaluate the usability of the admission and medical records modules of the three widely used HISs.

METHODS

The present study was a descriptive and cross sectional study conducted in 2016 in Iran. In this study, the usability of the admission and medical records module of the three HISs were evaluated and compared using heuristic evaluation method.

Research population

Three evaluated HISs (HIS1, HIS2, HIS3) were developed by three distinct companies, each of them has been operating on the design and implementation of HISs for about two decades in Iran. At the time of this study, HIS1, HIS2, and HIS3 were used in about 300, 90, and 40 hospitals and clinics, respectively.

Study setting

This study was carried out at three hospitals in Iran; the first hospital, a private general hospital with 230 beds in Yazd city with HIS1, the second hospital, an academic 460-bed heart hospital affiliated to Tehran University of Medical Sciences with HIS2, and the third hospital, an academic hospital with 530 beds affiliated with Qom University of Medical Sciences with HIS3.

Data collection tool

Xerox Corporation’s checklist was used to perform the heuristic evaluation (17). This checklist contains 252 items in the form of the 13 principles of usability. These principles and their descriptions are shown in Table 1.

Table 1.

Usability principles

Principle (heuristic)	Description
Visibility of system status (visibility)	The system should always keep user informed about what is going on, through appropriate feedback within reasonable time.
Match between system and the real world (match)	The system should speak the user’s language, with words, phrases and concepts familiar to the user, rather than system-oriented terms.
User control and freedom (control)	Users should be free to select and sequence tasks, rather than having the system do this for them.
Consistency and standards (consistency)	Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.
Help users recognize, diagnose, and recover from errors (undo)	Error messages should be expressed in plain language.
Error prevention (error)	Even better than good error messages is a careful design which prevents a problem from occurring in the first place.
Recognition rather than recall (memory)	The user should not have to remember information from one part of the dialogue to another.
Flexibility and efficiency of use (flexibility)	Allow users to tailor frequent actions. Provide alternative means of access and operation for users who differ from the “average” user.
Aesthetic and minimalist design (minimalist)	Dialogues should not contain information which is irrelevant or rarely needed.
Help and documentation (help)	Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation.
Skills	The system should support, extend, supplement, or enhance the user’s skills, background knowledge, and expertise.
Pleasurable and respectful interaction with the user	The user’s interactions with the system should enhance the quality of her or his work-life.
Privacy	The system should help the user to protect personal or private information.

The first ten principles of the checklist are based on the Nielsen’s usability heuristics, and this study was conducted only on the basis of Nielsen’s usability heuristics. Each item of the checklist was given one of the three answers of ‘Yes’, ‘No’, and ‘Not applicable’. Answer ‘Yes’ means that the item is considered in the user interface and there is no usability problem. Answer ‘No’ means that the item is not observed in the user interface and there is a usability problem (the heuristic is violated), and answer ‘not applicable’ means that this item is not applicable for the evaluated UI.

Stages of the study

This study was conducted in four stages:

First stage: The usability evaluation checklist was translated into Persian language. Then its validity was verified by three experts (a health information management specialist, a medical informatics specialist, and a computer engineering specialist), and was approved by them after being edited three times.

Second stage: Considering that the heuristic evaluation can be done with at least three evaluators (18), three expert users who had bachelor’s degrees in software engineering, health information technology (IT), and medical records were negotiated from each hospital. These users were fully familiar with HISs and had at least five years of experience with these HISs at the same hospital. Khajouei et al. in their study recommended that in heuristic evaluation it is better to recruit evaluators that are familiar with the system (19). The selected users (evaluators) respectively played a role in hospitals as HIS director, health information technology director, and medical record expert. After describing the research objectives for the evaluators, all sections and items of the checklist were explained to them. They expressed their consent to participate in the study.

Third stage: Evaluators independently inspected the user interface of admission and medical records module and completed the checklist based on it. For each item receiving ‘No’ response (the existence of a problem), its severity was rated by the evaluator based on the Nielsen’s rating scale (Table 2)(20). The evaluators were asked to rate the severity of each usability problem by considering the three factors of frequency (is the problem common or rare?), impact (is it difficult or easy for the end users to take over the problem?) and persistence (does it trouble the end users repeatedly or is it a one-time problem?) (21).

Table 2.

Nielsen’s severity rating scale for usability problems

Problem type	Severity	Description
No problem	0	I don’t agree that this is a usability problem at all.
Cosmetic	1	Need not be fixed unless extra time is available on project.
Minor	2	Fixing this should be given low priority.
Major	3	Important to fix, so should be given high priority.
Catastrophe	4	Imperative to fix this problem.

Forth stage (statistical analysis): Data collected from the evaluators was analyzed and reported in two following methods using descriptive statistics in Excel 2013:

First analysis method (aggregation): For each HIS, the total number of ‘No’ responses of the three evaluators divided by the sum of the number of items answered ‘Yes’ and ‘No’ were considered as the usability problems rate. The overall severity of problems for each HIS was derived from the average scores of the three evaluators.

Second analysis method (voting): For each HIS, the final response for each item was determined by voting between the three evaluators, so that if two or three evaluators answered ‘Yes’ to an item, the final result was considered ‘Yes’. If two or three evaluators answered ‘No’ to an item, the final result was considered ‘No’. In the rest of the cases the answer was ‘Not applicable’. The usability problems rate of each HIS was obtained from the ratio of the number of items assigned ‘No’ answer to the total number of items that were responded ‘Yes’ and ‘No’. The severity of each problem was also calculated from the average severity reported by the evaluators.

In both analytical methods, for each HIS, three heuristics that had the highest and the lowest number of problems and the greatest severity of problems, were classified into three separate groups:

“Highest rate of problems” group: It refers to the three heuristics that have the usability problems rate more than the others.

“Lowest rate of problems” group: It refers to the three heuristics that have the usability problems rate less than the others.

“Greatest severity of problems” group: It refers to the three heuristics that have the severity greater than the others.

RESULTS

Table 3 shows the results of the evaluation of three admission and medical records modules using the first analysis method (aggregation). The median of usability problems rates in HIS1, HIS2, and HIS3 were 30.4%, 24.5%, and 6.4%, respectively. The median severity of identified problems for HIS1, HIS2, and HIS3 were 2.8%, 1.6%, and 1.5%, respectively.

Table 3.

The rate and severity of the usability problems of the three HISsa for each of Nielsen ten heuristics using aggregation analysis method. aHIS: Hospital Information System

Usability heuristics	HIS1		HIS2		HIS3
	Rate (%)	Severity	Rate (%)	Severity	Rate (%)	Severity
1. Visibility	28.2	2.8	27.1	1.9	3.4	2.3
2. Match	33.3	2.6	20.6	1.3	2.8	0.5
3. Control	23.1	3.1	23.1	2	13.2	1.4
4. Consistency	14.6	2.6	23.2	1.3	9.2	1.7
5. Undo	46.6	2.1	53.4	1.4	8.2	1.6
6. Error	38.9	2.2	70.3	2.1	7.1	0.3
7. Memory	25	2.8	15	1.6	5.0	2.2
8. Flexibility	32.6	2.7	15.9	1.7	4.4	0.5
9. Minimalist	18.8	2.8	25.7	1.6	5.6	1.5
10. Help	45	2.8	39.7	1.4	29	1.2
Median (IQR1-IQR3)*	30.4 (23.6-37.5)	2.8 (2.6-2.8)	24.5 (21.2-36.6)	1.6 (1.4-1.9)	6.4 (4.6-0.9)	1.5 (0.7-1.7)

Table 4 shows the results of the usability evaluation of the admission and medical records modules of the three HISs by the second analysis method (voting). The median usability problems rates for HIS1, HIS2, and HIS3 were 12.3%, 13.6%, and 5.5%, respectively. The median severity of identified problems in HIS1, HIS2, and HIS3 were 2.4%, 1.4%, and 1.6%, respectively.

Table 4.

The rate and severity of the usability problems of the three HISs for each of Nielsen ten heuristics using voting analysis method.

Usability heuristics	HIS1		HIS2		HIS3
	Rate (%)	Severity	Rate (%)	Severity	Rate (%)	Severity
1. Visibility	12	2.7	18.5	1.7	3.5	2
2. Match	19.1	2.4	9.1	1.3	0	-
3. Control	0	-	14.3	1.8	8.7	1
4. Consistency	2.5	2.5	12.8	1.4	5.9	1.7
5. Undo	50	1.8	64.7	1.4	5	2
6. Error	33.4	2	72.7	2.2	7.1	0.5
7. Memory	12.5	2.1	0	-	2.5	2
8. Flexibility	0	-	0	-	0	-
9. Minimalist	0	-	0	-	8.3	1.5
10. Help	27.8	2.4	30.4	1.4	34.8	1.2
Median (IQR1-IQR3)*	12.3 (0.6-25.6)	2.4 (2.1-2.4)	13.6 (2.3-26.4)	1.4 (1.4-1.6)	5.5 (2.8-8.0)	1.6 (1.2-1.8)

Table 5 shows three heuristics that had the highest and the lowest number of problems and the greatest severity of problems for the three HISs. The heuristic of Help was placed in the “highest rate of problems” group in the three HISs. The heuristic of flexibility was placed in the “lowest rate of problems” group in the three HISs. The heuristic of visibility was placed in the “greatest severity of problems” group in the three HISs. The heuristics of undo, error, and help in both HIS1 and HIS2 were placed in the “highest rate of problems” group. Comparison of the rate and severity of usability problems in HIS2 showed that the heuristics of help and error were simultaneously placed in the “highest rate of problems” and “greatest severity of problems” groups.

Table 5.

Three principles with the highest and the lowest rate of usability problems and the greatest severity of problems in the three HISsa. aHIS: Hospital Information System

Groups	Cumulative analysis			Voting analysis
	HIS1	HIS2	HIS3	HIS1	HIS2	HIS3
Highest rate of problems	Undo Error Help	Undo Error Help	Control Consistency Help	Undo Error Help	Undo Error Help	Control Minimalist Help
Lowest rate of problems	Control Consistency Minimalist	Match Memory Flexibility	Visibility Match Flexibility	Control Flexibility Minimalist	Memory Flexibility Minimalist	Match Memory Flexibility
Highest severity of problems	Visibility Control Help	Visibility Control Error	Visibility Consistency Memory	Visibility Consistency Help	Visibility Control Error	Visibility Undo Memory

DISCUSSION

In this study, the usability of the admission and medical records modules of the three commonly used HISs were evaluated and the identified problems were analyzed and reported in terms of Nielsen’s ten principles. The results showed that the usability problems of HIS1 and HIS2 were significantly higher than those of HIS3. The median severity of HIS1 problems was significantly higher than the other two HISs. The heuristics of help, flexibility, and visibility were categorized in the “highest rate of problems”, “lowest rate of problems”, and “highest severity of problems” groups in the three HISs, respectively. The heuristics of undo, error, and help in HIS1 and HIS2 were identically categorized in the “highest rate of problems” group.

Despite the fact that HIS1 and HIS2 were used in more hospitals and clinics than HIS3, the results showed that the usability problems of these two HIS were significantly higher than HIS3. This finding shows that the large number of hospitals using one HIS is not a reason that the system has fewer defects compared with other systems. Also, although the number of hospitals using HIS1 is more than the other two HISs, the results showed that its median severity of problems was classified as major and more than two other HISs. In a similar study on another widely used HIS, the severity of 67% of problems was classified as major and catastrophe types (16). In another study, the average severity of problems of an electronic medication administration record system was reported to be major (15). The results of this study and similar studies show that, despite the widespread use of information systems in healthcare, some of them have usability problems that are highly severe. Since usability is one of the factors influencing the successful implementation of health information systems (22, 23), it is necessary to assess whether the implementation of information systems has been effective with the usability problems faces by many users.

The flexibility heuristic was placed in the group of “lowest rate of problems” in three evaluated HISs. This heuristic relates to the use of shortcuts (such as functional keys of F1-F12) for frequently used commands to accelerate user interaction with the system. Similarly, some other studies have shown that this heuristic has fewer problems than the other heuristics (16, 24). The low rate of problems associated with this heuristic may have been due to the evaluators expertise in interaction with HISs. Obviously, the more users have more experience in working with information systems, they will have more skills in interacting with the system than the novice users and will operate more efficiently (25, 26). In addition to using shortcuts, if users have the possibility of automatically entering data (e.g. based on previously entered values), they will interact with the system more quickly and make fewer errors while entering the data.

The help heuristic was placed in the group of “highest rate of problems” in three evaluated HISs. This result is consistent with the results of two previous studies (24, 27). This finding shows that HIS designers do not pay necessary attention to help and documentation and may consider it as an unnecessary and secondary functionality (28). In confirming this opinion, Alexander et al. showed that out of the 27 nursing information websites, only 11% of the them offered documentation and help (29). It should be noted that a guide in information systems can help user training, and the absence or weakness of it can lead to the user confusion while interacting with the system. In another study, we also showed that user training is one of the main requirements of electronic health records (30).

The heuristics of undo, error, and help in HIS1 and HIS2 were identically placed in the “highest rate of problems” group. Violation of these heuristics can increase user errors while interacting with the information system. The error heuristic is related to elimination of error-prone conditions or provision of a confirmation warning to users before an action are committed. Similarly, a study showed that error heuristic was not greatly considered in the UI of an electronic medication administration record system (15). Designers should note that consideration of this heuristic in the UI, such as displaying default values in the fields, displaying the allowed values for each field, and displaying the format and mask for the data, can prevent the user from making an error. The undo heuristic is relates to helping the user in recognizing, diagnosing, and recovering from errors. This heuristic emphasizes the relevance and the clarity of notifications and error messages. Error messages should also be made in such a way to inform the user of the severity and cause of errors and help him/her correct them.

The visibility heuristic was placed in the group of “highest severity of problems” in the three HISs. Consistent with this result, some similar studies reported that problems related to this heuristic were of major severity (14, 16). The high severity of problems of this heuristic reflects the undesirable consequences for users caused by the invisibility of the systems being evaluated. This heuristic is about understanding the current status of the system by the user and the possibility of deciding about next actions; so violating this heuristic may causes user confusion and dissatisfaction (10). To reduce the severity of problems related to this heuristic, it is recommended that the UI of information systems to be designed in such a way that it always informs user of system status and task progress through proper feedback.

This study has two strengths. First, the usability problems were identified using a checklist. The checklist can regulate the heuristic evaluation process and avoid subjective comments about user interface problems. Secondly, although the results of this study on the HISs problems may not be generalizable to other health information systems, these results are consistent with the results of similar studies performed on HISs in many cases (15, 16, 24, 27). This study also has two limitations that warrant consideration. Given that heuristic evaluation is performed only by expert evaluators without the participation of real users, the real problems of the users with the system may not be identified in this method. To reduce this limitation, evaluators were selected from individuals who were experts in HISs and were in contact with real users on a daily basis and familiar with their issues. Another limitation of this study is that since the checklist was used, there may be problems with evaluators that do not exist in the checklist. However, at the end of the evaluation process, the evaluators stated that the checklist was comprehensive and covered all existing problems.

The results of this study are applicable to chief information officers and IT managers of hospitals and HIS companies as follows. If the user interface problems identified in this study are solved in HISs and their usability is improved, the end-users of the systems will need less support from companies or hospitals IT experts. The results of this study can also be of great help to chief information officers and IT managers of hospitals to select new information systems with high usability.

Considering that in this study the evaluation was conducted by expert users, and the type of the identified problems may be different from those of the novice users, it is recommended that these HISs be evaluated in subsequent studies using the comments of the novice users and the results be compared with the results of the present study. Considering the wide use of HISs evaluated in a large number of hospitals in Iran and the existence of a significant number of usability problems in them, it is suggested that the usability principles be considered in the design of their user interfaces and their usability be evaluated by experts and end-users prior to implementation of them.

CONCLUSION

The results of this study and previous studies indicate that despite the widespread use of information systems in hospitals, these systems often have many usability problems. Also, the large number of hospitals that use one HIS cannot demonstrate its high usability to others. The most common usability problems with HISs are related to heuristics of “help and documentation”, “error prevention”, and “help users recognize, diagnose and recover from errors” and the least common problems are related to the “flexibility and efficiency of use” heuristics.