Usability Evaluation of an Admission, Discharge, and Transfer Information System: A Heuristic Evaluation.

BACKGROUND: Admission, discharge and, transfer (ADT) process is one of the most important hospital workflows. ADT system is a part of a hospital information system (HIS). AIM: The objective of this study was to evaluate the usability of the ADT system.
METHODS: The study performed at Mashhad University of Medical Sciences (MUMS) hospitals. Data collection instrument was a validated checklist of Pierotti heuristic evaluation. To determine the severity of usability problems, a hybrid of Nielson and Tampere unit for computer-human interaction (TAUCHI) severity scaling algorithm was used. Usability problems were divided into five categories (major, severe, minor, cosmetic, and technical). Six experts evaluated the ADT system independently. According to TAUCHI severity scale, if a feature has not yet been implemented in the ADT system, evaluators considered it a technical usability problem. Therefore, usability problems due to non-design feature in the ADT system were identified. Finally, the mean severity of each usability problems was calculated.
RESULTS: A total of 186 usability problems were identified. The frequency of major, sever, minor and cosmetic usability problems were 2, 65, 69 and 50, respectively. A total of 55 usability problems by the evaluators were recognised as technical problems. The highest mismatch with usability principles was related to the "recognition rather than recall". The range of the mean severity of usability problems was between 0-2.31.
CONCLUSIONS: Our result showed that although implementation of IHIS on a large scale, it still suffered from unresolved usability problems. Identification of usability problems and evaluation of their level of severity, which was simultaneously performed in this study, can be used as a guide to evaluate the usability of other HISs.

Introduction

Hospital information system (HIS) refers to a collection of integrated software systems used to collect, store, retrieve and present patients’ data and information in a hospital. HIS has several components such as radiology, lab and nursing information system [1]. Although using health information technology potentially reduces serious damage to patients [2], some of the previous studies revealed the unintended adverse consequences of this technology, including increased documentation time, inconsistency with clinical workflow, increased error rising in patients’ treatment [3].

Zheng states “Although health information systems promise healthcare improvement as well as medical error decrease, in the case where these systems are not used appropriately, it seems impossible to gain effective results. Factors, such as poorly designed user interface and inconsistency between system and work process, can cause unintended adverse consequences” [4].

Moreover, if an information system is unable to meet users’ basic expectations, it will miss their trust gradually, and its efficient use will be deteriorated time by the time [5]. Therefore, it is highly significant to remove the usability problems of health information systems and to prevent their unexpected adverse consequences [6]. Usability is one of the key dimensions for the software quality, especially HIS. Usability evaluation supports a collection of parameters determining the system quality [7]. Usability evaluation generally includes two steps: first, usability problems identification and the the determination of their severity [8]. Heuristic evaluation is one of the most common methods to identify usability problems. These methods find usability problems with a minimum amount of time, cost and resources [9]. In this method, a small group of evaluators, based on a predetermined checklist and according to the usability evaluation principles, evaluates the system [10]. Severity rating can determine which serious usability problems require to be fixed immediately and can be used to allocate the relevant resources to repair them [8].

In this study, we evaluated the usability of the ADT system utilised in MUMS hospitals.

Methods

In 2002, MUMS implemented a customised HIS, namely IHIS (Iranian Hospital Information System), with its subsystems, including nursing, pharmacy and ADT system. Now, the IHIS is implemented in 26 hospitals of Khorasan Razavi Province and 10 healthcare centres. All these healthcare organisations used the same version of IHIS, and its update is carried out by the HIS Group of the MUMS IT Center.

This study was descriptive and cross-sectional. The study was conducted on April 2017 in the five selective MUMS hospitals (two general and three speciality hospitals). The heuristic method was used to detect the ADT system usability problems.

Data collection instrument was a validated checklist of Pierotti heuristic evaluation. This checklist includes 13 heuristic evaluation principles and 292 questions [11] (Table 1).

Table 1

The Pierotti ’s Heuristic evaluation principles [11]

Visibility of system status (n = 29)	Flexibility and minimalist design (n = 16)
Match between system and the real world (n = 24)	Aesthetic and minimalist design (n = 12)
User control and freedom (n = 23)	Help and documentation (n = 23)
Consistency and standards (n = 51)	Skills (n = 21)
Help users recognize, diagnose, and recover from errors (n = 21)	Pleasurable and respectful interaction with the user (n = 14)
Error prevention (n = 15)	Privacy(n = 3)
Recognition rather than recall (n = 40)

To determine the severity of usability problems, a hybrid of Nielson severity rating scale [12] and Tampere unit for computer-human interaction (TAUCHI) severity rating scale [13] was used (Table 2).

Table 2

Hybrid severity scaling algorithm for usability problems

Severity scale	Definition
Major	Imperative to fix this before the product can be released (Score = 4) (10)
Severe	Important to fix. Therefore it should be given high priority (Score = 3) (10)
Minor	Fixing this should be given low priority (Score = 2) (10)
Cosmetic	It should be fixed to use the system to be as pleasant as possible (Score = 1) (10)
No problem	Usability problem do not exist (Score = 0) (10)
Technical	These usability problems are most likely due to technical problems with the system. Features that have not been implemented yet (Score = 0-4 according to evaluators’ opinion) (11)

Six experts were recruited as usability evaluators: One health information management specialist with two years of practical work experience with IHIS in a clinical setting, two master students in medical informatics with computer engineering background, one Ph.D. student in medical informatics with 7 years IHIS management experience and two information technology specialists with 15 years of IHIS management experience. All the usability evaluators had participated in the heuristic evaluation course. To improve the quality of the usability evaluations, several sessions were held to equalise the personal interpretations of the checklist items.

This study was conducted in two phases: First, each evaluator based on Pierotti checklist evaluated the user interface of the ADT system, and then evaluators determined the severity of each usability problem (according to hybrid severity scaling algorithm in Table 2). According to TAUCHI scale, if a feature has not been implemented in the ADT system yet, evaluators considered it a technical usability problem. Therefore, usability problems due to non-design or non-existence of a feature in the ADT system were identified, and the severity of them was determined. Second, the mean severity of each usability problem was calculated.

Results

Heuristic evaluation was conducted on the ADT system by 6 evaluators. A total of 210 usability problems were recognised. After eliminating the duplicates problems, 186 unique usability problems remained. The severity of the identified problems included: 2 (1%) major problems, 65 (35%) severe problems, and 69 (37%) minor problems and 50 (27%) cosmetic problems. A total of 55 usability problems identified by the evaluators were recognised as technical problems.

The range of the mean severity of usability problems was between 0-2.31. The “help and documentation” principle had the highest level of problem severity. The “error prevention” and “flexibility and minimalist design” were ranked as the 2nd and 3rd principles, respectively.

The “recognition rather than recall” principle with 32 problems and the mean severity of 1.08 had the highest number of usability problems. The “help and documentation” and “consistency and standards” were ranked as the 2nd and 3rd principles, respectively.

There was only one problem reported on the principle of “privacy” (Table 3). Table 4 showed a sample of the most important identified usability problems based on their severity.

Table 3

Frequency and the mean severity of identified usability problems

Heuristic evaluation principal	Number of usability problems	Mean (Range) of the severity of problems
Visibility of system status	Total= 19 (C=8, Mi=8, S=3, and Ma=0)	0.95 (0.00 – 2.31)
Match between system and the real world	Total=14 (C=1, Mi=6, S=7, and Ma=0)	1.16 (0.00 – 2.50)
User control and freedom	Total=11 (C=1, Mi=6, S=4, and Ma=0)	1.00 (0.00 – 3.00)
Consistency and standards	Total=21 (C=6, Mi=12, S=3, and Ma=0)	0.60 (0.00 – 2.50)
Help users recognize, diagnose, and recover from errors	Total=13 (C=2, Mi=5, S=6, and Ma=0)	1.25 (0.00 – 2.66)
Error prevention	Total=11 (C=0, Mi=7, S=3, and Ma=1)	1.46 (0.00 – 2.50)
Recognition rather than recall	Total=32 (C=20, Mi=5, S=6, and Ma=1)	1.08 (0.00 – 2.50)
Flexibility and minimalist design	Total=13 (C=2, Mi=4, S=7, and Ma=0)	1.74 (0.00 – 3.00)
Aesthetic and minimalist design	Total=7 (C=5, Mi=2, S=3, and Ma=0)	0.98 (0.00 – 2.50)
Help and documentation	Total=23 (C=2, Mi=7, S=14, and Ma=0)	2.31 (1.00 – 3.00)
Skills	Total=13 (C=1, Mi=4, S=8, and Ma=0)	1.42 (0.00 – 3.00)
Pleasurable and respectful interaction with the user	Total=8 (C=1, Mi=3, S=4, and Ma=0)	1.20 (0.00 – 3.00)
Privacy	Total=1 (C=1, Mi=0, S=0, and Ma=0)	0.22 (0.00 – 2.16)

Note: C = Cosmetic, Mi = Minor, S = Sever, and Ma = Major.

Table 4

Samples of the identified usability problems based on their severity

Samples of major usability problems

-The content of fields in the ADT did not match with the work process.

Samples of severing usability problems

-The ADT pages did not have appropriate titles.

-Extra data elements were displayed on the data entry pages.

-Data elements were not classified properly and did not have a logical sequence.

-Users did not have the choice of either clicking on menu items or using a keyboard shortcut

-The hand and eye movements between input devices were not minimised.

-When the users entered into a screen or dialogue box, the cursor was not positioned on fields and menus which users most likely to need.

-The origin of the system problems and their solutions was not demonstrated in the error messages.

-The ADT did not warn users if they made a potentially serious error.

Samples of minor usability problems

-Fields and menus were not visually distinct.

-If there were observable delays (greater than fifteen seconds) in the system’s response time, the user was not informed of the system’s progress.

-Patients’ information was not retrieved easily and correcting the mistakes was very difficult.

-Users could not customise the system colour coding.

-It was impossible to save patients’ information temporarily in the ADT.

Samples of cosmetic usability problems

-Various and distinctive colours and voices were not used in the ADT.

-Bold fonts were not used to attract users’ attention.

-Visible symbols for active window were not used.

Note: ADT= Admonition, discharge, and transfer.

Discussion

In this study, we used a heuristic evaluation checklist as well as a hybrid severity scaling algorithm to evaluate the ADT system interface. The study findings suggested many technical usability problems in the ADT system.

The results of this study showed that the “recognition rather than recall” and “consistency and standards” principles had the most usability problems, respectively. The findings of the study by Nabovati et al., on a heuristic evaluation of IHIS radiology and lab information subsystem [14] and Khajouei et al., on IHIS emergency subsystem [9] also reported the same problems. In the above two studies, these principles were reported to have the most frequent usability problems.

The findings of the study by Rezaei et al., [15] also revealed numerous usability problems in the principle of “recognition rather than recall”. The findings of the usability evaluations of Agharezaee [6], Abedi and Khajouei [16], Thyvalikakath [17] and Verheul [18] on other HISs as well as electronic health records also showed numerous usability problems in the principle of “consistency and standards”. As well, findings of Sadoghi [19], Meydani [20] and Asadi [21] illustrated that the standards are not adopted in HISs, while access to standard data in an organised format to provide appropriate and on-time healthcare service is highly significant [22]. It seems that the HISs have the same usability problems. Designers wishing to develop or update on HIS should pay particular attention to HIS’s usability problems.

In this study, the smallest number of usability problems was related to the principle of “privacy”, while in the study by Nabovati et al., the smallest number of usability problems was reported in the principle of “help and documentation” [14]. The reason for the difference between the present study and their study is possibly concerned with detecting technical usability problems in the present study. System help was not designed in IHIS. Thus, Nabovati et al. did not evaluate usability problems in the principle of “help and documentation”. In this study, features that have not been implemented, were considered technical usability problems, and the level of their severity was determined. Therefore, in the present study, the number of problems related to the “system help” principle was ranked 2nd. This indicates that the hybrid methods used in this study have the potential to detect a larger number of usability problems.

The findings of our study on the Pierotti heuristic evaluation checklist also reveal that the principle of “help and documentation” has the highest level of problem severity. The findings of previous studies on other HISs in Iran [6] [9] [14] [15] [16] are also indicative of either the absence of system help design or design deficiency. The users in the study by Kimiafar et al. reported the use of guidelines, instructions and educational papers as a solution to address this problem [23].

One of the key findings in the present study detecting serious (major and severe) usability problems in the ADT system. A total of 36.5% of the identified usability problems were classified as major and severe. Although it was lower than the severity rating reported in the study by Nabovati et al., (%66 of the identified usability problems in their study were classified as severe and major) [14], it is still a considerable concern. This is because the high severity of usability problems can negatively impact the users’ interaction with the system. This not only makes them dissatisfied but can also affect data entry and documentation quality. On the other hand, the studies by Khajouei et al., and Nabovati et al., were performed on other IHIS subsystems four and three years ago, respectively [9] [14]. Our study was conducted in 2017, which shows that IHIS still suffers from unresolved usability problems.

One of the major strengths of the current study was the use of professional evaluators, who had previous experience of practical work with IHIS in clinical settings. They were also able to recognise the system’s problems as well. Moreover, our usability evaluation method was an easier and cheaper method to study numerous usability problems in a rather short period. This makes it superior to other available methods. As well, the findings of the present study can be generalised to other IHIS subsystems since many of the IHIS features are similar in various subsystems. Therefore, the findings of this study can help system designers to overcome IHIS usability problems in future editions. The designers of other HISs can also benefit from the findings of this study: they can be informed of the usability problems and their effects on the users’ workflow. This can be used to prevent the same usability problems in others HISs. Identification of usability problems and evaluation of their level of severity, which was simultaneously performed in this study, can be used as a guide to evaluate the usability of other IHIS subsystems and other HISs.

In conclusion, in this study, we used a heuristic evaluation checklist as well as a hybrid severity scaling algorithm to evaluate the hospital information system interface. Our usability evaluation method was an easy and cheap method to study numerous usability problems in a rather short period. This makes it superior to other available methods. The findings of this study can help system designers to overcome hospital information system usability problems. As well, it can be used as a guide to evaluate the usability of other hospital information systems. As well, the current study performed at one the largest provinces in a developing country. Our result showed that although IHIS was implemented on a large scale in a developing country, it still suffered from unresolved usability problems.

According to the current study findings, it seems that designers wishing to develop or update on HIS should pay particular attention to HIS’s to the principles of “consistency and standards” and “recognition and recall” and produce a better HIS. Moreover, the repetition of problems in the principle of “help and documentation” in this study and other previous studies reveals the importance of caring about HIS education and the creation of appropriate instruction for using the system.