A Modified User-Oriented Heuristic Evaluation of a Mobile Health System for Diabetes Self-management Support.

Mobile health platforms offer significant opportunities for improving diabetic self-care, but only if adequate usability exists. Expert evaluations such as heuristic evaluation can provide distinct usability information about systems. The purpose of this study was to complete a usability evaluation of a mobile health system for diabetes patients using a modified heuristic evaluation technique of (1) dual-domain experts (healthcare professionals, usability experts), (2) validated scenarios and user tasks related to patients' self-care, and (3) in-depth severity factor ratings. Experts identified 129 usability problems with 274 heuristic violations for the system. The categories Consistency and Standards dominated at 24.1% (n = 66), followed by Match Between System and Real World at 22.3% (n = 61). Average severity ratings across system views were 2.8 (of 4), with 9.3% (n = 12) rated as catastrophic and 53.5% (n = 69) as major. The large volume of violations with severe ratings indicated clear priorities for redesign. The modified heuristic approach allowed evaluators to identify unique and important issues, including ones related to self-management and patient safety. This article provides a template for one type of expert evaluation adding to the informaticists' toolbox when needing to conduct a fast, resource-efficient and user-oriented heuristic evaluation.

Recent figures from the World Health Organization show that 347 million people in the world are affected by diabetes; this chronic disease is predicted to be the seventh leading cause of death in the world by the year 2030.[1] Data from the Department of Health and Human Services and the Centers for Disease Control and Prevention show that in the United States alone, the number of people living with the disease is 29.1 million. This number continues to grow rapidly.[2]

Of the people diagnosed with this disease, 90% have type 2 diabetes.[1] Factors such as poorly regulated glycemic levels have a large influence on patients’ conditions and are therefore vital to monitor for controlling the disease.[3] Type 2 diabetes is also lifestyle related to a large degree and can be self-managed to a certain extent in addition to using more conventional treatment procedures.[4]

Self-management is becoming increasingly important in diabetes care[5,6]; researchers found that self-management support should be integrated into patients’ everyday lives to achieve desired, improved patient outcomes.[7] As an adjunct to diabetes management, researchers highlighted the use of information and communication technology (ICT) and the development of applications for day-to-day self-care and disease management.[7,8] In support of that goal, the mobile health (mHealth) system for this study was developed as an individually based mobile and web support system for type 2 diabetes patients’ self-management.

Usability of eHealth systems

To facilitate their adoption, mobile healthcare applications and systems for chronic disease management must be usable.[9,10] Despite the increasing availability of self-management tools, many of the patient-operated ICT applications are still deficient in terms of usability.[11]

Completing an evaluation process in a self-management context for these kinds of systems requires an understanding of users and their needs when performing tasks in such a system.[12,13] Having this focus can help ensure that mobile applications are safe for clinical and patient use and possibly prevent user errors.[14] Authors thus argue that it is vital for user interfaces to be designed in a way that does not contribute to errors as this can also be a factor negatively affecting users’ experiences with the system.[9,15,16] Usability evaluations can therefore help to appropriately determine how well the application or system meets the clinical need and patients’ expectations and in safeguarding both the quality of care and patient outcomes.[17] These evaluations can be expert based, such as heuristic evaluation (HE) or cognitive walk-through, or empirical and user based evaluations, such as think-aloud methods involving user tests with actual system users.[18]

Heuristic Evaluation

Heuristic evaluation is one of the most common usability inspection methods completed by usability experts. Users are explicitly not part of this kind of method. Instead, experts apply the knowledge they have about usability principles, processes, and standards to evaluate systems.[19] Heuristic evaluation was first defined by Nielsen and Molich.[16] In this technique, usability experts evaluate an application to find usability problems, assign them to a specific category of heuristic and ascribe a severity rating. Nielsen[20] originally defined 10 heuristic categories and recommended assigning severity scores to a master list of usability violations.

Authors have attempted to modify and extend Nielsen’s techniques in different ways to achieve better results in various contexts. These include Zhang et al,[21] who came up with 14 heuristics by combining Nielsen’s 10 heuristics with Shneiderman’s eight “golden rules” to evaluate infusion pumps. Allen and colleagues[9] employed a more simplified use of the HE inspection method by having evaluators select only those heuristics they deemed appropriate for their assessment and assigned severity ratings for the usability problems on the fly instead of first creating a master list of all usability problems. Chattratichart and Brodie[22] extended the method to a technique they called HE-Plus, a directed approach using usability problem profiles to help evaluators focus their evaluations on specific types of problem areas to provide more consistent and reliable evaluation results.

Heuristic evaluation or expert usability evaluation can be useful because it provides a unique perspective and distinct information[19] and because it is a discount usability technique, meaning it is relatively quick, cost effective, and resource efficient.[16,23] However, as other authors have shown, the original method by Nielsen can be improved upon for better results. Critics of the technique, for example, indicate that many problems found with HE can be minor interface design problems[23,24] or of a more general nature.[25] User tests, in comparison, involve actual users and identify problems of a critical, qualitative nature. On the other hand, these are also more costly and time-consuming.[20,23,24,26]

In sum, current expert techniques require improvements to be able to find more severe usability problems of a critical nature for users. In this article, we addressed this gap. Our approach to accomplish this is by using a modified HE technique using its beneficial aspects and also focusing on the patient user and their needs in disease management and system information and interaction requirements to provide enhanced evaluation results. Our modifications involve (1) employing dual-domain experts (healthcare professionals and usability experts combined) as evaluators, (2) using realistic, validated user tasks with appropriate scenarios related to patients’ diabetes self-care, and (3) making severity ratings specific and in-depth across three severity rating factors by predicting each problem’s influence on patients with factors of impact, persistence, and frequency. Our intent was to explore whether the technique would be able to detect both crucial and context-related problems in patient self-management in addition to the more common, minor usability issues.

MATERIAL AND METHODS

System Description

The mobile system we evaluated was designed as a low-cost, convenient, personalized self-care management and tracking tool for use by a large number of patients with diabetes. It was also meant to function as support for conversations between patients and their healthcare providers. The system combined a web service and mobile phone solution for patients to send in self-management values, that is, glucose and blood pressure, via text messaging. The web user interface is divided into sections consisting of a Dashboard, Glucose Diary, Blood Pressure, Medication Adherence, an Exercise and Weight progress, and Appointment reminder view. Each has graphical representations of the different measurements and goals with progress indicators in red, yellow, and green. For example, the sections include a meter to visualize glucose readings, a blood pressure bar with systolic and diastolic values, a medication adherence section indicating how much of the prescribed medication was taken, and an exercise and weight progress section to show exercise and weight measures. Using their cell phones, patients can retrieve, enter, and edit their values and goals. Scenarios and tasks were developed based on these kinds of patient interactions and uses.

Expert Evaluators

The HE was performed by three expert evaluators who identified heuristic violations listed in Nielsen’s taxonomy.[20] According to Nielsen, three to five single-domain usability expert evaluators find, on average, between 74% and 87% of usability problems.[27] The number of usability problems found by dual-domain experts is even higher at 81% to 90%. Only two to three dual-domain evaluators are then deemed necessary.[27] These types of experts are seen as especially suitable in evaluating complex systems, such as those in the healthcare area, because they have usability expertise and extensive knowledge in the specific domain of application.[28,29] Each expert for this study was thus carefully selected based on dual-domain competency consisting of (1) extensive usability experience in health informatics, (2) being healthcare professionals (registered nurses [RNs]), and experience with (3) the patient group and their task requirements, and (4) diabetes self-management. As this was a HE evaluation, it involved only expert evaluators and no patients. Therefore, institutional review board approval was not required for this study.

Use Scenarios and Tasks

Scenarios and tasks outlined specific steps that evaluators used to interact with the diabetes self-management system in the HE. Tasks were based on real case scenarios to simulate how patients would use the system in a self-management process in a clinic or at home. To ensure that these were as realistic as possible, a panel also evaluated both scenarios and tasks. The panel included a physician with a diabetes specialty, a diabetes RN, a public health professional with chronic patient intervention systems expertise, and a diabetes patient. The panel verified and validated tasks for content validity and accuracy (content validity index of 0.91 of 1.0). The eight tasks and scenarios were disease specific and had varying levels of difficulty. For example, tasks consisted of viewing and locating glucose values on graphs, identifying and correcting collected glucose values, setting weight and exercise goals and medicine and appointment reminders, and viewing summary statements about medical measurements. Table 1 includes an example of a scenario and task.

Table 1

Example of a Scenario and Task Used in the Evaluation

Nielsen’s Heuristics

Similar to other method modifications, we selected Nielsen’s 10 heuristics for this study because they have been thoroughly tested, are widely accepted by user experience experts, and are fast and easy to apply. To attend to some of its shortcomings, our approach was to apply our specific modifications of (1) dual-domain experts (healthcare professionals, usability experts), (2) validated scenarios and user tasks related to patients’ self-care, and (3) in-depth severity factor ratings to determine if more critical issues could be found. The HE categories are listed in Table 2. Part of the original work with Molich,[16,30] the categories are Nielsen’s published work from 1994.[20] The evaluators used the 10 heuristics to categorize usability problems by employing the specific modifications.

Table 2

Heuristics for Usability Evaluation According to Nielsen[20]

Severity Rating Scale and Factors

Typically, HE techniques include assigning a single severity score. Instead, we divided severity ratings into factors of frequency, impact, and persistence.[20] The focus was on how each of the three different factors for each usability issue would influence the user in different ways, and separate averages were calculated for each. Subsequently, separate averages were calculated for each severity factor. This allowed for greater specificity about the severity of the problem and its impact on the specific diabetes patient users. Severity ratings ranged from 0 (not a problem) to 4 (usability catastrophe).[20] Specific descriptors for the scale for the severity rating and the severity factors are listed in Table 3.

Table 3

Scale for Severity Rating and Severity Factors[20]

After evaluators conduct their individual factor severity ratings, all ratings are summed and divided by the number of evaluators to arrive at an average severity rating for each usability problem. This rating is considered the overall severity rating, as shown in Table 4.

Table 4

Example of Severity Rating Scoring Table Including Nielsen’s Three Factors

Evaluation Procedure

Evaluators had identical instructional materials to learn the system and to ensure consistency across evaluators. Information materials consisted of a digital video on system modules, how to navigate the portal, a study design manual detailing each specific scenario and tasks to be performed, an application user manual, and an evaluation guide sheet. The study design manual also included materials on how to conduct the evaluation, the scenarios, and a usability task manual outlining how to navigate tasks. Providing specific scenarios and tasks to simulate the diabetes patient care process ensured that all experts had the same knowledge level about the functionality and user tasks.

The procedure itself was a two-part process. The evaluators first familiarized themselves with the system and its usage using the materials and training described above. Then, they performed the modified HE as visualized in Figure 1.

FIGURE 1

Modified HE process.

Each dual-domain expert evaluator performed the eight scenarios and tasks independently. After the evaluators detected a usability problem, they assigned each problem to a heuristic violation/s from the categories in Table 2. A master list was compiled, duplicate problems were removed, and the list was verified across the evaluators for accuracy. Then, each evaluator individually assigned severity scores to each problem by using the severity rating factors of frequency, impact, and persistence. These were also averaged by factor and combined into one severity rating for each usability problem as described above. Descriptive statistics were used to summarize heuristic violations and associated severity scores.

RESULTS

The HE resulted in a total of 129 usability problems and 274 heuristic violations. The usability problems by place of occurrence (view), number of heuristic violations, and mean severity ratings are summarized in Figure 2. The number of usability problems ranged from a low of 12 to a high of 34 across application views. The Dashboard view generated the most usability problems (34), followed by the Glucose Diary view (21), the Blood pressure view (20), and the Medication adherence view (15). Heuristic evaluation violations ranged from 25 to 69. The largest number of heuristic violations were was on the Dashboard view (69), the Glucose Diary view (49), the Blood pressure view (44), the Medication adherence view (31), and the Appointment reminder view (29).

FIGURE 2

Location frequency and averaged severity for usability problems and heuristic violations by place of occurrence.

The average severity ratings ranged from 2.7 to 3 on a scale of 0 to 4, with the Glucose Diary view and Medication adherence view having the highest at 2.9 and 3.0 respectively.

Heuristic Violations Across System Views

Of the 10 types of HE violations depicted in Figure 3, the categories of Consistency and Standards and Match Between the System and the Real World dominated at 24.1% (n = 66) and 22.3% (n = 61) respectively, followed by Aesthetic and Minimalist Design at 16.8% (n = 46) and Recognition Rather Than Recall at 11.7% (n = 32). The heuristic categories Recover 1.4% (n = 4) and Help 1.03% (n = 3) had the fewest violations across all views.

FIGURE 3

Frequencies of heuristic violations by heuristic category.

Severity Ratings Across System Views

Most severity ratings across views (Figure 4) consisted of major and catastrophic severity ratings. The most severely rated problems were located in the Dashboard view (n = 16, n = 4), Glucose Diary view (n = 12, n = 2), and the Blood pressure view (n = 12, n = 2). Most of the minor usability problems were similarly located in the Dashboard view (n =14) and the Glucose Diary view ( n= 7). The Appointment reminder view, however, came next with six issues. There were no cosmetic violations.

FIGURE 4

Severity ratings across system views.

Nature of Usability Problems and Prioritization

The modified HE evaluation revealed that most catastrophic ratings concerned disease-related task deficiencies and specific system-related shortcomings in displaying necessary information for patients. Some examples of these types of usability problems and comments provided by the evaluators are as follows:

Dashboard:

Each entry should, at minimum, have the time (not just the date) since many people with diabetes will do multiple glucose tests in one day.

(The total severity rating was 3.8 and factor rating 4 for all evaluators for frequency, and persistence.)

Glucose Diary view:

It is very difficult to read the time line on the graph because the numbers are too crowded which makes it difficult to distinguish and read specific dates. This is especially cumbersome for diabetes patients who often have visual concerns.

(The total severity rating was 4.0 and factor rating 4 on all factors of frequency, impact, and persistence.)

Blood Pressure view:

It is disadvantageous for the patient to not see the diastolic blood pressure reading in the graph to compare against; only the systolic value is shown.

When I hover I only can see the systolic value. If the whole BP is displayed and rated, what happens if only one value is abnormal?

(The total severity rating was 3.9 and factor rating 4 by all evaluators on frequency and persistence.)

Medication Adherence view:

With the indications…in percentages, it is difficult for a patient or health care provider to determine what medication was taken or not, which day and what time.

…Tallying up totals to say 100% of medications is an odd way to think about medication from a patient perspective. I wouldn’t say that I have taken 75% of my meds for the last month for instance. I need to know specifics and insulin or Metformin and that these are jointly displayed and tracked.

…I found that this display did not match my mental model of medication compliance, need individual information regarding medication.

(Total severity rating of 3.7 factor rating of 4 for two evaluators on each factor of across frequency, impact, and persistence.)

DISCUSSION

In this study, most usability problems were categorized as major issues. The largest volume of problems clustered in the categories of Consistency and Standards and Match Between the System and the Real World. Both of these categories indicate that the system requires better design to support effective decision making and action control for relevant patient user tasks.[31]

This evaluation uncovered specific concerns related to the disease-related information deficiencies of the system. These included, for example, that the Dashboard view allows only one daily entry and lacks a time for glucose readings on its current meter. These are fundamental issues for a diabetes application. Since patients with diabetes often have multiple readings in one day, this is a major usability problem related to essential information needed or patients’ self-management tasks. In the Glucose diary, the numbers on the timeline were too small, crowded together, and difficult to read. This is especially problematic for individuals with diabetes as they often have visual acuity issues. The likelihood of users performing errors is increased for both of these issues.

Using the in-depth severity factor ratings, the use of dual-domain evaluators and validated user tasks related to the care process enabled the evaluators to find a large volume of severe deficiencies in the system. Finding major issues with HE is in contrast to other authors’ work both within and outside the health domain who mainly found minor interface issues.[23-25] The identified major and severe usability problems found here require immediate attention for redesign to fulfill patients’ self-management needs. The methodological changes to HE may have enhanced evaluators’ abilities in finding important usability problems.

Usability testing with users is very helpful, but as was seen here, dual-domain experts add important, additional dimensions to any usability evaluation. In particular, they were able to understand tasks as part of the context of use for a chronic disease self-management system and were able to identify specific system design concerns and uncover distinct information needs related to these self-management tasks. Dual-domain experts may also be able to assist in identifying possible patient safety issues that patients may not identify. A case in point was that the medicine reminder was too nonspecific and provided insufficient information, making it possible for patients to commit errors in insulin dosages.

We recommend that dual-domain experts be employed in future HEs for chronic disease management systems whenever possible. These kinds of experts can identify unique and critical usability problems as well as deeper cognitive support issues and specific disease-related concerns. Dual-domain experts provide added value in uncovering pertinent issues.[28,29,32] This information can be used both during the iterative design process, during formative evaluation, summative testing, and for comparing different versions and applications.

Other methodological modifications can assist future usability evaluations. A standardized evaluation process and use of specific scenarios and tasks allowed for efficient evaluations across the experts and can also aid in reproducibility and generalizability. The modified severity factor rating method in this study also proved important as it allowed the evaluators to think about the specific impact of each individual usability problem and provided a more in-depth analysis of the specific usability problems. This modification provided an objective method to determine the importance of the usability problem in relation to others of a similar nature and could aid in problem prioritization.

Limitations

Although the modified HE process uncovered many major usability problems, other problems might have been detected had the number of dual-domain evaluators been expanded over the recommended two to three evaluators.[27] Both HE and user tests could also be combined to detect an optimal number of usability problems,[33] although it would entail higher costs. We wanted to provide an efficient, cost-effective method of modifying HE to also be able to identify more serious usability issues that could have an impact on patients in their disease management.

CONCLUSION

Consumer health systems and applications in mHealth should be evaluated for usability as well as medical adequacy. This article describes useful modifications to HE by modifying and deepening Nielsen’s techniques. Specifically, modifications were using dual-domain experts; employing validated, patient-centered self-care tasks and realistic care process scenarios; and using separate in-depth severity factor ratings. In particular, dual-domain experts can provide unique information related to the salient tasks for patient self-care and identify potential patient safety issues as well as determine how an application adheres to known usability guidelines.

This modified heuristic method can be used by other informaticists in healthcare who need to conduct a fast and resource-efficient heuristic process related to patient self-management. The results of the study show that a modified HE can uncover unique, critical issues in this context. This kind of evaluation may be done at any point in the system lifecycle. As an expert evaluation technique, HE should be included in any usability evaluation and is quite suitable for mHealth applications designed for chronic disease patients. Thus, it has an important place in usability evaluations. With the modifications provided here, Nielsen’s original techniques can be improved to achieve improved results. Techniques like the ones described here can be an important addition to any informaticists’ toolbox when determining chronic disease systems’ adequacy for patient self-management needs.