The Use of an Adapted Health IT Usability Evaluation Model (Health-ITUEM) for Evaluating Consumer Reported Ratings of Diabetes mHealth Applications: Implications for Diabetes Care and Management.

BACKGROUND: The aim of this paper is to present a usability analysis of the consumer ratings of key diabetes mHealth applications using an adapted Health IT Usability Evaluation Model (Health-ITUEM).
METHODS: A qualitative content analysis method was used to analyze publicly available consumer reported data posted on the Android Market and Google Play for four leading diabetes mHealth applications. Health-ITUEM concepts including information needs, flexibility/customizability, learnability, performance speed, and competency guided the categorization and analysis of the data. Health impact was an additional category that was included in the study. A total of 405 consumers' ratings collected from January 9, 2014 to February 17, 2014 were included in the study.
RESULTS: Overall, the consumers' ratings of the leading diabetes mHealth applications for both usability and health impacts were positive. The performance speed of the mHealth application and the information needs of the consumers were the primary usability factors impacting the use of the diabetes mHealth applications. There was also evidence on the positive health impacts of such applications.
CONCLUSIONS: Consumers are more likely to use diabetes related mHealth applications that perform well and meet their information needs. Furthermore, there is preliminary evidence that diabetes mHealth applications can have positive impact on the health of patients.

1. INTRODUCTION AND BACKGROUND

Type 2 diabetes (T2D) is one of the major chronic conditions impacting the health of millions of people worldwide. In the United States alone, there are nearly 26 million people living with diabetes and an additional 79 million adults living with pre-diabetic state (1). In Europe, according to the World Health Organization (WHO), there are approximately 60 million people living with diabetes. In the Middle East, where some of the highest rates of T2D exist, there are approximately 34 million people suffering from diabetes and its complications (2, 3). The International Diabetes Federation (IDF) estimates that by 2030, there would be 552 million people living with diabetes. This translates to approximately one in ten adults living with the disease and its subsequent complications or associated comorbid conditions (4). In today’s dynamic era, information systems and technology play an important role in health education, population health promotion, and chronic disease prevention. According to Darvish and colleagues (5), in a critical appraisal of emerging technologies, the key elements for the success of health informatics initiatives are their focus on health care promotion and wellness of individuals and the society as a whole, and network and Internet advanced systems.

1.1. Overview of mHealth Technologies

Mobile health (mHealth), as defined by the WHO, is an area of electronic health (eHealth) which provides health information and services through smart (mobile) phones (6). mHealth can be an invaluable approach in particularly underserved rural and remote locations in low- and middle-income countries, as well as in developed high-income countries where personal outreach and service is a challenge on the part of nurses, doctors, and community health care workers/providers. mHealth devices can be patient monitoring devices, mobile phones, tablet PCs, personal digital assistants (PDA), and other wireless devices. mHealth overcomes geographic barriers by allowing remote communications and treatment delivery and removes time barriers by allowing continuous monitoring of physiologic measurements. mHealth can be thought of as a tool of “wireless medicine” which is defined as the use of wireless technologies for the personal health care delivery and management of chronic conditions (7). Kallander et al (6) note that there are approximately five billion mobile phone users around the globe. Health care researchers and providers are realizing the potential of using mobile technologies for health services outreach and delivery.

1.2. Diabetes mHealth Applications

Today, there are over 7,000 mHealth applications within both the Android and iPhone markets (8). Although there is no accurate figure for the number of diabetes-related mHealth applications, it is estimated that their number amounts to hundreds. With the epidemic proportional increase in T2D rates globally, the number of new mHealth applications for diabetes will continue to grow (9). For instance, recently, a 400% increase in the number of mHealth applications for diabetes has been observed lately, from 60 in 2009 to 260 in 2011. These applications have been designed to help patients with monitoring blood glucose, body weight status, physical activity patterns, dietary habits, insulin use and other medications, blood pressure, education, and disease-related alerts and reminders (10). In diabetes care, although there is great potential for the use of mHealth technologies, one of the major challenges for advancing their implementation is the difficulty in finding the clinical effectiveness of such technologies (9, 11). As more mHealth application continue to be developed, more research work, including clinical and community-based trials, should be conducted on both their usability and their impacts on health. There are promising opportunities for mHealth in general and for T2D in particular, but without developing usable applications that can positively impact the health of patients, the benefits of these technologies will remain elusive (9, 12).

1.3. Usability

Usability is “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (13). In health care, the evaluation of mHealth applications is regarded as a critical success factor in the ongoing implementation and their use. In 2012, the Institute of Medicine produced a report entitled: “Health IT and Patient Safety: Building safer systems for better care” which discussed the importance of testing the usability of electronic medical records (EMRs) implemented in hospitals. The report highlighted the importance of carrying out usability assessments on health information technologies because not doing so could adversely affect patient care (14). To date, the research data on the usability of mHealth applications is lacking and about 95% of applications have not undergone usability testing (15, 16). As the development of mHealth applications increase, there will be a need to systematically evaluate and rate these applications by independent and credible organizations and researchers to assess the usability of such technologies.

1.4. Health ITEUM

According to Brown et al (15), the Health IT Usability Evaluation Model (Health-ITUEM) was developed based on the concepts of usability stemming from the ISO 92411-11 and the Technology Acceptance Model (TAM). It was developed to complete the missing information that existed in previous usability frameworks and models (15). The Health-ITUEM focuses on the assessment of usability through the following items: error prevention, completeness, memorability, information needs, flexibility/customizability, learnability, performance, competency and other outcomes. An adapted model of the Health-ITUM was later created and tested by Brown and colleagues (15). In their model, the authors added more detail by including positive and negative sentiments for each of the Health-ITUEM codes, in addition to a neutral code. The results of this refinement allowed assessing positive, negative, and neutral responses to the usability of mHealth applications which led to the development of 27 possible coding categories. In this study, we used an adapted Health-ITUEM model based on the work of Brown et al (15).

2. AIM OF THE STUDY

The main aim of this study is to present a usability analysis of the consumer ratings of key diabetes mHealth applications using an adapted Health-ITUEM. In addition, this study evaluates consumers’ rating reports on the impacts mHealth applications on their health. The results of this study will be of use to health care service providers, administrative hospital personnel interested in the use of diabetes mHealth applications, and academics and students interested to understand the current challenges facing the use of mHealth diabetes applications and their potential impact on health care service delivery and utilization.

3. METHODS

3.1. Study Design

A qualitative content analysis method was used to analyze publicly available consumer reported data posted on the Android Market, Google Play, for the diabetes mHealth applications. In general, content analysis is an analytic approach that can be used to analyze qualitative data. It is a systematic process of analyzing communication messages and making inferences based on the analysis (17, 18). Content analysis involves the interpretation of textual data that has been categorized into concepts. Once the identification of concepts or categories has taken place, they are categorized into themes based on their relationships with each other (19, 20).

3.2. mHealth Diabetes Applications

In 2013, Healthline (Healthline.com), a website offering objective and credible health information, rated the top 13 mHealth applications for diabetes. When applying the eligibility criteria, only four out of the 13 mHealth diabetes applications were included in the analysis for the current study. The applications were included if they were developed using the Android platform, were in English, and had more than 500 consumer review reports. Applications were excluded if they were in a language other than English, had less than 500 consumer review reports, and were based on the iPhone platform (Figure 1).

Figure 1

Selection Process for Diabetes mHealth Applications

Based on the eligibility criteria outlined above, the four mHealth applications included in this study were:

Fooducate – Healthy Food Diet and Nutrition Scanner. Fooducate grades food, explains what’s inside each product, and offers healthier alternatives. The system has a large database of UPC-based nutrition information with over 200,000 unique products.

Glucose Buddy – Diabetes Helper. This is a data storage utility for people with diabetes. The system can assess blood glucose (BG), carbohydrates (Carbs), monitor physical activities such as walking or running, Hemoglobin A1c (HbA1c), blood pressure, and body weight.

Calorie Counter PRO myNetDiary – With over 40 screens and a 510,000+ foods in a database to help consumers monitor their calories’ plans.

OnTrack Diabetes – this application allows consumers to quickly and easily keep track of blood glucose, hemoglobin A1c, food diary as well as the ability to see detailed graphs and reports to share with their physician Various types of information such as user rating, number of total consumer ratings, number of installs, and software version and cost were collected for each of the four mHealth applications employed in the current study (Table 1).

Table 1

Diabetes mHealth Application Information (As of February 20, 2014)

3.3. Coding Scheme

The Health-ITUEM concepts guided the analysis of the data based on the adapted work of Brown et al (15). Information needs, flexibility/customizability, learnability, performance speed, and competency were Health-ITUEM concepts that were included to categorize the data. Error prevention, completeness, memorability, and other outcomes were excluded because they were more software-related issues. These issues were beyond the scope of the current study as our focus was on consumer reported ratings of the four mHealth applications. Health impact was an additional category that was included in the study. Similar to the previous work of Brown et al (15), each of the concept codes were broken down into positive or negative codes. No neutral codes were included as consumer reports are mostly positive or negative. The concept codes for identifying a positive response was designated with a plus sign (+). Negative responses were designated with a minus sign (-). There was a total two codes for each of the six usability coding categories included, for a total of 12 possible codes. Table 2 illustrates an overview of the codes and sample quotes.

Table 2

Health-ITUEM Adapted Codes and Health Impact Code

3.4. Sampling

All consumers reported quotes from January 9 to February 17, 2014 for the four mHealth applications were included in the study. Software vendor reports, when found, were excluded from the sample. The study team did not make any contributions to the consumer reports that were part of the sample.

3.5. Data Extraction and Analysis

The postings were copied and pasted into an MS-Excel spreadsheet in a chronological order based on the most recent posting date. The quotes were not parsed into Excel and as a result some of the consumer reports had more than one code applied to it, and which ranged from 1 to 3 coding categories for each consumer quote reported. The smallest consumer report was 1 word and the largest was 128 words. Each quote was treated as a separate entity and was not referenced or connected with the previous quotes. The data were extracted independently by one researcher and another check was conducted by another researcher to verify the coding. Any discrepancies were discussed within the study team for resolution. No personal information or identifiers were used when reporting the results or during the content analysis process.

4. RESULTS

Of the 13 highly rated diabetes mHealth applications identified, 4 diabetes mHealth applications met the inclusion criteria (Figure 1). There were a total of 405 consumer posts identified across all the four mHealth diabetes programs within the study time period. Consumer reports were included in the study if they could be categorized into one of the six coding categories and were in English. Any consumer reports that did not give enough detail on the mHealth application and as a result could not be coded were excluded. Out of the 405 consumer posts that were identified, 100 were excluded. Thus, 305 consumer reports remained and were included in the current study (Figure 1).

Overall, the analysis shows that over 81% (N=327) of the consumer comments were positive on all four mHealth applications for both usability and health impacts as shown in Table 3. Only 19% (N=78) were negative. When excluding the consumer reports on the impacts on health and focusing on the usability, the analysis shows that around 76% (N=246) of consumer postings were rated positive. Only 24% (N=76) were rated negative. On information needs, 83% (N=109) of consumer reports across all four mHealth applications noted a positive impact of each of the mHealth applications on meeting the information needs of the consumer. Only 19% (N=22) of consumer reports noted that the mHealth applications did not meet the information needs.

Table 3

Summary of Health-ITUEM and Health Impacts

On flexibility and the ability to customize the application, 63% (N=12) of the postings reported a negative result; only 37% (N=7) reported a positive result. In regards to the learnability, all eight of the consumer reports were positive and the data came from only one mHealth application, Fooducate. With respect to performance speed, over 71% (N=103) of the consumer reports were positive with only 29% (N=42) reporting negative performance speed. As to competency in the use of the mHealth application, all (N=19) of the occurrences coded were positive. There were no negative occurrences that were reported.

With regards to the health impacts of the mHealth application, 96% (N=81) of the consumer data reported positive health outcomes as a result of using the diabetes mHealth applications. Only 4% (N=2) of consumer reports noted negative health impacts.

5. DISCUSSION

Mobile health is considered one of the newly ground-breaking and emerging health information technologies that have recently begun to transform and redefine how we as health care professionals and as a community think about accessibility and service delivery of health care. A larger and widespread expansion of this technology requires a new paradigm of thinking as to how health information systems and decision support systems can be best implemented and utilized for analyzing electronic personalized data gathered through mobile smartphones and devices — widely available in the market today in the hands of patients and individuals. mHealth is still in its infant stages and much needs to be learned. Most likely, it will be the next big entity in type 2 diabetes (T2D) prevention and management from the standpoint of health care service delivery and integration.

With the growth in the number of mHealth technologies, there is a need to evaluate the usability and health impacts of such technologies. The current study is the first of its kind to examine the usability and health impacts of mHealth applications on diabetes care using the Health-ITUEM model. In this study, overall performance and performance speed were the primary issues that consumers were commenting on. Almost 48% (N=145) of all the usability comments were related to issues around performance speed–which included comments relating to application crashing, ability for patients to carry out tasks, and cumbersome use.. The results show that performance speed is one of the most significant factors impacting the usability of diabetes mHealth applications. Based on these results, it is recommended that mHealth applications be tested for their performance speed to ensure that users are satisfied with speed in relation to carrying out tasks within the mHealth application. Previous studies also recommend that mHealth applications should be tested for performance, as an important aspect for users, in relation to the applications’ ability to perform tasks (18;19).

The current study found that information needs was a significant factor influencing the use of diabetes mHealth applications. Over 43% (N=131) of both negative and positive codes were related to information needs. This finding suggests that information needs is a significant factor in rating the usability mHealth diabetes applications. This is consistent with previous studies that show the importance of technology meeting the information needs of diabetic patients (20, 21). Based on these findings, future mHealth applications should focus on meeting the information needs of patients, particularly diabetic patients, by developing content relevant to the patient audience. Mobile health developers should not only focus on building applications for functionality, but also concentrate on the importance of content to meet the information needs of diabetic patients.

A major finding of this paper is the impact of the mHealth application on the health of patients. In the analysis of the data, there were a total of 83 comments coded for health of which 81 were positive and 2 were negative. These results show that the four mHealth applications have positive, although unverified, health outcomes. Consumer reports discussed weight loss, monitoring and control of diabetes care, improved eating behaviour and monitoring, improved overall health, and using the mHealth application as a patient education tool. These finding are encouraging as they demonstrate positive impacts of the mHealth diabetes applications on health, which has been neglected in the mHealth literature. Future work should focus on retrieving long-term consumer data on health outcomes from several applications to determine the health impacts of the mHealth applications.

The current research study demonstrates the importance of the usability categories of the adapted Health-ITUEM model. This model should be further utilized to evaluate the positive and negative impacts of mHealth applications in a variety of health care settings. The use of the Health-ITUEM model can be effective in the analysis of consumer ratings of mHealth applications. This adopted model excluded the use of various coding categories such as error prevention, completeness, memorability and other outcomes when coding consumer related postings. These coding categories can be used for rating the system as opposed to rating the consumer reports. Furthermore, health impacts were added to the Health-ITUEM model. Future work should focus on the development of this category and testing the relationship between the Health-ITUEM codes and health impacts. In their study on the usability mHealth application to patients with diabetes, Arsand et al (22) analyzed 10 mHealth applications. One of their recommendations was that mHealth applications should be further examined for their impacts on health.

The two main findings of the current study relate to information needs and application performance as the two primary factors influencing the perception of usability among online self-reported consumer postings. Across all four mHealth applications, information needs and application performance were rated highly positive with consumer rating of 4.4 out of 5 and above (Table 1). The positive results of the content analysis around usability of all four applications as well as the high consumer ratings and high number of downloads suggest a correlation between the content analysis results and consumer ratings. This finding is contrary to previous literature that suggests self-reported ranking systems are inadequate and unreliable in determining the mHealth application’s usability (23). The latter author argues that user ratings are largely “unregulated” and can be manipulated. However, the results of the current study indicate that with sufficient large sample size (of consumer reported data), such issues can be mitigated.

Arnhold, Quade and Kirch (24) conducted a recent systematic review of mobile applications for diabetes. The authors found that a great number of applications used for diabetes already exist. However, there are similar functionalities among the majority of these applications. There is a combination of only one or two functions in one application. The development of future diabetes-related mobile applications should involve both patients in conjunction with physicians, nurses (particularly in rural and remote areas), and health care providers. The authors note that the performance of multifunctional applications, in terms of usability by diabetic patients, was much worse compared with uni-functional or bi-functional applications. The latter being the most pervasive applications found in mobile smartphones or other electronic mobile devices such as Tablet PCs and iPads. Furthermore, it was found that the presence of an analysis or documentation function resulted in significantly lower usability scores by individuals and patients alike. This impacted the operation and accessibility features for diabetes applications, and thus limited their usability.

6. LIMITATIONS

There are a few limitations of our study. First, the data collection process could have been expanded to include a longer timeframe. This would have increased the validity of the results as to the representation of the study sample to a larger target population (external validity). Second, testing of the adapted Health-ITUEM coding scheme should have been more rigorous and an inter-rater reliability study should have been conducted. Third, this study should have examined more diabetic-related mHealth applications, in particular those supported only by the iPhone system. Last, a correlation study examining the relationship between usability and health impacts of each of the four implemented mHealth applications should have been conducted. This, however, was beyond the scope of the current study. A future study with a larger sample size to and longer time frame for data collection is warranted to assess such relationships.

7. CONCLUSION

Studying the usability and health impacts of mHealth applications has been neglected and marginalized within the mHealth (and eHealth) field. The study found that the highly rated mHealth applications primarily met the information needs of the users, and performed well for them. This research also found that consumer rated reports regarding mHealth can be used to analyze usability and health impact of mHealth applications. Despite the aforementioned limitations, the findings offer a critical lens on the use of consumer ratings to assess the usability and health impact of the mHealth diabetes applications. A future study should be carried out to examine the association between usability and health impacts for each of the four diabetes mHealth applications.