Evaluation of Mobile Health Applications Developed by a Tertiary Hospital as a Tool for Quality Improvement Breakthrough.

OBJECTIVES: To evaluate the mobile health applications (apps) developed by a single tertiary hospital in Korea with a particular focus on quality and patient safety.
METHODS: Twenty-three mobile health apps developed by Asan Medical Center were selected for analysis after exclusion of the apps without any relationship with healthcare or clinical workflow, the apps for individual usage, and the mobile Web apps. Two clinical informaticians independently evaluated the apps with respect to the six aims for quality improvement suggested by the United States Institute of Medicine. All discrepancies were resolved after discussion by the two reviewers. The six aims observed in the apps were reviewed and compared by target users.
RESULTS: Eleven apps targeted patients, the other 12 were designed for healthcare providers. Among the apps for patients, one app also had functions for healthcare providers. 'My cancer diary' and 'My chart in my hand' apps matched all the six aims. Of the six aims, Timeliness was the most frequently observed (20 apps), and Equity was the least observed (6 apps). Timeliness (10/11 vs. 10/12) and Patient safety (10/11 vs. 9/12) were frequently observed in both groups. In the apps for patients, Patient-centeredness (10/11 vs. 2/12) and Equity (6/11 vs. 0/12) were more frequent but Efficiency (5/11 vs. 10/12) was less frequent.
CONCLUSIONS: Most of the six aims were observed in the apps, but the extent of coverage varied. Further studies, evaluating the extent to which they improve quality are needed.

I. Introduction

Ubiquitous health (u-health) aims to provide innovative healthcare services that are consumer-oriented, customized, and accessible anytime, anywhere by ubiquitous computing [1]. More than 10 years have passed since the concept of u-health was proposed in Korea [1]. Despite very high expectations, u-health still is in its very early stages, in part because its clinical effectiveness has not been clearly demonstrated [12]. It is also struggling in Korea in part because of the complex debates about telemedicine. However, electronic health (e-health), a concept that is similar to u-health, which utilizes information systems by applying them to existing healthcare practices to improve the productivity of the work of the health service provider [1], has been growing in Europe and the United States even though issues regarding its clinical effectiveness have not yet been resolved [34567]. The e-health market has grown rapidly, especially after the emergence of mobile health (m-health) in 2009 [34578910]. m-Health services using both wearable devices and smartphones are also expanding [111213].

m-Health provides innovative healthcare services by using the unique characteristics of the smartphone such as mobility, connectivity, accessibility, and the capacity to gather and transmit data [91011121314], so it is considered a subtype of u-health in Korea [12]. Diverse u-health devices, including wearable devices, can be connected to smartphones, and sometimes the smartphone itself can be used as a u-health device [1516]. m-Health services and trials using smartphone sensors, such as cameras, accelerometers, and pedometers, are frequent now [131718].

Despite this rapid growth, challenges remain which must be addressed if m-health is to result in the hoped-for changes [3812]. According to Dr. Joseph Kvedar, Director of the Center for Connected Health of Partners Healthcare, one in ten U.S. patients has used a health tracker, but the majority stop using it within six weeks, and for m-health applications (apps), 80% are abandoned after only two weeks [19]. Several studies have suggested key features that encourage users to be more likely to continuously use such apps, but most trackers and apps have little impact, and more research is needed [192021]. m-Health must also be connected to personal health records or Electronic Health Records to provide a continuous care model and to improve patient outcomes [8], but such connectivity and interoperability remain major issues. The Apple HealthKit and ResearchKit have the potential to be breakthrough technologies in this area [22]. Another big challenge is determining the clinical effectiveness of m-health. Whether the use of m-health can improve patient outcomes must be determined [2356823]. Like telemedicine cases, not all m-health services can achieve this goal [324]. The common features of m-health that impact its outcomes must be researched [8]. Therefore, welldesigned clinical trials and meta-analyses are required to definitively determine the effectiveness of m-health [368]. Healthcare organizations in Korea have also developed many m-health apps and have provided m-health services [925]. However, few outcomes research studies have been done in this area. The six aims for quality improvement of the Institute of Medicine (IOM) in the United States can be applied to evaluating the benefits of m-health apps [26]. In 2001 the IOM recommended that "health care should be supported by systems that are carefully and consciously designed to produce care that is safe, effective, patient-centered, timely, efficient, and equitable" and "health information technology (HIT) must play a critical role to achieve the six aims [26]." Some studies reviewed HIT applications with respect to the six aims to evaluate their benefits and impacts [2728].

Asan Medical Center in Seoul, Korea, has developed more than 20 m-health apps since 2010. Because no other hospital in Korea has developed as many m-health apps as our hospital has, we evaluated the m-health apps developed by our hospital with a particular focus on the IOM's six aims with the intention of sharing our findings.

II. Methods

The information about the launch date, the date of the final update, and installed data for the apps was collected from the mobile apps server or hospital information system server with cooperation of the Ubiquitous Health Center in the hospital. We collected the information about the apps from the day of distribution of the app until December 31, 2014. The Institutional Review Board approved our study protocol.

1. App Selection and Categorization

Mobile apps developed by Asan Medical Center or in which there was a leading role of the hospital in the initial request were reviewed retrospectively. After excluding the apps with no relationship to healthcare or clinical workflow and the apps designed for individual usage, 23 apps were chosen for the final analysis. These 23 apps were categorized based on their target users (i.e., healthcare providers or patients) and their range of distribution (i.e., open to public or closed for inhospital use) (Figure 1). We divided the apps into two groups, according to the target user of the apps. In addition, the functions of the apps and the implementations of the six aims of the apps in each group were evaluated and compared.

Figure 1

Categorization of the 23 apps selected for analysis. The x-axis represents the target users, and the y-axis represents the range of distribution (Emergency in ordinary life1: emergency in ordinary life for patients, emergency in ordinary life2: emergency in ordinary life for clinicians).

2. The Six Aims for Quality Improvement

The functions and purposes of the apps were reviewed based on the six aims. The six aims selected by the IOM include the following:

1) Patient-centeredness

Providing care that is respectful of and responsive to individuals (i.e., apps for providing medical/health information to patients themselves, providing patients' authority to decide),

2) Effectiveness

Providing services based on scientific knowledge to all who could benefit, and refraining from providing services to those not likely to benefit (i.e., apps for verifying clinical guidelines and clinical information with evidence, apps for practice based on evidence, apps for clinical trials to make evidence),

3) Patient safety

Avoiding injuries to patients from the care that is intended to help them (i.e. apps for prevention, monitoring, and early detection of harmful events in the field of healthcare),

4) Timeliness

Reducing waits and sometimes harmful delays for both those who receive and those who give care (i.e., apps for providing information without limitations of place and/or time, bedside access to health information),

5) Efficiency

Avoiding waste, including waste of equipment, supplies, ideas, and energy (i.e., apps for fast and precise transmission of health information in clinical workflow to achieve good decision-making, apps for reducing cost, unnecessary processes and manpower),

6) Equity

Providing care that does not vary in quality because of personal characteristics, such as gender, ethnicity, geographic location, and socioeconomic status (i.e., apps for vulnerable population; the disabled, patients with chronic diseases, patients with mental diseases, pediatric patients, maternity patients, and the elderly).

3. Evaluation Questionnaires for the Apps

Using questionnaires, the 23 apps were reviewed to assess their operating system, target user, necessity of log-in, and user supporting method. These questionnaires, and the categories they included, were prepared by two clinicians who specialize in clinical informatics. Both clinicians designed the questionnaires and independently reviewed them. After the forms were filled out, the two reviewers discussed the results of each app and resolved all discrepancies.

III. Results

Table 1 presents the overall characteristics of the selected 23 apps. Eleven of the 23 apps were designed for use by patients and the general population, 12 targeted healthcare providers as users. Among the apps for patients, one app also had functions for healthcare providers. Figure 2 lists the number of users of each app. As expected, the public apps had more users.

Table 1

Characteristics of the 23 mobile apps analyzed, which were developed or released by Asan Medical Center (AMC)

OS: operating system, PHR: personal health record, OPD: outpatient department, EMR: Electronic Medical Record, HIV: human immunodeficiency virus.

aFinal update: the date of latest version, checked on June 28, 2015. bUser count: accumulated count of users, assembled from the day of distribution to December 31, 2014. cEmergency in ordinary life1: emergency in ordinary life for patients. Emergency in ordinary life2: emergency in ordinary life for clinicians.

Figure 2

Accumulated count of users from the day of distribution to December 31, 2014.

The frequency that the 23 apps involved the six aims is described in Table 2. 'My chart in my hand' and 'My cancer diary' were matched for all the six aims. The specific functions or purposes of the apps correlate with the six aims as follows:

Table 2

Compatibility of the mobile apps developed at Asan Medical Center with the six aims for quality improvement

Emergency in ordinary life1: emergency in ordinary life for patients, Emergency in ordinary life2: emergency in ordinary life for clinicians

1) Patient-centeredness

'My chart in my hand' and 'Pediatric cancer diary' provide medical data to patients and promote self-management by recording and analyzing personal health data (i.e., body weight, blood sugar level, and blood pressure level). 'Pharm consult' provides e-mail-based consultation for proper drug use and management of drug side effects. 'mAMIS' can help physicians show and explain the patient's information at the patient's bedside.

2) Effectiveness

Apps such as 'ASAN-in', 'Toxicology', and 'Emergency in ordinary life' provide evidence-based information to clinicians and/or patients. 'Pit-a-Pat', 'Quality of life', and 'Injini' are apps for clinical studies to build evidence.

3) Patient safety

'Wild plants in Korea' and 'My medications' are apps to prevent hazards encountered in daily life, and 'Ora-Quick' and 'Blood culture' prevent hazards in the clinical field. 'mAMIS', 'CPCR drugs', and 'Pediatric cancer diary' assist in the early detection and/or rapid management of hazardous events.

4) Timeliness

'm-AMIS' and 'My chart in my hand' provide access to patients' information independent of space and time. 'My medication' and 'My cancer diary' provide health information or support at the patient's bedside.

5) Efficiency

Apps such as 'AMC ARC', 'ASANnet', and 'ASAN talk' promote efficient networking and communication between healthcare providers. 'Emergency in ordinary life' and 'Patient transportation' help to create an efficient clinical workflow.

6) Equity

'Pediatric cancer diary', 'My chart in my hand', and 'Injini' provide care for vulnerable populations, such as cancer patients, pediatric patients, children with cognitive disorder or patients with chronic diseases.

The frequency of overlap with the functions of apps and the six aims present different dispositions according to the target user (Table 3) and the range of distribution (Figure 3). Timeliness (10 apps out of 11 patients-targeted apps vs. 10 out of 12 healthcare provider-targeted apps) and Patient safety (10/11 vs. 9/12) were frequently observed in both groups and evenly distributed. The remaining aims showed differences between patient-targeted apps and healthcare providertargeted apps. In the apps for patients, Patient-centeredness (10/11 vs. 2/12) and Equity (6/11 vs. 0/12) were more frequent, but Efficiency (5/11 vs. 10/12) was less frequent. All the apps matched with Equity, and all the apps except two that matched with Patient-centeredness were for patients. In contrast, the apps corresponding to Efficiency were predominantly those for healthcare providers. There was no Equity-matched app for healthcare providers.

Table 3

Apps that have functions or purposes that are compatible with the six aims according to the target user

Values are presented as number (%).

Figure 3

Numbers of applications that have functions or purposes that are compatible with the six aims according to the target user.

IV. Discussion

It was clear that many apps for healthcare address a number of the key quality aims. For example, Timeless was predominant due to the nature of smartphones, which offer accessibility and mobility. Desktop-based programs has a lot of limitations in terms of timeless. Patient safety was well implemented in both groups. This implies that m-health can be beneficial in the quality aims. Most patient-targeted apps covered Equity and Patient-centeredness. Most apps for healthcare providers satisfied Efficiency. Interestingly, there was no app for healthcare providers that satisfies all of the six aims, in contrast with the apps for patients, such as 'My cancer diary' and 'My chart in my hand.' In addition, Equity was not matched for any app for healthcare providers. This reminds us to be aware of vulnerable populations when developing an app, even for healthcare providers [29].

There were some limitations of this study. First, the questionnaires used to assess the apps were evaluated by two clinicians. Since one of the clinicians participated in the development of the apps, any matches between the apps and the six aims might be biased. Also, the other clinician had limited ability to evaluate some apps which ended their services. Therefore, further evaluation of apps is necessary, focusing on currently in-service apps. For more objective and precise evaluation of healthcare-related apps in the future, the assessment of the six aims should be performed by experts uninvolved in their development and should encompass patients who are currently using the apps. Second, we found a disparity in the number of users in accordance with the range of distribution and/or the date of launch. In addition, the number of users who installed an app does not necessarily correlate with the level of usage. Since the extent of usage or the activity of apps is difficult to evaluate precisely, more advanced measures should be developed.

In this study, most of the apps designed for patients provide health information or patient medical information and satisfy Patient-centeredness. These apps are also a good tool for gathering health information from patients. Unfortunately, the user data collected in this way have not been actively applied to clinical use until now. However, the patients' generated data in clinical apps, with the merits of Timeliness and Effectiveness, would promote user interaction. In addition, tailored treatments acquired through user interactions with these healthcare apps would contribute to better Patient-centeredness and Patient safety. Therefore, the evaluation of an individual app is essential for efficient management and continual improvement of the app.

In conclusion, although this study was performed using the apps developed by a single tertiary hospital in Korea, the evaluation results demonstrate the enormous potential of mhealth apps as important tools to achieve breakthrough of quality improvement. To achieve this breakthrough, there should be much more study to improve m-health apps.