Initial Usability and Feasibility Evaluation of a Personal Health Record-Based Self-Management System for Older Adults.

INTRODUCTION: Electronic personal health record-based (ePHR-based) self-management systems can improve patient engagement and have an impact on health outcomes. In order to realize the benefits of these systems, there is a need to develop and evaluate heath information technology from the same theoretical underpinnings.
METHODS: Using an innovative usability approach based in human-centered distributed information design (HCDID), we tested an ePHR-based falls-prevention self-management system-Self-Assessment via a Personal Health Record (i.e., SAPHeR)-designed using HCDID principles in a laboratory. And we later evaluated SAPHeR's use by community-dwelling older adults at home.
RESULTS: The innovative approach used in this study supported the analysis of four components: tasks, users, representations, and functions. Tasks were easily learned and features such as text-associated images facilitated task completion. Task performance times were slow, however user satisfaction was high. Nearly seven out of every ten features desired by design participants were evaluated in our usability testing of the SAPHeR system. The in vivo evaluation suggests that older adults could improve their confidence in performing indoor and outdoor activities after using the SAPHeR system. DISCUSSION/
CONCLUSION: We have applied an innovative consumer-usability evaluation. Our approach addresses the limitations of other usability testing methods that do not utilize consistent theoretically based methods for designing and testing technology. We have successfully demonstrated the utility of testing consumer technology use across multiple components (i.e., task, user, representational, functional) to evaluate the usefulness, usability, and satisfaction of an ePHR-based self-management system.

Introduction

Electronic personal health records (ePHRs) are intended to empower patients, strengthen patient engagement, and improve care coordination.1 Widespread interest in using ePHRs has been reported and their adoption has been increasing.2–4 A Markel foundation study in 2008 documented that an estimated 2.7 percent of adults in the United States were using an ePHR.4 However, a more recent investigation documented a rise in ePHR use rates from 11 percent in 2012 to 17 percent in 2013.3 While slightly more than a fivefold increase over five years is encouraging, this represents a relatively low use rate.

The federal electronic health record (EHR) incentive program, known as the “Meaningful Use” program, requires that health care organizations provide patients with the ability to “view, download, and transmit” their health information electronically.5 ePHRs that are tethered to an organization’s EHR can enable the transfer of patient information from an EHR to an ePHR.6 Since more patients are expected to use ePHRs based on the requirement of the incentive program, tackling issues that hinder ePHR adoption and use poses a significant challenge to health care systems and consumers. There are several barriers to the adoption and use of ePHRs including lack of access to a computer or the Internet, limited health literacy, data security, and data integration across multiple information management systems.3,4,7 There also exist system design issues, including representing technical information to meet the needs of consumers, which can impede ePHR adoption and use.3,4 In the meantime, health care providers and organizations are actively encouraging the adoption and use of ePHRs by patients.

As consumers of health care services, older adults can benefit from online access to their health- and health care information. Although older adults may use online services less than their younger counterparts due to biophysiologic changes of aging, studies have documented a steady increase in the proportion of older adults accessing health information online.9 This is encouraging because online access to health- and health care information has the potential to remove barriers associated with aging, including living with multiple chronic conditions or physical limitations that can result in being homebound.8 Because of the changes associated with aging, careful attention must be placed on addressing these changes with the intended users in order to develop health information technology (HIT) that can effectively support the ongoing, complex needs of older adults.8

Interactive self-management technology for older adults with a variety of chronic health conditions is reported to be acceptable and useful.10 ePHRs with integrated educational tools and self-monitoring and tracking systems are examples of interactive systems that can facilitate improved patient engagement, improve overall health, and reduce health care costs.4 However, few studies have evaluated the effectiveness and value of ePHR-based self-management systems.11 In order to realize the full benefit of ePHRs, including self-management of chronic health conditions, there is a need for the development and evaluation of HIT that is grounded in the same theoretical underpinnings.

In this paper the authors present an innovative, theoretically based approach to usability testing of self-management tools in an ePHR with older adults. The paper presents these methods in the context of a new ePHR module designed to prevent falls: the Self-Assessment via a Personal Health Record (SAPHeR) system.

Falls Among Community-Dwelling Elders

Falls and their sequela among older community-dwelling adults are a national problem.12 Notably, the fall-associated mortality rate in this population increased by more than 45 percent from 1999 to 2007.13 In 2000, nonfatal fall-related injuries were estimated to cost nearly 19 billion USD.14 Since the population of older adults in the United States is expected to increase from 35 million to 72 million between 2010 and 2030, preventing deaths and injuries from unintentional falls is a high-priority public health goal.15

Falls Prevention Programs

There are several empirically based interventions that can reduce falls among community-dwelling elders.16–18 These include regular participation in exercise programs, modification of the home environment, and pharmacotherapy (including the administration of vitamin D, calcium, and hormone replacement therapy for women).19,20 Although exercise programs have been found to be one of the most effective methods for preventing falls in the community, it is often difficult to engage older adults and maintain their participation over time in these programs.18–20 Studies describing community-based dissemination efforts commonly describe complex programs that may be effective but also may be difficult to maintain over time due to costly administrative requirements and the need for specialized personnel.18–20 The use of technology-based interventions may be one method that could address these problems by providing a way for participants to self-manage their falls prevention activities and monitor their progress over time. In addition, this type of intervention may reach a broader group, where community-based interventions are limited to those in a particular area where the intervention is being offered. We describe a falls-prevention self-management system that is integrated into a hospital-based ePHR that can be easily implemented and used by community-dwelling elders.

An ePHR-Based Falls-Prevention Self-Management System

The purpose of the SAPHeR system for community-dwelling older adults is the following: (1) educate them about the importance of fall prevention and support them in creating a safe home environment, (2) provide them with guidance on effective physical activities that promote falls risk reduction, (3) enable day-to-day monitoring of their personal (i.e., falls diary) and physical activities (i.e., exercise routine), and (4) have them self-assess their confidence in performing selected indoor and outdoor activities that could place them at risk of a fall.21 The SAPHeR system consists of four modules plus a sign-in and home page (Figures 1–4). Each module was developed to achieve one of the goals stated above. Module 1 (Figure 1) provides an overview of the content and features contained in the falls prevention system, including falls prevention instructions and interactive assessment and monitoring. The goal of this module is to provide information in a way that promotes easy navigation for the older adult. Module 2 (Figure 2) provides specific information and guidance about physical activities that can help prevent falls. For example, the user is able to access animated demonstrations with instructions of exercises, such as tai chi, to learn how to perform these activities at home. Module 3 (Figure 3) provides the users with a log to record specific falls-prevention activities they can perform through the use of the SAPHeR system, such as strength training exercises or personal notes and reminders. Module 4 (Figure 4) allows the users to self-assess their confidence in performing daily activities that could place them at risk of falling.

Figure 1.

System Navigation Page

Figure 2.

Falls Prevention Information Seeking

Figure 3.

Falls Prevention Monitoring

Figure 4.

Falls Prevention Self-Assessment

Research indicates that older adults perceive the risk of falling as obvious and at the same time deny personal risk while maintaining a desire for a competent, active, and independent lifestyle and identity.22 Therefore, the SAPHeR system is focused on enabling community-dwelling older adults to engage in self-management but also focus on the immediate benefits of strength, balance, and stability training, which have been reported as reasons for continued engagement related to improving mobility, health, confidence, mood, interest, and enjoyment.23

Development of the SAPHeR System

We applied the consumer-centered, participatory design approach that started by conducting four focus groups with target users to understand their knowledge related to the clinical domain of falls prevention.24,25 This was followed by four design sessions that focused systematically on the user, functional, task, and representational requirements as presented in the Zhang and Walji Task, User, Representation, Function (TURF) framework.26–30 The design sessions began by identifying the characteristics of potential end users—such as expertise, knowledge, skill, education levels, cognitive capacities and limitations, perceptual variations, ages, cultural background, etc. Building on the first design session, the second design session focused on cognitive activities to identify what functions (e.g., alerts, medical knowledge, communication, and other aids) should be supported by the SAPHeR system. In the third design session, a series of activities were conducted to identify the procedures and actions that participants carry out to achieve task goals by identifying representations as cues. We concluded our design sessions using a high-fidelity prototype of the SAPHeR system to identify whether we had constructed appropriate information display formats for given tasks based on the information we collected in the first three design sessions. A detailed report of the design approach is described elsewhere.24,25

Conceptual Framework

We used the TURF framework to inform the initial usability testing of the SAPHeR tool.30 According to Zhang and Walji, “usability” refers to the degree to which a particular system is useful, usable, and satisfying for the target user group.30 Task analysis consists of an in-depth identification, description, and breakdown of the specific tasks that users would complete using the tool. A user analysis consists of characterizing the target user group. This includes the intended demographic characteristics, a description of common computer aptitude among the intended user group and their typical computer use patterns. The representational analysis describes ways in which the tool will represent the desired information and related tasks to be completed by system users. Lastly, the functional analysis compares the proportion of desired functions with the actual functions or those that can be carried out using the tool.

Usability Definitions

Within the TURF framework, “usability” refers to a system’s degree of usefulness. Specifically, “usefulness” refers to how well a system supports the functions associated with a particular domain or field covered by the system and only those functions. It can be measured by determining the proportion of all domain functions that are actually included in the system. Whether or not a system is usable can be determined by whether it is easy to learn (i.e., learnability), is efficient to use, and provides for error recovery. In other words, can the user successfully complete the tasks the system was built to facilitate? User satisfaction under TURF refers to the users’ subjective impression of a systems usefulness and likeability for task completion.30

Methods

Study Design

We assessed the usability and in vivo feasibility of the SAPHeR system in two phases. First, a laboratory-based observational study was conducted to examine the initial usability of the system. Second, this was followed by an eight-week feasibility study. These components are described below.

For the laboratory-based component, a beta prototype of the SAPHeR system was incorporated into the test environment of the ePHR (MyNYP.org) in use at the New York Presbyterian Hospital system. Participants were asked to walk through specific tasks across the different modules that make up the SAPHeR system. Morae screen capture software was used to record participants’ interactions with the SAPHeR modules and to audio record their verbalizations (Techsmith, Okemos, Mich.). The feasibility component was conducted over eight weeks with the primary goal of evaluating in vivo use of the SAPHeR system and the potential influence of the system on fall self-efficacy, or “fear of falling.”

Participant Recruitment

We recruited community-dwelling older adults who were at least 55 years of age and lived in Northern Manhattan in the city of New York—regardless of fall history—for the usability and feasibility components of the study. Fall prevention requires risk reduction among older adults starting at the age of 55 years.31 Although the risk of falling is low at age 55, research has shown significant rates of injury after a fall for females beginning at this age.32 The views of both nonfallers and fallers as young as 55 years of age were thought to be important in the development of an ePHR-based fall prevention system. In the United States, 61.7 percent of households have someone 55 years of age or older who reports Internet use.33 Older adults were not eligible for the study if they were not able to walk independently with or without an assistive device. Older adults who did not access the Internet regularly were also excluded from the study.

Our goal was to recruit a total of 10 participants in accordance with expert recommendations for sample size in this type of usability testing.34 Recruitment for the usability study began by inviting participants from the initial SAPHeR design sessions. We then used a process of snowball sampling, identifying additional participants through the design session participants. In addition, we identified university employees that fit the age criteria who also resided in Northern Manhattan. Participants were asked to come to the university to participate in the usability testing. Each participant was given an explanation of the study protocol and signed an informed consent prior to initiating the protocol. Participants for the feasibility study were identified from a group who participated in a large-scale survey study in Northern Manhattan. Fifteen individuals participated in the feasibility study. All feasibility study participants had provided consent to participate in research prior to being contacted via telephone to participate. Participants were then invited to attend a training session on the use of the SAPHeR system at the Columbia University, School of Nursing.

Usability Testing Protocol

Prior to beginning the testing, two researchers (BS and RJL) identified specific tasks to be carried out by the participants for each module in the SAPHeR system. Two tasks were identified for the sign-in and home page. Five tasks were identified for Module 1 that focused on demonstrating the user’s ability to navigate efficiently through the SAPHeR system. Module 2 involved carrying out seven tasks that focused on the ability of the user to seek specific information about falls prevention activities. Module 3 required that the user complete five tasks that involved direct interaction with the system. Specifically, the user needed to enter information about performing different falls-prevention activities. In module 4, three tasks were completed that involved filling out a questionnaire that assessed their confidence in completing everyday activities without falling. A set of simple to complex tasks that participants were encouraged to walk through across the SAPHeR modules is presented in Table 1. Usability testing took place in a conference room in the Columbia University, School of Nursing. For the first two participants, two researchers and the participant were present for the testing. This allowed for both researchers to agree on specific issues, such as identifying when an error occurred during the testing. One researcher then conducted further testing. After obtaining informed consent, the researcher explained the testing process and provided an example of how the testing would be conducted. The researcher verbally guided participants by asking them to complete each task required (i.e., “Show me how you would find the falls prevention website and open it up”). Participants were asked to think out loud as they completed each task for each module. Their verbalizations were recorded along with their screen interactions as they carried out each task.

Table 1.

Examples of Simple to Complex Task Activities Performed by Usability Participants

SIMPLE (SIGN-IN/HOME PAGE)	MODERATE (MODULE 1)	COMPLEX (MODULE 4)
User goal:Access the falls-prevention website.	User goal:Access information about the consequences of falling and fall prevention.	User goal:Complete self-assessment questionnaire.
Task activity:Find and open the falls-prevention website. Recognize the falls-prevention image (i.e., woman falling). Recognize and read the words “falls prevention solutions” under the image of a woman falling. Click on image or words to access the falls-prevention website.	Task activity:Find more information about fall prevention. Recognize clickable links. At top right-hand side of page: underlined link At bottom left-hand side of page: last link under “featured sections” Image or text must be recognized as clickable links. Click on one of the links.	Task activity:Find the self-assessment questionnaire from the home page. Recognize the link on the right-hand side of the “What is your level of confidence” page. Click on the link.
	Task activity:Return to the system home page Recognize clickable link on right-hand side of “Falls prevention solutions home” page. Click on the link.	Task activity:Access the questionnaire. Recognize the text box on the “Take January Evaluation” page. Click on the link.
		Task activity:Complete the questionnaire. Subtask: Read the instructions. Recognize the instructions. Read instructions aloud. Subtask: Read first question and possible answers. Recognize the first question and possible answers. Read question and answers aloud. Subtask: Fill out the questionnaire. Recognize radio buttons as method for entering responses. Click one radio-button for each question. Subtask: Save responses. Recognize “Save My Answers text box. Click on “Save My Answers” save responses.

The researcher had a list of required tasks and the associated steps necessary to complete them. When participants had difficulty determining the next step in a task or chose an incorrect step, they were prompted toward the appropriate step by the researcher. When participants verbally stated that they could not complete a requested task, the researcher provided the minimal guidance necessary to complete the task. This enabled each participant to successfully complete each task while allowing the researchers to record all prompts and errors encountered during the testing process. After completing each module, the participants completed a user satisfaction survey—the Post Study E-Health Usability Questionnaire (PSHUQ).35 This questionnaire allowed the participant to rate the system across three domains: (1) system usefulness, (2) system quality, and (3) overall satisfaction.

Usability Data Analysis

The usability factors and criteria by which these factors were measured for each component of the analysis are described in Table 2. The criteria and related metrics were chosen based on the recommendations from the TURF framework as well as the recommendations of the International Standards Organization (ISO 9241-11) for components of software systems that define usability.30,36

Table 2.

TURF Framework Analysis Domains, Usability Measures, and Related Usability Metrics Applied to the SAPHeR Usability Study

DOMAIN	USABILITY MEASURE	CRITERION	METRICS APPLIED IN SAPHER STUDY
Task	Usable	Learnability	# hints/prompts required by user to complete tasks # errors by task per user
		Efficiency	# of task steps required by user Time on task per user
User	Satisfaction	Satisfying	Post-study E-Health Usability Questionnaire
Representational	Usable	Affordances	Proportion of affordances recognized by user
Functional	Usefulness	Design model comparisons	Proportion of domain functions in the SPAHER system vs. functions required by users

The task analysis measured how usable the system is. This was measured according to two criteria: learnability and efficiency. Learnability was measured as the number of hints, prompts, and errors for each task. Efficiency was measured according to the number of steps required by the participant to complete a task as well as the overall time taken to complete the tasks.

For the user analysis, participants were asked to complete a demographic and computer use questionnaire. The Post-Study E-Health Usability Questionnaire (PSHUQ) was used to evaluate user satisfaction with the SAPHeR system at the end of the usability testing session.35 Participants completed the questionnaire without assistance from the research team. The PSHUQ was adapted from the Post-System Study Usability Questionnaire (PSSUQ).37 The PSHUQ has undergone psychometric evaluation in a study evaluating the usability of an e-health website with Chronbach’s alpha coefficients exceeding .70 for two subscales (system usefulness (α = .958) and system quality (α = .933)).35 “System usefulness” measures the user’s ease of completing tasks and “system quality” measures the user’s satisfaction with the quality of information and the interface.35 The PSHUQ is an 18-item Likert-type scale with responses that can range from 1 (strongly agree) to 7 (strongly disagree). Lower scores overall indicate a higher degree of user satisfaction. An overall satisfaction score was obtained by calculating the average scores for each participant across both subscales, then calculating the aggregate mean and median satisfaction scores.

The representational analysis is used to evaluate how usable the system is for task completion. This is based on the ability of participants to recognize affordances built into an interface. The concept of “affordances” is related to the set of actions contained in an interface that enables ease of task completion in a particular context of use.38 It is grounded in Gibson’s theory of affordances, which is based on studies of visual perception.39 When combined with user knowledge, affordance recognition can be used to evaluate how well the features in an interface facilitate task completion.30 Affordances include visual cues (e.g., arrows) signaling possible actions and actionable objects (links) or widgets (pull-down menus) that can advance the user to the next state (or step) toward task completion.

For each task that we evaluated, we noted the features of the SAPHeR interface that were designed to provide cues to support action necessary for task completion by participants. Only features that were programmed in the SAPHeR modules were counted as affordances in our usability testing.

The functional analysis was used to evaluate system usefulness. The features desired by design participants were compared to the following: (1) all of the features in the SAPHeR system, (2) the features desired by them but not included in system, and (3) the features included in the system but not desired by the participants. The “user model” (i.e., features desired by design participants) and the “design model” (i.e., features included in the SAPHeR system) are compared in Table 4.

Table 4.

Comparison of User and Design Models of the SAPHeR System

USER MODEL	DESIGN MODEL
FALLS PREVENTION EDUCATION
Videos	Included*
Actionable images associated with educational text	Included
Medication management tool	Accessible via PHR**
PHYSICAL ACTIVITY INSTRUCTIONS/DEMONSTRATIONS
Tai chi	Included
Strengthening exercises	Included
Balancing exercises	Included
Community walking group	Included
Community exercise group	Included
Endurance exercises	Not included***
Not requested†	General physical activity information
SELF-ASSESSMENT/MONITORING
Not requested	Self-assessment tool
Monthly fall calendar	Included
Personal diary	Included
Physical activity	Included
SYSTEM NAVIGATION
Keyword navigation	Included
General search field	Not Included
Not requested	System navigation tool on each page
Not requested	Breadcrumb tool for navigation****
GENERAL HEALTH AND HEALTH CARE INFORMATION
Patient specific health information	Accessible via PHR
General health Information	Accessible via PHR
Physician information	Accessible via PHR
Physician locator for New York City	Not Included

Notes:

“Included” means that content or feature was a requirement of participants, and was planned and included in the system by designers.

“Accessible via PHR” means content or feature was a requirement of participants and existed in the personal health record (PHR) platform.

“Not Included” means content or feature was a requirement of participants and not included in the system.

“Not requested” means content or feature was not a requirement expressed by users but was included in the system.

A breadcrumb tool is a navigation aid that illustrates where the user is in a system, for example: Home Page ▸ Module 1 ▸ Module 2

Task Analysis Usability Scores

To evaluate learnability and efficiency of the SAPHeR system, usability data were first summarized for each module—including the system sign-in and home page activity. Because task steps varied from module to module, we normalized each participant’s usability data by calculating a single unitless measure to draw conclusions about learnability and efficiency of the system overall.40,41 Normalizing the data can improve the stability of the measurement (i.e., hints and prompts, errors, task steps, and time) and clarify interpretation (i.e., system use). Normalized usability scores can range from 1 (poorer performance) to −1 (better performance).

Interrater Reliability

Two researchers (BS and RJL) evaluated two of the participants’ usability data by coding each task step and identifying all hints, prompts and errors. Then, the coding results were compared to identify hints, prompts, errors, task steps, and time. All coding discrepancies were discussed until agreement was reached. The eight remaining recordings were divided between the two coders for analysis.

Feasibility Testing Protocol

Participants received a 60-minute training session on the use of the SAPHeR system in a lab setting. The training focused on an orientation of the system interface and functionality as well as the overall purpose of the system (i.e., falls prevention self-assessment and management). Participants were not given prescribed instructions in using the SAPHeR system. They were offered additional training in their homes, and a weekly courtesy call was made to inquire if they needed assistance using the system. Participants completed the modified falls-efficacy scale (MFES), programmed in the SAPHeR system (Module 4 [Figure 4]) after they received training on use of the system.21 During the last week of the feasibility study period, participants were called by a member of the research team and reminded to complete the MFES before the end of the week. The MFES is a 14-item questionnaire designed to measure an individual’s perceived fear of falling related to performing everyday indoor and outdoor activities (e.g., confidence in using public transportation or preparing a simple meal). The items are scored using an analog scale with not confident at all equal to 0, fairly confident equal to 5, and completely confident equal to 10. Total scale scores can range from 0–140 with results typically reported using an average across all 14 items (i.e., 0–10). Hill et al. reported sound psychometric properties of the MFES with internal consistency reliability 0.95 and test-retest reliability 0.95.21

Feasibility Data Analysis

Descriptive statistics were used to summarize the characteristics of feasibility participants, use of the SAPHeR system, and the change in MFES scores before and after the use of the system.

Results

User Analysis

We recruited ten participants for the usability testing. Their mean age was 72.7 years with a range of 60–78. Four participants were male and six were female. Five participants identified English as their primary language, three identified for Spanish, and two did not identify a primary language. Eight participants owned a computer and used the Internet regularly, while three reported that their family owned a computer. The highest education level they had obtained was a master’s degree (n=2). Three participants reported having a bachelor’s degree, three had completed high school, and one had completed the eighth grade.

User Satisfaction

Aggregate PSHUQ scores can range from 1 (strongly agree) to 7 (strongly disagree). Overall, participants were highly satisfied (Mean=1.58, Median=1.38) with the SAPHeR system. On average, participants agreed strongly (Mean=1.84, Median=1.8) that it was easy to complete tasks in the system. Moreover, they agreed strongly (Mean=1.43, Median=1.12) about the quality of the interface and the information provided in the system.

Task Analysis

Participants experienced the highest average number of errors (M=11.8) completing tasks related to self-monitoring in Module 4 (Table 3). They were asked to enter specific information about exercises performed and to log notes regarding their activities. The tasks associated with this module also required the most steps (M=28.7), took the longest time to complete (M=9.8 minutes), and required more hints and prompts (M=5.8) on average than did tasks associated with all other modules. Notably, participants’ ability to sign in to the SAPHeR system required the fewest prompts (M=1.0) in the shortest time (M=4.18 minutes) with the fewest errors (M=1.9) on average compared to all other models.

Table 3.

Average Learnability and Efficiency Results by SAPHeR System Component (N=10)

SYSTEM COMPONENT	LEARNABILITY				EFFICIENCY
	HINTS/PROMPTS		ERRORS		TASK STEPS		TASK TIME*
	M (SD)	RANGE	M (SD)	RANGE	M (SD)	RANGE	M (SD)	RANGE
Sign-In	1.0 (1.8)	0–6	1.9 (2.0)	0–6	9.9 (2.6)	5–14	4.2 (1.5)	1.8–6.2
Home page	1.6 (1.1)	0–3	2.6 (1.8)	0–5	7.4 (1.8)	5–11	4.3 (1.6)	1.9–6.0
Module 1†	2.3 (2.2)	0–7	5.1(4.6)	1–14	10.8 (3.9)	7–20	5.6 (1.9)	3.1–8.9
Module 2†	2.7 (1.9)	0–6	4.5 (3.2)	0–12	26.0 (2.9)	22–30	6.2 (2.6)	2.2–9.6
Module 3†	2.5 (2.0)	0–7	4.8 (2.4)	2–9	18.2 (5.1)	7–25	7.9 (2.2)	4.2–10.2
Module 4†	5.8 (3.1)	1–13	11.8 (4.7)	4–18	28.7 (8.4)	12–38	9.8 (4.1)	4.1–17.4

Note:

Task time is measured in minutes.

Module 1: System navigation page, Module 2: Falls prevention information seeking, Module 3: Falls prevention monitoring, Module 4: Falls prevention self-assessment

Figure 5 presents the normalized scores associated with the learnability of the SAPHeR system—including the frequency of hints/prompts and errors—across all tasks performed by participants. There was equal variation in the scores for hints/prompts (i.e., −1.2 to 2.4) and for the error scores (−1.1 to 2.3). All but one participant had scores for hints/prompts and errors that were <1.0, which indicates that users found it easy to complete tasks in the SAPHeR system overall.

Figure 5.

Composite Scores for Errors and Hints/Prompts across All SAPHeR System Components by Participant (N=10)

Figure 6 depicts the composite scores associated with system efficiency, including task steps and time to complete tasks across all tasks for each participant. Both task steps and time to complete tasks varied widely (i.e., −1.3 to 1.8 and −1.9 to 0.9, respectively). This variation is equal to or greater than the measures for learnability. Any efficiency in the SAPHeR system appears to be driven mostly by the number of steps needed to accomplish the tasks (i.e., 5 participant’s scores < 0). On the other hand, the inefficiency in the SAPHeR system may be attributed to the amount of time it took to accomplish the tasks (i.e., 7 participant’s scores > 0).

Figure 6.

Composite Scores for Task Steps and Task Times across All SAPHeR System Components by Participant (N=10)

Representational Analysis

Figure 7 shows the proportion of affordances recognized by participants within each module of the SAPHeR system. Affordances include indicators such as highlighted or underlined text, an icon, or an image indicating either actionable content or structure. Bold, underlined text was consistently recognized by our participants as being a way to navigate from one page to another. In contrast, the use of a backward arrow icon to navigate back to a previous page was consistently missed. Participants recognized the majority of affordances (i.e., > 60 percent) represented in each of the six system components. Participants were able to identify all or nearly all (97 percent) of the affordances when completing tasks associated with navigating through the SAPHeR system or completing the falls self-efficacy questionnaire. On the other hand participants recognized fewer affordances when signing in to the system (60 percent), entering self-monitoring data (61 percent), and navigating from the personal health record (PHR) home page (68 percent). In general, the affordances built into the system from component to component were enabling features for task completion.

Figure 7.

Proportion of Affordances Recognized by All Participants by SAPHeR System component

Note: Module 1: System navigation page, Module 2: Falls prevention information seeking, Module 3: Falls prevention monitoring, Module 4: Falls prevention self-assessment

Functional Analysis

Table 4 provides a list of features and content that make up the participants’ ideal model of the SAPHeR system (i.e., User Model), and whether these were included in the design model or already existed in the ePHR platform. The table also includes designer specified features and content that were not included in the participants’ conceptions and needs of the system. Fifteen of the twenty-two (68 percent) features and information sources included in the system were requirements of participants. An additional three (13.6 percent) features and information sources were required by participants but could not be included in the system due to technical limitations. The SAPHeR system designers included four (18.2 percent) features and information sources associated with fall prevention (e.g., self-assessment tool) and human-computer interaction best practices (e.g., system navigation tool). This comparison of the user and designer models revealed that most of the system should be useful to the intended user, community-dwelling older adults.

Feasibility Results

The 15 participants in the feasibility study of the SAPHeR system were predominately female (80.0 percent), on average 67 years of age, and mostly Hispanic and bilingual (53.3 percent). For the most part, participants had completed at least high school (i.e., 73.3 percent had a high school diploma, 20.0 percent had a college degree), and all reported accessing the Internet at least once weekly. Table 5 presents the changes on average in the modified falls-efficacy scale, or changes in confidence in performing a set of daily activities, before and after the use of the SAPHeR system by all 15 feasibility participants. Overall, there was an increase in the users’ confidence in performing both indoor and outdoor activities from 7.54 to 8.12. It appears that this increase can be attributed to a greater change in confidence in performing outdoor activities as compared to indoor activities (i.e., 6.73 to 7.47 versus 8.02 to 8.48, respectively). Based on the overall group differences, we calculated that the SAPHeR system could have a moderate effect (i.e., 0.32) on improving falls self-efficacy.

Table 5.

Average Overall and Subscale Scores of the Modified Falls Efficacy Scale (MFES) Before and After Use of the Self-assessment Via a Personal Health Record System (N=15)

	BEFORE M (SD)	RANGE	AFTER M (SD)	RANGE
Overall	7.54 (1.80)	2.5–10.0	8.12 (1.87)*	4.6–10.0
Indoor activities	8.02 (1.82)	3.33–10.0	8.48 (1.58)	5.0–10.0
Outdoor activities	6.73 (2.2)	1.0–10.0	7.47 (2.5)	4.0–10.0

Note: One sample t-test < .0001, moderate effect size of 0.32

Discussion

In this section, we discuss the use of the novel methodology employed and the results of the usability testing.

We have developed an innovative, consumer-health technology-usability evaluation based on an approach used to evaluate clinician-facing technology. Our innovative approach addresses the limitations of other usability testing methods that do not utilize consistent theoretically based methods to approach both design and system evaluation. We employed a conceptual framework known as TURF for the design and usability testing of the SAPHeR system.30 This framework incorporates several features including describing and predicting usability issues, providing usability definitions and measures, and ensuring built-in usability when applied to design. Unlike most usability evaluations that rely primarily on qualitative methods, we have demonstrated a mixed-methods approach to measure the usability of SAPHeR guided by the TURF framework. By implementing this framework, we realized efficiencies in system development. As a result the SAPHeR system required minimal changes after initial usability testing. Our results, while preliminary, demonstrate that a useful and usable consumer system can be achieved with the use of a systematic, theoretically based approach. The addition of the feasibility analysis further demonstrated system usefulness in vivo by indicating that actual system use has the potential to result in positive consumer outcomes.

Potential end users’ scores for learnability indicate that it was easy for them to use the SAPHeR system overall. While efficiency scores suggest that task performance was slow, our participants still successfully completed the assigned tasks. All but two of our potential end users were 75 years of age or older. Slower task performance may be the result of the normal aging process where a general slowing of cognitive processes often results in slower performance and increased errors when performing computer-based tasks.42 Other explanations could include low computer literacy even though all of the usability participants reported regular use of the Internet. Notwithstanding, in our case, the incidence of errors overall (i.e., across all SAPHeR system components) illustrates that our design has the potential to support the expected abilities of a wide range of older adults while minimizing the need for interface modifications.42 Still, errors could be minimized through the use of passive monitors (e.g., the use of a Fitbit during endurance activities) to support the task of data entry associated with monitoring physical activity.

The easy use of the SAPHeR system was further supported by our measure of satisfaction. Users on average reported that they were highly satisfied with the system. This result may be associated with the users’ experience of errors during its use. In other words, generally, as more errors were experienced more affordances were recognized allowing for successful task completion. Moreover, on average, greater errors were experienced in SAPHeR modules that contained more complex tasks (e.g., Module 4: falls prevention self-assessment). It is important to note that our satisfaction evaluation was done immediately after the laboratory-based usability testing, and user satisfaction could vary more broadly with extended use of the system.

In the representational analysis, we tested potential end users’ ability to recognize affordances that were built into the interface to facilitate task completion and navigation within and between the various SAPHeR components. Overall, our participants recognized more than 60 percent of the affordances across the system components. Those affordances that were recognized consistently included actionable text. For example, bold, red, large font was underlined to guide the user to a video that provided empirically based fall prevention activities. The text specifically stated, “Click here to watch a video”. Affordances that were not easily identified included smaller images and buttons such as a “sign-in” button and a “back” button that used a common backward arrow icon. While dependency on affordances across the SAPHeR components varied, the use of these cues may be reflected in ease of use (i.e., learnability) and efficiency (i.e., number of tasks steps) of the SAPHeR system overall. On the other hand, affordances in the system did not appear to have an impact on the time needed to accomplish tasks.

The findings from the representational analysis are consistent with interface design recommendations for older adults that endorse clear navigation aids, text-relevant images, high-contrast interface design, and large fonts.43,44 The ability to recognize these kinds of design features may be affected by typical changes that occur with aging.42,43 Specifically, cognitive changes such as altered perceptual ability combined with decreased visual acuity and difficulty with fine motor control may have an impact on the older adult’s ability to recognize, interpret, and click on some icons.42,45 Researchers have recommended that icons should be avoided when designing for older adults since they may not recognize them.45 Some of the icons that potential end users encountered in our study were a part of the ePHR test environment, and were not modifiable in any way. In our design, we did not rely on a single method to support navigation. Rather, we utilized many different cues to support various user abilities and preferences. Based on the representational analysis, when participants were able to successfully navigate assigned tasks without depending on affordances in general, we neither removed nor modified affordances intended to support these tasks.

The functional analysis in this study focused on information technology features that were desirable or undesirable by potential end users and the inclusion or exclusion of these features in the SAPHeR system. Nearly, 7 out of every 10 features desired by design participants were evaluated in our usability testing of the system. Half of these features were accessible directly within the system while the remaining features (e.g., general health information and personal health information) were present in the ePHR infrastructure and accessible from the SAPHeR system. Other features that were included in the usability testing but not expressly wanted by potential end users included a falls self-efficacy self-assessment tool, information about general physical activity, and tools (e.g., breadcrumbs and page-by-page navigation bar) to facilitate ease of use across the SAPHeR system components. Specific navigation aids were included because these are recommended to support usability for the older adult.42 And, still other features desired by potential end users were not included in the usability prototype, including a physician locator and a general search tool. These features were outside the technical scope of our study and not included due to technical limitations of the ePHR system infrastructure, respectively. When designing consumer-facing health information management or self-management systems, there exists a trade-off between including features that support individual (i.e., tailored) or population (i.e., targeted) health promotion and health care management. We think that we have achieved the latter of these two foci by addressing a large proportion of potential end users’ needs and incorporating evidence-based design and clinical domain features. This type of functional analysis can make apparent the design flaws that are related to user needs, technical limitations of existing infrastructures, and even designer bias.

We expect that our approach would be generalizable to other studies involving consumer HIT, provided there is a mechanism for engaging a group that can participate in repeated design sessions. Access to a local community organization with facilities that enabled Internet access and a comfortable space for meeting with participants was key to completing the initial design work. It is important to note that the participants in the usability evaluation should be similar to but not the same people that participated in the design sessions. In our case, we targeted a particular age group. Other studies may target those with a particular chronic disease or health issue. It may also be important in future work to test a design with participants with different characteristics to determine if findings from the design sessions can be generalized to those with varying attributes—for example, those in a similar age group but different racial or ethnic groups. If the HIT being designed can be altered according to the findings from the design sessions and usability testing, we think this approach could be applied in any consumer-facing development project.

The intentional use of the SAPHeR system by 15 community-dwelling older adults during an eight-week period in their homes establishes a proof-of-concept, and provides evidence for a future study on the effectiveness of this system for preventing falls. Participants determined on their own what content and functions they were exposed to during the eight-week evaluation period. The overall outcome was that community-dwelling older adults could benefit from exposure to the SAPHeR system. Specifically, exposure to the system may improve older adults’ confidence in performing everyday indoor and outdoor activities, or falls self-efficacy, thereby reducing their risk of a fall. We think that with further development of the SAPHeR system (i.e., integrating clinical assessment data with users’ self-assessment data to generate an automated, tailored plan of care) even greater improvements in falls self-efficacy can be realized by community-dwelling older adults.

Conclusion

We have developed an innovative usability testing approach that is informed by the same theoretical underpinnings we used to design a consumer-facing, ePHR-based falls-prevention system for community-dwelling-older adults (i.e., SAPHeR). Our usability testing approach builds on the principles of the TURF framework described by Zhang and Walji.30 By adapting the TURF framework for use with consumers, we have successfully demonstrated the utility of analyzing consumers’ technology use across multiple components (i.e., task, user, representational, functional) to evaluate the usefulness, usability, and satisfaction of the SAPHeR system. Moreover, it is likely that the system would be acceptable to a large proportion of community-dwelling older adults for use in preventing falls. For now, SAPHeR exists as a research interface. Our partners at New York Presbyterian Hospital are considering releasing SAPHeR for use among their older adult population. We are also considering using an open-source platform to make SAPHeR a plug-in system for wider adoption and evaluation of the system. Future research will consider the addition of real-time tailored recommendations for fall prevention based on specific user characteristics as well as additional user testing with a larger population to further develop our usability measures.