Usability of the Participation and Environment Measure Plus (PEM+) for Client-Centered and Participation-Focused Care Planning.

IMPORTANCE: The Participation and Environment Measure Plus (PEM+) is a new electronic health application to help caregivers contribute to client-centered and participation-focused care planning for their young child. The PEM+ is designed to help caregivers build on their participation assessment by determining priority activities, setting specific goals, and identifying strategies for goal attainment.
OBJECTIVE: To examine the usability of the PEM+.
DESIGN: A single-arm usability trial.
SETTING: Two early intervention and early childhood educational programs. PARTICIPANTS: Six caregivers of children ages 0-3 yr with developmental delay and receiving rehabilitation services. INTERVENTION: Caregivers who were eligible and enrolled were given access to the PEM+ for 1 wk and instructed to complete one iteration. OUTCOMES AND MEASURES: Usability was assessed according to caregivers' report of PEM+'s technical effectiveness and ease of use, ease of learning, and user satisfaction via the Usefulness, Satisfaction, and Ease of Use questionnaire.
RESULTS: All caregivers completed the PEM+ online once and in its entirety (mean completion time = 13.6 min). Mean technical effectiveness ratings ranged from 5.7 to 6.3 out of 7.0, and mean ease of use, ease of learning, and user satisfaction ratings were 4.4, 5.4, and 4.2 out of 6.0, respectively. CONCLUSIONS AND RELEVANCE: Caregivers of young children can navigate the PEM+. Trends in survey feedback informed optimizations for the PEM+ before further feasibility testing. WHAT THIS ARTICLE ADDS: This study provides preliminary evidence for a promising tool that can help to customize the way practitioners partner with families to create a plan for occupational therapy services.

Client-centered care, which is well established in occupational therapy, is a key component of health care reform that is intended to promote positive health outcomes, increase client satisfaction, and ensure efficient use of resources (Constand et al., 2014; Mroz et al., 2015). Client-centered care requires interprofessional provider teams to consider the needs and priorities of clients and their family members when creating and implementing a care plan. Team members must establish effective, efficient, and collaborative ways to communicate with families and with each other throughout the process of care (Amtmann et al., 2011; Brewer et al., 2014; Chiarello, 2017). As part of interprofessional care teams that work to improve client participation in occupation, occupational therapists are poised to reinforce client-centered care, beginning with initial care planning (Law et al., 1995; Mroz et al., 2015; Stoffel et al., 2017).

Barriers to client-centered care include time and resource constraints as well as the perceived power differential between providers and clients (Whalley Hammell, 2015). Technology-based assessments may help to minimize these barriers by providing diverse clients with options for how they engage in planning and monitoring participation-focused care for their child (Wang et al., 2016). Some assessments supporting client-centered practice, such as the Canadian Occupational Performance Measure, now provide an option for electronic administration (Donnelly et al., 2017)

The Participation and Environment Measure (PEM; Coster et al., 2012) is a participation-focused assessment that allows caregivers to evaluate their child’s participation (e.g., frequency and involvement in mealtime and whether change is desired), their perceptions of environmental supports and barriers to their child’s participation (e.g., sensory qualities in the home), and current strategies for supporting their child’s participation at home, at daycare or preschool, and in the community. It also can be administered electronically. The Young Children’s PEM (YC–PEM; Khetani, Graham, et al., 2015) is a psychometrically sound measure that yields a range of information relevant when developing a young child’s occupational profile. The YC–PEM does not, however, automatically yield a care plan that caregivers can use to facilitate the collaborative care process and improve their young child’s participation in occupations.

The PEM+ is an electronic health (e-health) application that addresses the need for flexible options to improve family-engaged care planning and promote client participation. It was designed with caregiver and provider input (Khetani, Cliff, et al., 2015, Khetani et al., 2017) to help caregivers use their YC–PEM results to determine activities that are a high priority for change, generate specific and meaningful goals, and identify strategies for goal attainment. The PEM+ was found to be accessible to caregivers when delivered by phone, which prompted the development and testing of a web-based version (Khetani et al., 2017).

The purpose of this study was to examine the usability of the PEM+—specifically, its technical effectiveness, ease of learning, ease of use, and user satisfaction. Usability is the degree to which products are effective, efficient, and satisfying for people to use (Jokela et al., 2003; Zhou et al., 2017). Usability testing is a crucial step when developing user-centered web applications so that design and layout issues that affect user navigation can be resolved before the tool’s feasibility and preliminary effects are tested (O’Malley et al., 2014). Study results informed revisions to the PEM+ prototype before further testing.

Method

Participants

The PEM+ is designed for use by caregivers of children 0–5 yr old whose participation may differ by age group (e.g., 0–3 yr and 4–5 yr; Dunst et al., 2002; Law et al., 2012). In accordance with recommended usability testing guidelines (Nielsen, 2012), we established a target recruitment of 5 or more caregivers of children 0–3 yr old for this first usability study.

Participants were 6 caregivers recruited through convenience sampling from two programs, one in Chicago and one in Denver. Chicago participants were recruited from Blue Bird Therapeutic Day School, an early childhood educational setting that includes intensive group and individual therapies. Denver participants were recruited from Rocky Mountain Human Services, the largest early intervention program in the Denver metropolitan area. Caregivers were eligible if they (1) were at least 18 yr old; (2) read, wrote, and spoke English; (3) had broadband internet access; and (4) had a child between 0 and 3 yr old who was accessing services for developmental delay. Ethics approval was obtained before recruitment and data collection (June–August 2017).

Intervention

Caregivers first completed the YC–PEM home and community sections online to evaluate their child’s current participation and whether they desired it to change. When caregivers desired change, they were asked to identify their current strategies for enabling the child’s participation in activities of that type. Once caregivers had evaluated their child’s participation, they evaluated environmental supports and barriers to participation in a setting. After completing the YC–PEM, caregivers were directed to complete one iteration of PEM+ online. The current PEM+ has five steps:

Rank, sort, or otherwise specify a priority activity;

Develop a goal for the child, including the type and amount of desired change in priority activity, using the YC–PEM and their own words;

Consider the relevance of existing strategies for goal attainment in the priority activity;

Consider environmental strategies for goal attainment; and

Review and confirm care plan content (Table 1).

Table 1.

PEM+ Approach for Remote Care Planning

PEM+ Step	Process
Step 1
Purpose	Prioritize problematic activities on the basis of PEM responses.
User tasks	User selects 1 of 3 routes for step completion:• Enter a specific activity.• Rank order activities from most to least problematic.• Sort problematic activities into “address now” or “later,” then select 1 “now” activity.
Step 2
Purpose	Set activity-specific goal for first priority activity.
User tasks	• Confirm reporting of child’s current participation level in priority activity and describe “current participation” in own words.• Set goal for child’s participation in priority activity via PEM scale and in own words.• Confirm or describe what can be achieved in 1 mo.
Step 3
Purpose	Create initial plan to achieve goals in specific priority activity.
User tasks	• Review or modify list of current activity-specific strategies from PEM completion.• Type in new strategies to add to the list.• Donate and collect additional strategies via strategy exchange.
Step 4
Purpose	Consider environmentally focused strategies to achieve goals in specific priority activity.
User tasks	• Review PEM and identify environmental supports or barriers to achieving goals.• Describe initial ideas for making changes to the child’s environment.
Step 5
Purpose	Finalize action plan and decide on next steps after PEM+ completion.
User tasks	• Review, edit, and confirm action plan.• Save, print, or email plan to share with providers, family members, and friends.• Opt to repeat PEM+ for another activity or stop for now.

Note. PEM = Participation and Environment Measure.

Measure

To test PEM+’s technical effectiveness, we used web analytics to record PEM+ completion rate and time. In addition, caregivers were prompted to complete a set of eight user tasks that corresponded with key technical features of the application (e.g., log out and back in) during PEM+ completion. They were then asked to rate each task on a 7-point scale ranging from 1 (very difficult) to 7 (very easy). To evaluate ease of use, ease of learning, and user satisfaction, caregivers completed the Usefulness, Satisfaction, and Ease of Use (USE) questionnaire (Lund, 2001) after completing one iteration of the PEM+. Caregivers rated 22 questions (e.g., “I quickly became skillful with it,” “It meets my needs”) on a modified 6-point scale ranging from 1 (strongly disagree) to 6 (strongly agree).

Data Collection

Participants were approached by study staff in person or via phone (N = 13). Eligible participants who consented to enroll completed an electronic demographic questionnaire to gather information about the caregiver and child (e.g., age, gender, race/ethnicity, household income and size, reason for service use). Caregivers then completed the YC–PEM home and community sections to access an online YC–PEM report with a web link to begin PEM+ (N = 6). After PEM+ completion, participants were routed into Research Electronic Data Capture (REDCap), a secure web application for capturing and managing data (Harris et al., 2009). Users completed the USE questionnaire in REDCap and were mailed $20.00 gift cards for their participation.

Data Analysis

Demographic and usability data collected via REDCap and an administrative dashboard were exported for analysis. Demographic characteristics and YC–PEM summary scores were reported for each user. To address the main study question, we derived usability estimates for the total study sample, according to technical effectiveness (PEM+ range, mean percentage completion rate, mean completion time, mean task difficulty rating) and USE summary scores (mean and range) for ease of use, ease of learning, and user satisfaction.

Results

Most caregivers were female, of Hispanic or Latino ethnicity, between 30 and 39 yr old, and residing with multiple children. Race, education, and income levels varied (Table 2).

Table 2.

Participant Characteristics

Characteristic	Gilly	Gale	Yara	Roslin	Jamie	Shae
Child’s age, yr	3.2	3.6	3.3	NA	NA	2.2
Caregiver age group, yr	30–39	30–39	40–49	30–39	40–49	30–39
Race	Caucasian	Caucasian	Other	NA	Asian	Caucasian
Ethnicity (Hispanic/Latino)	No	Yes	Yes	Yes	No	Yes
Education	College	Graduate degree	Graduate degree	College	Some college	High school
Income, $	>100,000	>100,000	NA	NA	NA	50,000–55,000
Children in the family	2	2	2	NA	3	3
Diagnosis	ASD	Not specified	ASD	Not specified	Not specified	Not specified
Desire change at home, %	62	100	100	54	100	0
Desire change in community, %	82	91	64	NA	64	18

Note. Pseudonyms were used to protect caregiver confidentiality. Desire change is the percentage of activities to which parents responded “Yes, desire change” when completing the Young Children’s Participation and Environment Measure. ASD = autism spectrum disorder; NA = not available.

All caregivers completed the PEM+ in its entirety. Caregivers completed one PEM+ iteration in 13.6 min on average (range = 3.3–22.0). Users identified priority activities in both the home and community settings. Caregivers who focused PEM+ on home participation identified “getting rest” as their priority activity. Caregivers who focused PEM+ on community participation selected a broader range of activities, including “shopping/errands,” “religious/spiritual gatherings,” and “community attractions.” The PEM+ sort feature was primarily used (67%) to identify a priority activity. Technical effectiveness of PEM+ features was rated an average of 6.1 (range = 5.7–6.3 out of 7.0). Tasks with the lowest ratings were accessing activity examples, knowing when work was saved, and accessing the parent-to-parent strategy exchange. Also, caregivers varied in how much assistance they needed to complete the PEM+ (e.g., logging in and out, reminders to continue writing their plan). Mean ease of use, ease of learning, and user satisfaction were rated 4.4, 5.4, and 4.2 (of 6.0), respectively (Table 3).

Table 3.

PEM+ Usability

Usability Parameter	Identification
	Gilly	Gale	Yara	Roslin	Jamie	Shae	Mean
Technical effectiveness
Use Q&A to identify activity examples	7.0	5.0	6.0	5.0	7.0	6.6	6.1
Knowing work has been saved	2.0	7.0	7.0	5.0	7.0	7.0	5.8
Request help via email	5.0	7.0	7.0	5.0	7.0	6.0	6.2
Request help via phone	5.0	7.0	7.0	5.0	7.0	6.3	6.2
Add new strategy	4.0	7.0	7.0	6.0	7.0	7.0	6.3
Access activity strategy exchange	6.0	7.0	7.0	5.0	6.0	7.0	6.3
Access environment strategy exchange	6.0	7.0	2.0	6.0	6.0	7.0	5.7
Logging in or out	4.0	6.0	7.0	6.0	6.0	7.0	6.0
USE questionnaire
Ease of use	2.9	4.7	4.5	5.1	3.5	5.8	4.4
Ease of learning	5.0	5.3	4.8	6.0	5.0	6.0	5.4
User satisfaction	3.0	3.7	4.0	5.7	3.0	5.7	4.2

Note. Technical effectiveness was rated on a 7-point scale ranging from 1 (very difficult) to 7 (very easy). USE Questionnaire items were rated on a 6-point scale ranging from 1 (strongly disagree) to 6 (strongly agree). PEM+ = Participation and Environment Measure Plus; Q&A = question and answer; USE = Usefulness, Satisfaction, and Ease of Use.

Discussion

Client-centered care is reinforced by offering clients accessible options for designing and monitoring care, thus encouraging intentional therapeutic interactions between providers and clients (Taylor, 2008). Technology-based tools with evidence of usability may reduce provider burden by improving the ease of gathering information about client need and related contextual factors (Wang et al., 2016) as well as by expanding provider reach for care processes such as care planning, a common task that typically requires significant investment of provider and client resources (Bailey, 2017).

In this study, all caregivers could complete the PEM+ care planning tool online on the first attempt and did so in less than half the time, on average, than it took to complete phone-based interviews using PEM+ (Khetani et al., 2017) or other interview-based assessments that are standard in usual care, such as the Routines Based Interview (McWilliam, 2009). PEM+ harnesses technology so that caregivers can have greater flexibility to clarify their care priorities, define goals, and identify and share strategies for goal attainment with service providers to facilitate the collaborative care planning process. This utility holds promise for reducing provider burden, in turn increasing the opportunity for more effective care planning. In addition, technology may help providers address known disparities in client access and use of services such as rehabilitation (Pew Research Center, 2018).

Similar to prior study phases, the web-based mode of administration was accessible to users for care planning across multiple settings (e.g., home, community) and types of activities (e.g., getting rest, community attractions). Caregivers who focused on improving their child’s home participation consistently prioritized improving their child’s participation in getting rest, a potentially relevant intervention target for families and a relatively new focus of occupation (American Occupational Therapy Association, 2014). Although caregivers used all three options to identify a high-priority activity in the first step (see Table 1), sorting continued to be a preferred option among the majority of caregivers who also had a higher rate of desire for change. This trend warrants future study with larger samples to examine the effect of caregivers’ rate of desire for change (i.e., high vs. low percentage of desire for change) on PEM+ completion trends.

Several trends in PEM+ survey feedback have informed optimizations to the electronic prototype before further testing. First, we found variability in how difficult it was for users to access examples of activities within each YC–PEM activity category. This finding prompted the development and integration of activity illustrations into the PEM+ prototype to visually prompt and focus users on the types of activities that commonly fall within a broader YC–PEM activity category of focus.

Second, we found variability in users knowing when their work was saved. This prompted clarification in wording to guide users once an iteration was completed, as well as the development of a PDF storage function so research staff could provide users with their reports on request.

Third, users varied with respect to their ability to access a parent-to-parent strategy exchange feature during the third and fourth steps of PEM+ (see Table 1) when identifying ways to support goal attainment. This point of user feedback has, in turn, prompted two considerations. First, we changed the structure of PEM+ to automatically show parents the strategy exchange in the fourth step. Second, accessible educational materials could help introduce caregivers to the task of identifying and appraising child and environmentally focused strategies for goal attainment before they access a strategy exchange.

Evidence regarding the impact of environment on young children’s participation is growing (Di Marino et al., 2017), but environment is a multidimensional concept and a relatively new focus of participation-focused intervention (Anaby et al., 2015; Chiarello, 2017). Although users complete an environmental assessment before beginning PEM+ (Khetani, Graham, et al., 2015), they may need support to apply this type of assessment information when developing a care plan. Last, users varied in how much assistance they needed to get through PEM+. This finding prompted the optimization of automated text and email messages based on user interactions with PEM+.

Limitations and Future Directions

Several of our study limitations informed optimizations to the PEM+ prototype or study design before beginning larger scale testing that is underway. Future testing will aim to recruit male and female caregivers across a broader age range. PEM+ is designed for caregivers of children ages 0–5 yr, but we were only successful in enrolling caregivers of children aged 0–3 yr. We have gained recruitment site clearance and ethics approval to expand inclusion criteria to caregivers of children 0–5 yr old. Second, caregivers from one of two recruitment sites experienced intermittent technical glitches, resulting in disruptions that may have resulted in lower ease of use and satisfaction ratings. A simplified electronic route was developed so that users can complete YC–PEM and PEM+ using a single web link. Last, a number of survey options for usability testing need further evaluation (Zhou et al., 2017). We adjusted the USE questionnaire to offer users a 7-point response scale. Because the USE has no established cutoff score, we will use Likert item responses and open-ended feedback.

Implications for Occupational Therapy Practice

This study has the following implications for occupational therapy practice:

E-health tools with evidence of usability, such as PEM+, can decrease response burden and promote family-engaged and participation-focused care processes.

The YC–PEM offers electronic administration so that caregivers can flexibly contribute their expertise when planning their child’s care.

The PEM+ is designed to extend the use of the YC–PEM by helping caregivers to specify and communicate their priority concerns and ideas for goal attainment.

Study results suggest that clients can navigate this e-health tool. Further optimization and testing are warranted.

Conclusion

This single-arm usability trial established the PEM+ as a promising tool for use by caregivers to support engagement in participation-focused care plan development. The results informed optimizations before further testing of the PEM+.