Usability, acceptability, and feasibility of the Implementation of Infant Pain Practice Change (ImPaC) Resource.

The Implementation of Infant Pain Practice (ImPaC) Resource is an eHealth tool designed to support infant pain practice change and ultimately enhance pain outcomes. The aim of this study was to determine users' perspectives on usability, acceptability, and feasibility of the ImPaC Resource. A descriptive prospective mixed-methods quality improvement study was conducted at a pediatric hospital in Canada. Individual "think aloud" interviews were conducted in a nonclinical environment (Phase A); "near live" testing was conducted while users interacted with the Resource in clinical setting (Phase B); individual "think-aloud" interviews were conducted in a nonclinical environment (Phase C). Outcomes included usability (System Usability Scale-SUS), acceptability (Acceptability E-Scale-AES), and feasibility. Interview transcripts were coded per a priori themes using deductive content analysis to create a structured categorization matrix. In Phase A, 10 clinicians interacted with the Resource in individual sessions. Median SUS score was 73.75 (range 52.5-92.5). In Phase B, four clinicians implemented the Resource in the neonatal intensive care unit (NICU) over 4 months. Median SUS score was 85 (82.5-92.5), and median AES score was 24 (21-24). In Phase C, an enhanced prototype was produced, and the same users from Phase B navigated the Resource in individual sessions. Median SUS score was 88.75 (85-95), and median AES score was 27.5 (25-29). Users considered the Resource as feasible for implementation, easy to navigate, engaging, intuitive, comprehensive, and evidence-based. Users highlighted the potential transferability of the Resource to other contexts and settings. The enhanced version of the ImPaC Resource was usable, acceptable, feasible, and met users' expectations and requirements. Results lead the way for evaluation of the Resource in a nationwide cluster randomized trial including 18 NICUs. This knowledge-rich platform is expected to enhance infant pain practices and outcomes in diverse clinical settings.

INTRODUCTION

Over the past three decades, there has been an exponential growth in research on reliable and valid pain assessment measures and effective and safe pain treatment interventions for hospitalized infants. Nonetheless, infants continue to undergo multiple painful procedures daily with insufficient treatment. , , Repetitive and/or untreated pain can result in short‐ and long‐term deleterious outcomes on health and development. This disconnect between research and clinical practice highlights an important research‐to‐practice gap due partly to ineffective implementation of new evidence.

To address this gap, a multidimensional implementation intervention, the Evidence‐based Practice for Improving Quality (EPIQ) tool, was developed and evaluated for pediatric pain practice change in 32 hospital units across Canada. EPIQ incorporated new evidence, implementation, and dissemination (ie, knowledge translation) strategies, and continuous quality improvement (CQI) methods to achieve improved pain assessment and treatment and reduced pain intensity in hospitalized children. , While promising, healthcare professionals (HCP) described challenges with the feasibility and cost effectiveness of EPIQ, improvements in pain assessment and treatment were only partially sustained 12‐36 months poststudy completion.

In response to these implementation and dissemination challenges and concurrent evolutions in technology, an eHealth intervention to support practice change in neonatal and infant pain, the Implementation of Infant Pain Practice Change (ImPaC) Resource, was developed. The Resource is an evidence‐based, seven‐step interactive eHealth tool designed to be self‐administered by a small group of healthcare professional (HCP) champions referred to as the change team within neonatal intensive care units (NICUs). The ImPaC Resource encompasses the following activities :

Step 1: The change team is asked to complete a checklist to ensure members know of expected responsibilities.

Step 2: Each member of the change team is asked to complete and reflect on the unit's readiness for change. This survey is adapted from the Alberta Context Tool (ACT) and is comprised of 34 items about: communication; space; culture; feedback process; and leadership. Upon completion, guidance on strategies that can be used to improve any suboptimal context areas is offered.

Step 3: The change team will conduct an audit on 10 medical records for infants who have been in the NICU. These audit data will be used as baseline data on pain assessment and management practices. Infant medical records are to be selected for a convenience sampling using a standardized approach. Based on the audit results, a pain assessment or pain management practice is targeted for practice change will be identified.

Step 4: The members of the change team will review the evidence about pain assessment or pain management included in the Resource. An aim statement will be developed to precisely articulate the expected percentage of change to be achieved (eg, 20%), and the interval of time required for achieving that change (eg, two months).

Step 5: The change team will select appropriate implementation strategies that will support the targeted evidence‐based pain management or assessment practice change. Educational and reminder implementation materials are downloadable and printable from the Resource, and the change team will decide on their use within the unit. Decisions are made regarding the target audience, intended number of individuals to reach, estimate cost and time for implementation.

Step 6: The change team will re‐audit 10 infant medical records in as per Step 3. The results of this post‐intervention audit will inform the percentage of change for the target practice change.

Step 7: The change team will examine the effectiveness of implementation strategies and identify a new practice change target and associated implementation strategies for the next change cycle.

The ImPaC Resource focuses on six validated infant pain measures and six pain management strategies. The goal of successfully implementing the Resource is to support evidence‐based practice change to improve infant pain practices, and thereby enhance infant pain outcomes across diverse clinical settings.

The Consolidated Framework for Implementation Research (CFIR) informed the Resource development as an implementation framework that guides the adoption and integration of evidence‐based (or best practice) health innovations into usual care through a quality improvement (QI) process. ,

A series of usability tests were undertaken to identify barriers to use of the Resource prior to its implementation and to guide refinements. Usability testing is defined as the extent to which a product can be used by targeted users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use. Usability testing has the potential to yield practical recommendations that can be applied to the structure, design, and redesign of eHealth tools, and to improve end users' acceptance and attitudes toward eHealth tools. Yet, usability testing has not consistently been employed to create advanced iterations of eHealth tools. In a scoping review of 133 articles reporting usability testing of eHealth tools, only 31% reported that test results were used to inform further iterations of the tools. In addition to assessing usability, measuring other implementation outcomes enables the understanding of implementation processes and enhances implementation efficiency. The present study sought to evaluate users' perspectives on the usability, acceptability, and feasibility of the ImPaC Resource to inform an enhanced version of the ImPaC Resource.

METHODS

A descriptive prospective mixed‐methods QI study was conducted between August 2017 and May 2019 at a single pediatric hospital in Canada. The study protocol was approved by the hospital's Quality Management Department.

The study included three phases. Phases A and C involved a “think aloud” approach in nonclinical environments, wherein an interviewer asked participants to verbalize their thoughts as they interacted with the ImPaC Resource. This strategy has been effective in identifying usability problems with a small number of users. In Phase B, the “near live” ImPaC Resource was implemented in a naturalistic clinical setting (ie, the NICU) involving participants who used it to guide carrying out real tasks , including pain assessment and treatment.

At each phase, several methods were used to collect quantitative and qualitative data (ie, observations, field notes, questionnaires, interviews, and focus groups). Interviews and focus groups provided in‐depth information on users' perceptions. Results and areas of Resource improvement were discussed within the research team. Modifications were implemented iteratively throughout the three phases of the study.

Phase A: “Think aloud” usability testing of the ImPaC Resource

Participants

Ten HCP employed on inpatient units that care primarily for infants participated in two cycles of “think aloud” sessions. Volunteer participants formed the convenience sample. Eligibility criteria required that HCPs were licensed, employed at the institution for at least one year, worked full‐time, and were familiar with using a computer. Trainees were excluded.

Procedures

Individual interview sessions were conducted in a nonclinical environment by a facilitator, one of the researchers (MB) who was knowledgable of the website and experienced with interviewing for research. Once written consent was obtained, participants were asked to engage in a 90‐ to 120‐minute session where they navigated through the ImPaC Resource using a prescribed scenario. The scenario was used to ensure participants were exposed to all features of the Resource, and to elicit comprehensive feedback on the outcomes of interest. As participants navigated through the Resource, the facilitator invited them to voice any comments, questions, or needs for clarification. Once the participant completed navigating the Resource, a 20‐ to 30‐minute semi‐structured interview was conducted to further elaborate on usability, barriers, and facilitators to using the Resource, feasibility and ease of use, navigational difficulties, content clarity, and suggestions for improvement. Sessions were audio and screen‐recorded with permission from participants.

Outcomes and measures

Usability was assessed using the System Usability Scale (SUS). , The SUS has 10 statements, each having a 5‐point Likert scale from “strongly disagree” to “strongly agree.” The SUS includes five positive statements (eg, “I found the various functions in this system well integrated”) and five negative statements (eg, “I found the system unnecessarily complex”), which alternate. Summary scores for the 10 statements range from 0 to 100, scores equal to or >70 indicate good usability. Data were entered in a centralized REDCap™ database.

Acceptability was measured in terms of satisfaction and understandability. These constructs were measured by single questions extracted from the Acceptability E‐Scale (AES). For both questions, answers varied from 1 (most negative) to 5 (most positive). Some examples of the statements included “How easy was the ImPaC for you to use?” and “Was the amount of time to complete the ImPaC Resource acceptable?”. Data were entered in a centralized REDCap™ database.

Feasibility was assessed in the semi‐structured interviews (Appendix 1).

Phase B: “Near live” usability testing of the ImPaC Resource

Members of the hospital's NICU pain management committee and QI team were invited to undertake the “near live” usability testing for the Resource. Four HCP from this committee/team agreed to participate as champions (ie, the change team).

Once written consent was obtained, the same facilitator (MB) engaged the change team in a 90‐minute orientation session that included an overview of the Resource and its' access. The change team completed the first two steps of the Resource at the orientation session and then was invited to navigate through the Resource and complete Steps 3 to 4 within a month, and Steps 5 to 7 over a 2‐ to 3‐month period. The facilitator communicated with at least one member of the change team biweekly (in person, by email, or phone) to discuss and address queries and difficulties. A 60‐minute focus group interview was conducted at the end of the four‐month study period by the facilitator who had interacted with the change team over the implementation of the Resource. The focus group interview guide comprised 12 questions and was based on the interview guide used in Phase A. The session was audio‐recorded with permission from participants.

Usability was assessed using the SUS , ; as described previously. Data were entered in a centralized REDCap™ database.

Acceptability was measured by the AES, a valid and reliable tool comprised of six items with answers that vary from 1 (most negative) to 5 (most positive) points. Summary scores equal to or > than 80% indicate the program is acceptable to users. AES items were reworded to adapt the measure to the evaluation of the ImPaC Resource. Data were entered in a centralized REDCap™ database.

Feasibility was assessed in the focus group described above. In addition, feasibility also included time spent on the Resource. The data were captured using an electronic weekly survey, completed independently by the change team members.

Phase C: “Think aloud” usability testing of the enhanced version of the ImPaC Resource

Following completion of refinements to the Resource, the four HCPs involved in Phase B were invited to evaluate the enhanced version of the Resource in “think aloud” sessions, as described in Phase A.

Individual sessions were conducted in a nonclinical environment by the same facilitator (MB). Once written consent was obtained, participants were asked to engage in a 60‐ to 120‐minute session where they navigated through the Resource. The facilitator asked participants to “think aloud” should they have any comments, questions, or needs for clarification while navigating through the website. Following this exercise, participants were interviewed to further capture their thoughts, likes and dislikes, perceived ease of use, navigational difficulties, content clarity, and suggestions for improvement. Interviews were conducted by the facilitator (MB) once users navigated through the Resource and lasted 10‐20 minutes. An interview guide comprised of nine questions was developed based on interview guides developed for Phases A and B (Appendix 1). Sessions were audio and screen‐recorded with permission from participants.

The same usability , and acceptability outcome measures were used as described in Phase B. Data were entered in a centralized REDCap™ database. Feasibility was assessed in the semi‐structured interviews.

Data analysis

All data were reviewed in the REDCap database for accuracy and completeness. Descriptive statistics were used to report on participant demographics and to evaluate quantitative data (ie, SUS and AES scores). Audio‐recordings from the think aloud sessions and interviews were transcribed verbatim and de‐identified by an independent, trained professional transcriber. A member of the research team reviewed the de‐identified transcripts for accuracy by comparing them verbatim to the audio and screen‐recordings.

Prototype revision and analyses were conducted between each of the three phases to allow for iterative improvements and refinements. Transcripts were contrasted with the relevant video recordings on the computer screen to ensure accuracy.

Two coders (MB and MR) individually reviewed a subsample of three transcripts and annotated them for possible themes. Annotated transcripts were then revised, and potential themes compiled and standardized. All themes were unified into a coding categories dictionary.

Coders reviewed and annotated each transcript using the code book. Two independent coders (MB and MR) read through the annotated transcripts and reviewed the corresponding video for each case. Initial themes were refined and collapsed, resulting in the final code book.

RESULTS

Participants

In Phase A, the ImPaC Resource prototype was tested in two cycles of usability testing with five participants per cycle. In Phase B, four participants interacted with the ImPaC Resource prototype during a 4‐month period. The four clinicians from Phase B participated in Phase C and evaluated the enhanced version of the ImPaC Resource. Participants' characteristics for all Phases are presented in Table 1.

TABLE 1

Study participants' characteristics

	N	(%)
Phase A (n = 10)
Gender
Female	10	100
Primary role
Staff nurse	2	20
Nurse practitioner	2	20
Occupational therapist	2	20
Quality leader	2	20
Charge nurse	1	10
Physician	1	10
Years of experience in the current role
1‐5	3	30
6‐10	2	20
>10	5	50
Years of professional experience in pediatric health care
1‐5	3	30
6‐10	1	10
>10	6	60
Phases B & C (n = 4)
Gender
Female	4	100
Primary role
Nurse practitioner	2	50
Staff nurse	1	25
Clinical pharmacist	1	25
Years of experience in the current role
1‐5	1	25
>10	3	75
Years of professional experience in pediatric health care
1‐5	1	25
>10	3	75

Usability, acceptability, and feasibility

Usability and acceptability scores were provided by all participants, across all three phases of testing. Table 2 summarizes the SUS and AES scores obtained from the study participants.

TABLE 2

Median (IQR) and range of System Usability Scores (SUS) and Acceptability E‐Scores (AES)

	Median (IQR)	Range
SUS
Phase A	73.7 (15)	52.5‐92.5
Phase B	85 (2.5)	82.5‐92.5
Phase C	88.75 (4.4)	85‐95
AES
Phase B	24 (1.5)	21‐24
Phase C	27.5 (1.7)	25‐29

Acceptability also included users' satisfaction and understandability of the Resource prototype and were assessed in Phase A. All participants were either satisfied (5/10, 50%) or very satisfied (5/10, 50%) with the prototype. The majority of participants considered the Resource prototype either easy to understand (6/10, 60%) or very easy to understand (2/10, 20%); the remainder were neutral (2/10, 20%).

In terms of time spent on the Resource, the team collectively spent 2.3 hours per week using the website.

Users' perceptions of the ImPaC Resource

In Phases A and C, users navigated through the Resource while sessions were audio‐recorded. For all three usability testing phases, a semi‐structured interview guide was used for individual interviews (Phases A and C) and the focus group (Phase B). The overarching aim of the interviews was to determine various aspects of feasibility. The guide included open‐ended probes. Transcripts were coded per a priori themes using deductive content analysis to create a structured categorization matrix. Pseudonyms were provided to ensure all participants remained anonymous. Code definitions and coded statement examples are provided in Table 3.

TABLE 3

Code definitions and users' perceptions of the ImPaC Resource

Code	Definition	Example Coded Statement
Feasibility	Refers to the extent to which the Resource can be effectively implemented into clinical settings	“The reporting piece…it's very time consuming and people use different systems so this will help standardize that nicely. You've given them a nice tool”
Flow and Navigation	Refers to the logical progression between steps as well as how easy the resource is to learn and understand	“The steps are very sequential and you know exactly what's to come”
Appeal and Engagement	Refers to how pleasant the Resource is to interact with, whether the design interface and overall layout are user friendly	“[The Resource is] user‐friendly and could definitely see them going up in their unit”
Progression and Prompts	Refers to the Web site layout and indicating messages	“very intuitive [and they] liked it more than version one”
Comprehensive Evidence‐Based Resource	Refers to the content and materials presented in the Web site	"It's obviously highly research‐based which is great. It's not just information that people would trust a lot of the tools that you're using are validated tools and that the resources provided are not just something that was created overnight."
Transferability	Refers to the extension and utility of the Resource in other contexts and settings	“The process of auditing and evaluating your unit's readiness for change, that's all about change. It doesn't necessarily have to do with pain. So I could see something like this, the structure, being used for something else”

Feasibility

All users found the Resource feasible for the setting, with three considerations: (a) the potential for poor documentation of painful procedures and related pain assessment and treatment strategies implemented, (b) high HCP turnover rate within intensive care units, and (c) accessibility of language. User A3 explained, “even though they do [pain assessments], they don't document it.” User A10 reiterated that “we hardly ever document the non‐pharmacological things that we do.” Participants from Phase C, such as user C2 state that “the reporting piece…it's very time consuming and people use different systems so this will help standardize that nicely. You've given them a nice tool.”

A user from Phase A, explained that “the biggest barrier will be the fact that the staff is contantly changing… so I think the struggle would be solidifying a team and then making progress” (user A1). The same user continued to explain that “I would like to say that even if you didn't have any knowledge of this process, I think it would be easy to learn and understand.” Users indicated that while their units do experience a constant change in staff, the Resource was effective given that it is easy to learn and pick up where left off by new members.

Finally, the language used in the Resource was described as “researcher‐oriented.” User A6 explained “this is very helpful for researchers and valid and stuff but the language is not going to be the language that nurses at the bedside are going to [use]… what it really means to be at the bedside.” Revisions were completed to make language more accessible before Phase C of usability testing. User C4 stated that “it looks like it's much easier to use… I think overall it's going to be easy to implement into practice now.”

Flow and navigation

Positive feedback describing navigation as both intuitive and logical were given in all three phases. Minimal difficulty in progressing through the steps was observed. User A1 explained “the steps are very sequential and you know exactly what's to come.” Similar comments were made during Phase B. For example, user B1 “liked the sequence and felt that it helped to integrate some of the concepts and actions” that needed to be completed. User C1 reiterated that “It seemed to flow quite nicely” and “it's precise,” “easy to read,” and “easy to understand” while evaluating the enhanced version of the Resource.

Appeal and engagement

Users suggested reducing the number of clicks required to navigate through the website. A common comment was that the Resource had a colorful platform. In addition, participants highlighted that the Resource was effectively segmented allowing for complex infant chart audits to be broken down and made more manageable for clinicians. Users A1 and A2 explained that they liked that it was “succinct” and “not too much to read all at once.” Participants enjoyed “the look of the Resource, the graphics and the information popping up on the screen.” User A3 stated that the knowledge translation tools provided were “user‐friendly and could definitely see them going up in their unit.”

Progression and prompts

In all three phases, the Resource was iteratively assessed and revised to accommodate user suggestions. Suggestions included save prompts related to entering patient data into chart audits, progression, and multi‐user warning prompts, and a navigation menu allowing end‐users to access different steps. Strong positive feedback was received about the enhanced version during Phase C of usability testing. User C2 described the Resource as “very intuitive” and they “liked it a lot more than [they] liked version one.”

Comprehensive evidence‐based resource

All participants described the Resource as providing concise, multimodal, and ready‐to‐use tools that were appropriate for clinical settings. Positive feedback regarding the accessibility of the knowledge and extensive user‐friendly library required minimal refinement throughout the iterative cycles of evaluation. User A1 stated “the Resource has a lot for everyone. For people who like auditory, people that want to watch the video and those who are interactive. But also just so many different strategies for getting people involved.” User B2 expanded to include “people learn from all different modes so I like that you thought of all the resources that are possible and already have them available.” A7 included that the Resource was “well laid out” and that “you could pick and choose from what you wanted.” User B1 explained “So I really liked that about the whole toolkit. It was easy to follow. Easy to work through. And you knew where to go to find things.”

Transferability

A recurring theme of transferability emerged as users highlighted the possibility for subsequent practice change interventions (ie, in various healthcare areas, hospital units, clinical care practices, and among a diverse range of users) to be modeled after the Resource. User B2 explained that “you could apply this to anything and not just pain. You could apply it to wound care. You could apply it to documentation, it's just a PDSA cycle.” User B4 reiterated “what I was thinking about while I was going through it and I know it's focused on pain, but it could be a very useful template for other practice changes.” More specific to the Resource, User A9 explained that “the process of auditing and evaluating your unit's readiness for change, that's all about change. It doesn't necessarily have to do with pain. So I could see something like this, the structure, being used for something else.” Finally, User B1 noted the potential to be used in remote areas, such as if “you're a centre where this is not part of your daily practice then it's great because it's a navigator to how to implement and evaluate change. I keep thinking about rural communities where maybe they don't have an educator, they're a small unit where they don't have established practices, something like this where it's on a website and it's easily accessible by anybody.”

Suggestions for prototype refinement

Required technical refinements were identified, discussed within the research team, and adapted based on end‐user feedback throughout each phase of usability testing.

Suggested enhancements to functionality and interface design from Phase A were improving visuals indicating progress and saved information through the Resource, allowing users to review prior activities, minimizing the number of clicks to complete the pain audit, improving the visibility of clickable icons, enhancing features for planning, and evaluating the implementation process. Phase A participants also highlighted the need for the audit tool to be embedded within the ImPaC Resource as opposed to documented in a separate location. In addition, clarification of gestational age of the infant versus corrected age was requested. No navigation errors or significant barriers to use were identified in Phase A.

After incorporating changes and enhancements suggested in Phase A, users from Phase B reinforced the need for improving visuals indicating progress through the Resource as well as improving the visibility of clickable icons, enhancing features for planning, and evaluating the implementation process. Furthermore, Phase B users' reinforced the need for incorporating the audit tool into the Resource, allowing for individual login sessions rather than a team login, and finally enhancing interaction between different steps and tasks throughout the Web site. Save prompts and mandatory fields were identified and implemented as well as requests for more variation in color. No navigation errors or significant barriers to use were identified in Phase B.

Phase C of testing the enhanced version of the Resource highlighted the need for clinicians to select multiple pain intervention strategies for a single painful procedure and a minor technical error (ie, being rerouted to a previous step) was corrected.

Table 4 details users' suggestions for prototype refinement and changes implemented, according to the phases of the study.

TABLE 4

Users' suggestions for prototype refinement and changes implemented

Design, aesthetics, and functionality	Examples of users' comments	Description of changes implemented
Phase A
Improving visuals indicating progress and information throughout the Resource	“I'm wondering if the steps could be listed somewhere. Just as a process.”	Permanent menu created to indicate progress and activities users are completing, have completed, and next steps
Allowing users to review prior activities	“Maybe giving a bit of functionality to be able to navigate or toggle back and forth”	Users may navigate through and review steps completed, library resources, and cycle archives at any time while logged in
Minimizing the number of clicks to complete the pain audit	“I guess there was a lot of clicking…it would be nicer if you had one screen that you could click through rather than opening a separate screen for each component”	Multiple‐choice options using skip logic implemented to cater options according to user input
Improving the visibility and functionality of clickable icons	“I think the downloading piece…duplicates and buttons when you don't need to have them.”	Standardized “save and next” and “generate report” buttons appear at the bottom of each step
Enhancing features for planning and evaluating the implementation process	“To have the ability to auto generate a run chart on the data that they've entered. I think that's the most important thing.”	Summary reports can be generated for infant chart audits, even before completion of the charts review
Clarification of gestational age versus corrected age	“These are gestational ages so you can have a 28‐weeker who is one month old. So, you can have here a gestational age and you have a postnatal age, the question or the issue is which one do you want?”	Corrected gestational age of infant is specified in chart audit
Phase B
Need for embedding the audit tool into the Resource	“The most difficult part was the audit tool…it made it so you couldn't submit or upload without all those areas being filled in.”	Infant chart audit process embedded within the Resource. Navigable between charts, procedure times, and can be edited/modified at any time during the step
Individual login sessions rather than a team login	“Different experiences based on role may lead you to have different perception and I think that needs to be considered. If you're asking people to log in as a group and complete these tools…or if it's more valuable to have the different members complete it separately and then for it to be analyzed.”	Individuals log in independently to complete Resource steps. However, they are oriented to work as a team throughout all the steps
Save prompts and mandatory fields were identified	“I wanted to make sure that it was saved” “Are you sure you've completed this patient before you go to the next?” [Mandatory fields]	Pop‐up messages indicating saved information displayed each time user clicks save Mandatory field prompts appear once user clicks to save information without inputting all required fields
Phase C
Need for clinicians to select multiple pain intervention strategies for a single painful procedure	“It only lets me pick one. So, I might have done sucrose, non‐nutritive sucking and swaddling all together but I can only pick the one. I would like the interventions to be combined.”	Multiple intervention combinations created as options for users completing infant chart audit

DISCUSSION

The aim of this study was to describe the three stages of usability testing of the ImPaC Resource, from it's earliest prototype to the most refined version. “Think aloud” and “near live” usability testing approaches generated unique insights and refinements to the ImPaC Resource. These approaches allowed for a multi‐faceted understanding of users' perceptions of the eHealth tool and how they interacted with the Web site during clinical care. Furthermore, “think aloud” and “near live” methods were well‐suited to moving through the enhanced version of the Resource as it was possible to identify strengths, areas of improvement, and required modifications to the Web site.

Usability and acceptability scores improved through the three phases of this study. The SUS was used to evaluate users' subjective perceptions of a wide range of technologies , while the AES explored items such as ease of use, understandability, satisfaction, and usefulness of eHealth interventions. The increasing SUS and AES scores obtained across the three phases of usability testing, along with feedback provided by users throughout the study, demonstrates progressive improvements based on participants' comments and suggestions.

In terms of feasibility, users from Phase B spend 2.3 hours a week navigating through the Resource. Users reported no major challenges to implementing the Resource and its related activities in their clinical setting, as part of their roles. In addition, interviewed users consistently described the potential of the Resource to be successfully implemented and used in the clinical setting across the three Phases of the study.

Usability testing is a critical step in the development and refinement of online interventions and solicits user's feedback to learn what does and does not work, and to explore potential gaps in information or functionality using iterative cycles. Each round of testing provided unique insights that informed improvements in overall navigation and design. Design improvements included progress indicators to support navigation and reduction in clicks for task completion. Substantial back‐end architecture changes suggested in Phases A and B were implemented for the enhanced version of the Resource, and tested during Phase C. These changes included incorporating the audit tool into the ImPaC Resource and creating individual login sessions.

Usability testing combined with iterative improvements culminated in an enhanced version of the Resource that appears to have successfully met users' expectations and requirements. Across the three phases of the study, users consistently felt the Resource was well organized, logical, visually appealing, and interactive as well as well suited for the neonatal and infant care units' environment. The ImPaC Resource was considered user friendly and comprised of comprehensive evidence‐based information. In addition, a strong relative advantage if the Resource was identified by end‐users. Participants highlighted the potential of the Resource to serve as a template for practice changes in other areas of clinical care and with different stakeholders, which confirms the relevance of its development based on dissemination, implementation, and QI principles.

Exploring and acknowledging users' needs in the development and refinement of researched‐led eHealth tools for pain assessment and management is associated with tool availability, and thus, with reducing potential for research waste. To our knowledge, the Resource is the first research‐led, theory‐driven eHealth tool to support clinicians in a practice change process on infant pain. Recently published reviews indicate a plethora of pain‐related eHealth interventions targeted at patients, families, and clinicians. , , However, none of the interventions described focus on the neonatal and/or pediatric pain practice change process.

Following the completion of usability testing, the ImPaC Resource will be evaluated in an effectiveness‐implementation hybrid type 1 study involving 18 level 2 and level 3 neonatal units across Canada (a recruitment video for the ImPaC Resource study can be accessed at https://youtu.be/hBSqDZe_3sc). The goal is to determine the intervention and implementation effectiveness of the Resource in changing neonatal pain practices in Canadian hospital NICU settings and how organizational context influences these outcomes. Further challenges will include ensuring the pain‐specific content continues to be evidence‐based and current, and then, the Resource is translated into different languages to make it available to a broad range of users. From a technology perspective, customizing the Resource to mobile devices may be a future requirement.

Future research will be required to determine whether the Resource would function equally well when delivered in different organizational contexts and to audiences with different social and cultural backgrounds. Furthermore, using the Resource as a platform to foster diverse practice change processes in health care is another important area to be developed and investigated.

Limitations

The three sets of usability testing were conducted in a single, highly specialized pediatric hospital in Canada. The institution is strongly committed to high‐quality pediatric pain prevention and treatment practices through an interdisciplinary Pain Centre that seeks to integrate best pain practices into clinical care, education, research, and training. Therefore, participants were recruited from a homogenous sample of users who are highly qualified for clinical care, as well as highly familiar with pediatric pain and QI processes.

The participants were also homogenous in terms of gender. Preferences in design and usage might be influenced by gender; thus, the lack of male participants in this study may limit generalizability of the findings. Although male personnel were involved with development of the prototype as well as the refined version of the Resource, further studies including male users are needed to extend the generalizability of our results.

In addition, all participants were computer literate and devices used to access the website were fully functional and up to date. Determining how less skilled users or users with limited access to appropriate devices or Internet access will interact with the Resource need further investigation.

CONCLUSION

The enhanced version of the ImPaC Resource is considered usable, acceptable, feasible, and met users' expectations and requirements. Blending different usability testing strategies (ie, “think aloud” and “near live”) enabled an iterative process and produced an enhanced version of the ImPaC Resource that is now ready for intervention evaluation.

CONFLICT OF INTEREST

The authors declare no conflicts of interest.