The importance of feasible outcome evaluations: Developing stakeholder-informed outcomes in a randomized controlled trial for children's respite workers receiving pain training.

Objective: Pain is common for children with intellectual and developmental disabilities. It is critical that caregivers have adequate pain assessment and management knowledge. The Let's Talk About Pain program has shown promise to provide pain-related knowledge and skills to respite workers; however, more systematic evaluation of the program is needed. This study aims to support Let's Talk About Pain's RCT development by using stakeholder input to help determine a feasible approach for collecting behaviorally based outcomes. A secondary aim is to discuss relevant considerations and implications for others in the disability field conducting similar work. Methods/Design: Four employees in children's respite organizations completed telephone interviews lasting approximately fifteen minutes and a questionnaire about feasible data collection approaches.
Results: The use of questionnaire and focus group methodology was determined to be the most feasible method to evaluate participants' pain-related approaches in practice. Conclusions: Special consideration should be made when making methodological-related choices during study development to help ensure study feasibility. The iterative approach described in this paper may also be helpful in clinical settings when designing program evaluations to enhance feasibility and suitability; it is particularly important for multifaceted organizations supporting individuals with complex needs including those with intellectual and developmental disabilities.

INTRODUCTION

Children with intellectual and developmental disabilities (I/DD) frequently experience pain and are often reliant on caregivers to assist with pain assessment and management. , It is therefore critical that caregivers of children with I/DD have access to relevant pain‐related knowledge and care approaches. Although most work has focused on primary caregivers and health providers, children with I/DD often spend time in a variety of settings with other caregivers. Recent work with secondary caregivers including respite workers for children with I/DD, residential support workers of adults with I/DD, and school nurses for children with I/DD has illuminated challenges with pain assessment and management including: inaccurate beliefs, , limited to no access to specialized pain education, lack of knowledge, , and role confusion with other support staff. Indeed, it seems that a knowledge‐to‐action gap exists for these caregivers.

Knowledge translation is one way to address this gap and is an important component of the research process. , Defined by the Canadian Institutes of Health Research (CIHR), knowledge translation is “a dynamic and iterative process that includes synthesis, dissemination, exchange and ethically sound application of knowledge.” The knowledge‐to‐action cycle has been used extensively to illustrate processes associated with knowledge translation. , The action phases focus on implementation or application of knowledge and include the following: identifying the problem; identifying, reviewing, and selecting the knowledge to implement; adapting or customizing the knowledge to the local context; assessing the determinants of knowledge use; selecting, tailoring, implementing, and monitoring interventions related to knowledge translation; monitoring knowledge use; evaluating outcomes or impacts of using the knowledge; and determining strategies for ensuring sustained use of knowledge. Phases in the action cycle can occur sequentially or simultaneously and interact with other action phases and/or knowledge creation.

To date, the primary researchers (LG and CMM) have developed and begun to implement a program of research for respite workers which targets several action phases described above. Researchers addressed the “identify the problem/identify, review, and select knowledge” and “assess barriers to knowledge use” phases by 1) gathering broad information about respite workers’ disability and pain‐related beliefs and care decisions and comparing these to young adults with limited to no experience supporting children with I/DD; ; 2) examining factors that contribute to respite workers’ pain assessment and management decisions; and 3) investigating the perceived pain training needs/preferences of children’s respite staff. Building off of these findings, “adapting knowledge to local context,” “selecting, tailor, and implement interventions,” and “evaluating outcomes” phases were initially undertaken by developing and successfully piloting the Let’s Talk About Pain program for respite workers supporting children with I/DD. As further work was needed in “monitoring knowledge use,” and “evaluating outcomes” action phases, a more systematic evaluation of Let’s Talk About Pain using a randomized controlled trial (RCT) was desired. Within the RCT, examination of both short‐ and longer‐term impact on pain‐related knowledge and perceptions as well as approaches in practice was considered important.

Although certain methods and procedures of the initial pre‐post Let’s Talk About Pain pilot study. could inform some RCT methodology, no information was available on the most feasible way to measure respite workers’ use of pain assessment and management strategies in practice. Thus, the study’s primary aim was to support Let’s Talk About Pain’s RCT development and the larger research program by gathering stakeholder input and determining how to best collect these more behaviorally based outcomes [ClinicalTrials.gov Identifier: NCT03421795]. This research was exploratory in nature; hence, there were no a priori hypotheses. In contextualizing this work and reporting the methods and results, a secondary aim of this manuscript is to discuss relevant considerations and implications for clinical researchers conducting similar work in the field. The Consolidated Criteria for Reporting Qualitative Research (COREQ) has been used as a guideline for reporting of this research.

METHOD

Participant recruitment

Following research ethics clearance from the institution’s research ethics board (REB16‐12‐599), participants were recruited by email. Eligible participants were as follows: (a) at least 18 years of age, (b) previous participants in the initial development or pilot study of Let’s Talk About Pain; and (c) part of a database of people interested in future research involvement. All participants therefore (a) had experience with the population of interest and (b) knew about the Let’s Talk About Pain program, suggesting that they would have adequate insight and backgrounds to address the research question. For reference, the development phase of Let’s Talk About Pain included 17 front‐line respite staff and five staff in children’s respite‐related management positions across three respite organizations. The pilot phase of Let’s Talk About Pain included 50 front‐line respite staff across two respite organizations. This means that a total of 67 front‐line respite staff and five respite‐related managers from five respite organizations were eligible. All eligible participants from this pool were sent emails about the study and participants were recruited in order of their email responses until data saturation was reached. Further potential respondents were thanked but informed the recruitment was closed.

Determining sample size when conducting qualitative research has been described as contextual in nature and dependent on several factors. Indeed, even instances of single samples have been argued to be “informative and meaningful”. For this study, researchers recruited participants and conducted interviews until data saturation was deemed to be reached (i.e., the point where there were no “new” data being generated by participants, where additional data collection could have been considered counterproductive ). In order to ascertain this, interviews were conducted and simultaneously reviewed during the recruitment process. Of note, there were few interview questions (see Table 1) and the questions were narrowly focused as the aim was to inform RCT methodology. All participants were also involved in the targeted research setting. As such, the potential for variability in participant responses may be somewhat more limited than is typical for interviews. Given that no new themes or information were identified in the third and fourth participant interviews, data saturation was believed to have been met and participant recruitment ceased.

Table 1

Interview discussion prompts.

Interview Guide Questions	Sample Probes
What do you think is the best way to gather this information from respite workers (e.g., observation, completion of additional questionnaires/checklists)?	Can you tell me a bit more about that?
Following a pain training workshop, how often do you think respite workers would have the opportunity to use new skills related to pain assessment (e.g., observing behavior) and management (e.g., using distraction) in their work setting?	Do you think they would have the opportunity to use these skills at least once per shift?
What factors are important to consider when deciding how to track respite workers’ use of new pain assessment and management skills?	Would these considerations be different depending on how we try to track this?
What types of challenges (organization‐related and staff‐related) do you think we might encounter when trying to track the use of these skills in respite settings?	What might help us to overcome some of those challenges?
Do you think it would be better to follow‐up with staff regarding use of their skills during or outside of work hours?	Why do you think ________ would be better?
Let’s return to our original question: Has your opinion regarding the best way to gather this information changed at all?	Why or why not?
Can you think of anything else that might be useful for us to know when deciding how we will assess/track respite worker’s use of pain assessment and management strategies?	Not applicable.

Procedures and materials

After providing informed consent, participants completed the following and were offered a $5 gift card.

A demographics questionnaire. Data gathered included the following: (a) general demographic information such as age and sex, and (b) information about participants’ employment positions and experience developing protocols, procedures, and staff evaluations in respite settings.

A semi‐structured telephone interview. First, researchers oriented each participant to the purpose of this study. Specifically, they were told that in light of the next steps in developing and evaluating the Let’s Talk About Pain program, researchers were seeking feedback from stakeholders on the most feasible way to measure how/whether respite workers are using pain‐related knowledge and skills at work following training completion. Second, participants then responded to a series of semi‐structured interview questions and probes posed by the researcher (see Table 1), with interviews being audio‐recorded and lasting approximately 15 minutes. All interviews were conducted by the first author, a female PhD candidate in clinical psychology with experience in pain and disability research as well as applied experience in respite settings. In addition to the participant and interviewer, a research assistant was present taking field notes during each interview. All field notes were expanded within 24 hours of each interview, and interviews were later transcribed and verified.

A postinterview questionnaire. This researcher‐generated questionnaire focused on potential follow‐up methodology identified by the research team a priori. These questions were posed in addition to the semi‐structured interview because researchers wanted to (a) ensure that explicit feedback about certain potential methods of interest was provided in case it did not come up in each interview; (b) gather descriptive quantitative data about perceived feasibility; and (c) provide participants with an additional opportunity to reflect and share any final thoughts that may have come to mind after completing the semi‐structured interview. Specifically, participants were asked to:

Rate how important they believe the application of evidence‐based pain assessment and management skills are in respite settings (0 = Strongly Disagree; 10 = Strongly Agree).

Review several possible approaches (e.g., observations, questionnaires) which researchers could use to measure skill use following training completion and rate their perceptions of the feasibility of each (0 = Not Feasible At All; 10 = Extremely Feasible) as well as respond to open‐ended questions about what makes each approach feasible/challenging, and other potential considerations.

Rank order a list of potential follow‐up methods based on preference (1 = Most Preferred; 6 = Least Preferred).

Provide any additional suggestions that come to mind regarding how to gather this information from respite workers in organizations.

Analyses

Quantitative analyses

Frequencies and descriptives were used to analyze closed‐ended and rating responses from demographic and postinterview questionnaires. Given the study’s sample size, medians and interquartile ranges have been reported.

Qualitative analyses

Inductive qualitative content analysis from an essentialist/realist epistemology , following the phases outlined by Elo and Kyngäs (preparation, organization, and reporting) was used to analyze open‐ended data from the postinterview questionnaire and participant interviews. All data including approximately 20.5 transcribed pages of data plus expanded field notes were analyzed concurrently during data collection in order to identify when data became saturated. Although responses to questions were initially analyzed separately, interview analyses were collapsed across questions due to overlap of participant responses. When reviewing the data, the researcher made notes and observations before freely generating initial categories associated with these notes. These initial categories were then broadened, and subcategories were applied where necessary. Finally, category descriptions were created to represent the different categories. The lead researcher engaged in the entire analysis process manually, and a research assistant also reviewed the process. As recommended in qualitative research, an audit trail in the form of a log was kept in order to document the content analysis and related decision‐making processes, multiple types of field notes were cross‐checked (i.e., condensed and expanded notes ), and researchers consulted regularly when interpreting the data.

RESULTS

Participants

Participants were two children’s respite workers and two managers of children’s respite programs (Median age: 50.00 years; interquartile range: 24 [1st quartile: 32.00, 3rd quartile: 62.50]; 4 female) with varying length of employment in respite organizations located in Southern Ontario, Canada (Median: 16.50 years; interquartile range: 28 [1st quartile: 4.25; 3rd quartile: 32.50]). All participants had experience developing respite care protocols (e.g., care plans, medication protocols), and three had experience evaluating staff performance in respite settings.

Participant results

Preferred data collection approach

Review of interview data revealed that participants believed several data collection approaches could potentially be feasible. This was echoed in participants’ questionnaire ratings of the feasibility of several approaches, with median ratings for four of five possible options falling between 7.5 and 9.5 out of 10 (10 = Extremely Feasible; see Table 2). Several participants also made suggestions about potential approaches not mentioned on the questionnaire. For example, two of four participants provided additional suggestions to collect data during staff meetings, and two of four participants commented on the potential for online data collection. One overarching approach was discussed by all participants as having considerable value. Specifically, participants highlighted the need for a multi‐method data collection approach which most commonly included direct observation as one method. In addition to direct observations, focus groups and/or interviews and/or questionnaires were suggested. For example: “…I’m thinking to do an observation and then do the interview with the staff – ya ‐ that would be a great step, to‐ you know, because after observation you would have a better picture ‐right?‐ of what and how we are doing, and then, you can interview, just to clarify what you saw.”

Table 2

Summary of staff ratings of perceived feasibility for various data collection approaches

Data Collection Approach	Median	Interquartile Range (1st Quartile, 3rd Quartile)	Range
Having a research assistant observe staff during shifts	7.50	3 (5.50, 8.00)	5‐8
Having a senior staff member or manager observe staff during shifts	9.50	4 (6.00, 10.00)	5‐10
Asking staff to complete a questionnaire at the end of shifts	9.50	4 (6.00, 10.00)	5‐10
Incorporating materials into children’s care profiles with routine paperwork	9.50	4 (6.00, 10.00)	5‐10
Asking staff to complete questionnaires periodically outside of work hours	5.00	7 (1.00, 7.50)	0‐8

Feasibility was assessed on a 0 to 10 Likert scale with higher scores reflecting higher perceived feasibility. Quantitative ratings on interview/focus group feasibility are not available, as these approaches were not explicitly listed on the postinterview questionnaire.

Data collection considerations and potential challenges

Participants identified three categories or types of considerations and potential challenges associated with follow‐up data collection methods:

Documentation Requirements. This category refers to any documentation that the organizations would need from researchers in order to move ahead with a specific data collection approach. Participants raised the issue of police record checks: “Do you have police clearances from the university or no?”. Consent from parents in order to be able to observe their children was also noted: “They [parents] would have to know you’re there and consent to you being there…”, as was the need to consult further with their human resources department: “I’ll ask our HR [Human Resources] if there is anything else that they can think of…”.

Logistics. This category refers to the needs of researchers and organizations to effectively coordinate resources. For example, time and adequate consideration of all shifts were raised: “I think it would depend how long it [data collection] takes…obviously if it’s going to take a long period of time it’s difficult to do it during work hours because, its‐ its busy. You don’t necessarily have the time to just kind of stop what you’re doing.”; “…as we’re working 24/7…I think that all three shifts have to be included…”. Staff availability was also an important consideration noted by participants: “It [observations] might be ‐ it would likely be a little difficult to coordinate just because…I think it depends on where you would be going to do an observation because there is ‐ there’s different clients coming in all the time, there’s different staff working all the time, the schedules are obviously only a 6 week schedule…so it could, it could be difficult.”.

Staff‐Specific Factors. This category refers to any considerations that are specific to staff who may be participating in the study and would likely occur regardless of the organization in which they are employed. For example, participants highlighted the importance of considering staff’s familiarity with clients as something that could impact certain data collection outcomes: “A lot of the time, especially on the weekends…with the respite, you get a lot of the part time employees that aren’t really familiar with the clients.”. They also acknowledged that staff may have varying levels of comfort with a given data collection approach: “Staff members would be uneasy with being watched by a senior staff/ supervisor and may not get an accurate view of what they are actually doing.”, as well as interest in study participation and engagement: “I think it [data collection outside of staff hours] would depend on the staff members, like some would and some wouldn’t.”.

An important response that occurred frequently while discussing considerations and barriers was that it may be difficult to collect data outside of work hours. For example, “It [data collection] would be easier within [work hours], just cause the staff have different schedules and different jobs.”; “We do not request the staff to do work for the agency that is unpaid. The work would need to be completed on site to pay staff for their time.” Further, it appeared that while many approaches may be feasible, some may be more or less feasible depending on the staff members or organization policies and structure. For example, “It depends on the children that are in on each weekend…”; “…it also depends on the person, what their designation is…not all staff are able ‐ from a policy standpoint ‐ to do certain things…”. For example, only certain staff may be able to administer medication, or organizations may have policies about the use of physical touch such as hugs.

Researcher interpretation and RCT follow‐up methodology decisions

In sum, several potentially feasible data collection approaches were acknowledged and discussed by participants. Considerations and challenges which could pose significant barriers were also identified. Participant responses were critical to consider when determining how to best collect more behaviorally based outcomes in the proposed RCT. Ultimately, researchers aligned with participants’ suggestions to adopt a multi‐method data collection approach and collect data during paid staff time wherever possible. The specific methodologies and concurrent considerations/barriers suggested by participants were then considered in the context of our multisite RCT spanning across Ontario, Canada. When reconciling participant recommendations within our context, issues related to ethics (e.g., participant confidentiality, ensuring staff do not feel coerced into participating) and other feasibility concerns (e.g., consideration of study resources available, distance between sites) were of primary concern. Although participants most frequently suggested direct observation as part of a multi‐method approach, this was unfortunately deemed by the research team as lacking feasibility for this specific project for several reasons. For example, the research ethics board had significant concerns about other staff or management engaging in direct observation, yet it would not be feasible for research assistants to complete these observations in a multi‐center RCT due to factors such as distance between sites, multiple shifts, and staff schedules. Furthermore, the need for parent consent when children are part of the observation would add an additional layer of complexity. A combination of questionnaire and focus group methodology was ultimately selected. We believed that questionnaire and focus group methodology would provide preliminary insight into participants’ pain assessment and management behavior that could later help to inform a feasible approach in a study where direct observation would be possible.

DISCUSSION

Considering the frequency of pain and associated challenges with assessment and management for children with I/DD, , caregivers who support these children require adequate pain‐related knowledge and skills. A program of research which aims to address the knowledge to action framework’s action phases is underway in attempt to address this knowledge gap. Systematic RCT evaluation of the evidence‐based Let’s Talk About Pain program is needed to further address several of these action phases including “monitoring knowledge use,” “evaluating outcomes,” and “sustaining use of knowledge.” The aim of the current study was to support the development of this RCT’s methodology, gathering stakeholder feedback regarding potentially feasible approaches for collecting behaviorally based outcome data from children’s respite workers. A secondary aim was to discuss relevant considerations and implications for others in the disability field conducting similar work.

Outcomes and implications

Participants suggested several potentially feasible data collection approaches including questionnaires, focus groups, interviews, and direct observations. In all cases, participants also identified a series of considerations and challenges that could render a specific data collection method more or less feasible. Participants did unanimously acknowledge value in employing a multi‐method data collection approach and expressed consistent views that data collection is likely most feasible during work hours. Further, there was acknowledgement by all participants of the inherent difficulties of finding an ideal “one size fits all” approach. Their perspective likely reflects the differences between and within staff and organizations that may influence feasibility and uptake of various programs and evaluation approaches. Indeed, organizations supporting children with I/DD may host a range of respite programs, support children with varying needs, and have different types of policies, procedures, and resources available to them which could ultimately impact the feasibility of various approaches.

Upon further consideration of research findings in the context of research ethics board standards and the larger RCT study design, we decided that a multi‐method approach using questionnaire and focus group methodology during paid staff time (where possible) was most feasible. As with any decision, advantages and disadvantages exist. For example, while this methodological approach would allow us to collect both qualitative and quantitative data at multiple time points, there would be no direct observation of skill use. Direct observation has the power to more accurately observe human behavior as opposed to tapping into what participants “think” they would do or “think they did.” However, real‐world observation also introduces numerous factors that may influence staff behavior (e.g., child’s history, staff familiarity with the child). If these factors are adequately addressed and observations are approached in a clear and structured way, these kinds of observations could help researchers learn more about perceived versus actual knowledge and skill application. Balancing methodology with feasibility, rigor, and ethical responsibilities can sometimes limit research design in larger‐scale studies. In other words, certain study designs may lend themselves better to particular methodologies.

In addition to informing our RCT, this work further supports the need for stakeholder involvement even in early stages of study design. It was undoubtedly critical to gather information from stakeholders and to know that the methods ultimately chosen were viewed as feasible and suitable in their eyes. There is a relative scarcity of the research on pain management in children with I/DD. Therefore, identifying and understanding research and evaluation‐related barriers may help ensure these endeavors can be completed as intended. While clinical researchers may be able to predict some barriers or adaptations required for a given context, end users are likely to have unique insights that can impact study design and aims. For example, one participant spoke about the need to include all three shifts (days, evening, and overnight) if researchers were to observe staff directly in a respite setting; this consideration was taken into account when thinking about the potential challenges with direct observation. Similar to Genik and colleague’s training development and pilot study, responses from participants in the current study provided important information and insight from front‐line staff and management that will maximize the feasibility of measuring the RCT outcomes.

Strengths, limitations, and recommendations for clinical researchers in the field

As part of the development of an RCT protocol targeting respite workers supporting children with I/DD, this study was designed to explore the feasibility of various data collection approaches directly with stakeholders including respite workers and children’s respite managers. Researchers selected a very specific pool of eligible participants who had experience with the population and setting of interest as well as familiarity with the training program to be examined in the RCT. In the case of the current study, this approach was helpful and may have contributed to data saturation with few participants. Clinically relevant issues were also identified including concerns about power differential that could occur if direct observations were conducted by management instead of researchers. Beyond more traditional research studies, appropriately managing these issues during implementation of a new intervention or a program evaluation would be advisable.

The use of interview methodology allowed for participants to respond in open‐ended ways with more rich and nuanced data, and questionnaires completed by participants then complemented this information. Indeed, the literature has revealed several benefits of stakeholder engagement for clinical researchers including increased participant interest in research, increased research and policy relevance, and help with goal setting. There are implications for quality improvement work within clinical settings as well. For example, connecting with stakeholders in the early stages of program evaluation can help those in clinical settings understand the types of approaches that are most likely to yield the desired information to match their program evaluation goals. In the case of the current project, the specific information gleaned from knowledgeable participants served as important considerations when balancing project aims, feasibility, rigor, and ethics. It is believed that this ultimately created a more feasible and meaningful research study. Clinical researchers are therefore encouraged to think critically about areas where informal or formal discussions with stakeholders may be a proactive way to identify and address factors related to outcomes and feasibility.

Of note, participants in this study were not given an opportunity to review the final data collection approaches and methods to be used when measuring the impact of the pain training on participants’ approaches in practice. Providing a “member check” with participants regarding the study or evaluation protocol could allow for an additional chance to catch any potential challenges or barriers related to a methodological approach. Beyond the direct benefit and implications for researchers, this approach may also benefit participants by allowing a feeling of ownership over the approach or feeling as though their concerns and experiences are validated by nature of shared themes across the larger participant group. Others approaching study design in this way may consider the value of this additional step based on their objectives, particularly in settings where stakeholders are invested in also contributing to the evaluation through their participation.

Importantly, this study included only a small number of participants within a relatively small geographic region. Although the ideal sample size for any qualitative study may vary, this study demonstrates the possibility for clinical researchers to conduct brief, directed, narrowly focused “pre‐studies” with key stakeholders to help inform their evaluation designs. As seen in this study, even a smaller scale study with four participants aided the researchers greatly in determining which methods may be most feasible in a respite setting and highlighted information that may not have otherwise been considered prior to participant recruitment or data collection. Of course, it is important to acknowledge that the researchers’ aims in this study were narrow and specific to support one aspect of RCT methodology development. It is quite possible that participants spanning a larger geographic area may have had different responses or preferences, or that broader research aims may have required additional data collection before saturation occurred. Clinical researchers conducting this type of research, particularly when employing qualitative methods, are encouraged to familiarize themselves with various types of data saturation and develop a clear approach that will help them determine when the data are saturated.

Finally, the postinterview questionnaire used in this study was such that it did not systematically vary details related to the proposed data collection approaches such as who would be collecting the data or when the data collection would occur. We encourage others to think carefully about the importance of gathering this information in a systematic way. For example, the omission of systematically varied details in the postinterview questionnaires did limit our understanding of the extent to which each detail may have influence participants’ feasibility ratings. For example, was an approach rated less feasible because of who would be collecting the data, the timing of data collection, or a combination of both?

CONCLUSION

In the context of a research program informed by the KTA framework, results from this study were used to inform aspects of the data collection methodology for a larger randomized controlled trial examining the impact of a pain training on respite workers’ knowledge, perceptions, and pain‐related approaches in practice with children with I/DD. Based on study results, a feasible approach using a combination of questionnaire and focus group methodology was selected and is incorporated into the RCT’s research protocol. Our method and findings support further development of a research program following a knowledge to action approach, and implications were outlined for clinical researchers conducting similar work to help drive research in this area. Enhancing the potential success of research and program evaluation focused on children with I/DD and/or their caregivers is imperative in order to improve the quality of life for these vulnerable and understudied children.

CONFLICTS OF INTEREST

The authors have no conflicts of interest to declare.