Building I-INTERACT-North: Participatory Action Research Design of an Online Transdiagnostic Parent-Child Interaction Therapy Program to Optimize Congenital and Neurodevelopmental Risk.

To adapt an existing virtual family-based mental health intervention learning platform (I-InTERACT-North), using participatory action research design, to meet the needs of parents and children with congenital, neonatal, and neurodevelopmental conditions that impact development. A purposive sample of parent knowledge users recruited from a large pediatric hospital (n = 21) and clinician stakeholders (n = 16) participated in one interview. An iterative process was adopted to implement feedback in the adaption of the learning platform. Qualitative thematic analysis was used to examine themes across participant feedback. Initial satisfaction with the adapted website was high. Qualitative results revealed four themes: acceptability, usability, recommendations, and dissatisfaction. Addressed with iterations, technical difficulties, wanting more information on content, resources, and intended audience were areas of dissatisfaction. This study reflects the importance of participatory action research methods in informing virtual mental health interventions. Future directions to improve the learning platform are discussed.

Children with neonatal brain injury and associated conditions are at heightened risk for long-term mental and physical health challenges (Bellinger & Newburger, 2010; Champigny et al., 2020; Gomes et al., 2014; Marlow et al., 2005; Roberts et al., 2019; Williams et al., 2017; Zhu et al., 2016). The most common neonatal brain injuries include neonatal stroke (affecting 1/4000 live births) and hypoxia-ischemic encephalopathy (HIE, 1–8/1000 live births; Kurinczuk et al., 2010; Lynch & Han, 2005). Similarly, newborns with congenital heart disease (CHD) and preterm birth (Dimitropoulos et al., 2013; McQuillen et al., 2003; Miller et al., 2005; Nelson & Lynch, 2004) are also at-risk for neurodevelopmental challenges. These conditions have well-known negative sequalae including attention and behavioral concerns in the early years (Champigny et al., 2020; Roberts et al., 2019; Steinman et al., 2009; Williams et al., 2017). Condition-specific guidelines for follow-up recommend early intervention services that strengthen behavioral and social functioning, as well as supporting parent wellbeing (Marino et al., 2012; Zhu et al., 2016). Yet very few parents access specific behavior or psychosocial support (Vyas et al., 2021; Williams et al., 2018). Despite the adverse psychosocial sequalae associated with these conditions, minimal interventions have been designed considering the needs and unique perspectives of these families.

Following a knowledge to action framework (Graham et al., 2006), preliminary work by our group highlighted high parent interest and need for mental health support following early brain injury. For example, parents expressed a strong interest in better understanding how their child’s condition affects behavior and social skills, how they as parents could better support these issues, and a clear preference for easily accessible virtual interventions over in-person and group therapies (Williams et al., 2018). Parents often identify their child’s mental health problems as primary concerns, yet services often do not integrate psychosocial treatment and support for families of children with neonatal brain injury into routine care (Marino et al., 2012; Mussatto, et al., 2018; Taylor et al., 2008; Woolf-King et al., 2018). Recent research has called increased attention to support parents of medically at-risk children outside of the traditional rehabilitative services focused solely on the child’s physical and language development (Peterson et al., 2021; Williams et al., 2019a, 2019b).

Internet-Based Interacting Together Everyday: Recovering After Childhood Traumatic Brain Injury (I-InTERACT Express) is an evidenced based virtual intervention created in 2006. The program is designed for parents of children ages 3 to 9 years old with mental health concerns about child behavior and parenting stress. It teaches specific parent skills and strategies to improve behavior and stress and provides an overall understanding of common outcomes associated with traumatic brain injury (TBI; Antonini et al., 2014; Wade, 2017). I-InTERACT Express combines seven online learning modules with seven videoconference sessions with a therapist who provides direct parent coaching (Wade et al., 2017). Given the alignment of the goals of the program with the priorities of our parents, (i.e., supported child behavioral health, empowered parents and provided psychoeducation regarding brain and behavior outcomes), the program offered a compelling model to consider for families of children with neonatal or congenital conditions.

As an initial step, our group conducted a feasibility trial of the I-InTERACT intervention renamed I-InTERACT-North to reflect the Canadian context. In this trial, therapists adapted content, but no changes were made to original 2006 learning platform (Burek et al., 2020). The adapted content provided by therapists included psychoeducation of common outcomes associated with congenital and neonatal brain injury, and related conditions, family impact, and the Canadian context (Burek et al., 2020). The initial feasibility trial of the I-InTERACT-North program indicated high parent satisfaction (100%) and acceptability (95%) with the program, as well as preliminary evidence of improved child behavior (d = 0.50), increased parent confidence (d = 0.45), and positive parenting behavior (d = 0.64). Overall, parents described the intervention as beneficial and highlighted the increased accessibility offered in the virtual modality. However, parents noted the learning platform needed significant improvements and updates (e.g., appearance, outdated infrastructure, accessibility) and removal of irrelevant traumatic brain injury content. Overall, families wanted a more accessible learning platform that could be accessed on portable devices, contained more relevant language, and reflected the diversity of the Canadian population that was not reflected in the original I-InTERACT Express program.

In direct response, the study team partnered with AboutKidsHealth at the Hospital for Sick Children to create preliminary adaptions of the I-InTERACT-North learning platform based on parent feedback from the initial feasibility trial (Burek et al., 2020). AboutKidsHealth is a health education resource producing and promoting online patient and family education, digital-based solutions, and digital learning opportunities with expertise in inclusive digital representation of underrepresented families in the Canadian context. Adaptations to I-InTERACT-Express were made based on content adapted by therapists in the feasibility trial (i.e., neonatal/congenital terminology and psychoeducation) and provided neonatal/congenital parent quotes and video testimonials. Learning platform inclusiveness of underrepresented families was first based on AboutKidsHealth expertise. However, the adapted learning platform in its entirety was not informed by parent knowledge users or clinical stakeholders.

Involving both patients and public stakeholders (e.g., parents, clinicians) in modification and adaption of intervention programs is essential to improve the quality and applicability of research (Ehde et al., 2013; van Meetern & Klem, 2018; Woodgate et al., 2018). Participatory action research (PAR) is an approach that involves stakeholders and knowledge users in all aspects of the research process, from study conceptualization to data collection, interpretation, and dissemination of findings (Fayad et al., 2018), which offers unique insight into research topics that may be inadvertently overlooked in typical research practice. Collaborating with various stakeholders and knowledge users is a fundamental tenant of the PAR approach (Wallerstein & Duran, 2006) and increases the efficiency and value of research by: increasing study relevancy to participants; improving participant recruitment and retention rates; diversifying representation in studies; and improving dissemination of research findings beyond traditional academic audiences (Crocker et al., 2018; Domecq et al., 2014; Morris et al., 2011). Attaining a better understanding of how the adapted I-InTERACT-North learning platform could be adapted and improved is integral in optimizing its applicability and potential efficiency of the overall intervention for families.

The purpose of the present study was to utilize PAR approaches (e.g., formal evaluations with knowledge users and stakeholders) to inform the adaptation and further development of the I-InTERACT-North learning platform specifically for children with neonatal, congenital, or neurodevelopmental conditions. Key knowledge users (e.g., parents) and stakeholders (e.g., clinicians) for whom the learning platform/intervention had direct relevance, were included to improve the applicability and value of the intervention learning platform. An iterative process consisting of initial evaluation, formal evaluation of the first and second iteration, consideration of redesign strategies of the first and second iterations, and re-launch of the learning program was used (Fayad et al., 2018). Interview questions were informed by a model of program evaluation design that actively engages stakeholders with purposefully open-ended questions to elicit participant feedback (Quinn Patton, 2014; Tracy, 2019). Parents were intentionally involved in the evaluation of the learning platform to ensure that it was applicable and valuable for the intended audience. The primary objective was to determine the acceptability of the adapted I-InTERACT-North learning platform from the perspectives of parents of a child with neonatal, congenital, or neurodevelopmental conditions, and clinician stakeholders through interview and focus group as well as questionnaire data. Based on results from initial phases (e.g., feasibility study), we hypothesized parents and clinicians will report ease in use, comprehension, as well as satisfaction with program content and the virtual service delivery modality.

Method

The present study utilized a convergent mixed-method research design engaging in both qualitative and quantitative methods (Creswell & Plano Clark, 2018). Ethics approval for the current prospective study (REB # 1000063660) was obtained through the research ethics board at the institution.

Participants

Eligible parents (knowledge users) from a large pediatric hospital in Canada and clinicians (stakeholders) were invited to participate between July 2020 to October 2020. A purposeful sample was selected among participants who had consented for follow-up from prior parent intervention and engagement studies. We focused on families with prior I-InTERACT-North experience but also families who were novel to the program. Families were eligible to participate if they met the following criteria: (1) parent of a child with a neonatal, congenital, neurological, and/or neurodevelopmental condition (i.e., neonatal stroke, CHD, HIE, preterm birth, Attention Deficit Hyperactivity Disorder; ADHD) and (2) English speaking. Exclusion criteria included lack of access to the internet. Clinicians were invited to participate between November 2020 and January 2021 from pediatric hospitals, universities, as well as community and private mental health clinics in Canada. Based on recommendations, we aimed to recruit 20 to 30 participants with high information power (Hertzog, 2008; Malterus et al., 2016). According to Malterus and colleagues, the more relevant information (e.g., aim of study, sample specificity, use of established theory, quality of dialog, analysis strategy) a sample provides, the smaller the sample size needs to be.

Procedure

Knowledge users (e.g., parents) were invited to participate in initial review of the adapted learning platform, given they are the intended audience of I-InTERACT-North. Eligible parents from prior studies who consented to be contacted for future research were contacted using an emailed information letter with follow-up by the clinical research coordinator. Parents were sent the study questionnaire using a secure web application (REDCap; Harris et al., 2009), the link to the learning platform to review, and were booked in for a one-hour semi-structured online focus-group or individual interview. All parents were invited to be a part of focus groups and were provided a variety of times and dates. After completion of the focus-group or individual interviews, parents were provided a monetary token of appreciation ($50 gift card). A purposive sample of clinician stakeholders was invited to participate to represent a variety of mental health settings (e.g., hospital, university, community based, private clinics) through online email invitations. To optimize inclusion, clinicians and parents who could not attend a focus group were invited to an individual interview or to submit written feedback. All participation took place virtually (i.e., electronic questionnaires, video conferencing, and online learning platform of interest).

An iterative process was adopted by the authors to implement knowledge user and stakeholder feedback (see Fig. 1) with a total of two iterations. In the first iteration, the focus was gathering feedback from knowledge users (parents) on the newly adapted learning platform. Changes were then made based on a thematic analysis of the interviews, field notes, and a parent questionnaire assessing ratings of satisfaction. Suggested revisions were reviewed by the primary, and senior author to ensure retention of program integrity and then the [removed for blind review] project management team for technical feasibility and then adapted on the learning platform. In the second iteration, the adapted learning platform was re-evaluated by a mixed group of parents and clinician stakeholders to assess acceptability and further feedback.

Fig. 1

I-InTERACT-North iterative process: learning platform development and evaluation

Measures

Website Evaluation Questionnaire

Parents completed a questionnaire assessing ratings of overall satisfaction for the learning platform of interest and specific components (e.g., the Website Evaluation Questionnaire; WEQ; Elling et al., 2012). The WEQ included questions about the ease of use and helpfulness of components of the online sessions (e.g., “the homepage clearly directs me towards the information I need”) and overall website experience (e.g., “I find this website easy to use”). The WEQ has been used in previous evaluations of the I-InTERACT program (Burek et al., 2020; Wade et al., 2006, 2009).

Interviews

Interviews followed a semi-structured cognitive-interviewing format and interview guides were informed by a model of program evaluation (Drennan, 2003; Quinn Patton, 2014; Tracy, 2019). All participants were given time/prompts to provide feedback not directly probed. Key investigators were excluded from this process to reduce researcher bias. Questions asked parents and clinicians to comment on their comprehension of content (e.g., “Were there any issues we should have included but did not?”), individual probes (e.g., “Overall, how would you describe your experience with the I-InTERACT-North learning platform?”; “What do you think of the topic coverage of the learning platform?”), identification of difficult to understand or irrelevant content (e.g., “Which modules were the hardest for you to understand?”), and recommendations to improve the learning platform (e.g., “What changes would you recommend?”).

All interviews (individual and focus groups) were conducted online using videoconference technology (Zoom). Prior to the interviews, all participants were instructed to interact with the website independently in the following manner: click through and explore every tab/page at own pace, take notes throughout noting any difficulties or components they wished to be included. The focus groups were facilitated by two team members (MD and MF) who were experienced in conducting focus groups and used interviewing techniques to minimize the power differential between the interviewer and respondents (e.g., establishing rapport, using appropriate language, active and relaxed body language; Morgan, 1997). The individual interviews were conducted by one team member (MF) using the same interviewing techniques as described above. During the focus-groups and individual interviews, the facilitators collected field notes about difficulties encountered, and ease of use and learning described by knowledge users/stakeholders regarding mobility through the learning platform’s psychoeducation, videos, and interactive components. Focus groups and interviews were audio recorded and transcribed verbatim by a research team member.

Analysis

Quantitative Assessment of Parent Learning Platform Experiences

To investigate parents’ impressions of the adapted I-InTERACT-North platform, frequency data were summarized from the WEQ. All quantitative data were analyzed using IBM Statistical Package for the Social Sciences version 27 (Armonk, NY: IBM Corp USA).

Qualitative Assessment of Knowledge User/Stakeholder Experiences

Parent and clinician interviews and written feedback were copied verbatim onto separate Microsoft Word documents to prepare for analysis. Data were analyzed independently by two investigators based on Braun and Clarke’s (2006) six thematic analysis phases: (a) familiarization of data, (b) initial code generation, (c) identifying themes, (d) reviewing themes, (e) labeling and defining themes, and (f) creating the report. An inductive approach was taken, and data were coded inclusively to ensure that context was preserved. An initial set of codes was produced in systematic fashion after coders read and re-read transcripts several times to help familiarize themselves with the data. These codes were examined for overarching concepts, involving a deeper examination and organization of the codes into broad higher-level themes. The authors then defined, refined, and explicated the themes and subthemes for further discussion. Thematic saturation was achieved, such that further analysis of the data revealed no new themes. For example, the overall theme of “happy with the program” was defined as “acceptability” and “useability.” Within that code, the coders identified subthemes including “enjoyed the learning platform,” “content easy to understand,” “learning platform was easy to navigate,” and “learning platform was aesthetically appealing.” This process was applied to all codes.

With respect to inter-rater reliability, each coder coded five transcripts independently prior to creating the initial set of codes together (and after reviewing all transcripts). Percent agreement was assessed by dividing the number of times coding was in agreement by the total number of code comparisons. Overall inter-coder agreement was established at 85% across 5 focus groups/interviews. The remaining transcripts were coded independently, reviewed together, and any disagreements were discussed to consensus with a third coder.

Results

A total of 23 parents were invited to participate and two parents declined due to lack of time. Parent knowledge users were exclusively mothers, had an undergraduate degree or above (86%), and English was their first language (95%). The mean age of parents was 38 years and 5 months (SD = 4 years, 8 months). Their child’s diagnosis included HIE (n = 12), CHD (n = 4), Stroke (n = 3), premature birth (n = 1), and ADHD (n = 1). Approximately half of the children were male (52%), and the mean age of children were 6 years and 9 months (SD = 1 year, 9 months). Eighteen (86%) parents had participated in the original I-InTERACT feasibility study and had previous knowledge of the program.

Thirty clinicians were invited to participate and 14 (47%) declined due to lack of time. A final sample of 16 clinicians participated in the study. Clinician stakeholders worked in a variety of mental health settings including pediatric hospitals (n = 13), universities (n = 2), and private practice (n = 1). The majority were practicing clinical psychologists (n = 6) or clinical neuropsychologists (n = 3) as well as a psychiatrist, developmental psychologist, psychometrist, neurologist, occupational therapist, department head, and executive director. We also had two participants in mental health executive administrative roles. Professional experience ranged from one to over 20 years of experience. A minority of clinician (n = 7) and parents (n = 1) were not available for interview but provided written feedback. Following the interviews, two major iterations were provided. Nineteen parents participated in the first iteration and two parents, and 16 clinicians participated in the second iteration. Results are summarized based on iteration and response.

Formal Evaluation: Iteration 1 (Questionnaire & Interviews), Version 1

Parent Learning Platform Satisfaction

Parents reported overall satisfaction with the adapted I-InTERACT-North learning platform as indicated in responses to the WEQ. All 21 parents found the information on the adapted learning platform helpful, and 20 parents (95%) found that the language used throughout was clear/easy to understand, provided sufficient and helpful information, and found the design attractive overall. However, three parents (14%) described the learning platform as unattractive and the information incomplete at the time of evaluation. See Table 1 for a detailed overview of WEQ results.

Table 1

Website Evaluation Questionnaire (WEQ) parent knowledge user responses (N = 21)

	Yesn (%)
Positive Website Experience
I find the information in the website helpful	21 (100%)
The language used in this website is clear to me	20 (95%)
The website provides me with sufficient information	20 (95%)
The homepage clearly directs me towards the information I need	20 (95%)
I find the information in this website easy to understand	20 (95%)
I like the way this website looks	20 (95%)
The website offers information that I find useful	20 (95%)
I find the design of this website appealing	20 (95%)
Under the hyperlinks, I found the information I expected to find there	20 (95%)
I know where to find the information I need on this website	19 (91%)
The homepage immediately points me to the information I need	19 (91%)
I find the information in this website precise	19 (91%)
I find the information in this website precise	19 (91%)
I consider this website user friendly	19 (91%)
I find the structure of this website clear	19 (91%)
I find this website easy to use	17 (81%)
Negative Website Experience
I think this website looks unattractive	3 (14%)
The information in this website is of little use to me	0 (0%)
I find the information in this website incomplete	3 (14%)
I had difficulty using this website	3 (14%)
I was constantly being redirected on this website while I was looking for information	2 (9%)
I find many words in this website difficult to understand	1 (5%)
It is unclear which hyperlink will lead to the information I am looking for	0 (0%)

Qualitative Findings

Results of the semi-structured interviews are outlined in Fig. 2, additional quotes can be found in Table 2. Qualitative findings reflected two overarching meta-themes (e.g., happy with the learning platform, suggested revisions), followed by an exploration of the themes and related subthemes.

Fig. 2

Overview of iterative process

Table 2

Interview common themes and supporting statement examples

Happy with the Learning Program	Acceptability	• “It had a lot of helpful tips and ideas that I didn’t even know would work for [Child].” [Parent of a 9-year-old girl with stroke] • “I liked the explanation for why you do things, like oh you should be direct or whatever it was and this is the effect that it will have for your child.” [Parent of a 4-year-old boy with HIE] • “It was very helpful, I still use them to this day.” [Parent of a 9-year-old girl with stroke] • “I think they really covered all the bases to accommodate, learn everything that you needed to learn. They did a really good job knowing what was needed to help the parent and help the child interact with each other with the sessions.” [Parent of an 8-year-old girl with stroke] • “Website could be useful to anybody too not just a child who’s had a brain injury like we use it with our daughter as well.” [Parent of an 8-year-old girl with stroke] • “Content was easy to understand flowed well, loved the videos.” [Clinical Neuropsychologist] • “I like the multimodal approach to knowledge and skill building–especially enforced by the live coaching. Great that it's coaching and not just observing.” [Psychiatrist] • “I just watched the testimonials video, it’s so convincing. I kind of want to learn the techniques too.” [Neurologist]
	Usability	• “it’s a lot more interactive, it’s a lot more family friendly.” [Parent of a 6-year-old girl with HIE] • “New one’s much nicer, easy to navigate and a lot better.” [Parent of a 9-year-old girl with stroke] • “…it's very straightforward, well placed, well detailed.” [Parent of a 8-year-old girl with stroke] • “I liked the different activities like the drag and drop kind of stuff.” [Parent of a 4-year-old boy with HIE] • “that you can use it on tablet [or] phone when you’re a busy family you’re not home often.” [Parent of a 8-year-old girl with stroke] • “I think it’s a very appealing website, I liked the videos, I like the pictures and the graphics.” [Parent of a 9-year-old girl with HIE] • “I think it was really easy to navigate and find the relevant information.” [Clinical Neuropsychologist] • “Although module 2 seem intimidating at first given the number of sections, I thought it was really well organized and easy to follow.” [Clinical Neuropsychologist] • “The site is easy to navigate. The use of pictures with the timeline of sessions is great.” [Clinical Psychologist]
Suggested Revisions	Recommendations	• “It’d only let me see some of my words, it didn’t let me see the whole sentence.” [Parent of a 4-year-old boy with HIE] • “What would have been handy or even like further reading or something.” [Parent of a 4-year-old boy with HIE] • “Should be a contact us part at the top where it says home about sessions.” [Parent of an 8-year-old boy with stroke] • “One-page printed cheat sheet type.” [Parent of an 8-year-old girl with CHD] • “How can we provide this information to newcomers so if there is any idea or any resources just to translate this information to different languages to become more accessible for newcomer population who don’t speak English at this point?” [Parent of a 9-year-old girl with HIE] • “Direct clinicians on how to refer, self-refer. How to get in touch.” [Clinical Neuropsychologist]
	Dissatisfaction	• “One part though that I thought was confusing when it was saying…talking about the positive opposites?” [Parent of a 4-year-old boy with HIE] • “The broken record. It didn’t seem like it helped.” [Parent of an 8-year-old boy with stroke] • “I found some of the videos like at the beginning there were like a million videos of special playtime.” [Parent of an 8-year-old boy with HIE] • “Some of the videos I found to be quite long.” [Parent of a 5-year-old boy with HIE] • “In module 5 you say you can do time-out anywhere, but you don’t cover doing it in public until module 6-I wonder if you could mention that they’ll learn it in module 6 since this is the hardest thing for parents to do.” [Developmental Psychologist]

Happy with the Learning Platform

Acceptability

In the first iteration, the majority of parents reported overall acceptability of the learning platform (version 1; V1). Two subthemes were identified within this theme.

Content was Easy to Understand/Helpful/Applicable Seventeen parents (89%) shared they specifically enjoyed the learning platform content and found it helpful and applicable. For example, one parent shared I think it really covered all the bases to accommodate, learn everything that you needed to learn. It did a really good job knowing what was needed to help the parent and help the child interact with each other with the sessions (parent of an 8-year-old with stroke).

Enjoyed Learning Platform Fourteen parents (74%) expressed they enjoyed the V1 learning platform. One parent of a 5-year-old with HIE expressed I liked the fact that there was parental involvement in the quotes as well as videos.

Usability

Learning Platform was Easy to Navigate/User Friendly Seventeen parents (89%) described the I-InTERACT-North learning platform (V1) as easy to navigate and user friendly. Parents noted how they enjoyed being able to access the learning platform on multiple devices (e.g., tablet, cellphone, laptop) and the ease of navigation: it was pretty well laid out as far as just looking at the headings, you knew what to expect (parent of a 5 year old with HIE).

Learning Platform was Esthetically Appealing Overall, twelve parents (63%) specifically noted they found the learning platform (V1) physically appealing in terms of the color scheme, presentation, animations, and photos throughout the learning platform, finding it overall “modern” and “inclusive.”

Suggested Revisions

Recommendations

Suggestion for Improvement Fifteen parent knowledge users (79%) described specific recommendations to improve the I-InTERACT-North learning platform (V1). Parents discussed adding a replay option to videos, one parent reported when the videos are done it would be really helpful to be able to play the video again if you wanted to (parent of a 4 year old with HIE). Other recommendations included adding printouts and further readings and making the contact page more accessible at the top of the home page. In addition, parents noted adding more detail to the Frequently Asked Questions (FAQ) and About pages such as clarifying that both parents are not required to participate in the intervention, describing who the intervention is intended for, editing wording to reduce repetition, and placing parent testimonials at the top of the page. Parents recommended improving the learning platform (V1) interface to reduce the number of clicks to exit the modules, adding words of encouragement throughout the learning platform (e.g., you are doing great, thank you for showing up), and adding a Resource page. One parent of a 9 year old with HIE suggested translating the learning platform: translate this information to different languages to make it more accessible for populations who don’t speak English.

Technical Difficulty Encountered Six parents (31%) noted encountering difficulties on the learning platform (V1). Specifically, two parents described difficulties with formatting on mobile devices (e.g., words being cut off), dead links, and being exited from the modules if they clicked outside the screen.

Dissatisfaction

Unhelpful, Confusing, or Disliked Content In the first iteration, nine parents (47%) described specific learning platform (V1) components as unhelpful, confusing, or displeasing. For example, three parents noted that they disliked having to enter their name and contact information at the end of each module, one parent of a 9 year old diagnosed with stroke described I don’t understand why at the end of every session I had to put in my email, my name, and my therapist’s name. I didn’t like that. In addition, two parents noted they disliked the use of video recommendations shown after a video was watched on the learning platform. Four parents noted they did not like the multiple links to the same page (e.g., Road Map and About links lead to the About page).

Redesign 1: Learning Platform Changes Made in Response to First Iteration As a response to knowledge user feedback, for the second version (V2), the study team removed the demographics from each module quiz (e.g., linking the account email address to responses), changed the video platform from Youtube to Vimeo to eliminate video recommendations, reduced the number of links and learning platform pathways. In addition, the study team ensured multi device compatibility, removal of dead links, and ensured a seamless user experience. Following knowledge user input, recommendations were added to the learning platform including adding a replay option for all videos, adding printouts to module tabs, creating a “Resource” page containing further resources, making the contact page more accessible to locate, adding more detail to the “About” and “FAQ” pages, and adding words of encouragement throughout the learning platform. The study team requested a quote on translation costs for the learning platform into additional languages. Estimated cost ranged between $29,000 for Inuktitut or Inuinnaqtun and $4,100 for French (M.F personal communication, June 2, 2020) and were not feasible at that time but is now included in all prospective grant applications.

Formal Evaluation: Iteration 2 (Interviews), Version 2

The adapted learning platform (V2) was then reviewed by 2 additional parents and 16 clinicians. Participants’ responses/discussion echoed the same themes from the V1 initial iteration [ (e.g., understand/helpful/applicable (2 parents, 13 clinicians), Enjoyed the learning platform (2 parents, 7 clinicians)]. Fifteen stakeholders (83%, 2 parents, 13 clinicians) stated they found the I-InTERACT-North learning platform (V2) content easy to understand, helpful, and applicable. Nine stakeholders (50%, 2 parents, 7 clinicians) voiced they overall enjoyed the learning platform and twelve stakeholders (67%, 1 parent, 11 clinicians) specifically described the adapted I-InTERACT-North learning platform as accessible and esthetically appealing.

There were novel additional recommendations and difficulties encountered. Five stakeholders (28%; 2 parents, 3 clinicians) stated they did not like some aspects of the adapted learning platform (V2) and found it unhelpful or confusing. Notably, one parent voiced; it would be good to say what [neuroplasticity] is, to define it. It’s a pretty medical term, I’m not in medicine so I wouldn’t know what that is (parent of a 4-year-old with HIE). Four clinicians found the inclusion criteria (e.g., child diagnosis, age, research component) unclear on the “About” page, and three clinicians noted the learning platform was incompatible with one web browser (Safari). In addition, thirteen stakeholders (72%; 1 parent, 12 clinicians) recommended adding more information about whom the program was appropriate for and the study design (e.g., referral information including age and diagnoses of child, about the team, research intervention), adding more diagnosis specific resources to the “Resources” page (e.g., CHD and behavior), and using more lay person language throughout the learning platform.

Redesign 2: Learning Platform Changes Made in Response to Second Iteration (Final Version) For the final version, the research team incorporated stakeholder feedback by including more lay language throughout the adapted I-InTERACT-North learning platform (e.g., changing neuroplasticity to resilience on the homepage) and clarity regarding the inclusion criteria and program on the “About” page. Note, clinician stakeholders recommended placing parent testimonials on the “About” page at the bottom whereas parent stakeholders recommended placing testimonials front and center. Given that the adapted learning platform is intended for families, the study team elected to keep the testimonials at the top of the page. The study team was unable to fix the incompatibility issue with Safari due to the learning software used in the build. Additional instructions regarding recommended web browsers are now provided to incoming families to the program [removed for blind review].

Discussion

In this study, knowledge users and stakeholders assessed a newly adapted virtual family-based mental health intervention, I-InTERACT-North with the goal of informing the adaptation of the web-based learning platform. Overall parent and clinicians reported they enjoyed the content, and found the language easy to understand, the design attractive, and the information helpful. Stakeholder and knowledge user interviews highlighted two overarching meta-themes—happy with the learning platform and suggested revisions. Listening to their feedback identified numerous refinements to learning platform components such as accessibility and applicability that ultimately improved the final adapted I-InTERACT-North learning platform.

Parents and clinicians were happy with the adapted I-InTERACT-North learning platform, finding it acceptable, specifically noting that the content was easy to understand, helpful, and usable (e.g., easy to navigate, esthetically appealing). Clinician stakeholders and parent knowledge users enjoyed being able to access the adapted learning platform on multiple devices and the modern esthetics, addressing two key recommendations from the original learning platform (Wade, 2017). Feedback was also provided regarding specific sections of the adapted learning platform they did not find helpful. Recommendations were provided to improve the site including adding more detail to the “About” page such as the intended audience (e.g., parents of children between 3 and 9 years of age with mental health concerns regarding child behavior and parenting stress), adding more additional resources, and words of encouragement throughout. Interestingly, parent knowledge users preferred testimonials be front and center throughout the learning platform whereas clinician stakeholders recommended placing testimonials at the bottom of each page. As the iterations progressed, stakeholders reported less website difficulties and recommendations. Interestingly, in the second iteration with primarily clinician stakeholders, the majority of recommendations were around referral information (e.g., clarifying inclusion and referral information on the About page) which may not be helpful for parents since they will be already referred to the program.

This study utilized a participatory action research design to include a purposive sample of parents and clinicians to ensure that essential perspectives were captured from those most likely to engage with the adapted I-InTERACT-North learning platform. Clinician stakeholders ranged in occupation, job setting, and number of years of experience, all providing a unique sample representative of pediatric mental health. We sought out clinician stakeholders and parent knowledge users who were familiar with the intervention (e.g., content, coaching, participants in initial feasibility trial), and those who were not. The only difference between the two groups were their familiarity with the I-InTERACT-North intervention. Having parents and clinicians engaged throughout the learning platform development process identified important obstacles (language, format) that may have been missed by adapting the program based on content alone. In theory, actively engaging parents and clinicians using PAR methods will lead to greater long-term acceptability and willingness of future families to engage in the I-InTERACT-North intervention. Moreover, providing the opportunity for clinicians to review the adapted I-InTERACT-North learning platform and contribute feedback in its development allowed for a deeper familiarization of the program that could potentially aid in future referrals.

There are several important limitations and directions for future research. First, the homogeneity of the sample may limit the generalizability of study findings. Stakeholders were predominantly women (e.g., mothers), spoke English, better educated, older, and given demands of the study had to have access to the internet and a device (e.g., computer, tablet, phone). Further, data regarding participant race or cultural background were not collected, and we acknowledge the high proportion of college and university educated parents, which limits generalizability. This is of particular relevance given the current racial context and the increased mental health disparities for racialized communities. However, the study team partnered with [removed for blind review] given their experience and expertise including underrepresented groups and collaborated with families to ensure they felt represented in the learning platform. We have also started to collect race and cultural demographic data in other studies using a diversity questionnaire based on the Vancouver Index of Acculturation (VIA; Ryder et al., 2000). The findings may also reflect parent and clinician stakeholder’s approval bias to the known scarcity of mental health resources for families in need. The majority of parent knowledge users participated in the initial feasibility trial (Burek et al., 2020); thus, participants previous success and overall satisfaction of the intervention may have influenced their view of the adapted I-InTERACT-North learning platform. This is a well-known challenge of PAR, i.e., having prior knowledge of the content and intervention, and reflecting on the applicability of the intervention on their own families. However, the semi-structured interviews did not direct interviewees to give personal responses, knowledge users and stakeholders shared positive, negative, and neutral experiences with the adapted I-InTERACT-North learning platform and interviews were led by a team member (MD) who did not have prior therapeutic relationship with the participants. The learning platform was also designed in English from a North American, single-payer health-care perspective, and may not translate to service delivery models in other countries and languages. Translating the learning platform and adding additional cultural aspects is an essential direction to increase representation of non-English speaking parental needs. Our group is currently trialing an interpreter approach for non-English speaking families and hopes to dedicate more resources to increasing inclusivity on the adapted I-InTERACT-North learning platform and program. Finally, due to the unintended timing due to the COVID-19 pandemic, many parent knowledge users were unavailable to participate in the second iteration due to competing time demands associated with work from home and monitoring their child’s remote learning. Similarly, many clinician stakeholders also declined due to similar time restraints.

With the growing emphasis on meaningfully engaging stakeholders in research, this study reflects the utility and importance of PAR methods in informing virtual family-based mental health intervention and provides a valuable roadmap for researchers interested in collaborating with stakeholders and knowledge users throughout the research process. Virtual mental health care such as that offered in the I-InTERACT-North program, is also ever-evolving at increasing speed; thus, continued active involvement of relevant stakeholders and knowledge users is needed to keep associated learning platform up to date and applicable. Future studies will explore the efficiency and inclusivity (e.g., less glitches, accessibility, translations) of the adapted intervention learning platform on participant success and parent and child outcome factors (e.g., child behavior, parent level of stress). Overall, partnerships between parents and study teams provide vital feedback to identify scientist blind spots, challenge traditional service delivery models and optimize family-informed mental health care.