Evaluation of parent and youth experiences in advisory groups as part of a mental healthcare clinical trial: protocol for a mixed-method study.

INTRODUCTION: Patient engagement in healthcare research is a necessity to ensure that research objectives align with priorities, outcomes and needs of the population under study, and to facilitate ease of implementation and adoption of findings. In clinical trials, there is an increasing focus on patient engagement during the planning and conduct of clinical trials due to the potential for ethical and methodological benefits. As patient engagement in clinical trials increases, there is a need to evaluate the approaches of these activities to contribute evidence on what is most appropriate and successful. The purpose of this study is to evaluate patient engagement processes and the activities of patient partners during and after a paediatric mental healthcare trial. METHODS AND ANALYSIS: Using a mixed-methods study design, we will evaluate patient partners' engagement activities across set time-points during the trial and after trial completion. In this study, the term 'patient partner' is inclusive of two groups of people with lived experience: (1) caregivers (parents, formal/informal caregivers and family), and (2) youth (aged 15-24 years). Engagement will be evaluated using the participant and project questionnaires of the Public and Patient Engagement Evaluation Tool (PPEET), followed sequentially by semi-structured interviews. Quantitative data from the PPEET questionnaire will be analysed and reported using descriptive statistics. Data from open-ended questions from the PPEET questionnaires and semi-structured interviews will be analysed using thematic analysis. ETHICS AND DISSEMINATION: Approval from Athabasca University Research Ethics Board will be obtained for this project. Findings will be disseminated at both academic and public venues whether in-person or online, and using platforms that are caregiver and youth friendly. TRIAL REGISTRATION NUMBER: NCT04902391. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Validated evaluation tool used to measure outcomes of patient engagement across a paediatric mental healthcare trial.

Semi-structured interviews to elicit patient engagement experience.

Dedicated resources for patient engagement from concept to end-of-grant knowledge translation activities.

Limitation includes the ability to sustain parent and youth engagement over the course of the trial.

Introduction

Within the last decade, research funding organisations in the UK, the USA, Canada, Australia and New Zealand have required that patient engagement be an integral part of the design, conduct, dissemination and implementation of findings emerging from social and healthcare research.1–5 Patient engagement occurs when patients become partners through meaningful and active collaboration across the research process.2 6 A recent systematic review has reported ethical, methodological and study quality benefits from engaging patients as partners in health research.6 These benefits have also been demonstrated through youth engagement in research, in particular, in the areas of health promotion,7 mental health,8 9 chronic pain10 and community development.11 Steps taken to recognise youths’ expertise within engagement practices in research are equally beneficial and include: (i) ensuring a youth friendly approach (eg, listening to what youth have to say and asking questions in a non-judgemental manner); (ii) recognising diversity among youth (eg, recruiting youth from different developmental stages and with varied sociodemographic characteristics); (iii) formalising recognition of contributions that are authentic for youth (eg, providing compensation for engagement, writing references for school/university/jobs, offering coauthorship on findings or providing certificates of achievement) and (iv) establishing youth-friendly environments (eg, creating respectful and welcoming spaces for ongoing communication among youth, adult researchers and adult patient partners whether virtual or in-person and ensuring support for youth who may find research topics to be potentially triggering).12 When youth have been engaged in research with these practical recommendations in place, the experiences are youth-friendly and the results are more likely to be implemented and adopted widely.12–14

There is the emerging evidence around the benefits to engage patients as partners in clinical trials aimed to evaluate non-pharmacological or pharmacological interventions.15–18 These benefits include: (i) more appropriate and sensitive/ethical research designs; (ii) more appropriate wording and timing of administration of research instruments and interventions; (iii) improved readability and accessibility of research materials (ethical benefits); (iv) more patient and/or caregiver relevant research outcomes or end points (methodological benefits); (v) improved recruitment, retention, diversity and trial experience/satisfaction of study participants; (vi) better adherence to the trial protocol and (vii) faster study completion (study quality benefits).18 Given this evidence, we saw the benefits of prioritising engagement with patients as partners with lived experience comprised youth and caregivers (discussed in more detail below in the section: Patient Engagement Plan) across the lifespan of our paediatric mental healthcare trial.

Overview: mental healthcare paediatric clinical trial

Trial development began in 2018, when members of our research team alongside a group of patients, families and healthcare providers identified the need to make improvements to paediatric mental healthcare in emergency departments (EDs).19 This group recognised EDs as a vital clinical care setting for children and youth experiencing acute mental health or substance misuse crises.20 Such crises were acknowledged to be stressful and overwhelming, leaving the youth and family vulnerable in the absence of appropriate care. The team reinforced the position that ED-based care should be: (1) family centred and multidisciplinary, (2) informed by evidence-based approaches that identify risk, inform ED care and disposition decision-making and (3) consider family needs and preferences.20 21 In response to this position, a novel ‘bundle’ of mental healthcare was created with feedback from patient partners (comprised separate groups of youth and parent/caregivers). The bundle components were discussed and prioritised through several in-person meetings and teleconferences that took place between 2017 and 2019.22 Patient partners also prioritised study outcomes and selected outcome measures that were most relevant for self-reporting in mental health, satisfaction with care and family functioning. The results of this engagement were applied to both the trial as well as quasi-experimental study that provided pilot data to inform the trial.22 The bundle combines three evidence-based approaches to ED care, as follows:

Ask Suicide-Screening Questions to identify suicide risk at ED triage and to facilitate faster access to appropriate treatment pathways.23–25

A clinical mnemonic (Home, Education, Activities/peers, Drug/alcohol, Suicidality, Emotions and behaviour, Discharge Resources), for a focused mental health psychosocial evaluation to guide ED care and disposition decision-making.21 26

A ‘choice’ appointment post-ED care, in accordance with the Choice And Partnership Approach to care. This is a family centred approach to mental health service organisation,27 and its use recognises the value of connecting children and youth to follow-up services after a crisis.

The bundle will be evaluated using a type 1 comparative effectiveness-implementation hybrid design.28 A cluster randomised trial will evaluate the effectiveness of the bundle compared with current standards of ED mental healthcare, while bundle implementation will be evaluated using multimethods to better understand barriers and facilitators to the adoption in ED care. The trial will take place in eight paediatric EDs across Canada in partnership with Pediatric Emergency Research Canada (PERC), a national research network. The Conjoint Health Research Ethics Board (CHREB), University of Calgary approved this study effective 4 January 2021 (reference ID: REB20-1825). This trial was registered at clinicaltrials.gov (Identifier: NCT04902391) with an estimated start date of 1 November 2021 and completion date of 1 August 2025. The protocol for this trial was submitted elsewhere for publication (see online supplemental appendix A for trial timeline).

A patient engagement plan was created for the trial that was informed by principles of patient engagement from many sources, including those specific to youth and caregivers.2 12 29–32 The plan will be evaluated in a trial substudy, which is described in ‘Patient engagement evaluation study’ section. The purpose of the patient engagement evaluation study is twofold1: (1) evaluate the engagement of youth and caregivers in the trial design, conduct, outcomes (and knowledge translation of outcomes) of the trial and (2) obtain research team members’ perspectives on the impact of patient engagement on all stages of the trial.

Patient and public involvement

The study protocol was conceptualised by JR (lead for patient engagement evaluation study) with support from AN (co-principal investigator of the primary mental healthcare paediatric trial). LM (caregiver lead) and MP (youth lead) with lived experience and extensive experience in patient engagement reviewed and provided feedback on this study protocol. According to the patient engagement plan described below, all caregiver and youth advisory members will be invited to provide feedback on both process and findings at different time-points across the primary mental healthcare paediatric trial. Feedback by advisory group members on both trial process and findings will be shared at the executive research team meetings for discussion and deliberation. Postexecutive research team meetings, a brief written report about how suggestions have been incorporated will be circulated to advisory group members.

For the patient engagement evaluation study described below, caregiver and youth advisory group members will be invited to partake in the interpretation of the aggregate results, editing and preparation of the manuscript for publication. Caregiver and youth advisory group members will be invited to contribute to the overall knowledge integration and dissemination plan. Both patient partner contributors (LM and MP) met the ICMJE criteria for authorship and as such we acknowledge their valuable contributions through coauthorship of this study protocol. Engagement of patient partners has ensured further accuracy, readability and relevance of this protocol to the science and practice of patient engagement in paediatric clinical trials.

Patient engagement plan

Overview

The patient engagement plan has two foci: (i) consultation in trial planning, whereby youth and caregivers had the opportunity as advisors to provide feedback on the trial design and (ii) partnership, whereby youth and caregivers will be partners alongside research team members during trial conduct, with both groups actively involved in collaborating and leading trial-related activities. These foci align with the practices recommended by the Canadian Institute of Health Research when developing partnerships between patients and researchers.33 We will use the term ‘patient advisor’ to denote those who are involved in a one-time engagement activity versus ‘patient partner’ who will provide feedback during patient engagement activities throughout the lifespan of the trial.

Participants and recruitment

We used a multipronged approach to recruit patient partners representing two groups of people with lived experience: (1) caregivers (including parents from diverse family forms), formal/informal caregivers and family and (2) youth (aged 15–24 years). We will specify the group we refer to within the protocol for particular instances as needed, but otherwise we mean both groups. Initial recruitment began in 2017 with youth and caregivers from two family advisory councils for child and adolescent mental health services in Calgary and Edmonton, Alberta, Canada invited to provide feedback and perspectives on bundle design, trial planning and grant submission. The participation of these advisors has concluded, with one partner staying as a team member for the trial, since funded.

The next phase of recruitment will focus on the participation of youth in a Youth Advisory Group and caregivers in a Caregiver Advisory Group; the groups will be active during trial conduct and post-trial knowledge translation activities. During youth and caregiver recruitment, we will prioritise equity, diversity and inclusion-based practices in patient engagement,34 and plan to recruit individuals from different demographic groups, based on age, gender identity, cultural identity, disability or education. We will set up two advisory groups to minimise power differentials among youth and caregivers and foster a safe space, whereby all members are comfortable sharing their perspectives.

Youth engagement liaisons/coordinators from the Centre for Addiction and Mental Health (https://www.camh.ca/) (led by MP) will assist with recruitment of youth to join the Youth Advisory Group. Caregiver recruitment will be overseen by JR. Convenience sampling will be used to recruit individuals through organisations who partnered with the team (Canadian Mental Health Association, Translating Emergency Knowledge for Kids (TREKK), and Children’s Healthcare Canada).

Youth and caregivers who indicate interest in participating will be invited to attend a virtual introductory meeting (one for youth; one for caregivers) with the trial co-leads (SF and AN) and patient engagement leads (JR, MP and LM). The meeting will provide an opportunity for individuals to learn about the trial and the goal of patient partner engagement. Those interested in joining the team as advisory group members will be asked to contact the trial’s patient engagement evaluation study lead (JR) and youth engagement lead (MP) to further discuss and confirm their interest. Informed consent will be obtained at that time. Patient partners can voluntarily revoke consent at any time during the course of the trial. Patient partners will be supported by the patient engagement leads throughout the course of the project, as described further under advisory group activities. The goal is to have membership in both advisory groups filled between November 2021 and January 2022; the trial begins February 2022.

Sample size

We could not find evidence in the literature to support best practice for optimal group size for patient engagement in research. Therefore, we will exercise pragmatic considerations and recommendations from team members and project liaisons with extensive experience working in patient engagement. In particular, the size of the group should foster meaningful engagement, so that we are able to have robust discussion, and patient partners can share ideas in a safe space. Conversely, we do not want a group too large that it is ineffective to engage partners in discussions. We anticipate a target size of 5–10 patient partners per advisory group will be sufficient to lead to a meaningful discussion.

Patient partners that do not continue for the full duration of planned patient engagement activities will be included in the sample if they are willing. Patient partners who attend the virtual introductory meeting with the study co-lead, but later decline to participate in an advisory group will be asked if they are willing to share confidentially the reasons that led to the decision to forego participation. If we have attrition of advisory group members at any stage of the engagement process, we want to have the opportunity to formally document their reasons that led to that decision as part of the evaluation of patient engagement.

Advisory group activities

Advisory groups will be co-chaired by MP, LM and JR. At the first meeting of each group, members will discuss group norms, establish terms of reference for the respective advisory groups, including role description and individual goals as they relate to the patient engagement activities. Meetings will occur at least four times per year for up to 3 years, which is aligned with key milestones within the trial (see online supplemental appendix B). Compensation will be provided at a set annual rate per year that was budgeted in the grant. Institutionally approved videoconferencing platforms will be used to exchange and share information about the study. Patient partners will have the ability to choose the types of engagement activities that they would like to participate in, and this will be determined on an individual and/or advisory group basis. These activities include co-leading the advisory group alongside the chairs, providing feedback on recruitment materials for the trial and telephone scripts used during data collection with trial participants, informing changes to recruitment and retention processes during trial conduct, providing impressions on the results and designing/preparing materials to disseminate trial results (eg, statements for social media use, infographics for families). Patient partners will also have the opportunity to apply for funding to attend conferences and courses related to children’s mental health (budgeted within the trial) and/or work with the patient engagement evaluation study lead (JR) to plan and conduct evaluation meetings.

Patient engagement evaluation study

Design

We are using a mixed-method design to evaluate the trial’s patient engagement plan.35 36 In particular, we chose the exploratory sequential mixed-method design because we will first complete data collection and analyses using quantitative methods followed by qualitative methods.37 This will involve surveying patient advisors and partners as well as research team members on engagement experiences followed by semi-structured interviews of patient partners to gather more detailed information on engagement experiences and activities highlighted in survey responses. In addition, we will place equal weighting on both quantitative and qualitative methods37 because our research purpose is to evaluate engagement and determine impact of that engagement to the paediatric mental healthcare trial. Thus, to do that effectively, we emphasise that the data collected and analysed using both methods will address the twofold purpose. We will use the Good Reporting of Mixed Method Study Criteria38 and the Guidance for Reporting Involvement of Patients and the Public39 to assess the quality of reporting of our mixed-method patient engagement evaluation study procedures and findings.

Framework

We are using a framework to guide the evaluation. This framework contains four domains—integrity of design and process; flexibility; mentorship; influence and impact—and was adapted from the published literature on patient and youth engagement.3240 Integrity of design and process will be assessed by examining advisory group representation (eg, Do members represent different experiences based on age, gender identity, cultural identity, disability or education?) and how group members are supported in patient engagement activities (eg, Are members compensated for their involved in activities? Is information produced at an appropriate education level for members?).32 Flexibility of advisory groups with caregivers and youth will be assessed by examining the terms of reference (role, structure, norms and frequency of interactions) of the groups and how those terms of reference are carried out across the research project.40 Mentorship will be assessed by asking patient partners about advisory group trial activities and opportunities that aligned with personal goals.40 Influence and impact will be assessed by examining whether patient engagement activities informed or changed trial decisions.32

Methods

Engagement Evaluation Tool

We will use the Public and Patient Engagement Evaluation Tool (PPEET) to evaluate patient engagement across our domains of interest: integrity of design and process, flexibility, mentorship and influence and impact. The PPEET comprises two questionnaires which will allow us to evaluate perspectives and experiences of advisory group members (participant questionnaire) and research team members (project questionnaire). The participant questionnaire also contains questions to evaluate one-time and ongoing patient engagement experiences, which will ensure that patient partners with different experiences contribute to the evaluation. At this time, we are aware of at least 27 tools to evaluate patient and public engagement.1 41 42 We chose the PPEET because it was used to evaluate patient and public engagement within the context of health research. The PPEET was highly rated by the Centre of Excellence on Partnership with Patients and the Public because of high scores in the domains of scientific rigour, patient and public perspective, comprehensiveness and usability.1 41 42

In table 1, we outline how our domains of interest will be assessed using the PPEET participant and project questionnaires. We also identify time-points of when each questionnaire will be administered. Advisory group members will be asked to fill out a six-item demographic questionnaire, so we can assess diversity of perspectives, as part of integrity of design and process. The questions include year of birth, sex, group of people or communities one identifies with, highest level of education completed, current work status and paid experience in a healthcare profession. Additionally, we will include a description about advisory members’ previous experience with patient engagement in research (information will be obtained when we recruit members).

Table 1

Overview of the approach to measuring patient engagement

Integrity of design and process; flexibility; mentorship
PPEET questionnaire	Questionnaire outcomes	Administration time-point
Participant questionnaire: one-time engagement activities (module A—parts A and B)	Advisory group members represent diverse range of views. Advisory group members are provided support that enable participation in engagement activities. Online platforms suitable for communication with patient partners. Clear and bidirectional communication achieved among patient partners and researchers. Mentorship opportunities created and tailored to patient partners’ interests.	At the end of the activity on the same day
Participant questionnaire: ongoing engagement activities (module B—parts A and B)		Between 3 and 6 months
Project questionnaire(module A—parts A, B and C)		Pre-trial conduct (pre-recruitment of trial participants)
Influence and impact
Participant questionnaire: one-time engagement activities (module A—parts C and D)	Engagement activities inform planning and/or decision making in the trial. Engagement activities improve patient partners’ knowledge of mental health and substance use crises in emergency departments. Engagement activities improve patient partners’ knowledge of the patient and family centred acute mental healthcare bundle and need for implementation. Increased confidence and trust of patient partners in members of the team. Increased confidence of patient partners in the study purpose. Increased confidence of patient partners in the impact of bundle implementation on mental health.	At the end of the activity on the same day
Participant questionnaire: ongoing engagement activities (module B—parts C and D)		Between 3 and 6 months
Project questionnaire(module B—parts A, B and C)	Clinicians, staff and researchers improve knowledge on patient partners’ perspectives about the bundle.	Post activity
Project questionnaire(module C—parts A, B and C)		3–6 months after engagement has concluded

PPEET, Public and Patient Engagement Evaluation Tool.

PPEET participant questionnaire

The one-time engagement questionnaire comprises 19 questions: 13 questions based on a five-point Likert scale (range: strongly disagree to strongly agree) and six open-ended questions. The questionnaire will be administered to each patient advisor immediately after the conclusion of an engagement activity. The ongoing engagement questionnaire will be administered to patient partners every 3–6 months during trial conduct. This questionnaire comprises a total of 20 questions: 14 questions based on a five-point Likert scale (range: strongly disagree to strongly agree) and six open-ended questions.

PPEET project questionnaire

We will use the PPEET three-module project questionnaire to assess involvement in patient engagement among the principal investigators, project manager, implementation site leads and youth engagement liaisons/coordinators. The first module (module A) consists of 12 questions on a five-point Likert scale (range: strongly disagree to strongly agree) and three open-ended questions to assess how and whether study team members have chosen appropriate patient engagement activities during the planning stage. This module will be administered prior to trial conduct and after the run-in period. The second module (module B) identifies how well patient engagement activities were executed (10 five-point Likert questions; six open-ended questions) and will be administered after the trial engagement activities are completed. The third module (module C) will evaluate the impact of patient engagement activities. This module includes four questions (five-point Likert scale; range: strongly disagree to strongly agree) and four open-ended questions, which will be administered 3–6 months after the completion of the engagement activities.

Semi-structured interviews

We will interview patient partners at the end of the trial after the completion of all engagement activities. Interview questions will be developed based on the themes that emerge from the open-ended responses collected at different time-point using the PPEET with the intent to gather more information on engagement experiences and activities.43 Patient partners will be offered the choice to participate in individual or focus group interviews that will be conducted by a research nurse/graduate student. Focus group interviews will be held separately for youth and caregivers. A final forum inviting all patient partners and research team members to attend will be held to discuss and obtain feedback on synthesised questionnaire results and interview findings.

Additional data sources: supporting documents and trial protocol documents

Supporting documents for the paediatric mental healthcare trial such as meeting agendas, minutes or tailored resources for advisory group members will be reviewed monthly and will serve as additional data sources to understand patient engagement activities. Trial protocol documents will be collected annually from the trial research coordinator to determine how feedback from advisory group members were used to inform/modify trial decisions.

Data collection and management

We will administer the PPEET online using the Research Electronic Data Capture (REDCap) tools44 hosted and supported by the Women and Children’s Health Research Institute at the University of Alberta. Data collected through this platform by a research nurse/graduate student will be de-identified and will not be tracked to any individual participant. In exceptional cases, such as limited participant access to online environments, the PPEET will be administered via telephone with answers entered by the interviewer into the REDCap database. Data will be stored in REDCap until exported for analysis.

Semi-structured interviews will be conducted via telephone/videoconferencing platform hosted by the University of Calgary (Zoom) and the audio will be recorded. Interview recordings will be transcribed verbatim, checked for accuracy against the original recordings and identifiers will be removed from the transcripts to maintain confidentiality. Data from the semi-structured interviews and the audio recordings will be stored securely in a server at the University of Alberta until the data are transcribed and subsequently analysed.

Field notes will be kept when additional data sources are reviewed. Supporting documents will be reviewed during regular advisory group meetings to keep informed of the trial progress. Trial protocol documents will be collected and reviewed annually from the trial research coordinator to note modifications that have been implemented across sites based on advisory group members’ feedback.

Data analysis

Data from the PPEET will be entered into SPSS V.27. We will use descriptive statistics (eg, means and SD, medians with IQR) to report group characteristics and responses to the Likert scale questions. Frequency and percentages will be reported for categorical information. Data from open-ended questions45 and semi-structured interviews will be imported into NVIVO V.12. We will undertake thematic analysis45 to analyse these data. Data from open‐ended questions will be grouped by question and then coded for common themes that arise across the data. The primary coder (research nurse or graduate student) and first author (JR) will meet at least three times during the development and application of the coding scheme as part of an iterative process to ensure coding reliability. Coding reliability by the study co-lead (AN) will be completed to resolve inconsistencies or disputes and to review independent coding of selected data excerpts.

Data from semi-structured interviews will be analysed using the coding scheme as a guide. Data that cannot be coded to existing codes or themes will be noted. In the later stages of analysis, the study co-lead (AN) will independently code data excerpts to maintain coding reliability. Advisory group members’ feedback on supporting documents and trial protocol documents will be reviewed and synthesised into field notes by a research nurse and research assistant, which will be reported as a narrative.

Ethics and dissemination

Ethical approval for this study was obtained through the Athabasca University Research Ethics Board (file no: 24575) for 1 year and is subject for renewal on an annual basis since the project is ongoing beyond 1 year. All participating advisory group members will receive an information sheet that will provide details on the purpose of the study, identify the potential risks/benefits and explain the voluntary nature of their participation. Patient advisors and partners can revoke consent from participating in the parent or youth advisory group and evaluation activities at any time. Patient advisors and partners may choose to omit particular questions while filling out the survey or during the interview (pertains only to patient partners). All data will be de-identified; therefore, individual participant data cannot be removed once collected. Data will be kept confidential. All data will be stored using secured software on a password-protected server and device.

Patient engagement evaluation findings will be shared with parent and youth advisory group members during virtual meetings. Advisory group members will be asked about how the results fit within the organising framework and whether the patient engagement outcomes were met. Dissemination of findings from this patient engagement evaluation study for the paediatric mental healthcare trial will be in the form of an academic publication in a reputable peer-reviewed journal, presentations at TREKK and PERC conferences/meetings, public presentations at appropriate venues and as posts or blogs on the research study website and online platforms that are parent and youth friendly. Research team members and advisory group members will be invited to coauthor, co-develop and co-present these findings targeted at multiple venues.