Developing an evaluation framework for assessing the impact of recovery colleges: protocol for a participatory stakeholder engagement process and cocreated scoping review.

INTRODUCTION: Recovery colleges (RCs) are mental health centres aimed at equipping people with skills to live a meaningful life despite the presence of mental distress. Unique to them is the aspect of cocreation; RCs are designed collaboratively with people of lived experiences of mental health and addictions and care providers. Despite established benefits, there remains a lack of empirical evidence on how RCs work and on their impact. AIMS: We aim to address this gap by designing a cocreated evaluation framework for RCs. This will be accomplished by engaging RC student/facilitators to provide perspectives on RCs/RC evaluation and cocreate a scoping review identifying evaluation gaps in the literature. Themes identified through these processes will form the evaluation framework. METHODS AND ANALYSIS: Two methodologies will be used to explore RC evaluation: student/facilitator engagement and a scoping review of current published and grey literature on RC evaluation. Engagement will be achieved using a participatory action research approach consisting of informant interviews of ~25 RC students/facilitators across Canada, which will be thematically analysed. The scoping review will follow methodology described by Arksey and O'Malley modified to support cocreation. Concurrent conducting of the engagement process and scoping review will allow RC students and peer facilitators the opportunity to shape RC evaluations, address gaps in the literature and codesign an evaluation framework focused on recovery-oriented processes and outcomes mattering most to RCs students/facilitators. ETHICS AND DISSEMINATION: Ethics approval was received for the RC student/facilitator engagement component from the Centre for Addictions and Mental Health Research Ethics Board (#042-2020) and Ontario Shores Centre for Mental Health Sciences (#20-013-B). Scoping review results will be copresented through national and international medical education conferences and published in open-access peer-reviewed journals. Furthermore, a dissemination strategy on evaluation for the national RC community will be created. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The findings of this review will be the first to provide a cocreated framework for how recovery colleges (RCs) should be evaluated with a focus on recovery-oriented processes and key outcomes important to RC students and facilitators.

Uniquely, this study consists of two simultaneous methodologies: engagement of RC students/facilitators and a cocreated scoping review of the evaluation literature on RCs.

Ultimately, the results of this project will contribute to the creation of an evaluation tool, which is meaningful to all stakeholders and can be transferred and tailored to RCs globally that are interested in the process and outcomes of cocreating with students and researchers.

While the wide inclusion criteria for student/facilitators will likely allow for rapid data collection, the results of the informant interviews may still be limited in that our sample of RC participants may not necessarily be representative and, furthermore, will be limited to individuals involved with Canadian RCs.

Our focus on English-speaking student/facilitators for the qualitative interviews may also limit the usefulness of our findings for non-English speaking settings.

Introduction

Personal recovery in mental health and addictions (MHAs) is defined as living a purposeful and meaningful life despite the presence of mental distress.1 While there are examples of recovery-oriented practices in Europe as early as 200 years ago, their appearance in North America at the government policy level did not occur until the late 1980s.2 This introduction was due to persistent consumer/survivor advocacy3 and supporting epidemiological evidence for substantial symptom reduction in mental illnesses previously thought to be ‘incurable’.4 The key conceptual shifts introduced by recovery are: (1) the focus of care is the patient not the symptoms and (2) individual empowerment and quality of life are as important as symptom reduction. A key goal of recovery is to improve social inclusion by increasing participation in important social and economic roles. However, despite nearly three decades since government recovery policies were introduced, people with MHA challenges still confront social inequities, including high rates of underemployment and low rates of educational achievement.5

In response, recovery colleges (RCs) were developed and implemented in 2009 in the UK. Drawing on educational theories such as transformative and constructivist learning,6 RCs offer courses providing information and teaching students skills for managing their mental health, well-being and navigating their daily lives. Current and former patients with lived experiences of mental health challenges become students once enrolled in the college/courses. RC courses can include budgeting, cooking, advocacy or coping with current events. All levels of RC development—programme planning, course design and delivery, and quality assurance—are cocreated by people with lived MHA experiences and/or other forms of relevant expertise (eg, providing MHA care). An essential contribution of RCs is the shift from ‘patients’ to ‘students’ and consequently from a therapeutic to an educational framework.

Only 10 years later, RCs have been established in countries around the world including Australia, Canada, Hong Kong, Japan, Ireland and the USA.7 There is promising evidence that recovery approaches, such as RCs, are more cost-effective than traditional mental health services, reduce both symptoms and hospitalisations and increase positive recovery outcomes.7 However, researchers note the need for both rigorous and meaningful evaluations of RCs and a recent systematised search found almost no robust evaluative research on how RCs work and what outcomes they produce.8

Under these conditions, the typical research response is to establish an evaluation framework, design corresponding methodologies and tools, collect and analyse data and use the results for programme development and improvement. However, the challenge for an initiative like an RC is how to cocreate an evaluation that is both driven by RC student perspectives and values and is scientifically sound.

In education, formal evaluations do not typically include student input. However, the RC philosophy requires more inclusive strategies to cocreate, which involves working with students throughout the programme development process. Thus, traditional ways of evaluation are insufficient from a recovery perspective since they position the RC participants’ contributions as ‘add-on’ (or worse, as token). Furthermore, recent literature suggests that stakeholder consultation provides tremendous value throughout the entirety of the research process and contributes to better knowledge translation beyond an academic setting.9–12

To address this, we will develop an active RC student/facilitator engagement process and conduct a cocreated scoping review of the scientific and grey literature to understand the extent and range of RC evaluation studies and identify any knowledge gaps. Our decision to include peer facilitators along with RC students is based on the fact that these individuals are not healthcare professionals, are often former RC students themselves and thus provide an important perspective on RCs.

We define cocreation as moving beyond consultation to breaking down the barriers that exist when participating in research.13 Consequently, every stage of the project will be designed and developed using mutually agreed-upon processes that encourage participation, leadership and decision making.

The intended results of our engagement process will include the qualitatively collected perspectives of RC students and peer facilitators on RCs and RC evaluation, as well as involvement of interested individuals to advise on how to shape and interpret the scoping review findings. The intended results of the cocreated scoping review will be to describe how RC evaluation activities are reflected in the scholarly and grey literature. The synthesis of the student/facilitator perspectives and scoping review findings through a cocreative process among the review team and the student/facilitator advisory is expected to identify both strengths and gaps in RC evaluation, which will then support discussion and serve as the basis for an evaluation framework.

Methods and analysis

Data and information collection for this study began at the end of 2020 after receiving institutional and ethics approval. While originally funded for 1 year, the funder provided a 1-year extension due to the COVID-19 pandemic. As of the writing of this protocol article, 20 qualitative interviews with students and facilitators are complete. In addition, the extraction of the peer-reviewed and grey literature for the scoping review is complete, and the synthesis and write-up processes are being codeveloped. Our estimated project end date is mid-2022.

Student/stakeholder engagement

Meaningful engagement in research involves restructuring power relations/imbalances and the verbal and non-verbal ways in which power is shared.14 For this study, the participatory action research (PAR) method will be used as a holistic way to engage RC students and peer facilitators as research partners. PAR uses an organic, iterative process to combine reflection, data collection and action to improve health and reduce health inequities by involving the individuals affected. PAR has three core elements: active participation of researchers and RC student/facilitators in the coconstruction of knowledge; promotion of self-awareness and critical awareness leading to individual, collective and/or social changes; and building alliances between researchers and students in all stages of research.15 Having RC student/facilitators and researchers collaborate throughout the entirety of the research process communicates the importance of moving beyond tokenistic engagement of patients and towards a collaborative, power-sharing model.

To understand the outcomes important to RC student/facilitators, in-depth interviews will be conducted with a convenience sample of approximately 25 individuals from RCs across Canada.16 The target of roughly 25 individuals was chosen guided by the concept of information power that focuses on factors such as the broadness or narrowness of the study aims, the kind of sample and the analysis strategies.17 Given the exploratory and developmental nature of this study, the continuous cocreation process and the in-depth qualitative analysis of key informant narratives, a target size of 25 should be sufficient to capture the experiences of RC students and facilitators and be more than adequate to reach saturation.18

Any student who has taken at least one course at a RC or any RC peer facilitator who is English speaking will be eligible to participate in the qualitative interview process. RC student/facilitators will be invited to participate in a one-on-one interview which will last a maximum of 1 hour. Interviews will be conducted by research analysts unaffiliated with the participant’s RC. Open-ended questions will be asked following an interview guide that covers the student’s experiences as a RC student and recommendations for how RCs should be evaluated. Examples of questions include: ‘How would you describe your experience as a student?’ or ‘In your opinion, how do you think this Recovery College could be evaluated?’ (see online supplemental file for the interview guide). Interviews will be conducted via telephone or using the Webex videoconferencing platform. They will be audio recorded to avoid misinterpretation and to ensure all information is captured. Original audio files will only be accessible to the principal investigator, research analysts and transcriptionist. Other research personnel will have access to transcribed data that will be deidentified and given a numerical code. All RC students and facilitators will be compensated for their time. At the end of each interview, we will ask participants if they would be interested in reviewing our findings to ensure their interpretive validity.19

Scoping review

Under conditions where a field is new or rapidly changing and therefore has little primary research, a common method for gaining a comprehensive sense of existing evidence and identifying knowledge gaps is to conduct a scoping literature review.20 Typically, scoping review stages involve determination of the research question(s), consultation with a librarian to develop and execute a comprehensive search strategy, screening and abstracting identified articles, discussing and synthesising the results and producing a scientific manuscript. The proposed scoping review will be conducted in six stages that have been adapted from Arksey and O’Malley as described further.20 The Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for scoping reviews will also be used to ensure the reporting and synthesising of the evidence meets the current standards for reporting on knowledge synthesis research.21

Stage 1: consultation/cocreation

In contrast to the methodology designed by Arksey and O’Malley in which consultation is the last step, it will be the first step in this review process. Stakeholder consultation will be reframed as ‘RC student/facilitator engagement’ and will be the first and non-optional step towards integrating RC student/facilitator participation throughout the review process. This will be done by ensuring that information and updates are PAR agenda items and through recruitment of RC student/facilitators for the scoping review team. RC student/facilitator inclusion will be prioritised in the scoping review, both in terms of integrating their perspectives on the research questions, as well as supporting their active participation. The entire team will be introduced to and trained on scoping reviews with sessions open to non-team observers interested in learning about these processes. To promote self-reflection and critical awareness of cocreation throughout the entirety of the scoping review, close attention will be paid to all aspects of the process by which research is conducted. For example, every team meeting will start with a ‘check-in’ where team members are encouraged to share how they are and what has been happening in their lives since the previous meeting. This style of communication establishes that all members are valuable contributors who also have lives outside of the project, equalises power relations across the different roles and types of expertise of the team members and furthermore provides unique perspectives to the team. Sharing personal daily experiences and individual challenges creates a culture of trust essential to fostering reciprocity and mutual respect.22 Discussion topics, both within the team and as part of the larger PAR process, will include whether the scoping review questions and search terms are adequate; whether the data abstraction plans and templates need revision; and how to analyse, interpret and communicate the results.

Stage 2: planning and research question

The primary question is: how have RCs been evaluated in the published and grey literature? Related questions include: what kinds of theoretical frameworks are used, what outcomes are measured, how robust is the evidence provided, what elements are examined (eg, best practices and fidelity measures) and, most importantly, how patients are or are not involved. Consultation with national and international RC networks will be made to ensure outcome studies are not missed. Covidence, a systematic review management software, will be used to import citations, screen titles and abstracts, upload references and for data abstraction (https://www.covidence.org/).

Stage 3: search strategy

Search terms will be developed in consultation with the research team, RC student/facilitators and a research librarian. The librarian (TR) will design a comprehensive search strategy, then will translate and execute the strategy for databases such as Medline, APA PsycInfo and the International Bibliography of the Social Sciences. A ‘wide angle’ approach will be used to ensure broad coverage and include all relevant disciplines and all articles regardless of study design, year or article type.11 To increase the validity, comprehensiveness and reproducibility of our results, a second librarian will review the search strategy using the Peer Review of Electronic Search Strategies checklist.20 23 In addition, article bibliographies will be examined to identify other relevant publications, and the grey literature will be searched to avoid publication bias and improve the scoping review’s breadth.

Stage 4: screening and article selection

The review team will discuss and determine the eligibility criteria and, if needed, modify these criteria. Then, a 5% random sample of the results from the search strategy (stage 3) will be used to calibrate and establish the reliability of our screening process. The review team will meet to compare their assessments and to modify the eligibility criteria as needed. There will be two levels of screening: a title and abstract review to identify articles meeting criteria and a full-text review of those passing the first screen. For both levels, two independent reviewers (plus interested RC students) will perform the screen, meet and compare results, and, where needed, discuss with a third reviewer to resolve any differences.

Stage 5: data abstraction

The review team will first cocreate and then test draft data abstraction forms on a random sample of 10 articles that have passed the screening stage. Information gathered will include study characteristics (eg, authorship, publication year, methods and overall findings). As with the screening, extraction will be performed by two independent reviewers (plus interested RC student/facilitators) and involve a third reviewer as needed to resolve any discrepancies. Again, as with the screening process, these discussions can lead to modifications of the abstraction forms if the team feels they are necessary. As recommended by Arksey and O’Malley, formal appraisal of study methodologies will not be conducted as the aim of a scoping review is to identify and organise existing literature.

Stage 6: synthesis

The literature will be charted by the research questions to identify key dimensions of RC evaluations, for example, the number of studies on specific elements, the teaching methods used and the degree of RC student involvement. This approach will allow identification of key evaluation gaps in the literature, which will drive the development of a research agenda and RC evaluation framework.

Both quantitative and qualitative data analyses will be used to map the literature and support a descriptive numerical summary and a thematic analysis. The descriptive numerical summary will include the total number of studies, the types of study design, publication year, characteristics of the study population and countries where the studies were conducted. The thematic analysis will resemble a qualitative content analysis to capture common themes and patterns in the data. We will use the thematic analysis approach of Braun and Clarke to review the transcribed interviews, generate codes and develop descriptive themes.24 Data analysis will start by having the audio recordings transcribed verbatim by a professional transcriber and relocated into Dedoose software. The entire team will read a selected transcript individually and then jointly discuss initial thoughts and reflections. Through this process, using subsequent transcripts, we will, as a team, code line by line and undergo several in-depth reviews that will be co-led by HH and SoS. This will result in the codevelopment of numerous open codes. Subsequently, these codes will be collapsed and grouped forming descriptive categories initially by two members of the team (HH and SoS) and presented to the team for discussion. Finally, the categories will be refined and converged to create central themes. The themes will be provided to the entire team for further feedback and refinement. The research team will engage in consistent and iterative dialogue throughout the entire coding process to ensure that themes are not generated from a few vivid examples but instead that the process and resulting themes are thorough, inclusive, comprehensive and reflective of the entire data set.

Patient and public involvement

Our team consists of service users, students, RC facilitators and researchers who are involved at every stage of the research process. This project is funded through a strategy for patient-oriented research initiatives which means that the research design, development and outcome measures are being developed using their priorities, experiences and preferences. All service users and students are being paid through the grant funding to reduce the burden required to participate in the research process.

Ethics and dissemination

Research ethics approval is not required for the scoping review portion of this study; however, ethics approval was received from the institutions participating in the student/facilitator engagement phase (REB #042–2020 and #20–013-B). In terms of informed consent for the engagement component, each student or peer facilitator will be emailed the consent form to view at their leisure. At the agreed-upon interview time, the research analyst will use a verbal consent script that explains the consent form and answer any questions before conducting the interview. All interviews will be conducted remotely using the Cisco Webex videoconferencing platform. The research analyst will ask the participant if they give their consent to participate in the recorded interviewing using audio-only or audio-video depending on the person’s preference. In addition, each participant will have been provided, both ahead of time and just prior to the interview, information on the risks of using videoconferencing platforms (ie, Webex) as per our Institution’s guide for virtual participant sessions.

Discussion and implications

The purpose of this newly funded project is to lay the groundwork for a larger operating grant. Strengths of this work include the fact that this groundwork will consist of a cocreated framework for evaluating RCs built using two simultaneous methodologies: the engagement of RC student/facilitators and a cocreated scoping review. This work will serve as a starting point for further theoretical refinement, if needed, and will guide development of supporting evaluation tools and operational processes. To that end, the student/facilitator engagement component will produce an advisory group to that will help shape the scoping review and will also be invited to oversee the subsequent larger grant. The scoping review will identify and summarise scholarly and non-scholarly manuscripts but will also serve as a vehicle for building a shared approach to research and evaluation and as an evidence source to support the larger future grant.

Our approach is not without limitations, and these will need to be considered in the interpretation and application of any findings. For instance, our convenience sampling for the qualitative interviews may not include the full range of RC students or facilitators and will include only Canadian participants. Furthermore, our interviews will only be conducted in English. Both of these could limit the usefulness of our results, particularly for RCs in non-Canadian or non-English-speaking settings.

Nevertheless, the current literature and team member experiences from previous projects indicate that the combination of student/facilitator engagement and a cocreated scoping review could make several unique contributions to the literature and, ultimately, to patient-centred care/recovery in MHA.25 With regard to research contributions, these could include: modification of the original research question, identification of new concepts and therefore key search terms not currently addressed by the literature, expansion or modification of screening and abstracting criteria, adding greater relevance and real-world applicability to interpreting the findings and recommending new dissemination methods for broader audiences. Tracking and documenting these contributions could serve to identify gaps in the scientific literature. Collectively, participating in engagement and cocreation should also yield benefits in terms of learning how to engage and work jointly with RC students and peer facilitators. Developing respectful communication and collaborative skills, and thus building a shared understanding, are critical for supporting the inclusion and synthesis of more heterogeneous perspectives and values.13 Cocreating an evaluative framework for RCs with student/facilitators provides a model for innovating and challenging traditional ways of creating evaluations that will support accurate and meaningful understandings of the mechanisms of action and the outcomes of strengths-based, recovery education as offered by RCs.