Early Psychosis Intervention-Spreading Evidence-based Treatment (EPI-SET): protocol for an effectiveness-implementation study of a structured model of care for psychosis in youth and emerging adults.

INTRODUCTION: While early psychosis intervention (EPI) has proliferated in recent years amid evidence of its effectiveness, programmes often struggle to deliver consistent, recovery-based care. NAVIGATE is a manualised model of EPI with demonstrated effectiveness consisting of four components: individualised medication management, individual resiliency training, supported employment and education and family education. We aim to implement NAVIGATE in geographically diverse EPI programmes in Ontario, Canada, evaluating implementation and its effect on fidelity to the EPI model, as well as individual-level outcomes (patient/family member-reported and interviewer-rated), system-level outcomes (captured in provincial administrative databases) and engagement of participants with lived experience. METHODS AND ANALYSIS: This is a multisite, non-randomised pragmatic hybrid effectiveness-implementation type III mixed methods study coordinated at the Centre for Addiction and Mental Health (CAMH) in Toronto. Implementation is supported by the Provincial System Support Program, a CAMH-based programme with provincial offices across Ontario, and Extension of Community Healthcare Outcomes Ontario Mental Health at CAMH and the University of Toronto. The primary outcome is fidelity to the EPI model as measured using the First Episode Psychosis Services-Fidelity Scale. Four hundred participants in the EPI programmes will be recruited and followed using both individual-level assessments and health administrative data for 2 years following NAVIGATE initiation. People with lived experience will be engaged in all aspects of the project, including through youth and family advisory committees. ETHICS AND DISSEMINATION: Research ethics board approval has been obtained from CAMH and institutions overseeing the local EPI programmes. Study findings will be reported in scientific journal articles and shared with key stakeholders including youth, family members, programme staff and policymakers. TRIAL REGISTRATION NUMBER: NCT03919760; Pre-results. © Author(s) (or their employer(s)) 2020. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Real-world design leveraging routinely collected administrative data, established network of early psychosis intervention programmes and routine fidelity assessments.

Implementation supported by regional specialists and Project Extension of Community Healthcare Outcomes, a digitally supported hub-and-spoke training model.

Collaboration with youth, family members and other stakeholders including government policymakers at all stages of the project.

Interpretation of findings limited by absence of a comparison group for individual-level (patient/family member-reported and interviewer-rated) outcomes.

Introduction

Psychosis typically has its onset in youth and emerging adults (YEAs), a critical developmental stage for exploring and solidifying future career trajectories, intimate relationships and worldviews.1–3 Psychosis is a group of symptoms characteristic of diagnoses of schizophrenia, schizoaffective disorder, substance-induced psychotic disorders and bipolar disorder or major depressive disorder with psychotic features; taken together, these disorders account for the greatest disability among all medical illnesses in YEAs in developed countries.4 Furthermore, these conditions account for high healthcare costs.5 They can have a profound effect on young people: evidence suggests that a recent diagnosis of psychotic disorder confers an 8–24-fold greater mortality rate among YEAs compared with the general population, mostly due to suicide.6 7 Even after accounting for suicide, people with these disorders experience a shortened lifespan attributable to downward socioeconomic drift and poorly treated comorbid physical health disorders.8–10 Consistent evidence has demonstrated that longer duration of untreated psychosis is associated with more severe symptoms, lower likelihood of remission, poor social functioning and global outcome.11

Early psychosis intervention (EPI) programmes are designed to reduce barriers to treatment and improve recovery from first-episode psychosis (FEP). In a naturalistic study in Ontario, receiving care in an EPI programme was associated with improved access to psychiatric care, fewer emergency department (ED) visits and reduced all-cause mortality.12 Additional benefits associated with EPI in the literature include improved psychosis symptoms and reduced risk of relapse, fewer hospital readmissions and increased employment rates.13–16 Evidence suggests that EPI is likely cost effective, and possibly even cost saving compared with alternatives.17 Despite the demonstrated benefits of EPI, disability associated with psychotic disorders has not improved,18 and recovery rates in EPI programmes remain low.19 Achieving consistent delivery of high-quality, evidence-based care in EPI programmes is a major challenge.20 This may be related to specific deficits in delivering recovery-based services, including case management coupled with individualised psychosocial interventions, family education and support, and supported education and employment. Even in clinical service delivery trials, only a minority of patients receive the full range of psychosocial interventions offered by the service.13 14 Effective implementation and sustainability of recovery-based care in real-world EPI settings remains a challenge.21–24

The province of Ontario, Canada has prioritised EPI care through a separate funding stream and the establishment of EPI programme standards,25 with over 50 EPI sites delivering care for both non-affective and affective psychosis (ie, bipolar disorder and major depressive disorder with psychotic features). These EPI programmes are coordinated through the Early Psychosis Intervention Ontario Network (EPION), which receives funding from the Ministry of Health and operational, professional and financial management support from the Provincial System Support Program (PSSP), based at the Centre for Addiction and Mental Health (CAMH) in Toronto. EPION and PSSP led a pilot study to measure fidelity to current EPI standards using the First Episode Psychosis Services—Fidelity Scale (FEPS-FS)26 completed by peer assessors.27 The study found that the assessed EPI programmes lacked a structured or manualised process for delivering recovery-oriented care, which aligned with a prior key informant survey in which Ontario EPI programmes cited a need for tools to support service delivery, access to staff training and guidance on implementing new practices.22 Despite being connected by EPION, these programmes felt they were lacking a community of practice.

NAVIGATE was developed and studied with funding from the National Institute of Mental Health Recovery After an Initial Schizophrenia Episode (RAISE) Initiative. A model that addresses the need for consistently delivered recovery-based services, NAVIGATE is a form of coordinated specialty care for FEP consisting of four key intervention components: (1) individualised medication management using a decision support tool, (2) a package of psychoeducation and a blend of evidence-based psychotherapies called ‘individual resiliency training’, (3) supported employment and education and (4) a family education programme.28 NAVIGATE operationalises current EPI standards using manualised protocols, and systematically applies the four components to all patients. Regular team reviews assess patient progress, fidelity and need for adjustments. NAVIGATE was evaluated from 2009 to 2014 in a cluster randomised controlled trial (RCT) involving 404 individuals with FEP in 34 community mental health centres across the USA with no pre-existing EPI programmes.21 Compared with usual care, NAVIGATE treatment provided greater improvement in symptoms and real-world functioning, including social functioning and engagement in educational and vocational training.21 29

CAMH, having implemented NAVIGATE in its EPI programme, partnered with PSSP, EPION, the Ontario Ministry of Health and former EPI service users on a Canadian Institutes of Health Research Strategy for Patient-Oriented Research Innovative Clinical Trials grant to evaluate implementation of NAVIGATE in diverse EPI sites across the province. Implementation support is provided by PSSP. Training is delivered by experts who developed the NAVIGATE model, as well as expert users from CAMH. Knowledge transfer is supported by the Extension of Community Healthcare Outcomes (ECHO) Ontario Mental Health at CAMH and the University of Toronto, which consists of specialist hubs connected with multiple spoke (learner) teams in remote areas through secure multipoint videoconferencing technology. ECHO is designed to facilitate an interprofessional community of practice, provide access to specialised expertise and overcome geographic barriers in relation to coordinated specialty care. It has evidence of clinical effectiveness in the USA,30 and high provider satisfaction, increased knowledge and self-efficacy in managing mental health and addictions in rural and remote settings in Ontario.31 This implementation plan prioritises affordability, adaptability, spread and sustainability, as well as patient and caregiver engagement. In addition to collecting individual-level data, the study will examine outcomes with relevant comparison groups in administrative data held at Institute for Clinical Evaluative Sciences (ICES) at dramatically lower time and cost compared with an RCT.

Objectives

Early Psychosis Intervention-Spreading Evidence-based Treatment (EPI-SET), a multisite non-randomised effectiveness-implementation hybrid type III trial,32 will evaluate the implementation and impact of NAVIGATE in five EPI programmes in Ontario, supported by PSSP’s provincial regional network of implementation experts, and training through ECHO. First, we will assess at a programme level whether implementation of NAVIGATE, as a structured model of care, leads to improved fidelity to the EPI standard. Because NAVIGATE has not been evaluated in Ontario, which, unlike the original RAISE Study setting in the USA, is characterised by established EPI programmes and a system of universal healthcare, we will also assess the effectiveness of NAVIGATE in improving individual-level outcomes over time, and system-level outcomes using quasi-experimental methods and administrative data. Lastly, we will measure youth and family member engagement in the study. Specifically, our primary objectives are to:

Assess implementation of NAVIGATE and whether this leads to improvement in fidelity to the EPI standards (using the FEPS-FS).26

Determine longitudinal change in functioning and symptoms in NAVIGATE patients.

Compare system-level outcomes, including days in hospital, ED visits, suicide attempts, mortality and system costs among patients receiving NAVIGATE compared with patients at other EPI sites not receiving NAVIGATE as well as patients with psychotic disorders who are not enrolled in EPI.

Evaluate youth and family member engagement in this implementation study according to established frameworks.

Methods and analysis

Study setting

Ontario is the most populous province in Canada, with a population of approximately 14 million and a system of universal healthcare delivered through the Ontario Health Insurance Plan (OHIP). Five EPI programmes have been selected to represent different geographic regions of Ontario. Each site offers services from a psychiatrist as well as other health professionals (nurses, occupational therapists and social workers) who provide case management and family support. The diversity of sites allows for the opportunity to examine subpopulations, including ethno-racial minority groups, Indigenous populations and rural residents. CAMH acts as the coordinating site.

Eligibility criteria

Mirroring the RAISE Study, we will recruit 400 patients (100 in months 6–12, 300 in months 12–24) into the study with a 2-year follow-up.33 Participants will be in their first year of treatment in the participating EPI programmes. Inclusion criteria are broad, reflecting the real-world populations served by the programmes.

Age range of 14–35 years.

Any Diagnostic and Statistical Manual of Mental Disorders (DSM-5) diagnosis that can manifest as FEP (schizophrenia, schizoaffective disorder, schizophreniform disorder, bipolar I disorder with psychotic features, major depressive disorder with psychotic features, substance-induced psychotic disorder or unspecified psychotic disorder).

Within first year of treatment at a participating EPI programme.

Absence of psychosis.

Inability to provide informed consent to participate in the research study.

Objective 1: implementation and evaluation

Implementation plan

Participating EPI programmes are guided through a facilitated, staged, change process.34 35 Regional PSSP facilitators lead implementation in collaboration with NAVIGATE trainers, CAMH NAVIGATE experts and the EPI programme staff lead at each site.36 ECHO will become increasingly involved over time to support NAVIGATE as part of routine practice, and this will also be monitored. The stages of our implementation process are summarised in table 1.

Table 1

Stages of implementation to be used in EPI-SET

Stage	Overarching goal	Steps
1. Exploration	To assess site capacity and need, build engagement	CAMH NAVIGATE experts, PSSP implementation specialists and ECHO team meet with each site to explain NAVIGATE, learn about their current staffing and service delivery processes and discuss how to integrate NAVIGATE into their practice. EPI staff complete a site readiness assessment survey and a programme fidelity review is conducted to learn more about site capacities and needs for NAVIGATE implementation.
2. Installation	To create structures and build capacity for implementation	EPI staff and PSSP facilitators meet for an intensive in-person and/or blended in-person and virtual staff training over several days co-led by CAMH NAVIGATE experts and NAVIGATE trainers along with youth and family partners, delivered in lectures, role playing and discussions. PSSP supports preparation, including staff allocation to each NAVIGATE role, how interventions will be documented, how documentation will be used in NAVIGATE supervision and how to prepare for ECHO sessions. The ECHO team works with each site to ensure setup and ongoing functioning of infrastructure for live videoconferencing.
3. Initial implementation	To trial and refine implementation	Each site will begin NAVIGATE delivery using feedback from various sources, including ECHO training and coaching, contact and progress notes, and staff meetings to refine the implementation and service delivery processes and to build staff skills. The PSSP implementation specialists will document progress, strategies and challenges to implementation in a structured log that they will share in regular meetings with the NAVIGATE experts and other facilitators for continuous improvement, mindful of site-specific factors and population-specific factors (eg, sex, race/ethnicity, rural vs urban) that may influence implementation. Staff feedback will be used to refine the implementation process.
4. Full implementation and sustainability	To stabilise practice so that the implemented practice is routine	NAVIGATE is fully embedded into the organisation and can be sustained with internal resources. The ECHO team will work in collaboration with Study sites via videoconferencing technology to create and sustain a community of practice for NAVIGATE implementation and spread beyond the duration of this study, such that it becomes routine practice. After each ECHO session, questionnaires will be used to evaluate satisfaction and inform ECHO modifications, and cases discussed during the sessions will generate implementation recommendations, with surveys approximately 3 months later to evaluate adherence to these recommendations. Pre-knowledge and post-knowledge tests and competence assessments will be used to assess how knowledge changes throughout the ECHO cycle.

CAMH, Centre for Addiction and Mental Health; ECHO, Extension of Community Healthcare Outcomes; EPI, early psychosis intervention; EPION, Early Psychosis Intervention Ontario Network; EPI-SET, Early Psychosis Intervention-Spreading Evidence-based Treatment; PSSP, Provincial System Support Program.

Implementation evaluation

The implementation evaluation is summarised in table 2.37 Fidelity to the EPI standard, our primary outcome, will be assessed using the FEPS-FS.26 The FEPS-FS is a validated measure of fidelity of service delivery to the current standard of EPI evidence-based practice based on 32 program-specific items (individual and team practices) rated on a 5-point scale from ‘not implemented’ to ‘fully implemented’. A rating of 4 is considered satisfactory adherence. Trained assessors will review site administrative data, data abstracted from client health records and conduct phone interviews with site staff to complete the FEPS-FS. Fidelity assessments will be conducted for each site at baseline, 1 year and 2 years post implementation. Descriptive statistics (percentages, means and ranges) will be reported for the total scale score and for subscale scores that align with NAVIGATE components. With a small number of sites, we cannot quantitatively test changes over time and across sites but will describe and qualitatively compare findings.

Table 2

Implementation evaluation measures and timeline

Measurement domain	Measure	Implementation stage
		Explore (stage 1)	Install (stage 2)	Initial implementation (stage 3)	Full implementation and sustainability(stage 4)
		Capacity and needs assessment	Plan and prepare	Trial and refine	Stabilise practice	Practice is routine
		Months1–3	Months4–6	Months7–12	Months13–24	Months 25–42
Implementation process
Implementation milestones, risk, action	Tracker tool based on NIRN framework90	·	·	·	·	·
ECHO implementation (clinical coaching)	Post-session questionnaires and phone calls	·	·	·	·	·
Implementation outcomes (organisational capacity)
Readiness to implement	RMT adapted38	·			·	·
Fidelity to EPI model	FEPS-FS26	·			·	·
Fidelity to NAVIGATE	Module checklist		·	·	·	·
Staff perceptions of value and feasibility*	CFIR adapted39 40				·
Implementation outcomes (staff capacity)
Staff knowledge and skills	ECHO survey	·			·	·

*Includes perceptions of ECHO support.

CFIR, consolidated framework for implementation research; ECHO, Extension of Community Health Outcomes; EPI, early psychosis intervention; FEPS-FS, First Episode Psychosis Services–Fidelity Scale; NIRN, National Implementation Research Network; RMT, readiness monitoring tool.

To determine whether implementation of NAVIGATE is associated with improvement in fidelity to the EPI standard, programme staff will document delivery of core modules for each of the four interventions. We will calculate the percentage of core modules completed per intervention per patient to assess penetration and to identify variations in delivery both within sites and across sites. Additionally, we will calculate frequency of team activities (eg, weekly meetings, direct supervision) and assess staff perceived competence in delivery of NAVIGATE using the readiness monitoring tool38 and the competency assessment questionnaires completed at the conclusion of each ECHO cycle (see below).

The consolidated framework for implementation research (CFIR)39 40 will be used to systematically assess contextual factors that are associated with effective implementation in relation to five major domains: intervention characteristics (eg, complexity, relative advantage), outer setting (eg, external policy, patient needs), inner setting (eg, resources, fit, leadership), staff characteristics (eg, knowledge, beliefs) and implementation process (eg, facilitation, planning, coaching). A semi-structured interview will guide data collection.39 41 Interviews will be conducted with EPI staff and organisation leaders at each site at the end of the active implementation and will be recorded and transcribed. The CFIR will provide the organising framework for qualitative data coding and analysis. At each analysis stage, coding and development of emergent themes will be conducted by multiple coders using NVivo software, with consensus achieved through discussion and deliberation. In combination with fidelity results, we will examine implementation facilitators and challenges within and across sites.

Following an established evaluation framework developed by ECHO that builds on continuing education programme evaluation,42–44 we will assess staff participation and retention, satisfaction with ECHO support and perceived changes in competence to deliver NAVIGATE. Competency assessment questionnaires will be administered to staff prior to participation in ECHO sessions and at the conclusion of each ECHO cycle to assess changes in attitudes, knowledge and self-efficacy (self-reported competence) in delivering NAVIGATE components.31

Objective 2: evaluation of individual-level outcomes

Study procedures

Participant recruitment will be initiated by the clinical team at each participating site who will obtain verbal permission from potential participants to be contacted by a member of the research team. The research team will then meet with participants via live two-way videoconference, consistent with ECHO infrastructure, to obtain informed consent. Participants will read the consent form online and have the opportunity to ask the research team questions in real time over videoconference. Participants will provide digital consent signatures by clicking a checkbox to indicate their consent. A copy of the signed consent will be emailed (or mailed, if requested) to the participant. A family member of each patient will also be invited to participate in the study. All interviewer-rated assessments will also be administered via two-way live videoconference by trained interviewers, in a manner identical to the approach used in the original RAISE Trial.21 This method of remote assessment is comparable to in-person assessments for both patient acceptability and reliability.45 De-identified study data, including interviewer-rated and participant/family member self-report assessments, will be entered and managed using research electronic data capture tools hosted at CAMH.46 47 It is anticipated that the first participant will complete the study in month 30, while the final participant will complete a 2-year follow-up at month 48.

Outcome measures

Individual-level outcome measures are outlined in table 3. Participants will be administered the Structured Clinical Interview for DSM-5 for formal diagnostic assessment,48 with information supplemented by the clinical team as needed. Demographics and a medical history will also be recorded at baseline.

Table 3

Participant and family member assessment tools and schedule*

Assessments	Construct	Who completes	Months from baseline
			0	6	12	18	24
Screening
Demographic form		Youth	·
SCID-548 (all modules)	Psychopathology	Interviewer	·
Medical history		Youth	·
Functional assessments
QLS49	Condition-specific quality of life	Interviewer	·	·	·	·	·
WHODAS 2.051 (self-administered version)	Generic quality of life	Youth	·	·	·	·	·
Clinical assessments
SCID-548 (mood disorder, psychotic disorder, substance use disorder, anxiety disorder and obsessive-compulsive and related disorder modules)	Psychopathology	Interviewer			·		·
BPRS52	Psychotic symptoms	Interviewer	·	·	·	·	·
CGI53	Overall illness severity and improvement	Interviewer	·	·	·	·	·
PHQ-954	Depression	Youth	·	·	·	·	·
AADIS55	Substance use	Youth	·	·	·	·	·
Service utilisation
SURF56	Service utilisation	Youth	·	·	·	·	·
Satisfaction, care quality and therapeutic relationship
OPOC-MHA58 (client version)	Satisfaction with services	Youth	·	·	·	·	·
STAR-P59	Therapeutic relationship	Youth	·	·	·	·	·
RSA60	Perceptions of recovery principles and overall quality of services	Youth	·	·	·	·	·
Family member-completed assessments
WHODAS 2.051 (proxy-administered version)	Generic quality of life	Family member	·	·	·	·	·
LSP-2061–63	General functioning	Family member	·	·	·	·	·
OPOC-MHA58 (caregiver version)	Satisfaction with services	Family member	·	·	·	·	·
S-CGQoL64	Caregiver quality of life	Family member	·	·	·	·	·

*Time indicates months after NAVIGATE initiation for each participant.

AADIS, Adolescent Alcohol and Drug Involvement Scale; BPRS, Brief Psychiatric Rating Scale; CGI, Clinical Global Impression; LSP-20, Life Skills Profile-20; OPOC-MHA, Client Ontario Perception of Care Tool For Mental Health and Addictions; PHQ-9, Patient Health Questionnaire-9; QLS, Quality of Life Scale; RSA, Recovery Self-Assessment; S-CGQoL, Schizophrenia Caregiver Quality of Life Questionnaire; SCID-5, Structured Clinical Interview for DSM-5; STAR-P, Scale to Assess Therapeutic Relationships—Patient Version; SURF, Service Use and Resource Form; WHODAS 2.0, WHO Disability Assessment Schedule 2.0.

Functioning will be assessed with the Heinrichs-Carpenter Quality of Life Scale (QLS), administered by semi-structured interview.49 The QLS is the most comprehensive measure of community functioning in schizophrenia populations50 and the primary outcome in the RAISE Trial.21 While the QLS is psychosis-specific, we will measure general functioning using the 12-item self-report WHO Disability Assessment Schedule 2.0 (WHODAS 2.0),51 consistent with DSM-5 recommendations for use across mental illnesses.48 The interviewer-rated Brief Psychiatric Rating Scale 24-item will be used to assess symptom severity52 and the Clinical Global Impression Scale will characterise overall illness severity and improvement.53Depression symptoms and severity will be assessed with the Patient Health Questionnaire-9 (PHQ-9).54 The Adolescent Alcohol and Drug Involvement Scale (AADIS) will be administered to characterise current substance use.55 The Service Use and Resource Form will be used to measure utilisation of mental health and other medical services across residential, inpatient and outpatient treatment settings, with an add-on item to assess medication adherence.56 Most of the patient-level assessment tools selected were those used in the original RAISE Study, with the exception of the PHQ-9 for depression (vs the Calgary Depression Scale for Schizophrenia,57 since Ontario EPI programmes treat affective as well as non-affective psychosis), and the addition of the AADIS. To measure satisfaction with their care, participants will complete the Client Ontario Perception of Care Tool For Mental Health and Addictions (OPOC-MHA).58 The Scale to Assess Therapeutic Relationships—Patient Version will be used to assess participants’ perception of the therapeutic relationship with their care team,59 and perspectives on the quality of care will be assessed using the participant-rated Recovery Self-Assessment Scale.60

To capture the caregiver’s perspective on the participant’s functioning, a family member will complete the 12-item proxy-administered version of the WHODAS 2.0, as well as the Life Skills Profile-20.61–63 Family members will also complete the caregiver version of the OPOC-MHA to assess their satisfaction with care and the Schizophrenia Caregiver Quality of Life Questionnaire64 to assess their quality of life.

Statistical analysis

First, we will undertake a pre-post analysis of all of the clinical outcome variables. Next, we will compare clinical outcomes common to both EPI-SET and the RAISE Trial with the aggregate data from the RAISE Study to assess comparability of the intervention in different jurisdictions, using a matching-adjusted indirect comparison model.65 In this approach, we adjust the population receiving the intervention to match the average baseline characteristics with a reference population using propensity scores. We then compare outcomes across balanced populations. This is facilitated by the use of individual patient data (ie, individual-level outcome measures as in the RAISE Study) and the collection of similar baseline characteristics that might influence outcome (eg, age, education, sex, baseline illness severity, duration of untreated psychosis). As with RAISE, the primary outcome of this subproject will be the total QLS Score. We will compare our sample with the matched RAISE sample on mean or median changes in patient-level outcome data. We will also identify subgroups of patients with different functional outcome and symptom trajectories using latent class growth analysis and latent growth mixture modelling.

Overall retention rate at each data point is expected to fall within the acceptable range for statistical correction. Every effort will be made to prevent dropouts/missing data and to complete relevant assessments for participants who drop out, including reasons for dropout. As is typical in longitudinal research, we anticipate non-random missing data. Sensitivity analysis will be conducted to evaluate the impact of non-random missing data. We will use a variety of software platforms to perform the described analyses including the Mplus platform (Muthen and Muthen, 2009–2016) and the R Project for Statistical Computing. As we anticipate missing data both within and across assessments, both cross-sectional and longitudinal missing data will be handled using best practice procedures including multiple imputation and full-information likelihood estimation.

Objective 3: evaluation of system-level outcomes

Data sources

The primary data collected for patients receiving NAVIGATE in the study sites will be linked deterministically to data sources held at ICES via unique OHIP number. Only NAVIGATE patients who have consented will have their program-level data linked to ICES data. The following ICES data sources will be used: the Registered Persons Database, which contains information on patient demographics and deaths; the OHIP database, which captures data for physician visits and respective billings66; the Canadian Institute for Health Information Discharge Abstract Database, which captures all non-psychiatric and non-adult psychiatric hospitalisations66 67; the Ontario Mental Health Reporting System, which captures adult psychiatric hospitalisations68; the National Ambulatory Care Reporting System, which captures ED visits and other ambulatory care69 and the Ontario Drug Benefits (ODB) claims data, which provides information on all outpatient prescriptions covered by ODB (based on financial need for those under 65 years of age and for young people up to age 25 who lack private insurance). These databases can be linked via the encrypted health card number such that all the information is available for each individual, and de-identified. The health administrative data used for this analysis are collected by the Ministry of Health and stored by ICES without patient consent with a number of protections in place that have been described elsewhere (www.ices.on.ca). Data will be stored and analysed onsite at ICES following procedures approved by Ontario’s Information and Privacy Commissioner.

Study design

This universal administrative data source allows us to compare outcomes among study participants and two comparison groups: (1) FEP patients in Ontario who have received EPI services at non-NAVIGATE sites and (2) FEP patients who have not received EPI services. Comparison groups will be identified using a validated algorithm that detects incident cases of psychotic disorders in the administrative data.70 We will use propensity score methods to ensure comparison populations are similar to NAVIGATE participants based on observed characteristics.71 A propensity score is defined as the probability that a person is in the ‘exposed’ category. In this case, the exposure condition is access to the NAVIGATE protocol. The propensity score is developed using logistic regression to model exposure to NAVIGATE as a function of observed covariates likely to affect the probability of receiving the intervention to yield a probability of NAVIGATE access for each subject, and therefore creates a scenario whereby individuals who do and do not receive NAVIGATE are comparable based on measurable variables within the propensity score model. The propensity score model will include sociodemographic characteristics, clinical factors and prior service use. Individuals who used EPI services within 5 years preceding the index date will be excluded.

Outcomes

Our primary outcome is number of psychiatric hospitalisation days in the year following the index date (NAVIGATE baseline). Secondary outcomes will include ED visits, suicide attempts, mortality, number of psychiatric hospitalisations, time to first psychiatric hospitalisation, visits to outpatient psychiatrists and visits to outpatient primary care providers, stratified as mental health versus non-mental health-related.72–74

Covariates

We will extract information on sociodemographic characteristics, including age, sex, neighbourhood-level income quintile and rurality of residence. Clinical covariates will include type of diagnosis (schizophrenia, mood disorder and so on), source of index diagnosis (hospitalisation, outpatient visit) and history of visits with alcohol-related or substance-related diagnoses. We will also measure prior service use for mental disorders, including the number of mental health-related visits to primary care providers, psychiatrist visits, mental health-related ED visits and psychiatric hospitalisations.

Our primary outcome for Objective 3 is number of psychiatric hospitalisation days in the year following NAVIGATE admission. We will compare the three groups (NAVIGATE, non-NAVIGATE EPI users and non-EPI users with FEP) across all covariates listed above. We will subsequently model total mental health-related hospital days. The regression model will be determined by the distribution of the dependent variables. If normally distributed, we will use linear regression; if the distribution assumes a Poisson distribution, we will use Poisson regression. Secondary outcomes will be modelled using logistic regression (for binary outcomes) or Cox proportional hazard modelling (for time-to-event outcomes).

Cost analysis

We will employ a costing algorithm available at ICES75 to estimate all direct patient-level healthcare costs incurred by the public third-party payer (Ontario Ministry of Health) across the three comparison groups. This will include costs of hospitalisations (both non-psychiatric and psychiatric); ED visits; physician services (ie, primary care, psychiatry and other) and diagnostic tests; outpatient prescription drugs for individuals covered under the ODB programme; home care; long-term care and other care. Further details on the methodology to calculate cost for each type of healthcare service can be found elsewhere.76 Costs will then be compared between groups to ascertain whether there are any cost savings associated with NAVIGATE. Furthermore, assuming that all subjects will incur healthcare costs, we will use a generalised linear model with a gamma distribution and a log link to model healthcare costs and determine the main cost drivers. Actual model parameters will be determined by the nature of the cost distributions.

Objective 4: evaluation of engagement of people with lived experience

Consistent with best practices for youth77–79 and patient80–85 engagement, opportunities for youth and family participation will range from ad hoc, limited commitment (eg, surveys or interviews) to full, ongoing participation with opportunities for research team membership and mentorship. A youth and family member who previously received EPI services at CAMH have collaborated on the project from its inception. When the study launched, youth and family advisory committees were formally established, comprising youth and family members (partners) from each study site. These committees meet monthly (via ECHO-style videoconference or in-person) to guide the research team on recruitment strategies, assessment and treatment protocols, outcome measures, interpretation of findings and dissemination of the trial learnings to knowledge users. A central research coordinator facilitates the meetings. Each committee assigns a lead who is also a member of the Central Steering Committee where decisions are made, and represents the views of their members. Partners are oriented to the study, their readiness to join the study team is explored, and they receive NAVIGATE training tailored to their learning needs. Scientific, clinical and programme leaders receive training on how to include patients and family members on a team in a meaningful manner. Partners are compensated for their involvement in this study.

Evaluation of youth and family engagement

In collaboration with the Patient Engagement Resource Centre,86 we selected the Public and Patient Engagement Evaluation Tool87 participant questionnaire for formative and summative evaluation of the strength of the partner engagement, and the PCORI engagement activity inventory (PECI)88 to capture partners’ contribution to the EPI-SET Study. The tools are adapted based on partners’ feedback. At baseline, we assess the partners’ intent to contribute to the various aspects of the research, and will compare this to their reported contribution using the PECI at three different times throughout the study. The baseline questionnaire also assesses the partners’ and research team members’ need for support to achieve authentic engagement, and resources are offered accordingly. Partners’ engagement experience with the advisory committee and within the Central Steering Committee meetings is assessed monthly for formative purposes, and used to correct process as required to optimise experience. A summative evaluation is conducted at the study end or when a member leaves the committee.

Patient and public involvement

Patient involvement of youth and families with lived experience was informed by current best practices in the research literature.77–85 A youth and family member who previously received EPI services collaborated on the project from its inception, with active participation on the project Steering Committee where high-level decisions about the project are made. They also lead the Youth Advisory Committee and Family Advisory Committee, respectively. These committees include patients who have received care from the local study sites. In monthly meetings in-person and by videoconference as well as over email, the youth and family members draw on their past experiences to help guide project decisions. Individual-level assessments completed by youth have all been reviewed and selected by the Youth Advisory Committee and assessments completed by caregivers by the Family Advisory Committee. Study principal investigators representing the different project objectives rotate through attending the Youth Advisory Committee and Family Advisory Committee meetings as another mechanism for regular bidirectional communication.

The perspectives of enrolled study participants are privileged through the many self-report assessments they are asked to complete including measures of their experiences receiving NAVIGATE. Study participants who indicate their interest in receiving published materials presenting results of the study will provide their email address and will be contacted as materials are published. For additional details on patient and public involvement, please see Objective 4.

Ethics and dissemination

The study was approved by the Research Ethics Board (REB) at Lakeridge Health, Niagara Region Public Health, North Bay Regional Health Centre, Health Sciences North and the Canadian Mental Health Association Waterloo Wellington, in addition to the coordinating centre, the Centre for Addiction and Mental Health. The study protocol was prepared according to SPIRIT guidelines (online supplementary file l).89 REB-approved protocol amendments will be posted on clinicaltrials.gov. The principal investigators, scientific committee and study team will meet regularly to review accrued data, data confidentiality, adherence to protocol design, recruitment and implementation. During meetings, the scientific committee will also review the enrollment of data, the accrual and integrity of clinical data, implementation and fidelity of NAVIGATE and any adverse event associated with the various components of the study.

The results of each of the four study objectives will be reported in scientific journal articles and shared with key stakeholders as they become available. The study has been discussed at regular EPION meetings (of programme managers and other representatives) and will be formally presented at its bi-annual conference. De-identified participant data are available on reasonable request for each objective except system-level data held at ICES. A publications committee will manage access to data. Requests can be made by contacting principal investigator Dr. Aristotle Voineskos at aristotle.voineskos@camh.ca.

Conclusions

This multisite non-randomised pragmatic hybrid effectiveness-implementation type III mixed methods study leverages collaboration with multiple stakeholders to evaluate implementation of NAVIGATE in diverse EPI programmes in Ontario. It is expected that implementation will lead to improved fidelity to the EPI model, a community of practice and model for continuous learning for EPI programmes in Ontario, and improvements in patient symptoms, functioning and service utilisation (ie, diversion from acute mental health treatment including hospitalisation days), with meaningful engagement of youth and family members in the research process. It is anticipated that this research will highlight key ingredients for spread and scale of the NAVIGATE model to additional settings with the goal of improving recovery for YEAs with psychosis.