Understanding and evaluating new models of Child and Adolescent Mental Health Services in South-East England: a study protocol for an observational mixed-methods study.

INTRODUCTION: Increased demand for Child and Adolescent Mental Health Services (CAMHS), alongside concerns that services should be better commissioned to meet the needs of the most vulnerable, has contributed to a requirement to transform services to improve accessibility, quality of care and health outcomes. Following the submission of government-mandated transformation plans for CAMHS, services in England are changing in how, where and by whom they are delivered. This protocol describes the research methods to be applied to understand CAMHS transformations and evaluate the impact on the use of mental health services, patient care, satisfaction, health outcomes and health resource utilisation costs. METHODS AND ANALYSIS: A mixed-methods approach will be taken in an observational retrospective study of CAMHS provided by a large National Health Service (NHS) mental health trust in South-East England (Oxford Health NHS Foundation Trust). Quantitative research will include descriptive analysis of routinely collected data, with difference-in-differences analysis supplemented with propensity score matching performed to assess the impact of CAMHS transformations from 2015 onwards. An economic evaluation will be conducted from a healthcare perspective to provide commissioners with indications of value for money. Qualitative research will include observations of services and interviews with key stakeholders including CAMHS staff, service users and guardians, to help identify mechanisms leading to changes in service delivery, as well as barriers and enabling factors in this phase of transformation. ETHICS AND DISSEMINATION: This project has been registered with NHS Oxford Health Foundation Trust as a service evaluation. Informed consent will be sought from all stakeholders partaking in interviews according to good clinical practice. A local data sharing protocol will govern the transfer of quantitative data. Study findings will be published in professional journals for NHS managers and peer-reviewed scientific journals. They will be discussed in seminars targeting CAMHS providers, managers and commissioners and presented at scientific conferences. © Author(s) (or their employer(s)) 2018. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The proposed study employs a mixed-methods approach to help understand and evaluate the complexity of Child and Adolescent Mental Health Services (CAMHS) transformation and to triangulate findings from different methods.

The study uses data from one large NHS Trust, which ensures an element of consistency in data collection and provision.

Data availability, however, is expected to be limited for some outcome measures.

CAMHS transformations are ongoing and the process of embedding changes will continue beyond this study period.

Introduction

The contribution to the global burden of disease from mental health conditions is large and increasing.1 2 Globally, depressive disorders are the single largest contributor to non-fatal health loss, while self-harm is the cause of nearly 800 000 deaths.3 The global cost of mental health conditions in 2010, both direct and indirect, was estimated at nearly $2.5 trillion and projected to increase to over $6 trillion by 2030.4 The onset of mental health conditions occurs largely by the age of 24 and prevalence in children and adolescents aged up to 18 is estimated at 13%,5 6 with some evidence of recent increases for emotional problems in females.7 Untreated mental health conditions negatively impact on development throughout the life course, making access to ‘adolescent-responsive’ and high-quality health systems crucially important.8

In England, the cost of mental health conditions is estimated to be £105.2 billion each year and the prevalence among children and young people aged 5–16 is estimated at just under 10%.9 10 However, there are rising concerns about the adequacy of healthcare services to meet the needs of this group. A recent UK Government review, Future in Mind, 11 identified increased demand for Child and Adolescent Mental Health Services (CAMHS), raised concerns that services are not commissioned to meet the needs of the most vulnerable, and made recommendations as to how services should change to (A) promote resilience, prevention and early intervention, (B) improve access to effective support, (C) care for the most vulnerable, (D) promote accountability and transparency, and (E) develop the workforce. This, along with other directives, such as the Five Year Forward View for Mental Health,12 highlighted the need to ‘transform’ services to improve accessibility, increase quality of care and improve health outcomes. Local areas were tasked with submitting ‘Transformation Plans’ to NHS England setting out how they would achieve this.

Across the country CAMHS are evolving to meet those plans, with many moving from a traditional ‘tiered’ service model where, depending on perceived need, a young person is allocated to a specific tier of service aligned with complexity/need, towards a more ‘integrated’ service model. A recent Green Paper, Transforming Children and Young People’s Mental Health Provision, has restated the commitment to ensuring children and young people get timely support in the most appropriate setting.13 With significant investment in the transformation of CAMHS, and with limited mental health budgets, it is essential that healthcare planning and commissioning understand the likely impact of CAMHS transformations on services and to assess whether they meet the objectives set in Future in Mind and the Five Year Forward View for Mental Health.

This protocol sets out our approach to conducting a large observational retrospective research study commissioned by two large Clinical Commissioning Groups in South-East England and the Oxford Collaboration for Leadership in Applied Health Research and Care to examine CAMHS provided by Oxford Health NHS Foundation Trust (Oxford Health) transforming from 2015 onwards.

Methods and analysis

Study aims/research questions

This study aims to understand how CAMHS transform in terms of organisation and delivery and assess the impact of transformation on the use of mental health services, patient care, satisfaction, health outcomes and health resource utilisation costs.

Specifically we will address the following research questions:

What components are being adopted as part of the CAMHS transformations?

How are the transformations being implemented and maintained?

What are the facilitators and barriers to transformation?

What impacts have the transformations had on service activity/pathways?

What is the impact of the CAMHS transformation on patient experience, health outcomes and costs?

To ensure relevance, this study will also reflect on the extent to which the service transformations support local areas to meet the objectives set out in Future in Mind. To do so, we will draw on quantitative and qualitative evidence from this study and map this against the themes from Future in Mind.

Study setting and service transformation

Oxford Health is one of the largest UK providers of CAMHS with commissioned services covering a population of 2 million in Oxfordshire, Buckinghamshire, and Swindon, Wiltshire, Bath and North-East Somerset (collectively referred to as SWBaNES). The latter, SWBaNES, had been three differently commissioned services, but has been commissioned as an integrated service across the three areas from 2018/2019. As set out in table 1, the five areas vary widely on demographics and other indicators.

Table 1

Characteristics of County & UA areas involved in study

Demographics	Oxfordshire	Buckinghamshire	SWBaNES
			Swindon	Wiltshire	Bath and North-East Somerset
% of population under 18 (2016)	20.9	22.9	22.7	21.5	18.7
% of population aged 65+ (2016)	17.7	18.3	15.5	20.8	18.8
% of population from ethnic minorities (2016)	9.5	12.8	13.3	3.6	3.9
Deprivation (IMD score, 2015)	11.5	9.8	17.9	13.5	12.1
% of children in low-income families, under 16 (2015)	10.0	8.9	14.0	10.3	10.5
% of looked-after children for which there is concern (2016/2017)	39.7	43.1	38.1	39.4	52.9

Source: Public Health England Fingertips, Public Health Outcomes Framework (County & UA).

IMD, Index of Multiple Deprivation; SWBaNES, Swindon, Wiltshire, Bath and North-East Somerset.

The CAMHS provided by Oxford Health share common transformation goals (eg, improve accessibility and early intervention), are all working towards a THRIVE model14 15 and have some similar core components of transformation, variously: a Single Point of Access (SPA) for referrals; a School InReach Service; changes to pathways for treating young people who need a more intense or targeted approach; and finally, Community InReach, where CAMHS work more closely with third-sector partner organisations.

However, they all differ in the intensity, timing and pace of transformation. In particular, there has been a stepwise roll-out of changes across the study geography: the transformation of CAMHS was started in 2015 in Buckinghamshire and in 2017/2018 in Oxfordshire. SWBaNES have commissioned a new CAMHS model for 2018/2019, but in the meantime have continued to deliver services based on a ‘traditional, tiered system’ in which CAMHS are accessed primarily through general practitioners with some referrals from school staff and social services, and Primary Child and Adolescent Mental Health Services act as gatekeepers for all of CAMHS. Roll-out has occurred according to a range of factors, such as local decision-making and capacity, not at random.

Study design and research methods

CAMHS is a complex intervention, involving different groups of health and social care professionals, treating a variety of conditions, traditionally delivered across multiple service tiers.16 In an observational, retrospective and comparative study, we will assess the impact of the ongoing CAMHS transformation in Buckinghamshire and Oxfordshire on a broad range of outcomes, including the use of mental health services, patient care, satisfaction, health outcomes and health resource utilisation costs, by comparing them with the CAMHS in SWBaNES where many components of transformation have not yet been adopted. In line with Medical Research Council (MRC) guidelines for evaluating complex interventions, we will take a mixed-methods approach.17

A Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework will be used for evaluating the ongoing CAMHS transformations. The RE-AIM framework is appropriate for studying complex interventions and has a particular focus on implementation in real-world settings and its different dimensions align well with the goals of this study.18 For instance, given the study focuses on service transformations spurred by government initiatives, adoption is mandated, but the components of the transformations adopted may differ. The overall RE-AIM framework has helped to structure the analysis plan and the role of the respective research methods (table 2). Quantitative and qualitative evidence will be integrated iteratively throughout the project with the involvement of all researchers, thereby building and developing a theory-based understanding of the service transformations to inform the ongoing analysis.19

Table 2

Analysis and method by research question

RE-AIM	Research question	Analysis	Method	Future in Mind
				Theme 1: Promoting resilience, prevention and early intervention	Theme 2: Improving access to effective support	Theme 3: Care for the most vulnerable	Theme 4: Accountability and transparency	Theme 5: Developing the workforce
Adoption	(1) What components have been adopted as part of the CAMHS transformations?	Qualitative	Document review	x	x	x
			Observation
			Interviews
Implementation/maintenance	(2) How are the transformations being implemented and maintained?	Qualitative	Document review				x	x
			Observation
			Interviews
		Quantitative	Descriptive statistics
Implementation	(3) What are the facilitators and barriers to transformation?	Qualitative	Document review				x	x
			Observation
			Interviews
		Quantitative	Descriptive statistics
Reach	(4) What impacts have transformations had on service activity/pathways?	Qualitative	Document review	x	x	x
			Observation
			Interviews
		Quantitative	Descriptive statistics
			Regression analysis
Effectiveness	(5) What is the impact of CAMHS transformation on patient experience, health outcomes and costs?	Qualitative	Interviews	x	x
		Quantitative	Descriptive statistics
			Regression analysis
			Economic evaluation

CAMHS, Child and Adolescent Mental Health Services; RE-AIM, Reach, Effectiveness, Adoption, Implementation and Maintenance.

Qualitative data and analysis

Using qualitative methods, we will identify and describe the components of the two novel CAMHS models and evaluate its implementation. Specifically we will look in detail at three core changes: the ‘Single Point of Access’ (SPA), School InReach and the new pathways for treatment (eg, ‘Getting More Help’). First, observations of a range of different CAMHS team members and key transformation meetings with commissioners and service managers will provide greater understanding of how the new transformations are organisationally adopted and implemented. Documentary analysis will occur in tandem with observations in a complementary way to provide greater understanding of the transformations. Key sources of information for the documentary analysis include plans, meeting agendas, reports and websites detailing the CAMHS transformations.

Observations will inform interview sampling by providing an initial contact with relevant stakeholders who can help establish contacts with other key workers. This snowball sampling method is well documented in social research and organisation studies.20 From this, a minimum of 30 CAMHS staff (including clinicians, team leaders, administrative staff and NHS managers), from across the range of core components of change, will be interviewed on their experiences, thoughts and perceptions of the transformations in Oxfordshire and Buckinghamshire. The RE-AIM framework will further guide specific questions around training, management and impact on staff. Alongside this, school staff and voluntary agency workers involved in the transformations will be interviewed. The interviews will highlight: (A) perceptions and understanding of ‘conventional’ CAMHS versus transformed services, (B) changes in work role and expectations of CAMHS staff made in response to the CAMHS transformations, and (C) changes in organisational processes in response to the CAMHS transformation and their ongoing development and response to changing need. In addition, interviewees will be asked to indicate their thoughts on the impact of the changing services on prevention, early intervention, self-management and consultation/liaison for young people and families.

Following this, at least 10 young people and parents/care providers will be engaged either through in-depth interviews or focus groups, to provide an insight into the patient perspective and experience of these changes. Participants will be recruited with the help of CAMHS staff either from the central SPA or staff in the different pathways in Oxfordshire and Buckinghamshire. CAMHS workers will request consent for the researcher to contact them and, from this, participants will be recruited for interview. In the interviews, participants will be asked about their experiences of the current provision, their awareness of the new transformations and their thoughts about this in relation to their care.

Interviews will be digitally recorded and transcribed verbatim. Data analysis will follow a modified grounded theory framework, using constant comparisons, to provide a rigorous approach to the generation of concepts. There are two central features of grounded theory: that it is concerned with the development of theory out of the data and, second, that it is iterative or recursive which means that data collection and analysis often proceeds together in tandem.20 21 Each interview will first be openly coded (examining, comparing, conceptualising and categorising data) before being more consistently coded, a process by which codes are grouped into larger codes. These codes are then used to generate themes or categories, and data are constantly compared and reviewed to see which themes they best fit with.

All data written up will be anonymised. Any identifiable data will also be removed or changed. Data will be held in a secure location at the University of Oxford for at least 5 years.

Quantitative data and analysis

Outcomes

Quantitative data will include service activity data from the electronic patient record system, routine outcome measures (ROM) and other local data sets such as community mental health teams. This study will analyse a range of outcomes across the patient’s pathway: including the use of mental health services, patient care, satisfaction and health outcomes. Table 3 provides a list of the main measures to be included in the analysis, data permitting and the expected sources and units of measurement.

Table 3

Quantitative data outcomes analysis plan

Analysis of outcomes
Outcome	Variable	Unit of measure	Level of analysis	Data source
Number of referrals to services	Number of referrals	Referrals per time period Individuals referred per time period	Service	CareNotes
Number of people entering CAMHS	Number of individuals	Number per time period	Service	CareNotes
Waiting times into CAMHS	Waiting times: referral and first session Waiting times: between treatment sessions	Average wait per time period	Individual	CareNotes
Consultations with CAMHS providers	Number of sessions per patient	Average number of sessions per time period	Individual	CareNotes
Patient engagement with CAMHS	Attendance at planned CAMHS meetings Rate of cancelled/missed appointments Planned discharge	Proportion attending planned sessions Proportion of appointments cancelled or missed Number of people discharged per time period	Individual	CareNotes
Paired clinical outcomes	Revised Child Anxiety and Depression Scale (RCADS) Strengths and Difficulties Questionnaire (SDQ)	Average treatment score per time period	Individual	True Colours (ROMs)
Patient satisfaction	Experience of Service Questionnaire	Average for time period (patient) Average for time period (parent)	Individual	True Colours (ROMs)
Re-referral rates	Re-referrals within 6 months Re-referrals within 1 year	Proportion of cases re-referred	Individual	CareNotes
Staff to referred patient ratio	WTE clinicians to referred patients	• Average ratio for time period	Service	CCG KPIs

CAMHS, Child and Adolescent Mental Health Services; CCG, Clinical Commissioning Groups; KPI, Key Performance Indicators; ROM, routine outcome measures; WTE, whole-time equivalent.

Data from the electronic patient record systems, CareNotes, are expected to be available for at least 3 years (from 2015) before which a different case recording system was used, from which we will have access to a further 5 years (from 2010). ROMs are expected to be available from 2015, but matched predata and postdata are likely to be more limited. The observation period for this study will therefore include a pretransformation period, transformation and at follow-up, which will be at least 6 months after transformation (longer in the case of Buckinghamshire). Data will be available at a patient level and analysed in repeated cross sections.

Comparators

Outcomes will be compared between CAMHS in Buckinghamshire, Oxfordshire and SWBaNES to understand the impact of the transformations (research question 5) in four pairwise comparisons (Box 1). Analyses will be on repeated cross sections, comparing pretransformation and post-transformation periods. When analysing at a service level, the data will be aggregated at lower level geographies within comparator areas (patient postcodes will be used to define the lower level areas). We will compare the CAMHS after having described and ordered them based on their type, degree and intensity of transformation as well as their position in the transformation process. As per the Study Setting, we expect SWBaNES not yet to have undergone transformation (hence it is the conventional CAMHS) and for Buckinghamshire to be more advanced in transformation than Oxfordshire. This will be verified by the qualitative work. In view of the complex nature of service transformation,22 the hypothesis will be that CAMHS at a more advanced stage of the transformation process will be more effective and cost-effective when compared with conventional CAMHS or those at an earlier phase.

Pairwise comparisons

Buckinghamshire versus SWBaNES

Oxfordshire versus SWBaNES

Buckinghamshire and Oxfordshire versus SWBaNES

Oxfordshire versus Buckinghamshire

SWBaNES, Swindon, Wiltshire, Bath and North-East Somerset.

Statistical analysis

Descriptive analyses will seek to uncover and illustrate changes in the service models following transformation to help answer the second and fourth research questions. Mean values before and after the service redesign will be reported, along with the SD. The difference in two or more mean values will be tested for statistical significance at the 95% level, using t-test or analysis of variance tests. Trends in proportions or rates over time will be illustrated in charts and tested with Χ2.

We will perform difference-in-differences analysis to assess the impact of CAMHS transformation on outcomes across repeated cross sections. Difference-in-differences analysis helps identify the effect of the intervention (CAMHS transformation), free from any secular time trend in the pre-post period.20 23 A common concern with the difference-in-differences approach is that the treatment and control groups may differ in ways that would influence the treatment effect: they may not fulfil the parallel trends assumption.20 23 Following MRC guidelines, we will supplement our difference-in-differences approach with propensity score matching (PSM) to reduce observed confounding at the individual and CAMHS level.17 PSM with difference-in-differences is a proven technique for providing unbiased effect estimates in such instances.23

Bias can occur between groups (the service areas) and/or across time (where the composition of the cohorts changes between repeated cross sections). Given that the transformations are intended to improve access, changes in the composition of groups over time may be integral to the transformation. As such the details of our approach to PSM will depend on the estimand of interest and the level of analysis.23 Specifically, where changes in group composition are relevant, we will conduct analysis at the ‘Service’ level for which we will not balance on individual-level variables that may have been affected by the transformation (eg, change in case mix), but will match on area-level characteristics (eg, deprivation); where change in group composition is not relevant for the estimand, we will conduct the analysis at an ‘Individual’ level, for which we will include individual-level variables in the PSM. See table 3 for the anticipated level of analysis for each outcome listed.

We will adopt a stepwise strategy to select the PSM technique that most reduces observed confounding between the transformation areas (Buckinghamshire and Oxfordshire) and the areas delivering conventional CAMHS (SWBaNES).24–27 The general approach will involve, first, including all possible confounding variables available in the data set, assuming the inclusion of covariates not associated with the treatment assignment would have little influence in the propensity score model.24 There is support in the literature for overspecification: that is including as many variables as possible, unless it is clear they do not belong.28 Second, matching the cohorts in each CAMHS using a range of the most commonly used PSM techniques; including Mahalanobis, 1-to-1, K-to-1, kernel, local linear regression, spline and inverse probability weighting techniques. Third, assessing the performance of each PSM technique on covariate balancing based on the mean and median percentage standardised bias as well as Rubin’s B (the absolute standardised difference of the means of the linear index of the propensity score in the treated and (matched) non-treated groups) and Rubin’s R (the ratio of treated to (matched) non-treated variances of the propensity score index).29 Finally, choosing the PSM technique that has the lowest values on these performance indicators. Where data are limited the likelihood for unobserved confounding will be high, therefore we will perform a doubly robust estimation to further reduce confounding by using regression analysis after performing the most suitable PSM technique and including the confounding variables listed above as covariates.30

Activity data from electronic patient records are likely to be almost complete for those accepted into services, but less so for outcome data and follow-up clinical measures are expected to be available only for a proportion of the sample.31 We will analyse patterns of missing data across different covariates and the consistency of these patterns across sites to understand the best approach to dealing with missing data.32 Depending on the extent and patterns of missing data, this may involve multiple imputation or complete case analysis. If complete case analysis is performed on clinical outcomes, where necessary we will also consider using inverse probability weighting to avoid selection bias or ensure this is incorporated in our approach to PSM.

Subgroup analysis will be performed to investigate variation between patients with certain characteristics. The selection of different groups to look at will be informed by available literature,31 as well as by areas of political interest (eg, vulnerable groups) to be highlighted from the qualitative research. Subgroups are expected to include age, severity, ethnicity, condition type, service pathways and a particular focus on vulnerable groups such as looked-after children.

Statistical analysis will be conducted in Stata V.14.

Economic evaluation

The main economic evaluation will be a cost-consequences analysis. The comprehensiveness of this approach is well suited to informing commissioning around CAMHS transformation since it can capture a range of outcomes beyond the quality-adjusted life-year that are important to commissioners, such as satisfaction with care. As a possible supplementary analysis, the feasibility of a cost utility analysis will be explored, with utilities derived by mapping scores from the Strengths and Difficulties Questionnaire (SDQ) using published algorithms.33

The CAMHS transformations are expected to have an impact mainly on mental and community health resource utilisation during the study period, therefore the economic evaluation will take an NHS perspective on costs, including: (A) admissions to mental health hospital and inpatient length of stay; (B) contacts with outpatient mental health clinics; (C) other contacts with CAMHS (including, for example, SPA); and (D) contacts with community mental health services. All costs will be sought from the finance department of Oxford Health. We will calculate unit costs for different services and activities based on staff salaries and overheads compared with the number of patients seen/engaged. Health resource utilisation costs will be calculated by multiplying the resource utilisation with unit costs of each service. In addition, we will look to incorporate the costs to develop and implement the CAMHS transformation (eg, labour costs for irregular sessions devoted to transformation, training costs and ICT support). This information will also be sought from the respective services in Oxford Health, however, particularly given that the same NHS Trust provides all of the services, it is not clear to what extent these costs can be demarcated and attributed to a given service transformation.

In our mixed-methods approach, the interpretation of the results of the economic evaluation will be supported by the findings from the qualitative research. This will be conducted so as to verify the hypothesis that the more a CAMHS service has progressed in the transformation process the more effective and cost-effective it will be, and also to consider the extent to which different components of the CAMHS transformation drive the impact on outcomes and costs and what the impact of the implementation process is on outcomes.

Patient and public involvement

We conducted extensive Patient and Public Involvement (PPI) activities ahead of this study. We presented our plans and questions to the Oxon Young Person’s advisory group (25 members) who gave feedback on the current transformation plan, in particular around how they wanted to access services, concerns about privacy when being seen outside of traditional CAMHS settings and about seeing non-CAMHS professionals. These same questions were asked to a further 15 secondary students. In addition, 60 teachers at different schools were asked what they and their students with mental health problems needed from services. This helped inform our understanding of the context for changes in services.

Ethics and dissemination

In accordance with local protocol, the project has been registered with Oxford Health as a Service Evaluation as recommended by the local Clinical Trials and Research Governance service. Further to this, informed consent will be sought from all stakeholders (staff members, children, young people and their parents/carers) who partake in interviews as part of this project according to Good Clinical Practice. Both the data provider (Oxford Health) and the University of Oxford are signatories to the Oxford Academic Health Science Network Data Sharing Agreement. A Data Sharing Protocol will supplement that overarching agreement. Quantitative data will be shared on a pseudonymised basis.

The findings will be published in professional journals for NHS managers and peer-reviewed scientific journals. They will be discussed in seminars targeting CAMHS providers, managers and commissioners and will be presented at scientific conferences. To ensure salience, the findings of the evaluation will be considered against the thematic objectives of the Future in Mind review (table 2) and, where relevant, services will be compared with national averages available from audit data (the Mental Health Services Data Set and indicators from the NHS England Mental Health Five Year Forward View Dashboard) to help contextualise the findings.

Discussion

The relative efficiency and effectiveness of different models of CAMHS delivery is largely unknown. There are few economic evaluations and those have often been of relatively low quality.34 One reason for this is a paucity of outcome and activity data as noted in Future in Mind (notably this study comes when data collection and reporting has recently been mandated nationally as part of the Mental Health Service Data Set and Mental Health Five Year Forward View Dashboard to help understand activity and outcomes in CAMHS).11 As such, it is now an ideal time to better develop the evidence base to determine the impact of the transformation of CAMHS services across the UK. This study may provide commissioners and clinicians involved in CAMHS with: (1) better understanding of service redesign around targeting care to individual patients; (2) better ways of accessing clinical and economic data to inform research, clinical practice and decision-making around service redesign; (3) improved innovation and added value to CAMHS by better embedding evidence-based practices; (4) help to assess the impact of commissioning decisions on the CAMHS pathway; and (5) the framework by which to evaluate and the first findings that could inform the development of CAMHS models nationally. This is consistent with the learning needed to inform local practice as set out in the recent Green Paper.13

However, the complex nature of CAMHS, and by consequence the service transformations taking place, is a challenge to evaluation. The changes to most components are unlikely to be discrete, making it difficult to fix points in time when the ‘new’ service models are in place. Similarly, while changes have taken place already at one site (Buckinghamshire) and are under way at another (Oxfordshire), the complexity of transformations means changes will continue and will likely be ongoing beyond the horizon of this study. The RE-AIM framework adopted in this study helps account for the fluid nature of the changes, specifically when considering adoption and maintenance. Moreover, this study will take a mixed-methods approach and incorporates a broad range of research questions and outcome measures. Under this approach, the quantitative and qualitative components will be taking place in parallel but iterative analysis of both data sets will ensure opportunities for results from one strand to inform the other. All of the ways in which this may occur cannot be specified in advanced, but examples include using early quantitative findings to inform the development of topic guides for staff and patient interviews, using quantitative results to test perceived effects of changes emerging from the qualitative component, or using findings from the qualitative analyses to help verify the timing of transformation for the quantitative analysis. The richness of the data being collected in this project and the mixed-methods approach increases the likelihood that the findings and the theoretical basis for changes can be validated by comparison with a different strand. Moreover, this research is drawing and building on connections with local stakeholders in service delivery. This embeddedness in clinical practice means the research is well placed to navigate the complexity of CAMHS and, ultimately, to bridge the divide between academic research and service delivery.

A specific challenge surrounding qualitative research is that the majority of research will occur when service transformations are already under way, therefore understanding of the changes at a given site may rely on recall from participants comparing the ‘conventional’ and transformed CAMHS models. Again, the mixed-methods approach will help by triangulating evidence where possible, such that perceptions of change can be compared with quantitative data.

The quantitative analysis faces a number of specific challenges common to service evaluation,35 in particular as it is non-experimental there is substantial risk of bias. We are accounting for potential observed as well as unobserved bias through statistical methods including PSM and difference-in-differences analysis. As above we will also seek to triangulate the quantitative findings with qualitative data.

Missing data are also a potential challenge; in particular data may be missing not at random. For instance, clinicians’ belief in the value of ROMs is known to be a measure of the likelihood of completion,36 which may vary systematically depending on factors such as the provision of training to promote collection. As per the methods, patterns of missing data will be analysed. Where the extent of missing data is significant, not at random and occurs mainly in the outcome variables, with no auxiliary variables, complete case analysis may be the most reliable approach. Data on missing data will in itself be of value to inform CAMHS, as improved data collection is one strand of Future in Mind.11

A further issue with our clinical outcomes recorded with SDQ is regression to the mean. That is, in the absence of treatment there would be some expected improvement in scores anyway which could obscure the true treatment effect, see for instance Edbrooke-Childs et al.31 To account for this, as noted in the Outcomes section, we will analyse both the unadjusted and adjusted treatment effects.

Future research could build on this work by expanding the scope and time frame of benefits and costs included. Studies have shown significant lifetime costs of mental health conditions in childhood which fall outside the scope of what can be observed in this study.37