Acceptability of the 'Crisis Toolbox': a skills-based intervention delivered in a Crisis Resolution and Home Treatment Team during COVID-19.

Crisis Resolution and Home Treatment Teams (CRHTTs) provide 24-hour, seven day per week support for people in crisis. The COVID-19 pandemic has placed significant demand on urgent care and increased the need for brief interventions in CRHTT settings with flexible methods of delivery. This evaluation aimed to examine client satisfaction with the 'Crisis Toolbox' (CTB), a brief, skills-based intervention delivered in one CRHTT during COVID-19. All participants who received the CTB completed a satisfaction questionnaire. Descriptive statistics were calculated to quantify acceptability and qualitative themes were generated using thematic analysis. Fifty-eight people participated, all of whom reported high levels of satisfaction with the CTB. Four qualitative themes also emerged relating to 'Active ingredients of the CTB', 'The therapeutic relationship', 'Service-user preferences' and 'Expectations and continuity of care'. The CTB appears to be a valued intervention. Further research is now needed to assess its clinical impact and effect on operational indicators.

Introduction

Services for people experiencing mental health crisis have undergone significant transformation over the last twenty years. Publication of The National Service Framework (NSF) for Mental Health [Department of Health (DoH), 1999], the National Health Service (NHS) Plan (DoH, 2000) and Mental Health Policy Implementation Guide (MHPIG) (DoH, 2001) identified the need for more accessible community-based acute care services. This led to the development of Crisis Resolution Home Treatment Teams (CRHTTs), which remain a key financial priority in the NHS Long Term Plan (NHS England, 2019).

CRHTTs aim to deliver 24-hour, seven day per week mental health support for people experiencing acute crisis. They provide rapid assessment in a least restrictive environment and offer an alternative to hospital admission through provision of immediate multidisciplinary care (DoH, 2000). Although several randomised controlled trials (RCTs) have reported some positive outcomes in relation to CRHTT service provision (Johnson et al., 2005; Stulz et al., 2020), issues with the model have also been highlighted such as a lack of continuity of care (Morant et al., 2017; Titheradge & Adrian Galea, 2019) and dissatisfaction with services (Chilman, Morant, Lloyd-Evans, Wackett & Johnson, 2021).

CRHTTs should provide psychological interventions to over 30% of its service users (Lloyd-Evans, et al., 2016). This is based on a wealth of evidence that psychologically informed interventions are effective for those presenting with self-harm and/or suicidal thoughts or behaviours (Yardley, McCall, Savage & Newton, 2019; Guthrie et al., 2001; McCabe, Garside, Backhouse, Xanthopoulou, 2018; Chopra et al., 2021). Brief interventions are recommended by the National Institute of Clinical Excellence (NICE) and are considered an indicator of service quality by the College Centre for Quality Improvement (CCQI) (Royal College of Psychiatrists, 2019). However, despite this, there remains a lack of psychology provision in CRHTTs across the UK (Lloyd-Evans et al., 2019) and significant barriers to access (Ebrahim, 2021). Moreover, studies evaluating brief interventions delivered in these settings are surprisingly sparse (Sjolie, Karlsson & Kim, 2010).

Recently, the unprecedented impact of Coronavirus disease 2019 (COVID-19) has led to an understandable increase in both distress (Daly & Robinson, 2021) and crisis presentations (Campion, Javed, Sartorius & Marmot, 2020). This has not only increased demand for mental health services (Cullen, Gulati & Kelly, 2020), but has forced them to transform and offer more services remotely, for example, via telephone (Zhou et al., 2020; Pereira-Sanchez et al., 2020).

Psychological interventions have been delivered by telephone since the mid-1970’s (Lester, 1974). Despite concerns regarding its potential impact on therapeutic alliance (Richards et al., 2006), itself a key determinant of positive experiences of CRHTTs (Middleton, Shaw, Collier, Purser & Ferguson, 2011) and treatment outcomes more generally (Martin, Garske & Davis, 2000), research has suggested that telephone interventions are commensurate in effectiveness and satisfactoriness to those delivered face-to-face (Mulligan et al., 2014; Rushton et al., 2019; Irvine et al., 2020). Indeed, there is evidence that brief interventions delivered via telephone can lead to improved outcomes for those presenting in crisis (Bidargaddi et al., 2015). However, it is noteworthy that service users often choose face-to-face over telephonic interventions (Rushton et al., 2020) and that preferences play a significant role in therapy attrition and outcomes (Swift, Callahan, Cooper & Parkin, 2018).

With the ongoing impact of COVID-19, there remains a need for CRHTTs to develop brief, accessible, psychologically informed interventions that are amenable to both face-to-face and telephonic delivery. By adapting and manualising an existing intervention utilised across several CRHTTs in the North West of England, one CRHTT in Greater Manchester recently took on this challenge to service provision. They adapted a brief, skills-based intervention, ‘The Crisis Toolbox’ (CTB) to help service users develop coping skills to manage mental health crises. These skills were drawn from existing evidence-based psychological interventions and were formulation driven (i.e., selected based on the needs of each individual service user). The CTB incorporated nine skills pertinent to crises management, including distress tolerance, problem solving, the STOPP skill, distraction techniques, self-soothing (Linehan, 2015), surf the urge (Marlatt & Donovon, 2005), sleep hygiene (Hauri, 1991), grounding (Lowen, 1958) and worry management (Butler & Hope, 2007).

Since its implementation in March 2020, and prior to this evaluation, 395 service users had accessed the CTB. Anecdotal evidence derived from service user feedback had been positive and suggested the CTB was acceptable; however, there has been no systematic evaluation of satisfaction with the CTB since its inception. Client satisfaction with services has long been recognised as an important marker of service quality (Blumenthal, 1996) and evaluations of such have become increasingly common place given the link between satisfaction and better mental health outcomes (Priebe & Miglietta, 2019). Therefore, we aimed to evaluate client satisfaction with the CTB to ascertain its current acceptability and inform its development.

Methods

Design

A mixed methods approach comprising a questionnaire and semi-structured interview was used to evaluate the CTB.

Intervention

The CTB.

The CTB consisted of up to three sessions of brief, psychologically informed skills and was routinely offered in the CRHTT during COVID-19 and beyond. The Service Operational Policy (SOP) identified that all service-users under the CRHTT should be considered for the CTB. The aim of the CTB was to provide service-users with a range of coping strategies to utilise when experiencing crisis. Nine skills pertinent to crisis management were included in the CTB, including: (1) Distress tolerance; (2) Problem solving; (3) The STOPP skill; (4) Surf the urge; (5) Distraction techniques; (6) Sleep hygiene; (7) Self-soothing; (8) Grounding; and (9) Worry management.

To ensure consistency, delivery of the CTB was designed and manualised so that different skills could be utilised depending on an individual’s presenting difficulties. The practitioner delivering CTB sessions used their clinical judgment and supervision to inform such decisions. Although the CTB was designed as a staff-wide intervention, amenable to both face-to-face and telephone delivery, in this evaluation, all CTB sessions were conducted by telephone by either an Assistant Psychologist (AP) or Trainee Associate Psychological Practitioner (TAPP) under the supervision of a Clinical Psychologist.

Procedure

Sampling and recruitment.

Opportunistic sampling was used to recruit participants from one CRHTT in Greater Manchester. All service-users who were referred to the CTB between February and May 2021 and who had attended at least one session (out of three offered) were invited to complete the CTBSQ and semi-structured interview.

Ethical considerations

Ethical approval

was not required as the study evaluated an existing intervention. The study was registered as a service evaluation with a local NHS Trust Research and Development Department. Prior to involvement, all participants were given information about the study, advised of confidentiality, the anonymity of their responses and of the availability of support should any questions cause distress. All participants gave informed consent to participate and were informed of their right to withdraw their participation or data at any time.

Materials

The Crisis Toolbox Satisfaction Questionnaire (CTBSQ) was developed by the research team and was used to measure satisfaction. The research team met to generate initial ideas for questionnaire items and domains. LM developed a pilot version of the CTBSQ and circulated this amongst the research team for feedback and refinement of item wordings. The final CTBSQ had good face validity.

The CTBSQ consisted of 10 closed ended and one open-ended question. Each item allowed individuals to discuss their experience of the CTB and allowed the research team to evaluate several facets of acceptability. Each closed question was scored using a 5-point Likert scale ranging from ‘strongly disagree’ to ‘strongly agree’. Participants were also asked one open-ended question: ‘How could the Toolbox be improved?’. Although there were no other open-ended questions, to capture as much feedback as possible, additional qualitative feedback received during administration of the CTBSQ was also recorded and included in the analysis. Please see Fig. 1. for the full questionnaire and interview schedule.

Fig. 1

The Crisis Toolbox Satisfaction Questionnaire (CTBSQ) and Interview Schedule

Three APs (KM, RR, DV) and one TAPP (MJ) delivered the CTB sessions. All participants were informed of the service evaluation during their first session and were asked if they wished to participate. Upon completion of their CTB sessions, participants who consented were contacted by an AP (KM) or TAPP (MJ) to confirm consent. To minimise desirability bias, all participants were contacted by a different AP or TAPP to those who completed their sessions. Semi-structured interviews were conducted via telephone and participants’ responses were recorded on paper and transcribed into an Excel database. Each interview lasted between fifteen and thirty minutes.

Analysis

Quantitative analysis.

Descriptive statistics (Mean / SD) were used to quantify participant endorsement of individual CTBSQ items, and percentages were calculated to indicate overall levels of satisfaction with the CTB.

Qualitative analysis.

Qualitative data was analysed by one TAPP (MJ) and two Clinical Psychologists (SN and LM). The analysis followed the six phases of empirical thematic analysis (Braun & Clarke, 2006). MJ undertook the initial coding of the data, which involved a line-by-line analysis of each participants’ responses, followed by a grouping of similar features and patterns within the data set. The final codes were refined, reviewed, and agreed upon by SN and LM to reduce bias and improve reliability. Through group discussion, it was considered how different codes meaningfully combined to form potential themes. These initial themes were then reviewed at a code level, considering how the themes fit for the entire data set to ensure their relevance, and further refined to form four main themes. These themes were identified at a latent level, and both inductive and theoretical approaches were used during thematic analysis.

Results

Demographics

A total of 58 participants provided feedback on the CTB (100% response rate). Of the 58 who provided feedback, 52 completed three sessions of the CTB (89.7%) and six completed two sessions (10.3%). Twenty-seven participants were male (46.6%) and 31 were female (53.4%). The CTB was delivered within an average of 23.3 days (SD = 9.7).

Quantitative analyses

Scores on the CTBSQ indicated high levels of satisfaction across all areas of the CTB. Mean scores for individual items ranged from 3.47 to 4.78 (1 = Strongly disagree and 5 = Strongly agree), (see Table 1). Participants agreed most with the statements ‘I felt listened to during my sessions’ and ‘I would recommend the Toolbox to others’. They agreed least with the statement ‘The number of sessions felt right’ (see Table 2).

Table 1

Mean and standard deviation of participant endorsement of individual CTBSQ items (1=Strongly disagree, 5=Strongly agree)

	Mean	Standard deviation
‘I felt listened to during my sessions’	4.76	0.43
‘I felt understood during my sessions’	4.60	0.62
‘The Toolbox met my expectations’	4.16	0.70
‘The content of my sessions was relevant to my needs’	4.36	0.87
‘The number of sessions felt right’	3.47	1.10
‘I have learned new skills to manage my distress’	4.41	0.73
‘I feel more confident in managing my distress’	3.83	0.86
‘I encountered no difficulties with the Toolbox’	4.45	0.65
‘The Toolbox has helped in my recovery’	4.02	0.93
‘I would recommend the Toolbox to others’	4.78	0.42

Table 2

Percentage of combined participant endorsement of individual CTBSQ items

	Strongly Agree (%)	Agree (%)	Neither agree nor disagree (%)	Disagree (%)	Strongly Disagree (%)
‘I felt listened to during my sessions’	75.86%	24.14%	0.00%	0.00%	0.00%
‘I felt understood during my sessions’	65.52%	31.04%	1.72%	1.72%	0.00%
‘The Toolbox met my expectations’	29.31%	60.34%	6.90%	3.45%	0.00%
The content of my sessions was relevant to my needs’	51.72%	39.66%	5.17%	0.00%	3.45%
‘The number of sessions felt right’	15.52%	44.83%	12.07%	25.86%	1.72%
‘I have learned new skills to manage my distress’	51.72%	41.38%	3.45%	3.45%	0.00%
‘I feel more confident in managing my distress’	18.97%	55.17%	15.52%	10.34%	0.00%
‘I encountered no difficulties with the Toolbox’	51.72%	43.11%	3.45%	1.72%	0.00%
‘The Toolbox has helped in my recovery’	29.31%	53.45%	10.34%	3.45%	3.45%
‘I would recommend the Toolbox to others’	77.59%	22.41%	0.00%	0.00%	0.00%

Qualitative analysis

Four main qualitative themes consisting of 11 sub-themes were identified using an empirical thematic analysis (Braun & Clarke, 2006). A description of each theme is outlined below:

1) Active ingredients of the CTB.

Where participants experienced the CTB as helpful, they identified several key features they believed contributed to its effectiveness.

The importance of tailoring interventions.

Some participants highlighted the value of learning skills that were adapted to their specific difficulties, suggesting a tailored, formulation driven, approach was important for them, “I found the skills helpful and felt they were tailored to my needs”.

Opportunities to acquire knowledge and skills.

Participants valued the opportunity to acquire new skills during their CTB sessions, which enabled them to gain new knowledge and confidence as they practiced, “I learnt new things which I hadn’t heard of or tried before”, “I have learnt other ways of trying to cope …. When I put the skills into practice, they were really helpful”.

The generalisable effects of the CTB.

One participant reported that they had been able to share the techniques with others who had similarly found them helpful, suggesting the positive effects of the CTB could be generalisable “I have been attending a group and a few of us have completed the CTB. We have all said how beneficial it was and have been sharing some of the skills with others”.

2) The therapeutic relationship.

All participants gave positive comments regarding the therapeutic relationship they formed during their CTB sessions and outlined some determinants of this.

Consistency.

Many participants highlighted how consistency was important for them to establish a positive therapeutic relationship, “I liked that I was talking to the same person, so I was able to build a rapport with them”.

Congruence.

Participants commented on the benefit of having someone to work with who seemed interested, curious and who was experienced as genuine, “It was nice to have someone contact me who seemed more concerned and didn’t just ask the same questions over and over about whether I could keep myself safe”.

Unconditional positive regard.

Participants spoke of the value of having space to talk in their CTB sessions, as well as the positive impact of having someone listen without judgement, “I was given time to talk and really benefited from it’”, “It was nice to have someone take the time to listen to you without judgement”.

3) Service-user preferences.

All participants reported preferences based on their experience of the CTB. This theme outlined these in terms of the delivery and duration of the CTB, and the suggestions made for its further development and improvement.

Mode of delivery.

Whilst participants acknowledged and understood the restrictions COVID-19 placed on face-to-face delivery of the CTB, some did communicate this would have been their preference if given the choice, “I know it had to be done remotely, but I would have preferred face-to-face sessions instead”.

Number of sessions.

Many participants stated that more sessions of CTB would have been helpful. Reasons for this varied and appeared idiosyncratic in nature. Some participants suggested more sessions would be needed for those who had no previous experience using skills, “Maybe more sessions for others who are new to the skills, I had covered some of the techniques before, so the number of sessions felt right, but others may need more”. Other participants felt their sessions ended prematurely, “A couple more sessions may have been helpful as I felt I was really getting somewhere with it as the sessions ended”. Participants offered additional suggestions, including the potential for check-in calls between sessions, “Brief check-in calls between sessions would be helpful to prompt practice”. Another identified that a follow-up session would be beneficial, “Maybe have a follow-up session 1–2 weeks later to see how we’re getting on with the skills?”. Others commented on the value of spacing between sessions, “The sessions were well spaced out and gave time to practice the skills in between”.

Content of sessions.

Almost all participants commented positively on the content of their CTB sessions. Some participants found the CTB most useful in its focus on present-day difficulties and problem solving, “I liked that it wasn’t in-depth psychological therapy, but focused more on current problems and how to fix them”. Other participants highlighted specific skills they found helpful or commented on how well their session time was used, “The sessions were very thorough and covered a lot. They gave me recommendations going forward”, “The worry postponement really helped!”. Where participants felt that certain skills covered during their sessions were not helpful for them, they considered how they might be helpful for others, “Some of the skills I didn’t feel were relevant to me, like self-soothing”, “It wasn’t helpful for me, but it might be helpful for others”. Some participants offered practical advice to help others feel less overwhelmed during the sessions, “It might be useful to send GAD/PHQ questions so we have these in front of us and can remember the answers. It would take some of the stress away”, “The resources were too much to take in at first. You could send specific skills after the sessions”. Other words used to describe the content of CTB included “calming”, “thorough” and “helpful”.

4) Expectations and continuity of care.

This theme outlined the importance of managing service user expectations during the initial referral for CTB sessions and the need for continuity of care upon its completion.

Expectations of the CTB.

Some participants reported holding different expectations of the CTB prior to commencement. Some stated they expected the CTB to be less directive, “I expected the sessions to be more like Counselling (talking and listening)”, whereas others reported a lack of expectation, “I didn’t know what to expect initially”.

Continuity of care.

Many participants spoke about the lack of contact they received from the CRHTT following CTB sessions and how this contributed to unhelpful uncertainty. One participant described this experience as being “left in limbo”, whilst others emphasised the importance of clarity in their care in terms of forward planning, “More continuous care from the team would be helpful. There was little contact after sessions, so I was unsure of next steps”.

Discussion

Summary of results.

This service evaluation aimed to determine the acceptability of the CTB, a brief, skills-based, intervention, delivered in one CRHTT during the COVID-19 pandemic. Fifty-eight participants who received the CTB completed a researcher designed questionnaire of satisfaction (CTBSQ) and results were analysed using descriptive statistics and thematic analyses.

The CTB received high scores across all items of the CTBSQ. Mean scores out of five ranged from 3.47 to 4.78 and the uniformity suggested the CTB was highly acceptable. Notably, the most endorsed items were ‘I felt listened to during my sessions’, ‘I felt understood during my sessions’ and ‘I would recommend the toolbox to others’ whereas the lesser endorsed items were ‘The number of sessions felt right’ and ‘I feel more confident in managing my distress’. These results were supported by four qualitative themes derived from thematic analyses, which highlighted participant experiences of the CTB. These included ‘Active ingredients of the CTB’, ‘The therapeutic relationship’, ‘Service-user preferences’ and ‘Expectations and continuity of care’. These findings are discussed in relation to previous research, future adaptations of the CTB, limitations of the evaluation and ideas for future research.

Almost all participants commented positively on the content of their CTB sessions. This suggests that the skills included in the CTB were acceptable to those presenting in crisis. This is not too surprising as CTB skills were derived from psychological approaches, including dialectical, cognitive, and behavioural therapies that have demonstrable efficacy for those at risk of self-harm and/or experiencing suicidal thoughts (Yardley et al., 2019; McCabe et al., 2018). However, the current findings also validate the acceptability and value of skills-based interventions delivered via telephone for this client group (Bidargaddi et al., 2015; Mulligan et al., 2014; Rushton et al., 2019; Irvine et al., 2020). This is an important finding as telephone interventions can increase access to psychological interventions (Irvine et al., 2020), which remains a significant challenge for CRHTT service provision and a frequent cause of patient dissatisfaction (Lloyd-Evans, et al., 2019; Chilman et al., 2021).

Participants also valued skills that were tailored to their needs. This suggests a formulation driven approach to delivery was key to ensure acceptability. The CTB sessions in this evaluation were delivered by AP’s or a TAPP supervised by a Clinical Psychologist. As the CTB was designed as a staff-wide intervention, this might have implications for its delivery from practitioners not trained in formulation nor supervised by a Clinical Psychologist.

Notably, participants most strongly highlighted the importance of the therapeutic relationship during their CTB sessions. In particular, the value of consistency, congruence and unconditional positive regard was reported by most participants, which closely resemble the Rogerian concepts of core conditions of therapy (Rogers, 1957). Therefore, akin to previous research examining other interventions (Lambert & Barley, 2001; Martin et al., 2000), a positive therapeutic relationship might be of fundamental importance for successful delivery of the CTB. These findings also resonate with previous research that has highlighted the importance of therapeutic relationships between staff and service users as a moderator of positive CRHTT outcomes (e.g., Morant et al., 2017; Middleton et al., 2011).

Future adaptations of the CTB.

Although this service evaluation confirmed the acceptability of the CTB in its current form, there are several adaptations services could make to increase this further.

Firstly, some participants identified that although telephone CTB was acceptable, their preference would have been for face-to-face intervention. As COVID-19 restrictions lift and services gradually return to normal, service users should be given the choice of telephone or face-to-face CTB, as a preference-based mode of delivery will likely ensure its acceptability. Secondly, some participants identified occasions where their expectations prior to commencing the CTB were not managed and described feelings of uncertainty upon completion due to a lack of continuity of care. Lack of continuity within CRHTT settings has been associated with poorer staff / patient relationships (Titheradge & Galea, 2019), which participants highlighted as a core feature of the CTB. Services could safeguard patient expectations through the creation and provision of CTB information leaflets and ensure an MDT plan is created and protocolised for all completing the CTB to maximise continuity of care within the CRHTT and beyond. Thirdly, many participants expressed a desire for more sessions. Services could consider extending this offer; however, offering more sessions could pose a challenge to service provision and result in the accrual of waiting lists and/or delays to discharge. Given the well documented issues with access to psychological interventions within CRHTT settings (Ebrahim, 2021), one solution could be the investment in staff training to increase psychological skills and thinking across the workforce and make the CTB a core component of every practitioner’s role.

Limitations.

Although the CTB was highly acceptable, there are some caveats to consider. Firstly, this evaluation included a relatively small sample size recruited from one CRHTT. Therefore, the generalisability of the outcomes to other teams remains unclear. Secondly, although the CTB was developed for both telephonic and face-to-face delivery, the current study only assessed acceptability of the former. Although qualitative results suggested face-to-face delivery might have been the preference for some, and therefore of at least comparable satisfactoriness, the acceptability of this mode of delivery remains speculative. Thirdly, as no clinical information was recorded for any participants (e.g., diagnosis), it is unclear whether the CTB is acceptable for all clinical groups. Lastly, although the CTB was designed as a staff-wide intervention, in this evaluation, the CTB was delivered exclusively by APs and a TAPP due to capacity issues across the team. It would be important for clinical teams to problem solve barriers with staffing resource to ensure the CTB can be delivered by all staff as this would increase its accessibility and validate its acceptability across all staff groups.

Conclusions

The CTB was a highly valued and acceptable intervention for those accessing CRHTT settings during the COVID-19 pandemic. To our knowledge this is one of the first studies to examine the acceptability of a brief intervention amenable to telephonic and face-to-face delivery in these settings in the UK. The results presented here are promising and suggest the CTB could improve access to psychological interventions in urgent care and be a vehicle to improve the standards of CRHTTs nationally. However, further research is now needed to assess the generalisability of the CTB, its clinical effectiveness and its effect on long term operational indicators, such as frequency of accident and emergency presentations, number of inpatient admissions, length of inpatient stay and rates of re-admission.