Children and Young People-Mental Health Safety Assessment Tool (CYP-MH SAT) study: Protocol for the development and psychometric evaluation of an assessment tool to identify immediate risk of self-harm and suicide in children and young people (10-19 years) in acute paediatric hospital settings.

INTRODUCTION: Currently, no standardised, evidence-based assessment tool for assessing immediate self-harm and suicide in acute paediatric inpatient settings exists. AIM: The aim of this study is to develop and test the psychometric properties of an assessment tool that identifies immediate risk of self-harm and suicide in children and young people (10-19 years) in acute paediatric hospital settings. METHODS AND ANALYSIS: Development phase: This phase involved a scoping review of the literature to identify and extract items from previously published suicide and self-harm risk assessment scales. Using a modified electronic Delphi approach, these items will then be rated according to their relevance for assessment of immediate suicide or self-harm risk by expert professionals. Inclusion of items will be determined by 65%-70% consensus between raters. Subsequently, a panel of expert members will convene to determine the face validity, appropriate phrasing, item order and response format for the finalised items.Psychometric testing phase: The finalised items will be tested for validity and reliability through a multicentre, psychometric evaluation. Psychometric testing will be undertaken to determine the following: internal consistency, inter-rater reliability, convergent, divergent validity and concurrent validity. ETHICS AND DISSEMINATION: Ethical approval was provided by the National Health Service East Midlands-Derby Research Ethics Committee (17/EM/0347) and full governance clearance received by the Health Research Authority and local participating sites. Findings from this study will be disseminated to professionals and the public via peer-reviewed journal publications, popular social media and conference presentations. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

To our knowledge, this is the first UK-based study to develop an assessment tool to ascertain immediate risk of suicide and self-harm in children and young people presenting to acute paediatric hospital settings in mental health crisis.

This study seeks to address the limitations of established assessment tools of immediate risk of suicide and self-harm in children and young people, in adherence to published psychometric evaluation guidelines.

This protocol is tailored towards an English-speaking population, therefore, further national and international testing and adaptations may be required to achieve wider generalisability.

An opportunistic sample of self-selected experts will be used to inform the construction of the suicide and self-harm risk assessment.

They may however have more motivation to take part than other populations and/or possess preconceived investments leading them to respond in a particular way.

Introduction

Mental health (MH) crisis has been defined as ‘an acute disruption of psychological homeostasis in which one’s usual coping mechanisms fail and there is distress and functional impairment’.1 This may include: extreme anxiety or panic attacks; psychotic episodes (including delusions, hallucinations, paranoia and/or hearing voices); hypomania or mania; other behaviours that feel out of control and acts of suicide or self-harm.2 MH crisis is recognised as a psychiatric emergency as it is ‘an acute disturbance of thought, mood, behaviour or social relationship that requires immediate intervention’.3 It is estimated to be the primary cause of around 5% of emergency department (ED) attendances4 with the most prevalent presenting conditions being self-harm and/or suicidal behaviours. For these conditions alone, data for children and young people (CYP) aged 10–19 years in England indicates the prevalence of suicide is 4.3 per 100 0005 and self-harm is 435.95 per 100 000.6

Care pathway for CYP in MH crisis

Nationally the care pathway for CYP in MH crisis varies considerably. The majority of acute and initial care is delivered by non-MH professionals in general healthcare settings,7 such as paediatricians and registered children’s nurses. Whereas specialist mental healthcare is delivered by professionals (including psychiatrists, nurses, social workers and psychologist) with specialist training, skills and knowledge in working with CYP with MH difficulties.8

For CYP, initial triage and care is delivered by paediatricians and registered children’s nurses working in acute paediatric settings,9 including EDs and paediatric inpatient wards. The main focus of this initial part of the CYP care pathway is to ensure that they are appropriately assessed to direct a plan of care and to ensure that they are safe until definitive and expert MH assessment is undertaken.

Professionals who provide care for CYP experiencing MH crisis identify, it can be one of the most complex and stressful duties undertaken in practice.10 This could be attributed to the acuity of the child’s status, as crisis is defined as a psychiatric emergency. However, health professionals’ responses could also be governed by the perception that poor experience and outcome of this acute phase of the CYP care pathway may negatively impact on their adherence with follow-up and future MH. In turn, there is accumulating evidence that non-adherence to follow-up is a predictor of poor outcomes, in terms of repeated self-harm and suicide but also in a variety of other psychosocial outcomes.

The assessment of CYP experiencing a MH crisis should involve: the identification of the main clinical and demographic features known to be associated with their MH crisis and identification of the key psychological characteristics associated with risk, in particular depression, hopelessness and continuing suicidal intent.11 In addition, any immediate physical health needs (such as wound care, use of antidotes, gastric irrigation and psychotropic medication should be addressed)12; and their safety should be maintained until expert assessment is undertaken by specialist MH professionals. However, in order to implement a plan of care where attempts can be made to mitigate immediate risks, health professionals (such as paediatricians and children’s nurses) need to be supported to make an informed assessment. Invariably this will have to be made in time limited circumstances and with CYP with potentially changing MH status. Therefore, the assessment should focus on identifying the most pertinent risks (ie, immediate risk of self-harm or suicide) and take into account risk factors, coping abilities and assessment of lethality of previous suicidal and self-harm behaviour.13 14 Such factors can be useful to differentiate between high-risk and low-risk suicidal and self-harm behaviours.15

Assessing risk of self-harm and suicidal intent

Evidence suggests that risk assessments are no more accurate at predicting risk than expert specialist MH professional clinical judgement in non-acute psychiatric outpatients.16 However, acute paediatric care settings present specific differences in utility, focus and context that make the application of an assessment of suicide and self-harm unique.7 For example, the assessment is usually made by non-MH experts who may lack specialist knowledge and experience to inform clinical decisions. Furthermore, assessments are focused on immediate risks of self-harm or suicide during receipt of acute paediatric care; not projected to weeks, months or years. Additionally, assessments are performed in time limited circumstances with CYP with potentially dynamic and fluctuating MH. Therefore, to enable implementation of a plan of care where immediate risks can be mitigated, healthcare professionals require appropriate support and guidance to inform their assessment. In current practice, risk assessments of CYP suicidal or self-harm behaviour in paediatric inpatient departments are typically undertaken through a variety of non-validated tools that are often created by the specific department, and then generally completed by non-MH clinicians.17 However, some acute paediatric departments do use previously validated suicide risk assessment tools, but these tools have not been validated in populations presenting to acute paediatric care, very rarely assess immediate suicide risk and rarely include items on self-harm.

As such, across the UK, there is currently no standardised, specifically designed risk assessment tool being used within paediatric inpatient settings that assesses the immediate risk of self-harm or suicide for CYP experiencing MH crisis.

The aim of this study is to develop and provide an initial psychometric evaluation of an evidence-based tool to assess the immediate safety of CYP (aged 10–19 years) in MH crisis presenting to acute paediatric inpatient hospital settings.

To identify relevant evidence-based self-harm and suicide risk assessment tools from published literature.

To identify, through expert consensus, the most pertinent items for inclusion in a CYP-Mental Health Safety Assessment Tool (CYP-MH SAT) to assess immediate risk of self-harm and suicide in CYP experiencing MH crisis.

To establish construct validity of a CYP-MH SAT through agreement from an expert panel.

To establish the psychometric properties of CYP-MH SAT in CYP presenting in MH crisis to the ED and inpatient wards in acute paediatric hospital settings.

Methods

This study incorporates three phases as outlined in figure 1:

Figure 1

Children and Young People (CYP)-Mental Health Safety Assessment Tool study flow chart. C-SSRS, Columbia suicide severity rating scale.

Phase 1: scoping review of existing evidence

The possible domains for the development of a CYP-MH SAT to be used by non-expert MH professionals in an acute paediatric setting were identified from clinician feedback and themes identified from the investigation and root cause analysis of the high-level incidents at a local National Health Service (NHS) Trust. From this, the immediate risk of (1) self-harm and (2) suicidal behaviour were identified as the most important factors to be assessed by the tool.

Subsequently, we conducted a scoping review of published literature to identify evidence-based items used to assess risk of self-harm and/or suicide in CYP in acute inpatient settings. From this review, 22 assessment tools were identified, of which 12 were accessible and retrieved.18–29

Importantly, none of 22 assessment tools identified in this scoping review were specifically developed for and tested on a population accessing paediatric inpatient care in MH crisis. Thus, all items from the 12 accessible assessment tools were collated and grouped into thematically similar subject headings using framework analysis.30 Following removal of duplicate items and collapsing of similar items, a total of 93 items were identified (themes, subthemes and items are outlined in online supplementary table 1). These items identified from the literature will now be used to inform the preliminary round of the Delphi survey, proposed to be undertaken in phase 2a.

Phase 2a: Delphi survey

The purpose of this phase is to gain consensus of the relevance of items identified from the scoping review (phase 1) using the Delphi method. The Delphi method involves successive questioning with a panel of experts and key stakeholders to develop consensus regarding a particular issue.31 For this study, experts and key stakeholders will be requested to rank the items according to their perceived relevance. Expertise will be defined as having experience of delivering care to CYP in MH crisis or having expertise in CYP MH. In order to canvass the views of a range of experts and key stakeholders, no stipulation will be made as to how long they had been working in this field. However, information pertaining to the characteristics of participants was collected to explore and contextualise the results.

To form the panel of experts and key stakeholders, we will aim to recruit a purposive sample of 20–30 professionals including: Psychiatrists, Paediatricians, Nurses, Clinical Support Workers, Academics, Public and Patient Involvement representatives. Participants will be recruited from three NHS Trusts and two non-NHS organisations (Higher Education Institution and Local council). We would like opinions from a range of differing staff types/grades and aim to recruit equal proportions of each staff type to prevent over-representation from one particular view point possibly shared by a group of individuals sharing the same characteristic (such as job type).32

Staff will be invited to take part by the site Local Collaborator via an email containing a participant information sheet, instructions for questionnaire completion and a link to a modified electronic Delphi survey (Bristol Online Survey: BOS software). Staff will be informed that completion of the survey indicates consent to take part.

The survey will consist of two sections: (1) demographics: email address, job title, staff grade, employing organisation, gender, date of birth and ethnicity; and (2) the list of items generated from phase 1 (see online supplementary table 1). Participants will be asked to rate each item in relation to its relevance in assessing immediate risk of self-harm and/or suicide for CYP presenting in MH crisis to acute paediatric hospital settings. Relevance to the assessment of immediate risk of suicide and/or self-harm for CYP in MH crisis in acute paediatric inpatient settings will be reported on a 4-point Likert scale (1-not relevant, 2-somewhat relevant, 3-quite relevant, 4-highly relevant)33 using a closed questioning procedure. The survey also allows participants to type any items they think should be included using open-ended questioning. The survey will take no longer than 15 minutes to complete and participants have 2 weeks to complete it.34 Staff will receive an email reminder 1 week before the return due date. Newly identified items from open-ended questioning will be included in the second iteration of the questionnaire to decipher consensus on the new items. Items will be removed if they do not achieve the prespecified consensus margin (≥65%–70%; median ≥3.25).35 Only items within the prespecific consensus margin will be redistributed in successive Delphi survey rounds. The survey will be distributed in three rounds to achieve consensus.36

Phase 2b: nominal group technique

The nominal group technique is widely used for problem identification, solution generation and decision-making37 and can be used in groups of varying sizes.38 In this phase, respondents from phase 2a will be invited to attend a 2-hour meeting and will be asked to provide their opinions regarding face validity of the CYP-MH SAT.39 Group facilitators (JCM, GMW and TC) will facilitate discussions regarding any required changes to the tool including item phrasing, attribution of suitable rating scales and question order. When disagreements occur, opinions with the highest consensus in the group will be prioritised. The CYP-MH SAT will be considered finalised for further psychometric testing once the changes subject to expert opinion have been implemented.

Phase 3: psychometric evaluative study

The study is a multicentre, psychometric evaluation study.

Eligibility and recruitment

CYP ‘cases’ and ‘non-cases’ will be recruited from acute paediatric hospital settings (EDs and wards) across three NHS trusts in the East Midlands, UK. CYP will be considered eligible for recruitment based on the criteria outlined in box 1.

Aged 10–19 years.

Approved National Health Service Trust site.

Cases: Presenting to acute paediatric hospital setting in mental health crisis as defined as: having reached ‘breaking point’, likely to harm themselves or others and behaviours that feel out of control, including: extreme anxiety or panic attacks; psychotic episodes (including delusions, hallucinations, paranoia and/or hearing voices); hypomania or mania and acts of suicide or self-harm; those who have engaged in non-suicidal self-injury or low-lethality self-harm.

Non-cases: Presenting to acute paediatric setting not in mental health crisis but presenting with a primary physical medical illness or injury. This is defined as a health problem having physical origins, which can be short term (eg, accidental injury, influenza, migraine, infections) or long term (eg, diabetes, asthma, arthritis).

Mental capacity to consent.

Parental/legal guardian consent and child assent (under 16 years old).

Young person consent (over 16 years).

Unable to speak or comprehend English language.

Currently receiving active end-of-life care or considered too medically ill by clinicians.

Non-cases: Current mental health crisis.

Registered clinical staff will make the initial approach to participants. The informed consent process will then be performed by trained registered clinical staff or research team members.

Sample size and justification

Based on needing 4–10 participants per questionnaire item in order to conduct the psychometric testing procedures,40 100–200 cases and 100–200 non-cases are required. We envisage the CYP-MH SAT that will be evaluated in phase 3 will not exceed 20 items.

Measures/outcomes

Two measures will be used in this study:

The CYP-MH SAT: This measure is being developed as part of this study (phases 1 and 2) and aims to assess immediate risk of self-harm and immediate risk of suicidality for CYP (aged 10–19 years) presenting to acute paediatric care. The total number of items and face validity will be established on completion of phase 2b of this study. The CYP-MH SAT will be completed with both cases and non-cases.

The Columbia Suicide Severity Rating Scale (C-SSRS): is a clinician rated, 19-item, measure of suicide risk.41 The C-SSRS is considered the current gold-standard assessment for suicide risk.42 The C-SSRS has good predictive validity, discriminant validity, convergent validity, and sensitivity and specificity.43 44 The C-SSRS will be completed with cases only to enable assessment of the level of agreement (convergent validity) with the CYP-MH SAT.

In addition to the measures, demographic details (date of birth, gender, ethnicity, NHS site, site setting, hospital number), primary reason for presentation to hospital (International Classification of Diseases-10 categories), MH crisis hospital presentation in the past 6 months and treating physician contact details will also be collected. All data will be collected by registered clinical staff members located in EDs and paediatric wards across three sites. All registered clinical staff members who take part in the study will have received CYP-MH SAT training. The specific training required to administer the tool will be dependent on its content and format that will be determined during the development of this study. The study steering group (composed of clinical experts) will provide guidance on the development of training resources. It is anticipated that a minimum of five people per clinical area per site (n=circa 30) registered clinical staff members will administer the tool. We have processes in place to ensure that data will only be collected once from participants during their inpatient admission. However, if they are admitted again during the data collection period and consent to participate in the study then they will be counted as a new participant. Data will be collected from October 2017 to May 2018 (as shown in study timeline, figure 2), with recruitment ending earlier if sample size achieved.

Figure 2

Study timeline.

Patient and public involvement

The inception and design of this study has been informed through meaningful engagement with patient, public and expert professionals. CYP, their parents and carers, and professionals from both acute paediatric care and child and adolescent mental health services have informed the topic and focus of the study through previous stakeholder engagement led by JCM and funded by NHS England (link to full report: http://eprints.nottingham.ac.uk/35284/1/Manningetal2015PCOMNHSEnglandFinal.pdf). In the development of this study, we have involved CYP to inform the focus, design and development of study information such as the participant information sheets.

We will continually refer to guidance on involving the public in research can be on the INVOLVE website (see: http://www.invo.org.uk/). We have referred to the British Council ‘Learn English’ for advice regarding writing for lay audiences and specific topic areas (see: https://learnenglish.britishcouncil.org/en/writing) and the involve resource page (see: http://www.invo.org.uk/makeitclear/support-and-resources/). These resources have and will be fully used in the construction of study documents, procedures and dissemination activities.

In addition, a group of clinical and research experts have been assembled to steer the direction of this study. This includes staff members who provide care for CYP experiencing MH problems and crisis events on a daily basis in a range of settings that include NHS, Local Government and charity organisations.

Analyses

Data will be analysed using IBM SPSS (version 22) using statistical techniques appropriate for cross-sectional research. For all study phases, counts (n) and proportions (%) will be calculated for categorical variables and descriptive statistics (mean, median, SD, range, IQR) for continuous variables.

Phase 2a/b

Mixed-methods analyses will be employed. Consensus agreement will be calculated using counts (n) and proportions (%) for a median relevance rating of 3.25 or above. Item content validity ratios (I-CVR)45 and Content Validity Indices (I/S-CVI)46 will be displayed as a decimal >0 and <1 (see figure 3). The CVR is a linear transformation of a proportional level of agreement on how many ‘experts’ within a panel rate an item as being ‘highly relevant’.47 A CVI demonstrates the proportion of staff ‘in agreement’ about the relevance (scoring either 3 or 4) of individual items.46 Field notes will be presented in narrative form48 to describe consensually agreed decision processes during the nominal group meeting.

Figure 3

Content validity equations. I-CVR, Item content validity ratio; I/S-CVI, content validity indices.

Phase 3

A standard psychometric approach49 50 will be taken to test the psychometric properties (ie, the reliability and validity) of the CYP-MH SAT. The factor structure will be analysed through exploratory factor analysis to explore the most parsimonious factor structure of the scale. Equivalence reliability will be assessed through Cronbach alpha (α). Concurrent validity will be determined through Pearson’s correlation coefficients (or Spearman’s rank for non-parametric data) between the CYP-MH SAT and the C-SSRS. Inter-rater reliability will be established through Kappa coefficients (K) between CYP-MH SAT and inter-rater version scores from Case data (ie, two healthcare professionals will complete the assessment for each CYP to allow for a comparison between assessor scores). Convergent, divergent and discriminant validity will be determined by InterItem Correlations within and between assessment items and subscales. In order to assess convergent validity, we will correlate total scale scores to explore the relationship between the CYP-MH SAT and the C-SSRS. Discriminant/divergent validity will be ascertained through the correlation of case and non-case total scores where we would expect the scale to be able to differentiate between those presenting at the ED and children’s inpatient wards in MH crisis and those presenting at the ED and children’s inpatient wards not in MH crisis. In the case of non-normally distributed data, equivalent non-parametric analyses will be undertaken.

Ethics and dissemination

Ethical review

The study is sponsored by the Nottingham University Hospitals NHS Trust, Research and Innovation department and funded by Nottinghamshire Clinical Commissioning Group. Neither the sponsor nor the funders will be involved in the analysis of study data or report writing.

The study did not commence prior to any approval mechanisms from the Sponsor Health Research Authority/Research Ethics Committee and Research and Development authorities. All study procedures will be conducted in line with Good Clinical Practice and NHS research ethical guidance.

Informed consent

All participants will be fully informed about the study prior to any participation in research activities in the form of a patient information sheet and have the opportunity to ask questions throughout the study.

For phases 2a and 2b of the study, staff consent will be implied through electronic completion of the modified e-Delphi questionnaire (phase 2a) and through staff attendance at the modified nominal group technique meeting (phase 2b).

For phase 3, all participation begins after a consent form has been signed and dated. In compliance with the Mental Capacity Act,51 nominee parental consent will be provided for children under 16 years with the opportunity to provide written assent for themselves. Young people over age 16 can consent for themselves unless assessed by the clinician to be too medically ill or mentally incapacitated. Written informed consent will indicate consent to notify the CYP’s hospital treating physician that they are participating in the study. All information will remain confidential and anonymised within the study team and regulatory authorities, unless there is indication of any risk of harm posed to the participant or others around them whereby members of the care team will be informed and local NHS reporting procedures followed. Anonymity will be ensured through the allocation of a study identification number to corresponding case report forms for each participant and separate storage of any patient identifiable data. All data will be stored on a secure server in encrypted format, with limited access permissions.

Potential participant distress and burden

While there is evidence that indicates performing a self-harm/suicide safety assessment is not harmful,52 we recognise that there is the possibility that the administration of the CYP-MH SAT and C-SSRS may cause some distress to participants. Therefore, only registered clinical staff will be trained in study procedures and will administer the measures. In the event that participants become too distressed their participation will be withdrawn. All participants will be made aware of their right to withdraw at any time and voluntary participation at study conception. All participants will continue to receive usual care from their care team during and following their involvement in the study. No care will be withheld or adapted for any participants taking part.

Local procedures will be adhered to in the case of participant disclosure throughout the course of the study. All staff will have received safeguarding training and know to share information regarding child protection as part of their professional practice with appropriate authorities (ie, social services).

Furthermore, for the CYP-MH SAT, we have integrated within the development phase of the tool, mechanisms to ensure that the items are appropriate and phrased sensitively. The e-Delphi procedure will apply a stringent prespecified consensus margin of ≥65%–70% consensus and a median value of 3.25, it is expected this will substantially reduce the amount of question items to enable a rapid assessment and minimise participant exposure. Furthermore, the tool will be scrutinised by the expert panel to ensure appropriate and sensitive terminology and phrasing will be used for each item.

Methods of disseminating findings

This paper serves as an important step in the dissemination of the findings by outlining the project background and aims, details of methods used and the practical challenges that may be faced and how these will be overcome.

We intend that the findings from this study will result in both the development and evaluation of a clinically appropriate and relevant MH SAT for use with CYP in MH crisis. This, in turn, will provide the NHS with an evidence based, assessment tool that could be employed to enhance the safety of CYP experiencing MH crisis admitted to an acute paediatric inpatient setting.

The research findings will be disseminated to regional, national and international audiences, including service users, clinicians, academics, service commissioners and policy-makers. Study findings will be disseminated in peer-reviewed, high-quality academic and professional journals in accordance with publishing guidelines and provide the opportunity for participants to gain access to the final publication and a summary of findings in lay format. Findings will be presented at local, national and international conferences.