Study designs and outcomes used in evaluation studies of hospital-presenting self-harm: protocol for a methodological systematic review.

INTRODUCTION: Self-harm is the most common risk factor for suicide, and so those who present to hospital following self-harm provide an opportunity for targeted clinical care interventions. Observational studies evaluating such interventions may be useful in overcoming limitations of controlled trials, but study design, statistical analyses and outcomes used must be appropriate. This methodological systematic review will describe, categorise, synthesise and compare the methodological aspects of studies evaluating interventions and aspects of clinical management following hospital-presenting self-harm in both observational and experimental (ie, controlled trials or quasi-experimental studies) study designs. METHODS AND ANALYSIS: Preferred Reporting Items for Systematic Reviews and Meta-Analysis-Protocol guidelines were followed in drafting this protocol. Search terms were developed (related to self-harm, hospital presentation and evaluation studies) and adapted for MEDLINE, PsycINFO, EMBASE, CINAHL, Cochrane Central Register of Controlled Trials and grey literature databases. Two reviewers will independently screen 100 titles/abstracts until consensus is reached, with the remaining screened by one reviewer. Full-text screening will be conducted independently by two reviewers. Data will be extracted by one reviewer, and a second will check all data extracted. Validated risk of bias tools will be used. Data synthesis will focus on the heterogeneity of outcomes used in individual studies. Descriptive summary statistics of the data (eg, key study characteristics, type and frequency of outcomes) will be provided in categorical format, using frequencies and percentages. Outcomes will be reported separately for trials (both randomised and non-randomised trials), observational and quasi-experimental studies. Categorisation of outcomes will be guided by Cochrane Effective Practice and Organisation of Care resources for reviews of health systems interventions. ETHICS AND DISSEMINATION: Results will be disseminated at national and international conferences and published in a peer-reviewed journal. Findings will be used to inform future studies in the area of hospital-presenting self-harm. Ethical approval is not required for this review. PROSPERO REGISTRATION NUMBER: CRD42020208714. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Utilising high quality and robust systematic search methods, this review will assess the quality of the methodological aspects of observational studies evaluating interventions following hospital-presenting self-harm including design, measures for confounding and outcomes used.

The range and quality of observational studies following hospital-presenting self-harm, compared with experimental designs, will be reported for all included studies.

As a first step in describing and obtaining consensus on core outcome measures in this area, this review will cover the suitability and quality of outcome measures used.

The focus on hospital-presenting self-harm only, although justifiable, is a potential limitation and the authors acknowledge from the outset that results will not be include studies involving other healthcare settings.

Review findings will allow for more standardisation in the evaluation and reporting of outcomes, thereby highlighting the potential of registry-based and routinely available data in informing service delivery and policy for this patient group.

Introduction

Each year, approximately 800 000 people die by suicide.1 The single most important risk factor for suicide is a history of self-harm, with those who engage in self-harm up to 50 times more likely to die by suicide than the general population. Therefore, research is often focused on the population who present to hospital following self-harm as a key intervention group. The incidence rates of hospital-presenting self-harm are highest among young people and females. Furthermore, approximately 16% of individuals will reattend hospital with a further episode of self-harm within 12 months.2

Over the past number of decades, several experimental trials have sought to evaluate interventions for individuals who present to hospital with self-harm, both in the hospital setting and following discharge. Such trials use both randomised and non-randomised controlled designs, and have focused on reducing the incidence of repeat self-harm and suicide, including brief hospital-based interventions3–6 and follow-up psychosocial interventions.7 8 However, most of this evidence comes from relatively small studies, and overall, there is a lack of recent evidence from large scale trials.9

Observational study designs are known to offer huge potential by evaluating the impact of exposures in real-world settings.10 11 Such studies are relatively inexpensive compared with trials, allow for multiple outcome measures12 and may overcome some ethical issues in treating individuals with complex physical and mental health concerns. Studies employing these designs have primarily utilised hospital records and data recorded via self-harm registries.13–19 However, a systematic review that examined aspects of routine clinical care following self-harm (eg, hospital admission or specialist follow-up) found little evidence for their role in reducing repeat self-harm and suicide,20 despite evidence from individual studies being used to inform several guidance.21 22

The reasons for a lack of evidence from observational studies may reflect the size and representativeness of the samples used, the quality of study design, as well as the consistency, quality and range of outcomes used. High-quality design and use of appropriate methods are imperative in order to strengthen causal influence and minimise risk of bias.23 Observational studies should include measures which adequately assign participants to exposed or comparison groups, as well as address both observed and unobserved confounding.10 While limited in most respects to routinely recorded data, observational and quasi-experimental studies have the potential to incorporate a wider range of outcomes than trials, including processes of care. Although there is a need to expand on and improve the quality of outcomes examined, particularly beyond using repeat self-harm as a sole indicator of patient outcomes, consensus on the type and quality of outcome measures is lacking.24–27

While work has been undertaken to address methodological issues in trials,28 to date, no study has undertaken to examine the design, analysis and choice of outcomes in other experimental and observational studies. In addition, little research has been undertaken to compare the quality of observational studies and their choice of outcome, as compared with experimental studies. This review will aim to provide guidance by describing the studies which have been conducted to date which evaluate interventions and aspects of clinical management following hospital-presenting self-harm via experimental (including controlled trials and quasi-experimental studies) and observational study designs. In particular, the study design and outcomes utilised will be examined. For studies examining changes at an individual, rather than aggregate level, these outcomes will be compared with those used in trials of the same population. To this end, the specific objectives of the study will be:

To describe the study design and outcomes used in evaluation studies with a focus on hospital-presenting self-harm.

To compare the outcomes used in observational studies with those used in experimental studies.

Methods and analysis

This protocol was prepared following Preferred Reporting Items for Systematic Reviews and Meta-Analysis-Protocol reporting guideline.29

It was not considered possible to involve patients or the public in the design, conduct, reporting or dissemination plans of this research.

Eligibility criteria

For this review, all primary research which aims to evaluate an aspect of clinical care for individuals presenting to hospital following self-harm will be included (hereafter referred to as evaluation studies), and where an outcome relating to hospital-presenting self-harm has been used. These evaluation studies can be experimental studies or observational, as long as there is a comparator/control group or control measures utilised. Studies published in the English language and meeting the following criteria will be eligible for inclusion.

Study design

Evaluation studies will be categorised as either experimental or observational studies. The term ‘experimental studies’ will be used to describe studies where the researcher intervenes during the study period, including randomised and non-randomised studies and quasi-experimental designs. The term ‘observational studies’ will be used to describe all study designs where the researchers are not acting on the study participants but observing relationships between factors and outcomes. These include non-controlled before-and-after studies and interrupted time series studies, cohort and case–control studies.30 31 Studies not considered eligible for inclusion are: cohort studies with a focus on aetiology, prevalence or incidence studies which do not seek to evaluate hospital or follow-up care, non-primary studies, single case reports, qualitative studies and reviews (including literature reviews, systematic reviews and meta-analyses).

Populations

Evaluation studies targeted at all individuals, regardless of age, presenting to hospital following self-harm will be included. Studies including self-harm in the general population, or patients presenting for suicidal thoughts or other mental-health related behaviours, will be excluded as they are considered beyond the scope of the present review. Should a study also include participants outside these parameters, these studies will not be excluded however only data relevant to the inclusion criteria will be extracted (where possible). For the purposes of this study, self-harm is defined as ‘an act with non-fatal outcome in which an individual deliberately initiates a non-habitual behaviour, that without intervention from others will cause self-harm, or deliberately ingests a substance in excess of the prescribed or generally recognised therapeutic dosage, and which is aimed at realising changes that the person desires via the actual or expected physical consequences’.32

Interventions

All evaluation studies with a focus on clinical care following hospital-presenting self-harm will be examined. An aim of the review is to identify and categorise intervention types, therefore categories cannot be predefined. However broadly, those eligible for inclusion will be interventions which are delivered to individuals presenting to hospital following self-harm. These may include brief interventions delivered within the hospital setting, routine aspects of clinical management (eg, next care procedures) or interventions delivered via follow-up care following presentation to hospital. In addition, population exposures such as changes in service delivery or reconfiguration of services will be included as population-level interventions.

Setting

Studies focused on emergency department attendances or admissions will be included. Presentations to other healthcare settings (eg, primary care, community-based care, outpatient clinics) and psychiatric hospitalisations (not admitted via emergency departments of general or acute hospitals) will be excluded. Interventions delivered in inpatient settings (either psychiatric or general) were considered beyond the scope of the present review largely due to the heterogeneity of reasons for inpatient admission. As mentioned previously, should a study include a variety of settings only data relevant to the inclusion criteria and review aim will be extracted.

Outcome measures

Studies relating to hospital-presenting self-harm will be eligible for inclusion. All clinical outcomes at an individual level (mortality, repeat attendance and admission), service level (admissions, assessments provided), economic outcomes and patient-reported outcomes are eligible for inclusion.

Patient and public involvement

No patients involved.

Search strategy

Search terms related to self-harm (eg, self-mutilation, attempted suicide and self-inflicted injury), hospital presentation (eg, hospital treated, emergency department and emergency medical services) and evaluation studies (eg, intervention, randomised controlled trial, observational study and evaluation study) were developed drawing on previous systematic reviews, expert opinion in the area and information retrieval specialists. For full details of the search strategy including use of truncation, phrase searching and subject headings, see online supplemental appendix 1. Terms were translated for the various databases included in the search strategy. Given the broad definition and varying terminology used to describe self-harm, extensive search terms were developed to increase search sensitivity (eg, attempted suicide, self-poisoning self-injury).

Data management

Covidence software will be used for data management, including deduplication, screening and data extraction.

Information sources

Databases to be searched will include Medline, EMBASE, PsycINFO, Cochrane Central Register of Controlled Trials and CINAHL. Key journals in the area will also be searched (including: CRISIS, Suicide and Life-Threatening Behaviour, Archives of Suicide Research, British Journal of Psychiatry, Lancet Psychiatry, JAMA Psychiatry, Psychological Medicine, Social Science and Medicine, Acta Psychiatrica Scandinavica). Grey literature databases (ETHoS, ClinicalTrials.gov Register, ProQuest Dissertation & Thesis A&I, OpenGrey.eu) and relevant organisation which hold data relating to hospital-presenting self-harm will be searched and screened. Various combinations of keywords will be used in Google and the first two pages of results for each keyword combination screened. Reference lists of included studies and relevant systematic reviews (identified via title/abstract screening) will be screened for additional eligible articles. In the case of conference abstracts and study protocols, should they satisfy the inclusion criteria, every effort will be made to retrieve the article reporting full results.

Academic and grey literature databases will be searched from January 2000 to July 2020. This starting point is informed by the establishment of several monitoring studies on suicide and suicidal behaviour in the 1990s, under the WHO/Euro multicentre study on parasuicide, which was the first large scale international collaboration on this topic.32–34 Prior to this date there would be few systems routinely recording data on hospital-presenting self-harm.

Data screening

Following deduplication and removal of obviously irrelevant records (ie, not related to hospital-presenting self-harm), titles of identified studies will be screened using the eligibility criteria by ER-M and PH. Initially, 100 of the titles and abstracts will be independently double screened and overall agreement will be assessed prior to completing the remainder of screening. If discussion does not resolve disagreements or uncertainty regarding inclusion, a third author will arbitrate. This process will be repeated until the team are satisfied that one author can proceed to single screening of the remaining records. Article full texts will be independently screened by two members of the review team and disagreements resolved by a third reviewer.

Data extraction and quality assessment

Standardised data extraction forms adapted from Cochrane Developmental, Psychosocial and Learning Problems (DPLP) templates will be used (online supplemental appendix 2) to obtain the following data from the included studies: author, year of publication, geographic setting (country), study design, sample size, study setting, participant characteristics (age, gender, method of self-harm, psychiatric diagnosis and sociodemographic information), statistical methods employed and the outcomes used will be extracted from the included studies. Details of the outcomes will be extracted using items adapted from Cochrane DPLP templates (primary and secondary outcome identified, outcome name, time points measured, time points reported, outcome definition, source of data (self-report, clinician interview/questionnaire, hospital records), unit of measurement, is the outcome/tool validated). Details of the intervention will be extracted using an adapted TIDieR checklist.35 Conclusions in terms of the presence or absence of an effect of the intervention, including the direction and strength of this effect will also be extracted. The data extraction form will be piloted on five of the articles and experience shared between reviewers. One reviewer will continue to extract the remainder of the data with a second reviewer checking each data extraction. Categorisation of the data will be conducted during data extraction, and further discussed and agreed by the review team based broadly on Cochrane guidance for the Effective Practice and Organisation of Care (EPOC) reviews of health systems interventions outcome categories.31

Risk of bias will be assessed using Risk Of Bias In Non-randomised Studies of Interventions (ROBINS-I)36 tool for non-randomised trials, observational cohort, or natural experiment designs and Cochrane risk of bias tool for randomised trials (RoB V.2.0)37 tool. Risk of bias assessment will be independently conducted by two researchers; disagreements will be discussed until agreement reached or a third researcher will arbitrate. A high risk of bias score will not result in exclusion of an included article, however risk of bias score will be taken into consideration when interpreting the results. Statistical analyses will be undertaken using SPSS V.26 and Stata/IC V12.

The quality of outcome measures identified will be assessed using a tool originally designed for use in development of a core outcome set for neonatal abstinence syndrome38 and adapted for use in the area of infant feeding.39 This tool consists of six binary items which relate to how outcomes assessed are clearly stated and defined, the rationale for outcome selection and how quality of outcome measurement was maintained.

Data synthesis

Data synthesis will focus on the heterogeneity of outcomes used in individual studies, rather than synthesising the evidence for interventions. Once the range of outcomes have been identified, they will be grouped into categories following discussion with the review team. Categorisation of outcomes identified will be guided by Cochrane EPOC guidance for reviews of health systems interventions.31 Cochrane EPOC guidance recommends that main outcomes for reviews for health systems interventions should cover the important outcomes that impact on individuals. Cochrane EPOC recommended outcome categories include: Patient outcomes; Quality of care; Utilisation, coverage or access; Resource use; Healthcare provider outcomes; Social outcomes; Equity; Adverse effects or harms. Descriptive summary statistics of the data (eg, key study characteristics, type and frequency of outcomes) will be provided in categorical format, using frequencies and percentages. Outcomes will be reported separately from experimental and observational studies. In addition, outcomes will be subcategorised based on whether they examine individual changes, population changes or changes in processes of care (eg, number of patients receiving psychosocial assessment following a hospital level change to improve care delivery).

Discussion

Interventional studies following hospital-presenting self-harm differ in their design, use of and operationalisation of outcomes. This review will contribute significantly to the field of suicide research by assessing the range and quality of such studies, and by highlighting the potential of routinely available data in informing service delivery and policy for those who present to hospital following self-harm. The results of this review will provide an insight into the quality and potential of observational and quasi-experimental studies to be used in place of trials, as well as assessing the comparability and consistency of outcomes measures used in such studies. A particular focus will be given to the outcomes utilised, as compared with trials. Unlike other areas of health services research and mental health research, there is no agreed set of core outcome measures (COMs) in interventional studies of self-harm, particularly for trial-based research. The results may be the first step in obtaining consensus on a set of COMs to be reported in interventions targeted at those presenting to hospital following self-harm. This methodology has been successfully applied to other areas of mental health research.40

By reviewing the studies which have been conducted in this area, it is expected that these findings will allow for more standardisation in the evaluation and reporting of outcomes, thereby resulting in improved research in this area. Furthermore, the potential of measures which are extracted via hospital records and self-harm registries will also be explored. Self-harm data are consistently used to inform policy, service delivery and clinical guidelines (including National Institute for Health and Care Excellence guidelines). However, such studies are dependent on data being routinely available, via self-harm registries.41 It is expected that this study could highlight the need for such data to be recorded on an ongoing basis to support research for this important patient group.

Ethics and dissemination

Results of the systematic review will be disseminated at national and international conferences and published in a peer-reviewed journal. Findings will be used to inform future studies in the area of hospital-presenting self-harm. No ethical approval was required for this systematic review.