Suicide after contact with a national digital mental health service.

BACKGROUND: The safety of mental health care provided remotely via the internet, in particular, the probability of suicide after contact, is not known.
METHOD: An observational cohort study of patients registered with the MindSpot Clinic an Australian national digital mental health service (DMHS), linked to the National Death Index. Measures included demographic information, the nature of contact, duration between last contact and death, scores on measures of psychological distress (K-10), depression (PHQ-9) and anxiety (GAD-7), and responses to questions about suicidal thoughts or plans for patients who died by suicide within two years of last contact with the service.
RESULTS: Sixty-four (0.11%) of 59,033 patients registered with the MindSpot Clinic between 1 January 2013 and 31 December 2016 died from suicide within two years of last contact. The mean time between last contact and death was 344 days. Fourteen patients died within 90 days of last contact, and 4 of 285 who were urgently referred for crisis service intervention at the time of contact or soon afterwards died within 2 years. Suicidal thoughts (OR: 2.59), a suicide plan (OR: 10.8), and a score of "3" to item 9 of the PHQ9 (OR: 16.4) were significantly associated with subsequent suicide. Patients who died by suicide were more likely to be male (OR: 3.2), middle-aged (35-45; OR: 2.3), separated or divorced (OR: 3.1), unemployed (OR: 3.1) or receiving disability benefits (OR: 5.1). Enrolling in an online treatment course was associated with reduced risk (OR: 0.38).
CONCLUSIONS: Although DMHS provide services to patients with severe symptoms of depression, only a small proportion died by suicide, and only a small number of those referred for urgent care, which suggests that the safety protocols of the clinic are relatively effective.

Introduction

There is a significant association between depression and subsequent suicide (Ribeiro et al., 2018). A large prospective study found that half of those who died by suicide had been diagnosed with a mental health condition or received treatment in the previous year (Simon et al., 2018), and another large study conducted in primary and outpatient care found that patients reporting frequent thoughts of death or self-harm in response to question 9 of the Patient Health Questionnaire (PHQ-9) were five times more likely to die by suicide in the following year, compared to patients not reporting those thoughts (Simon et al., 2016).

There has been a rapid recent growth in the number of digital mental health services (DMHS) worldwide in an attempt to increase access to evidence-based care (Titov et al., 2018), and in several countries DMHS now offer services to large numbers of patients as part of routine care (Titov et al., 2018). The effectiveness of therapist-guided DHMS is now well-established, with clinical outcomes comparable to high-quality face-to-face care (Titov et al., 2020; Andersson et al., 2019; Etzelmueller et al., 2020). However, a recent meta-analysis of contact with mental health services prior to suicide did not include any studies of suicide after contact with a DMHS (Walby et al., 2018), and little is known about the risk profiles of users of DMHS or the adequacy of procedures for identifying and managing risk.

Australia has several DMHS, including the MindSpot Clinic (MindSpot), funded by the Australian Department of Health. MindSpot delivers evidence-based assessment and treatment by trained mental health professionals to over 20,000 adults per year with anxiety, depression, and chronic pain (Titov et al., 2020; Titov et al., 2015; Titov et al., 2017). Most MindSpot users self-refer after being told about the service or conducting their own searches. Three-quarters report clinically significant symptoms of depression, often in the severe range, more than one-quarter report suicidal thoughts, nearly 4% report a current suicide plan (Titov et al., 2020), and around 0.5% are referred for crisis intervention after disclosing a plan to self harm, mostly at the time of assessment (Nielssen et al., 2015). However, the outcome of those urgent referrals, other than that services were available to take urgent referrals in all parts of Australia, and the risk of suicide after contact with DMHS such as MindSpot is not known.

Aims of the study

The aims of this study were to (1) identify the rate of suicide in the two years after last contact with a nationwide DMHS; (2) establish whether patients who were urgently referred to acute services because of increased suicide risk subsequently died by suicide, and (3) attempt to develop a predictive risk assessment model from the characteristics of those who died by suicide.

Methods

Patients and data linkage

Patients were Australian residents aged 18 years or older. A total of 62,611 people registered with MindSpot between 1 January 2013 and 31 December 2016 (Fig. 1), of whom 59,033 provided consent for their records to be used for research purposes and sufficient data for linkage. Patient details were forwarded to the Australian Institute for Health and Welfare (AIHW) to match with the National Death Index. The AIHW provided data on the fact of death and the cause of death by suicide using ICD-10 codes (X60-X84). At the time of the linkage data from the National Death Index was current until the end of 2018, hence the endpoint was defined as 1 January 2019 or the date of suicide. The files of patients who were found to have died from suicide within two years of last contact with MindSpot were then examined.

Fig. 1

Patient flow diagram.

MindSpot clinic procedures

Most patients self refer and are invited to complete an online assessment that includes a series of symptom questionnaires. Those with clinically significant symptom scores are then offered one of a series of self guided courses provided over eight weeks with clinician support as required. Patients enrolled in courses are asked to fill out symptom questionnaires in each week of the course and at 3 month follow up. The procedures for assessment, the nature of the treatment provided and the outcomes of treatment are described in detail elsewhere (Titov et al., 2020; Titov et al., 2015; Titov et al., 2017).

MindSpot Clinic safety protocols

MindSpot operates with a clinical governance framework and policies and procedures aligned with the Australian National Safety and Quality Digital Mental Health Standards (ACSQHC, 2020). The procedures for identification and management of patients at risk of harm to self or others at MindSpot are based on those of the Department of Health in the state of New South Wales (NSWMOH, 2014). Assessment of risk occurs at each stage of a patient's contact with MindSpot and is guided by patient responses to symptom questionnaires and clinical enquiry by trained therapists about thoughts of self-harm and other indicators of risk. Patients identified as being at risk are asked in more detail about symptoms and the presence of known risk and protective factors, and those patients who are unable to confirm their safety are referred to local crisis or emergency services. Those patients were free to re-engage with MindSpot at a later date once the crisis resolved. A detailed account of the safety procedures and the nature of urgent referrals have been published elsewhere (Nielssen et al., 2015).

Measures

Demographic information

As part of the registration and assessment process, twenty-three demographic and symptom variables are collected, and these formed the bases for the analysis of suicide risk. Demographic information included age, gender, marital status, country of birth (Australia or other), Indigenous status, level of education and employment.

Symptom questionnaires and clinical indicators of risk

The Patient Health Questionnaire 9-Item (PHQ-9) (Kroenke et al., 2001) Generalized Anxiety Disorder 7-Item Scale (Spitzer et al., 2006) and Kessler 10-Item Plus Scale (K-10) (Kessler et al., 2002) were administered at assessment to measure depression, anxiety, and general distress, respectively. Participants were asked series of questions about past and current health service use, including whether they had ever seen a mental health professional for symptoms of depression or anxiety, whether they spoke with a general practitioner about their mental health, and whether they were taking psychotropic medication. They were also asked about difficulties in employment, relationships, physical health or finances, and suicidal thoughts or plans for self-harm.

Clinical data extraction

Clinic records were examined to establish the nature of contact. Patient contact was defined as: (1) patients who completed an assessment only (n = 48,110), (2) patients who enrolled in a treatment course (n = 10,638), and (3) patients who were urgently referred for crisis intervention (n = 285), either at the time of assessment (n = 252) or during treatment (n = 33). Information about the service and treatment outcomes has also been published in detail elsewhere (Titov et al., 2020; Titov et al., 2015; Titov et al., 2017). The duration between the date of last contact, defined as most recent login or phone contact, and death was calculated in days.

Statistical analyses

The suicide rate, suicide risk factors, and the development of a predictive risk model followed three steps. First, the rate of suicide was estimated for the total sample, and three subgroups: (1) those who undertook an assessment only (2) those who enrolled in treatment and (3) those who required urgent referral due to assessed risk. Precautionary checks for differences between each of the four years of the study (2013–2016), and differences in the rate of suicide in the first and second years after contact with the service were conducted.

In a second step, the participant-level correlates of suicide risks were explored with a series of univariate logistic regressions, examining the heterogeneity of suicide risk associated with demographic characteristics and symptoms at presentation. An examination of correlates of risk was repeated in the assessment, treatment, and urgent referral subgroups to examine the effect of referral pathway and participation in treatment. Univariate model estimates were reported as a 1:100,000 per annum (pa), age-standardised, incidence risk ratio, to allow a ready comparison with population benchmarks (Ahmad et al., 2001).

Third, a predictive model of participant suicide risk was created for the sample as a whole, and each of the service subgroups, to identify combinations of characteristics associated with higher risk. In this step, participants were classified into subgroups characterised by several variables that compound suicide risk (multivariate, aggregated risk). Estimation of the participant risk was created with a Chi-square automatic interaction detection classification algorithm (CHAID) and evaluated within the sample as whole and within each of the subgroups.

In line with STROBE guidelines (von Elm et al., 2007), an additional series of sensitivity analyses were conducted to test the robustness of the suicide correlates to measurement variance; (1) re-estimation of suicide risk ratios without age standardisation, (2) re-modelling with missing data excluded (treated as a distinct category by default), and (3) reanalysis of suicide risk after controlling for any differences associated with the probability (propensity scores) of completing an assessment only, taking up treatment, or being referred to an emergency service. Results for these models are presented in the supplementary material.

Statistical analyses were performed using SPSS version 27, and p < .05 was considered statistically significant. Statistical power was determined at 0.8. Missing data in any patient variables were included in all analyses as a distinct category.

Ethical review

Approval to conduct the study was obtained from the Human Research and Ethics committees of Macquarie University (Reference No: 5201949936957) and the AIHW (EO2019/2/242). The MindSpot Clinic is registered on the Australian and New Zealand Clinical Trials Registry (ACTRN12613000407796).

Results

Estimation of suicide risk after contact with DMHS

In the first four years of operation, 59,033 consenting participants started an online assessment with the service (Fig. 1, Table 1). Of this total, 64 were found to have died by suicide within two years of last contact with the service, with an age-standardised estimate of suicide of 56.6 (95% CI 54.0 to 58.5) per 100,000 per annum (pa). Precautionary checks found the suicide risk ratio was stable over the four years of the study (Wald χ2 = 1.348, p = .704), and between the first and second years following contact with the service (Wald χ2 = 0.3745, p = .711).

Table 1

Characteristics of patients that died by suicide, irrespective of proximity of contact with MindSpot.

	Total	Benchmark comparison⁎
Number of patients	n = 64	–
Number of assessments completed	57/64 (81%)	–
Started a treatment course	7/64 (11%)a
Demographics (at assessment)
Mean age at time of assessment (SD)	38.2 years	35.7 years
Born in Australia	82%	78%
Aboriginal or Torres Strait Islander	4.7%	3.7%
Paid employment	44%	57%
University degree	25%	39%
Married/de-facto	27%	38%
Symptoms (at assessment)
K-10	36.5 (7.3)	31.8 (7.5)
PHQ-9	19.9 (5.6)	14.9 (6.2)
GAD-7	15.0 (4.8)	12.5 (5.2)
Score of 3 on item 9 of PHQ-9	35.0%	–
Reported suicidal thoughts	66%	32%
Reported suicide plan (of those reporting suicidal thoughts)	25%	4%
Service use (at assessment)
Previous or current health professional	87%	65%
Speaks with a GP about mental health	68%	47%
Current psychotropic medication	48%	27%
Main purpose for seeking service at MindSpot
Assessment and information	50%	67%
Treatment	43%	26%
Other	7%	7%
Main reason for using an online mental health service rather than traditional service
Convenience and cost	43%	34%
Privacy and anonymity	17%	33%
Other	40%b	33%

Comparison column from Titov et al. (2020). Lancet Digital Health 2, E582–E593.

Only 2 patients completed all five lessons of a MindSpot treatment course. Both patients also completed 3-month follow-up. At follow-up, scores on the PHQ-9 and GAD-7 for both these patients were below clinical thresholds.

Of the patients that reported other reasons for seeking online support, 26% (7/27) reported that face to face services had not helped them, or they needed additional support.

Of the 59,033 cases, 11,902 did not fully complete the assessment questionnaires (20.2%), resulting in incomplete data for a large part of the sample. Risk ratios for suicide for those who did not complete the assessment and later died by suicide (n = 14) were not different from the remaining assessment sample (OR = 1.04, p = .764, Wald χ2 = 0.09).

Of the total DMHS sample, 10,671 (18.1%) participated in treatment with the service, of whom 7 are known to have died by suicide. Those participating in treatment were found to have a suicide risk ratio of 35.8 (95% CI 32 to 40.1) per 100,000 pa, significantly lower than the assessment only group (OR = 0.628, p < .001, Wald χ2 = 597.7).

Outcome of crisis referrals

During the four year period, 285 participants (0.48%) required crisis intervention and were urgently referred to a local mental health service (249) or an emergency service (35 to police, 1 to ambulance). Of the 285, four subsequently died by suicide, resulting in a significantly higher risk ratio estimate (OR = 13.47, 750.3 per 100,000 pa).

Statistical power

Based on the observed incident rates, statistical power of 80% was determined for odds ratio tests with effects were at least 0.6 (OR > 0.6) of the rate in the assessment subgroup. The reduced sample size and incident rate in the treatment (n < 7) and referral subgroups (n < 5) meant that the refutation of non-significant effects was not reliable.

Correlates of suicide risk

Patient characteristics are presented in Table 1 for the whole sample total and each service subgroup. Results in Table 2 show that of the 23 demographic and clinical variables, most subsequent suicides were male, had severe symptom scores of depression, anxiety, and psychological distress, scored “3” on item 9 of the PHQ9, and reported suicidal thoughts and a suicide plan. These features were significantly associated with proportionally greater suicide risk. Patients who reported taking psychotropic medication, unemployment, recent separation or divorce, and a plan for self-harm were a minority of the suicide cases, but their risk ratios were high compared to the number of cases with those features.

Univariate logistic regression models found gender, educational attainment, employment, separation and indigenous status were associated with at least a 10-fold increased risk of suicide, and receiving disability benefit up to 20-fold greater. Among the clinical features, the most definitive single association with subsequent suicide was the disclosure of a suicide plan, estimated to result in a suicide ratio over fifty times the population rate (546.2, 95% CI 502.3 to 593.9).

Prediction of suicide in different paths of service use

A CHAID classification algorithm was used to profile suicide risk for combinations of patient variables for those who only had an assessment (Fig. 2) and those enrolled in treatment (Fig. 3). The algorithm identified three patient subgroups (tree nodes) showing compounded suicide risk for the sample as a whole. The most elevated risk was for males who had separated or divorced and reported a plan to self-harm (7/72, 4861 per 100,000 pa). Males with a plan to self-harm but who did not report separation or divorce had a smaller but still significant risk (7/1381, 253.4 per 100,000 pa). Among those who did not report a plan to self-harm, being unemployed or receiving disability benefit was associated with an increased risk (24/8298, or 144.2 per 100,000 pa), although with a high proportion of false positives (99.7%).

Fig. 2

CHIAD tree for assessment only patients.

Fig. 3

CHIAD tree for patients enrolled in treatment.

Similar patterns were found in those who only had an assessment, where again males who recently separated and had a plan to self-harm were most at risk (6/152, 1973.7 per 100,000 pa), and accounted for 0.32% of cases, but 11% of suicides. Those in middle-age (35–55), unemployed (9/1507, 298.6 per 100,000 pa), or receiving disability benefits (8/1725, 231.9 per 100,000 pa) were also at increased risk. Both models demonstrated effective cross-validation in a training-testing sample-partitioning (Table 2).

Risk modelling for those in treatment and the acute referral group were also attempted. In these groups, males formed the single significant predictor, accounting for 6 of 7 cases in treatment (6/2958, 101.4 per 100,000 pa) and all 4 in the referral group (4/103, 1941.7 per 100,000 pa). No covariates were identified in these models due to the low statistical power.

Finally, the sensitivity analyses of the results in Table 2 are presented for reference in supplementary online material. These models illustrate the robust estimation of the incidence risk ratio across the key demographic and symptom scale variables of Table 2, using different methods for analysing risk ratios.

Discussion

To our knowledge, this is the first study to examine suicide among former patients of a DMHS providing remote treatment using partly automated assessment and treatment. Despite a large number of patients who registered with the clinic reporting severe symptoms of depression, and suicidal thoughts and plans, only a small number died by suicide soon after contact with the service. The overall suicide rate of 56.6 per 100,000 per annum for the two years after last contact with the service is around five times the rate of the wider community (ABS, 2020), but is comparable to the rate of subsequent suicide reported among primary care and outpatient mental health service patients by Simon and colleagues (Simon et al., 2016). The findings help fill the gap in our knowledge of the potential adverse effects of digital interventions (Andersson and Titov, 2014) and will be of particular interest to regulatory bodies that are required to establish both the safety and quality of DMHS.

Outcome of acute referrals

MindSpot follows clinical procedures designed to identify and manage risk consistent with national standards for DMHS including escalation to specialist crisis services (ACSQHC, 2020). Examination of the clinic records of the 285 patients who were urgently referred to a crisis service found that of the 4 patients who subsequently died by suicide, but only one died soon after contact with the service, 90 days later. Services were found in all parts of Australia to take urgent referrals, and the finding that only one of the urgent referrals died by suicide soon after contact suggests that the crisis interventions of local emergency and mental health services were initially successful in addressing either suicidal intent or the reasons for disclosing suicidal plans in most cases.

Risk model derived from patient characteristics and symptoms

The strongest predictor of subsequent suicide was the disclosure of a suicide plan at the time of assessment, which had an odds ratio (OR) of 21.3 among those not progressing to treatment, followed by an OR of 16.4 for those with a score of “3” to question 9 of the PHQ-9, which is higher than the relative hazard reported by Simon et al. (2016), and may reflect a greater willingness to disclose suicidal thoughts to a DMHS service (Nielssen et al., 2015). Analysis of characteristics of those registering with a DMHS found middle age, male gender, indigenous status, and being unemployed or in receipt of disability payment all carried an increased risk of suicide. In many cases, disability payments reflect the diagnosis of a chronic psychiatric disorder, consistent with the findings of Weiser and colleagues that a diagnosed psychiatric disorder was the main predictor of subsequent suicide in a large sample of Israeli military recruits (Weiser et al., 2016). The study provided some evidence that enrolment in treatment was associated with a reduction in suicide risk (OR 0.38), either because treatment was effective, since participation in treatment by MindSpot has consistently achieved effect sizes of greater than 1.2, and a 50% reduction in symptoms scores (Titov et al., 2020), or because the characteristics of those who enrolled in treatment differ from those who do not (Moskalenko et al., 2020).

The CHAID analysis identified a subgroup of middle-aged recently separated males with a suicide plan, of whom 7/72 (10%) died by suicide within two years of contact, suggesting there may be value in targeted engagement of men who disclose this combination of risk factors (Torous and Walker, 2019). Other high-risk combinations included being middle-aged and either unemployed or on disability payments, although these characteristics were of limited utility due to the high rates of false positives (Nielssen et al., 2017). Many of those who went on to die by suicide did not complete an assessment and the suicides often took place long after contact, in circumstances that were not known. Moreover, the risk factors associated with suicide in the MindSpot sample were common among people who did not suicide, and even in high-risk populations, death by suicide is a comparatively rare event, leading to high rates of false positives in predictive models that significantly limit their clinical utility (Ryan et al., 2010; Belsher et al., 2019). Nevertheless, the current study identifies a range of factors, when combined, are associated with increased risk of suicide (e.g., being male, middle aged, unemployed or receiving a disability payment) that could potentially be explored and used to guide targeted engagement aimed at reducing risk.

Limitations

The study has several limitations. The first is that some patients did not provide consent for their data to be analysed, and a small number did not provide enough identifiable information for data linkage. Moreover, deaths by suicide are statistically rare events, and statistical analyses based on small numbers need to be considered with caution.

It should also be noted the follow-up period between the last contact and time of death in the current study ranged from one day to two years, and the risk of suicide is likely to continue for several years for some patients. A small number of patients completed more than one assessment, although the duration from last contact was taken from the most recent assessment, and the small number did not significantly affect the results. Moreover, the circumstances of suicide are complex, and the current study was unable to examine the nature of intervening events, including the effect of engagement with other mental health services, further life events or substance use. A further study using information about the circumstances of death derived from the National Coroner's Information System is planned. Another important limitation is that the results reported here are based on contact with an established and well-regulated DMHS in which services are delivered by trained mental health professionals, and may not generalise to fully automated services or services provided by unqualified staff. Despite these limitations, the study provides valuable information on risk factors associated with DMHS, and may assist in planning of regulatory frameworks and integration into existing health care systems for this emerging sector.

Conclusions and clinical implications

While the overall suicide rate in the current study is larger than the rate of suicide in the general population (ABS, 2020), it is comparable to studies of face-to-face care (Simon et al., 2016; Simon et al., 2018), with very few suicides at the time of or soon after contact with the service. However, the increased probability of subsequent suicide among patients who disclosed suicidal thoughts and plans and the identification of several groups with higher risk indicate the need to address the enduring risk of suicide in these patients, either by follow-up or referral for ongoing care. There are inherent limitations of suicide risk assessment including high false positive rates (Torous and Walker, 2019; Nielssen et al., 2017) and a lack of agreement regarding procedure to be followed after the identification of risk. The results also suggest that research trials that exclude potentially suicidal patients may be overly cautious since suicidal thoughts are a common symptom of depressive illness, and produce results that are not necessarily representative of routine care and deny effective treatment to some potential participants (McCall et al., 2019). However, there is clear benefit to most patients who engage in treatment with MindSpot, based on the reduction in symptom scores of those who engage in treatment, or as a result of the opportunity to refer people for urgent face to face care. Overall, these results indicate that if mandated risk identification, triage and management protocols are followed, the safety of DMHS delivered by mental health professionals is similar to that of face to face services.

The following are the supplementary data related to this article.

Table 2

Age standardised 1:100k annual risk ratio estimates of a suicide event within the total sample, and service subgroup.

Funding

The MindSpot Clinic is funded by the Australian Health Department. Professors Titov and Dear are authors and developers of the treatment courses used at the MindSpot Clinic but derive no personal or financial benefit from them.

Declaration of competing interest

The MindSpot Clinic is funded by the Australian Health Department.

Professors Titov and Dear are authors and developers of the treatment courses used at the MindSpot Clinic but derive no personal or financial benefit from them.

None of the other authors have any potential conflict of interest to declare.