Implicit Cognition Tests for the Assessment of Suicide Risk: a Systematic Review.

PURPOSE OF REVIEW: Suicide risk assessment is a challenge in clinical practice. Implicit measures may present with advantages with respect to explicit methods, and therefore may be useful for the assessment of suicide risk. We conducted a systematic review of 2 databases (PubMed and EMBASE) about implicit tests that measure suicide risk to explore their validity and reliability. RECENT
FINDINGS: Initial research revealed 321 articles. After the selection process, 31 articles were included in the review. The most death-related implicit cognition test used was the Death/Suicide Implicit association test (D/S IAT), followed by the Suicide Stroop Task. The Suicide Affect Misattribution Procedure (S-AMP) and the Death version of the Implicit Relational Assessment Procedure (D-IRAP) were also used. We found that the measures reviewed were generally valid for the assessment of past and future suicidal thoughts and behaviors, with statistically significant results regarding retrospective and prospective associations.

Introduction

A person dies by suicide every 40 s [1]. Suicide represents 1.5% of all deaths worldwide [2]. Suicide attempts (SAs) are estimated to be twenty times more prevalent than death by suicide. Suicide ideation (SI) is not only a recognized risk factor for SAs and death by suicide but also represents a public health problem in its own right [3]. Due to the social, economic, and psychological stresses of the Covid-19 crisis last year, rates of SI, SAs and death by suicide are expected to rise, especially among populations at risk, thus becoming an urgent concern for public health worldwide [4].

Suicide risk assessment is a challenge in clinical practice. Traditionally, it relies on patients’ self-report about their suicidal intentions [5, 6]. The results of a suicide risk assessment will lead to a specific intervention that addresses the patient’s suicidal intentions to prevent fatal outcomes [7]. Nonetheless, self-report measures for suicide risk assessment/estimation may present some limitations.

On the one hand, suicidal patients may be ambivalent about expressing their suicidal thoughts and hide crucial information during a structured clinical assessment. In a prospective study, 78% of the people who had died by suicide explicitly denied such intentions during the previous clinical evaluation [8]. Reasons why patients deny their suicidal intentions are broad; patients may not feel comfortable disclosing their intention to end their lives [9]. Alternatively, they may not be fully aware of their intention or even not be capable of verbalizing it [10]. Likewise, patients may underestimate the severity of their thoughts and their needs for clinical services [11]. Moreover, others may conceal certain information to avoid hospitalizations that would frustrate their suicidal plans [12].

On the other hand, suicide risk assessments and suicide screening are usually conducted in crowded psychiatric emergency departments with limited time and resources to perform clinical evaluations [7]. These conditions make it particularly challenging for the clinician to estimate and predict suicide risk [13]. In a retrospective study of people who died by suicide, results showed that clinicians classified suicide risk as low or absent in 80–90% of the cases [14].

In the last decades, literature has sought to investigate indirect markers of suicidal behavior that are not based on self-report [15•, 16]. In this context, the use of tests based on implicit cognition arises. Implicit cognition is the ensemble of judgments, assumptions, and associations made automatically and unconsciously [17]. Thus, tests based on implicit cognition are not subjected to introspection, so that individuals can hardly control or manipulate their responses [18]. Implicit cognition tests have a large background that started in 1935 with the traditional Stroop task, which have been used by cognitive psychologists to study attentional processes [19]. Nowadays implicit cognition tests have been extended to different measures, and there are specifically some related to suicide, such as the Suicide Stroop Task version [20]. In the context of suicide risk assessment, implicit measure tasks allow to assess implicit biases for suicide-related content in real time without having to ask participants directly. Tasks are usually associative and constitute a behavioral marker of suicide risk. Among the different implicit cognition tests, the most frequently used is the Implicit Association Test (IAT) [21, 22]. The IAT is based on the response time given by a person when performing an association task [23].

The first IATs were used to explore implicit attitudes on sensitive issues, such as racism [24]. The assumption behind the test is that it should be easier to associate two concepts when they are already related to each other in our minds [24]. For example, IAT studies show that many participants who claim not to have racist attitudes nevertheless responded more quickly and accurately when they were asked to associate White with Positive and Black with Negative than the opposite [21].

In 2010, Nock et al. [12] adapted the original IAT for suicide risk assessment, thus creating the Death/Suicide IAT (D/S IAT), which aimed to identify people who quickly associated suicide with the self.

Other tests based on implicit cognition in suicide risk assessment have emerged in later years, such as the Suicide Stroop task [20, 25–29], which evaluates the time it takes a person to identify the font color of death/suicide related words. Also, the Death version of the Implicit Relational Assessment Procedure (D-IRAP) [30], which is similar to the IAT in that participants have to pair stimulus under time and accuracy pressure. And the Suicide Affect Misattribution Procedure (S-AMP) [16, 31] that evaluates how participants misattribute a stimulus after being presented with a suicide/death-related prime.

The ease of implementing these implicit measures would allow clinicians to quickly assess real-time suicide risk like in emergency departments [13].

A recent meta-analysis explored the validity—discriminative and predictive value—of the D/S IAT, finding that the test is accurate when predicting both past and future suicide behaviors. However, they recommend that suicide risk be determined based on multiple sources of assessment and not solely based on the D/S IAT [32]. Additionally, another review tested the reliability and concurrent validity of the Suicide Stroop task and found poor psychometric properties [33]. Given the mixed results, it is important to systematically explore the value of implicit cognition tests for the discrimination and prediction of suicide behavior in order to clarify and standardize the relevance of their use in clinical and research settings. However, there are no systematic reviews about the use of implicit cognition tests for the assessment of suicide risk. Here, we perform a systematic review of the retrospective and prospective validity of implicit cognition tests for the assessment of suicide. We discuss the implications of our findings for clinical practice and future research.

Methods

This review followed the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines [34]. The review protocol was registered in the PROSPERO database (registration number CRD42020165368).

Inclusion/Exclusion Criteria

Inclusion criteria were as follows:

Original studies published in peer-reviewed journals that report measurable outcomes (e.g., score in a test).

Studies that use an implicit cognition test of death-related and/or self-injury-related variables (e.g., D/S IAT).

Studies that measure the past or future occurrence of any kind of suicidal behaviour (e.g., number of past suicide attempts).

Studies that explore the correlation between the results of the test and the occurrence of suicidal behaviour (e.g., statistical analysis of correlation such as Pearson correlation test).

Exclusion criteria were as follows:

Proofs of concepts, protocols for randomized clinical trials.

Studies that do not provide measurable outcomes.

There were no restrictions regarding language or publication date.

Search Strategy

We conducted a systematic literature search in two of the most accessible databases: PubMed and EMBASE. The last search date was 20 January 2021.

The following search terms were used: Implicit AND Suicide (suicide OR suicidal OR self-harm OR self-injury). The references of included studies were also screened.

Study Selection Process

The articles were selected if they were of relevance to the research question (i.e., the validity and reliability of implicit cognition tests for the assessment of suicide risk), met the inclusion criteria, and were of sufficient methodological quality. Eligible studies were critically appraised [35].

Studies were independently reviewed for inclusion by two authors (MM and APS). Any inconsistencies were resolved with the involvement of a third author (LGR). Agreement between reviewers was measured by intraclass correlation coefficient (ICC).

Study Selection and Data Extraction

The quality of all eligible studies was assessed independently by two reviewers (MM and APS). Discussion between reviewers resolved all discrepancies. Aspects assessed included methodological design, risk of bias, and quality of reporting. Data were identified, checked, and mined by two independent authors (MM and APS). Using pre-made tables, the following variables were collected: author; year of study publication; country; aims of study; sample size; sample characteristics; mean age of the sample; gender distribution of the sample; test used in the study; outcome; timeframe (retrospective association or prospective association); results and main findings.

Results

The initial search revealed 321 results. After screening, full-text review and study selection, 31 articles were finally included in the review (see Fig. 1). ICC among reviewers was 0.79 (95% CI 0.67–0.89).

Fig. 1

Flowchart of the bibliographical search

Characteristics of the Reviewed Studies

Table 1 summarizes the characteristics of the reviewed studies [3, 7, 12, 16, 20, 25–31, 36–54].

Table 1

Characteristics of the reviewed studies

Study	Country	Aims of study	Sample	Mean age (DS)	Gender distribution (% female)	Test
Barnes et al. [36]	USA	Retrospective and prospective association with suicide attempts	N = 163 psychiatrically hospitalized veterans with suicide behaviors Group 1 (recent SA): 27 Group 2 (without SA): 136	Group 1: 49.0 (12.3) Group 2: 45.6 (14.4)	Group 1: 7% Group 2: 6%	D/S IAT
Barnes et al. [37]	USA	Prospective association on future suicide attempt in comparison with other risk factors	N = 166 psychiatrically hospitalized veterans for suicide behaviors Group 1 (with PTSD): 92 Group 2 (without PTSD): 74	Total: 47.0 (13.8) Group 1: 45.0 (14.2) Group 2: 49.5 (12.8)	6%	D/S IAT
Becker et al. [25]	Germany	Prospective association with suicide attempts	N = 62 Group 1 (history of SAs): 31 Group 2 (controls): 31	Group 1: 38.4 (16.2) Group 2: 38.2 (14.6)	Group 1: 64% Group 2: 64%	Suicide Stroop Task
Cha et al. [26]	USA	Retrospective and prospective association with suicide attempts	N = 124 adults presented to a psychiatric emergency department Group 1 (previous SA): 68 Group 2 (no previous SA): 56	Group 1: 34.1 (10.5) Group 2: 35.1 (13.2)	Group 1: 42.6% Group 2: 35.2%	Suicide Stroop Task
Cha et al. [38]	USA	Prospective association with suicidal ideation before and after mood induction	N = 264 community-based adults Group 1 (current SI): 176 Group 2 (controls): 88	Group 1: 31.44 (12.14) Group 2: 35.58 (15.89)	Not reported	D/S IAT and Suicide Stroop Task
Chung and Jeglic [27]	USA	Retrospective association with suicide behaviors	N = 736 undergraduate students Group 1 (history of STB): 101 Group 2 (controls): 635	Group 1: 20.2 (4.1) Group 2: 19.9 (3.5)	Group 1: 81.2% Group 2: 66.6%	Suicide Stroop Task
Chiurliza et al. [39]	USA	Retrospective association with history of suicidal behaviors and depression	N = 1548 US military service members	Total: 30.03 (4.99)	8.5%	D/S IAT
Dickstein et al. [40]	USA	Retrospective association with suicide attempts and non-suicidal self-injury	N = 136 adolescents psychiatric inpatients Group 1 (made an SA but had never engaged in NSSI): 47 Group 2 (engaged in NSSI but had never made an SA): 46 Group 3 (controls): 43	Group 1: 15.85 (1.27) Group 2: 15.24 (1.26) Group 3: 15.47 (1.29)	Group 1: 66.0% Group 2: 82.6% Group 3: 60.5%	SI-IAT and a D/S subtest performance
Ellis et al. [41]	USA	Retrospective association with psychiatrics measures (depression severity and suicide ideation) Prospective association with suicide behaviors	N = 124 psychiatric inpatient with SI and complex treatment resistant disorders (diagnosed with multiple Axis I and Axis II disorders, and an average length of stay of 6 weeks)	Total: 34.51 (13.40)	56.5%	D/S IAT
Glenn et al. [42]	USA	Retrospective association with self-harm	N = 7015 adult volunteers Group 1 (did SI-IAT): 2063 Group 2 (did Death IAT): 2042 Group 3 (did Suicide IAT): 2124	Group 1: 27.04 (10.43) Group 2: 27.34 (10.83) Group 3: 27.58 (10.89)	Group 1: 67.9% Group 2: 66.9% Group 3: 67.0%	SI-IAT, Death IAT, and S-IAT
Glenn et al. [43]	USA	Retrospective and prospective association with suicide ideation and attempts	N = 276 adolescents admitted to a short-term residential treatment program	Not reported	Not reported	D/S IAT
Glenn et al. [44]	USA	Retrospective and prospective association with suicidal thoughts and behaviors	N = 141 adolescents as part of a previous study (2017) Group 1 (non-suicidal psychiatric controls): 35 Group 2 (lifetime SI): 12 Group 3 (past-year SI): 42 Group 4 (lifetime SA): 30 Group 5 (past-year SA): 22	Total: 17.52 (1.64) Group 1: 17.40 (1.63) Group 2: 18.00 (1.65) Group 3: 17.37 (1.67) Group 4: 18.03 (1.07) Group 5: 17.05 (2.10)	Total: 81.6% Group 1: 80.0% Group 2: 75.0% Group 3: 76.2% Group 4: 93.3% Group 5: 81.8%	Death IAT
Harrison et al. [45]	Australia	Retrospective association with suicidal behaviors and suicidal risk factors, and the mediation of life oriented beliefs	N = 408 undergraduate students	Total: 20.36 (4.72)	69.8%	D/S IAT
Harrison et al. [46]	Australia	Prospective association in the estimations of suicide risk and exploring the associations between risk and protective factors	N = 128 adults with SI and SA presented at emergency department	Not reported	61.7%	D/S IAT
Hussey et al. [30]	Ireland	Retrospective association with suicide ideation	N = 52 psychiatric patients and undergraduate students Group 1 (current SI): 23 Group 2 (control): 25	Group 1: 38.6 (12.4) Group 2: 18.7 (2.7)	Group 1: 56% Group 2: 56%	D-IRAP
Kene [47]	USA	Retrospective associations with suicide attempts	N = 100 forensic and civil inpatients at three psychiatric hospitals Group 1 (previous SA): 60 Group 2 (controls): 40	Total: 35.84 (11.44) Group 1: 36.13 Group 2: 35.40	Total: 63.0% Group 1: 45% Group 2: 25%	SI-IAT identity and attitude versions
Millner et al. [48]	USA	Reliability and validity of the Brief D/S IAT	N = 570 adult volunteers	Total: 41.9 (10.8)	69.3%	D/S B-IAT
Millner et al. [49]	USA	Reliability, retrospective and prospective associations with self-harm and suicide risk	N = 71 adolescents from psychiatric inpatient unit	Total: 14.8 (1.5)	57.1%	Death IAT, Suicide IAT, SI-IAT, Death SC-IAT, Suicide SC-IAT, and Suicide SC-IAT with pictures
Moreno et al. [50]	Spain	Validation of the D/S IAT and prospective associations with suicide attempt	N = 75 adults from psychiatric outpatients with history of STB	Total: 24.67 (13.50)	72.0%	D/S IAT
Nock and Banaji [3]	USA	Retrospective and prospective associations in suicidal behaviors	N = 89 adolescents from general and psychiatric population Group 1 (no SI): 38 Group 2 (current SI): 37 Group 3 (recent SA): 14	Total: 17.10 (1.92) Group 1: 16.9 (1.9) Group 2: 17.6 (1.6) Group 3: 16.2 (2.2)	Total: 76.4% Group 1: 73.7% Group 2: 75.7% Group 3: 85.7%	SI-IAT
Nock et al. [12]	USA	Retrospective and prospective associations with suicide attempts	N = 157 emergency department patients Group 1 (no SA in the past week): 114 Group 2 (SA in the past week): 43	Group 1: 35.1 (11.8) Group 2: 36.6 (12.6)	Group 1: 36.0% Group 2: 44.2%	D/S IAT
Podlogar et al. [7]	USA	Retrospective association with suicidal behaviors	N = 382 online sample with high demographic suicide risk (i.e., military service members and veterans, men over age 50, and LGBTQ young adults)	Total: 46.23 (19.11)	23.6%	D/S IAT
Rath et al. [51]	Germany	Validation of the D/S IAT and retrospective associations with suicide behaviors	N = 32 adults from general and psychiatric population Group 1 (psychiatric patients with depression and STB): 16 Group 2 (control): 16	Group 1: 35.8 (9.9) Group 2: 35.6 (10.1)	Group 1: 69% Group 2: 69%	D/S IAT
Rath et al. [52]	Germany	Retrospective and prospective associations with suicide attempt	N = 297 psychiatric inpatients Group 1 (inpatients with depressive disorders and lifetime or recent STB): 71 Group 2 (Inpatients with recent SA or severe suicidal crisis): 226	Group 1: 37.4 (14.4) Group 2: 35.4 (14.1)	Group 1: 73% Group 2: 57%	D/S IAT
Richard-Devantoy et al. [28]	Canada	Retrospective association with suicide attempts	N = 79 psychiatric outpatients Group 1: (patients with history of SA and mood disorders): 33 Group 2: (controls with history of mood disorders but no SA): 46	Group 1: 41.6 (10.8) Group 2: 42.8 (11.9)	Group 1: 57.6% Group 2: 65.2%	Suicide Stroop Task
Stewart et al. [29]	USA	Retrospective association with suicide thoughts and behaviors	N = 99 inpatient with recent STB	Total: 15.53 (1.34)	72%	Suicide Stroop Task
Tello et al. [53]	France	Retrospective and prospective associations with suicide attempts	N = 165 adults from psychiatric emergency department Group 1 (not recent SA): 115 Group 2 (recent SA): 50	Group 1: 43.93 (14.51) Group 2: 39.6 (14.63)	Group 1: 48.7% Group 2: 56%	S-IAT
Tucker et al. [16]	USA	Prospective associations with suicide ideation	N = 138 undergraduate students	Total: 19.25	68.12%	S-AMP
Wang et al. [54]	China	Retrospective association of depressives symptoms and the history with suicide behaviors	N = 155 adults from psychiatry department Group 1 (patients with depression): 130 Group 2 (control): 125	Group 1: 23.94 (9.90) Group 2: 25.68 (10.85)	Group 1: 72.30% Group 2: 69.60%	D/S IAT
Wells et al. [31]	USA	Retrospective association with suicidal ideation	N = 79 undergraduate students Group 1 (history of SI): 22 Group 2 (controls): 57	Total: 19.8 (3.3) Group 1: 20.5 (13.8) Group 2: 19.6 (3.1)	Total: 64.6% Group 1: 72.7% Group 2: 61.4%	S-AMP
Williams and Broadbent [20]	UK	Retrospective association with suicide attempts	Experiment 1: Group 1 (adults with recent overdose SA admitted to an emergency department): 25 Group 2 (controls): 25 Experiment 2: Group 1 (adults with recent overdose SA admitted to an emergency department): 18 Group 2 (controls): 25	Experiment 1: group 1: 31 (11.8) Experiment 2: group 1: 33.7	Experiment 1: group 1: 68% Experiment 2: group 1: 61%	Suicide Stroop Task

D/S death/suicide, D-IRAP Death version of the Implicit Relational Assessment Procedure, NSSI non-suicidal self-injury, PTSD posttraumatic stress disorder, SA suicide attempt, S-AMP Suicide Affect Misattribution Procedure, SI suicide ideation, S-IAT Suicide IAT, SI-IAT Self-Injury IAT, STB suicidal thoughts and behaviors

The sample size of the reviewed studies ranged between 32 [51] and 7015 [43]. All studies explored the relationship with past or future suicide behavior. The most common implicit test used was the D/S IAT [6, 12, 36–46, 48–51, 53, 54] followed by the Suicide Stroop Task [20, 25–29, 38]. Most studies employed a control group composed of participants or patients without presenting suicidal behaviors [3, 12, 20, 25–27, 30, 31, 36, 38, 40, 44, 51, 53, 54]. Additionally, four studies used undergraduate students [16, 27, 31, 45]. Three studies used community-based adults [38, 43, 48], one used military service members [39] and one used a high demographic suicide risk sample [6]. The majority of our studies reviewed used adult’s samples, but six studies used a sample with adolescents [3, 29, 40, 42, 44, 48]. Mean age across studies ranged from 14.8 to 49.5.

All but one of the tests reviewed were based on a computerized behavioral task and built their score out of participants’ reaction times employing different algorithms. The exception was the S-AMP, in which time was not registered.

Death and Suicide Implicit Association Test (D/S IAT)

The D/S IAT (Death/Suicide Implicit Association Test [12, 19] is a computer-based test that measures people reaction times when doing a categorization task. The task consists in classifying stimuli of the construct of “death/suicide” (i.e., die, dead, deceased, lifeless, and suicide) and “life” (i.e., alive, survive, live, thrive, and breathing) and the attributes of “me” (i.e. I, myself, my, mine, and self) and “not me” (i.e., they, them, their, theirs, and other). Participants are asked to sort stimuli as quickly as possible to their attribute/construct. Reaction times in correctly classifying the stimuli result in the D-score. Higher D-scores are supposed to indicate greater suicide risk. Although, originally the Death and Suicide IAT was abbreviated as D/S IAT [12], some authors [6, 42, 43, 48] use the abbreviation D-IAT (Death IAT).

Concurrent Validity of the D/S IAT

Four studies have reported the concurrent validity of the D/S IAT [6, 50, 51, 54]. Three studies employed the Beck Scale for Suicide ideation (BSS) as gold standard and found a positive correlation between the D/S IAT and the scale [6, 51, 54]. For their part, the study by Moreno et al. [50] used the Columbia Suicide Scale for Risk Assessment (CSSRS) as gold standard and found a positive correlation.

Retrospective Validity of the D/S IAT

Twenty-four studies explored the retrospective validity of the D/S IAT (see Table 2). Three studies explored the association between D/S score and SI. From this, two found a positive correlation [41, 42], while Chiurliza et al. [39] found no significant association. Regarding the association between the D/S IAT and a history of SA, there are also mixed results. Four studies found a positive correlation with higher D/S IAT score [6, 12, 45, 54]. However, Harrison et al. [46] only found this positive correlation when controlling for the number of past SAs (multiple past SAs, t = 2.12, p < 0.05). In contrast, two studies did not find a significant correlation between the D/S IAT and a history of SA [36, 43]. Moreover, the study by Rath et al. [52] found an inversed association between number past SAs and D/S IAT score (r = − 0.27, p < 0.01).

Table 2

Main findings regarding validity of implicit cognition tests

Study	Test	Outcome	Timeframe	Results	Main findings
Barnes et al. [36]	D/S IAT	SAs	6 m (retro)	NS	Higher D/S IAT score at baseline significantly predicted suicide attempts at 6 months follow-up. But were not significant in discriminate recent suicide attempts
			6 m (pro)	Multivariate: OR = 1.89 (95% CI: 1.15–3.12)**
Barnes et al. [37]	D/S IAT	SAs	6 m (pro)	All sample: OR = 4.30 (95% CI: 1.15–3.12)*** No PTSD: OR = 1.89 (95% CI: 1.42–13.0)** PTSD: NS	D/S IAT score was associated with likelihood of attempting suicide at 6 months follow-up in the full sample and in participants without PTSD but not in participants with PTSD
Becker et al. [25]	Suicide Stroop Task	SAs	Last year (retro)	Suicide words stimulus: F = 7.78, f = 0.14*** Negative words stimulus: NS Neutral words stimulus: NS	Participants with history of suicidal thoughts and behaviors took longer in naming color in suicide-related words compared with controls
Cha et al. [26]	Suicide Stroop Task	SAs	Lifetime (retro)	Suicide words stimulus: OR = 1.01 (95% CI: 1.00–1.01)* Negative words stimulus: NS Neutral words stimulus: NS	Suicide-specific attentional bias (greater interferences in suicide-related words) was positively associated with participants with history of SA but no with negative nor neutral words stimulus
			6 m (pro)	Suicide words stimulus: OR = 1.02 (95% CI: 1.00–1.03)* Negative words stimulus: NS Neutral words stimulus: NS	Suicide-specific attentional bias was positively associated with future suicide attempt
Cha et al. [38]	D/S IAT	SI	1 m (pro)	Bivariate: Pre-mood induction: OR = 4.56 (95% CI: 2.07–10.05)*** Post-mood induction: OR = 3.86 (95% CI: 1.89–7.85)*** Multivariate: Pre-mood induction: OR = 3.38 (95% CI: 1.26–9.10)* Post-mood induction: OR = 2.80 (95% CI: 1.19–6.57)*	D/S IAT score was associated with suicidal ideation at 1 month and at 6 months follow-up, both pre- and post-negative mood induction. Suicide Stroop task was not associated to suicidal ideation
			6 m (pro)	Bivariate: Pre-mood induction: OR = 2.96 (95% CI: 1.33–6.56)** Post-mood induction: OR = 4.47 (95% CI: 2.07–9.62)*** Multivariate: Pre-mood induction: NS Post-mood induction: OR = 2.77 (95% CI: 1.04–7.40)*
	Suicide Stroop Task		1 m (pro)	NS
			6 m (pro)	NS
Chiurliza et al. [39]	D/S IAT	SI	Lifetime (retro)	NS	D/S IAT score was not associated with a history of suicidal ideation
Chung and Jeglic [27]	Suicide Stroop Task	SAs	Lifetime (retro)	Suicide words stimulus: t = 2.18* (before controlling for mood symptoms) Suicide words stimulus: NS (after controlling for mood symptoms) Negative words stimulus: NS Neutral words stimulus: NS	In comparison to controls, participants with past suicide attempts showed greater interferences in suicide-related words but no significant when controlling for mood symptoms and no differences in the negative nor neutral-related words condition
Dickstein et al. [40]	SI-IAT	SA	1 m (retro)	Cutting subconstruct: NS D/S subconstruct: NS	Participants with past SA did not show significant differences in SI-IAT score compared with controls and NSSI groups
		NSSI	1 m (retro)	Cutting subconstruct: compared with TDC: Cohen’s d = 1.13** Compared with SA group: Cohen’s d = 1.06** D/S subconstruct: compared with TDC: Cohen’s d = 0.74** Compared with SA group: Cohen’s d = 0.56*	Participants with past NSSI had significantly higher score in SI-IAT (both in cutting and in D/S subconstruct) compared with controls and SA groups
Ellis et al. [41]	D/S IAT	SI	2 m (retro)	R = 0.50**	D/S IAT score was positively associated with past suicide ideation
			Average of 6 w (pro)	From admission to discharge: Cohen’s d = 0.27**	Change in D/S IAT performance were sensitive to change over the course of treatment and was significantly associated with SI at discharge
Glenn et al. [42]	SI-IAT	NSSI	Lifetime (retro)	t = 12.34*** AUC = 0.72 (95% CI: 0.69–0.75)***, sensitivity = 0.53, specificity = 0.82, PPV = 0.82, PPN = 0.59	SI-IAT score was higher in participants with previous NSSI and controls compare with participants without a history of NSSI nor controls SI-IAT was sensitive to recency and severity of NSSI
		STB	Recent (retro)	T = 7.55***
		NSSI		F = 161.10***
	Death IAT	NSSI	Lifetime (retro)	t = 8.09***	Death and S-IAT scores were significantly higher in individuals with a history of STB
		STB		t = 5.32***
	S-IAT	NSSI		t = 9.00***
		STB		t = 8.16*** AUC = 0.64 (95% CI: 0.60–0.67)***, sensitivity = 0.27, specificity = 0.87, PPV = 0.47, PPN = 0.73
Glenn et al. [43]	D/S IAT	SI	1 m (retro)	r = 0.20***	D/S IAT score was associated with past SI, but not with past SA At discharge, D/S IAT score significantly predicted SI among participants with longer treatment stay in psychiatric inpatient facility
			At discharge (pro)	Short treatment stay (± 9 days): NS Longer treatment stay (± 20 days): b = 5.50***
		SA	Lifetime (retro)	NS
Glenn et al. [44]	Death IAT	SI	12 m (retro)	OR = 1.71 (95% CI: 1.18–2.50)*	Death IAT scores was positively associated with the occurrence of past SI. And was significantly associated with occurrence of SI at 6 and 12 months follow-up
			6 m (pro)	OR = 1.48 (95% CI: 0.98–2.23)*
			12 m (pro)	OR = 1.58 (95% CI: 1.05–2.40)*
		SA	12 m (pro)	AUC = 0.85 (95% CI: 0.64–0.99)*	Death IAT scores were higher among participants who had an SA at 1 year follow-up
Harrison et al. [45]	D/S IAT	SI frequency	Lifetime (retro)	Multivariate: NS Bivariate: b = − 0.13**	D/S IAT scores were no longer associated with SAs or SI after controlling for SCB-S scores
		SI intensity		Multivariate: NS Bivariate: b = − 0.14*
		SA		Multivariate: NS Bivariate: b = − 0.14**
Harrison et al. [46]	D/S IAT	SA	Lifetime (retro)	Single: NS Multiple: t = 2.12*	D/S IAT score were higher in participants who had made multiple SAs D/S IAT score was not prospectively associated with SA and SI at the 3-month follow-up
			3 m (pro)	NS
			6 m (pro)	NS
		SI	3 m (pro)	NS
			6 m (pro)	NS
Hussey et al. [30]	D-IRAP	SI	Past week (retro)	Abstract version: “Death-negative”: NS Personal version: “My death-negative”: OR = 10.50 (95% CI: 2.34–47.03)**, sensitivity = 0.86, specificity = 0.64	Self-focused negative evaluations of death on the personal version of the D-IRAP significantly differentiated between individuals with SI and controls whereas evaluations of abstract death did not differentiate
Kene [47]	SI-IAT	SA	Lifetime (retro)	Identity version: NS Attitude version: NS	No significant differences between attempters and non-attempters on both attitude and identity version of the SI-IAT
Millner et al. [48]	Death IAT	SA	3 m (pro)	NS	After correction between-group differences in SI-IAT scores were not significant
	Suicide IAT		3 m (pro)	NS
	SI-IAT		Lifetime (retro)	NS
			3 m (pro)	NS
Millner et al. [49]	D/S B-IAT	SA	Lifetime (retro)	6 blocks: Cohen’s d = 0.43 (95% CI: 0.30–0.55)*, sensitivity = 0.46, specificity = 0.70 4 blocks: Cohen’s d = 0.38 (95% CI: 0.25–0.51)*, sensitivity = 0.45, specificity = 0.69	In comparison with the D/S IAT, D-BIAT had lower accuracy at distinguishing participants with past SA compare with participants without history of SA
			12 m (retro)	6 blocks: Cohen’s d = 0.46 (95% CI: 0.23–0.69)*; sensitivity = 0.59; specificity = 0.57 4 blocks: Cohen’s d = 0.47 (95% CI: 0.19–0.68)*, sensitivity = 0.62, specificity = 0.60
Moreno et al. [50]	D/S IAT	SA	3 m (pro)	OR = 73.50 (95% CI: 10.43–517.73)***, sensitivity = 0.89, specificity = 0.94, accuracy = 0.93	The D/S-IAT scores were prospectively associated with the likelihood of SA at 3-month follow-up
Nock and Banaji [3]	SI-IAT	SI	12 m (retro)	OR = 5.59 (95% CI: 2.09–26.98)***	SI-IAT score were significantly higher in participants with past STB. SI-IAT scores were prospectively associated with likelihood of attempting suicide at 6-month follow-up
			6 m (pro)	OR = 6.42 (95% CI: 1.19–34.58)*
		SA	12 m (retro)	OR = 10.91 (95% CI: 2.66–45.86)***
Nock et al. [12]	D/S IAT	SA	7 d (retro)	OR = 6.38 (95% CI: 1.02–39.93)*	Higher D/S IAT scores were retrospectively associated with recent SA, and prospectively associated with likelihood of SA at 6-month follow-up
			6 m (pro)	Continuous: OR = 30.68 (95% CI: 1.18–795.12)* Dichotomous: OR = 5.88 (95% CI: 1.32–26.26)*, sensitivity = 0.50, specificity = 0.81, PPV = 0.32, PPN = 0.90
Podlogar et al. [7]	D/S IAT	SAs	Lifetime (retro)	Before controlling BSS: OR = 1.58 (95% CI: 1.32–1.9)*** After controlling BSS: OR = 1.35 (95% CI: 1.12–1.6)* At least one SA: AUC = 0.64 (95% CI: 0.57–0.71) Multiple SA: AUC = 0.73 (95% CI: 0.64–0.82)	D/S-IAT score was associated with severity of past STB before and after controlling current SI
Rath et al. [51]	D/S IAT	SI	Lifetime/including current (retro)	NS	D/S IAT scores were not significantly different between depressed participants compare with controls
Rath et al. [52]	D/S IAT	SI	Lifetime	NS	D/S IAT score did not significantly differentiate between participants with lifetime SI nor lifetime single or multiple SA history D/S IAT score were inversely associated with number of SA
		SAs	Lifetime (retro)	Single SA: NS Multiple SA: r = − 0.27**
			3 m (pro)	r = 0.18*	D/S IAT score correlated positively with likelihood of SA at 3-month follow-up but not at 6 nor 12 months
			6 m (pro)	NS
			12 m (pro)	NS
Richard-Devantoy et al. [28]	Suicide Stroop Task	SAs	Lifetime (retro)	NS	Interference in suicide-related words was not associated with history of suicide attempts
Stewart et al. [29]	Suicide Stroop Task	SAs	Lifetime (retro)	Suicide word stimuli: Cohen’s d = 0.41* Negative word stimuli: NS Positive word stimuli: Cohen’s d = 0.53**	In comparison to participants with SI, participants with recent showed greater interference in suicide related word and positive related word
Tello et al. [53]	S-IAT	SAs	Lifetime (retro)	NS	S-IAT score did not differentiate between participants with history of SA compared with controls
			7 d (retro)	M = − 0.55 (95% CI: − 0.64, − 0.45)	Higher S-IAT score in participants with a recent SA compared with participants with lifetime SA and control groups
			6 m (pro)	OR = 5.58 (95% CI: 1.45–21.51)*, accuracy = 0.85, sensitivity = 0.94, specificity = 0.94, PPV = 0.55, NPV = 0.89	Higher S-IAT scores were associated with likelihood of SA at 6-month follow-up before and after controlling other predictors
Tucker et al. [16]	S-AMP	SI	15 d (retro)	0R = 1.19 (95% CI: 1.05–1.34)*	Suicide ideation was positively associated to suicide S-AMP scores
Wang et al. [54]	D/S IAT	SA	Lifetime (retro)	F = 7.65***	Higher D/S IAT score in participants with past SA than participants without SA
Wells et al. [31]	S-AMP	SI	Lifetime (retro)	Cohen’s d = 0.48*	S-AMP score did significantly differentiate between participants with history of SI than those without history of SI
Williams and Broadbent [20]	Suicide Stroop Task	SAs	Recent median of 39 h (retro)	Experiment 1: t = 2.04* Experiment 2: t = 2.19*	In both experiment (panel presentation) participants with recent suicide attempt showed a suicide-specific attentional bias (greater interferences in suicide-related words) in comparison with controls

BSS Beck Scale for Suicide Ideation, d days, D/S death/suicide, D-IRAP Death version of the Implicit Relational Assessment Procedure, m = month, NS nonsignificant, pro prospective, retro retrospective, SA suicide attempt, S-AMP Suicide Affect Misattribution Procedure, SBQ-R Suicidal Behaviors Questionnaire-Revised, SCBS The Survival and Coping Beliefs subscale, SI suicide ideation, S-IAT Suicide IAT, SI-IAT Self-Injury IAT, STB suicidal thoughts and behaviors, w week

*p < 0.05, **p < 0.01, ***p < 0.001

Prospective Validity of the D/S IAT

Regarding the prospective validity of the D/S IAT, eight out of eleven studies found a positive correlation between the D/S IAT score and the occurrence of suicidal thoughts and behaviours [12, 36–38, 41, 43, 44, 50] (see Table 2). For instance, in the study by Nock et al. [12], the likelihood of committing a SA at 6-month follow-up was higher when the D/S IAT score at baseline was above 0. Two studies [41, 42] explored how changes in D/S IAT scores could be associated with treatment received. Ellis et al. [41] found that the D/S IAT changed throughout the course of the treatment and that it significantly predicted SI at discharge (Cohen’s d = 0.27, p < 0.01). In the study by Glenn et al. [42], D/S IAT score significantly predicted SI at discharge among participants with a long stay in a psychiatric inpatient facility (b = 5.50, p < 0.001), but not in participants with a short stay. In the study by Rath et al. [52], they found a positive correlation when predicting SAs at 3-month follow-up (r = 0.18, p < 0.05) but not at 6 nor 12-month follow-up. Finally, two studies did not find any positive correlation between the D/S IAT score and the occurrence of future suicidal thoughts and behaviors [46, 48].

Test Reliability of the D/S IAT

Four studies explored the D/S IAT test reliability. Rath et al. [52] found an r of 0.22, and Millner et al. [48] found a Cohen’s d of 0.76 (0.74–0.78). Glenn et al. [43] reported good internal replication, without specifying numeric results. The test also showed good consistency in the study of Harrison et al. [49] with an r of 0.85.

D/S IAT Variants

Some studies used variants of the classic D/S IAT. For instance, Millner et al. [48] used the Suicide-IAT, which includes specific suicide methods stimuli (i.e., gunshot, hanging, overdose, cutting). They also included other minor variations, such as the Death Single Category–IAT (DSC-IAT), the Suicide Single Category–IAT (SSC-IAT), and the SSC-IAT with pictures. In this study, the found no significant association between the scores of the different tests used and past or future SAs [48].

There is also a brief version of the D/S IAT created by Millner et al. [49] and called the D/S B-IAT which have shown a good accuracy, though smaller than the original D/S IAT [49]. Tello et al. [53] and Glenn et al. [43] also used the Suicide-IAT. Tello et al. [53] found a high internal consistency of the test and a positive correlation with SAs at 6-month follow-up. In contrast, scores were not significantly associated with past SAs. Whereas, Glenn et al. [43] found a positive correlation with a history of suicidal thoughts and behavior (see Table 2).

D-IRAP

The IRAP is a computer-based test that measures the reaction time when participants are asked to paired stimulus under speed and accuracy pressure. The IRAP is based on the Relational Frame Theory (RFT) [55]. The premise of RFT is that the foundation components of cognition are relational rather than associative [30, 56].

In the study by Hussey et al. [30] they defined two separate death-evaluation IRAPS, the “personal IRAP” and the “abstract IRAP.” The “personal IRAP” is based on a theoretical supposition that suicide differs from homicide by the presence of a desire to die [57]. The “personal IRAP” included a reference to self (i.e., “my death” or “my life) whereas the “abstract IRAP” did not include that reference (i.e., “death” or “life”). In both tasks, the label stimuli (“my death” or “death”) are presented with either a positive (i.e., pleasant, enjoyable, lovely) or a negative target stimulus (i.e., horrible, upsetting, painful). Participants are asked to pair a label and a target stimulus on each trial as fast as possible. Time is registered in millisecond generating a D-IRAP algorithm, which is considered a variant of the D algorithm for the IAT [23].

Only one study (see Table 2) tested the D-IRAP and its association with suicidal thoughts and behaviors [30]. Authors found that recent SI was positively associated with higher scores in “my death-negative” trial from the personal version (self-focused) and could differentiate participants from the SI or control group (OR = 10.50, 95% CI: 2.34–47.03). However, the “death-negative” trial from the abstract version did not significantly distinguish between people with and without SI.

SI-IAT

The Self-Injury IAT (SI-IAT) follows the same theory that the D/S IAT, but constructs are oriented to self-injury, such as cutting. Participants are asked to categorize stimuli of the construct of “escape versus stay” (i.e., leave, quit versus hold on, remain, etc.), “cutting versus no cutting” (i.e., images of cut versus intact skin), “suicide versus life” (overdose, hanging versus live, survive, etc.), and “death versus life” (die, funeral versus live, survive, etc.) and the attributes of “me” (i.e., I, myself, my, mine, and self) and “not me” (i.e., they, them, their, theirs, and other) [3].

In the original study (see Table 2) by Nock and Banaji [3] higher SI-IAT score were positively associated with both past occurrence of suicidal thoughts and behaviors (OR = 5.59, 95% CI: 2.09–26.98) and likelihood of SAs at 6-months (OR = 6.42, 95% CI: 1.19–34.58) and 12-month follow-up (OR = 10.91, 95% CI: 2.66–45.86). Subsequent studies found a positive association between SI-IAT score and a history of non-suicidal self-injury [40, 43]. Suicide behavior, whether past or future, was not associated with the SI-IAT in the reviewed studies [47, 48].

S-AMP

The Suicide Affect Misattribution Procedure (S-AMP) is a computerized-based task that relies on participants’ misattributions based on their emotions, cognitions, and perceptions of themselves. The test assesses how people misattribute the relevance of a target stimulus after being presented with a prime stimulus. Participants are presented with four types of prime images: related to suicide (i.e., dead bodies provoked by suicide via overdose, firearm or overdose), negative but non-related to suicide (i.e., insects crawling on half-eaten food items), neutral (i.e., umbrella or barstool), and positive (i.e., children laughing and playing, flowers and blue skies). After the prime stimuli, participants are asked to quickly rate in a 4-point scale how the target stimulus (i.e., a Chinese pictograph) fits their self-concept (i.e. 1 = doesn’t fit me well, 2 = fits me a little, 3 = moderately fits me, 4 = fits me well). Higher scores represent greater self-identification with the prime stimuli. Thus, the test assesses how the prime influences the affective categorization of the target stimuli.

Two studies (see Table 2) used the S-AMP test [16, 31]. In the study by Tucker et al. [16], they found higher self-identification with suicide prime stimulus in participants with recent SA (OR = 1.19, 95% CI: 1.05–1.34). Similarly, Wells et al. [31] found a positive correlation between the S-AMP score and a history of SI (Cohen’s d = 0.48, p < 0.05).

Suicide Stroop Task

The classic Stroop task is a computer-based test that analyses the time it takes a person to identify the font color of words that indicate colors. Sometimes the word matches the font color and sometimes it does not, which increases the need to pay attention. The test assumes that the participant’s ability is interfered by the emotional salience of each word, creating an attentional bias [25]. In the Suicide Stroop test, colored words are relevant to the construct of death and suicide (i.e., death, suicide, funeral) and are compared to negative words (i.e., rejected, stupid, alone) and neutral words (i.e., museum, paper, engine). Larger response latencies are interpreted as an indicator of greater interference due to the semantic content of the words [26].

Seven studies (see Table 2) explored the validity of the Suicide Stroop Task [20, 25–29, 38]. In four studies, participants with a history of suicide behaviors showed greater interferences with suicide-related words stimulus [20, 25, 27, 38]. In the study by Stewart et al. [29], they found greater interferences in suicide and positive word in adolescents with a history of SAs compared with those with SI. In the study by Cha et al. [26], they found that the suicide words stimulus score was associated with both history of SAs (OR = 1.01, 95% CI: 1.00–1.01) and the occurrence of future SAs at 6-month follow-up (OR = 1.02, 95% CI: 1.00–1.03). However, two studies did not found any significance between the score in the Suicide Stroop Task and SI at 1 and 6-month follow-up [38], nor with a history of lifetime SAs [28].

Discussion

In this systematic review, we explored the evidence about the validity of death-related implicit cognition tests. We found that death-related implicit cognition tests were generally aimed on assessing past suicide thoughts and behaviors and showed great past prediction. However, only a few studies have tried its validity for future suicide thoughts and behaviors, and the most used test for this purpose was the D/S IAT showing adequate predictive power. Has shown in a recent meta-analysis of the discriminative and prospective utility of the D/S IAT, the authors stated that, although this test has sufficient predictive value, clinical decisions should not be based exclusively on it [32].

Stimuli and Performance

The death-related implicit cognition tests reviewed share the same aim to assess automatic attitudes and cognitions associated with suicidal thoughts and behaviors. However, they do so through different tasks and stimuli. On the one hand, the Suicide Stroop Task and the S-AMP aim to assess the implicitness by measuring the degree of interference that stimuli can generate, either by presenting a prime (a stimulus presented just before another) or by presenting two distinctive cues in a single stimulus (the color/meaning of a word). Therefore, the outcome of these tests will depend on a person’s cognitive ability to perceive and process these stimuli/cues to respond to the assigned task. On the other hand, the D/S IAT, the SI-IAT and the D-IRAP try to access the implicitness through the individual’s ability to assign words to the semantic field to which they belong, which requires knowing the meaning of each word. In addition, the D-IRAP, unlike the IATs, includes a referential component on the task (a reference to the self in association to words, such as “My death – positive/negative”), which could add complexity to the task because it requires an additional component. In this line, it seems convenient to evaluate at the neurobiological level whether the simplicity or complexity of elements included in death-related implicit cognition tests influence on individual performance and thus may affect the predictive capacity of the test.

Neurobiological and Cognitive Correlates

The mechanisms underlying the accuracy of implicit cognition tests have not yet been fully elucidated. Recent neuroimaging studies have found neuronal correlates with the IAT [58•, 59]. In one study, they used an adapted functional magnetic resonance imaging (fMRI) version of the S-IAT. Imaging results detected an increased activation in the bilateral insula, medial prefrontal cortex, middle occipital cortex, and parahippocampal gyri during the self-death condition compared with the self-life condition in the test [58•]. In another study, connectivity changes between the early visual cortex, amygdala, and anterior insula were observed during the S-IAT, and increased connectivity discriminated between participants with and without recent SAs [59].

Additionally, a study discovered that adults with mood disorders and a history of SAs had changes in default mode and basal ganglia activity during the emotional face-word Stroop task [60]. Furthermore, the original Stroop task has been related to the attentional bias, which is implicated in activating the specific brain region associated with affective disorders and plays a role in predicting future suicidal behavior [61, 62]. In the previous study by Thompson et al. [63], participants with a history of SA showed reduced activity in leftward frontal areas during the performance of the emotional Stroop task, which would suggest difficulties in the ability to regulate emotional processing. Moreover, participants with a history of SAs have, on average, worse memory, more impulsivity, and less cognitive flexibility [64•, 65]. According to these results, using the affective go/no-go performance, such dysfunctions were found in participants with current or past suicide thoughts and behaviors but were greater in participants with current SI [66•]. How exactly these findings affect the prediction of suicide behavior is still unknown, and future research is needed to explore brain regions and neural networks associated with suicide risk, which will aid in mapping the suicidal mind [67].

Clinical Implications

Death-related implicit cognition tests may be useful in assessing suicidal behavior by overcoming certain limitations of explicit methods. As argued in the study by Baucom et al. [68•] implicit cognition tests do not only not rely on the direct disclosure of suicide intention by patients, but also they do not require full face-to-face interaction with a clinician. Thus, the test is suitable for people who may be reluctant to express their suicidal thoughts or resistant to take a traditional suicide risk assessment with another person. Moreover, the tests we analyzed could be used in clinical settings when there are concerns that patients may be withholding crucial information that affects clinical decisions. In addition, implicit measures of suicide have the potential to be highly portable and are easy to administer. Accordingly, they could be applied by different kind of professionals in a large range of settings, as they do not require specific training, which makes these measures suitable not only for clinical, but also for research purposes.

Ease of implementation would allow clinicians and researchers to quickly assess suicide risk in real-time, particularly when time is restrained (i.e., in an emergency department). Findings indicate that screening methods did not identify many people who died by suicide, nor declared explicitly their suicide intentions [8, 69]. Hence, using an implicit cognition assessment method may allow to capture a more accurate information regarding suicide risk.

Although the tasks of these tests are relatively simple and do not require complex cognitive processing, some people with attentional or behavioral difficulties, such as high states of agitation in episodes of anxiety or aggressive non-cooperative behaviours, may impede the correct performance of these tests and therefore bias their results. Moreover, people with basic language difficulties or diminished cognitive capabilities should not be assessed through these tests. Professionals administering these measures must first consider the participants’ condition and ability to perform an automatic task in the required time.

This, despite their advantages, implicit measures should be administered combined with other well-known explicit measures to detect suicide risk accurately.

Future Lines of Research

All implicit cognition tests were computerized and consisted of relatively simple software. One of the fronts to be explored in the future could be integrating this software into smartphones in the form of mobile applications. This could increase the portability of the tests, thus facilitating their use outside the hospital environment—for example, for following-up patients after discharge. Mobile technology is increasingly being used for suicide prevention [70]. Implicit cognition tests could be a valuable addition to this field. Additionally, implicit cognition tests could be combined with other mobile health assessment tools, such as Ecological Momentary Assessment (EMA). EMA is based on self-report by users in their usual environment, without the direct supervision of a clinician, and is being increasingly employed in suicide research [71].

Strengths and Limitations

This is the first study to review all the available implicit cognition tests for the assessment of suicide risk. Among the limitations of the review, the heterogeneity of the studies—with different study designs, population, and implicit tests used—precluded performing a quantitative synthesis of the results. One of such limitations is that results regarding their psychometric properties of each test, such as their reliability and validity, are scarcely reported by the reviewed studies. Accordingly, future research shall address this gap.

Conclusions

Our review shows that the implicit cognition tests generally have good concurrent, retrospective, and prospective validity. However, an important caveat on the available research reviewed is that results on psychometric properties are scarce, limiting our capacity for a comprehensive analysis of all these tests. Nonetheless, overall, we can conclude that implicit cognition tests represent a potentially helpful tool for assessing suicide risk and could be an important complement to traditional measures used in clinical practice. Among their advantages are the ease of administration and interpretation, the fact that they do not depend on the explicit suicide intentions self-reported by the patient, and the fact that they yield an objective score that is not influenced by the subjectivity of the evaluator. Despite these benefits, there are still limitations, such as the lack of knowledge of their underlying mechanisms and the lack of familiarity of clinicians with this type of test, which may delay their implementation in routine clinical practice.