Lifeline Crisis Chat: Coding form development and findings on chatters' risk status and counselor behaviors.

OBJECTIVE: This study aimed to develop a reliable tool for the abstraction of data from crisis chat transcripts; to describe chatters' suicide risk status and selected counselor behaviors; and to examine the relationship of chatters' self-reported pre-chat suicidal thoughts to counselor behaviors and to chatters' disclosures of suicide risk during the chat conversation.
METHODS: Coders used an instrument developed for this study to abstract data from 1034 crisis chats handled by the National Suicide Prevention Lifeline Crisis Chat network in 2015. The relationship of transcript coding data to data from an automated pre-chat survey (PCS) was examined.
RESULTS: Lifeline Crisis Chat serves a young (median age = 21), high-risk population: 84.0% of chats (869/1034) came from chatters endorsing current or recent suicidal thoughts on the PCS. Counselors engaged in rapport-building on 93.3%, problem-solving on 70.1%, and suicide risk assessment on 67.7% of these 869 chats. Counselor risk assessment behavior, and the availability of information on suicide risk in the chat transcript, varied significantly by the chatter's PCS response.
CONCLUSION: Crisis counselors are able to implement keystones of Lifeline's crisis intervention model over the medium of online chat. Additional efforts are needed to ensure that suicide risk is assessed on every chat.

INTRODUCTION

Formally established in 2004, the National Suicide Prevention Lifeline (Lifeline or NSPL) (www.suicidepreventionlifeline.org) is a national network of over 180 crisis centers located in every state in the USA and accessible via the number 1‐800‐273‐TALK. The impact and effectiveness of telephone crisis services, such as those provided by Lifeline, are well‐established (see Halford et al., 2021 for a recent review). Crisis callers’ suicidality and emotional distress have been found to decrease from the beginning to the end of the crisis call (Gould et al., 2007; King et al., 2003; Mishara & Daigle, 1997); moreover, specific counselor behaviors, including building rapport by offering emotional support (“good contact”), and collaborative problem‐solving, have been linked to better caller outcomes (Mishara et al., 2007b), leading to their adoption as Lifeline best practices. This research has lent support to the ubiquitous use of Lifeline number as a suicide prevention resource, including in the U.S. Surgeon General’s 2012 National Strategy for Suicide Prevention, and to recent legislation designating 988 as the new dialing code for Lifeline beginning in July 2022. Evaluations have also identified the need for continued improvement, notably in the area of suicide risk assessment, which has not been observed to be conducted on all calls as would be consistent with Lifeline policy (Mishara et al., 2007b). A study of ten crisis lines in California found that current suicidal ideation was assessed on calls to Lifeline centers significantly more frequently than on calls to non‐Lifeline centers (77% vs. 52% of calls where the caller did not volunteer this information), with Lifeline centers nevertheless falling short of the standard of 100% (Ramchand et al., 2017).

In recognition of a cultural shift, particularly among younger people, toward digital forms of communication, crisis lines and helplines across the globe have begun providing services via text‐based platforms such as online chat and text (e.g., Kids Help Line in Australia (King et al., 2006); Kids Help Phone in Canada (Haner & Pepler, 2016); SAHAR (Barak & Bloch, 2006) and ERAN (Gilat & Shahar, 2007) in Israel; Kindertelefoon (Fukkink & Hermanns, 2009a, 2009b) and 113Online (Mokkenstorm et al., 2017) in the Netherlands; Nevada Crisis Call Center (Evans et al., 2013) and the Veterans Crisis Line (Predmore et al., 2017) in the United States). The Lifeline Crisis Chat (LCC) network began as a pilot in 2012 and was formally established in 2013. Currently, over 30 Lifeline centers handle crisis chats as well as calls, with the network responding to over 900,000 chats to date. Evaluations of chat services have found that users of crisis chat tend to be younger than callers to telephone crisis lines, and that crisis chatters are more likely than crisis callers to present severe mental health issues including suicidality (Fukkink & Hermanns, 2009a, 2009b; Haner & Pepler, 2016; Mokkenstorm et al., 2017). Interventions conducted over online chat are found to last longer on average than those conducted over the telephone (Fukkink & Hermanns, 2009b; Mokkenstorm et al., 2017). Counselors who conduct both text‐based and telephone interventions report that information gathering via text is more time‐consuming and that assessing and understanding clients’ problems is more difficult in the absence of non‐verbal cues (Bambling et al., 2008). Perhaps due to these challenges, text‐based interventions have been found to concentrate more on rapport‐building, information gathering, and storytelling, and less on goal exploration and action planning (Chardon et al., 2011; Mallen et al., 2011), and generally to use more person‐centered than problem‐centered interventions (Fukkink, 2011).

Few existing evaluations of crisis chat and other single‐session online counseling interventions have examined chatters’ suicide risk or counselors’ risk assessments. Mokkenstorm et al. (2017) used a coding scheme based on Mishara’s Silent Monitoring Study of Telephone Helplines (Mishara et al., 2007a, 2007b) to code transcripts of 526 chats to Amsterdam’s 113Online crisis chat service. Over 85% of chatters were identified as being in a suicidal, rather than non‐suicidal crisis, based on a combination of the chat content and the chatter’s answers to an optional pre‐chat questionnaire. Information on chatters’ suicidality in the first and last 10 minutes of the chats was missing for 64% of chats due to suicide not being discussed during those times, an omission the authors identify as being of great concern. Counselor behaviors in the domain of suicide risk assessment were not examined in this study.

The first aim of the current study was to develop a reliable methodology for the evaluation of LCC services, by adapting to this new medium the methodologies used in earlier hotline evaluations where coders listened in on live crisis calls and abstracted data in real time (“silent monitoring”; see e.g., Gould et al., 2013; Mishara & Daigle, 1997). Our Crisis Chat Transcript Abstraction Form is the first instrument developed specifically for the evaluation of text‐based crisis interventions to assess an extensive range of both chatter and counselor behaviors. The current study then used this newly developed methodology to abstract data from the transcripts of 1034 LCC chat interventions, focusing on chatters’ suicide risk status and counselor behaviors in the domains of risk assessment, rapport‐building, and collaborative problem‐solving. In light of earlier findings of inconsistencies in counselor risk assessment behavior in telephone crisis interventions (see e.g., Mishara et al., 2007a), of Lifeline’s efforts to improve and standardize these assessments (see e.g., Gould et al., 2013), and of evidence that crisis chatters tend to disclose higher levels of suicide risk than crisis callers (see e.g., Fukkink & Hermanns, 2009a), it is important to know to what degree suicide risk is assessed on Lifeline crisis chats. In light of challenges to rapport‐building in the absence of non‐verbal cues, and of earlier findings regarding a de‐emphasis on problem‐solving in text‐based interventions, the study further aimed to assess the degree to which these behaviors, cornerstones of Lifeline's crisis intervention model, appear in crisis chats. The availability of self‐report data on suicidal ideation from Lifeline's mandatory pre‐chat survey (PCS), data which are not available in the case of crisis calls, provided the unique opportunity to compare this data with information on chatters’ suicide risk discussed during the chat conversation, and to examine to what extent a chatter's response to the PCS question about suicidal thoughts can serve as a guide to the chatter's current need for a suicide‐focused intervention. Finally, the study examined the potential impact of the chatter's PCS self‐report regarding suicidal thoughts on counselors’ rapport‐building, problem‐solving, and risk assessment behaviors during the chat.

This study’s aims complement those of a recently published evaluation by our team which examined 13,130 linked pre‐ and post‐chat surveys completed by LCC chatters in 2017–2018 and found that two thirds of suicidal chatters reported finding the chat helpful, with just under half reporting they were less suicidal by the end of their chat (Gould et al., 2021).

METHODS

Sample

In 2015, the LCC network ranged between 23 and 25 participating centers. A stratified random sample of crisis chat interventions conducted by any center in the network was included in our study. The sample was stratified to reflect the proportion of chats coming in over Lifeline's two chat platforms, as a proxy for center, and was stratified such that in each month, half of the transcripts were from chats where the chatter had completed an optional post‐chat survey following the chat. (Stratification of the sample by post‐chat survey completion will enable a comparison of chatters who did and did not complete the post‐chat survey, the subject of a future paper.). Transcripts of crisis chat interventions were randomly selected by Lifeline staff subject to these stratifications and were de‐identified by Lifeline staff before being provided to the research team. For each month between July 2015 and December 2015, the Lifeline provided 210 chat transcripts, for a total of 1260 transcripts across 6 months. All transcripts included the chatter's responses to the PCS. Additional de‐identified transcripts provided by Lifeline were used for instrument development, coder training, and reliability testing purposes.

In the sample of 1260 transcripts provided by Lifeline for coding, two chats were included twice, yielding a total of 1258 unique transcripts. Of these, 224 transcripts (17.8%) were excluded from coding: 92 (7.3%) where the chatter left before speaking with a counselor; 89 (7.1%) deemed too short for coding due to their being <10 conversational turns long (a “turn” was defined as a timestamped message sent by either party, regardless of whether the speaker changed); 30 (2.4%) where the chatter was chatting out of concern for someone else (third‐party chats); 2 (0.2%) with no recorded answer to the PCS question about the chatter's suicidal ideation, which was essential to our analyses; and 11 (0.9%) excluded for other reasons, including technical glitches and, in one case, expressed confidentiality concerns. This left 1034 chat transcripts eligible for coding. Within the coded sample, the chatter had completed the optional post‐chat survey on 551 chats (53.3%), consistent with our stratification of the sample by post‐chat survey completion, described above.

Measures

Lifeline's Pre‐Chat Survey (PCS)

This survey, routinely administered by Lifeline to all individuals entering a Lifeline crisis chat, assesses the chatter's age, gender, and response to the question “Do you have thoughts of suicide?” (answer options = Yes, current (within the past 24 hours)/Yes, recent past (within the past few days)/No). (Please note that throughout this paper, chats will be grouped on the basis of the chatter's response to the PCS question, “Do you have thoughts of suicide?”, as follows: chatters who respond “Yes, current” are classified as CS (currently suicidal); chatters who respond “Yes, recent past” are classified as RS (recently suicidal); and chatters who respond “No” are classified as NS (not suicidal), according to their PCS response.)

Crisis Chat Transcript Abstraction Form

An instrument designed to abstract information from de‐identified transcripts of crisis chat interventions was developed for the current study (see “Procedures” below). The form is coded by two independent teams of trained coders, one team coding chatter characteristics and behaviors and a second team coding counselor behaviors (see Table 1 for a full list of items). When coding the chatter's current suicidal ideation and other suicide risk indicators, transcript coders were instructed to base their codes only on evidence from the body of the chat (i.e., the conversation between the chatter and the counselor), excluding information from the chatter's responses to the PCS, which were appended to the chat transcript and thus visible to the coders. A code of “Don't Know” (DK) for a suicide risk indicator indicates that the indicator either was not discussed during the chat or was discussed in such a way that clear and unambiguous information was not obtained. By contrast, chat counselor behaviors were coded as either present at any time during the chat, or absent from the chat, without the option of a “Don't Know” response.

TABLE 1

Data elements included on Crisis Chat Transcript Abstraction Form

Part I: Chatter Characteristics and Behaviors a Structural Overview: chat start date; start time; end time; length of chat in turns a Demographics: sexual minority status; military status; whether the center had knowledge of the chatter from prior chats Life Stressors Present and Discussed During Chat: breakup of intimate relationship; loneliness/isolation; sexual abuse; bullying; other violence or abuse (including physical and emotional); other interpersonal/relationship problem; death of someone close to them; exposure to suicide or to suicidal threats/behavior; financial problem/worries; financial barrier to mental health treatment access; problem with health insurance; unemployed/lost job; job stress (other than financial or interpersonal); problem with school; problem related to sexuality/gender identity; physical problem (e.g., pain, illness, disability); current homelessness a Suicide Risk Factors: lifetime suicidal ideation; current suicidal ideation (i.e., during chat); imminent risk indicated during chat; passive or suggested suicidal ideation during chat; attempt in progress; history of preparatory behavior toward future/past attempt; method chosen or considered; means available; place chosen or considered; chatter said they had a plan; history of suicide attempt(s); history of non‐suicidal self‐injury Additional Suicide Risk Factors: history of substance abuse; current intoxication; lifetime/current substance abuse treatment; history of psychiatric diagnosis; lifetime/current mental health treatment; lifetime/recent psychiatric hospitalization; reasons for living identified Mitigation of Imminent Risk: whether chatter agreed to any of the following steps: center to send emergency services; self‐ or third‐party rescue; get others involved now; transition to phone now; remove means; receive follow‐up call within 24 h; whether imminent risk was reduced without recourse to any of the first four steps listed here Chatter Behavior at End of Chat: expressed gratitude or appreciation; expressed feeling better or helped; accepted coping suggestion; agreed to pursue new behavioral health treatment; wanted to go to sleep; needed to go do something else; apologized/expressed feeling burdensome; expressed dissatisfaction with counselor or chat; objected to chat's ending; abandoned chat Chatter's Behavioral Changes by End of Chat: less overwhelmed; more hopeful; more confident/in control; less at risk of suicide

Part II: Counselor Behaviors Fostering Engagement/Rapport: created a safe and welcoming environment/affirmed chatter's current use of crisis chat; exhibited empathy/validated or normalized chatter's feelings; affirmed chatter's strengths Collaborative Problem‐Solving: explored what has worked for the chatter in the past/what the chatter had tried to do to solve/cope with the problem; asked what the chatter thought they might do to solve/cope with the problem; offered specific suggestions for the chatter's consideration; reviewed the action plan a Suicide Risk Assessment: Asked: Are you thinking about suicide?/Explored current suicidal thoughts; Asked: Have you had thoughts of suicide in the past/ever?/Explored past/lifetime suicidal thoughts; Asked: Have you ever attempted suicide?/Explored past suicidal behavior; Assessed whether chatter had a suicide plan/had thought of a method/Explored suicide plan; Assessed availability of means; Assessed whether an attempt was in progress/Explored attempt in progress; Explored suicidal intent Establishing Safety & Mitigating Risk: helped chatter develop a formal safety plan; suggested chatter remove access to means/disable suicide plan; suggested chatter transition from chat to phone (now); suggested chatter call 911/go to hospital (now)/offered to send help; asked whether chatter could stay safe Resources, Referrals, & Mental Health Treatment Promotion: explored chatter's openness to/experience with mental health treatment; suggested/explored ways to find a new mental health service provider; made a referral to a specific, new mental health service provider; provided self‐help or other resource(s)

These sections (within Chatter Characteristics and Behaviors: Structural Overview, Demographics, and Suicide Risk Factors; within Counselor Behavior: Suicide Risk Assessment) are coded for all chats, regardless of the chatter's response to “Do you have thoughts of suicide?” on the pre‐chat survey. The remaining sections are coded only for chats where the chatter's answer to that question was “Yes, current” or “Yes, recent past.”

Structural Overview: chat start date; start time; end time; length of chat in turns

Demographics: sexual minority status; military status; whether the center had knowledge of the chatter from prior chats

Suicide Risk Factors: lifetime suicidal ideation; current suicidal ideation (i.e., during chat); imminent risk indicated during chat; passive or suggested suicidal ideation during chat; attempt in progress; history of preparatory behavior toward future/past attempt; method chosen or considered; means available; place chosen or considered; chatter said they had a plan; history of suicide attempt(s); history of non‐suicidal self‐injury

Suicide Risk Assessment: Asked: Are you thinking about suicide?/Explored current suicidal thoughts; Asked: Have you had thoughts of suicide in the past/ever?/Explored past/lifetime suicidal thoughts; Asked: Have you ever attempted suicide?/Explored past suicidal behavior; Assessed whether chatter had a suicide plan/had thought of a method/Explored suicide plan; Assessed availability of means; Assessed whether an attempt was in progress/Explored attempt in progress; Explored suicidal intent

Procedures

The groundwork for the development of the Crisis Chat Transcript Abstraction Form began in 2012. A team comprised of MG, AL, TN, and RA began by reviewing an initial sample of de‐identified Lifeline chat transcripts. Data elements to be coded were adapted for the online chat medium from an earlier instrument designed for quality improvement monitoring of Lifeline crisis calls, and followed Lifeline's guidance on best practices for crisis interventions, which were then based on findings from Gould et al. (2007), Joiner et al. (2007), and Mishara et al. (2007b), and disseminated internally to Lifeline network. An iterative process of revision of the drafted form ensued, in which the four developers independently coded selected chat transcripts, identified and discussed any coding discrepancies, and made revisions to the form designed to improve both content validity and coder agreement. This continued until preliminary inter‐rater reliability exercises yielded acceptable results. A training manual was developed to codify the decisions made during this process and to enable new users to consistently interpret the form.

In the interest of enabling independent assessments, and of easing the burden of coder training, the decision was made to separate the coding of chatter characteristics and behaviors from the coding of counselor behaviors, with each to be coded by an independent coding team. Two teams of research assistants who had not participated in the development of the form, and who each had prior experience as a counselor on a crisis hotline, were trained to use the two parts of the form. Training was conducted using a training sample of chats handled by LCC centers in months not included in our study sample. For the initial 40‐hour trainings of each team, team members focused on one topic area at a time (see Table 1), coding chats first collaboratively, then independently, and coming together to discuss the rationale for their coding decisions and the reasons for any disagreements. Iterative revisions to the training manual were made in order to improve clarity and reduce coder disagreement. Less‐than‐satisfactory reliability on some items during preliminary inter‐rater reliability checks resulted in additional revisions, re‐training, and new rounds of practice. After each team had achieved favorable results on an inter‐rater reliability exercise involving 30 chat transcripts coded by each member of the team, that team began coding our study sample of 1034 chat transcripts. Additional inter‐rater reliability tests were conducted at the middle and end of coding for Chatter Characteristics and Behaviors, and at the end of coding for Counselor Behaviors. Following each reliability exercise, any item for which either the proportion agreement was <.80, or kappa was <.61 and proportion agreement was <.95, was subject to a booster training.

The chatter's responses to the PCS were visible to the transcript coders. However, in answering questions about the chatter's suicide risk, the coders were instructed to disregard the information from the PCS and to answer questions solely based on the conversation between the chatter and counselor. For chats defined as non‐suicidal based on the PCS (i.e., where the chatter answered “No” to “Do you have thoughts of suicide?”), only a limited number of items related to the chat's structural characteristics, the chatter's suicide risk status, and the counselor's risk assessment behaviors were coded. For chats defined as suicidal based on the PCS (i.e., where the chatter answered “Yes, current” or “Yes, recent past” to “Do you have thoughts of suicide?”), the coders coded the entire form. This is in keeping with the priority given by our research team to evaluating crisis interventions with suicidal individuals. Full coding of Chatter Characteristics and Behaviors in a transcript took approximately 15 minutes on average, while coding Counselor Behaviors took about half as long. Transcript codes were assigned and managed using REDCap electronic data capture tools hosted at the Data Coordinating Center, Columbia University Mailman School of Public Health (Harris et al., 2009). The current paper will present findings from the chatter risk status and counselor risk assessment sections, which were coded for the full sample of suicidal and non‐suicidal chats, and from two additional domains of counselor behavior, as previously noted. Coding results for the full battery of items coded for suicidal chats, including coders’ assessments of the extent to which suicide risk was reduced during the chat, will be presented in a future paper.

The final versions of the coding form and training manual are appended to this paper as supplements. The existence of a comprehensive training manual adds to this instrument's value for research purposes. Given the length and complexity of the form, simplification and streamlining are likely needed prior to using the instrument as a clinical or quality improvement tool.

The project's protocol was approved by the Institutional Review Board of the New York State Psychiatric Institute and the Columbia University Department of Psychiatry.

Statistical analyses

Each of the inter‐rater reliability exercises described above involved 30 randomly selected chat transcripts coded by all members of the respective coding team. Because these variables are all categorical, Fleiss’ kappa statistic was used to assess degree of agreement among raters (Fleiss, 1981). Proportion agreement observed was calculated as the proportion of pairs which agreed on the rating out of the total number of pairs. Proportion agreement expected is the proportion of agreement expected based on chance, given the distribution of the categories by each rater. Maximum kappa is the highest possible value for kappa given the observed marginal frequencies. If, for instance, each rater has the same number of Yesses, Nos, etc., then the maximum kappa will be 1 (perfect agreement). It is useful as a comparison to the value of kappa actually obtained. For items with no variability in responses (for example, where every coder responded “Don't Know” for every chat), the proportion agreement expected, Kappa, and Maximum Kappa could not be calculated.

Descriptive analyses of chat length and the chatter's PCS‐reported gender, age, and suicidal thoughts were conducted. Next, a series of Fisher's exact tests were performed to determine if suicide risk as coded by transcript coders differed by the chatter's response to the question about suicidal thoughts on the PCS (“Yes, current,” “Yes, recent,” and “No”). Fisher's exact tests were also performed to determine differences in coded counselor risk assessment behaviors in these three groups. For all analyses of multinomial data, Fisher's exact tests were performed rather than chi‐square tests because there were several instances where the expected frequency in one or more cells was less than or equal to 5. A total number of counselor behaviors in the domain of suicide risk assessment were calculated, and a Kruskal–Wallis test was run to determine if there was a difference in mean number of counselor behaviors in this domain across the three groups. Kruskal–Wallis analysis was chosen rather than a one‐way ANOVA because a normal distribution could not be presumed. Descriptive analyses of counselors’ rapport‐building and problem‐solving behaviors were conducted. Chi‐square tests were performed to determine differences in these coded counselor behaviors (which were coded only for chats where the chatter responded “Yes, current,” or “Yes, recent” on the PCS) between the two groups. For all analyses of dichotomous data, where all expected cell counts were above 5, chi‐square tests were performed. Finally, the relationship between counselor risk assessment behaviors and the availability of information on chatter risk status in the chat conversation was examined in a series of descriptive post hoc analyses.

RESULTS

Inter‐rater reliability

Results of inter‐rater reliability exercises conducted at the beginning and end of transcript coding are provided in Tables 2 and 3. Results of a third inter‐rater reliability exercise involving the chatter characteristics and behaviors codes, conducted midway through chat coding, are consistent with the results provided and are available upon request. Most chatter characteristics and behaviors items had observed proportions of agreement of 0.80 or greater across all three exercises (mean: 0.94; median: 0.96), and kappas of 0.61 or greater, representing substantial agreement or better (mean: 0.66; median: 0.73) (see Table 2). Nearly all counselor behavior items had observed proportions of agreement of 0.90 or greater across both exercises (mean: 0.96; median: 0.97) and kappas of 0.61 or better (mean: 0.83; median: 0.89) (see Table 3). Items with notably low kappa scores tended to be those with high proportion agreement but a very low variability in responses due to a low prevalence of the behavior or characteristic being coded (e.g., current homelessness, preparatory behavior, an attempt in progress at the time of the chat, or a counselor's providing a specific mental health referral). While adequate, reliability for items requiring an assessment of chatters’ emotional states (e.g., whether the chatter was more hopeful or less overwhelmed by the end of the chat) tended to be lower than for other items coded.

TABLE 2

Results of inter‐rater reliability exercises: Chatter characteristics and behaviors

Variable	Proportion agreement observed		Proportion agreement expected		Kappa		Maximum kappa
	Time 1 a	Time 3 a	Time 1	Time 3	Time 1	Time 3	Time 1	Time 3
Demographics
LGBTQ	1	1	.680	.936	1	1	1	1
Military status	.978	.978	.956	.956	.489	.489	.489	.489
Prior chats	.956	1	.894	b	.580	b	.580	b
Life stressors
Break‐up	.978	.978	.785	.708	.896	.924	.896	.924
Isolation	.889	.844	.680	.563	.653	.644	.653	.797
Sexual abuse	1	1	b	b	b	b	b	b
Bullying	1	1	.936	b	1	b	1	b
Other abuse	.933	.956	.895	.737	.365	.831	.365	.831
Relationship issue	.822	.800	.580	.506	.577	.595	.577	.730
Death	.978	.956	.895	.803	.788	.775	.788	.775
Other's suicide	.933	.956	.838	.838	.588	.726	.588	.726
Financial problem	.933	.889	.737	.680	.746	.653	.916	.653
Barrier to treat.	.978	.911	.957	.737	.489	.662	.489	.662
Insurance prob.	1	.978	.936	.857	1	.845	1	.845
Lost job	.911	.956	.838	.820	.451	.753	.863	.753
Job stress	.956	.844	.838	.753	.726	.371	.863	.640
School problem	.978	.933	.708	.753	.924	.730	.924	.910
Gender/Sexuality	.978	1	.895	.936	.788	1	.788	1
Physical problem	.978	.911	.857	.803	.845	.550	.845	.663
Homelessness	.978	.978	.978	.915	−.011	.738	−.011	.738
Suicide risk factors
Lifetime ideation	.956	.956	.680	.680	.861	.861	1	1
Current ideation	.811	.889	.508	.454	.616	.797	.864	.837
Imminent risk	.765	.875	.569	.605	.454	.684	.545	.789
Passive ideation	.833	.667	.500	.605	.667	.330	1	.598
Attempt in prog.	.978	1	.978	b	−.011	b	−.011	b
Prep for future	.972	1	.973	b	−.014	b	−.014	b
Prep for past	.944	.942	.946	.818	−.021	.682	.234	.841
Method chosen	.917	.913	.500	.500	.833	.826	.833	.884
Gun	1	.978	b	.857	b	.845	b	.845
Hanging	1	1	b	.936	b	1	b	1
Pills	1	1	.510	.680	1	1	1	1
Gas	1	1	.755	b	1	b	1	b
Drowning	1	1	.867	b	1	b	1	b
Bridge/Height	1	1	b	.936	b	1	b	1
Moving object	1	1	.867	b	1	b	1	b
Knife/Sharp	1	1	.592	.876	1	1	1	1
Means available	.758	c	.341	c	.632	c	.816	c
Place chosen	.889	1	.871	b	.140	b	.355	b
Said had plan	.917	.913	.556	.397	.812	.856	.812	.856
Prior attempt(s)	.917	.884	.463	.580	.845	.724	.922	.724
NSSI	.911	.911	.511	.609	.818	.773	.864	.773
Additional suicide risk factors
Substance abuse	.878	.956	.762	.895	.486	.577	.486	.577
Intoxicated	1	1	.936	b	1	b	1	b
SA tx (ever)	1	1	b	b	b	b	b	b
SA tx (current)	–	1	–	b	–	b	–	b
MH diagnosis	.911	.889	.583	.571	.787	.741	.893	.793
MH tx (ever)	.956	.911	.411	.449	.925	.839	.925	.839
MH tx (current)	.867	.846	.380	.396	.785	.746	.929	.767
Psych. hosp. (ever)	.978	.933	.753	.876	.910	.464	.910	.643
Recent discharge	.978	1	.915	b	.738	b	.738	b
Reasons for living	.889	.840	.639	.706	.693	.457	.754	.638
Steps agreed upon by chatter to mitigate imminent risk
Emergency rescue	1	1	b	b	b	b	b	b
Self‐rescue	1	1	b	b	b	b	b	b
Involve others now	1	1	.803	b	1	b	1	b
Move to phone	1	1	b	b	b	b	b	b
Remove means	1	1	b	.936	b	1	b	1
Follow‐up call	1	1	b	b	b	b	b	b
Risk reduced	1	1	.625	c	1	1	1	1
Chatter behavior at end of chat
Said thanks	.889	.933	.509	.502	.774	.866	1	.911
Felt better	.844	.889	.563	.556	.644	.750	.898	.850
Accepted coping	.844	.911	.694	.803	.492	.550	.928	.775
Accepted MH tx	.933	.911	.838	.753	.589	.640	.863	.820
Felt sleepy	.978	.956	.915	.936	.738	.310	.738	.310
Had to go	.911	.933	.694	.753	.710	.730	.910	.730
Apologetic	.911	.956	.857	.956	.380	−.023	.845	−.023
Dissatisfied	.978	.956	.785	.936	.896	.310	.896	.310
Ending too soon	1	1	b	b	b	b	b	b
Abandoned chat	.978	.933	.915	.753	.738	.730	.738	.910
Chatter's behavioral changes by end of chat
Less overwhelmed	.711	.911	.609	.838	.261	.451	.432	.863
More hopeful	.800	.800	.708	.753	.316	.191	.544	.391
More confident	.800	.844	.667	.667	.400	.533	.800	.666
Less at risk d	.767	.833	.442	.489	.582	.674	.841	.783

Inter‐rater reliability exercises, each involving 30 chats coded by three coders, were conducted at the beginning, middle, and end of transcript coding. Results of the first and third exercises are presented here. Results of the second exercise were consistent with these and are available upon request.

Coders agreed perfectly on these items; however, kappa could not be calculated due to a lack of variability in responses. This could happen, for example, if all coders answered “No” for all chats.

Too few cases to calculate reliability.

Dichotomized as “Not at all” vs. “A little” or “Moderately/A lot”.

TABLE 3

Results of inter‐rater reliability exercises: Counselor behaviors

Variable	Proportion observed		Proportion expected		Kappa		Maximum kappa
	Time 1 a	Time 2 a	Time 1	Time 2	Time 1	Time 2	Time 1	Time 2
Fostering engagement/Rapport
Welcome	.967	.867	.545	.580	.927	.683	.927	.841
Empathy	.900	.967	.745	.905	.608	.649	.608	.649
Strengths	.867	.900	.680	.567	.583	.769	.583	.923
Collaborative problem‐solving
Past skills	.867	.967	.536	.625	.713	.911	1	.911
Support solutions	.933	1	.509	.502	.864	1	1	1
Offer suggestions	.867	.833	.520	.501	.722	.666	.722	.933
Review plan	1	.933	.936	.876	1	.464	1	1
Suicide risk assessment
Current thoughts?	.900	.967	.625	.625	.733	.911	.733	.911
Prior thoughts?	.967	.900	.625	.745	.911	.608	.911	.608
Past attempt?	.967	1	.745	.680	.869	1	.869	1
Thoughts about method?	1	.933	.609	.502	1	.866	1	1
Available means?	1	1	.820	.680	1	1	1	1
Attempt in progress?	1	1	.609	.642	1	1	1	1
Intent to act?	1	.967	.820	.847	1	.782	1	.782
Steps suggested by counselor to establish safety & mitigate risk
Safety planning	1	1	.936	b	1	b	1	b
Remove means	1	1	.936	.936	1	1	1	1
Move to phone	1	1	b	b	b	b	b	b
Rescue	1	.967	.820	.794	1	.838	1	.838
Assess safety	1	.900	.722	.701	1	.666	1	.666
Resources, referrals, & mental health treatment promotion
Open to MH treatment?	.967	.900	.505	.505	.933	.798	.933	.933
Ways to access treatment	1	.967	.820	.905	1	.649	1	.649
Referral	.967	1	.967	b	−.017	b	−.017	b
Other resources	1	.900	.642	.701	1	.666	1	.666

Inter‐rater reliability exercises, each involving 30 chats coded by two coders, were conducted at the beginning and end of transcript coding. Results of both exercises are presented here.

Coders agreed perfectly on these items; however, kappa could not be calculated due to a lack of variability in responses. This could happen, for example, if all coders answered “No” for all chats.

Chat length and pre‐chat survey responses (gender, age, thoughts of suicide)

As noted in the description of our sample, chats lasting <10 conversational turns were excluded from coding. Coded chats ranged from 10 to 394 turns long (mean: 60.8 turns (SD = 42.24), median: 52 turns). Conversations ranged in duration from 2 to 203 minutes (mean: 45.2 min (SD = 27.40), median: 40 min). The conversational pace of the chats ranged from 0.4 to 7.0 turns/min (mean: 1.5 turns/min, median: 1.3 turns/min). When responding to the PCS, chatters in the 1034 coded chats self‐identified as 21.8% male, 70.5% female, 3.2% transgender, and 4.5% questioning. Age was provided by 1012/1034 chatters (range: 11–63; mean: 24.6 (SD = 10.41); median: 21). Over a quarter of chats (28.5%) were from chatters younger than 18, and the average age of the adult chatters in the sample was 28.4 (SD = 10.09, median = 25). Because no identifying information on chatters was collected, we are unable to know whether the 1034 chats are from 1034 unique chatters, or whether our sample contains multiple chats from repeat chatters. Chatters denied suicidal thoughts on the PCS of 16.0% of coded chats (N = 165, henceforth known as NS chats), endorsed current suicidal thoughts on 59.3% (N = 613, henceforth known as CS chats), and endorsed recent suicidal thoughts on 24.8% (N = 256, henceforth known as RS chats); thus, a total of 84.0% of coded chats had current or recent suicidal thoughts endorsed on the PCS (N = 869, henceforth known as CS/RS chats).

Chatters’ suicide risk according to transcript coders

Indicators of suicide risk disclosed by chatters during the chat conversation and coded by transcript coders are presented in Table 4. Transcript coders found evidence of a lifetime history of suicidal ideation in two thirds (66.7%) of chats, and of current suicidal ideation in nearly half (45.8%); however, nearly a third (32.9%) and nearly half (45.8%) of chats received codes of “Don't Know” for lifetime suicidal ideation and current suicidal ideation, respectively, because the transcript contained insufficient information on these topics for the coders to code either “Yes” or “No.” Explicit denials of either lifetime or current suicidal ideation (yielding codes of “No”) were infrequent (0.4% and 8.3%, respectively). Indicators of whether the chatter had chosen or considered a suicide method, or had a history of suicide attempts, showed a similar pattern: codes of “Don't Know” were frequent, and codes of “No” were rare (see Table 4). The availability of information about suicidal ideation and suicide method in the chat conversation varied significantly according to the chatter's response to the PCS question about suicidal thoughts. Current suicidal ideation was not assessed or discussed adequately during the chat to enable it to be coded “Yes” or “No” on a third of CS chats, on nearly 60% of RS chats, and on over 70% of NS chats. For over a quarter of the total sample (28.9%, N = 299), there was insufficient information in the body of the chat to determine whether the visitor had any suicidal ideation (lifetime or current).

TABLE 4

Coders’ assessment of chatter's suicide risk, by suicidal thoughts on pre‐chat survey

Indicators of chatter's suicide risk in body of chat (coded by evaluation team)	Do you have thoughts of suicide? (from pre‐chat survey)
	Yes, current (CS) N = 613	Yes, recent (RS) N = 256	No (NS) N = 165	Total N = 1034	Fisher's exact p
Lifetime suicidal ideation a
Yes	503 (82.1%)	148 (57.8%)	39 (23.6%)	690 (66.7%)	<.001
No	0	0	4 (2.4%)	4 (0.4%)
DK b	110 (17.9%)	108 (42.2%)	122 (73.9%)	340 (32.9%)
Current suicidal ideation
Yes	385 (62.8%)	76 (29.7%)	13 (7.9%)	474 (45.8%)	<.001
No	23 (3.8%)	29 (11.3%)	34 (20.6%)	86 (8.3%)
DK b	205 (33.4%)	151 (59.0%)	118 (71.5%)	474 (45.8%)

Includes current suicidal ideation, past suicidal ideation, and suicidal ideation of indeterminate timing.

A code of “Don't Know” indicates that there was not enough information on this topic in the chat transcript to justify a code of Yes or No.

Indications of imminent risk were coded as present or absent, with no option of a “Don't Know” response.

For chats containing evidence of current suicidal ideation, coders were instructed to assess the presence or absence of indications of imminent risk, defined as indications in the chat that the chatter may attempt suicide on the day/night of the chat (or may suffer the consequences of an already‐initiated attempt), unless urgent action is taken to reduce this risk. Of the 474 chats in which current suicidal ideation was coded as present, a quarter (N = 122) were identified by the coders as containing indications of imminent risk of suicidal behavior (see Table 4). This represents approximately 12% of the total sample of coded chats. Of chats coded as containing indications of imminent risk, 87.7% (107/122) were CS chats, 11.5% (14/122) were RS chats, and 0.8% (1/122) was an NS chat.

Very few CS chatters (3.8%) explicitly denied having current suicidal ideation once the chat had begun. Conversely, a small minority of NS chatters (7.9%) disclosed current suicidal ideation once the chat conversation had begun, with one showing indications of imminent suicide risk. A more substantial minority of RS chatters (29.7%) disclosed current suicidal ideation during the chat, with 18.4% of those giving indications of imminent risk.

Counselor behaviors: Suicide risk assessment

Frequencies of chat counselors’ asking about or exploring various facets of the chatter's suicide risk are presented in Table 5. Counselors engaged in one or more of our coded risk assessment behaviors on over 60% of chats, including just under three‐quarters of CS chats, just over half of RS chats, and approximately a quarter of NS chats. The most frequently assessed aspects of risk were current suicidal ideation and whether the chatter had a suicide plan. Counselors’ rates of engaging in five of the six coded risk assessment behaviors varied significantly according to the chatter's endorsement of suicidal thoughts on the PCS. Counselors assessed each of these five aspects of suicide risk most frequently on CS chats, and least frequently on NS chats.

TABLE 5

Counselor's suicide risk assessment behaviors, by suicidal thoughts on pre‐chat survey

Counselor's behaviors (coded by evaluation team)		Do you have thoughts of suicide? (from pre‐chat survey)
		Yes, current (CS) N = 613	Yes, recent (RS) N = 256	No (NS) N = 165	Total N = 1034	Fisher's exact p
Assessed current suicidal thoughts	Yes	210 (34.3%)	74 (28.9%)	26 (15.8%)	310 (30.0%)	<.001
Assessed past/lifetime suicidal thoughts a	Yes	112 (18.3%)	44 (17.2%)	18 (10.9%)	174 (16.8%)	.071
Assessed suicide plan/method	Yes	239 (39.0%)	54 (21.1%)	5 (3.0%)	298 (28.8%)	<.001
Assessed suicidal intent	Yes	71 (11.6%)	18 (7.0%)	4 (2.4%)	93 (9.0%)	<.001
Assessed past suicidal behavior	Yes	94 (15.3%)	28 (10.9%)	1 (0.6%)	123 (11.9%)	<.001
Assessed attempt in progress	Yes	75 (12.2%)	10 (3.9%)	4 (2.4%)	89 (8.6%)	<.001
At least one of the above		453 (73.9%)	135 (52.7%)	42 (25.5%)	630 (60.9%)	<.001
At least two of the above		233 (38.0%)	66 (25.8%)	11 (6.7%)	310 (30.0%)	<.001
Mean number of behaviors (out of 6)		1.31 (SD = 1.12)	0.89 (SD = 1.05)	0.35 (SD = 0.75)	1.05 (SD = 1.11)	K‐W 131.91, p < .001 †

Includes assessment of past suicidal thoughts and suicidal thoughts of indeterminate timing. Does not include assessment of current suicidal thoughts.

Kruskal–Wallis test with 2 df.

It may be noted that codable information about chatters’ suicide risk factors (Table 4) was available in chat transcripts more frequently than these factors were assessed by counselors (Table 5). To identify how information on chatters’ suicidal ideation was obtained, and to identify the reasons for its not being obtained, we examined the relationship between counselor's assessments of suicidal ideation and the availability of information on this topic in the chat transcripts of the 613 CS chats. In the 408/613 CS chat transcripts (66.6%) where information on current suicidal ideation was available (i.e., coded “Yes” or “No”), counselors had assessed current suicidal ideation on 179 (43.9%) and had assessed past or lifetime (but not current) suicidal ideation on 63 (15.4%). This indicates that on the remaining 166 chats (40.7%), chatters volunteered information about their suicidal ideation without being asked. In the 205/613 CS chat transcripts (33.4%) that were lacking information on chatters’ current suicidal ideation (i.e., coded “Don't Know”), counselors had failed to assess current suicidal ideation on 174 (84.9%), indicating that on the remaining 31 (15.1%), counselors had attempted to elicit this information but had not received a clear answer from the chatter.

Counselor behaviors: Rapport‐building and collaborative problem‐solving

Counselor behaviors in the domains of fostering engagement or rapport with chatters, and of collaborative problem‐solving, were coded only on the 869 CS/RS chats (as noted above under Procedures). Coders identified counselors as creating a safe and welcoming environment and/or affirming the chatter's current use of crisis chat on 578 (66.5%) of these chats, as exhibiting empathy/validating or normalizing the chatter's feelings on 730 (84.0%), and as affirming chatters’ strengths on 184 (21.2%), for an average of 1.72 (SD = 0.79) out of 3 rapport‐building behaviors per chat. At least one of these three rapport‐building behaviors was coded as present in 93.3% of chats (N = 811), and all three were coded as present in 14.6% of chats (N = 127).

In the domain of collaborative problem‐solving, coders identified counselors as exploring what the chatter had already tried to do to solve or cope with their problems/what strategies had worked for them in the past on 470 (54.1%) of the 869 chats, asking what the chatter thought they might do to solve or cope with their problems on 258 (29.7%), offering specific suggestions for the chatter's consideration on 347 (39.9%), and reviewing the action plan developed during the chat on 17 (2.0%). Counselors engaged in an average of 1.26 (SD = 1.06) out of four collaborative problem‐solving behaviors per chat. At least one problem‐solving behavior was coded as present in 70.1% of chats (N = 609), with all four coded as present in only 1.2% of chats (N = 10).

Counselors were more likely to be coded as asking chatters what they had already tried to do to solve or cope with their problems/what strategies had worked for them in the past on chats with RS chatters than on chats with CS chatters (60.9% vs. 51.2%; χ2 df=1 = 6.48, p = .011). (This behavior was not coded on chats with NS chatters.) None of the other counselor behaviors in the domains of rapport‐building and collaborative problem‐solving differed significantly by the chatter's response to the PCS question about suicidal thoughts.

DISCUSSION

We believe ours to be the first coding form to be developed specifically for abstracting data from text‐based crisis intervention transcripts, to assess an extensive range of chatter and counselor behaviors specific to crisis intervention, and to demonstrate substantial reliability.

During the development of our chat transcript abstraction form, items used for the coding of monitored crisis calls were selected and revised based on the kinds of information available in a chat transcript as opposed to a live audio call. Items that relied heavily on pace or tone of voice were omitted in favor of items that rely on careful reading (and re‐reading) of the written word. The somewhat lower reliability scores received by codes for changes in emotional states (more hopeful, less overwhelmed) reflect the difficulty of assessing these states in the absence of non‐verbal cues. Whereas we ultimately achieved substantial reliability in coding the presence of current suicidal ideation, it was surprisingly challenging to do so in light of the sometimes ambiguous wording of counselors’ risk assessment questions, which made it difficult to interpret the exact meaning of the chatter's response, particularly with respect to the timing of the chatter's thoughts. By contrast, the Lifeline's recommended prompt questions, “Are you thinking about suicide?” and “Have you had thoughts of suicide in the last two months?” make the intended timeframes explicitly clear. A challenge for counselors lies in adapting these questions to fit the flow of an individual conversation, without sounding robotic or pre‐programmed, but also without sacrificing the clarity and specificity of the information to be obtained.

In examining 1034 chats handled by Lifeline Crisis Chat centers in 2015, we found crisis chat chatters to be younger on average than crisis callers and to reveal suicidal ideation at higher rates, which is consistent with the results of earlier studies (Fukkink & Hermanns, 2009a; Haner & Pepler, 2016; Mokkenstorm et al., 2017). When excluding minors, who made up over a quarter of our sample, adult chatters’ median age was 25, compared with an estimated median age of over 34 for adult callers to a telephone crisis line (Mishara et al., 2007a). Chatters endorsed either current or recent suicidal thoughts on Lifeline's PCS for 84% of chats, which is markedly higher than the estimated 23% of Lifeline crisis calls on which callers acknowledged having suicidal thoughts on the day of or the day before their call, according to silent monitors listening to the calls (Gould et al., 2013). Review of the chat intervention transcripts (excluding information from the PCS) enabled trained chat coders to identify the presence of current suicidal ideation on just under half of chats. For most of the remaining chats, definite information on current suicidal ideation was lacking in the chat transcript.

Due to the substantial proportions of chats where current suicidal ideation was not discussed adequately during the chat to enable it to be coded as present or absent, and the fact that the “Don't Know” codes were differentially distributed by the chatters’ PCS response, we are unable to draw a conclusion about the overall concordance between chatters’ PCS self‐reports regarding suicidal ideation and their disclosures regarding suicidal ideation during the chat conversation. Even so, some notable differences were observed between the two sources, with nearly 30% of chatters who endorsed recent (rather than current) suicidal ideation on the PCS going on to reveal current suicidal ideation during the chat. This finding, and the finding that even a few chatters who did not endorse current suicidal thoughts on the PCS nonetheless went on to reveal imminent risk of suicidal behavior during the course of the chat, reinforce the importance of Lifeline's policy that counselors assess current suicide risk on all chats, regardless of the chatter's PCS response.

On average, crisis chat interventions appear to last longer than telephone crisis interventions and are likely to move at a slower pace. Lifeline chat counselors engage in rapport‐building on nearly every chat with a suicidal chatter and engage in problem‐solving on over two thirds of such chats, demonstrating what seems to be a more balanced approach than has been observed in other evaluations of online counseling interventions (Chardon et al., 2011; Fukkink, 2011; Mallen et al., 2011). As has been found in evaluations of telephone crisis interventions, inquiries into suicide risk did not appear to be conducted on every chat as would be consistent with Lifeline policy (Gould et al., 2013). Coders observed at least one of our coded risk assessment behaviors on approximately 60% of chats, with the frequency of counselor behavior in this area depending significantly on the chatter's response to the PCS question about suicidal thoughts. Low rates of individual risk assessment behaviors seem to reflect that in some cases, the information in question is already available due to its having been volunteered by the chatter. In other cases, if initial questioning resulted in a denial of suicide risk, counselors may have found further questioning to be uncalled for. Nonetheless, the substantial minority of chat transcripts in which the discussion of suicidal ideation during the conversation was insufficient for the coders to identify whether or not the chatter was currently suicidal seems to indicate a need for a more consistent and explicit focus on suicide risk.

Best practice counselor interventions such as building rapport or engagement with persons at risk, helping them to feel heard and cared for, and helping them to identify coping strategies and to problem‐solve about their situations, are likely to be protective against suicide risk whether or not they are explicitly suicide‐focused. Nonetheless, the Zero Suicide Institute recommends that treatment of suicidal individuals “should directly target and treat suicidal thoughts and behaviors” (Education Development Center, Inc., 2020), which depends on directly assessing the extent to which these thoughts and behaviors are present. While a crisis intervention is not a clinical or research interview dedicated solely to collecting diagnostic information about suicide risk, identifying the presence of modifiable risk factors such as current suicidal ideation, intent, and available means is critical to ensuring that the crisis intervention meets the present needs of the person‐at‐risk, to include enhancing his or her immediate safety if necessary (Carter & Spittal, 2018).

Because our sample was stratified based on chatters’ completion of Lifeline's optional post‐chat survey, our sample may not be completely representative of all Lifeline chats. However, chat transcripts with and without a completed post‐chat survey did not significantly differ with regard to the proportion of chats from chatters who endorsed suicidal ideation (either current or recent) on the PCS (82.8% of chats with a post‐chat survey, and 85.5% of chats without), or to the proportion of chat transcripts containing indications of imminent risk (11.4% of chats with a post‐chat survey, and 12.2% of chats without), indicating that these findings regarding chatters’ suicide risk are likely to be generalizable to all Lifeline chats. For the remainder of our findings, which concern the reliability of our coding form and the relationship between PCS data and transcript coding data, there is no reason to believe that these results would vary according to whether or not the chatter completed a post‐chat survey. Because chat transcripts were de‐identified by Lifeline staff before being provided to us, we are not able to identify the center or counselor handling the chat and could not nest our analyses within center or counselor. We are also unable to determine whether our sample contains multiple chats from the same crisis chatter, which means our unit of analysis is the chat rather than the chatter. We further cannot determine which chats in our sample are from chatters using the chat service for the first time, vs. from chatters with whom the chat counselors may be familiar based on previous contacts with them. However, Lifeline policy recommends that suicide risk assessments be performed on all crisis chats, regardless of whether the chatter had used the service before. Despite its limitations, this paper provides an unprecedented level of insight into crisis chatters’ suicide risk status and chat counselors’ behaviors, especially in the domain of suicide risk assessment.

CONCLUSION

In developing an instrument specifically for the evaluation of crisis chat interventions, we found that it was possible to reliably code an extensive array of chatter characteristics and behaviors and counselor behaviors. It appears that crisis chat counselors are able to use the online chat medium to build rapport with chatters and to collaborate with them on identifying coping strategies and solutions to their problems, in keeping with Lifeline's crisis intervention model. However, counselors were not observed to assess suicide risk on all chats and appeared to base their risk assessment activity to a significant degree on the chatter's PCS response. Although it may be tempting to view a mandatory, automated survey as a solution to the previously identified shortcomings in risk assessments during crisis interventions, our finding that a minority of chatters were observed to disclose higher levels of risk during the chat conversation than they had endorsed on the PCS indicates that the PCS should not be taken as a substitute for a direct inquiry into suicide risk and current safety during the chat. Additional efforts are therefore needed in the form of research, policy, and training to improve counselor adherence to Lifeline's long‐standing requirement that suicide risk be assessed during every crisis intervention.

AUTHOR CONTRIBUTIONS

Alison M. Lake: Conceptualization (equal); data curation (lead); formal analysis (supporting); investigation (equal); methodology (equal); project administration (lead); software (equal); validation (equal); writing – original draft (lead); writing – review and editing (lead). Thomas Niederkrotenthaler: Conceptualization (supporting); investigation (equal); methodology (equal); validation (equal); writing – review and editing (supporting). Rebecca Aspden: Conceptualization (supporting); Investigation (equal); methodology (equal); validation (equal); writing – review and editing (supporting). Marjorie Kleinman: Data curation (equal); formal analysis (lead); software (equal); validation (equal); writing – review and editing (supporting). Amanda M. Hoyte‐Badu: Investigation (equal); validation (equal); writing – original draft (supporting); writing – review and editing (supporting). Hanga Galfalvy: Conceptualization (supporting); formal analysis (equal); supervision (supporting); writing – review and editing (supporting). Madelyn S. Gould: Conceptualization (equal); formal analysis (equal); funding acquisition (lead); investigation (equal); methodology (equal); supervision (lead); validation (equal); writing – review and editing (equal).

Click here for additional data file.

Click here for additional data file.