Assessing Suicide Risk in a Pediatric Outpatient Behavioral Health System: A Quality Improvement Report.

Introduction: Standardized suicide risk assessment improves the detection of individuals at risk of suicide. We conducted a quality improvement initiative in a system of outpatient behavioral health practices affiliated with a free-standing children's hospital to implement standardized suicide risk assessment for new patients.
Methods: Clinicians received education in suicide risk assessment and were trained to use an evidence-based suicide risk assessment tool, the Columbia Suicide Severity Rating Scale (C-SSRS). We standardized workflow processes and integrated the C-SSRS in the electronic health record with a feature to communicate instances of elevated risk across care teams through a problem list. We analyzed C-SSRS responses and adherence to standardized processes and compared the percentage of patients with a suicide-related item on the problem list before and after implementation. We assessed clinician knowledge through a survey. All patients with identified suicide risk received treatment to reduce their risk of suicide in the context of usual care.
Results: For 3,972 new patient visits occurring postimplementation (November 2016-December 2018), the average monthly adherence to the standardized process was 97.7%. The mean monthly incidence of nonspecific active suicidal thoughts was 16%, aborted suicide attempts were 2%, and actual suicide attempts were 3%. The mean monthly incidence of a suicide-related item documented on the problem list was 5.66% in the postimplementation period compared with 1.47% in the 1-year preimplementation. Clinicians demonstrated statistically significant increases in knowledge about suicide risk factors and assessment. Conclusions: Standardization of suicide risk assessment processes improved detection and documentation of suicide risk in a pediatric outpatient behavioral health setting.

INTRODUCTION

Suicide is the second leading cause of death among young people in the United States.[1] Young people with behavioral health diagnoses are at higher risk for suicide than their peers without psychiatric illness.[2,3] As of July 2019, the Joint Commission National Patient Safety Goal 15.01.01 requires screening for suicidal ideation with a validated tool for patients aged 12 and over being evaluated or treated for behavioral health conditions as their primary concern.[4]

The Zero Suicide framework is a national effort to improve suicide prevention across healthcare settings and recommends that organizations implement standard practices to identify individuals at risk of suicide. Research from emergency department and urgent care settings[5-10] highlights the need for clinicians to use standardized suicide risk assessment procedures to increase opportunities to detect and intervene for suicide risk. Despite the evidence supporting standardized suicide risk assessment, few examples of standardized practices exist in pediatric outpatient specialty behavioral health clinics. Outpatient care often represents a patient’s entry point into mental health care. It allows clinicians to identify suicidal thoughts or behaviors and intervene in a setting where patient care is maintained over time. One institution’s successful approach focused on improving adherence to joint commission guidelines for risk screening and assessment practices in the pediatric emergency room and outpatient behavioral health care,[11] but literature is limited on how other clinical programs have implemented suicide risk screening and assessment based on local resources and needs.

In our institution’s outpatient behavioral health clinics, use of a validated, evidence-based suicide risk assessment tool was not standard practice before beginning the project described here. Most clinical team members had not recently participated in any formal training regarding evidence-based suicide risk assessment of children and adolescents. As an initial step to implement the Zero Suicide framework within our behavioral health clinics, we used an institutional quality improvement framework to implement standardized suicide risk screening processes.[12] Specifically, we aimed to increase the use of evidence-based suicide risk assessment to 95% for all new patient encounters in a network of pediatric outpatient behavioral health clinics within 6 months.

METHODS

Context

Our outpatient behavioral health program is part of a tertiary, university-affiliated free-standing children’s hospital system and includes 1 primary urban clinic and 5 satellite suburban clinics. The outpatient program serves children and adolescents ages 2 through 19 who present with behavioral health concerns through self-referral or referral from another clinician. Behavioral health providers include licensed psychiatrists, psychologists, clinical social workers, psychiatric nurse practitioners, and trainees.

Interventions

We convened a multidisciplinary QI team including psychology, psychiatry, clinical process improvement, and data science representatives. The team developed a driver diagram (Fig. 1) to identify primary drivers of the likelihood that a new patient encounter would include a standardized suicide risk assessment. We identified 3 primary drivers: (a) workflow that enables a standardized risk assessment, including electronic health record (EHR) support; (b) clinician knowledge of suicide risk factors and standardized risk assessment; and (c) clinician skill at conducting a standardized risk assessment. The team selected interventions targeting each of the primary drivers.

Fig. 1.

Driver diagram: project to improve suicide risk assessment in outpatient behavioral health clinics. BPA, best-practice advisory; C-SSRS, Columbia Suicide Severity Rating Scale; EHR, electronic health record.

Interventions to address each driver included: (a) workflow: (1) selecting an evidence-based suicide risk assessment instrument; (2) standardizing the expectation for the use of an evidence-based suicide risk assessment process across clinical programs (eg, ADHD clinic, mood, and anxiety disorders clinic); (3) embedding the instrument in the EHR; (4) requiring EHR completion of the instrument in the encounter documentation process; and (5) including EHR prompts to add suicide-related items to the EHR problem list to communicate cases of increased suicide risk to other members of the healthcare team; (b) Clinician Knowledge: educating clinicians about suicide risk and protective factors; and (c) Clinician Skill: training clinicians in the use of the evidence-based suicide risk assessment instrument.

The C-SSRS

The team selected an evidence-based suicide risk assessment tool, the pediatric version of the Columbia Suicide Severity Rating Scale (C-SSRS) Lifetime version,[13] based on a review of the existing literature, benchmarking with peer institutions, and recommendations from national guidelines from the Joint Commission and National Action Alliance for Suicide Prevention.[4,14] The C-SSRS is a semistructured clinician-administered interview tool that elicits information about suicidal ideation and behavior in the patient’s lifetime and past 1 month, based on responses directly from an individual and their caregivers. Responses are recorded by the interviewer using specific definitions of types and levels of severity of suicidal ideation and behavior. The C-SSRS Lifetime version can be administered in several minutes and is available in over 100 languages.

Staff Training

In June 2015, 3 project leaders (2 psychologists and 1 psychiatrist) completed training to become recognized as suicide prevention champions competent to train colleagues in suicide prevention principles and using the C-SSRS. Their “train-the-trainer” education included a 2-day in-person training sponsored by the Zero Suicide Institute (http://zerosuicideinstitute.com) and online C-SSRS training sponsored by the Columbia Lighthouse Project and Center for Practice Innovations Learning Community.[15]

Beginning in September 2016, 44 outpatient staff clinicians (psychiatrists, psychologists, nurse practitioners, and clinical social workers) completed a 3-hour training session led by the project leaders to increase their knowledge and skill in assessing suicide risk. The training included case illustrations, instruction in using the C-SSRS (including online training), integration in the EHR, and the revised suicide risk assessment process. Additionally, the project team conducted training over the subsequent 23 months with 185 other healthcare team members (social workers and behavioral health trainees), with whom pre/post measures of clinician knowledge and skill were completed. Once the initial training was completed, an annual training was introduced for trainees, new hires, and clinicians wishing to participate.

Standardizing Suicide Risk Assessment Process and C-SSRS Integration in EHR

Our team developed a standardized suicide risk assessment process, communicated via training sessions and in a document for ongoing reference. Before roll-out of the C-SSRS Lifetime version in the EHR, a small number of clinicians pilot-tested the C-SSRS for feasibility and integration into clinical workflow across a range of clinical programs and patient populations. Pilot tests revealed that the C-SSRS required more time than clinicians’ previous assessment strategies, and clinicians found they preferred to complete the assessment early in the visit to allow time for completion and so that patient reported information during the assessment could inform the rest of the visit. In mid-November 2016, the C-SSRS Lifetime version was integrated into the EHR (Epic, Epic Systems Corporation). Within the EHR, a best-practice advisory alert prompted clinicians to complete the C-SSRS at every new patient encounter. If a clinician attempted to close an encounter without completing the C-SSRS, the best-practice advisory prevented the encounter from closing and prompted the clinician to complete the C-SSRS. A clinician could close the encounter without completing the C-SSRS by providing a reason (eg, patient was younger than 7 years old, patient declined to participate in the assessment, or patient ended the visit before the assessment completion).

Identification of Severity of Suicide Risk

Clinicians scored the C-SSRS in keeping with guidelines. A best-practice advisory alert prompted clinicians to add a suicide-related problem list item (eg, suicide ideation, suicide attempt, and timeframe of either lifetime or past one month) if a patient endorsed higher-severity items on the C-SSRS, including any suicidal ideation with methods, intent to act, or plan, or any suicidal behavior item. The problem list is an EHR feature that communicates a patient’s diagnoses and health problems across all members of the healthcare team. It enables other clinicians (eg, primary care providers, medical subspecialists) to have situational awareness, screening, and follow-up of the patient’s suicide risk. We trained clinicians to inform patients that the suicide-related item would be added to their EHR and visible to other clinicians in the health system. Clinicians were also trained to review and update the problem list at every encounter. The problem list is not viewable by parents or other proxies in the EHR patient access portal. All notes that include confidential mental health-related information require specific written permission (from the patient if age 14 or older, or from the parent for younger patients) for release.

Study of the Interventions

We defined new patient visits as patients who were new to the practice or initiating an additional modality of care (eg, psychiatric evaluation before initiating medication for a patient receiving psychotherapy). Following integration of the C-SSRS and best-practice advisory within the EHR in November 2016 (postimplementation period: mid-November 2016–December 2018), data for endorsements of C-SSRS suicidal ideation and behavior items, and frequency of suicide-related items on the problem list were extracted for analysis. We included 1 year of data from the preimplementation timeframe for comparison (October 2015–October 2016, shown in Figures 2–4).

Fig. 2.

P-Chart: C-SSRS completion rate in EHR. C-SSRS, Columbia Suicide Severity Rating Scale; EHR, electronic health record.

Fig. 4.

X-Bar chart: time to close new patient encounter documentation in EHR. C-SSRS, Columbia Suicide Severity Rating Scale; EHR, electronic health record.

Measures

The primary outcome measure was identification and documentation of suicide risk, defined as inclusion of a suicide-related problem list item. Behavioral health notes have special privacy protections, and clinicians outside of behavioral health cannot see documentation of suicide risk if it is only present in behavioral health notes. Therefore, problem list documentation of suicide risk is an important component of situational awareness and patient safety for patients seen in subspecialty behavioral healthcare. The primary process measure was proportion of new patient evaluation visits with a completed C-SSRS. The goal was at least 95% adherence to the process within 6 months. A goal of 95% completion was selected, given the importance of consistent assessment for effective identification of at-risk patients, but anticipating that some patients may not be assessable (eg, patients under age 7). Secondary process measures were knowledge of suicide risk and protective factors and skills in using the C-SSRS among 185 social workers and trainees. We assessed these using a survey developed by our team that queried clinicians’ self-ratings of their knowledge, comfort, confidence in ability, and possession of necessary training to conduct risk assessments before and after training (Table 1).

Table 1.

Comparison of Clinician Suicide-related Knowledge and Skills Assessment Scores Before and After Training

Assessment Items	Pretraining Mean Score (SD)	Posttraining Mean Score (SD)	Difference
Knowledge about suicide*	2.22 (55% correct)	3.41 (85% correct)	1.19† (30%)
Comfort in asking questions about suicide‡	5.93 (1.44)	6.29 (0.53)	0.36†
Confidence in ability to assess suicide risk‡	5.37 (1.19)	6.06 (0.63)	0.69†
Received needed training to assess suicide risk‡	4.99 (2.04)	6.21 (0.47)	1.22†

*Scored as number of items correct out of 4 knowledge items.

†P < 0.05.

‡Respondent rating on a scale of 1 (“Strongly Disagree”) to 7 (“Strongly Agree”).

As a balancing measure, we collected data on the average number of days to close new patient visit encounter records in the EHR. We selected this measure as a proxy for clinician efficiency and timeliness of documentation.

Data Analysis

The outcome and process measures were analyzed using statistical process control charts. Rules were applied to identify special cause variation, including trends, points outside the control limits, and runs of 8 or more points above or below the centerline.[16] Paired t-tests were used to examine data collected from assessments of clinician knowledge and skills before and following training sessions.

Ethical Considerations

This project was undertaken as a quality improvement initiative and does not constitute human subjects research and was exempt from IRB review. This report was written in accordance with the SQUIRE 2.0 guidelines for quality improvement reporting.[17] All patients received clinical care directed toward reducing their suicide risk in usual clinical practice. The team has since undertaken additional quality improvement efforts to implement standardized practices supporting the “treat suicidal thoughts and behaviors directly” pillar of the Zero Suicide framework, which were beyond the scope of the suicide risk assessment efforts described in this article.

RESULTS

Completion of C-SSRS

The preimplementation period included 1,697 new patient visits, and the postimplementation period included 3,972 new patient visits. The mean patient age during preimplementation was 10.6 years (median 14.7 years; range 8.6 to 20.8), and the mean patient age during postimplementation was 11.5 years (median 11.3; range 2.1 to 30.9). Postimplementation, monthly completion of the C-SSRS in the EHR ranged from 95.5% to 99.5% (mean 97.7%) (Fig. 2). Reasons for not completing the C-SSRS included patients leaving before the appointment was completed, patients declining to complete the assessment, and clinician error. The mean incidence per month of patients endorsing a wish to be dead was 25%, and nonspecific active suicidal thoughts was 16%. The mean incidence per month of patients reporting aborted suicide attempts was 2%, and actual suicide attempts was 3%.

Suicide Risk-Related Item on EHR Problem List

During the preimplementation period, the mean incidence per month of a suicide-related problem list item was 1.47%. After implementation, the mean incidence per month was 5.66% (Fig. 3). There was a centerline shift, that is, 8 consecutive data points above the previous centerline, beginning in November 2016, which corresponded to C-SSRS implementation in the note template for new patient visits accompanied by a best-practice advisory alert preventing clinicians from closing encounters without C-SSRS completion.

Fig. 3.

P-chart: proportion of new patient visits with a suicide-related item on problem list in EHR. C-SSRS, Columbia Suicide Severity Rating Scale; EHR, electronic health record.

Clinician Suicide Prevention Knowledge and Skills

Of the 185 social workers and behavioral health trainees who received a pre- and posttraining assessment, 175 returned the survey (94.5% response rate). Among survey respondents, there was a statistically significant increase in mean scores for knowledge of suicide prevention principles, respondent self-assessment of comfort in asking questions about suicide, confidence in ability to assess for suicide risk, and belief that they received the training needed to assess for suicide risk (Table 1).

Balancing Measure: Time to Close Encounters

Before introduction of the C-SSRS, average time to close an encounter was 21 days. Six months after introduction of C-SSRS, there was a centerline shift downward. At that point, average time to close an encounter decreased to 19 days, suggesting that standardizing the suicide risk assessment and documentation process did not negatively affect the efficiency of documentation (Fig. 4).

DISCUSSION

We successfully established a program-wide, standardized suicide risk assessment process which included: (1) implementation of an evidence-based instrument (C-SSRS); (2) education in use of the C-SSRS and principles of suicide prevention and risk assessment; and (3) EHR-based tools to support use of the C-SSRS and documentation, and timely communication of findings. Using a standardized process and assessment instrument rather than existing variable approaches to risk assessment without embedded EHR support resulted in consistently high completion of risk assessments and improved detection and documentation of suicide risk. In addition, team members who completed training about youth suicide prevention and risk assessment demonstrated increased knowledge and reported increased comfort, preparation, and confidence regarding suicide risk assessment.

This project makes several significant contributions. First, limited research examines the lifetime prevalence of suicidal ideation and behavior among young people seeking specialty behavioral healthcare. Healthcare systems have only begun to address this critical issue, so this project starts to fill the knowledge gap about how to accurately assess suicide risk in young people. The focus on children and adolescents is crucial, given the increases in suicide rates among youth,[18-20] greater attention to youth suicide in the context of public discourse, and heightened focus on suicide prevention by regulatory agencies.[4] These factors underscore the need for a standardized suicide risk assessment among children and adolescents seeking behavioral healthcare, even in subspecialty behavioral health clinics, to better understand and address clinical needs. Other programs have adopted tiered approaches where young people are first screened for depression and then receive a suicide risk assessment if they are at risk for moderate to severe depression or suicide based on depression screening results.[21] Such tiered approaches are well-suited to settings like primary care, medical subspecialty care, and emergency department care. However, a tiered approach that begins with screening is not optimally suited for subspecialty behavioral healthcare. Given that patients are seeking behavioral healthcare for a known concern, beginning with suicide risk assessment is more time-efficient and more likely to identify patients at risk of suicide.

Second, this project demonstrates that a large, multidisciplinary, multisite pediatric behavioral health outpatient practice group can integrate and sustain an evidence-based assessment tool to increase consistency and standardization in assessing patients for suicide risk. Establishing a standardized process and expected workflow and supporting that workflow with EHR tools were strategies that supported this initiative’s success. The first EHR strategy we implemented was a hard-stop function forcing C-SSRS completion before closing an encounter. Prior literature has shown that hard-stop functions in the EHR can be powerful tools, but that they can contribute to unintended consequences such as delays in care.[22] In our clinical group, we had strong buy-in for the importance of a structured approach to suicide risk assessment and the hard stop did not appear to contribute to delays in documentation, nor did the study team receive reports of clinician dissatisfaction with the approach. Nevertheless, other groups seeking to implement hard-stops in the EHR should use hard-stops judiciously, given their known risk of unintended consequences. The second EHR strategy we implemented was a structured approach to inclusion of suicide risk on a patient’s problem list. More reliably identifying the presence of suicidal ideation and behavior is an essential element of developing responsive and effective treatment plans for patients with elevated risk. In an era of team- and systems-based care, clear communication of suicide risk via EHR tools is essential for patient safety. Further, an ongoing data collection system related to suicide risk factors contributes to data-driven progress monitoring.

Third, we found that establishing quality parameters around suicide risk assessment increased the consistency of assessment completion and increased suicide risk identification rates. Rather than assume that licensed practicing behavioral health clinicians have requisite knowledge and skill for conducting suicide risk assessments with children and adolescents, our project targeted establishing relevant knowledge and skill. Staff education was a core component of our intervention and was closely tied with implementing the standardized risk assessment tool. Finally, the project emphasized timely communication among clinicians/clinical teams by making the C-SSRS administration results available for all hospital system clinicians and highlighting higher risk concerns by populating them on a patient’s EHR problem list. We anticipate strong sustainability in the use of the C-SSRS as part of clinician practice, given the requirement for completion as part of new patient visits and ongoing sustained use throughout our postimplementation period.

Although this project contributes to the understanding of improving processes for suicide risk assessment among children and adolescents, some limitations apply. First, we only included English-speaking patients receiving outpatient care. Our findings may not generalize to patients who speak other languages or require different levels of behavioral health care such as intensive outpatient, partial hospital, inpatient psychiatric services, or in primary care or medical subspecialties. Second, the project focused on new patient visits, not the full range of outpatient behavioral health services such as therapy and psychiatric medication management, and rates of suicide risk detection may differ for patients seen in return visits. Future projects will use the C-SSRS with patients engaged in ongoing care (beyond the initial visit), at visits conducted in other settings and levels of care, and with patients who speak languages other than English.

CONCLUSIONS

A workforce of skilled clinicians with system-level support to reliably assess and identify suicide risk is a critical component of decreasing morbidity risk due to suicide ideation and attempts and mortality due to suicide. This project demonstrated how a standardized suicide risk assessment can be effectively implemented for children and adolescents seeking outpatient behavioral health care and can increase suicide risk detection and documentation.

Dr. Doupnik received support from the National Institute of Mental Health (K23MH115162). The project also received support from the Cardinal Health Foundation. Funders had no role in data collection, analysis, or decision to submit manuscript for publication.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.

ACKNOWLEDGMENTS

Assistance with the study: We sincerely thank the clinical teams who participated in this quality improvement initiative, Kristin McNaughton and Brenna Aredas for their important contributions to the preparation of this article, and Elizabeth Kauffman and Adam Rudofker for their critical work with data analysis and visualization.