Impact of Universal Suicide Risk Screening in a Pediatric Emergency Department: A Discrete Event Simulation Approach.

OBJECTIVE: The aim of this study was to use discrete event simulation (DES) to model the impact of two universal suicide risk screening scenarios (emergency department [ED] and hospital-wide) on mean length of stay (LOS), wait times, and overflow of our secure patient care unit for patients being evaluated for a behavioral health complaint (BHC) in the ED of a large, academic children's hospital.
METHODS: We developed a conceptual model of BHC patient flow through the ED, incorporating anticipated system changes with both universal suicide risk screening scenarios. Retrospective site-specific patient tracking data from 2017 were used to generate model parameters and validate model output metrics with a random 50/50 split for derivation and validation data.
RESULTS: The model predicted small increases (less than 1 hour) in LOS and wait times for our BHC patients in both universal screening scenarios. However, the days per year in which the ED experienced secure unit overflow increased (existing system: 52.9 days; 95% CI, 51.5-54.3 days; ED: 94.4 days; 95% CI, 92.6-96.2 days; and hospital-wide: 276.9 days; 95% CI, 274.8-279.0 days).
CONCLUSIONS: The DES model predicted that implementation of either universal suicide risk screening scenario would not severely impact LOS or wait times for BHC patients in our ED. However, universal screening would greatly stress our existing ED capacity to care for BHC patients in secure, dedicated patient areas by creating more overflow.

I. Introduction

1. Background

Suicide is the second leading cause of death among youth aged 10–24 years [1,2]. Effective July 1, 2019, the Joint Commission required suicide risk screening for all patients presenting to the emergency department (ED) with a behavioral health complaint (BHC) (National Patient Safety Goals [NPSG] 15.01.01) [3]. Among Medicaid-enrolled youth, nearly half of suicide decedents (44.8%, odds ratio of 2.87 compared to controls) had a healthcare visit in the month before their death [4]. These visits may represent a missed opportunity, and suicide risk screening in the ED and other healthcare settings is an especially important tool for youth involved in the child welfare system [5]. Screening for suicide risk during pediatric healthcare visits, in conjunction with corresponding referrals and interventions, might help prevent suicide [6,7]. Screening during ED and outpatient visits for non-behavioral health complaints has been recommended as a potential way to identify more patients at risk [6].

Two different system interventions have been considered at our pediatric center to help address suicide risk: universal suicide risk screening for ED patients ≥10 years old and hospital-wide patients ≥12 years old. Our existing system and the interventions under consideration involved ED-based evaluation by behavioral health specialists in secure patient care areas for any child at imminent risk for suicide.

2. Importance

The ED includes mental health professionals and secure patient care areas to offer safe and emergent evaluations of patients at imminent risk of suicide. However, these resources are limited. Furthermore, patients with behavioral health complaints in our setting are already subjected to long stays in the ED. It was unknown how the implementation of universal screening would impact the availability of ED resources or throughput for patients who currently require evaluation for BHC.

3. Goals of This Investigation

(1) To test the hypothesis that universal suicide risk screening (ED and hospital-wide) would increase the mean length of stay (LOS) for patients with a BHC by more than 1 hour in a discrete event simulation model of the ED and (2) to compare system performance (LOS, wait times, and secure unit overflow) between models of the existing system and both proposed universal screening scenarios (ED and hospital-wide).

II. Methods

1. Discrete Event Simulation Modeling

We used discrete event simulation (DES) modeling to build a computational representation of patient flow through the part of our ED where patients are treated for BHC. We sought to use DES to predict the system impact on ED patient flow from universal suicide risk screening, since DES is well-suited for testing theoretical changes to a complex system [8,9].

2. Study Setting

Patients 0–21 years old are treated in the study setting: a large, urban, free-standing, academic children’s hospital ED in the United States with approximately 90,000 annual visits.

3. Conceptual Modeling

We created a conceptual model representing the flow of patients who required evaluation by a behavioral health specialist in the ED. The flow of patients through the existing system is depicted in Figure 1, along with theoretical flows that represented anticipated system changes for two proposed scenarios: universal ED screening and universal hospital-wide screening.

Figure 1

Conceptual model for pediatric emergency department (ED) evaluations of patients with behavioral health complaints.

1) Universal ED screening

In this scenario, all ED patients ≥10 years old, including those who did not present for a BHC, would be screened. If imminent suicide risk were detected, then it was assumed that these children would then be evaluated by a behavioral health specialist in the secure area in the ED.

2) Universal hospital-wide screening

In this scenario, all ED patients ≥10 years old and all patients hospital-wide ≥12 years old (regardless of presenting complaint), would undergo screening. Similar to the first scenario, patients who screened positive for imminent risk of suicide in any outpatient clinic would be referred to the ED for an emergent evaluation by a behavioral health specialist in the secure area.

4. Data Collection and Model Parameters

Our team included local experts to ensure that the layout and general allocation of rooms and staff in the model accurately represented our ED. In order to model parameters for medical care, behavioral health evaluation, and disposition times, we utilized site-specific data from actual patients from January 1, 2017 to December 31, 2017. We randomly selected dates to create a 50/50 split of patient tracking data from 2017 for derivation and validation datasets. Our proposed universal screening scenarios include the Columbia-Suicide Severity Rating Scale (C-SSRS), one of five evidence-based screening tools recommended by the Joint Commission for suicide risk screening [10]. While a full assessment of suicide risk may require substantial time and resources, we did not include the duration of screening in our model, since this can be automated and a negative C-SSRS requires less than 3 minutes to complete [11]. A large trial of the C-SSRS screener in an urban adult health system guided our estimates for screen-positive rates in our urban pediatric setting [12]. The proportion of positive responses by ED patients to specific C-SSRS questions was reported, from which we based the proportion of positive screens that we would categorize as low-risk, requiring referral for outpatient resources; moderate risk, requiring a brief safety assessment as well as outpatient resources; or imminent risk, requiring transfer to a secure patient area with a full behavioral health evaluation for risk of suicide.

5. DES Model Building and Validation

Our DES model was built using Simio (v10.165), using parametric probability distributions for all processes, fitted to the shape of retrospective data. The DES model represents how a patient who requires a behavioral health evaluation will flow through the ED, using probability distributions to approximate the real-world variability around the duration of each evaluation and disposition decision. Staffing was built around a standard week-long ED schedule and patient arrival patterns incorporated the variability around each hour of the day, day of the week, and month of the year.

Patient flow through the model (including the existing system and proposed interventions) was verified by our study team. Validation was performed by simulating 1 year of system flow at steady state, and repeating this with 1,000 iterations to compare model output predictions of LOS, wait times for behavioral specialist assessment, and secure unit overflow to actual site-specific data. The ED psychiatry leadership indicated they would be most interested in predictions from a model with at least 95% accuracy; therefore, we set an a priori validation cut-off of within 5% of historical data for the mean of each model output metric.

Model outputs, including LOS, wait times, and secure unit overflow, are dependent on complex interactions between patient needs and available resources, defined by model parameters. We used a validation dataset, also from site-specific records between January 1, 2017 and December 31, 2017, to compare against output metrics from our model built with parameters defined by a separate derivation dataset from the same year. Validation served as the final error-checking step to demonstrate that our model would produce output metrics that adequately represented the real-world system.

6. Model Experiments and Outcome Measures

We simulated 1,000 iterations of 1 year of patients flowing through our model, incorporating the additional demands on behavioral health resources that we expected from the implementation of universal ED screening. We simulated another 1,000 iterations of 1 year, including changes to represent universal hospital-wide screening. For each scenario, we collected LOS, defined by ED arrival to disposition for BHC patients. We also collected wait times for behavioral health evaluation, defined by the time in queue after completing medical clearance, and the number of days each year with unit overflow, during which BHC patients exceeded the space available in secure patient areas (Figure 2). For each iteration, we included a warm-up period of 200 days to reach a steady state. Using descriptive statistics, we compared model outputs from the two proposed scenarios to the model representing the existing system. The existing system model assumes the current medical practice of suicide risk evaluation by a medical provider who then determines the need for behavioral health specialist evaluation.

Figure 2

Emergency department (ED) layout, including the secure patient care area dedicated to patients with behavioral health complaints, adjacent patient care locations (“Area C”), and the decontamination area, which serves as the primary overflow when the number of patients with behavioral health complaints exceeds the space available in secure patient areas.

7. Sensitivity Analyses

In our first sensitivity analysis, we explored how the use of additional ED personnel resources would impact our results. In the existing system, our behavioral health specialists are psychiatry-specific social workers, who perform all behavioral health evaluations in the ED. The addition of brief safety assessments for screen-positive patients would represent additional clinical work for our behavioral health specialists. We performed a theoretical experiment in which other ED social workers completed all brief safety assessments for screen-positive patients not at imminent risk.

We performed additional sensitivity analyses by varying the proportion of positive screens, as our estimates were based on adult data and the screen-positive rate of adolescents for suicide risk might be different.

This project was undertaken as a quality improvement initiative at Children’s National and it did not constitute human subjects research. As such, it was deemed exempt from oversight of the Institutional Review Board.

III. Results

1. Model Parameters

Based on annual volumes at our center, we estimated that universal ED screening of non-BHC patients would detect approximately 684 ED patients per year requiring emergent evaluation by a behavioral health specialist. Universal hospital-wide screening would detect approximately 2,989 outpatients per year requiring emergent behavioral health evaluations, in addition to the patients detected with universal ED screening. Other model parameters, including parameters for the probability distribution used for each process, are reported in Table 1.

Table 1

Model parameters

Process	Data source	Parameter	Service time distribution[c] (min)
Patients evaluated by psychiatry team
Arrival rate[a]	Derivation dataset (n = 1,058)	Poisson (λ)	-
Admission rate	Derivation dataset (n = 448 of 1,058)	Fixed 42.3% chance	-
Boarding rate[b]	Derivation dataset (n = 85 of 1,058)	Fixed 0.08% chance	-
Medical care for patients with behavioral health complaints	Derivation dataset (n = 1,049)	-	9 + Weibull (155, 1.21)
Complete behavioral health evaluation	Derivation dataset (n = 1,017)	-	39 + Weibull (99.3, 1.08)
Wait time for inpatient bed, admitted patients	Derivation dataset (n = 397)	-	9 + Weibull (527, 0.77)
Boarding time for emergency department (ED) boarders	Derivation dataset (n = 56)	-	137 + Weibull (1220, 1.27)
Nurse discharge process	Expert opinion		Triangular (5, 10, 15)

Variable arrival rate with distinct λ each hour, ranging from 0.001 to 1.089 patients per hour depending on month, day of week, and hour of day.

Boarding rate includes only patients who board in the ED prior to discharge home. Patients that board in the ED prior to admission are included in admission rate.

For distribution Weibull (beta, alpha), beta is scale parameter, alpha is shape parameter.

The Weibull distribution has a closed-form inverse cumulative distribution function given by: F-1(U) = b [−ln(1-U)]1/a

To generate the Weibull distributions, random subset of n = 100 from derivation dataset was used for: medical care, complete behavioral health evaluation, and wait time.

Parameters for the Triangular distribution include (minimum, mode, maximum).

2. Model Verification and Validation

Local stakeholders, including ED psychiatry leadership, reviewed our model and the anticipated patient flow for both proposed scenarios, verifying that patients moved from process to process appropriately. We validated the model, with mean LOS, wait times, and unit overflow within 5% of our validation dataset (Table 2).

Table 2

Model output performance against the ED validation dataset from 2017

Metrics for BHC patients	Validation dataset (n = 925)	Model output	Difference (%)
Wait time (hr)	3.04 (2.37)	2.91 ± 0.01	−4.11
Length of stay (hr)
Overall	10.81 (10.94)	11.28 ± 0.05 4.35
Admitted patients	15.58 (12.57)	15.58 ± 0.09	−0.01
Boarded patients	25.91 (12.52)	26.35 ± 0.24	1.69
Discharged patients	5.22 (2.75)	5.17 ± 0.02	−0.96
Secure unit overflow (day/yr)	52 (0)	52.90 ± 1.40	1.73

Values are presented as mean (standard deviation) or mean ± half-width of 95% confidence interval. ED: emergency department, BHC: behavioral health complaint.

3. Model Experimentation Outcomes

Patients with BHC in the ED had similar mean LOS in the model of the existing system (11.28 hours; 95% confidence interval [CI], 11.23–11.33 hours), universal ED screening (11.48 hours; 95% CI, 11.43–11.53 hours), and universal hospital-wide screening (11.88 hours; 95% CI, 11.82–11.94 hours). The number of days per year that BHC patients exceeded the space available in secure patient areas increased from 52.9 days (95% CI, 51.5–54.3 days) in the model of the existing system to 94.4 days (95% CI, 92.6–96.2 days) and 276.9 days (95% CI, 274.8–279.0 days) for universal ED screening and hospital-wide screening, respectively (Figure 3). The complete model outputs for the existing system and both universal screening scenarios are presented in Table 3.

Figure 3

Simio measure of risk and error (SMORE) plot, showing the number of days each year with overflow of patients with behavioral health complaints exceeding the capacity of secure patient care areas. The model outputs for our existing system, as well as models with universal emergency department (ED) screening and universal hospital-wide screening, are represented as a box-and-whisker plot demonstrating the differences in expected unit overflow between these models. The means are presented as orange circles, 95% confidence intervals (CIs) around means are represented by beige bars, and 95% CIs around the 25th and 75th percentiles are represented by blue bars.

Table 3

Outcome measures for models of the existing system and two proposed system changes

Metrics for BHC patients	Model output
	Existing system	Universal ED screening	Universal hospital-wide screening
	Wait time (hr)	2.91 ± 0.01	3.09 ± 0.02	3.57 ± 0.03
Length of stay (hr)
Overall	11.28 ± 0.05	11.48 ± 0.05	11.88 ± 0.06
Admitted patients	15.58 ± 0.09	15.76 ± 0.09	16.17 ± 0.09
Boarded patients	26.35 ± 0.24	26.38 ± 0.24	26.88 ± 0.25
Discharged patients	5.17 ± 0.02	5.33 ± 0.02	5.82 ± 0.03
Secure unit overflow (day/yr)	52.90 ± 1.40	94.40 ± 1.80	276.90 ± 2.10

Values are presented as mean ± half-width of 95% confidence interval.

ED: emergency department, BHC: behavioral health complaint.

4. Sensitivity Analyses

The predicted increase in unit overflow (to 93.5 days per year with universal ED screening; 95% CI, 91.9–95.1 days per year) was not mitigated by using ED social workers to relieve psychiatry-specific social workers from performing brief safety assessments (other metrics included in Table 4). Varying the proportion of positive screens, we again found minimal changes in LOS between the model of the existing system and universal ED screening. In all sensitivity analyses, the number of days of unit overflow was higher with universal screening than in the model of our existing system. As we increased the percentage of screen-positive, imminent-risk patients, unit overflow progressively worsened (see Supplementary Table S1).

Table 4

Outcome measures for models of the existing system and two proposed system changes, with support from social workers

Metrics for BHC patients	Model output
	Existing system	Universal ED screening[a]	Universal hospital-wide screening[a]
	Wait time (hr)	2.91 ± 0.01	3.08 ± 0.01	3.49 ± 0.03
Length of stay (hr)
Overall	11.28 ± 0.05	11.39 ± 0.05	11.85 ± 0.06
Admitted patients	15.58 ± 0.09	15.64 ± 0.09	16.16 ± 0.08
Boarded patients	26.38 ± 0.24	26.54 ± 0.24	26.95 ± 0.24
Discharged patients	5.17 ± 0.02	5.33 ± 0.02	5.75 ± 0.03
Unit overflow (day/yr)	52.90 ± 1.43	93.50 ± 1.60	266.00 ± 2.02

Values are presented as mean ± half-width of 95% confidence interval.

BHC: behavioral health complaint.

Social worker performs brief safety assessment. In our initial testing of proposed system changes, we programmed our model to have brief safety assessments performed by psychiatry social workers, responsible for all other behavioral health complaint management in the emergency department. Subsequent pre-implementation planning has proposed the use of emergency department social workers for these brief safety assessments to make sure psychiatry social workers remain available for complete behavioral health evaluations.

IV. Discussion

We developed a theoretical model that predicts significant system impacts of implementing universal screening for suicide risk. We validated the DES model and compared screening scenarios using mean LOS, wait times, and secure unit overflow as output metrics. Based on this DES model, we expect minimal changes to LOS for BHC patients after implementation of either universal suicide risk screening scenario. However, our model suggests that universal suicide risk screening in the ED could nearly double the number of days each year that BHC patients will exceed the secure patient areas dedicated to this population.

Although this work is theoretical, we employed strategic elements that are considered central to improving experiences for real patients, including policies and measurement, quality, and innovation and technology [13]. We evaluated LOS as our primary outcome because this patient-centered metric is a well-recognized quality measure and directly relevant to the experience of patients. Specifically, each additional hour of ED LOS is associated with a 0.7% decrease in top satisfaction ratings [14]. The focus of our current work was to predict the impact of two proposed universal screening scenarios in order to make the best decision for patients, safety, and the situational readiness of the ED. Future policies to improve ED outflows, such as decreasing wait times for an inpatient psychiatry bed, could have a greater impact on patient experience than any effort to increase secure unit capacity [15]. DES is an optimal tool for testing the impacts of such potential interventions [8].

The coronavirus disease 2019 (COVID-19) pandemic has been associated with increasing demand and decreasing access to mental health resources [16-18], which has created significant challenges as our team has worked to more efficiently discharge each patient with an appropriate plan for outpatient resources. Nevertheless, the emotional toll of the COVID-19 pandemic has also increased the urgency of addressing mental health needs [19]. Efforts by local stakeholders to screen more broadly for suicide risk, short of universal screening [20], have continued at our institution over the last year. Even before the COVID-19 pandemic, suicide rates in the United States have been rising steadily since 1999 [2,21,22], and are projected to increase further related to the pandemic [23,24].

As our systems continue to face new challenges, such as increasing numbers of patients needing mental health services and limited resources, computer-based modeling is an economically friendly and useful tool. DES has been used for decades to support staffing and operational planning in hospitals [25,26] and more recently to evaluate the impact of quality improvement efforts in the ED setting [27]. Incorporating insights from simulation modeling into the planning phases of quality improvement efforts can help avoid unintended negative consequences [28]. For example, although universal screening minimally impacted patients’ LOS in our model, the negative impact on secure unit overflow was significant and severe. During periods of unit overflow, patients with BHCs exceed the secure patient areas designed for them to be safely observed and managed. In our center, BHC patients who exceed the secure space can be safely managed in the ED area designed for decontamination. However, this space is not designed for patient care, and during periods of overflow it becomes unavailable for decontamination emergencies. Sufficient decontamination space is critical for an ED to respond to specific mass casualty incidents [29]. If universal screening had been suddenly implemented in the real world without the use of DES in planning our intervention, the situational readiness of our ED could have been compromised.

Our simulation work was impactful in two ways. First, ED psychiatry leadership proceeded with a phased approach of suicide risk screening, in part because our model demonstrated the potential for drastic worsening of behavioral health patient overflow out of the secure patient area [20]. Second, we have transitioned to a new model of ED social workers providing second-tier screening instead of routing newly identified cases to psychiatry social workers. The risk of seriously diminishing situational readiness by abruptly adopting universal screening may be relevant to stakeholders in EDs at other urban and suburban children’s hospitals, where pediatric mental health boarding is already a common problem [30]. The model outcomes from this study may not be generalizable to other centers, where unit overflow is a rare phenomenon at baseline. Simulation modeling prior to implementation of universal screening may help avoid other unanticipated negative consequences of new demands on scarce behavioral health resources.

This study has several limitations. The most important limitation of our model is that we treated the wait time for inpatient beds as a model input independent of the number of patients requiring admission. This assumption, which allowed us to constrain our modeling efforts to the ED as a system, was appropriate for our clinical setting because we have the ability to transfer admitted patients to other regional facilities. If we were unable to transfer patients to outside facilities, our model outputs would underestimate the ED flow impact of suicide risk screening. For this reason, our findings may not be generalizable to ED settings without the ability to transfer patients to outside psychiatric facilities.

We assumed that 100% of ED patients ≥10 years old and all patients hospital-wide ≥12 years old would be eligible for screening. Acutely ill ED patients were included in the model because they can be screened after they are stabilized. However, we did not exclude patients with respiratory failure or those taken for emergency surgery, who would be ineligible for screening in the real world, potentially exaggerating the detrimental impact of universal screening in our model.

Our model may require modifications as we gather additional observational data from our site’s experience with the C-SSRS. We did not perform direct observations to confirm the duration of processes in our model. However, site-specific patient tracking data are routinely used for quality improvement and operational decisions at our center and a team of local experts reviewed our retrospectively derived process durations. Although the C-SSRS has been extensively studied in adults, we have not yet assessed test characteristics or the proportion of positive screens in our ED [10-12]. We also did not perform direct observations to determine the duration of screening. We did not anticipate significant workflow or educational hurdles related to the completion of screening: bedside nurses at our institution are accustomed to the use of standardized screening tools during initial assessments and are trained in the use of 18 screening examinations, including universal screening for domestic safety and disease-specific screens such as the Westley Croup Score and Acute Concussion Evaluation. Furthermore, the ED psychiatry leadership estimated that the C-SSRS screener can be completed in under 3 minutes, which is less than 1% of the total mean LOS experienced by these patients.

The operational planning for universal suicide risk screening was informed by these modeling efforts. For example, in our models, psychiatry social workers perform a brief safety assessment for every screen-positive ED patient, in addition to their complete evaluation for existing and screen-positive patients at imminent risk of suicide. Currently, ED social workers complete these evaluations. In preparation for universal ED screening, the ED psychiatry leadership has made other efforts to optimize resource allocation, including support for medical providers to better discern which patients require behavioral health evaluations, the addition of an ED psychiatrist to safely discharge patients with BHCs who might otherwise be hospitalized, and more expeditious transfer of patients when our inpatient service is at capacity. Additionally, some outpatient clinics (notably, the outpatient psychiatry clinics) have already begun documenting suicide risk screens at least once annually for older patients. Based on these ongoing, incremental efforts to prepare for and increase suicide risk screening, the real-world detrimental system impacts of implementing universal screening would likely be less pronounced than our model predictions.

The overall lesson from our experience is that DES modeling suggested that an unacceptable increase in unit overflow would take place in response to an abrupt implementation of universal suicide risk screening. Incremental increases in screening are critical to maintaining the situational readiness of our ED, and successful implementation of screening should be accompanied by aggressive strategies to decrease waiting time for hospitalization, reduce boarding times, and minimize unnecessary admissions.