Identifying diverse concepts of discharge failure patients at emergency department in the USA: a large-scale retrospective observational study.

OBJECTIVES: Identifying patients who are at high risk for discharge failure allows for implementation of interventions to improve their care. However, discharge failure is currently defined in literature with great variability, making targeted interventions more difficult. We aim to derive a screening tool based on the existing diverse discharge failure models. DESIGN, SETTING AND PARTICIPANTS: This is a single-centre retrospective cohort study in the USA. Data from all patients discharged from the emergency department were collected from 1 January 2015 through 31 December 2017 and followed up within 30 days.
METHODS: Scoring systems were derived using modified Framingham methods. Sensitivity, specificity and area under the receiver operational characteristic (AUC) were calculated and compared using both the broad and restricted discharge failure models.
RESULTS: A total of 227 627 patients were included. The Screening for Healthcare fOllow-Up Tool (SHOUT) scoring system was derived based on the broad and restricted discharge failure models and applied back to the entire study cohort. A sensitivity of 80% and a specificity of 71% were found in SHOUT scores to identify patients with broad discharge failure with AUC of 0.83 (95% CI 0.83 to 0.84). When applied to a 3-day restricted discharge failure model, a sensitivity of 86% and a specificity of 60% were found to identify patients with AUC of 0.79 (95% CI 0.78 to 0.80).
CONCLUSION: The SHOUT scoring system was derived and used to screen and identify patients that would ultimately become discharge failures, especially when using broad definitions of discharge failure. The SHOUT tool was internally validated and can be used to identify patients across a wide spectrum of discharge failure definitions. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The Screening for Healthcare fOllow-U p Tool (SHOUT) scoring system is different than other tools reported in the literature and has more potential for applying to the general population.

The SHOUT scoring system was derived from a large sample size and is applicable to diverse concepts of discharge failure model, giving it broad application.

This is a relatively simple and easy scoring calculation to predict patients with different types of discharge failures.

To the best of our knowledge, this is the first reported emergency department discharge failure prediction tool that combined all validated discharge failure risk factors using a least absolute shrinkage and selection operator regression model, making it a more accurate model.

As a single-centre retrospective data analysis, limited and potential incorrect information, missing data and potential patient population selection bias cannot be avoided.

Introduction

Traditional practice recommends arranging timely clinic follow-up for patients who are discharged from the emergency department (ED). Such follow-up has been shown to improve patient-centred care specifically for disease prevention, monitoring and management.1 2 However, nearly one-third of ED patients who have sought access to healthcare rarely follow-up with their primary care physician (PCP) or specialist after ED discharge.2 Such patients were historically considered a discharge failure. However, the definition of discharge failure varies among most studies. A very broad definition used in previous studies included patients who had not shown for a clinic appointment after an index ED discharge, had no clinic appointment after an index ED discharge or had short ED returns (eg, 3, 7, 14 or 30 days).3–6 Such definitions may not be accurate since patients might use the ED for episodic acute care and have no urgent clinic follow-up needs. On the contrary, patients with extremely short ED returns or that had significant deviation from the usual treatment regimen following an index ED visit may have been inappropriately discharged in the first place from their initial ED visit. Alternatively, their ED return could be unrelated to the initial visit. We believe that a more restricted definition of discharge failure truly reflecting the value of arranging timely clinic follow-up should be applied. Unfortunately, such studies are lacking in the current literature.

Six screening tools have been reported currently to identify patients with high-risk potential for discharge failure.7–11 However, all of these tools are intended for screening older patients with poor-to-moderate discrimination, and none of them can be used in general patient populations. A majority of these tools used self-assessed questionnaires (eg, assistance with daily activity, healthcare recommendation for added assistance, having a predicted need for more help after ED discharge) and rarely linked screening with predictive risk factors.9–11 Many studies in the past have identified a variety of risk factors predictive of discharge failure.3 4 12 The most common validated ones are either social or medical factors. These common biological and social factors include insurance type, homelessness, lack of PCP, age, sex and race/ethnicity,4 5 7 12 13 whereas medical factors could be attributed to alcohol/drug history and chronic medical conditions.14 15 Given that validated risk factors have already been reported, the derivation of a tool associated with such factors might be beneficial.

Our goal is to identify patients at risks for discharge failures so that efficient interventions can be implemented to prevent ED returns, reduce cost and save healthcare resources. Therefore, we aim to (1) determine the differences in ED-discharged patients using either a broad or restricted discharge failure model and (2) derive novel tools associated with predictive risk factors for the initial screening of ED patients for discharge failures.

Methods

Study setting and design

This was a single-centre retrospective cohort study. The study hospital is a publicly funded county hospital and urban tertiary referral centre. The study hospital ED is a level-1 trauma centre, acute chest pain and comprehensive stroke centre whose ED volume reached approximately 1 20 000 visits annually. The ED also sponsors an Emergency Medicine (EM) residency programme. This study was approved by John Peter Smith Health Network Institutional Review Board.

Study participants

Patients who presented to study ED from 1 January 2015 to 31 December 2017 and were subsequently discharged after the index ED visit were included in this study. The study hospital system uses the same Electronic Medical Record (EMR) including ED, hospital and clinics. The medical record of all enrolled patients was retrieved automatically until 1 February 2018. This allowed all enrolled patients to have 30 days after the index ED discharge to follow-up. All patient data were electronically retrieved by data managers from the Department of Information Technology. We excluded patients during the index ED visits who (1) were admitted, (2) expired, (3) transferred to other facilities, (4) left without being seen, eloped or left against medical advice and (5) prisoners. Since this study mainly focused on the characteristics of discharge failure, we further divided discharged patients into two large categories of patients without discharge failure and patients with broad discharge failure potential. We defined patients without discharge failure as meeting all of the following criteria: (1) patients visited their PCP/specialist clinic within 1 year from the index ED discharge (under normal circumstance, at least one clinic visit should be ranged every year for regular check-up and screening); (2) patients visited their PCP/specialist clinics prior to their ED revisits and (3) patients had no ED revisits within 30 days.

Broad and restricted discharge failure models

In general, discharge failure was defined as ED revisits within a short period of time from the index ED visit (eg, 3 , 7, 14 or 30 days) and poor patient adherence to PCP or specialist clinic follow-up. We divided patients with discharge failure into broad and restricted categories. Patients with restricted discharge failures were confirmed discharge failure within 30 days from the index ED discharges. Whereas, patients with broad discharge failures included not only ones with confirmed discharge failures but also ones with discharge failure potential or uncertainty. Broad discharge failure was considered if patients met one of the following criteria: (1) patient had no PCP/specialist follow-ups from the index ED discharge; (2) patient had clinic follow-up longer than 1 year from the index ED discharge; (3) patients returned to the ED prior to their clinical follow-up; (4) patients with ED returns and clinic visits on the same day and (5) patients with ED returns within 30 days from the index ED discharge (see detail in online supplementary table 1). As mentioned above, multiple factors can impact patient follow-up after the index ED discharge (eg, patient conditions do not require clinical follow-up, patient ED condition completely resolved). Additionally, patients could revisit the ED appropriately or unrelated to their initial ED visit prior to their clinic follow-ups (eg, acute trauma). These patients might need to be excluded from the discharge failure category. Therefore, a more restricted discharge failure model was applied to the study patients. Restricted discharge failure was considered if patients met all the following criteria: (1) patients returned to the ED prior to their clinic follow-ups, (2) such ED revisits were within 30 days from the index ED discharge and (3) patients were discharged from their ED return and the visit reason was considered inappropriate ED utilisation. To satisfy diverse concepts of discharge failure in the literature, we expanded our restricted discharge failure models to the following four extended-restricted discharge failure groups: (1) restricted discharge failure with subsequent ED return of <3 days); (2) restricted discharge failure with ED return of <7 days; (3) restricted discharge failure with ED return of <14 days and (4) restricted discharge failure with ED return of <30 days.

Appropriateness of ED utilisation

New York University ED Algorithm (NYUA) was used in this study to determine the appropriateness of ED return visits.16 Briefly, the four major categories were used in NYUA: (1) emergent and not avoidable, considered appropriate ED visits; (2) primary care treatable, defined as care that can be safely provided in a primary care setting without the need for emergent treatment; (3) emergent care needed but preventable/avoidable, defined as patients whose disease conditions can be prevented/avoided if preventive care is received in a timely fashion and (4) non-emergent. Appropriate ED utilisation was considered if patients met the emergent not avoidable category criteria and inappropriate utilisation was determined if patients were classified within the other three categories. However, since NYUA is only used to determine the appropriateness of ED utilisation among ED-discharged patients, for patients who revisited ED within 30 days, appropriate ED utilisation was also considered if such patients were: (1) admitted to hospital, (2) moved to the operating room, (3) transferred to other facilities or (4) expired.

Variables

Variables chosen for model building were selected from previous studies and reviewed by clinicians experienced in healthcare quality studies to ensure consistent clinical significance.4–8 Patient general characteristics including age, sex and race/ethnicity were collected. Other patient and clinical variables included were: (1) patient total ED length of stay (LOS), divided into two categories of LOS equal to or less than 4 hours and LOS longer than 4 hours, (2) patient waiting room time in minutes, (3) mode of arrival, divided into two categories of healthcare-assisted arrival (ambulance or hospital/healthcare facility-arranged transportation) and other (private car, public transportation, taxi, wheelchair, ambulatory, police or unknown), (4) level of acuity, divided into three categories based on Emergency Severity Index (ESI) level including high (ESI 1–2), moderate (ESI 3) and low (ESI 4–5), (5) homeless status, (6) patient’s last vital signs at disposition (including heart rate, respiratory rate, blood pressure, oxygenation and temperature): divided into two categories of patients who had normal vital signs versus ones who had any abnormal vital signs (eg, heart rate <50 or >100, respiratory rate <8 or >20, systolic blood pressure <90 or >140 mm Hg, diastolic blood pressure <60 or >90 mm Hg, pulse oximetry <94%, temperature >100.4°F(38°C) or <96.8°F(36°C)), (7) next healthcare visit (eg, ED, PCP/specialist clinic or none) and its time interval from the index ED discharge, (8) whether patients had a PCP assigned, (9) number of medications prescribed on the index ED discharge, divided into two categories of patients who had prescriptions versus those who had none, (10) insurance status and (11) presence of chronic disease, with chronic disease conditions determined using the chronic condition indicator (CCI) for the International Classification of Diseases tenth revision, Clinical Modification. CCI was developed as part of the Healthcare Cost and Utilization Project by the Agency for Healthcare Research and Quality.17

Derivation and validation of SHOUT scoring system

To identify potential ED discharge failure patients, the Screening for Healthcare fOllow-Up Tool (SHOUT) scoring system was derived. Variables chosen for model building were selected from previous studies and reviewed by clinicians experienced in healthcare quality studies to ensure consistent clinical significance. We built five scoring systems using predictive logistic regression modelling. Each model predicted a specific outcome as defined above: broad discharge failure and 3-day, 7-day, 14-day and 30-day restricted discharge failure. In our sample, less than 5% of the patients had missing data on predictor variables (specific variables denoted in table 1, see online supplementary table 2). To build the predictive model for broad discharge failure, we used 50% of the data to train the model and 50% to test the model because we had a large sample size. We dichotomized the predictors for ease of use in clinical practice. Neither making the variables continuous nor including interaction terms added substantially to the model’s performance, and we preferred parsimony for generalizability. To avoid overfitting, we used the least absolute shrinkage and selection operator (LASSO) to fit the most informative but parsimonious model.18 The LASSO model predicted a patient’s probability of broad discharge failure, and we used a threshold value to classify the patient (0 or 1). Simple point scoring systems were then derived using methods described by Framingham with minor modifications.19 We used the receiver operating characteristic curve to define the threshold as the value that maximises the model’s sensitivity and minimises the false positive rate (1−specificity). Because the model’s primary purpose was to classify patients, we focused on the model’s discriminative abilities. Accuracy of the prediction was reported with sensitivity, specificity, positive and negative likelihood ratios. Scores were calculated among all patients in both the derivation and validation groups, the sensitivity, specificity, positive and negative likelihood ratios and AUC were compared between groups of different models in both the derivation and validation data.

Table 1

Study patient general characteristics

	Diverse discharge failure models					Control
	Broad n=194 270	Restricted (3 days) n=2086	Restricted (7 days) n=3518	Restricted (14 days) n=4957	Restricted (30 days) n=6715	No discharge failure n=33 357
Age, year
Mean(SD)	39 (16)	47 (14)	47 (14)	47 (14)	47 (14)	47 (14)
Median (IQR)	38 (27, 51)	49 (37, 58)	49 (37, 58)	49 (37, 57)	48 (37, 57)	49 (36, 58)
Race/ethnicity, n (%)
Non-Hispanic white	63 438 (33)	867 (42)	1424 (40)	1948 (39)	2573 (38)	9813 (29)
Others	130 832 (67)	1219 (58)	2094 (60)	3009 (61)	4142 (62)	23 544 (71)
Sex, n (%)
Male	96 882 (50)	1160 (56)	1868 (53)	2514 (51)	3238 (48)	12 141 (36)
Female	97 380 (50)	926 (44)	1650 (47)	2443 (49)	3477 (52)	21 216 (64)
PCP provider, n (%)
Yes	129 345 (67)	445 (21)	658 (19)	821 (17)	1014 (15)	3621 (11)
No	64 925 (33)	1641 (79)	2860 (81)	4136 (83)	5701 (85)	29 736 (89)
Homeless, n (%)
Yes	16 783 (9)	663 (32)	1090 (31)	1386 (28)	1694 (25)	1719 (5)
No	1 77 487 (91)	1423 (68)	2428 (69)	3571 (72)	5021 (75)	31 638 (95)
Means of arrival, n (%)
Healthcare-assisted	51 181 (27)	802 (39)	1268 (36)	1682 (34)	2106 (31)	5449 (16)
Others	141 688 (73)	1278 (61)	2242 (64)	3264 (66)	4597 (69)	27 840 (84)
Any insurance, n (%)
Yes	99 827 (51)	1757 (84)	2942 (84)	4112 (83)	5515 (82)	28 774 (86)
No	94 268 (49)	329 (16)	576 (16)	845 (17)	1200 (18)	4579 (14)
ESI level, n (%)
ESI (1,2,3)	150 277 (78)	1606 (77)	2654 (76)	3740 (76)	5062 (75)	27 621 (83)
ESI (4,5)	43 423 (22)	477 (23)	859 (24)	1211 (24)	1646 (25)	5680 (17)
Last vitals on discharge, n (%)
Normal	23 338 (13)	235 (12)	389 (12)	547 (12)	739 (11)	4287 (13)
Abnormal	162 312 (87)	1739 (88)	2968 (88)	4190 (88)	5690 (89)	28 172 (87)
Any chronic conditions, n (%)
Yes	91 194 (47)	1338 (64)	2277 (65)	3161 (64)	4224 (63)	18 649 (56)
No	103 076 (53)	748 (36)	1241 (35)	1796 (36)	2491 (37)	14 708 (44)
Prescriptions on discharge, n (%)
Yes	129 198 (67)	1177 (56)	2086 (59)	3050 (62)	4322 (64)	22 356 (67)
No	65 072 (34)	909 (44)	1432 (41)	1907 (38)	2393 (36)	11 001 (33)
Length of ED stay, n (%)
Equal or longer than 4 hours	127 708 (66)	808 (39)	1364 (39)	1907 (38)	2550 (38)	13 964 (42)
Less than 4 hours	66 538 (34)	1277 (61)	2153 (61)	3049 (62)	4164 (62)	19 393 (58)

ESI, Emergency Severity Index; PCP, primary care physician.

Data analysis

Student’s t-test was used to compare continuous variables while Pearson chi-square (χ2) analysis was used to compare categorical variables between groups. We plotted the inverse of the Kaplan-Meier survival curves for the frequency comparison of patients who returned to ED versus those who had clinic follow-up after the index ED discharge. Methods used to derive and validate scoring systems are addressed above. All descriptive and statistical analyses were conducted using STATA V.14.2. A p value <0.05 was considered statistically significant.

Patients and public involvement

Patients and the public were not directly involved in this study.

Results

General Information

A total of 227 627 ED-discharged patients were retrieved from the EMR with only 33 357 patients categorised to the without broad discharge failure group (figure 1). Overall, 85% (194 270/227 627) of patients were considered broad discharge failures and only 15% of ED-discharged patients had their clinic follow-ups within the study period. When restricted discharge failure models were applied, 3.0% (6715/227 627) of patients were considered restricted discharge failures within 30 days, 2.2% (4957/227 627) within 14 days, 1.5% (3518/227 627) within 7 days and 0.9% (2086/227 627) within 3 days. Patients who had restricted discharge failure were more likely white, homeless, relied on healthcare assisted transportations, had chronic disease conditions, had extended insurance coverage, and were less likely to have a PCP assigned in comparison to patients with broad discharge failures. Patients with no discharge failures tended to be female, not homeless, used less health-assisted transportation and had less PCP coverage (table 1). When reverse Kaplan-Meier curves were drawn among the study patients who had either ED returns or clinical follow-up visits after the index ED discharge, it showed that 24% of patients returned to the ED within 7 days, while 18% of patients had clinic follow-ups within 7 days. Similarly, 46% of patients returned to the ED within 30 days and 45% of patients had a clinic follow-up within 30 days. At 32 days, the curves crossed indicating that patients sought clinic visits more frequently than ED return visits after 32 days. The graph also showed a median of 38 days for subsequent ED returns in comparison to a median of 37 days for subsequent clinic follow-up in this cohort (figure 2). Our results indicated that a high frequency of ED returns occurred within the first 32 days from the index ED discharge.

Figure 1

The study flow diagram.

Figure 2

Time to next-event curve to determine the probability of subsequent events (ED return vs clinic follow-up) occurring among discharged patients. ED, emergency department; PCP, primary care physician.

Derivation of SHOUT scoring systems for diversity of discharge failure models

Nine independent variables predicting discharge failures were: (1) homelessness, (2) PCP status, (3) male sex, (4) history of chronic diseases, (5) lack of insurance, (6) low level of acuity (ESI 4–5), (7) White race/ethnicity, (8) arriving by health-assisted transportation and (9) abnormal vital signs at discharge. These factors were incorporated into the SHOUT scores for discharge failure models (table 2). These scores were applied back to the derivation data yielding good discriminations indicating the feasibility of using SHOUT scores for the initial screening of different discharge failure models (table 3).

Table 2

SHOUT scoring system for different discharge failure models

	Broad	Restricted (3 days)	Restricted (7 days)	Restricted (14 days)	Restricted (30 days)
Sex
Female	0	0	0	0	0
Male	2	1	1	1	1
Race/ethnicity
Non-Hispanic white	1	1	1	1	1.5
Others	0	0	0	0	0
PCP provider assigned
Yes	21	0	0	0	0
No	0	6	6	11	15.5
Homeless
Yes	7	5	5	7	9
No	0	0	0	0	0
Means of arrival
Healthcare assisted	6	1	1	1.5	1.5
Others	0	0	0	0	0
Any insurance
Yes	0	3	3	4	5
No	10.5	0	0	0	0
Last vital signs on discharge
Abnormal		1	1	1	1.5
Normal		0	0	0	0
ESI level
ESI (1,2,3)	0	0	0	0	0
ESI (4, 5)	1.5	1	1	2	3
History of chronic conditions
Yes	0	1	1	1.5	2
No	1	0	0	0	0
Score range	0–50	0–20	0–20	0–30	0–40
Predicted discharge failure	≥9	≥9	≥9	≥14	≥20

Table 3

Predictive performance of different discharge failure models in derivation study

Outcome	AUC (95% CI)	Sensitivity (%)	Specificity (%)	LR(+)	LR(−)
Broad	0.83 (0.83 to 0.84)	80	71	2.77	0.28
Restricted (3 days)	0.79 (0.78 to 0.80)	86	60	2.14	0.24
Restricted (7 days)	0.79 (0.79 to 0.80)	86	60	2.17	0.23
Restricted (14 days)	0.79 (0.78 to 0.80)	84	61	2.18	0.25
Restricted (30 days)	0.79 (0.78 to 0.79)	82	63	2.21	0.29

AUC, area under the receiver operational characteristic; LR, likelihood ratio; ESI, Emergency Severity Index; PCP, primary care physician; SHOUT, Screening for Healthcare fOllow-Up Tool.

Validation of SHOUT scoring system for discharge failure models

SHOUT scores were again applied back to the study validation data using different discharge failure models. First, AUC comparisons of SHOUT scores predicting patients with restricted discharge failure within 3, 7, 14 and 30 days were performed. Similar AUCs predicted patients with short-term restricted discharge failure in this cohort (table 4). Second, when the SHOUT score was applied to patients with broad discharge failures, higher AUC (0.84, 95% CI 0.84 to 0.84) yielded a sensitivity of 80%, specificity of 72%, positive likelihood of 2.85 and negative likelihood of 0.27 (table 4).

Table 4

Predictive performance of different discharge failure models in validation study

Outcome	AUC (95% CI)	Sensitivity (%)	Specificity (%)	LR(+)	LR(–)
Broad	0.84 (0.84 to 0.84)	80	72	2.85	0.27
Restricted (3 days)	0.79 (0.78 to 0.80)	85	60	2.13	0.25
Restricted (7 days)	0.80 (0.79 to 0.80)	87	61	2.20	0.22
Restricted (14 days)	0.79 (0.78 to 0.80)	85	62	2.21	0.24
Restricted (30 days)	0.79 (0.78 to 0.79)	82	63	2.22	0.29

AUC, area under the receiver operational characteristic; LR, likelihood ratio.

Discussion

Timely arrangement of post-ED follow-up is critical to ensure patient safety, monitor patient disease progression and adjust management properly.2 20 The SHOUT scoring system was derived and internally validated to differentiate patients with different discharge failure models and shown to be broadly applicable among the types of discharge failure patients. In this study, we categorised patients as either having broad discharge failure potential or having short-term restricted discharge failure. Our study findings add some evidence to the literature pool on capable of early recognising different ED discharge failure patients, thus could provide the potential to implement interventions earlier to prevent discharge failures.

It is reported that providing a follow-up appointment prior to the patient departing the ED can significantly increase follow-up care.21 However, making real-time PCP appointments among all ED-discharged patients might be a challenge with limited healthcare resources. Therefore, our study differentiated patients between ‘broad’ and ‘restricted’ discharge failures. Given the uncertainty of ED returns and poor adherence for clinic follow-up, we believe that the restricted discharge failure prediction tool with its higher sensitivity and small patient size can provide better ED administrative value (eg, capturing more patients and providing real-time patient PCP appointments at patient discharge). Whereas, a broad discharge failure tool with higher specificity and large patient size can better help with healthcare utilisation (eg, capturing more ‘true’ discharge failure patients and limiting the urgent needs for PCP follow-up). Different institutions can choose the one that better fits their own operational needs.

Risks identified in our study to predict patients with discharge failure have also been validated in previous studies to a certain level.3 4 13 Lack of insurance coverage, lack of a PCP, homelessness and chronic diseases are most commonly addressed in the literature with different discharge failure models.5 12 14 Lacking insurance coverage prevented patients from seeking healthcare follow-up and incentivized patients to use the ED as their medical home, which usually resulted in inappropriate ED utilisation.22 Patients with homelessness and chronic disease conditions more frequently had discharge failures due to the certain association between homeless patients and chronic disease conditions.23 Studies showed homeless patients had more chronic diseases in comparison to general population.24 Additionally, homeless patients tended to use ED more often as their medical home resulting in higher inappropriate ED utilisation.25 Our study also showed that patients of lower acuity (ESI 4–5), male and non-Hispanic-White ethnicity had more discharge failure. Similar findings reported that patients with lower acuity, male and White race/ethnicity had higher inappropriate ED visits and higher 72 hours ED returns.6 15 However, such findings are controversial in different studies probably due to different study patient populations.26

Though risks predictive of broad and restricted discharge failure seem similar, three risks had opposite effects on such predictions. Lack of PCP assignment, presence of chronic disease conditions and healthcare insurance coverage seemed to predict restricted discharge failure and protected patients from broad discharge failures. This might be partly due to current study hospital healthcare policies. The study hospital advocates for PCP assignments and clinic follow-up arrangements, provides charity insurance coverage among certain patients (eg, high psychosocial risks, homeless, low-income residents) and has developed outreach programmes for patients with special needs (eg, homeless, chronic heart failure outreach programmes).25 27 It has been reported that these patients had high risk of short-term ED returns (eg, 72 hours) both in the literature and in our own study.3 14 25 In addition, such policies are not uncommon across publicly funded or nonprofit hospitals in the USA.28 29 However, when applied to patients with long-term discharge failure potential, such effects protected against broad discharge failures. This is consistent to other reports in the literature.30 Therefore, we believe that the SHOUT score for broad discharge failure can be used more broadly in a diversity of hospital settings (eg, charity, public-funded, Veteran Affair, private or community hospitals). However, the SHOUT score for restricted discharge failure might be limited to public-funded hospitals with similar policies as the study hospital.

This study has several strengths: (1) large sample size was used and applicable to diverse concepts of discharge failures; (2) the LASSO regression model improved the accuracy of identifying independent risks; (3) relatively simple and easy scoring calculations to predict patients with discharge failures and (4) the SHOUT scoring system is different than other tools reported in the literature with more potential for applying to general population.

Our study has its own limitations. First, with a study design using a single-centre, retrospective data analysis, limited and potential incorrect information and potential patient population selection bias cannot be avoided. In this study, not all patients had EMR data after 1 year of post ED discharge, which might potentially affect the accuracy of SHOUT scores. In addition, we were unable to capture patient follow-up information if follow-up occurred outside of the study hospital system. Second, we were not able to include all potential variables that may predict study outcomes. However, ED providers are busy during clinical shifts with limited time to collect pertinent information. We intended to include convenient variables that can be common and easily identified within a short period to make it feasible for any ED. Third, though SHOUT scores can identify patients with potential risk of discharge failure, based on the AUC results, these models have good but not excellent discrimination. Using our recommended cut-off scores yielded fair sensitivities and specificities but not excellent ones. Considering such outcomes are multifactorial with the diversity of patient populations, it is challenging to derive scoring systems with both higher sensitivity/specificity and excellent discrimination. Such scoring systems might only be used as initial screening tools, and further multicentre external validation is warranted.

In summary, SHOUT might be used as initial screening to differentiate patients with different discharge failure models. It can be used to identify patients with broad and restricted discharge failure potentials. However, its use might be limited only in publicly funded or not-for-profit hospitals similar as the study hospital.