Patients leaving without being seen from the emergency department: A prediction model using machine learning on a nationwide database.

OBJECTIVE: The objective of this study was to develop a US-representative prediction model identifying factors with a greater likelihood of patients leaving without being seen.
METHODS: We conducted a retrospective cohort analysis using a 2016 nationwide emergency department (ED) sample. Patient factors considered for analysis were the following: age, sex, acuity, chronic diseases, weekend visit, quarter of presentation, median household income quartile for patient's zip code, primary/secondary insurance, total charges for the visit, and urban/rural household. Hospital factors considered were urban/rural location, trauma center/teaching hospital, and annual ED volume. Multivariable logistic regression was used to find significant predictors and their interactions. A random forest algorithm was used to determine the order of importance of factors.
RESULTS: A total of 32,680,232 hospital-based ED visits with 466,047 incidences of leaving without being seen were included. The cohort comprised 55.5% females, with a median (IQR) age of 37 (21-58) years. Positively associating factors were male sex (odds ratio [OR], 1.22; 99% confidence interval [CI], 1.17-1.26), lower acuity (P < 0.001), and annual ED visits ≥60,000 (OR, 1.44; 99% CI, 1.21-1.7) versus <20,000. Negatively associating factors were primary insurance being Medicare/Tricare or private insurance (P < 0.001); weekend presentations (OR, 0.87; 99% CI, 0.85-0.89); age >64 or <18 years (P < 0.001); and higher median household income for patient's zip code second (OR, 0.86; 99% CI, 0.77-0.97), third (OR, 0.8; 99% CI, 0.7-0.91), and fourth (OR, 0.7; 99% CI, 0.6-0.8) quartiles versus the first quartile. Significant interactions existed between age, acuity, primary insurance, and chronic conditions. Primary insurance was the most predictive.
CONCLUSION: Our derivation model reiterated several modifiable and non-modifiable risk factors for leaving without being seen established previously while rejecting the importance of others.

INTRODUCTION

Background

Leaving without being seen (LWBS) is a problem that almost all emergency departments (EDs), irrespective of size or location, face every day. LWBS is considered to be an indirect marker for 2 of 6 domains of healthcare quality defined by the US Institute of Medicine: timeliness and efficiency.

Studies characterizing the incidence of LWBS, although substantial, have been conducted predominantly in coastal urban teaching hospitals , , or in a particular US state. Subsequently, they have generalized findings for the whole country without any testing in place for external validity. Other studies have used data obtained from the US National Hospital Ambulatory Medical Care Survey (NHAMCS) to support their calculations of national prevalence of LWBS. , NHAMCS is based on visits to a reporting hospital rather than being based on populations, so use of the NHAMCS database for the purposes of prevalence determination introduces a systemic bias and is therefore fundamentally flawed. Also, several factors cited in those studies (eg, age, chronic conditions, location, primary insurance) are intuitively interrelated. However, efforts to study such interactions have been sparse. No study has focused on predictive analytics or used supervised machine‐learning tools to rank predictors.

Importance

LWBS is, by consensus, considered to be a lack of ability of an individual to access the healthcare system. It is also a liability risk because these patients usually have limited options of seeking care elsewhere. Payment and quality metrics often benchmark LWBS prevalence, making it important for ED administrators. The “bottom line” of the hospital is affected directly in the way of lost revenues from opportunities missed in delivering care and indirectly in the form of reimbursement penalties from a decrease in satisfaction scores. The literature on LWBS reveals that prolonged waiting times and duration of stay are by far the major reasons why patients leave after registration. This is a function of a multitude of rapidly shifting variables that need a multidisciplinary approach for control and optimization. Various interventions have been identified and demonstrated to reduce waiting times and throughput, but they are very intensive upon resources if applied broadly. , , Prior studies have demonstrated that specific attributes, such as living in a city, , being uninsured, being covered by Medicaid, , , or being an ethnic minority, , predispose to a higher risk of LWBS. We believe that appropriate “hot‐spotting” of efforts toward such types of at‐risk patients in at‐risk hospital EDs will economize resources spent toward interventions.

The Bottom Line

Patients who leave without being seen are a liability and loss of income. Based on 32 million US ED visits in 2016, positive predictors for leaving without being seen were male sex, low acuity, and high annual visits. Negative predictors were Medicare or private insurance, weekend visit,age extremes, and higher income.

Goals of this investigation

As a primary goal, we sought to identify intrinsic patient factors and hospital factors that are likely to increase their odds of LWBS. We aimed to fill the deficiencies identified in previous analyses by using a more contemporary and nationally representative data set: the Nationwide Emergency Department Sample (NEDS) developed for the Healthcare Cost and Utilization Project. The NEDS is the largest all‐payer ED database in the United States. It yielded national estimates of 32,680,232 hospital‐based ED visits for 2016. A capture of the characteristics of 466,047 LWBS visits provided an ample opportunity for analyses after stratifying for geography, demographics, and insurance type. Our secondary goal was to rank the most influential factors by the order in which they help predict LWBS rates using a machine‐learning algorithm.

METHODS

Data source

Annual data on LWBS status for ED patients were obtained from the 2016 NEDS developed from billing abstractions for the Healthcare Cost and Utilization Project. The ED sample includes the diagnosis and inpatient procedures coded using only International Classification of Diseases Tenth Revision (Clinical Modification / Procedure Coding System). It is weighted, estimates ∼145 million ED visits from 984 hospitals located in 36 states and the District of Columbia, and approximates a 20% stratified sample of US hospital‐owned EDs. Its size supports robust and parsimonious predictive modeling without compromising the power of the study.

Study design

This retrospective cross‐sectional study analyzed the records of 2016 US hospital‐based ED visits by patients with a valid value for the presence or absence of LWBS status. The Ethics Review Board of the Catholic Health Initiatives (Englewood, CO) determined the study to be not human research. Visits were characterized by the number of chronic conditions and patient acuity based on International Classification of Diseases Tenth Revision codes using the Chronic Condition Indicator and Clinical Classification Software Refined tools for International Classification of Diseases Tenth Revision CM diagnoses provided by the Healthcare Cost and Utilization Project. New binary categorical variables were created to reflect high acuity or low acuity based on whether the number of acute diagnoses was ≥2 or ≤2 and to represent the presence or absence of diagnosis codes for chronic conditions.

Statistical analyses

All analyses accounted for the sampling design of the NEDS. National estimates of ED visits associated with LWBS status and descriptive statistics regarding visit characteristics were calculated. Survey‐weighted multivariable logistic regression was used to examine the relationship between the patient characteristics and hospital characteristics associated with LWBS status. Bivariate methods, including the χ2 test and simple logistic regression, were used to identify the patient characteristics and hospital characteristics associated with LWBS status. Forward stepwise regression was employed to generate preliminary regression models and test for significant interactions. Multiple logistic regressions were carried out after exploratory analyses, and the resulting odds ratios (ORs) and test‐of‐fit were examined. All 2‐way and 3‐way interactions except for those between age and chronic conditions, age and type of insurance, and between acuity and type of insurance were found not to influence LWBS prediction significantly. A “main effects” model was created from only the variables from the sample data set, which themselves or by interactions with other variables had significant effects at the end of the elimination process, and interactions were not included. A “main effect with interactions included” model was created subsequently from the variables as well the interactions found to be significant from the initial exploratory analysis to address multicollinearity.

Finally, we used an advanced supervised machine‐learning algorithm called “random forest classifier” on a sample data set of 400,000 observations to rank the final set of independent variables in order of importance for predicting LWBS. To predict an observation, this algorithm first assigns the observation to a single “leaf” in a “decision tree.” Then, it uses that leaf to make a prediction based on the tree that contains the leaf and, finally, averages the predictions over several trees as shown in the Figure 1. A random selection of 3 explanatory variables was selected to test each possible split for each node in each tree within the “forest.” All regression estimates, ORs, and 99% confidence intervals (CIs) were calculated using survey procedures in SAS 9.4 (SAS Institute, Cary, NC). P < 0.01 was considered significant to avoid a type‐1 error from using a large data set.

FIGURE 1

Flow diagram for the random forest algorithm from a hundred decision trees. OOB, out of bag

RESULTS

Characteristics of study subjects

The 2016 national prevalence of LWBS was found to be 1.27%. A total of 32,680,232 hospital‐based ED visits with 466,047 LWBS incidences were included in the present study. The cohort consisted of 55.5% females, with a median (interquartile range) age of 37 (21–58) years. Male patients seemed to have higher odds of LWBS than females (OR, 1.15; 99% CI, 1.12–1.18). The pediatric age group as well as the Medicare‐qualifying age group >64 years was associated with lower odds of LWBS then the middle‐aged group (OR, 0.28; 99% CI, 0.24–0.32; and OR, 0.46; 99% CI, 0.42–0.50), respectively. Poverty was an important factor, with subjects having Medicaid or no insurance and those living in zip codes with median income at the lower end of the spectrum having higher odds of LWBS (Table 1). Visits from patients with chronic conditions (OR, 0.99; 99% CI, 0.87–1.11) or those presenting during weekends (OR, 0.84; 99% CI, 0.83–0.86) had lower odds of LWBS. Hospital EDs that were in urban areas and that had visits >60,000 saw more LWBS, whereas the teaching status or trauma status of the hospital did not matter according to initial crude ORs.

TABLE 1

Characteristics of patients seen at emergency departments 2016: bivariate model (weighted estimates)

Variable	LWBS = No	LWBS = Yes	Odds ratio (99% CI)
Male sex, no. (%)	53,309,851 (44.1)	723,630 (47.6)	1.15 (1.12–1.18)
Age group, y, no. (%)
0–17	24,574,342 (20.3)	115,770 (7.6)	0.28 (0.24–0.32)
18–35	32,923,423 (27.2)	548,734 (36.1)	Ref
36–64	41,236,349 (34.1)	687,839 (45.2)	1.01 (0.96–1.05)
64+	22,205,713 (18.4)	169,476 (11.1)	0.46 (0.42–0.50)
Primary insurance, no. (%)
Medicaid/self pay	52,994,630 (43.8)	868,232 (57.1)	Ref
Medicare/Tricare/Workman's Compensation	32,931,742 (27.2)	338,967 (22.3)	0.55 (0.49–0.60)
Private insurance	35,013,456 (29.0)	314,621 (20.7)	0.55 (0.54–0.56)
Median household income of home zip code, $, no. (%)
<$43,000	35,763,933 (41.6)	578,676 (47.9)	Ref
$43,000–$53,999	23,938,293 (27.9)	324,681 (26.9)	0.81 (0.80–0.82)
$54,000–$70,999	16,169,375 (18.8)	198,459 (16.4)	0.73 (0.72–0.74)
>$71,000	10,054,772 (11.7)	105,383 (8.7)	0.59 (0.58–0.62)
Patient lives in metro area	111,373,453 (92.1)	1,421,205 (93.4)	1.21 (0.99–1.46)
Chronic conditions, any	64,980,151 (53.7)	809,862 (53.2)	0.99 (0.87–1.11)
Acuity score ≤ 2	63,942,589 (52.9)	910,803 (59.8)	1.33 (1.22–1.44)
Initially seen on weekend	33,754,721 (27.9)	376,241 (24.7)	0.84 (0.83–0.86)
Hospital level characteristics
Annual ED volume, N, no. (%)
0–19,999	15,200,277 (12.6)	156,421 (10.3)	Ref
20,000–39,999	24,972,065 (20.6)	279,937 (18.4)	1.09 (0.96–1.29)
40,000–59,999	25,049,496 (20.7)	275,599 (18.1)	1.07 (0.86–1.28)
≥60,000	55,717,991 (46.1)	809,864 (53.2)	1.41 (1.39–1.44)
Hospital type urban, no. (%)	118,039,378 (77.32)	1,497,971 (78.25)	1.54 (1.16–1.93)
Hospital trauma center	38,222,378 (31.6)	506,589 (33.3)	1.08 (0.79–1.37)
Teaching hospital	67,355,234 (55.7)	905,876 (59.5)	1.18 (0.94–1.41)

The multivariable logistic regression model is from all the factors.

CI, confidence interval; ED, emergency department; LWBS, leaving without being seen; Ref, reference.

Main results

For the main effects model, patients from zip codes with higher median income quartiles had lower odds of LWBS than those from the lowest quartile, and the differences became more significant progressively. Patients in the age group of 36 to 64 years were at higher odds of LWBS than those in the reference age group of 18 to 35 years. However, patients aged <18 years or >65 years had significantly lower odds for LWBS than the reference group. Male sex, lower acuity at presentation, and presentation to very high volume EDs were found to have significantly higher odds of LWBS. Patients with chronic conditions at presentation as well as weekend presentations had significantly lower odds of LWBS than those presenting on business days. Subjects with Medicare, Tricare, or Workman's Compensation (MTW) and private insurance had significantly lower odds of LWBS compared with those with Medicaid or no insurance. The model satisfied the convergence criterion with a concordance of 63.1%.

The model for main effects with interactions included had a slightly better concordance (63.4%) with fewer tied (6.8% vs 6.9%) and discordant (29.8% vs 30%) observations. Once interactions were included, the presence of chronic conditions (maximum likelihood estimate [MLE], −0.05; P = 0.202) by itself was no longer significant in influencing predictions, but its interactions with age were, as shown in Table 2. For all age groups, the presence of chronic conditions negatively influenced LWBS (age <18 years = MLE, −0.47 [P < 0.001]; age 36−64 years = MLE, −0.15 [P < 0.001]; age >64 years = MLE, −0.58 [P < 0.001]). The interaction between being 35 to 64 years of age and having MTW as insurance had a significant positive effect on LWBS (MLE, 0.18; P < 0.001), whereas being >65 years of age and having MTW as insurance had a significant negative effect on LWBS (MLE, −0.31; P < 0.001). Presentation of lower acuity in subjects having MTW as the primary payor also had a positive effect on it (MLE, 0.24; P < 0.001).

TABLE 2

Interactions between predictors that significantly affect LWBS

Parameter	Maximum likelihood estimates	Standard error	t Value degree of freedom = 776	P‐Value > |t|
Age < 18 & chronic conditions	−0.47	0.05	−10.36	<0.01
Age 36–64 & chronic conditions	−0.15	0.02	−7.58	<0.01
Age > 64 & chronic conditions	−0.58	0.04	−14.79	<0.01
Age < 18 & MTW	−0.12	0.13	−0.92	0.36
Age < 18 & private insurance	0.13	0.06	2.28	0.02
Age 36–64 & MTW	0.18	0.03	6.47	<0.01
Age 36–64 & private insurance	−0.01	0.02	−0.29	0.77
Age > 64 & MTW	−0.31	0.06	−5.05	<0.01
Age > 64 & private insurance	0.02	0.08	0.26	0.79
Lower acuity & MTW	0.24	0.03	8.39	<0.01
Lower acuity & private insurance	0.06	0.02	2.48	0.01

These include the presence of chronic conditions and all age groups, MTW insurance and age groups 36–63 and >64, types of insurance, and lower acuity at presentation.

LWBS, leaving without being seen; MTW, Medicare/Tricare/Workman's Compensation.

Secondary results

A total of 100 trees were grown for classification of random forests, and 60% of the sample was selected for the bagging process at each step. “Pruning” was not carried out, so the number of observations read was equal to the number used. The misclassification rate of 0.013 meant that with the variables used, the forest was able to predict LWBS correctly ∼98.7% of the time. Table 3, which shows the loss reduction variable importance, ranks the variables as per their importance in predicting LWBS. Thus, the most important variable for prediction was the primary insurance followed, in decreasing order, by sex, presence of chronic conditions, and annual volume of ED visits. The out of bag margin turning negative starting from the fifth variable as shown on the table indicates that the last 4 variables were not helpful for prediction.

TABLE 3

Results from random forest classification showing loss reduction variable importance

	Loss reduction variable importance
Variable	Number of rules	Gini	Out‐of‐bag Gini	Margin	Out‐of‐bag margin
Primary insurance	7316	0.000038	0.00013	0.000077	0.00342
Male sex	7544	0.000013	0.00002	0.000025	0.00115
Chronic conditions at presentation	9389	0.000019	0.00001	0.000037	0.00100
Annual ED volume	15387	0.000032	−0.00001	0.000063	0.00000
Median household income of home zip code	16793	0.000031	−0.00001	0.000063	−0.00020
Low acuity at presentation	3685	0.000016	−0.00002	0.000032	−0.00241
Age	11012	0.000068	−0.00004	0.000135	−0.00038
Weekend presentation	9051	0.000013	−0.00008	0.000026	−0.00235

The out of bag margin being positive down the table until the “annual ED volume” variable indicate that the first 4 variables have the highest predictive utility.

ED, emergency department.

Limitations

The limitations of the present study stemmed primarily from the fact that administrative data were used for analyses. Unlike survey data collected primarily for research purposes, our data set missed out on several variables that could have been important for predictive modeling. For example, race/ethnicity and level of education, which have been shown to be predictors in previous studies, were not available for analyses. As urban and larger volume EDs capture more minority visits, the proportion of influence of these variables on LWBS that could be attributed to the diversity is important. Another resulting limitation was that, although we only considered patients who were not admitted, there was no way to separate those who left before final ED disposition from those who left before being seen by a physician. Therefore, both of these categories had to be analyzed as a single outcome variable. Similarly, we had to derive a dichotomous acuity level indirectly from the billing codes because actual triage levels were not available. Also, we assumed subjects from zip codes with higher median incomes would have higher incomes, which is not always true. Patients living in these zip codes also tend to have higher education opportunities. Thus, the effect observed might be attributed to education rather than residence location. Furthermore, we did not adjust for operational variables in our analysis, such as waiting times, duration of stay, and ED boarding because they were not available. Adjusting for these might have created better models. Finally, the data available did not allow us to analyze the time from a previous visit or to identify high users of EDs, both of which have been shown to be important LWBS predictors.

DISCUSSION

The main findings from our study were that that the patient's type of insurance, sex, age, poverty level, acuity of illness at presentation, presence of comorbidities, and presentation during a weekend influenced the risk for LBWS significantly. The only hospital factor that significantly impacted and helped predict LWBS was very high ED volumes. The acuity and presence of chronic conditions at presentation, age, and type of insurance had significant interactions for predicting LWBS. Including interactions in the model definitively improved prediction. The most important predictive factor for LWBS was the patient's type of insurance.

We scrutinized literature from the United States and overseas, mostly from countries with advanced healthcare systems, similar to that in the United States. Unlike our study, male sex has not been found to be a significant factor in studies carried out in the United States. However, some studies from Europe and the Middle East have reported male sex to be associated consistently with LWBS. , Pham et al analyzed data from NHAMCS and reported higher levels of LWBS at the extremes of age, which was corroborated by our analyses. Contrary to our findings, many studies show the pediatric population as being more predisposed to LWBS. , , Ding et al found that patients lacking insurance, self‐paying, or having Medicaid had higher rates of LWBS, and we placed these 3 categories into a single group for our analysis. Other studies conducted in the United States have also reported a lack of insurance for children and adults to be important factors. An association with poverty (which we found by analyzing subjects from zip codes with lower household incomes) was corroborated by a study by Hsia et al. Some studies have shown an urban location of patients in proximity to the hospital to be a significant factor, but this was refuted by our analysis. Similar to our analysis, other studies have pointed out lower acuity at presentation to have a significant association with LWBS, , , but no scholars have studied relationships with the presence of chronic conditions.

Time of presentation has been associated with LWBS. Some studies have cited summer/fall seasons, whereas others have found overnight or evening shifts , to have higher rates of LWBS. Hobbs et al reported findings similar to ours of higher rates of LWBS on weekends. Although we used the quarter of presentation for data stratification during the exploratory analysis, it was not used as an independent variable. Weiss et al constructed a national ED crowding scale that was positively associated with LWBS; this echoes our findings of very high annual volume EDs having a higher incidence. Unlike other studies, we did not show a relationship of LWBS with an urban location of the hospital. , Being a teaching hospital or trauma center have been shown to be factors favoring LWBS in certain studies. , However, our analysis did not find such associations.

Our study had 4 main strengths. The first was the large sample size and corresponding statistical power. Even after “cleaning” the data set by deleting all records with missing variables and admitted patients, we were left with a resultant data set of 27,061,841 records without and 120,939,829 visits with weights, thereby capturing 347,286 LWBS without and 1,521,820 LWBS with weights. This large sample size allowed for the inclusion of several potential predictors for the number of LWBS patients. In addition, the large sample size enabled more stable coefficient estimates in the regression analysis. Second, an analysis using a nationwide database enables much better external validity than those conducted in restricted geographic margins. Similarly, all subgroups of age (ie, pediatric, young adults, and old adults) were evaluated. The third strength of our study was the analysis of interactions between associated factors and how they shaped predictions. Lastly, we used machine‐learning algorithms to ascertain the most important factors to predict LWBS, which has never been done before.

In conclusion, a patient's insurance type and the annual ED volume of the hospital had the greatest impact on LWBS risk. Furthermore, our model reiterated several modifiable and non‐modifiable risk factors for LWBS established from previous studies while rejecting the importance of others. The main policy implications are to steer interventions such as early triage and early bedding preferentially toward patients having a higher risk for LWBS. The main research implications are to collect data so that operational factors can also be accounted for in the model. Further research is needed to integrate machine learning to analyze real‐time ED data so that LWBS prediction can be improved.

CONFLICTS OF INTEREST

The authors declare no conflicts of interest.