Odds of return: a prospective study using provider assessment to predict short-term patient return visits to the emergency department.

OBJECTIVE: Previous studies have assessed patient-level characteristics associated with emergency department (ED) return visits, but none have used provider assessment. We prospectively investigate whether clinical providers could accurately predict ED return visits.
METHODS: Prospective cohort study.
SETTING: Single academically affiliated urban county hospital. PARTICIPANTS: Discharged ED patients over a 14-month period with a provider assessment of the likelihood of patient return within 7 days of ED discharge. MAIN OUTCOME MEASURES: The primary outcome of interest was a return visit to the ED within 7 days. Additional outcome measures included a return visit within 72 hours and a return visit resulting in admission. We also measured the accuracy of provider gestalt, and provide measures of sensitivity, specificity, predictive values, and likelihood ratios.
RESULTS: Of the 11 922 ED discharges included in this study, providers expected 2116 (17.7%) to result in a return visit within 7 days. Providers were much more likely to perceive a return visit if the patient left against medical advice (OR: 5.97, 95% CI: 4.67 to 7.62), or was homeless (OR: 5.69, 95% CI: 5.14 to 6.29). Patients who actually returned were also more likely to be homeless, English speaking and to have left the ED against medical advice on the initial encounter. The strongest predictor of a return visit at both 72 hours and 7 days in multivariable modelling was provider assessment (OR: 3.77, 95% CI: 3.25 to 4.37; OR: 3.72, 95% CI: 3.29 to 4.21, respectively). Overall sensitivity and specificity of provider gestalt as a measure of patient return within 7 days were 47% and 87%, respectively. The positive and negative likelihood ratios were 3.51 and 0.61, respectively.
CONCLUSIONS: Clinician assessment was the strongest predictor of a return visit in this dataset. Clinician assessment may be used as a way to screen patients during the index visit and enrol them in efforts to decrease return visits. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This is the first study to report on the predictive quality of medical provider gestalt in determining the likelihood of a patient’s return to the emergency department (ED) after an initial index visit.

Among multiple demographic, social and clinical variables, the strongest predictor of the overall odds of return to the ED within 7 days of an initial visit was provider assessment.

Generalisability may be challenging given the sampling frame and non-random nature of the study.

Introduction

Emergency department (ED) return visits following discharge represent a commonly used quality metric, as they are presumed to result from mistaken diagnoses, lack of complete ED care, missed opportunities for provision of additional services or social needs, or lack of appropriate follow-up. In order to design effective interventions to mitigate the resource utilisation and potential harm associated with multiple ED visits, healthcare systems must first improve their ability to predict unplanned return visits prior to the initial discharge of patients.

Numerous studies have sought to identify predictors both of revisits to the ED1–3 and of revisits resulting in hospital admission.4–6 While several variables, including demographics such as older age, social factors such as homelessness, high-risk dispositions such as leaving against medical advice (AMA), and certain chronic medical conditions, have all been found to confer some risk of return after an initial ED visit, no tools currently exist to predict this outcome with a high level of accuracy.1 2 Similarly, no prior studies have investigated the use of provider assessment (ie, gestalt) to determine a patient’s risk of return.

Provider assessment has previously been used to evaluate the medical risks associated with a number of ED patient presentations, with variable results. Prior literature has weighed the degree to which providers can accurately predict patients at risk of suicide,7 opioid use or misuse,8 and high-risk medical events such as acute coronary syndrome9 and pulmonary embolism.10 The Wells’ criteria for risk of pulmonary embolism, currently in broad use in clinical emergency medicine, demonstrate a successful incorporation of provider gestalt.10

The aim of this study was to investigate how accurately a binary gestalt opinion from the treating clinical provider could predict an ED return visit, either alone or in concert with other objective patient characteristics. The more accurately this category of patients can be identified in advance, the more effective potential targeted interventions can be.

Materials and methods

We conducted a prospective cohort study of discharged patients from a single, urban, academically affiliated county hospital in California over a 14-month period from January 2018 through February 2019. Over the study period, clinical providers responsible for the care and disposition of each patient had the opportunity to predict whether a patient discharged from the ED was ‘more likely than average to return’ to the ED within 7 days of the index visit. Clinical providers were defined as attending level physicians, residents (from various specialties) and advanced practice providers.

‘More likely than average to return’ was defined as whether the provider perceived an increased risk of return to the ED in the proceeding 7 days after the index visit and was categorised as a simple ‘yes’ or ‘no’. ‘More likely than average’ was based on provider gestalt and defined in comparison with the historical baseline return rate of approximately 13% (based on internal quality metrics). Providers were exposed to an extensive education campaign, including meetings, posters and emails, explaining the study design and that the baseline average likelihood of 7-day ED return was 13% over the previous 12 months. Providers received no additional training on the topic other than how to fill in the answer via the electronic health record (EHR).

A screening prompt (‘RTED 7-day risk >13%?’) was embedded in the discharge tab of the patient’s ED EHR. Due to limitations of the EHR, we could not provide additional text. The provider’s answer was directly entered into the electronic medical system during the discharge via a ‘yes–no’ radio button following the above prompt. Answering this specific question on discharge was voluntary, did not preclude a patient’s final disposition and did not become part of the medical record.

All patients discharged from the ED with a recorded provider assessment of return likelihood, including those who left AMA or those absent on eventual discharge, were included in the study.

Patients were excluded from analysis if, during the study period, the provider did not assess the patient’s risk of return. Index visits resulting in a death in the ED were also excluded, as were visits resulting in a hospital admission, transfer to another facility (including to another hospital for admission, jail or prison, to urgent care or psychiatric emergency services), or duplicate encounters, and visits in which the patient presented but left without being triaged or seen by a provider.

Patient and public involvement

Patients or the public were not involved in the design, or conduct, or reporting, or dissemination plans of our research.

Outcome measures

The primary outcome measure was a return visit to the ED within 7 days after the initial ED discharge. Secondary measures included whether the return visit resulted in an unscheduled visit within 72 hours and hospital admission. We report both the 72-hour and 7-day time frames given the lack of consensus from prior studies on adverse events after ED discharge, and the lack of an agreed-upon quality metric.11–15 The measurement of return visits was limited to the single department under study, and as such does not account for the risk of return to any other ED.

Predictive characteristics

We assessed visit-level demographic information including age, sex, race/ethnicity (white, black/African American, Hispanic, Asian, American Indian/Alaskan Native, Native Hawaiian/Pacific Islander and other), insurance status (private, self-pay/uninsured, Medicaid, Medicare and other), language preference, homeless (yes or no) and access to a personal phone (yes or no). We also collected information on prior visits, including the total number of prior visits and the number of visits within 180 days of the index visit.

We created a dichotomous variable based on mode of arrival and whether or not patients presented by ambulance. Length of stay in the ED was calculated and presented in minutes and later made a categorical variable based on median length of stay. Discharge disposition was also included with a dichotomous variable created to differentiate those who left the ED AMA or were absent on discharge from those who were discharged. We also collected the primary discharge diagnosis and aggregated diagnoses based on a previously described organ system classification.16

Statistical analysis

We present overall numbers and percentages of all ED visits, as well as those meeting inclusion criteria. Initial comparisons of patient characteristics among those encounters perceived by providers to be more likely to result in a return with those perceived to be less likely were made using the Mantel-Haenszel summary Χ2 with univariable ORs presented to measure the strength of the relationship.

Univariable modelling was performed on patient characteristics and the presence or absence of an actual return visit within 72 hours and 7 days. A multivariable model was constructed with predictor variables chosen from univariable analysis if the p value was <0.10 and via purposeful variable selection, based on historical significance as confounders, including notable clinical, socioeconomic predictors and provider assessment.17 Other variables, while significant in univariable modelling, were eventually excluded owing to lack of good fit. In order to assess each model, we also calculated the area under the curve to assess for discrimination, Hosmer-Lemeshow statistics for goodness of fit and variance inflation factor statistics to assess for multicollinearity. Crude (unadjusted) and adjusted ORs and 95% CIs were used to assess the associations of individual variables with risk of both 72-hour and 7-day return. We also compared individual patient visit characteristics and provider-assessed risk of return with return visits resulting in an unscheduled hospital admission. Given the presupposition that gestalt is positively correlated with experience, we also analysed the relationship between provider assessment and risk of return at 7 days, stratified by provider type.

In multivariable modelling, the unit of analysis was at the patient visit level. However, owing to the large number of patients who presented multiple times throughout the study, we controlled at the individual patient level as well.

Additional measures of the accuracy of provider gestalt, including sensitivity, specificity, predictive values and likelihood ratios, were also calculated. Data analyses were performed with Stata (StataCorp, College Station, Texas, USA).

Results

Over the 14-month study period, there were 95 447 unique visits to the ED (table 1). Patients were majority male (N=57 411, 60.2%), English speaking (N=70 514, 77.1%) and had visited the ED at least once prior (N=58 396, 61.2%).

Table 1

Patient demographic characteristics among total unique presentations and those with return to emergency department (ED) risk assessed (N=11 922) over the 14-month study period

Characteristic	All patient presentations	Risk of return to ED assessed
	N=95 447 (%)	N=11 922 (%)
Median age (IQR)	42 (29–57)	40 (28–55)
Female sex	38 035 (39.8)	4858 (40.75)
Preferred language
English	70 514 (77.1)	8954 (75.1)
Spanish	16 419 (17.9)	2309 (19.4)
Other	4584 (5.0)	658 (5.5)
Working/listed phone	85 157 (89.2)	8610 (72.2)
Number of visits in 180 days preceding the index visit
0	37 051 (38.8)	5322 (44.6)
1	27 993 (29.3)	3150 (26.4)
2	10 410 (10.9)	1200 (10.1)
3	5289 (5.5)	628 (5.3)
4+	14 704 (15.4)	1622 (13.6)
Number of historical visits
0	37 051 (38.8)	5322 (44.6)
1	11 335 (11.9)	1092 (9.2)
2	5648 (5.9)	625 (5.2)
3	4519 (4.7)	510 (4.3)
4+	36 894 (38.7)	4373 (36.7)
Median length of stay in ED minutes (IQR)	259 (148–416)	249 (160–373)

Of the 71 342 patients discharged, left AMA and absent on eventual discharge, 10 190 (14.3%) returned to the ED within 7 days. A total of 11 922 (16.7%) patient encounters had a provider-assessed risk of return. These 11 922 patient encounters included a total of 10 113 unique patients seen.

Among 11 922 patient encounters, the majority were male (N=7064, 59.3%), non-white (N=9222, 77.4%) and English speaking (N=8954, 75.1%). For 45% (N=5322) of patients, this was their first visit to the study ED. Additionally, notable proportions were without a working phone (N=3312, 27.8%), homeless (N=2793, 23.4%), Hispanic (N=4258, 35.7%) and had Medicaid insurance (N=6960, 58.4%, table 2).

Table 2

Patient characteristics among unique patient encounters with return to emergency department (ED) risk assessed (N=11 922) over the 14-month study period

Characteristic	N=11 922
Race/ethnicity
White	2700 (22.7)
Black	2903 (24.4)
Hispanic	4258 (35.7)
Asian	1307 (11.0)
Native Hawaiian/Pacific Islander	165 (1.4)
American Indian/Alaskan Native	88 (0.7)
Other	426 (3.6)
Unknown	75 (0.6)
Insurance
Private	889 (7.5)
Medicare	1652 (13.9)
Medicaid	6960 (58.4)
Other public subsidised	357 (3.0)
None	1486 (12.5)
Other	577 (4.8)
Homeless	2793 (23.4)
Arrived by ambulance	2641 (22.2)
Discharge diagnosis
Trauma	2341 (19.6)
Abdominal pain	1359 (11.4)
Dyspnoea, cough, respiratory tract infection	1191 (10.0)
Atraumatic back and extremity pain	1091 (9.2)
Skin and soft tissue infection, dermatological, allergic	1024 (8.6)
Psychiatric, toxicological	923 (7.7)
Genitourinary, gynaecological, obstetric	854 (7.2)
Cardiac, chest pain	798 (6.7)
Head, neck	776 (6.5)
Other	1565 (13.1)
Discharged from ED	11 652 (97.7)

Overall, providers predicted that 2116 (17.7% of 11 922) discharged encounters were more likely than average to result in return within 7 days (table 3). Providers were much more likely to perceive a risk of return if the patient left AMA (OR: 5.97, 95% CI: 4.67 to 7.62), was homeless (OR: 5.69, 95% CI: 5.14 to 6.29), or had a primary psychiatric or substance use diagnosis (OR: 3.04, 95% CI: 2.64 to 3.51). Providers’ assessment of above-average risk of return was significantly associated with actual return to the ED within both 72 hours (OR: 6.05, 95% CI: 5.29 to 6.93) and 7 days (OR: 5.86, 95% CI: 5.23 to 6.56).

Table 3

Demographic and differences in physician assessment of return risk regarding unique individual patient encounter

Characteristic	No	Yes	OR (95% CI)
	N=9806 (82.3%)	N=2116 (17.7%)
Male sex	5563 (56.7)	1501 (70.9)	1.86 (1.68 to 2.06)
White race/ethnicity	2078 (21.2)	622 (29.4)	1.55 (1.39 to 1.72)
English language	7095 (72.3)	1859 (87.9)	2.76 (2.41 to 3.17)
Homeless	1658 (16.9)	1135 (53.6)	5.69 (5.14 to 6.29)
No working phone	2354 (24.0)	958 (45.3)	2.62 (2.38 to 2.89)
Insured	1275 (13.0)	211 (10.0)	1.35 (1.16 to 1.57)
Arrived by ambulance	2045 (20.9)	596 (28.2)	1.49 (1.34 to 1.66)
Primary diagnosis psychiatric or substance use	582 (5.9)	341 (16.1)	3.04 (2.64 to 3.51)
Left against medical advice*	122 (1.2)	148 (7.0)	5.97 (4.67 to 7.62)
Return visit at 72 hours	490 (5.0)	511 (24.1)	6.05 (5.29 to 6.93)
Return visit at 7 days	853 (8.7)	758 (35.8)	5.86 (5.23 to 6.56)

*Includes patients absent on eventual discharge

In sum, 1011 (8.4%) and 1611 (13.1%) patients with a recorded risk assessment returned to the ED within 72 hours and within 7 days of their index visit, respectively. The 7-day return rate among our sample was slightly less than the overall 7-day rate of return (14.3%) for patients seen and discharged over the same time period. The risks of return within both 72 hours and within 7 days were assessed at the univariable level (table 4). Patients who returned within 7 days were over twice as likely to have left AMA or prior to discharge on the preceding encounter, be homeless, and be primarily English speaking. The strongest predictor of a return visit both within 72 hours and within 7 days was provider assessment (OR: 6.05, 95% CI: 5.29 to 6.93; OR: 5.86, 95% CI: 5.23 to 6.56, respectively).

Table 4

Crude ORs and 95% CIs of return at 72 hours and 7 days per patient encounter

Characteristic	72 hours		7 days
	Unadjusted OR	95% CI	Unadjusted OR	95% CI
Provider-assessed risk	6.05	5.29 to 6.93	5.86	5.23 to 6.56
Homeless	4.27	3.74 to 4.87	4.04	3.62 to 4.51
English	2.56	2.11 to 3.12	2.43	2.09 to 2.83
≥1 visit last 180 days	2.63	2.27 to 3.06	2.76	2.45 to 3.11
Left AMA or prior to DC	2.36	1.71 to 3.26	2.21	1.67 to 2.91
Lack of phone	2.35	2.06 to 2.68	2.25	2.02 to 2.51
Psychiatric or substance use	2.25	1.86 to 2.72	2.27	1.93 to 2.66
Age ≥40	1.97	1.72 to 2.26	1.96	1.75 to 2.18
Male	1.75	1.52 to 2.02	1.70	1.52 to 1.90
Insurance	1.46	1.17 to 1.82	1.58	1.31 to 1.89
White	1.37	1.18 to 1.58	1.37	1.22 to 1.54
Initial LOS >249 min	1.20	1.05 to 1.36	1.30	1.17 to 1.45

AMA, against medical advice; DC, Discharge; LOS, length of stay.

While attendings (OR: 6.16, 95% CI: 5.11 to 7.44) provided a more accurate risk assessment than other providers, there was no significant statistical difference between categories of provider with respect to their ability to predict return at 7 days (online supplemental table 1).

The association between characteristics of the index visit and a return visit within 72 hours and within 7 days was also assessed through conditional logistic regression (table 5). Provider-assessed risk was the strongest predictor of a return visit with an adjusted OR of 3.78 (95% CI: 3.34 to 4.28). The odds of return among the homeless and those with at least one other visit in the last 180 days were twice as great as for domiciled patients, and for those patients who had no visits within the past 180 days (OR: 2.01, 95% CI: 1.76 to 2.30; OR: 2.10, 95% CI: 1.85 to 2.38, respectively). The model’s variable coefficients changed little when controlling for individual patients.

Table 5

Adjusted ORs and 95% CIs of return at 72 hours and 7 days from multivariable conditional regression modelling

Characteristic	72 hours*		7 days†
	Adjusted OR	95% CI	Adjusted OR	95% CI
Provider-assessed risk	3.77	3.25 to 4.37	3.72	3.29 to 4.21
Homeless	2.06	1.74 to 2.42	2	1.74 to 2.28
≥1 visit last 180 days	1.91	1.63 to 2.23	2.1	1.85 to 2.38
English speaker	1.43	1.17 to 1.76	1.42	1.21 to 1.68
Psychiatric or substance use	1.25	1.01 to 1.53	1.29	1.08 to 1.54
Lack of phone	1.21	1.03 to 1.40	1.2	1.06 to 1.37
Male	1.14	0.98 to 1.33	1.15	1.01 to 1.30

*Number of observations: 11 922; ROC: 0.7543.

†Number of observations: 11 922; ROC: 0.7490.

ROC, receiver operating characteristic.

Of the 1611 visits resulting in eventual return to the ED within 7 days, 182 (11.3%) were admitted to an inpatient service on the subsequent visit. The overall return visit rate resulting in hospital admission (among all patients included in the study) was 1.5% (182 out of 11 922). Characteristics including sex, race, insurance status or presentation by ambulance were similarly represented within revisits resulting in admission and revisits not resulting in admission.

The strongest predictors of hospital admission upon return were language and homelessness. English speakers were more likely to be admitted than non-English speakers (41 of 182, 22.5% vs 169 of 1429, 11.8%, p<0.0001); and homeless patients were less likely to be admitted than domiciled patients (62 of 182, 34.1% vs 732 of 1429, 51.2%, p<0.0001). Moreover, provider gestalt of the likelihood of revisit was not predictive of the revisit resulting in hospital admission (81 of 182, 44.5% vs 677 of 1429, 47.4%, p=0.465).

Sensitivity and specificity of provider gestalt as a measure of patient return within 7 days were 47% and 87%, respectively (online supplemental table 2). The positive and negative predictive values were 36% and 91%, and overall accuracy was 82%. The positive and negative likelihood ratios were 3.51 and 0.61, respectively.

Discussion

This study is the first to report on the predictive quality of medical provider gestalt in determining the likelihood of a patient’s return to the ED after an initial index visit. Our results confirmed the hypothesised predictive ability of providers, and in fact showed it to be the single biggest predictor of the primary outcome. This study also reconfirmed many previously known demographic and clinical factors associated with ED revisits. A new incorporation of clinician gestalt as part of a patient’s overall risk assessment for revisiting the ED may therefore be useful in addressing the growing problem of ED resource utilisation and could be used to direct resources toward mitigating patient return.

Patient revisits to the ED represent a significant strain on health systems in measurable ways.18 Revisits are associated with overcrowding, which has been shown to result in treatment delays and excessive mortality.19 20 Emergency care is often more expensive than outpatient medicine, and revisits may therefore be an important contributor to overall health costs as well.21

Potential ED overuse is not a small problem. Previous data suggest that upwards of 40% of patients seek care more than once in an ED over a 2-year period.22 Reducing inappropriate ED revisits could reduce US healthcare spending by as much as $32 billion each year.23–25 Although not the primary focus of the study, our data did reveal that over 13% of index visits resulted in a return visit within 7 days, which provides a sense of the scale of opportunity for improvement.

The reduction of ED revisits has long been considered a measure of a successful discharge and a strategy for increasing care quality.26 Previous data suggest that multiple revisits may be avoidable, but developing effective strategies to reduce discharge failure requires better predictability of the patient population susceptible to it.

Past literature has elucidated many of the demographic and clinical variables associated with high rates of return. This study reconfirms some of these characteristics, in that homelessness, substance use, psychiatric illness and prior visits all confer substantially increased odds of ED revisit.27 Additional studies have attempted to combine these variables in predictive modelling.28 29 While these previous models are useful, they lack the power of provider insight. To our knowledge, this is the first study to assess whether providers could holistically predict a patient’s risk of return. In comparison with other patient variables, provider gestalt was the single greatest predictor of patient return after an index visit to the ED.

Providers are often tasked with risk stratifying their patients, both formally and informally, but recent data have called into question the value of clinician gestalt and the problem of both provider hubris and either implicit or explicit bias.9 30 In the case of this study, provider assessment may be a powerful tool for intervention with little harmful side effect beyond social resource utilisation. Future studies may seek to include not only provider-based assessments, but patient-based assessments as well. Providers rarely ask patients specifically if their social needs are being met on discharge. A more complete assessment of a patient’s risk of ED return may ultimately incorporate not only demographic and clinical data, but also a provider’s gestalt and a patient’s own opinion of her own needs and capabilities.

Limitations

This study has several limitations. Our results are based on a convenience sample limited to 16.7% of all possible patient encounters. Clinical providers were not obligated to answer the question of whether they thought patients would re-present for emergency care, and thus may have actively decided not to answer in more difficult-to-assess cases, which risks a systematic bias and likely skews the estimates away from the null. The study validity is strengthened by the large and varied number of providers who participated and its embeddedness in the EHR. Additionally, this study was performed at a single hospital, with a specific patient population and significant rate of return visits to the ED. Results may not be generalisable to all emergency care settings. Similarly, this study lacks data on visit history at other hospitals within the follow-up period. Thus, discharges that resulted in ED presentations or admissions at other health facilities are not accounted for. This has been seen as a major limitation of prior studies. As a result, the conclusions most likely grossly underestimate the overall rate of ED revisits. Lastly, we were limited by a lack of clinical data in this study, including vital signs, clinical laboratory data and imaging results. Further studies that incorporate these variables may be warranted. Additionally, in this study, we also provide data on risk of ED revisits resulting in hospital admissions, though the question posed to providers did not solicit an estimation of that specific outcome. Further studies assessing clinician assessment of return visits resulting specifically in hospital admissions may be warranted.

Conclusions

Of all measured predictors of patient revisits to the ED that were included in this study, physician gestalt was the strongest. As in previous studies, homelessness and multiple prior visits remained strong predictors. Further programmes that include provider intuition could be used to support interventions aimed at strengthening the discharge process.