Prognostic value of a modified surprise question designed for use in the emergency department setting.

OBJECTIVE: Few reliable and valid prognostic tools are available to help emergency physicians identify patients who might benefit from early palliative approaches. We sought to determine if responses to a modified version of the surprise question, "Would you be surprised if this patient died in the next 30 days" could predict in-hospital mortality and resource utilization for hospitalized emergency department patients.
METHODS: For this observational study, emergency physicians responded to the modified surprise question with each admission over a five-month study period. Logistic regression analyses were completed and standard test characteristics evaluated.
RESULTS: 6,122 visits were evaluated. Emergency physicians responded negatively to the modified surprise question in 918 (15.1%). Test characteristics for in-hospital mortality were: sensitivity 32%, specificity 85%, positive predictive value 6%, negative predictive value 98%. The risk of intensive care unit use (relative risk [RR], 1.87; 95% confidence interval [CI], 1.45 to 2.40), use of 'comfort measures' orders (RR, 3.43; 95% CI, 2.81 to 4.18), palliative-care consultation (RR, 3.06; 95% CI, 2.62 to 3.56), and in-hospital mortality (RR, 2.18; 95% CI, 1.72 to 2.76) were greater for patients with negative responses.
CONCLUSION: The modified surprise question is a simple trigger for palliative care needs, accurately identifying those at greater risk for in-hospital mortality and resource utilization. With a negative predictive value of 98%, affirmative responses to the modified surprise question provide reassurance that in-hospital death is unlikely.

INTRODUCTION

A growing body of literature suggests that many patients visiting the emergency department (ED) have unmet palliative care (PC) needs, and that identifying these patients early in the course of disease will extend and improve quality of life, reduce hospitalizations and future ED visits [1-4]. As interest in improving PC in the ED grows, development of widely-applicable screening tools for PC needs is a priority. While there has been significant work on the assessment of PC needs in cancer patients visiting the ED [1,5], there are currently a paucity screening tools validated in the general ED population.

Dozens of “triggers” for PC needs assessments have been described in the literature. Such triggers are often based on diagnosis or symptoms, healthcare utilization patterns, psychosocial needs, and/or the clinical impressions of providers [3,6]. We believe an ideal initial screening tool or trigger for the fast-paced ED environment would be easy to administer in a brief period of time; easy to understand and interpret; require minimal training to apply; and would be specific enough that further assessment on patients screening positive would be feasible. A trigger based on clinical impression may be particularly well-suited to ED physicians, who regularly employ intuitive pattern-recognition techniques to make rapid clinical judgments.

The surprise question (SQ), “Would you be surprised if this patient died in the next year?” has been used as a PC screening tool in cancer, dialysis, and general populations to predict one-year mortality with moderate sensitivity and high specificity [5,7-10]. Unfortunately these studies were small with limited generalizability and notably high mortality rates in the populations studied. While the simplicity of this trigger makes it appealing for the ED environment, the one-year time frame is likely to have limited clinical utility and be conceptually challenging for the busy emergency physician. In an attempt to improve the clinical relevance of the SQ, while at the same time maintain its simplicity, here we proposed a modified ED version of the surprise question (ED-MSQ), “Would you be surprised if this patient died in the next 30 days?” for use in the ED. The ED-MSQ is a clinician-based tool that is simple to administer and easy for providers to interpret. If found to be reliable, the ED-MSQ could be applied easily in the ED setting. Therefore, our objective was to determine if the ED-MSQ, as answered by ED physicians at time of admission, could reliably predict increases in hospital resource utilization, in-hospital mortality and other surrogates for unmet PC needs. We hypothesized that in-hospital mortality, intensive care unit (ICU) use, utilization of a ‘comfort measures’ order set and PC consultation would be higher in the ‘No, I would not be surprised’ group as compared to the positive response group (Yes, I would be surprised).

METHODS

Study design

We conducted a retrospective review of prospectively collected data to address our research objectives. We developed a mechanism within our electronic health record (EPIC) to prospectively query emergency providers at the time of admission on their response to the ED-MSQ. At our institution all patients entering through the ED and subsequently admitted to hospital receive emergency provider generated ED transition admitting orders at the time of the admission decision. During this intervention, embedded within the ED transition order set was the ED-MSQ with a hard stop forcing providers to answer. As a result, providers were prospectively queried on nearly 100% of patients admitted through our ED. Patients taken directly to the operating room or cardiac catheterization laboratory bypassed this process. The study period lasted 5 months from August 1, 2013 through December 31, 2013. Emergency physician responses to the modified SQ were used to assign patients to one of two groups: those for whom physicians would be surprised if death occurred within 30 days (Yes, I would be surprised group) and those for whom physicians would not be surprised if death occurred within 30 days (No, I would not be surprised group). The Maine Medical Center (MMC) institutional review board exempted the study and waived the requirement for written informed consent.

Study setting

The study was conducted at MMC, a 637-bed tertiary care teaching hospital located in northern New England. The ED at MMC is a mixed adult and pediatric ED that treats approximately 65,000 patients annually and houses an emergency medicine residency training program.

At the time of this intervention, significant institutional efforts were focused on improving end-of-life care. The ED-MSQ intervention, as described here, was only part of a greater awareness leading to an acceleration of educational efforts targeting providers hospital wide.

Selection of the study sample

Patients were actively enrolled and admitting physicians queried by virtue of admission to the hospital through the ED. We obtained visit-level data through queries of MMC’s electronic medical records system (Epic Systems Corporation, Verona, WI, USA). As ED patient care occurs, the Epic system electronically captures time, date, demographic, and clinical data throughout the hospital stay, yielding an integrated electronic medical record. We analyzed de-identified data for all ED visits occurring during the study period spanning August 1, 2013 through December 31, 2013. Data analyzed for this manuscript followed the ‘go live’ of the ED-MSQ functionality in the electronic record by approximately 6 months.

Data obtained included patient demographic information, triage acuity, presenting complaint, patient disposition from the ED, patient disposition from the hospital, inpatient hospitalization unit, utilization of the institution’s ‘comfort measures’ order set, ordering of a PC consult, and emergency physician responses to the ED-MSQ. We excluded primary psychiatric visits, visits for patients who were not admitted to the hospital, and visits for patients who were less than 18 years of age and those taken directly to the operating theater or cardiac catheterization laboratory from the ED.

Outcome measures

The primary outcome measure of interest in this study was inpatient mortality which we defined as any patient death that occurred prior to discharge from the ED or inpatient hospital setting. Secondary outcomes of interest included markers of hospital resource utilization; placement of orders for ‘comfort measures’, PC consultation, ICU utilization and length of stay. We also investigated the association of responses with various patient demographic characteristics (Table 1).

Table 1.

Characteristics of the study subjects

Characteristics	All subjects (n = 6,122)	No, I would not be surprised (n = 918)	Yes, I would be surprised (n = 5,171)
Age category (yr)
18–29	380 (6.2)	11 (1.2)	367 (7.1)
30–39	408 (6.7)	17 (1.9)	390 (7.5)
40–49	618 (10.1)	27 (2.9)	586 (11.3)
50–59	951 (15.5)	75 (8.2)	873 (16.9)
60–69	1,108 (18.1)	135 (14.7)	963 (18.6)
70–79	1,234 (20.2)	222 (24.2)	1,009 (19.5)
80–89	1,065 (17.4)	298 (32.5)	760 (14.7)
≥ 90	358 (5.8)	133 (14.5)	223 (4.3)
Sex
Male	3,168 (51.7)	487 (53.1)	2,658 (51.4)
Female	2,954 (48.3)	431 (46.9)	2,513 (48.6)
ESI category
ESI-1	130 (2.2)	50 (5.7)	79 (1.6)
ESI-2	4,229 (71.2)	668 (75.6)	3,539 (70.5)
ESI-3	1,526 (25.7)	161 (18.2)	1,355 (27.0)
ESI-4	51 (0.9)	4 (0.5)	47 (0.9)
ESI-5	1 (> 0.01)	1 (0.1)	0 (0)
Race
White	5,852 (95.7)	30 (3.3)	233 (4.5)
Non-White/other	265 (4.3)	888 (96.7)	4,934 (95.5)
Ethnicity
Hispanic	45 (0.7)	8 (0.9)	36 (0.7)
Non-Hispanic	6,036 (99.3)	903 (99.1)	5,102 (99.3)
Payor type
Private	1,495 (24.4)	75 (8.2)	1,412 (27.3)
Public	4,263 (69.6)	824 (89.8)	3,418 (66.1)
Self-pay	364 (5.9)	19 (2.1)	341 (6.6)
Intensive care use
Yes	171 (2.8)	47 (5.2)	124 (2.4)
No	5,951 (97.2)	871 (94.9)	5,047 (97.6)
Comfort measures orderset use
Yes	109 (1.8)	54 (5.9)	55 (1.1)
No	5,980 (98.2)	864 (94.1)	5,116 (98.9)
Palliative care consult order
Yes	270 (4.4)	114 (12.4)	156 (3.0)
No	5,819 (95.6)	804 (87.6)	5,015 (97.0)
In-hospital mortality
Yes	157 (2.6)	107 (11.7)	50 (0.9)
No	5,932 (97.4)	811 (88.3)	5,154 (99.6)

Values are presented as number (%).

ESI, Emergency Severity Index.

Primary data analysis

We received the study data in a Microsoft Excel (Microsoft Corporation, Redmond, WA, USA) spreadsheet and later transferred the data into IBM SPSS Statistics ver. 22 (IBM Corp., Armonk, NY, USA) for analysis. To maximize sample size, we excluded cases with missing data on an analysis-by-analysis basis rather than excluding an entire case. Findings for categorical variables are reported as numbers and percentages with 95% confidence intervals (CIs). Continuous variables are reported as means and CIs or medians and interquartile ranges, as appropriate.

For our primary analysis, we first conducted univariate logistic regression analyses to examine the relationships between patient characteristics (age, race, ethnicity, sex, acuity level, and payor type), clinical resource utilization (ED length of stay, hospital length of stay, presence of a PC consult order, ICU utilization, and use of the institutional ‘comfort measures’ orderset) and emergency physician responses to the ED-MSQ with in-hospital mortality. Reference groups were chosen by temporal order or increasing values. Factors that were significant in univariate logistic regression analyses were then used as covariates in subsequent multivariate logistic regression modeling. Through multivariate modeling, we have estimated odds ratios and corresponding 95% CIs.

In addition to these primary analyses, standard test characteristics, including sensitivity, specificity, positive predictive value, and negative predictive value, were also computed. We also calculated the relative risk (RR) of experiencing each outcome (ICU utilization, PC consultation, ‘comfort measures’ order set use, and in-hospital mortality) based upon the physicians’ ED-MSQ response.

An a priori sample size calculation was not computed; rather, we used the sample available to us approximately six months following implementation of the ED-MSQ. Following standard sample size recommendations, the available sample of data from 6,122 cases is adequate to support the analyses we conducted [11,12].

RESULTS

Characteristics of the study subjects

During the study, there were 7,766 admissions to the hospital originating in the ED. We excluded from analysis 768 admissions for patients <18 years of age, and 876 primary psychiatric admissions, leaving data from 6,122 cases for analysis. Direct admissions to the hospital and patients taken directly to the operating room and cardiac catheterization laboratory were also excluded as they bypassed our admission process. The median age for those included was 66 years (interquartile range, 51 to 79); 3,168 (52%) were male and 5,852 (96%) were white. Emergency physicians responded ‘No, I would not be surprised’ if the patient died in the next 30 days for 918 cases (15%). Differences in age, Emergency Severity Index category, payor type, ICU utilization, ‘comfort measures’ order set use, PC consult ordering, and in-hospital mortality were all noted when comparing the characteristics of patients with negative (No, I would not be surprised) and affirmative (Yes, I would be surprised) physician responses (Table 2). In-hospital mortality was 2.6% during the study period (157 cases), with 107 deaths (68%) in the ‘No, I would not be surprised’ group and 50 (32%) in the ‘Yes, I would be surprised’ group.

Table 2.

Test characteristics for the modified surprise question by provider response for hospital mortality

Group		No. of patients	No, I would not be surprised responses	Sensitivity[a)]	Specificity[b)]	PPV[c)]	NPV[d)]
All patients		6,089	918 (15.1)	31.8 (24.7–39.8)	85.4 (84.4–86.3)	5.45 (4.1–7.1)	97.9 (97.5–98.3)
Age 18–59 years		2,346	130 (5.5)	17.1 (6.61–33.7)	94.63 (93.6–95.5)	4.62 (1.7–9.8)	98.7 (98.1–99.1)
Age ≥ 60 years		3,743	788 (21.1)	36.1 (27.6–45.3)	79.5 (78.1–80.8)	5.6 (4.1–7.4)	97.4 (96.7–97.9)
Female		2,944	431 (14.6)	31.9 (21.2–44.2)	85.8 (84.4–87.0)	5.1 (3.2–7.6)	98.1 (97.5–98.6)
Male		3,145	487 (15.5)	31.8 (22.3–42.6)	85.0 (83.7–86.2)	5.7 (3.9–8.2)	97.7 (97.1–98.3)
ICU	No	5,918	871 (14.7)	33.8 (23.0–46.0)	85.5 (84.6–86.4)	2.8 (1.8–4.1)	99.1 (98.8–99.3)
	Yes	171	47 (27.5)	30.2 (20.8–41.1)	75.3 (64.8–84.0)	55.3 (40.1–69.8)	51.6 (24.5–60.7)

Values are presented as number (%) or % (95% confidence interval).

PPV, positive predictive value; NPV, negative predictive value.

Percent inpatient deaths predicted.

Percent survivals to discharge predicted.

Percent ‘No, I would not be surprised’ responses correctly predicting inpatient death.

Percent “Yes, I would be surprised’ responses correctly predicting survival to discharge.

Characteristics of the modified SQ

The sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) for the ED-MSQ were computed for all subjects as well as for three subgroups based upon age (≥60 vs. 18–59 years old), subject sex, and ICU utilization (Table 2). For the entire cohort, the sensitivity of the ED-MSQ negative response (No, I would not be surprised) was 32% (95% CI, 25% to 40%) and the specificity was 85% (95% CI, 84% to 86%). The PPV was 5.5% (95% CI, 4% to 7%) and the NPV was 97.9% (95% CI, 97.5% to 98.3%).

Predictive value of the modified SQ

We computed the RR of ICU utilization, PC consultation, ‘comfort measures’ order set utilization, and in-hospital mortality for subjects with and without negative ED-MSQ responses (Table 3). Those with negative responses were at greater risk of being treated in the ICU (RR, 1.87; 95% CI, 1.45 to 2.40), of receiving PC consultation (RR, 3.06; 95% CI, 2.62 to 3.56), of ‘comfort measures’ order set use (RR, 3.43; 95% CI, 2.81 to 4.18) and of experiencing in-hospital mortality (RR, 2.18; 95% CI, 1.72 to 2.76).

Table 3.

Predictive value of the modified surprise question

Characteristics	All subjects (n = 6,122)	No, I would not be surprised (n = 918)	Yes, I would be surprised (n = 5,171)	Relative risk (95% CI)
Intensive care use
Yes	171 (2.8)	47 (5.1)	124 (2.4)	1.87 (1.45–2.40)
No	5,918 (97.2)	871 (94.9)	5,047 (97.2)
Comfort measures order set use
Yes	109 (1.8)	54 (5.9)	55 (1.1)	3.43 (2.81–4.18)
No	5,920 (98.2)	864 (94.1)	5,116 (98.9)
Palliative care consult order
Yes	270 (4.4)	114 (12.4)	156 (3.0)	3.06 (2.62–3.56)
No	5,819 (95.6)	804 (87.6)	5,015 (97.0)
In-hospital mortality
Yes	157 (2.6)	50 (5.4)	107 (2.1)	2.18 (1.72–2.76)
No	5,932 (97.4)	868 (94.6)	5,064 (97.9)

Values are presented as number (%).

CI, confidence interval.

In-hospital mortality

In univariate logistic regression analysis, the modified SQ response, patient age, payor type, ED length of stay, hospital length of stay, ICU utilization, ‘comfort measures’ order set utilization, and PC consult orders were significantly associated with inpatient mortality (Table 4). The odds of experiencing in-hospital death were 63% (95% CI, 48% to 74%) less likely for patients where the emergency physician indicated, ‘Yes, I would be surprised’ when compared to those for whom physicians indicated that the patient’s death within 30 days would not surprise them. Patients treated in the ICU died in-hospital 84 times more frequently (95% CI, 57 to 123) than patients who were not treated in the ICU while those with ‘comfort measures’ orders died 86 times (95% CI, 56 to 132 times) more frequently than patients without ‘comfort measures’ orders. In multivariate analysis, only age, hospital length of stay, ICU utilization and ‘comfort measures’ predictors were significantly associated with inpatient mortality (Table 5). The odds of inpatient mortality were 9 times greater (95% CI, 6 to 12) for patients with PC consult orders than for their counterparts without PC consults.

Table 4.

Univariate logistic regression analysis to predict in-hospital mortality

Predictor	OR (95% CI)
Modified ‘surprise question’ response (reference = ‘yes’)	0.367 (0.260–0.517)
Age category (yr, reference: 18–29)
30–39	0.931 (0.130–6.643)
40–49	3.743 (0.833–16.814)
50–59	3.853 (0.893–16.623)
60–69	4.900 (1.162–20.667)
70–79	5.355 (1.281–22.394)
80–89	7.376 (1.774–30.667)
≥ 90	11.183 (2.595–48.200)
Sex (reference: male)	1.195 (0.868–1.644)
Race (reference: White)	0.281 (0.069–1.141)
Ethnicity (reference: non-Hispanic)	0.892 (0.122–6.516)
Payor type (reference: private)
Public	1.806 (1.171–2.784)
Self-pay	0.819 (0.311–2.154)
ED length of stay	0.998 (0.997–0.999)
Hospital length of stay	1.033 (1.017–1.050)
Intensive care use (reference: no)	83.791 (57.285–122.562)
Comfort measures order set use (reference: no)	85.758 (55.715–132.001)
Palliative care consult order (reference: no)	8.476 (5.782–12.424)

OR, odds ratio; CI, confidence interval; ED, emergency department.

Table 5.

Multivariate logistic regression analysis to predict in-hospital mortality

Predictor	OR (95% CI)	P-value
Modified surprise question response (reference: yes)	0.975 (0.595–1.600)	0.921
Age ≥ 60 yr (reference: < 60 yr)	1.848 (1.057–3.232)	0.031[a)]
Payor type (reference: private)
Public	1.062 (0.594–1.901)	0.838
Self-pay	0.621 (0.189–2.047)	0.434
ED length of stay	1.000 (0.998–1.001)	0.447
Hospital length of stay	1.032 (1.007–1.057)	0.011[a)]
Intensive care use (reference: no)	104.837 (64.160–171.303)	< 0.001[a)]
Comfort measures order set use (reference: no)	78.791 (41.533–149.472)	< 0.001[a)]
Palliative care consult order (reference: no)	1.552 (0.828–2.910)	0.171

OR, odds ratio; CI, confidence interval; ED, emergency department.

Statistically significant predictor.

DISCUSSION

In our cohort a “No” response to the ED-MSQ, “Would you be surprised if this patient died in the next 30 days?” appears to offer prognostic value regarding increased resource utilization for admitted patients and appears to help predict inpatient mortality. The risk of in-hospital mortality, use of comfort-only order set, and discharge to hospice—all markers of unmet PC needs at time of ED visit—were all higher for patients in the “No, I would not be surprised” group. Our results demonstrate that the ED-MSQ may be used by ED physicians as a simple, effective time-of-admission trigger for a more comprehensive PC needs assessment.

A landmark study published in 2010 by Temel demonstrated that early PC for patients with metastatic lung cancer leads to reduction of symptoms, improved both quality of life and survival despite less-aggressive end-of-life care [13]. Since that time, a rapidly-expanding evidence base has supported these findings for a broad range of terminal diagnoses. More recently, the first randomized study of ED-triggered PC for patients with advanced cancer reproduced the key results of the Temel study, suggesting attention to unmet PC needs in the ED can help patients to live better and longer lives [1].

Despite evidence suggesting the benefits of PC are strongest when PC teams are engaged early in the course of terminal disease, PC is still underutilized to be of maximum benefit [5]. Emergency physicians, who have no long-term relationships with terminally-ill patients but who meet them in moments of crisis, may be uniquely positioned to objectively identify those unmet needs. Furthermore, as a point of access to medical care for patients across the socioeconomic and health literacy spectra, the ED has been identified as a key point for identification of unmet PC needs in vulnerable populations without access to high-quality outpatient care [14].

The ED-MSQ is simple to implement, requires no training to answer, and is easily incorporated into time-of-admission orders in an electronic medical record. Unlike diagnosis-based triggers, which are being used in current studies on ED-triggered PC assessments [15], the ED-MSQ may capture both those with known terminal illness and those with an unexpected change in health who may have a more urgent need for PC assessment. Given its high specificity for in-hospital mortality and hospice discharge, the ED-MSQ alone could be used to trigger the use of a more comprehensive assessment tool in a subset of inpatients. Although high sensitivity is generally considered to be an important characteristic of an initial screening tool, most hospitals do not currently have the resources to perform comprehensive PC assessments on the number of patients that would be identified with a highly-sensitive but poorly-specific tool.

The high NPV (97.9%; 95% CI, 97.5% to 98.3%) for in-hospital mortality observed in this cohort indicates that when emergency physicians provide an affirmative response to the ED-MSQ (yes, I would be surprised), the probability of surviving to hospital discharge is approximately 98%. This may be clinically useful for considering a physician’s desire to avoid failing to provide helpful resources to the sickest of patients, those who might die before leaving the hospital, particularly in environments with limited PC resources. Should a physician respond ‘yes, I would be surprised,’ it would be quite unlikely that their patient would expire during the hospitalization. In resource-limited settings without extensive PC resources, the ED-MSQ may help to identify those who are least likely to have acute need for early initiation of PC, therefore allowing for resources to be targeted at those who are most in need.

Given the low sensitivity and PPV for in-hospital mortality demonstrated in this population (which does not include cardiac catheterization patients and those taken directly to the operating room), providers should be aware of the severity of illness in their cohort population. We believe the value for this tool for the ED setting is two-fold. First, it provides a simple trigger for emergency clinicians to be thinking about early initiation of PC. In addition, it may provide value in targeting resources towards those most acutely in need.

Our study was limited to a single tertiary-care medical center ED, where the particular culture and practice patterns may influence responses to the ED-MSQ. Although the ED-MSQ was initiated at the time of admission for all patients admitted to the hospital from the ED, patients taken directly to the operating theater or cardiac catheterization laboratory, bypassed this typical admission process and therefore did not have the ED-MSQ answered. As this particular population likely represents a disproportionately ill segment of patients entering our institution, had they been included as part of this intervention, it is likely that the prognostic value of the ED-MSQ would have been stronger. In addition, the data used in this analysis were de-identified, rendering it impossible for us to account for multiple visits by the same patient. As a result, each encounter was handled as an independent visit.

It should also be noted that the ED-MSQ will likely work best to identify the sickest of patients with PC needs, one would suspect that it may not adequately identify those patients who are not terminally ill but have underlying conditions such as severe COPD, HIV and various malignancies that could also benefit from early PC services. Furthermore, as implemented here, the ED-MSQ does not help to identify those patients in need of PC services that are discharged from the ED. While we suspect there are patients discharged from our ED with unmet PC needs the described intervention was not designed to identify this segment of our population. We also acknowledge that in-hospital mortality may not fully represent true 30-day mortality statistics; unfortunately national death records were not available to us at the time of this pilot study.

While the focus of this study was not on identifying patients with PC needs early in their disease process, it was intended to identify these patients earlier in their hospital course. Future research should also focus on mechanism to optimize the identification and management of patients with unmet PC needs earlier in their illness trajectory.

Emergency physicians are uniquely positioned to improve symptom burden and quality of life in patients with terminal illness by identifying unmet PC needs at the time of admission. The ED-MSQ is a simple, clinician-based trigger for PC needs assessment that may help to identify patients with a significantly-higher risk of mortality and use of PC resources. We hope to further define the clinical implications of this tool with future investigations targeting the tool’s ability to predict outcomes in specific ED patient populations, as well as develop further work flow optimizations that help facilitate PC consultation based on the ED-MSQ’s responses.