Dynamic data in the ED predict requirement for ICU transfer following acute care admission.

Misidentification of illness severity may lead to patients being admitted to a ward bed then unexpectedly transferring to an ICU as their condition deteriorates. Our objective was to develop a predictive analytic tool to identify emergency department (ED) patients that required upgrade to an intensive or intermediate care unit (ICU or IMU) within 24 h after being admitted to an acute care floor. We conducted a single-center retrospective cohort study to identify ED patients that were admitted to an acute care unit and identified cases where the patient was upgraded to ICU or IMU within 24 h. We used data available at the time of admission to build a logistic regression model that predicts early ICU transfer. We found 42,332 patients admitted between January 2012 and December 2016. There were 496 cases (1.2%) of early ICU transfer. Case patients had 18.0-fold higher mortality (11.1% vs. 0.6%, p < 0.001) and 3.4 days longer hospital stays (5.9 vs. 2.5, p < 0.001) than those without an early transfer. Our predictive analytic model had a cross-validated area under the receiver operating characteristic of 0.70 (95% CI 0.67-0.72) and identified 10% of early ICU transfers with an alert rate of 1.6 per week (162.2 acute care admits per week, 1.9 early ICU transfers). Predictive analytic monitoring based on data available in the emergency department can identify patients that will require upgrade to ICU or IMU if admitted to acute care. Incorporating this tool into ED practice may draw attention to high-risk patients before acute care admit and allow early intervention.

Introduction

The majority of hospital admissions now come through the Emergency Department (ED) [1] and Emergency Medicine physicians usually provide the initial clinical determination of the appropriate destination for these patients. Misidentification of illness severity may lead to patients being admitted to an acute care unit then unexpectedly transferring to an ICU as their condition deteriorates [2]. While early unexpected ward-to-ICU transfers of ED admissions are infrequent (around 3% at 12 h post-admission) [3], these events result in an increased need for ventilator support [4], increased in-hospital and 30-day mortality [5-7], and prolonged length-of-stay (LOS) [5, 8]. The rates of ED floor admissions with early ICU transfer has been proposed as a quality metric because of these worsened outcomes [9, 10]. For these reasons, methods for identification of the trajectory of patients prior to admission and determining the appropriate level of care is important.

Guidelines for appropriate ICU admission are in use, but current tools have significant limitations. The Society of Critical Care Medicine (SCCM) provides guidelines which rely primarily on need for specific resources (i.e., mechanical ventilation or vasopressor requirements) [11]. As such, outside of these clear indications for ICU care, the decision to admit to a floor ward versus ICU is based on the patient’s perceived stability and consideration of other, often less clear, social and clinical determinants [12]. Early warning scores (EWS) exist for identifying acute and intensive care patients at risk for decompensation: CURB-65 predicts mortality due to pneumonia [13]; sequential organ failure score (SOFA) and quick SOFA (qSOFA) scores predict mortality due sepsis [14]; and Acute Physiology And Chronic Health Evaluation (APACHE) scores predict in hospital mortality at ICU admission [15]. It is unclear how well these scores predict unexpected ward-to-ICU transfers. While some previous studies have developed models for predicting unexpected ICU transfers, these studies are limited to patients presenting with specific disease entities [16], do not take advantage of vital sign and laboratory data [17, 18], and do not differentiate time of ward-to-ICU transfer after admission [18, 19].

Accurate identification of ED patients that will decompensate within 24 h if admitted to acute care would allow for targeted interventions: early ICU consultation, ICU or step-down unit admission, or closer follow-up monitoring. This early intervention may avert later clinical deterioration, improving patient outcomes. It has been previously shown that patients with “borderline” ICU admission criteria for pneumonia have better outcomes when admitted to the ICU [12]. Given this, the development of an predictive model to help identify patients at risk for unexpected ward-to-ICU transfer is warranted. Such a tool should utilize available information, provide the flexibility to handle missing information, and provide real-time analysis based on updated data. The objective of this study was to develop a predictive analytic tool to identify ED patients at risk for decompensation within 24 h following acute care admission. The long-term goal of this approach would be to provide real-time risk assessment for clinical decision support to physicians and aid determination of appropriate disposition.

Methods

Study design and setting

We undertook a retrospective cohort design to identify consecutive ED-to-inpatient admissions to the University of Virginia Medical Center (UVa), a 612-bed academic, tertiary-care center. Annual ED volume is approximately 60,000 with a 37.7% admission rate. Adult critical care units include a medical ICU, surgical-trauma ICU, coronary care ICU, thoracic and vascular surgery ICU, neuro/neurosurgical ICU, surgical-trauma intermediate care unit (IMU), thoracic and vascular surgery IMU, and a neuro/neurosurgical IMU. The primary outcome was acute care admission, followed within 24 h by transfer to IMU, ICU, or death (i.e., early ICU transfer). The time of inpatient admission and that of IMU or ICU transfer were determined using data warehouse admit-discharge-transfer records.

Selection of participants

Patients were enrolled between Jan. 1, 2012 and Dec. 31, 2016. The study included all adult patients (18 years old or greater) admitted through the ED to acute care: two general medicine wards, two cardiovascular medicine wards, and one ward each of cardiovascular surgery, medical/surgical, surgery, orthopedic/trauma, neurology, neurosurgery, hematology/oncology, and geriatric/palliative wards. We excluded patients admitted to pediatric units, labor and delivery, psychiatry, and post-anesthesia and care transfer units. Patients with DNR orders prior to, or within 24 h following, admission were excluded. This study was approved by the UVa Institutional Review Board with a waiver of consent.

The event population was patients with early ICU transfer following inpatient admission to acute care. All data available for patients during their ED stay were included in the analysis.

Prognostic features

Patient data were extracted from the electronic data warehouse. Only data elements available during the ED visit were used as feature variables. Dynamic data included flowsheet vital signs and laboratory measurements frequently available at admission. We extracted 7 nurse-documented vital signs: pulse rate (99.8% available), respiratory rate (99.4%), oxygen saturation (99.6%), temperature (96.3%), Glasgow coma scale (GCS, 33.4%), and systolic and diastolic blood pressure (99.7%). We extracted 12 laboratory results from basic metabolic panels and complete blood counts, both 92% available. These intermittent features were combined using sample-and-hold, and censored when the value was older than 24 and 48 h for vital signs and labs, respectively. Vital signs and laboratory measurements were treated as continuous variables in the models, except that GCS (available for 33.4% of admits) was treated as a binary feature of whether or not it was measured. We also included binary features indicating whether or not a less-frequently available laboratory test was available: complete metabolic panel (75% available), CBC with differential (37%), arterial blood gases (3%), lactate (28%), troponin I (16%), and prothrombin time (56%).

We included demographics as well as clinical characteristics available on admission. This included age, length of time boarding in the ED, and length of ED stay as continuous variables in the model. We also included the following as binary variables: male sex, presence of a blood culture order in the ED, administration of intravenous (IV) antibiotics in the ED, triage acuity category, admission at night (11 pm to 7 am), evening admission (3 pm to 11 pm), admitting Charlson comorbidity index (CCI) diagnoses [20], and use of supplemental oxygen on admission. The CCI diagnoses were determined using ICD9 and ICD10 codes [21, 22].

Finally, we included several pre-existing risk scores. CURB-65 estimates mortality due to pneumonia and is intended to identify patients requiring inpatient admission. The national early warning score (NEWS) is a scoring system designed for track-and-trigger use across the National Health Service [23]. The systemic inflammatory response syndrome (SIRS) criteria, SOFA and qSOFA are designed to identify patients with suspected infection at high risk for poor outcomes [14, 24, 25]. Frost et al. present a risk model for early ICU transfer [18]. The APACHE II score [26] identifies patients at high risk for in-hospital mortality [15]. The laboratory-based acute physiology score (LAPS) identifies patients at risk for early ICU transfer [27]. We calculated these scores based on data available in the ED prior to admission to acute care, and used the last score measured during the ED stay. For APACHE we were unable to determine chronic organ insufficiency and immunocompromise. For LAPS we were not able to identify high risk patients for whom different imputations should be used because the laboratory score preliminary model was not available [27].

Predictive analytics to predict early ICU transfer

We developed a predictive model on the entire cohort and used cross-validation to reduce bias in feature selection and performance characteristics. Only the last values available prior to admission were used for modeling. All modeling was performed in R [28] using the rms package [29]. We constructed a binary logistic regression model to identify patients at risk for early transfer. We first removed, blinded to the outcome, the most predictable features correlated more than R2 of 0.9 with other features. Missing values were imputed with median of all available values. Next, we built a model with all remaining features and removed features using fast backward elimination [30]. Another model was built with restricted cubic spline transformations using three knots of nonlinearity on each feature with enough unique values. The model (linear or non-linear) with the highest Akaike Information Criterion (AIC) was retained. We performed tenfold cross-validation with fast backward elimination to ensure unbiased feature selected, removing features eliminated and retraining the model. Finally, we calculated the predicted risk using leave-one-out cross validation [31]. Briefly, the first hospital admission was identified as index admission, and we built a model on the remaining N-1 admissions. The predicted risk for the index admission was estimated using this model, and the procedure repeated for each of N admissions. In this way risk estimates for each admission were out-of-sample.

Statistical analysis

Risk marker performance was evaluated on the cross-validated risk predictions for two use cases of acute care admission decision: (1) continuous risk estimation during stay in the ED and (2) threshold-based alerting strategies for use before admission [30]. For continuous risk estimation we calculated the area under the receiver operating characteristic (AUC) and the area under the precision-recall curve (AUCPR). The AUCPR was calculated using the PRROC package for R [32]. The AUC, also known as C-statistic, was calculated using the Mann–Whitney U test. The 95% confidence intervals for AUC and AUCPR were calculated using 200 bootstrap runs. For each bootstrap run, predicted risk and the associated outcome were resampled with replacement and the AUC was calculated. The 95% confidence interval was then defined as the 5th and 195th values of AUC.

For threshold-based alerting, we pre-specified a target rate for early ICU transfer (i.e., we set the sensitivity) and determined the number of alerts required to achieve that target. We selected the last estimated risk prior to admission. The threshold was determined using the pre-specified sensitivity and the resulting alert rate, positive predictive value (PPV), and specificity were recorded.

Results

Characteristics of the study subjects

There were 77,507 admissions through the ED during the study period. The inclusion and exclusion criteria yielded 42,332 admissions for analysis. Table 1 shows a breakdown of the study population inclusion and exclusion criteria. There were 496 events (1.2%), including 425 early ICU transfers, 62 early IMU transfers, and nine early deaths.

Table 1

Description of the study population

Unit	Excluded		Admits	Event (%)
All inpatient admits			77,507
w/o data	707
to IMU/ICU	15,132
Excluded units	12,867
DNR	6320
Age < 18 years	149
Total included			42,332	496 (1.2)
CVSurg			1814	36 (2.0)
Nsurg			2425	38 (1.6)
CVMed			7623	111 (1.5)
SSU			815	11 (1.3)
HemOnc			1224	16 (1.3)
Geri/Palli			3350	43 (1.3)
GenMed			11,351	126 (1.1)
Ortho/Tr			3099	34 (1.1)
Surgery			2396	23 (1.0)
Neuro			4486	38 (0.8)
MedSurg			3749	20 (0.5)

Table 2 shows characteristics of the study population, i.e., acute care admits with and without early ICU transfer. Acute care admits with early ICU transfer had 18.0-fold higher mortality and stayed in the hospital 3.4 days longer than acute care admits without early ICU transfer. We note that event patients with early IMU transfer did not have significantly different hospital length of stay or mortality than those with early ICU transfer: 5.7 vs. 6.2 days, p = 0.19 and 4.8% vs. 10.1%, (p = 0.18).

Table 2

Characteristics of the study population showing median and IQR

	Acute	ICU no vent or pressor	ICU	Acute w/ICU trans
Count	41,836	7513	10,530	496
Male	50.9%	57.8%	58.6%	55.2%
Age	57 (44–70)a	58 (45–71)a	58 (45–71)a	63 (50–74)
Hours in ED	5.2 (3.5–7.6)	4.9 (3.4–7.0)a	4.6 (3.1–6.7)a	5.5 (3.6–8.0)
Hours boarding	0.2 (0.1–1.6)a	1.7 (1.1–2.8)a	1.6 (1.1–2.8)a	0.3 (0.0–2.2)
CCI	0 (0–2)	0 (0–0)a	0 (0–0)a	0 (0–2)
APACHE IIb	4 (2–6)a	6 (3–9)	7 (3–11)a	5 (3–8)
Intubated during stay	1.5%a	12.9%a	33.6%a	24.8%
LOS	2.5 (1.3–4.5)a	4.1 (2.5–7.5)a	4.8 (2.7–9.1)a	5.9 (3.7–10.6)
Mortality	0.6%a	2.9%a	7.6%a	11.1%

aValue is significantly different (p < 0.05) from acute care with early ICU transfer

bAPACHE II score based on pre-admission data

Table 2 also shows characteristics of adult admits directly to ICU for comparison. A secondary analysis for ICU admits without mechanical ventilation or vasopressors is also shown, as this population was felt to more closely represent a patient population which could have been potentially triaged to acute care or “floor” status. Acute care admits with early ICU transfer are older and have significantly higher mortality and length of stay than any other group. Admits with early ICU transfer have 3.8-fold higher mortality and 1.8 days longer hospital stay than direct ICU admits without vasopressors or mechanical ventilation. Pre-admit APACHE II scores for these two groups are not significantly different. Acute care admits with early ICU transfer had 46% higher mortality (p = 0.004) and stayed in the hospital 1.1 days longer (p < 0.001) than admits directly to ICU without DNR orders, commensurate with findings of Liu et al. [5].

Figure 1 shows the characteristics of the risk marker model using cross-validated risk estimates. The features in the model are shown in Fig. 1a ordered by goodness-of-fit quantified by chi-squared minus degrees of freedom. No features were removed during cross-validation fast backward feature selection. The most important feature is high heart rate, followed by need for supplemental oxygen, longer boarding time, and abnormal sodium. The qSOFA and CURB65 scores were also selected for the model. Time series of predicted risk leading up to the time of admission is shown in Fig. 1b. Risk estimates are shown in risk relative to average risk of early ICU transfer. That is, 1.0 represents the average risk, 2.0 is twice the average risk, and so on. White points indicate that risk estimates are significantly higher than risk estimates for the same patients 3 h prior. Risk estimates for admits with early ICU transfer (solid) are significantly higher than risk estimates for control patients, and rise from about twice the average risk to three times the average during their stay in the ED. The risk is significant beginning about 6 h prior to inpatient admission. Figure 1c shows the density of risk estimates for patients with (solid) and without (dashed) early ICU transfer. Figure 1d shows the predicted versus observed risk of early ICU transfer for deciles of predicted risk. Better calibrated models are closer to the line of identity, and better discriminating models have larger separation between deciles. The risk model is well calibrated in the middle 80% of predicted risk, overestimates risk for high risk patients, and underestimates risk for low risk patients. The model has good discrimination: the observed risk of admits with the highest decile of predicted risk is 7.9-fold (95% CI 5.0–12.9) higher than admits in the lowest decile of predicted risk.

Fig. 1

a Components of risk marker ordered by goodness-of-fit measured by chi-square minus degrees of freedom. Net positive coefficients are shown in black, net negative coefficients are shown in white, and features with non-linear association are shown as triangles. b Mean time course of the risk marker near the time of admission to acute care. Data are shown for control (dashed lines) and event (solid lines) admits. c Density of log odds predicted by risk marker, shown for event (solid) and control (dashed) admits. d Observed and predicted relative risk for risk marker. The observed relative risk is shown as a function of the relative risk predicted by the risk marker model. Each point represents 10% of measurements

The AUC of the risk marker using cross-validated risk estimates was 0.697 (95% CI 0.671–0.723). The AUCPR of the risk marker using cross-validated risk estimates was 0.54 (0.044–0.073). Figure 2 shows the characteristics of alerts based on the risk marker model. For reference, there were 162.2 acute care admits through the ED per week, of which 1.9 admits had early ICU transfer. Selecting a sensitivity (dashed) or positive predictive value (solid) on the abscissa determines the number of alerts per week on the ordinate that would be required to obtain that performance. In order to identify 10% of ED patients that go on to have early ICU transfer, for example, about 1.6 alerts per week would sound and the PPV would be 11.8%. For a system that sent an alert once every 4 days, 1 in 8 would identify an admit that results in early ICU transfer and 10% of early ICU transfer admits would be identified.

Fig. 2

Characteristics of the risk model for alerting. The number of alerts per week is shown as a function of the PPV (solid) and sensitivity (dashed). The maximum value of the ordinate is the number of admits through the ED per week (162.2) and the dashed horizontal line is the number of admissions leading to early ICU transfer per week (1.9). The rate of admission leading to early ICU transfer is shown as a dashed vertical line and is the lower bound on PPV

Performance characteristics of existing EWS are shown in Table 3. The cross-validated AUC of the risk marker model is 45% higher than the best performing EWS, the NEWS score (the AUC of a random classifier is 0.5). The PRAUC of the risk marker model is 330% higher than the best performing EWS, the NEWS score (the PRAUC of a random classifier is the event rate, 1.2%). The risk marker model requires fewer than one third the number of alerts to achieve a sensitivity of 10%.

Table 3

Performance of existing early warning scores for predicting early ICU transfer based on data available in the emergency department

	PRAUC	AUC	Alarms per week	PPV (%)	Specificity (%)
UVa	0.054 (0.044–0.073)	0.697 (0.671–0.723)	1.6	11.8	99.1
NEWS	0.022 (0.018–0.026)	0.636 (0.615–0.662)	5.6	3.5	96.7
SIRS	0.019 (0.016–0.023)	0.607 (0.583–0.633)	6.2	3.1	96.3
qSOFA	0.018 (0.016–0.021)	0.587 (0.566–0.612)	7.2	2.7	95.7
CURB65	0.019 (0.015–0.026)	0.605 (0.582–0.630)	8.1	2.4	95.1
APACHE II	0.020 (0.016–0.026)	0.614 (0.591–0.638)	8.3	2.3	95.0
SOFA	0.017 (0.015–0.020)	0.592 (0.568–0.615)	9.4	2.0	94.3
LAPS	0.021 (0.018–0.024)	0.620 (0.597–0.648)	4.8	4.0	97.1
Frost	0.014 (0.013–0.016)	0.576 (0.553–0.601)	12.3	1.6	92.5

Discussion

Predicting which patients are at risk for unplanned ward-to-ICU transfer at the time of admission has the potential to significantly improve patient safety. In our population, patients admitted to a hospital ward then transferred to an ICU have significantly worse mortality than those admitted to the ICU directly, similar to findings in previous studies [5–8, 33]. Our model demonstrated the ability to identify patients at risk for ward-to-ICU transfers. If implemented, our model would allow an alert to be activated prior to admission for the subset of patients at high risk for transfer to an ICU. Implementation would result in fewer than 2 such alerts per week, which is important to prevent alarm fatigue. Despite being infrequent, these alerts would have the potential to decrease ward-to-ICU transfers by 10%, and significantly outperform existing models such as NEWS and LAPS.

Our model was built to perform continuous risk assessment. Risk estimates made by the model were significantly higher for cases than controls throughout the ED stay. In addition, the risk estimates for patients with subsequent ward-to-ICU transfer increased during their ED stay. These patients had twice the average risk six hours prior to admission, which rose to three times the average risk immediately prior to admission. This indicates that not only are risk estimates useful for identifying high risk patients, but that risk score trajectories may also be important.

Prior clinical decision support tools used in clinical practice have tended to be simple, point-based systems for ease of recall and use. Better integration profiles and standards for the hospital enterprise now allow for more complex integration of predictive models into clinical practice. More data points and complex weighting of variables can be used by integrating decision tools directly into data acquisition systems and the electronic medical record (EMR). This has been successfully implemented in monitoring for neonatal sepsis and showed a 20% mortality reduction just by making risk estimates available to clinicians at the bedside [34].

Some have expressed skepticism towards adoption of such clinical scoring systems, stating that clinical decision instruments are seldom favorably compared to physician gestalt [35]. While we recognize that scoring systems and risk scores cannot supplant physician experience, they can certainly provide valuable information to a treating provider. This is evidenced by reductions in rates of septic shock and morality associated with clinical implementation of predictive analytic tools [34, 36]. Our current work also largely sidesteps this issue, as the event population is virtually pathognomonic of a failure in gestalt with resultant inappropriate initial disposition and demonstrable harm. Additionally, in a previous analysis where point of care clinicians were interviewed and observed following implementation of continuous predictive analytics in the ICU, they noted that the analytics were viewed as a key piece of clinical information once adopted into workflow through routine documentation, reported as a risk score during patient rounds and handoffs of care [37].

Future studies will be directed towards implementation of a continuous predictive analytics tool in the ED to determine the impact on early acute care to ICU transfer. To date, there has not been a similar analysis conducted in an ED setting where the patient ratios, clinical presentations, and workflow patterns are very different than other environments. Due to the relative rarity but clinical relevance of the event of interest, it would be critical to incorporate implementation and study of continuous predictive analytics within existing clinical decision support frameworks. Specifically, alert risk thresholds could be established to aid providers in making the initial decision for floor versus ICU admit. Beyond the initial disposition decision, this score could be communicated to the oncoming admission team to guide management decisions early in the post-admit trajectory (e.g., through more frequent vital sign ascertainment or early attending assessment). Additionally, such a tool may assist task-saturated physicians in a supervisory role to appropriately triage attention to patients at higher risk of clinical decompensation. In order to impact distal patient outcomes of unintended ICU/IMU upgrade, technologies must move from passive surveillance to integrated clinical decision support [38].

While previous studies have examined ICU transfer at any point following admission [18, 19], we focused on ICU transfer in the first 24 h because these are patients whose outcomes may be most readily impacted by changes in ED care. The relative rarity of these events makes them difficult to predict. The rate of these early ICU transfer in our study population was 1.2%, which is on the lower end of the range seen in similar studies (1.2–4%) [18, 39]. Despite the low rate of these events, they are associated with significantly worsened outcomes. In our study population, hospital length of stay and mortality were both increased in patients with early ward-to-ICU, even relative to patients admitted directly to the ICU.

Our analysis demonstrated that factors specific to a patient’s current presentation were the most important to determine risk for early transfer. The most important features were high heart rate, need for supplementary oxygen, longer time boarding in the ED, and abnormal sodium. Existing EWS were incorporated into the model: qSOFA, an aggregate of vital signs (respiratory rate, Glasgow coma scale, and blood pressure) and CURB65 (age, blood pressure, respiratory rate, and higher urea nitrogen levels). Low calcium and the clinical decision to draw a troponin (irrespective of result) were also selected as features. This means that incorporating the latest vital sign and laboratory information into real-time prediction is essential to creating an accurate model of patient risk. Better identification of high-risk patients in the ED might lead to earlier or more targeted treatment, may prompt direction admission to the ICU, and could reduce early ICU transfer and the associated poorer outcomes.

This study is limited in that the model was developed and tested at a single center. Models may need to be tuned for a specific institution with a different population or with different clinical protocols. Our study was also limited by not incorporating continuous data from cardiorespiratory bedside monitors. Our results indicate that current patient state is important. Continuous cardiorespiratory dynamics provide the highest resolution look at patient physiology and would provide additional relevant information [40]. Additionally, the reason for upgrade to ICU-level care was not readily available in our dataset. While such information would help provide a more robust framework in which to interpret of the results, we believe the presence of meaningful markers of deterioration in the event population serves as an effective proxy for appropriateness of ICU transfer. We also recognize that there are other non-clinical features that may influence the decision to admit a patient to either the acute care or ICU following an ED visit, including bed availability, local practice patterns, and nurse staffing considerations [11]. Future study involving a multi-center or prospective design could offset some of this variability and provide more robust, granular data.

In summary, it is likely that predictive analytic monitoring based on ED data can help identify patients that will require upgrade to ICU or IMU if admitted to acute care. Incorporating this tool into ED practice may draw attention to these high-risk patients and allow early intervention. Intervention before admission may ameliorate or prevent deterioration and reduce the associated morbidity and mortality.