Real-time prediction of patient disposition and the impact of reporter confidence on mid-level triage accuracies: an observational study in Israel.

AIM: The emergency department (ED) is the first port-of-call for most patients receiving hospital care and as such acts as a gatekeeper to the wards, directing patient flow through the hospital. ED overcrowding is a well-researched field and negatively affects patient outcome, staff well-being and hospital reputation. An accurate, real-time model capable of predicting ED overcrowding has obvious merit in a world becoming increasingly computational, although the complicated dynamics of the department have hindered international efforts to design such a model. Triage nurses' assessments have been shown to be accurate predictors of patient disposition and could, therefore, be useful input for overcrowding and patient flow models.
METHODS: In this study, we assess the prediction capabilities of triage nurses in a level 1 urban hospital in central Israeli. ED settings included both acute and ambulatory wings. Nurses were asked to predict admission or discharge for each patient over a 3-month period as well as exact admission destination. Prediction confidence was used as an optimisation variable. RESULT: Triage nurses accurately predicted whether the patient would be admitted or discharged in 77% of patients in the acute wing, rising to 88% when their prediction certainty was high. Accuracies were higher still for patients in the ambulatory wing. In particular, negative predictive values for admission were highly accurate at 90%, irrespective of area or certainty levels.
CONCLUSION: Nurses prediction of disposition should be considered for input for real-time ED models. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study was conducted on a large cohort of patients, very few of whom were excluded from analysis, thus strengthening the reliability of the results.

To our knowledge, this is the first study of its kind conducted in Israel, and the fact that the data support that of previous studies from other regions is reassuring.

The study was limited to data collected from one emergency department (ED) in one institution in Israel and did not take into account the nurses’ experience or educational background, limiting both its internal and external validity.

We are unable to draw conclusions on prediction accuracy related to specific diseases or presentation (ie, chest pain/acute coronary syndrome) as this was beyond the scope of our study.

We believe that the results of the study indicate that predictions could be effectively used as part of a more holistic real-time, machine learning ED analysis tool as an accurate, cost-efficient and quick input metric.

Introduction

Overcrowding in the emergency department (ED) has become such a common phenomenon that it is become a routine working environment in many hospitals. The strain on staff and hospital resources has an impact on the ability to provide adequate medical services and directly correlates with the quality of patient care and overall hospital experience. Multiple studies have demonstrated that ED overcrowding has a negative effect on many outcomes, including patient mortality and waiting times,1 door to needle time in patients suspected of having acute myocardial infract (door to needle time is the elapsed time between the arrival of a patient with acute myocardial infarction (MI) to the hospital and the start of coronary arteries catheterisation). It is generally accepted that sub 90 min provides optimal outcomes,2 pain management3 and delays in antibiotic administration.4 Additionally, overcrowding is a major contributing factor in staff burnout.5

Overcrowding is, therefore, a frequent topic of internal auditing and research publications. In Israel, a national survey conducted in 2018 revealed that EDs on average operated at 104% capacity, with an average length of stay of 3.0 hours.6 The Tel Aviv Sourasky Medical Center (TASMC) ED, the location of this study, is a particularly busy inner-city hospital, with a patient length of stay of 3 hours on average and even higher for those requiring admission (51% staying over 5 hours).6

Improvement in real-time analysis and computational models of ED overcrowding are expected to facilitate better provision of medical treatment and allocation of resources, thus improving patient outcome in the ED as well as in the admitting hospital departments.1 3 There are many tools designed for retrospective analysis of ED disposition prediction and overcrowding.7 Several studies have shown that tools combining objective metrics with triage nurses’ disposition predictions are able to produce good patient admission prediction as early as at time of triage.8 In recent years, there have been attempts to construct real-time overcrowding models, often using triage scores and bed availability as inputs.9 10 Examples include The National Emergency Department Overcrowding Scale, the Emergency Department Work Index and the Risk Management, Economic Sustainability and Actuarial Science Development in Indonesia2.10 No study has, as of yet, compared the efficacy of these tools.

In TASMC’s ED, nurses triage patients using the Canadian Triage and Acuity Scale (CTAS), which is a model combining subjective metrics such as presenting complaint and severity of pain with objective metrics such as vital signs, evidence of bleeding, presence of rash, etc.11 CTAS levels range from 1 to 5 and indicate the urgency in which patients require medical attention. A score of 1 indicates patient who require immediate attention in the resuscitation bay, whereas 5 indicates non-urgent cases with the lowest priority. In the USA, for comparison, the Emergency Severity Index (ESI) triage method is the most commonly used.

Many studies have demonstrated that triage nurses are able to predict patient disposition with a high degree of accuracy, based on their experience and the limited information available to them at the time of triage. For example, Danette et al published a study, in which triage nurses were able to predict admission with 71.5% sensitivity and discharge with 88.0% specificity.12 The negative predictive value (NPV) for discharge was also particularly high at 90%. Predictions were most accurate for young patients and for patients with a low (level 1) or high (levels 4–5) ESI score.12 Another study looking at overall disposition predictions demonstrated similar results (sensitivity 75.6% and specificity 84.5%).13 Importantly, when nurses were asked to assign a level of confidence to their predictions, a high degree of certainty correlated with improved accuracy of disposition prediction (sensitivity 83.6%, specificity 93.1%, NPV 95%).13 However, a similar study from the UK was unable to demonstrate high accuracy of triage disposition predictions (sensitivity and specificity 68% and 85%, respectively).14 The accuracy of nurse triage in Israel has never been assessed in.

In addition to triage nurse predictions, several studies have explored the possibility of using objective metrics to predict patient disposition. A 2009 retrospective study examined 1100 patient cases in 6 medical centres, excluding trauma, psychiatric and obstetrics and gynecology patients. That study used a variant automatic prediction model available during triage: age over 60, chest pain, shortness of breath, dizziness, weakness or syncope, history of cancer, history of diabetes. Each variant was ascribed a weight with a total combined score of 0–14. When the total score was above 4 (34% of cases), the likelihood of admission was 77%, and when the score was above 5 (29% of cases), the likelihood rose to 80%.15

Another study attempted to build a prediction model based on data that are routinely collected during triage. This retrospective study included approximately 300 000 ED case files. Of these cases, 60% were used to train the model and 40% were used to validate it. The data used as input for training included demographic characteristics (age, sex, ethnicity), recent (<3 months) hospital admissions or ED visits, method of arrival, patient acuity category and the presence of chronic illness (eg, diabetes, hypertension, dyslipidaemia). The variables that were found to be significant for hospitalisation prediction were age, method of arrival and patient acuity category.16

The concept of combining triage predictions and admission prediction models was explored by Cameron et al. In their research, they compared the prediction ability of triage nurses to that of a simple clinical tool, the Glasgow Admission Prediction Score (GAPS) is a score based on age (a point is given for each decade) triage urgency level (20 points for level 1 and 5 points for level 3); 10 points are given if the patient was referred by a doctor to the ED; 5 points are given if the patient was brought in by ambulance or was admitted in the last 12 months. The model also gives a point for each point received by the National Early Warning Score, a score based on vital signs (NEWS score). This tool was found to be efficient in predicting admission.17. Their research demonstrated that in most cases, GAPS was superior at predicting patient admission outcome over triage nurses (accuracy of 0.810 vs 0.759).17 The exception was in cases where nurses were very with their prediction, supporting previous findings.13 The authors proposed a combination of both triage and admission prediction models. By allowing nurses to overrule GAPS when they were certain of their predication, overall accuracy was improved to 0.892.17 It is important to note that GAPS is not an objective tool as it takes into account the triage level as determined by the triage nurse.8 Riodan et al acknowledged that this in their 2017 publication examining patients with ESI level 3. They experimented with several variables including age, pulse, systolic blood pressure and pain in an attempt to build a regression model capable of predicting patient discharge.18

As with many areas of medicine, there is growing interest in the field of artificial intelligence, in particular, machine learning, to predict patient admission outcome at the triage level.19 One such study found that a trained algorithm outperformed classical methods, especially when predicting outcomes for patients with moderate scores (ie, CTAS level 3).19 20 This is a field that is expected to develop rapidly in the coming years. Another interesting study by Tahayori et al analysed the use of natural language processing (NLP) to predict the disposition of patients.21 The algorithm developed was applied to ED triage notes with a relatively high level of accuracy. Such tools are only as robust as the algorithm developed and the data that were input and used to train them, so, at present, it is necessary to continue to develop human approaches to data analysis.

Methods

This is a single centre, observational, retrospective study to determine the accuracy of nurse predictions of patient disposition and destination. Data were gathered between the period of 1 April 2019 and 30 June 2019 in TASMC ED, a tertiary hospital in central Israel, for all adult patients.

All the nurses who took part in this study were graduates of the Emergency Medicine Nursing Course. No data were collected on the nurses themselves. The medical team was blinded to the nurses’ triage predictions to avoid bias.

The participating nurses were asked to fill out a questionnaire that was embedded in the ED’s patient managing software. The nurses were aware of the study and completed the questionnaire in a short period of time with no interference with their work. For each patient, the nurse provided disposition predictions (admission or discharge), exact admission destination prediction (where relevant) and level of certainty in the predication (high, medium and low).

Patient demographic data were gathered (patients ID number, sex, age) as well as time of arrival and discharge from the ED (home vs admission), triage placement in the ambulatory wing or acute ED wing, triage level (1–5) according to CTAS, vitals (blood pressure, heart rate, oxygen saturation, respiratory rate, temperature) and pain level (according to Numeric pain Assessment Scale). Textual data regarding the reason of ED visit (ie, presenting complaint) were also included. Selection criteria included any patient visiting the ED and seen by the triage team in said period of time, excluding patients seen by the paediatric team.

Data were processed in order to calculate the sensitivity, specificity, positive predictive value (PPV), NPV and accuracy of nurses’ prediction as well as the influence of various patient characteristics on these parameters.

This manuscript was prepared in accordance with the Strengthening the Reporting of Observational studies in Epidemiology (STROBE) statement for improved reporting of outcomes from observational studies.

Results

Between April and June 2019, data were gathered from 33 685 ED visits, of which 11 143 were referred to the ambulatory wing (33%) and 22 542 to the acute department (67%). The average patient age was 51 years old. The men to women ratio was approximately 52:48. A total of 6566 cases (20%) had incomplete triage prediction forms and were excluded from the results. A total of 27 119 questionnaires were included in the analysis—19 146 (71%) acute and 7973 (29%) ambulatory. No statistically significant difference regarding disposition was found between the group that had complete triage prediction forms and the group that was excluded.

In the ambulatory wing of the ED, discharge was predicted in 7307 cases (92%), of which 6950 cases were actually discharged. For this group, the accuracy of nurse predictions was high with 95% accuracy rate. Nurses predicted hospital admission in 666 cases, of which only 312 were actually admitted. Here, the nurses’ predictive accuracy was much lower at 47%. Combined accuracy was 91%. PPV and NPV were 95% and 46%, respectively. For the purpose of this calculation, admission was defined as a positive test result and discharge negative. Sensitivity and specificity were 47% and 95%, respectively (figure 1).

Figure 1

Triage predictions according to wing.

In the acute wing of the ED, discharge was predicted in 13 145 cases, of which 10 816 were actually discharged (overall number of discharges 12 867), with a prediction accuracy of 82%. Hospital admission was predicted in 6001 cases, of which 3950 were actually admitted (overall number of admissions—6279), with a lower accuracy of 66%. Combined accuracy was 77%. PPV and NPV were 84% and 62%, respectively. Sensitivity was 63% and specificity was 84% (figure 1).

Nurses did not demonstrate a high level of accuracy in predicting the receiving admission department in the hospital at the time of triage for both acute and ambulatory wing settings. The exception to this was for admissions to the oncology department; however, this was a very small cohort (table 1).

Table 1

Disposition prediction accuracy by wing

	Acute wing			Ambulatory wing
	Number of cases predicted to be admitted	Actual number of admitted cases	Accuracy of triage predictions %	Number of cases predicted to be admitted	Actual number of admitted cases	Accuracy of triage predictions %
Surgery	687	275	40	41	3	7.3
Internal medicine	3345	1833	54.8	230	75	32.6
Ophthalmology	15	1	6.7	11	4	36.4
Cardiology	295	121	41	4	1	25
Orthopaedics	337	173	51.3	77	46	59.7
Oncology	9	2	22.2	1	1	100
ENT	97	24	24.7	60	15	25
Dermatology	79	25	31.6	109	53	48.6
Neurology	337	141	41.8	45	16	35.6
Urology	119	36	30.3	13	5	38.5
Neurosurgery	189	61	32.3	21	11	52.4

No significant difference was found in prediction accuracy between male and female patients in either wing. There was also no significant difference in the prediction accuracy for patients with normal vital signs (pulse, BP, oxygen saturation, temperature) compared with patients with abnormal vitals, remaining approximately 90% in the ambulatory wing and 76% in the acute wing. The exception to this was predictions in patients with abnormal temperatures in the ambulatory wing, which reduced prediction accuracy to 72%.

CTAS triage level had a significant influence on prediction accuracy (figure 2).

Figure 2

Effect of triage level on prediction accuracy.

As expected, with mid-CTAS levels (specifically level 3), predictions were less accurate. In the ambulatory wings, there was only one case of CTAS level 1 and less than 1% of cases were CTAS level 2. In comparison, 50% of cases were CTAS level 4.

In the acute wing, about 1% of patients were CTAS level 1. Most patients were CTAS levels 3 and 4 (50% and 38%, respectively). In this department, predictions in cases with a CTAS level 3 were particularly inaccurate.

The impact of the time of nurses’ working shift on the accuracy of prediction was also evaluated. Nurse shifts in the ED were divided into the morning (07:00–15:00), evening (15:00–23:00) and night (23:00–07:00). During the data collection period for this study, the ambulatory wing closed at 23:00; therefore, only morning and evening shifts were analysed there.

In the acute wing, average prediction accuracy was 85% during the night shift, significantly better than the evening (78%) and morning (71%) shifts. The total number of cases recorded in this study was similar for the morning and evening shifts, however, for the night shift; the number of cases was 50% smaller. There was no significant difference in the proportion of cases recorded as CTAS levels 1 and 2 between shifts, although a larger proportion of CTAS level 5 cases was seen during night shifts. For these patients, prediction accuracy was high and contributed to the overall higher accuracy level.

The degree of reporter certainty when making a prediction had a significant impact on accuracy (figure 3). In the ambulatory wings, when a nurse stated that the prediction was made with high certainty, the accuracy of the prediction was over 96%. Most predictions in this wing were stated to be highly certain or moderately certain (5235 and 2541 accordingly), and only a minority were given with low certainty (458, approximately 5.5%).

Figure 3

Effect of prediction certainty on prediction accuracy.

In the acute wing, a similar increase was observed for predictions reported as having a high degree of certainty—88% accurate, compared with 77% for the wing as a whole. In this wing, prediction uncertainty was considerably higher (2114, 11%), and the accuracy of these predictions was just 60% (compared with 70% in the ambulatory wing).

Importantly, CTAS level 3 cases with a high degree of reporter confidence were highly accurate (93% for ambulatory wing and 85% for acute wing), significantly greater than CTAS level 3 accuracies as a whole. It is important to point out that the likelihood of a high certainty prediction for triage level 3 cases is lower than average (figure 3, table 2A, B).

Table 2

Breakdown of triage level 3 cases in ambulatory and acute wards and the effect of prediction certainty

Prediction	Certainty level	% Rate	True disposition		Total	Accuracy %
			Discharge	Hospitalisation
(A) Triage level 3, ambulatory wing
Admission	Very certain	29%	15	39	54	72%
	Somewhat certain	52%	73	25	98	26%
	Not certain	19%	32	4	36	11%
Total		100%	120	68	188	36%
Discharge	Very certain	55%	806	27	833	97%
	Somewhat certain	38%	524	57	581	90%
	Not certain	7%	101	13	114	89%
Total		100%	1431	97	1528	94%
Grand total			1551	165	1716	87%
(B) Triage level 3, acute wing
Discharge	Very certain	34%	1747	255	2002	87%
	Somewhat certain	53%	2422	691	3113	78%
	Not certain	12%	498	223	721	69%
Total		100%	4667	1169	5836	80%
Admission	Very certain	31%	210	910	1120	81%
	Somewhat certain	56%	834	1154	1988	58%
	Not certain	13%	248	205	453	45%
Total		100%	1292	2269	3561	64%
Grand total			5959	3438	9397	74%

Discussion

The results of this study support the results of previous studies, namely that trained triage nurses can accurately predict patient disposition during the triage process. At the extremes of CTAS/ triage scores (1 and 5), these predictions were more accurate, as is to be expected. Additionally, reporter confidence is also positively correlated to prediction accuracy, potentially highlighting a particularly useful as well as easy metric to measure. We anticipate that the model we presented can be served as an important tool in predicting patient disposition from triage, thereby improving patient flow in the ED and reducing wait times. This system could be supplemented by machine learning and NLP, such as that presented in Tahayori et al to assist in early identification of patients who require hospitalisation and provide early notice to admitting hospital departments.

After a discussion with nurses who participated in the study, the structure of the questionnaire itself may be the cause of the inaccuracy in predicted admission destination. However, patients are not always admitted to the most suitable ward due to factors outside the control of the ED, such as bed availability. The subject of destination prediction and the varying limiting factors will be further evaluated in future studies.

Regarding the difference in the prediction accuracy between different shifts, it seems that the higher accuracy in the acute wing during night shifts may be in part due to a greater percentage of CTAS level 5 triage patients in that wing during this shift, as ambulatory patients are also seen there at night. As level 5 cases were predicted with a greater degree of accuracy, this may explain the results.

Careful consideration was given to the analysis of CTAS level 3 patients in this study. These patients represent a substantial percentage of presentations to the ED. In general, reporters struggled to accurately predict disposition for this group. It was demonstrated, however, that when the triage nurse was confident in their prediction for this group, the accuracy was also high. This metric may, therefore, allow for accurate predictions for subset of level 3 patients.

An additional study, ongoing at the time of writing, will evaluate the ability of triage predictions to improve the accuracy of a machine learning algorithm designed to predict overcrowding and patient disposition, especially in areas that demonstrated poor accuracy (ie, CTAS level 3).

This research demonstrated that it is possible to predict future discharge with a high degree of certainty for over 60% of ED patients even as early as initial triage. This group includes all ambulatory wing patients, patients at either extreme end of triage severity levels (1 and 5) and any patient for whom the triage nurse is certain of their prediction.

Limitations

The major disadvantage of the use of triage predictions as part of an overcrowding analysis tool is the added workload for nursing staff. It is our opinion that additional evidence of the effectiveness of this method is required before recommendations are made.

It is evident from the data concerning disposition predictions that they are, in general, not accurate enough in their raw form to greatly influence the management of the ED. However, it is our belief that such data can be used as a part of a real-time ED overcrowding analysis tool, capable of assisting bed managers and improving patient flow as well as allowing for better allocation of resources.

Conclusion

Triage nurses are able to accurately predict disposition with a high degree of accuracy, particularly for patients with on either extreme end of the CTAS score. With the introduction of prediction confidence as a metric, accuracy increased for all predictions, including those made for patients with middle-range CTAS scores. However, predictions for patient destination once admitted were not accurate. We believe that implementing these metrics into a machine learning overcrowding tool may improve overall performance and assist in maximising flow through the ED, thus decreasing length of stay.