Outcome-Based Validity and Reliability Assessment of Raters Regarding the Admission Triage Level in the Emergency Department: a Cross-Sectional Study.

INTRODUCTION: Emergency department (ED) is usually the first line of healthcare supply to patients in non-urgent to critical situations and, if necessary, provides hospital admission. A dynamic system to evaluate patients and allocate priorities is necessary. Such a structure that facilitates patients' flow in the ED is termed triage.
OBJECTIVE: This study was conducted to investigate the validity and reliability of implementation of Emergency Severity Index (ESI) system version 4 by triage nurses in an overcrowded referral hospital with more than 80000 patient admissions per year and an average emergency department occupancy rate of more than 80%.
METHOD: This prospective cross-sectional study was conducted in a tertiary care teaching hospital and trauma center with an emergency medicine residency program. Seven participating expert nurses were asked to assess the ESI level of patients in 30 written scenarios twice within a three-week interval to evaluate the inter-rater and intra-rater reliability. Patients were randomly selected to participate in the study, and the triage level assigned by the nurses was compared with that by the emergency physicians. Finally, based on the patients' charts, an expert panel evaluated the validity of the triage level.
RESULTS: During the study period, 527 patients with mean age of 54 ± 7 years, including 253 (48%) women and 274 (52%) men, were assessed by seven trained triage nurses. The degree of retrograde agreement between the collaborated expert panel's evaluation and the actual triage scales by the nurses and physicians for all 5 levels was excellent, with the Cohen's weighted kappa being 0.966 (CI 0.985-0.946, p < 0.001) and 0.813 (CI 0.856-0.769, p<0.001), respectively. The intra-rater reliability was 0.94 (p < 0.0001), and the inter-rater reliability for all the nurses was in perfect agreement with the test result (Cohen's weighted kappa were as follows: 0.919, 0.956, 0.911, 0.955, 0.860, 0.956, and 0.868; p < 0.001).
CONCLUSION: The study findings showed that there was perfect reliability and, overall, almost perfect validity for the triage performed by the studied nurses.

INTRODUCTION

Emergency department (ED) is usually the first line of healthcare supply to patients in non-urgent to critical situations and, if necessary, provides hospital admission (1-4). During the past few years, with regard to the Affordable Care Act, the need of EDs to address a wide range of healthcare needs has increased (2, 5-7). An overcrowded ED could lead to several problems including staff tiredness, prolonged waiting time, threatened patient safety, and lack of adequate privacy (8-12). Therefore, a dynamic system to evaluate patients and allocate priorities is necessary. Such a structure that facilitates patients’ flow in the ED is termed triage (13, 14). It is important to apply a reliable triage system. Different triage systems including the five-level triage systems, the Emergency Severity Index (ESI), and the Canadian Triage and Acuity Scale are applied worldwide (15-17). Among these systems, the ESI is unique as it evaluates both acuity and resource utilization. The ESI algorithm includes five levels of care, ranging from the most to the least critical status. While levels 1 and 2 are based on high acuity level, levels 3–5 emphasize on resource requirements (17-19). Unstable and critically ill patients are assigned to ESI level 1 and are visited by the physician immediately while patients classified as ESI level 5 are sent to a “fast track” unit (2). ESI-version-4 (ESI-v4) is a valid and reliable triage tool, used since 2005, and is more accurate for detecting patients in level 1 (20). This study was conducted to investigate the validity and reliability of implementation of this system by expert triage nurses in an overcrowded referral hospital with more than 80000 patient admissions per year and an average emergency department occupancy rate of more than 80%.

METHODS

This prospective cross-sectional study was conducted from October to December 2014 in a tertiary care teaching hospital and trauma center with an emergency medicine residency program. The study protocol was approved by the Emergency Medicine Department Research Council and Ethics Committee of Tehran University of Medical Sciences. The researchers did not interfere in the actions related to the admission of patients and were committed to maintaining the principles of secrecy.

Seven fully trained registered nurses willingly participated in this study. These nurses had an average of 8 years of experience. The participating nurses were blinded to the goals of the study. Sampling was performed based on consensus method; so all patients had been admitting during their shifts were enrolled and there was not any sex or age limitiation in this regard.

The nurses underwent a written test to evaluate the reliability of their assessment. Thirty written scenarios were handed to them, and they were asked to scale the triage levels in 30 min (inter-rater reliability). They were asked to participate in the same test for the second time after three weeks (intra-rater reliability). The scenarios were designed and standardized by an expert panel of five emergency medicine faculty members. The distribution of the triage levels was as follows: 3 scenarios of level 1 and level 5, 6 scenarios of level 2 and level 4, and 12 scenarios of level 3. During the selected shifts, the designated triage nurses ranked the patients based on the ESI-v4 algorithm. A group of attending emergency physicians who were the instructors of the triage course also individually ranked the same patients. The nurses and attending emergency physicians were kept blinded to the triage scores evaluated by each other. Based on the department policy, the triage scoring by the designated triage nurses was considered as final for the patients’ triage. One out of each 20 patients who visited the ED, on the specific days that the nurses were in charge, were randomly selected to be evaluated in this study. The medical charts and records of all patients were then discussed in an expert panel consisting of five attending emergency medicine faculty members. Based on the medical charts and the ESI level ranked by the nurses and physicians, the validity of their triage was discussed and finalized. Final utilization of resources, ED admission and discharge rates, mortality rate, and transfer to other hospital wards and intensive care unit (ICU) were also recorded.

Regarding the use of resources, patients who were expected to consume no resources were classified as ESI level 5, those who were likely to require one resource were ESI level 4, and those in need of two or more resources were designated as ESI level 3. Patients in levels 1 through 4 were admitted to the ED. Only patients in level 5 of triage, who were not going to utilize resources, were managed by a fast track unit without ED admission. However, these patients could be admitted based on the physician’s opinion.

Hospital admission was considered as more than 24 hours in the ED or transfer to another ward. A number of the patients stayed in the ED for days before being transferred to the wards or ICU due to hospital crowding.

ED demise meant patient’s death at any time in the ED.

The gathered data were analyzed using SPSS for Windows (Version 12.0. Chicago, SPSS Inc., released 2003). The distribution of the patients’ baseline characteristics has been presented as numbers and percentages. The Cohen weighted κ statistic and Spearman's rank correlation coefficient were used to evaluate inter-rater reliability between the triage nurses. The weighted κ values were interpreted as follow: slight agreement, κ less than 0.20; fair agreement, κ 0.21–0.40; moderate agreement, κ 0.41–0.60; substantial agreement, κ 0.61–0.80; and almost perfect agreement, κ greater than 0.80. The association of ED admission (admitted/discharged) with the ESI level was assessed with Kendall's τ. P-value < 0.05 was considered statistically significant.

RESULTS

During the study period, 527 patients with mean age of 54 ± 7 years, including 253 (48%) women and 274 (52%) men, were assessed by seven trained triage nurses. Final outcome of 527 patients and their triage level as assessed by a panel of experts reported in table 1. Among the 527 patients, 174 (33.01%) stayed in the ED for less than 24 hours and were either discharged (31.3%) or died (1.7%) in the ED. On the other hand, 353 (66.98%) patients were admitted for more than 24 hours and were transferred to either the ICU (3.79%) or other wards (63.18%).

Table 1

Final outcome of 527 patients and their triage level as assessed by a panel of experts

Rates	Emergency Severity Index (ESI)
	level 1	level 2	level 3	level 4	level 5
Emergency department admission (total) (%)	5 (5) (100)	81 (105) (77.1)	207 (274) (75.5)	59 (96) (61.5)	10 (47) (21.2)
Emergency department discharge (total) (%)	0 (5) (0)	24 (105) (22.8)	67 (274) (24.4)	37 (96) (38.5)	37 (47) (78.7)
Emergency department mortality (admitted) (%)	1 (5) (20)	5 (81) (6.1)	2 (207) (0.9)	1 (59) (1.6)	0 (10) (0)
ICU transfer (admitted) (%)	3 (5) (60)	11 (81) (13.5)	5 (207) (2.4)	1 (59) (1.6)	0 (10) (0)
Ward transfer (admitted) (%)	1 (5) (20)	65 (81) (80.2)	200 (207) (96.6)	57 (59) (96.6)	10 (10) (100)

Nurses performance based on case scenarios during two different tests three weeks apart have been shown in table 2. There was no significant difference between the triage levels estimated by each nurse for the 30 scenarios in the first and second test. Therefore, the intra-rater reliability was in perfect agreement among the nurses (Cohen’s weighted kappa= 0.94, Spearman’s p < 0.001).

Table 2

Nurses performance based on case scenarios during two different tests three weeks apart

Scenarios	Nurse A		Nurse B		Nurse C		Nurse D		Nurse E		Nurse F		Nurse G
	First time	Second time	First time	Second time	First time	Second time	First time	Second time	First time	Second time	First time	Second time	First time	Second time
Level 1 (n=3)	3	3	4	3	3	4	3	3	2	4	3	2	3	3
Level 2 (n=6)	4	5	6	5	5	6	6	6	5	7	6	6	5	4
Level 3(n=12)	14	13	11	13	11	11	11	12	15	14	11	13	10	10
Level 4(n=6)	5	6	6	7	7	5	7	6	5	3	6	5	8	9
Level 5(n=3)	4	3	3	2	4	4	3	3	3	2	4	4	4	4
Total (n=30)	30	30	30	30	30	30	30	30	30	30	30	30	30	30

Comparing the triage by physicians and nurses in 527 patients based on ESI were reported in table 3. Regarding the agreement between the nurses’ assessments for the written scenarios in the first test (inter-rater reliability), there was perfect agreement between the nurses’ assessments and the correct triage scores (Cohen’s weighted kappa for nurse A–G were 0.92, 0.96, 0.91, 0.95, 0.86, 0.96, 0.87, respectively; p < 0.001).

Table 3

Comparing the triage by physicians and nurses in 527 patients based on ESI

		Triage scoring by nurses (in number)
		Level 1	Level 2	Level 3	Level 4	Level 5	Total
Triage scoring by attending physicians(in number)	Level 1	3	1	1	0	0	5
	Level 2	0	73	23	6	3	105
	Level 3	0	11	212	44	7	274
	Level 4	0	3	13	69	11	96
	Level 5	0	0	2	12	33	47
	Total	3	88	251	131	54	527

All 527 patients were evaluated once by the designated triage nurses and then by the attending physicians who were the triage course instructors. The degree of agreement was nearly substantial between them (Cohen's weighted kappa= 0.609 %; p-value < 0.001). Three level 1 patients (33.3%) were correctly rated by the nurses; however, the remaining 6 were incorrectly rated to level 2 and 3. To protect the patients’ safety, they received level 1 care after being triaged by the emergency medicine physicians. For the triage level 2, 3, 4, and 5, there was substantial agreement between the assessment by the attending physicians and nurses (level 2–5 kappa 0.69, 0.77, 0.73, and 0.70, respectively; p < 0.001).

After the retrograde consideration of all 527 patients’ charts and records, and comparing them with the final resource utilization (table 4) and outcome (table 1), the expert panel established the reference retrograde triage level for the patients as outcome-based triage (table 5).

Table 4

Final resource utilization at different triage levels based on number (%)

Resources utilization (number)	Emergency Severity Index (ESI)
	level 1	level 2	level 3	level 4	level 5
	Number (%)
0	0	1 (5%)	6 (2.2%)	13 (13.5%)	31 (66%)
1	0	11 (10.5%)	73 (26.6%)	30 (31.3%)	12 (25.5%)
≥2	5 (100%)	93 (88.5%)	195 (71.2%)	53 (55.2%)	4 (8.5%)

Table 5

Patients’ final triage level

Triage	Emergency Severity Index (ESI)
	level 1	level 2	level 3	level 4	level 5
	Number (%)
Outcome based	9 (1.7)	86 (16.3)	247 (46.9)	133 (25.2)	52 (9.9)
Physicians	5 (0.9)	105 (19.9)	274 (52.0)	96 (18.2)	47 (8.9)
Nurses	3 (0.6)	88 (16.7)	251 (47.6)	131 (24.9)	54 (10.2)

The degree of agreement for the triage done by the nurses and physicians was almost in perfect agreement with the expert panels’ triage level. Cohen’s weighted kappa was 0.966 for the triage by nurses (CI 0.985–0.946, p < 0.001) and 0.813 (CI 0.856–0.769, p < 0.001) for that by the physicians.

Among the study participants, 9 died in the ED.

Only one patient was under-triaged to ESI level 3 by the nurses. Based on the expert panel’s opinion, the triage was incorrect, but the medical management was appropriate due to re-triage by the physician. This patient came to the ED with near-normal vital signs and abdominal pain. After admission, the signs and symptoms of diabetic ketoacidosis, were apparent (ESI level 2). The patient died due to arrhythmia 4 hours later in ED pod A (critical care area).

DISCUSSION

The current study was performed to assess the validity and reliability of triage by nurses using the ESI triage system. The inter-rater and intra-rater reliability, calculated for each nurse, showed perfect agreement. The degree of agreement between the triage done by the nurses and physicians with the expert panels’ triage level assessment was almost perfect.

Overcrowded EDs and limited health care resources can result in prolonged waiting time, interruption of treatment, and poor quality of patient management. Therefore, applying an appropriate triage system is inevitable to prevent resource wasting and delay in patients’ care. At present, five-level triage systems, such as ESI, are widely used as emergency triage tools. ESI is a comprehensive algorithmic triage system, able to predict hospital admission and resource use (10, 18, 20). Some studies have been done using the ESI. A Belgian study demonstrated a good inter-rater agreement (k = 0.72) between triage nurses and reference answers using ESI-v4. The maximum level of disagreement occurred in triage level 2 (21). Another retrospective study done in 780 pediatric patients showed a k value of 0.92 for agreement between the triage level assessed by the nurses and 0.78 for that by the nurses and physicians (22). In a study performed by Baumann et al., the reliability and validity of the ESI-v3 triage algorithm in a pediatric population were evaluated. In the reliability phase, the weighted kappa demonstrated an excellent agreement between the raters (23). During the validity phase, 510 patients were included in the final data analysis. Hospitalization, length of stay in the ED, and resource utilization were strongly associated with the ESI-v3 category. This study demonstrated that the ESI triage algorithm is a reliable and valid triage tool (24). In another study, the overall agreement between the scoring by nurses and the true ESI score was 0.89, and the ESI level correlated to the likelihood of hospital and ICU admission. In a study by Wuerz et al., 493 patients were enrolled, and 159 (32%) patients were hospitalized. The weighted kappa between the physicians and nurses was 0.80 (95% CI = 0.76–0.84). Resource use and hospitalization rates were strongly associated with the triage level. The results demonstrated that this five-level ESI triage instrument is both valid and reliable (25). Based on a study done by Martin et al., the ESI score assigned by nurses did not depend significantly on the level of experience and proper training and understanding of the ESI system; minimally experienced nurses could use it for triage. The overall agreement of the participating nurses with an expert nurse was 0.65 (26). The results are similar to that of our study. Our study showed an almost perfect agreement between the triage level determined by the nurses and retrograde outcome-based triage by the experts’ panel.

It seems that the validity of this study is substantially based on the final utilization of resources and outcome. Personal and systematic errors in implementing the algorithm are inevitable, especially in overcrowded EDs; as this study was performed in the overcrowded ED of a tertiary care teaching hospital, most of the patients demanded a large number of facilities, and their management was not possible without allocating resources conservatively. Some of the patients in level 5 were admitted to the hospital because of a request from another medical staff. On the other hand, some of the patients in level 4 or 5 suffered from lack of family or social support, and the nursing staff and emergency physicians had to admit them to the wards.

In this study, the classification of the length of stay as more or less than 24 hours had an influence on the validity; previous investigations have assessed duration of stay accurately on a quantitative scale, based on minutes or hours (23, 25, 27).

This study was designed for assessing reliability but was likely not suitable for evaluating the validity of nursing triage. So, it is recommended to conduct more studies, considering the exact length of stay, with larger sample size, in different centers involving non-referral hospitals.

CONCLUSIONS

The study findings showed that there was perfect reliability and, overall, almost perfect validity for the triage performed by the studied nurses.