Consistency of triage scores by presenting complaint pre- and post-implementation of a real-time electronic triage decision support tool.

OBJECTIVE: eCTAS is a real-time electronic decision-support tool designed to standardize the application of the Canadian Triage and Acuity Scale (CTAS). This study addresses the variability of CTAS score distributions across institutions pre- and post-eCTAS implementation.
METHODS: We used population-based administrative data from 2016-2018 from all emergency departments (EDs) that had implemented eCTAS for 9 months. Following a 3-month stabilization period, we compared 6 months post-eCTAS data to the same 6 months the previous year (pre-eCTAS). We included triage encounters of adult (≥17 years) patients who presented with 1 of 16 pre-specified, high-volume complaints. For each ED, consistency was calculated as the absolute difference in CTAS distribution compared to the average of all included EDs for each presenting complaint. Pre-eCTAS and post-eCTAS change scores were compared using a paired-samples t-test. We also assessed if eCTAS modifiers were associated with triage consistency.
RESULTS: There were 363,214 (183,231 pre-eCTAS, 179,983 post-eCTAS) triage encounters included from 35 EDs. Triage scores were more consistent (P < 0.05) post-eCTAS for 6 (37.5%) presenting complaints: chest pain (cardiac features), extremity weakness/symptoms of cerebrovascular accident, fever, shortness of breath, syncope, and hyperglycemia. Triage consistency was similar pre- and post-eCTAS for altered level of consciousness, anxiety/situational crisis, confusion, depression/suicidal/deliberate self-harm, general weakness, head injury, palpitations, seizure, substance misuse/intoxication, and vertigo. Use of eCTAS modifiers was associated with increased triage consistency.
CONCLUSIONS: eCTAS increased triage consistency across many, but not all, high-volume presenting complaints. Modifier use was associated with increased triage consistency, particularly for non-specific complaints such as fever and general weakness.

INTRODUCTION

Triage is a fundamental process for the safe and efficient management of patients where health care demands exceed available emergency department (ED) resources. The Canadian Triage and Acuity Scale (CTAS) is the standard used in all Canadian and many international EDs to aid in safely determining the priority by which patients should be assessed. , , , , , , The scale delineates 5 levels of acuity: level 1 (resuscitation), level 2 (emergent), level 3 (urgent), level 4 (less urgent), and level 5 (non‐urgent). , , , CTAS is similar to other triage algorithms including the Australasian Triage Scale and the Manchester Triage Scale (MTS) , which categorize patients based on perceived clinical urgency, but differs from other triage scales such as the Emergency Severity Index, which also incorporates the anticipated number of resources that may be required. CTAS and the MTS also differ from the other triage algorithms by including standardized presenting complaint lists. ,

Despite widespread adoption of CTAS guidelines, triage often relies on subjective judgment, and the process by which CTAS scores are assigned has been shown to vary significantly both within and between EDs. , , , , In 2015, the government of Ontario agreed to fund the development and implementation of a standardized, electronic application to reduce triage variability across the province. eCTAS is a real‐time electronic decisionsupport tool, designed to standardize the application of CTAS scores while respecting the nurse's autonomy in applying their clinical judgement. , The application requires the user to select a presenting complaint from a standardized list of 169 complaints and then displays a CTAS‐based template with complaint‐specific modifiers (eg, vital signs, respiratory distress, hemodynamic status, level of consciousness, pain score, bleeding disorder, and mechanism of injury) to help ensure high risk time‐sensitive conditions are not missed. This assists the user in assigning the appropriate CTAS score in real time.

We previously reported that eCTAS improves both interrater agreement and data accuracy without substantially increasing triage time. In a prospective, observational study including 1491 real‐time triage encounters in 7 EDs, we found interrater agreement was higher after eCTAS implementation compared to pre‐eCTAS (unweighted kappa 0.89 vs 0.63; quadratic‐weighted kappa 0.93 vs 0.79). The use of eCTAS significantly reduced the number of patients over‐triaged (12.0% vs 5.1%; ∆ 6.9, 95% confidence interval [CI] = 4.0‐9.7) and under‐triaged (12.6% vs 2.2%; ∆ 10.4, 95% CI = 7.9‐13.2), and this was consistent across all participating sites. Median triage time was 312 seconds pre‐eCTAS, compared to 347 seconds post‐eCTAS (∆ 35 seconds, 95% CI = 29‐40 seconds).

Given that CTAS is used to define ED case‐mix groups for comparative and benchmarking processes, triage accuracy and consistency are important, but different, considerations especially for regions that incorporate CTAS scores as part of their ED funding model. , , , Triage accuracy refers to how close the triage score is to the “truth” or reference standard, whereas triage consistency is a measure of variability and refers to how reproducible triage scores are within and between EDs. Despite widespread implementation (>90% of EDs) of this government mandated policy, it remains unknown if triage consistency has improved after the introduction of eCTAS across the province.

Goals of this investigation

The primary objective of this study was to assess differences in consistency of CTAS score distributions across institutions before and after e‐CTAS implementation. Secondary objectives were to determine if hospital ED volume, triage process or use of eCTAS modifiers were associated with triage consistency.

METHODS

Study setting and population

This was a retrospective cohort study using population‐based administrative databases from the province of Ontario from January 2016 to December 2018. All hospital EDs in Ontario that had implemented eCTAS for at least 9 months were included. Following a 3‐month stabilization period facilitating ED triage nurses familiarity with and consistent use of eCTAS, we compared data for 6 months post‐eCTAS implementation to the same 6‐month period the previous year (pre‐implementation) to account for potential seasonal variation, patient volume, and case‐mix.

We included triage encounters of adult patients aged 17 years and older if they had one of the following 16 pre‐specified high‐volume, presenting complaints: altered level of consciousness; anxiety/situational crisis; chest pain (cardiac features); confusion; depression/suicidal/deliberate self‐harm; extremity weakness/symptoms of cerebrovascular accident; fever; general weakness; head injury; hyperglycemia; palpitations; seizure; shortness of breath; substance misuse/intoxication; syncope; or vertigo. The 16 presenting complaints were a priori selected by a provincial steering committee to represent commonly encountered, high‐volume conditions that have a minimum allowable CTAS score (eg, none of the included complaints should be assigned a CTAS score of 5). The provincial steering committee consisted of emergency physicians, triage nurses, researchers, and ED educators/managers, including the provincial master CTAS trainer who had 23 years of ED nursing and triage experience. A small committee first suggested the list, and it was approved by the provincial steering committee through debate and consensus.

Prior to the start of the study, CTAS had been the standard triage process for all participating EDs for at least 15 years. Prior to the implementation of eCTAS, all triage nurses completed a mandatory 2‐hour training session consisting of didactic teaching and application practice using an online, interactive, simulated training environment with 10 standardized triage scenarios and real‐time instruction on how to incorporate vital signs and relevant modifiers. For ongoing eCTAS training, all triage nurses have access to a training environment as soon as they receive their access credentials and can continue to access this environment for future training and updates. There is also a help guide that includes videos built into the eCTAS tool. The guide allows nurses to search any aspect of the tool and access a visual full training step‐by‐step guide to the section in question.

Data sources

Data were obtained from province‐wide health administrative databases through Ontario Health (Cancer Care Ontario), an agency of the provincial Government of Ontario responsible for improving healthcare services. Pre‐eCTAS implementation data were obtained from the Canadian Institute of Health Information National Ambulatory Care Reporting System (CIHI‐NACRS). CIHI‐NACRS contains anonymized, abstracted data on all ED visits in Ontario. Post‐eCTAS implementation data were obtained through the provincial eCTAS database. Ontario has universal healthcare coverage for medically necessary care, therefore, these databases contain the majority of healthcare utilization in the province.

The Bottom Line

The Canadian Triage and Acuity Scale (CTAS) is used internationally to triage emergency department patients. The eCTAS is a real‐time electronic decision‐support tool, designed to standardize the application of CTAS scores. In this study of over 350,000 visits to 35 Canadian EDs, eCTAS improved triage consistency for some conditions but not others. Use of compliant‐specific modifiers was associated with increased triage consistency post eCTAS.

The study protocol was approved by the research ethics board at Mount Sinai Hospital in Toronto, Ontario, Canada.

Data analysis

The main exposure variable was the timing of the ED triage encounter, which was categorized as pre‐eCTAS or post‐eCTAS implementation. CTAS distributions were described using frequencies (%) and proportional differences were compared pre‐ and post‐eCTAS using chi‐square statistics and presented as deltas (∆ = post‐eCTAS % − pre‐eCTAS %) with 95% CIs. To determine consistency, the overall CTAS distribution for all 35 included EDs was first calculated for each presenting complaint, pre‐ and post‐eCTAS implementation. Then, the absolute difference in CTAS distribution for each presenting complaint was calculated for each hospital, pre‐ and post‐eCTAS, resulting in a pre‐eCTAS change score and a post‐eCTAS change score.

Figure 1 includes the data used to calculate the pre‐eCTAS change score for the presenting complaint of “shortness of breath” for one of the included high‐volume sites. The overall CTAS distribution for the presenting complaint of “shortness of breath” pre‐eCTAS was 3.3% for CTAS 1, 41.4% for CTAS 2, 49.3% for CTAS 3, 5.8% for CTAS 4, and 0.2% for CTAS 5. The pre‐eCTAS distribution for the same presenting complaint (shortness of breath) for all included hospitals was 2.9% for CTAS 1, 51.7% for CTAS 2, 43.1% for CTAS 3, 2.1% for CTAS 4, and 0.2% for CTAS 5 prior to eCTAS implementation. To calculate the pre‐eCTAS change score for that hospital, we summed the absolute difference (0.4 + 10.3 + 6.2 + 3.7 + 0), resulting in a pre‐eCTAS change score of 20.6%, which was rounded to 0.21 removing the percentage. The larger the ED change score, the more deviant the individual hospital CTAS distribution was from the overall CTAS distribution for all included sites. These change scores (n = 1120) were calculated for all 35 EDs, for all 16 presenting complaints, pre‐ and post‐eCTAS implementation. A paired‐samples t‐test was used to compare the pre‐eCTAS and post‐eCTAS change scores for each complaint, with each hospital acting as their own control. Mean pre‐eCTAS and mean post‐eCTAS changes scores are the average of the individual hospital change scores for each presenting complaint.

FIGURE 1

Data used to calculate the pre‐eCTAS change score for the presenting complaint of “shortness of breath” for one of the included high‐volume sites

Consistency ratios for the change score were also calculated (pre‐eCTAS change score/post‐eCTAS change score) for each hospital by presenting complaint, with a value >1.0 indicating an increase in triage consistency with the overall CTAS distribution for all 35 included EDs. Mean consistency ratios are the average of the individual hospital consistency ratios for each presenting complaint. Analysis of variance (ANOVA) was used to compare consistency ratios by hospital ED volume (low volume <30,000 annual ED visits; medium volume 30,000–49,999 annual ED visits; high volume 50,000–84,999 annual ED visits, and very high volume >85,000 annual ED visits). An independent samples t‐test was used to compare consistency ratios by the ED triage process prior to the implementation of eCTAS (paper‐based triage vs an electronic triage process).

We also captured the use of complaint‐specific clinical modifiers (ie, vital signs, respiratory distress, hemodynamic status, level of consciousness, pain score, bleeding disorder, and mechanism of injury) for each triage encounter post‐eCTAS implementation. Pearson product‐moment correlation coefficients were used to estimate the strength and direction of association between the use of modifiers and post‐eCTAS consistency change scores. Data analyses were performed using Stata 16.0 (StataCorp LP, College Station, TX).

RESULTS

Thirty‐five hospital EDs met the inclusion criteria of eCTAS use for at least 9 months. There were eight (22.8%) low volume sites, eight (22.8%) medium volume sites, 13 (37.1%) high volume sites, and six (17.1%) very high volume sites. Prior to eCTAS, 15 (42.9%) EDs used a paper‐based triage process, and 20 (57.1%) EDs used an electronic triage system. Of the 363,214 (183,231 pre‐eCTAS, 179,983 post‐eCTAS) triage encounters included, mean age (55.4 vs 55.6 years) and proportion of female patients (51.5% vs 51.8%) were similar pre‐ and post‐eCTAS implementation (Table 1).

TABLE 1

Cohort descriptive statistics for 363,214 triage encounters from 35 hospital emergency departments for 16 included presenting complaints

	Pre‐eCTAS			Post‐eCTAS
Presenting Complaint	n	Mean Age	Female	n	Mean Age	Female
Altered Level of Consciousness	4,289	64.0	47.9%	3,913	64.9	48.8%
Anxiety/Situational Crisis	9,212	38.0	53.6%	8,578	38.3	54.2%
Chest Pain (Cardiac Features)	35,012	55.9	49.5%	36,744	56.4	49.8%
Confusion	3,361	73.3	49.4%	3,053	74.3	48.8%
Depression/Suicidal/Deliberate Self Harm	11,108	34.5	52.6%	11,467	34.3	51.5%
Extremity Weakness/Symptoms of CVA	5,356	68.8	50.7%	5,545	67.8	51.4%
Fever	10,642	48.1	51.5%	9,553	48.8	51.6%
General Weakness	17,409	68.5	54.0%	18,821	68.4	54.7%
Head Injury	13,282	51.8	51.3%	13,818	52.4	52.6%
Hyperglycemia	1,856	55.6	50.1%	1,907	54.7	48.7%
Palpitations/Irregular Heartbeat	8,708	57.4	55.3%	8,784	55.7	56.0%
Seizure	4,573	42.6	40.6%	3,889	43.3	41.1%
Shortness of Breath	30,855	62.0	53.2%	28,893	62.0	53.3%
Substance Misuse/Intoxication	6,393	38.8	33.6%	5,966	38.9	32.3%
Syncope/Pre‐syncope	9,171	57.6	53.3%	9,294	58.1	53.9%
Vertigo	12,004	57.9	59.2%	9,758	58.1	60.6%
Grand Total	183,231	55.4	51.5%	179,983	55.6	51.8%

CVA, cerebrovascular accident; eCTAS, electronic Canadian Triage and Acuity Scale.

Figure 2 shows the distribution of CTAS scores pre‐ and post‐eCTAS implementation for the presenting complaint of “shortness of breath.” The distribution curves for the remaining 15 presenting complaints can be found in the supplementary appendix.

FIGURE 2

Distribution of CTAS scores pre‐ and post‐implementation of eCTAS by presenting complaint “Shortness of Breath”

Table 2 presents pre‐ and post‐eCTAS consistency change scores for each of the 16 presenting complaints. Compared to pre‐eCTAS, triage scores were more consistent with the overall CTAS distribution after the implementation of eCTAS for six (37.5%) presenting complaints: chest pain (cardiac features) (P < 0.001), extremity weakness/symptoms of cerebrovascular accident (P < 0.001), fever (P < 0.001), shortness of breath (P < 0.001), syncope (P = 0.02), and hyperglycemia (P = 0.03). Triage consistency was similar pre‐ and post‐eCTAS for altered level of consciousness, anxiety/situational crisis, confusion, depression/suicidal/deliberate self‐harm, general weakness, head injury, palpitations, seizure, substance misuse/intoxication, and vertigo.

TABLE 2

Pre‐ and post‐eCTAS consistency estimates for 16 presenting complaints

	Pre‐eCTAS	Post‐eCTAS
Presenting complaint	Change Score	Change Score	Delta (95% CI)	t
Chest pain (cardiac features)	0.27	0.01	0.26 (0.18 to 0.34)	6.55
Extremity weakness/symptoms of CVA	0.41	0.29	0.12 (0.06 to 0.17)	4.28
Fever	0.37	0.27	0.10 (0.06 to 0.14)	4.70
Shortness of breath	0.30	0.22	0.08 (0.05 to 0.11)	5.01
Hyperglycemia	0.43	0.36	0.07 (0.01 to 0.13)	2.24
Syncope/pre‐syncope	0.36	0.30	0.06 (0.01 to 0.12)	2.47
Depression/suicidal/deliberate self‐harm	0.36	0.31	0.05 (−0.02 to 0.11)	1.49
Seizure	0.40	0.35	0.05 (−0.05 to 0.14)	0.98
Anxiety/situational crisis	0.28	0.25	0.03 (−0.03 to 0.09)	1.05
Head injury	0.25	0.22	0.03 (−0.01 to 0.07)	1.72
General weakness	0.26	0.24	0.02 (−0.02 to 0.06)	1.01
Palpitations/irregular heartbeat	0.32	0.30	0.02 (−0.04 to 0.09)	0.79
Altered level of consciousness	0.38	0.36	0.02 (−0.09 to 0.13)	0.42
Substance misuse/intoxication	0.34	0.33	0.01 (−0.04 to 0.07)	0.42
Confusion	0.32	0.31	0.01 (−0.10 to 0.12)	0.21
Vertigo	0.26	0.29	−0.03 (−0.09 to 0.02)	–1.22

CVA, cerebrovascular accident; eCTAS, electronic Canadian Triage and Acuity Scale.

Figure 3 displays the average consistency ratios (pre‐eCTAS/post‐eCTAS change scores) for the 16 presenting complaints included in this study. All but one presenting complaint (vertigo) had consistency ratios >1.0, indicating increased consistency with eCTAS. Consistency ratios for each presenting complaint broken down by each of the 35 included EDs can be viewed in the supplementary appendix. Consistency ratios were similar across hospital ED volumes (low, medium, high, and very high volume) and previous triage processes (paper‐based vs electronic) for most presenting complaints. Consistency ratios were higher for EDs transitioning from paper‐based triage to eCTAS compared to EDs transitioning from an electronic triage system to eCTAS for the presenting complaints of chest pain (cardiac features) (93.7 vs 38.8; ∆ 54.9; 95% CI = 8.9–101.1) and head injury (1.8 vs 1.1; ∆ 0.7; 95% CI = 0.2–1.3).

FIGURE 3

Average consistency ratios (pre‐eCTAS/post‐eCTAS) for 16 presenting complaints

We found a statistically significant correlation between post‐eCTAS consistency change scores and use of modifiers for 12 of the included presenting complaints, particularly for non‐specific presenting complaints that are applicable to many different medical conditions (Table 3). The use of complaint‐specific modifiers was highly correlated with increased consistency post‐eCTAS for confusion, fever, general weakness, head injury, shortness of breath, and vertigo; moderately correlated with increased consistency post‐eCTAS for anxiety/situational crisis, depression/suicidal/deliberate self‐harm, hyperglycemia, palpitations/irregular heartbeat, and syncope/pre‐syncope; and weakly correlated with increased consistency for altered level of consciousness. Figure 4 displays the post‐eCTAS change scores and use of modifiers for “shortness of breath” for all 35 included EDs. Similar figures depicting the association between post‐eCTAS changes scores and use of modifiers for the other presenting complaints can be viewed in the supplementary appendix. There was no discernable pattern for low modifier use, based on hospital volume, previous triage method (paper‐based vs electronic) or presenting complaint.

TABLE 3

The association between post‐eCTAS consistency change scores and use of modifiers for 16 presenting complaints

Presenting complaint	Post‐eCTAS change score	Use of modifiers (%)	Pearson correlation	Strength of association
Shortness of breath	0.22	84.13	–0.85	High
Fever	0.27	89.52	–0.83	High
Confusion	0.31	67.21	–0.81	High
Head injury	0.22	77.77	–0.75	High
General weakness	0.24	71.34	–0.74	High
Vertigo	0.29	61.12	–0.73	High
Anxiety/situational crisis	0.25	74.32	–0.68	Moderate
Hyperglycemia	0.36	80.07	–0.66	Moderate
Syncope/pre‐syncope	0.30	72.35	–0.64	Moderate
Palpitations/irregular heartbeat	0.30	80.09	–0.60	Moderate
Depression/suicidal/deliberate self‐harm	0.31	83.37	–0.55	Moderate
Altered level of consciousness	0.36	78.20	–0.39	Weak
Seizure	0.35	77.09	–0.21	n/a
Extremity weakness/symptoms of CVA	0.29	61.26	0.10	n/a
Chest pain (cardiac features)	0.01	81.31	–0.04	n/a
Substance misuse/intoxication	0.33	72.54	–0.02	n/a

CVA, cerebrovascular accident; eCTAS, electronic Canadian Triage and Acuity Scale; n/a, not applicable.

FIGURE 4

Post‐eCTAS change score and use of modifiers by presenting complaint “Shortness of Breath” for 35 hospitals across Ontario.

LIMITATIONS

This study has several limitations. As of February 2020, eCTAS has been implemented in 114 (90%) EDs across the province. However, at the time of this study, only 35 EDs had implemented eCTAS for at least 9 months (3‐month stabilization period and at least 6 months of triage data using eCTAS) required to be included in this evaluation. It remains unknown if the overall consistency reported in this analysis is representative of all hospital EDs that have implemented eCTAS. The 16 presenting complaints included in this study were selected by a provincial steering committee to represent commonly encountered, high‐volume conditions that have a minimum allowable CTAS score (eg, none of the included complaints should be assigned a CTAS score of 5). It is possible that triage consistency may be different for other presenting complaints. It is also possible that some of the increased consistency observed post‐eCTAS implementation may be explained by the reduction in CTAS 5 use post‐eCTAS. Another limitation is the uncertainty in the underlying assumption that the distribution of presenting complaint severities was similar across the 35 participating sites pre‐ and post‐eCTAS implementation. We did not record the triage experience of the nurses included in this study, so we are unable to comment how this may have influenced triage consistency. We did not include pediatric (<17 years) triage encounters, so it is possible that our results are not generalizable to that age demographic. Finally, there are no known benchmarks for what constitutes a clinically important improvement in triage consistency.

DISCUSSION

This study evaluated the consistency of CTAS score distributions by presenting complaints 6 months pre‐ and post‐eCTAS implementation in EDs across Ontario. We found that a standardized, electronic approach to performing triage assessments increased consistency in CTAS scores for some high‐volume presenting complaints, but had a mixed effect without indication of reducing consistency. We also found the use of complaint‐specific modifiers was associated with increased triage consistency post‐eCTAS, particularly for some non‐specific presenting complaints such as shortness of breath, fever, and general weakness.

Triage decisions are usually made under conditions of uncertainty. After a brief clinical assessment, often based on incomplete or ambiguous information, triage nurses must quickly assign a CTAS score based on the perceived acuity of the patient. Historically, triage decisions were generally subjective, influenced by triage experience, patient volume, and current resource availability. Triage accuracy, consistency, and timeliness may influence patient outcomes. Under‐triage may contribute to delays in time‐sensitive interventions and lead to potentially avoidable clinical deterioration and misdiagnosis. Over‐triage, or labeling of patients with non‐urgent presentations to high acuity designations, may lead to overutilization of scarce hospital resources and influence physician decisions, including hospital admission. , , , , , , , , ,

Not all ED patients require a thorough and comprehensive triage. Patients who present with serious, life‐threatening illness or injury (eg, cardiac arrest) can be quickly assessed and triaged based on their presenting complaint and general appearance. However, for the majority of patients presenting to the ED, more information is required before a CTAS score can be assigned. In these cases, complaint‐specific modifiers (eg, vital signs, respiratory distress, hemodynamic status, level of consciousness, pain score, bleeding disorder, and mechanism of injury) help determine the severity of the presenting complaint to assign the most appropriate CTAS score. , , , Based on the presenting complaint selected, eCTAS presents the relevant modifiers to the triage nurse and then electronically documents the modifier(s) selected and CTAS score. We found the use of modifiers was associated with increased triage consistency post‐eCTAS, particularly for non‐specific complaints where patients may present with a wide spectrum of illness severity, such as shortness of breath, fever, and general weakness. For example, a patient who presents to the ED with “shortness of breath” with normal vital signs who appears well may be appropriately triaged as CTAS 4, whereas another patient with the same presenting complaint (shortness of breath) who is asthmatic with moderate respiratory distress and a fever would be triaged as CTAS 2. The modifiers and computer‐based prompts (eg, standard deviation from the norm for age‐specific vital signs) in eCTAS help guide the clinical decisionmaking of the triage nurse.

Previously, Lin and Worster suggested objective reliance on existing modifiers may greatly improve triage consistency and accuracy compared to subjective reliance on experience or intuition alone. Similarly, Brown et al, tested interrater reliability and accuracy of CTAS scores for 20 mental health scenarios and found accuracy improved when triage nurses used complaint‐specific modifiers. In contrast, nurses who assigned the correct score <40% of the time were less likely to use complaint‐specific modifiers or avoided their use altogether. The authors suggested the additional cues provided by the modifiers may help prompt triage nurses to consider higher acuity or risk presentations, encouraging a more detailed assessment by the nurse to support clinical decisionmaking. Although we can only speculate why some nurses did not enter modifier data, it seems likely to be related to perceived process time, improper education or triage efficiency. Unknown system‐level factors may also be a driving force behind these findings and future research should attempt to elucidate factors associated with use of modifiers.

CONCLUSIONS

In our study, a standardized, electronic approach to performing triage assessments increased consistency in CTAS scores for some high‐volume presenting complaints, but had a mixed effect without indication of reducing consistency. Modifier use varied substantially between hospitals and presenting complaints. ED sites that were least consistent with the overall CTAS distribution had the lowest use of modifiers across all presenting complaints. Findings from this study may be useful to optimize the use of eCTAS in EDs that failed to show improvements in consistency and guide triage nurse education. Regular audit and feedback and a targeted educational curriculum clarifying the importance of modifier selection should improve triage consistency, particularly for some non‐specific presenting complaints such as shortness of breath, fever, and general weakness.

CONFLICTS OF INTEREST

At the time of the study, SS, TA, JM, BF and NM were paid employees of Ontario Health (Cancer Care Ontario). HO is a paid advisor to the Ministry of Health of Ontario and in that capacity has provided leadership to the eCTAS project. SLM, CT, BB, LT, KG, AW, TA, MB and GG state no conflict of interest and have received no payment in preparation of this manuscript.

AUTHOR CONTRIBUTIONS

SLM, BB, HO, and NM conceived the study and designed the protocol. SLM, BB, SS, and GG supervised the conduct of the study. SLM, SS, TA, and BF managed the data, including quality control. SLM, CT, LT, AW, TA, and GG provided advice on study design and data analysis. BB and NM chaired the data oversight committee. BB, HO, KG, JM, AW, TA, MB, and GG provided clinical advice on study interpretation. SLM drafted the manuscript, and all authors contributed substantially to its revision. SLM takes responsibility for the paper as a whole.

Supporting Information.

Click here for additional data file.