Differences in emergency nurse triage between a simulated setting and the real world, post hoc analysis of a cluster randomised trial.

OBJECTIVES: In the TRIAGE trial, a cluster randomised trial about diverting emergency department (ED) patients to a general practice cooperative (GPC) using a new extension to the Manchester Triage System, the difference in the proportion of patients assigned to the GPC was striking: 13.3% in the intervention group (patients were encouraged to comply to an ED or GPC assignment, real-world setting) and 24.7% in the control group (the assignment was not communicated, all remained at the ED, simulated setting). In this secondary analysis, we assess the differences in the use of the triage tool between intervention and control group and differences in costs and hospitalisations for patients assigned to the GPC.
SETTING: ED of a general hospital and the adjacent GPC. PARTICIPANTS: 8038 patients (6294 intervention and 1744 control).Primary and secondary outcome measures proportion of patients with triage parameters (reason for encounter, discriminator and urgency category) leading to an assignment to the ED, proportion of patients for which the computer-generated GPC assignment was overruled, motivations for choosing certain parameters, costs (invoices) and hospitalisations.
RESULTS: An additional 3.1% (p<0.01) of the patients in the intervention group were classified as urgent. Discriminators leading to the ED were registered for an additional 16.2% (p<0.01), mainly because of a perceived need for imaging. Nurses equally chose flow charts leading to the ED (p=0.41) and equally overruled the protocol (p=0.91). In the intervention group, the mean cost for patients assigned to the GPC was €23 (p<0.01) lower and less patients with an assignment to the GPC were hospitalised (1.0% vs 1.6%, p<0.01).
CONCLUSION: Nurses used a triage tool more risk averse when it was used to divert patients to primary care as compared with a theoretical assignment to primary care. Outcomes from a simulated setting should not be extrapolated to real patients. TRIAL REGISTRATION NUMBER: NCT03793972. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Post hoc analysis of the TRIAGE trial in which patients were diverted from an emergency department to primary care.

This cluster randomised trial offered the opportunity to assess the difference in use of the same triage tool between real-world triage and simulated triage.

This trial was not designed to measure the outcomes of this post hoc analysis.

This is a single-centre study in the context of the Belgian healthcare system, potentially limiting the generalisability of the results.

Introduction

Worldwide, initiatives for collaboration between emergency departments (EDs) and sites for primary out-of-hours care (called general practitioners cooperatives or GPCs) have been installed to improve patient care, staff satisfaction, and to reduce crowding at the ED. In these initiatives, both services are located on the same site or in proximity of each other. They implement triage, defined as the sorting out and classification of patients or casualties to determine priority of need (urgency classification) and proper place of treatment (assignment to ED or GPC).1 The efficiency of such a triage has been studied in simulated settings (written case scenarios or live cases), retrospective case studies or in prospective interventional studies describing the situation before and after implementation of triage.2–5 A comparison of live cases versus paper case scenarios revealed a lower intraclass correlation for urgency classification in live triage assessments as compared with paper cases. Paper case scenarios generally receive lower triage scores (less urgent) than live cases.6 One of the few well validated and widely implemented triage systems is the Manchester Triage System (MTS). A multicentre prospective observational study revealed a sensitivity of 0.47–0.87 and a specificity of 0.84–0.94.7 In a meta-analysis, the agreement regarding written scenario assessment was substantial while it was almost perfect for assessment of real live cases.8

In 2018, a Cochrane review concluded that the current evidence concerning primary care professionals providing non-urgent care in EDs was insufficient to draw conclusions for practice or policy.9 It is unknown how well the efficiency of triage observed in simulated settings compares to everyday practice (the real world).10 Some studies found that the presence of a general practitioner (GP) leads to an improvement in the effectiveness and quality of care at the ED and is less expensive than the usual care method, as GPs use fewer resources than do usual ED staff.11–14 Again, it is unknown whether a simulated experiment can predict such a cost reduction.

The TRIAGE (Triaging and Referring in Adjacent General and Emergency Departments) trial studied the efficiency and safety of a newly developed extension (called eMTS) to the original MTS assigning low risk patients to ED or GPC. This trial was executed during weekends and bank holidays (from here on we refer to weekends and bank holidays as weekends). An eMTS triage results in three parameters: reason for encounter (presentational flow chart, eg, ‘abdominal pain’), discriminator (property of the complaint, eg, ‘mild pain’) and urgency category (ranging from one or to five). Each combination of parameters is linked to an ED or GPC assignment. Weekends were randomly allocated to the intervention group (patients were encouraged to follow their assignment to ED or GPC) or the control group (the assignment was not communicated, all remained at the ED, a simulated setting). This intervention led to the safe diversion of 9.5% of the included patients.15 In the intervention group, 838/6374 patients (13.3%) were assigned to the GPC, in the control group this was almost twice as much: 431/1744 (24.7%). We hypothesise that this remarkable difference was caused by a difference in use of the triage tool between intervention and control weekends.

The objective of this secondary analysis was to assess the differences in the use of the triage tool between the intervention and the control group of the TRIAGE trial. As a secondary objective, the difference in hospitalisations and costs between the intervention and control group for low urgency patients was studied.

Methods

We refer to the original article on the TRIAGE trial for details on the participants, the intervention and the study design.15 Another paper explores the characteristics of patients non-compliant to a GPC advice.16 Below we only describe those aspects important for the current article.

Study design

This study is a post hoc analysis of a cluster randomised controlled trial (RCT) executed from 1 March 2019 to 30 December 2019. This trial was randomised into 10 control and 27 intervention weekends. The intervention was triage by a nurse using a new extension to the MTS assigning low-risk patients to the GPC and all other patients to the ED. During intervention weekends, patients were encouraged to follow this assignment while it was not communicated during control weekends (all patients remained at the ED). In the intervention group, the triage had immediate consequences which is comparable to real world circumstances. The virtual triage in the control group had no consequences which is comparable to a simulated setting with live cases. The primary outcome was the proportion of patients assigned to and handled by the GPC during intervention weekends (9.5%, 95% CI 8.8% to 10.3%). The trial was randomised for the financial analysis (under review) and the secondary outcome: the proportion of patients assigned to the GPC during intervention (13.3%, 95% CI 12.5% to 14.2%) and control weekends (24.7%, 95% CI 22.7% to 26.8%).

Study tool

The study tool for the TRIAGE trial was based on the MTS (V.3.6), a validated tool for prioritisation.7 17 When using the MTS, the nurse starts by choosing a presentational flow chart(eg, chest pain). A flow chart consists of a list of terms called discriminators (eg, mild pain), the presence of which has to be checked top-down.

The eMTS adds site of treatment (ED or GPC) to this system in 42/53 flow charts, the remaining nine flow charts always lead to the ED. In 18 flow charts, additional discriminators led patients in urgency categories four or five to an assignment to the ED (when at least one supplementary discriminator is applicable) or GPC (when no supplementary discriminators are applicable). These additional discriminators were either already present in the original MTS (eg, mild pain) or newly introduced for the eMTS (eg, ‘right lower abdominal pain’ in the presentational flow chart for abdominal pain). See figure 1 for an example. In 26 flow charts, urgency category four and/or five were directly linked to an assignment to the GPC or ED. In all flow charts with possible assignment to the GP, the nurses had the option to choose for the discriminator ‘GP Risk’ linked to an assignment to the ED. ‘GP Risk’ was defined as an unspecified risk to assign the patient to the GPC according to the opinion of the triaging nurse, or because of age less than 3 months. Nurses were instructed to use this parameter when they thought there was a good reason to keep the patient at the ED, but this reason was not specified in the eMTS.

Figure 1

Example of the Manchester Triage System presentational flow chart for abdominal pain in children with the studied extension.29 ED, emergency department; GP, emergency department; GPC, general practice cooperative; PV, per vaginam. Image based on Emergency Triage: Mackway-Jones K, Marsden J, Windle J, Manchester Triage Group. Emergency triage. Third edition. Ed, 2014, ISBN 9781118299067 p. 66 with kind permission.

The eMTS was built into the ED’s computer decision support system (E.care ED 4.1). After choosing a flow chart and a discriminator, this system showed the advice ‘Assign to GPC’ when applicable. Afterwards, the nurse had to register the final assignment he/she had given (intervention weekends) or considered the most appropriate (control weekends). Nurses working at the study ED with a degree in emergency medicine and at least 1 year of experience at the study hospital were allowed to triage, they all participated in the study. These nurses followed a training on using the eMTS, patient communication skills focusing on refusal of the assignment and the study protocol.

Participants

Patients with an available national insurance number triaged by a nurse at the ED were included. Patients already admitted to the study hospital, those arriving by an ambulance staffed with a doctor or nurse, and patients referred to the ED by a doctor or nurse were excluded from the TRIAGE trial. Patients with a missing final assignment were also excluded from the current study. For the cost analysis, patients who were hospitalised were excluded because their invoice does not reflect their true costs at the ED, as it was not possible to differentiate their outpatient costs at the ED and the cost of their subsequent hospitalisation.

Setting

ED of a general hospital (AZ Monica, Deurne) with an annual census of 37 000 patients, and the adjacent GPC with an annual census of 10 000 patients (open during weekends only). Before 2019, ED triage in Belgium only involved urgency classification for prioritisation; patients were only assigned to primary care in experimental settings.18–20

Data collection

The following study tool parameters were collected: MTS flow chart (52 flow charts); MTS urgency level (1–5), chosen MTS discriminator (200 terms), computer-generated assignment (ED or GPC) and nurse-selected assignment (ED or GPC). Due to limitations of the used software at the ED, the collection of the discriminators was incomplete. When the nurse chose an original MTS discriminator (eg, ‘mild pain’), it was registered correctly. But when the nurse wanted to choose a specific newly introduced eMTS discriminator (eg, ‘right abdominal pain’) or the non-specific newly introduced discriminator ‘GP Risk’, they had to click the option ‘Not assigned to the GPC because of a specific newly introduced eMTS discriminator or GP Risk’. Afterward they were asked to write down a free-text motivation why they had chosen this term, but that field was not obligatory. In the results section, the specific newly invented eMTS discriminators and ‘GP Risk’ are reported together as additional discriminators in the eMTS linked to the ED. For the cost analysis, the total invoice cost and costs per cost category (physician fees, medical imaging, technical procedures, non-refundable items and medication) were studied. The number of hospitalisations was calculated using these invoices.

After reading all free-text motivations concerning overruling the protocol and the choice of additional discriminators in the eMTS linked to the ED, authors VV and SM independently divided them into categories. Afterwards, authors VV and SM reached consensus on the categories to use and the classification of all free texts.

Data were collected using iCARE data (Improving Care And Research Electronic Data Trust Antwerp), a Belgian database for out-of-hours care.21 22 The data for the costs were obtained directly from the financial department of the studied ED and GPC and were linked to the medical data based on admission time, sex and ZIP-code.

Outcomes

The first outcome of this study was the proportion of patients with study tool parameters leading to an assignment to the ED in the intervention and control weekends. The following parameters were studied: reason for encounter registered in the MTS as a presentational flow chart (9 flow charts always leading to ED vs 43 with possible assignment to the GPC), urgency category (1–3 always lead to ED, 4 or 5 might lead to the GPC) and discriminator (1197 discriminator-flow chart pairs linked to the ED, 175 to the GPC). The motivations for choosing additional discriminators in the eMTS linked to the ED and for overruling were studied using free-text fields. This outcome is the proportion of patients for which the nurse overruled a computer-generated assignment to the GPC in favour of the ED. A subanalysis for the most frequently used presentational flow chart (limb problems) was made using the same outcomes.

As a proxy for the need of specialist/secondary care in the group of patients assigned to the GPC, the number of hospitalisations and costs were studied. The total costs were studied as well as the different cost categories. The invoices were divided into the share of the invoice paid for by the patient, and the share of the cost refunded by the national health insurance (government costs).

Analysis

A χ2 test was used to compare proportions between intervention and control weekends. A two-sample Wilcoxon rank-sum (Mann-Whitney) test was used to assess the difference in the mean costs and distribution of costs for the intervention and control group. Density histograms were created to illustrate the skewness of the cost data. Data were analysed using JMP Pro V.15.0 (SAS institute) and Stata V.17.0 (StataCorp). The significance level for all tests was set at 0.05.

Patient and public involvement statement

A lay person volunteering at the ED of a hospital not participating in this study was involved in the study design, she gave advice about the study protocol and tool. An advisory board with stakeholders from EDs, GPCs and universities gave advice about the study design and discussed the interim analysis and gave feedback on the results.

Results

Population

In the TRIAGE trial, 9964 patients were assessed for eligibility, of which 1806 patients were excluded mainly because they were already triaged by a healthcare professional prior to arrival in the ED.15 For this paper, patients with a missing final assignment (N=338/8158, 4%) were also excluded leading to a population of 8038 (6294 intervention and 1744 control). The baseline characteristics of the patients in the intervention and in the controlgroup were similar (see online supplemental table 1).

Influence of the intervention on the selection of study tool parameters

See figure 2 for a summary. Nurses equally chose flow charts always leading to the ED for the control and the intervention group (2,5% vs 2.1%, p=0.41). However, an additional 3.2% of the patients in the intervention group were classified in a higher urgency category (mandatory assignment to the ED) as compared with the control group (p=0.02). For an additional 17.3% of the patients within urgency categories four and five, a discriminator leading to the ED was selected in the intervention group (p<0.01). Among those discriminators leading to the ED, newly introduced eMTS discriminators were selected more frequently than those already available in the original MTS (85.0% vs 15.0%) regardless of the allocation to intervention or control weekends (p=0.78).

Figure 2

Influence of the intervention on the selection of study tool parameters. ED, emergency department; GPC, general practice cooperative; ICT, Information and Communications Technology.

The motivation for choosing an additional discriminator in the eMTS linked to the ED was registered for 347/2207 (15.7%, see table 1) in the intervention and 39/534 (7.3%) in the control group. The most frequent motivation was the presence of a predefined eMTS discriminator, mainly the need for imaging according to the nurse. Out of the 202 patients with imaging as a motivation, 160 (79%) had a radiology cost on their invoice.

Table 1

Motivations for choosing an additional discriminator in the eMTS linked tothe ED (N=347) in the intervention group

Motivation	N	%
Presence of a predefined eMTS discriminator	243	63
Need for imaging according to the nurse	173	45
Need for a technical procedure according to the nurse	60	16
Physical observation by the nurse (eg, right abdominal pain)	4	1
Age (<6 months or in some flow charts >75 years)	5	1
History	1
Medical reason not predefined in the eMTS	127	33
Need for imaging according to the nurse	29	8
Recent medical care for the same problem	27	7
Physical observation by the nurse (eg, ‘patient looks bad’)	25	6
Patient’s history or current anamnesis contain worrisome elements	21	5
Need for a technical procedure according to the nurse	9	2
Other medical reasons	7	2
Mental illness such as anxiety disorder	5	1
Need for laboratory testing according to the nurse	4	1
Organisational	10	3
Other motivations	6	2

ED, emergency department; eMTS, extended Manchester Triage System.

Influence of the intervention on overruling the protocol

The nurses overruled the automated eMTS assignment of the software to the GPC in favour of the ED for 3.9% of the patients, there was no significant difference between intervention and control weekends (p=0.91). No motivations for overruling were given during control weekends while this information was available for 95/147 (64.6%) during intervention weekends. See table 2 for these motivations. The vast majority of these motivations was either organisational or medical.

Table 2

Motivations for over ruling the automated eMTS assignment (N=95)

Motivation	N	%
Organisational:	48	51
The emergency physician has already started helping the patient	9	9
The patient has a minor problem that can be resolved directly in the triage room	8	8
A companion needs ED care	8	8
Other organisational motivation	7	7
The GP refuses to see the patient	6	6
Patient arrived by ambulance	5	5
It is currently quiet at the ED	5	5
Medical reason	34	36
Communication problem	13	14

ED, emergency department; eMTS, extension Manchester Triage System; GP, general practitioner.

Influence of the intervention on the use of the flow chart for limb problems

There was no significant difference in the proportion of patients triaged with the flow chart limb problems (513/1740 or 29.5% vs 1691/6238 or 27.1%, p=0.05) or the selection of the three highest urgency categories (76/513 or 14.8% vs 252/1439 or 14.9%, p=0.96). There was, however, a marked difference in the selection of discriminators within urgency categories four and five. The additional discriminators in the eMTS linked to the ED (nurse perceived need for medical imaging, baby below 3 months or the unspecified GP Risk) were selected for an additional 15.2% of the patients (273/437 or 62.5% vs 1049/1351 or 77.7%, p<0.01) in the intervention group. The only original discriminator in the MTS linked to the ED for this presentational flow chart was ‘deformity’ defined in the MTS as abnormal angulation or rotation. ‘Deformity’ was selected equally in both groups (51/437 or 11.7% vs 180/1351 or 13.3%, p=0.37).

Influence of the intervention on hospitalisations in the study Hospital for patients assigned to the GPC

During intervention weekends,13 (1.0%) out of the 838 patients assigned to the GPC were hospitalised. During control weekends this proportion was significantly higher: 20/431 (1.6%, p<0.01).

Influence of the intervention on the costs for patients and government for patients assigned to the GPC

For this analysis, 1146 patients were included. Four patients were excluded because they had an unlikely low invoice, 33 were excluded because they were hospitalised, and the invoice was missing for 86 patients. Online supplemental appendix figures 1 and 2 illustrate the skewed distribution of the costs and indicates that the costs are more concentrated around zero during intervention weekends. The mean total cost during intervention weekends was €56 while this was €79 during control weekends (p<0.01). The Wilcoxon rank-sum test confirms that the distribution of costs among patients assigned to the GPC differs between intervention and control weekends. This difference was mainly driven by decreased use of medical imaging and technical procedures (p<0.01) during intervention weekends. See table 3 for details.

Table 3

Costs for patients assigned to the GPC

Cost	Intervention(n=760)	Control(n=386)	Total(n=1146)	P value Wilcoxon rank-sum test
Total cost (€)
Mean (SD)	55.58 (40.07)	78.74 (70.96)	63.38 (53.64)	<0.01
Median (IQR)	47.69 (39.22–52.17)	50.73 (48.73–92.00)	48.73 (39.22–52.53)
Physician fees (€)
Mean	46.92 (12.86)	46.46 (8.74)	46.77 (11.64)	0.49
Median	39.22 (39.22–52.17)	48.73 (42.85–48.73)	48.73 (39.22–48.73)
Medical imaging (€)
Mean	4.17 (23.03)	12.55 (36.63)	6.99 (28.61)	<0.01
Median	0.00 (0.00–0.00)	0.00 (0.00–0.00)	0.00 (0.00–0.00)
Technical procedures (€)
Mean	3.53 (15.05)	15.64 (42.59)	7.61 (28.16)	<0.01
Median	0.00 (0.00–0.00)	0.00 (0.00–22.78)	0.00 (0.00–0.00)
Non-refundable items (€)
Mean	0.30 (1.64)	1.75 (4.43)	0.79 (2.98)	<0.01
Median	0.00 (0.00–0.00)	0.00 (0.00–2.00)	0.00 (0.00–0.00)
Medication (€)
Mean	0.65 (3.53)	1.97 (5.59)	1.09 (4.38)	<0.01
Median	0.00 (0.00–0.00)	0.00 (0.00–0.52)	0.00 (0.00–0.00)

GPC, general practice cooperative.

Discussion

In this secondary analysis of the TRIAGE trial, we analysed how emergency nurses used the triage tool differently for the intervention and the control group, which led to a remarkable difference in the proportion of patients assigned to primary care (13% in the intervention vs 25% in the control group). We found that nurses did not choose more flow charts leading to the ED as compared with all other flow charts, but they classified more patients as urgent (plus 3.1%) and they selected more discriminators linked to the ED (plus 16.2%). The motivation for choosing additional study tool discriminators leading to the ED were mostly the nurse-perceived need of medical imaging or medical reasons not specified in the protocol. The nurses did not overrule the protocol more often in the intervention group but when they did, they registered the reason for over-ruling more rigorously. These reasons mainly concerned organisational issues. The number of hospitalisations and the costs for patients assigned to the GPC were lower in the intervention weekends.

The strength of this study lies in the unique opportunity found in a cluster RCT making it possible to study the differences between an intervention and control group. The high number of included patients in a real-world setting is another strength. The study design has important limitations as well: the trial was not designed to study the outcomes of this paper. On the contrary, the marked difference in triage between the intervention and control group was an unexpected finding. Due to ICT limitations, it was not always possible to know which additional discriminators of the eMTS were chosen (either newly invented eMTS discriminators or the nonspecific GP risk). The collected free-text values were only available for a minority of the patients where an additional eMTS discriminator was chosen. Also, it was registered in only 64.6% of the intervention patients where the protocol was overruled even though this was explicitly requested. This might cause an important bias. Finally, due to an ICT error during the first two intervention weekend, the chosen discriminator remains unknown for 5% of the patients within urgency categories 4 and 5.

A previous study revealed that paper case scenarios generally receive lower triage scores (lower urgency) than live cases.6 As our intervention group is comparable to the real world, it seems logical that nurses triaged even more risk averse in the intervention group as compared with the control group (similar to live cases). Our study does not allow to definitely answer the question why the nurses classified more patients as urgent by selecting other discriminators during intervention weekends. It is likely that they did this either consciously or subconsciously because of a desired ED outcome and thus triaged more risk averse. Previous qualitative research concerning the same trial reveals some reasons why nurses are sometimes reluctant to assign a patient to the GPC (Meysman J, Morreel S, Lefevere E, et al. Triaging and Referring In Adjacent General and Emergency departments (the TRIAGE-trial): A process evaluation of medical staff experiences in a nurse-led triage system. Submitted for publication). Some nurses reported that they found it very difficult in the beginning to refer patients to the GPC. They gained trust in the system after reassurance that low-risk patients they diverted were not sent back. One nurse compared the control group to ‘playing poker for chips’ and the intervention group to ‘playing poker for money’. Nurses also indicated that it is time-consuming and complex to divert patients to the GPC even though the ED was only 50 m away. Another possibility is the influence of the study hospital. When diverting patients to the GPC, the ED loses income. On the other hand, the intervention led to a relatively small but significant decrease in the workload which at times was very welcome. Another reason why nurses triaged differently in intervention weekends might be found in the theory of planned behaviour: whether or not a subjective norm (some patients should go to the GPC) leads to an intention (I want to divert these patients) and a behaviour (I have given the advice to go to the GPC) depends on the attitude of the nurse (how important is it to me to give this advice?) and the amount of perceived behavioural control (what are the consequences of this diversion?).23 This attitude and the perceived behavioural control were probably different in the intervention group, because in the control group the nurse’s decision had no influence on the patient’s treatment.

The most frequently chosen eMTS discriminator was the perceived need for medical imaging. Even when not specified in the eMTS, nurses often used the non-specific discriminator GP risk with medical imaging as motivation. This need is subjective but, in most cases (79%), the physician did order medical imaging. The studied tool can be improved by providing more specific guidelines on the need for medical imaging especially for the presentational flow chart for limb problems, for example, by implementing the Ottawa knee, midfoot and ankle rules which have been validated to rule out fractures at the ED24 25 and can be used by ED nurses.26

The admission rate of patients with an assignment to the GPC was low but significantly higher during control weekends. More medical imaging and technical procedures were used for patients with an assignment to the GPC during control weekends. The question whether or not this reflects a true medical need or a difference in clinical behaviour between ED and GPC physicians cannot be answered by the current study leaving the question open whether or not the more risk averse triage during intervention weekends leads to over triage. It also remains unknown whether this more risk averse triage during intervention weekends increased patients’ safety.

The control group of the TRIAGE trial can be regarded as simulated circumstances comparable to triage research using paper-based scenarios or retrospective observational studies. This study proves that the theoretical results of such research should be interpreted cautiously as the nurses are likely to triage more risk averse when it really comes down to diverting patients and not only writing down a theoretical assignment. This difference between simulated and real-world experiments is new for the research about triage but is well known in other fields of research. For example, laboratory experiments may both understate and exaggerate the importance of social preferences.27 In lottery experiments, researchers found that research subjects in laboratory circumstances typically underestimate the extent to which they will avoid risk in the real world.28

Conclusion

Triage nurses classify more patients as urgent and select more discriminators linked to the ED when they actually have to divert patients to primary care as compared with a theoretical assignment to primary care. Researchers should be aware that outcomes from a simulated triage setting should not be extrapolated to the real world.