Pediatric early warning score versus a paediatric triage tool in the emergency department: A reliability study.

AIM: In the paediatric emergency department (PED), it is important to correctly prioritize children for physician assessment. The pediatric early warning score (PEWS), although not a triage tool, is often used for PED triage. The scandinavian Rapid Emergency Triage and Treatment System-pediatric (RETTS-p) is a reliability tested triage tool. We aimed to compare PEWS and RETTS-p in a Norwegian PED.
DESIGN: A reliability study.
METHODS: The PED nurse routinely did PEWS observations, while the principal investigator concomitantly made RETTS-p observations. Inter-tool agreement was calculated for the complete PEWS and RETTS-p and for vital signs scores, disregarding the RETTS-p emergency symptoms and signs (ESS).
RESULTS: Rapid Emergency Triage and Treatment System-pediatric assigned a higher urgency than PEWS. The inter-tool agreement between PEWS and RETTS-p was low (weighted kappa [95% confidence interval [CI] = 0.32 [0.24-0.40]]). Weighted kappa (95% CI) was 0.50 (0.41-0.59) for PEWS and RETTS-p without ESS, indicating that PEWS is not equivalent to five-level triage tools.

INTRODUCTION

In 2018, 1.2 million patients were admitted to Norwegian hospitals through an emergency department (ED) (Norwegian Directorate of Health, 2019). The Norwegian Board of Health Supervision recommends the use of five‐level triage tools to prioritize ED patients for physician assessment. Also, in a paediatric emergency department (PED), standardizing the priority of physician assessment could improve patient safety. The Brighton Pediatric Early Warning Score (PEWS) is used in all Norwegian paediatric departments to evaluate medical risk in children 0–18 years of age (Akre et al., 2010; Monaghan, 2005). In this version of the PEWS, vital signs including respiratory rate, oxygen requirements, pulse and behaviour, are documented and evaluated over time. Although not a triage tool (Seiger et al., 2013), PEWS is also used for prioritizing patients for physician assessment in many Norwegian PEDs. Only a few departments have implemented the Manchester Triage Scale (Mackway et al., 2014), the South African triage scale (Engan et al., 2018) or the Rapid Emergency Triage and Treatment System‐pediatric (RETTS‐p) (Henning et al., 2016) for paediatric triage. Other paediatric triage systems include the Canadian Triage and Acuity Scale (Warren et al., 2008), the Emergency Severity Index (Wuerz et al., 2000, 2001) and the Australasian Triage Scale (Australasian College for Emergency Medicine, 2002).

BACKGROUND

For this study, we chose the Scandinavian RETTS‐p, a reliability tested five‐level triage tool used in at least two large Norwegian PEDs including one PED in our regional health trust. At the time our study was undertaken, RETTS‐p was the only paediatric triage system that had been studied in the Scandinavian and Norwegian setting. Thus, we decided to examine this particular triage tool. PEWS and RETTS‐p both include a table of age‐specific reference values for respiratory rate and pulse. The age categories and cut‐offs differ in the two systems (Table 1). The different cut‐offs between PEWS and RETTS‐p pose a didactic challenge in particular for the PED nursing staff who are required to triage children according to RETTS‐p and at the same time perform a PEWS as an initial assessment of children who proceed to be admitted. The answer to the question whether both tools are necessary in the PED significantly influences the working processes of PED nursing staff and nurses represent important stakeholders, together with patients and their families. The use of a single system in the PED and inpatient wards would allow for continuity in patient assessment (Seiger et al., 2013), offer a didactic advantage over the use of a different system for triage and improve PED nursing staff resource use. The primary aim of this study was to examine whether PEWS and RETTS‐p assign the same degree of urgency in patients in a large Norwegian PED. To investigate the potential effect of different respiratory rate and pulse classifications in the two systems, the agreement between PEWS and the vital signs part of RETTS‐p was examined in addition to the agreement between PEWS and the complete RETTS‐p tool.

TABLE 1

Age categories and reference values for respiratory rate and pulse in pediatric early warning score (PEWS) and the Rapid Emergency Triage and Treatment System‐pediatric (RETTS‐p)

PEWS			RETTS‐p
Age category	Respiratory rate/min	Pulse/min	Age category	Respiratory rate/min	Pulse/min
<1 month	40–55	100–160	0–2 month	30–60	100–165
1 month to < 13 month	35–45	100–160	3–5 month	30–55	100–160
			6–12 month	25–50	90–150
13 month to <4 year	25–35	90–130	1 year	25–40	90–140
			2 year	20–35	85–135
4 year to <7 year	20–24	70–120	3–5 year	18–28	80–125
7 year to <13 year	19–22	70–110	6–11 year	15–24	65–110
13 year to <19 year	14–19	55–95	12–18 year	12–20	55–110

A secondary aim was to examine the sensitivity and specificity of PEWS to identify the patients with high urgent triage levels defined by RETTS‐p. We hypothesized that there was a substantial agreement between the PEWS and RETTS‐p in identifying high‐risk patients, that is patients who require immediate physician attention.

THE STUDY

Design

A reliability study presented according to the Guidelines for Reporting Reliability and Agreement Studies (GRRAS) (Kottner et al., 2011).

Method

All included patients were observed and categorized with a modified Brighton PEWS (Akre et al., 2010; Monaghan, 2005) and RETTS‐p. The PED nurse routinely assessed each child with PEWS, while the RETTS trained and certified nurse principal investigator (PI) concomitantly performed RETTS‐p triage.

Setting

The PED is one of the largest in Norway, with 7.000 patient assessments per year. In the winter season, approximately 200 children are being assessed per week, on average 30 each day. The PED has not implemented a triage tool, but has since 2011 used the PEWS in the initial assessment of children with medical and surgical diagnoses (Solevåg et al., 2013). All nurses are formally trained in the use of PEWS and have ample experience with its use. The nurse PI is formally trained in RETTS and practised RETTS‐p with experienced RETTS‐p raters prior to initiation of the study.

The tools

There is a vast number of paediatric early warning systems, and PEWS can be very different in its comprehensiveness (i.e. requiring different levels of assessment) and ability to discriminate between patients, for example needing hospital admission or not (Roland et al., 2016; Solevåg et al., 2013). The simpler Brighton PEWS (Monaghan, 2005) and RETTS‐p both give a numeric score based on deviations from normal, including vital signs such as pulse (or heart rate) and respiratory rate (Table 1). Deviations from normal only affect the PEWS when, for example, respiratory rate is 10/min higher than the upper reference, while RETTS‐p escalates at any value outside the reference (Henning et al., 2016).

In the PEWS, a score of 0–3 is allocated to each of six physiological measurements: respiratory rate, respiratory effort, oxygen requirement, pulse, capillary refill time and level of consciousness. The score reflects the magnitude of deviation from normal, with a high score representing large deviations from normal. In our modified Brighton PEWS, a weighting score of two is added if the patient receives continuous positive airway pressure or high‐flow nasal cannula treatment. An additional score of two is added in the case of persistent postoperative vomiting. Based on the total score, the medical risk is categorized into three groups: low (PEWS 0–2), medium (PEWS 3–4 or any individual physiological parameter score of 3) and high (PEWS ≥5).

In contrast to other triage systems, the RETTS‐p does not specify a wait time in minutes, nor does the original Brighton PEWS (Monaghan, 2005). The RETTS‐p requires a step‐wise priority classification initiated by a quick physical examination (“vital signs”) to capture the signs of potential organ failure similar to the physiological measurements in PEWS, including respiratory rate and pulse. The next step is to use the Emergency Signs and Symptoms (ESS), where the PED nurse chooses among more than 100 potential presenting complaints and based on symptoms and signs, find one out of 40 ESS algorithms that fits best with the patient's complaints (Widgren, 2020). The ESS grade symptoms in different urgency levels based on the disease manifestation that fits best with the patient's symptoms or presenting complaint. Both “vital signs” and ESS classify the child in one out of five triage levels and the higher of the two determines the child's triage colour (red, orange, yellow, green or blue). Correct use of the RETTS‐p requires extensive training and experience. RETTS‐p has only been tested for reliability, not validity (Henning et al., 2016; Magalhaes‐Barbosa et al., 2019; Westergren et al., 2014).

For comparison of urgency between PEWS and RETTS‐p, we used the Norwegian PEWS guideline (Akershus University Hospital, 2014): (a) PEWS 0–2: continue scoring according to treatment plan; (b) PEWS 3: notify physician; (c) PEWS 4: medical review within 30 min; and (d) PEWS ≥5: urgent medical review. A kappa coefficient can only be calculated if both variables have the same number of categories. A paediatrician with ample experience with PEWS (ALS) thus developed priority levels for PEWS that corresponded to the RETTS‐p levels developed by St. Olavs Hospital in Norway (Henning et al., 2016) (Table 2). Patients who were referred from the primary physician or through internal hospital referral as subacute (often seen in the PED the day after the referral) or for simple blood work or follow‐up were prospectively excluded from the study. Thus, we did not use the RETTS‐p blue triage level (i.e. no need for triage) in our analyses.

TABLE 2

Comparable urgency/triage categories in pediatric early warning score (PEWS) and the Rapid Emergency Triage and Treatment System‐pediatric (RETTS‐p) developed for the purpose of this study

Urgency/triage category	PEWS	RETTS‐p
Need urgent assessment of paediatrician	PEWS ≥ 5	Red
Paediatrician assessment within 30–20 min	PEWS 3–4	Orange
Paediatrician assessment within 2 hr	PEWS 2	Yellow
Paediatrician assessment within 4 hr	PEWS 0–1	Green

Participants

Children and adolescents 0–18 years of age with medical and surgical diagnoses referred from the primary physician for acute medical attention were included.

Data collection

The project constituted the master thesis of the nurse PI (HB). The master programme curriculum allowed for observations to be made in April 2018 and January 2019 on day, evening and night shifts (N = 19). Patients were consecutively recruited during the shifts. PED nurses varied in age and experience, but these data were not systematically collected due to privacy concerns in a relatively small population of PED nurses (N = 25).

Pulse and oxygen saturation were collected with pulse oximetry, and the values were read at the same time for PEWS and RETTS‐p. Similarly, the respiratory rate was measured by the PED nurse and used for both PEWS and RETTS‐p triage. Thus, the two raters were not blind to each other's vital signs observations. The two raters assessed the child's general condition (level of consciousness), which differs in the two tools, independently.

Power calculation

We aimed for a sample size to detect a weighted kappa of 0.75 against a null hypothesis that kappa was 0.6 for the agreement between the most urgent PEWS and RETTS‐p levels. This required 199 paired observations with PEWS and RETTS‐p to achieve 90% power (Sim & Wright, 2005).

Analysis

Data were entered by ALS into the Statistical Package for the Social Sciences (SPSS) version 24 (IBM Corporation) for statistical analyses. ALS also performed the analyses and was blinded to patient characteristics and outcomes, except for the data obtained from the PEWS and RETTS‐p.

Summary measurements are expressed as the mean and range, unless otherwise stated. Weighted Cohen's kappa with 95% confidence interval (CI) was calculated to examine agreement between PEWS and the priority given by the RETTS‐p. To examine whether a potential difference between PEWS and RETTS‐p could be explained by the ESS in RETTS‐p, weighted kappa was calculated for the complete tools and for “vital signs” without ESS for RETTS‐p. Although the RETTS‐p is not validated, we assumed that it better identifies “red” children than PEWS, since RETTS‐p is a specifically designed triage tool that, like all other five‐level triage tools, has more information than physiological parameters.

Thus, the sensitivity of and specificity of PEWS to detect a red RETTS‐p was calculated.

The sensitivity and specificity of the two highest PEWS urgency levels to detect orange and red RETTS‐p patients, that is including children who may present more subtle signs of urgency, were also calculated. In addition, because neither the PEWS nor the RETTS‐p has been validated for PED triage and there is thus no true gold standard, we calculated the sensitivity of RETTS‐p to detect a red PEWS.

ETHICS

This was a reliability study of observations, with no change in the management of the patients. The PI was involved in clinical work in the department of paediatric and adolescent medicine and had signed a confidentiality agreement. The hospital privacy legislation authority approved the study and judged the study to be exempt from the need for written consent from the observed patients or their parents. The PI provided oral information to the PED nurse, the child and parents about the study. Only routinely measured information was collected for the PEWS and RETTS‐p rating.

RESULTS

We included 200 children with a mean (range) age 6 (0–18) years, of which 94 were girls. Weighted kappa (95% CI) for PEWS and RETTS‐p was 0.32 (0.24–0.40). RETTS‐p more often prioritized the patient to a higher degree of urgency than PEWS.

Table 3 shows PEWS triage levels in the rightmost column and RETTS‐p in the lower row (N = 200). PEWS classified most children as green, followed by yellow, orange and red. RETTS‐p classified most children as yellow, followed by orange, green and red. Among children with green PEWS, 45/132 (34%) were also green in RETTS‐p. Two children with red RETTS‐p triage level had green PEWS. One of these patients had ESS number 130 (head/neck/back injury/hanging/strangulation); the other had ESS number 150 (hypoglycaemia). In both patients, the ESS determined the triage level. Seven out of 20 (35%) patients with red RETTS‐p triage level were classified as red also by the PEWS. The total PEWS under triage (N = 108) and the total over triage rate (N = 12) resulted in 80/200 (40%) children receiving the same triage category with the two systems.

TABLE 3

Crosstab pediatric early warning score (PEWS) and the Rapid Emergency Triage and Treatment System‐pediatric (RETTS‐p)

		RETTS‐p				Total PEWS
		Green	Yellow	Orange	Red
PEWS	Green PEWS 0–1	45	54	31	2	132
	Yellow PEWS 2	4	13	10	3	30
	Orange PEWS 3–4	1	2	15	8	26
	Red PEWS ≥ 5	0	0	5	7	12
Total RETTS‐p		50	69	61	20	200

Table 4 shows PEWS triage levels in the rightmost column and RETTS‐p without ESS in the lower row (N = 200). More patients were classified with a low (green) triage level than with the full RETTS‐p tool. Weighted kappa (95% CI) for agreement between PEWS and RETTS‐p without ESS was 0.50 (0.41–0.59). Of the children with red RETTS‐p triage level (N = 16), seven (44%) were also classified as red by the PEWS. None of the patients with red RETTS‐p level had a green PEWS level when ESS was excluded. The total PEWS under triage (N = 59) and the total over triage rate (N = 24) resulted in 117/200 (59%) children receiving the same triage category with the two systems.

TABLE 4

Crosstab pediatric early warning score (PEWS) and vital signs in the Rapid Emergency Triage and Treatment System‐pediatric (RETTS‐p) without emergency signs and symptoms

		Vital signs RETTS‐p				Total PEWS
		Green	Yellow	Orange	Red
PEWS	Green PEWS 0–1	86	40	6	0	132
	Yellow PEWS 2	11	14	4	1	30
	Orange PEWS 3–4	1	7	10	8	26
	Red PEWS ≥ 5	0	3	2	7	12
Total RETTS‐p		98	64	22	16	200

Table 5 shows the sensitivity and specificity of a red PEWS to “detect” a red RETTS‐p. The sensitivity (95% CI) of PEWS to detect a red RETTS‐p level was 35 (16–59)%. The specificity (95% CI) of PEWS to detect a red RETTS‐p was 97 (93–99)%. The sensitivity (95% CI) of the two highest PEWS urgency levels to detect orange or red RETTS‐p level was 43 (32–55)%. The specificity (95% CI) of the two highest PEWS urgency levels to detect a orange or red RETTS‐p was 97 (92–99)%.

TABLE 5

Pediatric early warning score (PEWS) sensitivity and specificity for red Rapid Emergency Triage and Treatment System‐pediatric (RETTS‐p)

			RETTS‐p triage category		Total
			Not red	Red
PEWS ≥ 5 (red)	Not red	Count	175	13	188
		% within RETTS‐p red	97,2%	65,0%	94,0%
	Red	Count	5	7	12
		% within RETTS‐p red	2,8%	35,0%	6,0%
Total		Count	180	20	200
		% within RETTS‐p red	100,0%	100,0%	100,0%

The sensitivity (95% CI) of RETTS‐p to detect a red PEWS was 58 (29–84)%. The specificity (95% CI) of RETTS‐p to detect a red PEWS was 93 (88–96)%.

DISCUSSION

Pediatric early warning score has a proven value in the wards to detect clinical deterioration and provide stabilizing measures to prevent adverse outcomes. In the PED, there was a poor agreement between the PEWS and RETTS‐p in identifying high‐risk patients, that is patients who require immediate physician attention.

Since PEWS is performed in many PEDs as a baseline assessment, it has been used to prioritize patients for physician attention in many Norwegian hospitals. In this study, RETTS‐p assigned a higher triage level than PEWS. The agreement between PEWS and RETTS‐p was low with only one third of patients with a red RETTS‐p triage level being classified as most urgent by the PEWS. The agreement between PEWS and RETTS‐p improved if only vital signs were used in the assessment (excluding ESS), but the weighted kappa was still low. The different age categories and reference ranges for respiratory rate and pulse may at least partly explain this difference (Table 1).

When using the full RETTS‐p tool, a similar proportion of children had green, yellow and orange triage level, respectively. If too many children have the same triage level, the PED nurse has to prioritize between children within the same triage level. Alternatively, rapid review by a senior physician or re‐triage may be required. Almost one third of the children were classified as orange, that is should receive medical attention within 20 min. Also, one third of the patients were classified as yellow by the RETTS‐p. RETTS‐p without ESS categorized most children to green triage, then yellow, orange and red, that is the same pattern that was seen with PEWS. Our results cannot support a conclusion whether or not to include ESS or clinical parameters other than vital signs in determination of urgency level.

Sensitivity and specificity

With respect to “detecting” a red RETTS‐p, PEWS had a low sensitivity and high specificity. Red RETTS‐p is the least frequent triage level, but the level where agreement between the two tools is most crucial. Assuming that RETTS‐p correctly assigns a red level, a PEWS sensitivity of 35% is too low. A specificity of PEWS to detect a red RETTS‐p of 97% is not surprising considering the more conservative definition in PEWS of abnormal respiratory rate and pulse. In agreement with our results, a previous study concluded that a high PEWS should prompt a high degree of awareness, whereas a low PEWS does not exclude serious disease (Lillitos et al., 2016).

Age‐specific references for respiratory rate and pulse

The divergent definitions of normal and abnormal respiratory rate and pulse may explain some of the differences we found between PEWS and RETTS‐p triage levels. In particular, the fact that deviations from normal only affect the PEWS when, for example, respiratory rate is 10/min higher than the upper reference, while RETTS‐p escalates at any value higher than the reference (Henning et al., 2016) results in higher triage levels in RETTS‐p. In RETTS‐p, a respiratory rate 9/min above the upper reference can result in a red triage. Thus, the same patient may have a green PEWS triage level, even when ESS is not used in RETTS‐p.

Emergency signs and symptoms

In this study, ESS often escalated the “vital signs” triage level. Not all severe disease and injury results in abnormal vital signs. The history provided by the patient and parents about the presenting complaint may provide essential information about disease severity (Farrohknia et al., 2011). Other triage systems than RETTS‐p use risk factors such as newborn age or age <2–3 months, fever and chronically ill child, instead of ESS to have a second factor for triage (Engan et al., 2018).

In our PED, nurses do take the presenting complaint and history into account and may ask for physician assessment more rapidly than the PEWS indicates. However, correct evaluation of presenting complaint and history is dependent on knowledge and experience. Triage tools are meant to ensure that nurses’ evaluation of priority is less dependent on such knowledge and experience. Robust systems are important determinants of patient safety.

Strengths and limitations

Vital signs and clinical condition may change rapidly. Strengths of this study include a high number of observations and that RETTS‐p triage was prospectively carried out at the exact time as PEWS for each patient. However, comparison between the PED nurses’ routine PEWS assessment and the PI’s RETTS‐p triage may have introduced bias in the evaluation of the patient. The fact that the two raters were not blinded to each other's vital signs observations is another potential source of bias. Excluding some months from the study, because the PI was not available to perform observations, could have introduced selection bias. Other limitations of the study include its single centre nature. Norwegian paediatric departments differ in size and organization. Only a limited number of paediatric departments have a designated PED, most hospitals admit children from the same ED as adult patients. Both PEWS and RETTS‐p may be used and interpreted differently in hospitals with various experiences and workloads. These different communities of practice may influence the generalizability of results from one hospital. Also, the value of quick triage versus a more thorough (PEWS) assessment may be less in Norwegian PEDs that have a considerably lower number of patients in the waiting room than many international PEDs. Importantly, we used a triage system that has not been used extensively internationally. Zachariasse et al. (2019) found no clear difference in performance between the most commonly used and described systems, that is the Canadian Triage and Acuity Scale, Emergency Severity Index and Manchester Triage System. Thus, we speculate that five‐level triage systems have comparable psychometric properties, but perform differently with changing settings. However, it should be noted that vital sign measurement increases the sensitivity of the paediatric Canadian Triage and Acuity Scale (Warren et al., 2008), which might represent a difference compared with RETTS‐p where the ESS often escalated the acuity level assigned with vital signs scores.

In studies where the researcher makes observations, the validity of the results may be influenced by the study object's knowledge of being observed, the so‐called Hawthorne effect (Sedgwick & Greenwood, 2015). The PED nurse was asked to perform PEWS the way he/she routinely does, but it cannot be ruled out that he/she would have scored differently if the researcher was not present. Importantly, the study was not designed to examine what the correct (gold standard) triage level was for individual patients. The undertaking of validity studies of paediatric triage systems has been hampered by the lack of a reference standard for true urgency. Surrogate markers including hospitalization, resource use, intensive care unit admission, length of stay and expert opinion have been used (van Veen & Moll, 2009). A recent review and meta‐analysis included studies evaluating the Canadian Triage and Acuity Scale, Emergency Severity Index and Manchester Triage System in an unselected group of ED patients (Zachariasse et al., 2019). In Norway and the Scandinavian countries, ED patients are selected through prior assessment in primary care. (Zachariasse et al., 2019) defined low urgency as “discharge home after the ED visit.” Even in the selected ED population we investigated, approximately 50% of patients leave the ED without being admitted. RETTS‐p has been examined with regard to reliability, but has not been validated (Magalhaes‐Barbosa et al., 2019). The agreement between PEWS and RETTS‐p was low, but we cannot conclude from our data which tool assigns the correct triage level, although RETTS‐p was defined as “correct” for most statistical calculations. Based on our results, RETTS‐p systematically assigns a higher level of urgency than the PEWS. The rationale for this might be that it is safer to over triage than to under triage. However, too many children being classified as “urgent” may also negatively affect patient safety. Our results cannot support a conclusion whether or not to include ESS or clinical parameters other than vital signs in determination of urgency level.

CONCLUSIONS

Rapid Emergency Triage and Treatment System‐pediatric assigned a higher triage level than PEWS. The agreement between PEWS and RETTS‐p was low. The difference could be explained by ESS often assigning a higher urgency level, as well as different reference values for vital signs in the two systems. For the purpose of identifying patients classified as most urgent by RETTS‐p, PEWS has a low sensitivity. PEWS has not been validated for use in the emergency department, rather it was designed to detect changes over time. When using triage tools in the emergency department, it is recommended to use a five‐level tool such as RETTS‐p. Our results do not support that PEWS is equivalent to five‐level tools specifically designed for emergency department triage.

They suggest that only physiological parameters do not seem sufficient to discriminate high levels of urgency in the PED. However, as neither tool were validated, validation studies of both tools including clinical outcomes to assess the correct triage level for individual patients are needed to corroborate this conclusion. Future studies comparing PEWS and triage tools should aim to limit observer bias and improve observer accuracy, for example by performing independent observations with the tools.

CONFLICT OF INTEREST

The authors declare that they have no conflict of interest.

AUTHOR CONTRIBUTIONS

The first author (HB): formulation of hypotheses and design, conduction of the research presented in this paper, data analysis, and writing of paper. The second author (ALS): formulation of the hypotheses and design, writing of paper, general consultant, and co‐writing of the final version of the manuscript. The third author (MTS): formulation of the hypotheses, and design and study supervision. All authors: reading and approval of the final manuscript.