Diagnostic accuracy of PAT-POPS and ManChEWS for admissions of children from the emergency department.

BACKGROUND: The Pennine Acute Trust (PAT) Paediatric Observation Priority Score (PAT-POPS) is a specific emergency department (ED) physiological and observational aggregate scoring system, with scores of 0-18. A higher score indicates greater likelihood of admission. The Manchester Children's Early Warning System (ManChEWS) assesses six physiological observations to create a trigger score, classified as Green, Amber or Red.
METHODS: Prospectively collected data were used to calculate PAT-POPS and ManChEWS on 2068 patients aged under 16 years (mean 5.6 years, SD 4.6) presenting over 1 month to a UK District General Hospital Paediatric ED. Receiver operating characteristics (ROC) comparison, using STATA V.13, was used to investigate the ability of ManChEWS and PAT-POPS to predict admission to hospital within 72 h of presentation to the ED.
RESULTS: Comparison of the area under the ROC curve indicates that the ManChEWS ROC is 0.67 (95% CI 0.64 to 0.70) and the PAT-POPS ROC is 0.72 (95% CI 0.68 to 0.75). The difference is statistically significant. At a PAT-POPS cut-off of ≥2, 80% of patients had their admission risk correctly classified (positive likelihood ratio 3.40, 95% CI 2.90 to 3.98) whereas for ManChEWS with a cut off of ≥Amber only 71% of patients were correctly classified (positive likelihood ratio 2.18, 95% CI 1.94 to 2.45).
CONCLUSIONS: PAT-POPS is a more accurate predictor of admission risk than ManChEWS. Replacing ManChEWS with PAT-POPS would appear to be clinically appropriate in a paediatric ED. This needs validation in a multicentre study. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/.

There is a need for a specific emergency department early warning system for children.

Manchester Children's Early Warning System (ManChEWS) is one scoring system, which correctly identifies the deteriorating child, but can overtrigger which may lead staff to become immune to the score.

Paediatric Observation Priority Score (POPS) is a new scoring system, combining physiological measurement and clinical observation.

A Pennine Acute Trust (PAT) Paediatric Observation Priority Score (PAT-POPS) score ≥2 would correctly identify about 50% of children who need to be admitted from the ED, and a PAT-POPS score <2 would correctly identify 85% of cases who could be discharged from the ED.

PAT-POPS has slightly higher diagnostic accuracy for predicting the likelihood of admission than an existing tool often used in this population, ManChEWS.

PAT-POPS can be used for patients with either trauma or illness.

Introduction

Health professionals make judgements on whether children attending emergency departments (EDs) require hospitalisation, or can safely be sent home. These judgements require a complex assessment of the child's health and an estimation of the potential for improvement or deterioration. Since at least 2006 it has been recommended that early identification systems to recognise children developing critical illness should be used.1 Many paediatric early warning scores (EWS) use track and trigger systems, relying on repeated observations over time, intended for use with hospitalised children,2–9 to predict which children are likely to deteriorate, rather than who requires admission from an ED. There is a need for a specific ED early warning system, validated on ED patients.10 In the absence of an ED EWS, our ED used The Royal Manchester Children's Hospital Early Warning System (ManChEWS), a track and trigger system.11 Variation in six physiological parameters (RR, oxygen requirement, heart rate, systolic BP, capillary refill time and conscious level) is scored during routine nursing observations (figure 1). While ManChEWS correctly identifies the deteriorating child it overtriggers which may lead staff to become immune to the score.4

Figure 1

Manchester Children's Early Warning System (ManChEWS) scoring system.

The Pennine Acute Trust (PAT) Paediatric Observation Priority Score (PAT-POPS) is a new EWS, designed for use in children's ED, combining physiological measurement and clinical observation into an aggregate scoring system (figure 2). The target user of PAT-POPS is nursing staff in the ED as part of their post-triage initial assessment. PAT-POPS has been adapted from the Paediatric Observational Priority Score (POPS),12 13 designed by the Paediatric Emergency Medicine Leicester Academic Group.

Figure 2

North Manchester modified Pennine Acute Trust (PAT) Paediatric Observation Priority Score (PAT-POPS) chart.

The original POPS demonstrated utility as a patient safety system in an analysis of the admission length of 24 000 patients who attended a UK ED.14 With an area under the curve (AUC) for the receiver operating characteristic (ROC) curve of 0.8, it was concluded that POPS may assist in reducing unnecessary admissions and preventing episodes of missed or incorrect diagnoses.

Over a period of 1 month, we calculated PAT-POPS and ManChEWS scores for 2068 children attending our paediatric ED, and recorded whether the child was admitted or discharged. We compared the diagnostic accuracy of the two scoring methods in estimating the likelihood of admission or discharge. The overall aim of this study was to compare the diagnostic accuracy of PAT-POPS and ManChEWS for estimating whether or not hospital admission is required among children presenting to an ED with trauma or medical diagnoses.

Method

Participants

The study population was children aged under 16 years attending the ED of North Manchester General Hospital (part of PAT) during March 2012. Children who left the ED before they could be assessed for admission, or where insufficient data were available to calculate PAT-POPS and ManChEWS, were excluded.

Test methods

The reference standard for our study was admission to hospital within 72 h of first presentation at the ED. This was a prospective study of a consecutive series of patients: data collection was planned before the index tests and reference standard were performed. The decision on whether or not to admit a child to inpatient care was made by the clinician seeing the patient, using their subjective clinical experience as well as departmental guidelines, including ManChEWS. The disposal outcome (discharged or admitted) for each attendance was recorded on the electronic patient record. Information was collected on re-presentations, and where a child re-presented to the ED with the same clinical problem within 72 h and was admitted, we counted that as an admission for the original presentation. Data on admission were recorded by hospital staff in the hospital electronic record, and then extracted by the research team retrospectively.

ManChEWS is scored Green, Amber or Red.11 The score encourages observation monitoring and empowers nursing staff by providing clear instructions of what actions should be taken dependent on the status of the child. All observations must be within the normal range for the age of the child for the award of Green status. Any physiological parameters that are abnormal, but within the defined range, lead to Amber status. Any parameters that are very abnormal and which lie outside of the Green or Amber ranges result in a Red status, indicating that the child has potentially significant physiological disturbance.

PAT-POPS is assessed as a score between 0 (likely low risk of serious illness) and 18 (likely high risk of serious illness) and is a checklist which quickly scores acutely ill children on age-related physiological measures (heart rate, RR, temperature) and behavioural and risk-identifiers (such as oxygen saturations, breathing pattern, conscious level, nurse's judgement of how well the child is, child's behaviour) using easy to collect data (figure 2). Measurements of the physiological variables and subjective assessments necessary to calculate ManChEWS and PAT-POPS for each patient were taken by nursing staff in the ED either at the point of triage or during the child's assessment in the ED.

Before the study, nursing staff were trained in the use of PAT-POPS. Nursing staff were already familiar with ManChEWS. Patient data for the PAT-POPS and ManChEWS assessment were collected prior to the admission decision, so there was blinding to the outcome. The persons who transcribed the data from the electronic patient record to the PAT-POPS data sheet did so retrospectively, some time after admission, and so were not blinded to the outcome. The doctor making the admission decision was not blinded to the ManChEWS score and may have used it in their decision-making. Doctors were blinded to the PAT-POPS scores, which were recorded onto data capture sheets rather than directly into the patient's notes. The doctors received no training on PAT-POPS to minimise the chances of them using it as part of a decision-making process if they happened to see the data capture sheet.

Statistical methods

We report the age, gender and diagnosis of our sample of patients, using descriptive statistics. We have calculated the sensitivity and specificity of PAT-POPS and ManChEWS to predict admission and we present this data as comparative ROC curves. We report the positive and negative likelihood ratios at different cut points of PAT-POPS and ManChEWS.15 We present 95% CIs and p values, as appropriate. We have compared the sensitivity and specificity of PAT-POPS to predict admission for separate groups of children with illness or trauma, using ROC analysis, and followed the DeLong method to compare the AUC of ROC curves for PAT-POPS and ManChEWS.16 The data were entered into Microsoft Excel and analysed using STATA V.13 (StataCorp. 2013. Stata Statistical Software: Release 13. College Station, TX: StataCorp LP).

Ethics

This service evaluation was approved by the hospital's audit department and no formal ethical approval was required as no clinical decisions were made on the basis of the PAT-POPS data and we continued to manage each patient in accordance with our ED guidelines.

Results

Between 1 March 2012 and 31 March 2012, 2539 children attended the paediatric ED. We have omitted 87 children with no reference standard who left the ED without being seen, and 384 children with no PAT-POPS recorded. There are two likely reasons for not recording PAT-POPS. First, where there was no wait in the ED a disposal outcome decision may have been made before nursing assessment. Second, PAT-POPS was a new tool and some staff may have forgotten to calculate a PAT-POPS for each patient. The analysis is based on the remaining 2068 patients (figure 3). Among the 2068 children, the mean age was 5.6 years, median 4.1 years (range 1 day to 15 years), 954 (46%) presented with a trauma and 1114 (54%) with a medical condition.17 A comparison of children with and without a PAT-POPS recorded score indicates that the children who were assessed using PAT-POPS were, on average, younger (5.6 years compared with 8.2 years), and were less likely to attend with trauma (46.1% compared with 53.6%), more likely to re-attend within 72 h (3.7% compared with 1.6%) and more likely to be admitted (15.8% compared with 4.4%). All these differences are statistically significant (p<0.05), using Mann–Whitney or proportion tests, as appropriate. It appears that the 384 children who were not assessed for PAT-POPS were less ill than those who were assessed.

Figure 3

Study flow diagram.

Hospital admission

Of the 2068 children, 317 (15.3%) were admitted to hospital at first attendance. Seventy-six children (3.7%) re-attended the ED with the same complaint one or more times within 72 h of first attendance and 15 (19.7%) of these were admitted (3 of whom had also been admitted on first attendance, and then discharged). In total, 327 (15.81%) were admitted to hospital within 72 h of first attendance.

Test results

Measurement of the variables necessary to calculate ManChEWS and PAT-POPS was undertaken as soon as possible after arrival to the ED. Over 95% of patients were seen and either admitted or discharged within a period of 4 h from the time of arrival.

ManChEWS results

Across the 2068 patients, 1397 (67.55%) were Green, 573 (27.71%) Amber and 98 (4.74%) Red (table 1).

Table 1

A cross tabulation of Manchester Children's Early Warning System (ManChEWS) and Paediatric Observation Priority Score (POPS) scores by admission

	Admitted	Discharged	Total
ManChEWS score
Green	132	1265	1397
	9.45%	90.55%	100%
Amber	144	429	573
	25.13%	74.87%	100%
Red	51	47	98
	52.04%	47.96%	100%
PAT-POPS score
0	108	1164	1272
	33.0%	66.9%	61.5%
1	57	323	380
	17.4%	18.6%	18.4%
2	44	137	181
	13.5%	7.9%	8.8%
3	42	56	98
	12.8%	3.2%	4.7%
4	18	32	50
	5.5%	1.8%	2.4%
5	18	20	38
	5.5%	1.2%	1.8%
6	10	6	16
	3.1%	0.3%	0.8%
7	10	0	10
	3.1%	0.0%	0.5%
8	9	3	12
	2.8%	0.2%	0.6%
9	4	0	4
	1.2%	0.0%	0.2%
10	3	0	3
	0.9%	0.0%	0.2%
11	1	0	1
	0.3%	0.0%	0.1%
12	3	0	3
	0.9%	0.0%	0.2%
Mean PAT-POPS	2.3	0.6	0.9
SD	2.62	1.11	1.58
Total, N	327	1741	2068

PAT-POPS, Pennine Acute Trust (PAT) Paediatric Observation Priority Score.

PAT-POPS results

The mean PAT-POPS score across all 2068 patients was 0.9 (SD 1.58). The median was 0. Overall 62% of patients scored a PAT-POPS of 0. No patient had a PAT-POPS score of greater than 12 (table 1).

No adverse events were reported from performing any of the index tests or reference standard. There were no deaths of children in the ED during the study period.

Diagnostic accuracy of ManChEWS and PAT-POPS

Sensitivity and specificity

Sensitivity and specificity of ManChEWS to predict admission were calculated at each ManChEWS value (table 2). A Red ManChEWS score identifies around 16% of those who were admitted, and successfully identifies almost all of those who were not admitted. An Amber or Red score identifies almost 60% of those who were admitted, and 73% of those who were discharged from the ED. Looking at the positive likelihood ratios, children with a ManChEWS score of Amber or Red are twice as likely to be admitted as children assessed as Green. Children with a ManChEWS score of Red are almost six times as likely to be admitted, compared with children assessed as Green or Amber.

Table 2

Sensitivity and specificity of Manchester Children's Early Warning System (ManChEWS) and Paediatric Observation Priority Score (POPS) to predict admission to the emergency department (ED)

Cut point for admission decision	Sensitivity	Specificity	Likelihood ratio+	Likelihood ratio−
ManChEWS
Admit all (Green, Amber, Red)	100.00%	0.00%	1.00
Admit if Amber or Red	59.63%	72.66%	2.18	0.56
	(54.08 to 64.96)	(70.49 to 74.73)	(1.94 to 2.45)	(0.49 to 0.64)
Admit if Red	15.60%	97.30%	5.78	0.87
	(11.93 to 20.09)	(0.96 to 0.98)	(3.96 to 8.43)	(0.83 to 0.91)
Admit none	0.00%	100.00%		1.00
POPS
(≥0)	100.00%	0.00%	1.00
(≥1)	66.97%	66.86%	2.02	0.49
	(61.55 to 71.99)	(64.58 to 69.06)	(1.83 to 2.24)	(0.42 to 0.58)
(≥2)	49.54%	85.41%	3.40	0.59
	(44.01 to 55.09)	(83.65 to 87.02)	(2.90 to 3.98)	(0.53 to 0.66)
(≥3)	36.09%	93.28%	5.37	0.69
	(30.92 to 41.58)	(91.98 to 94.39)	(4.28 to 6.74)	(0.63 to 0.74)
(≥4)	23.24%	96.50%	6.63	0.80
	(18.85 to 28.28)	(95.49 to 97.29)	(4.84 to 9.09)	(0.75 to 0.84)
(≥5)	17.74%	98.33%	10.65	0.84
	(13.84 to 22.41)	(97.58 to 98.86)	(6.93 to 16.37)	(0.80 to 0.88)
(≥6)	12.23%	99.48%	23.66	0.88
	(8.98 to 16.40)	(98.98 to 99.75)	(11.59 to 48.29)	(0.85 to 0.92)
(≥7)	9.17%	99.83%	53.24	0.91
	(6.37 to 12.97)	(99.45 to 99.96)	n/a	(0.88 to 0.94)
(≥8)	6.12%	99.83%	35.49	0.94
	(3.87 to 9.44)	(99.45 to 99.96)	n/a	(0.91 to 0.97)
(≥9)	3.36%	100.00%	n/a	0.97
	(1.78 to 6.11)	(99.73 to 100)	n/a	(0.95 to 0.99)
(≥10)	2.14%	100.00%	n/a	0.98
	(0.94 to 4.55)	(100 to 100)	n/a	(0.96 to 0.99)
(≥11)	1.22%	100.00%	n/a	0.99
	(0.39 to 3.32)	(100 to 100)	n/a	(0.98 to 100)
(≥12)	0.92%	100.00%	n/a	0.99
	(0.24 to 2.88)	(100 to 100)	n/a	(0.98 to 1.00)
(>12)	0.00%	100.00%	n/a	1.00
N=2068

n/a, Not available. 95% CIs are shown in parentheses.

Sensitivity and specificity of PAT-POPS to predict admission were calculated at each PAT-POPS value (table 2). A PAT-POPS score of 9 or above correctly identifies all of those who were sent home, but has poor sensitivity. Sensitivity improves as the cut point lowers, without great loss of specificity until under 2. A PAT-POPS score of 2 or more successfully identifies 50% of those who were admitted, and 85% of those who were sent home. Looking at the positive likelihood ratios, children with a PAT-POPS cut point of 2 or more are more than three times as likely to be admitted as children with 0 or 1.

Receiver operating characteristics

Sensitivity was plotted against (1−specificity) for each possible cut-off of ManChEWS and PAT-POPS in ROC curves (figure 4). The area under the ROC curve for ManChEWS is 0.67 (95% CI 0.64 to 0.70) and for PAT-POPS is 0.72 (95% CI 0.68 to 0.75). The difference is statistically significant (p<0.01).

Figure 4

Receiver operating characteristics (ROC) curves for Manchester Children's Early Warning System (ManChEWS) and Pennine Acute Trust (PAT) Paediatric Observation Priority Score (PAT-POPS).

Use of PAT-POPS for trauma and medical patients

Sensitivity was plotted against (1−specificity) to draw separate ROC curves for the use of PAT-POPS among medical and trauma patients (figure not shown). The area under the ROC curve is 0.73 for medical patients and 0.69 for trauma patients. This analysis suggests that the usefulness of PAT-POPS to predict admission is similar among both trauma and medical patients, although slightly higher in medical patients. The pattern is repeated for ManChEWS: in medical patients, the area under the ROC curve is 0.69, and in trauma patients it is 0.66.

Limitations

This is a single-centre study and the results could have been different in another centre with an alternative arrangement of services and a different admission threshold. The outcome measure used in the study is whether the child was admitted to hospital, as assessed by an ED clinician. We endeavoured to increase the robustness of that measure by including any readmissions within 72 h. Future studies could consider level of inpatient care, admissions to other hospitals and length of stay.

Decision-making on a heterogeneous population of medical and trauma patients differs widely. There are some conditions which automatically trigger an admission regardless of the PAT-POPS (such as deliberate self-harm, child protection cases or a child with a fracture requiring operative management).

This study was based on patients who attended the ED during 1 month (March) and diagnoses in paediatrics do have a seasonal variation, with higher rates of respiratory conditions in winter and minor trauma in summer. March was chosen because this month coincided with some dedicated time made available in work schedules for members of the study team. In future studies sampling patients from throughout the year will be important.

Some patients who attended the ED during the month of the study did not have PAT-POPS recorded, because of an absence of some of the data used to calculate the score. The missing data could not be calculated retrospectively because PAT-POPS includes subjective nurse assessments of the child's behaviour and condition, which is not routinely collected.

Although our results show that PAT-POPS appears slightly better than ManChEWS at predicting admission of children from the ED, PAT-POPS scores are disadvantaged by nurses being less familiar with the PAT-POPS tool and hence more likely to make errors in scoring and clinicians not being blinded against the ManChEWS score.

Nonetheless, it is important to acknowledge that the differences between those two scoring systems may not be clinically significant.

Discussion

No universally validated children's EWS exists to predict likelihood of admission or discharge from the ED with illness or injury.

This research has demonstrated that, among children aged 0–16 years, PAT-POPS has slightly higher diagnostic accuracy for predicting the likelihood of admission than an existing tool often used in this population, ManChEWS, and that it can be used for patients with either trauma or illness.

The use of PAT-POPS in trauma patients as well as medical patients may sound counterintuitive: the decision to admit a patient with a trauma diagnosis from the ED can be based on factors and assessments that are quite different from those patients with a medical diagnosis (such as the need for operative intervention). Even though the original POPS was intended to be used on medical rather than trauma patients, the authors of this study believed that it would be interesting to evaluate how well PAT-POPS functioned on patients with a trauma diagnosis as well as a medical diagnosis. Some trauma patients would score highly in the categories of behaviour (if they were agitated or in pain) or nurse's judgement (a child with a severely deformed fractured limb).

A PAT-POPS score of 2 or more was associated with correct identification of 50% of children who were admitted from the ED, and a PAT-POPS score of less than 2 was associated with correct identification of 85% of children who were discharged from the ED. While this is a useful tool to aid decision-making on admissions in our population of patients, at a score of 2 or more, about half of children who were admitted would have been discharged, and 15% of those who were discharged home would have been admitted. Understanding which of these patients are more likely to be missed is a factor that should be addressed in future research. The results in this study compare favourably to previous studies. In a review of the performance of 10 Paediatric Early Warning Scores (PEWS) in predicting hospital admission of 17 943 children aged 0–16 years attending an ED, the area under the curve of the ROC curves (AUROCs) for the 10 PEWS ranged from 0.56 (95% CI 0.055 to 0.58) to 0.68 (0.66 to 0.69).18

The NHS Institute PEWS is a valid tool with good diagnostic accuracy in recognising children at risk of serious and life-threatening deterioration at triage in the ED, but further work is needed to determine whether other subjective measures have any value in paediatric early warning tools.19 In a study of children aged 0–16 years a PEWS score of ≥2 had a sensitivity of 37% and a specificity of 88%. The authors concluded that PEWS is of limited value in predicting admission (in a triage setting) in a population of undifferentiated disease. However, a low PEWS score has a high specificity, that is, a patient scoring <2 is unlikely to need admission.20 PAT-POPS has a larger AUROC than any of the 10 PEWS tested in the study, and it contains more subjective criteria than PEWS, making it an ideal candidate for future work to resolve the concerns that were highlighted in the above two studies.

The mean PAT-POPS score was 0.9 on a scale of 0–18 which may indicate that some of the items used to calculate the score may not be especially relevant, particularly at the lower end of the scale. Future work should investigate the weighted contributions of each of the components of PAT-POPS to the total score and whether there are some of the current components that could be either modified or removed without detrimental effect on the sensitivity and specificity reported in this study.

This initial study was undertaken as a service evaluation of a new tool and the intention is that, after this preliminary work, a multicentre study will be undertaken to investigate further. We recommend further research in two areas. First, refinement and validation of PAT-POPS should take place to ensure the various components in the score are combined together to make the most effective tool. Second, multicentre validation would be useful to determine if the effects found at our site are replicated elsewhere. In carrying out this further work there is a clear need to identify a gold standard outcome measure setting out clearly the criteria upon which the appropriateness, or otherwise, of an admission have been determined.

PAT-POPS has the potential to be an effective tool for use in deciding admission of children from the ED.

Further research could refine the tool and enhance its predictive accuracy.

There is a need to identify a robust gold standard outcome measure for hospital admission.