A NICE combination for predicting hospitalisation at the Emergency Department: a European multicentre observational study of febrile children.

BACKGROUND: Prolonged Emergency Department (ED) stay causes crowding and negatively impacts quality of care. We developed and validated a prediction model for early identification of febrile children with a high risk of hospitalisation in order to improve ED flow.
METHODS: The MOFICHE study prospectively collected data on febrile children (0-18 years) presenting to 12 European EDs. A prediction models was constructed using multivariable logistic regression and included patient characteristics available at triage. We determined the discriminative values of the model by calculating the area under the receiver operating curve (AUC).
FINDINGS: Of 38,424 paediatric encounters, 9,735 children were admitted to the ward and 157 to the PICU. The prediction model, combining patient characteristics and NICE alarming, yielded an AUC of 0.84 (95%CI 0.83-0.84).The model performed well for a rule-in threshold of 75% (specificity 99.0% (95%CI 98.9-99.1%, positive likelihood ratio 15.1 (95%CI 13.4-17.1), positive predictive value 0.84 (95%CI 0.82-0.86)) and a rule-out threshold of 7.5% (sensitivity 95.4% (95%CI 95.0-95.8), negative likelihood ratio 0.15 (95%CI 0.14-0.16), negative predictive value 0..95 (95%CI 0.95-9.96)). Validation in a separate dataset showed an excellent AUC of 0.91 (95%CI 0.90- 0.93). The model performed well for identifying children needing PICU admission (AUC 0.95, 95%CI 0.93-0.97). A digital calculator was developed to facilitate clinical use.
INTERPRETATION: Patient characteristics and NICE alarming signs available at triage can be used to identify febrile children at high risk for hospitalisation and can be used to improve ED flow. FUNDING: European Union, NIHR, NHS.

Advanced Paediatric Life Support

Area under the Curve

Emergency Department

Emergency Medical Services

Interquartile range

Management and Outcome of Fever in children in Europe

National Institute for Health and Care Excellence

Personalised Risk assessment in Febrile illness to Optimise Real-life Management across the European Union

Paediatric Early Warning System

Receiver Operating Characteristic curve

Serious bacterial infection

Evidence before this study

ED crowding is known to negatively impact quality of care for adult as well as paediatric patients, patient outcome and patient as well as health care worker satisfaction.

Although several studies have investigated the positive effects of admission prediction models on patient flow and ED crowding, those studies mainly focused on adults or specific paediatric patient groups such as asthma, used variables not available early on at the ED process and thus limiting the effect on patient flow, or were single-centre studies, limiting their generalisability.

Added value of this study

In a large multicentre study of almost 40,000 paediatric ED visits, we developed and validated a robust admission prediction model for febrile children attending the ED, based on patient characteristics and clinical alarming signs, which can be used to predict general ward and PICU admission directly at triage.

Implications of all the available evidence

The developed prediction model can be used at triage to identify febrile children attending the ED at high risk for hospitalisation and as such can be used to improve patient flow and reduce crowding. A digital calculator is available to facilitate clinical use.

Alt-text: Unlabelled box

Introduction

Fever in infants and children is one of the most common reasons to present to the ED, accounting for up to 20-30% of ED visits.[1]

Most children with fever will suffer from self-limiting viral illnesses, however, the clinical presentations of self-limiting viral and life-threatening bacterial infections may be identical, making diagnosis based on clinical judgement alone a difficult task. [1] If life-threatening infections are not recognised in time, this may have disastrous consequences, such as mortality, long-term morbidity or ICU admission, highlighting the daily challenge of caring for this broad group of children. As a result, an elaborate approach is often used, characterised by multiple investigations, evaluation of treatment effect at the ED or hospital admission for observation [1,2]. Unfortunately, such interventions are invasive, costly and are likely to prolong a child's visit to the ED, contributing to extended ED waiting times and ED crowding [1].

ED crowding can negatively impact ED length of stay, guideline adherence, quality of care for the individual patient, such as delay in administration of antibiotics or analgesics, patient outcome including mortality, health care costs, patient satisfaction and healthcare staff satisfaction. For example, several American studies estimated the financial burden of crowding to be as high as several million dollars per hospital, for example due to a longer hospital length of stay [3] or ambulances being diverted to other EDs [4].

Regarding patient and healthcare staff satisfaction, ED waiting times are one of the most important factors influencing parental satisfaction [1] and crowding is one of the factors contributing to health care staff “compassion fatigue”, a reduced capacity and interest in being empathetic for suffering individuals, that has a further negative impact on quality of care [5].

Several studies have shown a positive effect of interventions that reduce crowding on these outcome measures [1,[3], [4], [5], [6], [7], [8], [9], [10]] and interventions that expedite early admissions ensure earlier access to specialised care for those patients that need to be admitted anyway and are associated with increased patient and physician satisfaction [8,11].

An important approach in the reduction of ED crowding is the development of tools that can predict admission early on at the ED process, such as during triage [8].

Our aim was to identify risk factors for hospital admission in febrile children attending the ED that are available at triage and to use those risk factors to develop and validate a prediction model that can be used to improve patient flow by identifying febrile children with a high risk of hospitalisation. Furthermore, we aimed to develop a practical digital prediction tool, that can easily be implemented at the ED.

Methods

Study design

This study is part of the MOFICHE study (Management and Outcome of Febrile children in Europe), which is embedded in the PERFORM study (Personalised Risk assessment in Febrile illness to Optimise Real-life Management across the European Union, https://www.perform2020.org). The MOFICHE study is an observational multicentre study that studies the management and outcome of febrile children in Europe using routine data [12]. The study was approved by the ethics committees of the participating hospitals. The need for informed consent of individual patients was waived.

Study population and setting

All children aged 0-18 years presenting with fever to the ED (temperature >=38.0 ° C) or a history of fever in the 72 hours before the ED visit were included.

Twelve EDs from eight different European countries (Austria, Germany, Greece, Latvia, the Netherlands, Spain, Slovenia, and the United Kingdom) participated in the study. Participating hospitals were either tertiary university hospitals or large teaching hospitals. The characteristics of these hospitals are described in Appendix 1. Short stay units were not available at the participating hospitals at the time of data collection. Data were collected for at least one year, in the study period between January 2017 until April 2018. Data collection per month ranged from a one week per month sample to the entire month, depending on the number of ED visits per hospital (Appendix 1).

Sample size was estimated based on Riley et al. [13,14] Assuming 20 predictors, an admission prevalence of 30% and an expected R2 of 0.145 based on previous literature, a sample size of 1,139 with 342 admissions would be sufficient for any hospital admission. Based on a prevalence of 0.4% and a maximum achievable R2 of 0.05, a sample size of 7,724 with 31 admissions would be sufficient for ICU admission.

Data collection and definitions

Data were entered into the patient's records as part of routine care by the treating physician and nurses and were then manually extracted from these records and manually entered into an electronic case report form. The data entered into the case report form were specifically collected for the MOFICHE study according to a prespecified study protocol. The collected data included general patient characteristics (age, sex, comorbidity, medical care in the last five days, time of arrival, referral (self, primary care physician, Emergency Medical Services (EMS) or other), triage urgency, vital signs, presence of “red traffic light” symptoms for identifying risk of serious illness (alarming signs) (National Institute for Health and Care Excellence (NICE) guideline on fever [15] and disposition. Comorbidity was defined as a chronic underlying condition that is expected to last at least one year. Complex comorbidity was defined as a chronic condition in two or more body systems, malignancy, or immunocompromised patients [16]. Disposition was defined as patient destination after the ED: discharge, admission, or paediatric intensive care unit (PICU) admission. Any hospital admission was defined as general ward admission, or PICU admission. Long hospital admission was defined as general ward admission longer than 24 hours. Admission with an intervention was defined as any admission with either intravenous antibiotics, oxygen therapy, or one or more immediate life-saving interventions. Immediate life-saving interventions were adapted from Lee et al. [17] and were categorised into the following categories: airway and breathing support, electrical therapy (e.g. defibrillation), emergency procedures, hemodynamic support and emergency medications (Appendix 2).The NICE alarming signs include reduced consciousness, ill appearance, increased work of breathing, dehydration, age less than three months, non-blanching rash, meningeal signs, status epilepticus, and focal neurological signs. Different signs of dehydration (dry mucous membranes, sunken eyes, and reduced skin turgor) were grouped together as dehydration. Information on cyanosis of the skin was not available (Appendix 3).

Paediatric Early Warning System (PEWS) scores were calculated based on the PEWS developed by Parshuram (vital signs, capillary refill time, work of breathing, and oxygen therapy, combined into a score, Appendix 4) [18]. A modified PEWS was used as blood pressure was excluded from the PEWS as it was not routinely performed in our study. A previous study showed that a simplified PEWS not containing blood pressure showed similar performance in predicting PICU admission in comparison to the original full PEWS [19].

Data quality was improved and standardised by the use of a digital training module for treating physicians at the ED who assess febrile children, in order to reduce missing values and improve uniform data quality, including the clarification of the NICE alarming signs for the local research teams. Clinical data were entered into a standardised case report form by trained research team members. Furthermore, monthly teleconferences and biannual meetings were organised and quarterly reports of data quality were discussed with all ED partners.

Missing data

Patients with missing disposition (e.g. discharge, hospital admission) were excluded from the analysis. Missing determinants such as heart rate and respiratory rate were handled by using multiple imputation. Imputation was performed by using the MICE package in R, version 3.5. SPSS version 25 and R version 3.5.1 were used for the analysis of the data.

Data analysis and model construction

Data analysis was performed according to a pre-specified analysis plan. First, we performed a descriptive analysis for the frequency of general patient characteristics, vital signs, PEWS scores and presence of NICE alarming signs (Table 1). Patient characteristics between discharged and admitted children were compared using chi-squared-tests and Mann-Whitney tests. Results were deemed significant with a p-value < 0.05.

Table 1

patient characteristics, original data (N=38,424)*

	Discharged N = 28,531 N (%)	Any admission N = 9,892 N (%)	Admission>24 hours N = 7,258 N (%)	PICU admission N = 157 N (%)	Missing N (%)
Age in years, median (IQR)	2.9 (1.4-5.6)	2.3 (1.1-5.3)	2.3 (1.1-5.3)	2.4 (0.8-6.8)	0 (0.0)
Male	15,645 (54.8)	5,433 (54.9)	3,999 (55.1)	85 (54.1)	1 (0.0)
Comorbidity	3,786 (13.2)	2,265 (22.9)	1,703 (23.4)	79 (50.4)	366 (1.0)
Complex	835 (2.9)	804 (8.1)	612 (8.4)	42 (26.8)
Referral					1,161 (3.0)
Self	17,694 (62.0)	3,510 (35.5)	2,461 (33.9)	27 (17.2)
GP/private paediatrician	3,591 (12.6)	2,810 (28.4)	2,046 (28.2)	28 (17.8)
Emergency medical service	3,544 (12.4)	2,036 (20.6)	1,701 (23.4)	49 (31.2)
Other	2,746 (9.6)	1,332 (13.5)	879 (12.1)	48 (30.6)
Triage urgency					1,174 (3.1)
High: immediate, very urgent, intermediate	8,245 (28.9)	4,961 (50.1)	3,724 (51.3)	121 (77.1)
Low: non-urgent, standard	19,572 (68.6)	4,472 (45.2)	3,241 (44.7)	25 (15.9)
Vital signs
Tachycardia⁎⁎	6,239 (21.9)	3,324 (33.6)	2,381 (32.8)	95 (60.5)	3,492 (9.1)
Tachypnoea⁎⁎	3,398 (11.9)	2,276 (23.0)	1,562 (21.5)	70 (44.6)	8,773 (22.8)
Hypoxia, oxygen saturation ≤94%	259 (0.9)	596 (6.0)	414 (5.7)	33 (21.0)	5,558 (14.5)
Prolonged capillary refill >=3	100 (0.4)	323 (3.3)	217 (3.0)	32 (20.4)	4,414 (11.5)
Oxygen therapy	123 (0.4)	968 (9.8)	728 (10.0)	83 (52.9)	120 (0.3)
PEWS
5 or higher	481 (1.7)	994 (10.0)	695 (9.6)	61 (38.9)	16,639 (43.3)
NICE “red traffic lights” (alarming signs)
Decreased consciousness	44 (0.2)	156 (1.6)	93 (1.3)	32 (20.4)	378 (1.0)
Ill appearance	2,426 (8.5)	3,577 (36.2)	3,014 (41.5)	91 (58.0)	1,702 (4.4)
Increased work of breathing	1,434 (5.0)	1,825 (18.4)	1,186 (16.3)	76 (48.4)	4,857 (12.6)
Dehydration	746 (2.6)	1,154 (11.7)	885 (12.2)	19 (12.1)	6,956 (18.1)
Fever < 3 months of age	336 (1.2)	720 (7.3)	554 (7.6)	14 (8.9)	1,056 (2.7)
Rash: petechiae/non-blanching	695 (2.4)	415 (4.2)	283 (3.9)	13 (8.3)	4,394 (11.4)
Meningeal signs	21 (0.1)	116 (1.2)	90 (1.2)	7 (4.5)	2,029 (5.3)
Seizures	629 (2.2)	748 (7.6)	484 (6.7)	21 (13.4)	1,138 (3.0)
Status epilepticus	7 (0.0)	59 (0.6)	29 (0.4)	17 (10.8)	1,138 (3.0)
Focal neurology	25 (0.1)	108 (1.1)	60 (0.8)	18 (11.5)	2,438 (6.3)

All comparisons p<0.0001 except gender.

According to APLS cut-off values by age.

A prediction model for hospitalisation was constructed using multivariable logistic regression analysis. We used a stepwise approach in which models with the following sets of variables were tested separately, and then were subsequently combined into the final model: general patient characteristics (age, sex, comorbidity, medical care in the last five days, time of arrival, mode of referral), vital signs (heart rate, respiratory rate, capillary refill time, oxygen saturation, and temperature), PEWS scores, and NICE alarming signs (Table 2).

Table 2

Model construction and final model

	Model construction	Final model
General patient characteristics	Age, gender, comorbidity, medical care in the last 5 days, time of arrival, referral, triage urgency	Age, comorbidity, referral, triage urgency
Vital signs+	Temperature ≥38.0, tachycardia, tachypnoea, oxygen saturation ≤94%, capillary refill time > 3 seconds	Tachycardia, tachypnoea, oxygen saturation ≤94%, capillary refill time > 3 seconds
NICE alarming signs	Reduced consciousness, ill appearance, increased work of breathing, dehydration, fever < 3 months, non-blanching rash, meningeal signs, seizures/status epilepticus, focal neurological signs	Reduced consciousness, ill appearance, increased work of breathing, dehydration, fever < 3 months, non-blanching rash, meningeal signs, seizures/status epilepticus, focal neurological signs
PEWS	Heart rate, prolonged capillary refill, respiratory rate, work of breathing, oxygen saturation and oxygen therapy, combined into the modified PEWS score

according to APLS cut-offs by age

Vital signs were tested with predefined cut-off values according to APLS and PEWS reference values as these are age-dependent. Individual variables that did not improve the AUC of the model significantly were removed. As PEWS scores and vital signs have considerable overlap, we added PEWS scores and vital signs separately and not simultaneously to the other sets of variables and chose the best performing model as the final model.

We defined our outcome measures as: any admission, admission longer than 24 hours, PICU admission and admission with an intervention.

We determined the discriminative value of the model by calculating the area under the receiver operating characteristic (ROC) curves (AUC) and evaluated the predictive performance (sensitivity, specificity, positive and negative likelihood ratio (LR) and positive and negative predictive value (PV) of different risk thresholds to obtain a cut-off with a high specificity for hospital admission as our aim was to identify children at high risk for admission. The prediction model was derived from the complete set and the final model was subsequently validated using the “leave-one-setting-out cross-validation” method, a validation method that addresses between-setting heterogeneity [20], which is relevant in our study as it included different settings with a different patient case mix and different admission rates. With this method, the model is derived in all settings except one and then validated in the setting that was left out, repeated with each setting being left out once, leading to 12 validations; the separate results from these 12 validations are pooled [20] by using the R metaphor package. In addition to this, the prediction model was validated in a separate dataset from a previous study on febrile children at different European EDs [21]. This second dataset included data from 28 European EDs and 5,177 paediatric visits and consisted of a mixture of university (17 settings, 3,807 patients), teaching (10 settings, 1,299 patients) and non-teaching hospitals (1 setting, 71 patients), paediatric as well as mixed adult paediatric EDs and inner-city as well as regional hospitals. [21] Detailed information regarding this dataset is described in the original research article.

Role of the funding source

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No. 668303

The Research was supported by the National Institute for Health Research Biomedical Research Centre based at Imperial College (JH, ML) and at Newcastle Hospitals NHS Foundation Trust and Newcastle University (EL, ME).

The funders had no role in the study design, data collection, data analysis, interpretation, or writing of the report. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

Results

After excluding 56 paediatric visits with missing disposition, 38,424 paediatric encounters were included

The different EDs varied in patients who had any comorbidity (range: 5.2-65.6%), were self-referred (range: 0.6-94.9%), were triaged as high urgent (range: 8.8-89.9%) or were ill appearing (range: 0.9-50.3%) (Table 1). Comorbidity is described in more detail in Appendix 5.

In total, 9,735 children were admitted to a general ward (25.3%, range EDs 5.1-54.5%) and 157 were admitted to the PICU (0.4%, range EDs 0.1-4.0%). 7,258 children were admitted for longer than 24 hours (74.6% of general ward admissions) and 4,268 children were admitted with an intervention (43.1% of all admissions).

Children requiring admission were younger (median age 2.3 versus 2.9 years), more often had comorbidity (24.4 versus 14.2%) and more often were referred or brought in by Emergency Medical Services (EMS) (20.6 versus 12.4%) than those discharged home (p <0.001). Admitted children also had a higher triage urgency (50.1 versus 28.9%), more often had abnormal vital signs (tachycardia 33.6 versus 21.9, tachypnoea 23.0 versus 11.9%), a higher PEWS score (PEWS score of five or higher: 10.0 versus 1.7%) or had one or more NICE alarming signs present (for example ill appearance 36.2 versus 8.5%) than children that were discharged home (p <0.001, Table 1).

Prediction model

Multivariable odds ratios for patient characteristics, vital signs and NICE alarming signs in relation to hospital admission are shown in Table 3 and Appendix 6.

Table 3

multivariable odds ratios for any admission, admission>24 hours and PICU admission

		Any admission	admission>24hours	PICU admission
		OR (95% CI)	OR (95% CI)	OR (95% CI)
General patient characteristics
Age	< 3 months	8.8 (7.6-10.3)	7.1 (6.1-8.3)	3.1 (1.6-6.3)
	3-12 months	1.4 (1.3-1.5)	1.4 (1.3-1.5)	1.8 (1.1-2.9)
	1-4 years	Reference	Reference	Reference
	5-12 years	1.1 (1.0-1.1) #	1.1 (1.0-1.1) #	1.9 (1.2-3.0)
	≥12 years	1.4 (1.3-1.6)	1.4 (1.2-1.6)	1.4 (0.7-2.8) #
Comorbidity	Present	1.9 (1.8-2.1)	2.1 (1.9-2.2)	2.2 (1.4-3.3)
Referral	Self	Reference	Reference	Reference
	Primary care physician	1.6 (1.5-1.8)	1.4 (1.3-1.6)	1.5 (0.8-3.0) #
	Emergency medical service	1.3 (1.2-1.4)	1.3 (1.1-1.4)	1.8 (1.0-3.2) #
	Other	1.9 (1.7-2.1)	1.9 (1.7-2.2)	4.7 (2.5-8.6)
Triage urgency	High urgent	2.1 (1.9-2.2)	2.0 (1.9-2.2)	2.0 (1.2-3.6)
Vital signs
Tachycardia*	Present	1.1 (1.0-1.1) #	1.1 (1.0-1.2) #	2.4 (1.6-3.5)
Tachypnea*	Present	1.3 (1.2-1.5)	1.3 (1.2-1.4)	1.3 (0.8-2.0) #
Oxygen saturation	≤94%	2.6 (2.2-3.1)	1.9 (1.6-2.3)	3.0 (1.8-5.2)
Capillary refill time	≥3 seconds	2.0 (1.5-2.7)	1.1 (0.8-1.5) #	3.7 (2.2-6.4)
NICE alarming signs
	Reduced consciousness	2.2 (1.4-3.4)	0.7 (0.5-1.1) #	8.8 (4.6-16.6)+
	Ill appearance	4.9 (4.6-5.3)	4.4 (4.1-4.7)	5.6 (3.4-9.3)
	Increased work of breathing	3.1 (2.8-3.4)	1.9 (1.7-2.1)	3.7 (2.3-5.9)
	Dehydration	3.9 (3.5-4.4)	2.6 (2.3-2.9)	1.3 (0.7-2.5) #
	Non-blanching rash	2.1 (1.8-2.5)	1.7 (1.4-2.0)	2.7 (1.4-5.4)
	Meningeal signs	6.8 (3.8-12.2)	3.4 (2.3-5.2)	2.2 (0.7-6.2) #
	Seizures	3.5 (3.0-4.0)	2.1 (1.8-2.5)	1.5 (0.8-3.1) #
	Status epilepticus	7.3 (3.0-17.9)	1.5 (0.8-2.5) #	5.7 (2.1-15.1)
	Focal neurological signs	4.3 (2.3-8.0)	1.2 (0.7-2.1) #	2.1 (0.8-5.5) #

according to APLS age related cut-offs

all values: p <0.001, except

= p: 0.001

= non-significant.

When tested separately, NICE alarming signs (AUC 0.81, 95% CI 0.80-0.82) performed better than patient characteristics (AUC 0.72, 95%CI 0.71-0.72), vital signs (AUC 0.72, 95% CI 0.72-0.73) and PEWS (AUC 0.73, 95% CI 0.73-0.74), displayed in detail in Fig. 1).

Fig. 1

ROC curves of the separate risk factors and combined model with patient characteristics, vital signs and NICE alarming signs

Ia. Any admission

1b. Admission>24 hours

1c. PICU admission

Final model

The combination of general patient characteristics, vital signs, and NICE alarming signs (final model), yielded an AUC of 0.84 (95% CI 0.83-0.84) in predicting any hospitalisation.

Several variables, including some of the NICE clinical alarming signs, were removed from the final model, as they did not significantly improve the model when included in the final model (Tables 2 and 3, Appendix 6). Adding PEWS scores to the final model instead of vital signs did not improve the model and thus PEWS scores were left out in the final model. Variables included in the final model are displayed in Table 2.

Cross-validation, external validation and calibration

Using the leave-one-setting-out cross-validation method, the final model, performed equally well, with AUC's ranging from of 0.82 to 0.90 and a pooled AUC of 0.82 (95% CI 0.80-0.84, Appendix 7).

Validation in the separate dataset yielded an excellent pooled AUC of 0.90 (95% CI 0.89- 0.91) in the whole dataset and 0.91 (95% CI 0.90- 0.93) in the subset of non-university hospitals only.

Calibration differed between settings and ranged between poor-moderate to excellent with significant changes in intercept (range of -1.1 to +1.3) and slope (range of 0.6 to 1.6) (Appendix 8). Calibration was good for the external validation dataset (intercept 0.1, slope 0.9, p < 0.001).

An intercept of 0 and a slope of 1 mean perfect calibration, while an intercept <0 is suggestive of overestimation and an intercept >0 is suggestive of underestimation due to differences in the prevalence of the event, in this case hospital admission. A slope of <1 suggests that risk estimation by the model is too extreme (i.e., too high for patients with a high risk and too low for patients with a low risk) and a slope >1 suggests that risk estimation is too moderate. [22]

Adjustments in intercept and slope after calibration reflect differences between the different settings, for example due to differences in admission risk per setting, differences in sample size or case mix variables not complete captured by correction for ED setting in the model.

In our prediction model, overall calibration showed considerable unexplained heterogeneity; possibly caused by the factors mentioned above. To demonstrate the impact of different baseline admission risks, calibration was performed for settings depending on admission risk prevalence, analysing settings with low (<20%), intermediate (20-40%) and high (>40%) admission risk separately (Appendix 8). This analysis showed excellent calibration for settings with a low and intermediate admission risk but poor calibration for settings with a high admission. However, as an admission rate of >40% is only the case in a minority of European EDs [2,21], this supports the applicability of our prediction model in clinical practice.

Model performance at different thresholds

The model performed well for a rule-in threshold of 75% with a specificity of 99.0% (95% CI 98.9-99.1%, positive likelihood ratio of 15.1 (95% CI 13.4-17.1), and positive predictive value of 0.84 (95% CI 0.82-0.86) and a rule-out threshold of 7.5% with a sensitivity of 95.4% (95% CI 95.0-95.8), a negative likelihood ratio of 0.15 (95% CI 0.14-0.16), and a negative predictive value of 0.96 (95% CI 0.95-0.96). Detailed information on model performance at different thresholds is provided in Table 4.

Table 4

Sensitivity and specificity of the combined model (general patient characteristics, vital signs, NICE alarming signs) for the prediction of hospitalisation*

Risk threshold	N below threshold (%)	N above threshold (%)	Sensitivity in % (95% CI)	Specificity in % (95% CI)	Positive likelihood ratio (95% CI)	Negative likelihood ratio (95% CI)	Positive predictive value	Negative predictive value
Any admission
>5.5%	5,633 (15)	32,791 (85)	97.5 (97.2-97.8)	18.9 (18.4-19.3)	1.20 (1.19-1.21)	0.13 (0.12-0.15)	0.29 (0.29-0.30)	0.96 (0.95-0.96)
>7.5%	9,167 (24)	29,257 (76)	95.4 (95.0-95.8)	30.5 (30.0-31.1)	1.37 (1.36-1.39)	0.15 (0.14-0.16)	0.32 (0.32-0.33)	0.95 (0.95-0.96)
> 10%	13,968 (36)	24,457 (64)	89.6 (89.0-90.2)	45.4 (44.8-45.9)	1.64 (1.62-1.66)	0.23 (0.22-0.24)	0.36 (0.36-0.37)	0.93 (0.92-0.93)
>15%	20,834 (54)	17,591 (46)	80.0 (79.2-80.8)	66.1 (65.5-66.7)	2.36 (2.32-2.41)	0.30 (0.29-0.31)	0.46 (0.45-0.46)	0.91 (0.90-0.91)
> 25%	25,997 (68)	12,427 (32)	67.9 (67.0-68.8)	80.0 (79.5-80.4)	3.39 (3.30-3.48)	0.40 (0.39-0.41)	0.54 (0.53-0.55)	0.88 (0.87-0.88)
> 40%	30,646 (80)	7,778 (20)	49.6 (48.6-50.6)	90.0 (89.9-90.3)	4.94 (4.74-5.14)	0.56 (0.55-0.57)	0.63 (0.62-0.64)	0.84 (0.83-0.84)
> 50%	33,436 (87)	4,988 (13)	35.8 (34.9-36.8)	95.0 (94.7-95.2)	7.09 (6.70-7.51)	0.68 (0.67-0.69)	0.71 (0.70-0.72)	0.81 (0.81-0.81)
> 75%	36,625 (95)	1,799 (5)	15.3 (14.6-16.0)	99.0 (98.9-99.1)	15.1 (13.4-17.1)	0.86 (0.85-0.86)	0.84 (0.82-0.86)	0.77 (0.77-0.78)
Admission>24 hours
>4.0%	5,643 (15)	32,781 (85)	98.0 (97.7-98.3)	17.6 (17.2-18.1)	1.19 (1.18-1.20)	0.11 (0.09-0.13)	0.22 (0.21-0.22)	0.97 (0.97-0.98)
>5.0%	11,043 (29)	27,381 (71)	94.9 (94.3-95.4)	34.2 (33.7-34.8)	1.44 (1.43-1.46)	0.15 (0.14-0.17)	0.25 (0.25-0.26)	0.97 (0.96-0.97)
>6.0%	14,297 (37)	24,127 (63)	90.0 (89.4-90.8)	43.6 (43.0-44.1)	1.60 (1.58-1.62)	0.23 (0.21-0.24)	0.27 (0.27-0.28)	0.95 (0.95-0.95)
>10%	21,270 (55)	17,155 (45)	80.0 (79.0-80.9)	63.6 (63.0-64.1)	2.20 (2.16-2.24)	0.32 (0.30-0.33)	0.34 (0.33-0.35)	0.93 (0.93-0.93)
>17.5%	27,662 (72)	10,762 (28)	62.8 (61.7-63.9)	80.0 (79.6-80.5)	3.16 (3.07-3.25)	0.46 (0.45-0.48)	0.42 (0.41-0.43)	0.90 (0.90-0.91)
> 25%	32,103 (84)	6,321 (16)	43.9 (42.7-45.0)	90.0 (89.6-90.3)	4.4 (4.2-4.5)	0.62 (0.61-0.64)	0.50 (0.49-0.52)	0.87 (0.87-0.88)
> 40%	34,785 (91)	3,639 (9)	28.6 (27.5-29.6)	95.0 (94.7-95.2)	5.69 (5.36-6.05)	0.75 (0.74-0.76)	0.57 (0.55-0.59)	0.85 (0.85-0.85)
> 65%	37,473 (97)	951 (3)	8.7 (8.1-9.4)	99.0 (98.9-99.0)	8.63 (7.56-9.85)	0.92 (0.92-0.93)	0.67 (0.64-0.70)	0.82 (0.82-0.83)
PICU admission
> 0.1%	19,592 (51)	18,832 (49)	98.1 (94.1-99.5)	51.2 (50.7-51.7)	2.01 (1.96-2.0.6)	0.04 (0.01-0.11)	0.01 (0.01-0.01)	1.00 (1.00-1.00)
> 0.15%	28,761 (75)	9,663 (25)	94.9 (89.9-97.6)	74.5 (74.1-75.0)	3.73 (3.58-3.88)	0.07 (0.03-0.13)	0.02 (0.01-0.02)	1.00 (1.00-1.00)
>0.25	31,821 (83)	6,603 (17)	90.4 (84.5-94.4)	83.1 (82.7-83.4)	5.36 (5.07-5.66)	0.11 (0.07-0.19)	0.02 (0.02-0.03)	0.98 (0.97-0.98)
> 0.5%	34,483 (90)	3,941 (10)	82.1 (75.1-87.6)	90.0 (89.8-90.3)	8.27 (7.64-8.95)	0.20 (0.14-0.28)	0.03 (0.03-0.04)	1.00 (1.00-1.00)
>1.0%	36,432 (95)	1,879 (5)	0.71 (0.63-0.78)	95.1 (94.9-95.3)	14.5 (13.0-15.4)	0.31 (0.24-0.39)	0.06 (0.05-0.07)	1.00 (1.00-1.00)
>5%	37,946 (99)	478 (1)	49.0 (41.0-57.1)	99.0 (98.8-99.1)	46.8 (38.8-56.4)	0.51 (0.44-0.60)	0.16 (0.13-0.20)	1.00 (1.00-1.00)
Admission with an intervention⁎⁎
>2.5	5,738 (15)	32,685 (85)	98.0 (97.5-98.4)	16.6 (16.2-17.0)	1.17 (1.17-1.18)	0.15 (0.10-0.15)	0.13 (0.12-0.13)	0.99 (0.98-0.99)
>3.0	10,805 (28)	27,619 (72)	95.7 (95.0-96.2)	31.1 (30.6-31.6)	1.39 (1.38-1.40)	0.14 (0.12-0.16)	0.15 (0.14-0.15)	0.98 (0.98-0.99)
>4.0%	15,409 (40)	23,019 (60)	90.0 (88.7-90.6)	43.8 (43.2-44.3)	1.60 (1.57-1.62)	0.24 (0.22-0.26)	0.17 (0.16-0.17)	0.97 (0.97-0.97)
>6%	22,350 (58)	16,074 (42)	80.0 (78.8-81.2)	62.9 (62.4-63.4)	2.16 (2.12-2.20)	0.32 (0.30-0.34)	0.21 (0.21-0.22)	0.96 (0.96-0.96)
>10%	28,826 (75)	9,582(25)	65.6 (64.1-67.0)	80.0 (79.7-80.5)	3.29 (3.19-3.39)	0.43 (0.41-0.45)	0.29 (0.28-0.30)	0.95 (0.95-0.95)
>15%	32,980 (86)	5,444 (14)	48.2 (46.7-49.7)	90.0 (89.8-90.4)	4.86 (4.65-5.08)	0.58 (0.56-0.59)	0.38 (0.36-0.39)	0.93 (0.93-0.94)
>25%	35,223 (92)	3,201 (8)	35.0 (33.6-36.5)	95.0 (94.8-95.2)	7.02 (6.60-7.46)	0.68 (0.67-0.70)	0.47 (0.45-0.48)	0.92 (0.92-0.92)
>45%	37,407 (97)	1017 (3)	15.8 (14.7-16.9)	99.0 (98.9-99.0)	15.6 (13.8-17.7)	0.85 (0.84-0.86)	0.66 (0.63-0.69)	0.90 (0.90-0.91)

Baseline risk general admission: 26%; admission>24 hours: 19%, PICU admission 0,4%.

Defined as either intravenous antibiotic treatment, oxygen therapy or any immediate life-saving interventions (e.g. haemodynamic support or emergency medications)

Model performance for different types of admission and different patient groups

Separate analyses were performed for admission longer than 24 hours (AUC 0.84, 95% CI 0.83-0.84 and admission with an intervention (AUC 0.83, 95% CI 0.82-0.83) (Table 4, Fig. 1). Furthermore, the model was able to predict PICU admission with an AUC of 0.95 (95% CI 0.94-0.95) and with cross-validated AUC's ranging from of 0.90 to 0.99 and a pooled AUC of 0.95 (95% CI 0.93-0.98, Appendix 7). Detailed information on model performance at different thresholds is shown in Table 4.

Secondly, separate analyses were performed for different age groups and by focus of infection, which all showed results consistent with the analyses of the complete dataset.

Running the prediction model in a dataset in which patients with comorbidity were excluded, yielded a similar AUC of 0.84 (95% CI 0.84-0.85).

Lastly, separate analyses were performed by grouping hospitals together by admission rates. The model performed well in the hospitals with low (0-20% admissions, AUC 0.83, 95% CI 0.82-0.85) and intermediate (20-40% admissions, AUC 0.81, 95% CI 0.80-0.82) admission rates and moderately well in the hospitals with high admission rates of over 40% (AUC 0.78 (95% CI 0.77-0.79, Appendix 9).

Digital calculator

Based on the cross-validated and externally validated prediction rule described above, a digital calculator was developed to illustrate and facilitate clinical use (Fig. 2). A ≥ 75% admission risk was chosen as a cut-off to consider admission at triage as this corresponded with a specificity of 99.0% (95% CI 98.9-99.1%), a positive likelihood ratio of 15.1 (95% CI 13.4-17.1) and a positive predictive value of 0.84 (95% CI 0.82-0.86) (Table 4).

Fig. 2

Screenshot of digital admission risk calculator.

Clinical vignette

An 11-month-old, previously healthy girl, presents to the ED with fever and shortness of breath. She was referred by her GP. Her triage urgency is “very urgent”, she has a heart rate of 164 bpm, a respiratory rate of 60 bpm, oxygen saturation of 93% in room air and increased work of breathing.

Advice:

This girl has an 75% probability that she will be admitted.

An online demonstration of the digital calculator can be found at www.rimon.nl/arc.

Discussion

Main findings

We developed and validated a prediction model that can be used during the triage process for early admission of febrile children attending the ED. The combined model of vital signs and clinical alarming signs available at triage performed well and can identify febrile children at high risk for hospital admission. The prediction model can be used to improve flow by arranging admission during or immediately after triage for children classified high risk, even while they are still awaiting test results and treatment effects.

Although our prediction model was developed for “ruling-in” hospitalisation at triage, this does not imply that children not identified as “high-risk”, can safely be discharged. First of all, at a different threshold, the prediction model can be used to identify a group of children with a low admission risk. Children classified as low risk for hospitalisation still require evaluation, as they might still require diagnostic testing or treatment; however, they might classify for a “fast track” evaluation and identifying those children can help to improve patient flow and ED crowding as well. Secondly, there is a remaining “intermediate risk” group in which the decision to admit or discharge cannot be made at triage and further evaluation is needed before deciding on optimal disposition. Lastly, clinicians should be aware of the fact that the prediction model provides an estimation and can aid in expediting admission, but should not be used as a substitute for clinical judgement and a thorough evaluation.

We hypothesise that febrile children requiring hospitalisation are a heterogenous group and are admitted for different reasons, for example intravenous drug therapies, observation, unable to take oral fluid or medication, worried parents, or lack of transportation. As we created our prediction model based on current hospitalisation practices of febrile children, our model might be a better predictor of current admission practices than of actual necessary admission. However, the main aim of this study was to develop a prediction model that can be used to improve patient flow and ED crowding, regardless of the reason for admission. Furthermore, as the prediction model performed well for children admitted to the ICU, children admitted for over 24 hours and children that were admitted with a medical intervention, the prediction model seems to correspond well with necessary admissions. Lastly, many of the participating hospitals were university hospitals, in which available beds are scarce and actual admission is expected to correspond with necessary admissions.

Findings in relation to previous literature

Traditionally many studies have focused on either identifying children with severe bacterial infections or identifying which febrile children can be safely discharged. However, not all children with SBI have to be admitted and not all children that require hospital admission suffer from SBI, emphasizing the complexity of caring for febrile children and illustrating how hospital admission and serious bacterial infections are two different clinical outcomes.

In their review on clinical prediction models regarding the safe discharge of children at the ED, Irwin et al. outlined four possible outcomes of febrile children: serious bacterial infection (SBI) requiring admission (e.g. sepsis, meningitis), SBI but no admission required (e.g. urinary tract infection, uncomplicated pneumonia), presumed viral illness requiring admission (e.g. bronchiolitis with hypoxia, gastroenteritis with dehydration), or presumed viral illness not requiring admission (other viral infections) [23], [24], [25], [26], [27].

In line with this, previous studies showed that the NICE alarming signs seem to be better predictors for children requiring hospital admission than for SBI.

Regarding the use of vital signs in our prediction model, even though fever is known to impact heart rate and respiratory rate, a previous study showed that absolute heart rate was a better predictor for SBI than temperature corrected heart rate [28]. Furthermore, we included temperature in the original prediction model, but it was left out in the final model as it did not significantly improve the model.

The final model including general patient characteristics and vital signs did improve the model containing only NICE alarming signs for all types of admission. Furthermore, as these variables are routinely assessed at triage and can easily be incorporated into the “admission risk calculator”, adding them is not time-consuming at triage.

To our knowledge, two previous single-centre studies have looked into the prognostic value of NICE red alarming signs in predicting hospitalisation in febrile children, showing good performance [26,27].

In addition to these two studies, several previous studies have developed prediction models for admission of ED patients. However, many of those have focused on adult patients, specific paediatric patients such as children with asthma, are single setting studies, or use variables not available at triage, thus limiting their use to improve patient flow [8,[29], [30], [31], [32], [33]].

As our study focused on a broad group of febrile children and used a large cohort from different settings with large differences in patient case mix, the generalisability of our results is expected to be high [2,12,34,35]. Furthermore, our prediction model uses variables that are available at the triage process and can thus be used to improve patient flow and crowding.

Lastly, our prediction model can be automated and integrated into the triage process and, based on the “rule-out” and “rule-in” thresholds, can be used to provide advice regarding hospital admission to ED health care workers. A similar clinical calculator, the validated “feverkidstool” developed by our research group, can be used as an aid on antibiotic treatment in febrile children, showed high compliance, a reduction of inappropriate antibiotics and an improved targeting of antibiotic use [36,37].

Implications for clinical practice and research

Our prediction model can be used to improve patient flow and ED crowding by providing a tool that can be used at triage to initiate and expedite hospital admission for those children that will be admitted anyway. Test results and treatment effect can then further be monitored on the ward.

The negative effects of ED waiting times and crowding have motived UK policy makers to introduce the 4-hour rule, which states that 95% of all ED patients should be discharged or admitted within four hours [34].

Although this target has reduced waiting times, it shows that introduction of waiting time targets can also have unwanted side effects and other measures are also needed to improve ED crowding, flow, and patient care. For example, our previous European study showed that settings with a 4-hour rule have a significantly larger number of short admissions in comparison to settings that have not implemented such a rule [34].

Admitting high-risk patients at triage could potentially benefit all patients, as it can improve crowding as well as waiting times for both patients that were admitted at triage as well as patients remaining at the ED for further evaluation, by improving allocation of resources. Furthermore, it is expected to reduce health care costs, improve patient and health care worker satisfaction and possibly reduce unnecessary short admissions.

Studying the actual impact of the prediction model on outcomes such as patient safety, ED length of stay, ED crowding, use of resources, health care costs and patient and healthcare worker satisfaction is an important next step in implementation. Ideally, the prediction model would be integrated into the triage process, maximising the potential benefits of early admission.

Furthermore, assessment of clinician's acceptability and barriers and facilitators for implementation should be part of a future implementation study that includes the use of the digital calculator. Based on previous research, the following potential barriers can be identified for implementation of the prediction model: an intuitive rather than analytical decision-making process, patient factors not included in the prediction model and difficulty among clinicians dealing with probabilistic knowledge. Possible facilitators include adding an actionable recommendation rather than just showing the calculated risk, integrating and automating the prediction model with the clinician's workflow, providing the evidence behind the prediction model and using an outcome that is perceived as clinically relevant by physicians and patients [38]. Furthermore, the fact that we used a limited number of variables that are available at triage and variables that are already used in daily practice as they are part of the fever guideline is expected to increase ease of use and acceptability.

Based on these recommendations, we would suggest integrating the prediction model with the triage process and providing the clinician with an actual “admission advice” rather than merely a risk calculation. A previous study by our research group showed high compliance with a prediction model for antibiotic therapy [36,39].

Using a digital tool, either as part of the patient electronic health record or a stand-alone tool, offers benefits in terms of efficiency and accuracy but should be subjected to local and regional regulations, such as CE marking, before being implemented into routine care.

Strengths and limitations

The main strength of our study is the large number of patients and the fact that data were collected year-round from 12 EDs in eight European countries and included different hospitals with different patient case mixes, which largely increases the generalisability of the results. Furthermore, the model performed equally well after cross-validation and external validation, showing that the model performs well in settings with a different patient case mix and different admission rates. Although the prediction model was developed and cross-validated in a large number of hospitals with large differences in patient case mix, those were mainly university hospitals. However, external validation on a mixed set of other EDs, with a higher number of non-university and regional hospitals showed good performance. The validation was limited by the dominance of children with a low admission risk, thereby the validity of our model in febrile children with a high admission risk may need further confirmation. Furthermore, it would be valuable to validate the model in other settings, such as non-teaching hospitals and low- and middle-income settings.

The use of patient characteristics that are available directly at or within minutes after triage, increases the applicability of our prediction tool.

Although the alarming signs from the NICE fever guideline we used is targeted at children until the age of five and we included children until the age of 18, the majority of our study population was below the age of five. Furthermore, these alarming signs have been applied to older children in several previous studies [24,27] and have considerable overlap with alarming signs from the NICE sepsis guidelines for children of all ages [15].

Our results should be interpreted in light of the limitations of using routinely collected data. To improve data quality and completeness, all settings received instructions regarding the accurate documentation of patient characteristics such as NICE alarming signs and quality checks were preformed regularly. Except for “pale skin”, which was not routinely collected, the amount of missing data was limited. Data were missing with the following frequencies: general patient characteristics: 0-5%, vital signs: 0-23% NICE alarming signs 1-18% and disposition 0.1%. Missing data is described in more detail in Table 1.

The effects of missing data were further reduced by using multiple imputation for missing values [40].

Unfortunately, as the database was anonymised, there were no data available on recurrent admissions of the same child. However, we did try to adjust for revisits and “frequent visitors“ in two ways. First, any previous medical care in the last five days was included in the original model, but was left out of the final model as it did not significantly improve the final model.

Secondly, we performed a separate analysis in a dataset in which all children with comorbidity were excluded, which showed a similar AUC as in the original dataset.

Although we evaluated several potential predictors, it is possible that not all variables related to hospitalisation were captured. For example, blood pressure was only performed in a minority of children and thus was not included in the prediction model. Although a previous study showed that although hypotension is associated with serious illness in children, its sensitivity is limited as a routine measurement in all children attending the ED [41] and it is a late sign in critically ill children in comparison to adults.

Furthermore, it is possible that some children in our dataset were admitted for non-medical reasons, such as lack of transportation or inability to understand and follow-up on safety netting advice. As we did not collect data on these social reasons it is difficult to assess in what proportion social reasons had an impact on hospital admissions and whether this was different in the different settings. However, as most participating settings offered tertiary university care and available beds in these settings in general is limited the number of admissions for social reasons is expected to be low.

Although it is possible that different settings used different admission criteria [42], combining data from 12 settings and performing a cross-validation analysis and external validation, shows the robustness of our model.

In order to improve and standardise data quality a digital training was used to optimise clinical assessment. This might have impacted clinical care in the study period and might have improved the background rate of accurate decision-making regarding admission. However, the clinical alarming signs that were part of the training module and data collection were part of the fever guideline that was already applied at the participating EDs.

Conclusion

The combination of general patient characteristics, vital signs and NICE alarming signs available at triage, can be used to identify febrile children at high risk for hospitalisation at ED triage. A digital calculator is available to facilitate clinical use.

Contributors

All authors contributed to the design and data collection of the study. DB and NH verified and analysed the data. DB, NH and HM interpreted the data. DB, NH and HM drafted the manuscript. All authors critically evaluated and revised the manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. This publication is the work of the authors who will serve as guarantors for the contents of this paper.

Declaration of interests

DB, UB, EC, JD, ME, MF, NH, BK, FMT, HM, EL, ML, MP, IRC, FS, MT, CV, SY, DZ and WZ report grants from the European Union. Horizon 2020 research and innovation programme during the study conduct. FS reports a grant from the Slovenian Research Agency outside the submitted work.

MP reports a grant from Pfizer and financial support from Pfizer and Sanofi outside the submitted work. MF reports a grant from CSL Behring outside the submitted work. RN reports a grant from the National Institute for Health Research during the study conduct. ME reports financial support from the National Institute for Health Research Biomedical Research Centre based at Newcastle Hospitals NHS Foundation Trust and Newcastle University ng the study conduct. MT is a member of the Advisory Board of MSD and Pfizer, a member of the National Committee on Immunization Practices and a member of the national Scientific Advisory Group for the management of the pandemic.

All other authors declare no competing interests.

Data sharing statement

Individual participant data that underlie the results reported in this article, including a data dictionary, will be made available after de-identification to researchers who provide a methodologically sound proposal. Proposals should be directed to d.borensztajn@erasmusmc.nl.

To gain access, data requestors will need to sign a data access agreement.