Clinical prediction model to aid emergency doctors managing febrile children at risk of serious bacterial infections: diagnostic study.

OBJECTIVE: To derive, cross validate, and externally validate a clinical prediction model that assesses the risks of different serious bacterial infections in children with fever at the emergency department.
DESIGN: Prospective observational diagnostic study.
SETTING: Three paediatric emergency care units: two in the Netherlands and one in the United Kingdom. PARTICIPANTS: Children with fever, aged 1 month to 15 years, at three paediatric emergency care units: Rotterdam (n=1750) and the Hague (n=967), the Netherlands, and Coventry (n=487), United Kingdom. A prediction model was constructed using multivariable polytomous logistic regression analysis and included the predefined predictor variables age, duration of fever, tachycardia, temperature, tachypnoea, ill appearance, chest wall retractions, prolonged capillary refill time (>3 seconds), oxygen saturation <94%, and C reactive protein. MAIN OUTCOME MEASURES: Pneumonia, other serious bacterial infections (SBIs, including septicaemia/meningitis, urinary tract infections, and others), and no SBIs.
RESULTS: Oxygen saturation <94% and presence of tachypnoea were important predictors of pneumonia. A raised C reactive protein level predicted the presence of both pneumonia and other SBIs, whereas chest wall retractions and oxygen saturation <94% were useful to rule out the presence of other SBIs. Discriminative ability (C statistic) to predict pneumonia was 0.81 (95% confidence interval 0.73 to 0.88); for other SBIs this was even better: 0.86 (0.79 to 0.92). Risk thresholds of 10% or more were useful to identify children with serious bacterial infections; risk thresholds less than 2.5% were useful to rule out the presence of serious bacterial infections. External validation showed good discrimination for the prediction of pneumonia (0.81, 0.69 to 0.93); discriminative ability for the prediction of other SBIs was lower (0.69, 0.53 to 0.86).
CONCLUSION: A validated prediction model, including clinical signs, symptoms, and C reactive protein level, was useful for estimating the likelihood of pneumonia and other SBIs in children with fever, such as septicaemia/meningitis and urinary tract infections.

Introduction

Fever is among the most common presenting signs of illness in children. Between 10% and 20% of all paediatric visits to hospital emergency departments are due to febrile illnesses.1 2 3 To differentiate children who have a benign self limiting viral infection from the small proportion with serious bacterial infections, many prediction models have been proposed.4 5 6 7 8 9 Most of these prediction models have not, however, been validated,9 and those that have performed poorly in emergency department settings.10 Typically these models also attempt to predict the overall risk of serious bacterial infections and ignore the fact that many different types of bacterial infection are involved, each requiring discrete diagnostic and therapeutic management. In one of the most robust models to date, researchers showed that clinical signs and symptoms contribute differently to predicting the risk of particular serious bacterial infections—that is, pneumonia, urinary tract infection, and bacteraemia.11 In that study they also showed that a clinical model outperformed the clinician’s impression for assessing the risk of a serious bacterial infection. However, this prediction model, although accurate, involved the input of 26 clinical variables, which limits the feasibility of using the model in many clinical settings.11 This large number of clinical variables invites the development of a more practical prediction model with fewer variables. In addition, it would be worthwhile to include serum C reactive protein level in a prediction model, an important predictor of serious bacterial infections.12 13 14 C reactive protein is widely used in many emergency care settings in Europe and North America, and point of care versions of the test have been proved to be reliable when applied routinely in general practices.15 16 This potentially allows the use of rapid and minimally invasive C reactive protein tests in prediction models at the first clinical assessment.

We developed and externally validated a clinical prediction model, including both clinical characteristics and C reactive protein, to identify febrile children presenting to emergency care settings at increased risk of serious bacterial infections.

Methods

We performed a diagnostic study by first developing a clinical prediction model, with derivation and cross validation in two Dutch populations (Erasmus MC-Sophia and Haga-Juliana children’s hospitals, the derivation population), then externally validating the prediction model in a UK population (Coventry, broad validation population).17

Derivation populations used to develop prediction model

We prospectively enrolled all children (1 month to 15 years) presenting with fever at the emergency department of the Erasmus MC-Sophia children’s hospital, Rotterdam (2003-05), and the Haga-Juliana children’s hospital, the Hague (2007), the Netherlands. The Erasmus MC-Sophia children’s hospital is an inner city university hospital with 9000 annual emergency department visits (90% for basic paediatric emergency care). The Haga-Juliana children’s hospital is an inner city teaching hospital with 16 000 annual emergency department visits.

In Erasmus MC-Sophia febrile children were eligible to participate in the study if fever had been noted at home in the 24 hours before presentation or body temperature measured in the emergency department was ≥38.0°C.18 In Haga-Juliana children were eligible if their temperature when measured rectally in the emergency department was ≥38.0°C.19 In both populations we excluded children with chronic comorbidity (for example, malignancies, cystic fibrosis, severe psychomotor retardation) and those who received antibiotics in the week before the emergency department visit. In children who reattended the emergency department within five days of their first presentation we used only data from the first visit. Laboratory tests were obtained at the discretion of the attending emergency department doctor. Informed consent was obtained from the parents.18

Population used for external validation

To assess broad external validation of the prediction model we used a second large cohort of febrile children from a different setting. This cohort consisted of children (1 month to 15 years) who were recruited prospectively at the paediatric assessment unit at the University Hospital Coventry and Warwickshire NHS Trust, United Kingdom, in 2005 and 2006, if an acute infection was suspected by the parents, the referring clinician, or the triage nurse.20 The Coventry hospital is an inner city hospital delivering emergency care to 25 000 children annually. Children were eligible if fever was reported by the parents or was documented by the triaging nurse (≥38.5ºC). We excluded children with an increased risk of recurrent serious infections, including those with iatrogenic immunosuppression and haematological malignancies. Laboratory tests were obtained at the discretion of the attending emergency department doctor. Informed consent was obtained.20

Data collection

For the Erasmus MC-Sophia and the Haga-Juliana derivation populations trained nurses in paediatric emergency care routinely collected data using a structured data entry application form, which was customised for this study.21 For the Coventry validation population a triage emergency care nurse recorded data, including a questionnaire of presenting symptoms, prospectively in the medical or nursing records on arrival of the children.20 All data were collected blinded to final outcome measures.

Outcome measures

The outcome categories were pneumonia, other serious bacterial infections (SBIs), and no SBIs. Other SBIs included meningitis, septicaemia, urinary tract infections, and other (erysipelas, cellulitis, bacterial gastroenteritis, orbital cellulitis, bacterial upper airway infection, ethmoiditis, arthritis, osteomyelitis). The outcomes categories were chosen according to differences in clinical presentation, diagnostic management, and frequency of occurrence. Outcomes were defined a priori by positive bacteriological cultures results of normal sterile sites or by consensus diagnosis.22 In Erasmus MC-Sophia the diagnosis of pneumonia was based on the presence of nodular infiltrations or consolidation in the lung on chest radiographs assessed by two board certified radiologists blinded to clinical information (κ=0.64, 95% confidence interval 0.52 to 0.75). For the Haga-Juliana and the Coventry populations a single radiologist, not blinded to clinical information, interpreted the chest radiographs. We used a one week follow-up period to rule out the possibility of a missed diagnosis of serious bacterial infection and to avoid verification bias.23 Follow-up consisted of telephone contact or follow-up hospital visit and checking for repeat visits to the emergency department or hospital admissions. All data were collected independent of study outcome measures.

Predictor variables

We identified predictors of serious bacterial infection obtained from keynote papers on predictive research in children with fever, which were readily available at first assessment and have small interobserver variability.5 7 11 20 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 Desaturation was defined as oxygen saturation levels less than 94% measured by pulse oximetry.40 In the derivation populations the nurse scored ill appearance on a 2 point scale: ill versus non-ill appearance. In the Coventry validation population ill appearance was defined as a modified Yale score of more than 6.27 We included all candidate predictor variables in the model independent of their statistical contribution. Continuous variables were included as continuous to preserve maximal diagnostic information. We tested the linearity of continuous variables with restricted cubic splines using the Hmisc and Design library in R statistical software (R).41 42 A linear relation with the outcomes was found to be a good approximation for temperature and duration of fever; for C reactive protein we used a logarithmic transformation (lnCRP). We observed different risk profiles for children aged less than and more than 1 year, which we modelled as linear piecewise. This accounted for the differentiated risks and the uniqueness of signs and symptoms of children in all age groups. The risk of children aged less than 1 year was calculated: β(age <1 year)×age in years. The risk of children aged 1 or more years was calculated with: β(age <1 year)×1+β(age ≥1 year)×(age in years−1). Tachycardia and tachypnoea were defined according to age specific cut-offs of advanced paediatric life support.40 43 C reactive protein and duration of fever were truncated at the value of three standard deviations (upper limits: C reactive protein 225 mg/L, duration of fever six days).

Statistical analysis

We constructed a polytomous multivariable prediction model using the multinom function in the NNET library (R).42 Values for missing data were imputed 10 times using a multiple imputation process with the Mice logarithm (R). We assumed missing data to be missing at random. The imputation model included all candidate predictor variables, the outcome variable, and several relevant variables describing case mix of the patients, such as white blood cell count, admission to hospital, and antibiotic prescription. (See supplementary file appendix A for the results of the imputation process.)

Modelling procedure and validation

We developed a prediction model in the combined derivation populations of Erasmus MC-Sophia and Haga-Juliana (n=2717). We noted a decreasing incidence of children with septicaemia or bacterial meningitis: 21/1750 at Erasmus MC-Sophia (2003-05), 1/967 at Haga-Juliana (2007). Since we expect incidence to remain low as a result of current vaccination strategies,44 45 we introduced a 1:10 weighted analysis for children with septicaemia or bacterial meningitis in the Erasmus MC-Sophia population. We applied a cross validation by first deriving the prediction model in the Erasmus MC-Sophia population and then cross validating the prediction model in the Haga-Juliana population and vice versa. The prediction model was externally validated in the Coventry population. As the variable “duration of fever” was absent in the Coventry population, the prediction model was first refitted without this variable in the combined derivation populations.41

Model performance

To estimate the ability to discriminate between patients with different diagnoses we used the C statistic for pairs of patients. A pair consisted of either a patient with pneumonia and a patient without serious bacterial infection, or a patient with another serious bacterial infection and a patient without a serious bacterial infection.46 To assess calibration we compared the predicted risks of pneumonia and other SBIs with the observed proportions of pneumonia and other SBIs. We calculated diagnostic performance (sensitivity, specificity, positive and negative predictive values, and positive and negative likelihood ratios) for several risk thresholds (from 2.5% to 30%) of children with pneumonia compared with no SBI and other SBI and of children with other SBI compared with no SBI and pneumonia (epiR package in R).42 The diagnostic performance of the validation study was calculated after adjusting for the incidence of the outcome categories—that is, updating the model’s constants.

Presentation of the prediction model

A digital calculator was constructed to illustrate and accommodate potential clinical use. The calculator was based directly on the final polytomous regression formula.

Results

In the Erasmus MC-Sophia derivation population 2784 patients were eligible. Fifty three concerned control visits and were excluded. A further 722 were excluded because of chronic underlying disease and 259 owing to use of antibiotics in the week before the emergency department visit, resulting in 1750 included children (fig 1). In the Haga-Juliana derivation population 1182 patients were eligible. Thirty two concerned control visits and were excluded. A further 36 were excluded because of chronic comorbidity and 147 owing to use of antibiotics in the week before the emergency department visit, resulting in 967 included children (fig 1). The Coventry validation population consisted of 487 children with fever. Table 1 shows the general characteristics of the two derivation populations and the validation population. We observed a similar prevalence of pneumonia and other SBIs for the derivation populations but a substantially higher prevalence in the validation population. Populations differed for some characteristics, such as age, temperature, respiratory rate, heart rate, oxygen saturation, and ill appearance. Table 2 shows the distribution of the predictor variables for the three outcome categories and univariate polytomous regression coefficients in the combined derivation population.

Fig 1 Flowchart of derivation populations

Table 1

 General characteristics of included populations. Values are numbers (percentages) unless stated otherwise

Characteristics	Derivation population			External validation population
	Erasmus MC-Sophia (n=1750)	Haga-Juliana (n=967)		Coventry (n=487)
Predictor variables
Median (interquartile range) age (years)	1.8 (0.9-3.7)	1.5 (0.7-3.2)		2.5 (1.3-6.3)
Sex	n=1750	n=964		n=487
No (%) male	998 (57)	531 (55)		215 (44)
	n=1185	807		NA
Median (interquartile range) duration of fever (days)	2.0 (1.0-3.0)	2.0 (1.0-3.0)
	n=1699	n=967		n=483
Median (interquartile range) temperature (ºC)	39.0 (38.3-39.7)	38.8 (38.3-39.4)		38.1 (37.4-39.0)
	n=819	n=183		n=415
Median (interquartile range) respiratory rate (breaths/min)	36 (28-48)	48 (40-60)		34 (24-40)
	n=914	n=473		n=480
Median (interquartile range) heart rate (beats/min)	140 (120-160)	156 (140-172)		146 (122-163)
Oxygen saturation <94%	41 (2)	43 (4)		46 (9)
Capillary refill time >3 seconds	96 (5)	9 (1)		8 (2)
Chest wall retractions	97 (6)	108 (11)		91 (19)
Ill appearance	520 (30)	65 (7)		171 (73)
	n=780	n=317		n=209
Median (interquartile range) C-reactive protein (mg/L)	21 (7 to 54)	22 (7 to 56)		48 (18 to 123)
Outcome measures
Serious bacterial infection:	222 (13)	119 (12)		124 (25)
Pneumonia	105 (6)	66 (7)		59 (12)
Urinary tract infection	50 (3)	38 (4)		28 (6)
Septicaemia/meningitis	21 (1)	1 (0)		3 (1)
Other*	46 (3)	14 (1)		34 (7)
None	1528 (87)	848 (88)		363 (75)

When data were not available for all patients, total number of available data noted.

*Includes erysipelas, cellulitis, bacterial gastroenteritis, cellulitis orbitae, bacterial upper airway infection, ethmoiditis, arthritis, and osteomyelitis.

Table 2

 Predictor distribution and univariate polytomous odds ratios of derivation population (Erasmus MC-Sophia and Haga-Juliana combined). Values are numbers (percentages) unless stated otherwise

Variables	No SBI (n=2376)	Pneumonia (n=171)	Other SBIs* (n=170)	Odds ratio (95% CI)†
				Pneumonia	Other SBIs
Age:
Mean (95% CI) age	2.60 (2.49 to 2.71)	3.12 (2.71 to 3.53)	4.13 (3.52 to 4.75)	—	—
1 month to <1 year	—	—	—	5.89‡ (2.26 to 15.37)	0.42‡ (0.21 to 0.83)
≥1 year	—	—	1.01‡ (0.96 to 1.07)	1.18‡ (1.12 to 1.24)
Female	1008 (42)	80 (47)	98 (58)	1.19 (0.87 to 1.63)	2.09 (1.50 to 2.93)
Duration of fever (days)§	2.0 (1.9 to 2.1)¶	3.0 (2.7 to 3.3)¶	2.5 (2.2 to 2.8)¶	1.43 (1.31 to 1.57)	1.25 (1.12 to 1.38)
Temperature (°C)	38.9 (38.9 to 39.0)¶	39.3 (39.2 to 39.4)¶	39.1 (39.0 to 39.2)¶	1.71 (1.41 to 2.07)	1.22(1.00 to 1.50)
Tachypnoea	1377 (58)	127 (74)	98 (58)	2.10 (1.36 to 3.23)	0.98 (0.52 to 1.82)
Tachycardia	1015 (43)	86 (50)	73 (43)	1.36 (0.95 to 1.94)	1.03 (0.69 to 1.53)
Oxygen saturation <94%	61 (3)	23 (13)	0 (0 )	5.90 (3.55 to 9.80)	0.03 (0.00 to 12.44)
Capillary refill time >3 sec	85 (4)	7 (4)	13 (8)	1.15 (0.52 to 2.53)	1.25 (0.56 to 2.80)
Chest wall retractions	178 (8)	25 (15)	2 (1)	2.11 (1.35 to 3.32)	0.02 (0.00 to 1.31)
Ill appearance	475 (20)	53 (31)	57 (34)	1.80 (1.28 to 2.52)	1.62 (1.12 to 2.33)
lnCRP (ln(mg/L))**	2.52 (2.43 to 2.60)¶	3.86 (3.62 to 4.10)¶	4.16 (3.96 to 4.36)¶	2.26 (1.90 to 2.69)	3.02 (2.49 to 3.65)

SBI=serious bacterial infection.

*Other SBIs include septicaemia/meningitis, urinary tract infections, and others.

†No SBI as reference group.

‡Odds ratio: children <1 year=exp(β(age <1 year)×age in years), children ≥1 year=exp(1×β(age <1 year)+(age in years−1)×β(age ≥1 year)).

§Maximum duration six days.

¶Mean (95% CI).

**Maximum value ln(225) mg/L.

Model development and performance

Table 3 presents the multivariable odds ratios of the prediction model in the derivation populations. Important predictors of pneumonia were oxygen saturation less than 94% (odds ratio 4.92, 95% confidence interval 2.53 to 9.54), body temperature per °C (1.33, 1.05 to 1.69 per °C), tachypnoea (1.55, 0.99 to 2.42), duration of fever per day (1.24, 1.11 to 1.38 per day), and log transformation of C reactive protein (1.89, 1.58 to 2.27). The log transformation of C reactive protein (3.11, 2.50 to 3.87) was also an important predictor of other SBIs. Chest wall retractions and oxygen saturation less than 94% were useful to rule out the presence of other SBIs. The C statistic for the presence of pneumonia was 0.81 (95% confidence interval 0.73 to 0.88) and for other SBIs was 0.86 (0.79 to 0.92). Figure 2 shows that the predicted risks for pneumonia were higher in children who had pneumonia than in children with a diagnosis of no SBI. The predicted risks for pneumonia in children with other SBIs were also lower. The predicted risks for other SBIs ranged widely in children with a diagnosis of other SBI and were substantially higher than the predicted risks in children with a diagnosis of pneumonia and no SBI.

Table 3

 Polytomous odds ratios of final prediction model

Variables	Odds ratio (95% CI)
	Pneumonia	Other SBIs
Age:
1 month to <1 year	2.76* (0.99 to 7.69)	0.18* (0.08 to 0.39)
≥1 year	1.01* (0.95 to 1.08)	1.11* (1.05 to 1.18)
Female	1.14 (0.81 to 1.61)	2.01 (1.37 to 2.94)
Duration of fever (days)†	1.24 (1.11 to 1.38)	0.97 (0.86 to 1.10)
Temperature (°C)	1.33 (1.05 to 1.69)	0.98 (0.75 to 1.26)
Tachypnoea	1.55 (0.99 to 2.42)	0.90 (0.48 to 1.69)
Tachycardia	0.96 (0.63 to 1.46)	0.98 (0.62 to 1.56)
Oxygen saturation <94%	4.92 (2.53 to 9.54)	0.04 (0.00 to 19.22)
Capillary refill time >3 sec	0.84 (0.33 to 2.10)	1.35 (0.53 to 3.42)
Chest wall retractions	1.61 (0.93 to 2.76)	0.02 (0.00 to 1.85)
Ill appearance	1.18 (0.79 to 1.75)	1.31 (0.84 to 2.05)
lnCRP (ln(mg/L))‡	1.89 (1.58 to 2.27)	3.11 (2.50 to 3.87)
C statistic	0.81 (0.73 to 0.88)	0.86 (0.79 to 0.92)

SBIs=serious bacterial infections; lnCRP=log transformation of C reactive protein.

*Odds ratio: children <1 year=exp(β(age <1 year)×age in years), children ≥1 year=exp(1×β(age <1 year)+(age in years−1)×β(age ≥1 year)).

†Maximum six days.

‡Maximum ln(225) mg/L.

Fig 2 Predicted risks of pneumonia and other serious bacterial infections (SBIs) in children with a diagnosis of pneumonia, other SBIs, or no SBI

In the cross validation study the C statistic for pneumonia in the Haga-Juliana derivation population was 0.80 (95% confidence interval 0.70 to 0.90) and for other SBIs was 0.88 (0.76 to 0.99); the corresponding values in the Erasmus MC-Sophia derivation population were 0.77 (0.69 to 0.85) and 0.82 (0.76 to 0.88). Calibration for the cross validation study was adequate for both outcomes (see supplementary file appendix B).

A model including C reactive protein performed significantly better than a model including only clinical signs and symptoms. The C statistic of the clinical model for the prediction of pneumonia was 0.76 (0.68 to 0.84) and for other SBIs was 0.73 (0.66 to 0.80). Other model performance indicators also improved substantially by adding C reactive protein (see supplementary file appendix C). The complete case analysis (n=373, pneumonia n=58, and other SBIs n=44, see supplementary file appendix D) showed similar results, with a C statistic for pneumonia of 0.80 (0.67 to 0.93) and for other SBIs of 0.89 (0.79 to 1.00). Supplementary file appendix C shows the additional value of C reactive protein in the complete case analysis.

Low risk thresholds were useful to rule out the presence of pneumonia: a risk cut-off of 2.5% had a sensitivity of 0.90 (95% confidence interval 0.85 to 0.96) and a negative likelihood ratio of 0.24 (0.14 to 0.40, table 4 and fig 3). Risk thresholds of 10% or more were useful to identify children with pneumonia: for example, a risk threshold of 15% had a specificity of 0.92 (0.90 to 0.93) and positive likelihood ratio of 5.58 (4.31 to 7.23). Low risk thresholds could also rule out the presence of other SBIs: a risk threshold of 2.5% gave a sensitivity of 0.90 (0.84 to 0.96) and a negative likelihood ratio of 0.18 (0.11 to 0.30). Higher risk thresholds were useful to identify children with other SBIs: for example, a risk threshold of 15% had a specificity of 0.91 (0.90 to 0.93) and a positive likelihood ratio of 6.35 (5.16 to 7.82, table 4 and fig 3). Figure 4 shows the receiver operating characteristic curves for the prediction of pneumonia and other SBIs. The sensitivity and specificity for several risk thresholds are plotted in these receiver operating characteristic curves.

Table 4

 Diagnostic performance measures at different risk thresholds of final prediction model (derivation population)

Outcomes and risk thresholds	Sensitivity (95% CI)	Specificity (95% CI)	Predictive value (95% CI)			Likelihood ratio (95% CI)
			Positive	Negative		Positive	Negative
Pneumonia:
≥2.5%	0.90 (0.85 to 0.96)	0.40 (0.36 to 0.45)	0.09 (0.08 to 0.11)	0.98 (0.97 to 0.99)		1.51 (1.38 to 1.65)	0.24 (0.14 to 0.40)
≥5%	0.78 (0.71 to 0.86)	0.64 (0.61 to 0.67)	0.13 (0.11 to 0.15)	0.98 (0.97 to 0.99)		2.18 (1.96 to 2.44)	0.33 (0.24 to 0.46)
≥10%	0.60 (0.50 to 0.71)	0.84 (0.82 to 0.85)	0.20 (0.16 to 0.24)	0.97 (0.96 to 0.98)		3.73 (3.08 to 4.51)	0.47 (0.37 to 0.60)
≥15%	0.47 (0.37 to 0.56)	0.92 (0.90 to 0.93)	0.27 (0.21 to 0.33)	0.96 (0.95 to 0.97)		5.58 (4.31 to 7.23)	0.58 (0.48 to 0.70)
≥30%	0.16 (0.10 to 0.22)	0.99 (0.98 to 0.99)	0.43 (0.29 to 0.56)	0.95 (0.94 to 0.96)		11.07 (6.57 to 18.66)	0.85 (0.79 to 0.92)
Other SBIs:
≥2.5%	0.90 (0.84 to 0.96)	0.56 (0.53 to 0.60)	0.12 (0.10 to 0.14)	0.99 (0.98 to 1.00)		2.06 (1.87 to 2.26)	0.18 (0.11 to 0.30)
≥5%	0.82 (0.74 to 0.89)	0.72 (0.70 to 0.74)	0.16 (0.14 to 0.19)	0.98 (0.98 to 0.99)		2.92 (2.60 to 3.27)	0.26 (0.18 to 0.36)
≥10%	0.70 (0.61 to 0.79)	0.85 (0.84 to 0.87)	0.25 (0.20 to 0.29)	0.98 (0.97 to 0.99)		4.84 (4.10 to 5.71)	0.35 (0.26 to 0.46)
≥15%	0.55 (0.46 to 0.64)	0.91 (0.90 to 0.93)	0.30 (0.25 to 0.35)	0.97 (0.96 to 0.98)		6.35 (5.16 to 7.82)	0.49 (0.40 to 0.60)
≥30%	0.19 (0.12 to 0.26)	0.98 (0.97 to 0.99)	0.38 (0.26 to 0.50)	0.95 (0.94 to 0.96)		9.12 (5.54 to 15.01)	0.83 (0.76 to 0.91)

SBIs=serious bacterial infections.

Fig 3 Positive and negative likelihood ratios plotted for several risk thresholds for predicted risks of pneumonia and other serious bacterial infections (SBIs)

Fig 4 Receiver operating characteristic curve for risk of pneumonia and other serious bacterial infections (SBIs). Sensitivity and specificity of several risk thresholds of the prediction model are plotted

External validation of prediction model

As duration of fever was not recorded in the Coventry validation population, a model was refitted in the derivation populations without the variable “duration of fever”: the C statistic of this model for pneumonia was 0.80 (95% confidence interval 0.72 to 0.89) and for other SBIs was 0.86 (0.79 to 0.92), and all predictors had similar effects (data not shown). In the Coventry population the C statistic of this model to predict the presence of pneumonia was 0.81 (0.69 to 0.93). The discriminative ability of the prediction of other SBIs was substantially lower (0.69, 0.53 to 0.86). Overall, calibration was satisfactory (fig 5). The mean predicted risk of pneumonia in the Coventry population was 11% (95% confidence interval 10% to 13%) compared with an observed proportion of 12% (11% to 13%). The mean predicted risk of other SBIs was 16% (15% to 18%) compared with an observed proportion of 13% (12% to 15%). Agreement between the predicted and observed risks was reasonable given the low number of patients with pneumonia (n=59) or other SBIs (n=65). For the prediction of pneumonia, the confidence intervals of the observed probabilities covered the line of ideal calibration (fig 5). For other SBIs the confidence intervals of the observed probabilities covered the line of ideal calibration in the higher risk categories (fig 5). Low risk thresholds ruled out the presence of pneumonia. For example, a risk threshold of 2.5% or more had a negative likelihood ratio of 0.12 (0.02 to 0.84) and sensitivity of 0.99 (0.91 to 1.00, table 5). High thresholds were useful for ruling in pneumonia. For example, for a risk threshold of 15%, the positive likelihood ratio was 3.57 (2.58 to 4.94) and the specificity was 0.82 (0.78 to 0.87). Diagnostic performance measures for the risk of other SBIs were insufficient for either ruling in or ruling out the presence of other SBIs.

Fig 5 Calibration plot of predicted risks of pneumonia and other serious bacterial infections (SBIs) and observed frequencies (95% confidence interval). Triangles represent mean (predicted versus observed) risk estimates of outcomes by fifths of predicted risk. Dashed diagonal line represents ideal calibration. Distribution of predicted risks of patients with outcome (pneumonia n=59, other SBIs n=65) and other patients (pneumonia n=428, other SBIs n=422) is shown at bottom of graph

Table 5

 Diagnostic performance measures at different risk thresholds of final prediction model (validation population)

Outcomes and risk thresholds	Sensitivity (95% CI)	Specificity (95% CI)	Predictive value (95% CI)			Likelihood ratio (95% CI)
			Positive	Negative		Positive	Negative
Pneumonia:
≥2.5%	0.99 (0.91 to 1.00)	0.14 (0.10 to 0.18)	0.14 (0.11 to 0.17)	0.99 (0.91 to 1.00)		1.16 (1.10 to 1.22)	0.12 (0.02 to 0.84)
≥5%	0.91 (0.81 to 1.00)	0.38 (0.33 to 0.44)	0.17 (0.13 to 0.21)	0.97 (0.93 to 1.00)		1.48 (1.31 to 1.66)	0.23 (0.09 to 0.56)
≥10%	0.75 (0.61 to 0.90)	0.69 (0.63 to 0.74)	0.25 (0.18 to 0.31)	0.95 (0.92 to 0.98)		2.39 (1.92 to 2.99)	0.36 (0.21 to 0.59)
≥15%	0.63 (0.47 to 0.78)	0.82 (0.78 to 0.87)	0.33 (0.24 to 0.42)	0.94 (0.91 to 0.97)		3.57 (2.58 to 4.94)	0.45 (0.30 to 0.67)
≥30%	0.34 (0.21 to 0.48)	0.95 (0.92 to 0.98)	0.48 (0.31 to 0.66)	0.91 (0.88 to 0.94)		6.81 (3.70 to 12.55)	0.69 (0.56 to 0.85)
Other SBIs:
≥2.5%	0.80 (0.68 to 0.93)	0.33 (0.28 to 0.39)	0.16 (0.12 to 0.19)	0.92 (0.86 to 0.97)		1.21 (1.04 to 1.40)	0.59 (0.34 to 1.02)
≥5%	0.74 (0.61 to 0.87)	0.45 (0.40 to 0.51)	0.17 (0.13 to 0.22)	0.92 (0.87 to 0.97)		1.36 (1.14 to 1.64)	0.56 (0.36 to 0.89)
≥10%	0.70 (0.57 to 0.83)	0.63 (0.57 to 0.68)	0.22 (0.17 to 0.28)	0.93 (0.89 to 0.97)		1.87 (1.49 to 2.34)	0.49 (0.33 to 0.72)
≥15%	0.62 (0.48 to 0.77)	0.72 (0.67 to 0.77)	0.26 (0.19 to 0.33)	0.93 (0.89 to 0.96)		2.23 (1.69 to 2.92)	0.52 (0.36 to 0.75)
≥30%	0.42 (0.28 to 0.55)	0.90 (0.86 to 0.93)	0.38 (0.26 to 0.51)	0.91 (0.88 to 0.94)		4.00 (2.50 to 6.39)	0.65 (0.51 to 0.82)

SBIs=serious bacterial infections.

Discussion

We developed and externally validated a clinical prediction model to predict the presence of pneumonia and other serious bacterial infections (SBIs) in children with fever presenting to emergency care settings. Duration of fever, body temperature, tachypnoea, and low oxygen saturation were strong predictors of pneumonia. Chest wall retractions and low oxygen saturation were important to rule out the presence of other SBIs. Increased levels of C reactive protein were associated with the presence of both pneumonia and other SBIs. Low risk thresholds less than 2.5% were able to rule out, and high risk thresholds of 10% or more were able to rule in, the presence of pneumonia and other SBIs. The model was useful for ruling in and ruling out the presence of pneumonia in external validation; discriminative ability for other SBIs was lower.

Comparison with existing literature

Clinical prediction models in children fulfilling existing performance standards (that is, derivation, validation, broad validation, and implementation) are scarce.9 47 In addition, a systematic review concluded that none of the existing clinical prediction models for serious bacterial infections consisting of clinical signs and symptoms had added value in intermediate or high incidence settings, such as hospital emergency departments.10 Therefore in our study design we developed and validated a clinical prediction model considering many of the required quality items for clinical prediction rules.9 Also, most previous studies aggregate different types of serious bacterial infections into a single outcome category,38 and only the model by one group of researchers11 accounted for the heterogeneity of possible outcome categories. Our approach acknowledged that different types of serious bacterial infections involve different clinical management and diverse clinical risk assessment. We thus aimed to aid clinicians in distinguishing three categories of infections in febrile children: pneumonia, other SBIs, and no SBI (see supplementary file, a version of the calculator is also available at www.erasmusmc.nl/feverkidstool). In addition, the diagnostic values of clinical signs and symptoms have often been shown in children at risk of specific outcome measures.38 However, the challenge lies not only in identifying children with, for example, pneumonia among children with signs of a lower respiratory tract infection but rather in designating risk thresholds to all children with fever. Indeed, it has been shown that the incidence of radiographically proved pneumonia among children with fever without a clear focus is significant (5-9%),48 49 50 51 and was especially high in children with raised laboratory markers (13-18%).48 51 Besides, not only did our prediction model verify known predictors of pneumonia in children at increased risk of a lower respiratory tract infection, supporting the validity of our results,4 38 52 but perhaps more importantly it also simultaneously showed that the presence of these signs and symptoms was valuable for ruling out other SBIs. Next, while diagnostic tests are often deployed for the evaluation of febrile children, few studies have quantified the value of diagnostic tests in addition to clinical signs and symptoms.12 Previously reported clinical prediction rules on febrile children that combined both clinical features and laboratory markers either had poor external validity,5 18 targeted young infants,53 or included laboratory markers regardless of clinical signs and symptoms.54 55 56 C reactive protein is an important predictor of serious bacterial infections,12 13 14 and we determined its added value to clinical signs and symptoms. Our results confirmed that C reactive protein is a strong predictor of serious bacterial infections, and in most children C reactive protein considerably improved risk estimates compared with estimates based on clinical signs and symptoms only. We did not consider white blood cell count, as it has no added value when C reactive protein is available,12 38 54 55 57 58 59 nor procalcitonin, as it was not available at the emergency departments and no quantitative point of care test has yet been developed.12 14 60 61 Further research is required on these and other markers of serious bacterial infections.

Strengths and limitations of the study

The main strength of our study is that we performed a prospective, multicentre study involving large and diverse cohorts of children. Secondly, we derived and cross validated a prediction model using two large prospective cohorts and we assessed broad validation in an external population from a different country.41 These cohorts lacked uniformity of inclusion criteria, data collection, and outcome measures, which is commonly seen as a drawback when translating diagnostic research to clinical practice. However, this lack of uniformity reflects the diversity of clinical practice and thus forms a strong test of external validity.47 Thirdly, we included potential predictors in the model independent of their statistical contribution, limiting the risk of chance findings62 63 and increasing face validity and thereby ensuring clinical sensibility and compliance.39 We therefore chose to include heart rate in our model despite its limited effect. Fourthly, we incorporated the prediction model into an actual risk assessment tool for clinical practice.11

Our study has some limitations. Firstly, reference tests were ordered at the doctor’s discretion. We used a follow-up period and clinical consensus to ensure that diagnoses of serious bacterial infections were not missed.23 We believe this minimised the risk of verification bias. It is an approach commonly used in this area of research, as it would be unethical to perform invasive tests in children if not clinically indicated. Secondly, although our study uses routine clinical data collected by trained emergency department nurses, interobserver variability might influence the validity of the model.64 Thirdly, we specified three outcome categories—that is, pneumonia, other SBIs, and no SBI, and were not able to derive a prediction model for septicaemia or meningitis as separate outcome categories. Given further reductions in incidence of invasive bacterial infections in most high income countries,45 this is unlikely to be possible in future research. Also, a limited number of cases of other SBIs in the external validation population and the large diversity of diagnoses in this group restricted the model’s generalisability to other SBIs.63 Fourthly, C reactive protein was measured at the discretion of the doctor and, not surprisingly, was obtained more consistently in children at a higher risk of serious bacterial infection (see supplementary file appendix E). Excluding children without C reactive protein would have selected a subpopulation of children at high risk of serious bacterial infections, as shown in the complete case analysis, and thus limited the generalisability of our prediction model. Missing values of C reactive protein were imputed to be able to analyse observed values from the other predictors, which resulted in higher precision of predictor effects.65 66

Implications for clinical practice

The roles of alarming signs or pathognomic disease specific features of serious bacterial infections, such as petechiae, are limited in practice, as the prevalence of these features is low.38 67 68 Although these signs are highly specific, they have poor sensitivity and will apply only to a small number of children. We suggest that a more useful diagnostic strategy should involve prediction models that combine signs and symptoms and are particularly focused on providing better clinical discrimination for children in the so called “grey area”—that is, where diagnostic uncertainty is greatest. The proposed application of the model (a digital calculator, as shown in the supplementary file, also available at www.erasmusmc.nl/feverkidstool) reflects how we believe different risk levels of serious bacterial infection should be managed. Foremost, a prediction model should discriminate between children at high and low risk of serious bacterial infections (that is, pneumonia, other SBIs, including urinary tract infection and sepsis/meningitis compared with those without serious bacterial infections). This is helpful to identify children who may need further tests, such as additional blood tests or chest radiography, and guide the need for antibiotic prescribing. Also, prediction models should be combined with clear safety net instructions on when to access the health system if the clinical presentation changes, since no clinical prediction rule can achieve perfect sensitivity, especially as children present at different stages in the evolution of their illness.67 69 Considering the typical course of pneumonia and most other common serious bacterial infections and the diagnostic performance of low risk thresholds, we believe that a watchful waiting approach is acceptable in children at low risk. Furthermore, the harms of not routinely testing for C reactive protein should be weighed against the harms and costs of testing for C reactive protein in all febrile children. Even in children at a clinical low suspicion of serious bacterial infections, not carrying out tests for C reactive protein would still lead to missed cases of serious bacterial infections, albeit in only a small number of children. Moreover, although C reactive greatly improves risk predictions, this will not necessarily lead to better diagnostic or therapeutic management of febrile children if specific risk thresholds are not reached.70 At present no established risk thresholds for ordering additional diagnostic tests or initiating antibiotic therapy exist, posing an unanswered dilemma. In addition, the pretest probability of having a serious bacterial infection is determined by the incidence. Adjusting the prediction model for this pretest probability before implementation elsewhere enhances the risk predictions. Hence, risk thresholds should be chosen accordingly for each particular setting and each potential outcome category.

Implications for future research

We tried to create a framework and set a standard for future research in the topic of diagnostic research in febrile children. As many other potential diagnostic tests are being reported,12 14 61 71 their diagnostic value should always be evaluated in combination with clinical signs and symptoms and diagnostic tests that already have proved their additional value, and take into account more than one potential outcome category. We strongly suggest validating and updating prediction models, derived from prospective data, rather than developing new ones. A further validation study of our prediction model, in particular to improve on the discriminative ability of other SBI, is recommended. Finally, studies that focus on the impact of prediction models on clinical practice, ideally by trials of a clinical decision support system, are an essential final step when implementing prediction models in clinical practice.9 72 73 Therefore, the use and impact of this prediction model, as well as the calculator as a diagnostic tool, on clinical practice should be investigated in future studies.

Conclusion

Clinical signs, symptoms, and C reactive protein are useful for estimating the likelihood of pneumonia and other SBIs in children with fever through the presented prediction model. If further validated, this model may well support doctors in their diagnostic work-up and therapeutic decision making.

Many types of serious bacterial infections exist, each involving discrete diagnostic and therapeutic management

Clinical signs and symptoms in febrile children contribute differently to predicting the risk of different types of serious bacterial infections

C reactive protein is an important predictor of serious bacterial infections and is widely used in many emergency department settings in Europe and North America

A validated prediction model integrating clinical signs and C reactive protein gave risk estimates for the presence of both pneumonia and other serious bacterial infections in febrile children in an emergency setting

Risk thresholds ≥10% are useful to identify children with serious bacterial infections, whereas risk thresholds <2.5% are useful to rule out the presence of serious bacterial infections

Using polytomous regression modelling allows for distinguishing between several possible serious bacterial infections facilitating differentiated diagnostic and therapeutic management of febrile children