Validating a decision tree for serious infection: diagnostic accuracy in acutely ill children in ambulatory care.

OBJECTIVE: Acute infection is the most common presentation of children in primary care with only few having a serious infection (eg, sepsis, meningitis, pneumonia). To avoid complications or death, early recognition and adequate referral are essential. Clinical prediction rules have the potential to improve diagnostic decision-making for rare but serious conditions. In this study, we aimed to validate a recently developed decision tree in a new but similar population.
DESIGN: Diagnostic accuracy study validating a clinical prediction rule. SETTING AND PARTICIPANTS: Acutely ill children presenting to ambulatory care in Flanders, Belgium, consisting of general practice and paediatric assessment in outpatient clinics or the emergency department. INTERVENTION: Physicians were asked to score the decision tree in every child. PRIMARY OUTCOME MEASURES: The outcome of interest was hospital admission for at least 24 h with a serious infection within 5 days after initial presentation. We report the diagnostic accuracy of the decision tree in sensitivity, specificity, likelihood ratios and predictive values.
RESULTS: In total, 8962 acute illness episodes were included, of which 283 lead to admission to hospital with a serious infection. Sensitivity of the decision tree was 100% (95% CI 71.5% to 100%) at a specificity of 83.6% (95% CI 82.3% to 84.9%) in the general practitioner setting with 17% of children testing positive. In the paediatric outpatient and emergency department setting, sensitivities were below 92%, with specificities below 44.8%.
CONCLUSIONS: In an independent validation cohort, this clinical prediction rule has shown to be extremely sensitive to identify children at risk of hospital admission for a serious infection in general practice, making it suitable for ruling out. TRIAL REGISTRATION NUMBER: NCT02024282. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

Prospective multicentre validation study in almost 9000 illness episodes in children.

Examining sensitivity and specificity, that is, the proportion of true positives (sensitivity) and true negatives (specificity), which are correctly identified by the four-step decision tree.

Consecutive recruitment in three different settings covering the whole spectrum of acutely ill children seen at first contact.

Measuring standardised clinical features could have led to a workup bias.

Identification of admissions for serious infection depended on the quality of medical records and follow-up.

Introduction

Acute infection is the most common reason for children to attend ambulatory care and represents an important proportion of a general practitioner's workload.1

However, in primary care, <1% of children will be diagnosed with a serious infection.2 The incidence is assumed to be 5–10 times higher at the emergency department (ED).3

Serious infections in children are usually defined as sepsis (including bacteraemia), meningitis, pneumonia, complicated urinary tract infection, bacterial gastroenteritis with dehydration, osteomyelitis and cellulitis.4

These serious infections need to be distinguished from the vast majority of self-limiting infections in children because, although rare in children in developed countries, they are associated with considerable morbidity (eg, hearing loss, neurologic disability) and mortality.5

Furthermore, early recognition could improve prognosis of seriously ill children and prevent avoidable investigations and referrals in children without serious infection.5

Clinicians use signs and symptoms to initially assess the probability of a serious infection and decide on further management. On the basis of a prospective cohort of 4000 children, Van den Bruel et al6 derived a symptom-based four-step decision tree consisting of: the clinician's gut feeling ‘something is wrong’, ‘dyspnoea’, ‘temperature >39.95°C’ and ‘diarrhoea in children aged 1–2.5 years’.

The tree is considered positive if yes to any of these four sequential items is positive, with a sensitivity and negative predictive value (NPV) of nearly 100% in the original derivation study.6 Although the tree also demonstrated high sensitivity in a retrospective validation in another primary care data set using approximations for gut feeling and dyspnoea, prospective validation had not been performed as yet.7

In this study, we aim to prospectively validate this decision tree in a new and independent population of acutely ill children in ambulatory care.

Methods

Setting

This is a diagnostic accuracy study in ambulatory care (defined as general practice, paediatric outpatient clinics or ED).

Patients

Children aged 1 month to 16 years, presenting to a general practitioner (GP) or paediatrician in Flanders, Belgium, with an acute illness for a maximum of 5 days were included consecutively from 15 February 2013 to 28 February 2014. Children were excluded if the acute illness was caused by purely traumatic or neurological conditions, intoxication, a psychiatric problem or an exacerbation of a known chronic condition.

If a physician recruited less than five children over the 1-year study period, the assumption of consecutive inclusion was assumed to have been violated, leading to the exclusion of his or her data from the analysis.

When the same child was recruited twice within 5 days, we considered the second registration a consequence of the same illness episode and discarded the second registration from the analyses.

Index tests

We asked physicians to register diagnostic features based on previous research and consensus of an international team of clinicians and researchers,7 including all items of the National Institute of Health and Care Excellence (NICE) traffic light system, and vital signs (heart and breathing rate, temperature and capillary refill time) and pulse oximetry.7–9

In total, 74 diagnostic features were scored: 28 features obtained by history taking, 36 by clinical examination and 10 items relating to clinical decision-making (see online supplementary file 1).

In addition to the clinical prediction rule, clinicians were asked to rate whether the child appeared seriously ill and whether the parents considered their child's illness different from previous illnesses.6 All features were scored as ‘yes’ when present, ‘no’ when absent, and ‘?’ when they could not be evaluated.

Four-step decision tree

We asked physicians to score variables included in the four-step decision tree, as developed by Van den Bruel et al6 (figure 1).

Figure 1

Four-step decision tree developed by Van den Bruel et al.6 SI, serious infections; GP, general practitioner; yrs, years; red boxes, children testing positive on the decision tree; green box, children testing negative on the decision tree.

‘Something is wrong’ was defined as a subjective gut feeling of the physician that something is out of the ordinary. ‘Dyspnoea’ was defined as difficult or laboured breathing. ‘Body temperature’ was defined as the highest body temperature measured by parents or the physician during the illness episode. Before analysis, 0.5°C was added to temperatures measured under the axilla, or with a tympanic thermometer.10 11

‘Diarrhoea’ was defined as loose or watery stools, increased in frequency and volume.12

Vital signs

Temperature, respiratory rate, heart rate, oxygen saturation and capillary refill time were measured, each according to their respective standardised method.13

All GPs were provided with a paediatric finger pulse oximeter (CMS50QA, Contec Medical Systems, China) for use in children at least 3 years old (due to device limitations). Paediatricians were given the choice to use the provided finger pulse oximeter, or rather use their own large-size pulse oximeter appropriate for all ages.

Target condition

The target condition was hospital admission (>24 h) for a serious infection, which was one of the following:

Sepsis (including bacteraemia) with pathogenic bacteria isolated from haemoculture;

Meningitis with a positive lumbar puncture (pleocytosis in cerebrospinal fluid or identification of bacteria or a virus);

Appendicitis with a positive histological diagnosis;

Pneumonia with an infiltrate seen on chest X-ray;

Osteomyelitis (pathogens from bone aspirate or MRI or bone scan suggestive for osteomyelitis);

Cellulitis (acute suppurative inflammation of subcutaneous tissues);

Bacterial gastroenteritis with dehydration (pathogen isolated from stool culture);

Complicated urinary tract infection (>105/mL pathogens of a single species isolated from urine culture and systemic effects such as fever).

The outcome was verified by three complementary methods:

A search of the electronic medical records of all regional hospitals;

An interview with each participating GP;

A diary completed by parents for children recruited in general practice, recording the date of recovery.

If methods II and III showed evidence of a hospital admission initially not captured by method I, attempts were made to obtain information for this additional hospital admission. Children were considered as not having a serious infection if hospital records showed no evidence of serious infection. In cases when no definitive adjudication could be made on the basis of the above criteria, an adjudication committee consisting of clinicians with expertise in acute paediatric care assigned outcome by consensus, using all available information.

Sample size

Sample size calculations were based on the assumption that the prevalence and diagnostic value of the decision tree would be similar to those reported by Van den Bruel et al.

Assuming a prevalence of 0.9%, recruiting 6500 children would result in 59 cases. This would provide us with an error margin of 12% around an expected sensitivity of 97% (95% CI 85% to 100%).14

Statistical analysis

Accuracy of individual features

First, the accuracy of each diagnostic feature was analysed and reported using sensitivity, specificity, likelihood ratios and predictive values for both the GP and specialist setting (paediatric outpatient and ED). A correction of 0.5 was added to every cell in case of an empty cell in a 2×2 table.

We constructed receiver operating characteristic (ROC) curves for temperature, breathing rate, heart rate and oxygen saturation. In addition, these features were dichotomised on the basis of NICE guidance.13

Validation of the four-step decision tree

The four-step decision tree for any serious infection was validated in the entire group and in the three predefined settings separately being general practice, ambulatory paediatric care and EDs. In addition, we performed subgroup analyses for three infectious categories: pneumonia, complicated urinary tract infections and sepsis/meningitis.

We applied the same missing value categorisations for every decision tree variable as in the derivation study, namely missing values in the same category as ‘no’ or ‘unknown’.6

Optimised thresholds

We optimised the tree by recalibrating the thresholds of body temperature and age for the current data, using classification and regression tree (CART) analysis, and maximising sensitivity with a weighing factor of 75 for false negatives, while keeping the structure of the tree constant.

Pragmatic thresholds

To facilitate implementation in routine care, we created a decision tree with easy-to-remember thresholds for temperature and age:

Temperature of 40°C in the GP setting or 39.5°C in the specialist setting (instead of 39.95°C or 39.2°C);

Age below 3 years of age (instead of 3.3).

Sensitivity analyses were performed comparing the results of all illness episodes versus the first illness episode only to explore the effect, if any, of clustering based on recurring admissions in the same children.

Analyses were performed with Stata software (V.11.2; Stata Corp) and JMP Statistical Discovery (version Pro 11.1.1; SAS Institute Inc).

Ethics

Formal written informed consent was obtained for each child. We provided age-appropriate information leaflets and assent forms for minors below and above 12 years of age.

Results

Baseline characteristics

Children were recruited across Flanders at 92 GP surgeries, 6 outpatient paediatric clinics and 6 EDs, involving 276 physicians (170 GPs and 106 paediatricians): 33% were male, with a median clinical practice experience of 13 years (range 0–40 years).

We included 8664 new illness episodes in 7355 children between 15 February 2013 and 28 February 2014 (figure 2); 1322 children were included with two separate illness episodes, 525 children with three episodes and 379 with four or more episodes.

Figure 2

Flow chart of inclusions in recruited children (GP, general practitioner).

The children's median age was 2 years (IQR 1–4.1; total age range: 1 month to 16.9 years) and 3897 were boys (53%).

Outcome verification

We identified 1025 admissions to hospital for >24 h, of which 283 were for a serious infection (table 1). No patient died during this study.

Table 1

Baseline characteristics for children with or without a serious infection

Baseline characteristics	Serious infection (n=283)	No serious infection (n=8381)
Median age in years (IQR)	1.8 (0.8–4.2)	2 (1–4.1)
Sex, male (%)	150 (53.0)	4460 (53.3)
Recruited in general practice (n=3147)	11	3136
Recruited at a paediatric outpatient clinic (n=2895)	75	2820
Recruited at an emergency department (n=2622)	197	2425
Final outcome (admission >24 h with)
Sepsis	10	0
Meningitis	17	0
Appendicitis	15	0
Pneumonia	163	0
Osteomyelitis	0	0
Cellulitis	3	0
Bacterial gastroenteritis with dehydration	21	0
Complicated urinary tract infection	54	0
Non-serious infection	0	8381

The prevalence of serious infections was 3.3% (95% CI 2.9% to 3.7%), increasing significantly from 0.3% (95% CI 0.1% to 0.6%) in general practice over 2.6% (95% CI 2.0% to 3.2%) in paediatric outpatients to 7.5% (95% CI 6.5% to 8.5%) in the ED setting.

There were only 11 cases of serious infection in the GP setting, of which 8 had pneumonia, 2 had complicated urinary tract infections and 1 had appendicitis. Of the 27 cases of sepsis and meningitis identified in the specialist setting, 16 children had a viral meningitis (mostly enterovirus or herpes simplex), 1 had a bacterial meningeal infection (group B Streptococcus), 5 had Streptococcus pneumoniae sepsis, 1 had Haemophilus influenzae type B sepsis (despite evidence of prior immunisation), 1 had Neisseria meningitidis sepsis and 3 had uropathogenic sepsis (eg, Escherichia coli).

In the GP setting, only gut feeling, fever >1 day, eating or drinking less, and being less active had sensitivities above 80% (see online supplementary file 2). In ambulatory paediatrics and the ED, overall sensitivities were even lower, with only fever duration >1 day and fever not reducing to normal temperatures after antipyretics having sensitivities above 80%.

Red flags (specificity >99%) included reduced consciousness, bloody diarrhoea, inconsistent speech, abnormal skin turgor and fontanel tension, petechial rash, meningeal irritation, nasal flaring, cyanosis, reduced peripheral circulation and peritoneal irritation.

The areas under the ROC curves (AUC) for temperature, breathing and heart rate per setting were low (0.58–0.69), except for breathing rate in the GP setting (AUC=0.80; 95% CI 0.63 to 0.97), probably due to the high number of pneumonia cases in this setting (see online supplementary file 3).

Validation of the four-step decision tree

Figure 3 shows all diagnostic properties of the decision tree per setting. In general practice, sensitivity was 100% (95% CI 71.5% to 100%) and specificity 77.7% (95% CI 76.2% to 79.1%), and 23% of children seen by the GP tested positive on the tree. Sensitivity and specificity were lower in both specialist settings, although CIs overlap.

Figure 3

Validation results of the four-step decision tree for all serious infections. GP, general practice; Paed, paediatric outpatient clinic; ED, emergency department; prevalence, prevalence of serious infection within this setting; LR, likelihood ratio; PV, predictive value.

The diagnostic value of the tree for pneumonia, urinary tract infection and sepsis/meningitis is reported in table 2. For pneumonia, the diagnostic characteristics were almost identical to those for the composite outcome of serious infections, which is unsurprising since pneumonia cases made up 58% of all serious infections. Specificity was higher for complicated urinary tract infection (88.5%, 95% CI 87.3% to 89.5%).

Table 2

Results for pneumonia, urinary tract infection and sepsis/meningitis

Setting	Subgroups serious infections
	Pneumonia	UTI	Sepsis/meningitis
All
Sens	80.4 (73.4 to 86.2)	66.7 (52.5 to 78.9)	66.7 (52.5 to 78.9)
Spec	64.8 (63.8 to 65.8)	64.1 (63.1 to 65.2)	64.1 (63.1 to 65.2)
GP
Sens	100 (63.1 to 100)	100 (15.8 to 100)	No cases
Spec	79.2 (77.7 to 80.6)	88.5 (87.3 to 89.5)
Paed
Sens	84.3 (71.4 to 93.0)	73.3 (44.9 to 92.2)	73.3 (44.9 to 92.2)
Spec	59.9 (58.1 to 61.7)	59.3 (57.5 to 61.1)	59.3 (57.5 to 61.1)
ED
Sens	76.9 (67.6 to 84.6)	62.2 (44.8 to 77.5)	62.2 (44.8 to 77.5)
Spec	54.9 (53.0 to 56.9)	53.9 (51.9 to 55.8)	53.9 (51.9 to 55.8)

All diagnostic characteristics are given with their respective 95% CIs in brackets.

ED, emergency department; GP, general practice; Paed, paediatric outpatient clinic; sens, sensitivity; sepsis/meningitis, composite group of sepsis and meningitis cases; spec, specificity; UTI, complicated urinary tract infections.

For sepsis and meningitis, sensitivity was 69.6% (95% CI 47.1% to 86.8%) in the ED where a large majority of cases were seen.

Optimised and pragmatic thresholds

Figure 4 illustrates the threshold changes when (1) optimising the splits of the decision tree variables using CART, and (2) applying the pragmatic approach.

Figure 4

Validation results after applying optimised and pragmatic thresholds to the four-step decision tree. Yellow boxes, threshold changes after applying the optimisation using classification and regression tree analysis (CART); orange boxes, additional threshold changes after applying the pragmatic approach; sensitivity and specificity are given for every tree with their respective 95% CIs in brackets; y, years.

In the GP setting, using the pragmatic ‘temperature’ threshold of 40°C, sensitivity remained at 100% (95% CI 71.5% to 100%) and specificity was 83.6% (95% CI 82.3% to 84.9%), which is higher than the value obtained with the original tree (but lower than that with the optimal threshold (40.7°C) of 85.4% (95% CI 84.1% to 86.6%)).

In the specialist settings, these strategies increased sensitivity up to 92.0% (95% CI 83.4% to 97.0%), however at the expense of a lower specificity up to 44.8% (95% CI 43.0% to 46.7%).

The sensitivity analyses revealed similar sensitivities and specificities with overlapping CIs for all settings and chosen thresholds in the 7355 first hospital admissions only (84.9% of all episodes).

Discussion

Summary

Validating the four-step decision tree in a new and independent but similar population 9 years after the derivation study demonstrated a sensitivity and NPV of 100% in the GP setting, thus confirming its usefulness to rule out serious infections in general practice. This perfect sensitivity suggests that current practice could be improved by using the tree since 4 of the 11 children with a serious infection were initially not identified at first presentation.

A clinical decision tree that is able to rule out serious infections is especially useful in low prevalence situations. There were only 11 hospital admissions for a serious infection in the GP setting (0.3%), most of which were pneumonia (8 cases), and there were no cases of sepsis or meningitis. This very low prevalence is comparable to that in the derivation study (0.4% in the GP setting).6

In the paediatric outpatient clinic and ED settings, the tree did not provide a useful rule-out value, although sensitivity rose considerably to 92% in the paediatric outpatient clinic setting if the thresholds were optimised.

Using pragmatic thresholds allowed us to enhance overall clarity and ease of use, without losing diagnostic accuracy in the GP and paediatric outpatient settings.

Strengths and limitations

This was a prospective multicentre validation study of the four-step decision tree in a large and similar population of children. We included almost 9000 illness episodes, which makes this study one of the largest cohorts of children with acute illness.15 16

The Belgian healthcare system allows for unlimited access to paediatric outpatient clinics and EDs, alongside general practice. This provides us with a unique opportunity to examine acutely ill children in different urgent-access settings.

To ensure identification of all admissions for serious infection, the outcome was measured through three complementary strategies. Nonetheless, this verification depended on the quality of medical records and follow-up.

We asked the participating physicians to record a list of standardised clinical features, which could lead to additional testing and potentially facilitate a diagnosis of serious infection (workup bias), inflating sensitivity and specificity.17 For this reason, the outcome was defined as hospital admission for a serious infection, rather than hospital admission or serious infection in isolation.

Comparison with existing literature

Very few studies have validated clinical prediction rules of vital signs and symptoms in acutely ill children in primary care.5 Most research has been performed in secondary care, with varying results.8 15 18 19 To the best of our knowledge, there is only one prior study that conducted a retrospective validation in a low prevalence setting and found a sensitivity of 90% for the four-step decision tree.7

Implications for clinicians

Signs and symptoms are the first available tests to support clinical decision-making in primary care.20 The clinician's feeling that ‘something is wrong’ (gut feeling) is confirmed to be an important predictor of serious infection.6 Other red flags, such as cyanosis, poor peripheral circulation, meningeal irritation and petechial rash, are useful as they raise the probability of serious infections, but are rarely present.5

Physicians often choose not to measure vital signs, assuming them to be normal. However, vital signs might act as a red flag for serious infection, as suggested by most recent guidelines.13 21 The results of our study confirm this assumption.

The decision tree consisting of a gut feeling, dyspnoea, temperature >40°C and diarrhoea is able to safely exclude serious infection that warrants hospital admission in children in general practice.

However, 17% of acutely ill children will be labelled as potentially at risk of a serious infection, of whom 98% will be false positive. Consequently, appropriate additional strategies such as rapid laboratory testing or watchful waiting with adequate safety netting need to be put in place to reduce unnecessary referrals.

Implications for research

Blood tests are currently rarely performed in acutely ill children in primary care, because the result becomes available too late to influence clinical decision-making. In adults, rapid laboratory tests such as C reactive protein have been shown to be useful in improving the management of lower respiratory tract infections.22

Future research might be able to establish the exact role of such tests in the management of acutely ill children presenting to ambulatory care.