Risk stratification in children with cancer and febrile neutropenia: A national, prospective, multicentre validation of nine clinical decision rules.

BACKGROUND: Reduced intensity treatment of low-risk febrile neutropenia (FN) in children with cancer is safe and improves quality of life. Identifying children with low-risk FN using a validated risk stratification strategy is recommended. This study prospectively validated nine FN clinical decision rules (CDRs) designed to predict infection or adverse outcome.
METHODS: Data were collected on consecutive FN episodes in this multicentre, prospective validation study. The reproducibility and discriminatory ability of each CDR in the validation cohort was compared to the derivation dataset and details of missed outcomes were reported.
FINDINGS: There were 858 FN episodes in 462 patients from eight hospitals included. Bacteraemia occurred in 111 (12·9%) and a non-bacteraemia microbiological documented infection in 185 (21·6%). Eight CDRs exhibited reproducibility and sensitivity ranged from 64% to 96%. Rules that had >85% sensitivity in predicting outcomes classified few patients (<20%) as low risk. For three CDRs predicting a composite outcome of any bacterial or viral infection, the sensitivity and discriminatory ability improved for prediction of bacterial infection alone. Across all CDRs designed to be implemented at FN presentation, the sensitivity improved at day 2 assessment.
INTERPRETATION: While reproducibility was observed in eight out of the nine CDRs, no rule perfectly differentiated between children with FN at high or low risk of infection. This is in keeping with other validation studies and highlights the need for additional safeguards against missed infections or adverse outcomes before implementation can be considered.

Evidence before this study

Reduced intensity treatment of low-risk febrile neutropenia (FN) in children with cancer has been shown to be safe, improve quality of life and reduce costs of care. International paediatric FN guidelines recommend that centres adopt a validated risk stratification strategy and incorporate it into practice. While as many as 27 paediatric FN clinical rules (CDRs), designed to stratify patients into low and high risk of severe infection or medical complication, have been derived ongoing uncertainty remains as to the most safe and effective rule. This is largely because very few CDRs have undergone prospective, external validation. We searched PubMed, with no restrictions on language or publication date, using the search terms: “febrile neutropenia” AND “clinical decision rule” OR “risk prediction” AND “validation.” Only ten CDRs have undergone prospective, external validation, of which six were validated in multisite studies. These studies were conducted in Europe and India and no CDRs have been prospectively validated in Australia.

Added value of this study

This is the largest, multicentre prospective validation study of paediatric FN CDRs published to date and the first time the PICNICC CDR had been prospectively validated. In addition to assessing reproducibility (sensitivity and specificity) and discriminatory ability (AUC-ROC) we also provide a comprehensive assessment of clinical utility of each rule and report details of all clinically significant missed outcomes. Performance of each CDR for the prediction of a ‘likely bacterial infection’ and the impact of an overnight period of in-hospital observation is also provided. Although eight out of the nine validated CDRs were reproducible, overall discriminatory ability at FN presentation was poor. Reassuringly, performance of all CDRs improved after an overnight period of observation and for three CDRs sensitivity and discriminatory ability increased for predication of bacterial infection. The CDR's with the highest sensitivity tended to classify fewer FN episodes as low-risk.

Implications of all the available evidence

Currently no published paediatric CDR can perfectly predict infections or adverse outcomes in children presenting with FN. Given that there have been at least 27 attempts to derive such a rule, this quest for perfection is unlikely to be achieved using currently available clinical, radiological and biochemical parameters. Depending on the desired low-risk management strategy, a number of the validated CDRs could be incorporated into practice. For entirely home-based treatment, CDRs with the highest sensitivity and NPV should be used, while CDRs with lower sensitivity could be used to select suitable patients for early (<24 h) transfer to home-based care. However, irrespective of the approach, appropriate safe guards such as a period of in-hospital observation, together with a structured home-based program incorporating clear recommendations for readmission, remain paramount.

Alt-text: Unlabelled box

Introduction

Children with cancer and febrile neutropenia (FN) are a heterogeneous group with varying risk of infection. This heterogeneity is not always reflected in management, with many clinicians and centres treating all patients with intravenous antibiotics, irrespective of underlying risk [1,2]. This is contrary to international paediatric FN guideline recommendations that centres ‘adopt a validated risk stratification strategy and incorporate it into practice [3]. Such a strategy might facilitate reduced-intensity treatment within the first 24 h with oral antibiotics or home-based management in patients identified as low risk [4]. The benefits of this include improved quality of life, decreased exposure to nosocomial infections and reduced health costs [5,6].

As many as 27 paediatric FN clinical decision rules (CDRs), designed to stratify patients into low and high risk of severe infection or medical complication, have been derived [7], [8], [9], [10], [11], [12], [13]. However, ongoing uncertainty remains as to the most safe and effective rule [7]. Before a CDR can be used it must undergo validation to determine applicability in a new population and time period. This is especially important for CDRs designed to predict children with low-risk FN and trigger reduced-intensity treatment. As CDR performance in validation and implementation is usually lower than in derivation, a realistic expectation of a rules predictive ability may ensure appropriate safeguards are in place to protect against missed infections or adverse events [7].

Across Australia, home-based or reduced intensity treatment of children with FN identified as low-risk of infection or adverse outcome is not standard of care [2]. Availability of validated CDRs to assist in the identification of these patients has the potential to increase the uptake of dedicated low-risk FN care pathways. The objective of this study was to prospectively validate nine CDRs that predict infection or adverse outcome in children with solid-organ cancer or leukaemia. Performance of the CDRs at day 2 was also assessed.

Methods

This was a prospective, multicentre, non-interventional study (Australian New Zealand Clinical Trials Registry 12616001440415). All eight Australian tertiary paediatric hospitals participated. Children with solid-organ cancer or leukaemia on active treatment and who were admitted to hospital or presented to the emergency, outpatient or day-chemotherapy departments with fever or clinical instability were eligible for inclusion. Fever was defined as a temperature ≥38°C and neutropenia was defined as an absolute neutrophil count (ANC) <1000/mm3. Children with hematopoietic stem cell transplant (HSCT) within three months or receiving treatment antibiotics were excluded. Multiple, discrete FN episodes per patient were allowed. Methodology and reporting of results followed the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis statement (TRIPOD) [14].

Demographic, FN episode and outcome data were prospectively collected by the site research assistant (RA) from entirely electronic (1 site) and combined electronic and paper-based records (7 sites), and entered into REDCap. The RA was blinded to the CDR variables and outcome definitions and data accuracy was verified by the project manager and site investigators (oncology or infectious diseases physician).

Clinical variables were collected at two time points: presentation (day 1), which was 0–4 h from hospital presentation for outpatient-onset FN and from fever onset for inpatient-onset FN, and day 2. Data for day 2 were taken between 0900 and 1100am the morning following admission to replicate the period where clinical ward-round decisions are made. Infection outcomes were from clinical symptoms or microbiological samples taken ≤48 h of FN onset. Other outcome data were collected at the end of FN episode and on day 30. For episodes occurring outside ‘office’ hours, data was collected within 72 h.

Study definitions are outlined in Table 1. Other outcomes were defined according to the derivation studies (Table 2).

Table 1

Definitions used in study.

	Definition
Fever	A single temperature ≥38°C
Neutropenia	Absolute neutrophil count (ANC) <1000/mm3.
End of FN episode	Afebrile for more than 48 h, recovery of ANC beyond nadir and antibiotic cessation
Severely unwell	Severe sepsis or septic shock (as per Goldstein et al.) [15], altered conscious state (Glasgow Coma Score <15 or only responsive to voice or pain), documented as ‘severely unwell’ or equivalent in patient record or either blood pressure or respiratory rate within the mandatory emergency call range [16].
Bacteraemia[17].	A recognised pathogen (including organisms associated with mucosal barrier injury in the setting of mucositis or neutropenia) from ≥1 blood culture set or common commensals from ≥2 blood culture sets drawn on separate occasions [17].
Microbiologically documented infection[17]	An infection that was clinically detectable and microbiologically proven [17].
Clinically documented infection[17]	A site of infection that is diagnosed but its microbiological pathogenesis either cannot be proven or is inaccessible to examination. [17]
Likely bacterial infection[18]	Any infection with a microbiologically documented bacterial cause or that was clinically documented in categories typically attributed to bacterial infection, including pneumonia, skin and soft-tissue infection, osteomyelitis or myositis, enterocolitis, otitis media or externa, sinusitis, epididymoorchitis, central venous catheter pocket or tunnel infection, pharyngitis, perianal abscess or cellulitis, peritonitis, lymphadenitis, or culture-negative sepsis.

Table 2

Details of clinical decision rules undergoing prospective validation and key differences between derivation and validation datasets.

Rule	High risk variables	High risk outcome	Key differences in derivation and validation dataset
Rules predicting microbiologically defined infection
PICNICC [8]	Weighted variables for malignancy type, maximum temperature, clinically unwell, WCC, haemoglobin and AMC	Microbiologically defined infection	Developed from IPD meta-analysis from 22 studies with variable inclusion and exclusion criteria
Rules predicting adverse outcome
SPOG-AE [21]	Applied after 24 h. Total score ≥ 9 = high risk	Adverse outcome defined as a serious medical complication (death, ICU or other life-threatening complication) as a result of infection, MDI and radiologically confirmed pneumonia	-Excluded inpatient onset FN.
			-Fever: ≥ 38·5 °C once or ≥38·0 °C during ≥2 h
	Score for preceding chemotherapy more intensive than ALL maintenance =4; haemoglobin ≥ 90=5; WCC < 300 cells/mm3 = 3; platelet <50 G/L=3		-Neutropenia: ANC ≤ 500cells/mm3
			-Bacteraemia not defined*
Hakim [22]	Total score ≥ 24 = high risk	Proven invasive bacterial infection defined as isolation of a pathogen from a sterile body site or as proven by histology or culture-negative sepsis defined as a systemic response to a possible infection because of hemodynamic instability, focal or multiple organ involvement or altered mental status or lethargy	-Excluded inpatient onset FN
	Score for cancer diagnosis: AML=20, ALL/lymphoma = 7, solids = 0 points; Seriously unwell⁎⁎ = 14 points; temperature ≥39 °C = 11 points; ANC <100 cells/mm3 = 10 points	Bacteraemia defined as a recognized pathogen cultured from one or more blood cultures or common commensals cultured from two or more blood cultures	-Fever: ≥38·3 °C or ≥ 38·0 °C for ≥1 h
Alexander [23]	Any of following = high risk	Adverse outcome defined as identification of a pathogen or where there was a serious medical complication or death	-Excluded inpatient onset FN.
	AML, Burkitt lymphoma, ALL in induction, progressive or relapsed disease; Hypotension, tachypnea/hypoxia 94%; new CXR changes; altered mental status; severe mucositis; vomiting or abdominal pain; focal infection; other clinical reason for in-patient treatment		-Fever: >38·5 °C at presentation or within 6h
			-Neutropenia: ANC ≤ 500cells/mm3
			-Bacteraemia,* serious medical complication,* hypotension⁎⁎⁎ and focal infection# not defined
Klaassen [24]	AMC < 100 cells/mm3	Significant bacterial infection defined as blood or urine culture positive for bacteria, interstitial or lobar consolidation on CXR, or unexpected death from infection (patient not palliative)	-Excluded comorbidity on presentation inc. severe mucositis and pneumonia
			-Fever: > 38·5 °C once or > 38·0 °C or within 12h
Rules predicting bacteraemia
SPOG-bacteraemia [28]	Applied after 24 h	Bacteraemia defined as at least 1x positive blood culture	-Excluded inpatient onset FN.
	Score for shaking or chills = 5; Hb ≥90 = 3; platelet <50 G/L = 3; Other need for inpatient care = 3		-Fever: ≥ 38·5 °C once or ≥ 38·0 °C for ≥2h
			-Neutropenia: ANC ≤ 500cells/mm3
			-Definition of bacteraemia different
Ammann [27]	High risk: any of temperature >39·7 °C, comorbidity requiring inpatient care, WCC≤1000 cells/mm3, not in remission	Bacteraemia defined as at least 1x positive blood culture	-As for SPOG bacteraemia CDR
Baorto [25]	AMC<155 cells/mm3	Bacteraemia (not defined)*	-Excluded age <1y
			-Neutropenia – ANC < 500cells/mm3
Rackoff [26]	High risk: AMC < 100 cells/m3 and temperature ≥39 °C	Bacteraemia defined as a positive blood culture	-Excluded inpatient onset FN.
	Low risk = AMC ≥ 100 cells/mm3; intermediate risk = AMC < 100 cells/mm3 and temperature <39 °C		-Fever: ≥ 38·5 °C once or ≥ 38·0 °C 3x within 24h
			-Neutropenia: ANC < 500cells/mm3
			-Definition of bacteraemia different

PICNICC, Predicting Infectious ComplicatioNs In Children with Cancer; WCC, white cell count; AMC, absolute monocyte count; IPD, individual participant data; SPOG, Swiss Paediatric Oncology Group; AE, adverse event; ANC, absolute neutrophil count; ALL, acute lymphoblastic leukaemia; ICU, intensive care unit; MDI, microbiologically defined infection; AML, acute myeloid leukaemia; HSCT, haematopoietic stem cell transplant; SMC, serious medical complication; ICU, intensive care unit; PCR, polymerase chain reaction.

international consensus definition used for validation.52

defined as severe sepsis or septic shock (as per Goldstein et al.), 15 altered conscious state (Glasgow Coma Score <15 or only responsive to voice or pain), documented as ‘severely unwell’ or equivalent in the patient record or either the blood pressure or respiratory rate in the mandatory emergency call range.16

Hypotension defined according to VICTOR chart.16

Focal infection includes defined as upper respiratory tract infection (URTI), lower respiratory tract infection (LRTI), anorectal infection or central venous catheter (CVC) infection.

Children were managed according to local FN guidelines with piperacillin-tazobactam used as first-line empiric FN therapy. During the study period there was a piperacillin-tazobactam shortage across four sites and local guidelines were modified to include cefepime as first line at three sites and ceftazidime and flucloxacillin at one site. A formal low-risk FN pathway was not in use during the study period and cessation of antibiotics and hospital discharge was typically considered in patients with ANC recovery beyond nadir, negative cultures and at least a 24 h afebrile period.

Microbiological investigations were performed according to site FN guidelines. Across all sites this included: at least one blood culture set (all patients) and urine for culture; nasal swab for respiratory virus PCR; chest X-ray; stool for culture, Clostridioides difficile toxin assay and viral PCR; and skin or wound swab for culture and viral PCR (as indicated).

Identification of CDRs for validation

Twenty-seven potentially relevant studies were identified [[7], [8], [9], [10], [11], [12], [13],19,20]. Of these, nine CDRs were suitable for validation in this dataset (Table 2) [8,[21], [22], [23], [24], [25], [26], [27], [28]]. Insufficient information was collected for eight, with three incorporating C-reactive protein that is not routinely used for FN in Australia [9,20,27,[29], [30], [31], [32], [33]]. Nine studies only described individual variables for infection, [[10], [11], [12], [13],[34], [35], [36], [37], [38]] and in one, a central venous catheter (CVC) was used as a predictor of outcome [39]. As CVCs are present in >95% of paediatric oncology patients in Australia, this was deemed a priori as non-discriminatory [40].

Statistical analysis

To assess reproducibility, the sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of each CDR using the pre-defined thresholds and outcomes were calculated and compared to the derivation datasets.

For the Predicting Infectious ComplicatioNs in Children with Cancer (PICNICC) rule, previously reported recalibrated PICNICC variables were used and clinical utility was assessed by dichotomising at ≤ 10% chance of microbiologically defined infection (MDI) [40]. The PICNICC rule was derived from an individual participant data meta-analysis of 1101 FN episodes and uses different predictors for type of malignancy and log-transformed data. Full details of the model, including the original beta estimates is available elsewhere [8]. The threshold of 10% was derived from discussion with the collaborating members of the international PICNICC group. This included a series of clinically active paediatric cancer and infectious diseases research physicians, a parent whose child had undergone treatment for malignancy and who had experienced FN, and statisticians. It was agreed that up to a 10% risk of MDI would be an acceptable threshold for classification of low-risk.

For the Rackoff rule that stratified patients into >2 groups, both the low and intermediate and the intermediate and high-risk groups were combined for calculation of sensitivity and specificity [26]. Across all CDRs, data were reported separately for all FN episodes (inpatient and outpatient onset FN) and for outpatient-onset FN only.

To determine the overall discriminatory ability of the CDR's the AUC-ROC curve and likelihood ratios were calculated. For the PICNICC CDR, the scaled Briers score and the calibration slope were also reported. The scaled Briers score reflects the proportion of incorrectly assigned episodes, and the calibration slope estimates how precisely the predicted probability of infection meets the measured values [41]. Re-estimation of the odds ratios of the individual variables was made by logistic regression using the same covariates as the original model. All analyses were done using R version 3.2.0.

To assess clinical utility, the following missed outcomes were reported: bacteraemia, ICU admission, severe sepsis/septic shock and death. The ability of the CDRs to predict ‘likely bacterial infection’ and the impact of an overnight period of observation on sensitivity were also calculated.

The clinical utility of each CDR at day 2 was assessed using methodology described by the Swiss Paediatric Oncology Group (SPOG) [21]. Using variables collected at presentation, the sensitivity of the rule at day 2 (between 0900 and 1100am) was determined by combining the information on episodes with the outcome known at that time with the results of prediction on the remaining episodes. Clinical utility of the PICNICC CDR at day 2 was further assessed using variables collected on day 2 as previously described [40]. Episodes that had already been shown to have any of MDI, severe sepsis/septic shock or intensive care unit (ICU) admission before day 2 assessment were excluded from analysis, assuming that they would be pre-classified as high-risk irrespective of score.

Continuous data were presented as median and interquartile range. Fisher's exact test was used for comparisons of categorical data, including sensitivity and specificity between the derivation and validation cohorts. Confidence intervals were calculated for both the derivation and validated datasets using hybrid Wilson/Brown method [42]. The Newcombe-Wilson test with continuity correction was used for difference between proportions. A CDR was considered reproducible if there was no significant difference between either sensitivity or specificity in the derivation and validation cohorts.

Sample size

International data indicate that MDI occurs in between 18 and 25% of FN episodes [7,19]. Sample size estimates were based on validation of the PICNICC CDR as this was most recently derived and included individual participant data from six of the CDR included in this study [[21], [22], [23], [24],27,28]. For validation, 780 episodes of FN, with an estimated event rate of 18%, were required for 80% power to show that AUC-ROC of the PICNICC model is ≥ 0.7.

Role of funding source

The funding source had no involvement in study design, data collection, analysis or manuscript preparation or approval.

Ethics

The study had national and site specific Human Research Ethics Committee approval and informed patient consent was obtained.

Results

A total of 2124 episodes of fever or clinical instability in children with cancer were screened of which 858 FN episodes occurring in 462 patients were included (Figure 1, online supplement). Patient accrual occurred from 1 December 2016 to 31 January 2018. The number recruited exceeded the sample size by 10% as recruitment was delayed at two sites and accrual remained open to enable a minimum of 4 months of data collection per site.

Demographic data is available in Table 3. The primary cause of fever was bacteraemia in 111 (12·9%) episodes, non-bacteraemia MDI in 185 (21·6%), clinically defined infection in 80 (9·3%) and fever of unknown cause in 482 (56·2%). A viral upper respiratory tract infection was the most common non-bacteraemia MDI (n = 101) followed by infective enterocolitis (n = 39) and bacterial urinary tract infection (n = 13). A likely bacterial infection occurred in 198 (23·1%) episodes (proven in 167 and probable in 31).

Table 3

Demographic and outcome data.

	n = 858
Median age, years (IQR)	5.8 (3·5–10·7)
Female, n (%)	415 (48·4)
Aboriginal and/or Torres Strait Islander, n (%)	37 (4·3)
Acute leukaemia, n (%)	449 (52·3)
-Acute lymphoblastic leukaemia	375
-Acute myeloid leukaemia	67
-Other	7
Lymphoma, n (%)	66 (7·7)
-Non-hodgkin lymphoma (excl. Burkitts)	36
-Burkitts lymphoma	19
-Hodgkin lymphoma	11
Solid tumour, n (%)	343 (40·0)
-Ewing sarcoma	70
-Osteosarcoma	51
-Neuroblastoma	48
-Rhabdomyosarcoma	41
-Medulloblastoma	37
-Wilm's tumour	19
-Clear cell sarcoma (kidney)	15
-Other brain tumours	37
-Other solid tumour	25
Relapse/refractory disease, n (%):
-Leukaemia/lymphoma	102 (19·8)
-Solid tumour	31 (9·0)
-Allogeneic HSCT > 3 months prior, n (%)	5 (0·6)
-Autologous, n (%)	18 (2·1)
Central venous catheter in situ, n (%)	845 (98.5)*
-Implanted Port	454
-Tunnelled external catheter	358
-Peripherally inserted central catheter	32
-Non-tunnelled external catheter	5
Prophylaxis, n (%)
-Pneumocystis jirovecii pneumonia (PJP)	830 (96·7)
-Antifungal	290 (33·8)
-Antiviral	72 (8·4)
-Antibacterial (excl. for PJP)	22 (2·6)
Location of FN onset, n (%)
-Outpatient	581 (67.7)
-Inpatient ward	169 (19.7)
-Day chemotherapy/day medical	87 (10.1)
-Hospital in the home	21 (2.5)
Primary cause of FN, n (%)
-Bacteraemia	111 (12.9)
-Other microbiologically defined infection	185 (21.6)
-Clinically defined infection	80 (9.3)
-Fever without focus	482 (56.2)

IQR, interquartile range; excl, excluding; FN, febrile neutropenia.

4 patients had 2 CVCs.

Overall, severe sepsis occurred in 13 (1·5%) (≤4 h in 7 and >4 h in 6), ICU admission in 24 (2·8%) (median time to admission 9·2 h, IQR 4·5–157·6 h) and 30-day all-cause mortality in four (0·9%). There were no deaths attributed to infection.

Validation at presentation

Eight rules exhibited reproducibility (Table 4). Four showed this both in both sensitivity and specificity; PICNICC, Rackoff, Baorto and SPOG-bacteraemia CDRs. In the remaining, reproducibility was observed for sensitivity only in the Klaassen and Ammann CDRs and for specificity only in the SPOG-adverse event (AE) and Alexander CDRs. For the prediction of ‘likely bacterial infection,’ sensitivity improved for four of the reproducible CDRs: PICNICC, SPOG-AE, Alexander and Klaassen (Table 5).

Table 4

Sensitivity, specificity, positive predictive value and negative predictive value of derivation study (d) and prospective validation (Pv) cohort at febrile neutropenia presentation.

Rule	Epi-sodes	Out-come, n (%)	Low risk, n (%)	AUC (95% CI)	Sensitivity		Specificity		PPV,% (95% CI)	NPV,% (95% CI)	LR
					% (95% CI)	Dif from deriv.% (p value)	% (95% CI)	Dif from deriv.% (p value)
Rules predicting microbiologically defined infection
d-PICNICC [8]	909	236 (26·0)	163 (17·9)	–	91.5 (87·3–94·4)	–	21·2 (18.3–24·5)	–	29 (25·8–32·2)	87·7 (81·8–91·9)	1·2
Pv-PICNICC	858	296 (34·5)	155 (18·1)	0·56 (0·53–0·60)	87·2 (82·9–90·5)	4·3 (0·12)	20·8 (17·7–24·4)	0·4 (0·89)	36·7 (33·2–40·3)	75·5 (68·2–81·6)	1·1
Rules predicting adverse outcome
d-SPOG AE [21]	423	122 (28·2)	165 (39)	NA	91·8* (85·6–95·5)	–	51·5 (45·9–57·1)	–	43·3 (37·5–49·5)	93·9 (89·2–96·7)	1·9
Pv-SPOG AE	858	320 (37·3)	329 (38·3)	0·54 (0·51–0·58)	72·2* (67·0–76·8)	19·6 (<0·001)	44·6 (40·5–48·8)	6·9 (0·07)	43·7 (39·5–47·9)	73·0 (67·9–77·5)	1·3
d-Hakim [22]	323	47 (14·6)	223 (69)	NA	74·5 (60·5–84·7)	–	76·4 (71·1–81·1)	–	35·0 (26·4–44·7)	94·6 (90·8–96·9)	3·2
Pv-Hakim	858	151 (17·6)	693 (80·8)	0.69 (0·64–0·73)	41·7 (34·2–49·7)	32·8 (<0·001)	85·6 (82·8–88·0)	9·1 (<0·001)	38·2 (31·1–45·8)	87·3 (84·6–89·6)	2·9
d-Alexander [23]	104	22 (21·2)	55 (53)	NA	90·9 (72·2–98·4)	–	64·6 (53·8–74·1)	–	40·8 (28·2–54·8)	96·4 (87·7–99·4)	2·6
Pv-Alexander	858	306 (35·7)	354 (41·3)	0·51 (0·47–0·55)	63·7 (58·2–68·9)	27.2 (<0.01)	44·0 (39·9–48·2)	20·6 (<0·001)	38·7 (34·5–43·0)	68·6 (63·6–73·2)	1·1
d-Klaassen [24]	227	43 (18·9)	83 (36·6)	NA	83·7 (70–91·9)	–	41·5 (34·6–48·8)	–	25·3 (18·8–32·9)	91·6 (83·6–95·9)	1·4
Pv-Klaassen	858	135 (15·7)	207 (24·1)	0·59 (0·55–0·63)	85·2 (78·2–90·2)	1·5 (0·81)	25·9 (22·8–29·2)	15·7 (<0·001)	17·7 (14·9–20·8)	90·3 (85·6–93·7)	1·2
Rules predicting bacteraemia
d-SPOG bacteraemia [28]	423	67 (15·8)	54 (12·8)	NA	100* (94·6–100)	–	15·2 (11.8–19.3)	–	18·2 (14·6–22·4)	100 (93.4–100)	1·2
Pv-SPOG bacteraemia	858	111 (12·9)	133 (15·5)	0·63 (0·58–0·69)	94·6* (88·7–97·5)	5·4 (0·8)	17·1 (14·6–20·0)	1·9 (0·44)	14.5 (12.1–17.3)	95.5 (90.6–97.9)	1·1
d-Ammann [27]	348	85 (24)	100 (28·7)	NA	95·3 (88·5–98·2)	–	36·5 (30·9–42·5)	–	3·7 (27·1–38·7)	96·0 (90.2–98.4)	1·5
Pv-Ammann	858	111 (12·9)	139 (16·2)	0·57 (0·54–0·59)	95·5 (89·9–98·1)	0·2 (>0·99)	17·9 (15·4–20·9)	18·6 (<0·001)	14·7 (12·3–17·5)	96·4 (91.9–98.5)	1·2
d-Baorto [25]	1171	189 (16·1)	164 (14)	NA	94·7 (90·5–97·1)	NA	15·7 (13·5–18·1)	NA	17·8 (15·5–20·3)	93·9 (89·1–96·7)	1·1
Pv-Baorto	858	111 (12·9)	148 (17·2)	0·59 (0·56–0·62)	93·7 (87·6- 96·9)	1.0 (0.80)	18·9 (16·2–21·8)	3·2 (0·08)	14·6 (12·2–17·4)	95·3 (90·5–97·7)	1·2
d-Rackoff [26]	115	24 (20·9)	94 (81·7)	NA	41·7 (24·5–61·2)	–	87·9 (79·6–93·1)	–	47·6 (28·3–67·6)	85·1 (76·5–90·9)	3·5
Pv-Rackoff⁎⁎	858	111 (12·9)	691 (80·5)	0·63 (0·59–0·69)	35·1 (26·9–44·4)	6·5 (0·64)	82·9 (80·0–85·4)	5·0 (0·30)	23·4 (17·6–30·3)	89·6 (87·1–91.6)	2·1
Pv-Rackoff⁎⁎⁎	858	111 (12·9)	207 (24·1)	–	91·0 (84·2–95·0)	NA	26·4 (23·3–29·7)	NA	15·5 (12·9–18·5)	95·2 (91·3–97·4)	1·2

d, derivation study; Pv, prospective validation; CI, confidence interval; PPV, positive predictive value; NPV, negative predictive value; LR, likelihood ratio; Dif from deriv, difference from derivation study.

includes episodes with adverse event known at reassessment.

intermediate and low risk combined into a single low-risk group.

intermediate and high-risk combined into a single high-risk group.

Table 5

Sensitivity, specificity, positive predictive value and negative predictive value of prospective validation cohort at febrile neutropenia presentation to predict ‘likely bacterial infection.’

	Low-risk, n (%)	AUC-ROC (95% CI)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)	LR
PICNICC	155 (18·1)	0·66 (0·62–0.71)	90·9 (86·1–94·2)	20·8 (17·8–24·0)	26·6 (22·5–29·0)	88·4 (82·4–92·5)	1·2
SPOG-AE	329 (38·3)	0·61 (0·57–0·65)	75·8 (69·3–81·2)	47·9 (44·1–51·7)	30·4 (26·5–34·6)	86·8 (82·9–89·9)	1·5
Hakim	693 (80·8)	0·60 (0·56–0·64)	29·3 (23·4–36·0)	83·8 (80·8–86·4)	35·2 (28·3–42·7)	79·8 (76·7–82·6)	1·8
Alexander	354 (41·3)	0·57 (0·53–0·61)	69·7 (63·0–75·7)	44·6 (40·8–48·4)	27·4 (23·7–31·4)	83·1 (78·8–86·6)	1·3
Klaassen	207 (24·1)	0·57 (0·55–0·60)	87·4 (82·0–91·3)	27·6 (24·3–31·1)	26·6 (23·3–30·1)	87·9 (82·8–91·7)	1·2
SPOG-bact	134 (15·6)	0·60 (0·56–0·65)	90·9 (86·1–94·2)	17·6 (14·9–20·7)	24·9 (21·9–28·1)	86·6 (79·8–91·3)	1·1
Ammann	139 (16·2)	0·56 (0·54–0·58)	92·9 (88·5–95·7)	18·9 (16·1–22·1)	25·6 (22·5–28·9)	89·9 (83·8–93·9)	1·2
Baorto	148 (17·2)	0·55 (0·52–0·57)	89·9 (84·9–93·4)	19·4 (15·6–22·6)	25·1 (22·0–28·4)	86·5 (80·1–91·1)	1·1
Rackoff⁎⁎	691 (80·5)	0·60 (0·55–0·64)	27·3 (21·6–33·9)	82·9 (79·8–85·6)	32·3 (25·7–39·8)	79·2 (76·0–82·0)	1·6
Rackoff⁎⁎⁎	207 (24·1)	–	87·4 (82·0–91·3)	27.6 (24·3–31·1)	26·6 (23·3–30·1)	87·9 (82·8–91·7)	1·2

AUC-ROC is area under receiver operating characteristic curve; PPV, positive predictive value; NPV, negative predictive value; LR, likelihood ratio.

intermediate and low risk combined into a single low-risk group.

intermediate and high-risk combined into a single high-risk group.

When restricted to outpatient onset FN (n = 689), there was no significant difference in the sensitivity and specificity analyses when compared to the full FN cohort (n = 858) for all CDRs except the Alexander (sensitivity and specificity) and the Hakim rules (specificity only) (online supplement).

Across the nine CDRs the AUC-ROC ranged from 0·51 to 0·69. The AUC-ROC improved for three CDRs for the prediction of ‘likely bacterial infection’ (PICNICC, SPOG-AE, Alexander) (Table 5). Similarly, for prediction of bacteraemia alone, the AUC-ROC improved for the PICNICC CDR.

The recalibrated-PICNICC rule had a scaled Briers score of 27% (95% CI 25–30%) and calibration slope of 0·23 (95% CI 0·04–0·65). Calculation of the odds ratios for each of the individual PICNICC variables indicated tumour type (acute myeloid leukaemia, Ewing's sarcoma, osteosarcoma, Hodgkin lymphoma) and temperature were the strongest predictors of MDI (online supplement). When the first, or subsequent, episodes were assessed separately, there was no significant differences in the score (p = 0·65) or AUC-ROC for MDI (p = 0·63). Similarly, when the effect of individual sites was assessed, there was no moderator effect. For prediction of bacteraemia alone, the AUC-ROC improved to 0.70 (95% CI 0·63–0·75) (online supplement).

Two rules (SPOG-AE and SPOG-bacteraemia) were designed to be applied after a period of overnight observation. Applying these at presentation is associated with reduced sensitivity: from 72% to 55% (95% CI 50–61%) for SPOG-AE and from 100% to 92% (95% CI 85–96%) for SPOG-bacteraemia. For the Rackoff CDR which stratifies patients into three groups, the proportion of episodes with bacteraemia in the low risk group was 4·8%, increasing to 12% in the intermediate and 23% in the high-risk groups (p<0·05).

Validation at day 2

Day 2 assessment occurred a median 18·6 h after FN presentation (IQR 14·7–23·8 h). Using SPOG methodology, the adjusted sensitivity improved for all nine CDRs (Table 6) [21].

Table 6

Adjusted sensitivity of each clinical decision rule taking into consideration of outcomes known at Day 2 reassessment and details of missed outcomes for episodes classified as low risk (outpatient onset FN only, n = 689). Details of missed bacteraemia episodes after Day 2 assessment are available in online supplement.

Rule (number low risk)	Adjusted Day 2 Se. (95% CI)^	Missed outcomes in low-risk group			Other outcomes in low-risk group
		Missed outcomes, n (%)	Median time to diagnosis, h (IQR)*	Known at Day 2, n (%)	Bacteraemia n (%)		ICU admission, n (%)	Late onset severe sepsis, n (%)	30 day mortality, n (%)	Median LOS, days (IQR)
					All	>Day2
Rules predicting microbiologically defined infection
PICNICC [8] (140)	90·7 (86·3–93·8)	33 (23·6)	42.3 (19·2–52·4)	11 (33·3)	7 (5·0)	5 (3·6)	0	0	0	3·0 (2·1–4·7)
Rules predicting adverse outcome
SPOG-AE [21] (309)	NA	114 (36·9)⁎⁎	31·9 (12·9–53·4)	41 (36·0)	22 (7·1)⁎⁎	17 (5·5)	1⁎⁎⁎ (0·3)	0	2 (0·6)	3·9 (2·5–7·0)
Hakim [22] (589)	68·0 (58·4–76·2)	65 (11·0)	23·7 (17·1–39·0)	32 (49·2)	49 (8·3)	28 (4·8)	2⁎⁎⁎ (0·3)	1⁎⁎⁎ (0·2)	2 (0·3)	4·1 (2·7–7·1)
Alexander [23] (354)	69·0 (62·9–74·6)	111 (31·4)	37·6 (15·7–62·2)	37 (33·3)	29 (8·2)	13 (3·7)	2 (0·6)	2 (0·6)	0	3·9 (2·7–6·8)
Klaassen [24] (187)	90·7 (83·3–95·0)	18 (9·6)	36·6 (8·9–59·3)	9 (50)	8 (4·3)	4 (2·1)	0	0	0	2·9 (2·0–4·1)
Rules predicting bacteraemia
SPOG-bact [28] (137)	NA	9 (6·6)⁎⁎	27·8 (17·0–42·4)	3 (33·3)	9 (6·6)	6 (4·4)	1⁎⁎⁎	0	0	4·5 (2·6–6·8)
Ammann [27] (139)	96·0 (88·9–98·9)	5 (3·6)	26·6 (5·9–46·1)	2 (40)	5 (3·6)	3 (2·2)	0	0	0	2·9 (2·0–4·2)
Baorto [25] (135)	96·0 (88·9–98·9)	6 (4·4)	31·5 (8·5–66·6)	3 (50)	6 (4·4)	3 (2·2)	0	0	0	2·8 (2·0–3·7)
Rackoff [26] (low & int 547)	67·7 (55·4–76·3)	45 (8·2)	23·6 (16·9–37·1)	20 (44·4)	45 (8·2)	25 (4·6)	5 (0·9)#	1 (0·2)#,⁎⁎⁎	2 (0·4)#	4·2 (2·7–7·2)

Se, sensitivity; h, hour; IQR, interquartile range; ICU, intensive care unit; LOS, length of stay; AE, adverse event.

using SPOG methodology.

For composite outcomes the time to diagnosis of first outcome was used.

data presented for missed episodes at FN presentation.

Outcome known prior to day 2 assessment.

all missed episodes were classified as intermediate risk and ICU admission known prior to day 2 assessment in 3 out of 5.

The clinical utility of the PICNICC CDR at day 2 was also determined using variables collected on day 2. Repeat blood samples were not taken in 109 episodes and in a further 112 episodes, an MDI, severe sepsis or ICU admission prior to day 2 assessment was documented and were excluded. In episodes with missing and non-missing bloods, there was no significant difference in the proportion with an MDI (25·7% vs 21%, p = 0·26) and bacteraemia (7·3% vs 13·8%, p = 0·07). Using this methodology, the sensitivity, specificity, PPV and NPV of the PICNICC rule in the remaining 637 episodes was 93·5% (95% CI 88·5%−96·4%), 12·6% (95% CI 9·9–15·9%), 25·4% (95% CI 22·0–29·2%) and 85·9% (95% CI 76·0–92·2%), respectively, and 71 (11·1%) episodes were identified on day 2 as low risk.

Other clinically significant events

Details of the missed outcomes in episodes classified as low-risk are available in Table 6 and the online supplement (Table 4). In five rules, between one and five low-risk episodes required ICU-level care. In the SPOG-bacteraemia and SPOG-AE rules, these admissions occurred before day 2 assessment, and in the Rackoff rule three out of five occurred before day 2 assessment. For the missed bacteraemia episodes, the median time to initial pathogen identification was 31·2 h (IQR 24·0–42·8 h)

Discussion

This is the largest, multicentre prospective validation study of paediatric FN CDRs published to date. Each of the nine CDRs were rigorously assessed in a ‘real-world’ context using contemporary definitions and detailed information is available on all clinically-significant infections and adverse events. The PICNICC, Rackoff, Baorto and SPOG-bacteraemia CDRs, were reproducible with overlapping sensitivity and specificity in the derivation and validation datasets [8,25,26,28]. Only one CDR (Hakim) did not exhibit reproducibility in this population, in keeping with an earlier study [22,43]. Overall the discriminatory ability (as measured by AUC-ROC) and the likelihood of predicting the outcomes defined in the original derivation studies (measured by the likelihood ratio), for each CDR assessed was moderate, at best. However, across all CDRs designed to be implemented at FN presentation, the sensitivity improved at day 2 assessment after taking into consideration outcomes known at that time [8,[22], [23], [24], [25], [26], [27]]. Notably, for three rules predicting a composite outcome that included all MDIs, sensitivity, discriminatory ability and likelihood ratio also improved for prediction of ‘likely bacterial infection’ [8,21,23].

Although we have demonstrated that most CDRs exhibited reproducibility as evidenced by overlapping sensitivity or specificity, not all are suitable for inclusion in clinical FN pathways that support reduced-intensity treatment. In the classic trade-off between sensitivity and specificity, rules that had high (>85%) sensitivity in predicting outcomes resulted in very few patients being classified as low risk [25,27,28]. While this sensitivity may sit more comfortably with clinicians and patients, it is difficult to justify the time and effort required for successful implementation of a low-risk program incorporating rules which identify less than 20% of episodes as low risk [44]. Looking towards the reproducible rules with a higher proportion of episodes allocated as low-risk (i.e. SPOG-AE, Alexander and Rackoff), albeit with a lower sensitivity, factoring in additional safe guards such as a period of overnight observation may make these more palatable [21,23,26]. Such a pragmatic approach has been successfully described in the adult FN population where patients stratified as low-risk must also have stable underlying disease, no active infection or medical complication requiring in-hospital care and suitably resourced follow up before being eligible for home-based care [45].

Various pathways for reduced-intensity treatment of children with low-risk FN have been explored, ranging from entirely home-based management to early discharge after a period of in-hospital observation with either oral or intravenous antibiotics [4,46]. While these options have been shown to be safe in randomised trials, studies using more stringent risk assessment demonstrate lower rates of treatment failure [4]. The type of reduced-intensity treatment should be tailored to the patient and hospital and be accompanied by appropriate patient and clinician education [47]. Similarly, decisions about which validated CDR to incorporate into low-risk FN pathways require site-specific feasibility assessments. Factors such as timely manual white cell count differentials for accurate monocyte counts (i.e. Klaassen, Baorto, Rackoff CDRs) and access to electronic algorithms for calculation of complicated scoring systems (i.e. PICNICC CDR) require careful consideration in the implementation phase.

There is no international consensus as to the most important outcome to predict in children presenting with FN. Given that bacteria accounts for a significant proportion of infectious causes of FN and underscores the rationale for early introduction of broad-spectrum antibiotics, focusing our efforts on predicting bacterial infections may be the most sensible approach [40]. Three of the CDRs validated in this study include all MDI's as part of a composite outcome and therefore provide equal weight to bacteraemia and a viral upper respiratory illness [8,21,23]. Removing some of the ‘background noise’ of these viral infections, which do not require antibiotics, improves the performance of these rules and suggests that all MDIs may not be the most appropriate outcome to predict at FN onset.

Our study is unique as it provides substantial detail on all missed clinically significant outcomes. Reassuringly, the rate of missed severe sepsis or ICU admission was low, with the latter being known by day 2 assessment in three out of five CDRs [21,22,28]. Although the overall low number of these adverse outcomes may be influenced by in-hospital management with intravenous antibiotics, our rates are in keeping with studies of oral and home-based FN management strategies [4].

For the SPOG-AE and SPOG-bacteraemia CDRs, both designed to be implemented at Day 2, the sensitivity for prediction of adverse event or bacteraemia was considerably lower if applied at presentation [21,28]. Centres adopting either of these rules must be aware of this and ensure patients identified as low risk have an appropriate period of observation prior to transfer to home-based care. The Rackoff rule is also unique in that it stratifies patients into three groups: low, intermediate and high. With a sensitivity of 91% and NPV of 95% when the low-risk group is considered separately, implementation of a low-risk program utilising this rule could facilitate early discharge of these patients with consideration for early discharge in the intermediate group provided additional safety criteria are fulfilled.

Across all sites the number of children with cancer presenting to hospital with non-neutropenic fever (NNF) exceeded the number with FN. This burden is previously unrecognised as reflected by the paucity of NNF studies and the absence of guidelines [48]. To date, only one risk-prediction rule has been derived in children with NNF .[49]. Unlike FN CDRs, a higher ANC was associated with an increased risk of infection, highlighting that rules derived in children with FN are not applicable when the ANC >1.0.

This is the first prospective, multicentre validation of the PICNICC CDR and is the result of a national multidisciplinary collaboration. In addition to the PICNICC rule, the study was sufficiently powered to validate all eight CDRs. To replicate real life, we permitted multiple episodes per patient and there was no significant differences between the discriminatory values in first of subsequently captured episodes. A potential limitation is that inclusion criteria of all derivation studies was not replicated. However, the impact is likely to be small as a previous validation of six CDRs included in this study found no significant difference in sensitivity and specificity when different criteria, including both fever and neutropenia definitions, were used [43]. While we have shown there was no significant difference in CDR performance across study sites, these results may not be generalisable outside of both Australia and the original countries (predominantly European and North American) where they were derived. Finally, as few patients (<3%) received antibacterial prophylaxis results may not be generalisable to patients receiving fluoroquinolones and who may be at risk of breakthrough infections with antibiotic-resistant organisms.

Currently no paediatric FN CDR can perfectly predict infections or adverse outcomes in children presenting with FN. Given that there have been at least 27 attempts to derive such a rule, this quest for perfection is unlikely to be achieved using currently available clinical, radiological and biochemical parameters. While novel biomarkers or harnessing the research capabilities of electronic medical records may provide some hope in the future, clinicians could turn towards existing rules and explore ways to safely incorporate these into low-risk FN programs. Consideration should also be given to recalibration of these rules to refine their predictive ability, however this would require further revalidation. For rules such as PICNICC, SPOG-AE or SPOG-bacteraemia, provision of actual risk scores or percentages may also be of benefit, although further research is required to determine how this may impact patient- and clinician-level decision making.

Our study provides a contemporary and accurate understanding of nine CDRs in the Australian population. Results will inform formal implementation studies that incorporate clinical, economic and quality of life evaluation of low-risk FN management strategies. Although no single CDR performance was superior, we believe a number of the validated rules could be incorporated into practice, depending on the desired treatment strategy. For entirely home-based treatment, CDRs with the highest sensitivity and NPV should be used, while CDRs with lower sensitivity could be used to select suitable patients for early (<24 h) transfer to home-based care. However, irrespective of the approach and the CDR that is used, appropriate safe guards, together with a structured home-based program incorporating clear recommendations for readmission, together with rigorous evaluation, remain paramount.

Author contribution

All authors conceived and designed the analysis, ZA and GMH oversaw data collection, GMH and RP performed the analysis and all authors provided clinical interpretation of the findings. GMH and RP drafted the manuscript; all authors reviewed, edited and confirmed their acceptance of the final submitted version. The corresponding author (GMH) has full access to all the data in the study and had final responsibility for the decision to submit for publication

Declaration of competing interest

GMH reports grants from the Victorian Cancer Agency during the conduct of the study. FEB reports grants from The Royal Children's Hospital Foundation during the conduct of the study. RDAL reports grants from the NHMRC during the conduct of this study. KAT, MS, ZA and FM and RP have nothing to disclose.