Predictive Models of Fever, ICU Transfer, and Mortality in Hospitalized Patients With Neutropenia.

Neutropenia is a common side effect of myelosuppressive chemotherapy and is associated with adverse outcomes. Early Warning Scores are used to identify at-risk patients and facilitate rapid clinical interventions. Since few Early Warning Scores have been validated in patients with neutropenia, we aimed to create predictive models and nomograms of fever, ICU transfer, and mortality in hospitalized neutropenic patients.
DESIGN: Development of statistical prediction models and nomograms using data from a retrospective cohort study of hospitalized patients with neutropenia.
SETTING: University of Virginia Medical Center, a tertiary-care academic medical center in Charlottesville, VA. PATIENTS: The derivation and validation cohorts included hospitalized adult patients with neutropenia who were admitted to the inpatient wards between October 2010 and January 2015, and April 2017 and April 2020, respectively. We defined neutropenia as an absolute neutrophil count of less than 500 cells/mm3.
INTERVENTIONS: None.
MEASUREMENTS AND MAIN RESULTS: The derivation cohort included 1,531 hospital admissions in patients with neutropenia. Fever, ICU transfer, and in-hospital mortality occurred in 955 admissions (62%), 297 admissions (19%), and 147 admissions (10%), respectively. In the derivation cohort, the internally validated area under the curves with 95% CI for the fever, ICU transfer, and mortality models were HYPERLINK "callto:0.74%20(0.67-0.84),%200.77"0.74 (0.67-0.84), 0.77 (0.67-0.86), and HYPERLINK "callto:0.95%20(0.0.87-1.0"0.95 (0.0.87-1.0), respectively. The validation cohort included 1,250 admissions in patients with neutropenia. In the validation cohort, the area under the curve (95% CI) for the fever, ICU transfer, and mortality models were HYPERLINK "callto:0.70%20(0.67-0.73),%200.78"0.70 (0.67-0.73), 0.78 (0.72-0.84), and HYPERLINK "callto:0.91%20(0.88-0.94"0.91 (0.88-0.94), respectively. Using these models, we developed clinically applicable nomograms which detected adverse events a median of 4.0-11.4 hours prior to onset.
CONCLUSIONS: We created predictive models and nomograms for fever, ICU transfer, and mortality in patients with neutropenia. These models could be prospectively validated to detect high-risk patients and facilitate early clinical intervention to improve patient outcomes.

Neutropenia occurs in patients with hematologic malignancies and those receiving myelosuppressive chemotherapy. It is associated with severe infections, prolonged hospital stays, and increased mortality (1, 2). Neutropenic fever is an indicator of infection and is considered a medical emergency with mortality rates ranging from 4% to 21% (3, 4). Since rapid initiation of antibiotics and prompt ICU transfer of critically ill patients with neutropenia decreases morbidity and mortality, early identification of patients at high risk for fever, ICU transfer, or in-hospital mortality could improve patient outcomes (5–8).

Adverse clinical events such as cardiac arrest, unanticipated ICU transfer, or death are often preceded by subtle clinical and physiologic changes several hours before the event occurs (9–11). However, these changes are not always recognized or acted upon expediently. Early Warning Scores (EWS) have been developed to identify patient deterioration and alert clinicians to the need for escalated care (12). These EWS use weighted scoring systems, or nomograms, to identify subtle vital sign or laboratory value changes hours to days before the adverse event occurs, facilitating timely intervention and decreasing the risk of a bad outcome. Commonly used EWS such as the quick Sequential Organ Failure Assessment (qSOFA) score and the National Early Warning Score (NEWS) were developed to identify hospitalized patients at high risk for mortality. However, these EWS were not derived from or validated in neutropenic patients (13–15) and have been shown to have low-to-moderate predictive accuracy in other high-risk oncology populations (16). Current EWS designed to predict adverse events in neutropenic patients have significant limitations (17–20). The Multinational Association of Supportive Care in Cancer and Clinical Index of Stable Febrile Neutropenia scores are used to identify patients with neutropenic fever who are at high risk of serious medical complications and could benefit from hospital admission for IV antibiotic therapy and intensive monitoring (18, 20). However, these scores were designed for use in the outpatient setting and developed to triage patients only after fever has occurred. Other predictive models of adverse events during neutropenia are malignancy specific and require extensive knowledge of the patient’s demographics, treatment history, and current clinical condition, making them less practical for use in acute care settings and difficult to integrate into the electronic health record (EHR) (17, 19–25).

Due to the vulnerability of neutropenic patients and the limitations of current EWS, there has been a clarion call for improved identification of these high-risk patients (22). With the advent of the EHR, updated patient data are now available to the clinician in real time. Development of EWS to detect early changes in vital signs and laboratory values and predict adverse events could facilitate early clinical intervention and improve outcomes. Accordingly, we aimed to create predictive models to identify hospitalized neutropenic patients at high risk for the study endpoints of fever, ICU transfer, and mortality using routinely obtained vital signs and clinical laboratory values contained in the EHR. We then aimed to translate these models into a cumulative point scoring systems, or nomograms, that could be readily implemented into the EHR to predict the study endpoints.

MATERIALS AND METHODS

Study Design and Population

For initial model derivation and validation, we conducted a retrospective cohort study of adult patients admitted to the University of Virginia (UVA) Medical Center with neutropenia from October 2010 to January 2015. We obtained approval for the study from the UVA Institutional Review Board. We included all adult patients greater than or equal to 18 years old admitted to UVA who experienced at least one episode of neutropenia as defined by an absolute neutrophil count (ANC) of less than or equal to 500 cell/μL (3). In patients where ANC was unavailable, we used a total WBC count of less than or equal to 500 cells/μL as a surrogate measure for neutrophil count.

Data inclusion and Missing Values

For model derivation, we included 42 predefined variables which were limited to routinely obtained vital signs and laboratory values available in the EHR (Supplemental Table 1, Supplemental Digital Content 5, http://links.lww.com/CCX/A437). We excluded covariates which were not obtained in at least 50% of the patient population and hospital encounters in which fewer than 50% of the variables of interest were obtained. To avoid negative numbers and outliers, which could represent data entry errors or spurious values, we included values only if they were positive, within the 99th percentile for a given variable and within physiologically accepted limits. Since our goal was to predict the study endpoints of fever, ICU transfer, or death prior to their occurrence, we also excluded encounters in which the study endpoint occurred less than 12 hours after hospital admission. For ICU transfer, we also excluded all encounters in which a patient was directly admitted from the emergency department to the ICU. Using the last-observation-carry-forward method, we then randomly sampled variables from each patient encounter taken at a given time between 2 and 24 hours prior to the event of interest. For variables that did not have a result within the past 24 hours, we performed multiple imputation using the multivariate imputation by chained equations package in R to generate five imputed datasets (26).

Outcomes

For each patient in the derivation cohort, we identified the study endpoints of fever, ICU transfer, or death during hospitalization. We defined fever as the first episode of temperature greater than or equal to 100.4°F during the hospital admission, ICU transfer as the first instance of transfer from a general medical or surgical ward to an ICU during hospitalization, and death as all-cause in-hospital mortality (3). At our hospital, ICU transfer is initiated at the discretion of the treating physician in consultation with an intensivist, but it is generally related to an imminent need for intubation or initiation of vasopressors. Since the time of death was not consistently documented in the EHR, the time of hospital discharge served as a surrogate marker in patients who died.

Analyses

We used the chi-square test to compare proportions and the Student t test to compare continuous variables. We considered a two-sided p value less than of 0.05 significant for all statistical tests. We used R (R Foundation for Statistical Computing, Vienna, Austria) for all analyses, and we used the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) and Strengthening the Reporting of Observational Studies in Epidemiology checklist to analyze and report these models (27).

For each study endpoint, we built a logistic regression model using the entire derivation dataset with 10-fold cross-validation for model selection and validation (TRIPOD type 1b model) (27). To address the class imbalance in the ICU transfer and mortality cohorts, we randomly down-sampled the control patients in these groups during model derivation (28). We then used Bayesian model averaging to select the clinically significant variables that produced the model with the minimal Bayesian information criterion (29). To avoid multicollinearity, we calculated variance inflation factors (VIFs) for each variable. For variables with VIF greater than or equal to 3, we used clinical judgment to select the retained variable. We used the resulting variables to build the final logistic regression model, which we evaluated using the area under the curve (AUC).

After each predictive model was developed, we used the beta-coefficients from the logistic regression to create separate nomograms for fever, ICU transfer, and mortality. For each nomogram, we first determined the significant threshold for each covariate using the Class-Attribute Interdependence Maximization algorithm (30). We then assigned a weighted point score by dividing the beta-coefficients of the initial logistic regression by the smallest significant beta-coefficient and rounding each value to the nearest integer. The sum of the corresponding weighted points was added to create a cumulative point score for each nomogram. We evaluated each nomogram for the ability to distinguish between the study endpoints using the AUC and reported their performance at various thresholds using sensitivity and specificity as well as positive predictive value (PPV) and negative predictive value. In a final sensitivity analysis, we also evaluated each nomogram only in patients with malignancy to ensure the performance did not differ between patients with and without malignancy.

We used repeated measures from each patient encounter within the cohort to simulate prospective performance in a hospital setting. For each patient encounter, we sequentially scored each nomogram as the values became available. To account for missing data, we carried forward values according to their mean sampling rate; values that occurred beyond the carry-forward window of 24 hours were left unimputed to replicate a real-world application. For each nomogram threshold, we calculated the time to the study endpoint but discarded it if it was greater than or equal to 24 hours. We then calculated the median time to event for each threshold and generated a plot depicting the cumulative true positive rates for each score against the time to the event of interest.

Finally, we compared the performance of our fever nomogram with the Vitalpac Early Warning Score (ViEWS), an EWS which uses pulse rate, respiratory rate, systolic blood pressure, temperature, oxygen saturation, inspired oxygen (yes/no), and level of consciousness to predict mortality at 24 hours (31). As a threshold, we used the commonly accepted aggregated score of 5 to predict study endpoints (32). ViEWS was chosen over the systemic inflammatory response syndrome (SIRS) since fever and WBC are two components of the SIRS scoring system (33); ViEWS was chosen over qSOFA score since mental status, a primary component of the qSOFA score, was documented in fewer than 50% of the patients in our dataset (34). We also compared the ICU transfer and mortality nomograms to ViEWS since this was designed to predict mortality at 24 hours. We assessed the performance of the nomograms in the original derivation data set. We used repeated measures from each patient encounter within the cohort to simulate prospective performance in a hospital setting in the same way that we evaluated each nomogram.

Validation Cohort

We evaluated the performance of the fever, ICU transfer, and mortality models using a separate validation cohort of adult patients with neutropenia admitted to UVA from April 2017 to April 2020. We applied the logistic regression models to the validation cohort in order to determine AUC for each. We also applied the nomogram for each model to the validation cohort and determined the time-to-event for their respective outcome.

RESULTS

Baseline Patient Characteristics

We screened 73,959 inpatient admissions from September 1, 2010, to August 31, 2015, and identified 1,531 neutropenic episodes from 1,001 unique patients (Table ). The median (interquartile range [IQR]) age was 54 years (45–67 yr), and approximately half the patients were female. Overall, 86% of patients with neutropenia had an underlying malignancy, the most common of which was leukemia (32%) followed by non-Hodgkin’s lymphoma (15%) and myeloma (5%). We identified 215 patients (14%) with bloodstream infections (Supplemental Table 2, Supplemental Digital Content 6, http://links.lww.com/CCX/A438). For all hospitalizations, the median (IQR) length of stay was 13 days (4–19 d). The validation cohort consisted of 1,250 neutropenic episodes from 893 unique patients (Supplemental Table 3, Supplemental Digital Content 7, http://links.lww.com/CCX/A468). The patient demographics including age, sex, and type of malignancy were similar between the derivation and the validation cohorts.

Fever

Of the 955 patients in the derivation cohort who developed fever during hospitalization, 430 (45%) experienced fever more than 12 hours from the time of admission and were included in the analysis (Fig. and Table 1). The median (IQR) hospital stay was 23 days (10–30 d) for patients with fever compared with 9 days (3–11 d) for patients without fever (p < 0.001). Patients with fever were more likely to require ICU transfer (29% vs 8%; odds ratio [OR], 4.5, 95% CI, 3.1–6.5; p < 0.001). In-hospital mortality did not differ significantly between those with and without fever (10% vs 8%; OR 1.4; 95% CI, 0.9–2.2; p = 0.13). In total, 749 patients (80%) included in the fever model had malignancy. Patients with fever were more likely to have an underlying malignancy (83% vs 70%; OR 1.9; 95% CI, 1.3–2.6; p < 0.001), particularly acute leukemia (42% vs 26%; OR 2.1; 95% CI, 1.6–2.7; p < 0.001).

TABLE 1.

Characteristics of Hospitalized Episodes in Patients With Neutropenia Admitted to the University of Virginia (Derivation Cohort)

Characteristics	Total(n = 1,531)	Fever(n = 430)	ICU Transfer(n = 218)	Died(n = 124)
Age, yr, median (IQR)	54 (45–67)	55 (45–67)	62 (55–73)	60 (52–72)
Male sex, n (%)	780 (51)	224 (52)	124 (54)	70 (56)
Length of stay, d, median (IQR)	13 (4–19)	23 (10–30)	18 (7–24)	19 (7–24)
Fever, n (%)	955 (62)	—	188 (86)	91 (73)
ICU transfer, n (%)	297 (19)	124 (29)	—	63 (50)
Death, n (%)	147 (10)	46 (10)	59 (27)	—
Underlying condition, n (%)
Total malignancy	1323 (86)	357 (83)	168 (77)	82 (66)
Acute leukemia	496 (32)	182 (42)	88 (40)	32 (26)
Chronic leukemia	63 (4)	20 (5)	5 (2)	2 (2)
Myeloma	73 (4)	31 (7)	12 (6)	2 (2)
Hodgkin’s disease	32 (2)	31 (7)	5 (2)	3 (2)
Other lymphoma	248 (16)	9 (2)	30 (14)	22 (18)
Head and neck cancer	27 (2)	74 (17)	1 (<1)	2 (2)
Lung cancer	56 (4)	4 (1)	6 (3)	7 (6)
Breast cancer	35 (2)	8 (2)	0 (0)	1 (1)
Other solid tumor	264 (17)	5 (1)	33 (15)	28 (23)
Stem cell transplant	35 (2)	52 (12)	5 (2)	5 (4)
Aplastic anemia	58 (4)	14 (3)	8 (4)	7 (6)

IQR = interquartile range.

Dashes indicate no value is given since this was not a relevant result within the category.

In the multivariate analysis, hemoglobin (adjusted OR [aOR] 0.7; 95% CI, 0.6–0.9; p = 0.007) and platelet concentration (aOR 0.6; 95% CI, 0.5—0.7; p < 0.001) were negatively associated with fever; magnesium (aOR 1.5; 95% CI, 1.2–1.8; p < 0.001) heart rate (aOR 1.8; 95% CI, 1.5–2.2; p < 0.001), and body temperature (aOR 2.2; 95% CI, 1.8–2.7; p < 0.001) were positively associated with fever (Tables and ) (Supplemental Fig. 1, Supplemental Digital Content 1, http://links.lww.com/CCX/A433). The internally validated AUC (95% CI) of the fever logistic regression model was 0.74 (0.69–0.84). When the logistic regression model was applied to the validation cohort, the AUC (95% CI) was 0.70 (0.67–0.73). In the derivation cohort, ViEWS had an AUC (95% CI) of 0.50 (0.44–0.57).

The derived nomogram had an AUC (95% CI) of 0.77 (0.67–0.79) for the total population and 0.74 (0.63–0.75) for patients with malignancy. The risk of fever increased with higher nomogram scores (Fig. ). The PPV for the derived nomogram ranged from 0.46 to 0.87 with higher nomogram scores associated with higher PPVs (Table ). A score of greater than or equal to 6 predicted fever a median (IQR) of 7.5 hours (2.5–14.4 hr) hours before it occurred with a positive PPV of 0.63 (Table 4) (Supplemental Fig. 2, Supplemental Digital Content 2, http://links.lww.com/CCX/A434). In the validation cohort, the nomogram AUC (95% CI) was 0.75 (0.72–0.78). Using a threshold score of 6, the median (IQR) time to fever was 7 hours (2.4–14.1 hr). In the derivation cohort, using a threshold score of 5, ViEWS predicted fever a median (IQR) of 7.8 hours (4–13 hr) before it occurred.

TABLE 4.

Clinical Prediction Rule Performance for Neutropenic Fever, ICU Transfer, and Mortality Nomograms

Threshold	Sensitivity	Specificity	Positive Predictive Value	Negative Predictive Value	Median Time to Event, hr (IQR)
Fever
0	1.00	0.00	0.46	—	—
1	0.89	0.38	0.55	0.81	9.4 (4.0–16.0)
2	0.86	0.46	0.58	0.80	9.0 (3.8–15.7)
3	0.76	0.60	0.62	0.74	9.0 (3.7–15.7)
4	0.58	0.78	0.69	0.68	8.6 (3.5–15.4)
5	0.49	0.84	0.73	0.65	7.8 (2.8–14.6)
6	0.39	0.90	0.77	0.63	7.5 (2.5–14.5)
7	0.31	0.93	0.81	0.61	7.2 (2.2–14.3)
8	0.18	0.97	0.86	0.58	6.6 (1.7–13.8)
9+	0.17	0.97	0.87	0.57	6.7 (1.6–13.5)
ICU transfer
0	1.0	0.00	0.33	—	—
1	0.60	0.79	0.59	0.80	6.3 (2.3–14.8)
2	0.49	0.85	0.62	0.77	5.8 (2.2–13.6)
3	0.37	0.91	0.67	0.74	5.4 (2.1–12.9)
4	0.20	0.96	0.74	0.70	4.7 (2.0–12.3)
5	0.12	0.98	0.78	0.69	4.4 (1.9–11.3)
6+	0.07	0.99	0.82	0.68	4.0 (1.9–11.3)
Mortality
0	1.00	0.00	0.07	—	—
1	0.97	0.65	0.19	0.99	11.1 (6.0–17.0)
2	0.93	0.74	0.23	0.99	11.1 (6.0–17.0)
3	0.92	0.83	0.31	0.99	11.4 (5.9–17.0)
4	0.79	0.93	0.50	0.98	11.0 (5.8–17.0)
5	0.66	0.96	0.63	0.97	11.0 (5.9–17.0)
6	0.53	0.98	0.75	0.96	11.1 (6.0–17.0)
7	0.34	0.99	0.87	0.94	11.1 (6.0–17.0)
8	0.23	1.00	0.92	0.93	11.1 (6.0–17.0)
9	0.14	1.00	1.00	0.93	11.1 (6.0–17.0)
10	0.06	1.00	1.00	0.92	11.1 (6.0–17.0)
11+	0.03	1.00	1.00	0.92	10.7 (5.7–16.7)

IQR = interquartile range.

Dashes indicate no value is given since this was not a relevant result within the category.

ICU Transfer

Of the 297 patients who required ICU transfer, 218 (73%) were transferred more than 12 hours after admission and were included in the analysis (Fig. 1 and Table 1). The median (IQR) age for patients requiring ICU transfer was 62 years (55–73 yr) compared with 54 years (44–66 yr) for patients who did not (p < 0.001). The median (IQR) hospital stay was 18 days (7–24 d) for patients requiring ICU transfer compared with 14 days (4–19 d) for patients who did not (p < 0.001). Patients requiring ICU transfer had an increased prevalence of both fever (86% vs 57%; OR 4.6; 95% CI, 2.1–6.9; p < 0.001) and in-hospital mortality (27% vs 6%; OR 5.5; 95% CI, 3.8–8.1; p < 0.001). In total, 1,118 patients (77%) in the ICU transfer model had malignancy. The prevalence of malignancy did not differ significantly between those who required ICU transfer and those who did not (77% vs 77%; OR 0.99; 95% CI, 0.7–1.3; p = 1.0). However, acute leukemia was more common in those who were transferred to the ICU (40% vs 31%; OR 1.4; 95% CI, 1.1–1.9; p = 0.01).

Figure 1.

Flow diagram of neutropenic episodes included in the development of predictive models for (A) fever, (B) ICU transfer, and (C) mortality for hospitalized patients with neutropenia admitted to the University of Virginia from October 2010 to January 2015.

In the multivariate analysis, body temperature (aOR 1.2; 95% CI, 1.7–2.7; p < 0.001), respiratory rate (aOR 1.4; 95% CI, 1.1–1.8; p < 0.001), total bilirubin (aOR 1.2; 95% CI, 1.1–1.6; p = 0.003), and heart rate (aOR 2.1; 95% CI, 1.7–2.7; p < 0.001) were positively associated with ICU transfer (Tables 2 and 3) (Supplemental Fig. 3, Supplemental Digital Content 3, http://links.lww.com/CCX/A435). The internally validated AUC (95% CI) of the ICU transfer logistic regression model was 0.77 (0.67–0.86). When the logistic regression model was applied to the validation cohort, the AUC (95% CI) was 0.78 (0.72–0.84). In the derivation cohort, ViEWS had an AUC (95% CI) of 0.79 (0.75–0.83).

TABLE 2.

Performance Metrics for Models Predicting Fever, ICU Transfer, and Mortality in Neutropenic Patients

Variables	Coefficient	Adjusted OR	95% CI	Area Under the Curve	95% CI
Fever				0.74	0.69–0.84
Temperature (°F)	0.81	1.8	1.5–2.2
Heart rate (beats/min)	0.62	2.2	1.8–2.7
Magnesium (mg/dL)	0.40	1.5	1.2–1.8
Platelets (k/µL)	–0.46	0.6	0.5–0.7
Hemoglobin (g/dL)	–0.23	0.7	0.6–0.9
ICU transfer				0.77	0.67–0.86
Heart rate (beats/min)	0.78	2.1	1.7–2.7
Respiratory rate (breaths per minute)	0.37	1.4	1.1–1.8
Total bilirubin (mg/dL)	0.26	1.3	1.1–1.6
Temperature (°F)	0.22	1.2	1.0–1.5
Mortality				0.95	0.87-1.0
Blood urea nitrogen (mg/dL)	1.22	3.4	2.2–5.3
Heart rate (beats/min)	0.99	2.7	1.8–4.1
Total bilirubin (mg/dL)	0.67	1.9	2.2–5.3
Respiratory rate (breaths per minute)	0.61	1.8	1.3–2.5
WBCs (k/µL)	0.43	1.5	1.1–2.0
Systolic blood pressure (mm Hg)	–0.54	0.5	0.3–0.8
Total protein (g/dL)	–1.09	0.3	0.2–0.5
Oxygen concentration (%)	–1.01	0.3	0.1–0.6

OR = odds ratio.

TABLE 3.

Clinical Variables and Thresholds for Neutropenic Fever, ICU Transfer, and Mortality Nomograms

Variables	Cutoff	Points
Fever
Temperature (°F)	≥ 98.6	4
Heart rate (beats/min)	≥ 90	3
Magnesium (mg/dL)	≥ 3.0	2
Platelets (k/µL)	< 50	2
Hemoglobin (g/dL)	< 8.0	1
Maximum potential score		12
ICU transfer
Heart rate (beats/min)	≥ 110	3
Respiratory rate (breaths per minute)	≥ 22	2
Total bilirubin (mg/dL)	≥ 4.0	1
Temperature (°F)	≥ 100.4	1
Maximum potential score		7
Mortality
Blood urea nitrogen (mg/dL)	≥ 30	3
Total protein (g/dL)	< 4.5	3
Heart rate (beats/min)	≥ 110	2
Total bilirubin (mg/dL)	≥ 3.0	2
Oxygen concentration (%)	< 90	2
Respiratory rate (breaths per minute)	≥ 22	1
WBCs (k/µL)	≥ 15	1
Systolic blood pressure (mm Hg)	< 90	1
Maximum potential score		15

The derived nomogram had an AUC (95% CI) of 0.71 (0.69–0.76) for the total population and 0.74 (0.63–0.81) for patients with malignancy. The risk of ICU transfer increased with higher nomogram scores (Fig. 2). The PPV for the derived nomogram ranged from 0.33 to 0.82 with higher nomogram scores associated with higher PPVs (Table 4). A nomogram score greater than or equal to 6 predicted ICU transfer a median (IQR) of 4.0 hours (1.9–11.3 hr) before it occurred with a PPV of 0.68 (Table 4) (Supplemental Fig. 2, Supplemental Digital Content 2, http://links.lww.com/CCX/A434). In the validation cohort, the AUC (95% CI) was 0.76 (0.66–0.86). Using a threshold score of 6, the median (IQR) time to ICU transfer was 12.3 hours (6.8–16.7 hr). In the derivation cohort, using a threshold score of 5, ViEWS predicted ICU transfer a median (IQR) of 8.4 hours (10.5–19.4 hr) before it occurred.

Figure 2.

The frequency and associated event rate of (A) fever, (B) ICU transfer, and (C) mortality scores for hospitalized patients with neutropenia admitted to the University of Virginia from October 2010 to January 2015.

In-Hospital Mortality

Of the 147 in-hospital deaths, 124 occurred at least 12 hours from the time of admission and were included in the analysis (Fig. 1 and Table 1). Patients who died during hospitalization were more likely to develop fever (73% vs 62%; OR 1.7; 95% CI, 1.1–1.6; p = 0.01) and require ICU transfer (50% vs 16%; OR 5.2; 95% CI, 3.6–7.7; p < 0.001). Although the majority of patients who died had an underlying malignancy (n = 1153, 66%), patients who died were less likely to have an underlying malignancy compared with those who did not (66% vs 77%; OR 0.56; 95% CI, 0.37–0.83; p = 0.005).

In the multivariate analysis, total protein (aOR 0.3; 95% CI, 0.2–0.5; p < 0.001), systolic blood pressure (aOR 0.5; 95% CI, 0.3–0.8; p = 0.01), and oxygen concentration (aOR 0.3; 95% CI, 0.1–0.6; p < 0.001) were negatively associated with in-hospital mortality; WBC concentration (aOR 1.54; 95% CI, 1.1–2.0; p = 0.002), respiratory rate (aOR 1.8; 95% CI, 1.3–2.5; p < 0.001), total bilirubin (aOR 1.9; 95% CI, 1.3–2.5; p = 0.003), heart rate (aOR 2.7; 95% CI, 1.8–4.1; p < 0.001), and blood urea nitrogen (aOR 3.4; 95% CI, 2.2–5.3; p < 0.001) were positively associated with in-hospital mortality (Tables 2 and 3) (Supplemental Fig. 4, Supplemental Digital Content 4, http://links.lww.com/CCX/A436). The internally validated AUC (95% CI) of the mortality logistic regression model was 0.95 (0.87–1.0). The risk of death increased with higher nomogram scores (Fig. 2). When the logistic regression model was applied to the validation cohort, the AUC was 0.91 (95% CI, 0.88–0.94). In the derivation cohort, ViEWS had an AUC (95% CI) of 0.93 (95% CI 0.90–0.96).

The derived nomogram had an AUC (95% CI) of 0.94 (0.82–0.96) for the total population and 0.94 (0.87–0.95) for patients with malignancy. The PPV for the derived nomogram ranged from 0.07 to 1.0 with higher nomogram scores associated with higher PPVs (Table 4). A score greater than or equal to 5 predicted in-hospital mortality a median (IQR) of 11.0 hours (5.9–17.0 hr) before it occurred with a PPV of 0.96 (Table 4) (Supplemental Fig. 2, Supplemental Digital Content 2, http://links.lww.com/CCX/A434). In the validation cohort, the nomogram AUC (95% CI) was 0.92 (0.89–0.95). Using a threshold score of 5, the median (IQR) time to death was 12.0 hours (6.0–17.5 hr). In the derivation cohort, using a threshold score of 5, ViEWS predicted death a median (IQR) of 11.4 hours (5.9–17.1 hr) before it occurred.

DISCUSSION

Using readily available clinical data, we derived and both internally and externally validated predictive models of fever, ICU transfer, and in-hospital mortality in a large cohort of neutropenic patients. Using these models, we created nomograms that detected high-risk patients 4.0–11.4 hours prior to the adverse event. Each outcome was associated with a unique set of predictive variables allowing us to identify patients at highest risk for each study endpoint. Use of these nomograms could ultimately alert clinicians to patients at high risk for adverse events to facilitate targeted clinical interventions and improve morbidity and mortality (35–37).

Our fever nomogram included several variables which have previously been associated with either neutropenic fever or sepsis (33). Both anemia and thrombocytopenia are surrogate markers for bone marrow suppression, and low hemoglobin has been associated with neutropenic fever (38–41). Early increases in heart rate and temperature, below the fever threshold, were predictive of subsequent fever even when the values remained within physiologically normal ranges. Specifically, patients whose body temperature remained less than 98.6°F were at low risk of developing a fever over the next 24 hours compared with patients whose body temperature was greater than or equal to 98.6°F. Tachycardia is a common response to both systemic inflammation and infection and has been a defining criterion for sepsis (33). In our nomogram, a heart rate of greater than 90 was associated with fever development. Clinicians may not recognize subclinical changes in vital signs, including small increases in body temperature, until a critical threshold is reached (e.g. fever occurs). A nomogram alerting the clinician to these subtle changes could rapidly identify at-risk patients and facilitate early intervention prior to development of fever, mitigating the adverse outcome.

We found that hypermagnesemia was associated with neutropenic fever, and all patients with a magnesium greater than or equal to 3.0 meq/L developed fever within the next 24 hours. Magnesium is an important cofactor in many adenosine triphosphate-dependent cellular pathways including sepsis signaling pathways (42, 43). Hypermagnesemia is a known complication of tumor lysis syndrome (TLS), a condition in which patients with high grade malignancy who are treated with cytotoxic chemotherapy experience massive tumor cell death which releases large amounts of intracellular ions including potassium, phosphate, and magnesium. These patients have often received intensive chemotherapy leading to significant bone marrow suppression and prolonged neutropenia which could explain the association between hypermagnesemia and fever.

Increased respiratory rate is associated with both sepsis and pulmonary decompensation which is consistent with our finding that patients with tachypnea were more likely to require ICU transfer. Acute respiratory failure is one of the most common adverse events associated with neutropenia and is the most frequent cause of ICU transfer (44). ICU transfer is associated with poor outcomes in neutropenic patients with pneumonia or critical illness (7). Increased heart rate and temperature were included in both the ICU transfer and fever nomograms. We found that a heart rate of greater than or equal to 90 predicted fever while a heart rate of greater than or equal to 110 predicted ICU transfer. Elevated bilirubin was also associated with ICU transfer. Cholestasis is a common complication of severe sepsis due to hepatic hypoperfusion and endotoxin-mediated biliary dysfunction (34). Furthermore, in ICU patients with and without malignancy, hyperbilirubinemia is associated with increased mortality and is considered a surrogate marker of disease severity (44, 45).

Markers of end-organ damage including increased creatinine and elevated bilirubin are also associated with increased mortality in ICU patients with and without malignancy (15, 44–47). Many of the variables found in our mortality nomogram are also predictors of mortality in the general ICU population and are components of mortality risk scores including Sequential Organ Failure Assessment, NEWS, and Acute Physiology and Chronic Health Evaluation II (15, 46, 47). We also found an association between increased total WBC count and patient mortality. Malignancies associated with leukocytosis include acute leukemias such as acute myelogenous leukemia (AML) where leukocytosis is a predictor of disease severity and a risk factor for TLS at the initiation of treatment. Patients with AML who have hyperleukocytosis and leukostasis are at high-risk of mortality which is consistent with our findings (48, 49).

From each predictive model, we created a nomogram to allow for easy clinical implementation. The performance of these nomograms varied depending upon the chosen threshold. The appropriate threshold to differentiate between high and low risk patients should be tailored to specific patient populations and chosen based on the desired clinical intervention. In our population, at a threshold of 5 points, the fever nomogram had a sensitivity of 49% and specificity of 78% with a PPV of 73% and would identify at-risk patients a median of 11 hours before fever onset. However, if a threshold of 3 was chosen, the sensitivity would increase to 86%, whereas the PPV would decrease to 58%, thus increasing the number of false-positive results. Given the low event rate of adverse outcomes, the PPV is the most important metric to use to determine an appropriate threshold since choosing a threshold that maximizes sensitivity without considering the PPV could lead to increased false-positive rates and overtreatment of patients.

Given the lack of a suitable EWS comparator for our neutropenic models, we compared our nomograms with ViEWS which also uses readily available vital signs to predict adverse outcomes in hospitalized patients (31). Our fever model significantly outperformed ViEWS in predicting onset of fever with an AUC of 0.74 compared with an AUC of 0.5 for ViEWS. Although the AUC and time-to-event for our ICU transfer score was similar to that of ViEWS, the PPV of our ICU transfer nomogram were higher than the PPVs of ViEWS. For example, at a threshold of 4, our model had a PPV of 0.68, whereas ViEWS had a PPV of 0.45 at a threshold of 5. Our mortality model also had a similar AUC compared with ViEWS; however, our model also had higher PPVs compared with ViEWS. Since PPV is the most important performance metric for clinical alarms to identify true cases and minimize false alarms, our models out-perform previously developed models that were not specifically designed for those with neutropenia (50). Accordingly, after successful prospective validation, our nomograms could be used by clinicians to identify at-risk patients with neutropenia and allow for diagnostic and therapeutic interventions that could improve outcomes. This could be confirmed in a randomized clinical trial.

Our study had limitations. First, the patient population used in this study was confined to a single university tertiary care center, so it is unclear how our models and nomograms would translate to different patient populations. Consequently, we recommend external validation of each model in different healthcare settings. Second, we included patients with all causes of neutropenia rather than only patients with malignancy. In general, this is a strength which allows our nomograms to be applied to all hospitalized patients with neutropenia regardless of cause. In our sensitivity analysis of only patients with malignancy, we found that the models maintained similarly good performance. For the ICU transfer nomogram, the cause of transfer was not considered during nomogram development. At our institution, the decision to transfer is based on clinician discretion. Since we wanted to predict in-hospital ICU transfer, we excluded all patients directly admitted to the ICU from the emergency department, but during model development, we did not differentiate between patients who had a planned versus unplanned ICU transfer. Since healthcare systems may have different criteria for ICU transfer, external validation in different healthcare settings will be important. Similarly, the cause of death (e.g. unexpected death vs withdrawal of care) was not considered during development of the mortality nomogram. However, we expected that the physiologic changes that occurred prior to death that were captured by our model would be similar whatever the cause of death.

CONCLUSIONS

We created three predictive logistic regression models and nomograms to detect patients with neutropenia at high-risk for fever, ICU transfer, or in-hospital death which relied solely upon readily available clinical variables. Deployment of the nomograms in the EHR could allow for continuous monitoring of patients with neutropenia in order to detect adverse events and alert clinicians to subtle clinical deterioration prior to overt decompensation. This could ultimately facilitate the development of targeted interventions to improve patient outcomes.