Predicting outcome of patients with severe urinary tract infections admitted via the emergency department.

OBJECTIVE: To evaluate clinical prediction tools for making decisions in patients with severe urinary tract infections (UTIs).
METHODS: This was a retrospective study conducted at 2 hospitals (combined emergency department (ED) census 190,000). Study patients were admitted via the ED with acute pyelonephritis or severe sepsis-septic shock related UTI. Area under the receiver operating characteristic curve (AUROC) augmented by decision curve analysis and sensitivity of each rule for predicting mortality and ICU admission were compared.
RESULTS: The AUROC of PRACTICE was greater than that of BOMBARD (0.15 difference, 95% confidence interval [CI] = 0.09-0.22), SIRS (0.21 difference, 95% CI = 0.14-0.28) and qSOFA (0.06 difference, 95% CI = 0-0.11) for predicting mortality. PRACTICE had a greater net benefit compared to BOMBARD and SIRS at all thresholds and a greater net benefit compared to qSOFA between a 1% and 10% threshold probability level for predicting mortality. PRACTICE had a greater net benefit compared to all other scores for predicting ICU admission across all threshold probabilities. A PRACTICE score >75 was more sensitive than a qSOFA score >1 (90% versus 54.3%, 35.7 difference, 95% CI = 24.5-46.9), SIRS criteria >1 (18.6 difference, 95% CI = 9.5-27.7), and a BOMBARD score >2 (12.9 difference, 95% CI = 5-12.9) for predicting mortality.
CONCLUSION: PRACTICE was more accurate than BOMBARD, SIRS, and qSOFA for predicting mortality. PRACTICE had a superior net benefit at most thresholds compared to other scores for predicting mortality and ICU admissions.

INTRODUCTION

Background and importance

Acute pyelonephritis and urinary tract infections (UTIs) are common reasons for physician and hospital visits in the United States. UTIs account for over 2 million emergency department (ED) visits per year. Acute pyelonephritis accounts for over 250,000 hospital visits and over 100,000 hospital admissions per year with an annual treatment cost of $2.14 billion in 2005 ($2.9 billion in 2014 adjusted for inflation). , In the latest U.S. Department of Health and Humans Services National Inpatient Sample comparison report, kidney and UTIs were the sixth most common overall diagnosis and the second most common inpatient infection.

Published guidelines for UTIs do not detail indications for hospital admission. , , Experts describe practical but unvalidated individual objective markers of UTI severity including azotemia, inability to tolerate oral medications, age, comorbidities, an elevated WBC count, and abnormal vital signs. , , Others describe less‐objective markers of severity requiring admission including overall appearance and a subjective assessment of clinical toxicity and clinical signs of infection. , , , , , , ,

Although there are validated rules for identifying pneumonia severity in adults (Pneumonia Severity Index, CURB‐65), no widely accepted rules are available for identifying severity in pyelonephritis or urine infections. , , Fukushima et al found that the qSOFA score accurately predicted mortality in patients with pyelonephritis, but their study only evaluated patients with ureteral calculi. Stalenhoef created a modification of the Pneumonia Severity Index (PRACTICE score) to predict outcome in community‐acquired UTIs. Although this score lowered initial admission rates, the authors found that its use led to an increase in outpatient treatment failure. The BOMBARD score was devised to predict development of severe sepsis/septic shock and mortality based on readily available information at ED triage. In this study, UTIs were the second most common infection in patients with severe sepsis and septic shock. Identification of a clinical decision rule for risk stratification might allow for improved decisionmaking in patients who present to the ED with severe UTIs.

Goals of this investigation

The current study was performed to determine if any of these tools could be used to predict mortality in admitted patients with severe UTIs: acute pyelonephritis and sepsis‐associated UTIs. Our objective was to compare the ability of the PRACTICE score, BOMBARD score, qSOFA score, and SIRS criteria to predict in‐hospital mortality in this population (Supporting Information Table S1). A secondary goal was to determine if these criteria and scores could predict admission to an ICU.

Design and setting

This was a retrospective study conducted at 2 hospitals. Hospital A was a level 1 urban trauma center with a 2018 ED census of 102,000 patients with 65% of hospital admissions occurring via the ED. Hospital B was an urban community hospital with a 2018 ED census of 88,000 patients with 84% of hospital admissions occurring via the ED.

The Bottom Line

Acute pyelonephritis and urinary tract infections (UTIs) are common reasons for physician and hospital visits in the United States. In this study, UTIs were the second most common infection in patients with severe sepsis and septic shock. Identification of a clinical decision rule for risk stratification might allow for improved decisionmaking in patients who present to the ED with severe UTIs.

Study participants

Participants included consecutive patients with a final inpatient discharge diagnosis of severe UTI including acute pyelonephritis and severe sepsis/septic shock. Study patients comprised patients admitted via the ED between October 1, 2015 and September 30, 2018 with a final inpatient discharge diagnosis of severe sepsis/septic shock (ICD10 code R65.20, R65.21) with a UTI or acute pyelonephritis (ICD10 code N10). Hospital abstractors used the Centers for Medicaid and Medicare Services (CMS) SEP‐1 measure set with Sepsis‐2 consensus definitions for defining severe sepsis and septic shock. , Patients were included only if acute pyelonephritis or UTI was proven within 48 hours of admission. Patients were excluded if they were not admitted via the study hospital EDs or a complete set of ED vital signs was missing. Repeat admissions during the study period were excluded so that no single patient contributed more to each score relative to any other patient.

This proposal was approved by the hospitals’ Institutional Review Board.

Definitions and data collection

Data collection was completed by study authors using a standard data collection spreadsheet to abstract data. Severe UTI was defined using the European Association of Urology guidelines for urologic infections and included pyelonephritis plus sepsis‐related UTI. , Prior to data collection, consensus definitions were created for admission diagnostic categories, cerebrovascular disease (prior stroke or transient ischemic), malignancy (active hematologic cancer or metastatic cancer in past 12 month excluding basal cell and squamous skin cancer), severe liver disease (cirrhosis, hepatitis due to alcohol, elevated INR due to liver disease), congestive heart failure (any prior diagnosis), chronic renal disease, and immune deficiency (diabetes, transplant, HIV positive, absent spleen, sickle cell disease, and immune medication use [steroids, chemotherapy, immune modulators]). Uncomplicated UTI was defined using European Association of Urology guidelines and includes non‐pregnant women with no chronic renal disease, no anatomic (eg, ureteral stone, hydronephrosis, stents, nephrostomy tubes) functional urologic abnormalities (eg, neurogenic bladder), and no immune deficiency. Complicated UTIs included men, pregnant women, patients with an indwelling urinary catheter, patients with renal disease (transplant, chronic renal insufficiency, polycystic kidneys, renal or ureteral stones), and patients who were immunocompromised. The 2019 update of this guideline removed post‐menopausal women from the definition of complicated disease.

Mental status was documented as normal if terms on the templated ED record were circled including, “alert,” “oriented X 3,” and “normal mental status.” Mental status was documented as abnormal if any of these items were crossed out or if the terms confusion, abnormal mental status, not oriented, or disoriented were written on the ED record. Specific areas of each medical record that were reviewed included the index ED and inpatient record including all triage and ED vital signs (all of which are entered into an electronic health record), the ED and inpatient admission history and physical examination, clinical summary, laboratory results, electronic prescriptions, and consultant records for the index visit. Prior to record review, a 3‐hour training session took place that emphasized definitions and uniform chart reviews. Primary chart abstractors were told that the study was being conducted to develop a profile of pyelonephritis patients and not to analyze mortality, ICU admissions, or scores being evaluated (eg, PRACTICE). Initially, study abstractors simultaneously abstracted 20 charts to ensure uniformity. After every 10 charts were reviewed, data abstraction and data entry were re‐evaluated by principal investigators looking at missing data cells, historical features, and discrepancies between vital signs recorded for each score. After every 50 charts, coding rules were re‐reviewed with abstractors. The principal investigators arbitrated all coding questions on an ongoing basis. Any disagreements were settled by consensus of the 2 principal investigators and study abstractor.

For each patient the following data were recorded: age, sex, primary ED admitting diagnosis, final inpatient diagnosis, date of admission/discharge/death, past medical history (cerebrovascular disease, chronic renal disease, congestive heart failure, malignancy), pregnancy status, presence of current urologic obstruction (ureteral calculi or other obstruction), presence of neurogenic bladder including paralysis, urological instrumentation in prior 30 days, initial temperature, systolic blood pressure (SBP), heart rate (HR), shock index (HR/SBP), respiratory rate, oxygen (O2) saturation, mental status (abnormal defined as altered mental status or Glasgow coma scale [GCS] <15), WBC count, creatinine, lactate, urinalysis (WBCs, leukocyte esterase, nitrate), urine culture result from urine obtained within 48 hours of arrival, and blood culture from blood obtained within 48 hours of arrival. Absent initial historic features were imputed as absent. For septic shock‐severe sepsis patients, the presence or absence of organ dysfunction criteria within the first 3 hours after ED arrival was recorded to categorize severe sepsis‐septic shock as being present in the ED or after admission. The presence of sepsis‐related organ dysfunction criteria was evaluated by sequentially evaluating vital signs, lactate, other organ dysfunction laboratory parameters, and oxygenation. The first evidence of organ dysfunction was documented for this assessment (Supporting Information Table S1).

Initial recorded vital signs and clinical features were used to calculate initial PRACTICE, BOMBARD, qSOFA scores, and SIRS criteria.

Statistical analysis

Chi‐square analysis and Fisher exact test were used to compare proportions and the Wilcoxon rank‐sum test was used to compare continuous variables between patients who died in the hospital versus discharged alive.

The area under the receiver operating characteristic curve (AUROC) for predicting overall mortality and ICU admission were compared between the BOMBARD score, PRACTICE score SIRS criteria, and qSOFA score using the method of Delong. To further analyze the clinical usefulness of each score, the net benefit for multiple threshold probabilities were analyzed using decision curve analysis. Decision curve analysis consists of subtracting the proportion of all patients who are false‐positives from the proportion who are true‐positives, weighted by the relative harm of a false‐positive or false‐negative result. The value weighting (threshold probability) represents a decisionmaker's estimation (clinical preference) about the benefit to harm ratio between prediction of disease and unnecessary treatment. A lower threshold implies that the perceived harm of treatment is low compared to the benefits of predicting disease (eg, mortality). Conversely, a higher risk threshold occurs when the perceived harm from over‐treatment is high compared to the benefit of predicting disease.

Sensitivity, specificity, and likelihood ratios of each test at predicting mortality and ICU admission in all patients and in the subset with uncomplicated UTIs were compared using McNemar test.

Interrater reliability of chart abstraction for features included in each score was performed with a second reviewer abstracting 55 charts (28 with sepsis, 27 with acute pyelonephritis) chosen using a random number generator. This was the total number of charts required to detect a statistically significant difference (alpha <0.05. power = 0.8) between 2 raters assuming the proportion of positive ratings was between 0.1 and 0.9 and assuming the null hypothesis value of kappa was between 0 and 0.7.

All statistical analyses were performed using MedCalc Statistical Software v18.2.1 (MedCalc Software bvba, Ostend, Belgium, 2018) and Stata software (StataCorp. 2019. Stata Statistical Software: Release 16. College Station, TX; StataCorp LLC).

Study sample size calculation

In the validation study for the PRACTICE score, the AUROC for predicting mortality in individuals with UTIs was 0.91. Saeed et al found that the qSOFA score had an AUROC of 0.84 for predicting mortality in ED patients with suspected infections (22%–24% of whom had a UTI). Mortality was estimated at 5% for acute pyelonephritis and 27% for combined severe sepsis/septic shock. , Assuming a 50:50 mix of pyelonephritis and UTI‐related severe sepsis/septic shock cases, the mortality rate was estimated at 16%. Assuming this mortality rate, at least 732 cases would have to be enrolled in order to have 90% power (alpha = 0.05) to detect a 0.07 difference in accuracy for predicting mortality (AUROC 0.91 versus 0.84) between the PRACTICE and qSOFA scores.

RESULTS

Characteristics of study subjects

There were 1011 patients evaluated during this study: 540 with severe sepsis/septic shock and 471 with acute pyelonephritis. UTI was listed as the primary (first listed) ED diagnosis in 282 (59.9%) of acute pyelonephritis patients and 63 (13.4%) of sepsis patients. UTI was listed as any ED diagnosis in 350 (74.3%) of acute pyelonephritis patients and 263 (48.7%) of sepsis patients (Supporting Information Table S2). A total of 622 patients had complicated UTIs and 389 had uncomplicated UTIs. Overall mortality was 70 (6.9%) with 55 deaths in those with complicated UTIs (8.8%) and 15 deaths in those without complicated UTIs (3.9%). Two hundred ninety‐four patients (29.1%) were admitted to an ICU during their hospitalization. Sepsis‐related organ dysfunction was present during the initial ED evaluation (0–3 hours after arrival) in 407 severe sepsis‐septic shock patients (75.3%) with the other 133 severe sepsis‐septic shock patients developing organ dysfunction after admission or >3 hours after arrival.

In‐hospital mortality was associated with older age, male sex, ICU admission, a diagnosis of severe sepsis‐septic shock, and features of complicated UTIs. Comparison of other clinical features between groups is listed (Table 1).

TABLE 1

Baseline characteristics of patients

Feature	All patients (n = 1011)	In‐hospital death (n = 70) a	Alive and discharged (n = 941) a	Absolute difference (95% CI)
Age—y, median (IQR)	61 (40–76)	73 (66–84)	61 (39–75)	−14 (−19 to −9)
Sex male, no. (%)	375 (37.1)	36 (51.4)	339 (36)	−15.4 (−27.8 to −2.8)
ALF resident b	207 (20.5)	30 (42.9)	177 (18.8)	−24.1 (−36.6 to −12.2)
Cerebrovascular disease	145 (14.3)	14 (20)	131 (13.9)	−6.1 (−17.9 to 2.5)
Severe sepsis‐septic shock	540	68 (97)	472 (50.2)	−47 (−51 to −38.3)
Congestive heart failure	130 (12.9)	16 (22.9)	114 (12.1)	−10.7 (−22.8 to 1.6)
Diabetes (DM)	369 (36.5)	25 (35.7)	344 (36.6)	1 (−12 to 12.1)
Immune disorder excluding DM	196 (19.4)	21 (30)	175 (18.6)	−11.4 (−23.9 to 1)
Liver disease	32 (3.2)	7 (10)	25 (2.7)	−7.3 (−17.5 to 1.7)
Active or recent malignancy	79 (7.8)	13 (18.6)	66 (7)	−11.6 (−23.1 to −3.4)
Current pregnancy	5 (0.5)	0	5 (0.5)	.5 (−1.3 to 5.9)
Renal disease	202 (20)	21 (30)	181 (19.2)	−10.8 (−23.3 to −0.3)
Urological disease	314 (31.1)	19 (27.1)	295 (31.3)	4.2 (−8.3 to 14.3)
Altered mental status or GCS < 15	293 (29)	46 (65.7)	247 (26.3)	−39.5 (−50.5 to −26.7)
Initial temperature—centigrade, median (IQR)	37.2 (36.7–38.1)	36.4 (36.1–37)	37.2 (36.7–38.1)	0.9 (0.6–1.2)
Initial systolic blood pressure (SBP)—mm Hg, median (IQR)	120 (101–138)	103 (82–127)	121 (103–139)	16 (9–23)
Initial shock index (HR/SBP), median (IQR)	0.86 (0.69–1.07)	0.89 (0.72–1.24)	0.85 (0.69–1.06)	−0.06 (−0.13 to 0.01)
Initial heart rate (HR)—beats per minute, median (IQR)	104 (88–119)	101 (80–119)	104 (89–119)	4 (−2 to 11)
Initial respiratory rate (RR)—respirations/min, median IQR	19 (17–22)	20 (16–24)	18 (17–22)	0 (−2 to 1)
Urinary tract infection noted on ED record on admission	627	25 (35.7)	602 (64)	28.3 (16.2–38.9)
Positive urine culture from urine obtained in first 48 h of arrival	722 (71.4)	51 (71.4)	671 (71.3)	0 (−0.1 to 0.12)
Positive blood culture from blood obtained in first 48 h of arrival	307 (30.4)	20 (28.6)	287 (30.5)	2 (−0.11 to 0.12)
Complicated urinary tract infection	622 (61.5)	55 (78.6)	567 (60.3)	−18.3 (−27.5 to −6.2)
Duration hospitalization in days until discharge or death	5 (3–8)	5 (1–10)	5 (3‐8)	1 (0–2)
ICU admission during hospitalization c	294 (29.1)	62 (88.6)	232 (24.7)	−63.9 (−70.5 to –53.1)
White blood cell count in cells/mm3	13,100 (9600–17,900)	13,100 (7200–22,100)	13,100 (9700–17,700)	0 (−2000 to 2000)
Creatinine in mg/dL		2.5 (1.3–4.2)	1.3 (0.8–2)	−1 (−1.4 to −0.63)
Lactate—mmol/L	1.8 (1.1–3.1)	3.6 (2.1–6)	1.7 (1.1–2.9)	−1.6 (−2.1 to −1.1)
Lactate >2 mmol/L d	371/867 (42.8)	50/64 (78.1)	321/803 (40)	−38.2 (−25.3 to −47.8)
PRACTICE score >75	490 (48.5)	63 (90)	427 (45.4)	−44.6 (−51 to −34)
BOMBARD score >2	603 (59.6)	54 (77.1)	549 (58.3)	−18.9 (−28.2 to –6.5)
SIRS criteria >1	600 (59.3)	50 (71.4)	550 (58.5)	−13 (−23.4 to −0.3)
qSOFA score >1	235 (23.2)	38 (54.3)	197 (20.9)	−33.4 (−45.4 to −20.8)

Median with interquartile range (parenthesis) provided for continuous data, Number, and percent (parenthesis) for categorical data.

ALF, assisted living facility.

Of the 8 patients not admitted to the ICU who died, 5 were hospice patients or designated do not resuscitate in the ED, 2 were designated do not resuscitate within 24 hours of admission, and 1 95‐year‐old female with metastatic gastrointestinal cancer had a coexisting non‐STEMI and complicated sepsis‐related urinary tract infection with hydronephrosis due to a kidney stone.

Lactate was measured in the ED in 64 patients who died and 803 patients who lived.

Mortality prediction—all patients

The initial PRACTICE score (AUROC 0.79, 95% confidence interval [CI] = 0.76–0.81) was more accurate than the initial BOMBARD score (0.15 difference, 95% CI = 0.09–0.22), initial SIRS criteria (0.21 difference, 95% CI = 0.14–0.28), and initial qSOFA score (0.06 difference, 95% CI = 0–0.11) for predicting overall mortality (Figure 1; Table 2). The PRACTICE score was the only score with a greater net benefit compared to the option of treating all patients at all thresholds. The PRACTICE score had a greater net benefit compared to BOMBARD and SIRS criteria at all threshold probability levels. PRACTICE had a greater net benefit between a 1%–10% threshold probability whereas qSOFA had a superior net benefit at an 11%–16% threshold probability for mortality prediction in all patients (Supporting Information Figure S1).

FIGURE 1

AUROC comparisons for predicting mortality in severe urinary tract infections

TABLE 2

Comparison of operating characteristics and accuracy for predicting mortality in all patients

Score/criteria	Sensitivity c	Specificity d	Positive likelihood ratio	Negative likelihood ratio	AUROC b
BOMBARD >2	77.1% (65.5–86.3)	42.7% (38.6–45)	1.37 (1.17–1.55)	0.54 (0.35–0.83)	0.63 (0.6–0.66)
PRACTICE >75	90% (80.5–95.9)	54.7% (51.5–58)	1.9 (1.71–2.11)	0.19 (0.09–0.38)	0.79 (0.76–0.81)
qSOFA >1	54.3% (41.9–66.3)	79.2% (76.5–81.8)	2.59 (2.02–3.32)	0.58 (0.45–0.75)	0.73 (0.7–0.76)
SIRS >1	71.4% (59.4–81.6)	41.6% (38.5–44.9)	1.23 (1.05–1.44)	0.68 (0.47–0.99)	0.57 (0.54–0.6)

Initial criteria—obtained from first set of vital signs in the ED. Initial vital signs were obtained a median of 9 min (4–21 min, interquartile range/IQR) after triage arrival for all patients. 95% confidence interval (CI) within parentheses.

AUROC—the area under the receiver operating characteristic curve receiver operative curve (AUROC) of PRACTICE was greater than that of BOMBARD (0.15 difference, 95% CI = 0.09–0.22) and SIRS (0.21 difference, 95% CI = 0.14–0.28) and qSOFA (0.06 difference, 95% CI = 0–0.11) for predicting mortality. The AUROC of qSOFA was greater than BOMBARD (0.09 difference, 95% CI = 0.04–0.15) and SIRS (0.16 difference, 95% CI = 0.1–0.22) for predicting mortality. The AUROC of BOMBARD was greater than SIRS (0.06 difference, 95% CI = 0.01–0.12) for predicting mortality.

Initial sensitivity comparison—McNemar test. An initial PRACTICE score >75 was more sensitive than an initial qSOFA score >1 (35.7 difference, 95% CI = 24.5–46.9), initial SIRS criteria >1 (18.6 difference, 95% CI = 9.5–27.7) and an initial BOMBARD score >2 (12.9 difference, 95% CI = 5–12.9) for predicting mortality. An initial BOMBARD score >2 was more sensitive than an initial qSOFA score >1 (22.9 difference, 95% CI = 13–32.7) but not initial SIRS criteria >1 (5.7 difference, 95% CI = 0.3–11) for predicting mortality. Initial SIRS criteria >1 was more sensitive than an initial qSOFA score >1 (17.1 difference, 95% CI = 8.3–30) for predicting mortality.

Initial specificity comparisons—McNemar test. An initial qSOFA score >1 was more specific than an initial PRACTICE score >75 (24.5 difference, 95% CI = 21.7–27.2), an initial BOMBARD score >1 (37.5 difference, 95% CI = 34.4–40.6), and initial SIRS criteria >1 (37.6 difference, 95% CI = 34.5–40.7) for predicting mortality. An initial PRACTICE score >75 was more specific than an initial BOMBARD score >2 (13 difference, 95% CI = 10.8–15.1) and initial SIRS criteria >1 (13.1 difference, 10.9–15.3) for predicting mortality. Specificity of an initial BOMBARD score >2 did not differ from initial SIRS criteria >1 (0.1 difference, 95% CI = −0.1 to 0.3) for predicting mortality.

An initial PRACTICE score >75 was more sensitive than an initial qSOFA score >1 (90% versus 54.3%, 35.7 difference, 95% CI = 24.5–46.9), initial SIRS criteria >1 (90% versus 71.4%, 18.6 difference, 95% CI = 9.5–27.7), and an initial BOMBARD score >2 (90 versus 77.1, 12.9 difference, 95% CI = 5–12.9) for predicting mortality (Table 2). The 7 patient deaths with a PRACTICE score <75 are listed (Supporting Information Table S3).

Mortality prediction—uncomplicated infections

The initial PRACTICE score (AUROC 0.89, 95% CI = 0.84–0.91) was more accurate than a BOMBARD score (0.19 difference, 95% CI = 0.07–0.32) and SIRS criteria (0.33 difference, 95% CI = 0.22–0.44) but not qSOFA (0.07 difference, 95% CI = −0.03–0.17) for predicting overall mortality in patients with uncomplicated severe UTIs (Table 3). The PRACTICE score had a greater net benefit between a 1%–9% threshold probability whereas qSOFA had a greater net benefit at a 10%–15% threshold probability for mortality prediction in all patients with uncomplicated infections (Supporting Information Figure S1).

TABLE 3

Comparison of operating characteristics and accuracy for predicting mortality in patients with uncomplicated infections

Score/criteria	Sensitivity c	Specificity d	Positive likelihood ratio	Negative likelihood ratio	AUROC b
BOMBARD >2	80% (51.9–95.7)	48.1% (42.9–53.3)	1.54 (1.18–2.02)	0.42 (0.15–1.15)	0.7 (0.65–0.74)
PRACTICE >75	93.3% (66–99.6)	71.1% (66–75.6)	3.23 (2.62–3.98)	0.09 (0.01–0.62)	0.89 (0.86–0.92)
qSOFA >1	60% (32.9–82.5)	85.8% (81.7–89.1)	4.23 (2.61–6.86)	0.47 (0.25–0.87)	0.82 (0.78–0.86)
SIRS >1	66.7% (38.6–87)	40.4% (35.4–45.5)	1.12 (0.77–1.61)	0.83 (0.4–1.7)	0.56 (0.51–0.61)

Initial criteria obtained from first set of vital signs in the ED. Initial vital signs were obtained a median of 9 min (4–21 min, interquartile range/IQR) after triage arrival for all patients. 95% confidence interval (CI) within parentheses.

The initial PRACTICE score (AUROC 0.89, 95% CI = 0.84–0.91) was superior to BOMBARD (0.19 difference, 95% CI = 0.07–0.32) and SIRS criteria (0.33 difference, 95% CI = 0.22–0.44) but not qSOFA (0.07 difference, 95% CI = −0.03 to 0.17) for predicting overall mortality in patients with uncomplicated severe urinary tract infections (UTIs). The AUROC of the qSOFA score was superior to the BOMBARD score (0.13 difference, 95% CI = 0.05–0.2) and SIRS criteria (0.26 difference, 95% CI = 0.15–0.37) for predicting overall mortality. The AUROC of the BOMBARD score was superior to SIRS criteria (0.13 difference, 95% CI = 0.02 to 0.25) for predicting overall mortality in uncomplicated severe UTIs.

Initial Sensitivity comparisons—McNemar test. Sensitivity of an initial PRACTICE score >75 did not differ from a BOMBARD score >1 (13.3 difference, 95% CI = −3.9 to 30.5), SIRS criteria >1 (26.7 difference, 95% CI = 4.3–49.1), and a qSOFA score >1 (33.3 difference, 95% CI = 9.5–57.2) for predicting mortality in uncomplicated severe UTIs. Sensitivity of an initial BOMBARD score >2 did not differ from SIRS criteria >1 (13.3 difference, 95% CI = −3.9 to 30.5) or a qSOFA score >1 (20 difference, 95% CI = −0.2 to 40.2) for predicting mortality in uncomplicated severe UTIs. Sensitivity did not differ between SIRS criteria >1 and a qSOFA score >1 (6.7 difference, 95% CI = −6 to 19.3) predicting mortality in uncomplicated severe UTIs.

Initial Specificity comparisons McNemar test. An initial qSOFA score >1 was more specific than a PRACTICE score >75 (12.6 difference, 95% CI = 9.2–15.9), BOMBARD >2 (39.3 difference, 95% CI = 34.4–44.3), and SIRS >1 (45.5% difference, 95% CI = 40.4–50.5) for predicting mortality in uncomplicated severe UTIs. An initial PRACTICE score >75 was more specific than a BOMBARD score >2 (26.7% difference, 95% CI = 22.3–31.2), and SIRS criteria >1 (32.9 difference, 95% CI = 28.1–37.7) for predicting mortality in uncomplicated severe UTIs. An initial BOMBARD score >2 was more specific than SIRS >1 (6.1 difference, 95% CI = 3.7–8.6) for predicting severe sepsis/septic shock.

Sensitivity of an initial PRACTICE score >75 did not differ from a BOMBARD score >1, SIRS criteria >1, and a qSOFA score >1 for predicting mortality in uncomplicated severe UTIs (Table 3).

Intensive care unit admission prediction

Accuracy and operating characteristics of each criteria and score for predicting ICU admission are detailed for all patients and for patients with uncomplicated UTIs (Figure 2; Tables 4 and 5). The PRACTICE score had a superior net benefit compared to all scores for predicting ICU admission at all probability thresholds in all patients.

FIGURE 2

AUROC comparisons for predicting ICU admission in severe urinary tract infections

TABLE 4

Comparison of operating characteristics and accuracy for predicting ICU admission in all patients

Score/criteria	Sensitivity c	Specificity d	Positive likelihood ratio	Negative likelihood ratio	AUROC b
BOMBARD >2	74.9% (69.5–79.4)	46.6% (42.9–50.3)	1.42 (1.3–1.57)	0.53 (0.43–0.66)	0.66 (0.63–0.69)
PRACTICE >75	79.3% (74.2–83.7)	61.5% (57.6–65.1)	2.1 (1.85–2.3)	0.34 (0.27–0.43)	0.76 (0.73–0.79)
qSOFA >1	45.2% (39.5–51.1)	85.8% (83–88.2)	3.18 (2.55–3.96)	0.64 (0.57–0.71)	0.74 (0.71–0.77)
SIRS >1	67.3% (61.6–72.8)	44.4% (40.7–48.1)	1.21 (1.09–1.34)	0.74 (0.61–0.89)	0.59 (0.56–0.62)

Initial criteria obtained from first set of vital signs in the ED. Initial vital signs were obtained a median of 9 min (4–21 min, interquartile range/IQR) after triage arrival for all patients 95% confidence interval (CI) within parentheses.

The area under the receiver operating characteristic curve receiver operative curve (AUROC) of the initial PRACTICE score was greater than that of the BOMBARD score (0.1 difference, 95% CI = 0.06–0.14), SIRS (0.17 difference, 95% CI = 0.13–0.21), and but not the qSOFA score (0.02 difference, 95% CI = −0.01 to 0.05) for predicting ICU admission. The AUROC of qSOFA was greater than BOMBARD (0.08 difference, 95% CI = 0.05–0.11) and SIRS (0.15 difference, 95% CI = 0.11–0.19) for predicting ICU admission. The AUROC of BOMBARD was greater than SIRS (0.07 difference, 95% CI = 0.04–0.11) for predicting ICU admission.

Initial Sensitivity comparisons—McNemar test. An initial PRACTICE score >75 was more sensitive than an initial BOMBARD score >2 (3.7 difference, 95% CI = 1.6–5.9), initial SIRS criteria >1 (10.5 difference, 95% CI = 7–14.1), and initial qSOFA score >1 (50.3 difference, 95% CI = 45.3–55.2) for predicting ICU admission. An initial BOMBARD score >2 was more sensitive than initial SIRS criteria >1 (6.8 difference, 95% CI = 3.9–9.7) and SIRS criteria >1 (29.6 difference, 95% CI = 24.4–34.8) for predicting ICU admission. Initial SIRS criteria >1 was more sensitive than an initial qSOFA score >1 (22.8 difference, 95% CI = 18–27.6) for predicting ICU admission.

Initial Specificity comparisons McNemar test. An initial qSOFA score >1 was more specific than a PRACTICE score >75 (21.9 difference, 95% CI = 18.9–24.9), BOMBARD >2 (39.2 difference, 95% CI. 35.6–42.3), and SIRS >1 (41.6 difference, 95% CI = 38–45.2) for predicting ICU admission. An initial PRACTICE score >75 was more specific than BOMBARD >2 (17.3 difference, 95% CI = 14.5–20.1) and SIRS >1 (19.7 difference, 95% CI = 16.8–22.6) for predicting severe sepsis/septic shock. An initial BOMBARD score >2 was more specific than SIRS >1 (2.4 difference, 95% CI = 1.3–3.5).

TABLE 5

Comparison of operating characteristics and accuracy for predicting ICU admission in patients with uncomplicated infections

Score/criteria	Sensitivity c	Specificity d	Positive likelihood ratio	Negative likelihood ratio	AUROC b
BOMBARD >2	72% (60.4–81.8)	51.6% (45.9–57.2)	1.49 (1.24–1.78)	0.54 (0.37–0.79)	0.67 (0.62–0.72)
PRACTICE >75	66.7% (54.8–77.1)	77.1% (72–81.6)	2.91 (2.25–3.76)	0.43 (0.31–0.6)	0.81 (0.77–0.85)
qSOFA >1	44% (32.5–55.9)	90.7% (87–93.7)	4.76 (3.1–7.3)	0.62 (0.5–0.76)	0.78 (0.74–0.82)
SIRS >1	68% (56.2–78.3)	42% (36.5–47.7)	1.17 (0.98–1.41)	0.76 (0.53–1.09)	0.6 (0.54–0.64)

Initial criteria obtained from first set of vital signs in the ED. Initial vital signs were obtained a median of 9 min (4–21 min, interquartile range/IQR) after triage arrival for all patients 95% confidence interval (CI) within parentheses.

The initial PRACTICE score (AUROC 0.81, 95% CI = 0.77–0.85) was superior to BOMBARD (0.14 difference, 95% CI = 0.07–0.22) and SIRS criteria (0.22 difference, 95% CI = 0.14–0.29) but not qSOFA (0.03 difference, 95% CI = −0.03 to 0.08) for predicting ICU admission in patients with uncomplicated severe UTIs. The AUROC of the qSOFA score was superior to the BOMBARD score (0.11 difference, 95% CI = 0.06–0.17) and SIRS criteria (0.19 difference, 95% CI = 0.12–0.26) for predicting ICU admission. The AUROC of the BOMBARD score superior to SIRS criteria (0.07 difference, 95% CI = 0.01–0.14) for predicting ICU admission

Initial Sensitivity comparisons—McNemar test. An initial BOMBARD score >2 was more sensitive than a qSOFA score >1 (28 difference, 95% CI = 17.8–38.2), but not a PRACTICE score >75 (4 difference, 95% CI = −1.3 to 8.4) or SIRS criteria >1 (5.3 difference, 95% CI = 0.2–10.4) for predicting ICU admission in uncompleted UTIs. SIRS criteria >1 was more sensitive than an initial qSOFA score >1 (24 difference, 95% CI = 14.3–3.7) but not a PRACTICE score >75 (1.3 difference, 95% CI = −1.3 to 3.9) for predicting ICU admission in uncomplicated UTIs. An initial PRACTICE score >75 was more sensitive than initial an initial qSOFA score >1 (22.7 difference, 95% CI = 13.2–32.1) for predicting ICU admission.

Initial Specificity comparisons McNemar test. An initial qSOFA score >1 was more specific than PRACTICE >75 (11.2 difference, 95% CI = 7.7–14.6), BOMBARD >2 (41.1 difference, 95% CI = 35.6–46.5), and SIRS >1 (48.7 difference, 95% CI = 43.2–54.3) for predicting ICU admission. An initial PRACTICE score >75 was more specific than a BOMBARD score >2 (29.9 difference, 95% CI = 24.9–35) and SIRS criteria >1 (37.6 difference, 95% CI = 32.2–42.9) for predicting ICU admission. An initial BOMBARD score >2 was more specific than SIRS >1 (7.7 difference, 95% CI = 4.7–10.7) for predicting ICU.

Interrater reliability

The intraclass correlation coefficient and kappa for the chart abstraction of individual features and vital signs of analyzed scores and criteria were each 1 with the following exceptions: disabled immune system (k = 0.96, 95% CI = 0.89–1), malignancy (k = 0.9, 95% CI = 0.7–1), and renal disease (k = 0.77, 95% CI = 0.52–1). The kappa value for the total BOMBARD score was 0.99 (95% CI = 0.98–1), total PRACTICE score was 0.97 (95% CI = 0.93–1), total SIRS criteria was 1, and total qSOFA was 1. The intraclass correlation coefficient (ICC) for chart abstraction of the BOMBARD score, qSOFA score, and SIRS criteria was 1. The ICC for chart abstraction of the PRACTICE score was 0.99 (95% CI = 0.99–1).

LIMITATIONS

We chose to not study patients diagnosed with lower UTIs including simple UTIs, prostatitis, epididymitis, or orchitis unless those patients had or developed acute pyelonephritis or severe sepsis/septic shock during hospitalization because they were not included in guidelines defining severe UTIs and mostly comprise a group of patients already designated as having complicated UTIs (males). , Patient diagnoses were based on assigned billing codes that reflect review of charts by hospital coding abstractors supervised by physicians for documentation of pyelonephritis. In addition to clinical documentation of a urine infection, CMS guidelines using Sepsis‐2 consensus definitions requiring evidence of organ dysfunction were used by hospital coding abstractors to determine if patients had severe sepsis or septic shock. Whether or not a positive urinalysis indicates a significant infection can be subjective with signs and symptoms being unreliable. , Without radiological proof of an upper tract infection (eg, positive dimercaptosuccinic acid scan or diffusion weight magnetic resonance imaging), the diagnosis of pyelonephritis is largely clinical. For this reason, we chose the presence of both a positive urinalysis and inpatient diagnosis of pyelonephritis‐ or UTI‐related severe sepsis‐septic shock during the first 48 hours of admission to select patients.

Our study was not designed to distinguish between those patients who had pyelonephritis or sepsis syndromes in the ED or developed those disorders later during their hospitalization. Our goal was to identify all patients with the most serious adverse UTI‐related outcomes without regard to whether or not features of severe illness was present in the ED.

The outcome measure of ICU admission potentially reflects a decision process and not an entirely objective outcome like mortality. The decision to admit patients from the ED or transfer patients from an inpatient ward to an ICU can be affected by hospital protocols, ICU census, ICU and inpatient ward staffing, and other factors that might not be related to disease severity. However, experts who developed pneumonia severity scores (eg, PORT/Pneumonia Severity Index, CURB‐65) have identified ICU admission as a marker for identifying disease severity in patients admitted to the hospital. , For this reason, we chose ICU admission as an outcome measure to compare prediction tools in our study.

A drawback to using net benefit to make inferences about prediction scores is that threshold probabilities are not uniformly distributed in the range of interest. For many clinical scenarios, individuals may have a lower threshold probability for detecting and treating diseases associated with mortality and a higher threshold probability for treating less serious diseases. A lower threshold probability for mortality would tend to favor the PRACTICE score in our population since the outcome is death.

None of the studied criteria or scores contain indications for procedural intervention that might impact admission decisions. Patients with urinary foreign bodies (eg, stone, stent, and nephrostomy tube) and anatomic abnormalities comprise a subset of patients with complicated infections who might require admission regardless of clinical severity. It is uncertain if a clinical decision rule based on ED triage data would be useful in those with complicated disease. For this reason, we chose to separately evaluate the use of each test in the subset of patients with uncomplicated UTIs.

Overall mortality, and not sepsis‐related mortality, was a major outcome analyzed in this study. Because all patients had serious infections, study authors felt that accurately discriminating between infection‐induced mortality and unrelated mortality (eg, respiratory failure, myocardial infarction) was not possible.

Because this was a retrospective study, interrater reliability of the clinical assessment of individual features was not performed. Others have found that vital signs and prediction scores based on physiologic variables have good to excellent interrater reliability. , , , Ideally, clinical features should be collected prospectively with repeated measurements by multiple individuals to assess interrater reliability before implementation.

Before using a clinical decision tool, researchers should assess whether that tool improves clinical judgment. , Prediction tools often improve sensitivity in detecting pathology at the expense of decreased specificity. , This may lead to unnecessary testing, treatment, and hospital admissions.

DISCUSSION

There are no widely used screens to risk stratify or make admission decisions in patients with severe UTIs. Our study found that a previously described modified pulmonary severity index (PRACTICE) showed superior accuracy for predicting mortality and ICU admissions compared to the BOMBARD score, SIRS criteria, and qSOFA score. A previously derived cutoff (PRACTICE >75 versus ≤75) was more sensitive than each alternate score for predicting mortality. Conversely, our study found that an initial qSOFA score >1 had a higher positive likelihood ratio for predicting mortality (2.59 in all, 4.23 in uncomplicated) and was more specific than a PRACTICE score >75 for predicting mortality and ICU admission. The qSOFA score had a greater net benefit compared to PRACTICE at threshold probabilities above 10% whereas PRACTICE had greater net benefit below this level. Higher sensitivity of the PRACTICE score, a lower negative likelihood ratio (0.19 in all, 0.09 in uncomplicated), and superior mortality prediction at lower threshold probabilities compared to qSOFA indicate that the PRACTICE score is preferable for predicting a mortality in individuals with severe UTIs.

In the United States, 10%–30% of adults with pyelonephritis require hospitalization with an estimated cost of $2.9 billion per year. There are no guidelines that provide a clear distinction which patients with UTIs require admission to the hospital. Prior Infectious Disease Society of American guidelines recommended hospitalization for patients with nausea or vomiting that precludes oral intake, high fever (undefined), high WBC count (undefined), dehydration, or evidence of sepsis with no distinction between simple sepsis, severe sepsis, and septic shock. , The European Association of Urology recommends hospital admission if complicating factors cannot be ruled out by diagnostic procedures or patients have features of sepsis (no distinction between simple sepsis or severe sepsis/shock). , Textbooks describe many subjective indications for admission including “frailty,” “more than mild illness,” “severe illness,” lack of “adequate hydration,” “toxicity,” the “response to initial ED interventions,” “systemic signs of UTI,” and “poor social support.” , , , , , , , Because of the reliance on subjective markers of illness severity by many experts, an objective clinical score might serve to aid in decisionmaking. For this reason, we chose to analyze a previously published rule for making admission decision in patients with UTIs (PRACTICE), in addition to tools used to stratify patients at risk for sepsis (BOMBARD, qSOFA, and SIRS).

The original PRACTICE score was derived to predict adult ED patients with febrile UTIs who required initial hospital admission or failed initial home‐based treatment and required eventual hospitalization. Our population included patients who had or developed severe sepsis/septic shock and acute pyelonephritis regardless of whether or not a fever was present. We did not include non‐hospitalized ED patients unless that patient was later admitted to an index hospital. Since we included patients with UTI‐related severe sepsis/septic shock, hospital mortality rate in our study (6.9%) was higher than mortality in the original PRACTICE study (<1%).

Of the 7 deaths with a PRACTICE score ≤75, 6 had sepsis‐related organ dysfunction in the ED, 3 had co‐existing pneumonia and one had an uncomplicated UTI (Supporting Information Table S3). The single in‐hospital mortality with an uncomplicated UTI and an initial PRACTICE score ≤75 was a 52‐year‐old female that presented with a possible gastrointestinal bleed and an initial blood pressure of 83/35 mm Hg. Thus, there were alternate reasons to admit most patients with a low initial PRACTICE score who died.

A PRACTICE score >75 was more sensitive than other scores for predicting ICU admission. Sixty‐three (21.5%) ICU admissions had a PRACTICE score ≤75 with 8 deaths (11.4%) not admitted to the ICU. Five of these 8 were hospice patients or designated do not resuscitate in the ED, 2 were designated do not resuscitate within 24 hours of admission, and 1 95‐year‐old female with metastatic gastrointestinal cancer had a coexisting non‐STEMI and sepsis‐related UTI with a ureteral stone. It is likely that many of these cases would have been admitted to the ICU if they were not designated do not resuscitate. Due to the retrospective nature of this study, it is not possible to determine the effect of end‐of‐life care decisions on the aggressiveness of overall treatment, mortality, or likelihood patients would be admitted to an ICU.

Mortality was 1.3% in individuals with an initial PRACTICE score ≤75. This rate is higher than pneumonia‐related mortality rates in those with the lowest Pneumonia Severity Index risk classes of I and II (0.1% to 0.9%) and lower than those of pneumonia patients in the indeterminate Pneumonia Severity Index risk class III (0.9%–2.8%). In the subset with uncomplicated disease, mortality was 0.4% in those with an initial PRACTICE score ≤75. This mortality rate is consistent with cutoffs for discharging pneumonia patients home based on their Pneumonia Severity Index. These findings indicate that some individuals with a low PRACTICE score potentially could be managed as outpatients.

In summary, the PRACTICE score was more accurate and sensitive at predefined cutoffs than BOMBARD, SIRS, and qSOFA for predicting mortality in patients with severe UTIs. Prospective studies are needed to determine if PRACTICE can augment clinical judgement when considering treatment decisions in the broader group of ED patients with UTIs.

CONFLICTS OF INTEREST

The authors declare no conflicts of interest.

AUTHOR CONTRIBUTIONS

All authors (SGR, DDC, BG, DB, EH, JB, NT, MR, DY, CP, DB, and ORMC Sepsis study group) were involved in conception of study, design of the study, data collection and abstraction, drafting and revision of the manuscript. All authors analyzed the data and SGR performed statistical analyses. All authors take responsibility for the paper as a whole. SGR takes the final responsibility of the paper as a whole.

FIGURE S1 Decision curve analysis for the prediction of mortality and ICU admission. (A) Mortality prediction all patients. (B) Mortality prediction uncomplicated urinary tract infections (UTIs). (C) Prediction of ICU admission in all patients. (D) Prediction of ICU admission in uncomplicated UTIs. Yellow dotted line, net benefit for PRACTICE score; blue dotted line, qSOFA score; green dotted line, BOMBARD score; red dotted line, SIRS criteria. The solid blue line is the net benefit if all patients are treated, and the solid red line is the net benefit if no patients are treated.

Click here for additional data file.

Supplementary information

Click here for additional data file.

Supplementary information

Click here for additional data file.

Supplementary information

Click here for additional data file.