Net benefit of routine urine parameters for urinary tract infection screening: a decision curve analysis.

BACKGROUND: Whether routine urinary analysis has a net benefit for urinary tract infection (UTI) screening is unclear.
METHODS: Using the laboratory information system (LIS), we retrospectively extracted the data of urine culture and routine analysis between January 2017 and April 2017. Receiver operating characteristic (ROC) curve, logistic regression model, net reclassification improvement (NRI) and integrated discrimination improvement (IDI) were used to estimate the screening performance of routine urinary analysis. Decision curve analysis (DCA) was used to estimate the net benefit of routine urinary analysis.
RESULTS: A total of 927 specimens with 156 UTIs were included in the present study. The area under ROC curves (AUCs) of white blood cells (WBCs) and bacteria were 0.729 and 0.836, respectively. The logistic regression model incorporating WBCs, bacteria and nitrite together had an AUC of 0.851, which is significantly higher than that of WBCs. NRI and IDI analyses also indicated that WBCs, bacteria and nitrite, when used together, had better a screening performance than each single test alone. DCA revealed that 0.08 net benefit can be obtained for bacteria and the model, while the net benefit of WBCs is limited.
CONCLUSIONS: WBCs, bacteria and nitrite, when used together, can significantly improve the efficiency for UTI screening. Bacteria and the model incorporating WBCs, bacteria and nitrite have a net benefit in UTI screening, while the net benefit of WBCs, when used alone, is limited. 2020 Annals of Translational Medicine. All rights reserved.

Introduction

Urinary tract infection (UTI) is common in clinical practice and urine culture is the “gold standard” for its diagnosis (1). However, urine culture has its intrinsic disadvantages. First, it is time- and labor-consuming and does not facilitate early diagnosis. Second, a large portion of specimens sent for urine culture are negative and thus increase the workload of the clinical microbiology laboratory. Therefore, developing UTI screening tools to decrease the rates of unnecessary urine culture is of great value. However, it should be kept in mind that the screening tools are typical “double-edged sword”, although it can decrease the rates of unnecessary urine culture, some UTI patients may be missed. Therefore, it is necessary to evaluate whether the screening tool is of net benefit.

Routine urine analysis has been widely used in UTI screening for a long time. A previous meta-analysis indicated that white blood cells (WBCs) and bacteria, which are measured by automated urine analyzer, are useful tools for UTI screening (2). The sensitivity and specificity for WBCs in the previous meta-analysis were 0.87 and 0.67, respectively. For urine bacteria, the sensitivity and specificity were 0.92 and 0.60, respectively (2). Additionally, two previous studies also indicated that urine nitrite is of value for UTI diagnosis (3,4). However, to the best of our knowledge, this type of research has only reported the sensitivity, specificity and area under receiver operating characteristic (ROC) curve (AUC). These metrics, although they have been widely used in screening or diagnostic trials, only reflect the predictive accuracy of an index test and do not incorporate information on consequences. Additionally, it also remains unclear whether these tests, when used together, can improve the predictive accuracy and net benefit for UTI.

In this study, we investigated the predictive accuracy of a combined model incorporating WBCs, bacterial and nitrite using the logistic regression model, net reclassification improvement (NRI) and integrated discrimination improvement (IDI). Additionally, the net benefit of the model and the single test was analyzed using decision curve analysis (DCA).

Methods

Data collection

This is a retrospective observational study using the data stored in the laboratory information system (LIS) of our institution. All urine specimens sent to the clinical microbiology laboratory for culture between January 2017 and April 2017 were eligible. The exclusion criteria were the following: (I) specimens without routine urine analysis within 24 hours before or after urine culture; (II) contaminated specimens, defined as more than three types of bacteria identified. Of the remaining eligible cases, we extracted the following data from the LIS: age, sex, urine culture results, urine WBCs, bacteria and nitrite. Urine WBCs and bacteria were detected with Sysmex UF-1000i automated urine sediment analyzer. UTI was defined as more than 105 cfu/mL bacterial growth, as previously described (5).

This study was approved by the ethics committee of our institution (No. KY2018026) and the study was performed in accordance with the Declaration of Helsinki. Informed consent was waived because the retrospective design of this study.

Sample size estimation

According to our experience, the prevalence of UTI is approximately 15% in the urine specimens sent to clinical microbiology laboratory for culture. The alpha, AUC of ROC and its 95% confidence interval (95% CI) width were set at 0.05, 0.80 and 0.10, respectively. The calculated sample size was 780, including 117 UTI specimens from UTI patients. Sample estimation was performed with PASS 15 (NCSS Statistical Software, Kaysville, Utah, USA).

Statistical analysis

Because all continuous variables were not normally distributed (Kolmogorov-Smirnov test), they were expressed as median and interquartile and compared with the Mann-Whitney test. Categorical variables were expressed as absolute number and compared with the chi-square test. The ROC curve analysis was used to estimate the screening performance of WBCs and bacteria for UTI. We used the logistic regression analysis to incorporate WBCs, bacteria and nitrite into a model and estimated the screening performance of the model using ROC curve analysis. The threshold with maximum Youden index was chosen as the optimal threshold and the corresponding sensitivity and specificity were calculated. NRI and IDI were used to estimate whether the model, by incorporating WBCs, bacteria and nitrite, provides added values beyond a single test (6). DCA was used to estimate the net benefit of WBCs, bacteria and the model (7). All analyses were performed with Sigmaplot 12.0 (Systat Software Inc., San Jose, CA, USA), SPSS 18.0 (SPSS, Inc., Chicago, IL, USA) and R (https://www.r-project.org/, version 3.5.0). The packages used with R included PredictABLE (8) and rmda (9). A P value less than 0.05 was defined as statistically significant.

Results

Characteristics of the subjects

Between January 2017 and April 2017, 1,525 urine specimens were sent to our clinical microbiology laboratory for culture. Among them, 140 specimens were contaminated, and 569 specimens did not have results of urine flow cytometry. Finally, 927 specimens with 156 UTIs were included in the present study. The characteristics of these specimens are summarized in .

Table 1

Summary of specimens

Characteristics	UTI (n=156)	Non-UTI (n=771)	P value
Age [year]	61 [50–75]	57 [41–69]	<0.01
Sex (M/F)	56/100	316/455	0.27
WBCs (/μL)	190 (66–1,270)	32 (7–163)	<0.01
Bacteria (/μL)	6110 (698–17,413)	5/5 (14–381)	<0.01
Nitrite (Y/N)	73/83	48/723	<0.01
Pathogens
Escherichia coli	61
Klebsiella pneumoniae	8
Staphylococcus epidermidis	6
Streptococcus agalactiae	6
Enterococcus faecalis	5
Enterobacter cloacae	4
Others	26

Continues variables were expressed as median (interquartile) and compared with the Mann-Whitney U test. Category variables were expressed as an absolute number and compared with the chi-square test. WBC, white blood cell.

Assessment the performance of WBCs and bacteria for UTI screening using ROC curve analysis, NRI and IDI

is a ROC curve depicting the screening accuracy of WBCs and bacteria for UTI. The AUCs (95% CI) of WBCs and bacteria were 0.729 (95% CI: 0.690–0.769) and 0.836 (95% CI: 0.800–0.872), respectively. Next, we used logistic regression analysis to incorporate WBCs, bacteria and nitrite into a model. The coefficients and odds ratios (ORs) of WBCs, bacteria and nitrite are listed in .

Figure 1

ROC curve depicting the performance of WBCs, bacteria and their combination for UTI screening. ROC, receiver operating characteristic; WBC, white blood cell; UTI, urinary tract infection.

Table 2

Logistic regression analysis with WBCs, bacteria and nitrite

Parameters	β	OR (95% CI)	P
WBCs per 100/μL	0.011	1.011 (1.002–1.020)	0.015
Bacteria per 1,000/μL	0.040	1.041 (1.018–1.064)	<0.001
Positive nitrite	1.962	7.111 (4.212–12.005)	<0.001
Constant	–2.315	0.099	NA

WBC, white blood cell; OR, odds ratio; CI, confidence interval; NA, not applicable.

The AUCs, optimal sensitivities and specificities of WBCs, bacteria and the model are listed in . The AUC for this model was 0.851 (95% CI: 0.820–0.883), which was significantly higher than that of WBCs (P<0.001), but not significantly higher than that of bacteria (P=0.531).

Table 3

Screening accuracy of WBCs, bacteria and their combined model

Metrics	WBCs	Bacteria	Model
AUC (95% CI)	0.729 (0.690–0.769)	0.836 (0.800–0.872)	0.851 (0.820–0.883)
Cut-off	77.5/μL	594.1/μL	0.094
Sensitivity (95% CI)	0.724 (0.647–0.793)	0.763 (0.688–0.823)	0.814 (0.744–0.872)
Specificity (95% CI)	0.660 (0.626–0.694)	0.793 (0.762–0.821)	0.742 (0.710–0.773)

WBC, white blood cell; AUC, area under curve; CI, confidence interval.

Considering that the AUC of bacteria was significantly higher than that of WBCs, we further analyzed whether WBCs and nitrite could provide added value in addition to bacteria. Using IDI and NRI, we found that nitrite and WBCs could significantly improve the predictive accuracy of bacteria, with a continuous NRI of 0.642 (95% CI: 0.484–0.800; P<0.001) and an IDI of 0.070 (95% CI: 0.042–0.098; P<0.001).

Assessment of the clinical efficiency of WBCs and bacteria for UTI screening using DCA

is a decision curve depicting the clinical efficiency of WBCs, bacteria and their combination (the model) for screening UTI. Notably, the clinical efficiency of their combination was higher than that of both bacteria and WBCs alone. The net benefit of bacteria and the model was closed 0.08; at a threshold of 0.30, the net benefit of WBCs, bacteria and the model were 0.0025, 0.0419 and 0.0627, respectively (). Using the net benefit formula (7), we calculated that the number of false positive results per 100 patients decreased, with WBCs, bacteria and the model being approximately 1, 10 and 15, respectively, with no decrease in the number of patients with UTI.

Figure 2

Decision curve depicting the clinical efficiency of WBCs, bacteria and their combination. WBC, white blood cell.

Table 4

Net benefit of WBCs, bacteria and the model at a threshold of 0.30

Parameters	Corresponding threshold at probability of 0.30	Net benefit	Reduction in urine culture per 100 subjects
WBCs	4,677/μL	0.0025	0.58
Bacteria	12,910/μL	0.0419	9.78
Model	0.30	0.0627	14.63

WBC, white blood cell.

Discussion

In this study, we found that urine WBCs, bacteria and nitrite were useful screening tools for UTI screening. Using DCA, we found that bacteria had a high net benefit for UTI screening while the net benefit introduced by WBCs was low. Furthermore, combined use of WBCs, bacteria and nitrite had a higher net benefit than using bacteria alone.

Similar to a previous meta-analysis (2), we found that both WBCs and bacteria had a higher screening performance for UTI screening, and the screening performance of bacteria was significantly higher than that of WBCs. However, compared with previous studies, our study had two strengths. First, we found that combined use of WBCs, bacteria and nitrite improved the efficiency of UTI screening. The logistic regression model incorporating WBCs, bacteria and nitrite had a significantly higher AUC than that of WBCs. Furthermore, the results of IDI and NRI indicated that combined use of WBCs, nitrite and bacteria had a higher screening efficiency than the single parameter alone. Second, previous studies only calculated the AUC, optimal thresholds, and corresponding sensitivity and specificity of routine urine parameters for UTI screening. However, these metrics only concern the accuracy of an index test, and do not incorporate information on clinical consequence (7). In this study, we used DCA to estimate the net benefit of these parameters. Unlike traditional ROC curve analysis, DCA uses “treat all” and “treat none” as reference and can calculate the net benefit of an index test. We noted that the decision curve of WBCs nearly overlapped with the reference line (“treat all” and “treat none”), indicating that it has limited value in UTI screening, although its AUC was 0.729. By contrast, the decision curve of bacteria was notably above the reference line, indicating that introducing urine bacteria into the UTI screening procedure has a high net benefit. Furthermore, we noted that the decision curve of the model incorporating bacteria, WBCs and nitrite was above that of bacteria, indicating that combined use of WBCs, bacteria and nitrite has a significantly higher net benefit than that of a single one.

Our findings had some clinical implications. Because WBCs, bacteria and nitrite are usually tested together to screen UTI in clinical practice, the coefficients () of these parameters can be easily incorporated into the LIS and thus the probability of UTI can be easily calculated. Listing this probability into their laboratory reports may help clinicians better manage suspected UTI subjects.

Our study has two limitations. First, this is a single center study with a small sample size. The coefficients of WBCs, bacteria and nitrite in a logistic regression model may be affected by the prevalence of the UTI and the disease spectrum in the target population. Second, this is a retrospective study based on the specimens sent to the clinical microbiology laboratory, and no inclusion and exclusion criteria were defined on symptoms and signs. Therefore, the representativeness of the studied cohort may be negatively affected. Our results need to be validated by other centers. Despite these limitations, our study did analyze the screening performance of routine urinary parameters using DCA and thus can provide a new insight into UTI screening. Further prospective, multicenter studies with larger sample sizes are needed to validate the findings of this study.

Conclusions

In conclusion, our study indicates that urine bacteria had a high net benefit for UTI screening, while WBCs had limited value. Furthermore, combined use of bacteria, WBCs and nitrite had a higher net benefit than any single one of these parameters alone.

The article’s supplementary files as