Clinical accuracy of point-of-care urine culture in general practice.

OBJECTIVE: To assess the clinical accuracy (sensitivity (SEN), specificity (SPE), positive predictive value and negative predictive value) of two point-of-care (POC) urine culture tests for the identification of urinary tract infection (UTI) in general practice.
DESIGN: Prospective diagnostic accuracy study comparing two index tests (Flexicult™ SSI-Urinary Kit or ID Flexicult™) with a reference standard (urine culture performed in the microbiological department).
SETTING: General practice in the Copenhagen area patients. Adult female patients consulting their general practitioner with suspected uncomplicated, symptomatic UTI. MAIN OUTCOME MEASURES: (1) Overall accuracy of POC urine culture in general practice. (2) Individual accuracy of each of the two POC tests in this study. (3) Accuracy of POC urine culture in general practice with enterococci excluded, since enterococci are known to multiply in boric acid used for transportation for the reference standard. (4) Accuracy based on expert reading of photographs of POC urine cultures performed in general practice. Standard culture performed in the microbiological department was used as reference standard for all four measures.
RESULTS: Twenty general practices recruited 341 patients with suspected uncomplicated UTI. The overall agreement between index test and reference was 0.76 (CI: 0.71-0.80), SEN 0.88 (CI: 0.83-0.92) and SPE 0.55 (CI: 0.46-0.64). The two POC tests produced similar results individually. Overall agreement with enterococci excluded was 0.82 (0.77-0.86) and agreement between expert readings of photographs and reference results was 0.81 (CI: 0.76-0.85).
CONCLUSIONS: POC culture used in general practice has high SEN but low SPE. Low SPE could be due to both misinterpretation in general practice and an imperfect reference standard. Registration number: ClinicalTrials.gov NCT02323087.

Introduction

Urinary tract infection (UTI) is common in general practice and is the second leading reason for antibiotic prescriptions [1]. Patients with suspected UTI are usually treated with antibiotics, since it is a painful and bothersome condition and antibiotic treatment shortens the duration of the symptoms [2-4]. However, overtreatment can result in unnecessary side effects for the patient and increasing bacterial resistance [5-8]. Accurate diagnosis is essential for correct treatment, and combinations of symptoms and urine dipstick tests have proved inadequate for establishing or ruling out UTI [9-11]. This has led to the use of point-of-care (POC) urine culture in general practice in Scandinavia for both complicated and uncomplicated UTI. Most guidelines recommend treating uncomplicated UTI based on symptoms and urine dipstick findings [12,13]. However, in a recent study conducted in an outpatient setting in Norway, patients with suspected uncomplicated UTI were treated based on dipstick and symptoms, which lead to antibiotic treatment of almost all patients although 43% did not have confirmed UTI [14]. Performing additional urine tests to increase accuracy could potentially decrease overtreatment of both uncomplicated and complicated UTI. POC urine culture can usually be performed by practice staff and has the advantage of providing a definite result within 24 h if handled correctly, while sending urine to the microbiological department usually involves a delay of several days [15-17]. The Flexicult™ SSI-Urinary Kit test and ID Flexicult™ (SSI Diagnostica, Denmark) are available in general practice in Denmark and have proven accurate in several laboratory studies and one validation study, but remain to be tested in the daily practice setting [17,18].

The aim of this study was to determine the accuracy of chromogenic agar-based POC culture in identifying significant bacteriuria in women with symptoms of UTI and a positive dipstick finding (leucocytes or nitrites) in general practice. A secondary analysis of the results excluding enterococci was performed in order to take into account the potential multiplication of entercocci during transport to the reference laboratories in boric acid tubes [19-21]. A separate analysis was performed based on expert readings of photographs of POC culture plates from the study in order to investigate whether accuracy could be improved if the plates were read by an expert.

Materials and methods

This study is based on data from a randomized controlled trial, in which the design is described thoroughly in the protocol [22].

Recruitment of practices

An invitation letter was mailed to 200 randomly selected general practices in the Copenhagen area with the aim of recruiting 50 general practitioners (GPs) with experience in using POC culture. The GPs who were recruited participated in a pre-study instruction course on handling and reading both POC tests, and had to pass an online test prior to the inclusion of patients.

Recruitment of patients

Female adult patients, 18 years or older, presenting to their GP between 1 March 2015 and 1 May 2016 with dysuria, frequency or urgency, of 7 days duration or less, and for whom the GP suspected uncomplicated UTI, were included in the study. Exclusion criteria were negative dipstick analysis on leucocytes and nitrites, complicated UTI (except uncomplicated diabetes, elderly patients and recurrent UTI), previous participation in the study and patients presenting on a Friday (since the POC is read the following day).

Urine sampling and transportation

Having given informed consent, patients were randomized to one of the two POC tests and instructed to deliver a midstream urine sample without prior cleaning in accordance with Danish recommendations [23]. Part of the urine sample was inoculated immediately on the POC test and the remaining urine sample was sent to the microbiological department in a standardized boric acid container (Urine-Monovette®, Sarstedt, Germany).

POC tests (index test)

The ID Flexicult™ (SSI DIagnostica, Denmark) is a chromogenic agar allowing identification and quantification of: (1) Escherichia coli, (2) Other Enterobacteriaceae (Gram-negative rods), (3) Enterococci, (4) Proteus Spp., (5) Staphylococcus saprophyticus and (6) Pseudomonas aeruginosa. The plate is inoculated with freshly voided urine using a 10 μL loop-needle and incubated at 35 °C overnight. It is read the following day, but negative culture can only be determined after 24 h. Significant growth was prespecified as ≥103 colony-forming units per milliliter (CFU/mL) for E. coli and S. saprophyticus, 104 CFU/mL for other typical uropathogens in accordance with European consensus [24].

The Flexicult™ SSI-Urinary Kit (SSI Diagnostica, Hillerød, Denmark) is an agar dish consisting of one big well containing the same agar material as in the ID Flexicult™ and five small wells containing agar with one of five antibiotics: (1) trimethoprim, (2) sulfamethizole, (3) ampicillin, (4) nitrofurantoin and (5) mecillinam. The plate is inoculated by flooding with urine for 3–5 s and hereafter discarding superfluous urine. The plate is incubated and handled as the ID Flexicult™. Significant growth was prespecified (advised by manufacturer) to ≥103 CFU/mL for any uropathogen.

The GPs registered the index test as ‘significant growth of uropathogens’, ‘no significant growth of uropathogens’ or ‘inconclusive’. A positive result of the index test was defined as having ‘significant growth of uropathogens’, while ‘No significant growth of uropathogens’ or ‘inconclusive’ were labeled as negative.

Photographs of index tests

All index tests were photographed using a digital camera. The primary investigator (AH) interpreted photographs, and a separate analysis was performed with the result of the photograph reading by AH as the index test to investigate whether accuracy could be improved if plates were read by an expert unaffected by the patient history.

Reference test and laboratories

Urine samples were sent by a specialized delivery service to the reference microbiological laboratories at the Department of Clinical Microbiology, Copenhagen University Hospital, Herlev, Denmark or the Department of Clinical Microbiology, Copenhagen University Hospital, Hvidovre, Denmark. Urine samples were analyzed on Inoqul A™ Bi-plate (CHROMagar and blood agar) with 10 μL on each half of the agar. All samples were quantified. Significant growth was defined as growth of ≥103 CFU/mL for E. coli and S. saprophyticus, ≥104 CFU/mL for other typical uropathogens and ≥105 CFU/ml for possible uropathogens in accordance with European consensus [24]. Plates with growth of more than two uropathogens were labeled as mixed cultures. A positive result was defined as having significant growth of uropathogens, while all other results including mixed flora were labeled as negative.

Data collection and management

Information regarding symptoms, interpretation of culture (positive/negative/inconclusive) and identification, quantification and susceptibility pattern of possible uropathogens were recorded in case report forms by the GPs or their staff. The data were double-typed. Results from the microbiological department were obtained from the hospital laboratory system and linked with the case report forms from general practice using social security numbers.

Blinding

The interpreter of the POC index test in general practice was blinded to the result of the reference test, as far as the result of the reference test was not available before 2–3 days and the result of the index test was consistently recorded 24 h after the consultation. The interpreter of the reference test was likewise blinded to the result of the index test. AH was blinded to both the interpretation from general practice and the microbiological department when evaluating the photographs.

Statistical analysis

Sensitivity (SEN), specificity (SPE), positive predictive value (PPV), negative predictive value (NPV) and agreement (AGR, true positives + true negatives/all) were calculated. 95% confidence intervals (95% CI) for this collection of proportions were calculated with the exact method. Statistical analysis was performed using SAS version 9.4 for Windows 7, SAS Institute Inc (Cary, NC).

Results

Baseline data

Twenty general practices with a total of 45 GPs were recruited from the Copenhagen area. Only three were solo practices. Twenty practices recruited 341 female, non-pregnant patients with symptoms of UTI (mean age: 48.5 years). Data collection can be seen in the attrition flowchart (Figure 1). The prevalence of UTI was 72% according to the two index tests and 63% according to the reference standard. The most prevalent uropathogen in both general practice and the microbiological department was E. coli. In general practice, the second most frequent single uropathogen was enterococci; however, this uropathogen was not identified on reference cultures. See Table 1 for details.

Figure 1.

Attrition flowchart.

Table 1.

Characteristics of samples from 341 patients: test results and distribution of uropathogens from general practice and the microbiological department.

	Index tests	Reference
Both index tests, n = 341
Significant growth	246 (72%)	215 (63%)
E. coli	128 (38%)	176 (51%)
Enterococci	40 (12%)	0 (0%)
S. saprophyticus	15 (4%)	8 (2%)
Other single uropathogen	19 (6%)	25 (7%)
Two uropathogens	44 (13%)	6 (2%)
Index test 1: ID Flexicult™, n = 158
Significant growth	116 (74%)	103 (65%)
E. coli	71 (45%)	89 (56%)
Enterococci	16 (11%)	0 (0%)
S. saprophyticus	7 (4%)	2 (1%)
Other single uropathogen	10 (6%)	10 (6%)
Two uropathogens	12 (8%)	2 (1%)
Index test 2: SSI-Urinary Kit, n = 183
Significant growth	129 (70%)	112 (61%)
E. coli	57 (31%)	87 (48%)
Enterococci	23 (13%)	0 (0%)
S. saprophyticus	8 (4%)	6 (3%)
Other single uropathogen	9 (5%)	15 (8%)
Two uropathogens	32 (17%)	4 (2%)

Percentages in brackets.

Accuracy

Table 2 shows the measures of test accuracy for the various analyses. Overall agreement of POC urine culture with the reference was 0.76 (95% CI: 0.71–0.80). SEN was 0.88 (95% CI: 0.83–0.92) and SPE was 0.55 (95% CI: 0.46–0.64). The two tests produced similar results. Since estimation of enterococcal growth after transportation in boric acid was expected to pose a challenge, a subgroup analysis was performed without enterococci monocultures identified in general practice. This improved overall SPE from 0.55 to 0.71 without lowering SEN. Expert photograph reading by AH (including enterococci) increased SPE to 0.71, and agreement to 0.81 but did not change SEN.

Table 2.

Accuracy of point-of-care culture in relation to culture at the reference laboratories.

	N	PPV	NPV	SEN	SPE	AGR
All cultures	341	0.77 (0.71–0.82)	0.73 (0.63–0.81)	0.88 (0.83–0.92)	0.55 (0.46–0.64)	0.76 (0.71–0.80)
Enterococci in practice culture excluded	301	0.86 (0.81–0.91)	0.73 (0.63–0.81)	0.87 (0.82–0.92)	0.71 (0.61–0.80)	0.82 (0.77–0.86)
ID Flexicult™	158	0.79 (0.71–0.86)	0.76 (0.60–0.88)	0.90 (0.83–0.95)	0.56 (0.42–0.70)	0.78 (0.71–0.85)
SSI-Urinary Kit	183	0.74 (0.66–0.82)	0.70 (0.56–0.82)	0.86 (0.78–0.92)	0.54 (0.41–0.65)	0.73 (0.66–0.79)
All cultures – photo readings	309	0.84 (0.79–0.89)	0.75 (0.66–0.83)	0.87 (0.81–0.91)	0.71 (0.62–0.79)	0.81 (0.76–0.85)

PPV: positive predictive value; NPV: negative predictive value; SEN: sensitivity; SPE: specificity; AGR: agreement (true positives + true negatives/all).

95% confidence intervals in brackets.

Photograph evaluation of discrepancies between index tests and reference

83 index test results differed from the reference. 75 of these had a photograph acceptable for evaluation. When evaluating these photographs, 31 (41%) of discrepancies could be explained by incorrect interpretation of the culture plate, since the photograph reading corresponded to the reference while the interpretation in general practice did not. For enterococci identified in general practice, but with a negative reference culture, 13 out of 28 (46%) were overdiagnosed due to incorrect interpretation of the culture plate. In one case, the photograph was missing. Figure 2 shows six examples of culture plates with discrepancies between the index test and the reference.

Figure 2.

Examples of cultures diagnosed incorrectly in general practice according to the reference standard. A and D: Correctly answered as negative in general practice according to the photograph and as S. saprofyticus 104 CFU/mL and Citrobacter koseri 104 CFU/mL in the microbiological department. B and E: Correctly answered as E. coli 103 CFU/mL and Enterococcus faecalis 105 CFU/mL in general practice but as negative in the microbiological department. C and F: Incorrectly answered as significant growth in general practice, and as negative and mixed flora in the microbiological department.

Identification of uropathogens

Table 3 shows the identification of uropathogens and the agreement between results from general practice and the microbiological department. E. coli identified in general practice was also identified by the microbiological department in 114 out of 128 cases (89%). The total number of monoculture E. coli identified by the microbiological department was 176 and general practice identified 114 of these (65%). Sixty-two remaining cultures were reported as two uropathogens in 27 cases from general practice. Nineteen of these were E. coli and enterococci and this combination was reported as monoculture E. coli by the microbiological department according to their guideline. When this discrepancy in identification is taken into consideration, general practice identified 76% of the E. coli identified by the microbiological department.

Table 3.

Identification of uropathogens in general practice compared to the microbiological department.

	Reference (microbiological department)
General practice	E. Coli	Enterococci	S. saprophyticus	Other single pathogen	Two pathogens	No pathogens	Total
E. coli	114	0	3	1	2	8	128
Enterococci	7	0	2	2	0	29	40
S. saprophyticus	3	0	11	0	1	4	19
Other single pathogen	12	0	0	1	0	2	15
Two pathogens	27	0	2	0	1	14	44
No pathogens	13	0	7	4	2	69	95
Total	176	0	25	8	6	126	341

Discussion

Principal findings

This study on 341 symptomatic, female patients from general practice found that GPs can identify those with significant bacteriuria with an agreement of 0.76 using chromogenic agars as POC test. We found that enterococci posed a certain challenge since they were often identified in general practice (13% of cases) but not at all in the microbiological department. This study cannot accurately determine whether enterococci were overdiagnosed in general practice or underdiagnosed in the microbiological department, but the photograph readings suggest that both could be the case.

Strengths and weaknesses of the study

The interpreters of the index test and reference test were both sufficiently blinded, the tests were performed on the same sample of urine and all patients included in analysis were investigated with both the index test and the reference test. Twenty-two patients in the trial did not have the reference performed. This should not affect accuracy measures as, according to the participating GPs, it was due to forgetfulness. Since all patients were symptomatic and the interpretation of the reference standard corresponded to current consensus, a positive reference corresponds to the definition of having UTI. Verification and interpretation procedures, therefore, had low bias. However, the GPs had access to clinical information, which the interpreter of the reference test and the photographs did not. This could partly be the cause of the low SPE, since GPs were instructed only to include patients where UTI was suspected, leading to overestimation of UTI in general practice. All results were included in the analysis. We handled ambiguous results as negative in both the index test and in the reference test.

The study was conducted in the daily practice setting and GPs were obliged and motivated to screen all patients for eligibility. We had decided to include elderly patients, patients with uncomplicated diabetes and recurrent UTI to improve the applicability of our results. However, the inclusion period was quite long and practices were not active in recruiting at all times, which compromised obtaining a consecutive sample. Because our data came from a randomized controlled trial, the design was quite time-consuming; patients had to wait 1 day for the POC culture result before treatment could be initiated, causing some to refuse to participate. We do not know if they differed from included patients, but the mean age of patients refusing to participate was similar to those who did participate (52.0 vs. 48.5 years) We only included patients with symptoms and a positive dipstick result, since most patients with a negative dipstick result do not have UTI; however, we do not know anything about the performance of the tests for the group with a negative dipstick but strong symptoms, where urine culture could still be indicated. This could introduce spectrum bias if our results were applied to a population who were not screened with urine dipstick and therefore possibly had a lower prevalence of UTI [25]. The index and reference tests were performed as in daily clinical practice and threshold values were predefined. However, the reference has been shown to have limitations. The perfect reference test would have involved quantifying the bacteria of urine by means of serial dilution for every sample included in the study in general practice [26]. However, this is not feasible, and sending urine to the microbiological department is the ‘gold standard’ in daily practice. Also, since prevalence of UTI in this study was intermediate, reference standard misclassification would probably be low making our findings valid despite an imperfect reference standard [27]. All practices in the study had prior experience with performing POC culture and most were already using the Flexicult™ SSI-Urinary Kit on a daily basis. None of them had experience using the ID Flexicult™. However, The ID Flexicult™ did not exhibit a lower agreement than the Flexicult™ SSI-Urinary Kit, which suggests that our results could be applied to GPs with little prior experience in using any of the tests.

The photographs proved particularly advantageous in investigating the causes of low agreement. Without access to the photographs, all the wrong diagnoses would have been attributed to incorrect interpretation of the test results in general practice, but only 41% could be explained this way according to the photographs.

Findings in relation to other studies

Agreement in this study was higher than most studies investigating urine dipstick in symptomatic patients, but comparable to a recent study diagnosing UTI with a combination of dipstick and microscopy [11,28]. The field trial validation study of Flexicult™ SSI-Urinary Kit [17] does not report overall agreement, but reports discrepancies in quantification between index and reference as 16% before adjustment for various factors. That study does not report ambiguous results or the overall prevalence of UTI, and its results are therefore difficult to compare with our results; however, problems with their reference samples being transported in boric acid were also reported. One study on Uricult® and Sensicult® dipslides have shown higher SEN and SPE for symptomatic patients than we found [16]. The Uricult® study results were obtained from a single health center, which would be expected to yield better results than our multi-center study. They also reported data from a multi-center study, but ambiguous cultures were excluded from analysis. Another study on dipslide found a lower SEN and a higher SPE than this study [15]. The study included symptomatic patients and the prevalence of UTI was comparable to its prevalence in this study. However, they used a dipslide identical to the index test sent to the laboratory as reference. This would partly explain the higher SPE, but not the lower SEN. They suggest themselves that not all their samples were incubated at 37 °C. Previous research has shown that enterococci are at risk of multiplying when transported in boric acid containers [19,20]. This contradicts our finding as enterococci were often diagnosed in general practice but not by the microbiological department. Enterococci, identified in general practice and on the photograph, could possibly have been eliminated in the boric acid due to too little urine in the container resulting in too high a concentration of boric acid.

Meaning of the study and implications for practice

This study is one of the first to investigate the accuracy of chromogenic agars in general practice. We found SEN to be acceptable, but SPE was too low, since studies on dipstick have produced similar specificities, and a combination of symptoms and dipstick has even greater SPE [11]. However, overall agreement was higher than in most studies evaluating urine dipstick. Previous studies have shown that training can raise GPs’ accuracy in evaluating microbiological tests, and evaluation of the photographs suggests that this could also apply here [29]. It remains to be investigated whether improved microbiological diagnosis using chromogenic agars increases appropriate antibiotic prescribing. Two ongoing studies are investigating this with slightly different designs [22,30]. With the accuracy identified in this study, GPs would be expected to improve their antibiotic prescribing for UTI by performing a POC culture in addition to dipstick analysis before starting antibiotic treatment.