Prospective, observational study comparing automated and visual point-of-care urinalysis in general practice.

OBJECTIVE: Point-of-care testing (POCT) urinalysis might reduce errors in (subjective) reading, registration and communication of test results, and might also improve diagnostic outcome and optimise patient management. Evidence is lacking. In the present study, we have studied the analytical performance of automated urinalysis and visual urinalysis compared with a reference standard in routine general practice.
SETTING: The study was performed in six general practitioner (GP) group practices in the Netherlands. Automated urinalysis was compared with visual urinalysis in these practices. Reference testing was performed in a primary care laboratory (Saltro, Utrecht, The Netherlands). PRIMARY AND SECONDARY OUTCOME MEASURES: Analytical performance of automated and visual urinalysis compared with the reference laboratory method was the primary outcome measure, analysed by calculating sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) and Cohen's κ coefficient for agreement. Secondary outcome measure was the user-friendliness of the POCT analyser.
RESULTS: Automated urinalysis by experienced and routinely trained practice assistants in general practice performs as good as visual urinalysis for nitrite, leucocytes and erythrocytes. Agreement for nitrite is high for automated and visual urinalysis. κ's are 0.824 and 0.803 (ranked as very good and good, respectively). Agreement with the central laboratory reference standard for automated and visual urinalysis for leucocytes is rather poor (0.256 for POCT and 0.197 for visual, respectively, ranked as fair and poor). κ's for erythrocytes are higher: 0.517 (automated) and 0.416 (visual), both ranked as moderate. The Urisys 1100 analyser was easy to use and considered to be not prone to flaws.
CONCLUSIONS: Automated urinalysis performed as good as traditional visual urinalysis on reading of nitrite, leucocytes and erythrocytes in routine general practice. Implementation of automated urinalysis in general practice is justified as automation is expected to reduce human errors in patient identification and transcribing of results. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/

This is the first study that has compared the diagnostic performance of automated and visual urinalysis in routine general practice.

Automated urinalysis by experienced and routinely trained practice assistants in general practice performs as good as visual urinalysis for nitrite, leucocytes and erythrocytes.

Agreement of automated and visual urinalysis for leucocytes is rather poor when compared with the central laboratory reference standard.

Follow-up studies using clinical end points are needed to study the value of automated urinalysis on patient outcome.

Implementation of automated urinalysis in general practice is justified as automation is expected to reduce human errors in patient identification and transcribing of results.

Introduction

Urine examination is most commonly used for the diagnosis of urinary tract infection (UTI). It is requested in 2% of all primary care consultations,1 2 resulting in 1 320 000 urine tests per year in the Netherlands.3

For decades, urine test strips have been used for this examination. In most cases, the coloured pads of these test strips are read visually after being dipped in urine and compared with a reference colour chart. Studies have indicated that this subjective evaluation is only moderately accurate compared with the quantitative evaluation of urine with culture as a reference standard. In a systemic review the sensitivity of the nitrite dipstick test was 45–60% and the specificity 85–98%. Sensitivity of the urine dipstick test for leucocytes was slightly higher than for the nitrite test (48–86%), while the specificity was slightly lower (17–93%).2 Therefore, the recommendation in most primary care clinical practice guideline4 5 is, for otherwise healthy women with a strong suspicion of UTI, to perform a (semi)quantitative urine culture, or to perform urine sediment analysis if the result of the urine nitrite test strip is negative and either leucocytes or erythrocyte test is positive.

Recently, point-of-care testing (POCT) urine analysers have been introduced in primary care. Automated urinalysis might reduce errors in (subjective) reading, registration and communication of test results, and might therefore also improve diagnostic outcome and optimise patient management. So far, analytical performance of POCT for urine analysis in primary care has not been evaluated. In a previous study, we compared the analytical performance of six commercially available POCT urine analysers with the laboratory standard for the diagnosis of UTI.6 The overall performance of these urine analysers for nitrite, leucocytes and erythrocytes compared with the two reference methods in the laboratory was adequate and justifies the routine use of POCT urine analysers in primary care. The procedure of automatic reading through POCT is less susceptible to erroneous labelling and registration of test results in the medical record of the patient compared with manual handling and could therefore enhance patient safety.7

In the present study, we report the analytical validity and the user friendliness of automated urinalysis after implementation in routine general practice. Our prerequisite was that automated urinalysis can be broadly implemented in general practice if the results would be at least as good as routine urinalysis by visual reading.

Methods

Setting

The study was performed in six general practitioner (GP) practices in the Utrecht area in the Netherlands. Our aim was to collect 200 urine samples. We recruited 6 practices (1 single, 5 multiple partner practices) so that each could collect ∼40 urine samples within 2–4 weeks. The POCT urine analysers were validated at Saltro Diagnostic Centre, the leading primary care laboratory in the central region of the Netherlands, before installation at the GP practice by a POCT laboratory technician. This technician also trained the GP practice assistants, who routinely perform urine testing in the Netherlands, on use and technical aspects of the urine POCT device. For study purposes, automated and visual urinalysis for each urine sample would be performed by the same practice assistant. In total, 25 practice assistants were involved in the study. In the Netherlands, each practice employs 2–3 practice assistants whose routine tasks include in-office urinalysis test performance.

Testing procedure

This prospective study was performed with routine care urine samples from patients with suspected UTI (no selection was made on age, gender or otherwise). The study procedure was as follows: urine of patient with suspected UTI was tested by test strip visual reading by the practice assistant, following routine practice procedure, and results were documented. Indications for urine testing followed routine care (guideline on UTI, Dutch College of General Practitioners).4 Subsequently, urine strips were photographed, including date and time. Next, urine was tested with the POCT urine analyser by the practice assistant. Test results were printed with date and time to proof that visual reading was done before automated urinalysis. Samplewise data were checked to see if the visual readings were indeed blinded to the automated results (test time for visual urinalysis should be before the test time for automated urinalysis). This was the case in almost all of the readings. Left-over urine was transferred to a Stabilur urine tube and sent to the laboratory for reference testing on the Urisys 2400 within 8 hours.

Reference testing was done by Saltro Diagnostic Centre. The laboratory assistants were blinded to the test results gained in the practices. Saltro routinely uses Stabilur to preserve urine samples for particle counting. These tubes are also acceptable for test strip analysis within 8 hours on the day of urine collection.8 9

Urine analysers and tests

We used the Urisys 1100 (Roche Diagnostics, Risch-Rotkreuz, Switzerland) as POCT device. The Urisys 1100 uses reflectance photometry to semiquantitatively measure specific gravity, pH, leucocytes, nitrite, protein, glucose, ketone, urobilinogen, bilirubin and erythrocytes. This analyser was tested previously,6 together with five other devices, and found suitable for use in general practice. Its quality and user-friendliness was judged sufficiently for use in general practice; GP practice assistants and midwives reported the analysers as easy to use, while their susceptibility to flaws was considered low.6 For this study, the Combur 10-test strips (Roche) were used. Although these strips contain a total of 10 tests, we have focused this study on nitrite, leucocyte and erythrocyte tests.

For POCT urinalysis on the Urisys 1100, a test strip is dipped into urine. After excess urine is removed, the strip needs to be immediately placed in the analyser. After pressing the start button, the analysis starts. Results will be shown on screen or printed after 1 min.

For reference testing in the laboratory, we used the Urisys 2400 automated urine analyser (Roche Diagnostics, Risch-Rotkreuz, Switzerland). This analyser is routinely used for all (primary care) samples at Saltro Diagnostic Centre and measures, also using reflectance photometry, pH, leucocytes, nitrite, protein, glucose, ketones, urobilinogen, bilirubin, blood (erythrocytes/haemoglobin), clarity and specific gravity. The Urisys 2400 uses test strips from the Urisys 2400 cassette.

In this study, urinalysis by visual reading was performed with routinely used test strips. Three out of six GP practices participating in this study were routinely using Combur 7-test strip (Roche). Two practices were using Multistix from Siemens, one practice was using Multistix 8SG variant and the other was using the Multistix 5 variant. One practice was using Medi-Test URYXXON stick 10 (Macherey-Nagel). Four out of six practices routinely performed visual reading, whereas two practices were already routinely using automated urinalysis. However, for the study period they used the Urisys 1100 instead of their own POCT analyser.

User-friendliness

User-friendliness of automated urinalysis was assessed using a standardised user-friendliness questionnaire10 consisting of five questions concerning the user-friendliness of the analyser, test procedure and the susceptibility of flaws. Subsequently, practice assistants were asked if a POCT urine analyser was deemed useful in their daily practice and if they thought that using such an analyser would improve their productivity and efficiency, and whether the analyser would make urine analysis more precise. In total, 19 practice assistants completed the two questionnaires.

Statistical analysis

Analytical performance of automated and visual urinalysis was calculated and compared with that of the reference laboratory method. Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) were calculated for each method and compared with the reference method. Agreement between the automated urinalysis and the reference method was analysed by calculation of the Cohen's κ coefficient and its 95% CI. This was repeated for the visual urinalysis. Agreement was ranked as very good: κ=0.81–1.00; good: κ=0.61–0.80; moderate: κ =0.41–0.60; fair: κ=0.21–0.40 and poor: κ<0.20.11 κ's of automated urinalysis versus reference method, and visual urinalysis versus reference method were compared (in absolute numbers).

Results

Agreement with laboratory reference standard

In total, 234 urine samples from six GP practices (range 30–44 urine samples per practice) were analysed by 25 GP practice assistants. One of these practices did not perform the automated method for 6 of their 44 samples. Table 1 shows the cross-tables, sensitivity, specificity, PPV, NPV and κ between automated urinalysis and the laboratory reference test (Urisys 2400), and between visual urinalysis and the laboratory reference test (Urisys 2400).

Table 1

Sensitivity, specificity, PPV, NPV and κ for automated urinalysis versus the laboratory reference test (Urisys 2400), as well as for visual urinalysis versus the laboratory reference method

		Urisys 2400		Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)	κ (0.95 CI)
Visual
Nitrite		Positive	Negative
N=234	Positive	44	14	96% (85% to 99%)	93% (88% to 96%)	76% (63% to 86%)	99% (96% to 100%)	0.803 (0.710 to 0.896)
	Negative	2	174
Automated
Nitrite
N=227	Positive	40	9	91% (78% to 97%)	95% (91% to 98%)	82% (68% to 91%)	98% (94% to 99%)	0.824 (0.732 to 0.917)
	Negative	4	174
Visual
Leucocytes		Positive	Negative
N=234	Positive	125	5	61% (54% to 67%)	82% (63% to 94%)	96% (91% to 99%)	22% (15% to 31%)	0.197 (0.062 to 0.332)
	Negative	81	23
Automated
Leucocytes
N=227	Positive	130	4	65% (58% to 72%)	86% (67% to 96%)	97% (93% to 99%)	26% (17% to 36%)	0.256 (0.115 to 0.396)
	Negative	69	24
Visual
Erythrocytes		Positive	Negative
N=230	Positive	79	8	56% (47% to 64%)	91% (83% to 96%)	91% (83% to 96%)	56% (47% to 64%)	0.416 (0.303 to 0.529)
	Negative	63	80
Automated
Erythrocytes
N=229	Positive	91	8	65% (56% to 73%)	91% (83% to 96%)	92% (85% to 96%)	62% (53% to 71%)	0.517 (0.408 to 0.626)
	Negative	49	81

NPV, negative predictive value; PPV, positive predictive value.

Nitrite is detected similarly using the automated urinalysis compared with the visual urinalysis. Agreement for nitrite is high. κ for the automated urinalysis is 0.824, which is ranked as very good, whereas the κ for the visual urinalysis is 0.803, which is ranked as good.

The κ's for the leucocyte test are relatively low: 0.256 for automated urinalysis (ranked as fair) and 0.197 for visual urinalysis (ranked as poor). κ's for erythrocytes are higher: 0.517 (automated) and 0.416 (visual), both ranked as moderate.

The differences between automated and visual urinalysis in sensitivity, specificity, PPV and NPV are small. In general, the scores for automated urinalysis are slightly higher. However, the differences lie within the 95% CI and are therefore not significant.

The results of the two practices that were previously using a POC testing urine analyser did not differ from the overall results; in total, they performed 85 (35.8%) of the 237 samples.

User-friendliness and usability

A total of 19 practice assistants (76%) completed the questionnaire on user-friendliness and usability of the POCT urine analyser (Urisys 1100). Assistants found the POCT analyser easy to use (79%) and the results easy to read (84%). The liability of flaws during set-up/performing the analysis, and in reading the test results was considered to be low by 84% and 95% of the assistants (table 2). No one found the urine POCT analyser difficult to use or thought that the risk of flaws would increase.

Table 2

User-friendliness and usability of POCT urine analyser Urisys 1100

User-friendliness
1. Ease of performance:	Easy/moderate/difficult (%)
Use of analyser	79/21/0
Reading test result	84/16/0
2. Liability of flaws in procedure:	Small/moderate/large (%)
Set-up for analysis	84/16/0
Performing analysis	84/16/0
Reading test result	95/11/0
Usability
The POCT urine analyser improves my work …	Totally agree/agree/neutral/disagree/totally disagree (%)
Performance	11/16/37/32/0
Productivity	0/11/21/53/11
Efficiency	0/21/21/47/5
I believe that the POCT urine analyser is …
Useful for my work	11/37/32/11/5
More accurate than the current urine strip evaluation	11/26/26/32/0

POCT, point-of-care testing.

In total, 64% of assistants thought that the use of a POCT urine analyser would either improve their work performance or that it would not be altered by it (‘totally agree’ plus ‘agree’ plus ‘neutral’), 48% believed that automated urinalysis was useful for their work and 37% believed that automated urinalysis would be more accurate than the current strip evaluation, which was nearly as much as the assistants that were neutral or disagreed that it would improve accuracy. When asking about productivity and efficiency, 64% of assistants disagreed (‘disagree’ plus ‘totally disagree’) with the fact that automated urinalysis would improve productivity and 52% did not think it would improve efficiency (‘disagree’ plus ‘totally disagree’). Results are shown in table 2.

Among the general practices, there were two practices that were already using a POCT device for urinalysis. Their usability scores were distinctly different from the other practices (eg, 75% agreed or totally agreed that automated POCT would improve their performance; 100% indicate they believe that automated urinalysis is useful in their daily work).

Discussion

Summary

Automated urinalysis by experienced and routinely trained practice assistants in general practice performs as good as visual urinalysis for nitrite, leucocytes and erythrocytes if compared with the reference method. Agreement for nitrite is high for automated and visual urinalysis. κ's are 0.824 and 0.803 (ranked as very good and good, respectively). Agreement with the central laboratory reference standard for automated and visual urinalysis for leucocytes is rather poor (0.256 for automated and 0.197 for visual, respectively, ranked as fair and poor). κ's for erythrocytes are higher: 0.517 (automated) and 0.416 (visual), both ranked as moderate. The confirmation of the presence of nitrite (PPV) for automated urinalysis is as good as for visual reading. Additionally, the diagnostic value of a negative nitrite test (NPV) is as good for automated as for visual urinalysis. Automated urinalysis and visual urinalysis are equally good in confirming the presence of leucocytes and erythrocytes, and although both perform poorly in the exclusion of leucocytes and erythrocytes, the Urisys 1100 analyser was easy to use and considered to be not prone to flaws. The liability of flaws in the test procedure was scored as low. Taking together the analytical validation and the user-friendliness data, we conclude that automated urinalysis performs just as good as visual analysis of urine in general practice. Considering the expected advantages in reducing incorrect patient identification and result transcription,7 12 13 automated urinalysis can be implemented in general practice.

Strengths and limitations

This is the first study we know of that has compared the performance of automated and visual urinalysis in routine general practice. In light of this very common diagnostic procedure in general practice (over 1 320 000 tests annually in the Netherlands), the study results may directly impact on the quality of care and patient safety for patients with suspected UTI. We compared all test results with an acknowledged laboratory reference method, besides the comparison between automated urinalysis and visual urinalysis. We have previously evaluated and validated the POCT analyser, which showed that it could be used in daily practice.6

No formal power analysis was performed. Our prerequisite was that automated urine POCT could be implemented in general practice if the results show that automated urine POCT performs at least similarly to manual urine POCT. To show this, we have chosen for a pragmatic study that needed to be manageable in the current busy daily practice. We felt that by collecting 200 samples, we would be able to at least show that automated urine POCT performs as good as manual urine POCT. The relatively low number of samples could have affected the outcome, particularly the lack of (large) differences found.

This study has been performed by assistants that are trained in urinalysis. Not all countries have trained practice assistants for urinalysis and skills may differ per country. As the test procedure is not complex, visual reading is straightforward, and the POCT analyser is easy to use, we do not think that the results would be much different in other settings.

Visual inspection of urine test strips that were routinely used in the particular practices was allowed to comply with the regular practice procedures. As this may have introduced a possible extra variation in test results, this is a limitation of this study. All practices did use the same POCT urine analyser with identical test strips.

One could argue about the chosen method to measure agreement between automated and visual urinalysis. Measuring the κ between automated and visual urinalysis might look more sensible than measuring the agreement between the visual urinalysis and the reference method, as well as measuring the agreement between automated urinalysis and the reference method. Statistical experts have advised differently though, because comparison of two methods lacks ground if these are not compared with the reference standard.

Comparison with existing literature

Multiple studies, all conducted in a hospital laboratory setting, have shown that automated urinalysis, using different automated urinalysis systems (no POCT), is acceptable for the screening of urine samples, and thus avoiding unnecessary urine culture.14–17 Various studies have been performed on the performance of (now outdated) POCT urine analysers compared with laboratory reference method,18–21 but none of them were tested in general practice, and none of them compared automated urinalysis with the routine method of visual reading.

The guideline of the Dutch College of General Practitioners for UTI4 advises to perform a (semi)quantitative urine culture or to assess the urine sediment if the result of the urine nitrite test strip is negative and either leucocyte of erythrocyte test is positive. This advice is indicated for otherwise healthy women without a strong suspicion of UTI; this agrees with the flow chart of the primary care guidance document for UTI by the British Infection Association (BIA).5 This guideline also indicates, like Little et al,22 that there is no reason to apply a culture test for every urine sample. Antibiotics targeted after POCT urinalysis with a delayed prescription as backup was shown to reduce antibiotic use.22 This evidence-based information was the basis for our study. Our study shows that automated urinalysis leads to test results as accurate as with visual urinalysis.

Implications for practice

The study results legitimise the advice to use automated urinalysis for routine use in daily general practice as an alternative to visual reading of urine strips. Both automated urinalysis and visual urinalysis by skilled practice assistants (and physicians) perform well. In case of serious illness or ongoing symptoms, despite adequate management, POCT urinalysis cannot fully replace urine testing by high-standard urine analysers in a central laboratory, since the results of automated and visual reading of leucocytes and erythrocytes were fair to poor compared with the reference laboratory standard. Various guidelines worldwide indicate the importance of connective POC testing. They state that the use of connective automated POCT analysers greatly reduces the risk of incorrect patient identification, uncertified user and incorrect transcribing of results or failure to record results.7 12 13 With the knowledge that the automated device generates results as good as the routine visual urinalysis, we expect that introduction of automated urinalysis will improve the quality of urinalysis in general practice.

Preanalytic and postanalytic errors by the users of POCT are the main reason for errors in diagnostic testing.23 24 Automated urinalysis prevents most of the possible reasons of errors by the error-proof handling and automated processes, particularly when POCT devices are automatically connected to patient records and other databases. Data connectivity of diagnostic devices to patient records, and the information systems of GP practices, laboratories and hospitals to improve quality of data and communication of test results (with less repeated testing) are recommended in guidelines in the Netherlands.7 25

Overall, the good results of automated urinalysis for UTI hold promise for the analysis of other available analytes (such as ketones, glucose, protein, bilirubin and urobilinogen) that can be tested with the same urine strips. Further studies to define the clinical value of these analytes are needed as results of urinalysis in daily practice are used for UTI and for the work-up of patients with urine incontinence, micturition symptoms in men and other conditions.

The results from the questionnaire on user-friendliness suggest that the POCT urine analyser is easy to use and the liability of flaws is small. The usability questionnaire gives a less clear result; study assistants have some doubts on the time it would take to perform POCT urinalysis. Among the practices, there were two practices that already used a POC device for urinalysis. Their usability scores were distinctly different from the other practices, indicating that these users are more tolerant of its use.

Our study results show that the use of automated POCT for urine can be implemented in general practice. The analytical performance shows results similar to the routine visual urinalysis, and the user-friendliness shows that the POCT analyser is easy to use and the liability of flaws is small. Additionally, automation is expected to reduce human errors in patient identification and transcribing of results. Follow-up studies using clinical end points are needed to confirm that POCT urinalysis also improves patient outcomes and to detail the implementation of POCT urinalysis in daily general practice.