Evaluation of an Image Analysis Device (APAS) for Screening Urine Cultures.

While advancements have been made in some areas of pathology with diagnostic materials being screened using image analysis technologies, the reporting of cultures from agar plates remains a manual process. We compared the results for 2,163 urine cultures read by a reference panel of microbiologists, by the routine laboratory process, and by an automated plate reading system, APAS (LBT Innovations Ltd., South Australia). APAS detected colonies with a sensitivity of 99.1% and a specificity of 99.3% on blood agar, while on MacConkey agar, the colony detection sensitivity was 99.4% with a specificity of 99.3%. The device's ability to enumerate growth had an accuracy of 89.2%, and the morphological identification of colonies showed a high level of performance for the colony types typical of Escherichia coli and other enteric bacilli. On blood agar, lactose-fermenting colonies were morphologically identified with a sensitivity of 98.9%, while on MacConkey agar they were identified with a sensitivity of 99.2%. In this first clinical evaluation, APAS demonstrated high performance in the detection, enumeration, and colony classification of isolates compared with that for conventional plate-reading methods. The device found all cases reported by the laboratory and detected the most commonly encountered organisms found in urinary tract infections.

INTRODUCTION

Traditional culture techniques and the use of agar plates remain important tools in the identification of bacterial infections and are still in common use within the field of diagnostic microbiology. In other areas of the clinical laboratory, a combination of diagnostically effective screening tests and automation has led to improvements in throughput, precision, data handling, and patient safety (1, 2). These technologies are now assisting laboratories in meeting the many challenges being faced by diagnostic facilities around the world.

While automated specimen, plate handling, and plate imaging devices are now being introduced into the microbiology laboratory (3, 4), the reporting of cultures has remained a predominantly manual process. This contrasts with technologies now routinely applied within diagnostic devices in the fields of cytopathology and hematology (5, 6) where some prescreening using image analysis technologies has been introduced to streamline workflow and produce efficiencies.

APAS (LBT Innovations Ltd., South Australia) is an image analysis device dedicated to screening agar plates for growth. It is able to detect colonies, enumerate the various colony types present, and, with use of an interpretive algorithm, apply standard rules to assign each plate and case into convenient categories for further processing.

As infections of the urinary tract are common (7) and contribute to a significant proportion of the effort and resource use within clinical microbiology laboratories (8, 9, 10, 11, 12), the performance of APAS was assessed against the traditional manual reading method for culture plates inoculated with urine.

MATERIALS AND METHODS

Specimens.

Clean catch and catheter urine samples submitted for routine cultures to a clinical laboratory (Healthscope Pathology, Wayville, South Australia) over a 4-week period were included in the study. Referrals were sourced from community clinics and hospitals with a variety of age groups and clinical presentations represented. Specimens were collected in compliance with the laboratory's protocols, stored under laboratory controlled conditions, and tested on the day of receipt.

Instrumentation.

The APAS system used in this study consisted of a plate-handling mechanism, a lighting module, a digital camera, and analytical software. A high-quality monitor linked to the system was used for reviewing digital plate images.

From each digital image, the device estimated the total colony numbers and converted these to a CFU per milliliter value appropriate to the 1-μl inoculum used in this study. All colonies detected by the device were differentiated into one of the morphology groups listed in Table 1. Differentiation was based on the physical characteristics and color reactions associated with the agar formula, with the groupings designed as a guide for interpretation and not as a replacement for formal identification procedures.

TABLE 1

Colony identification performance by APAS compared with that of a reference panel

Colony morphologies on blood agar	Examples of colony morphology	Sensitivity (%)	Specificity (%)
Coliform-like colonies	Escherichia coli	98.9	83.9
Swarming colonies	Proteus mirabilis	97.2	99.9
Granular Gram-negative colonies	Pseudomonas aeruginosa	67.7	92.5
Staphylococcus-like colonies	Staphylococcus spp.	94	83.8
Small beta-hemolytic colonies	Streptococcus agalactiae	92.4	89.3
Small colonies	Enterococci, lactobacilli, corynebacteria	90	73.7
Colony morphologies on MacConkey agar
Lactose fermenters	Escherichia coli	99.2	98.1
Non-lactose fermenters	Proteus spp.	92.6	95.9

Additional information such as operator alerts for critical specimens and positive complementary tests can be used by the device for consideration within the decision algorithm. In this study, urine leukocyte counts performed by the laboratory were uploaded to APAS from the laboratory's information system and used to screen for cases of sterile pyuria.

The image analysis and leukocyte counts were collated by the device, and an interpretive algorithm was applied using rules based on published guidelines (13, 14). Cases were then segmented by the device into one of three groups: “positive” for plates requiring further work, “negative” for plates with a low probability of requiring further work, and “review” where an on-screen assessment by a microbiologist was required before a workflow or reporting decision was made.

All results were presented for on-screen review and electronically transferred to a database. Each report included the enumeration of growth and a summary of the colony morphologies present.

Culture and processing.

Urine cultures were prepared by inoculating 1 μl of well-mixed urine onto standard 90-mm plates of trypticase soy agar with sheep blood and MacConkey agar with crystal violet (Remel, Lenexa, KS, USA). The plates were incubated aerobically for 18 h at 35°C.

Following incubation, a reference panel of three experienced microbiologists independently and outside the routine workflow recorded the amount of growth and the colony morphologies found on the plates. The growth enumeration was estimated for the 1-μl inoculum and recorded in CFU per milliliter, while the colony morphologies present on each plate were reported as described in Table 1. A fourth microbiologist presented the plates to APAS for imaging and analysis.

The urine leukocyte counts obtained by the laboratory's standard method were electronically transferred to the device for use within the decision algorithm. A pyuria flag, defined as ≥50 × 103 leukocytes/ml, was used by the device to indicate possible cases of sterile pyuria. These cases were then reported by APAS as requiring further review by a microbiologist if growth was not detected.

Results generated by the laboratory from the routine workflow for the urine samples enrolled in the study were filed for comparison with the APAS results.

Analysis.

An independent statistician (Emphron Informatics Pty Ltd., Brisbane, Australia) analyzed the results by comparing the APAS findings with the consensus results from the reference panel. Where differences in individual panel member reports were present, a majority finding was used.

Performance values for growth detection, enumeration, and the differentiation of colony morphologies using sensitivity and specificity calculations were obtained. Sensitivity was defined as the number of true positives divided by the number of true positives plus false negatives expressed as a percentage, and specificity was the number of true negatives divided by the number of true negatives plus false positives and expressed as a percentage.

APAS and the reference panel interpreted the results by the application of published guidelines (13, 14).

An additional analysis was also performed to assess the ability of APAS to detect specific uropathogens. This was done by comparing the results from the device with the final reports issued by the laboratory.

RESULTS

Study participants and specimens.

A total of 2,163 urine samples from predominantly community-based patients, including 1,466 (68%) females and 697 (32%) males were examined in the study. The ages of the patients were diverse with 206 (9.5%) ≤20 years, 461 (21.3%) from 21 to 40 years, 398 (18.4%) from 41 to 60 years, 661 (30.6%) from 61 to 80 years, and 437 (20.2%) ≥80 years. Of the urine samples cultured, 2,156 (99.7%) were collected by the clean catch method and the remaining 7 (0.3%) by catheter.

Growth detection.

Colonies were detected by APAS on 1,603/1,618 blood agar plates reported by the reference panel as having growth and on 866/871 of the MacConkey agar plates. For blood agar, this resulted in a growth detection sensitivity of 99.1% and a specificity of 99.3%, while the performance for MacConkey agar showed a growth detection sensitivity of 99.4% and a specificity of 99.3%.

Differences between the device's ability to detect colonies at various growth levels were noted. At 105 CFU/ml, the colony detection sensitivity for blood agar was 100%, while at 104 CFU/ml and 103 CFU/ml, the sensitivities were 99.6% and 96.8%, respectively. For MacConkey agar, the differences were less marked, with detection sensitivities of 100% at 105 CFU/ml, 99.4% at 104 CFU/ml, and 97.9% at 103 CFU/ml.

On 15 blood agar plates where growth was reported by the reference panel but not by APAS, small numbers of colonies of ≤0.4-mm diameter at 103 and 104 CFU/ml were observed following technologist review of the images.

Growth enumeration.

The reference panel determined 450 cases with no growth, 413 with 103 CFU/ml, 492 with 104 CFU/ml, and 728 with 105 CFU/ml. For the purposes of comparison, 80 of the 2,163 cases were removed because APAS was unable to enumerate growth in the presence of moderate to high numbers of swarming colonies such as Proteus mirabilis. In such instances, these cases were referred by the device for review and definitive enumeration and assessment by a microbiologist.

APAS produced the correct enumeration in 1,859/2,083 (89.2%) of the remaining cases with differences in agreement noted at the various growth levels: 96.7% at 105 CFU/ml, 77% at 104 CFU/ml, 88.6% at 103 CFU/ml, and 91.1% at 0 CFU/ml.

Of the 224 enumeration discrepancies, APAS determined that 138 were greater than and 86 were less than the values reported by the reference panel, and 204 were within 1 log of the reference panel's consensus count.

Colony morphologies.

A number of different colony morphologies were identified by APAS, and the performance for each is listed in Table 1. For key urinary tract infection (UTI) pathogens on blood agar, coliform-like and swarming colonies (e.g., Proteus mirabilis) were identified with sensitivities of 98.9% and 97.2% and specificities of 83.9% and 99.9%, respectively. On MacConkey agar, lactose fermenters were identified with a sensitivity of 99.2% and a specificity of 98.1%.

Examples of the ability to differentiate colonies can be seen in the images found in Fig. 1 to 4.

FIG 1

Image of a blood agar plate following inoculation with urine and incubation for 18 h at 35°C that shows mixed growth of a Gram-negative bacillus and Gram-positive coccus.

FIG 4

APAS computer interpretation of mixed growth of lactose-fermenting colonies (red) and non-lactose-fermenting colonies (black) from the MacConkey agar plate shown in Fig. 3.

APAS computer interpretation of mixed growth of a Gram-negative bacillus (red) and Gram-positive coccus (black) from the blood agar plate shown in Fig. 1.

FIG 3

Image of a MacConkey agar plate following inoculation with urine and incubation for 18 h at 35°C that shows mixed growth of lactose- and non-lactose-fermenting colonies.

Segmentation of cases.

All 509 cases reported by the laboratory with significant growth were segmented as either positive or review by APAS and produced a segmentation sensitivity of 100%. A further 818 cases segmented as positive or review were reported by the laboratory as not having significant growth and contributed to the specificity of 66.9%.

If the leukocyte counts had not been included in the interpretive algorithm, the device would have found 506/509 cases and produced a segmentation sensitivity of 99.4%.

Organisms reported by the laboratory.

Growth of potentially significant organisms were reported by the laboratory from 509/2,163 (23.5%) specimens submitted. The most common pathogens were Escherichia coli (n = 341), Enterococcus faecalis (n = 38), Klebsiella pneumonia (n = 21), Proteus mirabilis (n = 19), and Pseudomonas aeruginosa (n = 19) (Table 2). Less frequently encountered isolates, including 3 cases of infection due to viridans streptococci, 3 cases of Candida spp. (including Candida albicans), and 1 case of Aerococcus urinae, were also reported.

TABLE 2

Organisms detected by APAS compared with those by the routine laboratory reports

Organism	No. of cases detected by APAS	No. of cases reported by the laboratory
Escherichia coli	339	341
Enterococcus faecalis	38	38
Klebsiella pneumoniae	21	21
Proteus mirabilis	19	19
Pseudomonas aeruginosa	18	19
Staphylococcus saprophyticus	14	14
Klebsiella oxytoca	8	8
Staphylococcus epidermidis	7	7
Streptococcus agalactiae	6	6
Enterobacter aerogenes	5	5
Citrobacter koseri	5	5
Enterobacter cloacae complex	3	3
Morganella morganii	3	3
Viridans streptococci	3	3
Candida albicans	2	2
Citrobacter freundii	2	2
Staphylococcus, coagulase negative	2	2
Acinetobacter spp.	1	1
Aerococcus urinae	1	1
Candida spp.	1	1
Enterococcus faecium	1	1
Raoultella spp.	1	1
Serratia liquefaciens	1	1
Serratia ureilytica	1	1
Staphylococcus aureus	1	1
Staphylococcus haemolyticus	1	1
Staphylococcus hominis	1	1
Streptococcus dysgalactiae	1	1
Total	506	509

Cases referred for review.

During the study, APAS referred 374/2,163 (17.3%) cases for onscreen review by a microbiologist. Of these, 256 cases were due to the presence of low numbers of coliform-like or lactose-fermenting colonies and a further 80 cases were referred for review because the device identified the presence of swarming colonies on blood agar. The remaining cases were referred for review because of the possible presence of a mixture of potential pathogens at a concentration of 104 CFU/ml.

DISCUSSION

Two measures were used to define the performance of this novel device during the study. The reference panel provided a consensus result for the enumeration and description of growth, while the final reports issued by the laboratory allowed comparisons to be made to ensure that APAS identified all significant instances of growth.

The organisms isolated during the study were representative of those found in urinary tract infections from community and hospitalized patients (15, 16, 17, 18). In this study, E. coli, Enterococcus faecalis, Klebsiella pneumonia, Proteus mirabilis, Pseudomonas aeruginosa, and Staphylococcus saprophyticus were the most commonly encountered organisms and represented 88.5% of the total isolates reported.

Growth was found by the reference panel and not by APAS on 15/1,618 blood agar and 5/871 MacConkey agar plates. On review, the colonies not captured by APAS on blood agar were seen on the images as having diameters of ≤0.4 mm and were present in concentrations of 103 and 104 CFU/ml. They were considered to be normal skin and urogenital organisms such as lactobacilli and corynebacteria. Colonies missed on MacConkey agar were small, partially suppressed nonfermenters.

This finding led to the conclusion that some small colonies and perhaps the early growth stages of slow-growing pathogens such as Aerococcus urinae and Corynebacterium urealyticum might be missed by APAS if cultures are only incubated for 18 h on the media used in this study. The consequence of this observation is that cases of complicated urinary tract infection, where slow-growing organisms may be expected, will need to be carefully managed by following established guidelines, including the inoculation of higher specimen volumes and additional culture media as well as the extension of incubation times (13).

The identification of colony morphologies showed a high level of performance. For morphologies representing E. coli and the other enteric bacilli on blood agar, an identification sensitivity of 98.9% was obtained, while on MacConkey agar, lactose-fermenting colonies were identified with a sensitivity of 99.2%. The colony morphology classed as “granular Gram-negative colonies” produced the lowest performance with a sensitivity of 67.7%. This category of colonies represented the classic large, rough colony type of Pseudomonas aeruginosa. When these colonies were assessed by APAS, they were generally categorized as coliform-like colonies, and the cases were redirected for a microbiologist's review. The result was that this type of misclassification did not compromise the overall interpretation and patient safety. This assigned morphology discrepancy is not dissimilar to the general practices of trained microbiologists, who may at times make subjective colony morphology classifications before full identification procedures are initiated.

A total of 818 cases found to be positive by APAS were reported by the laboratory as having either contaminants or normal urogenital flora present where typically greater than three organisms were present. This false-positive rate was responsible for the low specificity of 66.9%. However, as each of these cases was referred by APAS to a microbiologist for careful review in line with the laboratory's interpretive criteria, patient safety was not compromised. Thus, as a screening device, where sensitivity is more important than specificity (19), the segmentation of these cases by APAS was favorable and as such was considered acceptable within the general workflow of a modern laboratory.

The urine leukocyte count was used within the APAS decision algorithm for each specimen within the study. This optional feature identified three cases with apparently sterile pyuria. Further assessment of these cases by the laboratory showed that clinically relevant growth was detected on cultures inoculated with 10 μl urine as opposed to the 1 μl used in the study. Associating leukocyte counts with the culture report can enhance the diagnostic value of urine cultures as the finding of pyuria may indicate the presence of low numbers of uropathogens, slow-growing organisms, or more complex urinary tract pathology (13, 20, 21).

The recent introduction of plate-imaging systems for reading cultures offers a number of quality improvements and an image archive facility for diagnostic and teaching purposes (3, 4). Progressing from these imaging systems to fully automated culture plate readers will be challenging as there is a perception that machines cannot exercise the complex decision-making skills required to assess microbiological cultures (3). Additionally, little has been published regarding the accuracy and reproducibility of manual plate reading, so validating new technologies will require a better understanding of current manual reporting performances.

In conclusion, all cases of clinical infection were detected by APAS and its associated decision algorithm during the study. However, in a few cases, small colonies in low numbers were not detected, indicating the possibility that uropathogens in the early stage of growth or highly fastidious organisms may not be detected. To avoid this, specimen management and result validation strategies as described above may need to be implemented.

The introduction of automated plate assessment systems is likely to facilitate the rationalization of resources in the clinical laboratory and lead to workplace efficiencies and improved staff and patient safety through the reduction in plate handling and transcription of results. Furthermore, the screening of a large portion of samples as “negative” will allow experienced staff to use their time and skills for reviewing complex cases.

Such systems, as described here, will allow better segregation of tasks and use of skilled staff and will be a welcome addition to the routine clinical microbiology laboratory. Further studies are currently planned to assess and refine this new technology for routine use.