Evaluation of the FilmArray Blood Culture Identification Panel: Results of a Multicenter Controlled Trial.

Sepsis is a major cause of morbidity, mortality, and increased medical expense. Rapid diagnosis improves outcomes and reduces costs. The FilmArray blood culture identification panel (BioFire Diagnostics LLC, Salt Lake City, UT), a highly multiplexed PCR assay, can identify 24 etiologic agents of sepsis (8 Gram-positive, 11 Gram-negative, and 5 yeast species) and three antimicrobial resistance genes (mecA, vanA/B, and blaKPC) from positive blood culture bottles. It provides results in about 1 h with 2 min for assay setup. We present the results of an eight-center trial comparing the sensitivity and specificity of the panel with those of the laboratories' standard phenotypic identification techniques, as well as with molecular methods used to distinguish Acinetobacter baumannii from other members of the A. calcoaceticus-A. baumannii complex and to detect antimicrobial resistance genes. Testing included 2,207 positive aerobic blood culture samples, 1,568 clinical and 639 seeded. Samples were tested fresh or were frozen for later testing within 8 h after the bottles were flagged as positive by an automated blood culture system. At least one organism was detected by the panel in 1,382 (88.1%) of the positive clinical specimens. The others contained primarily off-panel organisms. The panel reported multiple organisms in 81 (5.86%) positive clinical specimens. The unresolved blood culture identification sensitivity for all target detections exceeded 96%, except for Klebsiella oxytoca (92.2%), which achieved 98.3% sensitivity after resolution of an unavoidable phenotypic error. The sensitivity and specificity for vanA/B and blaKPC were 100%; those for mecA were 98.4 and 98.3%, respectively.

INTRODUCTION

Bacteremia and sepsis constitute major health and financial burdens in the United States and internationally (1). In the United States, rates for hospitalization that included the diagnostic codes for sepsis or severe sepsis doubled between 2000 and 2008, partly because of the aging of the population (2, 3). In 2008, hospital costs for the treatment of sepsis were $14.6 billion, and the inflation-adjusted cost is increasing by almost 12% annually. Treatment of sepsis is complicated by the continuing increase in antibiotic resistance (4).

A definitive diagnosis of a bacteremic/septic episode is made from blood cultures. They consist of an aerobic and an anaerobic blood culture bottle inoculated with blood from the patient and incubated in an automated blood culture instrument. When the bottles are flagged as positive, identification (ID) and antibiotic susceptibility testing (AST) of the etiologic agents generally require 2 or more additional days. Patients who are treated rapidly with appropriate antibiotics have better outcomes, with decreased morbidity and mortality rates, hospital stay lengths, and hospital costs (5). Excess mortality attributable to inadequate antimicrobial therapy ranges from 10 to almost 40% (6, 7). Since decreasing the time to administration of appropriate antibiotics improves survival and decreases costs (6, 8–10), a goal of the clinical microbiology laboratory is to expedite organism ID and AST results. Molecular techniques play an increasing role in speeding these determinations (9, 11–13).

Between July 2012 and February 2013, eight clinical microbiology laboratories participated in a study of the BioFire FilmArray blood culture ID (BCID) system. BCID is a two-stage, highly multiplexed, nested PCR test that is carried out in a closed, disposable, single-use pouch. It requires about 2 min for assay setup and provides results in approximately 1 h. It is designed to identify simultaneously 24 etiologic agents of sepsis (eight Gram-positive, 11 Gram-negative, and five Candida species) as well as three antimicrobial resistance genes (mecA, vanA/B, and blaKPC, which encodes Klebsiella pneumoniae carbapenemase). In general, the comparator assays were the standard phenotypic organism ID methods utilized by the eight participating laboratories. However, molecular techniques were used for confirmation of Acinetobacter baumannii to the species level and detection of the antibiotic resistance genes. The results of this study were submitted to the U.S. Food and Drug Administration as part of the information required for the 2013 clearance of the system.

(These results were presented in part in 2013 at the 113th General Meeting of the American Society for Microbiology and at the 23rd European Congress of Clinical Microbiology and Infectious Diseases.)

MATERIALS AND METHODS

Study sites.

The eight clinical microbiology laboratories that participated in this study were in the following U.S. cities: New York, NY; Baltimore, MD (two sites); Galveston, TX; Chicago, IL; Detroit, MI; Salt Lake City, UT; and Los Angeles, CA.

Overall design.

The comparison had two arms. The prospective clinical arm tested the residual specimens from diagnostic patient blood cultures and compared them with the results of phenotypic analysis of the same specimens. The seeded arm tested specimens from similar bottles that had been injected with human blood (screened for pathogens such as HIV and the hepatitis B and C viruses; Bioreclamation, Westbury, NY) and a pure culture of a known microorganism. One laboratory tested only seeded cultures, and three analyzed only clinical specimens. The other four performed both types of testing. This study was approved by the Institutional Review Board at each site, and a waiver of informed consent was obtained.

All laboratories used the BD Bactec Plus Aerobic/F blood culture bottle (BD, Sparks, MD). For both the prospective and seeded arms, a positive Gram stain from the flagged blood culture bottle was required before the specimen was included in the study. BCID testing and the freezing of aliquots had to be initiated within 8 h after the bottle was flagged as positive by the automated blood culture system. This time interval was chosen for reproducibility on the basis of preliminary experiments that showed that BCID positivity was stable for at least 12 h. Specimens from both arms of the study were deidentified, coded with unique study numbers, and treated identically once they were flagged as positive by the blood culture instrument. Organisms from all positive blood culture bottles, both clinical and seeded, were subjected to standard blood culture ID and AST. The laboratory ID was generally regarded as the gold standard and as the basis for determination of results as true or false positive (TP or FP) or true or false negative (TN or FN), respectively. However, in the cases of A. baumannii and the antibiotic resistance genes, the comparator methods were molecular (see below). Tables 3, 4, and 6 show unresolved data. Resolution of discrepancies between the BCID and the comparator method is presented in Results and summarized in Table 5.

TABLE 3

Performance summary of the BCID panel versus the comparator assays for bacteria in both clinical and seeded positive blood cultures combined

Organism	Isolates detecteda: BCID/comparator		No. of results: BCID/comparator				Sensitivity or PPAb: TP/(TP + FN) (%)	95% CI	Specificity or NPAb: TN/(TN + FP) (%)	95% CI
	Clinical arm	Seeded arm	TP +/+	FP +/−	FN −/+	TN −/−
Gram-positive bacteria
Enterococcus	102/101	29/29	127	4	3	2,073	127/130 (97.7)	93.4–99.5	2,073/2,077 (99.8)	99.5–99.9
L. monocytogenes	0/0	36/36	36	0	0	2,171	36/36 (100)	90.3–100	2,171/2,171 (100)	99.8–100
Staphylococcus	780/797	2/1	770	12	28	1,397	770/798 (96.5)	95.0–97.7	1,397/1,409 (99.1)	98.5–99.6
S. aureus	257/257	0/0	253	4	4	1,946	253/257 (98.4)	96.1–99.6	1,946/1,950 (99.8)	99.5–99.9
Streptococcus	140/141	63/62	198	5	5	1,999	198/203 (97.5)	94.3–99.2	1,999/2,004 (99.8)	99.4–99.9
S. agalactiae (group B)	18/18	18/18	36	0	0	2,171	36/36 (100)	90.3–100	2,171/2,171 (100)	99.8–100
S. pneumoniae	26/25	12/12	36	2	1	2,168	36/37 (97.3)	85.8–99.9	2,168/2,170 (99.9)	99.7–100
S. pyogenes (group A)	8/7	31/31	38	1	0	2,168	38/38 (100)	90.7–100	2,168/2,169 (99.9)	99.7–100
Total	1,331/1,346	191/189	1,494	28	41	16,093	1,494/1,535 (97.3)	96.4–98.1	16,093/16,121 (99.8)	99.7–99.9
Gram-negative bacteria
A. baumannii	16/14	40/37	51	5	0	2,151	51/51 (100)	93.0–100	2,151/2,156 (99.8)	99.5–99.9
Enterobacteriaceae	307/310	187/188	490	4	8	1,705	490/498 (98.4)	96.9–99.3	1,705/1,709 (99.8)	99.4–99.9
E. cloacae complex	24/22	17/17	38	3	1	2,165	38/39 (97.4)	86.5–99.9	2,165/2,168 (99.9)	99.6–100
E. coli	149/148	6/5	150	5	3	2,049	150/153 (98.0)	94.4–99.6	2,049/2,054 (99.8)	99.4–99.9
K. oxytoca	6/6	54/58	59	1	5	2,142	59/64 (92.2)c	82.7–97.4	2,142/2,143 (99.9)	99.7–100
K. pneumoniae	74/71	37/34	102	9	3	2,093	102/105 (97.1)	91.9–99.4	2,093/2,102 (99.6)	99.2–99.8
Proteus	22/22	17/17	39	0	0	2,168	39/39 (100)	91.0–100	2,168/2,168 (100)	99.8–100
S. marcescens	22/22	55/55	76	1	1	2,129	76/77 (98.7)	93.0–100	2,129/2,130 (99.9)	99.7–100
H. influenzae	8/8	35/35	43	0	0	2,164	43/43 (100)	91.8–100	2,164/2,164 (100)	99.8–100
N. meningitidis	1/1	35/35	36	0	0	2,171	36/36 (100)	90.3–100	2,171/2,171 (100)	99.8–100
P. aeruginosa	52/52	0/0	51	1	1	2,154	51/52 (98.1)	89.7–100	2,154/2,155 (99.9)	99.7–100
Total	681/676	483/481	1135	29	22	23,091	1,135/1,157 (98.1)	97.1–98.8	23,091/23,120 (99.9)	99.8–99.9

Culture or culture and 16S rRNA gene sequencing for A. baumannii.

The data in these columns are based on the combined results from the clinical and seeded specimens. Sensitivity refers to performance with clinical specimens, while PPA refers to performance with seeded specimens. Despite the differences in names and usage, they were calculated identically. Likewise, specificity refers to performance with prospective samples, while NPA is used for seeded specimens. They were also calculated identically. These data were obtained in the initial analyses and have not been changed to reflect subsequent investigations of discordant results. Investigations of discordant results are described in Results and summarized in Table 5.

K. oxytoca is the only organism that apparently failed to meet the prespecified criterion of a sensitivity of >95%. However, this was due to an unavoidable error in phenotypic identification. When this was resolved, the sensitivity for this organism exceeded 98.3%.

TABLE 4

Performance summary of the FilmArray BCID panel versus culture for Candida spp. in positive blood cultures

Organism	Isolates detected: BCID/comparator		No. of results: BCID/comparator				Sensitivity or PPAa: TP/(TP + FN) (%)	95% CI	Specificity or NPAa: TN/(TN + FP) (%)	95% CI
	Clinical arm	Seeded arm	TP +/+	FP +/−	FN −/+	TN −/−
Candida albicans	20/16	48/48	64	4	0	2,139	64/64 (100)	94.4–100	2,139/2,143 (99.8)	99.5–99.9
Candida glabrata	14/12	37/37	49	2	0	2,156	49/49 (100)	92.7–100	2,156/2,158 (99.9)	99.7–100
Candida krusei	4/4	33/33	37	0	0	2,170	37/37 (100)	90.5–100	2,170/2,170 (100)	99.8–100
Candida parapsilosis	9/7	52/54	59	2	2	2,144	59/61 (96.7)	88.7–99.6	2,144/2,146 (99.9)	99.7–100
Candida tropicalis	3/3	36/36	39	0	0	2,168	39/39 (100)	91.0–100	2,168/2,168 (100)	99.8–100
All yeast isolates	49/42	207/208	248	8	2	10,777	248/250 (99.2)	97.1–99.9	10,777/10,785 (99.9)	99.9–100

Sensitivity and specificity refer to performance with the prospective specimens only. PPA and NPA refer to performance with the seeded specimens. These are unresolved data and do not reflect the further investigations discussed in Results.

TABLE 6

Comparison of FilmArray BCID resistance gene results to the prespecified comparator assay (PCR/sequencing directly from blood culture bottle)

Antimicrobial resistance gene(s)	Isolates detected: BCID/comparator		No. of results: BCID/comparator				Sensitivity or PPAa: TP/(TP + FN) (%)	95% CI	Specificity or NPAa: TN/(TN + FP) (%)	95% CI
	Clinical arm	Seeded arm	TP +/+	FP +/−	FN −/+	TN −/−
mecA in association with:
All Staphylococcus isolates detectedb	491/494	2/2	488	5	8	281	488/496 (98.4)	96.8–99.3	281/286 (98.3)	96.0–99.4
Staphylococcus and S. aureus isolates detected	137/139	0/0	137	0	2	118	137/139 (98.6)	94.9–99.8	118/118 (100)	96.9–100
vanA/B in association with Enterococcus isolates detected	36/36	28/28	64	0	0	67	64/64 (100)c	94.4–100	67/67 (100)	94.6–100
blaKPC in association with Enterobacteriaceae and/or A. baumannii and/or P. aeruginosa isolates detected	6/6d	33/33	39	0	0	558	39/39 (100)e	91.0–100	558/558 (100)	99.3–100

Sensitivity and specificity refer to performance with the prospective specimens only. PPA and NPA refer to performance with the seeded specimens. These are unresolved data.

Either Staphylococcus or S. aureus isolates detected or both.

Of the 64 Enterococcus isolates, 11 (17.2%) carried the vanB gene as determined by bidirectional sequence analysis.

All six isolates were identified as K. pneumoniae.

These isolates included 30 of K. pneumoniae, 2 of E. cloacae, and 1 of E. coli.

TABLE 5

Summary of outcomes of discordant organism result investigations

Result and analyte	No. of results	No. of investigationsa
		Comparator result confirmed	FilmArray result confirmed	Both results incorrectb	Inconclusive
BCID FP results
Enterococcus	4		1		3
Staphylococcus	12		11		1
S. aureus	4	0	1		3
Streptococcus	5	1	1		3
S. pneumoniae	2		1		1
S. pyogenes	1	1
A. baumannii	5	5
Enterobacteriaceae	4		1		3
E. cloacae complex	3			1	2
E. coli	5	1	2		2
K. pneumoniae	9	4		3	2
K. oxytoca	1				1
S. marcescens	1			1
P. aeruginosa	1				1
C. albicans	4				4
C. glabrata	2				2
C. parapsilosis	2				2
Total	65	12	18	5	30
% of total FP results		18	28	8	46
BCID FN results
Enterococcus	3	3
Staphylococcus	28	20	1		7
S. aureus	4	1	2		1
Streptococcus	5	3			2
S. pneumoniae	1		1
Enterobacteriaceae	8	4	3		1
E. cloacae complex	1	1
E. coli	3		3
K. pneumoniae	3	1		1	1
K. oxytoca	5	1	4
S. marcescens	1		1
P. aeruginosa	1		1
C. parapsilosis	2		2
Total	65	34	18	1	12
% of total FN results		52	28	2	18

The method used was investigation of possible sample mix-ups, reculturing from frozen aliquots of the blood culture bottle fluid, sequencing directly from the bottle if isolated organisms were expected, or sequencing from resubcultured isolated organisms if more than one isolate per bottle was present. For bacterial resolution, 16S rRNA gene sequences were analyzed and for yeast, ITS1 was analyzed. See Materials and Methods for further information. Three apparent blood culture bottle mix-ups and one apparent bottle selection error were resolved in favor of BCID.

See Results for further information regarding the indicated organism.

For the prospective clinical specimen arm, each blood culture came from a patient who had no previous specimen enrolled in the study. At the time of specimen enrollment, the following information was linked to the unique ID number assigned to the specimen: age and sex of the patient, type of blood collection (peripheral vein versus intravascular device), and time to positivity (incubation time until the bottle was flagged positive). The results were recorded in a fashion that allowed the uniquely numbered BCID results to be linked to the results from comparator phenotypic testing by a coordinator who was blind to the testing results. BCID analytic failures were also recorded to allow determination of the reliability of the system.

The seeded blood culture arm was used to ensure that organisms and resistance genes were analyzed in sufficient numbers to allow statistically valid determinations. Since it was not known prior to completion of the study how many total cultures or how many samples of each organism would be included in the prospective clinical arm, some organisms were seeded in excess of the minimum needed.

Seeded blood culture bottles were prepared by aseptically injecting them with 10 ml of human blood (Bioreclamation, Westbury, NY). The bottles were then inoculated with a pure culture of a known microorganism. The organism was prepared by suspension in normal saline to a turbidity comparable to a 0.5 McFarland standard (approximately 1.5 × 108 CFU/ml of bacteria or 1.5 × 106 CFU/ml of the Candida spp.). The suspension was diluted to approximately 1.5 × 102 CFU/ml, and the blood culture bottle was inoculated with 10 to 300 μl of the sample. The inoculum was adjusted on the basis of preliminary time-to-positivity data provided by BioFire to investigators in an attempt to have the bottles become positive following overnight incubation. Positive bottles were analyzed as described above. Seeded cultures determined phenotypically to be contaminated were eliminated from the analysis.

FilmArray BCID testing.

FilmArray BCID testing was performed as instructed by the manufacturer. Briefly, hydration solution and a sample from the blood culture bottle (100 μl) mixed with sample buffer were loaded into the BCID pouch and the pouch was placed in the FilmArray instrument. The instrument automatically performed nucleic acid extraction, highly multiplexed nested PCR, and product melt analysis with results reported in 1 h. BCID results for a specimen were displayed only if the two internal pouch controls for the run were valid. For general descriptions of the FilmArray system and of the methodologies used to develop the primers and probes for BCID and other FilmArray pouches, see references 14 and 15. All target sequences are longer than 200 bp and are proprietary, developed by BioFire for the FilmArray system. rRNA was not used as a target in the BCID assay.

BCID targets included the following genera or families of bacteria: Enterococcus, Staphylococcus (two targets), Streptococcus, and members of the family Enterobacteriaceae. There were also assays for organisms included in the above groups. Among the staphylococci was Staphylococcus aureus; among the streptococci were Streptococcus agalactiae, S. pneumoniae, and S. pyogenes, and the members of the family Enterobacteriaceae included the Enterobacter cloacae complex, Escherichia coli, Klebsiella oxytoca, K. pneumoniae, Proteus species, and Serratia species. Other targets for bacteria not included in the above groups included A. baumannii, Haemophilus influenzae (two targets) and Pseudomonas aeruginosa. Assays for five different Candida spp. (Candida albicans, C. glabrata, C. krusei, C. parapsilosis, and C. tropicalis) were also included.

If any of the three Staphylococcus assays (two genus assays or the S. aureus assay) was positive, the culture was identified as containing a member of the genus Staphylococcus. If the S. aureus assay was positive, S. aureus was also reported as detected, and the presence/absence of other staphylococci could not be determined. If S. aureus was reported as not detected by BCID, a Staphylococcus species other than S. aureus was assumed. Similarly, if BCID reported that either a Streptococcus species or a member of the family Enterobacteriaceae was detected and the sample was also reported as positive for a specific organism in that genus or family, the specific organism was presumed to be the one responsible for the positive genus/family BCID result. If a positive result for a specific member of the genus Streptococcus or the family Enterobacteriaceae was not reported, the analyte was presumed to be a Streptococcus species or a members of the family Enterobacteriaceae for which a specific assay was not included in the panel. Therefore, as with the staphylococci, the presence of a Streptococcus species or of a member of the family Enterobacteriaceae that did not have a specific assay in the BCID panel could not be ruled either in or out if a related organism with a specific assay was present.

Bacterial and fungal comparator methods.

For organism ID, with the exception of A. baumannii, the reference methods used to evaluate BCID performance in both the prospective and seeded arms of the study were the standard phenotypic ID procedures used in each laboratory. Automated methods included MicroScan Walk-Away (Siemens Medical Solutions, Deerfield, IL), Vitek 2 (bioMérieux, Durham, NC), and Phoenix (BD Diagnostics, Sparks, MD). Testing was performed according to the manufacturers' instructions. Manual techniques were performed according to the individual laboratory protocols, which conformed to the standards of the Clinical and Laboratory Standards Institute (16).

Phenotypic methods do not separate A. baumannii from other members of the A. calcoaceticus-A. baumannii complex (17), while BCID specifically targets A. baumannii. Therefore, the gold standard for the comparator identification of A. baumannii was a two-step process in which the phenotypic ID of a complex member was followed by bidirectional 16S rRNA gene testing to confirm its identity as A. baumannii sensu stricto. The 16S rRNA gene PCR was performed by research associates at BioFire who were blind to the original BCID and AST results. Amplicon cleanup and bidirectional sequencing were performed by Macrogen USA, Rockville, MD.

Antibiotic resistance gene detection methods.

The BCID resistance gene targets are within the sequences of the resistance genes, are at least 200 bp in length, and are proprietary. Antibiotic resistance gene results are reported as detected by BCID only if an organism known to carry that gene is also detected: Staphylococcus spp. for mecA; Enterococcus spp. for vanA/B; and A. baumannii, P. aeruginosa, or a member of the family Enterobacteriaceae for blaKPC. The comparator methods for these resistance genes were molecular rather than phenotypic and consisted of PCR assays performed with aliquots from the frozen blood culture bottles and targeting a sequence of the resistance gene distinct from that of the BCID target. The comparator PCR targets were developed by BioFire in conjunction with the design of their BCID targets; they were different from and did not overlap those of the BCID assays. PCR was performed by research associates at BioFire who were blind to the original BCID and AST results. Amplicon cleanup and bidirectional sequencing were performed by Macrogen USA, Rockville, MD.

Comparison of BCID resistance gene determinations with the results of AST of the isolated organisms by the phenotypic techniques is presented elsewhere (18).

Investigation of discrepancies.

Each discrepant result occurring between BCID and the comparator methods was investigated at BioFire by technologists who were blind to the results of the original assays. They applied one or more of the following strategies to frozen aliquots of the original blood culture specimens: direct molecular testing of the aliquot or subculturing and molecular testing of an organism grown from the aliquot. The molecular methods used included repeat BCID and/or sequencing of the 16S rRNA gene for bacteria or of the ITS1 region for yeast. Resolution of discrepant determinations from polymicrobial cultures by molecular techniques required isolation of the organisms prior to 16S rRNA gene or ITS1 analysis. Human error leading to a possible sample mix-up was also considered in the investigation of discrepant results. A sample mix-up involving two bottles, each containing one organism, would lead to two FP and two FN results.

Statistics.

Sensitivity and specificity were determined from standard two-by-two performance tables. Briefly, sensitivity (or positive percent agreement [PPA]) = 100TP/(TP + FN), while specificity (or negative percent agreement [NPA]) = 100TN/(TN + FP). Sensitivity and specificity are applicable to the prospective clinical cultures, and PPA and NPA are applicable to the seeded cultures. Both terms are included in the tables. However, for simplicity, only sensitivity and specificity are used in the text. The binomial two-sided 95% confidence intervals were calculated for both sensitivity and specificity by using the exact method of Clopper and Pearson (19). The number of patients enrolled in the prospective clinical study ensured that organisms with a prevalence rate of >2% could achieve statistical significance. For rare analytes, the seeded specimens ensured an adequate number of samples for analysis. The statistical analysis was conducted separately for each subgroup of each organism or resistance gene (clinical fresh, clinical frozen, overall clinical, seeded fresh, seeded frozen, and overall frozen) and for the combined total. Since no significant difference between the categories of each analyte was noted, they were combined in the performance summary tables (see Tables 3, 4, and 6).

RESULTS

Overview.

This study included 2,207 samples, 1,568 clinical and 639 seeded. Specimen testing by BCID was either initiated within 8 h after the bottles turned positive (821 or 52% of the patient specimens and 419 or 66% of the seeded cultures) or performed with aliquots of the fluid from the bottles that had been frozen within the 8-h time window (the remainder). Of the total number of samples tested by BCID, 41 (1.9%) needed repeat testing: 1 was aborted by the operator, 5 had instrument or software errors, internal control failure occurred in 16, and pouch hydration failures occurred in 19. All were successfully retested within the 8-h limit.

Table 1 shows all of the BCID targets and the number and percentage of each detected by BCID in the prospective clinical specimens from pediatric and adult patients. More than one organism was reported by BCID in 81 (5.2%) of the cultures, including two organisms in 74 cultures, three organisms in five cultures, and four organisms in two cultures.

TABLE 1

FilmArray BCID panel targets and prevalence of positive results by age group for the prospective clinical study arm

FilmArray BCID resulta	No. (%) of patients
	Totalc	Pediatricd	Adulte
Gram-positive bacteria
Enterococcus	102 (7)	5 (3)	97 (7)
Listeria monocytogenes	0 (0)	0 (0)	0 (0)
Staphylococcus	780 (50)	74 (50)	706 (50)
S. aureus	257 (16)	25 (17)	232 (16)
Streptococcus	140 (9)	21 (14)	119 (8)
S. agalactiae (GBS)	18 (1)	2 (1)	16 (1)
S. pneumoniae	26 (2)	3 (2)	23 (2)
S. pyogenes (GAS)	8 (1)	2 (1)	6 (<1)
Gram-negative bacteria
A. baumannii	16 (1)	1 (1)	15 (1)
Enterobacteriaceae	307 (20)	27 (18)	280 (20)
Enterobacter cloacaeb	24 (2)	4 (3)	20 (1)
Escherichia coli	149 (10)	16 (11)	133 (9)
Klebsiella oxytoca	6 (<1)	1 (1)	5 (<1)
Klebsiella pneumoniae	74 (5)	5 (3)	69 (5)
Proteus	22 (1)	0 (0)	22 (2)
Serratia marcescens	22 (1)	1 (1)	21 (1)
Haemophilus influenzae	8 (1)	3 (2)	5 (<1)
Neisseria meningitidis	1 (<1)	0 (0)	1 (0)
P. aeruginosa	52 (3)	4 (3)	48 (3)
Yeast
Candida albicans	20 (1)	1 (1)	19 (1)
Candida glabrata	14 (1)	0 (0)	14 (1)
Candida krusei	4 (<1)	1 (1)	3 (<1)
Candida parapsilosis	8 (1)	0 (0)	8 (1)
Candida tropicalis	3 (<1)	0 (0)	3 (<1)
Antimicrobial resistance genes
mecA	491 (31)	46 (31)	445 (31)
vanA/B	36 (2)	0 (0)	36 (3)
blaKPC	6 (<1)	0 (0)	6 (<1)

Regardless of results of comparator assays.

E. cloacae complex. Individual members are not separated.

n = 1,568 (100%).

Those <18 years old; n = 149 (10% of the total).

Those ≥18 years old; n = 1,419 (90% of the total).

The data from both prospective clinical samples and seeded specimens are combined (see Tables 3, 4, and 6). The columns marked clinical arm and seeded arm give the numbers of positive specimens determined by BCID over those of the comparator assay for each analyte in each study arm. The remainder of the columns include combined prospective and seeded results.

Summary of results for organisms not detected by the BCID panel.

Organisms detectable by the comparator assays for which the BCID pouch did not contain primers are called off-panel organisms (OPOs). A total of 223 OPOs were identified in 210 blood cultures. Of these, 186 cultures (11.9% of the prospective cultures) were BCID negative, including 8 that were BCID FN. Another 24 cultures contained both BCID panel organisms and OPOs. Of the OPOs, 109 (48.9%) were organisms traditionally regarded as skin contaminants, including 49 corynebacteria/diphtheroids, 33 bacilli, and 27 micrococci. Other OPOs were organisms phenotypically misidentified as organisms by assays in the BCID panel. These initially appeared to be BCID FN but were resolved as unidentifiable by the panel. Of these, 16 were Staphylococcus pettenkoferi, a Staphylococcus species not identified by the BCID panel (data not shown; for more information about undetectable organisms and cross-reactivity, see reference (18). Other organisms in this category included Raoultella ornithinolytica and R. planticola, which were identified phenotypically as Klebsiella spp. The Raoultella spp. are included in Table 2 and are also discussed in the section covering BCID results for Gram-negative organisms. An additional six organisms phenotypically identified as A. baumannii and thus giving apparent BCID FN results were shown by16S rRNA gene analysis to be other species of Acinetobacter. Since the molecular identification of A. baumannii was prespecified, these OPOs are not counted among the FN BCID results.

TABLE 2

OPOs not targeted by BCID and identified in the prospective clinical arm only

OPO	No. of isolates	OPO	No. of isolates
Abiotrophia or Granulicatella speciesa	7b	Lactobacillus species	3
Achromobacter xylosoxidans	1	Micrococcus species	27
Acinetobacter species (not A. baumannii)	23c	Moraxella catarrhalis	1
Actinomyces odontolyticus	2	Moraxella species	2
Actinomyces species	1	Mycobacterium fortuitum complex	1
Aerococcus species	3	Neisseria species	2
Aeromonas sobria	1	Paenibacillus species	1
Bacillus cereus	19	Pasteurella multocida	2
Bacillus species	14	Pasteurella species	1
Brevibacterium species	2	Pasteurella stomatis	1d
Brevundimonas species	2	Propionibacterium species	1
Burkholderia cepacia complex	2	Pseudomonas species	5
Candida kefyr	1	Pseudomonas stutzeri	1d
Capnocytophaga species	1	Raoultella ornithinolytica	4d
Chryseobacterium meningosepticum	1	Raoultella planticola	1d
Chryseobacterium indologenes	1	Rhizobium radiobacter	1
Chryseomonas luteola	1	Rhodococcus species	1
Corynebacterium jeikeium	1	Rothia (Stomatococcus) mucilaginosa	4
Corynebacterium species/diphtheroids	48	Sphingomonas mucosissima	1
Cryptococcus neoformans	2	Staphylococcus pettenkoferi	16e
Flavobacterium species	1	Stenotrophomonas maltophilia	10
Fusarium species	1	Weeksella virosa	1
Kocuria kristinae	1

Formerly nutritionally deficient streptococci.

One of seven Abiotrophia spp. was identified by 16S rRNA gene sequencing during discrepancy investigations. It had been misidentified phenotypically as a Streptococcus species.

Of 23 isolates, 17 were Acinetobacter spp. not A. baumannii and 10 were phenotypically A. baumannii. The latter were revealed to be A. pittii, A. nosocomialis, A. bereziniae, and A. radioresistens by 16S rRNA gene sequencing.

Other organisms identified during the resolution of discrepancies between BCID and the phenotypic comparator assays included one Pasteurella stomatis isolate (phenotypic ID, E. coli), one Pseudomonas stutzeri isolate (phenotypic ID, P. aeruginosa), one Raoultella planticola isolate (phenotypic ID, K. pneumoniae), and four R. ornithinolytica isolates (phenotypic ID, K. oxytoca).

Although all Staphylococcus spp. are intended to be detected by the panel, these species do not react with either of the Staphylococcus sp. target assays. Several less common Staphylococcus spp. are not detected or are detected with lower sensitivity (18).

Gram-positive bacteria.

Table 3 presents a summary of BCID results for Gram-positive and Gram-negative bacteria from the clinical and seeded arms compared to phenotypic determinations. These are the original data and are not altered by the resolution of the discrepancies. These data are presented in the context of the resolution of discordant results (see Table 5).

Enterococcus spp.

Relative to the comparator assay, BCID exhibited four FP and three FN results (sensitivity and specificity, 97.7 and 99.8%). The species detected in prospective specimens by phenotypic techniques included 56 of Enterococcus faecalis (55.4%; 1 missed by BCID), 37 of Enterococcus faecium (36.6%; 1 missed by BCID), 2 of Enterococcus avium, 2 of Enterococcus casseliflavus (1 missed by BCID), and 1 each of Enterococcus durans and Enterococcus gallinarum. Of the four specimens with BCID FP results for Enterococcus, one was also FN for E. coli. The frozen aliquot of the latter specimen was Gram stained again and recultured, and the fluid showed only Gram-negative rods and grew E. coli. On retesting by BCID, the results were concordant with the phenotypic results: negative for Enterococcus and positive for E. coli. This was presumed to be a bottle selection error at the time of original testing. The remaining three FP results came in bottles that also contained a Staphylococcus species, identified both by culture and by BCID, but no detectable Enterococcus species. Experiments conducted by BioFire subsequent to these investigations revealed that some staphylococci (i.e., S. epidermidis, S. capitis, and S. haemolyticus), if present at high levels, cross-react with the Enterococcus target because of sequence similarity of the assay primers (18). These particular isolates were not specifically tested and are listed (see Table 5) as inconclusive.

Listeria monocytogenes.

L. monocytogenes was not detected in any of the clinical cultures, so all 36 cultures positive for L. monocytogenes were seeded. The seed organisms included seven distinct clinical isolates and four reference strains: one proficiency test organism from the College of American Pathologists (CAP) and strains ATCC 43248, ATCC 35152, and ATCC 51779. Each was tested one to eight times. All seeds were detected. Thus, the sensitivity and specificity of BCID for L. monocytogenes were both 100%.

Staphylococci.

Staphylococci were detected by the phenotypic assays in 797 samples, while BCID detected 782. One specimen was seeded with S. epidermidis, and that was detected both phenotypically and by BCID, while one Staphylococcus species was detected in a bottle seeded with another organism. The unresolved sensitivity and specificity of BCID were 96.5 and 99.1%, respectively.

Of the 797 staphylococcal isolates that were identified phenotypically in the prospective clinical arm, 28 were not identified by BCID. Sixteen were determined by bidirectional 16S rRNA gene sequencing to be S. pettenkoferi, which was found on later investigation not to be detected by BCID (18). Results from specimens containing more than one Staphylococcus species could not be used to determine BCID sensitivity for this target. BCID sensitivities for unique, named Staphylococcus spp. were >99% for S. epidermidis (201 isolates), S. hominis (64 isolates), S. haemolyticus (19 isolates), S. lugdunensis (5 isolates), S. simulans (3 isolates), S. lentus (1 isolate), and S. cohnii (1 isolate). BCID also detected 18/20 S. capitis and 4/5 S. warneri isolates. BCID failed to detect the unique isolates identified as S. auricularis, S. cohnii, and S. sciuri. For organisms reported as coagulase-negative staphylococci by the comparator assay, including S. pettenkoferi, the BCID sensitivity was 90%.

S. aureus.

Specimens phenotypically identified as S. aureus were also included in the analysis for Staphylococcus spp. No cultures were seeded with S. aureus. The phenotypic assay detected 257 S. aureus isolates in the prospective clinical samples. BCID sensitivity and specificity were 98.4 and 99.8%. Analysis of four FN and four FP results revealed that one of each probably resulted from a specimen mix-up. S aureus was confirmed by molecular techniques from one of three remaining apparent BCID FN cultures. One apparent FN result was actually a Staphylococcus species that had been falsely identified by the comparator method as S. aureus. The other discrepancies remain unresolved.

Streptococci.

Positive phenotypic results for Streptococcus spp. were obtained in 141 prospective cultures, while BCID detected 140. Sixty-two seeded bottles were analyzed. The seeds were prepared from 45 distinct clinical isolates and three reference strains, including ATCC 12386 (S. agalactiae, five cultures), ATCC 19615 (S. pyogenes, nine cultures), and ATCC 4969 (S. pneumoniae, three cultures). BCID detected all of these and also reported one FP result. Overall, there were five FP and five FN Streptococcus genus assay results. The sensitivity and specificity of the Streptococcus genus assay for the organisms that were also detected by one of the specific streptococcal assays (S. agalactiae, S. pneumoniae, and S. pyogenes) were 100%, as described below. Of the remaining 153 isolates, 42 were identified phenotypically as viridans group streptococci, of which BCID detected 39 (92.9%). In the cultures that grew only a single Streptococcus species, BCID detected 8/9 Streptococcus mitis, 5/5 S. anginosus group, 5/5 S. oralis, 3/3 S. bovis, 2/2 Streptococcus group C, and 2/2 Streptococcus group G isolates. BCID also detected each of the unique isolates of S. canis, S. equinus, S. gordonii, S. parasanguinis, and S. salivarius. The five FN and five FP BCID results of the Streptococcus genus assay were further investigated. Three of the five apparent BCID FN cultures were subcultured and shown by bidirectional sequencing to contain Streptococcus spp.; two remain unresolved. The five cultures for which BCID initially appeared to be FP were recultured. One grew an organism identified by 16S rRNA gene sequencing as S. mitis. Another culture grew an Abiotrophia species formerly considered a nutritionally deficient Streptococcus species. Possible cross-reaction with the streptococcal primers in the BCID panel was not verified. In Table 5, which highlights the resolution of discrepancies, the Abiotrophia species and the three remaining BCID FP results remain unresolved.

S. agalactiae (group B Streptococcus).

Culture-positive results for S. agalactiae were obtained for 18 positive clinical specimens. Also included were 18 cultures seeded with S. agalactiae: 13 unique clinical isolates and one reference strain (five cultures). All were detected with no BCID FP or FN results, yielding 100% sensitivity and specificity of BCID for S. agalactiae.

S. pneumoniae.

Positive results for S. pneumoniae were obtained by the phenotypic comparator method for 25 prospective clinical cultures; all were detected by BCID, and there was one BCID FP result. Twelve cultures were seeded with unique clinical isolates of S. pneumoniae (one BCID FN result). One FP S. pneumoniae BCID result was also found in the cultures seeded with other organisms. The one FN result and one of the FP determinations occurred in sequential cultures and were resolved as specimen mix-ups. One FP result remained unresolved.

S. pyogenes (group A Streptococcus).

Positive results for S. pyogenes were obtained by the phenotypic comparator method for seven prospective clinical cultures. BCID detected all of these. A single BCID FP result was obtained. Also included were 31 S. pyogenes cultures seeded with 24 distinct clinical isolates, all of which were detected by BCID with no discrepancies. The unresolved sensitivity and specificity of BCID were 100 and 99.95%.

Subculture of the BCID FP specimen, followed by bidirectional 16S rRNA gene sequencing, identified it as S. dysgalactiae, which was not found to cross-react with the S. pyogenes target.

Gram-negative bacteria.

BCID was designed to detect nine Gram-negative bacteria and also targets members of the family Enterobacteriaceae that do not have a specific assay. The unresolved results for Gram-negative bacteria detected during this evaluation are presented in Table 3. Discrepancies are discussed below (for a summary, see Table 5).

A. baumannii.

A. baumannii was identified by phenotypic identification of an A. calcoaceticus-A. baumannii complex member, followed by bidirectional 16S rRNA gene PCR sequencing. After utilization of both comparator assays, BCID detected 51/51 cultures that were found to be positive for A. baumannii (sensitivity, 100%). Ten isolates phenotypically identified as members of the A. calcoaceticus-A. baumannii complex were not confirmed by 16S rRNA gene sequencing to be A. baumannii but rather proved to be A. pittii (four isolates), A. nosocomialis (four isolates), A. bereziniae (one isolate), and A. radioresistens (one isolate). All except the four A. pittii isolates are included among the Acinetobacter isolates not A. baumannii in Table 2. In addition, BCID falsely identified the four A. pittii isolates and one A. junii isolate as A. baumannii (a total of five FP results, specificity of 99.8%; see also Table 5).

Enterobacteriaceae.

Members of the family Enterobacteriaceae are targeted by a family level assay that detects most of the common genera/species present in human infections. In addition, the E. cloacae complex, E. coli, K. oxytoca, K. pneumoniae, Proteus spp., and S. marcescens all have their own specific assays. If the family level assay and/or one of the genus/species-specific assays was positive, BCID reported “Enterobacteriaceae detected.” As detailed in Table 3, the reference phenotypic methods detected 310 members of the family Enterobacteriaceae in the prospective cultures with four BCID FP and seven BCID FN results. In addition, 188 cultures were seeded with members of the family Enterobacteriaceae. One gave a BCID FN result, bringing the total number of FN results to eight. Two discrepant results were from sequential specimens and were resolved as specimen mix-ups. Another specimen was both FN for Enterobacteriaceae and FP for Enterococcus by BCID (discussed above as a bottle selection error). A third FN BCID Enterobacteriaceae determination came from a specimen that was reported to contain both an E. coli and a Pasteurella species isolate by phenotypic methods, but the E. coli isolate could not be detected on resubculture. The remaining FN specimens were analyzed by 16S rRNA gene sequencing directly from the bottle. Among the four definite BCID FN results were two Pantoea sp. isolates, one S. marcescens isolate, and one Citrobacter koseri isolate. Sequencing of the eighth FN isolate failed, and it remains unresolved. Of the four FP specimens, one was resolved by attribution to a specimen mix-up described above, while three came from mixed cultures, failed to grow on resubculture, and remain unresolved.

E. cloacae complex.

Individual members of the E. cloacae complex were not differentiated either by BCID or by phenotypic assays. Phenotypic assays detected the E. cloacae complex in 22 prospective clinical specimens and 17 cultures seeded with members of the E. cloacae complex, including 14 clinical isolates and one reference strain (ATCC 13407; three cultures). Relative to the phenotypic comparator methods, BCID detected 21 (95.5%) of the 22 positive clinical isolates and all 17 of the seeded cultures. The sensitivity and specificity were 97.4 and 99.9%, respectively. On resolution, the single FN culture grew an organism identified by bidirectional 16S rRNA gene sequencing as E. cloacae, but the presence of E. cloacae in the three FP samples was not confirmed. Instead, three other members of the family Enterobacteriaceae were detected: one Cedecea davisae isolate and two K. pneumoniae isolates.

E. coli.

Phenotypic assays detected E. coli in 148 of the prospective clinical specimens of which BCID detected 145. Five specimens were seeded with distinct clinical isolates that were included to increase the number of blaKPC-containing E. coli isolates, and both culture and BCID detected all five. Five BCID FP results were obtained, four among the 1,420 prospective clinical cultures that were negative for E. coli by phenotypic methods and one among the 634 cultures seeded with other organisms. Three FN results were also obtained. One FN result and one FP result were obtained with sequential specimens and were interpreted as specimen mix-ups. Among the four remaining FP cultures, one was confirmed to contain E. coli on subculture. Three contained other members of the family Enterobacteriaceae; two contained K. pneumoniae, and one contained E. cloacae. These are not known to cross-react in the BCID E. coli assay. Of the two remaining BCID FN samples, one was already discussed as a possible sample selection error (see the Enterococcus and Enterobacteriaceae sections). One E. coli BCID FN specimen is also discussed in the Enterobacteriaceae section.

K. oxytoca.

Phenotypic analysis and BCID detected K. oxytoca in six prospective clinical specimens. However, there was one FP BCID result and one FN BCID result. BCID also detected 54/58 cultures seeded with K. oxytoca. The overall unresolved sensitivity of the K. oxytoca assay was therefore 92.2% (Table 3). The five FN results and one FP results were investigated. No K. oxytoca could be grown from the frozen aliquot of the single FP specimen. One of the five apparent BCID FN results was confirmed to be K. oxytoca by bidirectional 16S rRNA gene sequencing. The other four were identified by 16S rRNA gene sequencing to be R. ornithinolytica, which cannot be distinguished from K. oxytoca by phenotypic methods (20, 21). After resolution of this unavoidable phenotypic error, the resolved sensitivity of the BCID assay for K. oxytoca was 98.3%.

K. pneumoniae.

The phenotypic assays detected K. pneumoniae in 71 prospective clinical specimens, of which BCID detected 68. BCID also detected 34/34 cultures seeded with individual clinical isolates. There were nine apparent BCID FP results and three FN results. Of the three FN BCID results, one was confirmed to be K. pneumoniae. One was found by sequencing to be Raoultella planticola, and one isolate did not grow for further analysis. The nine BCID FP results were investigated by attempting to isolate K. pneumoniae from the blood culture specimens, but no K. pneumoniae could be grown. Two isolates were identified as Enterobacter aerogenes, and two were identified as E. coli. Four BCID FP results, including two of seven that were also BCID positive for other members of the family Enterobacteriaceae, were not resolved.

Proteus.

In prospective clinical cultures, both the comparator phenotypic methods and BCID identified Proteus in 22/22 clinical cultures. BCID also detected Proteus in 22/22 seeded cultures. There were no FN or FP BCID results. Thus, the sensitivity and specificity were 100%.

S. marcescens.

Both the phenotypic methods and BCID identified S. marcescens in the same 22 prospective cultures, with one FP determination. BCID detected 54/55 S. marcescens isolates in seeded cultures with one FN result. The unresolved sensitivity and specificity were 98.7 and 99.9%, respectively. Both the FN and FP results were investigated and found to be due to organisms other than S. marcescens. The apparent FN specimen contained a member of the Serratia proteamaculans-S. grimesii complex that had been misidentified by the phenotypic methods as S. marcescens, while the apparent FP specimen contained an R. ornithinolytica isolate that was identified phenotypically as K. oxytoca but also cross-reacted with the S. marcescens assay (data not shown).

H. influenzae.

In prospective clinical cultures, both the phenotypic methods and BCID identified H. influenzae in the same 8 prospective clinical cultures and in 35/35 (100%) cultures seeded with 22 distinct clinical isolates. There were no FP or FN determinations. The sensitivity and specificity of BCID for H. influenzae were both 100%.

Neisseria meningitidis.

Both the phenotypic methods and BCID identified N. meningitidis in the same clinical culture and in 35/35 cultures seeded with six distinct clinical isolates and five ATCC reference strains: ATCC 13077 (10 cultures), ATCC 13090 (10 cultures), ATCC 13102 (4 cultures), ATCC 13103 (1 culture), and ATCC 35561 (4 cultures). The sensitivity and specificity of BCID for N. meningitidis were both 100%.

P. aeruginosa.

In prospective clinical cultures, both the phenotypic assays and BCID identified P. aeruginosa in 52 cultures. However, BCID reported one FN result and one FP result (sensitivity, 98.1%; specificity, 99.7%). No seeded cultures were included. The apparent BCID FN specimen was found to contain P. stutzeri, not P. aeruginosa. The BCID FP specimen did not contain detectable Pseudomonas, but rather S. aureus was detected.

Candida.

The unresolved results of BCID detection of the Candida species included in the panel are shown in Table 4. Resolution of discordant results is discussed in each organism section and summarized in Table 5.

C. albicans.

The phenotypic assays and BCID detected C. albicans in the same 16 clinical specimens. In addition, both the phenotypic and BCID assays detected 48/48 of cultures seeded with C. albicans. BCID also reported four FP C. albicans detections. C. albicans could not be grown from frozen aliquots of these specimens, although C. glabrata was grown from one. The sensitivity and specificity of BCID for C. albicans were 100 and 99.8%, respectively.

C. glabrata.

The phenotypic assays and BCID detected C. glabrata in the same 12 prospective clinical specimens and 37/37 cultures seeded with C. glabrata (27 clinical isolates, ATCC MYA-2950 [nine cultures], and ATCC 15126 [one culture]). BCID produced two FP results among the prospective clinical cultures (sensitivity, 100%; specificity, 99.9%). C. glabrata could not be grown from frozen aliquots of the two FP specimens, although C. albicans was grown from one.

C. krusei.

The phenotypic method and BCID detected C. krusei in the same 4 prospective clinical specimens and in all 33 seeded specimens (15 distinct clinical isolates, ATCC 14343 [15 cultures], and ATCC 6248 [3 cultures]). BCID did not report any FP or FN detections of C. krusei. Thus, the unresolved sensitivity and specificity were both 100%.

C. parapsilosis.

The phenotypic method and BCID both detected C. parapsilosis in the same seven clinical specimens and 52/54 (96.3%) seeded cultures (two apparent FN results). BCID also produced two FP results (sensitivity, 96.7%; specificity, 99.9%). The two FN results and the two FP results were investigated. Sequencing determined that the two apparent BCID FN samples both contained Candia metapsilosis, which is identified by phenotypic methods as C. parapsilosis. C. parapsilosis could not be grown from either of the two cultures with FP BCID results, although C. albicans, also detected by BCID, grew from one.

C. tropicalis.

Both the phenotypic methods and BCID detected C. tropicalis in the same three prospective clinical specimens and 36/36 cultures seeded with C. tropicalis (35 unique clinical isolates and 1 reference strain [CAP proficiency sample]), with no FP or FN results. Thus, the unresolved sensitivity and specificity were both 100%.

Resolution of discrepant organism identification.

The resolution of BCID FP and FN organism detection is discussed in the section relevant to each organism (Tables 3 and 4 contain unresolved data) and summarized in Table 5. There were 130 apparently discrepant BCID organism detections, 65 FP and 65 FN results. Bottle selection errors apparently occurred with seven (0.3%) of the bottles analyzed by BCID. One that occurred at the time of specimen selection is discussed in the sections on Enterococcus, Enterobacteriaceae, and E. coli. Sixteen additional discrepancies were attributed to three bottle mix-ups (two bottles each). The bottles involved in the mix-ups turned positive at approximately the same time and were erroneously picked by testing personnel. After investigation, discrepancies attributed to bottle selection errors were resolved in favor of BCID.

Of 130 discordant results, 36 were resolved in favor of BCID and 46 were resolved in favor of the comparator assay. In an additional six discordant results, both methods were incorrect because of cross-reactivity with closely related organisms or misidentification by phenotypic methods, leaving 42 discordant results unresolved. Of the BCID FP detections, 25/65 (38.5%) occurred in the 81 prospective clinical cultures that were polymicrobial, as did 16/65 (24.6%) of the BCID FN results.

Antibiotic resistance gene detection.

The results of the BCID antibiotic resistance gene assays are compared to data derived from gene sequencing directly from frozen aliquots from the blood culture bottles (Table 6). A comparison of the BCID resistance gene detection with the phenotypic assays utilized by the participating laboratories is presented elsewhere (18).

Methicillin resistance (mecA) gene detection in Staphylococcus spp.

The mecA PCR comparator assay detected mecA in 496 cultures containing staphylococci (494 clinical and 2 seeded) and BCID detected 488 (98.4%) with eight FN results. Of 286 Staphylococcus-containing clinical cultures in which the comparator assay did not detect mecA, BCID was concordant in 281 (98.3%) with five FP results. The sensitivity of mecA detection in all Staphylococcus spp. was 98.4%. The specificity was 98.3%. The BCID assay is also capable of detecting mecC, formerly known as mecALGAA251. No isolates with mecC were detected in this study.

mecA detection in S. aureus.

Of 139 prospective samples that were positive for mecA by the comparator molecular method, BCID detected mecA in 137 (sensitivity, 98.6%; specificity, 100%). The two isolates with apparent BCID FN results were phenotypically methicillin susceptible.

Vancomycin resistance in Enterococcus spp. (vanA/B).

Among the clinical specimens, BCID vanA/B results were concordant with the molecular comparator method for 36/36 positive and 67/67 negative cultures. In all 29 seeded cultures, vanA/B detection by comparator and BCID agreed (28 positive, 1 negative). The sensitivity and specificity of the vanA/B detection assay were both 100%.

blaKPC in Gram-negative bacteria.

BCID reported a total of 597 specimens to be positive for members of the family Enterobacteriaceae, A. baumannii, or P. aeruginosa. BCID and the comparator method were concordant for 6/6 positive specimens and 364/364 negative specimens. In cultures seeded with members of the family Enterobacteriaceae, A. baumannii, or P. aeruginosa, BCID and the comparator methods were concordant in 33/33 positive detections and in 194/194 negative determinations. The sensitivity and specificity of the BCID blaKPC detection assay are both 100%.

DISCUSSION

Rapid identification of the etiologic agents of bloodstream infection/sepsis and their resistance genes by molecular methods allows early targeted antibiotic therapy, which may improve patient outcomes, reduce mortality, and impact the length of stay of septic patients (5–8, 22, 23). This paper presents the results of a large (1,568 clinical and 639 seeded specimens), multicenter trial of the FilmArray BCID panel. Of the 2,207 BCID analyses performed, 98.1% were completed on the initial run. Repeat analyses of specimens with failed controls were successful in all cases, and no specimens were lost because the repeat test was not initiated within the 8-h time limit. The comparator assays for organism ID were the phenotypic assays used in the participating laboratories. For A. baumannii, 16S rRNA gene sequencing was prespecified to be the final arbiter. This strategy eliminated several organisms phenotypically misidentified as A. baumannii, but it also revealed cross-reactivity of BCID with A. pittii and A. junii.

We did not change the results presented in Tables 3, 4, and 6 after the resolution of discrepancies. The unresolved sensitivities of all BCID assays of organisms other than K. oxytoca were ≥96% (Tables 3, 4, and 6). After four apparent K. oxytoca BCID misses were determined to be R. ornithinolytica by 16S rRNA gene sequencing, the resolved sensitivity for K. oxytoca exceeded 98%. Phenotypic methods incorrectly identify R. ornithinolytica as K. oxytoca (20, 21).

Discordant results, both FP and FN, are a problem when the new assay may be more sensitive and specific than the supposed gold standard. The detection of discrepant results among the seeded bottles was not unexpected since bottles were seeded with archived clinical isolates that were identified phenotypically. These are subject to the same limitations of phenotypic testing as the prospective clinical specimens. Organisms phenotypically identified as K. oxytoca and K. pneumoniae and reported to be BCID misses proved to be R. ornithinolytica and R. planticola, as discussed above. In addition, it was determined by BioFire, subsequent to the research presented here, that 16/28 FN Staphylococcus sp. assay results were due to the failure of the BCID assay to detect S. pettenkoferi. For detailed information on other organisms that may cause cross-reactivity or organisms that may be undetectable or detected with reduced efficiency by BCID, see reference 18.

FilmArray BCID is capable of detecting organisms in mixed cultures. Of prospective clinical specimens, 81 (5.2%) contained multiple (two to four) BCID-detectable organisms. However, other BCID studies have shown that the presence of multiple organisms favors discordant BCID results (15, 24). We obtained BCID FP results with 37 (46%) of the specimens containing multiple organisms. In addition, although polymicrobial cultures represented approximately 10% of those in which Staphylococcus was detected, they were associated with 29% (8/28) of the FN results and 58% (7/12) of the FP results for Staphylococcus. A similar trend was seen for other target organisms.

Investigation of the 130 discrepancies between BCID and the comparator phenotypic assays was approached by investigating the possibility of human error, by growing the organisms from subcultures of frozen aliquots of the specimens, and/or by identification by sequencing methods either directly from aliquots of the frozen specimens or from colonies grown from these aliquots. These investigations revealed one bottle selection error and three possible sample mix-ups (six bottles). The supposed mix-ups led to seven FP and nine FN determinations that were resolved in favor of BCID (Table 5). Overall, 36 (27.6%) of the discrepant results were resolved in favor of BCID and 46 (35.3%) were resolved in favor of the comparator. In 6 (4.6%) cases, both identifications were incorrect, while 42 (32.3%) remain unresolved. The sensitivity and specificity of the BCID detections were not altered in Tables 3 and 4 regardless of the results of these resolution investigations.

There are several possible sources of FP results in nucleic acid-based assays, as molecular methods may detect the genetic material present in nonviable organisms. One cause of molecular FP results could be the ingredients in the blood culture bottles. Package inserts for blood culture bottles state that although the fluid in the bottles is sterile, it may contain nonviable, Gram-stainable organisms. These could theoretically also be detectable by molecular techniques. A preliminary study showed that uninoculated, charcoal-containing medium (BacT/ALERT FA FAN, aerobic; bioMérieux) gave multiple but infrequent FP BCID results (18). Any blood culture bottle type could theoretically contain nucleic acid sequences from nonviable organisms. BCID results that do not agree with the bottle Gram strains should be evaluated carefully. The resin-containing bottles used in this study (BD Bactec Plus Aerobic/F), gave no false-positive results when 92 bottles from 16 lots were screened (data not shown; 95% confidence interval, 0.00 to 0.04). However, some of the bottles seeded with one organism were BCID FP for an additional organism that might have come from the bottle medium.

Another source of these FP results could be rare microbial nucleic acids in the blood injected into the seeded bottles to create the simulated samples. In addition, BCID FP results with clinical samples could be caused by nonviable organisms present in the patients' blood, especially since we do not know whether the patients were on antibiotic therapy at the time of specimen collection. The patient might have originally experienced polymicrobial bacteremia, and one organism might still be able to grow in the blood culture bottles while a different, but no longer viable, organism could remain detectable by BCID. Alternatively, an organism that was no longer viable at the time of culture collection could have been detected by BCID in a patient specimen also containing a contaminating skin organism; these may be expected in about 3% of positive blood cultures (16). Thus, an overrepresentation of “biologic” BCID FP results would be expected in polymicrobial cultures, both in those in which BCID detected other panel organisms and in those containing only OPOs detected by culture.

The unresolved sensitivities and specificities for detection of mecA in all members of the genus Staphylococcus and for S. aureus alone exceeded 98%. The unresolved sensitivities and specificities for vanA/B and blaKPC were 100%. No blaKPC was detected in A. baumannii or P. aeruginosa, as blaKPC in these organisms is currently rare or absent in the United States but has been detected in Puerto Rico (25, 26).

Several previous studies of the BCID have been published (15, 24, 27–30). Most of these analyzed specimens from adults, but one (24) analyzed only pediatric specimens. The majority included 100 to 169 specimens; but one included 204 prospective specimens (27) and another included 111 prospective and 102 archived samples (15). All generally found excellent sensitivity and specificity, although the number of pathogens was limited. Altun et al. (27) showed the reproducibility and stability of the BCID results by retesting five positive bottles for up to 4 weeks. Two studies (15, 27) analyzed all cultures that were flagged as positive by the blood culture instruments, regardless of Gram stain results, and found that some of the bottles grew no organisms and were BCID negative as well. We did not analyze culture-negative bottles since our protocol required a positive Gram stain on bottles flagged as positive prior to BCID testing.

Although our investigation is the largest clinical study of BCID, it is limited by a low number of clinical isolates of certain organisms. No L. monocytogenes isolates were detected and only one N. meningitidis isolate was detected in the prospective clinical specimens. Nonetheless, the presence of these rare and extremely pathogenic organisms is highly significant clinically, and the inclusion of targets for them in this assay seems warranted. For some rare organisms and also for the resistance genes, many of the analyses were performed with seeded specimens. Some of these were archived clinical isolates. In many cases, this did not fulfill the desired numbers, and so reference strains were included, many of which were tested repeatedly. This is not ideal, but a ready alternative to attain statistically significant results does not seem available.

Contamination of blood cultures with skin organisms is unavoidable (16). While Staphylococcus spp. other than S. aureus are the most prevalent contaminating organisms, they may also be the etiologic agents of disease (16) and thus worthy of reporting. However, in this study, the numbers of other OPOs assumed to be skin contaminants (49 Corynebacterium spp., 33 Bacillus spp. [not B. anthracis] and 27 Micrococcus spp.; 16) exceeded many of the pathogens detected by BCID. Identification of a common blood culture contaminant can be useful clinically, as it may facilitate the discontinuation of antibiotics (5, 31). Thus, inclusion of such organisms in the BCID panel might be valuable.

Another limitation is that BCID did not report the resistance genes as detected unless an organism generally recognized as containing that gene was also detected. This seems sensible but may cause occasional problems. For example, it would have prevented the detection of the world's first vancomycin-resistant S. aureus strain. This was isolated in one of the participating laboratories and was found to have acquired vanA (32).

An additional limitation is that the study sites, although they are distributed widely throughout the United States, did not include other countries. Resistance mechanisms and organism strains are known to vary geographically, and information about the detectability of such variations could prove useful, as such organisms and resistance genes often disseminate widely over time (25, 26).