Comparison of five commonly used automated susceptibility testing methods for accuracy in the China Antimicrobial Resistance Surveillance System (CARSS) hospitals.

OBJECTIVE: The objective of this study was to evaluate the performance of five commonly used automated antimicrobial susceptibility testing (AST) systems in China (Vitek 2, Phoenix, Microscan, TDR, and DL).
MATERIALS AND METHODS: Two "unknown" isolates, S1 (ESBL-producing Escherichia coli) and S2 (KPC-producing Klebsiella pneumoniae), were sent to 886 hospitals in China for identification and AST. Using broth microdilution method (BMD) as gold standard, minimum inhibitory concentrations (MICs) were determined.
RESULTS: Most hospitals (392, 46.1%) used Vitek 2, followed by 16% each for Phoenix, Microscan, and DL systems, and 5.9% (50) used TDR system. MICs of 22 antimicrobials were evaluated for two study isolates plus three ATCC strains. Individual susceptibility results for three ATCC strains (n=1581) were submitted by 780 (91.2%) hospitals. For each AST system, 8.7% (6/69) to 13.0% (33/253) reported MICs outside the expected range for several drugs. For the two study isolates, TDR and DL systems performed the worst in MIC determination and susceptibility categorization of cefazolin and cefepime, while the Microscan system had difficulties in susceptibility categorization for aztreonam and ertapenem. Categorical agreements were >90% for most antimicrobials tested for both the isolates, among which, using BMD, no essential agreements were noted for ampicillin, piperacillin, cefazolin, cefuroxime, ceftriaxone, and trimethoprim/sulfamethoxazole. All AST systems except Vitek 2 showed unacceptable VMEs for cefazolin (S1 and S2) and major errors for ceftazidime, cefepime, and aztreonam (isolate S1), while Vitek 2 showed a high VME rate for cefepime (10.0%) and meropenem (6.2%) for S2.
CONCLUSION: None of the five automated systems met the criteria for acceptable AST performance, but Vitek 2 provided a relatively accurate and conservative performance for most of the antimicrobials.

Introduction

Global antimicrobial surveillance programs have provided important epidemiological and antimicrobial susceptibility data for bacterial infections.1,2 In August 2005, the multicentre nationwide China Antimicrobial Resistance Surveillance System (CARSS), which analyzes routine antimicrobial susceptibility data from participating hospitals and develops a quarterly report, was established. Since then, the program has provided important information on the antimicrobial susceptibility patterns of common clinical bacteria in China. Of late, the CARSS program has experienced a dramatic increase in the number of participating hospitals, which currently stands at 1427 in 31 provinces. However, this increase has brought challenges to the program, including concerns about the quality of data submitted by the hospitals due to differences in antimicrobial susceptibility testing (AST) methods and standards.

The choice of which AST method to use in clinical laboratories is mainly dependent on financial resources, labor efficiency, and workload of the laboratory itself.3 Other considerations include cost of equipment and reagents, ease of performance of the technique, flexibility in selection of drugs for testing, maneuverability of automated or semi-automated devices, and the veracity of the methodology.4 In addition, due to the increase in incidence and severity of bacterial infections, the need for more rapid and accurate methods for AST has never been greater.5 To this end, the development of a fully automated AST system is a major advancement, which significantly reduces scientists’ hands-on time, turnaround time, and variability, by using a standard operating procedure.6 Currently, there are five commonly used AST systems in China, including Vitek 2 (bioMérieux, Marcy-l’Étoile, France), Phoenix (Becton Dickinson Diagnostics, Sparks, MD, USA), MicroScan (Beckman Coulter, Brea, CA, USA), Tiandiren (Mindray TianDiRen, Changsha, China), and Dier (Zhuhai DL, Zhuhai, China). Therefore, to ensure accuracy and comparability of antimicrobial susceptibility data collected in the CARSS program, we studied the distribution and use of automated AST systems at each participating hospital.

Specifically, we evaluated the performance of five AST systems in CARSS participating hospitals, by using minimum inhibitory concentration (MIC) determination of several antimicrobials (including β-lactams, aminoglycosides, macrolides, lincosamides, quinolones, glycopeptides, and sulfonamides), against two isolates (Escherichia coli and Klebsiella pneumoniae), to monitor and improve the accuracy of the susceptibility data.

Materials and methods

Study design

This study evaluated the performance of five commonly used automated AST systems in China, including Vitek 2 (bioMérieux), Phoenix (Becton Dickinson Diagnostics), MicroScan (Beckman Coulter), TDR (Mindray TianDiRen) and DL (Zhuhai DL), among 886 hospitals in the CARSS program (2015–2016). MICs for two “unknown” isolates determined by each of the AST systems were compared with the reference method, broth microdilution method (BMD; Clinical Laboratory Standards Institute [CLSI]), which was only performed at three central laboratories – Peking Union Medical College Hospital (PUMCH), National Center for Clinical Laboratories, and Chinese PLA General Hospital.7

Test organisms

As part of a series of studies designed to ensure uniformity and quality of AST results for the CARSS program, two Gram-negative “unknown” isolates, named S1 (extended-spectrum beta-lactamase [ESBL]-producing E. coli) and S2 (carbapenemase-producing K. pneumoniae), obtained from routinely cultured bacteria at one of the central laboratories (PUMCH, Beijing, China), were copied and concurrently sent to 886 hospitals in 31 provinces of China (Figure 1) for identification and AST. The study was carried out in accordance with the institute’s guidelines and procedures, including ethics approval by the Human Research Ethics Committee and obtaining written informed consent from the patients involved. Each participating hospital was directed to use the most commonly used automated AST system, including relevant quality control (QC) strains, on the two isolates. Results were to be reported within a specified time period. QC strains used for BMD at the central laboratories were the CLSI-recommended American Type Culture Collection (ATCC) strains; E. coli ATCC 25922, E. coli ATCC 35218, and Pseudomonas aeruginosa ATCC 27853.

Figure 1

Geographic distribution of participating hospitals in the China Antimicrobial Resistance Surveillance System (CARSS) study.

MIC determination

MICs of several antimicrobial drugs against each of the two “unknown” isolates were determined at each participating hospital using the available AST system, as per manufacturers’ instructions.6 BMD MIC testing was performed in the three central laboratories according to the latest CLSI guidelines, using in-house prepared panels.7 The panels were incubated at 35°C in ambient air and read manually following 16–20 hours of incubation. The MIC of each antimicrobial agent was defined as the lowest concentration that inhibited visible growth of the organism.

Data analysis

Due to differences in CLSI versions used in each AST system, all data from each hospital were imported into WHONET 5.6 software and interpreted according to CLSI 2017 breakpoints at PUMCH, to ensure uniformity.8 BMD results were considered the reference standard. Essential agreement (EA) was defined as percentage of MICs within a single doubling dilution of the corresponding BMD result. Categorical agreement (CA) was the proportion of isolates classified in the same susceptibility category by BMD and the method under evaluation. AST error rates were calculated and reported as follows: very major error (VME, false susceptible result or an inability to detect resistance); major error (ME, false “resistant” result); and minor error (mE, an intermediate result reported as either susceptible or resistant and vice versa). According to CLSI recommendations, the acceptable inter-method VME, ME, and mE rates are 1.5%, 3%, and 10%, respectively.9

Results

An overwhelming majority (96.0%; 851/886) of the hospitals returned results within the specified time frame, among which 392 (46.1%) used Vitek 2, and 146 (17.2%), 133 (15.6%), 130 (15.3%), and 50 (5.9%) used Phoenix, Microscan, DL, and TDR systems, respectively (Table S1). The remaining 35 (4.0%) hospitals did not return results on time and were excluded. Both the “unknown” isolates were correctly identified by all the participating hospitals. Based on the antibiotics tested by each system, 22 antibiotics (Table 1) were analyzed. However, due to differences in AST cards, not all drugs were reported by each system. For example, susceptibilities to amoxicillin/clavulanic acid (AMC), cefotetan (CTT), ertapenem (ETP), imipenem (IPM), and nitrofurantoin (NIT) were not reported by TDR system; susceptibilities to cefuroxime (CXM), CTT, ETP, tobramycin (TOB), and NIT were not reported by Phoenix system; and susceptibilities to CTT, ETP, and TOB were not reported by DL system.

Table 1

Susceptibility results of three quality control strains for each antibiotic reported by five individual automated systems as interpreted by CLSI 2017

Antibiotic	Automated system	Totala	MIC outside reference range		Susceptibility category not determinableb
			N	%	N	%
AMP	Dier	126	0	0	69	54.8
	Microscan	136	0	0	69	50.7
	Phoenix	151	0	0	91	60.3
	Tiandiren	42	0	0	23	54.8
	Vitek 2	473	11	2.3	276	58.4
PIP	Dier	228	2	0.9	110	48.2
	Microscan	253	5	2	110	43.5
	Phoenix	273	3	1.1	129	47.3
	Tiandiren	69	0	0	36	52.2
	Vitek 2	758	13	1.7	279	36.8
AMC	Dier	126	0	0	25	19.8
	Microscan	136	0	0	30	22.1
	Phoenix	151	0	0	37	24.5
	Vitek 2	473	2	0.4	77	16.3
SAM	Dier	126	1	0.8	56	44.4
	Microscan	136	0	0	54	39.7
	Phoenix	151	0	0	60	39.7
	Tiandiren	42	0	0	14	33.3
	Vitek 2	473	6	1.3	202	42.7
TZP	Dier	228	2	0.9	212	93
	Microscan	253	8	3.2	229	90.5
	Phoenix	273	5	1.8	250	91.6
	Tiandiren	69	0	0	65	94.2
	Vitek 2	758	17	2.2	684	90.2
CAZ	Dier	221	4	1.8	157	71
	Microscan	244	10	4.1	168	68.9
	Phoenix	261	6	2.3	197	75.5
	Tiandiren	68	0	0	52	76.5
	Vitek 2	637	18	2.8	458	71.9
CRO	Dier	221	4	1.8	111	50.2
	Microscan	244	6	2.5	109	44.7
	Phoenix	261	3	1.1	134	51.3
	Tiandiren	68	0	0	39	57.4
	Vitek 2	637	15	2.4	293	46
CTT	Microscan	127	2	1.6	50	39.4
	Vitek 2	352	1	0.3	125	35.5
CXM	Dier	119	1	0.8	43	36.1
	Microscan	127	0	0	34	26.8
	Tiandiren	41	0	0	16	39
	Vitek 2	352	5	1.4	97	27.6
CZO	Dier	119	0	0	113	95
	Microscan	127	0	0	109	85.8
	Phoenix	139	2	1.4	129	92.8
	Tiandiren	41	1	2.4	36	87.8
	Vitek 2	352	5	1.4	316	89.8
FEP	Dier	221	8	3.6	193	87.3
	Microscan	244	11	4.5	209	85.7
	Phoenix	261	7	2.7	238	91.2
	Tiandiren	68	2	2.9	61	89.7
	Vitek 2	637	23	3.6	549	86.2
ATM	Dier	228	5	2.2	137	60.1
	Microscan	253	8	3.2	150	59.3
	Phoenix	273	5	1.8	168	61.5
	Tiandiren	69	1	1.4	45	65.2
	Vitek 2	758	16	2.1	413	54.5
ETP	Microscan	244	3	1.2	72	29.5
	Vitek 2	637	5	0.8	218	34.2
IPM	Dier	221	4	1.8	132	59.7
	Microscan	244	7	2.9	144	59
	Phoenix	261	3	1.1	157	60.2
	Vitek 2	637	12	1.9	383	60.1
MEM	Dier	221	1	0.5	160	72.4
	Microscan	244	9	3.7	168	68.9
	Phoenix	261	4	1.5	182	69.7
	Tiandiren	68	0	0	47	69.1
	Vitek 2	637	16	2.5	403	63.3
AMK	Dier	221	3	1.4	209	94.6
	Microscan	244	8	3.3	225	92.2
	Phoenix	261	3	1.1	244	93.5
	Tiandiren	68	0	0	67	98.5
	Vitek 2	637	20	3.1	588	92.3
GEN	Dier	221	1	0.5	215	97.3
	Microscan	244	7	2.9	226	92.6
	Phoenix	261	3	1.1	251	96.2
	Tiandiren	68	0	0	67	98.5
	Vitek 2	637	16	2.5	602	94.5
TOB	Microscan	244	6	2.5	162	66.4
	Tiandiren	68	0	0	48	70.6
	Vitek 2	637	9	1.4	414	65
CIP	Dier	221	6	2.7	195	88.2
	Microscan	244	11	4.5	209	85.7
	Phoenix	261	5	1.9	237	90.8
	Tiandiren	68	1	1.5	64	94.1
	Vitek 2	637	23	3.6	565	88.7
LVX	Dier	221	7	3.2	160	72.4
	Microscan	244	10	4.1	172	70.5
	Phoenix	261	6	2.3	186	71.3
	Tiandiren	68	1	1.5	49	72.1
	Vitek 2	637	24	3.8	434	68.1
SXT	Dier	221	5	2.3	127	57.5
	Microscan	244	15	6.1	129	52.9
	Phoenix	261	6	2.3	132	50.6
	Tiandiren	68	1	1.5	42	61.8
	Vitek 2	637	25	3.9	311	48.8
NIT	Dier	119	0	0	67	56.3
	Microscan	127	2	1.6	64	50.4
	Vitek 2	352	5	1.4	188	53.4

Notes:

Total number of individual susceptibility results obtained from hospitals for three QC strains for each antibiotic tested.

Susceptibility category not determinable: as interpreted using CLSI 2017 breakpoints.

Abbreviations: CLSI, Clinical Laboratory Standards Institute; MIC, minimum inhibitory concentration; AMP, ampicillin; PIP, piperacillin; AMC, amoxicillin/clavulanic acid; SAM, ampicillin/sulbactam; TZP, piperacillin/tazobactam; CZO, cefazolin; CXM, cefuroxime; CAZ, ceftazidime; CRO, ceftriaxone; FEP, cefepime; CTT, cefotetan; ATM, aztreonam; ETP, ertapenem; IPM, imipenem; MEM, meropenem; AMK, amikacin; GEN, gentamicin; TOB, tobramycin; CIP, ciprofloxacin; LVX, levofloxacin; SXT, trimethoprim/sulfamethoxazole; NIT, nitrofurantoin; QC, quality control.

Susceptibility results of QC strains reported by five automated systems using CLSI 2017 breakpoints

In total, 1581 individual MIC results for three ATCC strains were submitted by 780 (91.2%; 780/851) hospitals, with one (121, 14.2%), two (517, 60.5%) or three (142, 16.6%) ATCC strain results at the same time; the remaining 71 (8.3%) hospitals did not report any QC results due to various reasons (data not shown). The reported MICs were analyzed according to the CLSI 2017 M100S document.8 For the three QC strains reported in the study, only drugs with specified MIC reference ranges as per the CLSI 2017 document8 were analyzed. Most results were from Vitek 2 which was used by more laboratories (Table S2). Likewise, the least number (4.4%, 69) of results came from the TDR system. About 50% of the results were from Phoenix, Microscan, and DL systems, averaging 15.9% each (range 14.4%–17.3%).

Overall, results outside the reference MIC range for the three ATCC strains were reported in about 10.3% of the cases (range 8.7%–13.0%) in the five systems; highest (13.0%) was in Microscan, and the lowest (8.7%) in TDR (Table S3). The TDR system performed the best in accurately determining MIC values of the QC strains, with expected MIC values in 64.7% (11/17) of the cases (Tables S4–S8). The highest rate of MICs outside the reference range for the three ATCC strains was observed in trimethoprim/sulfamethoxazole (SXT) (3.6%; range, 2.3% [Phoenix] to 6.1% [Microscan]; Tables 1 and S3–S8). However, most MIC results from the AST systems were categorized as “not determinable” as the MICs given were not a specific value and thus could not be discriminated using the CLSI 2017 breakpoints. This was especially common in amikacin (AMK) (≥92.2% in each of the five systems), piperacillin/tazobactam (TZP) (≥90.2% in each), gentamicin (GEN) (≥92.6% in each), cefazolin (CZO) (≥85.8% in each), cefepime (FEP) (≥85.7%) and ciprofloxacin (CIP) (≥ 85.7% in each; Tables 1 and S4–S8).

Susceptibilities by BMD at central laboratories

Contrary to the poor performance of the AST systems in MIC determination of the three QC strains, the BMD method yielded excellent results, with all MICs (100%) in the expected reference ranges. Moreover, MICs of the two study isolates (S1 and S2) for several antimicrobials, as performed by each of the three central laboratories, were 100% in agreement (Table S9). S1 showed susceptibility or intermediate susceptibility to most of the antimicrobials except ampicillin (AMP), piperacillin (PIP), CZO, CXM, ceftriaxone (CRO), CIP, levofloxacin (LVX), and SXT, while S2 exhibited resistance to all the antibiotics, with intermediate susceptibility to only CTT.

Comparison of susceptibility profiles of the two study isolates for five automated systems

MICs of the two study isolates (S1 and S2) were interpreted and categorized using CLSI 2017 breakpoints via WHO-NET 5.6, a Windows-based database software.8 Data were analyzed and presented as percentages except in instances when the numbers were low (<10) (Table 2). Apart from the three commonly used susceptibility categories of resistant (R), intermediate susceptible (I), and susceptible (S), three additional categories of “R?”, “S?” and “non-susceptible (NS)” were generated. This was because the MICs given by each automated system could not be classified precisely as R/I/S according to CLSI 2017 breakpoints, mainly due to the drawbacks of the system itself.

Table 2

Antimicrobial susceptibility profiles of two CARSS isolates (S1 and S2) for five individual automated systems as per CLSI 2017 guidelines

Abx	Automated system	N	S1						S2
			Susceptibility category						Susceptibility category
			%R	%R?	%I	%S	%S?	%NS	%R	%R?	%I	%S	%S?	%NS
AMP	Microscan	133	94	0.8	3			2.2	94.8	0.7	3.7	0.7
	Phoenixc	143	95.1		4.2			0.7	96.5		2.8			0.7
PIP	Dier	27	96.3		3.7				92.6	3.7	3.7
	Microscan	129	93.8		3.1			3.1	99.2		0.8
	Phoenixc	142	94.4		4.9			0.7	95		3.5	1.4
AMC	Microscan	126	0.8			99.2			92.9	1.6	3.1	0.8		1.6
	Phoenixc	143	1.4		0.7	97.2		0.7	94.4		4.2	1.4
SAM	Dier	128	5.5		71.1	21.9	0.8	0.8	95.3	2.3	1.6	0.8
	Tiandiren	49	85.7		10.2	2		2	98		2
TZP	Dier	130	3.1			96.9			95.3		0.8	0.8		3.1
	Phoenixc	144	0.7			99.3			95.1	0.7	2.8	0.7		0.7
	Vitek 2	411	0			100			98.8		0.7			0.5
CAZ	Microscan	132	1.5			97.7		0.8	94.8	0.7	4.4
	Phoenixc	142	3.5	0.7		95.8			90.1		8.5	1.4
CTT	Microscan	8				8			1	1		6		1
	Vitek 2	351	0.3			99.7			3.1	0.3	21.1	75.5
CXM	Microscan	121	94.2		5	0.8			95	0.8	3.3	0.8
CZOa	Dier	130	0.8	96.9		2.3			0.8	96.9		2.3
	Microscan	131	95.4	0.8		3.8			95.5	0.7		3.7
	Phoenixc	138	97.1	0.7		2.2			97.1	0.7		2.2
	Tiandiren	50	4	92		4			4	92		4
FEP	Dier	130	22.3		23.1	40	14.6		86.8		10.1	0.8	2.3
	Microscan	132	90.2		1.5	3.8	4.5		97.8			0.7	1.5
	Phoenixc	144	38.2		59	1.4	1.4		97.9		1.4	0.7
	Tiandiren	50	74		2		24		94		2		4
	Vitek 2	439	0.7		1.8	97.5			13		76.8	10		0.2
ATM	Microscanb	131	44.3	0.8	2.3	9.2	43.5		99.3	0.7
ETP	Microscanb	121	1.7			2.5	95.9		99.2				0.8
	Vitek 2	296				98.6	1.4		99.7			0.3
IPM	Phoenixc	145	0.7	0.7		97.9	0.7		92.4		1.4	6.2
MEM	Phoenixc	143	0.7			98.6	0.7		93.7		1.4	4.9
	Tiandiren	50	4			96			88			10		2
AMK	Dier	125	1.6		0.8	97.6			97.6	0.8	0.8	0.8
	Microscan	134	0.7			98.5		0.7	96.3		2.2	1.5
	Phoenixc	145	0.7			99.3			95.9		2.8	0.7		0.7
GEN	Dier	130	2.3		0.8	96.9			97.7	0.8	0.8	0.8
CIP	Dier	129	98.4		0.8			0.8	99.2		0.8
	Microscan	134	97		2.2			0.7	95.6		4.4
	Vitek 2	439	100						99.8				0.2
LVX	Dier	129	42.6		55.8	0.8		0.8	89.1		10.9
	Microscan	134	73.1		25.4			1.5	94.1		3.7	2.2
	Tiandiren	50	94		4		2		100
	Vitek 2	439	92.5		7.3		0.2		99.5		0.5
SXT	Dier	129	96.9	2.3		0.8			96.9	2.3		0.8
	Microscan	122	96.7	2.5		0.8			97.5	1.6		0.8
	Phoenixc	142	96.5	1.4		2.1			96.5			3.5
NIT	Dier	47	2.1		2.1	93.6		2.1	83.3			2.1		14.6
	Microscan	10	1			7	2		9

Notes:

Dier and Tiandiren systems performed poorly in MIC categorization of CZO for both S1 and S2 isolates, with ≥92% of the results categorized as “R?.”

The Microscan system had difficulties in susceptibility categorization of ETP, followed by ATM, for isolate S1, with 95.9% (116/121) and 43.5% (57/131) of the results categorized as “S?” when applying the CLSI 2017 breakpoints.

The Phoenix system had overall lower percentages of uncategorized MICs for each antibiotic. About 53% (9/17) and 23.5% (4/17) of the antimicrobials tested had various uncategorized problems for both S1 and S2 isolates, respectively. Bold text represents the percentages of MICs that could not be classified precisely by each system.

Abbreviations: CARSS, China Antimicrobial Resistance Surveillance System; CLSI, Clinical Laboratory Standards Institute; R, resistant; I, intermediate susceptible; S, susceptible; NS, non-susceptible; Abx, antibiotic; AMP, ampicillin; PIP, piperacillin; AMC, amoxicillin/clavulanic acid; SAM, ampicillin/sulbactam; TZP, piperacillin/tazobactam; CAZ, ceftazidime; CTT, cefotetan; CXM, cefuroxime; CZO, cefazolin; FEP, cefepime; ATM, aztreonam; ETP, ertapenem; IPM, imipenem; MEM, meropenem; AMK, amikacin; GEN, gentamicin; CIP, ciprofloxacin; LVX, levofloxacin; SXT, trimethoprim/sulfamethoxazole; NIT, nitrofurantoin; MIC, minimum inhibitory concentration.

For instance, ≥92% of the results from hospitals using the DL and TDR systems reported an MIC of >4 for CZO on isolate S1, leading to the category of “R?” as per the CLSI 2017 breakpoints of “S≤16, R≥32”. Likewise, 14.6%, 24%, 4.5%, and 1.4% of the results from hospitals using DL, TDR, Microscan, and Phoenix systems, respectively, reported an MIC of ≤8 or ≤4 for FEP on isolate S1, which was categorized as “S?” as per CLSI 2017 breakpoints of “S≤2, R≥16.” And finally, for MIC results categorized as “NS” by CLSI 2017 breakpoints, the MICs were all reported as “> susceptible cut-off value” due to system inability to differentiate between “I” and “R.” For example, the CLSI 2017 breakpoints for NIT are “S≤32, R≥128,” yet 14.6% (7/48) of the DL system results were reported as MIC of >32 for isolate S2, which was consequently interpreted as “NS” (Table 2).

Overall, Vitek 2 performed the best as it correctly categorized MICs of most antibiotics tested (20/22; 90.9%) for isolate S1, followed by TDR (76.5%; 13/27). In contrast, Phoenix and Microscan systems only correctly categorized about 46% each of the antibiotics tested. For isolate S2, Vitek 2 (86.4%; 19/22) and TDR (82.4%; 14/17) systems still performed better than others (Table 2). For isolate S1, ESBL interpretation was reported in ≥83.6% of Microscan, DL, and Phoenix results, and only 18.0% of TDR users (Table S1).

Performance characteristics (MIC agreements and errors) amongst five automated systems

The overall performance characteristics of each AST system are shown in Table 3. For isolate S1, all systems showed a CA >90% for most of the antibiotics. Similar findings were reported for isolate S2, except for CZO (DL 0.8%, TDR 4%), FEP (Vitek 2 13.0%), and IPM (DL 59.1%). However, despite the above, considerable discrepancies in EA still existed. EAs were nearly zero for both the study isolates in all the AST systems for several antibiotics, though with high CA percentages (Table 3). For example, the MICs of these antibiotics were either too high (MIC >256 for AMP, PIP) or too low (MIC of 0.03 for ETP) for isolate S1, but still qualified the classification criteria, though the MIC could go beyond the concentration range within each automated AST system, leading to a relatively high CA along with an extremely low EA.

Table 3

Comparison of overall performance characteristics between each of the five automated systems and broth microdilution for the two CARSS isolates

Abx	Automated system	N	S1					S2
			EA (%)a	CA (%)b	VME (%)c	ME (%)d	mE (%)e	EA (%)a	CA (%)b	VME (%)c	ME (%)d	mE (%)
AMP	Dier	129	0	99.2			0.8	0	99.2			0.8
	Microscan	133	0	94			3	0	94.8	0.7		3.7
	Phoenix	143	0	95.1			4.2	0	96.5			2.8
	Tiandiren	50	0	100				0	100
	Vitek 2	439	0	99.8	0.2			0	100
PIP	Dier	27	0	96.3			3.7	0	92.6			3.7
	Microscan	129	0	93.8			3.1	0	99.2			0.8
	Phoenix	142	0	94.4			4.9	0	95	1.4		3.6
	Tiandiren	49	4.1	100				4.1	100
	Vitek 2	28	7.1	92.9			7.1	3.6	100
AMC	Dier	23	91.2	69.6		8.7	21.7	0	95.7			4.3
	Microscan	126	97.6	99.2		0.8		0	92.9	0.8		3.1
	Phoenix	143	98.6	97.2		1.4	0.7	0	94.4	1.4		4.2
	Vitek 2	28	100	100				0	100
SAM	Dier	128	96.1	71.1			27.4	0	95.3	0.8		1.6
	Microscan	10	9	8			2	0	10			1
	Phoenix	140	100	95			5	0	95.7			4.3
	Tiandiren	49	95.8	10.2			87.7	0	98			2
	Vitek 2	357	98.9	22.1			77.9	0	100
TZP	Dier	130	79.3	96.9		3.1		3.1	95.3	0.8		0.8
	Microscan	134	5.2	100				0	97	0.8		2.2
	Phoenix	144	98	99.3		0.7		0	95.1	0.7		2.8
	Tiandiren	50	0	96		4		4	96	4
	Vitek 2	411	100	100				0	98.8			0.7
CAZ	Dier	129	18.6	96.9		3.1		98.4	96.9	0.8		2.3
	Microscan	132	92.4	97.7		1.5		99.2	94.8	0		4.4
	Phoenix	142	96.5	95.8		3.5		98.6	90.1	1.4		8.5
	Tiandiren	50	10	96		4		96	94	4		2
	Vitek 2	385	99.2	99.2		0.5	0.3	97.2	100
CRO	Dier	130	0	100				0	100
	Microscan	133	0	100				0	100
	Phoenix	8	0	8				0	9
	Tiandiren	50	0	100				0	100
	Vitek 2	439	0	99.8			0.2	0	100
CTT	Microscan	8	0	8				8	0			7
	Vitek 2	351	0	99.7		0.3		81.7	21.1			78.6
CXM	Dier	129	0	99.2			0.8	0	99.2			0.8
	Microscan	121	0	94.2	0.8		5	0	95	0.8		3.3
	Tiandiren	50	0	100				0	100
	Vitek 2	143	0	100				0	100
CZO	Dier	130	0	0.8	2.3			0	0.8	2.3
	Microscan	131	0	95.4	3.8			0	95.5	3.7
	Phoenix	138	0	97.1	2.2			0	97.1	2.2
	Tiandiren	50	0	4	4			0	4	4
	Vitek 2	439	0	100				0	100
FEP	Dier	130	39.3	40		22.3	23.1	97.7	86.8	0.8		10.1
	Microscan	132	3.8	3.8		90.2	1.5	97	97.8	0.7
	Phoenix	144	45.2	1.4		38.2	59	95.7	97.9	0.7		1.4
	Tiandiren	50	0	0		74	2	96	94			2
	Vitek 2	439	98.8	97.5		0.7	1.8	69.9	13	10		76.8
ATM	Dier	28	92.8	92.8		3.6	3.6	0	100
	Microscan	131	9.9	9.2		44.3	2.3	0	99.3
	Phoenix	144	96.6	96.5		3.5		0	97.9	0.7		1.4
	Tiandiren	49	83.7	83.7		14.3	2	0	95.9	4.1
	Vitek 2	433	98.6	99.5		0.5		0	100
ETP	Microscan	121	0	2.5		1.7		0.8	99.2	0
	Vitek 2	296	0	98.6				3	99.7	0.3
IPM	Dier	128	0.8	99.2		0.8		13.3	59.1	4.7		36.2
	Microscan	134	2.2	100				87.5	96.3	0.7		3
	Phoenix	145	2.1	97.9		0.7		80	92.4	6.2		1.4
	Vitek 2	439	0.7	100				90.1	98.4	0		1.6
MEM	Dier	129	0.8	99.2		0.8		95.4	95.3	0.8		3.9
	Microscan	130	0.8	100				97.6	97.7	1.5		0.8
	Phoenix	143	0.7	98.6		0.7		93.1	93.7	4.9		1.4
	Tiandiren	50	0	96		4		88	88	10
	Vitek 2	161	95	100				83.9	83.9	6.2		9.9
AMK	Dier	125	76	97.6		1.6	0.8	0	97.6	0.8		0.8
	Microscan	134	94	98.5		0.7		0	96.3	1.5		2.2
	Phoenix	145	98.7	99.3		0.7		0	95.9	0.7		2.8
	Tiandiren	49	6.1	95.9		4.1		0	95.9	4.1
	Vitek 2	439	99.1	100				0.2	98.9	1.1
GEN	Dier	130	70	96.9		2.3	0.8	0	97.7	0.8		0.8
	Microscan	134	11.2	99.3		0.7		0	95.6	1.5		2.9
	Phoenix	145	98.6	99.3		0.7		0	96.6	0.7		2.7
	Tiandiren	50	2	96		4		0	96	4
	Vitek 2	439	100	100				0	98.9	1.1
TOB	Microscan	132	6.8	99.2		0.8		0	97.7	0.8		1.5
	Tiandiren	49	2	95.9		4.1		0	95.8	4.2
	Vitek 2	411	100	100				0.2	99	1
CIP	Dier	129	98.5	98.4			0.8	0.8	99.2			0.8
	Microscan	134	97	97			2.2	0	95.6			4.4
	Phoenix	144	97.2	97.2			2.8	1.4	97.2			2.8
	Tiandiren	49	100	100				2	100
	Vitek 2	439	100	100				0.5	99.8
LVX	Dier	129	99.3	42.6	0.8		55.8	89.1	89.1			10.9
	Microscan	134	100	73.1			25.4	94	94.1	2.2		3.7
	Phoenix	144	99.3	99.3	0.7			97.9	97.9			2.1
	Tiandiren	50	100	94			4	100	100
	Vitek 2	439	99.8	92.5			7.3	99.5	99.5			0.5
SXT	Dier	129	0	96.9	0.8			0	96.9	0.8
	Microscan	122	1.6	96.7	0.8			1.6	97.5	0.8
	Phoenix	142	0	96.5	2.1			0	96.5	3.5
	Tiandiren	50	0	100				0	100
	Vitek 2	439	0	99.8	0.2			0	99.3	0.7
NIT	Dier	47	93.6	93.6		2.1	2.1	2.1	83.3	2.1
	Microscan	10	7	7		1		1	9
	Vitek 2	439	99.6	100				100	100

Notes:

EA (%), percentage of MICs within a single doubling dilution of the MIC result determined by BMD.

CA (%), percentage of isolates classified in the same susceptibility category by BMD and the method under evaluation.

VME (%), false susceptible result or an inability to detect resistance.

ME (%), false “resistant” result.

mE (%), an intermediate result reported as either susceptible or resistant and vice versa.

Abbreviations: Abx, antibiotic; EA, essential agreement; CA, categorical agreement; VME, very major error; ME, major error; mE, minor error; AMP, ampicillin; PIP, piperacillin; AMC, amoxicillin/clavulanic acid; SAM, ampicillin/sulbactam; TZP, piperacillin/tazobactam; CAZ, ceftazidime; CRO, ceftriaxone; CTT, cefotetan; CXM, cefuroxime; CZO, cefazolin; FEP, cefepime; ATM, aztreonam; ETP, ertapenem; IPM, imipenem; MEM, meropenem; AMK, amikacin; GEN, gentamicin; TOB, tobramycin; CIP, ciprofloxacin; LVX, levofloxacin; SXT, trimethoprim/sulfamethoxazole; NIT, nitrofurantoin; MIC, minimum inhibitory concentration; BMD, broth microdilution method.

Comparison of incidence of errors for the five automated systems

For isolate S1, notable VMEs were observed in CZO (DL 2.3%, TDR 4%, Microscan 3.8%, and Phoenix 2.2%) and SXT (Phoenix 2.1%). High rates of MEs were observed in AMC (DL 8.7%), TZP (DL 3.1%), ceftazidime (CAZ, 3.1%–4% in DL, TDR, and Phoenix), FEP (22.3%–90.2% each for all except Vitek), aztreonam (ATM) (3.6%–44.3% each for all except Vitek), AMK and TOB (TDR 4.1% each). No VMEs and MEs were observed in most antibiotics tested by Vitek 2, say for four antibiotics (≤0.7% each; Table 3). Nevertheless, substantial mEs were observed in Vitek (77.9%) for ampicillin/sulbactam (SAM), along with DL (27.4%) and TDR (87.7%) systems. Categorizing by antibiotics, mEs were observed for the DL system in the following: AMC (21.7%), FEP (23.1%), and LVX (55.8%). Further mEs were reported for FEP (59.0%, Phoenix) and LVX (25.4%, Microscan). To sum up, the five AST systems showed comparable performance with BMD in 12 of 22 (Vitek), 7 of 17 (Phoenix), 4 of 22 (Microscan), 2 of 17 (TDR), and 2 of 19 (DL) antibiotics tested by each system for isolate S1. CIP was the only drug that satisfied the CLSI standards by all AST systems, with an MIC of 8, which was categorized as “R” by CLSI breakpoints (Table 3).

The situation was a bit worse for isolate S2, which exhibited almost total resistance to the antibiotics tested and with high MICs. Although no MEs were observed, VME rates were worse compared with isolate S1. High rates of VMEs were observed with meropenem (MEM; Vitek 6.2%, TDR 10.0%, Phoenix 4.9%), IPM (Phoenix 6.2%, DL 4.7%), and CZO (2.2%–4.0% each for all except Vitek). Notably, Vitek 2 had the highest interpretive errors in FEP (10.0% VME and 76.8% mEs). Substantial VMEs were also observed for CAZ, ATM, AMK, GEN, and TOB (4.0%–4.2%) by TDR system, LVX (2.2%) by Microscan system, SXT (3.5%) by Phoenix system, and NIT (2.1%) by DL system. A high rate of mEs was also observed in the DL system for FEP (10.1%), IPM (36.2%), and LVX (10.9%), and in Vitek 2 for CTT (78.6%). Overall, only four (CAZ, IPM, LVX, and NIT) of 22, three (CAZ, FEP, and LVX) of 17, three (CAZ, FEP, and MEM) of 22, two (FEP and LVX) of 17, and two (CAZ and MEM) of 19 antibiotics tested by Vitek 2, Phoenix, Microscan, TDR and DL systems met the recommended standards by CLSI (Table 3).

Discussion

With the popularization of laboratory automation testing, more clinical laboratories in China are using commercial automated systems for AST. MIC determination is of great importance in the selection of antimicrobial therapy to guide the most appropriate dosing regimen.10 However, differences in instrument manufacturers, software versions, including the built-in susceptibility breakpoints edition, and even AST panels, could make a difference in MIC results.11 Therefore, it is imperative to evaluate the accuracy of commonly used susceptibility testing methods. The five automated systems evaluated in this study are market leaders in China and many other countries. To the best of our knowledge, this is the first and largest study evaluating the accuracy of these automated systems for susceptibility testing of two commonly encountered Enterobacteriaceae, with 851 participating hospitals and using the latest CLSI 2017 breakpoints.

This study highlights three important points. First, not all hospitals in China seem to fully understand and appreciate the importance of the internal QC for MIC determination. E. coli ATCC 25922, E. coli ATCC 35218, and P. aeruginosa ATCC 27853 are the most commonly used QC strains for antimicrobial testing in clinical microbiology laboratories. However, in the present study, 71 (8.3%) of the hospitals did not return any results of the QC strains. To figure out possible reasons, we contacted these laboratories and identified three main reasons, including lack of QC strains, and forgetting to perform the test or to upload the data. In addition, even for those laboratories that tested the QC strains, the results were barely satisfactory for all the five AST systems, with a considerable proportion of MICs that were outside the acceptable range and some cases of “not determinable” results, especially for AMK, CIP, CZO, FEP, GEN, SXT, and TZP. This may be due to failure by scientists to properly follow standardized procedures during a series of technological processes.12 Also, the instrument itself has its own detection limit which may contribute to the category of “not determinable.”11 All these findings reinforce the need to increase awareness on the importance of accurately performing susceptibility testing of QC strains before MIC testing in China. There is also an urgent need for continual improvement of individual instruments, including software upgrades to incorporate the latest antimicrobial breakpoints.

Second, our results reveal that the five automated AST systems commonly used in China, except Vitek 2, are not reliable for correctly categorizing susceptibility profiles for certain drugs (CZO and FEP: DL and TDR system; ATM and ETP: Microscan system; IPM: Phoenix system), leading to wrong classifications as “R?,” “S?,” and “NS,” which has not been previously reported. Possible explanations for this include the use of the unified latest 2017 CLSI breakpoints for MIC interpretation in the present study, instead of the corresponding breakpoints within each individual instrument. In fact, the breakpoint versions used in the five AST systems in the participating hospitals ranged from CLSI 2009 to CLSI 2016. In addition, in the past several years, CLSI has revised the breakpoints for several antimicrobial agents commonly tested against Gram-negative bacteria. This includes a revision of the breakpoints for Enterobacteriaceae for the antimicrobials ATM, CZO, FEP, CAZ, CRO, ETP, IPM, and MEM.13 These updates in CLSI versions may lead to differences in MIC interpretation. For example, since 2012, the susceptible breakpoint for ETP has gone through two significant changes, from ≤2 (CLSI 2010) to ≤0.25 (CLSI 2011), and is now coming to ≤0.5.14–16 For Microscan system, 95.9% (116/121) of the susceptibility results for ETP were categorized as “S?,” with 95.7% (111/116) exhibiting a MIC of ≤2, and 4.3% (5/116) with a MIC of ≤1, which can all be categorized as “S” when using the CLSI 2010 breakpoints instead of “S?” according to CLSI 2017 breakpoints. Thus, it is incumbent upon the instrument manufacturer to keep pace with the breakpoint updates and make relevant improvements such as extending the detection limit and verifying the performance of the AST system with the revised breakpoints internally, to avoid the problem of uncategorized results.11 In this regard, our study has provided a preliminary examination in the five automated AST systems for future advancement and verification.

Third, based on our study, none of the commercial testing methods met the standards for all the antimicrobial agents tested as per CLSI recommended performance standards for commercial AST systems, when compared to BMD (EA ≥90%, CA ≥90%, VME ≤1.5%, ME ≤3.0%, mE≤10.0%).9 Generally, the TDR system was the least reliable, with significant VMEs in CZO, CAZ, ATM, MEM, AMK, GEN, and TOB, which were well above the acceptable ranges. On the contrary, Vitek 2 performed best among the five systems, with the least incidence of errors except for a high VME rate of 10.0% and 6.2% against FEP and MEM for isolate S2, respectively, which is consistent with previous studies. Lat et al reported a VME rate of 67.0% and 27.0% for FEP and MEM, respectively, for K. pneumoniae carbapenemase (KPC)-producing K. pneumoniae by Vitek 2 system, in comparison with BMD.17 The problem with MEM is not unique to Vitek 2 as the MIC results from Microscan, Phoenix, and TDR systems also tended to be several dilutions lower than those of BMD, resulting in high rates of VME (1.5% vs 4.9% vs 10.0%). The same problem was observed in Phoenix and TDR systems for IPM. Moreover, all systems except Vitek 2 had considerable difficulty in MIC determination of CZO, with high VMEs for both ESBL-producing E. coli and KPC-producing K. pneumoniae. Although CZO has not been previously studied, problems concerning other third-generation cephalosporins such as cefotaxime, CAZ, and CRO have been reported previously,10,13 and thus we speculate that it could be extrapolated to CZO as well.

A possible reason for these discrepancies in susceptibility results among automated systems may be the inoculum size effect, as a smaller inoculum was used in the automated methods compared with the BMD reference method. Several studies with the Vitek 2 system have revealed false susceptibility rates for Enterobacteriaceae, which were suspected to be due to a low inoculum size.11,13,17 This problem has also been reported with the Microscan and Phoenix systems,18,19 leading to the conclusion that low inoculum size has a major influence on the outcome of these systems, with false susceptibilities being reported. As for TDR and DL systems that are mainly used in China, there are limited studies on their performance in susceptibility testing. Therefore, further studies are needed to determine possible reasons for poor performances.

Limitations

Study limitations include possible selection bias as only two species of Enterobacteriaceae with two types of resistant phenotypes were used in the study. And finally, for uniformity, only the latest CLSI 2017 breakpoints were used for data analysis.

Conclusion

In conclusion, our findings reveal substantial discordance in susceptibility results between the tested methods and BMD, with none satisfying the criteria for acceptable AST performance. Each system has inherent advantages and limitations, and the results varied widely by antimicrobial drugs, software versions, and cards used. However, Vitek 2 system seemed to provide a relatively accurate and conservative assessment of MICs for most antimicrobials except FEP and MEM. Standardized MIC testing procedures including QCs, as well as the timely update of the systems in keeping with the CLSI breakpoints, are crucially important as highlighted by this study. Our study suggests that whatever automated AST system is used, laboratories must supplementally use the reference BMD for problematic antimicrobials.