Evaluation of fast-track diagnostics and TaqMan array card real-time PCR assays for the detection of respiratory pathogens.

Several commercial assays are now available to detect the nucleic acid of multiple respiratory pathogens from a single specimen. Head-to-head comparisons of such assays using a single set of standard specimens provide additional information about key assay parameters such as sensitivity, specificity and lower limits of detection, and help to inform the decision regarding which method to use. We evaluated two real-time PCR platforms: the Fast-track Diagnostics® (FTD) multiplex respiratory panel and a TaqMan array card (TAC) for simultaneous uniplex detection of multiple respiratory pathogens. Two sets of samples were used to evaluate the assays. One set was created by spiking pooled nasal wash or phosphate buffered saline with specified volumes of known concentrations of virus and/or bacteria. Clinical nasal wash specimens from children with lower respiratory tract illness comprised the other set. Thirteen pathogen targets were compared between the two platforms. Testing with a validation panel of spiked samples revealed a sensitivity of 96.1% and 92.9% for the FTD and TAC assays, respectively. Specificity could not be reliably calculated due to a suspected contamination of the sample substrate. Inter-assay agreement was high (> 95%) for most targets. Previously untested clinical specimens tested by both assays revealed a high percent agreement (> 95%) for all except rhinovirus, enterovirus and Streptococcus pneumoniae. Limitations of this evaluation included extraction of the validation samples by two different methods and the evaluation of the assays in different laboratories. However, neither of these factors significantly impacted inter-assay agreement for these sets of samples, and it was demonstrated that both assays could reliably detect clinically relevant concentrations of bacterial and viral pathogens.

Introduction

Background

While traditional microbiological methods continue to play an important role in determining the causes of respiratory tract infections, nucleic acid detection tests are now the diagnostic tools of choice for many respiratory pathogens. These advanced diagnostics offer several advantages over the traditional methods (Murdoch et al., 2012, Murdoch et al., 2010); they may be adapted to include additional targets as needed, can simultaneously detect multiple organisms, have the capacity to detect organisms that would be otherwise difficult to identify due to being fastidious, less viable or present only in very small amounts, can quantify the amount of organism in the sample, and can measure pathogen load (Bhat et al., 2012). Several commercial and in-house assays are now available that detect multiple pathogens, but there are limited published studies of head-to-head comparisons (Sakthivel et al, 2012). Here we present an evaluation of the commercial Fast-track Diagnostics® Respiratory 21 Kit (Fast-track Diagnostics, Luxembourg), a real-time multiplex PCR assay, and a TaqMan array card (TAC, formerly Taqman Low-Density Array or TLDA) (Life Technologies, Carlsbad, CA), a real-time multiple uniplex PCR platform produced by the National Center for Respiratory Diseases, U.S. Centers for Disease Control and Prevention (CDC) (Kodani et al., 2011). The evaluation came about as part of an assessment for a diagnostic platform for the Pneumonia Etiology Research for Child Health (PERCH), a large multi-center, standardized, case control study of the etiology of severe and very severe pneumonia in five African and two Asian countries (Adegbola and Levine, 2011).

Objectives

The purpose of this study was to conduct a head to head evaluation of the Fast-track Diagnostics (FTD) and TAC real time PCR respiratory pathogen assays. We chose to conduct an initial evaluation using a standardized validation panel so that sensitivity and specificity of these assays could be readily compared to a ‘gold standard’; in this case, culture or in-house PCR assays. Because our ultimate goal was to evaluate the assays for their potential use in a large field based epidemiologic study, we also wished to assess their performance in the evaluation of true clinical specimens. Therefore, we also evaluated a set of specimens obtained from children with respiratory illnesses, as described below.

Materials and methods

The FTD assay evaluated here includes 21 viral and bacterial targets multiplexed into six pools (Table 1 , with RSV A/B considered as two targets), though current commercially available versions of the FTD assay can detect up to 33 pathogens. The FTD assay can test up to 12 specimens per 8-hour run and includes internal positive and negative controls as well as an extraction control. The TAC assay evaluated was comprised of 21 uniplex PCR assays that are conducted simultaneously, in duplicate, on a single 384-well microfluidic card with eight ports. One of the ports is dedicated to a negative control and another can be dedicated to a positive control allowing for the testing of 6–7 clinical specimens per 2.5-hour run on a single card. Two internal positive control assays and four nucleic acid extraction control assays are included in the card (Kodani et al., 2011). The FTD platform requires a nucleic acid elution volume of 60 μL divided between the six PCR pools (10 μL template nucleic acid per pool) while the TAC platform requires 20 μL per of 48 PCR reactions (20 μL template nucleic acid added to each port).

Table 1

Pathogen targets included in the FTD and TAC respiratory assays.

	FTD	TAC
Adenovirus	✓	✓
Enterovirus	✓	✓
Human metapneumovirus	✓	✓
Influenza type A	✓	✓
Influenza type B	✓	✓
Parainfluenza type 1	✓	✓
Parainfluenza type 2	✓	✓
Parainfluenza type 3	✓	✓
Parechovirus	✓	✓
Respiratory syncytial virus types A and B	✓	✓
Rhinovirus	✓	✓
Streptococcus pneumoniae	✓	✓
Mycoplasma pneumoniae	✓	✓
Bocavirus	✓
Haemophilus influenzae type b	✓
Human coronavirus NL63	✓
Human coronavirus OC43	✓
Human coronavirus 229E	✓
Parainfluenza type 4	✓
Staphylococcus aureus	✓
Bordetella pertussis		✓
Chlamydophila pneumoniae		✓
Haemophilus influenzae		✓
Influenza type A H1		✓
Influenza type A H3		✓
Legionella pneumophila		✓
Legionella spp.		✓
Streptococcus pyogenes		✓

Two sets of samples were used in our evaluation of the assays: a validation panel containing respiratory pathogens of known concentrations, and a set of clinical specimens that had not undergone any other diagnostic assessment (Fig. 1 ).

Fig. 1

Head-to-head comparison using two sets of samples.

Validation panel

Several identical aliquots of the 81-sample validation panel (Online Supplementary Table 1) were constructed for use in a head-to-head evaluation of the FTD and TAC assays. Samples contained viruses of known quantities (samples 1–27, 37–42), known dilutions (samples 28–36), or both (samples 43–45) spiked in pooled nasal wash from healthy adults who had undergone physical examination demonstrating them to be free of respiratory symptoms and who were otherwise healthy as assessed by physical examination and screening laboratory tests. Sample concentration was confirmed by a combination of the following: use of a turbometer followed by confirmation with colony counts, plaque assays, and TCID50 methods. Five samples of nasal wash from healthy adults were not spiked with viruses and served as negative controls (samples 46–50). Part two of the panel (samples 51–81) included viruses, bacteria, and viral/bacterial mixtures of known quantities (samples 51–54, 62, 72, 73), dilutions (samples 55–61), or both (samples 68–71, 74–81) spiked into phosphate buffered saline (PBS), and an additional five negative controls consisting of PBS alone (samples 63–67). The panel was designed to evaluate the sensitivity of the assays within a range of pathogen concentrations commonly found in clinical specimens, and to assess potential inhibition due to matrix effects in nasal wash specimens. Panel design also took into consideration the limits of detection previously published for TAC (Kodani et al., 2011) as we well as those reported by the Fast Track manufacturer (ranging from 102 to 104  copies/mL, with an average limit of detection of 5.20 × 103  copies/mL). Samples containing combinations of 2 or 3 pathogens were included in the second part of the panel to simulate mixed infections. In total, the panel contained 48 single pathogen samples, 23 mixed pathogen samples, and 10 negative controls.

For the validation panel assessment, FTD was tested at the Johns Hopkins University School of Public Health, Center for Immunization Research (CIR) laboratory, and the TAC evaluation was performed at CDC, Division of Bacterial Diseases. For the FTD evaluation, total nucleic acid extraction was done with the NucliSENS MiniMag (BioMerieux, Marcy l'Etoile, France), eluting 500 μL of specimen into a final volume of 75 μL of nucleic acid. Real-time PCR was performed using an Applied Biosystems 7500® platform (Applied Biosystems, Foster City, CA). Cycling conditions were 50 °C for 15 min, 95 °C for 10 min, and 40 cycles of 95 °C for 8 s followed by 60 °C for 34 s. For the TAC evaluation, total nucleic extraction was done with an InviMag Bacteria DNA panel (Invitek, Germany) and the KingFisher ML extraction platform (Thermo Scientific, Waltham, MA), eluting 200 μL of specimen into a final volume of 120 μL of nucleic acid, and real-time PCR was performed using an Applied Biosystems 7900 HT real-time PCR platform. Cycling conditions were 45 °C for 10 min, 94 °C for 10 min, and 45 cycles of 94 °C for 30 s followed by 60 °C for 1 min. The AgPath-ID one-step master mix and enzyme panel (Applied Biosystems, Foster City, CA) was used for both platforms and an RT step was included for both methods.

Clinical specimens

Assay performance was further evaluated using a set of 149 nasopharyngeal wash specimens obtained from children 12–32 months of age with lower respiratory tract illness (inpatient and outpatient) in the United States between April and December, 2009. The method of nucleic acid extraction was the same for all clinical samples used in this evaluation (using KingFisher method described above, and performed by CDC Division of Bacterial Diseases). Half of the extracted nucleic acid was retained at CDC for TAC evaluation while the rest was sent to the Johns Hopkins University School of Public Health CIR laboratory to be tested by FTD.

For both the analytic and the clinical samples, positive FTD and TAC results were manually analyzed by adjusting the fluorescence threshold for each target. For FTD, a standard Ct cutoff of 35 was used to avoid weak positives. TAC was run and interpreted up to 45 cycles, with no Ct cutoff. If a positive Ct value appeared in at least one of two duplicate TAC wells, the result was considered positive for that target; a negative specimen was defined by the absence of amplification in both wells.

Analytic methods

Sensitivity was calculated for the assays using the validation panel. Concordance between methods was assessed by the Kappa statistic using results from both the validation panel samples and the clinical specimens. For targets with imperfect agreement, McNemar's χ2 test was used to test for independence.

Results

Results of validation panel assessment

Overall, both platforms exhibited high sensitivity for detecting known pathogens in the validation panel. All runs were valid according to the specifications for the negative, positive and extraction controls. The FTD platform detected 100 true positives out of 104 total possible pathogens and TAC detected 92 true positives out of 99 total possible pathogens (totals were adjusted to account for the availability of different targets in the two tests: the FTD platform did not include Legionella targets; the TAC platform did not include Staphylococcus aureus, parainfluenza virus type 4, coronavirus 229E or coronavirus OC43 targets). The overall sensitivity was 96.1% and 92.9% for FTD and TAC, respectively, which is closely aligned to the 100% sensitivity that has been reported for FTD and the 89% overall sensitivity that has been previously reported for TAC (Kodani et al., 2011, Prill et al., 2012, Weinberg et al., 2013).

Detection of single pathogens

Both platforms detected all validation samples containing a single pathogen (N = 48) with two exceptions; TAC was unable to detect the lowest concentration of influenza type A H3N2 (10 TCID50), and neither platform was able to detect enterovirus coxsackie B5 (1:1000 dilution) (Online Supplementary Table 2).

Detection of multiple pathogens

Both platforms were able to detect at least one of the pathogens included in all of the mixed validation samples (N = 23) (Online Supplementary Table 2). However, there were three mixed samples for which one of the pathogens could not be detected by the FTD platform, and five mixed samples for which one of the pathogens could not be detected by TAC. The FTD platform was unable to detect S. aureus (102  cfu/mL) in the presence of influenza type A (1:10 dilution) and unable to detect S. aureus (105  cfu/mL) in the presence of Haemophilus influenzae type B (105  cfu/mL) and rhinovirus (1:100 dilution). S. aureus and H. influenzae type B are in the same FTD multiplex pools; influenza type A and rhinovirus are not in this pool. TAC was unable to detect rhinovirus (1:10,000 dilution) in the presence of Streptococcus pneumoniae (105  cfu/mL), unable to detect S. pneumoniae at (102  cfu/mL) in the presence of either influenza type A (1:10 dilution) or adenovirus (1:10 dilution), and unable to detect H. influenzae type b (102  cfu/mL) in the presence of S. pneumoniae at (102  cfu/mL). Neither platform detected S. pneumoniae (102  cfu/mL) in the presence of H. influenzae (105  cfu/mL).

Specificity was not calculated using the validation panel as the nasal wash fluid used as the substrate for samples 1–45 of the validation panel is suspected to have contained rhinovirus, S. pneumoniae and S. aureus that had been previously undetected by culture. Rhinovirus was detected by TAC in all 45 of the 45 spiked nasal wash specimens and by FTD in 44 of the 45 spiked nasal wash specimens. Because the nasal wash was pooled from several healthy individuals, it is likely that specimens were included from individuals who were colonized by S. pneumoniae and/or S. aureus and who were asymptomatically shedding rhinovirus. The nasal wash negative controls came from a different pool and no rhinovirus was detected in these. The FTD assay detected S. pneumoniae in 41 specimens (there was insufficient quantity to test two of the samples on this target), and TAC detected S. pneumoniae in 31 of the same samples. S. aureus was detected by FTD in all of the spiked nasal wash samples (excluding the two samples with insufficient quantity for testing with this target), as well as in three of the negative nasal wash controls. Of the validation panel samples that were created by spiking PBS instead of nasal wash (i.e. samples 51–81), TAC and FTD each detected one false positive S. pneumoniae in different samples. The FTD platform also detected a false positive parainfluenza type 3 result in a S. pneumoniae and H. influenzae mixed sample.

All positive TAC results were positive in two of two wells apart from S. pneumoniae, which was positive in only one well for 24 of 43 (56%) positive results. The mean Ct value for specimens that were positive in only one well was 36.77 while the mean Ct value for a specimen positive by both wells was 33.49 (p < 0.01). Twelve of the validation panel samples were intentionally spiked with S. pneumoniae. Of these, the seven were present in quantities of 105  cfu/mL and were detected in both TAC wells. The remaining samples were present in quantities of 102  cfu/mL. Of these, two were detected by a single TAC well and three were not detected in either well.

For the comparative analysis we considered only the 13 targets that were included in both the TAC and FTD platforms (Table 2 ). For 12 of the 13 targets, there was nearly perfect agreement. Individual percent agreement for each of those 12 targets was 97% or greater, with kappa values ranging from 0.88 to 1.0. Percent agreement was lower (79.6%) for S. pneumoniae, with a kappa value of 0.59. The McNemar test for paired data indicates that S. pneumoniae is the only target for which the difference in detection between the two platforms was significant. However, when we included only the 31 PBS spiked validation samples, the percent agreement for S. pneumoniae increased to 87.1% and the difference between the two platforms was no longer significant.

Table 2

Comparison of Fast-track PCR and TAC results — validation kit.

Pathogen	Agreement (%)	Kappa value	Kappa 95% CI	McNemar's χ2	McNemar's p-value (exact)
Adenovirus	100.00	1.00	–a	–	–
Enterovirus	100.00	1.00	–	–	–
Human metapneumovirus	100.00	1.00	–	–	–
Influenza A	98.77	0.95	(0.86–1.00)	1.00	1.00
Influenza B	100.00	1.00	–	–	–
Mycoplasma pneumoniae	100.00	1.00	–	–	–
Parainfluenza 1	100.00	1.00	–	–	–
Parainfluenza 2	100.00	1.00	–	–	–
Parainfluenza 3	98.77	0.88	(0.66–1.00)	1.00	1.00
Parechovirus	98.77	0.00	–	–	–
Rhinovirus	97.53	0.95	(0.88–1.00)	0.00	1.00
Respiratory syncytial virus	100.00	1.00	–	–	–
Streptococcus pneumoniae	79.75	0.59	(0.41–0.76)	9.00	< 0.01

Not applicable.

Results of clinical specimen assessment

Among the clinical specimens (N = 149), a total of 163 and 216 positive results were detected by the FTD and TAC assays, respectively. Percent agreement was high (> 90%) for all but three of the pathogen targets (Table 3 ). Of those three (rhinovirus, enterovirus and S. pneumoniae), only rhinovirus and S. pneumoniae showed statistically significant differences. TAC detected 49 rhinovirus positive results whereas the FTD platform detected 18. Similarly, TAC detected 90 positive S. pneumoniae results whereas the FTD platform detected 75 in the same specimen set. The average Ct value of Rhinovirus positive and S. pneumoniae positive results detected by TAC but not by the FTD platform was 28.8 and 32.4, respectively. With the exception of S. pneumoniae and rhinovirus, fewer than 10% of the specimens tested positive for any of the 13 pathogens.

Table 3

Comparison of Fast-track PCR and TAC results — clinical specimens.

Pathogen	FTD		TAC		Agreement (%)	Kappa value	Kappa 95% CI	McNemar's χ2	McNemar's p-value (exact)
	Positive	Negative	Positive	Negative
Adenovirus	3	146	6	143	96.64	0.43	(0.02–0.84)	1.80	0.38
Enterovirus	9	140	12	137	89.93	0.23	(− 0.03 to 0.50)	0.60	0.61
Human metapneumovirus	4	145	5	144	99.33	0.89	(0.66–1.00)	1.00	1.00
Influenza A	12	137	13	136	99.33	0.96	(0.87–1.00)	1.00	1.00
Influenza B	1	148	1	148	100.00	1.00	–	–	–
Mycoplasma pneumoniae⁎	0	149	0	149	–	–	–	–	–
Parainfluenza 1	13	136	12	137	96.64	0.78	(0.60–0.97)	0.20	1.00
Parainfluenza 2⁎	0	149	0	149	–	–	–	–	–
Parainfluenza 3	10	139	13	136	97.99	0.86	(0.70–1.00)	3.00	0.25
Parechovirus	9	140	1	148	94.63	0.19	(− 0.13 to 0.51)	8.00	0.01
Rhinovirus	18	131	49	100	77.85	0.40	(0.25–0.55)	29.12	< 0.001
Respiratory syncytial virus	3	146	7	142	97.32	0.59	(0.23–0.95)	4.00	0.13
Streptococcus pneumoniae	75	74	90	59	88.59	0.77	(0.67–0.87)	13.24	< 0.001

No positive parainfluenza 2 or Mycoplasma pneumoniae results in this specimen set.

Discussion

Overall, both platforms demonstrated very high sensitivity and inter-assay agreement for detecting pathogens of known quantities in a set of spiked validation samples. TAC has been shown elsewhere to be 10-fold less sensitive than individual real-time PCR assays with the same primers and probes (Kodani et al., 2011), perhaps because of the smaller specimen volume in TAC wells compared to typical individual assays. However, the reduced sensitivity was only apparent in our evaluation for the lowest concentration of influenza A H3N2 and for a few pathogens included in mixed infection samples, and therefore this reduced sensitivity may not have much clinical relevance.

Inter-assay agreement was moderate for the S. pneumoniae target when all of the validation samples were taken into consideration, and this may reflect differences in assay design. Because rhinovirus, S. pneumoniae, and S. aureus were frequently detected at low levels in the first 50 samples, it is likely that the nasal wash pooled from healthy adults included specimens from one or more adults who were carriers of these agents. These pathogens were detected more frequently by the FTD assay than by TAC, and when these samples were removed from the analysis the agreement between the two platforms for detecting S. pneumoniae was excellent. Conversely, S. pneumoniae was detected more frequently by TAC in a set of 149 clinical specimens than it was by the FTD assay. The same was true of rhinovirus and enterovirus. The reduced sensitivity of the FTD rhinovirus species primers (particularly B and C strains), has been noted elsewhere (Sakthivel et al., 2012). Since the time of this evaluation, the rhinovirus primers in the Fast-track platform and the rhinovirus targets in the TAC platform have been updated. For all other targets, the inter-assay agreements in this evaluation were very high (> 95%), and were similar to the results of another head-to-head evaluation including the FTD platform (Anderson et al., 2013).

In this evaluation, there are at least two important limitations that may have affected the results. The first was that, due to logistical constraints, two different nucleic acid extraction procedures were used in the assessment of the validation panel. Due to differences in extraction volumes used and the recovery efficiencies of these methods, the concentration of primary specimen to eluent was quite different and may account for some of the differences in pathogens detected. However, it should be noted that a previous evaluation of the extraction methods founds their performance to be comparable (Yang et al., 2010). The other important limitation was that the evaluations were conducted in different laboratories and frozen samples had to be shipped between testing locations.

Conclusions

As a result of this evaluation, key knowledge gaps about the FTD and TAC respiratory platforms were addressed. It was demonstrated that both platforms could reliably detect clinically relevant concentrations of bacterial and viral pathogens spiked in nasal wash or buffer as well as individual pathogens present in mixed samples. Inter-assay agreement did not vary significantly when different nucleic acid extraction methods were used in comparison to the clinical specimen evaluation where nucleic acid extraction was held constant. Most importantly, the head-to-head evaluation described here provided a direct comparison of the two platforms that were unavailable elsewhere, and confirmed that either platform would be appropriate for detecting important pathogens from respiratory specimens. Since the time of this evaluation, both platforms have been adapted to include additional pathogen targets and both have been adopted for use in multi-center field based etiology studies among infants and children. The TAC platform has some operational advantages in that individual reactions can be updated without having to optimize multiplexed panels, there is a low sample volume requirement, and minimal pipetting steps are involved. An advantage of the FTD platform is the ability to use less specialized and more widely available real-time PCR equipment.

Conflict of interest

None to declare.