Comparison of FilmArray Respiratory Panel and laboratory-developed real-time reverse transcription-polymerase chain reaction assays for respiratory virus detection.

The FilmArray Respiratory Panel (Idaho Technology) is a highly multiplexed respiratory virus real-time polymerase chain reaction (PCR) assay. Eighty-four respiratory viruses identified by laboratory-developed real-time reverse transcription-PCR assays (LDA) or by viral cultures were mixed and tested by FilmArray to assess its performance. FilmArray identified 72 (90%) of 80 viruses also detected by LDA. Six of the 8 viruses not detected by FilmArray had PCR cycle threshold values >35. Compared to LDA, FilmArray showed comparable sensitivity when used to test serial dilutions of virus mixtures and good agreement with negative samples. With the use of 1 FilmArray instrument, 7 clinical samples could be analyzed and reported in an 8-h shift compared to 20 using LDA and 1 real-time detection instrument. While the FilmArray was rapid and easy to use, its low throughput and qualitative results may be a disadvantage in some clinical settings.

Introduction

Respiratory virus detection is necessary in the clinical setting and has been used for many years to follow the epidemiology of respiratory tract infections and to elaborate pediatric patient care (Byington et al., 2002, Vega, 2005). Recently, molecular assays such as conventional and real-time polymerase chain reaction (PCR) have been used to reliably and accurately detect respiratory viruses in clinical specimens and their use has resulted in an increase in the detection of respiratory viruses compared to conventional methods (Balada-Llasat et al., 2010, Freymuth et al., 2006, Kuypers et al., 2009, Kuypers et al., 2006]). This increased detection rate has expanded the importance of respiratory virus infections, particularly in severely immunocompromised patients (Kuypers et al., 2009, Peck et al., 2007). It is now clear that highly sensitive and rapid diagnosis of a broad range of respiratory viruses is essential for improving patient management. Many PCR methods that detect all common respiratory viruses in 1 assay, although highly sensitive, are complex and generally require several hours for performance of nucleic acid extraction, PCR amplification, and detection of multiple viral targets.

The Food and Drug Administration (FDA)–cleared FilmArray Respiratory Panel (Idaho Technology, Inc., Salt Lake City, UT) is a rapid and easy-to-use method that includes automated extraction of the specimen and highly multiplexed nested reverse transcription (RT)–PCR followed by melting curve analysis (Poritz et al., 2011). Detection of 12 respiratory viruses can be accomplished in a short turnaround time with minimal hands on time using a single reagent pouch. Laboratory validation of a new, highly multiplexed PCR panel can be laborious, expensive, and unreachable for many clinical laboratories. Standard validation usually includes a head-to-head comparison of unknown samples between one method and a gold standard or an expanded gold standard in the case of molecular assays. The FilmArray has previously been compared to other FDA-cleared and RUO commercial assays xTAG [Luminex Corp., Austin, TX, USA], Prodesse [Hologic Gen-Probe Inc., San Diego, CA, USA], and ResplexII v2.0 [Qiagen Inc., Valencia, CA, USA] with head-to-head comparisons for the more frequently detected viruses (Hayden et al., 2012, Loeffelholz et al., 2011, Rand et al., 2011). However, since respiratory viruses are seasonal and can be completely absent over a year, a large number of samples may need to be prospectively collected in order to validate all the targets in the PCR panel.

In this study, the performance of the FilmArray assay was evaluated using a panel of positive and negative samples, including a variety of less frequently detected viruses, which were initially retrospectively tested using laboratory-developed, multiplexed real-time RT-PCR assays (LDA). The hands-on and overall time and cost required to analyze specimens by each method were also determined.

Materials and methods

Specimen selection

A panel of 34 samples was prepared that contained 84 strains of 13 respiratory viruses (respiratory syncytial virus [RSV], human metapneumovirus [HMPV], influenza virus types A and B [FLU A/FLU B], human parainfluenza virus types 1–4 [HPIV1-4], human adenoviruses [HAdV], human rhinoviruses [HRV], human enteroviruses [HEV], human coronaviruses [HCoV], and human bocavirus [hBoV]) and 6 viruses not included in the respiratory virus panel (herpes simplex virus type 1 [HSV], varicella zoster virus [VZV], cytomegalovirus [CMV], WU polyomavirus [WUPyV], KI polyomavirus [KIPyV], and human parechovirus [hPeV]). The samples were prepared by mixing respiratory specimens (nasal washes, nasal swabs, bronchoalveolar lavages, sputum, and tracheal aspirates) that were previously positive by LDA and typed viral cultures (for HRV, HAdV, and HEV). Viruses were mixed in combinations that included 1 virus with high viral load and 1 with low viral load, estimated using the initial LDA PCR cycle threshold values (Ct). All viruses included in the mixed samples had estimated Ct values between 21 and 37 when tested by LDA. All positive clinical specimens had been collected between January 2008 and July 2011 and stored at −80 °C. Viral cultures were either ATCC (HAdV) or previously serotyped HRV and HEV, and were also stored at −80 °C. Serial 10-fold dilutions of virus mixtures, including a positive control mix containing all 12 respiratory viruses (RSV, FLU A, HPIV3, HMPV, HAdV, HCoV, HRV, hBoV, HPIV2, FLU B, HPIV 4, and HPIV 1) and a positive control mix containing 4 FLU viruses (FLU A subtypes H3, H1 seasonal, H1 2009 pandemic; and FLU B) were used to evaluate the relative sensitivity between the 2 methods. These controls have been used for respiratory virus PCR for a long time in our clinical laboratory; they are well characterized and their expected results are in the low positive range (Ct value of ~30). Clinical specimens negative for respiratory viruses were selected randomly from those collected between January 2011 and June 2011, and were stored at −80 °C. Negative samples tested by FilmArray included 12 prepared by combining 2 previously negative specimens, 1 single negative specimen, and 1 negative human cell control.

FilmArray Respiratory Panel assay

The FilmArray Respiratory Panel assay detects HAdV; hBoV; HCoV subtypes 229E, HKU1, OC43, and NL63; HMPV; FLU A subtypes H1 seasonal, H3 seasonal, and H1 2009 pandemic; FLU B; HPIV1-4; RSV; HRV; HEV; Bordetella pertussis; Chlamydophila pneumoniae; and Mycoplasma pneumoniae. However, HRV and HEV must be reported as indistinguishable since cross-positivity between those viruses is possible with the FilmArray Respiratory Panel assay. HRV and HEV are closely related viruses, which explains the difficulty of many multiplex assays to distinguish adequately those 2 groups of virus. The assay was performed as instructed by the manufacturer. The pouch was first hydrated by injecting 1 mL of water provided with the kit. Then, 300 μL of the sample was loaded in the pouch. The pouch was finally placed into the FilmArray instrument and the run started. The principle of the assay has been previously reported (Loeffelholz et al., 2011, Poritz et al., 2011). The FilmArray Respiratory Panel includes an RNA transcript from the yeast Schizosaccharomyces pombe that is carried through all stages of the test process. Each pouch also includes a DNA fragment dried into the wells of the array with its corresponding primers to more specifically monitor the second step of the nested PCR. Fifty-five mixed or single samples were run by the FilmArray Respiratory Panel assay.

Laboratory-developed real-time RT-PCR assays

Our LDA detect the same 13 respiratory viruses and viral subtypes detected by the FilmArray Respiratory Panel. Protocols have been previously published showing good specificity, reproducibility, and analytical sensitivity of 1000 viral copies/mL of specimen (Kuypers et al., 2005, Kuypers et al., 2006, Kuypers et al., 2007, Kuypers et al., 2009, Renaud et al., 2010, Renaud et al., 2011). HRV and HEV real-time RT-PCR assays did not show cross-positive results from previously tested patient samples [Renaud et al., 2011]. Briefly, all RT-PCR reactions were designed using TaqMan hydrolysis probes, most reactions were multiplexed with 2 or 3 targets, and all reactions were run on an ABI 7500 thermocycler (Life Technologies Corp., Carlsbad, CA, USA). Specimens used in this study were previously tested by the LDA and recorded as positive or negative for specific viruses, including the Ct value. Samples with discordant FilmArray results were repeat tested by the LDA for the discrepant target.

Results

Detection of respiratory viruses in mixed samples

We tested 34 mixed samples that contained 84 strains of 13 respiratory viruses. Two samples had 1 respiratory virus, 17 had 2 viruses, 13 had 3 viruses, 1 had 4 viruses, and 1 had 5 viruses. Viruses undetected by the respiratory panels (WUPyV, KIPyV, hPeV type 1, CMV, HSV, and VZV) were added to 4 samples without interfering with expected results. The FilmArray results were concordant with the initial LDA results for 72 (85.7%) of 84 viruses (Table 1 ). One sample containing FLU A seasonal H3 was equivocal by the FilmArray because the H3 target was positive and the pan-influenza A target was negative. All other 11 viruses with initial discordant results were negative by the FilmArray assay. After retesting the 12 samples that were equivocal or negative in the FilmArray by the LDA, 4 samples were negative (Table 2 ) and 8 were positive for the previously detected virus. After excluding the concordant negative samples, the positive agreement between FilmArray and the LDA was 72 (90%) of 80 viruses. Among the 8 viruses that were not detected by the FilmArray, adenovirus subtypes 2 (Ct = 35.3) and 6 (Ct = 24.9) have been identified by the manufacturer as having lower sensitivity than other respiratory viruses. The 6 remaining discrepant samples (HAdV type 8, FLU A seasonal H3 and pandemic 2009 H1, 2 strains of HMPV, and RSV) had Ct values between 33 and 39 when repeated by LDA.

Table 1

Correlation between expected results from laboratory-developed real-time RT-PCR/viral cultures and FilmArray Respiratory Panel assay.

Expected virus	Number tested	Number FilmArray positive
Adenovirus	11	7
A31	1	1
B3	1	1
B7	1	1
B21	1	1
C2	1	0
C5	1	1
C6	1	0
D8	1	0
E4	1	1
F41	1	1
Untyped	1	0
Bocavirus	1	1
Enterovirus	5	5
Coxsackievirus A16	1	1
Coxsackievirus B3	1	1
Coxsackiesvirus B5	1	1
Echovirus 6	1	1
Echovirus 9	1	1
Human rhinovirus	14	14
A2	1	1
A8	1	1
A30	1	1
A68	1	1
A80	1	1
B3	1	1
B6	1	1
B35	1	1
C	3	3
Untyped	3	3
Influenza B	5	5
Influenza A	14	11
Pandemic H1, 2009	7	6
Seasonal H1, 2008	1	1
Seasonal H3, 2009	6	4
Human metapneumovirus	6	4
Parainfluenza 1	4	3
Parainfluenza 2	4	4
Parainfluenza 3	5	5
Parainfluenza 4	1	1
Respiratory syncytial virus	8	6
A	3	3
B	4	2
Untyped	1	1
Coronavirus	6	6
229E	1	1
HKU1	2	2
NL63	2	2
OC43	1	1
Total	84	72

Table 2

Results of repeat laboratory-developed real-time RT-PCR assays for 12 samples with discrepant results by FilmArray.

Expected virus	Ct value of repeated laboratory-developed real-time RT-PCR	Final sample resolution
Adenovirus C2	35.3	Discordant
Adenovirus C6	24.9	Discordant
Adenovirus D8	33.0	Discordant
Human metapneumovirus	39.0	Discordant
Human metapneumovirus	35.8	Discordant
Influenza A pandemic 2009 H1	37.0	Discordant
Influenza A seasonal H3, 2009	37.2	Discordant
Respiratory syncytial virus B	37.4	Discordant
Adenovirus untyped	Negative	Concordant
Influenza A seasonal H3, 2009	Negative	Concordant
Respiratory syncytial virus B	Negative	Concordant
Parainfluenza 1	Negative	Concordant

Detection of respiratory viruses in dilutions of positive control mixes

A positive control mix that contained 12 respiratory virus targets was tested by both methods at 4 concentrations: undiluted, 1:10, 1:100, and 1:1000. The results are shown in Table 3 . Our LDA detected higher dilutions than did the FilmArray for HMPV, HPIV1, HPIV2, HAdV, HCoV, and HRV, while the FilmArray assay detected higher dilutions of FLU B and HPIV4. A positive control mix containing 5 influenza virus targets was tested by both methods at undiluted, 1:10, and 1:100 concentrations. The results are shown in Table 4 . The pandemic 2009 H1 subtype of FLU A was positive by the LDA and negative by FilmArray in the undiluted and 1:10 control, while the LDA was negative and the FilmArray was positive for FLU A H3 and FLU B at a dilution of 1:100.

Table 3

Results of laboratory-developed real-time RT-PCR assays and FilmArray Respiratory Panel assay for dilutions of the respiratory virus positive mix control.

Virus target	Undiluted		1:10		1:100		1:1000
	Real-time CT value	FilmArray result	Real-time CT value	FilmArray result	Real-time CT value	FilmArray result	Real-time CT value	FilmArray result
RSV	28.3	+	33.6	+	38.1	+	−	−
FLU A	30.4	+	34.5	+	37.8	+	−	−
PIV3	27.7	+	33.3	+	39.3	+	−	−
MPV	30.0	+	36.2	−	−	−	−	−
ADV	23.5	+	28.0	+	32.9	+	34.9	−
CoV	28.1	+	33.4	+	36.8	+	39.3	−
RHV	26.6	+	31.9	+	36.9	+	37.6	−
BCV	28.3	+	32.5	+	36.9	+	−	−
PIV2	28.7	+	34.7	−	38.3	−	−	−
FLU B	30.3	+	34.4	+	−	+	−	−
PIV4	28.5	+	34.7	+	−	+	−	−
PIV1	29.3	+	33.6	−	38.0	−	−	−

Table 4

Results of laboratory-developed real-time RT-PCR assays and FilmArray Respiratory Panel assay for dilutions of the influenza positive mix control.

Virus target	Undiluted		1:10		1:100
	Real-time Ct value	FilmArray result	Real-time Ct value	FilmArray result	Real-time Ct value	FilmArray result
Pan FLU A	30.1	+	33.0	+	36.8	+
FLU A H3	28.9	+	32.1	+	−	+
FLU A H1 seasonal	32.8	+	−	−	−	−
FLU A H1 2009 pandemic	34.8	−	37.8	−	−	−
FLUB	31.9	+	35.7	+	−	+

Evaluation of negative samples

The negative human cell control tested negative for all viruses by both the FilmArray Respiratory Panel assay and the laboratory-developed real-time RT-PCR assays. Twenty-five previously negative samples (12 mixes of 2 samples and 1 sample alone) were tested by the FilmArray Respiratory Panel assay. The FilmArray did not identify any respiratory viruses in these samples. However, it did identify 1 sample that was positive for M. pneumoniae. When the 2 specimens (both nasopharyngeal secretions) that were mixed together in that sample were tested separately with a laboratory-developed real-time PCR for M. pneumoniae, 1 sample was positive for M. pneumoniae with a Ct value of 31.4.

Required time and reagent costs

Testing 1 sample by the FilmArray Respiratory Panel required about 10 min of hands-on time to load the buffer, sample, and pouch into the instrument; to retrieve results; and to clean up. Results were available approximately 1 hour after loading the pouch. In an 8-h shift, 7 samples, which required 60 to 70 min of total hands-on time, could be run and reported by 1 technician. In comparison, automated nucleic acid extraction and real-time RT-PCR for 12 respiratory viruses (excluding HEV) could be performed on 20 clinical samples by 1 technician in an 8-h shift using 1 real-time instrument, with approximately 200 min of total hands-on time. The cost of a FilmArray instrument is comparable to that of a real-time sequence detection system. The cost of each reagent pouch was approximately 6-fold higher than the cost per sample of the reagents needed to perform the laboratory-developed multiplexed real-time RT-PCR assays for detection of 12 respiratory viruses.

Discussion

Highly multiplexed respiratory virus PCR panels such as the FilmArray Respiratory Panel are laborious and expensive to validate for use in clinical laboratories. Previous studies have demonstrated good correlation between the FilmArray assay and other respiratory virus detection methods including direct fluorescence antibody (DFA), xTAG, Prodesse, and Resplex II v2.0 (Hayden et al., 2012, Loeffelholz et al., 2011, Poritz et al., 2011, Rand et al., 2011). We designed a simplified validation strategy for the FilmArray Respiratory Panel in our laboratory using a selected panel of clinical samples previously tested by our well-characterized and validated LDA, which have been used in our clinical laboratory for more than 5 years. Additional viruses were selected from ATCC cultures for HAdV and from previously serotyped cultures for HRV and HEV, creating a broad range of viral subtypes to test by the FilmArray assay. Selection of HAdV, HRV, and HEV subtypes was based on the most frequently identified types and on the FilmArray's published sensitivity (FilmArray Respiratory Panel (RP) Instruction Booklet, 2011). This is the first time that specific HRV and HEV subtypes from various groups have been tested with the FilmArray assay. By mixing clinical specimens and viral cultures, we were able to test for 80 respiratory viral strains by the FilmArray using a minimal number of reagent pouches. Among the 59 pouches available for the evaluation, 3 could not be used owing to loss of vacuum and 1 had a software error at the end of the run, resulting in 55 pouches providing results and 1 sample lost. Our assay validation approach is complementary to the previous head-to-head comparisons of clinical specimens (Hayden et al., 2012, Loeffelholz et al., 2011, Rand et al., 2011) and allowed us to analyze samples with known viral loads and a large variety of strains. However, examination of unselected samples collected directly from patients remains the best evaluation method because it is closer to a real-life situation, although more expensive to perform.

The sensitivity of the FilmArray Respiratory Panel assay was comparable to our LDA for detection of 12 respiratory viruses in clinical samples and in dilutions of positive control mixes. In the analysis of 34 positive mixed clinical specimens, the FilmArray had results similar to those in another study comparing FilmArray with real-time PCR assays, identifying 90% of the 80 viruses detected by our LDA (Pierce et al., 2012). Six of 8 missed viruses had higher Ct values (>33) corresponding to lower viral loads. For all samples tested, agreement between the 2 methods was seen for all viruses with PCR Ct values ≤35 except for detection by real-time PCR and not by FilmArray of FLU A pandemic 2009 H1 with a Ct of 34.8, HPIV1 with a Ct of 33.6, HPIV2 with a Ct of 34.7, and HAdV subtypes 3, 6, and 8 with Ct values of 34.9, 24.9, and 33.0, respectively. In addition, our LDA detected RSV, HMPV, FLU A, HAdV, HRV, and HCoV strains with Ct values ranging from 35.3 to 39.3 that were not detected by FilmArray. However, FilmArray detected FLU A seasonal H3, FLU B, and PIV4 strains in samples with estimated Ct values ranging from 35.6 to 39.2 that were negative by real-time RT-PCR. Interestingly, none of the HRV subtypes tested cross-reacted with the HEV primers and probes in the FilmArray, but the 5 HEV subtypes showed cross-reactivity with the HRV primers and probes, which confirms the inability of the FilmArray assay to reliably differentiate the 2 groups. No other concerns about specificity were observed during the validation. Two missed HAdV subtypes, 2 and 6, are known to amplify in the FilmArray assay with reduced efficiency resulting in respective limits of detection 100 times and 10,000 times over the limit of detection for the other types. HMPV was not detected by FilmArray in 2 clinical samples, including 1 with a Ct value of 35.8. FilmArray did not detect HMPV in the 1:10 dilution of the control mix either, which had a PCR Ct of 36.2. Another study comparing FilmArray to DFA also showed a lower agreement for HMPV between the 2 methods with 2 of 6 HMPV-positive samples detected only by DFA (Poritz et al., 2011). The same study also showed a low agreement for HPIV1 with 5 of 11 positive samples detected only by DFA (Poritz et al., 2011). It is important to mention that the relative sensitivity of FilmArray was evaluated in our study using controls that contained 12 respiratory virus targets in the same sample and 5 influenza typing targets in the same sample. Amplification of numerous targets (over 5) within the same sample is uncommon and could have affected the sensitivity. Real-time PCR reactions contained primers and probes for 2 to 4 targets within each well, even though all 12 targets for respiratory panel and 5 targets for influenza panel were present in the samples.

In conclusion, the FilmArray Respiratory Panel performed comparably to LDA, including testing of samples with multiple viruses present at different concentrations. Advantages of the assay include its ease of performance with only minimal hands-on time, its small footprint in the laboratory, and its fast turnaround time (Table 5 ). The assay can be performed after very little training, and the instrument requires no calibration or maintenance. The FilmArray is also able to identify bacterial pathogens. Currently, the assay is only approved for testing nasal swab specimens. Laboratories testing other types of respiratory specimens would need to perform in-house validation. Other disadvantages include the assay's known decreased sensitivity for some HAdV types and its inability to differentiate some HEV and HRV types. In addition, the low throughput (1 sample per hour) and the qualitative results are characteristics that could limit its utility in some clinical laboratories, especially those in which quantitative results for respiratory virus detection in standardized nasal washes are used to follow respiratory virus infections in immunocompromised patients. As more data become available, quantitative or semiquantitative results may prove to be essential for interpreting from a clinical standpoint a positive molecular assay result for respiratory viruses, since merely the presence of a virus in a clinical sample may not have any significant implications. When a rapid result would impact patient care, the use of the FilmArray may be justified, in spite of the higher reagent costs. In other situations, use of multiplexed real-time RT-PCR assays can provide lower cost, higher throughput, and semiquantitative results.

Table 5

Comparison of selected characteristics of direct fluorescent antibody, laboratory developed real-time PCR, and FDA-approved respiratory panel assays.

Characteristic	DFAa	LDAb	XTAG	Prodesse	FilmArray
Rapid turnaround time for 1 sample	+++	++	+	++	+++
Ease of use	+	++	++	++	+++
Throughput	++	+++	+++	+++	+
Small laboratory footprint	+++	++	+	++	+++
Low cost	+++	++	+	+	+
Sensitivity	+	+++	+++	+++	+++
Quantitative result	+	+++	+	+++	+

DFA = Direct fluorescent antibody.

LDA = Laboratory developed assay.