Comparison of three commercial RT-PCR systems for the detection of respiratory viruses.

BACKGROUND: Due to the insensitivity of rapid tests for respiratory viruses, nucleic acid amplification tests are quickly becoming the standard of care. OBJECTIVES AND STUDY
DESIGN: The performance of the FilmArray Respiratory Panel (RP) and Verigene RV+ (RV+) were compared in a retrospective analysis of 89 clinical specimens previously determined to be positive for the following viruses by our test of record, Prodesse (Pro): influenza A (29, FluA), influenza B (13, FluB), respiratory syncytial virus (12, RSV), human metapneumovirus (10, hMPV), parainfluenza (14, PIV), and adenovirus (10, AdV). Samples positive for influenza A, B or RSV were tested by both methods, while the remainder were tested by RP only. True positives were defined as positive by two or more assays.
RESULTS: Limit of detection (LOD) analyses demonstrated Pro had the lowest LOD for all FluA strains tested, PIV1, PIV2 and AdV; RV+ had the lowest LOD for FluB; and RP had the lowest LOD for RSV, PIV3 and hMPV. Of the 55 samples tested by RV+, all 54 true positive samples were positive by RV+. Of the 89 samples tested by RP, 85 of the 88 true positive samples were positive by RP. From these results, the overall sensitivities for influenza A, B and RSV were 100% and 98% for RV+ and RP, respectively. The overall sensitivity of RP for all viruses was 97%.
CONCLUSIONS: In summary, these systems demonstrated excellent performance. Furthermore, each system has benefits which will ensure they will all have a niche in a clinical laboratory.

Background

Respiratory viruses are major contributors to morbidity and mortality in all age groups [1], [2], [3], [4]. Rapid identification is important for both therapeutic and infection control purposes. Traditional rapid viral diagnoses such as immunoassays produce quick results and are simple to perform; however, have sub-optimal sensitivity [5], [6], [7], [8]. Nucleic acid amplification assays (NAAT), which are relatively rapid and have greatly enhanced sensitivity, are becoming the method of choice [9]. More recently, less complex sample-to-result molecular platforms such as Xpert Flu Assay (Cepheid), RP (BioFire Diagnostics), RV+ (Nanosphere), eSensor Respiratory Viral Panel (GenMark Diagnostic) and Liat Influenza A/B (Iquum) have been introduced into the marketplace which have the potential to provide results in 1–4 h with random access capabilities [10], [11], [12], [13], [14], [15], [16], [17], [18], [19]. The main purpose of this study was to compare the performance of two new automated real-time PCR systems with a proven reliable comparator system [11], [13], [17], [20], [21].

Objectives

We compared the performance of two IVD-cleared rapid RT-PCR tests, RV+ and RP, to results obtained with Pro. Both rapid systems are highly automated and have random access capabilities.

Study design

Clinical samples

Clinical specimens were obtained from 89 patients (age range 45 d–86 yr, median 7 yr, 60% pediatric cases) who had respiratory specimens (90% nasopharyngeal, 8% lower respiratory tract) or stool (adenovirus, 2%) submitted for the detection of respiratory viruses and were previously determined to be positive by Pro. Specimens, collected between January 2008 and February 2012 and stored at −80 °C, were selected based on results from original testing. Samples were paired for differing viruses and tested as two-sample pools. Discrepant samples were retested individually, while samples discrepant for adenovirus were refereed by viral culture as previously described [22]. Specimens were defined as true positive if they were positive by two or more assays.

Viruses

The NATtrol™ RVP Verification Panel (ZeptoMetrix, Buffalo, NY) was used for limit of detection analysis by serial dilution (1:10) in normal human plasma for endpoint detection. Viral nucleic acid concentration determinations were based on quantified control viral RNA (Hologic Gen-Probe Inc., San Diego, CA) using Pro.

RV+

The Verigene® RV+ test (Nanosphere, Northbrook, IL) combines conventional PCR with a gold nanoparticle probes detection system. Initially, Test Cartridges, Extraction Trays, and Amplification Trays were loaded into Processor SP after thawing the Amplification Trays for 10–30 min. Specimen in viral transport (0.2 ml) was added to the extraction tray. Automated sample lysing and target amplification of approximately 10% of the lysate was performed on the Verigene Processor SP as a multiplex assay for the detection of FluA, 09H1, sH3, sH1, FluB, RSV-A and RSV-B. Within 5 min of completion of amplification, the RV+ test cartridge was removed from the processor, opened to release the hybridization slide, which was then inserted into the Verigene reader. Results were reported “detected” or “not detected” for each target, or a message of “No Call” (NC) was reported in the event of assay failure, such as an internal control failure. NC samples were retested to obtain a valid result.

RP

The FilmArray Respiratory Panel v1.6 (BioFire Diagnostics Inc., Salt Lake City, UT), which at the time of the clinical evaluation, was a comprehensive panel for the detection of 15 respiratory viral targets (AdV, coronaviruses [CoV] HKU1 and NL63, FluA, 09H1, sH3, sH1, FluB, hMPV, human rhinovirus/enterovirus (HRV), PIV1–4, and RSV) by real-time PCR. The closed system integrates extraction, amplification, and detection into a simple, hands free test. Briefly, 0.3 ml of specimen in viral transport was added to 0.5 ml sample buffer. The FilmArray pouch was injected with 0.5 ml of hydration solution and 0.3 ml of the diluted sample. The pouch was inserted into the FilmArray and within 70 min results were reported as detected, not detected or invalid for each target. Invalid samples were retested to obtain a valid result. In addition, a result of equivocal for FluA occurred when only one of the two FluA specific targets was amplified.

Pro

The Prodesse assays for respiratory viruses (Hologic|Gen-Probe, San Diego, CA) are a series of either multiplex or uniplex, real-time PCR assays, including ProFlu+ (detects FluA, FluB and RSV), ProFAST+ (differentiation of 09H1, sH3, and sH1), ProAdeno+, ProHMPV+ and ProParaflu+ (differentiates PIV1–3). Briefly, viral RNA was extracted from 0.2 ml of specimen in viral transport, along with a universal internal control, using NucliSens Magnetic Extraction Reagents and the easyMAG extractor (bioMerieux, Durham, NC) and eluted to a volume of 50 μl. RNA extract, 5 μl, was amplified on SmartCyclers (Cepheid, Carlsbad, CA) with the appropriate assay as requested for clinical purposes.

Statistical analysis

Probit analyses for the limit of detection with a 95% probability of detection were performed using SPSS version 8.0 (IBM, Armonk, NY). The chi-square test was performed using Microsoft Excel 2013 (Redmond, WA). P values of 0.05 were considered statistically significant. Confidence intervals (CI) were determined using Graph-Pad (La Jolla, CA).

Results

Limit of detection

Pro had the lowest LOD for all FluA viruses, while the LOD for RV+ was within 0.5 log copies/ml of Pro for the 09H1 and sH1, and more than 1 log greater for sH3 (Table 1 ). RV+ had the lowest LOD for FluB, but RP had the lowest LOD for RSV. The analytical sensitivity was mixed as to whether Pro or RP was more sensitive for PIV, hMPV and AdV.

Table 1

Limit of detection (copies/ml).

Virus	Pro	RP	RV+
FluA 09H1N1	3.68	5.42	4.08
FluA sH1N1	3.72	6.03	4.01
FluA sH3N2	3.29	5.72	4.67
FluB	5.29	4.93	4.22
RSV	4.54	3.42	4.89
PIV1	4.01	4.92	n.a.
PIV2	4.58	5.58	n.a.
PIV3	2.96	2.43	n.a.
hMPV	4.09	3.76	n.a.
AdV	2.55	2.59	n.a.

NATtrol panel members used: adenovirus 1, influenza A H1 Seasonal (A/NEWcAL/20/99; sH1), influenza A H3 Seasonal (A/Brisbane/10/07; sH3), influenza A H1N1 2009 (A/NY/02/2009; 09H1), influenza B (B/Florida/02/06), metapneumovirus 8 (Peru6-2003), parainfluenza 1–3, and respiratory syncytial virus A.

Performance with clinical samples

The determination of clinical performance involved 89 specimens previously positive for respiratory viruses by Pro including one sample originally positive for sH3 that became negative upon archiving (Table 2 ). The 54 samples positive for FluA, FluB or RSV and the negative sample were tested by both RP and RV+ as two sample pools. Initially, 24 sample pools were positive for the expected viruses by RV+, 1 was positive with 3 viruses, and 3 resulted as No Call. The NC pooled samples (6) were retested individually; five additional samples were positive for the expected virus while the sixth was positive for two viruses. The discrepant samples, which involved a pool of a sH1 sample and a sH3 sample additionally positive for 09H1 and a sH3 sample additionally positive for FluB, were retested individually. Upon repeat each sample was positive only for the expected virus.

Table 2

Comparison of RP and RV+ on clinical specimens.

Virus	n	RP					RV
		TP	TN	FP	FN	Invalid	TP	TN	FP	FN	Invalid
09H1N1	15	14		1a	1b		15		1a
sH3N2	12	12					12				1
sH1N1	2	2					2
Flu B	13	13					13		1a		3
RSV	11	11					11				2
hMPV	10	10
PIVR1	5	5
PIVR2	4	4
PIVR3	5	5
AdV	10	9			1
RSV/AdV	1	1c			1c		1
No virus	1		1					1
Total	89	86	1	1	3	0	54	1	2	0	6

Samples/pools positive for multiple viruses.

FluA equivocal.

Positive for RSV but negative for AdV.

When testing the same samples with RP, 51 of the 54 positive specimens were positive for the expected virus. One sample from a patient with dual RSV and AdV infection was only positive for RSV and one 09H1 sample was equivocal for FluA. In addition, one pool was positive for three viruses (an AdV sample and a sH3 sample additionally positive for 09H1). Again, the samples from this discrepant pool were tested individually. Upon repeat each sample was positive only for the expected virus. The additional 33 specimens positive for PIV, hMPV or AdV were tested with RP, which detected the appropriate virus in all but one sample positive for AdV, for a total AdV detection rate of 9/11 AdV positive samples. In addition, AdV was originally detected in all positive specimens by viral culture.

From these findings the sensitivities for FluA, B and RSV were 100% and 98% for RV+ and RP, respectively (95% CI = 92.1–100% and 89.3–100%, respectively). This difference in sensitivity was not statistical significance (two-tailed P value = 0.3129). The overall sensitivity of RP for all viruses was 97% (95% CI = 90.1–99.3%). The initial failure rates for these two assays with clinical samples were 10.7% and 0% for RV+ and RP. However, when including the LOD analysis, the failure rates were 9.7% and 1%, respectively. All failed results resolved with repeat testing.

Workflow and cost analysis

RP and RV+ both offer enhanced workflow, significantly reducing technical time and test-turn-around times as compared with Pro. Both systems are designed for on demand testing. The most cumbersome step for both was the test preparation, including sample addition. RP has an extra step of sample dilution, but the hands-on-time (HOT) was only 5 min (Table 3 ). RV+ had the additional steps of thawing for frozen reagents, loading the cartridges and trays, and one return to the instrument post-amplification for cartridge reading, for a total HOT of 21 min. Pro is more manual, including nucleic acid extraction and amplification reagent preparation, but has higher throughput capabilities. Cost analysis demonstrates that RP is the most expensive per test, however there are multiple targets reportable and billable per test, hence the cost/reportable is considerably lower than other methods.

Table 3

Comparison of labor and reagent costs based on list pricing.

RP			RV+			Pro
Test	NRa	Cost/test	Test	NR	Cost/test	Test	NR	Cost/test
RP	20	$129.00	RV+	6	$85.00	EasyMAG	–	$6.17
						ProFlu+	3	$53.10
						ProFAST+	3	$35.77
						ProHMPV+	1	$35.77
						ProParaflu+	3	$35.77
						ProAdeno+	1	$35.77
Total	20	$129.00	Total	6	$85.00	Total	11	$202.35
Cost/reportable		$6.45	Cost/reportable		$14.17	Cost/reportable		$18.40

Number of reportable.

Hands-on-time.

Test-turnaround-time.

One tenth of per run labor when testing10 specimens.

Five tests analyses for each sample.

Minutes per reportable.

Discussion

Since their introduction in 2011, there have been numerous studies regarding the performance of RV+ [10], [15], [23] and RP [11], [12], [14], [24]; however, only one compared the systems side by side [25] Interestingly, in this study both RP and Pro demonstrated significantly superior sensitivity for the detection of FluA as compared to RV+. Hence, it is surprising that in our study the LOD for all FluA strains tested were much lower with RV+ than RP; and both systems demonstrated excellent sensitivity with clinical specimens. Furthermore, Van Wesenbeeck et al. [25] evaluated the LOD of the different assays based on the virus titers in the clinical samples and were able to determine an LOD for RP and Pro for FluA in the range of 3.5–4.5 log RNA copies/ml, which is similar to our findings for Pro but not for RP. One potential reason for the discrepancy between studies is that all specimens in the Belgium study were collected between February and March 2012 and almost all FluA specimens subtyped as sH3. Hence, RV+ may perform suboptimally with the strain of sH3 circulating in Belgium during the winter of 2012. Our clinical samples would have been representative of the typical genetic diversity seen over a period of 4 years. In addition, these authors reported a higher than typical percentage of invalid results with RV+ (15.8%), which may indicate a system problem or improper handling of reagents which is critical for optimal performance.

Other samples in which the expected virus was not detected were two containing AdV. In our study, the LOD with RP for the virus strain in the NATtrol™ RVP Verification Panel was similar to Pro. However, it has been established that RP version 1.6 had low sensitivity for most group C AdV as well as select serotypes with all other groups [11], [15], whereas the version 1.7 kits demonstrated significant improvements in the range of AdV detected [26]. Interestingly, these two samples were still negative with version 1.7 kits; however, one was now negative with Pro, and the other had a Ct value of more than 37 which is very close to the LOD.

One limitation of these studies is that the clinical performance of Pro cannot be assessed as sample selection was based on Pro test results. In addition, assay specificity cannot be accurately assessed because the population was artificially skewed toward positive specimens rather than the annual respiratory virus positivity rate of 29% that we typically see at our institution. However, false positive results were seen with both RP and RV+. It is possible that the false positive results were an artifact of testing samples in pools. Contamination is another possible source for false positive results. RP is a closed, real-time PCR; hence there is no opportunity for contamination with residual amplicon. Of course, carry over contamination between samples is possible, but unlikely since the samples were all tested individually on one instrument. Although RV+ does involve post-amplification processing, the manufacturer has indicated that amplicons are not released from their cartridges.

In addition to the enhanced workflow, both RP & RV+ have many advantages and disadvantages (Table 4 ). RV+ allows for a relatively simple, highly sensitive option for detecting FluA, B and RSV at a more economical price. But since this system does not detect other respiratory viruses, a second system would be required if a full diagnostic profile is needed. RP has room temperature reagent storage and a total test time of little over an hour, making this system conducive for STAT testing. RP also detects a broad range of organisms, including HRV and CoV. After our studies were performed, the manufacturer added two more strains of CoV, as well as three bacterial targets. Both systems only test one specimen at a time per instrument; hence, multiple instruments are necessary for increased throughput.

Table 4

Advantages and disadvantages of the automated systems.

	Advantage	Disadvantage
RP	Highly automated with little HOTa	Multiple instruments cannot be linked
	TAT ∼ 1 h for STAT testing capabilities	Not as sensitive as other molecular tests
	Low cost per target	High cost per patient
	On demand testing
	Room temp storage for all components
	Subtypes influenza A virus strains
	Detects all respiratory viruses previously detected by culture
	Low percentage of invalid results
	Small foot print (1.94 sq. ft. per system)
RV+	Highly automated with minimal HOT.	Return to instrument required with strict timing
	Multiple instruments can be linked	One-sample/instrument
	Processors are stackable	Post-amplification processing
	Good sensitivity	Split storage conditions for kit components (refrigerated and frozen)
	Subtypes influenza A virus strains	Only detects influenza A, B and RSV
	Small foot print (2.87 sq. ft. per system, 1.21 sq. ft. each additional processer)

Hands-on-time.

Lastly, Pro offers the advantage of high throughput testing and a broad menu with the ability to select different analytes. It is difficult to compare the labor costs for this system with the two on-demand systems as the labor per specimen needed for Pro is highly dependent on the number of specimens tested. None-the-less, all three systems offer excellent sensitivity and each system offers different benefits to fit different laboratory needs. It is apparent that these highly automated assays demonstrate similar performance and greatly reduce the hands-on time.

Funding

We acknowledge the kind generosity of Nanosphere and BioFire Diagnostics Inc., for providing the reagents for this study.

Competing interests

The authors declare no competing interests.

Ethical approval

Obtained from Albany Medical Center.