Evaluation of the specificity and sensitivity of a potential rapid influenza screening system.

Influenza remains a serious worldwide health threat with numerous deaths attributed to influenza-related complications. It is likely that transmission of influenza and both the morbidity and mortality of influenza could be reduced if inexpensive but reliable influenza screening assays were more available to the general public or local medical treatment facilities. This report provides the initial evaluation of a pilot system designed by Lucigen Corp. (Middleton, WI, USA) as a potential rapid near point-of-care screening system for influenza A and influenza B. The evaluation of specificity and sensitivity was conducted on stored nasal swab samples collected from emergency department patients presenting with influenza-like symptoms at a large military academic hospital and on de-identified nasal swabs and isolated RNA from a local epidemiology laboratory. The gold standard for assessment of specificity and sensitivity was the Luminex® Respiratory Viral Panel. Published by Elsevier Inc.

Introduction

Seasonal influenza epidemics result in about 3–5 million cases and around 0.25–0.5 million deaths each year worldwide (WHO, 2009). In the United States, up to 300,000 people are hospitalized annually and 10,000–40,000 die from influenza-related complications (CDC, 1999). Diagnosis of influenza A and B by clinical manifestations alone is difficult because of overlapping symptoms from a variety of pathogens including rhinovirus (Arden and Mackay, 2010), coronavirus (Renois et al., 2010), parainfluenza virus (Lau et al., 2005), respiratory syncytial virus (Freymuth et al., 2004), adenovirus (Lina et al., 1996), metapneumovirus (Debur et al., 2010), enterovirus (Lina et al., 1996), and Streptococcus pyogenes (Yamada et al., 2010). Use of antibiotics to treat upper respiratory tract infections without proper diagnosis is a common practice in the United States (Franck and Smith, 2010, Linder et al., 2003). For instance, in a national ambulatory network study of 52,135 upper respiratory tract episodes identified, 65% received antibiotics (Gill et al., 2006), although respiratory viruses are responsible for approximately 80% of respiratory infection (Mahony, 2008). Use of antibiotics, which are of little benefit for viral infections, can play a major role in the development and spread of antibiotic-resistant microorganisms and substantially increase health care costs. Thus, development of a rapid, cost-effective, and accurate screening system for influenza virus available to the general population and/or local physicians and medical treatment facilities would likely reduce the misuse of antibiotic therapy for treatment of influenza infections. Rapid identification of influenza infections would further reduce transmission of virus including nosocomial infections, allow for timely antiviral therapy, and likely reduce the morbidity and mortality of influenza-related complications thereby lowering health care costs.

Currently, there are several methods used to diagnose influenza A and B including viral culture, lateral flow immunoassays (LFIAs), direct fluorescent antibody tests (DFAs), and nucleic acid tests (NATs). Diagnosis of influenza A or B through viral culture is effective but is labor intensive, time consuming, and may require highly trained personnel and several days to provide confirmed results (Balada-Llasat et al., 2011, Ruest et al., 2003). Current LFIA methods for influenza A or B have high specificity (90–100%), are cost effective, and results can be acquired within 15 to 30 min, but, unfortunately, these tests have been shown to exhibit variable sensitivity ranging from as low as 10% up to 100% (Louie et al., 2009). DFA influenza assays demonstrate sensitivities of 60% to 80%, and results can be obtained within 2 h, but require high level of technical proficiency to perform and culture to confirm the results (Cram et al., 1999). NATs offer high sensitivity and specificity for the diagnosis of influenza A or B (Jenny et al., 2010, Louie et al., 2009, Pabbaraju et al., 2008). However, use of NATs in the clinical sector may be prohibited because of high cost and delayed reporting of results as compared to LFIA and DFA. The lack of an affordable, near point-of-care, sensitive, and specific screening system for influenza virus created a void that was addressed by Lucigen Corp. (Middleton, WI, USA) by designing a first-generation system to test the feasibility of a nucleic acid lateral flow system for testing both influenza A and B.

The PyroScript® influenza A and B reagents require the use of a nucleic acid lateral flow (NALF) device for the detection of influenza A or B–amplified RNA molecules. NALF devices work in a manner analogous to lateral flow immunoassays. A visual result can be obtained within 2 h after sample collection allowing rapid interpretation of results without the need of complex or expensive instruments. The PyroScript® influenza screening reagents could be the alternative solution for a rapid influenza A and B test in the clinical sector.

Methods and materials

This research was conducted under an internal review board–approved protocol which allowed for the collection of clinical samples from patients presenting with influenza-like respiratory symptoms in the Emergency Department (ED) of Brooke Army Medical Center after informed consent was obtained. Transfer of de-identified specimens (stored virus or, in the case of H1N1 novel 2009 [swine flu] specimens, isolated viral RNA) from a local epidemiology reference laboratory was also approved. Twenty nasal swab samples collected from ED patients with influenza-like symptoms, 5 archived de-identified Streptococcus pyogenes bacterial specimens, and 75 de-identified specimens transferred from the local epidemiology laboratory were investigated. Although the viral samples obtained from the epidemiology reference laboratory were de-identified (all personal health information and personal identifiers removed), they were supplied with the etiologic agent of each sample identified. The laboratory personnel performing the extractions, Luminex RVP assay, and Lucigen PyroScript tests were blinded to the clinical results until the conclusion of all testing. These samples included 24 novel influenza A H1N1 RNA specimens and 12 influenza A H1N1 seasonal, 12 influenza A H3N2 seasonal, 15 influenza B seasonal; 5 adenovirus, 4 parainfluenza 3 viral specimens, and 3 negative samples (no virus detected). The 5 Streptococcus pyogenes culture–positive throat swab samples were confirmed via polymerase chain reaction (PCR) with specific primers for the ubiquitous proS gene as described before (Livezey et al., 2011) and were included in the study as influenza-negative controls. All samples were stored at −80 °C.

The objective of this preliminary study was to assess the specificity and sensitivity of the influenza A and B NALF systems as a potential rapid screening assay. This system, because of its simplicity and independence from expensive or complicated instrumentation, could be used in near point-of-care settings. Analysis time was considered, but did not include sample preparation time since our samples were stored specimens. For our convenience and reproducibility purposes, as well as for comparison to the Food and Drug Administration approved xTAG® (Luminex Molecular Diagnostics, Toronto, Ontario, Canada) Respiratory Viral Panel (RVP) test, nucleic acid isolation was performed by an automated system (NucliSENS® easyMAG® system, bioMérieux, Marcy l'Etoile, France) as described in Bolotin et al. (2009). The purified nucleic acids obtained were stored at −80 °C.

The gold standard for the specificity and sensitivity assessment was the results of the FDA approved RVP, a nucleic acid test. The RVP test had a sensitivity of 98.2% and a specificity of 96.4% in a study of 247 clinical samples when compared to DFA and culture results (Mahony et al., 2007). All viral isolates were amplified, hybridized, and detected with the Luminex® IS-200 instrument (Luminex, Austin, TX) using the xTAG RVP kit following the vendor's protocol. Briefly, 5 μL of purified nucleic acid from each sample was reverse transcribed in a 25-μL multiplex reverse transcription polymerase chain reaction (RT-PCR) tube. A multiplex target specific primer extension (TSPE) reaction was prepared per sample using 5 μL of treated RT-PCR. After TSPE, 3.5 μL of reaction product of each reaction was added directly to a micro-well containing 20 μL of RVP Luminex bead mixture for hybridization. Phycoerythrin reporter (100 μL) was added to each well containing hybridization product to identify the presence of a virus by assessment of the fluorescence in a Luminex IS-200. Results generated were analyzed using the software component of the kit (TDAS RVP-I). In addition, all novel H1N1 2009 RNA samples were assayed by a microarray technique (ElectraSense® assay, CombiMatrix, Corp., Mukilteo, WA, USA) to positively identify the H1(sw)N1 variant (Straight et al., 2010).

The PyroScript® Influenza reagents (Lucigen) were tested for all samples in this study. One reaction per test was prepared on ice in a 200-μL PCR tube as follows: 6.25 μL nuclease-free water, 12.50 μL PyroScript Isothermal 2× master mix, 1.25 μL 20× Primer Mix, and 5 μL of nucleic acid sample. The reaction was mixed and centrifuged briefly before incubating for 40 min at 72 °C in a thermal cycler (Eppendorf 5417R, Hamburg, Germany) preheated to 72 °C. The reaction was stopped at 4 °C on ice until the 200-μL PCR tube containing specific amplicons was loaded in a cartridge of a Type I BESt™ Cassette (BioHelix Corporation, Beverly, MA, USA) for detection according to the manufacturer's instructions. The influenza A or B–amplified molecules entered the DNA strips via capillary action. When the cassette was closed, the PCR tube released the 6-carboxy fluorescein (FAM) and biotin-labeled amplicons (flu A or flu B reagent) and detected via gold nanoparticle–labeled anti-FAM antibodies. Results were visually read after 10–15 min, photographed, and results recorded.

Results

Each ED sample was routinely tested using a rapid antibody assay (BinaxNOW, Alere, Waltham MA, USA), DFA, and viral culture by the BAMC clinical laboratory (data not shown). All seasonal influenza A samples were previously identified by the local epidemiology laboratory as influenza A H1N1 and H3N2 subtypes, respectively, as per their standard protocol (data not shown). Influenza B samples were similarly identified by the epidemiology laboratory. All 12 influenza A H1N1 seasonal viral samples were confirmed by the Luminex RVP assay as influenza A H1. Similarly, all 12 influenza A H3N2 seasonal viral isolates were confirmed as influenza A H3 by the RVP assay. Two additional samples collected at the BAMC ED were identified by the RVP as influenza A (no-subtype) and were classified as novel H1N1 influenza A, although this was not confirmed by PCR or microarray. Six of the 20 ED samples were identified by the Luminex RVP assay as positive for virus: 2 influenza A no-subtype (as discussed above), 2 rhinovirus, 1 parainfluenza 3, and 1 metapneumovirus. In addition, all 15 influenza B, 5 adenovirus, and 4 parainfluenza 3 samples from the epidemiology laboratory were also confirmed by the Luminex RVP assay. Fourteen of the 20 ED samples, plus the 3 samples from the epidemiology laboratory with no virus detected, and the 5 Streptococcus pyogenes bacterial samples were used as the 22 negative controls for both the influenza A and influenza B calculations (see Table 1 ).

Table 1

Result summary for the Luminex xTAG and PyroScript assays.

xTAG Identification (gold standard)	Number assayed	Positive by PyroScript
		Influenza A	Influenza B
Novel influenza Aa	25	22/25	2/25
Influenza A H1	12	12/12	2/12
Influenza A H3	12	11/12	0/12
Influenza B	15	1/15	14/15
Rhinovirus	2	0/2	0/2
Metapneumovirus	1	0/1	0/1
Adenovirus	5	0/5	0/5
Parainfluenza 3	5	1/5	1/5
No calls	17	1/17	1/17
Nonviral bacterial controls
Streptococcusb	5	0/5	0/5

Novel influenza A: 23 samples were obtained as RNA from a local epidemiology laboratory and detected as unsubtyped influenza A by the RVP assay and H1(sw)N1 variant confirmed by ElectraSense® microarray assay. Two clinical isolates were identified as influenza A (no-subtype) but were not further characterized.

Etiologic agent of Streptococcus in samples verified by PCR.

The PyroScript influenza A and B results and the RVP assay results (gold standard) were compared to estimate the sensitivity and specificity of the new reagents. Of all samples tested, the PyroScript influenza reagents properly identified 22/25 (88.0%) of the novel H1N1 samples, 23/24 (95.8%) of seasonal influenza A H1N1 or H3N2 samples, and 14/15 (93.3%) of the influenza B samples. The PyroScript influenza reagents detected 45/49 confirmed-positive influenza A samples and 14/15 true-positive influenza B samples resulting in 91.8% influenza A sensitivity and 93.3% influenza B sensitivity.

A total of 3 false-positive (positive by the Lucigen system but negative by the gold standard Luminex RVP assay) influenza A and 6 false-positive influenza B results were observed with the PyroScript influenza tests. In the case of influenza A, all samples not identified by the RVP assay as influenza A are considered as true negatives for calculation of the influenza A specificity. Conversely, all samples not identified by the RVP assay as influenza B are considered as true influenza B negatives. Thus, the specificity of the PyroScript influenza reagents are 50/53 (94.3%) for influenza A and 84/90 (93.3%) for influenza B tests.

Discussion

The recent 2009 influenza pandemic caused about 55 million human cases of novel 2009 influenza H1N1 with approximately 246,000 H1N1-related hospitalizations and about 11,160 H1N1-related deaths in the United States alone from April to December of 2009 according to estimates from the Centers for Disease Control and Prevention (CDC, 2010). The development of easy-to-use, cost-effective, and reliable new diagnostic methods accessible to primary health care providers for screening patients is needed for the rapid detection of influenza A and influenza B. Rapid screening for influenza at primary health care facilities would aid in sequestering infected patients from the general population to limit transmission and would reduce the unnecessary use of antibiotics provided to influenza patients misdiagnosed as having bacterial respiratory illnesses. The PyroScript influenza reagents in combination with the cartridge of Type I BESt™ Cassette tested in our study demonstrated excellent outcome parameters with greater than 90% specificity and sensitivity for both influenza A and B tests. Similar isothermal amplification tests in combination with Type I BESt™ Cassette have been developed and used to rapidly detect other pathogenic microorganisms including herpes simplex virus types 1 and 2 (Kim et al., 2011), Mycobacterium tuberculosis (Motre et al., 2011), human immunodeficiency virus (Tang et al., 2010), and toxigenic Clostridium difficile (Chow et al., 2008).

The 24 novel influenza A H1N1 samples were initially identified by the RVP at the epidemiology laboratory as influenza A no-subtype. Novel influenza A H1N1 is detected with the RVP assay as “influenza A no-subtype” due to recognition of the matrix gene, combined with lack of detection of either the seasonal H3 or the H1 target genes. Investigators have reported that during the 2009 influenza season all “influenza A no-subtype” cases detected with the RPV test were most likely novel influenza A H1N1 (Ginocchio and St George, 2009, Vinikoor et al., 2009). All 24 samples were confirmed via RT-PCR as novel influenza A H1N1 (data not shown) prior to being received at BAMC. Of these 24 RNA samples tested at BAMC by the ElectraSense®, 23 were confirmed as influenza A H1(sw)N1 variants (Straight et al., 2010). The novel influenza A H1(sw)N1 RNA sample not detected by this microarray technique was also reevaluated by the Luminex RVP assay, and no viral etiology could be identified. Therefore this sample was eliminated from this study.

Extraction of nucleic acids for the detection of influenza virus with automated robotic systems has been used by other investigators (Belongia et al., 2010, Bolotin et al., 2009, Bose et al., 2009, Yang et al., 2011), as was done in this study using the NucliSENS® easyMAG® system. The easyMAG® system requires at least 1 h to extract nucleic acids from 24 clinical samples including sample preparation time and 10-min viral/bacterial lysis time. Automated nucleic acid extraction systems are excellent instruments for use in reference laboratories where a larger number of samples are processed but are bulky and expensive to use in small laboratories or physician's offices. Alternative nucleic acid extraction methods must be incorporated into future PyroScript influenza screening systems to fulfill the time efficiency requirement of a rapid screening assay. In addition, the requirement to screen for influenza A and B separately with the tested PyroScript influenza system requires 2 reactions with different reagents and 2 detection cassettes. Ideal future versions of the PyroScript influenza screening system should also allow for simultaneous screening of both viruses and might include other respiratory virus targets.

The objective of this study was to determine the sensitivity and specificity of the PyroScript Influenza system. Loss of sensitivity occurs if the test assay does not produce a positive result when the gold standard assay is determined to be positive. The 91.8% influenza A and 93.3% influenza B sensitivity resulted from failure to identify 5 RVP-positive samples as positive by the PyroScript assay. There could be several reasons for the PyroScript not being able to identify some of the true positives. One such possibility could be due to stored RNA deterioration with time. Since insufficient sample remained for additional retesting by the RVP assay, the possibility cannot be excluded that the 3 H1N1 novel, the seasonal influenza A, and the seasonal influenza B samples that appear as false negatives could be from long-term storage–induced RNA degradation and/or freeze–thaw cycles (Botling et al., 2009, Ginocchio et al., 1997). Loss of specificity occurs due to a false-positive result (positive by the PyroScript assay but not the RVP). In this study, a total of 9 false positives were observed. One possibility for these false-positive results could be that the primers for RVP and PyroScript assay are not identical. An additional possibility is that there may have been a mutation in the viral genome where the RVP primers/probes bind which did not affect the isothermal amplification employed in the PyroScript tests. Another possibility could be nonspecific amplification by the PyroScript assay.

In conclusion, the PyroScript influenza reagents in combination with the cartridge of Type I BESt™ Cassette are self-contained disposable detection devices and do not require special instrumentation or complicated specimen handling to obtain a visual result within 2 h after nucleic acid isolation. As such, the PyroScript influenza A and B tested in this study show the potential to fill the screening void for rapid influenza diagnosis. Future development of the PyroScript influenza reagents, screening system, and the implementation of alternative nucleic acid extraction methods to provide a true point-of-care NAT for human influenza diagnostics is warranted.