Diagnostic performance of influenza viruses and RSV rapid antigen detection tests in children in tertiary care.

BACKGROUND: Rapid antigen detection tests (RADTs) are increasingly used to detect influenza viruses and respiratory syncytial virus (RSV). However, their sensitivity and specificity are a matter of debate, challenging their clinical usefulness.
OBJECTIVES: Comparing diagnostic performances of BinaxNow Influenza AB(®) (BNI) and BinaxNow RSV(®) (BNR), to those of real-time reverse transcriptase PCR (RT-PCR), virus isolation and direct immunofluorescence (D-IF) in paediatric patients. STUDY
DESIGN: Between November 2005 and September 2013, 521 nasal washings from symptomatic children (age <5 years) attending our tertiary care centre were tested, with a combination of the respective assays using RT-PCR as gold standard.
RESULTS: Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of BNI were 69% (confidence interval [CI] [51-83]), 96% [94-97], 55% [39-70] and 98% [96-99] respectively. Of eleven false-negative samples, RT-PCR Ct-values were higher than all RT-PCR positive test results (27 vs 22, p=0.012). Of twenty false-positive samples, none were culture positive and two tested positive in D-IF. Sensitivity, specificity, PPV and NPV for BNR were 79% [73-85], 98% [96-99], 97% [93-99] and 88% [84-91]. Of the 42 false-negative samples the median Ct-value was higher than that of all RT-PCR positive samples (31 vs 23, p<0.0001). Five false-positive samples were detected. Three of these tested positive for RSV in virus isolation and D-IF.
CONCLUSIONS: RADTs have a high specificity with BNR being superior to BNI. However, their relative low sensitivity limits their usefulness for clinical decision making in a tertiary care paediatric hospital.

Background

Influenza viruses and respiratory syncytial viruses (RSV) cause acute respiratory tract infections (ARTIs) in children, being a leading cause of hospitalization [1], [2], [3]. Identification of both viruses is important for disease management, as the presence of these infections may require specific treatment (i.e. oseltamivir) and hospital containment measures. The current gold standard for detection of these viruses is real-time reverse transcriptase PCR (RT-PCR) [4]. This is however not performed in all hospitals, as it requires a molecular diagnostic laboratory with specialized personnel and equipment. Instead, rapid antigen detection tests (RADTs) are used as these assays are easier and cheaper to perform and less time-consuming [5], [6], [7]. The performance of these tests depends on factors like time between disease onset and sampling, quality and type of specimen and epidemiological parameters [8]. Diagnostic value and clinical usefulness of RADTs for influenza diagnosis vary greatly [5], [6], [7], [9], [10], [11], [12]. This prompted us to evaluate the diagnostic performance of the routinely used RADTs (manufactured by Alere BinaxNOW®) for these two viruses as used in our tertiary care paediatric hospital.

Objectives

Comparing diagnostic performances of two RADTs, BinaxNow Influenza AB® (BNI) and BinaxNow RSV® (BNR), with those of RT-PCR in samples of paediatric patients attending our tertiary care centre with ARTIs for a period of almost eight consecutive years. Discrepant data were subsequently compared with those of virus isolation and direct immunofluorescence (D-IF) assays.

Study design

This study was conducted from November 2005 through September 2013, we identified paediatric patients between 0 and 5 years who attended Erasmus MC-Sophia’s emergency department, out-patient-clinic and those who were hospitalized in this period. To analyse the performance of the BNI and BNR compared to RT-PCR we selected 521 nasal washings of 489 patients with a median age of 4 months (minimum 0.03–maximum 58 months, lower interquartile range 1.6–upper interquartile range 9.8) and 55% (268/489) were male. Nasal washings were obtained during routine clinical practice in symptomatic children and were tested immediately after sampling by trained laboratory personnel using all four diagnostic methods. Multiple samples from the same patient were included in our analysis. Therefore patients are referred to as cases. Data regarding gender, age and hospital admission were obtained from the electronic patient files.

Ethics

Data collection and analyses were conducted on anonymized samples, which does not require further medical ethics review as consented by our Medical ethical board (MEC-2015-306).

Tests

RT-PCR gold standard

All nasal washings were tested for the presence of selected viruses by means of RT-PCR with primers and probes sets used in the routine setting of our department [13]. In short, RNA and DNA were extracted using MagnaPureLC (Roche Diagnostics, Almere, the Netherlands) and the total nucleic acid isolation kit. The extractions were internally controlled by addition of a known concentration of phocine distemper virus (PDV) and phocine herpes virus (PHV). Uni-plex RT-PCR was used to detect RSV-A, RSV-B, human rhinovirus (HRV), parainfluenza virus (PIV) type 3 (PIV-3), adenovirus (ADV), and human bocavirus (HBoV). Duplex reactions were performed combining influenza A virus and PDV, influenza B virus and human coronavirus (HCoV) OC43 (HCoVOC43), human metapneumovirus (HMPV) and PIV-2, HCoV229E and PIV-4, and HCoVNL63 and PIV-1. A cycle threshold value (Ct-value) of <40 was defined positive for any virus. RT-PCRs were developed in-house for influenza viruses and RSV-A and validated [13]. RSV-B primers and probes were used as reported by Dewhurst-Maridor et al. [14].

Rapid antigen detection tests (RADTs)

Alere BinaxNOW® Influenza A and B (BNI) and Alere BinaxNOW® RSV (BNR) (Scarborough, Maine, USA) are commercially available in vitro immunochromatographic assays for the qualitative detection with monoclonal antibodies directed against influenza A and B virus nucleoproteins and RSV fusion protein antigen, respectively. Nasal washings were obtained using standard protocols and rapid antigen testing was performed as described by the manufacturer. For our analyses the test results of BNI influenza A and influenza B were combined into a single influenza BNI dataset as influenza B was not encountered frequently with only four influenza B BNI positive samples, two of which were influenza B RT-PCR positive.

Virus isolation assay

Virus isolation assays were always performed in combination with D-IF. Madin-Darby Canine Kidney (MDCK) cell line (NBL-2) (ATCC® CCL-34™) and the human cell line HEp-2 (ATCC® CCL-23™) were used to isolate influenza viruses and RSV respectively. Virus cultures were regularly checked for cytopathic effect by light microscopy. Immunofluorescence with fluorescein isothiocyanate (FITC) labeled monoclonal antibodies was used to confirm the presence of influenza virus or RSV [15].

Direct immunofluorescence (D-IF) assays in clinical specimens

Cells were isolated from nasal washings, dried on microscope slides, and fixed with acetone. Subsequently, cells were stained with FITC conjugated monoclonal antibodies against influenza A virus, influenza B virus or RSV (IMAGEN™ Influenza A and B and IMAGEN™ RSV, Hampshire, United Kingdom). Specimens were incubated with FITC-conjugated antibodies for 15 min at 37 °C, subsequently excess reagent was washed off with phosphate buffered saline. The stained area was then mounted and viewed by fluorescent microscopy.

Comparison between tests

The focus of our study was to compare data obtained with two RADTs, BNI and BNR with those obtained by RT-PCR as gold standard. We defined false-negative tests as those for which the rapid test was negative and the gold standard RT-PCR positive; a false-positive test result was defined if the rapid test tested positive and the gold standard RT-PCR tested negative. We compared the available Ct-values in all respective categories of samples and analysed whether there was an association between Ct-values and RADTs results and hospitalization. For influenza all Ct-values were available, for RSV Ct-values were available for 183/204 (90%) of the performed tests. Missing Ct-values were from samples tested in 2005 and 2006 when routine input of Ct-values in our laboratory system was not yet performed and digital documentation was not available. Finally, false-negative and false-positive test results were compared to test results obtained with the other virus detection methods: virus isolation and D-IF assays.

Statistical analyses

The main outcomes of this study were the sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of the BNI and BNR rapid test results compared to RT-PCR during the total study period and during viral season (October 1st through March 31st). Ct-values were compared with Mann-Whitney U tests.

Results

Sensitivity and specificity of BNI

Of 521 nasal washings both influenza RT-PCR and BNI data were available. Most were obtained between September and March (Supplemental data Fig. S1, Fig. S2). Of these, 35 cases tested positive with RT-PCR (35/521, 7%, median Ct-value 22 [range] [17-39]) whereas 44 tested positive in the BNI (44/521, 8%). Of the 35 RT-PCR positive cases 24 also tested positive in the BNI (24/35, 69%, median Ct-value 21 [17-31]). The eleven RT-PCR positive (median Ct-value 27 [18-39]) and BNI negative cases were considered false-negative cases (11/521, 2%). Of the 486 RT-PCR negative cases, 20 were BNI positive, and were therefore considered false-positive cases (20/521, 4%). Considering RT-PCR as the gold standard, it may be concluded that the BNI has a relatively low sensitivity of 69% (confidence interval (CI): [51-83]) (24/35), a high specificity of 96% [CI: 94–97] (466/486), a low PPV of 55% [CI: 39–70] (24/44) and a high NPV of 98% [CI: 96–99] (466/477) (Table 1). We also calculated these parameters only with samples obtained in the period from October 1st through March 31st, when respiratory viruses are more prevalent in The Netherlands. Sensitivity and specificity decreased with 2% and 1% respectively (69%–67% and 96%–95%). PPV and NPV both decreased with 1% from 55% to 54% and 98%–97% respectively.

Table 1

Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of rapid antigen detection test BinaxNOW influenza AB® and BinaxNOW RSV® compared to gold standard RT-PCR tested in nasal washings of children between 0 and 5 years at Erasmus MC-Sophia from 2005 to 2013.

Rapid antigen detection tests	Sensitivity (%) [95% confidence interval; CI] n	Specificity (%) [95% CI] n	PPV (%) [95% CI] n	NPV (%) [95% CI] n
BinaxNow Influenza AB® vs RT-PCR n = 521	68.6 [51–83] 24/35	95.8 [94–97] 466/486	54.5 [39–70] 24/44	97.7 [96–99] 466/477
BinaxNow RSV® vs RT-PCR n = 514	79.4 [73–85] 162/204	98.4 [96–99] 305/310	97 [93–99] 162/167	88 [84–91] 305/347
Samples obtained in respiratory virus season
BinaxNow Influenza AB® vs RT-PCR n = 436	66.7 [48–82] 22/33	95.3 [93–97] 384/403	53.7 [37–69] 22/41	97.2 [95–99] 384/395
BinaxNow RSV® vs RT-PCR n = 428	80 [74–85] 160/200	98 [96–100] 224/228	98 [94–99] 160/164	85 [80–89] 224/264

Fig. S1

. Number of all BinaxNOW influenza AB® (BNI) and BinaxNOW RSV® (BNR) rapid tests performed and all positive test results for BNI and BNR in children between 0 and 5 years at Erasmus MC-Sophia from 2005 to 2013.

Fig. S2

.Number of all BinaxNOW influenza AB® (BNI) and BinaxNOW RSV® (BNR) rapid tests performed and all positive test results for BNI and BNR in children between 0 and 5 years at Erasmus MC-Sophia from 2005 to 2013 in months per years.

Sensitivity and specificity of BNR

Of 514 nasal washings both RSV RT-PCR and BNR data were available. Of these, 204 cases were RSV RT-PCR positive (204/514, 40%) with Ct-values available for 183 samples ranging from 14 to 39 (median Ct-value 23) and 167 were BNR positive (167/514, 32%) (Table 1). Hundred sixty-two samples were RT-PCR positive and BNR positive (162/514, 32%, median Ct-value 21 [14-35], no Ct-value available n = 15). Forty-two cases were considered false-negative (42/514, 8%, median Ct-value 31 [22-39], no Ct-value available n = 6). Of the 310 RT-PCR negative cases, five were BNR positive and considered false-positive cases (5/514, 1%). The overall test performance of BNR was relatively high with a sensitivity of 79% [CI: 73–85] (162/204), specificity of 98% [CI: 96–99] (305/310), PPV of 97% [CI: 93–99] (162/167) and NPV of 88% [CI: 84–91] (305/347). We also calculated these parameters during the respiratory virus season (October–March) and sensitivity increased with 1% (79%–80%), but the specificity remained the same (98%). PPV increased with 1% (from 97% to 98%) and NPV decreased with 3% from 88% to 85% respectively.

Discordant samples

False-negative rapid antigen detection tests

From the eleven false-negative BNI cases, influenza virus was successfully isolated in six cases (6/11, 55%, median Ct-value 25 [17-27]), three of which were also influenza D-IF positive. By means of RT-PCR, virus isolation or D-IF 7/11 (63%) samples tested positive for another virus, most frequently RSV (n = 3) or adenovirus ADV (n = 3) (Supplemental Table 1a).

Table 1a

.

BNR results were considered false-negative in 42 cases, in 25 (60%) of those RSV was cultured successfully. In these 25 cases the Ct-values ranged from 22 to 39 with a median of 32. In addition, in 20/42 (48%) cases, RSV D-IF tested positive (median Ct-value 29 [22-37]). Co-infections were found in 16/42 (38%) cases and most often HRV (n = 7). In four cases BNI tested positive for influenza, which could be confirmed for three samples by RT-PCR (Supplemental Table 1b).

Table 1b

.

False-positive rapid antigen detection tests

Of the 20 influenza false-positive cases, six tested positive in BNR (6/20, 30%). Moreover, six cases tested RSV RT-PCR positive (6/20, 30%) of which five were also BNR positive. In eight samples another respiratory virus than influenza virus or RSV was detected with RT-PCR (8/20, 40%). For two samples the D-IF was positive for influenza, in accordance with the BNI, but for both virus isolation did not yield influenza virus (Supplemental Table 2a).

Table 2a

.

For BNR only five false-positive cases were found. Three of these were RSV positive in virus isolation and D-IF (3/5, 60%). One case tested negative in all methods except for BNI (Supplemental Table 2b). The sensitivity (80%), specificity (99%) and PPV (99%) increased if we considered the three RSV positive virus isolations as true-positive cases, resulting in only two false-positive cases (2/514, 0,4%).

Table 2b

.

Test results and hospitalization

Of all 521 patients tested for BNI and BNR 361 patients (361/521, 69%) were hospitalized. Hospitalization rates were 16/24 (67%) and 115/162 (71%) for true-positive BNI and BNR cases respectively. False-negative test results did not seem to have a major impact on hospitalization with hospitalization rates of 7/11 (64% vs 69%, p = 1) and 25/42 (60% vs 71%, p = 0.1912) for BNI and BNR test results respectively (Table 2 ).

Table 2

BinaxNow Influenza AB® (BNI) and BinaxNOW RSV® (BNR) test results in relation to hospitalization of children between 0 and 5 years at Erasmus MC-Sophia from 2005 to 2013.

BNI	True-positive (%)	False-negative (%)	False-positive (%)
Admitted	16/24 (67)	7/11 (64)	16/20 (80)
Not admitted	8/24 (33)	4/11 (36)	2/20 (10)
Unknown	–	–	2/20 (10)
BNR
Admitted	115/162 (71)	25/42 (60)	5/5 (100)
Not admitted	47/162 (29)	17/42 (40)	–

Comparing the Ct-values of the respective categories, BNI and BNR cases that were false-negative displayed overall higher Ct-values (p = 0.012 vs p < 0.0001) (Fig. 1 A and B). However, no differences were found in Ct-values of hospitalized and non-hospitalized patients within the respective case groups (p > 0.5 for both BNI and BNR) (Fig. 2 A and B). The BNI test result did not differentiate for severe disease with three of the false-negative cases admitted to the paediatric intensive care unit (PICU), but also three false-positive cases. For BNR six patients were admitted to the PICU (6/42, 14%) despite a false-negative test result. None of the five BNR false-positive tested patients were admitted to the PICU (0/5).

Fig. 1

(A) Ct-values compared for BinaxNOW influenza AB® (BNI) rapid test positive results and rapid test negative results for influenza in children between 0 and 5 years at Erasmus MC-Sophia from 2005 to 2013. (B) Ct-values compared for BinaxNOW RSV® (BNR) rapid test positive results and rapid test negative results for RSV in children between 0 and 5 years at Erasmus MC-Sophia from 2005 to 2013. The number between () refers to all RT-PCR positive results, for 21 samples no Ct-values were available.

Fig. 2

(A) Ct-values in relation to BinaxNOW influenza AB® (BNI) rapid test results and hospitalization of children between 0 and 5 years at Erasmus MC-Sophia from 2005 to 2013. ns = no statistical significant difference. (B) Ct-values in relation to BinaxNOW RSV® (BNR) rapid test results and hospitalization of children between 0 and 5 years at Erasmus MC-Sophia from 2005 to 2013. ns = no statistical significant difference.

Discussion

The present study evaluated the diagnostic performance of BNI and BNR RADTs in a large number of symptomatic paediatric patients between 0 and 5 years attending our tertiary care paediatric hospital during almost eight consecutive years. By testing fresh nasal washings with RT-PCR, we found a relatively low sensitivity and PPV for BNI. The overall test performance of BNR scored higher for all these aspects. Both BNI and BNR false-negative cases displayed a significantly higher Ct-value compared to all RT-PCR positive and true-positive tested cases.

The accuracy of rapid tests is generally less than that of RT-PCR and virus isolation assays. However, RADTs are valuable as a point-of-care test for their ease of use, fast results and laboratory independence [8]. These advantages, and especially the high specificity are important for their use as surveillance tools for influenza outbreaks as recommended by the World Health Organization [8]. Indeed for surveillance purpose a high specificity is of importance, which we found to be the case in our study. However, for clinical management this is not sufficient. In theory, the decision to start antiviral therapy and to refrain from unnecessary further diagnostic testing and antibiotic use may be based on RADT [16], [17], [18]. In addition, rapid test results may result in more effective isolation and containment measures [6]. For this purpose assay sensitivity is of utmost importance. Therefore, we conclude that the relatively low sensitivity of the BNI in our tertiary care centre is worrisome. Of note, BNR test performance proved to be better compared to the BNI, although false-negative cases were detected. Based on our results we stopped using BNI and will use rapid PCR-based tests for detection of influenza virus and RSV.

We considered the clinical implications of a false-negative and false-positive test result in relation to hospitalization and found no significant differences between these groups and RT-PCR positive and BNI positive tested cases, indicating that the clinical observation is still pivotal in admission decision-making as suggested by current guidelines. In the present study we were not able to study effects of testing on treatment since antiviral medication was not routinely used in our hospital before 2009. Data on isolation and containment measures were not available.

Although the retrospective nature of our study has inherent limitations, we were able to include more than 500 samples of patients over time. Since, our study spanned a considerably longer period of time than previous studies, we were able to test the clinical feasibility of RADTs “in a real life” clinical setting and for different circulating virus subtypes during eight consecutive seasons. Overall, most studies reported a high specificity and high NPV versus a low sensitivity and low PPV for BNI, which is largely in agreement with our results [5], [6], [7], [9], [10], [11], [12]. We did find a relatively high sensitivity of 79% and an even higher PPV (97%) for BNR, which is in agreement with data obtained in other studies [5], [19], [20].

During the last decade, RT-PCR has become the gold standard for detecting the presence of respiratory viruses [4]. The downside of RT-PCR is the relatively long time (6–24 h) between sample collection and availability of test results [21]. This makes current RT-PCR formats less useful for admission decision-making and calls for faster methods. Indeed, new rapid point-of-care PCRs are being developed and implemented with a shorter turnaround time [22], [23], [24], [25], [26], [27]. Studies comparing their performances with those of RT-PCR would allow us to judge their potential for clinical decision-making.

In conclusion, we evaluated the performance of the RADTs BNI and BNR in a tertiary care paediatric hospital setting over eight consecutive years. We showed that sensitivity and PPV of BNI were relatively low (69% and 55%), whereas those of BNR were higher (79% and 97%) when compared to the respective gold standard RT-PCR. False-negative samples consistently displayed high Ct-values, although this did not influence whether patients were hospitalized or not. Given the relatively low sensitivity and PPV of the BNI we strongly advocate a restricted use of BNI or similar rapid influenza antigen detection assays in a tertiary paediatric care setting. In contrast, the higher sensitivity and PPV of the BNR rendered this rapid test more useful, albeit still less sensitive than RT-PCR.

Conflict of interest

PF participates in the IRIS trial, sponsored by Hoffmann–La Roche.

Funding

PF, MK and AO received funding from the EU FP7 project PREPARE (602525) for studies on the pathogenesis of viruses in children. PF and AO received funding from VIRGO project NWO the Netherlands (FES0908). AO received funding from N-RENNT DFG Germany.

Ethical approval

Data collection and analyses were conducted on anonymized samples, which does not require further medical ethics review as consented by our Medical ethical board (MEC-2015-306).