Diagnostic performance of two highly multiplexed respiratory virus assays in a pediatric cohort.

BACKGROUND: Rapid detection of respiratory viruses is important for management and infection control in hospitalized patients. Multiplex nucleic acid tests (NATs) have begun to replace conventional methods as gold standards for respiratory virus detection.
OBJECTIVE: To compare the performance of two large multiplex NATS, ResPlex II (RPII) and Respiratory Virus Surveillance kit with electrospray ionization mass spectrometry (RVS/MS) using nasopharyngeal aspirates (NPAs) from hospitalized children who had been tested previously with conventional methods. STUDY
DESIGN: Stored residual NPAs (N=306) were tested concomitantly by RPII and RVS/MS. Alternate NATs were used to adjudicate discordant results.
RESULTS: More viruses were detected with multiplex NATs (RPII, 110; RVS/MS, 109) than conventional assays (86); diagnostic gain was primarily for fastidious viruses (coronaviruses and enteroviruses [EVs]/human rhinoviruses [HRVs]). Total positive and negative agreement between the multiplex NATs for all viruses detected was quite high (86% positive agreement, 99% negative agreement). Most individual viruses were detected with fairly equivalent accuracy by the multiplex NATs, except for adenoviruses (RPII sensitivity 40%) and human metapneumovirus (RVS/MS sensitivity 42%). RPII had the advantage of detecting EVs and HRVs, however, it demonstrated considerable EV/HRV cross-reactivity (29 HRV-positive specimens by real-time PCR were positive for EV by RPII and 21 specimens positive for HRV only by RT-PCR were dual positive for EV/HRV by RPII). RPII also had reduced sensitivity for HRV detection (in 36 specimens, HRV was detected by RT-PCR but not by RPII).
CONCLUSIONS: Both multiplex NATs were promising, but had notable limitations.

Background

Timely respiratory virus detection in hospitals is important for patient management and infection control. Rapid, highly sensitive nucleic acid tests (NATs) are replacing conventional methods as the new gold standard. Respiratory virus NATs vary from singleplex to multiplex PCR and in either conventional or real-time formats. Multiplexed respiratory NATs are often preferable since they eliminate the need for separate NATs to identify respiratory viruses that often cause overlapping clinical presentations.

ResPlex II (RPII) and Respiratory Virus Surveillance kit with electrospray ionization mass spectrometry (RVS/MS) are two large multiplex NATs with different amplification/detection designs. RPII contains a low concentration primer pair that is complementary to viral targets and has a heterologous sequence. A second, high concentration primer pair contains sequences complementary to this heterologous sequence. This design favors specific target amplification early in PCR and amplicon amplification at later cycles, improving amplification efficiency and reducing background. Respiratory syncytial virus (RSV) A and B, influenza (INF) A and B, parainfluenza 1–4 (PIV1–4), human metapneumovirus (hMPV), coxsackieviruses/echoviruses (EV herein), human rhinovirus (HRV), adenovirus B and E (ADV), coronaviruses NL63, HKU1, 229E, and OC43 (referred to collectively as CoV), and bocavirus are detected.

The RVS/MS assay utilizes conventional PCR and electrospray ionization mass spectrometry (ESI/MS) to detect and identify respiratory viruses. ESI/MS is used to convert amplicon duplexes to single strands, and to determine nucleotide composition by measuring single strand masses. Observed masses are compared to a library of expected masses to identify amplified products. Each PCR reaction contains an exogenous internal control that can also be used to estimate target analyte concentration. RVS/MS detects similar viruses as RPII except it is designed to detect a broader array of ADVs (genotypes A–F) and it does not detect EVs and HRVs.

Objective

The purpose of this study was to compare the ability of RPII and RVS/MS to detect viruses in nasopharyngeal aspirates (NPAs) from clinically well-characterized, hospitalized pediatric patients that had previously been tested by conventional methods.

Study design

Clinical samples

NPAs were collected from infants and toddlers (N  = 306) upon admission to the hospital from December 2007 to March 2008 and submitted for conventional virologic testing. This retrospective cohort was selected for testing because it has been well characterized clinically. Residual samples were stored at −70 °C until nucleic acid testing; NATs were performed simultaneously.

Conventional methods

Cell pellets from processed NPAs were spotted onto slides and stained with antibodies to INF A and B, RSV, PIV 1–3, ADV, and hMPV (D3 Ultra Screening and ID reagents and hMPV reagent, Diagnostic Hybrids, Inc [DHI], Athens, OH). Supernatants were cultured by shell vial (R-Mix Too, DHI) and tube culture. R-Mix Too coverslips were stained at 48 h with D3 Ultra Screening reagent then with specific ID reagents when appropriate. Tube cultures were screened for cytopathic effect for 3 weeks; confirmatory staining was performed when appropriate.

Nucleic acid extraction

Total nucleic acid was extracted from NPAs using the QIAsymphony SP automated instrument, QIAsymphony Virus/Bacteria Mini Test Kit reagents, and the Complex 200 protocol (Qiagen Germantown, MD). Input and elution volumes were 350 μl (200 μl processed) and 110 μl.

RPII target amplification (Qiagen)

Amplification and detection reactions were performed according to manufacturer's research use only protocol. Process controls added by the QIASymphony during extraction included an internal control and reagents to detect human genomic DNA (proprietary target) that ascertained sample quality. Signal detection/raw data analysis Signals were detected with the Luminex 200 IS System using QIAplex MDD Software. In-house determination of the cut-off value that could be applied to distinguish positive results was required since software lacks this interpretive criterion. Four different cut-off values were studied including 3× background (bkgd), ≥50, ≥100, or ≥150 mean fluorescence intensity (MFI) values. For 3× background calculation, the negative control MFI for each assay was multiplied by three and used as the cutoff to define positive samples within that run. To decide which cut-off value was most appropriate, the results obtained with each cutoff were compared to RVS/MS, discrepant analysis with alternate PCR assays, and conventional test results. Two cutoffs, ≥100 MFI and 3× bkgd MFI, showed the best correlation with aggregate data from other test methods. The 3× bkgd cut-off was selected because it demonstrated excellent overall agreement with the other methods and it allowed inter-run data normalization by accounting for run-to-run variability in background fluorescence. For the purposes of this study, RSV A and B were not differentiated and bocavirus was not included in the analysis since its importance as a respiratory pathogen has not been firmly established.

RVS/MS (Ibis Biosciences, Inc., a part of Abbott Molecular, Des Plaines, IL)

Target amplification was performed as described previously using a Mastercycler ep gradient S (Eppendorf, Hamburg, DE). Amplification products were desalted and analyzed using the T5000 universal biosensor, a time-of-flight mass spectrometer.

Singleplex assays for enterovirus and rhinovirus

Two different real-time RT-PCR assays for EVs and HRVs were used as comparators for RPII since these viruses were not detected by RVS/MS. Assays were performed as described with the following modifications: the QIAsymphony was used for extraction and real-time PCR was performed with QuantiTect Virus Kit (Qiagen) and 7500 Real Time PCR system (50 °C for 30 min, 95 °C for 15 min and 45 cycles of 95 °C for 15 s and 60 °C for 1 min).

Discrepant analysis

Discordant results were adjudicated with alternative NATs (Table 1 ). Highly sensitive singleplex assays (5–50 copies/reaction as demonstrated by published or in-house data) were selected for use. Multiplex NAT (xTag RVP, Luminex, Austin, TX) was used when singleplex assays that met the above criteria were not readily available (Table 1).

Table 1

Assays used in discordant analysis.

	Virus	Reference	Methodologic changes to published/cleared assays
Singleplex NAT	ADV	13	Nucleic acid extraction with QIAsymphony; use of 2× Universal master mix (Applied Biosystems,) and 7500 Real Time PCR System for real-time PCR amplification
	hMPV	14	Nucleic acid extraction with QIAsymphony; use of Quantitect Virus kit (Qiagen) and 7500 Real Time PCR System for reverse transcription/real-time PCR amplification
Multiplex NATa	RSV	15	None
	CoV, PIV 4		Results obtained using RUO software

ADV – adenovirus; hMPV – human metapneumovirus; RSV – respiratory syncytial virus; CoV – coronavirus; PIV – parainfluenza virus.

xTag RVP.

Statistical methods and calculations

Data were stored in Excel (Microsoft, Redmond, WA) and analyzed using Stata, version 11.0 (StataCorp, College Station, TX). Positive and negative agreement (by virus and by total positives and negatives) were calculated using RPII as the test assay and RVS/MS as the comparator assay. Since this agreement analysis highlights deficiencies in virus detection by RPII but minimized detection deficiencies by RVS/MS, an assessment of sensitivity and specificity of each assay was also performed using the following definitions: true positive, positive by two NATs (either RPII plus RVS/MS or one multiplex tests plus one NAT used for discordant analysis) or one multiplex NAT (RPII or RVS/MS) and one conventional test; true negative, negative by RPII and RVS/MS; false positive, positive by one multiplex NAT (RPII or RVS/MS) but negative by the other multiplex NAT, by alternate NAT and by conventional tests. Due to the small number of INF B (2), INF A and B were combined as INF for agreement, sensitivity, and specificity calculations. Likewise, PIVs were analyzed as a single group (PIV) since the combined total number of PIV 1, 3 and 4 detected was four. CoVs were analyzed as a group since there is currently no clinical utility to distinguishing these viruses individually. EV and HRV discordant results were not adjudicated by a third assay, therefore, only agreement was determined. Confidence intervals were calculated using a web-based calculator. Positive signal statistical data provided in Supplemental data tables were calculated using Excel.

Results

Detection of viruses other than EVs and HRVs

The number of viruses detected by RPII and RVS/MS was similar (110 vs. 109, Table 2 ) and greater than the number detected by conventional methods (N  = 86). Viruses were detected in 101 (33%), 98 (32%), and 86 (29%) of 306 samples by RPII, RVS/MS, and conventional methods respectively (data not shown). RSV was most commonly identified by all methods (Table 2). The second and third most commonly detected viruses varied by multiplex. INF A, hMPV, and CoVs were the next most prevalent viruses found by RPII (INF A, 5.2% prevalence overall and 15% of all identified viruses; hMPV and CoVs, 4.2% prevalence overall and 12% of all identified viruses), while ADV and INF A were found by RVS/MS (ADV, 5.5% prevalence overall and 16% of all identified viruses; INF A, 4.9% prevalence overall and 14% of all identified viruses). CoVs were not detected by conventional methods but were fairly common (∼4% prevalence overall, ∼12% of all identified viruses) and similarly detected by both multiplex NATs.

Table 2

Viruses detected by RPII, RVS/MS and conventional methods.a

Virus	RP II	RVS/MS	Conventional methods
ADVb	6	17	5
CoV	13	14	NDc
hMPV	13	5	7
INF A	16	15	15
INF B	1	1	1
PIV	4	1	1
RSV	57	56	57
Totald	110	109	86

HRV/EV reported as separate analysis (Table 4).

ADV – adenovirus; CoV – coronaviruses; hMPV – human metapneumovirus; INF –influenza virus; PIV – parainfluenza virus; RSV – respiratory syncytial virus.

ND, not detected by conventional methods.

Total number of viruses detected by each method, including confirmed and unconfirmed detection events.

Table 4

Performance of RPII and Singleplex real-time PCRs for HRV and EV.

	RT-PCR dual positivea	RT-PCR HRV only	RT-PCR EV only	RT-PCR negative
RPII Dual positive	0	21	0	0
RPII HRV only	0	7	0	1
RPII EV only	14	29	1	2
RPII negative	0	36	1	194

Dual positive, HRV and EV detected.

EV/HRV not specified: either EV, HRV or both detected in a sample, but grouped as EV/HRV detection (i.e. specific virus not differentiated) for analytical purposes.

The relative diagnostic performance of the two multiplex NATs in individual samples was high and fairly equivalent for most viruses, as determined by agreement analyses and by assay sensitivity/specificity (Table 3 ). The lowest % positive agreement was observed for ADV; lowest % negative agreement was observed for hMPV. Resolution of discordant results with either an alternate NAT or conventional result further delineated the disparity in detection of these two viruses by the multiplex NATs (sensitivity of 40% for ADV by RPII and 42% for hMPV by RVS/MS).

Table 3

Comparison of RPII and RVS/MS diagnostic performance.

Virus	RPII + RVS/MS + (A)	RPII + RVS/MS-(B)	RPII-RVS/MS + (C)	RPII-RVS/MS-(D)	Positive % agreement (95% CI)a	Negative % agreement (95% CI)a	RPII		RVS/MS
							% Sensitivityb	% Specificityb	% Sensitivityb	% Specificityb
ADV	6	0	11 (9/2)c	289	35 (15–61)	100 (98–100)	40 (17–67)	100 (98–100)	100 (75–100)	100 (98–100)
CoV	11	2 (1/1)d	3 (2/1)e	290	78 (49–94)	99 (97–100)	100 (70–100)	100 (98–100)	92 (60–100)	100 (98–100)
hMPV	5	8f	0	293	100 (46–100)	97 (95–99)	100 (72–100)	100 (98–100)	42 (16–71)	100 (98–100)
INF	16	1g	0	289	100 (76–100)	100 (98–100)	100 (76–100)	99 (98–100)	100 (76–100)	100 (98–100)
PIV	1	3 (2/1)h	0	302	100 (55–100)	99 (97–100)	100 (55–100)	99 (97–99)	100 (55–100)	100 (98–100)
RSV	53	4 (3/1)c	3i	246	95 (84–99)	98 (96–99)	95 (85–99)	100 (98–100)	95 (85–99)	100 (98–100)
Totalj	92	18	17	1709	84 (76–90)	99 (98–99)	90 (82–94)	100 (99–100)	90 (83–95)	100 (99–100)

RPII and RVS/MS were used as test and comparator assays respectively. Positive % agreement, (A/A + C) × 100; negative % agreement, (D/D + B) × 100. 95% CI, 95% confidence interval.

Sensitivity and specificity were calculated using alternative NAT or conventional results to adjudicate discordant results as described in Section 3.8. Samples with insufficient volume for discordant analysis were excluded from sensitivity and specificity calculations.

Parentheses indicate number of discordant samples confirmed by third NAT or conventional test result/number of discordant samples that could not be resolved by alternate NAT due to insufficient residual volume.

Parentheses indicate number of discordant samples confirmed by third NAT/number of discordant samples not confirmed positive by third NAT.

Parentheses indicate number of discordant samples that could not be verified by alternate NAT due to insufficient residual volume/number of discordant samples not confirmed positive by third NAT.

All discordant samples confirmed for hMPV by third NAT (N = 7) or conventional test (N = 1).

Sample demonstrated low positive signal for INF A by RPII that was not corroborated by third NAT.

Numbers in parentheses indicate 2 samples with low positive signal for PIV by RPII that were not corroborated by third NAT/one PIV positive sample was QNS for discordant analysis.

All discordant samples confirmed for RSV by third NAT (N = 2) or conventional test (N = 1).

Total, sum of all results in columns A through D, including specimens with insufficient residual volume for discordant analysis.

EV/HRV detection by RPII

Aside from samples that were negative for both viruses by RPII and RT-PCR, there was little agreement of EV/HRV results in individual samples (Table 4 ). No samples were dual positive for HRV plus EV by RPII and by RT-PCR, only one sample was positive for EV by RPII and RT-PCR, and only 7 samples were positive for HRV by RPII and by RT-PCR. RPII detected EVs in a greater number of samples than RT-PCR (67 vs. 16). These data suggest cross-reactivity between RPII EV and HRV detection reagents rather than enhanced sensitivity of EV detection since 50/67 EV positive samples (21 dual positive and 29 EV only) contained only HRV by RT-PCR. Furthermore, a large proportion of RPII EV only/RT-PCR negative samples would be expected if EVs were more effectively detected by RPII, however only two such samples were identified.

In contrast to EV detection, RPII detected HRV in fewer samples than RT-PCR (29 vs. 107). Notably, 36 samples were found to contain HRV RNA by RT-PCR but were negative by RPII (Table 4). The mean Ct value for these HRV-positive samples was 36, the highest of any category found to contain HRV by RT-PCR in the 4 × 4 analysis (Supplemental data Table 1), suggesting that these samples contained low target RNA levels.

Fourteen samples were positive for HRV and EV by RT-PCR, but were only positive for EV by RPII. In a qualitative concordance analysis, these samples would appear to be false positive for RT-PCR results, however analysis of signal strength suggests the opposite. The mean and median HRV Ct values were the lowest of any category of samples found to contain HRV by RT-PCR (mean 24, median 21; see Supplemental data Table 1). These data suggest erroneous EV detection by RPII and RT-PCR in samples containing high HRV concentrations.

A simple 2 × 2 agreement table in which the specific virus or combination of viruses is not differentiated (Table 4, bottom panel, EV/HRV Not Specified) was constructed given the high degree of sequence concordance between EVs and HRVs and the absence of definitive sequence data for the studied samples. Positive agreement in this analysis was fairly high (72/109, 66%). Even in this analysis however, singleplex real-time RT PCRs were more sensitive since they detected EV/HRV in 37 samples that were negative by RPII, largely due to the number of samples that were positive by RT-PCR for HRV but negative by RPII.

Detection of co-infections

Co-infections with viruses other than EV/HRV were found in relatively few samples (RPII, 6/306, 1.9%; RVS/MS, 8/306, 2.6%) and all confirmed co-infections contained CoVs (specifically HKU1, Supplemental data Table 2). Co-infections with enteroviruses (EV and HRV) and other viruses were found in 18 samples by RPII (Supplemental data Table 3).

Conclusions

NATs are powerful tools in the diagnosis and management of respiratory virus infections due to their enhanced sensitivity and rapid time to result. Deep multiplexing also has the important advantage of allowing identification of not only the usual pathogens (RSV, INF A, ADVs) but also unsuspected, typically lower virulence viruses (PIVs, HRVs, CoVs) that can cause lower respiratory tract disease and can therefore be significantly pathogenic in highly susceptible hosts. In this performance comparison study on samples from hospitalized children, the highly multiplexed RPII and RVS/MS assays detected most viruses with fairly equivalent accuracy. The exceptions to this were hMPV and ADV which were more effectively detected by RPII and RVS/MS respectively. A low ADV detection rate by RPII compared to a different extensively multiplexed respiratory virus NAT (Seeplex) has been previously reported. Independent published confirmation of low hMPV detection rates by RVS/MS compared to an alternate NAT are lacking. Given that ADV and hMPV are highly prevalent and can cause serious disease, these potential diagnostic shortcomings are significant and represent opportunities for performance enhancement in subsequent generations of these assays. Unfortunately the prevalence of PIVs was too low to make any definitive conclusions regarding the relative efficacy of RPII and RVS/MS. It should also be noted that the study design precludes conclusions regarding true (vs. relative) accuracy and sensitivity. Therefore, while the accuracy of detection for most viruses was comparable and quite high, testing with gold standard NATs for each analyte might reveal additional detection deficiencies common to RPII and RVS/MS.

For RPII, the manufacturer does not supply a cutoff MFI value that allows differentiation of positive and negative results. Our method comparison study design allowed for analysis of multiple cutoffs as defined above. A variable cutoff based on intra-run background signal functioned well and offered the theoretical advantage of correcting for inter-run background variability. Additional studies with a larger cohort are required to conclusively establish its superiority.

To have maximal clinical utility, respiratory virus NATs should detect the broadest range of pathogenic targets. RPII provides better diagnostic potential than the current version of RVS/MS since it can detect EV/HRV. In fact, in this study, the prevalence of EV/HRV was highest of all viruses (∼37%, including RPII and RT-PCR positives). However, RPII has limitations with regard to EV/HRV detection. First, the data suggest cross-reactivity between RPII EV and HRV detection reagents as suggested by previous studies that noted discordant EV/HRV results in a limited number of samples and a high proportion of EV/HRV dual positives. Our data imply that cross-reactivity is more extensive than previously appreciated given the number of discordant (RPII EV+/RT-PCR HRV+) and RPII dual+/RT-PCR HRV only specimens.

One curious result was the number of specimens found to be dual positive by RT-PCR but positive only for EV by RPII. On a qualitative basis, these results would appear to suggest false positive HRV RT-PCR results. However signal strength data point to a different conclusion. The low mean and median HRV RT-PCR Ct values suggest that most of these samples likely contained HRV only, further implying issues with EV detection specificity (by RPII and RT-PCR) and sensitivity of HRV detection (by RPII). Published data and our unpublished observations with the EV RT-PCR reagents suggest a propensity for cross-reactivity. An internal study of EV and HRV RT-PCR reagents demonstrated HRV in 69/507 NPAs; 4/69 were also amplified by EV RT-PCR. VP1 sequence data was obtainable from 1/4 samples and demonstrated HRV. HRV RT-PCR specificity should be much greater than EV RT-PCR; the latter was developed by comparative analysis of a more extensive sequence library compared to EV RT-PCR and was shown to cross-amplify EV only in artificial samples spiked with high concentrations of synthetic transcripts.

Improved EV/HRV concordance between RPII and RT-PCR was observed when positive results were grouped (EV/HRV unspecified), which may be the most accurate approach for reporting RPII results. However even with grouped reporting, it was apparent that RPII missed approximately one third of infections detected by singleplex RT-PCR. A similar defect in EV/HRV detection compared to an alternate NAT has been reported. Subsequent versions of this assay would benefit from improved HRV sensitivity given reports of HRV causing clinically significant disease.11, 12

Few co-infections with viruses other than EV/HRV were observed. Strikingly, all confirmed non-EV/HRV co-infections included HKU-1 as a co-pathogen. The biological significance of this is unclear; no data are available in the literature. Among EV/HRV co-infections, no dominant non-enteroviral pathogen was observed. The identified viruses reflect the spectrum commonly observed in young children.

The greatest diagnostic gain of the extended panel NATs was largely for viruses that are more challenging to detect by conventional methods. Viruses such as RSV and FluA that are readily detectable by DFA and culture were equivalently detected by NATs and conventional methods. PIV prevalence was too low to make any conclusive statements regarding the relative efficacy of conventional vs. NAT panels but the data suggest improved detection by NATs.

Deeply multiplexed respiratory virus NAT replaced cell culture for use in immunocompromised populations in our center in the wake of the 2009 influenza pandemic due to the perceived benefit that a rapid, highly sensitive result would provide for patient management. After several years’ experience with this testing, it has become apparent that the characteristics of the optimal respiratory virus NAT include ability to detect broadest range of respiratory virus pathogens, highly accurate detection of each virus, low labor requirement, results available within a single shift, and low contamination rates. The increasing availability of deeply multiplexed commercial respiratory virus NATs offers the opportunity to compare performance characteristics. Our studies with RPII and the current version of RVS/MS suggest that they each have strengths but both demonstrated some limitations with regard to fitting the profile of the perfect assay.

Funding

This work was funded by R-Baby Foundation.

Competing interests

Reagent support was kindly provided by Qiagen and Ibis Biosciences/Abbott Molecular. C. Gaydos has received research funding from Abbott/Ibis. A. Valsamakis has received research funding from Abbott/Ibis and Qiagen; she has served on a scientific advisory board for Diagnostic Hybrids Inc.

Ethical approval

This study was approved by the Johns Hopkins School of Medicine Institutional Review Board (Study #NA_00021591).