Comparison of two nucleic acid amplification tests (NAATs) and two antigen tests for detection of SARS-CoV-2 from upper respiratory specimens.

There are numerous tests available for acute diagnosis of SARS-CoV-2, the virus that causes the disease COVID-19. These tests fall into two main groups: nucleic acid amplification tests (NAATs) and antigen-based assays. We evaluated the clinical performance of two rapid antigen assays (BD Veritor System for Rapid Detection of SARS CoV-2 and Abbott BinaxNOW COVID-19 Ag Card) and one NAAT (Hologic Aptima SARS CoV-2 Assay) by comparing them with the initial test of record, the Roche cobas SARS-CoV-2 assay; the antigen tests were also compared to Aptima. We tested remnant frozen specimens from patients suspected of SARS-CoV-2 infections (either due to symptoms or exposure) on the comparator platforms to evaluate assay performance across a wide range of positive results, including cobas cycle threshold (Ct) values ranging between 12 and 35. We tested 250 previous positive and 50 previous negative specimens and found 95.6% positive percent agreement (PPA) with the Aptima assay. The few discrepancies between the NAATs occurred only when Ct values were >32. Agreement was much lower for the rapid antigen tests, with 45.2%/47.3% PPA for the Veritor and 47.0%/47.0% PPA for the Binax compared to cobas/Aptima. Discrepancies occurred when cobas Ct values were >20 for Veritor and >25 for Binax. The negative percent agreement (NPA) was 100% for all assay comparisons. These data indicate similar performance between the cobas and Aptima NAATs but demonstrate that antigen-based assays may be insufficient to diagnose SARS-CoV-2 infection when lower levels of the virus are shed.

Introduction

The SARS-CoV-2 (Severe Acute Respiratory Syndrome Coronavirus 2) pandemic has shaped the world in a variety of ways and the response to testing demand has been to develop a myriad of tests to diagnose the virus [1], [2], [3]. The testing algorithm has been particularly troublesome because individuals carrying the virus may be asymptomatic [4], [5] or symptomatic, showing mild to severe symptoms including fevers, cough, and shortness of breath. Not knowing who might be carrying the virus due to the absence of symptoms has necessitated that both symptomatic and asymptomatic individuals at risk be tested for SARS-CoV-2.

Among those tests available for acute diagnostics are point-of-care (POC) rapid antigen assays and laboratory-run nucleic acid amplification-based assays [6]. In the U.S., Emergency Use Authorization (EUA) from the Food and Drug Administration (FDA) were granted, rather than full approvals, to expedite the use of newly developed assays. Many of these assays are now on the market without full understanding of their clinical sensitivity or specificity in real world use, leaving a relative lack of certainty as to how they perform in different patient populations [7], [8], [9]. Most nucleic acid amplification tests (NAATs) thus far have been based on real time reverse transcription polymerase chain reaction (RT-PCR) and are considered to be among the most sensitive and specific tests available [10], [11]. However, not all NAATs are based on real time RT-PCR. The assay available on the Hologic Panther instrument, the Aptima SARS-CoV-2 assay (Aptima), utilizes transcription-mediated-amplification (TMA) and does not require thermocycling at different temperatures to amplify the virus’ nucleic acids [12]. Like real time RT-PCR-based NAATs, this TMA-based NAAT is also among the most sensitive and specific assays available [13].

With the urgent need to increase testing capacity, antigen assays have gained attention for being lower-cost tests that provide rapid results. The BD Veritor System for Rapid Detection of SARS CoV-2 (Veritor) and the Abbott BinaxNOW COVID-19 Ag Card (Binax) are two available antigen assays that target the virus’ nucleocapsid protein but require sufficient levels of the viral protein to be present for detection, as antigen tests do not amplify the viral protein target as part of the test. The tests have been brought to market under EUA and are intended for use in symptomatic populations with the notion that asymptomatic individuals may not shed high enough levels of the virus to be detected by these assays. The performance of these assays as outlined in their EUA's suggests they may have similar performance as NAATs; however, some studies suggest that their sensitivity for detecting disease may be much lower [14].

Here, we examined the performance of two rapid antigen assays, Veritor and Binax, and compared them with the performance of two NAATs, the Roche cobas SARS-CoV-2 assay (cobas) and Aptima. Albeit “off label” for the rapid antigen assays, in order to directly compare the performance of these assays with a defined specimen collection, these tests were all evaluated using the same set of patient specimens in viral transport medium (VTM).

Materials and methods

Specimen collection and storage

A retrospective study using upper respiratory swab specimens (n = 300) from individuals with suspected SARS-CoV-2 infections (either due to clinical signs/symptoms or known exposure) was performed at the University of California San Diego Health Clinical Microbiology Laboratory. These specimens included nasopharyngeal (NP) swabs, mid-turbinate nasal swabs, and anterior nasal swabs collected in 2 mL VTM (Rocky Mountain Biologicals, Missoula, MT), each of which had been independently validated for use on the cobas and Aptima assays. The specimens were tested with the cobas assay within 24 h of collection; aliquots of each specimen were subsequently frozen at -80 °C until comparator testing. The use of the residual specimens for this study was approved by the UCSD Institutional Review Board under protocol #160524.

SARS CoV-2 assays

The cobas® SARS-CoV-2 real-time RT-PCR test was performed on the cobas 6800 instrument (Roche Molecular Diagnostics, Pleasanton, CA). Two SARS-CoV-2 nucleic acid targets are amplified by real time RT-PCR: the SARS-CoV-2 specific ORF1 a/b non-structural region (Ct1) and a conserved region of the envelope E-gene common to all SARS-like coronaviruses (pan-Sarbecoviruses) (Ct2). The Hologic Aptima SARS-CoV-2 Assay is a TMA assay that amplifies two distinct conserved regions of the ORF1 a/b gene and was performed on the Panther instrument (Hologic, Inc., San Diego, CA). Testing for cobas and Aptima was performed following manufacturer instructions. The BD Veritor™ System for Rapid Detection of SARS-CoV-2 is a POC chromatographic immunoassay that detects the SARS-CoV-2 nucleocapsid antigen. The Abbott BinaxNOW™ COVID-19 Ag Card is a lateral flow immunoassay that detects viral nucleocapsid antigen. Both Veritor and Binax are indicated for use with direct nasal swabs. However, as residual swab specimens used in this study were collected in VTM, the swabs from the rapid antigen assays were dipped into the VTM specimens until they were completely saturated. From this point, those swabs were used in the Binax and Veritor testing procedures, according to the manufacturer's instructions. Original qualitative results (positive or negative) and Ct values from cobas and qualitative and Relative Light Unit (RLU) results obtained from Aptima were recorded; only qualitative results were available for the rapid antigen assays.

Identification of specimens for testing

Fifty (50) cobas negative and 250 cobas positive specimens were selected for the study. All positive specimens were positive for both cobas targets (Ct1 and Ct2); 62 specimens with Ct1 values from 12 to <20, 61 from 20 to <25, 68 from 25 to <32, and 59 from 32 to 40 were included in the study. Frozen aliquots from all 300 specimens previously tested with cobas were tested using Aptima and Veritor; a subset of these (100 positive samples representative of the Ct groupings above and 20 negative) were tested using Binax.

Data analysis

Mean ages and standard deviation of the study population were determined (Microsoft Corporation). Positive percent agreement (PPA), negative percent agreement (NPA), overall percent agreement (OPA) and associated 95% confidence intervals (CI) were determined with cobas or Aptima serving as the reference method for the various assay comparisons.

Results

Study population

We examined 300 (250 positive and 50 negative) upper respiratory specimens (NP swabs, mid-turbinate nasal swabs, and anterior nasal swabs) collected from individuals with suspected SARS-CoV-2 infections. The individuals tested were from the following age groups: ≤ 25 years (n = 69), 26 to 45 years (n = 100), and 46 years and older (n = 131). The mean age for those who tested positive for SARS-CoV-2 was 43.4 ± 19.5 years, and 42.92 ± 28.9 years for those who tested negative (Table 1 ). The population included 48.4% females who tested positive, and 56.0% for those who tested negative. The vast majority of this sample set came from symptomatic individuals (86.7%), with 96.6% of the individuals testing positive being symptomatic, and 36.0% of the individuals testing negative being symptomatic.

Table 1

Characteristics of subjects based on cobas results.

	Positive (n = 250)	Negative (n = 50)
Age, mean ± SD	43.4 ± 19.5	42.9 ± 28.9
Female Sex, n (%)	121 (48.4)	28 (56.0)
Symptomatic, n (%)	242 (96.6)	18 (36.0)
Ct1 12 to <20, n (%)	62 (24.8)	N/A
Ct1 20 to <25, n (%)	61 (24.4)	N/A
Ct1 25 to <32, n (%)	68 (27.2)	N/A
Ct1 32 to 40, n (%)	59 (23.6)	N/A
Ct1 not detected	N/A	50

Agreement among NAATs and antigen assays

We first compared the Aptima assay to the cobas assay. Of the 250 swab specimens that were positive on the cobas assay, 239 also tested positive on the Aptima assay; all 50 cobas negatives were negative by Aptima (Table 2 ). The positive percent agreement (PPA), negative percent agreement (NPA), and overall percent agreement (OPA) were 95.6% (95% CI: 92.3% – 97.5%), 100% (95% CI: 92.9% – 100%), and 96.3% (95% CI: 93.6% – 97.9%), respectively.

Table 2

Positive and negative percent agreements.

		cobas
		Pos	Neg	Total			95% CIs
	Pos	239	0	239	PPA	95.6%	92.3%	97.5%
Aptima	Neg	11	50	61	NPA	100.0%	92.9%	100.0%
	Total	250	50	300	OPA	96.3%	93.6%	97.9%

We next evaluated two rapid antigen tests, Veritor and Binax, against the NAAT tests. Of the 250 cobas positives, 113 were positive on the Veritor assay, with 100% agreement among negatives. These data correspond to a PPA of 45.2% (95% CI: 39.2 – 51.4%), an NPA of 100% (95% CI: 92.9 – 100%), and an OPA of 54.3% (95% CI: 48.7- – 59.9%) (Table 2). Results for the Binax assay versus cobas were comparable, but only a subset (100 positive and 20 negatives) was tested due to limitations in reagent supplies. Of 100 cobas positive specimens, 47 were positive by Binax, and all 20 cobas negative specimens were negative. This corresponds to a PPA of 47% (95% CI: 37.5 – 56.7%), an NPA of 100% (95% CI: 83.9 – 100%), and an OPA of 55.83% (95% CI: 46.9 – 64.4%). Similar results were observed for the antigen versus Aptima assay comparisons, with Veritor demonstrating a PPA, NPA, and OPA of 47.3% (95% CI: 41.0% – 53.6%), 100% (95% CI: 94.1% – 100%), and 58.0% (95% CI: 52.4% – 63.5%), respectively, and Binax demonstrating a PPA, NPA, and OPA of 47.0% (95% CI: 37.8% – 56.5%), 100% (95% CI: 85.7% – 100%), and 58.3% (95% CI: 49.4% – 66.8%), respectively.

NAAT and antigen assay performance based on cobas Ct values

We next evaluated how each test compared to the cobas assay based on the cobas Ct1 values. We found that 100% of the specimens with cobas Ct1 values <32 tested positive on the Aptima assay (Fig. 1 ); of 59 specimens with Ct1 values ≥32, Aptima detected 48 (81.4%). All 11 discordant results occurred at Ct1 values >32.6, which is at or below the cobas assay limit of detection (LOD; average Ct1 value at LOD = 32.7) [15].

Fig. 1

Scatter plot demonstrating the cobas Ct1 values of specimens that test positive or negative on the Aptima, Veritor, and Binax assays. The y-axis shows the Ct1 values and the x-axis shows positive and negative results for each assay. Positive Aptima results are shown in dark blue, Veritor results in orange, and Binax results in purple. Negative Aptima results are shown in cyan, Veritor results in yellow, and Binax results in magenta. Positivity rates based on Ct1 results are demonstrated in the table below. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

For the antigen assays, discordant results occurred at lower Ct1 values. We observed that 62 of 62 (100%) specimens with Ct1 values <20 tested positive on the Veritor (Fig. 1). Only 45 of the 61 (73.8%) specimens with Ct1 values ranging from 20 to 24.99 tested positive, 5 of 68 (8.5%) with Ct1 values ranging from 25 to 31.99, and 1 of 59 (1.7%) with Ct1 values ranging from 32 to 40. A similar pattern was obtained in our evaluation of the Binax. We found that 20/20 (100%) specimens with Ct1 values <20 and 20/20 (100%) specimens with Ct1 values ranging from 20 to 24.99 tested positive on the Binax. However, only 7 of 40 (17.5%) with Ct1 values from 25 to 31.99 and 0 of 20 (0%) with Ct1 values from 32 to 40 tested positive on this assay.

Discussion

Since the beginning of the SARS-CoV-2 pandemic, NAATs have been the standard of care for diagnosing individuals with acute infections. In this study, we examined the performance of the Aptima and cobas NAAT assays, which are among the most sensitive assays used for acute SARS-CoV-2 diagnosis [12], [13], [16], [17], [18]. Previous studies have shown the Aptima assay has a high PPA (95–100%) to other real time PCR-based SARS-CoV-2 EUA assays [12], [13], [16]. Similarly, we found high agreement between the Aptima and cobas assays (95.6% PPA, 100% NPA), with only a small number of discrepant results, all of which were at or below the LOD of the cobas assay (Ct1 value >32). While these data might suggest that cobas has slightly higher sensitivity when there are lower levels of virus present, performance comparisons of other assays have shown that results often are not reproducible in the lower range of detection [19], [20], and samples with target levels at or below LOD will have increased variability in detection even on the same platform. We also note that these remnant samples had undergone a freeze/thaw between testing on the cobas and Aptima assays, which has the potential to compromise detection particularly in low target (high Ct value) samples, thus putting Aptima at a disadvantage in this comparison.

Rapid antigen assays offer the benefit of detecting the virus in less than 15 min but are believed to suffer from a relative lack of sensitivity compared to NAAT tests [21], [22]. Our comparison of the Veritor and Binax antigen assays with cobas demonstrated low PPAs (45.2 and 47%) for both assays (Table 2). Similar agreements were obtained when comparing each rapid antigen test to the Aptima assay. The antigen tests demonstrated 100% NPA with both NAATs. We did not observe any potential false positive results from the rapid tests in our evaluation, despite this being an issue noted for rapid antigen tests in other reports [21], [23]; however, our negative sample set was smaller, thus limiting the potential to identify false positives. Further analysis of antigen test performance revealed that as Ct1 values for the cobas assay increased, the likelihood that the rapid tests would result negative also increased. Above a Ct1 value of 20, Veritor detection rates decreased significantly as Ct1 values increased; for Binax, detection rates began to decrease when cobas Ct1 values were >25. It should be noted that in order to compare retrospective samples across assays, we could not use the swabs for primary collection per the antigen assay package inserts. Instead, the Veritor and Binax dry swabs were immersed into the VTM of the remnant sample and tested. We estimate the dilution effect of this method to be approximately a 1 to 1.5 log difference (i.e., the primary swab diluted into 2 mL VTM, with ~80–100 uL absorbed onto the swab for antigen assay testing, translating to a potential 3–4.5 Ct difference by real time PCR. Considering the dilution effect, Veritor detection rates may have improved up to Ct1 values of 23–24.5, and Binax detection rates may have improved up to Ct1 values of 28–29.5. Adjusting for this, overall PPAs could be higher, but likely still near 70%, and corresponds with previous studies in symptomatic patients [24]. Collectively, these data suggest that the antigen-based assays are capable of detecting SARS-CoV-2 from nasal specimens when Ct values are relatively low, but their correlation with NAATs is significantly reduced as Ct values increase. These data are in direct opposition to the performances outlined in the package inserts of each test but are in line with results being provided by other studies [21], [22], [24], [25].

While our data suggest relatively poor performances for rapid assays compared to the cobas NAAT, there were limitations that might have affected the performances of the rapid assays. In this study, we used swabs in VTM for all the tests. The use of the antigen tests outside of their recommended application could have affected sensitivity, as noted above. These tests also are authorized for use in symptomatic but not in asymptomatic individuals. We investigated their use in a sample set that included a small number of asymptomatic individuals, which could have affected the clinical agreement. However, the majority of samples from the positive sample group were obtained from symptomatic individuals (96.6%) and also contained all of the discordant results. Finally, our retrospective study required freezing and thawing of specimens, potentially resulting in degradation of viral nucleic acids and rupturing of virions causing initially positive specimens to provide false negative results [26]. This could especially impact samples with viral target levels at or below the LOD.

Because many of the SARS-CoV-2 acute diagnostic tests were developed rapidly, data available for sensitivity and specificity across broad groups of the population have not been available for these assays. As such, direct comparisons of assay performance help to provide greater understanding of how these assays perform across an array of positive and negative SARS-CoV-2 specimens. While each assay has its utility in the midst of this pandemic, these results add to a growing body of evidence that highlights the significant limitations in sensitivity of antigen tests as compared to molecular NAATs.

Declaration of Competing Interest

All authors have reviewed and approved the manuscript and have no conflicts of interest.