Field evaluation of seven SARS-COV-2 antigen assays.

There is a global demand for rapid diagnostic tests (RDTs) for Coronavirus disease 2019 (COVID-19), and the interest in their clinical compliance is growing. In this study, we evaluated the clinical compliance of seven different severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) antigen RDTs. Nasopharyngeal/oropharyngeal swab specimens from COVID-19-confirmed cases and reverse-transcription PCR (RT-PCR) screening were used to evaluate the performance of seven RDTs. Using the RT-PCR and RDT results, we predicted the cycle threshold (Ct) of each target gene (E, RdRP, and N genes) which 50% (Ct50) and 95% (Ct95) detection rates were achieved in the RDTs. A total of 482 specimens were enrolled in our study: 316 specimens from COVID-19-confirmed cases and 166 RT-PCR-negative specimens. The median values of Ct50 and Ct95 for the seven RDTs were in the ranges of ranged 24.3-30.9 and 19.3-22.6 for E, 25.5-31.5 and 20.9-24.0 for RdRP, and 26.8-32.3 and 22.7-25.7 for N, respectively. The RDTs showed acceptable compliance only for specimens with high viral burdens (Ct < 20). However, the false-negative rate increased by more than 50% for most of the RDTs in low-viral burden specimens (Ct> 30). These results suggest that RDTs should not be used without molecular assays for COVID-19 screening for asymptomatic patients because of their high false-negative rates.

Introduction

Since the rapid, worldwide spread of the novel coronavirus strain, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), Coronavirus disease 2019 (COVID-19) has become the leading public health concern [1]. Although molecular assays are considered the gold standard for the diagnosis of SARS-CoV-2 infection, the drawbacks of PCR-based assays, such as long turnaround times and high running cost, pose several problems in a “PCR-only”-based diagnostic testing program [2]. Antigen (Ag) rapid diagnostic tests (RDTs) for the detection of SARS-CoV-2 have become attractive options for several countries lacking resources and skilled laboratory staff to perform complex PCR assays [3]. Although many RDTs have been evaluated for their performance and are actually used in various countries [[3], [4], [5], [6]], there is still a growing demand for the evaluation of RDTs in real-world clinical settings. In this study, we evaluated the clinical compliance of seven SARS-CoV-2 antigen RDTs and compared their results with those of a molecular assay using nasopharyngeal swabs.

Materials and methods

We selected seven SARS-CoV-2 antigen lateral flow rapid assays from five manufacturers in this study: ASAN Easy Test® COVID-19 Ag (Asanpharm, Gyeonggi-do, Republic of Korea), AFIAS COVID-19 Ag and ichroma™ COVID-19 Ag (Boditech Med Incorporated, Gang-won-do, Republic of Korea), Epithod®AutoDx SARS-CoV-2 qAg and Epithod®616 COVID-19 Ag (DxGen, Gyeonggi-do, Republic of Korea), VERI-Q COVID-19 Ag Rapid Test (MiCo BioMed Co., Ltd., Gyeonggi-do, Republic of Korea), and STANDARD Q COVID-19 Ag (SD Biosensor, INC. Chungcheongbuk-do, Republic of Korea). Information about the seven RDTs is summarized in Supplementary Table S1. At the request of the manufacturers, the kits were anonymized and randomly labeled from A to G.

We collected the clinical specimens for which the SARS-CoV-2 PCR tests were to be performed at the Chung-Ang University Hospital (CAUH). Nasopharyngeal/oropharyngeal (NP/OP) swab specimens were collected from COVID-19-confirmed cases and reverse-transcription PCR (RT-PCR) screening cases. Nucleic acid extraction was performed using the NucliSens easyMAG instrument (bioMerieux, Marcy l'Etoile, France), and routine SARS-CoV-2 RT-PCR tests were performed using the Allplex™ 2019-nCoV Assay (Allplex; Seegene, Seoul, Republic of Korea) and a CFX96™ Real-Time PCR Detection System (Bio-Rad, CA, USA) according to the manufacturer's instructions. The Allplex™ 2019-nCoV Assay reports positive results only when all the target genes (E, RNA-dependent RNA polymerase (RdRP), and N) are amplified. The cycle thresholds (Cts) of RT-PCR-positive specimens are presented in Supplementary Table S2. The respiratory specimens were enrolled in this study consecutively; however the numbers of the tested specimens were different because the evaluation period for each RDT was different. This study was approved by the Institutional Review Board of the CAUH (IRB no.2012-009-439). This study was performed in accordance with the relevant guidelines and regulations, and the need for informed consent was waived according to the IRB (Institutional Review Board of the Chung-Ang University Hospital) policy.

We performed binomial logistic regression analysis for the evaluation of the clinical compliance of each SARS-CoV-2 RDT [7]. The logistic regression models were constructed based on the binomial results of the RDTs and the Ct of each of the three target genes included in the RT-PCR assay. Using the logistic regression analysis, we could predict the functions between the RDT results and the Ct value of each target gene, and estimate the Ct values at which 50% and 95% detection rates were achieved (Ct50 and Ct95). All statistical analyses were performed using R version 4.3.3 (http://www.R-project.org/).

Results

We enrolled 316 specimens obtained from COVID-19-confirmed cases and 166 screening specimens that had previously tested negative for SARS-CoV-2 by a PCR assay during the study period. The numbers of tested NP/OP specimens for each RDT are summarized in Table 1 . The results of binomial logistic regression analyses are presented in Fig. 1 (a, c, and e). The median values of Ct50 and Ct95 for seven RDTs were in the ranges of 24.3–30.9 and 19.3–22.6 for the E gene, 25.5–31.5 and 20.9–24.0 for the RdRP gene, and 26.8–32.3 and 22.7–25.7 for the N gene, respectively (Fig. 1 b, d, and f).

Table 1

Characteristics of clinical specimens enrolled in this study.

Kit	Number of tested samples	Specimens from confirmed cases	Specimens from screening tests
A	322	200	122
B	309	143	166
C	210	60	150
D	322	200	122
E	199	57	142
F	360	210	150
G	199	57	142

Fig. 1

Logistic regression analyses for the percentages of the positive results of each SARS-CoV-2 antigen assay according to the cycle threshold of E (a), RdRP (c), and N (e) genes in the SARS-CoV-2 RT-PCR assay. Cycle thresholds for each target gene at which 50% (Ct50, open circle) and 95% (Ct95, closed circle) detection rates were achieved and their corresponding 95% confidence intervals (b, d, f).

Ct, cycle threshold value; RdRP, RNA-dependent RNA polymerase; RT-PCR, reverse-transcription PCR; SARS-CoV-2, severe acute respiratory syndrome coronavirus.

Discussion

In this study, we assessed the clinical compliance of seven RDTs that have recently become available on the local market. These assays were officially cleared by the Ministry of Food and Drug Safety for export, and the advantages of these devices, such as the short turnaround time, user-friendliness, and cost-effectiveness, are expected to effect major changes in the screening and diagnosis of SARS-CoV-2 infections, especially in resource-limited countries [3,8].

It is well known that RDTs are less sensitive than RT-PCR, and the negative results of RDTs cannot rule out SARS-CoV-2 infection confidently, especially when the amount of virus in the respiratory specimens is low [2,[9], [10], [11]]. For specimens with Ct > 30, the false-negative rate increases by more than 50% for most of the RDTs according to the results of the binomial logistic regression analyses. However, all RDTs are capable of detecting the SARS-CoV-2 viral antigen accurately in NP/OP swabs for specimens with high viral load (Ct < 20). At the time of writing this article (May 2021), the medians and interquartile ranges (IQRs) of the Ct values were 23.7 (IQR 17.7–28.1) for the E gene, 24.3 (IQR 18.6–29.1) for the RdRP gene, and 24.8 (IQR 19.9–29.7) for the N gene in the SARS-CoV-2-positive specimens from patients diagnosed with COVID-19 for the first time in the CAUH. If the Ct95 value is regarded as the lower limit of detection, the RDTs tested in our study will only detect between 40.5% and 54.2% of RT-PCR-positive samples from COVID-19-confirmed cases (A: 54.2%, B: 45.2%, C: 41.3%, D: 41.3%, E: 47.1%, F: 47.1%, G: 40.6%). Our results indicate that these RDTs would be helpful for diagnosing SARS-CoV-2 infections with active virus replication in the upper respiratory tract. However, these RDTs should not be used for screening without the RT-PCR assay because they may not identify approximately half of the infected patients in real-world settings.

Despite the significant results demonstrating the clinical compliance of these seven COVID-19 RDTs, there are several limitations to this study. First, although there was some specimen overlap among the RDTs, all specimens were not identical for all the RDTs. Therefore, the RDTs evaluated in our study could not be analyzed using the same specimens, which may have affected the results. Second, we also used specimens from COVID-19-confirmed patients in the convalescent phase, and it is possible that the relatively lower amount of antigen in the NP/OP specimens would have influenced the results.

Conclusion

We evaluated seven commercially available SARS-CoV-2 RDTs, and these assays showed acceptable compliance only in specimens with high viral burden (Ct < 20). Although there is a global need for rapid and accurate diagnosis of SARS-CoV-2 infection using RDTs, our results suggest that “RDT-only”-based diagnostic testing should not be used for COVID-19 screening for asymptomatic patients because of their high false-negative rate.

Authors' contributions

Lee MK conceived of the presented idea and supervised this work. Kim TH and Kim HR and performed the computations and designed the figures. Lim YK and Kweon OJ wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Funding

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (Ministry of Science and ICT) (No. 2020R1A5A1018052). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Conflicts of interest

The authors have declared that no competing interests exist.