Evaluation of a SARS-CoV-2 lateral flow assay using the plaque reduction neutralization test.

As new tests and technologies advance our understanding and diagnostic capabilities of the severe acute respiratory syndrome coronavirus 2 and the coronavirus disease 2019, they must be appropriately validated to make sure test performance is following manufacturer claims. In this study, we evaluated the Vazyme 2019-nCoV IgG/IgM Detection Kit, which is a lateral flow assay (LFA), by the plaque reduction neutralization test (PRNT) using 100 patient plasma/serum samples. As compared to the PRNT results, the Vazyme LFA had 95.9% sensitivity and 96.1% specificity. Along with the increased need for rapid, effective, and affordable point of care tests to help provide meaningful epidemiological data, we demonstrated that the Vazyme LFA performed well on IgG detection but cannot be judged on the performance of IgM detection using PRNT alone. However, our observation of the low IgM-positive rate supported the poor performance of IgM detection of this LFA which led to the disapproval of its Emergency Use Authorization recently.

Introduction

The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), an emerging virus, has caused a global pandemic of coronavirus disease 2019 (COVID-19). Real-time, reverse transcriptase-polymerase chain reaction (RT-PCR) detection of SARS-CoV-2 nucleic acid in clinical specimens is the current diagnostic standard (Loeffelholz and Tang, 2020). While PCR is useful for the diagnosis of active infection, it lacks information regarding possible SARS-CoV-2 infection prior to testing. To address this need, various types of serologic tests have been developed to detect SARS-CoV-2 immunoglobulin (Ig)G and IgM antibodies, indicating prior infection (Li et al., 2020, Wu et al., 2020, ). Serological testing has proven to be an important component of the overall estimate of SARS-CoV-2 disease incidence and prevalence (Wu et al., 2020). These tests can help determine whether a person has been infected with SARS-CoV-2 and may help determine immune-status. While antibody tests do not diagnose active SARS-CoV-2 infection, they provide more realistic epidemiological information. In turn, this can help with vaccine evaluation, public health planning, and quarantine strategies (Okba et al., 2020).

In response to this need, many SARS-CoV-2 lateral flow, point-of-care antibody tests have been developed (Wu et al., 2020). Lateral flow assays (LFAs) are a rapid, cost-effective, and easy-to-use solution for facilities without the resources necessary for advanced platforms (Li et al., 2020). However, due to the emergent nature of the pandemic, many of these tests have not been thoroughly validated nor have they been approved by regulatory agencies (Okba et al., 2020). This has resulted in the distribution of tests that may or may not function according to the manufacturers’ claims and performance assessments are limited (Okba et al., 2020).

The plaque reduction neutralization test (PRNT) is the current gold standard for SARS-CoV-2 serology tests (Okba et al., 2020). Even so, most of the published literature evaluating SARS-CoV-2 LFAs, do not use PRNT as the comparison method (Li et al., 2020, Montesinos et al., 2020, Wu et al., 2020) due to its low throughput and biosafety level III restrictions. In this study, we evaluated 2019-nCoV IgG/IgM Detection Kit (Colloidal Cold-Based) (Nanjing Vazyme Medical Technology Co., LTD, China) for its sensitivity, specificity, and cross-reactivity using a PRNT assay. During the early stage of the COVID-19 pandemic, we searched for a reliable LFA to support a serosurveillance study. At that time, the Vazyme test was among a few LFAs under application for the Federal Drug Administration (FDA) Emergency Use Authorization (EUA). We, therefore, evaluated the Vazyme LFA test using the PRNT assay. After completing our evaluation, we found that the FDA did not approve the EUA for this test because of its poor IgM performance. Here we report our independent evaluation results of the LFA test.

Materials and methods

Samples

Serum/plasma samples (100) used in this study were either from patients with SARS-CoV-2 RT-PCR tests done or from collections before the COVID-19 pandemic. For assay cross-reaction evaluation, serum/plasma samples that were positive for other infections, antibodies, or vaccinations were used (Table 1 ). To assess interference, serum samples containing a certain amount of albumin or elevated cholesterol/rheumatoid factors were included as well (Table 1). The research protocols regarding the use of human serum/plasma specimens were reviewed and approved by the University of Texas Medical Branch Institutional Review Board (IRB).

Table 1

Antibody, infection, and vaccine status of individual patients with negative SARS-CoV-2 PRNT results.

Immune serum/plasmaa and interfering substances	Number of samples	Number tested positive on Vazyme LFA
bAlbumin 4.5 g/dL	3	0
Antinuclear antibodies (ANA)+	4	0
Anti-Cytomegalovirus IgG+	5	1
Anti-Epstein Barr Virus capsid or nuclear antigen IgG+	4	0
Anti-Hepatitis A virus Ab+	2	0
Anti-Hepatitis B virus surface antigen Ab+	9	0
Anti-Hepatitis C virus Ab+	4	0
Anti-Human immunodeficiency virus 1 Ab+	4	0
Cryptococcus neforomans Antigen+	1	0
Anti-Herpes simples virus 1 IgG+	3	0
Anti-Herpes simples virus 2 IgG+	2	0
Anti-Measles virus IgG+	5	0
Anti-Mumps virus IgG+	1	0
Anti-Parvovirus B19 IgG+	3	0
Anti-Parvovirus B19 IgM+	1	0
Anti-Rubella virus IgG+	6	0
Anti-Syphilis IgG+	4	0
Anti-Typhus Fever IgG+IgM+	1	0
Anti-Varicella zoster virus IgG+	9	0
Anti-West Nile virus IgG+	3	0
bElevated blood cholesterol	3	0
bElevated Rheumatoid Factor	3	0
cHuman coronavirus 229E	1	0
cHuman coronavirus HKU1+	1	0
cHuman coronavirus NL63+	1	0
cHuman coronavirus OC43+	1	0
cHuman Rhino/Enterovirus+	2	0
cInfluenza B+	1	0
cParainfluenza virus 4+	1	0
Yellow fever virus post-immunization	2	0

The immune sera are listed in alphabetical order. Samples tested positive for antibodies against specific pathogens are indicated with the prefix “anti,” whereas samples tested positive on antigens or pathogen nucleic acids are not indicated with the prefix.

Specimens with interfering substances.

Specimens collected within 1 to 6 months after PCR tested positive.

PRNT

A standard double-layer PRNT was performed with each serum/plasma sample (Muruato et al., 2020). Specifically, serial dilutions of serum/plasma samples (1:20) for the first dilution followed by serial 1:2 dilutions were mixed with an equal amount of wild-type SARS-CoV-2 virus suspension containing 100 plaque-forming units in 100 µL. After incubating the mixtures at 37°C for 1 hour, each virus-serum/plasma mixture was inoculated onto one well of a 6-well tissue culture plate containing a confluent monolayer of Vero E6 cells. After incubating the plate at 37°C for 1 hour, an agar overlay was added to the infected cell monolayer, and the plate was further incubated at 37°C for 2 days. When virus plaques became visible, a second overlay containing neutral red was added and incubated for 5 hours. Then neutral red was removed and plaques were counted. The antibody titer was determined as the serum/plasma dilution that inhibited 50% of the tested virus inoculum (PRNT50).

LFA

The assay was done following the manufacturer's instructions. Briefly, the test kits were warmed to room temperature, removed from the foil pouch, and placed horizontally on a flat surface. Using the provided dropper, 1 drop (∼20 µL) of serum/plasma and 3 drops (∼60 µL) of dilution buffer were added to the sample loading position. After 10 minutes, the test results were read, analyzed, and photographed.

mNeonGreen reporter SARS-CoV-2 virus neutralization test

This assay is a modified PRNT using mNeonGreen reporter SARS-CoV-2 instead of the wild-type virus in the neutralization test (Xie et al., 2020). Briefly, after infecting cells with the mixture of serum/plasma and reporter virus, the plates were incubated for 16 hours. Intracellular mNeonGreen fluorescence signals were measured using CytationTM 7 (BioTek, Winooski, VT). The infections without serum/plasma treatment were used as non-treatment controls. mNeonGreen fluorescence signals from the non-treatment controls were set at 100%. mNeonGreen fluorescence signals from each serum/plasma-treated samples were normalized to those from the non-treatment controls. The neutralization titers were defined by the serum/plasma dilution fold at which suppressed 50% of the mNeonGreen fluorescence signals of the non-treatment control (mNG-NT50).

Statistical analysis

Statistical analysis was performed using Medcalc stats (http://www.medcalc.org/calc/diagnostic_test.php).

Results

Fig. 1 displays the representative results of the 2019-nCoV IgG/IgM Detection Kit. None of the specimens resulted as an invalid test which would be indicated by a lack of the control line (top line). The second line shows IgG results and the bottom line shows IgM results. Any faint band was considered to be positive during this study. Although the package insert describes whole blood as a valid specimen type to be used, whole blood was not evaluated in this study.

Fig. 1

Pictures of representative lateral flow assay test results.

As compared to the PRNT results, the Vazyme 2019-nCoV IgG/IgM Detection Kit had 95.9% sensitivity (95% confidence interval [CI]: 86.0 to 99.5%) and 96.1% specificity (95% CI: 86.5 to 99.5%). Of the 49 positive Vazyme LFA tests, only 10 were positive for IgM antibodies and 9 out of those 10 samples were also positive for IgG. There were two samples, one with IgM(+) IgG(–) and one with IgM(+) IgG(+) that were negative by PRNT (Table 2 ). Of the 24 samples collected before the SARS-CoV-2 pandemic, none were positive for antibodies by either detection method.

Table 2

Overall comparison between 2 antibody detection methods.

Assay	PRNT (+)		PRNT (-)		Total
Vazyme LFA IgG and/or IgM (+)	47		2		49
IgG (+)		47		1	48
IgM (+)		8		2	10
Vazyme LFA (−)	2		49		51
Total	49		51		100

In the early stages of disease (days 0 to 5), the Vazyme LFA detected antibodies in only 60% of the samples. In the intermediate and later stages of disease (6+ days) the assay detected antibodies in 100% of the positive samples (Table 3 ). The discrepant results are further detailed in Table 4 . A newly developed mNG-NT (Muruato et al., 2020) was used as the third method to evaluate the discrepant results between PRNT and Vazyme LFA. Samples 596, 175, and 206 had consistent results between PRNT and mNG-NT while sample 733 had the same results between Vazyme LFA and mNG-NT. Therefore, based on additional available information on the number of days post-SARS-CoV-2 nucleic acid test that the serum/plasma samples were collected from the individuals, sample 596 with IgM positive by Vazyme LFA was most likely a false-positive due to the cross-reaction with Cytomegalovirus (CMV) IgG. Sample 733 was most likely to be a true IgG and IgM positive although PRNT was negative, while samples 175 and 206 were most likely to be false-negative by Vazyme LFA (Table 4).

Table 3

Vazyme LFA results distributed by days post-positive nucleic acid test.

Days post-NA+	No. of samples	LFA IgM(+)	LFA IgG(+)
0-5	10	2	7
6-10	14	2	14
11-15	15	5	15
16-20	6	0	6
20+	6	0	6

Table 4

Discrepant results.

Sample ID	PRNT titer	Vazyme LFA results	mNG-NT50 titer	Notes
596	<20	IgM (+)	<20	NA−a, CMV IgG+
733	<20	IgG and IgM (+)	38	Post-NA+5db
175	20	negative	66	Post-NA+0d
206	80	negative	115	Post-NA+0d

NA−: SARS-CoV-2 nucleic acid test negative.

Post-NA+= number of days since positive SARS-CoV-2 nucleic acid test.

Table 1 lists the common antibodies and common interfering substances encountered in human serum/plasma. A few specimens collected within 1 to 6 months of post-PCR positives for the common respiratory viral pathogens were also included. Vazyme LFA only produced one false-positive result.

Discussion

Immunoassays are an important component of SARS-CoV-2 testing and offer information about the true prevalence of the disease. We have demonstrated that the Vazyme LFA accurately detects SARS-CoV-2 antibodies with almost no evidence of cross-reactivity with other antibodies or common interfering substances except possible cross-reaction to CMV IgG (1 out of 5, 20%) in our cases. Architect SARS-CoV-2 IgG assay (Abbott, Abbott Park, IL), a chemiluminescent microparticle immunoassay, also reported 1 of 5 CMV IgG positive samples was tested to be SARS-CoV-2 IgG positive (Abbott Architect SARS-CoV-2 IgG assay package insert). However, the target for the antibody in Architect SARS-CoV-2 IgG assay is SARS-CoV-2 virus nucleocapsid protein, while Vazyme LFA targets the spike protein. To our knowledge, this is the first report to evaluate a LFA using the gold standard serology reference method PRNT while previous studies compared LFA to enzyme-linked immunosorbent assay (ELISA) (Montesinos et al., 2020, Tollånes et al., 2020, Wu et al., 2020). The most significant advantage of PRNT is detecting and measuring antibodies capable of neutralizing a target virus, while ELISA and other immunoassays cannot determine this neutralization capability.

While Vazyme LFA detected SARS-CoV-2 IgG with high sensitivity and specificity it did not appear to perform well with IgM detection. Our observation is in agreement with the poor IgM performance of this LFA, which led to the disapproval of its EUA by the FDA (https://www.accessdata.fda.gov/cdrh_docs/presentations/maf/maf3278-a001.pdf). Studies have shown that a positive IgM result can be detected as early as the fourth day after symptom onset of COVID-19 and last for at least a few weeks (Tollånes et al., 2020). Therefore, IgM should be present in most of the positive serum/plasma samples used in this study since about 70% of the samples were collected from symptomatic patients that were 6 to 20 days post-positive SARS-CoV-2 nucleic acid detection. However, IgM was only detected in 9 samples (18%). Unfortunately, PRNT only measures total antibodies and cannot differentiate between IgG and IgM, unless one type of Ig is depleted from serum/plasma samples before performing the assay. However, the depletion experiment was not performed in this study. Therefore, the sensitivity and the specificity of Vazyme LFA on IgM cannot be evaluated accurately. Despite our findings, other SARS-CoV-2 LFAs have also shown lower sensitivity for IgM antibodies, and this may be due to the lack of strength in the initial IgM signal during the early stages of SARS-CoV-2 infection (Sheridan, 2020). Since not all spike antibodies could neutralize virus infection, it is expected that higher titers of antibodies could be detected by spike protein-based binding assays (e.g., LFA, ELISA, or Luminex) than those detected by PRNT. However, there should be a correlation between the titers detected by the spike protein-based binding assays and the PRNT assay. Indeed, our results support such concordance. However, antibody response can be unpredictable and there is still much to learn about human responses to SARS-CoV-2 infection (Sheridan, 2020).

Although RT-PCR detection of SARS-CoV-2 nucleic acid in clinical specimens is the current diagnostic standard, this technology requires specialized instrumentation and highly trained personnel. It is usually performed only at larger hospitals, reference, and public health laboratories (Klein, 2002). Unfortunately, smaller hospitals, doctor's offices, and clinics may not perform complex testing on site. Many of these locations opt to send patients to larger hospitals for testing, often delaying patient care, treatment, and results. Rapid, effective, and affordable alternatives to RT-PCR and ELISA are necessary to help integrate decentralized testing and provide meaningful epidemiological data to public health officials during this unprecedented pandemic (Sheridan, 2020). Therefore, it is important to appropriately assess new SARS-CoV-2 assays and many of the point-of-care LFAs that have not been adequately tested. To meet the diagnostic need of the COVID-19 pandemic, many companies and academic labs developed different diagnostic platforms before or when the regulatory agency has stipulated or refined the standards for diagnostic products. Thus, thorough evaluations of these products are essential to ensure the accuracy of these diagnostic products. Early guidance from the regulatory agency is critical to improving the development of diagnostic tests, particularly during a pandemic crisis. In this study, we have demonstrated that the Vazyme LFA accurately detected SARS-CoV-2 IgG but not necessarily IgM. Therefore this LFA might be a reliable point-of-care test for IgG detection but not for IgM in facilities without resources for complex testing.

CRediT author statement

Xuping Xie: Methodology, Validation, Investigation, Writing - review and editing

Marisa C. Nielsen: Formal analysis, Data curation, Writing - Original draft, review and editing

Antonio E. Muruato: Investigation

Camila R. Fontes-Garfias: Investigation, Writing - review and editing

Ping Ren: Conceptualization, Resources, Formal analysis, Project administration, Writing - Original draft, review and editing, Visualization, Supervision, Project administration

Ethical approval

The project is approved by the the University of Texas Medical Branch Institutional Review Board (IRB) under an expedited review process with IRB # 20-0070 & 20-0143.

Informed consent

Not applicable.

Conflict of interest

All authors have no conflict of interest.