Comparison of commercial SARS-CoV-2 surrogate neutralization assays with a full virus endpoint dilution neutralization test in two different cohorts.

Determination of neutralizing antibody titers is still considered the gold standard for infection protection. A full virus neutralization test (VNT) with replication-competent, infectious SARS-CoV-2, is labor-intensive and requires Biosafety Level 3 certified laboratories. Therefore, several commercial SARS-CoV-2 surrogate virus neutralization tests (sVNTs) have been developed that aim to detect neutralizing antibodies targeting the receptor binding domain (RBD) of the viral spike glycoprotein (S). Neutralizing antibodies to the RBD block its interaction with the angiotensin-converting enzyme 2 (ACE2) receptor protein. Here, we compared a full virus neutralization test (VNT) with two SARS-CoV-2 surrogate virus neutralization tests (sVNT) and validated them in two cohorts of i) convalescent SARS-CoV-2-infected individuals and ii) COVID vaccinated individuals. The sVNTs showed highly different results both, compared to the VNT-titers and also between the two cohorts. This indicates that currently, sVNT provide a qualitative instead of a quantitative measurement of neutralizing antibodies. The findings in this work show that the cutoff levels for sVNTs might need to be readjusted for convalescent and vaccinated individuals.

Introduction

SARS-CoV-2, which belongs to the species SARS-related coronavirus (SARSr-CoV), was identified as the causative agent of COVID (Coronaviridae Study Group of the International Committee on Taxonomy of, 2020, Zhou et al., 2020) which was first recognized in December 2019 in Wuhan, China and has since spread to all parts of the world (Wang et al., 2020). Within the viral spike-protein (S), which is responsible for both receptor binding and glycoprotein mediated-fusion of the virus and host cell membranes, the receptor binding domain (RBD) mainly mediates its interaction with the host cell receptor (Shang et al., 2020, Yan et al., 2020). Epitope mapping showed that the majority of neutralizing antibodies are directed against the viral RBD as well as the N-terminal domain (NTD) of the spike protein, indicating that both regions are highly immunogenic (Liu et al., 2020). Furthermore, the RBD is the main target for many antiviral compounds (Letko et al., 2020).

Various assays are performed for the detection of neutralizing antibodies, using either replicating infectious virus (Mendoza et al., 2020) which requires Biosafety Level 3 certified laboratories or viral vectors pseudotyped with the spike glycoprotein (Crawford et al., 2020). Both, the conventional neutralizing assays as well as the pseudotyped viral vector–based assays are multistep procedures and labor-intensive. Meanwhile, several SARS-CoV-2 surrogate virus neutralization test (sVNTs) which have been developed based on similar principles, mainly antibody-mediated blockage of the interaction between the ACE2 receptor protein and the RBD (Kruttgen et al., 2022, Meyer et al., 2020). These sVNTs are less labor-intensive automated or semi-automated and can be used in routine laboratories. Several studies have shown that sVNT assays have good comparability to conventional NTs in terms of sensitivity and specificity (Abe et al., 2020, von Rhein et al., 2021). It is important to note, however, that the development of these tests and most of the related clinical studies were performed in the pre-COVID vaccination era. By now it is known that COVID vaccines elicit SARS-CoV-2-neutralizing antibody responses similar to or higher than the geometric mean neutralizing antibody titer after natural infections (Walsh et al., 2020). Here, we therefore compared the results with a full virus neutralization test (VNT) and validated two commercially available sVNTs in two different cohorts, i) convalescent SARS-CoV-2 infected individuals and ii) COVID vaccinated individuals.

Materials and methods

Characteristics of the study population were summarized in Table 1. The first group included a total of 71 residual blood samples of COVID-19 convalescents, donated in the period between November 2020 and March 2021. None of the convalescents was hospitalized due to the SARS-CoV-2 infection indicating a less severe clinical course. Ethical approval was obtained from the ethical committee of the HDZ NRW in Bad Oeynhausen (Reg. No. 670/2020). Informed consent was obtained from all plasma donors prior to blood sampling. The second group included blood samples – collected between January and February 2021 – from volunteers, young and elderly vaccinees (< 60/> 80 years of age) (n = 75) after a prime and boost vaccination campaign using BNT162b2 (Comirnaty®¸ BioNTech/Pfizer) and were collected between January and February 2021. The local ethics committee at Heinrich-Heine University Düsseldorf, Germany approved the study (study 2021–1287). Informed consent was obtained from all volunteers before sampling. This group was part of a SARS-CoV-2 vaccination study as previously published (Müller et al., 2021a, Müller et al., 2021b). Additionally, a smaller group of control sera from non-COVID individuals (n = 18) were obtained from healthy staff members. Both cohorts were tested for SARS-CoV-2 infection by VNT as well as by anti-Spike (DiaSorin Trimeric Spike IgG Assay) and anti-N-ELISA (Abbott Architect SARS-CoV-2 Nucleocapsid IgG). Members of the control group had no underlying chronic diseases or immunosuppressive therapies (data shown in Supplementary material).

Table 1

Characteristics of study populations.

Characteristics	Convalescents	Vaccinated	p	Control group
Total N (%)	71 (100)	75 (100)		18 (100)
Gender			0.83
Male N (%)	23 (32)	22 (29)		7 (39)
Female N (%)	48 (68)	53 (71)		11 (61)
Mean years (CI)	29.75 (27.2–22.3)	60.6 (55.3–66.0)	< 0.0001	40.5 (29.6–51.3)
Days after 2nd vac. (mean, CI)	n.a.	13 (12.1–13.9)		n.a.
Days after infection (mean, CI)	53 (48.7–57.1)	n.a.	< 0.0001*	n.a.
Date of sample	11/2020–3/2021	01/2021–04-2021		02-2021

n.a. = not applicable * days after 2nd vaccination compared with days after first symptoms of infection,

Samples were tested on the sVNT EUROIMMUN SARS-CoV-2 NeutraLISA assay (Lübeck, Germany) (test1). The underlying molecular basis is blocking the interaction between the SARS-CoV-2 HRP-conjugated RBD and the ACE2 receptor by neutralizing specific antibodies from patients’ serum. All samples were also tested by the sVNT ELISA from GenScript (Piscataway Township, USA), which has the same underlying molecular basis as the EUROIMMUN NeutraLISA (Tan et al., 2020) (test2). The tests were performed as specified by the manufacturer. In both tests, results are given semiquantitatively as the percentage of inhibition. The VNT cell culture assay was performed as previously described (Müller et al., 2021a, Müller et al., 2021b). In brief, a virus stock solution with a SARS-CoV-2 B.1 isolate (EPI ISL 425126) (Walker et al., 2020) was added to a final concentration of 100 TCID50/50 µl to heat-inactivated (56 °C) and serial diluted serum samples. The serum neutralization titer was determined by microscopic inspection as the highest serum dilution without a virus-induced cytopathic effect. All samples were tested in duplicate.

Results

Determination of sensitivity and specificity of the surrogate virus neutralization assays (sVNTs)

In order to assess the performance of the two sVNT against the conventional VNT, we analyzed 71 convalescent sera (64 positives, 7 negatives in VNT) and 75 post-vaccination sera (63 positives, 12 negatives in VNT). For the convalescent group, Test 1 (sVNT EUROIMMUN SARS-CoV-2 NeutraLISA) falsely identified 10 out of 64 positive sera as negative (sensitivity 0.844, (54/64)). There were six sera which showed equivocal results in the area between 25 % and 35 % binding inhibition, as defined by the manufacturer. No false positive results were found (specificity 1.0 (7/7)) ( Table 2 and Fig. 1A). Test 2 (sVNT ELISA from GenScript) falsely identified 3 out of 64 positive sera as negative (sensitivity 0.953 (61/64)). Here, 5 false positive results were found (specificity 0.286 (2/7)) (Table 2 and Fig. 1B).

Table 2

Sensitivity and specificity of the surrogate virus neutralization test (sVNT) as compared to the conventional neutralization test (VNT).

	Convalescent		Vaccinated		Control group
	Sensitivity	Specificity	Sensitivity	Specificity	Specificity
Test 1	0.844 (54/64)	1.000 (7/7)	1.000 (63/63)	0.667 (8/12)	1.000 (18/18)
Test 2	0.953 (61/64)	0.286 (2/7)	1.000 (63/63)	0.500 (6/12)	0.824 (15/18)

Fig. 1

Correlation between percent inhibition in the surrogate virus neutralization test (sVNT) and titers in conventional neutralization test (VNT) for COVID-19 sera of convalescent and vaccinated individuals. Correlation and linear regression analyses were performed in GraphPad Prism using Pearson’s correlation coefficients. Statistical significance was calculated using the two-tailed test. The data presented are the log of the neutralization titer for VNT and the % inhibition sVNT. Negative results in VNT are not shown. The dashed lines indicate the standard deviations of the linear regression plots A,B, correlation analysis of sera of convalescent individuals by sVNT and VNT in Test 1 (A) and Test 2 (B). C,D, correlation analysis of sera of vaccinated individuals by sVNT and VNT in Test 1 (C) and Test 2 (D). The dashed curves indicate the standard deviations of the linear regression plots. The dotted lines indicate the sVNT cutoffs of grayzone at 25 % and 35 % for Test 1 (A and C) and cutoff of 30 % for Test 2 (B and D), respectively.

For the post-vaccination group, test 1 identified 0 out of 63 positive sera as negative (sensitivity 1.0 (63/63)) and no sera were equivocal in the area between 25 % and 35 % binding inhibition. There were four false positive results (specificity 0.667 (8/12)) (Fig. 1C). Test 2 did not falsely identify positive sera as negative (sensitivity 1.0 (63/63)) but 6 false positive results were found (specificity 0.5 (6/12)) (Table 2 and Fig. 1D).

Furthermore, 18 negative sera were also tested in both sVNTs. While Test 1 identified all sera as negative (specificity 1.0 (18/18)), Test 2 showed three positive results with inhibition values of 30 %, 31 % and 32 % which is considered as indeterminate (specificity 0.824 (15/18)) (Table 2).

Correlation of the results obtained with surrogate and conventional neutralization assays

In order to evaluate whether the two semiquantitative sVNTs can be used to estimate the titer of neutralizing antibodies, we investigated the correlation between VNT values and binding inhibition values obtained from the sVNTs. For the positive samples from the convalescent group, both sVNTs showed a moderate correlation to the results of VNT (R2 = 0.368 and R2 = 0.397) with a broad range of sVNTs-values when compared to each dilution step in VNT (Fig. 1A and B). In contrast to the convalescent group, sera from the vaccinated group showed binding inhibition values towards the upper limit of quantification in both sVNTs leading to a narrow range of sVNTs-values when compared to each dilution step in VNT (Fig. 1C and D). This leads to similar correlation coefficients of R2 = 0.414 for Test 1 and R2 = 0.485 for Test 2 respectively.

Performance of sVNT ELISAs on convalescent patients versus vaccinees

In order to focus on the point that the two sVNTs produce different values depending on the study population, we divided both cohorts into 4 groups, depending on their VNT-titer: Group 1: titers < 10, group 2: titers > 10 to < 40, group 3: titers > 40 to < 160, group 4: titers > 160 ( Fig. 2). Test 1 showed highly variable inhibition percentage values between convalescents and vaccinees in titer groups 2–4 (p < 0.0001) (Fig. 2A). Similarly, Test 2 showed highly variable inhibition percentage values between convalescents and vaccinees in titer groups 2 and 4 (p < 0.001) (Fig. 2B). Taken together, both sVNTs binding inhibition values were significantly higher in vaccinees compared to convalescent patients.

Fig. 2

Comparison of virus neutralization test (sVNT) results in convalescent and vaccinated individuals. Box plots showing percent inhibition in the surrogate virus neutralization test (sVNT) stratified by results of sVNT %inhibition/VNT50 titre in convalescent (C) and vaccinates individuals (V) in Test 1 (A) and Test 2 (B). ns = not significant, ***p = 0.001, **** p < 0.0001. The data were analyzed using GraphPad Prism 8.02 in an unpaired t-test.

Comparison of the results obtained with surrogate neutralization assays and anti-spike-ELISA and anti-nucleocapsid-ELISA

Results obtained in the sVNT were compared with antibody levels in anti-spike-ELISA and anti-Nucleocapsid-ELISA. A correlation could be seen between sVNT and spike antibodies, while there was no correlation between sVNT and Nucleocapsid antibodies (see Supplementary results, Fig. S1).

Discussion

Surrogate virus neutralization tests, such as the EUROIMMUN SARS-CoV-2 NeutraLISA (Test 1) or GenScript ELISA (Test 2), have the advantage of simpler and faster detection of neutralizing antibodies in serum samples compared to the conventional VNT. They do not need BSL-3 facilities and are less labor-intensive than the conventional VNT. Previous studies have described a moderate to good correlation of these sVNT results with antibody titers measured by VNT (Abe et al., 2020, Tan et al., 2020, von Rhein et al., 2021). However, it should be noted that these tests were developed in the second half of 2020, i.e. at a time when there was no SARS-CoV-2 vaccine available and the alpha mutant was dominant. The isolate Wuhan Hu-1 served as the basis for the RBD of the spike protein used in commercial assays. In contrast to the reports of others (Abe et al., 2020, Kruttgen et al., 2022), the sVNT results for convalescent sera are not satisfactory as shown in this study. We observed a relatively high rate of false negatives and generally a broad range of values in the sVNT compared to the VNT. Additionally, Test 2 showed false positive results and thus, a low specificity (0.286 (2/7)). This could be further confirmed by the testing of sera of uninfected individuals, where 3 out of 18 sera were identified as false positive specificity 0.824 (15/18). Since the GenScript cPass sVNT was one of the first surrogate neutralization tests that received FDA emergency use approval, studies assessing its analytical performance rapidly followed. To study specificity and sensitivity of the assay, Jung and colleagues used a cohort of 25 RT-PCR positive individuals and 10 RT-PCR negative individuals. Interestingly, they report a sensitivity of 96% (24/25) and a specificity of 100 % (10/10) which is in contrast to our results (Jung et al., 2021). In a study conducted by Taylor and colleagues, the assay also provided highly sensitive and specific results. However, six samples from PCR positive individuals were classified as negative by cPass. Here, Taylor et al. suggest that these results are in fact true negatives since these individuals apparently did not seroconvert according to total IgG assays (Taylor et al., 2021).

Meanwhile, as of June 2022, more than half of the population in Europe has received a primary course of SARS-CoV-2 vaccinations plus at least one booster vaccination according to European Centre for Disease Prevention and Control COVID-19 Vaccine Tracker (accessed on June 9th 2022) and the predominant virus variant has shifted towards the Omicron variants. It has been shown that COVID-vaccines elicit higher titers of SARS-CoV-2 neutralizing antibodies than natural SARS-CoV-2 infections (Walsh et al., 2020). The same is true for total anti-SARS-CoV-2 IgG–titers, where, in our experience, the antibody titers of vaccinees are above the upper limit of quantification. Reasons for this could be, that the induction of the humoral immune response by vaccination follows a different route than by natural infection. Furthermore, while in natural infections antibodies are generated against a broad range of virus proteins and epitopes, in vaccinees only antibodies against the spike protein are produced (Haynes et al., 2021).

Our data clearly showed that sVNTs yielded highly variable results in convalescents compared to vaccinees when they are correlated to the VNT-titers. One possible way to resolve this might be to separately define cutoff levels for sVNTs inhibition in convalescent and vaccinated individuals. This may not be easily implemented in routine diagnostics, the main application for high throughput assays such as sVNTs, due to automated processes. The results from this study suggest that sVNTs in vaccinees may be considered as qualitative rather than quantitative measurement of neutralizing antibody capacity. Furthermore, the combinatorial use of neutralization assays and the assessment of total SARS-CoV-2 specific antibody titers can provide an approximation of the individuals’ immune status post-vaccination or post-infection. It is expected that an increasingly complex picture of immune responses due to vaccination with different vaccines, as well as natural infection, will arise in the population. Without constant adaptation and careful interpretation of the results of sVNTs we see the individual use of these tests for monitoring the SARS-CoV-2 specific immune response as problematic.

CRediT authorship contribution statement

Conceptualization: O. A., M. A., J.T. and L. M. Formal analysis: O. A., N. L. and L. M. Investigation: L. M., M. A., D.H., H.S., O.A. Writing – original draft: O. A., M. A., and L. M. Writing – review & editing: L. M., M. A., S. H., D. H., N. L., J. T., L. M. O. A., and H. S. Supervision: O. A., M. A., and H. S.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.