A method comparison of three immunoassays for detection of neutralizing antibodies against SARS-CoV-2 receptor-binding domain in individuals with adenovirus type-5-vectored COVID-19 vaccination.

OBJECTIVE: Detecting neutralizing antibodies targeting receptor-binding domain (RBD) is important for the assessment of humoral protection and vaccine efficacy after vaccination. We compared the performance of three surrogate immunoassays for detection of neutralizing antibodies targeting RBD.
METHODS: We analyzed 115 serum samples obtained from individuals with Ad5-vectored COVID-19 vaccination using two competitive enzyme-linked immunosorbent assays (Wantai BioPharm and Synthgene Medical Technology) and one competitive chemiluminescence assay (YHLO Biotech). Performance evaluation and methodology comparison were performed according to the Clinical and Laboratory Standards Institute related guidelines.
RESULTS: The precision met the manufacturers' statements. The linear range of the WANTAI was 0.0625-0.545 U/ml and the YHLO was 0.260-242.4 U/ml. The WANTAI's limit of blank (LoB) and limit of detection (LoD) were 0.03 and 0.06 U/ml, respectively. The YHLO's LoB and LoD were 0.048 and 0.211 U/ml, respectively. The correlations of semi-quantitative results of Synthgene with quantitative results of YHLO (ρ = 0.566) and WANTAI (ρ = 0.512) were medium. For YHLO and WANTAI, there was a good agreement (0.62) and a strong correlation (ρ = 0.931). Passing-Bablok analysis and Bland-Altman plot showed a positive bias (112.3%) of the YHLO compared to the WANTAI. The exclusion of samples >50 U/ml did not decrease bias.
CONCLUSION: These findings contribute to a deeper understanding of surrogate viral neutralization assays and provide useful data for future comparison studies.

INTRODUCTION

The outbreaks of coronavirus disease 2019 (COVID‐19) hit the world health, economy and society severely. Since June 2021, severe acute respiratory syndrome coronavirus‐2 (SARS‐CoV‐2) has caused more than 100 million infections and 3.6 million deaths.  Vaccines are one of the most effective approaches to prevent viral infection.  Nowadays, more than 200 vaccines for SARS‐CoV‐2 are being developed or in clinical trials. At least five vaccines including adenovirus‐vectored vaccine, inactivated virus vaccine, and recombinant proteins vaccine have been approved for clinical use by the Chinese Food and Drug Administration.

Serological testing played an important role in assessment of immunity in the vaccinated populations.  Most SARS‐CoV‐2 Ab detection assays are based on nanoparticle‐based lateral‐flow test (GNT) strip, enzyme‐linked immunosorbent assay (ELISA), chemiluminescence assay (CLIA), and electrochemiluminescence immunoassay (ECLIA). ,  The common antigens used as the target were spike (S) and nucleocapsid (N) due to the high immunogenicity.  These tests assist in evaluating the COVID‐19 vaccines. Recently, the receptor‐binding domain (RBD) of the SARS‐CoV‐2 S protein has been an emerging target.

Anti‐RBD antibodies, which are produced after vaccination, acted as the main neutralizing antibodies by blocking virus binding to the host angiotensin‐converting enzyme 2 (ACE2). The neutralizing antibodies targeting RBD levels have been used to evaluating humoral immune response following COVID‐19 vaccination. Nowadays, the serological surrogate immunoassays are being developed for the neutralizing antibodies targeting RBD. , , , ,  These surrogate viral neutralization assays (sVNTs) based on the same principle of competitive binding. , , Utilizing purified receptor‐binding domain from S protein and ACE2 receptor, these assays enable specific antibodies to block RBD binding to ACE2. Qualitative or quantitative determination could be achieved by immunolabeling. As the number of sVNTs is growing; however, the performances are not well‐known.

In this study, we compared the performances of three surrogate immunoassays including two competitive ELISA assay and one competitive CLIA assay for detection of the neutralizing antibodies targeting RBD in serums from vaccinated individuals. All three assays have received the mark on Conformité Européene. Methodology comparison and bias estimation were conducted.

METHODS

Subjects

From January 2021 through May 2021, 115 participants (78 males and 37 females; age 42.0 ± 7.5 years, range 20–68) who received one injection of recombinant adenovirus type‐5‐vectored COVID‐19 vaccine were well informed and enrolled in this study. Inclusion criteria and exclusion criteria of vaccination were as described (Table 1). Serum samples were collected 4 weeks after vaccination. Isolation of serum was achieved by centrifuging at 1700 g  for 10 min. Hemolyzed, icteric, or lipemic samples were excluded. Of the original 115 samples, finally, 92 had sufficient serum available for further analysis. Then, all samples were stored at −80℃ until analyzed. This study was performed with the approval of the Ethics Committee of Jinling Hospital (Approval No.: 2020DZWJWKT‐005).

TABLE 1

Inclusion criteria and exclusion criteria

Inclusion criteria	Exclusion criteria
Healthy adults, age range 18–60	Allergic to vaccine components
Negative for SARS specific IgM and IgG antibodies before vaccination	Received blood products in past 4 months
Negative for SARS‐CoV‐2 nucleic acid test before vaccination	Any history of mental illness or epilepsy

Serological testing of neutralizing antibodies targeting RBD

The neutralizing antibodies targeting RBD levels were measured by three different assays: the WANTAI SARS‐CoV‐2 NAbs ELISA Kit (Wantai BioPharm), the SARS‐CoV‐2 Virus Neutralization Test Kit (ELISA method) (Synthgene Medical Technology), and iFlash‐2019‐nCoV NAb CLIA assay (YHLO Biotech). The reagents and calibrators of individual assays were in the same lots. Samples were in parallel subjected to both ELISAs and CLIA assays according to the manufacturers’ instructions. The experiments were performed by certified medical laboratory technicians in one laboratory. For the two ELISA assays, the iMark microplate reader (BioRad) was used. For the CLIA, the chemiluminescence apparatus iFlash 3000 provided by YHLO was applied. Interpretation of results for the three assays were summarized in the Table 2. The WNATAI and YHLO assays are quantitative assays and the Synthgene assay is semi‐quantitative assay with qualitative interpretation.

TABLE 2

Results interpretation of assays for anti‐RBD antibodies

Manufacturer	Assay principle	Results interpretation	Reference
WANTAI	Competitive ELISA	Positive a : Binding inhibition rate ≥50% Substitute the A value into the regression equation to obtain the sample concentration	https://www.szabo‐scandic.com/en/wantai‐sars‐cov‐2‐nabs‐elisa‐neutralizing‐antibodies‐ce‐ivd
Synthgene	Competitive ELISA	Positive: Inhibition rate b ≥20%	http://en.syngenemed.com/product/64.html
YHLO	Competitive CLIA	Positive: ≥10 AU/ml	https://pdf.medicalexpo.com/pdf/shenzhen‐yhlo‐biotech‐co‐ltd/iflash‐2019‐ncov‐nab/107786‐233490.html

Abbreviations: CLIA, chemiluminescence assay; ELISA, enzyme‐linked immunosorbent assay.

Binding inhibition rate = (A value of Standard 0 U/ml − A value of specimen) × 100%/A value of Standard 0 U/ml.

Inhibition rate = (OD value of sample − OD value of negative control)/(OD value of positive control − OD value of negative control).

Positive : Binding inhibition rate ≥50%

Calibration protocol

For the WANTAI ELISA assay, calibration curve was performed according to the manufacturers’ instructions and fitted using four‐parametric logistic curves. For the CLIA assay, standard curve was re‐conducted by using a two‐fold serial dilution of 16 U/ml Wantai's kit standard to convert arbitrary units per millilitre (AU/ml) into units per millilitre (U/ml). The standard used is calibrated against NIBSC 20/136 standard18. The concentration of one Wantai unit (U/ml) is ≈25.7 IU/ml (NIBSC 20/136). Data were analyzed and plotted with GraphPad Prism 8.0 (GraphPad Software, Inc.). If the concentration of the SARS‐CoV‐2 neutralizing antibody targeting RBD in specimen exceeded the linear range, it is necessary to properly dilute the specimen with diluent.

Repeatability and within‐laboratory precision

The precision was evaluated using serum samples has analyte values near the concentrations the manufacturer used to establish the precision claims for the assay, by continuous measurement in quadruplicate for 5 consecutive days, according to the Clinical and Laboratory Standards Institute (CLSI) EP15‐A3 guideline. To validate the precision of assays, the repeatability, and intermediate precision were estimated through a one‐way analysis of variance and compared to the manufacturers’ claims.

Linearity assessment

Linearity assessment for the two quantitative assays was performed as described in the CLSI EP6‐A guideline. The sera sample with high (H) concentration was serially diluted with the low (L) concentration sample at ratios of L, 0.9L + 0.1H, 0.8L + 0.2H, 0.7L + 0.3H, 0.6L + 0.4H, 0.5L + 0.5H, 0.4L + 0.6H, 0.3L + 0.7H, 0.2L + 0.8H, 0.1L + 0.9H, H. These analytes took values equally spaced between them and the concentration range was 20% wider than the linear range reported by the manufacturers. All assay measurements were performed in triplicate. Then, a regression equation was calculated according to y = ax ± b, where y was the measured concentration and x was the expected concentration. When a ranged from 0.97 to 1.03 and R 2 was more than 0.95, with b closer to zero, it could be assumed that measurements were in the linear range.

Limit of blank and limit of detection

The limit of blank (LoB) and limit of detection (LoD) were determined according to the EP17‐A2 protocol. The initial LoB estimate was achieved with direct measurement of the zero‐level sample diluent (n = 20). Then, the desired concentration range of low‐level samples was identified as 1–5 times the initial estimated LoB. Five blank samples and five low‐level samples were detected by the two quantitative assays in four replicates over 3 days (n = 60). For the LoB assessment, the nonparametric analysis was used. The LoB estimate was calculated using the formula: LoB = X 57 + 0.5 × (X 57 − X58). For the LoD evaluation, the precision profile approach was adopted. The LoD was then calculated: LoD = LoB + 1.645 SD, where SD was estimated by the distribution of values measured in the serum pools with very low levels.

Statistical analysis

Methodology comparison and bias estimation were performed according to the CLSI EP9‐A3 protocol. For comparison, method data were analyzed and displayed. Statistical analysis was carried out with SPSS version 22.0 (IBM) and MedCalc Software (Mariakerke). The overall, positive, negative percent agreement and Cohen's κ coefficient were calculated to demonstrate the concordance between the three assays. κ values less than 0.40 mean poor agreement, those between 0.40 and 0.60 mean moderate agreement, those between 0.60 and 0.80 mean good agreement and those over 0.80 mean excellent agreement. Experimental data were analyzed using t‐test. Spearman's correlation coefficient was used as an assessment of the correlation between detected results of the three assays. A correlation coefficient was categorized as follows: |r| < 0.2, poor; 0.2 ≤ |r| < 0.4, weak; 0.4 ≤ |r| < 0.6, moderate; 0.6 ≤ |r| < 0.8, strong; 0.8 ≥ |r|, excellent.  Pairwise concurrence of the two quantitative assays was attained using Passing–Bablok regression models and Bland–Altman plots.

RESULTS

Standard curves

To quantitative the levels of the neutralizing antibodies targeting RBD in clinical samples, we first calculated standard curves of WANTAI and YHLO. The R 2 of WANTAI and YHLO was 0.9999 and 0.9905, respectively. The standard curves were presented in Figure 1A,B. As the ELISA from Synthgene were qualitative experiment, the results of Synthgene assay were only represented as negative or positive.

FIGURE 1

Calibration results. (A) Calibration curve for WANTAI ELISA. (B) Calibration curve for YHLO CLIA. CLIA, chemiluminescence assay; ELISA, enzyme‐linked immunosorbent assay

Repeatability and intermediate precision

Results for repeatability and intermediate precision of WANTAI, Synthgene, and YHLO were illustrated in Table 3. The data obtained showed satisfactory precision for the low, medium, and high levels, which were lower than those claimed by the manufacturers.

TABLE 3

Repeatability and intermediate precision results for three assays

Manufacturer	Assays	Measurands (U/ml)	Repeatability (% CV)		Intermediate precision (% CV)
			Results in our lab	Manufacuturer's statement	Results in our lab	Manufacuturer's statement
WANTAI	ELISA	0.1	12.3	<15.0	13.0	<15.0
		0.2	14.2		14.2
		0.4	12.6		13.5
Synthgene	ELISA	CL1	14.5	–	14.9	–
		CL2	13.9		15.0
YHLO	CLIA	2.1	9.0	<10.0	13.9	<15.0
		11.3	8.7		11.5
		87.9	9.8		14.2

Abbreviations: CL1, control 1; CL2, control 2; CLIA, chemiluminescence immunoassay; CV, coefficients of variation; ELISA, enzyme‐linked immunosorbent assay.

The WANTAI ELISA assay is reported to be linear in 0.0625–0.5 U/ml without sample dilution by the manufacturers. So, we mixed high‐level at 0.6 U/ml with low‐level serum pools at 0.0625 U/ml in various ratios. The WANTAI showed a good linear range between 0.0625 and 0.545 U/ml (R 2 = 0.9966, Figure 2A). Since no linearity information is available on YHLO, a wide range of values of tested mixes were prepared. The YHLO anti‐RBD‐specific antibody did not deviate from linearity in the entire tested range (0.260–242.4 U/ml) (R 2 = 0.9993, Figure 2B).

FIGURE 2

Linearity. (A) WANTAI’s linearity assessment using high level pools (0.6 U/ml) diluted with low level pools (0.0625 U/ml). (B) YHLO’s linearity assessment using high level pools (242.4 U/ml) diluted with low level pools (0.260 U/ml). The sera sample with high (H) concentration was serially diluted with the low (L) concentration sample at ratios of L, 0.9L + 0.1H, 0.8L + 0.2H, 0.7L + 0.3H, 0.6L + 0.4H, 0.5L + 0.5H, 0.4L + 0.6H, 0.3L + 0.7H, 0.2L + 0.8H, 0.1L + 0.9H, H

LoB and LoD

The LoB and the corresponding LoD for WANTAI, calculated as previously described, were 0.03 and 0.06 U/ml. The LoB and the corresponding LoD for YHLO were estimated to be 0.048 and 0.211 U/ml.

Comparison of clinical data between three different assays

Synthgene versus WANTAI

The detection results obtained by the WANTAI ELISA, Synthgene ELISA, and YHLO CLIA were reported in Table 4 and Figure 3. For Synthgene and WANTAI, the negative percent agreement (NPA) was 88.89% (95% confidence interval [CI], 68.36–98.01). The positive percent agreement (PPA) was 100.0% (95% CI, 95.58–100.0). The overall percent agreement (OPA) was 98.91% (95% CI, 94.09–99.81) with a κ value of 0.94 (95% CI, 0.81–1.0) (p < 0.001). The correlation of quantitative results of WANTAI with inhibition rate of Synthgene was moderate (ρ = 0.512, p < 0.001) (Figure 3A).

TABLE 4

Concordance between the detection results of three assays

Assays	Negative (n)	Positive (n)	Compared assays	NPA (%) (95% CI)	PPA (%) (95% CI)	OPA (%) (95% CI)	κ (95% CI)
WANTAI	9	83	YHLO	100.0 (70.08–100.0)	89.16 (82.47–95.85)	90.22 (82.45–94.77)	0.62* (0.40–0.84)
Synthgene	8	84	WANTAI	88.89 (68.36–98.01)	100.0 (95.58–100.0)	98.91 (94.09–99.81)	0.94* (0.81–1.0)
YHLO	18	74	Synthgene	100.0 (67.56–100.0)	88.10 (81.17–95.02)	89.13 (82.77–95.49)	0.56* (0.33–0.79)

Abbreviations: 95% CI, 95% confidence interval; NPA, negative percent agreement; OPA, overall percent agreement; PPA, positive percent agreement.

p < 0.001.

FIGURE 3

Graphic representation of correlation between the detection results of three assays. (A) The correlation of quantitative results of WANTAI ELISA with inhibition rate of Synthgene ELISA. (B) The correlation of quantitative results of YHLO CLIA with inhibition rate of Synthgene ELISA. (C) The correlation of quantitative results between WANTAI ELISA and YHLO CLIA. CLIA, chemiluminescence assay; ELISA, enzyme‐linked immunosorbent assay

Synthgene versus YHLO

As shown in Table 2, the NPA of these two assays in detecting the neutralizing antibodies targeting RBD was 100.0% (95% CI, 67.56–100.0). Meanwhile, the PPA was 88.10% (95% CI, 81.17–95.02). And the OPA was 89.13% (95% CI, 82.77–95.49) with a κ value of 0.56 (95% CI, 0.33–0.79) (p < 0.001). The correlation of quantitative results of YHLO with inhibition rate of Synthgene was medium (ρ = 0.566, p < 0.001) (Figure 3B).

YHLO versus WANTAI

The concordances between these two quantitative assays were shown in Table 2. The NPA and PPA were 100.0% (95% CI, 70.08–100.0) and 89.16% (95% CI, 82.47–95.85), respectively. And the OPA was 90.22% (95% CI, 82.45–94.77). The kappa statistical analysis indicated a good agreement (κ = 0.62, 95% CI: 0.40–0.84, p < 0.001). Spearman's analysis uncovered a strong correlation between the quantitative results tested by two assays (ρ = 0.931, p < 0.0001) (Figure 3C). However, the neutralizing antibodies targeting RBD levels determined by YHLO (0.492–315.6 U/ml) were significantly higher than those determined by WANTAI (0.087–342.3 U/ml) (p < 0.001). And of the positive samples, most had neutralizing antibodies targeting RBD levels less than 50 U/ml.

More importantly, the Passing–Bablok analysis and Bland‐Altman plot collectively revealed that bias for various immunoassays could not be ignored. To evaluate the pairwise concordance between assays, the selected sample concentrations were evenly distributed in the linear range (n = 54). In Passing–Bablok analysis, the intercept was 0.8663, 95% CI (−0.4000 to 2.0831). And the slope was 3.1911, 95% CI (2.7771–4.0348, Figure 4A). In Bland–Altman plot, the YHLO showed a positive mean bias% from the WANTAI, which was found to be 112.3% higher. Moreover, the 95% limits of agreement was very wide as shown in Figure 4B, emphasizing a poor concurrence. To our surprise, the exclusion of samples >50 U/ml did not decrease the bias, but large the bias (121.4%), with a slope of 4.5604 (95% CI, 4.0144–5.1969) and intercept of −0.3255 (95% CI, −1.8121 to 0.7019) (Figure 4C,D).

FIGURE 4

Passing–Bablok regression and Bland–Altman plots for YHLO and WANTAI. (A, B) Differences of all results between the two assays. (C, D) Differences of results at low concentrations between the two assays. regression line (blue), line of equality (black), mean proportional bias (red), 95% confidence interval (grey)

DISCUSSION

The neutralizing antibody targeting RBD level is a highly significant indicator for vaccine efficacy. Nowadays, many efforts have been devoted to developing high‐throughput assays for detection of SARS‐CoV‐2 neutralizing antibody which can allow for use in routine clinical laboratory. In this study, we introduced sVNTs targeting neutralizing antibodies against RBD and reported their different performances.

Of the three assays included in our study, the competitive ELISA assay manufactured by Wantai BioPharm has been used to identify individuals with an adaptive immune response to SARS‐CoV‐2 in phase I and II clinical trials for the recombinant adenovirus type‐5 vectored COVID‐19 vaccine. , Furthermore, it has been promisingly proven to propose reliable results with excellent specificity (99.60%) and sensitivity (100.00%) after being validated with vaccinated individuals. And its performance of the strong positive correlation with the pseudovirus‐based VNT (pVNT) has also been indicated in the studies (R 2 = 0.843). This correlation was much better than those of other commercial assays with neutralizing activity measurements (Spearman's ρ = 0.7075 ; Spearman's ρ = 0.81 ) and slightly superior to which between the cPass sVNT and pVNT (R 2 = 0.8374 ). The other similar ELISA assay manufactured by Synthgene Medical Technology, like WANTAI ELSIA, also permits 1–2 h turnaround time (TAT), a BSL‐2 laboratory, and broad access to high‐throughput assays. To our knowledge, no related Synthgene data has been published yet. Finally, the novel assay using one‐step competitive CLIA from YHLO Biotech adopts freely‐moved RBD‐coated magnetic beads and the acridinium ester‐labeled ACE2 to detect neutralizing antibody targeting RBD. It offers a more efficient (TAT <30 min), automated quantitative option versus the ELISA. , , A recent study on the efficacy of a heterologous prime–boost vaccination suggested that the YHLO’s CLIA assay could be used to determine the surrogate neutralization activity and it correlated closely with that in the cell culture‐based experiments (R 2 = 0.9256).  The YHLO CLIA seemed to be a better one among the surrogate viral neutralizing assays that have been reported.

First, the Synthgene ELISA kit is a newly developed kit, which can only provide qualitative or semi‐quantitative results in our study. As shown, this assay had the highest detection rate among the three assays. Our study could suggest that this assay is sufficiently sensitive to detect the presence of the neutralizing antibodies targeting RBD. We also observed great agreement between Synthgene ELISA and WANTAI ELISA. However, we only found general correlation between inhibition rate of Synthgene and quantitative results of WANTAI and YHLO. If quantification attempted in future development, we would further evaluate the accuracy of this assay.

Second, the YHLO CLIA assay is a fully automated technique. In the 92 serum samples, there were 9 samples negative for CLIA but positive for ELISAs. The CLIA seemed to be more specific than two ELISA assays. In comparison to WANTAI ELISA assay, we found that the linearity of YHLO CLIA assay has a much wider range and so no sample dilution is needed in analyzing the serum samples from adenovirus type‐5‐vectored COVID‐19 vaccine. Meanwhile, the mRNA vaccine may produce higher neutralizing antibodies (NAbs). As reported by Tenbusch et al., they found much NAbs with undiluted serum after two doses of BNT162b2 mRNA vaccine by using this YHLO CLIA assay. So, we have reason to believe that the YHLO CLIA sVNT could be also applied in monitoring immune response after mRNA vaccine vaccination without prior dilution. This assay has great potential for wide use in practice. Next, we evaluated the concurrence of the two quantitative assays. Generally, the neutralizing antibodies targeting RBD levels determined via YHLO CLIA were higher than those of WANTAI ELISA. The proportional biases were above 110.0% regardless of whether samples over 50 U/ml are excluded or not. Although the Cohen's κ and Spearman's ρ coefficients revealed a good concordance, the proportional errors cannot be ignored. The discrepant results might be owing to nonspecific interference proteins. The antigen coating concentration and the source or amount of antibody, can also influence the neutralizing antibody quantification. Overall, the antibody levels determined by YHLO CLIA and WANTAI　ELISA are not interchangeable, which means that these results from different systems cannot be compared directly.

Further, we also learned that some POCT kits by applying colloidal gold technique were successfully produced but not evaluated in our study, which significantly further shorten test time. As with above‐mentioned assays, our findings suggest that performance validation is required for any approaches and concurrence between assays should be carefully examined before adopted.

SARS‐CoV‐2 vaccine development yield a new clinical challenge to the test for vaccine evaluation. To date, no comparable published research in these three assays is available. Given the current research situation, we hope that this study provides useful data for future comparison studies. Moreover, joint efforts shall be warranted by developers and implementers to improve intermethod comparability.

Additionally, we would also like to highlight that, the neutralizing antibodies targeting RBD is not the only indicator of seroconversion after vaccination. Antibody against the N‐terminal domain (NTD) also involved in immune response. , Cellular immunity, are an equally important component as humoral immunity in acquiring immunity.  The vaccines immune responses varies depending on the vaccine types, the compositions, and vaccination routes.

The increasing novel SARS‐CoV‐2 variants present new insights. Although serologic tests could never fully substitute virus neutralization test, the above newly assays described are of practical utility to improve test capacity and long‐term monitor neutralizing antibody levels, facilitating large‐scale vaccine evaluation.

In conclusion, we found good agreement and correlation among three commercial immunoassays. The instrumented YHLO CLIA assay has a broader linear range and take advantage of simplicity and efficiency over manually ELISAs. Inconsistent results between the CLIA and ELISA indicated that they are not interchangeable in the determination of neutralizing antibodies targeting RBD. Our study contributed to a deeper understanding of these three sVNTs.

CONFLICT OF INTEREST

There is no conflict of interests.

AUTHOR CONTRIBUTIONS

Yang Yang, Guorui Liu, and Hui Chen conceived and designed this study. Weijun Jiang, Guorui Liu and Hui Chen carried out the experiments. Hui Chen and Wanwan Yu performed the main statistical analyses and wrote the manuscript. Wanwan Yu, Yang Yang, Guorui Liu, and Xiaojun Li reviewed and edited the manuscript.