Head-to-head comparison of two rapid high-throughput automated electrochemiluminescence immunoassays targeting total antibodies to the SARS-CoV-2 nucleoprotein and spike protein receptor binding domain.

BACKGROUND: Accurate anti-SARS-CoV-2 assays are needed to inform diagnostic, therapeutic, and public health decisions. The first manufacturer-independent head-to-head comparison of two rapid high-throughput automated electrochemiluminescence double-antigen sandwich immunoassays targeting total anti-SARS-CoV-2 antibodies against two different viral proteins, Elecsys Anti-SARS-CoV-2 (Elecsys-N) and Elecsys Anti-SARS-CoV-2 S (Elecsys-S) (Roche Diagnostics), was performed in a routine setting during the exponential growth phase of the epidemic's second wave.
METHODS: The diagnostic specificity of Elecsys-N and Elecsys-S was initially evaluated on a panel of 572 pre-COVID-19 samples, showing 100 % specificity of both assays. Elecsys-N/Elecsys-S head-to-head comparison used 3,416 consecutive blood samples from individuals that were tested for the presence of anti-SARS-CoV-2 within commercial out-of-pocket serologic testing.
RESULTS: Elecsys-N/Elecsys-S head-to-head comparison showed overall agreement of 98.68 % (3,371/3,416; 95 % CI, 98.23-99.03 %), positive agreement of 95.16 % (884/929; 95 % CI, 93.52-96.41 %), and a high kappa value of 0.996 (95 % CI, 0.956-0.976). Previous SARS-CoV-2 PCR positivity was identified in 14/24 (58.3 %) Elecsys-N negative/Elecsys-S positive individuals and in 4/21 (19.0 %) Elecsys-N positive/Elecsys-S negative individuals.
CONCLUSION: The first Elecsys-N/Elecsys-S head-to-head comparison showed excellent agreement of two highly specific and rapid high-throughput automated anti-SARS-CoV-2 assays. An important question is whether laboratories offering two different antibody assays could benefit from combining the assays; if so, should use be concomitant or sequential-and, in the latter case, in which order? Based on our results, we favor concomitant over sequential Elecsys-N/Elecsys-S use when testing individuals for anti-SARS-CoV-2 antibodies in high-incidence settings; for example, during the exponential or stationary growth phase of the COVID-19 epidemic.

Introduction

The availability of assays to detect antibodies against severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) created excitement and hope among the laboratory community, government leaders, and the public [1]. Unfortunately, early in the pandemic, the global market was flooded with antibody assays of unproven performance and various governments purchased large quantities of ineffective tests [2,3]. The situation improved with implementation of verification/authorization procedures and recommendations for antibody test utilization and result interpretation [[1], [2], [3], [4]].

Antibody tests are a useful diagnostic aid, primarily for patients that present later in the disease course and are negative for SARS-CoV-2 RNA, when a lower-respiratory-tract sample cannot be collected, for diagnosing multisystem inflammatory syndrome in children, and to screen potential donors for convalescent-phase plasma therapy [5,6]. Serologic testing may prove useful in determining immunity, stratifying individuals for vaccine receipt, and documenting vaccine response, which could inform return-to-work and travel decisions and other public health measures [5,6]. Finally, they play an important role in understanding the epidemiology, including seroprevalence at the local, national, and global levels [[5], [6], [7]].

Although several commercial anti-SARS-CoV-2 assays have received U.S. Food and Drug Administration (FDA) emergency-use authorization (EUA), most approved assays lack manufacturer-independent performance evaluations in peer-reviewed literature.

Here we present a manufacturer-independent head-to-head comparison of two rapid (18-minute) high-throughput automated electrochemiluminescence double-antigen sandwich immunoassays targeting total anti-SARS-CoV-2 antibodies against two different viral proteins: Elecsys Anti-SARS-CoV-2 (Elecsys-N) and Elecsys Anti-SARS-CoV-2 S (Elecsys-S) (Roche Diagnostics, Mannheim, Germany).

Elecsys-N is an assay for qualitative detection of total anti-SARS-CoV-2 antibodies against nucleoprotein (N) that received FDA EUA on May 3, 2020, and Conformitè Europëenne (CE) mark on April 28, 2020. The assay has been extensively evaluated by the manufacturer, showing 99.80 % (95 % confidence interval (CI), 99.69‑99.88 %) clinical specificity on 10,453 samples and 99.5 % (95 % CI, 97.0‑100 %) sensitivity on 185 samples obtained 14 days or later after SARS-CoV-2 PCR-confirmation. Elecsys-N has also been evaluated in several manufacturer-independent studies, with diagnostic specificity and sensitivity values spanning claims made by the manufacturer in most studies [1,5,[7], [8], [9], [10], [11], [12], [13], [14], [15], [16]], and it is consequently considered one of the most appropriate assays for seroprevalence surveys, especially in low-prevalence settings [6,17].

Elecsys-S is an assay for quantitative detection (linear range 0.4–250 U/mL) of total anti-SARS-CoV-2 antibodies against the spike (S) protein receptor binding domain (RBD), launched in Europe in September 2020. It received FDA EUA on November 25, 2020, and CE mark on September 17, 2020. The assay has been extensively evaluated by the manufacturer, showing 99.98 % (95 % CI, 99.91–100 %) specificity on 5,991 samples and 98.8 % (95 % CI, 98.1–99.3 %) sensitivity on 1,423 samples obtained 14 days or later after SARS-CoV-2 PCR-confirmation. As far as we know, no Elecsys-S evaluation data have been published in peer-reviewed literature yet.

This study evaluated Elecsys-N and Elecsys-S head-to-head in a routine setting during the exponential growth phase of the epidemic’s second wave. During the 84-day study period, the cumulative number of PCR-confirmed COVID-19 cases in Slovenia increased 18.4-fold, from 6,105 to 112,048 (https://www.nijz.si/sl/dnevno-spremljanje-okuzb-s-sars-cov-2-covid-19), providing a challenging but informative environment for evaluating two highly specific anti-SARS-CoV-2 assays directed against different SARS-CoV-2 antigens.

Material and methods

Before head-to-head comparison, the diagnostic specificity of Elecsys-N and Elecsys-S was internally evaluated in May and September 2020, respectively, on a panel of 572 samples collected prior to the emergence of COVID-19 (Table 1 ).

Table 1

Internal assessment of clinical specificity of Elecsys-N and Elecsys-S assays using 572 pre−COVID-19 serum samples.

		Elecsys-N		Elecsys-S
Panel/Cohort	n	Positive	Specificity	Positive	Specificity
Laboratory-confirmed acute human cytomegalovirus (CMV) (n = 6) or Epstein–Barr virus (EBV) (n = 16) infection	22	0	100 %	0	100 %
Pneumonia caused by Mycoplasma pneumoniae	15	0	100 %	0	100 %
Laboratory-confirmed pertussis	7	0	100 %	0	100 %
PCR-confirmed viral non-SARS-CoV-2 respiratory infections with coronavirus HKU1 (n = 1), NL63 (n = 5), 229E (n = 3), or OC43 (n = 1), influenza virus A (n = 3), influenza virus B (n = 4), respiratory syncytial virus (n = 4), or rhinoviruses (n = 7)	28	0	100 %	0	100 %
Serum samples collected for different medical reasons (testing for HIV (n = 210) and serological markers of viral hepatitis (n = 290)) before June 2019	500	0	100 %	0	100 %
Total number of samples	572	0	100 %	0	100 %

For head-to-head comparison, 3,416 consecutive blood samples received between October 1, 2020, and December 23, 2020 were tested in parallel using Elecsys-N and Elecsys-S on a cobas e411 analyzer following the manufacturer’s instructions, using cut-off values for positive results of ≥1.0 and ≥0.8 U/mL, respectively. Blood samples were obtained from the same number of individuals that attended out-of-pocket anti-SARS-CoV-2 testing with a commercial test provider. In contrast to SARS-CoV-2 RNA testing [18], which is fully covered by national health insurance, SARS-CoV-2 antibody testing in Slovenia is not reimbursed. Thus, the study population consisted of individuals that requested out-of-pocket anti-SARS-CoV-2 testing for several reasons: travel purposes; to check serological response after PCR-confirmed COVID-19 or clinically compatible but virologically non-confirmed COVID-19; recent contact with a person with COVID-19, but not eligible for PCR-testing; or pure curiosity.

Due to the inability of obtaining follow-up sample(s) from individuals with Elecsys-N/Elecsys-S discordant results, the national SARS-CoV-2 PCR notification database was consulted and all discrepant samples were additionally tested by two anti-SARS-CoV-2 antibody assays with excellent analytical performance proven in manufacturer-independent evaluations: SARS-CoV-2 Ab Elisa Kit (Wantai; Wantai Biological Pharmacy Enterprise Co, Beijing, China) detecting total antibodies against the S protein RBD [[19], [20], [21], [22]] and Architect SARS-CoV-2 IgG (Abbott; Abbott Diagnostics, IL, USA) detecting IgG antibodies against N protein [5,8,10,13,14]. Furthermore, 10 % and 5% randomly selected samples with Elecsys-N/Elecsys-S concordantly positive and concordantly negative results, respectively, were additionally tested by Wantai and Abbott.

A contingency table was constructed to assess overall and positive agreements with 95 % CIs. The level of agreement between both tests was assessed using kappa statistics. All statistical analyses were performed using Excel (Microsoft, Redmond, WA, USA) and R software version 3.2.5 (Free Software Foundation, Boston, MA, USA).

Results

Internal evaluation on the panel of 572 pre−COVID-19 samples showed 100 % specificity of both assays (Table 1). As shown in Table 2 , head-to-head Elecsys-N/Elecsys-S comparison showed overall agreement of 98.68 % (3,371/3,416; 95 % CI, 98.23–99.03 %), positive agreement of 95.16 % (884/929; 95 % CI, 93.52–96.41 %) and a high kappa value of 0.996 (95 % CI, 0.956–0.976). A total of 45/3,416 discordant results were observed (Table 2, Table 3).

Table 2

Results of head-to-head comparison of Elecsys-N (detecting total anti-SARS-CoV-2 antibodies against nucleoprotein) and Elecsys-S (detecting total anti-SARS-CoV-2 antibodies against the spike protein receptor binding domain) on 3,416 consecutive blood samples received between October 1, 2020 and December 23, 2020 using cut-off values for positive results of ≥ 1.0 and ≥ 0.8 U/mL, respectively.

		Elecsys-N
Elecsys-S		Positive	Negative
	Positive	884	24*
	Negative	21#	2,487

23/24 tested positive using supplemental Wantai test (detecting total anti-SARS-CoV-2 antibodies against the spike protein receptor binding domain) and 24/24 tested negative using supplemental Abbott test (detecting anti-SARS-CoV-2 IgG antibodies against nucleoprotein).

17/21 tested positive using supplemental Abbott test and 20/21 tested negative using supplemental Wantai test.

Table 3

Overview of anti-SARS-CoV-2 Elecsys-N and Elecsys-S testing results, available SARS-CoV-2 RNA PCR testing results and results of supplemental testing using Abbott and Wantai assays in 45 individuals with Elecsys-N/Elecsys-S discordant results. For SARS-CoV-2 RNA–positive individuals the date of the first recorded PCR positive result is presented, and for SARS-CoV-2 RNA–negative individuals the date of the last recorded PCR negative result is presented. N/A = no record in the national SARS-CoV-2 PCR notification database.

Sample ID	Testing date (M/D/Y)	Elecsys-N		Elecsys-S		SARS-CoV-2 RNA PCR		Abbott	Wantai
		Value (pos ≥ 1.0)	Result	Value (pos ≥ 0.80 U/mL)	Result	Nasopharyngeal swab collection date (M/D/Y)	PCR result	Result	Result
1	10/28/2020	0.104	NEG	1.16	POS	10/16/2020	POS	NEG	POS
2	10/29/2020	0.149	NEG	1.48	POS	11/02/2020	POS	NEG	POS
3	10/20/2020	0.089	NEG	1.61	POS	04/22/2020	NEG	NEG	NEG
4	11/01/2020	0.462	NEG	2.25	POS	04/30/2020	NEG	NEG	POS
5	12/04/2020	0.227	NEG	2.45	POS	04/24/2020	POS	NEG	POS
6	12/04/2020	0.090	NEG	2.63	POS	03/23/2020	NEG	NEG	POS
7	11/19/2020	0.421	NEG	3.27	POS	11/06/2020	POS	NEG	POS
8	10/29/2020	0.093	NEG	3.79	POS	04/12/2020	NEG	NEG	POS
9	10/24/2020	0.750	NEG	6.65	POS	09/10/2020	POS	NEG	POS
10	11/20/2020	0.568	NEG	7.30	POS	N/A	N/A	NEG	POS
11	11/20/2020	0.384	NEG	10.19	POS	10/14/2020	POS	NEG	POS
12	10/23/2020	0.259	NEG	11.62	POS	04/28/2020	POS	NEG	POS
13	10/30/2020	0.731	NEG	14.89	POS	10/16/2020	POS	NEG	POS
14	11/13/2020	0.862	NEG	28.14	POS	09/14/2020	POS	NEG	POS
15	10/29/2020	0.848	NEG	33.06	POS	09/01/2020	POS	NEG	POS
16	11/30/2020	0.655	NEG	9.75	POS	10/20/2020	POS	NEG	POS
17	11/30/2020	0.095	NEG	1.63	POS	10/17/2020	POS	NEG	POS
18	12/01/2020	0.597	NEG	6.89	POS	11/16/2020	POS	NEG	POS
19	12/08/2020	0.091	NEG	15.11	POS	N/A	N/A	NEG	POS
20	12/09/2020	0.143	NEG	2.06	POS	N/A	N/A	NEG	POS
21	12/10/2020	0.376	NEG	2.13	POS	N/A	N/A	NEG	POS
22	12/10/2020	0.355	NEG	4.08	POS	N/A	N/A	NEG	POS
23	12/14/2020	0.501	NEG	28.26	POS	N/A	N/A	NEG	POS
24	12/17/2020	0.775	NEG	6.42	POS	11/20/2020	POS	NEG	POS
25	11/05/2020	1.38	POS	< 0.400	NEG	N/A	N/A	POS	NEG
26	11/17/2020	1.44	POS	< 0.400	NEG	N/A	N/A	POS	NEG
27	10/22/2020	1.44	POS	< 0.400	NEG	08/21/2020	NEG	POS	NEG
28	11/09/2020	1.59	POS	0.446	NEG	10/24/2020	POS	POS	NEG
29	11/09/2020	1.65	POS	< 0.400	NEG	N/A	N/A	POS	NEG
30	11/01/2020	1.65	POS	< 0.400	NEG	10/20/2020	POS	POS	NEG
31	10/25/2020	1.67	POS	< 0.400	NEG	04/21/2020	NEG	NEG	NEG
32	11/06/2020	1.87	POS	< 0.400	NEG	N/A	N/A	POS	NEG
33	11/06/2020	2.35	POS	< 0.400	NEG	N/A	N/A	NEG	NEG
34	10/26/2020	2.41	POS	< 0.400	NEG	04/25/2020	NEG	POS	NEG
35	10/07/2020	2.45	POS	< 0.400	NEG	N/A	N/A	POS	NEG
36	11/04/2020	4.15	POS	0.462	NEG	N/A	N/A	POS	NEG
37	10/18/2020	4.75	POS	< 0.400	NEG	03/21/2020	NEG	NEG	NEG
38	10/21/2020	4.84	POS	< 0.400	NEG	05/25/2020	NEG	NEG	NEG
39	11/02/2020	5.97	POS	< 0.400	NEG	05/24/2020	NEG	POS	NEG
40	11/12/2020	7.47	POS	0.413	NEG	10/12/2020	NEG	POS	POS
41	11/18/2020	7.63	POS	< 0.400	NEG	08/04/2020	NEG	POS	NEG
42	11/20/2020	8.12	POS	0.626	NEG	11/20/2020	POS	POS	NEG
43	11/02/2020	10.56	POS	< 0.400	NEG	10/30/2020	NEG	POS	NEG
44	11/09/2020	10.86	POS	< 0.400	NEG	07/16/2020	NEG	POS	NEG
45	10/27/2020	25.73	POS	0.505	NEG	10/15/2020	POS	POS	NEG

Of 24 Elecsys-N negative / Elecsys-S positive samples, 23 (95.8 %) tested Wantai positive and all Abbott negative. Previous SARS-CoV-2 PCR positivity was identified in 14/24 (58.3 %) of Elecsys-N negative / Elecsys-S positive individuals (Table 3 ).

Of 21 Elecsys-N positive / Elecsys-S negative samples, 17 (80.9 %) tested Abbott positive and all but one tested Wantai negative. Previous SARS-CoV-2 PCR positivity was identified in 4/21 (19.0 %) of Elecsys-N positive/Elecsys-S negative individuals (Table 3).

All 124 Elecsys-N/Elecsys-S concordantly negative samples tested also negative using both Wantai and Abbott. Out of 87 Elecsys-N/Elecsys-S concordantly positive samples, 87 (100 %) and 85 (97.7 %) tested positive using Wantai and Abbott, respectively.

Conclusions

Accurate anti-SARS-CoV-2 assays are needed to inform diagnostic, therapeutic, and public health decisions [5,23]. When selecting antibody assays, virologists must consider not only sensitivity and specificity, but also prevalence in the tested population, the intended use of results, sample throughput, test complexity, reagent and instrument availability, and cost per reportable result [5]. Especially assays’ throughput and specificity are crucial parameters if large-scale antibody testing is desirable in a low-prevalence pre-vaccination environment [9,23].

This comparison showed high overall and positive agreement of two highly specific and rapid high-throughput automated assays. Equal distribution of Elecsys-N/Elecsys-S discordant results was observed. Such distribution of discordant results was confirmed by additional testing using two supplementary assays: Wantai detecting the equivalent total anti-S RBD antibodies as Elecsys-S and Abbott detecting IgG fraction of the total anti-N antibodies targeted by Elecsys-N. The recorded slight Elecsys-N/Abbott discordance is most probably a result of the presence of anti-N antibodies other than IgG detected by Elecsys-N and missed by Abbott. Thus, although we were unable to obtain follow-up sample(s) from individuals with discordant results, we strongly believe that not more than 5% of discordant results are due to false positivity of one of the Elecsys assays. This is supported by: (i) extremely high specificity of both Elecsys assays recorded in the manufacturer’s and manufacturer-independent evaluations [1,5,[7], [8], [9], [10], [11], [12], [13], [14], [15]], including this study; (ii) confirmation of the presence of targeted antibodies using supplementary serological assays in 40/45 samples with Elecsys-N/Elecsys-S discordant results; (iii) confirmation of previous COVID-19 in 18/45 individuals with discordant results through the national SARS-CoV-2 PCR notification database; (iv) distribution of Elecsys-N and Elecsys-S testing values in samples with discordant results not concentrated near the cut-off; and (v) high-incidence study settings in which anti-S-only and anti-N-only responders are not unusual in the early convalescent phase [[24], [25], [26]].

An important open question is whether laboratories offering different antibody assays could benefit from combining the assays; if so, should use be concomitant or sequential—and, in the latter case, in which order? Previous studies showed that a two-assay algorithm improves the positive predictive value compared with an individual assay alone while maintaining the negative predictive value [5,17,27]. Thus, the two-assay approach was recently recommended for identifying potential convalescent-phase plasma donors and assessing candidacy for experimental COVID-19 therapeutics in PCR-negative patients with respiratory symptoms [5]. As far as we know, the Elecsys-N and Elecsys-S manufacturer issued no recommendation for combination use, but the manufacturer’s unpublished data showed that concomitant use of both assays could increase overall sensitivity (some convalescent patients were anti-S-only and some anti-N-only responders) and that sequential use (initially Elecsys-N followed by Elecsys-S for N-positives) could improve positive predictive value to 100 % in low-prevalence settings. Based on our results, we favor concomitant over sequential Elecsys-N/Elecsys-S use when testing in high-incidence settings (e.g., during the exponential or stationary growth phase of the COVID-19 epidemic), which in February 2021 is still unfortunate reality in most of the world.

Funding

This study was funded by the Institute of Microbiology and Immunology, , and the (grants P3-0083 and V3-2034). The assays used in the comparative evaluation were not provided by the manufacturer, and the manufacturer had no role in the study design, data collection, analysis, interpretation of the results, writing of the article, or decision to submit the work for publication.

CRediT authorship contribution statement

Mario Poljak: Conceptualization, Methodology, Validation, Investigation, Writing - original draft, Writing - review & editing, Supervision. Anja Oštrbenk Valenčak: Methodology, Validation, Investigation, Data curation, Formal analysis, Writing - review & editing. Tina Štamol: Methodology, Validation, Investigation, Data curation, Writing - review & editing. Katja Seme: Conceptualization, Writing - original draft, Writing - review & editing, Supervision.

Declaration of Competing Interest

The authors report no declarations of interest.