Comparison of eight commercial, high-throughput, automated or ELISA assays detecting SARS-CoV-2 IgG or total antibody.

BACKGROUND: Many commercial assays, of different designs, detecting SARS-CoV-2-specific antibodies exist but with little experience with them.
OBJECTIVES: The aim of this study was to compare the performance of assays detecting IgG or total antibodies to N or S antigens, validated for routine use in France, with samples from subjects with more or less severe SARS-CoV-2 infection.
METHODS: Eight assays were used: Abbott Architect, DiaSorin Liaison®, bioMérieux Vidas®, Roche Elecsys Cobas®, Siemens Atellica®, BioRad Platelia ELISA, Epitope Diagnostics ELISA, and Wantai ELISA. The tested population included 86 samples from 40 hospitalized subjects and 28 outpatients at different time from symptom onset.
RESULTS: The positivity rate varied depending on the assay but was greater for all assays in hospitalized than non-hospitalized patients. Despite a good correlation between the assays, discrepancies occurred, without a systematic origin, even for samples taken more than 20 days after symptom onset. These discrepancies were linked to low antibody levels in pauci-symptomatic patients.
CONCLUSION: Whichever assay is chosen, a false negative result may need to be ruled out with another test in a risk situation.

Background

The SARS-CoV-2 (Severe acute respiratory syndrome coronavirus-2) responsible for COVID-19 (Coronavirus infectious disease 2019) emerged in December 2019 in Wuhan, China. The members of Coronaviridae family are enveloped positive RNA viruses; the 5’ part of the genome is encoding for a polyprotein cleaved in non-structural proteins forming the transcription and replication complex. The 3’ part of the genome in encoding for the structural proteins, spike (S), envelope (E), membrane (M) and the nucleoprotein (N). Serological assays target mainly the S and/or the N proteins which are considered as the target of neutralizing antibodies and immunodominant proteins, respectively.

While SARS-CoV-2 genome detection in respiratory specimens remain the only way to confirm current infection, serology might be useful for epidemiological studies and past SARS-CoV-2 infection determination in the absence of molecular diagnosis. Most of the companies producing immunodiagnostic assays have rapidly produced tests detecting SARS-CoV-2-specific IgG, IgM, IgA or total antibodies. Some of these assays have been evaluated by independent groups [[1], [2], [3]] but their comparison on the same samples has rarely been performed [[4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15]] or only for a small number of commercial assays, with the exception of point-of-care systems.

Objective

The objective of this paper was to perform a performance comparison of eight commercial assays of which five use automated instruments with CLIA or ELFA technology (Abbott Architect, DiaSorin Liaison®, bioMérieux Vidas®, Roche Elecsys Cobas®, and Siemens Atellica®), and three are microplate ELISA (BioRad Platelia, Epitope Diagnostics EDI™, Wantai).

Materials and methods

The characteristics of each assay are presented in Table 1 . For comparison, only the assays detecting IgG or total antibodies are presented. Half of them are directed against the N protein (Abbott Architect, Roche Elecsys Cobas®, Biorad Platelia, Epitope Diagnostics) and the other half against parts of the S protein (S1 + S2: Diasorin Liaison®, S1: biomérieux Vidas® or the receptor binding domain (RBD): Siemens Atellica®, Wantai). All are CE marked and their sensitivity and specificity were evaluated by the manufacturer (Table 1). Antibody detection kits were used according to manufacturers' instructions.

Table 1

Product description of the compared automated commercial antibody detection assays.

Manufacturer assay name	DiaSorin Liaison®	bioMérieux Vidas®	Siemens Atellica®	Wantai	Abbott Architect	Roche Elecsys	BioRad Platelia	Epitope Diagnostics EDI™
Antibody detected	IgG	IgG	Total Ig	Total Ig	IgG	Total Ig	Total Ig	IgG
recombinant labeled protein	S1 + S2	S1 + peptide	RBD	RBD	N	N	N	N
methodology	indirect CLIA (2 steps)	Sandwich ELFA (2 steps)	Sandwich CLIA (1 step)	Sandwich ELISA	indirect CMIA (2 steps)	Sandwich ECLIA (2 steps)	Sandwich ELISA (1 step)	Indirect ELISA (2 steps)
specimen type	serum/plasma	serum/plasma	serum/plasma	serum/plasma	serum/plasma	serum/plasma	serum/plasma	serum/plasma
sample volume	20 μL	100 μL	50 μL	100 μL	25 μL	20 μL	10 μL	1 μL
instrument used	Liaison® XL	Vidas®	Atellica®	manual or automated ELISA instrument	Architect	Cobas®	manual or automated ELISA instrument	manual or automated ELISA instrument
turnaround time	35 min	27 min	15 min	1.5 h	29 min	18 min	2 h	2 h
cutoff calculation basis	calibrator	standard	calibrator	negative control (min 0.19)	calibrator	calibrator	cut-off control (mean)	negative control (mean+0.18)
threshold	AU/mL; 12	ratio; 1	ratio; 1	ratio; 1.1	ratio; 1.4	ratio; 1	ratio; 1	≥ 1,1x (NC + 0.18)
grey zone	>12 - <15	ND	ND	≥0.9 - ≤1.1	ND	ND	≥ 0.8 - < 1	≥ 0.9x (NC + 0.18) < 1.1x (NC + 0.18)
measuring interval	3.8 - 400		upper limit: 10				upper limit 3.5
reported sensitivity: days from symptoms (number): %	≤ 5 (44): 25.0 %5–15 (52): 90.4%≥16 (39): 97.4%	≤7 (117): 45.3 %8–15 (44): 88.6%≥16 (29): 96.6%	<7 (89): 60.7 %7–13 (116): 97.5%≥14 (42): 100%	94.5 %	<3 (5): 0 %3–7 (10): 50%8–13 (34): 91.2%≥14 (73): 100%	<7 (116): 65.5 %7–13 (59): 88.1%≥14 (29): 100%	≤8 (8): 73 %>8 (39): 97.4 %	100 %
reported specificity	98.5 %	100 %	99.8 %	100 %	99.6 %	99.8 %	99.6 %	100 %

RBD: Receptor Binding Domain, CLIA: ChemiLuminescence ImmunoAssay, ELFA: Enzyme Linked Fluorescent Assay, CMIA: Chemiluminescence Microparticule ImmunoAssay, ECLIA: ElectroChemiLuminescence ImmunoAssay, ND: not determined.

Characteristics of the commercial antibody detection assays classified according to the targeted antigen and the class of antibodies detected. The technical details of the assay as well of the sensitivity and specificity data reported by the manufacturer are listed.

In the present study 82 residual serum or plasma samples from a population of 68 SARS-CoV-2 infected patients, confirmed by a positive RT-PCR (reverse transcriptase polymerase chain reaction), were used.

Results

The patient population included 40 infected hospitalized patients (including 25 in intensive care units) and 28 non hospitalized infected health care workers (HCW) (including physicians, nurses, and lab staff). Age of the patients ranged from 7 to 81 years (median = 51). Median age of hospitalized patients was greater (64 years, range: 7–81) than that of HCW (36 years, range: 25–59). In the HCW group, women proportion (22/28; 78.6 %) was higher than in the hospitalized group (11/40; 27.5 %). Seven patients were tested at different time points (ranging from 2 to 5). The delays between the dates of samples used for serology testing, and the beginning of documented symptoms varied from 4 to 52 days (37 ≤ 20 days, 45 > 20 days). The data were compared between 3 groups of delays from symptom onset (SO) : ≤ 15 days (16 samples), 16–20 days (21 samples), > 20 days (45 samples). The results of the eight assays, grouped according to the delay from SO and the patient population (hospitalized patients versus HCW), are shown in Fig. 1 . The positivity rate is detailed in Table 2 . Regardless of the assay, antibody reactivity was higher in hospitalized patients compared to non-hospitalized HCW. The positivity rate increased with time from SO but did not reach 100 % in this population of HCW, probably because antibody titer declined relatively rapidly in patients with mild disease and weak antibody response. Of the 45 samples taken in 40 patients more than 20 days after SO, 12 (26.7 %) were negative for antibody detection in at least one assay. Eleven of these 12 samples were from HCW. A low reactivity of positive results was observed in most cases. Six were from two HCW with four and five follow-up samples, respectively. In these two HCW, antibody kinetics either increased just above the threshold, or increased then decreased, or never reached the threshold, depending on the assay. In a single case of a hospitalized person with a sample taken more than 20 days after SO, negative results were obtained with two assays (Abbott Architect and BioRad Platelia) while all other assays gave positive results. Whatever the delay after SO, discrepancies between assays occurred (29.3 % of the samples) without being able to be systematically attributed to the sensitivity, to the target antigen (N or S) or to the class of antibodies detected (IgG or total). These discrepancies occurred mainly in samples from HCW (19/24, 79 %). Despite this, concordance between the tests was quite good and similar for all of them, with no clear distinction depending on whether the tests being compared detected the same or another viral target (Table 3 ). In addition, there was no better agreement between the tests detecting total antibodies (Siemens Atellica®, Wantai, Roche Elecsys Cobas®, BioRad Platelia) than between these tests and those detecting only IgG.

Fig. 1

Distribution of antibody detection values for the different assays according to the patient population and delay from symptom onset.

Each dot represents the value obtained with each serological assay. The assays’ results are presented according to the targeted antigens : S1 or S1+peptide ((A) Diasorin Liaison®, (B) bioMérieux Vidas®); Receptor Binding Domain ((C) Siemens Atellica®, (D) Wantai); or N protein ((E) Abbott Architect, (F) Roche Elecsys Cobas®, (G) Biorad Platelia, (H) Epitope Diagnostics EDI™). Results are expressed as a ratio of the sample signal to cut-off for all tests except the EDI™ assay which is expressed in optical density and the Diasorin Liaison® allowing the quantification of the antibodies in arbitrary units (AU/mL). Detection limit for each test is shown with a discontinued line. Black or red dots are used for samples from health care workers (HCW) or hospitalized (Hosp) patients respectively.

Table 2

Positivity rate of each assay according to the delay from onset of symptoms.

Days fromsymptom onset (N)	DiaSorinLiaison®	bioMérieuxVidas®	SiemensAtellica®	Wantai	AbbottArchitect	RocheElecsys	BioRadPlatelia	EpitopeDiagnosticsEDI™
≤ 15 (16)	56.3 %	81.3 %	73.3 %	100 %	62.5 %	68.8 %	80 %	81.3 %
16 – 20 (21)	71.4 %	90.5 %	81%	95 %	95.2 %	85.7 %	90.5 %	100 %
> 20 (45)	77.8 %	90.7 %	88.6 %	97.6 %	95.6 %	88.9 %	86.7 %	95.3 %

Percentage of antibody detection was expressed for each assay according to the delay expressed in days from symptom onset.

Table 3

Concordance between each assay.

	DiaSorin Liaison®	bioMérieux Vidas®	Siemens Atellica®	Wantai	Abbott Architect	Roche Elecsys	BioRad Platelia	Epitope Diagnostics EDI™
DiaSorin Liaison®	100 %	80 % (80)	82.5 % (80)	72.7 % (77)	78.05 % (82)	80.5 % (82)	81.5 % (81)	74.7 % (79)
bioMérieux Vidas®	80 % (80)	100 %	94.9 % (78)	90.7 % (75)	90 % (80)	87.5 % (80)	87.3 % (79)	89.6 % (77)
Siemens Atellica®	82.5 % (80)	94.9 % (78)	100 %	85.3 % (75)	86.1 % (79)	87.3 % (79)	82.5 % (80)	84.2 % (76)
Wantai	72.7 % (77)	90.7 % (75)	85.3 % (75)	100 %	90.9 % (77)	87 % (77)	89.5 % (76)	93.5 % (77)
Abbott Architect	78.05 % (82)	90 % (80)	86.1 % (79)	90.9 % (77)	100 %	92.7 % (82)	88.9 % (81)	89.9 % (79)
Roche Elecsys	80.5 % (82)	87.5 % (80)	87.3 % (79)	87 % (77)	92.7 % (82)	100 %	90.1 % (81)	84.8 % (79)
BioRad Platelia	81.5 % (81)	87.3 % (79)	82.5 % (80)	89.5 % (76)	88.9 % (81)	90.1 % (81)	100 %	87.2 % (78)
Epitope Diagnostics EDI™	74.7 % (79)	89.6 % (77)	84.2 % (76)	93.5 % (77)	89.9 % (79)	84.8 % (79)	87.2 % (78)	100 %

We compared assays by pairs. Considering only the samples tested with both assays (N), we considered as concordant the samples for which antibodies were detected positive or detected negative by both assays; when one assay gave a positive value whereas the other gave a negative value, the sample result was considered as discordant between the assays. Concordance was expressed as a percentage of samples detected positive or negative by the two considered assays.

Data are repeated for horizontal or vertical reading.

Discussion

In the present study all assays showed similar overall performance although the most and least sensitive tended to be the ELISA from Wantai and the Diasorin Liaison® assay, respectively, as confirmed by previous reports with different patient populations [4,6,15]. For the other assays the differences were not sufficiently consistent to claim that one had a better sensitivity than the others. Comparison of the same samples with these different commercial antibody detection systems showed that samples from SARS-Cov-2 infected patients could be negative for antibody detection by any of them. Antibodies undetectability could be linked to patient characteristics such as variable kinetics of antibody production against each viral target, strength of antibody response, antibody affinity, or assay characteristics such as the antigen nature and preparation, or duration of antigen-antibody incubation. Each of observed discrepancies could have had a different origin leading to no obvious and consistent interpretation. Discrepant results occurred mainly in patients with mild or asymptomatic infection, as antibody response might be weak and decreasing over time. Such dissociated responses between assays have already been described by others [4,8,11].

Thus, in absence of molecular diagnosis, but with suggestive clinical features, a negative antibody result may need to be evaluated with another assay to ensure that it was not a false negative result.

One limitation of our study was the small size and heterogeneous sample population, limiting determination of a true sensitivity for each assay. Specificity of these assays was not addressed in this report but has been evaluated in other studies [[4], [5], [6],8,9,13]. Even if some cross reactivity was observed, it was low, except with SARS-CoV virus, and often with different samples with each test compared.

Despite good overall performance, commercial assays need to be further evaluated with longitudinal samples from well characterized patients. In the first published studies, data came from severe patients but serology would be more useful in individuals with mild illness. Discrepancies between assays occurring mainly in this patient category, they should be the target of future studies aimed at correlating the data with the kinetics of N and S-specific antibodies, as well as their neutralizing capacity.

Credit author statement

MAT selected and collected the samples. MAT and VE interpreted the data and wrote the manuscript.

VI contributed to the biobank collection, and MPM to the technical organization. They both helped at interpreting the data and contributed to the final version of the manuscript.

AB and BL contributed to the finalization of the manuscript.

This work did not receive any specific grant from funding agencies, in the public, commercial or not-for-profit sectors. The assay kits were kindly provided by the manufacturers.

Declaration of Competing Interest

The authors report no declarations of interest.