Comparison of different serological assays for SARS-CoV-2 in real life.

BACKGROUND: The emergence of the global SARS-CoV-2 pandemic required the rapid and large-scale deployment of PCR and serological tests in different formats.
OBJECTIVES: Real-life evaluation of these tests is needed. Using 168 samples from patients hospitalized for COVID-19, non-hospitalized patients but infected with SARS-CoV-2, patients participating in screening campaigns, and samples from patients with a history of other seasonal coronavirus infections, we evaluated the clinical performance of 5 serological assays widely used worldwide (WANTAI®, BIORAD®, EUROIMMUN®, ABBOTT® and LIAISON®).
RESULTS: For hospitalized patients, all these assays showed a sensitivity of 100 % from day 9 after the symptoms onset. On the other hand, sensitivity was much lower for patients who did not require hospitalization for COVID-19 confirmed by PCR (from 91.6 % for WANTAI® to 69 % for LIAISON®). These differences do not seem to be due to the antigens chosen by the manufacturers but more to the test formats (IgG detection versus total antibodies). In addition, more than 50 days after a positive PCR for CoV-2-SARS the proportion of positive patients seem to decrease. We did not observe any significant cross-reactions for these techniques with the four other seasonal coronaviruses.
CONCLUSION: In conclusion, the evaluation and knowledge of the serological tests used is important and should require an optimized strategy adaptation of the analysis laboratories to best meet patient's expectations in the face of this health crisis.

Background

In December 2019, a new Betacoronavirus virus of the coronavirus family causing severe acute respiratory symptoms appeared in Wuhan, China [1]. The World Health Organization (WHO) has named the disease, coronavirus 2019 (COVID-19), and coronavirus 2 severe acute respiratory syndrome (SARS-CoV-2). The virus has spread rapidly around the world, with a huge impact on everyone's life.

Since the outbreak of coronavirus cases worldwide, a frantic race for the availability of PCR and serological tests has been launched by the entire community of in vitro diagnostic manufacturers [2]. Antibody tests, such as enzyme-linked immunosorbent assays (ELISA) or chemiluminescent assays (CLIA), can overcome some of these difficulties. Serological tests can detect past infection with CoV-2-SARS in patients for whom PCR could not be performed or for whom the nasopharyngeal swab result was falsely negative [3].

For serological tests, manufacturers have often demonstrated very good performance in terms of sensitivity and specificity [4,5]. However, for antibody testing in acute disease, the sensitivity is highly dependent on the kinetics of antibody development. Similarly, specificity is dependent on the type of samples selected to evaluate cross-reactions. It is necessary to evaluate these cross-reactions to other viruses of the coronavirus family. In addition, firms have adopted different strategies in terms of selecting their antigenic base and the type of immunoglobulins detected.

Objectives

The rapid availability of these tests then requires on-site evaluation by users to detect flaws in the results [6,7]. Thus, we evaluated five commercial serological tests widely used worldwide on samples from patients hospitalized for COVID-19, non-hospitalized patients but infected with SARS-CoV-2, patients participating in screening campaigns, and samples from patients with a history of other seasonal coronavirus infections.

Methods

Study design and cohort

The study was conducted at Amiens University medical Center. The study was approved by the institutional review board of the Amiens University Medical Center (number PI2020_843_0046, 21 April 2020).

Samples were derived from de-identified excess serum specimens sent to our clinical virology lab. Patient serum samples used in this study were submitted to the routine serology laboratory.

The assays were validated using serum samples from (i) patients hospitalized for COVID-19 (n = 20), non-hospitalized patients but PCR confirmed with SARS-CoV-2 (n = 58), patients participating in screening campaigns (n = 62), and samples from patients with a history of other seasonal coronavirus infections (n = 28).

Serological assays

The list and characteristics of the different serological tests evaluated are listed in Table 1 . The antigen used in the assay is SARS-CoV-2 nucleocapsid for ABBOTT® and BIORAD®, Spike 1 for EUROIMMUN®, Spike 1 and 2 for LIAISON® and receptor binding domain (RBD) for WANTAI®. ABBOTT®, EUROIMMUN® and LIAISON® detect immunoglobulin G while BIORAD® and WANTAI® detect total antibodies with double antigen bridging assay (DABA). A sample with a doubtful signal was tested a second time and if the result was still the same, the result was considered negative for our evaluation.

Table 1

List and characteristics of the différent serological assays.

	ABBOTT®	BIORAD®	EUROIMMUN®	LIAISON®	WANTAI®
Targeted viral antigen	Nucleocapsid	Nucleocapsid	Spike 1	Spike 1/ Spike 2	Receptor binding domain (RBD)
Immunoglobulins detected	IgG	Total antibodies	IgG	IgG	Total antibodies
Formats	CLIA Indirect antigen down	ELISA Double antigen bridging	ELISA Indirect antigen down	CLIA Indirect antigen down	ELISA Double antigen bridging
Positivity threshold	≥ 1.4	≥ 1	≥ 1.1	≥ 15	≥ 1

CLIA : Chemiluminescence Immunoassay.

ELISA : Enzyme-linked immunosorbent assay.

Data analysis

The demographic information of the 168 patients (sex, age) was extracted from the patient data software (detailed in supplementary Table 1).

Statistical analyses

Sensitivity was defined as the proportion of patients correctly identified as having SARS-CoV-2 infections. Percent of agreement and Kappa index were calculated with GraphPad software v5.1.

Results

Assays sensitivity and specificity

All samples from the 4 patient groups were run through the 5 CoV-2 SARS antibody detection kits. For the first group, with 20 patients hospitalized for COVID-19 with a positive nasopharyngeal SARS-CoV-2 PCR, all samples were positive with these serological assays evaluated (Fig. 1 A). Then for patients also screened for COVID-19 but not hospitalized and patients participating in screening campaigns, disparities between the tests were found. The figures ranged from 91.6 % (WANTAI®) to 69 % (LIAISON®) for the first group and from 40.3 % (WANTAI®) to 21 % (LIAISON®) for the second. These differences do not seem to be due to the antigens chosen by the manufacturers but more to the test formats (IgG detection versus total antibodies). We evaluated the specificity of the different techniques with respect to the four seasonal coronaviruses (OC43, HKU1, NL63, 229E) from 28 serum samples taken away from a PCR positive respiratory sample (between day 7 and day 1153). We observed only one positive sample (EUROIMMUN®) with a low index (1.45 S/Co). Thus, these 5 serological assays do not seem to present cross reactions with the other coronaviruses whatever the antigen selected for the detection of anti-SARS-CoV-2 antibodies.

Fig. 1

Percentages of positive patient samples for the 5 serological techniques evaluated according to patient group (A) and according to the delay between SARS-CoV-2 PCR and serology for the first two patient groups (B).

Then, for the two groups of patients for whom we had a SARS-CoV-2 PCR positive result, we compiled the percentage of positive results according to the delay between PCR and serology (Fig. 1B). For serology from 31 days after the PCR sample, the different techniques showed a positive signal for 100 % of patients with WANTAI® and EUROIMMUN® (21/21), 95 % with BIORAD® and LIAISON® (20/21) and 90 % with ABBOTT® (19/21). However for earlier samples (before D30) we observe a better sensitivity for the techniques searching for total antibodies (100 % between days 11–30 for WANTAI® and BIORAD® versus 60 %, 80 % and 60 % for EUROIMMUN®, ABBOTT® and LIAISON® respectively). Finally 50 days after diagnosis of SARS-CoV-2 by PCR, the percentage of positive sera decreases whatever the technique used with a sensitivity between 71.4 % (10/14) with the LIAISON® assays and 85.7 % (12/14) with the WANTAI® and BIORAD® assays. The raw index values (S/CO) of the different assays according to the four groups are shown in Fig. 2 . We can observe that regardless of the technique with the exception of LIAISON®, hospitalized patients have indexes at the highest possible value. For the following two groups of patients, the index values are very spread out and disparate according to the assays.

Fig. 2

Index of the different evaluated assays for the 4 groups of patients. The dotted lines correspond to the positivity thresholds defined by the different manufacturers and the continuous lines to the median values.

Agreement between serological assays

For these 168 samples divided into 4 groups for which the five serological techniques were performed, we compared the number of positive samples two by two and calculated the overall percent agreement (negative and positive samples) and Kappa index (Table 2 ). For positive samples, for all sera, the highest number was for WANTAI®/BIORAD® (n = 93) while the lowest number was for LIAISON®/EUROIMMUN® and LIAISON®/ABBOTT® (n = 70) (Table 2A). The BIORAD® and ABBOTT® techniques using the Nucleocapsid as an antigenic base had a good percentage of positive approvals (91.4 %) and a high kappa index (0.82) (Table 2B and C). All kappa indices were above 0.6 but with a range from 0.61 (LIAISON®/BIORAD®) to 0.82 (BIORAD®/ABBOTT®).

Table 2

Agreement between serological assays. Comparison of the number of positive samples by two techniques (A), the overall percent agreement (B) and the calculation of Kappa index (C).

A
ALL
n = 168	WANTAI	BIORAD	EUROIMMUN	ABBOTT	LIAISON
WANTAI	98
BIORAD	93	96
EUROIMMUN	83	80	88
ABBOTT	83	84	75	84
LIAISON	73	83	70	70	73

Evaluation of discrepancies between each serological assays

Finally, in order to better study the discrepancies in the positive results in the two groups for which we observed differences (Outpatients with SARS-CoV-2 positive PCR and Outpatients with no history of SARS-CoV-2 infection), we mentioned the indexes on two-by-two comparison histograms (supplementary Fig. 1). We observe most of the time for the positives samples with one assay, that these index numbers are low and rarely at signal saturation.

Finally, we analyzed more finely for these two groups the number of positive tests (from 1 to 5). We obtained for the 88 positive sera by the most sensitive technique (Wantai), 48 (55 %) sera positive with the five different assays (Table 3 ). However, for the rest of the positive samples, we observed significant differences between assays, and in particular for the LIAISON® assay, 90.5 % (48/53) samples found positive with this technique were also positive with the four other assays.

Table 3

Number of positive results by the different evaluated assays following the positive results of one assay in outpatients with SARS-CoV-2 positive PCR and outpatients with no history of SARS-CoV-2 infection.

Outpatients with SARS-CoV-2 positive PCR
	WANTAI	BIORAD	EUROIMMUN	ABBOTT	LIAISON
Number of positive samples	53	52	44	46	40
Number of other positive assays
4	36	36	36	36	36
3	4	4	3	4	1
2	9	7	4	5	3
1	4	5	1	1	0
0	0	0	0	0	0

Discussion

In this study, using 168 samples from a diverse group of patients in the SARS-CoV-2 pandemic, we compared the performance of five widely used serological tests from around the world. Many serological tests in different formats are now available and evaluated by different authorities but never with the same panel of samples. This allows us to compare these tests under real-life conditions with different categories of patients. It is clear that for patients requiring hospitalization for COVID-19, the humoral response to CoV-2-SARS is so exacerbated that all properly developed techniques will have 100 % sensitivity. The sensitivity problem can arise under two conditions. The first is when the antibody detection is too early in the course of the infection, especially for techniques that detect only IgG. The second condition concerns a percentage of SARS-CoV-2 infection with asymptomatic or mild forms, in which case IgG synthesis is absent or low while IgM is probably more frequently detected. Moreover, for this sensitivity problem, manufacturers have had to make new devices available in record time but probably with a preference for specificity over sensitivity in order not to suffer from bad publicity in case of false positive reactions. For example, we have tested the EUROIMMUN® IgA kit which shows a specificity of 90 % on the package leaflet which we have confirmed (data not shown). Antibody testing may therefore be relevant in the following settings: i) diagnosis of patients who seek medical attention more than a week after the onset of symptoms; ii) contact tracing; iii) determining potential immunity and risk of infection; and iv) sero-epidemiological studies to understand the extent of COVID-19 spread. There is a debate as to whether sensitivity or specificity should be preferred for an acute disease for which serology can only provide mainly epidemiological data. Perhaps we will soon have more sensitive techniques while maintaining a good specificity.

In terms of specificity, which we evaluated against other seasonal coronaviruses, all techniques gave excellent results. The different manufacturers have excellent specificity figures, but these must then be evaluated under real conditions because of the diversity of possible reactions and the non-exhaustive search for potential cross-reactions.

As observed in Fig. 1B on samples more than 50 days post-PCR, the percentage of positive results tends to decrease as recently described for neutralizing antibodies [8,9]. All these results raise the question of the role of humoral immunity in relation to cellular immunity in combating this infection and its persistence [10,11]. Although we cannot compare the periods in this Fig. 1B because we do not present a longitudinal follow-up, but an evaluation of this type in the future would be interesting. For hospitalized patients, the positivity of serological tests should be maintained over a longer period of time.

Finally, with the use of these serological assays in daily practice and compared to the results of our study we can affirm that they present good specificity. However, a negative result must always be interpreted with caution according to the clinical context of the serological research, the history of the patient in relation to the suspected or documented SARS-CoV-2 infection and also in relation to the technical characteristics of the diagnostic kits used. Thus, each diagnostic laboratory must adapt its antibody testing strategy to make a result as relevant as possible.

Funding

This work was supported by a grant from the Amiens university Medical center.

Declaration of Competing Interest

All authors have no conflict of interest to declare.