Comprehensive evaluation of eight commercial SARS-CoV-2 IgG assays.

Sensitivity and specificity of serological assays are key parameters for the accurate estimation of SARS-CoV-2 sero-prevalence. The aim of this study was to compare 8 readily available IgG antibody tests using a panel of well-defined serum samples of prepandemic and pandemic origin. A cross-reaction panel included samples of patients with recent infection with either of the endemic Coronaviruses 229E, NL63, HKU1, or OC43. Additionally, samples with high antibody levels against influenza virus, adenovirus, and during acute EBV infection were included. Previous infection with endemic coronaviruses caused a significant amount of cross-reactivity in two of the assays. In contrast, the confidence intervals for the assays of Abbott, DiaSorin, Euroimmun and Roche encompassed the value of 98% for samples with a previous endemic HCoV infection. For all assays, sensitivities were between 91.3% and 98.8%. Assay performance was independent of the usage of either nucleocapsid or spike proteins.

Introduction

Since its emergence in December 2019 in the Chinese province of Hubei (Zhu et al., 2020), the novel coronavirus SARS-CoV-2 caused significant mortality and morbidity leading to more than 100 million proven infections and 2.300.000 deaths in more than 200 countries world-wide (Di et al., 2020; World Health Organization). Coronaviruses are enveloped and have a large positive-sense, single-stranded RNA-genome (Cui et al., 2019). The disease resulting from SARS-CoV-2 infection, COVID-19, is mild in the majority of cases. However, infection may also result in severe disease courses with acute respiratory distress, multi-organ failure, and death (Di et al., 2020). In acute infections, diagnosis is confirmed by nucleic acid amplification testing (NAAT) of nasal or oropharyngeal swabs as well as deep respiratory specimen. Specific antibodies appear in sufficient amounts not earlier than 10 to 14 days after infection (Kellam and Barclay, 2020; Okba et al., 2020).

Asymptomatic infections as well as patients having rather mild symptoms may result in underestimating the total disease burden. Although ongoing testing for acute infections will provide a point prevalence of acute infections, false negative PCR results, i.e. due to inadequate sampling or low viral load, should be taken into account. The detection of specific IgG directed against SARS-CoV-2 allows the identification of previously infected persons and patients and thus helps to establish more precisely the true prevalence of SARS-CoV-2 in a population.

High specificity and sensitivity of serological tests are of paramount importance. Different antigenic sites on the structural proteins show a varying degree of homology to other members of the Orthocoronavirinae, which could cause false positive test results. Back in 2002/2003, cross-reactivity was noted in the SARS epidemic (Che et al., 2005). Notably, 4 other human Coronaviruses (HCoV), namely the alpha-coronaviruses HCoV-229E and HCoV-NL63 and the beta-coronaviruses HCoV-OC43 and HCoV-HKU1, circulate in the human population (Su et al., 2016). The endemic human coronaviruses are widespread causing predominantly mild infections of the upper respiratory tract (Gaunt et al., 2010; Walsh et al., 2013). Infections could give rise to cross-reactive antibodies depending on their affinity to the specific antigen and thus may influence the performance of SARS-CoV-2 serological assays.

The aim of the study was to compare different commercial assays on a panel of selected serum samples with a focus on an in-depth analysis of specificity and cross-reactivity.

Methods

Specimen

A panel of anonymized leftover serum samples (samples; n = 257) were tested for SARS-CoV-2 IgG antibodies. The panel included 140 pre-pandemic samples from patients with a high probability of cross-reactivity due to a prior infection with endemic coronaviruses (HCoV 229E (n = 17), HCoV HKU1 (n = 14), HCoV OC43 (n = 16), HCoV NL63 (n = 27)), an acute Epstein-Barr virus infection (EBV; n = 25) or high antibody levels against either influenza viruses A and B (n = 21) or adenovirus (n = 20). Furthermore, the panel included 117 samples of patients that were tested for an infection with SARS-CoV-2 by NAAT in 2020. Of these, 37 were randomly chosen samples of in-patients with no history of contact to SARS-CoV-2 infected individuals and a negative SARS-CoV-2 NAAT. Additionally, 80 samples of patients with a laboratory confirmed SARS-CoV-2 infection were included.

SARS-CoV-2 IgG testing

Samples were tested for SARS-CoV-2 specific-IgG with the following assays according to the manufacturer's recommendations: (1) SARS-CoV-2 IgG (Abbott; Chicago, IL), (2) LIAISON SARS-CoV-2 IgG (DiaSorin, Saluggia, Italy), (3) EDI Novel Coronavirus COVID-19 IgG ELISA (Epitope Diagnostics, Inc., San Diego, California, USA), (4) Anti-SARS-CoV-2-ELISA (IgG) and (5) Anti-SARS-CoV-2-NCP-ELISA (IgG) (Euroimmun, Lübeck, Germany), (6) recomWell SARS-CoV-2 IgG (Mikrogen GmbH, Neuried, Germany), (7) COVID-19 ELISA IgG (Vircell, Granada, Spain), (8) Elecsys Anti-SARS-CoV-2 assay (Roche, Basel, Switzerland). Assay specifications are provided in the manufacturer's manuals (Table 1 ).

Table 1

Specifications of the analyzed anti-SARS-CoV-2 IgG assays according to the manufacturers.

Manufacturer	Assay	Used Platforma	Antigen	Test Principle	Measurement	Interpretation		Specificity (according to manufacturer)	Sensitivity (according to manufacturerb)
Abbott	SARS-CoV-2 IgG	Abbott ARCHITECT i2000SR Sytsem	rN	CMIA	index (S/Co)	negative	< 1.4	99.63%	100% (after 14 days)
						positive	≥ 1.4
DiaSorin	Liaison SARS-CoV-2 S1/S2 IgG	DiaSoin LIAISON	rS1/rS2	CLIA	AU/ml	negative	< 12	98.5%	97.4% after 15 days
						indeterminate	12 - < 15
						positive	≥ 15
Epitope Diagnostics	EDI Novel Coronavirus COVID-19 IgG ELISA	manual	rN	ELISA	OD	negative	≤ negative cutoff	100%	100%
						indeterminate	> negative cutoff < positive cutoff
						positive	≥ positive cutoff
Euroimmun	Anti-SARS-CoV-2 ELISA (IgG)	EUROIMMUN Analyzer I	rS1	ELISA	ratio	negative	< 0.8	99%	93.8% (after 20 days)
						indeterminate	0.8 - < 1.1
						positive	≥ 1.1
Euroimmun	Anti-SARS-Cov-2 NCP ELISA (IgG)	EUROIMMUN Analyzer I	rN	ELISA	ratio	negative	< 0.8	99.8%	94.6% (after 10 days)
						indeterminate	0.8 - < 1.1
						positive	≥ 1.1
Mikrogen	recomWell SARS-CoV-2 IgG	manual	rN	ELISA	U/ml	negative	< 20	98.7%	98%
						indeterminate	20 - 24
						positive	> 24
Vircell	COVID-19 ELISA IgG	Vircell VIRCLIA	rS/rN	ELISA	antibody index	negative	< 4	98%	70% (after 7 days)
						indeterminate	4 - 6
						positive	> 6
Roche	Elecsys Anti-SARS-CoV-2c	Roche 8000/e602	rN	ECLIA	COI	negative	< 1.0	99.81%	100% (after 14 days)
						positive	≥ 1.0

r = recombinant; N = nucleocapsid protein; S = spike protein; S1 = subunit 1 of S; S2, Subunit 2 of S; CMIA = chemiluminescent microparticle immunoassay; ECLIA = electrochemiluminescence immunoassay, CLIA = chemiluminescence immunoassay; S/Co, Signal/Cutoff; AU = arbitrary units; OD = optical density; COI, cutoff index.

Platform that was used for the conduction of this study.

Classification of the timeframe after infection varied between the manufacturers. Total or last timeframe is given as indicated.

IgA, IgM, and IgG are detected.

SARS-CoV-2 detection

A published qRT-PCR targeting the viral E gene was used (Corman et al., 2020).

Endemic Coronavirus detection

Testing was done with a multiplex NAAT for respiratory viruses (NxTAG RPP, Luminex Corporation; Austin, TX) according to the manufacturer's instructions.

Endemic Coronavirus IgG antibody testing

IgG antibodies to endemic coronaviruses were detected using the recomLine SASR-CoV-2 IgG blot (Mikrogen GmbH, Neuried, Germany), according to the manufacturer's recommendations.

EBV testing

Acute EBV infection was diagnosed using EBNA-1 IgG, VCA IgG, and IgM CMIA (Architect System; Abbott, Chicago, IL) with subsequent confirmation by recomLine EBV IgG, IgA, and IgM blots (Mikrogen GmbH, Neuried, Germany), according to the manufacturer's recommendations.

Influenza virus A and B IgG antibody testing

IgG antibodies to influenza viruses were detected using the Anti-Influenza A Virus ELISA and the Anti-Influenza B Virus ELISA (Euroimmun, Lübeck, Germany) according to the manufacturer's recommendations. Serum samples with reactivity above 100 relative units/ml (RU/mL) for both viruses were selected (assay cutoff: negative < 16 RU/mL; positive ≥ 22 RU/mL).

Adenovirus IgG antibody testing

IgG antibodies to adenovirus were detected using the Anti-Adenovirus ELISA (Euroimmun, Lübeck, Germany) according to the manufacturer's recommendations. Serum samples with reactivity above 150 RU/mL were selected (assay cutoff: negative < 16 RU/mL; positive ≥ 22RU/mL).

Statistical analysis

Statistical analysis was performed using IBM SPSS Statistics for Windows, Version 25.0 (Armonk, NY: IBM Corp.). For the analysis of sensitivity and specificity, samples with equivocal results were considered positive. The 95% confidence interval (CI) was calculated using the Wilson score method. Positive (PPV) and negative (NPV) predictive values were calculated using Bayes’ theorem with an assumed prevalence of 5%. The correlation analysis was done using the Spearman-Rho test. The scatterplot matrix was built with R (R Core Team 2018) using the GGally package (Schloerke et al., 2018).

Results

Study population

In total, 257 samples of 222 individual patients were included into the analysis. The median age of the patients was 50.8 years with a range from 12 to 91. Of all individuals, 38.3% were female. The median time of detection of endemic coronaviruses before sample collection was 34 days (interquartile range 6.25–96.25 days, range 0–478). An upper respiratory tract infection was reported in 51% (38/74) and a lower respiratory tract infection in 38% (28/74) of the cases. Eleven percent (8/74) did not have a respiratory tract infection. Of the SARS-CoV-2 infected individuals, 15 samples were collected within the first 15 days after symptom onset, the remaining samples thereafter. An upper respiratory tract infection was reported in 26.3% (21/80) and a lower respiratory tract infection in 72.5% (58/80) of the cases. No respiratory symptoms were reported for one case.

Sero-reactivity

Specificity analysis

The majority of antibody assays delivered negative test results. However, sero-reactivity in the pre-pandemic samples differed considerably between the commercial assays (Table 2 ). Lowest specificities were seen for the HCoV NL63 subgroup (Epitope Diagnostics) and the HCoV HKU1 subgroup (Vircell). Reactive samples were mostly reactive in one of the assays, except for six samples that reacted in both the assays of Epitope Diagnostics and Mikrogen. The confidence interval for the assays of Abbott, DiaSorin, Euroimmun, and Roche encompassed a value of at least 98% for the samples with a known previous endemic HCoV infection. While high levels of antibodies against influenza viruses A and B and adenovirus were only associated with an increased cross-reactivity in one assay (Epitope Diagnostics), EBV infection was associated with specificities below 70% for two assays (Epitope diagnostics and Vircell). Again, reactive samples were mostly reactive in one of the assays with exception of one influenza sample that reacted in four of the assays (Epitope diagnostics, Abbott, Mikrogen, and Euroimmun (NCP)). Two other samples were reactive in the assays of Epitope diagnostics, Mikrogen, and Euroimmun (NCP). For pandemic samples (Table 3 , lower panel), the confidence interval included values of at least 98% for all but two assays (Epitope Diagnostics and Mikrogen). Different samples showed a positive reactivity in different assays except for two samples that were tested positive in the assays of Epitope diagnostics and Mikrogen. The specificities of all negative samples were 99.4% (CI 69.9%–99.9%, PPV 89.4) for Abbott, 97.7% (CI 94.3%–99.1%, PPV 67.6) for DiaSorin, 74% (CI 67.1%–79.9%, PPV 16.5) for Epitope Diagnostics, 97.7% (CI 94.3%–99.1%, PPV 68.5) for Euroimmun (S1), 97.7% (CI 94.3%–99.1%, PPV 69.3) for Euroimmun (NCP), 92.7% (CI 87.8%–95.7%, PPV 41) for Mikrogen, 84.2% (CI 78.1%–88.8%, PPV 24.8) for Vircell and 100% (CI 97.9%–100%, PPV 100) for Roche.

Table 2

Test performance using prepandemic sera.

Manufacturer	Abbott	DiaSorin	EpitopeDiagnostics	Euroimmun (S1)	Euroimmun (NCP)	Mikrogen	Vircell	Roche
HCoV 229E infection [n = 17]
positive	0	1	4	0	0	3	1	0
indeterminate	NA	0	0	1	0	0	1	NA
negative	17	16	13	16	17	14	15	17
specificity [%]	100	94.1	76.5	94.1	100	82.4	88.2	100
HCoV NL63 infection [n = 27]
positive	0	0	2	0	0	1	4	0
indeterminate	NA	0	6	1	1	0	1	NA
negative	27	27	19	26	26	26	22	27
specificity [%]	100	100	70.4	96.3	96.3	96.3	81.5	100
HCoV HKU1 infection [n = 14]
positive	0	0	1	1	0	1	4	0
indeterminate	NA	0	0	0	0	0	3	NA
negative	14	14	13	13	14	13	7	14
specificity [%]	100	100	92.9	92.9	100	92.9	50	100
HCoV OC43 infection [n = 16]
positive	0	0	3	0	0	2	2	0
indeterminate	NA	0	0	0	0	0	2	NA
negative	16	16	13	16	16	14	12	16
specificity [%]	100	100	81.3	100	100	87.5	75.0	100
total HCoV samples [n = 74]
positive	0	1	10	1	0	7	11	0
indeterminate	NA	0	6	2	1	0	7	NA
negative	74	73	58	71	73	67	56	74
specificity [%]	100	98.7	78.4	96	98.7	90.5	75.7	100
CI [%]	99.1-100	92.7-99.8	67.7-86.2	88.8-98.7	92.7-99.8	81.2-95.3	64.8-84	99.1-100
PPV [%]	100	78.7	19.2	55.6	80	34.8	17.6	100
acute EBV infection [n = 25]
positive	0	0	12	0	0	0	5	0
indeterminate	NA	0	2	0	0	0	3	NA
negative	25	25	11	25	25	25	17	25
specificity [%]	100	100	44.0	100	100	100	68.0	100
Influenza virus A/B IgG positive [n = 21]
positive	0	0	2	0	2	2	0	0
indeterminate	NA	2	2	0	0	0	1	NA
negative	21	19	17	21	19	19	20	21
specificity [%]	100	90.5	80.1	100	90.5	90.5	95.2	100
Adenovirus IgG positive [n = 20]
positive	1	0	1	1	1	1	0	0
indeterminate	NA	0	1	0	0	0	0	NA
negative	19	20	18	19	19	19	20	20
specificity [%]	95	100	90	95	95	95	100	100
total pre-pandemic samples [n = 140]
positive	1	1	25	2	3	10	16	0
indeterminate	NA	2	11	2	1	0	11	NA
negative	139	137	104	136	136	130	113	140
specificity [%]	99.3	97.9	74.3	97.1	97.1	92.9	80.1	100
CI [%]	96.1-99.9	93.9-99.3	66.5-80.1	92.9-98.9	92.9-98.9	87.4-96.1	87.4-96.1	97.1-100
PPV [%]	87.9	69.6	16.6	63.3	64.2	41.7	20.7	100

CI = 95% confidence interval; PPV = positive predictive value.

Specificity is provided as percent for the negative samples in total and stratified by subgroups.

Table 3

Test performance using pandemic sera.

Manufacturer	Abbott	DiaSorin	EpitopeDiagnostics	Euroimmun (S1)	Euroimmun (NCP)	Mikrogen	Vircell	Roche
pandemic samples of SARS-CoV-2 NAAT positive patients [n = 80]
positive	77	72	78	71	78	75	76	73
indeterminate	NA	1	0	5	1	2	3	NA
negativea	3(2)	7(4)	2(1)	4(3)	1(1)	3(1)	1(0)	7(6)
sensitivity [%]	96.3	91.3	97.5	95	98.8	96.3	98.8	91.3
CI [%]	89.6-98.7	83-95.7	91.3-99.3	87.8-98	93.3-99.8	89.6-98.7	93.3-99.8	83-95.7
pandemic samples from SARS-CoV-2 NAAT negative patients of 2020 [n = 37]
positive	0	1	8	0	0	2	0	0
indeterminate	NA	0	2	0	0	1	1	NA
negative	37	36	27	37	37	34	36	37
specificity [%]	100	97.3	73	100	100	91.9	97.3	100
CI [%]	90.6-100	86.2-99.5	57-84.6	90.6-100	90.6-100	78.7-97.2	86.2-99.5	90.6-100
PPV [%]	100	64	16	100	100	38.5	65.8	100

CI = 95% confidence interval; PPV = positive predictive value.

The number of samples that were tested negative in the respective assay are given. Numbers in brackets indicate serum samples collected < 15 days after symptom onset.

To further characterize the samples that showed cross-reactivity in any of the assays, all reactive samples were tested for IgG antibodies to endemic coronaviruses, adenovirus, influenza virus A and B, and for the EBV status. Comparison was done to the preselected samples for each chosen pathogen and to the pandemic samples of SARS-CoV-2 NAAT negative patients of 2020 (Table 4 ). The highest seroprevalence was detected for the preselected samples for all endemic Coronaviruses except for HCoV-HKU1 when compared to all cross-reactive samples. Cross-reactive samples with a prior infection with HCoV 229E showed the highest seroprevalence for any endemic coronavirus. The seroprevalence for any endemic coronavirus of the samples with a prior infection with endemic coronaviruses was 78.4 % (63.5 % for multiple HCoVs). A high seroprevalence was seen for adenovirus, influenza virus A and B and EBV. However, the mean sero-reactivity was highest in the preselected samples. No additional acute EBV infection could be detected.

Table 4

Serological characterization of cross-reactive samples.

			HCOV						EBV		Influenza virus A		Influenza virus B		Adenovirus
		n	229EP [%]	NL63P [%]	HKU1P [%]	OC43P [%]	any HCoVa P [%]	multiple HCoVbP [%]	P [%]	acute infection [%]	P [%]	Mean [RU/mL]	P [%]	Mean [RU/mL]	P [%]	Mean [RU/mL]
A	HCoV 229E infection	7	85.7	100	85.7	57.1	100	100	100	0	85.7	40.9	85.7	80.6	100	68.1
	HCoV NL63 infection	11	54.5	63.3	63.6	72.7	90.9	72.7	81.8	0	81.8	49.4	81.8	89.5	90.9	67
	HCoV HKU1 infection	8	25	37.5	25	12.5	50	25	87.5	0	37.5	34.1	75	107.3	62.5	91.5
	HCoV OC43 infection	7	71.4	85.7	85.7	71.4	85.7	85.7	100	0	100	65.9	85.7	106.3	85.7	67
	total HCoV samples	33	57.6	69.7	63.3	54.5	81.8	69.7	90.9	0	75.8	47.4	81.8	95.5	84.8	73.1
	acute EBV infection	17	29.4	29.4	35.3	23.5	52.9	41.2	100	100	76.5	52.8	76.5	77.6	82.4	114.8
	Influenza virus A/B IgG positive	6	33.3	66.7	50	33.3	83.3	50	100	0	100	144.2	100	174.5	66.7	62.7
	Adenovirus IgG positive	3	66.7	66.7	33.3	33.3	66.7	66.7	100	0	100	80.7	100	82.3	100	182
	pandemic samples from SARS-CoV-2 NAAT negative patients of 2020	13	53.8	46.2	61.5	46.2	76.9	69.2	84.6	0	84.6	79.2	100	115.2	84.6	79.2
	all cross-reactive samples	72	48.6	55.6	54.2	43.1	73.6	61.1	93.1	25	81.9	61.3	86.1	100.8	83.3	87.7
B	HCoV 229E infection	17	64.7	82.4	64.7	47.1	94.1	82.4	100	0	53	28.5	82.4	79	88.2	46.2
	HCoV NL63 infection	27	40.7	63	44.4	44.4	77.8	55.6	81.4	0	74.1	68.8	81.5	74.4	74.1	68.8
	HCoV HKU1 infection	14	35.7	42.9	42.9	28.6	57.1	35.7	85.7	0	50	83.3	71.4	86.5	64.3	83.3
	HCoV OC43 infection	16	56.3	62.5	81.3	75	81.3	81.3	93.8	0	87.5	70.6	81.3	103.2	87.5	70.6
	total HCoV samples	74	48.6	63.5	41.9	48.6	78.4	63.5	89.2	0	78.4	66.8	79.7	84	78.4	66.8
	acute EBV infection	25	28	24	28	45.3	56	40	100	100	76	48.4	80	92.6	76	109.4
	Influenza virus A/B IgG positive	21	19	42.9	42.9	33.3	61.9	38.1	100	0	100	150.9	100	169	66.7	75.4
	Adenovirus IgG positive	20	30	45	25	20	46	35	90	0	60	56.3	85	97	100	189.9
C	pandemic samples from SARS-CoV-2 NAAT negative patients of 2020	37	22.2	30.6	38.9	25	52.8	33.3	94,6	0	97.2	78.9	100	118.8	88,9	86.4

The seroprevalence of cross-reactive samples (A) was characterized for the indicated pathogens. For EBV the amount of acute infections is given. For Influenza virus A and B and adenovirus the mean reactivity is given. All parameters are also shown for the samples which were specifically selected for the respective pathogen (B) and for the samples of the pandemic samples of SARS-CoV-2 NAAT negative patients of 2020 (C). P, seroprevalence.

Reactive for at least one endemic coronavirus.

Reactive for at least two endemic coronaviruses.

Sensitivity analysis

The sensitivity of all tested assays was above 90% with highest sensitivities seen for the assays of Euroimmun (NCP) and Vircell (Table 3, upper panel). Fifteen samples, collected before day 15 after symptom onset included the majority of false negative test results. However, each of the tested samples was reactive in at least 2 of the assays. If only the samples taken ≥ 15 days after symptom onset were considered, the sensitivities were 100% [CI 94.4%–100%] for Euroimmun (NCP), 96.9% [89.5%–99.2%] for Mikrogen, and 98.5% [91.8%–99.7%] for the remaining assays. The negative predictive value was above 99% for each of the tested assays. Quantitative results were compared pairwise for each of the used assays (Fig. 1 ). Spearman-Rho correlation coefficients were highest for the assays of Euroimmun (S1) and Diasorin (ρ = 0.84), and Euroimmun (NCP) and Abbott (ρ = 0.81).

Fig. 1

Correlation of quantitative assay results of serum samples of patients with a known SARS-CoV-2 infection. Each scatter plot represents the test results of two assays to show interassay correlation (left part). Each dot represents one sample (pandemic samples of SARS-CoV-2 NAAT positive patients). Spearman-Rho correlation coefficient (ρ) is shown on the right part. All correlations are significant (P≤ 0.01), except for +(significant; 0.01> P≤ 0.05) and ‡(not significant). Variable distribution for each assay and test specific units are displayed on the diagonal. (Epitope D, Epitope Diagnostics; Euro, Euroimmun.)

Discussion

Antibody tests are generally suitable (1) to determine seroprevalence in a cohort of patients to be used for defining a denominator of the overall magnitude of regional past infections, (2) to identify blood donors which may serve for the preparation of convalescent plasma, and (3) to investigate rare complications of a SARS-CoV-2 infection, e.g., in children (Viner and Whittaker, 2020).

A pivotal quality of a serological test is high specificity in order to exclude false positive results. However, large discrepancies to the manufacturers’ specifications and to previous studies were determined for some of the assays when challenged with potential cross-reactive samples. In line with previous studies, the highest specificities were seen for the assays of Roche and Abbott, which were reported to be above 99% (Bryan et al., 2020; Espejo et al., 2020; Pflüger et al., 2020). Both were the most reliable assays in the cross-reaction panel. Of the two Euroimmun assays, the one using the spike protein (S1) was previously reported with specificities between 96% and 100% (Espejo et al., 2020; Haselmann et al.2020; Kohmer et al., 2020). However, one study used a cross-reaction panel of 37 samples reporting a specificity of 91.9% (Jääskeläinen et al., 2020). Evaluation of the Euroimmun (NCP) assay is scarce, however one study reported a specificity of 98.2% (Herroelen et al., 2020). Studies on the assay of Epitope Diagnostics mostly report low specificities between 78 and 86% (Haselmann et al., 2020; Krüttgen et al., 2020; Wechselberger et al., 2020; Whitman et al., 2020). However, one study used a modified threshold to obtain a specificity of 98.4% (Bundschuh et al., 2020). For the assays of Mikrogen, Vircell, and Diasorin specificities were reported to range between 97% and 100% (Krüttgen et al., 2020; Wechselberger et al., 2020), 94 and 98% (Alharbi et al., 2020; Wechselberger et al.2020), and 97% and 99.1% (Bonelli et al., 2020; Espejo et al., 2020; Kohmer et al., 2020), respectively. The choice of the antigen, either as complete protein or as a recombinant subunit, could profoundly influence the results of the respective assay. The investigated test systems used either the SARS-CoV-2 spike protein and/or the nucleocapsid protein. The spike protein is expected to be more specific while the nucleocapsid protein is expected to be more sensitive, because of the large abundance during an infection (Motley et al., 2020; Ou et al., 2020; Pflüger et al., 2020). Both proteins show a variable degree of homology to other members of the Orthocoronavirinae (Cui et al., 2019; Lu et al., 2020; Okba et al., 2020; Zuwała et al., 2015). Thus, a recent infection with a coronavirus other than SARS-CoV-2 may lead to false positive test results. Accordingly, high rates of reactive samples were detected for three of the eight tested assays (Epitope Diagnostics, Mikrogen, and Vircell). Interestingly, the alpha-coronaviruses HCoV 229E and HCoV NL63 appeared to have a similar impact on cross-reactivity, in spite of the fact that SARS-CoV-2 is a member of beta-coronaviruses. However, this could be a random effect of the small sample size for the individual coronavirus subgroups. Depending on the rate of coronavirus infections in a respiratory season and occurrence in a given geographical area, the positive predictive values of a given serological SARS-CoV-2 assay may therefore be influenced to varying extents. Performance of the assays was test specific and not dependent on the used antigen as assays using either the nucleocapsid or spike-protein (i.e., both Euroimmun assays) were able to demonstrate an equally high specificity. An acute infection with EBV did not lead to cross-reactivation except in two assays (Epitope Diagnostics and Vircell). Additionally, the presence of high levels of antibodies against influenza viruses A and B and adenovirus were only associated with a marked cross-reactivation in one of the assays (Epitope Diagnostics). The specificity of the assays was closest to the ones stated by the manufacturers when randomly selected samples from the diagnostic routine were used, equivalent to test validations using random blood donor samples. However, we show that a cross-reaction panel is necessary to thoroughly characterize the performance of a serological assay.

The sensitivity of all investigated assays was estimated to be above 90%. Especially, when only samples after 15 days of symptom onset were considered, the sensitivity was similar between the different assays. Additionally, based on an assumed 5% prevalence of SARS-CoV-2 infected individuals, the negative predictive value of all assays was above 99%. The highest sensitivities were calculated for the assays of Euroimmun (NCP) and Vircell. The comparison to other studies, however, is difficult due to the wide spectrum of collection time points after symptom onset. Means to increase sensitivity, may include the use of two assays as reported previously (Gudbjartsson et al., 2020). However, these results underline the recommendation by the World Health Organization not to apply serological assays for the determination of acute infections (WHO). Spearman-Rho correlation coefficients were moderate for most of the compared assay combinations. The highest coefficient was detected for the S-based assays of Diasorin and Euroimmun, which was also reported before (Pflüger et al., 2020), followed by the N-based assays of Abbott and Euroimmun. The best correlation of two assays using different proteins was calculated for both assays of Euroimmun. Interestingly, the assay of Roche only showed a single correlation that was above 0.5 with one of the other assays (Abbott). It is the only pan-Ig assay included in this study and thus, this may be due to an unknown proportion of IgA and IgM reactivity of the assay result. This may be an indication for different areas of application for different test systems: assays for the detection of (formerly infected) individuals and assays that allow the estimation of the level of the immune response after infection or upon vaccination (Kohmer et al., 2020; Weidner et al., 2020). For the later, S1-based assays may be most suitable since the neutralizing immunity is directed towards the spike protein. Interassay correlation may be a requirement for the establishment of international standards. However, correlation of a positive result to immunity is still not well understood. In order to infer humoral immunity, a virus neutralization assay needs to be applied, but a variable degree of correlation was reported (Bonelli et al., 2020; Jiang et al., 2020; Peterhoff et al., 2020). Furthermore, even neutralization tests may not allow a reliable prediction of the immunity against SARS-CoV-2 and its duration.

There are several limitations of this study. The seroprevalence for each of the endemic coronaviruses did not reach 100% in the preselected samples. Unfortunately, there is a lack of thoroughly validated test systems for the detection of the serological response towards endemic coronaviruses, which is also acknowledged by the manufacturer of the used assay. Thus, the results need to be interpreted with caution as neither false negative results nor cross-reactivity can be ruled out. However, the time frame of samples after the infection with an endemic coronavirus used in this study is similar to ones described before (Okba et al., 2020). Additionally, no seroconversion panel was included in this study to adequately describe the sensitivity in different periods after symptom onset. Nevertheless, the analysis of eighty samples allowed a valuable comparison of the investigated assays.

In conclusion, high specificity is of paramount importance and should be the main criteria for the choice of a respective test system. The investigated assays by Abbott, DiaSorin, Euroimmun, and Roche showed the highest estimated specificity, especially with regard to other human Coronaviruses. Other assays may be applied with a subsequent validation by the aforementioned tests.

Author contribution

Study design: UGL; Data collection and testing: MH, CL, MN, MM, CP, TB, ND, TG, VN; Data analysis: MH, UGL; Data interpretation: MH, MM, CL, AD, UGL; Writing: MH, MM, CP, TB, TG, AD, UGL.

Funding

Parts of this study were supported by a grant from the State Parliament of the Free State of Saxony to A.D.

Conflict of interest

All authors have no conflict of interest to declare.