Serological testing for SARS-CoV-2 antibodies in clinical practice: A comparative diagnostic accuracy study.

BACKGROUND: Serological tests are a powerful tool in the monitoring of infectious diseases and the detection of host immunity. However, manufacturers often provide diagnostic accuracy data generated through biased studies, and the performance in clinical practice is essentially unclear.
OBJECTIVES: We aimed to determine the diagnostic accuracy of various serological testing strategies for (a) identification of patients with previous coronavirus disease-2019 (COVID-19) and (b) prediction of neutralizing antibodies against SARS-CoV-2 in real-life clinical settings.
METHODS: We prospectively included 2573 consecutive health-care workers and 1085 inpatients with suspected or possible previous COVID-19 at a Swiss University Hospital. Various serological immunoassays based on different analytical techniques (enzyme-linked immunosorbent assays, ELISA; chemiluminescence immunoassay, CLIA; electrochemiluminescence immunoassay, ECLIA; and lateral flow immunoassay, LFI), epitopes of SARS-CoV-2 (nucleocapsid, N; receptor-binding domain, RBD; extended RBD, RBD+; S1 or S2 domain of the spike [S] protein, S1/S2), and antibody subtypes (IgG, pan-Ig) were conducted. A positive real-time PCR test from a nasopharyngeal swab was defined as previous COVID-19. Neutralization assays with live SARS-CoV-2 were performed in a subgroup of patients to assess neutralization activity (n = 201).
RESULTS: The sensitivity to detect patients with previous COVID-19 was ≥85% in anti-N ECLIA (86.8%) and anti-S1 ELISA (86.2%). Sensitivity was 84.7% in anti-S1/S2 CLIA, 84.0% in anti-RBD+LFI, 81.0% in anti-N CLIA, 79.2% in anti-RBD ELISA, and 65.6% in anti-N ELISA. The specificity was 98.4% in anti-N ECLIA, 98.3% in anti-N CLIA, 98.2% in anti-S1 ELISA, 97.7% in anti-N ELISA, 97.6% in anti-S1/S2 CLIA, 97.2% in anti-RBD ELISA, and 96.1% in anti-RBD+LFI. The sensitivity to detect neutralizing antibodies was ≥85% in anti-S1 ELISA (92.7%), anti-N ECLIA (91.7%), anti-S1/S2 CLIA (90.3%), anti-RBD+LFI (87.9%), and anti-RBD ELISA (85.8%). Sensitivity was 84.1% in anti-N CLIA and 66.2% in anti-N ELISA. The specificity was ≥97% in anti-N CLIA (100%), anti-S1/S2 CLIA (97.7%), and anti-RBD+LFI (97.9%). Specificity was 95.9% in anti-RBD ELISA, 93.0% in anti-N ECLIA, 92% in anti-S1 ELISA, and 65.3% in anti-N ELISA. Diagnostic accuracy measures were consistent among subgroups.
CONCLUSIONS: The diagnostic accuracy of serological tests for SARS-CoV-2 antibodies varied remarkably in clinical practice, and the sensitivity to identify patients with previous COVID-19 deviated substantially from the manufacturer's specifications. The data presented here should be considered when using such tests to estimate the infection burden within a specific population and determine the likelihood of protection against re-infection.

chemiluminescence immunoassay

coronavirus disease‐2019

electrochemiluminescence immunoassay

enzyme‐linked immunosorbent assay

lateral flow immunoassay

nucleoprotein

receptor‐binding domain of the SARS‐CoV‐2 spike glycoprotein

extended receptor‐binding domain of the SARS‐CoV‐2 spike glycoprotein

domain 1 or 2 of the SARS‐CoV‐2 spike glycoprotein

severe acute respiratory syndrome coronavirus 2

INTRODUCTION

Serological tests are a powerful tool in the monitoring of infectious diseases and the detection of host immunity. To help fight the recent coronavirus disease‐2019 (COVID‐19) pandemic representing our time's sincerest health and socioeconomic crisis, various serological assays have been brought to market in record time. , , , , Many of these tests were developed with the ultimate goal to monitor the infection burden within a community, assess vaccination responses, and determine the likelihood of protection against re‐infection. , Broad implementation of serological COVID‐19 tests has also been envisioned to assess the effectiveness of control strategies and facilitate decision‐making on the reopening of schools, cultural facilities, and businesses. , , , Further, such tests might form a basis for the issue of immunity passports, the authorization of international traveling, and the return of employees to work. , Numerous serological studies have recently been conducted, , , , and governments worldwide have ordered millions of serological tests to identify individuals with antibodies against severe acute respiratory syndrome coronavirus 2 (SARS‐CoV‐2) without prior in‐depth clinical validation of the assays.

To enable a meaningful application and interpretation of serological test results, such assays must (a) accurately identify patients with previous COVID‐19 and (b) correctly predict protective immunity acquired by previous infection or vaccination. , , Tests with inadequate performance characteristics will result in misinterpretation of data and might lead to questionable or even counter‐productive health policy decision. , Problematically, manufacturers of serological assays often provide diagnostic accuracy data generated through biased studies and claiming to have a sensitivity and specificity close to 100%. , , , , , , , Design characteristics that frequently lead to erroneous results are as follows: (a) case‐control design, (b) selected samples, (c) partial or differential verification, (d) inadequate flow and timing, and (e) retrospective data collection. , Thus, estimates of diagnostic accuracy are regarded as unreliable. , , Many organizations, including the WHO, now call for the development of reliable antibody tests and evaluation in appropriate diagnostic accuracy studies. ,

Here, we conducted a prospective cross‐sectional study in a real‐life clinical setting, stringently fulfilling the requirements of a diagnostic accuracy study, including (1) an adequately powered prospective design studying clearly defined clinical questions, (2) selection of a representative study population, (3) head‐to‐head comparison of all significant serological testing strategies, (4) rigorous choice and determination of reference standard, and (5) optimal flow and timing. Specifically, we assessed whether different serological testing strategies may (a) accurately identify patients with previous COVID‐19 and (b) correctly predict neutralizing antibodies against SARS‐CoV‐2.

METHODS

Study design and setting

The IA‐COVID study presented here is a prospective cross‐sectional study conducted in a real‐life clinical setting at a Swiss University Hospital (Figure 1). First ad interim results have been published previously. Inselspital is the largest tertiary hospital in Switzerland, covering a catchment area of more than 1 million inhabitants (German and French‐speaking areas). The study protocol was approved by the appropriate ethical committee (Kantonale Ethikkommission Bern #2020‐00683) and the institutional authorities. All participants signed informed consent. The study was conducted in accordance with the declaration of Helsinki.

FIGURE 1

Flowchart illustrating the study design and procedures of a prospective cross‐sectional study in a real‐life clinical setting (diagnostic accuracy study). Consecutive health‐care workers and inpatients with suspected or possible previous COVID‐19 were prospectively included. Abbreviations: ELISA, enzyme‐linked immunosorbent assay; ECLIA, electrochemiluminescence immunoassay; CLIA, chemiluminescence immunoassay; LFI, lateral flow immunoassay; N, nucleoprotein; RBD, receptor‐binding domain of the SARS‐CoV‐2 spike glycoprotein; S1/2, domain 1 or 2 of the SARS‐CoV‐2 spike glycoprotein; RBD+, extended receptor‐binding domain of the SARS‐CoV‐2 spike glycoprotein; RT‐PCR, real‐time PCR

Population

Two groups of individuals representing distinct target populations were included between April and November 2020 (Figure 1): (a) consecutive inpatients with suspected SARS‐CoV‐2 infection (ill patients at risk for complications) and (b) health‐care workers at Inselspital (healthy individuals tested for surveillance measures). The inclusion criteria for inpatients were as follows: (1) hospitalization in Inselspital, (2) tested at least once for SARS‐CoV‐2 by real‐time polymerase chain reaction (RT‐PCR; nasopharyngeal swab), (3) aged 18 or older, and (4) signed informed consent. Only inpatients with more than 4 days of residual sample material were considered. The inclusion criteria for health‐care workers were as follows: (1) medical staff at Inselspital since February 2020, (2) aged 18 or older, and (3) signed informed consent. Health‐care workers were contacted using e‐mail lists. A representative subset of patient samples was analyzed with the live SARS‐CoV‐2 neutralization assay (see below). Additionally, 102 anonymized biobank samples collected from inpatients between December 2018 and February 2019 were analyzed (Liquid Biobank Bern; www.biobankbern.ch). Health‐care workers were contacted by e‐mail lists.

Definition of diagnoses

We defined a positive RT‐PCR from a nasopharyngeal swab as the primary reference standard test representing a “confirmed SARS‐CoV‐2” infection (COVID‐19). We defined “SARS‐CoV‐2” as negative if (a) RT‐PCR was negative in all nasopharyngeal swabs conducted or (b) RT‐PCR not performed. Following applicable regulations, all inpatients and medical staff were supposed to be tested in case of symptoms. Biobank samples were classified as negative as they had been collected before the pandemic (ie, winter 2018/2019).

Handling of samples and collection of data

To ensure adequate pre‐analytical conditions, blood was taken following an established in‐house protocol. Samples were collected using plastic syringes containing serum or lithium heparin, respectively (S‐Monovette®, Sarstedt, Germany). Residual sample material was used in the case of inpatients; serum and heparin tubes were used in medical staff. Samples were immediately transported to the laboratory and centrifuged within 30 min using an established protocol.

Pseudonymized demographical, clinical, and laboratory data were extracted from the patient documentation and transferred by the Insel Data Science Center (IDSC). Additional clinical characteristics were retrieved to explore factors affecting immune response among inpatients. A REDCap database survey was constructed to collect data of medical personnel (demographic, symptoms, comorbidities, and risk factors).

Selection of immunoassays

To study all major serological testing strategies, we selected seven immunoassays with different analytical techniques (enzyme‐linked immunosorbent assay, ELISA; electrochemiluminescence immunoassay, ECLIA; chemiluminescence immunoassay; CLIA; and lateral flow immunoassay, LFI) and epitopes of the SARS‐CoV‐2 (receptor‐binding domain of the spike glycoprotein, RBD; S1/S2 domain of the spike glycoprotein; nucleocapsid, N). Key characteristics of the tests are given in Table 2. Considering the results of our own interim analysis and other publications, we decided to include assays capturing IgG and pan‐Ig.

TABLE 2

Diagnostic accuracy of various serological immunoassays for the presence of previous COVID‐19. Results of a diagnostic accuracy study are shown

Analytical technique	ELISA	ELISA	ELISA	ECLIA	CLIA	CLIA	LFI
Epitope	Anti‐RBD	Anti‐S1	Anti‐N	Anti‐N	Anti‐N	Anti‐S1/S2	Anti‐RBD+
Antibody subtype	IgG	IgG	IgG	Pan‐Ig	IgG	IgG	Pan‐Ig
Manufacturer	In‐house	Euroimmun	Epitope diagnostics	Roche diagnostics	Abbott	DiaSorin	Bühlmann
Numbers of patients	3637	3658	3654	3630	3630	3630	2589
True positives	152	168	126	164	153	160	110
False negatives	40	27	66	25	36	29	21
False positives	98	62	79	54	60	84	95
True negatives	3347	3401	3383	3387	3381	3357	2363
AUC (95% CI)	0.95 (0.93, 0.97)	0.97 (0.95, 0.98)	0.90 (0.86, 0.93)	0.94 (0.91, 0.97)	0.95 (0.93, 0.98)	0.95 (0.92, 0.97)	0.92 (0.88, 0.95)
Sensitivity (percent; 95%CI)	79.2 (72.7, 84.7)	86.2 (80.5, 90.7)	65.6 (58.4, 72.3)	86.8 (81.1, 91.3)	81.0 (74.6, 86.3)	84.7 (78.7, 89.5)	84.0 (76.6, 89.8)
Specificity (percent; 95%CI)	97.2 (96.5, 97.7)	98.2 (97.7, 98.6)	97.7 (97.2, 98.2)	98.4 (98.0, 98.8)	98.3 (97.8, 98.7)	97.6 (97.0, 98.1)	96.1 (95.3, 96.9)
Positive likelihood ratio	28	48	29	55	46	35	22
Negative likelihood ratio	0.21	0.41	0.35	0.13	0.19	0.16	0.17

Consecutive inpatients and medical personnel with suspected COVID‐19 were included in a prospective cross‐sectional study in a real‐life clinical setting (n = 3658). A positive RT‐PCR from a nasopharyngeal swab was defined as the primary reference standard test representing a “previous SARS‐CoV‐2” infection (COVID‐19). Abbreviations: ELISA, enzyme‐linked immunosorbent assay; ECLIA, electrochemiluminescence immunoassay; CLIA, chemiluminescence immunoassay; LFI, lateral flow immunoassay; N, nucleoprotein; RBD, receptor‐binding domain of the SARS‐CoV‐2 spike glycoprotein; S1/2, domain 1 or 2 of the SARS‐CoV‐2 spike glycoprotein; RBD+, extended receptor‐binding domain of the SARS‐CoV‐2 spike glycoprotein.

An in‐house anti‐RBD ELISA was developed and validated as previously described. An ELISA capturing antibodies against the S1 domain was employed (Euroimmune AG, Lübeck, Germany). Besides, an ELISA determining antibodies against N was used (Epitope Diagnostics Inc., San Diego, CA, USA). Three high‐throughput tests were selected: a pan‐Ig ECLIA capturing antibodies against N (Roche diagnostics, Rotkreuz, Switzerland), an IgG CLIA detecting anti‐S1/S2 antibodies (DiaSorin S.p.A., Saluggia, Italy), and an IgG CLIA measuring anti‐N antibodies (Abbott Laboratories, Sligo, Ireland). Aiming to use a sensitive and precise LFI, we employed a pan‐Ig LFI capturing antibodies against an extended RBD protein, which is suggested to be a superior antigen compared to canonical RBD constructs (BÜHLMANN Laboratories, Schönenbuch, Switzerland). The extended RBD is a structurally more confined construct with an even number of cysteines and thus does not form dimerization artifacts during recombinant protein purification. Moreover, it exhibits increased thermal stability and harbors an additional, linear neutralizing antibody epitope.

Detection of SARS‐CoV‐2 specific antibodies

Laboratory technicians and researchers were blinded to the RT‐PCR results and clinical details. All ELISA measurements were done on a DSX automated ELISA system device (DYNEX Technologies, Chantilly, VA, USA) as previously described; instructions of the manufacturers were followed.

For the in‐house ELISA, 96‐well plates were coated overnight at 4°C with 100 µl of 1 µg/ml RBD protein in PBS. For all incubations, that is, blocking and dilution of sera and secondary antibody, PBS containing 0.15% casein (SIGMA, Darmstadt, Germany; C7078‐500G) was used. All incubations lasted 30 min at ambient temperature. Between the incubation steps, plates were washed with 5 × 300 ml PBS containing 0.1% Tween. HRP‐conjugated secondary anti‐human IgG antibody (SIGMA, A0170) was added in a 1:10,000 dilution. Then, TMB‐substrate (SIGMA T4444) was added for 15 min, and the color reaction was stopped with an equal volume of 0.5 M H2SO4. Results were measured at OD450‐620, and samples with an OD >0.5 were considered positive. In all experiments, an internal positive serum was used at a 1:100 dilution. Over 98 measurements, the OD value of this positive control showed a CV of 16.8% (Data not shown).

Concerning the anti‐S1 ELISA measured at OD450‐620, antibody values were expressed as a ratio (ODsample/ODcalibrator), and ratios above 0.8 were assigned positive. With regard to the anti‐N ELISA, measurements were conducted at OD450‐620 and the cutoff was calculated by the following formula: 1.1*(mean ODnegative control + 0.10).

The anti‐N ECLIA was conducted on a Cobas 8000 analyzer (Roche diagnostics, Rotkreuz, Switzerland). The cutoff was calculated based on the calibrator measurements, and a cutoff index s/c ≥ 1.0 was considered positive. Anti‐N CLIA was determined on an Architect i2000SR (Abbott AG, Baar, Switzerland). The cutoff was determined by dividing the measurement of the samples by the mean calibrators result; an index ≥1.4 was considered positive. The anti‐S1/S2 CLIA was determined on a Liaison XL (DiaSorin S.p.A., Saluggia, Italy); arbitrary units ≥12.0 were classified positive.

The anti‐RBD+LFI was conducted as previously described. 10 µl serum was transferred to the application pad, and designated chase buffer was added. Results were analyzed after 15 min using the Quantum Blue® Reader (3rd generation; BÜHLMANN Laboratories, Schönenbuch, Switzerland) that measures the test line intensity of the LFI in gray values [GVs]. Based on the limit of blank plus four times the standard deviation, samples showing GVs of at least 1.5 are considered positive for SARS‐CoV‐2 antibodies. For this study, GVs were also measured and evaluated quantitatively.

Detection of SARS‐CoV‐2 by real‐time PCR

Nasopharyngeal swabs (Copan FLOQSwabs and Copan UTM Viral Transport medium; Copan, Brescia, Italy) were transported at room temperature and stored at 4°C until processing. Three different methodologies for nucleic acid testing (NAT) were employed in the study period. A laboratory‐developed test (LDT) PCR workflow was utilized and two commercial, fully automated assays (Seegene Allplex 2019‐nCoV Assay, Seegene, Seoul, Korea; Roche cobas® SARS‐CoV‐2 Assay, Roche Diagnostics, Rotkreuz, Switzerland). Details of all assays have been reported previously. For the LDT, the E‐gene and the RdRP‐gene were detected, and a cycle threshold of 40 was considered. RT‐PCR was done in clinical practice at the author's institution (Institute for Infectious Diseases, University of Bern, 3012 Bern, Switzerland). Laboratory technicians were not aware of the index test results.

Live SARS‐CoV‐2 neutralization assay

A representative subset of patients was selected for analysis using a live SARS‐CoV‐2 neutralization assay (n = 201). To identify potential differences between immunoassays, we included all complex cases (RT‐PCR+/anti‐S1‐; RT‐PCR‐/anti‐S1+; RT‐PCR+/asymptomatic patients; anti‐S1+/anti‐N‐; anti‐S1‐/anti‐N+) alongside randomly selected antibody‐positive and negative patients. For reasons of priority, we did not select biobank samples for the neutralization assay. Technicians and researchers conducting and interpreting the neutralization assay were blinded regarding RT‐PCR results, serological test details, and clinical details.

Neutralizing antibodies were determined by the inhibition of virus‐induced cytopathic effect (CPE). Methodological details have been described previously. Briefly, sera were incubated at 56°C for 30 min, centrifuged (13,000 g for 10 min), and diluted 1:8 in cell culture medium. Duplicates of fivefold serial dilutions were prepared in 96‐well plates and mixed with 100 TCID50 of infectious SARS‐CoV‐2 in equal volume (BetaCoV/France/IDF0372/2020; National Reference Centre for Respiratory Viruses (Institut Pasteur, Paris, France)). After incubation at 37°C for 1 h, dilutions were transferred to confluent Vero E6 cells (provided by Prof. Dr. Volker Thiel, University of Bern, Bern, Switzerland) and incubated at 37°C, 5%CO2 and >85% relative humidity for three days. CPE was assessed by crystal violet staining. Wells with an undisturbed cell layer were rated as (−), and signs of CPE were rated as (+). Full neutralization titer was defined as the serum dilution without signs of CPE (−) in both duplicates with a limit of detection (LOD) of 1:16.

Statistical analysis

Descriptive statistics were used to describe the distribution of serological test results in individuals with and without COVID‐19, overall and in subgroups. Diagnostic accuracy measures were calculated using pre‐defined cutoffs of the serological index tests in relation to the reference standard as defined above (RT‐PCR). Two‐by‐two tables were generated, sensitivities and specificities calculated, receiver‐operating characteristics (ROC) curves created, and c‐statistics calculated. Serological test results were standardized using z‐scores for test comparison in the context of the neutralization assay. Antibody response between subgroups was compared using unpaired t test; P‐values were reported without adjustment for multiple testing. As a power analysis, we used a method proposed by Bujang et al. We considered a prevalence of 6% (COVID‐19) and a power of 0.8 to detect differences in specificity of 0.05 between assays. Analyses were carried out using the Stata 14.2 statistical software (StataCorp. 2014. Stata statistical software: Release 14. College Station, Tx: StataCorp LP). Figures were created using Prism 6 (GraphPad Software, Inc., LaJolla, California, United States).

Patient and public involvement

The study protocol, the questionnaire, and the first results have been presented on different occasions at Inselspital, and participant feedback was requested and incorporated.

RESULTS

Characteristics of participants

Between April and November 2020, 3658 consecutive individuals were included, comprising 1085 inpatients with suspected SARS‐CoV‐2 infection and 2573 medical personnel (Figure 1). Among those, 195 were COVID positive (prevalence 5.3%). The median age was 46 years (standard deviation, SD, 16.2); 68% of the individuals were female (a high proportion of female nursing staff, in particular). The mean time since PCR was 59 days (SD 47). Detailed patient characteristics are given in Table 1.

TABLE 1

Patient characteristics. Consecutive inpatients and medical personnel with suspected COVID‐19 were included in a prospective cross‐sectional study in a real‐life clinical setting (n = 3658)

Characteristic	No. (%)
	All	COVID‐19 positive	COVID‐19 negative
No. (%)	3658 (100)	195 (5.3)	3463 (94.7)
Age, mean (SD), years	46.0 (16.2)	46.9 (17.0)	45.9 (16.2)
Female gender	2478 (68.4)	108 (55.4)	2370 (69.1)
Inpatients	1085 (29.7)	94 (48.2)	991 (28.6)
Medical personnel	2573 (70.3)	101 (51.8)	2472 (71.4)
Days since PCR, mean (SD), days	58.6 (47.0)	46.0 (48.6)	60.3 (46.5)
Hospitalization	1085 (29.7)	94 (48.2) a	991 (28.6) a
ICU	N/A b	32 (16.4)	N/A b
Mechanical ventilation	N/A b	26 (13.3)	N/A b
No symptoms	N/A b	27 (13.8)	N/A b
Fever	N/A b	94 (48.2)	N/A b

Corresponding to respective number of inpatients.

Information not available in COVID‐19‐negative inpatients.

Diagnostic accuracy for the presence of previous COVID‐19

The distribution of results obtained with various serological tests in patients with and without previous COVID‐19 is given in Figure 2; the respective cutoff levels are depicted as a gray line. The most significant overlap between COVID‐19‐positive and COVID‐19‐negative individuals was observed in the case of anti‐N ELISA. In contrast, little overlap can be seen in anti‐S1 ELISA and anti‐N ECLIA.

FIGURE 2

Distribution of serological testing results in patients with and without previous COVID‐19. IgG or pan‐Ig responses against the nucleoprotein (N), the receptor‐binding domain of the SARS‐CoV‐2 spike glycoprotein (RBD), the domain 1 or 2 of the SARS‐CoV‐2 spike glycoprotein (S1/S2), and the extended RBD domain are shown. Consecutive health‐care workers and inpatients with suspected or possible previous COVID‐19 were prospectively included (n = 3658; prevalence 5.3%)

The diagnostic accuracy of all tests in terms of receiver‐operating characteristics (ROC) curves is given in Figure 3. High area under the ROC curves (AUC) were observed for anti‐S1 ELISA (0.97; 95% confidence interval [CI] 0.95, 0.98), anti‐RBD ELISA (0.95; 95% CI 0.93, 0.97), anti‐S1/S2 CLIA (0.95; 0.92, 0.97), and anti‐N ECLIA (0.94; 0.91, 0.97). Lower AUC values were seen for anti‐N ELISA (0.90; 0.86, 0.93) and anti‐RBD+LFI (0.92; 0.88, 0.95).

FIGURE 3

Accuracy of various serological tests for the identification of patients with previous COVID‐19. Receiver‐operating characteristics (ROC) curves of IgG or pan‐Ig responses against the nucleoprotein (N), the receptor‐binding domain of the SARS‐CoV‐2 spike glycoprotein (RBD), the domain 1 or 2 of the SARS‐CoV‐2 spike glycoprotein (S1/S2), and the extended RBD domain are shown. Consecutive health‐care workers and inpatients with suspected or possible previous COVID‐19 were prospectively included (n = 3658; prevalence 5.3%)

The differences in diagnostic accuracy of various serological tests are illustrated in Figure 4 (panel A); detailed results are given in Table 2. The sensitivity to detect patients with previous COVID‐19 was ≥85% in anti‐N ECLIA (86.8%; 95% CI 81.1, 91.3) and anti‐S1 ELISA (86.2%; 80.5, 90.7), corresponding to 25 and 27 false‐negative results. Sensitivity was 84.7% in anti‐S1/S2 CLIA (78.7, 89.5), 84.0% in anti‐RBD+LFI (76.6, 89.8), 81.0% in anti‐N CLIA (74.6, 86.3), 79.2% in anti‐RBD ELISA (72.7, 84.7), and 65.6% in anti‐N ELISA (58.4, 72.3). The corresponding numbers of false‐negative results were 29 (anti‐S1/S2 CLIA), 21 (anti‐RBD+LFI), 36 (anti‐N CLIA), 40 (anti‐RBD ELISA), and 66 (anti‐N ELISA). Detailed diagnostic accuracy measures for all tests are given in Table 2.

FIGURE 4

Comparative diagnostic accuracy of various serological tests for (A) the identification of patients with previous COVID‐19, and (B) the presence of neutralizing antibodies. Abbreviations: ELISA, enzyme‐linked immunosorbent assay; ECLIA, electrochemiluminescence immunoassay; CLIA, chemiluminescence immunoassay; LFI, lateral flow immunoassay; N, nucleoprotein; RBD, receptor‐binding domain of the SARS‐CoV‐2 spike glycoprotein; S1/2, domain 1 or 2 of the SARS‐CoV‐2 spike glycoprotein; RBD+, extended receptor‐binding domain of the SARS‐CoV‐2 spike glycoprotein

The specificity was 98.4% in anti‐N ECLIA (98.0, 98.8), 98.3% in anti‐N CLIA (97.8, 98.7), 98.2% in anti‐S1 ELISA (97.7, 98.6), 97.7% in anti‐N ELISA (97.2, 98.2), 97.6% in anti‐S1/S2 CLIA (97.0, 98.1), 97.2% in anti‐RBD ELISA (96.5, 97.7), and 96.1% in anti‐RBD+LFI (95.3, 96.9). The corresponding numbers of false‐positive results were 54 (anti‐N ECLIA), 60 (anti‐N CLIA), 62 (anti‐S1 ELISA), 79 (anti‐N ELISA), 84 (anti‐S1/S2 CLIA), 98 (anti‐RBD ELISA), and 95 (anti‐RBD+).

Diagnostic accuracy for the presence of neutralizing antibodies

The accuracy of serological immunoassays for the presence of neutralizing antibodies was observed in a subgroup of complex patients (n = 201). The association between the antibody response (z‐scored) and serum dilutions at full neutralization of live SARS‐CoV‐2 is depicted in Figure 5.

FIGURE 5

Antibody responses and live SARS‐CoV‐2 neutralization. Z‐scored serological test results and serum dilutions at full neutralization are shown as determined in 201 selected individuals (non‐linear curve fitting). Abbreviations: ELISA, enzyme‐linked immunosorbent assay; ECLIA, electrochemiluminescence immunoassay; CLIA, chemiluminescence immunoassay; LFI, lateral flow immunoassay; N, nucleoprotein; RBD, receptor‐binding domain of the SARS‐CoV‐2 spike glycoprotein; S1/2, domain 1 or 2 of the SARS‐CoV‐2 spike glycoprotein; RBD+, extended receptor‐binding domain of the SARS‐CoV‐2 spike glycoprotein

The differences in diagnostic accuracy of various serological tests are illustrated in Figure 4 (panel B); detailed results are given in Table 3. The sensitivity to detect neutralizing antibodies was ≥85% in anti‐S1 ELISA (92.7%; 95%CI 87.3, 96.3), anti‐N ECLIA (91.7%; 86.0, 95.6), anti‐S1/S2 CLIA (90.3%; 84.3, 94.6), anti‐RBD+LFI (87.9%; 80.3, 93.4), and anti‐RBD ELISA (85.8; 91.0, 79.1), corresponding to 11, 12, 14, 13, and 21 false‐negative results, respectively. In contrast, sensitivity was 84.1% in anti‐N CLIA (77.2, 89.7) and 66.2% in anti‐N ELISA (58.0, 73.8). The corresponding numbers of false‐negative results were 23 and 50.

TABLE 3

Diagnostic accuracy of serological immunoassays for the presence of neutralizing antibodies

Analytical technique	ELISA	ELISA	ELISA	ECLIA	CLIA	CLIA	LFI
Epitope	Anti‐RBD	Anti‐S1	Anti‐N	Anti‐N	Anti‐N	Anti‐S1/S2	Anti‐RBD+
Antibody subtype	IgG	IgG	IgG	Pan‐Ig	IgG	IgG	Pan‐Ig
Manufacturer	In‐house	Euroimmun	Epitope diagnostics	Roche diagnostics	Abbott	DiaSorin	Bühlmann
Numbers of patients	197	201	197	188	188	188	157
True positives	127	140	98	133	122	131	95
False negatives	21	11	50	12	23	14	13
False positives	2	4	17	3	0	1	1
True negatives	47	46	32	40	43	42	48
Sensitivity (percent; 95%CI)	85.8 (91.0, 79.1)	92.7 (87.3, 96.3)	66.2 (58.0, 73.8)	91.7 (86.0, 95.6)	84.1 (77.2, 89.7)	90.3 (84.3, 94.6)	87.9 (80.3, 93.4)
Specificity (percent; 95%CI)	95.9 (86.0, 99.5)	92.0 (80.8, 97.8)	65.3 (50.4, 78.3)	93.0 (80.9, 98.5)	100 (91.8, 100)	97.7 (87.7, 99.9)	97.9 (89.2, 100)

A subset of complex patients was selected for analysis using the live SARS‐CoV‐2 neutralization assay because of limited resources (n = 201). Abbreviations: ELISA, enzyme‐linked immunosorbent assay; ECLIA, electrochemiluminescence immunoassay; CLIA, chemiluminescence immunoassay; LFI, lateral flow immunoassay; N, nucleoprotein; RBD, receptor‐binding domain of the SARS‐CoV‐2 spike glycoprotein; S1/2, domain 1 or 2 of the SARS‐CoV‐2 spike glycoprotein; RBD+, extended receptor‐binding domain of the SARS‐CoV‐2 spike glycoprotein

The specificity was ≥97% in anti‐N CLIA (100%; 91.8, 100), anti‐S1/S2 CLIA (97.7%; 87.7, 99.9), and anti‐RBD+LFI (97.9%; 89.2, 100), corresponding to 0, 1, and 1 false‐positive results, respectively. Specificity was 95.9% in anti‐RBD ELISA (86.0, 99.5), 93.0% in anti‐N ECLIA (80.9, 98.5), 92% in anti‐S1 ELISA (80.8, 97.8), and 65.3% in anti‐N ELISA (50.4, 78.3). The corresponding numbers of false‐positive results were 2, 3, 4, and 17, respectively.

Accuracy in salient subgroups of patients

The antibody response in salient subgroups of patients is illustrated in Figure 6 (only COVID‐19‐positive individuals are shown). Significant higher antibody concentrations were observed in males, older individuals, inpatients, and patients admitted to intensive care units, including ventilated patients. As a sensitivity analysis, we calculated the diagnostic accuracy of the anti‐S1 ELISA for the presence of COVID‐19 in various subgroups (Table 4). The sensitivity was higher in males (93.1%), probably reflecting more severe disease in this subgroup of patients. Other significant differences were not observed (Table 4).

FIGURE 6

Antibody response in salient subgroups of patients. Anti‐S1 ELISA results in patients with previous COVID‐19 are given. * p = 0.0005; ** p < 0.0001 

TABLE 4

Diagnostic accuracy of an anti‐S1 ELISA (IgG) for the presence of previous COVID‐19 in various subgroups. Results of a diagnostic accuracy study are given (n = 3637)

Subgroup	Numbers of patients	True positives	False negatives	False positives	True negatives	Sensitivity	Specificity
	No.	No.	No.	No.	No.	Percent (95%CI)	Percent (95% CI)
Patient population
Full study cohort	3658	168	27	62	3401	86.2 (80.5, 90.7)	98.2 (97.7, 98.6)
Inpatients	1085	80	14	15	976	85.1 (76.3, 91.6)	98.5 (97.5, 99.2)
Outpatients b	2573	88	13	47	2438	87.1 (79.0, 93.0)	98.1 (97.5, 98.6)
Gender
Male	1147	81	6	18	1042	93.1 (85.6, 97.4)	98.3 (97.3, 99.0)
Female	2478	87	21	43	2348	80.6 (71.8, 87.5)	98.2 (97.6, 98.7)
Age
<50 years	2201	89	17	42	2070	84.0 (75.6, 90.4)	98.0 (97.3, 98.6)
>50 years	1457	79	10	20	1348	88.8 (80.3, 94.5)	98.5 (97.8, 99.1)
Symptoms
Any symptom	1985	143	25	39	1751	85.1 (78.8, 90.1)	97.8 (97.0, 98.5)
No symptoms	2473	88	13	47	2425	87.1 (79.0, 93.0)	98.1 (97.5, 98.6)
Days since PCR
≥28	3116	96	14	55	2951	87.3 (79.6, 92.9)	98.2 (97.6, 98.6)
≥21	3205	107	15	56	3027	87.7 (80.5, 93.0)	98.2 (97.7, 98.6)
≥14	3304	128	15	57	3104	89.5 (83.3, 94.0)	98.2 (97.7, 98.6)

Health‐care workers.

To further explore factors affecting the immune response among individuals, we compared the lymphocyte count between responders (detectable anti‐S1 antibodies) and non‐responders (non‐detectable antibodies) in inpatients. No significant difference was found (mean 0.74 and 0.66 G/L, respectively; p = 0.55). Further, we retrieved information on immunosuppressive drugs. Three responders and two non‐responders took immunosuppressive drugs. Unfortunately, total antibody levels and other measures of the immune system were not available in a sufficient quality of data.

Biobank samples

Analyzing 102 anonymized biobank samples collected from inpatients between December 2018 and February 2019, serological test results were negative in all samples in case of anti‐S1 ELISA, anti‐N ECLIA, and anti‐S1/S2 CLIA. One positive test result was observed in the case of anti‐RBD ELISA and anti‐N CLIA (1.0%). Four positive test results were observed in case of anti‐N ELISA and anti‐RBD+LFI (3.9%).

DISCUSSION

We conducted a large prospective cross‐sectional study in a real‐life clinical setting stringently fulfilling the requirements of a diagnostic accuracy study and comparing all significant serological testing strategies. Sensitivities and specificities varied remarkably between different tests and were substantially different from manufacturer's specifications. The diagnostic accuracy in detecting patients with previous COVID‐19 was high in anti‐N ECLIA and anti‐S1 ELISA (sensitivity ≥85%; specificity ≥97%). The accuracy in detecting neutralizing antibodies was high in anti‐S1/S2 CLIA and anti‐RBD+LFI (sensitivity ≥85%; specificity ≥97%). Sensitivities and specificities obtained were consistent across various patient subgroups. With these diagnostic accuracy measures obtained in a real‐life clinical setting, we were able to fill a critical gap in knowledge, identified by many previous authors, systematic reviews, and institutions such as the WHO. , , , , , , ,

The study presented here adds important value as it was designed (1) as an adequately powered cross‐sectional study conducted in a real‐life clinical setting, (2) to answer clearly defined clinical questions, (3) to include a representative study population, (3) to conduct a head‐to‐head comparison of all significant serological testing strategies, (4) to select and determine the reference standard test rigorously (e) apply optimal flow and timing. Specifically, we assessed whether different serological testing strategies may (a) accurately identify patients with previous COVID‐19 and (b) correctly predict neutralizing antibodies against SARS‐CoV‐2. However, several potential limitations can be discussed. One might argue that we might have missed COVID‐19 in some asymptomatic health‐care workers because individuals were asked to perform a nasopharyngeal swab in case of symptoms consistent with COVID‐19. However, we do not believe that this might have affected the interpretation because it would not alter sensitivity and does not affect differences between different assays. Besides, a pre‐specified complex subset of patients was selected for the live neutralization assay. This procedure ensures that there is no overestimation of performance. Furthermore, assessing the effects of preexisting immunological characteristics on the immune response was beyond the focus of the current study, and the quality of data does not allow firm conclusions on this issue.

The sensitivities and specificities obtained in clinical practice were considerably lower compared to previous publications. For example, the manufacturer of the anti‐N ELICA claims a sensitivity of 100% (Elecsys Anti‐SARS‐CoV‐2 package insert; Roche diagnostics, Rotkreuz, Switzerland), and previous studies reported sensitivities between 96% and 100%. , , , , , However, these study populations do not reflect real‐life clinical practice, and the diagnostic accuracy measures can therefore not be applied to routine practice. In contrast, the sensitivity in our study, which was strictly designed to reflect clinical settings, was 87%. These differences in sensitivity translate into a completely different interpretation of serological test results in seroprevalence studies and individual patients. Applying the sensitivity provided by the abovementioned manufacturer (anti‐N ECLIA) to our study population, the number of COVID‐19 patients missed by the tests would be zero. In contrast, 25 COVID‐19 patients were missed by the same test in our population (13.2% of RT‐PCR‐positive individuals). The number of missed COVID‐19 patients was even higher in other tests (eg, anti‐N ELISA; n = 66; 34.3%). Accordingly, the specificity of the anti‐N ELICA was stated to be 99.8%, corresponding to 7 patients falsely claimed to have had COVID‐19 in our study cohort (false‐positives). In contrast, we observed 54 false‐positive individuals (falsely claimed to have COVID‐19). These values are very similar in the case of the other serological tests.

The sensitivities and specificities must be taken into account to interpret test results correctly. We would like to give two examples to illustrate how this could be done. In a seroprevalence study in a setting similar to our study cohort, one can estimate the true prevalence by adding the numbers of false‐negatives and subtracting the number of false‐positives as calculated using the diagnostic accuracy measures determined from our study. Likewise, the probability of neutralizing antibodies in individual patients can be estimated in a similar calculation.

Our data support previous knowledge that the majority of patients with COVID‐19 develop antibodies against epitopes of the SARS‐CoV‐2 and that these antibodies can be detected with a range of serological immunoassays. However, this does not apply to all patients and the clinical performance varies remarkably between different assays. Of note, even individuals who do not mount detectable antibodies against SARS‐CoV‐2 may have a robust local adaptive immune response in the nasal, ocular, and airway mucosae and might be well protected for extended periods of time.

A high diagnostic accuracy in terms of previous COVID‐19 was observed for anti‐N ECLIA and anti‐S1 ELISA. In terms of neutralizing antibodies, the accuracy was high in anti‐S1/S2 CLIA and anti‐RBD+LFI. Our data further confirm that the concentration of antibodies detected is strongly associated with the intensity of neutralizing antibodies, irrespective of assay technique. However, major questions remain which must be addressed in future studies: (1) can serological assays be used to distinguish between previous COVID‐19 and vaccination, and (2) what the accuracy of serological assays is to predict protective immunity at certain serological cutoff levels.

CONCLUSIONS

In conclusions, sensitivities and specificities varied remarkably between different tests and were substantially lower than the manufacturer's specifications. These diagnostic accuracy measures can be used to calculate the virus burden within a specific population and determine the likelihood of protection against re‐infection. Thus, our data might inform researchers, health professionals, and authorities to interpret seroprevalence studies and test results in individual patients.

CONFLICT OF INTEREST

The study was supported by a research grant of Bühlmann Laboratories (test kits). All authors declare that there is no conflict of interests. MN received research support from Roche diagnostics outside of the present work. AE is consulting for Bühlmann Laboratories AG and has received honoraria in this context.

AUTHOR CONTRIBUTION

MPH, DB, and AE developed the RBD immunoassay. MPH implemented and conducted all immunoassays. PB and FSR implemented and conducted the RT‐PCR. HRJ and BW performed virus neutralization assays. MN developed the study design, the protocol, and the analysis plan, analyzed, interpreted the data, and wrote the manuscript. TF and CL developed the biobanking solutions. OE provided clinical data. LD and JM included patients. MF and FSR provided infrastructure, contributed to study design, and intellectually reviewed the manuscript. MPH, AE, BW, MF, and CL contributed to study design, interpretation of the results, and drafting of the manuscript. All authors intellectually reviewed the manuscript. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. The corresponding author affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; no important aspects of the study have been omitted, and discrepancies from the study as originally planned have been explained.

COPYRIGHT/LICENSE FOR PUBLICATION

The Corresponding Author has the right to grant on behalf of all authors and does grant on behalf of all authors, a worldwide license to the Publishers and its licensees in perpetuity, in all forms, formats, and media (whether known now or created in the future), to (i) publish, reproduce, distribute, display, and store the Contribution, (ii) translate the Contribution into other languages, create adaptations, reprints, include within collections, and create summaries, extracts, and/or abstracts of the Contribution, (iii) create any other derivative work(s) based on the Contribution, (iv) to exploit all subsidiary rights in the Contribution, (v) the inclusion of electronic links from the Contribution to third party material where‐ever it may be located, and (vi) licence any third party to do any or all of the above.