Head-to-head validation of six immunoassays for SARS-CoV-2 in hospitalized patients.

BACKGROUND: Detecting SARS-CoV-2 antibodies may help to diagnose COVID-19. Head-to-head validation of different types of immunoassays in well-characterized cohorts of hospitalized patients remains needed.
METHODS: We validated three chemiluminescence immunoassays (CLIAs) (Liaison, Elecsys, and Abbott) and one single molecule array assay (SIMOA) (Quanterix) for automated analyzers, one rapid immunoassay RIA (AllTest), and one ELISA (Wantai) in parallel in first samples from 126 PCR confirmed COVID-19 hospitalized patients and 158 pre-COVID-19 patients. Specificity of the AllTest was also tested in 106 patients with confirmed parasitic and dengue virus infections. Specificity of the Wantai assay was not tested due to limitations in sample volumes.
RESULTS: Overall sensitivity in first samples was 70.6 % for the Liaison, 71.4 % for the Elecsys, 75.4 % for the Abbott, 70.6 % for the Quanterix, 77.8 % for the AllTest, and 88.9 % for the Wantai assay, respectively. Sensitivity was between 77.4 % (Liaison) and 94.0 % (Wantai) after 10 dpso. No false positive results were observed for the Elecsys and Abbott assays. Specificity was 91.1 % for the Quanterix, 96.2 % for the Liaison, and 98.1 % for the AllTest assay, respectively.
CONCLUSION: We conclude that low sensitivity of all immunoassays limits their use early after onset of illness in diagnosing COVID-19 in hospitalized patients. After 10 dpso, the Wantai ELISA has a relatively high sensitivity, followed by the point-of-care AllTest RIA that compares favorably with automated analyzer immunoassays.

Introduction

SARS-CoV-2 is a novel coronavirus causing the current pandemic of the acute respiratory disease termed COVID-19 [1]. During the acute stage of infection, detection of viral RNA in respiratory specimens by reverse transcriptase polymerase chain reaction (RT-PCR) is the diagnostic method of choice. However, RT-PCR results may be false negative, especially in hospitalized patients who usually present relatively late in the course of infection when viral loads are declining [[2], [3], [4]]. Detecting SARS-CoV-2 antibodies with reliable immunoassays can be of crucial added diagnostic value in these cases [5].

Several SARS-CoV-2 antibody tests are currently commercially available or in development and measure either total antibodies or specific immunoglobulin subclasses binding to the viral nucleocapsid (N) and/or spike (S), including the S1 and S2 subunits and receptor binding domain (RBD), protein. Diagnostic performance of these assays is likely dependent on antigen and technique used and type of antibody subclass detected [[6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20]].

Earlier studies reported the clinical applicability of different types of immunoassays for diagnosis of COVID-19 when added to regular PCR testing. However, validation studies in hospitalized patients were often hampered by small sample sizes, risk of bias, limited specificity testing and lack of parallel evaluation of different relevant assays [[6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20]]. In the current study, samples from a well-characterized cohort of hospitalized patients with PCR-COnfirmed COVID-19 were used for parallel validation of six assays for detection of SARS-COV-2 antibodies, including chemiluminescence-, ELISA- or single molecule array-based assays for high-throughput laboratory testing and a lateral flow-based rapid immunoassay (RIA) for point-of-care diagnostics. Specificity was evaluated using a comprehensive set of samples from pre-COVID-19 patients with documented relevant infectious and non-infectious conditions. This head-to-head validation may help to determine specific clinical applications of each immunoassay during the ongoing pandemic.

Materials and methods

Subjects and samples

The sensitivity panel of samples consisted of plasma and serum samples stored at the Amsterdam University Medical Centres (Amsterdam UMC) COVID-19 Biobank. These samples were first samples from patients with PCR-confirmed COVID-19 admitted to the COVID-19 ward or Intensive Care Unit (ICU) of the Amsterdam UMC during the first SARS-CoV-2 wave in the Netherlands in March and April 2020. Patient characteristics (age, sex, immune status, days post symptom onset (dpso), hospitalization location (i.e., COVID-19 ward or ICU) and PCR results were collected from electronic patient charts and the laboratory information management system. Hospitalization location was defined as the location where the patient resided when the first Biobank specimen was obtained. Immunocompromised status was defined as asplenia or splenic dysfunction, presence of a hematologic malignancy, a history of a bone marrow or solid organ transplantation, Human Immunodeficiency Virus (HIV) infection, and/or chronic steroid use (≥7.5 mg prednisone equivalent per day). In addition, nine hospitalized patients with negative SARS-CoV-2 PCR, but high clinical and radiological suspicion [21], were tested.

The specificity panel of samples consisted of serum samples obtained between January 1st 2016 and November 1st 2019 and stored in the serum bank at the department of Medical Microbiology & Infection Prevention of the Amsterdam UMC. These samples were from patients with a variety of laboratory-confirmed viral, bacterial or fungal infections and from patients from whom samples were obtained for specific clinical reasons (i.e., dialysis and fertility outpatient clinic). To test point-of-care applicability of the RIA for patients in tropical regions, the cohort was expanded with samples from patients with confirmed parasitic and dengue virus infections. Due to their limited sample volumes, these samples were only used for this immunoassay.

Procedures

Upper respiratory tract (i.e., nasopharyngeal and oropharyngeal swabs) and/or lower respiratory tract (i.e., sputum and broncho-alveolar lavage) specimens were obtained as part of routine clinical practice in the COVID-19 work up. Nasopharyngeal and oropharyngeal swabs were collected in E-swab or UTM viral transport medium (COPAN Diagnostics, Murrieta, CA, USA). SARS-CoV-2 RNA was extracted using the MagNA Pure 96 system (Roche, Penzberg, Germany). SARS-CoV-2 PCR targeted at the E-gene was performed according to a previously published protocol [22].

Four immunoassays for automated analyzers were validated, namely the Liaison SARS-CoV-2 S1/S2 IgG chemiluminescence immunoassay (CLIA) on the Liaison XL analyzer (Diasorin, Saluggia, Italy; S protein S1 and S2 subunit-based), the Elecsys Anti-SARS-CoV-2 total antibody electrochemiluminescence immunoassay (ECLIA) on the Cobas e601 analyzer (Roche Diagnostics, Rotkreuz, Switzerland; N protein-based), the SARS-CoV-2 IgG chemiluminescence microparticle immunoassay (CMIA) on the Architect iSR2000 analyzer (Abbott, Chicago, USA; N protein-based), and a pilot pre-commercial version of a total antibody SARS-CoV-2 single molecule array assay (SIMOA) on the HD-X platform (Quanterix Corp., Lexington, USA; S protein RBD and S1 subunit, and N protein-based). These immunoassays were included in the parallel validation, because their respective analyzers are widely used globally. Second, a lateral flow-based RIA, the Biozek COVID-19-IgG/IgM Rapid Test (Inzek International Trading BV, Apeldoorn, The Netherlands; N-protein based), was included that could accommodate SARS-CoV-2 diagnostics in areas with no or limited laboratory infrastructure. Last, the ELISA-based Wantai SARS-CoV-2 total antibody assay (Wantai Biological Pharmacy, Beijing, China; S protein RBD-based) was included for favourable performance in earlier studies that deserved further study [14,15,20]. Due to limited volumes only the sensitivity panel was tested on the Wantai ELISA. All immunoassays were performed according to the manufacturers’ instructions. Results in the equivocal range as indicated by the manufacturer (Liaison CLIA and Wantai ELISA) were considered negative. Results of the Biozek RIA were considered positive if either the IgM band, IgG band, or both were reactive.

Statistical analysis

Overall sensitivity was calculated with first samples of each COVID-19 patient. Stratified sensitivity was calculated for 0–10 dpso and >10 dpso time intervals of first sample collection, and for hospitalization location (i.e., COVID-19 ward or ICU). Continuous variables were expressed as median and interquartile range (IQR) and categorical variables as absolute numbers with proportions. Differences in median dpso between patients on the COVID-19 ward and ICU were analyzed with a Mann Whitney U test. Positive predictive value (PPV) and negative predictive value (NPV) were calculated with sensitivity and specificity in first samples for 10 % community seroprevalence. P < 0.05 was considered statistically significant. Analyses were performed using GraphPad Prism Version 8 (GraphPad, San Diego, CA, USA) and graphs were created using Adobe Illustrator (Adobe Corp. San Jose, CA, USA).

Ethics statement

Informed consent was obtained before storage and usage of residual materials from COVID-19 patients in accordance with the Amsterdam UMC COVID-19 Biobank protocol approved by the local institutional review boards. Ethical review was waved for anonymized use of stored diagnostic specimens for specificity testing in accordance with Dutch law.

Results

Patients and samples

A total of 126 patients with PCR-confirmed COVID-19 were included for sensitivity testing. Of these patients, 73 (58 %) were admitted to the COVID-19 ward and 53 (42 %) to the ICU (Table 1 ). Median dpso was not different between both groups of patients (13 (IQR 8) versus 12 (IQR 6) dpso, respectively (P = 0.49)). Median dpso was 8 (IQR 4) for first samples obtained between 0–10 dpso, and 15 (IQR 6) for first samples obtained after 10 dpso.

Table 1

Characteristics of patients with RT-PCR confirmed COVID-19.

		Total (n = 126)	COVID-19 ward (n = 73)	ICU (n = 53)
Female sex, n (%)		30 (29)	20 (27)	10 (19)
Median age, years (IQR)		64 (15)	65 (15)	62 (16)
Immunocompromised, n (%)a		9 (7)	6 (8)	3 (6)
First positive RT-PCR	URT, n (%)	76 (60)	55 (75)	21 (40)
	LRT, n (%)	50 (40)	18 (25)	32 (60)
Days post symptom onset	0–10, n (%)	42 (33)	25 (34)	17 (32)
	>10, n (%)	84 (67)	48 (66)	36 (68)
	Median (IQR)	12 (5)	13 (8)	12 (6)

RT-PCR = reverse transcriptase polymerase chain reaction; URT = upper airway tract; LRT = lower airway tract; NA = not applicable; ICU = intensive care unit.

Defined as asplenia or splenic dysfunction, presence of (hematologic malignancy, a history of a bone marrow or solid organ transplant, HIV-infection, and/or chronic steroid use (≥ 7.5 mg prednisone equivalent per day).

Specificity was assessed using 158 stored samples, of which 116 were from patients with confirmed bacterial, viral, or fungal infections, and 42 from dialysis patients or individuals visiting the fertility outpatient clinic. An additional 106 samples from patients with confirmed parasitic and dengue virus infection were tested with the RIA.

Performance of the immunoassays

Overall sensitivity in first samples was 70.6 % for the Liaison CLIA, 71.4 % for the Elecsys ECLIA, 75.4 % for the Abbott CMIA, 70.6 % for the Quanterix SIMOA, 77.8 % for the Biozek RIA, and 88.9 % for the Wantai ELISA (Fig. 1 A; Table 2 ). Between 0–10 dpso, sensitivity was between 57.1 % (Liaison CLIA) and 78.6 % (Wantai ELISA), which increased to sensitivities between 77.4 % (Liaison CLIA) and 94.0 % (Wantai ELISA) in samples collected after 10 dpso (Fig. 1B and C; Table 2). Sensitivity was higher in ICU patients compared to patients admitted to COVID-19 wards (Table 2). Separate sensitivities for the Biozek RIA IgM versus IgG are given in Supplemental Table 1. Ten patients tested negative in all assays (Supplemental Table 2), of whom three were immunocompromised and for six patients samples were collected before 10 dpso. Of nine patients with high clinical and radiological suspicion, six (66.7 %) tested positive in the Liaison CLIA and seven (77.8 %) in the other immunoassays.

Fig. 1

Sensitivity of all immunoassays in first samples of patients with RT-PCR confirmed COVID-19.

Bars represent values and whiskers represent 95 % confidence intervals for A) overall sensitivity, B) sensitivity between 0–10 days post symptom onset, and C) sensitivity after 10 days post symptom onset. CLIA = chemiluminescence immunoassay; ECLIA = electrochemiluminescence immunoassay; CMIA = chemiluminescence microparticle immunoassay; SIMOA = single molecule array assay; RIA = rapid immunoassay; ELISA = enzyme-linked immuno sorbent assay.

Table 2

Performance of immunoassays in first samples of patients with RT-PCR confirmed COVID-19.

		Automated analyzer immunoassays				Biozek RIA	Wantai ELISAd
		Liaison	Elecsys	Abbott	Quanterix
Sensitivitya	Overall	70.6 (61.7–78.4)b	71.4 (62.7–79.1)	75.4 (66.9–82.6)	70.6 (61.9–78.4)	77.8 (69.5–84.7)	88.9 (82.1–93.8)
	0–10 dpso	57.1 (41.0–72.3)	45.2 (29.9–61.3)	57.1 (41.0–72.3)	50.0 (34.2–65.8)	61.9 (45.6–76.4)	78.6 (63.2–89.7)
	>10 dpso	77.4 (67.0–85.8)	84.5 (75.0–91.5)	84.5 (75.0–91.5)	81.0 (70.9–88.7)	85.7 (76.4–92.4)	94.0 (86.7–98.0)
	COVID-19 ward	65.8 (53.7–76.5)	69.9 (58.0–80.1)	69.9 (58.0–80.1)	64.4 (52.3–75.3)	69.9 (58.0–80.1)	84.9 (74.6–92.2)
	ICU	77.4 (63.8–87.7)	73.6 (59.7–84.7)	83.0 (70.2–91.9)	79.2 (65.9–89.2)	88.7 (77.0–95.7)	94.3 (84.3–98.8)
Specificity		96.2 (91.9–98.6)c	100 (97.7–100)	100 (97.7–100)	91.1 (85.6–95.1)	98.1 (94.6–99.6)	NT
PPV		67.4 (48.3–82.0)	100	100	47.0 (34.7–59.7)	82.0 (59.7–93.3)	NT
NPV		96.7 (95.7–97.5)	96.9 (96.0–97.7)	97.3 (96.4–98.0)	96.5 (95.5–97.4)	97.5 (96.6–98.2)	NT

Data presented as % (95 % CI). RIA = Rapid immunoassay; ELISA = Enzyme-linked immunosorbent assay; dpso = days post symptom onset; ICU = intensive care unit; PPV = positive predictive value; NPV = negative predictive value; NT = not tested.

Calculated with first samples of each patient.

Equivocal (n = 9) is considered negative.

Equivocal (n = 2) is considered negative.

Equivocal (n = 1) is considered negative.

No false positive results were observed for the Abbott CMIA and the Elecsys ECLIA (Table 2). Specificity was 91.1 % for the Quanterix SIMOA, 96.2 % for the Liaison CLIA, and 98.1 % for the Biozek RIA. Wantai ELISA specificity was not tested. Detailed specificity data with all different conditions listed are shown in Supplemental Table 3a. Specificity for the Biozek RIA was 94.3 % in parasitic and dengue virus infection, with four false positive IgM (i.e., specificity 96.2 %) and two false positive IgG results (i.e., specificity 98.1 %), respectively (Supplemental Table 3b).

At 10 % community seroprevalence, PPV was 67.4 % for the Liaison CLIA, 100 % for the Elecsys ECLIA, 100 % for the Abbott CMIA, 47.0 % for the Quanterix SIMOA, 82.0 % for the Biozek RIA (Table 2). NPV was over 95 % for these immunoassays (Table 2).

Conclusions

In this validation study, six immunoassays for detection of SARS-COV-2 antibodies were validated head-to-head in identical sample sets from PCR-Confirmed COVID-19 patients and pre-COVID-19 patients. Sensitivity was limited in specimens obtained between 0–10 dpso for all immunoassays, ranging from 45.2 % to 78.6 %, increasing to between 77.4 % and 94.0 % after 10 dpso. Specificity ranged from 91.1 % to 100 %, whereby relatively low specificities were noted for the Liaison CLIA, the pilot version of the Quanterix SIMOA and IgM detection of the Biozek RIA. Importantly, the relatively simple lateral flow-based Biozek RIA showed similar or better overall performance compared to the automated immunoassays evaluated.

The diagnostic performance of the Wantai total antibody ELISA is currently unparalleled by other immunoassays [14,15,20]. Literature shows that IgG antibodies directed against N protein precede those against S protein during the course of infection in hospitalized patients [23,24]. Interestingly, the Wantai ELISA is developed with a double sandwich-, or total antibody bridging design that combines detection of IgM, IgA, and IgG antibodies directed against the S protein RBD. A recent report showed higher sensitivity and specificity of an S protein RBD-targeted total antibody bridging assay compared to its N protein equivalent, also in patients with mild disease [25]. Early detection of IgM and IgA directed against S protein RBD most like determined this difference and may also explain the relatively high performance of the Wantai ELISA. Observed sensitivities for the Abbott CMIA, Elecsys ECLIA, and Biozek RIA were similar to those reported earlier, indicating robust performance in different cohorts, especially at increasing dpso [[7], [8], [9], [10],[15], [16], [17], [18],[26], [27], [28]]. Similar to our observations, earlier studies of the Abbott CMIA and Biozek RIA reported detection of SARS-CoV-2 antibodies in up to 90 % of clinically suspected, but PCR-negative patients [17,18]. Overall, these findings support the notion that these immunoassays may be helpful in diagnosing COVID-19 in the absence of PCR confirmation, especially later during the course of infection when viral loads are declining whilst antibodies emerge. As absence of detectable antibodies in early illness does not rule out infection, follow-up samples should obviously be collected in case of negative results. Caution is warranted in immunocompromised patients, since our results suggest lower sensitivity in these patients.

Specificities of the Liaison CLIA and Quanterix SIMOA were low compared to the other immunoassays, but similar to those reported in earlier studies [7,8,28]. In one of these studies, low Liaison CLIA specificity was due to cross-reactivity with two samples from the 2003 SARS-CoV outbreak, which also affected specificity of the Architect CMIA and Elecsys ECLIA [7]. The Quanterix SIMOA we evaluated is a pilot version of a total antibody assay targeted at both RBD and S1 subunit of the S protein, as well as the N protein. A recent validation of this assay similarly showed a low specificity of 94 % in pre-pandemic samples, which appeared to be mostly due to the false positive IgM results [29]. Similarly, IgM detection by the Biozek RIA showed poor specificity in our and other studies, suggesting that only the IgG result can reliably be used for diagnostic purposes [10,13,17,27]. Limited specificity has also been reported for other IgM detection assays, irrespective of the antigen used [30].

Specific test properties determine the clinical applicability of each immunoassay. The CLIAs and SIMOA for random access automated analyzers enable high-throughput testing, but are costly and demand laboratory infrastructure and trained personnel. ELISA-based tests such as the Wantai ELISA require substantial hands-on laboratory time by trained personnel and lacks the flexibility of random access analyzers. The Biozek RIA can directly deliver a result within ten minutes from a finger prick blood sample. Sensitivity and specificity of IgG detection in this assay were comparable to more expensive and technically demanding CLIAs. Specificity of the IgG result was also high in samples from patients with confirmed tropical parasitic and dengue virus infection, indicating that this assay can also reliably be used as an affordable point-of-care test in low-to-middle-income countries where those infections are endemic and resources and laboratory infrastructure are limited.

Strengths of this study include the head-to-head comparison of identical sample sets to allow unbiased comparison of their performance in well-characterized cohorts of hospitalized patients. A weakness of the study is that repeat samples were not available from all patients to allow complete evaluation of serological responses and temporal antibody kinetics.

In conclusion, limited sensitivities early after onset of illness limits the use of all evaluated immunoassays in diagnosing COVID-19 in patients presenting with moderate and severe disease but may contribute to diagnosis later during the course of infection. Performance of tested assays varied, with the Wantai ELISA performing best and the point-of-care RIA comparing remarkably favorably compared to automated analyzer immunoassays.

Funding

Abbott, Diasorin, Roche Diagnostics, and Quanterix provided reagents free of charge. These companies had no part in the design of the study, data analysis, and writing of the manuscript.

Declaration of Competing Interest

Dr. Teunissen has a collaboration contract with ADx Neurosciences and Quanterix and performed contract research or received grants from AxonNeurosciences, Biogen, Boehringer, Brainstorm Therapeutics, Celgene, EIP Pharma, Esai, Janssen prevention center, Roche, Toyama, and Vivoryon.

Dr. de Jong reports personal fees from Roche, personal fees from Cidara, personal fees from Vertex, personal fees from Janssen, and personal fees from GSK, outside the submitted work.

Dr. de Bree reports grants and other from Gilead Sciences, outside the submitted work.