Evaluation of Six Commercial Mid- to High-Volume Antibody and Six Point-of-Care Lateral Flow Assays for Detection of SARS-CoV-2 Antibodies.

Coronavirus disease (COVID) serological tests are essential to determine the overall seroprevalence of a population and to facilitate exposure estimates within that population. We performed a head-to-head assessment of enzyme immunoassays (EIAs) and point-of-care lateral flow assays (POCTs) to detect severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) antibodies. Demographics, symptoms, comorbidities, treatment, and mortality of patients whose sera were used were also reviewed. Six EIAs (Abbott, Affinity, Bio-Rad, DiaSorin, Euroimmun, and Roche) and six POCTs (BTNX, Biolidics, Deep Blue, Genrui, Getein BioTech, and Innovita) were evaluated for the detection of SARS-CoV-2 antibodies in known COVID-19-infected individuals. Sensitivity of EIAs ranged from 50 to 100%, with only four assays having overall sensitivities of >95% after 21 days after symptom onset. Notably, cross-reactivity with other respiratory viruses (parainfluenza virus [PIV-4] [n = 5], human metapneumovirus [hMPV] [n = 3], rhinovirus/enterovirus [n = 1], CoV-229E [n = 2], CoV-NL63 [n = 2], and CoV-OC43 [n = 2]) was observed; however, overall specificity of EIAs was good (92 to 100%; all but one assay had specificity above 95%). POCTs were 0 to 100% sensitive >21 days after onset, with specificity ranging from 96 to 100%. However, many POCTs had faint banding and were often difficult to interpret. Serology assays can detect SARS-CoV-2 antibodies as early as 10 days after symptom onset. Serology assays vary in their sensitivity based on the marker (IgA/IgM versus IgG versus total) and by manufacturer; however, overall only 4 EIAs and 4 POCTs had sensitivities of >95% >21 days after symptom onset. Cross-reactivity with other seasonal coronaviruses is of concern. Serology assays should not be used for the diagnosis of acute infection but rather in carefully designed serosurveys to facilitate understanding of seroprevalence in a population and to identify previous exposure to SARS-CoV-2.

INTRODUCTION

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) was first detected as an unknown cause of pneumonia in December 2019 (1). By 8 January 2020, the Chinese Center for Disease Control and Prevention officially announced a novel coronavirus to be the cause of the outbreak seen in Hubei province (2). The disease spread quickly; at the time of writing (1 July 2020), there were 10,357,662 confirmed cases and 508,055 deaths associated with SARS-CoV-2 globally, which has affected nearly every country in the world (3).

The ability to rapidly diagnose disease, isolate infected patients, and employ contact tracing strategies to mitigate spread of the virus is vital to slowing the spread of infection. Public health laboratories and acute diagnostic laboratories globally have rapidly developed and implemented diagnostic tests to identify COVID-19 disease. In the acute phases of illness, molecular detection of the virus is the primary tool for early and accurate diagnosis of disease (4, 5), as antibody production is usually delayed or absent in the acute phase. Serological assays are now being developed as an epidemiological tool for population-based serosurveys and identification of remote infection (6). However, the full extent of SARS-CoV-2 infection in large populations has yet to be determined due to limited testing (5, 7) and the presence of asymptomatic infection. Therefore, serosurveys must be well designed to best represent the population of interest. To this end, accurate and high-throughput serology assays that can be integrated into laboratory information systems are key to facilitating these large-scale studies and improving the understanding of the true proportion of the population that has recovered from COVID-19.

The primary objective of this study was to conduct a direct comparison of six high- to mid-volume commercial enzyme immunoassays (IgG only or IgG with IgM or IgA or total antibody) and six lateral flow point-of-care assays (IgG and IgM) for detection of SARS-CoV-2 antibodies. All assays were tested against the same panel of serology samples from patients with confirmed COVID-19 and a group of negative controls. High- to mid-volume enzyme immunoassays were also evaluated with a separate panel of convalescent-phase sera to evaluate cross-reactivity to common respiratory viruses and non-SARS-CoV-2 coronaviruses. These performance data, coupled with clinical data from SARS-CoV-2-positive patients, make our evaluation panel particularly robust and significantly add to the current understanding of serology assays for SARS-CoV-2.

MATERIALS AND METHODS

Assay evaluation.

We evaluated six commercial high- to mid-volume kits: two chemiluminescence immunoassays (a chemiluminescent microparticle immunoassay [CMIA]/chemiluminescence immunoassay [CLIA] [SARS-CoV-2 IgG assay; Abbott Laboratories, Abbott Park, IL, USA] and SARS-CoV-2 S1/S2 IgG [DiaSorin, Stillwater, MN, USA]), three enzyme-linked immunosorbent assays (ELISAs; the EDI novel coronavirus COVID-19 IgM and IgG ELISA [Epitope Diagnostics Inc., supplied by Affinity Diagnostics Corp., Toronto, ON, Canada], a novel coronavirus COVID-19 IgM and IgG assay [DRG International Inc., supplied by Bio-Rad, Hercules, CA, USA], and an anti-SARS-CoV-2 ELISA IgA and IgG assay (Euroimmun, Mississauga, ON, Canada]), and one electrochemiluminescence immunoassay (ECLIA; anti-SARS-CoV-2 [Roche Diagnostics, Indianapolis, IN, USA]) for detection of SARS-CoV-2 antibodies. All assays were assessed for detection of SARS-CoV-2 IgG antibodies. Additionally, the Affinity and Bio-Rad tests were assessed for detection of IgM, and the Euroimmun test was assessed for detection of IgA antibodies. Testing was performed as per manufacturer specifications, and cutoffs were determined as described in the package inserts. All values greater than the published cutoff were considered positive. Importantly, all kits were assessed using the same patient samples from single-use aliquots, which negated analyte degradation due to increased freeze-thaw cycles and allowed direct comparison among a large number of commercially available serology assays. For the purposes of this study, CLIA, ECLIA, and ELISAs are referred to as enzyme immunoassays (EIAs). Where specified, targets of EIAs are listed in Table 1.

TABLE 1

Serology assays used in this study

Company	Antibody class detected	Type of assay	Volume and estimated TATa	Assay target	Approvals	Notes
Abbott	IgG	CMIA	High throughput; 45 min per sample	Recombinant antigen nucleocapsid protein	FDA EUA granted 26 April 2020; HC approved 14 May 2020	A cleaning of the instrument before and after running SARS-CoV-2 samples is required (∼40 min total)
Affinity	IgM and IgG	ELISA	Mid-volume; 4 h per 96-well plate	Recombinant antigens of the RBD and spike protein	CE marked	Package insert recommends testing each sample in duplicate
Bio-Rad	IgM and IgG	ELISA	Mid-volume; 4 h per 96-well plate	Antibodies recognizing recombinant nucleocapsid proteins and peptides	Submitted to HC	Package insert recommends testing each sample in duplicate
DiaSorin	IgG	CLIA	High throughput; 40 min per sample	IgG antibodies directed against the S1 and S2 domains of the spike protein	FDA EUA granted 24 April 2020; HC approved 12 May 2020
Euroimmun	IgA and IgG	ELISA	Mid-volume; 4 h per 96-well plate	Recombinant S1 domain of the structural protein	FDA EUA granted 4 May 2020; CE marked	Package insert recommends testing each sample in duplicate
Roche	IgG	ECLIA	High throughput; 45 min per sample	Recombinant protein representing the nucleocapsid antigen	FDA EUA granted 2 May 2020
BTNX	IgM and IgG	Lateral flow	POCT; 15 min per sample	Target unspecified
Biolidics	IgM and IgG	Lateral flow	POCT; 15 min per sample	Recombinant protein, target unspecified
Deep Blue	IgM and IgG	Lateral flow	POCT; 15 min per sample	Target unspecified	Removed from FDA EUA
Genrui	IgM and IgG	Lateral flow	POCT; 15 min per sample	Target unspecified
Getein BioTech	IgM and IgG	Lateral flow	POCT; 15 min per sample	Recombinant nucleocapsid and spike proteins
Innovita	IgM and IgG	Lateral flow	POCT; 15 min per sample	Target unspecified

TAT, turnaround time.

Additionally, we assessed six point-of-care lateral flow tests (POCTs): Rapid Response (BTNX, Markham, Ontario, Canada), the 2019 nCoV IgM/IgG detection kit (Biolidics Limited, Singapore), the SARS-CoV-2 IgG/IgM Ab test kit (Anhui Deep Blue Medical Technology Co., Ltd., Anhui, China), the Novel Coronavirus IgG/IgM test kit (Genrui; Genrui Biotech Inc., Shenzhen, China), the One Step Test for Novel Coronavirus (Getein Biotech Inc., Nanjing, China), and the 2019-nCoV Ab test (Innovita Biological Technology Co. Ltd., Qian'an, Hebei, China). All assays were assessed for the detection of IgM and IgG antibodies. A positive result was determined by any banding detected for either IgM or IgG. Faint banding was considered positive. Assays where the control line was absent were considered invalid. Testing was performed as per manufacturer specifications. Results were read independently by two laboratorians, and when there was a discrepancy, a third laboratorian reading was used as an arbitrator (+/−/− was considered equivocal, +/−/+ was considered positive). Sensitivities at various time intervals and specificities, as well as binomial exact 95% confidence intervals and Fisher’s exact tests, were calculated (Microsoft Excel and STATA v.15 [StataCorp, College Station, TX, USA]).

Precision and reproducibility studies.

Patient sera from 4 patients positive for SARS-CoV-2 by reverse transcriptase real-time PCR (rRT-PCR) were pooled and used as the positive control, while patient sera from 4 patients with sera collected prior to 1 November 2019 (from otherwise healthy individuals with specimens sent for immunity screening) were pooled to create the negative control. Reproducibility was assessed by running replicates of 3 in triplicate for all EIAs. Precision was determined with quality control (QC) material provided by the manufacturer (positive and negative controls), which was run singly each day of testing (Abbott and DiaSorin), or in triplicate for each day of testing (Affinity, Bio-Rad, and Euroimmun). As no QC material was provided for the Roche assay, the pooled positive and pooled negative controls were used as positive and negative controls.

Sample collection.

Negative samples were retrieved from bio-banked sera stored at the public health laboratory (Alberta Precision Laboratories) in Alberta collected before 1 November 2019. To develop a panel of positive sera from patients with COVID-19, serum samples were collected from hospitalized patients confirmed to be positive for SARS-CoV-2 upon nasopharyngeal swab or endotracheal aspirate testing by rRT-PCR. Samples were collected, spun down (3,000 rpm for 10 min), aliquoted into single-use aliquots, and frozen at −80°C until the time of testing.

Eleven COVID-19 positive patients had serum collected at multiple time periods; however, only one sample per patient was used per time interval to calculate assay sensitivity. When more than one serum sample from the same individual was within a given time interval, only the most recently collected serum sample was included.

To evaluate cross-reactivity of the EIA serology tests with other respiratory viruses, convalescent-phase sera (either retrieved from stored sera or prospectively collected) were used (note that the cross-reactivity panel was not assessed on the POCTs). The sera were from patients who had tested negative for COVID-19 by in-house rRT-PCR but positive for other viruses as follows (with the number of sera used): influenza A virus (n = 5), influenza B virus (n = 5), respiratory syncytial virus (RSVA, n = 6; RSVB, n = 1), rhinovirus/enterovirus (n = 6), human metapneumovirus (hMPV; n = 5), parainfluenza virus (PIV-1 and PIV-4; n = 4), CoV-229E (n = 6), CoV-NL63 (n = 11), CoV-OC43 (n = 7), or CoV-HKU1 (n = 7). One patient was positive for multiple viruses (RSVA and enterovirus/rhinovirus). All non-COVID-19 respiratory virus testing was done using the Luminex respiratory pathogen panel (RPP; NxTAG respiratory pathogen panel; Luminex, Austin, TX, USA).

Chart review.

To obtain baseline demographic variables and outcomes for the patients with confirmed SARS-CoV-2 infection, a retrospective chart review of each patient’s electronic medical record was performed by two study team members using the provincial electronic medical record (Epic Systems Corporation, Verona, WI, USA). All symptoms listed were at presentation to hospital, and death was attributed to COVID-19 if it occurred within 30 days of symptom onset. Basic summary statistics (proportions, median, and range) were calculated using Microsoft Excel.

Determining the date of symptom onset.

The date of symptom onset for each case of laboratory-confirmed COVID-19 was determined via history-taking using a standardized history intake form by a member of the Alberta Health Services Communicable Diseases Team (Public Health). All serum samples were stratified by this date to determine the number of days between collection of serum and time of symptom onset.

Ethics.

This study received ethics approval from both the University of Calgary and University of Alberta Health Research Ethics Boards. Certification approval numbers are REB20-0516 and Pro00099818, respectively.

RESULTS

Population demographics.

Serum from 28 patients who tested positive for SARS-CoV-2 by rRT-PCR were used in this study. The mean age of patients was 70.1 (range, 34 to 102 years), with a majority being male (57%; Table 2). Seven percent of patients were ambulatory, while most were hospitalized (93%), and 35% were admitted to the intensive care unit (ICU). Of those who were hospitalized, 27% required mechanical ventilation and 92% developed COVID-19-associated pneumonia. The most common comorbidities in the cohort were hypertension (64%), dyslipidemia (57%), and hypothyroidism (36%). All dates of symptom onset were reported earlier than the date of diagnostic sample collection (mean, 16 days [range, 2 to 48 days]). The time from the date of symptom onset to the date of hospitalization ranged from 0 to 19 days, with a mean of 5 days after symptom onset. Recent travel was reported in 14% of all cases, with the United States being the most frequent location of travel (Table 2).

TABLE 2

Demographic and clinical variables of patients with confirmed SARS-CoV-2 infection (COVID-19) (n = 28)

Variable	Value
Age (yrs)
Mean	70.1
Median	73
Range	34–102
Female (n [%])	12 (43)
Type of specimen used in diagnosis
Nasopharyngeal (n [%])	27 (96)
Endotracheal suction (n [%])	1 (4)
Hospitalized (n [%])	26 (93)
Duration of hospitalization (days)
Range	4–51
Mean	17
Median	11
Time from symptom onset to hospitalization (days)
Range	−29 to 19b
Mean	5
Median	5
Hospitalized patients (n = 26)
ICU admission required (n [%])	9 (35)
Need for mechanical ventilation (n [%])	7 (27)
Pulmonary embolism (n [%])	1 (4)
Development of COVID-19 pneumonia (n [%])
Yes	26 (92)
No	1 (4)
Unknown	1 (4)
Development of acute respiratory distress syndrome (n [%])
Yes	13 (46)
No	14 (50)
Unknown	1 (4)
Died (n [%])	9 (32)
Receipt of investigational treatments (n [%])	8 (29)
HCQ alone	3 (37)
HCQ + AZT	2 (25)
LPV/r	1 (13)
HCQ + LPV/r	2 (25)
Viral copathogen (coronavirus-NL63) (n [%])	1 (4)
Symptoms at presentation (n [%])
Fever	17 (61)
Cough	24 (86)
Dyspnea	26 (93)
Myalgias	10 (36)
Abdominal pain	6 (21)
Diarrhea	4 (14)
Sore throat	2 (7)
Chest pain	10 (36)
Malaise	18 (64)
Anorexia	9 (32)
Comorbidities (n [%])
Hypertension	18 (64)
Diabetes mellitus	7 (25)c
COPD	4 (14)
Coronary artery disease	4 (14)
Valvular disease	2 (7)
Obesity (BMI > 30 kg/m2)	4 (14)
Chronic renal disease	7 (25)
Hypothyroid	9 (36)
Asthma	6 (24)
Congestive heart failure	7 (25)
Atrial fibrillation	4 (14)
Dyslipidemia	16 (57)
Cancer	4 (14)
HIV	0
Travel-related exposures (n [%])
Yes	4 (14)
No	23 (82)
Unknown	1 (4)
Location of travel (n = 4) (of those who travelled) (n [%])
United States	2 (50)
United Arab Emirates	1 (25)
Within Canada	1 (25)
Contact with traveler (n [%])
Yes	6 (21)
No	21 (75)
Unknown	1 (4)
Infection related to outbreak in long-term-care/continuing-care facility (n [%])	9 (32)

Abbreviations: HCQ, hydroxychloroquine; AZT, azithromycin; LPV/r, lopinavir/ritonavir; COPD, chronic obstructive pulmonary disease; BMI, body mass index; HIV, human immunodeficiency virus.

Negative due to one health care-acquired case of COVID-19, which occurred 29 days after hospital admission. If this case is removed, the range is 0 to 19 days.

Median HBA1c (glycated hemoglobin; reported for diabetic patients only), 7.2%; range, 4.2 to 10.9%.

Performance characteristics of EIAs.

In total, 46 samples from 28 different patients testing positive for SARS-CoV-2 by rRT-PCR and 50 negative samples from serum samples stored prior to 1 November 2019 were run on each assay. Overall, the positivity rate for each assay increased over time (Table 3). With the exception of the Affinity assay (100% detection), all assays performed poorly prior to 7 days after symptom onset (range, 40 to 60%; data not shown). However, this improved over time, and all assays had at least 80% sensitivity (range, 80 to 100%) after 21 days (Table 3). The earliest time to detection was seen when assays had a combination of IgG with either IgM or IgA. For example, Affinity IgM and IgG assays had sensitivities of 76% and 62%, respectively, when considered individually at 0 to 14 days after symptom onset; however, when results were combined (with either IgM or IgG being positive in a sample), the overall sensitivity increased to 90% (Table 3). Notably, sera collected 46 and 48 days after symptom onset still had IgM detectable with both Bio-Rad and Affinity Assays and IgA detectable with the Euroimmun assay.

TABLE 3

Performance of six SARS-CoV-2 EIAs by date of serum collection relative to date of symptom onset

Assay	Antibody class	Positive samples																									Negative samples (collected pre–Nov 2019)
		0–14 days						15–21 days						>21 days						All time points
		Neg	Equ	Pos	Sens	Sensb	CIc	Neg	Equ	Pos	Sens	Sensb	CIc	Neg	Equ	Pos	Sens	Sensb	CIc	Neg	Equ	Pos	Sens	Sensb	CIc	nd	Neg	Equ	Pos	Spec	Spece	CIc
Abbott	IgG	6	0	15	71	71	48–89	2	0	9	82	82	48–98	0	0	10	100	100	69–100	8	0	34	81	81	66–91	42	49	0	1	98	98	89–100
Affinity	IgM	5	0	16	76	76	53–92	0	0	11	100	100	72–100	0	0	8	100	100	63–100	5	0	35	88	88	73–96	40	47	0	0	100	100	92–100
	IgG	8	0	13	62	62	38–82	3	1	7	64	73	31–89	1	0	7	88	88	47–100	12	1	27	68	70	51–81	40	47	0	0	100	100	92–100
	IgM/IgG	2	0	19	90	90	70–99	0	0	11	100	100	72–100	0	0	8	100	100	63–100	2	0	38	95	95	83–99	40	47	0	0	100	100	92–100
Bio-Rad	IgM	9	0	12	57	57	34–78	4	1	6	55	64	23–83	5	0	5	50	50	19–81	18	1	23	55	57	39–70	42	49	1	0	98	100	89–100
	IgG	4	0	17	81	81	58–95	1	2	8	73	91	39–94	0	0	10	100	100	69–100	5	2	35	83	88	69–93	42	50	0	0	100	100	93–100
	IgM/IgG	4	0	17	81	81	58–95	1	0	10	91	91	59–100	0	0	10	100	100	69–100	5	0	37	88	88	74–96	42	50	0	0	100	100	93–100
DiaSorin	IgG	11	1	9	43	48	22–66	3	0	8	73	73	39–94	1	0	9	90	90	55–100	15	1	26	62	64	46–76	42	48	1	1	96	98	86–100
Euroimmun	IgA	7	0	14	67	67	43–85	2	0	9	82	82	48–98	0	1	7	88	100	47–100	9	1	30	75	78	59–87	40	46	4	0	92	100	81–98
	IgG	11	0	10	48	48	26–70	3	0	8	73	73	39–94	1	0	7	88	88	47–100	15	0	25	63	63	46–77	40	50	0	0	100	100	93–100
	IgA/IgG	5	0	16	76	76	53–92	2	0	9	82	82	48–98	0	0	8	100	100	63–100	7	0	33	83	83	67–93	40	50	0	0	100	100	93–100
Roche	Total Ab	7	0	14	67	67	43–85	3	0	8	73	73	39–94	2	0	8	80	80	44–97	12	0	30	71	71	55–84	42	50	0	0	100	100	93–100

Abbreviations: Neg, number of negative samples; Equ, number of equivocal samples; Pos, number of positive samples; Sens, percent sensitivity; CI, confidence interval; Spec, percent specificity; Ab, antibody.

Sensitivity if equivocal results are considered positive.

Confidence intervals calculated for sensitivity and specificity where equivocal results are considered negative.

Two invalid samples were observed for Affinity and for Euroimmun (total n = 40).

Specificity if equivocal results are considered positive.

Only the Affinity IgM assay was able to detect antibodies with >95% sensitivity before 21 days after symptom onset. After 21 days after symptom onset, four assays (Abbott, Affinity, Bio-Rad, and Euroimmun) achieved >95% sensitivity; all four had 100% overall sensitivity (Table 3). However, due to the relatively small sample size, confidence intervals show a substantial overlap between time periods. All assays, with the exception of the Euroimmun IgA assay had specificities of >95% for samples collected from patients before 1 November 2019 (Table 3).

Reproducibility for the EIAs was excellent; all assays showed 100% concordance for all samples. Likewise, precision of the assays was high, with all assays having 100% qualitative agreement for positive and negative controls.

Performance characteristics of POCTs.

The same validation panel used for the EIAs was used for all POCTs, with the exception of one specimen (collected on day 13 after symptom onset), as the volume of serum was exhausted following evaluation of the EIAs. As with the EIAs, performance of the POCTs was poor <7 days after onset, with sensitivities ranging from 40 to 60% (data not shown). The performance of the assays increased over time, and all POCTs had >75% sensitivity after 14 days after onset and >90% sensitivity after 21 days (Table 4). However, only 4 assays had overall sensitivities that would be acceptable for use in a clinical laboratory (>95%) after 21 days after symptom onset (Table 4). Of note, IgM was poorly detected by a number of POCTs; Getein detected IgM in only 1 of 42 positive specimens, while Biolidics detected IgM in 10 to 12% of positive samples. The highest sensitivities for IgM detection among the POCTs were consistently reported for BTNX, Deep Blue, and Genrui; these kits had statistically significantly higher overall (i.e., all-time-point calculation) IgM sensitivities than Getein (P < 0.001 for all), Biolidics (P ≤ 0.004 for all), and Innovita (P ≤ 0.009 for all) (Table 4).

TABLE 4

Performance of six SARS-CoV-2 lateral flow assays (POCTs) by date of serum collection relative to date of symptom onset

Assay	Antibody class	Positive samples																									Negative samples (collected pre–Nov 2019)
		0–14 days						15–21 days						>21 days						All time points
		Neg	Equ	Pos	Sens	Sensb	CIc	Neg	Equ	Pos	Sens	Sensb	CIc	Neg	Equ	Pos	Sens	Sensb	CIc	Neg	Equ	Pos	Sens	Sensb	CIc	nd	Neg	Equ	Pos	Spec	Spece	CIc
BTNX	IgM	7	5	8	40	65	19–64	3	1	6	60	70	26–88	0	4	6	60	100	26–88	10	10	20	50	75	34–66	40	50	0	0	100	100	93–100
	IgG	8	1	11	55	60	32–77	3	0	7	70	70	35–93	1	0	9	90	90	55–100	12	1	27	68	70	51–81	40	50	0	0	100	100	93–100
	IgM/IgG	6	1	13	65	70	41–85	2	0	8	80	80	44–97	0	1	9	90	100	55–100	8	2	30	75	80	59–87	40	50	0	0	100	100	93–100
Biolidics	IgM	14	2	4	20	30	6–44	8	0	2	20	20	3–56	6	3	1	10	40	0–45	28	5	7	18	30	7–33	40	48	1	1	96	98	86–100
	IgG	6	1	13	65	70	41–85	2	0	8	80	80	44–97	0	0	10	100	100	69–100	8	1	31	78	80	62–89	40	50	0	0	100	100	93–100
	IgM/IgG	6	1	13	65	70	41–85	2	0	8	80	80	44–97	0	0	10	100	100	69–100	8	1	31	78	80	62–89	40	50	0	0	100	100	93–100
Deep Blue	IgM	5	4	11	55	75	32–77	3	1	6	60	70	26–88	0	0	10	100	100	69–100	8	5	27	68	80	51–81	40	49	1	0	98	100	89–100
	IgG	11	3	6	30	45	12–54	3	1	6	60	70	26–88	1	0	9	90	90	55–100	15	4	21	53	63	36–68	40	50	0	0	100	100	93–100
	IgM/IgG	5	4	11	55	75	32–77	2	1	7	70	80	35–93	0	0	10	100	100	69–100	7	5	28	70	83	53–83	40	50	0	0	100	100	93–100
Genrui	IgM	6	0	14	70	70	46–88	2	0	8	80	80	44–97	0	0	10	100	100	69–100	8	0	32	80	80	64–91	40	48	2	0	96	100	86–100
	IgG	10	0	10	50	50	27–73	3	0	7	70	70	35–93	1	0	9	90	90	55–100	14	0	26	65	65	48–79	40	50	0	0	100	100	93–100
	IgM/IgG	6	0	14	70	70	46–88	2	0	8	80	80	44–97	0	0	10	100	100	69–100	8	0	32	80	80	64–91	40	50	0	0	100	100	93–100
Getein BioTech	IgM	19	0	0	0	0	0–18	9	0	1	10	10	0–45	10	0	0	0	0	0–31	38	0	1	3	3	0–13	39	50	0	0	100	100	93–100
	IgG	11	0	8	42	42	20–67	3	0	7	70	70	35–93	0	0	10	100	100	69–100	14	0	25	64	64	47–79	39	50	0	0	100	100	93–100
	IgM/IgG	11	0	8	42	42	20–67	3	0	7	70	70	35–93	0	0	10	100	100	69–100	14	0	25	64	64	47–79	39	50	0	0	100	100	93–100
Innovita	IgM	9	8	3	15	55	3–38	5	2	3	30	50	7–65	5	3	2	20	50	3–56	19	13	8	20	53	9–36	40	50	0	0	100	100	93–100
	IgG	12	4	4	20	40	6–44	3	1	6	60	70	26–88	1	3	6	60	90	26–88	16	8	16	40	60	25–57	40	50	0	0	100	100	93–100
	IgM/IgG	8	7	5	25	60	9–49	2	1	7	70	80	35–93	1	3	6	60	90	26–88	11	11	18	45	73	29–62	40	50	0	0	100	100	93–100

Abbreviations: Neg, number of negative samples; Equ, number of equivocal samples; Pos, number of positive samples; Sens, percent sensitivity; CI, confidence interval; Spec, percent specificity.

Sensitivity if equivocal results are considered positive.

Confidence intervals calculated for sensitivity and specificity where equivocal results are considered negative.

Control failure on one Getein BioTech sample was reported (n = 19 for the 0- to 14-day time frame).

Specificity if equivocal results are considered positive.

Reading the lateral flow assays was often challenging; many kits frequently produced only very faint lines. Because package inserts did not indicate the appropriate density of bands to call a positive result, for this study, any visualization of a band for either IgM or IgG was considered positive. This approach may have overcalled the sensitivity of the lateral flow assays. Equivocal bands were observed in 9% (3 of 32) of positive samples for Biolidics, 9% (3 of 32) for BTNX, 15% (5 of 33) for Deep Blue, 0% (0 of 33) for Genrui, 4% (1 of 26) for Getein GP BioTech, and 37% (11 of 29) for Innovita.

Cross-reactivity studies with EIAs.

An additional 62 serum samples were used to assess cross-reactivity with other respiratory viruses on EIAs. Of those, 15 sera were collected prior to the first case of SARS-CoV-2 diagnosis in Alberta, and 47 were collected after the first case of SARS-CoV-2 was detected in Alberta. All samples collected after the first case were confirmed to be from patients who tested negative for SARS-CoV-2 by rRT-PCR on nasopharyngeal swab testing. The time from an RPP-positive result to serum collection ranged from 11 to 135 days (mean 45 days) from the date of the original RPP result.

Overall, all assays performed well, with only a few samples showing cross-reactivity. Most notably, the Bio-Rad IgM assay showed cross-reactivity with hMPV for a serum sample collected 31 days after the RPP and with PIV-4 for a serum sample from 120 days post-RPP, while the Bio-Rad IgG assay showed cross-reactivity with rhinovirus/enterovirus (sample collected 48 days post-RPP) and CoV-229E (sample collected 14 days post-RPP). DiaSorin showed cross-reactivity with PIV-4 in a sample collected 54 days post-RPP. The Euroimmun IgA assay showed cross-reactivity with CoV-NL63 (two samples collected 37 and 46 days post-RPP from different patients), CoV-OC43 (two samples collected 15 and 49 days post-RPP from different patients), CoV-229E (sample collected 13 days post-RPP), and PIV-4 (sample collected 54 days post-RPP), while the IgG assay showed cross-reactivity to PIV-4 (sample collected 54 days post-RPP). The Affinity IgM assay showed cross-reactivity to PIV-4 (sample collected 120 days post-RPP). Abbott, Roche, and the Affinity IgG assay did not show any cross-reactivity to other respiratory viruses (Table 5). Overall, PIV-4 and CoV-229E were most cross-reactive across assays, with additional cross-reactivity noted for CoV-NL63 and CoV-OC43 in one assay (Table 5).

TABLE 5

Cross-reactivity of high- to mid-volume serological EIAs with sera from patients infected with other respiratory viruses, by antibody class

Virus	Abbott G		Affinity M		Affinity G		Bio-Rad M		Bio-Rad G		DiaSorin G		Euroimmun A		Euroimmun G		Roche G
	Neg	Pos	Neg	Pos	Neg	Pos	Neg	Pos	Neg	Pos	Neg	Pos	Neg	Pos	Neg	Pos	Neg	Pos
Influenza A virus	5	0	5	0	5	0	4	0	5	0	5	0	5	0	5	0	5	0
Influenza B virus	5	0	5	0	5	0	5	0	5	0	5	0	5	0	5	0	5	0
RSV A	6	0	6	0	6	0	6	0	6	0	6	0	6	0	6	0	6	0
RSV B	1	0	1	0	1	0	1	0	1	0	1	0	1	0	1	0	1	0
Rhinovirus/enterovirus	6	0	6	0	6	0	6	0	5	1	6	0	6	0	6	0	6	0
hMPV	5	0	5	0	4	1	4	1	4	1	4	0	5	0	5	0	5	0
PIV	4	0	3	1	4	0	3	1	4	0	3	1	3	1	3	1	4	0
Coronavirus 229E	6	0	6	0	6	0	5	0	5	1	6	0	5	1	6	0	6	0
Coronavirus NL63	11	0	11	0	11	0	11	0	11	0	11	0	9	2	11	0	11	0
Coronavirus OC43	7	0	7	0	7	0	7	0	7	0	7	0	5	2	7	0	7	0
Coronavirus HKU1	7	0	7	0	7	0	7	0	7	0	7	0	7	0	7	0	7	0

Abbreviations: Neg, number of negative samples; Pos, number of positive samples.

Time course of antibody development.

To evaluate the progression of antibody development, 11 patients in our study had serial serum samples collected (Table 6). Samples ranged from 5 to 29 days after symptom onset, and patients had between 2 and 6 samples collected over time. Overall, Abbott detected 63.6% (7 of 11 patients) of the earliest sample drawn from a patient, Affinity detected 100%, Bio-Rad detected 72.7%, Euroimmun detected 45.5%, DiaSorin detected 36.4%, and Roche detected 63.6% (Table 6). Interestingly, despite four different samples collected from patient 6 (ranging from 18 to 29 days after symptom onset), antibodies were never detected by the Roche assay. Likewise, patient 7 antibodies were not detected by the DiaSorin assay; however, the two samples were collected at days 6 and 8, which is relatively early during the course of infection (Table 6). Once a patient was positive by an assay, all sera from subsequent collection days were also positive; and in no instances did a patient go from positive to negative for this time frame.

TABLE 6

Blood collection over time for SARS-CoV-2-positive PCR patients from time of symptom onset to serum collection date, reported by EIA platform

Day of sample collection is indicated with a gray box with black outline. The earliest detection of antibodies by an assay is indicated as follows: AB, Abbott; AF, Affinity; B, Bio-Rad; D, DiaSorin; E, Euroimmun; R, Roche.

DISCUSSION

We conducted a head-to-head comparison of 12 different serology assays for detection of SARS-CoV-2 antibodies, in 161 different samples from 143 patients. We found that Abbott, Affinity, and Bio-Rad assays had the highest clinical sensitivity and specificity among the EIAs. Despite the relatively small panel size and the wide statistical confidence intervals, we believe that these data are very useful and informative for evaluation, comparison, and validation purposes. Most EIAs and POCTs performed well after 21 days after symptom onset; however, and most importantly, no single assay was sensitive enough to detect antibodies <7 days after symptom onset. Even assays with IgM or IgA components were unable to detect antibodies reliably before 14 days from time of symptom onset (Tables 3 and 4). Furthermore, many of the assays did not perform at a level that is considered acceptable for laboratory diagnostics (>95% sensitivity and specificity). The use of these assays for diagnostic testing is therefore not recommended.

It is worth noting that these performance characteristics were determined using a cohort of COVID-19-positive patients which represented mostly hospitalized patients with high mortality rates. Preliminary studies have suggested that more severe infection may entail higher seroconversion rates and antibodies may develop earlier than in mild infection (8). Therefore, our study data may artificially improve performance characteristics of the assays. Further studies should be done for mostly nonhospitalized COVID-19-positive patients to ensure that findings remain consistent across all cohorts.

As the time course for antibody development has not yet been fully determined for SARS-CoV-2, it is important to compare equivalent start points when analyzing and comparing these data. The sensitivity of serology assays decreases significantly when time of symptom onset rather than time from first PCR positive result is used as the start of infection. In our study, time of PCR positivity was 5.3 days after date of symptom onset on average (range, 0 to 19 days). This time difference has the potential to significantly skew the data. If the time of the first positive PCR result rather than the time from symptom onset had been used as the starting point for this study, all EIAs and POCTs would have been shown to perform better earlier in the disease course. However, using a positive PCR result to define the start of the infection would be an inaccurate reflection of the development of antibody in an individual. Therefore, whenever possible, studies should strive to include performance data in relation to date of onset, to allow true comparison of SARS-CoV-2 testing data.

Cross-reactivity was observed with the respiratory viruses PIV-4, hMPV, rhinovirus/enterovirus, and most importantly CoV-229E, CoV-NL63, and CoV-OC43 in some of the commercial EIAs. To date, only small panels assessing cross-reactivity have been tested, and very little cross-reactivity with coronaviruses other than SARS-CoV-1 has been shown (9). Our larger panel of convalescent-phase sera suggests that cross-reactivity with other respiratory viruses may influence SARS-CoV-2 serology results, particularly following a recent respiratory infection (within 13 to 49 days post-RPP). As the majority of individuals have been exposed to the endemic coronaviruses 229E, NL63, OC43, and HKU1 during their lifetime (10), the cross-reactivity of these viruses in particular should be evaluated. In our hands, two different assays showed cross-reactivity to CoV-229E (Euroimmun IgA and Bio-Rad IgG), and one assay (Euroimmun IgA) showed cross-reactivity to two different patient samples positive for CoV-NL63 and an additional two different patient samples positive for CoV-OC43 (Table 5). An additional study found a single patient sample to be cross-reactive to coronavirus-OC43 (6), and although the sample number tested was small (n = 5), these results are consistent with what we observed in our panel. The potential cross-reactivity of the endemic coronaviruses with SARS-CoV-2 is a significant drawback. Cross-reactivity to other coronaviruses should therefore be considered when interpreting serological results, particularly when determining SARS-CoV-2 prevalence of infection.

We assessed the ability of EIAs and POCTs to detect antibodies against SARS-CoV-2 in human sera; however, no correlation with neutralizing antibodies was performed. While a few neutralizing-antibody studies have been completed and have shown that some antibodies are protective (namely, against the receptor-binding domain [RBD] of the spike domain) (11), the assays that we evaluated in this study detect total, not specifically neutralizing, antibodies. Additionally, not all assays target the same antigen-specific antibodies, and therefore, the kinetics of detection may not be equivalent between assays (Table 1). As antigen presentation and trafficking will vary between different epitopes, variability between assays is expected. We therefore recommend caution in the use of these serology results as indicators of immunity, and we recommend that further studies be done to measure appropriate immunity markers. This will be particularly important for the POCTs, as the antigens used are often not described (Table 1). Additionally, because SARS-CoV-2 is a recently emerged virus, there are few data on the longevity of immunity following infection. Here, we found that both IgM, IgA, and IgG were detected in samples >45 days after symptom onset, suggesting that antibodies are detectable for extended periods; however, more extensive long-term evaluations on the level and duration of immunity following infection are needed.

While the use of serology assays in clinical testing is currently under debate, some recommendations as to the appropriate use of these assays are beginning to emerge. The American Society for Microbiology (ASM) and the WHO recently published similar recommendations against using serology testing for diagnosis of acute infection (12, 13). A natural delay is seen from the time a patient is exposed to the virus to the time the patient starts creating antibodies against the virus. Therefore, regardless of the sensitivity of the assay, there will always be a delay between infection and the development of antibodies. Recent studies suggest that most patients seroconvert between 7 and 14 days, with IgM and IgA being detected as early as 3 and 4 days after symptom onset, respectively (9). However, we found that detection of antibodies earlier than 14 days was unreliable but that the performance of all serology assays improved over time. To this end, both the Centers for Disease Control (14) and the Public Health Agency of Canada (15) have also recommended against using serology assays as an aid in the diagnosis of acute infection. However, serology assays will be helpful in understanding the prevalence of SARS-CoV-2 infection in the population, the timeline of antibody development in different patient populations, and the longevity of the antibody response.

Here, we present results for a comprehensive serology panel consisting of sera from known COVID-19-positive patients and known negatives. We evaluated six different commercial EIA platforms and six POCTs with the same serum panel to give an accurate comparison across all platforms. Based on our results, serology assays should not be used for the diagnosis of acute infections but rather in carefully designed serosurveys to facilitate understanding of seroprevalence in a population and to identify previous exposure to the virus.