Comparative performance data for multiplex SARS-CoV-2 serological assays from a large panel of dried blood spot specimens.

The extent of the COVID-19 pandemic will be better understood through serosurveys and SARS-CoV-2 antibody testing. Dried blood spot (DBS) samples will play a central role in large scale serosurveillance by simplifying biological specimen collection and transportation, especially in Canada. Direct comparative performance data on multiplex SARS-CoV-2 assays resulting from identical DBS samples are currently lacking. In our study, we aimed to provide performance data for the BioPlex 2200 SARS-CoV-2 IgG (Bio-Rad), V-PLEX SARS-CoV-2 Panel 2 IgG (MSD), and Elecsys Anti-SARS-CoV-2 (Roche) commercial assays, as well as for two highly scalable in-house assays (University of Ottawa and Mount Sinai Hospital protocols) to assess their suitability for DBS-based SARS-CoV-2 DBS serosurveillance. These assays were evaluated against identical panels of DBS samples collected from convalescent COVID-19 patients (n = 97) and individuals undergoing routine sexually transmitted and bloodborne infection (STBBI) testing prior to the COVID-19 pandemic (n = 90). Our findings suggest that several assays are suitable for serosurveillance (sensitivity >97% and specificity >98%). In contrast to other reports, we did not observe an improvement in performance using multiple antigen consensus-based rules to establish overall seropositivity. This may be due to our DBS panel which consisted of samples collected from convalescent COVID-19 patients with significant anti-spike, -receptor binding domain (RBD), and -nucleocapsid antibody titers. This study demonstrates that biological specimens collected as DBS coupled with one of several readily available assays are useful for large-scale COVID-19 serosurveillance. Crown

Introduction

While molecular and antigen tests remain the standard for diagnosing active coronavirus disease 2019 (COVID-19) [1,2], the breadth and depth of the pandemic will be better understood with the help of serological testing for the presence of antibodies against the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) [3]. Serological assays currently authorized or under review by Health Canada require serum or plasma to be collected using standard phlebotomy (https://www.canada.ca/en/health-canada/services/drugs-health-products/covid19-industry/medical-devices/testing/serological.html). This requirement poses significant challenges in implementing population-based serosurveillance due to cost [4], increased risk of exposures during in-person biological specimen collection [5], shortages of healthcare resources [6], as well as geographical and structural barriers especially in northern, remote or isolated settings [7, 8, 9]. Serological tests based on point-of-care (POC) lateral flow immunoassays (LFIAs) would circumvent the need for phlebotomy, but their performance is currently sub-optimal [10]. In contrast, dried blood spot (DBS) specimens are an attractive choice for utilization in serological procedures considering their simplicity [11], acceptability for self-sampling [10, 11, 12, 13, 14], stability at room temperature [15, 16], and proven clinical utility for other viral infections including HIV, hepatitis C, and hepatitis B [17, 18].

Although not used as frequently for SARS-CoV-2, earlier DBS validation studies have reported encouraging performance [4, 14, 15, 19, 20, 21, 22, 23]. DBS were found to be comparable to serum specimens with most studies reporting a sensitivity and specificity of ≥90% and 100% respectively [4, 15, 19, 22, 23]. The serological tests under investigation were largely based on single antigens for the S1 subunit of the spike glycoprotein [4, 5, 24, 25], receptor binding domain (RBD) [22, 24], or nucleocapsid phosphoprotein [15, 21, 26, 27], but these tests may be of limited use in the context of population-based serosurveillance [28]. Antibody responses following a SARS-CoV-2 infection are dynamic in nature and vary according to the antigen of interest. Studies conducted in healthcare workers estimate that the half-life for anti-spike antibodies was more than double the half-life of anti-nucleocapsid antibodies [29, 30], findings that suggest trends in antibody decline have possible impacts on seroprevalence estimates and for follow-up testing in suspected cases of multisystem inflammatory syndrome [28, 31, 32]. Furthermore, it is difficult to resolve infection-acquired immunity from vaccine-induced immunity without the nucleocapsid antigen or other non-spike antigens [33, 34]. Orthogonal testing could be considered but is likely impractical for DBS given the limited amount of blood that is collected on a standard DBS card (∼400 μL) compared to phlebotomy (∼10 mL/tube). DBS sampling in combination with multi-antigen (multiplex) serological testing may be the most appropriate option for serosurveillance since it addresses all concerns related to biological specimen collection and single antigen-based testing [28]. To date, all reports of DBS validation studies with multiplex serological tests have been limited to in-house developed tests [23, 35] and MESO electrochemiluminescence immunoassays (MSD, Rockville, MD) [36]. Direct comparative performance data on multiplex SARS-CoV-2 serological tests derived from identical DBS specimens are significantly lacking.

Here, our objective is to provide performance data for the BioPlex 2200 SARS-CoV-2 IgG (Bio-Rad, Hercules, CA), V-PLEX SARS-CoV-2 Panel 2 IgG (MSD), and Elecsys Anti-SARS-CoV-2 (Roche, Basel, Switzerland) commercial multiplex assays, as well as for two in-house multiplex assays (University of Ottawa and Mount Sinai Hospital protocols) to evaluate their ability to detect SARS-CoV-2 antibodies in DBS for the purpose of serosurveillance. All the assays are evaluated against identical panels of DBS collected from convalescent COVID-19 patients (n = 97) and individuals undergoing routine sexually transmitted and bloodborne infection (STBBI) testing prior to the COVID-19 pandemic (n = 90). Performance is quantified using sensitivity, specificity, positive predictive values (PPV), and negative predictive values (NPV).

Materials and methods

Ethics statement

All experiments were carried out in accordance with relevant guidelines and regulations. Written informed consent was obtained from all participants who provided blood samples. All participants were 18 years of age or older. Ethical approval was obtained from the Health Canada and Public Health Agency of Canada Research Ethics Board (no. 2020–022P).

Dried blood spot panel

A DBS panel consisting of 97 unique SARS-CoV-2 antibody positive and 90 unique SARS-CoV-2 negative DBS specimens was used to assess performance defined in terms of sensitivity, specificity, negative predictive values (NPV), and positive predictive values (PPV) [37]. The assays are described in greater detail in Table 1 . All testing sites were blinded to the status of the DBS specimens and they were asked to perform testing as per standard operating procedures. SARS-CoV-2 antibody positive blood was collected from convalescent COVID-19 patients (positive diagnostic nucleic acid test) at Mount Sinai Hospital (Toronto, Canada) using standard phlebotomy. DBS specimens were collected during the first wave of the pandemic before the availability of vaccines. Therefore, these patients were likely infected by wild type SARS-CoV-2. Several variants of concern (VOCs) have been detected in Canada since our study, but they have not impacted assay performance (personal communications). Blood from BD Vacutainer EDTA tubes (BD, Mississauga, Canada) was directly spotted onto each circle of a Whatman 903 Proteinsaver cards (5 × 75 μL; GE Healthcare, Boston, MA) using pipettes, air-dried at room temperature for a minimum of 3 h (maximum 24 h), and stored at -80 °C in Bitran SARANEX bags (Fisher Scientific, Ottawa, Canada) with desiccant. The commercial assays under investigation require 1 or 4 × 6 mm punches corresponding to approximately 14 μL and 56 μL of whole blood respectively. The in-house assays only required 1 × 3.2 mm punches or approximately 7.5 μL of whole blood. DBS were prepared by applying multiple drops to fill each circle instead of overlaying drops to facilitate uniform blood distribution and drying. Each DBS panel was visually inspected prior to shipment to ensure sample quality was satisfactory to minimize punch-to-punch variability. DBS specimens were prepared from venous blood to minimize patient discomfort i.e., prevent concurrent finger and venous punctures.

Table 1

Overview of the SARS-CoV-2 multiplex serological assays evaluated in this study.

	In-house	In-house	Bioplex 2200 SARS-CoV-2 IgG	V-PLEX SARS-CoV-2 Panel 2 (IgG)	Elecsys Anti-SARS-CoV-2	Elecsys Anti-SARS-CoV-2
Manufacturer	University of Ottawa	Mount Sinai Hospital	Bio-Rad Laboratories	Meso Scale Diagnostics	Roche	Roche
Testing site	CI3	Mount Sinai Hospital	NML at JC Wilt	NML at JC Wilt	NML at CSCHAH	NMAL at CSCHAH
Assay principal	CLIA	CLIA	MFI	ECLIA	ECLIA	ECLIA
Format	384-well plates	384-well plates	Individual tubes	96-well plates	Individual tubes	Individual tubes
Antigen(s)	S, RBD, N	S, RBD, N	S1, RBD, N	S, RBD, N	RBD	N
Antibody class	IgG	IgG	IgG	IgG	Total antibody	Total antibody
Output	S/CO	RR	U/mL	AU/mL	U/mL	COI
Thresholds
Non-reactive (S)	S/CO < 1.0	<3SD mean of control	<10 U/mL	<1,960 AU/mL	<0.8 U/mL	COI <1.0
Reactive (S)	S/CO ≥ 1.0	≥3SD mean of control	>10 U/mL to ≤100 U/mLa	≥1,960 AU/mL	≥0.8 U/mL	COI ≥1.0
Non-reactive (RBD)	S/CO < 1.0	<3SD mean of control	<10 U/mL	<538 AU/mL	<0.8 U/mL	COI <1.0
Reactive (RBD)	S/CO ≥ 1.0	≥3SD mean of control	>10 U/mL to ≤100 U/mLa	≥538 AU/mL	≥0.8 U/mL	COI ≥1.0
Non-reactive (N)	S/CO < 1.0	<3SD mean of control	<10 U/mL	<5,000 AU/mL	<0.8 U/mL	COI <1.0
Reactive (N)	S/CO ≥ 1.0	≥3SD mean of control	>10 U/mL to ≤100 U/mLa	≥5,000 AU/mL	≥0.8 U/mL	COI ≥1.0
Reported sensitivity %b,c95% CI (LL, UL)			0–7 days: 81.3 (64.7, 91.1)8–14 days: 96.3 (81.7, 99.3)≥15 days: 93.9 (90.1, 96.2)	-	0–6 days: 88.67–13 days: 85.514–20 days: 89.221–27 days: 98.328–34 days: 100>35 days: 1000–13 days: 86.1 (80.3, 90.7)Overall: 98.8 (98.1, 99.3)	0–6 days: 60.2 (52.3, 67.8)7–13 days: 85.3 (78.6, 90.6)≥14 days: 99.5 (97.0, 100)
S	94.4	100.0	-	0–14 days: 84.2 (68.7, 94.0)≥15 days:98.3 (95.1, 99.6)	-	-
RBD	89.0	100.0	-	0–14 days: 71.1 (54.1, 84.6)≥15 days: 98.3 (95.1, 99.6)	-	-
N	78.6	100.0	-	0–14 days: 71.1 (54.1, 84.6)≥15 days: 93.8 (89.1, 96.8)	-	-
Reported specificity %b95% CI (LL, UL)			99.9 (99.64, 99.99)	-	99.97 (99.9, 100)	99.8 (99.7, 99.8)
S	98.9	98.0	-	99.5 (97.2, 100)	-	-
RBD	100.0	99.0	-	98.5 (95.7, 99.7)	-	-
N	99.3	94.0	-	100 (98.2, 100)	-	-
Instrument time	10.5 hours/5,760 samples	5–6 hours/384 samples	4 hours/96 samples	6 hours/96 samples	5 hours/96 samples	5 hours/96 samples

CI3: The Centre for Infection, Immunity, and Inflammation in Ottawa, Canada; Mount Sinai Hospital in Toronto, Canada; NML at JC Wilt: National Microbiology Laboratory at the JC Wilt Infectious Diseases Research Centre in Winnipeg, Canada; NML at CSCHAH: National Microbiology Laboratory at the Canadian Science Centre for Human and Animal Health in Winnipeg, Canada; CLIA: Chemiluminescence immunoassay; MFI: Multiplex flow immunoassay; ECLIA: Electrochemiluminescence immunoassay; S: Spike; S1: Spike S1 subunit; RBD: Receptor binding domain; N: Nucleocapsid; RR: Relative ratio; S/CO: Signal to cutoff ratio: COI: Cutoff index; SD: Standard deviation; 95% CI (LL, UL): 95% confidence interval (lower limit, upper limit).

Values >100 U/mL also considered reactive, but above the upper limit of the measuring interval.

Clinical performance reported in serum/plasma.

Clinical performance reported for days between symptom onset and sample collection (Bio-Rad Laboratories SARS-CoV-2 IgG) or days after diagnosis with positive PCR test (both Elecsys and V-PLEX SARS-CoV-2 assays).

EUROIMMUN anti-SARS-CoV-2 ELISA IgG

All DBS specimens were tested using the EUROIMMUN anti-SARS-CoV-2 ELISA IgG assay (EUROIMMUN, Lübeck, Germany) according to the manufacturer’s instructions for DBS prior to distribution to each testing site. DBS specimens were punched (1 × 6 mm) using a semi-automated BSD600 Ascent puncher (BSD Robotics, Brisbane, Australia) into 2 mL 96-well polypropylene plates (ThermoFisher Scientific, Waltham, MA). DBS punches were eluted in 500 μL of Sample Buffer (EUROIMMUN) overnight at 4 °C without agitation. Afterwards, plates were incubated at room temperature for 30 min with agitation (400 RPM) and 100 μL of DBS eluate was transferred directly into microplate wells coated with SARS-CoV-2 recombinant structural protein (S1 domain; EUROIMMUN). This plate was shaken at 400 rpm for 1 h at 37 °C during the first incubation step. Results were evaluated by calculating the ratio of the extinction of the patient sample over the extinction of the calibrator. A ratio of <0.8, ≥0.8 to ≤1.1, and ≥1.1 was interpreted as negative, grey zone, and positive respectively. The anti-SARS-CoV-2 ELISA IgG assay from EUROIMMUN was considered as the reference test since it is the only assay approved by Health Canada for use with serum which also has a manufacturer developed protocol for use with DBS (CE marked).

In-house University of Ottawa protocol

DBS specimens were tested using an in-house chemiluminescent direct enzyme-linked immunosorbent assay (CLIA) within the bio-containment CL2+ facility at the Department of Biochemistry, Microbiology, and Immunology (University of Ottawa, Ottawa, Canada) according to laboratory developed protocols validated during earlier work [23]. DBS specimens were punched (1 × 3.2 mm) manually or with semi-automated punchers (PerkinElmer DBS puncher, PerkinElmer, Woodbridge, Canada; BSD6000 Ascent puncher, BSD Robotics) into 2 mL 96-well propylene plates (ThermoFisher Scientific). DBS punches were eluted in 100 μL of PBS containing 1% Triton X-100 (PBST) for a minimum of 4 h up to a maximum of 16 h with agitation 400 RPM at room temperature. Afterwards, DBS eluates were centrifuged at 216 x g for 2 min and diluted with PBST containing 2% milk using a 1:2 ratio. A signal-to-cutoff ratio (S/CO) < 1 and ≥1 was interpreted as negative and positive respectively.

In-house Mount Sinai Hospital protocol

DBS specimens were tested using an in-house CLIA at the Lunenfeld-Tanenbaum Research Institute (Mount Sinai Hospital, Sinai Health, Toronto, Canada) according to laboratory developed protocols validated during earlier work [23]. DBS specimens were punched (1 × 6 mm) manually or with a semi-automated BSD6000 Ascent puncher (BSD Robotics) into 2 mL 96-well polypropylene plates (ThermoFisher Scientific). DBS punches were eluted in 160 μL of PBS containing 0.1% Tween 20 and 1% Triton X-100 for a minimum of 4 h with agitation (150 rpm). Afterwards, DBS eluates were centrifuged at 1,000 x g for 30 s, transferred to new 2 mL 96-well plates, and diluted in 1.3% Blocker BLOTTO buffer (ThermoFisher Scientific) using a 1:4 ratio. Results were interpreted as follows. Spike: a relative ratio (RR) < 0.46 and ≥0.46 was interpreted as negative and positive respectively; RBD: a RR < 0.27 and ≥0.27 was interpreted as negative and positive respectively; nucleocapsid: a RR < 0.74 and ≥0.74 was interpreted as negative and positive respectively. Cutoffs represent three standard deviations from the mean of the log10 distribution of the RR from the negative controls for each antigen [23].

Bio-Rad BioPlex 2200 SARS-CoV-2 IgG

DBS specimens were tested using the BioPlex 2200 SARS-CoV-2 IgG assay (Bio-Rad Laboratories, Hercules, CA) at the NLHRS (Public Health Agency of Canada, Winnipeg, Canada) according to the manufacturer’s instructions coupled with BioPlex 2200 SARS-CoV-2 IgG Calibrators diluted in BioPlex 2200 Wash Buffer (Bio-Rad) using a 1:8 ratio. DBS specimens were punched (1 × 6 mm) using a semi-automated BSD600 Ascent puncher (BSD Robotics) into 400 μL 96-well polypropylene plates (ThermoFisher Scientific). DBS punches were eluted in 130 μL of DPBS containing 0.5% BSA and 0.05% Tween 20 overnight at 4 °C without agitation. Afterwards, plates were incubated at room temperature for 30 min with agitation (400 RPM) and 100 μL of DBS eluate was transferred into 2 mL microtubes for direct loading onto the BioPlex 2200 system (Bio-Rad). A result of <10 U/mL and ≥10 U/mL was interpreted as negative and positive respectively. DBS punching and elution protocols were validated using an earlier DBS panel [9] and took into consideration minimal input volumes for the BioPlex 2200 system.

MSD V-PLEX SARS-CoV-2 panel 2 IgG

DBS specimens were tested using the V-PLEX SARS-CoV-2 Panel 2 IgG assay (MSD, Rockville, MD) at the NLHRS (Public Health Agency of Canada, Winnipeg, Canada) according to the manufacturer’s instructions. DBS specimens were punched (1 × 6 mm) using a semi-automated BSD600 Ascent puncher (BSD Robotics) into 400 μL 96-well polypropylene plates (ThermoFisher Scientific). DBS punches were eluted in 100 μL of DPBS containing 0.5% BSA and Tween 20 overnight at 4 °C without agitation. Afterwards, plates were incubated at room temperature for 30 min with agitation (400 RPM) and 25 μL of DBS eluate was transferred directly into MULTI-SPOT 96-well plates spotted with SARS-CoV-2 antigens (MSD). Results were interpreted as follows unless stated otherwise. Spike: a result of <4.6 AU/mL and ≥4.6 AU/mL was interpreted as negative and positive respectively; RBD: <2.1 AU/mL and ≥2.1 AU/mL was interpreted as negative and positive respectively; Nucleocapsid: <9.7 AU/mL and ≥9.7 AU/mL was interpreted as negative and positive respectively. Cutoffs as well as DBS punching and elution protocols were established using an earlier DBS panel [9].

Roche Elecsys Anti-SARS-CoV-2

DBS specimens were tested using the Elecsys Anti-SARS-CoV-2 (nucleocapsid) and Elecsys Anti-SARS-CoV-2 S (RBD) assays (Roche Diagnostics, Basel, Switzerland) at the National Microbiology Laboratory (NML; Public Health Agency of Canada, Winnipeg, Canada) according to the manufacturer’s instructions. DBS samples were punched (4 × 6 mm) using a semi-automated BSD600 Ascent puncher (BSD Robotics) into 2 mL 96-well polypropylene plates (ThermoFisher Scientific). DBS punches were eluted in 370 μL of DPBS containing 0.5% BSA and 0.05% Tween 20 overnight at 4 °C with agitation (400 RPM). Afterwards, plates were incubated at room temperature for 30 min with agitation (400 RPM) and 250 μL of DBS eluate was transferred into 2 mL microtubes for direct loading onto a cobas e 411 analyser (Roche Diagnostics). A result of <0.8 U/mL and ≥0.8 U/mL was interpreted as negative and positive respectively with the Elecsys Anti-SARS-CoV-2 S. A cutoff index <1.0 and ≥1.0 was interpreted as negative and positive respectively with the Elecsys Anti-SARS-CoV-2 assay. DBS punching and elution protocols were validated using an earlier DBS panel [9].

Statistical analysis

Continuous data were summarized using the median and interquartile range (IQR), while categorical data were presented using exact numbers and proportions. Performance expressed in terms of sensitivity, specificity, NPV, and PPV was computed using the anti-SARS-CoV-2 ELISA IgG assay (EUROIMMUN) as the reference test. PPV and NPV were also computed using a prevalence ranging from 0.5% to 50% as follows: PPV = (sensitivity x prevalence)/[(sensitivity x prevalence) + ((1 – specificity) x (1 – prevalence))]; NPV = (specificity x (1 – prevalence))/[(specificity x (1 – prevalence)) + ((1 – sensitivity) x prevalence)] [38].

The strength of agreement between the index test and the reference test was quantified with kappa coefficients (https://www.graphpad.com/quickcalcs/kappa1/?K=2) and interpreted as follows: <0 = no agreement, 0–0.20 = slight agreement, 0.21–0.40 = fair agreement, 0.41–0.60 = moderate agreement, 0.61–0.80 = substantial agreement, and 0.81–1.00 = almost perfect agreement [39]. Confidence intervals for kappa coefficients were computed according to Fleiss [40].

Receiver operating characteristic (ROC) curves were computed using the Wilson/Brown method [41]. Test performance was quantified according to the area under the ROC curve (AUC) and interpreted as follows: <0.5 = not useful, 0.5–0.6 = bad, 0.6–0.7 = sufficient, 0.7–0.8 = good, 0.8–0.9 = very good, and 0.9–1.0 = excellent [41, 42]. Youden’s J statistic was computed as follows: J = sensitivity + specificity – 1. Prism version 9.0.0 (GraphPad Software, San Diego, CA) was used for all data analysis and visualisation.

Data availability

All data generated or analysed during this study are included in this published article (and its Supplementary Information files).

Results

COVID-19 patient population

Demographic and clinical characteristics of the COVID-19 patient population are provided in Table 2 . The patient population included 21 outpatients (21.6%), 51 inpatients (52.6%), and 25 patients admitted to the intensive care unit (ICU; 25.8%). Patient age ranged from 24 to 97 years (median = 62) and 48 (49.5%) were female. Days between symptom onset and sample collection ranged from 1 to 188 days (median = 21). According to the EUROIMMUN anti-SARS-CoV-2 IgG assay (Figure 1 ), antibody readings were higher in the ICU patient group (median S/CO = 6.1; IQR = 4.8, 7.5) compared to the outpatient group (median S/CO = 4.6; IQR = 2.5, 5.6) and inpatient group (median S/CO = 4.8; IQR = 4.8, 7.5). Antibody readings were trending higher in DBS collected 15–28 days between symptom onset and sample collection (median S/CO = 5.6; IQR = 3.9, 6.7) and lower in DBS collected ≤14 days (median S/CO = 4.7; IQR = 2.8, 6.0) or >28 days (median S/CO = 5.1; IQR = 2.6, 6.5) between symptom onset and sample collection however, these observations were not statistically significant (Figure 1). Furthermore, antibody readings did not vary significantly according to sex or age (Figure 1).

Table 2

Demographic and clinical characteristics of COVID-19 patients.

Variable	≤14 daysa (n = 26)	15–28 days (n = 36)	>28 days (n = 35)	All patients (n = 97)
Median age (IQR)	62.0 (57.0, 70.5)	70.0 (58.0, 77.8)	55.0 (39.0, 67.0)	62.0 (50.0, 75.0)
Female sex (%)	9 (34.6)	22 (61.1%)	18 (51.4)	48 (49.5)
Outpatient (%)	-	-	21 (60.0)	21 (21.6)
Inpatient (%)	9 (34.6)	26 (72.2)	6 (17.1)	51 (52.6)
ICU (%)	17 (65.4)	10 (27.8)	8 (22.9)	25 (25.8)
Median S/CO (IQR)b	4.7 (2.8, 6.0)	5.6 (3.9, 6.7)	5.1 (2.6, 6.5)	5.1 (3.3, 6.3)

ICU: Intensive care unit; IQR: Interquartile range; S/CO: Signal to cutoff ratio.

Days between symptom onset and sample collection.

EUROIMMUN anti-SARS-CoV-2 IgG assay.

Figure 1

EUROIMMUN anti-SARS-CoV-2 ELISA IgG signal to cutoff ratios (S/CO) from DBS collected from COVID-19 patients. A) Highest level of care; B) Days between symptom onset and sample collection; C) Sex; D) Age categories. Bars represent the median and interquartile range (IQR). Test cutoffs are represented by dashed lines. Significant differences between S/CO means are represented by ∗∗ (p < 0.002) and ∗∗∗∗ (p < 0.0001) while non-significant differences are represented by ns. S/CO values are log2 transformed to aid with visualisation.

COVID-19 patient DBS versus pre-COVID-19 DBS

SARS-CoV-2 antibody readings from the two in-house and three commercial multiplex assays are summarized in Figures 2 and S1 Table. Both in-house assays could clearly distinguish DBS from COVID-19 patients and those collected pre-COVID-19 regardless of antigen, days between symptom onset and sample collection, or level of care (Figure 2 ). Antibody readings were generally highest in the ICU patient group and DBS collected 15–28 days between symptom onset and sample collection (S1 Table). Among the commercial assays, the BioPlex and V-PLEX assays clearly differentiated DBS from COVID-19 patients and those collected pre-COVID-19, while the separation was not as clear with the Elecsys assay (Figure 2).

Figure 2

SARS-CoV-2 antibody readings in DBS from convalescent COVID-19 patients and DBS collected pre-COVID-19. A) University of Ottawa protocol; B) Mount Sinai Hospital protocol; C) Bio-Rad BioPlex 2200 SARS-CoV-2 IgG; D) MSD V-PLEX SARS-CoV-2 Panel IgG; E) Roche Elecsys Anti-SARS-CoV-2. Bars represent the median and interquartile range. Test cutoffs are represented by dashed lines. Antibody readings are log2 transformed to aid with visualisation.

Sensitivity and specificity

Sensitivity and specificity were computed using cutoffs established elsewhere or those recommended by the manufacturers (Tables 3, S2 Table). Overall sensitivity of the assays ranged from 96.9 to 99.0%, 82.5–97.9%, and 55.7–97.9% for the spike, RBD, and nucleocapsid antigens respectively. The highest sensitivities were achieved in DBS samples taken from severely ill patients (outpatients or ICU patients) and samples taken 15–28 or >28 days between symptom onset and sample collection (S2 Table). Most false negative results were reported in DBS collected ≤14 days post symptom onset. Overall specificity of the assays ranged from 92.2 to 100.0%, 93.0–100.0%, and 92.2–100.0% for the spike, RBD, and nucleocapsid antigens respectively (Table 3 ). All assays were in “almost perfect” agreement with the reference test based on kappa coefficients except the Elecsys assay which only had “moderate” agreement (nucleocapsid, kappa = 0.547; 95% CI = 0.441, 0.653).

Table 3

Summary performance statistics in dried blood specimens.

Spike	University of Ottawa protocol	Mount Sinai Hospital protocol	Bio-Rad BioPlex 2200 SARS-CoV-2 IgG	MSD V-Plex SARS-CoV-2 Panel 2 IgGa	Roche Elecsys Anti-SARS-CoV-2
Sensitivity (n/N) % (95% CI)	95/9797.9 (92.8, 99.6)	96/9799.0 (94.4, 99.9)	94/9796.9 (91.3, 99.2)	95/9797.9 (92.8, 99.6)
Specificity (n/N) % (95% CI)	83/9092.2 (84.8, 96.2)	86/9095.6 (89.1, 98.3)	90/90100.0 (95.9, 100.0)	85/8698.8 (93.7, 99.9)
PPV (n/N) % (95% CI)	7/9093.1 (86.5, 96.6)	4/9096.0 (90.2, 98.4)	0/90100.0 (96.1, 100.0)	1/8699.0 (94.3, 99.9)
NPV (n/N) % (95% CI)	2/9797.6 (91.8, 99.6)	1/9798.9 (93.8, 99.9)	3/9796.8 (90.9, 99.1)	2/9797.7 (92.0, 99.6)
Kappa (95% CI)	0.903 (0.841, 0.965)	0.946 (0.900, 0.993)	0.968 (0.932, 1.000)	0.967 (0.930, 1.000)
RBD	University of Ottawa protocol	Mount Sinai Hospital protocol	Bio-Rad BioPlex 2200 SARS-CoV-2 IgG	MSD V-Plex SARS-CoV-2 Panel 2 IgGa	Roche Elecsys Anti-SARS-CoV-2
Sensitivity (n/N) % (95% CI)	95/9797.9 (92.8, 99.6)	95/9797.9 (92.8, 99.6)	92/9794.8 (88.5, 97.8)	95/9797.9 (92.8, 99.6)	80/9782.5 (73.7, 88.8)
Specificity (n/N) % (95% CI)	87/9096.7 (90.7, 99.1)	87/9096.7 (90.7, 99.1)	89/9098.9 (94.0, 99.9)	80/8693.0 (85.6, 96.8)	90/90100.0 (95.9, 100.0)
PPV (n/N) % (95% CI)	3/9096.9 (91.4, 99.2)	3/9096.9 (91.4, 99.2)	1/9098.9 (94.2, 99.9)	6/8694.1 (87.6, 97.2)	0/90100.0 (95.4, 100.0)
NPV (n/N) % (95% CI)	2/9797.8 (92.2, 99.6)	2/9797.8 (92.2, 99.6)	5/9794.7 (88.1, 97.7)	2/9797.6 (91.5, 99.6)	17/9784.1 (76.0, 89.8)
Kappa (95% CI)	0.946 (0.900, 0.993)	0.946 (0.900, 0.993)	0.936 (0.885, 0.986)	0.912 (0.852, 0.972)	0.819 (0.739, 0.900)
Nucleocapsid	University of Ottawa protocol	Mount Sinai Hospital protocol	Bio-Rad BioPlex 2200 SARS-CoV-2 IgG	MSD V-Plex SARS-CoV-2 Panel 2 IgGa	Roche Elecsys Anti-SARS-CoV-2
Sensitivity (n/N) % (95% CI)	95/9797.9 (92.8, 99.6)	89/9791.8 (84.6, 95.8)	87/9789.7 (82.1, 94.3)	94/9796.9 (91.3, 99.2)	54/9755.7 (45.8, 65.2)
Specificity (n/N) % (95% CI)	83/9092.2 (84.8, 96.2)	87/9096.7 (90.7, 99.1)	90/90100.0 (95.9, 100.0)	84/8697.7 (91.9, 99.6)	90/90100.0 (95.9, 100.0)
PPV (n/N) % (95% CI)	7/9093.1 (86.5, 96.6)	3/9096.7 (90.8, 99.1)	0/90100.0 (95.8, 100.0)	2/8697.9 (92.7, 99.6)	0/90100.0 (93.4, 100.0)
NPV (n/N) % (95% CI)	2/9797.6 (91.8, 99.6)	8/9791.6 (84.3, 95.7)	10/9790.0 (82.6, 94.5)	3/9796.6 (90.3, 99.1)	43/9767.7 (59.3, 75.0)
Kappa (95% CI)	0.903 (0.842, 0.965)	0.882 (0.815, 0.950)	0.893 (0.829, 0.957)	0.945 (0.898, 0.993)	0.547 (0.441, 0.653)
Spike or RBD	University of Ottawa protocol	Mount Sinai Hospital protocol	Bio-Rad BioPlex 2200 SARS-CoV-2 IgG	MSD V-Plex SARS-CoV-2 Panel 2 IgGa	Roche Elecsys Anti-SARS-CoV-2
Sensitivity (n/N) % (95% CI)	95/9797.9 (92.8, 99.6)	96/9799.0 (94.4, 99.9)	94/9796.9 (91.3, 99.2)	95/9797.9 (92.8, 99.6)
Specificity (n/N) % (95% CI)	81/9090.0 (82.1, 94.6)	84/9093.3 (86.2, 96.9)	89/9098.9 (94.0, 99.9)	79/8691.9 (84.1, 96.0)
PPV (n/N) % (95% CI)	9/9091.3 (84.4, 95.4)	6/9094.1 (87.8, 97.3)	1/9098.9 (94.3, 99.9)	7/8693.1 (86.5, 96.6)
NPV (n/N) % (95% CI)	2/9797.6 (91.6, 99.6)	1/9798.8 (93.6, 99.9)	3/9796.7 (90.8, 99.1)	2/9797.5 (91.4, 99.6)
Kappa (95% CI)	0.882 (0.814, 0.949)	0.925 (0.870, 0.979)	0.957 (0.916, 0.999)	0.901 (0.838, 0.964)

Four pre-COVID-19 DBS omitted from analysis due to insufficient quantity for testing.

Similar observations were made using multiple antigen consensus-based rules to establish overall positivity. Overall sensitivity of the assays ranged from 96.9 to 99.0% if using spike or RBD. Most false negative results were reported in DBS collected ≤14 days post symptom onset. Therefore, sensitivity was typically higher in DBS collected ≥15 days post symptom onset. Overall specificity of the assays ranged from 90.0 to 98.9% if using spike or RBD All assays were in “almost perfect” agreement with the reference test based on kappa coefficients (Table 3).

Predictive values

Predictive values for the two in-house and three commercial multiplex assays with DBS are summarized in Tables 3 and S2 Table. All assays achieved overall positive predictive values (PPV) of >93%, >94%, and >93% with the spike, RBD, and nucleocapsid antigen respectively (Table 3). All assays achieved overall negative predictive values (NPV) of >97%, >84%, and >67% with the spike, RBD, and nucleocapsid antigen respectively (Table 3). PPV and NPV improved for all assays in DBS collected ≥15 days post symptom onset (S2 Table). Similar PPV were observed using multiple antigen consensus-based rules. More specifically, all assays achieved an overall PPV of >91% with spike or RBD antigens above the cutoff (Table 3). All assays achieved an overall NPV of >96% with spike or RBD antigens above the cutoff (Table 3).

Granted prevalence has a significant impact on PPV and NPV, predictive values were re-computed using a wide range of prevalence estimates (Figure 3 ). As expected, all assays achieved a wide range of PPV and NPV depending on prevalence and antigen. In general terms, the spike and RBD antigens would result in the lowest proportion of false negatives and false positives in low and high prevalence settings respectively. Likewise, commercial assays would offer better PPV and NPV in low prevalence settings compared to the in-house assays. Using multiple antigen consensus-based rules would perform similarly to using spike or RBD alone (Figure 3).

Figure 3

Positive and negative predictive values according to prevalence. A) University of Ottawa protocol; B) Mount Sinai Hospital protocol; C) BioPlex 2200 SARS-CoV-2 IgG (Bio-Rad); D) V-PLEX SARS-CoV-2 Panel 2 IgG (MSD); E) Elecsys Anti-SARS-CoV-2 (Roche).

ROC curve analysis

According to receiver operating characteristic (ROC) curves (Figure 4 ), all assays demonstrated excellent performance (area under the curve >95%) regardless of antigen, days post symptom onset, and highest level of care suggesting that performance could be improved by adjusting cutoffs (Figure 4).

Figure 4

Receiver operating characteristic (ROC) curves. ROC curves are presented for n = 90 SARS-CoV-2 antibody negative DBS specimens and n = 97 SARS-CoV-2 antibody positive DBS specimens. Four SARS-CoV-2 antibody negative DBS specimens were removed from the V-PLEX SARS-CoV-2 Panel 2 IgG (MSD) analysis due to insufficient quantity for testing. A) University of Ottawa protocol; B) Mount Sinai Hospital protocol; C) BioPlex 2200 SARS-CoV-2 IgG (Bio-Rad); D) V-PLEX SARS-CoV-2 Panel 2 IgG (MSD); E) Elecsys Anti-SARS-CoV-2 (Roche).

Discussion

Our study describes the performance of three commercial and two in-house multiplex assays for the qualitative analysis of SARS-CoV-2 antibodies in a well pedigreed panel of DBS specimens. These assays were evaluated using a panel of 187 unique DBS specimens collected from 97 convalescent COVID-19 patients and 90 individuals undergoing routine STBBI testing pre-COVID-19. Overall, higher antibody responses were observed in ICU patients and in DBS collected >14 days between symptom onset and sample collection which is consistent with the current literature. Higher antibody titers are typically reported in severely ill patients compared to patients exhibiting milder forms of COVID-19 [43, 44, 45, 46]. Seroconversion typically occurs within 2 weeks post-infection [46]. Taken altogether, we expected some heterogeneity in performance according to disease severity and the number of days between symptom onset and sample collection. Future assessments should incorporate DBS specimens collected during multiple waves of the pandemic since VOCs may elicit varying levels of immune responses compared to wild-type SARS-CoV-2 [47,48], especially the omicron variant which has been associated with milder disease severity [49].

Overall sensitivities ranged from 96.9% to 99.0%, 82.5%–97.9%, and 55.7%–97.9% for the spike, RBD, and nucleocapsid antigens respectively. A similar observation was made using a consensus-based rule (spike or RBD) to establish overall seropositivity. While varying levels of performance are expected based on the antigen of interest [28, 29, 30], our ROC curve analysis suggests that better performance could be achieved by adjusting laboratory-developed or manufacturer recommended cutoffs for DBS specimens [5, 21, 50, 51]. These observations are comparable to performance estimates reported by others with commercial assays like the Roche Elecsys [15, 21]. However, we are the first group to report performance data on the BioPlex 2200 SARS-CoV-2 IgG with DBS specimens to the best of our knowledge. We also clearly demonstrate that laboratory developed assays have the potential to perform as well as or better than commercial assays [23, 52]. Future inter-laboratory comparisons will be conducted with DBS collected from the latest iteration of the Canadian COVID-19 antibody and Health Survey (April 1 2022 to August 31 2022; https://www.statcan.gc.ca/en/survey/household/5339).

Although adjusting cutoffs might be helpful in improving performance, this approach may be inadequate in population-wide serosurveys where SARS-CoV-2 infected individuals may be mostly asymptomatic. Multi-antigen approaches have been proposed as a better alternative especially since they allow to distinguish between vaccine-induced immunity and infection acquired immunity [28]. We did not observe marked improvements in assay performance using a two-antigen strategy (spike or RBD) compared to using the spike or RBD antigen alone, contrary to other studies that have shown improvements in performance using a multiplex approach to establish seropositivity [28, 53, 54]. This could be due to our choice of patient population which consisted of convalescent COVID-19 patients with elevated antibody titers for spike, RBD, and nucleocapsid.

In low prevalence settings, a high PPV is critical and tests with high specificity (≥99%) should be chosen to limit the number of false positives. This may no longer be a significant issue for the spike and RBD antigens due to high vaccine uptake [55], but it is still an important consideration for the nucleocapsid antigen to allow the distinction between infection- and vaccine-induced antibody responses [34]. In this study, overall specificity ranged from 92.2% to 100% for the nucleocapsid antigen. The highest specificities (100%) were observed with the BioPlex and Elecsys assays while the lowest specificities (<97%) were observed with the in-house assays resulting in lower PPV in low prevalence settings (<5%). However, this could be improved by adjusting cutoffs as mentioned above.

At the time of this writing, information regarding SARS-CoV-2 antibody seroprevalence in Canadians was lacking but, overall seroprevalence was estimated at 3.6% (95% CI = 2.6,4.2) during the first wave of the pandemic [56, 57]. The BioPlex 2200 SARS-CoV-2 IgG, MSD V-PLEX SARS-CoV-2 Panel 2 IgG, or Roche Elecsys are therefore mostly like better suited for epidemiological studies in low prevalence settings compared to the in-house assays. However, further validation in asymptomatic populations and longer delays between symptom onset and DBS collection (>6 months) will be required to substantiate our findings especially if SARS-CoV-2 becomes endemic [58]. Other laboratories seeking to implement their own serosurveillance may benefit from adopting commercial assays versus in-house assays due to their performance and lower technical barriers to implementation. For example, the BioPlex 2200 system is a fully automated system capable of delivering results for all three antigens simultaneously while the in-house assays require customized liquid handling platforms where three singleton assays are necessary to deliver results for all three antigens.

Our study revealed acceptable performance of three multiplex SARS-CoV-2 antibody assays: BioPlex 2200 SARS-CoV-2 IgG, MSD V-PLEX SARS-CoV-2 Panel 2 IgG, or Roche Elecsys. Several false positives were observed with the in-house assays resulting in low PPV in low prevalence settings, but it would be important to note that the University of Ottawa and Mount Sinai Hospital enforce a requirement that two or more of the individual assays pass their cutoffs for a sample to be considered seropositive overall [23]. In this study, our findings suggest that performance could improve by simply adjusting cutoffs for DBS specimens. Adjusting manufacturer’s cutoff values will have to be undertaken independently by laboratories and is likely not generalizable. While others have reported greater performance using a multiplexed approach versus single-antigens, we did not make this observation. This is likely due to the patient population that was chosen to make up the DBS panel. Our findings will have to be substantiated with further validation in asymptomatic populations and DBS specimens collected during the latest wave of the pandemic to confirm assay performance. Nonetheless, our study demonstrates that DBS specimens coupled with several different assays are useful tools for large-scale application of epidemiological COVID-19 studies.

Declarations

Author contribution statement

François Cholette: Conceived and designed the experiments; Analyzed and interpreted the data; Wrote the paper.

Christine Mesa, Philip Lacap, Marc-André Langlois, Karen Colwill, Anne-Claude Gingras, Carla Osiowy and John Kim: Conceived and designed the experiments; Analyzed and interpreted the data.

Michael Drebot: Conceived and designed the experiments.

Rissa Fabia, Angela Harris, Hannah Ellis, Karla Cachero, Lukas Schroeder, Elizabeth Giles and Jacqueline Day: Performed the experiments.

Corey Arnold, Yannick Galipeau: Performed the experiments; Analyzed and interpreted the data.

Allison McGeer, Yves Durocher, Catherine Hankins, Bruce Mazer: Contributed reagents, materials, analysis tools or data.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Data availability statement

Data included in article/supplementary material/referenced in article.

Declaration of interests statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.