Longitudinal evaluation of laboratory-based serological assays for SARS-CoV-2 antibody detection.

Serological assays for SARS-CoV-2 infection are now widely available for use in diagnostic laboratories. Limited data are available on the performance characteristics in different settings, and at time periods remote from the initial infection. Validation of the Abbott (Architect SARS-CoV-2 IgG), DiaSorin (Liaison SARS-CoV-2 S1/S2 IgG) and Roche (Cobas Elecsys Anti-SARS-CoV-2) assays was undertaken utilising 217 serum samples from 131 participants up to 7 months following COVID-19 infection. The Abbott and DiaSorin assays were implemented into routine laboratory workflow, with outcomes reported for 2764 clinical specimens. Sensitivity and specificity were concordant with the range reported by the manufacturers for all assays. Sensitivity across the convalescent period was highest for the Roche at 95.2-100% (95% CI 81.0-100%), then the DiaSorin at 88.1-100% (95% CI 76.0-100%), followed by the Abbott 68.2-100% (95% CI 53.4-100%). Sensitivity of the Abbott assay fell from approximately 5 months; on this assay paired serum samples for 45 participants showed a significant drop in the signal-to-cut-off ratio and 10 sero-reversion events. When used in clinical practice, all samples testing positive by both DiaSorin and Abbott assays were confirmed as true positive results. In this low prevalence setting, despite high laboratory specificity, the positive predictive value of a single positive assay was low. Comprehensive validation of serological assays is necessary to determine the optimal assay for each diagnostic setting. In this low prevalence setting we found implementation of two assays with different antibody targets maximised sensitivity and specificity, with confirmatory testing necessary for any sample which was positive in only one assay.

Introduction

Following the emergence and spread of the novel coronavirus 2019, SARS-CoV-2, a range of diagnostic assays have been developed. Real-time PCR (RT-PCR) testing for SARS-CoV-2 RNA is the gold standard for diagnosis of acute infection, with serological testing reserved for specific scenarios (e.g., epidemiological surveillance; resolving indeterminate RT-PCR findings). A range of commercial serological assays are now available, with the majority targeting either the nucleocapsid antigen (N), the spike protein (S), or a component of the spike protein such as the receptor binding domain (RBD). The RBD is the distal area of the spike protein that binds with host angiotensin-converting enzyme 2 receptors (ACE2-R) to facilitate viral cell entry. Antibodies which recognise this region are more likely to be neutralising, which has been supported by good correlation between S1 or RBD targeted antibodies and neutralisation assays. , To date, however, the optimal target antigen for commercial diagnostic assays has not yet been determined, and may vary depending on: (i) whether the purpose of testing is to determine exposure to the virus (non-neutralising or neutralising antibodies) or putative immunity (neutralising antibodies), (ii) at what time point sampling occurs following infection, and (iii) the vaccination status of the patient. Understanding the performance characteristics of the assays in a number of different settings, and with different clinical cohorts, is important in designing testing algorithms relevant for the local population and in providing accurate clinical interpretation of results for clinicians.

The average time to seroconversion following SARS-CoV-2 symptom onset is 10–15 days, with the majority of patients seroconverting by 21 days. , Antibody titres may correlate with severity of illness, with lower titres observed in patients who remain asymptomatic throughout their infection. , The longevity of the antibody response to SARS-CoV-2 is unknown, although based on studies of the related seasonal beta-coronaviruses as well as SARS-CoV-1 and MERS-CoV, is expected to last between 1 and 5 years. , , Antibody titres against SARS-CoV-2 N protein have been shown to rise earlier than those directed against the S protein following infection, and recent evidence suggests that the half-life of anti-N antibodies may be shorter. ,

At the time of writing Australia has successfully controlled the COVID-19 pandemic; as of 24 November 2020, COVID-19 cases are sporadic (due to international importation in quarantined travellers) or associated with small community clusters. The state of Victoria has experienced the largest number of COVID-19 cases, reporting 73% of the 27,835 cases confirmed in Australia since the first case on 26 January 2020. , Despite two ‘waves’ of infections, first in March to April and then again between June to September, the period prevalence of COVID-19 for Victoria (∼0.4%) remains low compared to international reports. Longitudinal sampling of RT-PCR confirmed COVID-19 cases in Victoria therefore likely reflects the dynamics of SARS-CoV-2 antibody generation following a single exposure to the virus. To date, there are few published data on the longitudinal performance characteristics of serological assays for COVID-19. Accordingly, we undertook a laboratory validation of three commercial assays, the Architect SARS-CoV-2 IgG (Abbott, USA), Liaison SARS-CoV-2 S1/S2 IgG (DiaSorin, Italy) and Cobas Elecsys Anti-SARS-CoV-2 (Roche, Switzerland), using sera from RT-PCR SARS-CoV-2 positive cases. Sera were further characterised by a surrogate virus neutralisation assay (sVNT) or an in-house microneutralisation assay. Longitudinal performance post-infection was determined by serial testing of a cohort of individuals infected in wave one of the Victorian outbreaks. Further, we also describe the ‘real-world’ clinical performance of two commercial assays in two cohorts of healthcare workers.

Methods

Ethics approval

Ethical approval for this project was obtained from the Melbourne Health Human Research Ethics Committee (RMH HREC QA2020052) and Monash Health (QA/64587).

Laboratory validation sample panels

A standardised serum panel was used to test all three assays, with positive sera further characterised by sVNT, apart from one sample for which there was insufficient volume (Supplementary Table 1, Appendix A). The panel consisted of sera from: (i) 131 SARS-CoV-2 RT- PCR positive patients; (ii) an age-stratified cohort of 200 pre-pandemic samples; and (iii) 31 potentially cross-reactive sera collected from patients with other acute infections collected before December 2019 (Supplementary Dataset, Appendix A). Each patient contributed no more than a single sample per time interval, with 69 patients contributing single bleeds and 62 patients contributing serial samples. Paired SARS-CoV-2 antibody titres for a cohort of 45 patients who contributed both early (day 15–90 post-symptom onset) and late (day 121–210 post-symptom onset) convalescent bleeds were examined for significant change.

Commercial serological assays

Three commercially available assays were assessed: the Abbott Architect SARS-CoV-2 IgG and the DiaSorin Liaison SARS-CoV-2 S1/S2 IgG which both detect IgG antibodies, and the Roche Cobas Elecsys Anti-SARS-CoV-2 assay which detects total antibodies. Results for the three assays were compared to the GenScript SARS-CoV-2 Surrogate Virus Neutralisation Test (sVNT) total antibody assay, one of the confirmatory assays employed by the reference laboratory [Victorian Infectious Diseases Reference Laboratory (VIDRL)].

Both the Abbott and Roche assays detect antibodies targeted against the nucleocapsid (N) protein, while the DiaSorin assay detects antibodies against the spike protein (S1/S2 subunits) (Supplementary Table 2, Appendix A). Testing was undertaken on the Abbott Architect, the DiaSorin Liaison and the Roche Cobas platforms, as per the manufacturer's instructions for use (IFU).14, 15, 16 For the purposes of this study, semi-quantitative results are reported as signal-to-cut-off ratios (s/co), arbitrary units per mL (AU/mL), and as a cut-off index (coi), for the Abbott, DiaSorin, and Roche assays, respectively. The DiaSorin assay incorporates an equivocal zone; in this current study equivocal results were considered positive. Following our initial testing, Abbott released a Product Information Letter on 27 October (PI1060-2020) updating the IFU to include an optional editable greyzone which laboratories could choose to set between 0.49 and <1.40 index. Final results for the Abbott are reported without, and then with inclusion of this greyzone (0.49–<1.40). In clinical practice, assay results should be reported qualitatively according to the respective IFU as: positive or negative (Abbott); positive, equivocal or negative (DiaSorin); or reactive or non-reactive (Roche).

The sVNT is an enzyme-linked immunosorbent assay (ELISA) utilising a horseradish peroxidase indicator. SARS-CoV-2 specific antibodies in test sera inhibit binding of the RBD to fixed ACE2 receptors, and are detected by a reduction in the optical density reading compared to control samples. , Similar to our previous work, we conducted a first round of testing on all samples following the IFU. Subsequently, samples within 10% of the 20% inhibition cut-off value were repeated in duplicate, with the final qualitative result determined by the majority result.

The Abbott and DiaSorin assays were also tested against serum samples obtained from healthcare workers (HCW) at the Royal Melbourne Hospital at different time periods during the Victorian COVID-19 outbreaks. The Roche platform was not available for this purpose. In total, 2764 samples were tested (1972 samples obtained between 27 April 2020 and 10 May 2020, and 792 samples obtained between 24 August 2020 and 29 October 2020). Positive or equivocal results from the period April–May were confirmed by microneutralisation (n=48). Positive or equivocal results from August–October were confirmed at the reference laboratory (VIDRL), according to their in-house confirmatory testing algorithm which involved screening with three ELISA assays: Euroimmun spike IgA and IgG (Euroimmun, Germany), and the Wantai total antibody assay (Beijing Wantai Biological Pharmacy Enterprise, China), with confirmation by sVNT (n=64).

All samples were tested by trained scientists at either the Royal Melbourne Hospital (RMH) microbiology laboratory or at VIDRL, and followed the IFU.

Microneutralisation testing

Microneutralisation was performed using a previously described in-house method at the University of Melbourne.20, 21, 22, 23 Samples were tested in duplicate, with a third test undertaken if titres were greater than two dilutions apart. The geometric mean of the two titres closest in value is then taken as the final titre result; titres 40 or above were considered positive.

RT-PCR

SARS-CoV-2 RNA was detected by at least two of three assays, specifically the Coronavirus Typing assay (AusDiagnostics, Australia), Respiratory Pathogens 12-well assay (AusDiagnostics), the Xpert Xpress SARS-CoV-2 (Cepheid, USA) or an in-house real-time assay at the reference laboratory, using previously published primers.24, 25, 26 All initial positive results were confirmed by re-extraction from the primary sample and amplification in at least one of the remaining three assays.

Statistical analysis

All analyses were conducted using Graphpad Prism (version 9.0; GraphPad Software, USA). Binomial 95% confidence intervals (CI) were calculated for all proportions. Differences in non-normally distributed continuous data were calculated using the Kruskal–Wallis test. A multiple linear regression model (least squares method) considering age, gender, admission status and time of collection following symptom onset was used to assess for significant association of independent variables to the measured antibody level on the different assays. Correlation between assays was determined by the Spearman r calculation for non-parametric data, with values closest to 1.0 indicating closest correlation.

Results

Comparison of commercial assays with RT-PCR

Using RT-PCR proven SARS-CoV-2 infection as the reference standard, the observed sensitivity for all three commercial assays was below that reported by the manufacturers, although was not significantly different (Table 1 ). As expected, all assays had higher sensitivity at >14 days post-symptom onset, with sensitivities ranging from 88.1–100% (95% CI 75–100%) between days 15 and 150 (Table 1). The observed specificity for pre-pandemic and potentially cross-reactive samples was consistent with those reported in the IFUs, ranging from 98.7–100% (95% CI 96.3–100%).

Table 1

Laboratory performance for the Abbott (Architect SARS-CoV-2 IgG), DiaSorin (Liaison SARS-CoV-2 S1/S2 IgG) and Roche (Cobas Elecsys Anti-SARS-CoV-2) SARS-CoV-2 serological assays in an Australian cohort

Characteristic	Abbott	DiaSorin	Roche
Sensitivity by time interval in days from symptom onset (n) [95% CI]
0–7 d (51)	17.6% [9.6, 30.3]23.5% [12.8, 37.5]a	7.8% [3.1, 18.5]	17.6% [9.6, 30.3]
8–14 d (30)	53.3% [36.1, 69.8]70.0% [50.6, 85.3]a	40.0% [24.6, 57.7]b	53.3% [36.1, 69.8]
15–30 d (42)	95.2% [84.2, 99.2]97.6% [87.4, 99.9]a	88.1% [75.0, 94.8]b	92.9% [81.0, 97.5]
31–90 d (39)	92.3% [79.7, 97.3]97.4% [86.5, 99.9]a	92.3% [79.7, 97.3]b	100% [91.0, 100]
121–150 d (11)	100% [74.1, 100]100% [74.1, 100]a	100% [74.1, 100]	100% [74.1, 100]
151–210 d (44)	68.2% [53.4, 80.0]90.9% [78.3, 97.5]a	88.6% [76.0, 95.4]b	97.7% [88.2, 99.9]
Specificity (n) [95% CI]
Cross-reactive specimens (31)	100% [88.8, 100]100% [88.8, 100]a	96.8 [83.3, 99.9]	100% [88.8, 100]
Pre-pandemic controls (200)	100% [98.2, 100]99.5% [97.3, 99.99]a	99.0% [96.4, 99.9]	100% [98.2, 100]
Overall (231)	100% [98.4, 100]99.6% [97.6, 99.99]a	98.7% [96.3, 99.7]	100% [98.4, 100]

CI, confidence interval (Clopper–Pearson).

Results when equivocal zone employed (0.49–<1.40) as per Abbott Diagnostics Product information Letter PI1060-20202, with equivocal results considered positive.

One sample in each time interval (different participants) with an equivocal result on the DiaSorin assay was considered positive.

Of note, sensitivity for one of the assays (Abbott) fell to 68.2% (95% CI 53.4–80.05) for samples collected more than 151 days after symptom onset (Fig. 1 ), with 10 sero-reversion events for patients who initially recorded positive results with early convalescent serum (Fig. 2 ). When results in the equivocal range were considered as positive results, the sensitivity of the Abbott assay with late convalescent serum increased to 90.9% (95% CI 78.3–97.5%) and negated 9 of 10 sero-reversions. The DiaSorin assay demonstrated a lower sensitivity early, at only 40.0% (95% CI 24.6–57.7%) for samples collected in the second week following symptom onset, but remained high at 88.6% (95% CI 76.0–95.4%) after 151 days (Fig.1), with only one sero-reversion observed (Fig. 2). The Roche assay performed well at all time points, with sensitivity of both early and late convalescent serum (day 15–210 following symptom onset) ranging between 92.9–100% (95% CI 80.5–100%) (Fig. 1).

Fig. 1

Semi-quantitative score for serum samples on each assay by time interval using a Log10 scale. (A) Signal-to-cut-off ratio on the Abbott (Architect SARS-CoV-2 IgG); (B) arbitrary units per mL on the DiaSorin (Liaison SARS-CoV-2 S1/S2 IgG); (C) cut-off index on the Roche (Cobas Elecsys Anti-SARS-CoV-2) SARS-CoV-2. Horizontal dashed line indicates cut-off for the assay, with the shaded area on (B) indicative of the equivocal range for the DiaSorin assay. Figures placed above each time interval indicate the qualitative sensitivity (%) for the assay at that time interval, reflecting the reporting of the assay in a diagnostic laboratory. ns, not significant. ∗∗∗∗ p<0.0001 for a difference between early and late convalescent time points for both the Abbott and Roche, as calculated by the Kruskal–Wallis test.

Fig. 2

Comparative semi-quantitative score for serial serum samples from a cohort of 45 patients who had both early convalescent (from 15–90 day post-symptom onset) and late convalescent (from 121–210 days post-symptom onset) serum collection using a Log10 scale. Horizontal dashed line indicates cut-off for the assay, with samples above the line reported as positive/reactive, samples below the line reported as negative/non-reactive and the shaded area in (B) indicative of the equivocal range for the DiaSorin assay. Each circle represents a serum sample, blue pairs indicate antibody scores dropping and teal pairs indicate antibody scores rising between early and late samples; triangles represent samples that have qualitatively sero-reverted from positive on early convalescent testing to negative on late convalescent testing. ns, not significant; ∗∗∗∗ p<0.0001.

Early and late antibody values were compared in 45 patients for whom serial bleeds were available. The geometric mean for antibody scores fell from 5.3 s/co (95% CI 4.3–6.5) to 2.2 s/co (95% CI 1.6–3.1) for the Abbott (p<0.0001), rose from 25.14 coi (95% CI 18.2–34.7) to 67.6 coi (95% CI 46.3–98.7) for the Roche (p<0.0001) and did not significantly change for the DiaSorin assay (Fig. 2). The geometric mean for the sVNT reduced significantly from a percentage inhibition score of 59.86 to 52.54 (p<0.01) (Supplementary Fig. 2, Appendix A). The DiaSorin assay demonstrated the best correlation with sVNT (Spearman r=0.92), compared to the Abbott and Roche assays (Spearman r scores of 0.84 and 0.78, respectively; Supplementary Fig. 1, Appendix A). Further characteristics of the sVNT assay can be found in the Supplementary Data (Appendix A).

Multiple linear regression demonstrated that increasing age was significantly associated with a higher antibody level for the Abbott and sVNT, with p values of <0.01 and <0.05, respectively, but not for the DiaSorin or Roche assays (Supplementary Table 4, Appendix A). Admission to hospital was associated with an increased antibody level for the DiaSorin (p<0.05) only. No difference in antibody level was observed by sex. All assays apart from the Abbott showed an association between antibody levels and increasing time of collection from symptom onset (p<0.0001; Supplementary Table 4, Appendix A).

Clinical testing

Of 1972 HCWs tested in May using the Abbott and Diasorin assays, only one sample (0.05%) was positive in both assays and confirmed as positive by microneutralisation testing. Of 792 HCWs tested between August and October, 41 (5.2%) samples were considered positive, testing positive in either assay and confirmed by the sVNT. In total, 38 samples were positive on the Abbott assay, while 38 were positive and one was equivocal on the DiaSorin assay (Table 2 ).

Table 2

Clinical performance of the Abbott (Architect SARS-CoV-2 IgG) and DiaSorin (Liaison SARS-CoV-2 S1/S2 IgG) serological assays in an Australian cohort

Characteristic	May 2020 community period prevalence ∼0.03%a(n=1972) [95% CI]		August – October 2020 community period prevalence ∼0.4%b(n=792) [95% CI]
	Abbott	DiaSorin	Abbott	DiaSorin
Proportion of cases detected (true positive/composite positivec)	100% (1/1)	100% (1/1)	92.7% (38/41)	95.1% (39/41)d
Positive predictive valuee (true positive/all positive)	8.3% (1/12)	2.7% (1/37)	90.5% (38/42)	67.2% (39/58)

CI, confidence interval (Clopper–Pearson).

Period prevalence January–May 2020 and b period prevalence January–October 2020 determined according to RT-PCR confirmed case count, Victorian Department of Health and Human Services.

Composite positive case numbers = those samples testing positive in either assay and then confirmed either by microneutralisation or by surrogate virus neutralisation.

One equivocal sample counted as positive.

As compared to microneutralisation or testing at the reference laboratory by surrogate virus neutralisation.

Of the 2764 samples tested, 15 of 54 positive samples (27.8%; 95% CI 16.5–41.6%) from the Abbott assay and 55 of 95 positive samples (57.9%; 95% CI 47.3–68.0%) tested on the DiaSorin assay did not confirm and were considered false positives (Fig. 3 ). Median s/co for false positive result for the Abbott assay was 2.5 (range 1.4–7.2) and 19.5 AU/mL for the DiaSorin assay (range 12.4–345). Local PCR testing guidelines did not cover regular testing of the HCW cohort prior to the initial sero-survey in May 2020. Of the 23 HCW with false positive results between August 2020 and October 2020, all HCW had negative PCR results during the 6 months prior to serological sampling (median number of PCR tests 4, range 1–9), while only two were documented contacts of a COVID-19 case in the workplace. As expected, the positive predictive value (PPV) varied at each testing time point relative to community prevalence (Table 2). If the optional greyzone was adopted for the Abbott assay, 39 of 41 confirmed cases would have been detected in the August to October period, with an extra 26 samples in May and 19 samples in this period requiring confirmatory testing. Of note, all sera positive by both the Abbott and DiaSorin assays were later confirmed at the reference laboratory.

Fig. 3

Testing outcomes following the clinical implementation of the Abbott and DiaSorin assays.

Discussion

Here we present results from a longitudinal assessment of SARS-CoV2 antibody detection with three commonly used serological testing platforms, and a recently introduced sVNT assay. Using sera from 131 RT-PCR positive patients, we observed differential sensitivities between the assays at different time points. Most notably, the sensitivity of the Abbott assay decreased significantly when testing late convalescent sera (greater than 4 months from initial infection), compared to the Roche and DiaSorin assays. This is likely due to the different antigens used in each assay, and the antibody isotype detected. The Abbott and Roche assays both employ an N antigen target, compared to the DiaSorin assay which uses an S antigen target. A previous study by Wheatley et al. demonstrated that antibodies targeted against the N antigen have a half-life of 85 days compared to 229 days for those against the S protein, when tested in an in-house assay. Further, a study by Lumley et al. of 522 health care workers also observed a half-life of antibodies detected in the Abbott assay of 85 days. This study reported higher rates of sero-reversion, with 33% at 3 months and 53% at 6 months, compared to the 27% at 7 months described in our current study. Another laboratory validation study utilising serial samples from 97 participants noted the decrease in sensitivity of the Abbott assay relative to the Roche and DiaSorin, with a sensitivity of 71% reported for samples at the earlier time-frame of >81 days post-symptom onset, and a 15% sero-reversion rate. Samples were not collected as late as those reported in our study, and the optional equivocal range not employed. Of note, the Roche assay detects total antibody, whereas the Abbott assay detects only IgG; this may also partly explain the differential longitudinal sensitivity between these two assays.

For the Abbott assay, incorporation of the equivocal range improved the analytical sensitivity of the assay at all time points and particularly for late convalescent samples, albeit at some cost to specificity. While this specificity loss appears marginal in our laboratory validation (100%–99.6%), application in widespread clinical practice may be more problematic and suggests that more results may need to be confirmed with ‘gold standard’ neutralisation testing. Despite high analytical specificity of both the Abbott and DiaSorin assays (100% and 98.7%, respectively), due to the extremely low prevalence of COVID-19 in our setting, there was a high proportion of false positive results in clinical practice (27.8% and 57.9% for the Abbott and DiaSorin, respectively). Consistent with this observation, current local guidelines recommend confirmatory testing for all positive results, and follow up sampling at an appropriate interval. A four-fold rise in titre, neutralisation assay, immunofluorescent assay or microsphere assay are all suggested as confirmatory tests; however, in practice each reference laboratory may develop their own in-house algorithm according to local capacity and capability. Analogous to molecular testing, where specificity is often confirmed by utilising two different specific RNA or DNA targets, we found 100% specificity for clinical samples which had antibodies detected by both the Abbott (N) and DiaSorin (S) targets. For laboratories that have access to more than one platform this may prove a reliable and fast method to confirm positive results, if formal neutralisation or reference laboratory testing is not easily accessible or timely (Fig. 4 ). Our laboratory validation would suggest that samples positive on a single assay should be sent for confirmatory testing, as only a small number are likely to be true positive results. As expected, given the spike protein target, the DiaSorin assay results correlated best with sVNT. The sVNT has previously been shown to correlate strongly with virus neutralisation testing, with a correlation R of 0.8629. As our understanding of what constitutes immunity evolves, this may prove to be an added advantage for assays that detect spike-based antibodies, compared to N antigen-based assays.

Fig. 4

Proposed algorithm for SARS-CoV-2 serological testing in a routine diagnostic laboratory in a low prevalence setting.

Some studies have shown higher antibody titres in patients with more severe COVID-19 disease. , Our analysis did not demonstrate consistent associations between age, gender or disease severity and antibody levels across the different serological assays. While this may be partly due to the heterogeneity of antibody targets and isotypes across the assays, it may also be that findings observed using ‘gold standard’ neutralising assays are not necessarily replicated by diagnostic laboratories using commercial assays.

Limitations of this study are the inclusion of patients with predominantly mild COVID-19 disease; it is possible our results may not be generalisable to all populations with COVID-19. Other limitations include limited information on medical co-morbidities and/or concurrent use of medications (including steroids) which may affect antibody production. Systematic PCR testing was not undertaken for the HCW cohort, some samples determined to be false positive results may reflect past infection with waning antibodies and loss of/or failure to produce neutralising antibodies, and thus be ‘true positive’ results. Notable strengths include the longitudinal nature of the validation, use of a standardised serum panel on all assays, and reporting of the clinical application of two of the assays.

In summary, our data describe the differential performance characteristics of three automated laboratory based assays for sera collected over a 7 month period following COVID-19 infection. Overall, the Roche assay demonstrated the highest sensitivity when considering all time points, with high specificity. Clinical utility of the Abbott assay was highest early post-infection, while that of the DiaSorin assay was higher later post-infection. Testing algorithms should consider a number of factors, including the intended purpose of testing, the time interval at which sera is collected, and expected prevalence in the sample population. In this low prevalence setting we found implementation of two assays with different antibody targets maximised sensitivity and specificity, with confirmatory testing necessary for any sample which was positive in only one assay.