Clinical evaluation of four commercial immunoassays for the detection of antibodies against established SARS-CoV-2 infection.

A comparison of the clinical performance of the Elecsys Anti-SARS-CoV-2, Liaison SARS-CoV-2 S1/S2 IgG, Access SARS-CoV-2 IgG and Vitros Immunodiagnostic Products Anti-SARS-CoV-2 IgG immunoassays for the diagnosis of COVID-19 infection was performed. Patient sera were collected at least 6 weeks following onset of COVID-19 infection symptoms. Negative control specimens were stored specimens from those without COVID-19, collected in April-May 2019. Sensitivity and specificity with 95% confidence intervals (CI) were calculated. Linear regression was used to examine the relationship between the magnitude of serological response and clinical characteristics. There were 80 patients from whom 86 sera specimens were collected; six patients had duplicate specimens. There were 95 negative control specimens from 95 patients. The clinical sensitivity of the Elecsys assay was 98.84% (95% CI 93.69-99.97), specificity was 100% (95% CI 96.19-100.00); the Liaison assay clinical sensitivity was 96.51% (95% CI 90.14-99.27), specificity was 97.89% (95% CI 92.60-99.74); the Access assay clinical sensitivity was 84.88% (95% CI 75.54-91.70), specificity was 98.95% (95% CI 94.27-99.97); and the Vitros assay clinical sensitivity was 97.67% (95% CI 91.85-99.72), specificity was 100% (95% CI 96.15-100.00). A requirement for hospitalisation for COVID-19 infection was associated with a larger Vitros, Liaison and Access IgG response whilst fever was associated with a larger Elecsys response. All assays evaluated with the exception of the Access assay demonstrated similar performance. The Elecsys assay demonstrated the highest sensitivity and specificity.

Introduction

The COVID-19 pandemic caused by the novel coronavirus, SARS-CoV-2, has spread globally. Key to control of the pandemic is accurate and timely diagnosis coupled with infection control measures to limit transmission. The gold standard for diagnosis relies on the detection of the presence of SARS-CoV-2 nucleic acid most commonly by polymerase chain reaction (PCR). Immunoassays may serve as an adjunct to diagnosis in patients who present late in their illness with only low levels of nucleic acid present in the upper respiratory tract, as well as having a key role in epidemiological serosurveys to determine disease prevalence.

In this study, we sought to assess and compare the clinical performance of four commercially available immunoassays using a panel of sera collected from patients who had recovered from their COVID-19 infection.

Materials and methods

Ethical considerations

This study was reviewed and approved by the Austin Health Human Research Ethics Committee (HREC/65529/Austin-2020).

COVID-19 infected patients

All patients diagnosed with COVID-19 infection and managed at Austin Health (Melbourne, Australia) between March and April 2020 were prospectively enrolled into a clinical database (database reference number CD20002). All patients had at least one combined naso- and oro-pharyngeal swab tested for the presence of SARS-CoV-2 nucleic acid by polymerase chain reaction (PCR) using a qualitative assay. There were three different assays in use during this period due to reagent shortages: AusDiagnostics Coronavirus Typing (8-well) assay, performed at Austin Pathology; in house E gene assay with primers based on Corman et al. performed at the Microbiological Diagnostic Unit Public Health Laboratory; and in house RdRP primer assay and E gene assay based on Corman et al. performed at the Victorian Infectious Diseases Reference Laboratory.

Patients were invited to participate in this study following recovery from COVID-19 infection, at least six weeks following onset of symptoms (Recovery cohort). This duration following infection was chosen due to existing institutional infection control policies. Clinical details regarding date of onset of symptoms, date of first nucleic acid detection by PCR, fever and respiratory tract symptoms, requirement for hospital or intensive care unit admission, and patient co-morbidities were obtained from the clinical database.

In addition to the Recovery cohort, a small subset of patients who required admission to hospital due to COVID-19 infection had remnant sera saved from routine blood tests collected during their hospitalisation to investigate the timing of antibody detection.

All samples were collected in serum separator tubes (Becton Dickinson, USA), sera harvested and stored at –80°C until testing was performed. The same aliquot was used for all four assays. Each aliquot underwent one freeze thaw cycle.

Negative controls

These were stored sera collected from patients in April–May 2019. Further information on the presence of other antibodies present in these specimens can be found in Appendix A.

Correlation and precision studies

Only samples with numerical results within the linear range of each assay were included in the statistical analysis for assay correlation. Residual serum from COVID-19 infected patients was used to create two serum pools, and each pool run in replicates of 10 on each assay.

Instrumentation and analysis

The platforms required for the performance of the assays chosen were previously in use at Austin Health. Specimens were analysed using four immunoassays according to the manufacturer's instructions. Interpretation of results was performed according to the manufacturer's instructions. The operator was blinded to the SARS-CoV-2 infection status of the specimens being analysed.

The Elecsys Anti-SARS-CoV-2 assay (Roche Diagnostics, Germany) was performed on the Cobas e801. This is an electrochemiluminescence immunoassay for the detection of total antibodies against SARS-CoV-2 using a recombinant protein representing the nucleocapsid (N) antigen. Results are reported as numeric values in form of a cut-off index (COI) with COI values <1.0 interpreted as non-reactive or negative for anti-SARS-CoV-2 antibodies and COI ≥1.0 interpreted as reactive or positive for anti-SARS-CoV-2 antibodies.

The Liaison SARS-CoV-2 S1/S2 IgG assay (DiaSorin, Italy) was performed on the Liaison XL. This is an indirect chemiluminescence immunoassay for the detection of IgG anti-S1 and IgG anti-S2 specific antibodies to SARS-CoV-2. Results are reported as antibody concentrations in arbitrary units (AU/mL) with concentrations of <12.0 AU/mL interpreted as negative; concentrations of ≥12.0 to <15.0 AU/mL interpreted as equivocal; and concentrations of ≥15.0 AU/mL interpreted as positive.

The Access SARS-CoV-2 IgG assay (Beckman Coulter, USA) was performed on DxI800. This is a paramagnetic particle chemiluminescent immunoassay for the detection of IgG antibodies to SARS-CoV-2 using a recombinant protein specific for the receptor binding domain (RBD) of the S1 protein. Results are reported as signal sample/cut-off (S/CO) with S/CO values <0.8 interpreted as non-reactive; S/CO values ≥0.8 to <1.0 interpreted as equivocal; and S/CO values ≥1.0 interpreted as reactive for SARS-CoV-2 IgG antibodies.

The Vitros Immunodiagnostic Products Anti-SARS-CoV-2 IgG assay (Ortho Clinical Diagnostics, France) was performed on Vitros 5600. This is an immunometric assay for the detection of SARS-CoV-2 IgG using SARS-CoV-2 spike protein S1 antigen. Results are reported as signal sample/cut-off (S/CO) with S/CO values <1.00 interpreted as non-reactive and S/CO values ≥1.00 interpreted as reactive for SARS-CoV-2 IgG antibodies.

Samples were analysed in batches, with calibration and quality control performed to each manufacturer's recommendations.

Statistical analysis

Patient characteristics were presented as median (interquartile range; IQR) and frequency (percentage). Clinical sensitivity was determined using the SARS-CoV-2 PCR as the reference standard. To determine clinical specificity and cross-reactivity, specimens collected in April–May 2019 were assumed to be negative, as these were collected prior to the emergence of SARS-CoV-2 in late 2019.

The relationship between the magnitude of response and clinical characteristics was explored using linear regression. Some outcomes required a transformation using natural logarithm to enable better model fit. Results are expressed as change in the outcome (in original units or % where transformation was required) with 95% confidence intervals (CI).

Relationship between days from symptom onset and development of immune response was evaluated in a sample of seven participants with serial blood samples and is presented graphically. Analyses were performed using Stata 16.1 (StataCorp, USA). Correlation studies were analysed using Analyse-it for Microsoft Excel 5.30.1, build 7059.19477.

Results

Numerical results varied across the four assays. The range of results obtained were: Elecsys 0.07–156 COI; Liaison <3.80–1430 AU/mL; Access 0.01–57.5 S/CO; Vitros 0.01–23.9 S/CO. Passing–Bablok regression graphs and equations were derived for each assay pair (Supplementary Fig. 1A–F, Appendix A).

Assay precision [from lowest to highest coefficient of variation % (CV)] for serum pool A was (assay/mean value/CV%): Elecsys/42.5/1.0; Vitros/15.3/1.8; Liaison/77.7/2.8; Access/13.8/4.4; and for serum pool B: Elecsys/116/0.4; Vitros/19.5/1.3; Access/26.0/2.7; Liaison/137/5.6.

Clinical sensitivity and specificity of the assays evaluated

For the analysis of clinical sensitivity and specificity, only specimens that had testing on all four assays were included. There were 80 patients from whom 86 sera specimens were collected in the Recovery cohort. Six patients had two duplicate specimens. The clinical characteristics of this cohort is shown in Table 1 . These specimens were collected 45–130 days after symptom onset. The majority of patients had COVID-19 infection that did not require hospitalisation and were managed in the community only. All patients had fever and/or respiratory tract symptoms; none were asymptomatic. Only two patients were immunosuppressed. There were 95 negative control specimens from 95 patients.

Table 1

Clinical characteristics of the Recovery cohort (n=80)

Characteristic	n (%)
Sex, male	42 (52%)
Age, years, median (IQR)	54.5 (35, 63)
Ethnicity, white	75 (94%)
Days from symptom onset until serum specimen collected, median (IQR)	83 (76, 88)
Days from first PCR positive until serum specimen collected, median (IQR)	77 (72, 81.5)
Days from symptom onset until first PCR positive specimen collected, median (IQR)	5.5 (3, 7)
Age adjusted Charlson comorbidity index, median (IQR)	1 (0, 2)
Fever present	45 (56%)
Respiratory symptomsa	77 (96%)
Requiring hospital admission	24 (30%)
Requiring intensive care unit admission	4 (5%)
Immunosuppressed	2 (3%)

IQR, interquartile range.

Respiratory symptoms defined as the presence of cough, sore throat, rhinorrhoea, or shortness of breath.

For the Recovery cohort, the values obtained on the immunoassays were (assay/median/IQR): Elecsys/82.05 COI/(47.3, 109); Liaison/97.65 AU/mL/(44, 235); Access/5.41 S/CO/(1.535, 21.335); and Vitros/13.4 S/CO/(7.26, 18.5) (Fig. 1 ; Supplementary Table 1, Appendix A).

Fig. 1

Immunoassay values for the Recovery cohort of patients previously diagnosed with COVID-19 infection by nucleic acid detection. Results for the Elecsys, Access and Vitros assays are graphed using the left y-axis. Results for the Liaison assay are graphed using the right y-axis. Line and bars for each cohort represents the median with interquartile range.

There was one Recovery specimen that tested non-reactive on all assays, one specimen that only tested reactive on the Elecsys assay and one specimen that tested reactive on the Elecsys and Vitros assays (Supplementary Table 1, Appendix A).

For the Elecsys assay, all Recovery specimens apart from one specimen were reactive (85/86) and all negative control specimens were non-reactive (95/95). The clinical sensitivity was 98.84% (95% CI 93.69–99.97) and specificity was 100% (95% CI 96.19–100.00).

For the Liaison assay, there were 83/86 Recovery specimens that were reactive and 93/95 negative control specimens that were non-reactive. One negative control specimen was reactive (this specimen was also reactive for hepatitis B surface antibody) and one specimen was equivocal (this specimen was also reactive for hepatitis A total antibody, measles IgG, VZV IgG, EBV VCA IgG, EBV NA IgG, HSV type 1 IgG, HSV type 2 IgG and ANA 1:160 speckled). The clinical sensitivity was 96.51% (95% CI 90.14–99.27) and specificity was 97.89% (95% CI 92.60–99.74).

For the Access assay, there were 73/86 Recovery specimens that were reactive, five specimens that were equivocal and eight specimens that were non-reactive. To calculate clinical sensitivity, when the equivocal results were grouped together with the reactive results, the clinical sensitivity was 90.70% (95% CI 82.49–95.90). If only reactive results were considered, the clinical sensitivity was 84.88% (95% CI 75.54–91.70). There were 94/95 non-reactive negative control specimens; one specimen was reactive (this specimen was also reactive for hepatitis A total antibody and hepatitis B surface antigen). The clinical specificity was 98.95% (95% CI 94.27–99.97).

For the Vitros assay, 84/86 Recovery specimens were reactive; the clinical sensitivity was 97.67% (95% CI 91.85–99.72). All (95/95) negative control specimens tested non-reactive; the specificity was 100% (95% CI 96.15–100.00).

In the six patients in the Recovery cohort who had duplicate specimens collected, there appeared to be a fall in the S/CO index or concentration in many instances, except with the Vitros assay where S/CO values demonstrated little change (Supplementary Table 1, Appendix A).

Seroconversion in patients requiring admission with serial samples

There were six patients with 79 specimens (median 11 serum specimens; IQR 6, 18) who had serial residual sera opportunistically stored during their inpatient admission and analysed on the four assays being evaluated (Supplementary Fig. 2A–D, Appendix A). In most patients seroconversion occurred approximately 15 days after the onset of symptoms, however there is considerable variability, especially among Roche results.

Effect of clinical variables on the magnitude of serological response

These results are demonstrated in Supplementary Table 2A,B (Appendix A).

On univariate analysis age, comorbidities, fever and being hospitalised were associated with larger Elecsys response. After adjustment for these variables, only the presence of fever showed significant association; those with fever had for 18.84 COI units larger response (95% CI 0.95, 36.74; p=0.039).

After adjustment for age, comorbidities, immunosuppression, time since first PCR positivity, hospitalisation and ICU admission, variables associated with a larger Vitros S/CO result were greater age [1 year increase in age associated with 0.11 S/CO larger response (95% CI 0.00, 0.20), p=0.052] and requiring hospitalisation [5.0 S/CO larger response (95% CI 2.09, 7.93), p=0.001], while a longer duration from first PCR positive test was possibly associated with smaller response (increase in duration of 1 day associated with a decrease of 0.1 S/CO (95% CI –0.21, 0.01), p=0.085).

Larger Liaison concentration results (after adjustment for age, comorbidities, immunosuppression, hospitalisation, ICU admission, time since first PCR positivity and time since symptom onset) were associated with hospitalisation [result larger for 149.36% (95% CI 48.20%, 319.57%), p=0.001], while longer duration from first PCR positive test was associated with lower IgG concentration [increase in duration of 1 day associated with a decrease for 4.51% (95% CI –7.88%, –1.02%), p=0.012].

After adjustment for age, comorbidities, hospitalisation, time since first PCR positivity and time since symptom onset, larger Access S/CO responses were associated with older age [1 year increase in age associated with 3.15% increase in response (95% CI 0.70%, 5.66%), p=0.012] and hospitalisation [increase in 245.4% (95% CI 74.57%, 583.4%), p=0.001].

Discussion

This head-to-head study examining four commercial high-throughput and widely available assays shows that the Elecsys, Vitros and Liaison assays demonstrate good clinical sensitivity (>96%) for the detection of SARS-CoV-2 antibodies up to 130 days following onset of symptoms in COVID-19 infection. Of the four assays, the Access assay showed the lowest sensitivity; this assay targets the S1 protein and two other assays evaluated, Vitros and Liaison, also targeted this protein. The sensitivity issues seen with the Access assay may have been due to an issue with the single batch of reagents used. There was only one patient who was non-reactive on all four serological assays consistent with other studies showing that a proportion of patients may fail to seroconvert.5, 6, 7, 8, 9, 10, 11 We did not have access to an earlier specimen and it is unknown whether the patient failed to seroconvert or had a sero-reversion event. No testing on reference immunoassays such as neutralisation tests or immunofluorescence assays were performed in this study.

This study differs from other studies , that have examined the utility of serology assays in that we had few specimens soon after COVID-19 symptom onset but instead had specimens collected distant from symptom onset, with the majority of specimens obtained from patients who did not require hospitalisation for their COVID-19 infection. Hence our assessment of seroconversion kinetics was restricted, however these results demonstrate a measure of serological response durability, although longer follow up is required.

We found that clinical patient factors can influence the magnitude of the serological response although this varied depending on the assay used. In three assays (Vitros, Liaison and Access) we found that hospitalisation was associated with a larger IgG response, suggesting that patients with a more severe illness will mount a greater response. This finding has been described by others.12, 13, 14 Although this association with hospitalisation was not found with the Elecsys assay, the presence of fever was associated with a greater response on this assay, also suggesting that a systemic inflammatory response is associated with a serological response. Because none of the patients in our study were asymptomatic, we were not able to ascertain whether a serological response could be detected in the asymptomatic patient population using the assays being evaluated in this study. ,

Increased age appeared to be associated with greater serological response measured by the Vitros and Access assays and we hypothesise that this may be due to previous infection with other coronaviruses, however this requires further investigation with a larger dataset.

Negative correlation with the duration from initial illness and IgG response with the Liaison assay (and possibly the Vitros assay) have implications for the choice of assays in seroprevalence surveys, as extensive data on the durability of serological response is yet to be studied given the recency of pandemic onset. This decay in serological response has also been described by other authors although it is not known whether this indicates loss of immunity. The impact on detection sensitivity of a longer duration from symptom onset to serological evaluation is unknown and follow up specimens of longer duration are required to assess this further.

All assays demonstrated good specificity (>97%), in particular the Elecsys and Vitros assays, although this study assessed a relatively small number of negative specimens and we did not have access to sera from patients infected with other endemic coronaviruses, limiting analysis of cross reactivity. Despite this high specificity, use of serological assays in an area of low prevalence such as Australia will be associated with lowered positive predictive value and it is likely that serological diagnosis will remain an adjunct to nucleic acid detection for the diagnosis of SARS-CoV-2 infection in specific settings such as retrospective diagnosis where nucleic acid testing is not detected or not performed due to mild or asymptomatic infection. ,

Conclusion

In conclusion, we demonstrate that the assays evaluated demonstrated similar performance, with the exception of the Access assay. The Elecsys assay demonstrated the highest sensitivity and specificity.