Comparative analysis of three laboratory based serological assays for SARS-CoV-2 in an Australian cohort.

Many unanswered questions remain regarding the role of SARS-CoV-2 serological assays in this unfolding COVID-19 pandemic. These include their utility for the diagnosis of acute SARS-CoV-2 infection, past infection or exposure, correlation with immunity and the effective duration of immunity. This study examined the performance of three laboratory based serological assays, EUROIMMUN Anti-SARS-CoV-2 IgA/IgG, MAGLUMI 2000 Plus 2019-nCov IgM/IgG and EDI Novel Coronavirus (COVID-19) IgM/IgG immunoassays. We evaluated 138 samples from a reference non-infected population and 71 samples from a cohort of 37 patients with SARS-CoV-2 confirmed positive by RT-PCR. The samples were collected at various intervals of 0-45 days post symptoms onset (PSO). Specificity and sensitivity of these assays was 60.9%/71.4% (IgA) and 94.2%/63.3% (IgG) for EUROIMMUN; 98.5%/18.4% (IgM) and 97.8%/53.1% (IgG) for MAGLUMI; and 94.9%/22.5% (IgM) and 93.5%/57.1% (IgG) for EDI, respectively. When samples collected ≥14 days PSO were considered, the sensitivities were 100.0 and 100.0%; 31.0 and 82.8%; 34.5 and 57.1%, respectively. Using estimated population prevalence of 0.1, 1, and 10%, the positive predictive value of all assays remained low. The EUROIMMUN Anti-SARS-CoV-2 IgA lacked specificity for acute diagnosis and all IgM assays offered poor diagnostic utility. Seroconversion can be delayed although all patients had seroconverted at 28 days in our cohort with the EUROIMMUN Anti-SARS-CoV-2 IgG. Despite this, with specificity of only 94% this assay would not be satisfactory for seroprevalence studies in the general Australian population given this is likely to be currently <1%.

Introduction

SARS-CoV-2, a n class="Species">novel coronavirus, emerged in December 2019, and has rapidly spread worldwide resulting in the World Health Organization (WHO) declaration of a pandemic in March 2020. SARS-CoV-2 is a beta coronavirus closely related to SARS-CoV (∼90% homology) which caused an outbreak of respiratory illness in 2003 and has since disappeared. Whole genome sequencing has demonstrated up to 33–47% homology with other circulating human beta coronaviruses. , The clinical spectrum of infection from SARS-CoV-2 is wide, from asymptomatic infection to severe respiratory illness and death.

Nucleic acid testing (NAAT) with real-time reverse transcription polymerase chain reaction (RT-PCR) on a range of respiratory tract samples has been the mainstay of diagnosis for SARS-CoV-2. , Although PCR may be prolonged, most patients become PCR negative after 10–14 days. , Serological testing may be useful for delayed diagnosis in patients who are PCR negative or are greater than 14 days from their onset of symptoms as mentioned in the Infectious Diseases Society of America (IDSA) guidelines.7, 8, 9 Additional uses for serological testing for SARS-CoV-2 include serosurveillance and epidemiological studies, outbreak investigation and contact tracing, vaccine development and use of convalescent plasma for therapy.10, 11, 12 It is still unknown whether development of an antibody response confers immunity and if so, what the longevity of immunity will be, particularly as reports of reinfection are emerging. ,

The most common antigens used in serological assays to date have been the spike glycoproteins, n class="Gene">S1 comprising the receptor binding domain (RBD) and S2 required for fusion, and the nucleocapsid protein. Some assays have targeted IgG only, or in combination with IgM or IgA. These assays have been in the form of rapid tests (e.g., lateral flow immunochromatographic test) or laboratory-based assays including enzyme-linked immunosorbent assays (ELISA), chemiluminescence immunoassays (CLIA), immunofluorescent assays and neutralisation assays. , The platform and performance of the various antibodies is important when choosing an assay for a specific purpose and population.

Sullivan Nicolaides Pathology (part of the SOn class="Gene">NIC healthcare group) services both community and hospital patients predominantly from the states of Queensland and Northern New South Wales. As the ‘curve has flattened’ attention has turned to the role of serology in this evolving pandemic. We performed a comparative analysis of the EUROIMMUN Anti-SARS-CoV-2 IgG/IgA, MAGLUMI 2000 Plus 2019-nCov IgG/IgM and EDI Novel Coronavirus (COVID-19) IgG/IgM plate immunoassays that are currently available to our laboratory.

Methods

Ethics

Ethics approval from the Sullivan Nicolaides Pathology Low Risk Ethics Committee allowed for prospective serum collection from n class="Species">SARS-CoV-2 RT-PCR positive patients.

Patient cohorts

The following serum panels were established.

Specificity analysis

This cohort included 48 patient samples held at –80oC and collected prior to November 2019 (pre-COVID-19) from Queensland and NSW healthy populations and 90 samples prior to November 2019 (pre-COVID-19) from a range of patients with other confirmed respiratory illnesses, acute infections, or autoimmune disease with potential for cross reactions.

Sensitivity analysis

Serum and plasma samples were collected from laboratory confirmed RT-PCR positive cases of COVID-19. Serial serum samples were obtained from a number of n class="Species">patients. Demographic details including age, gender, severity (hospitalisation versus community care) and host immune status were collected. Where available, stored serum samples prior to December 2019 from these SARS-CoV-2 infected patients were tested to demonstrate seroconversion and included in the descriptive analysis.

Serological testing

The six serological assays evaluated are described in Table 1 . These assays were chosen because they were among the first available for use. Qualitative results and index values reported by the instrument were used in analyses except for the Epitope assays where the optical density cut-offs for each run were converted to indices to standardise comparison of assays.

Table 1

Assays for evaluation

Assay	Manufacturer	Technology	Antigen	Cut-offs
EUROIMMUN Anti-SARS-CoV-2 IgG (FDA-EUA, CE-IVD, TGA)	EUROIMMUN Medizinische Labordiagnostica (Germany)	Enzyme immunoassay (ELISA)	Recombinant structural protein (S1 domain)	Negative: <0.8Borderline: 0.8–1.1Positive: >1.1
EUROIMMUN Anti-SARS-CoV-2 IgA (CE-IVD, TGA)	EUROIMMUN Medizinische Labordiagnostica (Germany)	Enzyme immunoassay (ELISA)	Recombinant structural protein (S1 domain)	Negative: <0.8Borderline: 0.8–1.1Positive: >1.1
EDI Novel Coronavirus (COVID-19) IgGa (CE-IVD)	Epitope Diagnostics (USA)	Enzyme immunoassay (ELISA)	Recombinant full length nucleocapsid protein	Negative: <0.8;Equivocal: 0.8–1.0Positive: >1.0
EDI Novel Coronavirus (COVID-19) IgMa (CE-IVD)	Epitope Diagnostics (USA)	Enzyme immunoassay (ELISA)	Recombinant full length nucleocapsid protein	Negative: <0.8;Equivocal: 0.8–1.0Positive: >1.0
MAGLUMI 2000 Plus 2019-nCov IgG (CE-IVD)	Snibe Diagnostic, China	Chemiluminescence immunoassay	CoV-S (spike) and CoV-N (nucleocapsid)	Negative: <0.9 Equivocal: 0.9–1.1 Positive: >1.1
MAGLUMI 2000 Plus 2019-nCov IgM (CE-IVD)	Snibe Diagnostic, China	Chemiluminescence immunoassay	CoV-S (spike) and CoV-N (nucleocapsid)	Negative: <0.9Equivocal: 0.9–1.1Positive: >1.1

Results normalised to an index after OD cut-offs established for each run to allow comparability across platforms.

Testing was performed exactly as per the manufacturer's instructions, with the use of automated plate washers and an optical density reader. Serial serum samples from individual patients were tested in parallel. Samples were tested in single replicate but if there were discordant results, they were tested in triplicate.

RT-PCR testing

Testing for SARS-CoV-2 was performed using an in-house developed Taqman assay targeting the E gene. All positive samples then underwent three supplementary RT-PCRs targeting the n class="Gene">N gene.

Data management and statistical analysis

Sensitivity, specificity, positive and negative predictive values for each assay were calculated for all samples and greater than 14 days post symptom onset (PSO). Equivocal results were included as positive results for analysis. Binomial confidence intervals (CI) were calculated for all proportions (Clopper–Pearson exact) and level of agreement was calculated using Kappa statistics. Statistical analyses were performed using Graphpad Prism Version 8 (GraphPad Software, USA).

Results

Patient selection and demographics

A total of 209 samples from 175 patients were tested, with 138 samples iene">n the specificity analysis and 71 samples from 37 n class="Species">patients in the sensitivity analysis (Table 2 ).

Table 2

Samples

Cohort	Characteristics	Purpose	Samples/Patients
1	SARS-CoV-2 RT PCR positive patients	Sensitivity analysis	71/37
2	Other non-COVID-19 infections (pre-COVID-19)	Specificity analysis	90/90
3	Pre-COVID-19 controls	Specificity analysis	48/48

Fourteen serum samples were available from days PSO 0–7, seven available from days PSO 8–14, 11 from days PSO 15–21, 15 from days PSO 22–28, and 24 samples from >28 days PSO. The samples were grouped into 0–14 days PSO (n=21) and >14 days PSO (n=50). Although 71 samples were available, only 20 samples collected <14 days PSO and 29 samples collected >14 days PSO were included in the analysis because multiple samples from the same patient were excluded. Serial samples were available for 20 n class="Species">patients, with 11 patients having greater than two samples available from different time points.

All patients except one were managed in the community without requiring hospitalisation. One n class="Species">patient was asymptomatic and two patients in the cohort were immunocompromised. Seventeen of the patients were male and 20 were female. The median age within the cohort was 46 years (range 20–71 years).

Specificity

Table 3 demonstrates the pre-COVID-19 samples used for the specificity panel and the cross-reactivity results with the EUROIMMUn class="Gene">N Anti-SARS-CoV-2 IgG and IgA assays, the MAGLUMI 2000 Plus 2019-nCov IgG and IgM assays, and the EDI Novel Coronavirus IgG and IgM assays, as well as the specificity of the assays. With respect to the IgM/IgA assays, the EUROIMMUN IgA assay was the least specific at 60.9%, with cross-reactivity with other infective agents, autoimmune antibodies and importantly also with the healthy population with 44% testing positive (Table 3). However, if the borderline/equivocal results are removed from the positives, the specificity is 74%. The MAGLUMI IgM and the EDI IgM reported a specificity of 98.6% and 94.9%, respectively. For the IgG assays, the MAGLUMI IgG performed the best with the highest specificity at 97.8%. The EUROIMMUN IgG recorded a specificity of 94.2% and the EDI Novel Coronavirus IgG had a specificity of 93.5%. The MAGLUMI assays had the fewest cross-reactive samples (Fig. 1 , Table 3).

Table 3

Samples used for specificity panel and cross-reactivity results. Numbers indicate equivocal and positive results for each assay

Specificity panel (sample number)	MAGLUMI 2000 plus 2019-nCov IgM	MAGLUMI 2000 plus 2019-nCov IgG	EUROIMMUN anti-SARS-CoV-2 IgA	EUROIMMUN anti-SARS-CoV-2 IgG	EDI novel COVID-19 IgM	EDI novel COVID-19 IgG
Respiratory patients (n=55)
Adenovirus (5)			2
Influenza A (20)			3	1	2	1
Influenza B (19)			5
Parainfluenza (5)			2	1		1
RSV (5)	1		3
Mycoplasma pneumoniae (5)			3	1
Bordetella pertussis (5)			4	1
Potential cross-reacting sera (n=35)
CMV (10)			2	1		2
EBV (10)			4			1
Parvovirus (5)			1
Rheumatoid factor (5)		2	2
Antinuclear antibody (5)			2		1
Healthy population (n=48)	1	1	21	2	1	2
Total specificity (n=138)Confidence interval	98.6 [94.9, 99.2]	97.8 [93.8, 99.6]	60.9, [52.2, 69.1]	94.2 [88.9, 97.5]	94.9 [89.8, 97.9]	93.5 [88.0, 96.9]

CMV, cytomegalovirus; EBV, Epstein–Barr virus; RSV, respiratory syncytial virus.

Fig. 1

Comparison of serological assays ≤14 days PSO and >14 days PSO. Median with interquartile range. The dotted line represents the respective cut-off values recommended by the manufacturer for positive and negative results. Grey zone represents the range with equivocal results. Numbers included: specificity (n=138); ≤14 days (n=20); >14 days (n=29).

Sensitivity

The overall sensitivity of the assays and the sensitivity when tested at <14 days PSO and >14 days PSO was calculated (Table 4 ). For both of the IgM assays, sensitivity was low at both time points and 39 (79%) patients did not develop a measurable IgM response (Fig. 1). The total sensitivity of the EUROIMMUn class="Gene">N IgA was 71.4% and 100% >14 days PSO, with development of IgA antibodies sooner than the MAGLUMI and EDI IgM assays. In addition, IgM antibodies did not develop earlier than IgG antibodies (Fig. 2 ). Of the IgG assays, the EUROIMMUN IgG had the highest sensitivity overall at 63.3% and 100% >14 days PSO, with all patients seroconverting (Fig. 1). Two patients did not seroconvert until day 31 and day 37 PSO; these patients were not immunosuppressed and had a mild illness. The MAGLUMI IgG and EDI IgG had poorer sensitivity at 82.8% and 79.3% >14 days PSO, respectively. Figure 3 demonstrates the comparative performance of the assays overall using a receiver operator curve. The EUROIMMUN IgG performed the best with an area under the curve (AUC) of 0.88 [confidence interval (CI) 0.801,0.938].

Table 4

Comparative sensitivity performance of serological assays

Test assay	Samples/patients total	Sensitivity	Samples/patients >14 days PSO	Sensitivity >14 days PSO	Samples/patients <14 days PSO	Sensitivity <14 days PSO
MAGLUMI 2000 Plus 2019-nCov IgM	71/37	18.4 [8.8,32.0]	49/37	31.0 [15.3,50.8]	20/20	0.00 [0.0]
MAGLUMI 2000 Plus 2019-nCov IgG	71/37	53.1 [38.3, 67.5]	49/37	82.8 [64.2, 94.2]	20/20	10 [1.23, 31.70]
EUROIMMUN Anti-SARS-CoV-2 IgA	71/37	71.4 [56.7, 83.4]	49/37	100.0 [88.1, 100.0]	20/20	30 [11.89, 54.28]
EUROIMMUN Anti-SARS-CoV-2 IgG	71/37	63.3 [48.3, 76.6]	49/37	100.0 [88.1, 100.0]	20/20	10 [1.23, 31.70]
EDI Novel Coronavirus IgM	71/37	22.5 [11.8, 36.6]	49/37	34.5 [17.9, 54.4]	20/20	5 [0.13, 24.87]
EDI Novel Coronavirus IgG	71/37	57.1 [42.2, 71.2]	49/37	79.3 [60.3, 92.0]	20/20	25 [8.66, 49.10]

Confidence intervals in square brackets.

Fig. 2

Timeframe of antibody development days post symptom onset: 0–7 (n=14), 8–14 (n=7), 15–21 (n=11), 22–28 (n=15), >28 (n=24). Total (n=71).

Fig. 3

Receiver operator curve (ROC) including all six assays (n=209).

Four patients had serial nasopharyngeal samples for RT-PCR that remained positive for up to 43 days. All of these n class="Species">patients developed an IgG response by day 14 PSO. All PCR positive patients in the cohort developed an IgG response with the EUROIMMUN IgG assay but not with the MAGLUMI IgG or EDI IgG.

Substantial agreement was demonstrated between the IgG assays. The kappa between EUROIMMUN IgG and EDI IgG and MAGLUMI IgG was 0.76 (CI 0.58,0.94) and 0.621 (CI 0.41,0.833), respectively, and between the MAGLUMI IgG and EDI IgG was 0.842 (CI 0.693,0.991).

Positive and negative predictive values

The positive (PPV) and negative predictive values (NPV) were calculated using an estimated population prevalence of 0.1%, 1% and 10% (Table 5 ). The current estimated population prevalence in Australia is <0.1%.

Table 5

Positive predictive value (PPV) and negative predictive value (NPV) calculated if prevalence of COVID-19 in the population is 0.1%, 1%, 10%, >14 days post onset of symptoms

Test assay	PPV (%) 0.1% prevalence	NPV (%) 0.1% prevalence	PPV (%) 1% prevalence	NPV (%) 1% prevalence	PPV (%) 10% prevalence	NPV (%) 10% prevalence
MAGLUMI 2000 Plus 2019-nCov IgM	2.2	99.9	18.3	99.3	71.1	92.8
MAGLUMI 2000 Plus 2019-nCov IgG	3.6	99.9	27.5	99.8	80.7	98.1
EUROIMMUN Anti-SARS-CoV-2 IgA	0.3	100.0	2.5	100.0	22.1	100.0
EUROIMMUN Anti-SARS-CoV-2 IgG	1.7	100.0	14.8	100.0	65.7	100.0
EDI Novel Coronavirus IgM	0.7	99.9	6.4	99.3	42.9	92.9
EDI Novel Coronavirus IgG	1.2	99.9	11.0	99.8	57.5	97.6

Positive predictive value (PPV) and negative predictive value (NPV) calculated if prevalence of n class="Disease">COVID-19 in the population is 0.1%, 1%, 10%, >14 days post onset of symptoms

Discussion

Here we report the performance characteristics of six recently available immunoassays for the detection of SARS-CoV-2 antibodies. All three IgG assays had a specificity greater than 90% with the MAGLUMI IgG reporting the highest specificity at 97.8%. Both MAGLUMI and EDI IgM assays had specificity greater than 94%; however, the EUROIMMUn class="Gene">N IgA was poorer, even when the borderline results were removed from the positive group, with marked cross-reactivity to multiple different sample types with no specific pattern, including a significant proportion of the tested healthy population. This suggests that an IgA response (via EUROIMMUN assay) would not be reliable in predicting acute infection and the high false positivity rate limits its utility. Other studies have reported variable results and differences may reflect sample size variation; however, there is a paucity of robust evaluation data for comparison. , Interestingly, there is increased sequence homology of SARS-CoV-2 to other beta coronaviruses in the region of the nucleocapsid protein compared to the spike proteins; however, this pattern of cross reactivity was not assessed directly in this study because samples from other human coronavirus infections were not available for analysis.

Currently in Australia the prevalence of COVID-19 is low; therefore, a highly specific assay is essential to avoid a high false positive rate when used for diagnosis or confirmation of past exposure. All assays had poor positive predictive value <2% (when estimated population prevalence 0.1%), still higher than the probable current prevalence in Australia, suggesting these assays may not be suitable for population-wide screening and use of a secondary confirmation assay would be required. Iene">n this study the overall sensitivity of the IgM assays was low, and although the EUROIMMUn class="Gene">N IgA was more sensitive than the IgM assays, the poor specificity reduces the overall diagnostic accuracy of the test precluding its widespread use. The performance of the IgG assays was better, particularly when used >14 days PSO. The EUROIMMUN IgG ELISA assay performed the best with a sensitivity at >14 days of 100%. The sensitivity of the IgM assays prior to 14 days PSO was low and IgM did not peak earlier than IgG, suggesting there is limited utility in testing for an IgM response over and above an IgG response when using these assays. Other studies have suggested development of IgM may occur earlier using a nucleocapsid antigen target compared to the spike glycoproteins; however, this was not observed in our study with poor IgM response demonstrated for both assays. , The possibility of antigenic evolution over time and changes in viral structural proteins affecting performance of assays will also need to be constantly reassessed as more genomic information about SARS-CoV-2 becomes available.

All PCR positive patients in the cohort developed an IgG response with the EUROIMMUn class="Gene">N IgG assay, although there were two patients who had delayed seroconversion out to 37 days. This suggests the need to perform convalescent testing out to 4–6 weeks. Neither of these patients were hospitalised and had mild clinical illness. Some studies have suggested that asymptomatic or mild cases may have an attenuated or delayed antibody response. , Within our cohort, one patient was asymptomatic and was tested as a contact of a confirmed case. This patient developed IgG antibodies at day 14 PSO via EUROIMMUN, however neither MAGLUMI nor EDI detected IgG in this patient. The immunosuppressed population may also fail to seroconvert and develop an antibody response. In our cohort, there were two immunosuppressed patients, both of whom developed an IgG response on all three IgG assays.

The use of serology for outbreak investigation has been demonstrated in Singapore as a tool for contact tracing and completing epidemiological links. On a larger scale, seroprevalence studies may be used to ascertain population prevalence. Importantly, the concept of ‘immunity passports’ must be used cautiously. Protective antibodies can only be definitively diagnosed using neutralisation assays which are not routinely available. It has been speculated that the components of the spike proteins are most likely to produce neutralising antibodies due to the ability to prevent binding of the RBD to the target n class="Species">human receptor (ACE-2). It is possible that development of IgG targeting the S1 antigen, such as in the EUROIMMUN IgG assay, may reflect protection but this cannot be interpreted without measurement of neutralising antibodies. Other studies have demonstrated variable correlation between binding antibodies and neutralising antibodies, depending on severity of illness, but further evaluation is required. , In addition, the level of antibody required to result in protection is yet to be elucidated. Isolated reports of SARS-CoV-2 reinfection with genetically distinct strains have recently emerged. In one report from Hong Kong, a patient did not have an original antibody response by 10 days following the original infection and was antibody negative on diagnosis of reinfection 4 months later; however, he then did develop an antibody response by day 5, possibly representing an anamnestic response.

Limitations of this study include the relatively small number of positive patient samples and the single laboratory study, with the prevalence in Australia much lower than many other countries. The cohort also included n class="Species">patients with milder clinical illness who were not hospitalised. However, this single laboratory services a geographically diverse patient population. Although the specificity panel included a large number of patients with other respiratory illness, no confirmed patients with other human coronavirus infections, e.g., HCoV-NL63 and HCoV-HKU1, were available for inclusion. There was also no reference standard used for serology testing, e.g., neutralisation assay for comparison, which may limit the strength of conclusions that can be drawn.

Conclusions

This comparative analysis demonstrates superiority of the EUROIMMUN Anti-n class="Species">SARS-CoV-2 IgG ELISA assay over the MAGLUMI 2000 Plus 2019-nCov IgG and the EDI Novel Coronavirus IgG for SARS-CoV-2 antibody testing. Inclusion of a specific IgM or IgA in any diagnostic algorithm is limited in our observations. Although serological testing will be useful in delayed diagnoses and confirmation of past exposure in selected settings, the low prevalence of this infection in the Australian population means that the positive predictive value when applied to the general population is concerning and caution must be taken if widespread testing occurs.

Conflicts of interest and sources of funding

The authors state that there are no conflicts of interest to disclose.