Diagnostic performance of seven rapid IgG/IgM antibody tests and the Euroimmun IgA/IgG ELISA in COVID-19 patients.

OBJECTIVES: To evaluate the diagnostic performance of seven rapid IgG/IgM tests and the Euroimmun IgA/IgG ELISA for antibodies against severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in COVID-19 patients.
METHODS: Specificity was evaluated in 103 samples collected before January 2020. Sensitivity and time to seropositivity was evaluated in 167 samples from 94 patients with COVID-19 confirmed with RT-PCR on nasopharyngeal swab.
RESULTS: Specificity (confidence interval) of lateral flow assays (LFAs) was ≥91.3% (84.0-95.5) for IgM, ≥90.3% (82.9-94.8) for IgG, and ≥85.4% (77.2-91.1) for the combination IgM OR IgG. Specificity of the ELISA was 96.1% (90.1-98.8) for IgG and only 73.8% (64.5-81.4) for IgA. Sensitivity 14-25 days after the onset of symptoms was between ≥92.1% (78.5-98.0) and 100% (95.7-100) for IgG LFA compared to 89.5% (75.3-96.4) for IgG ELISA. Positivity of IgM OR IgG for LFA resulted in a decrease in specificity compared to IgG alone without a gain in diagnostic performance, except for VivaDiag. The results for IgM varied significantly between the LFAs with an average overall agreement of only 70% compared to 89% for IgG. The average dynamic trend to seropositivity for IgM was not shorter than for IgG. At the time of hospital admission the sensitivity of LFA was <60%.
CONCLUSIONS: Sensitivity for the detection of IgG antibodies 14-25 days after the onset of symptoms was ≥92.1% for all seven LFAs compared to 89.5% for the IgG ELISA. The results for IgM varied significantly, and including IgM antibodies in addition to IgG for the interpretation of LFAs did not improve the diagnostic performance.

Introduction

The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is the cause of coronavirus disease 2019 (COVID-19), an acute respiratory syndrome that was first identified at the end of 2019 in Wuhan, China, and quickly evolved into a pandemic. The current gold standard for the diagnosis of COVID-19 is the detection of viral RNA in respiratory tract samples [1]. However, the sensitivity of nucleic acid amplification techniques is <100%. False negatives can occur, especially when using nasopharyngeal swabs (positivity rate estimated at 54–74%) because of difficulty in sampling; false negatives can also occur in patients with low viral loads (especially in patients who present at day 8 or later) and in mild cases [1].

Detection of antibodies has been proposed as an additional diagnostic tool which could help in the diagnosis of patients with suspected COVID-19 who have a negative PCR result, or in whom no respiratory sample for PCR was taken at the time of acute illness (e.g. due to lack of adequate resources during an outbreak). Seroconversion for SARS-CoV-2 is estimated to occur 7–14 days after the onset of symptoms, when the sensitivity of the PCR decreases [3,4]. Detection of antibodies could be useful in patients in whom a past asymptomatic, atypical or mild infection is suspected. Antibody tests can provide epidemiological information about the number of affected individuals and can guide control measures taken by governments [2,5,6].

There are currently two main ways of investigating these antibodies: by enzyme-linked immunosorbent assay (ELISA) and by lateral flow assay (LFA). At the end of March 2020 the first ELISA, the Euroimmun IgA and IgG ELISA, received CE marking. Although ELISA is a long-established method for antibody detection, disadvantages include a longer turn around time, the need for a laboratory environment, and higher labour costs needed to produce a result. LFAs, on the other hand, are medical diagnostic tests which can be used at the point of care or in the laboratory and typically give a response in less than 15 min.

In the first quarter of 2020 more than 100 so called ‘rapid tests’ for the detection of IgM/IgG antibodies were marketed. There are, however, important concerns about the quality and diagnostic performance of rapid tests for SARS-CoV-2. At the end of March, the Spanish government said they had returned a shipment of rapid antigen LFAs after they were found to be unreliable [7], and at the beginning of April the British government reported problems with the performance of antibody LFAs [8]. As a result of these problems, doctors and regulators throughout the world started to look with suspicion at rapid tests for COVID-19.

The aim of this study was to critically evaluate the diagnostic performance of seven rapid LFA tests for professional use only to detect SARS-CoV-2 antibodies, as well as the Euroimmun IgA/IgG ELISA. We determined the specificity, the sensitivity, and the time to seropositivity of IgM and IgG.

Materials and methods

Patient selection

This study was performed at the University Hospital Leuven and approved by the local ethics committee (protocol number S63897). To assess specificity, we selected samples from 103 patients collected before January 2020 as negative controls. These included (a) a disease control group of 49 consecutive patients with a respiratory infection who had a PCR test for respiratory pathogens in the period September to November 2019, the serum samples being collected between day 1 and day 40 after the PCR test; (b) 14 samples from patients with a confirmed non-SARS-CoV-2 coronavirus infection collected 12–42 days after the positive PCR; and (c) 40 samples from patients with antibodies against other pathogens—e.g. cytomegalovirus (CMV), Epstein–Barr virus (EBV), human immunodeficiency virus (HIV)—from routine serology testing (Supplementary Material Table S1). All samples were stored at –20°C until use.

For analysis of sensitivity and dynamic trend to seropositivity, a total of 167 samples from 94 patients who presented at the University Hospitals Leuven with a clinical suspicion of COVID-19 in March and April 2020, and who were diagnosed with COVID-19. Only patients positive for SARS-CoV-2 with real-time polymerase chain reaction (RT-PCR) on nasopharyngeal swabs (UTM®, Copan, Italy) and for whom residual samples were available were included. RT-PCR was performed using an in-house method complying with the WHO guidelines [9]. Two patients who were initially considered for the study were excluded because of treatment with rituximab for a B-cell malignancy.

Data collection and data analysis

For the 94 COVID-19 patients the date of symptom onset, clinical classification (critical versus non-critical) and basic demographic information (male/female, age) were recorded. The group consisted of 66 male and 28 female patients with a median age of 67.5 years (range 23–90 years). The median time between onset of symptoms and admission to the hospital was 7 days (80% of patients were admitted the day of the first positive RT-PCR result). Twenty-nine patients (35%) were classified as critical if mechanical ventilation was required or in the case of fatality.

The online Supplementary Material details information about the LFAs and ELISAs (Supplementary Material Table S2) and data analysis. We calculated the positive likelihood ratio (LR+) = sensitivity/(1 – specificity) as a measure of the diagnostic performance of a test.

Results

Specificity

The specificity (95% confidence interval, 95%CI) of LFAs varied between 91.3% (84.0–95.5) and 100% (95.7–100) for IgM, 90.3% (82.9–94.8) and 99.0% (94.2–100) for IgG, 85.4% (77.2–91.1) and 99.0% (94.2–100) for IgM OR IgG, and 97.1% (91.4–99.4) and 100% (95.7–100) for IgM AND IgG (Table 1 ). The specificity was >95% for four LFAs for IgM, five LFAs for IgG, two LFAs for the combination IgM OR IgG (either one positive), and all seven LFAs for the combination of IgM AND IgG (both positive). The specificity of the ELISA was 96.1% (90.1–98.8) for IgG and only 73.8% (64.5–81.4) for IgA. Given the low specificity of the IgA ELISA, this assay was not tested further. Multi-G IgM and Prima IgG were the only assays with more than one false-positive result in the 14 samples from non-SARS-CoV-2 coronaviruses (two and three, respectively) (Supplementary Material Table S3).

Table 1

Overall diagnostic performance of the different assays for IgM alone, IgG alone, IgM OR IgG, and IgM AND IgG

IgM	n	Clungene	OrientGene	VivaDiag	StrongStep	Dynamiker	Multi-G	Prima
Sensitivity (LR+) [95%CI]	153	39.2% (4.5)[31.8–47.1]	72.5% (15)[65.0–79.0]	65.4% (+∞)[57.5–72.5]	32.0% (33)[25.1–39.8]	69.3% (14)[61.6–76.1]	43.8% (5.0)[36.2–51.7]	56.2% (8.3)[48.3–63.8]
Day 0–6	37	16.2% (1.9)[7.3–31.5]	40.5% (8.0)[26.3–56.5]	35.1% (+∞)[21.8–51.3]	10.8% (11)[3.7–25.3]	46.0% (9.5)[31.0–61.6]	27.0% (3.1)[15.2–43.1]	43.2% (6.4)[28.7–59.1]
Day 7–13	78	42.3% (4.8)[32.0–53.4]	75.6% (16)[65.0–83.9]	64.1% (+∞)[53.0–73.9]	33.3% (34)[23.8–44.4]	66.7% (14)[55.6–76.2]	44.9% (5.1)[34.3–55.9]	56.4% (8.3)[45.4–66.9]
Day 14–25	38	55.3% (6.3)[39.7–69.9]	97.4% (20)[85.3–100]	97.4% (+∞)[85.3–100]	50.0% (52)[34.9–65.1]	97.4% (20)[85.3–100]	57.9% (6.6)[42.2–72.2]	68.4% (10)[52.5–81.0]
Specificity [95%CI]	103	91.3%[84.0–95.5]	95.1%[88.9–98.2]	100%[95.7–100]	99.0%[94.2–100]	95.1%[88.9–98.2]	91.3%[84.0–95.5]	93.2%[86.4–96.9]

Sensitivity and dynamic trend to seropositivity

The sensitivity of LFAs (IgM, IgG, IgM OR IgG, and IgM AND IgG) and the IgG ELISA was <50% during the first week after onset of symtoms (days 0–6) except for the Prima IgM OR IgG (Table 1). Prima IgM OR IgG had a sensitivity of 56.8% (40.9–71.3), but a specificity of only 85.4% (77.2–91.1). The sensitivity of all the assays increased during the second week (days 7–13). After 2 weeks (days 14–25) the sensitivity of the LFAs ranged between 50.0% (34.9–65.1) and 97.4% (85.3–100) for IgM, 92.1% (78.5–98.0) and 100% (89.1–100) for IgG, 97.4% (85.3–100) and 100% (89.1–100) for IgM OR IgG, and 50.0% (34.9–65.1) and 94.7% (81.8–99.5) for IgM AND IgG (Table 1). While the combination of IgM OR IgG increased the overall sensitivity of LFAs compared to either antibody class alone, this resulted in a decrease in the LR + for all the assays except VivaDiag (due to its good specificity for both IgM and IgG).

The performance of the IgM LFAs varied greatly, with an overall sensitivity ranging from 32.0% (25.1–39.8) (StrongStep) to 72.5% (65.0–79.0) (OrientGene). This large variation was accompanied by an overall agreement in the results between the different IgM LFAs of only 70%. For IgG, the agreement was notably higher, with an average agreement of 89% between the different assays (Table 2 ).

Table 2

Percentage agreement between the different lateral flow assays (LFAs) for IgM and IgG in 94 COVID-19 patients (153 samples for sensitivity)

% Agreement [95%CI]	IgM
	OrientGene	VivaDiag	StrongStep	Dynamiker	Multi-G	Prima
Clungene	64.1%[56.2–71.2]	68.6%[60.1–75.5]	73.2%[65.7–79.6]	66.0%[58.2–73.1]	64.1%[56.2–71.2]	63.4%[55.5–70.6]
OrientGene		83.7%[76.9–88.7]	58.2%[50.2–65.7]	85.0%[78.4–89.8]	63.4%[55.5–70.6]	68.0%[60.2–74.9]
VivaDiag			65.4%[57.5–72.5]	96.1%[91.5–98.4]	68.0%[60.2–74.9]	72.5%[65.0–79.0]
StrongStep				62.8%[54.9–70.0]	60.8%[52.9–68.2]	57.5%[49.6–65.1]
Dynamiker					69.3%[61.6–76.1]	73.9%[66.4–80.2]
Multi-G						81.0%[74.1–86.5]

The average dynamic trend to seropositivity for IgM antibodies was not shorter than for IgG antibodies (Fig. 1 and Supplementary Material Fig. S1). The dynamic trend to seropositivity for IgG followed the same pattern for all seven LFAs and the Euroimmun IgG assay, but the trends for the different LFAs varied strongly for IgM.

Fig. 1

Dynamic trend to seropositivity for IgM and for IgG for the different assays in 154 samples from 86 patients. This graph represents the cumulative positivity rate after onset of symptoms in patients with COVID-19. Of note, the average time to seroconversion in this figure lags behind the true time of serconversion by a couple of days since patients were not tested daily and a patient is only considered to have seroconverted after the first positive result. Eighteen samples from days 0–4 are included in the analysis, but not shown on the graph.

Diagnostic performance of IgG LFAs and ELISA 14–25 days after the onset of symptoms

The sensitivity of all seven IgG LFAs was >92.1% (78.5–98.0) and for four IgG LFAs even ≥97.4% (85.3–100) in samples taken 14–25 days after the onset of symptoms. Moreover, in this time window, all seven IgG LFAs had an LR+ ≥10. The sensitivity of the IgG ELISA 14–25 days after the onset of symptoms—89.5% (75.3–96.4)—was lower than that of the seven IgG LFAs, although the difference did not reach statistical significance. This can be attributed to a slower time to seroconversion for the ELISA (Fig. 1). Between day 3 and day 17 after the onset of symptoms, nine patients tested negative with the Euroimmun IgG ELISA but positive with all seven LFAs. The six samples tested days 18–25 after the onset of symptoms were positive for IgG with all assays including the Euroimmun IgG ELISA.

Diagnostic performance of LFAs at the time of admission to the hospital

In the 63 diagnostic samples, sensitivity ranged from 7.9% (3.1–17.7) to 46.0% (34.3–58.2) for IgM and from 25.4% (16.2–37.4) to 39.7% (28.5–52.0) for IgG. The sensitivity of LFAs for IgM OR IgG was higher but did not reach 60% for any test. Furthermore, when only the two assays with an LR+ ≥10 for IgM OR IgG were considered (VivaDiag and StrongStep), the sensitivities at the time of admission were only 30.2% (20.2–42.4) and 31.7% (21.5–44.1), respectively (Table 3 ).

Table 3

Diagnostic performance of lateral flow assays (LFAs) at time of admission to the hospital (63 samples from 63 patients)

Sensitivity (LR+) [95%CI]	Clungene	OrientGene	VivaDiag	StrongStep	Dynamiker	Multi-G	Prima
IgM	17.5% (2.0)[9.9–28.8]	46.0% (9.5)[34.3–58.2]	30.2% (+∞)[20.2–42.4]	7.9% (8)[3.1–17.7]	36.5% (4.2)[25.7–48.9]	36.5% (4.2)[25.7–48.9]	44.4% (6.5)[32.8–56.7]
IgG	25.4% (13)[16.2–37.4]	33.3% (4.9)[22.2–44.4]	27.0% (27)[17.5–39.1]	30.2% (31)[20.2–42.4]	25.4% (26)[16.2–37.4]	30.2% (10)[20.2–42.4]	39.7% (4.1)[28.5–52.0]
IgM OR IgG	30.2% (3.1)[20.2–42.4]	50.8% (5.8)[38.8–62.7]	30.2% (30)[20.2–42.4]	31.7% (16)[21.5–44.1]	36.5% (7.5)[25.7–48.9]	42.9% (3.7)[31.4–55.2]	57.1% (3.9)[44.9–68.6]
IgM AND IgG	12.7% (13)[6.3–23.4]	28.6% (9.8)[18.8–40.8]	27.0% (+∞)[17.5–39.1]	6.3% (+∞)[2.1–15.7]	25.4% (26)[16.2–37.4]	23.8% (+∞)[14.9–35.7]	27.0% (14)[17.5–39.1]

Discussion

The sensitivities of the seven LFAs included in our study for IgG were at least as good as the first CE-marked IgG ELISA during the first 3 weeks after the onset of symptoms, with a faster seroconversion for IgG LFAs. Seropositivity was >92% with all seven IgG LFAs 14–25 days after the onset of symptoms. The specificity for IgG was >97% for five of the seven LFAs, which can be considered very good given the challenging nature of the control samples used in our evaluation. The performance of the IgM LFAs, however, varied greatly, and the average dynamic trend to seropositivity was not shorter than that for IgG. For the LFAs, including IgM also did not improve the diagnostic performance. The low specificity of the IgA ELISA has since been confirmed by the manufacturer who now recommends not using the IgA ELISA for screening of asymptomatic persons.

Initial reports suggested that IgM antibodies against SARS-Cov-2 might appear earlier than IgGs, and that measuring both IgM and IgG would improve the diagnosis of SARS-Cov-2 infection [1,10]. To et al., however, found that more patients had earlier seroconversion for IgG than for IgM. In addition, they also found a 100% seroconversion for IgG antibodies, but not for IgM, 14 days after the onset of symptoms in 16 patients for whom serial serum samples were available [3]. Recently, Long et al. reported 100% seroconversion for IgG after 19 days [11]. Our results confirm these observations in a group of more than 80 patients and suggest that the antibody response to SARS-CoV-2 might be comparable to the response to SARS-CoV-1 where the three antibodiy classes IgA, IgG and IgM seroconverted simultaneously, or even 1 day earlier for IgG [12].

Combining the results of IgG LFAs and IgM LFAs did not improve the diagnostic performance, questioning the rationale for measuring IgM anti-SARS-CoV-2 antibodies. The fact that the specificity of two of the seven LFAs was <90% for IgM OR IgG (either one positive) could explain concerns that have been raised regarding the specificity of LFAs. Concerns regarding the sensitivity of LFAs might be attributable to the fact that these assays have been used in the emergency department. Zhao et al. claimed that antibody detection (using ELISA) could be used as a diagnostic test complementary to PCR, even in patients presenting in the first week from the onset of symptoms [13]. Antibody testing with LFAs at the time of admission could also be useful in resource-limited countries where PCR is not readily available. However, the diagnostic performance at the time of admission in our study was not very good when both sensitivity and specificity, expressed as LR+, were taken into account. The two LFAs IgM OR IgG with an LR+ ≥10 at the time of admission had a sensitivity of only 30.2% and 31.7%.

The low sensitivity at the time of admission in our study is not surprising given that the median time of admission in our study was 7 days after the onset of symptoms and seroconversion typically occurs 7–14 days after the onset of symptoms [3]. Our results also confirm a recent report by Cassaniti et al. who did not recommend the use of a SARS-CoV-2 IgM/IgG LFA for detection of COVID-19 in patients presenting at the emergency department, stating a sensitivity of <20% in this patient population [14]. The discussion about whether or not IgM/IgG LFAs should be used in the emergency department raises the question about the intended use of IgM/IgG LFAs for the detection of antibodies against SARS-CoV-2. Despite that all seven of the tested assays had a CE mark, none of the assays included information about the intended clinical use other than that the assays are for the detection of antibodies against SARS-CoV-2. Such a vague intended use, which might have contributed to the current discussion about the diagnostic performance of LFA, will no longer be accepted for CE marked after May 2022 when the IVD (In Vitro Diagnostics) regulation 2017/746 enters into force. One of the new requirements of the IVD regulation is that manufacturers will be required to document the clinical evidence and the clinical benefit.

This study is to our knowledge the first peer-reviewed study to compare the diagnostic performance and time to seropositivity of a series of LFAs with ELISA. A strength of our study is that we evaluated the diagnostic performance using a set of 103 selected samples for specificity and 167 samples from 94 COVID-19 patients for sensitivity and dynamic trend to seropositivity. Most peer-reviewed studies evaluating the diagnostic performance of antibody tests used a limited number of samples, and many studies did not include samples from patients with a respiratory infection including non-SARS-CoV-2 coronaviruses for specificity. Another strength of our study is that we investigated the added value of measuring IgM with LFA.

There are some limitations to our study. First, our control group included only a limited number of samples from patients with frequent respiratory infections such as influenza, Mycoplasma pneumoniae, and Chlamydia pneumoniae. A second limitation is that the samples used to evaluate specificity were challenging, and that specificity in a routine laboratory setting will most likely be higher. A third limitation is that we did not study the antibody response in asymptomatic persons or mild patients.

The main expected use of antibody testing in the coming months is to confirm past COVID-19 in patients, to determine (herd) immunity, and in epidemiological studies [15]. Our results suggest that detection of IgG antibodies can be very useful if performed at least 18 days after onset of symptoms or, in asymptomatic persons, after the end of an outbreak. There is currently no clear evidence that measuring IgA or IgM is useful. Our results even suggest that it might be better not to measure IgM or IgA since this could result in a significant number of false-positive results without a significant gain in diagnostic performance. Important questions remain regarding the use of antibody testing for epidemiological purposes. Can someone be colonized with SARS-CoV-2 without developing IgG antibodies? In this case, would this person be protected against reinfection? Finally, it is still not clear whether IgG antibodies are protective against reinfection [16].

Conclusions

We found that the sensitivity for the detection of IgG antibodies 14–25 days after the onset of symptoms was >92% for all seven LFAs compared to 89.5% for the IgG ELISA. Five LFAs even had a sensitivity and specificity of ≥94.7%. The average time to seropositivity for IgM was not shorter than for IgG, and including IgM antibodies for LFA resulted in a decrease in specificity without a gain in diagnostic performance in all the assays except for one (VivaDiag). Our results suggest that the development of LFAs that measure only IgG is warranted to avoid false-positive results for IgM.

Author contributions

PV devised the study, collected data and drafted the manuscript, JVE collected data and drafted the manuscript, all other authors aided in collecting data and critically reviewing the manuscript.

Transparency declaration

PV reports personal fees from , outside the submitted work, and is a senior clinical investigator of the FWO-Vlaanderen. KL reports personal fees and non-financial support from , personal fees and non-financial support from MSD, personal fees from SMB Laboratoires, personal fees from Gilead, and personal fees from FUJIFILM Wako, outside the submitted work. The other authors state no conflicts of interests. The research did not receive any specific grant from funding agencies in the public, commercial or not-for-profit sectors.