Evaluation of three immunochromatographic tests for rapid detection of antibodies against SARS-CoV-2.

Lateral flow immunoassays (LFIA) for rapid detection of specific antibodies (IgM and IgG) against SARS-CoV-2 in different human specimens have been developed in response to the pandemic. The aim of this study is to evaluate three immunocromathographic assays (Sienna®, Wondfo® and Prometheus®) for detection of antibodies against SARS-CoV-2 in serum samples, considering RT-qPCR as a reference. A total of 145 serum samples from 145 patients with clinical suspicion of COVID-19 were collected: all of the samples were tested with Sienna®, 117 with Wondfo® and 89 with Prometheus®. The overall results of sensitivity, specificity, positive predictive value and negative predictive value obtained were as follows: 64.4%, 75%, 85.5% and 47.8% with Sienna®; 45.2%, 81.8%, 80.5% and 47.4% with Wondfo® and 75.5%, 12.5%, 51.4% and 29.4% with Prometheus®. The accuracy of the test for Sienna®, Wondfo® and Prometheus® was 67.6%, 59% and 47.2%, with a prevalence of COVID-19 of 69.7%, 62.4% and 55.1% respectively. Sensitivity of the three tests (Sienna®, Wondfo® and Prometheus® respectively) along the three different stages was 36.6%, 18.8% and 68.6% in the early stage (first week); 81.3%, 74.1% and 90.9% in the intermediate stage (second week) and 100%, 83.3% and 100% in the late stage (third week). The results demonstrate that even though Prometheus® presented a high sensitivity, the specificity was notably lower than the other two tests. Sienna® showed the greatest contrast between sensitivity and specificity, achieving the best accuracy, followed by Wondfo®. The sensitivity of the three ICT assays was higher in late stages of the disease.

Introduction

In late December 2019, a novel coronavirus was identified as the etiological agent of anew pneumonia [1, 2]. The etiological agent, named as SARS-CoV-2, rapidly spread to other cities in China and to other countries worldwide and on 11 March World Health Organization (WHO) declared the outbreak as a pandemic. This situation has forced many countries to adopt firm measures in order to promote early detection of COVID-19 and early isolation of the cases, track contacts and encourage distancing measures.

Real-time reverse transcription quantitative polymerase chain reaction (RT-qPCR) has been established as the gold standard for microbiological diagnostic of SARS-CoV-2 infection, targeting at least two different regions of SARS-Cov-2 genome in order to avoid cross-reactivity with other coronavirus and the potential genetic drift of SARS-CoV-2 [3-6]. RT-qPCR tests presented a high specificity with a low probability of false positive; however, sensitivity relies on different factors as specimen site, method of collection, viral load and time from the onset of symptoms [3, 7]. An increasing number of cases with negative RT-qPCR and clinical features consistent with COVID-19 pneumonia has been reported [8-10].Therefore, supplementary diagnostic approaches are needed to reduce the number of false-negative cases, which is essential for the epidemiologic control of the disease [11].

Several studies focused on antibody response against SARS-CoV-2 suggested that IgM can be detected during the first week since the onset of the symptoms, although the IgM detection rate is highly variable at this early stage [12-14]. IgG can be detectable after 8 days since the onset of the symptoms, and after 14 days, over 90% of cases present antibodies against SARS-CoV-2 [13-15]. However, the strength of antibody response depends on several factors (nutritional state, severity of disease, immune status...) and it has been observed that some patients do not produce detectable levels of antibodies [13, 15, 16].

Therefore, serological methods had limited utility for early diagnosis of COVID-19, since the sensitivity of the assay is low during the first days and increases with time. However, serologic tests could play an important role in confirmation and late diagnostic of COVID-19, mainly in patients with repetitive negative RT-qPCR, as well as giving information about the immune status of asymptomatic patients and contributing to the determination of the prevalence and mortality rate [17]. Combination of RT-qPCR and IgM/IgG detection methods could provide a suitable approach to COVID-19 diagnosis [18].

Several studies suggested that acquired immunity is protective against SARS-CoV2 re-infection [19, 20], though some reports described cases of post-recovery positive nasopharyngeal RT-qPCR [21-23]. Currently, is being discussed if the recurrence of positive SARS-CoV-2 RT-qPCR in recovered patients with subsequent negative RT-qPCR is due to prolonged viral shedding with false negative results of RT-qPCR, or a possible re-infection [21, 24, 25]. In this context, the use of serologic test for the follow-up of the immune response in those cases may be useful to a better understanding of the acquired immunity and the possibility of re-infection.

Lateral flow immunoassays (LFIA) for rapid detection of specific antibodies (IgM and IgG) against SARS-CoV-2 in different human specimens (whole blood, serum and plasma) have been developed in response to the pandemic [18, 26]. This technique is quick and simple to perform and does not require special equipment. In addition, some studies have shown good concordance between the results obtained using whole blood samples (fingerstick blood) and serum or plasma samples so can be used as point-of-care (POC) immunodiagnostic tests, which makes it suitable for community surveillance [18, 26]. The use of these POC immunodiagnostic tests for massive detection of antibody response to SARS-Cov-2 in the population would be an important step towards a better understanding of the outbreak’s situation, which would help us to establish the right policies in the control of the COVID-19 pandemic [27]. However, the utility of these LFIA assays is questionable due to the lack of official performance validation to evaluate the sensitivity and specificity of the tests, which are highly variable between the different commercially available serologic tests [15, 28].Therefore, further studies are needed to assess the sensitivity and specificity of the serologic test launched to the international market.

The aim of this study is to evaluate three immunocromathographic assays for IgM and IgG detection of COVID-19 in serum samples.

Material and methods

Immunochromathographic strip assay

Three different immunochromathographic (ICT) assays for qualitative detection of IgG and IgM antibodies against SARS-CoV-2 were performed in accordance with the manufacturer’s protocols for each device. The kits evaluated were as follows:

Wondfo®, SARS-CoV-2 Antibody Test (Lateral flow method). (Luogang District, Guangzhou, China).

T&D Diagnostics, Sienna®, 2019-nCoV IgG/IgM Rapid Test Cassette (Halifax, Nova Scotia, Canada).

Prometheus® Bio Inc., 2019-nCoV IgG/IgM Test Cassette (Zhejiang, China).

In summary, for the three of them, 10 μl of serum and 2 drops of buffer solution were added to the corresponding loading area in the cassettes. After a short time (no longer than 20 min), the interpretation of the results was done by two observers, based on the appearance of a coloured band. According to manufacturer’s instructions, the appearance of a blurred band was considered as a positive result (weak positives).

Sienna® and Prometheus® kits show IgM and IgG bands separately, while Wondfo® detected total antibodies in one band.

RT-qPCR

Nasopharyngeal swabs were collected, and RNA was extracted using an automated system and analysed using selected RT-qPCR commercial kits routinely used for diagnosis of COVID-19, which are addressed to detect the common targets of SARS-CoV-2: nucleocapside (N), spike (S), envelope (E), Orf1ab and RdRp genes.

RT-qPCR has been considered the reference for the evaluation of the ICT strip assays.

Samples and patients

Serum samples were collected between the 8th of March and the 2nd of April at Hospital Universitario La Paz (Madrid, Spain), and were centrifuged at 3000 rpm during 10 min.

The evaluation was performed in two groups of patients. The first group was integrated by 95 healthcare workers at Hospital Universitario La Paz, who attended the occupational health consultation for the first time between the 24th March and the 2nd of April referring symptoms compatible with COVID-19. The second group included 50 patients randomly selected who were admitted to the Emergency Department of the Hospital with positive RT-qPCR or high clinical suspicion of COVID-19. In the first group of patients, serum samples and nasopharyngeal swabs were collected at the same time in 82 patients, while in the other 13 patients, the average time since the nasopharyngeal swab collection and the serum extraction was 7.5 days. Regarding the second group, in 48 patients, serum samples were taken days after the swab collection in an average time of 4.3 days, while in two patients, both samples were collected at the same time.

In addition, 20 extra serum samples of randomly selected patients from 2018 were tested as negative control with Prometheus® and Sienna® test. Unfortunately, it was not possible to carry out this evaluation with Wondfo® test due to lack of reagents.

Statistical analysis

The statistical analysis was performed overall and divided in three different stages according to the time since the symptoms onset: early stage (first week), intermediate stage (second week) and late stage (third week).

The statistical test parameters were calculated according to their definitions, with additional plots generated using GraphPad Prism (version 8). The results from the negative controls were excluded from the overall statistical analysis.

Results

A total of 145 serum samples from 145 patients were collected: 89 samples, which belonged to the group of healthcare workers, were tested with the three ICT assays, 28 samples (6 from the first and 22 from the second group of patients) were tested with Sienna® and Wondfo® and the other 28 samples from the second group of patients were tested only with Sienna. Therefore, 145 samples were tested with Sienna®, 117 with Wondfo® and 89 with Prometheus®.

Demographic and clinical characteristics of the patients included in the study are collected in Table 1.

Table 1

Demographic and clinical characteristics of patients included in the study

	Healthcare workers			Patients admitted to the Emergency Department
Characteristics	Sienna® (n = 95)	Wondfo® (n = 95)	Prometheus® (n = 89)	Sienna® (n = 50)	Wondfo® (n = 22)
Female sex, n	74	74	68	23	10
Median age, years (range)	43 (21–79)	43 (21–79)	42 (21–79)	50 (28–98)	72.5 (46–98)
Hospitalized, n	2	2	2	47	19
Pneumonia, n	12	12	12	48	20
Median time between onset of symptoms until serum sample collection, days (range)	5 (1–24)	5 (1–24)	5 (1–21)	11 (3–18)	11 (5–18)
Median time between serum and nasopharyngeal aspirate sample collections, days (range)	0 (0–17)	0 (0–17)	0 (0–7)	4 (0–13)	5 (2–9)
Positive RT-qPCR patients, n	55	55	49	46	18
Negative RT-qPCR patients, n	40	40	40	4	4

The overall results of sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) obtained were as follows: 64.4%, 75%, 85.5% and 47.8% with Sienna®; 45.2%, 81.8%, 80.5% and 47.4% with Wondfo® and 75.5%, 12.5%, 51.4% and 29.4% with Prometheus®. The accuracy of the test was 67.6% with Sienna®, 59% with Wondfo® and 47.2% with Prometheus®, with a prevalence of COVID-19 of 69.7%, 62.4% and 55.1% respectively (Table 2). The results of prevalence, accuracy, sensitivity, specificity, PPV and NPV of Prometheus® and Sienna® for IgG and IgM detection separately are also available in Table 2.

Table 2

Diagnostic test accuracy of the three ICT tests for detection of IgM and IgG independently, and in global

	Sienna® IgM (CI 95%)	Sienna® IgG (CI 95%)	Sienna® global (CI 95%)	Wondfo® (CI 95%)	Prometheus® IgM (CI 95%)	Prometheus® IgG (CI 95%)	Prometheus® global (CI 95%)
Prevalence	0.697 (0.614–0.769)	0.697 (0.614–0.769)	0.697 (0.614–0.769)	0.624 (0.529–0.71)	0.551 (0.442–0.655)	0.551 (0.442–0.655)	0.551 (0.442–0.655)
Accuracy	0.628 (0.543–0.705)	0.614 (0.529–0.692)	0.676 (0.592–0.75)	0.59 (0.495–0.679)	0.449 (0.345–0.558)	0.528 (0.42–0.634)	0.472 (0.366–0.58)
Sensitivity	0.554 (0.452–0.652)	0.515 (0.414–0.615)	0.644 (0.541–0.735)	0.452 (0.337–0.572)	0.673 (0.523–0.796)	0.469 (0.328–0.616)	0.755 (0.608–0.862)
Specificity	0.795 (0.642–0.897)	0.841 (0.693–0.928)	0.75 (0.594–0.863)	0.818 (0.668–0.913)	0.175 (0.079–0.334)	0.6 (0.434–0.747)	0.125 (0.047–0.276)
PPV	0.862 (0.748–0.931)	0.881 (0.765–0.947)	0.855 (0.752–0.922)	0.805 (0.646–0.906)	0.5 (0.376–0.624)	0.59 (0.422–0.74)	0.514 (0.394–0.632)
NPV	0.438 (0.328–0.553)	0.43 (0.325–0.541)	0.478 (0.358–0.601)	0.474 (0.359–0.591)	0.304 (0.141–0.53)	0.48 (0.339–0.624)	0.294 (0.114–0.56)
N	145			117	89

Detection rate of IgM, IgG and IgM/IgG antibodies against SARS-CoV-2 with the three ICT strip assays in positive and negative RT-PCR patients along three periods of time since the onset of symptoms is shown in Table 3.

Table 3

Detection of IgM, IgG and IgM/IgG antibodies against SARS-CoV-2 with the three ICT strip assays in positive and negative RT-PCR patients along three periods of time since the onset of symptoms

			PCR
			Early stage			Intermediate stage			Late stage
			Positive	Negative	Total	Positive	Negative	Total	Positive	Negative	Total
Sienna® (n = 142)	IgM	Positive	13	4	17	33	2	35	8	3	11
		Negative	28	22	50	15	7	22	1	6	7
		Total	41	26	67	48	9	57	9	9	18
	IgG	Positive	11	1	12	33	4	37	7	2	9
		Negative	30	25	55	15	5	20	2	7	9
		Total	41	26	67	48	9	57	9	9	18
	IgM/IgG	Positive	15	4	19	39	4	43	9	3	12
		Negative	26	22	48	9	5	14	0	6	6
		Total	41	26	67	48	9	57	9	9	18
Wondfo® (n = 114)	IgM/IgG	Positive	7	3	10	20	4	24	5	1	6
		Negative	30	23	53	7	5	12	1	8	9
		Total	37	26	63	27	9	36	6	9	15
Prometheus® (n = 88)	IgM	Positive	20	22	42	10	5	15	2	6	8
		Negative	15	3	18	1	2	3	0	2	2
		Total	35	25	60	11	7	18	2	8	10
	IgG	Positive	14	10	24	7	4	11	1	2	3
		Negative	21	15	36	4	3	7	1	6	7
		Total	35	25	60	11	7	18	2	8	10
	IgM/IgG	Positive	24	23	47	10	6	16	2	6	8
		Negative	11	2	13	1	1	2	0	2	2
		Total	35	25	60	11	7	18	2	8	10

When the test was performed in patients with less than 1 week since the symptoms onset, the results obtained with Sienna® (n = 67), Wondfo® (n = 63) and Prometheus (n = 60) respectively were as follows: sensitivity 36.6%, 18.9% and 68.8%; specificity 84.6%, 88.5% and 8%; PPV 78.9%, 70% and 51.1% and NPV 45.8%, 43.4% and 15.4%.

In patients which presented between 7 and 14 days since the onset of symptoms, sensitivity increased to 81.3% with Sienna® (n = 57), 74.1% with Wondfo® (n = 36) and 90.9% with Prometheus (n = 18). Specificity, PPV and NPV were 55.6%, 90.7% and 35.7% respectively with Sienna®; 55.6%, 83.3% and 41.7% with Wondfo® and 14.3%, 62.5% and 50% with Prometheus®.

In patients with more than 14 days since the symptoms onset (n = 18), sensitivity and NPV reached 100%, with Sienna® (n = 18) and Prometheus® (n = 10), and 83.3% and 88.9% with Wondfo® (n = 15). Specificity and PPV were 66.7% and 75% with Sienna®, 88.9% and 83.3% with Wondfo® and 25% and 25% with Prometheus®.

These results of the variation of the test parameters along the three different periods, including IgG and IgM detection separately, are available in supplementary material (Table 1).

Antibody detection rate based on the number of days after onset of symptoms of the three ICT assays is determined and summarized in Figs. 1, 2, and 3.

Fig. 1

Diagnostic test parameters of Sienna® broken down into early, intermediate and late stage

Fig. 2

Diagnostic test parameters of Prometheus® broken down into early, intermediate and late stage

Fig. 3

Diagnostic test parameters of Wondfo® broken down into early, intermediate and late stage

Detection rates of total antibodies (IgM/IgG) obtained with Sienna® and Wondfo® by the two groups of patients along the three stages since the symptoms onset are collected in Table 4.

Table 4

Detection rates of total antibodies (IgM/IgG) with Sienna® and Wondfo® in the two groups of patients among the three different stages

	Healthcare workers		Patients admitted to the Emergency Department
	Sienna® (n = 93)	Wondfo® (n = 93)	Sienna® (n = 49)	Wondfo® (n = 21)
	IgM/IgG	IgM/IgG	IgM/IgG	IgM/IgG
Early stage	14/61 (22.9%)	8/61 (13.1%)	5/6 (83.3%)	2/2 (100%)
Intermediate stage	10/18 (55.5%)	10/18 (55.5%)	33/39 (84.6%)	15/18 (83.3%)
Late stage	8/14 (57.1%)	5/14 (35.7%)	4/4 (100%)	1/1 (100%)
Total	32/93 (34.4%)	23/93 (24.7%)	42/49 (85.7%)	18/21 (85.7%)

The calculation of the different test parameters among the three stages was performed excluding three patients because the time since the onset of symptoms was unknown. The three of them were included in the global evaluation of Sienna® and Wondfo® and one was included in the evaluation of Prometheus®.

The 20 serum samples from 2018 included as negative control of Sienna® and Prometheus® resulted in one positive sample with an IgM band using Sienna® while seven were positive (2 IgG and IgM, 3 IgG and 2 IgM) with Prometheus®.

Discussion

Diagnosis of SARS-CoV-2 remains challenging in most fields. First, COVID-19 patients present a range of unspecific clinical symptoms that are similar to those produced by common respiratory tract pathogens. Second, laboratory diagnosis is currently based on RT-qPCR detection of viral RNA from nasopharyngeal samples, which has limitations of sensitivity, especially in later stages of the disease. Therefore, it is essential to optimize laboratory diagnosis, including serological tests as a complementary technique of RT-qPCR for SARS-CoV-2 diagnosis.

In this study, we have investigated the diagnostic value of detection of SARS-CoV-2 IgM and IgG antibodies in different stages of the disease, using three ICT strip assays, in comparison with RT-qPCR.

It has been described that during the course of the disease, the sensitivity of the RT-qPCR technique tends to decrease in nasopharyngeal samples, due to a reduction of viral load in this location, while antibodies production raise and serological techniques show a remarkable increase of the sensibility [29] .

We found that the most sensitive test was Prometheus®, whose sensitivity was significantly higher during the second and third week of the disease. Sienna® and Wondfo® presented a slightly lower sensitivity which also increased following the same pattern through the evolution of the disease. During the early stages of the disease (first week), the antibody detection rate with the three LFIA test was low, as it has been observed in different published studies [13, 14, 26, 30].

With Prometheus® and Sienna®, IgM positive rate was higher than IgG throughout the 3 weeks. However, we have observed that IgM presented a higher number of false positives, especially in the case of Prometheus test. The most specific test was Wondfo®, followed by Sienna® and last Prometheus®. Both last showed higher specificity in IgG detection.

Considering the high rate of false positives when using Prometheus® test, an extra 20 serum samples of randomly selected patients from 2018 were studied as negative controls and the results confirmed a high percentage of false positive with this assay (35%). The same was performed using Sienna®, obtaining a much lower rate of false positives (5%). Unfortunately, it was not possible to carry out this evaluation with Wondfo® test due to lack of reagents. This indicated the need of further studies to evaluate the specificity and cross-reactions of serological test using negative controls.

These results demonstrate that even though Prometheus® presented a high sensitivity, the specificity was notably lower than the other two tests. Sienna® showed the greatest contrast between sensitivity and specificity, achieving the best accuracy, followed by Wondfo®.

However, there are several limitations in the present study. The current pandemic situation has led to an interrupted availability of the tests, so it has not been possible to carry out the evaluation uniformly with the three kits in the same number of samples. Moreover, the differences between the two groups of patients included in the study may be a potential bias. On one side, in the group of healthcare workers, probably more aware of disease symptoms than the general population, the time elapsed since the onset of symptoms until the attendance to the occupational consultation was considerably lower than in those patients who attended the Emergency Department, which were in more advanced stages of the disease with higher rates of antibodies production. Furthermore, in most healthcare workers, nasopharyngeal swabs and serum samples were collected at the same time and the time between the onset of symptoms and serum sample collection was lower; hence, the sensitivity of serologic tests is reduced. On the other side, patients included in the second group, who were admitted to the Emergency Department, presented a previous positive RT-qPCR or a high clinical suspicion of COVID-19, and consequently the pre-test probability of antibodies production in this group of patients was higher. Finally, the use of RT-qPCR as reference technique for the evaluation of serological test should be taken with care, since both techniques have different time windows for positivity during the course of the infection and RT-qPCR presented a substantial rate of false negatives. In contrast, the low rate of false positives using RT-qPCR enables a consisting evaluation of the sensitivity of serological tests in COVID-19 patients with positive RT-qPCR.

Our study highlights the importance of performing the serological tests when the time since the onset of symptoms is larger than 1 week, since the sensitivity of the three ICT assays was notably higher during the intermediate and late stages of the disease.

(PDF 510 kb)