Analytical performances of five SARS-CoV-2 whole-blood finger-stick IgG-IgM combined antibody rapid tests.

Facing the ongoing pandemic caused by SARS-CoV-2, there is an urgent need for serological assays identifying individuals previously infected by coronavirus disease 2019 (COVID-19), including rapid diagnostic tests (RDTs). We herein compared five new CE-IVD-labeled commercially available SARS-CoV-2 whole-blood finger-stick IgG/IgM combined RDTs, in parallel according to the manufacturers' instructions, with two serum panels obtained from 48 patients with confirmed COVID-19 (panel I) and from a group of 52 patients randomly selected, for whom serum samples collected before the COVID-19 epidemic (from October 1 to November 30, 2019) were negative for SARS-CoV-2 IgG (panel II). We found a sensitivity of 95.8 %, 91.6 %, 92.3 %, 97.9 % and 91.4 %, and a specificity of 98.1 %, 86.5 %, 100 %, 98.1 % and 84.6 %, for BIOSYNEX COVID-19 BSS (IgG/IgM) (Biosynex Swiss SA, Freiburg, Switzerland), Humasis COVID-19 IgG/IgM Test (Humasis Co., Ltd., Gyneonggi, Republic of Korea), LYHER COVID-19 IgM/IgG Rapid Test (Medakit Ltd, Hong Kong, China), SIENNA™ COVID-19 (IgG/IgM) Rapid Test Cassette (Salofa Oy, Salo, Finland) and NG-BIOTECH COVID-19 (IgG/IgM) (NG-Biotech, Guipry, France), respectively. Commercially available SARS-CoV-2 IgG/IgM combined RDTs have a sufficient sensitivity for identifying individuals with past SARS-CoV-2 infection, but some RDTs may lack of specificity, with risk of false positivity mainly for the IgM band.

Introduction

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), a novel coronavirus that causes Coronavirus Disease 2019 (COVID-19), started in Wuhan province, China, in December 2019. It was declared by the World Health Organization (WHO) as global pandemic on March 11th, 2020 (WHO, 2020a). SARS-CoV-2 infection has caused a worldwide viral pandemic, with > 2,046,876 deaths and > 95,918,344 cases reported internationally, with France reporting 2,914,725 cases and 70,686 deaths, as of 18th January 2021(Worldometer, 2021).

Control of the outbreak in both community and hospital setting has mainly relied on the availability of highly sensitive and specific nucleic acid amplification-based molecular testing for SARS-CoV-2 (Péré et al., 2020; Wölfel et al., 2020). Furthermore, it was demonstrated that serological testing looking for specific SARS-CoV-2 IgG and/or IgM antibodies could be useful for confirming the diagnosis and care of COVID-19 (Amanat et al., 2020; Long et al., 2020; Petherick, 2020). On March 2nd, 2020, the WHO recommended serological testing in addition to molecular diagnosis, for investigating on-going outbreaks as well as for the diagnosis of strongly suspected patients of SARS-CoV-2 infection with a negative molecular test for SARS-CoV-2 RNA (WHO, 2020b). Otherwise, antibody tests for SARS-CoV-2 could constitute one of the keys to fight the SARS-CoV-2 epidemic, in particular to surpass the deconfinement period (Petherick et al., 2020). Seropositivity to SARS-CoV-2 antigens would also allow the identification of previously infected individuals, including asymptomatic patients, a priori considered to be healed and protected against new reinfection (Petherick et al., 2020), although this use remains controversial (Bélec et al., 2020).

Recently, rapid diagnostic tests (RDTs) for IgG/IgM antibodies produced during COVID-19 have been developed (Abduljalil, 2020; La Marca et al., 2020), and reports have shown that COVID-19 IgG/IgM lateral flow immunoassays may be a reliable tool to diagnose SARS-CoV-2 infection from 14 days following the onset of symptoms (Candel et al., 2020; Charlton et al., 2020; Dellière et al., 2020; Montesinos et al., 2020; Nicol et al., 2020; Pérez-García et al., 2020).

In some countries, rapid diagnostic testing for COVID-19 has been incorporated into the current local guidelines for testing asymptomatic contacts of positive cases, at day 13 of home surveillance (Zainol Rashid et al., 2020). These easy to use IgG/IgM combined test kits allow rapid screening with capillary blood sample. The tests are simple, qualitative, visually interpretable, and give a result within 10−15 min. A positive serology allows determining whether a person has already been infected by SARS-CoV-2. Serologic tests will be needed to assess the immune response of vaccine candidates and to map levels of immunity in communities. These RDTs are particularly interesting for low resource settings, such as at patients’ bedside or at any other locations where laboratory facilities are lacking.

The analytical performances of RDTs for both SARS-CoV-2-specific IgG and IgM detection should be investigated before use on local serum panels, as recommended for other rapid tests, such as HIV RDTs (WHO, 2012, 2015). We herein investigated and compared the analytical performances of 5 CE IVD-labeled immunochromatographic RDTs for SARS-CoV-2-specific IgG and IgM detection using serum panels from patients with COVID-19 hospitalized during the SARS-CoV-2 infection outbreak in last March 2020 and from patients hospitalized in Paris, France, collected prior the 2019 COVID-19 outbreak.

Material and methods

Study population

Two groups of patients were included in the study.

Group 1 (confirmed cases of SARS-CoV-2 infection)

A panel of 48 serum samples was prospectively constituted during the epidemic period of SARS-CoV-2 infection in France between March and April 2020 from inpatients (30 males; 18 females, mean age, 62.3 years) of Hôpital Européen Georges Pompidou, Paris, France, suffering from COVID-19 with positive detection of SARS-CoV-2 from a nasal swab by molecular diagnosis (Allplex™ 2019-nCoV Assay, Seegene, Seoul, Korea). Samples were collected at least 4 weeks after the onset of clinical signs suggestive of COVID-19 (mean 36.5 days). All sera were furthermore found positive for SARS-CoV-2 IgG by CE-IVD Abbott SARS-CoV-2 IgG assay (Abbott GmbH, Rungis, France) detecting IgG against SARS-CoV-2 nucleoprotein used on Architect analyzer (Abbott Architect™ i2000), according to manufacturer's instructions. This panel was considered as “true positive sera”.

Group 2 (true negative controls)

A panel of 52 serum samples randomly selected from patients (37 males; 15 females, mean age, 59.4 years) who had a serum sample taken for other serologic studies, from September 1st to November 30th, 2019 (Prior to the first cases of COVID-19 were reported in France). All sera were negative for Abbott SARS-CoV-2 IgG reference serology. Thirteen had previous pulmonary infection with endemic coronavirus [n = 12; 229E (n = 2), OC43 (n = 4), NL63 (n = 6)] or metapneumovirus (n = 1) detected by the real-time Allplex™ Respiratory Panel RT-PCR assay (Seegene). Eighteen showed positive for malaria serology. Nineteen had IgG against cytomegalovirus (CMV). This panel was considered as “true negative sera”.

Rapid diagnostic tests

The following commercially available CE IVD-labeled finger-stick whole-blood RDTs for IgG and IgM antibodies to SARS-CoV-2 detection were used in the study:

BIOSYNEX® COVID-19 BSS (IgG/IgM) (Biosynex Swiss SA, Freiburg, Switzerland), targeting the receptor-binding domain (RBD) of the spike surface protein of SARS-CoV-2,

Humasis COVID-19 IgG/IgM Antibody Test (Humasis Co, Ltd., Republic of Korea, manufactured by MT Promedt Consulting GmbH, Saint-Ingbert, Germany),

LYHER COVID-19 IgM/IgG Rapid Test (Medakit Ltd, Hong Kong, China),

SIENNA™ COVID-19 IgG/IgM Rapid Test Cassette (Salofa Oy, Salo, Finland, manufactured by T & D Diagnostics Canada Pvt, Ltd., Halifax, Novia Scotia, Canada),

NG-Test® IgG-IgM COVID-19 (NG BIOTECH laboratories, Guipry, France).

The nature of antigens used is only indicated in the instructions for use of BIOSYNEX® COVID-19 BSS (IgG/IgM).

All assays were performed following the instructions of the respective manufacturers, with 10 μL of serum sample, and read after 10 min (BIOSYNEX® COVID-19 BSS (IgG/IgM), LYHER COVID-19 IgM/IgG Rapid Test and SIENNA™ COVID-19 IgG/IgM Rapid Test Cassette) or 15 min (Humasis COVID-19 IgG/IgM Antibody Test and NG-Test® IgG-IgM COVID-19) by two clinical microbiologists blinded regarding the sample groups. Indeterminate readings were further read by a third microbiologist. The intensities of IgG and IgM bands were measured by eye from 1 to 3 crosses.

In addition to their CE IVD mark, the BIOSYNEX® COVID-19 BSS (IgG/IgM), Humasis COVID-19 IgG/IgM Antibody Test, LYHER COVID-19 IgM/IgG Rapid Test and NG-Test® IgG-IgM COVID-19 are approved for medical use by the French Ministry of Health (Ministère des Solidarités et de la Santé, Paris, France; https://covid-19.sante.gouv.fr/tests). However, for the tests Humasis COVID-19 IgG/IgM Antibody Test and NG-Test® IgG-IgM COVID-19, it is stipulated that “this rapid test cannot be interpreted for the detection of IgM” (Ministère des Solidarités et de la Santé, Paris, France).

Statistical analysis

All data were entered into an Excel file and analyzed on SPSS 20.0 (Chicago, IL). Test results in group 1 were considered as “true positive” (TP) in case of positive test result and “false negative” (FN) in case of negative test result. Test results in group 2 were considered as “true negative” (TN) in case of negative test results and “false positive” (FP) in case of positive test result. The sensitivity and specificity were calculated for each serologic test using the results from group 1 and group 2 patients, respectively.

First, the percent agreement was estimated to measure crude agreement between the test results and the expected results without taking into account the agreement due solely to chance. Next, the kappa concordance, using the Cohen’s κ coefficient formula (Cohen, 1960), was calculated to measure agreement by taking into account the agreement due to chance. Thus, the degree of agreement was determined as ranked by Landlis and Koch (Landlis and Koch, 1977). The accuracy of each serologic test was estimated by Youden’s J index (J = sensitivity + specificity − 1) (Youden, 1950). The results were presented as a 95 % confidence interval (CI) using the Wilson score bounds (Newcombe, 1998).

Although there is no international consensus on the minimum sensitivity and specificity of RDTs for COVID-19 serological diagnosis, the French technical agency for medical practice and biological tests, the so-called Haute Autorité de Santé (HAS, Saint-Denis, France), has proposed on April 16th, 2020, that analytical performances of such rapid tests should demonstrate minimum clinical sensitivity of 90 % (or even 95 %) and clinical specificity of 98 % (Haute Autorité de Santé, 2020).

Ethical statement

Our non-interventional study was carried out in accordance with the Declaration of Helsinki with no additional sampling to usual procedures. Serum sample specimens were obtained in France only for standard diagnostic following medical prescriptions and care. Under these conditions, the study was exempt from informed consent application, according to the French public health code (Code de la Santé Publique, article L 1121−1.1; https://www.legifrance.gouv.fr/). Data analyses were carried out using anonymized database.

Results

IgG and IgM reactivity of rapid diagnostic tests

The results of IgG and IgM reactivity of the 5 study RDTs for SARS-CoV-2-specific antibodies detection by using the per-epidemic serum panel positive for SARS-CoV-2 IgG and the pre-epidemic serum panel are depicted in Table 1 .

Table 1

Analytical performances of 5 rapid diagnostic tests for IgG and IgM to SARS-CoV-2.

BIOSYNEX COVID-19 BSS (IgG/IgM)					Humasis COVID-19IgG/IgM Test					LYHER COVID-19 IgM/IgGRapid Test					SIENNA (IgG/IgM)					NG-BIOTECH COVID-19 (IgG/IgM)
Results	TP	FP	TN	FN	Results	TP	FP	TN	FN	Results	TP	FP	TN	FN	Results	TP	FP	TN	FN	Results	TP	FP	TN	FN
N = 100	46	1	51	2	N = 100	44	7	45	4	N = 50a	24	0	24	2	N = 100	47	1	51	1	N = 99a	43	8	44	4
Sensitivity IgG/IgM	95.8 %		[90.2−100.0%]*		Sensitivity IgG/IgM	91.6 %		[83.8−99.5%]		Sensitivity IgG/IgM	92.3 %		[82.1−100.0%]		Sensitivity IgG/IgM	97.9 %		[97.9−100.0%]		Sensitivity IgG/IgM	91.4 %		[83.5−99.5%]
SpecificityIgG/IgM	98.1 %		[94.3 %–100.0 %]*		SpecificityIgG/IgM	86.5 %		[77.3−95.8%]		SpecificityIgG/IgM	100.0 %		[79.6−100.0%]		SpecificityIgG/IgM	98.1 %		[87.9 %–100.0 %]		SpecificityIgG/IgM	84.6 %		[74.8−94.4%]
Agreement§	97 %		Excellent		Agreement	89 %		Good		Agreement	96 %		Excellent		Agreement	98 %		Excellent		Agreement	87 %		Good
Concordance$	0.94		Almost perfect		Concordance	0.78		Substantial		Concordance	0.92		Almostperfect		Concordance	0.95		Almost perfect		Concordance	0.75		Substantial
Accuracy£	93.9 %		Excellent		Accuracy	78.1 %		Acceptable		Accuracy	92.3 %		Excellent		Accuracy	96.0 %		Excellent		Accuracy	76.0 %		Acceptable
HAScriteriaμ	Yes				HAS criteria	No				HAScriteria	Yes				HAScriteria	Yes				HAScriteria	No

FN: False negative; FP: False positive; HAS: Haute Autorité de Santé (High Authority of Health); T: Total; TN: True negative; TP: True positive; WHO: World Health Organization.

The availability of the LYHER COVID-19 IgM/IgG Rapid Test was limited and 1 out of 2 sera were randomly selected to be tested by this test; the NG-BIOTECH COVID-19 (IgG/IgM) was tested with only 47 sera of group 1, because the quantity of 1 serum of this group was insufficient.

95 % confidence intervals in brackets were calculated using the Wilson score bounds.

Agreement = TP + TN / TP + FP + TN + FN; agreement in expressed in percentage.

The Cohen’s κ coefficient calculation was used to estimate the concordance (Cohen, 1960) and interpreted according the Landis and Koch scale (Landis & Koch, 1977) as follows: < 0 as indicating no agreement, 0–0.20 as slight, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1 as almost perfect agreement.

Accuracy was estimated by Youden’s J index = sensitivity + specificity – 1; accuracy is expressed in percentage.

HAS criteria: According to the specifications defining the methods for evaluating the performance of serological tests detecting the antibodies directed against SARS-CoV-2 of April 16, 2020 by the so-called Haute Autorité de Santé (HAS, Saint-Denis, France; https://www.has-sante.fr/upload/docs/application/pdf/2020-04/cahier_des_charges_test_serologique_covid19.pdf), the minimum clinical sensitivity of a serological test must be 90 % (or even 95 %) and its minimum specificity 98 %.

All false positive results were due to weak IgM reactivity at 1 cross. There was no IgG reactivity in our series.

Interestingly, the IgM reactivity was associated with other infections for Humasis COVID-19 IgG/IgM Antibody Test [5 of 7 (71.4 %): seropositivity for CMV (n = 2) and malaria (n = 1), infection with OC43 coronavirus (n = 1) and OC43 coronavirus and metapneumovirus (n = 1)] and NG-BIOTECH COVID-19 (IgG/IgM) [5 of 8 (62.5 %): seropositivity for CMV (n = 1) and malaria (n = 2), infection with NL63 coronavirus (n = 1) and OC43 coronavirus and metapneumovirus (n = 1)], respectively.

Analytical performances

Analytical performances are shown in the Table 1.

We found a sensitivity of 95.8 %, 91.6 %, 92.3 %, 97.9 % and 91.4 %, and a specificity of 98.1 %, 86.5 %, 100 %, 98.1 % and 84.6 %, for BIOSYNEX COVID-19 BSS (IgG/IgM), Humasis COVID-19 IgG/IgM Test, LYHER COVID-19 IgM/IgG Rapid Test, SIENNA™ COVID-19 (IgG/IgM) Rapid Test Cassette and NG-BIOTECH COVID-19 (IgG/IgM), respectively.

The concordance of the 5 study RDTs were excellent: almost perfect, respectively, for 3 tests, and good and substantial, respectively, for the 2 remaining RDTs. The accuracy to diagnose a “true positive” or a “true negative” serum showed similar distribution, with 3 study RDTs with excellent accuracy, and the 2 remaining with only acceptable accuracy. Finally, the 3 out 5 study RDTs fulfilling the acceptance criteria of the Haute Autorité de Santé, showed the best analytical performances, including sensitivity, specificity, agreement, concordance and accuracy. The CE IVD Humasis COVID-19 IgG/IgM Antibody Test and NG-BIOTECH COVID-19 (IgG/IgM), approved by the French Ministry of Health, did not respect the Haute Autorité de Santé criteria in our hands.

Discussion

In our study, we showed that rapid CE IVD-marked RDTs for the detection of circulating antibodies of the IgG and IgM isotypes against SARS-CoV-2 could harbor weak positive reactivity with sera collected from European patients before the 2019 outbreak of COVID-19, always for the IgM isotype, leading to possible final misinterpretations, i.e. false-positive test results. Taken together, our observations point the risk of false positive reactivity when using currently available RDTs for SARS-CoV-2 serological screening, especially for the IgM band, even if the test is CE IVD-labeled and approved by national health authorities. These findings have important implications for the choice of the best COVID-19 serological assays. Finally, the need for validating the analytical performances of COVID-19 RDTs using local serum panels well documented and mastered by the biologist should be underlined.

COVID-19 infection can be detected indirectly by measuring the host humoral immune response to SARS-CoV-2 infection, SARS-CoV-2-specific IgM indicating acute or recent infection and SARS-CoV-2-specific IgG on-going or past-infection (Sethuraman et al., 2020). Positive reactivity, particularly for the IgM band, and resulting false positive test results were variably observed for the 5 study COVID-19 RDTs used with pre-epidemic sera. These observations are reminiscent of the high rates of unspecific serological reactivity with immunochromatographic RDTs that have been previously reported for various infectious diseases, such as HIV (Aghokeng et al., 2009; Jenabian et al., 2017) or syphilis (Mbopi-Keou et al., 2019). In our study, about 50 % of patients exhibited positive reactivities and evidence of recent infections with common cold coronaviruses or other respiratory virus, or positive results for HIV, HBV, CMV, malaria, or HCV serologies. These associations allow us to hypothesize that a variety of mechanisms related to infectious diseases may be involved to account for false positive IgM results, such as cross-reacting antibodies against common coronaviruses (Woo et al., 2004), or disturbances affecting the B cell-driven immunity during infectious diseases (Mori et al., 1987; Bouvet and Dighiero, 1998; Klarkowski et al., 2014; Mbopi-Keou et al., 2014). However, in the remaining study positive reactivities, no obvious co-infectious could be found, suggesting that other causes of unspecific RDT results could be also involved such as autoimmunity, concurrent infection by other pathogens, interfering substances such as rheumatoid factor and other yet unknown factors (Landry, 2016). It is well recognized that most IgM based serological assays for several infectious diseases diagnosis suffer from higher false positive rates relative to IgG-based assays (Landry, 2016), as in our study COVID-19 RDT.

Diagnostic test performances must be examined within the specific settings where the tests will be used, and this may be relevant to account for possible differences due to environmental and racial factors (Mbopi-Keou et al., 2019).

RDTs for detection of SARS-CoV-2-specific antibodies have been widely developed and are of variable quality. Currently, there is greater than 200 RDTs available for the detection of antibodies against SARS-CoV-2 (Covid-19 - Foundation for Innovative New Diagnostics; available at: https://www.finddx.org/). Many manufacturers do not reveal the nature of antigens used. These tests are purely qualitative in nature and can only indicate the presence or absence of SARS-CoV-2 antibodies. In order to make the testing more available, the principal regulatory agencies such as the Food and Drug Administrations (FDA, US) and the European Center for Disease Prevention and Control have removed demanding requirements for agency review of SARS-CoV-2 assays, providing a double-edged sword, with test not properly vetted and with highly variable performances. All the serological tests in our study have the CE IVD mark and four of them have been approved by the French Ministry of Health, i.e. they have theoretical clinical sensitivity of at least 90 % and specificity of at least 98 % (Haute Autorité de Santé, 2020). However, even for compliant CE-marked RDTs, their performance may vary in the routine testing laboratory in comparison with the performance study of the manufacturer done for the purposes of CE-marking. Among our 5 study RDTs for COVID-19 serology, Humasis COVID-19 IgG/IgM Test and NG-Test® IgG-IgM COVID-19 showed poor analytical performances assessed by high false positive test results rates with the European (13.4 % and 15.4 %, respectively) pre-epidemic sera, in accordance with the restricted use of both tests to IgG, not IgM by the French Ministry of Health. Higher specificity at 95.3 % was previously reported for the NG-Test® IgG-IgM COVID-19, but the difficulty for reading the IgM band of this test was underlined (Nicol et al., 2020).

Among the 5 study RDTs, the tests LYHER COVID-19 IgM/IgG Rapid Test, BIOSYNEX® COVID-19 BSS (IgG/IgM) and SIENNA™ COVID-19 (IgG/IgM) Rapid Test Cassette showed the lowest rates of false positive test results with European pre-epidemic sera (0%, 1.9 %, 1.9 %, respectively). Cautionary opinion on currently available SARS-CoV-2 serological assay was previously expressed (Farnsworth and Anderson, 2020), despite the excitement of the lay public emphasizing that the deployment of insufficiently validated assays, particularly inaccurate ones, could have negative consequences for public health.

Our study has some limitations. In particular, the number of blood samples is limited. Thus, the number of negative specimens to be analyzed should be sufficiently high, especially if the prevalence of COVID-19 in the community is low and if the risk of false positive results is low, as for example for specific antibodies of the IgG isotype (Farnsworth and Anderson, 2020). In addition, the positive sera from group 1 were collected from patients at least 4 weeks after the onset of clinical signs, late in the kinetic infection and probably containing only SARS-CoV-2-specific IgG. Thus, we can consider that the study results refer exclusively to a specific phase of the infection (at least 4 weeks after the beginning of SARSD-CoV-2 infection), not allowing sufficient evaluation of the analytical parameters of the IgM band.

In practice, clinical validation of the diagnostic performance of RDTs for COVID-19 serology in real-life should be always carried out in a sufficiently large number of target population subjects before introducing them into the routine diagnosis, as strongly recommended by the European Centre for Disease Prevention and Control (ECDC, 2020). Our study in a hospital setting confirms the need for COVID-19 RDTs to undergo validation on the field in the specific environments in which they will be deployed, with local serum panel corresponding to given environments and human habitats.

In conclusion, the immunochromatographic RDTs for SARS-CoV-2 serological screening are simple, cheap and fast. They do not require qualified personnel for interpretation and could be done in hospitals as well as primary care facilities. However, in view of the great profusion of COVID-19 RDTs on the market, it is important to evaluate them locally before use, in order to confirm their analytical performances, independently of those asserted by the manufacturer.

Funding

No grant was received from the test manufacturers.

Author statement

The authors of this paper have no financial or personal relationship with other people or organizations that could inappropriately influence or bias the content of this paper.

Contributions

IP, HP, DV and LB have conceived and designed the research; RSMB performed the experiments; STW performed statistical analyses; HP, DV, LB and RSMB analyzed the results and drafted the manuscript.

Declaration of Competing Interest

The authors report no conflicts of interest. The authors alone are responsible for the content and the writing of the paper.