Performance evaluation of antibody-based point-of-care devices intended for the identification of immune responses to SARS-CoV-2.

The novel coronavirus outbreak caused by the severe acute respiratory syndrome coronavirus (SARS-CoV-2) was first identified in December of 2019 in Wuhan, China. The local outbreak quickly rose to pandemic level that has spread to more than 188 countries with more than 19 million cases and 732,467 deaths worldwide. The current recommendation for testing is RT-PCR based tests of nasopharyngeal or alternatively nasal- and/or oropharyngeal swabs that detects infection with SARS-CoV-2 to diagnose acute infection. However, there is an urgent need for a quick and accurate antibody-based point-of-care test method to quickly identify evidence of SARS-CoV-2 infection among people who might be missed through active case finding and surveillance efforts. Serology tests measure the presence of antibodies in serum after infection. Here we compared the performance characteristics of 6 commercially available antibody-based point-of-care devices and their potential for identification of individuals infected at some time by SARS-CoV-2.

Introduction

There is growing recognition of the importance for serology testing for SARS-CoV-2, not only for seroprevalence purposes, but also to help to guide “return to work” or other COVID-19 exposure decisions. Although most testing is performed in the laboratory environment, there have been a large number of rapid diagnostic serology tests that have been made available (Anand Shah and Jeff Shuren, 2020). Many of these tests have not been reviewed by the Food and Drug Administration (FDA). Several more have been reviewed by the FDA and have received Emergency Use Authorization (EUA), although some of these tests have had their EUA revoked ((FDA) USFDa, 2020). These instances point to the high variability in the performance of these “Point of Care” (POC) devices. Conceptually, POC devices offer many advantages such as the ability to detect IgM and IgG from a very small amount of blood, plasma or serum, and provide results within 10 to 15 minutes. We had the opportunity to evaluate the performance characteristics of a limited number of rapid test devices and compare them to a high complexity test used in our laboratory.

Materials and methods

Serum samples

Studies were performed on sera from deidentified specimens submitted to the Wadsworth Center, New York State Department of Health (NYSDOH) shortly after the first reported COVID-19 case in New York (March 1, 2020). These submitted specimens were from COVID-19 patients, as determined using RT-PCR to detect SARS-CoV-2 RNA. Due to the timing of specimen submission, most sera were “early acute” and not all specimens were antibody positive. These specimens are labeled as “PCR Positive” sera in this study. We also obtained sera from healthy, COVID-19 convalescent patients who were at least 25 days post symptom onset. These specimens were all reactive at various levels in the comparator assay, New York SARS-CoV Microsphere Immunoassay (MIA). The MIA has FDA EUA and its development and performance characteristics are described in (FDA U, 2020). These specimens are labeled as “Positive by MIA” in this study. For specificity and control sera, we used (1) pre-COVID-19 sera purchased, submitted for clinical testing, or gifted to the Wadsworth Center that were characterized by positive identification for known pathogens or autoantigens; (2) sera obtained during the COVID-19 pandemic from individuals experiencing respiratory infections, but with a negative SARS-CoV-2 RT-PCR result (a kind gift from Dr E. Hod, Columbia University Medical Center, NY).

MIA

Specimens were assessed for the presence of total antibody using the New York SARS-CoV MIA. For the present study, the original determination of COVID-19 positivity was based on reactivity to the highly cross-reactive SARS-CoV-1 nucleocapsid protein (N), however subsequent analyses showed that all the MIA reactive specimens had antibodies reactive with the SARS-CoV-2 N protein and all but one specimen also had antibodies reactive with the receptor binding domain of SARS-CoV-2 spike protein. The assay detects total CoV-specific antibodies in human sera using an antihuman immunoglobulin reagent (reactivity to IgG, IgA, and IgM; Life Technologies; Grand Island, NY), but was also modified to separately identify IgG or IgM, where noted. Analysis was performed using a FLEXMAP 3D Analyzer (Luminex Corp.) and the results are provided as median fluorescence intensities (MFI). The specificity and sensitivity of the assay are 99% and 88%–93%, respectively (FDA U, 2020; Yang et al., 2020).

POC testing method

Testing was performed at room temperature (20°–25°C) according to manufacturer's instructions. Briefly, serum (please refer to Table 1 for exact amounts) was directly pipetted into the sample well of the cassette, followed by 2 to 3 drops of buffer solution. The results in the form of pink/purple colored lines were read visually within 10 to 15 minutes. Test results were considered valid if the internal control line was visible. Weak signals for IgM and/or IgG was/were considered positive.

Table 1

List of SARS COVID19 antibody rapid diagnostic tests evaluated in this study. All devices can be stored at room temperature.

Test name	2pc?>Company and location	Requires pre-dilution of sample	Volume sera/cassette	Devices IgM/IgG
BioMedomics COVID-19 IgM/IgG tapid test	BioMedomics Inc., Morrisville, NC	No	10 µL	1 device for both IgM/IgG
Rapid response COVID-19 IgG/IgM Test	BTNX Inc., Markham, ON L3R 4G7Canada	No	5µL	1 device for both IgM/IgG
Coronavirus IgG/IgM antibody (COVID-19) Test	UCP Biosciences Inc. San Jose, CA	No	10 µL	1 device for both IgM/IgG
ZEUS rapid SARS-CoV-2 IgM/IgG Test Cartridgea	ZEUS Scientific, Branchburg, NJ	Yes	65 µL	Separate IgM and IgG
Coronavirus (COVID-19) IgM/IgG Rapid Testa	RayBiotech Life, Peachtree Corners, GA	Yes	25 µL	Separate IgM and IgG
STANDARD Q COVID-19 IgM/IgG Duo	SD Biosensor, INC. Gyeonggi-do, Republic of Korea	No	10 µL	Separate IgM and IgG

Note: Zeus Scientific and RayBiotech testing require sample dilution before adding onto the cassette's sample well.

Calculations

Specificity and sensitivity were calculated as follows:

Results and discussion

All sera used in this study were first characterized using the NYS SARS-CoV MIA. The serum samples used to measure the performance of the point-of-care tests (POCTs) in this study were grouped as outlined in Table 2 .

Table 2

Groups of specimens used in evaluating antibody-based point-of-care (POC) testing.

Group	Category	Description	Number of samples (N)
1	Low positive MIA	6SD> MFI <20,000	10
2	Intermediate positive MIA	20,000> MFI <30,000	10
3	Strong positive MIA	MFI >30,000 MFI	10
4	PCR positive specimens	Convalescent serum specimens from SARS COV-2 RT-PCR confirmed patients. This does not imply the patient, or the serum specimen was PCR positive at the time of collection.	14
5	Specificity panel	Antibody positive for other infectious agents	∼62
6	Other coronaviruses: OC43, NL63, HKU1	Acute serum specimens determined by RT-PCR	5

Performance characteristics of POC tests based upon MIA reactivity

To determine the sensitivity of the POC devices in this study, we asked 3 fundamental questions: first, if there were any antibodies reactive with SARS-CoV-2 in the serum specimens, could the POC tests detect them; second, at what level must antibodies be present in the sera in order for the POC tests to be able to detect them; third, since all of the POC tests purport to detect, individually, both IgM and IgG, how well did they do so in IgM+/IgG+ sera? To address these questions, we analyzed sera from known convalescent COVID-19 positive individuals, as determined by RT-PCR positivity, and at least 21 days postsymptom onset. The sera were assessed using the NY SARS-CoV-2 MIA which provides a signal (MFI) that is proportional to antibody abundance. Using reactivity to the N antigen as a standard, the sera were categorized as being low, intermediate, or high antibody sera (Table 2). All but one of the sera also had varying amounts of antibodies reactive with the receptor binding domain antigen. In addition, all specimens were IgM and IgG positive, except for 2 specimens in the low and one specimen in the intermediate groups that lacked IgM.

As indicated in Table 1, 3 of the POC tests detected IgM and IgG on a single device, while the other 3 POC tests required separate devices to singly detect IgM and IgG. IgM and IgG are scored separately regardless of whether the bands are on one device or separate devices. Possible outcomes are no bands (nonreactive), IgM+/IgG-, IgM-/IgG+ or IgM+/IgG+. The overall sensitivity and the summary of IgM and IgG individually detected for each of the devices is shown in Table 3 . The POC devices had a large amount of variability in their detection levels (80%–97% over 30 samples), even in specimens with high antibody abundance. Overall, IgG was better detected than IgM. Even greater variability was seen when looking at IgM detection by itself (one band/device) or with IgG (2 bands/devices), or, conversely, IgG detection by itself or with IgM. The distribution of IgM (or IgG) alone versus detecting both antibody classes is shown in Fig. 1 .

Table 3

Summary of sensitivity results. MIA pos samples were separated into high intermediate and low based on total Ig reactivity to the N antigen as described in materials and methods.

POC device	SARS COV-2 antigen used	Any antibody			Total IgM			Total IgG			Sensitivity % (95% CI)
		H	I	L	H	I	L	H	I	L
BioMedomics	RBD of Spike protein	10/10a	7/10	7/10	9/10	5/9	5/8	9/10	7/10	4/10	80.0 (60.8–91.5)
BTNX	Proprietary	10/10	9/10	10/10	10/10	9/9	10/8	10/10	7/10	5/10	96.7 (80.0–99.8)
UCP Biosciences	Proprietary	14/14	11/11	9/10	14/14	8/10	8/8	14/14	10/11	8/10	97.1 (83.3–99.8)
Zeus POC	Proprietary	10/10	9/10	7/10	3/10	2/9	3/8	10/10	8/10	5/10	86.7 (68.3–95.6)
RayBiotech	N protein	9/10	9/10	6/10	9/10	8/9	5/8	6/10	8/10	6/10	80.0 (60.8–91.5)
SD Biosensor	Proprietary	10/10	10/10	9/10	10/10	5/9	6/8	10/10	10/10	8/10	96.6 (80.9–99.8)

CI = confidence interval; H = high-; I = intermediate-, L = low antibody abundance based on NY SARS-CoV MIA; RBD: receptor binding domain; N protein = nucleocapsid.

Denominators represent numbers of samples tested which contain antibody regardless of class, IgM/IgG respectively.

Fig. 1

Diagnostic performance of antibody-based point-of-care devices on MIA positive specimens.Thirty positive specimens were previously characterized by MIA: 10 low positives (6SD-20,000 MFI), 10 intermediate positive (20,000–30,000 MFI), and 10 strong (high) positiv (30,000–40,000 MFI). These 30 specimens were then tested for a side-by-side evaluation by 6 different antibody-based rapid diagnostic tests and results were recorded for IgM and IgG, IgM, IgG, and no reactivity to either IgM and/or IgG. (n = 30 for all 6 different tests except for UCP Biosciences where a total of 11 intermediate and 14 strong positive specimens were tested). Ig = immunoglobulin; MIA = microsphere immunoassay; SD = standard deviation

For each individual device, we summarize how the original questions were addressed:

Biomedomics: Without regard to isotype, this device could detect Ab in 80% (95% confidence interval [CI]: 60.8–91.5) of the 30 specimens tested. At the lowest level of Ab abundance, based upon N antigen reactivity in the MIA, this device could detect Ab in 70% of the specimens. For separate detection of IgM versus IgG, both had a wide range in positive results over the different antibody abundance groups (IgM 56%–90%; IgG 40%–90%).

BTNX: Without regard to isotype, this device could detect Ab in 96.7% (95% CI: 80.0–99.8) of the 30 specimens tested. At the lowest level of Ab abundance, this device could detect Ab in 100% of the specimens. For separate detection of IgM versus IgG, IgM was found in every positive specimen, including 3 specimens that were IgM negative in the MIA. The IgM detected in those three specimens could either be IgM not detected in the MIA or false IgM positives. IgG detection was more varied and ranged from 50% to 100% of the specimens in the different antibody abundance groups.

UCP Biosciences: Without regard to isotype, this device could detect Ab in 96.7% (95% CI: 83.3–99.8) of the 35 specimens tested. At the lowest level of Ab abundance, this device could detect Ab in 90% of the specimens. For separate detection of IgM versus IgG, positivity for both isotypes ranged from 80% to 100% over the different Ab abundance groups.

Zeus:Without regard to isotype, this device could detect Ab in 86.7% (95% CI: 68.3–95.6) of the 30 specimens tested. At the lowest level of Ab abundance, this device could detect Ab in 70% of the specimens. For separate detection of IgM versus IgG, IgM was only detected in 22% to 38% of the specimens in the various Ab abundance groups, while IgG detection ranged from 50% - 100%.

RayBiotech: Without regard to isotype, this device could detect Ab in 80% (95% CI: 60.8–91.5) of the 30 specimens tested. At the lowest level of Ab abundance, based upon N antigen reactivity in the MIA, this device could detect Ab in 60% of the specimens. For separate detection of IgM versus IgG, both had a wide range in positive results over the different antibody abundance groups (IgM 63%–90%; IgG 60%–80%). We note that one specimen in each of the high and low Ab abundance groups did have a weakly reactive IgM band, which the kit considers inconclusive with a reflex to an alternative testing method.

SD Biosensor: Without regard to isotype, this device could detect Ab in 96.70% (95% CI: 80.9–99.8) of the 30 specimens tested. At the lowest level of Ab abundance, based upon N antigen reactivity in the MIA, this device could detect Ab in 90% of the specimens. For separate detection of IgM versus IgG, IgM positivity had a larger variation among the Ab abundance groups (56%–100%) than did IgG positivity (80%–100%).

Performance characteristics of POC tests from SARS-CoV-2 PCR confirmed patients

To further characterize the rapid diagnostic tests, we also examined their ability to detect antibody production in sera from early acute COVID-19 patients. Sera from 14 individuals, who had tested positive for SARS-CoV-2 RNA in an RT-PCR test, were tested at the Wadsworth Center using the MIA and additionally tested using the POC devices. Of these sera, the MIA detected antibodies at a high level in 3 specimens and did not detect antibodies in 11 specimens. The failure to detect antibodies could be due to the sensitivity limit of the MIA or because the COVID-19 individuals had not (yet) produced antibodies. In addition, 2 of the sera had an indeterminate result in the MIA, which could be due to either a very low level of COVID-19-specific antibodies in the sera or Ab cross-reactivity with other antigens. Given the recent infection with SARS-CoV-2, the former explanation is the likeliest. In testing the POC devices, we asked the question: in these instances of known COVID-19 infection, could the devices detect Abs when the MIA could not; that is, would they perform better than the comparator assay? As shown in Fig. 2 , for any of the POC devices, Abs were only detected in the specimens which were reactive in the MIA. No Abs were detected in samples found to be either nonreactive or indeterminant using the MIA. Not all of the rapid tests, however, could detect in Abs in even the MIA positive specimens. The BioMedomics, BTNX, UCP Bioscience, and Zeus devices were Ab positive for all 3 specimens. However, while the first 3 devices detected both IgM and IgG in each of the 3 specimens, the Zeus devices detected IgG, but not IgM, in 2 of the 3 sera. This result is consistent with those obtained using convalescent sera, which showed the Zeus device to be the least sensitive in identifying IgM positive sera. Of the remaining devices, the RayBiotech device detected IgM and IgG in 2 of the 3 MIA positive specimens. We also note that 2 additional specimens had a weakly reactive IgM band, which the kit considers inconclusive with a reflex to an alternative testing method. Due to insufficient specimen volume, only one of the MIA reactive specimens could be tested using the SD Biosensor device and that specimen was found to be positive for both IgM and IgG.

Fig. 2

Diagnostic performance of antibody based point-of-care (POC) devices on serum from patients that had previously tested positive by PCR for SARS Cov-2. Fourteen specimens were tested on the POC devices and the percentage of positive and negative IgM, IgG, and IgM + IgG results were recorded. # Due to limited number of RayBiotech devices, only 12 specimen were tested. ## There were a total of 16 specimens tested on the SD Biosensor devices. Ig = immunoglobulin; PCR = ploymerase chain reaction.

Cross-reactivity/analytical specificity

Having determined the relative abilities of the POC tests to detect Abs that were present in positive sera, we next wished to determine if the devices provided positive results when COVID-19 Abs were not present. To assess specificity, we evaluated the devices using a panel known to be negative COVID-19, but to also have Abs to other infectious agents or autoimmune markers. We examined 23 separate analytes with an average of 3 serum specimens per category. Of the tested devices, BioMedomics, UCP Biosciences and RayBiotech had the highest number of false positive results, with an overall false positive rate of 11% (n = 63), 19% (n = 62) and 22.5% (n = 40), respectively. The false positive rate for RayBiotech's IgM cassettes (including the inconclusive results) was 20%, whereas the false positive rate for the IgG cassettes was 2.5% (overall 22.5%). POC devices from BTNX and Zeus Scientific had overall false positive rates of 1.6%, while SD Biosensor had a false positive rate of 0%. (Table 4 and Supplemental Tables S1–6). We also had the opportunity to evaluate most of the POC devices using a small panel of sera from individuals which had recently been diagnosed, using a molecular respiratory pathogen panel test, as being infected with a non-SARS-COV-2 coronavirus (OC43 (n = 2), NL63 (n = 2), and HKU1 (n = 1)). None of the specimens was reactive for any of the tests, except for a positive IgM result in the UCP Biosciences device with the serum from a recent infection with HKU-1. Although the number of specimens tested was low (5, in total) it is encouraging that the false positivity rate is low with these related coronaviruses. We should note however, that, due to low specimen volumes, only three specimens NL63 (n = 2) and OC43 (n = 1) were tested on the SD Biosensor devices. Also, due to the limited number of devices, none of these specimens were tested on the RayBiotech devices.

Table 4

Summary of specificity results.

Specificity (95%CI)	Any antibody	IgM	IgG
Biomedomics	56/63 (88.9%) (77.8–95.0)	5/63 (91.8%) (81.1–96.9)	5/63 (91.8%) (81.1–96.9)
BTNX	62/63 (98.4%) (90.3–99.9)	0/63 (100%) (92.8–100)	1/63 (98.4%) (90.3–99.9)
UCP Biosciences	50/62 (80.6%) (68.2–89.1)	9/62 (85.4%) (73.7–92.7)	5/62 (91.9%) (81.4–96.9)
Zeus Scientific	60/61 (98.4%) (90.0–99.9)	0/61 (100%) (92.6–100)	1/61 (98.3%) (90.0–99.9)
RayBiotecha	31/40 (77.5%) (61.1–88.5)	8/40 (80%)b (63.8–90.3)	1/40 (97.5%) (85.2–99.8)
SD Biosensor	61/61 (100%) (92.6–100)	0/61 (100%) (92.6–100)	0/61 (100%) (92.6–100)

Due to several inconclusive results of the RayBiotech devices, 2 different sets of performance measures were calculated (please see Supplemental Table 4).

IgM results on the RayBiotech devices were inconclusive. Per kit instructions these samples should be tested using an alternative method to confirm results.

The ease of use, fast turnaround time, and relative low cost make the use of POC tests an attractive option for serology (serosurveys and companions to molecular tests for determining exposures). However, as this small survey shows, there is great breadth in performance, both with respect to sensitivity and also specificity. In our opinion, the performance of many of these devices do not match up to high complexity tests and caution must be exercised in the choice of a particular rapid test for determining antibody responses to SARS-CoV-2.

Funding support

This work was supported by the New York State Department of Health.

Declaration of competing interest

Authors have no conflict of interest.

Compliance with ethical standards

Studies were performed on remnant sera from deidentified clinical specimens, submitted to the NYSDOH, as part of routine clinical testing for SARS COV-2. Testing was done in a CLIA-certified clinical laboratory. Testing of human specimens was in response to a declared Public Health Emergency and is not considered to be research. All procedures performed in this study involving human serum samples were done in accordance with the ethical standards of the institutional research committee of the NYSDOH.