Clinical evaluation of SARS-CoV-2 point-of-care antibody tests.

The aim of this study was to assess the analytic and clinical performance of four rapid lateral flow point-of-care tests (POCTs) for identifying SARS-CoV-2-specific antibodies. A retrospective study was conducted between 22 January and 30 March 2020 on 132 serum samples for SARS-CoV-2-specific antibody detection referred to a tertiary referral hospital laboratory in New South Wales. Multiple sera were tested from 20 confirmed or suspected COVID-19 patients with SARS-CoV-2-specific antibodies detected by immunofluorescence (IFA) or neutralisation, and 71 SARS-CoV-2 uninfected individuals. We measured the sensitivity and specificity for detection of SARS-CoV-2 IgM and IgG antibodies for each POCT in comparison to positive SARS-CoV-2-specific IFA and viral neutralisation, our current laboratory benchmark tests. All POCTs were found to have a low analytic sensitivity for SARS-CoV-2 antibodies, ranging from 27.3% to 58.2%, with a specificity between 88.3% and 100%, and a low clinical sensitivity from 45% to 65%, with a clinical specificity between 87.3% and 100%. All POCTs had an increased sensitivity when specimens were collected more than 14 days from onset of symptoms. The detection using point-of-care testing of SARS-CoV-2-specific antibodies after disease onset lagged behind IFA by a range of 0-9 days. POCTs promise the benefit of providing quick easy testing for SARS-CoV-2-specific antibodies. However, their poor sensitivity and delayed antibody detection make them unsuitable as a diagnostic or screening tool alone. Crown

Introduction

The current coronavirus disease 2019 (COVID-19) outbreak caused by a novel coronavirus named severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), was first reported in Wuhan, China in December 2019.

The primary means of laboratory diagnosis for COVID-19 disease is nucleic acid testing (NAT) on deep nasal, nasopharyngeal and throat swabs, or lower respiratory tract specimens, using real-time reverse-transcriptase polymerase chain reaction (RT-PCR) during the acute symptomatic phase of illness.

Detection of SARS-CoV-2-specific antibodies is another method to identify recent or past infection with SARS-CoV-2. The SARS-CoV-2 envelope proteins trigger antibodies that are neutralising; the most important is thought to be the spike protein (S), which is responsible for attachment, fusion and viral entry into host cells, and is an obvious target for serology test development. Other potential targets include the nucleocapsid protein (N).

Antibodies to SARS-CoV-2 are detected 7–10 days post-illness onset with studies showing the majority of patients seroconverting by weeks 2–3; this can vary depending on factors including the patient's immune status and disease severity. Serology alone is not recommended for acute diagnosis of COVID-19, though is likely to be useful in the confirmation of recent or past COVID-19 infections (for example, in patients presenting seven or more days from symptom onset). Serology has proven useful in detecting convalescent cases to aid establishing epidemiological links between clusters. It is uncertain whether the presence of SARS-CoV-2-specific antibodies indicates immunity from further infection, and how long antibodies persist following acute infection.

There is widespread interest in the use of point-of-care tests (POCTs). A media release by the Commonwealth Minister for Health in late March stated that the Australian Government had ordered 1.5 million POCTs to expand Australia's testing capacity for COVID-19 disease. Potential benefits include a rapid turnaround time (as short as 15 minutes) and ease of performance, especially advantageous in remote and rural settings. Most commercially available POCTs are based on detection of SARS-CoV-2 antigens or antibodies, and are generally rapid lateral flow assays (LFA) that detect IgM and/or IgG.

Twenty-two POCTs have been listed by the Australian Therapeutic Goods Administration (TGA) for use in Australia, and are undergoing an expedited post-marketing evaluation on the Australian Register of Therapeutic Goods. Previous experience with antigen-detecting LFA for influenza have shown reduced sensitivity compared to NAT.9, 10, 11 Concerns regarding the lack of robust validation of POCTs and the significant consequences of their misapplication has led to several bodies including the World Health Organization, the TGA, the Public Health Laboratory Network and The Royal College of Pathologists of Australasia to caution against their use for diagnosis of COVID-19 disease.

This study aimed to assess the analytic and clinical performance of POCTs in identifying SARS-CoV-2-specific antibodies, and so to help determine their role in the Australian setting.

Material and methods

We conducted a retrospective study evaluating the clinical sensitivity and specificity of four commercial lateral flow assay devices marketed as POCTs for the detection of SARS-CoV-2-specific antibodies.

Principle of tests

We tested four different POCTs: OnSite COVID-19 IgG/IgM Rapid Test (CTK Biotech, USA), 2019-nCov Antibody Test (Innovita Tangshan Biological Technology, China), SARS-CoV-2 Antibody Test Strip (Changsha Sinocare, China), Standard Q COVID-19 IgM/IgG Duo Test and Standard Q COVID-19 IgM/IgG Combo Test (SD Biosensor, Republic of Korea). All are lateral flow assays that detect IgG and IgM antibodies against SARS-CoV-2 in whole blood, plasma or serum. The OnSite, Innovita and Standard Q Combo tests have separate lines containing anti-human IgG and IgM monoclonal antibodies, while the Sinocare test has a single line with both antibodies. The Standard Q Duo tests have separate individual devices for IgG and IgM, but are essentially the same test as the Standard Q Combo. There is a control line which must develop colour; if absent the test is invalid. They are promoted as tests to be used by healthcare professionals and the TGA restricts their use to medical practitioners.

All POCT were performed using serum according to the manufacturer's protocol. Test results were interpreted as negative, equivocal or positive depending on appearance of the band on the testing strip. Equivocal results were included as positive tests in our analyses, as the presence of a faint band is likely to be conservatively interpreted as positive by clinicians.

The in-house immunofluorescence assay (IFA) and neutralisation antibody assay were used as the benchmark. Compared to NAT-confirmed cases, the sensitivity and specificity of detecting one or more classes of SARS-CoV2 antibodies (IgG, IgM or IgA) by IFA when collected 14 days or more after symptoms onset was 91.3% (95% Cl 84.9–95.6) and 98.9% (98.4–99.8) respectively. The principles of viral neutralisation and the IFA have been previously described.13, 14, 15

Sample selection

Sixty serum samples were obtained from COVID-19 NAT-confirmed patients (n=17) or COVID-19 suspected patients without positive NAT tests (n=3) who were symptomatic household contacts of confirmed cases. The NAT method used was an in-house nucleic acid test targeting E gene and RdRp: the primers have been described elsewhere. Both initial samples (n=20) and convalescent samples (n=40) were included. All confirmed and suspected COVID-19 patients had SARS-CoV-2-specific IgM, IgG and/or IgA antibodies detected by both IFA and neutralisation on one or more serum samples. Clinical data were available including date of symptom onset, SARS-CoV-2 NAT results and possible exposure history. In SARS-CoV-2 seropositive samples, the presence of SARS-CoV-2-specific IgM and/or IgA were regarded as suggestive of a recent infection. The detection of SARS-CoV-2-specific IgG alone could not reliably be used to differentiate a past infection from a recent infection. SARS-CoV-2-specific IgG can often precede IgM or IgA following acute infection, and so the test should be interpreted within the context of the patient's date of symptom onset and potential exposure. This pattern has also been documented in SARS and SARS-CoV-2.17, 18, 19

There were 18 patients with SARS-CoV-2-specific IgM and/or IgA detected by IFA, suggestive of a recent infection. A past infection was defined as a resolution of symptoms in addition to being 10 days or more from first positive NAT with the absence of a positive IFA IgM/IgA. Two patients had isolated SARS-CoV-2-specific IgG detected, with one patient likely having a past infection, as onset of symptoms predated serology by 50 days. The second patient likely had a recent infection with IgG development prior to appearance of IgM or IgA.

Negative control samples (n=72) were collected from stored baseline testing for asymptomatic staff submitted at the start of the pandemic (n=9), symptomatic patients found to be SARS-CoV-2 NAT negative (n=21), serum samples submitted before the pandemic (June to August 2019) for respiratory pathogen testing found to be positive for Mycoplasma pneumoniae antibodies (n=8) or influenza A antibodies (n=19), and samples positive for rheumatoid factor (n=15). These negative control samples were negative (titre <10) for SARS-CoV-2-specific IgG, IgM and IgA by IFA and for SARS-CoV-2 neutralising antibody.

A total of 132 serum samples from 91 patients were tested by the OnSite COVID-19, Sinocare SARS-CoV-2, Standard Q COVID-19 Duo and Combo antibody tests. Only 100 samples from 60 patients (including 60 samples from confirmed or suspected patients and 40 negative control samples) were tested by the Innovita 2019-nCov Antibody Test, due to a limited availability of testing kits.

Data analysis

Tang et al. highlighted the importance of distinguishing ‘analytic’ and ‘clinical’ sensitivity when assessing methods to detect COVID-19, namely the ability of an assay to detect SARS-CoV-2-specific antibodies when present in sera, versus the ability of an assay to identify a patient's overall immune status to the virus. The analytical sensitivity and specificity of each POCT was calculated independently for the presence of SARS-CoV-2-specific IgG and IgM in comparison to IFA. Clinical sensitivity was defined as the assessment of the overall development of antibodies in individual patients, and was measured as the detection of any antibody in any sample from a given patient. In addition, the clinical sensitivity and specificity of the test's ability to ascertain a correct immune status was assessed. Furthermore, we compared the sensitivity rate of specimens collected more than 14 days following onset of symptoms.

We also evaluated the median IFA antibody titre of the samples which were positive on each of the POCTs. Binomial 95% confidence intervals were calculated for all proportions.

We calculated the window periods from the onset of symptoms until antibody detection in three patients with serial collections who initially had negative serology.

Results

Patient characteristics

Characteristics of all 91 patients including 20 COVID-19 confirmed or suspected patients and 71 uninfected patients are summarised in Table 1 .

Table 1

Patient characteristics (n=91)

Characteristics	n
Age, years	Median 46 (range 3–91)
Gender
Male	34
Female	57
IFA assays performed	132
IFA assays positive	55
IFA assays negative	77
SARS-CoV-2 NAT positive patients	17
SARS-CoV-2 NAT negative patients	25
SARS-CoV-2 NAT not done	49
Patients with recent infections	19
Patients with past infections	1
Negative patients	71
Influenza A antibodies	19
Mycoplasma pneumoniae antibodies	8
Rheumatoid factor positive	15
Serum samples collected from onset of symptoms
Week 1 (0–7 days post-onset of symptoms)	7
Week 2 (8–14 days post-onset of symptoms)	24
Greater than 2 weeks (>14 days post-onset of symptoms)	29
Patients with >1 serology test	9
Patients with evidence of seroconversion	3
Patients with 4-fold rise in IgG	6

NAT, nucleic acid testing.

When individually assessing the 60 samples from the 20 confirmed or suspected COVID patients, 55/60 samples had IgG detected and of those, 49 also had SARS-CoV-2-specific IgM and/or IgA detected. In 6/60 samples, only IgG was detected. The majority of samples (48%) were collected greater than two weeks post-onset of symptoms. Three patients had evidence of SARS-CoV-2-specific antibody seroconversion on serial testing and four-fold rise of IgG, a further three patients had a four-fold IgG rise (see Table 2 ).

Table 2

Analytical and clinical performance of the OnSite COVID-19 IgG/IgM Rapid, Innovita IgG/IgM 2019-nCoV, Sinocare SARS-CoV2 and Standard Q COVID-19 Duo and Combo tests using IFA IgG and/or IgM as the reference standard for individual samples and patients

	Sensitivity[95% CI]	Specificity[95% CI]	PPV[95% CI]	NPV[95% CI]
OnSite COVID-19 Test
IgG
Overall	54.5% [40.6, 68.0]	98.7% [93.0, 100.0]	96.8% [80.8, 99.5]	75.2% [69.5, 80.3]
>14 days PSO	82.8% [64.2, 94.2]	98.4% [91.5, 100]	96.0% [77.3, 99.4]	92.5% [84.8, 96.5]
IgM
Overall	50.0% [33.8, 66.2]	100.0% [96.1, 100.0]	100%	82.1% [77.1, 86.3]
>14 days PSO	60.7% [40.6, 78.5]	100.0% [94.4, 100]	100%	85.3% [78.6, 90.2]
IgG and/or IgM
Overall	58.2% [44.1, 71.4]	98.7% [93.0, 100.0]	97.0% [81.8, 99.6]	76.8% [70.7, 81.9]
>14 days PSO	85.7% [67.3, 96.0]	98.4% [81.5, 100]	96.0% [77.4, 99.4]	93.9% [86.2, 97.5]
Patientsa	65.0% [40.8, 84.6]	98.7% [92.4, 100.0]	93% [64.4, 98.9]	91% [84.6, 94.8]
Innovita 2019-nCoV Test
IgG
Overall	27.3% [16.1, 41.0]	95.6% [84.9, 99.5]	88.2% [64.4, 96.9]	51.8% [57.5, 56.1]
>14 days PSO	44.8% [26.4, 64.1]	93.6% [78.6, 99.2]	86.7% [61.6, 96.3]	64.4% [56.3, 71.8]
IgM
Overall	35.0% [20.6, 51.7]	100.0% [94.0, 100]	100.0%	69.8% [64.8, 74.3]
>14 days PSO	42.9% [24.5, 62.8]	100.0%	66.7% [59.2, 73.4]	73.3% [60.3, 83.9]
IgG and/or IgM
Overall	36.4% [23.8, 50.4]	95.6% [84.9, 99.5]	90.9% [71.2, 97.6]	55.1% [49.9, 60.2]
>14 days PSO	62.1% [42.3, 79.3]	93.6% [78.6, 99.2]	90.0% [69.6, 97.3]	72.5% [62.1, 80.9]
Patientsa	45.0% [ 23.1, 68.5]	97.2% [ 90.2, 99.7]	82% [51.4, 94.0]	86% [80.8, 90.3]
Standard Q COVID-19 Duo and Combo Tests
IgG
Overall	52.7% [38.8, 66.5]	100% [95.3, 100.0]	100%	74.76% [69.1, 79.7]
>14 days PSO	72.4% [52.8, 87.3]	100% [94.3, 100]	100%	88.7% [81.4, 93.4]
IgM
Overall	47.5% [ 31.5, 63.9]	95.7% [89.2, 98.8]	82.6% [3.97, 30.07]	80.73% [75.7, 85.0]
>14 days PSO	48.3% [29.5, 67.5]	96.8% [89, 99.6]	87.5% [63.0, 96.7]	74.1% [72.1, 88.9]
IgG and/or IgM
Overall	53.6% [39.7, 67.0]	97.4% [90.8, 99.7]	93.8% [78.9, 98.4]	74.0% [68.2, 79.1]
>14 days PSO	68.0% [46.5, 85.1]	97.2% [90.3, 99.7]	89.5% [67.9, 97.2]	89.7% [83.2, 93.4]
Patientsa	60.1% [36.1, 80.9]	97.2% [90.2, 99.7]	85.7% [59.4, 96.1]	89.6% [83.4, 93.7]
Sinocare SARS-CoV2 Test
IgG/M (Total)
Overall	45.5% [32.0, 59.5]	88.3% [79.0, 94.5]	73.5% [58.5, 84.6]	69.4% [63.7, 74.5]
>14 days PSO	65.5% [45.7, 82.1]	85.7% [74.6, 93.3]	67.9% [52.2, 80.3]	84.4% [76.4, 90.0]
Patientsa	55.0% [23.1, 68.5]	87.3% [90.2, 99.7]	55% [51.4, 95.0]	87% [80.8, 90.3]

CI, confidence interval (Clopper-Pearson); PPV, positive predictive value; NPV, negative predictive value; PSO, post-symptom onset.

Detection of any antibody in any sample from a given patient.

Comparative analytical performance of POCTs with IFA

Performance characteristics of the commercial POCT IgG and IgM assays compared to IFA are summarised in Table 2. All had poor sensitivities for IgG and IgM, with relatively high specificities. Our evaluation had significantly lower sensitivity than the stated performances provided by the manufacturer for all POCTs (see Table 3 ). However, the Standard Q Duo and Combo tests had increasing sensitivity with symptom onset >7 and >14 days, which is a trend we observed in our analysis. The only false positive IgG using the OnSite assay was from a patient with known positive Mycoplasma pneumoniae IgM. The Sinocare test also demonstrated some cross-reactivity with samples positive on M. pneumoniae and influenza A serology. The Standard Q tests had two false positives: one was a patient with known rheumatoid factor, and the other from a SARS-CoV-2 uninfected patient. Furthermore, there were nine samples that were positive for IgA by IFA, but negative for IgM by IFA, OnSite and Innovita assays. The standard Q IgM assay detected two samples. All nine of these samples also had SARS-CoV-2-specific IgG detectable by IFA, yet none were detected by the Innovita IgG assay, only 3/9 were detected by both the OnSite IgG assay and Standard Q IgG assay and one detected by Sinocare assay.

Table 3

Sensitivity reported by manufacturers

	Sensitivity[95% CI]	Specificity[95% CI]
OnSite COVID-19 Test
IgG	96.86% [93.7, 98.5]	100% [98.8, 100]
IgM	78% [72.1, 83.0]	99.4% [97.8, 99.8]
IgG and/or IgM	96.86 [97.8,99.8]	99.4% [97.8, 99.8]
Innovita 2019-nCoV Test
IgG and/or IgM	87.3% [92.0, 80.4]	100% [94.2, 100]
Standard Q COVID-19 Duo Test
IgG and/or IgM
Symptom onset
<7 days	68.9% [53.4, 81.8]	95.1% [91.8, 97.4]
7–14 days	88.0% [53.4, 81.8]
>14 days	99.1% [95.1, 100]
Standard Q COVID-19 Combo Test
IgG and/or IgM
Symptom onset
<7 days	69.1% [52.9, 82.4]	96.2% [93.2, 98.2]
7–14 days	89.39% [79.4, 95.6]
>14 days	96.9% [91.1, 99.4]
Sinocare SARS-CoV2 Test
IgG/M (total)	96.3%	99.6%

CI, confidence interval.

There was a trend to improved detection rates with increasing titres of IgA, IgG and IgM by IFA (Fig. 1, Fig. 2 ), and the window periods from the time of illness onset to detection of antibody were delayed by at least two days in each of the POCTs when compared to IFA in the three patients where serial serology samples were available (Table 4 ).

Fig. 1

Positive rate of IgG point-of-care test results by SARS-CoV-2 specific IgG immunofluorescence assay titre in overall and in samples collected >14 days post-onset of symptoms.

Fig. 2

Positive rate of IgM point-of-care test results by SARS-CoV-2 specific IgM immunofluorescence assay titre.

Table 4

Window periods for three patients

Patienta	Window periods (days)
	IFA	OnSite	Innovita	Standard Q	Sinocare
1	11	13	14	13	13
2	12	14	21	14	14
3	13	16	16	13	16

Patients with serial collections and initial negative serology.

When analysing samples collected greater than 14 days from onset of symptoms, there was an increase in sensitivity across all POCT devices in both IgG, IgM and total antibody (see Table 2).

Twenty patients had antibody detectable by IFA and neutralisation at one or more time points after illness onset. Thirteen of these had any antibody detectable at any time point by the OnSite assay (sensitivity 65%), 12 by the Standard Q assay (sensitivity 60%), nine by the Innovita POCT (sensitivity 45%) and 11 by the Sinocare POCT (sensitivity 55%) (Table 2).

Discussion

Our evaluation of the OnSite COVID-19, Innovita 2019-nCoV, Standard Q COVID-19 and Sinocare SARS-CoV-2 tests was undertaken to determine their efficacy in an Australian setting during the first wave of the COVID-19 pandemic.

The manufacturers of the POCTs have listed high sensitivities of 96.9% (OnSite), 87.3% (Innovita), 96.3% (Sinocare) and 99.1% (Standard Q) in their package information. Our findings contradicted these reports, as despite an increase in sensitivity in specimens collected more than 14 days post-onset of symptoms, all four POCTs had an unacceptably low analytic and clinical sensitivity for SARS-CoV-2-specific antibodies, with a delay in antibody detection.

A recent post-market evaluation of two LFA POCTs released by the Doherty Institute also shows poor total antibody sensitivity with 56.9% and specificity 95.6% for the OnSite test and 51.8% sensitivity and 97.8% for VivaDiag COVID-19 IgM/IgG Rapid Test. They compared POCTs against both NAT and viral neutralisation. However, benefits of evaluating POCTs against IFA and viral neutralisation are that both tests are detecting and quantifying antibodies, and IFA (along with ELISA-based assays) have historically been the most widely used standard for serological diagnosis.

Our study is an independent validation of POCTs for COVID-19 infection in an Australian setting. Currently there are limited data reporting the diagnostic performance and independent validation of POCTs for diagnosis of SARS-CoV-2 in clinical samples. One of the largest studies used a LFA POCT device on 525 cases with a sensitivity of 89% and specificity of 91%. A small study of 39 randomly selected patients from a high-prevalence area screening centre (German Red Cross), compared an LFA to NAT with a sensitivity of 36.4% and specificity of 88.9%. The slight superiority in sensitivity of the combined IgG-IgM antibody test over individual IgG or IgM antibody test was seen in a previous study.

Since the appearance of IgG may pre-date IgA or IgM responses in SARS-CoV-2 infection as seen previously in SARS,17, 18, 19 it is difficult to classify an individual as having recent or past infection from a serological test at a single time point. We attempted to assess whether a patient had a recent or past infection as part of clinical sensitivity, we determined that this was not possible using POCTs. We recommend serial measurements using a quantitative assay (like IFA or neutralisation) to detect seroconversion or a rising titre as a more reliable method for timing infection than looking for the presence of SARS-CoV-2 specific IgM. Furthermore, we found SARS-CoV-2-specific IgA was frequently present in the absence of SARS-CoV-2-specific IgM by IFA, and in cases when they were both present, IgA was present in a higher titre, making it a more reliable serological marker of SARS-CoV-2 infection. In patients with recent infections when analysed using IFA IgA as a benchmark, IFA IgG had a sensitivity of 87.3%. There was a small increase in sensitivity across all POCTs when assessing the presence of IgG, which tends to predate IgM and IgA (Supplementary Fig. 1 and Table 1, Appendix A).

There has been speculation regarding the application of POCTs to detect SARS-CoV-2 specific-antibody in combination with PCR to produce an ‘immunity passport’ to guide infection control and as part of strategies to ease lock-down measures. This assumes that past SARS-CoV-2 infection conveys immunity. However, the degree of immunity post-infection, and the seroprevalence in the Australian community are both unknown. The current POCTs are not sufficiently sensitive to be utilised in this capacity and further development of POCTs is required.

Limitations

The main study limitation is the relatively modest number of samples available for testing early in the pandemic in NSW. However, we were able to test these samples against four different devices. The timing of serum collection of initial and convalescent samples were not standardised, and not all patients had convalescent samples collected. Ideally, serial testing from well characterised serum panels would be used to most accurately define window periods.

Conclusion

In summary, there is a need for robust serological assays for SARS-CoV-2 to supplement diagnosis by NAT and help guide public health interventions. Despite potential benefits of POCTs, their current poor sensitivity and delayed SARS-CoV-2 specific antibody detection make them unsuitable to be utilised as a diagnostic tool, as an accurate marker of past infection, or in seroprevalence surveys.