Evaluation of diagnostic accuracy of 10 serological assays for detection of SARS-CoV-2 antibodies.

Antibody detection is essential to establish exposure, infection, and immunity to SARS-CoV-2, as well as to perform epidemiological studies. The worldwide urge for new diagnostic tools to control the pandemic has led to a quick incorporation in clinical practice of the recently developed serological assays. However, as only few comparative studies have been published, there is a lack of data about the diagnostic accuracy of currently available assays. We evaluated the diagnostic accuracy to detect Ig G, Ig M+A, and/or IgA anti SARS-CoV-2 of 10 different assays: lateral flow card immunoassays, 4 enzyme-linked immunosorbent assay (ELISA), and 3 chemiluminescent particle immunoassays (CMIA). Using reverse transcriptase polymerase chain reaction (RT-PCR) for COVID-19 as gold standard, sensitivity, specificity, PPV, and NPV were determined. Each assay was tested in 2 groups, namely, positive control, formed by 50 sera from 50 patients with SARS-CoV-2 pneumonia with positive RT-PCR; and negative control, formed by 50 sera from 50 patients with respiratory infection non-COVID-19. Sensitivity range of the 10 assays evaluated for patients with positive COVID-19 RT-PCR was 40-77% (65-81% considering IgG plus IgM). Specificity ranged 83-100%. VPP and VPN were respectively 81-100% and 61.6-81%. Among the lateral flow immunoassays, the highest sensitivity and specificity results were found in Wondfo® SARS-CoV-2 Antibody Test. ELISA IgG and IgA from EUROIMMUN® were the most sensitive ELISA. However, poor results were obtained for isolated detection of IgG. We found similar sensitivity for IgG with SARS-CoV-2 for Architect by Abbott® and ELISA by Vircell®. Results obtained varied widely among the assays evaluated. Due to a better specificity, overall diagnostic accuracy of the assays evaluated was higher in case of positive result. On the other side, lack of antibody detection should be taken with care because of the low sensitivity described. Highest diagnostic accuracy was obtained with ELISA and CMIAs, but they last much longer.

Introduction

A new coronavirus from the betacoronavirus family (subgenus Sarbecovirus) has emerged in the last few months. It has been denominated as severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) because of a high phylogenetic similarity to the SARS-CoV, first identified in China’s Guangdong province in 2002[1]. The disease caused by SARS-CoV-2 infection has been named by international consensus COVID-19 (coronavirus disease 2019)[2].

SARS-CoV-2 was described for the first time in December 2019 in Wuhan City, Hubei Province, China[3]. Due to high contagiousness, virulence, and issues to identify infected people, an extremely fast worldwide spreading broke. On March 11, 2020, the World Health Organization (WHO) officially declared the pandemic situation[4]. In the first week of July 2020, there were 216 affected countries; 11,514,395 people infected worldwide; and 535,185 deaths from this virus[5].

From the beginning, a significant effort has been done from laboratories worldwide to develop and commercialize specific diagnostic assays. Consequently, viral RNA reverse transcriptase polymerase chain reaction technique (RT-PCR SARS-CoV-2) of nasopharyngeal exudate samples was available from the very first weeks of infection. Due to its high accuracy to identify genetic material, this technique was and is still considered the gold standard in the diagnosis of the infection[6]. Despite this, RT-PCR has several limitations as it can only assess the presence of RNA but not that of infectious viral particles. On the other side, negative result cannot rule out a previous infection, not even assess the immune status of the individual against the infection. This last can only be reported with the detection of IgG, IgM, and IgA antibodies against SARS-CoV-2. Theoretically, these data would allow us to know if the individual has been in contact with the virus and if there is a current infection. This information would be much more accurate to know the risk of developing and transmitting the disease. At the individual level, it would allow actions to prevent and treat the infection. At the community level, essential information would be obtained to control the pandemic, by allowing epidemiological studies to be carried out in the general population and in specific sources of transmission (as health workers). For this reason, the development and commercialization of techniques to detect antibodies has constituted the second step in the laboratory diagnosis of SARS-CoV-2 infection. Due to its recent and progressive availability, information about the diagnostic accuracy of these assays is not enough. Because of this, the use of one over another in the different microbiology departments is determined most of the times by accessibility rather than scientific evidence. The publication of comparative studies between the different assays is essential to determine their diagnostic precision and, therefore, its true usefulness.

Aim

To determine the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of these ten assays (3 rapid and 7 ELISA/chemiluminescence) using the nasopharyngeal exudate RT-PCR for COVID-19 as gold standard.

Methods

Patients and serological samples

This descriptive study was conducted with serological samples obtained from patients, both hospitalized and outpatient, of La Princesa University Hospital, a tertiary level hospital in Madrid, Spain. Two groups were formed to evaluate the assays. The positive control group included serum samples (one per patient) obtained from the first 50 consecutive patients treated in that hospital between March and May 2020. The negative control group was formed with serum samples from 50 consecutive patients treated previously to the beginning of the SARS-CoV-2 pandemic (November 2019).

Characteristics of positive control group

All of them fulfilled the current (in that moment) criteria for COVID-19[7-9]. The average evolution time from the onset of symptoms was 10 days. All of them had positive RT-PCR for COVID-19 in a respiratory sample.

Negative control group

All of them had clinical suspicion of pneumoniae (at least 2 of fever, productive cough, dyspnea) and radiology findings non-compatible with SARS-CoV-2 infection (pulmonary infiltrates compatible with a community acquired pneumonia or ground-glass opacity pulmonary infiltrates compatible with a process of another viral etiology). In all of these patients, a microbiological evaluation of the suspected infectious etiology was performed. All the samples of this group had been previously frozen at − 20 °C. This is a standard of care that allows future serological assessments in case of need.

Antibody detection assays evaluated/antibody testing

The Microbiology Department of La Princesa University Hospital has 2 types of antibody detection assays at disposal. On one hand, lateral flow immunoassay cards, whose main advantage is the speed in obtaining results. Our service has had 3 different ones, i.e., WONDFO®, SGTi-Flex®, and Innovita®. On the other hand, chemiluminescence or ELISA detects IgG antibodies, and, in some cases, they also allow the detection of IgM, Ig A, and/or Ig M+A. These assays theoretically have greater diagnostic precision due to their detection methodology, but they take considerably longer. Of these second ones, the following have been evaluated: VIRCLIA (IgG and IgM+A) and ELISA (IgG and IgM+A), both from VIRCELL®; EUROIMMUN® ELISA (IgG and IgA); and the ABBOTT® chemiluminescence technique, currently only available for the detection of IgG. All assays have CE (European conformity in vitro device) marking for in vitro diagnosis. All have been carried out in the Microbiology Department of La Princesa University Hospital.

Lateral flow immunoassay card

Three assays were evaluated as follows: Wondfo® SARS-CoV-2 Antibody Test (Guangzhou Wondfo Biotech Co., Ltd), SGTi-flex® COVID-19 IgM/IgG (Sugentech, Inc.), and Innovita® 2019 n-CoV Ab Test Colloidal Gold (Biological Technology Co.). In each of them, the manufacturer’s instructions regarding the volume of sample and buffer to be dispensed were followed. The results of each test were visibly evaluated after 15 min. No result was invalidated.

Chemiluminescence

Three assays were evaluated following the manufacturer’s instructions. Cut-offs were calculated according to the manufacturer. Values above the cut-off were considered positive.

VIRCLIA IgG Monotest, VIRCLIA IgM+A Monotest (Vircell®, Spain, S.L.)

Both assays use recombinant antigens from the spicule (protein S) and the nucleocapsid (protein N). VIRCLIA processes 24 samples determining IgG and IgM+A simultaneously in 3 h. It is the maximum number of samples that can be processed at the same time. The manufacturer reports a global IgG sensitivity of 56% and a specificity of 99%; and for IgM+A, an overall sensitivity of 63% and a specificity of 99%.

SARS-CoV-2 IgG Architect (Abbott®)

This assay uses the nucleocapsid protein (protein N) as antigen. Architect has a higher work performance and is able to manage a greater number of samples in less time. The manufacturer reports a sensitivity of 86.36% in patients with an evolution of 8–13 days after the onset of symptoms and 100% in those with more than 14 days of evolution and a specificity of 99.63%.

No result in both chemiluminescence was invalidated.

ELISA

Four assays were evaluated. All ELISAs were performed automatically on DS2 (Alere®), which automatically calculated the optical densities of the samples and measured at 450 nm. The cut-offs were calculated according to the manufacturer. Values above the cut-off were considered positive. Vircell® VIRCLIA and ELISA processed sera also required manufacturer’s recommended 30 min at 56 °C for discomplementation.

COVID-19 ELISA IgG, COVID-19 ELISA IgM+IgA (Vircell®, Spain, S.L.)

Both Vircell techniques use recombinant spike (protein S) and nucleocapsid (protein N) antigens.

In 4 h, it is capable of evaluating 92 samples, determining both IgG and IgM+A simultaneously. It is the maximum number of samples that can be processed at the same time. The manufacturer reports an overall sensitivity for IgG of 58% and a specificity of 98%; and for IgM+A, an overall sensitivity of 66% and a specificity of 99%.

No result was invalidated.

EUROIMMUN® ELISA Anti SARS-Co-V2 IgG, and Anti SARS-Co-V2 IgA (Medizinische Labordiagnostika AG)

They use a recombinant spike protein antigen (protein S). The manufacturer reports a sensitivity for IgG of 33% in patients with < 10 days after the onset of symptoms and 80% in patients with > 10 days after the onset of symptoms and a specificity of 98.5%, and for IgA a sensitivity of 50% in patients with < 10 days after the onset of symptoms and 100% in patients with > 10 days after the onset of symptoms and a specificity of 92.5%. For IgG, a maximum of 92 samples can be studied at the same time. For IgA, only 45 samples can be processed at a time. The manufacturer indicates if a larger number is processed, decreases in adsorption can happen. Turnaround time until results were released is 3 h for both techniques.

Results from Negative Control IgA were invalidated and no more samples to retest were available.

Statistical analysis

Descriptive analysis of the variables was performed: (a) discrete variables were described as percentage and 95% confidence interval and (b) continuous variables (normal distribution, mean and range; non-normal distribution, median and range). Sensitivity and specificity intervals calculated are those of Clopper-Pearson for binomial distributions. Both PPV and NPV and their confidence intervals are calculated based on a prevalence of 50% (which we set ourselves). PPV and NPV intervals are calculated following the method proposed by Mercaldo et al[10].

Kappa coefficient was calculated to measure inter-rater agreement between tests.

Results

Patients database from both groups are summarized in Table 1.

Table 1

Basal data from positive and negative control groups, % (n)

Group	Sex	Age (mean, range)	Department of origin	Clinical criteria	Radiological criteria	Microbiological criteria
Positive control	Male 62% (31/50)	59.62 (27–75)	Emergency 76% (38/50)	98%* (49/50)	94%** (47/50)	PCR SARS-CoV-2 100% (50/50)
Negative control	Male 54% (27/50)	60.84 (17–96)	Internal medicine 70% (35/50)	100% (50/50)	98%*** (49/50)	34% (17/50) Yes**** 66% (33/50) No

*The most common symptoms in mild to moderate patients were fever, fatigue, and dry cough, followed by other symptoms including headache, nasal congestion, sore throat, myalgia, and arthralgia. A minority of patients had gastrointestinal symptoms. Four patients died during the recovery (two men and two women).

**Mild patients also manifested unilateral and focal ground-glass opacity (GGO) which gradually developed to bilateral or multilobular lesions. As the disease progressed further, GGOs evolved to consolidation lesions, presenting mixed pattern or pure consolidation

***No radiological data of pneumoniae. Microbiological exam was confirmatory of non-SARS-CoV-2 origin

****Streptococcus pneumoniae (n = 4), Chlamydophila pneumoniae (n = 4), Mycoplasma pneumoniae (n = 2), CMV (n = 2), Rhinovirus/Enterovirus (n = 2), Parainfluenza virus (n = 1), MTB (n = 1), Mycobacterium fortuitum (n = 1)

Test results are summarized in Table 2.

Table 2

Comparative table of COVID-19 serology assays of Microbiology Department

Assay	Sensitivity	Specificity	PPV	NPV	Kappa
Wondfo (n = 100)	76 (61.83–86.94)	100 (92.89–100)	100	80.65 (71.79–87.22)	0.76 (0.57–0.95)
SGTi (N = 100)
IgG	40 (26.41–54.82)	100 (92.89–100)	100	62.5 (57.07–67.64)	0.4 (0.24–0.56)
IgM	70 (55.39–82.14)	90 (78.19–96.67)	87.5 (74.93–94.25)	75 (66.04–82.23)	0.6 (0.41–0.79)
IgG+IgM	74 (59.66–85.37)	90 (78.19–96.67)	88.1 (76.02–94.53)	77.59 (68.24–84.79)	0.64 (0.45–0.83)
Innovita (n = 100)
IgG	44 (29.99–58.75)	98 (89.35–99.95)	95.65 (75.50–99.37)	63.64 (57.71–69.18)	0.42 (0.26–0.58)
IgM	52 (37.42–66.34)	100 (92.89–100)	100	67.57 (60.96–73.55)	0.52 (0.35–0.69)
IgG+IgM	58 (43.21–71.81)	98 (89.35–99.95)	96.7 (80.42–99.51)	70 (62.70–76.41)	0.56 (0.38–0.74)
VIRCLIA Vircell (n = 96)
IgG	47.92 (33.29–62.81)	95.83 (85.75–99.49)	92 (74.15–97.88)	64.79 (58.23–70.84)	0.44 (0.26–0.61)
IgA+M	62.5 (47.35–76.05)	95.83 (85.75–99.49)	93.75 (79.15–98.34)	71.88 (63.84–78.72)	0.58 (0.39–0.77)
IgG+IgA+M	64.58 (49.46–77.84)	93.75 (82.80–98.69)	91.18 (77.20–96.93)	72.58 (64.21–79.61)	0.58 (0.39–0.77)
ELISA Vircell (n = 96)
IgG	64.58 (49.46–77.84)	95.83 (85.75–99.49)	93.94 (79.71–98.39)	73.02 (64.77–79.93)	0.60 (0.41–0.79)
IgA+M	77.08 (62.69–87.97)	83.33 (69.78–92.52)	82.22 (70.69–89.87)	78.43 (68.07–86.12)	0.60 (0.40–0.80)
IgG+IgA+M	81.25 (67.37–91.05)	81.25 (67.37–91.05)	81.25 (70.31–88.80)	81.25 (70.31–88.80)	0.62 (0.42–0.83)
ELISA Euroimmun (n = 90)
IgG	37.78 (23.77–53.46)	100 (92.13–100)	100	61.64 (56.14–66.87)	0.38 (0.22–0.54)
IgA	88.9	-	-	-	-
IgG+IgA	88.9	-	-	-	-
IgG Architect (n = 100)
IgG	52 (37.42–66.34)	100 (92.89–100)	100	67.57 (60.96–73.55)	0.52 (0.35–0.69)

Lateral flow immunoassays card: Wondfo, SGTi, Innovita

ELISA: Vircell, Euroimmun

Chemiluminescence: Virclia, Architect

Immunochromatography lateral flow type

In Wondfo® SARS-CoV-2 Antibody, total antibody sensitivity was 76%, with a specificity of 100%, PPV of 100%, and NPV of 81%.

In SGTi-flex® COVID-19 IgM/IgG, which separates the two antibodies, in the case of IgG, a sensitivity of 40%, specificity of 100%, PPV of 100%, and NPV of 62.5% were obtained; and for IgM, a sensitivity of 70%, specificity of 90%, PPV of 87.5%, and NPV of 75%. If we consider an Ig of the two positive (IgG or IgM), the sensitivity rises to 74%, the specificity to 90%, PPV to 88.1%, and NPV to 77.6%.

Innovita® 2019 n-CoV Ab Test Colloidal Gold also discriminates between the two antibodies. The results obtained for IgG were sensitivity of 44%, specificity of 98%, PPV of 95.7%, and NPV of 63.6%. The results obtained for IgM were as follows: sensitivity 52%, specificity 100%, PPV 100%, and NPV 67.6%. The results obtained for IgG or IgM, considering one of the two Ig positive, were as follows: sensitivity 58%, specificity 98%, PPV 96.7%, and NPV 70%.

In VIRCLIA IgG Monotest Vircell®, the results were as follows: sensitivity of 48%, specificity of 96%, PPV of 92%, and NPV of 65%.

In VIRCLIA IgM+A Monotest Vircell®, the results were as follows: sensitivity 63%, specificity 96%, PPV 94%, and NPV 72%. If we consider a sample as positive if it has IgG and/or IgM+A, the results were as follows: 65% sensitivity, 94% specificity, 91% PPV, and 73% NPV.

In SARS-CoV-2 IgG Architect Abbott®, it should be noted that, despite not being advised by the manufacturer, the technique was tested with discomplemented serum because it was the last assay to be available, and previously, the serum had been discomplemented as it was an essential requirement to analyze Vircell’s ELISA and CLIA techniques. The results obtained for IgG were a sensitivity of 52%, specificity of 100%, PPV of 100%, and NPV of 68%.

In COVID-19 ELISA IgG Vircell®, the results were as follows: 65% sensitivity, 96% specificity, 94% PPV, and 73% NPV.

In COVID-19 ELISA IgM+A Vircell®, the results were as follows: sensitivity 77%, specificity 83%, 82% PPV, and 78% NPV. If we consider a sample as positive if it has ELISA IgG and/or IgM+A, the results were as follows: sensitivity 81%, specificity 81%, PPV 81%, and NPV 81%.

In EUROIMMUN® ELISA Anti SARS-Co-V2 IgG, the results were as follows: sensitivity of 37.8%, specificity of 100%, PPV of 100%, and NPV of 61.6%.

In EUROIMMUN® ELISA Anti SARS-Co-V2 IgA, a fail dispensing the stopping solution during the performance of negative control group caused the invalidation of the assay. Therefore, only sensitivity data are presented for this antibody (IgA) and the IgG+IgA group. The sensitivity for IgA was 88.9%. If we consider a sample as positive if it has an ELISA IgG and/or IgA, as in the previous cases, it should be noted that only the positive control have been taken into account because in the negative control, we do not have IgA. In this case, the sensitivity is 88.9%. The IgA and IgG+IgA sensitivity results coincide because there is no patient who has positive IgG and negative IgA.

Discussion

Current diagnosis of active SARS-CoV-2 infection is established with clinical data (symptoms, imaging tests, epidemiological context) and a RT-PCR of respiratory samples. Antibody detection assays are destined to assess the immunological status against the virus of the individual and, by extension, of the community. We agree with Woloshin et al.[11] that inaccurate diagnostic tests undermine efforts at containment of the pandemic.

As the development of acquired defenses against the virus seems to provide security, the first target for these new techniques is to be highly specific. Even when Woloshin et al.[11] recommended new tests to be highly sensitive, we assume that in the case of antibody detection, the consequences of a false positive are worse than those of a false negative. The unfounded belief of presenting immunity carries an obviously much higher risk. Therefore, we believe that it is important to analyze the results obtained from this prism.

Given the exceptional situation in which this pandemic is developing, the evaluation of the assays has been limited by their availability, as the development and commercialization of specific reagents for SARS-CoV-2 has not been simultaneous. It has been considered a priority to use the same serum when evaluating all the assays, since, in our opinion, this increases the validity of the results obtained. For this reason, it was necessary to preserve the serum extracted from patients. In our case, it was frozen, being defrosted with each assay examined. Due to the high number of them to be evaluated, this freezing and thawing process was carried out more than what, in our opinion, is desirable. We believe that this may limit the quality of the results obtained.

As expected, lateral flow immunoassays showed worse diagnostic accuracy than ELISA and chemiluminescence test, due to their lower sensitivity. In our study, the test that showed the best result was the one that detected total antibodies, Wondfo® SARS-CoV-2 Antibody, with a sensitivity of 76%, and a specificity of 100%, PPV of 100%, and NPV of 81%, the Kappa value was 0.76. The rest presented similar results, although a little lower. Innovita® 2019 n-CoV Ab Test Colloidal Gold presented significantly worse values (sensitivity 58%, specificity 98%, PPV 96.7%, NPV 70%). These results are in line with previously published series[12], in which, however, results of Innovita® 2019 n-CoV Ab Test Colloidal Gold are comparable to the others tests.

Finally, when evaluating this group of tests, 2 facts should be considered. Lateral flow immunoassays provide a quick result and, furthermore, they can be performed by less experienced staff. Both advantages make them more useful in certain clinical situations. However, outside of these contexts or the lack of availability, we believe that their use is not justified against chemiluminescence and ELISAs, taking into account their higher diagnostic accuracy and larger capacity.

Global specificity for ELISA assays was > 95% (> 99% with Wondfo, EUROIMMUN® ELISA Anti SARS-Co-V2 IgG, and SARS-CoV-2 IgG Architect Abbott®). EUROIMMUN ELISA IgG and IgA was, in global terms, the ELISA with the highest sensitivity (88.9%) and specificity (100%). IgA values were high, in line with previously published results[13, 14]. However, the sensitivity for the isolated detection of IgG was very low (37.8%), in contrast to other studies (67–93.8%)[13, 14]. Something similar happened with COVID-19 ELISA IgG Vircell®. Facing Kohmer et al.[13] results (100% sensitivity, 95.2% specificity), in our study, the sensitivity was 65% (95% CI 49.46–77.84), maintaining a high specificity of 96% (95% CI 85.75–99.49) with 94% PPV (95% CI 79.71–98.39).

Despite a high specificity, in general, the sensitivity of chemiluminescence assays was lower than ELISAs. However, it should be considered that Architect’s results were significantly limited by the discomplementation and freeze-thaw process. We expect these results to be better when performed in optimal conditions. In addition, it is important to consider that the Architect is the equipment with the largest capacity (it processes the greatest number of samples in the least time). This, together with the fact that it does not need a discomplementation process, adds speed to the test and reduces the response time by the Microbiology Department. A last advantage is that it also permits other determinations to be carried out in parallel on the same sample.

When analyzing results obtained in this study, we believe it is important to take several aspects into account. On one hand, during the infection, the identification of antibodies against SARS-CoV-2 is time-dependent. From the seventh day after the onset of symptoms, they are detected in 40% of patients. This percentage rises significantly to more than 90%, from day 14[15]. In our case, the samples of the positive control were obtained from patients with an average clinical evolution of 10 days, which can mainly affect the sensitivity of the techniques.

On the other hand, sensitivity was not exactly as expected. As Woloshin et al.[11] have previously highlighted, pre-clinical sensitivity may be overestimated under the EUA (Emergency Use Authorization) from FDA (Federal Drug Administration).

We also are aware that the sample size is not especially large, what may limit the results obtained, and may justify the differences observed with previous studies.

Finally, the descriptive study methodology limits the interpretation of the observed differences.

In any case, as it has been highlighted in the introduction, there is a lack of evidence regarding the diagnostic accuracy of antibody detection assays. In the current pandemic context with high morbidity and mortality rates, scientific data obtained in a real clinical context regarding the real accuracy of these diagnostic tests is highly needed. Our study is, of those published, one which evaluates at the same time a larger number of tests. This provides a direct comparison of their results in the same sample. Furthermore, they were evaluated in a real clinical context such as the one experienced in the last months, with the limitations that were found. We believe that this helps to reflect the real usefulness of these tests.

Regardless of these results and those obtained to date, larger comparative and randomized studies are now more than needed, as this pandemic seems to be a long way to go.