Retrospective serosurveillance for anti-SARS-CoV-2 immunoglobulin during a time of low prevalence: A cautionary tale.

We performed a retrospective screening of 428 serum samples for anti-SARS-CoV-2  immunoglobulin during a period of low prevalence. Employing two different serological tests yielded discrepant results for 10 samples; highlighting an increased risk of potential  false positive results and the need for further confirmatory testing before publication of data. Crown

Dear Editor,

We read with interest the study in the Journal by Gurgel and colleagues from Sergipe, Brazil investigating asymptomatic circulation of SARS-CoV-2 in Northeastern Brazil prior to the first case reported on the 26th February 2020. The authors obtained 987 anonymised serum samples collected between January and April 2020 and tested them for anti-SARS-CoV-2 IgG and IgM antibodies using two in vitro diagnostic tests: the Nantong lateral flow immunochromatography test and iChroma2 lateral flow sandwich detection immunofluorescence COVID-19 antibody test. The study found 16 (1.6%) participants who tested positive on both assays with seven (43.8%) being IgM positive, three (18.8) IgG positive, and six (37.5%) positive for both. The authors state that positive samples were not screened for the presence of SARS-CoV-2 antigen. The paper concludes that SARS-CoV-2 may have been circulating among the lower income population of the southern districts of Sergipe prior to the first confirmed cases.

A similar paper by Apolone and colleagues detailed the retrospective testing of 959 asymptomatic individual samples from Italy between September 2019 to March 2020 when the first confirmed case occurred in February 2020. Using a receptor-binding domain specific enzyme-linked immunosorbent assay (ELISA), which detects Spike (S) protein immunoglobulin, the study found 111 (11.6%) positive samples, with the majority being in February (20%) followed by October (16.3%). The authors claim that SARS-CoV-2 could have been circulating in a high rate of asymptomatic carriers prior to the first confirmed case.

In a study similar to Gurgel and Apolone, we present the data from a small retrospective serosurveillance project of the Leicestershire patient population. 428 randomly selected serum samples (mean age: 39, s.d 23.3, range 0–95) from 1st October 2019 to 31st March 2020 were selected for serological testing. All samples were from patients undergoing viral serological screening and had previously been stored at −40 °C. All samples were thawed at 4 °C prior to testing on the DiaSorin SARS-CoV-2 S1/S2 Assay (Diasorin Ltd., Dartford, England) a chemiluminescent assay (CLIA) detecting IgG antibodies to the SARS-CoV-2 spike protein S1/S2 domain and all positive samples repeated on the Siemens Aptima SARS-CoV-2 Total Assay (Siemens, Erlangen, Germany) an ELISA detecting IgG and IgM antibodies to the SARS-CoV-2 spike protein S1 receptor-binding domain. Diasorin state a sensitivity and specificity of 97% (95% CI 86.8–99.5%) and 98.9% (97.5–99.2%), Siemens state theirs as 96.4% and 99.9% respectively. The Diasorin and Siemens assays were directly compared in a study analysing the performance of five SARS-CoV-2 serological assays with Diasorin being reported to have a sensitivity and specificity of 95% (92.8–96.7) and 98.6% (97.6–99.2) and Siemens as 98.1% (96.6–99.1) and 99.9 (99.4–100).

Our screening revealed 10 (2.3%) anti-SARS-CoV-2 IgG positive samples using the Diasorin assay; of which 5 were from patients sampled in 2019 (Table 1 ). However, upon testing on the Siemens assay all 10 samples were anti-SARS-CoV-2 IgG/IgM negative, all samples were tested in triplicate with no inconsistency in results. Additional archived samples suitable for SARS-CoV-2 PCR testing from IgG positive patients were sought, but none were available.

Table 1

Characteristics of SARS-CoV-2 IgG positive patients from retrospective serum sampling of Leicester population.

Patient	Sex	Age When Sample Taken (years)	Alive/Deceased	Co-morbidities	Symptoms	Recent Travel History	Serum Collection Date	DiaSorin SARS-CoV-2 S1/S2 Assay Results	Siemens SARS-CoV-2 IgG Assay Results
1	F	3	Alive	Non-resolving tonsillitis	Fever, sore throat, inability to ingest solids	Unknown	29/10/2019	Positive	Negative
2	F	5	Alive	Unknown	Unknown	Unknown	07/11/2019	Positive	Negative
3	M	18	Alive	Unknown	Asymptomatic	Unknown	22/11/2019	Positive	Negative
4	F	52	Alive	None	Asymptomatic	Unknown	04/12/2019	Positive	Negative
5	M	35	Alive	Hepatitis B Positive	Asymptomatic	Unknown	04/12/2019	Positive	Negative
6	M	0	Alive	Unknown	Unknown	Unknown	14/02/2020	Positive	Negative
7	F	37	Alive	Medical Termination of Pregnancy	Vomiting and abdominal pain	Unknown	19/02/2020	Positive	Negative
8	M	60	Alive	Latent TB, hypertension, gout	Asymptomatic	Unknown	22/02/2020	Positive	Negative
9	F	38	Alive	Endometrial polyp	Unknown	Unknown	04/03/2020	Positive	Negative
10	M	56	Alive	ESRF, T2DM, chronic osteomyelitis of right foot	Unknown	Unknown	11/03/2020	Positive	Negative

A systematic review by Bastos and colleagues examined 40 studies on the diagnostic accuracy of serological testing for COVID-19; examining the overall sensitivity and specificity of samples using ELISAs, lateral flow immunoassays (LFIAs) and CLIAs. They reported that LFIAs have demonstrated the lowest sensitivity (66%) compared to ELISA (84.3%) and CLIAs (97.8%), but that specificity varied minimally between methods (96.6–99.7%). Although Bastos and colleagues highlight a high level of agreement between the methods, they note a high risk of patient selection bias for 98% of the analyses in the 40 studies, because most studies selected cases and controls from different populations and did not use random or consecutive sampling.

This has significant implications for the interpretation of assay performance characteristics, as the prevalence of infection in these studies will not be reflective of those in populations where these assays may be applied. Whilst there is a clear connection between prevalence and the positive and negative predictive values of a diagnostic test, prevalence can also impact sensitivity and specificity due to the spectrum effect. This is particularly important where the comparator test used to define the gold standard is imperfect (such as using SARS-CoV-2 PCR results to define likely anti-SARS-CoV-2 antibody status) and when the test is used in a population where prevalence may differ significantly from those in the initial evaluations (such as a lookback exercise to identify potential early cases of SARS-CoV-2 infection). Care must therefore be taken when interpreting results from SARS-CoV-2 serological assays in scenarios where prevalence differs significantly from those in the original evaluations, as seen in our data when samples were taken during a period of very low prevalence, resulting in 10 potential false positive results using the DiaSorin assay.

Our study and those mentioned highlight the difficulties of interpreting serological results during low levels of prevalence but also the need for additional molecular testing. A positive or negative PCR result gives strength to serological results; however, when working with a novel agent molecular testing can be difficult to perform given the required time to develop a specific assay. ,

Gurgel used two LFIAs during their study but did not use a different serological technique for confirmation. Apolone used a single ELISA test and again did not use a different serological technique for confirmation. We used two separate assays which employ different serological detection techniques revealing that reliance on a single technique can lead to false results, a problem exacerbated during periods of low prevalence.

Our study along with Gurgel and Apolone were conducted retrospectively when SARS-CoV-2 would have had a low prevalence within the population and be at risk of higher rates of false positive results. Therefore, while high specificity of these antibody tests during periods of higher disease prevalence makes them ideal for seroprevalence work assessing how widely spread COVID-19 is within the population and determine the degree of asymptomatic transmission, caution must be urged in their use for identifying COVID-19 introduction to a population.

Authors’ contributions

PWB and JWT conceived the original study idea. PWB and OF performed the testing of samples. PWB and VB performed the analysis of data. All authors discussed the result, reviewed and revised drafts of the manuscript and gave final approval to submit the manuscript.

Ethical approval and consent to participate

Ethical approval and patient consent was not required for this study.

Declaration of Competing Interest

No authors have any conflicting interests to declare.