Comparison of diagnostic accuracies of rapid serological tests and ELISA to molecular diagnostics in patients with suspected coronavirus disease 2019 presenting to the hospital.

OBJECTIVES: To assess the diagnostic performance of rapid lateral flow immunochromatographic assays (LFAs) compared with an ELISA and nucleic acid amplification tests (NATs) in individuals with suspected coronavirus disease 2019 (COVID-19).
METHODS: Patients presenting to a Dutch teaching hospital were eligible between 17 March and 10 April 2020, when they had respiratory symptoms that were suspected for COVID-19. The performances of six different LFAs were evaluated in plasma samples obtained on corresponding respiratory sample dates of NATs testing. Subsequently, the best performing LFA was evaluated in 228 patients and in 50 sera of a historical patient control group.
RESULTS: In the pilot analysis, sensitivity characteristics of LFA were heterogeneous, ranging from 2/20 (10%; 95% CI 0%-23%) to 11/20 (55%; 95% CI 33%-77%). In the total cohort, Orient Gene Biotech COVID-19 IgG/IgM Rapid Test LFA had a sensitivity of 43/99 (43%; 95% CI 34%-53%) and specificity of 126/129 (98%; 95% CI 95%-100%). Sensitivity increased to 31/52 (60%; 95% CI 46%-73%) in patients with at least 7 days of symptoms, and to 21/33 (64%; 95% CI 47%-80%) in patients with C-reactive protein (CRP) ≥100 mg/L. Sensitivity and specificity of Wantai SARS-CoV-2 Ab ELISA was 59/95 (62%; 95% CI 52%-72%) and 125/128 (98%; 95% CI 95%-100%) in all patients, respectively, but sensitivity increased to 38/48 (79%; 95% CI 68%-91%) in patients with at least 7 days of symptoms.
CONCLUSIONS: There is large variability in diagnostic test performance between rapid LFAs, but overall limited sensitivity and high specificity in acutely admitted patients. Sensitivity improved in patients with longer existing symptoms or high CRP. LFAs should only be considered as additional triage tools when these may lead to the improvement of hospital logistics.

Introduction

In December 2019 the outbreak of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) started in Wuhan in China [1], but coronavirus disease 2019 (COVID-19) spread rapidly to other countries [2]. The first infected patient in the Netherlands was detected on 27 February 2020 [3]. Accurate diagnostics are fundamental in the fight against this increasing pandemic. Moreover, hospitals would benefit from rapid detection of this virus infection in individuals who present acutely to hospitals with respiratory symptoms suspected for COVID-19. Time delay in the establishment of diagnosis increases logistic challenges and causes stagnation of patient flow in emergency departments because these individuals cannot be transferred to appropriate hospital wards or intensive care units (ICUs) when the results of the diagnostic tests are still pending [4].

Nucleic acid amplification tests (NATs) are the reference standard because of the high specificity, although sensitivity may depend on the timing of disease presentation, sampling location and severity of illness [5]. Nevertheless, it usually takes about 4–24 hours before laboratory-based results become available, depending on specific NAT platforms and laboratory organization. Therefore, numerous lateral flow immunochromatographic assays (LFAs) have been introduced onto the market, and some countries have stocked up on such rapid tests. These LFAs detect the presence of IgM and IgG against SARS-CoV-2.

This study aimed to assess the diagnostic performance of LFAs, and compare these to an ELISA and NATs in individuals with suspected COVID-19.

Methods

Patients presenting to a teaching hospital in the Netherlands were eligible between 17 March 2020 and 10 April 2020 when they had respiratory symptoms that were suspected for respiratory tract infection. Samples were taken from the oral cavity and subsequently from the nasal cavity using the same nasopharyngeal swab; this was tested by NATs. In some cases, sputum samples were tested, because of persisting clinical suspicion of COVID-19 despite a negative NAT on nasopharyngeal swabs. NATs were performed according to the national reference method that was established after international collaboration [6], or by the CE-IVD kit GeneFinder™ COVID-19 Plus RealAmp Kit using the Sample to Result Platform ELITe InGenius®. The Institutional Review Board waived the need for informed consent because tests were performed on samples that had been acquired for routine clinical care (IRB protocol number 2020-034), and according to hospital procedure all patients were informed about the possibility of an opt-out if they had objections against the use of left-over material for research to improve or validate diagnostic testing procedures. The study was conducted in accordance with Helsinki Declaration as revised in 2013.

First, in a pilot phase, 20 NAT-positive and 5 NAT-negative patients were retrospectively selected for which six LFAs were performed on heparin plasma samples obtained upon hospital presentation (see Supplementary material, Fig. S1), which corresponded to the dates of molecular testing. LFAs were included from Boson Biotech, Cellex, Dynamiker Biotechnology, Orient Gene Biotech, Prometheus Bio and Wantai Rapid Test. Any visible band for IgG, IgM or unspecified immunoglobulin was indicative for a positive result. Second, based on the sensitivity and specificity results in the pilot study, the best performing LFA was further evaluated in an extended cohort of randomly selected patients. Third, this LFA was prospectively tested in consecutive patients between 6 April and 10 April. Fourth, specificity was also tested in a historical control group of randomly selected sera of 50 adult patients in September 2019 as SARS-CoV-2 was not circulating at that time. Finally, samples were also analysed by the Wantai SARS-CoV-2 Ab ELISA kit, which detects total antibodies, and interpreted according to the manufacturer's instructions. Both clinical information and reference standard results were unavailable to the performers of LFAs and the ELISA.

All analyses were performed using SAS 9.2 (Cary, NC, USA). We compared groups using non-parametric tests for continuous variables and chi-square test or Fisher's exact test for categorical variables as appropriate. Values of p that were <0.05 were considered to be statistically significant.

Results

In the pilot study, sensitivity characteristics of LFA were very heterogeneous, ranging from 2/20 (10%; 95% CI 0%–23%) to 11/20 (55%; 95% CI 33%–77%)) (Table 1 ). We decided to continue with the Orient Gene Biotech COVID-19 IgG/IgM Rapid Test (OGBRT) as it had the highest sensitivity.

Table 1

Sensitivity of lateral flow immunochromatographic assays in pilot study

Lateral flow immunochromatographic assay	Sensitivity	Specificity
Boson Biotech Rapid 2019-nCoV IgG/IgM Combo Test Card	10/20 (50%; 95% CI 28%–72%)	5/5 (100%; 95% CI 48%–100%)
Cellex qSARS-CoV-2 IgG/IgM Cassette Rapid Test	4/20 (20%; 95% CI 3%–38%)	5/5 (100%; 95% CI 48%–100%)
Dynamiker Biotechnology 2019-nCOV IgG/IgM Rapid Test	2/20 (10%; 95% CI 0%–23%)	5/5 (100%; 95% CI 48%–100%)
Orient Gene Biotech COVID-19 IgG/IgM Rapid Test Cassette	11/20 (55%; 95% CI 33%–77%)	5/5 (100%; 95% CI 48%–100%)
Prometheus Bio 2019-nCOV IgG/IgM Rapid Test	4/20 (20%; 95% CI 3%–38%)	5/5 (100%; 95% CI 48%–100%)
Wantai SARS-CoV-2 Ab	10/20 (50%; 95% CI 28%–72%)	5/5 (100%; 95% CI 48%–100%)

A total of 111 patients (including the 25 from the pilot study) were retrospectively selected between 16 March and 29 March. Subsequently, 117 consecutive patients were prospectively included between 6 April and 10 April. In total, 228 individuals were included with a median age of 61 years (interquartile range (IQR) 46–74 years), 117 (52%) were male, 21 (9%) were admitted to the ICU within 24 hours and median C-reactive protein (CRP) upon hospital presentation was 31 mg/L (IQR 7–95 mg/L) (see Supplementary material, Table S1). Median time from symptom onset to sample collection was 7 days (IQR 4–14 days).

OGBRT had an overall sensitivity of 43/99 (43%; 95% CI 34%–53%) and specificity of 126/129 (98%; 95% CI 95%–100%) (Table 2 ). Sensitivity increased to 31/52 (60%; 95% CI 46%–73%) in patients with at least 7 days of symptoms, and to 21/33 (64%; 95% CI 47%–80%) in patients with CRP ≥100 mg/L upon presentation. However, there was no significant difference between patients requiring ICU care within 24 hours after presentation and the remaining patients. Of the 43 individuals positive for both OGBRT and NAT, 14 were both IgG and IgM positive, 10 were only IgG positive and 19 were only IgM positive.

Table 2

Performance characteristics of Orient Gene Biotech LFA and Wantai ELISA in patients with suspected COVID-19 compared with nucleic acid amplification tests as reference standard

	All patients	Time from symptom onset to sample collection <7 days a	Time from symptom onset to sample collection ≥7 days a	p	Non-ICU	ICU	p	CRP <100 mg/L	CRP ≥100 mg/L	p
Orient Gene Biotech LFA
Sensitivity	43/99 (43; 95% CI 34–53)	11/39 (28; 95% CI 14–42)	31/52 (60; 95% CI 46–73)	<0.01	35/83 (42; 95% CI 32–53)	8/16 (50; 95% CI 26–75)	0.56	22/66 (33; 95% CI 22–45)	21/33 (64; 95% CI 47–80)	<0.01
Specificity	126/129 (98; 95% CI 95–100)	39/40 (98; 95% CI 93–100)	48/50 (96; 95% CI 91–100)	0.96	122/124 (98; 95% CI 96–100)	4/5 (80; 95% CI 45–100)	<0.01	109/111 (98; 95% CI 96–100)	17/18 (94; 95% CI 84–100)	0.33
Wantai ELISAb
Sensitivity	59/95 (62; 95% CI 52–72)	19/39 (49; 95% CI 33–64)	38/48 (79; 95% CI 68–91)	<0.01	49/80 (61; 95% CI 51–72)	10/15 (67; 95% CI 43–91)	0.69	36/65 (55; 95% CI 43–68)	23/30 (77; 95% CI 62–92	0.047
Specificity	125/128 (98; 95% CI 95–100)	39/40 (98; 95% CI 93–100)	48/50 (96; 95% CI 91–100)	1.00	121/123 (98; 95% CI 96–100)	4/5 (80; 95% CI 45–100)	0.11	108/110 (98; 95% CI 96–100)	17/18 (94; 95% CI 84–100)	0.37

Abbreviations: CRP, C-reactive protein; ICU, intensive care unit; LFA, lateral flow immunochromatographic assays.

Data are presented in absolute numbers (percentages). Of note, sensitivity was 17/24 (71%) in patients with ≥7 days from symptom onset to sample collection and CRP was ≥100 mg/L.

In some patients time from symptom onset was undetermined or unavailable. In the subgroup of patients with time from symptom onset to sample collection ≥14 days, sensitivity and specificity of the LFA were 9/14 (64%; 95% CI 39%–89%) and 30/32 (94%; 95% CI 85%–100%), respectively, whereas sensitivity and specificity of the ELISA were 6/12 (50%; 95% CI 22%–78%) and 30/32 (94%; 95% CI 85%–100%), respectively.

5/228 (2%) samples were unavailable for ELISA.

The ELISA showed sensitivity and specificity of 59/95 (62%; 95% CI 52%–72%) and 125/128 (98%; 95% CI 95%–100%), respectively. Sensitivity increased to 38/48 (79%; 95% CI 68%–91%) in patients with at least 7 days of symptoms, and to 23/30 (77%; 95% CI 62%–92%) in patients with CRP ≥100 mg/L. Overall agreement between the LFA and the ELISA was 195/223 (87%; 95% CI 83%–92%). In 21 NAT-positive patients the ELISA was positive and the LFA was negative, whereas in three NAT-positive patients the ELISA was negative and the LFA was positive (see Supplementary material, Fig. S2).

In the randomly selected historical control sera, the LFA and the ELISA specificities were 49/50 (98%; 95% CI 94%–100%) and 50/50 (100%; 95% CI 100%–100%), respectively; LFA showed a very weak IgG line in one sample.

Discussion

This study shows that the sensitivity of LFA was low in patients suspected for COVID-19 presenting to the hospital, but it improved in patients with at least 7 days of symptoms and in those with CRP levels >100 mg/L upon presentation. Specificities of LFAs and the ELISA were very high, and fulfilled a frequently used criterium of at least 98%. The ELISA had a higher sensitivity compared with LFAs.

Several countries, including Spain and the United Kingdom, have purchased one or more of these LFAs. However, our study findings underline that cautiousness is required when considering implementation of such tests. Interestingly, Cellex Rapid Test, which is currently the only rapid diagnostic test that is US Food and Drug Administration approved, performed less well than OGBRT in our pilot study. Another rapid test was reported to have a sensitivity <20% in acute patients referred to an emergency department [7]. Other studies showed higher sensitivities of LFAs up to 90% in unspecified patient groups with more time between disease onset and testing or missing information regarding timing of sampling [8,9]. Test performance characteristics as provided by manufacturers were higher than those observed in our study, which is related to a different selection of positive and negative controls. In our study we primarily included consecutive patients presenting to the hospital, which represents clinical practice and clinical sensitivity (i.e. diagnosing COVID-19 upon hospital presentation) rather than analytical sensitivity (i.e. detecting the presence of antibodies at that moment). The observed higher sensitivity in patients with at least 7 days of symptoms is in line with findings from other studies [10,11].

There are some study limitations to consider. This study included a wide comparison of six different LFAs, an ELISA and NATs, but more tests are available on the market. Nevertheless, both LFAs and the ELISA were limited in sensitivity, suggesting that antibody production is not always detectable or at least not yet detectable during the early phase of infection. Second, NAT as reference standard remains suboptimal, and it remains possible that in some cases actual infections were missed. In some patients NATs were only positive in sputum and negative in nasopharynx, whereas the majority of patients were only tested from nasopharyngeal swabs. Third, the subgroup of patients admitted to the ICU was limited, precluding definite conclusions in this group.

In conclusion, the high specificity of LFAs may contribute to rapidly confirm the presence of COVID-19, and accelerate decision-making in emergency rooms and routing to appropriate hospital wards. Yet negative LFA results are unreliable to exclude COVID-19 because of the limited sensitivity of these tests. Therefore, these LFA tests cannot replace molecular diagnostics in acute-care settings, but should only be used as an additional triage tool when improvement of hospital logistics is expected and their limitations are carefully considered.

Transparency declaration

The authors declare no conflicts of interest.

Funding

No funding.

Authors' contributions

DSYO and JGHK contributed to the conception and design of the study. DSYO, SJM and FAL acquired the data. DSYO and SJM analysed the data. All authors contributed to the interpretation of the data. DSYO drafted the first manuscript and all other authors revised it critically for important intellectual content. All authors approved this manuscript version to be submitted.