Analytical sensitivity and clinical sensitivity of the three rapid antigen detection kits for detection of SARS-CoV-2 virus.

BACKGROUND: Numerous rapid antigen detection (RAD) kits for diagnosing COVID-19 patients are available in the market recently.
OBJECTIVE: To compare analytical sensitivity and clinical sensitivity for the three commercially available RAD kits. STUDY
DESIGN: Analytical sensitivity for the detection of SARS-CoV-2 virus was determined by limit of detection (LOD) using RT-PCR as a reference method. Clinical sensitivity was evaluated by using respiratory specimens collected from confirmed COVID-19 patients.
RESULTS: The LOD results showed that the three RAD kits varied from 102-105 fold less sensitive than RT-PCR. Clinical sensitivity of RAD kits ranged from 22.9 %-71.4 % for detecting specimens from COVID-19 patients.
CONCLUSIONS: Although RAD kits were less sensitive than RT-PCR, understanding the clinical characteristics of different RAD kits can guide us to obtain suitable specimens for testing. The likelihood of positive results for RAD kits will be higher.

Introduction

The rapid diagnosis of COVID-19 patients is essential to reduce the disease spread. Besides RT-PCR, rapid antigen detection (RAD) kits for qualitative determination of SARS-CoV-2 antigen are available. RAD kits are less expensive than RT-PCR. In addition, the test results of RAD kits can be available without specialized instrument and interpreted within 30 min. Currently, WHO does not recommend specific commercial RAD kit for patient care. However, researches of different RAD kits are encouraged especially for the performance of sensitivity [1,2].

We previously evaluated a RAD kit for SARS-CoV-2 on May 2020 [3]. Since then, many new RAD kits are commercially available. Data on the performance of RAD kits varied between different studies and different brands [[4], [5], [6], [7], [8], [9]]. The purpose of this evaluation is to assess analytical sensitivity of the three SARS-CoV-2 RAD kits by means of limit of detection (LOD) using a set of serial tenfold dilution samples; and clinical sensitivity in detecting SARS-CoV-2 virus in different types of respiratory specimens. A direct comparison for these three kits has not been performed.

Methods

RAD kits for SARS-CoV-2 detection

The RAD kits selected in the present study were all commercially available in Hong Kong. These kits were either evaluated in previous studies or demonstrated good clinical performance mentioned in the kit inserts. The three kits evaluated were: (1) COVID-19 Ag Respi-Strip (Coris Bioconcept, Belgium), (2) NADAL COVID-19 Ag Test (Nal Von Minden GmbH, Germany), (3) Standard Q COVID-19 Ag (SD Biosensor, Korea). For the ease of communication, ‘Coris’, ‘NADAL’ and ‘Standard Q’ stand for these three kits respectively. The Standard Q was gifted by Roche for evaluation. It is available for sale in Hong Kong and is distributed by Roche. The name of the kit is ‘SARS-CoV-2 Rapid Antigen Test’.

These three kits employ a lateral flow test format to detect viral antigen by the immobilized coated SARS-CoV-2 antibody on the device. The test results can be interpreted without a reader at 15 min. The intended use for these kits is for the detection of SARS-CoV-2 virus in either nasopharyngeal swabs or secretions.

The procedures were carried out according to manufacturer’s instructions. For the NADAL, since our specimens were placed in viral transport media or phosphate-buffered saline, 350 μL specimen volume was mixed with 10 drops of extraction buffer (personal communication with Nal Von Minden GmbH). The subsequent procedures were carried out according to the manufacturer’s instructions.

Limit of detection for RAD kits

To determine LOD between different kits, a respiratory specimen was selected to perform a serial tenfold dilution. The selected specimen was nasopharyngeal aspirate and throat swab (NPA & TS) which was obtained from the Hong Kong COVID-19 patient, hCoV-19/Hong Kong/VM20030475/2020. To minimize the number of freeze-thaw cycle, the three kits were tested in parallel for this set of serial dilution at the same time.

The BIOCREDIT COVID-19 Ag (RapdGEN Inc. Korea) kit was also tested as a reference method simultaneously. Virus concentrations in each dilution were estimated from cycle threshold (Ct) value as described [3].

Respiratory specimens used for accessing clinical sensitivity

Ideally, the specimens selected should be tested in parallel for the three kits. In order to evaluate the three kits simultaneously, a total specimen volume of 800 μL is needed (Coris = 100 μL, NADAL = 350 μL, Standard Q = 350 μL). Due to the limited quantity for each specimen, the same specimen cannot be tested for all kits. Since both Coris and Standard Q shared similar operating procedures, they were tested together using the same set of specimens. Another set of specimens were tested for NADAL.

From March 16, 2020 to August 15, 2020, respiratory specimens from COVID-19 patients collected by the Public Health Laboratory Services Branch (PHLSB) in Hong Kong were retrieved for this evaluation. All of the specimens were confirmed with SARS-CoV-2 infection by RT-PCR as described [3].

Total number of 280 archive specimens with sufficient quantity (≥450 μL) was selected for this evaluation. These comprised four types of respiratory specimen: (1) throat saliva, (2) nasopharyngeal swab and throat swab (NPS & TS), (3) NPA & TS, and (4) NPS. These 280 archive specimens were divided into two sets, each set comprised 35 throat saliva, 35 NPS & TS, 35 NPA & TS and 35 NPS. The viral load for these two sets of specimens were roughly the same, specimens of various Ct values up to ∼34 were selected (which is close to the limit of detection for RT-PCR). The technicians performing the RAD kits were blinded to the RT-PCR results.

Respiratory isolates used for evaluating cross-reactivity

To evaluate the cross-reactivity of the RAD kits, 13 non-SARS-CoV-2 respiratory virus isolates were tested. They were influenza A(H1pdm09), influenza A(H3), influenza B, adenovirus, coronavirus type OC43, coronavirus type 229E, parainfluenza virus type 1, parainfluenza virus type 2, parainfluenza virus type 3, parainfluenza virus type 4, respiratory syncytial virus, rhinovirus and enterovirus.

Results of RAD kits interpretation

All of the kits evaluated were not involved using readers for determining a positive or negative result. Visual output of the devices was read by two technicians. In case of disagreement, the visual output was read by the third technician.

Results

The LOD of the reference RAD kit, BIOCREDIT COVID-19 Ag was 105 fold less sensitive than RT-PCR (BIOCREDIT COVID-19 Ag: 10−2; RT-PCR: 10−7) which was concordant to our previous study in May 2020.

As shown in Table 1 , the three RAD kits shared the same LOD at 10−2, they were BIOCREDIT COVID-19 Ag, Coris and NADAL. The LOD of Standard Q was 10−5. The corresponding Ct values for LOD at 10−2 and 10−5 were 18.44 and 28.67 respectively. Although viral culture was not performed in the present study, the Standard Q was 102 fold less sensitive than RT-PCR, it corresponded to the LOD of viral culture based on our results reported previously.

Table 1

Comparison of RT-PCR and rapid antigen detection kits for the limit of detection of SARS−COV-2 virus.

Dilutionb	Test resultsa
	RAD kit				RT-PCRc
	BIOCREDIT	Coris	NADAL	Standard Q
10−1	POS	POS	POS	POS	15.15
10−2	POS	POS	POS	POS	18.44
10−3	NEG	NEG	NEG	POS	21.54
10−4	NEG	NEG	NEG	POS	24.64
10−5	ND	ND	ND	POS	28.67
10−6	ND	ND	ND	NEG	31.90
10−7	ND	ND	ND	ND	34.87
10−8	ND	ND	ND	ND	NEG

POS, positive; NEG, negative; ND, not done.

Serial tenfold dilution of the respiratory specimen, NPA & TS, obtained from the Hong Kong COVID-19 patient, hCoV-19/Hong Kong/VM20030475/2020.

RT-PCR were tested twice with identical results. The Ct values shown were the mean of both runs.

Since the cut-off of 18.57 for BIOCREDIT COVID-19 Ag was utilized in our previous study in May 2020, it was also used in the present study for consistency to classify specimens as ‘high viral load’. Specimens with Ct values between 18.57 and 28.67 were classified as ‘normal viral load’ while specimens >28.67 were classified as ‘low viral load’.

Of the 280 specimens tested, 72, 132 and 76 specimens were classified as ‘high viral load’, ‘normal viral load’ and ‘low viral load’ specimens respectively. The ratio was approximately 1:2:1 which was similar to the viral load distribution of specimens collected by PHLSB. RAD kits showed high sensitivity (77.8 %–100 %) for high viral load specimens but low sensitivity (0–11.1 %) for low viral load specimens. For normal viral load specimens, one kit was capable of having sensitivity of ≥75 % while the other two kits ranged from 6.3 %–56.3 %. Review of the Ct values showed that specimens missed by the RAD kits had relatively high Ct values (Supplementary Figures). Invalid results were found for six specimens while 83.3 % (5/6) belonged to throat saliva. The corresponding mean Ct value, Ct value range and sensitivity for NPA & TS, NPS & TS, throat saliva and NPS specimens among the four group of specimens tested for Coris, NADAL and Standard Q were shown in Table 2 .

Table 2

Performance characteristics of the rapid antigen detection kits for the presence of SARS-CoV-2 virus in respiratory specimens.

Specimen typea	Coris					Nadal					Standard Q
	Ct value		No. of specimens		sensitivity	Ct value		No. of specimens		sensitivity	Ct value		No. of specimens		sensitivity
	mean	range	tested	positive		mean	range	tested	positive		mean	range	tested	positive
NPA & TS
High	16.70	13.90−18.24	9	7	77.8 %	16.27	11.17−18.20	9	9	100 %	16.70	13.90−18.24	9	9	100 %
Normal	24.07	19.90−28.65	16	1	6.3 %	23.77	19.19−28.67	18	3	16.7 %	24.07	19.90−28.65	16	12	75.0 %
Low	31.83	28.69−33.81	10	0	0 %	31.50	29.17−33.69	8	0	0 %	31.83	28.69−33.81	10	0	0 %
All	24.39	13.90−33.81	35	8	22.9 %	23.61	11.17−18.20	35	12	34.3 %	24.39	13.90−33.81	35	21	60.0 %
NPS & TS
High	15.96	12.94−18.43	9	9	100 %	15.81	11.82−18.49	9	9	100 %	15.96	12.94−18.43	9	9	100 %
Normal	23.72	19.75−28.59	16	5	31.3 %	23.60	19.72−28.29	16b	3	18.8 %	23.72	19.75−28.59	16	15	93.8 %
Low	32.04	29.00−34.24	10	0	0 %	31.96	28.88−33.99	10	0	0 %	32.04	29.00−34.24	10	1	10 %
All	24.10	12.94−34.24	35	14	40 %	23.98	11.82−33.99	35	12	34.3 %	24.10	12.94−34.24	35	25	71.4 %
NPS
High	16.38	14.56−18.07	9	9	100 %	16.05	13.30−17.84	9	9	100 %	16.38	14.56−18.07	9	9	100 %
Normal	23.44	18.98−27.51	16	5	31.3 %	23.31	18.89−28.38	16	9	56.3 %	23.44	18.98−27.51	16	13	81.3 %
Low	31.73	28.91−34.47	10	0	0 %	31.56	28.81−33.80	10	0	0 %	31.73	28.91−34.47	10	1	10 %
All	23.99	14.56−34.47	35	14	40 %	23.80	13.30−33.80	35	18	51.4 %	23.99	14.56−34.47	35	23	65.7 %
throat saliva
High	15.30	11.46−18.37	9b	8	88.9 %	15.27	11.32−18.36	9c	7	77.8 %	15.30	11.46−18.37	9	9	100 %
Normal	23.21	18.68−28.62	17b	3	17.6 %	23.20	18.66−28.62	17	4	23.5 %	23.21	18.68−28.62	17d	15	88.2 %
Low	31.06	29.18−33.31	9	0	0 %	31.04	29.16−33.30	9b	0	0 %	31.06	29.18−33.31	9	1	11.1 %
All	23.20	11.46−33.31	35	11	31.4 %	23.18	11.32−33.30	35	11	31.4 %	23.20	11.46−33.31	35	25	71.4 %

‘High’, means specimens with Ct values <18.57 of SARS-CoV-2 virus RT-PCR; ‘Normal’, Ct values between 18.57 and 28.67; ‘Low’, Ct values >28.67.

The control band was failed to appear for one specimen.

The control band was failed to appear for two specimens.

The nozzle cap was blocked for one specimen, 100 μL of suspension was added directly to the device. The final test result was positive.

In the cross-reactivity test using virus isolates, all were tested negative by the RAD kits.

Discussion

The purpose of this study is to evaluate if the commercially available RAD kits are suitable for diagnosing SARS-CoV-2 infection, however, ranking of RAD kits is not our objective. Our study demonstrated a good correlation between LOD and clinical sensitivity. It means that the sensitivity of RAD kits is viral load dependent. Viral load depends on whether the patients are symptomatic and the course of the disease. It should be noted that although our data indicated that RAD kits were capable of detecting SARS-CoV-2 virus in throat saliva, majority of invalid results were found in throat saliva which should be due to the viscous nature of throat saliva. Similar findings were also observed in our previous study [3]. Throat saliva is not recommended as a specimen of choice where the collection of other types of specimens is available.

Understanding the clinical characteristics of different RAD kits can guide us to obtain suitable specimens for testing. The performance of RAD kits could be tailored by targeting a specific patient group. Specimens obtained ≤5 days after symptom onset are most likely contain high viral loads [10]. We have analyzed the proportion of specimen for the two cut-offs, Ct ≤18.57 and Ct ≤28.67, according to days after symptom onset from ≤ day 1 to ≤ day 7. A total of 3932 specimens were available for analysis. These specimens comprised throat saliva, NPS & TS, NPA & TS, NPS collected from 31 Jan 2020 to 13 Sep 2020. Information of date of symptom onset for each specimen was collected [11]. We found that viral loads of some specimens collected early after symptom onset have similar viral loads for specimens collected late after symptom onset. It means that if we set a cut off by means of ≤ day X after symptom onset for performing RAD kits, we will miss another group of specimens having a similar high viral load. We found that for Ct ≤18.57, testing specimens obtained ≤5 days after symptom onset will likely to match the high viral load specimens while keeping the minimum proportion of specimens having a similar high viral load that were missed. For Ct ≤28.67, specimens obtained ≤7 days after symptom onset are optimum (Supplementary Tables). This observation was concordant to the WHO guideline that symptomatic cases should be tested within the first 5–7 days of illness [2].

The Standard Q might be suitable for use as an aid to early diagnosis of SARS-CoV-2 infection in terms of analytical and clinical sensitivity. This kit is listed in the ‘WHO Emergency Use Listing for In vitro diagnostics (IVDs) Detecting SARS-CoV-2’ [12]. The exact time period after symptom onset for use is not mentioned by the Standard Q. A recent study showed that the sensitivity can be decreased from 83.9 %–76.3 % when specimens obtained ≤5 and ≤7 days after symptom onset respectively [13]. Our data showed that although the prevalence of specimens with Ct ≤28.67 decreased from 81.8 %–79.9 % when specimens obtained ≤5 and ≤7 days after symptom onset respectively, the number of specimens with Ct ≤28.67 that were skipped for testing was decreased from 18.2 %–9.4 % (Table S2). Therefore, we recommended specimens obtained ≤7 days after symptom onset for use with the Standard Q. Then, the RAD kit can serve as a COVID-19 filter (filtered out of the infected persons and prevent spread to the others). The balance between turnaround time and sensitivity should be considered. A diagnostic test with high analytical sensitivity can still be failed to be a COVID-19 filter [14].

The limitation of this study is to use Ct value to estimate viral load. Ct values for a given target RNA can vary between RT-PCR assays and both intra- and inter-PCR runs. However, the same in-housed developed RT-PCR assay is consistently used for diagnosing SARS-CoV-2 infection in PHLSB and also our evaluations, variations of Ct values can be minimized. Although Ct is a semiquantitative value, it can also be used for guiding infection control and public health decisions [15,16].

It is expected that many RAD kits for diagnosing SARS-CoV-2 infection are emerged in this exploding field. Access to laboratory services based on rapid diagnostic tests alone is common in developing countries [17]. With the lack of regulatory policies, laboratories took the responsibility for governance and quality control for rapid diagnostic tests [18]. The difficulty of evaluating RAD kits is limited by collecting a large number of positive specimens of different types. The LOD can predict the performance of RAD kits and it has been utilized in RAD kits for diagnosing influenza [19]. In future, the LOD approach will be performed first in our setting. When the LOD results demonstrate better results than the reference method, a full evaluation using clinical specimens of various concentrations will be performed. It should be noted than when determining clinical sensitivity, if only specimens of high viral load were selected, every kit will have a 100 % clinical sensitivity [20].

In conclusion, adoption of RAD kit for clinical use should be guided by evidences. When the study is well-performed to understand the clinical characteristics, the RAD kit will be safe to use.

CRediT authorship contribution statement

Gannon CK Mak: Conceptualization, Methodology, Validation, Investigation, Writing - original draft, Writing - review & editing. Stephen SY Lau: Validation, Investigation. Kitty KY Wong: Validation, Investigation. Nancy LS Chow: Validation, Investigation. CS Lau: Resources, Supervision. Edman TK Lam: Supervision. Rickjason CW Chan: Supervision. Dominic NC Tsang: Supervision, Writing - review & editing.

Declaration of Competing Interest

None.