Comparative diagnostic performance of different rapid antigen detection tests for COVID-19 in the real-world hospital setting.

BACKGROUND: The availability of accurate and rapid diagnostic tools for COVID-19 is essential for tackling the ongoing pandemic. The aim of this study was to quantify the performance of different available types of antigen-detecting rapid diagnostic tests (Ag-RDTs) in the real-world hospital setting.
METHODS: In this retrospective analysis the diagnostic performance of seven different Ag-RDTs was compared with the real-time reverse transcription quantitative polymerase chain reaction (RT-qPCR) in terms of sensitivity, specificity and expected predictive values.
RESULTS: A total of 321 matched Ag-RDT-RT-qPCR samples were analyzed retrospectively. The overall sensitivity and specificity of Ag-RDTs were 78.7% and 100%, respectively. However, a wide range of sensitivity estimates by brand (66.0-93.8%) and cycle threshold (Ct) cut-off values (Ct <25: 96.2%; Ct 30-35: 31.1%) was observed. The optimal Ct cut-off value that maximized sensitivity was 29.
CONCLUSIONS: The routine use of Ag-RDTs may be convenient in moderate-to-high intensity settings when high volumes of specimens are tested every day. However, diagnostic performance of the commercially available tests may differ substantially.

Introduction

The ongoing SARS-CoV-2 pandemic has been associated with a significant burden and unprecedented pressure on healthcare systems (McArthur et al., 2020, Greene et al., 2020, Rahimi and Talebi Bezmin Abadi, 2020). The availability of accurate and rapid diagnostic tools for COVID-19 is therefore essential for both active monitoring of cases and contact tracing strategies in order to reduce the circulation of the COVID-19 causative agent (Greene et al., 2020, Rahimi and Talebi Bezmin Abadi, 2020, Venter and Richter, 2020, Hu et al., 2021).

Etiological diagnosis of SARS-CoV-2 infection may be performed by directly identifying the viral ribonucleic acid or antigens or by indirectly identifying specific antibodies. The direct methods include, for example, real-time reverse transcription quantitative polymerase chain reaction (RT-qPCR), transcription-loop-mediated isothermal amplification, and antigen-based tests. Indirect diagnostics, useful for establishing previous exposure to the virus and/or vaccine, may be done through enzyme-linked immunoassay, chemiluminescence immunoassay and other serological tests (Venter and Richter, 2020, Russo et al., 2020).

Owing to its high diagnostic performance, RT-qPCR is the “gold standard” assay for the laboratory diagnosis of both symptomatic and asymptomatic COVID-19 cases (Russo et al., 2020). To the best of our knowledge, the RT-qPCR technique is included in all principal international diagnostic protocols, including that issued by the World Health Organization (WHO) (2020a). However, from a public health perspective, RT-qPCR has some limitations, including relatively high cost, need for qualified personnel and a suboptimal turnaround time (Russo et al., 2020). To address RT-qPCR shortcomings, antigen-detecting rapid diagnostic tests (Ag-RDTs) have been quickly developed and are increasingly common in the clinical setting with many brands available (World Health Organization (WHO, 2020a; ECDC, 2020). If Ag-RDTs are accurate, they may have a greater public health impact than RT-qPCR for the following reasons: (i) no need for high-level technical expertise and laboratory capacity; (ii) may be performed locally in a decentralized locality with the associated logistic advantages; (iii) may facilitate timely decisions regarding quarantine and/or treatment regimens and epidemiological investigations of novel clusters (Dinnes et al., 2021; ECDC, 2020).

On the other hand, Ag-RDTs are less sensitive than RT-qPCR (ECDC, 2020). A recent Cochrane review (Dinnes et al., 2021) highlighted considerable between-study variability in sensitivity and specificity estimates; these also varied by Ag-RDT brand and viral load. For instance, a subgroup analysis by viral load defined by the cycle threshold (Ct) quantified a 53.8% absolute difference in sensitivity between samples with Ct ≤25 and >25 (Dinnes et al., 2021). The WHO (2020b) recommends that SARS-CoV-2 Ag-RDTs have a minimum of 80% sensitivity and 97% specificity. The European Centre for Disease Control and Prevention (ECDC) (2020) has recently proposed a more conservative threshold of ≥90% for the sensitivity parameter, especially in low-incidence settings.

Our rationale for this retrospective analysis is the paucity of data comparing the performance of Ag-RDTs in the hospital setting. Moreover, the ECDC (2020) has recommended that European countries perform an independent and setting-specific evaluation of Ag-RDTs before widespread implementation. Thus, our primary goal was to assess the diagnostic accuracy of Ag-RDTs in a real-world setting compared with RT-qPCR. Moreover, considering the recent significant evolution of Ag-RDTs, we also aimed to compare the diagnostic accuracy of different Ag-RDTs assays.

Methods

Study design and setting

We conducted this retrospective analysis at the regional reference laboratory for COVID-19 diagnostics located in San Martino Policlinico Hospital, Hygiene Unit (Genoa, Italy). To be included in the study, samples had to have been tested by both Ag-RDT (in urgent routine modality) and RT-qPCR assay (following confirmation of the Ag-RDT output). All available matched samples collected between July and December 2020 were eligible. Swabs were performed using a flocked probe and were eluted in the universal transport medium (UTM, Copan Diagnostics Inc, US). All tests were performed within 8 h from the arrival of samples at the laboratory.

No formal ethical approval for this retrospective study was needed because it was conducted as part of routine SARS-CoV-2 testing.

Real-time reverse transcription quantitative polymerase chain reaction

Each sample underwent the extraction-free RT-qPCR on Nimbus IVD (Seegene Inc., Republic of Korea) using the Allplex™ SARS-COV-2 Assay kit (Seegene Inc., Republic of Korea), following the manufacturer’s protocol. The obtained material was tested for SARS-CoV-2 through a one-step multiplex RT-qPCR on Bio-Rad CFX96™ thermal cycler (Bio-Rad Laboratories, US). Gene amplifications were then tested by FAM (E gene), HEX (internal control), Cal Red 610 (RdRP – RNA-dependent RNA-polymerase gene) and Quasar 670 (N gene) fluorophores. Final results were interpreted using the 2019-nCoV viewer (Seegene Inc., Republic of Korea) following the manufacturer’s instructions. Samples showing Ct values <35 for the target genes were considered positive.

Antigen-detecting rapid diagnostic tests

The Ag-RDTs used were classified into 2 categories. The first was lateral flow immunochromatographic tests (LFTs) and included: STANDARD Q COVID-19 Ag (SD Biosensor, Republic of Korea), Humasis COVID-Ag Test (Humasis, South Korea), COVID-19 Antigen Rapid Test Prima Professional (Prima Lab, Switzerland) and BIOCREDIT COVID-19 Ag (RapiGEN Inc., Republic of Korea). The second was fluorescent immunoassay (FIA) tests and included: STANDARD™ F COVID-19 Ag FIA (Biosensor, Republic of Korea), LumiraDx SARS-CoV-2 Ag Test (LumiraDx UK Ltd) and FREND COVID-19 Ag test (NanoEntek, Republic of Korea).

The Ag-RDTs were performed following the manufacturer’s instructions.

Data analysis

RT-qPCR was considered as a reference standard against Ag-RDTs (Foundation for Innovative New Diagnostics (FIND, 2020). On the basis of Ct values, and therefore viral load, RT-qPCR positive samples were divided into 3 categories: <25 (high load), 25–29.9 (medium load) and 30–34.9 (low load). Infectiousness is likely associated with high viral loads with Ct values of <25/30, and Ag-RDTs are expected to perform best in these cases (World Health Organization (WHO, 2020a; ECDC, 2020; Dinnes et al., 2021).

Diagnostic performance of Ag-RDTs was assessed through calculation of sensitivity and specificity overall and stratified by assay type and Ct value categories. A receiving operation curve was then constructed and the area under the curve quantified. An optimal cut-off of the Ct value was estimated using Youden’s J statistic. The expected positive (PPV) and negative (NPV) predictive values were calculated from the overall sensitivity and specificity and hypothetical positivity prevalence of 0.5%, 1%, 5%, 10% and 20% (ECDC, 2020; Dinnes et al., 2021).

All analyses were performed using R stats packages v. 4.0.3 (R Core Team, 2020).

Results

Characteristics of samples analyzed

A total of 321 samples were included in the analysis. Of these, 21.2% (n = 68) were found negative by RT-qPCR. The remaining 78.8% (n = 253) tests were positive: 41.1% (95% CI: 35.0%–47.4%), 41.1% (95% CI: 35.0%–47.4%) and 17.8% (95% CI: 13.3%–23.1%) had low, medium and high Ct values, respectively. The summary distribution of Ct values of the positive samples is shown in Figure 1. Of the Ag-RDTs used 37.1% (n = 119) and 62.9% (n = 202) belonged to LFT and FIA types, respectively.

Figure 1

Distribution of the cycle threshold values of RT-qPCR positive samples, by gene region.

Ct: cycle threshold; RdRp: RNA-dependent RNA-polymerase; RT-qPCR: reverse transcription quantitative polymerase chain reaction.

Overall diagnostic performance of Ag-RDTs used

Table 1 reports raw data on the performance of the Ag-RDTs analyzed: the proportion of false-negative results was 21.3% (n = 54), while no false-positive specimens were revealed. The overall sensitivity and specificity were therefore 78.7% (95% CI: 73.2%–83.3%) and 100% (95% CI: 94.7%–100%), respectively. The expected NPVs at the positivity rate of 0.5%, 1%, 5%, 10% and 20% were 99.9%, 99.8%, 98.9%, 97.7% and 94.9%, respectively, while the expected PPV was constantly 100%. The receiving operation curve constructed using the average Ct value vs positive Ag-RDT showed an area under the curve of 0.88 (95% CI: 0.82–0.93) with an optimal Ct cut-off value of 29.

Table 1

Two-per-two table on the performance of the antigen-detecting rapid diagnostic tests used (results are reported as % (n)).

Ag-RDTs	RT-qPCR		Total
	Positive	Negative
Positive	62.0 (199)	0 (0)	62.0 (199)
Negative	16.8 (54)	21.2 (68)	38.0 (122)
Total	78.8 (253)	21.2 (68)	100 (321)

Ag-RDT: antigen-detecting rapid diagnostic test; RT-qPCR: reverse transcription quantitative polymerase chain reaction.

Sensitivity of Ag-RDTs according to the cycle threshold and test type

There was a significant (as shown by non-overlapping 95% CIs) relationship between sensitivity and Ct values: low Ct value specimens were associated with a sensitivity of 96.2%, while those with high Ct values had a sensitivity of only 31.1%. The use of FIA tests was associated with a lower number of false negatives, independently from Ct. However, LFTs performed reasonably well at low-to-medium Ct values (Table 2 ).

Table 2

Sensitivity of the antigen-detecting rapid diagnostic tests used, by assay type and cycle threshold category (results are reported as % (95% CI)).

Ag-RDT type (n)	Cycle threshold
	<25	25–29.9	<30	30–34.9	<35
LFT (119)	91.7 (80.5–96.7)	70.7 (55.5–82.3)	82.0 (72.8–88.6)	9.5 (2.7–28.9)	68.2 (59.0–76.2)
FIA (202)	100 (93.6–100)	88.9 (78.8–94.5)	94.1 (88.4–97.1)	50.0 (31.4–68.6)	86.7 (80.2–91.3)
Any (321)	96.2 (90.5–98.5)	81.7 (73.2–88.0)	88.9 (84.0–92.5)	31.1 (19.5–45.7)	78.7 (73.2–83.3)

Ag-RDT: antigen-detecting rapid diagnostic test; FIA: fluorescent immunoassay; LFT: lateral flow immunochromatographic test.

On the other hand, there was a high between-brand heterogeneity of results (Table 3 ). The best performing assays were STANDARD Q COVID-19 Ag and FREND COVID-19 Ag.

Table 3

Sensitivity of the antigen-detecting rapid diagnostic tests used, by brand and cycle threshold (Ct) category (results are reported as % (95% CI)).

Antigen-detecting rapid diagnostic tests	n	Ct <30	Ct <35
STANDARD Q COVID-19 Ag	16	100 (78.5–100)	93.8 (71.7–98.9)
Humasis COVID-Ag	21	89.5 (68.6–97.1)	85.7 (65.4–95.0)
COVID-19 Antigen Rapid Test Prima Professional	50	78.6 (64.1–88.3)	66.0 (52.2–77.6)
BIOCREDIT COVID-19 Ag	23	64.3 (38.8–83.7)	39.1 (22.2–59.2)
STANDARD F COVID-19 Ag	60	100 (92.9–100)	86.7 (75.8–93.1)
LumiraDx SARS-CoV-2 Ag	23	73.7 (51.2–88.2)	69.6 (49.3–84.4)
FREND COVID-19 Ag	60	100 (92.9–100)	93.3 (84.1–97.4)

Discussion and conclusions

In the present study, we have evaluated the diagnostic performance of different Ag-RDTs in a real-world hospital setting. The overall sensitivity of Ag-RDTs was approximately 79%, while the specificity was 100% with no false-positive results. On the other hand, the sensitivity of Ag-RDTs increased up to 96.2% for high viral load (Ct <25) samples. Therefore overall sensitivity was driven by a relatively high frequency of false-negative results among specimens with Ct values of 30–35. On average, FIA tests performed better than LFTs, although a substantial between-brand heterogeneity was observed.

To the best of our knowledge, there is still an ongoing debate on which RT-qPCR cut-off value should be used to indicate that a specimen is positive, weakly positive or negative. In this regard, a recent systematic review (Jefferson et al., 2020) has established that Ct values were significantly lower in samples producing live virus culture. Bullard et al. (2020) have estimated that the infectiousness (defined in that study by growth in the Vero cell culture) has been significantly reduced at Ct >24; this may mean that RT-qPCR positivity persists beyond infectiousness. In our study, to align with the previous body of primary research identified in available systematic reviews (Van Walle et al., 2020, Dinnes et al., 2021), we used a more conservative Ct threshold of 35. Indeed, we found that the optimal Ct-value cut-off that maximized sensitivity was 29; the same Ct cut-off with a sensitivity of 92% has been recently documented by Nalumansi et al. (2021).

A systematic review on the diagnostic accuracy of Ag-RDTs against RT-qPCR performed by ECDC researchers (Van Walle et al., 2020) has documented a high variability of sensitivity estimates (range 29%–93.9%), while the specificity was constantly high (range 98.8%–100%). The more recent Cochrane review (Dinnes et al., 2021) has reported a wider range for both sensitivity (0%–94%) and specificity (90%–100%). The pooled results obtained by Dinnes et al. (2021) are consistent with the sensitivity estimates obtained in our study. For instance, for samples with Ct values <25, we calculated sensitivity of 96.2% (95% CI: 90.5%–98.5%) which is in line with the Cochrane review estimate of 94.5% (95% CI: 91.0%–96.7%). By contrast, as compared with the Cochrane review, we found a higher overall sensitivity for specimens with Ct values >25 (66.4% [95% CI: 58.5%–73.5%] vs 40.7% [95% CI: 31.8%–50.3%]). This finding may be partially explained by one-third of the Ag-RDTs belonged to high performing FIA tests (STANDARD Q COVID-19 Ag and FREND COVID-19 Ag), while Dinnes et al. (2021) mainly dealt with LFTs. Apart from their higher sensitivity, FIA tests also have an advantage of a shorter readout time. On the other hand, these rapid tests may have higher purchase costs; a critical assessment of the local epidemiological situation with clear cost-benefit reasoning may aid the decision-making process.

More generally, our results are consistent with the recent suggestions proposed by the ECDC (2020). Thus, in low COVID-19 prevalence, Ag-RDTs would be associated with low PPVs and high NPVs. Ag-RDTs may be useful to screen positive patients with high viral loads; these, however, should be subsequently confirmed by RT-qPCR. Negative tests may not require a subsequent RT-qPCR. By contrast, in settings with high incidence, positive results will most probably identify true positives with no need for subsequent RT-qPCR, while for negative tests the ECDC suggests an immediate RT-qPCR (ECDC, 2020). Again, cost-benefit reasoning should be applied in this circumstance.

We acknowledge that our study may suffer from some limitations. First, although the overall sample size was comparatively large (the medium sample size of 77 studies included in the Cochrane review by Dinnes et al. (2021) was 182), it was skewed to RT-qPCR positive samples. It has been suggested (Foundation for Innovative New Diagnostics (FIND, 2020) that in retrospective validation studies of SARS-CoV-2 Ag-RDTs a minimum number of both RT-qPCR positive and negative samples should be 100. In this study we had only 68 negative samples. On the other hand, shifting a positivity cut-off to Ct <30 (Dinnes et al., 2021; ECDC, 2020) would produce a total of 105 RT-qPCR negative samples. Thus we believe our study results are sufficiently powered. Second, during the study period, the availability of single Ag-RDTs differed substantially; therefore, some tests were underrepresented. In particular, a post hoc power analysis for the observed sensitivity revealed that the estimates reported for COVID-19 Antigen Rapid Test Prima Professional (Prima Lab, Switzerland), BIOCREDIT COVID-19 Ag (RapiGEN Inc., Republic of Korea) and LumiraDx SARS-CoV-2 Ag Test (LumiraDx UK ltd.) were likely underpowered (α < 0.8). Third, the last specimen analyzed was collected in December 2020; therefore, the SARS-CoV-2 population at the time of study may not represent the current situation. The rapid diffusion of novel SARS-CoV-2 variants (Kupferschmidt, 2021, Mahase, 2021) may interfere with the accuracy of Ag-RDTs available; therefore, continuous monitoring of the performance of Ag-RDTs is warranted. Finally, for privacy reasons, we could not link laboratory test results to the sociodemographic and clinical features of patients.

To conclude, our study demonstrated that some Ag-RDTs fulfilled the required diagnostic performance criteria proposed by the WHO (2020a; 2020b) and ECDC (2020), especially in patients with high viral loads. Rapid point-of-care antigen tests are therefore valuable in everyday clinical practice for their ease of use with minimum training time, availability of results in a few minutes and very high specificity. We believe that the use of Ag-RDTs may be particularly convenient in moderate-to-high intensity settings when high volumes of specimens must be tested every day. In fact, our results suggest that at the positivity rate of 10%–20% the expected PPV and NPV are close to being optimal. In such circumstances, positive rapid test samples do not need to be further confirmed by RT-qPCR, which would undoubtedly alleviate pressure on specialized healthcare facilities, especially in resource-constrained settings.

Statement of ethics

The study was conducted according to the guidelines of the Declaration of Helsinki. Ethical review and approval were waived for this retrospective analysis because it was based on routine COVID-19 testing.

Conflicts of interest

The authors declare no conflict of interest regarding this publication.

Funding/support

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author contributions

Concept and design: Bruzzone, Orsi, Da Rin, Icardi. Acquisition, analysis and/or interpretation of data: De Pace, Caligiuri, Ricucci, Guarona, Pennati, Boccotti. Drafting of the manuscript: Domnich, Bruzzone. Statistical analysis: Domnich, Orsi.