A nationwide analytical and clinical evaluation of 44 rapid antigen tests for SARS-CoV-2 compared to RT-qPCR.

BACKGROUND: The SARS-CoV-2 pandemic has resulted in massive testing by Rapid Antigen Tests (RAT) without solid independent data regarding clinical performance being available. Thus, decision on purchase of a specific RAT may rely on manufacturer-provided data and limited peer-reviewed data.
METHODS: This study consists of two parts. In the retrospective analytical part, 33 RAT from 25 manufacturers were compared to RT-PCR on 100 negative and 204 positive deep oropharyngeal cavity samples divided into four groups based on RT-PCR Cq levels. In the prospective clinical part, nearly 200 individuals positive for SARS-CoV-2 and nearly 200 individuals negative for SARS-CoV-2 by routine RT-PCR testing were retested within 72 h for each of 44 included RAT from 26 manufacturers applying RT-PCR as the reference method.
RESULTS: The overall analytical sensitivity differed significantly between the 33 included RAT; from 2.5% (95% CI 0.5-4.8) to 42% (95% CI 35-49). All RAT presented analytical specificities of 100%. Likewise, the overall clinical sensitivity varied significantly between the 44 included RAT; from 2.5% (95% CI 0.5-4.8) to 94% (95% CI 91-97). All RAT presented clinical specificities between 98 and 100%.
CONCLUSION: The study presents analytical as well as clinical performance data for 44 commercially available RAT compared to the same RT-PCR test. The study enables identification of individual RAT that has significantly higher sensitivity than other included RAT and may aid decision makers in selecting between the included RAT. FUNDING: The study was funded by a participant fee for each test and the Danish Regions.

Introduction

During the current SARS-CoV-2 pandemic, massive testing has been applied with the aim to contain the spread of the virus. The test strategy was emphasized by the WHO in March 2020 as the backbone of the global pandemic response together with isolation and contact tracing that combined with social distancing and hand hygiene would allow “to extinguish” the pandemic [1].

The unprecedented demand for testing has led to a shortage of reagents and manufactured tests for reverse transcription polymerase chain reaction (RT-PCR) and other nucleic acid amplification technology (NAAT) methods including point of care testing (PoCT). Thus, lateral flow based rapid antigen tests (RAT) have been deployed extensively due to delivery of results within minutes, availability, low cost and the ease of use that allow non-healthcare trained individuals to perform the test outside of healthcare facilities [2,3].

Several approaches to evaluate real-life performance of RAT has been applied, including studies on cultured virus, retrospective studies evaluating analytical performance data and clinical evaluations [4], [5], [6], [7], [8], [9], [10], [11], [12], [13].

Viral culture studies have shown that limit of detection (LOD) of RAT differs significantly and varies depending on spike variants of SARS-CoV-2 [4,14]. SARS-CoV-2 variants contain several mutations in the nucleocapsid gene together with mutations in the spike gene, which may account for the difference in LOD [15].

Clinical studies have shown that sensitivity of a single RAT may vary at least 20 percentage point between non-symptomatic and symptomatic individuals [9,[16], [17], [18]]. As most clinical studies only include a single or few different RAT and as the tested population varies between different clinical studies regarding SARS-CoV-2 prevalence, proportion of symptomatic versus non-symptomatic individuals, vaccination status and demographic profile, comparing performance data between different studies is difficult [[9], [10], [11],19,20]. Due to the absence of clinical studies that allow for comparison of the different performances of commercially available RAT and the huge amount of financial resources spent on RAT, the Danish Regions initiated a nationwide clinical study comparing different RAT. This study combines a retrospective analytical sensitivity and specificity study on 100 negative and 204 positive frozen samples with a prospective clinical study on sensitivity and specificity, in which nearly 200 SARS-CoV-2 positive individuals and 200 SARS-CoV-2 negative individuals were tested by the included RAT and routine RT-PCR within 72 h of the initial RT-PCR test.

Methods

For detailed description of protocol and methods, please refer to Supplementary data and [21].

Study design

A total of 44 RAT from 26 manufacturers, were included in the study.

Participants

Individuals, who tested positive for SARS-CoV-2 by routine RT-PCR performed by a public test provider were included in the study.

Retrospective analytical RAT testing

Samples in universal transport medium (UTM) from SARS-CoV-2 RT-PCR positive individuals participating in the prospective arm of the study were collected by out-patient testing teams. All samples were collected as deep oropharyngeal swabs and immediately stored at −80 °C until further processing. In total, 50 samples with Cq <25; 54 samples with Cq between 25 and 30; 50 samples with Cq between 30 and 35 and 50 samples with Cq between 35 and 40 were prepared. One hundred pooled SARS-CoV-2 RT-PCR negative routine clinical samples were prepared by pooling 1000 routine UTM samples that had tested negative for SARS-CoV-2, thus allocating ten clinical samples for each negative pooled sample (Fig. 1 A).

Fig. 1

STARD diagram. A, retrospective study part reporting RAT results compared to RT-PCR stratified into Cq ranges. B, pro-spective clinical study part reporting data from individuals, who were either included due to a positive RT-PCR test for SARS-CoV-2 or being tested as part of screening for SARS-CoV-2. For each RAT the total number of tests is reported together with the distribution compared to RT-PCR. FN false negative, FP false positive, Inv invalid RAT, TN true negative, TP true positive. N anterior nasal cavity sampling, OP deep oropharyngeal sampling, S saliva.

Prospective clinical RAT testing

From each included individual, deep oropharyngeal swabs, anterior nasal cavity swabs and saliva were collected for RAT testing together with a deep oropharyngeal swab for RT-PCR testing. All tests procedures were conducted as instructed by the manufacturer employing utensils provided by the manufacturer. Samples collected from other anatomical sites than Conformitè Europëenne (CE)-marked by the manufacturer were collected and handled as instructed by each manufacturer.

Prospective RT-PCR

Deep oropharyngeal swabs for RT-PCR were collected in a NEST disposable sampler inactivation transport medium with an oropharyngeal specimen collection swab (Wuxi NEST Biotechnology Co., Ltd, Wuxi City, China) and sent to RT-PCR testing.

Analysis

RT-PCR targeting two genetic sequences in SARS-CoV-2 and a human control target was used as gold standard. In the retrospective part of the study, samples were grouped into four Cq ranges based on the highest Cq level of the two E- and N-targets. In the prospective part of the study, samples were grouped into three Cq ranges based on the average Cq level between the two N-targets.

Sensitivity and specificity were calculated for each individual RAT in relation to RT-PCR. All p-values less than 0.05 were considered statistically significant.

Role of the funding source

The participating companies had no influence on study design, sample collection, analysis, interpretation, drafting of manuscript, or decision on publishing.

Results

Participants and SARS-CoV-2 variants

A total of 3797 individuals were included in the prospective part of the study between January 18th, 2021 and September 9th, 2021 to allow each RAT to be tested on 200 SARS-CoV-2 positive and 200 SARS-CoV-2 negative individuals. 2045 individuals that had just tested positive for SARS-CoV-2 were included in the study within 72 h and 1752 SARS-CoV-2 expected RT-PCR negative individuals were included after screening at a regional test center. Among the newly positive individuals for SARS-CoV-2, twelve individuals were excluded due to missing sample for RT-PCR (n = 3) or a negative RT-PCR test including the human control target (n = 9). Among the remaining 2033 individuals, 140 (6.9%) were negative by the concomitant RT-PCR at the time of RAT testing (Fig. 1B), resulting in inclusion of 1893 positive individuals among the previous SARS-CoV-2 positive individuals. In the screening group, forty-three individuals were excluded, eight samples were missing for RT-PCR and thirty-five samples were negative for all targets including the human sampling control. Thus, a total of 1709 SARS-CoV-2 negative or positive individuals by RT-PCR were included in the screening group. The SARS-CoV-2 prevalence in the screening group was 0.64%, as 11 out of 1709 individuals were positive for SARS-CoV-2 by RT-PCR (Fig. 1B). Participating individuals were predominantly between 18 and 70 years of age with few individuals above 70 years of age. Study participants were mainly unvaccinated at the time of inclusion.

The alpha variant (B.1.1.7) was the predominating variant among SARS-CoV-2 positive individuals included in the study and accounted for 90.8% (1729 of 1904) of all cases (Supplementary data).

Analytical sensitivity and specificity of RAT

The overall analytical sensitivity differed significantly between the 33 included tests from 25 manufacturers. The analytical sensitivity varied from 2.5% (95% CI 0.5–4.8) for the Lituo saliva RAT to 42% (95% CI 35–49) for the Acro RAT. All RAT had a specificity of 100% (Fig. 2 ). The overall mean sensitivity for the tests was 25% (95% CI 20–31), which reflects that RAT detected almost none of the samples with Cq >35 by RT-PCR (n = 50). For RT-PCR samples with Cq 30–35 (n = 50), the mean sensitivity was 0.7% (95% CI 0.0–1.7) but increased at Cq 25–30 (n = 54) with a mean sensitivity of 30% (95% CI 19–41) and a mean sensitivity of 70% (95% CI 60–80) for samples with RT-PCR Cq <25 (n = 50) (Supplementary data).

Fig. 2

Overall analytical sensitivity and specificity of 33 RAT. Sensitivity and specificity are reported as mean with 95% CI and is based on 50 samples with Cq <25, 54 samples with Cq between 25 and 30, 50 samples with Cq between 30 and 35, 50 samples with Cq >35 and 100 samples negative by RT-PCR.

Clinical sensitivity and specificity of RAT

In the prospective part of the study, the overall clinical sensitivity varied significantly among the 44 included tests from 26 manufacturers; from 2.5% (95% CI 0.5–4.8) for the Quidel saliva RAT to 94% (95% CI 91–97) for the Acon Flowflex RAT, with a mean overall sensitivity of 69% (95% CI 64–74). Specificity varied among the RAT from 98% (95% CI 95–99) for the Abbott nasal RAT to 100% for 28 RAT (Fig. 3 ). According to CE-mark data, the sensitivities for all tests were reported to be 81% (95% CI 69–89) or above and specificities were reported from 97% (95% CI 93–98) and above (Supplementary data).

Fig. 3

Overall clinical sensitivity and specificity of 44 RAT. Sensitivity and specificity are reported as mean with 95% CI. “N” is an anterior nasal cavity test, “OP” is a deep oropharyngeal test and “S” is a saliva based RAT. “ST” is self-test, “I” indicates an instrument read-out test and “visual” means that the RAT is evaluated visually instead of by instrumental read-out.

The sensitivity of the Acon Flowflex RAT did not differ significantly from 16 of the other RAT from 14 manufacturers, when multiple testing was taken into account. The remaining 27 RAT (including the visual and automated read-out of the BD veritor RAT) had a significantly lower overall sensitivity compared to the Acon Flowflex RAT (Table 1 and Supplementary data).

Table 1

Comparison of clinical sensitivity between RAT. Green fields are non-significant differences, red are significant differences in clinical sensitivity. ”N” anterior nasal cavity test, ”OP” deep oropharyngeal test, ”S” saliva. Sensitivity is reported in brackets. Details for calculation are included in Supplementary data.

Image, table 1

The mean overall sensitivity was 81% (95% CI 75–87) for strong positive samples with Cq <15, which decreased to 71% (95% CI 63–78) for medium positive samples with Cq 15–20 and 44% (95% CI 31–57) mean sensitivity for weak positive samples with Cq >20 (Supplementary data). Three out of the four included self-test were among the 17 most sensitive tests in the study, whereas the Wantai saliva/buccal RAT was among the five tests with the lowest sensitivity in the study. The sensitivity of RAT with automated readout was distributed over the full range of sensitivities with the Lumira Dx and Quidel nasal RAT performing among the 17 best tests of the study.

Discussion

In our study, we used deep oropharyngeal swabs collected for the analytical sensitivity and specificity study. Thirty-three RAT were compared using the same 204 SARS-CoV-2 positive samples and we anticipated that the analytical sensitivity data would correlate with the ranking in clinical sensitivity among the 44 RAT in the prospective part of the study. The prospective study included a higher number of different RAT compared to the retrospective study, as several manufacturers provided identical RAT to be used on several anatomical sampling sites. Interestingly, we saw similarities between analytical sensitivity and clinical sensitivity on SARS-CoV-2 positive individuals, but several tests performed differently between the retrospective and prospective part of study. As an example, the Acro RAT presented the highest analytical sensitivity, but performed significantly worse than the Acon Flowflex RAT in the prospective part of the study. Our data shows that anterior nasal cavity RAT in general had higher sensitivity compared to deep oropharyngeal RAT and RAT using saliva for testing performed the worst. This may indicate that the Acro RAT using deep oropharyngeal swabs is underperforming in the prospective part of the study as anterior nasal cavity swabs may be a better testing material. A more general concern regarding the comparison of retrospective and prospective results is that the sampling step, release of material and lysis material from the swab in the lysis buffer are removed from the comparison in retrospective studies, which may lead to underperformance of clinically optimized RAT in analytical studies.

The primary strength of this study is that we can identify RAT that perform significantly better than other RAT included in the study, as we report analytical as well as clinical performance data for a large number of commercially available RAT and compare the RAT performance to the same RT-PCR test on the same samples.

This study has several limitations. First, it is important to emphasize that the study was designed to compare performance differences between the included RAT. The data reported in this study cannot be used to predict the sensitivity of a certain RAT in a specific clinical setting. Second, as the out-patient testing teams knew the RT-PCR result of the included individual prior to sampling and testing by RAT, and as even a weak band should be regarded as a positive test result, the prospective part of the study is biased towards overestimating the sensitivity of each RAT. Third, the current study is likely to overestimate the clinical sensitivity of the included RAT as the study was performed prior to vaccination of the participating age groups. It has previously been shown that vaccination accelerates viral clearance, which may reduce the amount of viral particles released by infected individuals and thereby narrows the time period for a positive RAT [22]. Finally, at the time of the study, the predominant variant of SARS-CoV-2 was the B.1.1.7 alpha variant with few cases of the delta variant (B.1.617.2). It has previously been shown that different SARS-CoV-2 variants influence the analytical sensitivity of different RAT [14], which may translate to differences in clinical sensitivity of RAT if other SARS-CoV-2 variants are predominant [9].

In conclusion, this study compares analytical and clinical sensitivities to RT-PCR of 46 commercially available RAT and enables identification of individual RAT that has significant higher sensitivity than other included RAT. The study demonstrates significant differences in analytical as well as clinical sensitivities between the included RAT but cannot be used for prediction of the clinical sensitivity in a specific clinical setting. The results reported in this study may guide decision makers prior to purchase of RAT for population screening.

Authors' contributions

UVS, JGL, and JDK conceptuated the study. UVS and JGL were responsible for the funding acquisition. Acquisition of data was conducted by UVS, MWF, TDL, CBJ, KG, CNA, KKM, CSJ, HL, MBJ, US, AK, NSK, The National Danish RAT testing group, MLS, FHJ, RLJ, CMGM, NS, and JGL. Local supervision of research activity was conducted by MWF, CNA, KKM, AC, NS, CSJ, HL, MBH, US, AK, and NSK. Overall project administration was conducted by UVS, MWF, AC, CSJ, JDK, and JGL. Data was curated by CBJ, KG, MWF, CNA, KKM, CSJ, US, NSK, UVS, and JGL. Data was validated by or under supervision of MWF, CBJ, CNA, KKM, CSJ, HL, US, NSK, RLJ, JGL, and UVS. TK and UVS performed the formal analysis of data.

UVS and JGL drafted the manuscript with substantial input from JDK. All authors read and commented on the manuscript and all authors approved the final manuscript prior to submission. All authors agree to be held accountable for all aspects of the work.

Ethics committee approval

The study was evaluated by the National Committee on Health Research Ethics in the Danish Capital Region to be a method validation study without the need of approval of the committee (decision H-20,068,579). Access to test results for research was granted by the Capital Region of Denmark – Research and Innovation (R-20,083,753) and contact to participants without prior consent from the individual was granted by the board of directors at the Hospitals at which the participating DCM are situated.

Declaration of Competing Interest

Lars Østergaard has within 36 months received speaker fees from Sanofi-Pasteur and Pfizer. None of the other authors have any personal conflict of interests to report. The project received a participation fee from each of the participating companies to cover the cost of the project.