Diagnostic performance of a SARS-CoV-2 rapid antigen test in a large, Norwegian cohort.

BACKGROUND: Rapid antigen tests (RATs) may be included in national strategies for handling the SARS-CoV-2 pandemic, as they provide test results rapidly, are easily performed outside laboratories, and enable immediate contract tracing. However, before implementation further clinical evaluation of test sensitivity is warranted.
OBJECTIVES: To examine the performance of Abbott's Panbio™ COVID-19 Ag Rapid Test Device for SARS-CoV-2 testing in a low to medium prevalence setting in Norway. STUDY
DESIGN: A prospective study comparing the results of the Panbio RAT with PCR in 4857 parallel samples collected at a SARS-CoV-2 test station in Oslo, and from COVID-19 outbreaks in six Norwegian municipalities.
RESULTS: A total of 4857 cases were included in the study; 3991 and 866 cases from the test station and the outbreak municipalities, respectively. The prevalence at the test station in Oslo was 6.3 %, and the overall sensitivity of the RAT was 74 %. Increased sensitivity was observed in patients who experienced symptoms (79 %) and when considering samples with viral loads above estimated level of infectivity (84 %), while it was lower in asymptomatic persons (55 %). In the outbreak municipalities, the overall prevalence was 6.9 %, and the total sensitivity of the RAT was 70 %.
CONCLUSIONS: Our results indicate that the test correctly identified most infectious individuals. Nevertheless, the sensitivity is considerably lower than for PCR, and it is important that the limitations of the test are kept in mind in the follow-up of tested individuals.

Introduction

The Norwegian strategy to fight the spread of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is reliant upon rapid identification of infected individuals, isolation of cases, contact tracing and quarantine. In this respect, SARS-CoV-2 rapid antigen tests (RATs) could represent a good alternative to polymerase chain reaction (PCR) given its short turnaround time, which enables earlier initiation of isolation and infection control measurements. However, further data on the clinical performance of RATs is needed. Most RATs are based on lateral flow immunochromatography using antibodies to target the SARS-CoV-2 nucleocapsid protein. Compared to PCR, the sensitivity of RATs is lower. A key question is whether a positive RAT result may correlate with infectiousness and correctly identify infectious persons in specific settings. The World Health Organization has suggested that RATs could play a significant role in patient management and surveillance of coronavirus disease 2019 (COVID-19) given a sensitivity of ≥ 80 % and specificity of ≥ 97 % [1]. Analytical sensitivity and specificity measures given by the manufacturers do not necessarily reflect the actual performance of the test. Furthermore, evaluation studies from other countries might not be transferable to national outbreak settings, and independent and setting-specific validations of RATs before their implementation are therefore recommended by the European Centre for Disease Prevention and Control [2]. Thus, a field evaluation of Abbott’s Panbio™ COVID-19 Ag Rapid Test Device (Panbio RAT) in Norwegian settings was initiated. The aim of the evaluation was to study the RAT’s performance compared to SARS-CoV-2 PCR in 1) a low to medium prevalence setting at the COVID-19 test station Aker in Oslo, and 2) in outbreaks in more rural areas.

Materials and methods

Study population, setting and sampling strategy

The evaluation study of the Panbio RAT diagnostic performance was conducted from 30th October to 25th November 2020. Individuals who signed up for a COVID-19 test at Aker test station in Oslo were given information about the study at the Oslo municipality website and invited to participate. Individuals were eligible to participate if they were aged 10 years or older, competent to consent, and able to read and understand Norwegian. Persons without a Norwegian personal identification number were excluded from participation. Specially trained personnel at Aker test station performed the testing. Two parallel combined throat/nasopharyngeal swabs were obtained from each person. The first swab was sent to Oslo University Hospital (OUH) for routine SARS-CoV-2 PCR. The other was examined with the Panbio RAT at the test station, according to the manufacturer’s instructions. All study participants were surveyed about known exposure and symptom duration. Municipality physicians in charge of infection control, experiencing COVID-19 outbreaks during the study period, were asked to organize their municipality’s participation in the study. Testing was performed following the same procedure as Aker test station, except that the first swab was sent to the local microbiology laboratory instead of to OUH.

Laboratory analyses

At the Department of Microbiology at OUH, extraction of viral RNA was performed with the automated workstation Tecan Fluent 1080 (Tecan Trading AG, Switzerland) using an in-house extraction protocol based on a standard method with magnetic beads [3], developed at the Norwegian University of Science and Technology. The eluate was analysed with reverse transcriptase PCR for the SARS-CoV-2 E-gene (modified from Corman et al. [4]) using Aria Dx Real-Time PCR System (Agilent Technologies LDA, Malaysia). In cases of positive RAT results not confirmed by PCR, the PCRs of the samples were repeated. Additionally, they were analysed with the Cobas® SARS-CoV-2 kit on the Cobas® 6800 system (Roche Diagnostics GmbH, Mannheim, Germany).

In order to investigate which cycle threshold (ct) value a concentration of one million RNA copies/mL represented, we examined a dilution series of an RNA standard (Human 2019-nCoV RNA, made available through European Virus Archive Global, EVAg) [4]. In our PCR, this concentration corresponded to a ct value of approximately 30 (data not shown).

Statistics

Sample size was calculated based on the formula by Malhotra et al. [5]. A sample size of 4000 was deemed acceptable.

Clinical data, as well as results from the RATs, were compiled and delivered to the Norwegian Institute of Public Health (NIPH) on a weekly basis. Data from Aker test station was subsequently merged with the Norwegian Laboratory database at NIPH, using a personal identifier, in order to obtain the corresponding PCR results. Data analysis was performed using Stata version 16 (StataCorp. 2019. Stata Statistical Software: Release 16. College Station, TX: StataCorp LLC). Data was summarized with descriptive statistics. Sensitivity and specificity with 95 % confidence intervals, as well as positive and negative predictive value, were computed using the PCR as the reference standard. Agresti-Coull confidence intervals are shown. Bivariate associations between independent categorical variables and RAT results were calculated using Chi-Square tests. For independent numerical variables Mann Whitney U tests were used to compare medians in two groups.

Ethics

The study was presented to the Data Protection Officer for Research at the Norwegian Directorate of Health, who gave advice on how to safeguard privacy aspects in the study. All persons who met the inclusion criteria at Aker test station or in the outbreak municipalities were asked to give oral consent before taking part in the study.

Results

Aker test station in Oslo

During the study period, a total of 5412 samples were collected for PCR analysis. Out of the 5412 cases, 4025 gave their consent to participate in the study, giving a participation rate of 74 %. Among the 4025 cases, 83 individuals were tested twice, and two individuals were tested three times. These were all considered independent cases in subsequent analysis. Of the 4025 samples, 3998 were successfully matched to their corresponding PCR results in the database. As one PCR result was inconclusive, and six antigen tests were either inconclusive or defective, a total of 3991 cases were successfully included in the study.

As shown in Table 1 , 6.3 % of the included cases tested positive for SARS-CoV-2 by PCR. The majority of the participants (n = 2475) reported symptoms of COVID-19 at the time of testing, with a symptom duration ≤ 5 days (n = 2143), and there was a higher proportion of PCR positive samples among symptomatic cases compared to asymptomatic (8.0 % vs. 3.3 %).

Table 1

Descriptive characteristics of the 3991 cases included from Aker test station in Oslo.

	Total, n	PCR negative, n (%)	PCR positive, n (%)
n	3991	3741 (93.7)	250 (6.3)
Exposed
No	2234	2143 (95.9)	91 (4.1)
Yes	1423	1284 (90.2)	139 (9.8)
Unknown	325	305 (93.9)	20 (6.2)
Missing	9	9 (100)	0
Symptoms
No	1408	1361 (96.7)	47 (3.3)
Yes	2475	2276 (92.0)	199 (8.0)
Unknown	101	97 (96.0)	4 (4.0)
Missing	7	7 (100)	0
Symptom duration
≤ 5 days	2143	1965 (91.7)	178 (8.3)
> 5 days	327	306 (93.6)	21 (6.4)
Unknown	5	5 (100)	0

The overall sensitivity of the RAT was 74.4 % (Table 2 ). Of the 3741 PCR negative cases, three were RAT positive (1.2 % of the RAT positive cases). For these three samples, repeated PCRs on two different platforms were both negative, confirming that they were truly PCR negative and thus RAT false positives. The specificity of the RAT was thus 99.9 % (CI 95 %: 99.7–99.9). As shown, the sensitivity of the RAT was markedly higher in symptomatic than in asymptomatic cases (78.9 % versus 55.3 %). The median ct value was significantly lower in the symptomatic cases (24.5 versus 28.2; p = 0.001), although the range was similar in both groups (17.5–37.8 versus 16.2 - 39.0) (data not shown). When examining only samples with ct values below the estimated threshold of infectivity (ct < 30), the sensitivity of the RAT was 83.3 %.

Table 2

Test performance (sensitivity) of the Panbio RAT compared to PCR, - overall and at different ct values and clinical cutoffs.

	Total, n	RAT negative, n	RAT positive, n	Sensitivity, % (95 % CI)
PCR positive	250	64	186	74.4 (69–79)
PCR negative	3741	3738	3
ct < 30	204	33	171	83.8 (78–88)
ct ≥ 30	3787	3769	18
PCR positive symptomatic	199	42	157	78.9 (73–84)
Duration ≤ 5 days	178	36	142	79.8 (73–85)
Duration > 5 days	21	6	15	71.4 (50–86)
ct < 30, symptom duration ≤ 5 days	153	19	134	87.6 (81–92)
Asymptomatic PCR positive	47	21	26	55.3 (41–69)
Exposed PCR positive	139	38	101	72.7 (65–79)

At a prevalence of 6.3 %, the positive predictive value (PPV) of the RAT was 0.984, while its negative predictive value (NPV) was 0.983. Fig. 1 illustrates how the PPV and NPV are affected by different prevalence rates, and a sharp decrease in PPV at prevalence rates below 1 % is depicted.

Fig. 1

Positive predictive value (PPV) and negative predictive value (NPV) at different prevalence rates of SARS-CoV-2, given a sensitivity of 74.4 % and a specificity of 99.9 %.

Among cases that tested positive for SARS-CoV-2 by PCR, a comparison of RAT negative and RAT positive cases showed that the presence of COVID-19 symptoms was significantly associated with a positive RAT result (p < 0.001), while the duration of symptoms (≤ 5 days vs. > 5 days), was not (Table 3 ). The mean and median ct values were significantly lower among the RAT positive cases compared to the RAT negative cases (p < 0.001), and as illustrated in Fig. 2 , a significantly larger fraction of the RAT positive cases had ct values in the mid and lower range, while higher ct values were more prevalent among the RAT negative cases. However, there was a considerable overlap between the two groups.

Table 3

Comparison of the RAT negative and positive PCR positive cases.

	Total, n	RAT negative, n (%)	RAT positive, n (%)	p-value*
n	250	64 (25.6)	186 (74.4)
Exposed
No	91	21 (23.1)	70 (76.9)	0.469
Yes	139	38 (27.3)	101 (72.7)
Unknown	20	5 (25.0)	15 (75)
Symptom
No	47	21 (44.7)	26 (55.3)	< 0.001
Yes	199	42 (21.1)	157 (78.9)
Unknown	4	1 (25.0)	3 (75.0)
Symptom duration
≤ 5 days	178	36 (20.2)	142 (79.8)	0.375
> 5 days	21	6 (28.6)	15 (71.4)
ct values
Mean (SD)	25.8 (4.7)	29.9 (4.7)	24.4 (3.9)	< 0.001
Median	25.3	29.8	23.8
Min – Max	16.16 - 38.99	17.5 - 38.27	16.16 - 38.99

The categories “Unknown” were not included in the chi square tests.

Fig. 2

Distribution of ct values from the SARS-CoV-2 PCR in RAT negative and positive cases.

Outbreaks

Cases from outbreaks in six Norwegian municipalities were included. A total of 866 RATs were performed, 304 in Farsund (28 PCR positive), 75 in Rana (18 PCR positive), 404 in Våler/Åsnes (9 PCR positive), 54 in Lindesnes (1 PCR positive), 21 in Vindafjord (3 PCR positive), and 8 in Lurøy (1 PCR positive). Among the 806 participants with a negative PCR result, no false positive RAT result was recorded in any municipality, yielding an overall specificity of 100 % (95 % CI 99.5–100 %). Among the 60 participants with a positive PCR result, 42 tested positive with the RAT, resulting in an overall sensitivity of 70 % (95 % CI: 57–81) (data not shown).

Discussion

In this study, we assessed the performance of the Panbio RAT for SARS-CoV-2 compared to PCR in 4857 parallel samples. In the 3991 cases from Aker test station in Oslo the overall sensitivity of the RAT was 74.4 %. However, the sensitivity was higher among people who reported COVID-19 symptoms (78.9 %) and in samples with high viral loads (sensitivity 83.8 % in samples with ct values < 30).

Recent large prospective studies on the Panbio RAT have included from 913 to 4138 participants and were performed in settings with relatively high SARS-CoV-2 prevalence, ranging from 9.6 % to 40.1 % [[6], [7], [8], [9], [10], [11], [12]]. The prevalence in Oslo during the study period was low to medium with a prevalence rate in all samples from Aker test station at 5.7 % (unpublished data), which made it particularly important to conduct an independent evaluation. The overall sensitivity reported by the aforementioned studies varied between 60.5 % and 90.5 % [[6], [7], [8], [9], [10], [11], [12]]. Studies comprising mainly symptomatic persons described higher sensitivity, all above 80 % [6,8,9,12], whereas the populations in the two studies reporting the lowest sensitivities (60.5 % and 71.4 %) both contained large fractions of asymptomatic individuals [7,10]. Our results also indicate a lower sensitivity among asymptomatic individuals, as we found the sensitivity of the RAT to be 55.3 % in this group. The ct values were higher in the PCR positive samples from the asymptomatic persons compared to those with symptoms (median 28.2 versus 24.5, respectively), suggesting that lower viral loads could at least partly explain the lower sensitivity found in this group. In a recent study, sensitivity numbers (75–93 %) equal to that for symptomatic patients was found for patients with presymptomatic or early asymptomatic infections [13]. However, for asymptomatic patients late in the course of disease, the sensitivity was very low (26 %). This may indicate that the Panbio RAT is equally sensitive for infectious individuals regardless of whether they have symptoms or not [13].

The test principle of most RATs is based on antibodies targeting the SARS-CoV-2 nucleocapsid protein, while the reference standard PCR is based on amplification of RNA to millions of copies. The sensitivity of RATs is thus expected to be lower. It has, however, been proposed that RATs could be used as a test of infectiousness instead of clinical disease [14], and modelling studies have shown that in this context, sensitivity is less important than test frequency and turn-around time [14,15]. Viral loads around one million RNA copies per mL (or per swab) of respiratory secretions have been proposed as a reasonable cutoff for evaluating infectiousness, as replicating virus is rarely detected in samples with viral loads below this limit [4,[16], [17], [18], [19]]. Using an RNA standard, we found that one million RNA copies/mL roughly corresponds to a ct value of 30 in our PCR. In the samples with ct values below 30, we found the RAT’s sensitivity to be 83.8 %. This is in line with previous studies which also found an association between high viral load (ct values < 30 or viral load > 106 copies/mL) and increased sensitivity, with reports ranging from 80.0 % to 98.0 % [[6], [7], [8], [9], [10], [11], [12]]. This indicates that the majority of infectious cases can be correctly identified with the RAT. Nevertheless, in our study more than 15 % of the potentially infectious individuals (ct values < 30) received negative RAT results, underscoring the fact that negative results should be interpreted with caution. Alternative strategies that could compensate for this limitation are repeated RAT or parallel PCR testing.

Several studies have shown that SARS-CoV-2 viral load peaks in the first week after symptom onset [20]. After one week, viral loads in upper respiratory tract samples have been shown to drop below one million copies/swab [19]. Our data shows that for SARS-COV-2 infected patients with symptom duration less than five days, the sensitivity of the RAT was about 80 %, and when restricting the analysis to individuals with an expected contagious viral load (ct value < 30), the sensitivity was 87.6 %. This is in line with other studies of the Panbio RAT, where sensitivity ranged from 77.2 % to 95.8 % when only patients with symptom duration of less than one week were considered [[6], [7], [8], [9], [10], [11], [12]].

In our study, the overall specificity of the antigen test was found to be very high (99.9 %). This is consistent with previous reports, which all showed specificity numbers close to 100 % [[6], [7], [8], [9], [10], [11], [12]]. False positive results are thus rare. However, in low prevalence settings (<1 %), the proportion of false positives still becomes notable.

This is to our knowledge the largest prospective study comparing the performance of the Panbio RAT for SARS-CoV-2 with PCR. In addition, a major strength of our study is that most of the testing was conducted in routine testing facilities, which involved highly experienced testing personnel and performance of the RATs in realistic settings for future use. Moreover, the study was carefully planned for, thus securing well-functioning logistics, standardized routines and adequate training in performance of the RAT. The indication for testing in these facilities is liberal, and one limitation of the study is therefore that the reasons for testing may vary considerably between the participants. A poor indication for testing as well as a high proportion of asymptomatic individuals in the study population could partly explain why we find a lower sensitivity of the RAT than previous studies. The large study population increases the statistical power. However, with a SARS-CoV-2 prevalence rate at 6.3 % the number of PCR positive cases is still limited for some stratifications.

Conclusions

We have conducted one of the largest studies to date comparing the Panbio RAT with PCR in a low to medium prevalence setting and found that the test results are in line with previous field evaluations in higher prevalence settings. It is important to be aware of the limitations of the RAT and to keep in mind that the sensitivity is lower compared to the well-established PCR tests. Nevertheless, in some situations this might be outweighed by the advantages of identifying infectious individuals faster and thus allowing for rapid isolation and contact tracing.

Funding

The study was funded by the Norwegian health authorities as part of the national strategy to increase the overall test capacity in Norway.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.