Accuracy and ease-of-use of seven point-of-care SARS-CoV-2 antigen-detecting tests: A multi-centre clinical evaluation.

BACKGROUND: Antigen-detecting rapid diagnostic tests (Ag-RDTs) for SARS-CoV-2 are important diagnostic tools. We assessed clinical performance and ease-of-use of seven Ag-RDTs in a prospective, manufacturer-independent, multi-centre cross-sectional diagnostic accuracy study to inform global decision makers.
METHODS: Unvaccinated participants suspected of a first SARS-CoV-2 infection were recruited at six sites (Germany, Brazil). Ag-RDTs were evaluated sequentially, with collection of paired swabs for routine reverse transcription polymerase chain reaction (RT-PCR) testing and Ag-RDT testing. Performance was compared to RT-PCR overall and in sub-group analyses (viral load, symptoms, symptoms duration). To understandusability a System Usability Scale (SUS) questionnaire and ease-of-use (EoU) assessment were performed.
FINDINGS: 7471 participants were included in the analysis. Sensitivities across Ag-RDTs ranged from 70·4%-90·1%, specificities were above 97·2% for all Ag-RDTs but one (93·1%).Ag-RDTs, Mologic, Bionote, Standard Q, showed diagnostic accuracy in line with WHO targets (> 80% sensitivity, > 97% specificity). All tests showed high sensitivity in the first three days after symptom onset (≥87·1%) and in individuals with viral loads≥ 6 log10SARS-CoV2 RNA copies/mL (≥ 88·7%). Usability varied, with Rapigen, Bionote and Standard Q reaching very good scores; 90, 88 and 84/100, respectively.
INTERPRETATION: Variability in test performance is partially explained by variable viral loads in population evaluated over the course of the pandemic. All Ag-RDTs reach high sensitivity early in the disease and in individuals with high viral loads, supporting their role in identifying transmission relevant infections. For easy-to-use tests, performance shown will likely be maintained in routine implementation. FUNDING: Ministry of Science, Research and Arts, State of Baden-Wuerttemberg, Germany, internal funds from Heidelberg University Hospital, University Hospital Charité - Universitätsmedizin Berlin, UK Department of International Development, WHO, Unitaid.

Evidence before this study

We conducted a PubMed, Web of Science Core Collection, bioRvix, FIND and medrXiv search for studies evaluating Ag-RDT performance from 12/2019 until 04/2021. The search terms used were taken from the living systematic review that is led by our group (www.diagnosticsglobalhealth.org): In total we found 63 studies pertaining to the tests evaluated here, with SD Biosensor Standard Q being assessed in the most (34) studies, Bionote in 4, Fujirebio in 6, Rapigen in 8, SD Biosensor Standard F in 7, Bioeasy in 3 and Mologic in 1. Only 16 of the studies (with SD Biosensor Standard Q included in 12 of these and the others in two or less) performed tests according to manufacturer's instructions, thus providing comparable results. Overall, the data generated primarily on SD Biosensor Standard Q suggests that Ag-RDTs can achieve high sensitivity and excellent specificity.

Added value of this study

Our study presents the comparative analysis of seven Ag-RDTs, evaluated in a multi-centre, clinical accuracy study with over 7000 participants in two countries. For three of the Ag-RDTs, Mologic, Fujirebio and Bionote, this is the first manufacturer independent study. We assessed accuracy overall and in predefined subgroups (according to viral load, presence of symptoms and symptoms duration). We found three tests to meet the requirements formulated for WHO EUL and a fourth test came very close. Sensitivity was particularly high in the first days of symptom onset and when viral loads were high. The comparative system usability and ease-of-use assessment complement the accuracy assessment of the tests and highlight critical factors to facilitate widespread use of Ag-RDTs in point-of-care settings.

Implications of all the available evidence

Evidence from this study was used to inform the WHO EUL procedure. Furthermore, the high sensitivity demonstrated within the first days of symptoms, when most transmission occurs, support the role of Ag-RDTs for public health relevant screening. In addition, the high ease-of-use of some of the tests suggests that their accuracy will likely be retained when implemented in routine settings for diagnosis of persons presenting with symptoms suggestive of COVID-19.

Alt-text: Unlabelled box

Introduction

Early in the coronavirus disease 2019 (COVID-19) pandemic the World Health Organization (WHO) highlighted fast and accessible testing as critically important for effective pandemic control With the global shortage of vaccines as well as immune escape variants, testing remains an essential public health tool.

The gold standard for the diagnosis of SARS-CoV-2 infections is laboratory-based reverse transcription polymerase chain reaction (RT-PCR), which is highly sensitive but requires extensive laboratory infrastructure, expensive materials and skilled staff. These aspects limit RT-PCR scalability and implementation in many settings, especially those with low-resources.

Antigen-detecting rapid diagnostic tests (Ag-RDTs) deliver results quickly and if easy to use, meet the characteristics required for a public health testing tool., The utility of these tests for pandemic control through mass testing or testing to protect (e.g. in high risk settings such as hospitals), to release (e.g. contact testing) and to enable (e.g. regular school or workplace testing) has been suggested in different studies.5, 6, 7 For the WHO Emergency Use Listing (EUL), Ag-RDTs are required to meet targets of at least 80% sensitivity and 97% specificity, to be used to diagnose SARS-CoV-2 infections. As of October 2021 four Ag-RDTs have been approved by the WHO for emergency-use and are being implemented in various settings to support RT-PCR testing., Numerous other Ag-RDTs have been developed since and are seeking WHO EUL.

This manufacturer-independent, multi-centre cross-sectional clinical accuracy study assessed the performance and ease-of-use of seven Ag-RDTs, selected for evaluation by FIND, the WHO collaborating centre for COVID-19 diagnostics, to inform the review of global decision makers (e.g., WHO, Global Fund) and expedite the availability of tests. FIND opened expressions of interest (EOI) to select Ag RDTs to be evaluated, prioritizing those that were considered most relevant to low- and middle-income country implementation based on supplier submitted information on manufacturing capacity, a history of offering products in LMIC, projected price, and performance. The seven products included in this manuscript represent the first seven selected, with the exception of another test, the Abbott Panbio, the results of which had been previously reported.

Methods

Ethics

The study protocol was approved in March 2020 by the ethical review committee at Heidelberg University Hospital for the study sites in Germany (Heidelberg and Berlin) (Registration S-180/2020). For the study sites in Brazil (Rio de Janeiro, Marica, Guapimirim and Macae), the study protocol was approved in April 2020 by the National Commission of Research Ethics (Registration 3.953.368). All study participants provided written informed consent. The study was registered in the German Clinical Trial Registry (DRKS00021).

Clinical diagnostic accuracy study

This study is reported following the Standards for Reporting Diagnostic accuracy studies (STARD) and the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement.,

Study design, setting and participants

The primary objective of this study was to estimate the sensitivity and specificity of seven Ag-RDTs compared to RT-PCR as reference standard. Participant recruitment and sample collection were conducted in six sites, two in Germany (Heidelberg and Berlin) and four in Brazil (Rio de Janeiro, Marica, Guapimirim and Macae). Heidelberg performed the study at a drive-in testing station whereas the Berlin site was located at a clinical ambulatory facility. In Brazil, the study sites were located at the COVID-19 Diagnostic Centre in Rio de Janeiro and three community testing clinics in Marica, Guapimirim and Macae.

Consecutive participants screened for the study were adults (age ≥ 18 years) identified as at risk for SARS-CoV-2 infection based on exposure to RT-PCR confirmed SARS-CoV-2 positive individuals (independent of symptoms) or individuals presenting with symptoms suggestive of COVID-19 disease. The detailed symptoms asked are available in the supplementary material (Table S2). Participants were excluded if they had previously been diagnosed with a SARS-CoV-2 infection, vaccinated against SARS-CoV-2 or if their command of either English, or the local language was insufficient to give written informed consent. An interview focusing on symptoms and comorbidities was performed. The study protocol is available upon request.

Enrolment

The enrolment target was at least 100 RT-PCR positive participants for each Ag-RDT. The number of participants per Ag-RDT varied due to changing prevalence across sites and time. As the goal of the evaluation was to provide data to the global decision makers as quickly as possible, an Ag-RDT evaluation was stopped when the minimum number of positive cases were reached, regardless of the distribution by country. Furthermore, interim analyses were performed at predefined sample sizes and an evaluation was stopped if the predefined performance criterion for specificity (≥ 97%) was not met and a root-cause analysis did not provide a reason other than poor specificity attributable to the test itself.

Sample collection and testing

All participants received paired sample collection for RT-PCR testing and one Ag-RDT. The sample collection was performed by trained professionals.

RT-PCR testing and quantification of viral loads

In Heidelberg RT-PCR samples were collected via nasopharyngeal (NP) swab (oropharyngeal (OP) only in case of clinical contra-indications for NP sampling). Berlin performed combined OP/NP sample collection for routine testing as per institutional recommendations (OP first, followed by NP with the same swab). In Brazilian sites, NP sampling was done. While in Germany sampling for RT-PCR preceded sampling for Ag-RDT, in Brazil the sequence was reversed.

Swabs for RT-PCR testing were transported to the referral laboratory in recommended viral transport solutions on the same day within hours of sampling. The assays were performed according to routine procedures at the referral laboratory and varied due to supply scarcity. In Heidelberg, the SARS-CoV-2 assay from TibMolbiol (Berlin, Germany), the Allplex SARS-CoV-2 Assay from Seegene (Seoul, South Korea) or the Abbott (Illinois, US) RealTime 2019-nCoV assay were performed. In Berlin the Roche cobas SARS-CoV-2 assay (Pleasanton, CA United States) on the cobas® 6800 or 8800 system or the SARS-CoV-2 assay from TibMolbiol (Berlin, Germany) were performed. In Brazil the CDC 2019-Novel Coronavirus Real-Time RT-PCR Diagnostic Panel assay was used. Staff performing the RT-PCR tests were blinded to results of Ag-RDTs and vice versa.

Quantification of viral load was performed either using a SARS-CoV-2 RNA standard (RKI, Berlin, Germany) or a SARS-CoV-2 plasmid control (2019-nCoV_N_Positive control, Cat nr 10,006,625, IDT in Brazil) with a defined viral load in order to set up a standard curve. Conversion of the Ct-values into viral load was performed using RT-PCR with defined amounts of standardized quantified SARS-CoV-2 RNA or plasmid. Based on this testing of standardized material, the Ct-values of the three SARS-CoV-2 RT PCR assays performed differ by ± 1.5 Ct with the same amount of virus present between runs and different PCR assays.

Antigen-detecting testing

Seven Ag-RDTs were evaluated sequentially. Each participant presenting at the testing facilities underwent only one Ag-RDT. Sample collection was collected as NP swab for six of the seven Ag-RDTs under evaluation (as per IFU). One Ag-RDT, Mologic, recommended sample collection per anterior nasal swab. The Ag-RDT was performed directly after sampling in dedicated areas of sample collection sites. For Ag-RDTs with visual read-out, the results were interpreted by two trained operators, each blinded to the result of the other. If discrepant results were reported, both operators re-read the result and agreed on a final result. Invalid results were repeated once if sufficient buffer solution was available as per IFU. Persistent invalid Ag-RDT results were reported separately.

Seven Ag-RDTs assessed:

BIOCREDIT COVID-19 AG (RapiGEN Inc., Gyeonggi-do, Korea; henceforth Rapigen).

STANDARD F COVID-19 Ag FIA (SD Biosensor Inc., Gyeonggi-do, Korea; henceforth Standard F).

STANDARD Q COVID-19 Ag Test (SD Biosensor Inc., Gyeonggi-do, Korea; in Europe distributed by Roche; henceforth Standard Q).

Bioeasy 2019-nCoV Ag Fluorescence Rapid Test Kit, (Shenzhen Bioeasy Biotechnology Co. Ltd., Guangdong Province, China; henceforth Bioeasy).

ESPLINE® SARS-CoV-2 (Fujirebio Inc., Tokyo, Japan; henceforth Fujirebio).

Mologic COVID-19 Rapid Antigen Test (Mologic Ltd., Bedford, United Kingdom; henceforth Mologic).

NowCheck COVID-19 Ag Test (BioNote Inc., Gyeoggi-do, Korea.; henceforth Bionote).

All Ag-RDTs evaluated were cassette-based lateral flow assays for the detection of SARS-CoV-2 viral antigen. Rapigen, Standard Q, Fujirebio, Mologic and Bionote produce a colour change that can be read with the naked eye. Standard F and Bioeasy utilize a fluorescent readout, using a proprietary reader. The IFUs were followed for all Ag-RDTs (one exception pipettes usage for liquid transfer per manufacturer's request for Bioeasy). For sample collection, proprietary swabs provided with kits were used for all tests except Fujirebio (manufacturer recommended commercially available swabs: IAMP COVID-19 Sample Collection Device NP, Atila BioSystems, CA, USA). The readout was recorded within the recommended time (eight to thirty minutes across tests). A list of all seven Ag-RDTs and their characteristics is available in the supplement material (Table S3).

System usability scale and ease-of-use assessment

To understand the usability of Ag-RDTs, a standardized System Usability Scale (SUS) questionnaire and ease-of-use assessment (EoU), specifically developed for this study, were performed. Laboratory personnel from each study site (at least three operators per test) were invited to complete the surveys. The SUS questionnaire, the EoU questionnaire and survey evaluation matrix are provided in the supplement material questionnaire S1 and S2 and Fig. S1. A SUS score greater 68 is interpreted as above average.

Statistical analysis

For the primary analysis, the pooled sensitivity and specificity of the Ag-RDTs were calculated using a fixed-effects model, given that variability between countries was limited. We compared the Ag-RDT results to RT-PCR from samples collected on the same day. The 95% confidence interval (CI) was calculated using Wilson's method. A priori subgroup analyses were performed for Ag-RDTs where more than 100 RT-PCR positive results were available. Subgroup analyses combined data from all sites per test and included symptom duration (≥ or < 7days, ≥ or < 3days), viral load (< or ≥6 log10 SARS-CoV-2 RNA copies/ml), and symptomatic/asymptomatic (if at least 10 asymptomatic participants were RT-PCR positive). While the transmissibility of SARS-CoV-2 is not only explained by viral load, high viral load is strongly correlated with it. We chose the cut-off of ≥6 log10 SARS-CoV-2 RNA copies/ml as a marker of high viral load. A posthoc analysis of sensitivity was done using a general linear model including test types, viral load and symptom duration. Inter-operator variability was assessed for tests read with the naked eye. Cohen's κ statistic was used to calculate agreement of positive and negative results between the two independent readers. The analysis was conducted with the statistical software R Version 4·03 (R Foundation for Statistical Computing, Vienna, Austria).

For the usability assessment, the SUS score was calculated taking the mean of all answers for each test. For the EoU assessment, responses were scored on a predefined numerical scale and the mean values were summarized in a heat map. Both assessments were analysed with Microsoft Excel.

Role of funders

The funders and test manufacturers had no role in the study design, the data collection, the analysis, and decision to publish this study. Staff from the study sponsor, FIND, contributed to the study design, data analysis, and/or the interpretation of the data.

Results

In total, 8389 participants with suspected SARS-CoV-2 infections were screened for participation, with 7471 participants included in the analysis (study flow available in supplement Fig. S2). In Germany, a total of 5742 participants were enroled from Apr 17th 2020 to Mar 31st 2021. Enrolment in Brazil from July 27th to Sept 16th 2020 included a total of 1729 participants.

The total number of participants enroled per test were: (1) 1715 Rapigen (1239 Germany, 476 Brazil); (2) 1129 Standard F (676 Germany, 453 Brazil); (3) 2110 Standard Q (1710 Germany, 400 Brazil); and in Germany only (4) 729 Bioeasy; (5) 723 Fujirebio and (6) 665 Mologic; and in Brazil only (7) 400 Bionote. Of note, the evaluation of Bioeasy was stopped prior to reaching 100 RT-PCR positives as predefined specificity performance criteria were not met.

Study population

We present clinical and demographic characteristics by test overall and by test per country in Table 1 ((a) for tests evaluated in both countries; (b) for those only evaluated in one country). The median age of participants was 36 years (interquartile range (IQR): 28–49) with 3813 participants (51·1%) being female. On the day of testing, 5859 participants reported at least one symptom consistent with COVID-19, with more participants being asymptomatic in Germany (25·6%) compared to Brazil (2·2%). Average duration of symptoms at the time of testing was 4·5 days (IQR: 2–5 days). Of the 7471 analysed participants, 954 (12.8%) tested RT-PCR positive, with substantial difference in positivity rates between Germany (8.9%) and Brazil (25.7%). Median viral load was lower in Brazil (5·9 log10SARS-CoV2 RNA copies/mL) versus Germany (7·9 log10SARS-CoV2 RNA copies/mL). A total of 634 participants (66·4%) had a viral load >6 log10 SARS-CoV2 RNA copies/mL.

Table 1

Study population characteristics.

(a) Cohorts for tests evaluated in both Germany and Brazil
	Rapigen			Standard F			Standard Q
Characteristics	Overall	Brazil	Germany	Overall	Brazil	Germany	Overall	Brazil	Germany
n	1715	476	1239	1129	453	676	2110	400	1710
Age Information available:	n = 1712			n = 1127			n = 2107
Median (IQR)	38 (28–52)	43 (32–56)	36 (28–50)	36 (28–47)	38 (27–48)	35 (28–46)	34 (28–44)	37 (28–46)	34 (28–43)
Gender n (%) Information available:	n = 1692			n = 1120			n = 2100
Female	833 (49·2)	221 (46·7)	612 (50·2)	628 (56·1)	268 (59·2)	360 (54·0)	1060 (50·5)	229 (57·5)	831 (48·8)
Male	859 (50·8)	252 (53·3)	607 (49·8)	492 (43·9)	185 (40·8)	307 (46·0)	1040 (49·5)	169 (42·5)	871 (51·2)
Comorbidities n (%) Information available:	n = 1714			n = 1129			n = 2110
Yes	733 (42·8)	297 (62·5)	436 (35·2)	325 (28·8)	143 (31·6)	182 (26·9)	504 (23·9)	100 (25·0)	404 (23·6)
No	981 (57·2)	178 (37·5)	803 (64·8)	804 (71·2)	310 (68·4)	494 (73·1)	1606 (76·1)	300 (75·0)	1306 (76·4)
PCR Result n (%) Information available:	n = 1715			n = 1129			n = 2110
Positive	142 (8·3)	117 (24·6)	25 (2·0)	159 (14·1)	120 (26·5)	39 (5·8)	232 (11·0)	106 (26·5)	126 (7·4)
Negative	1573 (91·7)	359 (75·4)	1214 (98·0)	970 (85·9)	333 (73·5)	637 (94·2)	1878 (89·0)	294 (73·5)	1584 (92·6)
Reporting symptoms n (%) Information available:	n = 1699			n = 1119			n = 2094
Yes	1203 (70·8)	470 (98·7)	728 (59·5)	938 (83·8)	421 (93·6)	516 (77·1)	1887 (90·1)	396 (99.7)	1491 (87·9)
No	496 (29·2)	6 (1·3)	495 (40·5)	181 (16·2)	29 (6·4)	153 (22·9)	207 (9·9)	1 (0.3)	206 (12·1)
Symptoms duration in days Information available:	n = 1166			n = 925			n = 1848
Median (IQR)	4 (2–6)	5 (4–7)	3 (2–4)	4 (3–5)	4 (3–6)	3 (2–5)	4 (2–5)	5 (4–6)	3 (2–5)
Viral Load (log10 SARS-CoV2 RNA copies /mL) Information available:	n = 142	n = 159	n = 232
Median (IQR)	5·9 (4·0–7·5)	5·6 (3·8–7·3)	6·9 (5·5–8·1)	6·3 (4·9–7·5)	5·9 (4·5–7·2)	7·4 (6·0–8·2)	6·5 (5·0–7·6)	5·6 (4·4–6·5)	7·2 (5·9–8·3)

Invalid Ag-RDT results are included in study population characteristics.

Performance across tests

Three Ag-RDTs met the WHO targets for EUL of sensitivity > 80%. Summary results are presented in Figure 1. Mologic had a sensitivity of 90·1% (95% CI: 85·1%−93·6%) detecting 173 out of 192 positive RT-PCR cases. Bionote had a sensitivity of 89·2% (91 out of 102 detected; 95% CI: 81·7%−93·9%) and Standard Q was 81·9% sensitive (190 out of 232 detected; 95% CI: 76·4%−86·3%). Fujirebio had a sensitivity of 79·5% (89 out of 112; 95% CI: 71·1%−85·9%), just missing the WHO target. CIs for all four above mentioned tests were overlapping. Standard F showed a sensitivity of 75·5% (120 out of 159 detected; 95% CI: 68·2%−81·5%), and Rapigen had a sensitivity of 70·4% (100 out of 142 detected; 95% CI: 62·4%−77·3%).

Figure 1

Overall Performance for Ag-RDTs. N = Total number of cases included in analysis, TP = True Positives, FN = False Negatives, TN = True Negatives, FP = Faklse Positives, dashed red lines = WHO TPP cut-offs.

Mologic and Fujirebio showed a specificity of 100% (0 false-positives for both tests), Rapigen was 99·7% specific (4 false-positives/1573, 95% CI: 99·3%−99·9%) and Standard Q was 99·0% specific (18 false-positives/2110, 95% CI: 98·5%−99·4%). For Standard F and Bionote specificity was 97·2% (27 false-positives/970, 95% CI: 96·0%−98·1%) and 97·3% (8 false-positives/298, 95% CI: 94·8%−98·6%), respectively.

For Bioeasy, the evaluation was aborted after an interim analysis due to low specificity: 93·1% (49 false-positives/712, 95% CI: 91·0%−94·8%). Thus, the sensitivity estimates were calculated from a smaller sample size (66·7%, 10 out of 15 detected), resulting in wide CIs (95% CI: 41·7%−84·8%).

Invalid results were rare across all tests (0–2·4%), with the highest number reported for Mologic with 16 invalids results out of 665 participants (2·4%). Bioeasy had two invalid tests out of 729 (0·3%). The remaining five tests had no invalid results.

For tests with visual read-out, the following kappa results were reported: Tests with no discordance (kappa 1.0) were Rapigen, Fujirebio, and Bionote. Standard Q had minor discordance with kappa 0.997 and Mologic with kappa 0.996.

Subgroup analysis of sensitivity

The summary results are presented in Figure 2. Bioeasy was excluded from the subgroup analysis due to insufficient number of RT-PCR positive cases.

Figure 2

Performance of Ag-RDTs based on subgroup analysis. D+ = Number of PCR positive cases inlcuded in analysis, TP = True Positives, FN = False Negatives.

Subgroup analysis by viral load

Sensitivity analysed by viral load (as estimated from CT-values) showed consistently high sensitivities in participants with high viral load (≥6 log10 SARS-CoV2 RNA copies/mL) ranging from 88·7% to 94·8%. CIs were overlapping for all six tests. In comparison, there was more variability in performance for participants with viral load <6 log10 SARS-CoV2 RNA copies/mL and overall lower sensitivities ranging from 25·0% to 81.8%, with a sensitivity of 25% (95% CI: 12·0%−44·9%) for Fujirebio and 36·4% (95% CI: 15·2%−64·6%) for Mologic. The CI of Fujirebio did not overlap with that of Standard Q, Standard F and Bionote in this analysis. Figure 3 presents an overview of viral loads compared to Ag-RDT test results. In the supplement, Figs. S3–S14 depict further detail on the correlation of Ag-RDT results for the six tests (with sufficient number of RT-PCR positive cases) with viral load as measured by CT-value for all RT-PCR positive participants across CT-values and for those with symptoms by days since symptom onset. The data confirms the findings as outlined above and highlight the variability of viral load observed across the sequential evaluations.

Figure 3

Viral load for each test compared to Ag-RDT results. FN = False Negatives, TP = True Positives.

Subgroup analysis by symptom duration

Across the six Ag-RDTs, the sensitivity was highest for participants with symptom duration ≤3 days ranging from 87·1% to 95·1% and was similar across the six tests (CIs overlapping). In the first three days of symptoms, Bionote and Mologic showed a sensitivity of 95·1%; (95% CI: 83·9%−98·7%) and 93·8% (95% CI: 87·8%–97·0%), respectively. Variability of sensitivity was higher in participants with symptom duration of > 3 days and overall sensitivity was lower (69·7%−88·5%).

Sensitivity was slightly lower for participants with symptom duration of ≤ 7 days (compared to first 3 days), ranging from 77·5%−93·2%. Again CIs were overlapping for all tests. The sensitivity of tests for participants reporting > 7 days of symptoms decreased and ranged between 50·0% and 75·0%.

Subgroup analysis by presence of symptoms

When the analysis focused on participants with symptoms only, the overall sensitivity by test slightly increased to 72·7%–92·0% for all six Ag-RDTs. The sensitivity in participants where no symptoms were reported ranged from 40·0% to 100%, but the analysis was limited by few reported cases in asymptomatic participants (50). Viral loads of the asymptomatic participants ranged between 2·08 and 9·2 log10 SARS-CoV2 RNA copies/mL (supplementary material Fig. S15).

A post hoc analysis using a generalized linear model, controlling for symptom duration (3 categories 0–3, 4–7days. > 7days), viral load (continuous, log transformed) and age (continuous), only found a significant differences for a pairwise comparison of Mologic and Rapigen (p = 0.02) and for symptom duration (results available in the supplement).

SUS and ease-of-use assessment

Summary results are presented in Figure 4. The SUS results varied widely across the seven evaluated Ag-RDTs with scores ranging from 57 to 90 out of 100. The Ag-RDT considered most unsuitable for point-of-care testing was Mologic, despite the fact that it was the only test performed with anterior nasal sampling rather than NP sampling. The operators reported repeated problems with kit components, particularly when transferring the sample-buffer mix onto the lateral flow cassette. Bionote, Standard Q and Rapigen all performed similarly well, with Rapigen being the most-user friendly test with a score of 90. Fujirebio scored lower (75/100) due to its limited throughput owing to its additional steps (buffer-specimen needs 5 min incubation time and to start reaction of later flow assay a convex button in the Ag-RDT has to be pressed), longer time to readout (30 min), and no time window for result interpretation (readout at 30 min as per IFU). The tests that required a reader, Standard F and Bioeasy, scored overall slightly lower, primarily because of a reader limiting the throughput. Batch testing was considered cumbersome for all tests, because of the time-sensitive steps and the coordination needed to run tests in parallel. For all tests but Rapigen, the allowed storage temperature was a maximum of 30 °Celsius (Rapigen 40 °Celsius), which was considered to pose a potential major difficulty, particularly in countries with hotter climates. Also, the lower bound for operating temperature (Fujirebio at 20 °Celsius; other Ag-RDTs 15 °Celsius) was considered problematic particularly in countries with more moderate climate, especially where testing is performed outdoors.

Figure 4

System Usability Score and Ease-of-Use assessment results. The SUS score for each test is the mean score of all respondents who filled in the SUS (supplement material). The heat map includes the different aspects of the tests which were assessed by at least 3 respondents in the EoU survey (supplement material). The heat map was generated using a pre-defined matrix (Fig. S2). Number of participants: GE – Germany and BRA – Brazil Rapigen: 8 (6 GE, 2 BRA), SDF: 13 (7 GE, 6 BRA), SDQ: 13 (6 GE, 7 BRA), Bioeasy: 8 (8 GE, 0 BRA), Fujirebio: 6 (6 GE, 0 BRA), Bionote: 3 (0 GE, 3 BRA), Mologic: 6 (6 GE, 0 BRA).

Discussion

This multi-centre accuracy study provides an overview of the performance and ease-of-use of seven Ag-RDTs for the diagnosis of SARS-CoV-2. The findings of this study supported WHO decisions regarding EUL. Three tests (Mologic, Bionote and Standard Q) met sensitivity > 80% and specificity > 97% recommended by the WHO. Fujirebio came close to the sensitivity target with 79·5% and met the specificity target. When assessing groups by viral load, all tests presented high sensitivity (> 88·6%) on samples with higher viral loads (≥ 6 log10 SARS-CoV-2 RNA copies/mL). When testing occurs in the first week of infection, viral loads and correspondingly sensitivity are highest (86·8%–93·2% for the four tests with highest sensitivity). For some tests an insufficient number of samples with lower viral loads were included (21% for Fujirebio and 5·7% for Mologic), which demonstrates that populations tested were not directly comparable across all tests. For Mologic and Fujirebio also the overall mean viral load was much higher than average (7·8 and 8·3 log10SARS-CoV-2 RNA copies /mL), likely resulting in an overestimate of overall sensitivity. In contrast, for Rapigen, the split between high and low viral load samples was even, likely resulting in an underestimate of sensitivity. All tests included in the subgroup analysis performed very well (> 88%) in the first three days of symptoms. The early phase of illness is the period when most transmission occurs, and the high sensitivity of Ag-RDTs in comparison to RT-PCR during this period is important for detecting infectious cases., Our findings are largely confirmed by a limit of detection (LOD) study by Cubas-Atienzar et al. with the exception of results on Fujirebio. The Fujirebio test was more sensitive in the analytical evaluation and comparable to Mologic with an LOD using dry swabs of ≤ 5.0 × 102 pfu/ml only, while Bionote and Standard Q reach an LOD of 1.0–5.0 × 103 pfu/ml. The lower clinical sensitivity of Rapigen and Standard F in our study is in line with a lower LOD of ≥ 1.0 × 104 in the analytical study.

We acknowledge that viral loads were consistently higher in Germany than in Brazil. We speculate that the patients presented earlier in their disease. This is also suggested by the lower median duration since symptom onset in Germany (Table 1). The performance of tests studied was, nevertheless, lower in Germany (Figure 2). This suggests that factors other than viral load are at play affecting the performance. Viral loads were standardized in similar ways across the sites, which makes this unlikely to play a role. Virus circulating at the time of the study were only the wild type in Brazil and in Germany primarily wild type and alpha only in the first months of 2021, when Fujirebio and Mologic were evaluated. Both viruses are thought to have similar kinetics in the first week, thus differences in virus circulating are unlikely to have affected results. Thus, in summary, from the data available, we are not able to fully explain this finding.

Viral load kinetics have been described to be similar in asymptomatic and symptomatic infections. Our study included asymptomatic cases, but numbers were small, the day of exposure and thus likely the day of illness onset was unknown, explaining the variability in sensitivity. Other studies confirm that Ag-RDTs show comparable performance between asymptomatic and symptomatic cases with similar viral loads. This let us to combine data from sites, despite the fact that the percentage of asymptomatic was highly variable between sites in Brazil and Germany, which was likely attributable to the different capacity for contact tracing and testing.

Our study also shows excellent specificity for some of the Ag-RDTs (Standard Q, Fujirebio, Rapigen and Mologic), with results ≥ 99·9%. However, the results of Bioeasy demonstrate, that evaluating specificity as part of an independent validation is equally important to evaluating sensitivity, which was not appreciated by all regulators in the pandemic. Overall, our findings confirm the results from systematic reviews and meta-analyses showing variable performance across different Ag-RDTs particularly in respect to specificity and sensitivity for low viral load samples.31, 32, 33

In addition, our study demonstrates the importance of considering the ease-of-use of a test. From our experience with point-of-care testing, only the tests with a high SUS score, are likely to translate these study findings into successful performance in real world implementation. In addition to ease-of-use, the range of permitted storage and operating temperatures are critical. Temperatures both higher and lower than the recommended temperature ranges are a problem, particularly as testing facilities in the pandemic were often make-shift (in containers or drive-through) and especially in countries with extreme climates (e.g. in the Global South).,

Taking diagnostic accuracy assessed by viral load and ease-of-use together, Standard Q appears to be the test suitable for implementation, consistent with its WHO EUL. The specificity of Bionote was lower (97·3%), however, the sample size tested was smaller (resulting in lower precision around the estimate) and the evaluation was performed only in one site. While Mologic was highly sensitive, it was evaluated in participants with mostly high viral load and the assessment of performance in participants with low viral load was limited. Furthermore, the Ag-RDT had the highest number of invalid test results during evaluation. In addition, the ease-of-use of Mologic needs to be further optimized (e.g. kit components and buffer transferral), in order to ensure reliable performance, particularly when implemented in routine settings. AN sampling for Mologic was reported as user-friendly.

Overall, our study has several strengths. The population enroled for testing was representative of the pandemic dynamics observed in Germany and Brazil at the respective times. The tests were performed at point-of-care at all sites, thus mimicking the real-world challenges of near patient testing. The comprehensive ease-of-use assessment with a standardized SUS-tool and questionnaire, developed specifically for this study, captured the differences between the tests and highlighted important points for operationalization of the tests.

However, the study also has several limitations. First, the prevalence of positive cases, viral load distribution and the percentage of symptomatic participants varied substantially over the course of the study amongst tested individuals and also differed between the sites and countries. This likely reflects different phases of the pandemic, with variability in the health care system's ability to track and trace (primarily reflected in the number of asymptomatic tested), in patient behaviour (e.g., presenting earlier in their illness) and recommendations for testing. Thus, the comparisons of performance across tests are limited by the fact that they were done sequentially and not on the same participant. However, it should be acknowledged that the swabbing procedure is uncomfortable, and few participants would likely consent to multiple additional swabs. Most early exclusions were due to refusal of a second swab after routine swabbing for RT-PCR testing. In addition, if multiple swabs are done sequentially in the same location on one participant, this could affect sample quality. Furthermore, variability in strains circulating is to be expected, although this is unlikely to affect Ag-RDTs performance. It has to be acknowledged that the Ag-RDTs could only be evaluated based on the most common strains (WT and B 1.1.7) circulating at the time of evaluation. This limits the generalizability of the diagnostic accuracy to other variants that have since evolved. An analysis by virus variants for Fujirebio is under submission (personal communication Andreas Lindner). To date no virus variant has been shown to affect performance of commonly used Ag-RDTs.,

In addition, not all Ag-RDTs were evaluated in both countries, which also limits comparability, however subgroup-analyses by viral load and symptom duration enable more comparable results. In addition, Ag-RDT results were compared to RT-PCR, using different SARS-CoV-2 assays, limiting the comparability of the results. Furthermore, we also acknowledge, that the RT-PCR reference standard has its limitations, as it is not always a meaningful test when considering viable virus and risk of transmission. By using the RT-PCR reference standard (instead of for example viral culture), we might have underestimated the performance of the Ag-RDTs when it comes to detection of viable virus. To account for the variability, CT-values were translated into viral load; however this transformation comes with its own limitations., Lastly, our study was performed excluding vaccinated individuals and those with prior infections, given the uncertainty around possibility of break-through infections at the time of protocol writing. Today, we know that vaccinated individuals have slightly lower viral loads in the case of break through infections. This is likely to diminish the sensitivity of Ag-RDT in this population group overall, however, the tests should continue to detect the most infectious individuals.

In summary, our prospective, multi-centre diagnostic accuracy study demonstrates high sensitivity particularly early in a SARS-CoV-2 infection and high specificity for six different Ag-RDTs in comparison to RT-PCR. With a fast turn-around, these well performing and easy-to-use tests can be a useful screening tool to identify SARS-CoV-2 cases rapidly, and contribute to pandemic control. Further implementation research and economic evaluations are needed to translate the study findings into optimized testing strategies.

Declaration of interests

We declare no competing interests.