Rapid antigen test for SARS-CoV-2: results of validation and use in real life.

Aim: To verify a SARS-CoV-2 rapid antigen test (RAT) compared with PCR. Materials & methods: Validation of RAT included 2295 subjects. Next matching of RAT with the PCR was checked in 13,852 subjects referred to PCR after being positive in RAT.
Results: Sensitivity and specificity of RAT were 77.38 and 99.10%, respectively. A 74.60% of RAT positive results were confirmed with PCR.
Conclusion: The test met WHO susceptibility criteria in a group of symptomatic subjects. In terms of specificity, it met requirements in all subjects. The concordance of RAT with PCR in real life was in line with our verification data.

The gold standard for the microbiological diagnosis of COVID-19 is the detection of SARS-CoV-2 virus RNA by RT-PCR after nucleic acid isolation and reverse transcription PCR. Due to various, especially logistical reasons, the use of PCR is not always possible. Rapid antigen tests (RATs) for the detection of SARS-CoV-2 antigens (RAT) are a complement to PCR. They are cheaper, the execution time is shorter (up to 30 min), it can be performed by staff without prior thorough laboratory training [1-10].

The RAT, selected by the Ministry of Health through a public procurement procedure, has a European compliance (CE) and in vitro diagnostic (IVD) certificate. In order to obtain the CE certificate, manufacturer performed clinical validation on 273 subjects, there is no official reference. Of the 73 positives by PCR, 70 were also positive by RAT. All PCR-positive subjects were classified as symptomatic. All 200 PCR-negative subjects were also RAT negative and were defined as asymptomatic. The manufacturer thus stated 95.9 sensitivity and >99.9% specificity of the test, thus satisfying the analytical test criteria in the procurement procedure, the national recommendations for the use of RAT and the recommendations of the WHO [11].

The purpose of our study was to verify whether the RAT meets the manufacturer’s indications (95.9% sensitivity and specificity >99.9%), whether the RAT meets the national recommendations for the use of the RAT in terms of sensitivity and specificity of the test (at least 90% sensitivity in independent verification and >98% specificity), to verify whether the RAT meets WHO recommendations (>80% sensitivity and specificity greater than 97%) and to determine the sensitivity and specificity of the test in the asymptomatic population [12-15].

The study was performed at the end of the second wave of the epidemic when the predominant virus type was a wild-type SARS-CoV-2.

Materials & methods

Implementation of the verification process

The National Laboratory for Health, Environment and Food (NLZOH), in accordance with the instructions of the Ministry of Health, proceeded with a verification protocol for assessing the acceptability of the RAT antigen SARS-CoV-2 Test Kit, Shenzen Ultra-Diagnostics Biotec. Co., Ltd. This is an immunochromatographic test performed on the principle of lateral flow, as the detection reagent is colloidal gold bound to antibodies. The reference RT-qPCR method was Allplex 2019-nCoV, manufactured by Seegene Inc., South Korea. RNA was extracted from clinical samples with the STARMAG UNIVESAL kit (Seegene Inc., South Korea).

The survey lasted from 11 January to 9 February 2021. In that time in the country was the prevalent B.258.17 variant SARS CoV-2. We followed the validation protocol recommended by the WHO (www.finddx.org/covid-19/pipeline).

The validation was carried as a prospective cohort study; it took place at several sampling points of a mass testing for general asymptomatic population and hospitals and community health centers, where symptomatic patients were tested. Consecutive subjects were invited to participate. Subjects, who agreed to participate were explained the aim and the protocol of the study. They signed informed consent. Before taking swabs, the subjects were interviewed about the presence/absence and type of symptoms, data on the duration of symptoms, age, gender and possible contacts. Two nasopharyngeal swabs were taken, one for RAT and the other for RT-qPCR.

For analyzing the data participants were divided in two main groups (asymptomatic and symptomatic) and then there were four subgroups according to viral load. (Ct) in PCR test (Ct value below or equal to 25, Ct value below or equal to 30, Ct value below or equal to 33 and Ct value above 30). Each participant could be included in more than one group.

Sampling took place in the National Laboratory for Health, Environment and Food at the locations of Murska Sobota, Maribor, Celje, Novo mesto, Ljubljana, Kranj and Koper Nova Gorica, at the National Institute of Public Health in Ljubljana, at the entry points of the Health Centers Kranj, Celje, Maribor, Murska Sobota, Krško and Jesenice, points of mass testing Kranj and Cerklje na Gorenjskem and biathlon workers testing at the Congress Center in Bled.

Analysis of data from national databases

In the second part of the study we analyzed the matching of RAT and the RT-qPCR results of subjects, who were invited for PCR testing after being positive in routine RAT performed during mass testing of asymptomatic individuals. At the National Institute of Public Health (NIJZ), we obtained anonymous data on test results from the Central Register of Patient Data.

Statistical analysis

Data are presented as a mean and standard deviation.

The sensitivity and specificity of the test were calculated according to the following formula. Results were given with a 95% CI. The 95% CIs were calculated to assess the level of uncertainty introduced by sample size, using the Wilson’s score method.

The sensitivity was calculated according to the following formula: (A/E) × 100. The sensitivity of RAT tells what proportion of PCR-positive samples is RAT positive.

Specificity expressed as a percentage was calculated according to the following formula: (D/F) × 100. The specificity of RAT tells what proportion of PCR negative samples is RAT negative.

Results

Result of the verification process

In the period from 11 January to 9 February 2021, we collected and analyzed samples from 2295 subjects (37.38% men).

In 413 symptomatic cases, the following symptoms were reported: cold: 48.91%, cough: 45.03%, muscle pain: 35.11%, fever: 34.62%, sore throat: 32.92%, altered smell or taste: 20.82%, dyspnea: 15.73% and digestive problems: 14.04%.

A total of 19 people (0.8%) reported reinfection of new coronavirus infection.

In the whole group the sensitivity of RAT was 77.38% (95% CI: 72.26–81.95%) and specificity 99.10% (95% CI: 98.57–99.46%). In a subgroup of 413 symptomatic subjects the sensitivity of RAT was 86.43% (95% CI: 80.88–90.86%) and the specificity 98.60% (95% CI: 95.96–99.71%). In 1882 subjects without pronounced disease signs the sensitivity of RAT was 60.38% (95% CI: 50.41–69.75%) and the specificity 99.16% (95% CI: 98.62–99.53%) (Table 1). Sensitivity of RAT was lower in subjects with low viral load (Ct ≥ 30), regardless of being symptomatic (53.33%) or asymptomatic (27.27%).

Table 1.

Rapid antigen test sensitivity in groups of positive symptomatic, positive asymptomatic subjects and all positive together with different Ct values for the E gene in PCR.

Subgroup of subjects symptomatic	RAT positive (n)	PCR positive (n)	Sensitivity (%)
Ct ≤25	88	90	97.80 (95% CI: 100–94.80%)
Ct ≤30	139	143	97.20 (95% CI: 99.9–94.50%)
Ct ≤33	160	172	93.02 (95% CI: 96.82–89.22%)
Ct ≥30	24	45	53.33 (95% CI: 67.91-38.75%)

RAT: Rapid antigen test.

Result of analysis of data from national databases

The number of all positive subjects with RAT in that period was 15,685. Of those, 1833 were not verified by PCR. Of 13,852 (88.31%) checked by PCR, 10,333 (74.60%) were positive and 3519 (25.40%) were negative or the result of PCR was unclear. The concordance of RAT with PCR is 74.6%, thus 25.40% of RAT were falsely positive.

Discussion

The purpose of the validation of SARS-CoV-2 antigen test Shenzen Ultra – Diagnostics Biotec. Co., Ltd. was to confirm the analytical capabilities of the method as defined by the manufacturer, as in a different test conditions the performance of the test might be different. Beside, we obtained additional information on the performance characteristics of the test in different subgroups of subjects.

The RAT subjected to verification belongs to the group of immunochromatographic tests based on the reaction of the antibody bound to the test field with the virus specific antigen present in the sample. Studies have shown that HATs are 102–105-fold less sensitive than RT-PCR [2]. The PCR method is more sensitive as it is an amplification reaction according to a log-scale. The number of copies of the virus is doubled in each cycle, which allows less than 100 copies of the virus in the sample to give a positive reaction in the test. Best results obtained for the E gene and RdRp gene assay (5.2 and 3.8 copies per reaction at 95% detection probability) [1]. The virus concentration in the sample can be approximately estimated using the Ct values of the positive sample. The Ct value is the number of cycles in which the target is multiplied to such an extent that the measured signal exceeds the threshold separating the negative from the positive result. Ct values between different PCR tests are not completely comparable for a number of technical reasons. Nevertheless, an approximate correlation exists: the lower the Ct value, the higher the virus concentration in the sample. It is therefore quite understandable that RAT detects infections more successfully among those subjects in whom the virus has multiplied to high concentrations, and the calculated sensitivity of the test consequently depends on the clinical presentation of the infection of the subject population [1-6].

Test sensitivity depends on case mix of subjects studied, according to the viral load. The higher the proportion of subjects with a high viral load in the test group, the better is the calculated sensitivity of the RAT. As can be seen from the results, the sensitivity is highest in the group of symptomatic PCR-positive subjects with low Ct values, namely 97.8%, and the lowest in the group of asymptomatic subjects with high Ct values, namely 27.27%.

Cycle 25 and 33 were selected according to the recommended WHO protocol, and Cycle 30 was chosen based on some views that it is unlikely that a person with such a PCR result would still excrete infectious viral particles. WHO recommends that the test should achieve more than 80% sensitivity and specificity greater than 97% at Ct values up to 30, and that the lower limit of the 95% CI for sensitivity be above 80% [11].

That the sensitivity of the rapid test is the worst in the group of subjects with low viral load (Ct above 30) is the expected result, as the analytical sensitivity (detection threshold) in RATs is up to 100,000-times less (threshold higher) than the analytical sensitivity of PCR [2]. In the verification procedures, they approach the manufacturer’s requirements in the group of subjects with lower Ct values and therefore higher viral loads. If the proportion of those with lower viral loads in the group of subjects is higher, the verification procedure usually fails to confirm the manufacturer’s statements. This is also in our case.

After introduction of preventive COVID-19 testing of asymptomatic population with RAT in many subject the test was positive. To discriminate between asymptomatic infected subjects and subjects with a false positive RAT the decision was made to confirm all positive RAT with a PCR. It turned that the rate of false positive RAT in real life is 25.40%, which is also in line with our verification data.

The accuracy of COVID-19 antigen RDTs Shenzhen Bioeasy Biotechnology Co. Ltd was assessed in a prospective studies across multiple, independent sites using consecutive enrolment. The results were compared with diagnostic RT PCR result, which was used for clinical management. The evaluation study was made in Germany with 729 subjects, 86.1 % symptomatic, 2.1 % positivity COVID-19. Clinical sensitivity was 66,7%, specificity 93.1%. Evaluation was stopped after preliminary analysis indicated specificity below 97% [16].

In a German study RAT showed slightly higher sensitivity in adults (30.4%; 95% CI: 18.8–90.9%) than in children (20.8%; 95% CI: 7.1–42.2%). The sensitivity was 80% (59.3–93.2%) with Ct value <30. True positive RAT had a statistically lower Ct value in reference RT PCR (p < 0.001) compared with false negative RAT. The specificity was excellent. With the right cut-off the RAT can be useful to distinguish SARS-CoV-2-infected subjects who can transmit the infection from noninfectious people, enabling appropriate triage while waiting for the RT-PCR result [17].

Our verification results showed that the RAT meets the manufacturer’s specifications regarding the sensitivity of the test in the group of symptomatic subjects with Ct values up to 30; however, regarding specificity verification results did not meet the manufacturer’s specifications in any group. The test meets the criteria of the national guidelines for the use of RAT regarding sensitivity in the group of subjects with Ct values up to 33. In terms of specificity the test meets the criteria of the national guidelines for the use of RAT in all tested groups. The test meets WHO susceptibility criteria with and without a 95% CI in a group of symptomatic subjects. In terms of specificity, it meets the requirements both in the groups of symptomatic and asymptomatic subjects.

In the group of asymptomatic subjects, the high specificity (86.43%) of RAT was confirmed, which, when performed correctly, means that it gives very few false-positive results. The sensitivity of the test in the whole group of asymptomatic subjects was 60.38%. The test detects subjects without symptoms and with a high viral load as well as subjects with pronounced symptoms and a high viral load. Unlike the PCR test, RAT detects subjects with low viral load poorly, whether or not they show signs of disease. From a comparison with the results of the other verifications listed in Table 2, it can be concluded that the RAT gives comparable results with other already verified RATs. The verification confirms that, when using RAT, it is of the utmost importance that the value of the negative result is correctly presented to the subjects, which should not lead to a feeling of false security. The basic purpose of using RAT is to detect people with a high viral load and send them to self isolation as soon as possible.

Table 2.

Comparison of rapid antigen tests results of different manufacturers performed according to the WHO protocol.

The mark of the verified test according to the legend	1	1S	2	2	2	3	3	4	5	5
Testing country	SI	SI	BR	DE	CH	CH	DE	BR	BR	DE
All tested (%)	2295	413	400	1263	529	535	1108	400	476	1239
Proportion of symptomatic (%)	18,0	100	98,7	84,6	99,8	99,8	64,5	100	98,7	59,9
PCR positive (%)	13,3	48,2	26,5	3,7	36,1	21,5	9,6	25,5	25	2
RAT positive within PCR-positive sensitivity (%)	77,4	86,4	88,7	76,6	89	85,5	90,8	89,2	74,4	52
RAT positive within PCR-positive Ct ≤ 33 (%)	92,9	93,0	91,9	87,8	91,8	89,7	88,3	91,4	82,5	61,9
RAT positive within PCR-positive Ct ≤ 25 (%)	97,7	97,8	95,9	100	97,2	96,8	95,8	94,8	90,9	80
RAT negative within all PCR-negative specificity (%)	99,1	98,6	97,6	99,3	99,7	100	99,9	97,3	98,95	100

1 – Antigen SARS-CoV-2 Test Kit, Shenzen Ultra-Diagnostics Biotec. Co. Ltd all subjects.

1S – Antigen SARS-CoV-2 Test Kit, Shenzen Ultra-Diagnostics Biotec. Co. Ltd symptomatic subjects.

2 – STANDARD Q COVID-19 Ag Test, SD Biosensor Inc.

3 – Panbio COVID-19 Ag rapid test device, Abbott.

4 – NowCheck COVID-19 Ag test, Bionote, Inc.

5 = BIOCREDIT COVID-19 Ag, RapiGEN, Inc.

BR: Brazil; CH: Switzerland; DE: Germany; RAT: Rapid antigen test; SI: Slovenia.

The use of rapid lateral flow antigen testing for SARS-CoV-2 has been questioned with uncorroborated reports of poor sensitivity. In a national systematic evaluation of sensitivity and specificity Peto et al. reported, that only orient Gene, Deepblue, Abbott and Innova SARS-CoV-2 Antigen Rapid Qualitative Test have desirable performance characteristics with viral antigen detection of >90% at 100,000 RNA copies/ml [18]. The remaining 60 lateral flow diagnostics (LFDs) showed unsatisfactory results [18].

Our point is that even tests for home use must have the required sensitivity and specificity, because otherwise it is misleading and a false sense of security. However, there are some concerns whether RT-PCR is a real gold standard test for COVID-19. Namely, in many individuals viral RNA fragments can be detected for weeks months without any evidence of active viral replication [19,20].

Conclusion

The verification results showed that:

The test meets the manufacturer’s specifications regarding the sensitivity in symptomatic subjects with Ct values up to 30. However, the test did not meet the manufacturer’s specifications in terms of specificity in any group.

The test satisfied the criteria of the national guidelines for the application of the RAT regarding sensitivity in the group of subjects with Ct values up to 33 only. In terms of specificity the test satisfied the criteria both symptomatic and asymptomatic subjects.

The test satisfied the WHO sensitivity criteria with and without a 95% CI in the group of symptomatic subjects. In terms of specificity, it met requirements both in the groups of symptomatic and asymptomatic subjects.

In the group of asymptomatic subjects, high specificity of RAT was confirmed. When performing the test correctly it gives few false positive results.

Unlike PCR test, RAT poorly detected subjects with a low viral load, regardless of whether they show signs of illness or not.

From a comparison with the results of the other verifications listed in Table 2, we can conclude that RAT verified in our study gave comparable results with other already verified RATs.

The gold standard for the microbiological diagnosis of COVID-19 is the detection of SARS-CoV-2 virus RNA by RT-PCR.

Rapid antigen tests (RATs) for the detection of SARS-CoV-2 antigens (RAT) are a complement to PCR with its advantages and limitations.

The purpose of the validation of SARS-CoV-2 Antigen Test Shenzen Ultra – Diagnostics Biotec. Co., Ltd was to confirm the analytical capabilities of the method as defined by the manufacturer.

The sensitivity of RAT in the whole group of subjects in the verification analysis was 77.38% (95% CI: 72.26–81.95%) and specificity 99.10% (95% CI: 98.57–99.46%).

The sensitivity was high (97.20%) in symptomatic subjects with Ct ≤30, but only 27.27% in asymptomatic subjects with Ct ≥33.

The correspondence of RAT with PCR in real life was 74.60% within 15,685-RAT positive subjects and it is in line with our validation data.

Our point is that even tests for home use must have the required sensitivity and specificity, because otherwise it is misleading and a false sense of security.