Sensitivity of severe acute respiratory syndrome coronavirus type 2 rapid antigen point-of-care tests in vaccinated patients.

BACKGROUND AND IMPORTANCE: Rapid antigen point-of-care tests (antigen POC tests) are frequently used to detect COVID-19 infections. Based on clinical impressions, we suspected that the sensitivity of antigen POC tests might be lower in vaccinated patients.
OBJECTIVE: To evaluate the sensitivity of antigen POC tests in vaccinated patients. DESIGN, SETTING AND PARTICIPANTS: We retrospectively evaluated all patients over 18 years of age that tested positive for severe acute respiratory syndrome coronavirus type 2 (SARS-CoV-2) in November 2021 at our institution, whose antigen POC test result had been recorded. We considered patients who were either fully vaccinated or unvaccinated, as well as whether they were symptomatic. OUTCOME MEASURE AND ANALYSIS: We calculated the sensitivity of antigen POC tests in vaccinated and unvaccinated as well as in symptomatic and asymptomatic patients and compared the results.
RESULTS: A total of 4080 reverse transcription PCR tests were performed in our institution in November 2021, of which 360 patients tested positive for SARS-CoV-2. Of those, 234 patients met inclusion criteria and were further evaluated. The sensitivity of the antigen POC test was only 39.4% [95% confidence interval (CI), 31.3-48.1%] in vaccinated patients and differed significantly from the sensitivity of 53.3% (95% CI, 43.9-62.4%) in unvaccinated patients (difference of sensitivities, 13.9%; 95% CI, 1.1-26.1%). In symptomatic patients, the sensitivity increased by only 11.8% from 45.7% in all patients (95% CI, 39.5-52.1%) to 57.5% in symptomatic patients (95% CI, 49.9-64.7%). In asymptomatic patients, the antigen POC test was only able to detect SARS-CoV-2 positive patients in 16.4% of the cases (95% CI, 9.4-27.1%).
CONCLUSION: Point-of-care antigen tests are likely not useful for ruling out SARS-CoV-2 infection, especially in vaccinated and asymptomatic patients, potentially due to lower viral load. Moreover, the use of these tests might lead to a false sense of security, especially when used by the public as part of a public health testing strategy.

Introduction

Rapid antigen point-of-care tests (antigen POC tests) have become a mainstay in fighting the Coronavirus Disease 2019 (COVID-19) pandemic. Timely diagnosis of Severe Acute Respiratory Syndrome Coronavirus Type 2 (SARS-CoV-2) infection can help to accelerate the treatment of patients, contribute to the protection of healthcare workers, and prevent the spread of the disease. Additionally, early detection allows for more appropriate allocation of limited resources for isolating patients.

Antigen POC tests are relatively inexpensive, easy to perform, and return results in approximately 15 min. Although reverse transcription PCR (RT-PCR) is the recommended gold-standard for diagnosis of SARS-CoV-2 infection, it is time-consuming, requires special equipment, and requires specially trained staff, thereby limiting the timely availability of this test [1]. As a result of these limitations, some hospitals have adopted a strategy whereby patients are swabbed in the emergency department, and antigen POC tests and RT-PCR are performed simultaneously at the time of hospital admission [2]. Additionally, antigen POC tests are frequently embedded in routine testing strategies that enable the social life of staff outside of the hospital.

Manufacturers indicate a high sensitivity of typically around 95% for their tests, but several studies have already demonstrated that in ‘real world’ settings, the sensitivity of antigen POC tests is much lower and depends on the viral load of the tested patients. Patients with low RT-PCR cycle threshold-values (Ct-values) are potentially more infectious, with a higher viral load, and are, therefore, more reliably detected by antigen POC tests [3,4].

Since the end of 2020, several COVID-19 vaccines have become available. In patients with SARS-CoV-2 infection that have been previously vaccinated, the disease is less severe [5], and this likely impacts the sensitivity of antigen POC tests due to altered viral dynamics and viral load in these patients when compared with unvaccinated patients [6]. Therefore, antigen POC tests may have a lower sensitivity in vaccinated patients and may be of less value in the detection or for rule-out of SARS-CoV-2 infection in this population.

We conducted this study to evaluate and compare the sensitivity of antigen POC tests in vaccinated and unvaccinated individuals, hypothesizing that antigen POC tests would be less sensitive in vaccinated individuals.

Methods

Study design

The study was designed as a retrospective cohort study and was conducted during December 2021. We evaluated all patients that tested positive for SARS-CoV-2 in our institution during November 2021 for the fulfillment of inclusion criteria. The study was approved by the ethics committee of the state chamber of physicians of Baden–Wuerttemberg on 7 December 2021 with the study number F-2021-162 and had been preregistered in the German Clinical Trials Register under DRKS00027348.

Participants

Based on our sample size calculation, we included all patients that had tested RT-PCR positive for SARS-CoV-2 in November 2021 in our institution, that had an antigen POC test with recorded result done on the same day, that had been either fully vaccinated or not vaccinated, and that were over 18 years of age. Fully vaccinated was defined as either receiving two doses Comirnaty (BioNTech/Pfizer, Mainz, Germany), Spikevax (Moderna Biotech, Cambridge, Massachusetts, USA), Vaxzevria (Oxford/Astra-Zeneca, Cambridge, UK), a combination of these, one dose of COVID-19 Vaccine Jansen (Johnson & Johnson, New Brunswick, New Jersey, USA), or a combination of convalescence and one dose of any vaccine, as well as all of the above that had already received a third (booster) dose.

Procedures

We collected the results of RT-PCR testing (Ct-values) and antigen POC testing, as well as the immunization status. In fully vaccinated patients, we also recorded the time of last COVID-19 vaccination and the vaccine used. We recorded whether these patients were symptomatic (dyspnea, cough, fever, loss of smell, ageusia, abdominal discomfort, diarrhea, vomiting, fatigue, headache, back pain, limb pain, and cold) or asymptomatic.

The standard antigen POC test in our institution is the NADAL COVID-19 rapid antigen test (nal von minden GmbH, Moers, Germany). The manufacturer states a high sensitivity of 97.6% in high viral loads between Ct-values of 20 and 30 and a specificity of more than 99.9%. Patients received an oropharyngeal swab and all testing was done according to the manufacturer’s recommended process [7]. Simultaneously, patients received a second swab for RT-PCR testing from the same site. RT-PCR testing was done either with Xpert Xpress SARS-CoV-2 for GeneXpert (Cepheid, Sunnyvale, California, USA) for detection of E-gene and N2-gene, BD SARS-CoV-2 Reagent for BD max (Becton Dickinson, Franklin Lakes, New Jersey, USA) for detection of N1-gene and N2-gene or ViroKey SARS-COV-2 RT-PCR v2.0 (Vela Diagnostics, Singapore) with Rotor-Gene-Q (Qiagen, Venlo, The Netherlands) for detection of N-gene and ORF1a-region. The tests were all performed according to the manufacturers’ recommended processes. Both tests were collected simultaneously from the same site (oropharyngeal) in the emergency department at the time of hospital admission with the intention to detect infectious patients early during hospital stay (antigen POC test) and to not miss infectious patients (RT-PCR).

In a comparison of commercially available antigen POC tests, Scheiblauer et al. [8] reported a sensitivity of 83.3% at Ct-values 25 or less for the antigen test used in this trial. We used this value as the presumed sensitivity for unvaccinated patients. Based on our clinical impressions, we predicted a sensitivity of only 65% for vaccinated patients with breakthrough infections. Using Fisher’s exact test, a type-I-error of 5%, a power of 80%, and the assumption that 25% more vaccinated than unvaccinated patients would be enrolled, we obtained a sample size of 93 for unvaccinated and 115 for vaccinated patients, respectively [9]. Based on this analysis, our goal was to include at least 100 unvaccinated and 125 vaccinated patients in our trial.

Outcomes

Our primary endpoint was the sensitivity of the antigen POC test used in our institution in vaccinated and unvaccinated patients as well as a comparison between these groups. Secondary endpoints were the Ct-values determined by RT-PCR testing in vaccinated and unvaccinated patients, as a possible explanation for potential differences in sensitivity in vaccinated versus unvaccinated individuals. To calculate sensitivity, we defined patients with a SARS-CoV-2 positive test result by RT-PCR to be true positives. This was decided based on the very high sensitivity and specificity of RT-PCR, and is consistent with prior research comparing antigen POC tests with RT-PCR results.

Statistical analysis

The data were collected from our hospital information system (Orbis, Dedalus Healthcare GmbH, Bonn, Germany) and transferred to a spreadsheet in Excel (Excel, Microsoft Corporation, Redmond, Washington, USA). Statistical analysis was done with R (R Core Team 2021, Vienna, Austria). Patient characteristics were described using mean, SD, median, and interquartile range (IQR). Confidence intervals (CIs) for sensitivities of antigen POC tests as well as for differences of sensitivities were calculated by the method of Wilson. Differences between Ct-values of different groups were calculated with mean and Welch two-sample t-test.

Results

A total of 4080 RT-PCR tests were performed in our institution in November 2021 of which 360 patients tested positive for SARS-CoV-2. Of those, 234 patients met inclusion criteria and were enrolled for further evaluation.

Sex distribution was comparable, with 111 women (47.4%) and 123 men (52.6%) participants. The mean age was 62.8 (SD, 21.2; median, 69; IQR, 33) for women and 63.8 (SD, 18.7; median, 66; IQR, 30.5) for men participants.

Sensitivity of antigen point-of-care tests in unvaccinated versus vaccinated patients

The sensitivity of the antigen POC tests used in our institution differed significantly between unvaccinated (53.3%; 95% CI, 43.9–62.4%) and vaccinated (39.4%; 95% CI, 31.3–48.1%) patients (95% CI of difference of sensitivities, 1.1–26.1%). The overall sensitivity of the antigen POC tests for all patients was 45.7% (95% CI, 39.5–52.1%). Antigen POC test results of SARS-CoV-2 positive patients with respect to their immunization status are presented in Table 1.

Table 1

Antigen point-of-care test results in SARS-CoV-2 positive patients and immunization status

Point-of-care test result/ Immunization status	Vaccinated (n)	Unvaccinated (n)
Antigen POC positive	39.4% (50)*	53.3% (57)*
Antigen POC negative	60.6% (77)	46.7% (50)

POC, point of care.

*95% CI of difference = 1.1–26.1%.

Sensitivity of antigen point-of-care tests in symptomatic and asymptomatic patients

The sensitivity of the antigen POC test used in our institution differed significantly between symptomatic (57.5%; 95% CI, 49.9–64.7%) and asymptomatic (16.4%; 95% CI, 9.4–27.1%) patients (difference, 41.1%; 95% CI, 28.0–51.1%). Antigen POC test results in symptomatic and asymptomatic SARS-CoV-2 positive patients are presented in Table 2.

Table 2

Antigen point-of-care test results in symptomatic and asymptomatic SARS-CoV-2 positive patients

Point-of-care test result/ Presence of typical symptoms	Symptomatic (n)	Asymptomatic (n)
Antigen POC positive	57.5% (96)*	16.4% (11)*
Antigen POC negative	42.5% (71)	83.6% (56)

POC, point of care.

*95% CI of difference = 28.0–51.1%.

Association between vaccination status and presence of typical symptoms of COVID-19

As shown in Table 3, vaccinated patients with COVID-19 breakthrough infections presented significantly less frequently with typical symptoms (vaccinated, 62.2%; 95% CI, 53.3–70.2% vs. unvaccinated, 82.2%; 95% CI, 73.9–88.3%; difference, 20.0%; 95% CI, 8.5–30.6%).

Table 3

Association between vaccination status and presence of typical symptoms of COVID-19

Presence of typical symptoms/ Immunization status	Vaccinated (n)	Unvaccinated (n)
Symptomatic	62.2% (79)*	82.2% (88)*
Asymptomatic	37.8% (48)	17.8% (19)

*95% CI of difference = 8.5–30.6%.

Difference of reverse transcription PCR cycle threshold values between positive and negative antigen point-of-care test results

Ct-values (N-gene, ORF1a-region, N1-gene, N2-gene, and E-gene) in patients tested positive with antigen POC tests were all significantly lower than for patients that were tested negative with antigen POC tests (N-gene: P = 7.7 × 10−5; ORF1a: P = 1.5 × 10−4; N1-gene: P = 4.2 × 10−5; N2-gene: P = 4.0 × 10−16; E-gene: P = 9.0 × 10−13). Figure 1 shows the means and 95% CI of Ct-values in antigen POC test positive and negative tested patients.

Fig. 1

Means and 95% CI of Ct-values in antigen POC test positive and negative patients (*P < 0.001). CI, confidence interval; Ct, cycle threshold; POC, point of care.

Difference of reverse transcription PCR cycle threshold values between symptomatic and asymptomatic patients

An analysis of the RT-PCR Ct-values of symptomatic and asymptomatic patients showed a significant difference only for the N1-gene (N-gene: P = 0.295; ORF1a: P = 0.246; N1-gene: P = 0.016; N2-gene: P = 0.056; E-gene: P = 0.133). Nevertheless, Ct-results were consistently higher in asymptomatic patients. Ct-values of symptomatic and asymptomatic patients are presented in Table 4.

Table 4

Mean (SD) cycle threshold-values in symptomatic and asymptomatic patients

Nucleotid sequence/ Presence of typical symptoms	Symptomatic	Asymptomatic
N-gene	24.80 (7.11)	26.70 (5.83)
ORF1a-region	24.00 (7.13)	26.22 (5.82)
N1-gene	21.74 (5.10)	26.82 (6.14)
N2-gene	25.38 (5.23)	27.77 (6.07)
E-gene	23.90 (5.29)	26.90 (6.83)

Difference between reverse transcription PCR cycle threshold-values of vaccinated and unvaccinated patients

An analysis of RT-PCR Ct-values of vaccinated and unvaccinated patients showed significantly lower Ct-values in unvaccinated patients only for N-gene and ORF1a-region (N-gene: P = 0.012; ORF1a: P = 0.019; N1-gene: P = 0.800; N2-gene: P = 0.417; E-gene: P = 0.372) results. Ct-values in vaccinated and unvaccinated patients are presented in Table 5.

Table 5

Mean (SD) cycle threshold-values in vaccinated and unvaccinated patients

Nucleotide sequence/ Immunization status	Vaccinated	Unvaccinated
N-gene	27.49 (6.10)	23.23 (5.96)
ORF1a-region	26.97 (6.02)	22.91 (6.10)
N1-gene	23.81 (5.78)	24.38 (6.67)
N2-gene	25.46 (5.04)	26.13 (5.84)
E-gene	23.78 (4.92)	24.60 (6.01)

Discussion

In this study, we evaluated the sensitivity of the antigen POC test used in our institution in vaccinated and unvaccinated patients.

The main findings were as follows.

The sensitivity of the antigen POC tests differed significantly between vaccinated and unvaccinated patients with a sensitivity of only 39.4% in fully vaccinated patients.

In patients with typical symptoms of COVID-19, the sensitivity of the antigen POC tests only increased by 11.8% (from 45.7 to 57.5%). In asymptomatic patients, the antigen POC test was only able to detect SARS-CoV-2 positive patients in 16.4% of the cases.

The Ct-values of RT-PCR tests were significantly lower in patients that tested positive by antigen POC test.

The COVID-19 pandemic is dynamic and so is our knowledge and understanding of this disease. With the availability of vaccines against COVID-19, each subsequent wave of the pandemic is different than those that preceded it. As the number of people vaccinated against COVID-19 rises, the number of patients presenting to the emergency department with breakthrough infections is on the rise as well.

The gold-standard for the detection of SARS-CoV-2 infection is the RT-PCR. However, RT-PCR testing takes time (at least an hour, but often more), and resources are limited, especially for PCR tests that deliver results quickly, while the patient is still in the emergency department. Thus, antigen POC tests are frequently used in conjunction with RT-PCR to provide rapid diagnosis of a SARS-CoV-2 infection. Clinical impressions prompted us to evaluate the sensitivity of these tests in vaccinated patients with breakthrough infections. The sensitivity of various antigen POC tests has previously been evaluated, but their sensitivity in vaccinated patients has not been reported.

Our results show a significant lower sensitivity for antigen POC tests in vaccinated patients (39.4%) when compared with unvaccinated patients (53.3%; 95% CI of difference, 1.1–26.1%). We hypothesize that this is due to altered viral dynamics in vaccinated patients. Previous findings by Chia et al. [6] showed similar peak-Ct-values in vaccinated and unvaccinated patients with SARS-CoV-2 infection, but viral loads decreased faster in vaccinated individuals, with higher Ct-values in RT-PCR testing as a surrogate parameter. In our study, significantly higher Ct-values were found only for tests detecting the N-gene and the ORF1a-region in vaccinated patients. However, this study was not powered for this research question and was not standardized regarding the onset of symptoms or designed to investigate viral dynamics.

It has previously been reported that the sensitivity of antigen POC tests is limited likely due to the tests depending on a high viral load for reliable detection of SARS-CoV-2 infected patients [3,4], and our results are consistent with these findings. In our study, Ct-values all were higher in asymptomatic patients, vaccinated patients were significantly less frequently symptomatic and the antigen POC tests performed very poorly in asymptomatic patients, detecting only 16.4% infected patients. On the other hand, even in symptomatic patients, the sensitivity of the test is only minimally increased by 11.8% from an overall sensitivity of 45.7% in all patients to 57.5%. This finding occurred in spite of symptomatic patients having consistently lower Ct-values in RT-PCR testing.

In specimens with a high concentration of viral RNA, fewer RT-PCR copy cycles are needed to produce a detectable amount of the virus. The Ct value of an RT-PCR reaction is the number of cycles at which the PCR product is detectable over and above the background signal. Therefore, a lower Ct-value implies a higher viral load in the specimen and a higher infectivity in the patient of whom the specimen was collected from. While this interpretation of Ct-values is generally accepted, it is unclear if patients with high Ct-values above 30 are still able to transmit the virus [10]. The manufacturer of the antigen POC test used in this trial states a sensitivity of 97.6% in viral loads between Ct-values of 20 and 30. Based on the findings of our study (Fig. 1) and in accordance with previous findings [4], most false-negative antigen POC test results corresponded to Ct-values between 25 and 30.

These findings have significant implications with regard to patient management and healthcare staff management. Many hospital systems have been using routine antigen POC testing to determine safety for unvaccinated staff to come to work, as well as appropriateness for staff to return to work after high-risk exposure. Further, these tests are often being utilized as a first-line measure for patient cohorting in the hospital. Given our findings and those of prior researchers demonstrating variable and sometimes inadequate sensitivity among commercially available antigen POC tests, we argue that these tests must be used with caution. A negative antigen POC test provides patients and staff with a false sense of security, as well as potentially leading to inappropriate patient cohorting which may in fact lead to harm for other patients. The routine use of these tests in public healthcare to enable ‘safe’ visits to public places like theaters, restaurants, or nursing homes should also be questioned, as end-users usually have no understanding of test theory and negative tests provide them with a false sense of security.

These findings also highlight the need for urgent evaluation of the many different commercially available antigen POC tests specifically in vaccinated and asymptomatic patients.

Limitations

The study retrospectively evaluates the sensitivity of one antigen POC test in patients presenting in one hospital. Therefore, the findings of this study might be of limited value for other antigen POC tests as well as for other populations.

There might be differences related to the point in time at which vaccinated and unvaccinated SARS-CoV-2 positive patients visit the emergency department. Since the study was not standardized regarding the onset of symptoms or designed to study the viral dynamics of SARS-CoV-2 infections, it may be possible that our findings in part reflect these circumstances. However, this does not change the clinical implications of our findings.

Preanalytical factors must also be considered, as antigen POC tests and RT-PCR were performed from two oropharyngeal swabs, whereas nasopharyngeal swabs were considered to be the gold-standard of detection of a SARS-CoV-2 infection. However, a recent literature review and meta-analysis found a very similar performance for oropharyngeal and nasopharyngeal swabs [11], and the swabs were taken from the same site, on the same day, by trained staff and in most cases by the same person at the same time.

The pandemic is dynamic, with new virus variants impacting clinical presentation, infectivity, the efficacy of vaccines, and the accuracy of assay methods. It is therefore unclear to what extent these findings will be relevant to future virus variants as the predominant virus variant at the time the study was conducted has been the Delta Variant (B.1.617.2). However, we believe that our findings demonstrate that negative antigen POC test results must be interpreted with caution.

Conclusion

In summary, antigen POC tests are not useful for rule-out and do not consistently identify patients that can be deisolated, especially in vaccinated and asymptomatic patients.

Acknowledgements

M.B. and B.K. conceived the study, designed the trial, and obtained ethical approval. M.B. and P.S. collected the data. M.K. provided statistical advice and analyzed the data. M.B. and A.D.S. drafted the manuscript and all authors contributed substantially to its revision. M.B. takes responsibility for the paper as a whole.

Conflicts of interest

There are no conflicts of interest.