Evaluation of Panbio rapid antigen test for SARS-CoV-2 in symptomatic patients and their contacts: a multicenter study.

BACKGROUND: Point-of-care rapid tests to identify SARS-CoV-2 can have clinical benefits.
METHODS: A cross-sectional study in adults visiting emergency services or screening sites of referral hospitals for COVID-19 to validate the diagnostic performance of a rapid antigen test for SARS-CoV-2 (Abbott's Panbio) compared with reverse transcription-polymerase chain reaction (RT-PCR) testing. Tests were performed by health personnel in a routine situation during a COVID-19 outbreak.
RESULTS: A total of 1060 participants (mean age 47, 46% with a self-reported comorbidity) were recruited from 8 hospitals in Mexico. Participants provided 1060 valid Panbio rapid test-RT-PCR test pairs with 45% testing positive in the RT-PCR. Overall sensitivity of the Panbio test was 54.2% (95% CI 51%-57%), and 69.1% (95% CI 66%-73%) for patients during the first week of symptoms. Sensitivity depended on viral load (cycle threshold (Ct) of RT-PCR) and days of symptoms. With a Ct ≤25, sensitivity was 82% (95% CI, 76%-87%). Specificity of the Panbio test was >97.8% in all groups.
CONCLUSIONS: The Panbio rapid antigen test for SARS-CoV-2 had good specificity but low sensitivity. A negative test requires confirmation with RT-PCR, especially for testing after the first week of symptoms.

Introduction

Rapid tests to identify infectious agents are beneficial since they allow decisions to be made at the point-of-care (POC) for treatment selection or separation of cohorts to avoid cross-infection. Rapid tests are especially useful in emergency situations, such as those experienced in the COVID-19 pandemic. For influenza and other respiratory viruses, rapid tests are readily available and have shown clinical benefits (Benirschke et al., 2019; Rahamat‐Langendoen et al., 2019; Shengchen et al., 2019; Wabe et al., 2019), although not in all evaluations (Schechter-Perkins et al., 2019). Rapid tests at POC can be employed to screen asymptomatic people, diagnose people with symptoms suggestive of disease, or for contact tracing and epidemiological purposes in persons with contact with suspected or confirmed cases. These situations, in which the pre-test probabilities of SARS-CoV-2 infection are very different, have different demands (Watson et al., 2020). A recent Cochrane review showed the urgent need for prospective and comparative evaluation of rapid COVID-19 tests (Dinnes et al., 2020). Having a reliable rapid test is desirable, especially in places with poor infrastructure or without access to standard laboratory tests and at reference sites, especially those with potential for patients with similar clinical manifestations but with infection with a different virus (Kubina and Dziedzic, 2020). Rapid tests could also be performed on the same subject on several occasions and at a low cost to detect and isolate positive cases and for epidemiological surveillance, even at a lower sensitivity than reverse transcription-polymerase chain reaction (RT-PCR) tests (Mina et al., 2020).

Several international regulatory bodies have authorized the emergency use of rapid tests for the presumptive diagnosis of SARS-CoV-2 infection based on the identification of antigens. While the overall recommendation is to confirm results with tests considered the gold standard, such as PCR, a readily available result obtained with a rapid antigen test can aid in making several important decisions in the clinical care workflow.

Here, we assessed the performance of the Panbio rapid antigen test for SARS-CoV-2 (Abbott Diagnostics Korea, Inc. Ref. 41FK10) administered by health personnel as a diagnostic tool in symptomatic patients who arrived at the emergency rooms (ER) and outpatient clinics of referral hospitals for COVID-19 and in symptomatic or asymptomatic contacts of patients diagnosed with SARS-CoV-2.

Methods

We conducted an observational, cross-sectional study across 8 tertiary care referral hospitals for COVID-19 in Mexico, comparing the performance of the Panbio rapid antigen test with the gold standard RT-PCR test. The participating institutions were part of the Mexican National Institutes of Health (NIH) network, including the National Institute of Respiratory Diseases (INER), National Institute of Cancer (INCAN), National Institute of Cardiology (INCIC), High Specialty Regional Hospital Ciudad Salud in Tapachula Chiapas, High Specialty Regional Hospital of Mérida Yucatán (HRAE Mérida), High Specialty Regional Hospital in Ixtapaluca State of Mexico (HRAE Ixtapaluca), National Institute of Medical Sciences and Nutrition (INCMNSZ), and the National Institute of Perinatology (INPER). The protocol was revised and approved by a single Institutional Review Board designated by the NIH authority for this study.

Inclusion criteria

Patients aged ≥18 presenting with respiratory symptoms consistent with COVID-19 or influenza syndrome at the emergency rooms or screening sites of the participating hospitals and who provided written informed consent to participate were included in the study regardless of hospitalization status. Contacts of confirmed COVID-19 cases presenting to the same sites for evaluation (mostly with respiratory symptoms but some asymptomatic) were also enrolled, all providing a written informed consent to participate.. Participants were recruited on weekdays, during the morning-afternoon shift, from the 8 participating institutions.

Gold standard

For this study, the “Berlin” SARS-CoV-2 RT-PCR methodology recommended by the World Health Organization (WHO) (Corman et al., 2020) was considered the gold standard. All participating institutions implemented this test in situ and were accredited by the corresponding national authority (National Epidemiological Reference Institute) based on the detection of 4 SARS-CoV-2 markers: the N, E, ORF and RdRp genes. The RT-PCR test for SARS-CoV-2 was performed according to the Berlin protocol on nasopharyngeal swab samples taken with a synthetic fiber swab and flexible shaft and sent to the laboratory in a transport tube. In all cases, the cycle threshold (Ct) for the different gene targets was requested. Laboratory personnel were blinded to results for the rapid tests.

Procedures in the emergency rooms, outpatient clinics and with contacts

After participants provided their written informed consent, the Panbio rapid test and RT-PCR were performed in the ER, triage, or at the usual reception site for probable patients with COVID-19. Samples were obtained within minutes of each other with different swabs, following instructions for Panbio and RT-PCR. The Panbio test swab with the sample was inserted into the extraction tube with 300 uL of fluid buffer and processed immediately at the same place where it was taken. For RT-PCR, the specimens were inserted inside the universal viral transport tube with 2 mL of media (Becton, Dickinson, NJ, USA) and sent to the laboratory.

Rapid SARS-CoV-2 antigen test

The Panbio COVID-19 Ag Rapid Test Device (nasopharyngeal) (Abbott Diagnostics Korea, Inc. Ref. 41FK10) was evaluated. This test does not require additional equipment and is approved by the corresponding regulatory agencies in the United States, Europe and Mexico. The manufacturer reports a sensitivity of 93.3% (95% CI of 83%–98%) and a specificity of 99.4% (95% CI, 95%–99.3%) with a detection limit of 1.5 × 101.8 median tissue culture infectious dose (TCID50)/mL (Abbot 2020).

Healthcare personnel at the ER or the triage area carried out the tests following the manufacturer's instructions. A nasopharyngeal swab with a flexible shaft included with the kit was utilized to obtain the specimen with a gentle rub and roll movement. The swab was then inserted into the extraction tube and processed immediately where taken.

All participating healthcare personnel were trained in obtaining nasopharyngeal swabs and in the use of the rapid test. In all cases, a stopwatch was utilized to record the exact time of the reading. The result and a photograph of the cassette were incorporated into the database for control and later verification if necessary. Clinical information was retrieved using a REDCap database (REDCap 10.31-2021), including the WHO COVID-19 severity classification (World Health Organization 2020), the use of respiratory support (oxygen by nasal prongs or high flow), mechanical ventilation, and the presence and duration of respiratory symptoms. For hospitalized patients, the results of routine laboratory tests and clinical data were subsequently collected, including a complete blood count, serum electrolytes, glucose, urea, creatinine, electrocardiogram, liver function tests, urinalysis, and chest computed tomography.

A sample number of 600 subjects, with at least 300 symptomatic patients, and 300 contacts were considered in the protocol (Hajian-Tilaki 2014, Bujang 2016), allowing for a study power of 0.8 even given an assumed prevalence of COVID-19 positivity of 20%, and rapid test sensitivity and specificity of 80%.

Blinding: The laboratory processing the RT-PCR test and the administrator of the Panbio rapid antigen test did not know the result of the other test.

Statistical analysis

The initial comparison was performed in a 2 × 2 table between the positive and negative tests by RT-PCR and the positive and negative tests by the Panbio rapid test, with sensitivity, specificity, and positive and negative predictive value. We also evaluated concordance (kappa statistic) between the RT-PCR and the Panbio test.

Patients, contacts and participating hospitals were considered as subgroups. The impact of duration of symptoms on the positivity of the rapid test was assessed, with positivity as a dependent variable as a function of the duration of symptoms, taking into account the Ct of the RT-PCR test.

Age, sex, days from the onset of symptoms (if they occurred), symptoms, disease severity (based primarily on the type of support needed), time of arrival at the ER, time of the test, time of the result, day and the time of obtaining the RT-PCR result were taken as covariates on order to observe modifying effects on sensitivity and specificity.

Analysis was performed with STATA v13.0 statistical software, with summary statistics for diagnostic tests performed by DISGT and DIAGTEST procedures. The MIDAS procedure was utilized to analyze diagnostic performance across recruiting sites.

Results

We obtained 1069 rapid test results (from 1069 patients) and 1060 rapid and RT-PCR test pairs from 1060 patients (597 women and 463 men; mean age 46.6±16 years) were further analyzed (see flow diagram). We recruited 378 study participants at INER, 246 at INCAN, 32 at INCIC, 41 at Ciudad Salud Tapachula Chiapas, 40 at HRAE Mérida, 80 at HRAE Ixtapaluca, 150 at INCMNSZ, and 93 at INPER. Ct values were reported for 386 of 472 positive RT-PCR tests (81.8%). Ct values were not available for some participating hospitals (mainly outside of Mexico City) that sent RT-PCR samples to a state laboratory for processing. In the analysis, we utilized the lowest Ct of those reported for different genes.

Table 1 describes the clinical characteristics of the study participants.

Table 1

Principal characteristics of participants contributing to Panbio rapid antigen test-reverse transcription-polymerase chain reaction (RT-PCR) test pairs (Means and SD or percentage and N)

	Total (1060)	Female (597)	Male (463)
Age	47 (16)	47 (16)	46 (17)
Days of symptoms	6 (8)	6 (10)	6 (5)
Asymptomatic subjects	14% (145)	14% (84)	13% (61)
Fever	33% (350)	28% (169)	39% (181)
Dry cough	37% (391)	38% (225)	36% (166)
Cough and phlegm	11% (114)	10% (58)	12% (56)
Phlegm	9% (98)	10% (58)	9% (40)
Dyspnea	24% (261)	22% (132)	28% (129)
Any comorbidity	46% (485)	46% (272)	46% (213)
Diabetes	14% (148)	14% (81)	14% (67)
Hypertension	18% (195)	19% (115)	17% (80)
Obesity	10% (102)	10% (58)	10% (44)
Any cancer	10% (101)	10% (62)	8% (39)
Previous smoking	27% (284)	20% (121)	35% (163)
Current tobacco smoker	10% (107)	7% (43)	14% (64)
Former smoker	18% (178)	14% (79)	25% (99)
Cigarettes per day in smokers	4(4)	3(3)	5(5)
Influenza vaccine 2020-2021	49% (515)	51% (304)	46% (211)
COVID vaccine	5% (49)	5% (28)	5% (21)
SpO2%	90(10)	91(10)	89(10)
Breathing frequency	21(6)	21(5)	22 (7)
Oxygen use in the moment of the test	18% (190)	13% (79)	24% (111)
Site of recruitment
Emergency room visit	57% (601)	59% (350)	54% (251)
Outpatient clinic	35% (366)	34% (202)	35% (164)
Hospitalized	9% (93)	8% (45)	10% (48)
Follow-up
Remained ambulatory	79% (847)	84% (449)	75% (348)
Re-admission to hospital	1% (15)	2% (9)	1% (6)
Positive RT-PCR test	45% (472)	41% (247)	49% (225)

Percentage (%) or mean (standard deviation) are shown.

From 1060 valid test pairs, 1 per participant, 915 were obtained from participants with any respiratory symptom (86.3%) and 145 (13.7%) from asymptomatic participants (usually contacts of a positive relative or coworker). Among participants reporting any symptom, 72% had symptoms lasting for <1 week.

Approximately half (57%) of participants requested attention in the ER or triage system, and 35% at an outpatient service. After initial screening, 79% of all participants were managed as outpatients. At the time of recruitment, only 49 participants (5%) had received 1 dose of the SARS-CoV-2 vaccine (none had complete vaccination) and 46% of participants presented at least 1 comorbidity, as follows: 18% had hypertension, 14% diabetes, 10% obese, and 10% current smokers.

From all the test pairs analyzed, 44.5% were positive in the RT-PCR test and 25% in the Panbio rapid test. Positivity of the rapid test was strongly associated with Ct (a surrogate for viral load) (Figure 1 ), (Table S1) with a sensitivity of 82% (95% CI, 76%–87%) in the presence of a Ct ≤25 (Figure 2 ). Positivity also depended on the days since the onset of symptoms (Figure 3 ) (Table S2), with an initial sensitivity of 69.1% (95% CI 63%–74%) in the rapid test during the first week of symptoms thereafter exhibiting a progressive decline.

Figure 1

Dot graph of cycle threshold (Ct) (red circles) with median value and 25% and 75% percentiles (blue lines) in individuals with negative (left side) or positive tests (right side). Those with positive rapid tests have lower Ct values.

Figure 2

Mean sensitivity of the Panbio rapid test as a function of cycle thresholds (Ct) of the reverse transcription-polymerase chain reaction and 95% CI error bars. Sensitivity of the rapid test decreases with Ct values, indicating lower viral loads.

Figure 3

Mean sensitivity of the Panbio rapid test as a function of the duration of respiratory symptoms in participants positive for the reverse transcription-polymerase chain reaction, and 95% CI error bars. Sensitivity drops significantly with duration of symptoms.

Table 2 shows the diagnostic performance of the rapid test for the whole group, stratified by time since the onset of symptoms. Overall sensitivity of the Panbio test was 54.2% (95% CI, 51%–57%) with a positive likelihood ratio of 35.7, a negative likelihood ratio of 0.46, and a receiver operating characteristic curve area of 0.77. For participants presenting during the first week of symptoms, sensitivity was 69.1% (95% CI, 66%–73%), decreasing considerably with a longer duration of symptoms. The presence of symptoms was a predictor of positivity for the rapid test; sensitivity of 58.1% in participants with symptoms, compared with 26.3% in persons with no symptoms but with positive RT-PCR. On the other hand, specificity of the Panbio rapid test was >97.8% in all groups.

Table 2

Diagnostic performance of the Panbio rapid antigen test

Group	Test pairs	Prevalence of COVID-19 positivity (%) (RT-PCR)	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)
All	1060	44.6	54.2 (51.2–57.2)	98.5 (97.7–99.2)	96.6 (95.5–97.7)	72.8 (70.2–75.5)
First week of symptoms	710	38.7	69.1 (65.7–72.5)	98.4 (97.5–99.3)	96.5 (95.1–97.8)	83.4 (80.7–86.2)
2nd week of symptoms	147	68.7	43.6 (35.3–51.5)	97.8 (95.4–100)	97.8 (95.4–100)	44.1 (36.1–52.1)
More than 1 week of symptoms	203	68.5	36.7 (30.1–43.3)	98.4 (96.7–100)	98.1 (96.2–100)	41.7 (34.9–48.5)
No symptoms	146	39.0	26.3 (19.1–33.4)	98.9 (97.2–100)	93.8 (89.8–97.7)	67.7 (60.1–75.3)
Symptoms*	908	45.5	58.1 (54.9–61.3)	98.4 (97.6–99.2)	96.8 (95.6–97.9)	73.8 (70.9–76.7)

PPV = positive predictive value, NPV = negative predictive value, RT-PCR = reverse transcription-polymerase chain reaction. Test pairs include a Panbio rapid antigen test and RT-PCR with minutes of each other, obtained from 1060 patients, one per patient.

Concordance (kappa statistic) between the Panbio rapid test and RT-PCR (Table S3) during the first week of symptoms was 0.72 (SE 0.04), but across all participants, it was 0.53 (SE 0.03). For participants presenting during the second week of symptoms, sensitivity was 0.31 (SE 0.06) and 0.13 (SE 0.07) for participants with >2 weeks of symptoms.

In a logistic regression model, a positive result on the RT-PCR was predicted by a positive result on the Panbio rapid test (odds ratio (OR) 98; 95% CI, 47–203), by days of symptoms (OR 1.08; 95% CI, 1.05–1.11), male sex (OR 1.5; 95% CI, 1.03–2.08), and age (OR 1.01; 95% CI, 1.00–1.03), and these same variables were associated with the Ct (Pseudo R2 0.38).

In multivariate logistic regression, modeling positivity for the Panbio rapid test, the most important predictor was the RT-PCR test and especially the Ct (adjusted odds ratio (aOR) 0.79; 95% CI, 0.75–0.84), but also the days of symptoms (aOR 0.88; 95% CI, 0.83–0.94) adjusting for age, sex and comorbidities (Pseudo R2 0.34).

The diagnostic characteristics of the Panbio rapid test varied across sites, with substantial heterogeneity in sensitivity (I2 80.9; 95% CI, 68–93) and specificity (I2 73.5; 95% CI, 54–92), with a relevant influence of centers with fewer study participants (Figure 4 ).

Figure 4

Forest plot depicting combined sensitivity and specificity as a function of participant hospitals. Combined sensitivity was 0.53 (range 0.42–0.63) with I2 of 80.2 (range 67–93), whereas specificity was consistent with an overall value of 0.99 (0.93–1.0) and I2 of 73 (range 55–92)

Discussion

We present the results of a validation experiment of a commercial POC rapid antigen rapid test for SARS-CoV-2 infection (Panbio), revealing high variability in sensitivity with respect to time from symptoms onset and viral load. In our study, during the first week of symptoms, the sensitivity of the Panbio rapid antigen test was 69.1%, although the overall sensitivity was 54.2%, much lower than that reported by the manufacturer (Abbott, 2020). The test was performed in patients with a longer duration of symptoms or with no respiratory symptoms (predominantly contacts of symptomatic cases); such patients were common in ERs or at screening sites during outbreak peaks when access to hospitals was difficult. On the other hand, the specificity was consistently high in all subgroups.

In studies conducted elsewhere with the Panbio test, lower sensitivities have also been found, ranging from 45% in pediatric patients with <5 days of symptoms compatible with COVID-19 (Villaverde et al., 2021), 48% in household contacts of positive cases, 37% in non-domestic contacts (Torres et al., 2021), and up to 73% in primary-care patients (Linares et al., 2020). In a meta-analysis, days of symptoms were strongly associated with the Panbio rapid test result, with sensitivity of 86.5% if symptoms were present for <7 days and 54% if symptoms were present for >7 days (Linares et al., 2020). In a study conducted under normal working conditions, the sensitivity of the Panbio test was 73% in the Netherlands, 81% in Aruba, and >95% in Aruba if the Ct was <32 (Gremmels et al., 2021). In another study in symptomatic patients in primary care, sensitivity was 80% and specificity 100% (Albert et al., 2021). In all studies, the specificity has been 100% or close to 100%.

The Canadian authorities issued recommendations that took into account sensitivities incorporated into the annex to the Panbio test (Interim guidance on the use of the Abbott PanbioTM COVID-19 Antigen Rapid Test, 2021) before independent assessments of the test... and sensitivities in the abbott inserts were much higher than those found later.

An update of a Cochrane review (Dinnes et al., 2021), including 78 studies (20 pre-prints) and 24 087 samples (approximately one-third positive for SARS-CoV-2), confirmed a substantial variation in sensitivity according to the presence or absence of symptoms (72% vs 58%), first versus second week of symptoms onset (78% vs 51%), and a Ct of ≤25 versus >25 (94.5% vs 40.7%) (Dinnes et al., 2021)() available rapid antigen tests (several) but with a section on Panbio test. Sensitivities reported for rapid antigen tests for SARS-CoV-2 from different manufacturers ranged from 34% to 88%, but specificities were in general >99% (Dinnes et al., 2021)(). The pooled sensitivity of the Panbio test (11 studies) was 75.1% in symptomatic patients during the first week of symptoms but dropped to 58% in asymptomatic individuals (Dinnes et al., 2021)(). Specificity was 99.5%. In the first week of symptoms, the sensitivity in our study was 69.1%, and specificity was 98.5%. It is relevant that only 1 study included in the Cochrane analysis (Alemany et al., 2021) had a test sensitivity with a lower limit of confidence >80%, the recommended cut-off point by the WHO for a rapid antigen test.

We observed a strong association between the Panbio rapid test result and viral load (estimated with the Ct) with no positive results in samples with a Ct of >39. It is also relevant that in our study, the Panbio test was performed in patients with <5 days of symptoms but also with >5 days from the onset of symptoms, a group in which positivity decreases significantly due to an expected lower viral load. Nevertheless, even in this group in which sensitivity is considerably lower, a positive test would be highly informative.

It is noteworthy that with symptoms of longer duration, the cost-effectiveness of applying POC rapid antigen tests would drop progressively. Thus, given that specificity is high, the greatest clinical advantage of utilizing a rapid antigen test would present when the result is positive, in which case, a confirmatory RT-PCR would not be needed.

In the presence of symptoms compatible with COVID-19, or a period of high incidence of infection in the community, a negative RT-PCR test result would be unreliable, even during the first week of symptoms. In these cases, the RT-PCR test is usually repeated a second or even third time (Ramdas et al., 2020). As the infection progresses, the viral and nucleic acid load tends to decrease, and the RT-PCR test tends to be negative, while antibody titers against SARS-CoV-2 begin to appear (Ghaffari et al., 2020; Kilic et al., 2020; Laureano and Riboldi 2020; Shyu et al., 2020). Thus, confirming a SARS-CoV-2 infection ideally involves both the positivity of an RT-PCR test (not necessarily the first test) and the consideration of a combination of epidemiological variables, including the rate of community transmission, the presence of compatible symptoms and the presence of antibodies.

False-positive RT-PCR test results have rarely been reported and are attributed to contamination with viral genetic material in any of the steps between sampling and processing. On the other hand, in the case of a negative antigen test, confirmation with RT-PCR will be required, especially in the presence of compatible symptoms, a high rate of community infections, or in the case of persons with direct contact with confirmed COVID-19 patients. In any case, according to our observations, if a Panbio rapid antigen test were employed, 69.1% of the RT-PCR tests would be avoided during the first week of symptoms, which represents a considerable saving. The latter represents an enormous advantage in settings where decision-making is needed and where the lack of infrastructure and high costs render it difficult to implement molecular testing.

Our study was observational, subject to all possible biases of cross-sectional studies; it was performed predominately during the daytime, the shift with more available staff and in-training personnel. Testing was carried out with the Mexican health system and personnel under stress, a very different situation from that of rapid tests performed under controlled circumstances in a laboratory by the same expert personnel, which could explain, at least in part, the reduced sensitivity found in our study. On the other hand, our results are expected to be closer to what can be observed during outbreaks that saturate or overwhelm screening sites and ERs, that is, more demanding circumstances than those found under strictly controlled laboratory testing. Furthermore, overall RT-PCR positivity in participating hospitals was relatively high (44.6%, ranging from 27% to 93% in different hospitals), allowing for a proper evaluation of sensitivity and specificity. Actual conditions of use may be even more demanding than those present in our study, for example, if testing includes primary care, community hospitals, during all shifts including weekends and nights, characterized by the presence of fewer personnel, and especially if the overall positivity of the tests or community transmission is low.

Conclusions

We have shown that the sensitivity of the Panbio SARS-CoV-2 rapid antigen test is limited, with an overall estimated value of 69.1% in the present study. As expected, sensitivity was highly associated with time from the onset of symptoms and with viral load. The low sensitivity and high specificity observed in the Panbio test make it necessary to confirm all negative results, especially in the presence of symptoms compatible with COVID-19 and in settings with a high community infection rate. Nevertheless, the Panbio rapid antigen test could be beneficial to screen for positive tests and to reduce the number of cases to be confirmed, enabling rapid clinical decision-making.

Figure 5

Figure 5

STUDY FLOW.RT-PCR=real time polymerase chain reaction. Description of the participants, a total of 1060, each providing one pair of PanbioTM-RT-PCR tests, and their results.