A method comparison study of the high throughput automated HISCL® SARS-CoV-2 antigen assay using nasopharyngeal swab samples from symptomatic and asymptomatic subjects against conventional RT-PCR.

Our study aim was to evaluate the performance of the automated Sysmex HISCL® severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) antigen assay against reverse-transcription polymerase chain reaction (RT-PCR). We tested 277 remnant frozen nasopharyngeal swab samples, stored in universal transport medium (UTM), yielding a sensitivity of 94.9% against historical RT-PCR results with cycle threshold (Ct ) < 30, and a sensitivity of 76.7% for Ct  < 35, and specificity of 100% (all Ct values) confirming compatibility of UTM-diluted samples with the assay system. Thereafter, we prospectively collected 141 nasopharyngeal swab samples in UTM from healthcare workers and 1369 paired swabs (400 UTM; 969 dry) from individuals at a public health testing center, with the first swab (UTM) reserved for RT-PCR, yielding a positivity rate of 4.6%. HISCL assay performance using UTM swabs was superior to dry swabs, with a sensitivity of 100% (95% confidence interval [CI] 71.5%-100%) at Ct  < 30 versus 92.3% (95%CI 81.5%-97.9%), and a specificity of 99.3% (95% CI 98.1-99.89) against 83.3% (95%CI 80.7%-85.6%). We conclude that this antigen assay is suitable for high throughput facilities where the primary indication for testing is to rule out infection with low RT-PCR Ct values (proxy for high viral loads) to curb viral spread.

INTRODUCTION

The coronavirus disease 2019 (COVID‐19) pandemic has provided many lessons since its emergence in China in December 2019, notably that diagnostic testing is a critical cornerstone in safeguarding the future of global health. An early breakthrough was heralded by the public sharing of the severe acute respiratory syndrome coronavirus 2 (SARS‐CoV‐2) virus genome sequence facilitating the rapid development of reverse transcription‐polymerase chain reaction (RT‐PCR) testing protocols for its detection, in nasopharyngeal swab samples, through global scientific community collaboration. These early protocols laid the foundation for case confirmation, but the rapid spread and geographic scale of infection soon overwhelmed the diagnostic capacity of laboratories equipped to conduct such relatively complex in‐house molecular tests. This provided the impetus for the unprecedented speed of development of simplified commercial RT‐PCR‐based COVID‐19 testing kits. Access was facilitated by fast‐tracked emergency use listing by the World Health Organization (WHO) or emergency use authorization by the United States Food and Drug Administration. Numerous SARS‐CoV‐2 RT‐PCR kits have since become available from multiple manufacturers.

Accurate and rapid widescale COVID‐19 testing is essential to contain this continuously evolving pandemic. Viral nucleic acid detection by RT‐PCR, with its excellent sensitivity and specificity, makes it the most reliable SARS‐CoV‐2 detection method. However, a major limitation is the sophisticated laboratory infrastructure requirement and relatively long result turnaround time. Pre‐analytical nucleic acid extraction, sometimes manual, is required, and most commercial kits are optimized for batch testing, taking 3–4 h on average. As the pandemic evolved, indications for testing expanded from diagnosis for symptomatic individuals, to screening of asymptomatic individuals as part of public health measures to re‐open economies, on the premise that asymptomatic and presymptomatic individuals can transmit the virus. Consequently, it became evident that RT‐PCR testing accessibility was insufficient, paving the way for the next wave of diagnostic tool development, namely rapid antigen tests (RAT).

Antigen tests detect viral proteins directly, in contrast to RT‐PCR‐based viral nucleic acid detection requiring thermal amplification. Antigen tests are thus inherently less sensitive than RT‐PCR for SARS‐CoV‐2 detection, but this is offset by superior accessibility. RAT are mostly lateral flow assays in single test formulation, with or without a strip reader, providing results within approximately 15 min, simple to use, and thus suitable for use outside of laboratories.

Both RT‐PCR and antigen tests detect the presence of viral particles, but neither can confirm active infection with a live virus capable of infecting others. Only viral culture, beyond the realm of routine diagnostics, can achieve that. RT‐PCR cycle threshold (C t) values, which provide a viral load approximation, have been used as a surrogate to estimate the probability of infectivity, the higher the C t value the lower the viral load and the lower the likelihood of infectivity. A C t value of approximately 30 is generally accepted as the threshold beyond which transmission is unlikely, , an important consideration where the primary indication of screening tests is to identify asymptomatic infected individuals capable of onward transmissibility.

Head‐to‐head comparisons with established lab‐based RT‐PCR methods, the de facto gold standard, show that RAT sensitivity varies substantially amongst manufacturers. Whilst some have shown excellent performance , many fail to detect RT‐PCR positive samples with C t > 25. , , , Although RAT has revolutionized access, their suboptimal sensitivity limits their effectiveness in curbing viral spread.

In contrast, automated immunoassay antigen tests provide high throughput and have been reported to have better sensitivity than RAT for C t > 25. Although automated antigen tests, like RT‐PCR are laboratory‐based tests, they do not require highly skilled operators, can be conducted within routine clinical laboratories, and have a significantly shorter time to result availability. As such automated antigen testing, although not yet widely established, has the potential to fill an important gap in the COVID‐19 diagnostic landscape.

Here, we present a performance evaluation of the HISCL® SARS‐CoV‐2 antigen assay on the HISCL®‐5000 analyser (Sysmex Corporation), compared with routine RT‐PCR testing on nasopharyngeal swab samples obtained from symptomatic healthcare workers tested as part of COVID‐19 workplace requirements and individuals presenting to a public health testing center.

MATERIALS AND METHODS

Study design and objectives

This study was conducted by the Department of Medical Microbiology at the Radboud University Medical Center (RadboudUMC) in Nijmegen, the Netherlands.

Part 1: Retrospective testing of frozen bio‐banked nasopharyngeal swab samples collected in viral transport medium (VTM)

Whereas the HISCL antigen assay was designed for dry swab testing, nasopharyngeal swabs for SARS‐CoV‐2 RT‐PCR are conventionally collected in VTM. We thus conducted a pilot study to assess the suitability of VTM samples for HISCL antigen testing. Frozen residual samples from historical standard nasopharyngeal swab collections, obtained from patients with suspected SARS‐CoV‐2 infection and stored in 3 ml VTM (GLY or Universal Transport Medium [UTM]) after RT‐PCR testing, were selected for antigen testing.

Part 2: Prospective testing of freshly collected nasopharyngeal swab samples

Part 2a: Comparative COVID‐19 testing in a hospital setting on a single swab collected in UTM

Our aim was to evaluate the HISCL antigen test in line with standard COVID‐19 sample collection and testing procedures. Single nasopharyngeal swabs prospectively collected from symptomatic healthcare workers and patients, placed in 1.5 ml UTM (Copan), underwent antigen testing, and compared with routine RT‐PCR conducted in parallel.

Part 2b: Comparative COVID‐19 testing at a public health testing center (PHTC) of paired nasopharyngeal swabs (dry vs. UTM)

Here, the original aim was to compare dry swab (study‐specific sample) and UTM swab (routine sample) antigen testing, with RT‐PCR (UTM). Two nasopharyngeal swabs were collected from consenting adults presenting for testing (irrespective of symptoms). The first swab was placed in 1.5 ml UTM, the second swab was kept dry and placed in the HISCL specific buffer, exclusively for antigen testing, upon receipt in the laboratory. Logistically it proved impossible to test the UTM sample with the antigen assay as the routine RT‐PCR testing was conducted at a location remote to RadboudUMC. This study part was thus limited to a dry swab antigen test comparison with RT‐PCR testing on a separate, but simultaneously collected UTM swab sample.

Part 2c: Comparative COVID‐19 testing at a PHTC on two separate nasopharyngeal swabs collected per subject (UTM vs. UTM)

A study addendum was made to compensate for the low SARS‐CoV‐2 positivity rate encountered during the period of hospital sample collection. Two swabs were collected per subject, both in UTM. As per part 2b, the first swab was referred for routine RT‐PCR, and used as a comparator for antigen testing on the second UTM swab at RadboudUMC. RT‐PCR was repeated on the same UTM sample used for antigen testing, if discrepancies with the RT‐PCR result obtained on the first swab sample tested elsewhere were encountered.

Study sample numbers and selection criteria

Part 1

Archived samples were selected for testing as follows: 50 RT‐PCR negative from symptomatic subjects and 200 RT‐PCR positive aiming at equal distribution across the disease severity spectrum from asymptomatic to critical (ICU admission or died). Any documented SARS‐CoV‐2 variants were specifically included. Samples without a recorded RT‐PCR result, insufficient volume or >1 prior freeze‐thaw cycles were excluded.

Part 2

For the prospective study our target was 100 positive samples for comparative analysis for both settings. Any adult (≥18 years) with suspected COVID‐19, irrespective of symptom severity, or asymptomatic individuals undergoing a screening test were included. At the PHTC an additional inclusion criterion was a willingness to have a second swab taken.

HISCL SARS‐CoV‐2 Ag assay description

Testing with the HISCL® SARS‐CoV‐2 Ag Assay kit (Sysmex Corporation) was conducted on the HISCL®‐5000 (Sysmex Corporation) automated immunoanalyser. The acronym “HISCL” refers to High Sensitivity Chemiluminescence Enzyme Immunoassay (CLEIA). HISCL®‐5000 is a high throughput analyser (200 samples/hour) with results available within 17 min and stat position for immediate testing and a continuous loading option. The test principle is a two‐step sandwich CLEIA targeting the SARS‐CoV‐2 N protein. The luminescence signal intensity is recorded as a ratio relative to the background signal, referred to as the cutoff index (C.O.I.), with <1.0 defined as negative, and ≥1.0 as positive. Although the assay is not intended to be quantitative, in‐house studies have shown an excellent correlation between C.O.I. value and viral copy number (the greater the C.O.I, the greater the viral load, and the lower the RT‐PCR C t value; and vice versa).

The original kit design utilized dry nasopharyngeal swabs as the input sample, with a manual preanalytical extraction step using a proprietary tube prefilled with 500 µl extract solution. In brief, the swab tip is immersed into the extract solution tube, manually squeezed to optimize viral material release, left to stand for 3 min, and then centrifuged for 5 min at 2000 g. Thereafter the provided filter screw cap is attached, the sample inverted and allowed to drop into a sample cup for placement on the analyser. If a sample gives a positive result close to the C.O.I. threshold (1 to 10) repeat testing is recommended.

Routine RT‐PCR testing

The TIB Modular Sarbeco E gene assay (TibMolbiol) was used for routine RT‐PCR testing throughout the study as per laboratory standard operating procedures. Testing was conducted on the Roche Flow Solution for molecular diagnostics platform for all prospectively collected samples, and most of the bio‐banked samples. Due to material shortages, the laboratory temporarily switched to using the Roche Cobas 4800 system, during the period of the bio‐bank archive samples accumulation.

Testing procedures

Part 1. Frozen bio‐banked samples

Samples were selected from the freezer archive as per the selection criteria and thawed on the day of testing. RT‐PCR was repeated on the first 50 thawed specimen samples to ensure that viral particle degradation during storage did not unduly bias test performance. The prestorage RT‐PCR repeat testing Ct values of the previously frozen sample were compared. As Ct values were similar, repeat RT‐PCR was only performed on the remaining samples if the results were discordant.

As the original HISCL antigen assay was designed for dry swab testing, we tested various UTM with extract solution dilutions on a subset of samples to find the optimal ratio for testing without undue loss of sensitivity due to dilution (too much UTM) and invalid results (insufficient extract solution buffer), settling on 300 µl UTM sample in 500 µl HISCL extract solution for the method comparison and all other testings.

HISCL antigen testing was conducted as per manufacturer instructions, with the adaptation described above for UTM swabs. The testing procedure followed is outlined in Figure 1A. Additionally, we conducted repeatability testing using 10 replicate measurements for three sample pools: PCR negative, low viral load C t~27, and high viral load C t~20 and stability testing with daily testing over 6 days using a pool of positive samples (C t 24–29) at room temperature (~20°C) and 4°C.

Figure 1

Schematic outline of the study sample processing workflows. (A) Part 1—frozen bio‐banked nasopharyngeal swab samples; (B) Part 2a—prospectively collected single nasopharyngeal swabs (UTM) in the hospital setting; (C) Part 2b—prospectively collected double nasopharyngeal swabs (UTM and dry) in the public health testing center. Here a subset of samples underwent filtration only. (D) Part 2c—prospectively collected double nasopharyngeal swabs (UTM only) in the public health testing center. RT‐PCR, reverse transcription‐polymerase chain reaction; UTM, universal transport medium

Information on swab type used for collection and whether storage was in UTM or GLY was not available for individual samples.

Part 2: Prospective testing of fresh nasopharyngeal swabs

Nasopharyngeal swabs were collected in accordance with standard local procedures using Copan FLOQSwab Minitip swabs.

The workflow for the prospective testing of the fresh nasopharyngeal swab samples collected within the hospital setting and the PHTC is shown in Figure 1B–D. Dry swabs received at RadboudUMC laboratory were stored at approximately 4°C and placed within HISCL extract solution within 24 h of collection. If these samples, together with those received in UTM were not tested on the day of collection, they were stored at −70°C for ≤72 h before testing. HISCL antigen testing was conducted as per manufacturer instructions, with the adaptation for the UTM swabs previously described. We additionally tested a sub‐set of samples using filtration only, instead of centrifugation and filtration, to assess if preanalytical processing could be shortened without compromising analytical performance.

Study period

The bio‐banked samples were collected throughout 2020, before the emergence of the Delta variant. The sample analysis and prospective sample collection took place between April and September 2021, when the Delta variant dominated.

Statistical methods

Data analysis was done using MedCalc® Statistical Software version 12.0.1.0 (MedCalc Software Ltd.). We reported the case numbers detected with each test, the mean value, and range. Logistic regression analysis quantified the C.O.I. and C t value association and correlation estimated with Pearson correlation coefficient (r). Diagnostic accuracy was quantified using sensitivity, specificity, false‐negative and false‐positive counts, and their 95% confidence intervals (95%CI). We assessed antigen assay performance against C t cutoffs of < 30 and < 35 as these C t values are in general use as cutoffs assessing the probability of culturable virus in the sample and therefore infectiousness.

RESULTS

Part 1—Retrospective analysis of frozen bio‐banked nasopharyngeal swab samples

Method comparison

In total 277 remnant previously frozen nasopharyngeal swab samples underwent antigen testing. Repeat RT‐PCR on 50 positive samples revealed no significant difference (p = 0.6854) in the mean historical C t value (26.76, 95%CI 24.99–28.52) compared with repeat testing of freshly thawed samples (mean 26.62; 95%CI 24.82–28.43). The original RT‐PCR result was thus used for method comparison for all samples. Comparison of antigen assay C.O.I. values with RT‐PCR Ct values showed good correlation (coefficient of determination 0.7639) across the disease severity spectrum as well as for SARS‐CoV‐2 mutants (n = 29) (Figure 2). The overall study sample characteristics and antigen assay performance comparison with RT‐PCR are shown in Table 1.

Figure 2

Comparison of HISCL SARS‐CoV‐2 antigen assay C.O.I values with RT‐PCR C t values for retrospective testing of bio‐banked nasopharyngeal swabs. The open black dots represent samples with wild‐type SARS‐CoV‐2 infections. The red dots represent samples with the B1.1.177 mutation (n = 22/29) and the blue dots collectively represent samples. C.O.I., cutoff index; RT‐PCR, reverse‐transcription polymerase chain reaction; SARS‐CoV‐2, severe acute respiratory syndrome coronavirus 2

Table 1

Characteristics and SARS‐CoV‐2 testing results of nasopharyngeal swab samples frozen in 3 ml viral transport medium (Part 1)

SARS‐CoV‐2 infection status	Symptoms	Sample number N	RT‐PCR	HISCL	Sensitivity %			Specificity %
			C t value Mean [min–max]	C.O.I Mean [min–max)	C t < 30% [95%CI] (p/TP)	C t < 35% [95%CI] (p/TP)	C t < 40% [95%CI] (p/TP)	C t < 40% [95%CI] (n/TN)
Negative		50	Negative	<1	94.9% [90.1%–97.8%] (148/156)a	76.7% [70.4%–82.2%] (161/210)	71.4% [65.0%–77.2%] (162/227)	100% [92.9%–100%] (50/50)
Positive	Mild/asymptomatic	100	28.4 [20.5–38.0]	479 [<1–10807]
	Moderate	52	23.2 [14.6–41.9]	3097 [< 1–30146]
	Severe	49	27.1 [14.1–41.0]	1418 [<1–34257]
	Critical	26b	27.2 [11.6–39.8]	2318 [<1–37812]

Abbreviations: CI, confidence interval; C.O.I., cutoff index; n, HISCL antigen assay negative; p, HISCL antigen assay positive; RT‐PCR, reverse‐transcription polymerase chain reaction; SARS‐CoV‐2, severe acute respiratory syndrome coronavirus 2; TN, true negative (RT‐PCR negative); TP, true positive (RT‐PCR positive).

False negative sample Ct values ranged from 26.89 to 29.9 (n = 7), with one outlier at C t 18.42 (repeat testing of this sample was not possible).

9 of the 26 critical patients died.

Repeatability and stability testing

Repeatability results were as follows: negative sample pool C.O.I. consistently < 1.0, low viral load (C t~27) sample pool C.O.I. mean 21.43 (SD 0.3773, CV 1.7%) and high viral load (C t~20) sample pool C.O.I. mean 2674.9 (SD 78.9, CV 2.9%).

Stability for a low positive sample pool (C t 24–29), with baseline C.O.I. of 68, was good for storage at 4°C up to 6 days and somewhat reduced but still acceptable at approximately 20°C. Using the Figure 2 regression equation, we calculated that the loss in antigen detectability is equivalent to a change in C t value of 0.1 at 24 h at 4°C and approximately 20°C, and 0.2 and 0.4 for 4°C and approximately 20°C respectively after 6 days.

Part 2a and 2c—Prospective analysis of freshly collected nasopharyngeal swab samples in UTM

The study was performed during a low transmission phase with just 141 samples collected for testing within the hospital (Part 2a) and four testing positive by routine RT‐PCR. Because of the low positivity rate (2.8%), a further 400 UTM swab samples were collected at the PHTC (Part 2c), with an RT‐PCR positivity rate of 3.0% (12/400). Comparison of routine RT‐PCR and antigen test results of all swab samples (n = 541) collected in 1.5 ml UTM, using C t < 30 as the reference, revealed an initial sensitivity of 83.3% (9/11) (95%CI 51.6%–97.9%) and specificity of 93.1% (498/525) (95%CI 90.1%–95.4%). After repeat antigen testing for C.O.I. values of 1–10, as per manufacturer instructions (n = 27), and repeat RT‐PCR testing on discordant samples (n = 4) where the initial RT‐PCR testing was conducted on a paired swab sample (Part 2c), the sensitivity was 100% (95%CI 71.5%–100%) and specificity 99.3% (95%CI 98.1%–99.8%). The overall study sample characteristics and antigen assay performance on fresh swabs collected in 1.5 ml UTM are shown in Table 2.

Table 2

Characteristics and SARS‐CoV‐2 testing results of nasopharyngeal swab collected fresh in 1.5 ml universal transport medium (Part 2a and Part 2c)

Sample origin	SARS‐CoV‐2 infection status	Sample number N	RT‐PCR	HISCL	Sensitivity %			Specificity %
			C t value Mean [min–max]	C.O.I Mean [min–max)	C t < 30% [95%CI] (p/TP)	C t < 35% [95%CI] (p/TP)	C t < 40% [95%CI] (p/TP)	C t < 40% [95%CI] (n/TN)
Hospital patients and healthcare workers (Part 2a)	Negative	137	Negative	<1	100%a, b [71.5%–100%] (11/11)	84.6%a, b [54.6%–98.1%] (11/13)	75%a, b [47.6%–92.7%] (11/16)	99.3%a, b [98.1%–99.8%] (522/525)
	Positive	4	37.2 [26.3–41.2]	0.1 [<1–4.5]
Public health testing center subjects (Part 2c)	Negative	388	Negative	0.1 [<1–17.7]
	Positive	12	21.4 [12.6–33.4]	636 [<1–3700]

Abbreviations: CI, confidence interval; C.O.I., cutoff index; n, HISCL antigen assay negative; p, HISCL antigen assay positive; RT‐PCR, reverse‐transcription polymerase chain reaction; SARS‐CoV‐2, severe acute respiratory syndrome coronavirus 2; TN, true negative (RT‐PCR negative); TP, true positive (RT‐PCR positive).

The sensitivity data shown here includes repeat RT‐PCR testing for samples with disagreement between original PCR and HISCL antigen testing.

Repeat testing for low positive values.

Part 2b—Prospective analysis of freshly collected dry nasopharyngeal swab samples at the PHTC

A total of 969 samples were collected with an RT‐PCR positivity rate of 5.67% (55/969). Of these, 748 were processed using filtration only, giving a sensitivity of 92.9% (39/42; 95%CI 80.5%–98.5%) and specificity of 72.8% (508/706, 95%CI 68.8%–75.5%). The remaining 221 samples were processed as per manufacturer instructions, using centrifugation and filtration pre‐analysis and repeat testing of seven samples with low positive results. This sample subset gave a sensitivity of 90.0% (9/10, 95%CI 55.5%–99.5%) and specificity of 98.1% (206/211, 95%CI 95.2%–99.5%). The overall sensitivity and specificity for dry swabs (Part 2b), irrespective of pre‐analytical processing method, and using C t < 30 as a reference, gave a sensitivity of 92.3% (48/52, 95%CI 81.5%–97.9%) and specificity of 83.3% (761/914, 95%CI 80.7%–85.6%). With all C t values considered, the sensitivity was 90.9% (50/55, 95%CI 80%–97%). Repeat RT‐PCR testing of the four false‐negative samples (C t 23.7–28.5) was not possible as the corresponding UTM samples were not retrievable from PHTC RT‐PCR testing facility. The overall study sample characteristics and antigen assay performance on dry swabs are shown in Table 3.

Table 3

Characteristics and SARS‐CoV‐2 testing results of freshly collected dry nasopharyngeal swabs (Part 2b)

Sample origin	SARS‐CoV‐2 infection status	Sample number N	RT‐PCR C t value mean [min–max]	HISCL	Sensitivity %			Specificity %
				C.O.I Mean [min–max)	C t < 30% [95%CI] (p/TP)	C t < 35% [95%CI] (p/TP)	C t < 40% [95%CI] (p/TP)	C t < 40% [95%CI] (n/TN)
Public health testing center subjects	Negative	914	Negative	0.1 [<1–40.1]	92.3%b, a [81.5–97.9] (48/52) 92.9%a, c [80.5%–98.5%] (39/42) 90.0%a, d [55.5%–99.5%] (9/10)	90.9%a, b [80.0%–97.0%] (50/55) 93.2%a, c [81.3%–98.6%] (41/44) 81.8%a, d [48.2%–97.7%] (9/11)	90.9%a, b [80.0%–97.0%] (50/55)	83.3%a, b [80.7%–85.6%] (761/914) 72.8%a, c [68.8%–75.5%] (508/706) 98.1%a, d [95.2%–99.5%] (206/211)
	Positive	55	18.5 [7.13 ‐ 32.4]	7319 [<1–46 442]

Abbreviations: CI, confidence interval; C.O.I., cutoff index; n, HISCL antigen assay negative; p, HISCL antigen assay positive; RT‐PCR, reverse‐transcription polymerase chain reaction; SARS‐CoV‐2, severe acute respiratory syndrome coronavirus 2; TN, true negative (RT‐PCR negative); TP, true positive (RT‐PCR positive).

Repeat testing for low positive values

All dry swabs combined

Dry swabs processed with filtration only

Dry swabs processed with centrifugation and filtration, as per manufacturer instructions.

We confirmed that blood contamination did not influence antigen testing as the 8.2% (82/969) visibly bloody samples after dry swab immersion in extract solution tubes showed no significant difference in performance (p > 0.05).

The correlation between antigen assay and RT‐PCR results for all 1787 samples tested in this study, with an overall 16.7% RT‐PCR positivity rate, is shown in Figure 3.

Figure 3

Comparison of HISCL SARS‐CoV‐2 antigen assay C.O.I. values against RT‐PCR C t values for Part 1, 2a, and 2c and 2b samples combined (n = 1787). The dotted line represents the cutoff between positive (on or above) and negative (below) antigen test results. C.O.I., cutoff index; RT‐PCR, reverse‐transcription polymerase chain reaction; SARS‐CoV‐2, severe acute respiratory syndrome coronavirus 2

Positive predictive value (PPV) and negative predictive value (NPV) of the HISCL SARS‐CoV‐2 antigen assay

Predictive values were calculated for the prospective arm of the study using 5% and 25% disease prevalence rates (Table 4).

Table 4

Comparison of PPV and NPV at different disease prevalence rates for HISCL SARS‐CoV‐2 antigen assay tests performed on freshly collected nasopharyngeal swabs

Sample collection	Positive Test	Low disease prevalence (5%)			High disease prevalence (25%)
		PPV (%) [95%CI]	NPV (%) [95%CI]	Accuracy (%) [95%CI]	PPV (%) [95%CI]	NPV (%) [95%CI]	Accuracy (%)[95%CI]
1.5 ml UTM	CT < 30	90.2 [74.9–96.6]	100	99.5 [95.0–99.9]	98.3 [95.0–99.4]	100	99.6 [98.6–99.94]
	CT < 35	88.6 [71.1–96.1]	99.2 [97.2–99.8]	98.7 [97.3–99.5]	98.0 [94.0–99.4]	95.1 [84.4–98.6]	95.7 [93.7–97.3]
Dry Swaba	CT < 30	66.7 [45.1–83.0]	99.5 [96.7–99.9]	97.3 [94.1–99.0]	92.7 [83.9–96.9]	96.7 [82.0–99.5]	95.7 [92.2–98.0]
	CT < 35	64.5 [42.3–81.9]	99.0 [96.7–99.7]	96.8 [93.6–98.7]	92.0 [82.3–96.6]	94.2 [82.1–98.3]	93.7[89.6–96.5]
All combined	CT < 30	84.1 [71.5–91.7]	99.7 [98.3–99.96]	98.9 [97.8–99.5]	97.1 [94.1–98.6]	98.4 [90.2–99.8]	98.1 [96.8–98.9]
	CT < 35	82.2 [68.4–90.8]	99.1 [97.9–99.6]	98.3 [97.1–99.1]	96.7 [93.2–98.4]	94.7 [87.9–97.8]	95.1 [93.4–96.5]

Abbreviations: CI, confidence interval; NPV, negative predictive value; PPV, positive predictive value; SARS‐CoV‐2, severe acute respiratory syndrome coronavirus 2.

The NPV and PPV for dry swab testing was calculated using the sensitivity and specificity values obtained for samples processed strictly in accordance with manufacturer instructions.

DISCUSSION

RT‐PCR, with its unsurpassed diagnostic accuracy, remains the undisputed gold standard for the detection of viral particles. RAT although highly accessible, easy‐to‐use and rapid, are not scalable, and have highly variable sensitivity, ranging from 58% in asymptomatic individuals to 72% during the first few days of symptoms, and 94.5% for C t ≤ 25, based on a meta‐analysis incorporating 48 studies. For testing to be fully effective, it must be accurate and accessible. A recent study has suggested that antigen assays performed on high throughput immunoanalysers may offer a viable solution for patient screening, or in situations where RT‐PCR testing is not readily available.

In this present study, we evaluated the performance of the automated Sysmex HISCL® CLEIA‐based SARS‐CoV‐2 antigen assay against routine RT‐PCR. This assay was originally designed for use with dry nasopharyngeal swabs which is misaligned with conventional sample collection and COVID‐19 testing workflows. Consequently, we conducted feasibility testing on frozen bio‐banked nasopharyngeal swab samples, originally collected in 3 ml VTM, from symptomatic and asymptomatic individuals, obtaining an overall sensitivity of 71.4% (95%CI 65.0%–77.2%) and 94.9% (95%CI 90.1%–97.8%) for samples with C t < 30. Seven false‐negative samples (C t 26.89–29.9) were attributed to antigen dilution in 3 ml UTM, and one sample (C t 18.42), despite having a B.1.1.177 mutation, to a handling error, rather than loss of assay sensitivity for the mutant antigen as 21/22 such mutants were correctly detected (Figure 2). Specificity was 100% (95%CI 92.9%–100%). These results confirmed assay compatibility for use with VTM stored swab samples.

In the prospective arm of our study, we evaluated dry swabs and swabs collected in 1.5 ml UTM, in anticipation that a lesser dilution factor would enhance sensitivity. Antigen test performance on dry swabs, when performed strictly in accordance with manufacturer instructions, with an RT‐PCR threshold of C t < 30, gave a sensitivity of 90.0% (95%CI 55.5%–99.5%) and specificity of 98.1% (95%CI 95.2–99.5). Here, the dry swab is placed directly into the test‐specific extraction buffer which destroys viral RNA integrity, thereby precluding RT‐PCR testing. Dry swab antigen test performance was thus compared with RT‐PCR testing on paired nasopharyngeal swabs, with the first swab always reserved for RT‐PCR. Our study design, therefore, leaves in question whether the four “false negative” results obtained were due to analytical sensitivity or sampling issues. The inclusion of a relatively high viral load sample (C t 23.7) amongst the false negatives suggests sampling as the most plausible explanation, notably as this was confirmed as a cause of discordant results during the UTM arm of this study. Here, the initial pooled analysis (hospital and PHTC samples) revealed two false‐negative results based on the first swab RT‐PCR result. Repeat RT‐PCR testing of the same sample used for antigen testing, was negative, confirming that preanalytical issues, such as swab collection, play a critical role in test performance, as has been highlighted by others. In the retrospective part of our study, positive samples included 12.8% mutants, mostly Alpha, with no difference in assay performance, and Delta was the predominant circulating variant during the prospective study, indicating that the HISCL assay can detect multiple variants. Monitoring the performance of diagnostic assays with changing variants is crucial and in case of reducing performance, assay modifications may be needed.

Although there is no reference standard for infectiousness, evidence exists that viral transmission is far less likely at C t ≥ 30, hence this threshold has been adopted by some healthcare authorities to gauge transmission risk. Using C t < 30 as the threshold for positivity, the HISCL antigen assay performance for UTM swab samples was excellent with 100% sensitivity (95%CI 71.5%–100%) and an overall specificity of 99.3% (95%CI 98.1%–99.8%). A multicentre evaluation of the Elecsys® SARS‐CoV‐2 Antigen assay, a similar CLEIA‐based automated immunoassay, also showed high sensitivity (93.7%, 95% CI 89.7–96.5) at viral RNA concentrations of ≥ 104 copies/ml, equivalent to approximately C t < 29.9.

Unlike RT‐PCR, antigen assays directly measure viral material in the sample without amplification, hence sensitivity is strongly influenced by the infection kinetics and resultant viral concentration in the upper respiratory tract. Presence and duration of symptoms, and anatomical location of swabbing all play a role. Assay performance comparisons from different studies, where study cohort composition varied widely, is thus difficult. Notwithstanding that, to the best of our knowledge, there are no published studies of side‐by‐side automated antigen assay comparisons within the same patient group, when all C t values were considered, the HISCL antigen assay still performed favorably (overall sensitivity 74.7%, total 1787 samples and 16.7% RT‐PCR positivity) compared with other automated antigen assays where reported sensitivities ranged from 41.2% to 65.9% , , , overall, although all still falling short of the 80% sensitivity WHO recommendation. In a world without resource constraints, RT‐PCR would be the test of choice for every testing situation, however, as resource availability is unequal, there is a role for antigen tests, both rapid and automated immunoassays, such as the HISCL assay. Choice of an assay is influenced by infrastructure, affordability, testing indication (symptomatic, contact testing, and screening), and disease prevalence, which we have seen rises and falls. We calculated PPV and NPV at 5% and 25% disease prevalence, the prevailing RT‐PCR positivity rates during prospective sample collection for our study, and during the fourth wave of COVID‐19 infections in the Netherlands in January 2022, respectively. Assay performance for UTM swabs had a superior predictive value over dry swabs, irrespective of disease prevalence. Under high prevalence conditions, both NPV and PPV for UTM swabs exceeded 95%, even at C t < 35. The assay would thus be suitable for confirmation of infection, as well as ruling‐out infection in support of public health disease containment measures. At low prevalence, NPV is significantly better than PPV ( > 99% vs. >88%). Here, the greatest clinical utility would be infection exclusion, with the mass testing nature of such screening well supported by the high throughput capacity of the HISCL automated analyser.

Our study has limitations. It was conducted at a time when the SARS‐CoV‐2 positivity rate was rapidly declining, hence positive‐to‐negative sample ratio was low. Also, we were unable to directly compare antigen test performance on paired dry and UTM swabs, against RT‐PCR on the same UTM swab. This would have required participants consenting to three swabs which was considered unacceptable. Furthermore, we could not stratify assay sensitivity based on the presence and duration of symptoms, as such data was not available.

As the fourth wave of rising SARS‐CoV‐2 infections gained momentum in its surge around the globe, a new highly mutated SARS‐CoV‐2 strain identified as B.1.1.529, and subsequently named Omicron, was first reported to WHO from South Africa on 24 November 2021 triggering global alarm. This unpredictability of the SARS‐CoV‐2 pandemic has once again reinforced the critical importance of diagnostic testing as key to all ongoing efforts to contain the pandemic. Whilst rapid and reliable results are vital for case management, such diagnostic samples are equally critical for surveillance, including detection of new variants, which as we saw with Delta, and now Omicron, change the pandemic dynamic and the public health response needed for containment. In this regard, we strongly recommend the use of swabs collected in UTM, rather than dry swabs, to facilities considering the HISCL antigen testing platform, as this had superior performance, and caters for RT‐PCR confirmatory testing, if needed, and genotyping, which is again being undertaken with increasing frequency in support of tracking the global trajectory of Omicron.

CONFLICTS OF INTEREST

Joachim Linssen and Marion Münster are full‐time employees of Sysmex Europe GMBH who funded the study. The other authors have no conflicts of interest to declare.

ETHICS STATEMENT

The use of the frozen bio‐banked samples and residual nasopharyngeal swab samples prospectively collected for routine RT‐PCR testing was covered by existing national and local policy for laboratory test method validation using remnant specimens. The public health testing center arm of the study, which entailed the collection of an additional nasopharyngeal swab exclusively for study purposes was approved by the Radboud University Medical Center Ethics Review Board with approval number 2021‐8288. Participants were provided with a participant information sheet (in Dutch) and provided oral informed consent.

AUTHOR CONTRIBUTIONS

Joachim Linssen, Janette Rahamat‐Langendoen, and Heiman Wertheim conceptualized and designed the study. Marion Münster wrote the protocol. Janette Rahamat‐Langendoen liaised with the public health center and obtained ethical approval for the study. Study material procurement was undertaken by Joachim Linssen. Claire Schapendonk and Paul Daemen retrieved the bio‐banked samples. Sample preparation and processing on the HISCL analyser was done by Claire Schapendonk, Paul Daemen and Joachim Linssen. Data management and statistical analysis was done by Joachim Linssen. Data analysis and review was done by Joachim Linssen, Marion Münster, Janette Rahamat‐Langendoen and Heiman Wertheim. The manuscript was written by Marion Münster and the final version was edited and approved by all authors.