Comparison of Roche and Lumipulse quantitative SARS-CoV-2 antigen test performance using automated systems for the diagnosis of COVID-19.

BACKGROUND: Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) continues to spread worldwide. Here, we evaluated the performance of two quantitative antigen (Ag) tests, the Roche and Lumipulse Ag test, using automated platforms.
METHODS: We collected 637 nasopharyngeal swab samples from 274 individuals. Samples were subjected to quantitative reverse transcription PCR (RT-qPCR) as well as the Roche and Lumipulse Ag tests.
RESULTS: When RT-qPCR was used as a reference, the overall concordance rate of the Roche Ag test was 77.1% (491/637) with 70.0% (341/487) sensitivity and 100% specificity (150/150);. When inconclusive results of the Lumipulse Ag test were excluded, the overall concordance rate of Lumipulse Ag test was 88.3% (467/529) with 84.8% (330/389) sensitivity and 97.9% (137/140) specificity. The overall concordance rate between the Roche and Lumipulse Ag tests was 97.9% (518/529) with 96.7% (322/333) sensitivity and 100% (196/196) specificity. Quantitative Ag levels determined using the Roche and Lumipulse Ag tests were highly correlated (R2 = 0.922). The Roche and Lumipulse Ag tests showed high concordance up to 9 days after symptom onset, with progressively lower concordance thereafter.
CONCLUSIONS: The Roche and Lumipulse Ag tests showed equivalent assay performance and represent promising approaches for diagnosis of coronavirus disease 2019.

Introduction

Quantitative reverse transcription PCR (RT-qPCR) is a highly sensitive and specific assay and is considered the gold standard test for detecting severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). In addition, RT-qPCR can detect viral RNA at a very low copy number by amplification over 40 cycles. Therefore, RT-qPCR is a powerful tool for the diagnosis of SARS-CoV-2 infection.

Different approaches are, however, needed for infection control and public health measures (Mina et al., 2021). For example, RT-qPCR can detect viral RNA for several weeks in patients with coronavirus disease 2019 (COVID-19) (Hirotsu et al., 2020a, He et al., 2020, van Kampen et al., 2021). Although shedding of viral RNA can be prolonged in clinical samples, viable virus shedding is relatively short (Cevik et al., 2021). Therefore, detection of viral RNA does not necessarily indicate the presence of live infectious viruses (Cevik et al., 2021). Moreover, transmission events are thought to occur within a short period, likely a few days before and immediately after symptom onset (Cheng et al., 2020). Therefore, clinical samples generally do not contain culture-positive viruses (i.e., potentially “contagious” virus) from eight to ten days after symptom onset (Wolfel et al., 2020, La Scola et al., 2020, Million et al., 2020, Bullard et al., 2020, van Kampen et al., 2020, Perera et al., 2020).

We previously evaluated the accuracy of the Lumipulse antigen (Ag) test, an automated system that quantitatively measures nucleocapsid Ag using a chemiluminescent enzyme immunoassay (Hirotsu et al., 2021, Hirotsu et al., 2020b). The proportion of samples with positive results using the Lumipulse Ag test decreased rapidly around nine days after the onset of symptoms (Hirotsu et al., 2021), suggesting that this assay may be suitable for diagnosing COVID-19 patients when infectious viruses are present. Recently, Roche launched another Ag test platform for detecting nucleocapsid Ag. In this study, we compared the performance of the Roche and Lumipulse Ag tests to that of RT-qPCR using a panel of 637 samples.

Materials and methods

Patients and samples

A total of 637 nasopharyngeal swab samples were collected from 274 individuals (139 samples from 135 uninfected individuals and 498 samples from 139 patients with SARS-CoV-2-infection). On average, 3.6 samples (range, 1–14 samples) were collected from each patient with SARS-CoV-2. Of 139 SARS-CoV-2-infected patients, 123 were symptomatic (i.e., reported fever, cough, sore throat, fatigue, and/or headache), and 16 were asymptomatic. All 637 samples were subjected to RT-qPCR, the Roche Ag test, and the Lumipulse Ag test.

The Institutional Review Board of the Clinical Research and Genome Research Committee at Yamanashi Central Hospital approved this study and the use of an opt-out consent method (Approval No. C2019-30). The requirement for written informed consent was waived due to the study’s observational nature and the urgent need for improved COVID-19 diagnostics. Participation in the study was optional.

Sample collection and processing

All nasopharyngeal swab samples were collected using cotton swabs and placed in 3 mL of viral transport media (VTM) obtained from Copan Diagnostics (Murrieta, CA, USA), and 700 μL of the VTM were used for the Lumipulse Ag test immediately after sample collection. The residual VTM was temporarily stored at 4 °C, and 200 μL of the VTM were used for nucleic acid extraction within two hours after sample collection. In addition, 300 μL of freeze–thaw VTM were used for the Roche Ag test.

Lumipulse Ag test

The Ag levels were determined quantitatively with the Lumipulse SARS-CoV-2 Ag test (Fujirebio, Inc., Tokyo, Japan) as we previously reported (Hirotsu et al., 2021, Hirotsu et al., 2020b). In brief, 700 μL of the VTM samples were briefly vortexed, transferred into a sterile tube, and centrifuged at 2000 × g for 5 min. Aliquots (100 μL) of the supernatant were used for testing on the LUMIPULSE G600II automated system (Fujirebio). For samples with an Ag level >5000 pg/mL, the samples were diluted with the kit diluent and re-tested, and the Ag level was calculated, taking the dilution factor into account. Of 487 PCR-positive samples, 94 samples (19.3%) were re-tested. Samples with an Ag level ≥10 pg/mL were considered positive, samples with ≥1.0 pg/mL and <10.0 pg/mL were labeled inconclusive, while a result of ≤1.0 pg/mL was considered negative as per the manufacturer’s guidelines.

Roche Ag test

To measure antigen levels, samples were subjected to the Elecsys® SARS-CoV-2 Antigen Assay on a cobas® 8000 (e801 module) automated platform (Roche, Basel, Switzerland) in accordance with the manufacturer’s protocol with minor modifications. In brief, we transferred 300 μL of VTM into the sample cup (Hitachi, Tokyo, Japan) and added 30 μL of SARS-CoV-2 Extraction Solution C (Roche). After mixing for 5 s, the mixture was incubated for 5 min at room temperature. A 30 μL aliquot of the mixture was used for measuring antigen levels. This assay uses the electrochemiluminescence immunoassay principle. Samples with a cut-off index (COI) of <1.0 were considered negative, and those with a COI ≥1.0 were considered positive. Samples with COIs >17,000 were diluted with diluent and re-tested. COI values were then calculated, taking the dilution factor into account. Of 487 PCR-positive samples, four samples (0.8%) were re-tested.

Viral nucleic acid extraction

Total nucleic acid was isolated from the samples using the MagMAX Viral/Pathogen Nucleic Acid Isolation Kit (Thermo Fisher Scientific, Waltham, MA, USA) on the KingFisher Duo Prime System (Thermo Fisher Scientific) as we previously described (Hirotsu et al., 2020c, Hirotsu et al., 2020d). Briefly, we added 200 μL of VTM, 5 μL of proteinase K, 265 μL of a binding solution, 10 μL of total nucleic acid-binding beads, 0.5 mL of wash buffer, and 0.5–1 mL of 80% ethanol to each well of a deep-well 96-well plate. The nucleic acids were eluted with 70 μL of elution buffer. The total nucleic acids were immediately subjected to RT-qPCR.

RT-qPCR

According to the protocol developed by the National Institute of Infectious Diseases (NIID) in Japan (Shirato et al., 2020), we performed one-step RT-qPCR to detect SARS-CoV-2. This PCR amplifies the nucleocapsid gene of SARS-CoV-2 (NC_045512.2). The reaction mixture comprises 5 μL of 4× TaqMan Fast Virus 1-Step Master Mix (Thermo Fisher Scientific), 1.0 μL of 10 μM forward primer (5′-AAATTTTGGGGACCAGGAAC-3′), 1.4 μL of 10 μM reverse primer (5′-TGGCAGCTGTGTAGGTCAAC-3′), 0.8 μL of 5 μM probe (5′-FAM-ATGTCGCGCATTGGCATGGA-TAMRA-3′), 6.8 μL of nuclease-free water, and 5 μL of nucleic acid sample in a 20-μL total volume. Therefore, the expected amplicon size is 158 bp. For the internal positive control, the human ribonuclease P protein subunit p30 (RPP30) gene was used (Integrated DNA Technologies, Coralville, IA, USA) (Hirotsu et al., 2020e).

The RT-qPCR assays were conducted on a StepOnePlus Real-Time PCR System (Thermo Fisher Scientific) with the following cycling conditions: 50 °C for 5 min for reverse transcription, 95 °C for 20 s, and 45 cycles of 95 °C for 3 s and 60 °C for 30 s. The threshold was set at 0.2.

A threshold cycle (Ct) value was assigned to each PCR reaction, and the amplification curve was visually assessed. According to the national protocol (version 2.9.1) (Shirato et al., 2020), we deemed a sample to be positive when a visible amplification plot was observed, whereas a sample was deemed negative when no amplification was observed.

The absolute copy number of viral loads was determined using serially diluted DNA control targeting the N gene of SARS-CoV-2 (Integrated DNA Technologies, Coralville, IA) as previously described (Hirotsu et al., 2020e). The limit of detection of RT-qPCR using the primer/probe was considered as two copies according to the previous report (Shirato et al., 2020).

Statistical analysis

Sensitivity and specificity were calculated, excluding inconclusive results of the Lumipulse Ag test. Fisher’s exact test was used to assess differences among groups. Values of p < 0.05 were considered statistically significant. Cohen’s kappa (κ) coefficients and 95% confidence intervals (CI) were calculated using R version 3.6.2 (http://www.r-project.org/). Cohen’s κ values greater than 0.81 were interpreted as almost perfect agreement (Landis and Koch, 1977).

Results

Comparison of RT-qPCR, Roche and Lumipulse Ag tests

A total of 637 samples were analyzed using RT-qPCR, the Roche Ag test, and the Lumipulse Ag test. RT-qPCR was positive in 487 (76.5%) and negative in 150 (23.5%) samples. The Roche Ag test was positive in 341 (53.5%) and negative in 296 (46.5%) samples. The Lumipulse Ag test was positive in 333 (52.3%), inconclusive in 108 (17.0%), and negative in 196 (30.8%) samples.

We first examined concordance between RT-qPCR and Ag test results. Both Roche and Lumipulse Ag tests showed positive results for samples with high viral loads (Figure 1(A)). For the Roche Ag test, the average COI value of PCR-positive samples (n = 487) was 1.1 log10 COI (range: −0.44 to 4.9 log10 COI) (Figure 1(B)), and that of PCR-negative samples (n = 150) was −0.35 log10 COI (range: −0.54 to −0.02 log10 COI) (Figure 1(C)). For the Lumipulse Ag test, the average Ag level of PCR-positive samples was 2.1 log10 pg/mL (range: −2.0 to 6.5 log10 pg/mL) (Figure 1(D)) and that of PCR-negative samples was −0.89 log10 pg/mL (range: −2.0 to 1.3 log10 pg/mL) (Figure 1(E)).

Figure 1

Comparison of the results of RT-qPCR, the Roche Ag test, and the Lumipulse Ag test. (A) Overview of results of the Roche and Lumipulse Ag tests for RT-qPCR-positive samples. (B–E) Dot plots show SARS-CoV-2 Ag levels in the tested samples. (B, C) COI values of the Roche Ag test were plotted in samples positive by RT-qPCR (n = 487) (B) or negative by RT-qPCR (n = 150) (C). (D, E) Ag levels determined using the Lumipulse Ag test were plotted for samples positive by RT-qPCR (D) or negative by RT-qPCR (E). The red dashed lines indicate the decision threshold for the antigen test.

Abbreviations: RT-qPCR, reverse transcription-quantitative polymerase chain reaction; Ag, antigen; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; COI, cut-off index.

When RT-qPCR was used as a reference, the overall concordance rate of the Roche Ag test was 77.1% (491/637), sensitivity was 70.0% (341/487), and specificity was 100% (150/150) (κ = 0.524, 95% CI, 0.500–0.524) (Table 1 ). When inconclusive results of the Lumipulse Ag test were excluded, the overall concordance rate of the Lumipulse Ag test was 88.3% (467/529), sensitivity was 84.8% (330/389), and specificity was 97.9% (137/140) (κ = 0.733, 95% CI, 0.690–0.750) (Table 1). When the Lumipulse Ag test was used as a reference, the overall concordance rate of the Roche Ag test was 97.9% (518/529), sensitivity was 96.7% (322/333), and specificity was 100% (196/196) (κ = 0.811, 95% CI, 0.739–0.863) (Table 1). These results suggested that both the Roche and Lumipulse Ag tests were highly consistent.

Table 1

Comparison of the performance of RT-qPCR, the Roche Ag test, and the Lumipulse Ag test for detection of SARS-CoV-2.

RT-qPCR vs. Roche Ag test
	Positive by Roche Ag	Negative by Roche Ag
Positive by RT-qPCR	341	146
Negative by RT-qPCR	0	150

RT-qPCR, quantitative reverse transcription-polymerase chain reaction; Ag, antigen; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2.

Correlations between Ag levels, viral loads, and threshold cycle (Ct) values

We next examined correlations between the COI values of the Roche Ag test, viral loads, and Ct values determined by RT-qPCR. Positive correlations were observed between the Roche COI value and viral load (R2 = 0.805) and between the Roche COI value and Ct value (R2 = 0.815) (Figure 2(A)). Similarly, positive correlations were observed between Lumipulse Ag level and viral load (R2 = 0.837) and between Lumipulse Ag level and Ct value (R2 = 0.851) (Figure 2(B)), as we previously reported (Hirotsu et al., 2021).

Figure 2

Correlation between Ag levels and viral load or Ct values. (A–C) The dot plots show quantitative values for the Ag test and viral copy numbers determined by RT-qPCR. (A) Correlation between COI values determined using the Roche Ag test and viral loads (left) or Ct values (right) determined by RT-qPCR. (B) Correlations between antigen levels determined using the Lumipulse Ag test and viral loads (left) or Ct values (right) determined by RT-qPCR. (C) Correlations between COI values determined using the Roche Ag test and antigen levels determined using the Lumipulse Ag test.

Abbreviations: Ag, antigen; Ct, threshold cycle; RT-qPCR, reverse transcription-quantitative polymerase chain reaction; COI, cut-off index.

Both the Roche and Lumipulse Ag tests target the nucleocapsid protein and quantitatively assess Ag levels. To examine whether quantitative Ag levels were correlated, we compared the results of the Roche and Lumipulse Ag tests. The Lumipulse Ag levels and Roche COI values were highly correlated (R2 = 0.922) (Figure 2(C)). These results suggested that the Roche COI value reflected viral load and the amount of nucleocapsid Ag level with high accuracy.

Relationship between Ag test performance and viral load

To examine the relationship between viral load and the results of Ag tests, we calculated the number of viral copies used in each Ag test. The volume of VTM used for Ag test measurements was 100 μL for Lumipulse and 27.3 μL for Roche; therefore, the estimated number of viral copies per test is lower for the Roche Ag test.

Compared with RT-qPCR, the Roche Ag test was positive and showed 100% concordance for samples containing ≥5 log10 copies (99/99 samples), 99% concordance for samples containing 4–5 log10 copies (69/70), 97% concordance for samples containing 3–4 log10 copies (75/77), 81% concordance for samples containing 2–3 log10 copies (57/70), 41% concordance for samples containing 1–2 log10 copies (31/70), and 10% concordance for samples containing <1.0 log10 copies (31/70) (Table 2 ). Clearly, the positive concordance rate gradually declined with decreasing viral load.

Table 2

Relationship between SARS-CoV-2 Ag test performance and viral load.

RT-qPCR	Roche Ag test			Lumipulse Ag test
Viral Load (log10 copies/test)	Tested sample (n)	Positive (n)	Positive rate (%)	Tested sample (n)	Positive (n)	Inconclusive (n)	Positive rate (%)	Cumulative rate of positive and inconclusive (%)
≥5	99	99	100%	131	131	0	100%	100%
4–5	70	69	99%	79	78	1	99%	100%
4–3	77	75	97%	73	61	12	84%	100%
3–2	70	57	81%	70	37	28	53%	93%
2–1	75	31	41%	75	21	32	28%	71%
<1	96	10	10%	59	2	25	3%	46%

RT-qPCR, quantitative reverse transcription-polymerase chain reaction; Ag, antigen; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2.

Compared with RT-qPCR, the Lumipulse Ag test was positive and showed 100% concordance for samples containing ≥5 log10 copies (131/131 samples), 99% concordance for samples containing 4–5 log10 copies (78/79), 84% for samples containing 3–4 log10 copies (61/73), 53% concordance for samples containing 2–3 log10 copies (37/70), 28% concordance for samples containing 1–2 log10 copies (21/75), and 3% concordance for samples containing <1.0 log10 copies (2/59) (Table 2). These results indicated that RT-qPCR-positive samples with low viral loads were often judged as inconclusive using the Lumipulse Ag test (Table 2).

Relationship between Ag test performance and time since symptom onset

This study included 468 RT-qPCR-positive samples collected from 123 symptomatic patients infected with SARS-CoV-2 and 19 RT-qPCR-positive samples from 16 asymptomatic patients. To examine the relationship between time since symptom onset and concordance rate, the 468 samples were assessed in more detail (Table 3 ).

Table 3

Relationship between SARS-CoV-2 Ag test performance and time since symptom onset.

Days after symptom onset	RT-qPCR	Roche Ag test			Lumipulse Ag test				Fisher’s exact test (p-value)*
	Positive (n)	Positive (n)	Negative (n)	Positive rate (%)	Positive (n)	Inconclusive (n)	Negative (n)	Positive rate (%)
0–3	137	128	9	93%	122	10	5	89%	0.285
4–6	98	80	18	82%	81	14	3	83%	1
7–9	99	74	25	75%	71	21	7	72%	0.635
10–12	73	34	39	47%	33	28	12	45%	0.87
13–15	28	11	17	39%	10	11	7	36%	1
16–18	14	2	12	14%	2	6	6	14%	1
>19	19	0	19	0%	0	5	14	0%	1
Total (n)	468	329	139		319	95	54

RT-qPCR, quantitative reverse transcription-polymerase chain reaction; Ag, antigen; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2.

*According to Fisher’s exact test, there were no significant differences in the positive rates of the Roche and Lumipulse Ag tests.

The concordance rates of RT-qPCR-positive samples were 93% vs. 89% (Roche vs. Lumipulse) in samples obtained 0–3 days after symptom onset, 82% vs. 83% in samples obtained 4–6 days after symptom onset, 75% vs. 72% in samples obtained 7–9 days after symptom onset, 47% vs. 45% in samples obtained 10–12 days after symptom onset, 39% vs. 36% in samples obtained 13–15 days after symptom onset, 14% vs. 14% in samples obtained 16–18 days after symptom onset, and 0% for both assays in samples obtained 19 days or more after symptom onset (Table 3). Compared with RT-qPCR, there were no significant differences in the positive concordance rates of the Roche and Lumipulse Ag tests over this period (Table 3, p > 0.05, Fisher’s exact test).

Discussion

In this study, we compared two Ag quantification tests (Roche and Lumipulse) approved for the diagnosis of COVID-19 in Japan. Both Ag tests could accurately detect SARS-CoV-2 Ag in RT-qPCR-positive samples with high viral loads. In addition, Ag levels correlated with viral loads and Ct values determined by RT-qPCR. The performance of the Roche and Lumipulse Ag tests was nearly equivalent, indicating that both assays have high diagnostic accuracy up to nine days after the onset of symptoms. Furthermore, both the Roche and Lumipulse Ag tests can process large numbers of specimens using automated systems. The Roche Ag test can produce results in 18 min for a single sample and measure up to 300 tests per hour on the e801 module. The Lumipulse Ag test can produce results in 30 min for a single sample and process 60–120 samples per hour. Compared with RT-qPCR, the advantage of these automated assays is their low cost, scalability, rapid turnaround time, low hands-on-time requirements, and lower error rates (Omata et al., 2021).

Early detection and isolation of super-spreaders excreting live virus are essential to reduce the spread of SARS-CoV-2. RT-qPCR is an extremely sensitive assay and can detect very tiny amounts of RNA in clinical specimens (Sethuraman et al., 2020, Wang et al., 2020). Therefore, even in patients at the late stages of infection or after recovery, persistent viral excretion can be detected by RT-qPCR (Sethuraman et al., 2020). However, this may reflect the presence of a non-infectious virus or viral debris. Meanwhile, Roche and Lumipulse Ag tests have high detection rates up to nine days after symptom onset. Based on previous studies (van Kampen et al., 2021, Wolfel et al., 2020, Perera et al., 2020), the live virus can be isolated for approximately 8–10 days after symptom onset. This coincides with the period that an infected individual can transmit the virus to others (He et al., 2020). Therefore, the point when an Ag quantification test becomes negative may indicate a period of reduced infectivity (Pekosz et al., 2021).

The Roche Ag test results in a negative or positive test result, whereas the Lumipulse Ag test results in a positive, inconclusive, or negative result. The cut-off threshold for an inconclusive result on the Lumipulse Ag test is between 1.0 pg/mL and 10 pg/mL. This threshold setting was shown to be helpful in screening tests for the community, with high sensitivity and negative predicted value (Gili et al., 2021). The inconclusive results were frequent, especially in samples with low viral load (Figure 1(A)). In our results, 90.7% (98/108) of the inconclusive results by Lumipulse Ag test were positive by RT-qPCR, but only 17.6% (19/108) were positive by the Roche Ag test (Table 1). In other words, most of the inconclusive results by the Lumipulse Ag test were found to be negative by the Roche Ag test. Although the relationship between sensitivity and specificity is a trade-off based on which threshold is set, the two tests are likely to differ in samples with low viral load. These results provide valuable insights into the interpretation of the results of the two Ag quantification tests.

Ag tests have limitations. The sensitivity of Ag tests is lower than that of RT-qPCR. Therefore, it may be impossible to identify an infected patient at the very early or later phases of infection using Ag tests. However, it is sometimes difficult to detect the virus even by RT-qPCR. For example, multiple mutations have accumulated in circulating SARS-CoV-2 variants compared with the original strain identified in Wuhan, China (Shu and McCauley, 2017, Hadfield et al., 2018, European Centre for Disease Prevention and Control, 2020, Tegally et al., 2020, Faria et al., 2020; Hirotsu and Omata, 2021a,b). These mutations inhibit PCR amplification, particularly if the mutation occurs at the annealing location of the primer and/or probe (Bal et al., 2021, Filkins et al., 2021). Thus, RT-qPCR may not be able to detect all variants, and therefore it would be optimal to conduct both PCR and Ag tests to complement the shortcomings of each assay and enable accurate COVID-19 diagnosis.

In summary, quantitative Ag tests using automated systems offer rapid results, which can be used for prompt implementation of infection control and isolation measures. Rapid and accurate decision-making is of great importance in public health and will help control the spread of SARS-CoV-2.

Conflict of interest

The authors have no conflicts of interest to declare.

Funding source

This study was supported by a Grant-in-Aid for the Genome Research Project from Yamanashi Prefecture (to Y.H. and M.O.), Grant-in-Aid for Early-Career Scientists 18K16292 (to Y.H.), and Grant-in-Aid for Scientific Research (B) 20H03668 (to Y.H.) from the Japan Society for the Promotion of Science(JSPS) KAKENHI, a Research Grant for Young Scholars (to Y.H.) from Satoshi Omura Foundation, the YASUDA Medical Foundation (to Y.H.), the Uehara Memorial Foundation(to Y.H.), and Medical Research Grants from the Takeda Science Foundation(to Y.H.). The funders had no role in study design, data collection, analysis, decision to publish, or manuscript preparation.

Ethical approval statement

The Institutional Review Board of the Clinical Research and Genome Research Committee at Yamanashi Central Hospital approved this study and the use of an opt-out consent method (Approval No. C2019-30). The requirement for written informed consent was waived due to the study's observational nature and the urgent need for improved COVID-19 diagnostics. Participation in the study was optional.

Contributions

YH contributed to study design, data collection, data analysis, and writing – review & editing.

HS and MM contributed to sample preparation and data collection.

MH and HM contributed to supervision.

TT, YK, and YM contributed to providing resources.

MO contributed to supervision and writing – review & editing.