Comparison of commercial assays and laboratory developed tests for detection of SARS-CoV-2.

The global COVID-19 pandemic has led to the rapid development of tests for detection of SARS-CoV-2. Studies are required to assess the relative performance of different assays. Here, we compared the performance of two commercial assays, the cobas® SARS-CoV-2 (Roche Diagnostics) and Xpert® Xpress SARS-CoV-2 (Cepheid®) tests, and a laboratory developed RT-PCR test adapted for use on the Hologic® Panther Fusion® (Hologic®) instrument as well as Bio-Rad and QIAGEN real-time PCR detection systems. Performance characteristics for each test were determined by testing clinical specimens and reference material. All assays detect the pan-Sarbecovirus E (envelope structural protein) gene plus a SARS-CoV-2-specific target. The limit of detection for the E gene target varied from ∼2 copies/reaction to >30 copies/reaction. Due to assay-specific differences in sample processing and nucleic acid extraction, the overall analytical sensitivity ranged from 24 copies/mL specimen to 574 copies/mL specimen. Despite these differences, there was 100 % agreement between the commercial and laboratory developed tests. No false-negative or false-positive SARS-CoV-2 results were observed and there was no cross-reactivity with common respiratory viruses, including endemic coronaviruses.

Introduction

In December 2019, a previously unknown Betacoronavirus was isolated from a cluster of patients with pneumonia (Zhu et al., 2020). The rapid spread of this virus (SARS-CoV-2) has caused a global pandemic of coronavirus disease (COVID-19). Diagnostic testing guides treatment decisions, informs public health strategies, and is essential to the pandemic response. In Canada, interim orders expedited Health Canada licensing of the cobas® SARS-CoV-2 (Roche Diagnostics) and Xpert® Xpress SARS-CoV-2 (Cepheid®) tests. In Manitoba, Canada, the Cadham Provincial Laboratory (CPL) and Shared Health Diagnostic Services clinical microbiology laboratory have also implemented a laboratory developed RT-PCR test (LDT) that has been modified for compatibility with three different platforms. The use of multiple assays has enabled our laboratories to increase testing capacity, despite the limited supply of commercial test kits and LDT reagents. At the same time, platform-specific differences in assay performance have the potential to compromise SARS-CoV-2 testing (Marie et al., 2020). In this study, we compared the performance characteristics of five molecular assays for SARS-CoV-2 detection.

MATERIALS and METHODS

Description of the commercial RT-PCR assays and LDT variants

The cobas® SARS-CoV-2 (Roche Diagnostics) assay detects the conserved pan-Sarbecovirus E (envelope structural protein) gene and the SARS-CoV-2-specific orf1a (non-structural region) gene plus an added internal control RNA (Roche Molecular Systems Inc., 2020). Testing, including nucleic acid extraction and RT-PCR, is performed on a cobas® 6800 instrument (Roche).

The Xpert® Xpress SARS-CoV-2 (Cepheid®) assay detects the E gene and the SARS-CoV-2-specific N2 region of the N (nucleoprotein) gene, and includes both a sample processing control and a probe check control. Testing is performed on a GeneXpert® Dx instrument (Cepheid®) (Cepheid, 2020).

The LDT is a two-step RT-PCR assay. The first (screening) step targets the E gene and, as an endogenous control, the human RNase P gene (Corman et al., 2020). The second (confirmatory) step targets the SARS-CoV-2-specific N1 region of the N gene (US Centers for Disease Control and Prevention, 2020). The three variants, LDT-1, LDT-2, and LDT-FUS, target the same gene sequences, but have been modified for use with different extraction protocols and RT-PCR instruments (see supplemental material for complete details). LDT-1 combines nucleic acid extraction with MagMAX™ reagents on a KingFisher™ instrument (Thermo Scientific™) and RT-PCR performed on a Bio-Rad CFX96 real time PCR detection system. LDT-2 combines nucleic acid extraction on a QIAcube® (QIAGEN) using the QIAamp® DNA Mini Kit (LDT-2D) or RNeasy® Mini Kit (LDT-2R) with RT-PCR performed on a Rotor-Gene® Q real-time PCR-cycler (QIAGEN). LDT-FUS was developed for compatibility with the Panther Fusion® open access system (Hologic®). Nucleic acid extraction and RT-PCR are performed in tandem by the Panther Fusion® instrument.

Description of specimens and reference material

Analytical test characteristics were evaluated using three types of samples:1) A reference panel of simulated specimens derived from cultured SARS-CoV-2 virus that was inactivated by gamma irradiation and then added to viral transport medium (VTM) containing simulated respiratory secretion medium Bose et al. (2016); 2) a convenience set of clinical specimens submitted to CPL for routine viral diagnostic testing (i.e., nasopharyngeal swabs in VTM), and 3) AccuPlex™ SARS-CoV-2 Reference Material (SeraCare), which is recombinant viral RNA encapsulated in a replication-deficient mammalian virus.

Positive and negative agreement

Agreement between assays was assessed using clinical specimens and a set of simulated specimens positive for SARS-CoV-2. The same panel of five simulated specimens was tested with all assays. It was not possible to test the same set of clinical specimens with all assays, but every clinical specimen was tested with the reference standard LDT-1 assay. Each assay was evaluated using at least 10 SARS-CoV-2-positive specimens, 10 SARS-CoV-2-negative specimens, and archived clinical specimens positive for other common respiratory viruses, including endemic coronaviruses (i.e., O43, NL63, 229E), human rhinovirus, influenza (A/H1N1, A/H3 and B), respiratory syncytial virus, human metapneumovirus, parainfluenza virus, and adenovirus.

Limit of detection

Limit of detection (LOD) studies used serial dilutions (1000 to 10 genome copies/mL) of AccuPlex™ SARS-CoV-2 Reference Material (SeraCare) in VTM. For each dilution, at least three technical replicates were performed.

Results

All methods demonstrated 100 % agreement with LDT-1 results (Table 1 ). There were no false-negative or false-positive results, and no cross-reactivity with circulating respiratory viruses, including endemic coronaviruses. Ct values for the E gene target differed between assays. With the simulated and clinical specimens, values for LDT-FUS were consistently higher than the LDT-1, whereas LDT-2 values trended lower. LOD was determined for the E gene (all assays), as well as N1 (LDT-FUS), N2 (Xpert® Xpress) and orf1a (cobas®). LOD was initially calculated as genome copies/reaction (Table 1). The E gene LOD varied from 2 copies/reaction for the LDT-2 to ∼10 copies/reaction for the LDT-1, LDT-FUS and cobas®. For the LDT FUS, the secondary, N gene, target performed better, with an LOD of 5.1 copies/reaction, whereas the orf1a target used in the cobas® system, was less sensitive at only 31.4 copies/reaction. The ‘black box’ nature of the Cephied cartridge precluded accurate calculations for the Xpert® Xpress, but LOD was estimated to be ≤30 copies/reaction for the E gene and ≤7.5 copies/reaction for the N2 gene target.

Table 1

Comparison of SARS-CoV-2 Assays.

	LDT-1	LDT-2D	LDT-2R	LDT-FUS	cobas®	Xpert®
Clinical Panel
# Clinical Specimens Tested	177	45	45	44	50	38
# Positive for SARS-CoV-2	65	20	20	10	15	20
# Negative for SARS-CoV-2	112	25	25	34	35	18
# Positive for Other Respiratory Viruses	57	15	15	21	21	18
Positive Percent Agreement	Reference	100	100	100	100	100
Negative Percent Agreement	Reference	100	100	100	100	100
Mean ΔCt vs LDT-1 (95 % CI)	Reference	−2.3 (−2.9 to −1.7)	0.03 (−0.7 to 0.7)	1.9 (1.2 to 2.7)	0.6 (−0.1 to 1.4)	−0.9 (−1.4 to −0.3)
Reference Panel
# Simulated Specimens Tested	5	5	5	5	5	5
Mean ΔCt vs LDT-1 (95 % CI)	Reference	−1.8 (−2.6 to −1.1)	−2.1 (−2.4 to −1.8)	2.5 (1.7 to 3.3)	1.2 (0.8 to 1.6)	−0.6 (−1.2 to −0.03)
LOD (Analytical Sensitivity)
E gene LOD (copies per reaction)	9.1	2.0	2.0	10.4	9.7	≤30
E gene LOD (copies per mL VTM)	455	172	100	574	24	100
2nd target	N1	N1	N1	N1	orf1a	N2
2nd target LOD (copies per reaction)	n.d.	n.d.	n.d.	5.1	31.4	≤7.5
2nd target LOD (copies per mL VTM)	n.d.	n.d.	n.d.	235	85	25

Due to assay-specific differences in sample processing and nucleic acid extraction, the amount of clinical material used per reaction varied between tests. As such, E-gene LODs were also calculated as genome copies/mL specimen. By this measure, the cobas® (24 copies/mL) was the most sensitive assay, the LDT-2 (100–172 copies/mL) and Xpert® Xpress (∼100 copies/mL) were intermediate, whereas the LDT-1 (455 copies/mL) and LDT-FUS (574 copies/mL) were the least sensitive.

Discussion

Sensitive and specific assays for SARS-CoV-2 detection are essential to the global response to the COVID-19 pandemic. Testing guides clinical and public health interventions. Laboratories with molecular diagnostic capacity have implemented LDTs for SARS-CoV-2 and access to new commercial RT-PCR tests has been accelerated though ‘Interim Orders’ in Canada and the U.S. Food and Drug Administration’s ‘Emergency Use Authorization’ process. In Manitoba, Canada, the availability of multiple assays has allowed testing capacity to increase despite shortages in LDT reagents and limited supplies of commercial test kits. At the same time, the rapid implementation of these different tests has raised concerns that results may be compromised by differences among tests. In this study, we compared two commercial assays and three variations of a LDT. Despite some test-specific differences in Ct values and the LOD for the common E gene target, 100 % agreement was observed. There were no false-negative or false-positive results, and no cross-reactivity with common respiratory viruses. Analytical sensitivity studies indicated that the LOD for the E gene ranged from 2 copies/reaction (LDT-2) to ≤30 copies/reaction (Xpert® Xpress). Values for LDT-1, cobas® and LDT-FUS, were 9.1, 9.7, and 10.4 copies/reaction, respectively. However, the overall analytical sensitivity was influenced by assay-specific differences in sample processing and nucleic acid extraction. When input specimens volumes were considered, the LOD for the E gene ranged from 24 copies/mL for the cobas® system to 574 copies/mL specimen for the LDT-FUS. With the Xpert® Xpress and LDT-FUS, sensitivity was improved by the use of the secondary, N gene target.

There are several limitations to this study. Due to the clinical material available, it was not possible to test all clinical specimens with all assays. However, a common panel of simulated SARS-CoV-2 specimens was tested with all platforms. The assay-specific trends (e.g., difference in Ct values relative to the LDT-1 reference standard) were equivalent for the simulated specimens and clinical specimens. The apparent difference in performance with the LDT-2R assay was due a change in the extraction procedure. The LDT-2R values for the clinical specimens were associated with manual extraction, whereas the lower LDT-2R values for the simulated specimens, and all LDT-2D results, were associated with automated extraction on a QIAcube®. We assessed potential cross-reactivity of the Cepheid® assay against human rhinoviruses, but not other viruses. However, a multicenter evaluation of that assay showed no cross-reactivity with other respiratory pathogens, including human coronaviruses (Loeffelholz et al. (2020)). Recent reports describe performance differences between the cobas® and Xpert® systems for samples that contain low levels of target (Lowe et al., 2020; Moran et al., 2020). The majority (75 %) of our clinical specimens were strong positives (LDT-1 Ct ≤ 30) and only two samples had Ct values >35. As such, the excellent agreement we observed may not be generalizable to testing of weakly positive clinical material.

In summary, despite measurable differences in analytical sensitivity, the cobas® SARS-CoV-2 (Roche Diagnostics), Xpert® Xpress SARS-CoV-2 (Cepheid®) and three variations of a LDT performed equivalently and showed 100 % agreement when testing simulated and clinical specimens. This suggests that the concurrent use of these platforms does not compromise SARS-CoV-2 detection. In contrast, the availability of multiple systems has enabled us to increase testing capacity, despite supply chain disruptions. Similarly, there is the potential to implement testing algorithms that exploit the strengths of each system. For example, the Xpert® and Hologic® Fusion® platforms support on-demand testing of critical specimens, whereas the cobas® 6800 is optimized for high throughput batches. Our LDT has proven to be very flexible, and is readily adapted to the various nucleic acid extraction and RT-PCR instruments available in our health system.

Funding Statement

This work was funded by Manitoba Health and Shared Health in Manitoba, Canada.

CRediT authorship contribution statement

Kerry Dust: Conceptualization, Methodology, Investigation, Writing - review & editing. Adam Hedley: Conceptualization, Methodology, Investigation, Writing - review & editing. Kim Nichol: Conceptualization, Methodology, Investigation, Writing - review & editing. Derek Stein: Conceptualization, Methodology, Investigation, Writing - review & editing. Heather Adam: Conceptualization, Methodology, Investigation, Writing - review & editing. James A. Karlowsky: Supervison, Writing - review & editing. Jared Bullard: Supervison, Writing - review & editing. Paul Van Caeseele: Supervison, Writing - review & editing. David C. Alexander: Conceptualization, Methodology, Investigation, Writing - review & editing, Data curation, Writing - original draft.

Declaration of Competing Interest

None.