Comparison of the clinical sensitivity and specificity of two commercial RNA SARS-CoV-2 assays.

OBJECTIVES: This study aimed to compare the performance of the NeuMoDx™ SARS-CoV-2 Assay, implemented on the NeuMoDx 96 Molecular System, with that of the ThermoFisher TaqPath™ COVID-19 CE-IVD RT-PCR Kit (reference method).
METHODS: Overall, 450 nasopharyngeal swab samples, previously tested using the reference method, were tested by the NeuMoDx Assay, and the clinical sensitivity and specificity of the assay were analyzed.
RESULTS: By retrospective statistical analysis of all valid results, the NeuMoDx Assay had a clinical specificity of 100% (95% confidence interval [CI]: 98.65-100.00) and a clinical sensitivity of 98.73% (95% CI: 95.47-99.85).
CONCLUSIONS: The NeuMoDx SARS-CoV-2 Assay demonstrated comparable analytical and clinical performance to the ThermoFisher TaqPath COVID-19 CE-IVD RT-PCR Kit. The NeuMoDx 96 Molecular System is well suited for automating medium-throughput routine SARS-CoV-2 testing or as an addition to high-throughput systems to allow fast-tracking for highly urgent clinical samples.

Introduction

In response to the COVID-19 pandemic, researchers have developed several diagnostic assays for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Reverse transcriptase polymerase chain reaction (RT-PCR) is the gold standard for diagnosis of active SARS-CoV-2 infections because of its high sensitivity and specificity (Park et al., 2020). Automation in molecular diagnostics enables scaling of testing capacity, which is critical for enabling a large number of tests (Eigner et al., 2019).

However, many published validation studies for SARS-CoV-2 assays have low sample numbers, differences in processes, and lack of validation by independent third parties (Vandenberg et al., 2021). In 2019, legislation brought UK RT-PCR testing under regulation by the United Kingdom Accreditation Service, against standards imposed by the Department of Health and Social Care (DHSC). The most stringent standard applied to travelers from destinations deemed high risk, on day 2 after arrival (Department of Health and Social Care, 2021).

The NeuMoDx™ SARS-CoV-2 Assay, as implemented on the NeuMoDx 96 Molecular System, is an automated, random-access, real-time RT-PCR test for detection of SARS-CoV-2 RNA, with a standard turnaround time of 80 minutes and a maximum throughput of 144 samples per 8 hours. This study evaluated the performance of the NeuMoDx Assay in comparison with the ThermoFisher TaqPath™ COVID-19 CE-IVD RT-PCR Kit, using the DHSC standards (Department of Health and Social Care, 2021).

Methods

Overall, 450 blinded nasopharyngeal swabs, previously tested using the ThermoFisher TaqPath COVID‑19 CE‑IVD RT‑PCR Kit (A48067), were provided by the UK Biocentre (Milton Keynes, UK); 175 were positive and 275 were negative for SARS-CoV-2 RNA. Samples were stored at –70°C, then transported to the Harefield laboratory (Uxbridge, UK) for testing using the NeuMoDx Assay (NeuMoDx SARS-CoV-2 test strip 300800). The manufacturer's method (NeuMoDx Molecular, 2021) was used without modification (Supplementary Methods; Table S2). Samples were collected and processed in April 2020 (1 week apart between sites), coinciding with a period of low SARS-CoV-2 community prevalence.

Results

Samples

Of the 450 samples, 3 were excluded because of invalid results per the NeuMoDx manufacturer's specifications (Table S3); 447 were therefore included in the primary analysis (Table S4). Of these, 12 had discordant results between assays, related to high cycle threshold (Ct) values using the reference method (Table S5a and S5b).

Low-level positives were excluded in a secondary analysis (Supplementary Methods); these results are presented here. In this secondary analysis, only 2 samples had discordant results.

NeuMoDx Assay performance

The clinical specificity of the NeuMoDx Assay was 100% (n/N=272/272; Table 1 ), meeting the DHSC threshold (Table S6). The sensitivity of the NeuMoDx Assay was 98.73% (n/N=155/157). Because the threshold for day 2 testing (99%) would equate to 155.43 of 157 samples, and only a whole number of samples can be positive, this was considered to satisfy the DHSC threshold (Table S6).

Table 1

Clinical sensitivity and specificity of the NeuMoDx SARS-CoV-2 Assay versus the reference method.

	Frequency (n/N)	Percentage (%)	Exact 2-sided 95% CI (%)
			Lower limit	Upper limit
Primary analysis: includes samples positive at the LoD per the reference method
Specificity	272/272	100.00	98.65	100.00
Sensitivity	163/175	93.14	88.33	96.41

CI, confidence interval; LoD, limit of detection.

The NeuMoDx 96 Molecular System provided turnaround times of 80 minutes and a throughput of 144 samples every 8 hours in a routine diagnostic setting.

Limit of detection

The LoD for the NeuMoDx Assay was 150 copies/mL (Table S1), in agreement with the manufacturer's evaluation (NeuMoDx Molecular, 2021). This exceeds the DHSC threshold (Department of Health and Social Care, 2021).

Discussion

In our study, the clinical specificity and sensitivity of the NeuMoDx Assay demonstrated similar analytical and clinical performance to the reference method and met the acceptance criteria for all the DHSC standards (Table S6).

The observed shift in Ct values between assays may be partially explained by sample transport and freeze-thaw. Unfortunately, because of the number of samples needed, it was impossible to mitigate this. Freeze-thaw can have marked effects: 1 study reported a 0.41 increase in Ct and a change in 10.2% of sample results after just 1 freeze-thaw cycle (Li et al., 2020).

The 2 discordant results could also be accounted for by freeze-thaw or transcription errors. It was not possible for either laboratory to repeat test the discordant samples; this represents a study limitation.

Other factors potentially affecting results are the use of off-label collection devices (see Supplementary Methods), samples not being tested in parallel, and a modest sample size. Furthermore, this study assumed that the reference method is 100% sensitive and specific; however, samples were tested when prevalence was low, and therefore, the probability of false positives was higher.

Importantly, the criteria for reference method positivity and low-level positivity were internally determined. Diagnostic laboratories frequently select cut-offs for assays, above which Ct values are considered negative (Sule and Oluwayelu, 2020); in this case, Ct ≥30 was applied. Furthermore, most diagnostic laboratories will verify the assay's LoD before implementation into routine testing (Burd, 2010): here, the internally determined reference method LoD was 100 copies/mL compared with the manufacturer-stated 250 copies/mL (ThermoFisher Scientific, 2021), which may be due to lack of standardization of SARS-CoV-2 reference materials. The recent availability of an International Standard for SARS-CoV-2 RNA may help alleviate these challenges (World Health Organization, 2020).

Because routine SARS-CoV-2 testing remains critical, the NeuMoDx Assay has demonstrated good clinical sensitivity and specificity on a platform well suited for automated clinical testing in our laboratory.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Authors’ contributions

ML contributed to data acquisition and analysis. PB contributed to study design, data acquisition, analysis, and interpretation. AO contributed to data acquisition, analysis, and interpretation. JK contributed to data analysis. RD contributed to study design and data interpretation. All authors were involved in drafting, critical revision, and final approval of the manuscript.

Ethical approval

Not applicable.

Declaration of Competing Interest

The authors report no competing interests.