Multicenter evaluation of the NeuMoDx™ SARS-CoV-2 Test.

The SARS-CoV-2 virus has caused millions of confirmed COVID-19 cases worldwide and hundreds of thousands of deaths in less than 6 months. Mitigation measures including social distancing were implemented to control disease spread, however, thousands of new cases continue to be diagnosed daily. To resume some suspended social activities, early diagnosis and contact tracing are essential. To meet this required diagnostic and screening capacity, high throughput diagnostic assays are needed. The NeuMoDx™ SARS-CoV-2 assay, performed on a NeuMoDx molecular system, is a rapid, fully automated, qualitative real-time RT-PCR diagnostic test with throughput of up to 288 tests in an 8 -h shift. The assay received emergency use authorization from the FDA and is used in some large testing centers in the US. This paper describes the analytical and clinical performance of the assay at three centers: Johns Hopkins Hospital, St. Jude Children's Research Hospital, and the Wadsworth Center.

Introduction

SARS-CoV-2 is a novel Betacoronavirus that emerged in Wuhan, China in December 2019. The virus has been associated with a wide spectrum of disease severity, from mild to critical with high mortality [[1], [2], [3]]. The virus was first isolated from bronchoalveolar lavage specimens from three patients with pneumonia and its genome was characterized by whole genome sequencing [4]. The full genome was deposited to public access databases early in January 2020, which facilitated the rapid development of diagnostic molecular assays. As the laboratory testing capacity in the US increased dramatically in late February and during March 2020, thousands of cases were diagnosed daily, defining the extent of the outbreak and confirming large-scale community spread. Molecular diagnosis of SARS-CoV-2 remains the gold standard for diagnosing COVID-19 and is invaluable for infection control. These assays target various genes that include the nucleocapsid (N), envelope (E), spike (S), and RNA dependent RNA polymerase (RdRp) genes. The general recommendation is to target at least two genes to enhance the sensitivity and specificity of detection [5]. In general, the analytical sensitivity of commercially available molecular methods has been shown to be comparable [[5], [6], [7], [8], [9]].

Different nucleic acid extraction methods and varying levels of automation have also been employed and paired with molecular detection methods, providing alternative approaches for SARS-CoV-2 diagnosis with the choices and combinations determining assay complexity and turn-around time. Closed, fully automated systems that received Emergency Use Authorization (EUA) for SARS-CoV-2 assays include the Roche Cobas [[10], [11], [12], [13], [14], [15]], the Hologic Panther Fusion and Aptima [9,10,14,[16], [17], [18]], the Abbott [[19], [20], [21]], and the NeuMoDx. The NeuMoDx SARS-CoV-2 assay (NeuMoDx™ Molecular, Ann Arbor, Michigan) is a fully automated, rapid real-time RT-PCR that detects two conserved regions of the non-structural protein (Nsp) 2 and N genes. The NeuMoDx systems are random-access platforms that integrate nucleic acid extraction, target amplification, detection and reporting of results [22,23]. There are currently no reports that evaluated the performance of the NeuMoDx SARS-CoV-2 assay. In this study, we describe an evaluation of the performance characteristics of the NeuMoDx SARS-CoV-2 test in three laboratories that validated the assay for clinical use.

Materials and methods

Specimens

The study protocol was reviewed and/ or approved by local institutional/ethical review boards at each site. Clinical specimens referred for SARS-CoV-2 testing were diagnosed by the standard of care (SOC) test at each site (Table 1 ), those sites included Johns Hopkins Hospital, St. Jude Children’s Research Hospital, and the Wadsworth Center, New York State Department of Health. Site specific specimen transport media and preanalytical procedures are provided in Table 2 . Specimens were collected between March 1st and April 15th, 2020. Specimen types were nasopharyngeal swabs collected in viral transport media. Residual portions of 212 diagnostic specimens were tested with the NeuMoDx SARS-CoV-2 assay.

Table 1

Comparator reverse transcription-PCR methods for SARS-CoV-2 detection used in this study.

Site	Test name	Regulatory approval status	Targetgenes	Specimen types	Reference
Wadsworth Center	New York SARS-CoV-2 Real-time Reverse Transcriptase (RT)- PCR Diagnostic Panel	FDA EUA	N (N1, N2)	NPS	https://www.fda.gov/media/135847/download
Johns Hopkins HospitalandSt. Jude Children’s Research Hospital	RealStar® SARS-CoV-2 RT-PCR Kit 1.0 (Altona Diagnostics, Hamburg, Germany)	FDA EUA	S, E	NPS	https://www.fda.gov/media/137252/download

N Nucleocapsid gene. S Spike gene. E Envelope gene. NPS Nasopharyngeal swabs. EUA Emergency Use Authorization.

Table 2

Site-specific specimen types, collection, transport, and pre-analytical procedures.

Site	Specimen sources	Transport medium	Specimen processing for SOC testing	Specimen storage	Specimen processing for NeuMoDx testing
Wadsworth Center	NPS	Viral transport medium	110 μL of NPS were extracted by NUCLISENS® easyMAG®or EMAG and eluted in 110 μL	After collection, specimens are stored and shipped at 4°Residual portions of specimens are stored at -70°	300 μL of each specimen is treated with 300 μL of viral lysis buffer. NeuMoDx module used is for pretreated specimens
Johns Hopkins Hospital	NPS	Viral transport medium	500 μL of NPS were extracted by NUCLISENS® easyMAG®or EMAG and eluted in 50 μL	After collection, specimen are stored at 4°. Residual portions of specimens are stored at -70	500 μL of each specimen is treated with 500 μL of the the viral lysis buffer. NeuMoDx module used is for pretreated specimens
St. Jude Children’s Research Hospital	NPS	Viral transport medium	200 μL NPS were extracted by NUCLISENS® easyMAG® and eluted in 50 μL	After collection, specimen are stored at 4°. Residual portions of specimens are stored at -70°	300 μL of each specimen is treated with 300 μL of viral lysis buffer. NeuMoDx module used is for pretreated specimens

NPS Nasopharyngeal swab.

Laboratory methods

SARS-CoV-2 testing was performed in real time using a site-specific SOC method prior to testing with the NeuMoDx SARS-CoV-2 test at each site. Notably, both Wadsworth Center and Johns Hopkins Hospital laboratories used the NeuMoDx 288 system, however St. Jude Children’s Research Hospital laboratory used the NeuMoDx 96 molecular platform, which has essentially half the throughput capacity of the former. SOC methods and specimen storage conditions are described in Table 2.

Results

Analytical evaluation

At Wadsworth, pooled SARS-CoV-2 negative NPS specimens were spiked with 10 fold serial dilutions of a SARS-CoV-2 previously positive specimen and each dilution was tested in triplicate. Side by side comparisons were run between the NeuMoDx assay and the CDC panel assay (https://www.fda.gov/media/134922/download) in one experiment (Table 3a ), and between the NeuMoDx, CDC, and New York SARS-CoV-2 real-time assay in a second experiment (Table 3b ). Results showed the NeuMoDx assay to have an analytical sensitivity as good as, or better than, either of these two assays.

Table 3a

Analytical sensitivity of the NeuMoDx assay: comparison with the CDC assay.

Sample dilution	NeuMoDx			CDC assay
	N	Nsp2	SPC2	N1	N2	RP
1:10	22.8	23.7	ND	26.1	25.8	27.1
	22.9	23.8	ND	26.2	25.8	26.9
	22.9	23.8	ND	26	25.94	27
1:1E2	26.4	27.2	25.5	29.6	29.2	26.8
	26.5	27.3	25.6	29.4	29.4	26.8
	26.3	37.3	25.6	29.3	29.2	26.7
1:1E3	29.3	29.8	26.8	32.6	32.5	26.6
	29.4	29.9	27.3	32.3	32.4	26.4
	29.1	29.5	26.6	32.7	32.8	26.7
1:1E4	30.7	30.9	26.8	35.6	35.5	26.5
	31.1	31.2	27.5	25.7	36.8	26.6
	31	31.2	27.2	38	35.8	26.4
1:1E5	ND	ND	27	ND	39.3	26.4
	32.4	ND	27.6	ND	ND	26.5
	ND	ND	27.6	ND	ND	26.4
1:1E6	ND	ND	27.7	ND	ND	25.8
	ND	ND	27.4	ND	ND	26.4
	ND	ND	27.5	ND	ND	26.3

*ND target not detected. N Nucleocapsid gene. Nsp2 Non-structural protein 2. SPC2 Sample Process Control. RP RNase P gene control.

Table 3b

Analytical sensitivity of the NeuMoDx assay: comparison with the CDC and NY assays.

Sample dilution	NeuMoDx			CDC assay			NY assay
	N	Nsp2	SPC2	N1	N2	RP	N1	N2	RP
1:10	23.1	23.7	ND	26.2	25.6	27.2	25.5	25.7	28.2
	22.6	22.7	ND	25.9	25	27.2	25	25	27.8
	23	22.9	ND	25.5	25	27	24.7	24.7	27.9
1:1E2	26.3	26.6	ND	29.3	28.6	27	28.3	28.8	28
	26.7	26.8	ND	29.3	28.5	27	28.4	29.1	28
	27.7	27.4	ND	29.4	28.8	25.7	28.6	28.6	27.8
1:1E3	30.1	30.2	28.1	32.4	31.9	27	31.1	32	28
	29.2	29.6	26.9	30.2	30.2	26.8	29.7	30.2	27.8
	29.7	30.1	27.6	32.8	32	26.5	31.5	32.1	28.1
1:1E4	31.9	31.9	28.3	37.4	35.4	26.9	ND	35.2	27.9
	32.3	32.2	27.9	34.6	36.2	26.9	36.1	37	27.9
	32.3	32.2	28	ND	35.1	26.4	34.1	36.8	27.7
1:1E5	33	33.1	27.9	ND	ND	27	ND	ND	27.7
	ND	33	28.9	ND	37.6	27.1	ND	ND	27.8
	32.3	32.2	27.6	ND	38.4	26.6	36.1	ND	27.7
1:1E6	ND	ND	27.6	ND	ND	26.8	37.2	ND	27.9
	ND	ND	29.1	ND	37.5	27	ND	37.7	28
	ND	ND	28.2	ND	ND	26.9	ND	ND	28.3

*ND target not detected. N Nucleocapsid gene. Nsp2 Non-structural protein 2. SPC2 Sample Process Control. RP RNase P gene control.

At Johns Hopkins Hospital, a positive SARS-CoV-2 clinical specimen was serially diluted and two dilutions were quantified by the EUA approved BioRad ddPCR assay following the EUA package insert (https://www.fda.gov/media/137579/download) in triplicate in each tested dilution (Table 4 ). Each dilution was tested by the NeuMoDx assay in replicates to verify the lower limit of detection of the assay (Table 4). The data showed that the assay can detect 100 % of the replicates at 1267 copies/ mL (N1) and 1392 copies/ mL (N2). Table 5 shows the reproducibility of the assay within and between 3 different runs at the lower limit of detection dilution.

Table 4

Analytical sensitivity of the NeuMoDx SARS-CoV-2 assay.

Average copies as determined by ddPCR						NeuMoDx average Ct value
N1	N2	Specimen dilution	Number tested	Number detected	% detected	N	Nsp2
		1:1E2	6	6	100	20.4	20.8
		1:1E3	6	5(1 invalid)	100	24.1	24.6
		1:1E4	6	6	100	27.6	27.9
10,956	10,759	1:1E5	8	8	100	29.5	29.7
1,267	1,392	1:1E6	25	25	100	34.7	35.2
		1:1E7	25	16	64	35.3	36.0

Table 5

Reproducibility of the NeuMoDx SARS-CoV-2 assay.

Run 1			Run 2			Run 3
Number of replicates	NCt (st dev)	Nsp2 Ct (st dev)	Number of replicates	NCt (st dev)	Nsp2 Ct (st dev)	Number of replicates	NCt (st dev)	Nsp2 Ct (st dev)
7	34.9	35.3	6	35.3	35.9	7	35.2	35.8
St dev	(0.5)	(0.6)		(0.6)	(0.9)		(0.6)	(0.8)

St dev: standard deviation.

St. Jude Children’s Research Hospital used the Exact Diagnostics standard (http://www.exactdiagnostics.com/sars-cov-2-standard.html) to verify the lower limit of detection of the assay. The data showed that the limit of detection using this material is less than 200 copies/ mL (Table 6 ).

Table 6

Analytical sensitivity of the NeuMoDx SARS-CoV-2 assay using Exact SARS-CoV-2 Standard.

				NeuMoDx average Ct value
Exact standard dilution in copies/ mL	Number tested	Number detected	% detected	N	Nsp2
20,000	7	7	100	28.7	29.5
2,000	7	7	100	31.7	32.1
400	20	20	100	32.3	32.2
200	25	25	100	35.3	36.1

Clinical performance

The NeuMoDx SARS-CoV-2 clinical performance evaluation was performed on a total of 212 patient specimens across the three sites and the results were compared to the SOC test at each site. Specimens included 106 SARS-CoV-2-positive and 106 negative samples, all nasopharyngeal swabs.

Compared to all SOC methods combined, the positive agreement of the NeuMoDx SARS-CoV-2 test was 105/106 (99 %) and the negative agreement was 97/106 (91.5 %) (Table 7 ). The only discordant positive result had one gene target detected by the RealStar® SARS-CoV-2 RT-PCR Kit 1.0 (E gene, Ct = 36.38) which is considered presumptive positive. A discrepancy analysis of 6 of the 9 discordant negative specimens (3 had insufficient volume for repeat testing) using a third assay (the CDC SARS-CoV-2 panel) revealed that all were positive by the CDC assay (Table 8 ), indicating a negative agreement of ≥ 97 % and better sensitivity of the NeuMoDx SARS-CoV-2 assay than the RealStar® SARS-CoV-2 as well as comparable sensitivity to the CDC panel assay confirming our analytical performance evaluation.

Table 7

Clinical specimens’ agreement of NeuMoDx SARS-CoV-2 test and comparator RT-PCR tests.

	NeuMoDx No./Comparator No.
Comparator (targets)	Pos/Pos	Pos/Neg	Neg/Pos	Neg/Neg
All methods	105	9	1	97
St. JudeAltona (S, E)	9	0	1	0
HopkinsAltona (S, E)	81	9	0	92
Wadsworth Center (N1, N2)	15	0	0	5

Table 8

Discrepancy analysis of specimens negative by the SOC and positive by the NeuMoDx™ SARS-CoV-2 assay.

RealStar® SARS-CoV-2 RT-PCR Kit 1.0		NeuMoDx™ SARS-CoV-2		SARS-CoV-2 CDC Diagnostic Panel assay
E gene	S gene	N gene	Nsp2	N1	N2
ND	ND	29.9	30.1	33.2	33.9
ND	ND	31.3	31.1	34.3	37.4
ND	ND	ND	32.1	35.6	38.8
ND	ND	30.9	31	34	36.4
ND	ND	30.9	30.8	33.9	35.6
ND	ND	32.8	ND	34.6	36.9

*ND target not detected.

Discussion

The selection decision to implement an automated molecular detection system is complex, requiring consideration of initial cost, supply chain of reagents, maintenance costs, physical size, throughput capacity, hands-on time, complexity of operation, and reported performance in the literature or other sources. On-site evaluations are essential to confirm satisfactory performance. The COVID-19 pandemic has brought unprecedented pressure on clinical testing facilities to install and implement testing on large scale automated equipment, in order to accommodate testing volumes not seen in any previous outbreak. Laboratories have increasingly relied on reports from other facilities’ experiences with tests and instruments, to assist with internal decisions.

In this collaborative effort, we sought to compare the experiences with one large molecular analyzer, the NeuMoDx, across three diverse testing sites. Different methods were performed to assess the analytical performance in the three sites which showed equivalent or better analytical sensitivity if compared to the CDC panel assay, and a lower limit of detection that was different when different materials were used. Using a clinical specimen quantified by ddPCR compared to the Exact quantified standard resulted in about one log higher calculated limit of detection, however, the average Ct values at the LOD were comparable in both cases (Table 4, Table 6). The clinical performance of the NeuMoDx SARS-CoV-2 test was superior to the SOC testing where the assay was capable of diagnosing 9 false negatives, 6 of them were confirmed by a third assay. Overall, the data indicate that the analytical and clinical performance of the NeuMoDx SARS-CoV-2 test, meets or exceeds that of other assays used at each study site. In addition, implementing testing on the automated NeuMoDx platform resulted in significant reductions in labor and turn-around time as well as an increase in throughput capacity as noted by the impact of the assay on the Johns Hopkins turn-around time (Fig. 1 ). This was largely due to using a fully automated assay with minimal hands on time to gradually replace a manual assay that required an extraction step, followed by manual PCR, analysis, and reporting of the results.

Fig. 1

Effect of NeuMoDx on COVID-19 testing workflow and Turn-around time at Johns Hopkins Hospital. Turn-around time was calculated from specimen collection to reporting of results.

As the next phase in the control of the COVID-19 pandemic requires large scale diagnostic capacity and asymptomatic screening with contact tracing, high throughput testing has become a critical need. Different methodologies are under development including highly multiplexed next generation sequencing, of which, the Illumina COVIDSeq test was the first to receive an EUA by the FDA. Although these methods may offer the required scalability and reduced cost compared to PCR-based methods, pre-analytical specimen preparation and extraction are still required and post-analytical data analysis and results reporting require extensive validation. Closed, fully automated systems offer the required scalability and automation, however, supply chain has been an issue, limiting their large-scale implementation. Different closed high throughput PCR platforms are available for SARS-CoV-2 testing which include the Roche COBAS [24], and Hologic systems (Panther Fusion and Aptima) [9,25]. The NeuMoDx system’s major advantage is combining a continuously loaded instrument feature with a shorter run time (first result in 70 min). The assay is also capable of generating cumulative reports with Ct values which has been very valuable for investigating certain cases and for research based questions.

In conclusion, our study shows that the NeuMoDx SARS-CoV-2 assay has comparable or better analytic and clinical performance to the RealStar, the CDC Panel, and the New York SARS-CoV-2 assays. Implementing the assay was associated with a positive impact on the workflow which assisted with scaling up testing within the limits of the company’s supply chain. The clinical sensitivity of COVID-19 molecular diagnostics in different patients’ populations is still an area of research.

CRediT authorship contribution statement

Heba H. Mostafa: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Resources, Data curation, Writing - original draft, Visualization, Supervision, Project administration, Funding acquisition. Daryl M. Lamson: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Resources, Data curation, Writing - review & editing, Visualization, Supervision, Project administration. Katharine Uhteg: Software, Validation, Formal analysis, Data curation. Melissa Geahr: Software, Validation, Formal analysis. Linda Gluck: Software, Validation, Formal analysis. Jessica N. Brazelton de Cárdenas: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation. Elizabeth Morehead: Software, Validation, Formal analysis. Michael Forman: Software, Validation, Formal analysis. Karen C. Carroll: Conceptualization, Methodology, Investigation, Resources, Writing - review & editing, Visualization, Supervision, Project administration, Funding acquisition. Randall T. Hayden: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Resources, Data curation, Writing - review & editing, Visualization, Supervision, Project administration, Funding acquisition. Kirsten St. George: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Resources, Data curation, Writing - review & editing, Visualization, Supervision, Project administration, Funding acquisition.

Declaration of Competing Interest

KCC: Scientific advisory board for Pattern Diagnostics, Inc. and Scanogen, Inc., Research funds paid to the institution from BD Diagnostics, LBT Innovations, GenePOC, Inc., MeMed, Inc.

KSG: Research support from ThermoFisher and has a royalty generating collaborative agreement with Zeptometrix.

RTH: Advisory boards for Roche Molecular and Quidel Corporation.

HHM: Research funds from DiaSorin Molecular and Bio-Rad Laboratories.