Clinical evaluation of the GSD NovaPrime® SARS-CoV-2 RTq-PCR assay.

Faced with the emergence of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), high-throughput respiratory tests are in high demand. We evaluated the clinical performance of the GSD NovaPrime® SARS-CoV-2 RTq-PCR assay, a new assay that detects 2 specific RNA sequences of the nucleocapsid (N) gene. It was assessed using 99 nasopharyngeal samples and compared in parallel with the Allplex® assay. Among those samples, 72 and 27 were included in the positive (PPA) and negative (NPA) percent agreement analyses, respectively. In case of discordance, samples were reanalyzed with another amplification technique, the Aptima® SARS-CoV-2 assay. Cross-reactivity, including specimens positive for another respiratory virus and collected before the COVID-19 outbreak, was also evaluated (n = 32). Based on the patients' clinical history, the Ct (cycle threshold) values obtained, and the results of the Aptima® assay, the clinical performances were deemed satisfactory, with the PPA reaching a minimum percentage of 87.5% and the NPA reaching 100%. No cross-reactivity with other respiratory viruses was observed.

Introduction

More than two years after the discovery of the SARS-CoV-2 virus (Chan et al., 2020), the COVID-19 pandemic, which hit the world with an unprecedented economic and social health crisis, collected heavy tolls. On April 17, 2022, Johns Hopkins’ University assessment reported that the number of confirmed cases exceeded 504 156 480, the number of deaths worldwide had reached 6 197 169, and the vaccine doses administered had reached 11 167 654 005 (Johns Hopkins University, 2022). The fight against the virus will last for many more months, and the strategies put in place by the various health authorities should not be relaxed. Among them, screening is fundamental; above all, it is a diagnostic tool, but it is also a risk management tool in addition to social distancing, personal protective equipment and hygiene measures (Sciensano, 2021; World Health Organisation, 2021). RT–qPCR is considered the gold standard in biological testing, but its capacity can be limited by shortages of reagents or disposables, instrument saturation and lack of qualified staff. To save reagents and/or increase testing capacities, various strategies have been developed, such as sample pooling, parallel acquisition of new molecular biology techniques and/or extraction-free SARS-CoV-2 detection (Lohse et al., 2020; Lübke et al., 2020; Mancini et al., 2020; Yan et al., 2020). Among all these strategies, adding an additional back-up molecular test from another vendor would strengthen the testing capability of a clinical laboratory.

Faced with the health emergency, many companies have participated in this global effort toward PCR diagnostic test development. To date, numerous CE marked tests have been marketed (World Health Organisation, 2020). It is essential for laboratories to independently validate these methods before broad introduction into routine clinical practice. In this context, an increasing number of independent validations of RT–qPCR tests have been published by analyzing nasopharyngeal samples with different techniques targeting various regions of the SARS-CoV-2 genome, such as the helicase (Hel), N (nucleocapsid), transmembrane (M), E (envelope) and S (spike) genes (World Health Organisation, 2020). Hemagglutinin esterase (HE), open reading frames ORF1a and ORF1b and RNA-dependent RNA polymerase (RdRP) are other genes that encode structural proteins and represent alternative targets for COVID-19 diagnosis (Tang et al., 2020). It seems to have been accepted by the scientific community that at least 2 targets should be used when testing for SARS-CoV-2 using RT–PCR in clinical laboratories to avoid false-negative (FN) results (Vanaerschot et al., 2020). At the end of January 2022, testing capacities in Belgium and various European countries were close to saturation. Even though the number of tests carried out is decreasing, the positivity rate for Belgium remains high (23%) (Sciensano, 2022).

The aim of our study is twofold: First, in this context of uncertain evolution of the pandemic we wish to show the importance of not limiting ourselves to a single diagnostic method in clinical laboratories; rather, several diagnostic methods should be used for validation of results and to allow for a greater volume of tests while maintaining a 24-hour SARS-CoV-2 test result turnaround time. The second objective of our study is to verify the usefulness of adding a third reflex molecular technique in the face of difficult interpretations of samples with high Ct (cycle threshold) (Ct > 35).

We retrospectively evaluated the clinical performance of a new RT–qPCR method called GSD NovaPrime® SARS-CoV-2 (NovaTec®, Immundiagnostica GmbH, Dietzenbach, Germany) and compared it to the Allplex® SARS‐CoV‐2 Assay RT‐qPCR® (Seegene®Technologies, Seoul, South Korea).

Materials and methods

Study design

This study was conducted from November 25 to December 10, 2020, at the clinical biology laboratory of the Iris Hospitals South (HIS-IZZ, Brussels, Belgium). The 2 nucleic amplification techniques used routinely in our lab were used as comparison methods: Allplex® assay and Aptima® SARS-CoV-2 assay (Hologic, San Diego, CA).

Samples

The clinical performance of the NovaPrime® kit was assessed using 99 clinical samples. Among those, 72 samples positive by the Allplex® assay were included in the positive percent agreement (PPA) analysis. The remaining 27 samples negative by the Allplex® assay were included in the negative percent agreement (NPA) analysis.

The cross-reactivity evaluation covered 32 samples from nasopharyngeal aspirations collected in 2018, before the COVID-19 outbreak, and positive for another respiratory virus (Blairon et al., 2021). These 32 samples came from COVID-19-negative patients who had other active viral infections that could be considered confounding factors. They were all positive for at least one and up to 3 different viruses. The number of positive samples per pathogen was as follows: enterovirus/human rhinovirus n = 9, human metapneumovirus (hMPV) n = 4, influenza A n = 11, influenza B n = 3, and respiratory syncytial virus (RSV) n = 16. Viral agents were identified by the Belgian National Influenza Center using 2 in-house RT–qPCR assays (RT–qPCR for influenza A/B and multiplex RT–qPCR for RSV A, RSV B, hMPV and enterovirus/human rhinovirus) (Fischer et al., 2021).

The discrepant samples were reanalyzed with the Aptima® assay. Clinical information (symptoms presence/absence at the time of testing, known COVID-19 history, etc.) of the discordant cases was also analyzed when the patient's record was accessible.

All 99 samples (72 positive and 27 negative) tested in parallel with NovaPrime® and Allplex® assays were fresh nasopharyngeal swabs from routine testing; they included UTM-RT swabs (Copan SpA, Brescia, Italy) and Vacuette virus stabilization tubes (Greiner Bio-One International GmbH, Kremsmünster, Austria). No freezie-thaw steps were required, since we used fresh samples daily. The 32 clinical samples used for the cross-reactivity evaluation were collected before the beginning of the COVID-19 outbreak and were stored at −20°C.

Analytical procedures

Regarding the samples used to evaluate the clinical performance of the NovaPrime® assay after sampling collection, specimens were sent to the laboratory and were directly extracted according to the routine extraction protocol. On the same day, the RNA extracts were amplified first with the Allplex® method and then with the NovaPrime® method.

Samples from the cross-reactivity study followed the same analytical flow after thawing. After following the same extraction procedure as the samples from the routine clinical collection, the extracts were amplified first with the Allplex® method and then with the NovaPrime® method.

Extraction protocol

To facilitate the use of the new NovaPrime® kit and save time in reporting results to prescribers if this test were to be used in clinical routine, the extraction protocol (REp) was the same for both Allplex® and NovaPrime® assays and was performed using the STARMag Viral DNA/RNA 200 C kit (Seegene Technologies, Seoul, South Korea) with a Nimbus extraction platform (Seegene Technologies, Seoul, South Korea) according to the manufacturer's instructions. Then, extracted nucleic acids were used in Allplex®/NovaPrime® reactions according to the manufacturer's instructions.

Master mix NovaPrime®

The NovaPrime® kit contains specific primers and probes labeled with fluorescent reporter and quencher dyes for amplification and simultaneous detection of specific RNA sequences representing 2 specific regions of the SARS-CoV-2 N gene. However, as the extracts were prepared according to the Seegene automated extraction method, the reaction setup was performed based on the manufacturer's recommendations, and EC (extraction control, which represents the internal control) was added after the extraction step as follows: 5 µL of E-MIX (RT–PCR enzyme mix) + 3 µL of PP (primer-probe mix) +1 µL of EC.

Amplification

cDNA synthesis and amplification were performed with a CFX96 C1000 thermal cycler (Bio–Rad Laboratories, Hercules, CA). The fluorophores used with the Allplex® kit were FAM, Cal Red 610, Quasar 670 and HEX for detecting the E gene, RNA-dependent RNA polymerase (RdRP) gene/S gene, N gene and internal control, respectively. The interpretation of the results and Ct calculations were performed with Seegene SARS-CoV-2 Viewer software version 3.19.003.010 (Seegene Technologies, Seoul, South Korea). Targets detected with a Ct less than 40 were considered positive. A sample was considered positive if at least one of the targets sought was declared positive.

With the NovaPrime® assay, the fluorophores FAM and Cy5 detected the N gene and EC, respectively. Valid samples showing FAM targets with a Ct less than or equal to 38 were considered positive.

Analysis of discrepancies

The true-positive (TP), true-negative (TN), false-positive (FP) and false-negative (FN) categories were determined based on the Allplex® reference standard used in this study. In cases of discordant (FN or FP) results between the NovaPrime® and Allplex® assays, an Aptima® assay was performed with the Panther system (Hologic, San Diego) on the same sample, as long as the remaining volume was sufficient to follow the manufacturer's instructions. The Hologic Panther SARS-CoV-2 transcription mediated amplification test (TMA) amplifies and detects 2 conserved regions of the ORF1ab gene in the same reaction. The 2 regions are not differentiated, and amplification of either or both regions leads to a relative light unit (RLU) signal. The assay results are determined by a cut-off based on the total RLU and the kinetic curve type. Specimens analyzed by Panther are recorded as “positive” or “negative” with an associated RLU value or “invalid.”

Statistical analyses

Statistical analyses were carried out using MedCalc version 10.4.0.0 (MedCalc Software, Ostend, Belgium). The results (positive or negative) of NovaPrime® RT–qPCR were compared with those of Allplex® to determine the PPA and NPA. A more detailed comparison of the Ct values concerning the target of the N gene common to these 2 assays was carried out by applying a Wilcoxon test. A P value <0.05 was considered statistically significant. The clinical performance of the NovaPrime® assay was examined using receiver operator characteristic (ROC) curves. The ROC area under the curve (AUC) was calculated as the positive TP and FP fractions determined according to known positive results with the Allplex® method chosen arbitrarily as the reference method.

Results

To approach PPA as accurately as possible, the clinical performance results of the NovaPrime® assay are presented in 4 steps.

First, based solely on the analytical results interpretation, notices transmitted by the firms Seegene and NovaTech, that is, including all Ct results obtained for each target sought (E, RdRP/S and N genes) with the Allplex® assay and only the Ct results obtained for the N gene with the NovaPrime® assay, the NovaPrime® assay showed an overall PPA of 84.7% compared to the Allplex® assay. Eleven out of 72 samples were classified as FN (Table 1 ). Among these 11 samples, 9 with Ct > 36 were positive for only 1 or 2 viral targets on the Allplex®, and 2 showed the following Ct values for the E, RdRP/S and N gene targets: 31.92, 32.15, and 30.6 and 37.53, 35.11, and 33.26, respectively (Table 1). Analysis of the ROC curve also confirmed a sensitivity of 84.7% (95% CI: 74.3-92.1%), and the adapted cutoff ≤37.84 was identical to that transmitted by the manufacturer ≤38 (Fig. 2).

Table 1

Analysis of discordant results between Allplex®, NovaPrime®, and Panther® technique based on Ct and available clinical information.

Allplex®				NovaPrime®			Panther®		Available clinical data at time of testing
E gene (Ct)	RdRP/S gene (Ct)	N gene (Ct)		Result	N gene (Ct)	Ct result interpretationa	ORF1ab gene (RLU)	RLU result interpretationa
(-)	(-)	36.6		P	(-)	FN	583	TP	Chest symptoms and pain for 2 weeks
(-)	38.13	36.06		P	(-)	FN	ND	ND	Degradation of the general condition. COVID-19 diagnosis known for a month.
37.86	(-)	36.74		P	(-)	FN	ND	ND	Isolation output. Asymptomatic. History of COVID-19 diagnosed >2 weeks.
(-)	(-)	37.1		P	(-)	FN	ND	ND	Patient admitted to intensive care. Hospitalization >1 month for COVID-19 infection.
37.47	(-)	36.35		P	(-)	FN	990	TP	Asymptomatic. International travel. History of COVID-19 diagnosed >2 weeks.
31.92	32.15	30.6		P	(-)	FN	1049	TP	Asymptomatic. COVID-19 diagnosed >3 weeks.
38.64	(-)	36.6		P	(-)	FN	606	TP	Asymptomatic. International travel. Unknown medical history.
37.53	35.11	33.26		P	(-)	FN	572	TP	Hospitalization for total hip arthroplasty. Unknown COVID-19 history.
(-)	(-)	36.21		P	(-)	FN	672	TP	Neurological symptoms. Coming from a nursing home. Unknown COVID-19 history.
36.92	(-)	(-)		P	(-)	FN	(-)	FN	Asymptomatic. Unknown COVID-19 history.
38.21	(-)	(-)		P	(-)	FN	(-)	FN	Asymptomatic. Presented 1 month before clinical symptoms suggestive of COVID-19.

(-) = not detected; ND = not determined; P = positive; FN = false negative; TP = true positive.

The interpretation of the raw results was made by comparing with the reference method chosen arbitrarily in this study (Allplex®).

Fig. 2

Clinical performance of the NovaPrime® assay (n = 99 samples). The adapted cut-off was ≤37.84. ROC curve reported excellent specificity of 100% (95% CI: 87.2–100%) and sensitivity of 84.7% (95% CI: 74.3–92.1%) with this adapted cut-off.

Second, focusing only on Ct values obtained with the N gene target common to these 2 assays, the NovaPrime® assay showed a PPA of 87.1% compared to the Allplex assay. The distribution of the different Ct values obtained with both methods is shown in Fig. 1 . The median global Ct with NovaPrime® (95% CI) was 24.43 (22.08-27.47). According to the Wilcoxon test, the Ct values obtained with the NovaPrime® and Allplex® tests were not significantly different (P = 0.354).

Fig. 1

Distribution of Ct values obtained with both RT-qPCR methods for the detection of SARS-CoV-2. The global dispersion of Ct with NovaPrime® assay is 21.3% <20 cycles, 31.1% between 20 and 25 cycles, 19.7% between 25 and 30 cycles, 23% between 30 and 35 cycles, 4.9% >35 cycles.

Third, the introduction of a third molecular method made it possible to reclassify some of the 11 discordant cases obtained with the NovaPrime® method compared with the Allplex® method. Among these 11 cases, 2 also came back negative with the Aptima method, 3 unfortunately could not be analyzed, and 6 were positive with this third method (including 5/6 with low RLU < 1000). Based solely on this third Aptima assay used to settle these discordant cases, 9 samples are finally considered FN, and the calculated PPA was 87.5%.

The last step to determine the accuracy of the Novaprime® assay more closely called for the analysis of the clinical history of these 9 patients. A total of 5/9 had already had a history of COVID-19 diagnosed for at least 2 weeks (Table 1). Interestingly, the 3 cases for which an Aptima® assay could not be performed all belonged to these cases with a known history of COVID-19. Combining both the Aptima result and the patients’ clinical history, we finally confirmed 9 discordant cases (6 positive with the Aptima® assay and 3 with a positive history of COVID-19 diagnosis) as FN, and the PPA reached a minimum percentage of 87.5%.

Regarding the NPA analysis, since none of the samples analyzed turned out to be an FP, is the NPA was 100%.

Regarding the cross-reactivity study, all 32 nasopharyngeal aspirations positive for one or more respiratory viruses were negative with both the NovaPrime® and Allplex® assays.

Discussion

RT–qPCR: selection of genetic detection targets

In December 2020, many mutations in the S protein spread across Europe, but other mutations involving the N, E, or RdRP gene have also been identified (Pachetti et al., 2020; Plante et al., 2020; Rahman et al., 2021; Tung and Limtung, 2020; Vankadari, 2020; Ziegler et al., 2020). Today, the variant of concern (VOC) that predominates in Europe is, according to the classification of the World Health Organization (WHO), the Omicron variant (B.1.1.529), which has characteristic mutations affecting the S protein (European Centre for Disease Prevention and Control, 2022).

The NovaPrime® kit evaluated in this study targets 2 specific regions of the SARS-CoV-2 N gene. The ability to detect at least 2 different protein S targets is an advantage of this assay, given that all the VOCs that have emerged since the beginning of the pandemic had mutations affecting the S protein.

Potential difficulties in interpreting discordant cases

In some cases, during extended testing (and an increasing number of asymptomatic persons tested), a positive RT–PCR result can be difficult to interpret and can raise subsequent questions related to the necessity of quarantine measures, contact tracing and real-time epidemiological monitoring (Sciensano, 2020). In our study, the clinical performance evaluation of the NovaPrime® assay was performed carefully in 4 steps to approach the PPA as accurately as possible. (1) First, based only on the raw interpretation of the Ct results of the Allplex® assay for all genes (E, RdRP/S and N gene) chosen arbitrarily as a reference, the PPA of the NovaPrime® assay was 84.7%. (2) By comparing only the Ct values of the N gene targets common to these 2 tests, the NovaPrime® assay showed a PPA of 87.1%. (3) By introducing a third molecular method (the Aptima® assay) to reclassify discordant cases, the recalculated PPA was 87.5%. (4) Finally, based on the patients’ clinical history and the results obtained with the Aptima® assay for discordant samples (characterized by low viral loads), the PPA reached a minimum percentage of 87.5%. Combining the clinical history of the patients with the Ct value obtained is essential to help in the interpretation of discordant cases. A total of 4/11 discordant cases had no prior COVID-19 infection reported, and 6/11 patients were asymptomatic, except for one case with neurological symptoms that were not suggestive of COVID-19. High Ct values were observed with the Allplex® method (>36 Ct in 9 cases/11), suggesting that those samples contained a low viral load, but it was difficult to interpret those results regarding the potential infectivity of the patients. These 9 individuals could just as easily have been infectious as they could have been only bearing traces of a previous infection. Other authors have also shown the importance of transmitting a Ct value to the clinician to help him interpret the results and make the best clinical decision (Tom and Mina, 2020). Although it is commonly accepted that high Ct values (>35) are frequently associated with an old infection and, therefore, with an absence of contagiousness (Centers for Disease Control, 2020; Singanayagam et al., 2020), it is important to remain cautious. Indeed, it is also possible that the patient is at the very beginning of an infection or even of a reinfection if a clinical history is known. In such a situation, a new assay within 24 or even 48 hours should be offered to the patient. A drop in Ct will then confirm an active infection and contagiousness (Tom and Mina, 2020). The introduction of another molecular technique may also help to interpret some of these complex cases.

This third analysis carried out on the same sample also offers the advantage of being able to provide a result and a faster interpretation. It might even avoid retesting the patient 24 to 48 hours later. In our study, the analysis of the 11 discordant cases by the Aptima® technique revealed 2 negative results, reinforcing our suspicion of old infections.

Study limits

Finally, the present study had some limitations. Further studies are needed to confirm whether this kit can detect viruses carrying the new mutations that target the N gene.

Other additional studies are necessary to confirm the clinical and analytical performance of this assay. Another limitation of this study is related to the absence of samples from other viruses of the Coronaviridae family. Given the scarcity of these samples, only samples positive for enterovirus/human rhinovirus, hMPV, influenza A, influenza B and RSV were tested, and cross-reactivity was not seen. Finally, the storage conditions of clinical samples used to assess cross-reactivity should have ideally been at −80°C rather than -20°C. The quality of the extracted RNA has a crucial effect on the performance of the RT–qPCR system. Working on the same fresh RNA extract is mandatory for comparing the 2 techniques so as not to introduce biases. This was only possible for 99/131 clinical samples.

Another limitation of our study requires consideration. In the absence of a true reference standard method for SARS-CoV-2 detection that is well recognized (Axell-House et al., 2020; Mitchell et al., 2020), we arbitrarily evaluated the NovaPrime® assay based on the results obtained with the Allplex® method, which was already validated and implemented in our laboratory routine. Many studies have used different approaches for establishing an arbitrary reference standard for diagnostic assays, such as using composite/consensus reference standards, comparing results of a new test to one of the RT–PCR results (Axell-House et al., 2020) or by considering the CDC PCR assay as a reference standard (Kanwar et al., 2021).

Conclusion

To our knowledge, this study is the first to report an external clinical evaluation of the GSD NovaPrime® SARS-CoV-2 RTq-PCR assay compared to the Allplex® SARS-CoV-2 assay. Our results suggest that the NovaPrime® assay is a reliable method for the diagnosis of SARS-CoV-2 infection. This study also highlights the usefulness of introducing an additional molecular technique in clinical laboratories as a back-up assay to not only allow for continued performance of diagnostic tests in the event of supply disruption of other reagents but also to help with the interpretation of some difficult cases. Positive results are indicative of the presence of SARS-CoV-2 RNA, and clinical correlation with patient history and other diagnostic information, including the result of an additional molecular technique, is mandatory to assess the infection status of the patient.

Author contribution statement

Marie Tré-Hardy: Conceptualization, Investigation, Formal analysis, Writing - original draft, Writing - review & editing. Sébastien Piteüs: Methodology, Formal analysis. Ingrid Beukinga: Writing - review & editing. Laurent Blairon: Conceptualization, Investigation, Formal analysis, Writing - review & editing.

Funding

NovaPrime® tests were provided at no cost by NovaTec company (NovaTec® Immundiagnostica GmbH, Dietzenbach, Germany). The company had no role in study design, data collection and analysis, decision to publish, or preparing the manuscript.

Informed consent

According to Belgian Health Public Law (article 3§2 of the Law of May,7 2004 relating to experiments on humans), this type of study did not require specific informed consent or ethics committee approval.

Acknowledgments

The authors thank all the members of the clinical laboratory staff for technical assistance.

Declaration of competing interest

The authors have no relevant competing interest to disclose in relation to this work.