Lack of sensitivity of an IVD/CE-labelled kit targeting the S gene for detection of SARS-CoV-2.

OBJECTIVES: New molecular tests for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) are being rapidly launched in response to the coronavirus disease 2019 (COVID-19) pandemic. The aim of this study was to evaluate the analytical and clinical performance of the VIASURE SARS-CoV-2 S gene RT-PCR Kit on the BD Max™ system and to compare results with those obtained with the cobas® SARS-CoV-2 test on the cobas® 6800 system.
METHODS: For testing the analytical performance, reference material was used. Clinical samples (n = 101) obtained from individuals with symptoms compatible with COVID-19 were studied. Oropharyngeal and nasopharyngeal swabs were collected by using either ESwab™ or UTM™ collection systems.
RESULTS: When the analytical performance was evaluated, the sample containing the lowest SARS-CoV-2 concentration tested negative with the VIASURE test whereas results obtained with the cobas® test were found to be concordant with the results expected. Six out of the 101 clinical samples (5.9%) showed an inhibition with the VIASURE test. When analysing the remaining 95 clinical samples, 27 were found to be negative with both assays. Of 68 samples that were positive with the cobas® test, the VIASURE test missed 21 (30.9 %) samples. All of those 21 samples had shown Ct values ≥ 31 with the cobas® 6800 system. None of the samples tested positive with the VIASURE test and negative with the cobas® test.
CONCLUSIONS: The VIASURE test was impaired by a lack of sensitivity and a relatively high number of invalid results. When using the VIASURE test for routine testing, a significant number of COVID-19-positive samples would have been missed.

Introduction

The coronavirus disease 2019 (COVID-19) pandemic has caused an extremely high demand for molecular severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) testing of respiratory samples. This situation puts high pressure on manufacturers of molecular diagnostics to place new assays rapidly on the market to meet this expectation. Both laboratories involved in this study, in Graz and Salzburg, are major COVID-19 diagnostic institutions in the Federal State of Styria and Salzburg in Austria, respectively. We have been testing for SARS-CoV-2 since the end of January 2020 and initially used the real-time RT-PCR protocol established by Corman et al. [1]. As a result of the significant increase of sample numbers, we introduced commercially available assays including the fully automated In Vitro Diagnostic Medical Device/Conformité Européenne (IVD/CE)-labelled (and US Food and Drug Administration-approved) cobas® SARS-CoV-2 test on the cobas® 6800 system (Roche Molecular Diagnostics, Branchburg, NJ, USA). The analytical performance of this assay in clinical specimens was recently evaluated [2]. The assay uses a two-target quantitative RT-PCR system targeting the ORF1 (target 1), a non-structural region that is unique to SARS-CoV-2, and a conserved region in the structural protein envelope E gene for pan-164 Sarbecovirus (target 2) detection [2]. In March 2020, the IVD/CE-marked VIASURE SARS-CoV-2 S gene RT-PCR Kit from CerTest Biotec (Zaragoza, Spain) targeting the S gene was launched [3]. This test has been designed for the BD MAX™ diagnostic platform (Becton Dickinson, Franklin Lakes, NJ, USA) also integrating nucleic acid extraction, purification, amplification and detection. The latter platform has been widely used for the detection of bacterial and viral pathogens from clinical samples [4,5], including our diagnostic laboratories. As part of ongoing efforts to optimize the workflow of our COVID-19 diagnostics with different assays, we aimed to introduce the VIASURE SARS-CoV-2 S gene RT-PCR to be run on the BD MAX™ platform. Here we report on the results of independent validations of this newly CE-marked automated SARS-CoV-2 S gene RT-PCR test from CerTest Biotec in our two laboratories.

Methods

In order to evaluate the clinical performance of the VIASURE test at the laboratories in Graz and Salzburg, cohorts of cobas® 6800 positive clinical samples with Ct values ranging from 20.82 to 37.85 and negative samples were compiled and retested by the VIASURE test according to the manufacturer's instructions. Oropharyngeal and nasopharyngeal swabs were received from hospitalized individuals, health-care workers or symptomatic individuals in the community presenting symptoms compatible with COVID-19. All samples were collected by using either ESwab™ or UTM™ collection systems (COPAN spa, Brescia, Italy). In total, 101 cobas®-positive and -negative samples were additionally analysed with the VIASURE test. For this, 200 μL of sample transport medium were retested within 48 h (stored at 2–8°C) after sampling by using VIASURE test lots numbers NCO124-004 and NCO124-010 in Graz and Salzburg, respectively.

Samples of the Quality Control for Molecular Diagnostics (QCMD) 2020 Coronavirus Outbreak Preparedness EQA Pilot Scheme (https://www.qcmd.org/) were tested with both the VIASURE SARS-CoV-2 S gene RT-PCR test (lot no. NCO124-004) on the BD MAX™ platform and the cobas® 6800 SARS-CoV-2 RT-PCR assay (lot no.G06568) on the cobas® 6800 platform according to the manufacturers' instructions at the Graz laboratory. The panel included samples containing different concentrations of SARS-CoV-2, coronavirus NL63 and coronavirus OC43 in transport medium, as well as blank transport medium.

The study was carried out in accordance with The Code of Ethics of the World Medical Association (Declaration of Helsinki). Only pseudonymized leftover specimens from routine diagnostics were used to validate the performance of the VIASURE SARS-CoV-2 S gene Real-Time PCR Detection KIT with no informed consent sought.

Results

Performance of the VIASURE SARS-CoV-2 S gene RT-PCR Kit with samples of an external quality assurance (EQA) pilot scheme

Results obtained with the cobas® 6800 SARS-CoV-2 test were consistent with the results expected. With the VIASURE SARS-CoV-2 S gene RT-PCR Kit, all samples containing SARS-CoV-2 concentrations between 2000 and 200 000 copies/mL tested positive; however, the sample containing the lowest concentration (200 copies/mL) could not be detected.

Performance of the VIASURE SARS-CoV-2 S gene RT-PCR Kit with clinical samples

Six out of the 101 samples (5.9%) showed an inhibition (unresolved result) with the VIASURE test (five cobas®-positive samples and one cobas®-negative sample) and were excluded from further analysis. When analysing the remaining 95 clinical samples, 27 were found to be negative with both assays. From 68 samples positive with the cobas® SARS-CoV-2 test, the VIASURE test missed 21 (30.9 %) samples (Fig. 1 ). Samples found to be negative with the VIASURE kit but positive with the cobas® kit showed significantly higher cobas® 6800 cycle threshold (Ct) values than those testing positive with both of the kits (p < 0.05, t-test). The VIASURE test missed 52.5% (21/40) of samples positive with the cobas® test showing a Ct value of ≥31. All samples that tested negative with the cobas® test were also found to be negative with the VIASURE SARS-CoV-2 test.

Fig. 1

VIASURE SARS-CoV-2 S gene RT-PCR test results of COVID-19 samples diagnosed with the cobas® SARS-CoV-2 RT-PCR test. In three independent analyses, a total of 101 cobas® SARS-CoV-2 test-positive and -negative throat samples were tested with the VIASURE SARS-CoV-2 S gene test. Each dot represents a cobas® test-positive sample with the corresponding Ct value of target 1 (if target 1 was negative, target 2 is depicted). Both cobas® and VIASURE test-negative samples as well as inhibited samples in the VIASURE test are not depicted. The numbers of VIASURE test positives from cobas® test positives are shown below the corresponding dots above the abscissa. In the right upper corner of the figure an example of an amplification curve of the cobas® system of a sample that was negative in the VIASURE test is shown. The respective sample in the first analysis from Graz is marked with an asterisk. The horizontal line marks a Ct value of 31. It indicates the lowest Ct value at which conflicting test results between the VIASURE test and the cobas® test started to appear.

The false-negative results with the VIASURE test from both laboratories were communicated independently to BD. As proposed by the manufacturer, the Salzburg laboratory investigated whether sensitivity might be improved with a higher sample input volume and at the same time an adjusted higher volume for extraction according to the manufacturer's instructions. Increasing the input volume with 21 samples from 200 μL to either 500 μL or 750 μL resulted in four (two negative and two inhibited) and two (one negative and one inhibited) discrepant test results, respectively, compared with the cobas® test.

Pharyngeal swabs with cobas® 6800 SARS-CoV-2 Ct values ≥ 31 are common in individuals with COVID-19

We then aimed to evaluate the potential impact of false-negative results obtained in our compiled COVID-19 samples with the VIASURE assay (Fig. 1) for routine COVID-19 testing. All Ct values of throat swabs that had tested positive for SARS-CoV-2 RNA with the cobas® test at the Graz laboratory between 27 March and 2 April 2020 were analysed. We found that 50.9% (298/586) of positive samples showed Ct values ≥ 31 (Fig. 2 ). Those samples included 224 newly diagnosed patients. Among those were 162 newly diagnosed patients in the community and 62 newly diagnosed patients from hospitals. Seventy-four samples were derived from previously diagnosed patients. When taking into account that 52.5% (21/40) of cobas® test-positive samples with Ct values ≥ 31 were not detected by the VIASURE assay (Fig. 1), one could estimate that approximately 26.7% of our COVID-19 samples would have been missed by this assay.

Fig. 2

Distribution of Ct values of SARS-CoV-2 RNA-positive pharyngeal swabs diagnosed with the cobas® 6800 system during 1 week of daily testing in Graz. The total number of quantitative PCR-positive samples detected during 1 week in the Graz laboratory is depicted below the corresponding dots above the abscissa. The horizontal line marks a Ct value of 31. It indicates the lowest Ct value, at which conflicting test results between the VIASURE test and the cobas® test started to appear (Fig. 1).

Discussion

In contrast to SARS-CoV-1, the causative agent of COVID-19, SARS-CoV-2, replicates efficiently in the upper respiratory tract leading to high viral loads in throat swabs during the first week of symptoms [6]. However, in most individuals with mild disease the viral load rapidly declines during the second week of illness [6]. Therefore, the high proportion of throat swabs with a relatively low viral load (Ct values ≥ 31) derived from hospitalized patients and patients in the community in our study is not surprising. COVID-19 respiratory tract samples with Ct values ≥ 31 were also observed by others in different SARS-CoV-2 RT-PCR assays [7,8]. Apart from the dynamics of the viral load in a short period of time, the sampling procedure of oropharyngeal or nasopharyngeal swabs is likely to add to highly variable viral loads detected in those specimens. Therefore, highly sensitive real-time RT-PCR are most important for any case detection and preventive measures. For COVID-19 RT-PCR intended for use by health professionals, the Directive 98/79/EC devices (IVD) is currently applied [9]. According to this Directive, a CE-mark can be affixed to the test, when the manufacturer declares that the requirements of the Directive are fulfilled (declaration of conformity) without involving a notified body. The CE-marked VIASURE SARS-CoV-2 S gene Real-Time PCR Kit for use with the BD MAX™ is an automated system in which cartridges are used for the full processing of single samples [3]. According to the Instructions for Use, the clinical sensitivity and specificity of the VIASURE SARS-CoV-2 S gene RT-PCR Kit was tested using four nucleic acids isolated from SARS-CoV-2-positive respiratory samples and 15 respiratory samples from individuals with clinical suspicion of COVID-19 disease or other respiratory diseases. From this panel, four SARS-CoV-2-positive samples were detected and results were confirmed by two manual assays [3]. Direct comparison of all test results obtained with this limited number of specimens with a reference method was not reported [3].

Our study revealed a high percentage of false-negative COVID-19 samples with the VIASURE SARS-CoV-2 S gene Real-Time PCR Kit, although CE marked [9] when validated with an appropriate target population and reference method. The small sample size is a limitation of our study, but is also due to an extreme shortage of SARS-CoV-2 assay reagents during the study period. Although the VIASURE Real-Time PCR Cartridges are provided with an internal PCR inhibition control, positive controls with the SARS-CoV-2 target gene are not provided. A possible reason for missing detection may be sensitivity issues related to the S gene target, which might be less suitable for SARS-CoV-2 diagnostics or nucleic acids extraction and removal of inhibitors. The latter two being essential for the reliability of results in routine molecular diagnostics [10]. Sensitivity issues are also of major importance when it comes to pooling of samples for SARS-CoV-2 testing. As reported recently, pooling affects the sensitivity leading to an increase of the Ct value and the false-negative rate [11].

Given the high numbers of newly CE-marked COVID-19 RT-PCR assays, the European Commission very recently communicated guidelines on COVID-19 in vitro diagnostic tests and their performance [12]. Although the Directive 98/79/EC defines a number of test evaluation data that have to be provided by the manufacturer to document test performance, the European Commission recommends carrying out additional clinical validation of COVID-19 diagnostic tests by comparison with a reference method including scientific peer review [12]. The results of our study strongly support this approach before introducing a COVID-19 test into clinical routine.

Transparency declaration

The authors have stated that there are no conflicts of interest.

Funding

None.

Authors' contribution

LM conceptualized the laboratory work, conducted laboratory tests, analysed data, prepared the figures and wrote the manuscript; EL conceptualized the laboratory work and analysed data; ES analysed data; KA analysed data and prepared the figures; DS and MB conducted laboratory tests; MM conceptualized the laboratory work, supervised laboratory tests and analysed data; IS conceptualized the overall study, analysed data and wrote the first manuscript draft; HK analysed data and edited the manuscript.