Evaluation of the diagnostic performance of nine commercial RT-PCR kits for the detection of SARS-CoV-2 in Colombia.

The ongoing severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic has led to the design and development of multiple reverse-transcription polymerase chain reaction kits aimed to facilitate the rapid scale-up of molecular testing for massive screening. We evaluated the diagnostic performance of nine commercial kits, which showed optimal performance and high discriminatory power. However, we observed differences in terms of sensitivity, specificity, and E gene Ct Values and discuss these results in light of the influence of SARS-CoV-2 genetic variability and its potential impact in current molecular diagnostic assays.

diagnostic precision

negative likelihood ratio.

positive likelihood ratio

negative predictive value

positive predictive value

reverse‐transcription polymerase chain reaction

INTRODUCTION

Similar to other human coronaviruses, the severe acute respiratory syndrome coronavirus 2 (SARS‐CoV‐2) virus genome is composed of a positive‐sense single‐stranded RNA of around 30,000 nucleotides (30 kb) in size. This fairly large genome consists of two segments: (i) a large segment formed by two open reading frames (ORF1a and ORF1b) that are translated into two polyproteins resulting in 16 nonstructural proteins (NSP), including the RNA dependent RNA polymerase (RdRp) and (ii) a shorter segment that encodes for structural proteins such as spike (S), envelope (E), membrane (M), and nucleocapsid (N) as well as other accessory proteins. Many of these structural and nonstructural coding genes have been used as diagnostic targets in a variety of nucleic acid amplification‐based tests including reverse‐transcription polymerase chain reaction (RT‐PCR) as well as other amplification formats such as isothermal‐based assays.

Given its sensitivity, specificity, and practicability as compared to viral culture, testing for SARS‐CoV‐2 infection relies predominantly on RT‐PCR as recommended by the World Health Organization. In fact, to date, RT‐PCR‐based assays account for approximately 77.2% of all nucleic acid amplification tests authorized by the Food Drugs Administration (FDA) under emergency use authorization. With the arrival of the second pandemic wave and the worldwide emergence of new “variants of concern and interest,” RT‐PCR has become a powerful tool for surveillance and control of coronavirus disease 2019 (COVID‐19). Moreover, massive testing has expanded its use beyond high‐complexity laboratory settings to different healthcare centers as well as epidemiological and clinical research facilities.3, 5, 6

So far, 381 commercial kits have been developed and commercialized worldwide (FIND). A number of these have already been endorsed by the CE‐IVD and FDA and are currently available in the market. These kits test for different molecular target regions and are designed based on the SARS‐CoV‐2 genome and assembly characteristics (fluorophores and preservation reagents). Due to genomic variability and variable performance between viral targets, some of these kits target various regions across the viral genome to ensure redundancy and improve sensitivity. Although, information on the analytical performance is available for most, characteristics regarding the diagnostic performance of these kits are still scarce and, therefore demands independent assay evaluations before massive diagnostic implementation. In Colombia, the number of laboratories authorized to perform SARS‐CoV‐2 molecular testing increased from 22 in April 2020 to 162 currently. Hence, it was essential to evaluate the diagnostic performance of various RT‐PCR kits. This study aims to provide a reliable comparison of the diagnostic performance of nine RT‐PCR kits from different manufacturers (most frequently used in Colombia) to evaluate the impact of its potential use for massive screening while considering the genomic variation landscape of SARS‐CoV‐2 in the country. Nine commercial kits were included in the study, registered on the FIND web page. Characteristics of the evaluated kits are depicted in Table S1.

METHODS

The size of positive and negative clinical samples necessary for evaluation of the diagnostic performance was calculated, using the sensitivity and specificity values previously reported for each kit (Table S1). Runs of at least 94 samples (49 positive and 45 negative) and two controls (positive and negative controls) were included for each kit.

The samples included in this study were collected between June and September 2020. Due to the emergency and urgent need to evaluate the diagnostic performance of the kits, the runs were made within the daily routine of the microbiology laboratory of the Universidad del Rosario in which 1200 samples of nasopharyngeal swabs were processed daily in 2020. For this reason, three panels of 94 samples were used (Table S1). The samples included for testing were processed avoiding freeze/thaw and ran side‐by‐side with the reference method. The standard reference test was based on the detection of the E gene using the primers and probe described in the Berlin Charité protocol as recommended elsewhere. Each of the kits was tested following the manufacturer's instructions.

The operating characteristics of the molecular tests were estimated by comparing against standard diagnosis (E gene amplification of the Berlin‐Charité protocol). Sensitivity, specificity, positive (LR+) and negative likelihood ratio (LR−), predictive values (PV), diagnostic precision (DP), and Kappa index (K) were estimated for each kit. The Ct values ​​for the E gene (reference method) were compared with the amplification performance of the commercial kits (estimating area under the curve [AUC]). Shapiro–Wilk was used as the normality distribution test. Due to the overdispersion of Ct values of E gene, medians and quartiles are presented by box plots, comparisons are based on the Mann–Whitney test between E gene Ct values from reference test and E gene Ct values from commercial RT‐PCR kits. A p value at less than 0.05 was considered statistically significant. All analyses were performed using R software.

RESULTS

The operating characteristics for each commercial RT‐PCR assay are presented in Table 1. The results obtained for all commercial kits were optimal in terms of sensitivity and specificity. Three kits (QuantuMDx, Inbios, JN Medsys) showed sensitivity values ​​higher than 95.0% (Table 1). E gene amplification performance was compared between GeneFinder, Seegene, Inbios, and PCL assays in parallel to the reference assay (Figure 1A–D). The GeneFinder (AUC: 0.97), Inbios (AUC:0.95), and PCL (AUC: 0.92) kits were found to have outstanding performance, with Seegene displaying an excellent performance. The mean Ct values for the E gene were compared in all samples analyzed (Figure 1E–H).

Table 1

Operating characteristics of commercial RT‐PCR kits for SARS‐CoV‐2 diagnosis

Commercial kits		Operating characteristics
Name of the kit	Manufacturer	Sensitivity	Specificity	PPV	NPV	DP	LR +	LR−	Kappa index
		(95% CI)	(95% CI)	(95% CI)	(95% CI)	(95% CI)	(95% CI)	(95% CI)	(95% CI)
QuantuMDx SARS CoV 2 RT PCR Detection Assay	QuantuMDx	100.00	95.56	96.08	100.00	97.87	22.5	Undefined	0.9573
									(0.7553–1.159)
		(92.73–100.00)	(85.17–98.77)	(86.78–98.92)	(91.8–100.00)	(92.57–99.41)	(8.444–59.95)
GeneFinder™ COVID‐19 Plus RealAmp Kit	GeneFinder	93.88	100.00	100.00	93.75	96.81	Undefined	0.06122	0.9362
								(0.03186–0.1177)	(0.7345–1.138)
		(83.48–97.9)	(92.13–100.00.0)	(92.29–100.00)	(83.16–97.85)	(91.03–98.91)
Allplex™ 2019‐nCoV Assay	Seegene	87.76	100.00	100.00	88.24	93.62	Undefined	0.1224	0.8728
		(75.76–94.27)	(92.13–100.00.0)	(91.8–100.00)	(76.62–94.5)	(86.77–97.04)		(0.08833–0.1698)	(0.6723–1.073)
MiRXES Fortitude Kit 2.1	MiRXES	87.76	97.78	97.73	88.00	92.55	39.49	0.1252	0.8515
		(75.76–94.27)	(88.43–99.61)	(88.19–99.6)	(76.19–94.38)	(85.42–96.35)	(5.527–282.1)	(0.09024–0.1738)	(0.6504–1.052)
Coronavirus COVID‐19 genesig® Real‐Time PCR assay	Genesig	93.88	97.78	97.87	93.62	95.74	42.24	0.06262	0.9149
		(83.48–97.9)	(88.43–99.61)	(88.89–99.62)	(82.84–97.81)	(89.56–98.33)	(5.934–300.7)	(0.03255–0.1205)	(0.7129–1.117)
Novel Coronavirus (2019‐nCoV) Nucleic Acid Diagnostic Kit	Sansure	94.83	91.67	94.83	91.67	93.62	11.38	0.05643	0.8649
		(85.86–98.23)	(78.17–97.13)	(85.86–98.23)	(78.17–97.13)	(86.77–97.04)	(5.909–21.91)	(0.0292–0.109)	(0.6628–1.067)
Smart Detect™ SARS‐CoV‐2 rRT‐PCR Kit	Inbios	96.55	100.00	100.00	95.0	97.92	Undefined	0.03448	0.9568
		(88.27–99.05)	(90.82–100.00)	(93.58–100.00)	(83.5–98.62)	(92.72–99.43)		(0.01294–0.09188)	(0.757–1.157)
ProTect™ COVID 19 PCR Kit	JN Medsys	100.00	78.95	87.88	100.00	91.67	4.75	Undefined	0.8192
									(0.6225–1.016)
		(93.79–100.00)	(63.65–88.93)	(77.86–93.73)	(88.65–100.00)	(84.41–95.72)	(3.718–6.069)
PCL COVID19 Speedy RT‐PCR	PCL	95.74	94.44	95.74	94.44	95.18	17.23	0.04506	0.9019
		(85.75–98.83)	(81.86–98.46)	(85.75–98.83)	(81.86–98.46)	(88.25–98.11)	(6.456–46.01)	(0.01685–0.1205)	(0.6868–1.117)

Note: When the specificity is 100% the positive likelihood ratio is undefined. When the sensitivity is 100% the negative likelihood ratio is undefined.

Abbreviations: DP, diagnostic precision; LR+, positive likelihood ratio; LR−, negative likelihood ratio; NPV, negative predictive value; PPV, positive predictive value; RT‐PCR, reverse‐transcription polymerase chain reaction; SARS‐CoV‐2, severe acute respiratory syndrome coronavirus 2.

Figure 1

E gene diagnosis performance and Ct values comparison between reference assay and commercial RT‐PCR kits. (A–D) ROC curves of RT‐PCR kits for SARS‐CoV‐2 diagnosis (A) GeneFinder™ COVID‐19 Plus RealAmp Kit (GeneFinder) (B) Allplex™ 2019‐nCoV Assay (Seegene) (C) Smart Detect™ SARS‐CoV‐2 rRT‐PCR Kit (Inbios) (D) PCL COVID19 Speedy RT‐PCR (PCL). Green line: Reference test. Red line: commercial test (E–H) E gene Cycle threshold values comparison (E) GeneFinder™ COVID‐19 Plus RealAmp Kit (GeneFinder) (F) Allplex™ 2019‐nCoV Assay (Seegene) (G) Smart Detect™ SARS‐CoV‐2 rRT‐PCR Kit (Inbios) (H) PCL COVID19 Speedy RT‐PCR (PCL). The outliers were removed from the graph for convenience. *p < 0.05. ROC, receiver operating characteristic; RT‐PCR, reverse‐transcription polymerase chain reaction; SARS‐CoV‐2, severe acute respiratory syndrome coronavirus 2

DISCUSSION

Of all assays evaluated, the Seegene and Sansure kits have been previously assessed in terms of their analytical and diagnostic performance13, 14, 15, 16 as well as sample pooling. Diagnostic performance results obtained for both of these kits in previous studies are in concordance with those observed in our current study.13, 14, 16

Different studies comparing the analytical performance of diverse viral targets included in the detection of SARS‐CoV‐2, based on the amplification of the E gene using the Berlin Charité protocol have demonstrated a higher sensitivity of this specific target when compared with the N and Rd/Rp targets.10, 18, 19 This translates into a higher efficacy in terms of analytical performance making it a suitable target for screening tests, as observed with the excellent performance of the E‐gene‐based commercial kits GeneFinder, Inbios, PCL, and Seegene (Figure 1A–D) assessed in this study.

Along these lines, the Genefinder and PCL kits showed lower Ct medians compared with the reference test (with significant statistically differences) (Figure 1E–H), suggesting a higher sensitivity as reflected also by the limits of detection (Table S1). In contrast, the Inbios and Seegene kits did not reveal any differences with respect to the reference test for which detection limits of 100 copies per reaction, 316 genomic equivalents per reaction, and 625 copies per ml have been reported.18, 19, 20

In conclusion, as previously described by in silico analyses, genomic variability of SARS‐CoV‐2 could affect the diagnostic accuracy of currently available diagnostic tests, particularly in the context of emerging variants.2, 21 Despite inherent variableness in diagnostic performance, all the RT‐PCR kits evaluated in this study were found suitable for SARS‐CoV‐2 genomic detection in Colombia. However, in those scenarios where highly sensitive detection of SARS‐CoV‐2 is required, any of the E‐gene inclusive kits (GeneFinder, Seegene, Inbios, and PCL) have proved to offer a potential advantage for improving test sensitivity as shown in this study. Continued monitoring and a multi‐target approach are needed to prevent the effects of genetic variability on test sensitivity.

CONFLICT OF INTERESTS

The authors declare that there are no conflict of interests.

Supplementary information.

Click here for additional data file.