Assessment of 12 qualitative RT-PCR commercial kits for the detection of SARS-CoV-2.

The emergence of the novel coronavirus, the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in the late months of 2019 had the officials to declare a public health emergency leading to a global response. Public measurements rely on an accurate diagnosis of individuals infected with the virus by using real-time reverse transcriptase-polymerase chain reaction (RT-PCR). The aim of our study is to relate the fundamental clinical and analytical performance of SARS-CoV-2 (RT-PCR) commercial kits. A total of 94 clinical samples were selected. Generally, 400  µl of each respiratory specimen was subjected to extraction using ExiPrep 96 Viral RNA Kit. All kits master mix preparation, cycling protocol, thermocycler, and results interpretation were carried out according to the manufacturer's instructions of use and recommendations. The performance of the kits was comparable except for the LYRA kit as it was less sensitive (F = 67, p < .001). Overall, four kits scored a sensitivity of 100% including: BGI, IQ Real, Sansure, and RADI. For specificity, all the tested kits scored above 95%. The performance of these commercial kits by gene target showed no significant change in CT values which indicates that kits disparities are mainly linked to the oligonucleotide of the gene target. We believe that most of the commercially available RT-PCR kits included in this study can be used for routine diagnosis of patients with SARS-CoV-2. We recommend including kits with multiple targets in order to monitor the virus changes over time.

European Conformity‐In Vitro Diagnostic; FDA

cycling threshold

US Food and drug Administration (Emergency Use Authorization)

envelope gene

limit of detection

nucleocapsid

large open reading frame

RNA‐dependent RNA polymerase

real‐time reverse transcriptase polymerase

Research Use Only

severe acute respiratory syndrome coronavirus 2

Spike gene

INTRODUCTION

The emergence of the novel coronavirus, the severe acute respiratory syndrome coronavirus 2 (SARS‐CoV‐2) in the late months of 2019 had the officials to declare a public health emergency leading to a global response. The new virus (SARS‐CoV‐2) has been identified as a new strain of Betacoronvirus from group 2B with approximately 70% genetic similarity to the severe acute respiratory syndrome coronavirus (SARS‐CoV). Coronaviruses are positive‐sense RNA viruses that have 14 open reading frames (ORFs) encoding structural and nonstructural proteins (Nsps). Structural proteins include spike (S), envelope (E), membrane (M), nucleocapsid (N) proteins plus eight accessory proteins. The ORF1ab encodes 15 nonstructural proteins including the RNA‐dependent RNA polymerase (RdRP) protein which is important for virus replication and transcription. In response to the burden of the current pandemic, strict measurements were globally implemented to stop further transmission of the virus. These measurements rely on an accurate diagnosis of individuals infected with the virus by using real‐time reverse‐transcriptase polymerase chain reaction (RT‐PCR). The most common targets for detecting SARS‐CoV‐2 by (RT‐PCR) assays in diagnostic laboratories are the ORF1ab gene, the RdRP gene, the E gene, the N gene, and the S gene.

The SARS‐CoV‐2 pandemic caused a huge burden on the societal, financial, and healthcare systems in the sphere, and various measures were implemented to control its spread. Most of the control measures mainly depend on the precise testing of the individuals infected by the virus. The real‐time RT‐PCR method of detection is the most common and reliable test in detecting viral genome, therefore the world health organization (WHO) has recommended the use of this method as the gold standard during the current time. Many SARS‐CoV‐2 RT‐PCR diagnostic kits have become commercially available in the market; however, the evaluation of most of the kits' performance is not available to the public sector.

In the Saudi Center for Disease Prevention and Control (SCDC) Laboratories, we have performed an evaluation of twelve RT‐PCR SARS‐CoV‐2 commercial kits from various manufacturers. For this study, most of selected kits were European Conformity in vitro diagnostic medical devices certified (CE‐IVD). A concise panel of 94 clinical samples was used to evaluate the performance of these commercial kits. The goal of our study is to relate the fundamental clinical and analytical performance of chosen kits of RT‐PCR from distinct manufacturers. The manufacturers enrolled in our assessment were TIB MOLBIOL, Altona Diagnostics, Thermo Fisher Scientific, Solgent, Quidel, BGI, OPTOLANE Technology, Kogene Biotech, Sansure Biotech, Novacyt/Primer design, GeneReach Biotechnology, and KH Medical. The 12 commercial kits amplified five unique targets in the SARS‐CoV‐2 genome, including: N, E, S, RdRP, and ORF1ab/PP 1ab genes. This study will provide an abundant assessment on the detection of SARS‐CoV‐2 by using different kits and targets to show the importance of accurate testing during these difficult times.

METHODOLOGY

Samples selection

This study encompasses 12 commercial RT‐PCR SARS‐CoV‐2 detection kits available in the market, sent by the manufacturers free of charge and not for marketing purposes. It is provided to the SCDC for evaluation, approval, and accreditation to use in medical laboratories in Saudi Arabia. None of the manufacturers were involved in the assessment or data analysis. A total of 94 nasopharyngeal and oropharyngeal clinical samples were selected with variation in their cycling threshold (CT) values. The ethical approval was obtained from our institution the Saudi CDC Number: (SCDC‐IRB‐A012‐2020).

Samples preparation and RT‐PCR procedure

Generally, 400 μl of each respiratory specimen was extracted using ExiPrep 96 Viral RNA Kit, and ExiPrep 96 Lite Automated NA Purification System (Bioneer). The nucleic acid (RNA) extraction process was performed twice, with 100 μl elution volume for each, then the extracted RNA was pooled and stored at −80°C until use. No exogenous internal controls were added to the extraction. The kits with endogenous internal control such as (BGI, RADI & Sansure) were used to check extraction success. Real‐Time PCR amplification process was monitored by master mix internal controls. All the kit's master mix preparation, cycling protocol, thermocyclers and results interpretation were carried out according to the manufacturer's instructions of use and recommendations (details of the compared kits and instruments used are summarized in Table 1). When the result was inconclusive or invalid, as per the manufacturer's results interpretation, retesting was performed, and no PCR inhibition was noted.

Table 1

Overview of the SARS‐CoV‐2 detection kits encompasses in this study

Company	Kit	Country	Regulatory status	Gene target	RNA template vol.a	Reaction vol.a	Thermocyclera	Cycling timea	Positivity cut off (CT value)a
TIB MOLBIOL	LightMix Modular Wuhan CoV E gene & RdRP gene + Multiplex RNA Master (Roche Diagnostics	Germany	RUOa	E & RdRP	5 µl	10 µl	LightCycler 480 II Instrument (Roche)	1 h	<39
Altona	RealStar SARS‐CoV‐2	Germany	FDA (EUA)b	E & S	10 µl	20 µl	LightCycler 480 II Instrument (Roche)	2 h	ND
Diagnostics
			CE‐IVDc
Thermo Fisher Scientific	TaqPath COVID‐19 RT‐PCR Kit + TaqPath 1‐Step Multiplex Master Mix (Applied Biosystems)	USA	CE‐IVD	N, Orf1b &S	5 µl	20 µl	ABI 7500 Fast (Applied Biosystems)	1 h:10 min	≤37
Solgent	DiaPlexQ Novel Coronavirus (2019‐nCoV) Detection Kit	Korea	FDA (EUA) CE‐IVD	N & Ofr1a	5 µl	20 µl	ABI 7500 Fast (Applied Biosystems)	2 h	≤40
Quidel	Lyra SARS‐CoV‐2 assay	USA	CE‐IVD	Orf1ab (PP1ab)	5 µl	15 µl	LightCycler 480 II Instrument (Roche)	1 h:20 min	≤40
BGI	BGI Real‐Time Fluorescent RT‐PCR kit for SARS‐CoV‐2	China	FDA (EUA) CE‐IVD	Orf1ab	10 µl	20 µl	LineGene 9600 Plus (BIOER)	1 h:45 min	≤38
OPTOLANE	KAIRA 2019‐nCoV Detection Kit	Korea	CE‐IVD	E & RdRP	5 µl	15 µl	ABI 7500 Fast (Applied Biosystems)	1 h:10 min	≤36/≤37.5
Technology
Kogene	PowerChek™ 2019‐nCoV Real‐time PCR Kit	Korea	FDA (EUA	E & RdRP	5 µl	15 µl	ABI 7500 Fast (Applied Biosystems)	2 h	≤37
Biotech			CE‐IVD
Sansure Biotech	Sansure COVID‐19 Nucleic Acid Test Kit	China	FDA (EUA)	N &Orf1ab	20 µl	30 µl	ABI 7500 Fast (Applied Biosystems)	2 h	≤40
Novacyt/Primer design	Genesig coronavirus COVID‐19 Real‐Time PCR Assay	UK	FDA (EUA)	RdRP	8 µl	12 µl	LightCycler 480 II Instrument (Roche)	1 h:20 min	ND
			CE‐IVD
GeneReach Biotechnology	IQ REAL SARS‐CoV‐2 Qualitative System	Taiwan	NDd	Orf1ab	2 µl	23 µl	ABI 7500 Fast (Applied Biosystems)	1 h:40 min	ND
KH Medical	RADI COVID‐19 Detection Kit	Korea	CE‐IVD	S & RdRP	15 µl	15 µl	ABI 7500 Fast (Applied Biosystems)	1 h	≤40

Abbreviations: CE‐IVD, European Conformity‐In Vitro Diagnostic; COVID‐19, coronavirus disease 2019; E, envelope gene; FDA (EUA), U.S Food and drug Administration (Emergency Use Authorization); N, nucleocapsid gene; ND, not determined; ORF, an open reading frame; RdRP, RNA‐dependent RNA polymerase gene; RUO, Research Use Only; S, spike gene; SARS‐CoV‐2, severe acute respiratory syndrome coronavirus 2.

Variables were according to manufacturers.

Data analysis

For statistical analysis, data was collected and analyzed using GraphPad Prism, version 8.4. Descriptive analysis was done on the reported CT and results were compared by the commercial RT‐PCR kits and targets. Analysis of variance (ANOVA) test was used to detect the significance of the CT values reported by the commercial RT‐PCR kits and targets. Box plots and bar graphs were used to show the distribution of CT values and detection of results by the different commercial kits. For sensitivity and specificity analysis, samples were considered positive by the gold standard if the majority (>6 kits) agreed on the result. The sensitivity and specificity were calculated according to the Trevethan 2017 reference study. The golden standard result was used conducting kappa agreement tests between the kits. All p values reported are two‐sided and were considered to be statistically significant at alpha less than .05.

RESULTS

The summary of the SARS‐CoV‐2 detection by multiple kits

The 94 samples were all processed, and the results were interpreted as recommended by each of the kits’ manufacturer recommendations. Kits with single targets were reported positive if their CT were below the cutoff value. All samples were tested by all kits except for the RADI, we only had enough for 55 samples, therefore we selected the enrolled samples randomly. The summary of the detection results is shown in Figure 1. Multiple diagnostic kits show that almost 60% of the samples were positive by most of the commercial kits except for LYRA as it detected 44% as positive samples. The majority of the Kits (>6 kits) agreed on 63 positive and 31 negative samples.

Figure 1

The summary of the SARS‐CoV‐2 detection results by the 12 commercial diagnostic kits. The bar graph shows a similar proportion by the kits, except for the LYRA kit, as it shows more negative results than positive. SARS‐CoV‐2, severe acute respiratory syndrome coronavirus 2

Positive samples by CT values

The qualitative analysis of the positive samples’ CT values showed a significant difference across the used kits (ANOVA, F = 67, p value < .001). Figure 2 shows the distribution of the CT values reported by the commercial kits and their targets. The lowest CT values were mostly reported with KAIRA (E and RdRP gene target), PowerCheck (RdRP gene target), and TaqPath (N gene target). The highest positive CT values were mostly reported with Genesig (RdRP gene target) PowerCheck (E gene target), and IQ REAL (ORF1ab gene target).

Figure 2

Box plot of the positive samples’ cycling threshold (CT) values across the 12 diagnostic kits (gene targets). The whisker above the box plot represents the upper limit of the 95% confidence interval, while the bottom line represents the lower limit of the 95% confidence interval, the red line represents the cutoff value reported in the kit user guide. The lowest CT values were mostly reported with KAIRA (E gene target), Sansure (N gene target), and TaqPath (N gene target). The highest positive CT values were mostly reported with PowerCheck (RdRP gene target), LightMix (E gene target), and Genesig (RdRP gene target). ANOVA test was significant by the CT value reported by the multiple kits. ANOVA, analysis of variance

Qualitative analysis of the positive samples by targets

By target, commercial kits showed similar performance and reported comparable CT values. The commercial kits had 5 different gene targets which were N, E, S, RdRP, and ORF1ab/PP1ab. The results of the ANOVA test showed that by target the CT values of the commercial kits were similar (F = 1.1, p < .05). Figure 3 shows the distribution of CT values by the target. Overall, the lowest CT values were mostly reported with kits targeting the N gene. The highest CT values were mostly reported with kits targeting the RdRP gene. Table 2A shows the overall CT summary reported by kits targeting the ORF1ab gene which were BGI, Sansure, IQ REAL, DiaPlexQ, LYRA, and TaqPath. For the ORF1ab gene target, the qualitative analysis showed a similar result across the different kits (ANOVA, F = 1, p value = .39). Table 2B shows the overall CT summary reported by kits targeting the RdRP gene which were KAIRA, LightMix, Genesig, RADI, and PowerCheck. For the RdRP gene target, the kits showed a significant difference in CT values reported by the kits (ANOVA, F = 2, p value = .03). The change was detected with the Genesig kit which reported distinctive CT values compared with KAIRA, LightMix, and PowerCheck. Table 2C shows the overall CT summary reported by kits targeting the E gene which were reported with KAIRA, LightMix, RealStar, and PowerCheck. For the E gene target results, the kits showed a significant difference in CT values reported by the kits (ANOVA, F = 20, p value < .001). The change was detected with KAIRA and RealStar CT values which were different compared to LightMix, PowerCheck, and each other. Table 2D shows the overall CT value summary by kits targeting the N gene which were reported with TaqPath, DiaPlexQ, and Sansure. For the N gene target CT values’ analysis, the kits showed no significant difference in CT values reported by the kits (ANOVA, F = 1.27, p value = .27). Similar results were reported with the S gene target as shown in Table 2E, no significance was detected (ANOVA, F = 1.75, p value = .18).

Figure 3

Box plot of the positive samples cycling threshold (CT) values across the testing kits targets. The whisker above the box plot represents the 95% upper confidence interval, while the bottom line represents the lower 95% confidence interval. The lowest CT value was reported with kits targeting the N gene. The highest CT values were mostly reported with kits targeting the RdRP gene

Table 2

Commercial Kits detecting SARS‐CoV‐2 gene targets by CT values summary

A. Commercial kits targeting ORF1ab/PP1ab gene
	BGI	Sansure	IQ REAL	DiaPlexQ	TaqPath	LYRA
No. samples	64	62	63	64	60	42
Mean	25.39	24.7	27.40	25.50	24.07	29.74
Std. deviation	7.39	6.12	6.36	6.99	7.32	5.96
Global ANOVA test	F  = 0.45, p value = .76

Abbreviations: ANOVA, analysis of variance; CT, cycling threshold.

Significant p value less than .05.

Significant p value less than .005.

Significant p value less than .001.

Sensitivity and specificity of the commercial kits

The summary of the sensitivity and specificity tests for the 12 commercial kits is shown in Table 3 based on 63 positive samples and 31 negative samples. Overall, four kits scored a sensitivity of 100% including: BGI, IQ Real, Sansure, and RADI. The lowest sensitivity was observed with the LYRA kit with 66%. For specificity, all the tested kits scored above 95%. For the Kappa agreement tests, the highest score was 100% and was observed with IQ Real and RADI kits, the other kits scored an agreement above 90% except for LYRA which had the poorest agreement with 57%.

Table 3

Sensitivity and specificity statistical summaries for the performance of 12 commercial kits

Commercial Kits	Sensitivity (95% CI)	Specificity (95% CI)	Fisher test (p value)	Kappa (95% CI)
1. BGI	100 (94%–100%)	97 (83%–99%)	<.0001	0.97 (0.93–1)
2. RealStar	95 (86%–98%)	100 (89%–100%)	<.0001	0.93 (0.85–1)
3. Genesig	95 (86%–98%)	100 (89%–100%)	<.0001	0.93 (0.85–1)
4. IQ Real	100 (94%–100%)	100 (94%–100%)	<.0001***	1 (NA)
5. KAIRA	98 (91%–99%)	97 (84%–99%)	<.0001***	0.95 (0.89–1)
6. PowerCheck	97 (89%–99%)	97 (83%–99%)	<.0001***	0.95 (0.89–1)
7. Sansure	100 (94%–100%)	97 (83%–99%)	<.0001***	0.98 (0.92–1)
8. DiaPlexQ	98 (91%–99%)	100 (89%–100%)	<.0001***	0.98 (0.92–1)
9. TaqPath	95 (86%–98%)	100 (89%–100%)	<.0001***	0.93 (0.85–1)
10. LightMix	95 (86%–98%)	100 (89%–100%)	<.0001***	0.93 (0.85–1)
11. LYRA	66.6 (54%–78%)	100 (89%–100%)	<.0001***	0.57 (0.4–0.7)
12. RADI	100 (88%–100%)	100 (88%–100%)	<.0001***	1 (NA)

Abbreviation: CI, confidence interval.

Significant p value less than .001.

DISCUSSION

During the current stage of the coronavirus disease 2019 (COVID‐19) pandemic, many testing kits were developed and available commercially, in our study we were able to evaluate the performance of 12 commercial kits in detecting the SARS‐CoV‐2 virus. The performance of the kits was comparable except for the LYRA kit as it was less sensitive. Particularly, the performance of BGI, IQ Real, Sansure, and RADI kits were the most sensitive. These kits had more than one target except for IQ Real and BGI. This finding indicates the importance of having a confirmatory gene to ensure the sensitivity of diagnostic testing. For specificity, only four commercial kits scored less than 100%, these four were BGI, KAIRA, PowerCheck, and Sansure, all of them had multiple targets except for BGI. This finding indicates that increasing the number of targets is not necessarily needed for high specificity however, the designed oligo has the biggest influence. Moreover, the performance of these commercial kits by gene target showed no significant change in CT values which indicates that kits disparities are mainly linked to the choice of the gene target. From the five targets, ORF1ab/PP1ab, S, and N genes reported similar results by the different kits, however, RdRP and E gene targets showed significant differences by the reported CT values. In consideration of time, the shortest tests were KAIRA, LightMix, and RADI with 60 min, however, these kits did miss few positive samples.

Indeed, other studies evaluating the performance of RT‐PCR kits are in concordance with our results. Comparing commercial kits showed similar results in detection, however, different targets did show variation in CT values. Moreover, kits with multiple targets such as Realstar, Taqpath, LightMix, and Sansure in another evaluation study did show higher sensitivity and specificity than with other kits detecting a single target. In our study, we did not evaluate the limit of detection (LOD) which can play a big role, as with several of the commercial kits approved for the pandemic situation many were approved without evaluation with appropriate numbers of samples. Another important variation source in many PCR kits results that we did not manage to evaluate is variation in instrument and technicians, in a multiple center euro surveillance study, the same commercial kits showed variation in results between different centers, many of these centers had different instruments and technicians. Overall, the variation in CT values can rise from multiple areas such as extraction methods, the gene target, the oligo design, and technical aspects linked to reproducibility.

One of the biggest challenges in the diagnosis of SARS‐CoV‐2 in clinical settings is the high rate of false‐negative cases. As known with RNA viruses the mutation rate is high compared with other pathogens. Moreover, the nucleotide mutation rate reported for SARS‐CoV‐2 was 8E‐04 substitution per site per year. , Many studies have already shown the high evolution rate of SAR‐CoV‐2 in many cases and linked its evolution with its origin. Detection methods using RT‐PCR are based on a fixed target, however, as the pandemic proceeds the virus is changing and many cases are missed. In one study, evaluating multiple RT‐PCR targets has shown a high loss of sensitivity, with a total of 11,627 cases missed due to variations in genetic code. In one of the biggest studies worldwide tracking the mutations of SARS‐COV‐2, one mutation D614 was decreasing and another mutation G614 mutation was growing, the new mutation has lower CT values which indicate a change in virus virulence. In a genomic surveillance study, a rise in mutation located in the ORF gene was linked to the Middle East SARS‐CoV‐2 cases. Overall, these studies indicate a high number of genetic mutations in SARS‐CoV‐2 globally.

This rise will cause many positive cases to be missed in the RT‐PCR, which will cause a challenge with using these fixed commercial RT‐PCR kits. In one case for a systematic SARS‐CoV‐2 case, the patient was negative in many RT‐PCR tests, however, he was only positive by the antibody test, moreover, the case was later investigated with sequencing and the virus infecting the patient had two major mutations one located at the NP genes and another located at the ORF gene. The escalation of these different mutations in SARS‐CoV‐2 at multiple geographical locations indicates the need for genetic screening periodically in each country to count for these changes as they may significantly play a role in choosing the commercial diagnostic kit used at the testing centers. Furthermore, the importance of revealing the sequence and design of the oligonucleotides in commercially available kits’ description is essential at this time as the virus keeps spreading and evolving.

We are aware that our study may have several limitations. The first one is that we were not able to study the cross‐reactivity of these commercial kits with other viruses which can significantly alter the results of the detection. Even though our conducted study implemented a comparative evaluation it is still not a comprehensive study as we could not include all RT‐PCR diagnostic kits available at the market and the performance of some kits may vary depending on the extraction method. By changing the extraction method, some results may vary. Another area of limitation in our study is our sample size which is limited to the number of tests per kit. In our future work, we hope to evaluate genetic changes in the virus and its effect on detecting the virus by the available commercial kits.

CONCLUSION

In conclusion, we believe that most of the commercially available RT‐PCR kits included in this study can be used for routine diagnosis of SARS‐CoV‐2 patients. Most of the kits succeeded in detecting the virus, however, few distinctions were found with specific kits and targets. Kits with multiple targets have been shown to be more sensitive and specific as they counted for target variations. Moreover, we recommend that regardless of the laboratory choice of diagnostic commercial kit for the clinical detection of patients with COVID‐19, the need for a good plan for validation and collaboration with exterior laboratories is essential to monitor the changes in the virus, procedures, technicians, and the different kits performances.

AUTHOR CONTRIBUTIONS

All authors contributed to the manuscript equally. AT conducted and planned the experiments. DO ran the analysis and wrote and reviewed the manuscript. TA wrote the methodology section, and MT wrote the introduction section. FA, MA helped with carrying the experiments. AA reviewed the manuscript.

CONSENT TO PUBLICATION

All authors have reviewed the final version of the manuscript and approve it for publication.

ETHICS APPROVAL AND CONSENT TO PARTICIPATE

Ethical approval was obtained from the Saudi Center for Diseases Control and prevention ethical committee under the department of public health research and statistics division, (IRB number: SCDC‐IRB‐A012‐2020). Participant consent was waived.

PEER REVIEW

The peer review history for this article is available at https://publons.com/publon/10.1002/jmv.26900.