Analytical performance evaluation of five RT-PCR kits for severe acute respiratory syndrome coronavirus 2.

BACKGROUND: We aimed to evaluate the analytical performance of five commercial RT-PCR kits (Genekey, Daan, BioGerm, Liferiver, and Yaneng) commonly used in China, since such comparison data are lacking.
METHODS: A total of 20 COVID-19 confirmed patients and 30 negative nasopharyngeal swab specimens were analyzed by five kits. The detection ability of five RT-PCR kits was evaluated with 5 concentration gradients diluted by a single positive sample. The limit of detection was evaluated by N gene fragment solid standard. Two positive clinical specimens were used to evaluate the repeatability and imprecision. Finally, we used six human coronaviruses plasmid and four respiratory pathogens plasmid to check for cross-reactivity.
RESULTS: The positive detection rate was 100% for Genekey, Daan, and BioGerm,and 90% for Liferiver and Yaneng in 20 clinical SARS-CoV-2 infection. The coincidence rate of five kits in 10 negative samples was 100%. The detection rate of target genes for Daan, BioGerm, Liferiver, and Yaneng was 100% from Level 1 to Level 3. In Level 4, only Daan detection rate was 100%. In Level 5, five kits presented poor positive rate. The limit of detection declared by each manufacturer was verified. The repeatability for target genes was less than 5% and so did the total imprecision. There is no cross-reactivity of five kits with six human coronaviruses and four respiratory pathogens for ORF1ab and N gene.
CONCLUSIONS: Five RT-PCR kits assessed in this study showed acceptable analytical performance characteristics and are useful tools for the routine diagnosis of SARS-CoV-2.

INTRODUCTION

Coronavirus disease 2019 (COVID‐19), caused by the severe acute respiratory syndrome coronavirus 2 (SARS‐CoV‐2), had caused more than 5 million cases confirmed infections and 337 736 reported deaths until May 24, 2020, as reported by World Health Organization. The epidemic has spread to over 180 countries worldwide, and case numbers predicted to rise in the coming months. , To facilitate identification of infected people and ensure appropriate clinical treatment, the Chinese government and World Health Organization (WHO) recommended nucleic acid detection of SARS‐CoV‐2 as a gold standard.

The diagnosis of COVID‐19 is mainly confirmed by positive transcription polymerase chain reaction test (RT‐PCR), currently the most reliable diagnostic test for SARS‐CoV‐2 detection by targeting the open reading frame (ORF), envelope (E), nucleocapsid (N), and spike(S) genes. As of May 14, 2020, National Medical Products Administration (NMPA) of China has approved 32 detection kits for COVID‐19, and 19 of these kits were developed by RT‐PCR. Due to expanding pandemic and demand, RT‐PCR kits are being developed and placed on the market with limited validation on clinical samples. Some suspected patients exhibited typical clinical pneumonia symptoms, such as fever, cough, myalgia, fatigue, or image characteristics, but were negative in RT‐PCR testing in clinical practice. , This raises the question of whether there are differences in detection performance among RT‐PCR testing kits. ,

RT‐PCR testing kits diagnostic accuracy is depending on many factors, such as skilled laboratory staff, sample types and collection, transportation conditions, the test kit quality, and so on. Wang et al reported that oropharyngeal swabs showed lower positive rate than nasopharyngeal swabs. Pan et al found that thermal inactivation adversely affected the efficiency of RT‐PCR for SARS‐CoV‐2 detection samples with low viral loads. Wang et al compared the limit of detection (LoD) of five RT‐PCR kits using real viral RNA. However, there is still a lack of performance evaluation of RT‐PCR kits from different manufacturers under the strict quality evaluation system.

In the work, we presented the analytical performance evaluations of five RT‐PCR kits using nasopharyngeal swabs samples from patients with confirmed SARS‐CoV‐2 infection, and negative nasopharyngeal swabs samples (Figure 1). Our work may promote implementation and standardization across public health laboratories globally and be helpful for immediate decisions about clinical care and preventive measures.

Figure 1

Flowchart of the study

MATERIALS AND METHODS

Sources of specimen

Clinical specimens were obtained through the Shenzhen Luohu Hospital Group Medical Laboratory. A total of 50 specimens (nasopharyngeal swabs), including 20 positive clinical samples and 30 negative clinical samples, were collected between February and May of 2020 from hospitalized patients during the Luohu outbreak of COVID‐19. The study design has been approved by the Ethics Committees from the Shenzhen Luohu Hospital Group. Written informed consent was not required for this study because RNA leftovers from nasopharyngeal swabs specimens were exclusively intended for daily test items.

Commercial RT‐PCR kits studied

The five commercial RT‐PCR kits were selected for the study(Table 1): Liferiver novel coronavirus nucleic acid detection kit (Triple fluorescence PCR, Lot no.: P20200509, Shanghai Zhijiang Biotechnology co., Ltd.); Daan novel coronavirus nucleic acid detection kit (Fluorescent PCR, Lot no.: 2 020 012, Da An Gene Co., Ltd. Of Sun Yat‐sen University); BioGerm novel coronavirus nucleic acid detection kit (Double fluorescence PCR, Lot no.: 20200427D, Shanghai BioGerm medical technology co., LTD.); Genekey novel coronavirus nucleic acid detection kit (Dual fluorescence PCR method, Lot no.: 2 020 040 201, Shenzhen Genekey biotechnology Co); and Yaneng novel coronavirus nucleic acid detection kit (Fluorescence PCR, Lot no.: NV20200220‐1, Yaneng biotechnology Shenzhen co., LTD.). This study only focused on the corresponding Lot number reagent.

Table 1

Product characteristics of five kits

Kit	Liferiver	Daan	BioGerm	Genekey	Yaneng
Limit of detection (copies/mL)	1000	500	1000	1000	1000
Target genes	ORF1ab/N/E	ORF1ab/N	ORF1ab/N	ORF1ab/N	ORF1ab/N/S
Number of cycles/Cutoff value(Ct value)	45/43	45/40	40/38	40/37	45/40
Internal control	Yes	Yes	Yes	Yes	Yes
Volume of DNA input	5ul	5ul	5ul	5ul	5ul

RNA extraction and quantitative RT‐PCR assay

To use the patient's nasopharyngeal swab as nucleic acid detection sample, all samples were dealt with nucleic acid extraction and purification kit (Health Gene Technologies Co. Ltd) on a Smart LabAssist‐32 platform (Taiwan Advanced Nanotech Inc). All operation steps followed manufacturer recommended protocols. The reaction procedure of ABI7500 was set according to the reagents specifications of different manufacturers, and the test results were judged as negative or positive. During each run, both positive and negative controls were included to ensure that proper PCR responses are not subjected to carryover.

Positive and negative coincidence rates

To compare the detection capability for clinical specimens, a total of 30 specimens were separated for investigation using five RT‐PCR kits, all of which are based on TaqMan technology. Daan, BioGerm, and Genekey target ORF1ab and N gene. Liferiver and Yaneng target ORF1ab/N/E and ORF1ab/N/S gene, respectively. The positive and negative results were interpreted based on the manufacturer's instructions.

Detection ability

To further confirm the detection ability of the five kits, a positive clinical specimen (Ct: ORF1ab 26.99, N: 28.19) was diluted with 5‐fold using RNase‐free water, and the resulting dilution is considered as Level 1. A total of 5 concentration levels, Level 1, Level 2(1/3 Level 1), Level 3(1/9 Level 1), Level 4(1/27 Level 1), and Level 5 (1/81 Level 1), were tested with 5 replicates per concentration with the five kits. Each target gene was tested based on cycle threshold (Ct) values and amplification curves obtained during RT‐PCR. Then, the detection rate (number of positive results/total number of measurements) was used to evaluate the detection ability of the five kits.

Limit of detection

The limit of detection (LoD) is the lowest concentration level that can be detected with specified kit. The N gene fragment solid standard (GW‐CRPM001, GeneWell, China) was diluted into LoD that manufacturer claimed of each kit (500 copies/mL for Daan and 1000 copies/mL for others) using RNase‐free water. Twenty replicates at LoD were prepared following the sample preparation procedure given in each kit. A detection rate of ≥95% for positive results means that the results conform to the LoD that the manufacturers claimed.

Repeatability and imprecision

Two positive clinical specimens (P1: low positive and P2: moderate positive) were used to evaluate the repeatability and imprecision. According to modified EP15‐A protocol, each sample was tested five times a day for five days. Coefficient of variation (CV) and standard deviation (SD) were calculated separately for each kit.

Cross‐reactivity

A panel consisting of six human coronaviruses plasmid (HCoV‐229E, HCoV‐OC43, HCoV‐NL63, HCoV‐HKU1, SARS‐CoV, and MERS‐CoV) and four respiratory pathogens plasmid (human parainfluenza viruses, rhinovirus, influenza A virus, influenza B virus) were used to check for cross‐reactivity. Among them, the four respiratory pathogens are the common respiratory viruses detected in patients that may cause similar clinical symptoms and lead to false‐positive diagnostic results. The panel was added into 30 negative clinical samples. Positive quality control in the kit was used as internal control, and blank control was set.

Statistical analysis

The statistical software R version 3.6.1 (http://www.r‐project.org) was used for data evaluation. Excel 2010 and GraphPad Prism5.0 (GraphPad Software, Inc) were used for statistical analysis.

RESULTS

Clinical performance of the five kits

The distribution of Ct values of ORF1ab and N gene in 20 positive clinical samples by five kits was shown in Figure 2. According to the interpretation standard of each manufacturer's specification, the positive coincidence rate for 20 clinical samples of Genekey, Daan and BioGerm was 100% (20/20). The positive rate of ORF1ab gene fragment for Liferiver and Yaneng in 20 clinical samples with in‐house confirmed SARS‐CoV‐2 infection was both 90% (18/20). However, for the other two target genes, the positive rate of Liferiver was 100% (20/20), while Yaneng was 90% (18/20). According to interpretation rules of Liferiver and Yaneng, the positive rate for them was both 90%. The coincidence rate of five kits in 10 negative samples was 100% (10/10). Liferiver had the highest Ct value than four other kits in 20 positive clinical samples for ORF1ab gene. For N gene, Genekey had the highest Ct value than four other kits in 20 positive clinical samples. Importantly, the within‐run CV was less than 5% for the Ct value about ORF1ab/N gene in 20 clinical samples detected by five kits.

Figure 2

The Ct value distribution of five kits in 20 positive specimens. A. ORF1ab gene, B. N gene

The detection results of five concentrations were shown in Table 2. The results showed that the positive coincidence rate was 100% from Level 1 to Level 3 for kits included in the study, while the detection ability of the five kits differs in Level 4 and Level 5 substantially. The detection rate of ORF1ab gene of five kits was 100% from Level 1 to Level 3. In Level 4, the detection rate of Daan was 100%, but the Ct values of ORF1ab gene were all in the gray region (38 ≤ Ct<40). In Level 5, five kits presented poor positive rate, only one of five replicates was detected for Daan, BioGerm and Yaneng. Figure 3 shows that Ct values of RT‐PCR negatively correlated with viral load.

Table 2

Detection limit of five SARS‐CoV‐2 RT‐PCR kits

Kits	Target genes	Different concentrations(copies/ml) of SARS‐CoV‐2
		Level 1	Level 2	Level 3	Level 4	Level 5
Genekey	ORF1ab/N	100%	100%	100%	0%	0%
Daan	ORF1ab/N	100%	100%	100%	100%	20%
BioGerm	ORF1ab/N	100%	100%	100%	80%	20%
Liferiver	ORF1ab/N/E	100%	100%	100%	60%	0%
Yaneng	ORF1ab/N/S	100%	100%	100%	80%	20%

Figure 3

Relationship between the Ct value and 5 concentration gradients. A. ORF1ab gene, B. N gene

To explore the LoD of five kits, N gene fragment solid standard was diluted to two separate final concentrations (500 copies/mL and 1000 copies/mL). Daan detected 20/20 replicates at 500 copies/mL. BioGerm, Yaneng, Genekey, and Liferiver detected 20/20 replicates at 1000 copies/mL. Ct values (Daan:33.04 ± 0.18, BioGerm:32.46 ± 0.24, Genekey:35.45 ± 0.88, Liferiver:33.04 ± 0.64, Yaneng:36.61 ± 0.79) were less than the positive cutoff value of each kit. Following guidelines in document EP17‐A, LoD for each kit was determined at the lowest concentration level with a positive result detection rate of 95%. In our study, the LoD declared by each manufacturer was verified.

The results of five kits for repeatability and imprecision evaluation were shown in Table 3. The repeatability for ORF1ab gene was from 0.11%(P1) to 0.74%(P2) and N gene was from 0.12% (P1) to 1.11% (P2), respectively. The total imprecision for ORF1ab gene was from 0.14% (P1) to 1.1% (P2) and N gene was from 0.16% (P1) to 0.52% (P2). Genekey has the best imprecision for ORF1ab gene with a CV value 0.24% in P1 and 0.49% in P2. BioGerm has the best repeatability for ORF1ab gene in both P1 (0.11%) and P2 (0.45%). Five kits have superior repeatability and imprecision for N gene in P1 and P2 expect Liferiver which has poorest repeatability and imprecision. It worth note that Genekey has a poorer imprecision for N gene in P2 (0.56%), rank only second to Liferiver. However, It has the best imprecision for N gene of all kits in P1. The repeatability was all less than 5%, and imprecision was less than 10% for all the five kits. The repeatability and imprecision of ORF1ab and N genes in P2(low positive sample) were generally higher than that in P1(strong positive sample).

Table 3

Repeatability and imprecision of the five kits

Kits	Sample	ORF1ab fragment					N fragment
		Mean	Repeatability		Total imprecision		Mean	Repeatability		Total imprecision
			SD	CV（%）	SD	CV（%）		SD	CV（%）		SD	CV（%）
Genekey	P1	25.2	0.04	0.16	0.06	0.24	25.23	0.03	0.12		0.03	0.14
Daan		24.47	0.05	0.2	0.16	0.66	23.95	0.05	0.2		0.05	0.2
BioGerm		25.53	0.03	0.11	0.18	0.69	26.38	0.07	0.23		0.08	0.3
Liferiver		25.52	0.04	0.13	0.07	0.28	22.95	0.07	0.28		0.11	0.5
Yaneng		25.46	0.06	0.19	0.11	0.42	26.45	0.07	0.25		0.1	0.38
Genekey	P2	32.57	0.17	0.52	0.17	0.52	32.98	0.21	0.57		0.21	0.57
Daan		32.74	0.22	0.64	0.26	0.79	31.92	0.13	0.38	0.18		0.55
BioGerm		33.28	0.15	0.45	0.25	0.76	33.75	0.16	0.43	0.17		0.5
Liferiver		33.09	0.25	0.74	0.25	0.75	31.50	0.36	1.11	0.36		1.11
Yaneng		34.18	0.24	0.65	0.23	0.67	34.63	0.12	0.34	0.16		0.47

Abbreviations: CV, coefficient of variation; SD, standard deviation.

Six human coronaviruses and four respiratory pathogens were selected as interfering substance to evaluate the cross‐reactivity. No positive result was obtained in testing of 30 negative clinical samples by using five kits for ORF1ab and N gene.

DISCUSSION

Rapid detection of infectious virus will be useful for outbreak investigations, disease surveillance, and health risk analysis. Based on China's experience in COVID‐19 prevention and control in the past 6 months, RT‐PCR‐based method to detect SARS‐CoV‐2 nucleic acid has been introduced in routine diagnosis to screen SARS‐CoV‐2 virus‐infected patients. Recently, some papers have been published on the efficacy of RT‐PCR assays. However, with the emergence of new detection kits, their performance has not yet been systematically assessed. In this study, we evaluated 5 commercially available RT‐PCR kits for the identification of SARS‐CoV‐2 in clinical samples. The positive identification rate for the five RT‐PCR kits varied from 18 to 20 out of 20 samples (Figure 2), with Genekey, Daan, and BioGerm performing better(20/20), followed by Liferiver and Yaneng (18/20). The negative identification rate of 5 kits was 100% (10/10). The results showed that 5 kits detection results of clinical samples were almost consistent. Liferiver targets a combination of the E gene, N gene, and ORF1ab gene. The positive identification rate of E gene and N gene was 100% (20/20), but ORF1ab gene was not detected in 2 positive samples. ORF1ab gene is a non‐structural region that is unique to SARS‐CoV‐2. Almost all RT‐PCR kits use it as a necessary target. Positive results should not be reported according to Liferiver kit instructions. Another study showed that the E gene has a 20% (2/20) missed detection rate for the kit. In addition, we used 1 low positive sample and 1 moderate positive sample to evaluate five kits repeatability and imprecision, and the CV value was less than 5%.

LoD is the lowest analyte concentration. Daan kit has the lowest LoD than other four kits. We evaluated the LoD of these four kits using N gene fragment solid standard. All RT‐PCR kits performed satisfactorily regarding LoD. Notably, we performed our analysis using only N gene sample and we therefore advise that medical laboratories conduct more extensive target gene validations upon implementation of novel RT‐PCR kits. A study evaluated LoD of six China National Medical Products Administration (NMPA) approved RT‐PCR kits for SARS‐CoV‐2. The study showed that the LoD of Liferiver and Daan kit was 484 copies/mL and BioGerm kit was 968 copies/mL. The conclusions conform to LoD declared in the instructions of Liferiver, Daan, and BioGerm. However, another study demonstrated that the LoD of clinical samples was significantly higher than the LoD declared in the instructions of Daan. The possible reason for the difference is that there are batch differences in the continuous optimization of the kit.

To evaluate the detection ability, a positive sample was serially diluted in simulated clinical matrix. A total of 5 concentration levels, with 3‐fold serial dilutions between the levels, were tested in replicates of 5. All RT‐PCR kits were able to positively identify the sample with concentration from Level 1 to Level 3. At Level 4 and Level 5, the detection ability decreases to varying degrees. Only Daan kit was a 100% detection rate. This may result from Daan kit the lowest LoD than other four kits. Ct value of qRT‐PCR is a semi‐quantitative measurement of viral load and inversely proportional to quantity of target gene. In other words, a high Ct value represents a low viral load and vice versa. In clinical applications, Ct value is negative association with SARS‐CoV‐2 probability of disease progression. ,

SARS‐CoV‐2 is a beta coronavirus belonging to the family of Coronaviruses with at least 70% similarity in genetic sequence to SARS‐CoV‐1. , ORF1/a gene is unique to SARS‐CoV‐2, whereas E gene was pan‐Sarbecovirus detection and the expected cross‐reactivity with the SARS‐CoV‐1 E gene. There were no cross‐reactions by the five kits for ORF1ab and N gene with six human coronaviruses and four respiratory pathogens. Considering a variety of clinical scenarios, multiplex PCR‐based assay that can simultaneously detect many pathogens including SARS‐CoV‐2 may become a regular detection method.

The study has some limitations. First, the 20 clinical specimens from SARS‐CoV‐2 confirmed patients were coming from same region. Geographical underrepresentation may influence the results. Second, we used the same virus RNA extraction kit for five kits. It might affect some kits. Third, the 20 positive clinical specimens were all nasopharyngeal swabs specimens. While a nasopharyngeal swab is the preferred testing method, further investigation to evaluate RT‐PCR kit performance using other specimen types is needed.

In conclusion, we believe that five RT‐PCR kits included in this study have good diagnostic performance and are useful tools for the routine diagnosis. We advise that the high‐risk populations, such as people who are close contact with COVID‐19 patients, patients during later stages of the infection, people who have fever but have not been detected and caregivers, should be detected by low LoD kits.

CONFLICT OF INTEREST

The authors declare that they have no competing interests.

AUTHOR CONTRIBUTIONS

Dayang Chen, Wei Wu, and Mengmeng Wang analyzed the data and drafted the paper. All the authors participated in the experiment. All authors read and agreed the final version of the article.

ETHICAL APPROVAL

The study design has been approved by the Ethics Committees from the Shenzhen Luohu Hospital Group.