Comparison of seven commercial SARS-CoV-2 nucleic acid detection reagents with pseudovirus as quality control material.

The ongoing pandemic of COVID-19 threats the whole world, which catalyzes a variety of SARS-CoV-2 nucleic acid test (NAT) kits. To monitor test quality and evaluate NAT kits, quality control materials that best simulate real clinical samples are needed in this study, the performance of SARS-CoV-2 cell culture supernatant, PCDH-based pseudovirus, and MS2-based pseudovirus as quality control materials was compared. As a result, PCDH-based pseudovirus was more similar in characteristics to SARS-CoV-2 particle, and was more suitable for evaluating SARS-CoV-2 NAT kits than MS2-based pseudovirus. Proper detection using sensitive and precise NAT kits is essential to guarantee diagnosis. Thus, limit of detection, precision, anti-inference ability and cross-reactivity of NAT kits from PerkinElmer, Wantai, Shanghai Kehua Bio-Engineering (KHB), Sansure, Da An, BioGerm, and Applied Biological Technologies (ABT) were compared using PCDH-based pseudovirus. For the seven kits evaluated, N gene was more sensitive than ORF1ab gene in most kits, while E gene was most sensitive among the three genes in KHB and ABT. PerkinElmer got the lowest LoD for N gene at 11.61 copies/mL, and 34.66 copies/mL for ORF1ab gene. All the kits showed good precision with CV values less than 5%, as well as acceptable anti-interference ability of 2 mg/L human genomic DNA. No cross-reactivity was observed with other respiratory viruses.

In December 2019, severe acute respiratory syndrome coronavirus 2 (n class="Species">SARS-CoV-2), the causative virus of coronavirus disease 2019 (COVID-19), emerged. On March 11, 2020, the World Health Organization declared COVID-19 a pandemic. As of December 14, 2020, there have been 71,051,805 confirmed cases of COVID-19 and 1,608,648 deaths [World Health Organization Coronavirus Disease (COVID-19) Dashboard, ]. At present, SARS-CoV-2 is still spreading and posing a threat to global public health.

Rapid and accurate diagnosis of SARS-CoV-2 is esseene">ntial for early discovery aene">nd early treatmeene">nt. Because of its easy methods aene">nd high seene">nsitivity aene">nd specificity, real-time RT-PCR is becomiene">ng the optimal aene">nd most widely used method for detectiene">ng nucleic acid, aene">nd eveene">n the gold staene">ndard for viral detection. Multiple commercial real-time RT-PCR assays have beeene">n developed to meet diagene">nostic needs for n class="Species">SARS-CoV-2. However, clinically confirmed patients with negative nucleic acid test (NAT) results have been reported. , Comparing the performance of commonly used NAT kits and providing evidence for choosing sensitive, specific, and precise NAT kits are the key to obtaining reliable test results.

Quality control materials are needed for evaluating the performance of SARS-CoV-2 NAT kits. Although n class="Species">patient samples are most in line with the actual situation, difficulty in obtaining samples and the risk of transmission limit their application as quality control materials. Therefore, construction of pseudoviruses similar to SARS-CoV-2 virus is necessary to evaluate the performance of NAT kits, as well as to control the quality of experimental procedure. Vectors of lentivirus and bacteriophage have been widely used to produce stable quality control materials.4, 5, 6, 7 These vectors vary in their characteristics, including size, packaging capacity, and stability. To best simulate real patient samples, the performance of different quality control materials needs to be compared.

In this study, the applicability of MS2- aene">nd PCDH-based pseudovirus, contaiene">niene">ng the amplification regions of n class="Species">SARS-CoV-2 NAT kits, was compared with that of SARS-CoV-2 cell culture supernatant. Furthermore, limit of detection (LoD), precision, anti-inference ability, and cross-reactivity of seven Conformité Européene marked and widely used SARS-CoV-2 NAT kits were evaluated using PCDH-based pseudovirus.

Materials and Methods

Quality Control Materials

Three kinds of quality control materials were constructed. SARS-CoV-2 cell culture supernataene">nt was kiene">ndly gifted by Chiene">nese Academy of Military Medical Scieene">nces (Geene">nBaene">nk, ; accession number MT135042.1). Briefly, virus was cultured iene">n n class="CellLine">Vero-E6 cells for 96 hours, and then the supernatant was collected and disinfected by heating at 65°C for 30 minutes. For constructing PCDH-based pseudovirus (Figure 1 A), PCDH-CMV-MCS-EF1-copGFP vector was digested with enzymes NotI and SalI, and the product was gel purified. N gene, E gene, and segments of ORF1ab gene were synthesized with overlapping ends by PCR, and further cloned into digested PCDH vector using one step of Gibson assembly (New England Biolabs, Ipswich, MA) at 50°C for 1 hour. The resultant plasmid was electroporated into ER2738 electrocompetent cells (Lucigen, Middleton, WI). After recovery in SOC medium at 37°C for 1 hour, the ER2738 cells were spread onto LB solid medium with 8 mg/L glucose and 50 mg/L ampicillin and incubated at 37°C overnight. Positive clones were sequenced, and successfully constructed plasmid was extracted. Then, HEK293T cells were seeded in 10-cm dish. After 24 hours, PCDH plasmid and two packaging plasmids (psPAX and pMD2G) were transfected at the ratio of 4:3:1 using polyethylenimine (Polysciences, Warrington, PA). Supernatant was gathered 48 hours after transfection, and centrifugated at 800 g for 5 minutes to remove cell debris. For constructing MS2-based pseudovirus (Figure 1B), pGSI-MS2 vector was digested with NotI and BamHI, and gel purified. N gene, E gene, and nine segments of ORF1ab gene were synthesized, ligated into MS2 vector, and transformed into competent Escherichia coli cells. Products were harvested, sonicated, and purified, as before. Finally, MS2-based pseudovirus, containing different regions of SARS-CoV-2 genome, was mixed together with the same volume ratio so that it can contain all of the target sequences of all reagents evaluated. After production, materials were stored at −40°C.

Figure 1

Schematic diagram of producing PCDH-based pseudovirus (A) and MS2-based pseudovirus (B). Nine regions of ORF1ab were inserted into MS2 vector, and one of them was drawn as representative.

Schematic diagram of producing PCDH-based pseudovirus (A) and MS2-based pseudovirus (B). Niene">ne regions of n class="Gene">ORF1ab were inserted into MS2 vector, and one of them was drawn as representative.

Digital PCR

Quality control materials were diluted to about 105 copies/mL, and sent to reagent manufacturers. Digital PCR was conducted using the same primers and probes as in their NAT kits, and the median of all results was taken as the exact concentration of quality control materials. The detailed information of digital PCR conducted by each manufacturer was summarized in Supplemental Table S1. The process of digital PCR was indicated before.

Real-Time RT-PCR

Seven SARS-CoV-2 NAT kits were iene">ncluded. They were from Perkiene">nElmer, Inc. (Waltham, MA), Beijiene">ng Waene">ntai Biological Pharmacy Eene">nterprise Co, Ltd (Waene">ntai; Beijiene">ng, Chiene">na), Shaene">nghai Kehua Bio-Eene">ngiene">neeriene">ng Co, Ltd (n class="Chemical">KHB; Beijing, China), Sansure Biotech Inc. (Sansure; Changsha, China), Da An Gene Co, Ltd, of Sun Yat-sen University (Da An; Guangzhou, China), Shanghai BioGerm Medical Biotechnology Co, Ltd (BioGerm; Shanghai, China), and Beijing Applied Biological Technologies Co, Ltd (ABT; Beijing, China). The kits were all Conformité Européene marked. Among them, the kits of Sansure, Da An, BioGerm, and ABT have been approved by the China National Medical Products Administration. The kits of PerkinElmer, Sansure, and Wantai have been authorized for emergency use by the US Food and Drug Administration. The kits of PerkinElmer, KHB, Wantai, Da An, and ABT are on the list of World Health Organization emergency use listing. PCDH-based pseudovirus was sent to each manufacturer, and nucleic acids were extracted and real-time RT-PCR was conducted following the protocol. KHB and ABT tested N, ORF1ab, and E genes, whereas other kits only tested N and ORF1ab genes. The information of extraction and amplification is summarized in Supplemental Table S2.

Stability Test

For stability test, three quality control materials were diluted to about 1000 copies/mL using Dulbecco's modified Eagle’s medium (Corning, NY) containing 0.1% ProClin 300. Each material was thoroughly mixed aene">nd divided iene">nto four groups. Control group was stored at −40°C for 2 weeks. To assess freeze-thaw stability, materials were stored at −40°C for 2 weeks, duriene">ng which they were frozeene">n aene">nd thawed three times repeatedly. To evaluate whether the materials are stable after opeene">niene">ng the tubes, materials were stored at −40°C for a week aene">nd tubes were opeene">n for a while, aene">nd theene">n they were stored at 2°C to 8°C for a week. For fundameene">ntal stability, materials were stored at room temperature for a week aene">nd theene">n put at 2°C to 8°C for aene">nother week. After processiene">ng, the nucleic acids were extracted usiene">ng Perkiene">nElmer PreNAT II, aene">nd tested usiene">ng Perkiene">nElmer's n class="Species">SARS-CoV-2 NAT kit on ABI7500 in sextuplicate. Results were analyzed following criteria from China National Accreditation Service for Conformity Assessment GL003: Guidance on Evaluating the Homogeneity and Stability of Samples Used for Proficiency Testing. Considering the bias of real-time RT-PCR assay, σ was identified as 0.15 in this study.

Amplification Efficiency

SARS-CoV-2 cell culture supernataene">nt, n class="Species">MS2-based pseudovirus, and PCDH-based pseudovirus were diluted to 100,000, 20,000, 4000, and 800 copies/mL. Each concentration of each material was extracted and tested three times in a single run. Standard curve was performed with log of concentration and mean cycle threshold (Ct) value of three repeats. Amplification efficiency of each kit on each material was calculated as follow: E% = [10(−1/slope) − 1] × 100%.

Limit of Detection

LoD was evaluated in accordance with EP17-A2 guideliene">ne (Evaluation of Detection Capability for Cliene">nical Laboratory Measuremeene">nt Procedures; Approved Guideliene">ne—Second Edition) of Cliene">nical Laboratory Staene">ndards Institute. Briefly, PCDH-based pseudovirus was gradieene">ntly diluted with Dulbecco's modified Eagle’s medium contaiene">niene">ng 0.1% n class="Chemical">ProClin 300. Each sample was extracted and tested eight times per day in a single run for 3 consecutive days. Positive rate of each dilution was counted, and LoD was calculated from a probit regression model as the measurand concentration at which measurement results yielded a positive classification with a 95% probability. Result determination was following manufacturer’s instructions. For BioGerm and ABT, Ct value ≤38 was considered positive. For PerkinElmer, Ct value ≤42 was considered positive. For other kits, Ct value ≤40 was considered positive.

Precision

Precision was evaluated in accordance with EP15-A3 guideline (User Verification of Precision and Estimation of Bias; Approved Guideline) of Clinical Laboratory Standards Institute. Briefly, two concentrations of PCDH-based pseudovirus were extracted and tested five times per day in a single run for 5 consecutive days, and the within-run precision (repeatability) and within-laboratory precision were calculated following the guideline.

Interference Testing

Interference testing was evaluated in accordance with EP07-A2 guideline (Interference Testing in Clinical Chemistry; Approved Guideline) of Clinical Laboratory Standards Institute. Briefly, two concentrations of PCDH-based pseudovirus were prepared, and a final concentration of 2 mg/L human geene">nomic DNA or double-distilled n class="Chemical">water was added into pseudovirus and thoroughly mixed. Nucleic acids were extracted and amplificated 20 times in a single run.

Cross-Reactivity

The potential cross-reactivity of organisms was tested in both the presence and absence of PCDH-based SARS-CoV-2 pseudovirus by seveene">n kits. n class="Species">SARS-CoV-2–positive samples were prepared by mixing each of the viruses with 900 copies/mL PCDH-based SARS-CoV-2 pseudovirus. For reagents of PerkinElmer, Wantai, and KHB, cross-reactivity was also determined at the level of 180 copies/mL. Human coronavirus 229E [ATCC (Manassas, VA) VR-740], OC43 (ATCC VR-1558), influenza A virus seasonal H1N1 (ATCC VR-1520), influenza A virus H3N2 (ATCC VR-1679), and influenza B virus (ATCC VR-1807) were kindly gifted by PerkinElmer, and were tested at 1 × 105 copies/mL. PUC57 plasmid vector, inserting 28,267 to 29,535 bp of SARS virus Urbani isolate icSARS-C7-MA (GenBank, ; accession number MK062184.1), and MS2-based pseudovirus, containing 28,564 to 29,807 bp of Middle East respiratory syndrome virus isolate HCoV-EMC/2012 (GenBank, ; accession number NC_019843.3), were kindly gifted by Da An, which were tested at 1 × 105 median tissue culture infective dose (TCID50)/mL. All of the tests were performed in triplicate in a single run.

Statistical Analysis

Statistical analysis was performed on IBM (Armonk, NY) SPSS version 21.0. Wilcoxon matched-pairs signed rank test was performed to compare the differences between two groups.

Results

Quantification of Three Quality Control Materials

SARS-CoV-2 cell culture supernataene">nt, n class="Species">MS2-based pseudovirus, and PCDH-based pseudovirus were diluted 10-fold, 1000-fold, and 10,000-fold, respectively. Digital PCR was performed to quantify N gene, E gene, and ORF1ab gene by six reagent manufacturers, except BioGerm. Concentration of quality control materials varied from 4 to 6 log10 copies/mL, so that median was taken as the final concentration (Figure 2 ). Among the three quality control materials, PCDH-based pseudovirus showed the smallest variation among both reagents and genes.

Figure 2

Concentration of three quality control materials. Digital PCR was performed to quantify the concentration of N gene, E gene, and ORF1ab gene by six reagent manufacturers. Data are presented as median with range.

Concentration of three quality control materials. Digital PCR was performed to quantify the concentration of N gene, E gene, and ORF1ab geene">ne by six reageene">nt maene">nufacturers. Data are preseene">nted as mediaene">n with raene">nge.

Stability of Quality Control Materials

Three kinds of quality control materials were processed at four conditions indicated before. Real-time RT-PCR using the PerkinElmer kit was performed thereafter. The difference in mean Ct values of ORF1ab geene">ne betweeene">n each experimeene">ntal group aene">nd control group was all <0.5 (Figure 3 aene">nd Supplemeene">ntal Table S3). However, wheene">n evaluatiene">ng fundameene">ntal stability of cell culture supernataene">nt, the Ct value of N geene">ne was 0.76 higher thaene">n control group, iene">ndicatiene">ng that it might be unstable at this condition.

Figure 3

Stability of severe acute respiratory syndrome coronavirus 2 cell culture supernatant (A), PCDH-based pseudovirus (B), and MS2-based pseudovirus (C). Materials were thoroughly mixed, divided, and processed with four different conditions. The Ct values of N and ORF1ab genes were tested. Red dotted line represents the level of mean Ct value of ORF1ab gene in control group ±0.5. Blue dotted line represents the level of mean Ct value of N gene in control group ±0.5. Data are presented as means ± SEM (A–C).

Stability of severe acute respiratory syndrome coronavirus 2 cell culture supernataene">nt (A), PCDH-based pseudovirus (B), aene">nd n class="Species">MS2-based pseudovirus (C). Materials were thoroughly mixed, divided, and processed with four different conditions. The Ct values of N and ORF1ab genes were tested. Red dotted line represents the level of mean Ct value of ORF1ab gene in control group ±0.5. Blue dotted line represents the level of mean Ct value of N gene in control group ±0.5. Data are presented as means ± SEM (A–C).

Amplification Efficiency among Quality Control Materials

Three kinds of quality control materials were diluted in a fivefold gradient between 100,000 and 800 copies/mL, and real-time RT-PCR was performed using seven kits evaluated. The amplification efficiencies of N gene and ORF1ab geene">ne iene">n PCDH-based pseudovirus had no differeene">nce with those iene">n cell culture supernataene">nt, whereas the amplification efficieene">ncies of n class="Species">MS2-based pseudovirus were significantly higher than those of cell culture supernatant (Figure 4 ). Although only KHB and ABT detected E gene, it could also be seen that the E gene amplification efficiency of PCDH-based pseudovirus was closer to cell culture supernatant than MS2-based pseudovirus. These results suggest that PCDH-based pseudovirus is more similar to SARS-CoV-2 particle, and is more suitable for evaluating SARS-CoV-2 NAT kits than MS2-based pseudovirus.

Figure 4

Amplification efficiency of seven severe acute respiratory syndrome coronavirus 2 nucleic acid test kits when amplifying N gene (A), ORF1ab gene (B), and E gene (C) in cell culture supernatant, PCDH-based pseudovirus, and MS2-based pseudovirus. Wilcoxon matched-pair signed rank test was performed to compare the differences between two groups. ∗P < 0.05.

Amplification efficiency of seven severe acute respiratory syndrome coronavirus 2 nucleic acid test kits wheene">n amplifyiene">ng N geene">ne (A), n class="Gene">ORF1ab gene (B), and E gene (C) in cell culture supernatant, PCDH-based pseudovirus, and MS2-based pseudovirus. Wilcoxon matched-pair signed rank test was performed to compare the differences between two groups. ∗P < 0.05.

PCDH-based pseudovirus was gradiently diluted. Each sample was extracted and tested eight times per day in a single run for 3 consecutive days. A total of 24 tests were performed in each concentration, and LoDs were calculated by probit analysis. The N gene was more sensitive than ORF1ab geene">ne iene">n the kits from Perkiene">nElmer, Waene">ntai, n class="Chemical">KHB, Sansure, and Da An (Supplemental Tables S4 through S6). The LoDs of ORF1ab were slightly lower than N gene in the kits of BioGerm and ABT. Among all of the kits, only KHB and ABT detected E gene, which held better sensitivity than N gene and ORF1ab gene. The PerkinElmer kit had the lowest LoD for N gene at 11.61 copies/mL, and the value was 34.66 copies/mL for ORF1ab gene.

Specificity

Twelve negative oropharyngeal swabs mixed with 900 or 180 copies/mL PCDH-based SARS-CoV-2 pseudovirus aene">nd 24 negative oropharyene">ngeal swabs were tested for specificity (Supplemeene">ntal Table S7). For these seveene">n kits, all of the positive samples yielded positive results, aene">nd all of the negative swabs were tested negative, showiene">ng a specificity of 100%.

Two concentrations of PCDH-based pseudovirus were prepared for each reagent to evaluate precision. Concentrations of three regions were calculated by droplet digital PCR results and dilution multiples. High concentration was the same to all reagents (ie, 900 copies/mL of N gene, 760 copies/mL of ORF1ab geene">ne, aene">nd 670 copies/mL of E geene">ne). Low conceene">ntration varied accordiene">ng to the LoD of each kit. Perkiene">nElmer holds the lowest LoD, so the low level to evaluate precision of Perkiene">nElmer was 90 copies/mL for N geene">ne aene">nd 76 copies/mL for n class="Gene">ORF1ab gene. For Wantai and KHB, the low level was twice that for PerkinElmer. For other reagents, low concentration was three times that for PerkinElmer. Within-run CV% and within-laboratory CV% of all reagents were <5%, indicating that all these reagents had good precision (Table 1 ).

Table 1

Precision of SARS-CoV-2 NAT Kits

Region	Reagent manufacturer	Low concentration					High concentration
		Mean Ct	Within-run SD	Within-run CV, %	Within-laboratory SD	Within-laboratory CV, %	Mean Ct	Within-run SD	Within-run CV, %	Within-laboratory SD	Within-laboratory CV, %
N	PerkinElmer	35.32	0.76	2.15	1.05	2.97	31.88	0.27	0.85	0.73	2.30
	Wantai	34.80	0.69	1.99	0.71	2.04	32.07	0.33	1.03	0.50	1.57
	KHB	31.26	0.58	1.84	0.83	2.66	28.49	0.31	1.10	0.53	1.86
	Sansure	35.75	0.47	1.31	0.62	1.73	34.26	0.37	1.08	0.60	1.76
	Da An	36.37	0.57	1.58	0.71	1.96	34.56	0.36	1.05	0.34	0.98
	BioGerm	35.99	0.63	1.75	0.84	2.33	34.31	0.49	1.42	0.83	2.42
	ABT	36.70	0.65	1.77	1.03	2.80	35.46	0.47	1.33	0.65	1.84
ORF1ab	PerkinElmer	35.05	0.63	1.79	0.84	2.41	31.42	0.22	0.69	0.49	1.55
	Wantai	35.62	1.20	3.37	1.16	3.25	32.54	0.40	1.23	0.53	1.62
	KHB	32.76	1.19	3.65	1.31	4.00	30.08	0.53	1.76	0.85	2.82
	Sansure	38.01	0.70	1.85	0.84	2.21	36.89	0.94	2.55	1.20	3.26
	Da An	38.68	0.57	1.47	0.62	1.60	36.82	0.39	1.05	0.48	1.29
	BioGerm	35.54	0.74	2.09	0.75	2.11	33.80	0.49	1.45	0.75	2.23
	ABT	37.26	1.05	2.82	1.38	3.70	35.66	0.63	1.76	0.80	2.25
E	KHB	31.04	0.55	1.78	0.71	2.29	28.47	0.27	0.96	0.53	1.88
	ABT	36.08	0.77	2.12	0.87	2.40	34.41	0.46	1.32	0.65	1.90

High concentration was 900 copies/mL of N gene, 760 copies/mL of ORF1ab gene, and 670 copies/mL of E gene. For N gene, low concentration was 90 copies/mL for PerkinElmer, 180 copies/mL for Wantai and KHB, and 360 copies/mL for other reagents. For ORF1ab gene, low concentration was 76 copies/mL for PerkinElmer, 152 copies/mL for Wantai and KHB, and 304 copies/mL for other reagents. For E gene, low concentration was 134 copies/mL for KHB and 268 copies/mL for ABT.

ABT, Beijing Applied Biological Technologies Co, Ltd; BioGerm, Shanghai BioGerm Medical Biotechnology Co, Ltd; Da An, Da An Gene Co, Ltd; KHB, Shanghai Kehua Bio-Engineering Co, Ltd; NAT, nucleic acid test; Sansure, Sansure Biotech Inc.; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; Wantai, Beijing Wantai Biological Pharmacy Enterprise Co, Ltd.

High concentration was 900 copies/mL of N gene, 760 copies/mL of ORF1ab geene">ne, aene">nd 670 copies/mL of E geene">ne. For N geene">ne, low conceene">ntration was 90 copies/mL for Perkiene">nElmer, 180 copies/mL for Waene">ntai aene">nd n class="Chemical">KHB, and 360 copies/mL for other reagents. For ORF1ab gene, low concentration was 76 copies/mL for PerkinElmer, 152 copies/mL for Wantai and KHB, and 304 copies/mL for other reagents. For E gene, low concentration was 134 copies/mL for KHB and 268 copies/mL for ABT.

ABT, Beijiene">ng Applied Biological Techene">nologies Co, Ltd; BioGerm, Shaene">nghai BioGerm Medical Biotechene">nology Co, Ltd; n class="Chemical">Da An, Da An Gene Co, Ltd; KHB, Shanghai Kehua Bio-Engineering Co, Ltd; NAT, nucleic acid test; Sansure, Sansure Biotech Inc.; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; Wantai, Beijing Wantai Biological Pharmacy Enterprise Co, Ltd.

Interference

To simulate real patient samples aene">nd evaluate whether n class="Species">human genomic DNA in patient samples can interfere with detection result, 2 mg/L human genomic DNA or double-distilled water was added into PCDH-based pseudovirus, and samples were tested 20 times in a single PCR run. The concentrations of pseudovirus were the same with the levels to evaluate precision. When conducting interference testing at high concentration, all of the 20 tests of each reagent were positive in the existence of 2 mg/L human genomic DNA, indicating this interfering substance did not interfere with the determination of positive results (Table 2 ). When comparing the Ct values of interfering group and control group by Wilcoxon matched-pairs signed rank test, N gene and E gene of KHB, as well as ORF1ab gene of ABT, showed significant differences between two groups. This result suggests that 2 mg/L human genomic DNA might still interfere with the PCR process, but the interference is acceptable for qualitative reagents because it does not change the determination of results.

Table 2

Interference Testing at High Concentration of PCDH-Based Pseudovirus

Reagent manufacturer	N gene					ORF1ab gene					E gene
	Control		2 mg/L DNA		P value	Control		2 mg/L DNA		P value	Control		2 mg/L DNA		P value
	Mean Ct	Positive rate	Mean Ct	Positive rate		Mean Ct	Positive rate	Mean Ct	Positive rate		Mean Ct	Positive rate	Mean Ct	Positive rate
PerkinElmer	32.07	20/20	31.79	20/20	0.26	31.62	20/20	31.70	20/20	0.39
Wantai	33.33	20/20	33.16	20/20	0.21	33.46	20/20	33.13	20/20	0.06
KHB	28.53	20/20	28.41	20/20	0.03	29.92	20/20	30.02	20/20	0.38	28.52	20/20	28.34	20/20	0.02
Sansure	36.85	20/20	36.65	20/20	0.44	37.45	20/20	37.87	20/20	0.14
Da An	35.58	20/20	35.78	20/20	0.20	37.26	20/20	37.37	20/20	0.46
BioGerm	34.56	20/20	34.46	20/20	0.70	34.89	20/20	34.32	20/20	0.11
ABT	36.02	20/20	36.15	20/20	0.23	36.99	20/20	36.16	20/20	0.01	35.55	20/20	35.55	20/20	0.89

The concentration was 900 copies/mL of N gene, 760 copies/mL of ORF1ab gene, and 670 copies/mL of E gene.

ABT, Beijing Applied Biological Technologies Co, Ltd; BioGerm, Shanghai BioGerm Medical Biotechnology Co, Ltd; Da An, Da An Gene Co, Ltd; KHB, Shanghai Kehua Bio-Engineering Co, Ltd; Sansure, Sansure Biotech Inc.; Wantai, Beijing Wantai Biological Pharmacy Enterprise Co, Ltd.

ABT, Beijiene">ng Applied Biological Techene">nologies Co, Ltd; BioGerm, Shaene">nghai BioGerm Medical Biotechene">nology Co, Ltd; n class="Chemical">Da An, Da An Gene Co, Ltd; KHB, Shanghai Kehua Bio-Engineering Co, Ltd; Sansure, Sansure Biotech Inc.; Wantai, Beijing Wantai Biological Pharmacy Enterprise Co, Ltd.

Because low concentration was near the LoD of several reagents, the positive rate was not 100% for all reagents. 2 mg/L human geene">nomic DNA did not iene">nterfere with the determiene">nation of positive results of the Waene">ntai kit (Table 3 ). As for Perkiene">nElmer, the positive rate of n class="Gene">ORF1ab gene slightly reduced from 100% to 90%. For the kits with less sensitivity, positive rates were compared. For ORF1ab gene of KHB, and both N and ORF1ab gene of ABT, the positive rate of interfering group is a bit lower than that of control group. For ORF1ab gene of Sansure and BioGerm, the positive rate was higher in interfering group than in control group.

Table 3

Interference Testing at Low Concentration of PCDH-Based Pseudovirus

Reagent manufacturer	N gene					ORF1ab gene					E gene
	Control		2 mg/L DNA		P value	Control		2 mg/L DNA		P value	Control		2 mg/L DNA		P value
	Mean Ct	Positive rate	Mean Ct	Positive rate		Mean Ct	Positive rate	Mean Ct	Positive rate		Mean Ct	Positive rate	Mean Ct	Positive rate
PerkinElmer	34.44	20/20	35.40	20/20	0.004	35.24	20/20	34.48	18/20	0.006
Wantai	36.07	20/20	35.75	20/20	0.26	36.21	19/20	35.73	20/20	0.45
KHB	31.14	20/20	30.78	20/20	0.02	32.55	20/20	33.42	18/20	0.14	31.3	20/20	30.77	20/20	0.01
Sansure	38.66	19/20	37.56	20/20	0.001	38.96	12/20	38.51	16/20	0.17
Da An	37.19	18/20	37.83	19/20	0.17	38.92	16/20	39.04	17/20	0.61
BioGerm	36.62	16/20	36.17	18/20	0.25	35.68	13/20	35.85	17/20	0.31
ABT	37.56	12/20	38.42	9/20	0.02	38.71	12/20	38.00	10/20	0.28	37.11	17/20	37.17	19/20	0.95

For N gene, the concentration was 90 copies/mL for PerkinElmer, 180 copies/mL for Wantai and KHB, and 360 copies/mL for other reagents. For ORF1ab gene, the concentration was 76 copies/mL for PerkinElmer, 152 copies/mL for Wantai and KHB, and 304 copies/mL for other reagents. For E gene, the concentration was 134 copies/mL for KHB and 268 copies/mL for ABT. Undetermined results were excluded.

ABT, Beijing Applied Biological Technologies Co, Ltd; BioGerm, Shanghai BioGerm Medical Biotechnology Co, Ltd; Da An, Da An Gene Co, Ltd; KHB, Shanghai Kehua Bio-Engineering Co, Ltd; Sansure, Sansure Biotech Inc.; Wantai, Beijing Wantai Biological Pharmacy Enterprise Co, Ltd.

For N geene">ne, the conceene">ntration was 90 copies/mL for Perkiene">nElmer, 180 copies/mL for Waene">ntai aene">nd n class="Chemical">KHB, and 360 copies/mL for other reagents. For ORF1ab gene, the concentration was 76 copies/mL for PerkinElmer, 152 copies/mL for Wantai and KHB, and 304 copies/mL for other reagents. For E gene, the concentration was 134 copies/mL for KHB and 268 copies/mL for ABT. Undetermined results were excluded.

ABT, Beijiene">ng Applied Biological Techene">nologies Co, Ltd; BioGerm, Shaene">nghai BioGerm Medical Biotechene">nology Co, Ltd; n class="Chemical">Da An, Da An Gene Co, Ltd; KHB, Shanghai Kehua Bio-Engineering Co, Ltd; Sansure, Sansure Biotech Inc.; Wantai, Beijing Wantai Biological Pharmacy Enterprise Co, Ltd.

When mixing 229E, OC43, seasonal n class="Species">H1N1, H3N2, influenza B, SARS, and Middle East respiratory syndrome with 900 or 180 copies/mL SARS-CoV-2 pseudovirus, all seven kits yielded expected results, indicating that these respiratory viruses do not cause false-negative results of detecting SARS-CoV-2 (Table 4 ). Organisms were also tested in the absence of SARS-CoV-2, and no false-positive test results were obtained. These results suggest all of the kits are designed for the specific detection of SARS-CoV-2, with no expected cross-reactivity to other coronaviruses or human influenza viruses mentioned.

Table 4

Cross-Reactivity of SARS-CoV-2 NAT Kits

Region	Reagent manufacturer	High concentration								Low concentration
		Control	229E	OC43	Seasonal H1N1	H3N2	Influenza B	SARS	MERS	Control	229E	OC43	Seasonal H1N1	H3N2	Influenza B	SARS	MERS
N	PerkinElmer	32.54	32.36	32.60	32.59	32.54	33.01	32.75	32.37	35.50	35.60	35.46	35.65	35.72	35.12	35.60	36.18
	Wantai	32.06	32.19	32.07	32.03	31.92	32.41	32.57	32.18	36.17	36.68	36.06	35.70	34.80	35.57	36.30	35.61
	KHB	26.31	26.80	26.39	26.48	26.37	26.78	26.19	26.89	29.30	30.02	29.73	29.48	29.44	29.17	28.95	29.25
	Sansure	33.01	32.12	33.15	32.88	32.76	33.03	33.08	32.97
	Da An	34.81	35.23	34.79	35.05	34.74	35.70	34.86	35.34
	BioGerm	33.45	33.27	33.13	33.11	33.24	33.33	33.73	34.11
	ABT	33.17	33.30	33.42	33.37	33.14	33.37	32.56	34.34
ORF1ab	PerkinElmer	32.40	32.39	32.43	32.76	32.53	32.60	32.61	32.38	35.47	35.37	35.16	35.07	35.49	35.49	35.33	35.99
	Wantai	31.82	31.55	31.94	31.84	32.10	32.13	31.90	32.03	35.04	34.68	34.62	35.56	35.31	34.39	35.17	34.76
	KHB	29.10	27.69	28.44	28.31	28.33	28.54	28.37	28.85	31.71	33.76	32.52	30.88	32.31	31.76	30.54	32.85
	Sansure	33.95	33.64	34.18	34.37	34.06	33.70	34.40	34.10
	Da An	36.05	36.22	35.60	35.95	35.63	36.53	35.72	35.74
	BioGerm	33.03	32.72	33.46	32.71	33.32	33.36	32.47	33.80
	ABT	34.08	34.35	32.85	34.88	33.37	34.09	32.89	35.50
E	KHB	26.56	26.70	26.26	26.39	26.47	26.92	26.33	26.86	29.42	30.37	30.12	29.53	29.48	29.24	28.74	29.20
	ABT	33.14	34.08	33.22	34.20	32.51	34.12	32.97	34.66

A total of 1 × 105 copies/mL of human coronavirus 229E (ATCC VR-740), OC43 (ATCC VR-1558), influenza A virus seasonal H1N1 (ATCC VR-1520), influenza A virus H3N2 (ATCC VR-1679), and influenza B virus (ATCC VR-1807), as well as 1 × 105 median tissue culture infective dose (TCID50/mL of pUC57 plasmid containing SARS coronavirus fragments and MS2-based MERS coronavirus pseudovirus, were mixed with PCDH-based SARS-CoV-2 pseudovirus. High concentration was 900 copies/mL of N gene, 760 copies/mL of ORF1ab gene, and 670 copies/mL of E gene. Low concentration was 180 copies/mL of N gene, 152 copies/mL of ORF1ab gene, and 134 copies/mL of E gene. Each concentration was extracted and tested three times in a single run, and mean Ct value was presented.

ABT, Beijing Applied Biological Technologies Co, Ltd; BioGerm, Shanghai BioGerm Medical Biotechnology Co, Ltd; Da An, Da An Gene Co, Ltd; KHB, Shanghai Kehua Bio-Engineering Co, Ltd; MERS, Middle East respiratory syndrome; NAT, nucleic acid test; Sansure, Sansure Biotech Inc.; SARS, severe acute respiratory syndrome; SARS-CoV-2, SARS coronavirus 2; Wantai, Beijing Wantai Biological Pharmacy Enterprise Co, Ltd.

A total of 1 × 105 copies/mL of human coronavirus 229E (ATCC VR-740), OC43 (ATCC VR-1558), n class="Species">influenza A virus seasonal H1N1 (ATCC VR-1520), influenza A virus H3N2 (ATCC VR-1679), and influenza B virus (ATCC VR-1807), as well as 1 × 105 median tissue culture infective dose (TCID50/mL of pUC57 plasmid containing SARS coronavirus fragments and MS2-based MERS coronavirus pseudovirus, were mixed with PCDH-based SARS-CoV-2 pseudovirus. High concentration was 900 copies/mL of N gene, 760 copies/mL of ORF1ab gene, and 670 copies/mL of E gene. Low concentration was 180 copies/mL of N gene, 152 copies/mL of ORF1ab gene, and 134 copies/mL of E gene. Each concentration was extracted and tested three times in a single run, and mean Ct value was presented.

ABT, Beijiene">ng Applied Biological Techene">nologies Co, Ltd; BioGerm, Shaene">nghai BioGerm Medical Biotechene">nology Co, Ltd; n class="Chemical">Da An, Da An Gene Co, Ltd; KHB, Shanghai Kehua Bio-Engineering Co, Ltd; MERS, Middle East respiratory syndrome; NAT, nucleic acid test; Sansure, Sansure Biotech Inc.; SARS, severe acute respiratory syndrome; SARS-CoV-2, SARS coronavirus 2; Wantai, Beijing Wantai Biological Pharmacy Enterprise Co, Ltd.

Discussion

SARS-CoV-2 is aene">n eene">nveloped siene">ngle-straene">nded positive-seene">nse RNA virus with a geene">nome leene">ngth of about 29.9 kb. To construct quality control materials that best simulate n class="Species">SARS-CoV-2 particles, vectors of lentivirus and phage are used in this study. Recombinant lentivirus vector is developed on the genome of HIV-1, in which the parental HIV-1 envelope is substituted with vesicular stomatitis virus G glycoprotein. According to the instruction, a PCDH lentivector can hold 3 to 5 kb of insert, and high-titer (about 109 infectious units/mL) concentrated viral particles can be generated. In this study, N gene, E gene, and parts of ORF1ab gene of SARS-CoV-2, with a total length of 4276 bp, were inserted into the PCDH vector and packaged into 293T cells. The titer of pseudovirus produced was about 1 × 109 copies/mL. Because the three regions were inserted together, it is assumed that the concentrations of three regions are the same. However, although droplet digital PCR can precisely quantify nucleic acids, the process is subject to bias and variance. Therefore, the concentrations of three regions are not identical.

Generally, packaging 500 bp of RNA is efficient using MS2, whereas it does not work well wheene">n packagiene">ng 1.5 kb of RNA. Although n class="Species">MS2-based armored L-RNA technology could package >2 kb RNA sequence, the packaging capacity of MS2 is still not as good as lentivirus. Because of the limitation, N gene, E gene, and nine parts of ORF1ab gene were inserted into the MS2 vector separately, and different MS2-based pseudoviruses were mixed together to form quality control materials. This raised the problem that the packaging efficiencies of different gene fragments were not exactly the same, so that the concentrations of pseudovirus containing different regions varied widely when mixing in the same volume ratio. In this study, digital PCR showed that the concentration of ORF1ab gene was about sixfold lower than N gene, and fourfold lower than E gene. The big difference between the levels of different genes made it not suitable to evaluate NAT kits using mixed MS2-based pseudoviruses.

SARS-CoV-2 particles are geene">nerally spherical, with some pleomorphism, aene">nd diameter varies from about 60 to 140 nm. Leene">ntivector is a spherical virus betweeene">n 80 aene">nd 120 nm iene">n diameter, eene">ncapsulatiene">ng a siene">ngle-straene">nded positive-seene">nse RNA of 7 to 12 kb iene">n leene">ngth. , n class="Species">MS2 is icosahedral capsid self-assembled from 180 copies of a single coat protein and measures 22 to 29 nm in diameter. Lentivirus-based pseudovirus is closer in size to SARS-CoV-2 particle, which makes it better to mimic SARS-CoV-2 virus during extraction and amplification. Consistent with this, the amplification efficiencies of PCDH-based pseudovirus in the seven kits evaluated are close to that of SARS-CoV-2 cell culture supernatant, suggesting that PCDH-based pseudoviruses are more similar to SARS-CoV-2 virus presented in real patient samples. For laboratories having the need to evaluate SARS-CoV-2 NAT kits, the characteristics among different pseudoviruses should be considered. Different evaluation materials may get different results.

SARS-CoV-2 is highly pathogeene">nic, aene">nd must be isolated aene">nd cultured iene">n Biosafety level-3 laboratory. This meaene">ns that most laboratories caene">nnot produce cell culture supernataene">nt as quality control materials. Moreover, virus needs to be iene">nactivated sufficieene">ntly before usiene">ng iene">n BSL-2. Poteene">ntial biological hazard of cell culture supernataene">nt makes pseudovirus safer as quality control material. To use as quality control material, supernataene">nt was heated at 65°C for 30 miene">nutes. However, 1000 copies/mL of heated n class="Species">SARS-CoV-2 cell culture supernatant was unstable when staying at room temperature for a week and then putting at 2°C to 8°C for another week. It has been proven that the Ct values in specimens from diagnosed COVID-19 patients increase after thermal incubation. Furthermore, about half of weak-positive samples containing about 700 copies/mL SARS-CoV-2 RNA tested negative after heating at 56°C for 30 minutes. Combining these lines of evidence, heat inactivation might not be suitable for producing quality control materials with SARS-CoV-2 cell culture supernatant.

With the outbreak of COVID-19, commercial kits detectiene">ng n class="Species">SARS-CoV-2 RNA also burst. Sensitivity is one of the most concerning questions about NAT kits. The median viral load in posterior oropharyngeal saliva or endotracheal aspirate was 5.2 log10 copies/mL. Salivary viral load was highest during the first week after symptom onset and subsequently declined, with a slope of –0.15 log10 copies/mL per day. For detecting viral RNA at the late time of disease, kits with adequate sensitivities are needed. Of the seven kits in this study, Da An and BioGerm reached the analytical sensitivity claimed in their instructions, which is 500 and 1000 copies/mL, respectively. LoD of Sansure, KHB, and ABT was higher than LoD in their instructions, which is claimed to be 200, 20, and 200 copies/mL, respectively. The analytical sensitivity declared of PerkinElmer and Wantai is 30 and 50 copies/mL, respectively, which is between the LoD of N and ORF1ab genes performed in our study. The sensitivities of ORF1ab gene were generally less than N and E genes. In addition, SARS-CoV-2 expresses nine subgenomic mRNAs containing N and E genes, whereas ORF1ab can only transcript from the whole genome. The subgenome results in more N and E gene copies than the less sensitive ORF1ab gene. Therefore, when detecting patient samples with low viral titers, it is normal to detect positive N or E gene with a negative result of ORF1ab gene. Samples should be retested in this situation following the instructions. The extraction systems were different in this study, so the LoD reflected the combined performance of both the extraction process and PCR process. Most kits do not restrict the matching extraction system. Therefore, extraction systems should be taken into consideration for sensitivity in laboratories detecting SARS-CoV-2 RNA. To increase analytical sensitivity, extraction systems can be optimized by increasing the volume ratio of elution to extraction or by increasing the extraction efficiency.

All of the seven kits showed high within-run precision and within-laboratory precision with the CV value of <5%, indicating a good repeatability of results. This result prompts that an indeterminate result should be confirmed with another more sensitive reagent, rather than multiple tests with one reagent. Specificity of each reagent was 100%, demonstrating that false-positive results are unlikely to appear. However, real patient samples are more complicated, aene">nd iene">nterfereene">nce of various substaene">nces may result iene">n false-positive results.

Sample type is diverse in COVID-19 n class="Species">patients. Virus RNA can be detected in nasopharyngeal and oropharyngeal swabs, sputum, bronchoalveolar lavage, endotracheal aspirate, blood, stool, and urine. In all these samples, swabs contain more cells that might be more susceptible by human genomic DNA interference. A total of 2 mg/L human genomic DNA showed no interference with the determination of positive results at the level of 900 copies/mL. For ORF1ab gene of KHB, and N gene of ABT, interfering group got a lower positive rate and a higher mean Ct value of detectable repeats than that of control group, suggesting that 2 mg/L human genomic DNA might interfere with the determination of result at levels near LoD in these two reagents. For ORF1ab gene of Sansure and BioGerm, the positive rate was higher in interfering group. However, interference substances are complicated in real patient samples; it is hard to say whether they would be better in detecting patient specimens with SARS-CoV-2 RNA at concentrations near LoD.

In summary, PCDH-based pseudovirus was characterized more closely to SARS-CoV-2, aene">nd was more suitable to evaluate n class="Species">SARS-CoV-2 NAT kits than mixed MS2-based pseudovirus. To produce quality control materials using SARS-CoV-2 cell culture supernatant, it should be considered that fundamental stability might be affected after heating inactivation. As for the seven kits evaluated, N gene was more sensitive than ORF1ab gene in most kits, whereas E gene was most sensitive among the three genes in KHB and ABT. PerkinElmer got the lowest LoD for N gene at 11.61 copies/mL, and the value was 34.66 copies/mL for ORF1ab gene. All of the kits showed good precision, with CV values of <5%, as well as acceptable anti-interference ability of 2 mg/L human genomic DNA. No cross-reactivity was observed with other respiratory viruses.