Interlaboratory assessment of quantification of SARS-CoV-2 RNA by reverse transcription digital PCR.

The pandemic of the novel coronavirus disease 2019 (COVID-19) has caused severe harm to the health of people all around the world. Molecular detection of the pathogen, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), played a crucial role in the control of the disease. Reverse transcription digital PCR (RT-dPCR) has been developed and used in the detection of SARS-CoV-2 RNA as an absolute quantification method. Here, an interlaboratory assessment of quantification of SARS-CoV-2 RNA was organized by the National Institute of Metrology, China (NIMC), using in vitro transcribed RNA samples, among ten laboratories on six different dPCR platforms. Copy number concentrations of three genes of SARS-CoV-2 were measured by all participants. Consistent results were obtained with dispersion within 2.2-fold and CV% below 23% among different dPCR platforms and laboratories, and Z' scores of all the reported results being satisfactory. Possible reasons for the dispersion included PCR assays, partition volume, and reverse transcription conditions. This study demonstrated the comparability and applicability of RT-dPCR method for quantification of SARS-CoV-2 RNA and showed the capability of the participating laboratories at SARS-CoV-2 test by RT-dPCR platform.

Introduction

The ongoing coronavirus disease 2019 (COVID-19, originally named 2019-nCoV) pandemic due to severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has caused a worldwide health hazard since late 2019 over many countries and territories [1]. Globally, there have been 207,784,507 confirmed cases of COVID-19 in 20 months after the outbreak, including 4,370,424 deaths, reported to WHO [2]. The pathogen named SARS-CoV-2 is a novel member of the β-coronavirus genus, with a positive-sense, single-stranded RNA genome of ~30 kb, sharing approximately 50% sequence homology with the middle east respiratory syndrome coronavirus (MERS-CoV), 79% sequence homology with SARS-CoV, and 93.3% and 96.1% sequence identity with SARS-like bat coronaviruses RmYN02 from Rhinolophus malayanus and RaTG13 from Rhinolophus affinis [3, 4].

The accurate and timely diagnostic of SARS-CoV-2 is crucial for the control of further disease outbreaks. Reverse transcription quantification real-time PCR (RT-qPCR) was used as the gold standard method to detect SARS-CoV-2 RNA in the infection cases. However, RT-qPCR is a relative quantitative method that uses Cq values to illustrate the initial amount of template in the reaction [5]. This may result in variety when different primer and probe sets or commercial kits are used by operators among different laboratories. Thus, lots of work have been done to evaluate and assess the performance of the RT-qPCR method [6-11].

Furthermore, as a supplement to RT-qPCR, other methods based on PCR have been developed, validated, and evaluated, among which reverse transcription digital PCR (RT-dPCR) has been proposed [12]. Through partitioning the PCR solution into small reactions and applying Poisson statistics to the proportion of the negative partitions, RT-dPCR allows absolute quantification of RNA without depending on standard curves [13, 14]. Moreover, RT-dPCR has the superiority in sensitivity and resistance to inhibitors [13]. Thus, it was reported that RT-dPCR could be used in numerous situations:(1) as a complementary tool for the screening, monitoring, and diagnosis of COVID-19 patients, in particular for patients at a low viral load with ambiguous RT-qPCR results [15-19]; (2) for quantification of the aerodynamic nature of SARS-CoV-2 [20]; (3) for understanding of SARS-CoV-2 gene expression using RT-dPCR assays designed to quantify different target regions spanning genome of SARS-CoV-2 [21]; (4) being resistant to reaction inhibition, for quantification of SARS-CoV-2 viral load from crude lysate without nucleic acid purification [22].

To evaluate the application and improve the wider implementation of RT-dPCR for detection of SARS-CoV-2, some work had been done to compare RT-qPCR and RT-dPCR [23-25]. However, there is not much research to assess the comparability of RT-dPCR between different laboratories and platforms. Inactivated virus particles had been used as a reference standard quantified by RT-dPCR on multiple commercialized platforms, and a CV% of about 40% was observed [26]. In order to exclude the effect of the complex extraction step of virus which may introduce extra dispersion to the comparison results, in this study, an interlaboratory assessment on quantification of in vitro transcribed SARS-CoV-2 RNA by RT-dPCR of six different platforms has been organized and analyzed by the National Institute of Metrology, China (NIMC). Participants were requested to measure the copy number concentration of open reading frame (ORF) 1ab gene, nucleocapsid (N) gene, and envelope (E) gene. Laboratories could choose to use methods developed by their own or supplied by the organizer, which used SARS-CoV-2 primer and probe sets of World Health Organization (WHO) (for E gene) and China CDC (for ORF 1ab and N gene), to demonstrate their capability in the field of SARS-CoV-2 measurement by RT-dPCR.

Materials and methods

Preparation of SARS-CoV-2 RNA sample

A mixture of three in vitro transcribed RNA fragments containing ORF, E gene, and N gene of SARS-CoV-2 was prepared as the comparability study materials.

Sequences containing partial ORF 1ab gene (NC_045512.2, positions 13201-15600), full length E gene, and N gene of SARS-CoV-2 were synthesized by BGI (Beijing, China) to generate in vitro transcribed RNA molecules. These sequences were cloned into pBluescript II SK(+) vectors.

The SARS-CoV-2 ORF 1ab gene, E gene, and N gene plasmids were linearized with 15 U/μL BamHI (1010S), 10× K buffer (1010S, both Takara) and nuclease-free water in a final reaction volume of 100 μL for 3 h at 30 °C. The digest was separated by gel electrophoresis and the corresponding bands were purified using the Universal DNA Purification Kit (DP214, Tiangen Biotech (Beijing) Co., Ltd) with 30 μL of elution buffer. DNA concentration was estimated using NanoDrop.

To generate positive sense strand RNA, in vitro transcription (IVT) was performed using the MEGAscript T7 kit (AM1334, Thermo Fisher). The mixture contained 7.5 mM of each of ATP, CTP, GTP, and UTP; 1× reaction buffer; 2 μL T7 enzyme mix; and 8 μL (approximately 0.2 to 1.1 μg) of plasmid. Incubation was performed at 37 °C for 4 h followed by TURBO DNase treatment. The resulting RNA was purified using the MEGAclear™ Kit (AM 1908, Thermo Fisher). RNA transcripts were eluted with 100 μL of RNase-free water. Successful in vitro transcription was confirmed by analyzing the 1000-fold dilution using 2100 Bioanalyzer RNA 6000 Pico kit (Agilent) and the nucleic acid concentration was estimated using NanoDrop. Three RNA stock solutions were prepared at approximately 1 × 1010copy/μL in RNA storage solution (AM7001, Thermo Fisher) and stored at −70 °C along with the neat RNA stock. The study material was prepared using the one-step RT-dPCR result as an initial indicator of copy number concentration of ORF 1ab, E gene, and N gene. The three stocks of SARS-CoV-2 RNA transcripts were thawed, mixed, and diluted in RNA storage solution with 5 ng/μL of total RNA extracted from 293T human cells. The 293T human cell line was purchased from the National Infrastructure of Cell Line Resource and cultured by NIMC.

Homogeneity of the transcribed RNA sample

Homogeneity of the study material was assessed by the previously established one-step RT-dPCR method [19] which used the published ORF 1ab and N gene assays of the Chinese Center for Disease Control and Prevention and E gene assay of WHO. The primers and probes used are listed in Table S1. Under the same condition, ten units of the study materials were assessed, and three RT-dPCR replicates were analyzed per unit. Homogeneity data was analyzed by ANOVA.

Stability of the transcribed RNA sample

To determine the stability of the SARS-CoV-2 RNA study material during sample distribution, we performed a short-term stability (STS) study at a range of storage temperatures and times. The RNA samples were maintained at −70 °C (reference temperature), 4 °C, 27 °C, or on dry ice for 3 and 7 days. Three replicate units were included per condition. Stability was assessed by one-step RT-dPCR mentioned earlier, with measurements of ORF 1ab and E gene. Three RT-dPCR replicates were analyzed per unit.

The long-term stability (LTS) of the study material was assessed by analyzing the RNA samples after 3, 4, and 9 months of storage, respectively, at the reference storage temperature of −70 °C. The sampling and analysis methods were the same as STS, except that all the three genes were measured in the LTS study.

Reference value (RV) and uncertainty of transcribed SARS-CoV-2 RNA sample

The RV of the study material was represented as the copy number concentration of ORF 1ab, E gene, and N gene of SARS-CoV-2 RNA. The assignments of the copy number values for each gene were carried out at 2 days using the one-step RT-dPCR mentioned earlier by NIMC. Four vials of samples were tested each day with five replicates for each vial.

Uncertainties of SARS-CoV-2 RNA were calculated by taking into account the following factors: characterization of the samples (uchar, including replicate measurement of target genes, volume of droplet, and calibration of pipettor), homogeneity, and stability of the samples (ubb and ults). Expand uncertainties (U) were obtained by multiplying standard uncertainty (u) by a coverage factor (k) of 2 using Eq. (1):

Participants

Ten laboratories participated in this comparability study and all of the laboratories submitted copy number concentration results of SARS-CoV-2 RNA within the designated time. Two laboratories reported results of two genes (E and N) and other eight laboratories reported results of all three genes. All the participating laboratories are denoted by serial numbers in this paper.

Instrumentation, methods, and reagents of the participants

A total of six different digital PCR platforms, including QX200 (Bio-Rad, CA, USA), SG-2000 (RainSure, Suzhou, China), Accu ONE (ZhenZhun, Shanghai, China), Drop Maker M1 (TargetingOne, Beijing, China), MicroDrop-100 (Forevergen, Guangdong, China), and DQ24 (Sniper, Suzhou, China), were used by ten laboratories in this comparability study.

For the primer and probe assays, the organizer supplied premix of the three genes (duplex of ORF 1ab and E gene, single-target of N gene) to all the participants, in case for any laboratory being not able to perform the detection due to lack of assays, but they could still choose to use the supplied assays or their own established assays. Of the ten laboratories, the majority (9/10) used the supplied assays, with one laboratory using their own established assay.

For the analytical method, nine laboratories employed one-step RT-dPCR approach using kits corresponding to the specific dPCR platforms. One laboratory employed a two-step RT-dPCR approach, where complementary DNA (cDNA) was synthesized in a separate reaction tube prior to dPCR.

All the specific details about the assay performance of the laboratories are listed in Table 1.

Table 1

Summary of the RT-dPCR platforms and methods of the participants

ID	Platform	Approach	RT-PCR mix	Primer/probe	Reaction/sample volume	Thermal cycling conditions
1	QX200	One-Step	One-Step RT-ddPCR Kit for Probes (Bio-Rad)	Supplied by NIMC	20 μL/3 μL	45 °C, 60 min; 95 °C, 10 min; 40 cycles of (95 °C, 30 s, 56 °C, 1 min); 98 °C, 10 min
2	QX200	One-step	One-Step RT-ddPCR kit for Probes (Bio-Rad)	Supplied by NIMC	20 μL/4 μL	45 °C 10 min; 95 °C 3 min; 40 cycles of (95 °C 15 s, 58 °C 30 s); 98 °C 10 min
3	QX200	One-step	One-Step RT-ddPCR kit for Probes (Bio-Rad)	Developed by participant	20 μL/2 μL	46 °C 60 min; 95 °C 10 min; 40 cycles of (95 °C 30 s, 58 °C 60 s); 98 °C 10 min
4	SG-2000	One-step	RT-ddPCR Master Mix for Probes (RainSure)	Supplied by NIMC	25 μL/4 μL	50 °C 20 min; 95 °C 10 min; 40 cycles of (95 °C 30 s, 55 °C 60 s); 98 °C 10 min
5	Accu ONE	Two-step	DigitalAmp PCR Master Mix (Zhenzhun)	Supplied by NIMC	20 μL/1.25 μL	37 °C 20 min; 95 °C 10 min; 39 cycles of (95 °C 30 s, 58 °C 45 s)
6	Drop Maker M1	One-step	RT-ddPCR Master Mix for Probes (TargetingOne)	Supplied by NIMC	30 μL/1 μL	55 °C 15 min; 95 °C 10 min; 40 cycles of (95 °C 30 s, 58 °C 1 min); 98 °C 10 min
7	MicroDrop-100	One-step	One-Step RT-ddPCR mix for Probes (Forevergen)	Supplied by NIMC	20 μL/2 μL	55 °C 30 min; 95 °C 10 min; 45 cycles of (95 °C 30 s, 60 °C 60 s); 98 °C 10 min
8	QX200	One-step	One-Step RT-ddPCR kit for Probes (Bio-Rad)	Supplied by NIMC	20 μL/2 μL	42 °C 60 min; 95 °C 10 min; 40 cycles of (95 °C 30 s, 58 °C 60 s); 98 °C 10 min
9	QX200	One-step	One-Step RT-ddPCR kit for Probes (Bio-Rad)	Supplied by NIMC	20 μL/4 μL	45 °C 10 min; 95 °C 5 min; 40 cycles of (95 °C 15 s, 58 °C 30 s); 98 °C 10 min
10	DQ24*	One-step	One-Step RT-ddPCR kit for Probes (Bio-Rad) - E gene	Supplied by NIMC	20 μL/4 μL	45 °C 10 min; 95 °C 5 min; 40 cycles of (95 °C 15 s, 58 °C 30 s); 98 °C 10 min
			One-Step RT-dPCR mix (Sniper) - ORF and N gene	Supplied by NIMC	20 μL/4 μL	37 °C 30 min; 95 °C 5 min; 40 cycles of (95 °C 15 s, 58 °C 30 s)

*Two different kits/conditions were used by participant 10

Results and discussion

Assignment and assessment of the study materials

The study materials consisted of three in vitro transcribed RNA fragments of SARS-CoV-2 ORF 1ab gene, E gene, and N gene, which are commonly targeted by molecular diagnostic tests, buffered in RNA storage solution with extracted total RNA of 293T human cell line as background. The length, purity, and integrity of each transcript had been checked and confirmed before mixing and preparing.

According to the genome structure of SARS-CoV-2 [27], the length of ORF 1ab, E gene, and N gene is 21,290 nt, 228 nt, and 1260 nt, respectively. In this study, the transcripts cover the full length of E gene and N gene and a partial fragment of ORF 1ab gene. ORF 1ab cannot be detected by assays designed out of the location described in the “Preparation of SARS-CoV-2 RNA sample” section.

The copy number concentration values of three genes of SARS-CoV-2 RNA were determined through RT-dPCR [19] by NIMC and denoted as the RVs of the study materials. The RT-dPCR method specifically designed to detect ORF 1ab, E gene, and N gene was developed by NIMC and had been validated and demonstrated to be applicable to clinical detection of SARS-CoV-2 RNA. Uncertainty of the samples derived from characterization, uchar, potential between-unit heterogeneity, ubb, and potential degradation after long-term storage, ults. The assigned values and uncertainties of the three genes of SARS-CoV-2 RNA are listed in Table 2. The RVs of ORF 1ab, E gene, and N gene were 2277, 1196, and 1596 copy/μL, respectively; the relative expanded uncertainty (k = 2) values were 14%, 17%, and 15%, respectively.

Table 2

RVs and uncertainties of the study materials

Gene	RV (copy/μL)	Relative uncertainty (%)					u (copy/μL)	U (k = 2) (copy/μL)
		uchar,rel	ubb,rel	ults,rel	urel	Urel (k = 2)
ORF 1ab	2277	2.1	0.7	6.6	7.0	14	159	319
E	1196	4.2	0.9	7.0	8.3	17	99	199
N	1596	4.8	0.9	5.5	7.4	15	118	236

To evaluate the differences in copy number concentration between tubes of the study materials, homogeneity study was carried out using the RT-dPCR method. No significant difference between tubes was observed (Fig. S1) and the samples were thought to be homogeneous enough. Participants were provided with three units of the study materials.

The effect of shipping conditions on copy number concentration of the study materials was assessed through STS study by RT-dPCR. ORF 1ab and E gene were measured after being stored at 4 °C, 27 °C, and on dry ice for 3 and 7 days. Results showed no systematic effect on copy number concentration (Fig. S2). Thus, the samples were shipped on dry ice and were ensured to be received within 7 days and under the protection of dry ice. LTS study was carried out to assess the value of the samples after being stored below −70 °C for a period of time. No significant change of the value was found for ORF 1ab, E gene, and N gene after 9 months (Fig. S3).

Copy number concentration results of participants

Overview of the participants’ results

The participants were required to submit the copy number concentration of SARS-CoV-2 RNA. For ORF 1ab gene, eight of ten laboratories submitted the results; for E gene and N gene, all the ten laboratories submitted the results. The submitted results and expanded uncertainties (k = 2) of the participants are plotted in Fig. 1. Among the ten laboratories, five used Bio-Rad QX200 dPCR platform, and the others used five different dPCR platforms.

Fig. 1

Participants’ results for copy number concentration of SARS-CoV-2 RNA ORF 1ab gene, E gene, and N gene. Results of the five laboratories using dPCR platforms of Bio-Rad are indicated with red circle, and results of the five laboratories using other platforms with blue diamond. Solid and dashed horizontal lines indicate RV and expanded uncertainty (95% level of confidence, k = 2) of each gene

For ORF 1ab gene, five laboratories’ results agreed with the RV within expanded uncertainty at a 95% level of confidence, with interlaboratory reproducibility expressed as CV% of 17%. For E gene, nine laboratories’ results agreed with the RV within expanded uncertainty at a 95% level of confidence with CV% of 17%. For N gene, six laboratories’ results agreed with the RV within expanded uncertainty at a 95% level of confidence with CV% of 23%.

Of the three target genes, N gene displayed the largest dispersion of the dataset with a ratio of highest to lowest value being 2.2, and ORF 1ab and E gene showed a similar ratio of 1.6 and 1.8. The distribution of the participants’ results is displayed in Fig. 2. This corresponds a variation less than 1.2 of Cq value in qPCR. The results indicate the quantification results from multiple platforms showed high degree of consistency, although different reagents and protocols were used.

Fig. 2

Scatter diagram showing the copy number concentration of different gene targets obtained from all participants

Potential reasons for discrepancies of the results

Five out of 10 participants used Bio-Rad dPCR platform, and the results of these participants were scattered rather than clustered in Fig. 1. This indicates platform is not the primary reason for the difference of the results. Of the possible reasons which caused dispersion of the results, partition volume is a key factor for the accurate quantification of RNA/DNA by the dPCR method according to Poisson distribution [28]. All the participants used a partition volume designated by the software of each platform. It is thus deduced that calibration of the partition volume may be helpful and necessary to improve the consistence of different dPCR platforms.

The reverse transcription step is another vital factor for quantification of RNA. In this study, One-Step RT-ddPCR Advanced Kit for Probes of Bio-Rad which integrates reverse transcriptase with DNA polymerase in one tube was used by laboratories 1, 2, 3, 8, and 9. RT temperature of 45 °C was used by these laboratories except 46 °C used by laboratory 3 and 42 °C used by laboratory 8. It is noted that the result of laboratory 8 was significantly lower than that of other labs using the same dPCR platform, which may suggest RT efficiency of One-Step kit of Bio-Rad is higher at RT temperature of 45 °C than that of 42 °C. This was further investigated by NIMC, by setting RT temperature at 42 °C, 45 °C, 48 °C, and 50 °C. It is shown that quantification results of both ORF 1ab and E gene are slightly higher at 45 °C than other temperatures (Fig. S4), indicating that 45 °C may be an optimal RT temperature for one-step RT-ddPCR kit of Bio-Rad. Other laboratories used reverse transcriptases suitable for corresponding platforms, with RT temperature ranging from 37 to 55 °C.

One laboratory (laboratory 5) used a two-step method whose result is the highest for N gene but rather low for E gene (result for ORF 1ab gene had not been submitted). Indeed, it is the same case in laboratories 2, 5, and 7, which used a one-step approach. This means one- or two-step approach has no evident impact on the results. However, the deviation varies by genes, due to the different RT and PCR amplification efficiencies of different assays.

Laboratory 10 used one-step RT-dPCR mix of both Bio-Rad (for E gene) and Sniper (for ORF 1ab and N gene), because no positive signals were observed for E gene using Sniper’s mix but normal positive and negative signals were obtained when changing the Sniper mix to Bio-Rad mix. The possible reason may be that the assay of E gene developed on Bio-Rad dPCR platform was not applicable for reagent of Sniper platform.

Of the two laboratories that did not submit results of ORF 1ab gene, laboratory 3 used assays developed by their own and perhaps the amplification region of primers and probe was not within the genome location of the transcripts; laboratory 5 used assays supplied by NIMC which were developed on QX200 dPCR platform of Bio-Rad and it may not be applicable for Accu ONE dPCR platform of ZhenZhun.

Z′ scores of participants’ results

Considering more than one dPCR platform was used in the comparability study, the organizer did not supply a standard protocol along with the primers and probes to the participants, including the minimum amount of sample, the thermal cycling conditions, and the reverse transcription strategy. Thus, dispersion of results was unsurprising. In order to evaluate the participants’ results by a method determined by the RVs as well as the dispersion of the participants’ results, a Z′ score was introduced to the data analysis [29].

Z′ score was calculated by Eq. (2):

where X is the participant’s result, X0 is the RV, s is the estimation of the dispersion of the participants’ results, and u is the standard uncertainty of the RV.

s was calculated by Eq. (3):

where NIQR is the normalized interquartile range and IQR is the interquartile range.

This evaluation method depends on uncertainty of the RV and dispersion of all the results.

Assessment of the results followed these principles:

The result is as satisfactory when │Z′│≦2

The result is ambiguity when 2 < │Z′│ < 3

The result is not as satisfactory when │Z′│≧ 3.

The Z′ scores of all the participants for the three target genes are displayed in Fig. 3 and listed in Table 3. For the three target genes, all the participants’ results were satisfactory with │Z′│≦2. Five laboratories for ORF 1ab gene, six for E gene, and eight for N gene had a │Z′│≦1, indicating these laboratories got closer results to RV.

Fig. 3

Z′ scores of the participants

Table 3

Summary of the results and Z′ scores of the participants

ID	ORF 1ab			E			N
	Results (copy/μL)	U (k = 2)	Z′	Results (copy/μL)	U (k = 2)	Z′	Results (copy/μL)	U (k = 2)	Z′
1	1771	196	−0.9	1024	72	−0.9	1666	275	0.2
2	1920	62	−0.6	1045	37	−0.8	880	27	−1.8
3	/	/	/	1205	95	0.0	1935	369	0.8
4	1603	93	−1.2	1182	73	−0.1	1781	112	0.5
5	/	/	/	954	81	−1.2	1977	102	1.0
6	1503	50	−1.4	796	24	−2.0	1098	41	−1.2
7	2394	204	0.2	1317	144	0.6	1265	129	−0.8
8	1520	103	−1.4	900	97	−1.5	1227	98	−0.9
9	2277	96	0.0	1196	102	0.0	1596	154	0.0
10	2411	182	0.2	1446	115	1.3	1571	96	−0.1

The Z′ score is a common value to evaluate performance of testing laboratories. The ten laboratories provided consistent result for quantification of copy number concentration of SARS-CoV-2 RNA by RT-dPCR of six different dPCR platforms according to the calculated Z′ scores.

Conclusion

Development of reliable diagnostic methods of SARS-CoV-2 is essential to control the disease spread. The current study provides evidence that the RT-dPCR method using multiple platforms can be applied to measure ORF 1ab, E gene, and N gene of SARS-CoV-2 RNA. Using in vitro transcribed RNA as study materials, ten laboratories took part in the interlaboratory comparison study of SARS-CoV-2 RNA. All the reported results were as satisfactory from an evaluation method of Z′ scores. The dispersion among different dPCR platforms and laboratories was within 2.2-fold, suggesting RT-dPCR can be potentially considered as a reference method for quantification of SARS-CoV-2 RNA to improve the measurement comparability. The comparison result shows the participating laboratories have developed capability of SARS-CoV-2 test by RT-dPCR platform. Calibration of the partition volume and controlling of the performance, like RT steps under optimum conditions, are required to further improve the interlaboratory comparability of SARS-CoV-2 RNA tests by dPCR platforms.

(DOCX 229 kb)