Evaluation of factors contributing to variability of qualitative and quantitative proficiency testing for SARS-CoV-2 nucleic acid detection.

The pandemic caused by SARS-CoV-2 has led to unprecedented social and economic disruption. Many Nucleic acid testing (NAT) laboratories in China have been established to control the epidemic better. This proficiency testing (PT) aims to evaluate the participants' performance in qualitative and quantitative SARS-CoV-2 NAT and to explore the factors that contribute to differences in detection capabilities. Two different concentrations of RNA samples (A, B) were used for quantitative PT. Pseudovirus samples D, E (different concentration) and negative sample (F) were used for qualitative PT. 50 data sets were reported for qualitative PT, of which 74.00% were entirely correct for all samples. Forty-two laboratories participated in the quantitative PT. 37 submitted all gene results, of which only 56.76% were satisfactory. For qualitative detection, it is suggested that laboratories should strengthen personnel training, select qualified detection kits, and reduce cross-contamination to improve detection accuracy. For quantitative detection, the results of the reverse transcription digital PCR (RT-dPCR) method were more comparable and reliable than those of reverse transcription quantitative PCR (RT-qPCR). The copy number concentration of ORF1ab and N in samples A and B scattered in 85, 223, 50, and 106 folds, respectively. The differences in the quantitative result of RT-qPCR was mainly caused by the non-standard use of reference materials and the lack of personnel operating skills. Comparing the satisfaction of participants participating in both quantitative and qualitative proficiency testing, 95.65% of the laboratories with satisfactory quantitative results also judged the qualitative results correctly, while 85.71% of the laboratories with unsatisfactory quantitative results were also unsatisfied with their qualitative judgments. Therefore, the quantitative ability is the basis of qualitative judgment. Overall, participants from hospitals reported more satisfactory results than those from enterprises and universities. Therefore, surveillance, daily qualitiy control and standardized operating procedures should be strengthened to improve the capability of SARS-CoV-2 NAT. .

Introduction

As of July 2022, over 568,77 million confirmed cases and over 6.38 million deaths caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) have been reported by WHO [1], [2], [3]. Now, SARS-CoV-2 is constantly mutating, and the rapid propagation and concealment of mutants such as Omicron [4], Alpha, and Beta [5] have made epidemic prevention and control extremely difficult [6], [7]. In addition, the rapid spread of the virus has led to unprecedented social and economic disruption, with the government ordering schools and businesses to close.

Apart from epidemiologic clustering and virus‐specific diagnostic testing, there is little clinically that distinguishes SARS-CoV-2 infection from other severe viral respiratory infections such as severe acute respiratory syndrome or influenza [8]. Nucleic acid testing (NAT) is the preferred method for detecting SARS-CoV-2 [9]. Suitable specimens for this testing include nasopharyngeal swabs, oropharyngeal swabs, nasal washes, and nasal aspirates [10], [11]. This test methodology, also considered the gold standard of testing, has been the mainstay of COVID-19 diagnosis in the world [12]. NAT is a complex procedure, and its accuracy depends on well-defined laboratory practices and quality systems [13]. Under the continuous spread of the epidemic, the urgent and massive demand for testing led to rapid development and validation of commercial and laboratory-developed assays to detect SARS-CoV-2. Testing laboratories should accurately and promptly identify SARS-CoV-2 by their established methods. However, given the massive number of laboratories doing SARS-CoV-2 testing using different methods and kits, reliability and accuracy of results have always been a concern. Accurate results of SARS-CoV-2 testing are used for patient management, infection control in health care settings, and surveillance data that drive decisions on community-wide sheltering orders. Therefore, the accuracy of the testing results is the cornerstone of epidemic prevention and control.

Globally, it is almost impossible to deploy such a wide range of testing capabilities through administrative means and ensure the accuracy of testing results. Therefore, an objective evaluation of system-wide product and laboratory measurement capability is needed. PT is an essential tool for monitoring the diagnostic proficiency of laboratories and providing results that allow the implementation of improved testing [14]. It is the procedure in which a government agency or professional institution sends identical samples to participating laboratories for blind analyses, and the results are then compiled and used as the basis for judging the testing ability of individual laboratories [15], [16]. In this study, we designed a SARS-CoV-2 quantitative PT in parallel with a qualitative PT to evaluate the factors affecting the performance of laboratory testing to provide information and guidance for the current clinical application of SARS-CoV-2 testing. Challenges and experience with implementing inter-laboratory comparison across a vast network of COVID-19 testing laboratories are also discussed.

Methods

Participants

The PT for SARS-CoV-2 detection was initiated and implemented by The National Institute of Metrology of China (NIM). Forty-eight laboratories, mainly those engaged in SARS-CoV-2 nucleic acid testing, registered for qualitative PT, and 42 of them registered for quantitative PT. These participants include laboratories from hospitals, academics, and enterprises. For the confidentiality of the PT, the laboratories were coded in sequence from NA01-001 to NA01-048. All participants were distributed in 14 provinces and cities in China (Supplementary Fig. S1).

Samples used in the PT

This PT consists of two parts: qualitative detection and quantitative measurement. The samples used are divided into two types to satisfy different application directions: purified RNA extracted from inactivated SARS-CoV-2, including samples A and B, and pseudovirus obtained by transfection cells by segmental construction of novel coronavirus sequences onto lentivirus vectors including sample D and E [17] (ORF1ab was longer in total length, it consisted of pseudoviruses of different segments). Pseudovirus samples were designed to evaluate the detection, including the extraction step.

Samples A and B were used for quantitative measurement, while samples D, E, and F were used for qualitative detection. Samples A and B were SARS-CoV-2 RNA samples with different concentrations. Samples D and E were diluted from a pseudovirus RM (NIM-RM5207), and sample F was a negative control. Two vials of each sample were distributed to all the participants. The reference values are listed in Table 1 .

Table 1

The information of panel samples.

Category	Sample name	Type	Copy number concentration ± expanded uncertainty (Copies/mL, k=2)
			ORF1ab Gene##	N Gene
Quantitative PT	A	SARS-CoV-2 Genomic RNA*	(2.60±0.28) ×105	(6.67±0.80) ×105
	B	SARS-CoV-2 Genomic RNA**	(7.00±2.00) ×103	(2.50±1.00) ×104
Qualitative PT	D	Pseudovirus#	(266∼2719) (7.55±2.50) ×102	(5.91±2.81) ×102
			(2720∼8554) (7.45±2.30) ×102
			(8555∼13024) (9.09±0.28) ×102
			(13025∼18039) (1.15±0.34) ×103
			(18040∼21552) (4.91±1.50) ×102
	E	Pseudovirus#	(266∼2719) (3.77±2.50) ×102	(2.95±1.41) ×102
			(2720∼8554) (3.73±2.30) ×102
			(8555∼13024) (4.55±0.28) ×102
			(13025∼18039) (5.73±0.34) ×102
			(18040∼21552) (2.45±1.50) ×102
	F	TE buffer	/	/

Sample A was certified reference materials GBW (E) 091099 (This reference material was the purified RNA genome of the 2019-nCoV, including the entire length of ORF1ab, N, and E genes. For details, please refer to supplemental File S2, or visit https://www.ncrm.org.cn/Web/Ordering/MaterialDetail?autoID=20123)

Sample B was diluted 30 times from GBW (E) 091099.

Sample D and E were diluted from NIM-RM5207 in different batches, and D was diluted 110 times, E diluted 220 times.

ORF1ab gene in samples D and E includes different segments with different concentrations. The final result is judged according to different fragments; refer to the certificate of NIM-RM5207 in supplemental File S1, or visit https://www.ncrm.org.cn/Web/Ordering/MaterialDetail?autoID=23949.

PT procedure

Since the primary purpose of the proficiency testing project is to examine the comprehensive testing ability of the laboratory. Participating laboratories can choose different methods, such as digital PCR (RT-dPCR) [18] or real-time quantitative PCR (RT-qPCR) [19], and use a kit commonly used in the laboratory or laboratory-developed tests. Samples D and E should be extracted before detection, and the extraction method should also be selected independently. The operating protocol referred to the manufacturer’s instructions.

When the RT-dPCR method was used, the PCR reaction was prepared according to the manufacturer’s instructions. Briefly, five μL of the sample was added to 15 μL of PCR reaction solution, and the PCR reaction mixture was transferred to generate droplets and amplified. Three replicated RT-dPCR measurements were required for each sample. The copy number concentration of ORF1ab and N were reported for samples A and B. Positive, or adverse results were reported for samples D, E, and F.

When using RT-qPCR, five μL of the sample was added to 20 μL of PCR reaction solution for amplification. The qualitative results of samples D, E, and F were directly judged by CT value according to the instruction of the testing kits. The standard curve should first be established for quantitative results using a serial dilution of certified SARS-CoV-2 reference material. Then test according to the instructions. Three replicated measurements were conducted for each vial. The copy number concentration of ORF1ab and N in samples A and B can be calculated according to its corresponding target’s standard curve.

Data analysis

For qualitative PT, each participant was assessed according to the consistency of sample attributes. The quantitative results of PT often need to be transformed into capability statistics for interpretation and comparison with other determined objectives. Its purpose is to measure deviations from the assigned values based on competency assessment criteria. As RMs were used in the PT, the reference values of RM can be used directly for the assigned values of the PT. Therefore; the z'-score was used to estimate the performance of each laboratory when the participant’s result did not affect the assigned value [20], [21]. The z'-score fraction formula is as follows:

x is the proficiency test result from the participant;

X is the assigned value;

σ is the standard deviation for proficiency assessment (determined by a normalized interquartile range of the participant);

u is the uncertainty of the assigned value.

Each participant was assessed according to the z'-score evaluation criteria shown in Table 2 .

Table 2

z'-score evaluation criteria [20], [22], [23]

z'-score	Judgment
| z' | ≤ 2	Satisfactory result
2 < | z' | < 3	Questionable result (Warning signal)
| z' | ≥ 3	Unsatisfactory result (Action signal)

When the result of the participant is not used for the determination of the assigned value, the z'-score can be calculated

Based on the fact that the logarithmic function is a monotonic increasing function in its definition domain, the relative relationship of data will not be changed after logarithmic processing. Therefore, the copy number of quantitative PT was processed by log10 and analyzed by violin and box diagram.

Homogeneity and stability of the PT samples

The previously established digital PCR method [18] was used to analyze the stability and homogeneity [18]. The homogeneity and stability of the six samples were tested at NIM. For samples A and B, the homogeneity was evaluated by F-test. For pseudovirus samples D and E, the whole vial would be extracted at once but not necessary to evaluate the homogeneity within the bottle. Therefore, the standard deviation among different vials was calculated as S, and the S was compared with the method’s repeatability (R). The result showed that all the samples were homogeneous (supplemental Table S1).

Short-term stability was assessed to ensure stability during the transportation. Copy number concentration under 4℃ was compared with the reference temperature of -70℃ for five days. Three vials were randomly selected at each time point under 4℃. Trend analysis was used to check if a significant change occurred. The results showed that the samples were stable at 4℃ for five days (supplemental Table S2). Therefore, the sample was shipped with dry ice to ensure the temperature was below 4℃, and all participants received samples within four days.

3. Results

Overall evaluation of qualitative PT

Fifty sets of valid data (including ORF1ab/N gene detection for sample D/E/F) were submitted from 48 participants (two of which provide two data sets with two methods). The overall accuracy was 88.00%, 84.00%, and 92.00% for samples D, E, and F, respectively (Table 3 ). Among them, eleven participants presented false negatives (either for sample D or E), indicating that problems existed in the operation or testing process of the laboratory personnel involved. Sample F was a negative control, but 4 participants reported false positives indicating contamination occurred in these laboratories. Only 74.00% of the participants reported correctly for all samples, which is relatively lower compared with 83.10% of 2019-nCoV nucleic acid detection inter-laboratory quality evaluation conducted by the National Health Commission Clinical Laboratory Center in 2020. We speculated that the reason was that the copy number concentrations in our study were relatively lower, especially the copy number concentration in sample E even lower than the detection limit of testing kits used by some participants. Interestingly, the false negative rate increased from 12.00% to 16.00%, decreasing the concentration from about 600 copies/mL (Sample D) to 300 copies/mL (Sample E), as sample E was more challenging regarding its low copy number concentration.

Table 3

Overall performance in SARS-CoV-2 qualitative proficiency testing.

Sample	Sample D	Sample E	Sample F	All correct
Total number	50	50	50	50
False negative/positive number	6	4+4*	4	13
Correct rate	88.00%	84.00%	92.00%	74.00%
False negative/positive rate	12.00%	16.00%	8.00%	26.00%

*The detection limit of the kits used by some participating laboratories is higher than the gene concentration of sample E

Factors impacting testing performance in qualitative PT

CT distributions had no preference

Among the 50 sets of data, eight were detected by RT-dPCR and 42 by RT-qPCR. RT-qPCR was the primary method used for the detection of SARS-CoV-2 nucleic acid. The distribution of CT values and detection kits, equipment, and extraction kits used by all participants in the qualitative PT were plotted (Fig. 1 , Fig. S2), and no CT values preference was observed. The CT values were distributed similarly and concentrated between 35 and 40 for two targets in two samples (Fig. 1A). The majority of CT values of sample E were slightly higher than that of sample D, which is consistent with the two-fold difference in concentration of the two samples. Significant fluctuation of CT values was observed for those participants using DaAn test kits and the CF96 RT-qPCR platform. No significant factors directly affected the accuracy were found in the qualitative PT.

Fig. 1

Violin diagram of CT value distribution for qualitative PT CT value of ORF1ab and N in samples D and E (A); CT value of ORF1ab in Sample D with different nucleic acid detection kits. (B), different participants (C) and on different equipment (D). Less than or equal to 3 categories were not included in the statistics.

Hospitals have better testing capabilities and quality control systems

Among the participants, 31 were from enterprises, 11 from hospitals, and six from academics (research institutes and universities). The accuracy rates were 100%, 83.33%, and 64.52% for participants from hospitals, academics, and enterprises, respectively (Table 4 ). For SARS-CoV-2 nucleic acid testing, hospitals have better testing capabilities and quality control systems with rigorous and accurate results. Some enterprises (14/31) were the in vitro diagnostic (IVD) manufacturers, who focus on testing kit development, and some enterprises (17/31) were the independent clinical laboratories (ICL, authorized and unauthorized). The error rate of IVD manufacturers was 28.60%, while the error rate of ICL was as high as 41.20% (Table 4). Furthermore, this indicates that enterprise participants, mainly from ICL, must improve the quality system, including personnel training on qualifications, quality control, and supervision.

Table 4

Classification of the participants in SARS-CoV-2 qualitative proficiency testing.

Classification	Kit (50 sets) *		Method (50 sets) *		Institution (48)
	Approved	RUO**	RT-qPCR	RT-dPCR	Enterprises		Hospitals	Universities
Kind	12	11	/	/	ICL	IVD	/	/
Number of tests (Set)	37	13	42	8	17	14	11	6
Number of errors (Set)	5	6	8	3	7	4	0	1
Correct rate	86.49%	53.85%	80.95%	62.50%	58.82%	71.43%	100.00%	83.33%
Error rate	13.51%	46.15%	19.05%	37.50%	41.18%	28.57%	0.00%	16.67%

Exclude errors caused by sample F.

RUO, research use only.

Approved kits for better fault tolerance and compatibility

All participants used twenty-three different testing kits, of which 12 were approved by the National Medical Products Administration (NMPA), and 11 were not approved yet. In the face of major infectious diseases worldwide, most enterprises and academics have actively participated in the development and production of kits, and the overall R&D capabilities of the nucleic acid detection industry have been improved. Most participants chose the approved testing kits for the PT (Table 4). Excluding the errors in the detection of sample F entirely caused by the contamination of laboratory operations, considering comprehensive factors such as the type of kits and the level of participants [24], the error rate was 13.51% and 46.15% for those using approved kits and RUO kits, respectively. The low error rate of approved SARS-CoV-2 nucleic acid detection kits indicates that approved detection kits are relatively stable, reliable, and fault-tolerant. National supervision contributes significantly to the improvement of the accuracy of SARS-CoV-2 detection.

Additionally, RT-dPCR showed a higher error rate than RT-qPCR because the kits used for RT-dPCR were under development, as none of them were approved by NMPA by the time of running the PT.

Overall evaluation of quantitative PT

A total of 42 laboratories participated in the quantitative PT. A few laboratories submitted partial results (submit ORF1ab or N only). The initial copy number concentration is shown in Fig. S3. The result of the z'-score for samples A and B was shown in Fig. 2 . The satisfactory results of ORF1ab and N in sample A accounted for 77.22%, the questionable results for 20.25%, and the unsatisfactory results accounted for 2.53%. The satisfactory results of ORF1ab and N in sample B accounted for 93.67%, the questionable results accounted for 1.27%, and the unsatisfactory results accounted for 5.06% (Fig. 2.). The quantitative results of ORF1ab in sample A showed a higher percentage of unsatisfactory/questionable than others. The possible reason is that the entire length of the ORF1ab gene was about 21kbp; more diverse primers and probes were used compared with the N gene (1.3kbp). Different primer/probes may lead to inconsistent amplification efficiency. It was observed that most participants used RT-qPCR with a standard curve to report the copy number concentration. Therefore, errors in preparing standard curves for each target were introduced due to unreasonable dilution gradients and additional operations, leading to a significant deviation in the quantitative result. Furthermore, the majority of z'-score was negative for the two targets of the samples, indicating RNA degradation occurred during the quantitative process.

Fig. 2

z'-score diagram for quantitative PT. The red dotted line is the absolute value of 2, and the blue dotted line is 3. Incomplete display when it is higher than 4.

The data was shown as logarithm ten due to the scattered copy number concentration. Through boxplot and violin plot analysis (Fig. 3 .), it was found that the medians of the tested samples A-N and B-ORF1ab were located in the middle of the entire boxplot, and the distances between the upper and lower edges are similar, indicating that the overall data distribution level of the samples was relatively uniform. One outlier was checked out for sample B-N (Fig. 3A). It was found that the ORF1ab results of sample B were concentrated on two peaks according to the violin plot (Fig. 3B). This indicates the results were biased and showed different aggregation areas in different laboratories. The quantitative capacity of the participants needs further improvement.

Fig. 3

Trend of quantitative PT results. A) Box diagram of samples A and B. B) Violin diagram of samples A and B. X-axis represented sample types; Y-axis was the result of log10 (lg) processing for different sample copy numbers (After logarithmic processing, the relative relationship of data will not change).

Factors impacting testing performance in quantitative PT

Digital PCR method with better comparability for quantitative measurement

For quantitative PT, 30 participants reported results with RT-qPCR and 9 with RT-dPCR. Therefore the total number of RT-dPCR tests was 36 times, and the number of unsatisfactory/questionable tests accounted for 8.33%. On the other hand, the total number of RT-qPCR tests was 122, of which 16.39% were unsatisfactory/questionable tests. Generally, RT-qPCR has more strict requirements for operators and other factors due to its quantification relying on an external standard curve, which may cause an increase in the error rate. Therefore, results obtained from the RT-dPCR method were more comparable and reliable for quantitative measurement because RT-dPCR provides quantification results independent of an external standard.

In the entire quantitative proficiency testing, the hospital’s satisfactory rate was as high as 92.86%, with only two questionable or not unsatisfactory (Table 5 ). Moreover, the satisfaction data of universities was only 75.00% (Table 5), which indicates that colleges and universities were primarily scientific research and exploration and compared with hospitals, they lack quality system construction. Therefore, regardless of qualitative or quantitative research, a relatively complete quality system and special training of personnel can ensure the accuracy of different methods and capabilities.

Table 5

Summary result of SARS-CoV-2 quantitative proficiency testing.

Classification	Type					Method		Institution **
	Sample A		Sample B		Both*	RT-qPCR	RT-dPCR	Enterprises		Hospitals	Universities
	ORF1ab	N	ORF1ab	N	/	/	/	ICL	IVD	/	/
Number	39	40	39	40	37	122	36	50	60	28	20
Satisfactory	23	38	37	37	21	102	33	44	50	26	15
Questionable	15	1	0	1	16	15	2	6	6	2	3
Unsatisfactory	1	1	2	2		5	1	0	4	0	2
Percentage of satisfactory	58.94%	95.00%	94.87%	92.50%	56.76%	83.61%	91.67%	88.00%	83.33%	92.86%	75.00%
Percentage of unsatisfactory/ questionable	41.03%	5.00%	5.13%	7.50%	43.24%	16.39%	8.33%	12.00%	16.67%	7.14%	25.00%

ORF1ab and N genes were calculated separately.

Submit complete data for sample A and sample B. If a laboratory was dissatisfied with one data, it was dissatisfied.

a number of experimental results were submitted by different institutions.

The higher the concentration, the greater the fluctuation of the quantitative results

The fluctuation was compared between samples A and B to explore the relations between variability and different sample concentrations. The fluctuation of ORF1ab and N in sample A was much more extensive than that of sample B. By excluding the outliers (Grubbs’ Test), the data of sample A dispersed in a range of 85 and 223 folds for ORF1ab and N, respectively. The results of sample B were scattered in a range of 50 and 106 folds for ORF1ab and N, respectively. Additionally, for sample A, 18 out of 79 were unsatisfactory/questionable, accounting for 22.78%. For sample B, 5 out of 79 were dissatisfied, accounting for 6.33%. The proportion of unsatisfactory was higher for sample A with high concentration, and most of them were used by the RT-qPCR method, indicating that the higher the sample concentration, the greater the fluctuation for quantitative determination.

The establishment of the standard curve of each laboratory is quite different

The participants selected nine kinds of reference materials using RT-qPCR [25]. One hundred sixteen standard curves need to be constructed, of which 74.20% were reference materials using NIM (GBW(E)091099, GBW(E)091089, and GBW(E)091090) (Fig. 4 A). Linear correlation was analyzed between CT value and copy number under the same conditions of reference material (GBW(E)091099), target gene and detection kit (DaAn). The correlation coefficients were 0, 0.32, -0.30, and 0.10 for A-ORF1ab, A-N, B-ORF1ab, and B-N, respectively (Fig. 4B). It was found that | R2 | was close to 0, indicating that the correlation between the establishment or operation of standard curves in different laboratories was very poor. Preparing the standard curve is a critical factor that leads to a big difference in quantitative results. Thus the procedure should be standardized.

Fig. 4

Proportion of reference materials used (A) and correlation analysis between CT and copies value(B). B) Analyzed the data in the same conditions of reference material, target gene, and detection kit. The correlation was reflected in the personnel operation and the establishment of the standard curve.

Accurate quantitative measurement ensuring reliable qualitative detection

To further explore the impact of quantitative measurement on qualitative detection, we compared the satisfactory and unsatisfactory participants in both quantitative and qualitative PT. There were 22 laboratories with both correct (satisfactory) qualitative and quantitative results, accounting for 95.65% of the satisfactory quantitative PT. Most of the laboratories with correct quantitative results in quantitative PT can report correctly in qualitative PT (Fig. 5 A). There were six laboratories with both incorrect (unsatisfactory) qualitative and quantitative results, accounting for 85.71% of the unsatisfactory qualitative PT (Fig. 5B). Most laboratories with incorrect qualitative results were also unsatisfied with quantitative results. Thus, quantitative ability is the basis of qualitative judgment.

Fig. 5

Analysis of laboratories with correct (A) and incorrect (B) results in qualitative and quantitative PT. Only analyze the data involved in both qualitative and quantitative PT and exclude the error results of sample F (cross-contamination) and sample E caused by detection limit issue.

4. Discussion

Evaluation of PT results from a large group of laboratories helps assess the performance of test methods applied in various conditions [26]. The significance of this measure is compounded when a test is new and widely practiced, and the results have several applications. Through this PT, it was found that the overall results are not very satisfactory regarding qualitative or quantitative PT. Only 74.00% of the laboratories were utterly correct in qualitative PT, and the number of satisfactory quantitative participants accounted for 56.76% (all genes were satisfied). Especially the copy number concentration fluctuated in the range of tens to hundreds of times. Given the spread of such a large-scale worldwide epidemic, it is still tricky to promptly establish a complete and accurate detection system. Therefore, on the one hand, it is necessary to strengthen the training and management of personnel within the laboratory. On the other hand, it is necessary to deepen the supervision and construction of the quality system comprehensively to cope with the outbreak of a series of infectious diseases such as SARS and MERS (Fig.6 ) [27].

Fig. 6

Main factors contributing to the PT. The red box represents the problems in this PT, and the dotted line on the left represents the solution direction.

National supervision and the development of various verification projects play an essential role in improving nucleic acid detection capabilities

By analyzing some laboratories with poor performance, incorrect results can be attributed to two main reasons: 1) Cross-contamination. Many laboratories reported false positive results for sample F, indicating sample contamination was caused by personnel operation and other reasons. 2) Low sensitivity of NAT (Fig. 6). Eleven laboratories report false negatives for samples D or E. On the one hand, the test kits used have not been registered and approved by NMPA, with the performance not sufficiently verified and evaluated. On the other hand, the stability and sensitivity of the detection kit are insufficient, and the detection limit is not enough. However, testing laboratories from hospitals showed a better performance than those from enterprises in this PT. Testing laboratories from hospitals have been engaged in medical and health testing for a long time and often participate in various verification evaluations, its supervision and management are rigorous. Meanwhile, these participants used approved testing kits with more stable performance, leading to better accuracy. A lack of enough evidence for the unapproved testing kits could easily lead to an inaccurate test. Therefore, systems such as “CE” certification based on “self-declaration” are temporarily unsuitable for China. National supervision and PT program can be essential in promoting and optimizing infection disease detection.

Accurate quantitative detection is the cornerstone of qualitative judgment, and the ability of quantitative detection needs to be improved

Through this PT, we found that there are still many contents worth discussing low-concentration samples and accurate quantitative detection. Several unsatisfactory results were reported for samples D and E. Participating laboratories used different extraction and elution volumes according to different kit instructions (supplemental Table S3). Due to the extraction steps, it is easy to cause a low amount of extracted samples by low extraction efficiency [28], [29], which may result in false negatives. Furthermore, the CT values of low-concentration samples were near the gray area, and it was not easy to decide whether it was negative or positive without further checking. Thus, using the weak positive control material in the daily control is necessary. For quantitative measurement, the deviation of high-concentration samples was extensive, resulting in many unsatisfactory results. Although participating laboratories can give good results for negative and positive judgments of high concentration samples, the capability for quantitative measurement is still poor. Especially for these IVD manufacturers, the limit of detection (LoD) of the testing kits cannot be determined accurately without qualified quantitative capability. All qualitative detection results were judged according to the testing kits with reliable LoD. Therefore, it is necessary further to improve the quantitative measurement capability of NAT for SARS-CoV-2.

The establishment of standardized processes and traceability systems is the basis for ensuring accurate nucleic acid detection

Overall, establishing a traceability system and a standardized process is also critical to comprehensively improve the accuracy of nucleic acid detection, to ensure the kit’s stability and personnel qualification. Before starting the detection, the problems caused by the equipment should be eliminated first, so the calibration should be carried out in time. At the same time, focus on strengthening the operation training of testing personnel, especially implementing reference materials and standard procedures. Good traceability delivery and system establishments are essential to support and prerequisites for ensuring nucleic acid detection.

In conclusion, this study showed that not all laboratories could obtain a satisfactory result, especially those from ICL need further improve their quality system, including a selection of qualified testing kits, personal training, standardizing operation, daily quality control, and regular external evaluation, etc. However, continuing to participate in various verification programs, improving the overall detection capability of the laboratory, paying attention to technical details, and staff training can significantly avoid testing errors and provide help for epidemic prevention and control.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.