Analytical Performance Evaluation of Three Commercial Rapid Nucleic Acid Assays for SARS-CoV-2.

PURPOSE: To cope with the SARS-CoV-2 epidemic, several rapid nucleic acid assays have been approved for use, but the analytical performance has not been well evaluated. In this report, two key performance parameters, analytical sensitivity (limit of detection) and reproducibility, of three approved rapid nucleic acid assays were assessed using heat-inactivated SARS-CoV-2 culture supernatants quantified by digital PCR.
METHODS: The LOD (limit of detection) and reproducibility of three approved rapid nucleic acid assays using their own instruments were assessed, while the LOD and reproducibility of two assays on a 7500 Real-Time instrument were assessed at the same time.
RESULTS: Using their own instruments, 100% of samples with 1150 copies/mL viral RNA could be detected by the Da An and Coyote assays, while 90% of samples could be detected by the Ustar assay; yet, for 525 copies/mL and 287.5 copies/mL viral RNA, the detection rate of the Ustar assay was higher than that of either the Da An or Coyote assays. However, the three assays did not produce statistically significant results with the three different concentrations of viral RNA (P=0.46, 0.46 and 0.46). Using a 7500 Real-Time instrument, Da An and Coyote assays did not produce statistically significant results with the 1150, 525 and 287.5 copies/mL viral RNA (P>0.99, >0.99 and >0.99). The positive and negative detection rates of the three assays in the intra- and inter-assay stages were 100% on both their own instruments and the 7500 real-time PCR instrument.
CONCLUSION: Positive or strongly positive samples can be detected by the rapid nucleic acid assay, but the analytical performance should be optimized, and comprehensive evaluations are also required.

Introduction

The novel SARS coronavirus 2 (SARS-CoV-2), which can cause lethal coronavirus disease, poses a great threat to human health and life1,2 worldwide. The early detection of new infections affects patient management and the control of transmission; thus, rapid and easy-to-use diagnostic assays are urgently needed.3 The Real-time PCR-based diagnostic testing of respiratory specimens represents the diagnostic standard of SARS-CoV-2 infection;4,5 however, this technique is time-consuming (even up to 24 h for the result) and not always available, and the technical requirements can usually only be met by special training and in special PCR laboratories.

Currently, SARS-CoV-2 antigen-detecting rapid diagnostic tests are being extensively researched and applied; these require no special training and generate results quickly, in contrast to molecular-based testing.6–8 However, they are less methodologically sensitive than PCR.9–11 Thus, molecular-based tests are expected to be an accurate and reliable method for SARS-CoV-2 detection and convenient or conducive to widespread use. To cope with the SARS-CoV-2 epidemic, several rapid, molecular-based testing kits for SARS-CoV-2 have been produced. In this study, we provide a direct comparison of the key performance characteristics of three commercial rapid, molecular-based assays using a quantitative RNA reference panel for the assessment of their analytical sensitivity (limit of detection, LOD) and reproducibility.

Materials and Methods

Preparation of the Reference Standard Candidate

Reference standard material, heat-inactivated SARS-CoV-2 culture supernatants (BetaCoV/Wuhan/IPBCAMS-WH-01/2019, GenBank: MT019529.1) grown in Vero cells (ATCC CCL-81), was provided by Division II of In Vitro Diagnostics for Infectious Diseases, Institute for In Vitro Diagnostics Control, National Institutes for Food and Drug Control. The initial concentration of viral RNA was 460,000 copies/mL, quantified by digital PCR, and was diluted in viral transport media for the calculation of LOD and reproducibility.

Evaluation of the LOD and Reproducibility of Commercial Diagnostic Assays

For verification of the LOD, the reference material containing SARS-CoV-2 RNA was diluted to three concentrations, which were above, equal to and below the LOD of the three assays. Then, 10 replicates of each concentration of viral RNA were tested. The positive detection rates at each viral RNA concentration were calculated to compare the LODs of the three assays. Both intra- and interassay reproducibilities were assessed at concentrations higher than the LOD with 20 replicates as well as 20 negative clinical samples confirmed by real-time PCR. Five replicates of each assay were applied in the intra-assay stage, and four replicates of the intra-assay stage were included in the inter-assay stage. The positive and negative detection rates were calculated to compare the reproducibility of the three assays. In all tests, the negative and positive control material was consistently detected.

Da an Kit Assay

A total of 100 μL of diluted, heat-inactivated SARS-CoV-2 culture supernatants was added to 25 μL of nucleic acid extraction and purification reagent. Real-time reverse transcription PCR (rRT-PCR) for SARS-CoV-2 detection was performed using a Detection Kit for 2019-nCoV (PCR-Fluorescence) (Da An Gene Co., Ltd., Guangzhou, China). Each rRT-PCR had a final volume of 25 µL, including 5 µL of nucleic acid purification product and 20 µL of rRT-PCR mixture. The Da An kit detects the open reading frame 1ab gene (ORF1ab) and the nucleocapsid protein gene (N). The results could be concluded as positive when the Ct values of both the ORF1ab and N genes were ≤30; otherwise, if the Ct values of both the ORF1ab and N genes were >30, a negative result was considered. Results with ORF1ab or N gene Ct values ≤30 would be concluded as presumptive positive, and retesting would be required; if results with ORF1ab or N gene Ct values ≤30 were detected a second time, a positive result would be considered. The whole detection time was approximately 40 min.

Coyote Kit Assay

A total of 15 μL of diluted, heat-inactivated SARS-CoV-2 culture supernatants was added to 15 μL of respiratory tract specimen treatment reagent. rRT-PCR for SARS-CoV-2 detection was performed using a 2019-nCoV nucleic acid detection kit (PCR-Fluorescence) (Coyote Bioscience Co., Ltd., Beijing, China). Each rRT-PCR had a final volume of 52 µL on its own instrument, including 15 µL of purified nucleic acid product and 37 µL of rRT-PCR mixture, while each rRT-PCR had a final volume of 50 µL on its own instrument, including 15 µL of purified nucleic acid product and 35 µL of rRT-PCR mixture. The Coyote kit detects the ORF1ab and N genes. The results could be concluded as positive when the Ct values of both the ORF1ab and N genes were ≤27; otherwise, when the Ct values of both the ORF1ab and N genes were >27, a negative result was considered. Results with ORF1ab or N gene Ct values ≤27 would be concluded as presumptive positive, and retesting would be required; if results with ORF1ab or N gene Ct values ≤27 were detected a second time, a positive result would be considered. The whole detection time was approximately 40 min.

Ustar Kit Assay

One millilitre of extraction reagent was transferred into a 2019-nCoV automatic detection tube, and then 500 μL of diluted, heat-inactivated SARS-CoV-2 culture supernatant was added to the detection tube. Isothermal amplification real-time fluorescence assay was conducted on its supporting testing instrument. The Ustar kit detects the ORF1ab and N genes. The results could be concluded as positive when the Ct values of either the ORF1ab or N gene or both genes were ≤40; otherwise, when the Ct values of both the ORF1ab and N genes were N/A, a negative result was considered. When the result was “Invalid” or “No results”, retesting was required. The whole detection time was approximately 60 min.

Real-Time Reverse Transcription PCR Assay

The principle of both the Da An Gene and Coyote 2019-nCoV nucleic acid detection assays was real-time reverse transcription PCR (rRT-PCR). To compare the LOD and reproducibility of these two assays on the same platform, amplifications with the same samples using two reagents were carried out according to the manufacturers’ instructions on a 7500 Real-Time PCR instrument (Applied Biosystems, San Francisco, CA, USA).

Statistical Analysis

Comparisons between two groups were made using the Mann–Whitney U-test, and comparisons between three groups were made using analysis of variance (ANOVA). A p value less than 0.05 (two sided) was considered statistically significant. The analyses were performed using Prism 7.0 (GraphPad, La Jolla, CA, USA) software.

Results

Comparison of the LODs of the Three Rapid Nucleic Acid Assays

According to the manufacturers’ instructions, the LODs of the Da An and Ustar assays were both 500 copies/mL, and the LOD of the Coyote assay was 400 copies/mL; thus, the initial concentration of viral RNA (460,000 copies/mL) was diluted to 1150 copies/mL, 525 copies/mL and 287.5 copies/mL. Samples of the three concentrations were detected ten times using the three rapid nucleic acid assays separately. The results showed that 100% of samples with 1150 copies/mL viral RNA could be detected by the Da An and Coyote assays, while 90% of samples could be detected by the Ustar assay. The result of one sample showed a positive amplification curve without a Ct value. For samples with 525 copies/mL viral RNA, the results were different: 90% samples could be detected by both the Da An and Coyote assays, while 100% of samples could be detected by the Ustar assay. The ORF1ab gene of one sample could not be detected using the Da An assay; neither the N gene nor the ORF1ab gene of another sample could be detected using the Coyote assay, while all ten samples could be detected using the Ustar assay. For samples with 287.5 copies/mL viral RNA, 60%, 70% and 80% of those were detected by the Da An, Coyote and Ustar assays, respectively. Specifically, four samples and the N gene of three samples could not be detected using the Da An assay; three samples and the ORF1ab gene of two samples were not detected using the Coyote assay; and both N and ORF1ab gene of one sample, the N gene of three samples and the ORF1ab gene of one sample were not detected using the Ustar assay, besides one. Overall, the three assays were not significantly different when testing 1150 copies/mL, 525 copies/mL and 287.5 copies/mL of viral RNA (P=0.46, 0.46 and 0.46) (Table 1).

Table 1

Comparison of the LODs of the Three Rapid Nucleic Acid Assays

Assay	Copy Number of the RNA Transcript (Copies/mL)
	1150						525						287.5
	Da An		Coyote		Ustar		Da An		Coyote		Ustar		Da An		Coyote		Ustar
	N	ORF1ab	N	ORF1ab	N	ORF1ab	N	ORF1ab	N	ORF1ab	N	ORF1ab	N	ORF1ab	N	ORF1ab	N	ORF1ab
1	+/27.12	+/25.44	+/20.88	+/20.13	+/22.5	+/21.00	+/30.33	+/27.69	+/20.75	+/20.23	+/9.67	+/11.17	–	+/28.16	+/23.08	+/21.36	+/19.67	–
2	+/27.15	+/28.39	+/20.86	+/20.00	+/9.50	+/9.67	+/30.60	+/28.59	+/21.99	+/22.03	+/10.00	+/10.50	+/29.29	+/29.67	+/26.13	–	+/22.00	–
3	+/27.61	+/28.00	+/21.35	+/20.26	+/8.83	+/10.17	+/30.36	+/29.28	+/22.06	+/22.35	+/20.33	+/24.00	–	–	–	+/18.52	–	–
4	+/26.18	+/26.06	+/19.89	+/20.43	+/9.33	+/8.83	+/24.85	+/26.57	–	–	+/9.83	+/10.33	–	+/28.2	–	–	+/12.83	+/11.67
5	+/27.37	+/25.12	+/21.19	+/22.52	No value#	+/11.37	+/26.75	–*	+/21.47	+/22.62	+/12.00	+/10.5	+/27.7	+/27.55	–	–	–	+/30.33
6	+/26.40	+/25.06	+/22.06	+/21.35	+/11.33	+/10.67	+/26.09	+/27.04	+/24.09	+/26.45	+/11.00	+/10.5	–	–	+/23.34	+/22.55	+/30.00	+/13.00
7	+/25.62	+/25.16	+/20.65	+/19.75	No value	No value	+/27.44	+/26.93	+/21.54	+/17.65	+/13.17	+/12.67	–	–	+/24.28	+/22.74	No value	No value
8	+/26.59	+/25.45	+/19.90	+/19.47	+/11.00	+/10.67	+/26.86	+/26.12	+/21.52	+/20.08	+/13.33	+/11.17	+/29.41	+/29.52	+/21.97	+/19.45	–	+/14.67
9	+/26.90	+/25.14	+/21.32	+/20.36	+/10.5	+/10.67	+/27.04	+/27.09	+/22.10	+/22.93	+/8.83	+/10.00	–	–	–	–	–	+/17.67
10	+/25.72	+/25.86	+/20.37	+/19.21	+/11.83	+/11.37	+/27.45	+/27.59	+/23.05	+/21.57	+/11.17	+/11.50	–	+/25.95	+/23.25	–	+/14.00	-

Notes: –*, Could not be detected. No value#, Could be detected, but without a Ct value.

Comparison of the Reproducibility of the Three Rapid Nucleic Acid Assays

The reproducibilities of the three rapid nucleic acid assays were assessed by calculating the positive and negative detection rates of both the N and ORF1ab genes in the intra- and inter-assay stages. Samples with 1150 copies/mL viral RNA were used to detect the positive detection rates. The results showed that both the N and ORF1ab genes of samples with viral RNA could be detected by the three assays conducted on their own instruments. In the negative samples, no genes could be detected by the three assays. Thus, the positive and negative detection rates of the three assays in the intra- and inter-assay stages were 100%.

Comparison of the LOD and Reproducibility of Two Rapid Nucleic Acid Assays Conducted on a 7500 Real-Time PCR Instrument

The Da An and Coyote assays were performed on a 7500 Real-Time PCR Instrument, following the manufacturer’s instructions. The results showed that 90% and 100% of samples with 1150 copies/mL viral RNA could be detected by the Da An and Coyote assays, respectively, 100% and 90% of samples with 525 copies/mL viral RNA could be detected by the Da An and Coyote assays, respectively, and 80% and 90% of samples with 287.5 copies/mL viral RNA could be detected by the Da An and Coyote assays, respectively. Specifically, one sample with 1150 copies/mL viral RNA could not be detected by the Da An assay, but all samples with the same concentration of viral RNA could be detected by the Coyote assay. However, one sample with 525 copies/mL viral RNA could not be detected by the Coyote assay, but the N gene of one sample with the same concentration of viral RNA could be detected by the Da An assay. For samples with 287.5 copies/mL viral RNA, two samples and the N gene of two samples could not be detected using the Da An assay, and one sample and the N gene of one sample were not detected using the Coyote assay. Overall, two assays were not significantly different in the samples with 1150 copies/mL, 525 copies/mL and 287.5 copies/mL viral RNA (P>0.99, >0.99 and >0.99) (Table 2).

Table 2

Comparison of the LODs of Two Rapid Nucleic Acid Extraction Methods Conducted on a 7500 Real-Time PCR Instrument

Copy Number of the RNA Transcript (Copies/mL)	1150				525				287.5
	Da An		Coyote		Da An		Coyote		Da An		Coyote
Assay	N Gene	ORF1ab Gene	N Gene	ORF1ab Gene	N Gene	ORF1ab Gene	N Gene	ORF1ab Gene	N Gene	ORF1ab Gene	N Gene	ORF1ab Gene
1	+/27.31	+/26.38	+/25.93	+/22.59	+/28.24	+/28.74	+/24.47	+/22.25	–	–	+/25.68	+/22.70
2	+/26.91	+/26.20	+/26.46	+/22.72	+/27.11	+/28.38	+/25.96	+/23.21	–	+/28.69	+/21.48	+/25.41
3	+/25.96	+/27.64	+/24.35	+/22.15	+/27.28	+/27.67	+/23.27	+/24.89	+/28.24	+/28.53	+/25.17	+/22.56
4	+/25.74	+/26.40	+/26.29	+/22.60	+/27.16	+/27.14	+/24.94	+/22.45	+/26.69	+/26.71	+/26.03	+/24.03
5	+/25.39	+/25.90	+/26.03	+/23.70	–	+/28.36	+/26.7	+/23.94	–	–	+/26.14	+/25.23
6	–*	–	+/26.66	+/23.39	+/26.63	+/27.51	–	–	+/28.33	+/28.74	–	+/24.96
7	+/26.52	+/25.91	+/24.87	+/21.8	+/28.23	+/28.41	+/25.33	+/22.73	–	+/27.71	+/25.00	+/23.24
8	+/26.48	+/25.66	+/26.37	+/22.22	+/27.09	+/27.88	+/27.56	+/22.70	+/28.15	+/28.76	–	–
9	+/25.80	+/25.12	+/27.13	+/23.28	+/27.14	+/26.10	+/25.79	+/23.20	+/27.97	+/28.46	+/25.65	+/23.73
10	+/27.16	+/26.71	+/24.58	+/20.94	+/26.90	+/26.96	+/25.18	+/22.42	+/28.14	+/28.53	+/26.15	+/23.65

Note: –*, Could not be detected.

With regard to reproducibility, both the N and ORF1ab genes of samples with 1150 copies/mL viral RNA could be stably detected, and in the negative samples, no gene could be detected by either assay. Thus, the positive and negative detection rates of the two assays in the intra- and inter-assay stages were 100%.

Discussion

The rapid nucleic acid assay has high application value in the field during quarantine amid public health emergencies. The Ebola virus Congo epidemic demonstrated the need for POCT in the prevention and control of new outbreaks.12 To date, the COVID-19 epidemic has already spread to more than 240 countries worldwide. To prevent and control the epidemic, several rapid nucleic acid assays have been approved for use, which has greatly promoted the development and application of rapid molecular diagnosis worldwide. The superiority of the rapid nucleic acid assay was its excellent timeliness, as the time spent on the entire detection protocol was no more than one hour for three assays in this study. There was no difference between the Da An and Coyote assays in the verification of LOD and reproducibility when the assays were conducted on either their own instruments (P=0.46) or the 7500 Real-Time PCR Instrument (P>0.99) (Table 2). Additionally, the three assays exhibited good reproducibility, as the positive and negative detection rates were 100% in the intra- and inter-assay stages. However, compared to conventional rRT-PCR, the rapid nucleic acid assay is less sensitive, of which the LODs are mostly 400~1000 copies/mL.13,14 In this study, 90% and above samples with 525 copies/mL viral RNA, close to the LODs, could be detected by the three assays, but when the viral RNA was diluted to 287.5 copies/mL, all three assays showed a decreased positive detection rate. Therefore, for positive samples with low viral load, false-negative detection may be possible using the rapid nucleic acid assay clinically. Notably, the sensitivity of the Ustar assay was higher than that of the other two assays, and perhaps the sample volume was an important influencing factor. The Ustar assay is a POCT molecular diagnostic assay and needs 500 μL of the sample for detection, while the other two assays are rapid nucleic acid assays but not POCT, requiring 100 μL and 15 μL for the Da An and Coyote assays, respectively. Moreover, the rapid nucleic acid assay can only detect two to six samples at the same time, which makes it not applicable for large population screening. In the LOD evaluation, the results of the two samples showed a positive curve without a Ct value, which may be caused by data collection error within the instruments; therefore, the instrument performance should be improved.

Although the rRT-PCR test has become the standard method for SARS-CoV-2 infection diagnosis, in clinical practice, it is important to take into account the moment of the suspected infection, the patient’s medical history, the symptoms and the general clinical picture for a successful outcome of the diagnostic test.15,16 Molecular tests will have a greater chance of being positive at the onset of symptoms, while the search for IgM and IgG antibodies begins to yield positivity at approximately one month after the presumed infection, and the levels of these immunoglobulins remain high for long periods of time.17 Therefore, IgM and IgG antibody tests exhibited an effective complement to the false-negative results from a nucleic acid test for SARS-CoV-2 infection diagnosis after onset.18–20

In summary, this study was a simple comparison of the LODs and reproducibilities of three commercial rapid nucleic acid assays. The initial sample was preserved in a −80 °C freezer for a few days, so the results may be affected by the freshness of the samples, the sample dilution ratio, the operation process, and other factors. Therefore, our study does not represent the overall detection performance of these assays.

Conclusion

Rapid nucleic acid assays can be a powerful complement to conventional real-time PCR, but their performance still needs to be optimized, and comprehensive evaluations are also required. In general, the most appropriate assay should be selected according to the local epidemic trend of disease, the actual situation of the clinical samples, the purpose of detection, and the laboratory conditions.