Development and Evaluation of AccuPower COVID-19 Multiplex Real-Time RT-PCR Kit and AccuPower SARS-CoV-2 Multiplex Real-Time RT-PCR Kit for SARS-CoV-2 Detection in Sputum, NPS/OPS, Saliva and Pooled Samples.

Rapid and accurate detection of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is essential for the successful control of the current global COVID-19 pandemic. The real-time reverse transcription polymerase chain reaction (Real-time RT-PCR) is the most widely used detection technique. This research describes the development of two novel multiplex real-time RT-PCR kits, AccuPower® COVID-19 Multiplex Real-Time RT-PCR Kit (NCVM) specifically designed for use with the ExiStation™48 system (comprised of ExiPrep™48 Dx and Exicycler™96 by BIONEER, Korea) for sample RNA extraction and PCR detection, and AccuPower® SARS-CoV-2 Multiplex Real-Time RT-PCR Kit (SCVM) designed to be compatible with manufacturers' on-market PCR instruments. The limit of detection (LoD) of NCVM was 120 copies/mL and the LoD of the SCVM was 2 copies/μL for both the Pan-sarbecovirus gene and the SARS-CoV-2 gene. The AccuPower® kits demonstrated high precision with no cross reactivity to other respiratory-related microorganisms. The clinical performance of AccuPower® kits was evaluated using the following clinical samples: sputum and nasopharyngeal/oropharyngeal swab (NPS/OPS) samples. Overall agreement of the AccuPower® kits with a Food and Drug Administration (FDA) approved emergency use authorized commercial kit (STANDARD™ M nCoV Real-Time Detection kit, SD BIOSENSOR, Korea) was above 95% (Cohen's kappa coefficient ≥ 0.95), with a sensitivity of over 95%. The NPS/OPS specimen pooling experiment was conducted to verify the usability of AccuPower® kits on pooled samples and the results showed greater than 90% agreement with individual NPS/OPS samples. The clinical performance of AccuPower® kits with saliva samples was also compared with NPS/OPS samples and demonstrated over 95% agreement (Cohen's kappa coefficient > 0.95). This study shows the BIONEER NCVM and SCVM assays are comparable with the current standard confirmation assay and are suitable for effective clinical management and control of SARS-CoV-2.

Introduction

Coronavirus disease 2019 (COVID-19) was first detected in Wuhan, China in 2019, and its outbreak has spread to other countries which led to a global pandemic [1]. The virus that causes COVID-19 was named severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), which is the seventh known coronavirus that can infect humans [2]. According to World Health Organization (WHO), as of June 30, 2021, approximately 181 million people were confirmed with SARS-CoV-2 infection, and 3.9 million were dead world-wide. (https://covid19.who.int/).

The key strategy for controlling outbreaks of COVID-19 is early and accurate detection of SARS-CoV-2 in the community. Real-time reverse transcription polymerase chain reaction (Real-time RT-PCR) a gold standard method in the detection of various viral diseases, is also the most reliable and accessible method for the diagnosis of SARS-CoV-2 infection [3]. SARS-CoV-2 is a positive-sense single-strand RNA virus that consists of RNA-dependent RNA polymerase (RdRp) in an ORF1ab [4], envelope, nucleocapsid, spike, and membrane protein. The genes of these regions have been chosen as the target for the detection of SARS-CoV-2 [5]. RNA viruses have a high tendency for multiple mutations. The mutations in the RNA sequence can decrease the detection ratio of the primers and probes, which may lead to the increased false-negative rates. Thus, the primers and probes in this study are designed to target multiple conserved regions to minimize the false negatives caused by mutations.

Two novel multiplex Real-Time RT-PCR kits, AccuPower® COVID-19 Multiplex Real-Time RT-PCR Kit (Cat No. NCVM-1111, BIONEER, Korea) and AccuPower® SARS-CoV-2 Multiplex Real-Time RT-PCR Kit (Cat No. SCVM-2112, BIONEER, Korea), were used in this study to detect the three viral genes of SARS-CoV-2 (RdRp gene, E gene, and N gene). The NCVM is a premixed product that is specifically designed to be used with the ExiStation™48 system (ExiPrep™48 Dx & Exicycler™96, BIONEER, Korea), in which the test is processed automatically from RNA extraction to PCR detection and confirmation. The SCVM has been developed for use with various other manufacturers’ PCR instruments. The limit of detection (LoD), cross-reactivity, and precision of the AccuPower® kits (SCVM and NCVM) were evaluated with SARS-CoV-2 positive materials as well as commonly used human clinical samples (sputum samples and nasopharyngeal swab (NPS)/oropharyngeal swab (OPS) samples). In addition, the clinical performance of the AccuPower® kits was verified for use in the NPS/OPS specimen pooling test and with saliva samples.

Materials and methods

Primers and probe design

Primers and probes were designed to detect SARS-CoV-2 RNA according to two guidelines, the WHO Interim guideline and the KDCA (Korea Disease Control and Prevention Agency) guideline. The primers and probes target three different genes of SARS-CoV-2 (RdRP gene, N gene, and E gene). In silico analysis for inclusivity was conducted by comparing primers and probes for an alignment with all COVID-19 sequences (n = 3037) in the GISAID database as of April 9th, 2020. The MUSCLE alignment was generated by multiple sequence alignment and viewed in Jalview. In silico analyses were performed against the updated standard database (n = 1,060,413 May 31, 2021) of the National Center for Biotechnology Information to confirm the current coverage of primers and probes. The coverage change of primers and probes was not significant. The target genes and coverage of each primer or probe are stated in Table 1.

Table 1

The target genes and coverage of primer and probe of the AccuPower® kits.

Target Gene	Oligomer	Coverage (%)	Max. Coverage (%)	Current Coverage (%)
E gene	Forward primer	99.97%	100%	99.94%
	Reverse primer	99.96%
	Probe	99.96%
RdRp gene 1	Forward primer	99.99%	100%	99.94%
	Reverse Primer1	99.93%
	Reverse Primer2	100%
	Probe	100%
RdRp gene 2	Forward primer	99.99%	100%	99.89%
	Reverse primer	99.98%
	Probe	99.98%
N gene	Forward primer	96.67%	99.97%	95.06%
	Reverse primer	99.40%
	Probe	99.89%

AccuPower® Kits

AccuPower® kits contain specific primer, specific dual-labeled fluorogenic (TaqMan®) probe, DNA polymerase, reverse transcriptase, dNTPs, and stabilizer. Primers and fluorescent probes attach specific sequences, which distinctively appear in the SARS-CoV-2 gene. TaqMan® probe contains the fluorescence in the 5’ end and the quencher in the 3’ end so the fluorescence is not released in the usual state. The fluorescence signal is emitted as the 5’-3’ exonuclease in DNA polymerase detaches from the probe, while the fluorescence and the quencher detach during PCR.

The RNA presence can be detected by fluorescence signals. The NCVM is a freeze-dried premixed product for use only with the ExiStation™48 system (ExiPrep™48 Dx & Exicycler™96). The SCVM is a master-mix product, which can be used with various PCR instruments. PCR reaction of AccuPower® kits was conducted according to each manufacturer’s protocol.

Analytical performance evaluation

The LoD, cross-reactivity, and precision were analyzed using the AccuPlex™ SARS-CoV-2 Verification Panel (Virus-Like Particles, SeraCare, USA) for the NCVM, and SARS-Related Coronavirus 2 (Isolate USA-WA1/2020) for the SCVM. The RNA of the SARS-CoV-2 Panel was extracted after dilution with Pan-Sarbecovirus gene and SARS-CoV-2 gene negative NPS/OPS or sputum matrix, then the RT-PCR process was performed. RNA extraction and PCR were performed on the ExiStation™ 48 system. RNA extraction of the SARS-Related Coronavirus 2 was performed using the ExiPrep™ 48 DX (BIONEER, Korea) after dilution of the NPS/OPS or sputum matrix, following the manufacturer’s instructions.

Limit of detection

The LoD measurement was performed, following the CLSI guideline EP17-A2 [6]. The LoD for the Pan-Sarbecovirus gene and the SARS-CoV-2 gene were determined by measuring the RNA level in the AccuPlex™ SARS-CoV-2 Verification Panel (Virus-Like Particles, SeraCare, USA) using the NCVM, and the SARS-Related Coronavirus 2 (Isolate USA-WA1/2020) using the SCVM.

The AccuPlex™ SARS-CoV-2 Verification Panel were serially diluted to 6 concentration levels (240, 200, 160, 120, 80, 40 copies/mL) for the NCVM testing. Each dilution was tested in total 40 replicates, 20 replicates per lot, 2 lots. The SARS-Related Coronavirus 2 (Isolate USA-WA1/2020) also were serially diluted to 6 concentration levels (6, 5, 4, 3, 2, 1 copies/μL) for the SCVM testing. Each dilution was tested in total 60 replicates, 20 replicates per lot, 3 lots. Both LoD tests were performed on the Exicycler™96 (BIONEER, Korea) and determined using hit-rate analysis as the concentration at the lowest dilution that can be detected with >95% probability.

Cross-reactivity

The cross-reactivity test was performed according to WHO EUL guidance [7] and Food and Drug Administration (FDA) guideline [8]. The specificity of AccuPower® kits for SARS-CoV-2 detection was evaluated by in silico analysis followed by testing the SARS-CoV-2 control with respiratory disease-related viruses and bacteria genes (concentration above 1.0x106 copies/mL). Altogether, 29 respiratory disease-related microorganisms were tested using the NCVM and 38 were tested using the SCVM. The cross-reactivity test was performed using the Exicycler™96.

Precision

The repeatability was tested according to the CLSI Guideline EP05-A3 [9] and EP15-A3 [10]. The repeatability of the AccuPower kits performance was evaluated by intra-assay and inter-assay variations at different concentrations of NPS/OPS, sputum, and saliva samples.

Ethics statement and sample collection

Authors confirm that this paper has complied with PLOS policies on research ethics, as stated in submission guide. Since all identification information of samples was anonymized by replacing it with the unique identification code of the institution, researchers have performed the clinical performance test without risks of identifying or tracing personal information with the sample. The protocol of the study was approved by the Institutional Review Board (IRB) of Myongji Hospital and IRB of Kangwon National University Hospital. Since samples were residual sample with guaranteed anonymity and there was no need to collect additional samples for this study, there was no risk to research subjects in accordance with ‘Bioethics and Safety Act’ and Article 13 of ‘Enforcement Rules of the In Vitro Diagnostic Medical Devices Act’. Hence, IRBs confirmed that this study does not need subject consent. This study was conducted in accordance with the principles of the Declaration of Helsinki, 1964.

Myongji Hospital Institutional Review Board (IRB) approved the use of surplus clinical samples for the NPS/OPS specimen-pooling (IRB No.: MJH 2020-12-028) and clinical performance evaluation (IRB No.: MJH 2020-12-029) tests. Kangwon National University Hospital IRB approved the use of saliva and NPS/OPS samples for the clinical performance testing of the AccuPower® kits (IRB No.: KNUH-2021-03-014), which were either surplus samples or de-identified patient samples drawn after informed consent (Fig 1). RNA in the NPS/OPS and sputum samples was extracted using the ExiPrep™48 Viral DNA/RNA Kit (BIONEER, Korea) with the ExiPrep™48 Dx instrument and the RNA of saliva samples was extracted using the ExiPrep™48 Fast Viral RNA Kit (BIONEER, Korea) with the ExiPrep™48 Dx instrument. Previously, it was determined whether each sample was SARS-CoV-2 positive or negative by the confirmation test.

Fig 1

Summary of clinical specimens analyzed in this study.

NPS, Nasopharyngeal Swab; OPS, Oropharyngeal Swab.

Clinical performance evaluation test

The clinical performance of AccuPower® kits was evaluated by comparing the PCR result of each AccuPower® kit to that of the confirmation test with confirmed positive or negative samples. A total of 343 sputum samples and 343 NPS/OPS samples were collected for clinical performance evaluation. The distribution of the Ct value of positive samples was described in Supplementary S1 Fig. At least 30% of positive samples had Ct values within the cut-off Ct value—10. The confirmation test was performed with the STANDARD™ M nCoV Real-Time Detection kit, the Q-Sens® COVID-19 Detection Kit V2 (CancerRop, Korea), or the Allplex™ 2019-nCoV Assay (Seegene, Korea) beforehand at the sample collection institute. Presuming the result of the confirmation test was true, the clinical sensitivity and specificity of AccuPower® kits were calculated. In addition, to validate the application of AccuPower® kits as alternative diagnostic kits for SARS-CoV-2, the detection rate of confirmed positive and negative samples was compared to samples tested using the AccuPower® kits and the reference kit (STANDARD™ M nCoV Real-Time Detection kit). STANDARD™ M nCoV Real-Time Detection kit is approved by the WHO and the FDA for SARS-CoV-2 detection for Emergency Use Authorization (EUA) and officially approved the first kit by the Ministry of Food and Drug Safety (MFDS), Korea. The RNA in samples was extracted using the ExiPrep™48 Viral DNA/RNA Kit with the ExiPrep™48 Dx instrument. PCR reaction was conducted using the Exicycler™96.

Nasopharyngeal Swab and Oropharyngeal Swab Specimen Pooling Test

A 5-pool test on NPS/OPS samples was performed to evaluate the performance of the AccuPower® kits on the pooled sample. A total of 180 samples (30 positive samples and 150 negative samples) were tested individually and in pools of 5 samples with the reference kit (STANDARD™ M nCoV Real-Time Detection kit) and the AccuPower® kits. At least 25% of the positive samples had Ct values within the cut-off Ct range of 2~3. The 30 positive pooled samples and 30 negative pooled samples were prepared. Experimental positive pools were created using 80 μL from one SARS-CoV-2 positive specimen mixed with 4 negative patient specimens (80 μL each) for a total volume of 400 μL. Experimental negative pools included 5 negative patient specimens (80 μL each). The RNA in pooled samples was extracted using ExiPrep™48 Viral DNA/RNA Kit with the ExiPrep™48 Dx instrument. PCR reaction was conducted using the Exicycler™96.

The AccuLoader™ (BIONEER, Korea) which was designed to automatically load samples into a reaction well was used to pool NPS/OPS specimens to reduce human error and carry-over/cross-over contamination (Fig 2). A tablet PC connected to the AccuLoader™ controls the instrument, which consists of a barcode reader, a contamination shield cover, and a contamination prevention filter. The user inputs the sample loading positions and volumes into the PC, then, and after reading the 1D barcode attached to each sample collection tube, the instrument directs the well-plate to automatically move to the proper position which prevents sample from being loaded in the wrong well. A contamination shield cover protects other wells from splashes that may occur during sample loading and a contamination prevention filter minimizes cross-contamination from tip contact. For the pooling test, users input the number of samples and the pooled volume, and the software calculated the required volume of each sample in the pool.

Fig 2

NPS/OPS specimens pooling with the AccuLoader™.

AccuLoader™ recognizes the information of each sample by scanning the barcode on a sample tube, then guides the user to dispense samples in the correct well.

Clinical Performance of AccuPower® kits with Saliva Samples

The clinical performance of AccuPower® kits with saliva samples was evaluated by analyzing the correlation between the PCR results of saliva samples and that of NPS/OPS samples. Saliva and NPS/OPS samples were collected in pairs from each patient. Patients were in a variety of stages in COVID-19 from asymptomatic period to 22 days after symptom onset. Altogether, 47 positive and 40 negative saliva and NPS/OPS paired samples were collected and stored. The RNA was extracted using the ExiPrep™48 Fast Viral RNA Kit with the ExiPrep™48 DX instrument. PCR reaction was conducted using the Exicycler™96.

Saliva samples were collected and stored using the Saliva Collection Kit (BIONEER, Korea), which was developed to collect, transport, and preserve saliva specimens for extraction of human genomic DNA, bacterial genomic DNA, and viral DNA/RNA for disease detection. Collection kits were gently inverted 5 times after saliva collection to properly mix the saliva and the preservation buffer.

Statistical analysis

Statistical analyses were conducted with R Studio 1.3.1093. In addition, the 2x2 contingency-table method was used for analyzing sensitivity, specificity, agreement, and Cohen’s kappa coefficient. Kendall’s W test was used for analyzing the correlation between PCR results in saliva samples and those in NPS/OPS samples.

Results

The LoD of NCVM was 120 copies/mL for the Pan-Sarbecovirus gene (hit rate = 95%) and 120 copies/mL for the SARS-CoV-2 gene (hit rate = 95%) using the ExiStation™48 System (ExiPrep™48 Dx & Exicycler™96). The LoD of SCVM was 2 copies/μL for the Pan-Sarbecovirus gene (hit rate = 95%), 2 copies/μL for the SARS-CoV-2 gene (hit rate = 95%) using the ExiCycler™96. (Table 2). The LoD of reference kit was 0.25 copies/μL for the E gene and ORF1ab gene in NP swab, and 0.25 copies/μL for the E gene and 0.125 copies/μL for the ORF1ab gene in Sputum.

Table 2

Characteristic and PCR condition for LoD test of the AccuPower® kits.

RT-PCR kit	Mix type	RNA extraction	PCR equipment	Target genes	Limit of detection
		equipment
AccuPower® COVID-19 MultiplexReal-Time RT-PCR Kit	Premix	ExiStation™ 48	ExiStation™ 48	Pan-Sarbecovirus gene SARS-CoV-2 gene	Pan-Sarbecovirus gene:NPS/OPS, Sputum 120 copies/mL
					SARS-CoV-2 gene:NPS/OPS, Sputum 120 copies/mL
AccuPower® SARS-CoV-2 Multiplex Real-Time RT-PCR Kit	Master mix	ExiPrep™48 DX	Exicycler™96	Pan-Sarbecovirus gene SARS-CoV-2 gene	Pan-Sarbecovirus gene:NPS/OPS, Sputum 2 copies/μL
					SARS-CoV-2 gene:NPS/OPS, Sputum 2 copies/μL
STANDARD™ M nCoV Real-Time Detection kit	Master mix	QIAamp Viral RNA mini kit	CFX96™Dx System	ORF1ab geneE gene	E gene:NP swab 0.25 copies/μLSputum 0.25 copies/μLORF1ab gene:NP swab 0.25 copies/μLSputum 0.125 copies/μL

Based on in silico analysis, the designed primer and probe sequences were not expected to have a significant PCR amplification by other respiratory disease-related microorganisms, and this proved to be correct. The AccuPower® kits showed no positivity to respiratory disease-related viruses or bacteria except Pan-Sarbecovirus gene positivity on NATrol Coronavirus-SARS Stock (qualitative, NATSARS-ST / 2003–00592). Since NATrol Coronavirus-SARS Stock originally contains the E gene, it was concluded that none of the AccuPower® kits exhibited cross-reactions with other respiratory viruses or bacteria (Table 3).

Table 3

List of pathogens tested for cross-reactivity.

No	Organism	No	Organism
1	Human Influenza virus A H3N2	20	Parainfluenza virus 4a
2	Human Influenza virus A H1N1	21	Chlamydia pneumoniae
3	Human Influenza virus B (Texas/6/11) (Victoria)	22	Haemophilus influenzae
4	Human Coronavirus 229E	23	Legionella pneumophila
5	Human Coronavirus NL63	24	Streptococcus pneumoniae
6	Human Coronavirus OC43	25	Streptococcus pyrogenes
7	Human Respiratory syncytial virus A	26	Bordetella pertussis
8	Human Respiratory syncytial virus B	27	Mycoplasma pneumoniae
9	Human Rhinovirus 14 (type B)	28	Pooled Human nasal wash—to represent diversemicrobial flora in the human respiratory tract
10	Human Metapneumovirus(hMPV)	29	NATtrol Coronavirus-SARS Stock (qualitative) (NATSARS-ST/ 2003–00592)
11	Human Adenovirus type 3 (type B)	30	Enterovirus 70
12	Enterovirus 71	31	Coxsackievrus B5
13	Mycobacterium tuberculosis H37Rv	32	Echovirus 25
14	MERS-CoV	33	Human Parachovirus 3
15	Human coronavirus HKU1(HCOV-HKU1)	34	Mycobacterium fortuitum
16	Adenovirus(71)	35	Mycobacterium intracell
17	Parainfluenza virus 1	36	Mycobacterium gordonae
18	Parainfluenza virus 2	37	Mycobacterium chelonae
19	Parainfluenza virus 3	38	Pneumocystis jirovecii(PJP)

Cross reactivity was evaluated using both in silico analysis and by PCR test. Cross reactivity of NCVM was tested with 29 pathogens (No.1 - 29). Cross-reactivity of SVCM was tested with 38 pathogens (No.1 - 38).

The repeatability of the AccuPower® kits was analyzed by comparing Ct values of the PCR result within runs, between runs, and between days in each matrix at each concentration. The standard deviation (SD) of Ct values in each condition was calculated. The SD result indicates no significant difference in Ct values for positive controls within-run, between-run, between-day, and total precision. (S1 Table).

The clinical performance of the NCVM was evaluated by comparing it to the confirmation test. For sputum samples, the clinical sensitivity of NCVM was 97.50% (95% CI: 93.72% - 99.31%) and the clinical specificity was 98.36% (95% CI: 95.28% - 99.66%). For NPS/OPS samples, the clinical sensitivity the NCVM was 100.00% (95% CI: 97.72% - 100.00%) and the clinical specificity was 98.91% (95% CI: 96.11% - 99.87%) (Table 4).

Table 4

Clinical sensitivity and specificity evaluation results for the AccuPower® kits in Sputum or NPS/OPS specimens.

RT-PCR Kit	Specimen type	N	Sensitivity (%)	95% CI(%)	Specificity (%)	95% CI (%)
AccuPower® COVID-19 MultiplexReal-Time RT-PCR Kit	Sputum	343	97.50	93.72–99.31	98.36	95.28–99.66
	NPS/OPS	343	100.00	97.72–100.00	98.91	96.11–99.87
AccuPower® SARS-CoV-2 Multiplex Real-Time RT-PCR Kit	Sputum	343	98.75	95.56–99.85	99.45	96.99–99.99
	NPS/OPS	343	98.75	95.56–99.85	98.91	96.11–99.87

The correlation of the PCR result of the NCVM and that of the reference kit was also evaluated. For sputum samples, the positive percentage agreement was 96.27% (95% CI: 92.07% - 98.62%), the negative percentage agreement was 98.90% (95% CI: 96.07% - 99.87%), the total percentage agreement was 97.38% (95% CI: 95.08% - 98.79%), and the Cohen’s kappa coefficient was 0.95. For NPS/OPS samples, the positive percentage agreement was 98.14% (95% CI: 94.65% - 99.61%), the negative percentage agreement was 98.90% (95% CI: 96.07% - 99.87%), the total percentage agreement was 98.25% (95% CI: 96.23% - 99.36%), and the Cohen’s kappa coefficient was 0.97 (Table 5).

Table 5

Clinical agreement evaluation results for the AccuPower® kits with STANDARD™ M nCoV Real-Time Detection kit.

RT-PCR Kit	Specimen type	N	PPA(%)	95% CI(%)	NPA (%)	95% CI (%)	Cohen’s kappa
AccuPower® COVID-19 Multiplex Real-Time RT-PCR Kit	Sputum	343	96.27	92.07–98.62	98.90	96.07–99.87	0.95
	NPS/OPS	343	98.14	94.65–99.61	98.90	96.07–99.87	0.97
AccuPower® SARS-CoV-2 Multiplex Real-Time RT-PCR Kit	Sputum	343	97.52	93.76–99.32	99.45	96.96–99.99	0.96
	NPS/OPS	343	98.14	94.65–99.61	99.45	96.96–99.99	0.97

PPA, Positive Percentage Agreement; NPA, Negative Percentage Agreement

The clinical performance of the SCVM was evaluated by comparing it to the confirmation test. For sputum samples, the clinical sensitivity of the SCVM was 98.75% (95% CI: 95.56% - 99.85%) and the clinical specificity was 99.45% (95% CI: 96.99% - 99.99%). For NPS/OPS samples, the clinical sensitivity of the SCVM was 98.75% (95% CI: 95.56% - 99.85%) and the clinical specificity was 98.91% (95% CI: 96.11% - 99.87%) (Table 4).

The correlation between results of the SCVM and that of the reference kit was evaluated. For sputum samples, the positive percentage agreement was 97.52% (95% CI: 93.76% - 99.32%), the negative percentage agreement was 99.45% (95% CI: 96.96% - 99.99%), the total percentage agreement was 98.25% (95% CI: 96.23% - 99.36%), and the Cohen’s kappa coefficient was 0.96. For NPS/OPS samples, the positive percentage agreement was 98.14% (95% CI: 94.65% - 99.61%), the negative percentage agreement was 99.45% (95% CI: 96.96% - 99.99%), the total percentage agreement was 98.54% (95% CI: 96.63% - 99.53%), and the Cohen’s kappa coefficient was 0.97 (Table 5).

The correlation of Ct values in target genes among NCVM, SCVM, and STANDARD™ M nCoV Real-Time Detection kit was analyzed by correlation analysis with the plot in R studio and showed significant correlations with coefficient of determination (R2) at ≥0.97 (S2 Fig). An analysis of Ct values was also performed using the ANOVA test with NCVM, SCVM and STANDARD™ M nCoV Real-Time Detection kit. The ANOVA test showed significant difference between NCVM and STANDARD™ M nCoV Real-Time Detection kit(P<0.05) due to differences in reagent formulation and qRT-PCR protocol. NCVM is a premixed product and STANDARD™ M nCoV Real-Time Detection kit is master mixed product and used touch down qRT-PCR protocol. There was no significant difference between the SCVM and STANDARD™ M nCoV Real-Time Detection kit in the ANOVA test(P>0.05). SCVM and STANDARD™ M nCoV Real-Time Detection kit are master mix type and used touch down qRT-PCR protocol (S3 Fig).

Pooled samples were prepared as described above (S2 Table). Samples were then tested individually using the AccuPower® kits and the reference kit. The test showed 100% positive and negative agreement of the AccuPower® kits with the reference kit. The clinical performance of the AccuPower® kits was also tested using in 5-pooled samples and evaluated by comparing the PCR results from the AccuPower® kits to that of the reference kit. For NCVM, the positive percentage agreement was 93.30% (95% CI: 77.93% - 99.18%) and the negative percentage agreement was 100.00% (95% CI: 88.43% - 100.00%) with the reference test of pooled samples.

The positive and negative agreement between pooled samples and individual samples was also analyzed. For NCVM, the positive percentage agreement of pooled samples was 93.30% (95% CI: 77.93% - 99.18%) and the negative percentage agreement was 100.00% (95% CI: 88.43% - 100.00%), compared to individual samples. The average Ct value of pooled samples was 1.65 higher than the average Ct value of individual samples in the Pan-Sarbecovirus gene and 1.90 higher in the SARS-CoV-2 gene (Fig 3A). The degree of association between the Ct value of individual samples and the Ct value of pooled samples was analyzed by regression analysis and expressed as y = 0.887x+4.885, R2 = 0.9700 in the Pan-Sarbecovirus gene and y = 0.983x+2.375, R2 = 0.9955 in the SARS-CoV-2 gene (Fig 3B).

Fig 3

The comparison of Ct values in individual samples and pooled samples in NPS/OPS specimen pooling test.

(A) Ct values of individual and pooled samples in the AccuPower® kits. (B) The regression analysis with Ct values in individual samples and pooled samples in the AccuPower® kits.

For SCVM, the positive percentage agreement was of pooled samples 90.00% (95% CI: 73.47% -97.89%) and the negative percentage agreement was 100.00% (95% CI: 88.43% - 100.00%) compared to the reference test. In addition, the positive percentage agreement was 90.00% (95% CI: 73.47% -97.89%) and the negative percentage agreement was 100.00% (95% CI: 88.43% - 100.00%) in pooled sample, compared to individual samples. The average Ct value of pooled samples was 0.94 higher than the average Ct value of individual samples in the Pan-Sarbecovirus gene and 1.25 higher in the SARS-CoV-2 gene (Fig 3A). The degree of association between the Ct value of individual samples and the Ct value of pooled samples was analyzed by regression analysis and expressed as y = 0.922x+2.841, R2 = 0.9716 in Pan-Sarbecovirus gene and y = 1.017x+0.840, R2 = 0.9976 in SARS-CoV-2 gene (Fig 3B). The results of the pooling tests are described in Table 6. The disagreement between the PCR results of pooled samples and those of individual samples has occurred only in samples with low concentration (Ct>Cutoff Ct—3) of target genes (S3 Table).

Table 6

The pooling test evaluation results for the AccuPower® kits.

Major discordance	Individual test (reference)
	No.positive	No.negative	No.positive	No.negative
	Individual test (NCVM)		Pool test (NCVM)
Positive	30	0	28	0
Negative	0	150	2	30
PPA (%)	100.00% (95% CI: 88.43% - 100.00%)		93.30% (95% CI: 77.93% - 99.18%)
NPA (%)	100.00% (95% CI: 97.57% - 100.00%)		100.00% (95% CI: 88.43% - 100.00%)
Accuracy (%)	100.00%		96.67%
cohen’s kappa (κ)	1.00		0.93
Major discordance	Individual test (reference)
	No.positive	No.negative	No.positive	No.negative
	Individual test (SCVM)		Pool test (SCVM)
Positive	30	0	27	0
Negative	0	150	3	30
PPA (%)	100.00% (95% CI: 88.43% - 100.00%)		90.00% (95% CI: 73.47% -97.89%)
NPA (%)	100.00% (95% CI: 97.57% - 100.00%)		100.00% (95% CI: 88.43% - 100.00%)
Accuracy (%)	100.00%		95.00%
cohen’s kappa (κ)	1.00		0.90

The clinical performance of the AccuPower® kits for use with saliva samples was evaluated by comparing the PCR results in saliva samples to those in NPS/OPS samples. The Ct values of each AccuPower® kit using both kinds of samples from diverse stages of COVID-19 are shown in Fig 4A.

Fig 4

Clinical performance of the AccuPower® kits in saliva and NPS/OPS samples.

(A) Ct values of saliva and NPS/OPS samples from patients in various stages of COVID-19. (B) Clinical performance comparison of the AccuPower® kits in saliva and NPS/OPS samples shown in scatter plots. (C) Ct value comparison by boxplot among the NCVM and SCVM. ASX, Asymptomatic.

The PCR results of each AccuPower® kit used with each type of sample were verified by comparison to the confirmation test. The NCVM PCR results for NPS/OPS samples showed 97.87% (95% CI: 88.71% - 99.95%) positive agreement, the 100.00% (95% CI: 91.19% - 100.00%) negative agreement, and 98.85% (95% CI: 93.76% - 99.97%) total agreement with a Cohen’s kappa coefficient of 0.98, compared to the PCR results of the confirmation test for NPS/OPS samples. The NCVM PCR results for saliva samples showed 100.00% (95% CI: 92.45% - 100.00%) positive agreement, 100.00% (95% CI: 91.19% - 100.00%) negative agreement, and 100.00% (95% CI: 95.85% - 100.00%) total agreement with a Cohen’s kappa coefficient of 1.00, when compared to the PCR results of the confirmation test for NPS/OPS samples. The comparison of the NCVM PCR results between NPS/OPS samples and saliva samples showed that 100.00% (95% CI: 92.29% - 100.00%) positive agreement, 97.56% (95% CI: 87.14% - 99.94%) negative agreement, and 98.85% (95% CI: 93.76% - 99.97%) total agreement with Cohen’s kappa coefficient of 0.98.

The SCVM PCR results for NPS/OPS samples showed 97.87% (95% CI: 88.71% - 99.95%) positive agreement, 100.00% (95% CI: 91.19%- 100.00%) negative agreement, and 98.85% (95% CI: 93.76% - 99.97%) total agreement with a Cohen’s kappa coefficient of 0.98, compared to the PCR results of the confirmation test for NPS/OPS samples. The SCVM PCR results for saliva samples showed 100.00% (95% CI: 92.45% - 100.00%) positive agreement, 100.00% (95% CI: 91.19% - 100.00%) negative agreement, and 100.00% (95% CI: 95.85% - 100.00%) total agreement with a Cohen’s kappa coefficient of 1.00, compared to the PCR results of the confirmation test for NPS/OPS samples. The comparison of the SCVM PCR results between NPS/OPS samples and saliva samples showed 100.00% (95% CI: 92.29% - 100.00%) positive agreement, 97.56% (95% CI: 87.14% - 99.94%) negative agreement, and 98.85% (95% CI: 93.76% - 99.97%) total agreement, with a Cohen’s kappa coefficient of 0.98. The results indicate the adequacy of the AccuPower® kits for the use with saliva samples (Table 7).

Table 7

Clinical performance evaluation results of the AccuPower® kits in saliva samples.

	Specimen	Positive	Negative	PPV (%)(95% CI %)	NPV (%)(95% CI %)	Cohen’s kappa
SD Kit	NPS/OPS	47	40	-	-	-
NCVM	NPS/OPS	46	41	97.87(88.71% - 99.95%)	100.00(91.19% - 100.00%)	0.98
	Saliva	47	40	100.00(92.45% - 100.00%)	100.00(91.19% - 100.00%)	1.00
	NPS/OPS-saliva agreement			100.00(92.29% - 100.00%)	97.56(87.14% - 99.94%)	0.98
SCVM	NPS/OPS	46	41	97.87(88.71% - 99.95%)	100.00(91.19% - 100.00%)	0.98
	Saliva	47	40	100.00(92.45% - 100.00%)	100.00(91.19% - 100.00%)	1.00
	NPS/OPS-saliva agreement			100.00(92.29% - 100.00%)	97.56(87.14% - 99.94%)	0.98

PPA, Positive Percentage Agreement; NPA, Negative Percentage Agreement

The scatter plots of Ct values in paired NPS/OPS and saliva specimens were analyzed for each AccuPower® kit (Fig 4B). The correlation of Ct values in paired NPS/OPS and saliva specimens showed no significant difference in the Pan-Sarbecovirus gene and the SARS-CoV-2 gene. Kendall’s W was over 0.5 in the Pan-Sarbecovirus gene (W = 0.639 for NCVM, 0.596 for SCVM) and the SARS-CoV-2 gene (W = 0.613 for NCVM, 0.589 for SCVM), showing a high degree of agreement. In addition, the association of Ct values in each type of sample with days from the onset of COVID-19 was examined (Fig 4C). Up to 10 days from the onset of COVID-19, Ct values of saliva samples were lower than those of NPS/OPS samples. On the other hand, after 10 days from the onset of COVID-19, Ct values of NPS/OPS samples were lower than those of saliva samples.

Discussion

The COVID-19 pandemic became the catalyst for the development of more rapid and accurate detection methods for SARS-CoV-2 to better support the clinicians and front-line healthcare professionals [11]. While effective vaccines have been developed, the availability of high-quality diagnostic methods remains essential [12]. Many studies continue to target a more efficient, reliable, and sensitive detection method for SARS-CoV-2. On the other hand, multiplex RT-PCR viral RNA detection assays have been developed for fast and reliable SARS-CoV-2 detection [13-16].

The analytical performance of two AccuPower® kits (NCVM as the premix type and SCVM as the master mix type) was evaluated. The NCVM was designed to be used on a closed system (ExiStation™48 system) to provide full automation and contamination-free and error-free results. The LoD of the NCVM was 120 copies/mL for the Pan-Sarbecovirus gene and the SARS-CoV-2 gene as determined by the ExiStation™48 system. On the other hand, the SCVM was designed to be used on an open system compatible with other manufacturers’ PCR instruments. The LoD of the SCVM was 2 copies/ul for the Pan-Sarbecovirus gene and the SARS-CoV-2 gene as determined by the Exicycler™ 96. In addition, The LoD was determined by multiple alternative PCR instruments (CFX96™ Real-Time PCR Detection System (Bio-rad, USA), Applied Biosystems 7500 Fast Real-time PCR Instrument system (Thermo Fisher Scientific, USA), QuantStudio™5 Real-Time PCR Instrument (Thermo Fisher Scientific), Exicycler™384 (BIONEER, Korea), Exicycler™96 Fast (BIONEER, Korea)) and the results demonstrated comparable performance (S4 Table). Cross-reactivity was tested with 29 respiratory disease-related viruses and bacteria genes were performed for the NCVM, and with 38 for the SCVM, respectively. The results showed that there was no detectable cross-reactivity in AccuPower® kits, Precision of the two AccuPower® kits was evaluated and the results showed high within-run, between-run, between-day, and total precision.

This study provides the data to support the usability of the AccuPower® kits for the detection of SARS-CoV-2 RNA in sputum and NPS/OPS clinical samples as evidenced by equivalency with the confirmation test, which was performed in the collection institute beforehand, and with the reference kit (STANDARD™ M nCoV Real-Time Detection kit). Specifically, the results of the AccuPower® kits were validated with over 95% agreement with both the confirmation test and the reference kit. On the other hand, the NPS/OPS specimen pooling test showed the PCR result of the pooled samples of 5 individuals had over 90% agreement with the individual samples by the AccuPower® kits. The PCR results of the AccuPower® kits in saliva samples showed over 95% positive and negative agreement with those in the NPS/OPS samples, confirming the usability of the AccuPower® kits for the saliva samples as well. In addition, the correlation between Ct values and days from the onset of COVID-19 in both NPS/OPS and saliva samples indicates the saliva samples are more suitable for detection up to 10 days from the onset of COVID-19 and the NPS/OPS samples are more suitable for detection after 10 days from the onset of COVID-19 for SARS-CoV-2 detection. This result is consistent with the previous studies on the COVID-19 saliva samples [17]. In brief, the analytical and clinical performance of the AccuPower® kits showed they are as effective in the SARS-CoV-2 detection kit as the current standard confirmation test including the reference kit.

A previous study presented the LoD of AccuPower® kits without the appropriate number of replications, using quantified specimens RNA for the test [18]. On the contrary, in this study, The LoD test was performed with 20 replicates, which is recommended in the CLSI guideline, using SARS-CoV-2 verification panel and SARS-Related Coronavirus 2. In addition, the clinical performance of each AccuPower® kit was tested with clinical samples of various Ct and verified by comparing them to the performance of STANDARD™ M nCoV Real-Time Detection kit, which obtained WHO and FDA approval for emergency use and MFDS official approval and showed high agreements (>95%). STANDARD™ M nCoV Real-Time Detection kit showed higher clinical sensitivity with Allplex™ 2019-nCoV Assay in a comparison study [15]. Also, the clinical performance evaluation of AccuPower® kits, assuming the result of Allplex™ 2019-nCoV Assay, which was one type of test in the confirmation test, as true, indicated high sensitivity and specificity (>95%) (S5 Table). Thus, it may be suggested that AccuPower® kits have equivalent clinical sensitivity with Allplex™ 2019-nCoV Assay.

In this study, AccuPower® kits has some limitations. The first is AccuPower® kit uses the same fluorescent dye for the RdRp gene and the N gene, so individual detection is impossible. In another thing, in the clinical trial using saliva, there was no EUA RT-PCR assay claiming saliva at the time of the test, so comparison by kit between saliva-saliva was not possible. Therefore, if a comparison with a reference kit claiming saliva is performed later, this part can be supplemented.

In conclusion, this study describes the successful development of two multiplex real-time RT-PCR methods, NCVM and SCVM, for the diagnosis of SARS-CoV-2. Simultaneous targeting of three viral genes (RdRP, N, and E) by the AccuPower® kits provides an accurate, reliable, and easy-to-use SARS-CoV-2 detection test.

NCVM was used as an automated system from RNA extraction to PCR with a vacuum dried premix type kit to minimize the hands-on step and user errors. SCVM can be used with other manufacturers’ PCR instruments such as CFX96™ Dx Real-Time PCR Detection System(Bio-Rad), Applied Biosystems 7500 Fast Real-time PCR Instrument system(Thermo Fisher Scientific), QuantStudio™5 Real-Time PCR Instrument (Thermo Fisher Scientific).

The AccuPower® kits demonstrate the analytical performance characteristics expected of a valid diagnostic assay. The clinical performance of the AccuPower® kits was comparable to the gold standard confirmation test, including the reference kit. In addition, the specimen pooling test with n = 5 showed the ability of the AccuPower® to process high volume samples cost-effectively for use as a surveillance tool. The clinical performance test of AccuPower® kits in saliva samples demonstrated the usability of the AccuPower® kits with saliva samples and the saliva samples being more adequate than NPS/OPS samples for early detection (before 10 days from the onset of symptom) of COVID-19. These data demonstrate that the AccuPower® assays can be used for the fast and dependable detection of the SARS-CoV-2 virus.

Ct value variation in collected clinical samples.

Nashpharyngeal and Oropharyngeal swab specimens Ct distribution (Above). Sputum specimens Ct distribution (Below). *SD: STANDARD™ M nCoV Real-Time Detection kit, Cutoff (Ct): 36 **CancerRop: Q-Sens® COVID-19 Detection Kit V2, Cutoff (Ct): 40 ***Seegene: Allplex™ 2019-nCoV Assay, Cutoff (Ct): 40

(TIF)

Click here for additional data file.

Correlation Analysis with plot among NCVM, SCVM, and STANDARD™ M nCoV Real-Time Detection kit.

Ct values among assays (NCVM, SCVM and STANDARD™ M nCoV Real-Time Detection kit) showed high correlation with a Pearson R2 correlation coefficient ≥0.97.

(TIF)

Click here for additional data file.

Comparison between NCVM, SCVM and STANDARD™ M nCoV Real-Time Detection kit.

Results of ANOVA test showed significant difference between NCVM and STANDARD™ M nCoV Real-Time Detection kit (p<0.05) whereas no significant difference between SCVM and STANDARD™ M nCoV Real-Time Detection kit (p>0.05).*SD: STANDARD™ M nCoV Real-Time Detection kit.

(TIF)

Click here for additional data file.

Precision evaluation results for the AccuPower® kits.

Precision evaluation results of AccuPower® COVID-19 Multiplex Real-Time RT-PCR Kit and AccuPower® SARS-CoV-2 Multiplex Real-Time RT-PCR Kit.

(TIF)

Click here for additional data file.

Individual samples and pooled samples in swab specimen pooling test.

(TIF)

Click here for additional data file.

The agreement between individual samples and pooled samples in the AccuPower® kits.

(TIF)

Click here for additional data file.

Limit of detection of SCVM in various PCR instruments.

(TIF)

Click here for additional data file.

Clinical sensitivity and specificity evaluation results for the AccuPower® kits in Sputum or NPS/OPS specimens, compared to Allplex™ 2019-nCoV Assay.

(TIF)

Click here for additional data file.

(PDF)

Click here for additional data file.

(XLSX)

Click here for additional data file.

8 Nov 2021

21 Dec 2021

Responding to Reviewers

Thank you for reviewing the manuscript. We respond to all of your comments as follows.

< Journal requirements >

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

�  As you commented, we uploaded files labeled ‘Response to Reviewers’, ‘Revised Manuscript with Track Changes’ and ‘Manuscript’. Uploaded files were written in PLOS ONE style templates.

2. Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement.

�  As you commented, we removed funding-related text. (page 27)

3. Please state what role the funders took in the study. If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript." If this statement is not correct you must amend it as needed. Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.”

�  The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

4. Please include your full ethics statement in the ‘Methods’ section of your manuscript file. In your statement, please include the full name of the IRB or ethics committee who approved or waived your study, as well as whether or not you obtained informed written or verbal consent.

�  As you commented, the ethics statement has been added to the 'Method_Ethics Statement and Sample Collection' section (page 8).

Ethics Statement and Sample Collection

Authors confirm that this paper has complied with PLOS policies on research ethics, as stated in submission guide. Since all identification information of samples was anonymized by replacing it with the unique identification code of the institution, researchers have performed the clinical performance test without risks of identifying or tracing personal information with the sample. The protocol of the study was approved by the Institutional Review Board (IRB) of Myongji Hospital and IRB of Kangwon National University Hospital. Since samples were residual sample with guaranteed anonymity and there was no need to collect additional samples for this study, there was no risk to research subjects in accordance with ‘Bioethics and Safety Act’ and Article 13 of ‘Enforcement Rules of the In Vitro Diagnostic Medical Devices Act’. Hence, IRBs confirmed that this study does not need subject consent. This study was conducted in accordance with the principles of the Declaration of Helsinki, 1964.

Myongji Hospital Institutional Review Board (IRB) approved the use of surplus clinical samples for the NPS/OPS specimen-pooling (IRB No.: MJH 2020-12-028) and clinical performance evaluation (IRB No.: MJH 2020-12-029) tests. Kangwon National University Hospital IRB approved the use of saliva and NPS/OPS samples for the clinical performance testing of the AccuPower® kits (IRB No.: KNUH-2021-03-014), which were either surplus samples or de-identified patient samples drawn after informed consent (Fig 1). RNA in the NPS/OPS and sputum samples was extracted using the ExiPrep™48 Viral DNA/RNA Kit (BIONEER, Korea) with the ExiPrep™48 Dx instrument and the RNA of saliva samples was extracted using the ExiPrep™48 Fast Viral RNA Kit (BIONEER, Korea) with the ExiPrep™48 Dx instrument. Previously, it was determined whether each sample was SARS-CoV-2 positive or negative by the confirmation test.

< Review Comments to the Author >

Reviewer #1

1. In Table 2, the data for LoD test of the reference kit should be included for better comparison with that of two AccuPower® kits.

�  As you kindly recommended, we added LoD data of the reference kit (STANDARD M nCoV Real-Time Detection kit, SD BIOSENSOR) in clinical trials for comparison (page 13, Table 2). LoD results of STANDARD M nCoV Real-Time Detection kit were referenced from Instruction for Use (IFU) Rev1 (SD BIOSENSOR, May 14, 2020).

RT-PCR kit Mix type RNA extraction PCR equipment Target genes Limit of detection

equipment

AccuPower® COVID-19 Multiplex

Real-Time RT-PCR Kit 　Premix ExiStation™ 48 ExiStation™ 48 Pan-Sarbecovirus gene SARS-CoV-2 gene Pan-Sarbecovirus gene:

120 copies/mL

SARS-CoV-2 gene:

120 copies/mL

AccuPower® SARS-CoV-2 Multiplex Real-Time RT-PCR Kit Master mix ExiPrep™48 DX Exicycler™96 Pan-Sarbecovirus gene SARS-CoV-2 gene Pan-Sarbecovirus gene:

2 copies/μL

SARS-CoV-2 gene:

2 copies/μL

STANDARD™ M nCoV Real-Time Detection kit Master mix QIAamp Viral RNA mini kit CFX96™Dx System ORF1ab gene

E gene ORF1ab gene:

NP swab 0.25 copies/μL

Sputum 0.125 copies/μL

E gene:

NP swab 0.25 copies/μL

Sputum 0.25 copies/μL

2. In general, it is suitable that the cutoff Ct value for real-time RT-PCR is set to 35. Please explain the rationality that the cutoff Ct value is set to 40 in this study.

�  For the cutoff Ct value of Accupower® kits, the optimal cutoff Ct value is set by drawing an ROC curve using the LoD(Limit of Detection) analysis result, the low titer positive clinical sample result, and the LoB(Limit of Blank) result.

As a result of ROC Curve confirmation, in the case of NCVM, the Pan-Sarbecovirus gene was 40 Ct (specificity: 100.0%, sensitivity: 94.9.0%, AUC:98.9%), and the SARS-CoV-2 gene was 39 Ct (specificity: 100.0%, sensitivity:96.8%, AUC:98.8%). In the case of SCVM, the Pan-Sarbecovirus gene is 35 Ct (specificity: 100.0%, sensitivity:99.4%, AUC:99.9%), and the SARS-CoV-2 gene is 34Ct (specificity: 100.0%, sensitivity:98.6%, AUC:99.5%) was confirmed.

AccuPower® COVID-19 Multiplex Real-Time

RT-PCR Kit (NCVM) AccuPower® SARS-CoV-2 Multiplex Real-Time

RT-PCR Kit (SCVM)

There is a difference in Ct depending on whether SCVM and NCVM use qRT-PCR protocol. Unlike NCVM, SCVM uses the Touch down qRT-PCR protocol to confirm that the Ct value of SCVM is pulled by about 3-5 Ct compared to NCVM at the same concentration. Therefore, when setting the cutoff Ct, the Ct values of LoD, LoB, and low titer positive samples differ by about 3-5 Ct, so the cutoff Ct value for NCVM was 40 Ct (or 39 Ct) and SCVM was found to be 35 Ct (or 34 Ct).

For a similar example, the STANDARD™ M nCoV Real-Time Detection kit (SD BIOSENSOR) which uses the same touch down PCR protocol as SCVM has the cutoff of 36Ct. On the other hand, Allplex™ 2019-nCoV Assay (Seegene) which does not use a touch down PCR protocol has a cutoff of 40Ct.

AccuPower® COVID-19 Multiplex Real-Time RT-PCR Kit AccuPower® SARS-CoV-2 Multiplex Real-Time RT-PCR Kit

qRT-PCR Protocol

Cutoff Ct value Pan-Sarbecovirus gene: 40 Ct

SARS-CoV-2 gene: 39Ct Pan-Sarbecovirus gene: 35 Ct

SARS-CoV-2 gene: 34Ct

Similar Product of other manufacturer Allplex™ 2019-nCoV Assay (Seegene)

Cutoff Ct: 40Ct STANDARDTM M nCoV Real-Time Detection kit (SD BIOSENSOR)

Cutoff Ct: 36Ct

3. In S3 Fig, the comparison of the average Ct values should be performed between STANDARD™ M nCoV Real-Time Detection kit and NCVM.

S3 Fig. is the analysis of Ct value for two types of kits, SCVM and the STANDARD™ M nCoV Real-Time Detection kit (SD), both of which are master mix type. NCVM, which is a premix type, was excluded from the analysis due to differences in reagent formulation and in qRT-PCR protocol (with or without touch down PCR protocol).

As you commented, the results of analyzing the Ct value between the STANDARD™ M nCoV Real-Time Detection kit (SD) and NCVM S3 Fig. are added as follows. In the ANOVA analysis of SCVM VS SD, the p-value is >0.05, which is not significant, but in the ANOVA analysis of NCVM VS SD, the p-value is <0.05 due to differences in reagent formulation and qRT-PCR protocol as described above.

In addition, since the previously submitted S3 Fig. only analyzed some of samples used in the clinical trial, S3 Fig. is now corrected with the re-analysis result including the total number of samples. (S3 Fig.)

4. In view of the less sensitivity of two AccuPower® kits in detecting pooled samples, it is better to compare the average Ct values of NCVM and SCVM with that of the reference kit in pooled samples in Fig 3.

The reference kit (STANDARD™ M nCoV Real-Time Detection kit, SD BIOSENSOR) did not claim pooling samples as usable samples and therefore we could not compare two AccuPower® kits to STANDARD™ M nCoV Real-Time Detection kit.

Additionally, according to FDA guidelines, the results of individual samples in the test kit should be compared to those of reference kits (comparator method) and the results of pooled samples in the test kit should be compared to those of individual samples in the same kit (Expected result based on individually test result). We present a guideline for test and analysis in sample pooling, which compares Ct values between individual and pooled samples.

5. In Fig 4C, the data of Ct value comparison mentioned in figure legend between saliva and NPS/OPS samples detected by STANDARD™ M nCoV Real-Time Detection kit is missing.

The reference kit (STANDARD™ M nCoV Real-Time Detection kit, SD BIOSENSOR) was not tested as it did not claim Saliva samples as usable samples. The legend of Figure 4C was written incorrectly and was corrected as follows.

Fig 4. Clinical performance of the AccuPower® kits in saliva and NPS/OPS samples (A) Ct values of saliva and NPS/OPS samples from patients in various stages of COVID-19. (B) Clinical performance comparison of the AccuPower® kits in saliva and NPS/OPS samples shown in scatter plots. (C) Ct value comparison by boxplot among the NCVM and SCVM. ASX, Asymptomatic.

6. The advantage and disadvantage of two AccuPower® kits compared to the commercial kit and the range of their clinical application should be addressed in discussion section.

We are already addressed advantage of AccuPower® kits. As you commented we added limitations(or disadvantage) of two AccuPower® kits. (Page 26)

In this study, AccuPower® kits has some limitations. The first is AccuPower® kit uses the same fluorescent dye for the RdRp gene and the N gene, so individual detection is impossible. In another thing, in the clinical trial using saliva, there was no EUA RT-PCR assay claiming saliva at the time of the test, so comparison by kit between saliva-saliva was not possible. Therefore, if a comparison with a reference kit claiming saliva is performed later, this part can be supplemented.

In conclusion, this study describes the successful development of two multiplex real-time RT-PCR methods, NCVM and SCVM, for the diagnosis of SARS-CoV-2. Simultaneous targeting of three viral genes (RdRP, N, and E) by the AccuPower® kits provides an accurate, reliable, and easy-to-use SARS-CoV-2 detection test.

NCVM was used as an automated system from RNA extraction to PCR with a vacuum dried premix type kit to minimize the hands-on step and user errors. SCVM can be used with other manufacturers' PCR instruments such as CFX96™ Dx Real-Time PCR Detection System(Bio-Rad), Applied Biosystems 7500 Fast Real-time PCR Instrument system(Thermo Fisher Scientific), QuantStudio™5 Real-Time PCR Instrument (Thermo Fisher Scientific).

The AccuPower® kits demonstrate the analytical performance characteristics expected of a valid diagnostic assay. The clinical performance of the AccuPower® kits was comparable to the gold standard confirmation test, including the reference kit. In addition, the specimen pooling test with n=5 showed the ability of the AccuPower® to process high volume samples cost-effectively for use as a surveillance tool. The clinical performance test of AccuPower® kits in saliva samples demonstrated the usability of the AccuPower® kits with saliva samples and the saliva samples being more adequate than NPS/OPS samples for early detection (before 10 days from the onset of symptom) of COVID-19. These data demonstrate that the AccuPower® assays can be used for the fast and dependable detection of the SARS-CoV-2 virus.

7. The primers and probe information are missing and should be included in the paper.

Primer and probe information (sequence, etc.) cannot be disclosed as confidential data of the manufacturer.

Minor comments

1) Have the authors verified that the assays are equally sensitive in detecting different circulating SARS-CoV-2 variants?

SARS-CoV-2 variants analysis was performed as an in-silico analysis.

For sequence In-silico analysis of AccuPower® kis alignment was performed for the following SARS-CoV-2 strain in the GISAID as Aug 30 to Sep 30, 2021(Location : North America, USA).

SARS-CoV-2 alpha variants (known as B.1.1.7, n=9,064), beta variants (known as B.1.351, n=477), delta variants (known as B.1.617.2, n=37,317), gamma variants (known as P.1, n=2,058), Mu variants (known as B.1.621, n=2,756) and lambda variants (known as C.37, n=229) were analyzed for coverage and homology with Accupower kit. Jalview was used for analysis.

The coverage of the using oligomers was analyzed individually by comparing and analyzing a total sequence, except to un-sequenced results (e.g –NNNNN-). The Table below shows the results of this coverage analysis.

Target oligo Alpha variant (n=9,064) Beta variant (n=477) Delta variant (n=37,317) Gamma variant (n=2058)

match Coverage match Coverage match Coverage match Coverage

E gene set1 Forward primer 9053 99.85% 462 100.00% 37247 99.81% 1,945 100.00%

Reverse primer 9064 100% 462 100.00% 37284 99.91% 1,944 99.95%

Probe 9062 99.95% 462 100.00% 37241 99.80% 1,944 99.95%

RdRP-1 Forward primer 9039 99.7% 462 100.00% 36685 98.31% 1,945 100.00%

Reverse primer 1 0.00% 0.00% 0 0.00% 0 0.00% 0 0.00%

Reverse primer 2 9048 99.8% 462 100.00% 37237 99.79% 1,940 99.74%

Probe 9062 99.7% 462 100.00% 37239 99.79% 1,940 99.74%

SARS-CoV-2 (RdRP set 2) Forward primer 9063 99.99% 462 100.00% 37264 99.76% 1,945 100.00%

Reverse primer 9064 100% 462 100.00% 37315 99.99% 1,945 100.00%

Probe 9062 99.98% 462 100.00% 37274 99.88% 1,945 100.00%

SARS-CoV-2 (RdRP set 3) Forward primer 9064 100% 462 100.00% 37243 99.80% 1,940 99.74%

Reverse primer 9064 100% 462 100.00% 37285 99.91% 1,945 100.00%

Probe 9058 99.93% 461 99.78% 37229 99.76% 1,944 99.95%

SARS-CoV-2 (N gene) Forward primer 9062 99.95% 462 100.00% 36837 98.71% 1,916 98.51%

Reverse primer 9060 99.93% 462 100.00% 37089 99.39% 1,906 97.99%

Probe 9060 99.98% 461 99.78% 37172 99.61% 1,929 99.18%

Target oligo Lambda variant (n=229) mu variant (n=2,756)

match Coverage match Coverage

E gene set1 Forward primer 229 100% 2,756 100%

Reverse primer 229 100% 2,756 100%

Probe 229 100% 2,756 100%

RdRP-1 Forward primer 229 100% 2,756 100%

Reverse primer 1 229 0.00% 0.00% 0.00%

Reverse primer 2 229 100% 2,756 100%

Probe 229 100% 2,756 100%

SARS-CoV-2 (RdRP set 2) Forward primer 229 100% 2,756 100%

Reverse primer 229 100% 2,756 100%

Probe 229 100% 2,756 100%

SARS-CoV-2 (RdRP set 3) Forward primer 229 100% 2,756 100%

Reverse primer 229 100% 2,756 100%

Probe 229 100% 2,756 100%

SARS-CoV-2 (N gene) Forward primer 228 99.56% 2,756 100%

Reverse primer 229 100% 2,756 100%

Probe 229 100% 2,756 100%

2) Could the authors provide more detailed data regarding the cross-reactivity test?

Several potentially cross-reactive organisms were spiked in simulated clinical matrix with presence of SARS-CoV-2 RNA (3X LoD concentration, Virus-Like Particles, Isolate USA-WA1/2020, Heat Inactivated). Then, spiked matrix was compared to control matrix (Sputum & NPS/OPS) which contained SARS-CoV-2 RNA and no spiked organisms. Each sample for each organism was tested in three replicates. Ct values of the control matrix were averaged and set as the control value. And Ct values of each sample was averaged and set as each testing group value. Difference between average value of testing group and the control was calculated to evaluate whether all samples were within the specified acceptance criteria or not. It was confirmed that the CV (%) and the % change of microbial interferent result are smaller than the acceptance criterion. All tested organisms were shown no influence in performance of the AccuPower® COVID-19 Multiplex Real-Time RT-PCR Kit and AccuPower® SARS-CoV-2 Multiplex Real-Time RT-PCR Kit at the 3XLoD concentrations evaluated with regards to analytical specificity and sensitivity.

List of organisms to be tested

No Organism Reference No Organism Reference

1 Human Influenza virus A H3N2 KBPV-VR-71 20 Parainfluenza virus 4a KBPV-VR-69

2 Human Influenza virus A H1N1 KBPV-VR-76 21 Chlamydia pneumoniae ATCC 53592

3 Human Influenza virus B (Texas/6/11) (Victoria) 0810242CFHI 22 Haemophilus influenzae ATCC 31441

4 Human Coronavirus 229E KBPV-VR-9 23 Legionella pneumophila ATCC 33152

5 Human Coronavirus NL63 0810228CF 24 Streptococcus pneumoniae ATCC 49619

6 Human Coronavirus OC43 0810024CF 25 Streptococcus pyrogenes ATCC700294

7 Human Respiratory syncytial virus A KBPV-VR-41 26 Bordetella pertussis ATCC 12742

8 Human Respiratory syncytial virus B KBPV-VR-42 27 Mycoplasma pneumoniae ATCC 15531D

9 Human Rhinovirus 14 (type B) KBPV-VR-39 28 Pooled Human nasal wash - to represent diverse microbial flora in the human respiratory tract 991-13-P-1

10 Human Metapneumovirus(hMPV) 0810156CF 29 NATtrol Coronavirus-SARS Stock (qualitative) (NATSARS-ST/ 2003-00592) 324780

11 Human Adenovirus type 3 (type B) KBPV-VR-62 30 Enterovirus 70 KBPV-VR-55

12 Enterovirus 71 KBPV-VR-56 31 Coxsackievrus B5 KBPV-VR-17

13 Mycobacterium tuberculosis H37Rv NAT MTB(H37RV)-ST 32 Echovirus 25 KBPV-VR-24

14 MERS-CoV 0505-0002 33 Human Parachovirus 3 0810147CF

15 Human coronavirus HKU1

(HCOV-HKU1) ATCC VR-3262SD 34 Mycobacterium fortuitum ATCC6841

16 Adenovirus(71) VR-1D 35 Mycobacterium intracell ATCC13950

17 Parainfluenza virus 1 KBPV-VR-44 36 Mycobacterium gordonae KCTC9513

18 Parainfluenza virus 2 KBPV-VR-45 37 Mycobacterium chelonae KCTC9505

19 Parainfluenza virus 3 KBPV-VR-68 38 Pneumocystis jirovecii(PJP) NATPJI-ERC

Microbial interference and Endogenous interference studies:

- SARS-CoV-2 gene with interfering substance must be not detected for base matrix.

- Hit Rate at SARS-CoV-2 panel (3xLoD) : The observed positive hit rate for the 3XLoD Positive samples must be equal to 100%.

- 3XLoD Positive samples should be analyzed CV(%) < 5.

- The % change (% change; [Test Ct mean values-Baseline Ct mean values]/ baseline Ct x 100) for the SARS-CoV-2 RNA (3xLoD) must be less than ± 10% change to be considered stable.

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

18 Jan 2022

Development and Evaluation of AccuPower® COVID-19 Multiplex Real-Time RT-PCR Kit and AccuPower® SARS-CoV-2 Multiplex Real-Time RT-PCR Kit for SARS-CoV-2 Detection in Sputum, NPS/OPS, Saliva and Pooled Samples.

PONE-D-21-30277R1

Dear Dr. Kim,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Feng Gao

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

Reviewer #2: In the revised version, the authors have addressed all the comments. I don't have further comments.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

2 Feb 2022

PONE-D-21-30277R1

Development and Evaluation of AccuPower COVID-19 Multiplex Real-Time RT-PCR Kit and AccuPower SARS-CoV-2 Multiplex Real-Time RT-PCR Kit for SARS-CoV-2 Detection in Sputum, NPS/OPS, Saliva and Pooled Samples.

Dear Dr. Kim:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Feng Gao

Academic Editor

PLOS ONE