Evaluation of Factors that Affect the Performance of COVID-19 Molecular Assays Including Presence of Symptoms, Number of Detected Genes and RNA Extraction Type.

BACKGROUND AND AIMS: Rapid and accurate detection of COVID-19 is crucial for mitigation of the pandemic. We evaluated the performance of six molecular kits and the effect of several factors on the performance of the kits.
MATERIALS AND METHODS: Two hundred and four nasopharyngeal samples were collected from participants aged ≥18 years at the Baruch Padeh Medical Center Poriya, Israel, between June and August 2020. Samples were tested by: Allplex 2019-nCOV Assay (Seegene), Real-Time Fluorescent RT-PCR Kit for Detecting SARS-2019-nCoV (BGI Genomics), Xpert® Xpress SARS-CoV-2 test (Cepheid), Simplexa® COVID-19 Direct Kit (Focus Diagnostics), BD SARS-CoV-2 Reagents for BD MAX™ System (BD), and Logix Smart™ Coronavirus Disease 2019 (COVID-19) Test kit (CO-DIAGNOSTICS).
RESULTS: Xpert® Xpress SARS-CoV-2 test and Logix Smart™ COVID-19 Kit had the highest (91.2%) and the lowest (74.5%) sensitivity, respectively. Symptoms were a predictor of a positive result. Traditional assays had a higher minimum cycle threshold (min Ct), i.e. detected lower viral load, compared to rapid assays (p = 0.012). Samples of symptomatic participants had lower min Ct, than samples of asymptomatic participants (p < 0.001). Additionally, the more genes were detected, the lower the min Ct (p < 0.001), indicating that a greater percentage of the viral genome was amplified.
CONCLUSIONS: Taken together, most assays had overall good performance. Since several factors affect the performance of kits, each laboratory must be familiar with its kit's limitations in order to produce the most reliable results.

Key Points

Introduction

More than 1 year ago, an outbreak of a severe acute respiratory syndrome Coronavirus 2 (SARS-CoV2) emerged in Wuhan, China, and rapidly spread globally [1]. Considering there were no specific therapies, one successful strategy for mitigating this pandemic spread was a rapid, accurate and sensitive detection of the virus, enabling early isolation of infected individuals [2].

To this end, a real-time reverse-transcriptase PCR (RT-PCR) using nasopharyngeal samples was chosen as the gold standard for COVID-19 detection and several assays have been rapidly developed [3, 4]. Among these assays are the traditional RT-PCR assays, that require sample preparation, including virus deactivation and RNA extraction, prior to amplification and detection of the viral genome. Other diagnostic tests are automated, rapid molecular assays, that include an automated process for all steps of RT-PCR from sample preparation to a final result.

Nevertheless, infection control is still limited since RT-PCR only assesses the viral RNA presence in the sample and cannot indicate on the virus viability and infectivity [5]. Thus, a negative result cannot exclude a previous infection, and must be combined with the patient’s history, clinical observations and available epidemiological information [2]. On the other hand, a patient with a positive result is not necessarily infectious.

In this study, we compared the performance of four traditional molecular assays and two automated, rapid tests that have been used worldwide by experienced clinical diagnostic laboratories to detect COVID-19: Allplex 2019‐nCOV Assay (Seegene), Real‐Time Fluorescent RT‐PCR Kit for Detecting SARS-2019-nCoV (BGI Genomics), Simplexa® COVID-19 Direct Kit (Focus Diagnostics), BD SARS-CoV-2 Reagents for BD MAX™ System (BD), Xpert® Xpress SARS-CoV-2 test (Cepheid), and The Logix Smart™ Coronavirus Disease 2019 (COVID-19) Test Kit (CO-DIAGNOSTICS). We also investigated the effect of several factors on the performance of the kits.

Materials and Methods

Study Population

The study group included 204 patients aged ≥18 years, who were admitted to the Baruch Padeh Medical Center Poriya, Israel, between June 2020 and August 2020 for COVID-19 molecular testing.

The Israeli Ministry of Health approved this study as part of validation of kits for detection of COVID-19. Since there was no utilization of personal data of the participants (except for the presence of symptoms that were linked to specimens and these were de-identified), the Israeli Ministry of Health waived the need for Helsinki Ethics Committee approval and for participants’ consent.

Out of 204 participants, 102 were positive and 102 were negative for the presence of COVID-19 RNA, according to the AllplexTM 2019‐nCOV Assay, which was considered our reference kit. Of the 102 COVID-19-positive participants, 77 were tested due to symptom presentation, while the other 25 had no symptoms and were tested due to exposure to an infected individual. Symptoms included fever or chills, cough, shortness of breath/difficulty breathing, fatigue, muscle or body aches, headache, recent loss of taste or smell, sore throat, congestion or runny nose, nausea or vomiting, and diarrhea.

Sample Collection

Samples were collected from the tonsillar areas and the posterior pharynx of patients using synthetic fiber swabs with thin flexible plastic shafts. The swabs were inserted into test tubes containing Universal Transport Medium for viruses (UTM) and sent to the clinical microbiology laboratory at the Baruch Padeh Medical Center, Poriya, Israel, for the detection of COVID-19.

Molecular Testing for COVID-19

Xpert Xpress SARS-CoV-2 Test

The Xpert Xpress SARS-CoV-2 test (Cepheid, Sunnyvale, CA, USA) was performed using the GeneXpert Instrument Systems (Cepheid, Sunnyvale, CA, USA) that perform automated sample preparation, nucleic acid extraction, amplification, and detection of target sequences using real-time PCR assays. The Xpert Xpress SARS-CoV-2 test was performed inside a disposable cartridge that contains RT-PCR reagents according to the manufacturer’s instructions. Briefly, the sample tube was mixed and then 300 µL of the sample-in-UTM was transferred to the sample chamber of the Xpert Xpress SARS-CoV-2 cartridge using the supplied transfer pipette. The GeneXpert cartridge was loaded onto the GeneXpert Instrument System platform and results were obtained within 47 min. Results were automatically interpreted by the instrument’s software. Table S1 in the Electronic Supplementary Material (ESM) describes the characteristics of all tested kits, including the specific gene targets.

Simplexa™ COVID-19 Direct Test

The Simplexa™ COVID-19 Direct test (Focus Diagnostics, Cypress, CA, USA) was performed on the LIAISON® MDX instrument (Focus Diagnostics), which, like the GeneXpert Instrument systems, performs an automated RT-PCR reaction from sample preparation to a final result. The kit is supplemented with 24 reaction mix vials and a Direct Amplification Disc, which contains two wedges for each sample, with one wedge for the reaction mix (designated as ‘R’) and the second wedge designated as ‘S’) for the sample. For each sample, the content of one reaction mix tube (50 µL) was transferred to the ‘R’ wedge, followed by pipetting 50 µL of the sample-in-UTM to the ‘S’ wedge. Then, the disk was loaded on the LIAISON® MDX instrument. Results were obtained within 90 min and automatically interpreted by the instrument’s software.

BD SARS-CoV-2 Reagents for BD MAX™ System

The BD SARS-CoV-2 Reagents for BD MAX™ System (BD Diagnostics, Franklin Lakes, NJ, USA) is a real-time RT-PCR test intended for the qualitative detection of COVID-19 RNA in respiratory samples. This assay includes RNA extraction, RT-PCR reaction and detection of the viral nucleic acid. For each sample, 750 µL of the sample-in-UTM was transferred to a BD MAX™ TNA-3 Sample Buffer Tube provided with the kit. For each sample, one Unitized Reagent Strip was loaded with one BD MAX™ ExK™ TNA-3 Extraction Tube, one BD MAX™ TNA MMK Master Mix Tube and one BD SARS-CoV-2 Reagents for BD MAX™ System Primers and Probes Tube. The Unitized Reagent Strips and required number of BD MAX™ PCR Cartridges were placed into the BD MAX™ System (BD Diagnostics). The Sample Buffer Tubes were then placed into the BD MAX™ System Racks corresponding to the Unitized Reagent Strips. Thermal cycling was performed at 58 °C for 20 min, followed by 95 °C for 5 min, and 45 cycles of 95 °C for 5 s, and 58 °C for 40 s.

The BD MAX™ System automatically interprets the test results when reaction ends (within ~59 min).

RNA Extraction for Non-Automated Assays

Viral RNA was extracted from 140 µL of the sample-in-UTM using the QIAamp Viral RNA Kit and the QIAcube automated spin-column purification kit (QIAGEN GmbH, Hilden, Germany) according to the kit's protocol using the QiIAqube instrument (QIAGEN).

AllplexTM 2019‐nCOV Assay

The Allplex™ 2019-nCoV Assay (Seegene, Seoul, South Korea) was designed to detect the COVID-19 RNA in human respiratory samples. This assay was chosen by the Israel Ministry of Health as the gold standard for COVID-19 detection. It was therefore used as the reference standard for this analysis. For each sample, 5 µL of 2019-nCoV MOM (containing oligonucleotides and amplification and detection reagent), 5 µL of RNase-free Water, 5 µL of 5X Real-time One-step Buffer, and 2 µL of Real-time One-step Enzyme were added to a well of 96 PCR microplate. Then, 8 μL of RNA from each sample or positive or negative controls was transferred into the specific well. The PCR plate was loaded into a Bio Rad CFX96TM Real-Time Detection System (Bio Rad), and reaction conditions were set as follows: one cycle at 50 °C for 20 min, followed by a cycle at 95 °C for 15 min and 45 cycles at 94 °C for 15 s and 58 °C for 30 s. Results were obtained within ~70 min. Interpretations of results was performed according to the manufacturer's instructions.

Real‐Time Fluorescent RT‐PCR Kit for Detecting SARS-CoV-2 Assay

The Real‐Time Fluorescent RT‐PCR Kit for Detecting SARS-CoV-2 (BGI Genomics, Yantian District, Shenzhen, China) was designed to detect the COVID-19 RNA in human respiratory samples. For each sample, 18.5 µL of SARS-CoV-2 Reaction Mix and 1.5 µL of SARS-CoV-2 Enzyme Mix were transferred into a well of 96 PCR microplate. Then, 10 μL of RNA from each sample or positive or negative control was transferred into the specific well. The PCR plate was loaded into a Bio Rad CFX96TM Real-Time Detection System (Bio Rad) and reactions conditions were set as following: 1 cycle at 50 °C for 20 min, followed by a cycle at 95 °C for 10 min and 40 cycles at 95 °C for 15 s and 60 °C for 30 s. Results were obtained within ~ 60 min. Interpretation of results was performed according to the manufacturer’s instructions.

The Logix Smart™ Coronavirus Disease 2019 (COVID-19) Test Kit

The Logix Smart™ Coronavirus Disease 2019 (COVID-19) Test kit (CO-DIAGNOSTICS, Lake City, UT, USA) was designed to detect the COVID-19 RNA in human respiratory samples. For each sample, 5 µL of Master Mix were transferred into a well of 96 PCR microplate. Then, 5 μL of RNA from each sample or positive or negative control was transferred into the specific well. The PCR plate was loaded into a Bio Rad CFX96TM Real-Time Detection System (Bio Rad) and reaction conditions were set as follows: one cycle at 45 °C for 15 min, followed by one cycle at 95 °C for 2 min and 50 cycles at 95 °C for 3 s and 55 °C for 32 s. Results were obtained within ~47 min. Interpretation of results was performed according to the manufacturer's instructions.

Statistical Analysis

Each sample was tested twice by each assay and there was no difference in the results of the two repeats.

Since our laboratory is a clinical microbiology laboratory, we did not evaluate copies/mL and only performed validation of the assays. We rely on the evaluation that was performed by each manufacturer regarding copies/mL and limit of detection.

We used the Allplex™ 2019-nCoV Assay as the reference method for calculating sensitivity, specificity, and negative and positive predictive values. Therefore, specimens that were found to be positive or negative by the Allplex™ 2019-nCoV Assay were defined as ‘True Positive’ or ‘True Negative’, respectively. Fisher’s exact test was applied for analyzing the differences between the sensitivities of the assays. Agreement rates were calculated as the percentage of samples that had the same results as the reference kit, the Allplex™ 2019-nCoV Assay, out of the total samples.

A Wilcoxon rank sum test was performed to analyze differences in number of positive results between symptomatic and asymptomatic participants. Odds ratio analysis was performed to investigate whether symptoms are predictive of a positive result.

A Chi-square test was applied to investigate the association between agreement and kit type (vs. rapid test) or between agreement and similarity of genes.

A Wilcoxon rank sum test was performed to analyze differences in the minimum cycle threshold (min Ct) (equates to the highest viral load) of agreement cases and disagreement with the reference kit or between symptomatic and asymptomatic participants. A paired t-test was performed to analyze differences in the mean minimum Ct value between traditional and rapid tests.

A repeated-measures ANOVA was applied to test differences in Ct values between PCR kits based on number of detected genes.

Statistical significance was determined with p value < 0.05. Data were analyzed using the R (R Core Team, 2020) software, version 4.0.2.

Results

Two hundred and four participants were enrolled in the study, with 102 positive and 102 negative for the presence of COVID-19 RNA, according to the AllplexTM 2019‐nCOV Assay, which was considered our reference kit. Of the 102 COVID-19-positive participants, 77 were tested due to the presence of symptoms, while the other 25 had no symptoms and were tested due to exposure to/contact with an infected individual.

Performance of Kits, in Comparison with Allplex™ 2019-nCoV Assay

Our main aim was to evaluate the performance of traditional and rapid molecular tests for COVID-19 detection, compared to our reference kit, the Allplex™ 2019-nCoV Assay.

The samples of all 102 COVID-19- negative participants were detected as negative by Logix Smart™ COVID-19 Kit, Xpert® Xpress SARS-CoV-2 test and Simplexa®COVID-19 Direct Kit.

Two (1.96%) out of 102 negative samples were detected as positive by the BD SARS-CoV-2 Reagents for BD MAX™ System (BD) and the Real-Time Fluorescent RT-PCR Kit for Detecting SARS-2019-nCoV had 8 (7.8%) false-positive results. All kits had false-negative results, with Logix Smart™ COVID-19 Kit having the highest (26/102, 25.5%) and the Xpert® Xpress SARS-CoV-2 test having the lowest (9/102, 8.8%) number of false-negative results (Table S2, ESM).

As presented in Table 1, among the rapid tests, the highest agreement levels with the Allplex™ 2019-nCoV Assay occurred with the Xpert® Xpress SARS-CoV-2 test (95.6%), which also had the highest sensitivity (91.2%) and the highest negative predictive value (92%) of all tests.

Table 1

Performance of kits in comparison to Allplex™ 2019-nCoV assay (n = 204)

Assay	Agreementa	Sensitivity	Specificity	PPV	NPV
Traditional assays
Real-Time Fluorescent RT-PCR Kit for Detecting SARS-2019-nCoV	90.2 (184/204)	88.2 (80.3–93.8)	92.2 (85.1–96.6)	91.8 (85.2–95.6)	88.7 (82.1–93)
Logix Smart™ COVID-19 Kit	87.3 (178/204)	74.5 (64.9–82.6)	100 (96.4–100)	100	79.7 (73.8–84.5)
BD SARS-CoV-2 Reagents for BD
MAX™ System	91.2 (186/204)	84.3 (75.8–90.8)	98.04 (93.1–99.8)	97.7 (91.6–99.4)	86.2 (80–90.75)
Rapid assays
Xpert® Xpress SARS-CoV-2 test	95.6 (195/204)	91.2 (84–96)	100 (96.4–100)	100	92 (85.8–95.5)
Simplexa®COVID-19 Direct Kit	89.7 (183/204)	79.4 (70.3–86.8)	100 (96.4–100)	100	83 (76.8–87.7)

PPV positive predictive value, NPV negative predictive value

All values in the table are presented as percentages with confidence intervals in parentheses

a Agreement = percentage of samples with the same results as obtained by the reference kit, the Allplex™ 2019-nCoV Assay

The lowest sensitivity assay among all assays was the Logix Smart™ COVID-19 Kit, with 74.5% sensitivity. All assays had high specificities and positive predictive values. The negative predictive values were in the range of 79.7–92.

Among the traditional tests, the BD SARS-CoV-2 Reagents for the BD MAX™ System had the highest agreement (91.2%) levels with the Allplex™ 2019-nCoV Assay and the highest sensitivity (88.2%).

Percentage of True Positive Results Detected by the Different Assays

Among the traditional molecular assays, the Real-Time Fluorescent RT-PCR Kit for Detecting SARS-2019-nCoV had detected the highest percentage (88.2%) of true positive (TP) results (according to the reference kit). Additionally, this assay had detected the highest percentage of TP results among both symptomatic and asymptomatic participants (93.5% and 76%, respectively) (Table 2).

Table 2

Percentage of true positive results for each kit

Assay	No. of true positive results (n, %)
	All positive participantsn = 102	Symptomatic participantsn = 77	Asymptomatic participantsn = 25
Traditional molecular assays
Real-Time Fluorescent RT-PCR Kit for SARS-2019-nCoV	90 (88.2)	72 (93.5)	19 (76)
Logix Smart™ COVID-19 Kit	76 (74.5)	58 (75.3)	18 (72)
BD SARS-CoV-2 Reagents for BD MAX™ System (BD)	86 (84.3)	69 (89.6)	17 (68)
Rapid molecular assays
Xpert® Xpress SARS-CoV-2 test	93 (91.2)	75 (97.4)	18 (72)
Simplexa®COVID-19 Direct Kit	81 (79.4)	70 (91)	12 (48)

The percentage of TP results of symptomatic participants was higher than that of asymptomatic participants in all the traditional assays.

Regarding the rapid molecular assays, the highest percentage of TP results was detected by the Xpert® Xpress SARS-CoV-2 test for all participants and for symptomatic and asymptomatic participants (91.2%, 97.4% and 72%, respectively). As with the traditional assays, the percentage of TP detected by the rapid tests was higher when participants had symptoms.

Percentage of True Positive Results According to the Number of Detected Genes

We were interested whether the percentage of TP results detected by the assays is affected by the number of detected genes. We categorized the number of genes detected according to the number of genes each kit detects (and not according to the test results of each gene). Therefore, three-genes-kit refers to Allplex 2019‐nCOV Assay, two-genes-kit refers to Simplexa® COVID-19 Direct Kit (Focus Diagnostics), BD SARS-CoV-2 Reagents for BD MAX™ System (BD), and Xpert® Xpress SARS-CoV-2 test (Cepheid).

As shown in Table 3, the percentage of TP results increased with the number of detected genes. Additionally, a higher number of TP results were obtained among asymptomatic participants with three genes (25, 100%) compared to two genes (22, 88%). Similarly, the number of TP results among symptomatic participants increased with the number of genes detected (Table 3).

Table 3

Percentage of true positive results according to number of detected genes

No. of detected genes	Average no. of true positive results (n, %)
	All participants	Symptomatic participants	Asymptomatic participants
1	93 (91.2)	71 (92.2)	22 (88)
2	98 (96.1)	76 (98.7)	22 (88)
3	102 (100)	77 (100)	25 (100)

The Association Between Symptoms and Positive Results

One of this study’s aims was to investigate whether the kits’ results are affected by symptom presentation. For each positive sample, we looked at the number of positive tests by specimen and compared these numbers between symptomatic and asymptomatic participants. As shown in Fig. 1, most positive samples among symptomatic participants were interpreted as positive by either five or six kits, while samples of asymptomatic participants were determined as positive by four or five kits (p < 0.001).

Fig. 1

Boxplot of sum of positive results for all kits, divided according to symptoms presentation. The minimum, maximum and median (indicated by the bold line) of the numbers of positive tests by specimen are presented. ***p < 0.001

Next, we evaluated the odds ratio, with symptoms as predictive of a positive result (Table 4). Logistic regression analysis revealed that the odds ratio (OR) for positive results when the participant had symptoms was higher than 1, with Xpert® Xpress SARS-CoV-2 having the highest OR (14.6). The OR was statistically significant for all kits except of the Logix Smart™ COVID-19 Kit.

Table 4

Odds ratio analysis results with symptoms as predictor of a positive result

Assay	Logistic regression
	OR	p value
Traditional molecular assays
Real-Time Fluorescent RT-PCR Kit for Detecting SARS-2019-nCoV	3.7 (1.06–13.3)	<0.05
Logix Smart™ COVID-19 Kit	1.2 (0.4–3.2)	Ns
BD SARS-CoV-2 Reagents for BD MAX™ System	4.1 (1.32–12.6)	<0.05
Rapid molecular assays
Xpert® Xpress SARS-CoV-2	14.6 (3.2–103.7)	<0.01
Simplexa® COVID-19 Direct Kit	9.3 (3.3–28.6)	<0.001

OR odds ratio, ns non-significant

Factors That Affect Agreement With the Gold Standard Assay

One aim was to investigate factors that may influence the agreement with our reference kit, the Allplex™ 2019-nCoV Assay. First, we looked at the min Ct for each sample. We divided the positive results into results with agreement with the Allplex™ 2019-nCoV Assay and results with disagreement, and compared the min Ct of each group. We found that when the kits disagreed with the reference kit, the min Ct was higher compared to the agreement cases (min Ct = 38.9 and 31.26, respectively) (p < 0.001) (Fig. 2).

Fig. 2

Boxplot of minimum CT value (for all kits) in agreement with the Allplex™ 2019-nCoV Assay. The minimum, maximum and median (indicated by a bold line) are presented. ***p < 0.001

Second, we tested whether the kit’s type (rapid test or traditional assays) affects the agreement of the kit with the reference kit. A Chi-square test that compared the proportion of agreement between rapid tests (88.2%) or traditional assays (85.5%) with the reference kit did not find any association between type of kit and agreement with the Allplex™ 2019-nCoV Assay (p = 0.238)

Third, we investigated whether the agreement of kits with the Allplex™ 2019-nCoV Assay was affected by similarity of detected genes. No difference was found in the proportion of results with agreement between kits with common genes (86%) as detected by the Allplex™ 2019-nCoV Assay and kits with different genes (86.9%) (p = 0.75).

Factors That Affect the Min Ct

In Real-Time PCR, the Ct indicates reciprocally on the viral load. High Ct values may indicate low viral load that can be found at the beginning or at the end of disease. Additionally, for each test there is a range of accepted Ct for a positive result interpretation. Therefore, we wanted to explore factors that might influence the min Ct for each sample.

First, we compared the min Ct between the rapid tests and the traditional assays, and found that the traditional assays had a higher min Ct compared to the rapid tests (30.75 and 29.12, respectively) (p = 0.012) (Fig. 3A).

Fig. 3

Boxplot of minimum Ct value (for all kits) for different factors. The minimum, maximum and median (indicated by a bold line) are presented for each graph. (a) The effect of method types on minimum (Min) Ct, *p < 0.05; (b) the effect of symptom presence on Min Ct, ***p < 0.001; and (c) the effect of number of detected genes on Min Ct, ***p< 0.001 for the comparison of 3-gene kits and 2-gene kits with 1-gene kit, #p < 0.05 for the comparison of 3-gene kits with 2-gene kits. One-gene kits refer to Real‐Time Fluorescent RT‐PCR Kit for Detecting SARS-2019-nCoV (BGI Genomics) and The Logix Smart™ Coronavirus Disease 2019 (COVID-19) Test kit (CO-DIAGNOSTICS); 2-gene kits refer to Simplexa® COVID-19 Direct Kit (Focus Diagnostics), BD SARS-CoV-2 Reagents for BD MAX™ System (BD), and Xpert® Xpress SARS-CoV-2 test (Cepheid); the 3-gene kit is the Allplex™ 2019-nCoV Assay

Second, we found an association between symptoms presence and min Ct, with samples from symptomatic participants having lower min Ct than samples of asymptomatic participants (29.4 and 33.44, respectively) (p < 0.001) (Fig. 3B).

Another factor that affected the min Ct was the number of genes detected by kits. The more genes that were detected, the lower the min Ct (p < 0.001) (Fig. 3C); the average min Ct values were 33.9, 28.8, and 26.57 for one-gene, two-gene and three-gene kits, respectively. A post hoc analysis revealed a significant difference between one-gene-kits and two-genes kits (p < 0.001), between one-gene kits and three-genes kits (p < 0.001) and between two-gene kits and three-gene kits (p = 0.016).

Discussion

The main aim of the current study was to evaluate the performance of six molecular tests for COVID-19 detection and to allocate factors that may affect the tests’ performance. The Food and Drug Administration Emergency Use Authorizations (FDA-EUA) have approved various molecular tests for the detection of COVID-19 [6]. These assays differ in many characteristics, starting from the sample type and the sample medium, through the extraction procedure, number and identity of detected genes, turnaround time, number of samples per run, and limit of detection [6]. Therefore, any laboratory that performs COVID-19 molecular tests should be familiar with its kit’s advantages and disadvantages.

We found that Xpert® Xpress SARS-CoV-2 test had the highest sensitivity among all tests, compared to the reference kit. This finding was not surprising in light of data from previous publications that have presented even higher sensitivities of this assay (97–100%), compared to the current study [7-9]. Additionally, a recent report that compared the performances of seven different primer-probe sets concluded that primers that target the N2 or the E genes, as in the Xpert® Xpress SARS-CoV-2 test, have higher sensitivity, compared to primers with other gene targets [10].

The least sensitive assay among all assays was the Logix Smart™ COVID-19 Kit, with 74.5% sensitivity. However, we have not find any previous publication for a reference. Nevertheless, it was shown that primers targeted against the RdRp gene were less sensitive compared to primers targeted against N2 and E genes [10]. Thus, this evidence can explain the relatively low sensitivity of the Logix Smart™ COVID-19 Kit, which amplifies the RdRp gene.

It should be mentioned that although various studies have evaluated some of the assays that were tested in the current study, several differences in study design (i.e., the extraction process, the study participants, etc.) may affect the different performance measures’ results. For example, in the current study, all assays were compared to the Allplex™ 2019-nCoV Assay. Different studies have used other assays as their reference kit [7–9, 11, 12]. Therefore, the performance of a specific kit may change when compared to a different assay.

It is known that various factors can contribute to variations in kit performances, including different gene targets and different primer-probe sets, the need for RNA extraction step, limits of detection, etc. [5].

One common factor that affected all tests was the presence of patient symptoms. As was shown earlier, the percentage of true positive results was higher among symptomatic participants compared to asymptomatic participants in all assays. As we also saw, symptomatic patients had lower Ct values compared to asymptomatic patients, so we assume the lower Ct values contribute to the higher TP percentage. Additionally, the OR analysis has found that symptoms were predictive of a positive result in most kits. These findings are probably associated with viral load differences; it was shown that the highest viral load was detected at symptom onset and decreased during the first 10 days [13]. Another study has reported that the Ct was lower when sampling was performed close to symptom onset and increased as the gap between symptom onset and sampling day increased [7]. Therefore, we believe that symptomatic participants have a higher viral load compared to asymptomatic patients, leading to a lower Ct determination and higher probability of interpretation as a positive result.

Another factor that was investigated in the current study with regard to kit performance is the number of detected genes. We showed that the larger the number of genes detected, the larger the number of true positive results, among all participants and among the sub-groups of symptomatic and asymptomatic participants. A former study has presented an enhanced sensitivity of assays with two gene targets compared to a one-gene target kit [12]. It is known that the COVID-19 virus has evolved over quite a short period of time. Whenever there is a change in the genome area to which the primer-probe set is usually attached, the test's sensitivity may decrease. Therefore, detecting more than one gene target can reduce the risk for sensitivity decrease due to viral evolution. As proof of this assumption, a recent study has screened available SARS-CoV-2 genomes, and found single nucleotide mutations that influenced the annealing of all RT-PCR assays tested in the study. They concluded that reduction in kit performance can be minimized with the implementation of more than one gene target [14].

Our next aim was to investigate factors that affect the agreement of kits with the reference kit. We found that disagreement cases were characterized with higher min Ct values. In addition to the fact that each kit has a different cut-off, there is an ongoing debate regarding the cut-off Ct that should be used to interpret a positive result. The reason for this argument is associated with evidence that the viral load and therefore the virus culturing rate decreases as the Ct increases [13, 15–18]. For example, Singanayagam et al. [13] found a probability of 8% to culture the virus from samples with a Ct above 35 [13]. Thus, we assume that the disagreement cases in our study represent samples with a low viral load.

Other factors that we explored in relation to agreement with the reference kit were the kit's type (rapid test or traditional assays) and similarity of detected genes. No associations were found between these factors and agreement with the Allplex™ 2019-nCoV Assay.

Our last goal was to investigate factors that affect the min Ct. First, to our surprise, we found a higher min Ct in the traditional assays compared to the rapid tests. It is known that tests with no nucleic acid step usually contain a large number of amplification inhibitors [19]. Therefore, we expected a slightly higher Ct in the rapid tests. Our results may suggest that the tested rapid tests overcome this disadvantage of amplification inhibitors.

Second, samples from symptomatic participants had a lower min Ct than samples from asymptomatic participants. This finding supports our assumption that symptomatic patients have a higher viral load compared to asymptomatic patients.

The last factor that affected the min Ct was the number of genes detected by kits. The more genes that were detected, the lower the min Ct. As mentioned earlier, as the virus has gained a considerable number of mutations since its first appearance, the binding efficiency of several primer-probe sets to their target genes may have been reduced, which may affect kit performance. Therefore, detecting more genes will compensate for a reduction in kit performance.

The study has several limitations. First, we used the Allplex™ 2019-nCoV Assay as the reference kit based on a decision by the Israeli Ministry of Health. A wider evaluation should be performed with other kits serving as the reference kit. Second, we had a limited number of samples and specifically samples of asymptomatic patients.

Conclusions

In conclusion, our results suggest that most assays for COVID-19 detection are characterized with overall good performance measurements. The differences in the various kit performances are derived from both controlled (such as sampling efficacy) and uncontrolled factors (presence of symptoms). Therefore, each laboratory must be familiar with its kit disadvantages and limitations in order to produce the most reliable results.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 19 kb)

We evaluated the sensitivity and specificity of six molecular kits that detect COVID-19 and investigated several factors that may affect the performance of the kits.

We found that Xpert® Xpress SARS-CoV-2 test had the highest sensitivity (91.2%).

The performance of the kits was affected by the presence of symptoms, number of genes the kit detected, and type of RNA extraction.