Evaluation of six different rapid methods for nucleic acid detection of SARS-COV-2 virus.

In the current coronavirus disease 2019 (COVID-19) pandemic there is a mass screening of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) happening around the world due to the extensive spread of the infections. There is a high demand for rapid diagnostic tests to expedite the identification of cases and to facilitate early isolation and control spread. Hence this study evaluates six different rapid nucleic acid detection assays that are commercially available for SARS-CoV-2 virus detection. Nasopharyngeal samples were collected from 4981 participants and were tested for the SARS-CoV-2 virus by the gold standard real-time reverse-transcription polymerase chain reaction (RT-PCR) method and with one of these six rapid methods of detection. Evaluation of the rapid nucleic acid detection assays was done by comparing the results of these rapid methods with the gold standard RT-qPCR results for SARS-COV-2 detection. AQ-TOP had the highest sensitivity (98%) and a strong kappa value of 0.943 followed by Genechecker and Abbot ID NOW. The POCKIT (ii RT-PCR) assay had the highest test accuracy of 99.29% followed by Genechecker and Cobas Liat. Atila iAMP showed the highest percentage of invalid reports (35.5%) followed by AQ-TOP with 6% and POCKIT with 3.7% of invalid reports. Genechecker system, Abbott ID NOW, and Cobas Liat were found to have the best performance and agreement when compared with the standard RT-PCR for COVID-19 detection. With further research, these rapid tests have the potential to be employed in large-scale screening of COVID-19.

BACKGROUND

The coronavirus disease 2019 (COVID‐19) pandemic has affected more than 112 million people worldwide and continues to spread and remains a public health challenge. The real‐time reverse‐transcription polymerase chain reaction (RT‐PCR) remains the gold standard for testing severe acute respiratory syndrome coronavirus 2 (SARS‐COV‐2) and is approved by both the WHO and Centers for Disease Control and Prevention. But this method requires a well‐established lab setup, expensive instruments, well‐trained and skilled manpower, and long hours. In the current scenario of the pandemic, with such large numbers being affected conducting these tests with limited lab capacities is challenging. The lab facilities are overburdened, and molecular testing is time‐consuming, which delays reporting and that in turn impacts the containment of the spread. Hence there is a demand for alternative testing strategies that are rapid and less sophisticated.

In SARS‐CoV‐2, human‐to‐human transmission is through droplets or direct contact and its symptoms are very similar to flu, hence molecular tests are critical to differentiate and detect COVID‐19 infections. In the early stages of infection, the viral load is usually high in patients and studies show that a single swab can contain more than a million viral particles, hence nucleic acid testing is the most efficient form of testing in the early stages, and identifying infections earlier is vital. As mass testing, early detection, and isolation are crucial for containing the spread of infection, evaluation of these rapid tests becomes imperative.

Currently, there are many nucleic acid detection assays that have obtained emergency authorization by the US Food and Drug Administration to detect SARS‐CoV‐2 and have been largely implemented around the globe.

The study compares six different molecular tests that detect nuclear RNA of SARS‐COV‐2 and evaluate them against the standard RT‐PCR.

MATERIALS AND METHODS

This study obtained ethical approval from the Institutional Review Board of the Department of Health, Abu Dhabi. The positive samples were collected from patients at the COVID‐19 quarantine facilities who were tested positive for COVID‐19 and were either asymptomatic or had mild symptoms. The negative samples were collected from people who visited the lab for screening for COVID‐19 infections for travel and other purposes. The participants were approached randomly and whoever was willing to give consent was included in the study. After getting informed consent from the participants, their nasopharyngeal swabs (NPS) were collected. The standard RT‐PCR requires a wet swab, for which a sterile nasopharyngeal swab was used for collection and then was placed into the universal transport medium. A dry swab is required for Abbott ID NOW and Atila iAMP COVID‐19 detection tests. The Atila iAMP COVID‐ 19 Detection Kit was provided with a dry swab that was used for collection and these swabs were transported to the lab in RNAase/DNAase free Eppendorf tubes. The FLOQSwabs (COPAN Flocked Swabs) dry swab was used for sample collection for Abbott ID NOW. This was done as a bedside test immediately after collection. All samples that were collected outside the lab were transported to the lab immediately.

The standard RT‐PCR was performed in the lab, where RNA extraction was performed by the automated machine MGISP−960 as per the manufacturer's instructions. After the RNA extraction, 10 μl of the sample extract is added to 20 μl of the master mix (BGI RT‐PCR fluorescence KIT). Both, the extraction method and the BGI RT‐PCR fluorescence KIT were verified in‐house. The real‐time fluorescent RT‐PCR was done using the Bioer LineGene 9600 Plus Fluorescent Quantitative Detection System.

A total of 4981 participants were tested for the SARS‐CoV‐2 virus by the gold standard RT‐PCR method for SARS‐CoV‐2 detection and with one of these six rapid methods of detection. The rapid tests were evaluated by comparison of the rapid nucleic acid detection assays results with the standard RT‐PCR results for SARS‐COV‐2 detection. Details of the six rapid detection methods are given in Table 1.

Table 1

The six rapid nucleic acid detection assays

Test	Principle	Gene detected	Sample	TAT	Throughput
The Abbott ID NOW COVID‐19 assay	NEAR isothermal amplification	RdRp	Dry	13	1
Atila iAMP COVID‐19 detection	OMEGA amplification	ORF‐1 N	Dry	60	96
AQ‐TOP Plus COVID‐19 Rapid Detection Kit	Loop isothermal nucleic acid amplification technology (LAMP)	ORF‐1 N	Wet	20	96
Genechecker PCR system‐UF 300–RT PCR system	Microfluidic chip‐based PCR method	N RdRp	Wet	45	4
Cobas Liat system SARS‐CoV‐2 and Influenza A/B nucleic acid test	Automated multiplex real‐time RT‐PCR assay	ORF‐1 N	Wet	20	1
POCKIT SARS‐CoV‐2 (orf lab) (RT‐ii PCR) assay	Insulated isothermal polymerase chain reaction(ii‐PCR)	ORF‐1	Wet	85	8

Abbreviations: COVID‐19, coronavirus disease 2019; RT‐PCR, real‐time reverse‐transcription polymerase chain reaction; SARS‐CoV‐2, severe acute respiratory syndrome coronavirus 2.

Statistical analysis

The results of the standard RT‐qPCR were used as a standard reference and the results obtained from the rapid methods were compared to the standard reference reports.

Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy were calculated for each method. Cohen's kappa values of agreement were calculated along with positive percentage agreement (PPA), negative agreement percent (NPA), and overall agreement percentage. SPSS statistical software was used for all statistical analysis.

RESULTS

The sensitivity, specificity, PPV, NPV, and accuracy were calculated for the six rapid methods by comparing with the standard RT‐PCR results. The results are shown in Figure 1.

Figure 1

Comparison of the rapid methods of nucleic acid detection

AQ‐TOP and Genechecker had the highest sensitivity and PPV followed by the sensitivity of Abbott ID NOW and Cobas Liat. The Atila iAMP COVID‐19 test had the lowest sensitivity. The POCKIT had the highest accuracy followed by Genechecker and Cobas Liat.

The positive, negative, and overall agreement percentage along with Cohens' kappa value were calculated for the six methods and are presented in Table 2.

Table 2

Agreement of the rapid tests results with the standard reference test results

	Standard reference test		Kappa (k) (p value)	Positive agreement (%)	Negative agreement (%)	Overall Agreement (%)
Molecular assays (n)	Positive	Negative
Atila iAMP (127)
Positive	30	2	0.391	44.11	96.61	68.50
Negative	38	57	(<0.001)
AQ‐TOP (212)
Positive	102	4	0.943	98.07	96.29	91.15
Negative	2	104	(<0.001)
Genechecker (1128)
Positive	120	8	0.938	95.23	99.20	98.75
Negative	6	994	(<0.001)
Abbott ID NOW (686)
Positive	158	16	0.906	95.18	96.92	96.50
Negative	8	504	(<0.001)
Cobas Liat (524)
Positive	35	11	0.830	94.59	97.74	97.52
Negative	2	476	(<0.001)
POCKIT (2131)
Positive	65	8	0.893	90.27	99.61	99.29
Negative	7	2051	(<0.001)

The positive percentage agreement with the standard RT‐qPCR was strongest with AQ‐TOP followed by Genechecker. Similarly, the negative agreement was best with POCKIT, Cobas Liat, and Genechecker. AQ‐TOP. The overall agreement was highest with POCKIT and the Atila iAMP had the poorest agreement with the standard RT‐PCR results.

The number of positive cases missed, and details of the results reported invalid by the rapid tests are shown in Table 3. Atila iAMP showed the highest percentage of invalid reports missing a large proportion (44.7%) of positive cases followed by AQ‐TOP with 7% invalid reports. The Cobas Liat test did not show any invalid reports.

Table 3

Invalid reports and positive cases missed by the rapid methods

Rapid methods (n)	Invalid reports (n (%))	Positive cases reported by standard RT‐qPCR missed by the rapid tests (n (%))
Atila iAMP COVID‐19 detection (197)	70 (35.5)	55 (44.7)
AQ‐TOP COVID‐19 Rapid Detection (226)	14 (6.1)	3 (2.8)
Genechecker‐UF 300–RT PCR system (1133)	5 (0.4)	5 (3.1)
The Abbott ID NOW COVID‐19 assay (689)	3 (0.4)	2 (1.2)
Cobas Liat (524)	0	0
POCKIT SARS‐CoV‐2 (orf lab) Premix Reagent (2212)	81 (3.7)	10 (1.3)

Abbreviations: COVID‐19, coronavirus disease 2019; RT‐qPCR, quantitative real‐time reverse‐transcription polymerase chain reaction; SARS‐CoV‐2, severe acute respiratory syndrome coronavirus 2.

DISCUSSION

In this ongoing COVID‐19 pandemic with the extensive transmission of infection, rapid molecular tests are very crucial for early identification and isolation. The molecular assays evaluated in this study are all rapid with low complexity and thus require less hands‐on time, which is the need of the hour, hence assays were evaluated in comparison with the standard RT‐PCR.

Our study found that the sensitivity of the Atila iAMP assay was very low. The Atila iAMP assay has been claimed to have around 87% PPA and 100% NPA with the reference RT‐PCR assay, but in this study, we found that the sensitivity was very low, and it reported a high number of invalid results, missing more than 40% of positive cases when compared with the standard RT‐qPCR results. The reasons for a high rate of invalid reports were unclear; however, the lack of extraction and amplification inhibitors, which can lead to inadequate amplification could be reasons that need further exploration.8, 9

The agreement percentage (kappa agreement) with the NPS results was low. In this assay, 93 positive samples were reported as either negative or invalid and the mean Ct value of these positive samples was 31.54 ± 4.84. The test missed weak/low viral load samples, which had Ct values more than 30. This was supported by the study at Stanford, which showed Atila had lower sensitivity and required a high volume of the nucleic acid eluate. Further studies have shown that molecular tests using the LAMP technology have reported false‐negative results in low viral load. However, the assay showed high specificity, which can be explained by the four different primers that are used in this assay to detect six different sequences of the RNA of SARS‐COV‐2. This test can detect SARS‐CoV‐2 RNA directly from samples and does not require prior RNA extraction, thus reducing the time to run along with the ability to process 96 samples per run and providing the results within 60 min. While there are positive aspects, the sensitivity and reliability of the method are important parameters for screening, hence with lower sensitivity higher percentages of false‐negative results would be expected and the high percentage of invalid results makes the test less reliable as a rapid method, for these reasons the test cannot be widely employed as a screening test.

In our study, the Abbott and AQ‐TOP both used the LAMP technology and had high sensitivity, specificity, PPA, NPA, and the kappa coefficient was strong. Similar studies have reported LAMP assays to be identical with the RT‐PCR tests and reported similar sensitivity and specificity. The drawback of AQ‐TOP was that it showed 6% invalid reports, which is relatively higher than the other tests. But this method is quick with faster amplification and high throughput, 96 samples can be tested in each run, results are easily readable, and requires less specialized equipment. It further has the ability to amplify multiple targets in a single reaction.

The Abbott ID NOW had an accuracy of 96.5%, with sensitivity and specificity compared with the standard RT‐qPCR and the lowest percentage of invalid reports compared with other methods. It also showed a strong agreement with the RT‐PCR results. A study by Rhoads in the United States reported similar findings, which showed that the assay had a good PPA of 94%. Another study done in New York had shown that when compared with Xpert Xpress, ID NOW revealed good PPA only when the sample had low Ct values and the PPA was low in the higher Ct value samples or when the viral load was low. Similar reports were published by another study that validated RT‐LAMP, which showed that samples having Ct values less than 30 showed 100% sensitivity, while samples with Ct values more than 35 showed only 54% concordance with the RT‐qPCR results. However, in our study, we found that the concordance with the RT‐qPCR results did not change with the Ct values. Furthermore, the Abbott ID is compact and can be used as a bedside test, and it is rapid as it detects the positive results in just 7 min and negative results in 13 min, although only one sample can be tested per run.

The Genechecker had high sensitivity and concordance with the RT‐qPCR reports, with the least percentage of invalid reports. It has a statistically significant 0.938 Cohen's kappa value and thereby strong agreement with the results of the standard reference test. Furthermore, this method has a turnaround time of 45 min and four samples can be tested per run. But the preparation of the samples requires 5–10 min and requires manual pipetting, which is recommended to be done in a biosafety cabinet. Therefore, though Genechecker is a compact instrument it cannot be used as a bedside test. Also for efficient usage, four samples need to be run together as there would be wastage of the kit cartridges. There are not many studies on the evaluation of Genechecker but our study shows the method has great potential as a rapid detection method.

The Cobas Liat automated RT‐PCR showed the PPA of 94.5% and accuracy of 97.5% with the lowest PPV of 76% and the results can be obtained in 20 min. Another study that evaluated the clinical performance of Cobas Liat showed that its accuracy was 98.6%, positive percent agreement was 100%, and negative percent agreement was 97.4%, which was very similar to our study report. However a recent report warns of potential false positives with Cobas rapid test that was alleged to sporadic assay tube leakage or abnormal PCR cycling in the reaction tubes. This might explain the high false positives and lowest PPV observed in our study with Cobas Liat.

The POCKIT SARS‐CoV‐2 (orf lab) (RT‐ii PCR) assay had the highest overall agreement with the reference RT‐PCR method. The turnaround time is 85 min with a throughput of eight samples per run. The positive study that validated its clinical performance comparing it to standard RT‐PCR assay also showed that positive agreement was 96.8% and kappa value of 0.93. However, in our study, the positive agreement was only 90%; this might be because the number of positive samples tested was only 75 samples compared with 2137 negative reports. The test reported a high number of invalid reports, which was 3.7% of the total samples.

STRENGTHS AND LIMITATIONS

The study has evaluated commercial assays available in the market for rapid detection of SARS‐COV‐2, which is the need of the hour. All the results are compared with the gold standard RT‐PCR recommended for SARS‐COV‐2 detection, which gives an insight into the performance of these rapid methods based on the standard comparison.

The limitation is that the number of samples tested with each rapid method varies, which was based on the availability of kits and reagents. Furthermore, this study did not consider the severity of symptoms and the days of infection to correlate with the test results.

CONCLUSION

In summary, the evaluation of six rapid methods of detection of SARS‐COV‐2 shows that the Atila OMEGA amplification method had low sensitivity with a high number of invalid, which made the test unreliable. While AQ‐TOP had good sensitivity and a rapid turnaround time of 20 min; the percentage of invalid reports was high with this method. The POCKIT had the highest overall agreement percentage with the standard RT‐PCR but with lower sensitivity than most other rapid tests and a high number of invalid reports. AQ‐TOP, Abbott, and GeneChecker had high sensitivity a strong kappa value, and a rapid turnaround time of 20, 15, and 45 min, respectively. Cobas Liat, Abbott ID NOW, and Genechecker system had the least number of invalid reports; however, the throughput for each is 1, 1, and 4 samples, respectively. Further studies analyzing all these parameters are needed, to consider implementing these rapid tests on a large scale for the detection of SARS‐CoV‐2.

CONFLICT OF INTERESTS

The authors declare that there are no conflict of interests.

ETHICS STATEMENT

The Ethics approval was obtained from the Department of Health (DOH) Institutional Review Board (IRB), Abu Dhabi. All methods were carried out in accordance with relevant guidelines and regulations. The DOH IRB, Abu Dhabi. Informed consent was obtained from all participants in the study as per the regulations of the DOH IRB, Abu Dhabi.

AUTHOR CONTRIBUTIONS

Sally A. Mahmoud: conceptualization, methodology, validation, reviewing and editing, supervision, and project administration. Subhashini Ganesan: Conceptualization, methodology, writing original draft, validation, reviewing, and editing. Esra Ibrahim: Conceptualization, methodology, validation, reviewing, and editing. Bhagyashree Thakre: Methodology, validation, reviewing, and editing. Juliet G. Teddy: Methodology, validation, reviewing, and editing. Preety Raheja: Methodology, validation, reviewing, and editing. Walid Z. Abbas: Reviewing, editing, supervision, and project administration.