Exploring beyond the limit: How comparative stochastic performance affects retesting outcomes in six commercial SARS CoV-2 nucleic acid amplification tests.

Objectives: To examine the comparative stochasticity profile of six commercial SARS-CoV-2 nucleic acid amplification tests (NAATs) and how this may affect retesting paradigms.
Methods: Commercial quality control (QC) material was serially diluted in viral transport media to create a panel covering 10-10,000 copies/ml. The panel was tested across six commercial NAATs. A subset of high cycle threshold results was retested on a rapid PCR assay to simulate retesting protocols commonly used to discriminate false positives.
Results: Performance beyond the LOD differed among assays, with three types of stochasticity profiles observed. The ability of the rapid PCR assay to reproduce a true weak positive specimen was restricted to its own stochastic performance at the corresponding viral concentration.
Conclusion: Stochastic performance of various NAATs overlap across low viral concentrations and affect retesting outcomes. Relying on retesting alone to discriminate false positives risk missing true positives even when a more sensitive assay is deployed for confirmatory testing.

Introduction

Interpreting weak positive results from SARS-CoV-2 nucleic acid amplification tests (NAATs) can be challenging as there are concerns that high cycle threshold (ct) value (‘weak’) positives could be potentially false positives (e.g. due to non-specific amplification) [1]. At different stages of the pandemic, some jurisdictions retest to “confirm” such results, with initial positives being over-ridden by subsequent negatives from retesting [1,2]. However, weak positive results may mean the samples contain small quantities of nucleic acids, beyond the limit of detection (LOD) of the NAATs. At these low levels, NAATs’ performance is probabilistic, depending on capture of viral material through sampling and the assay chemistry. An appreciation of the comparative stochastic performance of various NAATs is therefore important to understand the benefits and pitfalls of confirmatory retesting. Research comparing the stochasticity of different NAATs and its implication in confirmatory retesting is lacking [3], [4], [5]. We therefore examined the stochastic performance of six commercial SARS-CoV-2 NAATs using serial dilutions of a commercial QC material and examined its relationship to retesting outcomes.

Methods

Abbott Alinity m SARS-CoV-2 (Alinity m system), Abbott RealTime SARS-CoV-2 (m2000 system), Cepheid Xpert Xpress SARS-CoV-2 (GeneXpert IV system), Hologic Aptima SARS-CoV-2 (Panther instrument), Roche cobas SARS-CoV-2 (cobas 6800 system), and Seegene Allplex SARS-CoV-2 (Seegene STARlet/BioRad CFX instruments) assays were evaluated in this study. All assays and materials were used as per the manufacturer's instructions for use. Briefly, commercial lyophilised QC material registered with Therapeutic Goods Administration (Australia) for use as an in vitro diagnostic device (Microbiologics, Minnesota, USA, catalogue number: HE0065N) was rehydrated and serial-diluted with viral transport media (KangJian, Jiangsu, China, catalogue number: KJ502‐19) to the described concentrations (Table 1 ) for testing across six NAATs between January and May 2021. The six NAATs were run in parallel six times, each time with freshly prepared serially-diluted QC material, (except the cobas assay which was run twice due to reagent availability issue). To better demonstrate probabilistic detection beyond LOD with limited assay availability during pandemic times, more replicates were ran at close intervals of lower viral copies concentration (rather than higher concentrations). Fewer samples were also run on the Xpert due to test cartridge scarcity.

Table 1

Performance of six SARS-CoV-2 nucleic acid amplification tests.

Copies/ml	Seegene	Aptima	Realtime	cobas	Xpert	Alinity m
10,000	100% (9/9)	100% (9/9)	100% (9/9)	NA	NA	100% (9/9)
1000	100% (19/19)	100% (19/19)	100% (19/19)	NA	NA	100% (19/19)
500	100% (10/10)	100% (10/10)	100% (10/10)	NA	NA	100% (10/10)
250	70% (7/10)	100% (10/10)	100% (10/10)	100% (10/10)	NA	100% (10/10)
125	60% (6/10)	60% (12/20)	85% (17/20)	100% (10/10)	NA	100% (10/10)
100	46% (13/28)	28% (11/39)	83% (24/29)	100% (20/20)	100% (19/19)	97% (28/29)
75	45% (9/20)	30% (6/20)	55% (11/20)	75% (15/20)	90% (9/10)	100% (20/20)
50	20% (6/30)	30% (6/20)	43% (13/30)	75% (15/20)	53% (8/15)	90% (27/30)
25	5% (1/19)	0% (0/10)	30% (3/10)	35% (7/20)	0% (0/5)	85% (17/20)
10	0% (0/19)	0% (0/19)	0% (0/19)	10% (2/20)	13% (3/24)	23% (8/39)

Some laboratory services utilised the Xpert for a rapid retest of specimens with high ct values to “confirm” initial positive results. To examine such a retesting protocol, a subset of the low concentration samples, tested positive with high ct on PCR-based NAATs, were retested on the Xpert assay. The TMA-based Hologic Aptima assay produces RLU values that are not linear against viral concentrations so there is no equivalent of a high ct value and was therefore excluded from the retesting component of the study [6].

Results

We observed three different stochasticity profiles in NAATs beyond the concentrations where they detect 100%: gradual decline in detection probability (Seegene, Aptima, Realtime), quick drop off (Xpert), or maintenance of high detection probability until very low viral concentrations (cobas, Alinity m). For example, the Seegene assay detected 100% at 500 copies/mL and took a five-fold reduction in specimen analyte concentration (100 copies/mL) until detection probability halves. In contrast, the Xpert detected 100% at 100 copies/mL but detection probability halves as soon as analyte concentration halves (50 copies/mL), while the Alinity m detected with high probabilities until 10 copies/mL.

In the retesting experiments, a panel of 35 specimens at 10 to 100 copies/mL, initially tested positive with high ct values, were retested. The retesting assay, Xpert, gave a positive result at a probability similar to that at the corresponding concentrations as in its initial assessment (Tables 1 & 2 ).

Table 2

Retesting of high ct results from four PCR-based SARS-CoV-2 tests using the Xpert assay.

Xpert SARS CoV-2 Results					Xpert retesting total sensitivity
Copies/ml	Seegene	Realtime	cobas	Alinity m
100	86% (6/7)	NA	NA	100% (5/5)	92% (11/12)
50	67% (4/6)	40% (2/5)	NA	57% (4/7)	56% (10/18)
25	100% (1/1)	0% (0/2)	NA	NA	33% (1/3)
10	NA	NA	0% (0/2)	NA	0% (0/2)

Discussion

The current study provides the first analysis of the comparative stochastic performance across six commercial SARS-CoV-2 NAATs. Retesting of positive specimens was performed in a similar manner to previously described testing protocols [2,7,8]. The focus of this study was not to verify the LODs, but to examine a range of commonly utilised commercial assays’ differing stochastic characteristics beyond the LODs. The study also demonstrated that the assays’ probabilistic detection overlaps significantly across low analyte levels.

There have been concerns regarding results with high ct values to be false positives [1,2]. Many services therefore retest, using either the same assay and/or another assay with better sensitivity, to discriminate these false positives [2,7]. Retesting protocols assume positive results that cannot be reproduced (“confirmed”) to be false [2,7]. Thus, some NAATs are essentially deployed as “confirmatory”. However, none of the NAATs evaluated in this study were validated by the manufacturers as confirmatory assays, and there is a lack of guidance on the process of confirming SARS-CoV-2 molecular diagnosis. Further validation of the NAATs as laboratory developed confirmatory tests will be required.

The current study demonstrates that even with a more sensitive retesting assay, a true positive from a less sensitive initial NAAT need not be “confirmed” (Table 2). The retesting assay can only “confirm” positive results at a probability consistent with its intrinsic sensitivity at the corresponding concentrations. Moreover, the stochastic performances of many assays overlap over lower concentrations (Table 1). In our case, while Xpert has a 5-fold lower LOD (100 copies/mL vs 500 copies/mL) than Seegene, Seegene still detect samples with viral concentrations beyond the LOD of Xpert at high enough frequencies. At 50 copies/mL, Xpert only detects about twice as much as Seegene, and may miss a true positive detected by the initial Seegene assay at high enough frequency.

It is also worth considering how the Xpert assay is applied for retesting when the initial weak positives came from an assay having a higher sensitivity/better stochastic performance, e.g., at 50 copies/mL both the Alinity m and the cobas were more likely to produce a positive result than the Xpert (Table 1).

High ct values results may represent, in addition to false positives, various clinically significant scenarios, including early infection or ongoing viral shedding. Using retesting results to over-ride the initial positive results on the assumption that discrepant results are due to non-specific NAAT reactions, may erroneously call some true positives as false positives. Fundamentally, the relative stochastic performances of the two assays will determine the probability of reproducing a positive result on any assay combinations. Given vaccinated individuals can have lower viral loads when infected [9], high ct value results may be increasingly encountered with increasing vaccination coverage. Laboratories should be aware of how the stochastic performance of their assays affect their retesting protocols to avoid erroneously over-riding an otherwise valid positive result.

Our study has several strengths. Using commercially available control material, we addressed reproducibility issues raised by some researchers [3,10]. Our study design also allowed easy comparison of the evaluated NAATs, and resolved conflicting findings regarding relative sensitivity in previous studies [[4], [5], [6],[11], [12], [13]]. Serial dilutions at close intervals of low viral concentration levels were tested, generating consistent data to demonstrate stochasticity. There are several limitations, however. Only control materials utilizing inactivated virus were tested. Therefore, the matrix effect was not investigated, and the relative sensitivity of the various NAATs may not reflect real life conditions. However, the matrix effect will not invalidate the concept of stochasticity as demonstrated in this study. Another limitation of this study is that only a relatively small number of replicates were tested, and retesting was limited due to availability of assay kits.

In summary, our study illustrated that, even with a more sensitive “confirmatory” assay, true positives need not be reproducible on retesting. The more sensitive “confirmatory” assay only reproduced a ‘true’ result based on its intrinsic stochastic performance. Caution must be exercised if relying solely on retesting to identify false positives, and additional means e.g., testing additional patient samples should be considered.

No external funding was received for this study. Roche, Abbott, and Seegene have provided DH with conference support unrelated to current study. Roche, Abbott, Seegene, Cepheid and Hologic have provided VCS with assay kits targeting other pathogens (not SARS-CoV-2) for research studies. No conflict of interest identified for other authors.

Percentage positive (number positive/number tested). Colour intensity represents detection probability (0–19%, 20–39%, 40–59%, 60–79%, 80–95%, 96–100%). The highest 20% segment (80–100%) is sub-divided to identify sensitivity of > 95% which would indicate it satisfies the criteria for being at or above the LOD. Grey/NA – Not Assessed. Left to right columns: least sensitive to most sensitive assays.

Percentage positive (number positive/number tested). Colour intensity represents detection probability (0–19%, 20–39%, 40–59%, 60–79%, 80–95%, 96–100%). Grey/NA – Not Assessed.

CRediT authorship contribution statement

Hiu Tat Chan: Conceptualization, Methodology, Formal analysis, Resources, Writing – original draft, Writing – review & editing, Supervision. Marco H.T. Keung: Methodology, Investigation, Data curation, Formal analysis, Writing – review & editing. Ivy Nguyen: Investigation, Data curation, Writing – review & editing. Ellen L.O. Ip: Methodology, Resources, Writing – review & editing, Supervision. Su M. Chew: Conceptualization, Methodology, Writing – review & editing. Danielle Siler: Conceptualization, Writing – review & editing. Marion Saville: Methodology, Resources, Writing – review & editing, Funding acquisition. David Hawkes: Conceptualization, Methodology, Formal analysis, Resources, Writing – original draft, Writing – review & editing, Supervision, Funding acquisition.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:

Roche, Abbott, and Seegene had in the past provided one of the authors (DH) with conference support. Roche, Abbott, Seegene, Cepheid and Hologic had in the past provided VCS pathology with assay kits targeting other pathogens (not SARS-CoV-2) for research studies.