A Study of Analytical and Clinical Sensitivity of Aptima SARS-CoV-2 Assay (Hologic) and Proposals of Complementary Tests for SARS-CoV-2 Detection in Low Viral Load Specimens.

Early and accurate detection of SARS-CoV-2 is important for diagnosis and transmission control. The use of high-throughput and automated testing allows laboratories to better deliver diagnostic testing given manpower and resource limitations. We validated the clinical and analytical performance of the Hologic Panther Aptima SARS-CoV-2 assay with an emphasis on detection of specimens with low viral loads. The clinical performance was evaluated using 245 clinical specimens, against a comparator PCR-based laboratory developed test (LDT). The analytical performance was determined by replicate testing of contrived samples in a ten-fold dilution series (CT values 32-42, based on LDT). The Aptima assay had 96.7% overall percent agreement, 100% negative percent agreement and 88.1% positive percent agreement. It was able to consistently detect SARS-CoV-2 in contrived samples with CT = 32 by LDT (calculated 2354 copies/mL). The 95% limit of detection of the Aptima assay was estimated to be at LDT CT = 33 (equivalent to 870 copies/mL). The relative light units (RLU) × 1000 for 52 true positive clinical specimens was 962.2 ± 181.5, and that for the 186 true negative specimens was 264.6 ± 14.3. The Aptima assay was a reliable method with a high overall percent agreement against our comparator LDT. We propose that samples reported as negative by the Aptima assay with RLU > 350 be tested by a secondary method, in order to improve detection of samples with very low viral loads.

Introduction

Since the first reported detection of SARS-CoV-2 in December 2019, SARS-CoV-2 has rapidly spread across the world as a global pandemic. At the time of writing in July 2021, the number of global cases has reached 192,284,207, with 4,136,518 reported deaths [1].

Reliable and sensitive laboratory detection methods for SARS-CoV-2 play a critical role in surveillance and controlling viral transmission [2]. The current gold standard and the vast majority of existing test methods in use are nucleic acid amplification tests [2], which are predominantly based on real-time reverse transcription-polymerase chain reaction (qRT-PCR). The Aptima SARS-CoV-2 assay (Hologic, Panther Systems) uses target capture, transcription-mediated amplification (TMA), and dual kinetic assay technologies for the amplification and detection of the target virus. The assay targets two conserved regions of the open reading frame (ORF1ab) region of the SARS-CoV-2 viral genome and is run on the Panther Instrument [3]. It received the United States Food and Drug Administration (US FDA) Emergency Use Authorization on May 14, 2020 [4].

While there has been a recent emergence of comparison of different molecular testing methods for the virus [2, 5–13], the clinical performance of the Aptima assay has not been extensively-characterized. In this study, we aimed to validate its performance against standard qRT-PCR for the detection of SARS-CoV-2 in clinical specimens, with an emphasis on test performance in samples with low viral loads.

Materials and Methods

Specimen Collection

Nasopharyngeal flocked swabs (n = 195) were obtained from persons with clinical symptoms of respiratory tract infection, from screening of asymptomatic individuals in high-risk groups in the community and migrant workers residing in dormitories. Swabs were placed into 3 mL universal transport medium (UTM) and maintained in a chilled cooler box before and during shipping to the testing site (Changi General Hospital, Singapore) on the same day of collection. Samples were processed within 16 h of sample receipt. The primary test method was a laboratory-developed test (LDT) with confirmatory testing of low-viral load positive samples by a second PCR assay. Samples were concurrently tested on the Hologic Panther system. In order to further establish performance for samples with a low viral load, an additional 50 retrospective UTM samples (nasopharyngeal, n = 2; nasopharyngeal and oropharyngeal, n = 13; nose and throat, n = 34; throat, n = 1) archived at − 70 °C were tested simultaneously on the Hologic Panther System [3] and the LDT. All experiments were run without replicates to be reflective of the actual diagnostic practice.

Laboratory-Developed Tests

Nucleic acids were extracted from 150 μL of UTM using the Viral DNA/RNA isolation kit on the EX3600 automated nucleic acid extraction system (LifeRiver™, China), according to manufacturer’s instructions. The primary LDT qRT-PCR assay was designed to detect the N-gene (SARS-CoV-2 nucleocapsid protein) with an internal control (bacteriophage MS2). The fragments were amplified in a total volume of 30 μL containing 1 X reaction mix (SuperScript™ III Platinum One Step qRT-PCR system, Invitrogen, Life Technologies, USA), 0.4 μM of each N-gene primer, 0.16 μM of N-gene probe, 0.125 μM of each MS2 primer, 0.125 μM of MS2 probe (Integrated DNA Technologies, Singapore) (Supplementary Table 1), 1 μL of enzyme mix, and 10 μL of RNA extract. The PCR conditions were cDNA synthesis at 55 °C for 10 min, pre-denaturation at 95 °C for 3 min, followed by 45 cycles of denaturation at 95 °C for 15 s, annealing at 58 °C for 30 s. The reactions were run on Rotor-Gene Q (Qiagen, Germany) with positive and non-template controls, and fluorescence signal acquired on the yellow and crimson channels for target and internal control assays, respectively. A cycle threshold (CT) value < 32 for the N-gene was interpreted as a positive signal for the target, while newly diagnosed positive samples with CT ≥ 32 for the N-gene were confirmed by a second test method: either a second LDT qRT-PCR targeting the ORF1ab gene [14] or Xpert® Xpress SARS-CoV-2 (GeneXpert, USA) which targets both the E and N genes as per manufacturer’s instructions. This is to rule out any false positives defined by the reference method—N-gene LDT (±secondary confirmatory testing).

Briefly, for the secondary LDT qRT-PCR assay, the fragments were amplified in a total volume of 25 μL containing 1 X master mix (QuantiTect Probe RT-PCR Kit, Qiagen, Germany), 0.4 μM of each ORF primer, 0.20 μM of ORF probe (Integrated DNA Technologies, Singapore) (Supplementary Table 1), 0.25 μL of RT mix, and 5μL of RNA extract. The PCR conditions were cDNA synthesis at 50 °C for 30 min, pre-denaturation at 95 °C for 15 min, followed by 45 cycles of denaturation at 94 °C for 15 s, annealing at 55 °C for 45 s. The reactions were run on Rotor-Gene Q (Qiagen, Germany) with positive and non-template controls, and fluorescence signal acquired on the green channel for the target. A CT value ≤ 35 for the ORF1ab gene was interpreted as a positive signal for the target. When performed as a second test method of confirmatory testing, a CT value > 35 was also interpreted as a positive signal for SARS-CoV-2.

Hologic Panther Aptima SAR2-CoV-2 Assay

A 1000 μL aliquot of the UTM specimen was transferred into an Aptima Multitest Tube containing 2.9 mL Specimen Transport Medium and then loaded onto the Panther Instrument and analysed per manufacturer’s instructions. In brief, an aliquot (360 μL) was sampled from each reaction and processed with the Target Capture Reagent (containing an internal control). The purified nucleic acid was then used as the template for TMA reactions. The hybridisation of chemiluminescent probes to the amplicons emits light measured by a luminometer in relative light units (RLUs). Assay results were interpreted based upon the manufacturer’s determined total RLU cut-off and reported as ‘positive’, ‘negative’ or ‘invalid’. Samples with discordant results from LDT underwent repeated testing on Aptima assay but only the results from the first run were used in the analysis.

In order to determine the analytical sensitivity of the Aptima assay, contrived samples of CT = 32, was prepared by diluting a strongly positive UTM specimen containing the SARS-CoV-2 virus. From the contrived samples of CT = 32, further dilutions were performed to obtain contrived samples of assumed CT = 34, 35, 36, 38, 40 and 42. The copy numbers were derived from a separate experiment involving the running of a standard curve of serially-diluted synthesized positive plasmid control containing a portion of the SARS-CoV-2 nucleocapsid gene. Twenty replicates each for samples with CT = 32, 34 and 35, and only one reaction for each of the remaining CT values were run on the Panther Instrument. The RLU values were used in the analysis and compared with the CT values to determine the limit of detection (LOD).

Statistics

Sample size was estimated according to observer agreement study design with Cohen’s kappa (κ) method. We postulated that the discordance between “Aptima assay” and “reference method” was 0.2 with strong agreement (κ = 0.8) and width of confidence interval set at 0.2; a minimum of 222 samples were required [15].

Overall percent agreement (OPA), positive percent agreement (PPA), and negative percent agreement (NPA) were performed with the N-gene LDT (±secondary confirmatory testing) serving as the reference method. The 95% LOD of Aptima assay was estimated using Probit regression. All analyses were performed using SPSS v20.0 (IBM Corp.Armonk, NY).

Results

Clinical Validation

For all the tested 245 samples (prospective, 195; retrospective, 50), OPA was 97.1% (238/245), with 100% NPA (186/186) and 88.1% PPA (52/59). Among the 59 positive samples (prospective, 9; retrospective, 50) detected by the LDT, four (6.8%) were strong positives (CT < 24), 27 (45.8%) were moderate positives (24 ≤ CT < 32), and 28 (47.5%) (CT ≥ 32) were weak positives. For positive samples with CT < 32 as tested by the LDT (n = 31), there was 100% PPA. For positive samples with CT ≥ 32 (n = 28), seven samples were reported as negative on the first run of the Aptima assay, with four being reported as positive upon repeat testing. The Cohen’s kappa value was 0.919 (Z = 14.43, P < 0.001), indicating almost perfect overall agreement [16] (Table 1). Among the seven false negative samples, four were retrospective specimens and three were prospective specimens. Upon repeat testing on Aptima assay, three retrospective specimens and one prospective specimen turned positive.

Table 1

Performance characteristics of the Hologic Panther Aptima SARS-CoV-2 assay

Against the reference standard (Laboratory-developed assay ± secondary confirmatory assay)
CT < 24 (n = 4)
False positives	0	True positives	4
False negatives	0	True negatives	0
PPA (%)	100
NPA (%)	Undefined
24 ≤ CT < 32 (n = 27)
False positives	0	True positives	27
False negatives	0	True negatives	0
PPA (%)	100
NPA (%)	Undefined
CT ≥ 32 (n = 28)
False positives	0	True positives	21
False negatives	7	True negatives	0
PPA (%)	75.0
NPA (%)	Undefined
Negative (n = 186)
False positives	0	True positives	0
False negatives	0	True negatives	186
PPA (%)	Undefined
NPA (%)	100
Overall (n = 245)
False positives	0	True positives	52
False negatives	7	True negatives	186
PPA (%)	88.1
NPA (%)	100
Cohen’s kappa	0.919
OPA	97.1
NPA	100
PPA	88.1

The RLU (× 1000) [mean ± SD (min–max range)] for 52 true positive clinical specimens was 962.2 ± 181.5 (587–1133), and that for the 186 true negative specimens was 264.6 ± 14.3 (239–415). The RLU (× 1000) [mean ± SD (min–max range)] for seven false negative clinical specimens was 462.1 ± 92.1 (261–541). Correspondingly, they were also only weakly positive as defined by their CT values (34.1 ± 0.7). Figure 1 shows the boxplots of the distribution of the RLU values for the positive and negative clinical specimens defined by Aptima assay. Table 2 summarizes the results of the clinical validation and Table 3 shows the CT and RLU values of the nine positive prospective specimens.

Fig. 1

Boxplots of the RLU(× 1000) values for clinical specimens (positive/negative) defined by Aptima assay (- denotes outlier of true negative; *Denotes false-negative results). The cluster of false-negatives (circled) could be differentiated from the true negatives on the boxplot. Diagram was generated using SPSS v20.0

Table 2

RLU values of the Hologic Panther Aptima SARS-CoV-2 assay for clinical specimens

Sample	Call	n	Average: Panther RLU (× 1000)	SD: Panther RLU (× 1000)	Range: Panther RLU (× 1000)
Overall (n = 245)
Positive clinical specimens (n = 59)	Negative	7	462.1	92.1	261–541
	Positive	52	962.2	181.6	587–1133
	All	59	902.9	237.6	261–1133
Negative clinical specimens (n = 186)	Negative	186	264.6	14.3	239–415
	Positive	0	–	–	–
Positive clinical specimens by type (n = 59)
Positive retrospective specimens (n = 50)	Negative	4	489.0	19.6	462–509
	Positive	46	978.2	181.3	587–1133
	All	50	939.1	219.5	462–1133
Positive prospective specimens (n = 9)	Negative	3	426.3	146.7	261–541
	Positive	6	839.7	141.5	682–1067
	All	9	701.9	246.2	261–1067
Positive clinical specimens by CT group (n = 59)
Positive samples CT < 32 (n = 31)	Negative	0	-	-	-
	Positive	31	1047.4	127.3	587–1133
Positive samples CT ≥ 32 (n = 28)	Negative	7	462.1	92.1	261–541
	Positive	21	836.5	178.6	596–1110
	All	28	742.9	229.7	261–1110

Table 3

CT and RLU values of the nine positive prospective specimens

Sample code	CT of LDT	RLU (× 1000) of aptima assay	Result call of aptima assay	Aptima assay (against LDT)
P200059913	33.5	898	POS	True positive
P200059975	33.9	770	POS	True positive
P200059961	31.0	1067	POS	True positive
P200063310	35.0	261	NEG	False negative
P200063312	34.6	541	NEG	False negative
P200063435	33.8	893	POS	True positive
P200063405	34.0	477	NEG	False negative
P200063314	34.1	728	POS	True positive
P200063358	34.6	682	POS	True positive

Analytical Validation

The results of the contrived samples (CT = 32–42) and the profile of their RLU values are shown in Table 4. The copy numbers of contrived samples with CT = 32 were quantitated to be 2354 copies/mL. For each of the 20 replicates tested (CT values = 32, 34 and 35), the detection rate by the Aptima assay was 100%, 85%, and 75%, respectively. The 95% LOD of the Aptima assay was estimated to be 870 copies/mL (LDT-equivalent of CT = 33). All of the single samples with LDT CT ≥ 36 were not detected by the Aptima assay.

Table 4

RLU values of the Hologic Panther Aptima SARS-CoV-2 assay for contrived samples

Sample	Copies/mL	Call	n	Average: panther RLU (× 1000)			SD: panther RLU (× 1000)		Range: panther RLU (× 1000)
Contrived samples CT = 32 (n = 20)	2354	Negative	0	–			–		–
		Positive	20	1071.6			82.7		810–1140
Contrived samples CT = 34 (n = 20)	589	Negative	3	509.3			5.0		504–514
		Positive	17	820.3			162.9		618–1111
		All	20	773.7			187.9		504–1111
Contrived samples CT = 35 (n = 20)	294	Negative	5	469.8			60.9		392–559
		Positive	15	744.1			132.3		564–1011
		All	20	675.6			168.9		392–1011
Contrived samples CT = 36 (n = 1)	147	Negative	1	338			–		–
		Positive	0	–			–		–
Contrived samples CT = 38 (n = 1)	37	Negative	1	379			–		–
		Positive	0	–			–		–
Contrived samples CT = 40 (n = 1)	9	Negative	1	265			–		–
		Positive	0	–			–		–
Contrived samples CT = 42 (n = 1)	2	Negative	1	273			–		–
		Positive	0	–			–		–
Contrived samples in total (n = 64)	–	Negative	12	427.7			97.5		265–559
		Positive	52	895.0			191.0		564–1140
		All	64	807.4			254.9		265–1140

The RLU (× 1000) [mean ± SD (range)] among the replicates of the contrived samples (negative by the Aptima assay) with CT = 34 and 35 was 509.3 ± 5.0 (504–514) and 469.8 ± 60.9 (392–559), respectively. The RLU (× 1000) [mean ± SD (range)] for the 10 readings of contrived samples with CT = 34–38 (negative by the Aptima assay) was 459.4 ± 70.0 (338–559). The RLU (× 1000) values of contrived samples with CT = 40 and CT = 42 were 265 and 273, respectively. The RLU (× 1000) [mean ± SD (range)] for the non-detected and detected contrived samples were 427.7 ± 97.5 (265–559) and 895.0 ± 191.0 (564–1140), respectively.

Discussion

This present study compared the clinical sensitivity and specificity, and analytical sensitivity of the Aptima assay against the LDT as the reference gold standard. The results showed almost perfect agreement (Cohen’s kappa of 0.919), excellent NPA (100%) and acceptable PPA (88.1%) for all the clinical specimens tested. While the assay was highly specific, it might miss detecting samples with low viral load. All the four strong positives (CT < 24) were symptomatic of respiratory tract infections. The vast majority (92.9%) of the 28 weak positives (CT ≥ 32) were accounted by surveillance swabs of confirmed cases whose viral loads might be reducing over time (n = 16) and asymptomatic individuals from the screening of migrant workers in dormitories (n = 10).

Smith and colleagues evaluated the Aptima assay against two qRT-PCR comparators (Hologic Panther Fusion and BioFire Defense) using 150 retrospective and prospective specimens. The Aptima showed 100% NPA and 94.7% PPA, with a non-significant trend towards slightly lower sensitivity compared with the other two assays [2]. Mostafa et al. compared the analytical sensitivity of the Aptima assay against seven other commercial assays, and concluded that the Aptima assay could detect 100% of replicates at nucleocapsid concentration of ~ 1200–1400 copies/mL, with three comparator systems detecting all replicates at one log10 lower concentration. However, in this comparison, only contrived replicates were tested and the number of replicates tested was not equivalent for all of the commercial systems [5]. Schneider et al. reported 95.7% PPA and 100% NPA of the Aptima assay against real-time RT-PCR [17].

In our study, analytical validation shows that the Aptima assay was able to consistently detect SARS-CoV-2 in contrived samples up to CT = 32 (2354 copies/mL) (100%), beyond which the sensitivity dropped to 85% and 75% for samples with CT = 34 (589 copies/mL) and CT = 35 (294 copies/mL), respectively. The range of RLU (× 1000) values of contrived samples with CT = 32 (810–1140) were comparable with the clinical specimens with CT = 32.0 ± 0.1 (633–1024). All single samples run for CT ≥ 36 turned out negative by the Aptima assay. The 95% LOD (870 copies/mL) is within the range reported in published studies (500–1000 copies/mL) [2]. The LOD of the assay per manufacture’s package was 600 NDU/mL for FDA SARS-CoV-2 Reference Panel Testing and 83 copies/mL for the test sample [3].

It is of note that there are numerous real-time PCR assays (commercial and LDT) available for use around the world. The CT values of these different assays are not directly comparable since they do not use identical primers/probes, master mix, and amplification conditions and instruments. The Infectious Diseases Society of America and Association for Molecular Pathology joint statement (March 12, 2021) advised against the use of published SARS-CoV-2 PCR CT values for correlating with disease severity and transmissibility and for clinical decision-making [18]. Therefore, it is imperative to highlight that the CT values presented in this report cannot be extrapolated to the real-time PCR testing platforms in other laboratories or be applied for clinical decision-making purposes.

The RLU values generated by the Aptima assay were analysed in relation to the assay’s ability to detect targets of varying viral load. The RLU (× 1000) values of true positive samples (587–1133) were significantly higher from those of true negative samples (239–415), with no values overlapping between the two datasets. This indicates that the assay could differentiate clearly between positive and negative samples. The RLU (× 1000) values of the seven false-negative clinical specimens ranged from 261 to 541, with only one sample with RLU (× 1000) of <300. For the analytical replicates, the RLU (× 1000) values of the undetected replicates with CT ≤ 35 ranged from 392 to 559. For detected replicates with CT ≤ 35, the lowest RLU (× 1000) was 564. Based on the data from the study, samples which are tested by the Aptima assay with RLU values that fall between 350 and 563 may be tested by an alternative method to potentially improve the detection sensitivity of this assay. In a low-prevalence setting, this approach would better leverage the high-throughput and automation of the Hologic Panther platform, while minimising the risk of missing positive samples with a low viral load. The adoption of this approach would successfully detect six out of the seven false negatives, giving rise to a false negative rate of 0.4% (1/245) among all samples tested. This would not translate to a sizable number of undetected cases in a low-prevalent environment.

The Aptima assay is a high-throughput, automated test which reduces the requirements for trained laboratory personnel in molecular biology to perform the testing. Other advantages include the closed tube system which reduces the risk of extrinsic contamination, as well as the ability to continually load samples during testing, allowing results to be generated in 3.5 h. These advantages need to be balanced against a potential loss of sensitivity for the detection of samples with a very low viral load.

We recognize the limitations associated with the study. Firstly, this is a single-site study based on relatively small sample size (n = 245). However, this sample set comprised specimens of varying viral load from individuals across demographic profiles, which is reflective of the real-world situation. Secondly, the majority of the positive cases (50/59) were pre-selected by our current standard method; hence the clinical sensitivity of the Aptima assay might be an overestimate due to the potential omission of weakly positive samples (called as negative by the LDT ± secondary confirmatory assay). Based on the data from the contrived samples, we expect this bias to be minimal from the ascertained LOD of the Aptima assay. Thirdly, this study reports the performance of the assay up till the time of testing, which may vary depending on the circulating variants, including newly emerging strains of SARS-CoV-2 and their prevalence, which change over time. Lastly, the Hologic Panther system is a floor-standing instrument which may not be a feasible option for small-scale laboratories with space constraints.

Conclusion

To the best of our knowledge, this is the first report offering insights on the interpretation of the RLU values of the Aptima assay and its performance in samples with low viral loads. The Aptima assay was a reliable assay with a high overall percent agreement against our comparator LDT. We propose an additional step for laboratories which use the Aptima assay (samples reported as negative with RLU > 350 be tested by a secondary method), which may combine the benefits of automated testing and also potentially improve the detection of SARS-CoV-2.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 40 kb)