Impact of coronavirus disease 2019 (COVID-19) pre-test probability on positive predictive value of high cycle threshold severe acute respiratory coronavirus virus 2 (SARS-CoV-2) real-time reverse transcription polymerase chain reaction (RT-PCR) test results.

OBJECTIVES: Performance characteristics of SARS-CoV-2 nucleic acid detection assays are understudied within contexts of low pre-test probability, including screening asymptomatic persons without epidemiological links to confirmed cases, or asymptomatic surveillance testing. SARS-CoV-2 detection without symptoms may represent presymptomatic or asymptomatic infection, resolved infection with persistent RNA shedding, or a false-positive test. This study assessed the positive predictive value of SARS-CoV-2 real-time reverse transcription polymerase chain reaction (rRT-PCR) assays by retesting positive specimens from 5 pre-test probability groups ranging from high to low with an alternate assay.
METHODS: In total, 122 rRT-PCR positive specimens collected from unique patients between March and July 2020 were retested using a laboratory-developed nested RT-PCR assay targeting the RNA-dependent RNA polymerase (RdRp) gene followed by Sanger sequencing.
RESULTS: Significantly fewer (15.6%) positive results in the lowest pre-test probability group (facilities with institution-wide screening having ≤3 positive asymptomatic cases) were reproduced with the nested RdRp gene RT-PCR assay than in each of the 4 groups with higher pre-test probability (individual group range, 50.0%-85.0%).
CONCLUSIONS: Large-scale SARS-CoV-2 screening testing initiatives among low pre-test probability populations should be evaluated thoroughly prior to implementation given the risk of false-positive results and consequent potential for harm at the individual and population level.

Widespread laboratory testing for SARS-CoV-2, the cause of the COVID-19 pandemic, has led to the observation of positive real-time reverse transcription PCR (rRT-PCR) test results in persons without symptoms. These results may represent active presymptomatic (patients who later develop symptoms) or asymptomatic (patients who never develop symptoms prior to or following testing) infections, resolved infections with persistent viral RNA shedding, or false-positive laboratory tests. The likelihood of a false-positive rRT-PCR result increases as pre-test probability of the condition it is designed to detect decreases. Examples of low pre-test probability scenarios include asymptomatic groups with no known exposure to COVID-19 cases and communities with low prevalence of COVID-19. Furthermore, a positive rRT-PCR result nearing the assay limit of detection (LOD) has a greater likelihood of being falsely positive.

False-positive results can be attributable to preanalytical errors (eg, specimen contamination or aliquoting errors), analytical errors (eg, quality assurance failures, reagent contamination, or nonspecific assay signal), or postanalytical errors (eg, improper result interpretation or transcription). False-positive results can have unintended consequences on individual well-being and the public health response including outbreak declaration and modelling, case reporting, and resource allocation.

The cycle threshold (Ct) value, an indirect measure of viral load, and its application to test interpretation has become an important tool public health tool. Together with available clinical and epidemiological factors, Ct values can help determine appropriate public health follow-up (eg, contact tracing and/or outbreak declaration) for asymptomatic patients. However, multiple studies have shown that Ct values overlap between symptomatic, presymptomatic, and asymptomatic cases and that time from initial infection to testing is the most significant determinant of Ct value. Presymptomatic persons may have comparable viral loads to symptomatic individuals and may be just as likely to infect others; hence, their identification has implications for public health management.

In Ontario, Canada, (population, ˜14.7 million), the country’s first COVID-19 case presented to a hospital on January 23, 2020. The first pandemic wave peaked in April 2020 and was characterized by disproportionate impact on congregate settings including residents in long-term care (LTC), retirement homes, and some workplaces. During the first wave, asymptomatic screening programs and policies were implemented within some LTCs, acute-care facilities (eg, hospitals), and workplace settings. This widespread testing brought into focus the interpretation and implications of positive SARS-CoV-2 rRT-PCR results with high Ct values because many of these settings had both low prevalence and low pre-test probability of COVID-19.

In this study, we evaluated the relative burden of false-positive test outcomes when testing persons in low pre-test probability settings by exploring the likelihood of a reproducible positive test result upon retesting specimens having high rRT-PCR Ct values, stratified by pre-test probability.

Methods

The Public Health Ontario (PHO) laboratory, the Ontario provincial public health and reference laboratory, conducts ˜25% of the province’s SARS-CoV-2 testing. Specimens are submitted from acute care, community, institutional, and occupational settings, and from outbreaks. Specimen data were obtained from the PHO laboratory information system (LIS).

In total, 122 specimens from unique patients aged 10–99 years (median, 53.5) who underwent clinical testing between mid-March and July 2020 were included in the analysis. All specimens included were initially positive by rRT-PCR with Ct value ≥35 using either (1) a laboratory-developed test (LDT) targeting the envelope (E) gene, or (2) a commercial assay targeting the E and open reading frame 1ab (ORF1ab) genes (cobas SARS-CoV-2, Roche Diagnostics, Germany).

The interpretation of results for the LDT E gene rRT-PCR assay was based on prior validation data, which determined an LOD of 192 copies/mL of primary sample (95% CI 16 to 2,392 copies/ml of specimen), corresponding to Ct values between 34.8 and 38.7. Based on these data, LDT Ct values ≤38.0 are reported as detected, Ct values≥40.0 are reported as not detected, and Ct values between 38.1 and 39.9 are reported as indeterminate. Indeterminate results may be due to low viral target quantity approaching the assay LOD, failed viral RNA extraction, or nonspecific reactivity (ie, false signal). When important to clinical or public health management, repeated testing is recommended.

Specimens tested with the cobas SARS-CoV-2 rRT-PCR assay were reported as detected or not detected; the manufacturer does not include an indeterminate range. Although the Ct value for detected specimens is provided by the instrument, the maximum number of cycles of PCR amplification used in the assay is proprietary. Any specimen with a Ct value provided for E and/or ORF1ab target is considered SARS-CoV-2 detected. As determined by PHO Laboratory’s verification, the E gene 95% LOD of the cobas SARS-CoV-2 rRT-PCR was −3.9985 log copies/mL (95% CI, −3.1696 to −4.8265). This finding was similar to that for the ORF1ab gene 95% LOD −4.1175 log copies/ml (95% CI, 3.5875 to −4.6475), and several logs lower than the LDT E gene LOD.

To be included in the study, specimens had to have a high Ct value of ≥35 on either the LDT or cobas rRT-PCR assay E gene target. An E gene Ct value of ≥35 was chosen as a conservative estimate of lack of infectivity based on other studies using different assays reporting that a Ct of >34 indicates an individual is not likely to be infectious at the time of diagnostic testing. Specimens were further classified into 5 groups of differing pre-test probability of COVID-19 based on the presence of symptoms, prior laboratory detection of SARS-CoV-2, and epidemiological links to other positive cases. Information was collected from the PHO laboratory requisition. Table 1 describes the groups, ordered from highest pre-test probability (group 1) to lowest pre-test probability (group 5). Groups 1–4 were tested throughout the study period (March–June 2020), whereas group 5 was tested beyond the peak of the first pandemic wave (May–July 2020). Groups 4 and 5 only included asymptomatic cases, the former from facilities with outbreaks of ≥10 positive cases, and the latter from facilities that underwent institution-wide screening or outbreak investigations with ≤3 positive cases.

Table 1.

Study Patient Categories and Definitions

Group a	Category	Definition	Pre-test probability
1 b	Confirmed cases with second positive specimen of high Ct value	Persons who initially tested positive with a low Ct value (<30) and had a subsequent test with a high Ct value (≥35)
2	Symptomatic patient with high Ct positive specimen	Having a positive test with high Ct value (Ct ≥35) and at least one symptom as noted in the PHO LIS
3 c	Asymptomatic with exposure to probable or confirmed case	Indicated as asymptomatic in the PHO LIS. Tested due to exposure to probable or confirmed case OR residing at same address as another positive case
4	Asymptomatic at a facility with ≥10 positive cases	Indicated as asymptomatic in the PHO LIS and tested as part of an outbreak with at least 10 positive cases
5	Facility with institution-wide screening, with ≤3 positive cases, all asymptomatic	Tested as part of an outbreak or screening investigation having three or fewer asymptomatic positive tests and no symptomatic positive cases in PHO LIS

Note. Ct, cycle threshold, PHO, Public Health Ontario, LIS, Laboratory Information System.

Group 1 represents patients with highest pre-test probability and group 5 represents those with lowest pre-test probability.

20 patients were symptomatic, 2 were asymptomatic at time of first test, and 1 did not have symptom information available at time of their first test.

Group 3 contains specimens from institutional outbreaks (as well as nonoutbreaks); thus, some specimens could also be classified in the group 4 (facility ≥10 positive cases) category.

The study data set was produced by manually reviewing a line list of positive specimens of appropriate Ct values available at PHO Laboratory, Toronto, that met inclusion criteria. Group 1, representing the highest pre-test probability group, consisted of persons who had a confirmed infection with a prior positive result at a Ct value <30. Group 5, which consisted of asymptomatic positive cases in facilities undergoing screening with ≤3 positive cases identified, was defined as the lowest pre-test probability group due to the asymptomatic status and lower likelihood of exposure to COVID-19. No group 5 facilities had an outbreak status at the time of screening, which was confirmed by review of the provincial public health information system for the reporting and surveillance of communicable diseases. Group 5 was thus chosen as the reference group for statistical analyses. Decreasing levels of pre-test probability were attributed from group 2 to group 4.

Specimens included in this study were retested with an LDT endpoint–nested RT-PCR assay targeting the RNA-dependent RNA polymerase (RdRp) gene, followed by Sanger sequencing of amplicons with an expected size of 192 base pairs. This assay was adapted from a previously published Middle East respiratory syndrome coronavirus (MERS-CoV)–nested PCR: an outer primer and newly designed inner primers targeting SARS-CoV-2 were used for both amplification and sequencing (Table 2). The LOD determined during validation was similar to that of the E gene LDT rRT-PCR, at 256 copies/mL of primary specimen (95% CI, 37.92–1733 copies/mL). This parameter was chosen as the confirmatory assay in this study because it was previously developed and validated at the PHO laboratory and was used to confirm Ontario’s early cases of SARS-CoV-2 infection. In addition, it targeted a different gene than the SARS-CoV-2 rRT-PCR assays outlined above, with reproducibility of detection across multiple gene targets more likely to represent a true-positive result.

Table 2.

SARS-COV-2 Laboratory Developed RdRp-Nested PCR Primers in Use at the PHO Laboratory

SARS-CoV-2 RdRp-Nested PCR Primers	Sequence 5’ to 3’	Primer Position Aligned With SARS-CoV-2 b
Nested PCR outer primers	TGCCATTAGTGCAAAGAATAGAGC	15078–15101 bp
	GCATGGCTCTATCACATTTAGG	15319–15298 bp
Nested PCR inner primers	GCACCGTAGCTGGTGTCTCT	15104–15123 bp
	AATCCCAACCCATAAGGTGA	15295–15276 bp

Note. PCR, polymerase chain reaction; RdRp, RNA-dependent RNA polymerase.

Protocol was adapted from Corman et al 2012.

NCBI reference sequence NC_045512.2.

The proportion of specimens detected in each group by the RdRp gene–nested PCR assay relative to group 5 (reference) was calculated using the Fisher exact test with Bonferroni correction to adjust for multiple comparisons. Per group median and range of Ct values were compared using the Wilcoxon rank-sum test. Results were considered significant at a level of 0.05. All analyses were conducted using the SAS Enterprise Guide version 8.3.

The PHO Ethics Review Board determined that this project was exempt from research ethics committee review because it describes analyses that were completed at PHO Laboratory as part of routine clinical respiratory testing during the first wave of the COVID-19 pandemic in Ontario and were therefore considered public health practice, not research.

Results

Table 3 describes the results of the specimens overall and by group. After retesting specimens using the RdRp gene–nested PCR assay with Sanger sequencing, results varied according to pre-test probability. Overall, 66 (54.1%) of 122 specimens had an RdRp gene detected. Highest pre-test probability groups (1 and 2) had the highest proportion of reproducible positive results (18 of 23, 78.3%, and 17 of 20, 85.0%, respectively), and all groups (1–4) had significantly more positives reproducible on the RdRp assay compared to group 5. Across all groups, there was a significant difference (P < .01) in E gene Ct values among specimens that were reproducible on the RdRp gene nested PCR (median Ct, 36.2; range, 35.0–40.6) compared to those that were not reproducible (median Ct, 37.5; range, 35.2–39.8). SARS-CoV-2 was detected in the RdRp gene–nested PCR in 55 (75.3%) of 73 specimens initially tested using the cobas rRT-PCR assay, whereas it was only detected in 11 (22.4%) of 49 specimens tested by the E gene LDT rRT-PCR assay (Table 3).

Table 3.

Initial E Gene PCR and RdRp PCR Results Stratified by Patient Category

Group a	No.	Patients		Detected by RdRp PCR		Not Detected by RdRp PCR
		Age, Median Years (Range)	Ct on Initial E Gene PCR, Median (Range)	No. (%)	Ct on Initial E Gene PCR, Median (Range)	No. (%)	Ct on Initial E Gene PCR, Median (Range)	P Value b
1	23 e	52 (14–99)	36.9 (35.0–38.4)	18 (78.3)	36.7 (35.0–38.3)	5 (21.7)	38.1 (35.9–38.4)	<.0001
2	20 f	68.5 (26–94)	36.6 (35.0–38.3)	17 (85.0)	36.9 (35.03–38.3)	3 (15.0)	36.3 (35.6–37.4)	<.0001
3	15 g	38 (10–93)	36.1 (35.4–38.0)	10 (66.7)	36.0 (35.4–37.2)	5 (33.3)	37.5 (36.0–38.0)	.0078
4	32 h	57.5 (15–97)	37.5 (35.4–40.6)	16 h (50.0)	36.6 (35.4–40.6)	16 (50.0)	37.7 (35.5–38.3)	.035
5	32 i	46 (17–95)	36.9 (35.2–39.8)	5 (15.6)	36.2 (35.6–37.5)	27 (84.3)	37.0 (35.2–39.8)	(ref)[c]
Total	122 j	53.5 (10–99)	36.9 (35.0–40.6)	66 (54.1)	36.2 (35.0–40.6)	56 (45.9)	37.5 (35.2–39.8)	<.0001 d

Note. PCR, polymerase chain reaction; Ct, cycle threshold; E gene PCR, envelope gene real-time reverse-transcription PCR; RdRp PCR, RNA-dependent RNA polymerase gene end-point PCR with Sanger sequencing; LDT, laboratory-developed test.

Refer to Table 1 for group definitions.

P values compare proportion detected in each group to group 5, as the reference group.

Represents reference group to which other groups are compared.

P value compares groups 1 to 4 combined to group 5 as the reference group.

Among the 23 positives, 14 of 17 and 4 of 6 detected by the Roche and LDT assay, respectively, were confirmed by the RdRp assay.

Among the 20 positives, 13 of 14 and 4 of 6 detected by the Roche and LDT assay, respectively, were confirmed by the RdRp assay.

Among the 15 positives, 10 of 14 and 0 of 1 detected by the Roche and LDT assay, respectively, were confirmed by the RdRp assay.

Among the 32 positives, 15 of 23 and 1 of 9 detected by the Roche and LDT assay, respectively, were confirmed by the RdRp assay.

Among the 32 positives, 3 of 5 and 2 of 27 detected by the Roche and LDT assay, respectively, were confirmed by the RdRp assay.

Among all positive specimens, 55 (75.3%) of 73 and 11 (22.4%) of 49 tested by the Roche and LDT assay, respectively, were confirmed by the RdRp assay.

Discussion

This study was conducted to ascertain the impact of different COVID-19 pre-test probabilities on the likelihood that high Ct rRT-PCR results (Ct ≥35) will be reproducibly positive on a laboratory-developed nested PCR and Sanger assay targeting a different SARS-CoV-2 gene. We documented a much lower rate of reproducible high Ct-positive tests among patients in the lowest pre-test probability group of asymptomatic persons included (ie, within an institution with ≤3 positive patients identified through screening). Among this group, only 5 (15.6%) of 32 specimens were also positive by RdRp-nested PCR. This finding contrasts with the higher pre-test probability groups (ie, symptomatic, exposed to a case, and/or in a facility with ≥10 cases), in which 50%–85% of E gene rRT-PCR positive specimens remained positive by RdRp-nested PCR.

Although we documented a significant difference in the E gene Ct values among specimens that were reproducible in the RdRp gene–nested RT-PCR (median Ct, 36.2; range, 35.0–40.6) compared to those that could not be confirmed (median Ct, 37.5; range, 35.2–39.8), the absolute difference is too small to be used to inform clinical or public health decisions on cases and likely depends on the assay(s) used.

Lack of detection with the RdRp gene–nested PCR assay does not necessarily imply a false-positive E gene rRT-PCR result and does not definitively infer false positivity at the individual level. In general, specimens with Ct values well below the assay cutoff for positivity (eg, Ct values <35 with the positivity cutoff set at Ct 38.0) are less likely to be falsely positive. If the result is near the assay positivity cutoff, repeated testing of the same specimen may yield a negative result because assay performance near the LOD is not consistently reproducible. Furthermore, different assays perform differently on the same specimen with virus quantity near their assay LOD. However, when applied at a group level, these results provide an indication of the potential relative contribution of false-positive test results that may occur in different settings characterized by pre-test probability.

In general, the positive predictive value (PPV) of COVID-19 PCR assays is excellent among patients with high pre-test probability, approaching 100%. However, when testing asymptomatic patients with low pre-test probability in low prevalence settings, the PPV is inherently different. For example, if community prevalence of SARS-CoV-2 is 1% with a rRT-PCR test sensitivity of 90% and specificity of 99%, the PPV of a positive test is only 47.6%. If prevalence were to increase to 5% or 10%, then the PPV increases significantly to 82.6% and 90.9%, respectively. Serosurveys in Ontario using residual convenience specimens found a low adjusted monthly seroprevalence of 1.1% among specimens received in June, July and again in August 2020. These results provide further evidence of low community prevalence for SARS-CoV-2 in Ontario during the study period.

Analysis of results from >100,000 SARS-CoV-2 tests conducted at the PHO laboratory for asymptomatic screening programs (including long-term care homes, retirement homes, childcare settings, hospitals, settings with migrant workers, and correctional institutions) during the same period as this study identified a positivity rate of 0.2% (unpublished data). Nearly 70% of positive tests had Ct values ≥35, suggesting that true positivity is likely to be lower, given the potential for false-positive high Ct results in these low-prevalence settings.

The limitations of this study include small sample size and the use of a nonrandomized sampling method, which may limit the generalizability of our findings. All specimens in groups 2–5 were the first specimen submitted to PHO for that individual; however, an earlier specimen could have tested positive elsewhere. This would increase the pre-test probability of that specimen regardless of the group to which the individual’s sample was assigned. For similar reasons, all positive cases from individual institutions might not have been captured in our study if some testing for additional cases was done elsewhere or individuals declined testing. To substantiate that the low-prevalence institutional settings had ≤3 cases, the public health database was checked for outbreak-related cases associated with these settings within a 3-week period. We assumed that the database was correctly updated and that an outbreak had been declared if the number of cases identified by the asymptomatic screening program became >3.

The PHO laboratory obtains clinical information on cases (eg, symptoms, contact with COVID-19 cases) from the specimen requisition submitted to the PHO laboratory, which may not always be accurate and could not be validated. This may have resulted in case misclassification.

The median age varied across patient groups from 38 to 68.6 years (Table 3). The attack rates varied in different age groups during the first wave of the pandemic, which may have affected pre-test probability in the different groups in our study.

Groups 1–4 were tested throughout the study period (March–June 2020), whereas group 5 was tested beyond the peak of the first pandemic wave (May–July 2020). This factor introduces a potential bias to the study because the pre-test probability was inherently lower in group 5 independent of the clinical setting we attempted to evaluate and because testing was conducted in this group when disease prevalence was lower in the community.

Specimens included in this study were stored at −80˚C for weeks to months prior to conducting the RdRp gene–nested RT-PCR assay. RNA degradation during storage and freeze–thaw is possible and was more likely to affect specimens that were close to the LOD, resulting in a negative RdRp RT-PCR in a specimen that was true rRT-PCR positive at the time of initial testing.

Specimen inclusion was based on E gene Ct value at the time of rRT-PCR. Determination of Ct values for LDTs rely on interpretation by the reporting technologist, which can introduce variability into the assignment of the Ct value. Thus, reporter bias may have influenced the analysis of specimens included in this study. In addition, the cobas SARS-CoV-2 rRT-PCR assay has a formal LOD that is several logs lower than that of the E gene LDT and the nested RdRp assay. This may have introduced selection bias because only a subset of specimens were tested with this assay, and misclassification bias may have occurred if the secondary test was less sensitive than the index test. At the PHO laboratory, we did not observe a difference in positivity between the cobas assay and the E gene LDT assay, suggesting that their LODs are closer than formally documented (unpublished data). However, we did observe a higher rate of reproducibility among specimens originally tested by the Roche cobas assay in this study (Table 3). This was likely partly due to 27 (55%) of 49 LDT-positive specimens included in the study arising from group 5 patients, the lowest pre-test probability group.

Despite these limitations, the results presented here are an important step toward quantifying the magnitude of false-positive test results in low-prevalence settings, which will increasingly become the norm in many countries with increased vaccination and widespread testing, including broad testing in low pre-test probability populations. Currently, few studies have attempted to ascertain prevalence through probability-based population-level surveillance studies rather than initiating a study in an area known to have low prevalence. Examples of targeted low-prevalence studies include examination of potential SARS-CoV-2 wastewater detection and serosurveillance studies in low-prevalence areas.

The results of this study have implications for informing future testing approaches, including the utility of broad screening with PCR-based tests in settings with low pre-test probability. For example, in Ontario, this work has been used to inform recent public health approaches, resulting in the discontinuation of unnecessary public health management, such as case isolation, contact tracing, and outbreak declaration, for asymptomatic SARS-CoV-2 rRT-PCR–positive persons with low pre-test probability who are negative on retesting.

In conclusion, SARS-CoV-2 Ct values can be of use when interpreting positive laboratory results derived from patients with low pre-test probability, in particular asymptomatic persons with no epidemiological link to a confirmed COVID-19 case and/or low community COVID-19 prevalence. Healthcare providers, public health professionals, policy makers, and the public will benefit from ongoing education to understand that false-positive tests will occur when testing asymptomatic individuals during periods of low community prevalence of SARS-CoV-2. These false-positive tests and unnecessary public health actions likely outweigh the benefits from the low numbers of true cases detected among these populations. Once high levels of vaccination coverage are achieved and low test positivity are observed among persons with clinical indications for testing (symptomatic persons or asymptomatic contacts of confirmed cases), cessation of screening of asymptomatic persons without epidemiological risk factors for SARS-CoV-2 infection should be considered after conducting a risk assessment at the jurisdictional level.