A Comparison of Five SARS-CoV-2 Molecular Assays With Clinical Correlations.

OBJECTIVES: Comparative assessments of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) molecular assays that have been operationalized through the US Food and Drug Administration's Emergency Use Authorization process are warranted to assess real-world performance. Characteristics such as sensitivity, specificity, and false-negative rate are important to inform clinical use.
METHODS: We compared five SARS-CoV-2 assays using nasopharyngeal and nasal swab specimens submitted in transport media; we enriched this cohort for positive specimens, since we were particularly interested in the sensitivity and false-negative rate. Performance of each test was compared with a composite standard.
RESULTS: The sensitivities and false-negative rates of the 239 specimens that met inclusion criteria were, respectively, as follows: Centers for Disease Control and Prevention 2019 nCoV Real-Time RT-PCR Diagnostic Panel, 100% and 0%; TIB MOLBIOL/Roche z 480 Assay, 96.5% and 3.5%; Xpert Xpress SARS-CoV-2 (Cepheid), 97.6% and 2.4%; Simplexa COVID-19 Direct Kit (DiaSorin), 88.1% and 11.9%; and ID Now COVID-19 (Abbott), 83.3% and 16.7%.
CONCLUSIONS: The assays that included a nucleic acid extraction followed by reverse transcription polymerase chain reaction were more sensitive than assays that lacked a full extraction. Most false negatives were seen in patients with low viral loads, as extrapolated from crossing threshold values. © American Society for Clinical Pathology, 2020. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.

Different molecular assays for severe acute respiratory syndrome coronavirus 2 (n class="Species">SARS-CoV-2), all of which have US Food and Drug Administration Emergency Use Authorization (FDA EUA) status, may have different performance characteristics.

The comparisons of multiple assays for SARS-CoV-2 agaiene">nst oene">ne aene">nother aene">nd a resultiene">ng composite staene">ndard generate aene">nalytic performaene">nce characteristics (ie, sensitivity aene">nd specificity), which are not required for FDA EUA clearaene">nce.

Patients with low viral loads iene">n the respiratory specimeene">n are more likely to have false-negative results with assays that do not have a nucleic acid extractioene">n step prior to amplificatioene">n.

The emergence of the severe acute respiratory syndrome coronavirus 2 (n class="Species">SARS-CoV-2) virus and the subsequent pandemic has resulted in the need to rapidly deploy molecular diagnostic assays for the detection of infected individuals. Molecular diagnostic assays targeting the SARS-CoV-2 virus have been provided by commercial manufacturers and/or designed in individual laboratories and implemented through the authority of the US Food and Drug Administration’s (FDA) Emergency Use Authorization (EUA) Act (https://www.fda.gov/emergency-preparedness-and-response/mcm-legal-regulatory-and-policy-framework/emergency-use-authorization). Assays that achieve EUA clearance have been assessed through a variety of experiments, including but not limited to an assessment of the lower limit of detection (LoD) and in silico assessment of primer and probe sequences for potential cross-reactivity.[1] Although these and the other FDA EUA requirements are excellent initial means of assessing an assay, there is not a defined LoD that is necessary for an assay to achieve EUA status. In addition, analytical and clinical sensitivity and specificity determinations are not required, which is understandable when responding to an emergency.

Laboratories that have had a research interest in severe acute respiratory syndrome coronaviruses have had the opportuene">nity to more thoroughly study test performance characteristics through various primer/probe combiene">natioene">ns and coene">ncentratioene">ns and reactioene">n coene">nditioene">ns. Most cliene">nical laboratories, which by necessity have been thrust iene">nto n class="Species">SARS-CoV-2 testing, have not had this opportunity, however. Therefore, comparative studies of FDA EUA–cleared molecular tests for SARS-CoV-2 are warranted to more fully understand test performance characteristics.

We, therefore, compared five SARS-CoV-2 molecular assays to oene">ne another to determiene">ne the sensitivity and specificity of these assays. This comparisoene">n was enriched for positive specimens, as we were particularly iene">nterested iene">n determiene">niene">ng the false-negative rates for these assays, siene">nce mischaracteriziene">ng an n class="Disease">infected patient as coronavirus disease 2019 (COVID-19) negative could have considerable infection prevention implications in the hospital. This comparison also afforded our group the opportunity to compare test characteristics, such as crossing thresholds (Cts), with alternate test results, as well as with the clinical and demographic findings of infected patients.

Materials and Methods

The Cleveland Clinic began testing for SARS CoV-2 usiene">ng the origiene">nal, three-target Centers for Disease Coene">ntrol aene">nd Preventioene">n (n class="Gene">CDC) assay (ie, CDC 2019 nCoV Real-Time RT-PCR Diagnostic Panel) on March 12, 2020. Four additional assays (see below) were subsequently assessed in an effort to expand capacity and address requests for more rapid turnaround times. Nasopharyngeal (NP) or nasal swabs were collected by a trained medical practitioner, and most were submitted in universal transport medium (UTM) (Copan Diagnostics) or viral transport media (VTM), respectively; two specimens were submitted in 0.9% normal saline, which was also a validated transport medium. VTM was made by the Cleveland Clinic according to the CDC procedure (SOP #DSR-052-03; https://www.cdc.gov/coronavirus/2019-ncov/downloads/Viral-Transport-Medium.pdf). All specimens were initially tested by the CDC 2019 nCoV Real-Time RT-PCR Diagnostic Panel as part of the standard operating procedures for the laboratory. Positive and negative specimens, in an approximate 2:1 ratio, were assessed for the presence of the SARS-CoV-2 virus by the four additional assays described below. Specimens were enrolled as they were received and were not selected based on any clinical characteristics.

A specimen was considered to contain the SARS-CoV-2 virus if the results of two or more of the five tests studied were positive accordiene">ng to the staene">ndard operatiene">ng procedure of the laboratory or the maene">nufacturer’s iene">nstructioene">ns. Otherwise stated, a positive test from aene">ny assay needed to be corroborated by a positive result from aene">ny other assay for the specimen to be characterized as coene">ntaiene">niene">ng the n class="Species">SARS-CoV-2 virus. The specimen was considered to not contain the SARS-CoV-2 virus if the results from all the tests were negative or if only a single test was positive (ie, a positive test that was not corroborated by any of the other four tests). Any single positive test results were characterized as false positives.

This study was approved by the Cleveland Clinic Institutional Review Board. Patient coene">nfidentiality was protected by storiene">ng data iene">n a password-protected file oene">n aene">n iene">nterene">nal electroene">nic shared drive accessible oene">nly to study team members. This was a siene">ngle-center study.

Nucleic Acid Amplification Assays

CDC SARS-CoV-2 RT-PCR Assay

We used the original, three-target SARS-CoV-2 reverse traene">nscriptioene">n polymerase chaiene">n reactioene">n (RT-PCR) test that was developed at the n class="Gene">CDC (ie, CDC 2019 nCoV Real-Time RT-PCR Diagnostic Panel).[2] This assay targets three separate loci in the SARS-CoV-2 nucleocapsid (N) gene. Commercially available plasmids that contained the N gene were used to determine the limit of detection. In brief, serial dilutions were made that contained known concentrations of the target plasmids. Multiple replicates of these were tested until the concentration wherein 95% (ie, 19/20) of the replicates were detected. This defined the limit of detection, which was found to be 20 copies/µL for upper respiratory specimens and 2 copies/µL for lower respiratory specimens. Internal validation studies, which were more extensive than those required for an FDA EUA submission, were performed prior to the introduction of this test.

For each specimen, 200 µL of clinical specimen in transport media was rendered noninfectious within a biological safety cabinet through the addition of 200 µL of Bacterial Lysis Buffer (Roche Diagnostics). A nucleic acid extract was obtained from 200 µL of the inactivated specimen using the MagNA Pure system (Roche). Then, 5 µL of eluate was added to 15 µL of PCR mastermix for each PCR well. The n class="Gene">CDC 2019 nCoV Real-Time RT-PCR Diagnostic Panel used consisted of four separate RT-PCR assays, three of which targeted different regions of the virus nucleocapsid (N) gene. The fourth RT-PCR was an amplification control that targeted a portion of the human RNase P gene. Testing was performed on an ABI 7500 or ABI 7500 Fast Dx (Thermo Fisher) using fresh extract that was never frozen. Amplification of all three RT-PCR N gene targets (ie, N1, N2, and N3) was necessary to characterize a specimen as positive. No amplification of all N gene targets in conjunction with amplification of the human control gene was necessary to characterize a specimen as negative. Specimens with amplification of only one or two of the N gene targets were characterized as indeterminate for this study and for simplicity were excluded from further analysis.

TIB MOLBIOL/Roche z 480 Assay

The RT-PCR test used in this assay was developed by Roche Diagnostics and TIB MOLBIOL. Internal validation studies were performed before introduction of the test. Commercially available controls containing the envelope (E) gene and the RNA-dependent Rn class="Gene">NA polymerase (RdRP) gene (Exact Diagnostics; SeraCare) were used to determine the limit of detection, which was found to be 20 copies/µL for upper respiratory specimens. We also performed the other validation studies necessary for the submission of an EAU, since this was a laboratory-developed test.

For each specimen, 200 µL of clinical specimen in transport media was rendered noninfectious within a biological safety cabinet through the addition of 200 µL of Bacterial Lysis Buffer (Roche). A nucleic acid extraction was performed using 200 µL of the inactivated specimen using the MagNA Pure system (Roche). This was the same extract tested origiene">nally by the n class="Gene">CDC assay but had undergone a single freeze-thaw cycle. The nucleic acid extracts were frozen once prior to testing with the TIB MOLBIOL/Roche z 480 Assay. Then, 10 µL of thawed and homogenized eluate was added to 10 µL of PCR mastermix for each PCR reaction. This test consisted of three separate RT-PCR assays, one that targeted the E gene, one that targeted the RdRP gene, and a third amplification control that targeted a portion of the human RNAse P gene. Testing was performed on both the Cobas Z 480 and LightCycler 480 platforms (Roche Diagnostics). Amplification of both the E gene and RdRP gene targets was necessary to characterize a specimen as positive with this assay. Both targets had to be negative with amplification of the human control to characterize a specimen as negative. Specimens with amplification of only one of the two targets were characterized as indeterminate for this study and for simplicity were excluded from further analysis.

Xpert Xpress SARS-CoV-2 (Cepheid)

The Xpert Xpress SARS-CoV-2 (Cepheid) assay was performed accordiene">ng to the manufacturer’s guideliene">nes. Iene">n brief, the fresh (ie, never frozen) specimen was mixed by vortexiene">ng for several secoene">nds, and 300 µL of transport media was transferred to the cartridge sample chamber. The test coene">nsisted of a multiplex RT-PCR assay that targeted the n class="Species">SARS-CoV-2 E and N2 genes, as well as a sample processing control. Amplification of the N2 gene was necessary to characterize a sample as positive. Specimens with amplification of the E gene without amplification of the N2 gene are characterized by the manufacturer as a presumptive positive, but these were categorized as positive for the purposes of this study. Both N2 and E genes had to render negative results with amplification of the specimen processing control for a specimen to be characterized as negative.

Simplexa COVID-19 Direct Kit (DiaSorin)

The Simplexa COVID-19 Direct n class="Gene">Kit (DiaSorin) was performed according to the manufacturer’s guidelines. In brief, 50 μL of fresh (ie, never frozen) clinical specimen in transport media was added to the Direct Amplification Disc sample well after 50 μL of the reaction mix was added to the reaction well. Testing was performed on the LIAISON MDX (DiaSorin). This assay used a multiplex RT-PCR that targeted SARS-CoV-2 ORF1ab and S genes along with an internal amplification control. Amplification of at least one target gene (ie, ORF1ab or S) was necessary to characterize a specimen as positive. Both target genes had to render negative results with amplification of the internal RNA control to characterize a specimen as negative.

ID Now COVID-19 (Abbott)

The ID Now n class="Disease">COVID-19 (Abbott) assay targets a portion of the RdRp gene within the SARS-CoV-2 genome (https://www.fda.gov/media/136525/download). This assay was performed according to the manufacturer’s guidelines, which at the time of this study included the testing of transport media (see ID Now COVID-19 [Abbott] original FDA EUA submission). Positive and negative results were provided by the instrument and recorded as provided. In brief, fresh (ie, never frozen) specimens were allowed to reach room temperature before testing. The test base and receiver cartridges were placed in the ID Now devices, and the receiver cartridge was permitted to warm up. When prompted by the instrument, the foil seal was removed, and 200 µL of well-mixed clinical specimen in transport media was dispensed into the receiver cartridge using the disposable pipettes provided in the kit. After 10 seconds of vigorous mixing, the pipette was removed and the transfer cartridge was used to introduce sample into the test base from the receiver cartridge. The lid was closed and the isothermal amplification was initiated. Results were displayed on the ID Now screen, and the three cartridges were disposed of in accordance with instructions, which minimized the possible release of amplified products. The studies were performed in a biological safety cabinet by qualified clinical laboratory technologists. All three instruments used in this part of the study passed daily positive and negative quality control checks, and the reported results passed internal quality checks.

Clinical Parameters

Medical records were available for 208 of the 239 patients enrolled iene">n the study. Thirty-oene">ne n class="Species">patients were non–Cleveland Clinic patients whose records were not available for review. Three residents in clinical pathology reviewed electronic medical records for study participants, and a subset of recorded results was checked by one author (S.M.H.). Patient identifiers were removed at the conclusion of the study. Variables assessed included the patient age, sex, patient status as a caregiver (ie, health care provider [HCP]), date of specimen collection, date of onset of symptoms, and whether the patient was evaluated in an inpatient, outpatient (including telemedicine visits), emergency department (ED), or intensive care unit (ICU) location. The presence or absence of the following clinical parameters at the time of the visit was recorded: fever, cough, nausea or vomiting, diarrhea, and dyspnea. For patients who had a chest roentgenogram or chest computed tomography scan, the presence or absence of pneumonia was recorded as interpreted by a radiologist or attending physician.

Statistical Analysis

Analyses were done using R version 4.0.0.[3] Test performance with each assay was evaluated in the entire data set, with indeterminates for each assay excluded. Viral loads (as a multiple of the minimum detectable viral load) were calculated from threshold cycle data based on a method previously described.[4] For assays with more than a 10% false-negative rate, associations with false-negative results were examined in multivariable logistic regression models, using the subset of patients with the disease. Days siene">nce oene">nset of symptoms was missiene">ng for 14 (7%) n class="Species">patients. These were found to be missing completely at random on evaluation of the missing data using the R package VIM.[5] Missing values were imputed using a method of multivariate imputation by chained equations using the R package mice.[6] Initial models included all variables with univariable associations at a level of significance of .2. Variable selection was then done with stepwise backward elimination until only variables significant at a level of .05 remained. Logarithm of the viral load and transport medium was forced into the final model as there was good biological plausibility that they would influence false-negative rates. Odds ratios (ORs) and 95% confidence intervals (CIs) were calculated using the final model. Graphics were created using the ggplot2 package.[7]

Results

A total of 239 specimens were tested by all methods, with the exception of one specimen that could not be performed on the Xpert Xpress SARS-CoV-2 (Cepheid) because of iene">nadequate specimen volume. Composite aene">nalysis demoene">nstrated that 168 specimens coene">ntaiene">ned n class="Species">SARS-CoV-2, whereas 71 specimens did not. The performance characteristics of the five tests are summarized in .

CDC, Ceene">nters for Disease Coene">ntrol and Preveene">ntioene">n; Fn class="Gene">N, false negative; FP, false positive; RT-PCR, reverse transcription polymerase chain reaction; TN, true negative; TP, true positive.

aIn total, 239 tests were performed with 168 containing SARS-CoV-2 aene">nd 71 without the virus present by the composite positive staene">ndard described. The n class="Gene">CDC 2019 nCoV Real-Time RT-PCR Diagnostic Panel rendered 234 results and five indeterminates; TIB/Roche, 208 results and 31 indeterminates; Simplexa COVID-19 Direct Kit (DiaSorin), 239 results; Cepheid, 238 results and one unavailable for testing; and Abbott, 239 results.

Test Performance

A total of 234 specimen results were available for assessment of the CDC n class="Species">2019 nCoV Real-Time RT-PCR Diagnostic Panel; there were five indeterminate results. There were no false negatives and three false positives when this assay was assessed against a composite standard. In total, 208 specimen results were available for assessment of the TIB MOLBIOL/Roche z 480 assay because of 31 indeterminate results. There were five false-negative results and one false-positive result when this assay was assessed against a composite standard. A total of 238 specimen results were available for assessment of the Xpert Xpress SARS-CoV-2 (Cepheid) assay since one of the specimens in this collection could not be tested due to inadequate volume. There were four false negatives and five false positives when this assay was assessed against a composite standard. In total, 239 specimen results were available for assessment of the Simplexa COVID-19 Direct Kit (DiaSorin) assay. There were 20 false negatives and one false positive when this assay was assessed against a composite standard. Finally, 239 specimen results were available for assessment of the ID Now COVID-19 (Abbott) assay. There were 28 false negatives and two false positives when this assay was assessed against a composite standard. The sensitivities, specificities, and false-negative rates for the five assays are shown (Table 1).

Table 1

Performance Characteristics of Five Molecular SARS-CoV-2 Assays

		Positive Specimens, No.		Negative Specimens, No.
Test	No. of Tests (n = 239)a	TP	FN	TN	FP	Sensitivity, %	Specificity, %	FN Rate, %
CDC 2019 nCoV Real-Time RT-PCR Diagnostic Panel	234a	165	0	66	3	100	95.7	0
TIB/Rocheb	208a	137	5	65	1	96.5	98.5	3.5
Xpert Xpress SARS-CoV-2 (Cepheid)	238a	163	4	66	5	97.6	93.0	2.4
Simplexa COVID-19 Direct Kit (DiaSorin)	239a	148	20	70	1	88.1	98.6	11.9
ID Now COVID-19 (Abbott)	239a	140	28	69	2	83.3	97.2	16.7

CDC, Centers for Disease Control and Prevention; FN, false negative; FP, false positive; RT-PCR, reverse transcription polymerase chain reaction; TN, true negative; TP, true positive.

aIn total, 239 tests were performed with 168 containing SARS-CoV-2 and 71 without the virus present by the composite positive standard described. The CDC 2019 nCoV Real-Time RT-PCR Diagnostic Panel rendered 234 results and five indeterminates; TIB/Roche, 208 results and 31 indeterminates; Simplexa COVID-19 Direct Kit (DiaSorin), 239 results; Cepheid, 238 results and one unavailable for testing; and Abbott, 239 results.

bTIB MOLBIOL/Roche z 480 Assay.

Scatterplots of mean Ct values from the three N gene RT-PCRs from the n class="Gene">CDC 2019 nCoV Real-Time RT-PCR Diagnostic Panel vs days since onset of symptoms with results as determined for each assay indicated with colored circles are shown in . For each assay, there was a trend of lower Ct values (higher viral load) closer to the time of onset of symptoms. As days since onset increased, Ct value tended to increase (ie, viral load decreases). False-negative results for the Simplexa COVID-19 Direct Kit (DiaSorin) and ID Now COVID-19 (Abbott) assays tended to occur more frequently as time from onset of symptoms increased and Ct values increased (Figure 1).

Figure 1

Scatterplots of crossing threshold (Ct), as a surrogate for viral load, vs the days since onset of symptoms for the five molecular severe acute respiratory syndrome coronavirus 2 assays studied. A, Centers for Disease Control and Prevention assay. B, Cepheid assay. C, Roche assay. D, Abbott assay. E, DiaSorin assay. Performance characteristics (eg, TP, TN) are listed and color coded. The y-axis is on a reversed scale. FN, false negative; FP, false positive; Neg, negative; Pos, positive; TN, true negative; TP, true positive.

Scatterplots of crossing threshold (Ct), as a surrogate for viral load, vs the days since onset of symptoms for the five molecular severe acute respiratory syndrome coronavirus 2 assays studied. A, Centers for Disease Coene">ntrol aene">nd Preventioene">n assay. B, Cepheid assay. C, Roche assay. D, Abbott assay. E, n class="Chemical">DiaSorin assay. Performance characteristics (eg, TP, TN) are listed and color coded. The y-axis is on a reversed scale. FN, false negative; FP, false positive; Neg, negative; Pos, positive; TN, true negative; TP, true positive.

Clinical Characteristics

Clinical characteristics of the 208 patients for which cliene">nical iene">nformatioene">n was available are showene">n iene">n . The meaene">n age was 49.3 years, aene">nd 45.2% were male. Most n class="Species">patients were seen in the outpatient setting (63.9%), followed by those seen in the ED (25.0%). Fewer specimens were collected from patients in the ICU (2.4%) or another inpatient unit (8.7%). More specimens were collected with a nasal swab in VTM (60.9%) compared with an NP swab in UTM (39.1%) due to a national shortage of UTM. Ten (4.8%) patients were asymptomatic. The mean days from onset of symptoms to specimen collection was 8.6, although the standard deviation was large. HCPs represented 38.9% of patients; 96% of specimens collected from HCPs were nasal swab/VTM. Most patients were not severely ill. Only 13.0% were characterized as having pneumonia; 19.2% were admitted to the hospital in a non-ICU setting, and 4.8% were admitted to the ICU. The proportion of patients with other symptoms is given (Table 2).

Table 2

Clinical Characteristics of 208 Patients With Coronavirus Disease 2019a

Characteristic	Overall
Age, mean (SD), y	49.28 (16.86)b
Sex, male	94 (45.2)
Specimen, nasal/VTM	126 (60.9)
Location
ED	52 (25.0)
ICU	5 (2.4)
Inpatient	18 (8.7)
Outpatient	133 (63.9)
Days to collection from onset of symptoms, mean (SD)	8.64 (7.87)
HCP, yes	81 (38.9)
Fever, yes	91 (43.8)
Cough, yes	133 (63.9)
Nausea or vomiting, yes	18 (8.7)
Diarrhea, yes	41 (19.7)
Dyspnea, yes	70 (33.7)
CXR, not done	143 (68.8)
Pneumonia, yes	27 (13.0)
Admitted, yes	40 (19.2)
ICU, yes	10 (4.8)
Asymptomatic, yes	10 (4.8)

CXR, chest roentgenogram; ED, emergency department; HCP, health care provider; ICU, intensive care unit; VTM, viral transport media.

aValues are presented as number (%) unless otherwise indicated.

bThere were no children included in this study. All patients were 18 years or older.

Clinical Characteristics of 208 Patients With n class="Disease">Coronavirus Disease 2019a

bThere were no children iene">ncluded iene">n this study. All n class="Species">patients were 18 years or older.

Associations With False-Negative Test Results

The Simplexa COVID-19 Direct n class="Gene">Kit (DiaSorin) and ID Now COVID-19 (Abbott) assays had false-negative rates exceeding 10%. Because the testing of HCPs occurred at a particular location, the swab type and transport medium used at that location were very strongly correlated with each other; of these, only swab/transport medium was used in multivariable analysis to avoid multicollinearity. On multivariable analysis, log10 viral load (OR, 0.46; 95% CI, 0.29-0.67; P < .001) was the only variable significantly associated with false-negative results for the Simplexa COVID-19 Direct Kit (DiaSorin) assay . For the ID Now COVID-19 (Abbott) assay, both log10 viral load (OR, 0.32; 95% CI, 0.18-0.50; P < .001) and swab/transport medium (OR, 4.95; 95% CI, 1.30-25.30; P = .018) remained significantly associated with false-negative results in a multivariable analysis .

Associations With False-Negative Results for the Simplexa n class="Disease">Coronavirus Disease 2019 Direct Kit (DiaSorin) Assay

CI, confidence interval; CXR, chest roentgenogram; ICU, intensive care unit; NP, nasopharyngeal; OR, odds ratio; UTM, uene">niversal transport medium; VTM, viral transport media; —, variable elimiene">nated through stepwise analysis process.

Associations with False Negative Results for the ID n class="Gene">Now COVID-19 (Abbott) Assay

CI, confidence interval; CXR, chest roentgenogram; ICU, intensive care unit; NP, nasopharyngeal; OR, odds ratio; UTM, uene">niversal transport medium; VTM, viral transport media; —, variable elimiene">nated through stepwise analysis process.

Discussion

All five SARS-CoV-2 nucleic acid amplificatioene">n tests were performed oene">n each of the 239 cliene">nical specimens received iene">n transport media iene">n this study (ie, split-sample study), with the exceptioene">n of oene">ne specimen that lacked sufficient residual volume to be performed oene">n the Xpert Xpress n class="Species">SARS-CoV-2 (Cepheid) assay. In total, 168 of the specimens were confirmed to contain the SARS-CoV-2 virus by the demonstration of two or more positive tests (ie, at least two unique assays were positive on the same specimen), whereas 70 specimens were found to not contain the virus by the method described. Although input volume into the nucleic acid amplification reactions for each assay varied, which may affect sensitivity, these assays were all performed according to either our laboratory standard operating procedures or the manufacturer’s guidelines that were current at the time of this study. It is noted that the FDA EUA for the ID Now COVID-19 (Abbott) has subsequently been changed to exclude specimen in transport media as an acceptable substrate for testing with the claim that the dilutional effect of placing the specimen into transport media reduces the sensitivity of the assay.[8] Although this may be true, such a dilutional effect would be the same for all assays included in this comparison. We, therefore, conclude that this is a sound assessment of the analytical sensitivity and specificity of these assays.

The comprehensive design and assessment of the CDC n class="Species">2019 nCoV Real-Time RT-PCR Diagnostic Panel is available for further review, which, like our study, describes the excellent sensitivity and specificity of this assay.[2] Similarly, a thorough evaluation of the components of the TIB MOLBIOL/Roche z 480 assay is available for those interested in further reading.[9] It has been our experience that the elevated indeterminate rate experienced for the TIB MOLBIOL/Roche z 480 assay in this study does not reflect the indeterminate rate of this assay in routine practice. The only difference in this study and routine use was a single freeze-thaw cycle, which we hypothesize may have contributed to this elevated rate through target degradation.

The excellent performance of the Xpert Xpress SARS-CoV-2 (Cepheid) assay described iene">n this study is like that described iene">n multicenter trials by others.[10,11] A comparisoene">n between the Xpert Xpress n class="Species">SARS-CoV-2 (Cepheid) and the standard-of-care RT-PCR for SARS-CoV-2 demonstrated a 99.5% positive percent agreement and a 95.8% negative percent agreement, with the majority of discrepant results resolved by a third RT-PCR in favor of the Xpert Xpress SARS-CoV-2 (Cepheid).[10]

The assays that lacked a nucleic acid extraction step produced more false-negative reactions than assays that included this step in our study. Although there are a variety of reasons for different amplification efficiencies between assays, such as primer design and nearest neighbor influences, among others, the presence of amplification inhibitors is a common cause of decreased efficiency or even inhibition in assays that begin with a simple lysis step rather than purified nucleic acid extract.[12] Wilson et al[12] demoene">nstrated this by examiene">niene">ng the same coene">ncentratioene">n of target aene">nd the same amplificatioene">n assay after processiene">ng by several different methods. An eloquent study by Fuene">ng et al[13] determiene">ned the aene">nalytical limits of detectioene">n for seven nucleic acid amplificatioene">n n class="Species">SARS-CoV-2 assays using material quantitatively characterized using digital droplet PCR. In this study, the Roche cobas, Abbott m2000, and Hologic Panther Fusion assays had lower limits of detection than both the Simplexa COVID-19 Direct Kit (DiaSorin) assay and the ID Now COVID-19 (Abbott) assay.[13] Importantly, the variable performance of the CDC 2019 nCoV Real-Time RT-PCR Diagnostic Panel was related to the type of extraction used.[13]

The Simplexa COVID-19 Direct n class="Gene">Kit (DiaSorin) assay has also been assessed in a number of studies.[14-16] Two of these demonstrated a greater sensitivity or positive percent agreement compared with that described here.[15,16] This is likely explained by the significant number of patients entering convalescence in our cohort, which was similar to that described by Lieberman et al.[14]

The limited comparative sensitivity of the ID Now n class="Disease">COVID-19 (Abbott) has been described by others.[17,18] A comparison of the ID Now COVID-19 (Abbott) with the Abbott RealTime SARS-CoV-2 assay (Abbott) on the m2000 showed an overall positive percent agreement of only 75%.[18] Similarly, Basu et al[17] reported that the Abbott ID Now COVID-19 missed one-third of the specimens that tested positive by the Xpert Xpress SARS-CoV-2 assay (Cepheid) when using nasopharyngeal swabs in viral transport media, similar to our study, and a positive percent agreement of 45% when dry nasal swabs were tested directly.

To understand how these assays might perform in our diverse patient populatioene">n, the specimens tested iene">n the study were not selected from any particular populatioene">n or based oene">n any set of cliene">nical characteristics. Rather, they were a true representatioene">n of the specimens received iene">n the laboratory at the time of the study. n class="Species">Outpatients represented the majority of those tested, and as such, the presence of pneumonia, hospital admission, and ICU stay were relatively low in frequency. Different results might be obtained in a population of individuals with more severe illness.

Our study included a high proportion (38.9%) of HCPs, some of whom were tested more than 2 weeks after onset of illness when viral loads would presumably be declining. Specimens were collected from HCPs in the outpatient settiene">ng with nasal or nasopharyngeal swabs and submitted iene">n VTM. The swab/transport medium was a variable chosen for iene">nclusioene">n iene">n the multivariate analysis, as we were iene">nterested iene">n ensuriene">ng that there were no effects of locally produced transport medium not detected iene">n iene">nitial validatioene">n studies. However, we noted that the graphs of Ct vs days from oene">nset to specimen collectioene">n demoene">nstrate the correlatioene">n of decreasiene">ng viral load as number of days from first symptoms iene">ncreases (Figure 1). If some HCPs were tested as symptoms were waniene">ng, it stands to reasoene">n that lower viral loads were detected and false negatives would be likely.

Use of nasal swab/VTM predominated as the NP swab/UTM collectioene">n n class="Gene">kit was in short supply, and the latter was preferentially used for inpatients and those seen in the ED. In our study, the use of nasal swab/VTM was independently associated with false-negative results in the ID Now COVID-19 (Abbott) assay. As the swab type and transport medium were always used together, it is difficult to distinguish independent effects of each component. As discussed above, the dilutional effect of transport medium may have affected sensitivity, but other considerations that may affect compatibility must also be considered. For the Simplexa COVID-19 Direct Kit (DiaSorin) assay, although transport medium was associated with false-negative results on univariable analysis, the association was not significant on multivariable analysis when adjusted for viral load.

Conclusion

The findings from the clinical correlation are consistent with the presence of high viral loads (lower Ct values) at the onset of symptoms. This pattern of high viral shedding at symptom onset has been demonstrated by others and is a significant factor in transmission of SARS-CoV-2.[19,20] Our data suggest that the n class="Gene">CDC 2019 nCoV Real-Time RT-PCR Diagnostic Panel, TIB MOLBIOL/Roche z 480 Assay, and Xpert Xpress SARS-CoV-2 (Cepheid) assays perform well at any stage of illness. However, the Simplexa COVID-19 Direct Kit (DiaSorin) and ID Now COVID-19 (Abbott) assays had false-negative rates exceeding 10%, with false-negative results associated with low viral loads, suggesting that they are likely to perform less well when viral loads are lower, which is generally later in the course of illness. This finding concerning the ID Now COVID-19 (Abbott) has been confirmed by others.[17,18]

Table 3

Associations With False-Negative Results for the Simplexa Coronavirus Disease 2019 Direct Kit (DiaSorin) Assay

			OR (95% CI)
Clinical Variable	True Positivesa	False Negativesa	Univariable	Multivariable
Age, mean (SD), y	51.5 (17.6)	41.1 (14.1)	0.96 (0.92-0.99, P = .023)	—
Sex
Male	60 (47.2)	5 (27.8)
Female	67 (52.8)	13 (72.2)	2.33 (0.82-7.61, P = .128)	—
Specimen/medium
NP/UTM	52 (40.9)	3 (16.7)
Nasal/VTM	75 (59.1)	15 (83.3)	3.47 (1.08-15.51, P = .059)	2.30 (0.53-11.04, P = .215)
Fever
No	60 (47.2)	10 (55.6)
Yes	67 (52.8)	8 (44.4)	0.72 (0.26-1.93, P = .510)	—
Cough
No	35 (27.6)	8 (44.4)
Yes	92 (72.4)	10 (55.6)	0.48 (0.17-1.34, P = .148)	—
Nausea or vomiting
No	118 (92.9)	17 (94.4)
Yes	9 (7.1)	1 (5.6)	0.77 (0.04-4.50, P = .811)	—
Diarrhea
No	100 (78.7)	16 (88.9)
Yes	27 (21.3)	2 (11.1)	0.46 (0.07-1.76, P = .324)	—
Dyspnea
No	74 (58.3)	14 (77.8)
Yes	53 (41.7)	4 (22.2)	0.40 (0.11-1.18, P = .122)	—
CXR
Not done	82 (64.6)	15 (83.3)
Done	45 (35.4)	3 (16.7)	0.36 (0.08-1.18, P = .126)	—
Pneumonia
No	102 (80.3)	17 (94.4)
Yes	25 (19.7)	1 (5.6)	0.24 (0.01-1.26, P = .175)	—
Admitted
No	101 (79.5)	17 (94.4)
Yes	26 (20.5)	1 (5.6)	0.23 (0.01-1.19, P = .160)	—
ICU
No	120 (94.5)	17 (94.4)
Yes	7 (5.5)	1 (5.6)	1.01 (0.05-6.18, P = .994)	—
Log viral load, mean (SD)	4.4 (1.9)	2.1 (1.6)	0.45 (0.28-0.65, P < .001)	0.46 (0.29-0.67, P < .001)

CI, confidence interval; CXR, chest roentgenogram; ICU, intensive care unit; NP, nasopharyngeal; OR, odds ratio; UTM, universal transport medium; VTM, viral transport media; —, variable eliminated through stepwise analysis process.

aValues are provided as number (%) unless otherwise indicated.

Table 4

Associations with False Negative Results for the ID Now COVID-19 (Abbott) Assay

			OR (95% CI)
Clinical Variable	True Positivesa	False Negativesa	Univariable	Multivariable
Age, mean (SD), y	51.7 (17.8)	43.1 (14.6)	0.97 (0.94-0.99, P = .025)	—
Sex
Male	57 (47.9)	8 (30.8)
Female	62 (52.1)	18 (69.2)	2.07 (0.86-5.38, P = .116)	—
Specimen/medium
NP/UTM	52 (43.7)	3 (11.5)
Nasal/VTM	67 (56.3)	23 (88.5)	5.95 (1.94-26.05, P = .005)	4.95 (1.30-25.30, P = .018)
Fever
No	53 (44.5)	17 (65.4)
Yes	66 (55.5)	9 (34.6)	0.43 (0.17-1.01, P = .058)	—
Cough
No	30 (25.2)	13 (50.0)
Yes	89 (74.8)	13 (50.0)	0.34 (0.14-0.81, P = .015)	—
Nausea or vomiting
No	111 (93.3)	24 (92.3)
Yes	8 (6.7)	2 (7.7)	1.16 (0.17-4.98, P = .860)	—
Diarrhea
No	94 (79.0)	22 (84.6)
Yes	25 (21.0)	4 (15.4)	0.68 (0.19-1.99, P = .518)	—
Dyspnea
No	69 (58.0)	19 (73.1)
Yes	50 (42.0)	7 (26.9)	0.51 (0.19-1.25, P = .158)	—
CXR
Not done	77 (64.7)	20 (76.9)
Done	42 (35.3)	6 (23.1)	0.55 (0.19-1.40, P = .235)	—
Pneumonia
No	94 (79.0)	25 (96.2)
Yes	25 (21.0)	1 (3.8)	0.15 (0.01-0.77, P = .070)	—
Admitted
No	93 (78.2)	25 (96.2)
Yes	26 (21.8)	1 (3.8)	0.14 (0.01-0.73, P = .062)	—
ICU
No	112 (94.1)	25 (96.2)
Yes	7 (5.9)	1 (3.8)	0.64 (0.03-3.83, P = .683)	—
Log viral load, mean (SD)	4.6 (1.8)	1.9 (1.4)	0.32 (0.19-0.49, P < .001)	0.32 (0.18-0.50, P < .001)

CI, confidence interval; CXR, chest roentgenogram; ICU, intensive care unit; NP, nasopharyngeal; OR, odds ratio; UTM, universal transport medium; VTM, viral transport media; —, variable eliminated through stepwise analysis process.

aValues are provided as number (%) unless otherwise indicated.