Identification of SARS-CoV-2 in a Proficiency Testing Program.

OBJECTIVES: At the onset of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic in the United States, testing was limited to the Centers for Disease Control and Prevention-developed reverse transcription polymerase chain reaction assay. The urgent and massive demand for testing prompted swift development of assays to detect SARS-CoV-2. The objective of this study was to assess the accuracy of these newly developed tests.
METHODS: The American Proficiency Institute sent 2 test samples to 346 clinical laboratories in order to assess the accuracy of SARS-CoV-2 assays. The positive sample, containing 5,175 viral copies/mL, was fully extractable with SARS-CoV-2 viral capsid protein and RNA. The negative sample, with 3,951 viral copies/mL, contained recombinant virus particles with sequences for targeting human RNAase P gene sequences.
RESULTS: Of the laboratories submitting results, 97.4% (302/310) correctly detected the virus when present and 98.3% (296/301) correctly indicated when the virus was not present. Among incorrect results reported in this proficiency challenge, 76.9% (10/13) were likely related to clerical error. This accounts for 1.6% (10/611) of all reported results.
CONCLUSIONS: Overall performance in this SARS-CoV-2 RNA detection challenge was excellent, providing confidence in the results of these new molecular tests and assurance for the clinical and public health decisions based on these test results. © American Society for Clinical Pathology, 2020. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.

Overall performance in this SARS-CoV-2 RNA detection challenge was excellent, providing confidence in the results of these new molecular tests.

Demand for quality SARS-CoV-2 tests is universal. Laboratories from 46 states and 4 countries participated in the first US assessment of test accuracy.

Over 30 tests methods were reported by the more than 300 respondents in this challenge.

In December 2019 a cluster of respiratory disease cases were recognized in Wuhan, China.[1] By January 2020, the cause of the infections was identified as a novel coronavirus that was later designated severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2).[2] The first confirmed case of SARS-CoV-2 infection in the United States was identified in Washington in a traveler returning from Wuhan.[3] As of June 1, 2020, over 1.7 million cases and over 102,000 deaths have been reported in the United States.[4] The rapid and widespread transmission of the virus led to unprecedented social and economic disruptions as governments ordered schools and businesses closed.

At the onset of the US SARS-CoV-2 epidemic, testing was limited to the Centers for Disease Control and Prevention (CDC) and state public health laboratories using the CDC-developed reverse transcription (RT) polymerase chain reaction (PCR) assay. In early 2020, the urgent and massive demand for testing led to rapid development and validation of commercial and laboratory-developed assays to detect SARS-CoV-2.

In the United States, each clinical laboratory is required to verify performance of Food and Drug Administration (FDA)-approved and/or validated laboratory diagnostic tests. However, there is no standard for the number of samples to be included in a verification or the acceptable performance level.[5] This means there is considerable variation among laboratory verification studies, leading to concerns about reliability of test results.

Globally, SARS-CoV-2 molecular tests are being performed in laboratories with a range of experience and technical capacity for nucleic acid amplification testing. The commercial and laboratory-developed tests, even if validated and approved for use by a regulatory body, have little performance history in wide deployment. Therefore, an objective measure of system-wide product and laboratory quality is needed.

Results of SARS-CoV-2 testing are used not just for patient management but also for infection control in health care settings and for surveillance data that drive decisions on community-wide sheltering orders. SARS-CoV-2 test results are the cornerstone of contact tracing activities to control ongoing disease transmission. This study was undertaken to evaluate the performance of laboratories enrolled in a SARS-CoV-2 RT-PCR proficiency testing (PT) program and to assess reliability of test kits and methods.

The American Proficiency Institute (API) is a PT provider approved by the federal Centers for Medicare and Medicaid Services under the Clinical Laboratory Improvement Amendments of 1988 (CLIA ’88).[5] API provides PT samples and performance analysis to over 20,000 laboratories with over 350 programs. Subscribing laboratories are shipped samples and instructed to test them in the same manner as patient specimens. Each laboratory reports its results to API. API then provides performance feedback to subscribers. Analysis of PT results has been useful in the past to reveal deficiencies in testing quality and has led to recommendations that improved testing accuracy.[6,7] The ability of laboratories to correctly detect the presence or absence of SARS-CoV-2 has not previously been studied by CLIA-approved PT organizations. Data from these studies are important because they provide a snapshot of current laboratory practices and accuracy.

In this report we present the results of the first US study of SARS-CoV-2 accuracy by API participant laboratories from the 2020 First Test Event.

Materials and Methods

Data were acquired from a single PT event, 2020 First Test Event, by API. As part of this PT event, 2 samples (both noninfectious and manufactured by SeraCare) were shipped overnight in May 2020 to 346 laboratories enrolled in the API program. These subscribers represented domestic and international commercial laboratories, public health laboratories, clinics, hospitals, and diagnostic kit manufacturers.

The positive sample (COV-01), containing 5,175 viral copies/mL, was fully extractable with SARS-CoV-2 RNA encoding viral capsid protein and RNase P. It was prepared to be compatible with assays targeting the following regions: ORF1a, RdRp, S (spike), E (envelope), and N (nucleocapsid). The negative sample (COV-02), with 3,951 viral copies/mL, contained recombinant virus particles with sequences for targeting sequences from human RNase P gene. The proficiency samples were formulated in viral transport media consisting of Tris-buffered saline, with added antimicrobial agents, glycerol, and human proteins.

Participating testing sites were located in 50 clinical point-of-care testing sites, 182 hospital-based laboratories, 37 independent laboratories, 3 diagnostic kit manufacturers, and 5 government care facilities; 17 testing sites did not indicate laboratory type. Participating laboratories were located in 46 US states and 4 international sites (Malaysia, Pakistan, Taiwan, and Vietnam.)

Laboratories were instructed to submit the interpreted results (detected or not detected) for COV-01 and COV-02, and to provide the instrument and test kit used for testing the samples. Since laboratories using traditional PCR methods could participate, cycle threshold (Ct) values were not requested on the report. Of the 346 laboratories that received proficiency materials, 310 submitted results by the reporting deadline. The results from these samples were processed with proprietary software developed at API.

Results

Correct positive results were reported by 302 laboratories (97.4%), with 8 laboratories incorrectly reporting negative results for COV-01 (2.6%). Negative results were reported by laboratories using Applied Biosystems/Quidel Lyra SARS-CoV-2 and Luminex ARIES SARS-CoV-2 and had correct negative results for COV-02. For COV-02, 306 laboratories correctly reported negative results (98.3%) . The 9 laboratories who reported a testing problem with sample COV-02 all used the BioGx SARS-CoV-2 reagent on the BD Max System. BioGx users indicated that their results for sample COV-02 were “unresolved” due to an internal control failure and were thus nonreportable. The BioGx SARS-CoV-2 reagent (along with several other testing methods) requires human RNase P to be present in a sample to serve as an endogenous nucleic acid extraction control, which is present in all properly collected patient samples. Sample COV-02 did contain 3,951 copies/mL of RNase P; however, this was very near the limit of detection for the BioGx reagent, resulting in 9 of 25 laboratories using the reagent recovering levels of RNase P below the required threshold of detection.

shows that all types of laboratories performed well, with consensuses higher than 90%. The manufacturer’s category was excluded due to a low number of participants. False-negative results were reported by 4 independent laboratories, 4 hospital-based laboratories, and 1 diagnostic kit manufacturer. False-positive results were reported by 1 hospital-based laboratory and 4 independent laboratories. All 5 laboratories that reported false-negative results for COV-01 also reported a false-positive result for COV-02, indicating probable clerical errors during testing or reporting.

Among incorrect results reported in this proficiency challenge, 76.9% (10/13) were likely related to clerical error. This accounts for 1.6% (10/611) of all reported results.

Discussion

Before passage of CLIA ’88, participation in PT was voluntary for many clinical laboratories. With the implementation of the CLIA ’88 rules, PT evolved from an educational self-assessment tool to a measure that is fundamental for trend analysis, risk management, and laboratory accreditation. Performance on PT is a vital, objective indicator of the quality of clinical testing.

Monitoring and analyzing PT results from a large group of participating clinical laboratories helps to assess the accuracy of test methods applied in a variety of settings and individual laboratory performance. The significance of this measure is compounded when a test is new and widely practiced and results have several applications. SARS-CoV-2 RNA amplification test results are used for patient management, infection control in health care settings, contact tracing, and epidemiologic surveillance data. SARS-CoV-2 test results are the cornerstone of contact tracing activities, community-wide sheltering orders, and control of ongoing disease transmission during this historic pandemic.

With an overall consensus greater than 97%, this SARS-CoV-2 RNA detection challenge indicates excellent accuracy by test participants when evaluated by testing method and type of laboratory. The results of this study should provide confidence in clinical laboratory test results for patient management and public health decisions.

There are several limitations to this study. While PT materials are designed to mimic patient specimens and be stable, they are not exactly the same matrix and do not have the entire viral genome for safety considerations. This limitation is demonstrated by the challenges that laboratories using Applied BioSystems/Quidel Lyra reported. SARS-CoV-2 is not an analyte that CLIA requires PT for at this time. Therefore, the results of this voluntary program do not reflect performance across all laboratories performing SARS-CoV-2 amplification assays. Laboratories may have deployed multiple testing methods, but results are reported for only 1 method per laboratory. This study may not accurately represent the true scope of method deployment. Finally, Ct values were not collected. Ct value comparison for sites using the same quantitative PCR method may be valuable to assess interlaboratory variability.

Negative results for the COV-01 sample were reported by laboratories using Applied Biosystems/Quidel Lyra SARS-CoV-2 and Luminex ARIES SARS-CoV-2; these laboratories had correct negative results for the COV-02 sample. Upon further review, sample COV-01 lacked the target region (pp1ab) of the SARS-CoV-2 genome that is detected by the Quidel Lyra test system. Future PT challenge samples should include target gene sequences detected by all FDA-approved SARS-CoV-2 RNA assays. One of 6 laboratories using Luminex ARIES SARS-CoV-2 reported a negative result for COV-01 but reported expected results for COV-02. This result pattern is not typical of clerical errors, and laboratories using this method should closely monitor test performance through heightened ongoing verification activities.

Five testing sites reported both a false negative for COV-01 and a false positive for COV-02, suggesting clerical errors. Typical rate for clerical errors in reporting PT results is historically about 1%. While clerical errors do not reveal lack of sensitivity or specificity of the test method, they have an equal impact on treatment, infection control, and disease control efforts. Laboratories that do not meet expected performance on PT challenges due to clerical errors must investigate and correct the processes from which these errors arose.

Table 1

Overall Performance by Test Method of 310 Testing Sites Participating in SARS-CoV-2 RNA Detection Proficiency Testing Challengea

Method	COV-01, No. (% Correct)	COV-02, No. (% Correct)
Abbott m2000/Abbott RealTime SARS-CoV-2 assay	5 (100)	5 (100)
Agilent AriaMx/Light Power IVASARS-CoV-2	1 (100)	1 (100)
Applied Biosystems PCR/Altona RealStar SARS-CoV-2	1 (100)	1 (100)
Applied Biosystems PCR/CDC 2019-nCoV RT-PCR	13 (100)	13 (100)
Applied Biosystems PCR/Lab Corp COVID-19 RT-PCR	1 (0)	1 (0)
Applied Biosystems PCR/NY Wadsworth SARS-CoV-2 RT-PCR	1 (100)	1 (100)
Applied Biosystems PCR/Quest Diagnostics SARS-CoV-2	1 (100)	1 (100)
Applied Biosystems PCR/Quidel Lyra SARS-CoV-2	2 (0)	2 (100)
Applied Biosystems PCR/Thermo Fisher TaqPath COVID-19	9 (100)	9 (100)
BD Max /BD SARS-CoV-2	5 (100)	5 (100)
BD Max /BioGx SARS-CoV-2	25 (100)	16 (100)b
Bio-Rad CFX/ADT LyteStar 2019-nCOV	1 (100)	1 (100)
Bio-Rad CFX/CDC 2019-nCoV RT-PCR	1 (100)	1 (100)
Bio-Rad CFX/Curative-Korva SARS-CoV-2	1 (100)	1 (100)
Bio-Rad CFX/Logix Smart SARS-CoV-2	1 (100)	1 (100)
Bio-Rad CFX/Thermo Fisher TaqPath COVID-19	3 (100)	3 (100)
Cepheid Xpert Xpress SARS CoV-2	189 (99.5)	189 (99.5)
DiaSorin Simplexa COVID-19	4 (100)	4 (100)
GenMark ePlex SARS-CoV-2	2 (100)	2 (100)
Hologic Panther Fusion SARS-CoV-2	1 (100)	1 (100)
Luminex ARIES SARS-CoV-2	6 (83.3)	6 (100)
Luminex NxTag CoV	2 (50)	2 (50)
Mesa Biotech Accula SARS-CoV-2	1 (100)	1 (100)
NeuMoDx SARS-CoV-2	2 (100)	2 (100)
QIAstat-Dx Respiratory SARS-CoV-2 Panel	5 (100)	5 (100)
QuantStudio/CDC 2019-nCoV RT-PCR	1 (0)	1 (100)
QuantStudio/LabTurbo AIO COVID-19	1 (100)	1 (100)
QuantStudio/Thermo Fisher TaqPath COVID-19	2 (50)	2 (50)
Roche cobas 6800, 8800/cobas SARS-CoV-2	16 (100)	16 (100)
Roche cobas Z480/cobas SARS-CoV-2	1 (100)	1 (100)
Roche LightCycler/CDC 2019-nCoV RT-PCR	3 (100)	3 (100)
Sacace PCR/Sansure Biotech (2019-nCOV)	1 (100)	1 (100)
Sentosa SA201/ViroKey SARS-CoV-2	2 (100)	2 (100)
Total	302 (97.4)	296 (98.3)

COVID-19, coronavirus disease 2019; RT-PCR, reverse transcription polymerase chain reaction.

aSample COV-01 contained 5,175 copies/mL of SARS-CoV-2 virus; sample COV-02 was negative for SARS-CoV-2.

bNine laboratories reported a testing problem with sample COV-02 due to lack of internal amplification control.

Table 2

Overall Performance by Type of Laboratory of 310 Testing Sites Participating in SARS-CoV-2 RNA Detection Proficiency Challengea

Method	COV-01, No. (% Correct)	COV-02, No. (% Correct)
Clinic/Physician office laboratory	50 (100)	50 (100)
Hospital laboratory ≤100 beds	57 (100)	57 (100)
Hospital laboratory 101-200 beds	47 (97.9)	47 (100)
Hospital laboratory 201-300 beds	44 (95.5)	44 (97.7)
Hospital laboratory 301-400 beds	23 (100)	23 (100)
Hospital laboratory >400 beds	19 (100)	19 (100)
Independent laboratory	41 (90.2)	41 (90.2)
Manufacturer	3 (66.7)	3 (100)
Point of care	1 (100)	1 (100)
VA hospital/clinic	6 (100)	6 (100)
Not indicated	17 (100)	17 (100)
Total	310	310

aSample COV-01 contained 5,175 copies/mL of SARS-CoV-2 virus; sample COV-02 was negative for SARS-CoV-2.