Analysis of the specificity of a COVID-19 antigen test in the Slovak mass testing program.

AIMS: Mass antigen testing programs have been challenged because of an alleged insufficient specificity, leading to a large number of false positives. The objective of this study is to derive a lower bound of the specificity of the SD Biosensor Standard Q Ag-Test in large scale practical use.
METHODS: Based on county data from the nationwide tests for SARS-CoV-2 in Slovakia between 31.10.-1.11. 2020 we calculate a lower confidence bound for the specificity. As positive test results were not systematically verified by PCR tests, we base the lower bound on a worst case assumption, assuming all positives to be false positives.
RESULTS: 3,625,332 persons from 79 counties were tested. The lowest positivity rate was observed in the county of Rožňava where 100 out of 34307 (0.29%) tests were positive. This implies a test specificity of at least 99.6% (97.5% one-sided lower confidence bound, adjusted for multiplicity).
CONCLUSION: The obtained lower bound suggests a higher specificity compared to earlier studies in spite of the underlying worst case assumption and the application in a mass testing setting. The actual specificity is expected to exceed 99.6% if the prevalence in the respective regions was non-negligible at the time of testing. To our knowledge, this estimate constitutes the first bound obtained from large scale practical use of an antigen test.

Introduction

While PCR-tests are usually considered as the gold standard to detect infection with the SARS-CoV-2 coronavirus in terms of sensitivity as well as specificity, antigen tests (Ag-Tests) offer practical advantages in terms of costs, logistics and speed [1]. Because Ag-Tests may play a major role in large scale testing strategies [2, 3] in populations with low prevalence, besides their sensitivity also the specificity is of significant interest, especially as resulting low positive prognostic values may lead to confusion and public distrust into the testing strategy. The objective of this study is to obtain a lower bound of the specificity of the SD Biosensor Standard Q Ag-Test based on data of mass tests in Slovakia and infer Ag-Test specificity from a large sample of the general population. The lower bound for the specificity is obtained by making the conservative assumption that potentially all positive results are false positive results. This is in contrast to other studies which use a PCR-test as a reference to estimate sensitivity and specificity of an Ag-Test [4-7].

Methods

From late October to early November 2020, Slovakia undertook large scale testing of its population [8]. Participants were tested at specially set up locations by medical personnel via nasopharyngeal swab sampling and received their results after a short waiting period. For testing, the SD Biosensor Standard Q Ag-Test was used [9, 10]. The tests in Slovakia were divided into three phases. In the first pilot phase, testing was only conducted in certain particularly affected counties. In Phase 2 (31.10.–1.11.) all Slovak counties were tested. In Phase 3 (6.11.–8.11.) all heavily affected counties (those with > 0.7% prevalence during Phase 2) were tested again. In this retrospective study, we use publicly available data on the outcome of tests in Phase 2 on county level [11]. Participation in testing was voluntary, but it was a condition to avoid quarantine. Persons that were quarantined due to a previous positive PCR-test for COVID-19 or due to being a close contact of such a person were excluded from the test.

Statistics

To derive a lower bound for the specificity we made the conservative (“worst case”) assumption that potentially all positive results constitute false positive results. For each of the 79 counties, we compute the rate of positive tests together with two-sided binomial Clopper-Pearson confidence intervals at the Bonferroni-adjusted significance level of alpha = 0.05/79 (adjusted for the number of counties; see e.g. [12]). The minimum upper bound of these adjusted confidence intervals is an upper 97.5% confidence bound for the test positivity rate. This will be in a county with low disease activity and a large sampled population. Under the conservative assumption that all positive results constitute false positives, it is also an upper bound on the false positive rate of the test and defines a lower bound for the true specificity. Denoting prevalence, sensitivity and specificity by r, s and p, this holds under the assumption that s + p ≥ 1 (i.e., that the probability of a positive test result is larger for a SARS-COV2 positive than for a SARS-COV2 negative subject). This is fulfilled for any test that performs not strictly worse than chance. Then the overall rate of positive results is given by q = r s + (1 − r)(1 − p) and it follows that q ≥ 1 − p, where 1 − p is the false positive rate of the test. Consequently, 1 − q is a lower bound for the specificity p.

Results

In phase 2, all residents aged 10 to 65 throughout Slovakia were invited to get tested and 3 625 332 participated (about 66% of the population of Slovakia). Among 79 administrative counties of Slovakia, participation rate varied from 39% (Košice III county) to 78% (Senec county) of inhabitants. Fig 1 depicts the test positivity rate of the individual counties ordered according to the upper bound of the simultaneous Bonferroni adjusted 95% confidence interval from low to high incidence. The lowest upper bound (obtained for Rožnava) is 0.40% (see Table 1). In terms of specificity (instead of false positives) this implies that with 97.5% confidence, the specificity of Standard Q is higher than 99.6%. As a sensitivity analysis we also consider the five counties with the lowest upper bound for the positivity rate. All imply specificities above 99.54%. These counties include regions with both relatively low participation rate (Rožňava, Revúca and Veľký Krtíš rank 65., 67. and 57. in participation among the 79 counties) and relatively high participation rate (Bratislava IV and Bratislava III rank 13. and 4.).

Fig 1

SARS-CoV-2 antigen detected incidence in Slovakian mass testing by county.

Test positivity rates among the 3 625 332 tested persons in the 79 counties with simultaneous Bonferroni adjusted 95% confidence intervals. Counties are ordered from low to high incidence by the upper confidence bound. Data source: [11]. The corresponding data are tabled in S1 Table.

Table 1

The five counties with the lowest upper bound on the positivity rate of antigen tests.

County	Positivity rate upper bound simultaneous 95% CI	Number of tests	Number of positive tests	Participation, % of inhabitants tested	Population 2019 [13]
Rožňava	0.40%	34307	100	55%	62131
Revúca	0.41%	21419	58	54%	39537
Bratislava IV	0.43%	65861	229	67%	97792
Bratislava III	0.45%	49788	175	72%	69479
Veľký Krtíš	0.46%	24282	76	56%	43263

Discussion

In this study the lower bound for the specificity of the SD Biosensor Standard Q AG test was 96.4%. This worst case analysis provides only a lower bound of the true specificity since it neglects entirely the true incidence of COVID-19. As the antigen tests in Slovakia were not controlled directly using accompanying PCR-tests, we refrain from an attempt to subdivide the observed positives into true and false positives.

The above estimate is consistent with the information provided by the manufacturer [14], stating 0.32% (0.01, 1.78) as a false positive rate—a very broad CI. The derived upper bound from the mass testing is more informative and appears relevant in that it constitutes the first (to the best of our knowledge) bound obtained from large scale practical use of Standard Q and also suggests better performance of Standard Q compared to previous studies: a large study (with 2347 SARS-CoV-2-free samples based on a PCR-test), suggests a specificity of 99.3% (CI 98.6–99.6) [7]. Other available specificity estimates are based on an order of magnitude smaller samples (99.2% (CI 97.1–99.8) [4] and 100% [6] and, in addition, are from study populations with high incidence rates (according to PCR-testing). A point estimate of 98.53% is given by [5], based on 100 SARS-Cov2-free samples with other respiratory viruses present and 35 samples from healthy volunteers.

Data from Phase 2 was used in this study, because the general countrywide testing was performed irrespective of regional incidence rates, covering also regions with potentially very low incidence. The data has also been used to investigate spatial patterns of the spread of COVID-19 [10] and to demonstrate that both mass-scale testing and restrictive measures contributed to the sharp drop in COVID-19 incidence [9]. However, to avoid a positively biased specificity estimate, compared to [9] we use an updated, more detailed dataset, where an erroneous positivity rate for Bratislava IV has been corrected. The binomial distribution assumption underlying the computation of the Clopper-Pearson confidence intervals, is based on the assumption of independent events that could be violated if, e.g. testing stations were operating with different quality. However, the Bonferroni-correction is a strictly conservative approach to derive simultaneous confidence bounds that account for the selection of the county with the lowest upper bound to derive the estimate. It is well understood in the epidemiology literature that imperfect reference criteria lead to a systematic bias of the true incidence [15]. Specifically, the sensitivity and specificity of tests are systematically underestimated if the criterion (SARS-CoV-2 infection) cannot be assessed directly and tests are compared to a surrogate criterion as gold standard which is subject to errors. This could be a contributing factor for the relatively higher specificity found in Slovak data compared to other studies. However, there are other conceivable factors that could bias the results: different quality of swab sampling and handling, low temperature in outdoor testing stations, deviations in the production over time and data quality.

While the analysis is based on a worst case assumption, considering all positives to be false positives, the obtained estimate appears relevant in that it constitutes the first bound obtained from large scale practical use of Standard Q. It suggests better specificity of Standard Q compared to previous studies and these findings can support the planning and justification of future mass testing programs. Besides test specificity, the sensitivity is essential for the evaluation of the overall utility of mass testing strategies. Unfortunately, based on the mass testing data set no estimate of the test sensitivity can be obtained and, to our knowledge, no sensitivity estimates have been provided in a mass-testing setting so far such that further studies are needed.

County, number of tests, number of positive tests, positivity rates, simultaneous Bonferroni adjusted 95% confidence intervals, and standard errors of the positivity rates in the 79 counties.

Counties are ordered from low to high incidence by the upper confidence bound. Data source [11].

(PDF)

Click here for additional data file.

26 May 2021

PONE-D-21-14542

Analysis of the specificity of a COVID-19 antigen test in the Slovak mass testing program

PLOS ONE

Dear Dr. Posch,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

I obtained the comment from one reviewer but could not obtain it from the other reviewer. To progress the smooth review process, I have determined to move to the next round for the review.

Please submit your revised manuscript by Jul 08 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Etsuro Ito

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

and

2. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

3. We note you have included a table to which you do not refer in the text of your manuscript. Please ensure that you refer to Table 1 in your text; if accepted, production will need this reference to link the reader to the Table.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: In presenting argument in the introduction that antibody testing can be an alternative for RT-CPR, it may be helpful to note that the test evaluates different things: active infection vs history of infection (including current), with different delays between tests being positive (longer for serology). This reviewer agrees that serology can be a very attractive alternative for reasons cited by the authors, but additional caveats other than SN (sensitivity) and SP (specificity) related to meaning of the test in terms of inference of onset of incident infection should be clearly presented.

Can you please be clearer in the introduction how your approach is “in contrast to other studies which use a PCR-test as a reference to estimate sensitivity and specificity of an Ag-Test4”?

One alternative to Bonferroni correction that seems appealing to this reviewer is to model rate of positive tests using binomial regression with random effect of county, and maybe some fixed effects that can account for rate of positive tests, such as numbers quarantined per county, positivity rates from Phase 1, age structure, economic indicators, percent of county tested, outdoor temperature, count/proportion of outdoor test sites, date of test (the usual confounders in epidemiology that can affect both willingness to test and chance of having been infected plus those mentioned in the discussion by the authors). The fixed intercept of the model would overall rate of positive tests and can be made county-specific through use of other random and fixed covariates; upper percentile of these modelled estimates can be used authors did with their Bonferroni-adjusted confidence interval. One may be also tempted to model spatial correlation, if descriptive statistics support its existence.

Instead of just giving upper bound of 95%CI, it would be more informative to give both the estimate per county and associated standard error. This should allow anyone to estimate different percentile if for their purposes 97.5% is not appropriate. This may already be in the supplemental materials, so maybe this is just a matter of pointing the reader in the right direction.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Igor Burstyn

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

12 Jul 2021

We thank the referee for the helpful remarks and revised the manuscript accordingly. In addition, we added a table with all estimates in the supporting information and formatted the document according to the PLOS ONE formatting guidelines. Please see the submitted point to point answer for details.

Submitted filename: Point to Point reply.pdf

Click here for additional data file.

14 Jul 2021

Analysis of the specificity of a COVID-19 antigen test in the Slovak mass testing program

PONE-D-21-14542R1

Dear Dr. Posch,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Etsuro Ito

Academic Editor

PLOS ONE

21 Jul 2021

PONE-D-21-14542R1

Analysis of the specificity of a COVID-19 antigen test in the Slovak mass testing program

Dear Dr. Posch:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Prof. Etsuro Ito

Academic Editor

PLOS ONE