Validation testing to determine the sensitivity of lateral flow testing for asymptomatic SARS-CoV-2 detection in low prevalence settings: Testing frequency and public health messaging is key.

Lateral flow devices (LFDs) are quickly being implemented for use in large-scale population surveillance programs for SARS-CoV-2 infection in the United Kingdom. These programs have been piloted in city-wide screening in the city of Liverpool and are now being rolled out to support care home visits and the return home of University students for the Christmas break. Here, we present data on the performance of LFDs to test almost 8,000 students at the University of Birmingham between December 2 and December 9, 2020. The performance is validated against almost 800 samples using PCR performed in the University Pillar 2 testing lab and theoretically validated on thousands of Pillar 2 PCR testing results performed on low-prevalence care home testing samples. Our data show that LFDs do not detect infections presenting with PCR Ct values over 29 to 30 as determined using the Thermo Fisher TaqPath asssay. This may be of particular importance in detecting individuals that are either at the early, or late stages of infection, and reinforces the need for frequent, recurrent testing.

Introduction

In November 2020, the United Kingdom government announced a plan to introduce mass-scale population testing for Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) infection using Lateral Flow Devices (LFDs) [1]. The most commonly utilised of these is an LFD manufactured and marketed by Innova Medical Group, a subsidiary of Xiamen Biotime Biotechnology Company, which was the first device to pass a 4-phase validation. The LFD is a rapid lateral flow device based on colloidal gold immunochromatography designed to detect the presence of SARS-CoV-2 nucleocapsid antigens in nasopharyngeal swabs [2]. The test can provide a result within 30 minutes allowing rapid testing on a mass scale.

The Innova LFD has very quickly been put into implementation by the Department for Health and Social Care (DHSC) and was employed in the city of Liverpool to deliver an ambitious mass-scale surveillance project of the city over a 2-week period [3]. Data from the city council [4] show that 71,684 LFD tests were performed alongside 51,855 gold-standard PCR tests (a total of 119,054 residents tested) with 439 people testing positive (0.37% positivity rate). The LFD tests are now being used in to support people visiting relatives in Care Homes and are being rolled out to support testing of University students and secondary school pupils across the UK [1].

In order to support the use of the Innova LFD test, the University of Oxford and Public Health England (PHE) performed a series of validation trials of the LFD, benchmarking their performance against reverse transcription PCR (RT-PCR) using swabs from a number of research trials in the UK [5]. These included comparative testing on samples taken for the NIHR FALCON study [6], which aims to evaluate diagnostic platforms [6], and bespoke trials including PHE, hospital, military staff, and schools. The key headline findings of the validation report were that the LFD had a Limit of Detection (LoD) of around 100 plaque forming units/ml or 100,000 RNA copies/ml [7]. In the report, it is not made clear which RT-PCR assay is used in the comparison, but the cycle threshold (Ct) value given of 25.5 equating to 100 pfu/ml is not obtained from the ThermoFisher COVID-19 TaqPath assay [8] employed in the majority of Pillar 2 testing labs in the UK. As such, the validation report may not fully indicate the potential of the performance of the LFD against the vast majority of COVID PCR testing done in the UK through Pillar 2. This study aimed to assess the efficacy of the Innova LFD platform for the purposes of mass screening of asymptomatic individuals, using a purpose-built facility at the University of Birmingham. We determine through RT-PCR validation that Innova LFD platform captures all cases in which individuals have high quantities of viral RNA present on the swab, as presented to us (i.e., N gene Ct <30) and recommend that frequent, routine testing, paired with clear public health messaging, is key to the successful implementation of LFD testing in order to reduce overall community burden of SARS-CoV-2.

Methods

Lateral flow device testing of students

As part of the national plan to test students for SARS-CoV-2 before the 2020 Christmas break, the DHSC provided University of Birmingham with Innova LFDs to test 15,000 students. Using a University-owned booking system, a total of 7,189 students were tested between December 2 and December 9. Students were provided with a sterile nasopharyngeal swab and under supervision from a trained member of the University testing team, swabbed both tonsils and a single nasal cavity. The swab was passed through an opening in a plastic screen to a designated test area, where it was immediately processed according to the Innova protocol [2]. Briefly, this involves adding 6 drops of sample buffer to a sample tube, immersing and agitating the patient’s swab inside the buffer to homogeneity, and loading 2 drops of the final solution onto the test cartridge. Tests were performed by trained members of the University testing team drawn from postgraduate and final year undergraduate students in the College of Medical and Dental Science in the University, supervised by highly experienced postdoctoral researchers. In total, a team of 18 test operatives oversaw 36 testing booths, with a student attending a booth every 10 minutes. A further 7 staff acted as results recorders logging the test results via a barcode through a mobile phone device and result recording app, which were both supplied by DHSC.

Validation of lateral flow device test results by PCR testing

University of Birmingham is home to a national Pillar 2 testing laboratory (i.e., swab testing for the wider population), termed Turnkey lab, which conducts SARS-CoV-2 PCR diagnostics on behalf of DHSC [9]. The laboratory uses the ThermoFisher COVID-19 TaqPath assay used routinely in the Lighthouse laboratory testing network and tests a range of samples from mobile and stationary test sites [9]. On each day of testing, 90 residual LFD test samples (buffer solution in which the nasopharyngeal swab is resuspended to perform the test) were selected for confirmatory PCR testing, which constituted a single 96-well plate including positive and negative controls. All positive samples were chosen for confirmatory PCR, and the remainder were randomly selected samples. All samples were completely anonymous to the testing team with no identifying labels and were arbitrarily numbered from 1 to 90 each day. Sterile water (approximately 350 μl) was added to the samples to bring them to the 500 microlitres required for automated RNA extraction and tested according to Pillar 2 laboratory protocol [9]. The use of anonymised waste samples from student testing in this study was allowed under ethics gained to aid assay development (NRES Committee West Midlands—South Birmingham 2002/201 Amendment Number 4, 24 April 2013).

Statistical analysis of PCR validation

The efficient stratified study design involved verification of all Innova test positives with RT-PCR and a random sample of 720/7,187 Innova test negatives. Estimates of sensitivity and prevalence with 95% confidence intervals were obtained using maximum likelihood inverse probability weighted logistic regression to account for the sampling design, with conversion of the estimated odds to probabilities. Weights of 1 for Innova test positives and 9.98 (7,187/720) for Innova test negatives were used. Expected numbers of cases were computed from the prevalence estimate. The estimate of specificity was obtained without weighting as no Innova test positives were observed in those with negative PCR. Exact binomial methods were used to compute confidence intervals for test yield and specificity.

Theoretical validation of LFD performance against Pillar 2 PCR test data

As part of Pillar 2 testing, our Turnkey laboratory also conducts PCR testing as part of the national Care Home surveillance plan implemented by DHSC to test all care home staff and residents to assist in control of COVID-19 transmission in UK care homes [10]. Between October 25 and November 5, the Birmingham Turnkey laboratory processed a total of 19,176 PCR tests on home test and care home samples from across the UK. Of these, 641 samples tested positive for SARS-CoV-2 using the cutoff of 2 of 3 gene targets amplifying at a Ct value of 35 or under [9]. This gives a positivity rate of 3.3%, around the rate that might be reasonably be expected in a large random surveillance of the UK population at that moment in time.

Results

Lateral flow testing of University of Birmingham students

A total of 7,189 students voluntarily attended the asymptomatic student testing centre between December 2 and December 9. Students were refused a test if they had any symptoms of COVID-19 and were referred to a local test site for PCR testing. Results of 4 samples were void (as defined by the manufacturer’s protocol [2]), and 2 samples tested positive for SARS-CoV-2 by lateral flow, a prevalence of 0.03% (95% CI 0.02% to 0.10%) in the students volunteering for a test (Table 1).

Table 1

Table of results for Lateral Flow Device testing of University of Birmingham students and confirmatory PCR testing of approximately 10% of samples.

	LFD tests		PCR	Validation
Day	Positive	Negative	Positive	Negative
02/12/2020	0	630	0	90
03/12/2020	2	589	2	89
04/12/2020	0	1,102	1	88
05/12/2020	0	860	1	89
06/12/2020	0	610	0	90
07/12/2020	0	813	2	88
08/12/2020	0	1,259	2	88
09/12/2020	0	1,320	0	90
Totals	2	7,183	8	712

Lateral flow results validation by pillar 2 PCR

The 2 samples positive by Lateral Flow and another 718 randomly selected negative samples were transported to the University Turnkey laboratory for PCR testing (9.9% of sample total). Of the 720 samples tested by PCR, 8 were positive: The 2 positive by Lateral Flow and a further 6 samples negative on the LFD (Table 1).

Our PCR validation data suggest a true prevalence rate in the student population tested of 0.86% (95% CI 0.40% to 1.86%). The overall sensitivity of the test in the tested student population was observed to be 3.23% (95% CI 0.60% to 15.59%). We estimate that there would have been 62 cases in the 7,185 students, of which 60 were missed. There were no false positive results, observed specificity was 100% (95% CI 99.48% to 100.00%).

We further investigated the PCR testing discrepancy by extracting the Ct values for the amplification curves for the 8 PCR positive samples (Table 2). Our data show that the 6 samples testing false negative by Lateral Flow all had Ct values >29, while the 2 true positive samples had Ct values of 20 and 25. We collated the RT-PCR raw data from 5 technical replicates of assays performed on the Qnostics SARS-CoV-2 analytical Q-panel– 01 [11] and generated average Ct values for each of the known viral titres provided in the panel (Fig 1). Using these data, we determined that, at 100 viral copies per ml (the designated LoD for the Innova LFD [2,7]), the equivalent Ct values for the Pillar 2 PCR assay would be a Ct of 30.8 based on the N gene target, roughly in line with our PCR validation data.

Table 2

Pillar 2 PCR Ct values of confirmatory samples positive for SARS-CoV-2.

The samples which tested positive on Lateral Flow device are in grey columns.

Well Number	ORF1ab Ct	N gene Ct	S gene Ct
38	32.501118	33.89259	34.815563
1	25.160471	25.678833	25.034386
57	21.319279	22.638311	20.582413
36	28.538937	29.359957	29.123411
38		32.216614	33.92468
42	27.669174	29.401642	27.895124
14	30.770878	31.718863	31.998856
34	30.917858	31.794565	31.35588

Fig 1

Analytical sensitivity and specificity of the Birmingham Turnkey lab RT-PCR pipeline.

This was assessed against the commercial Qnostics SARS-CoV-2 analytical Q-panel– 01. Ct values are a median of 5 independent technical replicates, and figures in parentheses indicate the percentage of replicates returning a PCR positive for that given gene target (Ct <35). Ct, cycle threshold; RT-PCR, reverse transcription PCR; SARS-CoV-2, Severe Acute Respiratory Syndrome Coronavirus 2.

Sensitivity of lateral flow device in the student population by Ct value

From our data, the LFD test yield is 2.8 per 10,000 (0.3 to 10 per 10,000 tests). The sensitivity of the LFD in the tested population differs greatly dependent on the viral titre of the person tested. At a PCR Ct value <29, the sensitivity was 100% (95% CI 15.8 to 100). However, at a Ct < = 29, this dropped to 9.1% (1.03 to 49.1), and at Ct <33 dropped again to 5.01% (0.78 to 32.14).

Extrapolation of pillar 2 PCR data to theoretically evaluate lateral flow device performance

We collated the raw RT-PCR data for all 641 of our positive samples as of November 5 and ranked them according the N gene Ct value (S1 Table). We then plotted the distribution of Ct values for our 641 positive samples (Fig 2). Using the LoD of 100 pfu/ml, we determined that this would correlate with an N gene Ct value of 30.8 plus one other gene target amplifying at a Ct <35. By applying this theoretical level of performance to the LFD, we determine that 99 of our positive samples would not be able to be detected by the Innova LFD given that the Ct value of N gene is above 30.8. This equates to 15.44% of our true positive RT-PCR samples would not be detected using the Innova LFD. This means that, theoretically, the Innova LFD (when compared to Pillar 2 samples from low-prevalence, asymptomatic population screening similar to student and care home surveillance) would successfully detect 84.56% of all infections.

Fig 2

Graph plotting raw Ct values (y-axis) for all 641 positive samples in the Birmingham Turnkey lab (y-axis).

Ct values for each of the targets (Orf1, N, S) are plotted, with a sample only called positive if at least 2 of the 3 targets have a Ct <35. The red line indicates the N gene Ct value equating to 100 viral copies/ml, the previously determined LoD for the Innova LFD. Ct, cycle threshold; LFD, Lateral Flow Device; LoD, Limit of Detection.

Discussion

Expansion of mass testing for SARS-CoV-2 is an issue many countries are facing, with LFDs seen as a viable accompaniment to the gold standard of RT-PCR tests as a way to increase capacity and screen asymptomatic populations. The University of Birmingham deployed a purpose built testing facility in order to screen its student population before they travelled home in the lead up to the 2020 holiday period. A total of 7,183 students were tested over an 8-day period, with 2 positive tests; a prevalence of 0.03% (95% CI 0.02% to 0.10%). Approximately 10% of tests carried out across the study were sent to an onsite Pillar 2 testing lab (termed Turnkey) for RT-PCR validation. The validation, carried out using the ThermoFisher COVID-19 TaqPath assay, confirmed the 2 positive LFD results. This gave a false positive rate of rate 0%, therefore indicating a 100% (95% CI 99.48% to 100.00%) specificity of the test in this study. The high specificity rate suggests that what we are seeing are true positive samples, and this is exemplified in the recent report published by the DHSC suggesting an LFD specificity rate of 99.72% [12].

There has been some controversy over sensitivity and specificity of the Innova LFD platform [13,14]. Importantly, the studies that are often cited in such controversies relied on RT-PCR analysis of a swab taken on a separate day, whereas our study analysed the same prepared sample that provided a positive LFD result. Swabbing technique can largely impact the accuracy of diagnostic platforms; research shows that the accuracy of RT-PCR testing is significantly increased when the swab is taken by a trained scientist (79%) when compared to swabs taken by members of the public (58%) [7]. As such, a large part of the variation in sensitivity and specificity rates may be confounded by differences in swabbing technique, rather than by the technical limitations of the Innova LFD platform itself. From a public health perspective, our positive sample size was thankfully small; however, for a true validation of false positivity, an ideal dataset would include many more positive samples and, importantly, rely on a single swab to reduce variation.

By screening samples that were deemed negative by LFD, RT-PCR validation detected 6 samples that were determined to be false negatives (Table 1). This suggests a true prevalence of 0.86% (95% CI 0.40% to 1.86%) within the cohort tested and when extrapolating from these data, represents 60 potentially missed positive cases. Using this information, we explored the LOD for LFDs. Given the barriers to culturing SARS-CoV-2, we did not determine a viral titre for these samples; however, we correlated N gene Ct values with viral titre by using the Qnostics SARS-CoV-2 analytical Q-panel. This validated the limit of detection of Innova LFD devices as 100 pfu/ml, which correlated to an N gene Ct value of approximately 30. Unfortunately, the Qnostics panel contains irradiated virus which as a result cannot be detected on the Innova LFD, meaning a cross validation of the LoD could not be performed on matching material; however, our data are concordant with larger-scale studies which have comparatively reported the LoD of the Innova test [7,15,16].

Importantly, all of the false negative samples had N gene Cts above or close to 30 and therefore above or close to the limit of detection for the Innova LFDs. These may be representative of individuals at the very early or very late stages of infection, who may have a relatively lower titre of SARS-CoV-2 in the swabbed sites at the time the swab was taken. It is therefore imperative that individuals undergo regular, routine testing, to accurately remove infected individuals from community transmission pools.

First reports of variants of concern (VOCs) were published following the closure of this test site in December; however, the UK VOC (also referred to as the Kent VOC) was known to be circulating in late September [17]. Since then, numerous other VOCs have been identified through genomic sequencing of positive cases worldwide; most notably with in the UK with COVID-19 Genomics UK Consortium (COG-UK) [18]. While this study did not look at VOCs specifically, research carried out by PHE showed that the Innova LFD platform could detect the UK VOC [19]. Recent emergence of the South African VOC can also be detected on this platform [20]. With these datasets in mind, we would suggest that both the UK and South African VOCs can be reliably detected using the Innova LFD platform but as more variants emerge, these will require their own rapid evaluation by relevant public health bodies in their respective countries. The caveat to this is whether the comparative Ct values and LOD are the same for SARS-CoV-2 variants as they are with the wild type, something out of scope in this project.

Short of national restrictions, control of case numbers within a population relies heavily on diagnostic testing capacity. Multipronged approaches to mass testing using a suite of platforms including LFDs will be a huge benefit to public health and SARS-CoV-2 surveillance, provided the tests are deployed correctly and that public health messaging is clear and accurate. The Innova LFD device should “not” be considered as a green-light test—that is to say that a result from an LFD should not allow individuals to participate in an activity that they would otherwise not participate in, if their infective status was not known. The LFD device can provide an individual with an idea of their colonisation state “at that time,” but as shown by the limit of detection, individuals with a low associated N gene Ct (e.g., people at the start or end of an infection) may test negative. However, when used regularly, in the correct red-light fashion, LFDs can be a highly effective tool in reducing overall community burden, with a particular benefit to places of work and study and any other venues that would have a relatively high proportion of attendance despite potential COVID-secure restrictions.

Conclusions

Our data show that the Innova LFD can successfully detect SARS-CoV-2 infection in people with a viral titre above approximately 100 viral copies/ml. However, as determined at our site using the ThermoFisher COVID-19 TaqPath assay, it is incapable of detecting infection at comparable PCR Ct values of 30 and over. These levels of infection are indicative of very early or very late stages of infection, and as such, we would strongly recommend that LFD testing is used to screen people at very regular frequency and that a negative result should not be used to determine that someone is free from SARS-CoV-2 infection.

Raw PCR Ct values for all SARS-CoV-2–positive samples analysed in the University of Birmingham Pillar 2 testing lab between October 1 and November 5, 2020.

(XLSX)

Click here for additional data file.

29 Jan 2021

Dear Dr. McNally,

Thank you for submitting your manuscript entitled "Validation testing to determine the effectiveness of lateral flow testing for asymptomatic SARS-CoV-2 detection in low prevalence settings" for consideration as a Research Article by PLOS Biology.

Your manuscript has now been evaluated by the PLOS Biology editorial staff and I am writing to let you know that we would like to send your submission out for external peer review. Given the timeliness of the topic and also in your best interest, it would not be productive to invite a major revision, but if reviewers consider that the work almost ready to be published, we would be happy to move forward.

However, before we can send your manuscript to reviewers, we need you to complete your submission by providing the metadata that is required for full assessment. To this end, please login to Editorial Manager where you will find the paper in the 'Submissions Needing Revisions' folder on your homepage. Please click 'Revise Submission' from the Action Links and complete all additional questions in the submission questionnaire.

Please re-submit your manuscript within two working days, i.e. by Jan 31 2021 11:59PM.

Login to Editorial Manager here: https://www.editorialmanager.com/pbiology

During resubmission, you will be invited to opt-in to posting your pre-review manuscript as a bioRxiv preprint. Visit http://journals.plos.org/plosbiology/s/preprints for full details. If you consent to posting your current manuscript as a preprint, please upload a single Preprint PDF when you re-submit.

Once your full submission is complete, your paper will undergo a series of checks in preparation for peer review. Once your manuscript has passed all checks it will be sent out for review.

Given the disruptions resulting from the ongoing COVID-19 pandemic, please expect delays in the editorial process. We apologise in advance for any inconvenience caused and will do our best to minimize impact as far as possible.

Feel free to email us at plosbiology@plos.org if you have any queries relating to your submission.

Kind regards,

Paula

---

Paula Jauregui, PhD,

Associate Editor

PLOS Biology

12 Mar 2021

Dear Dr. McNally,

Thank you very much for submitting your manuscript "Validation testing to determine the effectiveness of lateral flow testing for asymptomatic SARS-CoV-2 detection in low prevalence settings" for consideration as a Research Article by PLOS Biology. As with all papers reviewed by the journal, yours was evaluated by the PLOS Biology editors as well as by an Academic Editor with relevant expertise and by independent reviewers. The reviewers appreciated the attention to an important topic.

Based on the reviews (below), we will probably accept this manuscript for publication, provided you satisfactorily address the points raised by the reviewers. Please also make sure to address the following data and other policy-related requests.

In particular, reviewer #1's major concern is the lack of empirical validation of the presence of infectious virus in the tested samples. We understand that this may not be possible to do retrospectively, but you should then discuss the implications and limitations of this study. For this issue, the Academic Editor also recommends to find in the literature examples of measurements of the same samples of SARS-CoV-2 for infectivity and for the levels of RNA as measured by PCR. Both reviewers also agree that you should add a discussion, add references, and provide clarifications throughout the manuscript. Reviewer #1 also says that it is critical to determine the LOD of both devices in parallel with the same panel to make sure the PCR threshold is accurately estimated, as this is crucial for the theoretical evaluation and the overall conclusions of the study. Please address the reviewers' concerns.

In order to make the manuscript more accessible to a general readership, we think that it would help adding a few sentences at the end of the abstract to explain the implications that Lateral Flow Devices do not detect infections presenting with PCR Ct values over 29-30, what % of this population was detected and undetected, and if any recommendation can be gleaned from it. We also recommend a more declarative title modifying it to include an idea of the conclusions of the work.

Please also provide a blurb which will be included in our weekly and monthly Electronic Table of Contents, sent out to readers of PLOS Biology, and may be used to promote your article in social media. The blurb should be about 30-40 words long and is subject to editorial changes. It should, without exaggeration, entice people to read your manuscript. It should not be redundant with the title and should not contain acronyms or abbreviations.

Please also address the following data and other policy-related requests.

ETHICS STATEMENT:

-- Please include information about the form of consent (written/oral) given for research involving human participants. All research involving human participants must have been approved by the authors' Institutional Review Board (IRB) or an equivalent committee, and all clinical investigation must have been conducted according to the principles expressed in the Declaration of Helsinki.

DATA POLICY:

You may be aware of the PLOS Data Policy, which requires that all data be made available without restriction: http://journals.plos.org/plosbiology/s/data-availability. For more information, please also see this editorial: http://dx.doi.org/10.1371/journal.pbio.1001797

Note that we do not require all raw data. Rather, we ask that all individual quantitative observations that underlie the data summarized in the figures and results of your paper be made available in one of the following forms:

1) Supplementary files (e.g., excel). Please ensure that all data files are uploaded as 'Supporting Information' and are invariably referred to (in the manuscript, figure legends, and the Description field when uploading your files) using the following format verbatim: S1 Data, S2 Data, etc. Multiple panels of a single or even several figures can be included as multiple sheets in one excel file that is saved using exactly the following convention: S1_Data.xlsx (using an underscore).

2) Deposition in a publicly available repository. Please also provide the accession code or a reviewer link so that we may view your data before publication.

Regardless of the method selected, please ensure that you provide the individual numerical values that underlie the summary data displayed in the following figure panels as they are essential for readers to assess your analysis and to reproduce it: Figure 1. We note that you have provided the data, but please note that the numerical data provided should include all replicates AND the way in which the plotted mean and errors were derived (it should not present only the mean/average values).

Please also ensure that figure legends in your manuscript include information on where the underlying data can be found, and ensure your supplemental data file/s has a legend.

Please ensure that your Data Statement in the submission system accurately describes where your data can be found.

As you address these items, please take this last chance to review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the cover letter that accompanies your revised manuscript.

We expect to receive your revised manuscript within two weeks.

To submit your revision, please go to https://www.editorialmanager.com/pbiology/ and log in as an Author. Click the link labelled 'Submissions Needing Revision' to find your submission record. Your revised submission must include the following:

-  a cover letter that should detail your responses to any editorial requests, if applicable, and whether changes have been made to the reference list

-  a Response to Reviewers file that provides a detailed response to the reviewers' comments (if applicable)

-  a track-changes file indicating any changes that you have made to the manuscript.

NOTE: If Supporting Information files are included with your article, note that these are not copyedited and will be published as they are submitted. Please ensure that these files are legible and of high quality (at least 300 dpi) in an easily accessible file format. For this reason, please be aware that any references listed in an SI file will not be indexed. For more information, see our Supporting Information guidelines:

https://journals.plos.org/plosbiology/s/supporting-information

*Published Peer Review History*

Please note that you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out. Please see here for more details:

https://blogs.plos.org/plos/2019/05/plos-journals-now-open-for-published-peer-review/

*Early Version*

Please note that an uncorrected proof of your manuscript will be published online ahead of the final version, unless you opted out when submitting your manuscript. If, for any reason, you do not want an earlier version of your manuscript published online, uncheck the box. Should you, your institution's press office or the journal office choose to press release your paper, you will automatically be opted out of early publication. We ask that you notify us as soon as possible if you or your institution is planning to press release the article.

*Protocols deposition*

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Please do not hesitate to contact me should you have any questions.

Sincerely,

Paula

---

Paula Jauregui, PhD,

Associate Editor,

pjaureguionieva@plos.org,

PLOS Biology

------------------------------------------------------------------------

Reviewer remarks:

Reviewer #1: SARS-CoV-2 testing optimization.

Reviewer #2: Diagnosis and surveillance of communicable diseases.

Reviewer #1: Ferguson et al. compare the effectiveness of a lateral flow device with PCR for large-scale screening of asymptomatic individuals in the UK. Test results obtained with the LFD and PCR were compared, and findings were extrapolated to estimate how many false negative results would theoretically have been obtained with the LFD in comparison to PCR. This is an important study providing empirical data on performance of LFD in comparison to PCR. However, I think there is more potential in this work if the authors would have validated their findings with infectivity assay and by placing their findings in a broader context by discussing the implications and limitations of the current study. Moreover, the limit of detection experiment needs to be clarified.

Major

1. My major concern regarding this study is the lack of empirical validation of the presence of infectious virus in the tested samples. The authors conclude that the LFD could not detect infections with Ct values over Ct 29-30. Based on previous studies we know that this is in line with the typical threshold of samples containing infectious virus, but it would have been much stronger if the authors could provide this evidence. Ct values can differ drastically between labs and different PCR platforms, and therefore it would be preferable to empirically determine the correlation between infectivity and Ct values for this particular study. I understand that these experiments are very challenging and that they might not be possible retrospectively, but at the minimum the authors should discuss the true implications and limitations of this study, which is currently lacking in the manuscript. Particularly, a discussion on the application of LFD for SARS-CoV-2 screening is needed as it can still be used as an effective tool despite the lower sensitivity in comparison to PCR. Please cite previous studies accordingly throughout the manuscript.

2. The LOD experiments require some clarifications. The authors tested the Qnostics Q-panel to estimate the Ct value that would match the LOD of the LFD (100 copies/mL). However, it seems striking to me that not all 3 PCR targets were able to consistently detect the panel at 100 copies/mL, while the LOD for the PCR should be much more sensitive than the LFD. It looks like the panel consists of inactivated virus, but I wonder if the reported concentrations are for PFU/mL or RNA copies/mL. This would make a huge difference as the authors report in the introduction that the LOD of the LFD is 100 PFU/mL or 100.000 RNA copies/mL. It would be critical to determine the LOD of both devices in parallel with the same panel to make sure the PCR threshold is accurately estimated, as this is crucial for the theoretical evaluation and the overall conclusions of the study. Please also clarify whether the LOD was tested in 3 (main text) or 5 (legend table 3) replicates.

3. I would suggest to extend the introduction by (1) describing background on the use and performance of lateral flow tests for SARS-CoV-2 screening of asymptomatic individuals (key publications are missing), (2) stating the overall aims of the study more clearly at the end, (3) summarizing the main findings.

4. The methods currently lack (1) an ethics statement, and (2) brief description to the Innova, RNA extraction, and TaqPath protocols.

Minor

1. Title suggestion: Validation of the effectiveness of the Innova lateral flow device for asymptomatic SARS-CoV-2 detection in low prevalence settings. Perhaps even making the title more specific by replacing "effectiveness" by "sensitivity" or another more specific term.

2. Line 91: Please specify how much the sample was diluted by adding 500 uL of water. Was this dilution factor taken into account when comparing LFD and PCR results?

3. Line 173: please rephrase

4. Fig1: remove zeros after decimal on Y-axis. X-axis label should read "number of tested samples"

5. Line 35 and Line 190: Ct values can differ significantly between different labs and PCR platforms. Please specify that the observed Ct value threshold is specific for the used PCR platform.

Reviewer #2: In this article, Ferguson et al. provide useful information on the clinical performance of a widely used LFD in the UK, the Innova LFD. The article was clear and well-written, but would benefit from additional methodological detail, and a Discussion section. Comments are below:

Introduction

1. The first sentence needs a reference.

2. Can the authors provide some more context about why the Innova LFD was 'principal' in being rolled out? This isn't immediately obvious for a non-UK audience.

3. 'Data from the city council shows that 71,684 LFD tests were performed alongside 51,855 gold-standard PCR tests' - this makes it sounds like tests were performed at the same time in the same individuals - can the authors clarify the number of individuals tested. Not clear from this sentence.

4. Lines 49-50 - needs ref.

5. Line 55 - what is the FALCON trial?

6. Lines 56-57 needs references.

7. Lines 60-61 - why can't the authors access this information - bit speculative otherwise.

Methods

8. Lines 79-80 - what was the result recording app? More detail required here.

9. Please provide information on what 'pillar' laboratories are - again, not clear to a non-UK audience.

10. Line 86: Why were 90 samples selected - was this figure derived statistically?

11. Line 86: How do the authors know that the solution is saline? Please reference the Innova IFU (I'm aware that other LFD solutions are not saline)

12. Line 99 - should read 720/7187

Results

13. Line 119: please provide criteria for deeming a sample void.

14. Line 144: RT-PR should be RT-PCR

15. Line 159: 'From our data the Lateral Flow Device test yield is 2.8 per 10,000 (0.3 to 10 per 10,000).' - not clear what the units here are? Tests?

Conclusions

16 Line 190: 'However it is incapable of detecting infection at comparable PCR Ct values of 30 and over.' - this needs qualifying to say that this is only based on an analysis using the TaqPath assay - it may be different with other assays.

I was a bit surprised not to see a Discussion here - there are obviously lots of facets to discuss (implications of false pos/neg; logistical roll out of tests; impact of variants of concern on LFD detection; frequency of testing). The authors should include a broader discussion of their findings here - not enough just to have a brief concussion section.

29 Mar 2021

Submitted filename: Reviewers_comments.docx

Click here for additional data file.

1 Apr 2021

Dear Dr. McNally,

On behalf of my colleagues and the Academic Editor, Bill Sugden, I am pleased to say that we can in principle offer to publish your Research Article "Validation testing to determine the sensitivity of lateral flow testing for asymptomatic SARS-CoV-2 detection in low prevalence settings: testing frequency and public health messaging is key." in PLOS Biology, provided you address any remaining formatting and reporting issues. Please ensure that the figure legends in your manuscript include information on where the underlying data can be found.

The rest of the formatting and reporting issues will be detailed in an email that will follow this letter and that you will usually receive within 2-3 business days, during which time no action is required from you. Please note that we will not be able to formally accept your manuscript and schedule it for publication until you have made the required changes.

Please take a minute to log into Editorial Manager at http://www.editorialmanager.com/pbiology/, click the "Update My Information" link at the top of the page, and update your user information to ensure an efficient production process.

PRESS

We frequently collaborate with press offices. If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximise its impact. If the press office is planning to promote your findings, we would be grateful if they could coordinate with biologypress@plos.org. If you have not yet opted out of the early version process, we ask that you notify us immediately of any press plans so that we may do so on your behalf.

We also ask that you take this opportunity to read our Embargo Policy regarding the discussion, promotion and media coverage of work that is yet to be published by PLOS. As your manuscript is not yet published, it is bound by the conditions of our Embargo Policy. Please be aware that this policy is in place both to ensure that any press coverage of your article is fully substantiated and to provide a direct link between such coverage and the published work. For full details of our Embargo Policy, please visit http://www.plos.org/about/media-inquiries/embargo-policy/.

Thank you again for supporting Open Access publishing. We look forward to publishing your paper in PLOS Biology.

Sincerely,

Paula

---

Paula Jauregui, PhD

Associate Editor

PLOS Biology