Sensitivity of RT-PCR testing of upper respiratory tract samples for SARS-CoV-2 in hospitalised patients: a retrospective cohort study.

Background: This study aimed to determine the sensitivity and specificity of reverse transcription PCR (RT-PCR) testing of upper respiratory tract samples from hospitalised patients with coronavirus disease 2019 (COVID-19), compared to the gold standard of a clinical diagnosis.
Methods: All RT-PCR testing for severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) in NHS Lothian, Scotland, United Kingdom between the 7 th of February and 19 th April 2020 (inclusive) was reviewed, and hospitalised patients were identified. All upper respiratory tract RT-PCR tests were analysed for each patient to determine the sequence of negative and positive results. For those who were tested twice or more but never received a positive result, case records were reviewed, and a clinical diagnosis of COVID-19 allocated based on clinical features, discharge diagnosis, and radiology and haematology results. For those who had a negative RT-PCR test but a clinical diagnosis of COVID-19, respiratory samples were retested using a multiplex respiratory panel, a second SARS-CoV-2 RT-PCR assay, and a human RNase P control.
Results: Compared to the gold standard of a clinical diagnosis of COVID-19, the sensitivity of a single upper respiratory tract RT-PCR for COVID-19 was 82.2% (95% confidence interval 79.0-85.1%).   The sensitivity of two upper respiratory tract RT-PCR tests increased sensitivity to 90.6% (CI 88.0-92.7%). A further 2.2% and 0.9% of patients who received a clinical diagnosis of COVID-19 were positive on a third and fourth test; this may be an underestimate of the value of further testing as the majority of patients 93.0% (2999/3226) only had one or two RT-PCR tests. Conclusions: The sensitivity of a single RT-PCR test of upper respiratory tract samples in hospitalised patients is 82.2%. Sensitivity increases to 90.6% when patients are tested twice.  A proportion of cases with clinically defined COVID-19 never test positive on RT-PCR despite repeat testing. Copyright:

Introduction

The coronavirus disease 2019 (COVID-19) pandemic in Europe has already caused significant morbidity and mortality, not least within the United Kingdom. As well as causing large numbers of community-acquired cases, severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2) has also been shown to circulate effectively within hospitals , necessitating the creation of COVID-19 specific areas. An estimate of the sensitivity of reverse transcription PCR (RT-PCR) testing for SARS-CoV-2 is therefore critical. Overestimation of sensitivity by clinical staff, and a lack of use of testing results in combination with clinical features of their presentation, may lead to patients with disease being incorrectly diagnosed, and placed in non-COVID-19 areas with the subsequent risk of infection to others; underestimation of the sensitivity by clinical staff may lead to patients who are SARS-CoV-2 negative being erroneously placed in COVID-19 areas.

A recent meta-analysis estimates the sensitivity of reverse transcription polymerase chain reaction (RT-PCR) testing of upper respiratory tract samples as 89%, but this meta-analysis, and a subsequent one highlight a number of limitations in the literature. These include small sample size (<100 patients with COVID-19) , reliance on RT-PCR itself as the gold standard for diagnosis , use of computed tomography (CT) scans rather than clinical criteria as a gold standard for the diagnosis of COVID-19 , and absence of comprehensive RT-PCR testing for all included patients . Finally, only a single study to our knowledge has examined the cumulative sensitivity of repeat testing for SARS-CoV-2 . Here we examine in a large, comprehensive dataset the sensitivity of RT-PCR testing of upper respiratory tract specimens for COVID-19, compared to the gold standard of clinical diagnosis.

Methods

Data source

All RT-PCR testing conducted for SARS-CoV-2 in NHS Lothian between the 7 th of February and 19 th April 2020 (inclusive) was reviewed. NHS Lothian covers a population of 907,580 people and during the period of the study the Royal Infirmary of Edinburgh was the only regional centre conducting SARS-CoV-2 testing. Hospitalised patients were identified by cross-matching patient identification numbers against the NHS Lothian TrakCare Patient Clinical Management System database. In this study we comply with the principles of the STARD reporting guidelines for diagnostic accuracy studies.

Data collection

For the analysis of all patients (community and hospitalised), RT-PCR tests for SARS-CoV-2 conducted by the Royal Infirmary of Edinburgh virology laboratory (the only laboratory in the region conducting testing at this point) were identified.

For hospitalised patients, a record of samples sent for RT-PCR testing in the study period were identified, and only unambiguous positive or negative results, as authorised by laboratory staff, selected. Testing patterns were allocated for each patient, determining the sequence of RT-PCR tests and whether each test had yielded a negative or positive result ( Table 1).

Table 1.

Classification of test results.

Description	Classification
Single negative test	Classified as a true negative.
Initial positive test, with or without subsequent testing.	Classified as a true positive. Clinical records reviewed to confirm that met case definition.
More than one negative test, no positive test result at any point	Clinical records reviewed to identify whether should be classified as true negative, or potential false negative based on clinical diagnosis.
A series of one or more negative tests followed by a positive test, with or without subsequent testing.	Clinical records reviewed to identify whether a single or multiple clinical presentations. If two distinct clinical presentations with independent testing, treated as discrete episodes, and test classified as a true positive. If a single episode, test classified as a false negative.

Case definitions

Hospitalised patients with a single negative test result were classified as a true negative, as clinical guidelines in place at the time specified that if there was clinical suspicion of COVID-19, an RT-PCR test should be repeated if the first test was negative. For those who initially tested negative on one or more occasions and then positive, case records were reviewed to determine whether this represented two discrete presentations or the same presentation. If they were classified as two distinct presentations, the negative followed by positive test was treated as a single positive test.

For those tested twice or more but who never received a positive result from RT-PCR testing, case records were reviewed, and a clinical diagnosis of COVID-19 was allocated based on a discharge diagnosis from the clinical team (or death certificate documentation) and clinical review. Clinical features, radiology reports and haematology results were reviewed. A positive clinical diagnosis was based on European Centres for Disease Control (ECDC) and World Health Organisation (WHO) criteria . Based on previously published studies , cases were judged to be more likely to represent COVID if a chest X-ray showed patchy bilateral infiltrative changes, or a CT scan showed ground glass changes and if there was lymphopaenia in the presence of a normal neutrophil count . Case records were reviewed by two clinicians (EW and TCW); if a consensus decision could not be reached, the case records were reviewed by a third clinician (DW) to arrive at a final clinical diagnosis. For patients classified as a possible false negative, their initial respiratory sample was retested using a multiplex respiratory panel, a second SARS-CoV-2 RT-PCR assay on the SeeGene platform as detailed below, and a human RNase P RT-PCR.

For patients who tested positive on their initial test, case records were reviewed to ensure they met the clinical criteria for COVID-19, as described above. As for those who tested negative on two or more occasions, a positive RT-PCR case was not part of the diagnostic criteria for COVID-19 infection. If they did not meet these clinical criteria, the samples IDs were matched against samples which had undergone whole genome sequencing (WGS) as part of the COVID-19 Genomics UK sequencing consortium . If WGS had been completed successfully for a sample, this was assumed to represent a true positive. For those that had not, RT-PCR re-testing was conducted using the SeeGene platform as detailed below.

Laboratory methods

Samples were collected and added to viral transport media (Remel MicroTest M4RT). A volume of 110 µL of eluate containing purified RNA was obtained following automated extraction carried out on the NucliSENS® easyMag® (bioMérieux) using an ‘off-board’ extraction where 200 µL of the sample was added to 2 ml of easyMAG lysis buffer. Overall, 94.0% (5418/5763) of tests on hospitalised patients were conducted using a modified in-house RT-PCR (Drosten, Eurosurveillance ), 5.8% (337/5763) were conducted using the Allplex™2019-nCoV Assay from SeeGene (Seoul, South Korea), and 0.15% (8/5763) using the Abbott RealTime SARS-CoV-2 assay (Des Plaines, IL), with cut-off for diagnosis a threshold cycle (Ct) of 40 or less.

Further characterisation of possible false negatives

The Luminex Panel NxTAG® Respiratory Pathogen Panel (Texas, United States) was used to re-test the original extracted RNA for suspected false negatives (cases which met the clinical case criteria but had negative RT-PCR testing). Multiplex real-time PCRs were carried out on positive extracts using the ABI real-time system, model 7500 (Applied Biosystems, Warrington, United Kingdom), as part of routine testing using assays developed in-house and/or adapted from published methods . The same samples were also re-tested using the Allplex™2019-nCoV SeeGene Assay, and using a human RNase P control . For samples that tested positive using the SeeGene assay, Ct values for human RNase P were compared to negative results using a Welch two-sample t-test in R version 3.4.1 and plotted using GraphPad Prism version 6.04 for Windows (GraphPad Software, La Jolla California USA).

For patients who tested positive for a new respiratory pathogen, the case records were reviewed to ascertain whether the diagnosis was best explained by SARS-CoV-2 infection or the subsequently identified respiratory pathogen. Convalescent serology samples (>14 days after onset of symptoms), if available, were analysed using the Abbott SARS-CoV-2 IgG assay on the Abbott Architect platform .

Statistical analyses

The sensitivity was calculated as the proportion of true positives detected on initial testing and re-testing of suspected false negatives, divided by the number of true positives added to convincing false negatives, as estimated on the basis of further respiratory testing and serology testing. The specificity was calculated by dividing true negatives by the number of true negatives added to those judged to be false positives, on the basis of repeat RT-PCR retesting. The positive predictive value was determined by dividing the number of true positive by the sum of the true positives and false positives. The negative predictive value was calculated by dividing the number of true negatives by the sum of the true negatives and false negatives. Confidence intervals for these estimates were calculated using a two-sided exact binomial test with a confidence level of 0.95, implemented in R .

Ethics statement

As part of the study protocol, specimens and associated clinical data were collected and anonymized before additional molecular/serological testing in accordance with local ethical approval (South East Scotland Scottish Academic Health Sciences Collaboration Human Annotated BioResource reference no. 10/S1402/33). As the study formed part of a service evaluation, with no publication of patient identifiable information, the need for informed consent was waived by the local Caldicott Guardian.

An earlier version of this article can be found on medRxiv (DOI: https://doi.org/10.1101/2020.06.19.20135756).

Results

A total of 10,601 RT-PCR tests with unambiguous results for SARS-CoV-2 for 8311 patients were conducted on upper respiratory tract specimens by the Royal Infirmary of Edinburgh laboratories between the 7 th of February and the 19 th of April 2020 on patients for whom data was available. “In addition to these, there were 37 tests with ambiguous or indeterminate results; of these 25 were in hospitalised patients. These results were not included in our analysis.” These tests included community testing for patients who were never admitted to hospital, and testing for patients outside NHS Lothian for Boards that did not perform their own SARS-CoV-2 testing. From this testing, 1667 patients received a positive result for SARS-CoV-2 testing ( Table 2). The overall sensitivity of an initial upper respiratory tract RT-PCR test for the whole cohort (using a gold standard of an eventual molecular diagnosis of SARS-CoV-2 on upper respiratory tract RT-PCR) is 91.8%, rising to 98.4% after 2 tests.

Table 2.

Summary of testing for all patients.

Testing pattern	Number	% all patients	% of positive patients
Single negative test	5665	68.1	NA
More than 1 negative test	979	11.8	NA
Single positive test	1531	18.4	91.8
Initial negative test followed by positive test	110	1.3	6.6
Positive test after two or more negative tests	26	0.3	1.6

Testing for other respiratory pathogens

Of the total cohort, 3226 patients were hospitalised in NHS Lothian. The data analysis for these patients is summarised in the flowchart in Figure 1. In total, 73 patients received a clinical diagnosis of COVID-19 but did not receive a positive RT-PCR result at the time. The RNA extract used for the initial SARS-CoV-2 RT-PCR was retested for common respiratory pathogens using the Luminex assay. Out of the 73, nine (12.3%) tested positive for a respiratory pathogen. On clinical review, all nine cases were judged to be better explained by this new diagnosis rather than COVID-19 ( Table 3).

Figure 1.

Flowchart for patients undergoing upper respiratory tract RT-PCR testing in NHS Lothian.

URT: upper respiratory tract; RT-PCR: reverse transcription polymerase chain reaction; WGS: whole genome sequencing; COG-UK: COVID-19 Genomics Consortium.

Table 3.

Positive results for other respiratory viruses on re-testing of initial sample.

Respiratory pathogen	Number of cases
Influenza B	3
Human rhinovirus/enterovirus	2
Parainfluenza virus 1	1
Parainfluenza virus 3	1
Parainfluenza virus 4A	1
Human coronavirus NL63	1
Total	9

Retesting with the Seegene assay

Retesting of the remaining 64 samples from suspected false negative cases with the Seegene assay for SARS-CoV-2 showed 27 (42.2%) of these were positive. Of the 37 remaining samples that neither tested positive for a respiratory pathogen nor for SARS-CoV-2 on repeat testing, all showed a positive result for human RNase P. Comparing Ct values for human RNase P for the samples that tested positive compared to those that tested negative (for SARS-CoV-2 on the Seegene assay) showed no difference using a Welch two sample t-test (p=0.49, Figure 3. The flowchart in Figure 2 summarises the results for the patients with a clinical assessment of COVID-19 but negative initial RT-PCR testing. For an initial test, the sensitivity of RT-PCR for SARS-CoV-2 infection was 82.2% (95% confidence interval 79.0–85.1%) with a specificity of 100% (CI 99.9–100%). The positive predictive value of an initial test was 100%; the negative predictive value of an initial test was 95.7% ( Table 4).

Figure 3.

Comparison of Ct values for human RNase P in URT sample RT-PCR tests which were positive or negative for SARS-CoV-2.

Mean and standard deviation shown, p= 0.49 using Welch two sample t-test.

Figure 2.

Flowchart for patients meeting clinical diagnosis of COVID-19 but with negative upper respiratory tract RT-PCR testing.

Table 4.

A 2 x 2 contingency table to calculate sensitivity and specificity of URT RT-PCR for SARS-CoV-2 detection on initial testing.

		COVID-19 (Clinical assessment & re-testing)
		Positive	Negative
Test (RT-PCR)	Positive	531	0
	Negative	115	2580

Repeat testing

Sensitivity increased to 90.6% (CI 88.0-92.7%) after two consecutive tests ( Table 5), with a specificity of 100% (CI 99.9–100%). Increasing to three tests captured an additional 14/646 (2.2%) patients, and up to four tests an additional 6/646 (0.9%). This is a potential underestimate, as in this cohort there were 20 patients with a clinical diagnosis of COVID-19 who were tested twice with consecutive negative results, who might have yielded a positive result on a third test. The positive predictive value of two tests was 100%, and the negative predictive value 97.7%.

Table 5.

Sensitivity of 2 URT RT-PCR tests for the diagnosis of COVID-19.

		COVID-19 (Clinical assessment & re-testing)
		Positive	Negative
Test (RT-PCR)	Positive	585	0
	Negative	61	2580

Lower respiratory tract samples

We examined data for a subset of 67 patients >16 years of age admitted to an Intensive Care Unit in NHS Lothian from the 6 th March until the 5 th of April 2020 with a discharge diagnosis of COVID-19. All tested positive on upper or lower respiratory tract RT-PCR testing. The sensitivity of an initial RT-PCR test in this cohort was 76.1% (51/67 positive, CI 64.1–85.7%). After two RT-PCR tests, sensitivity increased to 89.5% (60/67 positive, CI 79.7–95.7%). Four patients never tested positive on URT RT-PCR (6.0%). A total of 34 patients had a lower respiratory tract (LRT) sample sent for RT-PCR: the sensitivity of this initial test was higher than that of upper respiratory tract testing at 94.1% (32/34 positive). This dataset, with the extra information offered by the increased availability of LRT specimens, supports the overall findings from the study.

Convalescent serology

Out of the cohort of 64 patients who received a clinical diagnosis of COVID-19 with initial negative testing, and negative testing for other viruses, convalescent serology (>14 days after onset of symptoms) was available for seven patients. Of these, four were positive (57.1%).

Discussion

Summary of principal findings

Here we show, using a comprehensively examined dataset, that the sensitivity of RT-PCR testing of upper respiratory tract specimens for the diagnosis of COVID-19 is 82.2% on initial testing, and 90.6% after two consecutive tests. Subsequent tests showed a small increase in diagnostic yield (2.2% for three tests and a further 0.9% for four tests), although this may represent an underestimate, as a number of patients given a diagnosis of COVID-19 based on clinical criteria were only tested twice.

Findings of the present study in light of what has been published before

A previous meta-analysis gives a pooled sensitivity for RT-PCR of 89% (CI 81–94%) for the diagnosis of COVID-19 ; our results sit at the lower range of this estimate, but with overlapping confidence intervals. As highlighted in the introduction, the included studies suffer from a number of limitations including reliance on RT-PCR itself as the diagnostic gold standard, which would lead to an increase in the estimated sensitivity. We are not aware of any studies which have used a clinical diagnosis of COVID-19 against which to assess the sensitivity of RT-PCR. Here we show that the sensitivity of an initial test is lower than reported in this meta-analysis, but that the chance of a false negative result (17.8%) is lower than the 29% estimated in a subsequent meta-analysis using a subset of studies included in 2. These widely varying estimates highlight the importance of more data to inform our understanding of the strengths and weaknesses of RT-PCR testing.

Strengths and limitations

The strengths of the study include the large dataset of both COVID-19 positive and negative patients, and extensive further testing to rule out false negative RT-PCR results and alternative diagnoses in those patients given a clinical diagnosis of COVID-19. We also studied whether inadequate sampling might be a possible explanation for false negatives. However in a cohort of 37 possible false negatives all samples had detectable RT-PCR for human RNase P, with no difference between this group and those that tested positive for SARS-CoV-2, showing that this was not a factor in determining the sensitivity of RT-PCR in this population.

A limitation of the study is that the WHO/ECDC case definition of COVID-19 is likely to be highly sensitive but have low specificity. This means that a number of the cases we identified as potential false negatives could in fact represent other case presentations (a false positive in terms of the clinical diagnosis), and thus underestimate the sensitivity of the assay. This interpretation is supported by the findings from serology, where four out of seven patients who met the clinical case criteria and had a convalescent serology sample had a positive serological test. Conversely, we did not examine the case records of the 1837 patients who tested negative on a single occasion, some of whom are likely to have received a clinical diagnosis of COVID-19, or may have had atypical COVID-19 disease. An increased number of false negatives would lead to a decreased sensitivity for the assay and therefore the sensitivity of this assay may be less than what we report.

A final limitation is that this is a retrospective, not prospective study, and that cases were not blinded to RT-PCR COVID-19 status at the point of assessment of whether they met the case criteria.

Meaning of the study and understanding possible mechanisms

The result from our study suggest that there may be a small proportion of patients with SARS-CoV-2 infection who meet the clinical case definition but never test positive on RT-PCR testing. It is possible that, in patients with severe disease, infection is entirely in the LRT, or that by time of presentation in the disease course the virus may only be present at very low levels in the upper respiratory tract ; this is supported by our findings in the ICU cohort, where 6.0% of patients never tested positive on upper respiratory tract RT-PCR.

Implications for practice or policy, and suggestions for future research

Reliance on RT-PCR testing may result in patients with COVID-19 being inappropriately labelled with alternative diagnoses. These possibly infectious patients will subsequently pose a risk to healthcare workers and other patients. A more detailed picture of the sensitivity of RT-PCR testing will be aided by comprehensive serological testing of hospitalised patients with suspected infection.

Data availability

Underlying data

As part of a service evaluation project, this study received local Caldicott Guardian approval but no permission was granted for publication of any patient identifiable information. Therefore, the raw data underlying the analyses has not been made publicly available. Anonymised data will be provided to researchers at accredited institutions who wish to conduct their own analysis or run meta-analyses after consultation with the local Institutional Review Board. Requests for access to the data should be made to Kate Templeton ( kate.templeton@nhslothian.scot.nhs.uk).

I have no further comments. They addressed most of my previous comments.

Is the work clearly and accurately presented and does it cite the current literature?

Partly

If applicable, is the statistical analysis and its interpretation appropriate?

Partly

Are all the source data underlying the results available to ensure full reproducibility?

No

Is the study design appropriate and is the work technically sound?

Partly

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

NA

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

The sensitivity of RTT-PCR for COVID-19 or SARS-CoV-2 infection is an estimate of much debate, so any study providing more information on what the actual sensitivity (and specificity) should be is very much desired. Although I wonder whether this is the study that will really provide us with the right answers, the conclusion ( “Reliance on RT-PCR testing may result in patients with COVID-19 being inappropriately labelled with alternative diagnoses.”) will probably still be true.

Abstract, Results: “The sensitivity of two upper respiratory tract RT-PCR tests increased sensitivity to 90.6%”. Please rephrase. Suggestion: “Using two upper respiratory tract RT-PCR tests increased sensitivity to 90.6%”.

Introduction: “Overestimation of sensitivity may lead to patients with disease being incorrectly diagnosed”. I think it is not necessarily the overestimation leading to misdiagnosis, but rather the sensitivity (and actually the fact that a test results in any false negative) in itself.

The authors state that they comply with the STROBE guidelines for reporting, but as this is a diagnostic accuracy study, they should comply to STARD.

From the methods section it is not direct clear who are the tested population: only hospitalized patients, or also people in the community, or both? Please be clear.

The authors state that they ‘identified’ URT samples. What does this mean? Does this also mean that this is a retrospective study?

Only unambiguous results were used, but this leads to a high risk of bias and I think that at least the total number of ambiguous test results should have been reported. Also, what happened to samples from patients with one or more ambiguous test results. Were these patients excluded, or only the ambiguous samples? And how many of the patients with only one sample also had an ambiguous sample result?

Probably due to the retrospective nature, the assessment of the reference standard/gold standard was not blinded for the index test result (as the way the results were verified depended on whether there were one or more negative test results). This may have led to high risk of bias, especially when the index test (RT-PCR) was part of the clinical diagnosis criteria.

Please make it explicit whether the RT-PCR was part of the clinical diagnosis criteria.

Please make it explicit whether the outcomes and diagnoses were known when the RT-PCR results were assessed.

I was surprised that single negative test results were considered true negative and multiple negative test results were checked extra. To me, two or three negative test results would provide more certainty than only one negative test result.

Please check the estimates of sensitivity and specificity reported in the abstract and in the full text. It seems that they are different. Maybe they are not, but then it may be helpful to explain the numbers a bit more in the full text.

Also, I had difficulty in getting the numbers of tested patients right…Is it correct that over 5000 patients were not included in the study? That is the majority of the eligible patients. Why was this? And could that have led to biased results?

Title versus abstract: the title only mentions sensitivity, while the aim of the study (at least in the abstract) involves both sensitivity and specificity. Please be consistent in this. Also, if the aim of the study was to investigate specificity, why were potential false positives not further followed up? So maybe elucidating the specificity was not the aim of this study after all.

Limitations: the retrospective design and the lack of blinding should also be mentioned as limitations.

General remark: also minor detail. I think that using the abbreviation URT is not necessary and does not improve the readability of the manuscript.

Is the work clearly and accurately presented and does it cite the current literature?

Partly

If applicable, is the statistical analysis and its interpretation appropriate?

Partly

Are all the source data underlying the results available to ensure full reproducibility?

No

Is the study design appropriate and is the work technically sound?

Partly

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Partly

Reviewer Expertise:

Clinical epidemiology; co-author of QUADAS-2, the risk of bias tool for diagnostic accuracy studies.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

The coronavirus disease 2019 (COVID-19) pandemic is a serious global problem and proper diagnosis of individuals is a key step in the control strategy. As stated by the authors, a wrongful diagnosis could either send positive individuals into non-COVID-19 areas where they could be a risk of infection to others or send negative individuals into COVID-19 areas with a risk of infection unto themselves.

Using a large dataset, the authors highlight the importance of (i) not relying on a single method for diagnosis as a gold standard, either clinical or RT-PCR alone, and (ii) RT-PCR repeat testing that markedly improves test sensitivity.

The single negative test classified as a true negative, were these cases that never met a clinical diagnosis of COVID-19 (hence not suspected as a false negative)? Or is it that no other samples were available/collected? Would the edit below bring more clarity?

“Patients with a single negative test result were classified as a true negative

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Yes

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Molecular Epidemiology and Bioinformatics.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.