Performance of formal smell testing and symptom screening for identifying SARS-CoV-2 infection.

BACKGROUND: Altered sense of smell is a commonly reported COVID-19 symptom. The performance of smell testing to identify SARS-CoV-2 infection status is unknown. We measured the ability of formal smell testing to identify SARS-CoV-2 infection and compared its performance with symptom screening.
METHODS: A convenience sample of emergency department patients with COVID-19 symptom screening participated in smell testing using an eight odor Pocket Smell Test (PST). Participants received a SARS-CoV-2 viral PCR test after smell testing and completed a health conditions survey. Descriptive analysis and receiver operating characteristic (ROC) curve models compared the accuracy of smell testing versus symptom screening in identifying SARS-CoV-2 infection.
RESULTS: Two hundred and ninety-five patients completed smell testing and 87 (29.5%) had a positive SARS-CoV-2 PCR test. Twenty-eight of the SARS-CoV-2 positive patients (32.2%) and 49 of the SARS-CoV-2 negative patients (23.6%) reported at least one of seven screening symptoms (OR = 1.54, P = 0.13). SARS-CoV-2 positive patients were more likely to have hyposmia (≤5 correctly identified odors) than SARS-CoV-2 negative patients (56.1% vs. 19.3%, OR = 5.36, P<0.001). Hyposmia was 52.9% (95% CI 41.9%-63.7%) sensitive and 82.7% (95% CI 76.9%-87.6%) specific for SARS-CoV-2 infection. Presence of ≥1 screening symptom was 32.2% (95% CI 22.6%-43.1%) sensitive and 76.4% (70.1%-82.0%) specific for SARS-CoV-2 infection. The ROC curve for smell testing had an area under the curve (AUC) of 0.74 (95% CI 0.67-0.80). The ROC curve for symptom screening had lower discriminatory accuracy for SARS-CoV-2 infection (AUC = 0.55, 95% CI 0.49-0.61, P<0.001) than the smell testing ROC curve.
CONCLUSION: Smell testing was superior to symptom screening for identifying SARS-CoV-2 infection in our study.

Introduction

Mitigating community transmission of SARS-CoV-2 has proven challenging, in part because of the high proportion of infected individuals who are asymptomatic or presymptomatic. Asymptomatic and presymptomatic individuals may cause 40% of new SARS-CoV-2 infections [1, 2], and asymptomatic SARS-CoV-2 infection occurs about 50% of the time [3]. These cases evade the ubiquitous symptom-based screening strategies used by employers, schools, and businesses. Theoretically, symptom-based screening for SARS-CoV-2 infection is only as sensitive as the percentage of infected cases with symptoms. A more sensitive screening tool that also identifies asymptomatic cases could focus clinical testing and quarantine activities and better mitigate community transmission.

A frequently reported symptom by people with confirmed SARS-CoV-2 infection is altered sense of smell or taste. In a cohort of 2.6 million people in the United Kingdom, 65% of participants with a positive SARS-CoV-2 viral test reported a subjective loss of smell making it the most frequently reported symptom [4]. However, people are often unaware that their sense of smell is diminished [5]. Smell testing, an objective method of measuring olfaction, is more likely to identify diminished olfaction than self-report [6]. Approximately 20% of the general population has olfactory dysfunction on objective testing [7]. The discrepancy in self-reported smell alteration compared to measured smell alteration was seen in a study of patients hospitalized with COVID-19 in Iran, where 59 of 60 patients had altered olfaction on smell testing, but only 21 of the 60 reported alteration in smell or taste function [8].

The comparative accuracy of smell testing versus symptom screening to identify whether someone is infected with SARS-CoV-2 is unknown. We hypothesized that smell testing using “scratch and sniff” odor cards is superior to symptom screening in identifying SARS-CoV-2 infection. To test this hypothesis, we prospectively conducted smell testing and symptom screening of ambulatory patients who then received a SARS-CoV-2 viral PCR test to measure the performance of smell testing and symptom screening for identifying SARS-CoV-2 infection.

Methods

Study population

We prospectively enrolled adult patients who sought care at the emergency department of an academic medical center. Our convenience sample included two groups of patients with anticipated SARS-CoV-2 testing: 1) patients with reported COVID-19 exposure or a positive symptom screen at triage (≥1 of the following self-reported symptoms chosen for their prevalence in COVID-19 illness [9]: fever, shortness of breath, cough, chills, sore throat, loss of taste and/or smell, or body aches) or 2) planned hospital admission (all admitted patients tested for SARS-CoV-2 regardless of admitting diagnosis for infection prevention). We excluded patients with an altered level of consciousness and minors. Study enrollment via convenience sampling occurred throughout the week by multiple clinicians and researchers. Participants provided written informed consent prior to study data collection. The study protocol was reviewed and approved by the University of Kentucky Institutional Review Board (protocol #61519).

Sample size

We used Buderer’s formula [10] to estimate the sample size with α = 0.05, a marginal error of 0.1 and the hypothesis that 85% of people with SARS-CoV-2 infection have measurable loss of smell. This hypothesis was based on self-reported loss of smell in 65% of those with SARS-CoV-2 infection [4] and measured alteration in smell in 98% of hospitalized COVID-19 patients [8]. At the time of study design, SARS-CoV-2 test positivity was 5% which yielded a sample size of 980 patients. During study recruitment local SARS-CoV-2 test positivity increased to about 10%, and an interim analysis of our convenience sample showed SARS-CoV-2 positivity of 25%. This increase in disease prevalence reduced the estimated sample size to 196 patients, and we ended study recruitment with 308 patients.

Study procedures and data collection

Prior to study enrollment patients completed hospital protocol-driven SARS-CoV-2 symptom screening. With informed consent, a member of the study team provided a brief survey and a self-administered smell test. The survey collected data on patient demographics, health conditions potentially affecting sense of smell, and self-reported problems with smell. We used the validated National Health and Nutrition Examination Survey (NHANES) pocket smell test (PST) (Sensonics, Inc., Haddon Heights, NJ) a self-administered “scratch and sniff” smell test [11]. Each participant completed versions A and B of the PST, and each version had four distinct odors (listed in Fig 1) to identify from a multiple choice list of smells. SARS-CoV-2 viral testing occurred after smell testing as part of routine clinical care. Study clinicians were blinded to the results of the smell tests. Hospital staff obtained a nasopharyngeal swab for SARS-CoV-2 RT-PCR testing with either the Abbott Alinity m2000 (Abbott Laboratories, Santa Clara, CA) or BD Max (Beckton Dickinson, Franklin Lakes, NJ) platform. RT-PCR testing happened for all patients with a positive symptom screen, close contact with a known COVID-19 case, or planned hospital admission per hospital protocol. Study and clinical data were entered into REDCap, a secure, cloud-based data storage platform.

Fig 1

Smell testing odor discrimination by SARS-CoV-2 infection status.

Blue squares represent the proportion of SARS-CoV-2 positive patients that correctly identified the odor; red diamonds the proportion of SARS-CoV-2 negative patients that identified the odor. 95% confidence intervals for the point estimates are shown with whiskers.

Statistical analysis

We excluded nine study encounters (2.9%) because the patient had previously participated in the smell testing study. We additionally removed four of the remaining 299 participants (1.3%) from the analysis due to incomplete smell testing data. Data for the other variables was complete for the remaining 295 participants. We described our patient sample using means, proportions, and standard deviations. We used chi-squared tests to assess the distribution of patients across categorical variables, t tests for normally distributed continuous variables, Wilcoxon rank-sum test for non-normally distributed variables and used logistic regression and odds ratios to describe associations with our primary outcome, SARS-CoV-2 status, and the classifier variables of odor discrimination and screening symptoms. We classified patients who correctly identified six or more of the eight odors as normosmic and those who correctly identified fewer than 6 odors as hyposmic per the NHANES classification system [5]. We calculated the sensitivity, specificity, and predictive values of smell testing, self-reported symptoms, and measured fever in identifying patients with SARS-CoV-2 infection as determined by viral RT-PCR testing. We used receiver operating characteristic (ROC) curves and calculated the area under the curve (AUC) to compare the performance of six screening approaches for predicting SARS-CoV-2 infection. The models used the following classifiers: 1) number of the eight PST odors accurately identified; 2) presence of hyposmia defined as correctly identifying five or fewer odors correctly on the PST; 3) number of self-reported screening symptoms (out of seven); 4) presence of measured fever (body temperature ≥100.4°F); 5) two-odor (smoke and soap) performance on the PST; and 6) adjusted model that used the eight PST odors adjusted by the covariates age, gender, corticosteroid nasal spray use, measured fever, and cough. We developed the two-odor model by calculating the performance (AUC) of each of the eight PST odors to identify SARS-CoV-2 infection and using the two odors with the best performance (largest AUC). For the adjusted model age and gender were selected a priori and the other covariates were included because they were statistically associated with smell testing performance (corticosteroid nasal spray use) or with SARS-CoV-2 infection (fever and cough). We conducted a ROC subgroup analysis that excluded patients with reported previous positive SARS-CoV-2 test or COVID-19 exposure and a second subgroup analysis of asymptomatic patients to assess the predictive performance of smell testing in asymptomatic patients. To compare the predictive performance of ROC models we used a chi-squared statistic to test for differences in AUC between models. Reported P-values are two-sided, and we considered P<0.05 statistically significant. Author JWK conducted the analyses with STATA version 15.1 (StataCorp, College Station, TX).

Results

From October 16, 2020, to February 15, 2021, 295 unique patients completed smell testing, and 87 (29.5%) of those patients tested positive for SARS-CoV-2 by RT-PCR. Study participants were 52.2% (154/295) female, 83.3% (246/295) white, and had a mean age of 45.6 years (SD = 17.8; IQR 30–59). There were no statistically significant demographic differences between the participants that tested positive for SARS-CoV-2 and the participants that tested negative (Table 1). Seventy-seven (26.1%) participants reported at least one symptom, which included 49 (23.6%) SARS-CoV-2 negative patients and 28 (32.2%) SARS-CoV-2 positive patients (OR = 1.54; P = 0.13). Cough and loss of taste and/or smell were the only symptoms significantly more common in SARS-CoV-2 positive patients (Table 1).

Table 1

Patient demographics and symptoms by SARS-CoV-2 infection status.

	SARS-CoV-2 Negative (N = 208)		SARS-CoV-2 Positive (N = 87)
Characteristic	n	%	n	%	P
Demographic
Age (years; mean, SD)	44.5	17.2	48.1	19.1	0.16
Female	111	53.4%	43	48.3%	0.43
Race					0.11
Asian	0	0.0%	2	2.3%
Black	28	13.5%	17	19.5%
White	178	85.6%	68	77.0%
Multiracial	1	0.5%	0	0.0%
Hispanic	7	3.4%	3	3.5%	0.77
Symptom/exposure screening
Previous positive SARS-CoV-2 test or exposure to someone with COVID-19	8	3.9%	20	23.0%	<0.001
Reported fever	16	7.7%	10	11.5%	0.29
Shortness of breath	25	12.0%	17	19.5%	0.09
Cough	22	10.6%	22	25.3%	0.001
Chills	16	7.7%	11	12.6%	0.18
Sore throat	13	6.3%	9	10.3%	0.22
Loss of taste and/or smell	5	2.4%	9	10.3%	0.003
Body aches	22	10.6%	12	13.8%	0.43
Any reported symptom	49	23.6%	28	32.2%	0.13
Maximum recorded temperature in ED (mean, SD in°F)	98.5	0.94	98.8	1.2	0.01
Recorded fever in ED (T > = 100.4°F (38°C))	8	3.9%	7	8.1%	0.13

ED = emergency department; SD = standard deviation

Most patients (n = 213; 72.2%) had normal smell function (PST score of 6 to 8) on smell testing. Hyposmia was more common in older patients (OR 1.17 per 10 years of age, 95% confidence interval (CI) 1.01–1.35, P = 0.03), patients who reported a history of loss of smell (OR 3.44, 95% CI 1.56–7.61, P = 0.002), and in patients who used nasal sprays (OR 2.12, 95% CI 1.16–3.91, P = 0.02) (S1 Table). We did not observe associations between hyposmia and health conditions that can affect the sense of smell (S1 Table).

Of the 87 patients testing positive for SARS-CoV-2, 71 (81.6%) misidentified at least one odor in the PST. Individual odor identification accuracy by patient SARS-CoV-2 status appears in Fig 1. SARS-CoV-2 positive patients were more likely to have hyposmia (misidentify at least three of the eight odors) as compared to SARS-CoV-2 negative patients (52.3% vs 17.3%; OR = 5.36; 95% CI 3.08–9.43; P<0.001). Hyposmia was 52.9% (95% CI 41.9%-63.7%) sensitive and 82.7% (95% CI 76.9%-87.6%) specific for SARS-CoV-2 infection. Symptom screening (≥1 symptom) was 32.2% sensitive (95% CI 22.6%-43.1%) and 76.4% specific (95% CI 70.1%-82.0%) for SARS-CoV-2 infection, and measured fever had 8.0% sensitivity (95% CI 3.3%-15.9%) and 96.2% specificity (95% CI 92.6%-98.4%) for SARS-CoV-2 infection. The positive predictive values (PPV) and negative predictive values (NPV) for SARS-CoV-2 infection with smell testing, symptom screening, and measured temperature are shown in Table 2 under the scenarios of 1%, 5%, and 10% prevalence of infection.

Table 2

Sensitivity, specificity, and predictive values of smell testing, symptom screening, and body temperature measurement for identifying SARS-CoV-2 infection.

	Smell testing (hyposmia: PST≤5)			Symptom screening (≥1 symptom)			Measured temperature (temp≥100.4°F)
	SARS-CoV-2 Prevalence			SARS-CoV-2 Prevalence			SARS-CoV-2 Prevalence
	1.0%	5.0%	10.0%	1.0%	5.0%	10.0%	1.0%	5.0%	10.0%
PPV	3.0%	13.9%	25.3%	1.4%	6.7%	13.2%	2.1%	9.9%	18.9%
NPV	99.4%	97.1%	94.0%	99.1%	95.5%	91.0%	99.0%	95.2%	90.4%
False positive rate	17.3%			23.6%			3.8%
False positives (per 1,000 screened)	171	164	156	233	224	212	38	37	35
False negative rate	47.1%			67.8%			92.0%
False negatives (per 1,000 screened)	5	24	47	7	34	68	9	46	92
Correctly classified	824	812	797	760	742	720	953	917	873

PST = pocket smell test; PPV = positive predictive value; NPV = negative predictive value

The receiver operating characteristic (ROC) curve for smell testing to classify SARS-CoV-2 infection yielded an area under the curve (AUC) of 0.74 (95% CI 0.67–0.80: Table 3 and Fig 2A). The ROC curve using reported symptoms to classify SARS-CoV-2 infection had an AUC of 0.55 (95% CI 0.49–0.61: Fig 2B). The measured fever ROC curve had an AUC of 0.52 (95% CI 0.49–0.55: Fig 2C).

Table 3

Receiver Operating Characteristic (ROC) models and subgroup analyses for predicting SARS-CoV-2 infection.

Model	Variable(s)	Observations	AUC	95% CI		P*
8-odor	8 odors	295	0.74	0.67	0.80
Hyposmia	Hyposmia (PST≤5)	295	0.68	0.62	0.74	0.003
Symptoms	7 symptoms	295	0.55	0.49	0.61	<0.001
Fever	Body temperature ≥100.4°F	295	0.52	0.49	0.55	<0.001
2-odor	Smoke + soap odors	295	0.75	0.69	0.80	0.60
Adjusted	8 odors, age, gender, corticosteroid nasal spray use, measured fever, cough	295	0.79	0.73	0.85	0.02
Subgroup analyses
No symptoms	8 odors	218	0.76	0.69	0.84
No COVID test/exposure	8 odors	267	0.75	0.68	0.81

AUC = area under the curve; CI = confidence interval; PST = pocket smell test

*As compared to the 8-odor model

Fig 2

Receiver Operating Characteristic (ROC) curves to identify SARS-CoV-2 infection: a) Smell testing with 8 odors; b) Symptom screening with 7 self-reported symptoms; c) Temperature screening for fever (≥100.4°F); d) Smell testing with 2 odors (smoke and soap).

We evaluated two additional ROC models that used smell testing as the classifier. The hyposmia (PST score≤5) model had an AUC of 0.68 (95% CI: 0.62–0.74), and a 2-odor model using the odors smoke and soap (largest independent AUCs for predicting SARS-CoV-2 infection; S1 Fig) had an AUC of 0.75 (95% CI 0.69–0.80: Fig 2D). The sensitivity, specificity, and predictive values of the 2-odor model were similar to the 8-odor model (data not shown). A multi-classifier ROC model adding classifiers significantly associated with SARS-CoV-2 infection (corticosteroid nasal spray use, cough, measured fever) plus age and gender to the 8-odor model marginally increased the AUC to 0.79 (95% CI 0.73–0.85). ROC smell testing subgroup analyses that excluded patients with reported symptoms (AUC = 0.76; 95% CI 0.69–0.84) and patients with reported COVID-19 exposure or previous positive SARS-CoV-2 test (AUC = 0.75; 95% CI 0.68–0.81) performed similarly to the 8-odor ROC model using the full data set.

Discussion

Smell testing identified SARS-CoV-2 infection with greater sensitivity and specificity than COVID-19 symptom screening in our population. The prospective design of our study, which used objective measurements of smell paired with SARS-CoV-2 PCR testing yielded robust estimates of the performance of smell testing in symptomatic and asymptomatic patients. Our study is unique in that it prospectively assessed patient sense of smell prior to ascertaining SARS-CoV-2 status; previous work in this area has tested olfaction in known COVID-19 patients [8, 12, 13] or assessed self-reported alterations in smell in patients with known SARS-CoV-2 infection [4, 14–16], with neither approach supporting an accurate assessment of smell testing as a screening tool for SARS-CoV-2 infection in symptomatic and asymptomatic patients.

An optimal SARS-CoV-2 screening test should perform well regardless of health status and preexisting health conditions. Preexisting health conditions, like allergic rhinitis and chronic sinusitis, that may affect olfaction did not affect patient smell testing performance (S1 Table). The consistent performance of smell testing in our pragmatic cohort suggests that implementation of this screening tool does not need individual-level information (e.g., history of allergic rhinitis or age) to adjust smell testing results nor are more complex predictive models with multiple variables needed.

An ideal SARS-CoV-2 smell screening tool would be of low cost and require minimal time to administer. The 8-odor PST took less than 2 minutes to complete for most participants. Our study suggests that we can further streamline smell testing without affecting the performance of the screening test, as a simplified smell testing model based on the two odors with the largest individual AUCs (smoke and soap) performed similarly to the eight-odor model in identifying SARS-CoV-2 infection. A self-administered two-odor smell test is inexpensive and efficient, making it feasible to screen many people quickly.

Smell testing more accurately identified SARS-CoV-2 infection in the subgroup of asymptomatic patients as compared to the entire sample of symptomatic and asymptomatic patients, although this was not statistically significant. The performance of smell testing in asymptomatic patients suggests its utility as a SARS-CoV-2 screening tool in asymptomatic populations, like employees at congregate work settings. Larremore et al. modeled the effectiveness of smell testing to limit SARS-CoV-2 transmission and found every third day smell testing more effective than weekly RT-PCR testing when smell testing sensitivity was 75%, which is similar to the sensitivity of the two-odor model [17]. Interestingly, formal smell testing in our study was more sensitive than antigen testing in identifying SARS-CoV-2 infection in asymptomatic people. According to a large Cochrane meta-analysis, antigen testing was 58% sensitive and over 99% specific for SARS-CoV-2 when compared to RT-PCR in people without symptoms [18].

Symptom screening to identify SARS-CoV-2 infection was barely better than the flip of a coin in our study, which agrees with Gerkin et al. who found non-olfactory, non-gustatory symptoms unhelpful in identifying SARS-CoV-2 infection [19]. Menni et al. used self-reported symptoms and SARS-CoV-2 test data collected via an app to develop a symptom-based algorithm with 65% sensitivity in identifying SARS-CoV-2 infection [4]. This large study of self-reported data is limited by selection bias (those who chose to enroll via the app are likely not representative of the general population) and measurement bias, in that only a small subset (0.64%) of self-selected participants were tested for SARS-CoV-2. The large untested fraction of participants had a substantially lower frequency of reported symptoms suggesting few asymptomatic participants received SARS-CoV-2 tests. A second, similar app-based study found that self-reported symptoms were 70% sensitive in identifying SARS-CoV-2 infection using Menni et al.’s symptom-based algorithm and had the same shortcomings, namely selection and measurement bias with only 1.1% of the participants reporting SARS-CoV-2 test results [16].

Our study has several limitations. First, we recruited a convenience sample of patients seeking care at the emergency department, and we cannot generalize the findings of our study to other populations, such as asymptomatic people in the general population. However, in a subgroup analysis that excluded patients with reported COVID-19 symptoms, smell testing was as good and potentially better at identifying SARS-CoV-2 infection compared to its performance in the entire sample. Second, some of our participants with positive SARS-CoV-2 PCR tests may have been previously infected with SARS-CoV-2 with prolonged viral shedding and recovery of normal olfaction, which would decrease the calculated sensitivity of smell screening for SARS-CoV-2 infection. A subgroup analysis that excluded patients with self-reported prior SARS-CoV-2 test and/or recent COVID-19 exposure demonstrated similar smell testing performance compared to the primary analysis. Third, the PST provides four multiple choice options for each odor, forcing the test subject to provide a response even when they are unable to determine the odor. This characteristic of the PST inflates PST scores leading to potential underascertainment of hyposmia and decreased sensitivity of smell testing for SARS-CoV-2. Fourth, the epidemiologic context of the pandemic during our study period (e.g., predominant circulating virus variants and vaccine coverage) may influence study results and impact their generalizability to other pandemic contexts.

Conclusions

In conclusion, smell testing performed well in identifying SARS-CoV-2 infection, while symptom screening and measured temperature, which are widely used in many settings, did not reliably discriminate SARS-CoV-2 infection in our population. A subset of two odors from the smell test (smoke and soap) performed similarly in identifying SARS-CoV-2 infection as the full eight-odor test and could be a simple, affordable screening tool. Additional studies on the performance of smell testing in asymptomatic populations, e.g., healthy workers, can validate the use of smell screening to risk stratify people for SARS-CoV-2 clinical testing.

Demographics, health history, symptoms, and SARS-CoV-2 status by smell testing performance.

(DOCX)

Click here for additional data file.

Receiver Operating Characteristic (ROC) curves for predicting SARS-CoV-2 infection by individual odor.

(DOCX)

Click here for additional data file.

(CSV)

Click here for additional data file.

31 Dec 2021

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Muhammad Tarek Abdel Ghafar, M.D

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match.

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

Reviewer #5: Yes

Reviewer #6: Yes

Reviewer #7: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

Reviewer #5: Yes

Reviewer #6: Yes

Reviewer #7: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

Reviewer #5: Yes

Reviewer #6: Yes

Reviewer #7: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

Reviewer #4: Yes

Reviewer #5: Yes

Reviewer #6: Yes

Reviewer #7: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: All in all a sound report discussing utility of the Pocket Smell Test vs. symptom-based screening for SARS-CoV-2. Not an exceptionally innovative or surprising study, but sound, practical, and acceptable pending revisions and statistical review.

Limitations:

-Acknowledged power analysis, but still a single-institution, smaller study, ethnically homogenous population. And only adults as well? Pregnant vs. non-pregnant? Needs to be expanded to larger, more diverse demographics in further work, and limitations should be mentioned.

-Another limitation related to time. The study took place between October 2020 and February 2021. The results may not still apply to a more vaccinated population and/or amidst quite different SARS-CoV-2 strains.

General:

-Why not focus results (in the Abstract and Table 2) moreso on the 2-odor test? Higher sensitivity, lower false negative rate, slightly not significantly higher AUC. Seems a better screening test than 8-odor? The 8-odor test is ~50% sensitive, better than symptom-based screening but far from ideal amidst the pandemic.

-It would be quite interesting to compare PSTs to home COVID-19 tests at this stage of the pandemic. This seems worth mentioning, from a cost aspect as well.

-Given age differences in hyposmia vs. normosmia populations, were the PSTs equally sensitive and specific in both groups, or more false positives in older populations and false negatives in younger populations? This seems worth mentioning if used as a screening tool.

Abstract:

-"Twenty-eight had a SARS-CoV-2 positive patients (31.8%) and 52 SARS-CoV-2 negative patients" miswording?

-"Smell testing is superior to symptom screening for identifying SARS-CoV-2 infection." Perhaps clarifying "In this study" given limitations as above

Introduction:

-Line 84: ambulatory; this study was ED/hospital-based

Methods:

-Line 96: enrollment (line 96)

-Line 133 : out of N=308

-Lines 135, 147: chi-squared

-148: p-values

Results:

-Line 155: sample or populations?

-Table 1: fine for "ED in maximum recorded temperature" since defined below table

Discussion:

-Line 242: et al.

-Line: 243: three day; three-day or every three day?

-Line 246: Cochrane

Reviewer #2: Thank you for giving me the opportunity to review this interesting piece of work which can be implicated in public health. With problems in detecting COVID19 and its diagnosis, such work may present promising directions for public health.

Reading through the abstract gives the reader the impression that this is a half-baked work while after reading the paper the truth is that the abstract falls short of fully capturing and introducing your work. I strongly recommend rewriting the whole abstract to reflect the true level of your work. Simple edits or adding few statements will not do the work.

The methods are written in thorough details.

Table 1: since you are not doing any form of randomization, I think there is no need to compare both groups statistically. This fallacy creates the impression that they were randomized which is not true for your study. I recommend removing the p-values as their interpretation is problematic.

Table 2: the table has so much information that makes it very difficult to read and follow through. I strongly recommend separating the table into two tables.

It is better that p-values to be reported without leading zeros.

I also suggest introducing the benefits/ implications of this test (convenient, accessible, cheap) in the introduction and putting it within the context of public health responses.

Reviewer #3: *There are some ambiguous sentences in the abstract section. For example: “…Twenty-eight had a SARS-CoV-2 positive 49 patients (31.8%) …” And the abstract should be clearer, and if possible, fascinating with parsimony.

* Eligibility criteria are not clearly elucidated; inclusion and exclusion criteria should be detailed. This is particularly important as the study focused in patients visiting the emergency department.

* In the limitation section/discussion part, some points should be raised, including: use of this olfaction-based screening should be deferred in those with reported baseline smell problems or smell loss (diminution).

Reviewer #4: This diagnostic accuracy study recruited 304 COVID suspects for comparing the sensitivities and specificities of symptom screening and smell test against test positivity. It then reported that the sensitivities of 8 – odor PST, 2 – odor test and symptom screen were 53.4%, 72.7% and 31.8% respectively; the specificities were 83.8%, 74.5% and 75.9%, respectively. The AUC for PST was 0.68, which improved to 0.79 for multiple variables. It then concluded that smell testing, particularly the two – item smell test could be a simple, affordable screening tool.

It must be noted that the convenience sample of patients is really made up of two distinct groups: one group consisted of COVID suspects while the second group consisted of patients for admission who may or may not be COVID suspects. While there was no separate descriptions of the symptoms and COVID test results of these two groups of patients, I think that the second group is the group that clinically reflects the general population for COVID screening, wherein the pretest probability of turning positive is lower, compared to the first group.. This may be the reason why the authors conducted subgroup analyses for the 8 – odor PSTs of asymptomatic and unexposed patients. Their AUCs were comparable with the 8 – odor PST for the entire sample which probably meant that the two groups of sampled patients could really be treated as one, although this was not pointed out.

I note that only 2.3% of the 216 COVID negative patients and 10% of the 88 COVID positive patients reported anosmia or ageusia. This does not appear to be consistent with other studies which reported anosmia as the most common symptom. I think this merits some explanation.

I think the authors should explain why the 2 odor test had a higher sensitivity but lower specificity compared to the 8 odor test because this is counter intuitive.

The results section in the abstract is also difficult to understand for me, the third sentence particularly. I think they should stick to the key findings in the abstract.

The authors cut short the study when the prevalence of COVID went up to 25% positivity. How does this affect the applicability of the study in a non-surge setting? Should sensitivity analyses be done?

I also did not note if this research was approved by the ethics committee of their institution.

Reviewer #5: The study is interesting and well conducted expecialyy in the methods. The authors conclude that smell testing was able to identify SARS-CoV-2 infection with a 73% of sensitivity and 75% od specifity even if was not investigated on asymptomatic subjects. However Bianco et al showed that alteration of sense of smell represent a prodromal symptom of infection, they coul add this reference as follow:Bianco, Maria Rita, et al. "Alteration of smell and taste in asymptomatic and symptomatic COVID-19 patients in Sicily, Italy." Ear, Nose & Throat Journal 100.2_suppl (2021): 182S-185S.

Reviewer #6: This study is apt and timely. It stands to contribute significantly to existing body of knowledge. However, the following revisions should be considered;

Title: The may benefit from inclusion of location of the study.

Abstract: The background to the abstract is somewhat over summarized with the objective(s) of the study not stated .

Introduction: The authors should consider differentiating asymptomatic from presymptomatic state of SARS-CoV-2 infections.

Methods:

1.The exclusion criteria used in this study should be stated clearly

2. It is unclear the measures used to mitigate against testing any participant more than once.

3. The authors also need to bring out clearly how the smell test was conducted, were the research clinicians trained on

conducting the smell test?

4. It is unclear if the those with anosmia/smell anomalies from any other conditions were identified and excluded from the study.

5.It appears a test of difference of means (unpaired student t test) was used for the comparison of age as well as temperature between the two categories but the authors presented it as if it was chi square.

6.It appears the age of the study participants in the two groups are skewed, it is imperative that the test of assessment of normality used should be stated otherwise an appropriate non parametric test should be used and stated.

Results: The two additional ROC model used " hyposmia & a 2-odor model" were not mentioned in the methods, it is important that mention is made of it and the rationale for it provided in the methods before stating such in the results.

Additional comment: It is important that the authors bring to light the limitation(s) associated with convenience sampling used particularly with regards to generalization of the findings

Reviewer #7: This is a well executed study and well written paper that examines a question many front line clinicians have asked themselves: how predictive is loss of smell of a diagnosis of COVID-19?

The authors assessed smell using a validated clinically relevant tool.

Couple of suggestions

-would suggest including in table 1 the smell test results frequencies (each one and aggregate).

-Also in table 1 would include variables that are significant in table s1.

-what is the difference between history and symptom of loss of smell, the latter presumably without formal testing? The numbers appear small for both. What happens when both are combined? How good is a model of history and/or symptom of loss of smell/taste plus 2 smell test (or are numbers too small)? How do authors explain the discrepancy between self reported loss of smell and smell test results?

-Table 2 is difficult to follow and should be perhaps split into two separate table or presented in a simplified table?

-a secondary multivariate analysis (logistic regression of loss of smell, perhaps stratified on COVID-19 status?) that includes variables with significant differences in bivariate analysis from table 1 and table s1 could help identify confounders that one has to think about when using this approach.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes: Subah Abderehim Yesuf

Reviewer #4: Yes: Jose Acuin

Reviewer #5: Yes: Eugenia Allegra

Reviewer #6: Yes: Tolulope Olumide Afolaranmi

Reviewer #7: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: Review report smell.docx

Click here for additional data file.

14 Feb 2022

February 14, 2022

Dear Editor and Reviewers,

Thank you for your many helpful comments and questions regarding our manuscript titled Performance of formal smell testing and symptom screening for identifying SARS-CoV-2 infection.

We have provided our responses following your comments. When appropriate we indicate the line numbers where we revised the manuscript.

In response to a concern from Reviewer #6 regarding participants being smell tested more than once (multiple visits to the ED), we reviewed our non-deidentified data set and found nine instances where a patient participated a second time. We excluded those second visits from our analytic data set, reran the analyses, and updated the tables and figures. Results were minimally (non-significantly) changed.

Regards,

James W Keck, MD MPH

Assistant Professor

Department of Family and Community Medicine

University of Kentucky

Comments to the Author Authors’ Response

Reviewer 1

Acknowledged power analysis, but still a single-institution, smaller study, ethnically homogenous population. And only adults as well? Pregnant vs. non-pregnant? Needs to be expanded to larger, more diverse demographics in further work, and limitations should be mentioned.

Authors’ Response

This limitation was acknowledged (lines 273-4). As mentioned in the methods (line 88) this study was limited to adults. Pregnant women were eligible, but we did not enroll any. As shown in table 1, 17% (49/295) participants were non-white and 3% (10/295) were Hispanic.

Another limitation related to time. The study took place between October 2020 and February 2021. The results may not still apply to a more vaccinated population and/or amidst quite different SARS-CoV-2 strains.

Authors’ Response

Added additional limitation statement: “Fourth, the epidemiologic context of the pandemic during our study period (e.g., predominant circulating virus variants and vaccine coverage) may influence study results and impact their generalizability to other pandemic contexts.” (lines 285-288)

Why not focus results (in the Abstract and Table 2) moreso on the 2-odor test? Higher sensitivity, lower false negative rate, slightly not significantly higher AUC. Seems a better screening test than 8-odor? The 8-odor test is ~50% sensitive, better than symptom-based screening but far from ideal amidst the pandemic.

Authors’ Response

We adhered to our study protocol for the analysis which was designed to assess the 8-odor pocket smell test as a diagnostic/screening tool for SARS-CoV-2 infection. As defined in the protocol we individually assessed the discriminatory performance of each of the eight odors and reported those results including a subgroup analysis looking at the two odors with best performance.

It would be quite interesting to compare PSTs to home COVID-19 tests at this stage of the pandemic. This seems worth mentioning, from a cost aspect as well.

Authors’ Response

We agree and the manuscript you reviewed included a comparison of smell testing to the performance of antigen based tests (lines 255-8).

Given age differences in hyposmia vs. normosmia populations, were the PSTs equally sensitive and specific in both groups, or more false positives in older populations and false negatives in younger populations? This seems worth mentioning if used as a screening tool.

Authors’ Response

Although the age distribution of participants with hyposmia was on average 5 years older than those with Normosmia, the PST discriminated SARS-CoV-2 infection equally well across age groups. We ran two sub-analyses that split our sample into younger/older age groups (50th percentile and 75th percentile) and generated ROC models. The AUC was quite similar across groups: 50th percentile 0.71 vs 0.75 (older group) and 75th percentile 0.71 vs 0.77 (older group). 95% CIs were overlapping. We mention in the discussion that the robust performance of the PST across age groups (lines 236-239).

Abstract:

-"Twenty-eight had a SARS-CoV-2 positive patients (31.8%) and 52 SARS-CoV-2 negative patients" miswording?

Authors’ Response: Corrected to read “Twenty-eight of the SARS-CoV-2 positive patients (31.8%)”

-"Smell testing is superior to symptom screening for identifying SARS-CoV-2 infection." Perhaps clarifying "In this study" given limitations as above

Authors’ Response: Added “in our study.”

Introduction:

-Line 84: ambulatory; this study was ED/hospital-based

Authors’ Response: Health care (at least in the USA) received in an emergency room is considered ambulatory care – see this WebMD page as a reference: https://www.webmd.com/health-insurance/terms/ambulatory-patient-services

Methods:

-Line 96: enrollment (line 96)

-Line 133: out of N=308

-Lines 135, 147: chi-squared

-148: p-values

Authors’ Response: Corrections made.

Results:

-Line 155: sample or populations?

-Table 1: fine for "ED in maximum recorded temperature" since defined below table

Authors’ Response: Changed “sample” to “participants. Changed “emergency department” to ED in Table 1.

Discussion:

-Line 242: et al.

-Line: 243: three day; three-day or every three day?

-Line 246: Cochrane

Authors’ Response: Corrections made. Changed to “third day”

Reviewer #2

Reading through the abstract gives the reader the impression that this is a half-baked work while after reading the paper the truth is that the abstract falls short of fully capturing and introducing your work. I strongly recommend rewriting the whole abstract to reflect the true level of your work. Simple edits or adding few statements will not do the work.

Authors’ Response: There were several typos in the results section of the abstract. These have been corrected which should improve the readability of the abstract.

Table 1: since you are not doing any form of randomization, I think there is no need to compare both groups statistically. This fallacy creates the impression that they were randomized which is not true for your study. I recommend removing the p-values as their interpretation is problematic.

Authors’ Response

Because of the study design (assessment of diagnostic tool) we could not use randomization to minimize the risk of confounders. Hence it is important to describe and compare the attributes of the two groups (Sars-CoV-2 positive and negative) to identify potential confounders that would need to be controlled for in the analysis.

Table 2: the table has so much information that makes it very difficult to read and follow through. I strongly recommend separating the table into two tables.

Authors’ Response

To simplify the table we removed the sensitivity and specificity rows as those values are listed in the text. We also removed the “2-odor” screening test because this was a sub-analysis of the smell testing performance. We believe the various performance metric of the screening tests are best compared side-by-side in a single table.

It is better that p-values to be reported without leading zeros.

Authors’ Response

PLOS One journal publishes P-values with leading zeros.

I also suggest introducing the benefits/ implications of this test (convenient, accessible, cheap) in the introduction and putting it within the context of public health responses. We mention these benefits in the discussion (lines 241-44).

Authors’ Response

As the focus of the study was evaluating smell testing discriminatory performance (and not cost or accessibility) we did not discuss these potential attributes in the introduction.

Reviewer #3

There are some ambiguous sentences in the abstract section. For example: “…Twenty-eight had a SARS-CoV-2 positive 49 patients (31.8%) …” And the abstract should be clearer, and if possible, fascinating with parsimony.

Authors’ Response

There were several typos in the results section of the abstract. These have been corrected which should improve the readability of the abstract.

Eligibility criteria are not clearly elucidated; inclusion and exclusion criteria should be detailed. This is particularly important as the study focused in patients visiting the emergency department.

Authors’ Response

Because we wanted a diverse participant sample, we had few inclusion criteria. Those criteria are listed in the Methods-study population section and included 1) adults 2) seeking care at the study hospital with anticipated SARS-CoV-2 testing due to either 3) self-reported COVID-19 symptom on screening or close contact with a known case or 4) planned hospital admission.

In the limitation section/discussion part, some points should be raised, including: use of this olfaction-based screening should be deferred in those with reported baseline smell problems or smell loss (diminution).

Authors’ Response

Given the previously published inconsistency in self-reported versus measured loss/alteration of smell (see intro and references 5 and 8) and a similar finding in our study, we do not believe excluding individuals with self-reported smell problems is useful. To confirm this, we conducted a sub-analysis (n=267) restricted to participants reporting NO history of smell problems. The AUC for the ROC model was 0.72 – slightly less (but not statistically different) than the AUC for the entire sample which included people reporting a history of smell problems. This suggests the 8-odor smell testing screening approach performs similarly when including participants with a reported history of smell problems.

Reviewer #4

It must be noted that the convenience sample of patients is really made up of two distinct groups: one group consisted of COVID suspects while the second group consisted of patients for admission who may or may not be COVID suspects. While there was no separate descriptions of the symptoms and COVID test results of these two groups of patients, I think that the second group is the group that clinically reflects the general population for COVID screening, wherein the pretest probability of turning positive is lower, compared to the first group. This may be the reason why the authors conducted subgroup analyses for the 8 – odor PSTs of asymptomatic and unexposed patients. Their AUCs were comparable with the 8 – odor PST for the entire sample which probably meant that the two groups of sampled patients could really be treated as one, although this was not pointed out.

Authors’ Response

We agree with your comment. We wanted to include asymptomatic patients (not only COVID-19 suspects by symptom screening) to assess the performance of smell testing in both groups. As you point out, we conducted a sub-group analysis of the asymptomatic patients and found that smell testing performed similarly in this group.

I note that only 2.3% of the 216 COVID negative patients and 10% of the 88 COVID positive patients reported anosmia or ageusia. This does not appear to be consistent with other studies which reported anosmia as the most common symptom. I think this merits some explanation.

Authors’ Response

A limitation of symptom-based screening (like self-reported alteration in smell or taste) is that it misses the 50% or more of people infected with SARS-CoV-2 that are either pre-symptomatic at the time point of screening or have an asymptomatic infection. Since we smell tested and PCR-tested asymptomatic patients, a smaller proportion of our sample reported altered smell or taste as compared to the studies we mention in the discussion (lines 248-257) that suffered from substantial selection and measurement biases.

I think the authors should explain why the 2 odor test had a higher sensitivity but lower specificity compared to the 8 odor test because this is counter intuitive

Authors’ Response

As indicated in Table 2, for calculating the sensitivity and specificity of the 8-odor model we used the NHANES definition of hyposmia (misidentifying 3 or more odors) as our cut point. For the 2-odor model, we used misidentification of 1 odor as the cut point. This choice of cut points affected the calculated sensitivity and specificity of the smell tests. If for instance we defined abnormal smell testing performance on the 8-odor test as misidentifying one or more odors, the test would be very sensitive but not specific for SARS-CoV-2 infection.

The results section in the abstract is also difficult to understand for me, the third sentence particularly. I think they should stick to the key findings in the abstract.

Authors’ Response

There were several typos in the results section of the abstract. These have been corrected which should improve the readability of the abstract.

The authors cut short the study when the prevalence of COVID went up to 25% positivity. How does this affect the applicability of the study in a non-surge setting? Should sensitivity analyses be done?

Authors’ Response

The discriminatory performance of a screening or diagnostic test (sensitivity and specificity) is independent of disease prevalence. We included in Table 2 three levels of disease prevalence to show how the positive and negative predictive values of the different screening approaches change depending on disease prevalence.

I also did not note if this research was approved by the ethics committee of their institution.

Authors’ Response

We received Institutional Review Board (equivalent of ethics committee) approval as noted in lines 94-96.

Reviewer #5

The study is interesting and well conducted expecialy in the methods. The authors conclude that smell testing was able to identify SARS-CoV-2 infection with a 73% of sensitivity and 75% od specifity even if was not investigated on asymptomatic subjects. However Bianco et al showed that alteration of sense of smell represent a prodromal symptom of infection, they coul add this reference as follow: Bianco, Maria Rita, et al. "Alteration of smell and taste in asymptomatic and symptomatic COVID-19 patients in Sicily, Italy." Ear, Nose & Throat Journal 100.2_suppl (2021): 182S-185S.

Authors’ Response

We, in fact, investigated smell testing in asymptomatic patients. As described in Table 1, many of our participants had no symptoms (were asymptomatic or prodromal). We agree that altered smell can occur in presymptomatic (prodomal) patients, which is why we conducted formal smell testing and found it more sensitive than reported symptoms in identifying SARS-CoV-2 infection.

Reviewer #6

Title: The may benefit from inclusion of location of the study.

Authors’ Response

We felt that including a location in the title might distract readers as the objective of the study was to evaluate the performance of a SARS-CoV-2 screening test (smell testing) and not describe a location-based phenomenon

Abstract: The background to the abstract is somewhat over summarized with the objective(s) of the study not stated

Authors’ Response

We added a statement in the abstract background about the objective: We evaluated the performance of formal smell testing to identify SARS-CoV-2 infection. (lines 35-36)

Introduction: The authors should consider differentiating asymptomatic from presymptomatic state of SARS-CoV-2 infections.

Authors’ Response

We are unclear what this reviewer means by differentiating the asymptomatic and presymptomatic state of infection. We believe these are commonly used terms that the readers of PLOS One will understand.

Methods:

1.The exclusion criteria used in this study should be stated clearly

2. It is unclear the measures used to mitigate against testing any participant more than once.

3. The authors also need to bring out clearly how the smell test was conducted, were the research clinicians trained on

conducting the smell test?

4. It is unclear if the those with anosmia/smell anomalies from any other conditions were identified and excluded from the study.

5.It appears a test of difference of means (unpaired student t test) was used for the comparison of age as well as temperature between the two categories but the authors presented it as if it was chi square.

6.It appears the age of the study participants in the two groups are skewed, it is imperative that the test of assessment of normality used should be stated otherwise an appropriate non parametric test should be used and stated.

Authors’ Responses

1. Added an exclusion criteria sentence “We excluded patients with an altered level of consciousness and minors.” (line 94)

2. Thank you for bringing this to our attention. We reviewed the source data and using the unique medical record number of each patient-participant identified 9 instances of repeat encounters with smell tasting data. We kept the data from the first of the two encounters and re-ran the data analysis with the 295 unique participants. There were small, statistically insignificant changes across most of our descriptive measures. All results in the manuscript and supporting material have been updated using the de-duplicated data set.

3. As noted in the methods section (lines 113-115) the PST smell test is designed to be self-administered. Participants scratch and odor strip and circle one of four multiple choice options to identify the odor.

4. We asked participants about a history of loss of smell (not recent) and reported those results in supplemental table S1. 28 participants reported a history of loss of smell. We did not exclude these patients from our primary analysis.

5. The descriptive statistics of age and temperature appearing in table 1 were generated with t tests. For measures of association of the linear predictor variables with the binary outcome variable (SARS-CoV-2 infection) we used logistic regression. The methods section (lines136-7) was updated to reflect this.

6. Thank you for bringing this to our attention. We inspected the age distribution visually and statistically and it was not normally distributed. We used the Wilcoxon rank-sum test to assess difference in age means between samples and updated the methods and results to reflect this analysis.

Results: The two additional ROC model used " hyposmia & a 2-odor model" were not mentioned in the methods, it is important that mention is made of it and the rationale for it provided in the methods before stating such in the results.

Authors’ Response

We added substantial additional detail to the methods section to describe the six ROC models presented in table 3 (lines 143-153)

Additional comment: It is important that the authors bring to light the limitation(s) associated with convenience sampling used particularly with regards to generalization of the findings

Authors’ Response

We recognize this limitation and mentioned it first in our paragraph describing study limitations (lines 273-275), stating “we recruited a convenience sample of patients seeking care at the emergency department, and we cannot generalize the findings of our study to other populations, such as asymptomatic people in the general population.”

Reviewer #7

would suggest including in table 1 the smell test results frequencies (each one and aggregate). Also in table 1 would include variables that are significant in table s1.

Authors’ Response

We report the smell test result frequencies in figure 1 as point estimates with 95% CI bars by SARS-CoV-2 test status. We believe reporting the same data in tabular form would be redundant.

what is the difference between history and symptom of loss of smell, the latter presumably without formal testing? The numbers appear small for both. What happens when both are combined?

Authors’ Response

The symptom “loss of taste and/or smell” was part of the emergency department’s COVID-19 screening questionnaire as described in the methods section (lines 90-92). The “history of loss of smell” was a question on our patient survey (methods section lines 112-113) about pre-existing health conditions and was intended to identify patients with underlying olfactory disorder. Since one question asks about an acute symptom and the other about a pre-existing condition, combining the responses would not be appropriate.

How good is a model of history and/or symptom of loss of smell/taste plus 2 smell test (or are numbers too small)?

Authors’ Response

Using the three classifiers 1) soap odor, 2) smoke odor, and 3) reported loss of taste and/or smell, a ROC model has an AUC of 0.76 (295 observations) – which is essentially the same as the 2-odor ROC model (AUC=0.75).

How do authors explain the discrepancy between self reported loss of smell and smell test results?

Authors’ Response

We addressed this in the introduction – lines 72-77. In short, people are often not aware that they have a diminished sense of smell, hence the interest in formal smell testing as a SARS-CoV-2 screening tool.

Table 2 is difficult to follow and should be perhaps split into two separate table or presented in a simplified table?

Authors’ Response

To simplify the table we removed the sensitivity and specificity rows as those values are listed in the text. We also removed the “2-odor” screening test because this was a sub-analysis of the smell testing performance. We believe the various performance metric of the screening tests are best compared side-by-side in a single table.

a secondary multivariate analysis (logistic regression of loss of smell, perhaps stratified on COVID-19 status?) that includes variables with significant differences in bivariate analysis from table 1 and table s1 could help identify confounders that one has to think about when using this approach.

Authors’ Response

We reported an “adjusted” ROC model that adjusted for age, gender, corticosteroid nasal spray use, measured fever, and reported cough as described in methods (lines 149-150) and in results (lines 216-218). We adjusted for age and nasal spray use because these were associated with smell test performance, and we adjusted for fever and cough because these were associated with SARS-CoV-2 infection (as reported in table 1 and S1.

Submitted filename: Response to reviewers.docx

Click here for additional data file.

1 Mar 2022

16 Mar 2022

Reviewer #1

Line 125: Figure 1 -- caption meant to be here in the methods section?

+ Yes – per PLOS One author guidelines the figure caption should be placed after the paragraph where it is first reference. Figure 1 is referenced in the preceding paragraph.

Lines 131-132: understood but not quite grammatically clear

+ Removed clause “based on unique medical record number” to improve sentence syntax.

Lines 152: understood perhaps specifying what "best performance" means

+ Clarified that the performance metric we used was AUC (area under the curve)

Line 234: interesting perhaps unexpected given that nasal steroids did impact performance, correct? And assuming most people with rhinitis on nasal steroids?

+ Looking at S1 Table, about 50% of sample reported seasonal allergies (believable in Kentucky). However, only 12% reported nasal steroid use. We modified this sentence (lines 234-36) to represent our analysis more accurately, which found that smell testing performance was similar between patients with/without sinusitis and seasonal allergies. We didn’t measure whether either condition was an effect modifier of the discriminatory ability of smell testing for SARS-CoV-2 infection.

Line 276: or negative PCR and sustained olfactory dysfunction that would decrease the already high specificity, yes? but no need to specifically note to avoid wordage.

+ We agree that this scenario is also possible and would decrease the specificity of the smell test.

Reviewer #3

Discussion: There is some ambiguity in line (≠250 and ≠251): …the utility of this using this as..?

+ We revised the sentence to read “The performance of smell testing in asymptomatic patients suggests its utility as a SARS-CoV-2 screening tool in asymptomatic populations, like employees at congregate work settings” for clarity.

Reviewer #6

The quality of this manuscript will be improved if more information is added to the background to the abstract as well as the conclusion of the abstract.

+ We minimally revised the background section of the abstract for improved clarity. Respectfully, we are unclear on what additional information would be helpful in the abstract. An abstract provides a concise description of the study and invites the reader to review the full manuscript for greater detail. As PLOS One is open access, anyone can read the manuscript in its entirety.

Submitted filename: Response to reviewers #2.docx

Click here for additional data file.

30 Mar 2022

Performance of formal smell testing and symptom screening for identifying SARS-CoV-2 infection

PONE-D-21-38950R2

Dear Dr. Keck,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Muhammad Tarek Abdel Ghafar, M.D

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

4 Apr 2022

PONE-D-21-38950R2

Performance of formal smell testing and symptom screening for identifying SARS-CoV-2 infection

Dear Dr. Keck:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Prof Muhammad Tarek Abdel Ghafar

Academic Editor

PLOS ONE