Impact of COVID-19 on Otolaryngology Literature.

OBJECTIVES/HYPOTHESIS: To understand the effect of the COVID-19 pandemic on the volume, quality, and impact of otolaryngology publications. STUDY
DESIGN: Retrospective analysis.
METHODS: Fifteen of the top peer-reviewed otolaryngology journals were queried on PubMed for COVID and non-COVID-related articles from April 1, 2020 to March 31, 2021 (pandemic period) and pre-COVID articles from the year prior. Information on total number of submissions and rate of acceptance were collected from seven top-ranked journals.
RESULTS: Our PubMed query returned 759 COVID articles, 4,885 non-COVID articles, and 4,200 pre-COVID articles, corresponding to a 34% increase in otolaryngology publications during the pandemic period. Meta-analysis/reviews and miscellaneous publication types made up a larger portion of COVID publications than that of non-COVID and pre-COVID publications. Compared to pre-COVID articles, citations per article 120 days after publication and Altmetric Attention Score were higher in both COVID articles (citations/article: 2.75 ± 0.45, P < .001; Altmetric Attention Score: 2.05 ± 0.60, P = .001) and non-COVID articles (citations/article: 0.03 ± 0.01, P = .002; Altmetric Attention Score: 0.67 ± 0.28, P = .016). COVID manuscripts were associated with a 1.65 times higher acceptance rate compared to non-COVID articles (P < .001).
CONCLUSIONS: COVID-19 was associated with an increase in volume, citations, and attention for both COVID and non-COVID articles compared to pre-COVID articles. However, COVID articles were associated with lower evidence levels than non-COVID and pre-COVID articles. LEVEL OF EVIDENCE: 3 Laryngoscope, 132:1364-1373, 2022.

INTRODUCTION

The COVID‐19 pandemic has led to an upsurge in publications aiming to elucidate the clinical features, pathophysiology, and management of the disease. Due to the urgency of the pandemic, the scientific community has been forced to balance rigorous, time‐consuming research with the need for rapid publication and dissemination of knowledge. ,

Striking this balance has been challenging, and multiple studies have analyzed research challenges within the COVID‐19 literature. , , One study investigating the highest impact medical journals found that COVID‐19 publications were associated with lower levels of evidence and higher citation rates than that of non‐COVID‐19‐related research. Pandemic‐related research has also been criticized for small sample sizes, lack of control groups, and premature access to pre‐prints. Additionally, a high retraction rate of COVID‐19 publications has caused great alarm in the scientific community, affecting even the most prominent medical journals. , , Though COVID‐19 publication quality has been criticized, the research community has been ultimately successful in identifying treatments and developing vaccines. , , ,

The otolaryngology (ENT) research community has been similarly affected by the pandemic. Two publications have demonstrated an increase of 58% and 42% in ENT publications from German and Italian universities during the pandemic, compared to prior years. , Another study of the top 20 ENT journals found that 75% of the 166 COVID‐19 related publications were either editorials, comments, or letters, which generally correspond to lower levels of scientific evidence. Additionally, little attention has been given to non‐traditional, but increasingly influential, measures of ENT research impact, like social media reach. , The comprehensive impact of COVID‐19 on ENT research quantity and quality has yet to be formally described. We sought to elucidate these effects by analyzing the publications of 15 top‐ranked ENT journals during the COVID‐19 pandemic.

MATERIALS AND METHODS

Journal Selection

We identified the top 20 otolaryngology journals by h5‐index as listed on Google Scholar. From this list, we excluded the Otolaryngology Clinics of North America because it does not publish peer‐reviewed publications. We also excluded open access journals (i.e., Brazilian Journal of Otorhinolaryngology and Journal of Otolaryngology‐Head & Neck Surgery). After initial data collection was performed, we excluded Hearing Research and Audiology and Neurotology because neither journal published any COVID‐related articles. Thus, 15 journals were included in the final analysis.

Data Collection

This study met exemption criteria established by The George Washington University Institutional Review Board. We collected data from PubMed, utilizing its Entrez Programming Utilities (E‐utilities) interface. The articles from the aforementioned 15 otolaryngology journals were divided into three groups to compare the impact of COVID on otolaryngology literature. These article groups will be henceforth referred to as COVID articles, non‐COVID articles published during the pandemic, and pre‐COVID articles.

COVID articles from PubMed were queried from April 1, 2020 to March 31, 2021 using the search terms: COVID, COVID 19, COVID‐19, Coronavirus, Coronavirus 19, Coronavirus‐19, or SARS‐COV‐2. Non‐COVID articles were queried on PubMed from April 1, 2020 to March 31, 2021 by excluding COVID search terms. Pre‐COVID articles were queried from April 1, 2019 to March 31, 2020. While COVID‐19 was known as early as December 2019, we defined the “pandemic period” from April 1, 2020 to March 31, 2021 to account for publication lag. We separately queried publication data from PubMed on common ENT topics using the search terms otitis media, obstructive sleep apnea, chronic sinusitis, hoarseness, and sensorineural hearing loss from April 1, 2011 to March 31, 2021.

Data on unique COVID and non‐COVID article submission and acceptance count were requested from 9 of the top‐ranked ENT journals. Resubmissions were excluded. Data included the number of articles submitted and accepted for the pre‐COVID (April 1, 2019–March 31, 2020), non‐COVID (April 1, 2020–March 31, 2021), and COVID (April 2020–March 31, 2021) groups. Seven of the nine journals provided complete data.

Article Characteristics

Data collected from PubMed for each article included authorship information, publication date and type, and overall citation count at time of data collection (May 27, 2021) and within 120 days of publication. The top 5 countries by number of COVID articles published were identified for analysis and the remainder of countries were grouped together.

Publication type refers to PubMed‐specified terms and were grouped into five categories for analysis that aimed to parallel Oxford level of evidence groups. , “Meta‐analysis/Reviews” included Meta‐Analysis, Systematic Review, Review, and Practice Guideline publication types. “Randomized Control Trials” included Randomized Controlled Trial, Clinical Trial, Clinical Trial Phases I‐IV, Controlled Clinical Trial, and Equivalence Trial publication types. “Observation/Cohort/Other Studies” included Evaluation Study, Comparative Study, Multicenter Study, Observational Study, Validation Study, and Clinical Study publication types. “Case Studies” included Case Report and Twin Study publication types. “Miscellaneous” included all other publication types, a majority of which were Letter, Comment, and Editorial publication types. The generic publication type Journal Article is listed for a majority of publications without a more specific tag and were included in the “Observation/Cohort/Other Studies” group.

Altmetric (London, UK) was used to obtain information on article “attention.” The Altmetric Attention Score is a weighted function of article mentions across mainstream news outlets, social media websites, blogs, policy documents, and other online platforms. We also used Altmetric data to obtain the number of times an article is mentioned specifically on social media platforms.

Statistical Analysis

Descriptive statistics were used to compare article features using Chi‐square test for categorical variables and ANOVA test for continuous variables. Multivariate regression was performed to analyze the association between COVID publications and citation/attention metrics. All analyses were completed using Python (Python Software Foundation, Delaware, United States) and R (R Foundation for Statistical Computing, Vienna, Austria). A two‐sided P‐value less than .05 was considered statistically significant. All data collection and analysis code are available at https://bit.ly/3fXwfOy.

RESULTS

Publication Trends

The query on PubMed returned 9,847 articles. Three studies with “veterinary” publication types were excluded, resulting in 9,844 articles for final inclusion: 759 COVID articles, 4,885 non‐COVID articles, and 4,200 pre‐COVID articles. This corresponds to a 34.4% increase in publications from the pre‐pandemic period (April 1, 2019–March 31, 2020) to the pandemic period (April 1, 2020–March 31, 2021). During the pandemic period, over three times as many COVID articles were published as those on other popular ENT topics (Table I). However, during the pandemic period, linear regression of monthly publications demonstrated a gradual decline in COVID publications over time (−6.26 ± 1.42 publications per month, P < .001). No significant change was seen in the monthly number of non‐COVID publications over time (P = .571).

TABLE I

Number of Publications by Selected ENT Topic.

Topic	Number of Publications During COVID period (April 1, 2020–March 31, 2021)	Number of Publications pre‐COVID period (April 1, 2019–March 31, 2020)	Number of Publications in Past Decade (April 1, 2011–March 31, 2021)
COVID	759	0	0
Chronic sinusitis	120	211	2,067
Hoarseness	26	21	242
Obstructive sleep apnea	175	148	1,525
Otitis media	167	155	1,754
Sensorineural hearing loss	237	279	2,865

Journal of Laryngology & Otology (30.0% COVID publications, n = 66) and Otolaryngology‐Head and Neck Surgery (29.6%, n = 175) had the highest proportion of COVID publications during the pandemic period, while Otology & Neurotology (2.3%, n = 9) and Ear and Hearing (2.0%, n = 3) had the fewest (Fig. 1). Most journals published more COVID “miscellaneous” articles and fewer COVID “Observation/Cohort/Other studies” than those among non‐COVID or pre‐COVID articles (Fig. 2).

Fig. 1

COVID and non‐COVID publication by journal. Percent of non‐COVID versus COVID publication by 15 ENT journals from April 1, 2020 to March 31, 2021 demonstrate that certain journals published a higher portion of COVID articles, especially J Laryngol Otol (30.0% COVID publications, n = 66), Otolaryngol Head Neck Surg (29.6%, n = 175), Int Forum of Allergy Rhinol (21.4%, n = 49), JAMA Otolaryngol Head Neck (21.4%, n = 59), and Head Neck (20.6%, n = 104).

Fig. 2

Publication type by journal and publication period. Data acquired from PubMed demonstrates that a majority of top ENT journals published more “Miscellaneous” and fewer “Observation/cohort/other” studies than non‐COVID and pre‐COVID articles. Additionally, journals tended to publish more “meta‐analysis/review” articles on COVID.

Manuscript Submissions and Acceptances

Seven journals contributed data on submitted and accepted publications for the COVID, non‐COVID, and pre‐COVID groups. There was a 38.8% increase in number of submissions from the pre‐pandemic period (10,538 manuscripts) to the pandemic period (14,622 manuscripts), with only a 21.2% increase in the number of accepted publications (2,567 increased to 3,111 publications). In 5 of 7 journals, COVID articles had a significantly higher acceptance rate than non‐COVID articles (Fig. 3). Overall, COVID articles had a 1.65 times higher acceptance rate than non‐COVID articles (P < .001). There was a significant decrease in acceptance rate from pre‐COVID to non‐COVID groups in 4 of 7 journals (Fig. 3).

Fig. 3

COVID, non‐COVID, and pre‐COVID submissions. Number of unique submissions and accepted publications from April 2019–April 2020 (pre‐COVID) and April 2020–April 2021 (non‐COVID and COVID) show that a majority of selected top ENT journals had higher acceptance rates for COVID articles, compared to non‐COVID articles. Percent of articles accepted annotated on graph. Chi‐square P‐values identifying significant differences in acceptance rates between time period for each journal is shown on the figure as follows: *P < .05, **P < .01, ***P < .001.

Two journals were only able to provide partial data. International Journal of Pediatric Otorhinolaryngology noted a 36.3% acceptance rate in 2019, 33.0% in 2020, and approximately 30% for COVID articles. European Archives of Oto‐Rhino‐Laryngology noted a pre‐COVID acceptance rate of 26.0% and a combined COVID and non‐COVID acceptance rate of 25.7%.

Publication Statistics

Comparison of COVID, non‐COVID, and pre‐COVID article features is shown in Table II. Meta‐analysis/reviews and miscellaneous publication types made up a larger portion of COVID publications than non‐COVID and pre‐COVID (Meta‐analysis/reviews: 17.1%, 7.2%, and 8.6%, respectively, P < .001; miscellaneous: 21.3%, 8.7%, 8.2%, P < .001). Fewer non‐COVID randomized control trials (n = 73, 1.5%) were published compared to pre‐COVID randomized control trials (n = 137, 3.3%, P < .001). The United States published a larger portion of COVID publications (29.0%) and non‐COVID (23.2%) publications than pre‐COVID publications (15.5%, P < .001). Altmetric Attention Score is significantly higher in COVID articles than non‐COVID or pre‐COVID (24.9 vs. 2.0 vs. 2.9 points, P < .001). When analyzing the top 100 most cited publications in each group, most of the previous trends persisted (Table II).

TABLE II

Publication Statistics.

Variable	COVID † Publications, n (%)	Non‐COVID † Publications, n (%)	Pre‐COVID † Publications, n (%)	P
Among all publications	n = 759	n = 4,885	n = 4,200
Publication type				<.001*
Meta‐analysis/reviews	130 (17.1)	352 (7.2)	363 (8.6)
Randomized control trials	2 (0.3)	73 (1.5)	137 (3.3)
Observational/cohort/other studies	438 (57.7)	3,880 (79.4)	3,150 (75.0)
Case studies	27 (3.6)	155 (3.2)	206 (4.9)
Miscellaneous	162 (21.3)	425 (8.7)	344 (8.2)
Top 5 countries by most COVID articles				<.001*
United States	220 (29.0)	1,133 (23.2)	653 (15.5)
Italy	89 (11.7)	181 (3.7)	153 (3.6)
France	25 (3.3)	99 (2.0)	97 (2.3)
India	24 (3.2)	124 (2.5)	70 (1.7)
Canada	21 (2.8)	161 (3.3)	161 (3.8)
Others	380 (50.0)	3,187 (65.2)	3,066 (73.0)
h5‐index of journal (mean ± SD)	40.7 ± 8.5	39.7 ± 9.1	39.0 ± 9.2	<.001*
Altmetric Attention Score (mean ± SD)
Total score	24.9 ± 173.2	2.0 ± 18.2	2.9 ± 26.5	<.001*
Score 3 mo after publication	2.1 ± 14.9	0.6 ± 15.7	0.1 ± 1.0	<.001*
Score 6 mo after publication	5.0 ± 23.1	1.2 ± 17.0	0.2 ± 1.5	<.001*
Social media posts	21.0 ± 211.5	2.3 ± 6.5	3.2 ± 10.8	<.001*
Among top 100 most cited publications	n = 100	n = 100	n = 100
Publication type				.020*
Meta‐analysis/reviews	25	21	25
Randomized control trials	1	3	6
Observation/cohort/other studies	61	73	65
Case studies	0	1	0
Miscellaneous	13	2	4
Top 5 countries by most COVID articles				<.001*
United States	28	17	14
Italy	25	8	8
France	6	1	2
India	0	0	0
Canada	3	6	4
Others	38	68	72
h5‐index of journal (mean ± SD)	42.7 ± 6.8	43.1 ± 6.9	42.4 ± 7.5	.777
Altmetric Attention Score (mean ± SD)
Total score	87.7 ± 255.7	9.0 ± 37.9	17.5 ± 52.0	<.001*
Score 3 mo after publication	3.7 ± 19.1	2.0 ± 12.8	1.1 ± 5.6	.396
Score 6 mo after publication	10.5 ± 38.8	5.8 ± 36.8	1.5 ± 6.2	.124
Social media posts	48.8 ± 135.7	6.0 ± 13.3	14.8 ± 31.2	<.001*

P < .05.

COVID publications refers to publications queried from April 1, 2020–March 31, 2021 regarding COVID‐19. Non‐COVID refers to publications during the COVID time period (April 1, 2020–March 31, 2021) that are unrelated to COVID‐19. Pre‐COVID refers to publications unrelated to COVID in the 1 year period before COVID (April 1, 2019–March 31, 2020).

SD = standard deviation.

Citation Statistics

Total citation count and citation count 120 days postpublication are detailed in Table III. COVID publications were cited significantly more overall (P < .001) and at 120 days (P < .001), and this significance persisted when broken down by publication type (Table III). For the top 100 most cited articles in each group, COVID articles were still cited more often overall (46.8 vs. 3.4 vs. 8.0 citations/article, P < .001) and at 120 days postpublication (16.0 vs. 1.0 vs. 0.4 citations/article, P < .001). This trend persisted in all publication types except “miscellaneous” articles. Table IV lists the 10 most cited nonmiscellaneous COVID articles. They focus on characterizing olfactory and gustatory symptoms and outlining safe practices for otolaryngologic procedures.

TABLE III

Citation Statistics.

Variable	COVID † Citations/Article (mean ± SD)	Non‐COVID † Citations/Article (mean ± SD)	Pre‐COVID † Citations/Article (mean ± SD)	P
Among all publications
Overall citations per paper
Total	8.25 ± 33.92	0.20 ± 0.68	0.88 ± 1.65	<.001*
Within 120 d	2.82 ± 12.11	0.06 ± 0.31	0.04 ± 0.21	<.001*
Publication type
Meta‐analysis/reviews
Total	12.16 ± 25.79	0.43 ± 1.12	1.79 ± 3.11	<.001*
Within 120 d	4.41 ± 11.91	0.08 ± 0.33	0.7 ± 0.31	<.001*
Randomized control trials
Total	9.50 ± 4.95	0.22 ± 0.51	1.18 ± 1.67	<.001*
Within 120 d	1.50 ± 0.71	0.04 ± 0.20	0.04 ± 0.19	<.001*
Observation/cohort/other studies
Total	8.61 ± 40.40	0.20 ± 0.67	0.85 ± 1.44	<.001*
Within 120 d	2.79 ± 13.83	0.06 ± 0.32	0.04 ± 0.21	<.001*
Case studies
Total	3.37 ± 2.84	0.12 ± 0.41	0.40 ± 0.74	<.001*
Within 120 d	1.26 ± 1.51	0.02 ± 0.18	0.00 ± 0.00	<.001*
Miscellaneous
Total	4.94 ± 20.60	0.08 ± 0.35	0.29 ± 1.14	<.001*
Within 120 d	1.92 ± 7.38	0.04 ± 0.25	0.03 ± 0.20	<.001*
Among top 100 most cited publications
Overall citations per paper
Total	46.78 ± 83.80	3.38 ± 2.38	8.00 ± 4.24	<.001*
Within 120 d	16.03 ± 30.17	1.01 ± 1.37	0.41 ± 0.70	<.001*
Publication type
Meta‐analysis/reviews
Total	45.60 ± 45.62	3.86 ± 2.24	9.80 ± 7.03	<.001*
Within 120 d	17.92 ± 22.76	0.71 ± 0.96	0.36 ± 0.70	<.001*
Randomized control trials
Total	13.0 ± N/A‡	2.00 ± N/A‡	6.67 ± 1.37	<.001*
Within 120 d	2.0 ± N/A‡	0.00 ± N/A†	0.33 ± 0.52	.015*
Observation/cohort/other studies				—
Total	48.39 ± 99.84	3.32 ± 2.49	7.37 ± 2.59	<.001*
Within 120 d	15.57 ± 34.52	1.08 ± 1.46	0.42 ± 0.70	<.001*
Case studies
Total	N/A‡	3.00 ± N/A‡	N/A‡
Within 120 d	N/A‡	2.00 ± N/A‡	N/A‡
Miscellaneous
Total	44.08 ± 61.77	3.00 ± 1.41	9.00 ± 3.16	.392
Within 120 d	15.62 ± 22.14	2.50 ± 2.12	0.75 ± 0.96	.344

P < .05.

COVID publications refers to publications queried from April 1, 2020–March 31, 2021 regarding COVID‐19. Non‐COVID refers to publications during the COVID time period (April 1, 2020–March 31, 2021) that are unrelated to COVID‐19. Pre‐COVID refers to publications unrelated to COVID in the 1 year period before COVID (April 1, 2019–March 31, 2020).

N/A noted for mean if no publications in category and for standard deviation if only 1 publication noted in that category.

SD = standard deviation.

TABLE IV

Ten Most Cited COVID Articles.

Citations	Citations at 120 d	Title	Journal	Publication Type
735	252	Olfactory and gustatory dysfunctions as a clinical presentation of mild‐to‐moderate forms of the coronavirus disease (COVID‐19): A multicenter European study.	Eur Arch Otorhinolaryngol	Observational/cohort/other studies
244	85	Association of chemosensory dysfunction and COVID‐19 in patients presenting with influenza‐like symptoms.	Int Forum Allergy Rhinol	Observational/cohort/other studies
208	113	Safety recommendations for evaluation and surgery of the head and neck during the COVID‐19 pandemic.	JAMA Otolaryngol Head Neck Surg	Meta‐analysis/reviews
207	62	Smell dysfunction: A biomarker for COVID‐19.	Int Forum Allergy Rhinol	Observational/cohort/other studies
145	40	Objective evaluation of anosmia and ageusia in COVID‐19 patients: Single‐center experience on 72 cases.	Head Neck	Observational/Cohort/Other Studies
126	54	Endonasal instrumentation and aerosolization risk in the era of COVID‐19: Simulation, literature review, and proposed mitigation strategies.	Int Forum Allergy Rhinol	Observational/cohort/other studies
126	31	The prevalence of olfactory and gustatory dysfunction in COVID‐19 patients: A systematic review and meta‐analysis.	Otolaryngol Head Neck Surg	Meta‐analysis/reviews
111	36	Self‐reported olfactory loss associates with outpatient clinical course in COVID‐19.	Int Forum Allergy Rhinol	Observational/cohort/other studies
108	33	COVID‐19 anosmia reporting tool: Initial findings.	Otolaryngol Head Neck Surg	Meta‐analysis/reviews
103	52	Surgical considerations for tracheostomy during the COVID‐19 pandemic: Lessons learned from the severe acute respiratory syndrome outbreak.	JAMA Otolaryngol Head Neck Surg	Meta‐analysis/reviews

Table excludes miscellaneous publication types.

Multivariate regression analysis of the relationship between article characteristics and citations and attention metrics is detailed in Table V. When controlling for publication type, country of publication, and journal published, COVID publications were cited 7.25 ± 1.27 (P < .001) more times than pre‐COVID articles, while non‐COVID articles were cited 0.64 ± 0.36 fewer times (P < .001). When analyzing citations within 120 days postpublication, both COVID and non‐COVID articles were associated with 2.75 ± 0.45 (P < .001) and 0.03 ± 0.01 (P = .002) more citations than pre‐COVID articles. This trend persisted with the 3‐month Altmetric Attention Score. When analyzing social media posts per article, no difference existed between pre‐COVID and non‐COVID publications although COVID publications were posted 18.3 ± 8.3 more times than pre‐COVID articles (P < .001) when controlling for the same article characteristics.

TABLE V

Multivariate Regression of Article Characteristics on Citation and Attention Metrics.

Variable Coefficient ± Standard Error	Citations/Paper	Citations/Paper Within 120 d	Social Media Posts	Altmetric Attention Score at 3 mo
Publication period
Pre‐COVID	ref.	ref.	ref.	ref.
Non‐COVID	−0.64 ± 0.36 ***	0.03 ± 0.01 **	−0.10 ± 0.48	0.67 ± 0.28 *
COVID	7.25 ± 1.27 ***	2.75 ± 0.45 ***	18.32 ± 8.28 *	2.05 ± 0.60 **
Publication type
Observation/cohort/other studies	ref.	ref.	ref.	ref.
Meta‐analysis/reviews	1.00 ± 0.41 *	0.25 ± 0.17	0.24 ± 0.82	1.57 ± 1.33
Randomized control trials	0.25 ± 0.13	0.00 ± 0.03	1.34 ± 0.65 *	−0.17 ± 0.10
Case studies	−0.54 ± 0.13 ***	−0.12 ± 0.05 *	12.10 ± 13.79	0.06 ± 0.28
Miscellaneous	0.62 ± 0.13 ***	−0.26 ± 0.14	−5.28 ± 1.76 **	−0.80 ± 0.21 ***
Top 5 countries by most COVID articles
United States	ref.	ref.	ref.	ref.
Canada	0.67 ± 0.70	0.25 ± 0.24	2.35 ± 2.13	0.11 ± 0.73
France	3.95 ± 3.21	1.37 ± 1.11	2.29 ± 2.66	−0.08 ± 0.25
India	−0.38 ± 0.22	−0.13 ± 0.08	−1.03 ± 0.56	−0.01 ± 0.19
Italy	1.47 ± 0.74 *	0.42 ± 0.24	−0.26 ± 2.21	−0.14 ± 0.25
Others	0.13 ± 0.16	0.04 ± 0.06	0.25 ± 0.89	0.12 ± 0.19
Journal
Acta Otolaryngol	−0.26 ± 0.19	−0.01 ± 0.06	−1.36 ± 0.47 **	−0.33 ± 0.16 *
Am J Rhinol Allergy	−0.38 ± 0.23	−0.05 ± 0.08	0.22 ± 0.56	−0.40 ± 0.18 *
Ann Otol Rhinol Laryngol	−0.44 ± 0.19 *	−0.09 ± 0.06	−1.61 ± 0.71 *	−0.41 ± 0.19 *
Auris Nasus Larynx	−0.39 ± 0.23	−0.08 ± 0.08	−2.31 ± 1.16 *	−0.51 ± 0.18 **
Clin Otolaryngol	−0.20 ± 0.26	−0.03 ± 0.09	1.68 ± 0.77 *	0.13 ± 0.20
Ear Hear	0.45 ± 0.18 *	0.26 ± 0.07 ***	1.18 ± 0.72	−0.22 ± 0.20
Eur Arch Otorhinolaryngol	0.15 ± 0.51	0.08 ± 0.17	−1.17 ± 0.43 **	0.19 ± 0.24
Head Neck	0.54 ± 0.30	0.17 ± 0.10	0.11 ± 0.51	−0.42 ± 0.21 *
Int Forum Allergy Rhinol	1.99 ± 0.82 *	0.63 ± 0.28 *	5.31 ± 1.81 **	3.64 ± 2.57
Int J Pediatr Otorhinolaryngol	−0.37 ± 0.17 *	−0.05 ± 0.06	−1.56 ± 0.47 **	−0.47 ± 0.18 **
J Laryngol Otol	−1.13 ± 0.26 ***	−0.31 ± 0.09 ***	−3.09 ± 1.26 *	−0.28 ± 0.25
JAMA Otolaryngol Head Neck Surg	0.68 ± 0.53	0.34 ± 0.26	32.94 ± 9.75 **	1.27 ± 0.38 **
Laryngoscope †	ref.	ref.	ref.	ref.
Otol Neurotol	0.26 ± 0.21	0.05 ± 0.07	0.17 ± 0.46	−0.09 ± 0.27
Otolaryngol Head Neck Surg	−0.10 ± 0.31	−0.05 ± 0.10	−1.71 ± 0.78 *	−0.55 ± 0.26 *

All significant values are bolded for clarity.

P < .05.

P < .01.

P < .001.

Laryngoscope was chosen as the reference group because this journal had the most publications overall in our dataset (n = 1,486).

Notably, the country of publication was generally not associated with citation or attention metrics (Table V). Similarly, publication type and journal of publication had variable associations with only minor effect sizes on citation and attention metrics (Table V). However, more journals were significantly associated with the Altmetric Attention Score at 3 months and the number of social media posts than were associated with citations per article (Table V). A separate multivariate regression found no significant relationship between citations per article and social media posts (β coefficient of 0.02 ± 0.15, P = .908) when controlling for publication type, COVID group, journal, and country.

DISCUSSION

COVID‐19 had a significant impact on ENT publications. There was a 34.4% increase in publications during the COVID period compared to the year prior with a greater focus on COVID‐19 than other common ENT topics. However, we found overall research output on COVID‐19 in ENT is already beginning to decline. We utilized citations per article as a marker for impact, publication type as a proxy for quality, and Altmetric Attention Score and social media mentions as a representation of attention across a variety of media (e.g., social media, Wikipedia, blogs, and policy documents). While we acknowledge the imperfect nature of these metrics, we find strong evidence that the COVID‐19 pandemic period was correlated with higher impact and attention for both COVID and non‐COVID ENT articles, as well as a dramatic increase in volume of publications.

Both COVID (n = 759) and non‐COVID articles (n = 4,885) contributed to the increase in publications (pre‐COVID, n = 4,200). There was a corresponding increase in submission in non‐COVID articles compared to the pre‐COVID period, which may be due to increased research productivity among surgeons as elective surgeries were postponed. , , There was a significant increase in “Observation/Cohort/Other” articles, which often includes research that can be completed remotely, such as retrospective analyses. There were 685 more publications in the non‐COVID group compared to the pre‐COVID group, of which the United States produced 480. Thus, the United States represented a larger share of non‐COVID ENT publications (23%) than in the prior year (16%). This increase in research productivity was likely due to the pandemic's limiting effect on clinical practice, particularly elective procedures, of which the U.S. cancelled more than other countries. , , ,

Studies have demonstrated lower levels of evidence among COVID compared to non‐COVID articles. Among the top medical journals, COVID publications were 186‐fold more likely to be of lower evidence. High‐ and low‐ranking journals were equally as likely to publish COVID articles that were ultimately retracted. Another study found 72 retractions of COVID‐19 articles, including those published in well‐regarded journals like the New England Journal of Medicine and The Lancet. These retractions heighten the concern regarding COVID research quality. We found that ENT COVID publications were more likely to be of “Miscellaneous” and “Meta‐analysis/Reviews” publication types. While meta‐analysis and reviews are generally associated with higher levels of evidence, we believe this to be the exception for COVID articles. Given the relatively fewer number of “Randomized Controlled Trial” and “Observational/Cohort/Other Study” publication types, these review articles are likely based on weaker evidence.

Notably, we found that the COVID‐19 pandemic impacted non‐COVID ENT publications. Compared to the pre‐pandemic period, fewer non‐COVID “Meta‐analysis/Review” and “Randomized Control Trial” articles were published during COVID‐19. This change in publication pattern may be due to logistical challenges associated with COVID‐19. For example, it has been more difficult to run clinical trials during COVID‐19 due to safety challenges, patient appointment interruptions, and the shift in focus to COVID‐related research. ,

Our study showed that COVID articles generally had a higher acceptance rate than non‐COVID articles across several top ENT journals. Within ENT, COVID articles have focused on reducing the spread of disease during ENT procedures and on investigating olfactory symptoms, which is consistent with our findings. However, COVID‐19 publications across a variety of scientific disciplines have been associated with lower levels of evidence and a shorter time to publication, likely due to the urgency of the pandemic. , While COVID‐19 research has led to the eventual discovery of important treatments like remdesivir, they have also led to misuse, as in the case of hydroxychloroquine. It is important to understand the limitations and cost of COVID‐19 on ENT research, specifically noted by the lower levels of evidence in COVID articles and the decrease in randomized control trials and meta‐analysis/reviews in non‐COVID publications.

Nonetheless, during COVID‐19, an increase in the impact and attention was noted among top ENT journals for both COVID and non‐COVID articles. While the pandemic period was associated with more citations and attention for COVID articles, it was also associated with a modest increase in citations and attention score for non‐COVID articles when compared to pre‐COVID articles. This suggests that the readership brought to ENT journals from COVID articles had a spillover effect for non‐COVID articles. Overall citations per article and attention is higher in pre‐COVID than non‐COVID articles likely because pre‐COVID articles have been in circulation for a longer period of time. Generally, the journal of publication had little to no impact on citations per article. Int Forum Allergy Rhinol and Ear Hear were associated with the highest citations per article, while JAMA Otolaryngol Head Neck Surg was associated with the highest attention scores and number of social media mentions per article. Publication in JAMA Otolaryngol Head Neck Surg was associated with 32 more social media posts per article, likely due to their large online presence associated with the JAMA Network. However, it is important to note, we found no relationship between social media posts and citations per publication when controlling for journal and article characteristics, which differs from published findings. , , , This difference could be due to our analysis involving only the highest impact journals in a small specialty. Thus, scientific readership is relatively fixed and social media may have a more limited impact on citations. Additionally, publication type had little to no impact on citation or attention metrics.

The major limitation of this study is in its analysis of the level of evidence of published articles. Because we utilized PubMed “publication type” tags, we were not able to directly analyze the level of evidence as specified by the Oxford Center for Evidence‐Based Medicine. While we lacked the ability to differentiate nuances in studies, such as nonrandomized cohort studies (level of evidence III) versus observational studies with dramatic evidence (level of evidence II), we grouped publication types in categories that generally correspond to appropriate levels of evidence. An additional limitation is that publishing patterns may have changed in the study timeframe within the journals. For example, the Int Forum Allergy Rhinol and the Laryngoscope had new editors‐in‐chief during the study period. While these changes may affect how individual journals made editorial decisions, we believe the overall trends in our analysis are relatively unaffected.

CONCLUSION

COVID‐19 resulted in dramatic changes in otolaryngology research. There was a significant increase in publications among both COVID and non‐COVID articles but a decrease in the publication of clinical trials. COVID articles drew significantly more focus than non‐COVID articles from researchers and the general public, despite being of lower evidence levels. However, non‐COVID research was also cited more often and received more attention than in the prior year.