Quality of reporting of otorhinolaryngology articles using animal models with the ARRIVE statement.

Research involving animal models is crucial for the advancement of science, provided that experiments are designed, performed, interpreted, and reported well. In order to investigate the quality of reporting of articles in otorhinolaryngology research using animal models, a PubMed database search was conducted to retrieve eligible articles. The checklist of the ARRIVE (Animal Research: Reporting of In Vivo Experiments) guidelines was used to assess the quality of reporting of articles published in ear, nose and throat (ENT) and multidisciplinary journals. Two authors screened titles, abstracts, and full texts to select articles reporting otorhinolaryngology research using in vivo animal models. ENT journals ( n = 35) reported a mean of 57.1% adequately scored ARRIVE items (median: 58.3%; 95% confidence interval [CI; 53.4-60.9%]), while articles published in multidisciplinary journals ( n = 36) reported a mean of 49.1% adequately scored items (median: 50.0; 95% CI [46.2-52.0%]). Articles published in ENT journals showed better quality of reporting of animal studies based on the ARRIVE guidelines ( P < 0.05). However, adherence to the ARRIVE guidelines is generally poor in otorhinolaryngology research using in vivo animal models. The endorsement of the ARRIVE guidelines by authors, research and academic institutes, editorial offices and funding agencies is recommended for improved reporting of scientific research using animal models.

Research involving animal models plays an important role in scientific innovation provided that the experiments are designed, performed, interpreted, and reported well. Recently, a lack of transparency in scientific publications using animal models has been identified.[1] Crucial experimental design elements are frequently ignored in scientific publications, rendering experiments irreproducible.

Kilkenny et al. reported that 59% of investigated articles using animal models failed to adequately report a research hypothesis, objective, or the number and characteristics of the animals used (i.e. species/strain, sex, and age/weight).[2] The vast majority of these articles did not report randomization of animals (87%) or blinding of researchers (86%). A similar study revealed that 72% of articles using animal models did not report randomization of animals and 98% did not report blinding of researchers.[3] Others have reported deficiencies in reporting important methodological parameters that render experiments irreproducible.[4] Inadequate reporting greatly hinders the conclusions drawn in scientific publications, especially when they cannot be reproduced due to non-transparent methodology.[5] Studies may in fact be well-designed and well-conducted. However, when poorly reported they may lead to inaccurate interpretation and translation.[6,7]

In an attempt to optimize the overall quality of reporting of scientific research using animal models, the ARRIVE (Animal Research: Reporting of In Vivo Experiments) guidelines were developed.[8] These guidelines aim to improve the experimental design, analysis, and reporting of research using animal models.

Using the checklist of the ARRIVE guidelines as a scoring tool, our primary aim was to evaluate the quality of reporting of articles using animal models in otorhinolaryngology research. Furthermore, the quality of reporting of articles published in ear, nose and throat (ENT) journals was compared with those published in multidisciplinary journals.

Methods

Journal selection

The quality of reporting of articles which describe animal experiments in otorhinolaryngology research was compared between two journal categories: ENT journals and multidisciplinary journals. Based on ISI Web of Knowledge impact factors (www.webofknowledge.com, date inspected: 12 June,2015), the five ENT journals with the highest impact factors in 2012 were selected: Ear & Hearing (Ear Hear), Journal of the Association for Research in Otolaryngology (JARO), Head & Neck – Journal for the Sciences and Specialties of the Head and Neck (Head Neck), Hearing Research (Hear Res), and Audiology & Neurotology (Audiol Neurotol). None of these journals implemented the ARRIVE guidelines in the ‘Instructions to Authors’ (date inspected: 12 June 2015). The top five multidisciplinary journals in 2013 were Nature, Science, Nature Communications (Nat Commun), Proceedings of the National Academy of Sciences of the United States of America (PNAS) and Scientific Reports (Sci Rep). Two journals (Nature and Nat Commun) recommended the ARRIVE guidelines when documenting animal studies (date inspected: 12 June 2015). The included journals and their impact factors are summarized in Table 1.

Table 1.

Impact factors of the top five ENT and multidisciplinary journals.

Journal	Impact factor*
ENT journals
Ear & Hearing (Ear Hear)	3.262
Journal of the Association for Research in Otolaryngoloy (JARO)	2.952
Head & Neck (Head Neck)	2.833
Hearing Research (Hear Res)	2.537
Audiology & Neurotology (Audiol Neurotol)	2.318
Multidisciplinary journals
Nature	42.531
Science	31.477
Nature Communications (Nat Commun)	10.700
Proceedings of the National Academy of Sciences of the United States of America (PNAS)	9.809
Scientific Reports (Sci Rep)	5.078

Source: ISI Web of Knowledge 2012, Journal Citations Reports (JCR) via www.webofknowledge.com (accessed on 12 June 2015). ENT: ear, nose and throat.

Search strategy

A PubMed database search was conducted on June 12, 2015 using four predefined filters. First, an adapted version of the ENT filter developed by the Cochrane ENT group was used to retrieve articles conducting research in otorhinolaryngology.[9] Second, a filter was applied to only retrieve research using animal models.[10] Subsequently, date restrictions were applied per journal category to limit the amount of retrieved articles. We searched PubMed for articles published in ENT journals in the year 2014. Since less otorhinolaryngology related articles are published in multidisciplinary journals, we searched for articles conducting animal experiments in otorhinolaryngology research published in multidisciplinary journals from 2010 to 2014. It is important to note that the ARRIVE guidelines were first published in 2010. Thus, studies published in multidisciplinary journals in 2010 might have been written prior to the publication of these guidelines. An analysis was performed to investigate correlations between year of publication and quality of reporting.

The complete search syntax with specific filters is outlined in Supplemental digital content 1 (see http://journals.sagepub.com/doi/full/10.1177/0023677217718862 for all supplementary materials in this article).

Study selection

Two authors (SFLK and JPMP) independently screened titles, abstracts and full texts of the retrieved articles and selected those reporting in vivo animal experiments. To be considered for inclusion, studies must have assessed preclinical phases of diseases or disorders commonly treated by otorhinolaryngologists. Discrepancies between the two reviewers were discussed until consensus was reached.

Scoring articles

To assess the quality of reporting of articles, two authors (AB and SFLK) independently scored articles using the checklist from the ARRIVE guidelines. The checklist contains 20 points, some with subsections (a, b, c or d). Subsections were considered as separate items for scoring, yielding a total of 38 items. Two items on the ARRIVE checklist (10c and 15b) were optional and were rarely applicable. Therefore, to standardize our assessment of quality of reporting in all articles, these two items were excluded from the analysis. In total, 36 items were scored for each article.

Supplemental digital content 2 summarizes the scoring criteria per item. Articles were reviewed in order to extract all provided information. This included supplementary information; available online or in appendices. No more than five articles per journal category were scored consecutively to distribute possible learning effects evenly across the two journal categories.

Inter-observer agreement

Cohen's kappa value for inter-observer agreement was evaluated to analyze discrepancies among the scorers. Cohen's kappa was calculated for the complete dataset, and per item.

Data analysis

Descriptive statistics including median and mean scores of adequately reported ARRIVE items were calculated. A two-tailed Mann–Whitney U-test for two independent samples was used to evaluate significant differences between the two journal categories. Chi-square analysis was used to evaluate each ARRIVE item between the two journal categories. For the articles published in multidisciplinary journals, a correlation between year of publication and quality of reporting was investigated using Spearman's rank correlation coefficient (Spearman's rho).

Statistical tests were performed using the SPSS v20 statistics package (IBM, Armonk, NY, USA). Statistical significance was set at 5%.

Results

Search and study selection

The combined search syntaxes (Supplemental digital content 1) yielded 51 articles published in ENT journals, and 63 articles in multidisciplinary journals. Figure 1 summarizes the search and study selection process.

Figure 1.

Flow chart demonstrating study selection. ENT: ear, nose and throat.

Of the 51 articles retrieved from ENT journals, 11 were not primary research articles and five did not involve in vivo animal experiments. Therefore, 35 articles from ENT journals were included in the analysis (Hear Res: n = 15, JARO: n = 11, Head Neck: n = 9).

Of the 63 articles retrieved from multidisciplinary journals, 18 were not related to otorhinolaryngology research, five did not report on primary research and four did not include in vivo animal experiments. Therefore, 36 articles were included in the analysis (PNAS: n = 24, Nature: n = 4, Nat Commun: n = 4, Sci Rep: n = 3, Science: n = 1). Eight articles were published in multidisciplinary journals that endorse the ARRIVE guidelines. Six articles were published in multidisciplinary journals in 2010, eight in 2011, eight in 2012, nine in 2013 and five in 2014.

The numbers of retrieved and selected articles per journal are summarized in Table 2.

Table 2.

Retrieved articles by search and included articles following study selection.

	Retrieved	Included
ENT journals
Hear Res	26	15
JARO	13	11
Head Neck	11	9
Ear Hear	1	0
Audiol Neurotol	0	0
Total	51	35
Multidisciplinary journals
PNAS	39	24
Nature	13	4
Nat Commun	6	4
Sci Rep	3	3
Science	2	1
Total	63	36

ENT: ear, nose and throat.

Overall quality of reporting scores

The 35 articles published in ENT journals reported a mean of 57.1% adequately scored items (95% confidence interval [CI]: 53.4–60.9%; median: 58.3%). The 36 articles published in multidisciplinary journals reported a mean of 49.1% adequately scored items (95% CI: 46.2–52.0%; median: 50.0%). The overall difference between the journal categories was statistically significant (Mann–Whitney U-test, P = 0.001), suggesting that ENT journals adhered better to the ARRIVE guidelines.

For the articles published in multidisciplinary journals, there was no statistically significant correlation between the year of publication and the number of adequately reported ARRIVE items (P = 0.083). Moreover, there was no significant difference in the quality of reporting between the eight articles published in multidisciplinary journals that endorsed the ARRIVE guidelines (Nature, Nat Commun), and the 28 articles published in journals that did not endorse the ARRIVE guidelines: 51.4% (95% CI: 45.6–57.2%, median: 54.2%) compared with 48.4% (95% CI: 45.1–51.7%, median: 50.0%), respectively.

Quality of reporting for specific items

When examining ARRIVE items separately, five items (6a, 7a, 9c, 14, 18b) were scored significantly higher in the articles published in ENT journals (Figure 2, Table 3). These items assessed if the study reported the number of experimental and control groups (6a), information on the drug dose, site and route of administration, and surgical procedure and equipment used (7a), welfare-related assessments and interventions carried out prior, during, or after experiments (9c), information on health status of animals prior to treatment or testing (14) and study limitations (18b).

Figure 2.

Adequately reported scores per item according to the ARRIVE guidelines. *Significant difference (chi2) between journal type. Abstr: abstract; Adv ev: adverse events; Alloc: allocation of animals; Backgr: background; Exp an: experimental animals; Basel: baseline data; Exp out: experimental outcomes; Exp proc: experimental procedures; Fund: funding; Gen: generalisability; H & H: housing and husbandry; Interpr: interpretation; Nb an: number of animals; Obj: objectives; Study des: study design; SS: sample size; Stats: statistical method.

Table 3.

Proportion of adequately reported ARRIVE items per journal category.

Item:	1: Title	2: Abstract	3a: Background	3b: Background	4: Objective	5: Ethic	*6a: Study design	6b: Study design	6c: Study design	*7a: Exp proc	7b: Exp proc	7c: Exp proc	7d: Exp proc	8a: Exp an	8b: Exp an	9a: H & H	9b: H & H	*9c: H & H
Articles published in ENT journals (n = 35)	69%	91%	100%	86%	91%	94%	83%	23%	89%	89%	9%	46%	17%	83%	49%	20%	20%	46%
Articles published in multidisciplinary journals (n = 36)	56%	89%	100%	69%	83%	92%	53%	11%	72%	56%	3%	42%	11%	61%	69%	17%	25%	17%
P value chi2 (2-tailed)	0.330	1.000	NA	0.155	0.478	1.000	0.011*	0.220	0.135	0.003*	0.357	0.813	0.514	0.064	0.094	0.767	0.778	0.011*
Item:	10a: SS	10b: SS	11a: Allocation	11b: Allocation	12: Exp outcome	13a: Stats	13b: Stats	13c: Stats	*14: Basel	15a: Nb an	16: Outcome	17a: Adv ev	17b: Adv ev	18a: Interpretation	18b: Interpretation	18c: Interpretation	19: Gen	20: Fund
Articles published in ENT journals (n = 35)	74%	0%	20%	11%	97%	89%	80%	14%	29%	71%	94%	20%	6%	97%	49%	9%	97%	97%
Articles published in multidisciplinary journals (n = 36)	56%	6%	14%	3%	100%	78%	58%	11%	3%	86%	97%	6%	6%	100%	22%	3%	94%	100%
P value chi2 (2-tailed)	0.137	0.493	0.541	0.199	0.493	0.343	0.072	0.735	0.003*	0.155	0.614	0.085	1.000	0.493	0.026*	0.357	1.000	0.493

Significant difference (chi2) between journal categories. Adv ev: adverse events; Basel: baseline data; Gen: generalisability; Fund: funding; Exp an: experimental animals; Exp outcome: experimental outcome; Exp proc: experimental procedures; H & H: housing and husbandry; n: number; Nb an: number of animals; SS: sample size; Stats: statistical methods.

Several items were not adequately reported in both journal categories: 10 items were reported less frequently than 20% in both journal categories (Figure 2). These items include the time of day when experiments were carried out (7b), the rationale behind the choice of the specific anesthetic, its dose and route of administration opted for (7d), information regarding housing of animals (9a), sample size calculation (10b), allocation of the animals to groups (11a,b), methods used to assess whether the data met the assumptions of the statistical approach (13c), reporting of adverse events (17a,b), and implications of the experimental methods or findings for the replacement, refinement or reduction of the use of animals in research (18c).

Out of a total number of 2556 scored items, 158 (6.1%) were scored differently. Cohen's kappa value for inter-observer agreement was 0.87 (standard error = 0.10). A Cohen's kappa score between 0.61 and 0.80 suggests a good agreement between independent scorers.[11] Cohen's kappa value for inter-observer agreement per item is presented in Supplemental digital content 3. The inter-observer agreement was high for most items, and there were no Cohen's kappa values lower than 0.3.[11]

Discussion

The present study evaluated the quality of reporting of scientific publications using animal models in otorhinolaryngology research. Articles published in ENT journals adhered better to the ARRIVE guidelines than articles published in multidisciplinary journals. Therefore, articles published in multidisciplinary journals with high impact factors do not have a superior overall quality of reporting in otorhinolaryngology research using animal models. Similarly, MacLeod et al. have identified significantly fewer reporting of randomization in articles published in journals with high impact factors.[12] Our findings are contrary to reports investigating the quality of reporting of randomized controlled trials[13] and systematic reviews[14] in otorhinolaryngology research, where ENT journals underperformed.

Interpretation of results

Although ENT journals showed better quality of reporting, adherence to the ARRIVE guidelines is generally poor in otorhinolaryngology research for both journal categories. Items such as choice of the specific anesthetic, dose and route of administration (7d) and information regarding the housing of animals (9a) were rarely (<20% of all studies) reported. This information is essential for accurate replication of animal experiments, as it may influence study outcomes.[15] Prager et al. reported that housing and husbandry information of animals have the potential to influence responses of rodents, and thus alter study outcomes.[16] Our findings also revealed that sample size calculation for the number of animals chosen per group (10b) and allocation of the animals to groups (11a,b) were rarely reported (<20% of all studies). These two items are essential for optimizing statistical design, and for fulfilling ethical obligations, as they aim to reduce potential bias and the number of animals used in research.[1,17]

Articles published in multidisciplinary journals often described additional experiments alongside the animal model. As such, the animal experiment could have not been the primary focus of the study. Nevertheless, all multidisciplinary journals included had a methodology section containing information relating to the animal experiments. These sections do not have word limits that may have justified the missing information.

Similar outcomes are found in other disciplines. Gulin et al. performed a quality assessment review of animal studies for Chagas disease by comparing studies published before and after the publication of the ARRIVE guidelines. In line with our findings, their study revealed that items such as randomization (16%) and sample size calculations (7%) were rarely reported.[18] Ting et al. investigated interventional animal studies in rheumatology and reported missing information such as randomization (17%), sample size calculation (0%), allocation (0%), housing, husbandry and welfare-related information (5%), and implications for replacement, refinement or reduction of the use of animal assessments (0%).[19] These items are essential to reduce bias in scientific research, and to make experiments transparent and replicable.[20] Furthermore, Schwarz et al. reviewed publications on preclinical research for the treatment of mucositis/peri-implantitis, Freshwater et al. conducted reviews on animal research published in plastic surgery journals, and Tsilidis et al. investigated the reporting of animal models for neurological diseases.[21-23] All studies concluded that there is an urgent need for improving the quality of reporting when using animal models.

Methodological considerations

Strengths of our study include a search strategy that could be reproduced to evaluate the quality of reporting of animal studies in other disciplines. To account for learning effects, the two authors who independently scored 2556 items did not score more than five articles consecutively per journal category. The limitations of the study include firstly a subjective assessment by the two independent scorers. The scorers were also not blinded to which journal category the paper belonged. However, the high inter-observer agreement demonstrated that both reviewers had fairly similar judgment (Supplemental digital content 3). Second, in order to obtain a sufficient amount of articles, we included articles published in multidisciplinary journals from 2010–2014, whereas we included articles published in ENT journals in 2014 only. Since the ARRIVE guidelines were first published in 2010, articles published that year could not have had access to these guidelines. Nevertheless, a subanalysis revealed no correlation between the year of publication and the quality of reporting. A third limitation is that Nature and Nat Commun have recommended that authors use the ARRIVE checklist (date inspected: 12 June 2015). However, no statistical difference was found in the quality of reporting between articles published in multidisciplinary journals that endorsed the ARRIVE guidelines and those that did not. Finally, our findings were only based on eight journals (ENT journals: Hear Res: n = 15, JARO: n = 11, Head Neck: n = 9; multidisciplinary journals: PNAS: n = 24, Nature: n = 4, Nature Comm: n = 4, Science Rep: n = 3, Science: n = 1).

Reporting guidelines

Evidence that clinical trials lacked crucial methodological information led to the development of the Consolidated Standards for Reporting Trials (CONSORT) statement, which is now implemented by many journals and funding agencies. Implementing the CONSORT statement has been shown to drastically improve the quality of reporting of clinical trials.[24-27] By contrast, the development of the ARRIVE guidelines did not enhance quality of reporting when comparing articles appearing before and after the ARRIVE guidelines were published.[28] Baker et al. showed that reporting of animal research in PLoS journals, which have been early proponents of the ARRIVE guidelines, still remained low.[28] In our sample, we also showed that there was no improvement in the quality of reporting with increasing year of publication (2010–2014). Therefore, we recommend a stronger endorsement of the ARRIVE guidelines from authors, journal editors and funding agencies.

Conclusion

Although articles using animal models published in ENT journals have better quality of reporting scores than those published in multidisciplinary journals, adherence to the ARRIVE guidelines is generally poor in otorhinolaryngology research. There is an urgent need to improve the quality of reporting in otorhinolaryngology research using animal models. Editorial endorsement of the ARRIVE guidelines from authors, research and academic institutes, editorial offices, and funding agencies is warranted to optimize quality of reporting.