Influential variables in the Journal Impact Factor of Dentistry journals.

OBJECTIVE: The aim of this contribution is to determine what variables influence the position, by quartiles of the impact factor, as a quality indicator of a journal in the field of Dentistry.
METHODS: To this end, 24 journals included in Journal Citation Reports, 6 pertaining to each quartile were selected by a stratified sampling and then an ordinal regression model was estimated stepwise considering the journal impact factor quartile as response variable.
RESULTS: The estimation procedure concluded that the average number of papers published yearly by a journal and the percentage of systematic reviews are the most significant variables to be considered, along with the factor representing the journal's degree of adherence to recommendations by the International Committee of Medical Journal Editors. CONCLUSIONS/CLINICAL SIGNIFICANCE: Systematic reviews have significant effect on the Journal Impact Factor position of a journal as well as adherence to ICMJE recommendations, while papers publishing clinical trials bear no influence on this factor. Greater yearly average of published papers in a journal means a higher impact factor.

Introduction

What factors can influence the position that a journal occupies in the list of Journal Citation Reports (JCR) of a certain field? This is a question of great importance for editors and authors of scientific papers, as it may be determinant for positive evaluations in competitive calls for researching projects and University positions. In this sense, a recent paper by Robinson-Garcia et al. [1] analyze seven world university rankings and performed a principal component analysis in order to show that ranking scores can be explained by the number of publications and citations received by the institution.

On the other hand, one of the most used indicator of journal quality is the Journal Impact Factor (JIF). Others, such as the Eigenfactor Score, Article Influence Score or 5-year JIF, are increasingly taken into account because of the limitations and disadvantages of the JIF [2, 3, 4]. But, in most cases, the numerical value of the JIF per se, is not as important as the journal's position on a list specified by order statistics such as deciles, terciles, quartiles, etc. The research system in Spain, for example, assesses a classification by quartiles, sometimes also considering journals included in the first decile.

Some backgrounds in the application of regression models with categorical response can be found in the work by Bornmann and Daniel [5] where they examined following this methodology the peer review process at a Chemistry journal for evidence of potential sources of bias. Moreover Bravo et al. [6] tested an ordered logistic model with a cumulative link function to predict whether a paper was accepted, invited for resubmission with minor revision, with major revision, or rejected on the basis of authors’ reputation. A previous paper on this topic by Valderrama et al. [7] took into account as a dependent variable the tercile, and as explanatory ones the h-index of the editor-in-chief, the percentage of papers reporting research supported by an external grant, and the scope and the internal structure of the journal. Factors such as the language in which the paper is written [8] or the effect of self-citations [9, 10] may also prove influential.

The field Dentistry, Oral Surgery and Medicine has been selected in this study because it could be considered as a global representation of Medicine but in oral area; in fact its journals take in very diverse fields, from basic science to clinical and community studies and it appears to have grown in a similar evolution to Medicine [11]. On the other hand, its self-citation rate indicates a healthy scientific publishing environment [12].

The aim of this paper is to analyze, by statistical methods, which variables are the most significant in terms of journal position impact, concretely we will estimate an ordinal regression model to do it. Both, categorical variables (factors) and numerical ones (covariables), having greater influence on the JIF position of a journal in the field of Dentistry were selected from a broader set of variables including the average number of papers published yearly by the journal, the percentage of systematic reviews and clinical trials, the degree of adherence to International Committee of Medical Journal Editors (ICMJE) recommendations, and category of the journal according to its subject area.

Methodology

The field Dentistry, Oral Surgery and Medicine included in 2018 a set of 90 journals. Because of the calculation of values of the variables considered in the study is laborious, we selected a sampling fraction of a quarter, what it supposes by approximation 24 journals, through stratified random sampling by quartiles with the same affixation, so that 6 journals pertained to each quartile (see Table 1). The dependent categorical variable was precisely the quartile. The 2018 JIF of each journal was obtained from Web of Science (WoS) Journal Citation Reports [13] and number of narrative reviews, systematic reviews and clinical trials from WoS Core Collection database with access date 2019-06-03. For each journal, the 2013–2017 term was selected.

Table 1

Sampled journals together with their JIF and quartile (Q), covariables and factors considered in the study, and probability of belonging to each quartile according to the estimated model (EstQ18).

Journal	JIF18	Q18	Covariables				Factors		Probability of belonging to Q				EstQ18
			Av pap	% NR	% SR	% CT	ICMJE	Categ	Q1	Q2	Q3	Q4
1. J Dent Res	5.125	1	215.2	16.64	5.11	5.39	2	1	0.840	0.159	0.001	0.000	1
2. J Clin Periodont	4.164	1	140.4	12.96	10.83	20.94	2	2	0.771	0.228	0.001	0.000	1
3. Oral Oncol	3.730	1	215.8	18.91	4.26	4.17	2	2	0.715	0.284	0.002	0.000	1
4. J Dent	3.280	1	177.4	12.51	9.70	14.99	2	1	0.929	0.071	0.000	0.000	1
5. J Endod	2.833	1	326.8	6.61	3.24	6.55	2	3	0.995	0.005	0.000	0.000	1
6. J Periodont	2.768	1	206.8	7.64	6.00	14.60	2	2	0.873	0.126	0.001	0.000	1
7. Comm Dent Oral Epidem	2.278	2	65.0	7.69	7.08	6.46	2	3	0.012	0.732	0.253	0.003	2
8. Int J Paediatric Dent	2.057	2	65.8	8.51	6.38	10.33	2	1	0.008	0.650	0.338	0.004	2
9. BMC Oral Health	2.048	2	139.0	4.75	4.03	7.91	2	3	0.041	0.873	0.086	0.001	2
10. J Cranio Maxillofac Surg	1.942	2	246.6	4.19	2.50	6.91	2	1	0.556	0.440	0.003	0.000	1∗
11. Angle Orthod	1.880	2	152.2	4.07	3.55	10.51	0	3	0.015	0.771	0.212	0.002	2
12. Int J Oral Maxillofac Implants	1.734	2	188.0	8.30	6.60	7.45	1	2	0.222	0.764	0.014	0.000	2
13. Int Dent J	1.628	3	56.2	11.03	2.49	6.41	2	1	0.000	0.022	0.721	0.257	3
14. Acta Odontol Scand	1.565	3	131.2	6.55	2.44	7.47	1	1	0.000	0.058	0.831	0.111	3
15. J Appl Oral Sci	1.506	3	86.0	3.49	1.40	5.35	2	1	0.000	0.009	0.535	0.456	3
16. J Dent Education	1.506	3	153.4	3.65	2.22	2.87	2	1	0.007	0.635	0.353	0.004	2∗
17. Australian Dental J	1.282	3	92.2	18.87	2.82	4.99	1	1	0.000	0.016	0.659	0.325	3
18. Int J Dent Hyg	1.233	3	49.0	9.80	8.16	25.31	1	3	0.001	0.146	0.811	0.043	3
19. Cranio-J Craniomand Pract	1.144	4	54.2	10.70	1.85	2.95	1	2	0.000	0.000	0.051	0.949	4
20. Community Dent Health	1.079	4	49.0	5.31	1.63	3.67	0	3	0.000	0.001	0.096	0.903	4
21.Orthodont Craniofac Res	0.946	4	39.8	13.07	7.54	6.03	2	3	0.004	0.516	0.473	0.007	3∗
22. J Orof Orthopedics Fortsch	0.927	4	47.4	1.69	1.69	18.99	0	3	0.000	0.001	0.102	0.897	4
23. Oral Health Prev Dent	0.902	4	58.8	9.86	0.34	12.93	1	3	0.000	0.000	0.000	1.000	4
24. Australian Orthodont J	0.269	4	30.2	5.30	0.00	3.31	2	3	0.000	0.000	0.000	1.000	4

Average number of papers published annually (Av pap); percentage of narrative reviews (%NR) systematic reviews (%SR) and clinical trials (CT); adhesion to ICMJE recommendations (ICMJE); and journal category (Categ).

The highest probability of belonging to a quartile is marked in bold.

Wrong classification.

The following document types were excluded from the total: meeting, abstracts, corrections, retraction, unspecified, proceeding paper, reference materials, news and bibliography. The remaining documents were considered as total number of contributions of the journal. In document types, the number of clinical trials and reviews was obtained, and the number of systematic reviews by delimiting the topic systematic review∗ with the option refine results.

Once the number of narrative and systematic reviews and clinical trials was obtained, the corresponding percentages were calculated by dividing over the total of contributions. Because percentages are variables bounded between 0 and 1, a logit transformation was applied to the percentage of systematic reviews and clinical trials in order to obtain new variables in the interval (-∞,∞):

Likewise, the average number of contributions published annually was estimated by considering the total number of documents published in the last five years and dividing by five. All these data can be found in Table 1.

To evaluate the degree of adhesion to ICMJE recommendations, after reading the instructions to authors of each journal, a value was assigned to each journal according to the scale: Not mentioned (0), recommended (1) or required (2). A code was then assigned to the journals on the basis of the following classification:

Category 1: General scope including Odontopediatrics and Gerodontology

Category 2: Oral Medicine, Oral Surgery, Implantology and Periodontology

Category 3: Orthodontics, Restorative Dentistry, Endodontics, Prosthetics, Public Health and others

Once the final data had been collected, an ordinal regression model was estimated by means of a stepwise method to indicate the probability that a given journal would belong to a certain quartile. Calculations were performed by means of R software, version 3.4.4, for x86_64-pc-windows-gnu (www.R-project.org), as describes R Core Team [14]. The goodness of fit of the set of explanatory variables depends on two measures:

The pseudo coefficient of determination of Nagelkerke [15].

The correct classification rate (CCR) of quartile estimated by the model [16].

Results and discussion

The stepwise method for estimating the ordinal regression model provided as significant variables on the JIF quartile position the following ones: Adherence to ICMJE recommendations (factor), average number of papers published annually in the journal (covariable 1) and percentage of systematic reviews (covariable 2). The associated R2 Nagelkerke coefficient was 0.839 (near to 1) and the Correct Classification Rate 87.5%. On the other hand, the excluded variables (category and percentage of narrative reviews and clinical trials) have not significant effect on the JIF quartile position of the journal. Besides the above information, Table 1 gives the estimated probability of a journal belonging to each of the four quartiles, together with the one it actually pertains to. This situation can also be visualized in Figure 1.

Figure 1

Probability of journals belonging to each quartile.

In spite of the limitations and criticism surrounding the JIF as a bibliometric indicator of journal quality, in the framework of this paper it was considered as a reference variable since it is still the gold standard in Bibliometrics [17]. Our overall aim was to explain the position of a journal according to its quartile on the Journal Citation Reports list, within the field Dentistry, Oral Surgery and Medicine. The set of explanatory variables included: average number of papers published yearly in the journal, percentage of reviews, systematic reviews and clinical trials, adherence degree to ICMJE recommendations, and category of the journal according to its subject area. After introducing all of these as independent variables in the ordinal regression model, it was observed that neither reviews, clinical trials nor journal category had a significant influence on the response; for this reason we proceeded to re-estimate the model using a stepwise criteria for the selection of variables.

The new ordinal model presented a Nagelkerke coefficient somewhat lower than the complete one including all the variables because, as with the linear determination coefficient, the higher its value will be as more variables are considered in the model. But, on the other hand, its CCR is higher going from 84.4% when all variables are introduced in the model to 87.5% with the three selected ones.

It is logical to think that the greater the number of issues published in a journal, the greater the probability of receiving more citations. In the calculation of the JIF this would be corrected by the denominator, however, this only includes citable items (articles and reviews). A journal that publishes many issues it is also likely that in addition to articles and reviews include other issues such as conference papers, letters, proceeding, abstracts, etc., that can be cited and would be part of the numerator boosting the JIF, but they are not taken into account to be part of the denominator, which would lead to a questionable increase in the JIF. In fact this is one of the criticisms that this bibliometric receives [18].

Review papers, and particularly systematic reviews including meta-analysis, constitute the best scientific evidence and are highly useful guides for researchers and practitioners when clinical decisions must be made. They facilitate access to knowledge while serving as a source of citations that increase the JIF. This fact is, in part, confirmed by Miranda and García-Carpintero [19] proving that review papers are cited more frequently than regular research articles, especially in the Biomedical field. In fact, more and more review articles are being published that can reach epidemic proportions [20].

The excess or abuse of these publications may be detrimental to the publication of original and less citable research results than the reviews, which could contribute to reducing the global influence of the journal in the scientific community [21]. An example would be the non-influence of the percentage of clinical trials in the model, when it is a type of quality clinical research. On the other hand, the lack of following specific recommendations to enhance the quality of a paper increases the possibility that unreliable or biased studies be published. Given that ICMJE recommendations have been approved by over 2600 biomedical journals around the world [22], they can be considered to reflect quality, which justifies their inclusion as an influential indicator for determining the JIF.

When interpreting the results, it must be taken into account that this research has focused on the dental area. It would be interesting to test in further studies if these conclusions could be applied in other fields in Medicine or Nursing.

Conclusions

After estimating the ordinal regression model from recorded data, the main conclusions of the study can be summarized as follows:

Systematic reviews have significant effect on the JIF position of a journal

Greater yearly average of published papers in a journal means a higher impact factor

High quality journals usually conform to ICMJE recommendations

Papers publishing clinical trials bear no influence on the JIF of a journal

Declarations

Author contribution statement

Pilar Valderrama: Conceived and designed the experiments; Performed the experiments.

Manuel Escabias: Contributed reagents, materials, analysis tools or data.

Mariano J. Valderrama: Analyzed and interpreted the data; Wrote the paper.

Evaristo Jiménez-Contreras: Analyzed and interpreted the data.

Pilar Baca: Conceived and designed the experiments; Wrote the paper.

Funding statement

This work was supported by project MTM2017-88708-P of the Secretaría de Estado de Investigación, Desarrollo e Innovación, Ministerio de Economía y Competitividad, Spain, and researching groups CTS-167 and FQM-307 of PAI Junta de Andalucía, Spain.

Competing interest statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.