Ranking of addiction journals in eight widely used impact metrics.

Background and aims: Journal metrics assess impact upon the research literature, and are now used to assess individual researchers in hiring and promotion decisions. This study compared the ranking of addiction journals according to eight widely used metrics; assessed the correlations between journal rankings; and assessed changes over time in metric scores.
Methods: Data pertaining to the 2020 scores on eight metrics for 43 journals were obtained and the top 20 ranking in each compared and the correlations between rankings assessed. The Impact Factor was employed to assess changes over time.
Results: Ignoring the two categorization systems used by some metrics, 31 journals appeared in at least one metric top 20 and 11 in all eight. The top rank in each was occupied by one of three journals. Three-quarters of the correlations between rankings were above 6.0. The number of journals with an Impact Factor rose from 23 in 1997 to 38 in 2020, and the journals added tended to focus on addictions other than alcohol and drugs or have a specific focus. Conclusions and discussion: The results indicate a concentration of journals at the top of the metrics and moderate to strong agreement between them, but almost three-quarters of journals appeared in at least one metric. The longitudinal data reflect both a broadening and specialization of the addiction field. The study limitations include exclusion of some journals and metrics.

Introduction

The use of metrics to assess the quality and impact of academic journals first occurred almost one hundred years ago and their widespread use was facilitated by the introduced in the 1960s of the Journal Impact Factor (IF) (Archambault & Lariviére, 2009; Garfield, 2006). Since then, a number of other journal metrics have been developed in an attempt to address the limitations of the IF, such as inclusion of self-citations, limited time-period coverage and failure to account for variation in citation behavior across disciplines (Delgado López-Cózar & Cabezas-Clavijo, 2013; Roldan-Valadez, Salazar-Ruiz, Ibarra-Contreras, & Rios, 2018). Although they have been the subject of considerable criticism due to lack of transparency, the ease with which they can be manipulated and gamed, and their poor association with journal quality (Brembs, Button, & Munafò, 2013; Delgado López-Cózar, Robinson-Garcia, & Salina-Torres, 2014; Edwards & Roy, 2017), such metrics appear to be too well-established within academia to discard and, over the years, their use has expanded beyond the assessment of journal performance. For example, they continue to be used to assess the quality of individual candidate's research in faculty hiring, promotion and tenure decisions, and by funding agencies when reviewing grant applications (McKiernan et al., 2019; Roldan-Valadez, Salazar-Ruiz, Ibarra-Contreras, & Rios, 2018).

The problems with using journal metrics for such purposes have been highlighted in recent documents produced by professional associations designed to improve the assessment of academic research and individual scholars, such as the Leiden Manifesto (Hicks, Wouters, Waltman, de Rijcke, & Rafols, 2015) and the San Francisco Declaration of Research Assessment (DORA; American Society for Cell Biology, 2013). These include their insensitivity to the heterogeneity among research fields and to the socio-economic and cultural context within which research is conducted, the myopia they encourage in reviewers when conducting assessments of colleagues, and the erroneous sense of precision they create in measures that are conceptually ambiguous and subject to random fluctuation over time. In additional, when remuneration in the form of bonuses and salary increases is tied to publishing in specific “high impact” journals, a perverse incentive is introduced into the academic rewards structure that can undermined research integrity (Moher et al., 2020).

Much of the criticism of journal metrics has focused on the IF, the most commonly used metric in journal assessment and the one most frequently highlighted in journal editorial material (Hicks et al., 2015). Some of this criticism is focused on the way in which the IF is calculated, specifically its counting of self-citations and popular science articles, the inclusion of letters and editorials in the numerator but not the denominator of the calculation, and the extent to which it can be inflated by the publication of certain types of articles (e.g., reviews) over others (e.g., original research) (Dong, Loh, & Mondry, 2005; Liu & Fang, 2020; Roldan-Valadez et al., 2018). Brembs et al. (2013) present a very detailed critique of the IF, focused on the degree to which it actually reflects the quality of the research published by journals. Reviewing literature primarily from medicine and neuroscience, they report that the IF is a poor predictor of methodological rigor when assessed by statistical power of the analyses reported, adherence of articles to reporting guidelines and publication of studies that over-estimate true effect sizes. Tressoldi, Giofre, Sella, and Cumming (2013) also found that use of good statistical practices, such as the reporting of effect sizes and confidence intervals rather than P-values (recommended by groups such as the International Committee of Medical Journal Editors (ICMJE) and CONSORT), varied widely across journals, whether their IF was high or low.

Problems with the specific criteria and calculation method used in the IF have been addressed by the development of other metrics that have introduced modifications such as only counting citations of peer-reviewed publications, using different data sources for citations, and using more complex algorithms or longer citation windows (Liu & Fang, 2020; Pajić, 2015; Roldan-Valadez, Salazar-Ruiz, et al., 2018). A recent series of studies focused on journals from within a range of medical specialties including radiology, nuclear medicine, neurosciences and gastroenterology found that the IF failed to predict the two-year-ahead total citations of journals, whereas other metrics (such as the Eigenfactor score) were significant predictors of this outcome (see Roldan-Valadez, Irbe-Arteaga, & Rios, 2018, for details). However, the issues raised by Brembs et al. (2013) will likely affect any metric that becomes commonly used: pressure to publish novel positive results that are more likely to be unreliable will remain and these results will be submitted to the journals most highly ranked on any metric that has displaced the IF as the coin of the realm. And there is no guarantee that these journals will be any better than those with high IFs at ensuring adherence to reporting and statistical guidelines and identifying in the review process manuscripts that over-estimate effect sizes or selectively report positive results. In addition, according to Campbell's law, whichever metric is used in academic decision-making (such as where to publish, who to fund, who to promote), it will be subject to gaming and to corruption of the very thing, in this case quality, it is intended to assess (Campbell, 1979).

Given the widespread and varied uses and misuses of journal metrics, it is worth examining their application within specific disciplines and trying to assess whether they tell us anything of value. Such research has been conducted within a number of academic disciplines, including emergency medicine, rheumatology, general medicine and psychiatry (Diaz, Soares, Brambilla, Young, & Selvaraj, 2021; Rizkallahm & Sin, 2010; Robriguez, Chasn, Wong, & Monto, 2020; Tazegul, Emre, Öğüt, Yazısız, 2021). To date, no such assessment of the ranking of addiction journals by metrics has been conducted and the present paper addresses this gap in the literature. Specifically, it examines the ranking of 43 journals that appear on eight of the most widely used metrics. It also presents changes over a 24-year period in one of the metrics assessed.

Methods

Data pertaining to 2020 rankings of the following eight metrics were collected:

Google Scholar h5-index, which “is the h-index for articles published in the last 5 complete years. It is the largest number h such that h articles published in 2016–2020 have at least h citations each” (Google Scholar, 2021).

Journal Citation Reports Journal Impact Factor (IF), which is “a ratio that divides a journal's received citations by a count of its published articles” (Web of Science Group, 2021). The 2-year IF was selected rather than the 5-year, as this is the metric more commonly used.

Eigenfactor, which “is based on the number of times articles from the journal published in the past five years have been cited in the JCR year, but it also considers which journals have contributed these citations so that highly cited journals will influence the network more than lesser cited journals” (Incites Journal Citation Reports Help, 2021).

Journal Citation Indicator (JCI), which “provides a field-normalized measure of citation impact where a value of 1.0 means that, across the journal, published papers received a number of citations equal to the average citation count in that subject category” (Szomszor, 2021).

Scimago Journal and Country Rank H-index, which “expresses the journal's number of articles (h) that have received at least h citations” (Scimago Journal and Country Rank, 2021a, 2021b).

CiteScore, which “counts the citations received in 2017–2020 to articles, reviews, conference papers, book chapters and data papers published in 2017–2020, and divides this by the number of publications published in 2017–2020” (Elsevier, 2021).

SCImago Journal Rank (SJR), which “weights each incoming citation to a journal by the SJR of the citing journal, with a citation from a high-SJR source counting for more than a citation from a low-SJR source” (Elsevier, 2021).

Source Normalized Impact per Paper (SNIP), which compares “each journal's citations per publication with the citation potential of its field, defined as the set of publications citing that journal” (Elsevier, 2021).

With the exception of the JCI, which was introduced in 2021, detailed descriptions of these metrics can be found in Rolden-Valadez et al. (2018) and Delgado López-Cózar and Cabezas-Clavijo (2013).

The h5-index data were obtained from Google Scholar, the IF, Eigenfactor and JCI scores from Clarivate Journal Citation Reports through the Web of Science, the H-index data from the Scimago Journal and Country Rank, and the CiteScore, SJR and SNIP data were obtained from Scopus. The Google Scholar, Web of Science and Scopus datasets were accessed through the Texas A&M Library and the Scimago H-index data were downloaded from the Internet. Data pertaining to the 2020 JCI ranking were downloaded on January 28, 2022. Data pertaining to the 2020 rankings on the other seven metrics were downloaded between July 30 and August 4, 2021.

The current paper uses a format similar to Diaz et al. (2021) in that it first presents the top 20 ranking of journals in each metric. It then presents the correlations (with 95% confidence intervals and P-values) between the eight metrics' rankings of all 43 journals. In describing the strength of a correlation, the thresholds proposed by Villaseñor-Almaraz, Islas-Serrano, Murtata, and Roldan-Valadez (2019) were used. Specifically, 0.80 and above was considered a “very strong” correlation, 0.60–0.79 a “moderately strong” correlation, 0.30–0.59 a “fair” correlation, and less than 3.0 a “poor” correlation.

Kendall's Tau-B, as implemented in Stata 17 (StataCorp, 2021), was used to estimate the correlations between the journal ranks. A Bonferroni correction was used to adjust the P-values for multiple comparisons, and Efron's Bootstrap method was used to estimate confidence intervals. These methods were chosen for several reasons. First, the normal approximation for the sampling distribution of Kendall's Tau is achieved for smaller sample sizes than Spearman's rho. Thus, the P-values will be more accurate for smaller sample sizes. Second, Kendall's Tau can be interpreted as the probability of observing concordant pairs. A concordant pair is defined as both ranks for one observation being larger (or smaller) than both ranks for another observation. For example, if the h-5 index rank of Journal A is 1 and its IF rank is 4, the pair is (1, 4). In contrast, the ranks on the same two indices for Journal B, are (2, 7). In this hypothetical case, both ranks of the pair for Journal A are smaller than the ranks of the pair for Journal B, so the rank pairs are concordant. Third, pairs of ranks are sometimes neither concordant or discordant which results in a tie. Kendell's Tau B adjusts for such ties among the ranked pairs. Fourth, Hollander and Wolfe (1999) have suggested using Efron's Bootstrap method to estimate confidence intervals for Kendall's Tau. Confidence intervals based on the bootstrap method have been shown to produce accurate estimates when the assumptions underlying the normal approximation are not met. All of the confidence intervals presented are bias-corrected and used bootstrap sample sizes of 5,000.

The JCR IF was used to assess changes over time in metric scores. These data were downloaded from the Clarivate Journal Citation Reports webpage through the Web of Science on August 23–25, 2021. The JCR IF was chosen for this longitudinal analysis as data for it are available as far back as 1997 and because it is the most common metric used in evaluating journals and the one most emphasized in journal editorials (Brembs et al., 2013; Hicks et al., 2015).

The data reported pertain to journal metrics and are available online. No IRB approval is required for such data that is in the public domain.

Results

Four of the eight metrics have an addiction or substance abuse category. Specifically, Google Scholar ranks the top 20 journals in various categories based on their h5-index scores, including one called “Addiction”, while Clarivate JCR employs Institute for Scientific Information categories, one of which is “Substance Abuse”. In 2020, 54 journals were included in this category but only 41 had an IF score. The Substance Abuse category is not exclusively focused on substances but includes behavioral addiction journals such as the Journal of Gambling Studies. Since the Eigenfactor and JIC data are from the same source as the Clarivate JCR data, each of the 41 Substance Abuse journals with an IF score can also be ranked in the JCR Substance Abuse category according to their Eigenfactor and JIC scores. The other four metrics do not rank journals in an addiction- or substance abuse-specific category, hereafter referred to as the A/SA category.

The top 20 ranked journals in each metric is listed in Tables 1S and 2S in Appendix. Table 1 shows the top 20 journals according to the four metrics with an A/SA category, along with addiction journals that would appear in these were they categorized as “Addiction” or “Substance Abuse” journals by Google Scholar and Clarivate JCR, respectively. A total of 23 journals appear in one or both of the A/SA categories across the four metrics, and 13 appear in all four. Three A/SA journals appear in three of the metric top 20s, five in two, and two appear in just one.

Table 1S.

Twenty topped ranked journals: Google scholar addiction category and clarivate JCR substance abuse category

Google Scholar Addiction h5-index (Rank)a	Clarivate JCR Substance Abuse IF (Rank)	Clarivate JCR Substance Abuse Eigenfactor (Rank)	Clarivate JCR Substance Abuse JIC (Rank)
Addiction (1)	Tobacco Control (1)	Drug & Alcohol Dependence (1)	Tobacco Control (1)
	Journal of Behavioral Addictions (2)c
Drug & Alcohol Dependence (2)	Alcohol Research - Current Reviews (2)	Addiction (2)	Addiction (2)
Tobacco Control (3)c
Addictive Behaviors (3)	Addiction (3)	Addictive Behaviors (3)	Alcohol Research - Current Reviews (3)
International Journal of Drug Policy (4)	International Journal of Drug Policy (4)	Nicotine & Tobacco Research (4)	Nicotine & Tobacco Research (4)
	Cannabis & Cannabinoid Research (5)c		Journal of Behavioral Addictions (5)c
Journal of Behavioral Addictions (4)	Drug & Alcohol Dependence (5)	International Journal of Drug Policy (5)	International Journal of Drug Policy (5)
Nicotine & Tobacco Research (4)	Harm Reduction Journal (6)	Tobacco Control (6)	Addiction Biology (6)
			Cannabis & Cannabinoid Research (7)c
Alcoholism-Clinical & Experimental Research (7)	Addiction Biology (7)	Alcoholism-Clinical & Experimental Research (7)	Addictive Behaviors (7)
International Journal of Mental Health & Addiction (8)	Nicotine & Tobacco Research (8)	Journal of Substance Abuse Treatment (8)	Drug & Alcohol Dependence (8)
		Journal of Behavioral Addictions (9)c
Addiction Biology (9)	Addictive Behaviors (9)	Substance Use & Misuse (9)	American Journal of Drug & Alcohol Abuse (9)
Journal of Substance Abuse Treatment (10)	International Journal of Mental Health & Addiction (10)	Addiction Biology (10)	Journal of Gambling Studies (10)
Harm Reduction Journal (11)	American Journal of Drug & Alcohol Abuse (11)	Psychology of Addictive Behaviors (11)	Harm Reduction Journal (11)
Journal of Gambling Studies (12)c
Psychology of Addictive Behaviors (12)	Journal of Substance Abuse Treatment (12)	Drug & Alcohol Review (12)	Journal of Substance Abuse Treatment (12)
Journal of Studies on Alcohol & Drugs (12)	Substance Abuse (13)	Journal of Studies on Alcohol & Drugs (13)	Psychology of Addictive Behaviors (13)
Drug & Alcohol Review (12)	Journal of Addiction Medicine (14)	Journal of Addiction Medicine (14)	Addiction Research & Theory (14)
Cannabis & Cannabinoid Research (15)	Journal of Gambling Studies (15)	Substance Abuse (15)	Alcoholism-Clinical & Experimental Research (15)
American Journal of Drug & Alcohol Abuse (15)	Addiction Science & Clinical Practice (16)	American Journal of Drug & Alcohol Abuse (16)	Journal of Addiction Medicine (16)
Alcohol & Alcoholism (17)	Alcoholism-Clinical & Experimental Research (17)	Harm Reduction Journal (17)	Journal of Studies on Alcohol & Drugs (17)
American Journal on Addictions (18)	Drug & Alcohol Review (18)	American Journal on Addictions (18)	Addiction Science & Clinical Practice (18)
Substance Abuse (18)	Psychology of Addictive Behaviors (19)	Alcohol & Alcoholism (19)	Drug & Alcohol Review (19)
Journal of Addiction Medicine (20)	American Journal on Addictions (20)	Journal of Gambling Studies (20)	Alcohol (20)
Substance Use & Misuse (20)b

Notes:

a Same rank number in parentheses indicates a tied ranking based on the h5-index score.

b Substance Use & Misuse is not listed in the top 20 Addiction journals for 2020 posted online by Google Scholar although it has the same h5-index score of 31 as the Journal of Addiction Medicine (this is likely because its h5-median score is lower, 39 versus 53). It is included in the table because these rankings are based only on the h5-index score.

c Non-category journals that would be in Top 20 based on their metric score (rank if included). These are addiction journals that are not included in the specific categories employed by these metrics (“Addiction” in the case of Google Scholar and “Substance Abuse” in the case of Clarivate JCR IF and Eigenfactor) whose scores would mean they would appear in the top 20 were they included in the category. Their ranks, were they included, is shown in parenthesis.

Table 2S.

Twenty top ranked journals in metrics without a specific addiction or substance abuse categorya

Scimago H-index (Rank)b	Scopus CiteScore (Rank)b	Scopus SJR (Rank)	Scopus SNIP (Rank)
Addiction (1)	Addiction (1)	Tobacco Control (1)	Tobacco Control (1)
Drug & Alcohol Dependence (2)	Tobacco Control (2)	Addiction (2)	Addiction (2)
Alcoholism-Clinical & Experimental Research (3)	Journal of Behavioral Addictions (3)	Journal of Behavioral Addictions (3)	Journal of Behavioral Addictions (3)
Addictive Behaviors (4)	Addiction Biology (4)	Drug & Alcohol Dependence (4)	Journal of Gambling Studies (4)
Journal of Studies on Alcohol & Drugs (5)	Alcohol Research-Current Reviews (5)	International Journal of Drug Policy (5)	International Journal of Drug Policy (5)
Tobacco Control (6)	International Journal of Drug Policy (6)	Journal of Gambling Studies (6)	Addictive Behaviors (6)
Nicotine & Tobacco Research (7)	Nicotine & Tobacco Research (7)	Addictive Behaviors (7)	Harm Reduction Journal (7)
Psychology of Addictive Behaviors (8)	Addictive Behaviors (8)	Journal of Substance Abuse Treatment (8)	Drug & Alcohol Dependence (8)
Journal of Substance Abuse Treatment (9)	Drug & Alcohol Dependence (9)	Addiction Biology (9)	International Gambling Studies (9)
Alcohol & Alcoholism (10)	American Journal of Drug & Alcohol Abuse (10)	American Journal of Drug & Alcohol Abuse (10)	American Journal of Drug & Alcohol Abuse (10)
Alcohol Research-Current Reviews (11)	Journal of Gambling Studies (11)	Addiction Science & Clinical Practice (11)	Psychology of Addictive Behaviors (11)
Alcohol (12)	Psychology of Addictive Behaviors (12)	Harm Reduction Journal (12)	Cannabis & Cannabinoid Research (12)
International Journal of Drug Policy (13)	Harm Reduction Journal (13)	Nicotine & Tobacco Research (13)	Journal of Substance Abuse Treatment (13)
Journal of Gambling Studies (13)	Journal of Substance Abuse Treatment (13)	Psychology of Addictive Behaviors (14)	Nicotine & Tobacco Research (14)
Substance Use & Misuse (13)	Alcoholism-Clinical & Experimental Research (15)	Alcoholism-Clinical & Experimental Research (15)	Adicciones (15)
Addiction Biology (16)	Adicciones (16)	Journal of Addiction Medicine (16)	Addiction Science & Clinical Practice (16)
American Journal on Addictions (17)	Addiction Science & Clinical Practice (17)	Alcohol Research-Current Reviews (17)	Journal of Addiction Medicine (17)
Drug & Alcohol Review (18)	Drug & Alcohol Review (18)	Cannabis & Cannabinoid Research (18)	Substance Abuse Treatment, Prevention & Policy (18)
American Journal of Drug & Alcohol Abuse (19)	American Journal on Addictions (19)	International Gambling Studies (19)	Drug & Alcohol Review (19)
Journal of Addictive Diseases (20)	Addiction Research & Theory (20)	Journal of Studies on Alcohol & Drugs (20)	Alcoholism-Clinical & Experimental Research (20)
	International Gambling Studies (20)
	Journal of Studies on Alcohol & Drugs (20)

Notes:

a Based on the scores for 43 journals (18 from both the Google Scholar Addiction Top 20 and the Clarivate JCR Substance Abuse Top 41, 2 from just the Google Scholar Addiction Top 20, and 21 from just the Clarivate JCR Substance Abuse Top 41).

b Same rank number in parentheses indicates a tied ranking based on the metric score.

Table 1.

Twenty top ranked journals: Google Scholar Addiction category and Clarivate JCR Substance Abuse category by IF Eigenfactor and JIC

Journal	Google Scholar Addiction	JCR Substance Abuse
	h5-index	IF	Eigenfactor	JIC
4 Addiction/Substance Abuse Metrics
Addiction	1	3	2	2
Addiction Biology	9	7	10	6
Addictive Behaviors	3	9	3	7
Alcoholism – Clinical & Experimental Research	7	17	7	15
American Journal of Drug & Alcohol Abuse	15	11	16	9
Drug & Alcohol Dependence	2	5	1	8
Drug & Alcohol Review	12	18	12	19
International Journal of Drug Policy	4	4	5	5
Journal of Addiction Medicine	20	14	14	16
Journal of Substance Abuse Treatment	10	12	8	12
Harm Reduction Journal	11	6	17	11
Nicotine & Tobacco Research	4	8	4	4
Psychology of Addictive Behaviors	12	19	11	13
3 Addiction/Substance Abuse Metrics
American Journal on Addictions	18	20	18	–
Journal of Studies on Alcohol & Drugs	12	–	13	17
Substance Abuse	18	13	15	–
2 Addiction/Substance Abuse Metrics
Addiction Science & Clinical Practice	–	16	–	18
Alcohol & Alcoholism	17	–	19	–
Alcohol Research – Current Reviews	–	2	–	3
International Journal of Mental Health & Addiction	8	10	–	–
Substance Use & Misuse	20c	–	9	–
1 Addiction/Substance Abuse Metric
Addiction Research & Theory	–	–	–	14
Alcohol	–	–	–	20
Not Included in Google Scholar Addiction Category a
Journal of Gambling Studies	(12)	15	20	10
Tobacco Control	(3)	1	6	1
Not Included in Google Scholar Addiction Category b
Cannabis & Cannabinoid Research	15	(5)	–	(7)
Journal of Behavioral Addictions	4	(2)	(9)	(5)

Notes:

a Journals not included in the Google Scholar Addiction category whose score (shown in parentheses) is high enough that they would appear in the top 20 if included in the category.

b Journals not included in the Clarivate JCR Substance Abuse category whose scores (shown in parentheses) are high enough that they would appear in the top 20 of the three metrics if included in the category.

c Substance Use & Misuse is not listed in the top 20 Addiction journals for 2020 posted online by Google Scholar although it has the same h5-index score of 31 as the Journal of Addiction Medicine (this is likely because its h5-median score is lower, 39 versus 53). It is included in the table because these rankings are based only on the h5-index score.

Four of the differences in the top 20s result from the way in which journals are classified in Google Scholar and JCR. Tobacco Control is not in the Google Scholar h5-Index top 20, as it is classified as a Public Health, not Addiction, journal. Were it included in the Addiction category, it would be the third highest ranked. The Journal of Gambling Studies is also not classified as an Addiction journal, but would be joint twelfth on the metric with three other journals if it was included. Similarly, two of four Google Scholar journals not in the Clarivate JCR top 20 are not classified in the Substance Abuse category: the Journal of Behavioral Addictions is classified in Psychiatry and Cannabis & Cannabinoid Research in Pharmacology and Pharmacy. If the former was included in the Substance Abuse category, it would have the second highest IF score, the ninth highest Eigenfactor score, and the fifth highest JIC score. The IF score of Cannabis & Cannabinoid Research would be the fifth highest in the Substance Abuse category, and its JIC score the seventh highest.

Table 2 shows the top 20 rankings in the four metrics without an A/SA category. The rankings were based on scores for 43 journals, specifically 18 that appear in both the Google Scholar Addiction top 20 and the Clarivate JCR Substance Abuse top 41, two from just the former, and 23 from just the latter. Thus, no journal is excluded from any of these rankings due to a specific type of categorization being used by the metric. Of the 25 journals that appear at least once in the A/SA top 20s, two are absent from all the metrics without a category (Substance Abuse and International Journal of Mental Health & Addiction). Four additional journals appear in these top 20s, one of which appears in two of the four (Adicciones) and one in three (International Gambling Studies).

Table 2.

Twenty top ranked journals in metrics without a specific addiction or substance abuse categorya

Journalsa (Grouped by Number of Table 1 Metrics they Appear in)	H-indexd	CiteScored	SJR	SNIP
4 Metrics in Table 1
Addiction	1	1	2	2
Addiction Biology	16	4	9	–
Addictive Behaviors	4	8	7	6
Alcoholism-Clinical & Experimental Research	3	15	15	20
American Journal of Drug & Alcohol Abuse	19	10	10	10
Drug & Alcohol Dependence	2	9	4	8
Drug & Alcohol Review	18	18	–	19
International Journal of Drug Policy	13	6	5	5
Journal of Addiction Medicine	–	–	16	17
Journal of Behavioral Addictionsb	–	3	3	3
Journal of Gambling Studiesc	13	11	6	4
Journal of Substance Abuse Treatment	9	13	8	13
Harm Reduction Journal	–	13	12	7
Nicotine & Tobacco Research	7	7	13	14
Psychology of Addictive Behaviors	8	12	14	11
Tobacco Controlc	6	2	1	1
3 Metrics in Table 1
American Journal on Addictions	17	19	–	–
Cannabis & Cannabinoid Researchb	–	–	18	12
Journal of Studies on Alcohol & Drugs	5	20	20	–
Substance Abuse	–	–	–	–
2 Metrics in Table 1
Addiction Science & Clinical Practice	–	17	11	16
Alcohol & Alcoholism	10	–	–	–
Alcohol Research-Current Reviews	11	5	17	–
International Journal of Mental Health & Addiction	–	–	–	–
Substance Use & Misuse	13	–	–	–
1 Metric in Table 1
Addiction Research & Theory	–	20	–	–
Alcohol	12	–	–	–
0 Metrics in Table 1
Adicciones	–	16	–	15
International Gambling Studies	–	20	19	9
Journal of Addictive Diseases	20	–	–	–
Substance Abuse Treatment Prevention & Policy	–	–	–	18

Notes:

a Based on metric scores for 43 journals: 18 from both the Google Scholar Addiction Top 20 and the Clarivate JCR Substance Abuse Top 41, 2 from just the Google Scholar Addiction Top 20, and 21 from just the Clarivate JCR Substance Abuse Top 41.

b Journal not included in the Clarivate JCR Substance Abuse category.

c Journal not included in the Google Scholar Addiction category.

d Same rank number indicates a tied ranking based on the metric score.

Ignoring the Google Scholar and JCR categories and just focusing on the scores for each journal, there are 11 journals in the top 20 of all eight metrics. The top rank is occupied by one of three journals: Tobacco Control has top place on four of the eight metrics, Addiction on three, and Drug & Alcohol Dependence on one. The Journal of Behavioral Addictions, which is not included in the JCR Substance Abuse category appears in three of the four non-category metrics and it occupies third place in each of these. Its scores are among the five highest in six of the seven metrics shown in Tables 1 and 2, and ninth in the other. The only metric from which it is absent from the top 20 is the Scimago H-index.

Table 3 shows the correlations between the eight metric ranks for all 43 journals. Only three of the 28 exceeded the 0.80 threshold set for a very strong correlation: h-5 index↔Eigenfactor; CiteScore↔JCI; CiteScore↔JCR). Seven correlations were in the fair range (0.30–0.59), including the only one not statistically significant (H index↔SNIP; 0.32, P=0.07). The remaining 18 correlations fell in the moderately strong range (0.60–0.79). Six of the seven h5-index, JCI, CiteScore and SJR correlations exceeded the moderate threshold of 0.60, as did five of the IF, Eigenfactor and SNIP correlations. The weakest correlations were between the H-index and the other seven metrics, with just one exceeding the moderate threshold and the other six being fair (i.e., below 0.60).

Table 3.

Kendall's Tau-B rank correlations, bootstrap 95% confidence intervals and significance testsa among journal ranks on eight metrics

	h5-index	IF	Eigenfactor	JCI	H-index	CiteScore	SJR	SNIP
h5-index	1.00	0.68 [0.50, 0.78]	0.80 [0.66, 0.87]	0.69 [0.52, 0.78]	0.59 [0.39, 0.72]	0.68 [0.49, 0.79]	0.69 [0.47, 0.80]	0.62 [0.41, 0.74]
IF		1.00	0.57 [0.37, 0.70]	0.78 [0.64, 0.86]	0.38 [0.16, 0.55]	0.75 [0.58, 0.83]	0.73 [0.57, 0.84]	0.62 [0.39, 0.76]
Eigenfactor			1.00	0.60 [0.41, 0.70]	0.63 [0.47, 0.75]	0.61 [0.42, 0.73]	0.64 [0.47, 0.76]	0.51 [0.30, 0.65]
JCI				1.00	0.45 [0.23, 0.59]	0.84 [0.72, 0.90]	0.76 [0.65, 0.85]	0.66 [0.44, 0.78]
H-index					1.00	0.48 [0.30, 0.62]	0.45 [0.26, 0.61]	0.32 [0.09, 0.51]
CiteScore						1.00	0.82 [0.72, 0.88]	0.70 [0.48, 0.82]
SJR							1.00	0.75 [0.61, 0.84]
SNIP								1.00

Notes:

a All but two of the Bonferroni corrected P-values were <0.001. The exceptions were H-index↔IF (0.38, P <0.01) and H-index↔SNIP (0.32, P = 0.07).

As noted above, the JCR IF was chosen for the longitudinal analysis because it is the most widely used journal metric. A search of the homepages of the journals (accessed directly through Google and not through a university library portal) conducted on August 26–27, 2021 confirmed its wide use among addiction journals, with all 43 reporting their 2020 JCR IF score. This was often accompanied by the 5-year score and the journal ranking in the Substance Abuse category. Of the other six metrics examined, the CiteScore appeared on 24 homepages, SNIP on 17 and the SJR on 16. The Google Scholar, Eigenfactor and h-index scores were each only cited once.

Figure 1 shows the number of journals included in the JCR Journal Substance Abuse category since 1997, when there were 23, all of which had IF scores. By 2020, there were 41 with an IF score and an additional 13 with no score. The latter is due to the inclusion of those journals covered by the ESCI that have yet to receive an impact factor (Editor Resources, 2022). Since 1997, 21 journals with IF scores have been added to the category, two-thirds of these after 2009. Two of the 23 journals that appeared in the Substance Abuse category in 1997 were subsequently removed permanently, as was one of the 21 added after this date; only the latter discontinued publication.

Fig. 1.

Number of journals in JCR substance abuse category (1997–2020)a

Notes:

a Journals that changed name could appear twice in the Substance Abuse category for some years (e.g, both Alcohol Research and Health and Alcohol Research-Current Reviews both appear in 2014, although they are the same journal). In such cases, only one journal is included in the count for that year.

b It is not possible from the JCR data to know if any of the 23 journals in 1997 were newly added that year.

The types of journals included in the Substance Abuse category changed over time. All 23 journals listed in 1997 pertained to alcohol and/or drugs or addiction/substance abuse. Three of the 21 introduced after this time focus exclusively on tobacco, two on dual diagnosis, one on heroin and two on gambling. Five of the remaining 13 that pertain more generally to addictions have a specific focus on policy, harm reduction, ethnicity, nursing and Nordic studies. The two journals not included in the Substance Abuse category, and hence not in Fig. 1, also reflect these trends, one being focused on behavioral addictions (Journal of Behavioral Addictions) and one exclusively on cannabis (Cannabis & Cannabinoid Research).

Figure 2 presents details of the 10 addiction journals with the highest JCR IF over the period 1997 to 2020, irrespective of whether or not they are listed in its Substance Abuse category. Twenty-eight journals were in the top 10 for at least one year over this period of time. Only two journals have been consistently in the top 10 over the entire 24-year period (Addiction and Drug & Alcohol Dependence). Three journals entered the top 10 soon after being classified as Substance Abuse journals and remained throughout subsequent years (the International Journal of Drug Policy, Nicotine & Tobacco Research, and Tobacco Control). The Journal of Behavioral Addictions, not classified as a Substance Abuse journal, had an IF that placed it fourth among addiction journals four years after it was first published. It remained in the top 10 in all subsequent years, and had the second highest IF in 2020.

Fig. 2.

Addiction journals with the 10 highest JCR Impact Factors (1997–2020)

Notes:

a Journal of Behavioral Addictions and Cannabis & Cannabinoid Research are not in the JCR Substance Abuse category.

b Date is the first year the journal was published under its present, or any former, tile(s).

c This journal was titled the Journal of Studies on Alcohol before 2007.

d This journal changed its title to Substance Use & Misuse after 1997.

e The Journal of Substance Abuse was discontinued after 2001.

f This journal’s previous titles were Alcohol Health & Research World (until 2000) and Alcohol Research & Health (2001–2013).

Discussion

There are two expectations one might have when examining the performance of journal metrics in any academic discipline. On the one hand, one would anticipate that if there are detectable differences in the reach and influence of journals, the various metrics would be sensitive to these and able to identify those with the most impact on the research literature. On the other hand, the various metrics have been developed, at least in part, from perceived shortcomings of those that already existed and therefore they have different inclusion criteria (e.g., concerning self-citations) and use different methodologies and data sources (Pajić, 2015); accordingly, one would expect them to identify different journals in their rankings. The current paper lends some support to the former view. The correlations between the metrics are mostly moderate to strong. Also, ignoring the omissions that arise from the Google Scholar and JCR categorization systems, a quarter of the journals examined have scores which would place them among the top 20 on all eight metrics.

However, while there is a degree of agreement across metrics, it should also be noted that almost three-quarters of the 43 journals appear in at least one of the eight metrics, presumably because they apply different criteria in their selection process. This raises the question as to which metric might be “best”, and the answer will depend largely upon the purpose one has in mind. Those using such metrics should review them to identify differences, such as the extent to which they emphasize medical and biological journals compared to those from the social sciences as well as different types of addictive behaviors. For example, there are no gambling journals in the Google Scholar Addiction h5-index. In contrast, gambling journals are more prominent in CiteScore, SJR and SNIP, so researchers in this area of research might find these more useful and relevant in assessing journal impact.

Previous studies that have reported bivariate correlations between various metrics, including the IF, Eigenfactor, SNIP and CiteScore, have typically found these to be statically significant (with an Alpha of 0.05). Brown and Gutman (2019) reported that all correlations between seven metrics for 14 occupational therapy journals exceeded 0.75. They noted the similarity of these findings to previous bibliometric comparisons in nuclear medicine, rehabilitation and pediatric neurology journals. Villaseñor-Almaraz and colleagues compared six metrics for 122 journals categorized by the Web of Knowledge as radiology, nuclear medicine and medical imaging (Villaseñor-Almaraz et al., 2019). All bivariate correlations were 0.06 and above and statistically significant, but they found the IF did “not show the best correlation between other metrics” (Villaseñor-Almaraz et al., 2019, page 495). Diaz et al. (2021) compared the scores on five metrics of 50 top ranked psychiatry journals. They reported a wider range of correlations (0.43–0.96), but, as in the studies from other disciplines, all were statistically significant. In the present study, we reported effect sizes and adjusted the P-values for the number of comparisons made. Even with the latter adjustment, all but one of the correlations were statistically significant. In contrast to Villaseñor-Almaraz et al. (2019), the correlations between the IF and other metrics did not stand out as especially weak; rather the weakest set of correlations were found for the H-index. The inclusion of confidence intervals lent caution to interpreting these correlations, however, as most estimates were fairly imprecise. For example, the lower confidence interval of all three very strong correlations fell within the moderately strong range. Future studies should report confidence intervals in addition to P-values, as the latter are almost always “statistically significant” and they are uninformative about the precision of the correlations.

The longitudinal IF data show there are a handful of journals that have consistently remained in the top ten since 1997. However, new tobacco-focused journals and more specialized drug journals (e.g., focused on harm reduction or policy) entered the top 10 starting in the last decade, and displaced some of the more established journals such as Alcohol & Alcoholism and the Journal of Studies on Alcohol and Drugs. The two journals not categorized as Substance Abuse journals make this trend towards specialization even more pronounced. It is not clear why these journals are excluded from this category, or why Tobacco Control is not classified as an Addiction journal by Google Scholar. In this regard, the classification systems are not very useful in identifying the top journals in the field of addiction/substance abuse research.

Despite the criticism it has received and the existence of alternatives, the present study found that the IF was the only metric featured on all addiction journal public webpages. Journals are, presumably, highlighting their impact factor because they wish to attract researchers to publish with them. In turn, researchers want to be able to state that they have published in a “high impact” journal, presumably because they are being assessed by their colleagues and grant reviewers using this metric (Dong et al., 2005). So, despite the limitations of the IF, it is clearly so well-ingrained in the various systems by which academics are judged (university hiring and promotion committees, funding agencies, awards committees) that it seems unlikely to be discarded any time soon. This appears to be another aspect of the incentive and reward system of modern academia, like gift authorship, publication mills, lack of transparency, emphasis on producing novel positive results (Edwards & Roy, 2017; Nosek, Soies, & Motyl, 2014), that is widely acknowledged to be a problem, but that there is little willingness to correct.

This is not to be dismissive of the concerted efforts to correct the misuse of journal metrics by editors, grant reviewers and hiring and promotion committees, most noticeable among these being the Leiden Manifesto, the San Francisco DORA, and the Hong Kong Principles for evaluating researchers (American Society for Cell Biology, 2013; Hicks et al., 2015; Moher et al., 2020). And, indeed, examples of the adoption by specific North American and European universities and grant review bodies of the principles described in these documents can be found (see Moher et al., 2020 for details). These principles have yet, however, to achieve widespread adoption among academic institutions. A 2018 survey of 96 research institutions in Great Britain found that 75 had not endorsed the 2013 DORA and had no policy pertaining to the use of research metrics in evaluations (Gaind, 2018). A recent North American survey of promotion and tenure documentation from universities showed 40% of those from research-intensive institutions mentioned the journal impact factor, with over 80% of these being supportive of its use and more than 60% believing it to be an indicator of research quality.

It should be noted, however, that the current study found that some addiction journals do follows the DORA and Leiden Manifesto recommendations and place their IF within a broader context and explain the formula used by a number of different metrics. The Journal of Drug Issues is most thorough in this regard, describing a wide range of JCR, Google Scholar and Scopus metrics on its homepage (Journal of Drug Issues, 2021). The Journal of Behavioral Addictions presents two years of data for six metrics (Journal of Behavioral Addictions, 2021), and Taylor & Francis also has a link on the webpages of its addiction journals that contains their scores on four metrics, with a description of each (see, for example, Journal of Substance Use, 2021). Other journals should follow suit and provide data on a range of metrics and guide prospective authors in their interpretation of these. As noted by Pajić (2015), the various metrics are likely capturing different dimensions of a multifaceted phenomenon called “journal quality”, and therefore consulting a large number is preferable to consulting just one.

Whatever metrics one uses, their limitations should be acknowledged. Primary among these is their inappropriate use to evaluate individual researchers. It is a clear example of an ecological fallacy as it entails using data about a larger unit of analysis (journals) to make a judgement about a smaller unit of analysis (individual researchers who published in these), and should be avoided. Similarly, the journal metric tells one nothing about the quality of any individual paper published by that journal, or even the extent to which it has been cited (Dong et al., 2005).

The analysis also highlighted some potential problems with the categories used by some metrics. Journal rankings usually imply some form of disciplinary or subject-area classification, and such categorization of journals also helps limit temporal fluctuations over time (Pajić, 2015). Also, categories provide a shortcut to anyone trying to identify high impact journals within a specific discipline, such as addiction research. However, the omission of key journals from these undermines their utility. For example, those searching the Google Scholar Addiction category for a place to publish a tobacco- or gambling-related manuscript will not find Tobacco Control or the Journal of Gambling Studies, even though each has a high IF. While it is true that Tobacco Control has a policy-focus, another prominent policy-focused journal (the International Journal of Drug Policy) is included in the Addiction category.

Gambling is included in the most recent versions of the Diagnostic and Statistical Manual of Mental Disorders (DSM) and International Classification of Diseases (ICD) among addiction/substance use disorders, with gaming disorder also in the latter (Saunders, 2017). This makes the exclusion of the the Journal of Gambling Studies from the Google Scholar Addiction category more noticeable than it would have been a decade ago, when gambling was considered an “impulse control disorder” by DSM and categorized in the Habit and Impulsive Control Category of ICD. Thus, an additional downside of categorizing journals within metrics is that these cannot keep abreast of such changes in nosology.

Similar problems with consistency and incompatibility with recent diagnostic developments are evident in the JCR Substance Abuse category, which does not include the Journal of Behavioral Addictions and Cannabis & Cannabinoid Research. Cannabis is clearly a “substance” and other substance-specific journals appear in the category (e.g., Alcoholism-Clinical & Experimental Research and Tobacco Control). In the case of the Journal of Behavioral Addictions, it might be argued that behaviors are not substances, but the Journal of Gambling Studies is included in the JCR Substance Abuse category. Moreover, the Journal of Behavioral Addictions publishes papers that address the intersection of behavioral and substance addictions. And, as noted above, behavioral addictions are now included in both of the major psychiatric diagnostic systems as addiction/substance use disorders. While this may explain why the journal is included in the Psychiatry category, this logic would apply to practically all journals in the JCR Substance Abuse category.

Beyond academics looking for journals in which to publish their research, the Google Scholar and JCR categories are also convenient for those conducting studies of the editorial practices of high impact journals, and the quality of research published in these (e.g., Gorman, 2020; Wayant, Tritz, Horn, Crow, & Vassar, 2021). But such studies will miss key influential journals in the discipline and present an incomplete picture if they rely on only one metric. The present study provides guidance on the selection of journals in future studies of this type that use bibliometrics to generate a sample of journals, with a view to this process being more comprehensive.

Among the study limitations is the focus on a set of the most commonly used metrics, meaning that others, such as the Normalized Eigenfactor and Article Influence Score were not included. Also, not all addiction journals appear in the Google Scholar or JCR A/SA categories, and therefore some which might be important in niche areas of addiction studies such as historical research (e.g., Contemporary Drug Problems) educational research (e.g., Journal of Drug Education), or sexual addiction (e.g., Sexual Addiction and Compulsivity) were omitted. While these might have scores on some of the seven metrics studied, they are typically similar to those of the lower ranked journals, so it is unlikely any very high impact addiction journals were excluded. Finally, a few studies conducted in other fields of research have tried to assess the association between metric scores and quality (assessed in terms of adherence to reporting and statistical standards in published papers), but this was beyond the scope of the present paper. However, recent studies of addiction journals have found that adherence by authors to reporting practices designed to improve transparency and replication and curtail selective outcome reporting is poor, even among high-impact journals (Adewumi, Vo, Tritz, Beaman, & Vassar, 2021; Gorman, 2020; Vassar, Roberts, Cooper, Wayant, & Bibens, 2020).

Funding source

None.

Authors' contribution

DMG conceptualized and designed the study, collected the data, and drafted the manuscript. JCH analyzed the data. Both authors interpreted the results and reviewed and edited the final manuscript. Both authors had full access to all data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Conflict of interest

The authors declare no conflict of interest, financial or otherwise, relevant to the subject of this article.