Article-Level Metrics.

In the era of digitization and Open Access, article-level metrics are increasingly employed to distinguish influential research works and adjust research management strategies. Tagging individual articles with digital object identifiers allows exposing them to numerous channels of scholarly communication and quantifying related activities. The aim of this article was to overview currently available article-level metrics and highlight their advantages and limitations. Article views and downloads, citations, and social media metrics are increasingly employed by publishers to move away from the dominance and inappropriate use of journal metrics. Quantitative article metrics are complementary to one another and often require qualitative expert evaluations. Expert evaluations may help to avoid manipulations with indiscriminate social media activities that artificially boost altmetrics. Values of article metrics should be interpreted in view of confounders such as patterns of citation and social media activities across countries and academic disciplines.

INTRODUCTION

Currently available journal and author metrics do not reveal scientific merit and influence of individual articles. Some of these metrics, particularly the notorious journal impact factor (JIF), have major drawbacks which limit their use for research evaluation and academic promotion.12 In the era of digitization and mass proliferation of online resources, article-level metrics are increasingly proposed and employed to evaluate the dissemination of scientific information and research impact.34 While traditional metrics such as citations are slow to reveal scholarly use of articles and their overall implications, several alternative metrics have emerged to reveal ‘attractive’ articles in a timely manner and aid in adjusting research and journal publishing strategies.56 All metrics are dependent on journal digitization and promotion which expand the readership network. Use of an individual article can be increased by professional promotion through widely visible online platforms, information aggregators, digital libraries, and bibliographic databases.7

Academic disciplines and countries with low citation metrics may particularly benefit from complementing article citations with alternative metrics and adjusting their research and development strategies accordingly. As an example, quantifying article citations along with online attention scores and reuse in the social sciences, humanities and nursing yields more objective evaluation of research implications in these traditionally low-cited academic areas.89

Current online technologies offer equal opportunities for improving visibility and promotion of individual articles. The assignment of digital object identifiers (DOI) is one such universally applicable technology offering benefits to all stakeholders of science communication.10 The DOI assignment positively influences metrics, which are generated by Altmetric.com and Plum Analytics, the two information giants that aggregate attention surrounding digitally accessible articles.1112 Notably, an analysis of 496,665 PubMed-indexed articles demonstrated that a high percentage of articles (40.5%) published in the last 50 years lack DOI.13 The same study pointed to a much better situation with DOI assignment in the U.S., the U.K. and the Netherlands than in Russia, the Czech Republic and Romania.

The expanded use of online access and evaluation tools brings about a major change in the selection, categorization, and reuse of scholarly articles.14 Global bibliographic databases and online platforms such as Scopus and PubMed are now equipped with digital tools to capture citations and overall attention to individual articles. Researchers and authors may distinguish ‘trending’ articles and ‘hot’ topics in their fields by ranking articles with linked citations and overall online attention records.15

The move toward article-level metrics is well accepted by experts who recognize academic value of citation alternatives, particularly bookmarking counts at the Mendeley reference management platform.16 Mendeley metrics are employed for article impact evaluation similar to citations, keeping in mind that bookmarking counts accumulate earlier and that not all of them result in eventual citations.17

Considering the wide variety of currently discussed established and emerging metrics, we aimed to comprehensively overview article-level metrics and their potential implications for upgrading research evaluation and editorial strategies.

VIEWS AND DOWNLOADS

Article access data such as HTML views and PDF and XML downloads are basic quantitative metrics that reflect readers' interest toward certain topics. Most online journal platforms are now equipped with tools for quantifying real-time usage metrics.18 These metrics can be monitored over time to visualize temporal and geographic trends of article usage.1920 Variations in use of HTML, PDF, and XML article formats can be also reported to guide publishers over the readership preferences and priorities for reuse, archiving, and marketing.19

Views and downloads are important for non-indexed start-up journals that may change their online interface in line with their readership preferences and online activities. For indexed journals, these basic metrics aid in revealing influential articles in a short period (weeks, months) while expecting citations.21 A comparative analysis of the access to the same set of journal articles through Web of Science and Springer online interface revealed the user preference of quick and free access facilitated by the publisher website and common search engine.22

Interestingly, article characteristics such as title length, number of authors, and keywords do not confound download counts. Advanced publisher interface has more influence on article usage metrics.23 Article views are increasingly dependent on publisher access and distribution preferences. Publishers that tag their articles with DOI may set various online navigation routes through journal platforms and indexing databases. They may direct visits to HTML links before downloading PDFs. The availability of various access routes, particularly for articles archived by PubMed Central and other digital repositories, skews the usage statistics, making journal website visits a less reliable metric for journals with expanded archiving.24

Publisher and journal integration with social media sites significantly increases journal webpage visits and article downloads, particularly within the first days of new content publication.25 Twitter and Facebook are now instrumental for directing public interest toward freely accessible articles and increasing their usage.26

Usage metrics displayed at journal websites and other distribution platforms variably correlate with citation metrics.27 Top-downloaded articles are usually those from high-impact journals.28 Variable strength of such associations is also reported across academic disciplines, with a strong correlation between downloads and citations in the social sciences and humanities.29 A moderate correlation between usage metrics in the first year and citations in the two years post-publication was reported at a multidisciplinary open-access mega journal Scientific Reports.30

CITATIONS

Raw citation counts have long been used for quantifying and comparing the impact of scholarly articles. In the absence of other reliable tools, citations have been viewed as the only indicators of article utility and “quality.”31 Relevant citations indicate that index items are read and critically appraised by citing authors.32 Currently, online platforms of numerous journals display article citations recorded by Scopus, Web of Science, Google Scholar, and Dimensions. The counts may differ due to variable volumes of sources covered by each database and search engine, and particularly due to the high selectivity of Web of Science and Scopus.33 The ease of tracking citations at Web of Science and Scopus has allowed distinguishing most-cited articles, visualizing trending topics, and recording citation classics across subject categories. Related analyses often focus on 100 highly-cited items published within several decades. Such analyses allow visualizing citation networks and setting research priorities. High citation counts often reflect the approval of evidence-based research by professionals.3435 Regardless of their numbers, certain citations add to the scientific prestige of cited articles. These are primarily citations from clinical trial reports, practice guidelines, policy documents, and systematic reviews.21 As a rule, reference lists of all these influential articles are thoroughly checked in view of their relevance, validity, and scientific prestige. Importantly, citing and referencing patterns have changed over the past two decades.32 The digitization, indexing, and language of individual articles are now the driving forces behind the citation counts.36 The availability of online multidisciplinary databases with advanced search engines and comprehensive literature coverage facilitates the retrieval and subsequent citation of articles. In fact, an analysis of referencing and citing characteristics of nearly 1 million articles demonstrated that expanded and updated reference lists with high share of publications recorded in Web of Science positively correlate with citation impact.37 The interdisciplinary nature of current research and authors' and readers' expanded professional networks are believed to confound modern-day citation impact.38 Although citations are more reliable metrics than views and downloads, these are still far from being optimal indicators of article use and influence. There are numerous subjective factors and variable editorial strategies which affect citation counts. Access to bibliographic databases and search engines differ across countries, resulting in limited citation activities in disadvantaged regions. Also, numerous journal instructions contain points on how their authors should limit or expand reference lists. Reviewers of highly prestigious journals request their authors to replace or omit references to non-Anglophone articles, even if these are relevant and better explore the context.39 All these subjective factors may undermine the reliability of citation metrics.40 Raw citation counts are highly dependent on academic fields, necessitating field normalization by correcting for a number of related (average) references (citation score normalized by cited references) or citations (mean normalised citation score).4142 The widely discussed Leiden Manifesto stressed the importance of the field normalization and endorsed the percentiles method for quantifying an article's impact in its field.43 The percentiles method estimates the article rank on the basis of its listing among 1%, 10%, or 20% of highly-cited items in a certain well-defined and ‘homogenous’ academic field. The experiments with various normalization approaches have distinguished the percentile citation scores as less field- and time-biased than the scores normalized for mean (average) citations in a field.44 The same experiments have pointed to the time-normalization as an approach for revealing influential articles in an academic field with numerous citations within a short post-publication period.

The relative citation ratio (RCR) is now a widely discussed article-level citation metric proposed by experts at the U.S. National Institutes of Health (NIH) Office of Portfolio Analysis.45 This is a non-proprietary metric that is field and time-adjusted and benchmarked to NIH-funded articles. Its algorithm is based on a network of PubMed citations which are tracked by the NIH iCite analytic web tool (https://icite.od.nih.gov/). The nominator of the RCR is the total annual citations of an article while the denominator is the average annual citations received by NIH-funded articles in the same field. An RCR equal to one represents the field normalization and any values above one point to better performance of an index article compared to 50% NIH-funded articles.46 The main advantage of RCR is that it ignores journal impact indicators. Two large biomedical funding agencies, the Welcome Trust in the U.K. and Fondazione Telethon in Italy have already experimented with the RCR for analyzing their research grant outcomes.47 Several bibliometric analyses have found correlation between the RCR and established field-normalized citation indicators such as mean normalized citation score, citation percentile, and F1000 score.4849 Nonetheless, the RCR has an important limitation due to ignoring citations outside the PubMed platform, particularly those in higher citation fields which may decrease RCR values.5051

EXPERT EVALUATIONS

F1000Prime (https://f1000.com/prime/home) is a subscription-based expert evaluation system for articles in biology and medicine. It is a post-publication tool of Faculty 1000 (a U.K.-based publisher) that was launched in 2002 to help biomedical researchers find the most significant and potentially impactful articles in their field.52 The articles in the system are rated by faculty members from all over the world as exceptional (3 stars), very good (2) and good (1). Total scores are sums of the stars received from all recommending experts. The evaluated articles are additionally tagged with remarks such as “good for teaching,” “interesting hypothesis,” “new finding.” The experts may add brief comments and highlight potential implications.53 The top three areas with most recommendations include cell biology, molecular medicine, and genetics.53 Although the expert evaluation system is a sustainable alternative to citation-based ranking and social-media commenting, there are some limitations related to its subscription model, closed circle of peer evaluators, and focus on “high-quality” and well-cited biomedical items.5455

ALTERNATIVE METRICS

Alternative article metrics, or altmetrics, offered by Altmetric.com (https://www.altmetric.com/) and Plum Analytics (https://plumanalytics.com/) are increasingly used for the assessment of the broad academic and societal impact of scholarly works.56 The resultant metrics of the same articles generated by different platforms may differ due to varying tracking patterns. Articles in some disciplines, particularly in the social sciences, humanities, and biomedicine, with a broad public engagement in online sharing and commenting, may garner high values of altmetrics.57 The main advantages of the altmetrics are their early and real-time reflections of individual article uses and societal implications.

Altmetric.com services are currently utilized by most large publishers for monitoring their trending articles with high values of the altmetric attention score (AAS). Although the algorithm for the AAS calculation has not been publicized, its values are freely available and displayed at the center of the Donut Badges of published articles. The Donut Badge is a multicolor circle that depicts sources of attention to scholarly items (e.g., light blue represents Twitter activities).58 The Altmetrics.com algorithm weighs outreach and attractiveness of publicly shared and discussed articles, processing data from a wide variety of news outlets, blogs, policy documents, and social media. Although citation counts are not processed for the AAS calculation, these are still drawn from the Dimensions platform (https://app.dimensions.ai) and displayed along with the alternative metrics. Comparative analyses of alternative metric provider services demonstrated the accuracy of covering article mentions at blog posts, news items, and tweets by Altmetric.com.59

Plum Analytics offers the PlumX tool that generates the Plum Print, an infographic with aggregated information on article usage, captures, mentions, social-media attention, and citations.60 Plum Analytics was acquired by Elsevier in 2017 to display article metrics on Scopus. PlumX accurately reflects metrics from EBSCO Information Services and Mendeley reference management platform. The latter was acquired by Elsevier in 2013 and rapidly became one of the most popular resources in medicine.2761

Mendeley reader counts are displayed by both Altmetric.com and PlumX. The reader counts drawn from the Mendeley platform (https://www.mendeley.com/research-papers/), accurately reflect article bookmarks by Mendeley users.62 An analysis of 20,000 randomly chosen articles from various scientific fields demonstrated that 63% of these items were bookmarked and saved into individual online libraries by Mendeley users and that related reader counts better reflected their impact than other alternative metrics.63 Mendeley users save article links in their individual libraries and manage reference lists for own articles, which may include some of the saved items. Manipulations with artificially boosting Mendeley reader counts are unlikely.

Mendeley reader counts predict future citations with variable degree of certainty across academic disciplines and countries.64 The reader counts increase over time due to the rapidly growing Mendeley user numbers, necessitating the normalization for assessing the alternative impact of individual articles.65 The Mean Normalized Reader Score was proposed to normalize Mendeley reader counts against year of publication and subject category.66

Twitter is another popular alternative channel for dissemination and promotion of scholarly articles.6267 The ease of generating tweets, attaching graphics, tagging relevant users and publicly sharing information attract millions of users worldwide.68 Twitter activity can be viewed as microblogging which is a convenient and concise format for commenting on scholarly articles. Tweets with professional comments on article contents weigh more than those without.3 Also, tweets and retweets of active users with numerous followers weigh more than activities of users with small and disorganized networks. Accordingly, article tweets from group (journal) accounts receive more attention and expert comments than those from individual accounts.6970 Specifics of professional topics and professionals' heavy workload may limit the expansion of scholarly activities and networking on Twitter across some academic disciplines.71 At the same time, tweets in large numbers can be indiscriminately generated by the so-called bots (automated accounts, spambots) and trolls (human users) that may spread misinformation, diminish value of public engagement, and skew altmetrics.727374

Correlation analyses of tweets and citations have demonstrated variable results. A strong and statistically significant association was found for articles of prolific authors in medicine.75 However, an analysis of 1.4 million articles covered by PubMed and Web of Science did not reveal any such association, pointing to variable promotion practices in indexed journals and distinct features of traditional and alternative impact metrics.76

Similar to time and field-normalization for citations and Mendeley reader counts, Twitter counts normalization was proposed for cross-field and cross-country comparisons. The normalization method is based on the percentile principle which is applicable to journals with at least 80% of articles mentioned on Twitter.77 Twitter percentiles ranging from zero to 100 are particularly applicable to biomedical, health, life and earth science journals. For papers published in 2012, three top journals with 100% of tweeted articles were identified: The New England Journal of Medicine (16,908 total tweets for 215 papers), The Lancet (10,750 tweets for 233 papers), and The BMJ (12,469 tweets for 325 papers).77

Scholarly activities on social media are associated with prestige of individual articles. Blog and news posts are particularly contributing to institution and country prestige.78 Social-media references are similar to traditional citations in that both metrics positively correlate with JIF and international collaboration.78 Blogs and news outlets may also create a coverage bias. A recent analysis of more than 100,000 randomly chosen publications processed by altmetrics providers pointed to a coverage bias due to uneven representation of countries, languages, and academic disciplines, with over 65% of blogs representing English-speaking countries and over 75% publishing in English.79

The so-called societal impact of articles is becoming an integral part of research evaluation in most developed countries. Publishers integrating their journal platforms with popular social-media sites pave the way for attracting online attention and boosting related scores.80 Skilled authors and advocates of alternative distribution routes are now also able to share and promote their research works while waiting for citations.

While social-media activities are gaining momentum, some journal editors suggest entrusting article and journal promotion responsibilities to social media editors who can streamline the post-publication communication.81 The new editor (ambassador) roles relate to exposing potentially influential articles to a broad public attention.828384

CONCLUSION

Article-level metrics are increasingly recognized as reflections of real-life influence of scholarly research output. While digitization and aggregation of online information is becoming ubiquitous, any activity surrounding published articles can add value to these metrics. Article metrics aggregate information on individual item usage, expert evaluation, public attention, and citations. Arguably, the concept of article metrics is an attempt to prevent misuse of the JIF and provide research evaluators with a diversity of reliable and widely applicable tools. Comprehensively covering all currently available article metrics and publishing aggregate information along with articles may aid in distinguishing influential items and enriching research evaluation which is moving away from exclusively citation metrics calculations.

Current altmetrics providers, particularly Altmeric.com and Plum Analytics, have emerged as organizations supplying all stakeholders of scholarly communications with reliable metrics and links to post-publication communication threads. While social-media activities are diversifying, altmetrics platforms grow and integrate with journal and database platforms. Numerous open-access journals are currently partnering with altmetrics providers to display dynamically changing article influence in a real-time mode.

Information surrounding published articles cannot be exhaustively covered by altmetrics providers. Emerging online communication and networking platforms with advanced technical specifications are gradually included in the aggregation platforms to generate more comprehensive altmetrics. Similar to traditional citations, altmetrics can be manipulated by artificially increasing views, downloads, and social-media mentions. Limitations of article metrics can be also due to differing social-media practices across academic disciplines and countries. Even within a discipline some article types may be more attractive for social media than others, confounding the metrics calculation and necessitating context evaluation. Hence, any quantitative evaluation of article influence should be complemented by qualitative expert evaluations.