Indicators of Evaluating Research at Article Level: Recommendation for Effective Evaluation of APJCP’ Scientificzzm321990Performances

Today, research is seen as an investment to promote innovation and maintain sustainable social-economic development in all societies. The growth of scientific products and the expansion of knowledge in different scientific fields have entailed more attention to assessments and the impact evaluation of both outcome and process of researchers in all fields. In light of this need, policymakers in the medical field have paid more attention to evaluating the outcomes of research in terms of its impact on the society using many different indicators. In this short communication, the performance of scholarly published scientific products are discussed and the indicators that measure such impacts are evaluated and recommendation is given to APJCP’ editorial board on how to align its activities toward achieving better impact and scientometric measures for the journal. Creative Commons Attribution License

Research is seen as an investment to promote innovation and maintain sustainable social-economic development (Banzi et al., 2011). The growth of scientific products and the expansion of knowledge in different scientific fields entail more the need of effective assessments strategies and valid evaluations tools, tools that can provide a valid answer to the question of “how investment in research can bring the desired outcome, impact? (Ovseiko et al., 2012). Normally administrative bodies are mainly interested in evaluating the outcomes of their investment in research for the society (Hanney et al., 2004). In light of this need, policymakers in the medical field have paid more attention to how to evaluate outcomes of research seeking better tools and metrics. Such a valid tool can improve the impact of future investments in research and indirectly benefit the research community, reasonable budgeting, efficient resource allocation, and finally better decision-making (Gordon and Bartley, 2016; Zachariah et al., 2014). In addition, a valid evaluation of the effects of research uncovers those researchers that have a real impact on pushing knowledge boundaries to the new states. It also paves the way to evaluate the performance of universities and research institutes and help better decisions on allocation of funds, budgeting, employing and capacity building (Leimu and Koricheva, 2005). In addition to research institute and public administrators, publisher of scientific journals (as the immediate beneficiary of published research report) are interested in finding out which scientific productions have been more effective. A publisher wants to know which papers have been more effective in improving the scientific weight of the journal measured. The Asian Pacific Journal of Cancer Prevention as part of its own self-evaluation is in a need to develop its own strategies to evaluate its scientific performance and impact on its ultimate goal of impacting cancer control measures in Asia and the Pacific region. The present short communication will discuss the APJCP’s scientometric needs, surveys the key indicators of impact assessment that could best serve the APJCP’s future performance. Given that the fact that the journal is indexed in the Scopus database, the indicators introduced by this indexing database are mainly evaluated and discussed in this paper.

One of the common ways to evaluate the impact of a research is to employ bibliometric indicators (Brutscher et al., 2008; Hanney et al., 2004). Bibliometric gives a general picture of a research visibility and relies on the indicators to determine its impact and compare the results with those of other research coming from other organizations and projects (Krapels et al., 2015). One of the key indicators of bibliometric is a citation and it has been used for many years to measure the effectiveness and usefulness of a research report. Citation indicates the effectiveness of a paper in the academic world and the extent to which the paper is recognized by fellow researchers. In addition, a citation is an indicator of the scientific credit gained by a paper; it shows the scientific value, effect, and applicability of the paper in a specific field (Hanney, 2005; Cheek et al., 2006; Nieminen et al., 2006). In fact, a citation is the best crude metrics to measure the extent of effect and recognition.

In light of this, indexing databases in the world utilize many citation based indicators such as citation impact, percent of documents cited, percent of documents in top 1 percent, percent of documents in top 10%, hot & highly cited papers (based on the number of papers and received citations), journal impact factor (immediacy and 5-year), Eigen-factor, article influence, journal and category normalized citation impact, Cite Score metrics, SCImago Journal Rank (SJR), Source-Normalized Impact per Paper (SNIP), H-index of authors and institutions, and field-weighted citation impact to evaluate the research performance of authors and organizations.

Knowing about the importance of citations in the assessment, it is reasonable for the authors and organizations to find ways to increase citations for their scientific works. They try to publish papers with higher chances of citations and higher impact.

Although, using citations and other traditional indicators are still a mainstream metrics among policymakers to measure the effect of research, they are not adequate to make a reliable assessment of the impact of many research outcomes in the field of health and medicine (Brutscher et al., 2006; Hanney et al., 2004). Just reliance on above indicator has been criticized. There are several reasons for this including the fact that the indicators do not encompass non-traditional forms of research communications such as books, drafts, technical reports, conference representations, posters, and datasets and these forms are neglected in the assessments. In addition, in some specific fields of science such as humanities and social sciences, these forms are actually the main outputs of research works. Additionally, along with academic addresses, every research work has its own potential users in the society who can evaluate the paper’s impact. Thereby, a research outcome should be given the opportunity to publish its findings via other ways of communication such as papers with large groups of readers. Some of these papers even without an impact factor can play a critical role in the publication and promotion of science (Hanney, 2005; Krapels et al., 2015; Wooding et al., 2004). For instance, although Cochrane library has been one of the main information sources for a long time, it does not have a high impact factor. The point is that the information published by this organization is widely used mainly in developing guidelines, policy making, and clinical decisions (products are not cited by mainstream journal article publication) (Bunn, 2010). In addition, the extent to which an article is read is not taken into account in the assessment of a research report, but the article might still be highly useful in practice by professionals. Another limitation of these indicators is the ambiguity in the interpretation of the indicators. Although citation is an indicator of a paper’s quality, without taking the field and/or context into account, it cannot be considered as an index of quality. Such index is highly dependent on the time that an article is exposed to other scientists. This time while might well be over many years, however, it is a consensus among scientomerist that a duration 2 years is needed for a paper to receive the volume of citations that such a paper deserve (Krapels et al., 2015; Rosenkrantz et al., 2017).

Taking into account the role of information technology in the expansion and development of social media, the development of a ground for easier and faster publication, and the emergence of online research communication, it is essential to introduce new indicators to measure the impact of research along with the traditional indicators. Spotting this need, a new group of measures and metrics called altmetrics and article-level metrics (ALMs) have been introduced to evaluate and assess the impact of papers. In addition to citations, these metrics cover other indicators such as views, downloads, sharing, saving, and discussions.

Galician has listed altmetrics including the count of downloads, the number of clicks with the focus on readability, publications, weblog re-usage, social media, and social bookmarks indicators (Galligan et al., 2013).

altmetrics is an attempt to introduce alternative indicators, without emphasis on citations and the impact factor of a journal, to analyze research quantitatively by introducing supplementary indicators from different databases. In addition, altmetrics introduce measures at the article level by monitoring the available evidence and activities on social media. Therefore, it is possible to accurately evaluate the impact of any document.

Altmetrics has drawn a great deal of attention due to the limitations of the traditional indicators. The former provides more information about the use and reuse of research works by academic members. But, the altmetrics have the potential to be used as a valuable supplementary for the traditional bibliometric indicators. Moreover, altmetrics can show a real-time count of the readers of a specific article (Rosenkrantz et al., 2017).

Along with the expansion of research through altmetrics and the development of the web and social media, different tools have been proposed to survey the indicators and databases, and among them, Story impact, Plum analytics, altmetrics explorer, and Plus impact explorer are notable. These tools gather data from published works from different sources.

One of the databases that utilize both traditional and new indicators is Scopus. Recently, Scopus database has added a new section to its site known as PlumX Metrics, which reports indicators at the three levels of an author, article, and journal. These indicators and their implication at the three levels are introduced here:

Field-weighted citation impact: Citations are weighted based on the field and subject category of the article. This indicator compares the number of citations of an article with the average citations of similar papers indexed in Scopus. So that when the index is higher (lower) than 1, the citation of the article is higher (lower) than the average world citation of similar articles. Where the index is equal to 1, the citation of the article is the same as the average world citation of similar articles.

Citation benchmarking: An indicator of the citation ranking of the articles compared to other articles in the same field, duration since publication and the type of the article. In other words, the indicator compares the citations of a paper with the average citations of similar papers. When the indicator is higher than 99%, the paper is among the top 1% papers in terms of citations and ranking. The date of publication, the subject, and type of documents are taken into account in computing this indicator.

Usage

This indicator in this group provides the information beyond the citations of a paper. The information includes times that abstract is viewed, clicked, downloaded, full text viewed, HTML viewed, supporting data viewed, PDF viewed, a video played, link outs from.

Captures

This indicator highlights the return to a paper by the readers. The importance of these indicators lies in the fact that they show the chance of making reference to the paper by the readers in the future. Such a return is implicated by placing the article in bookmarks, favorites, watchers, subscribers, exports/saves, readers, and followers.

Mentions

The indicators in this group indicate the level of engagement with a research work. The key indicators in this group are forum topic counts, comments, blog mentions, news mentions, links, and reviews.

Social media

The indicators in this group refer to the number of addressees gained by a research product. Among the main indicators in this field are likes, shares, likes and comments, ratings, recommendations, scores, and Tweets.

Citation

The indicators in this field indicate the number of citations of paper archives. Some of the indicators in this field are citation indicators, patent citations, clinical citations, and policy citations.

Along with the advantages, Altmetrics has some disadvantages as well. Some of these disadvantages pertain to the fact that the altmetric indicators are new depend on many facts of uncertainty such as the paucity of supporting evidences, dependence on promotion of research results, lack of structure in the available tools, ambiguity of computation methods, lack of transparency of the quality indicators, lack of well-defined standards, high risk of false interpretation, high diversity of the indicators, high diversity of the data sources to obtain the indicators, repetitious articles that are not monitored in social media, and a lack of integrity of the information provided by different tools (Bornmann, 2014; Frank et al., 2012).

Despite the fact that citations basically indicates the visibility of a paper (Vincent et al., 2011), results of studies have shown that with higher altmetrics indicators such as reviews and uploads, and the paper has a higher chance of drawing attention on the web and higher chance of being cited (Nieminen et al., 2006; Thelwall et al., 2014; Rosenkrantz et al., 2017). In the light of this fact, authors may need to make their products more accessible on the web and via social media. Doing so, the authors would enjoy higher visibility and citations along with the advantage of having their works evaluated.

In summary, the recommendation to APJCP’s board members is to align its approach in its evaluation of its performance on citation as the key indicator and take into account the shortcoming of just citation by utilizing supplementary indicators (altmetrics) to evaluate journals performance. This is highlighted by the fact that the major indexing authorities have already adopted this strategy of in evaluation a journal’s performance (https://service.elsevier.com/app/answers/detail/a_id/12031/supporthub/scopus/#plumx).