COVID-19 citation pandemic within the psychological knowledge domain.

Presented study explores the knowledge domain of psychological research published in 2020 and 2021. Metadata from 156,942 psychology papers available in Scopus were analyzed using citation analysis and bibliographic mapping techniques. Having in mind the ubiquity of the COVID-19 pandemic and the numerous ways it has affected people's lives, the fact that COVID-19-related papers represent only 2% to 7% of the total academic production in psychology may seem rather low. However, these papers have attracted much more attention from the public than non-COVID papers. They were also cited two to eight times more frequently, depending on the measure used, and account for 16% to 19% of total citations to psychology papers. Results show that early-stage researchers and those who had fewer articles in Scopus have benefited more from publishing COVID papers. They have managed to boost their average citation rates to the level of their colleagues who were much longer active and previously had higher citedness. Results indicate that the authors citing behavior largely follows the cumulative advantage pattern. Psychological research in general is mainly focused on mental health, anxiety, depression, and stress. This trend is even more fostered due to the pandemic since some of these topics are often analyzed as typical emotional reactions to COVID-19. Other relevant issues are also very well covered, except for the question how scientific results are communicated to the public. The role of "hot" papers was elaborated from the perspective of research evaluation practice. Supplementary Information: The online version contains supplementary material available at 10.1007/s12144-022-03146-3.

Introduction

COVID-19 pandemic is probably the greatest challenge the world has faced since the World War II. It has caused global crisis that affected all aspects of people’s lives. At the beginning of the pandemic, researchers, media, and public were mainly focused on the physical health consequences of the infection and its implications for the mental wellbeing. However, two years later, with the pandemic still raging and more than 6 million deaths worldwide due to COVID-19, it is obvious that the pandemic will have more far-reaching and long-term social, economic, and even political consequences. Our daily routines and typical behaviors have dramatically changed and are being disrupted because of the imposed measures of social distancing, travel restrictions, and lockdowns. Most of those behaviors, being largely based on interpersonal contacts, moved to the various platforms for online education, conferencing, shopping, business, and communication. Economic activity of most countries felt significantly while job losses reached levels similar to those of the previous global financial crises. Trust in government and public institutions is being questioned since they are frequently perceived as responsible for failing to restrain the pandemic consequences. Finally, new potentially problematic behaviors have emerged such as anti-vaccination, anti-mask movements, and non-compliance with, or opposing to COVID-19 pass politics.

Being such a pervasive and ubiquitous issue, COVID-19 pandemic has attracted attention of researchers in all scientific fields. There are currently more than 230.000 documents related to this subject indexed in the Scopus database and this body of knowledge is exponentially growing. In 2021 the number of papers related to COVID-19 nearly doubled compared to 2020. Although the largest proportion of these papers is expectedly coming from the fields of medicine, biochemistry, and immunology, researchers from various social science fields are increasingly shifting their focus towards topics related to the current pandemic. Table 1 shows the number of papers published in 2020 and 2021 indexed in Scopus. COVID-19 pandemic is obviously no longer a predominant matter of physical health and medical treatment, but it is becoming a worldwide and diverse challenge that requires interdisciplinary approach and the mobilization of experts from other fields of science, such as business, economics, psychology, information technology, and decision sciences. Perhaps the most illustrating fact is that the largest increase in the number of papers has occurred in the field of environmental science, indicating that COVID-19 pandemic has become a matter of our global setting.

Table 1

Number of papers related to COVID19 indexed in Scopus

Subject Area	Number of papers
	2020	2021	Increase in %
Environmental Science	3393	8789	159.03
Business, Management and Accounting	2298	5586	143.08
Psychology	2708	6452	138.26
Economics, Econometrics and Finance	1813	4297	137.01
Computer Science	4649	10,526	126.41
Decision Sciences	1214	2423	99.59
Social Sciences	10,625	21,164	99.19
Immunology and Microbiology	4929	9784	98.50
Health Professions	1871	3520	88.13
Nursing	3642	6814	87.10
Biochemistry, Genetics and Molecular Biology	7823	13,558	73.31
Pharmacology, Toxicology and Pharmaceutics	3599	5623	56.24
Neuroscience	2343	3619	54.46
Medicine	58,007	83,153	43.35

Papers may be classified into multiple categories

COVID-19 Pandemic From the Psychological Perspective

In the most cited psychology paper related to COVID-19, Van Bavel et. al. offered a valuable review of major research areas in which social and behavioral sciences can support COVID-19 pandemic response (Bavel et al., 2020). The first one is related to the management of threat perception. Initial stages of pandemics in general are characterized by the negative emotional reactions to the threat, particularly the inability to objectively estimate health risks when the virus is novel and unknown. Higher intolerance of uncertainty is shown to be related to weaker self-control, lower problem-focused coping, and higher anxiety, thus increasing the probability of an individual to perceive the pandemic as threatening (Taha et al., 2014).

The second relevant topic for psychological research is social and cultural influence on behavior during the pandemic. We are witnessing every day how different cultural settings can affect both the policy decisions set by the governments and the emotional and behavioral reactions of people, for example to imposed lockdowns, vaccination, and wearing masks. It is obvious that some countries have managed to cope with the pandemic more efficiently than others, and that some populations and regions seem to be more resilient than others. Studies have shown that much of these variations may be attributed to cultural factors, particularly the differences on dimension collectivism vs. individualism (Chen et al., 2021). COVID-19 policy implementation in countries where more emphasis is put on the wellbeing of the group were more successful than in the countries where personal freedoms are accentuated.

In relation to the previous issue, very relevant research topic is how to improve effectiveness of leadership on various levels. It is of crucial importance to build a resilient trust with leaders in order to promote socially responsible behavior and align individual and collective interests, both within personal or work groups, and nationwide. Social norms and social identities are critical for maximizing the effect of public health messages and hence it is important to clearly define target groups, appeal to higher-order group values, provide fair and solid public support, insist on balance between injunctive and descriptive norm information, and coordinate all actions with group members who will communicate messages not merely as some impersonal representatives of the authorities, but also as “one of us” (Neville et al., 2021). It is of crucial importance to offer an assistance and to cooperate with community support groups since behavioral reactions to the pandemic are a function of dynamic group processes, rather than a commonly expected and easily predictable tendency to respond to emergency situations (Drury et al., 2021).

In line with all above, very important question is how scientific results are communicated to the public. The virus pandemic was accompanied by an unpresented production and outreach of news, recommendations, admonitions, and statistical reports that grew to become a kind of “infodemic” (Banerjee & Meena, 2021). Although being informed about the precautions and actions is of vital importance for reducing stress and anxiety in such situations (Wang et al., 2020), studies have shown that continuous exposure to infection-related information, both those coming from the media and from social networks, can induce certain level of confusion due to the ambiguity of information (Garfin et al., 2020). This is particularly relevant having in mind that spread of useful and reliable information on COVID-19 shows similar pattern to that of dissemination of dubious data, misinformation, fake news, and rumors about conspiracy theories (Cinelli et al., 2020).

Finally, psychological research should also focus on issues related to the effective coping with stressors other than virus itself. Potential stressors caused by the pandemic may include social isolation, financial difficulties, disruption of educational processes and the quality of learning, lack of intimate relationships, travel restrictions, limited access to various events, managing information on relevant topics such as vaccination, and many more. Recent study revealed significant individual differences among young adults in coping styles and psychosocial functioning, confirming the importance of problem-focused coping, but only when combined with both resilience and coping flexibility, i.e. seeking information and finding diverse ways to detach from the stressful situation (Shigeto et al., 2021). This may include various mitigation strategies such as music listening (Kong & Wong, 2021), developing new interests (Pigaiani et al., 2020), using humor (Savitsky et al., 2020), staying connected through social media (Ellis et al., 2020), being physically active (Faulkner et al., 2020), and others.

Exploring “hot” topics in Science

There is no doubt that COVID-19 pandemic has become a globally relevant social and political issue as well as a prominent research topic in different scientific fields. Since the seminal work of Eugene Garfield on citation indexes as tools for investigating association of ideas (Garfield, 1955), various bibliometric techniques have been used to detect and explore such emerging areas of scientific research. Probably the easiest way to get this type of insight is to analyze the content of the most cited papers in a certain period. Clarivate Analytics offers two indicators based on papers’ citation rate in their Web of Science (WoS) databases. The first, called Highly Cited Papers, shows what papers are among the top 1% most cited papers in their respective field, published in a specific year. Another commonly used indicator is called Hot Papers and indicates that a paper is receiving the number of citations, within two months period after publication, that places it in the top 0.1% of papers published in the same field. It became common for scientific journals to recommend articles to the readers based on their citation rate. Furthermore, research evaluation is largely based on citation rates of journals, institutions, and individuals (Moed, 2005).

Although citation counts cannot be considered entirely objective measures of intellectual influence and research quality (Paulus et al., 2018; Zuckerman, 1987), examination of high-performance papers provides valuable framework for identification and differentiation of key and emerging topics, centers of excellence, and influential researchers (Tijssen et al., 2006). It should be noted, though, that comparisons based on citation measures are appropriate only within a single field or similar fields, since the dynamics of citation exchange largely differs among disciplines. Social sciences and humanities are generally known to have slower pace of theoretical development reflected in the higher mean reference age, higher citation rate of older literature, and lower volume of citations, when compared to “hard” sciences, such as biomedicine, life sciences, and engineering (Chi, 2019; Nederhof, 2006). In this context, psychology has a somewhat specific “bordering” status in the group of social sciences as psychologists tend to show patterns of citing and publishing behavior similar to that of the researchers in medicine and life sciences (Bonaccorsi et al., 2017; Engels et al., 2012; Yang & Zheng, 2019). Nevertheless, citation analysis has already been used to explore research fronts and predominant concepts in educational psychology (Walberg, 1990), evolutionary psychology (Webster et al., 2009), personality psychology (Allik, 2013), social psychology (Kruglanski et al., 2017), and cognitive psychology (Rothermund & Koole, 2018).

More comprehensive insight into emerging topics in scientific research can be obtained by using various mapping techniques. In this type of analysis, large volumes of bibliographic metadata are visualized in the form of graphs, usually based on the frequency and cooccurrence of keywords, but also on the strength of relationship among journals, institutions, researchers, or countries. Such bibliographic maps are useful means to explore the state, dynamics, and structure of scientific disciplines (Börner et al., 2003). One of the most illustrative examples of the knowledge domain visualization is the analysis of more than one million documents from the WoS by Boyack et al. (Boyack et al., 2005). The resulting map of this “backbone of science”, based on journals cocitation and intercitation, confirmed the aforementioned “bordering” position of psychology in the system of disciplines, showing it acts as a form of hub that makes connections between the field of social sciences and other disciplines. Bibliographic mapping has become a useful research tool in psychology as it may be used to explore patterns of collaboration (Quayle & Greer, 2014), topics’ disciplinary reach (Rusk & Waters, 2013), research hotspots (Waqas et al., 2019), growth of publications (Anglada-Tort & Sanfilippo, 2019), and even a methodological meta-language used in psychological research (Flis & van Eck, 2018).

Purpose of the Study

The purpose of this study is twofold. The first goal is to compare the citation rates of psychology articles on COVID-19 pandemic to those related to other topics and published since the pandemic outbreak. The intention was not only to estimate the impact and outreach of the COVID-related papers in psychology, but also to investigate the effects of moving research focus towards emerging “hot” topics on research evaluation practice and the assessment of research quality. The other goal is to explore the domain of research related to COVID-19 and to identify key topics and concepts of interest for psychologists. Results will be elaborated in the context of the aforementioned major research areas in which social and behavioral sciences can support COVID-19 pandemic response. This is not the first analysis of COVID-19 research domain using bibliometric techniques. However, previous studies didn’t focus on citation rates and were more general in scope (Aristovnik et al., 2020). In some of them, the authors didn’t manage to overcome database limitations in the number of retrieved articles (Zyoud & Al-Jabi, 2020), and some were focused only on a single topic, such as mental health (Maalouf et al., 2021; Mækelæ et al., 2020).

Method

The analyses presented in this paper covered all articles and reviews published in 2020–2021 in psychology journals indexed in Scopus (SUBJAREA(PSYC)). All the data were downloaded in December 2021, using the Python wrapper for the Scopus RESTful API called pybliometrics (Rose & Kitchin, 2019). Documents were divided into two samples. The first consisted of 7,909 papers containing the term “COVID-19” in title, abstract, or keywords (TITLE-ABS-KEY(COVID-19)), and the second of 149,033 papers not containing this term (NOT TITLE-ABS-KEY(COVID-19)). Apart from the typical metadata, such as author ID, date of publication, citation counts, and keywords, additional journal data and alternative article level metrics (PlumX) were also retrieved. Each article record was complemented with journal’s 2019 values of SNIP (Source Normalized Impact per Paper) and SJR (SCImago Journal Rank), and the number of policy citations, captures (e.g. adding to Mendeley), mentions in news and blogs, and social media indicators, such as tweets and Facebook likes.

In the second phase, the focus was on profiles of the authors of retrieved documents. The total number of researcher names with unique Scopus ID extracted from the metadata was 351,093. Since the retrieval of all author profiles was not feasible due to the Scopus quota limitations, the analysis was limited to the authors of highly cited papers. Only papers ranked in the first quartile in both samples were considered, specifically 1,111 COVID papers and 13,788 non-COVID papers. This resulted in the lists of 5,822 authors in the first sample, and 50,719 in the second. Author profiles contained the number of published papers, number of citations, number of coauthors, h-index, and the number of years active defined as difference between the publication year of his or her most recent and earliest paper available in Scopus. Additionally, for 5,822 authors of highly cited COVID papers, metadata from all their papers available in Scopus were downloaded. Bibliographic data were obtained using the Scopus AuthorRetrieval API for a total of 354,175 papers. Since the Scopus data contain information on the exact day of publication, it was possible to calculate average monthly citation rate for each of these papers. This was done to complement papers’ citation count and obtain a more precise indicator because the analyzed period was relatively narrow and lots of papers were published a short time before the data retrieval. The exact sample sizes varied among analyses due to missing data for some variables.

Finally, all articles from the first sample were used to create the map of COVID-19 knowledge domain in psychology. Maps were created using VOSviewer, a software tool for constructing and visualizing bibliometric networks (Van Eck & Waltman, 2010). VOSviewer uses a novel smart local moving algorithm intended for modularity-based community detection in large networks (Waltman & van Eck, 2013). Topic modelling approach was also considered as a clustering technique, but it produced too many artefacts. When applied to bibliometrics data, topic modeling is known to reveal similarities other than semantic ones, while co-word maps show components that can be designated more easily (Leydesdorff & Nerghes, 2017).

Due to the known fact that bibliometric distributions are highly skewed (Seglen, 1992), most of the analyses were conducted using nonparametric statistical procedures available in Python SciPy and Pingouin libraries. Plotly library was used for visualization, PyProcessMacro for generating regression models, and NetworkX (Hagberg et al., 2008) for exploring the structure of the keywords network.

Results

The results are presented in four sections. The first contains various comparisons of COVID and non-COVID papers’ bibliometric features. The second is focused on the analysis of the profiles of authors who published highly cited psychology papers in 2020 and 2021. The third section explores possible regression models for predicting papers’ citation rate in both samples. Finally, the fourth section contains bibliometric mapping analysis of the COVID-19 knowledge domain in the field of psychology.

Citation Rates and Popularity of COVID and Non-COVID Papers in Psychology

Table 2 shows bibliometric features of COVID and non-COVID psychology papers published in 2020 and 2021. All indicators suggest that COVID papers have significantly higher citation rate and popularity. They are cited almost 4 times more often that non-COVID papers published in the same period. This ratio is even higher in the group of papers cited 10 or more times. More than 15% of COVID and less than 4% of non-COVID papers are in this category. Almost 59% of COVID papers were cited at least once, which indicates significantly higher citation probability compared to psychology papers not related to COVID-19 (χ2 (1, 156,942) = 282.23, p < 0.01). Citation distribution in the group of COVID papers is slightly more skewed due to the larger proportion of highly cited papers. Only 863 papers (≈11%) account for three thirds of all citations to COVID papers. In the group of non-COVID papers, this percentage is 20%. The fact that not only mean, but also the median citation count for COVID papers is higher, further confirms the difference in citation probabilities of two samples of papers.

Table 2

Bibliometric features of COVID and non-COVID psychology papers available in Scopus

	COVID	Non-COVID
Number of papers	7909	149,033
Median number of authors	4	3
Citation rate of the most cited paper	1312	512
Average citation rate	7.29	1.94
Citation rate skewness	20.99	20.11
Percentage of cited papers	58.83	49.13
Percentage of papers cited 10 + times	15.13	3.79
Average (and median) citation rate of cited papers	12.39 (3)	3.94 (2)
Average number of policy citations	0.17	0.02
Average number of reader count	67.51	18.16
Average number of mentions in news and blogs	0.90	0.24
Average number of social media mentions	31.13	15.32
Average SNIP of journals	1.28	1.27
Average SJR of journals	0.97	1.11
Average SJR of journals for “hot” papers	1.50	2.51

PlumX metrics are in line with the conclusion that COVID papers have larger popularity than non-COVID papers, both within the group of fellow researchers, and within the general public. Although psychology papers are in general rarely cited in policy documents, this number is much higher for COVID papers. They are also more than two times more often mentioned in news, blogs, and social media. The number of captures (e.g. readers in Mendeley) basically reflects the ratio of citation counts for two groups of papers. In fact, correlation between the number of captures and the number of citations is high and significant in both samples: 0.79 for COVID and 0.66 for non-COVID papers. On the other hand, other PlumX metrics have much lower correlations with citation counts (Table 3). All correlation coefficients are statistically significant at 0.01 level, but mainly due to the very large sample of papers. Hence, their practical significance is low, which indicates that PlumX metrics indeed captures aspects of papers’ impact other than the one measured by traditional citation counts.

Table 3

Correlations among papers’ citation rates and PlumX Metrics indicators

	Citations	Policy citations	Captures	Mentions	Social media
Citations	1.00	0.33	0.79	0.27	0.25
Policy citations	0.12	1.00	0.33	0.24	0.18
Captures	0.66	0.12	1.00	0.26	0.25
Mentions	0.14	0.08	0.15	1.00	0.29
Social media	0.23	0.07	0.30	0.18	1.00

* Upper triangular contains correlations for COVID papers and lower for non-COVID papers. All correlation coefficients are significant at the 0.01 level.

While most of the indicators presented in Table 2 are higher for COVID papers, differences in impact measures for journals were analyzed papers were published show a different pattern. Average journals’ SNIP values do not differ significantly for COVID and non-COVID papers, while the average SJR values are statistically significant lower for COVID papers. This difference is even more pronounced when only “hot” psychology papers are considered. For the purposes of this analysis, “hot” is defined as being ranked in the top 1% of all papers, both COVID and non-COVID, based on the citation count. At the time of the analysis, this threshold was 21 citations. SJR is generally regarded as a measure of a journal’s prestige (Colledge et al., 2010). It does not only limit the proportion of journal’s self-citations included in the calculation, but also considers the impact of citing journals. Hence, the difference between the average SJR of all vs. “hot” papers shows that journal’s prestige is one of the key determinants of the article’s probability to be cited. However, in the case of COVID papers, the topic of the paper significantly boosts its citation rate. As a frame of reference, it should be noted that the top 20 psychological journals in 2019 had SJR values between 3.8 and 10.2. Values of approximately 1.3 and above place a journal in the first quartile in the field of psychology.

Finally, it is worth mentioning that roughly two thirds of psychology papers are authored by more than two authors. Only 11% of papers were written by a single author. Median number of authors is significantly higher in the group of COVID papers where the quarter of articles were written by the teams of more than five researchers.

Research profiles of authors of highly cited papers published in 2020 and 2021

This section explores the overall productivity and citedness of 56,541 authors of the most cited psychology papers published in 2020 and 2021. Almost 60% of 5,822 COVID authors published only one (≈37%) or two COVID papers (≈21%). On the other hand, 379 of them (≈6%) published 10 or more COVID papers. The mean value of the number of COVID papers per author is 3.54 and the median is 2. When it comes to the proportion of COVID papers within the total number of published papers, for 785 COVID authors (≈13%) there were only COVID papers available in Scopus. For 1,441 authors (≈25%), COVID papers account for half or more of their productivity. In the case of citation rates, the situation is somewhat different. For almost 40% of the analyzed authors citations to COVID papers account for 50% or more of their total citation count. Average citation rate for COVID papers published by COVID authors is more than double that for their papers not related to COVID-19: 43.85 vs. 20.45 citations per paper respectively. Similar ratio may be observed if median values are compared: 20 vs. 12 respectively. This difference is even more pronounced when the average monthly citation rates are considered. On average, COVID papers were cited 3.13 times each month, which is almost eight times more often than non-COVID papers published by COVID authors. The latter were cited only 0.37 times per month on average. Regardless of the measure used, Wilcoxon signed-rank tests indicate significant differences in citation rates between COVID and non-COVID papers published by COVID authors.

Figure 1 shows the relationship among several bibliometric properties of the COVID authors research profiles after 0.5% of upper outliers for average citation rate and active years were removed. Proportion of COVID papers is plotted on the x-axis, average citation rates for all papers on the y-axis, bubble size denotes the number of papers available, and the color of the bubbles indicates the number of active years. Red horizontal line marks the median value of the COVID authors’ average citation rate. The most noticeable trend is that senior researchers have in general lower proportion of COVID papers. This is somewhat expected given the fact that early-stage researchers are initiating their careers in the era of COVID-19 as an extremely relevant and attractive research topic. On the other hand, the pandemic as a research topic has probably offered an opportunity for some senior authors to publish in international journals and COVID papers may have been their first papers available in Scopus. However, currently available data provide no means to test this hypothesis.

Fig. 1

Relationships among several properties of the COVID authors bibliometric profiles *

Nevertheless, the size and the color of bubbles on Fig. 1 indicate that the authors who are less experienced in publishing papers in international journals have benefited more from covering COVID-related topics. Many of them managed to achieve above median citation rate simply by publishing a few COVID articles, thus reaching the level of citedness of researchers who were longer active and published significantly more papers. Consequently, correlation between the number of years active and the proportion of citations to COVID papers within the author’s total citation rate is very high and negative (ρ = -0.85, p < 0.01). It is relatively high and significant even if it is calculated partially, controlling for the number of documents available in Scopus (ρ = -0.45, p < 0.01).

In order to further explore the impact of COVID papers and their effect on authors’ citedness, the comparisons were made between the profiles of authors who published most cited COVID or non-COVID papers in 2020 and 2021. For 1,454 authors who appeared on both lists, only the most cited paper was taken into account, which means that those who have received more citations for a COVID paper appeared only on the list of COVID authors. Figure 2 shows the correlation between the authors’ average citation rate with and without their most cited paper published in 2020 and 2021. In general, this correlation is very high (ρ = 0.97, p < 0.01). However, it is even higher in the group of authors whose most cited papers are not related to COVID-19 (ρ = 0.98) than in the group of COVID authors (ρ = 0.81). As it can be seen from the Fig. 2, this is mostly due to the fact that COVID papers significantly boosted average citation rate of COVID authors. Median citation rate with the most cited paper for COVID authors is significantly higher compared to the citation rate without this paper (16.3 vs. 11.7), while this ratio is opposite for the non-COVID authors (13.6 vs. 14.3). Finally, the size of the bubbles on Fig. 2 indicates larger increase in the citation rate for the group of authors who have shorter period of activity measured by the number of active years.

Fig. 2

Correlation between the authors’ average citation rate with and without the most cited COVID or non-COVID paper published in 2020–21 *

Predicting Citation Rates for Highly Cited Psychology Papers

Correlations between the (monthly) citation rates of highly cited psychology papers and other variables used in this study are statistically significant, but generally rather low. This means that possibility to predict papers’ citation rates is questionable. It may be due to the simple fact that presented analysis has covered only 25% of the most cited articles and the range restriction may have affected these results. Several regression models were tested and none of them sufficiently well explained papers’ monthly citation rate as a criterion. Nevertheless, some useful information may be obtained. Table 4 shows Spearman correlation coefficients among the various bibliometric indicators of the most cited psychology papers. Correlation patterns obviously differ between COVID and non-COVID papers. Most notably, journal’s SJR and citation rate of the most cited author of the paper have stronger predictive power in the group of COVID papers. Although it is not directly related to the purpose of this study, it should be noted that Hirsch index does not provide much more additional information on the authors’ impact compared to the authors’ average citation rate.

Table 4

Correlations among various papers’ bibliometric indicators*

	MCR	SJR	ACR	PCR	MCA	AY	H	NA
Monthly citation rate (MCR)	1.00	0.30	0.17	0.17	0.22	0.00	0.16	0.16
Journal’s 2019 SJR (SJR)	0.19	1.00	0.30	0.34	0.35	0.06	0.25	0.25
Authors’ average citation rate (ACR)**	0.11	0.31	1.00	0.82	0.93	0.67	0.95	0.25
Authors’ papers’ ACR (PCR)	0.12	0.36	0.81	1.00	0.74	0.58	0.76	0.17
CR of the most cited author (MCA)	0.10	0.30	0.93	0.71	1.00	0.56	0.85	0.55
Average authors’ active years (AY)	0.01	0.11	0.67	0.58	0.58	1.00	0.74	0.00
Average authors’ Hirsch index (H)	0.12	0.28	0.96	0.76	0.86	0.71	1.00	0.15
Number of authors (NA)	0.00	0.09	0.22	0.08	0.52	0.05	0.15	1.00

* Upper triangular contains correlations for COVID papers and lower for non-COVID papers. All correlation coefficients are significant at the 0.01 level.

** Authors’ average citation rates were calculated excluding the most cited paper published in 2020–21.

Possible interaction effects of some of the indicators were tested using the model in which journal’s SJR was used as a predictor, article type (COVID vs. non-COVID) and citation rate of the most cited author as moderators, and article’s average monthly citation rate as a criterion. Average range of the authors’ active years and the average citation rates of the authors’ previously published papers were used as controlling variables. Variables in the tested model explained 21% of the criterion variance (R2 = 0.21, F(9, 14,601), p < 0.01). Although the coefficient of determination is rather low, it should be noted that the interaction effect of SJR and MCA is significant. Conditional effects of these variables are shown in Table 5. In the set of non-COVID papers, effect of SJR on papers’ monthly citation rate is significantly lower compared to COVID papers. It is also practically the same over all levels of MCA, that is at the 10th, 50th, and 90th percentile of the MCA distribution. On the other hand, probability for COVID papers to be highly cited increases with the increase of journal’s SJR and the citation rate of the most cited authors of the paper. This result suggests that citation rates of COVID papers to some extent follow the pattern of the so-called cumulative advantage.

Table 5

Conditional effects of SJR on papers’ monthly citation rate for different moderator levels

MCA (percentile)	COVID	Effect	SE	t	p	LLCI	ULCI
392 (10)	No	0.04	0.01	3.88	0.00	0.01	0.06
392 (10)	Yes	0.26	0.05	5.48	0.00	0.16	0.35
4847 (50)	No	0.04	0.01	4.12	0.00	0.02	0.06
4847 (50)	Yes	0.37	0.04	8.32	0.00	0.28	0.46
34,693 (90)	No	0.05	0.01	4.31	0.00	0.02	0.07
34,693 (90)	Yes	1.16	0.05	22.74	0.00	1.06	1.26

Regression models for predicting the citation rate of analyzed papers have most probably failed due to the large skewness of bibliometric distributions and low discriminative power of citation counts for papers that are sparsely cited because they made up most of the sample. Nevertheless, some categorical predictions seem reasonable since certain types of papers have larger probability to be cited. Figure 3 shows the median citation rate of COVID and non-COVID papers in relation to their type (articles vs. reviews) and accessibility (open access vs. restricted access).

Fig. 3

Median, quartile range, and non-outlier range of citation rate for different types of psychology papers

Kruskal–Wallis test indicates that differences in citation rates of various types of papers are statistically significant (H(7) = 2162.04, p < 0.01). Although COVID papers have much larger citation rate variability, they are generally more often cited than non-COVID papers. The most frequently cited category of papers is COVID open access reviews. Reviews are significantly more often cited in the group of non-COVID, but not in the group of COVID papers. On the other hand, providing the open access to an article significantly increases its citation rate for COVID papers, but not for non-COVID papers.

COVID-19 Knowledge Domain Visualization

The final part of the results section contains exploratory analysis of the COVID-19 knowledge domain in the field of psychology. Figure 4 shows the map of author keywords from 7909 COVID papers. Node sizes on the graph denote keywords frequency, and the edges represent the strength of their connection based on the number of cooccurrences. Some nodes were removed to improve the readability of the map. Full-scaled network is stored in the JSON file available as a supplementary material for the article (Supplementary S1). It can be uploaded to the VOSviewer online web application (https://app.vosviewer.com/) and studied in more detail.

Fig. 4

Cooccurrence map of author keywords from 7909 psychology papers related to COVID-19

VOSviewer’s algorithm has classified COVID papers into as much as 25 clusters which indicates that researchers have covered a wide variety of COVID-19-related topics. However, five of them are dominant and mental health is a central topic that ties them all together. The largest cluster of papers covers the issues of emotional responses to the COVID-19 pandemic (red). Stress, anxiety, and depression are the most studied reactions to the threat and uncertainty, but also to the lockdown, loneliness, and quarantine measures. Second largest and more heterogenous cluster of papers covers the problems related to social and cultural influence on behavior during the pandemic (green). The main topics in this cluster are the use of social media, social distancing, conspiracy theories, social norms, attitudes towards vaccination, and risk perception and risk communication.

From a descriptive point of view, the network visualization suggests that preventive behaviors are often studied in the context of personality traits, usually within the Big Five and HEXACO models. Personality traits seem to be recognized as relevant factors in emotion regulation processes and the development of effective strategies for coping with stressors and fear. Psychological well-being and distress appear as the most relevant facets of mental health. Third cluster (blue) is comprised of papers covering the use of digital technologies as learning environments (online education, blended learning, virtual reality). The fourth and fifth clusters (yellow and violet) are smaller regarding the scope and size. Papers from these clusters are dealing with the problems of using digital technologies for therapy interventions (online health) and negative behaviors caused by the COVID pandemic (drug abuse).

It should be noted that clusters made up of less frequent keywords are more heterogenous in nature. For example, papers dealing with the problems of physical patients are classified into the fifth cluster (cancer, caregivers, dementia, Alzheimer’s disease). Interestingly, one segregated cluster is also noticeable and is relatively large (gray). Papers from this cluster cover rather specialized issues of mitigating COVID-19 effects in prison population. Three more clusters are worth noticing, although they are relatively small compared to the previous. They cover the issues of COVID effects in work setting, domestic violence and child abuse, and leadership, decision making, and crisis management. The complete list of author keywords with their frequencies and cluster memberships is provided as a supplementary material (Keyword clusters.csv).

In order to further explore specific aspects of COVID-19 research in psychology, most frequent author keywords from COVID papers were compared to those from non-COVID papers. Table 6 shows degree centrality measures for the most common keywords in COVID and non-COVID papers. Higher values indicate higher centrality of a term, which means that it has more connections to other terms. Values are normalized by dividing by the maximum possible degree in a simple graph n-1 where n is the number of terms. Basically, the most popular and the most frequently studied topics are similar between the two sets of papers. Mental health, stress, depression, anxiety, and well-being are the most represented subjects in psychological research. Topics that frequently appear in non-COVID papers but are not to the same extent covered in the context of COVID-19, are memory and cognition, autism spectrum disorder, mindfulness, and ageing.

Table 6

Degree centrality (DC) for 30 most frequent keywords in COVID and non-COVID papers

COVID papers					Non-COVID papers
Keyword	DC	Keyword	DC		Keyword	DC	Keyword	DC
mental health	0.50	universities	0.21		adolescents	0.86	assessment	0.56
anxiety	0.49	children	0.20		depression	0.81	drug (ab)use	0.54
stress	0.43	healthcare workers	0.19		anxiety	0.76	emo. regulation	0.54
depression	0.40	loneliness	0.19		children	0.74	motivation	0.54
well-being	0.35	technology use	0.19		mental health	0.68	PTSD	0.54
lockdown	0.39	social media	0.18		well-being	0.68	trauma	0.54
quarantine	0.31	emotions	0.17		meta-analysis	0.67	ageing	0.53
adolescents	0.28	adolescents	0.18		autism	0.66	sleep quality	0.53
online health	0.28	public health	0.17		emotions	0.65	culture	0.52
resilience	0.27	parenting	0.16		stress	0.65	mindfulness	0.52
coping	0.25	PTSD	0.16		interventions	0.62	qualit. research	0.52
online education	0.24	sleep quality	0.16		cognition	0.61	resilience	0.52
fear	0.23	social support	0.16		parenting	0.6	education	0.51
distress	0.22	students	0.16		longitudinal study	0.59	social support	0.50
social distancing	0.22	trauma	0.15		personality traits	0.57	students	0.50

Research in personality, although appearing in the map in Fig. 4, does not have such a central position as in the group of non-COVID papers. As it may have been expected, psychological research related to COVID-19 is more focused on the problems of psychological distress and fear caused by the pandemic, particularly the effects of social isolation and health threat. Degree centrality for the term “adolescents” indicates that significant proportion of research in both sets of papers relies on samples constituted of young people, particularly students. Finally, degree centrality distributions differ between the two groups of papers in a sense that values are more skewed for COVID papers. This is partially due to the larger sample of non-COVID papers, but also because COVID-related research in psychology is mostly focused on mental health and most of the studies have covered to some extent the problems of anxiety, depression, and stress as the most common emotional reactions to the pandemic. With a few mentioned exceptions, COVID-related research in psychology basically reflects the topics of non-COVID research, although non-COVID papers are grouped into larger and more homogeneous clusters (see Non-COVID.png and Non-COVID.json in supplementary material).

Discussion

The primary motivation behind this study was related to the problem of evaluating research excellence, particularly in the context of “hot” and popular topics. It was shown that COVID papers have attracted much more attention by the public and significantly more citations than papers not related to COVID-19. This is to be expected having in mind the importance and numerous ways the pandemic has affected people’s lives. On the other hand, relatively small proportion of COVID papers within the total academic production in psychology does not seem to reflect the general ubiquity of the pandemic. The proportion was 2% in 2020 and it increased to 7% in 2021. This could partially explain high citation rates of COVID papers since the academic community is simply starving for new knowledge on COVID-19. COVID papers have accounted for 16% of all citations to psychology papers in 2020, and more than 19% in 2021. However, this may also imply that citations to these publications should be distributed more uniformly. On contrary, citation distribution for COVID papers is more skewed than that of non-COVID papers. The fact that only 11% of COVID papers account for three thirds of all citations to this category of publication, raises the question of the authors’ motivation behind the decision what to cite. Results presented in this paper partially support the notion that citing is not always the way to acknowledge intellectual contribution of fellow researchers, but is also motivated by other, non‐academic factors (Bornmann & Daniel, 2008). Although COVID papers are generally published in journals of lower prestige than non-COVID papers, correlation between the citation rate of these papers and the SJR of journals in which they were published are higher. Having in mind that the effect of the type of paper is also more significant for COVID papers, it may be concluded that authors are simply deciding to cite papers that are already highly cited and influential (higher SJR), more easily accessible (open access), and provide convenient and easy to read outlines of the current topic (review articles). Citations to COVID paper seem to follow the patterns of cumulative advantage or the co-called Matthew effect in science (Merton, 1968).

Regarding the motivation behind the authors’ citing and publishing behavior, it is also evident that a significant proportion of researchers has recognized they could “profit” from publishing papers on COVID-19. Depending on the measure used, COVID papers are cited two to eight times more often than non-COVID papers by the same authors. For some 13% of the authors, COVID papers are the only publications available in Scopus. Early-stage researchers or those who had fewer published paper before the pandemic, have more COVID papers and have benefited more from publishing them. This could raise the question of the reliability of the results presented in those papers since the answers to one of the most challenging problems in current psychology are offered by the researchers who are less experienced in conducting the research of the international level of significance. Even if this statement is too harsh, decision makers should bear in mind that certain publishing strategies may be used to significantly boost commonly used measures of scientific excellence. By publishing mostly or exclusively COVID papers, a significant number of COVID authors have reached the average citation rates of their colleagues who were more active, more cited, and more productive before the COVID-19 pandemic.

The second purpose of this study was to explore the knowledge domain of psychological research published during 2020 and 2021. Citation analysis and bibliographic mapping techniques were used to estimate the impact of “hot” papers in psychology and explore the most popular topics discussed. In general, all five major research areas mentioned in the introduction are covered to some extent with a few additional research questions that were opened. Presented analysis shows that mental health, anxiety, and depression are the most often investigated subjects in psychology in general and COVID-19 pandemic has only further fostered this area of scientific investigation. Management of threat perception seems to be the most elaborated subject in the context of COVID-19. This should be considered a positive trend since recent studies indicate that current models of anxiety are not sufficient to fully explain levels and breadth of uncertainty distress caused by the pandemic as an objective threat that provokes the felling of insecurity (Freeston et al., 2020). Furthermore, the scale and duration of the pandemic have transformed the very notion of uncertainty into a multifaceted issue that is not related merely to the perception of threat of infection. It should also cover other aspects of (intolerance of) uncertainty, such as economic (Altig et al., 2020), social (Olivera-La Rosa et al., 2020), and academic (Murphy et al., 2020). While the problem of mitigating economic uncertainty seems to be underrepresented in the current psychological research, social and academic uncertainty are covered by the significant proportion of papers. Two distinctive clusters are observable on bibliographic map, one related to social norms, politics, trust, and similar social issues, and the other to the implementation of digital technologies in educational context.

Regarding the social and cultural influence on behavior during the COVID-19 pandemic, the authors have mainly focused on the effects of preventive behaviors, social distancing, the role of social media, the influence of conspiracy theories, and the attitudes towards vaccination. Researchers have also covered the issue of psychological factors underlying compliance with COVID-19 guidelines. This is relevant not only because social and cultural factors may be considered as determinants of the adherence to COVID-19 regulations but also because they are shown to be manifested through variations in distress level and other psychological outcomes related to the perception of and satisfaction with the measures imposed by governmental institutions (Mækelæ et al., 2020). In relation to the role of the government in controlling COVID-19, it should be noted that the problem of how to improve effectiveness of leadership in emergency situations seems to be somewhat underrepresented compared to the other topics. Even smaller number of papers have dealt with the question of how scientific results are communicated to the public, for example through various citizen science projects that are shown to be useful in improving the efficacy of science communication and boosting the engagement of public in the context of health issues (Sadiković et al., 2020).

Finally, the issue of effective coping with stressors is also very well covered. Coping is most frequently studied in the context of loneliness, fear, uncertainty, and burnout. Most often investigated stress mitigation strategy was physical activity. “Quarantine” and “lockdown” are among the keywords having the highest frequency and centrality. In relation to that, telepsychology and online therapy have emerged as a relevant topic that sparsely appears in non-COVID papers. Presented results have shown that research in personality in the context of COVID-19 has lower (relative) centrality compared to non-COVID papers. This term occupied the “social” cluster along with the keywords related to social media, social distancing, preventive behavior, compliance with COVID guidelines, risk perception, and propensity for conspiracy theories. Personality traits are most often studied from the perspective of the Big Five and HEXACO models. “Social” cluster is connected to the cluster related to stress management and mental health through some form of a hub comprised from subjects of coping and psychological well-being. Finally, word frequency analysis has revealed that adolescents, particularly students, are very common sample in psychological research which could be a relevant information in the context of discussion on the generalizability and validity of research results in psychology.

The third contribution of this study is the evaluation of the models for predicting papers’ citation rates in psychology. Although some previous studies have presented models that are reportedly able to predict future citation rate of papers (Bai et al., 2019; Ponomarev et al., 2014; Ruan et al., 2020), this study provided no support for this. One possible reason is that presented analysis has used rather simple regression model and very limited set of possible predictors. However, having in mind that citation distributions are highly skewed and that roughly the half of the papers are not cited at all (51% in the non-COVID and 42% in the COVID sample) it seems that the cost–benefit ratio of this practice is simply too low. Citing behavior is in the first place a psychological phenomenon and it may be influenced by practically unlimited set of factors. Features of the paper, journal, citing and cited author, early citation rate, and the publication date are just some of them. Too much effort is required to obtain all this information and to adjust their parameters according to the specific area of research or specific set of papers.

On the other hand, this study has shown that some categorical predictions are possible. Type of the paper may be used to predict whether it will be highly cited, and it seems that some authors had this in mind when they were choosing to publish COVID vs. non-COVID paper, article vs. review or open access vs. restricted access publication. Regarding this last dichotomy, there were high expectations of an open access in excelling the availability of results and the speed of knowledge dissemination regarding COVID-19. Several funders and open access journals have even offered a possibility to financially support and to publish COVID articles for free. However, it seems that the current crisis did not motivate researchers to embrace the open science practice more deeply (Brainard, 2021).

Limitations of the Study

The main limitation of this study is that some of the analyses were performed on a limited set of (highly cited) papers due to Scopus quota restrictions. This may have affected presented results, particularly those related to predicting articles citation rates. Furthermore, queries used to retrieve articles may have resulted in some papers being falsely included or excluded from the sample. The fact that bibliometric distributions are extremely skewed also puts certain limitations to the results. Another potentially confounding issue is papers’ monthly citation rate. For some of them, information on the exact day of publication was set to January 1st and hence they could have had in fact less time to be cited that the date of publication indicated. Additional limitation stems from the fact that this is a preliminary analysis conducted in an early stage of publications’ life and hence further longitudinal exploration is necessary to obtain more comprehensive insight into the differences between the papers that cover new and emerging topics and those that are not. Finally, one should bear in mind that bibliometric mapping and clustering techniques are more of an explorative method and that different sets of papers and different parameters could have yield different results. However, results presented in this study support the existence of at least five solid clusters of COVID papers in psychology, regardless of what method was used.

Below is the link to the electronic supplementary material.

Supplementary S1 (JSON 1915 KB)

Supplementary S2 (JSON 21745 KB)

Supplementary S3 (PNG 7998 KB)

Supplementary S4 (CSV 89 KB)