Sharing emotion while spectating video game play: Exploring Twitch users' emotional change after the outbreak of the COVID-19 pandemic.

This paper examines how the COVID-19 pandemic associates with Twitch users' emotion, using natural language processing (NLP) as a method. Two comparable sets of text data were collected from Twitch internet relay chats (IRCs): one after the outbreak of the pandemic and another one before that. Positive emotion, negative emotion, and attitude to social interaction were tested by comparing the two text sets via a dictionary-based NLP program. Particularly regarding negative emotion, three negative emotions-anger, anxiety, and sadness-were measured given the nature of the pandemic. The results show that users' anger and anxiety significantly increased after the outbreak of the pandemic, while changes in sadness and positive emotion were not statistically significant. In terms of attitude to social interaction, users used significantly fewer "social" words after the outbreak of the pandemic than before. These findings were interpreted considering the nature of Twitch as a unique live mixed media platform, and how the COVID-19 pandemic is different from previous crisis events was discussed based on prior literature.

Introduction

After the novel coronavirus (COVID-19) invaded the world unexpectedly, ordinary lives were changed dramatically across both offline and online spaces. Since March 2020, when the virus was officially defined as a “pandemic” by the World Health Organization (WHO), many countries asked people to “stay at home” as an administrative order or nationwide campaign (Bokat-Lindell, 2020). Most social activities in offline spaces were restrained to a certain degree, and the phenomenon made the term lockdown as a common word, which refers to “a set of measures aimed at reducing transmission of COVID-19 that are mandatory, applied indiscriminately to a general population and involve some restrictions on the established pattern of social and economic life” (Haider et al., 2020, p. 2). In the period of lockdown, social distancing (i.e., staying physically distant from others) has been widely perceived as one of the must-keep rules, and avoiding another person on the street became a more thoughtful behavior than saying hello in close proximity.

Still, people have their social nature during a pandemic; this desire is partially if not entirely indulged in the online space. Facebook released that their total messaging had increased more than 50% in many of the countries hit by the virus between February and March 2020 when the COVID-19 began to spread globally (Schultz & Parikh, 2020). On another popular social media platform, TikTok, the number of monthly active users grew sharply from 500 million to 700 million from January to June in 2020, emerging as a way to refresh oneself from pandemic stress, sometimes even mocking the pandemic situation (Isaac & Frenkel, 2020; Sherman, 2020). Based on the increased use of social media platforms, global internet usage increased by up to 40% between January and March 2020, when the virus was spreading severely around the globe (Obringer et al., 2021).

Of a host of social media platforms that have been influenced by the COVID-19 pandemic, the current study focuses on how the pandemic was associated with Twitch use, which is an interactive live broadcast platform especially popular for game-playing streams. Since it was launched in June 2011, Twitch has been positioned as a unique mixed media platform, combining game play, each streamer's characteristic hosting, and live chat between users (Hamilton et al., 2014). Particularly regarding the COVID-19 pandemic, the platform is worth paying attention to, as it had significant changes in number of users, channels, and hours watched during the period. According to TwitchTracker, which is a website that tracks and analyzes statistics about Twitch, total hours watched on Twitch grew 53% between December 2019 and December 2020 (TwitchTracker, 2021). Within the same period, Twitch's number of streamers increased by 70%, and their number of users increased by 63% on average (TwitchTracker, 2021).

The current relationship study advances down two different research avenues by exploring how the current COVID-19 pandemic has been associated with Twitch users' emotion. First, it extends the domain of research on the association between crisis events and people's emotion on social media to a new type of media. In fact, previous studies in crisis event research have explored users' emotion on social media platforms, yet they collected data from mostly text-based media platforms such as Twitter (Murthy & Longwell, 2013) or Reddit (Ashokkumar & Pennebaker, 2021). Conversely, the current study examines messages in a multi-modal platform, Twitch, wherein users watch vivid game play videos, listen to audio, and write their messages all in real time. Given that multiple researchers note the necessity of researching diverse social media platforms not because of their number of users but the technology they afford to users (Lampinen, 2016), what this study examines will contribute to research on the association between crisis events and people's emotion on social media.

The other avenue that this study will advance is research on Twitch. Since its launch in 2011, a considerable body of research has explored communication on Twitch across diverse disciplines (Diwanji et al., 2020; Hamilton et al., 2014; Leith, 2021; Oh et al., 2020; Sjöblom & Hamari, 2017). While the earlier studies are more focused on introducing the platform as a new type of media (Hamilton et al., 2014) or examining users' motivation to visit the platform (Sjöblom & Hamari, 2017), recent studies have explored a variety of phenomena on Twitch based on its unique media form (Diwanji et al., 2020; Leith, 2021; Oh et al., 2020). However, to our best knowledge, none of them has explored how a society-level event is manifested in Twitch. A substantial proportion of the previous studies have explored how different channels (streamer) or content (game) make a unique community on Twitch and how the community's culture or other types of patterns (e.g., linguistic pattern on chat) vary (Hamilton et al., 2014; Leith, 2021). Therefore, this study on the pandemic expands the topics of the research avenue from community-level or game-level to society-level.

Two broad research questions are addressed in this study. First, we examine if there is a change in Twitch users' emotion in their chats before and after the outbreak of the pandemic. Prior research on crisis events found that people's sentiment is significantly influenced by crisis events such as the September 11 attacks, the Sandy Hook Elementary School shootings, and Hurricane Sandy (Cohn et al., 2004; Miller, 2015). As uses and gratification theory suggests that people deliberately choose media to satisfy their needs (Katz et al., 1973), it is possible that Twitch users visit the platform to pick up their emotion, which might be, in general, more depressed than before during the pandemic. Thus, the first research question is proposed as follows:

Is there a significant change in Twitch users' emotion before the outbreak of the COVID-19 pandemic and their emotion after that?

Second, we examine changes in social interaction after the pandemic. The latest research on the pandemic and social media argues that the COVID-19 pandemic is different from most of the prior crisis events (Ashokkumar & Pennebaker, 2021). Crisis events usually make people come together and cope with the crisis as a community (Drury, 2018). Meanwhile, in the current COVID-19 pandemic, being social together is not a form of solidarity enabling people to overcome together but rather something to avoid, a catastrophe represented by the term “social distance” (Ashokkumar & Pennebaker, 2021). Therefore, we propose the following research question to inquire about people's attitude about social interaction during the pandemic:

Is there a significant association between the COVID-19 pandemic and Twitch users' attitude change on social interaction?

To examine those research questions, the current study uses data from the synchronous chatting feature of Twitch, the so-called Internet Relay Chat (IRC). More than 55,000 messages across the chats before and after the pandemic were collected, and they were analyzed with the natural language processing (NLP) software Linguistic Inquiry and Word Count (LIWC). Based on prior literature, it is assumed that we can to some degree gauge users' emotion and attitudes by examining their chat texts (Weisz et al., 2007).

Literature review

Twitch and synchronous chat

While Twitch does not provide exclusively gaming content, a dominant proportion of its streams have been related to video games since it originally diverged from the game section of Justin.tv in 2011 (Leith, 2021). In fact, as of December 2020, gaming content comprises 84% of the total streams on Twitch, although the content categories on Twitch have been diverse recently including multiple non-game categories (e.g., Just Chatting, Music; TwitchTracker, 2021). Because of this characteristic, a number of the early studies on Twitch consider the platform as a venue for video games, also known as Electronic-sport (E-Sport; Kaytoue et al., 2012). They recognize that Twitch provides an unprecedentedly unique media environment: streamers not only play games but also host their streams; users do not just watch the game play but share their thoughts via IRC; a group of users in the same community share experiences from the streaming content, and finally share their community identity (Edge, 2013; Hamilton et al., 2014; Nascimento et al., 2014). Of the studies, Hamilton et al. (2014) define Twitch as a “live mixed medium” using McLuhan's (1994) notion of hot media and cool media. With the concept of definition, which is the extent of being filled with data, McLuhan defines hot media as media with high definition (e.g., radio) and cool media as media with low definition (e.g., television). In accordance with the concepts, Hamilton et al. (2014) describes Twitch as a hybrid media mixed with both hot and cool media, defining Twitch as a platform “conjoining game graphics (high fidelity), live webcam video (medium fidelity), and chat (low fidelity)” (p. 1318).1 Whereas the contemporary media environment makes it difficult to apply McLuhan's media distinction due to the blurry boundaries of the two categories (hot media vs cool media), Hamilton and colleagues' analysis applying them has been often employed as a basic guide for later studies on Twitch.

Of the three components suggested by Hamilton et al. (2014), many studies on Twitch have focused on the chat feature (Diwanji et al., 2020; Oh et al., 2020). It is not only because of the availability of the text data but also because the chat texts are a feature that can distinguish Twitch from other social media platforms. Though a myriad of prior studies on social media have already explored a variety of texts from multiple platforms such as Twitter (Bollen et al., 2011; Wang et al., 2012), Facebook (Kramer, 2012; Moreno et al., 2011), and another video-based platform YouTube (Hajar, 2016), most of them are texts from asynchronous communication on social media. Having said that, Twitch's texts are the results of live conversation which might be closer to face-to-face conversation than other social media texts.

Meanwhile, unfortunately, chat texts from Twitch are not very amenable for meaningful analysis (Ford et al., 2017). When researchers collect chat messages from popular streamers' chat rooms, the inundation of chat texts there is an inevitable challenge, which Hamilton et al. (2014) describe as a “waterfall of text.” Indeed, through our testing data collection from the channel of TommyInnit—who was one of the top-tier streamers on total followers (4.3 million followers as of March 2021)—we found that there could be more than 150 messages newly added within 1 minute in popular chat rooms. Considering Jones et al.’s (2008) finding that users in a chat room can absorb about “30 messages per minute” (p. 330) at best, the amount we found means that it is almost impossible for researchers to observe a coherent conversation in popular Twitch chat rooms, not to speak of the overwhelming quantity of the texts in Twitch streams lasting 3–6 hours on average (Emergence, 2020).

NLP analysis for Twitch chat

In order to overcome the hurdles, a number of researchers have employed computerized text analysis, since it allows them to deal with a large number of texts within a relatively short time. Twitch provides an interface for their IRC, so drawing chat texts from a stream for computational analysis is relatively simple (Twitch, 2021). With that being said, the specific way to analyze the texts in a computational language, namely NLP, can vary across researchers. Nonetheless, it is worth noting that a substantial proportion of the studies has selected the dictionary-based approach (Diwanji et al., 2020; Kobs et al., 2020; Oh et al., 2020). This is somewhat surprising given that the general trend of NLP research has been using more and more complex machine learning and deep learning techniques by supervised learning such as word embedding (which represents words as dense vectors; Baden et al., 2020; Levy & Goldberg, 2014; Li & Yang, 2018) or unsupervised learning such as Latent Dirichlet Allocation (LDA; Maier et al., 2018). The unique tendency can be explained by a couple of reasons based on Twitch's characteristics. First, as aforementioned, there are a large number of speakers in Twitch chats, which makes it difficult for supervised/unsupervised models to discern a certain type of linguistic pattern from the aggregated texts. Second, even if each individual message, not the whole text in the chat, is employed as unit of analysis, it is still difficult because a substantial number of the messages are too short to grasp a pattern, let alone a myriad of emoji-only messages. Accordingly, those reasons give the advantage to simple dictionary-based text analysis, which is more convenient to analyze short Twitch texts word-by-word over more complex NLP techniques that try to grasp linguistic patterns of texts.

Of the different types of dictionary-based NLP programs, lately, multiple studies have used Linguistic Inquiry and Word Count (LIWC) to predict users' emotion (Diwanji et al., 2020; Leith, 2021; Oh et al., 2020). LIWC is developed using social psychology and linguistics, and its fundamental mechanism is operated by word count based on pre-existing dictionaries (Tausczik & Pennebaker, 2010). The program is especially advantageous for researchers to measure emotion as their dictionaries cover a variety of emotion beyond binary sentiment (positive/negative), including anger, anxiety, and sadness to name a few. Oh et al. (2020) explored the cultural difference between English-speaking users and Korean-speaking users employing four variables in LIWC: I-words, we-words, social words, and affection. Recently, Leith (2021) measures verbal immediacy of the chats from the most popular 50 channels from game and nongame categories respectively and displays that there are para-social relationships between Twitch streamers and their users. The current study uses LIWC as an NLP analysis tool as well since it also aims at analyzing Twitch users' emotion using their chat texts.

Twitch and the COVID-19 pandemic

The COVID-19 pandemic is an unprecedented event in human history. Nevertheless, we can glean some clues to people's emotional change during the pandemic from prior literature that addressed society-level crisis events. Cohn et al. (2004) collected, from 1,084 users living in the United States, diaries from an online journaling service which were written from 2 months prior to the September 11 to 2 months after that. Through text analyses using LIWC, the authors find that the September 11 significantly influenced U.S. people's emotion including positivity and psychological distancing. Miller (2015) conducted a content analysis of 2,207 YouTube comments on three tragic events that happened in the United States in 2012: the Sandy Hook Elementary School shootings, the Aurora theater shootings, and Hurricane Sandy. The study displayed the difference in people's grieving emotion between man-made and natural calamities. In general, those studies about crisis events show that the events negatively influence people's emotion, and that can be measured by their online texts.

In line with that, there has been lately emerging research that explores people's emotion during the COVID-19 pandemic collecting texts from social media platforms. Of the studies, a noteworthy one was conducted by Ashokkumar and Pennebaker (2021). They examine how the pandemic affected people's social and psychological states during the first three months after the first U.S. death by the COVID-19 was reported. The authors analyzed 1.8 million Reddit comments in the months preceding and following the outbreak of the COVID-19 pandemic and compared those data to comparable 1.1 million Reddit comments uploaded during the same time period in 2019 using LIWC. The results show that the negative emotions—anxiety, anger, sadness—significantly increased, and conversely, positive emotion significantly decreased after the first death by the virus.

While the current study recognizes Twitch as a distinguishable platform with its live and interactive streaming service (Hamilton et al., 2014), we hypothesize that our findings from Twitch texts related to users' emotional change will be consistent with other social media platforms. It is mainly because texts in IRCs also reflect people's emotions as with other social media platforms (Whitty, 2002). Thus, we propose the following hypotheses for negative emotion and positive emotion, respectively:

Users in Twitch chats will show more negative emotion after the outbreak of the pandemic than before.

Users in Twitch chats will show lower positive emotion after the outbreak of the pandemic than before.

As discussed in the introduction, one of the important differences between the COVID-19 pandemic and other crisis events may be the perspective on social interaction. It had been somewhat commonsensical until the pandemic came that social interaction is an important way to bolster each other when a crisis comes. The literature on crisis events also supported the belief. By observing the Blitz spirit2 in London during the Second World War, Fritz (1996) discovered a “community of sufferers” and suggests that people are integrated together regardless of their background under crisis events; their social interaction plays a therapeutic role for the sufferers. In line with that, Drury (2018) argues that when mass emergencies such as the September 11 attacks and the 2005 London bombings happen, social support is helpful to cope with the disaster, stating that “groupness confers benefits and individualism increases risk” (p. 71). Having said that, one of the salient changes in our life since the outbreak of the COVID-19 pandemic is keeping social distance between people (Centers for Disease Control and Prevention [CDC], 2020). Ashokkumar and Pennebaker's (2021) recent work on the pandemic shows that this difference can create emotional differences from other crisis cases. The authors indeed found that people weakened their relationships with their communities after the outbreak of the pandemic, analyzing texts on Reddit and data from an additional survey, which contrasts to the findings of most prior literature on crisis events.

We believe that Ashokkumar and Pennebaker's finding is important to extend the knowledge on categorization for crisis events and to define the nature of the current pandemic. Thus, the present study seeks to confirm whether the finding is also valid on another social media platform. Therefore, we propose the following hypothesis:

Users in Twitch chats will use fewer social words after the outbreak of the pandemic than before.

Method

The current study analyzes Twitch users' chat texts to see if their emotion and attitude to social interaction changed since the outbreak of the COVID-19 pandemic. To measure the change, two comparable sets of chat texts were collected from streams before the outbreak of the pandemic and streams after that, respectively. Since a number of recent studies analyzing Twitch texts used LIWC and supported its validity (Leith, 2021; Oh et al., 2020), the current study also employs LIWC2015 as an analytical tool. Specific dictionary categories in the program to gauge positive and negative emotions were decided following Ashokkumar and Pennebaker's (2021) study on the pandemic. In addition, the social words category on LIWC was measured to look into the attitude change on social interaction. Overall, interpretation of the results for each category follows the official manual of LIWC2015 (Pennebaker et al., 2015).

Measures

Anger, anxiety, and sadness

Of a total of 93 dictionary categories in LIWC2015, there are multiple categories related to negative emotion. Although a blanket category, negative emotion, broadly covers each of the negative emotion sub-categories—anxiety, anger, and sadness—the present study measured the three variables independently following Ashokkumar and Pennebaker (2021). Actually, it is true that those three emotions are all negative but significantly distinguishable from one another, and we assume that the difference can be helpful to understand the current pandemic.

As aforementioned, the basic mechanism of LIWC is based on word count, and each category in LIWC has a list of dictionary words. A score for each category refers to an averaged number of words in 100 words that belong to the word list of the category. For instance, if an anger score for a given text is 5.00, it means that there are, on average, five anger-related words for every 100 words in the text. Put differently, the scale is a percent probability, whereby we can easily grasp how often words in a certain category appear in the text. There are 230 words in the anger dictionary (e.g., hate, kill), 116 words in the anxiety dictionary (e.g., worried, fearful), and 136 words in the sadness dictionary (e.g., crying, grief). Internal reliability of each category in LIWC2015 was measured by the developers. The corrected alpha scores for anger, anxiety, and sadness are 0.53, 0.73, and 0.70, respectively (Pennebaker et al., 2015).3 While those scores do not reach the conventional threshold for reliability, Pennebaker and colleagues argue that internal reliability for linguistic patterns intrinsically cannot be as high as normal reliability scores from other types of data. Following studies verify the validity of the dictionary categories in LIWC2015 and lend support to the original authors' argument (Leith, 2021; Oh et al., 2020).

Positive emotion

For positive emotion, LIWC has only a single category, positive emotion. Positive emotion is measured by word count of the related words, and the score refers to a percent probability. Its dictionary is composed of 620 words, and the internal reliability score was 0.64 (corrected alpha).

Social words

While the previous variables, both positive emotion and negative emotion, followed the method with which Ashokkumar and Pennebaker (2021) operationalized LIWC's categories to gauge people's emotional change during the pandemic, the operationalization for social interaction in this study is different from what they did. To see social connection, Ashokkumar and Pennebaker measured four social group categories—friends, family, city, and country; however, we consider that those four categories are fit for their text data—comments on city-related communities on Reddit—but not for the current study. Thus, this study decides to use the social words category in LIWC to measure people's attitudes to social interaction. The dictionary has 756 words, including words indicating other people (e.g., they, friends) and other types of words related to social interaction (e.g., talk, Instagram). The internal reliability score of the category was 0.86 (corrected alpha).

Data collection

There were some obstacles to collecting randomized data from Twitch. By Twitch's policies, each channel can display a streamed video only for two months, and the video is automatically deleted from the streamer's list of previous broadcasts after the period. The only way to save a video longer than two months is that the streamer designates the streamed video as a “highlight.” As might be expected, most streamers do not regularly save their streamed videos as their highlight videos. In fact, many of the highlight videos are very special cases for the streamers, far from a random sampling. The time point when we tried to collect our data is over two months after WHO's official declaration of the COVID-19 pandemic on March 11, 2020, which this study recognizes as the point of the outbreak of the pandemic. Thus, this limitation during the data collection made us give up random sampling and consider another alternative for sampling to overcome the influence of possible confounding factors.

Control variables

Though we could not find a single study directly suggesting control variables on users' emotion in Twitch chats, some of the relevant studies gave us helpful hints. First, we found that multiple studies suggest that content type of videos as an important factor to users. In their study, Sjöblom et al. (2017) carried out a survey responded to by 1091 Twitch users and pinpointed that content type is a critical factor affecting Twitch users' motivation over other factors. Recently, Herring and Chae (2021) explore YouTube users' comments during the COVID-19 lockdown, and their results display that the linguistic patterns in users' comments significantly vary by content type. The other possible control variable is channel (streamer) on Twitch. The significant variance of viewers' characteristics across channels has been supported by a number of studies. As aforementioned, many of the early studies on Twitch focus on community formation by channel (Edge, 2013; Hamilton et al., 2014). In line with that, recently, Wulf et al. (2020) and Leith (2021) argue that Twitch users have para-social relationships with the streamers, whereby they were influenced by their persona (i.e., streamer).

In order to ensure if the content type and the streamer are necessary to be controlled in our main study, we conducted two preliminary studies each of which tested one of those. In the first preliminary study, we collected chat texts from the channels of eight Twitch streamers who together played a multiplayer game Among Us at the same time. Then, the scores of the five variables for the main study (anger, anxiety, sadness, positive emotion, social words) were compared by channel using LIWC. In the second preliminary study, we collected chat data from a single stream broadcasted by one streamer in which the streamer provided five different types of content including non-gaming free talk. Again, the scores of the five variables were compared by content type. As a result, the preliminary study about channels showed that channel is significantly associated with all of the five variables (p < .05). The other preliminary study manifested that content type is significantly associated with anger, anxiety, and social words but not with sadness and positive emotion.

Sampling

Based on the findings of the preliminary studies, we decided to methodologically control both channel and content type in the main study. Put differently, the data for the main study were collected from a single streamer's streams playing the same game before and after the COVID-19 pandemic in order to rule out the influence of the two control variables. After searching through a host of streamers on Twitch, the chat data were collected from a channel of an American female streamer, DizzyKitten. She began her streaming in September 2013, and as of October 2021, about 658,000 Twitch users follow her channel (DizzyKitten, 2021). She broadcasts her Twitch channel around five days a week as a professional streamer, and the spectrum of her content is diverse, ranging from ASMR (autonomous sensory meridian response, which means broadcasts inducing tingling responses) to a variety of video games. From her content, a series of streams playing Stardew Valley, which is a simulation role-playing game released in 2016 and supports both single-player and multiplayer modes (see Fig. 1 ), were selected. The gameplay of the game is reviving a dilapidated farm, allowing players to grow crops, raise livestock, socialize with the neighbors, and so on ("Stardew Valley," 2021). Whereas Twitch streamers often play multiple video games in one stream, and the stream is automatically saved on their channels, DizzyKitten created an independent video collection for Stardew Valley on her channel; after she had played the game on her channel, she filtered out a part solely related to the game play from the stream, excluding non-game part and any part related to other games, and then added the edited video into her Stardew Valley collection, numbering the video through the title (e.g., “Stardew Valley (part 16)”).

Fig. 1

Screenshot of DizzyKitten's Playing Stardew Valley. Note. The image was captured from the clip “Stardew Valley (part 3)” on DizzyKitten's channel.

The streamer and the game were selected because her Stardew Valley collection has some distinguishing merits. First, the collection includes all of the streams of her playing the game without any omissions. Second, as of October 2021, 22 videos are in the collection uploaded since late 2019, and they were appropriately distributed to compare before and after the pandemic (see Appendix A for details). She started to play the game from early December 2019, and until WHO declared COVID-19 a pandemic on March 11, 2020, she played the game in 11 streams. Then, she re-started to play the game from late December 2020, and as of the date for the data collection (February 21, 2021), she has played the game in another 11 streams after WHO's pandemic announcement. Lastly, the game is relatively unobtrusive compared to first-person shooter games or other types of action games wherein the users are often overwhelmed by vivid and speedy actions, and the users' chats mostly focus on the game play scene by scene. In DizzyKitten's Stardew Valley streams, the users discussed more diverse things beyond reaction to the game play and often talked to the streamer or to other users about non-game topics (e.g., “I keep saying, after the pandemic, I will just go and sit in a diner for 24 h”) although the game play was still the major topic in the chats.

As with the preliminary studies, the current study takes each chat message as unit of analysis. The data were drawn via Twitch's IRC interface using Python, and all of the messages across the 22 streams were collected without any exception. The data collected from the chats before the pandemic are 30,983 messages during the 11 streams for about 30 hours and 30 minutes, and those after the pandemic are 24,854 messages during the other 11 streams for about 30 hours and 45 minutes. Although it is not the focus of the present study, it may be worth mentioning that the number of messages per minute after the pandemic (13.47 messages) is 20% smaller than that before the pandemic (16.93 messages). Taken together, the final sample size is a total of 55,837 messages from the 22 streams for 61 hours and 15 minutes, which were broadcast from December 2019 to February 2021. Accordingly, an average of 15.19 messages per minute newly appeared in the chatrooms; the amount is considerably less than Jones et al.’s (2008) threshold for the number of messages possible to read in 1 min, which is 30 messages. In other words, it is not plausible that the users could not communicate well with others because of the amount of the messages.

Procedure

After the collection of the chat data, first, emojis in the messages were converted. While emojis in the customized emoji set for DizzyKitten are automatically changed into a certain word (e.g., “dizzyHey”, “dizzyHmm”) in the processing of data collection via Twitch IRC interface, other general emojis (e.g., , ) were not converted automatically. Thus, using a Python package emoji, the remaining emojis in the chat messages were changed into corresponding words (e.g., to “red_heart”, to “thumbsup”). Then, the cleaned messages were analyzed by LIWC referring to previous studies using LIWC for Twitch texts (Leith, 2021; Oh et al., 2020). As with the preliminary studies, the scores of the five categories—anger, anxiety, sadness, positive emotion, and social words—were measured for each message. After that, the average scores for the five variables were calculated for the before-pandemic group and for the after-pandemic group, respectively. Finally, it was tested whether the difference in the average scores is statistically significant for each variable respectively. Since we have a categorical independent variable having two groups (before the pandemic vs after the pandemic) and continuous dependent variables (scores of the five categories), t-tests were employed as an appropriate statistical test. The t-tests were conducted using a statistical program R (version 3.6.0).

In addition, for each dependent variable, we compared our data to the reference data offered by LIWC2015 to understand how the scores of our two text groups are positioned relative to general texts (or speech). To give a sense of the degree to which language varies across settings, the developers of LIWC collect reference texts from six genres—blogs, expressive writing, novels, natural speech, The New York Times, Twitter—and additionally shared means of the six genres as “grand means”; overall, their sampling totaled over 231 million words (Pennebaker et al., 2015). To preclude possible confusion, it is necessary to mention that, whereas comparing values across genres within the same dictionary category (e.g., anger, sadness) can be valid, comparing values across the categories is meaningless since the number of words in each dictionary varies by category.

Results

Negative emotion

For the total of 55,837 messages including both before- and after-the-pandemic messages, the average scores of the negative emotion variables, anger, anxiety, and sadness, were 0.51 (SD = 1.47), 0.08 (SD = 4.34), and 0.22 (SD = 2.71), respectively (see Table 1 for overall comparison). Comparing the before- and after-the-pandemic groups, the average scores after the pandemic for anger (M = 0.60, SD = 4.68), anxiety (M = 0.10, SD = 1.64), and sadness (M = 0.224, SD = 2.78) were increased by 36.0%, 35.7%, and 2.4% respectively relative to the average scores before the pandemic for anger (M = 0.44, SD = 4.04), anxiety (M = 0.07, SD = 1.32), and sadness (M = 0.219, SD = 2.66). Though all of the directions of the change correspond to what H1-1 posits, the statistical significance of the results varies across the variables. Whereas the difference of sadness between the groups before and after the pandemic did not reach the conventional threshold for statical significance, the differences for anger and anxiety were statistically significant (anger: t(49338.68) = −4.20 p < .001; anxiety: t(47118.18) = −2.06 p = .04). Thus, H1-1, which predicts that the scores of the negative emotion variables increase after the pandemic, was partially supported.

Table 1

Overall comparison between before- and after-the-pandemic groups.

	Negative Emotion			Positive Emotion	Social Words∗∗∗
	Anger∗∗∗	Anxiety∗	Sadness
Before	0.44	0.07	0.22	6.23	7.20
After	0.60	0.10	0.22	6.07	6.03
Total	0.51	0.08	0.22	6.16	6.68

∗The p-value of the difference between before and after the pandemic is less than .05.

∗∗∗The p-value of the difference between before and after the pandemic is less than .001.

As shown in Appendix B, the average anger score of our before-the-pandemic data (0.44) is lower than that of any other reference genres offered by LIWC2015 (Twitter: 0.75, blogs: 0.68, grand mean: 0.54, novels: 0.51, expressive writing: 0.49, the NY Times: 0.47), except for natural speech (0.36). Meanwhile, the average anger score of our after-the-pandemic data (0.60) is higher than all of the reference genres other than Twitter and blogs. When it comes to anxiety (see Appendix B), both of the average scores (before the pandemic: 0.07, after the pandemic: 0.10) were lower than any other genres (expressive writing: 0.50, novels: 0.44, grand mean: 0.31, blogs: 0.27, the NY Times: 0.25, Twitter: 0.24, natural speech: 0.14). In consistent with that, both of the average scores of sadness (before the pandemic: 0.219, after the pandemic: 0.224) are lower than all of the reference genres (novels: 0.55, expressive writing: 0.50, blogs: 0.44, Twitter: 0.43, grand mean: 0.41, the NY Times: 0.29, natural speech: 0.23; see Appendix B).

Positive emotion

The average score of positive emotion across all of the 55,837 messages was 6.16 (SD = 16.93). Comparing the two groups, the average score of positive emotion after the pandemic (M = 6.07, SD = 16.62) was decreased by about 3% from that before the pandemic (M = 6.23, SD = 17.18). Although the direction of the change is consistent with H1-2, the result of the t-test showed that the change is not statically significant. Therefore, H1-2 predicting that the score of positive emotion decreases after the pandemic was not supported.

Additionally, the result values were compared to LIWC's means of reference genres (see Appendix C), as done for negative emotion. In stark contrast to anxiety and sadness, both of the average scores of positive emotion (before the pandemic: 6.23, after the pandemic: 6.07) were higher than any other reference genres (Twitter: 5.48, natural speech: 5.31, grand mean: 3.67, blogs: 3.66, novels: 2.67, expressive writing: 2.57, the NY Times: 2.32).

Social words

The average score of social words across both before- and after-the-pandemic groups was 6.68 (SD = 13.91). The average score after the pandemic (M = 6.03, SD = 12.95) was decreased by 16% from that before the pandemic (M = 7.20, SD = 14.60), which is congruent with what H2 predicts. The difference between them is statically corroborated by a t-test, t(55275.69) = 10.04, p < .001. Thus, it supported H2 that predicts decrease of social words after the pandemic.

Discussion

This study was undertaken to examine how the COVID-19 pandemic associates with Twitch users' emotion using natural language processing. Overall, the suggested hypotheses were partially supported. Regarding H1-1, which predicts that Twitch users' negative emotion increases after the outbreak of the pandemic, the results varied across the three negative emotions. Both anger and anxiety were significantly increased whereas the increase of sadness was not statistically significant. The result for H1-2, which predicts the decrease of Twitch users' positive emotion after the outbreak of the pandemic, did not reach the conventional threshold for statistical significance while the direction of the change corresponds to the prediction. Last, the result for H2 expecting the decrease of social words was consistent with the hypothesis.

Positive and negative emotions

It needs to be explored why the results of the three negative emotions were not consistent. There are some possible rationales, and one of them could be explained by the intrinsic feature of sadness distinguishable from anger and anxiety. A thread of supporting discussion can be found from the psychology literature on motivation and emotion. Based on Izard's (1972) argument on the categorization of human emotions, Schwartz and Weinberger (1980) explore relations among six different emotions—happiness, sadness, anger, fear, depression, and anxiety—and test if “depression situations elicited a more complex pattern of emotions than did sadness situations” (p. 182). In the study, the participants significantly distinguish between depression situations and sadness situations, and the result corroborates that depression situations entail more anger, fear, and anxiety compared to sadness situations. In a similar vein, it is possible that the pandemic is perceived as closer to a depression situation than a sadness situation although the separation cannot be very clear in the pandemic case.

The difference among the three negative emotions can be also distinguished by commonsensical thinking. Anxiety is an anticipatory reaction to an unhappy event that will possibly happen in the future (Izard, 1972), and anger is a reaction elicited when people feel that they have been offended or injured (Lazarus, 1991). By comparison with them, sadness is often aroused as an ex post facto emotion after a tragic event happened. Applying this division to the pandemic situation, anger could be aroused by the inconvenience in the wake of lockdown and stay-at-home order, and anxiety could be elicited by the fear of the possible virus transmission or other pandemic-oriented risks. While the extent varies by individual, many people might not be free from those two emotions after the outbreak of the pandemic. Having said that, sadness is often raised by a personal sorrow such as losing a family member to the COVID-19. Assuming that most Twitch users were less likely to be under personal misery but more likely to feel the inconvenience or the fear of the pandemic, the distinguished results are understandable.

When it comes to positive emotion, the direction of the change coincides with H1-2 as the score decreased after the declaration of the pandemic, but the change is not statistically significant. To look into the result precisely, it is necessary to first understand that both before- and after-pandemic scores of positive emotion are higher than the average score of any type of writing that LIWC suggested as references. The average value of positive emotion across both the before- and after-the-pandemic data is 6.16, and it is higher than the comparable average values from any reference genres including Twitter (5.48), blogs (3.66), and the grand mean (3.67). It indicates that, regardless of whether it is before or after the pandemic, Twitch's chat is relatively positive. Furthermore, it stands in stark contrast to the fact that the average values of each of the three negative emotions (anger, anxiety, sadness) across the before- and after-the-pandemic groups are all lower than the corresponding grand mean of LIWC2015.

The relatively positive aspect of Twitch chats may be related to why positive emotion was not significantly changed after the declaration of the pandemic. Though two of the negative emotions significantly changed, and positive emotion did not, the comparison of their values to the references denote that positivity is a more salient sentiment than negativity on Twitch. In fact, it parallels the finding of prior literature. Applying the uses and gratifications theory as a theoretical framework, Sjöblom and Hamari (2017) test if Twitch users have different motivations to visit the platform and find that the tension release need is the strongest predictor of how many hours users watched Twitch streams and how many streamers' channels the participants watched. Following that, it is commonsensical that users coming to release their tension display more positive emotion over negative emotion. Therefore, compared to negative emotion, positive emotion in Twitch chats could be more rigid and less swayed by an external crisis situation because it is more related to the intrinsic function of the platform. Although it is probable that Twitch users' usual positive emotion decreased after the outbreak of the pandemic, visiting Twitch itself could offset the decrease and maintain a certain level of positiveness.

Across the results of both positive and negative emotion, it is notable that the directions of the changes for all of the four emotion variables were congruent with what the hypotheses predict, regardless of their statistical significance (again, see Table 1). It means that our results corresponded to the findings from the prior literature on crisis events which argues that people become more negative and less positive in general when going through crisis events (Cohn et al., 2004; Miller, 2015). While the notion could sound somewhat obvious, the meaning can be more profound when it comes to the unique nature of Twitch. The previous studies on crisis events collected the text data from an online diary platform (Cohn et al., 2004), subreddits about cities on Reddit (Ashokkumar & Pennebaker, 2021), or other types of online platforms wherein users can share their thoughts around their daily life which might be influenced by the crisis events. With that being said, Twitch chat stems from an essentially distinct media environment from those platforms. At the audio-visually hectic venue, users seem more likely to discuss the issue on the stream at the moment. Prima facie, the game-oriented, fandom-based, and messy chat seems to have nothing to do with the society-level crisis. Nonetheless, our results indicate that the pandemic was significantly associated with the underlying emotion in Twitch users' chats. Particularly given that the game Stardew Valley has an uplifting mood with cheerful music, and the streamer was bright in general, the association with the pandemic is somewhat surprising.

Future research could extend this finding, which is drawn from the snapshot results based on static texts, to a multidimensional association by connecting users' emotion with their motivation to come to Twitch. While the current study found the significant emotional change after the outbreak of the pandemic, it did not address how the emotion is associated with their motivation. It is probable that, after the outbreak of the pandemic, users are more likely to visit Twitch in order to refresh their negative emotion due to the lockdown. In fact, that assumption corresponds to the core proposition of uses and gratification theory that people selectively use media and gratify their needs (Katz et al., 1973). Furthermore, the motivational research could elucidate how media dependency is affected by crisis events, which might make people more emotionally vulnerable than usual (Ball-Rokeach & DeFleur, 1976).

In fact, the findings of this study are not in line with the general perspective of the research on Twitch. As aforementioned, in the body of literature, researchers focus on the point that Twitch has different communities by channel or by game, and the members in the user (follower) group are likely to share unique culture, attitude, and other patterns (Hamilton et al., 2014; Hilvert-Bruce et al., 2018). By comparison, our findings highlight that Twitch users are not just “video game nerds” who are maniacally absorbed in the game play or the game community, but rather they share their daily life emotion even during their game-spectating time. It indicates that Twitch is not only a unique platform that provides the live mixed media but also a venue where researchers can find people's general emotion regarding a society- or nation-level event, as they do so on Facebook (Paton & Irons, 2016) or on Twitter (Murthy & Longwell, 2013). Furthermore, the evidence from this study suggests that future research on crisis events could extend its research area to more diverse platforms and communities that seem not directly relevant to the events.

Being social during the pandemic

Our hypothesis about attitude to social interaction is built on Ashokkumar and Pennebaker's (2021) recent research on the COVID-19 pandemic and predicts decrease of social words after the outbreak of the pandemic. The result corresponds to the hypothesis; hence, it lends support to Ashokkumar and Pennebaker's finding that people felt less connected to friends and their community after the pandemic started. Nonetheless, they acknowledged in the paper that their finding is contrasting to the popular belief that a collective crisis increases solidarity of the community and also the findings from prior literature on crisis events demonstrating that the popular belief holds (Drury, 2018; Drury et al., 2009; Phan & Airoldi, 2015; Zaki, 2020). Ashokkumar and Pennebaker (2021) surmised that it might be because of the unique situation of the COVID-19 pandemic, stating “[l]ack of trust in the government and increasing polarization during the crisis” (p. 22). We believe that the same reasoning is also valid to interpret the results of the current study particularly given the fact that the U.S. society went through serious political and social events during the pandemic, which brought unprecedentedly extreme repercussions, including the killing of George Floyd by police in May 2020 and the storming of the U.S. Capitol in January 2021.

Along the lines of the discussion on the uniqueness of the COVID-19 pandemic on being social, the current study suggests another possible rationale for the phenomenon: Restraints on social interaction in person during the pandemic affect people's thoughts and attitude to overall social interaction including online interaction. Though the governmental policies to cope with the virus restricted physical interaction between people, it is possible that the general mood of the society at the time, which treated being social as a temporary taboo or a threat to the safety of a community, could influence people's attitude to overall social interaction regardless of whether it is in-person interaction or not. Provided that people who try to be social during the pandemic were sometimes considered even unthoughtful, the decrease of social words in the Twitch chats seems somewhat natural. We expect that future research could shed light on how restraints on physical interaction between people affect their attitude to online interaction.

Limitation

As every study does, the present study has notable limitations. Above all, the limited sample size is a point that future studies could improve. The study focused on one streamer's streams before and after the pandemic. That is to say, the findings of this study do not show a general trend on the platform at all. Although more than 55,000 messages over 61 hours in this study cannot be underrated, it is recommended that future studies collect data from multiple channels across different content types. It will be possible by collecting data at regular intervals—less than 2 months to avoid the disappearance of the videos. Although the confounding factors—streamer and game—that this study found might be still hurdles to future studies, the researchers could control them in a larger sample by statistical control techniques such as multi-level analysis.

Another notable limitation in the study is its way to handle emojis. As might be expected, our text data have a host of emojis since a unique emoji set by channel is a feature that many Twitch users enjoy. While the present study interpreted the emojis following the original label of each emoji (e.g., dizzyComfy, dizzyHey), it is probable that the meanings of the emojis inside the channel community are different from the labels, as with other Twitch jargon (Ford et al., 2017). Thus, it could be better that future studies first investigate the practical meaning of each emoji in the streamer's emoji set and apply them in their analysis.

Last but not least, it is true that NLP programs including LIWC still have limitations as a research tool for social science. As widely discussed in multiple disciplines, the current NLP techniques cannot interpret some types of pragmatic practices such as sarcasm or twisted humor (Baden et al., 2020). Likewise, in the present study, there could be multiple messages that their convoluted expressions were not correctly interpreted by LIWC. That is to say, though LIWC's dictionary-based analysis enabled us to analyze more than 55,000 messages in a relatively short time, on the other side of the coin, the risk of misinterpretation was unavoidable for now. We hope that future studies can analyze text data from diverse social media platforms with more advanced NLP techniques covering the pragmatic practices.

Credit author statement

Seung Woo Chae performed conceptualization, methodology, formal analysis, investigation, resources, writing - original draft, writing - review & editing, and visualization. Sung Hyun Lee performed conceptualization, writing - original draft, and writing - review & editing.

Declarations of competing interest

No potential conflict of interest was reported by the authors.

Period	Date	Video Title	Video length (Hour:Min:Sec)	Number of chat messages
Before the pandemic	Dec 4, 2019	Stardew Valley (part 1)	02:40:17	2633
	Dec 6, 2019	Stardew Valley (part 2)	02:48:06	3132
	Dec 25, 2019	Stardew Valley (part 3)	02:35:51	4195
	Dec 27, 2019	Stardew Valley (part 4)	03:27:55	4081
	Jan 15, 2020	Stardew Valley (part 5)	03:25:53	2967
	Jan 17, 2020	Stardew Valley (part 6)	01:23:38	1172
	Jan 20, 2020	Stardew Valley (part 7)	03:39:45	3051
	Jan 27, 2020	Stardew Valley (part 8)	01:52:04	2058
	Feb 2, 2020	Stardew Valley (part 9)	02:45:25	2295
	Feb 4, 2020	Stardew Valley (part 10)	02:56:41	2841
	Mar 4, 2020	Stardew Valley (part 11)	02:54:17	2558
After the pandemic	Dec 26, 2020	Stardew Valley (part 12)	03:10:53	4051
	Dec 28, 2020	Stardew Valley (part 13)	02:49:47	2672
	Dec 30, 2020	Stardew Valley (part 14)	04:20:42	3790
	Jan 4, 2021	Stardew Valley (part 15)	04:40:12	3864
	Jan 6, 2021	Stardew Valley (part 16)	01:50:23	1226
	Jan 11, 2021	Stardew Valley (part 17)	02:28:07	2098
	Jan 27, 2021	Stardew Valley (part 18)	03:46:01	2568
	Jan 28, 2021	Stardew Valley (part 19)	01:12:38	745
	Jan 30, 2021	Stardew Valley (part 20)	01:24:46	903
	Jan 31, 2021	Stardew Valley (part 21)	00:56:28	796
	Feb 3, 2021	Stardew Valley (part 22)	04:05:12	2141

Note. All information above were collected from the Twitch channel, DizzyKitten.