Analyzing Public Opinion and Misinformation in a COVID-19 Telegram Group Chat.

We analyze a Singapore-based COVID-19 Telegram group with more than 10000 participants. First, we study the group's opinion over time, focusing on five dimensions: participation, sentiment, negative emotions, topics, and message types. We find that participation peaked when the Ministry of Health raised the disease alert level, but this engagement was not sustained. Second, we investigate the prevalence of, and reactions to, authority-identified misinformation in the group. We find that authority-identified misinformation is rare, and that participants affirm, deny, and question misinformation. Third, we explore searching for user skepticism as one strategy for identifying misinformation, finding misinformation not previously identified by authorities.

The novel coronavirus pandemic (COVID-19) is an ongoing global health event. In this rapidly unfolding crisis, many people turn to social media for real-time information and support that cannot be found elsewhere. Before the pandemic, more than 60% of Singaporeans were consuming news via social media. This figure is expected to rise as more people stay at home.

At the same time, health officials and academics have warned that misinformation about the pandemic presents a risk to public health and public action. In Singapore, clarifications about misinformation are displayed prominently on the official COVID-19 updates website. The Government has also used the Protection from Online Falsehoods and Manipulation Act (POFMA) to correct claims about COVID-19.

We study a Singapore-based Telegram group chat that was created to discuss COVID-19. We focus on the first six weeks of the group’s existence, which represents the “first wave” of the pandemic in Singapore. The number of confirmed cases grew from 4 to 153, mostly imported from China. For two of those weeks, Singapore had the most number of confirmed cases in the world outside China. During this period, the Ministry of Health raised the Disease Outbreak Response System Condition (DORSCON) level from yellow to orange.

The rest of the article proceeds as follows. In the following two sections, we review the relevant literature and describe our data collection process. Then, in the next section, we discuss RQ1: How does group opinion change over time? We find that group participation peaked when the government raised the disease alert level from yellow to orange. The section titled “MISINFORMATION” focuses on misinformation. Specifically, we ask: RQ2a: How prevalent is fact-checked misinformation in the group? and RQ2b: How do group users react to fact-checked misinformation? We find that fact-checked misinformation is rare, and that users affirm, question, and deny misinformation. Additionally, we explore three topics related to RQ2b (“reactions to misinformation”). First, searching for user skepticism may be one strategy for identifying misinformation; second, users demonstrate cynicism in addition to skepticism; and third, users use the term “POFMA” in new ways.

RELATED WORK

Group Chats

Researchers have analyzed the general patterns of interaction in group chats. Others have studied the use of group chats for specific purposes, for example, teacher–student interaction and medical team communication. Previous papers that examine misinformation in group chats mostly do so in the context of political elections.

Social Media and Pandemics

Public health authorities use social media and other information and communication technologies for diagnostic efforts, coordination, and risk communication. Other studies focus on the public instead of health authorities. Chew and Eysenbach studies the types and source of content shared during the 2001 H1N1 outbreak; Strekalova analyzes engagement with posts from the Centers for Disease Control and Prevention Facebook channel during the Ebola outbreak.

Reactions to Misinformation

Media consumption is a complex phenomenon influenced by media literacy, need for cognition, political orientation, prior beliefs, and more. People are not passive consumers; they seek to affirm, deny, and/or question (mis)information on social media. Exposure to misinformation may cause people to become skeptical or cynical about social media.

Contribution

In the study of social media, most papers have focused on public platforms like Facebook and Twitter, rather than chat platforms like Telegram. Studies that do analyze group chats do not focus on pandemics, and studies that focus on pandemics do not analyze group chats. As far as we know, this article is one of the first to analyze a group chat during a pandemic.

We also contribute to the literature on reactions to misinformation, in particular the ability of online communities to challenge and question misinformation. We suggest searching for user skepticism as one approach to finding misinformation.

DATA COLLECTION FROM TELEGRAM

Telegram

Telegram is an instant messaging service that facilitates groups of up to 200000 members and positions itself as a platform that protects user privacy and free expression. The combination of large group sizes, partial visibility, and anonymity plausibly facilitates the spread of misinformation.

Data Collection

Several Singapore-based public Telegram groups emerged after Singapore’s first confirmed case of COVID-19 on January 23, 2020. We found the groups by searching “Singapore Coronavirus Telegram” on Google and Telegram. Some groups were: SG Fight COVID-19 (http://t.me/sgVirus), Wuhan COVID-19 Commentary (http://t.me/WuhanCOVID), and SG Fight Coronavirus (http://t.me/sgFight).

In this article, we focus on SG Fight COVID-19 because it has the most members and discussion. The other groups are characterized by one-to-many news broadcasts. From January 19 to March 8, 2020, we retrieved messages with the Python Telethon API. In total, we collected 48050 messages from 10765 users. There were 10765 system-generated messages, leaving us with 37285 mixed-media messages (26153 text, 1928 images, 276 videos, 36 audios, 8830 links, and 62 files).

Data Limitation

Telegram allows deletion of messages for up to 48 h. Some messages might have been deleted prior to the time of collection. While reliable demographic data about Singaporean Telegram users is not available, our personal experience is that Telegram is mostly used by people below 65 years old. Thus, participants in our group chat may be more digitally literate compared to other chat platforms.

RQ1: HOW DOES GROUP OPINION CHANGE OVER TIME?

Participation

Active Participants: For each week, we look at the number of users who sent at least one message, and the total number of messages. We exclude bots that forward news. User participation peaked in week 2 (see Figure 1). The most popular timing of messages is between 12–1 p.m. and 8–10 p.m.

Figure 1.

Active participation over time.

Lifespan of Participants: We take the ten most active participants for each week and search their activity in other weeks. Of the 50 most active participants, 10 were active for one week and 20 were active for two weeks.

Discussion: User participation peaked in week 2. During this week, the disease emergency level raised from yellow to orange. Prior to the official announcement, a press release was leaked and messages discussing the announcement were circulating on the group chat since 10.30 a.m. We note that the spike in group activity coincides with a government announcement. This illustrates the importance of unified and coherent public health communication. The incident demonstrates how misinformation and rumors can be reclassified as new information emerges. Even information that is eventually classified as true can cause alarm, especially if it is shared in an untimely and haphazard manner. Users voiced their request for better public communication (Feb 7: “We just needed a legit mouthpiece to broadcast the news”).

The decrease in group activity over time and short life span of participation suggests that the group did not meet users’ needs for information. Users may have stopped relying on the group and turned to other sources. During our period of study, the government started using the official WhatsApp channel (https://www.go.gov.sg/Whatsapp) to broadcast pandemic updates. Its subscription grew from 7000 to 900000 between January and April 2020 (https://www.mci.gov.sg/pressroom/news-and-stories/pressroom/2020/4/gov-sg-launches-new-channels-to-keep-the-public-informed-about-covid-19). We postulate that activity rises between 12–1 p.m. and 8–10 p.m. because people use their devices at mealtimes. Furthermore, the Ministry of Health usually releases daily updates at 8 p.m., triggering a flurry of discussion.

Sentiment

We perform phrase-level analysis using the MPQA lexicon (https://mpqa.cs.pitt.edu/lexicons/subj_lexicon/) before combining the results to obtain overall sentence-level sentiment. This method was adopted because Telegram texts are short and conversational.

We observe more negative sentiment than positive sentiment (see Figure 2). From Week 2 to Week 3, positive and especially negative sentiment increased. The rise in negative sentiment corresponds with the DORSCON orange weekend, demonstrating the impact of public health messaging on group opinion.

Figure 2.

Sentiment dimension (percentage) over time.

Psychological Dimensions

To measure psychological dimensions, we use the 2015 version of Linguistic Inquiry and Word Count (LIWC) https://liwc.wpengine.com/). We focus on negative emotions (see Figure 3). Anxiety fell over time, whereas sadness and anger increased over time. We speculate that group chat members were becoming more certain that the pandemic is a serious event.

Figure 3.

Negative emotions over time.

Message Types

Topic Clustering: To identify top word clusters each week, we use Latent Dirichlet Allocation. Clusters are chosen based on coherence scores and manual assessment. Participants generally discuss the essential regional and world events.

Two topics consistently appear in all weeks: cases and masks. The number of cases is a focal point in a pandemic (Feb 10: “[...]the ship has 60 new cases[...]”; Feb 13: “A new case in NUS![...]”). Throughout this study, the word “cases” is almost always accompanied by “confirmed.” In week 6, we observe for the first time the keyword “true cases.” Public health experts have warned that confirmed cases may not be the same as true cases, especially in countries that are under-testing. The group chat reflects this distinction.

The second popular topic is masks. In the first two weeks, participants discourage wearing masks (e.g., Feb 9: “Not sick don’t wear mask.”). In later weeks, the keyword “masks” is no longer observed with “don’t,” suggesting that participants are starting to support mask-wearing (27 Feb: “[...]Wear a mask to protect ourselves”). In parallel, we observe that participants encourage others to stay home (8 Mar: “[...]Adults try to work from home.”) Discussion of socially responsible behaviors in later weeks indicates that messaging from public health authorities is working.

Text Clustering: To understand the structure of discussion, we perform text clustering using the LIWC feature set to identify the message types. We reduce our LIWC feature set into a two-dimensional space using singular value decomposition with k=20 by the use of the elbow rule heuristic, before clustering the points using k-nearest-neighbors. As shown in Figure 4, users participate in the following five main clusters.

Figure 4.

Message types over time.

News reposts: “#asiaone #singapore 3 new coronavirus cases [...]”

Short netspeak: “ah, ok. Agree!”

General discourse: “so much agree. Is incompatible to profit with ncov”

Questions: “can the new virus spread through aerosol transmission?” // “who declare emergency aldy??”

Medical Conversations: “if you really sick go see doc” // “this virus have r0=4 [...]”

Over the weeks, message composition did not vary significantly. Medical conversation was more predominant in weeks 5 and 6, when there was a significant increase in the number of cases in Singapore and in Europe.

MISINFORMATION

RQ2a: How Prevalent Is Fact-Checked Misinformation in the Group?

We refer to the list of corrections about COVID-19 provided by the Singapore Government (https://www.gov.sg/factually, https://www.moh.gov.sg/covid-19/clarifications) and the Singaporean fact-checking agency Black Dot Research (https://blackdotresearch.sg/covid-watch/). Between January 24 and March 8, 2020, the two organizations listed 27 corrections (see Table 2). We adopt this approach of relying on lists from fact-checking parties from.

TABLE 2.

Misinformation identified by fact-checking organizations, and reactions.

Description	Source			Reaction
	Government Correction	BlackDot Research	Telegram Group	Affirm	Deny	Question	Unrelated
Foreign domestic worker’s death	50	50
Visit MOH office	50
States Times Review Facebook Page	50		50	501
CNA Asia graphics	50		50	501	501	501	501
Case at Lucky Plaza	50		50	501	505	502	502
CNA Tweet on school closure	50	50	50			501
Death in Singapore on Feb 7	50		50		502	501	502
List of places to avoid on Feb 1	50		50	503	5010	502	509
Validity of maskgowhere.sg site	50
Singapore ran out of face masks	50
5 Singaporeans contracted the virus locally on Jan 30	50	50
Woodlands mrt closed for disinfection	50		50		503	502	501
Death of 65yo man	50
Individual died of virus	50
100 travelers from Wuhan denied entry	50	50
Not to visit hospitals	50	50
Suspected case at EastPoint Mall	50	50	50		501	502	502
Garlic Cure		50	50	501	501		504
Mice vaccine		50
NTUC Fairprice selling fresh bat meat		50
Doctors in Thailand find cure		50
ncov can be spread through poop		50	50	501	502		509
Pets can be infected		50	50	501	501	502	506
Chinese prostitute asked to name her clients		50
Tankers not allowed to dock in Singapore		50
4 doctors issued a health advisory		50	50	501			502
Quarantined order for returning Chinese citizens		50	50	501	501

To search for messages discussing fact-checked misinformation, we focus on the five days surrounding each piece of misinformation. We filter for keywords, then manually screen the results. For example, to identify messages “challenging the validity of the maskgowhere.gov.sg site,” we perform automatic keyword filtering for messages containing “maskgowhere,” which returns six results. Since all six messages discuss the site instead of challenging its validity, our final result is 0.

Fact-checked misinformation is rare on the group chat. In total, 13 (out of 27) pieces of authority-identified misinformation are discussed in the group chat (see Table 2). These pieces of misinformation are contained in 84 messages, representing 0.2% of all messages.

RQ2b: How Do Users React to Fact-Checked Misinformation?

We categorize messages responding to fact-checking misinformation into four categories: affirms (propagate misinformation), denies (refute the misinformation), questions (asks about the misinformation), and unrelated.

Out of the 84 messages responding to misinformation, 11% of messages affirm the information, whereas 45% question or deny it. Stories that elicit mixed responses are those that have a direct impact on personal lives: confirmed case at Lucky Plaza and list of places to avoid. Many messages make a broader warning about misinformation in general (Feb 1 - “Guys, stop all the BS fake news and fear mongering[...]”).

Users are aware that misinformation exists. They seek to deny or question specific pieces of misinformation, and warn others about misinformation in general.

We report the following three observations from users’ responses to fact-checked misinformation.

Observation 1: Detecting misinformation by searching for skepticism: So far, we have depended on lists of misinformation provided by fact-checkers. However, fact-checkers have limited resources and cannot check all content. The total volume of misinformation exceeds misinformation that has been identified by fact-checkers.

Since users respond to fact-check misinformation with skepticism, we attempt to detect misinformation by searching for skepticism.

Because Telegram messages and Twitter comments are both conversational in nature, we use a Twitter Disbelief dataset and method described by Jiang et al. The dataset consists of manually annotated Twitter comments that reflect belief or disbelief in reaction to misinformation. We annotate each message with LIWC-values, and enhance the annotations with ComLex values, a contextual lexicon built with social media comments to misinformation (https://shanjiang.me/resources/#misinformation). Following Jiang, we construct a feature vector of 92-LIWC+300-ComLex values for each message. Then, we construct a logistic regression classifier in Python scikit-learn using the LIWC+ComLex annotated Twitter messages from the Disbelief dataset as the training data. Following which, we apply the classifier on LIWC+ComLex annotated Telegram messages, yielding 14730 messages expressing disbelief, which is 40% of all messages. We filter for messages that have at least five words or contain media.

By manually looking through the messages, we identify 72 pieces of misinformation. Many of these pieces of misinformation have not been publicly flagged by authorities, for example, a claim that the government censored #wuhanvirus on Instagram, and that a person collapsed at Paya Lebar. In general, the misinformation found falls into five categories, which we present in Table 1.

TABLE 1

Misinformation found by searching for user skepticism.

Category	Misinformation
Natural cures	Natural medical care from eucalyptus oil Traditional Chinese Medicine cures COVID-1
Case counts	Seven new cases at Lorong Chuan “New case at Changi” “4000 people locked in Grace Assembly of God church to contain the spread” A close contact of a confirmed case was not quarantined
Government crisis responses	“Government lacks masks and canned food” “[...]government is restricting sales of masks for richer people to buy first” Singaporeans who transit in Hong Kong have to be quarantined Singapore supplies masks to China
Conspiracy theories	“Dr. Boyle from Harvard law school took on an offensive biological warfare agent” “Virus was leaked by Chinese communists [..]”
Facts about the virus	“smokers have higher chance of getting virus” “[...]seizure and die immediately”

Observation 2: Skepticism and cynicism: As previously mentioned, users challenge and question misinformation, which constitutes skepticism. Skepticism is disbelief in social media but not a rejection of it. It involves doubt and careful clarification, and is thought to be healthy for the society.

However, healthy skepticism can turn into unhealthy cynicism. Cynicism is distrust and rejection of the media. It involves disengagement and disillusionment. In our group, we observe messages indicating cynicism, for example, the following.

Feb 07: “what’s the exact symptoms to worry/look out actually (as there’s quite a lot of fake news spreading, etc.)”

Jan 30: “At this point idk whats fake news anymore.”

Further research can investigate the distinction between skepticism and cynicism and clarify “right” level of skepticism that social media users should have.

Observation 3: POFMA You! The POFMA is a Singapore law passed in 2019 aimed at correcting misinformation. In total, there are 107 uses of the word “POFMA,” suggesting users are aware of the law and its consequences. In 63 instances, participants use “POFMA” in new ways. Originally, the name of a law, the term is being used as a verb (“POFMA me”), an entity with agency (“POFMA busy”), and a byword for misinformation (“that doesn’t look like censoring but pofma”).

CONCLUSION

We analyzed a Telegram group chat about COVID-19 in Singapore. The group was most active from February 3 to 9. This week is notable because a press release was leaked prior to a raised health alert. This demonstrates the importance of coherent and unified public health communication.

Authority-identified misinformation is rare on the group chat. Messages that discuss these pieces of misinformation tend to be skeptical. Our findings support the idea that users are not passive consumers of (mis)information; they seek to question and/or challenge online content. Thus, searching for user skepticism may be one strategy for detecting misinformation online.

This article is a preliminary attempt to investigate an ongoing and dynamic crisis. It contributes to the small but growing literature about misinformation in group chats. Although interest in misinformation about the pandemic may fall as the pandemic subsides, observations about reactions to misinformation and how misinformation might be detected may be of continuing interest to readers. A single Telegram group is not representative of the rest of the population, so it is hard to tell if the findings are generalizable. Further research can include more group chats and comparison with public platforms.