A two-step rumor detection model based on the supernetwork theory about Weibo.

Based on the supernetwork theory, a two-step rumor detection model was proposed. The first step was the classification of users on the basis of user-based features. In the second step, non-user-based features, including psychology-based features, content-based features, and parts of supernetwork-based features, were used to detect rumors posted by different types of users. Four machine learning methods, namely, Naive Bayes, Neural Network, Support Vector Machine, and Logistic Regression, were applied to train the classifier. Four real cases and several assessment metrics were employed to verify the effectiveness of the proposed model. Performance of the model regarding early rumor detection was also evaluated by separating the datasets according to the posting time of posts. Results showed that this model exhibited better performance in rumor detection compared to five benchmark models, mainly owing to the application of the supernetwork theory and the two-step mechanism.

Introduction

Since the rapid development of computer technology in recent years, several famous social media platforms, such as Twitter and Weibo, enable individuals to post their viewpoints on global events. Despite the conveniences brought by these platforms, they also pose some serious problems that should be addressed, e.g., how to verify the credibility of certain viral information [38]. According to Bondielli and Marcelloni [2], rumors refer to the most widely studied false information on the Internet, which could be defined as the information that was not confirmed by official sources yet in social media platforms. The spreading of rumors may cause serious issues and disrupt public order [2]. For example, on April 23, 2013, a rumor about two explosions injuring Barack Obama caused a serious crash in the US stock market. Therefore, there is an urgent need for a method that can classify rumors in an accurate and timely manner. In general, there are three relevant aspects in this regard: spammer detection [25, 40], rumor source detection [28, 44], and rumor detection [1, 5, 21, 34]. Related existing studies offered some valuable findings. For instance, Luo et al. [21] proposed a novel post-based augmentation representation for rumor detection, which achieved a high accuracy rate of 80.09% in empirical test. Alzanin and Azmi [1] put forward a semi-supervised system for detecting rumors in Arabic language social networks, which achieved an F1 score of 80% in the experiment. Existing rumor detection models are frequently established based on a wide range of features corresponding to account information, content of posts, and some other aspects [7, 16, 30, 39]. However, there is still no effective methods or frameworks to connect these features. In addition, most studies suggested that only the users with spammer features will post rumors. However, the fact is that some normal users who believe rumors may also contribute to the spreading of rumors to some extent. Ignoring this important point would decrease the effectiveness of the model, to a large extent.

The supernetwork theory was proposed by Sheffi in 1985 [10]. It is multi-layered, multi-leveled, multi-dimensional, and multi-attributed, and it has various degrees of congestion and coordination [4, 15]. As the supernetwork can be applied to reflect the interaction between different networks, a more comprehensive description of a whole system could be achieved [15]. With respect to rumor detection area, supernetwork theory seems to be an effective tool for reasonably classifying the features contained in rumors or rumor posters. In recent years, we firstly used the supernetwork theory to analyze the issues related to online public opinions [4, 19, 24, 37], elucidating the understanding of online public opinion analysis.

In the present paper, we proposed a two-step rumor detection model based on the supernetwork theory and the findings of previous studies, in which a three-layer supernetwork considering user, psychology, and viewpoint was established. Several features that can reflect the interaction within one subnetwork and among different subnetworks were presented. In particular, for the two-step mechanism in detection process, we first separated the selected features into two aspects: user-based and non-user-based features. By using user-based features, we classified the users into two groups. Then, the non-user-based features were used to detect rumors for each group. In addition, several assessment metrics and commonly applied machine learning methodologies were employed to verify the effectiveness of our proposed model.

The main contributions of this paper are as follows

(1) This paper firstly employed the supernetwork theory for the rumor detection in an overall and systematic way, which provides a way of reclassifying the features of rumors and considering their relationships.

(2) A two-step mechanism that is useful for detection rumors posted by normal users was proposed. By employing this mechanism, the detection performance of the model could be improved, to a large extent.

(3) A set of new features focusing on the relationships among different features of rumors were presented in this paper in order to comprehensively describe the online posts.

(4) Results show that our proposed model could achieve better performance compared to five benchmark models in rumor detection.

The remainder of the present paper is organized as follows: In Sect. 2, a number of related papers are analyzed and compared. In Sect. 3, the model is presented. Experiment results are presented and discussed in Sect. 4. In Sect. 5, a conclusion is drawn for the study.

Related work

Rumor detection

Previous studies focusing on rumors mainly consist of two aspects: rumor source detection and rumor detection. The aim of the former is to find the users who firstly generated the rumors in social media [44], while the latter is to distinguish whether the online posts are rumors or not [34]. Although issues about rumors have been investigated intensively in multiple disciplines, the number of studies on the detection of rumors in social media is still limited [2]. In general, rumor detection could be viewed as a binary classification problem using machine learning algorithms, in which some features obtained from social media data are considered [2, 7, 26]. Some previous studies stated that different learning algorithms may achieve similar results [15, 16]. Therefore, it seems that the selection of features refers to one of the most important steps in the rumor detection [26, 39]. In the present paper, we selected 12 relevant papers published in recent years and developed a framework of their employed features, shown in Table 1.

Table 1

Employed rumor features in 12 relevant studies

Papers	Content	User	Diffusion	Topic	Multimedia	Network	Number of features
[39]	√	√	√				19
[30]	√	√			√		7
[36]	√			√			12
[16]	√	√				√	9
[20]	√	√					22
[11]	√	√	√				23
[3]	√	√	√	√	√		29
[1]	√	√	√	√	√		41
[29]	√	√	√			√	24
[21]	√	√	√				38
[23]	√	√	√	√		√	11
[12]	√	√		√			14

some of above papers proposed several categories that are not included in Table . However, it seems that they can be integrated, to a large extent. For example, in Liang et al.[16], a behavior-based set of features is proposed, including verified users, average number of followers per day, and average number of posts per day, which are related to user profile. Therefore, we classified these features into the user-based group

It can be noticed from Table 1 that the features of rumors considered by previous studies mainly consist 6 classifications, namely, Content, User, Diffusion, Topic, Multimedia, and Network. However, it is difficult to find clear connections between these categories. The main reason for this confusion, as we mentioned in the introduction section, is the lack of an effective method to connect these features. Moreover, existing papers reported that only users with spammer features will post rumors [30]. Nevertheless, the fact is that some normal users who believe rumors also play an important role in accelerating the spreading process of rumors. Hence, setting all indexes together as the basis of rumor detection may negatively influence the accuracy performance of the model. In contrast, although existing studies focusing on rumor detection are relatively fewer, papers relevant to spammer detection could shed light on the understanding of this area, especially for investigation of user-based features. This is because the mechanism of models, applied methods, and employed datasets in spammer detection research are largely similar with the ones used in rumor detection [15, 25, 27, 35].

Features of rumors

Feature selection plays an important role in the rumor identification. In this section, we will briefly summarize the features of rumors that have been discovered by existing studies. First, we need to ensure the research subject of this paper, because there are some differences between the rumors in different social media platforms. Twitter and Weibo are two main micro-blogging platforms, in which rumors present a serious concern. Yang et al. [39] provided an excellent comparison between Twitter and Weibo by considering the linguistic and forwarding aspects. This is significantly useful for future research focusing on this area, especially for the feature selection step. In addition, we found that most existing studies have used Twitter as the premise of study. Therefore, in the present paper, we decide to analyze the rumor detection problem within the Weibo platform. In addition, an official account named “Weibo Refutes Rumor” has been created in recent years, which provides annual reports to summarize the rumors statistics [14]. This report is of great help for feature selection. Based on the findings of previous studies and our own understanding of rumors, we present three main categories of features that should be considered.

(1) User-based features

With respect to the features of users who post rumors, according to the 2019th annual report published by the “Weibo Refutes Rumor,” nearly all data that can be found on the personal page are contained, such as gender, age, type of account, location, following count, and follower count. Some other features proposed by previous studies can be also classified into the user-based category. For instance, Alzanin and Azimi [1] put forward a feature named “user effect” to measure whether a given user is a broadcaster or a receive. In addition, there was a widespread misunderstanding in previous studies that only users with spammer features will mostly post rumors, e.g., the theory suggested by Sunstein [31] that ‘rumor spreaders are persons who want to get attention and popularity’. However, the fact is that some normal users who have been blinded by rumors may also boost the spreading of rumors. Therefore, detection of this specific group of users should be one of important research focus areas.

(2) Content-based features

In general, the content-based features are ones related to the contents of posts, including but not limited to whether the content contains a URL, the frequency of question marks, and the number of arbitrary words and tentative words, in which most are calculated according to the keywords within the posts. These features seem to be the most direct proof to exactly judge whether a certain post is a rumor or not, and this has been fully explored by previous studies [3]. In addition, it is important to note that the diffusion-based set of features, e.g., the number of retweets and comments, are commonly employed to detect rumors [7, 14, 34]. However, the propagation process of rumors in social media is extremely that cannot be effectively and comprehensively represented by existing features, to a large extent. Similar findings were also reported by Kwon et al. [11]. In other words, these features can at most be set as some kinds of subsidiary information of the contents. Therefore, we classify these features into the content-based group.

(3) Psychology-based features

According to Ma and Liu [24] and Liu et al. [19], psychological-based features are significantly important for online opinion analysis. Jain et al. [9] proposed a rumor detection model that only considered psychological features of online posts, and the model achieved appreciable results. Therefore, in the present paper, unlike most previous studies [20, 39] that the psychological-based features are classified into the content category, we created a new set of features relevant to the psychological aspect. In addition, with respect to the emotional tendency of online posts, some existing papers only consider two kinds of emotions: positive and negative [21], which could not effectively describe the complex emotional expression contained in online posts, especially for those in Chinese language. Thus, in this paper, we put forward a psychological dictionary that contains a large number of keywords related to five common kinds of emotions, including identification, denial, doubt, anger, and excitement, in order to provide a more comprehensive representation of the psychological features of a given online post.

Supernetwork

With respect to previous studies focusing on the application of network theory in rumor detection problem, Liang et al. [16] provided the first related work, in which a network among users was established, considering publishing, reposting, and commenting behaviors. However, they only focused on the network with one layer, which cannot effectively describe the online posts. In particular, this kind of relationship is just one of the various driving forces of online post evolution, while the interpretation of real cases is much more complex than it can represent, to a large extent. In addition, it seems that existing studies have covered all elements separately that can be found in online posts, such as user attributes, posting time, content and keyword [2, 3], but the relationships between these elements, for example, different posts can have same keywords and different users can have same emotions, were generally neglected. These shortcomings may decrease the accuracy of rumor identification.

In recent years, the supernetwork theory has been employed in various areas, including online communication [19], transportation [18], knowledge [42], and supply chain [6]. A general form of the supernetwork model is shown in Fig. 1. According to Ma and Liu [24], the supernetwork is multi-layered, multi-dimensional, and multi-attributed. It is more complex than existing networks that describe the interaction between different networks, and thus providing a more comprehensive expression of a given system. With respect to issues about social media, the online public opinion supernetwork recently established by existing studies offers valuable inspiration. By using the attributes of supernetwork theory, a four-layer supernetwork considering user, information, psychology, and viewpoint features of online public opinion was proposed, in which the subnetworks are connected by superedges [19]. We also extended this model to the applications of opinion leader identification [24], rumor spreading intervene [32], link prediction [19], coupling mechanism analysis [37], post-informative quality evaluation [4], online water army detection [15]. Based on the empirical results of these studies, it can be confirmed that the supernetwork theory can be applied to comprehensively and effectively describe the features and their relationships of online posts in social media, which could improve the detection performance of rumors, to a large extent. Thus, in this paper, the supernetwork theory was employed to putforward the rumor detection model.

Fig. 1

A general form of supernetwork Notes there are four different layers in Fig. 1; nodes with similar color are included in the similar layer; black edges measure the interactions between two nodes within different layers; blue, orange, green, and gray edges measure the interactions between two nodes within a similar layer

Model

In this section, a two-step rumor detection model based on the supernetwork theory will be demonstrated in detail. In specific, the model is set with two steps: the first step is to classify the types of users based on a series of user profile features; the second step is to detect rumors posted by different types of users on the basis of some normal features and supernetwork-based features.

Rumor detection supernetwork model

With respect to the supernetwork theory regarding the application in social media, a general form of the model is displayed in Fig. 2 [15], in which there are four layers of networks: social subnetwork, environment subnetwork, psychological subnetwork, and viewpoint subnetwork. In particular, the social subnetwork refers to the communication relationship among individuals participating in the discussion; the environment subnetwork refers to the dissemination process of information; psychological subnetwork refers to the psychological transformation according to the contents of the posts; viewpoint subnetwork refers to the keywords within the post. The edge between two keywords means that they are included in a similar post.

Fig. 2

A simple form of online public opinion supernetwork model [7] Notes (1) Social subnetwork: each blue dot measures an individual; (2) Environment subnetwork: each purple star measures a message; (3) Psychological subnetwork: each yellow triangle measures a kind of psychology; (4) Viewpoint subnetwork: each green square measure a keyword

According to previous studies [4, 15, 37], this supernetwork could comprehensively describe the complex relationships between diffident features of online posts, and accordingly being useful for data mining. That is to say, by employing the supernetwork theory, more useful information can be mined and discovered from online posts, especially for the interaction and effect between different attributes contained in online posts, such as the relation between user and psychology. In addition, thanks to the unique strengths of supernetwork, namely, multi-layered, multi-leveled, multi-dimensional, and multi-attributed, a better reflection of the complex and dynamic natures of online posts can be achieved, and thus leading to a more comprehensive and accurate tool for detecting rumors from a large number of online posts. Moreover, the supernetwork theory has shown great performance in addressing data mining problems, as an example, Lian et al. [15] applied the supernetwork in identifying water armies of online posts, which provided strong evidence for the potential effectiveness of this theory in rumor detection area, to a large extent. Besides, since we need to verify the credibility of every selected post and each post is set as a separated unit, the dissemination process of information will not be considered. Therefore, based on the valuable achievements of previous studies and the theoretical guidance of the supernetwork, we established a supernetwork with three layers of subnetworks: social subnetwork, psychological subnetwork, and viewpoint subnetwork, to create a rumor detection system of rumor, as shown in Fig. 3.

Fig. 3

Rumor detection supernetwork with three layers Notes (1) Social subnetwork: each blue dot measures an individual; (2) Psychological subnetwork: each yellow triangle measures a kind of psychology; (3) each green square measure a keyword

In our built supernetwork, all subnetworks are undirected networks. In Fig. 3, lines within the same subnetwork are normal edges, and lines within different subnetworks are superedges. A superedge measures a complete post, with its user, psychology, and content. For example, the superedge SE1 is a post published by user a1 with the psychology p1, and consists of keywords of k1, k3, and k4. Moreover, within a similar subnetwork, the connections between different units are measured by a 0–1 matrix, in which 0 indicates no connection and 1 indicates there is a connection. There are three matrixes, shown in formulas 1, 2, and 3. In particular, for the social subnetwork, the social matrix is established according to the followers and the relationship between different users. For instance, in Fig. 3, a1 is followed by a2, so S (a1, a2) and S (a2, a1) are equal to 1. With respect to the psychological subnetwork, in the present paper, the psychology of a certain post is identified by employing a lexicon, consisting a large number of keywords related to five common kinds of emotions, including identification, denial, doubt, anger, and excitement. This mechanism is inspired by Vosoughi [36], who proposed a classifier of speech-acts that can distinguish six different categories. In addition, valuable understandings on the sentiment analysis of Weibo are also significantly helpful for the creation of the lexicon [13, 17, 22]. In most cases, posts in social media always contain more than one emotion. For instance, the anger and denial psychologies frequently appear simultaneously [33]. Moreover, the psychological matrix is established based on whether one post contains different emotions. In the example shown in Fig. 3, p1 and p2 are in one post, so P (p1, p2) and P (p2, p1) are equal to 1. In terms of the viewpoint subnetwork, we establish another new lexicon, named Rumor Lexicon. This lexicon is based on the keywords of 1329 rumors detected recently by the “Weibo Refutes Rumor” official account (the time period is from January 1, 2016 to July 20, 2017). After the word segmentation process, the top 200 words with the most frequency are selected to compose the lexicon, shown as Fig. 4. In addition, it should be noticed that this lexicon is fully composed of Chinese words. In order to make it more convenient for reading, we translated these Chinese words to English, in which some Chinese words have similar English meaning.

Fig. 4

Word cloud graph of 200 high frequency words of rumors

It can be found that, most of these words are easy to attract the attention of netizens, such as “death,” “accident,” and “attack.” The keyword matrix built in the viewpoint subnetwork is based on the Rumor Lexicon, and the relationship coefficient measures whether one post contains two keywords. For instance, as shown in Fig. 3, k1 k3, and k4 are included in one post, and therefore, we can get V (k1, k3) = V (k3, k1) = V (k1, k4) = V (k4, k1) = V (k3, k4) = V (k4, k3) = 1.

Features

Based on the rumor detection supernetwork, we can create a new set of features called supernetwork-based features, as follows:

(1) Social Subnetwork Clustering Coefficient

This indicator measures the aggregation degree of a specific user within the social subnetwork, which can be calculated through formula (4).where C is the clustering coefficient of user i, k is the number of users connecting to user i, and n is the number of edges that actually exist between these k users [41]. In addition, the connection relationship of each two users within the social subnetwork can be obtained through the social matrix.

(2) Psychology complexity

This indicator reflects the complexity degree of the psychology for a certain post, which can be calculated through formula (5).where PC is the psychology complexity degree of post m, p is the strength of ith psychology included in post m, N is the total number of psychologies contained in post m, and P (p, p) is the connection relationship between p and p within the psychological subnetwork. In specific, p is the ratio of the number of words with ith psychological tendency and the number of emotional words that are contained in post m.

(3) Rumor Keywords Density

This indicator represents the density of rumor-related keywords within a certain post, which can be obtained through formula (6).whereRKDis the rumor keywords density of post m, k is the strength of the hth rumor keyword included in the post m, M measures the total number of rumor-related keywords within post m, and V(k, k) is the connection relationship between k and k within the viewpoint matrix. It should be noted that the proposed Rumor Lexicon is established according to 1329 different rumors, and k is the ratio of the number of hth rumor keyword and the number of rumor keywords that are contained in post m. In addition, the left half of formula (6) is used to give the weight of each keyword regarding the co-occurrence feature, where the one with higher co-occurrence frequency will have greater weight.

(4) Superedge Similarity

This indicator reflects the similarity degree of a certain post. First, we need to obtain the similarity between two posts from a supernetwork perspective, which is based on the superedge (as mentioned above, a superedge measures a complete post, including the user, psychology, and content). It can be calculated through formula (7).where JBS measures the Jaccard-based superedge similarity between post m and post o (JBS∈ [0,1]), |SE ∩ SE| is the intersection between superedges SE and SE, and |SE ∪ SE| measures the union between superedges SE and SE. The intersection and union are measured by the number of nodes included in the superedges. This formula is based on the Jaccard Similarity Coefficient proposed by Jaccard [8]. In fact, Liang et al. [16] applied this formula to calculate a user behavior-based feature named the Number of Possible Sources, in which the intersection and union are represented by the keywords. However, to some extent, our proposed method is more advanced because it can gain the similarity within multidimensional networks. Therefore, we can have a similarity matrix JB_Sim between two different posts, as shown by formula (8).

Based on the similarity matrix, we can calculate the similarity degree of a certain superedge (post) through the following formula:where SS measures the similarity degree of superedge m, and M is the total number of superedges.

With respect to the features selected in this paper, two groups are considered. The first group is based on the valuable achievements of previous studies. Then, the supernetwork model is applied to re-classify existing features. In addition, due to that the research subject of the present paper is to detect rumors in the Weibo, only features that are specific to this social media platform are chosen. Moreover, we also learned much useful knowledge from some previous studies focusing on the spammer and Internet water army detection. For the second group of features, supernetwork theory allowed us to propose some new features that can measure the relationships within and between different subnetworks, which yet are not considered by previous works. In particular, four new supernetwork-based features, namely, social subnetwork clustering coefficient, psychology complexity, rumor keywords density, and superedge similarity, were introduced. Finally, 29 features were selected in the present paper, as shown in Table 2.

Table 2

List of rumor-related features selected in this paper

Group	Serial	Description
User-based features	F1	Whether the user's identity is verified by Weibo or not
	F2	Whether a user account is an authentic account or not
	F3	Whether the user has personal descriptions or not
	F4	Gender
	F5	Type of user
	F6	Age
	F7	Ratio between following and follower
	F8	Location
	F9	Time between the registration time and posting time
	F10	Time between this post and last post
	F11	Number of previous posts
	F12	Posted by PC or mobile
	F13	Average time between each two contiguous posts
Psychology-based features	F14	Psychology of posts (based on the psychological lexicon)
Content-based features	F15	Whether it contains multimedia or not
	F16	Number of the URLs
	F17	Number of specific symbols (@, *, #, etc.)
	F18	Original or forward
	F19	Length of post
	F20	Number of comments
	F21	Number of forwards
	F22	Number of “liked”
	F23	Time between the first comment and the last comment
	F24	Time between the first forward and the last forward
	F25	Time between the first “liked” and the last “liked”
Supernetwork-based features	F26	Social Subnetwork Clustering Coefficient
	F27	Psychology Complexity
	F28	Rumor Keywords Density
	F29	Superedge Similarity

Our proposed rumor detection system is shown in Fig. 5. First, the data were selected from Weibo by employing the crawler technique. In particular, we chose several heated topics and crawled their related online posts based on some relevant keywords. The posts of each topic are set as a dataset. Then, according to the verified rumors of the chosen topics published by the official account “Weibo Refutes Rumor,” we separated each dataset into two aspects: the true posts and false posts (rumors). Then, we employed three students to manually labeled abnormal users from the data. Hence, we have four categories: normal users who post rumors; abnormal users who post rumors; normal users who post true information; abnormal users who post true information. Moreover, each dataset is set as a case to verify the effectiveness of the proposed method. In general, it is significantly difficult to detect rumors within the posts of a certain topic because their contents are largely similar, thereby making the conclusion more convincing.

Fig. 5

The framework of Two-Step Supernetwork Rumor Detection System

Two-step mechanism

In addition, as mentioned above, apart from abnormal users, some normal users who post rumors should also be considered. Here, the abnormal users can be seen as possible rumormongers, who are always featured with some special user-based features, such as no personal description and short time between the registration time and posting time. Thus, simultaneously used the user-based and non-user-based features to detect rumors may influence the performance of the model. Thus, we proposed a two-step mechanism in the rumor detection process, displayed as follows:

Step 1: user classification

Based on the user-based features and the social subnetwork clustering coefficient shown in Table 2, we divided each dataset into two groups: normal users and abnormal users. This is a binary classification problem. According to previous studies, we used the four most popular machine learning methods to train the first classifier (Naive Bayes, Neural Network, Support Vector Machine, and Logistic Regression).

Step 2: rumor detection

Then, by using the psychology-based features, content-based features, and supernetwork-based features, except the social subnetwork clustering coefficient, combined with the above four machine learning methods, we trained the second classifier to detect the rumors within normal user group and abnormal user group. Finally, we obtained the whole rumors within the posts of a certain topic as the output.

Experiment

Data collection

The dataset for this study included 219,367 online posts and their user account information related to four recent hot topics, including “Illegal vaccine scandal in Shandong, China,” “The establishment of Xiongan New Area,” “RYB kindergarten teachers abused children Feeding tablets”, and “COVID-19.” Related information of these four cases can be found in Tables 3 and 4. In addition, to adapt machine learning methods, each dataset was divided into two aspects: training set and testing set. According to previous practices [25], a ratio of 50% between these two sets was employed. In addition, according to previous studies, early rumor detection refers to another important metric when evaluating the performance of the rumor detection models, as it could provide the best chances for prevention of rumors [21, 34, 43]. Luo et al. [21] stated that the core idea of early rumor detection refers to the small amount of data in the early stage. Thus, each dataset was divided into three datasets based on the posting time of posts: top-20% dataset, top-50% dataset, and 100% dataset. For instance, with respect to the case of “COVID-19”, top-20% dataset contains posts collected between December 20, 2019 to January 3, 2020, top-50% dataset contains those between December 20, 2019 to January 25, 2020, and 100% dataset contains those between December 20, 2019 to March 1, 2020.

Table 3

Rumor information of four topics

Topic	Rumor information
Illegal vaccine scandal in Shandong, China	1. Disabled children have a clear causal relationship with vaccination
	2. More than 1000 children are left with lifelong disability in each year
	3. It is life-threatening to fight a non-refrigerated vaccine
	4. Adverse reactions are all due to problems with the vaccine
	5. The second type of vaccine is unsafe
	6. Chinese domestic vaccine is low-tech
	Source: http://blog.sina.com.cn/s/blog_96a0474d0102w9xu.html
The establishment of Xiongan New Area	1. A picture showing a dead male who jumped out of windows widely transmitted via the Internet. The reason for suicide is that he sold his house located in Xiong’an City before the establishment of “Xiong’an District” (The price of houses located in Xiong’an City grew rapidly after the establishment of “Xiong’an District”)
	2. 87 Central Government-owned enterprises will move to Xiong'an New District
	3. The policy of car plate lottery will be implemented in Baoding City in June
	4. The Capital Third Airport will be located in Baoding City, Xushui District
	Source: http://finance.ifeng.com/a/20170630/15500612_0.shtml
RYB kindergarten teachers abused children	1. Children are forced-fed white pills that “made them go to sleep”
	2. Children were subjected to naked “health checks” at the daycare
	3. Children were sexually molested
	4. A Chinese military regiment was involved in molesting the children Source: https://weibo.com/weibopiyao?refer_flag=1005055013_&is_hot=1
COVID-19	1. A graduate student from Wuhan Virus Research institute, Chinese Academy of Sciences is the Patient Zero of COVID-19
	2. Zhong Nanshan did not wear a mask in the hospital
	3. Zhong Nanshan predicts the time for lifting restrictions
	4. Normal travel would be recovered in March 16, 2020
	5. A module hospital would be built in Beijing Jishuitan Hospital
	6. A woman spat at the doorknob of a residential area in Wuhan
	7. Mutation virus appeared in Wenzhou
	8. A man infected by COVID-19 coughed hundreds of times in a market
	9. Government works in Tianmen, Hubei Province dumped the radishes donated by other provinces into the garbage station
	10. Floating workers to Shanghai could not enter the community
	11. People without a pass would be segregated
	12. 6 points deducted for driving without mask
	13. Dozens of mask manufacturers have stopped production in Changyuan, Henan Province
	14. 30,000 sheep donated by Mongolia were driven to Er Lian Hao Te, Inner Mongolia Source: https://weibo.com/1866405545/IwA26hnGC?from=page_1002061866405545_profile&wvr=6&mod=weibotime&type=comment#_rnd1614583135228

Table 4

Description of the data

Topic	Time period	keywords	Number of posts
			Rumor	Non-rumor	Total
Illegal vaccine scandal in Shandong, China	March 18, 2016 to March 24, 2016	illegal vaccine; Shandong Province	5441	17,571	23,012
The establishment of Xiongan New Area	April 1, 2017 to April 7, 2017	Xiongan; New Area	12,219	37,878	50,097
RYB kindergarten teachers abused children	November 12, 2017 to November 27, 2017	Red Yellow Blue (RYB) kindergarten; Child; Abuse	5736	7846	13,582
COVID-19	December 20, 2019 to March 1, 2020	COVID-19; Coronavirus	42,005	90,671	132,676

Comparison results

To measure the performance of the proposed rumor detection model, following common practice [11, 12, 20], four methods were used, including Accuracy (Acc.), Precision (P.), Recall (R.), and Fa-score (Fa). Their formulae are shown in (10), (11), (12), and (13), respectively.where Correct_Rumors measures the number of correctly identified rumors, InCorrect_Rumors and InCorrect_NonRumors are the number of rumors and non-rumors that are misjudged. Rumors and non-rumors are the total number of rumors and non-rumors within one dataset, while a is a constant parameter that was always set as 1 in previous studies to equally weight Precision and Recall [16]. In the present paper, F1-score (a = 1), F2-score (a = 2), and F0.5-score (a = 0.5) were used. In particular, F2-score means that the Precision is double weighted compared to Recall, while F0.5-score means that the Recall is double weighted compared to Precision.

Five existing models were selected as benchmarks, including those proposed by Yang et al. [39], Sun et al. [30], Liang et al. [16], Ma and Luo [23], and Lathiya et al. [12]. Four machine learning methods were used to train the classifiers, including Naive Bayes (NB), Neural Network (NN), Support Vector Machine (SVM), and Logistic Regression (LR). In specific, for the Support Vector Machine method, the RBF kernel function with γ = 0.5 and tenfold cross-validation is used; while for Neural Network method, a three-layer Back Propagation Neural Network was applied (the activation function of the hidden layer and the output layer were both the sigmoid function). The training function and the performance function were, respectively, set as the Levenberg–Marquardt and the Mean Squared Error (MSE), and the number of hidden layers and output layers was both set as 1. In addition, the number of neurons in each hidden layer and output layer was set as 4 and 1, respectively.

It should be noticed that, for the feature “the number of event verbs” contained in Sun et al. [30], the verb lexicon of ICTCLAS (version: 2016) was used to achieve this feature. The number of duplications was also calculated based on the ICTCLAS. For the strong negative words, because no details were provided in their paper, an alternative way was implemented. Specifically, based on our own understanding, we selected 50 strongest negative words from the negative word lexicon contained in the ICTCLAS to achieve this task. Additionally, the multimedia-based feature was realized through their introduced approach. For Ma and Luo [23], we used the words contained in the Rumor Lexicon to replace their proposed bad words. With respect to Lathiya et al. [12], ICTCLAS was used to build the part-of-speech tags of posts.

To test the contribution of the two-step mechanism and the supernetwork-based features to the accuracy improvement of the proposed model, we created three different models, as follows.

Model A: model with two-step mechanism but without supernetwork-based features;

Model B: model with supernetwork-based features but without two-step mechanism;

Model C: model with two-step mechanism and supernetwork-based features.

With each method for each model and each dataset run 50 times, the average results were taken as the final results. An overview of performance evaluation results considering four machine learning methods and five assessment metrics is shown in Table 5 (the model with the best performance for each of metric is labeled in bold) and the average results are displayed in Fig. 6. It can be noticed that our proposed model (Model C) can be adopted to provide significantly better effectiveness in rumor detection compared to the other models. In particular, the average values of Accuracy, Precision, Recall, F1-score, F0.5-score, F2-score gained by the proposed model are 85.22%, 68.47%, 84.59%, 75.14%, 70.89%, and 82.4%, respectively. In addition, it seems that the results of five benchmark models taken in this paper are poorer than those in their own experiments, which is probably due to the large number of relatively similar posts selected in the present paper. In addition, with respect to the relatively weak performance achieved by the model proposed by Sun et al. [30], that only a small number of pictures appeared in the selected online posts and our tests focusing on the early rumor detection seem to be main reasons. Moreover, results show that our proposed two-step mechanism and supernetwork-based features are significantly helpful for improving the utility of the rumor detection model. In particular, the addition of the former increases the Accuracy, Precision, Recall, F1-score, F0.5-score, F2-score by an average of 5.98%, 9.57%, 5.84%, 8.59%, 9.3%, and 6.63%, respectively; while the latter increases by an average of 5.81%, 9.32%, 5.62%, 8.5%, 9.14%, and 6.47%, respectively. Additionally, it can be noticed that the performance of Model A and Model B were not as effective as we expected, especially when compared to the model introduced by Liang et al. [16], Ma and Luo [23]. The possible reasons were summarized as followings:

Table 5

Comparison results of the models

		Acc. (%)	P. (%)	R. (%)	F1(%)	F0.5(%)	F2(%)
NB	Model A	80.19	60.22	79.45	67.59	62.79	76.55
	Model B	78.96	58.55	78.07	66.07	61.20	75.12
	Model C	85.21	68.11	84.98	75.19	70.69	82.73
	[39]	77.48	56.59	76.62	64.28	59.30	73.57
	[30]	78.31	56.93	77.60	65.40	59.99	74.73
	[16]	82.94	64.36	83.00	71.91	67.07	80.38
	[23]	85.14	70.55	79.42	74.56	72.06	78.37
	[12]	79.45	59.02	75.85	66.04	61.58	73.59
NN	Model A	79.20	59.20	78.00	66.16	61.59	75.04
	Model B	79.06	58.72	77.77	66.06	61.30	74.91
	Model C	83.72	70.33	81.68	72.64	71.61	79.57
	[39]	76.16	54.53	74.95	62.28	57.24	71.80
	[30]	78.20	56.90	78.12	65.18	59.83	74.98
	[16]	82.45	64.12	81.17	70.85	66.50	78.70
	[23]	83.64	67.26	77.38	71.84	69.00	76.18
	[12]	81.80	64.41	74.39	68.95	66.14	73.22
SVM	Model A	79.48	58.97	79.92	67.01	61.79	76.75
	Model B	79.85	59.92	79.52	67.47	62.57	76.56
	Model C	86.67	70.98	86.78	77.42	73.29	84.59
	[39]	76.64	55.44	74.06	62.65	57.99	71.25
	[30]	76.84	54.83	73.93	62.50	57.58	71.22
	[16]	80.40	60.46	78.81	67.82	63.09	76.20
	[23]	82.63	64.38	82.44	71.52	66.93	79.81
	[12]	80.25	61.45	76.48	67.46	63.61	74.30
LR	Model A	78.78	58.22	78.50	65.81	60.84	75.34
	Model B	79.11	58.43	79.65	66.60	61.30	76.45
	Model C	85.30	68.47	84.91	75.32	70.97	82.69
	[39]	78.46	58.11	77.35	65.60	60.77	74.46
	[30]	78.93	57.87	78.55	66.16	60.84	75.59
	[16]	81.24	61.49	81.33	69.51	64.37	78.53
	[23]	81.45	63.59	80.47	69.76	65.69	77.77
	[12]	79.15	58.99	75.31	65.55	61.35	72.97

Fig. 6

Performance of the models based on average results of four learning algorithms

(1) For the Model A, the supernetwork-based features were not considered, which means that our established Rumor Lexicon was not applied. However, this lexicon is important for detecting rumors as it contains 200 top frequent words selected from 1329 rumors detected recently by the “Weibo Refutes Rumor” official account [23]. In addition, despite the contribution of the supernetwork theory on the feature classification that could make this process more logical and suitable, the provided possible way that could measure the relationships within and between different subnetworks are much more important. Therefore, the lack of other supernetwork-based features might also lead to the weak performance of Model A, to some extent.

(2) For the Model B, the two-step mechanism was not applied, which indicates that the user-based features were not used in this model. However, according to previous studies, this kind of features play a significant role in the rumor detection index system [26, 29, 34]. Thus, it seems that this missing factor could be cited as the main reason for the weak performance of Model B. Nevertheless, when compared to Model A and the models proposed by Yang et al. [39] and Sun et al. [30], relatively similar performance was gained through Model B, which also proves the effectiveness of our proposed supernetwork-based features in detecting rumors.

Conclusion

Rumors have been one of the most urgent necessities in online social networks, as they could cause potential damages to the nation, society and the individual [2]. Thus, it is of great significance to detect the rumors early, automatically and accurately. In this paper, a two-step rumor detection model based on the supernetwork theory was proposed. In particular, a three-layer supernetwork consisting of user, psychology, and keywords subnetworks was established, and the connections within one subnetwork and between different subnetworks were considered, in order to provide a comprehensive description of online posts in social media. A set of 29 features consisting of four categories (user-based group, psychology-based group, content-based group, and supernetwork-based group) were applied for rumor detection. We also put forward a two-step mechanism in the model, which is helpful for identifying rumors posted by normal users, and thus considerably improving the effectiveness of the model. Four datasets containing online posts related to four recent topics and four commonly applied machine learning methodologies were employed to compare the detection performance of the model with five previous models. In addition, the performance of the model regarding early rumor detection was evaluated by separating datasets based on the posting time. According to the experiment results, our proposed rumor detection model showed significantly better performance compared to other five previous models, and the two-step mechanism and supernetwork-based features had important contributions. In specific, our proposed model achieved in Accuracy of 86.67% and F2-score of 84.59% when the SVM was employed. It triumphed some baselines on four datasets with up to 5.11% and 7.65% improvement, respectively, on Accuracy and F1-score in average.

However, this paper is just a general attempt to explore the contribution of the theoretical principle for solving rumor detection problem. We only used online posts from Weibo to evaluate and compared the performance of the proposed model, which may reduce the assessment credibility. In addition, in the present paper, features were roughly classified into two categories: user-based features and non-user-based feature, in order to achieve our proposed two-step mechanism. However, how to distinguish normal users and abnormal users is a considerably difficult task, which cannot be effectively addressed by simply using several user-based features.

In our future studies, more theories from psychological and behavioral areas would be considered for identifying the features of rumors. Data collected from different social media platforms, such as Twitter and Facebook will be used to evaluate the performance of the models. We attempt to optimize the two-step mechanism by paying more attention on how to more scientifically classify the features and find more useful features in order to further improve the accuracy of the experiment. In addition, deep learning algorithms will be applied in our future works.