A Multi-RNN Research Topic Prediction Model Based on Spatial Attention and Semantic Consistency-Based Scientific Influence Modeling.

Computer science discipline includes many research fields, which mutually influence and promote each other's development. This poses two great challenges of predicting the research topics of each research field. One is how to model fine-grained topic representation of a research field. The other is how to model research topic of different fields and keep the semantic consistency of research topics when learning the scientific influence context from other related fields. Unfortunately, the existing research topic prediction approaches cannot handle these two challenges. To solve these problems, we employ multiple different Recurrent Neural Network chains which model research topics of different fields and propose a research topic prediction model based on spatial attention and semantic consistency-based scientific influence modeling. Spatial attention is employed in field topic representation which can selectively extract the attributes from the field topics to distinguish the importance of field topic attributes. Semantic consistency-based scientific influence modeling maps research topics of different fields to a unified semantic space to obtain the scientific influence context of other related fields. Extensive experiment results on five related research fields in the computer science (CS) discipline show that the proposed model is superior to the most advanced methods and achieves good topic prediction performance.

1. Introduction

In recent years, with the rapid development of computer science and technology, the number of papers in many research fields of computer science discipline has been increasing rapidly. These research fields influence each other and promote their own development [1]. Tracking the research progress and predicting the research topic trend of these research fields are of great significance. It has important reference value for scientific and technological innovation decision-making [2] and helps to guide government agencies to formulate scientific development strategies and policies. It is also of great significance for researchers to keep up with the rapid development of research [3].

The increasing number of publications and the rapidly changing research trend make it difficult to keep up with the development trend of scientific research of different research fields. In recent years, tracking and understanding the evolution of scientific research topic have attracted extensive attention [4, 5]. For example, based on the datasets of information retrieval publications, Chen et al. study how topics evolve by analyzing topic trends, evolution dynamics, and semantic words [6]. A topic evolution algorithm is proposed, including topic segmentation and topic dependency relations calculation [7] to effectively discover important topics and reflect the evolution of important research topics. Soumya et al. propose an effective method to discover the development trend of science by using graph-based subject classification of academic publications [8]. However, little effort has been made to predict the future research topic trend. The existing prediction methods for future topics are mainly based on expert evaluation. In essence, predicting the trend of future research topic is a time series prediction problem [9-11]. A few studies have been carried on predicting the trend of the future research topic. For example, the traditional time series prediction method ARIMA [12] has been employed to predict the development trend of research topics of conference papers on computer science discipline, which contains a total of 5982 papers over 17 years. Saman et al. construct a scientific knowledge network by using the keywords of articles in computer science journals and conferences and use the link prediction method to predict the future structure of the keyword networks [13]. With the development of deep learning, some Recurrent Neural Networks such as GRU and LSTM have been extensively studied in sequence modeling [14-16] and applied in evolution analysis and prediction tasks [17, 18]. For example, Chen et al. take the computer conferences as the research objects [19] to deploy GRU to model the topic sequences and propose a correlated neural influence (CONI) model. Specifically, Recurrent Neural Networks encode conference research topics into a hidden state which is a dense and low-dimensional vector (each dimension represents an attribute feature of the conference topic) to capture the research interests of the conference. At the same time, CONI verifies that the future topic trend of a conference is influenced by its peer conferences and models the scientific influence context of a conference topic by calculating the similarity of topics among the conference and its peer conferences.

However, the above methods of research topic sequence modeling based on the Recurrent Neural Network do not distinguish the importance of different attributes of a field's research topic. Intuitively, each attribute of a field's topic is not equally important. What is more, research topics of different fields are also different, which should be modeled by different Recurrent Neural Network. The existing scientific sequential modeling of research topic employs Recurrent Neural Network to model sequences of all fields by the same Recurrent Neural Network chains, which share the same parameters, which leads to poor topic prediction precision. So, topic sequences of different research fields should be distinguished using different Recurrent Neural Network chains. And, inspired by semantic consistency modeling [20, 21], when using the research topics of related fields to model the scientific influence context of research topics of a field, we need to transform them into consistent semantic space to calculate the similarity.

Based on the above discussion, this paper proposes a research topic trend prediction model based on spatial attention and semantic consistency-based scientific influence modeling (SASC). SASC employs multiple different RNN chains, which have their own parameters to model research topics of different field. Spatial attention employs a self-attention network to generate different spatial attention weight to distinguish the importance of the different attributes of topics in different research fields, which can learn fine-grained topic representation. Semantic consistency-based scientific influence modeling applies a linear transformation to achieve semantic consistency learning. It maps the research topics of each field to a consistent semantic space and obtains scientific influence context by calculating the similarities of topics among the field and its related fields.

The contributions of this paper are as follows:

We propose a topic representation method of different fields based on spatial attention. The spatial attention mechanism gives different weights to different attributes of a field's topic to distinguish the importance of each attribute to achieve fine-grained topic representation.

We employ multiple different RNN chains that model different field research topic and propose a semantic consistency-based scientific influence modeling method that can map research topics of different fields into a comparable feature space to model the interactive scientific influence context among different fields to improve the quality of scientific influence context.

We contribute a research topic prediction dataset including publications of five fields in the computer science discipline and will make it available to the public. We conduct experiments on the dataset to demonstrate the effectiveness of the proposed topic prediction model. Experimental results show that the proposed model can greatly improve the precision of topic prediction.

The rest of this paper is organized as follows: Section 2 discusses related work, and Section 3 describes preliminaries. Section 4 introduces the research topic trend prediction model based on spatial attention and semantic consistency-based scientific influence modeling in detail. Section 5 reports experiments results and analysis, and we summarize this work in Section 6.

2. Related Work

2.1. Scientific Research Trend Prediction

For research trend prediction, people have done some exploration. First, citation prediction has been widely studied. For example, based on the characteristics of highly cited papers, Yan et al. applied a regression model to study the interesting citation count prediction [22]. Li et al. use the comprehensive semantic representation of peer-reviewed data learning papers to establish a neural prediction model to improve the citation prediction performance [23]. Second, the prediction of rise and fall of the topic attracted many scholars. Prabhakaran et al. train topic models and a rhetorical function classifier to map topic models onto their rhetorical roles. It verified that the topic's rhetorical function is highly predictive of its eventual growth or decline [24]. Instead of themes, concepts are used to construct a model to predict their rise and fall trends [25], taking rhetorical features into account. In addition, other types of scientific research trend prediction tasks have also been focused on. For instance, Rotolo et al. define how to classify technology as “emerging” technology, determine five characteristics of the emergence of new technology [26], and identify the main empirical methods used to detect and study emerging technologies. Users' actions across sessions are studied [27] to reveal correlations among various behavioral signals and build a specialized model for download prediction. Xie proposes a learning model [28] to predict the number of researchers' collaborators by fitting the evolution trend of the number of researchers' collaborators. A two-step solution is proposed to solve the emerging topic prediction problem. In the first step, the future popularity score is introduced, which is a new indicator reflecting the impact and growth to predict candidate topics. The second step selects the popular novel topic with domain characteristics from the candidate topics [29]. This paper focuses on the prediction of research topics in scientific research and proposes a prediction model of research topics.

2.2. Attention-Based Time Series Prediction

Attention mechanism has been widely used in time series prediction tasks. Currently, a key problem of attention-based time series prediction is to represent and learn the spatiotemporal relationship of time series. Researchers employ attention mechanisms based on different spatiotemporal characteristics from different application perspectives. A reverse temporal attention model is employed using electronic health record data, which can achieve high predictive precision while maintaining interpretability [30]. Based on the two-level neural attention mechanism, a recursive neural network is used to predict the readings of geographical sensors in the next few hours [31]. It considers the readings of multiple sensors, meteorological data, and spatial data to predict air quality and water quality. The effectiveness of attention-based Recurrent Neural Network (RNN) for short-term and long-term prediction of dissolved oxygen was studied, which systematically discussed and compared the application of dissolved oxygen prediction methods based on spatial attention, temporal attention, spatiotemporal independent attention, and spatiotemporal joint attention [32]. Shi et al. propose a novel end-to-end attention-based Periodic-Temporal Neural Network [33] to capture the spatial, short-term, and long-term cycle dependence and achieve accurate traffic prediction. A multistage attention spatiotemporal graph network traffic prediction model is proposed [34] to dynamically capture the spatial correlation in the same ordered neighborhood and different neighborhoods. In addition, a time attention mechanism is used to extract dynamic time dependence. Generally speaking, different time series prediction networks based on attention mechanism can be applied to different tasks. This paper studies the feature attention mechanism of different fields in computer science so as to achieve accurate research topic prediction of different fields.

2.3. Scientific Influence Modeling

Measuring the scientific influence is very important for the development of science and the allocation of resources. Some scientific influence indicators such as h-index [35] and g-index [36] have been proposed to evaluate the influence of scholars or journals. Zhu et al. introduced j-index [37] to model the topic level academic influence according to the novelty of each article and its contribution to the cited article. A novel method is proposed to quantify the higher-order citation influence of publications to quantify and visualize citation flows among disciplines and to assess their degree of interdisciplinarity considering both direct and indirect citations [38]. Hu et al. construct time-aware weighted graphs [39] to quantify the importance of links established at different times to fuse the rich information in a mutual reinforcement ranking framework to rank the future influence of multiobjects simultaneously. The above methods do not use the scientific influence to explore the topic prediction of future research trends; only a small number of studies have explored this topic. The correlated neural influence (CONI) model [19] is proposed to integrate the scientific influence of the peer conferences to predict research topics of the conference. It is proved that peer conferences of a conference have an important influence on the future topic prediction of the conference. However, it does not consider the semantic space consistency of different conference topics when modeling the scientific influence context of peer conferences, which leads to poor influence context quality. By mapping topics from different research fields to a consistent semantic space, we can improve the quality of scientific influence context so as to achieve more accurate research topic prediction.

3. Preliminaries

3.1. Recurrent Neural Network

Recurrent Neural Network [40] (RNN) can deal with the long and orderly input sequence of text data. It simulates the order in which a person reads an article, reads every word from the beginning to the end, and encodes the useful information into the state variable so that it has a certain memory ability and can help better understand the later text.

In the vanilla RNN model, there is a serious problem in the process of training; that is, the gradient disappears or the gradient explodes. In order to solve the problem, LSTM [41] and GRU [42] are proposed. The structures of vanilla RNN, LSTM, and GRU are shown in Figure 1.

Figure 1

The structure of Recurrent Neural Network. (a) Vanilla RNN, (b) LSTM, and (c) GRU.

In Figure 1(a), o is the output of RNN, and the calculation formulas are as follows:where x represents the element of step t in the input sequence and h and h are, respectively, the output of RNN in the t and t−1 time step. U, V, and W are parameters.

In Figure 1(b), the existence of a gate mechanism enables LSTM to visually model the long-distance dependence in the sequence. By learning the gate parameters, the network can find the appropriate internal storage behavior. The calculation formulas are as follows:where W, W, W, W, b, b, b,  and b are parameters.

In Figure 1(c), GRU has only two gates, reset gate R and update gate Z. R and Z jointly control how to get the new hidden state h from the previous hidden state h. The calculation formulas are as follows:where W, W, and W are parameters.

3.2. Attention Mechanism

Attention mechanism is widely used in various tasks of natural language processing (NLP) based on deep learning. Bahdanau et al. applied attention mechanism to machine translation task for the first time [43]. Then, attention mechanism has become a research hotspot of neural network. Attention refers to the use of attention to extract sentence attention information without any additional information. Attention mechanism also achieved good results in various tasks. It has a very good performance in many NLP tasks.

The essence of attention can be described as a mapping from an input (query) to a series of (key-value) pairs, as shown in Figure 2. The first stage is to calculate the similarity between the query and each key to get the weight. The common similarity functions are dot product, splicing, perceptron, and so on. The second stage is to normalize these weights by using the softmax function. Finally, the weight and the corresponding key-value are weighted to get the final output. At present, in NLP research, key and value are often the same; that is, key equals value.

Figure 2

The framework of attention mechanism.

4. Research Topic Prediction Model Based on Spatial Attention and Semantic Consistency-Based Scientific Influence Modeling

4.1. Problem Definition of the Prediction Model

For a certain research field, the research topics are the words that can fully reflect the research hotspots of the field. In this work, research topics are the words that are representative nouns or adjectives that appear frequently in papers of this field. For example, for research field i at year t, we collect the titles of all the papers of this research field, remove the stop words, and then use words with word frequency greater than one as research topics.

For the collection of papers P = {f, f,…, f} in computer science discipline involving n fields, fi stands for the i research field. The vocabulary size of P is v. One-hot vector  f ∈ R is employed to represent the topic words of the t year in fi field, where f = {c1,  c2,…, c, c}, c is normalized word frequency of w, and c is calculated as follows: where tf(w) is word frequency of topic word w in fi field and num is the number of all the topic words of fi field.

Research topic prediction is to predict future research topics based on historical observations. This can be formulated as a time series prediction problem as follows.

Given one-hot vector  f, f ∈ R, which, respectively, represent the research topic of fi field at year t and year t + 1. Given f as the model input, we aim to learn a mapping function prediction such that , resulting in an accurate topic prediction precision of . In other words, the model is trained to predict target topic series in the t+1 time step based on the feature series from the past t time steps. The topic prediction model is optimized by approximating the predicted topic distribution to the target topic distribution f.

In the computer science discipline, the research topic of one field will change with the development of other related fields. The research topics of a field in year t + 1 should be predicted according to its own research topics before year t + 1 and the research topics of related fields before year t. Recurrent Neural Networks encode field research topic into a hidden state which is a dense and low-dimensional vector to express the research interests of each field. Each dimension represents an attribute feature of the field topic. The importance of each attribute of each research field's topics is different. When representing research topics of each field, we should distinguish the importance of each attribute of different research field's topics.

At the same time, different field has different research topics and belongs to different semantic spaces. When selecting the scientific influence context of related fields, the transformation of semantic space should be fully considered to obtain the optimal scientific influence context. Thus, in this paper, based on Recurrent Neural Network, we employ multiple different RNN chains which have their own parameters to model research topics of different field and propose a topic prediction mode based on spatial attention and semantic consistency-based scientific influence modeling (SASC) to enhance the precision of research topic prediction. The model is shown in Figure 3.

Figure 3

Research topic prediction model based on SASC.

4.2. Spatial Attention-Based Sequential Modeling of Field Research Topic

In order to track the research progress of each field and explore its sequence characteristics, RNN is deployed to model the research topic sequence. It takes the research topic of the current time step as the input and iteratively encodes the research topic into a hidden state to capture the research topic of the field. The sequences of all fields are modeled by multiple Recurrent Neural Network chains. Suppose that there are three research fields, i, j, and k; taking field i as an example, this paper introduces how our model updates the status of hidden research topics according to historical research topics.

Given the topic sequence of research field i, f={f1,  f2,…,  f}, where f is the research topic of the tth year of research field i and f ∈ R. Word embedding matrix ∅ is employed to transform f into a dense low-dimensional vector in order to avoid the curse of dimensionality when the vocabulary size increases where ∅∈R. The research topic of the t year of research field i is represented by x.where x=(x, x,…,x) ∈ R . Taking the research topics embedding x as the input, the hidden state h which captures research topics of field i at year t is iteratively updated. The calculation is as follows:where RNN has different variants such as vanilla RNN, GRU, and LSTM, and in this work, we use LSTM. Each dimension of h represents different feature attributes of field topic. Take the research topic of Artificial Intelligence field as an example. The research topic may be affected by a variety of factors, such as topic frequency and popularity. Different feature attributes have different effects on the final topic representation and cannot be treated equally. So, we employ spatial attention to calculate attention weight to distinguish the importance of each attribute of field topic. The spatial attention mechanism is deployed into conventional RNN-based topic sequence modeling to differentiate the importance of each attribute sequence of field research topic. Since any attribute value at any time has its corresponding weight, the topic representation of field i after the spatial attention weighting is . In the same way, the research topic of the t-1th year of research field j and field k can be represented to be and . The calculation is inwhere  W, W, W, W, W, W W, W, and W are hyperparameters.

4.3. Scientific Influence Context Modeling Based on Semantic Consistency

For a certain field, Recurrent Neural Network is deployed to capture the research topics of this field [44]. The future research topics of a field will be affected by the research topics of other related fields. Therefore, in addition to tracking the research topics within the field, we also need to track the research topics of its related fields and calculate the influence context of other fields on this field. Through the deployment of the attention mechanism, we can effectively select the scientific influence context of related fields [45]. The scientific influence modeling based on semantic consistency is shown in Figure 4.

Figure 4

Scientific influence modeling based on semantic consistency.

Given research topics of the t year of field ix , when predicting future research topics, scientific influence context of research topics of the t−1 year of related fields to field i should be learned. In fact, for field j and k, and , respectively, express their research topics of the t−1 year. Moreover, due to the different research topics of different fields, their semantic space is not in the comparable space. When calculating the scientific influence among fields, we need to map them to the comparable semantic space and then calculate the influence context to ensure the selection of the optimal influence context. Therefore, we model the influence context based on semantic space consistency.

Firstly, we map x, h, and h to the same semantic space by linear transformation. Thus, x, h, and h is transformed to be , , and . The calculation is as follows:where W, W,  and W are parameters.

Then, the influence of fields j and k on field i is s and s, which are calculated as follows:where ⊙ is an elementwise multiplication. The influence relationship among fields is represented as matrix G. It is supposed that the evolution of research topics in field i is influenced by the research topics in all relevant fields. G ∈ R; line i of G indicates that field i is affected by all related fields. So, if i ≠ j, G=1; otherwise, G=0. At the same time, we learn an influence parameter vector λεR which represents the strength of field i affected by field j. Scientific influence context influ of research topic of field i is calculated as follows:

However, and influ do not belong to the consistent semantic space, so we map influ to be which is in the consistent space as before fusion.where Winf is a parameter.

The softmax function is employed to output the predicted distribution of research topics for field i at the next time step t+1. The hidden state h and the influence context vector are concatenated and fed to the softmax predictor as follows:where W and b are parameters.

4.4. Training of the Topic Prediction Model

We use the generalization of multinomial logistic loss as the objective function as in equation (13), which minimizes the Kullback–Leibler divergence [46] between the predicted topic word distribution and the real word distribution f.where s refers to a specific research field and m is a research field related to s. The model is trained by minimizing the loss of research topic sequences of all research fields. We use the backpropagation algorithm to optimize the parameters.

5. Experiments Results and Analysis

5.1. Dataset and Preprocessing

We crawl the data of arXiv1 from 2006 to 2020 in all fields of computer science subject, with a total of 319078 papers. We abstract the papers from five fields: Computation and Language (CL), Computer Vision and Pattern Recognition (CV), Machine Learning (ML), Information Retrieval (IR), and Artificial Intelligence (AI). The title of a paper can best reflect the topic of a paper. So, we only use the title of each paper as the text to extract topic words to train the topic prediction model. Specifically, we first remove stop words for papers of each research field, then count the frequency of every word appearing in each research field, and finally use the words with a frequency greater than 1 as topics. The statistical data are shown in Table 1.

Table 1

The statistics of datasets.

Field	Total papers	Period time
Computation and Language (CL)	22633	2006–2020
Computer Vision and Pattern Recognition (CV)	50465	2006–2020
Machine Learning (ML)	72072	2006–2020
Information Retrieval (IR)	28154	2006–2020
Artificial Intelligence (AI)	8667	2006–2020

5.2. Evaluation Metrics

In order to evaluate the prediction performance of the model, the real topic words and the predicted topic words are evaluated based on the following metrics:

Root Mean Squared Error (RMSE). RMSE is the root mean squared error on the test set.where i stands for the research field, t stands for year, c is the real distribution of topic word of field i at year t, and is the predicted distribution of research topic words for field i at year t.

Precision@n. In the predicted n topic words, the correct probability of prediction is as follows:where tr is the number of topic words predicted correctly and fr is the number of topic words predicted incorrectly.

5.3. Compared Methods

We compare our method with four kinds of the prediction method. The first kind of prediction method is the classical time series prediction method ARIMA [47]. The second kind of prediction method is the topic prediction method based on Recurrent Neural Network LSTM and GRU. The third kind of prediction method is encoder-decoder-based research topic prediction that we refer to literature [48]. Encoder-decoder-based research topic prediction includes encoder-decoder (ENDE) [49], DARNN [50], and Temp-Attn-RNN [32]. The fourth kind of prediction method is a topic prediction method based on correlated neural influence (CONI) modeling [19].

Classical time series prediction method.

ARIMA. ARIMA is a widely used time series prediction method. For each research field, the frequency dynamics of each topic word at each year is regarded as the time series, and the ARIMA individually predicts the frequency of each word at the next year.

Prediction method based on Recurrent Neural Network.

LSTM. Topic prediction model based on LSTM models the research topics of every year of each field into time series and uses gated units to capture long-term dependencies in the process of topic prediction.

GRU. Topic prediction model based on GRU merges different gated units of LSTM and also combines the cell state and the hidden state, which leads to fewer parameters and easy convergence and is suitable for scenarios with smaller amounts of data.

Prediction method based on encoder-decoder.

ENDE. This method was originally used for machine translation, and we deployed it to predict research topics of different fields. It encodes field topics into fixed-length vectors, and the decoder is responsible for predicting future research topics.

DARNN. DARNN is a dual-stage attention-based RNN encoder-decoder for single-step time series prediction. It employs multilayer perceptron as attention to capture spatial correlations and long-term dependencies.

TARNN. Based on the encoder-decoder method, a temporal attention mechanism is employed on the hidden states of the encoder to obtain and learn more robust temporal relationships.

Topic prediction method based on correlated neural influence modeling.

CONI. Correlated neural influence (CONI) modeling can integrate the scientific influence of the related field and jointly model the topic evolution of all related fields in a unified Recurrent Neural Network framework. We use LSTM to model topic time series of different fields.

5.4. Experiment Settings

We treat the data of the first 2006–2019 as training set and the 2020 years' papers as testing set. In the process of training the model, we use the data from 2006 to 2018 to predict the data of 2019 to train the research topic prediction model. We removed the stop words from all the data. The word embedding is pretrained based on the 319078 papers of all fields of computer science. The implementation of Word2Vec is employed. In particular, we employ skip-gram with setting the dimension to 100, window size to 5, minimum count to 5, and a subsampling threshold of 10−2. The skip-gram model is trained for 5 iterations on the target corpus. The proposed network was implemented using the PyTorch framework. Adam optimizer is used to train the network. We adopt dropout technology to prevent overfitting. The other parameters are settled for their best performances in experiments.

5.5. Comparisons of Different Topic Prediction Models

In this section, we give the prediction results of the traditional time series prediction model ARIMA, the topic prediction model based on Recurrent Neural Networks LSTM and GRU, the encoder-decoder-based topic prediction models ENDE, TARNN, and DARNN, and the topic prediction model CONI. The topic prediction precision, RMSE, and average precision, average RMSE of baselines, and the proposed method SASC on the five research fields are shown in Tables 2 and 3.

Table 2

RMSE of different models.

Model	RMSE
	CL	CV	ML	IR	AI	Average
ARIMA	1.179e−4	2.967e−4	5.822e−4	3.640e−4	3.893e−4	3.500e−4
GRU	4.385e−4	2.899e−4	2.411e−4	1.792e−4	2.122e−4	2.720e−4
LSTM	4.470e−4	2.879e−4	1.312e−4	2.032e−4	1.987e−4	2.540e−4
ENDE	4.317e−4	2.944e−4	2.306e−4	2.034e−4	2.127e−4	2.750e−4
TARNN	4.375e−4	3.013e−4	2.385e−4	2.050e−4	2.094e−4	2.780e−4
DARNN	4.319e−4	2.923e−4	2.331e−4	2.070e−4	2.046e−4	2.740e−4
CONI	4.196e−4	2.558e−4	2.377e−4	2.049e−4	1.942e−4	2.620e−4
SASC	3.981e−4	2.384e−4	0.970e−4	1.577e−4	0.701e−4	1.920e−4

Table 3

RMSE and precision of different models of different fields.

(a) Precision of different models of CL field
	p@10	p@20	p@30	p@40	p@50	p@60	p@70	p@80
ARIMA	0.6000	0.5000	0.5333	0.5000	0.4600	0.4667	0.4857	0.4750
GRU	0.7000	0.8000	0.8000	0.7250	0.7200	0.6500	0.6571	0.6750
LSTM	0.7000	0.8000	0.7667	0.7750	0.7200	0.7167	0.7000	0.7250
ENDE	0.6000	0.6000	0.6000	0.5500	0.5800	0.5667	0.5286	0.5500
TARNN	0.6000	0.6000	0.6000	0.5500	0.5600	0.5500	0.5286	0.5375
DARNN	0.6000	0.6000	0.6000	0.5500	0.5800	0.5667	0.5286	0.5375
CONI	0.7000	0.8000	0.7667	0.7250	0.6800	0.6333	0.6286	0.6750
SASC	0.7000	0.9000	0.9000	0.8500	0.8000	0.8167	0.8143	0.8000
(b) Precision of different models of CV field
ARIMA	0.8000	0.6000	0.6000	0.6250	0.5400	0.5500	0.5714	0.6250
GRU	0.9000	0.6000	0.6667	0.6250	0.5600	0.5667	0.6000	0.6375
LSTM	0.9000	0.6000	0.6667	0.6250	0.5800	0.5833	0.6000	0.6375
ENDE	0.9000	0.6000	0.6667	0.6250	0.5800	0.5667	0.6000	0.6375
TARNN	0.9000	0.6000	0.6667	0.6250	0.5600	0.5833	0.6000	0.6250
DARNN	0.9000	0.6000	0.6667	0.6250	0.5800	0.5667	0.5857	0.6500
CONI	0.9000	0.6500	0.7000	0.6750	0.6800	0.7000	0.6857	0.7000
SASC	1.0000	0.9000	0.8333	0.8750	0.8400	0.8333	0.8714	0.8500
(c) Precision of different models of ML field
ARIMA	0.6000	0.5500	0.5333	0.6000	0.5800	0.5500	0.5286	0.6125
GRU	0.7000	0.6000	0.6000	0.7000	0.6200	0.5500	0.6143	0.6250
LSTM	0.8000	0.8500	0.7667	0.7750	0.7400	0.7500	0.7714	0.7875
ENDE	0.7000	0.6000	0.6000	0.7000	0.6200	0.5667	0.6143	0.6250
TARNN	0.7000	0.6000	0.6000	0.7000	0.6200	0.5833	0.6143	0.6375
DARNN	0.7000	0.6000	0.6000	0.7000	0.6200	0.5833	0.6143	0.6250
CONI	0.7000	0.6000	0.6000	0.7000	0.6200	0.5833	0.6143	0.6250
SASC	1.0000	0.9000	0.9000	0.9000	0.9000	0.9000	0.8714	0.9250
(d) Precision of different models of IR field
ARIMA	0.7000	0.5500	0.6333	0.7000	0.7000	0.7000	0.7000	0.7250
GRU	0.8000	0.6000	0.7000	0.8000	0.7600	0.8167	0.8000	0.7875
LSTM	0.8000	0.6000	0.6333	0.7250	0.7600	0.7500	0.7571	0.7625
ENDE	0.8000	0.5500	0.6667	0.7000	0.7600	0.7000	0.7429	0.7625
TARNN	0.8000	0.6000	0.6667	0.7250	0.7600	0.7167	0.7429	0.7500
DARNN	0.8000	0.5500	0.6333	0.7250	0.7600	0.7167	0.7429	0.7500
CONI	0.8000	0.5500	0.6333	0.7250	0.7600	0.7167	0.7625	0.7556
SASC	0.8000	0.7500	0.8000	0.8000	0.8000	0.8333	0.8143	0.8000
(e) Precision of different models of AI field
ARIMA	0.5000	0.5500	0.5000	0.5750	0.6000	0.6667	0.6857	0.6375
GRU	0.7000	0.6500	0.5000	0.5750	0.6200	0.6833	0.7000	0.6750
LSTM	0.6000	0.7000	0.5000	0.5750	0.6200	0.6667	0.7000	0.6625
ENDE	0.6000	0.6500	0.5333	0.5750	0.6200	0.6500	0.7000	0.6500
TARNN	0.6000	0.6500	0.5333	0.5750	0.6200	0.6500	0.6714	0.6750
DARNN	0.6000	0.6500	0.5000	0.5750	0.6200	0.6833	0.7000	0.6500
CONI	0.6000	0.6500	0.5000	0.6000	0.6400	0.6667	0.7000	0.6500
SASC	0.9000	0.8500	0.8333	0.9000	0.9000	0.8667	0.8429	0.8250
(f) Average precision of different models of five fields
ARIMA	0.6400	0.5500	0.5600	0.6000	0.5760	0.5867	0.5943	0.6150
GRU	0.7600	0.6500	0.6533	0.6850	0.6560	0.6533	0.6743	0.6800
LSTM	0.7600	0.7100	0.6667	0.6950	0.6840	0.6933	0.7057	0.7150
ENDE	0.7200	0.6000	0.6133	0.6300	0.6320	0.6100	0.6371	0.6450
TARNN	0.7200	0.6100	0.6133	0.6350	0.6240	0.6167	0.6314	0.6450
DARNN	0.7200	0.6000	0.6000	0.6350	0.6320	0.6233	0.6343	0.6425
CONI	0.7400	0.6500	0.6400	0.6850	0.6760	0.6600	0.6714	0.6825
SASC	0.8800	0.8600	0.8533	0.8650	0.8480	0.8500	0.8429	0.8400

Table 2 shows the RMSE value and average RMSE value of our proposed method SASC and baselines in five research fields of the computer science discipline. It can be seen from the table that the RMSE value and average RMSE value of SASC in all research fields are the smallest, except that the RMSE value of SASC in the CL field is not as good as ARIMA. It can be concluded that, in the process of training and optimization of our proposed model SASC, the distribution of the predicted topics gradually approaches the topic distribution of the real research field. This indicates that the topic prediction model we proposed is effective.

Table 3 shows the topic prediction precision and average precision of ARIMA, GRU, LSTM, ENDE, TARNN, DARNN, CONI, and SASC. It can be concluded from the table that the precision of the topic prediction model based on Recurrent Neural Network is significantly higher than ARIMA, which indicates that topic sequence modeling using Recurrent Neural Network is helping to improve the precision of topic prediction. Furthermore, the precision of the topic prediction model based on the Recurrent Neural Network is better than that based on encoder-decoder. The precision of the topic prediction model based on correlated neural influence (CONI) modeling is similar to the topic prediction model based on Recurrent Neural Network. The precision of SASC greatly exceeds the topic prediction model based on correlated neural influence modeling and Recurrent Neural Network. The difference between CONI and RNN-based topic prediction models is that CONI considers that the research topic of a field is affected by its related fields and models the scientific influence context. The difference between SASC and CONI is that SASC not only considers the context of scientific influence in related fields but also considers the consistency of subject space in different fields. That is, different fields have different topic spaces and need to be modeled separately. This indicates that although CONI considers scientific influence context modeling, the topic prediction precision of which is not greatly improved because it does not consider that research topics in different fields should belong to different topic space. SASC is effective in predicting research topics of different fields by employing multiple different RNN chains to capture topics of different research fields and using spatial attention mechanisms to model the representation of field topics and mapping different field topics to a unified semantic space to obtain scientific influence context.

Next, we report the change curve of average precision (Precison@10, Precison@20, Precison@40, and Precison@60) and average RMSE of five research fields of the topic prediction model ARIMA, GRU, LSTM, ENDE, TARNN, DARNN, CONI, and SASC with the increasing number of iterations. The change curve is shown in Figure 5.

Figure 5

The change curve of precision of different prediction models. (a) Precision@10, (b) Precision@20, (c) Precision@40, and (d) Precision@60.

As can be seen from Figure 5(a), at the beginning of model training, the topic prediction precision of each model shows a trend of rapid improvement. When the number of iterations reaches a certain number, the topic prediction precision of SASC is still improving, while the precision of other prediction models is stable. Figures 5(b), 5(c), and 5(d) reflect the same rule as Figure 5(a). It can be concluded that SASC combined with semantic consistency scientific modeling and spatial attention field topic representation has higher prediction precision.

Figure 6 shows the change of the average RMSE of each topic prediction model in five research fields with the increase of the number of iterations. It can be seen from Figure 6 that, with the increase of the number of iterations, the average RMSE of each model of five fields shows a downward trend, and the RMSE of SASC decreases the fastest. It shows that our model SASC has a good performance in topic prediction.

Figure 6

The change curve of RMSE of different prediction models.

5.6. Ablation Study

To further validate the effectiveness of SASC, we make comparisons with variants of SASC as follows:

SASC without spatial attention (SASC-SA): to evaluate the effect of multi-RNN field topic representation based on semantic consistency-based scientific influence modeling on model performance, we evaluate the performance of a variant of SASC that does not use spatial attention when predicting research topics. By removing the spatial attention, the model is not able to distinguish the influencing factors of field research topic representation. This model employs multiple RNN chains to represent different field topic and maps the research topic of each field to a consistent semantic space, and then the scientific influence context among fields is modeled by calculating topic similarity. We refer to this model as SASC-SA.

SASC without semantic consistency (SASC-SC): to evaluate the effect of field topic representation based on spatial attention on model performance, we evaluate the performance of a variant of SASC that does not use multi-RNN field topic representation based on semantic consistency-based scientific influence modeling when modeling scientific influence context of related fields. This model uses spatial attention to distinguish the importance of attributes of field research topic. We refer to this model as SASC-SC.

We compare the precision of SASC, SASC-SA, and SASC-SC in each field and the average precision of the five fields. The experimental results are shown in Figure 7. At the same time, we also compare the RMSE of SASC, SASC-SC, and SASC-SA in each research field and the average RMSE of the five fields. The experimental results are shown in Table 4.

Figure 7

Precision comparison of SASC with two of its variants. (a) CL, (b) CV, (c) ML, (d) IR, (e) AI, and (f) AVERAGE.

Table 4

RMSE of SASC and its variants.

Model	RMSE
	CL	CV	ML	IR	AI	Average
SASC	3.981e−4	2.384 e−4	0.970 e−4	1.577 e−4	0.701 e−4	1.920 e−4
SASC-SA	4.084 e−4	2.439 e−4	1.521 e−4	1.615 e−4	1.063 e−4	2.140 e−4
SASC-SC	4.668 e−4	2.447 e−4	0.914 e−4	1.583 e−4	1.011 e−4	2.120 e−4

Figure 7 shows the precision comparison of SASC with two of its variants. The performance of both SASC-SC and SASC-SA is worse than SASC. We believe that SASC-SC uses spatial attention to distinguish different importance of each attribute of field topics. But it is not able to solve the problem that the space of research topics of different research fields is inconsistent. So, the performance of SASC-SC is worse than SASC. SASC-SA first employs multiple RNN to model different research fields and maps research topics of these different fields to a consistent and comparable semantic space. So, the scientific influence context can be obtained by calculating the topic similarity among research fields. However, it ignores the importance of different attribute of field topics on the expression of field topics, so the precision of topic prediction is worse than SASC. SASC uses spatial attention to distinguish the importance of field topic attributes on topic expression, employs multiple RNN chains to distinguish research topics of different fields, and models the scientific influence context based on semantic consistency of topic so as to obtain the best topic prediction performance.

Table 4 shows RMSE and average RMSE of topic prediction model SASC with two of its variants of five research fields. The RMSE and average RMSE of both SASC-SC and SASC-SA are higher than the full model SASC. This further shows the effectiveness of our model SASC.

5.7. Case Study: Effectiveness of Research Topic Trend Prediction

In this part, we use the best topic prediction model SASC to predict the research topics of three fields in 2020 and give the true topic words in 2020. As shown in Table 5, it can be seen that our model has high precision in topic prediction compared with the real topics in five research fields in 2020.

Table 5

Predicted top 10 research topics of CL, CV, ML, IR, and AI fields in 2020.

Field	Predicted research topics	True research topics
CL	Neural, speech, semantic, learning, networks, evaluation, classification, recognition, detection, and extraction	Learning, neural, generation, translation, speech, knowledge, recognition, detection, classification, and extraction
CV	Learning, networks, convolutional, neural, recognition, detection, segmentation, estimation, adversarial, and classification	Learning, detection, neural, segmentation, networks, recognition, classification, adversarial, convolutional, and estimation
ML	Learning, neural, networks, adversarial, detection, reinforcement, classification, prediction, graph, and optimization	Learning, neural, networks, reinforcement, detection, graph, adversarial, classification, optimization, and prediction
IR	Ranking, semantic, learning, recommendation, search, neural, retrieval, information, query, and analysis	Recommendation, semantic, learning, ranking, retrieval, search, embedding, information, query, and recommender
AI	Learning, networks, language, machine, deep, neural, reinforcement, data, model, and knowledge	Learning, machine, reinforcement, deep, language, neural, topic, networks, knowledge, and model

6. Conclusion

In this paper, we employ multiple different RNN chains which model different field research topic and propose a research topic prediction model based on spatial attention and semantic consistency-based scientific influence modeling. Based on the Recurrent Neural Network topic feature sequence modeling method, spatial attention is employed to distinguish the importance of different topic characteristics of a research field to express the fine-grained research topic of a field. Based on the representation of topics in different research fields, semantic consistency-based scientific influence modeling is used to map research topics of different fields into a comparable feature space to improve the quality of scientific influence context. Specifically, research topics in different research fields are in different semantic spaces, and they are mapped to the consistent semantic space to model interactive scientific influence context. The experimental results on the five research fields in the computer science discipline demonstrate the effectiveness of our proposed model.