| Literature DB >> 34960530 |
Metaknowledge Enhanced Open Domain Question Answering with Wiki Documents.
Shukan Liu1,2, Ruilin Xu2, Li Duan2, Mingjie Li3, Yiming Liu2.
Abstract
The commonly-used large-scale knowledge bases have been facing challenges in open domain question answering tasks which are caused by the loose knowledge association and weak structural logic of triplet-based knowledge. To find a way out of this dilemma, this work proposes a novel metaknowledge enhanced approach for open domain question answering. We design an automatic approach to extract metaknowledge and build a metaknowledge network from Wiki documents. For the purpose of representing the directional weighted graph with hierarchical and semantic features, we present an original graph encoder GE4MK to model the metaknowledge network. Then, a metaknowledge enhanced graph reasoning model MEGr-Net is proposed for question answering, which aggregates both relational and neighboring interactions comparing with R-GCN and GAT. Experiments have proved the improvement of metaknowledge over main-stream triplet-based knowledge. We have found that the graph reasoning models and pre-trained language models also have influences on the metaknowledge enhanced question answering approaches.Entities:
Keywords: graph modeling; graph neural networks; knowledge graph; metaknowledge; question answering
Mesh：
Year: 2021 PMID： 34960530 PMCID： PMC8703336 DOI： 10.3390/s21248439
Source DB: PubMed Journal: Sensors (Basel) ISSN： 1424-8220 Impact factor: 3.576
1. Introduction
2. Related Work
2.1. Knowledge Base Question Answering (KBQA)
2.2. Graph Neural Networks for Graph Embedding
3. Approach
Figure 1An overview of our approach. There are basically four steps for our approach: (1) generating metaknowledge; (2) building a metaknowledge network; (3) graph modeling and encoding; and (4) graph reasoning.
3.1. Generating Metaknowledge
, , , , , which represent the title, section titles, summary, or paragraphs (referred to as metaknowledge hierarchical elements in Ref. [7]).
Suppose Wiki document , where denotes the paragraphs set in the document ( is the total number of paragraphs); denotes the hierarchical elements, then each paragraph hierarchically belongs to their upper hierarchical elements (e.g., section titles). Furthermore, this work extracts entities and relations paragraph by paragraph using Stanza [30] and OpenNRE [31], then links the metaknowledge semantic elements to hierarchical elements with a document structure (Figure 2).
Figure 2An example of metaknowledge extracted from Wiki documents. The number on the arrows indicates the hierarchical levels of the relations, which are also the weight of edges.
Each document metaknowledge is saved as a JSON file converted from Python dictionary (denoted as metaknowledge dictionary), the data structure is shown in Listing 1.
Listing 1Denotations of keys in metaknowledge dictionary.
Denotations of keys in metaknowledge dictionary.
The weights of hierarchical vertices and edges are set as negative, which are the opposite number of their level, for instance, weights of the 1st-level hierarchical vertices are −1, and the 2nd-level vertices are −2. In contrast, the semantic vertices and edges are set as positive, which are the exact level of the hierarchical vertices they belong to.
The denotations of keys in metaknowledge dictionary are shown in Table 1.
Table 1Denotations of keys in metaknowledge dictionary.
Entities Relations
ENT_ID
type
content
weight
title
up_id
Entity ID
REL_ID
type
head_ID
tail_ID
weight
Relation ID
3.2. Building Metaknowledge Network
For documents , the semantic association between and is denoted as , then the metaknowledge network built on is denoted as . When building a metaknowledge network from document metaknowledge, to avoid the loss of hierarchy caused by semantic entity fusion, this work only establishes semantic association between hierarchical vertices.
Supposing that are two semantically associated hierarchical vertices in document metaknowledge and , their textual embedding vectors are:
where denotes the pre-trained language models (PLMs), such as BERT [32], RoBERTa [33], XLNet [34], etc.
Then, we use cosine similarity to calculate the semantic association between two hierarchical vertices:
To decide the appropriate tolerance threshold, we uses BERT as the PLM. Taking “South Africa” as the keyword, this work retrieves 10 relevant Wiki documents using Wikipedia Search. Through hand-craft selection, 78 groups of associated metaknowledge hierarchical vertices are picked up. The cosine similarity of semantic embedding in each group is encoded by BERT. Then, the statistical results of this test are shown in Figure 3.
Figure 3Test to decide metaknowledge association tolerance.
Figure 3 indicates that the cosine similarity between associated hierarchical vertices is basically in the range of ; therefore, this work adopts .
Meanwhile, we use S-MART (https://github.com/kkteru/r-gcn (accessed on 9 December 2021)) to obtain relevant entities to q, called seed entities . Then, a retrieval starts in order to find the directly connected semantic vertices and hierarchical vertices , and the latter extends to the top level hierarchical vertex (see also Knowledge Retriever in Figure 1). Then, we get the question subgraph :
3.3. Metaknowledge Encoding
In this work, we propose a Graph Encoder for Metaknowledge (GE4MK) to encode the text-described document metaknowledge. For document metaknowledge , the features of each vertex could be divided into three parts: (1) the semantic features of itself, including its textual content and its entity type ; (2) the hierarchical features of itself, including the semantic features of the upper hierarchical vertex that belongs to, and the title’s semantic features ; (3) the semantic features of relations between and its k nearest 1-hop neighboring vertices.
Consequently, the vertex features of can be described as:
where
The output of these PLMs is a -dimensional dense semantic vector. The in Equation (3) indicates a 2-layer MLP, which transforms the concatenation of from to , D indicates the dimension of feature space which is manually set depending on the using PLM; for instance, in this work, we set . and are linear transformations, where in Stanza and in .
For the convenience of calculation, we use matrices to describe all the vertices and edges (also the entities and relations) features in the document metaknowledge , so the isolated vertices features (ignoring its neighbors) are
and the type features of vertices are:
Meanwhile, the relation type features are denoted as:
where ; for vertex (entity) x, if y is one of the k nearest one-hop neighbors, then indicates the type number in of the relation between vertices x and y; otherwise,
Therefore, for all the vertices in , their features are:
where concat indicates concatenation by column, and indicates a linear transformation from to in Equation (8).
When considering the semantic information in relations, we define the Semantic Relation Matrix of as:
Using to indicate the adjacency matrix of , then:
where denotes GE4MK, and only includes semantic relation, not hierarchical relations. Then, we use GE4MK to encode from text-described data to matrices:
3.4. Graph Reasoning: MEGr-Net
Inspired by R-GCN[rgcn] and GAT[gat], according to the complex semantic and hierarchical relations, this work proposes a graph-attention-based model MEG-Net (Metaknoledge Enhanced Graph reasoning Network) in order to perform reasoning on the question subgraph (Figure 4).
Figure 4The attention mechanism of MEGr-Net. (a) Self-attention; (b) attention aggregation.
Relational Graph Attention Layer (R-GAL) is the basic part of MEGr-Net, and the output is the vertex state features under k-heads attention influence. We denote the total number of vertices as , the vertex features input to R-GAL as (F is the dimension of vertex state space), and the relations as ( is the dimension of edge state space).
We firstly consider the interaction between vertex and its neighbors (attention heads). The semantic relation matrix from is transformed into relation features matrix :
where indicates deleting all the empty relations, indicate the neighbors of .
The attention mechanism is denoted as ; then, we calculate the attention coefficients:
where is the vertex weight matrix, and is the edge weight matrix. These two matrices realize the parallel computation of linear transformation on each vertex. indicates the interaction of the relation between vertex and its neighbor , as well as itself (self-attention). In MEGr-Net, the masked-attention mechanism is used to distribute the attention interaction to the neighbors of , so the masked-attention coefficient is:
The MEGr-Net sets the attention mechanism as a single-layer feed forward network (FFN) with parameters and LeakyReLU activate function, then:
Next, updating the vertex features of :
where is a nonlinear function, and we use the ELU in MEGr-Net.
When considering the multi-heads attention, we have:
where indicates the concatenation of k vertex features of under its k-neighbors attention interaction.
In the last R-GAL, we calculate the average features instead of concatenating, and use the logistic sigmoid to normalize the output features into as the probability that indicates vertex is the answer entity:
For the efficiency of computation, we use matrices in MEGr-Net to describe the whole progress: First, the vertex features matrix of question subgraph is multiplied by the vertex weight matrix for state space transformation. Then, an all-combination is used to concatenate and in Equation (16):
We do the same operation to :
Then, the attention coefficient vector:
Updating the vertices’ state:
and aggregating:
Finally, the logistic sigmoid:
The vector indicates the probability that each node in the question subgraph is the correct answer. In other words, the MEGr-Net turns question reasoning into a node classification task, and it picks the vertex whose probability is the highest as the most probable answer.
4. Experiments
To verify the effectiveness of metaknowledge network in open domain question answering, this section carries out experiments on a subset of WebQuestionsSP, analyzes the experimental variables including: (1) triplet-based knowledge and metaknowledge, (2) various graph reasoning models, and (3) several pre-trained language models.
4.1. Datasets and Set-Ups
This work uses the open domain natural language question answering dataset WebQuestionsSP [35] for experimental analysis, which includes 4737 questions in natural language. At present, there is no well-established large-scale metaknowledge base and metaknowledge network, so we have to build it from scratch by the approach designed in Section 3.1 and Section 3.2. For the fact that the entities and relations extracted by open source NLP models naturally have quality disadvantages, the metaknowledge network we build in this work has an innate weakness when comparing with the finely-built large-scale knowledge bases such as FreeBase and WikiData. Consequently, to make hierarchical metaknowledge and non-hierarchical triplet-based knowledge comparable on the same track, considering the data quality limitation, we adopt the general approach in the construction of knowledge graph, that is, deleting all hierarchical nodes and relationships, retaining only semantic entities and relations in metaknowledge and integrating them to form a non-hierarchical triplet-based knowledge network.
Meanwhile, the process of extracting metaknowledge from Wiki documents, constructing a metaknowledge network, retrieving and encoding question subgraphs takes a big amount of time and computing resources. For example, in the previous experiment, it took an average of 2 h for VRAM GPU and 2 × 12 Core, 24 Threads CPU to build a metaknowledge network from five Wiki documents relevant to a question and complete subgraph retrieval and its encoding. Considering the data quality and hardware, this section scaled down the dataset to 2.5% of WebQuestionsSP, that is, 250 questions in natural language. In addition, it was divided into 150 for the training set, 50 for the cross validation set and 50 for the test set. In this section, it is referred to as WebQuestions . The training parameters of MEGr-Net are shown in Table 2.
Table 2The training parameters of MEGr-Net.
Parameters Values Epochs 200 Learning Rate 5 × 10 − 3 Attention Heads k 8 Dimension of Entity Features F ′ 1000 Dimension of Relation Features F r ′ 500 Hidden Units 1000
The semantic encoder is deployed on Server #1. The metaknowledge generation, metaknowledge network construction framework and MEGr-Net are deployed on Server #2 (see also Appendix A). A Tesla V100 GPU (with 32 GB VRAM) is used for training, which takes 13.5 d (325 h).
This work takes the average accuracy (avg. Acc.) as the evaluation index.
4.2. Experimental Control Groups
This section analyzes the impact of different experimental variables on MbQA from the following three aspects:
Hierarchical metaknowledge and non-hierarchical triplet-based knowledge. This is the focus of this section, that is, what improvement hierarchical metaknowledge can make on open domain question answering compared with non-hierarchical triplet-based knowledge—in other words, whether metaknowledge and metaknowledge network have superiority in open domain QA tasks. As described in Section 3.1, considering the extraction quality of open domain entities and relationships by open source NLP models, this section uses the same data and extraction models to build a metaknowledge network (referred to as MK-Net in the experiment) and triplet knowledge base (referred to as Tri-KB) by the metaknowledge structure proposed in the beginning of Section 3 and the general triplet-based knowledge structure, respectively.
Graph reasoning model. MEGr-Net, based on GAT, essentially achieves an improvement of graph data with complex relationships, like metaknowledge. Meanwhile, it partially adopts the relationship processing approach in R-GCN. Therefore, this section takes GAT and R-GCN as test baselines and compares them with MEGr-Net. To explain the impact of (meta)knowledge extraction quality on the results, this section introduces the results of DrQA [21] and GRAFT-Net [22] on the entire WebQuestionsSP as a reference.
Pre-trained language models (PLMs). The input of MEGr-Net is the question subgraph encoded by GE4MK, and its semantic features mainly come from the text embedding vector encoded by the PLM in GE4MK. Therefore, different PLMs may exert different impact on the semantic feature richness of the problem subgraph. This section takes BERT as the baseline and RoBERTa [33] and ALBERT [36] as the control groups.
4.3. Results and Analysis
The results on control group #1 on WebQuestions are shown in Table 3. The results show that, with the same data quality, the hierarchical metaknowledge achieves better results than non-hierarchical triplet knowledge in open domain question answering (+16.9% Tri-KB).
Table 3Results on Control Group #1.
(Meta) Knowledge Network IH-Acc # Tri-KB 0.483
MK-Net
0.652
The WebQuestions dataset which we use in the experiment is part of the whole WebquestionsSP, so we use average In-House Accuracy (IH-Acc.) for avg. Acc.
The results on control group #2 are shown in Table 4. For GAT, the relationship matrix in MEGr-Net and the relationship weight matrix in R-GAL are removed in this section. Modifications have been made to R-GCN for the tasks in this section.
Table 4Results on Control Group #2.
Graph Reasoning Models Acc. Baselines GAT (MK-Net) 0.608 (IH † R-GCN # 0.601 (IH)
MEGr-Net (MK-Net) 0.652 (IH)
DrQA ☆ 0.215
GRAFT-Net ☆ 0.687
Model from https://github.com/kkteru/r-gcn (accessed on 9 December 2021). The data are cited from [22], respectfully. IH indicates that the experiments are based on the dataset WebQuestions , and if not annotated that indicates that the experiments are based on the dataset WebQuestionsSP.
As can be seen from the results, MEGr-Net achieves better performance than the baselines in the reasoning of hierarchical graph data with complex semantic relationships, such as a metaknowledge network (+4.4%GAT, +5.1%R-GCN). Meanwhile, compared with GRAFT-Net, which uses the complete FreeBase as the knowledge base and integrates the document (doc) and KB features, MEGr-Net still lags behind, indicating that it still needs to be improved in MbQA, especially in the integration with MRC method (see also Section 4).
The results on control group #3 are shown in Table 5 (see Appendix B for the source of the pre-training parameter file of the pre-training language model). From the results, the PLMs (ALBERT , RoBERTa ) with large-scale parameters perform better, indicating that the larger the PLMS used by the graph encoder, the finer the fine tuning and the richer the semantic features of the question subgraph, the better performance will be achieved in MbQA.
Table 5Results on Control Group #3.
MEGr-Net PLMs IH-Acc. Baseline +BERT B A S E 0.652 +ALBERT B A S E 0.646 +BERT L A R G E 0.670 +RoBERTa L A R G E 0.692 +ALBERT X X L A R G E
0.708
As shown in Figure 5, the combination of MEGr-Net and ALBERTXXLARGE achieved the best results (+5.6% MEGr-Net+BERT ) and gained better performance than GRAFT-Net using LSTM [lstm] as a text encoder, which proves that PLMS based on Transformers [28] is better than LSTM in MbQA.
Figure 5Results of graph reasoning models and PLMs in MbQA.
Generally, the metaknowledge network with document directory hierarchy can significantly improve the existing methods in KBQA, which is basically consistent with the view that titles play a positive role in question answering in [22]. Meanwhile, finer PLMs can improve the semantic feature representations of question subgraphs and achieve better results in question answering. This is also consistent with the view and experimental results in [37].
5. Discussion
From the overall results of this work, metaknowledge basically solves the problems of triplet-based knowledge with weak structural logic, and provides a new idea for the theoretical and practical research of knowledge engineering. Meanwhile, it must also be noted that, as a relatively new research field, there are still some urgent problems that need to be solved in the future work.
5.1. Metaknowledge and Metaknowledge Network Modeling
The metaknowledge and metaknowledge network modeled by the single dimension network in this work (Section 3.2 and Section 3.3) is a compromising strategy to reduce the complexity of the model under the current realistic conditions of mainstream GNN models. In fact, according to the our concept, the metaknowledge network should be a multi-dimensional hyper-graph with hierarchical structure (Figure 6). The metaknowledge network expressed by that type of graph model includes two dimensions: hierarchical dimension and semantic dimension. The hierarchical dimension is in the outer layer, which includes all hierarchical nodes and relationships; the semantic dimension is in the inner layer, which includes all semantic nodes and relationships subordinate to the hierarchical nodes. Ref. [38] proposes an embedding framework MINES for multi-dimensional networks with hierarchical structure, which uses a hierarchical structure for multi-dimensional network embedding; Ref. [37] proposes an open domain question answering method based on hyper-edge fusion. These documents show the feasibility of graph reasoning on the metaknowledge network expressed by the hierarchical multi-dimensional hyper-graph. This metaknowledge modeling method needs to be further studied and explored.
Figure 6Metaknowledge that modeled by multi-dimensional hyper-graph with hierarchical structure.
5.2. MbQA and Graph Reasoning
Limited by the extraction effect of open-source NLP models, MbQA has insufficient advantages over KBQA. At least under the existing conditions, there is still a huge gap between the metaknowledge network and mutual large-scale knowledge bases such as freebase from the perspective of data quality. Therefore, MbQA may be more suitable for in-domain QA tasks (such as question answering on laws and regulations). The fine-tuned NLP models will significantly improve the extraction quality of metaknowledge semantic elements. At the same time, the structural logic of metaknowledge network makes it have the ability to deal with complex relationships. Therefore, the role of the metaknowledge network in multi-hop QA tasks is also a direction worthy of research. In terms of graph reasoning models for question answering tasks, MEGr-net relies on the hierarchical features contained in metaknowledge to supplement the short board relying only on semantic features (KBQA). On this basis, documents [22] and pre-trained language models [39] can continue to be integrated into graph reasoning to select the best from the best and enhance the effect of MbQA.
In general, the metaknowledge enhanced question answering is a brand-new method for solving the problem caused by triplet-based knowledge, and it improves the capability of current knowledge bases (which are also the knowledge engines of intelligent voice interaction devices). In the foreseeable future, this method could be the antidote to help intelligent voice devices get rid of the problems that they are not so good when answering complex questions asked by users, and make great progress in the interaction with human users.
6. Conclusions
Facing the problems in current open domain QA tasks caused by the loose knowledge association and weak structural logic of triplet-based knowledge, this work makes pivotal innovations on metaknowledge enhance question answering: (1) Metaknowledge extraction and metaknowledge network construction, where we present the approach of generating metaknowledge and building metaknowledge network from Wiki documents automatically. (2) Metaknowledge and metaknowledge network modeling, where we generally consider several different kinds of features from reasoning performance related aspects including semantic features such as textual content, entity type, relations, and along with hierarchical features. (3) MEGr-Net, which is proposed for question answering, which aggregates both relational and neighboring interactions compared with R-GCN and GAT. Experiments have proved the improvement of metaknowledge over main-stream triplet-based knowledge. We have found that the graph reasoning models and pre-trained language models also have influences on the metaknowledge enhanced question answering approaches.
Table A1Hardware and software environments of Server#1.
Server #1: Providing BERT Embedding Service Hard-ware Env. CPU 2 × Intel Xeon E5-2678 v3 (48) @ 3.300 GHz RAM 32 GB GPU 4 × NVIDIA GV102 (11 GB VRAM) Software Env. OS Ubuntu 18.04.5 LTS Python Python 3.6.5: Anaconda PyTorch 1.6.0 (for GPU) TensorFlow 1.15.0 (for GPU)
Table A2Hardware and software environments of Server#2.
Server #2: Main Experimental Environment Hard-ware Env. CPU 2 × Intel Xeon Silver 4210R (40) @ 3.200 GHz RAM 256 GB GPU 4 × NVIDIA Tesla V100S (32 GB VRAM, using 1) Software Env. OS Ubuntu 20.04.2 LTS Python Python 3.7.7: Anaconda PyTorch 1.9.0 (for GPU) TensorFlow 1.15.0 (for GPU)
3 in total
1. Metaknowledge.
Journal: Science
Date: 2011-02-11 Impact factor: 47.728
2. Multi-Modal Explicit Sparse Attention Networks for Visual Question Answering.
Journal: Sensors (Basel)
Date: 2020-11-26 Impact factor: 3.576
3 in total
, , , which represent the title, section titles, summary, or paragraphs (referred to as metaknowledge hierarchical elements in Ref. [7]).
Suppose Wiki document , where denotes the paragraphs set in the document ( is the total number of paragraphs); denotes the hierarchical elements, then each paragraph hierarchically belongs to their upper hierarchical elements (e.g., section titles). Furthermore, this work extracts entities and relations paragraph by paragraph using Stanza [30] and OpenNRE [31], then links the metaknowledge semantic elements to hierarchical elements with a document structure (Figure 2).
Figure 2An example of metaknowledge extracted from Wiki documents. The number on the arrows indicates the hierarchical levels of the relations, which are also the weight of edges.
Each document metaknowledge is saved as a JSON file converted from Python dictionary (denoted as metaknowledge dictionary), the data structure is shown in Listing 1.
Listing 1Denotations of keys in metaknowledge dictionary.
Denotations of keys in metaknowledge dictionary.
The weights of hierarchical vertices and edges are set as negative, which are the opposite number of their level, for instance, weights of the 1st-level hierarchical vertices are −1, and the 2nd-level vertices are −2. In contrast, the semantic vertices and edges are set as positive, which are the exact level of the hierarchical vertices they belong to.
The denotations of keys in metaknowledge dictionary are shown in Table 1.
Table 1Denotations of keys in metaknowledge dictionary.
Entities Relations
ENT_ID
type
content
weight
title
up_id
Entity ID
REL_ID
type
head_ID
tail_ID
weight
Relation ID
3.2. Building Metaknowledge Network
For documents , the semantic association between and is denoted as , then the metaknowledge network built on is denoted as . When building a metaknowledge network from document metaknowledge, to avoid the loss of hierarchy caused by semantic entity fusion, this work only establishes semantic association between hierarchical vertices.
Supposing that are two semantically associated hierarchical vertices in document metaknowledge and , their textual embedding vectors are:
where denotes the pre-trained language models (PLMs), such as BERT [32], RoBERTa [33], XLNet [34], etc.
Then, we use cosine similarity to calculate the semantic association between two hierarchical vertices:
To decide the appropriate tolerance threshold, we uses BERT as the PLM. Taking “South Africa” as the keyword, this work retrieves 10 relevant Wiki documents using Wikipedia Search. Through hand-craft selection, 78 groups of associated metaknowledge hierarchical vertices are picked up. The cosine similarity of semantic embedding in each group is encoded by BERT. Then, the statistical results of this test are shown in Figure 3.
Figure 3Test to decide metaknowledge association tolerance.
Figure 3 indicates that the cosine similarity between associated hierarchical vertices is basically in the range of ; therefore, this work adopts .
Meanwhile, we use S-MART (https://github.com/kkteru/r-gcn (accessed on 9 December 2021)) to obtain relevant entities to q, called seed entities . Then, a retrieval starts in order to find the directly connected semantic vertices and hierarchical vertices , and the latter extends to the top level hierarchical vertex (see also Knowledge Retriever in Figure 1). Then, we get the question subgraph :
3.3. Metaknowledge Encoding
In this work, we propose a Graph Encoder for Metaknowledge (GE4MK) to encode the text-described document metaknowledge. For document metaknowledge , the features of each vertex could be divided into three parts: (1) the semantic features of itself, including its textual content and its entity type ; (2) the hierarchical features of itself, including the semantic features of the upper hierarchical vertex that belongs to, and the title’s semantic features ; (3) the semantic features of relations between and its k nearest 1-hop neighboring vertices.
Consequently, the vertex features of can be described as:
where
The output of these PLMs is a -dimensional dense semantic vector. The in Equation (3) indicates a 2-layer MLP, which transforms the concatenation of from to , D indicates the dimension of feature space which is manually set depending on the using PLM; for instance, in this work, we set . and are linear transformations, where in Stanza and in .
For the convenience of calculation, we use matrices to describe all the vertices and edges (also the entities and relations) features in the document metaknowledge , so the isolated vertices features (ignoring its neighbors) are
and the type features of vertices are:
Meanwhile, the relation type features are denoted as:
where ; for vertex (entity) x, if y is one of the k nearest one-hop neighbors, then indicates the type number in of the relation between vertices x and y; otherwise,
Therefore, for all the vertices in , their features are:
where concat indicates concatenation by column, and indicates a linear transformation from to in Equation (8).
When considering the semantic information in relations, we define the Semantic Relation Matrix of as:
Using to indicate the adjacency matrix of , then:
where denotes GE4MK, and only includes semantic relation, not hierarchical relations. Then, we use GE4MK to encode from text-described data to matrices:
3.4. Graph Reasoning: MEGr-Net
Inspired by R-GCN[rgcn] and GAT[gat], according to the complex semantic and hierarchical relations, this work proposes a graph-attention-based model MEG-Net (Metaknoledge Enhanced Graph reasoning Network) in order to perform reasoning on the question subgraph (Figure 4).
Figure 4The attention mechanism of MEGr-Net. (a) Self-attention; (b) attention aggregation.
Relational Graph Attention Layer (R-GAL) is the basic part of MEGr-Net, and the output is the vertex state features under k-heads attention influence. We denote the total number of vertices as , the vertex features input to R-GAL as (F is the dimension of vertex state space), and the relations as ( is the dimension of edge state space).
We firstly consider the interaction between vertex and its neighbors (attention heads). The semantic relation matrix from is transformed into relation features matrix :
where indicates deleting all the empty relations, indicate the neighbors of .
The attention mechanism is denoted as ; then, we calculate the attention coefficients:
where is the vertex weight matrix, and is the edge weight matrix. These two matrices realize the parallel computation of linear transformation on each vertex. indicates the interaction of the relation between vertex and its neighbor , as well as itself (self-attention). In MEGr-Net, the masked-attention mechanism is used to distribute the attention interaction to the neighbors of , so the masked-attention coefficient is:
The MEGr-Net sets the attention mechanism as a single-layer feed forward network (FFN) with parameters and LeakyReLU activate function, then:
Next, updating the vertex features of :
where is a nonlinear function, and we use the ELU in MEGr-Net.
When considering the multi-heads attention, we have:
where indicates the concatenation of k vertex features of under its k-neighbors attention interaction.
In the last R-GAL, we calculate the average features instead of concatenating, and use the logistic sigmoid to normalize the output features into as the probability that indicates vertex is the answer entity:
For the efficiency of computation, we use matrices in MEGr-Net to describe the whole progress: First, the vertex features matrix of question subgraph is multiplied by the vertex weight matrix for state space transformation. Then, an all-combination is used to concatenate and in Equation (16):
We do the same operation to :
Then, the attention coefficient vector:
Updating the vertices’ state:
and aggregating:
Finally, the logistic sigmoid:
The vector indicates the probability that each node in the question subgraph is the correct answer. In other words, the MEGr-Net turns question reasoning into a node classification task, and it picks the vertex whose probability is the highest as the most probable answer.
4. Experiments
To verify the effectiveness of metaknowledge network in open domain question answering, this section carries out experiments on a subset of WebQuestionsSP, analyzes the experimental variables including: (1) triplet-based knowledge and metaknowledge, (2) various graph reasoning models, and (3) several pre-trained language models.
4.1. Datasets and Set-Ups
This work uses the open domain natural language question answering dataset WebQuestionsSP [35] for experimental analysis, which includes 4737 questions in natural language. At present, there is no well-established large-scale metaknowledge base and metaknowledge network, so we have to build it from scratch by the approach designed in Section 3.1 and Section 3.2. For the fact that the entities and relations extracted by open source NLP models naturally have quality disadvantages, the metaknowledge network we build in this work has an innate weakness when comparing with the finely-built large-scale knowledge bases such as FreeBase and WikiData. Consequently, to make hierarchical metaknowledge and non-hierarchical triplet-based knowledge comparable on the same track, considering the data quality limitation, we adopt the general approach in the construction of knowledge graph, that is, deleting all hierarchical nodes and relationships, retaining only semantic entities and relations in metaknowledge and integrating them to form a non-hierarchical triplet-based knowledge network.
Meanwhile, the process of extracting metaknowledge from Wiki documents, constructing a metaknowledge network, retrieving and encoding question subgraphs takes a big amount of time and computing resources. For example, in the previous experiment, it took an average of 2 h for VRAM GPU and 2 × 12 Core, 24 Threads CPU to build a metaknowledge network from five Wiki documents relevant to a question and complete subgraph retrieval and its encoding. Considering the data quality and hardware, this section scaled down the dataset to 2.5% of WebQuestionsSP, that is, 250 questions in natural language. In addition, it was divided into 150 for the training set, 50 for the cross validation set and 50 for the test set. In this section, it is referred to as WebQuestions . The training parameters of MEGr-Net are shown in Table 2.
Table 2The training parameters of MEGr-Net.
Parameters Values Epochs 200 Learning Rate 5 × 10 − 3 Attention Heads k 8 Dimension of Entity Features F ′ 1000 Dimension of Relation Features F r ′ 500 Hidden Units 1000
The semantic encoder is deployed on Server #1. The metaknowledge generation, metaknowledge network construction framework and MEGr-Net are deployed on Server #2 (see also Appendix A). A Tesla V100 GPU (with 32 GB VRAM) is used for training, which takes 13.5 d (325 h).
This work takes the average accuracy (avg. Acc.) as the evaluation index.
4.2. Experimental Control Groups
This section analyzes the impact of different experimental variables on MbQA from the following three aspects:
Hierarchical metaknowledge and non-hierarchical triplet-based knowledge. This is the focus of this section, that is, what improvement hierarchical metaknowledge can make on open domain question answering compared with non-hierarchical triplet-based knowledge—in other words, whether metaknowledge and metaknowledge network have superiority in open domain QA tasks. As described in Section 3.1, considering the extraction quality of open domain entities and relationships by open source NLP models, this section uses the same data and extraction models to build a metaknowledge network (referred to as MK-Net in the experiment) and triplet knowledge base (referred to as Tri-KB) by the metaknowledge structure proposed in the beginning of Section 3 and the general triplet-based knowledge structure, respectively.
Graph reasoning model. MEGr-Net, based on GAT, essentially achieves an improvement of graph data with complex relationships, like metaknowledge. Meanwhile, it partially adopts the relationship processing approach in R-GCN. Therefore, this section takes GAT and R-GCN as test baselines and compares them with MEGr-Net. To explain the impact of (meta)knowledge extraction quality on the results, this section introduces the results of DrQA [21] and GRAFT-Net [22] on the entire WebQuestionsSP as a reference.
Pre-trained language models (PLMs). The input of MEGr-Net is the question subgraph encoded by GE4MK, and its semantic features mainly come from the text embedding vector encoded by the PLM in GE4MK. Therefore, different PLMs may exert different impact on the semantic feature richness of the problem subgraph. This section takes BERT as the baseline and RoBERTa [33] and ALBERT [36] as the control groups.
4.3. Results and Analysis
The results on control group #1 on WebQuestions are shown in Table 3. The results show that, with the same data quality, the hierarchical metaknowledge achieves better results than non-hierarchical triplet knowledge in open domain question answering (+16.9% Tri-KB).
Table 3Results on Control Group #1.
(Meta) Knowledge Network IH-Acc # Tri-KB 0.483
MK-Net
0.652
The WebQuestions dataset which we use in the experiment is part of the whole WebquestionsSP, so we use average In-House Accuracy (IH-Acc.) for avg. Acc.
The results on control group #2 are shown in Table 4. For GAT, the relationship matrix in MEGr-Net and the relationship weight matrix in R-GAL are removed in this section. Modifications have been made to R-GCN for the tasks in this section.
Table 4Results on Control Group #2.
Graph Reasoning Models Acc. Baselines GAT (MK-Net) 0.608 (IH † R-GCN # 0.601 (IH)
MEGr-Net (MK-Net) 0.652 (IH)
DrQA ☆ 0.215
GRAFT-Net ☆ 0.687
Model from https://github.com/kkteru/r-gcn (accessed on 9 December 2021). The data are cited from [22], respectfully. IH indicates that the experiments are based on the dataset WebQuestions , and if not annotated that indicates that the experiments are based on the dataset WebQuestionsSP.
As can be seen from the results, MEGr-Net achieves better performance than the baselines in the reasoning of hierarchical graph data with complex semantic relationships, such as a metaknowledge network (+4.4%GAT, +5.1%R-GCN). Meanwhile, compared with GRAFT-Net, which uses the complete FreeBase as the knowledge base and integrates the document (doc) and KB features, MEGr-Net still lags behind, indicating that it still needs to be improved in MbQA, especially in the integration with MRC method (see also Section 4).
The results on control group #3 are shown in Table 5 (see Appendix B for the source of the pre-training parameter file of the pre-training language model). From the results, the PLMs (ALBERT , RoBERTa ) with large-scale parameters perform better, indicating that the larger the PLMS used by the graph encoder, the finer the fine tuning and the richer the semantic features of the question subgraph, the better performance will be achieved in MbQA.
Table 5Results on Control Group #3.
MEGr-Net PLMs IH-Acc. Baseline +BERT B A S E 0.652 +ALBERT B A S E 0.646 +BERT L A R G E 0.670 +RoBERTa L A R G E 0.692 +ALBERT X X L A R G E
0.708
As shown in Figure 5, the combination of MEGr-Net and ALBERTXXLARGE achieved the best results (+5.6% MEGr-Net+BERT ) and gained better performance than GRAFT-Net using LSTM [lstm] as a text encoder, which proves that PLMS based on Transformers [28] is better than LSTM in MbQA.
Figure 5Results of graph reasoning models and PLMs in MbQA.
Generally, the metaknowledge network with document directory hierarchy can significantly improve the existing methods in KBQA, which is basically consistent with the view that titles play a positive role in question answering in [22]. Meanwhile, finer PLMs can improve the semantic feature representations of question subgraphs and achieve better results in question answering. This is also consistent with the view and experimental results in [37].
5. Discussion
From the overall results of this work, metaknowledge basically solves the problems of triplet-based knowledge with weak structural logic, and provides a new idea for the theoretical and practical research of knowledge engineering. Meanwhile, it must also be noted that, as a relatively new research field, there are still some urgent problems that need to be solved in the future work.
5.1. Metaknowledge and Metaknowledge Network Modeling
The metaknowledge and metaknowledge network modeled by the single dimension network in this work (Section 3.2 and Section 3.3) is a compromising strategy to reduce the complexity of the model under the current realistic conditions of mainstream GNN models. In fact, according to the our concept, the metaknowledge network should be a multi-dimensional hyper-graph with hierarchical structure (Figure 6). The metaknowledge network expressed by that type of graph model includes two dimensions: hierarchical dimension and semantic dimension. The hierarchical dimension is in the outer layer, which includes all hierarchical nodes and relationships; the semantic dimension is in the inner layer, which includes all semantic nodes and relationships subordinate to the hierarchical nodes. Ref. [38] proposes an embedding framework MINES for multi-dimensional networks with hierarchical structure, which uses a hierarchical structure for multi-dimensional network embedding; Ref. [37] proposes an open domain question answering method based on hyper-edge fusion. These documents show the feasibility of graph reasoning on the metaknowledge network expressed by the hierarchical multi-dimensional hyper-graph. This metaknowledge modeling method needs to be further studied and explored.
Figure 6Metaknowledge that modeled by multi-dimensional hyper-graph with hierarchical structure.
5.2. MbQA and Graph Reasoning
Limited by the extraction effect of open-source NLP models, MbQA has insufficient advantages over KBQA. At least under the existing conditions, there is still a huge gap between the metaknowledge network and mutual large-scale knowledge bases such as freebase from the perspective of data quality. Therefore, MbQA may be more suitable for in-domain QA tasks (such as question answering on laws and regulations). The fine-tuned NLP models will significantly improve the extraction quality of metaknowledge semantic elements. At the same time, the structural logic of metaknowledge network makes it have the ability to deal with complex relationships. Therefore, the role of the metaknowledge network in multi-hop QA tasks is also a direction worthy of research. In terms of graph reasoning models for question answering tasks, MEGr-net relies on the hierarchical features contained in metaknowledge to supplement the short board relying only on semantic features (KBQA). On this basis, documents [22] and pre-trained language models [39] can continue to be integrated into graph reasoning to select the best from the best and enhance the effect of MbQA.
In general, the metaknowledge enhanced question answering is a brand-new method for solving the problem caused by triplet-based knowledge, and it improves the capability of current knowledge bases (which are also the knowledge engines of intelligent voice interaction devices). In the foreseeable future, this method could be the antidote to help intelligent voice devices get rid of the problems that they are not so good when answering complex questions asked by users, and make great progress in the interaction with human users.
6. Conclusions
Facing the problems in current open domain QA tasks caused by the loose knowledge association and weak structural logic of triplet-based knowledge, this work makes pivotal innovations on metaknowledge enhance question answering: (1) Metaknowledge extraction and metaknowledge network construction, where we present the approach of generating metaknowledge and building metaknowledge network from Wiki documents automatically. (2) Metaknowledge and metaknowledge network modeling, where we generally consider several different kinds of features from reasoning performance related aspects including semantic features such as textual content, entity type, relations, and along with hierarchical features. (3) MEGr-Net, which is proposed for question answering, which aggregates both relational and neighboring interactions compared with R-GCN and GAT. Experiments have proved the improvement of metaknowledge over main-stream triplet-based knowledge. We have found that the graph reasoning models and pre-trained language models also have influences on the metaknowledge enhanced question answering approaches.
Table A1Hardware and software environments of Server#1.
Server #1: Providing BERT Embedding Service Hard-ware Env. CPU 2 × Intel Xeon E5-2678 v3 (48) @ 3.300 GHz RAM 32 GB GPU 4 × NVIDIA GV102 (11 GB VRAM) Software Env. OS Ubuntu 18.04.5 LTS Python Python 3.6.5: Anaconda PyTorch 1.6.0 (for GPU) TensorFlow 1.15.0 (for GPU)
Table A2Hardware and software environments of Server#2.
Server #2: Main Experimental Environment Hard-ware Env. CPU 2 × Intel Xeon Silver 4210R (40) @ 3.200 GHz RAM 256 GB GPU 4 × NVIDIA Tesla V100S (32 GB VRAM, using 1) Software Env. OS Ubuntu 20.04.2 LTS Python Python 3.7.7: Anaconda PyTorch 1.9.0 (for GPU) TensorFlow 1.15.0 (for GPU)
3 in total
1. Metaknowledge.
Journal: Science
Date: 2011-02-11 Impact factor: 47.728
2. Multi-Modal Explicit Sparse Attention Networks for Visual Question Answering.
Journal: Sensors (Basel)
Date: 2020-11-26 Impact factor: 3.576
3 in total
, which represent the title, section titles, summary, or paragraphs (referred to as metaknowledge hierarchical elements in Ref. [7]).
Figure 2An example of metaknowledge extracted from Wiki documents. The number on the arrows indicates the hierarchical levels of the relations, which are also the weight of edges.
Listing 1
Denotations of keys in metaknowledge dictionary.
|
|
Table 1
Denotations of keys in metaknowledge dictionary.
| Entities | Relations | ||
|---|---|---|---|
|
| Entity ID |
| Relation ID |
3.2. Building Metaknowledge Network
Figure 3Test to decide metaknowledge association tolerance.
3.3. Metaknowledge Encoding
3.4. Graph Reasoning: MEGr-Net
Figure 4The attention mechanism of MEGr-Net. (a) Self-attention; (b) attention aggregation.
4. Experiments
4.1. Datasets and Set-Ups
Table 2
The training parameters of MEGr-Net.
| Parameters | Values |
|---|---|
| Epochs | 200 |
| Learning Rate | 5 |
| Attention Heads | 8 |
| Dimension of Entity Features | 1000 |
| Dimension of Relation Features | 500 |
| Hidden Units | 1000 |
4.2. Experimental Control Groups
4.3. Results and Analysis
Table 3
Results on Control Group #1.
| (Meta) Knowledge Network | IH-Acc |
|---|---|
| Tri-KB | 0.483 |
|
|
|
The WebQuestions dataset which we use in the experiment is part of the whole WebquestionsSP, so we use average In-House Accuracy (IH-Acc.) for avg. Acc.
Table 4
Results on Control Group #2.
| Graph Reasoning Models | Acc. | |
|---|---|---|
| Baselines | GAT (MK-Net) | 0.608 (IH |
| R-GCN | 0.601 (IH) | |
| MEGr-Net (MK-Net) | 0.652 (IH) | |
| DrQA | 0.215 | |
| GRAFT-Net | 0.687 | |
Model from https://github.com/kkteru/r-gcn (accessed on 9 December 2021). The data are cited from [22], respectfully. IH indicates that the experiments are based on the dataset WebQuestions , and if not annotated that indicates that the experiments are based on the dataset WebQuestionsSP.
Table 5
Results on Control Group #3.
| MEGr-Net | PLMs | IH-Acc. |
|---|---|---|
| Baseline | +BERT | 0.652 |
| +ALBERT | 0.646 | |
| +BERT | 0.670 | |
| +RoBERTa | 0.692 | |
| +ALBERT |
|
Figure 5Results of graph reasoning models and PLMs in MbQA.
5. Discussion
5.1. Metaknowledge and Metaknowledge Network Modeling
Figure 6Metaknowledge that modeled by multi-dimensional hyper-graph with hierarchical structure.
5.2. MbQA and Graph Reasoning
6. Conclusions
Table A1
Hardware and software environments of Server#1.
| Server #1: Providing BERT Embedding Service | ||
|---|---|---|
| Hard-ware Env. | CPU | 2 × Intel Xeon E5-2678 v3 (48) @ 3.300 GHz |
| RAM | 32 GB | |
| GPU | 4 × NVIDIA GV102 (11 GB VRAM) | |
| Software Env. | OS | Ubuntu 18.04.5 LTS |
| Python | Python 3.6.5: Anaconda | |
| PyTorch | 1.6.0 (for GPU) | |
| TensorFlow | 1.15.0 (for GPU) | |
Table A2
Hardware and software environments of Server#2.
| Server #2: Main Experimental Environment | ||
|---|---|---|
| Hard-ware Env. | CPU | 2 × Intel Xeon Silver 4210R (40) @ 3.200 GHz |
| RAM | 256 GB | |
| GPU | 4 × NVIDIA Tesla V100S (32 GB VRAM, using 1) | |
| Software Env. | OS | Ubuntu 20.04.2 LTS |
| Python | Python 3.7.7: Anaconda | |
| PyTorch | 1.9.0 (for GPU) | |
| TensorFlow | 1.15.0 (for GPU) | |
3 in total
1. Metaknowledge.
Journal: Science Date: 2011-02-11 Impact factor: 47.728
2. Multi-Modal Explicit Sparse Attention Networks for Visual Question Answering.
Journal: Sensors (Basel) Date: 2020-11-26 Impact factor: 3.576
3 in total
北京卡尤迪生物科技股份有限公司 © 2022-2023.