Making Question Answering More Natural - Research on Natural Answer Generation System Based on Copy and Retrieval Mechanism

Making machines as intelligent as humans has always been the goal of researchers. Since the concept of intelligence is difficult to define accurately, Turing proposed the famous Turing Test: if a machine can have a conversation with a human and cannot be identified as a machine, then the machine is said to be intelligent. The Turing Test has always been used as a symbol of testing artificial intelligence. The question-answering system itself is a scene of the Turing Test. If we have an intelligent question-answering system like a human, it is equivalent to passing the Turing Test. Therefore, the research on question-answering systems has always received great attention.

Traditional knowledge question-answering provides precise answer entities for questions raised by users (using natural language), for example, for the question "Where was Tagore born?", it returns "Kolkata". However, simply providing such a solitary answer entity is not a very friendly interaction method. Users prefer to receive complete answers expressed in natural language sentences, such as "Indian poet Tagore was born in Kolkata". Natural answers can be widely used in knowledge service fields such as community question-answering and intelligent customer service. The generation of natural answers in knowledge question-answering has very clear practical significance and strong application background.

Compared with returning answer entities, returning natural answers in knowledge quizzes has the following advantages:

1. Ordinary users prefer self-contained answers rather than partial pieces of information.

2. Natural answers can provide some form of explanation for the process of answering questions, and can also invisibly increase users' acceptance of the system.

3. Natural answers can also provide contextual information associated with the answer (such as "Indian poet" in the natural answer above).

4. Complete natural language sentences can better support subsequent tasks such as answer verification and speech synthesis.

However, it is not easy to let the knowledge question-answering system generate answers in natural language. At present, most of the language generation models based on deep learning are based on the models of numerical calculation based on raw data. How to integrate the external knowledge base represented by symbols into the process of generating natural answers is a big challenge. In addition, the answers to many questions need to use multiple facts in the knowledge base, and different semantic units (words, entities) of a natural answer may need to be obtained through different channels. Answering such complex questions that require the use of multiple modes to extract and predict semantic units brings greater challenges to the generation of natural answers.

In order to solve these problems, Dr. He Shizhu, Liu Cao, Liu Kang and Zhao Jun from the Institute of Automation of the Chinese Academy of Sciences published a paper titled "Generating Natural Answers by Incorporating Copying and Retrieving Mechanisms in Sequence-to-Sequence Learning" at this year's ACL2017, proposing an end-to-end question answering system COREQA. It is based on an encoder-decoder deep learning model and introduces a copying and retrieval mechanism for complex questions that require multiple facts to answer. It obtains different types of vocabulary from different sources using different vocabulary acquisition modes such as copying, retrieval and prediction, corresponding to different parts of the answer vocabulary in complex questions, thereby generating natural answers to complex questions.

Figure 1: COREQA framework

So how do we do this? Here is an example of the question “Do you know where Jet Li is from?”

1. Knowledge retrieval: First, we need to identify the entity words contained in the question. Here, we identify the entity word: Jet Li. Then, we retrieve the relevant triples (topic, attribute, object) from the knowledge base (KB) based on the entity word. For the entity Jet Li, we can retrieve triples such as (Jet Li, gender, male), (Jet Li, birthplace, Beijing), (Jet Li, nationality, Singapore), etc.

2. Encoder: In order to generate answers, we need to encode the questions and retrieved knowledge into vectors for subsequent deep generative models to use.

Question encoding: A bidirectional RNN (Bi-RNN) is used to represent questions in two ways: one is to piece together the RNN state vectors in the two directions to get a vector sequence Mq; the other is to take out the last vector of each directional RNN and piece them together to get a vector q, which is used to represent the question sentence as a whole.

Knowledge encoding: The memory network is used to encode the knowledge triplets obtained in the knowledge retrieval phase. For a triplet, three vectors are used to represent each part, and then they are concatenated together to form a vector fi to represent the triplet, and Mkb is used to represent all triplet vectors.

3. Decoder: Next, a natural answer is generated based on the encoding vector of the answer and knowledge. Although the natural answer is a sequence of words, different words may need to be obtained through different channels. For example, for the answer to the above question, "Jet Li was born in Beijing and is now a citizen of Singapore.", the word "Jet Li" needs to be copied from the source question, the entity words "Beijing" and "Singapore" need to be retrieved from the knowledge base, and other words such as "born", "in", and "now" need to be predicted by the model. Therefore, here, based on the standard sequence-to-sequence model, three word acquisition modes (including copying, retrieval, and prediction) are integrated, and they are modeled with a unified model, so that various modes compete and influence each other, and finally generate the best natural answer to complex problems.

▲ Figure 2: COREQA decoding example

In order to test the effectiveness of the model, the paper conducted experiments on a simulated dataset (question-and-answer data constructed from 108 question-and-answer template rules) and a real dataset (239,934 community question-and-answer data obtained from Baidu Knows), and achieved good results in both automatic and manual evaluation.

Talking about future work, Dr. He Shizhu said: "Currently, the COREQA model is still too dependent on learning data. From the experimental results, it can be seen that it can have almost perfect performance on simulated artificial data, but it is still unsatisfactory on real data. The reason is that the model is essentially a fit to the original data, learning a mapping function from an input question (word sequence) to an output answer (word sequence), especially non-entity words (that is, words that are not copied and retrieved) are often predicted inaccurately. This is the biggest problem of the model. We plan to add some external artificial knowledge to adjust the model and improve the existing model. Another shortcoming is that currently only the knowledge base represented in the form of triples can be used, and it is assumed that the answer entity is the object part of the triple. In fact, this assumption does not hold for many problems. Another possible direction of improvement is to use knowledge bases with different representations. In addition, the model can also be applied to tasks such as machine translation, allowing language generation models to interact with external knowledge resources."

As for the future development of question-answering systems, Dr. He Shizhu also has some of his own views: "As far as I know, in real engineering practice, question-answering systems still use templates and rules, and rarely or never use statistical models, let alone deep learning models. At present, in the research community, question-answering systems almost all use deep learning models, or even completely end-to-end methods. The reason is that I personally think that the question-answering system is a system engineering, not a pure research task. At present, the research community does not have a unified paradigm for question-answering systems (unlike information retrieval, machine translation, information extraction and other tasks). Therefore, in the future, question-answering systems may need to summarize one or several general paradigms and processes, which can be decomposed into several subtasks, which will make it easier to promote the research and development of question-answering. In addition, the question-answering system cannot achieve practical effects, and its problems have not been analyzed clearly. Is it the incomplete knowledge resources, the heterogeneity of knowledge representation, or the challenge of understanding natural language problems? Finally, I think that for tasks such as question-answering systems that require a lot of knowledge, when the data scale is difficult to expand on a large scale, integrating statistical models and prior knowledge (extracted knowledge base, language knowledge, common sense, etc.) is a feasible development direction." \

Paper author: He Shizhu, Liu Cao, Liu Kang, Zhao Jun (Institute of Automation & University of Chinese Academy of Sciences)

Special Correspondent | Wang Zhe (University of Science and Technology of China)

Welcome to view the original paper:

Generating Natural Answers by Incorporating Copying and Retrieving Mechanisms in Sequence-to-Sequence Learning