Nowadays, we are making huge progress in the field of Artificial Intelligence. Since the rise of artificial neural networks, new astonishing frontiers are continuously being discovered. The development is so fast that overall no major technical limits are in sight. In the light of these developments, Question Answering (QA) is revisited in this blog article. Our goal is to depart from traditional approaches to QA where the solutions typically rely on a structured knowledge-base (KB). Examples of such systems are IBM Watson, WolframAlpha or even Siri. In their backbone, all these chatbot-like systems typically use some structured KBs like a SQL Database, RDF Triple Store, Graph Database or XML-Files. Their predominant achievement is “merely” the translation of natural language questions into appropriate machine language queries without any usage of higher “semantics” and intelligence. What if we skip this intermediate step of translation and try to answer the human queries directly with the machines? More specifically, can we build a system which can be trained to read like humans and answer arbitrary questions on text passages? This might sound like a science-fiction movie. But certainly, with this approach, we will be one step closer to the greater aim of AI.
Classic methods for QA
Ultimately, everything is a form of QA—all human problems can be cast into that format. Within computer science, QA is an interdisciplinary field of Text Mining, Information Retrieval and Natural Language Processing (NLP). Its aim is to automatically answer questions posed by human users in natural language. QA can take any dimension and semantic complexity. From simple questions about numbers, dates and names over grammatical co-references to more abstract semantic questions, like “What is the meaning of life?”, QA has no limits by default. Because of its wide span, it is one of the most difficult tasks in the field of Text Mining: Parsing, Spam Detection in Emails, Sentiment Analysis and the likes of it have been solved successfully in recent years, but overall QA is still counted among the partly-unsolved tasks of Text Mining. Nevertheless, the advance of QA is closely related to the progress of AI.
Traditionally, the task of QA has been tackled with NLP methods. At its core, usually, a structured source of knowledge has been utilized. The main working direction has been to map an incoming question of the user written in natural language into a formal query which can be dealt by a machine. This mapping has been performed with a two-stage approach. First, by extracting handcrafted features out of the raw question typically with bag-of-words and their TF-IDF. Next, by classifying these features typically with some linear classifiers like Support Vector Machines (SVMs) to create a formal query (in SQL, SAPRQL or XQuery) for extracting the desired information out of the underlying KB.
Current trends towards human-like QA
Since the Deep Revolution there has been a dramatic shift in this field. We can observe a change of trends in the research community and industry. Neural networks have been used with great success in a wide variety of NLP problems and predominantly replaced the traditional methods by establishing a new state-of-the-art. The tedious task specific feature engineering has been replaced by the word vector conversion, hence joining the aforementioned two-stage approach into one unified end-to-end trainable process.
With the usage of word vectors, automatic QA is breaking new records. Neural nets evolve into systems which learn to read text and “comprehend” its content such that it can be utilized for further tasks—justlike humans when they read an exam paper and are able to answer arbitrary questions on it. Particularly for QA, we focus on the following set up: Given a text and a question both in pure, natural language, the goal is to find the answer out of the raw text. In other words, the task is to locate the subsequence which contains the answer to the question. We call this class of tasks human-like QA, i.e. QA directly on text passages, rather on any (semi-) structured KB.
This very approach appears to be promising for the future of full AI. Such an intelligent domain-independent system can open up new frontiers in the way organizations work today. It has the potential of reshaping the foundations of communication systems currently used in firms which suffer from the asymmetric distribution of information. It can be applied in any business unit where a huge amount of time is consumed finding the right answer out of the textual archives. Just as it is the case in HR units where existing rule-based business processes cannot fully capture the semantics of natural language. Imagine all the potentials arising from saving the manpower consumed in reading bulk of documents by such a human-like QA system. People might finally start focusing on the stuff that really matters.
I am Saj, Co-Founder and CTO at 1789 — Beyond Revolution, a strategic consultancy with a focus on helping organizations creating new structures by utilizing current technological potentialities. Do you agree or disagree with my thoughts? Do you want to share your stories? Please leave a comment below!