Junichi Fukumoto

WoZ Simulation of Interactive Question Answering

QACIAD (Question Answering Challenge for Information Access Dialogue) is an evaluation framework for measuring interactive question answering (QA) technologies. It assumes that users interactively collect information using a QA system for writing a report on a given topic and evaluates, among other things, the capabilities needed under such circumstances. This paper reports an experiment for examining the assumptions made by QACIAD. In this experiment, dialogues under the situation that QACIAD assumes are collected using WoZ (Wizard of Oz) simulating, which is frequently used for collecting dialogue data for designing speech dialogue systems, and then analyzed. The results indicate that the setting of QACIAD is real and appropriate and that one of the important capabilities for future interactive QA systems is providing cooperative and helpful responses.

Are open-domain question answering technologies useful for information access dialogues?---an empirical study and a proposal of a novel challenge

There are strong expectations for the use of question answering technologies in information access dialogues, such as for information gathering and browsing. In this paper, we empirically examine what kinds of abilities are needed for question answering systems in such situations, and propose a challenge for evaluating those abilities objectively and quantitatively. We also show that existing technologies have the potential to address this challenge. From the empirical study, we found that questions that have values and names as answers account for a majority in realistic information-gathering situations and that those sequences of questions contain a wide range of reference expressions and are sometimes complicated by the inclusion of subdialogues and focus shifts. The challenge proposed is not only novel as an evaluation of the handling of information access dialogues, but also includes several valuable ideas such as categorization and characterization of information access dialogues, and introduces three measures to evaluate various aspects in addressing list-type questions and reference test sets for evaluating context-processing ability in isolation.

Answering questions of Information Access Dialogue (IAD) task using ellipsis handling of follow-up questions

In this paper, we propose ellipsis handling method for follow-up questions in Information Access Dialogue (IAD) task of NTCIR QAC3. In this method, our system classifies ellipsis patterns of question sentences into three types and recognizes elliptical elements using ellipsis handling algorithm for each type. In the evaluation using Formal Run and Reference Run data, there were several cases which our algorithm could not handle ellipsis correctly. According to the analysis of evaluation results, the main reason of low performance was lack of word information for recognition of referential elements. If our system can recognize word meanings correctly, some errors will not occur and ellipsis handling works well.

An evaluation of question answering challenge (QAC-1) at the NTCIR workshop 3

The Question Answering Challenge (QAC) was carried out as the first evaluation task on question answering of the NTCIR Workshop 3 [Fukumoto2002] [NTCIR]. Question answering in an open domain is a task for obtaining appropriate answers to given domain independent questions written in natural language from a large corpus. The purpose of the QAC was to develop practical QA systems in an open domain focusing on research of user interaction and information extraction. A further objective was to develop an evaluation method for the question answering system and information resources for evaluation. To evaluate QA technologies, there are several technical aspects to consider for the extraction of answer expressions from knowledge sources. Question type is one aspect of the QA system evaluation. In QAC-1, question types are defined as a noun or noun phrase which indicates names of persons, organizations, and various artifacts and facts, such as money, size, date and so on. Moreover, information related to these can also be considered as answer candidates: for example, names of persons, their affiliations, age and status can be an answer; and for names of organizations, their annual profit, year of establishment and so on. Another aspect to consider is how many answer expressions exist in the knowledge sources. In QAC-1, there may be multiple answers or no answers to questions in general. This aspect makes development of a QA system difficult because, QA system has to check all answer candidates very carefully. User interaction technology requires actual interaction between the computer and person. In actual QA between people, there are typically several interactions which take place in order to confirm the intention of the questions and so on. In QAC-1, we gave one follow-up question for the first question. It will be necessary to resolve an ellipsis in the follow-up question which is frequently occurs in Japanese sentences.

Recommendation System for Alternative-Ingredients Based on Co-occurrence Relation on Recipe Database and the Ingredient Category

Many people often cook a dish with a cooking recipe on Websites and magazines. The listed ingredients in the recipe sometimes can not be prepared. This paper proposes a recommendation system for alternative ingredients. The recommendation ingredients based on co-occurrence frequency of ingredients on recipe database and ingredient category stored in a cooking ontology. The result of the subjective evaluation experiments showed 88% appropriateness for alternative-ingredients recommendation.

Automatic Throwing of Questions Generated from Posted Comments to Activate Text Based Discussions

It is more difficult to activate text based discussions than face-to-face discussions because participants tend to hesitate give their opinions. To activate text based discussions, it is necessary for participants to give their opinions many times. We suppose that participants will give their opinions if questions are thrown to discussions. This paper proposes an agent that throws questions automatically in text based discussions. The questions are generated from comments posted by discussion participants previously in a discussion. The agent chooses a comment including a word that appears frequently in previous posted comments. The agent adds “why” or “how” to the beginning of the chosen comment, then the agent throws the comment as a question to the discussion participants. We experimentally verified that the agent could activate discussions by throwing questions.

“Co-occurrence Relation” and “Ingredient Category” Recommend Alternative-Ingredients

Websites and magazines are now becoming more useful for general people to cook a dish everyday. Sometimes, we have inconvenience for such information because the listed ingredients in the recipe can not be prepared. This paper proposes a recommendation method of alternative ingredients towards such situation. The recommendation is realized by considering co-occurrence of ingredients on recipe database and ingredient category stored in a cooking ontology. The subjective evaluation experiments showed 88% appropriateness for the alternative-ingredients recommended by our proposed method. Also, the recommended ingredients were used in a real cooking, the good tastes were confirmed through the workshop.

A Generation Method of Back-Channel Response to Let a Chatting Bot Be a Member of Discussions in a Text-Based Chat

This paper proposes a method to generate backchannel responses of a chatting bot in a text-based chat. We have been studied methods of activating text-based discussions. In our previous work, we have proposed a method to post questions by a chatting bot to activate discussions. Posting questions by the chatting bot could obtain answers from discussion participants. Some of the answers contributed to increasing the number of topics in discussions. However, all questions are not answered carefully by the discussion participants. Some of the questions are passed through by the discussion participants. Careful answering is required to activate text-based discussions. We thought that the discussion participants did not recognize the chatting bot as a member of a discussion because the chatting bot posted only questions. The chatting bot seemed not to listen to the others comments.Therefore, the discussion participants did not answer all questions carefully. We suppose that the chatting bot is recognized as a member of a discussion and questions of the chatting bot are answered carefully by discussion participants if the chatting bot posts back-channel responses to the others comments. The chatting bot posts back-channel responses with three satisfied conditions that have been obtained from preliminary experiments. In evaluation experiments, we compared the chatting bot of the proposed method with a chatting bot that gives only questions. We evaluated the efficiency of the two chatting bots for activating text-based discussions. We evaluated the degree of activation of a discussion using the number of topics obtained in the discussion and the rate of carefully answered questions. When using the proposed chatting bot, both the number and the rate were higher than those of the comparative chatting bot. The results indicate that discussions are more activated by posting both questions and back-channel responses by the chatting bot. The three types of back-channel response were posted appropriately instead of a condition: when a comment with a question mark is posted, the chatting bot posts a back-channel response as representing an agreement with the posted comment.

The Difference of Word Importance before and after Bookmark for Novel Abstract in Each Reading Progress

Abstract This paper describes an index and target periods to generate a novel abstract for each reading progress. As reading a long novel, the reading is often interrupted as putting a bookmark into the book. Sometimes, the information described in the read part is forgotten away from the reader’s memory. Then the abstract for each reading progress is effective to remind the reader of the story. The goal of this study is to automatically generate the abstract for each reading progress. In this paper, the difference of the word importance before and after bookmark is considered as the index to select the sentences for the abstract. Also, from where the sentences should be selected is compared as discussing the output results. Through the experiment, it was confirmed that the abstract in which the sentences were selected from the period just before the bookmark was better than the abstract generated from all part of the past story. Moreover, it was suggested that not paragraphs but sentences might be appropriate unit to generate the abstract for each reading progress.

Analysis of Dialogues Difficulty in Anime Comparing with JLPT Listening Tests

Abstract More and more people choose to learn Japanese as a foreign language (JFL) in the world. The traditional teaching materials have focused on training for the skills of writing and reading in Japanese. However, it seems that the JFL learners would like to train their speaking and listening rather than writing and reading because the speaking and listening are necessary to make conversations. We use Japanese animation which is called “Anime” in the world as a new teaching material of Japanese. Anime has a huge amount of scenes in which characters speak with their voice in the standard and clear pronunciations. It means that Anime has many good examples of Japanese dialogues for JFL learners to improve their speaking and listening. The goal of our research is to propose a new method to classify dialogues in Anime depending on the dialogues’ degrees of difficulty. In this paper, we analyzed the words and expressions used in the Anime dialogues and the scripts of listening tests of the previous Japanese Language Proficiency Tests (JLPT). Each word and expression were classified into different Japanese language levels. We used the analysis results of JLPT as a standard reference to be compared with Anime dialogues. The analysis results showed that the genres of Anime had effects on using words and expressions. We also found that the scripts of listening tests of high-level included many high-level words and expressions.