Emilio Sanchis

A statistical approach to spoken dialog systems design and evaluation

In this paper, we present a statistical approach for the development of a dialog manager and for learning optimal dialog strategies. This methodology is based on a classification procedure that considers all of the previous history of the dialog to select the next system answer. To evaluate the performance of the dialog system, the statistical approach for dialog management has been extended to model the user behavior. The statistical user simulator has been used for the evaluation and improvement of the dialog strategy. Both the user model and the system model are automatically learned from a training corpus that is labeled in terms of dialog acts. New measures have been defined to evaluate the performance of the dialog system. Using these measures, we evaluate both the quality of the simulated dialogs and the improvement of the new dialog strategy that is obtained with the interaction of the two modules. This methodology has been applied to develop a dialog manager within the framework of the DIHANA project, whose goal is the design and development of a dialog system to access a railway information system using spontaneous speech in Spanish. We propose the use of corpus-based methodologies to develop the main modules in the dialog system.

EXTRACTING SEMANTIC INFORMATION THROUGH AUTOMATIC LEARNING TECHNIQUES

We present an approach for the development of Language Understanding systems from a Transduction point of view. We describe the use of two types of automatically inferred transducers as the appropriate models for the understanding phase in dialog systems.

Error handling in a stochastic dialog system through confidence measures

In this work, we present an approach to take advantage of confidence measures obtained during the recognition and understanding processes of a dialog system, in order to guide the behavior of the dialog manager. Our approach allows the system to ask the user for confirmation about the data which have low confidence values associated to them, after the recognition or understanding processes. This technique could help to protect the system from recognition or understanding errors. Although the number of confirmation turns could increase, it would be less probable for the system to consider data with a low confidence value as correct. The understanding module and the dialog manager that we have used are modelled by stochastic automata, and some confidence measures are proposed for the understanding module. An evaluation of the behavior of the dialog system is also presented.

A stochastic approach to dialog management

In this article, we present an approach to the development of a stochastic dialog manager. The model used by this dialog manager to generate its turns takes into account both the last turns of the user and system, and the information supplied by the user throughout the dialog. As the space of situations that can be presented in the dialogs is too large, some techniques for reducing this space have been proposed. This system has been developed in the DIHANA project, whose goal is the design and development of a dialog system to access a railway information system using spontaneous speech in Spanish. A training corpus of 900 dialogs, that was acquired through the Wizard of Oz, was used to learn the models. An evaluation of the dialog manager is also presented

Multi-label Text Classification Using Multinomial Models

Traditional approaches to pattern recognition tasks normally consider only the unilabel classification problem, that is, each observation (both in the training and test sets) has one unique class label associated to it. Yet in many real-world tasks this is only a rough approximation, as one sample can be labeled with a set of classes and thus techniques for the more general multi-label problem have to be explored. In this paper we review the techniques presented in our previous work and discuss its application to the field of text classification, using the multinomial (Naive Bayes) classifier. Results are presented on the Reuters-21578 dataset, and our proposed approach obtains satisfying results.

User simulation in a stochastic dialog system

We present a new methodology of user simulation applied to the evaluation and refinement of stochastic dialog systems. Common weaknesses of these systems are the scarceness of the training corpus and the cost of an evaluation made by real users. We have considered the user simulation technique as an alternative way of testing and improving our dialog system. We have developed a new dialog manager that plays the role of the user. This user dialog manager incorporates several knowledge sources, combining statistical and heuristic information in order to define its dialog strategy. Once the user simulator is integrated into the dialog system, it is possible to enhance the dialog models by an automatic strategy learning. We have performed an extensive evaluation, achieving a slight but clear improvement of the dialog system.

The Incorporation of Confidence Measures to Language Understanding

We present an approach to the definition and application of confidence measures to the speech understanding module in a spoken dialog system, which answers queries about railway timetables and prices by telephone in Spanish. Some experiments have been carried out, and the results in terms of understanding accuracy depending on the threshold of confidence considered, are presented.

SPOKEN QA BASED ON A PASSAGE RETRIEVAL ENGINE

This paper presents a Passage Retrieval-based approach for the development of spoken Question Answering systems. Question Answering can be seen as a particular aspect of Information Retrieval where the users information need are not satisfied by means of a document, but by a portion of text. Currently, the performance of typical Question Answering systems is rather poor, if compared with other Information Retrieval tasks, however, it is predictable that these system will be used in combination with some speech interface when high precision will be achieved. The most important evaluation competitions for Question Answering, such as TREC and CLEF, still do not have a spoken Question Answering track, therefore this work presents a novel approach to this task in order to study the influence of recognition errors over Question Answering systems.

A wordnet-based indexing technique for geographical information retrieval

This paper presents an indexing technique based on Word-Net synonyms and holonyms. This technique has been developed for the Geographical Information Retrieval task. It may help in finding implicit geographic information contained in texts, particularly if the indication of the containing geographical entity is omitted. Our experiments were carried out with the Lucene search engine over the GeoCLEF 2006 set of topics. Results show that expansion can improve recall in some cases, although a specific ranking function is needed in order to obtain better results in terms of precision.

N-gram vs. keyword-based passage retrieval for question answering

In this paper we describe the participation of the Universidad Politecnica of Valencia to the 2006 edition, which was focused on the comparison between a Passage Retrieval engine (JIRS) specifically aimed to the Question Answering task and a standard, general use search engine such as Lucene. JIRS is based on n-grams, Lucene on keywords. We participated in three monolingual tasks: Spanish, Italian and French. The obtained results show that JIRS is able to return high quality passages, especially in Spanish.