Kyungduk Kim

Data-driven user simulation for automated evaluation of spoken dialog systems

This paper proposes a novel integrated dialog simulation technique for evaluating spoken dialog systems. A data-driven user simulation technique for simulating user intention and utterance is introduced. A novel user intention modeling and generating method is proposed that uses a linear-chain conditional random field, and a two-phase data-driven domain-specific user utterance simulation method and a linguistic knowledge-based ASR channel simulation method are also presented. Evaluation metrics are introduced to measure the quality of user simulation at intention and utterance. Experiments using these techniques were carried out to evaluate the performance and behavior of dialog systems designed for car navigation dialogs and a building guide robot, and it turned out that our approach was easy to set up and showed similar tendencies to real human users.

Recent Approaches to Dialog Management for Spoken Dialog Systems

A field of spoken dialog systems is a rapidly growing research area because the performance improvement of speech technologies motivates the possibility of building systems that a human can easily operate in order to access useful information via spoken languages. Among the components in a spoken dialog system, the dialog management plays major roles such as discourse analysis, database access, error handling, and system action prediction. This survey covers design issues and recent approaches to the dialog management techniques for modeling the dialogs. We also explain the user simulation techniques for automatic evaluation of spoken dialog systems.

MMR-based Active Machine Learning for Bio Named Entity Recognition

This paper presents a new active learning paradigm which considers not only the uncertainty of the classifier but also the diversity of the corpus. The two measures for uncertainty and diversity were combined using the MMR (Maximal Marginal Relevance) method to give the sampling scores in our active learning strategy. We incorporated MMR-based active machine-learning idea into the biomedical named-entity recognition system. Our experimental results indicated that our strategies for active-learning based sample selection could significantly reduce the human effort.

Hybrid approach to robust dialog management using agenda and dialog examples

Spoken dialog systems have difficulty selecting which action to take in a given situation because recognition and understanding errors are prevalent due to noise and unexpected inputs. To solve this problem, this paper presents a hybrid approach to improving robustness of the dialog manager by using agenda-based and example-based dialog modeling. This approach can exploit n-best hypotheses to determine the current dialog state in the dialog manager and keep track of the dialog state using a discourse interpretation algorithm based on an agenda graph and a focus stack. Given the agenda graph and multiple recognition hypotheses, the system can predict the next action to maximize multi-level score functions and trigger error recovery strategies to handle exceptional cases due to misunderstandings or unexpected focus shifts. The proposed method was tested by developing a spoken dialog system for a building guidance domain in an intelligent service robot. This system was then evaluated by simulated and real users. The experimental results show that our approach can effectively develop robust dialog management for spoken dialog systems.

A Frame-Based Probabilistic Framework for Spoken Dialog Management Using Dialog Examples

This paper proposes a probabilistic framework for spoken dialog management using dialog examples. To overcome the complexity problems of the classic partially observable Markov decision processes (POMDPs) based dialog manager, we use a frame-based belief state representation that reduces the complexity of belief update. We also used dialog examples to maintain a reasonable number of system actions to reduce the complexity of the optimizing policy. We developed weather information and car navigation dialog system that employed a frame-based probabilistic framework. This framework enables people to develop a spoken dialog system using a probabilistic approach without complexity problem of POMDP.

Hybrid user intention modeling to diversify dialog simulations

This paper proposes a novel user intention simulation method which is data-driven but can integrate diverse user discourse knowledge to simulate various types of user behaviors. A method of data-driven user intention modeling based on logistic regression is introduced in the Markov logic framework. Human dialog knowledge is designed into two layers, domain and discourse knowledge, and integrated with the data-driven model in generation time. Three types of user knowledge, i.e., cooperative, corrective and self-directing, are designed and integrated to generate behaviors of corresponding user-types. In experiments to investigate the patterns of simulated users, the approach successfully generated cooperative, corrective and self-directing user intention patterns.

Unsupervised Spoken Language Understanding for a Multi-Domain Dialog System

This paper proposes an unsupervised spoken language understanding (SLU) framework for a multi-domain dialog system. Our unsupervised SLU framework applies a non-parametric Bayesian approach to dialog acts, intents and slot entities, which are the components of a semantic frame. The proposed approach reduces the human effort necessary to obtain a semantically annotated corpus for dialog system development. In this study, we analyze clustering results using various evaluation metrics for four dialog corpora. We also introduce a multi-domain dialog system that uses the unsupervised SLU framework. We argue that our unsupervised approach can help overcome the annotation acquisition bottleneck in developing dialog systems. To verify this claim, we report a dialog system evaluation, in which our method achieves competitive results in comparison with a system that uses a manually annotated corpus. In addition, we conducted several experiments to explore the effect of our approach on reducing development costs. The results show that our approach be helpful for the rapid development of a prototype system and reducing the overall development costs.

Modeling confirmations for example-based dialog management

This paper proposes a method to model confirmations for example-based dialog management. To enable the system to provide a confirmation to the user in an appropriate time, we employed a multiple dialog state representation approach for keeping track of user input uncertainty and implemented a confirmation agent which decides when the information gathered from the user contains an error. We developed a car navigation dialog system to evaluate our proposed method. Evaluations with simulated dialogs show our approach is useful for handling misunderstanding errors in example-based dialog management.

Correlation-based query relaxation for example-based dialog modeling

Query relaxation refers to the process of reducing the number of constraints on a query if it returns no result when searching a database. This is an important process to enable extraction of an appropriate number of query results because queries that are too strictly constrained may return no result, whereas queries that are too loosely constrained may return too many results. This paper proposes an automated method of correlation-based query relaxation (CBQR) to select an appropriate constraint subset. The example-based dialog modeling framework was used to validate our algorithm. Preliminary results show that the proposed method facilitates the automation of query relaxation. We believe that the CBQR algorithm effectively relaxes constraints on failed queries to return more dialog examples.

Hybrid Approach to User Intention Modeling for Dialog Simulation

This paper proposes a novel user intention simulation method which is a data-driven approach but able to integrate diverse user discourse knowledge together to simulate various type of users. In Markov logic framework, logistic regression based data-driven user intention modeling is introduced, and human dialog knowledge are designed into two layers such as domain and discourse knowledge, then it is integrated with the data-driven model in generation time. Cooperative, corrective and self-directing discourse knowledge are designed and integrated to mimic such type of users. Experiments were carried out to investigate the patterns of simulated users, and it turned out that our approach was successful to generate user intention patterns which are not only unseen in the training corpus and but also personalized in the designed direction.