Aysu Ezen-Can

A Tutorial Dialogue System for Real-Time Evaluation of Unsupervised Dialogue Act Classifiers: Exploring System Outcomes

Dialogue act classification is an important step in understanding students’ utterances within tutorial dialogue systems. Machine-learned models of dialogue act classification hold great promise, and among these, unsupervised dialogue act classifiers have the great benefit of eliminating the human dialogue act annotation effort required to label corpora. In contrast to traditional evaluation approaches which judge unsupervised dialogue act classifiers by accuracy on manual labels, we present results of a study to evaluate the performance of these models with respect to their performance within end-to-end system evaluation. We compare two versions of the tutorial dialogue system for introductory computer science: one that relies on a supervised dialogue act classifier and one that depends on an unsupervised dialogue act classifier. A study with 51 students shows that both versions of the system achieve similar learning gains and user satisfaction. Additionally, we show that some incoming student characteristics are highly correlated with students’ perceptions of their experience during tutoring. This first end-to-end evaluation of an unsupervised dialogue act classifier within a tutorial dialogue system serves as a step toward acquiring tutorial dialogue management models in a fully automated, scalable way.

Combining Task and Dialogue Streams in Unsupervised Dialogue Act Models

Unsupervised machine learning approaches hold great promise for recognizing dialogue acts, but the performance of these models tends to be much lower than the accuracies reached by supervised models. However, some dialogues, such as task-oriented dialogues with parallel task streams, hold rich information that has not yet been leveraged within unsupervised dialogue act models. This paper investigates incorporating task features into an unsupervised dialogue act model trained on a corpus of human tutoring in introductory computer science. Experimental results show that incorporating task features and dialogue history features significantly improve unsupervised dialogue act classification, particularly within a hierarchical framework that gives prominence to dialogue history. This work constitutes a step toward building high-performing unsupervised dialogue act models that will be used in the next generation of task-oriented dialogue systems.