Milan Gnjatović

Therapist-Centered Design of a Robot’s Dialogue Behavior

Significant research effort has already been invested in the field of robot-assisted therapy for children with developmental disorders, and the researchers generally agree that therapists should be involved in the development of assistive robotic tools. However, relatively little attention has been devoted to robots’ capacity to autonomously engage in a natural language dialogue in the context of robot-assisted therapy. This paper focuses on this desideratum. It introduces a programming platform that enables the therapist to design a robot’s dialogue behavior. To the extent that the platform is domain-independent, it enables the therapist to flexibly model (1) the interaction domain and the lexicon, (2) the interaction context, and (3) the robot’s dialogue strategy. To the extent that the platform is therapist-centered, it is motivated by real-life difficulties that therapists encounter while trying to specify a robot’s dialogue behavior and can be used by nontechnical therapists in a user-friendly and intuitive manner. In addition, the platform (4) enables the therapist to test dialogue strategies independently of therapeutic settings, and (5) provides estimated cognitive load placed on the child while trying to process the therapist’s dialogue acts.

Adaptive multimodal interaction with industrial robot

This paper reports a spoken natural language dialogue system that manages the interaction between the user and the industrial robot ABB IRB 140. To the extent that the dialogue system is multimodal, it uses three communication modalities: (i) spoken language (automatic speech recognition and text-to-speech synthesis), (ii) visual recognition of the figures and determination of their positions, and (iii) typed text. To the extent that the dialogue system is adaptive, it takes the verbal and spatial contexts into account in order to adapt its dialogue behavior and to process spontaneously formulated user commands of different syntactic forms without explicit syntactic expectations. The industrial robot is slightly modified and enabled to manipulate over graphical figures, following the instructions of the dialogue system.

Cognitively-inspired representational approach to meaning in machine dialogue

One of the most fundamental research questions in the field of human–machine interaction is how to enable dialogue systems to capture the meaning of spontaneously produced linguistic inputs without explicit syntactic expectations. This paper introduces a cognitively-inspired representational model intended to address this research question. To the extent that this model is cognitively-inspired, it integrates insights from behavioral and neuroimaging studies on working memory operations and language-impaired patients (i.e., Brocas aphasics). The level of detail contained in the specification of the model is sufficient for a computational implementation, while the level of abstraction is sufficient to enable generalization of the model over different interaction domains. Finally, the paper reports on a domain-independent framework for end-user programming of adaptive dialogue management modules.

A novel split-and-merge algorithm for hierarchical clustering of Gaussian mixture models

The paper presents a novel split-and-merge algorithm for hierarchical clustering of Gaussian mixture models, which tends to improve on the local optimal solution determined by the initial constellation. It is initialized by local optimal parameters obtained by using a baseline approach similar to k-means, and it tends to approach more closely to the global optimum of the target clustering function, by iteratively splitting and merging the clusters of Gaussian components obtained as the output of the baseline algorithm. The algorithm is further improved by introducing model selection in order to obtain the best possible trade-off between recognition accuracy and computational load in a Gaussian selection task applied within an actual recognition system. The proposed method is tested both on artificial data and in the framework of Gaussian selection performed within a real continuous speech recognition system, and in both cases an improvement over the baseline method has been observed.

Human-like Robot MARKO in the Rehabilitation of Children with Cerebral Palsy

Actual research in the field of robot-supported therapy is dominantly oriented on systems for clinical neurorehabilitation of motor disorders and therapy of difficulties related to autism. However, very little attention is dedicated to the functional development of the therapeutic robot, which would be capable of participating, actively and intelligently, in a verbal dialogue of natural language with a patient and therapist. In this paper an approach is presented for incorporating the human-like robot MARKO in the physical therapy for children with cerebral palsy (CP). The mechanical design of the robot MARKO is briefly described and its context aware cognitive system which connects modules for sensorimotor system, speech recognition, speech synthesis and robot vision is presented. The robot is conceived as a child’s playmate, able to manage three-party natural language conversation with a child and a therapist involved. Traditional CP physical therapies are usually repetitive, lengthy and tedious which results in a patient’s lack of interest and disengagement with the therapy. On the other hand, treatment progress and the improvement of the neural functionality are directly proportional to the amount of time spent exercising. The idea is to use the robot to assist doctors in habilitation/rehabilitation of children, with a basic therapeutical role to motivate the children to practice therapy harder and longer. To achieve this, the robot must fulfill several requirements: it must have hardware design which provides sufficient capabilities for demonstration of gross and fine motor skills exercises, it must have appropriate character design to be able to establish affective attachment of the child, and it must be able to communicate with children verbally (speech recognition and synthesis), and non-verbally (facial expressions, gestures).

End-User Design of Emotion-Adaptive Dialogue Strategies for Therapeutic Purposes

Two fundamental nontechnical research questions related to the development of emotion-aware dialogue systems are how to identify different kinds of emotional reactions that can be expected to occur in a given interaction domain, and how the system should react to the emotional user behavior. These questions are especially important for dialogue systems used in medical treatment of children with developmental disorders. The paper reports on an adaptive dialogue system that allows the therapist to flexibly design and test dialogue strategies. Our aim was to achieve a balance between the ease-of-use of the system by nontechnical users and the flexibility to adapt the system to different therapeutic settings. The system builds on our previous work, and its functionality is explained by means of example.

Application of dimensional emotion model in automatic emotional speech recognition

This paper reports on the application of the dimensional emotion model in automatic emotional speech recognition. Using the perceptron rule in combination with acoustic features, an approach to speech-based emotion recognition is introduced, which can classify the utterance with respect to the valence-arousal (V-A) dimensions of its emotional content. The mapping of 5 discrete emotion classes onto the 3-class emotional clusters in the V-A space was adopted. Two corpora of acted emotional speech were used to compare recognition results: the Berlin Emotional Speech Database (in German) and the Corpus of Emotional and Attitude Expressive Speech (in Serbian). The experimental results show that the discrimination of emotional speech along the arousal dimension is better than the discrimination along the valence dimension for both corpora.

Encoding of Spatial Perspectives in Human-Machine Interaction

A spatial context is often present in speech-based human-machine interaction, and its role is especially significant in interaction with robotic systems. Studies in the cognitive sciences show that frames of reference used in language and in non-linguistic cognition are correlated. In general, humans may use multiple frames of references. But since the visual sensory modality operates mainly in a relative frame, most of users normally and preferably use relative reference frame in spatial language. Therefore, there is a need to enable dialogue systems to process dialogue acts that instantiate user-centered frames of reference. This paper introduces a cognitively-inspired, computational modeling method that addresses this research question, and illustrates it for a three-party human-machine interaction scenario. The paper also reports on an implementation of the proposed model within a prototype system, and briefly discusses some aspects of the models generalizability and scalability.

A prototype conversational agent embedded in Android-based mobile phones

This paper reports certain aspects of the design and implementation of a prototype conversational agent embedded in Android-based mobile phones. The embedded conversational agent is intended to enable a speech-based interface between the user and the mobile phone, and thus to make the use of the phone more intuitive for nontechnical users. The interaction domain is basic handling of the textual messages and call history. The paper discusses and illustrates the implementation aspects of loading the data from the mobile phone and their representation in the prototype system, and the design aspects of a dialogue strategy.

Toward Conscious-Like Conversational Agents

Although considerable effort has been already devoted to studying various aspects of human-machine interaction, we are still a long way from developing socially believable conversational agents. This paper identifies some of the main causes of the current state in the field: (i) socially believable behaviour of a technical system is misinterpreted as a functional requirement, rather than a qualitative, (ii) the currently prevalent statistical approaches cannot address research problems of managing human-machine interaction that require some sort of contextual analysis, and (iii) the structure of human-machine interaction is unjustifiably reduced to a task structure. In addition, we propose a way to address these pitfalls. We consider the capability of a technical system to simulate fundamental features of human consciousness as one of the key desiderata to perform socially believable behaviour. In line with this, the paper discusses the possibilities for the computational realization of (iv) unified interpretation, (v) learning through interaction, and (vi) context-dependent perception in the context of human-machine interaction.