Emilia I. Barakova

Communicating emotions and mental states to robots in a real time parallel framework using Laban movement analysis

This paper presents a parallel real time framework for emotions and mental states extraction and recognition from video fragments of human movements. In the experimental setup human hands are tracked by evaluation of moving skin-colored objects. The tracking analysis demonstrates that acceleration and frequency characteristics of the traced objects are relevant for classification of the emotional expressiveness of human movements. The outcomes of the emotional and mental states recognition are cross-validated with the analysis of two independent certified movement analysts (CMAs) who use the Laban movement analysis (LMA) method. We argue that LMA based computer analysis can serve as a common language for expressing and interpreting emotional movements between robots and humans, and in that way it resembles the common coding principle between action and perception by humans and primates that is embodied by the mirror neuron system. The solution is part of a larger project on interaction between a human and a humanoid robot with the aim of training social behavioral skills to autistic children with robots acting in a natural environment.

Expressing and interpreting emotional movements in social games with robots

This paper provides a framework for recording, analyzing and modeling of 3 dimensional emotional movements for embodied game applications. To foster embodied interaction, we need interfaces that can develop a complex, meaningful understanding of intention—both kinesthetic and emotional—as it emerges through natural human movement. The movements are emulated on robots or other devices with sensory-motor features as a part of games that aim improving the social interaction skills of children. The design of an example game platform that is used for training of children with autism is described since the type of the emotional behaviors depends on the embodiment of the robot and the context of the game. The results show that quantitative movement parameters can be matched to emotional state of the embodied agent (human or robot) using the Laban movement analysis. Emotional movements that were emulated on robots using this principle were tested with children in the age group 7–9. The tests show reliable recognition on most of the behaviors.

Social training of autistic children with interactive intelligent agents

The ability of autistic children to learn by applying logical rules has been used widely in behavioral therapies for social training. We propose to teach social skills to autistic children through games that simultaneously stimulate social behavior and include recognition of elements of social interaction. For this purpose we created a multi-agent platform of interactive blocks, and we created appropriate games that require shared activities leading to a common goal. The games included perceiving and understanding elements of social behavior that non-autistic children can recognize. We argue that the importance of elements of social interaction such as perceiving interaction behaviors and assigning metaphoric meanings has been overlooked, and that they are very important in the social training of autistic children. Two games were compared by testing them with users. The first game focused only on the interaction between the agents and the other combined interaction between the agents and metaphoric meanings that are assigned to them. The results show that most of the children recognized the patterns of interaction as well as the metaphors when they were demonstrated through embodied agents and were included within games having features that engage the interest of this user group. The results also show the potential of the platform and the games to influence the social behavior of the children positively.

Using an emergent system concept in designing interactive games for autistic children

This paper features the design process, the outcome, and preliminary tests of an interactive toy that expresses emergent behavior and can be used for behavioral training of autistic children, as well as for an engaging toy for every child. We exploit the interest of the autistic children in regular patterns and order to stimulate their motivational, explorative and social skills. As a result we have developed a toy that consists of undefined number of cubes that express emergent behavior by communicating with each other and changing their colors as a result of how they have been positioned by the players. The user tests have shown increased time of engagement of the children with the toy in comparison with their usual play routines, pronounced explorative behavior and encouraging results with improvement of turn taking interaction.

Long-term LEGO therapy with humanoid robot for children with ASD

To utilise the knowledge gained from highly specialised domains as autism therapy to robot-based interactive training platforms, an innovative design approach is needed. We present the process of content creation and co-design of LEGO therapy for children with autism spectrum disorders performed by a humanoid robot. The co-creation takes place across the disciplines of autism therapy, and behavioural robotics, and applies methods from design and human-robot interaction, in order to connect state-of-the-art developments in these disciplines. We designed, carried out and analyzed a pilot and final experiment, in which a robot mediated LEGO therapy between pairs of children was mediated by a robot over the course of 10 to 12 sessions. The impact of the training on the children was then analysed from a clinical and human-robot interaction perspective. Our major findings are as follows: first, game-based robot scenarios in which the game continues over the sessions opened possibilities for long-term interventions using robots and led to a significant increase in social initiations during the intervention in natural settings; and second, including dyadic interactions between robot and child within triadic games with robots has positive effects on the childrens engagement and on creating learning moments that comply with the chosen therapy framework.

Trends in measuring human behavior and interaction

Present day applications demand that behavioral measurements are performed in natural environments, where the measuring devices are thoroughly integrated into everyday objects and activities. Understanding users’ behavior in different contexts could be a goal of measuring human behavior, as well as a means for designing user experiences that utilize on pervasive measuring technology. The aim of this thematic issue on ‘Measuring human behavior and interaction’ is to summarize emerging trends and common problems across different branches of social, behavioral sciences, and interaction design that involve measurements of human behavior. It is inspired by the Measuring Behavior 2010 conference [1], which provided a broad and interdisciplinary forum for novel methods to define, measure, and analyze human, animal, and machine behavior. The papers in this thematic issue of Personal and Ubiquitous Computing are a selection of work presented at or related to the scope of the conference, but also relevant for the readers of PUC journal. One increasingly important issue in the measurement of behavior in computer (or machine)–human interaction is that of the ecological relevance of a study. If a test participant is asked to perform a task in a laboratory environment, then the behavior might be measured almost 100% correctly, but have limited meaning for real-life settings. On the other hand, measurements carried out in the subject’s natural environment are often hampered by the lack of controlled conditions, inadequate replication, and all sorts of technical issues caused by the difficulties of mobile and active subjects. Several of the papers in this thematic issue address that problem in various ways. Maly et al. show how the problem can be addressed by integrating and visualizing several different sources of data recorded from mobile subjects, giving much better power of interpretation than a narrow range of measurements. Their experiment measured navigation of visually impaired subjects in large-scale complex real-world environments. Navigation in such environments is qualitatively different from that of artificial small-scale environments, so laboratory or virtual reality studies have a large risk that their results will be without explanatory power in the real world. Kukka et al. present a study measured in an entirely natural situation, where the use of information services provided to people in a city center was measured, and despite the lack of laboratory conditions, information about the strategies of information seekers could be derived. As in many behavioral studies, that one also proved that the behavior of the people studies was actually quite different from what the subjects themselves said that their behavior would be. Studies in natural environments do not have to be on a large city-wide scale. Zillmer’s paper shows the insights, which can be gained in an oral care campaign by the automated measurement of people brushing their teeth. Such studies are often carried out purely by questionnaire E. I. Barakova (&) Department of Industrial Design, Eindhoven University of Technology, Eindhoven, The Netherlands e-mail: e.i.barakova@tue.nl

User-friendly robot environment for creation of social scenarios

This paper proposes a user-friendly framework for designing robot behaviors by users with minimal understanding of programming. It is a step towards an end-user platform which is meant to be used by domain specialists for creating social scenarios, i.e. scenarios in which not high precision of movement is needed but frequent redesign of the robot behavior is a necessity. We show by a hand shaking experiment how convincing it is to construct robot behavior in this framework.

Engaging autistic children in imitation and turn-taking games with multiagent system of interactive lighting blocks

In this paper game scenarios that aim to establish elements of cooperative play such as imitation and turn taking between children with autism and a caregiver are investigated. Multiagent system of interactive blocks is used to facilitate the games. The training elements include verbal description followed by imitation of video-modeled play episodes. By combining this method with the tangible multiagent platform of interactive blocks (i-blocks) children with autism could imitate play episodes that involved turn taking with a caregiver. The experiment showed that most of the children managed to imitate the play scenarios after video modeling, and repeat the behaviors with the tangible and appealing block platform. When all the actions were well understood by the autistic children, they performed willingly turn taking cooperative behaviors, which they normally do not do.

Mirror neuron framework yields representations for robot interaction

Common coding is a functional principle that underlies the mirror neuron paradigm. It insures actual parity between perception and action, since the perceived and performed actions are equivalently and simultaneously represented within the mirror neuron system. Based on the parity of this representation we show how the mirror neuron system may facilitate the interaction between two robots. Synchronization between neuron groups in different structures of the mirror neuron system are on the basis of the interaction behavior. The robotic simulation is used to illustrate several interactions. The resulting synchronization and turn taking behaviors show the potential of the mirror neuron paradigm for designing of socially meaningful behaviors.

Improving Collaborative Play between Children with Autism Spectrum Disorders and Their Siblings: The Effectiveness of a Robot-Mediated Intervention Based on Lego® Therapy.

The aim of the study was to investigate the effectiveness of a brief robot-mediated intervention based on Lego® therapy on improving collaborative behaviors (i.e., interaction initiations, responses, and play together) between children with ASD and their siblings during play sessions, in a therapeutic setting. A concurrent multiple baseline design across three child–sibling pairs was in effect. The robot-intervention resulted in no statistically significant changes in collaborative behaviors of the children with ASD. Despite limited effectiveness of the intervention, this study provides several practical implications and directions for future research.