Pedro Sequeira

Emotion-based intrinsic motivation for reinforcement learning agents

In this paper, we propose an adaptation of four common appraisal dimensions that evaluate the relation of an agent with its environment into reward features within an intrinsically motivated reinforcement learning framework. We show that, by optimizing the relative weights of such features for a given environment, the agents attain a greater degree of fitness while overcoming some of their perceptual limitations. This optimization process resembles the evolutionary adaptive process that living organisms are subject to. We illustrate the application of our method in several simulated foraging scenarios.

ION Framework --- A Simulation Environment for Worlds with Virtual Agents

Agents cannot be decoupled from their environment. An agent perceives and acts in a world and the model of the world influences how the agent makes decisions. Most systems with virtual embodied agents simulate the environment within a specific realization engine such as the graphics engine. As a consequence, these agents are bound to a particular kind of environment which compromises their reusability across different applications. We propose the ION Framework, a framework for simulating virtual environments which separates the simulation environment from the realization engine. In doing so, it facilitates the integration and reuse of the several components of the system. The ION Framework was used to create several 3D virtual worlds populated with autonomous embodied agents that were tested with hundreds of users.

What can i do with this?: finding possible interactions between characters and objects

Virtual environments are often populated by autonomous synthetic agents capable of acting and interacting with other agents as well as with humans. These virtual worlds also include objects that may have different uses and types of interactions. As such, these agents need to identify possible interactions with the objects in the environment and measure the consequences of these interactions. This is particularly difficult when the agents never interacted with some of the objects beforehand. This paper describes SOTAI - Smart ObjecT-Agent Interaction, a framework that will help agents to identify possible interactions with unknown objects based on their past experiences. In SOTAI, agents can learn world regularities, like object attributes and frequent relations between attributes. They gather qualitative symbolic descriptions from their sensorial data when interacting with objects and perform inductive reasoning to acquire concepts about them. We implemented an initial case study and the results show that our agents are able to acquire valid conceptual knowledge.

Learning by appraising: an emotion-based approach to intrinsic reward design

In this paper, we investigate the use of emotional information in the learning process of autonomous agents. Inspired by four dimensions that are commonly postulated by appraisal theories of emotions, we construct a set of reward features to guide the learning process and behaviour of a reinforcement learning (RL) agent that inhabits an environment of which it has only limited perception. Much like what occurs in biological agents, each reward feature evaluates a particular aspect of the (history of) interaction of the agent history with the environment, thereby, in a sense, replicating some aspects of appraisal processes observed in humans and other animals. Our experiments in several foraging scenarios demonstrate that by optimising the relative contributions of each reward feature, the resulting “emotional” RL agents perform better than standard goal-oriented agents, particularly in consideration of their inherent perceptual limitations. Our results support the claim that biological evolutionary adaptive mechanisms such as emotions can provide crucial clues in creating robust, general-purpose reward mechanisms for autonomous artificial agents, thereby allowing them to overcome some of the challenges imposed by their inherent limitations.

Emergence of Emotional Appraisal Signals in Reinforcement Learning Agents

The positive impact of emotions in decision-making has long been established in both natural and artificial agents. In the perspective of appraisal theories, emotions complement perceptual information, coloring our sensations and guiding our decision-making. However, when designing autonomous agents, is emotional appraisal the best complement to the perceptions? Mechanisms investigated in affective neuroscience provide support for this hypothesis in biological agents. In this paper, we look for similar support in artificial systems. We adopt the intrinsically motivated reinforcement learning framework to investigate different sources of information that can guide decision-making in learning agents, and an evolutionary approach based on genetic programming to identify a small set of such sources that have the largest impact on the performance of the agent in different tasks, as measured by an external evaluation signal. We then show that these sources of information: (i) are applicable in a wider range of environments than those where the agents evolved; (ii) exhibit interesting correspondences to emotional appraisal-like signals previously proposed in the literature, pointing towards our departing hypothesis that the appraisal process might indeed provide essential information to complement perceptual capabilities and thus guide decision-making.

Emerging social awareness: Exploring intrinsic motivation in multiagent learning

Recently, a novel framework has been proposed for intrinsically motivated reinforcement learning (IMRL) in which a learning agent is driven by rewards that include not only information about what the agent must accomplish in order to “survive”, but also additional reward signals that drive the agent to engage in other activities, such as playing or exploring, because they are “inherently enjoyable”. In this paper, we investigate the impact of intrinsic motivation mechanisms in multiagent learning scenarios, by considering how such motivational system may drive an agent to engage in behaviors that are “socially aware”. We show that, using this approach, it is possible for agents to learn individually to acquire socially aware behaviors that tradeoff individual well-fare for social acknowledgment, leading to a more successful performance of the population as a whole.

FearNot! demo: a virtual environment with synthetic characters to help bullying

This demo features FearNot!, a school-based Virtual Learning Environment (VLE) populated by synthetic characters representing the various actors in a bullying scenario. FearNot! uses emergent narrative to create improvised dramas with those characters. The goal is to enable children to explore bullying issues, and coping strategies, interacting with characters to which they become affectively engaged. Through their appearance, behaviours and affect, these characters are able to trigger empathic relations with the user. FearNot! is used for Personal and Health Social Education (PHSE) for children aged 8--12, in the UK, Portugal and Germany.

Can a child feel responsible for another in the presence of a robot in a collaborative learning activity

In order to explore the impact of integrating a robot as a facilitator in a collaborative activity, we examined interpersonal distancing of children both with a human adult and a robot facilitator. Our scenario involves two children performing a collaborative learning activity, which included the writing of a word/letter on a tactile tablet. Based on the learning-by-teaching paradigm, one of the children acted as a teacher when the other acted as a learner. Our study involved 40 children between 6 and 8 years old, in two conditions (robot or human facilitator). The results suggest first that the child acting as a teacher feel more responsible when the facilitator is a robot, compared to a human; they show then that the interaction between a (teacher) child and a robot facilitator can be characterized as being a reciprocity-based interaction, whereas a human presence fosters a compensation-based interaction.

Children's peer assessment and self-disclosure in the presence of an educational robot

Research in education has long established how children mutually influence and support each others learning trajectories, eventually leading to the development and widespread use of learning methods based on peer activities. In order to explore childrens learning behavior in the presence of a robotic facilitator during a collaborative writing activity, we investigated how they assess their peers in two specific group learning situations: peer-tutoring and peer-learning. Our scenario comprises of a pair of children performing a collaborative activity involving the act of writing a word/letter on a tactile tablet. In the peer-tutoring condition, one child acts as the teacher and the other as the learner, while in the peer-learning condition, both children are learners without the attribution of any specific role. Our experiment includes 40 children in total (between 6 and 8 years old) over the two conditions, each time in the presence of a robot facilitator. Our results suggest that the peer-tutoring situation leads to significantly more corrective feedback being provided, as well as the children more disposed to self-disclosure to the robot.

Discovering Social Interaction Strategies for Robots from Restricted-Perception Wizard-of-Oz Studies

In this paper we propose a methodology for the creation of social interaction strategies for human-robot interaction based on restricted-perception Wizard-of-Oz studies (WoZ). This novel experimental technique involves restricting the wizards perceptions over the environment and the behaviors it controls according to the robots inherent perceptual and acting limitations. Within our methodology, the robots design lifecycle is divided into three consecutive phases, namely data collection, where we perform interaction studies to extract expert knowledge and interaction data; strategy extraction, where a hybrid strategy controller for the robot is learned based on the gathered data; strategy refinement, where the controller is iteratively evaluated and adjusted. We developed a fully-autonomous robotic tutor based on the proposed approach in the context of a collaborative learning scenario. The results of the evaluation study show that, by performing restricted-perception WoZ studies, our robots are able to engage in very natural and socially-aware interactions.