Sachiyo Arai

Experience-based reinforcement learning to acquire effective behavior in a multi-agent domain

In this paper, we discuss Profit-sharing, an experience-baised reinforcement learning approach (which is similar to a Monte-Carlo based reinforcement learning method) that can be used to learn robust and effective actions within uncertain, dynamic, multi-agent systems. We introduce the cut-loop routine that discards looping behavior, and demonstrate its effectiveness empirically within a simplified NEO (non-combatant evacuation operation) domain. This domain consists of several agents which ferry groups of evacuees to one of several shelters. We demonstrate that the cut-loop routine makes the Profit-sharing approach adaptive and robust within a dynamic and uncertain domain, without the need for predefined knowledge or subgoals. We also compare it empirically with the popular Q-learning approach.

Multi-agent reinforcement learning for planning and scheduling multiple goals

Recently, reinforcement learning has been proposed as an effective method for knowledge acquisition of multiagent systems. However, most research on multiagent systems applying a reinforcement learning algorithm, focus on a method to reduce complexity due to the existence of multiple agents and goals. Although these pre-defined structures succeeded in lessening the undesirable effect due to the existence of multiple agents, they would also suppress the desirable emergence of cooperative behaviors in the multiagent domain. We show that the potential cooperative properties among the agent are emerged by means of profit-sharing (J. Grefenstette, 1988; K. Miyazaki et al., 1994) which is robust in the non-MDPs.

Guiding inference through relational reinforcement learning

Reasoning plays a central role in intelligent systems that operate in complex situations that involve time constraints. In this paper, we present the Adaptive Logic Interpreter, a reasoning system that acquires a controlled inference strategy adapted to the scenario at hand, using a variation on relational reinforcement learning. Employing this inference mechanism in a reactive agent architecture lets the agent focus its reasoning on the most rewarding parts of its knowledge base and hence perform better under time and computational resource constraints. We present experiments that demonstrate the benefits of this approach to reasoning in reactive agents, then discuss related work and directions for future research.

Semantic Web Service Architecture Using Multi-agent Scenario Description

Current research issues on web services have come to center around flexible composition of existing services. Under the initiative of industry, flexible composition framework has been developed on a workflow model where flow of the processes and bindings among services should be known beforehand. In short, its framework realizes flexible composition within the available services that are not widely opened. This paper focuses on two limitations of current web service composition. One limitation is that it’s hard to represent multi-agent scenarios consisting of several concurrent processes because it is based on a workflow model. The other limitation is that once composed, web service cannot be reused or transferrd for other requesters, because there is no function to put the semantics on composite web service. To overcome these limitations, we have developed scenario description language Q, which enables us to realize a web service composition reflecting a multi-agent’s context not as a workflow but a scenario. Furthermore, we developed a system that translates multi-agent scenario to DAML-S, and that registers the translated DAML-S as a new Web service. We also discuss the availability of our system for designing an application to C-Commerce and Digital Cities.

Multi-agent reinforcement learning for planning and conflict resolution in a dynamic domain

1.1 Problem Domain Non-combatant evacuation operations, or NEOs, have been used to test a variety of coordination strategies. Though real-world NEOs have many constraint and resource conflicts, the domain used in this study models multiple transportation vehicles which transfer groups of evacuees to safe shelters. Each transport is operated asynchronously by an autonomous agent, which makes its own decision based on locally available information.

Teamwork formation for keepaway in robotics soccer (reinforcement learning approach)

In this paper, we discuss guidelines for a reward design problem that defines when and what amount of reward should be given to the agents, within the context of reinforcement learning approach. We take keepaway soccer as a standard task of multiagent domain which requires skilled teamwork. The difficulties of designing reward for good teamwork are due to its features as follows: i) since it is a continuing task which has no explicit goal, it is hard to tell when reward should be given to the agents, ii) since it is a multiagent cooperative task, it is hard to make a fair share of the reward for each agent’s contribution. Through some experiments, we show that reward design have a major effect on the agent’s behavior, and introduce the reward function that makes agents perform keepaway successfully.

Encouragement of Right Social Norms by Inverse Reinforcement Learning

This study is intended to encourage appropriate social norms among multiple agents. Effective norms, such as those emerging from sustained individual interactions over time, can make agents act cooperatively to optimize their performance. We introduce a “social learning” model in which agents mutually interact under a framework of the coordination game. Because coordination games have dual equilibria, social norms are necessary to make agents converge to a unique equilibrium. As described in this paper, we present the emergence of a right social norm by inverse reinforcement learning, which is an approach for extracting a reward function from the observation of optimal behaviors. First, we let a mediator agent estimate the reward function by inverse reinforcement learning from the observation of a master’s behavior. Secondly, we introduce agents who act according to an estimated reward function in the multiagent world in which most agents, called citizens, have no way to act. Finally, we evaluate the effectiveness of introducing inverse reinforcement learning.

Learning for human-agent collaboration on the semantic Web

Semantic Web is a challenging framework to make Web information machine readable or understandable, but it seems not enough to make humans requirements for collecting and utilizing information automatically. The Agent technology becomes hopeful approach to bridge the gap between humans and machines. Agents may be autonomous and intelligent entities that may travel among agents and human. They get the requirements from human or other agents, and offer an appropriate solution through consulting among them. The main difference between agent and ordinary software development is the issue of coordination, cooperation and learning. This issue is very important for utilizing the Web information. We attempt to give an overview and research challenges with respect to the combination of machine learning and agent technologies with semantic Web from the perspective of interaction as well as interoperability among agents and humans.

Estimation of Reward Function Maximizing Learning Efficiency in Inverse Reinforcement Learning.

Inverse Reinforcement Learning (IRL) is a promising framework for estimating a reward function given the behavior of an expert.However, the IRL problem is ill-posed because infinitely many reward functions can be consistent with the expert’s observed behavior. To resolve this issue, IRL algorithms have been proposed to determine alternative choices of the reward function that reproduce the behavior of the expert, but these algorithms do not consider the learning efficiency. In this paper, we propose a new formulation and algorithm for IRL to estimate the reward function that maximizes the learning efficiency. This new formulation is an extension of an existing IRL algorithm, and we introduce a genetic algorithm approach to solve the new reward function. We show the effectiveness of our approach by comparing the performance of our proposed method against existing algorithms.

Estimation of the heterogeneous strategies from action log

Agent-based crowd simulation is a widely used technique for designing and evaluating human-in-the-loop situations such as evacuation plans and building design. Valid evaluation requires a correct model of an individual agents action rule, which causes human behavior in a crowd. However, in general, designing a specific action rule of each agent depends strongly on a trial-and-error approach because a real crowd shows diverse behaviors. To avoid trial-and-error approaches, we specifically examine an automated method to estimate an agents strategy to select a goal state from trajectories extracted from a humans action log. The previous method assumes a homogeneous strategy, meaning that all agents have a common strategy, but to reproduce the diversity of a real crowd it is more natural to assume a heterogeneous strategy: not all agents have the same strategy. Our proposed method of estimating individual and different strategies of agents can estimate a heterogeneous strategy that incorporates readability by evolutionary computation, even for trajectories that are the result not only of homogeneous strategies but also of heterogeneous strategies. The experiment results demonstrate the validity of our method. Additionally, some cases exhibiting multiple strategies will be extracted for a single trajectory. They show applicability to actual action log data.