Minoru Asada

Incremental Coevolution With Competitive and Cooperative Tasks in a Multirobot Environment

Coevolution has been receiving increased attention as a method for simultaneously developing the control structures of multiple agents. Our ultimate goal is the mutual development of skills through coevolution. The coevolutionary process is, however, often prone to settle into suboptimal strategies. The key to successful coevolution has thus far been unclear. This paper discusses how several robots can emerge cooperative and competitive behavior through coevolutionary processes. In order to realize successful coevolution, we propose two ideas: multiple schedules for incremental evolution and fitness sharing based on the method of importance sampling. To examine this issue, we conducted a series of computer simulations. We have chosen a simplified soccer game consisting of two or three robots as a testbed for analyzing a problem in which both competitive and cooperative tasks are involved. We show that the proposed fitness evaluation allows robots to evolve robust behaviors in cooperative and competitive situations

Modular Learning Systems for Behavior Acquisition in Multi-Agent Environment

There has been a great deal of research on reinforcement learning in multirobot/agent environments during last decades1. A wide range of applications, such as forage robots (Mataric, 1997), soccer playing robots (Asada et al., 1996), prey-pursuing robots (Fujii et al., 1998) and so on, have been investigated. However, a straightforward application of the simple reinforcement learning method to multi-robot dynamic systems has a lot of issues, such as uncertainty caused by others, distributed control, partial observability of internal states of others, asynchronous action taking, and so on. In this paper we mainly focus on two major difficulties in practical use : unstable dynamics caused by policy alternation of other agents curse of dimension problem The policy alternation of others in multi-agent environments may cause sudden changes in state transition probabilities of which constancy is needed for behavior learning to converge. Asada et al. (Asada et al., 1999) proposed a method that sets a global learning schedule in which only one agent is specified as a learner with the rest of the agents having fixed policies to avoid the issue of the simultaneous learning. As a matter of course, they did not consider the alternation of the opponent’s policies. Ikenoue et al. (Ikenoue et al., 2002) showed simultaneous cooperative behavior acquisition by fixing learners’ policies for a certain period during the learning process. In the case of cooperative behavior acquisition, no agent has any reason to change policies while they continue to acquire positive rewards as a result of their cooperative behavior with each other. The agents update their policies gradually so that the state transition probabilities can be regarded as almost fixed from the viewpoint of the other learning agents. Kuhlmann and Stone (Kuhlmann and Stone, 2004) have applied a reinforcement learning system with a function approximator to the keepaway problem in the situation of the RoboCup simulation league. In their work, only the passer learns his policy is to keep the ball away from the opponents. The other agents (receivers and opponents) follow fixed policies given by the designer beforehand. The amount of information to be handled in multi-agent system tends to be huge and easily causes the curse of dimension problem. Elfwing et al. (Elfwing et al., 2004) achieved the cooperative behavior learning task between two robots in real time by introducing the

Identification and Evaluation of the Face System of a Child Android Robot Affetto for Surface Motion Design

Faces of android robots are one of the most important interfaces to communicate with humans quickly and effectively, as they need to match the expressive capabilities of the human face, it is no wonder that they are complex mechanical systems containing inevitable non-linear and hysteresis elements derived from their non-rigid components. Identifying the input-output response properties of this complex system is necessary to design surface deformations accurately and precisely. However, to date, android faces have been used without careful system identification and thus remain black boxes. In this study, the static responses of three-dimensional displacements were investigated for 116 facial surface points against a discrete trapezoidal input provided to each actuator in the face of a child-type android robot Affetto. The results show that the response curves can be modeled with hysteretical sigmoid functions, and that the response properties of the face actuators, including sensitivity, hysteresis, and dyssynchrony, were quite different. The paper further proposes a design methodology for surface motion patterns based on the obtained response models. Design results thus obtained indicate that the proposed response properties enable us to predict the design results, and that the proposed design methodology can cancel the differences among the response curves of the actuators. The proposed identification and quantitative evaluation method can be applied to advanced android face studies instead of conventional qualitative evaluation methodologies.

1A1-S07 Cluster analysis of complex and various skin flow fields around human lips for enriching robot facial expressions

To enrich human-like robot facial expressions, robot designers should know how much and to which directions each human facial part moves for various facial expressions. However, combinations of facial part movements are too complex to determine appropriate design parameters of robot facial deformation mechanisms such as the number, locations, and orientations of actuators for realizing intended reproducibility of each human facial expression. This paper proposes an analysis method of various human facial expressions to determine the design parameters for human-like face robots. The proposed method superimposes several measured human skin flow fields, and extracts the direction of the first principal component of the motion vectors for each facial marker. Its contribution rate was 86%, which indicates each facial part moves almost only in one direction even when a human exhibits several facial expressions. Then, the proposed method estimates which face area moves similarly through cluster analysis of a vector field of the first principal component vector of each facial marker.

2A2-D18 Learning Dynamic Throwing Motion of A Wheeled Inverted Pendulum utilizing Whole Body Dynamics

We apply reinforcement learning to a wheeled inverted pendulum robot that acquires dynamic throwing motion utilizing whole body dynamics. Large number of parameters are needed to be calibrated so that the robot becomes able to throw a ball far away utilizing its own body dynamics while it keeps standing. We investigated the learning process of the throwing motion by application of a policy gradient method with a dynamics simulater.

Behavior Acquisition in RoboCup Middle Size League Domain

The RoboCup middle size league is one of the leagues that have the longest histories in RoboCup. This league has unique features, for example, bigger robots (around 45cm square) plays on the largest field (say, 18m×12m in 2007), any global sensory system is not allowed to use, all robots have on-board vision systems and controllers. Each robot plays based on its own sensory information, and it can share some information with teammates and a coach box located outside the playing field over wireless communication, then, shows some cooperative behaviors among them during the game. This chapter briefly introduces research activities in RoboCup middle size league. A variety of research topics have been attacked in this league. Some of them are common to other real robot leagues such as small size and 4-legged leagues. For example, robust real-time onboard vision system, precise localization based on vision system, and design of cooperative behavior are actively investigated in RoboCup middle size league. On the other hand, skill and cooperative/competitive behavior acquisition/emergence based on machine learning techniques is also well-studied. The latter is focused on in this chapter. First, a purposive behavior acquisition of a single robot based on machine learning technique is introduced. Reinforcement learning is one of machine learning techniques and extensively studied to be applied to acquisition of robot behavior like shooting a ball into a goal. It has a simple framework and algorithm to be applied to robots however some difficulties exist in practical use because of its simplicity. In order to overcome these problems, some modular learning and hierarchical systems have been proposed. Not only reinforcement learning but also evolutional methods have been investigated as well. Some examples will be shown. Next, studies on cooperative/competitive behavior acquisition based on machine learning techniques are introduced. Application of machine learning to multi-agent system usually has some difficulties because of complex dynamics of the system. The complexity is induced by decision making by multi-players, growing amount of information to decide an action by an individual, perceptual aliasing, and so on. In order to reduce the complexity, wireless communication between teammates is commonly used. In case of unavailability of communication between players, for example, lack of communication with opponents,

2P1-B07 Inferring other's intention based on estimated state value of self

Recognition of other agent intention in a multiagent environment is a very important issue to realize social activities, for example, imitation learning, understanding intention, cooperative/competitive behavior, and so on. Conventional approaches to infer the other agent intention need a precise trajectory in Cartesian or joint space that is sometimes hard to measure from the viewpoint of an observer. It is also difficult to estimate a same intention but with different realizations because they try to match just a certain trajectory during the trial. We propose a novel method of inference of other agent’s intention based on state value estimation. The method does not need a precise world model or coordination transformation system to deal with view dependency. This paper shows an observer can infer an intention of other not by precise object trajectory in Cartesian space but by estimated state value transition during the observed behavior.