Hajime Murao

Q-Learning with adaptive state segmentation (QLASS)

Q-learning is an efficient algorithm to acquire adaptive behavior of the robot without a priori knowledge of the sensor space and the task. However, there is a problem in applying the Q-learning to the task in the real world-how to construct the state space suitable for the Q-learning without knowledge of the sensor space? In this paper we propose Q-learning with adaptive state segmentation (QLASS). QLASS provides a method to segment the sensor space incrementally, based on sensor vectors and reinforcement signals. Experimental results show that QLASS can segment the sensor space effectively to accomplish the task. Furthermore, we show the obtained state space reveals the fitness landscape.

Modeling and genetic solution of a class of flexible job shop scheduling problems

We consider an extended class of flexible job shop scheduling problems. First, we translate the problem into a mathematical programming formula, i.e., a mixed-integer programming problem. This makes it possible to apply standard packages of mixed integer programming solvers and, while lots of computational time is required in general, to obtain the optimal schedule. Then, in order to seek the schedules close to the optimal for larger-scale problems, we newly design a solution method by adopting genetic algorithms based on the formula. Through some computational experiments, the effectiveness and the possibility of the proposed approach are examined.

An application of branch-and-bound method to deterministic optimization model of elevator operation problems

In this paper, we propose a framework for obtaining the optimal solution of an elevator operation problem by applying branch-and-bound method, where it is assumed that all information about the passengers are given. The problem is solved by determining the assignments of passengers to elevators and the processing order of passengers for each elevator. The validity of an existing rule to decide a car service is examined by comparing the results with the optimal one.

A hybrid neural network system for the rainfall estimation using satellite imagery

Hybrid neural networks composed of a self-organizing map (SOM) and three-layered feedforward neural networks have been developed and applied for rainfall estimation using satellite imagery. The SOM classifies an input vector extracted from satellite imagery, then one of the feedforward neural networks is chosen according to the class to give the rainfall estimation. In order to train the hybrid neural network, adjoining seas of Japan were selected as testing area. Hourly GMS infrared imagery data and simultaneous ground truth data (the area average of rainfall observations and radar/raingage composite data) were collected from AIP/l/sup 2/ data sets. The SOM is trained to classify the textural feature vectors extracted from the imagery data, and tuned by learning vector quantization method. The feedforward neural networks are trained to give the estimation by back propagation algorithm. Fairly good correlation coefficients about 0.8 are obtained between the estimation and corresponding ground truth for the unlearned test set. Furthermore, SOM with a recurrent structure for processing the temporal information has been proposed and tested.

A coevolutionary approach to adapt the genotype-phenotype map in genetic algorithms

This article introduces a coevolutionary approach to genetic algorithms (GAs) for exploring not only within a part of the solution space defined by the genotype-pheno-type map, but also the map itself. In canonical GAs with a fixed map, how large an area of the solution space can be covered by possible genomes, and consequently how better solutions can be found by a GA, rely on how well the geotype-phenotype map in designed, but it is difficult for designers of the algorithms to design the map without a priori knowledge of the solution space. In the proposed algorithm, the genotype-phenotype map is improved adaptively during the search process for solution candidates. It is applied to 3-bit deceptive problems such as of typical combinatorial optimazation problems. These are well known because their difficulty for GAs can be controlled by the genotype-phenotype map, and this shows a fairly good performance compared with a conventional GA.

Walking pattern acquisition for quadruped robot by using modular reinforcement learning

We apply the reinforcement learning to acquire a gait pattern of a quadruped locomotion robot. The advantage of the reinforcement learning for such a problem is that no exact robot model is needed for calculating the prescribed teaching signals, but one simply needs to evaluate the results of trials and generates the reinforcement signals. As a result, the robot can acquire by itself a walking pattern suitable to its structure, dynamics and environments. We use here a tightly coupled modular actor-critic structure with stochastic gradient ascent. The computer simulations show that it could generate various stable walking pattern suitable to the environment and dynamics of the robot. We also apply the proposed method to an experimental real robot and deal with the learning process for getting the walking pattern.

A reinforcement learning with switching controllers for a continuous action space

Reinforcement learning (RL) attracts much attention as a technique for realizing computational intelligence such as adaptive and autonomous decentralized systems. In general, however, it is not easy to put RL to practical use. This difficulty includes the problem of designing a suitable action space for an agent, i.e., satisfying two requirements in trade-off: (i) to keep the characteristics (or structure) of an original search space as much as possible in order to seek strategies that lie close to the optimal, and (ii) to reduce the search space as much as possible in order to expedite the learning process. In order to design a suitable action space adaptively, in this article, we propose a RL model with switching controllers based on Q-learning and an actor-critic to mimic the process of an infant’s motor development in which gross motor skills develop before fine motor skills. Then a method for switching controllers is constructed by introducing and referring to the “entropy.” Further, through computational experiments by using a path-planning problem with continuous action space, the validity and potential of the proposed method have been confirmed.

Online scheduling of a multi-robot system by using genetic algorithms

This paper proposes an online method to schedule the movement of robots in a multi-robot system. The authors introduce a multi-agent based search method to cope with small modifications of the system during the execution of a schedule which is preliminarily obtained by a genetic algorithm (GA). They apply the method to a welding plant, in which a number of seams are processed simultaneously by several robots, but there is no one-to-one relation between the seems and the robots. A GA is used to assign the welded seams to the robots and to schedule the welding order of the seams of each robot in a way which minimizes the overall welding time. The task of the proposed multi-agent based search method is to cope with troubles of the robots and sudden changes of the seams for which a online modification of the schedule is necessary. As a result of computer simulations, the proposed method shows fairly good results for perturbations in the system during the task.

Incremental state acquisition for Q-learning by adaptive Gaussian soft-max neural network

We propose an adaptive Gaussian soft-max neural network to construct a state space suitable for Q-learning to accomplish tasks in continuous sensor space. In the proposed method, a state of Q-learning is defined by a hidden neuron of the neural network which is used to estimate resulting sensor signals of actions. The learning agent starts with single state covering whole sensor space and a new state is generated incrementally by adding a new hidden neuron when difference between the estimated sensor signal and incoming one exceeds a given threshold. Simulation results show that the proposed algorithm is able to construct the sensor space effectively to accomplish the task.

Reinforcement learning for dynamic environment: a classification of dynamic environments and a detection method of environmental changes

Engineers and researchers are paying more attention to reinforcement learning (RL) as a key technique for realizing computational intelligence such as adaptive and autonomous decentralized systems. In general, it is not easy to put RL into practical use. In prior research our approach mainly dealt with the problem of designing state and action spaces and we have proposed an adaptive co-construction method of state and action spaces. However, it is more difficult to design state and action spaces in dynamic environments than in static ones. Therefore, it is even more effective to use an adaptive co-construction method of state and action spaces in dynamic environments. In this paper, our approach mainly deals with a problem of adaptation in dynamic environments. First, we classify tasks of dynamic environments and propose a detection method of environmental changes to adapt to dynamic environments. Next, we conducted computational experiments using a so-called “path planning problem” with a slowly changing environment where the aging of the system is assumed. The performances of a conventional RL method and the proposed detection method were confirmed.