Tadashi Horiuchi

Fuzzy interpolation-based Q-learning with continuous states and actions

This paper proposes a new method of Q-learning where fuzzy inference is introduced to calculate the Q-function that evaluates the state/action pairs so as to enable us to deal with continuous-valued pairs and continuous-valued states and actions. In this method, the Q-function is updated using the steepest descent method. Our proposed method is applied to a cart-pole balancing system, which demonstrates considerable improvements in its control performance with the aid of the fuzzy inference.

A NOVEL HYBRID FRAMEWORK OF COEVOLUTIONARY GA AND MACHINE LEARNING

In this paper, we will propose a novel framework of hybridization of Coevolutionary Genetic Algorithm and Machine Learning. The Coevolutionary Genetic Algorithm (CGA) which has already been proposed by Handa et al. consists of two GA populations: the first GA (H-GA) population searches for the solutions in given problems, and the second GA (P-GA) population searches for effective schemata of the H-GA. The CGA adopts the notion of commensalism, a kind of co-evolution. The new hybrid framework incorporates a schema extraction mechanism by Machine Learning techniques into the CGA. Considerable improvement in its search ability is obtained by extracting more efficient and useful schemata from the H-GA population and then by incorporating those extracted schemata into the P-GA. We will examine and compare two kinds of machine learning techniques in extracting schema information: C4.5 and CN2. Several computational simulations on multidimensional knapsack problems, constraint satisfaction problems and function optimization problems will reveal the effectiveness of the proposed methods.

Fuzzy interpolation-based Q-learning with profit sharing plan scheme

We have previously (1996) proposed fuzzy interpolation-based Q-learning where fuzzy rules are used to represent Q-function (action utility function), in order to enable us to treat continuous-valued states and actions. In this paper, we will introduce the idea of profit sharing plan (PSP) used in classifier systems into the fuzzy interpolation-based Q-learning in order to accelerate the speed of learning and will discuss its effectiveness through applications to control problems such as cart-pole balancing problems.

Coevolutionary GA with schema extraction by machine learning techniques and its application to knapsack problems

The authors introduce a novel coevolutionary genetic algorithm with schema extraction by machine learning techniques. Our CGA consists of two GA populations: the first GA (H-GA) searches for the solutions in the given problems and the second GA (P-GA) searches for effective schemata of the H-GA. We aim to improve the search ability of our CGA by extracting more efficiently useful schemata from the H-GA population, and then incorporating those extracted schemata in a natural manner into the P-GA. Several computational simulations on multidimensional knapsack problems confirm the effectiveness of the proposed method.

A two-stage self-organizing map with threshold operation for data classification

This paper presents a two-stage self-organizing map algorithm with threshold operation. Kohonens basic SOM algorithm (BSOM) is simple and effective for data classification problems of high-dimensional data. But inactivated cells appear for specific input data and it causes to decline the ability of data classification. In order to solve this problem, BSOM with threshold operation (THSOM) was proposed recently. The THSOM algorithm, however, tends to loose topological structure of input data. Our two-stage self-organizing map algorithm inherits both good points of BSOM and THSOM. Numerical simulations reveal that the two-stage SOM can achieve small clustering error and high topology preservation in comparison with BSOM and THSOM.

A Study on Two-stage Self-Organizing Map suitable for Clustering Problems

This paper presents a two-stage self-organizing map algorithm what we call two-stage SOM which combines Kohonens basic SOM (BSOM) and Aokis SOM with threshold operation (THSOM). In the first stage of Two-stage SOM, we use BSOM algorithm in order to acquire topological structure of input data, and then we apply THSOM algorithm so that inactivated code-vectors move to appropriate region reflecting the distribution of the input data. Furthermore, we show that two-stage SOM can be applied to clustering problems. Some experimental results reveal that Two-stage SOM is effective for clustering problems in comparison with conventional methods

A study on skill acquisition in trailer-truck steering problem by reinforcement learning

This paper presents an attempt to apply reinforcement learning to a trailer-truck steering problem as one of the skill acquisition problems. Because the learning agent in this problem needs to learn long sequences of actions to reach the goal, it is necessary for the agent to acquire proficient skills for steering. We construct the simulation environment for the problem and try to acquire the steering operations by reinforcement learning. Furthermore, two kinds of action selection methods, Boltzmann selection and e-greedy selection, are examined to reveal the difference between them through the simulation experiments.

Fuzzy Control as Self-Organizing Adaptive Constraint-Oriented Problem Solving

By introducing the notion of constraint-oriented fuzzy inference, we will show that it provides us ways of fuzzy control methods that has abilities of adaptation, learning and self-organization. The basic supporting techniques behind these abilities are “hard” processing by Artificial Intelligence or traditional computational framework and “soft” processing by Neural Network, Genetic Algorithm and Reinforcement Learning techniques. In the former processing, Qualitative Reasoning and Instance Generalization by Symbolic Reasoning play important role, while by the latter processing, fuzzy control becomes capable of learning, adaptation and evolutional self-organization.

A study on vision-based mobile robot learning by deep Q-network

Deep Q-network (DQN) is one of the most famous methods of deep reinforcement learning. DQN approximates the action value function by Convolutional Neural Network (CNN) and updates the action value function by Q-learning. In this research, we apply DQN to robot behavior learning the simulation environment. We realize that the mobile robot learns to acquire good behaviors such as avoiding the wall and moving along the center line by using high-dimensional visual information as input data. We propose a method which reuses the best target network so far in case learning performance suddenly falls. Moreover, we incorporate Profit Sharing method to DQN in order to accelerate the learning. Through the simulation experiment, we confirm the effectiveness of our method.

Experimental Study on Behavior Acquisition of Mobile Robot by Deep Q-Network

∗Graduate School of Information Science, Nara Institute of Science and Technology 8916-5 Takayama-cho, Ikoma, Nara 630-0192, Japan E-mail: sasaki.hikaru.rw3@is.naist.ac.jp ∗∗Department of Control Engineering, National Institute of Technology, Matsue College 14-4 Nishi-ikuma, Matsue, Shimane 690-8518, Japan E-mail: horiuchi@matsue-ct.ac.jp ∗∗∗Department of Information Engineering, National Institute of Technology, Matsue College 14-4 Nishi-ikuma, Matsue, Shimane 690-8518, Japan E-mail: kato@matsue-ct.ac.jp