Masanori Takeda

Continuous valued Q-learning for vision-guided behavior acquisition

Q-learning, a most widely used reinforcement learning method, normally needs well-defined quantized state and action spaces to converge. This makes it difficult to be applied to real robot tasks because of poor performance of learned behavior and a further problem of state space construction. This paper proposes a continuous valued Q-learning for real robot applications, which calculates the contribution values for estimating a continuous action value in order to make motion smooth and effective. The proposed method obtained a better performance of desired behavior than the conventional real-valued Q-learning method, with roughly quantized state and action. To show the validity of the method, we applied the method to a vision-guided mobile robot of which the task is to chase a ball. Although the task was simple, the performance was quite impressive. A further improvement is discussed.

Continuous valued Q-learning method able to incrementally refine state space

The conventional reinforcement learning method has problems in applying to real robot tasks, because such method must be able to represent the values in terms of infinitely many states and action pairs. In order to represent an action value function continuously, a function approximation method is usually applied. In our previous work (2000), we pointed out that this type of learning method potentially has a discontinuity problem of optimal actions for a given state. In this paper, we propose a method for estimating where a discontinuity of the optimal action takes place and for refining a state space incrementally. We call this method an continuous valued Q-learning method. To show the validity of our method, we apply the method to a simulated robot.

Improvement Continuous Valued Q-learning and Its Application to Vision Guided Behavior Acquisition

Q-learning, a most widely used reinforcement learning method, normally needs well-defined quantized state and action spaces to converge. This makes it difficult to be applied to real robot tasks because of poor performance of learned behavior and further a new problem of state space construction. We have proposed Continuous Valued Q-learning for real robot applications, which calculates contribution values to estimate a continuous action value in order to make motion smooth and effective [1]. This paper proposes an improvement of the previous work, which shows a better performance of desired behavior than the previous one, with roughly quantized state and action. To show the validity of the method, we applied the method to a vision-guided mobile robot of which task is to chase a ball.