Roland Hafner

Reinforcement learning for robot soccer

Batch reinforcement learning methods provide a powerful framework for learning efficiently and effectively in autonomous robots. The paper reviews some recent work of the authors aiming at the successful application of reinforcement learning in a challenging and complex domain. It discusses several variants of the general batch learning framework, particularly tailored to the use of multilayer perceptrons to approximate value functions over continuous state spaces. The batch learning framework is successfully used to learn crucial skills in our soccer-playing robots participating in the RoboCup competitions. This is demonstrated on three different case studies.

Neural Reinforcement Learning Controllers for a Real Robot Application

Accurate and fast control of wheel speeds in the presence of noise and nonlinearities is one of the crucial requirements for building fast mobile robots, as they are required in the MiddleSize League of RoboCup. We will describe, how highly effective speed controllers can be learned from scratch on the real robot directly. The use of our recently developed neural fitted Q iteration scheme allows reinforcement learning of neural controllers with only a limited amount of training data seen. In the described application, less than 5 minutes of interaction with the real robot were sufficient, to learn fast and accurate control to arbitrary target speeds.

Reinforcement learning on an omnidirectional mobile robot

With this paper we describe a well suited, scalable problem for reinforcement learning approaches in the field of mobile robots. We show a suitable representation of the problem for a reinforcement approach and present our results with a model based standard algorithm. Two different approximators for the value function are used, a grid based approximator and a neural network based approximator.

Making a Robot Learn to Play Soccer Using Reward and Punishment

In this paper, we show how reinforcement learning can be applied to real robots to achieve optimal robot behavior. As example, we enable an autonomous soccer robot to learn intercepting a rolling ball. Main focus is on how to adapt the Q-learning algorithm to the needs of learning strategies for real robots and how to transfer strategies learned in simulation onto real robots.

Learning to dribble on a real robot by success and failure

Learning directly on real world systems such as autonomous robots is a challenging task, especially if the training signal is given only in terms of success or failure (reinforcement learning). However, if successful, the controller has the advantage of being tailored exactly to the system it eventually has to control. Here we describe, how a neural network based RL controller learns the challenging task of ball dribbling directly on our middle-size robot. The learned behaviour was actively used throughout the RoboCup world championship tournament 2007 in Atlanta, where we won the first place. This constitutes another important step within our Brainstormers project. The goal of this project is to develop an intelligent control architecture for a soccer playing robot, that is able to learn more and more complex behaviours from scratch.

Die Brainstormers: Entwurfsprinzipien lernfähiger autonomer Roboter

ZusammenfassungDas “Brainstormers” Projekt wurde 1998 gestartet mit dem Ziel, lernfähige autonome Agenten in komplexen Umgebungen am Beispiel Roboterfußball zu erforschen. Dabei hat die Bearbeitung der vielfältigen Fragestellungen, die sich in dieser sehr dynamischen und verrauschten Umgebung ergeben, zu einer Vielzahl neuartiger Methoden und theoretischer Ergebnisse geführt.

Cognitive concepts in autonomous soccer playing robots

Computational concepts of cognition, their implementation in complex autonomous systems, and their empirical evaluation are key techniques to understand and validate concepts of cognition and intelligence. In this paper we want to describe computational concepts of cognition that were successfully implemented in the domain of soccer playing robots and show the interactions between cognitive concepts, software engineering and real-time application development. Beside a description of the general concepts we will focus on aspects of perception, behavior architecture, and reinforcement learning.