Gregory Kuhlmann

Reinforcement Learning for RoboCup Soccer Keepaway

RoboCup simulated soccer presents many challenges to reinforcement learning methods, including a large state space, hidden and uncertain state, multiple independent agents learning simultaneously, and long and variable delays in the effects of actions. We describe our application of episodic SMDP Sarsa(λ) with linear tile-coding function approximation and variable λ to learning higher-level decisions in a keepaway subtask of RoboCup soccer. In keepaway, one team, “the keepers,” tries to keep control of the ball for as long as possible despite the efforts of “the takers.” The keepers learn individually when to hold the ball and when to pass to a teammate. Our agents learned policies that significantly outperform a range of benchmark policies. We demonstrate the generality of our approach by applying it to a number of task variations including different field sizes and different numbers of players on each team.

Practical Vision-Based Monte Carlo Localization on a Legged Robot

Mobile robot localization, the ability of a robot to determine its global position and orientation, continues to be a major research focus in robotics. In most past cases, such localization has been studied on wheeled robots with range finding sensors such as sonar or lasers. In this paper, we consider the more challenging scenario of a legged robot localizing with a limited field-of-view camera as its primary sensory input. We begin with a baseline implementation adapted from the literature that provides a reasonable level of competence, but that exhibits some weaknesses in real-world tests. We propose a series of practical enhancements designed to improve the robot’s sensory and actuator models that enable our robots to achieve a 50% improvement in localization accuracy over the baseline implementation. We go on to demonstrate how the accuracy improvement is even more dramatic when the robot is subjected to large unmodeled movements. These enhancements are each individually straightforward, but together they provide a roadmap for avoiding potential pitfalls when implementing Monte Carlo Localization on vision-based and/or legged robots.

Graph-Based Domain Mapping for Transfer Learning in General Games

A general game player is an agent capable of taking as input a description of a games rules in a formal language and proceeding to play without any subsequent human input. To do well, an agent should learn from experience with past games and transfer the learned knowledge to new problems. We introduce a graph-based method for identifying previously encountered games and prove its robustness formally. We then describe how the same basic approach can be used to identify similar but non-identical games. We apply this technique to automate domain mapping for value function transfer and speed up reinforcement learning on variants of previously played games. Our approach is fully implemented with empirical results in the general game playing system.

Keepaway soccer: from machine learning testbed to benchmark

Keepaway soccer has been previously put forth as a testbed for machine learning. Although multiple researchers have used it successfully for machine learning experiments, doing so has required a good deal of domain expertise. This paper introduces a set of programs, tools, and resources designed to make the domain easily usable for experimentation without any prior knowledge of RoboCup or the Soccer Server. In addition, we report on new experiments in the Keepaway domain, along with performance results designed to be directly comparable with future experimental results. Combined, the new infrastructure and our concrete demonstration of its use in comparative experiments elevate the domain to a machine learning benchmark, suitable for use by researchers across the field.

The UT austin villa 2003 champion simulator coach: a machine learning approach

The UT Austin Villa 2003 simulated online soccer coach was a first time entry in the RoboCup Coach Competition. In developing the coach, the main research focus was placed on treating advice-giving as a machine learning problem. Competing against a field of mostly hand-coded coaches, the UT Austin Villa coach earned first place in the competition. In this paper, we present the multi-faceted learning strategy that our coach used and examine which aspects contributed most to the coachs success.

From pixels to multi-robot decision-making: A study in uncertainty

Mobile robots must cope with uncertainty from many sources along the path from interpreting raw sensor inputs to behavior selection to execution of the resulting primitive actions. This article identifies several such sources and introduces methods for (i) reducing uncertainty and (ii) making decisions in the face of uncertainty. We present a complete vision-based robotic system that includes several algorithms for learning models that are useful and necessary for planning, and then place particular emphasis on the planning and decision-making capabilities of the robot. Specifically, we present models for autonomous color calibration, autonomous sensor and actuator modeling, and an adaptation of particle filtering for improved localization on legged robots. These contributions enable effective planning under uncertainty for robots engaged in goaloriented behavior within a dynamic, collaborative and adversarial environment. Each of our algorithms is fully implemented and tested on a commercial off-the-shelf vision-based quadruped robot. c 2006 Elsevier B.V. All rights reserved.

Toward autonomous scientific exploration of ice-covered lakes—Field experiments with the ENDURANCE AUV in an Antarctic Dry Valley

Chemical properties of lake water can provide valuable insight into its ecology. Lakes that are permanently frozen over with ice are generally inaccessible to comprehensive exploration by humans. This paper describes the integration of several novel and existing technologies into an autonomous underwater robot, ENDURANCE, that was successfully used for gathering scientific data in West Lake Bonney in Taylor Valley, Antarctica, in December 2008. This paper focuses on three novel technological and algorithmic solutions. First, a robust position estimation system that uses an acoustic beacon to complement traditional dead-reckoning is described. Second, a novel vision-based docking algorithm for locating and ascending a vertical shaft by tracking a blinking light source is presented. Third, a novel profiling system for measuring water properties while causing minimal water disturbance is described. Finally, experimental results from the scientific missions in 2008 in West Lake Bonney are presented.