Scott Niekum

Learning and generalization of complex tasks from unstructured demonstrations

We present a novel method for segmenting demonstrations, recognizing repeated skills, and generalizing complex tasks from unstructured demonstrations. This method combines many of the advantages of recent automatic segmentation methods for learning from demonstration into a single principled, integrated framework. Specifically, we use the Beta Process Autoregressive Hidden Markov Model and Dynamic Movement Primitives to learn and generalize a multi-step task on the PR2 mobile manipulator and to demonstrate the potential of our framework to learn a large library of skills over time.

Incremental Semantically Grounded Learning from Demonstration

Much recent work in robot learning from demonstration has focused on automatically segmenting continuous task demonstrations into simpler, reusable primitives. However, strong assumptions are often made about how these primitives can be sequenced, limiting the potential for data reuse. We introduce a novel method for discovering semantically grounded primitives and incrementally building and improving a finite-state representation of a task in which various contingencies can arise. Specifically, a Beta Process Autoregressive Hidden Markov Model is used to automatically segment demonstrations into motion categories, which are then further subdivided into semantically grounded states in a finite-state automaton. During replay of the task, a data-driven approach is used to collect additional data where they are most needed through interactive corrections, which are then used to improve the finite-state automaton. Together, this allows for intelligent sequencing of primitives to create novel, adaptive behavior that can be incrementally improved as needed. We demonstrate the utility of this technique on a furniture assembly task using the PR2 mobile manipulator.

Learning grounded finite-state representations from unstructured demonstrations

Robots exhibit flexible behavior largely in proportion to their degree of knowledge about the world. Such knowledge is often meticulously hand-coded for a narrow class of tasks, limiting the scope of possible robot competencies. Thus, the primary limiting factor of robot capabilities is often not the physical attributes of the robot, but the limited time and skill of expert programmers. One way to deal with the vast number of situations and environments that robots face outside the laboratory is to provide users with simple methods for programming robots that do not require the skill of an expert. For this reason, learning from demonstration (LfD) has become a popular alternative to traditional robot programming methods, aiming to provide a natural mechanism for quickly teaching robots. By simply showing a robot how to perform a task, users can easily demonstrate new tasks as needed, without any special knowledge about the robot. Unfortunately, LfD often yields little knowledge about the world, and thus lacks robust generalization capabilities, especially for complex, multi-step tasks. We present a series of algorithms that draw from recent advances in Bayesian non-parametric statistics and control theory to automatically detect and leverage repeated structure at multiple levels of abstraction in demonstration data. The discovery of repeated structure provides critical insights into task invariants, features of importance, high-level task structure, and appropriate skills for the task. This culminates in the discovery of a finite-state representation of the task, composed of grounded skills that are flexible and reusable, providing robust generalization and transfer in complex, multi-step robotic tasks. These algorithms are tested and evaluated using a PR2 mobile manipulator, showing success on several complex real-world tasks, such as furniture assembly.

Genetic Programming for Reward Function Search

Reward functions in reinforcement learning have largely been assumed given as part of the problem being solved by the agent. However, the psychological notion of intrinsic motivation has recently inspired inquiry into whether there exist alternate reward functions that enable an agent to learn a task more easily than the natural task-based reward function allows. This paper presents a genetic programming algorithm to search for alternate reward functions that improve agent learning performance. We present experiments that show the superiority of these reward functions, demonstrate the possible scalability of our method, and define three classes of problems where reward function search might be particularly useful: distributions of environments, nonstationary environments, and problems with short agent lifetimes.

Automatic detection and classification of features of geologic interest

The volume of data that planetary rovers and their instrument payloads can produce will continue to outpace available deep space communication bandwidth. Future exploration rovers will require science autonomy systems that interpret collected data in order to selectively compress observations, summarize results, and respond to new discoveries. We present a method that uses a probabilistic fusion of data from multiple sensor sources for onboard segmentation, detection and classification of geological properties. Field experiments performed in the Atacama desert in Chile show the systems performance versus ground truth on the specific problem of automatic rock identification.

Concepts for science autonomy during robotic traverse and survey

Future Mars rovers will have the ability to autonomously navigate for distances of kilometers. In one sol a traverse may take a rover into unexplored areas beyond its local horizon. Naturally, scientists cannot specify particular targets for the rover in an area they have not yet seen. This paper analyzes what they can specify: priorities that provide the rover with enough information to autonomously select science targets using its onboard sensing. Several autonomous science operational modes and priority types are discussed. We also introduce a science priority language. A team of scientists was asked to use the language in specifying targets for a teleoperated rover, and qualitative results are reported.

Active articulation model estimation through interactive perception

We introduce a particle filter-based approach to representing and actively reducing uncertainty over articulated motion models. The presented method provides a probabilistic model that integrates visual observations with feedback from manipulation actions to best characterize a distribution of possible articulation models. We evaluate several action selection methods to efficiently reduce the uncertainty about the articulation model. The full system is experimentally evaluated using a PR2 mobile manipulator. Our experiments demonstrate that the proposed system allows for intelligent reasoning about sparse, noisy data in a number of common manipulation scenarios.

Evolution of reward functions for reinforcement learning

The reward functions that drive reinforcement learning systems are generally derived directly from the descriptions of the problems that the systems are being used to solve. In some problem domains, however, alternative reward functions may allow systems to learn more quickly or more effectively. Here we describe work on the use of genetic programming to find novel reward functions that improve learning system performance. We briefly present the core concepts of our approach, our motivations in developing it, and reasons to believe that the approach has promise for the production of highly successful adaptive technologies. Experimental results are presented and analyzed in our full report [3].

Online Bayesian changepoint detection for articulated motion models

We introduce CHAMP, an algorithm for online Bayesian changepoint detection in settings where it is difficult or undesirable to integrate over the parameters of candidate models. CHAMP is used in combination with several articulation models to detect changes in articulated motion of objects in the world, allowing a robot to infer physically-grounded task information. We focus on three settings where a changepoint model is appropriate: objects with intrinsic articulation relationships that can change over time, object-object contact that results in quasi-static articulated motion, and assembly tasks where each step changes articulation relationships. We experimentally demonstrate that this system can be used to infer various types of information from demonstration data including causal manipulation models, human-robot grasp correspondences, and skill verification tests.

Asking for Help Effectively via Modeling of Human Beliefs

Autonomous robots deployed around humans must be able to ask for help when problems arise. However, people may have incorrect mental models of the robots» capabilities or task, making them unable to help. We propose a data-driven method to estimate humans» beliefs after hearing task-related utterances and build sets of utterances that influence people towards useful help in expectation. We present an example to show our method selects effective utterances when the desired help is much different than a person expects.