Sonia Chernova

A survey of robot learning from demonstration

We present a comprehensive survey of robot Learning from Demonstration (LfD), a technique that develops policies from example state to action mappings. We introduce the LfD design choices in terms of demonstrator, problem space, policy derivation and performance, and contribute the foundations for a structure in which to categorize LfD research. Specifically, we analyze and categorize the multiple ways in which examples are gathered, ranging from teleoperation to imitation, as well as the various techniques for policy derivation, including matching functions, dynamics models and plans. To conclude we discuss LfD limitations and related promising areas for future research.

An evolutionary approach to gait learning for four-legged robots

Developing fast gaits for legged robots is a difficult task that requires optimizing parameters in a highly irregular, multidimensional space. In the past, walk optimization for quadruped robots, namely the Sony AIBO robot, was done by handtuning the parameterized gaits. In addition to requiring a lot of time and human expertise, this process produced sub-optimal results. Several recent projects have focused on using machine learning to automate the parameter search. Algorithms utilizing Powells minimization method and policy gradient reinforcement learning have shown significant improvement over previous walk optimization results. In this paper we present a new algorithm for walk optimization based on an evolutionary approach. Unlike previous methods, our algorithm does not attempt to approximate the gradient of the multidimensional space. This makes it more robust to noise in parameter evaluations and avoids prematurely converging to local optima, a problem encountered by both of the previously suggested algorithms. Our evolutionary algorithm matches the best previous learning method, achieving several different walks of high quality. Furthermore, the best learned walks represent an impressive 20% improvement over our own best hand-tuned walks.

Interactive policy learning through confidence-based autonomy

We present Confidence-Based Autonomy (CBA), an interactive algorithm for policy learning from demonstration. The CBA algorithm consists of two components which take advantage of the complimentary abilities of humans and computer agents. The first component, Confident Execution, enables the agent to identify states in which demonstration is required, to request a demonstration from the human teacher and to learn a policy based on the acquired data. The algorithm selects demonstrations based on a measure of action selection confidence, and our results show that using Confident Execution the agent requires fewer demonstrations to learn the policy than when demonstrations are selected by a human teacher. The second algorithmic component, Corrective Demonstration, enables the teacher to correct any mistakes made by the agent through additional demonstrations in order to improve the policy and future task performance. CBA and its individual components are compared and evaluated in a complex simulated driving domain. The complete CBA algorithm results in the best overall learning performance, successfully reproducing the behavior of the teacher while balancing the tradeoff between number of demonstrations and number of incorrect actions during learning.

Confidence-based policy learning from demonstration using Gaussian mixture models

We contribute an approach for interactive policy learning through expert demonstration that allows an agent to actively request and effectively represent demonstration examples. In order to address the inherent uncertainty of human demonstration, we represent the policy as a set of Gaussian mixture models (GMMs), where each model, with multiple Gaussian components, corresponds to a single action. Incrementally received demonstration examples are used as training data for the GMM set. We then introduce our confident execution approach, which focuses learning on relevant parts of the domain by enabling the agent to identify the need for and request demonstrations for specific parts of the state space. The agent selects between demonstration and autonomous execution based on statistical analysis of the uncertainty of the learned Gaussian mixture set. As it achieves proficiency at its task and gains confidence in its actions, the agent operates with increasing autonomy, eliminating the need for unnecessary demonstrations of already acquired behavior, and reducing both the training time and the demonstration workload of the expert. We validate our approach with experiments in simulated and real robot domains.

Multi-thresholded approach to demonstration selection for interactive robot learning

Effective learning from demonstration techniques enable complex robot behaviors to be taught from a small number of demonstrations. A number of recent works have explored interactive approaches to demonstration, in which both the robot and the teacher are able to select training examples. In this paper, we focus on a demonstration selection algorithm used by the robot to identify informative states for demonstration. Existing automated approaches for demonstration selection typically rely on a single threshold value, which is applied to a measure of action confidence. We highlight the limitations of using a single fixed threshold for a specific subset of algorithms, and contribute a method for automatically setting multiple confidence thresholds designed to target domain states with the greatest uncertainty. We present a comparison of our multi-threshold selection method to confidence-based selection using a single fixed threshold, and to manual data selection by a human teacher. Our results indicate that the automated multi-threshold approach significantly reduces the number of demonstrations required to learn the task.

Mobile human-robot teaming with environmental tolerance

We demonstrate that structured light-based depth sensing with standard perception algorithms can enable mobile peer-to-peer interaction between humans and robots. We posit that the use of recent emerging devices for depth-based imaging can enable robot perception of non-verbal cues in human movement in the face of lighting and minor terrain variations. Toward this end, we have developed an integrated robotic system capable of person following and responding to verbal and non-verbal commands under varying lighting conditions and uneven terrain. The feasibility of our system for peer-to-peer HRI is demonstrated through two trials in indoor and outdoor environments.

Learning equivalent action choices from demonstration

In their interactions with the world robots inevitably face equivalent action choices, situations in which multiple actions are equivalently applicable. In this paper, we address the problem of equivalent action choices in learning from demonstration, a robot learning approach in which a policy is acquired from human demonstrations of the desired behavior. We note that when faced with a choice of equivalent actions, a human teacher often demonstrates an action arbitrarily and does not make the choice consistently over time. The resulting inconsistently labeled training data poses a problem for classification-based demonstration learning algorithms by violating the common assumption that for any world state there exists a single best action. This problem has been overlooked by previous approaches for demonstration learning. In this paper, we present an algorithm that identifies regions of the state space with conflicting demonstrations and enables the choice between multiple actions to be represented explicitly within the robotpsilas policy. An experimental evaluation of the algorithm in a real-world obstacle avoidance domain shows that reasoning about action choices significantly improves the robotpsilas learning performance.

From Deliberative to Routine Behaviors: A Cognitively Inspired Action-Selection Mechanism for Routine Behavior Capture

Long-term human—robot interaction, especially in the case of humanoid robots, requires an adaptable and varied behavior base. In this paper, we present a method for capturing, or learning, sequential tasks by transferring serial behavior execution from deliberative to routine control. The incorporation of this approach leads to the natural development of complex and varied behaviors, with lower demands for planning, coordination and resources. We demonstrate how this process can be performed autonomously as part of the normal function of the robot, without the need for an explicit learning stage or user guidance. The complete implementation of this algorithm on the Sony QRIO humanoid robot is described.

Confidence-Based Multi-Robot Learning from Demonstration

Learning from demonstration algorithms enable a robot to learn a new policy based on demonstrations provided by a teacher. In this article, we explore a novel research direction, multi-robot learning from demonstration, which extends demonstration based learning methods to collaborative multi-robot domains. Specifically, we study the problem of enabling a single person to teach individual policies to multiple robots at the same time. We present flexMLfD, a task and platform independent multi-robot demonstration learning framework that supports both independent and collaborative multi-robot behaviors. Building upon this framework, we contribute three approaches to teaching collaborative multi-robot behaviors based on different information sharing strategies, and evaluate these approaches by teaching two Sony QRIO humanoid robots to perform three collaborative ball sorting tasks. We then present scalability analysis of flexMLfD using up to seven Sony AIBO robots. We conclude the article by proposing a formalization for a broader multi-robot learning from demonstration research area.

Teaching collaborative multi-robot tasks through demonstration

Humanoid robots working alongside humans in everyday environments is a long standing goal of the robotics community. To achieve this goal, methods for developing new robot behaviors that are intuitive and accessible to non-programmers are required. In this paper, we present a demonstration-based method for teaching distributed autonomous robots to coordinate their actions and perform collaborative multi-robot tasks. Within the presented framework, each robot learns an individual policy from teacher demonstrations using a confidence-based algorithm. Based on this learning approach, we contribute three techniques for teaching multi-robot coordination using different information sharing strategies. We evaluate and compare these approaches by teaching two Sony QRIO humanoid robots to perform three collaborative ball sorting tasks.