Dmitry Berenson

HERB: a home exploring robotic butler

We describe the architecture, algorithms, and experiments with HERB, an autonomous mobile manipulator that performs useful manipulation tasks in the home. We present new algorithms for searching for objects, learning to navigate in cluttered dynamic indoor scenes, recognizing and registering objects accurately in high clutter using vision, manipulating doors and other constrained objects using caging grasps, grasp planning and execution in clutter, and manipulation on pose and torque constraint manifolds. We also present numerous severe real-world test results from the integration of these algorithms into a single mobile manipulator.

Object recognition and full pose registration from a single image for robotic manipulation

Robust perception is a vital capability for robotic manipulation in unstructured scenes. In this context, full pose estimation of relevant objects in a scene is a critical step towards the introduction of robots into household environments. In this paper, we present an approach for building metric 3D models of objects using local descriptors from several images. Each model is optimized to fit a set of calibrated training images, thus obtaining the best possible alignment between the 3D model and the real object. Given a new test image, we match the local descriptors to our stored models online, using a novel combination of the RANSAC and Mean Shift algorithms to register multiple instances of each object. A robust initialization step allows for arbitrary rotation, translation and scaling of objects in the test images. The resulting system provides markerless 6-DOF pose estimation for complex objects in cluttered scenes. We provide experimental results demonstrating orientation and translation accuracy, as well a physical implementation of the pose output being used by an autonomous robot to perform grasping in highly cluttered scenes.

Task Space Regions: A framework for pose-constrained manipulation planning

We present a manipulation planning framework that allows robots to plan in the presence of constraints on end-effector pose, as well as other common constraints. The framework has three main components: constraint representation, constraint-satisfaction strategies, and a general planning algorithm. These components come together to create an efficient and probabilistically complete manipulation planning algorithm called the Constrained BiDirectional Rapidly-exploring Random Tree (RRT) – CBiRRT2. The underpinning of our framework for pose constraints is our Task Space Regions (TSRs) representation. TSRs are intuitive to specify, can be efficiently sampled, and the distance to a TSR can be evaluated very quickly, making them ideal for sampling-based planning. Most importantly, TSRs are a general representation of pose constraints that can fully describe many practical tasks. For more complex tasks, such as manipulating articulated objects, TSRs can be chained together to create more complex end-effector pose constraints. TSRs can also be intersected, a property that we use to plan with pose uncertainty. We provide a detailed description of our framework, prove probabilistic completeness for our planning approach, and describe several real-world example problems that illustrate the efficiency and versatility of the TSR framework.

Manipulation planning on constraint manifolds

We present the Constrained Bi-directional Rapidly-Exploring Random Tree (CBiRRT) algorithm for planning paths in configuration spaces with multiple constraints. This algorithm provides a general framework for handling a variety of constraints in manipulation planning including torque limits, constraints on the pose of an object held by a robot, and constraints for following workspace surfaces. CBiRRT extends the Bi-directional RRT (BiRRT) algorithm by using projection techniques to explore the configuration space manifolds that correspond to constraints and to find bridges between them. Consequently, CBiRRT can solve many problems that the BiRRT cannot, and only requires one additional parameter: the allowable error for meeting a constraint. We demonstrate the CBiRRT on a 7DOF WAM arm with a 4DOF Barrett hand on a mobile base. The planner allows this robot to perform household tasks, solve puzzles, and lift heavy objects.

Grasp planning in complex scenes

This paper combines grasp analysis and manipulation planning techniques to perform fast grasp planning in complex scenes. In much previous work on grasping, the object being grasped is assumed to be the only object in the environment. Hence the grasp quality metrics and grasping strategies developed do not perform well when the object is close to obstacles and many good grasps are infeasible. We introduce a framework for finding valid grasps in cluttered environments that combines a grasp quality metric for the object with information about the local environment around the object and information about the robots kinematics. We encode these factors in a grasp-scoring function which we use to rank a precomputed set of grasps in terms of their appropriateness for a given scene. We show that this ranking is essential for efficient grasp selection and present experiments in simulation and on the HRP2 robot.

Manipulation planning with Workspace Goal Regions

We present an approach to path planning for manipulators that uses Workspace Goal Regions (WGRs) to specify goal end-effector poses. Instead of specifying a discrete set of goals in the manipulators configuration space, we specify goals more intuitively as volumes in the manipulators workspace. We show that WGRs provide a common framework for describing goal regions that are useful for grasping and manipulation. We also describe two randomized planning algorithms capable of planning with WGRs. The first is an extension of RRT-JT that interleaves exploration using a Rapidly-exploring Random Tree (RRT) with exploitation using Jacobian-based gradient descent toward WGR samples. The second is the IKBiRRT algorithm, which uses a forward-searching tree rooted at the start and a backward-searching tree that is seeded by WGR samples. We demonstrate both simulation and experimental results for a 7DOF WAM arm with a mobile base performing reaching and pick-and-place tasks. Our results show that planning with WGRs provides an intuitive and powerful method of specifying goals for a variety of tasks without sacrificing efficiency or desirable completeness properties.

Humanoid motion planning for dual-arm manipulation and re-grasping tasks

In this paper, we present efficient solutions for planning motions of dual-arm manipulation and re-grasping tasks. Motion planning for such tasks on humanoid robots with a high number of degrees of freedom (DoF) requires computationally efficient approaches to determine the robots full joint configuration at a given grasping position, i.e. solving the Inverse Kinematics (IK) problem for one or both hands of the robot. In this context, we investigate solving the inverse kinematics problem and motion planning for dual-arm manipulation and re-grasping tasks by combining a gradient-descent approach in the robots pre-computed reachability space with random sampling of free parameters. This strategy provides feasible IK solutions at a low computation cost without resorting to iterative methods which could be trapped by joint-limits. We apply this strategy to dual-arm motion planning tasks in which the robot is holding an object with one hand in order to generate whole-body robot configurations suitable for grasping the object with both hands. In addition, we present two probabilistically complete RRT-based motion planning algorithms (J+-RRT and IK-RRT) that interleave the search for an IK solution with the search for a collision-free trajectory and the extension of these planners to solving re-grasping problems. The capabilities of combining IK methods and planners are shown both in simulation and on the humanoid robot ARMAR-III performing dual-arm tasks in a kitchen environment.

Human-robot collaborative manipulation planning using early prediction of human motion

In this paper we present a framework that allows a human and a robot to perform simultaneous manipulation tasks safely in close proximity. The proposed framework is based on early prediction of the humans motion. The prediction system, which builds on previous work in the area of gesture recognition, generates a prediction of human workspace occupancy by computing the swept volume of learned human motion trajectories. The motion planner then plans robot trajectories that minimize a penetration cost in the human workspace occupancy while interleaving planning and execution. Multiple plans are computed in parallel, one for each robot task available at the current time, and the trajectory with the least cost is selected for execution. We test our framework in simulation using recorded human motions and a simulated PR2 robot. Our results show that our framework enables the robot to avoid the human while still accomplishing the robots task, even in cases where the initial prediction of the humans motion is incorrect. We also show that taking into account the predicted human workspace occupancy in the robots motion planner leads to safer and more efficient interactions between the user and the robot than only considering the humans current configuration.

A robot path planning framework that learns from experience

We propose a framework, called Lightning, for planning paths in high-dimensional spaces that is able to learn from experience, with the aim of reducing computation time. This framework is intended for manipulation tasks that arise in applications ranging from domestic assistance to robot-assisted surgery. Our framework consists of two main modules, which run in parallel: a planning-from-scratch module, and a module that retrieves and repairs paths stored in a path library. After a path is generated for a new query, a library manager decides whether to store the path based on computation time and the generated paths similarity to the retrieved path. To retrieve an appropriate path from the library we use two heuristics that exploit two key aspects of the problem: (i) A correlation between the amount a path violates constraints and the amount of time needed to repair that path, and (ii) the implicit division of constraints into those that vary across environments in which the robot operates and those that do not. We evaluated an implementation of the framework on several tasks for the PR2 mobile manipulator and a minimally-invasive surgery robot in simulation. We found that the retrieve-and-repair module produced paths faster than planning-from-scratch in over 90% of test cases for the PR2 and in 58% of test cases for the minimally-invasive surgery robot.

Herb 2.0: Lessons Learned From Developing a Mobile Manipulator for the Home

We present the hardware design, software architecture, and core algorithms of Herb 2.0, a bimanual mobile manipulator developed at the Personal Robotics Lab at Carnegie Mellon University, Pittsburgh, PA. We have developed Herb 2.0 to perform useful tasks for and with people in human environments. We exploit two key paradigms in human environments: that they have structure that a robot can learn, adapt and exploit, and that they demand general-purpose capability in robotic systems. In this paper, we reveal some of the structure present in everyday environments that we have been able to harness for manipulation and interaction, comment on the particular challenges on working in human spaces, and describe some of the lessons we learned from extensively testing our integrated platform in kitchen and office environments.