Youding Zhu

Constrained optimization for human pose estimation from depth sequences

A new 2-step method is presented for human upper-body pose estimation from depth sequences, in which coarse human part labeling takes place first, followed by more precise joint position estimation as the second phase. In the first step, a number of constraints are extracted from notable image features such as the head and torso. The problem of pose estimation is cast as that of label assignment with these constraints. Major parts of the human upper body are labeled by this process. The second step estimates joint positions optimally based on kinematic constraints using dense correspondences between depth profile and human model parts. The proposed framework is shown to overcome some issues of existing approaches for human pose tracking using similar types of data streams. Performance comparison with motion capture data is presented to demonstrate the accuracy of our approach.

ONLINE TRANSFER OF HUMAN MOTION TO HUMANOIDS

Transferring motion from a human demonstrator to a humanoid robot is an important step toward developing robots that are easily programmable and that can replicate or learn from observed human motion. The so called motion retargeting problem has been well studied and several off-line solutions exist based on optimization approaches that rely on pre-recorded human motion data collected from a marker-based motion capture system. From the perspective of human robot interaction, there is a growing interest in online motion transfer, particularly without using markers. Such requirements have placed stringent demands on retargeting algorithms and limited the potential use of off-line and pre-recorded methods. To address these limitations, we present an online task space control theoretic retargeting formulation to generate robot joint motions that adhere to the robots joint limit constraints, joint velocity constraints and self-collision constraints. The inputs to the proposed method include low dimensional normalized human motion descriptors, detected and tracked using a vision based key-point detection and tracking algorithm. The proposed vision algorithm does not rely on markers placed on anatomical landmarks, nor does it require special instrumentation or calibration. The current implementation requires a depth image sequence, which is collected from a single time of flight imaging device. The feasibility of the proposed approach is shown by means of online experimental results on the Honda humanoid robot — ASIMO.

Head pose estimation for driver monitoring

Head pose estimation is important for driver attention monitoring as well as for various human computer interaction tasks. In this paper, an adaptive head pose estimation method is proposed to overcome difficulties of existing approaches. The proposed method is based on the analysis of two approaches for head pose estimation from an image sequence, that is, principal component analysis (PCA) and 3D motion estimation. The algorithm performs accurate pose estimation by learning the subject appearance on-line. Depth information is used effectively in the algorithm to segment the head region even in a cluttered scene and to perform 3D head motion estimation based on optical flow constraints.

Kinematic self retargeting: A framework for human pose estimation

This paper presents a model-based, Cartesian control theoretic approach for estimating human pose from a set of key features points (key-points) detected using depth images obtained from a time-of-flight imaging device. The key-points represent positions of anatomical landmarks, detected and tracked over time based on a probabilistic inferencing algorithm that is robust to partial occlusions and capable of resolving ambiguities in detection. The detected key-points are subsequently kinematically self retargeted, or mapped to the subjects own kinematic model, in order to predict the pose of an articulated human model at the current state, resolve ambiguities in key-point detection, and provide estimates of missing or intermittently occluded key-points. Based on a standard kinematic and mesh model of a human, constraints such as joint limit avoidance, and self-penetration avoidance are enforced within the retargeting framework. Effectiveness of the algorithm is demonstrated experimentally for upper and full-body pose reconstruction from a small set of detected key-points. On average, the proposed algorithm runs at approximately 10 frames per second for the upper-body and 5 frames per second for whole body reconstruction on a standard 2.13GHz laptop PC.

A Bayesian Framework for Human Body Pose Tracking from Depth Image Sequences

This paper addresses the problem of accurate and robust tracking of 3D human body pose from depth image sequences. Recovering the large number of degrees of freedom in human body movements from a depth image sequence is challenging due to the need to resolve the depth ambiguity caused by self-occlusions and the difficulty to recover from tracking failure. Human body poses could be estimated through model fitting using dense correspondences between depth data and an articulated human model (local optimization method). Although it usually achieves a high accuracy due to dense correspondences, it may fail to recover from tracking failure. Alternately, human pose may be reconstructed by detecting and tracking human body anatomical landmarks (key-points) based on low-level depth image analysis. While this method (key-point based method) is robust and recovers from tracking failure, its pose estimation accuracy depends solely on image-based localization accuracy of key-points. To address these limitations, we present a flexible Bayesian framework for integrating pose estimation results obtained by methods based on key-points and local optimization. Experimental results are shown and performance comparison is presented to demonstrate the effectiveness of the proposed approach.

Online and markerless motion retargeting with kinematic constraints

Transferring motion from a human demonstrator to a humanoid robot is an important step toward developing robots that are easily programmable and that can replicate or learn from observed human motion. The so called motion retargeting problem has been well studied and several off-line solutions exist based on optimization approaches that rely on pre-recorded human motion data collected from a marker-based motion capture system. From the perspective of human robot interaction, there is a growing interest in online and marker-less motion transfer. Such requirements have placed stringent demands on retargeting algorithms and limited the potential use of off-line and pre-recorded methods. To address these limitations, we present an online task space control theoretic retargeting formulation to generate robot joint motions that adhere to the robotpsilas joint limit constraints, self-collision constraints, and balance constraints. The inputs to the proposed method include low dimensional normalized human motion descriptors, detected and tracked using a vision based feature detection and tracking algorithm. The proposed vision algorithm does not rely on markers placed on anatomical landmarks, nor does it require special instrumentation or calibration. The current implementation requires a depth image sequence, which is collected from a single time of flight imaging device. We present online experimental results of the entire pipeline on the Honda humanoid robot - ASIMO.

3D head pose estimation with optical flow and depth constraints

A method is presented to estimate 3D head poses effectively by hybrid sensing of depth and gray information. Depth information is used to generate clean head segmentation even in a cluttered scene. Based on the segmentation result, sparse optical flow at head region is extracted and used for 3D head motion estimation in video rate. This method is shown to be effective through experiments on video sequences. Our method provides an alternative way for 3D head pose estimation from an image sequence in the current computer vision literature. Moreover, the depth information is incorporated into the estimation step as regularization for noisy motion estimation problem.

Constrained closed loop inverse kinematics

This paper introduces a kinematically constrained closed loop inverse kinematics algorithm for motion control of robots or other articulated rigid body systems. The proposed strategy utilizes gradients of collision and joint limit potential functions to arrive at an appropriate weighting matrix to penalize and dampen motion approaching constraint surfaces. The method is particularly suitable for self collision avoidance of highly articulated systems which may have multiple collision points among several segment pairs. In that respect, the proposed method has a distinct advantage over existing gradient projection based methods which rely on numerically unstable null-space projections when there are multiple intermittent constraints. We also show how this approach can be augmented with a previously reported method based on redirection of constraints along virtual surface manifolds. The hybrid strategy is effective, robust, and does not require parameter tuning. The efficacy of the proposed algorithm is demonstrated for a self collision avoidance problem where the reference motion is obtained from human observations. We show simulation and experimental results on the humanoid robot ASIMO.

Estimating pose from depth image streams

Capturing pose from observation can be an intuitive interface for humanoid robots. In this paper, a method is presented for estimating human pose from a sequence of images taken by a single camera. The method is based on a machine learning technique and it partitions human body into a number of clusters. Body parts are tracked over the image sequence while satisfying body constraints. An active sensing hardware is used in both methods to capture a stream of depth images at video rates, which are consequently analyzed for pose extraction. Experimental results are shown to validate our approach and characteristics of our approach are discussed