Giulio Marin

Hand gesture recognition with leap motion and kinect devices

The recent introduction of novel acquisition devices like the Leap Motion and the Kinect allows to obtain a very informative description of the hand pose that can be exploited for accurate gesture recognition. This paper proposes a novel hand gesture recognition scheme explicitly targeted to Leap Motion data. An ad-hoc feature set based on the positions and orientation of the fingertips is computed and fed into a multi-class SVM classifier in order to recognize the performed gestures. A set of features is also extracted from the depth computed from the Kinect and combined with the Leap Motion ones in order to improve the recognition performance. Experimental results present a comparison between the accuracy that can be obtained from the two devices on a subset of the American Manual Alphabet and show how, by combining the two features sets, it is possible to achieve a very high accuracy in real-time.

Hand gesture recognition with jointly calibrated Leap Motion and depth sensor

Novel 3D acquisition devices like depth cameras and the Leap Motion have recently reached the market. Depth cameras allow to obtain a complete 3D description of the framed scene while the Leap Motion sensor is a device explicitly targeted for hand gesture recognition and provides only a limited set of relevant points. This paper shows how to jointly exploit the two types of sensors for accurate gesture recognition. An ad-hoc solution for the joint calibration of the two devices is firstly presented. Then a set of novel feature descriptors is introduced both for the Leap Motion and for depth data. Various schemes based on the distances of the hand samples from the centroid, on the curvature of the hand contour and on the convex hull of the hand shape are employed and the use of Leap Motion data to aid feature extraction is also considered. The proposed feature sets are fed to two different classifiers, one based on multi-class SVMs and one exploiting Random Forests. Different feature selection algorithms have also been tested in order to reduce the complexity of the approach. Experimental results show that a very high accuracy can be obtained from the proposed method. The current implementation is also able to run in real-time.

Hand gesture recognition with depth data

Depth data acquired by current low-cost real-time depth cameras provide a very informative description of the hand pose, that can be effectively exploited for gesture recognition purposes. This paper introduces a novel hand gesture recognition scheme based on depth data. The hand is firstly extracted from the acquired depth maps with the aid also of color information from the associated views. Then the hand is segmented into palm and finger regions. Next, two different set of feature descriptors are extracted, one based on the distances of the fingertips from the hand center and the other on the curvature of the hand contour. Finally, a multi-class SVM classifier is employed to recognize the performed gestures. The proposed scheme runs in real-time and is able to achieve a very high accuracy on depth data acquired with the Kinect.

Time-of-Flight and Structured Light Depth Cameras

This book provides a comprehensive overview of the key technologies and applications related to new cameras that have brought 3D data acquisition to the mass market. It covers both the theoretical principles behind the acquisition devices and the practical implementation aspects of the computer vision algorithms needed for the various applications. Real data examples are used in order to show the performances of the various algorithms. The performance and limitations of the depth camera technology are explored, along with an extensive review of the most effective methods for addressing challenges in common applications. Applications covered in specific detail include scene segmentation, 3D scene reconstruction, human pose estimation and tracking and gesture recognition. This book offers students, practitioners and researchers the tools necessary to explore the potential uses of depth data in light of the expanding number of devices available for sale. It explores the impact of these devices on the rapidly growing field of depth-based computer vision.

Reliable Fusion of ToF and Stereo Depth Driven by Confidence Measures

In this paper we propose a framework for the fusion of depth data produced by a Time-of-Flight (ToF) camera and stereo vision system. Initially, depth data acquired by the ToF camera are upsampled by an ad-hoc algorithm based on image segmentation and bilateral filtering. In parallel a dense disparity map is obtained using the Semi-Global Matching stereo algorithm. Reliable confidence measures are extracted for both the ToF and stereo depth data. In particular, ToF confidence also accounts for the mixed-pixel effect and the stereo confidence accounts for the relationship between the pointwise matching costs and the cost obtained by the semi-global optimization. Finally, the two depth maps are synergically fused by enforcing the local consistency of depth data accounting for the confidence of the two data sources at each location. Experimental results clearly show that the proposed method produces accurate high resolution depth maps and outperforms the compared fusion algorithms.

Feature Descriptors for Depth-Based Hand Gesture Recognition

Depth data acquired by consumer depth cameras provide a very informative description of the hand pose that can be exploited for accurate gesture recognition. A typical hand gesture recognition pipeline requires to identify the hand, extract some relevant features and exploit a suitable machine learning technique to recognize the performed gesture. This chapter deals with the recognition of static poses. It starts by describing how the hand can be extracted from the scene exploiting depth and color data. Then several different features that can be extracted from the depth data are presented. Finally, a multi-class support vector machines (SVM) classifier is applied to the presented features in order to evaluate the performance of the various descriptors.

Operating Principles of Structured Light Depth Cameras

This chapter uses the camera virtualization approach to offer a unified treatment of various structured light depth cameras. Readers will learn how these cameras can differ in characteristics like number of cameras, baseline, position of the projector, and projected patterns. We also present fundamentals of illuminator design, the most critical component of structured light depth cameras, using the concept of uniqueness, and explore its implementation by wavelength, range, time, and space multiplexing. Various non-idealities of structured light depth cameras are also explained. In the last part of the chapter, theoretical ideas are applied to an analysis of the most popular structured light depth camera products in the market, like the Primesense Camera used in the Kinect v1 and other more recent products, such as the Intel RealSense F200 and R200.

Scene Segmentation Assisted by Depth Data

Segmentation, or detecting scene elements within an image, can be drastically simplified by combining depth and color data. This approach delivers segmentation tools which outperform techniques based on color alone. This chapter shows how consumer depth camera data can be used for three different tasks. The first is video matting, the separation of foreground objects from the background. The second is scene segmentation, the partitioning of color images and depth maps into different regions corresponding to scene elements. The third is semantic segmentation, the task of segmenting the framed scene and associating each segment to a specific category of object. We present various algorithms and methodologies for both single frame, color, and depth video sequences.

3D Scene Reconstruction from Depth Camera Data

Obtaining a 3D model of the world around us is a challenging task but consumer depth cameras are poised to make 3D reconstruction available to everyone. The distinct characteristics of data provided by these cameras require a rethinking of the algorithms and procedures for 3D reconstruction, since 3D modeling from consumer depth camera data requires the registration and fusion of many noisy frames. In this chapter, we discuss several approaches targeted to depth cameras. Additionally, solutions for pre-processing, pairwise, and global registration as well as fusion of views are presented, including the KinectFusion approach and its extension to dynamic scenes. The related simultaneous localization and mapping problem (RGB-D SLAM) is also briefly touched upon.

Human Pose Estimation and Tracking

Human pose estimation and tracking is one of the most intriguing yet challenging applications of consumer depth cameras. After an overview of common human hand and body models, we introduce approaches for pose recovery from a single frame, starting from the popular method based on Random Decision Forests proposed by Shotton et al. and used for the Microsoft Kinect. Various pose tracking approaches are then presented to recover the human pose configuration over time from a sequence of frames. We discuss some of the main solutions available today, including algorithms based on numerical optimization methods, filtering approaches, and recent advances based on Markov Random Fields.