Sebastian Handrich

Multi stereo camera data fusion for fingertip detection in gesture recognition systems

In this paper we present our results of fingertip detection to realize an automatic gesture recognition system by using a multi stereo camera setup. The online framework detects automatically the hands and the face of the user based on depth and color information. To estimate the spatial position and the joints of fingers a 3D hand model was generated. We used the Iterative Closest Point (ICP) algorithm to calculate the distance error between the model and 3D input data. In addition a separation and evaluation of hand and fingertip movements was implemented. To solve the general problem of self-occlusion we developed a multi stereo camera system to increase the information density. The required calibration is presented by using ICP algorithm and Genetic Algorithm.

A Robust Method for Human Pose Estimation Based on Geodesic Distance Features

In this work, we propose a real-time capable and robust method for human pose estimation based on geodesic distance features from depth images. Although a lot of work has been done in the field of the human pose estimation, it remains a challenging task - especially because of the high variability of human poses and self occlusions. The pose estimation focuses on the upper body, as it is the relevant part for a subsequent gesture and posture recognition and therefore the basis for a real human-machine-interaction. A graph-based representation of the 3D point cloud data is determined which allows for the measurement of pose-independent geodesic distances on the surface of the body. Based on these distances we determine feature points that are used for the adaptation of a kinematic skeleton model of the human upper body. The method does not need any pre-trained pose classifiers and can therefore track arbitrary poses as long as the user is not turned away from the camera.

Multi hypotheses based object tracking in HCI environments

Gesture recogntion plays an important role in Human Computer Interaction (HCI). However, in most HCI systems the user is limited to use only one hand or two hands under optimal conditions. Challenges are for instance non-homogeneous backgrounds, hand-hand or hand-face overlapping or brightness modifications which will be met in real HCI scenarios. In this work, we propose a novel method that solves the ambiguities due to the hand overlap robustly based on multi-hypotheses object association. The results of tracking build the basis for further feature extraction and gesture recognition.

Prerequisites for integrating unsupervised and reinforcement learning in a single network of spiking neurons

Most artificial neural network architectures learn either via unsupervised or reinforcement learning but rarely via both. However, the brain effectively integrates both types of learning. We describe which prerequisites are necessary in a spiking network architecture in order to integrate both learning mechanisms and present a network which meets these requirements. In a nut shell, the network has a winner-take-all type output layer resembling the motor output and an excitatory feedback layer which extends the firing of the input layer until after the end of external stimulation resembling the function of the hippocampus.

Improving of gesture recognition using multi-hypotheses object association

Gesture recognition plays an important role in Human Computer Interaction (HCI) but in most HCI systems, the user is limited to use only one hand or two hands under optimal conditions. Challenges are for instance non-homogeneous backgrounds, hand-hand or hand-face overlapping and brightness modifications. In this research, we have proposed a novel approach that solves the ambiguities occurred due to the hand overlapping robustly based on multi-hypotheses object association. This multi-hypotheses object association builds the basis for the tracking in which the hand trajectories are computed and this leads us to extract the features. The gesture recognition phase takes the extracted features and classifies them through Hidden Markov Model (HMM).

Multi-objective parameter estimation of biologically plausible neural networks in different behavior stages

An essential behaviour of biological neural networks is the switching between different dynamical stages i.e. during development, learning, attention or memory formation. This seems to be a key element in understanding the balance of stability and flexibility of biological information systems and should also be implemented in biologic plausible artificial neural networks. The parameter estimation of such artificial networks to fit it to the biological behavior in the different stages is a multi-objective problem. We show a multi-population genetic algorithm to get useful parameter combinations with an adapted cross population estimation of fitness and recombination of genes. The algorithm is tested on parameter fitting of a working memory model and further application of dopamine modulated learning is discussed.

Combining Supervised, Unsupervised, and Reinforcement Learning in a Network of Spiking Neurons

The human brain constantly learns via mutiple different learning strategies. It can learn by simply having stimuli being presented to its sensory organs which is considered unsupervised learning. In addition, it can learn associations between inputs and outputs when a teacher provides the output which is considered as supervised learning. Most importantly, it can learn very efficiently if correct behaviour is followed by reward and/or incorrect behaviour is followed by punishment which is considered reinforcement learning. So far, most artificial neural architectures implement only one of the three learning mechanisms — even though the brain integrates all three. Here, we have implemented unsupervised, supervised, and reinforcement learning within a network of spiking neurons. In order to achieve this ambitious goal, the existing learning rule called spike-timing-dependent plasticity had to be extended such that it is modulated by the reward signal dopamine.

A biologically plausible winner-takes-all architecture

Winner-takes-all (WTA) is an important mechanism in artificial and biological neural networks. We present a biologically plausible two layer WTA architecture with biologically plausible spiking neuron model and conductance based synapses. The excitatory neurons in the WTA layer receive spiking signals from an input layer and can inhibit other excitatory WTA neurons via related inhibitory neurons. The connections from the input layer to WTA layer can be trained by Spike-Time-Dependent Plasticity to discriminate between different classes of input patters. The overall input of the WTA neurons are controlled by synaptic scaling.

Low cost calibration of stereo line scan camera systems

We present a new calibration method for 3d measurement systems consisting of two co-planar aligned line scan cameras. As appliance we use a calibration target whose geometric 2d and 3d parameters are only approximately known. As our input data we take from that target a sufficient number of single captured image lies in different geometric positions. Then we calculate 2d points within the joint viewing plane of the cameras both by triangulation of corresponding image points and by intersection with the lines on the target surface. The distances between the triangulation points and the related intersection points are minimized by linearizing and least squares adjustment. In the result we obtain all relevant parameters of the inner and outer orientation as well as the 2d and 3d geometry parameters of the calibration target with high accuracy.

Human bodypart classification using geodesic descriptors and random forests

A new approach to classify human body parts in depth images is proposed. The approach is based on geodesic descriptors. Such a descriptor randomly samples the local geodesic neighborhood of each depth pixel. During a training phase, a random forest classifier learns the correct body part from these descriptors. The experimental evaluation shows that we can robustly classify 19 body parts for several different poses and body proportions. We further compare our approach and the classification based on geodesic distance features to those that were used in previous works.