Tobias Jaeggli

Distinct Mechanisms for Coding of Visual Actions in Macaque Temporal Cortex

Temporal cortical neurons are known to respond to visual dynamic-action displays. Many human psychophysical and functional imaging studies examining biological motion perception have used treadmill walking, in contrast to previous macaque single-cell studies. We assessed the coding of locomotion in rhesus monkey (Macaca mulatta) temporal cortex using movies of stationary walkers, varying both form and motion (i.e., different facing directions) or varying only the frame sequence (i.e., forward vs backward walking). The majority of superior temporal sulcus and inferior temporal neurons were selective for facing direction, whereas a minority distinguished forward from backward walking. Support vector machines using the temporal cortical population responses as input classified facing direction well, but forward and backward walking less so. Classification performance for the latter improved markedly when the within-action response modulation was considered, reflecting differences in momentary body poses within the locomotion sequences. Responses to static pose presentations predicted the responses during the course of the action. Analyses of the responses to walking sequences wherein the start frame was varied across trials showed that some neurons also carried a snapshot sequence signal. Such sequence information was present in neurons that responded to static snapshot presentations and in neurons that required motion. Our data suggest that actions are analyzed by temporal cortical neurons using distinct mechanisms. Most neurons predominantly signal momentary pose. In addition, temporal cortical neurons, including those responding to static pose, are sensitive to pose sequence, which can contribute to the signaling of learned action sequences.

Learning Generative Models for Multi-Activity Body Pose Estimation

We present a method to simultaneously estimate 3D body pose and action categories from monocular video sequences. Our approach learns a generative model of the relationship of body pose and image appearance using a sparse kernel regressor. Body poses are modelled on a low-dimensional manifold obtained by Locally Linear Embedding dimensionality reduction. In addition, we learn a prior model of likely body poses and a dynamical model in this pose manifold. Sparse kernel regressors capture the nonlinearities of this mapping efficiently. Within a Recursive Bayesian Sampling framework, the potentially multimodal posterior probability distributions can then be inferred. An activity-switching mechanism based on learned transfer functions allows for inference of the performed activity class, along with the estimation of body pose and 2D image location of the subject. Using a rough foreground segmentation, we compare Binary PCA and distance transforms to encode the appearance. As a postprocessing step, the globally optimal trajectory through the entire sequence is estimated, yielding a single pose estimate per frame that is consistent throughout the sequence. We evaluate the algorithm on challenging sequences with subjects that are alternating between running and walking movements. Our experiments show how the dynamical model helps to track through poorly segmented low-resolution image sequences where tracking otherwise fails, while at the same time reliably classifying the activity type.

Discrimination of locomotion direction in impoverished displays of walkers by macaque monkeys

A vast literature exists on human biological motion perception in impoverished displays, e.g., point-light walkers. Less is known about the perception of impoverished biological motion displays in macaques. We trained 3 macaques in the discrimination of facing direction (left versus right) and forward versus backward walking using motion-capture-based locomotion displays (treadmill walking) in which the body features were represented by cylinder-like primitives. The displays did not contain translatory motion. Discriminating forward versus backward locomotion requires motion information while the facing-direction/view task can be solved using motion and/or form. All monkeys required lengthy training to learn the forward-backward task, while the view task was learned more quickly. Once acquired, the discriminations were specific to walking and stimulus format but generalized across actors. Although the view task could be solved using form cues, there was a small impact of motion. Performance in the forward-backward task was highly susceptible to degradations of spatiotemporal stimulus coherence and motion information. These results indicate that rhesus monkeys require extensive training in order to use the intrinsic motion cues related to forward versus backward locomotion and imply that extrapolation of observations concerning human perception of impoverished biological motion displays onto monkey perception needs to be made cautiously.

Tracker trees for unusual event detection

We present an approach for unusual event detection, based on a tree of trackers. At lower levels, the trackers are trained on broad classes of targets. At higher levels, they aim at more specific targets. For instance, at the root, a general blob tracker could operate which may track any object. The next level could already use information about human appearance to better track people. A further level could go after specific types of actions like walking, running, or sitting. Yet another level up, several walking trackers can be tuned to the gait of a particular person each. Thus, at each layer, one or more families of more specific trackers are available. As long as the target behaves according to expectations, a member of a higher up such family will be better tuned to the data than its parent tracker at a lower level. Typically, a better informed tracker performs more robustly. But in cases where unusual events occur and the normal assumptions about the world no longer hold, they loose their reliability. In such cases, a less informed tracker, not relying on what has now become false information, has a good chance of performing better. Such performance inversion signals an unusual event. Inversions between levels higher up represent deviations that are semantically more subtle than inversions lower down: for instance an unknown intruder entering a house rather than seeing a non-human target.

Learning generative models for monocular body pose estimation

We consider the problem of monocular 3d body pose tracking from video sequences. This task is inherently ambiguous. We propose to learn a generative model of the relationship of body pose and image appearance using a sparse kernel regressor. Within a particle filtering framework, the potentially multimodal posterior probability distributions can then be inferred. The 2d bounding box location of the person in the image is estimated along with its body pose. Body poses are modelled on a low-dimensional manifold, obtained by LLE dimensionality reduction. In addition to the appearance model, we learn a prior model of likely body poses and a nonlinear dynamical model, making both pose and bounding box estimation more robust. The approach is evaluated on a number of challenging video sequences, showing the ability of the approach to deal with low-resolution images and noise.

Analysis of Human Locomotion based on Partial Measurements

A lot of computer vision applications have to deal with occlusions. In such settings only a subset of the features of interest can be observed, i.e. only incomplete or partial measurements are available. In this article we show how a learned statistical model can be used to make a prediction of the unknown (occluded) features. The probabilistic nature of the framework also allows to compute the remaining uncertainty given an incomplete observation. The resulting posterior probability distribution can then be used for inference. Additional unknowns such as alignment or scale are easily incorporated into the framework. Instead of computing the alignment in a preprocessing step, it is left as an additional uncertainty, similar to the uncertainty introduced by the missing values of the measurement. It is shown how the technique can be applied to the analysis of human loco-motion, when body parts are occluded. Experiments show how the unobserved body locations are predicted and how it can be inferred whether the measurements come from a running or walking sequence.

Model-based sparse 3D reconstruction for online body tracking

In this paper a new approach to 3D human body tracking is proposed. A sparse 3D reconstruction of the subject to be tracked is made using a structured light system consisting of a precalibrated LCD projector and a camera. At a number of points-of-interest, easily detectable features are projected. The resulting sparse 3D reconstruction is used to estimate the body pose of the tracked person. This new estimate of the body pose is then fed back to the structured light system and allows to adapt the projected patterns, i.e. decide where to project features. Given the observations, a physical simulation is used to estimate the body pose by attaching forces to the limbs of the body model. The sparse 3D observations are augmented by denser silhouette information, in order to make the tracking more robust. Experiments demonstrate the feasibility of the proposed approach and show that the high speeds that are required due to the closed feedback loop can be achieved.