Clemens Blumer

Automated analysis of spine dynamics on live CA1 pyramidal cells

Dendritic spines may be tiny in volume, but are of major importance for neuroscience. They are the main receivers for excitatory synaptic connections, and their constant changes in number and in shape reflect the dynamic connectivity of the brain. Two-photon microscopy allows following the fate of individual spines in brain slice preparations and in live animals. The diffraction-limited and non-isotropic resolution of this technique, however, makes detection of such tiny structures rather challenging, especially along the optical axis (z-direction). Here we present a novel spine detection algorithm based on a statistical dendrite intensity model and a corresponding spine probability model. To quantify the fidelity of spine detection, we generated correlative datasets: Following two-photon imaging of live pyramidal cell dendrites, we used serial block-face scanning electron microscopy (SBEM) to reconstruct dendritic ultrastructure in 3D. Statistical models were trained on synthetic fluorescence images generated from SBEM datasets via point spread function (PSF) convolution. After the training period, we tested automatic spine detection on real two-photon datasets and compared the result to ground truth (correlative SBEM data). The performance of our algorithm allowed tracking changes in spine volume automatically over several hours. Using a second fluorescent protein targeted to the endoplasmic reticulum, we could analyze the motion of this organelle inside individual spines. Furthermore, we show that it is possible to distinguish activated spines from non-stimulated neighbors by detection of fluorescently labeled presynaptic vesicle clusters. These examples illustrate how automatic segmentation in 5D (x, y, z, t, λ) allows us to investigate brain dynamics at the level of individual synaptic connections.

Occlusion-aware 3D Morphable Face Models.

We propose a probabilistic occlusion-aware 3D Morphable Face Model adaptation framework for face image analysis based on the Analysis-by-Synthesis setup. In natural images, parts of the face are often occluded by a variety of objects. Such occlusions are a challenge for face model adaptation. We propose to segment the image into face and non-face regions and model them separately. The segmentation and the face model parameters are not known in advance and have to be adapted to the target image. A good segmentation is necessary to obtain a good face model fit and vice-versa. Therefore, face model adaptation and segmentation are solved together using an EM-like procedure. We use a stochastic sampling strategy based on the Metropolis-Hastings algorithm for face model parameter adaptation and a modified Chan-Vese segmentation for face region segmentation. Previous robust methods are limited to homogeneous, controlled illumination settings and tend to fail for important regions such as the eyes or mouth. We propose a RANSAC-based robust illumination estimation technique to handle complex illumination conditions. We do not use any manual annotation and the algorithm is not optimised to any specific kind of occlusion or database. We evaluate our method on a controlled and an “in the wild” database.

Efficient object tracking by condentional and cascaded image sensing

We introduce a robust multi-object tracking for abstract multi-dimensional feature vectors. The Condensation and the Wavelet Approximated Reduced Vector Machine (W-RVM) approach are joined to spend only as much as necessary effort for easy to discriminate regions (Condensation) and measurement locations (W-RVM) of the feature space, but most for regions and locations with high statistical likelihood to contain the object of interest. The new 3D Cascaded Condensation Tracking (CCT) yields more than 10 times faster tracking than state-of-art detection methods. We demonstrate HCI applications by high resolution face tracking within a large camera scene with an active dual camera system.

Coarse-to-fine particle filters for multi-object human computer interaction

Efficient motion tracking of faces is an important aspect for human computer interaction (HCI). In this paper we combine the condensation and the wavelet approximated reduced vector machine (W-RVM) approach. Both are joined by the core idea to spend only as much as necessary effort for easy to discriminate regions (Condensation) or vectors (W-RVM) of the feature space, but most for regions with high statistical likelihood to contain objects of interest. We adapt the W-RVM classifler for tracking by providing a probabilistic output. In this paper we utilize condensation for template based tracking of the three-dimensional camera scene. Moreover, we introduce a robust multi-object tracking by extensions to the condensation approach. The novel coarse-to-flne condensation yields a more than 10 times faster tracking than state-of-art detection methods. We demonstrate more natural HCI applications by high resolution face tracking within a large camera scene with an active dual camera system.

Occlusion-Aware 3D Morphable Models and an Illumination Prior for Face Image Analysis

Faces in natural images are often occluded by a variety of objects. We propose a fully automated, probabilistic and occlusion-aware 3D morphable face model adaptation framework following an analysis-by-synthesis setup. The key idea is to segment the image into regions explained by separate models. Our framework includes a 3D morphable face model, a prototype-based beard model and a simple model for occlusions and background regions. The segmentation and all the model parameters have to be inferred from the single target image. Face model adaptation and segmentation are solved jointly using an expectation–maximization-like procedure. During the E-step, we update the segmentation and in the M-step the face model parameters are updated. For face model adaptation we apply a stochastic sampling strategy based on the Metropolis–Hastings algorithm. For segmentation, we apply loopy belief propagation for inference in a Markov random field. Illumination estimation is critical for occlusion handling. Our combined segmentation and model adaptation needs a proper initialization of the illumination parameters. We propose a RANSAC-based robust illumination estimation technique. By applying this method to a large face image database we obtain a first empirical distribution of real-world illumination conditions. The obtained empirical distribution is made publicly available and can be used as prior in probabilistic frameworks, for regularization or to synthesize data for deep learning methods.

3D CASCADED CONDENSATION TRACKING FOR MULTIPLE OBJECTS

The Condensation and the Wavelet Approximated Reduced Vector Machine (W-RVM) approach are joined by the core idea to spend only as much as necessary effort for easy to discriminate regions (Condensation) and measurement locations (W-RVM) of the feature space, but most for regions and locations with high statistical likelihood to contain the object of interest. We unify both approaches by adapting the W-RVM classifier to tracking and refine the Condensation approach. Additionally, we utilize Condensation for abstract multi-dimensional feature vectors and provide a template based tracking of the three-dimensional camera scene. Moreover, we introduce a robust multi-object tracking by extensions to the Condensation approach. The new 3D Cascaded Condensation Tracking (CCT) for multiple objects yields a more than 10 times faster tracking than state-of-art detection methods. In our experiments we compare different tracking approaches using an active dual camera system for face tracking.

Probabilistic Morphable Models

Abstract 3D Morphable Face Models have been introduced for the analysis of 2D face photographs. The analysis is performed by actively reconstructing the three-dimensional face from the image in an Analysis-by-Synthesis loop, exploring statistical models for shape and appearance. Here we follow a probabilistic approach to acquire a robust and automatic model adaptation. The probabilistic formulation helps to overcome two main limitations of the classical approach. First, Morphable Model adaptation is highly depending on a good initialization. The initial position of landmark points and face pose was given by manual annotation in previous approaches. Our fully probabilistic formulation allows us to integrate unreliable Bottom-Up cues from face and feature point detectors. This integration is superior to the classical feed-forward approach, which is prone to early and possibly wrong decisions. The integration of uncertain Bottom-Up detectors leads to a fully automatic model adaptation process. Second, the probabilistic framework gives us a natural way to handle outliers and occlusions. Face images are recorded in highly unconstrained settings. Often parts of the face are occluded by various objects. Unhandled occlusions can mislead the model adaptation process. The probabilistic interpretation of our model makes possible to detect and segment occluded parts of the image and leads to robust model adaptation. Throughout this chapter we develop a fully probabilistic framework for image interpretation. We start by reformulating the Morphable Model as a probabilistic model in a fully Bayesian framework. Given an image, we search for a posterior distribution of possible image explanations. The integration of Bottom-Up information and the model parameters adaptation is performed using a Data Driven Markov Chain Monte Carlo approach. The face model is extended to be occlusion-aware and explicitly segments the image into face and non-face regions during the model adaptation process. The segmentation and model adaptation is performed in an Expectation-Maximization-style algorithm utilizing a robust illumination estimation method. The presented fully automatic face model adaptation can be used in a wide range of applications like face analysis, face recognition or face image manipulation. Our framework is able to handle images containing strong outliers, occlusions and facial expressions under arbitrary poses and illuminations. Furthermore, the fully probabilistic embedding has the additional advantage that it also delivers the uncertainty of the resulting image interpretation.