Geremy Heitz

Discriminative learning of Markov random fields for segmentation of 3D scan data

We address the problem of segmenting 3D scan data into objects or object classes. Our segmentation framework is based on a subclass of Markov random fields (MRFs) which support efficient graph-cut inference. The MRF models incorporate a large set of diverse features and enforce the preference that adjacent scan points have the same classification label. We use a recently proposed maximum-margin framework to discriminatively train the model from a set of labeled scans; as a result we automatically learn the relative importance of the features for the segmentation task. Performing graph-cut inference in the trained MRF can then be used to segment new scenes very efficiently. We test our approach on three large-scale datasets produced by different kinds of 3D sensors, showing its applicability to both outdoor and indoor environments containing diverse objects.

Statistical shape model generation using nonrigid deformation of a template mesh

Active shape models (ASMs) have been studied extensively for the statistical analysis of three-dimensional shapes. These models can be used as prior information for segmentation and other image analysis tasks. In order to create an ASM, correspondence between surface points on the training shapes must be provided. Various groups have previously investigated methods that attempted to provide correspondences between points on pre-segmented shapes. This requires a time-consuming segmentation stage before the statistical analysis can be performed. This paper presents a method of ASM generation that requires as input only a single segmented template shape obtained from a mean grayscale image across the training set. The triangulated mesh representing this template shape is then propagated to the other shapes in the training set by a nonrigid transformation. The appropriate transformation is determined by intensity-based nonrigid registration of the corresponding grayscale images. Following the transformation of the template, the mesh is treated as an active surface, and evolves towards the image edges while preserving certain curvature constraints. This process results in automatic segmentation of each shape, but more importantly also provides an automatic correspondence between the points on each shape. The resulting meshes are aligned using Procrustes analysis, and a principal component analysis is performed to produce the statistical model. For demonstration, a model of the lower cervical vertebrae (C6 and C7) was created. The resulting model is evaluated for accuracy, compactness, and generalization ability.

Shape-Based Object Localization for Descriptive Classification

Discriminative tasks, including object categorization and detection, are central components of high-level computer vision. However, sometimes we are interested in a finer-grained characterization of the object’s properties, such as its pose or articulation. In this paper we develop a probabilistic method (LOOPS) that can learn a shape and appearance model for a particular object class, and be used to consistently localize constituent elements (landmarks) of the object’s outline in test images. This localization effectively projects the test image into an alternative representational space that makes it particularly easy to perform various descriptive tasks. We apply our method to a range of object classes in cluttered images and demonstrate its effectiveness in localizing objects and performing descriptive classification, descriptive ranking, and descriptive clustering.

3D segmentation of cell boundaries from whole cell cryogenic electron tomography volumes

Cryogenic electron tomography (cryo-ET) has gained increasing interest in recent years due to its ability to image whole cells and subcellular structures in 3D at nanometer resolution in their native environment. However, due to dose restrictions and the inability to acquire high tilt angle images, the reconstructed volumes are noisy and have missing information. Thus, features are unreliable, and precision extraction of the cell boundary is difficult, manual and time intensive. This paper presents an efficient recursive algorithm called BLASTED (Boundary Localization using Adaptive Shape and Texture Discovery) to automatically extract the cell boundary using a conditional random field (CRF) framework in which boundary points and shape are jointly inferred. The algorithm learns the texture of the boundary region progressively, and uses a global shape model and shape-dependent features to propose candidate boundary points on a slice of the membrane. It then updates the shape of that slice by accepting the appropriate candidate points using local spatial clustering, the global shape model, and trained boosted texture classifiers. The BLASTED algorithm segmented the cell membrane over an average of 93% of the length of the cell in 19 difficult cryo-ET datasets.

Object separation in x-ray image sets

In the segmentation of natural images, most algorithms rely on the concept of occlusion. In x-ray images, however, this assumption is violated, since x-ray photons penetrate most materials. In this paper, we introduce SATISφ, a method for separating objects in a set of x-ray images using the property of additivity in log space, where the log-attenuation at a pixel is the sum of the log-attenuations of all objects that the corresponding x-ray passes through. Our method leverages multiple projection views of the same scene from slightly different angles to produce an accurate estimate of attenuation properties of objects in the scene. These properties can be used to identify the material composition of these objects, and are therefore crucial for applications like automatic threat detection. We evaluate SATISφ on a set of collected x-ray scans, showing that it outperforms a standard image segmentation approach and reduces the error of material estimation.