Richard Souvenir

Manifold clustering

Manifold learning has become a vital tool in data driven methods for interpretation of video, motion capture, and handwritten character data when they lie on a low dimensional, nonlinear manifold. This work extends manifold learning to classify and parameterize unlabeled data which lie on multiple, intersecting manifolds. This approach significantly increases the domain to which manifold learning methods can be applied, allowing parameterization of example manifolds such as figure eights and intersecting paths which are quite common in natural data sets. This approach introduces several technical contributions which may be of broader interest, including node-weighted multidimensional scaling and a fast algorithm for weighted low-rank approximation for rank-one weight matrices. We show examples for intersecting manifolds of mixed topology and dimension and demonstrations on human motion capture data.

Learning the viewpoint manifold for action recognition

Researchers are increasingly interested in providing video-based, view-invariant action recognition for human motion. Addressing this problem will lead to more accurate modeling and analysis of the type of unconstrained video commonly collected in the areas of athletics and medicine. Previous viewpoint-invariant methods use multiple cameras in both the training and testing phases of action recognition or require storing many examples of a single action from multiple viewpoints. In this paper, we present a framework for learning a compact representation of primitive actions (e.g., walk, punch, kick, sit) that can be used for video obtained from a single camera for simultaneous action recognition and viewpoint estimation. Using our method, which models the low-dimensional structure of these actions relative to viewpoint, we show recognition rates on a publicly available data set previously only achieved using multiple simultaneous views.

A Survey of Manifold Learning for Images

Many natural image sets are samples of a low-dimensional manifold in the space of all possible images. Understanding this manifold is a key first step in understanding many sets of images, and manifold learning approaches have recently been used within many application domains, including face recognition, medical image segmentation, gait recognition and hand-written character recognition. This paper attempts to characterize the special features of manifold learning on image data sets, and to highlight the value and limitations of these approaches.

Image distance functions for manifold learning

Many natural image sets are samples of a low-dimensional manifold in the space of all possible images. When the image data set is not a linear combination of a small number of basis images, linear dimensionality reduction techniques such as PCA and ICA fail and non-linear dimensionality reduction techniques are required to automatically determine the intrinsic structure of the image set. Recent techniques such as ISOMAP and LLE provide a mapping between the images and a low-dimensional parameterization of the images. This paper specializes general manifold learning by considering a small set of image distance measures that correspond to key transformation groups observed in natural images. This results in more meaningful embeddings for a variety of applications.

On Manifold Structure of Cardiac MRI Data: Application to Segmentation

We develop theory and algorithms to incorporate image manifold constraints in a level set segmentation algorithm. This provides a framework to simultaneously segment every image of data sets that vary due to two degrees of freedom - such as cardiopulmonary MR images which deform due to patient breathing and heartbeats. We derive two formulations: a 4D level set which loosely couples the level set function between neighbors in the 2D image manifold and a multilayer level set function which uses different levels of the level set function to represent shapes that shrink or grow. We characterize the set of shape manifolds that the multilayer level set function can represent, and derive the evolution equations for both frameworks. We offer results of segmenting the left ventricle in cardiopulmonary MRI; by automatically discovering the 2D manifold structure of the image set then simultaneously segmenting every frame. Both extensions improve on frame-by-frame approaches, and a comparison of the results offers insight into their strengths and weaknesses.

Wide-Area Image Geolocalization with Aerial Reference Imagery

We propose to use deep convolutional neural networks to address the problem of cross-view image geolocalization, in which the geolocation of a ground-level query image is estimated by matching to georeferenced aerial images. We use state-of-the-art feature representations for ground-level images and introduce a cross-view training approach for learning a joint semantic feature representation for aerial images. We also propose a network architecture that fuses features extracted from aerial images at multiple spatial scales. To support training these networks, we introduce a massive database that contains pairs of aerial and ground-level images from across the United States. Our methods significantly out-perform the state of the art on two benchmark datasets. We also show, qualitatively, that the proposed feature representations are discriminative at both local and continental spatial scales.

Manifold learning for 4D CT reconstruction of the lung

Computed tomography is used to create models of lung dynamics because it provides high contrast images of lung tissue. Creating 4D CT models which capture dynamics is complicated because clinical CT scanners capture data in slabs that comprise only a small part of the tissue. Commonly, creating 4D reconstruction requires stitching together different lung segments based on an external measure of lung volume. This paper presents a novel method for assembling 4D CT datasets using only the CT data. We use a manifold learning algorithm to parameterize each slab data with respect to the breathing cycle, and an alignment method to coordinate these parameterizations for different sections of the lung. Comparing this data driven parameterization with physiological measurements captured by a belt around the abdomen, we are able to generate slightly smoother reconstructions.

Viewpoint manifolds for action recognition

Action recognition from video is a problem that has many important applications to human motion analysis. In real-world settings, the viewpoint of the camera cannot always be fixed relative to the subject, so view-invariant action recognition methods are needed. Previous view-invariant methods use multiple cameras in both the training and testing phases of action recognition or require storing many examples of a single action from multiple viewpoints. In this paper, we present a framework for learning a compact representation of primitive actions (e.g., walk, punch, kick, sit) that can be used for video obtained from a single camera for simultaneous action recognition and viewpoint estimation. Using our method, which models the low-dimensional structure of these actions relative to viewpoint, we show recognition rates on a publicly available dataset previously only achieved using multiple simultaneous views.

Selecting Degenerate Multiplex PCR Primers

Single Nucleotide Polymorphism (SNP) Genotyping is an important molecular genetics technique in the early stages of producing results that will be useful in the medical field. One of the proposed methods for performing SNP Genotyping requires amplifying regions of DNA surrounding a large number of SNP loci. In order to automate a portion of this method and make the use of SNP Genotyping more widespread, it is important to select a set of primers for the experiment. Selecting these primers can be formulated as the Multiple Degenerate Primer Design (MDPD) problem. An iterative beam-search algorithm, Multiple, Iterative Primer Selector (MIPS), is presented for MDPD. Theoretical and experimental analyses show that this algorithm performs well compared to the limits of degenerate primer design and the number of spurious amplifications should be small. Furthermore, MIPS outperforms an existing algorithm which was designed for a related degenerate primer selection problem.

Isomap and Nonparametric Models of Image Deformation

Isomap is an exemplar of a set of data driven non-linear dimensionality reduction techniques that have shown promise for the analysis of images and video. These methods parameterize each image as coordinates on a low-dimensional manifold, but, unlike PCA, the low dimensional parameters do not have an explicit meaning, and are not natural projection operators between the high and low-dimensional spaces. For the important special case of image sets of an unknown object undergoing an unknown deformation, we show that Isomap gives a valuable pre-processing step to find an ordering of the images in terms of their deformation. Using the continuity of deformation implied in the Isomap ordering allows more accurate solutions for a thin-plate spline deformation from a specific image to all others. This defines a mapping between the Isomap coordinates and a specific deformation, which is extensible to give projection functions between the image space and the Isomap space. Applications of this technique are shown for cardiac MRI images undergoing chest cavity deformation due to patient breathing.