Charless C. Fowlkes

Contour Detection and Hierarchical Image Segmentation

This paper investigates two fundamental problems in computer vision: contour detection and image segmentation. We present state-of-the-art algorithms for both of these tasks. Our contour detector combines multiple local cues into a globalization framework based on spectral clustering. Our segmentation algorithm consists of generic machinery for transforming the output of any contour detector into a hierarchical region tree. In this manner, we reduce the problem of image segmentation to that of contour detection. Extensive experimental evaluation demonstrates that both our contour detection and segmentation methods significantly outperform competing algorithms. The automatically generated hierarchical segmentations can be interactively refined by user-specified annotations. Computation at multiple image resolutions provides a means of coupling our system to recognition applications.

Learning to detect natural image boundaries using local brightness, color, and texture cues

The goal of this work is to accurately detect and localize boundaries in natural scenes using local image measurements. We formulate features that respond to characteristic changes in brightness, color, and texture associated with natural boundaries. In order to combine the information from these features in an optimal way, we train a classifier using human labeled images as ground truth. The output of this classifier provides the posterior probability of a boundary at each image location and orientation. We present precision-recall curves showing that the resulting detector significantly outperforms existing approaches. Our two main results are 1) that cue combination can be performed adequately with a simple linear model and 2) that a proper, explicit treatment of texture is required to detect boundaries in natural images.

Spectral grouping using the Nystrom method

Spectral graph theoretic methods have recently shown great promise for the problem of image segmentation. However, due to the computational demands of these approaches, applications to large problems such as spatiotemporal data and high resolution imagery have been slow to appear. The contribution of this paper is a method that substantially reduces the computational requirements of grouping algorithms based on spectral partitioning making it feasible to apply them to very large grouping problems. Our approach is based on a technique for the numerical solution of eigenfunction problems known as the Nystrom method. This method allows one to extrapolate the complete grouping solution using only a small number of samples. In doing so, we leverage the fact that there are far fewer coherent groups in a scene than pixels.

Globally-optimal greedy algorithms for tracking a variable number of objects

We analyze the computational problem of multi-object tracking in video sequences. We formulate the problem using a cost function that requires estimating the number of tracks, as well as their birth and death states. We show that the global solution can be obtained with a greedy algorithm that sequentially instantiates tracks using shortest path computations on a flow network. Greedy algorithms allow one to embed pre-processing steps, such as nonmax suppression, within the tracking algorithm. Furthermore, we give a near-optimal algorithm based on dynamic programming which runs in time linear in the number of objects and linear in the sequence length. Our algorithms are fast, simple, and scalable, allowing us to process dense input data. This results in state-of-the-art performance.

From contours to regions: An empirical evaluation

We propose a generic grouping algorithm that constructs a hierarchy of regions from the output of any contour detector. Our method consists of two steps, an oriented watershed transform (OWT) to form initial regions from contours, followed by construction of an ultra-metric contour map (UCM) defining a hierarchical segmentation. We provide extensive experimental evaluation to demonstrate that, when coupled to a high-performance contour detector, the OWT-UCM algorithm produces state-of-the-art image segmentations. These hierarchical segmentations can optionally be further refined by user-specified annotations.

Using contours to detect and localize junctions in natural images

Contours and junctions are important cues for perceptual organization and shape recognition. Detecting junctions locally has proved problematic because the image intensity surface is confusing in the neighborhood of a junction. Edge detectors also do not perform well near junctions. Current leading approaches to junction detection, such as the Harris operator, are based on 2D variation in the intensity signal. However, a drawback of this strategy is that it confuses textured regions with junctions. We believe that the right approach to junction detection should take advantage of the contours that are incident at a junction; contours themselves can be detected by processes that use more global approaches. In this paper, we develop a new high-performance contour detector using a combination of local and global cues. This contour detector provides the best performance to date (F=0.70) on the Berkeley Segmentation Dataset (BSDS) benchmark. From the resulting contours, we detect and localize candidate junctions, taking into account both contour salience and geometric configuration. We show that improvements in our contour model lead to better junctions. Our contour and junction detectors both provide state of the art performance.

Three-dimensional morphology and gene expression in the Drosophila blastoderm at cellular resolution I: data acquisition pipeline

BackgroundTo model and thoroughly understand animal transcription networks, it is essential to derive accurate spatial and temporal descriptions of developing gene expression patterns with cellular resolution.ResultsHere we describe a suite of methods that provide the first quantitative three-dimensional description of gene expression and morphology at cellular resolution in whole embryos. A database containing information derived from 1,282 embryos is released that describes the mRNA expression of 22 genes at multiple time points in the Drosophila blastoderm. We demonstrate that our methods are sufficiently accurate to detect previously undescribed features of morphology and gene expression. The cellular blastoderm is shown to have an intricate morphology of nuclear density patterns and apical/basal displacements that correlate with later well-known morphological features. Pair rule gene expression stripes, generally considered to specify patterning only along the anterior/posterior body axis, are shown to have complex changes in stripe location, stripe curvature, and expression level along the dorsal/ventral axis. Pair rule genes are also found to not always maintain the same register to each other.ConclusionThe application of these quantitative methods to other developmental systems will likely reveal many other previously unknown features and provide a more rigorous understanding of developmental regulatory networks.

Multiresolution models for object detection

Most current approaches to recognition aim to be scale-invariant. However, the cues available for recognizing a 300 pixel tall object are qualitatively different from those for recognizing a 3 pixel tall object. We argue that for sensors with finite resolution, one should instead use scale-variant, or multiresolution representations that adapt in complexity to the size of a putative detection window. We describe a multiresolution model that acts as a deformable part-based model when scoring large instances and a rigid template with scoring small instances. We also examine the interplay of resolution and context, and demonstrate that context is most helpful for detecting low-resolution instances when local models are limited in discriminative power. We demonstrate impressive results on the Caltech Pedestrian benchmark, which contains object instances at a wide range of scales. Whereas recent state-of-the-art methods demonstrate missed detection rates of 86%-37% at 1 false-positive-per-image, our multiresolution model reduces the rate to 29%.

A Conserved Developmental Patterning Network Produces Quantitatively Different Output in Multiple Species of Drosophila

Differences in the level, timing, or location of gene expression can contribute to alternative phenotypes at the molecular and organismal level. Understanding the origins of expression differences is complicated by the fact that organismal morphology and gene regulatory networks could potentially vary even between closely related species. To assess the scope of such changes, we used high-resolution imaging methods to measure mRNA expression in blastoderm embryos of Drosophila yakuba and Drosophila pseudoobscura and assembled these data into cellular resolution atlases, where expression levels for 13 genes in the segmentation network are averaged into species-specific, cellular resolution morphological frameworks. We demonstrate that the blastoderm embryos of these species differ in their morphology in terms of size, shape, and number of nuclei. We present an approach to compare cellular gene expression patterns between species, while accounting for varying embryo morphology, and apply it to our data and an equivalent dataset for Drosophila melanogaster. Our analysis reveals that all individual genes differ quantitatively in their spatio-temporal expression patterns between these species, primarily in terms of their relative position and dynamics. Despite many small quantitative differences, cellular gene expression profiles for the whole set of genes examined are largely similar. This suggests that cell types at this stage of development are conserved, though they can differ in their relative position by up to 3–4 cell widths and in their relative proportion between species by as much as 5-fold. Quantitative differences in the dynamics and relative level of a subset of genes between corresponding cell types may reflect altered regulatory functions between species. Our results emphasize that transcriptional networks can diverge over short evolutionary timescales and that even small changes can lead to distinct output in terms of the placement and number of equivalent cells.

A gene expression atlas of a bicoid -depleted Drosophila embryo reveals early canalization of cell fate

In developing embryos, gene regulatory networks drive cells towards discrete terminal fates, a process called canalization. We studied the behavior of the anterior-posterior segmentation network in Drosophila melanogaster embryos by depleting a key maternal input, bicoid (bcd), and measuring gene expression patterns of the network at cellular resolution. This method results in a gene expression atlas containing the levels of mRNA or protein expression of 13 core patterning genes over six time points for every cell of the blastoderm embryo. This is the first cellular resolution dataset of a genetically perturbed Drosophila embryo that captures all cells in 3D. We describe the technical developments required to build this atlas and how the method can be employed and extended by others. We also analyze this novel dataset to characterize the degree and timing of cell fate canalization in the segmentation network. We find that in two layers of this gene regulatory network, following depletion of bcd, individual cells rapidly canalize towards normal cell fates. This result supports the hypothesis that the segmentation network directly canalizes cell fate, rather than an alternative hypothesis whereby cells are initially mis-specified and later eliminated by apoptosis. Our gene expression atlas provides a high resolution picture of a classic perturbation and will enable further computational modeling of canalization and gene regulation in this transcriptional network. Summary: Drosophila bicoid mutant embryos show severe patterning phenotypes, but individual cells retain wild-type fates - as judged by expression of key patterning genes at single-cell resolution.