Shawn D. Newsam

Bag-of-visual-words and spatial extensions for land-use classification

We investigate bag-of-visual-words (BOVW) approaches to land-use classification in high-resolution overhead imagery. We consider a standard non-spatial representation in which the frequencies but not the locations of quantized image features are used to discriminate between classes analogous to how words are used for text document classification without regard to their order of occurrence. We also consider two spatial extensions, the established spatial pyramid match kernel which considers the absolute spatial arrangement of the image features, as well as a novel method which we term the spatial co-occurrence kernel that considers the relative arrangement. These extensions are motivated by the importance of spatial structure in geographic data. The methods are evaluated using a large ground truth image dataset of 21 land-use classes. In addition to comparisons with standard approaches, we perform extensive evaluation of different configurations such as the size of the visual dictionaries used to derive the BOVW representations and the scale at which the spatial relationships are considered. We show that even though BOVW approaches do not necessarily perform better than the best standard approaches overall, they represent a robust alternative that is more effective for certain land-use classes. We also show that extending the BOVW approach with our proposed spatial co-occurrence kernel consistently improves performance.

A texture descriptor for browsing and similarity retrieval

Image texture is useful in image browsing, search and retrieval. A texture descriptor based on a multiresolution decomposition using Gabor wavelets is proposed. The descriptor consists of two parts: a perceptual browsing component (PBC) and a similarity retrieval component (SRC). The extraction methods of both PBC and SRC are based on a multiresolution decomposition using Gabor wavelets. PBC provides a quantitative characterization of the texture’s structuredness and directionality for browsing application, and the SRC characterizes the distribution of texture energy in di!erent subbands, and supports similarity retrieval. This representation is quite robust to illumination variations and compares favorably with other texture descriptors for similarity retrieval. Experimental results are provided. ( 2000 Elsevier Science B.V. All rights reserved.

A bimodal distribution of two distinct categories of intrinsically-disordered structures with separate functions in FG nucleoporins

Nuclear pore complexes (NPCs) gate the only conduits for nucleocytoplasmic transport in eukaryotes. Their gate is formed by nucleoporins containing large intrinsically disordered domains with multiple phenylalanine-glycine repeats (FG domains). In combination, these are hypothesized to form a structurally and chemically homogeneous network of random coils at the NPC center, which sorts macromolecules by size and hydrophobicity. Instead, we found that FG domains are structurally and chemically heterogeneous. They adopt distinct categories of intrinsically disordered structures in non-random distributions. Some adopt globular, collapsed coil configurations and are characterized by a low charge content. Others are highly charged and adopt more dynamic, extended coil conformations. Interestingly, several FG nucleoporins feature both types of structures in a bimodal distribution along their polypeptide chain. This distribution functionally correlates with the attractive or repulsive character of their interactions with collapsed coil FG domains displaying cohesion toward one another and extended coil FG domains displaying repulsion. Topologically, these bipartite FG domains may resemble sticky molten globules connected to the tip of relaxed or extended coils. Within the NPC, the crowding of FG nucleoporins and the segregation of their disordered structures based on their topology, dimensions, and cohesive character could force the FG domains to form a tubular gate structure or transporter at the NPC center featuring two separate zones of traffic with distinct physicochemical properties.

Spatial pyramid co-occurrence for image classification

We describe a novel image representation termed spatial pyramid co-occurrence which characterizes both the photometric and geometric aspects of an image. Specifically, the co-occurrences of visual words are computed with respect to spatial predicates over a hierarchical spatial partitioning of an image. The representation captures both the absolute and relative spatial arrangement of the words and, through the choice and combination of the predicates, can characterize a variety of spatial relationships. Our representation is motivated by the analysis of overhead imagery such as from satellites or aircraft. This imagery generally does not have an absolute reference frame and thus the relative spatial arrangement of the image elements often becomes the key discriminating feature. We validate this hypothesis using a challenging ground truth image dataset of 21 land-use classes manually extracted from high-resolution aerial imagery. Our approach is shown to result in higher classification rates than a non-spatial bagof- visual-words approach as well as a popular approach for characterizing the absolute spatial arrangement of visual words, the spatial pyramid representation of Lazebnik et al. [7]. While our primary objective is analyzing overhead imagery, we demonstrate that our approach achieves state-of-the-art performance on the Graz-01 object class dataset and performs competitively on the 15 Scene dataset.

Geographic Image Retrieval Using Local Invariant Features

This paper investigates local invariant features for geographic (overhead) image retrieval. Local features are particularly well suited for the newer generations of aerial and satellite imagery whose increased spatial resolution, often just tens of centimeters per pixel, allows a greater range of objects and spatial patterns to be recognized than ever before. Local invariant features have been successfully applied to a broad range of computer vision problems and, as such, are receiving increased attention from the remote sensing community particularly for challenging tasks such as detection and classification. We perform an extensive evaluation of local invariant features for image retrieval of land-use/land-cover (LULC) classes in high-resolution aerial imagery. We report on the effects of a number of design parameters on a bag-of-visual-words (BOVW) representation including saliency- versus grid-based local feature extraction, the size of the visual codebook, the clustering algorithm used to create the codebook, and the dissimilarity measure used to compare the BOVW representations. We also perform comparisons with standard features such as color and texture. The performance is quantitatively evaluated using a first-of-its-kind LULC ground truth data set which will be made publicly available to other researchers. In addition to reporting on the effects of the core design parameters, we also describe interesting findings such as the performance-efficiency tradeoffs that are possible through the appropriate pairings of different-sized codebooks and dissimilarity measures. While the focus is on image retrieval, we expect our insights to be informative for other applications such as detection and classification.

Retrieval using texture features in high-resolution multispectral satellite imagery

Texture features have long been used in remote sensing applications to represent and retrieve image regions similar to a query region. Various representations of texture have been proposed based on the Fourier power spectrum, spatial co-occurrence, wavelets, Gabor filters, etc. These representations vary in their computational complexity and their suitability for representing different region types. Much of the work done thus far has focused on panchromatic imagery at low to moderate spatial resolutions, such as images from Landsat 1-7 which have a resolution of 15-30 m/pixel, and from SPOT 1-5 which have a resolution of 2.5-20 m/pixel. However, it is not clear which texture representation works best for the new classes of high resolution panchromatic (60-100 cm/pixel) and multi-spectral (4 bands for red, green, blue, and near infra-red at 2.4-4 m/pixel) imagery. It is also not clear how the different spectral bands should be combined. In this paper, we investigate the retrieval performance of several different texture representations using multi-spectral satellite images from IKONOS. A query-by-example framework, along with a manually chosen ground truth dataset, allows different combinations of texture representations and spectral bands to be compared. We focus on the specific problem of retrieving inhabited regions from images of urban and rural scenes. Preliminary results show that 1) the use of all spectral bands improves the retrieval performance, and 2) co-occurrence, wavelet and Gabor texture features perform comparably.

Feature selection in scientific applications

Numerous applications of data mining to scientific data involve the induction of a classification model. In many cases, the collection of data is not performed with this task in mind, and therefore, the data might contain irrelevant or redundant features that affect negatively the accuracy of the induction algorithms. The size and dimensionality of typical scientific data make it difficult to use any available domain information to identify features that discriminate between the classes of interest. Similarly, exploratory data analysis techniques have limitations on the amount and dimensionality of the data they can process effectively. In this paper, we describe applications of efficient feature selection methods to data sets from astronomy, plasma physics, and remote sensing. We use variations of recently proposed filter methods as well as traditional wrapper approaches, where practical. We discuss the general challenges of feature selection in scientific datasets, the strategies for success that were common among our diverse applications, and the lessons learned in solving these problems.

A texture descriptor for image retrieval and browsing

Image texture is an important visual primitive in image search and retrieval applications. To characterize texture information of images, a texture descriptor is proposed to capture both the perceptual information and the statistics of textures. The descriptor consists of two parts: a Perceptual Browsing Component (PBC) and a Similarity Retrieval Component (SRC). The emphasis of the paper is to introduce the extraction method for the PBC which is based on multidimensional decomposition of images using Gabor filters. By analyzing the one dimensional projections resulting from filtered images, PBC gives a quantitative measure about the structuredness of textures and also identifies the directionality and coarseness (scale). Our experimental results demonstrate that PBC can capture these texture properties quite well.

Comparing SIFT descriptors and gabor texture features for classification of remote sensed imagery

A richer set of land-cover classes are observable in satellite imagery than ever before due to the increased sub-meter resolution. Individual objects, such as cars and houses, are now recognizable. This work considers a new category of image descriptors based on local measures of saliency for labelling land-cover classes characterized by identifiable objects. These descriptors have been successfully applied to object recognition in standard (non-remote sensed) imagery. We show they perform comparably to state-of-the-art texture descriptors for classifying complex land-cover classes in high- resolution satellite imagery while being approximately an order of magnitude faster to compute. This speedup makes them attractive for realtime applications. To the best of our knowledge, this is the first time this new category of descriptors has been applied to the classification of remote sensed imagery.

Proximate sensing: Inferring what-is-where from georeferenced photo collections

The primary and novel contribution of this work is the conjecture that large collections of georeferenced photo collections can be used to derive maps of what-is-where on the surface of the earth. We investigate the application of what we term “proximate sensing” to the problem of land cover classification for a large geographic region. We show that our approach is able to achieve almost 75% classification accuracy in a binary land cover labelling problem using images from a photo sharing site in a completely automated fashion. We also investigate 1) how existing geographic knowledge can be used to provide labelled training data in a weakly-supervised manner; 2) the effect of the photographers intent when he or she captures the photograph; and 3) a method for filtering out non-informative images.