Reza Bahmanyar

A Comparative Study of Bag-of-Words and Bag-of-Topics Models of EO Image Patches

The large volume of detailed land cover features, provided by high resolution Earth observation (EO) images, has attracted considerable interest in the discovery of these features by learning systems. In this letter, we perform latent Dirichlet allocation on the bag of words (BoW) representation of collections of EO image patches to discover their semantic-level features, the so-called topics. To assess the discovered topics, the images are represented based on the occurrence of different topics, called bag of topics (BoT). The value added by BoT to the BoW model of image patches is then measured based on existing human annotations of the data. In our experiments, we compare the classification accuracy results of BoT and BoW representations of two different remote sensing image data sets, a multispectral optical data set and a synthetic-aperture-radar data set. Experimental results demonstrate that BoT can provide a compact and semantically meaningful representation of data; it either causes no significant reduction in the classification accuracy or increases the accuracy by a sufficient number of topics.

The Semantic Gap: An Exploration of User and Computer Perspectives in Earth Observation Images

Research on the semantic gap has considered differences between user and computer image interpretations and proposed methods to bridge it. These methods have been verified by comparing results to reference data or by measuring the degree of user acceptance. Although these methods result in a narrower semantic gap between computers and users, the resulting model for a specific user and search goal may still not be satisfactory to other users. Through an image annotation task with users, we find that this discrepancy is caused by the subjective biases present in the bridging methods, which we refer to as the “linguistic semantic gap.” Based on our findings, efforts to bridge the semantic gap should include different user perspectives to compensate the individual subjective biases, by increasing the diversity of data sets used in the domain. Moreover, models derived from proposed bridging methods could be stored and further used by other systems.

Farness preserving Non-negative matrix factorization

Dramatic growth in the volume of data made a compact and informative representation of the data highly demanded in computer vision, information retrieval, and pattern recognition. Non-negative Matrix Factorization (NMF) is used widely to provide parts-based representations by factorizing the data matrix into non-negative matrix factors. Since non-negativity constraint is not sufficient to achieve robust results, variants of NMF have been introduced to exploit the geometry of the data space. While these variants considered the local invariance based on the manifold assumption, we propose Farness preserving Non-negative Matrix Factorization (FNMF) to exploits the geometry of the data space by considering non-local invariance which is applicable to any data structure. FNMF adds a new constraint to enforce the far points (i.e., non-neighbors) in original space to stay far in the new space. Experiments on different kinds of data (e.g., Multimedia, Earth Observation) demonstrate that FNMF outperforms the other variants of NMF.

Discovery of Semantic Relationships in PolSAR Images Using Latent Dirichlet Allocation

We propose a multilevel semantics discovery approach for bridging the semantic gap when mining high-resolution polarimetric synthetic aperture radar (PolSAR) remote sensing images. First, an Entropy/Anisotropy/Alpha-Wishart classifier is employed to discover low-level semantics as classes representing the physical scattering properties of targets (e.g., low-entropy/surface scattering/high anisotropy). Then, the images are tiled into patches and each patch is modeled as a bag-of-words, a histogram of the class labels. Next, latent Dirichlet allocation is applied to discover their higher level semantics as a set of topics. Our results demonstrate that topic semantics are close to human semantics used for basic land-cover types (e.g., grassland). Therefore, using the topic description (bag-of-topics) of PolSAR images leads to a narrower semantic gap in image mining. In addition, a visual exploration of the topic descriptions helps to find semantic relationships, which can be used for defining new semantic categories (e.g., mixed land-cover types) and designing rule-based categorization schemes.

Building Outline Extraction Using a Heuristic Approach Based on Generalization of Line Segments

Efficient and fully automatic building outline extraction and simplification methods are highly demanded for three-dimensional model reconstruction tasks. In spite of the efforts put into developing such methods, the results of the recently proposed methods are still not satisfactory, especially for satellite images, due to object complexities and the presence of noise. Dealing with this problem, in this article, we propose a new approach that detects rough building boundaries (building mask) from Digital Surface Model data and then refines the resulting mask by classifying the geometrical features of the high spatial resolution panchromatic satellite image. The refined mask represents finer details of the building outlines, which are close to the original building edges. These outlines are then simplified through a parameterization phase wherein a tracing algorithm detects the building boundary points from the refined masks and a set of line segments is fitted to them. After that, for each building, the existing main orientations are determined based on the length and arc lengths of the buildings line segments. Our method is able to determine the multiple main orientations of complex buildings. Through a regularization process, the line segments are then aligned and adjusted according to the buildings main orientations. Finally, the adjusted line segments are intersected and connected to each other in order to form a polygon representing the buildings outlines. Experimental results demonstrate that the computed building outlines are highly accurate and simple, even for large and complex buildings with inner yards.

Evaluating the Sensory Gap for Earth Observation Images Using Human Perception and an LDA-Based Computational Model

High resolution Earth Observation (EO) images contain detailed information, making it possible to recognize objects. However, issues such as the sensory gap (the difference between a real life scene and its sensory interpretation) cause difficulties for object recognition. In EO, this gap is rather wide due to sensor resolution, image perspective, scale and field of view (FOV). In this work, human perceptual and computational evaluations of the sensory gap are presented. For the human perceptual evaluation, user labels describing image patch content are gathered and analyzed. Results highlight issues caused by the sensory gap, e.g., FOV (image patch size) limits the contextual clues which can be used to disambiguate objects. The effect of FOV is then computationally analyzed as the difference between the scene context discovered by Latent Dirichlet Allocation from content within a certain FOV and the ground truth. Results indicate that increasing the FOV decreases the sensory gap.

Multisensor Earth Observation Image Classification Based on a Multimodal Latent Dirichlet Allocation Model

Many previous researches have already shown the advantages of multisensor land-cover classification. Here, we propose an innovative land-cover classification approach based on learning a joint latent model of synthetic aperture radar (SAR) and multispectral satellite images using multimodal latent Dirichlet allocation (mmLDA), a probabilistic generative model. It has already been successfully applied to various other problems dealing with multimodal data. For our experiments, we chose overlapping SAR and multispectral images of two regions of interest. The images were tiled into patches and their local primitive features were extracted. Then each image patch is represented by SAR and multispectral bag-of-words (BoW) models. The BoW values are both fed to the mmLDA, resulting in a joint latent data model. A qualitative and quantitative validation of the topics based on ground-truth data demonstrate that the land-cover categories of the regions are correctly classified, outperforming the topics obtained using individual single modality data.

Land-cover change detection using local feature descriptors extracted from spectral indices

An effective monitoring and analysis of ecosystems requires developing new tools and knowledge. In this paper, we propose an approach for detecting land-cover changes using satellite Image Time Series. This approach represents each image by spectral indices and then extracts local features of these representations. Next, a clustering technique (e.g., k-means) is applied to the extracted features, where the resulting clusters are assumed to refer to land-cover classes. The land-cover change is then obtained by counting the number of times an assigned class to each point changes along the time series. For our experiments, we use a collection of Landsat-5 images captured every second month from October 2009 to August 2010 over the protected area of the Doñana National Park in southwestern Spain, which is the largest sanctuary for migratory birds in western Europe. Results demonstrate that the proposed approach can detect the occurring changes in the main land-cover categories along the assessed time series.

Earth Observation Image Semantic Bias: A Collaborative User Annotation Approach

Correctly annotated image datasets are important for developing and validating image mining methods. However, there is some doubt regarding the generalizability of the models trained and validated on available datasets. This is due to dataset biases, which occur when the same semantic label is used in different ways across datasets, and/or when identical object categories are labeled differently across datasets. In this paper, we demonstrate the existence of dataset biases with a sample of eight remote sensing image datasets, first showing they are readily discriminable from a feature perspective, and then demonstrating that a model trained on one dataset is not always valid on others. Past approaches to reducing dataset biases have relied on crowdsourcing, however this is not always an option (e.g., due to public-accessibility restrictions of images), raising the question: How to structure annotation tasks to efficiently and accurately annotate images with a limited number of nonexpert annotators? We propose a collaborative annotation methodology, conducting image annotation experiments where users are placed in either a collaborative or individual condition, and we analyze their annotation performance. Results show the collaborators produce more thorough, precise annotations, requiring less time than the individuals. Collaborators labels show less variance around the consensus point, meaning their assigned labels are more predictable and likely to be generally accepted by other users. Therefore, collaborative image annotation is a promising annotation methodology for creating reliable datasets with a reduced number of nonexpert annotators. This in turn has implications for the creation of less biased image datasets.

Land-cover evolution class analysis in Image Time Series of Landsat and Sentinel-2 based on Latent Dirichlet Allocation

Satellite Image Time Series (SITS) are widely used in monitoring the Earths changes for various applications such as land-cover evolution analysis. In this paper, we propose an approach based on Latent Dirichlet Allocation (LDA) which considers spatial and spectral information to measure the land-cover changes in multispectral SITS. For our experiments, we focus on the vegetation dynamics of the Doñana National Park (in southwestern Spain) using a Landsat and a Sentinel-2 SITS dataset. The proposed approach represents each image by Normalized Difference Vegetation Index (NDVI) and tiles it into smaller patches. The patches are then modeled as Bag-of-Words (BoW) and LDA is applied to them in order to discover the latent structure of the image. The divergence between the latent structures of any two consecutive images is then considered as the measure of change. Results show that the changes measured by the proposed approach can represent the vegetation dynamics of the region of interest. Moreover, comparing the results obtained from the two datasets demonstrates that using high-level information allows the proposed approach to measure the changes independent of the sensor. This will support long-term monitoring through combining various available data.