Gui-Song Xia

Transferring Deep Convolutional Neural Networks for the Scene Classification of High-Resolution Remote Sensing Imagery

Learning efficient image representations is at the core of the scene classification task of remote sensing imagery. The existing methods for solving the scene classification task, based on either feature coding approaches with low-level hand-engineered features or unsupervised feature learning, can only generate mid-level image features with limited representative ability, which essentially prevents them from achieving better performance. Recently, the deep convolutional neural networks (CNNs), which are hierarchical architectures trained on large-scale datasets, have shown astounding performance in object recognition and detection. However, it is still not clear how to use these deep convolutional neural networks for high-resolution remote sensing (HRRS) scene classification. In this paper, we investigate how to transfer features from these successfully pre-trained CNNs for HRRS scene classification. We propose two scenarios for generating image features via extracting CNN features from different layers. In the first scenario, the activation vectors extracted from fully-connected layers are regarded as the final image features; in the second scenario, we extract dense features from the last convolutional layer at multiple scales and then encode the dense features into global image features through commonly used feature coding approaches. Extensive experiments on two public scene classification datasets demonstrate that the image features obtained by the two proposed scenarios, even with a simple linear classifier, can result in remarkable performance and improve the state-of-the-art by a significant margin. The results reveal that the features from pre-trained CNNs generalize well to HRRS datasets and are more expressive than the low- and mid-level features. Moreover, we tentatively combine features extracted from different CNN models for better performance.

Unsupervised Feature Learning Via Spectral Clustering of Multidimensional Patches for Remotely Sensed Scene Classification

Scene classification plays an important role in the interpretation of remotely sensed high-resolution imagery. However, the performance of scene classification strongly relies on the discriminative power of feature representation, which is generally hand-engineered and requires a huge amount of domain-expert knowledge as well as time-consuming hand tuning. Recently, unsupervised feature learning (UFL) provides an alternative way to automatically learn discriminative feature representation from images. However, the performances achieved by conventional UFL methods are not comparable to the state-of-the-art, mainly due to the neglect of locally substantial image structures. This paper presents an improved UFL algorithm based on spectral clustering, named UFL-SC, which cannot only adaptively learn good local feature representations but also discover intrinsic structures of local image patches. In contrast to the standard UFL pipeline, UFL-SC first maps the original image patches into a low-dimensional and intrinsic feature space by linear manifold analysis techniques, and then learns a dictionary (e.g., using K-means clustering) on the patch manifold for feature encoding. To generate a feature representation for each local patch, an explicit parameterized feature encoding method, i.e., triangle encoding, is applied with the learned dictionary on the same patch manifold. The holistic feature representation of image scenes is finally obtained by building a bag-of-visual-words (BOW) model of the encoded local features. Experiments demonstrate that the proposed UFL-SC algorithm can extract efficient local features for image scenes and show comparable performance to the state-of-the-art approach on open scene classification benchmark.

Shape-based Invariant Texture Indexing

This paper introduces a new texture analysis scheme, which is invariant to local geometric and radiometric changes. The proposed methodology relies on the topographic map of images, obtained from the connected components of level sets. This morphological tool, providing a multi-scale and contrast-invariant representation of images, is shown to be well suited to texture analysis. We first make use of invariant moments to extract geometrical information from the topographic map. This yields features that are invariant to local similarities or local affine transformations. These features are invariant to any local contrast change. We then relax this invariance by computing additional features that are invariant to local affine contrast changes and investigate the resulting analysis scheme by performing classification and retrieval experiments on three texture databases. The obtained experimental results outperform the current state of the art in locally invariant texture analysis.

Learning High-level Features for Satellite Image Classification With Limited Labeled Samples

This paper presents a novel method addressing the classification task of satellite images when limited labeled data is available together with a large amount of unlabeled data. Instead of using semi-supervised classifiers, we solve the problem by learning a high-level features, called semisupervised ensemble projection (SSEP). More precisely, we propose to represent an image by projecting it onto an ensemble of weak training (WT) sets sampled from a Gaussian approximation of multiple feature spaces. Given a set of images with limited labeled ones, we first extract preliminary features, e.g., color and textures, to form a low-level image description. We then propose a new semisupervised sampling algorithm to build an ensemble of informative WT sets by exploiting these feature spaces with a Gaussian normal affinity, which ensures both the reliability and diversity of the ensemble. Discriminative functions are subsequently learned from the resulting WT sets, and each image is represented by concatenating its projected values onto such WT sets for final classification. Moreover, we consider that the potential redundant information existed in SSEP and use sparse coding to reduce it. Experiments on high-resolution remote sensing data demonstrate the efficiency of the proposed method.

AID: A Benchmark Data Set for Performance Evaluation of Aerial Scene Classification

Aerial scene classification, which aims to automatically label an aerial image with a specific semantic category, is a fundamental problem for understanding high-resolution remote sensing imagery. In recent years, it has become an active task in the remote sensing area, and numerous algorithms have been proposed for this task, including many machine learning and data-driven approaches. However, the existing data sets for aerial scene classification, such as UC-Merced data set and WHU-RS19, contain relatively small sizes, and the results on them are already saturated. This largely limits the development of scene classification algorithms. This paper describes the Aerial Image data set (AID): a large-scale data set for aerial scene classification. The goal of AID is to advance the state of the arts in scene classification of remote sensing images. For creating AID, we collect and annotate more than 10000 aerial scene images. In addition, a comprehensive review of the existing aerial scene classification techniques as well as recent widely used deep learning methods is given. Finally, we provide a performance analysis of typical aerial scene classification and deep learning approaches on AID, which can be served as the baseline results on this benchmark.

Accurate Junction Detection and Characterization in Natural Images

Accurate junction detection and characterization are of primary importance for several aspects of scene analysis, including depth recovery and motion analysis. In this work, we introduce a generic junction analysis scheme. The first asset of the proposed procedure is an automatic criterion for the detection of junctions, permitting to deal with textured parts in which no detection is expected. Second, the method yields a characterization of L-, Y- and X- junctions, including a precise computation of their type, localization and scale. Contrary to classical approaches, scale characterization does not rely on the linear scale-space. First, an a contrario approach is used to compute the meaningfulness of a junction. This approach relies on a statistical modeling of suitably normalized gray level gradients. Then, exclusion principles between junctions permit their precise characterization. We give implementation details for this procedure and evaluate its efficiency through various experiments.

Dirichlet-Derived Multiple Topic Scene Classification Model for High Spatial Resolution Remote Sensing Imagery

Due to the complex arrangements of the ground objects in high spatial resolution (HSR) imagery scenes, HSR imagery scene classification is a challenging task, which is aimed at bridging the semantic gap between the low-level features and the high-level semantic concepts. A combination of multiple complementary features for HSR imagery scene classification is considered a potential way to improve the performance. However, the different types of features have different characteristics, and how to fuse the different types of features is a classic problem. In this paper, a Dirichlet-derived multiple topic model (DMTM) is proposed to fuse heterogeneous features at a topic level for HSR imagery scene classification. An efficient algorithm based on a variational expectation-maximization framework is developed to infer the DMTM and estimate the parameters of the DMTM. The proposed DMTM scene classification method is able to incorporate different types of features with different characteristics, no matter whether these features are local or global, discrete or continuous. Meanwhile, the proposed DMTM can also reduce the dimension of the features representing the HSR images. In our experiments, three types of heterogeneous features, i.e., the local spectral feature, the local structural feature, and the global textural feature, were employed. The experimental results with three different HSR imagery data sets show that the three types of features are complementary. In addition, the proposed DMTM is able to reduce the dimension of the features representing the HSR images, to fuse the different types of features efficiently, and to improve the performance of the scene classification over that of other scene classification algorithms based on spatial pyramid matching, probabilistic latent semantic analysis, and latent Dirichlet allocation.

Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources

Central to the looming paradigm shift toward data-intensive science, machine-learning techniques are becoming increasingly important. In particular, deep learning has proven to be both a major breakthrough and an extremely powerful tool in many fields. Shall we embrace deep learning as the key to everything? Or should we resist a black-box solution? These are controversial issues within the remote-sensing community. In this article, we analyze the challenges of using deep learning for remote-sensing data analysis, review recent advances, and provide resources we hope will make deep learning in remote sensing seem ridiculously simple. More importantly, we encourage remote-sensing scientists to bring their expertise into deep learning and use it as an implicit general model to tackle unprecedented, large-scale, influential challenges, such as climate change and urbanization.

A hierarchical scheme of multiple feature fusion for high-resolution satellite scene categorization

Scene categorization in high-resolution satellite images has attracted much attention in recent years. However, high intra-class variations, illuminations and occlusions make the task very challenging. In this paper, we propose a classification model based on a hierarchical fusion of multiple features. Highlights of our work are threefold: (1) we use four discriminative image features; (2) we employ support vector machine with histogram intersection kernel (HIK-SVM) and L1-regularization logistic regression classifier (L1R-LRC) in different classification stages, respectively. The soft probabilities of different features obtained by the HIK-SVM are discriminatively fused and fed into the L1R-LRC to obtain the final results; (3) we conduct an extensive evaluation of different configurations, including different feature fusion schemes and different kernel functions. Experimental analysis show that our method leads to state-of-the-art classification performance on the satellite scenes.

Bag-of-Visual-Words Scene Classifier With Local and Global Features for High Spatial Resolution Remote Sensing Imagery

Scene classification has been studied to allow us to semantically interpret high spatial resolution (HSR) remote sensing imagery. The bag-of-visual-words (BOVW) model is an effective method for HSR image scene classification. However, the traditional BOVW model only captures the local patterns of images by utilizing local features. In this letter, a local-global feature bag-of-visual-words scene classifier (LGFBOVW) is proposed for HSR imagery. In LGFBOVW, the shape-based invariant texture index is designed as the global texture feature, the mean and standard deviation values are employed as the local spectral feature, and the dense scale-invariant feature transform (SIFT) feature is employed as the structural feature. The LGFBOVW can effectively combine the local and global features by an appropriate feature fusion strategy at histogram level. Experimental results on UC Merced and Google data sets of SIRI-WHU demonstrate that the proposed method outperforms the state-of-the-art scene classification methods for HSR imagery.