Gustau Camps-Valls

Unsupervised Deep Feature Extraction for Remote Sensing Image Classification

This paper introduces the use of single-layer and deep convolutional networks for remote sensing data analysis. Direct application to multi- and hyperspectral imagery of supervised (shallow or deep) convolutional networks is very challenging given the high input data dimensionality and the relatively small amount of available labeled data. Therefore, we propose the use of greedy layerwise unsupervised pretraining coupled with a highly efficient algorithm for unsupervised learning of sparse features. The algorithm is rooted on sparse representations and enforces both population and lifetime sparsity of the extracted features, simultaneously. We successfully illustrate the expressive power of the extracted representations in several scenarios: classification of aerial scenes, as well as land-use classification in very high resolution or land-cover classification from multi- and hyperspectral images. The proposed algorithm clearly outperforms standard principal component analysis (PCA) and its kernel counterpart (kPCA), as well as current state-of-the-art algorithms of aerial classification, while being extremely computationally efficient at learning representations of data. Results show that single-layer convolutional networks can extract powerful discriminative features only when the receptive field accounts for neighboring pixels and are preferred when the classification requires high resolution and detailed results. However, deep architectures significantly outperform single-layer variants, capturing increasing levels of abstraction and complexity throughout the feature hierarchy.

Compensatory water effects link yearly global land CO2 sink changes to temperature.

Large interannual variations in the measured growth rate of atmospheric carbon dioxide (CO2) originate primarily from fluctuations in carbon uptake by land ecosystems. It remains uncertain, however, to what extent temperature and water availability control the carbon balance of land ecosystems across spatial and temporal scales. Here we use empirical models based on eddy covariance data and process-based models to investigate the effect of changes in temperature and water availability on gross primary productivity (GPP), terrestrial ecosystem respiration (TER) and net ecosystem exchange (NEE) at local and global scales. We find that water availability is the dominant driver of the local interannual variability in GPP and TER. To a lesser extent this is true also for NEE at the local scale, but when integrated globally, temporal NEE variability is mostly driven by temperature fluctuations. We suggest that this apparent paradox can be explained by two compensatory water effects. Temporal water-driven GPP and TER variations compensate locally, dampening water-driven NEE variability. Spatial water availability anomalies also compensate, leaving a dominant temperature signal in the year-to-year fluctuations of the land carbon sink. These findings help to reconcile seemingly contradictory reports regarding the importance of temperature and water in controlling the interannual variability of the terrestrial carbon balance. Our study indicates that spatial climate covariation drives the global carbon cycle response.

Multimodal Classification of Remote Sensing Images: A Review and Future Directions

Earth observation through remote sensing images allows the accurate characterization and identification of materials on the surface from space and airborne platforms. Multiple and heterogeneous image sources can be available for the same geographical region: multispectral, hyperspectral, radar, multitemporal, and multiangular images can today be acquired over a given scene. These sources can be combined/fused to improve classification of the materials on the surface. Even if this type of systems is generally accurate, the field is about to face new challenges: the upcoming constellations of satellite sensors will acquire large amounts of images of different spatial, spectral, angular, and temporal resolutions. In this scenario, multimodal image fusion stands out as the appropriate framework to address these problems. In this paper, we provide a taxonomical view of the field and review the current methodologies for multimodal classification of remote sensing images. We also highlight the most recent advances, which exploit synergies with machine learning and signal processing: sparse methods, kernel-based fusion, Markov modeling, and manifold alignment. Then, we illustrate the different approaches in seven challenging remote sensing applications: 1) multiresolution fusion for multispectral image classification; 2) image downscaling as a form of multitemporal image fusion and multidimensional interpolation among sensors of different spatial, spectral, and temporal resolutions; 3) multiangular image classification; 4) multisensor image fusion exploiting physically-based feature extractions; 5) multitemporal image classification of land covers in incomplete, inconsistent, and vague image sources; 6) spatiospectral multisensor fusion of optical and radar images for change detection; and 7) cross-sensor adaptation of classifiers. The adoption of these techniques in operational settings will help to monitor our planet from space in the very near future.

Kernel Manifold Alignment for Domain Adaptation

The wealth of sensory data coming from different modalities has opened numerous opportunities for data analysis. The data are of increasing volume, complexity and dimensionality, thus calling for new methodological innovations towards multimodal data processing. However, multimodal architectures must rely on models able to adapt to changes in the data distribution. Differences in the density functions can be due to changes in acquisition conditions (pose, illumination), sensors characteristics (number of channels, resolution) or different views (e.g. street level vs. aerial views of a same building). We call these different acquisition modes domains, and refer to the adaptation problem as domain adaptation. In this paper, instead of adapting the trained models themselves, we alternatively focus on finding mappings of the data sources into a common, semantically meaningful, representation domain. This field of manifold alignment extends traditional techniques in statistics such as canonical correlation analysis (CCA) to deal with nonlinear adaptation and possibly non-corresponding data pairs between the domains. We introduce a kernel method for manifold alignment (KEMA) that can match an arbitrary number of data sources without needing corresponding pairs, just few labeled examples in all domains. KEMA has interesting properties: 1) it generalizes other manifold alignment methods, 2) it can align manifolds of very different complexities, performing a discriminative alignment preserving each manifold inner structure, 3) it can define a domain-specific metric to cope with multimodal specificities, 4) it can align data spaces of different dimensionality, 5) it is robust to strong nonlinear feature deformations, and 6) it is closed-form invertible, which allows transfer across-domains and data synthesis. To authors’ knowledge this is the first method addressing all these important issues at once. We also present a reduced-rank version of KEMA for computational efficiency, and discuss the generalization performance of KEMA under Rademacher principles of stability. Aligning multimodal data with KEMA reports outstanding benefits when used as a data pre-conditioner step in the standard data analysis processing chain. KEMA exhibits very good performance over competing methods in synthetic controlled examples, visual object recognition and recognition of facial expressions tasks. KEMA is especially well-suited to deal with high-dimensional problems, such as images and videos, and under complicated distortions, twists and warpings of the data manifolds. A fully functional toolbox is available at https://github.com/dtuia/KEMA.git.

Exploitation of SAR and Optical Sentinel Data to Detect Rice Crop and Estimate Seasonal Dynamics of Leaf Area Index

This paper presents and evaluates multitemporal LAI estimates derived from Sentinel-2A data on rice cultivated area identified using time series of Sentinel-1A images over the main European rice districts for the 2016 crop season. This study combines the information conveyed by Sentinel-1A and Sentinel-2A into a high-resolution LAI retrieval chain. Rice crop was detected using an operational multi-temporal rule-based algorithm, and LAI estimates were obtained by inverting the PROSAIL radiative transfer model with Gaussian process regression. Direct validation was performed with in situ LAI measurements acquired in coordinated field campaigns in three countries (Italy, Spain and Greece). Results showed high consistency between estimates and ground measurements, revealing high correlations (R2 > 0.93) and good accuracies (RMSE < 0.83, rRMSEm < 23.6% and rRMSEr < 16.6%) in all cases. Sentinel-2A estimates were compared with Landsat-8 showing high spatial consistency between estimates over the three areas. The possibility to exploit seasonally-updated crop mask exploiting Sentinel-1A data and the temporal consistency between Sentinel-2A and Landsat-7/8 LAI time series demonstrates the feasibility of deriving operationally high spatial-temporal decametric multi-sensor LAI time series useful for crop monitoring.

An Emulator Toolbox to Approximate Radiative Transfer Models with Statistical Learning

Physically-based radiative transfer models (RTMs) help in understanding the processes occurring on the Earth’s surface and their interactions with vegetation and atmosphere. When it comes to studying vegetation properties, RTMs allows us to study light interception by plant canopies and are used in the retrieval of biophysical variables through model inversion. However, advanced RTMs can take a long computational time, which makes them unfeasible in many real applications. To overcome this problem, it has been proposed to substitute RTMs through so-called emulators. Emulators are statistical models that approximate the functioning of RTMs. Emulators are advantageous in real practice because of the computational efficiency and excellent accuracy and flexibility for extrapolation. We hereby present an “Emulator toolbox” that enables analysing multi-output machine learning regression algorithms (MO-MLRAs) on their ability to approximate an RTM. The toolbox is included in the free-access ARTMO’s MATLAB suite for parameter retrieval and model inversion and currently contains both linear and non-linear MO-MLRAs, namely partial least squares regression (PLSR), kernel ridge regression (KRR) and neural networks (NN). These MO-MLRAs have been evaluated on their precision and speed to approximate the soil vegetation atmosphere transfer model SCOPE (Soil Canopy Observation, Photochemistry and Energy balance). SCOPE generates, amongst others, sun-induced chlorophyll fluorescence as the output signal. KRR and NN were evaluated as capable of reconstructing fluorescence spectra with great precision. Relative errors fell below 0.5% when trained with 500 or more samples using cross-validation and principal component analysis to alleviate the underdetermination problem. Moreover, NN reconstructed fluorescence spectra about 50-times faster and KRR about 800-times faster than SCOPE. The Emulator toolbox is foreseen to open new opportunities in the use of advanced RTMs, in which both consistent physical assumptions and data-driven machine learning algorithms live together.

Mapping Leaf Area Index With a Smartphone and Gaussian Processes

Leaf area index (LAI) is a key biophysical parameter used to determine foliage cover and crop growth in environmental studies. Smartphones are nowadays ubiquitous sensor devices with high computational power, moderate cost, and high-quality sensors. A smartphone app, which is called PocketLAI, was recently presented and tested for acquiring ground LAI estimates. In this letter, we explore the use of state-of-the-art nonlinear Gaussian process regression (GPR) to derive spatially explicit LAI estimates over rice using ground data from PocketLAI and Landsat 8 imagery. GPR has gained popularity in recent years because of its solid Bayesian foundations that offer not only high accuracy but also confidence intervals for the retrievals. We show the first LAI maps obtained with ground data from a smartphone combined with advanced machine learning. This letter compares LAI predictions and confidence intervals of the retrievals obtained with PocketLAI with those obtained with classical instruments, such as digital hemispheric photography (DHP) and LI-COR LAI-2000. This letter shows that all three instruments obtained comparable results, but PocketLAI is far cheaper. The proposed methodology hence opens a wide range of possible applications at moderate cost.

A Survey on Gaussian Processes for Earth-Observation Data Analysis: A Comprehensive Investigation

Gaussian processes (GPs) have experienced tremendous success in biogeophysical parameter retrieval in the last few years. GPs constitute a solid Bayesian framework to consistently formulate many function approximation problems. This article reviews the main theoretical GP developments in the field, considering new algorithms that respect signal and noise characteristics, extract knowledge via automatic relevance kernels to yield feature rankings automatically, and allow applicability of associated uncertainty intervals to transport GP models in space and time that can be used to uncover causal relations between variables and can encode physically meaningful prior knowledge via radiative transfer model (RTM) emulation. The important issue of computational efficiency will also be addressed. These developments are illustrated in the field of geosciences and remote sensing at local and global scales through a set of illustrative examples. In particular, important problems for land, ocean, and atmosphere monitoring are considered, from accurately estimating oceanic chlorophyll content and pigments to retrieving vegetation properties from multi- and hyperspectral sensors as well as estimating atmospheric parameters (e.g., temperature, moisture, and ozone) from infrared sounders.

Emulation of leaf, canopy and atmosphere radiative transfer models for fast global sensitivity analysis

Physically-based radiative transfer models (RTMs) help understand the interactions of radiation with vegetation and atmosphere. However, advanced RTMs can be computationally burdensome, which makes them impractical in many real applications, especially when many state conditions and model couplings need to be studied. To overcome this problem, it is proposed to substitute RTMs through surrogate meta-models also named emulators. Emulators approximate the functioning of RTMs through statistical learning regression methods, and can open many new applications because of their computational efficiency and outstanding accuracy. Emulators allow fast global sensitivity analysis (GSA) studies on advanced, computationally expensive RTMs. As a proof-of-concept, three machine learning regression algorithms (MLRAs) were tested to function as emulators for the leaf RTM PROSPECT-4, the canopy RTM PROSAIL, and the computationally expensive atmospheric RTM MODTRAN5. Selected MLRAs were: kernel ridge regression (KRR), neural networks (NN) and Gaussian processes regression (GPR). For each RTM, 500 simulations were generated for training and validation. The majority of MLRAs were excellently validated to function as emulators with relative errors well below 0.2%. The emulators were then put into a GSA scheme and compared against GSA results as generated by original PROSPECT-4 and PROSAIL runs. NN and GPR emulators delivered identical GSA results, while processing speed compared to the original RTMs doubled for PROSPECT-4 and tripled for PROSAIL. Having the emulator-GSA concept successfully tested, for six MODTRAN5 atmospheric transfer functions (outputs), i.e., direct and diffuse at-surface solar irradiance ( E d i f , E d i r ), direct and diffuse upward transmittance ( T d i r , T d i f ), spherical albedo (S) and path radiance ( L 0 ), the most accurate MLRA’s were subsequently applied as emulator into the GSA scheme. The sensitivity analysis along the 400–2500 nm spectral range took no more than a few minutes on a contemporary computer—in comparison, the same analysis in the original MODTRAN5 would have taken over a month. Key atmospheric drivers were identified, which are on the one hand aerosol optical properties, i.e., aerosol optical thickness (AOT), Angstrom coefficient (AMS) and scattering asymmetry variable (G), mostly driving diffuse atmospheric components, E d i f and T d i f ; and those affected by atmospheric scattering, L 0 and S. On the other hand, as expected, AOT, AMS and columnar water vapor (CWV) in the absorption regions mostly drive E d i r and T d i r atmospheric functions. The presented emulation schemes showed very promising results in replacing costly RTMs, and we think they can contribute to the adoption of machine learning techniques in remote sensing and environmental applications.

Principal polynomial analysis.

This paper presents a new framework for manifold learning based on a sequence of principal polynomials that capture the possibly nonlinear nature of the data. The proposed Principal Polynomial Analysis (PPA) generalizes PCA by modeling the directions of maximal variance by means of curves, instead of straight lines. Contrarily to previous approaches, PPA reduces to performing simple univariate regressions, which makes it computationally feasible and robust. Moreover, PPA shows a number of interesting analytical properties. First, PPA is a volume-preserving map, which in turn guarantees the existence of the inverse. Second, such an inverse can be obtained in closed form. Invertibility is an important advantage over other learning methods, because it permits to understand the identified features in the input domain where the data has physical meaning. Moreover, it allows to evaluate the performance of dimensionality reduction in sensible (input-domain) units. Volume preservation also allows an easy computation of information theoretic quantities, such as the reduction in multi-information after the transform. Third, the analytical nature of PPA leads to a clear geometrical interpretation of the manifold: it allows the computation of Frenet-Serret frames (local features) and of generalized curvatures at any point of the space. And fourth, the analytical Jacobian allows the computation of the metric induced by the data, thus generalizing the Mahalanobis distance. These properties are demonstrated theoretically and illustrated experimentally. The performance of PPA is evaluated in dimensionality and redundancy reduction, in both synthetic and real datasets from the UCI repository.