Patrik O. Hoyer

Non-negative sparse coding

Non-negative sparse coding is a method for decomposing multivariate data into non-negative sparse components. We briefly describe the motivation behind this type of data representation and its relation to standard sparse coding and non-negative matrix factorization. We then give a simple yet efficient multiplicative algorithm for finding the optimal values of the hidden components. In addition, we show how the basis vectors can be learned from the observed data. Simulations demonstrate the effectiveness of the proposed method.

Emergence of Phase- and Shift-Invariant Features by Decomposition of Natural Images into Independent Feature Subspaces

Olshausen and Field (1996) applied the principle of independence maximization by sparse coding to extract features from natural images. This leads to the emergence of oriented linear filters that have simultaneous localization in space and in frequency, thus resembling Gabor functions and simple cell receptive fields. In this article, we show that the same principle of independence maximization can explain the emergence of phase- and shift-invariant features, similar to those found in complex cells. This new kind of emergence is obtained by maximizing the independence between norms of projections on linear subspaces (instead of the independence of simple linear filter outputs). The norms of the projections on such independent feature subspaces then indicate the values of invariant features.

Topographic Independent Component Analysis

In ordinary independent component analysis, the components are assumed to be completely independent, and they do not necessarily have any meaningful order relationships. In practice, however, the estimated independent components are often not at all independent. We propose that this residual dependence structure could be used to define a topo-graphic order for the components. In particular, a distance between two components could be defined using their higher-order correlations, and this distance could be used to create a topographic representation. Thus, we obtain a linear decomposition into approximately independent components, where the dependence of two components is approximated by the proximity of the components in the topographic representation.

A two-layer sparse coding model learns simple and complex cell receptive fields and topography from natural images

The classical receptive fields of simple cells in the visual cortex have been shown to emerge from the statistical properties of natural images by forcing the cell responses to be maximally sparse, i.e. significantly activated only rarely. Here, we show that this single principle of sparseness can also lead to emergence of topography (columnar organization) and complex cell properties as well. These are obtained by maximizing the sparsenesses of locally pooled energies, which correspond to complex cell outputs. Thus, we obtain a highly parsimonious model of how these properties of the visual cortex are adapted to the characteristics of the natural input.

Independent component analysis applied to feature extraction from colour and stereo images.

Previous work has shown that independent component analysis (ICA) applied to feature extraction from natural image data yields features resembling Gabor functions and simple-cell receptive fields. This article considers the effects of including chromatic and stereo information. The inclusion of colour leads to features divided into separate red/green, blue/yellow, and bright/dark channels. Stereo image data, on the other hand, leads to binocular receptive fields which are tuned to various disparities. The similarities between these results and the observed properties of simple cells in the primary visual cortex are further evidence for the hypothesis that visual cortical neurons perform some type of redundancy reduction, which was one of the original motivations for ICA in the first place. In addition, ICA provides a principled method for feature extraction from colour and stereo images; such features could be used in image processing operations such as denoising and compression, as well as in pattern recognition.

A multi-layer sparse coding network learns contour coding from natural images

An important approach in visual neuroscience considers how the function of the early visual system relates to the statistics of its natural input. Previous studies have shown how many basic properties of the primary visual cortex, such as the receptive fields of simple and complex cells and the spatial organization (topography) of the cells, can be understood as efficient coding of natural images. Here we extend the framework by considering how the responses of complex cells could be sparsely represented by a higher-order neural layer. This leads to contour coding and end-stopped receptive fields. In addition, contour integration could be interpreted as top-down inference in the presented model.

Image feature extraction by sparse coding and independent component analysis

Sparse coding is a method for finding a representation of data in which each of the components of the representation is only rarely significantly active. Such a representation is closely related to the techniques of independent component analysis and blind source separation. In this paper, we investigate the application of sparse coding for image feature extraction. We show how sparse coding can be used to extract wavelet-like features from natural image data. As an application of such a feature extraction scheme, we show how to apply a soft-thresholding operator on the components of sparse coding in order to reduce Gaussian noise. Methods based on sparse coding have the important benefit over wavelet methods that the features are determined solely by the statistical properties of the data, while the wavelet transformation relies heavily on certain abstract mathematical properties that may be only weakly related to the properties of the natural data.

Equivalence of Some Common Linear Feature Extraction Techniques for Appearance-Based Object Recognition Tasks

Recently, a number of empirical studies have compared the performance of PCA and ICA as feature extraction methods in appearance-based object recognition systems, with mixed and seemingly contradictory results. In this paper, we briefly describe the connection between the two methods and argue that whitened PCA may yield identical results to ICA in some cases. Furthermore, we describe the specific situations in which ICA might significantly improve on PCA

Sparse code shrinkage for image denoising

Sparse coding is a method for finding a representation of data in which each of the components of the representation is only rarely significantly active. Such a representation is closely related to redundancy reduction and independent component analysis, and has some neurophysiological plausibility. We show how sparse coding can be used for denoising. Using methods reminiscent of wavelet theory, we show how to apply a soft-thresholding operator on the components of sparse coding in order to reduce Gaussian noise. Our method has the important benefit over wavelet methods that the transformation is determined solely by the statistical properties of the data. The wavelet transformation, on the other hand, relies heavily on certain abstract mathematical properties that may be only weakly related to the properties of the natural data. Experiments on image data are reported.

Topographic independent component analysis as a model of V1 organization and receptive fields

Abstract Independent component analysis (ICA) has been recently used as a model of natural image statistics and V1 simple cell receptive fields. Here we show how to extend the ICA model to explain V1 topography as well. This is done by relaxing the independence assumption and ordering the basis vectors so that vectors with strong higher-order correlations are near each other. This is a new principle of topographic organization, and may be more relevant to natural image statistics than the more conventional topographic ordering based on Euclidean distances. For example, our ordering leads to simultaneous emergence of complex cell properties: topographic neighborhoods act like complex cells.