Christoph Zetzsche

Fundamental limits of linear filters in the visual processing of two-dimensional signals

Since Lettvin, Maturana, McCulloch and Pitts (1959), neurophysiologists have known that the visual system contains detectors that respond to stimulus features such as “bugs”, line ends, bars, comers, etc. (Hubel & Wiesel, 1965). During the same period, the theory of linear systems has been applied successfully to the analysis and modelling of visual functions (DeValois & DeValois, 1980). Interestingly, however, there is a fundamental incompatibility between these two approaches that has not yet received adequate attention. Linear filtering, even if modified by common nonlinear&& like thresholding or rectification, will generally confound straight signals with signals that show essentially two-dimensional variations. This principle deficiency is illustrated for a curvature detector recently suggested by Dobbins, Zucker and Cynader (1987, 1989), which is based on a nonlinear combination of linear filters. However, the problem can be solved by using the mathematical formalism of differential geometry. We employ the concept of “Gaussian curvature” of surfaces to derive a class of physiologically plausible operators for the detection of two-dimensional signal variations. Two essential properties of these detectors turn out to be necessary: the use of “and” operations, that are impossible with linear filters, and a specific “compensation principle” corresponding to inhibitory interactions between orientation selective filters. One example for the encoding of essentially two-dimensional signal variations is the detection of curvature. According to a recent hypothesis by Dobbins et al. (1987, 1989), this can be accomplished by using the difference between the outputs of two simple cells with different receptive field sizes to generate “endstopped” responses that proportionally vary to stimulus length and curvature. It can be shown, however, that this particular model, as well as any essentially linear system, is subject to response ambiguities in that it is always possible to find a stimulus of zero curvature that erroneously elicits a response. Consider the stimulus configuration shown in Fig. la. While the curved lines and short bars give rise to appropriate responses of the Dobbins et al. detector, it also reacts erroneously to certain straight stimuli on the right side (Fig. lb). The corresponding critical spectral area within which such false responses can occur is indicated in Fig. 2. The very reason for the occurrence of such ambiguous responses has to be sought in a fundamental limitation of linear filters in the processing of two-dimensional signals. Such signals can be classified into three elementary categories: (1) constant signals that show no variation at all; (2) intrinsically onedimensional signals that are constant along one orientation and can, therefore, be completely characterized by their variation along the orthogonal orientation (here: lD-signals); (3) actually two-dimensional signals that vary along all orientations (here: ZD-signals). Obvious examples of 1-D signals are straight lines, straight edges, or sinusoidal gratings with arbitrary orientation. Curved lines, curved edges and junctions, intersections, terminations, etc. are typical 2D-signals (Marko, 1974; Julesz, 1981). An essential requirement for all detectors which encode ZD-signal properties is that they should not erroneously respond to ID-signals. Curvature detectors, for example, should not respond to straight stimuli. We will show that such an unambiguous detection of ZD-signal properties necessarily employs “and” operations. Such

Object and scene analysis by saccadic eye-movements : an investigation with higher-order statistics

Based on an information theoretical approach, we investigate feature selection processes in saccadic object and scene analysis. Saccadic eye movements of human observers are recorded for a variety of natural and artificial test images. These experimental data are used for a statistical evaluation of the fixated image regions. Analysis of second-order statistics indicates that regions with higher spatial variance have a higher probability to be fixated, but no significant differences beyond these variance effects could be found at the level of power spectra. By contrast, an investigation with higher-order statistics, as reflected in the bispectral density, yielded clear structural differences between the image regions selected by saccadic eye movements as opposed to regions selected by a random process. These results indicate that nonredundant, intrinsically two-dimensional image features like curved lines and edges, occlusions, isolated spots, etc. play an important role in the saccadic selection process which must be integrated with top-down knowledge to fully predict object and scene analysis by human observers.

Low-Level Integration of Auditory and Visual Motion Signals Requires Spatial Co-Localisation

It is well known that the detection thresholds for stationary auditory and visual signals are lower if the signals are presented bimodally rather than unimodally, provided the signals coincide in time and space. Recent work on auditory–visual motion detection suggests that the facilitation seen for stationary signals is not seen for motion signals. We investigate the conditions under which motion perception also benefits from the integration of auditory and visual signals. We show that the integration of cross-modal local motion signals that are matched in position and speed is consistent with thresholds predicted by a neural summation model. If the signals are presented in different hemi-fields, move in different directions, or both, then behavioural thresholds are predicted by a probability-summation model. We conclude that cross-modal signals have to be co-localised and co-incident for effective motion integration. We also argue that facilitation is only seen if the signals contain all localisation cues that would be produced by physical objects.

Scene analysis with saccadic eye movements: top-down and bottom-up modeling

The perception of an image by a human observer is usually modeled as a parallel process in which all parts of the image are treated more or less equivalently, but in reality the analysis of scenes is a highly selective procedure, in which only a small subset of image locations is processed by the precise and efficient neural machinery of foveal vision. To understand the principles behind this selection of the ‘‘informative’’ regions of images, we have developed a hybrid system that consists of a combination of a knowledgebased reasoning system with a low-level preprocessing by linear and nonlinear neural operators. This hybrid system is intended as a first step towards a complete model of the sensorimotor system of saccadic scene analysis. In the analysis of a scene, the system calculates in each step which eye movement has to be made to reach a maximum of information about the scene. The possible information gain is calculated by means of a parallel strategy which is suitable for adaptive reasoning. The output of the system is a fixation sequence, and finally, a hypothesis about the scene.

The atoms of vision: Cartesian or polar?

The inherent structure of the encoding in early stages of the visual system is investigated from a combined information-theoretical, psychophysical, and neurophysiological perspective. We argue that the classical modeling in terms of linear spatial filters is equivalent to the assumption of a Cartesian organization of the feature space of early vision. We show that such a linear Cartesian feature space would be suboptimal for the exploitation of the statistical redundancies of natural images since these have a radially separable probability-density function. Therefore a more efficient representation can be obtained by a nonlinear encoding that yields a feature space with polar organization. This prediction of the information-theoretical approach regarding the organization of the feature space of early vision is confirmed by our psychophysical measurements of basic discrimination capabilities for elementary Gabor patches, and the necessary nonlinear operations are shown to be closely related to cortical gain control and to the phase invariance of complex cells. Finally, we point out some striking similarities between the polar representation in visual cortex and basic image-coding strategies pursued in shape-gain vector quantization schemes.

Statistical dependence between orientation filter outputs used in a human-vision-based image code

We present an image coding scheme based on the properties of the early stages of the human visual system. The image signal is decomposed via even and odd symmetric frequency and orientation selective bandpass filters in analogy to the quadrature phase simple cell pairs in the visual cortex. The resulting analytic signal is transformed into a local amplitude and local phase representation in order to achieve a better match to its signal statistics. Both intra filter dependencies of the analytic signal and inter filter dependencies between different orientation filters are exploited by a suitable vector quantization scheme. Inter orientation filter dependencies are demonstrated by means of a statistical evaluation of the multidimensional probability density function. The results can be seen as an empirical confirmation of the suitability of vector quantization in subband coding. Instead of generating a code book by use of an conventional designalgorithm we suggest a feature specific partitioning of the multidimensional signal space matched to the properties of human vision. Using this coding scheme satisfactory image quality can be obtained with about 0. 78 bit/pixel.

Intrinsic two-dimensional features as textons

We suggest that intrinsic two-dimensional (i2D) features, computationally defined as the outputs of nonlinear operators that model the activity of end-stopped neurons, play a role in preattentive texture discrimination. We first show that for discriminable textures with identical power spectra the predictions of traditional models depend on the type of nonlinearity and fail for energy measures. We then argue that the concept of intrinsic dimensionality, and the existence of end-stopped neurons, can help us to understand the role of the nonlinearities. Furthermore, we show examples in which models without strong i2D selectivity fail to predict the correct ranking order of perceptual segregation. Our arguments regarding the importance of i2D features resemble the arguments of Julesz and co-workers regarding textons such as terminators and crossings. However, we provide a computational framework that identifies textons with the outputs of nonlinear operators that are selective to i2D features.

Nonlinear and higher-order approaches to the encoding of natural scenes

Linear operations can only partially exploit the statistical redundancies of natural scenes, and nonlinear operations are ubiquitous in visual cortex. However, neither the detailed function of the nonlinearities nor the higher-order image statistics are yet fully understood. We suggest that these complicated issues can not be tackled by one single approach, but require a range of methods, and the understanding of the crosslinks between the results. We consider three basic approaches: (i) State space descriptions can theoretically provide complete information about statistical properties and nonlinear operations, but their practical usage is confined to very low-dimensional settings. We discuss the use of representation-related state-space coordinates (multivariate wavelet statistics) and of basic nonlinear coordinate transformations of the state space (e.g., a polar transform). (ii) Indirect methods, like unsupervised learning in multi-layer networks, provide complete optimization results, but no direct information on the statistical properties, and no simple model structures. (iii) Approximation by lower-order terms of power-series expansions is a classical strategy that has not yet received broad attention. On the statistical side, this approximation amounts to cumulant functions and higher-order spectra (polyspectra), on the processing side to Volterra–Wiener systems. In this context we suggest that an important concept for the understanding of natural scene statistics, of nonlinear neurons, and of biological pattern recognition can be found in AND-like combinations of frequency components. We investigate how the different approaches can be related to each other, how they can contribute to the understanding of cortical nonlinearities such as complex cells, cortical gain control, end-stopping and other extraclassical receptive field properties, and how we can obtain a nonlinear perspective on overcomplete representations and invariant coding in visual cortex.

Higher-order statistics of natural images and their exploitation by operators selective to intrinsic dimensionality

Natural images contain considerable statistical redundancies beyond the level of second-order correlations. To identify the nature of these higher-order dependencies, we analyze the bispectra and trispectra of natural images. Our investigations reveal substantial statistical dependencies between those frequency components which are aligned to each other with respect to orientation. We argue that operators which are selective to local intrinsic dimensionality can optimally exploit such redundancies. We also show that the polyspectral structure we find for natural images helps to understand the hitherto unexplained superiority of orientation-selective filter decompositions over isotropic schemes like the Laplacian pyramid. However any essentially linear scheme can only partially exploit this higher-order redundancy. We therefore propose nonlinear i2D-selective operators which exhibit close resemblance to hypercomplex and end-stopped cells in the visual cortex. The function of these operators can be interpreted as a higher-order whitening of the input signal.

Direct detection of flow discontinuities by 3D curvature operators

Abstract We present an approach for the direct detection of flow discontinuities which avoids explicit computation of a dense optic flow field. It is based on regarding the time varying image as a hypersurface in four-dimensional space and on using the Gaussian curvature properties of this hypersurface as a direct indicator for the presence of motion discontinuities. An easy to implement, nonlinear operator is suggested and possible extensions of the basic scheme are discussed.