Jeffrey S. Perry

Edge co-occurrence in natural images predicts contour grouping performance.

The human brain manages to correctly interpret almost every visual image it receives from the environment. Underlying this ability are contour grouping mechanisms that appropriately link local edge elements into global contours. Although a general view of how the brain achieves effective contour grouping has emerged, there have been a number of different specific proposals and few successes at quantitatively predicting performance. These previous proposals have been developed largely by intuition and computational trial and error. A more principled approach is to begin with an examination of the statistical properties of contours that exist in natural images, because it is these statistics that drove the evolution of the grouping mechanisms. Here we report measurements of both absolute and Bayesian edge co-occurrence statistics in natural images, as well as human performance for detecting natural-shaped contours in complex backgrounds. We find that contour detection performance is quantitatively predicted by a local grouping rule derived directly from the co-occurrence statistics, in combination with a very simple integration rule (a transitivity rule) that links the locally grouped contour elements into longer contours.

Contour statistics in natural images: Grouping across occlusions

Correctly interpreting a natural image requires dealing properly with the effects of occlusion, and hence, contour grouping across occlusions is a major component of many natural visual tasks. To better understand the mechanisms of contour grouping across occlusions, we (a) measured the pair-wise statistics of edge elements from contours in natural images, as a function of edge element geometry and contrast polarity, (b) derived the ideal Bayesian observer for a contour occlusion task where the stimuli were extracted directly from natural images, and then (c) measured human performance in the same contour occlusion task. In addition to discovering new statistical properties of natural contours, we found that naïve human observers closely parallel ideal performance in our contour occlusion task. In fact, there was no region of the four-dimensional stimulus space (three geometry dimensions and one contrast dimension) where humans did not closely parallel the performance of the ideal observer (i.e., efficiency was approximately constant over the entire space). These results reject many other contour grouping hypotheses and strongly suggest that the neural mechanisms of contour grouping are tightly related to the statistical properties of contours in natural images.

Visual search: The role of peripheral information measured using gaze-contingent displays

Two of the factors limiting progress in understanding the mechanisms of visual search are the difficulty of controlling and manipulating the retinal stimulus when the eyes are free to move and the lack of an ideal observer theory for fixation selection during search. Recently, we developed a method to precisely control retinal stimulation with gaze-contingent displays (J. S. Perry & W. S. Geisler, 2002), and we derived a theory of optimal eye movements in visual search (J. Najemnik & W. S. Geisler, 2005). Here, we report a parametric study of visual search for sine-wave targets added to spatial noise backgrounds that have spectral characteristics similar to natural images (the amplitude spectrum of the noise falls inversely with spatial frequency). Search time, search accuracy, and eye fixations were measured as a function of target spatial frequency, 1/f noise contrast, and the resolution falloff of the display from the point of fixation. The results are systematic and similar for the two observers. We find that many aspects of search performance and eye movement pattern are similar to those of an ideal searcher that has the same falloff in resolution with retinal eccentricity as the human visual system.

Gaze-contingent real-time simulation of arbitrary visual fields

We describe an algorithm and software for creating variable resolution displays in real time, contingent upon the direction of gaze. The algorithm takes as input a video sequence and an arbitrary, real-valued, two-dimensional map that specifies a desired amount of filtering (blur) at each pixel location relative to direction of gaze. For each input video image the follow operations are performed: (1) the image is coded as a multi-resolution pyramid, (2) the gaze direction is measured, (3) the resolution map is shifted to the gaze direction, (4) the desired filtering at each pixel location is achieved by interpolating between levels of the pyramid using the resolution map, and (5) the interpolated image is displayed. The transfer function associated with each level of the pyramid is calibrated beforehand so that the interpolation produces exactly the desired amount of filtering at each pixel. This algorithm produces precision, artifact-free displays in 8-bit grayscale or 24-bit color. The software can process live or prerecorded video at over 60 frames per second on ordinary personal computers without special hardware. Direction of gaze for each processed video frame may be taken from an eye-tracker, from a sequence of directions saved on disk, or from another pointing device (such as a mouse). The software is demonstrated by simulating the visual fields of normals and of patients with low vision. We are currently using the software to precisely control retinal stimulation during complex tasks such as extended visual search.

Real-time simulation of arbitrary visual fields

This report describes an algorithm and software for creating and displaying, in real time, arbitrary variable resolution displays, contingent on the direction of gaze. The software produces precise, artifact-free video at high frame rates in either 8-bit gray scale or 24-bit color. The software is demonstrated by simulating the visual fields of normal individuals and low-vision patients.

Variable-Resolution Displays for Visual Communication and Simulation

The spatial resolution of the human visual system decreases as a function of angular distance from the direction of gaze. This fact can be exploited in various applications to increase image compression, to increase image processing speed, and to decrease access time for image data. This paper describes a multiresolution pyramid method for creating variable resolution displays in real time using general purpose computers. The location of the high resolution region(s) can be dynamically controlled by the user with a pointing device (e.g., a mouse or an eye tracker) or by an algorithm. Our method has a number of advantages: high computational speed and efficiency, smooth artifact-free variable resolution, and compatibility with other image processing software/hardware. Applications to video communications (MPEG) and graphic simulation are described.

Humans make efficient use of natural image statistics when performing spatial interpolation.

Visual systems learn through evolution and experience over the lifespan to exploit the statistical structure of natural images when performing visual tasks. Understanding which aspects of this statistical structure are incorporated into the human nervous system is a fundamental goal in vision science. To address this goal, we measured human ability to estimate the intensity of missing image pixels in natural images. Human estimation accuracy is compared with various simple heuristics (e.g., local mean) and with optimal observers that have nearly complete knowledge of the local statistical structure of natural images. Human estimates are more accurate than those of simple heuristics, and they match the performance of an optimal observer that knows the local statistical structure of relative intensities (contrasts). This optimal observer predicts the detailed pattern of human estimation errors and hence the results place strong constraints on the underlying neural mechanisms. However, humans do not reach the performance of an optimal observer that knows the local statistical structure of the absolute intensities, which reflect both local relative intensities and local mean intensity. As predicted from a statistical analysis of natural images, human estimation accuracy is negligibly improved by expanding the context from a local patch to the whole image. Our results demonstrate that the human visual system exploits efficiently the statistical structure of natural images.

Natural systems analysis

The environments we live in and the tasks we perform in those environments have shaped the design of our visual systems through evolution and experience. This is an obvious statement, but it implies three fundamental components of research we must have if we are going to gain a deep understanding of biological vision systems: (a) a rigorous science devoted to understanding natural environments and tasks, (b) mathematical and computational analysis of how to use such knowledge of the environment to perform natural tasks, and (c) experiments that allow rigorous measurement of behavioral and neural responses, either in natural tasks or in artificial tasks that capture the essence of natural tasks. This approach is illustrated with two example studies that combine measurements of natural scene statistics, derivation of Bayesian ideal observers that exploit those statistics, and psychophysical experiments that compare human and ideal performance in naturalistic tasks.

High-Order Statistics for Point Prediction in Natural Images

Results are presented for a simple conditional-moments method that directly measures high-order statistics of natural images. In four estimation tasks significant increases in performance are obtained in comparison to traditional methods.