Ellen C. Hildreth

Theory of Edge Detection

A theory of edge detection is presented. The analysis proceeds in two parts. (1) Intensity changes, which occur in a natural image over a wide range of scales, are detected separately at different scales. An appropriate filter for this purpose at a given scale is found to be the second derivative of a Gaussian, and it is shown that, provided some simple conditions are satisfied, these primary filters need not be orientation-dependent. Thus, intensity changes at a given scale are best detected by finding the zero values of ∇2G(x, y)* I(x, y) for image I, where G(x, y) is a two-dimensional Gaussian distribution and ∇2 is the Laplacian. The intensity changes thus discovered in each of the channels are then represented by oriented primitives called zero-crossing segments, and evidence is given that this representation is complete. (2) Intensity changes in images arise from surface discontinuities or from reflectance or illumination boundaries, and these all have the property that they are spatially localized. Because of this, the zero-crossing segments from the different channels are not independent, and rules are deduced for combining them into a description of the image. This description is called the raw primal sketch. The theory explains several basic psychophysical findings, and the operation of forming oriented zero-crossing segments from the output of centre-surround ∇2G filters acting on the image forms the basis for a physiological model of simple cells (see Marr & Ullman 1979).

The Measurement of Visual Motion

Abstract Visual motion provides useful information about the surrounding environment, which biological visual systems have evolved to extract and utilize. The first problem in analysing visual motion is the measurement of motion; this has proved to be surprisingly difficult. The human visual system appears to solve it efficiently using a combination of at least two different methods. These methods are discussed, together with some unsolved problems and their possible implications for neurophysiological studies.

Comutations underlying the measuremnt of visual motion.

Abstract The organization of movement in a changing image provides a valuable source of information for analyzing the environment in terms of objects, their motion in space, and their three-dimensional structure. A description of this movement is not provided to our visual system directly, however; it must be inferred from the pattern of changing intensity that reaches the eye. This paper examines the problem of motion measurement, which we formulate as the computation of an instantaneous two-dimensional velocity field. Initial motion measurements occur at the location of significant intensity changes, as suggested by Marr and Ullman [1]. These measurements provide only one component of velocity, and must be integrated to compute the two-dimensional velocity field. A fundamental problem for this integration is that motion is not determined uniquely from the changing image. An additional constraint of smoothness of the velocity field is formulated, based on the physical assumption that surfaces are generally smooth, allowing the computation of a unique solution. A theoretical analysis of the conditions under which this computation yields the correct velocity field suggests that the solution is physically plausible; empirical studies show the results to be consistent with human motion perception.

The detection of intensity changes by computer and biological vision systems

Abstract This article describes the implementation of a theory for the detection of intensity changes, proposed by Marr and Hildreth (Proc. R. Soc. London, Ser. B 207, 1980, 187–217). According to this theory, the image is first processed independently through a set of different size operators, whose shape is the Laplacian of a Gaussian, ▿2G(x, y). The loci, along which the convolution outputs cross zero mark the positions of intensity changes at different resolutions. These zero-crossings can be described by their position, slope of the convolution output across zero, and two-dimensional orientation. The set of descriptions from different operator sizes forms the input for later visual processes, such as stereopsis and motion analysis. There are close parallels between this theory and the early processing of information by the human visual system.

The Computation of the Velocity Field

The organization of movement in the changing retinal image provides a valuable source of information for analysing the environment in terms of objects, their motion in space, and their three-dimensional structure. A description of this movement is not provided to our visual system directly, however; it must be inferred from the pattern of changing intensity that reaches the eye. This paper examines the problem of motion measurement, which we formulate as the computation of an instantaneous two-dimensional velocity field from the changing image. Initial measurements of motion take place at the location of significant intensity changes. These measurements provide only one component of local velocity, and must be integrated to compute the two-dimensional velocity field. A fundamental problem for this integration stage is that the velocity field is not determined uniquely from information available in the changing image. We formulate an additional constraint of smoothness of the velocity field, based on the physical assumption that surfaces are generally smooth, which allows the computation of a unique velocity field. A theoretical analysis of the conditions under which this computation yields the correct velocity field suggests that the solution is physically plausible. Empirical studies show the predictions of this computation to be consistent with human motion perception.

Recovering Heading for Visually-Guided Navigation

We present a model for recovering the direction of heading of an observer who is moving relative to a scene that may contain self-moving objects. The model builds upon an algorithm proposed by Rieger and Lawton, based on earlier work by Longuet-Higgins and Prazdny. The algorithm uses velocity differences computed in regions of high depth variation to locate the focus of expansion, which indicates the observers heading direction. We relate the behavior of the model to psychophysical observations regarding the ability of human observers to judge heading direction, and show how the model copes with self-moving objects in the environment.

Human heading judgments in the presence of moving objects

When moving toward a stationary scene, people judge their heading quite well from visual information alone. Much experimental and modeling work has been presented to analyze how people judge their heading for stationary scenes. However, in everyday life, we often move through scenes that contain moving objects. Most models have difficulty computing heading when moving objects are in the scene, and few studies have examined how well humans perform in the presence of moving objects. In this study, we tested how well people judge their heading in the presence of moving objects. We found that people perform remarkably well under a variety of conditions. The only condition that affects an observer’s ability to judge heading accurately consists of a large moving object crossing the observer’s path. In this case, the presence of the object causes a small bias in the heading judgments. For objects moving horizontally with respect to the observer, this bias is in the object’s direction of motion. These results present a challenge for computational models.

Incremental rigidity scheme for recovering structure from motion: position-based versus velocity-based formulations

Perceptual studies suggest that the visual system may use a rigidity assumption in its recovery of three-dimensional structure from motion. Ullman [Perception 13, 255 (1984)] recently proposed a computational scheme that uses this assumption to recover the structure of rigid and nonrigid objects in motion. The scheme assumes the input to be discrete positions of elements in motion, under orthographic projection. We present formulations of Ullmans method that use velocity information and perspective projection in the recovery of structure. Theoretical and computer analyses show that the velocity-based formulations provide a rough estimate of structure quickly but are not robust over an extended time. The stable long-term recovery of structure requires disparate views of moving objects.

The Perceptual Buildup of Three-Dimensional Structure from Motion

We present a set of psychophysical experiments that measure the accuracy of perceived three-dimensional (3-D) structure derived from relative motion in the changing two-dimensional image. The experiments are motivated in part by a computational model proposed by Ullman (1984), called theincremental rigidity scheme, in which an accurate 3-D structure is built up incrementally, by considering images of moving objects over an extended time period. Our main conclusions are: First, the human visual system can derive an accurate model of the relative depths of moving points, even in the presence of noise in their image positions; second, the accuracy of the 3-D model improves with time, eventually reaching a plateau; and third, the 3-D structure currently perceived appears to depend on previous 3-D models. Through computer simulations, we relate the results of our psychophysical experiments with the predictions of Ullman’s model.

Recovering Three-Dimensional Structure from Motion with Surface Reconstruction

This paper addresses the computational role that the construction of a complete surface representation may play in the recovery of 3-D structure from motion. We first discuss the need to integrate surface reconstruction with the structure-from-motion process, both on computational and perceptual grounds. We then present a model that combines a feature-based structure-from-motion algorithm with a smooth surface interpolation mechanism. This model allows multiple surfaces to be represented in a given viewing direction, incorporates constraints on surface structure from object boundaries, and segregates image features onto multiple surfaces on the basis of their 2-D image motion. We present the results of computer simulations that relate the qualitative behavior of this model to psychophysical observations. In a companion paper, we discuss further perceptual observations regarding the possible role of surface reconstruction in the human recovery of 3-D structure from motion.