David Marr

Theory of Edge Detection

A theory of edge detection is presented. The analysis proceeds in two parts. (1) Intensity changes, which occur in a natural image over a wide range of scales, are detected separately at different scales. An appropriate filter for this purpose at a given scale is found to be the second derivative of a Gaussian, and it is shown that, provided some simple conditions are satisfied, these primary filters need not be orientation-dependent. Thus, intensity changes at a given scale are best detected by finding the zero values of ∇2G(x, y)* I(x, y) for image I, where G(x, y) is a two-dimensional Gaussian distribution and ∇2 is the Laplacian. The intensity changes thus discovered in each of the channels are then represented by oriented primitives called zero-crossing segments, and evidence is given that this representation is complete. (2) Intensity changes in images arise from surface discontinuities or from reflectance or illumination boundaries, and these all have the property that they are spatially localized. Because of this, the zero-crossing segments from the different channels are not independent, and rules are deduced for combining them into a description of the image. This description is called the raw primal sketch. The theory explains several basic psychophysical findings, and the operation of forming oriented zero-crossing segments from the output of centre-surround ∇2G filters acting on the image forms the basis for a physiological model of simple cells (see Marr & Ullman 1979).

A Theory of Cerebellar Cortex

1. A detailed theory of cerebellar cortex is proposed whose consequence is that the cerebellum learns to perform motor skills. Two forms of input—output relation are described, both consistent with the cortical theory. One is suitable for learning movements (actions), and the other for learning to maintain posture and balance (maintenance reflexes).

A Computational Theory of Human Stereo Vision

An algorithm is proposed for solving the stereoscopic matching problem. The algorithm consists of five steps: (1) Each image is filtered at different orientations with bar masks of four sizes that increase with eccentricity; the equivalent filters are one or two octaves wide. (2) Zero-crossings in the filtered images, which roughly correspond to edges, are localized. Positions of the ends of lines and edges are also found. (3) For each mask orientation and size, matching takes place between pairs of zero-crossings or terminations of the same sign in the two images, for a range of disparities up to about the width of the mask’s central region. (4) Wide masks can control vergence movements, thus causing small masks to come into correspondence. (5) When a correspondence is achieved, it is stored in a dynamic buffer, called the 2½-D sketch. It is shown that this proposal provides a theoretical framework for most existing psychophysical and neurophysiological data about stereopsis. Several critical experimental predictions are also made, for instance about the size of Panum’s area under various conditions. The results of such experiments would tell us whether, for example, co-operativity is necessary for the matching process.

Directional Selectivity and its Use in Early Visual Processing

The construction of directionally selective units, and their use in the processing of visual motion, are considered. The zero crossings of ∇2G(x, y) ∗ I(x, y) are located, as in Marr & Hildreth (1980). That is, the image is filtered through centre-surround receptive fields, and the zero values in the output are found. In addition, the time derivative ∂[∇2G(x, y) ∗ l(x, y)]/∂t is measured at the zero crossings, and serves to constrain the local direction of motion to within 180°. The direction of motion can be determined in a second stage, for example by combining the local constraints. The second part of the paper suggests a specific model of the information processing by the X and Y cells of the retina and lateral geniculate nucleus, and certain classes of cortical simple cells. A number of psychophysical and neurophysiological predictions are derived from the theory.

Artificial Intelligence -- A Personal View

The goal of Artificial Intelligence is to identify and solve tractable information processing problems. In so doing, two types of theory arise. Here, they are labelled Types 1 and 2, and their characteristics are outlined. This discussion creates a more than usually rigorous perspective of the subject, from which past work and future prospects are briefly reviewed.

A theory for cerebral neocortex

It is proposed that the learning of many tasks by the cerebrum is based on using a very few fundamental techniques for organizing information. It is argued that this is made possible by the prevalence in the world of a particular kind of redundancy, which is characterized by a ‘Fundamental Hypothesis’. This hypothesisis used to found a theory of the basic operations which, it is proposed, are carried out by the cerebral neocortex. They involve the use of past experience to form so-called ‘classificatory units’ with which to interpret subsequent experience. Such classificatory units are imagined to be created whenever either something occurs frequently in the brain’s experience, or enough redundancy appears in the form of clusters of slightly differing inputs. A(non-Bayesian) information theoretic account is given of the diagnosis of an input as an instance of an existing classificatory unit, and of the interpretation as such of an incompletely specified input. Neural models are devised to implement the two operations of diagnosis and interpretation, and it is found that the performance of the second is an automatic consequence of the model’s ability to perform the first. The discovery and formation of new classificatory units is discussed within the context of these neural models. It is shown how a climbing fibre input (of the kind described by Cajal) to the correct cell can cause that cell to perform a mountain-climbing operation in an underlying probability space, that will lead it to respond to a class of events for which it is appropriate to code. This is called the ‘spatial recognizer effect’. The structure of the cerebral neocortex is reviewed in the light of the model which the theory establishes. It is found that many elements in the cortex have a natural identification with elements in the model. This enables many predictions, with specified degrees of firmness, to be made concerning the connexions and synapses of the following cortical cells and fibres: Martinotti cells; cerebral granule cells; pyramidal cells of layers III, V and II; short axon cells of all layers, especially I, IV and VI; cerebral climbing fibres and those cells of the cortex which give rise to them; cerebral basket cells; fusiform cells of layers VI and VII. It is shown that if rather little information about the classificatory units to be formed has been coded genetically, it may be necessary to use a technique called codon formation to organize structure in a suitable way to represent a new unit. It is shown that under certain conditions, it is necessary to carry out a part of this organization during sleep. A prediction is made about the effect of sleep on learning of a certain kind.

Analysis of Occluding Contour

Almost nothing can be deduced about a general three-dimensional surface given only its occluding contours in an image, yet contour information is easily and effectively used by us to infer the shape of a surface. Therefore, implicit in the perceptual analysis of occluding contour must lie various assumptions about the viewed surfaces. The assumptions that seem most natural are (a) that the distinction between convex and concave segments reflects real properties of the viewed surface; and (b) that contiguous portions of contour arise from contiguous parts of the viewed surface – i. e. there are no invisible obscuring edges. It is proved that, for smooth surfaces, these assumptions are essentially equivalent to assuming that the viewed surface is a generalized cone. Methods are defined for finding the axis of such a cone, and for segmenting a surface constructed of several cones into its components, whose axes can then be found separately. These methods provide one link between an uninterpreted figure extracted from an image, and the 3-D representation theory of Marr & Nishihara (1977).

Representation and Recognition of the Movements of Shapes

The problems posed by the representation and recognition of the movements of 3-D shapes are analysed. A representation is proposed for the movements of shapes that lie within the scope of the Marr & Nishihara (1978) 3-D model representation of static shapes. The basic problem is how to segment a stream of movement into pieces, each of which can be described separately. The representation proposed here is based upon segmenting a movement at moments when a component axis, e. g. an arm, starts to move relative to its local coordinate frame (here the torso). For example, walking is divided into a segment of the stationary states between each swing of the arms and legs, and the actual motions between the stationary points (relative to the torso, not the ground). This representation is called the state─motion─state (SMS) moving shape representation, and several examples of its application are given.

Cooperative computation of stereo disparity

Perhaps one of the most striking differences between a brain and today’s computers is the amount of “wiring.” In a digital computer the ratio of connections to components is about 3, whereas for the mammalian cortex it lies between 10 and 10,000 (1).

Bandpass channels, zero-crossings, and early visual information processing

Under appropriate conditions zero-crossings of a bandpass signal are very rich in information. The authors examine here the relevance of this result to the early stages of visual information processing, where zero-crossings in the output of independent spatial-frequency-tuned channels may contain sufficient information for much of the subsequent processing.